Mixture of Latent Experts Using Tensor Products
Authors: Zhan Su
Fengran Mo
Prayag Tiwari
Benyou Wang
Qiuchi Li
Jian-Yun Nie
Jakob Grue Simonsen
Venue: Accepted by TMLR ()

Abstract: In multi-task learning, the conventional approach involves training a model on multiple tasks simultaneously. However, the training signals from different tasks can interfere with one another, potentially leading to \textit{negative transfer}.  
To mitigate this, we propose a novel \textit{latent-expert} approach (\texttt{TensorPoly}), that balances parameter efficiency with nuanced routing methods. For \textit{experts}, we reparameterize Low-Rank Adaptation (\texttt{LoRA}) by employing an entangled tensor through the use of tensor product operations and name the resulting approach \texttt{TLoRA}. For \textit{routing function}, we tailor two innovative routing functions according to the granularity: \texttt{TensorPoly-I} which directs to each rank within the entangled tensor while \texttt{TensorPoly-II} offers a finer-grained routing approach targeting each order of the entangled tensor.  The experimental results from the multi-task T0-benchmark demonstrate that: 1) all latent-expert approaches surpass the corresponding dense approaches, highlighting the potential of modular language models to mitigate negative inference in multi-task learning and deliver superior outcomes. 2) \texttt{TensorPoly-I} achieves higher parameter efficiency in adaptation and outperforms other modular LMs, which shows the potential of our approach in multi-task transfer learning \footnote{The code is released: \url{https://github.com/microsoft/mttl}}.

Published in Transactions on Machine Learning Research (10/2024)
Mixture of Latent Experts Using Tensor Products
Zhan Su
zhan, zhan. swddd ku, dk
University of Copenhagen, Denmark
Fengran Mo
University of Montreal, Quebec, Canada
Prayag Tiwari
prayag, Sannyag.üuurlBtier..cm
School of Information Technology, Halmstad University, Sweden
Benyou Wang
wanglengoou@culk.ct..cm
The Chinese University of Hong Kong, Shenzken, China
Qiuchi Li
qiuchi.lidda duddiiik..a dk
University of Copenhagen, Denmark
Jian- Yun Nie
nie wtt.Biio.monttrallcoom ca
University of Montreal, Quebec, Canada
Jakob Grue Simonscn
simonsen Is, dk
University of Copenhagen, Denmark
Reviewed on OpenReview: https: opwp:eviee..........mmmm net/ forum? id= Sgze
Abstract
In multi-task learning, the conventional approach involves training model multiple lasks simultaneously. However, the training signals from different tasks can interfere with one another, potentially leading to negative transfer. To mitigate this, we propose a novel latent-erpert approach (TensorPoly). that balances parameter efficiency with nuanced rout- ing methods. For experts, we reparameterize Low-Rank Adaptation (LoRA) by employing an entangled tensor through the use of tensor product operations and name the resulting ap- proach TLORA. For routing function, we tailor two innovative routing functions according to the gramularity: TensorPoly-I which directs to each rank within the entangled tensor while TensorPoly-II offers finer-grained routing approach targeting each order of the entangled tensor. The experimental results from the multi-task T0-benchmark demonstrate that: 1) all latent-expert approaches surpass the corresponding dense approaches, highlighting the potential modular language models to mitigate negative inference in multi-task learning and deliver superior outcomes. 2) TensorPoly-I achieves higher parameter efficiency in adaptation and outperforms other modular LMs, which shows the potential of approach in multi-task transfer learning
1 Introduction
Recently, the de facto paradigm for natural language understanding (NLU) tasks has centered on leveraging
large language models (LLMs) (He et al.) 2021 that are pre-trained on vast corpus of unlabelled data and subsequently finc-tuned for specific tasks (Qiu et a... 2020 Yo et al. 2021). While this approach has significantly advanced the field, it often requires substantial computational resources and may not efficiently
The code released: https: https://github.ccom http:/ttww..ccc/ccccooaatttttttt

---

Published in Transactions on Machine Learning Research (10/2024)
transfer knowledge across diverse tasks. In addition, fine-tuning tasks independently would lead to nega- live transfer, where the lack of shared information across tasks makes it difficult to achieve compositional generalization Ponti et al., 2023).
Comparison of Across Dense and
69.5
68.3
68.8
69-
SOlA resulss
Paly
68.0
Avi Score
67
66.8
Accuracy
Potsor
66.
66
67.5
85
67-
LoRA
64
66.5
TensorPoly-kz
AdapterSoup
66-
63
62
LoRA
TLORA
Poty
MeR
TemorPoly-l
PersarPolys
Poly
SK
500K SM
50M
Adaptation Parameters
Comparison between the dense models (LoRA, TLoRA) and latent-cxpert approaches (Poly, MHR, TensorPoly-I, TensorPoly-II). Poly/MHDD use LoRA as the modules, TensorPoly-I and Te use TLoRA as the modules. Right: Adaptation parameters across different approaches in the fine-tuning process.
Figure 1: Left:
To address the aforementioned issues, there have been, roughly, two general lines of research. The first line aims to mitigate the computation and memory issue using lightweight alternatives known as parameter- efficient fine-tuning (PEFT), which updates only a small number of extra parameters while keeping most pre-trained parameters frozen (Houlsby et al. 2019b; Li & Liang 2021 Hu et al. 2021). However, these solutions need to train an adapter for each task, which does not take into account the fact that test tasks may require solving different combinations of sub- sub-problems compared training tasks (Vu et al. 2020).
thus failing to achieve compositional generalization (Rosenbaum et al., 2019 Ponti
2021
The second line is to facilitate information sharing across multiple tasks using multi-task learning (MTL) approaches [Caruana 1997 Zhang & Yang 2021 Liu et al.) 2019). which simultaneously train the model on several tasks, allowing it to learn shared representations for all tasks involved. However, MTL necessi- tates access to all training tasks during the training phase, mcaning that incorporating new tasks requires retraining the model from scratch. This requirement significantly increases the computational burden and limits the flexibility of the model to adapt to new tasks efficiently.
A promising approach to address the above issues is the adoption of modular language models PPfeiffer et al. 2023 Ponti et al. 2023; Caccia et al. 2023). where modules are typically implemented as PFFTs for different tasks (Hu et al.
2021 Houlsby et al. 2019a Bach et al. 2022). Information flow is conditionally routed to subset of these modules, which are then aggregated for the given task. This design facilitates the
positive transfer and systematic generalization (Peifferr et al. 2023). Recently, Poly and MHR were designed handle diverse tasks by leveraging different combinations of latent experts (Ponti al. 2023 Caccia let al. 2023). Given [T] tasks, there are only [AC] < [T] experts trained. We treat these [K] experts as latent experts, and each task-specific adapter be obtained by linear combination of latent experts. During both the multi-task pre-trained and fine-tuning, Poly implements adapters with LoRAs and concurrentyy optimizes the LoRA inventory and routing function. MHR partitions the LoRAs into multiple heads and uses a finer-traiedd routing among these heads. However, LoRA adapters are still limited in parameter efficiency, especially with expert libraries involving a huge number of adapters. In addition, we notice that previous approaches only use lincar combinations of experts, which implicitly assume that the given task has a lincar relationship with the expert modules, whereas the relations could be much more complicated in practice. To this end, the question of developing modular language model that balances parameter efficiency with
2

---

Published in Transactions on Machine Learning Research (10/2024)
complex routing methods is critical for advancing the scalability and functionality of multi-task transfer learning.
To answer this question, we devise a new variant Poly model: TensorPoly, which mixes experts using tensor product (Smolensky, 1990). an operation that maps two vectors in spaces V and W to a vector in the tensor product space + W of the two vector spaces, associated with bilinear map x + V @ W ($3.3). This process enables the capture of higher-order interactions and structural relationships between imput spaces (Kye 2023 Tensor product has been successful as strategy for compressing word embeddings Panaah et al. 2019: Gan et al. 2022). To achieve higher parameter-efficintt adapter, we have reparameterized LoRA (Hu et al., 2021] adapters by employing an "entangled" tensor structure, where the high hidden and intermediate dimensions are decomposed into smaller dimensions in tensor product form. Consequently, the training matrix M E Rdxr in LoRA is reparameterized into a finer-grained tensor L E Rxxx V2)x named TLoRA (1.1) This reparameterization allows for more nuanced manipulation of the model's parameters, facilitating a more efficient adapter process for more complex tasks. The entangled tensor configuration introduces two critical hyper-parameters: the tensor rank (R) and the tensor order Leveragin these parameters, we develop two distinct routing functions designed to select modules on varying levels of granularity. As depicted in Figure 3 TensorPoly-I employs routing mechanism that assigns distribution scores to different tensor ranks, facilitating the selection of modules based on their rank gramularity. Further advancing this concept, we propose more refined routing function, TensorPoly-II which largets even finer-grained tensors activated modules. Each module is associated with specific order of the entangled tensor. Once modules are selected via the routing function, they are aggregated through a tensor product operation, enabling a sophisticated and dynamic assembly of modular skills.
To evaluate the effectiveness and parameter efficiency of our approach, we apply our methods against a series of competitive baselines on TO (Sanh al. 2021), a widely used benchmark in multi-task transfer learning covering high variety of language understanding tasks. Our experiments reveal several key insights: i) Modular language models such as Poly, MHR and TensorPoly frameworks consistently outperform tradi- tional PFFT approaches LoRA and TLoRA (Figure [], underscoring the effectiveness of modular LLMss in facilitating positive transfer across multi-task environments. iij TensorPoly-I demonstrates competi- live results against Poly and MHR, while simullaneously achieving higher parameter elficiency in adaptation. This efficiency gain highlights the benefits of our tensorized module approach in achieving high performance with lower parameter overhead. Wi comparative analysis between TensorPoly-I and TensorPoly-II indicates that the latter's finer-grained routing mechanism does not contribute to improved performance. This outcome suggests that while granularity in module selection is valuable (Caccia et. al., 2023). there is complexity threshold beyond which additional granularity may not yield further benefits.
In summary, our contributions are as follows:
We propose TLoRA that achieves competitive results while only using O(N xrx [ VaT x Rt) training parameters compared to LoRA Old x r), highlighting our approach's high parameter efficiency.
We propose novel modular LM TensorPoly, which balances parameter efficiency with tensor prod- uct routing. The evaluation results on TO benchmark demonstrate that TensorPoly-I can surpass other strong modular LMs, underscoring the critical role of tensor product routing in scenarios involving multi-task transfer learning.
By only fine-tuning the routing function. TensorPoly-I can surpass the TLoRA with only 8.6k training parameters, achieving extremely parameter-efficient fine-tuning.
2
Related Work
Parameter-efficient Fine-tuning
Parameter-officient fino-tuning (PEFT) methods facilitate efficient adaptation of LLMs without updating all the training parameters, thereby reducing the memory and computation cost (Zhang et al., (023b). One kind of PEFT approach focuses on adding modules to LLMs, and only these small modules will be trained while
3

---

Published in Transactions on Machine Learning Research (10/2024)
the backbone model is kept frozen and shared across tasks. For example, Adapter Tuning inserts small neural modules (adapters) between the layers of the basic model (Houlsby et al. 2019a), whereas Prefix Tuning and Prompt Tuning add tunable vectors to the imput or hidden layer of the base model (Li & Liang] 2021 Lester et al. 2021). Another kind of rescarch model is the incremental update of the pre-trained weights in a parametter-eficient way, without modifying the model architecture. Bitfit fixes all training parameters and only fine-tunes the additive bias term (Zaeen et al.) 2021). Diff Pruning learns a task-specific "diff vector" that extends the original pre-trained parameters. As the number of tasks increases, Diff' Pruning only requires storing small diff vector for each task (Guo al. 2020). The seminal paper Hu et al. (2021) proposes method named LoRA that parameterizes incremental weights A as a low-rank matrix by the product of the down projector matrix and up projector matrix. LoRA achieves comparable or even superior performance to full fine-tuning (Hu et al. 2021p. [Zhang et. al. (2023a) demonstrates that weight matrices in the top layers are more important than those in the bottom layers. They propose Adaptive Low Rank Adaptation (AdaLoRA), a new method that dynamically allocates the parameter budget among weight matrices during LoRA-like fine-tuning (Zhang et al.) 2023a AdaLoRA adjusts the rank of incremental matrices for different layers. Liu et al. (2024) introduces new approach DoRA to investigate the inherent differences between full fine-tuning and LoRA. Baziotis et al. (002)) finds hyper-adapters are more parameter efficient than regular adapters, reaching the same performance with up to 12 times less parameters.
Compared to the works above, our approach TLoRA takes a novel perspective by reparameterzing LoRA using tensor products and utilizing finer-grained tensors as modules.
Multi-task Learning
The key to MTL is information sharing across tasks. For this purpose, AdapterSoup Chronopoulou et al. (2023) trains each adapter for each domain, and performs weight-space averaging of adapters trained on different domains. Huang et al. (2023) introduce LoRAhub to aggregate the LoRA modules trained on diverse tasks. They first train & group of LoRA modules that are specialized in each task, then randomly select a subset of modules, and finally learn a set weights to combine these LoRA models using gradient-free optimization. AdapterFusion (Pfeiffer et al.) 2020) proposes two-stage algorithm that leverages knowledge from multiple tasks. Similarly to LoRAhub, group of task-specific adapters learn to encapsulate the lask-specific information, and in the second stage, fusion layer combines the trained adapters. Ponti ett al. (023)) introduces a variable-size module routing mechanism, Poly, based on the assumption that each task correlates with a specific subset of latent skills drawn from a comprehensive inventory of modules. Building upon Poly. (Caccia et al. 2023) introduces a finer- grainne multi-head routing function MHR where the experimental findings underscore the significance of the routing function during the pre-training phase. Mfxturee-f-expert (MoE) methods such as TaskMoE' Kudugunta et al. (2021) learn a routing function that allocates modules to tasks end-to-end.
We propose TensorPoly-I and TensorPoly-II, two novel routing mechanisms based on TLoRA. TensorPoly-I is a variant of Poly using TLoRA as the modules. In this sctting, each rank of the entangled tensor corresponds to a separale expert. TensorPoly-II is a finer-grained routing function largeting each order of the entangled tensor.
Expert Merging
Once the experts are activated in the forward pass, we need to aggregate their outputs. There is increasing focus on aggregating adapters from different domains through expert merging. The simplest operation of merging is averaging the weights of different experts, where the weight of each expert is set according to the
routing probability generated by the router (MeMahan et. al. 2017 Choshen et al. 2022: Matena & Raffel
2022
Chronopoulou et al. 2023 Huang et al. 2023 Muqeeth et al. 2023; OStapenko et al. 2023 2024).
Poly Ponti et al. (2023) uses the latent experts and integrates the experts by averaging the weights. MHR
Caccia et al. (2023) partitioned the LoRA experts into different heads, which are eventually concatenated to obtain merged expert.
We devise two lensor product routing functions TensorPoly-I and TensorPoly-II. Once each expert is activated, routing function will aggregate the expert weights as an entangled tensor.
4

---

Published in Transactions on Machine Learning Research (10/2024)
experts
Share expert
experts
Task Data
Task dataset
Task dateset
routing
Figure 2: Compare with three training paradigms in multi-task transfer learning, Left is the private training, for each task, we train the corresponding expert individually. Middle is the shared version, for all the tasks, we train an expert continually, and as a result, we only get one expert. Right is the latent experts model, for the tasks, we train subset of "latent" experts, so each corresponding expert can be seen as a linear combination of these latent experts.
3 Background
In multiple transfer learning tasks, we define a set of tasks as T= (T1...., ? This set is divided into two subsets train Tirain and test Trest- The goal of multi-task transfer learning is to apply the knowledge from the training tasks Terain to the test tasks within Trest. This process involves two main phases. Building top of foundation model, the first phase consists of multi-task pre-training using the dalaset in tasks Terain- The second consists of few-shot adaptation, where the learned adapters are fine-tuned independently on each test task Trest- We follow the procedure from RRaffel et al. 2020 and formulate each task as a lext-to-text problem.
3.1 Module: LoRA
Lora is recently proposed adapter architecture that achieves a competitive balance between performance and parameter efficiency (Hu et al. 2021 Mahabadi et al., 2021). For each linear transformation corresponding to the query (q), key (k), value (v), and output (o) of the self-attention layers, LoRA modifics the base model parameters as follows:
Wox + 8 A(B) T
(LoRA)
where Wo the (frozen) weights of the pre-trained model (e.g. T5 (Raffel al., 2020)) ). A, Rdxr are low-rank learnable parameters and 1 is tunable scalar hyperparameter. We schematize LoRA in Figure
3.2 Polytropon (Poly): Mixture of Latent Experts with Linear Combination
Poly/ MHR addresses the multi-task problem by softly sharing latent experts across tasks. Each Poly layer contains 1) inventory of latent experts M (oi...., Φm) with [M] < [T]: 2) routing function r(-) that chooses which subset of the experts to combine for each task. Each latent expert corresponds to LoRA adapter, where Φi are its associated paramcters A(1), B()) [gdxr, r(-) is implemented as task- = Zr,: E [RMM is a routing vector of task Tr, with cell Zrj being the probability logits of using module 6, for task T, in the current layer. Differently from mixture- of-experts (Fodus et al., 2022), which perform token-level top-k routing, Z converges to a binary matrix, defining soft partition over modules. This is achieved by using a siimel-sigmoid distribution (Jang al. 2017) during training, with 2 Gumbel(7,, (7..))- At each forward pass, Poly be defined
module routing matrix Z E R//|x|MI
5

---

Published in Transactions on Machine Learning Research (10/2024)
A Ea B E a
(Poly)
and A B(, AT, Bd RR We normalize the mixing coefficients Zrd for each task where a; E, Z.,
to ensure that the number of active modules does not affect the norm of AT, Br. Overall, this approach enables different subsets of modules to be activated for the current layer and combined in a task-specific way. Following [LoRA] the output of the Poly layer is added to the output of the original layer of the frozen backbone: W6x + sAT(B") TZ.)...
3.3 Tensor, Tensor Product, Entangled Tensor
Tensor. The tensor A is multi-dimensional array of elements (called components) of R. each being denoted by its integer coordinates in the array; c.g., for two-dimensional array, the component at position i.j E N is denoted Aaj- The order of a tensor is how many indices it has (e.g., a vector is a first-order tensor, a matrix M is a second-order tensor, etc.).
The tensor product V W of two vector spaces V and W is a vector space to which is associated a bilinear map V x W + V & that maps a pair of vectors (u, w), v E V, w E W to a vector in V & W, denoted as v 6 w. We can create tensor product spaces by more than one application of tensor product, U V & W, with arbitrary bracketing since the tensor product is associative. The tensor product space of N vector spaces in such form is said to have tensor order of N.
Tensor Product.
N
H,y = H41 2 H2 8.. 8 Hx
(1)
j=1
Entangled Tensor. The vectors in N-order tensor product space @ Hi in the form /f 1/ E Hj, and referred to simple lensors. Vectors represented as the sum of multiple simple tensors are called enlangled tensors:
Ujk
Σ
Vik
U2A 3
@ UNk
(2)
where lensor rank R is the smallest number of simple tensors that sum up v. For example, &es S is a tensor of rank
v2
3.4 Tensorized Vector using Entangled Tensor
Any training parameter Rd can be expressed as entangled tensor of rank R and order N by:
Ujk. Here, Uik R yielding resultant vector of dimension p g.N. If there exists a value of
N such that d= p. the storage requirements are efficiently managed, consuming only RNg 0 (R logg/q) parameters. More generally, we take the smallest possible value g satisfying gd > d, i.e. g = Vd), and cut off the excess part of the generated vector to produce v E Rdd
For example, we would like to compress a 512-dim vector (d = 512) with tensor of order N and rank = Then we can the dimension of each small vectors as q Vd = In this way, the vector can be represented with total number of RNg 48 parameters, leading to significant, parameter reduction.

---

Published in Transactions on Machine Learning Research (10/2024)
TensorPoly-I
TensorPoly-Il
3x5x5
3x5xS
3x5
A,A
3x25x5
3x625K5
625X5 Jown
Σ
625
Σ
W diown
A2A
R3A
3x25xx5
LoRA
Z IRlT
Σ
LoRA
A4X
Map
A.p
Plaues
Wipp
Z IR RR
Figure 3: TensorPoly-I and TensorPoly-II. We illustrate how to reparameterize the LoRA matrix R625x5 with 4 tensor A E 3 In this case, the tensor rank R = tensor order N = 4. For TensorPoly-I. the routing function Z is designed to select which rank of the entangled tensor is activated for a given task. Conversely, TensorPoly-In introduces a more granular control by selecting tensor rank and tensor order.
4 Methods: TensorPoly
TensorPoly-I and TensorPoly-II based on the TLoRA module (5 4.1 TensorPoly-I employs routing mechanism that assigns distribution scores to tensor ranks while TensorPoly-II assigns distribution scores to the finer-grained tensor order.
We propose two variant TensorPoly models:
4.2
4.1
TLoRA
To achieve higher paramcter efficiency, reparamcterize the LoRA using the tensor product, which is widely used in compressing the word embedding (Panahi et al. 2019 Gan et al. 2022). Essentially, the low- rank matrices A and B are further reparameterized to entangled tensors of rank R and order N, following For a given input x, TLoRA modifies the projection output / as:
3.3
Whx + s. sA((B
N
R
N
T
(TLoRA)
Whe s
Σ
Aik
B
k=
where R is the lensor rank, N is the order of entangled Lensor respectively. A,B E RNxxxx Vd)xR) are fourth-order tensors, which refer to the training parameters of the modules. Ai,k, BLi R V7T) are indices of fourth-order lensors, referring to specific rank and order in entangled lensor, depicted in Fig
4.2
TensorPoly: Mixture of Latent Experts using Tensor Products
Latent-capert approaches have been proven effective in the few-shot multi-task transfer learning (Pntt al. 2023 Caccia et 20.3) Poly merges the latent experts by averaging the weight. MHR partitioned the LoRA into several heads and use piecewise linear aggregation (i.e., linear within disjoint intervals). Instead of using a linear combination, we propose TensorPoly models that incorporate TLoRA as the core module. Specifically, we propose a TensorPoly-I and TensorPoly-II according the routing gramularity. In cach layer, TensorPoly-I use a routing matrix Z E R determine which rank within the entangled tensor be activated for given task. In this case, each rank A& corresponds to "expert" in the forward

---

Published in Transactions on Machine Learning Research (10/2024)
pass. Upon activation of the selected rank, the model computes linear combination of the selected rank, weighted by lactor oz
A
AI, AI, A2, 60... 0 AN,K
i=1
(TensorPoly-)
AT
ak = an +... ORAR
1=1
merge the rank ft
The expert As E xxxx Val is a third-order tensor. The same for Σ an where 0 Σ, Rrs
Contrast. to the TensorPoly-I with routing matrix, the routing for TensorPoly-II is conceptualized third-order tensor Z R(T|xNxR which offers finer-grained level in dirccting the model's focus across different ranks and orders of the lensor space. This sophisticated routing mechanism facilitates the selection of finer-grained tensor elements, which are then aggregated through a tensor product operation.
R A a, = a1,1A,, a
k=1
merge the rank
(TensorPoly-II)
A
A A
Ar
merge the order N
Each unit of Z is 0s,k E,
Brick &rgga
which will be routed to the specific order and rank in the entangled tensor.
In this case, "expert" Ai, E B Va)
corresponds to each order in the entangled tensor.
5 Experiments
To evaluate the effectiveness of our approaches, perform experiments multi-task transfer learning datascts TO benchmark (Sanh et al. 2021), which is widely used in few-shot generalization approaches. In addition, diverse array of lasks in To benchmark can help us lest. the generalization ability across dif- ferent tasks. We conduct a comparative analysis between routing approaches (Poly, MHR, TensorPoly-I, TensorPoly-II) and their corresponding "single-expert" (without routing function) version as detailed in 5.2 We also investigate how parameter efficiency and effectiveness are influenced by hyper- parameters rank R and order N in entangled tensor (.5.3.1).
5.1 Datasets and Evaluation
Datasets To evaluate the generalization capabilities of our models, we adopt the same benchmarking strategy as (Liu al. 2022), utilizing subset of tasks designated held-out from the multitask training, This benchmark encompasses diverse array of tasks, including sentence completion (COPA (Roemmele et al. 2011h. H-SWAG (ellers et al.) 2019) and Story Cloze (Sharma et al. 2018) datasets), natural lan- guage inference (ANLI (Nic et al. 2019). CB DD Marneffe et al. 2019) and RTE (Dagan et al. 2005)), ) coreference resolution (WSC (Levesque el al. 2012). Winogrande (Skkauuchi el. al. 2021)), and word sense disambiguation (WIC (Pilehvar Camacho- Collados 2018)). For each task, our evaluation strategy in- volves constructing sets of five few-shot training examples, which are generated by sampling subsets from each dataset using different seeds. We then report the median performance. should be noted that the prompt examples from each dataset using the prompt templates from P3 (Bach et al. 2022
8

---

Published in Transactions on Machine Learning Research (10/2024)
Natural Language Inference
Sentence Completion H-SWAG
Co-reference
WSD
Model
ACC
RTE
ANLII
ANLI2
ANLI3
COPA
Story
WSC
Wino
WiC
Baselines (w/o pre-train)
FullFT
79.8
46.6
41.3
40.0
$1.0
93.8
65.4
56.5
57.7
63.3
BitFit (with LN)
72.2
57.1
36.5
35.3
36.6
75.0
29.5
88.6
61.5
56.6
51.7
54.6
LayerNorm
71.8
57.1
36.5
35.1
36.3
76.0
29.6
88.7
63.5
49.4
52.2
54.2
Adapter
76.2
87.5
45.1
40.4
35.3
84.0
41.9
91.7
65.4
54.7
55.5
61.6
Compacter
75.8
82.1
40.8
37.4
35.8
84.0
93.5
64.4
55.5
55.2
61.0
Compacter
76.9
82.1
41.7
38.3
36.9
86.0
46.3
93.5
65.4
55.1
54.1
61.5
Prompt(10) (10)
52.7
66.1
34.2
33.5
33.5
67 0
29.9
84.2
54.8
51.9
51.6
50.9
Prompt(100)
48.0
53.6
33.4
33.8
33.3
60.0
26.8
74.0
60.6
51.1
50.0
47.7
Prefix tuning
68.6
84.0
43.3
37.5
36.5
71.0
42.1
90.2
52.0
54.2
57.8
FishMask (0.2%)
76.9
83.9
43.7
39.7
37.2
$2.0
44.1
94.2
63.5
54.5
52.5
61.1
Fish (0.02%)
75.5
76.8
39.9
38.1
36.2
84.0
38.2
93.6
61.5
53.9
53.5
59.0
SAID
69.0
80.4
40.4
35.4
35.5
77.0
36.7
89.3
61.5
52.7
55.0
57.5
LoRA
78.3
85.7
45.1
41.0
39.5
88.0
47.1
93.6
60.6
56.8
55.2
62.8
(IA)*
78.0
87.5
48.6
40.8
40.8
87.0
49.4
94.7
68.3
59.8
56.0
64.6
w/ pre- pre-train
LoRA
81.9
89.3
41.2
40.3
41.3
93.7
59.8
96.2
66.0
67.9
56.8
66.8
(IA)*
82.2
89.9
45.8
41.6
41.2
91.7
53.4
94.2
70.8
63.3
53.9
66.2
TLoRA
80.7
90.5
39.9
40.9
41.2
93.0
95.3
66.3
67.4
57.3
66.1
Poly
84.7
46.0
42.8
42.7
93.0
96.6
70.1
59.9
68.8
MHR
85.2
90.5
44.7
42.3
42.8
94.7
63.3
96.7
70.5
70.6
59.8
69.2
TensorPoly-I
85.2
91.7
45.0
42.5
42.5
96.7
63.1
96.6
68.6
69.8
60.6
69.3
TenserPoly-II 84.7
44.4
41.0
42.0
94.3
95.7
67.6
59.9
68.0
Table 1: Results on the TO few-shot benchmark. All the results in our implementation are the median score of 3 random seods [0, 1024, 42]. For all the bascline scores, we quote the results from Liu ct al. (0022 The value bold is the best score.
Evaluation For the evaluation of our models, we employ the rank classification methodology as outlined by the Liu et al. (2022 study, This approach involves ranking the model's log- bog-prbbabilitees for all possible label strings associated with each task. The model's prediction deemed correct the label string with the highest log-pro Dog-probability ranking corresponds the correct answer. This method allows for nuancod asscssment of the model's predictive accuracy by examining its ability prioritize the correct label over others based their calculated log- probabilities, offering precise measure of its understanding and processing of the task at hand.
5.2 Baselines
Method
Pre- Pre-Tranning d²
Fine- Fine-Tuning d' 2dr VdRR
FullFT
LoRA
2dr 2Nr VdR
TLoRA
2Nr
Poly
2dr S + |T]
2dr|S| + S
TensorPoly-I
2Nr[ V/d R R
2Nr[ Vdl R R
TensorPoly-II
2Nr VdJR | RN
2Nr[ Vd R RN
Table 2: Number of parameters (per layer) used for each method. d is the input and output dimension of the training parameters. We assume they are identical. is the rank in the LoRA, where < d. N and R are the order and rank of entangled tensors respectively. S is the number of modules in Poly.
In our comparative analysis, we initially set the benchmark by evaluating the performance of the TLoRA model against the traditional full fine-tuning approach, referred to FullFT. To facilitate fair comparison

---

Published in Transactions on Machine Learning Research (10/2024)
with baseline methodologies, we have chosen the TO-3B model, consistent with the approach described in the TA3 paper by Liu et al. (2022) In the FullFT scenario, we do not freeze any parameters of the pre- trained model, nor do we insert any adapters, allowing for a comprehensive update of the model's parameters during fine-tuning. Subsequently, we contrast our method against a suite of established parameter- fine-tuning (PEFT) baselines to study the efficiency and effectiveness of each in lerms of training parameter utilization. These baselines include: Adapter, as introduced by Houlsby inserting trainable layers while keeping the pre-trained model's parameters fixed; BitFit by Zaken et al.)
al. (2019b), which involves
(2021), which only fine-tunes the bias terms within the model; LoRA proposed by Hu et al. (2021), adjusting the low-rank adaptations of the weight matrices; Compacter and +ompacter++ by Karimi Mahabadi et al. (2021), which extend the adapter methodology with compact and efficient training strategies; Prompt tuning (Lester et al. 2021) and Prefix luning Li & Liang (2021) add some tunable vectors to the input or hidden layer of the base model; FishMask by [Sung et al. (2021), identifying and training a subset of parameters; Intrinsic SAID as described by Aghajanyyn et al. (2020), focusing on intrinsic sparse activations; and IA3 (Liu al. 2022), emphasizing adaptability and elliciency.
We perform comparative analysis between routing approaches TensorPoly, Poly Poly/MHR /HR and dense models without routing, specifically TLoRA and LoRA. This comparison aims to evaluate the impact of routing techniques on model performance and efficiency. By contrasting these models, we seek to evaluate how the dynamic allocation of tasks to specific experts in TensorPoly and Poly/MHH compares to the dense models.
5.3 Results
Tab. D presents the mean downstream accuracy for 11 held-out tasks in the TO benchmark. We reported most of the results from (Liu et al. 2022). When evaluating the performance of various PEFT approaches against single-cxpert performances, it is observed that many PEFT strategies achieve similar outcomes while utilizing a significantly smaller subset of training parametens compared to the FullFT method. Furthermore, the LoRA method within our training framework further illustrates the efficiency of these methods. Specifically, TLoRA achieves a competitive score of 66.1, closely trailing the original LoRA's score of 66.8, while requiring only fewer training parameters used by LoRA. This demonstrates that TLoRA not only matches the effectiveness of LoRA in terms of performance but also achieves higher parameter efficiency.
In our analysis, we initially contrast the modular model against the dense model, followed by a comparison of routing- based approaches with single adapter strategy. Within this context, we utilize both LoRA and TLoRA as baselines for these routing techniques. According to the results presented in Tab. D the Poly model demonstrates superior performance over LoRA by a margin of 2.0 points. Moreover, TensorPoly exhibits an improvement of 3.2 points over the base TLoRA model, underscoring the superiority of routing in enhancing multi-task generalization within multi-task transfer learning.
When evaluating the various modular models, TensorPoly-I stands out by not only par with the recent stale-of-the-art achievements but also by oulperforming TensorPoly-II, despite the latler employing more granular routing function. This finding is particularly noteworthy, as it suggests that the increased specificity of the routing function in TensorPoly-II does not necossarily translate to superior performance. We will discuss this in Section 5.4
5.3.1 Rank and Order Analysis
For TensorPoly, rank R and order N correlate with the number of experts. Tab. 2 illustrates how the tensor rank R and the tensor order N affect the training and adaptation parameters. As illustrated in Figure 4 (left), when the tensor rank is set to 1, it corresponds to the original TLoRA configuration. An increase in lensor rank necessitates a larger number of training parameters. Consequently, there is a notable enhancement in performance. In addition, we investigate if the rank will give more capacity to the model. As depicted in Figure (right), as the increase of the rank, we can get a lower validation loss score during the multi-task pre-training. This progression underscores the direct correlation between the tensor rank and the capability of the model.
10

---

Published in Transactions on Machine Learning Research (10/2024)
60.14
0.86
TLORA
68.70 70
TemoorPPlyy (0
TemsorP (12-00
Aborray
07
Validdtion
0.82
0.78-
68.5
06
0.78
06
500
1000
1500
2000
pretraining steps
Figure 4: Rank analysis in the TensorPoly-I, Left denotes the average accuracy over 11 held-out tasks according to different rank. Right is the validation loss in the multi-task pre-training process.
The tensor order N, correlates with the granularity of training parameters (experts) (44.2) a tensor of order yields finer-grained module compared to of order 2. Our research examines the impact of varying the tensor orders performance outcomes. For simplicity, and 4. Our results demonstrate that tensor of order 2 oulperforms one of order 4 by margin of 1.7 for the TenSorPoly-I. For TenSorPoly-II, an order-2 tensor exceeds the performance of an order-4 tensor by 4.8, suggesting that a balance must be struck between the gramularity of experts and its efficacy. Although higher-order lensors may conserve training parameters, this comes at the cost of diminished performance.
constrain
analysis to tensors of order 2
Model
Multi-Tase Params
Adaptation Params
ACC
LoRA
2.2M
2.2M
66.8
TLoRA
2
1.4M
1.4M
66.1
Poly
17M
2.2M
68.8
TensorPoly-I
8
2
12.2M
L.4M
68.7
TensorPoly-I
20
27.8M
1.4M
69.3
TensorPoly-I
8
4
4.3M
1.4M
66.9
TensorPoly-II
2
13.3M
14M
67.9
TensorPoly-II
20
2
33.2M
1.4M
68.0
TensorPoly-II
4
7.6M
1.4M
63.1
Table 3: Rank and Order analysis. The Adaptation parameter is the number of parameters required to learn new downstream task. Multi-Task params is the number of additional parameters that must be conserved after multi-task pre-training to enable transfer to downstream task.
5.3.2 Routing Analysis
Routing Only In the paper Caccia et al. 2023). the MHR study demonstrates that fine-tuning solely the routing function can yield competitive outcomes. This insight provides valuable perspective for our investigation into various routing stratogics within the TensorPoly framework. In line with this approach, we focus exclusively on fine-tuning the routing function during the few-shot adaptation process, indicated by the notation This methodological choice allows to isolate the impact of the optimization of the routing function on the overall performance of the TensorPoly model, thereby offering clearer understanding of how dynamic routing contributes to the adaptability and efficiency of the model in few-shot learning scenarios.
As detailed in Tab. 4 an initial observation reveals that the routing parameters necessitate a small number of training parameters, achieving extreme parameter efficiency. TensorPoly-I-2 achieves accuracy score of 66.3 while only using 8.6k adaptation paraneters. TensorPoly-II- achieves 65.4 with 17.3k parameters. MHR-z achieve the 68.3 with 220K adaptation parameters. The results indicate that TensorPoly-I-z
11

---

Published in Transactions on Machine Learning Research (10/2024)
Model
modules
routing
Adaptation Params
ACC
LoRA
2.2M
66.8
TLoRA
1.4M
66.1
Modular LMs
Poly-ı
2.2M
68.8
MHR-/
2.2M
68.6
TensorPoly-I-u
1.4M
69.0
Ten-orPoly-II-j1
1.4M
68.2
Poly-:
X
3.5k
65.4
MHR-z
X
220K
68.3
TemsorPoly-I-1
8.6k
66.3
TemsorPoly-II-2
17.3k
65.4
Poly
2.2M
68.8
MHR
2.2M
69.2
TensorPoly-I
1.4M
69.3
TensorPoly-11
1.4M
68.0
We compare several fine-tuning approaches. -H represents we only fine-tune the modules. -2 means only fine-tune the routing functions. We set the order N 2 for this comparison.
Table 4:
achieve competitive results LoRA with higher parameter efficiency. Notably, the average accuracy of TemsorPoly-I-2, and TensorPoly-II-z lag behind their counterparts where both the modules and the routing function undergo fine-tuning. This disparity highlights the critical role that modules play in the fine-tuning process for TensorPoly.
Modules Only To verify whether the routing is important in few-shot fine-tuning, we discard the routing function and average the pre-trained modules in the fine-tuning process, indicated by the notation -y. The result in Tab. shows that there is only a slight decrease for MHR-p and TensorPoly-I-A compared to their counterparts where both the modules and the routing function are fine-tuned. This result is consistent with the conclusion in Caccia al. (2023).
5.4 Discussion
5.4.1
Investigate more about the latent expert approach.
Since we add more training parameters (we use more experts), it is necessary to investigate if the improvement is caused by adding more training parameters. To investigate this, In Caccia al. (2023). they assign binary module allocation to each data point in minibatch, disregarding task information. At test time, the learned modules are averaged into one single module. This approach is named random-/4, the random-ju performs similar to single LoRA, which proved that the "routing' function is important for latent-expert approach in multi-lask transfer learning.
5.4.2 Why TensorPoly-II underperform the TensorPoly-I7
Finer-graines routing has been shown to be effective in multi-head routing function (Caccia et al.) 2023 However, our experimental results underscore that the finer-grained lensor product routing did not contribute to improved final performance our models. This discrepancy prompts a future line of inquiry: we plan explore whether there exist specific benchmarks or conditions under which the tensor-product interaction could demonstrate its purported benefits. Identifying such scenarios will be crucial harnessing the potential advantages of tensor product routing in modular language models.
12

---

Published in Transactions on Machine Learning Research (10/2024)
MHR(heads=64)
0.86
Poly(S=20)
Lora
0.84
TLoRA
TensorPoly-((R220)
Validation Loss
0.82
Tensorpoly-I(R=20)
0.8
0.76
0.74
1000
1500
2000
Multi-task pretraining steps
Figure 5: Validation loss compare across different models.
In the current methodology, each tensor-based module not specialized towards any particular domain. the future, we intend to explore a more tailored training strategy. This will involve dedicating each tensor a specific domain and subsequently aggregating these domain-specialized tensors using tensor product operations. Our objective is assess whether this domsin-specific aggregation approach can yield superior generalization capabilities.
6 Conclusion
We introduce a novel modular language model named TensorPoly. This model incorporates tensorized mod- ules, specifically TLoRA, to significantly reduce the number of training parameters required by the traditional LoRA approach. We employ two distinct strategies for aggregating activated modules: TensorPoly-I directs to each rank and more finely routing, named TensorPoly-II targets each order of the tensor. Our evalua- tion across various multi-task learning scenarios reveals that modular language models, such as TensorPoly, surpass the performance of single-adapter models. This underscores the importance of sharing task infor- mation through a routing function in multi-task learning contexts. Notably, TensorPoly-I achieves state- of-the-art results, highlighting the effectiveness of the TensorPoly framework. However, TensorPoly-II does not outperform TensorPoly-I our experimental scttings, suggesting areas for further invostigation in future research.
Broader Impact Statement
The research reported in this paper proposes a novel type language model, and primarily a theoreti- cal contribution accompanicd by experiments to show the practical usefulness on the model, especially mitigation of negative inference in multi-task learning. Thus, the impact is primarily in generic modelling and empirical performance in situations where existing approaches (e.g., LoRAs) used, but does not per create new scenarios, raise concerns, ethical or otherwise, beyond those already present for LoRAs.
Acknowledgments
We acknowledge the support of Alessandro Sordoni for optimizing the code and discussion.
13

---

Published in Transactions on Machine Learning Research (10/2024)
References
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. ar preprint or Xrv::2012. 12..1132 13255, 2020.
Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. Promptsource: An integrated development environment and repository for natural language prompts. arXiv preprint arXm::202.0112 2022.
Christos Baziotis, Mikel Artetxe, James Cross, and Shruti Bhosale. Multilingual machine translation with hyper-adapters. Xrxiv preprint or #Xxu:2205.10833 2022.
Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Percira, Nicolas Le Roux, and Alessandro Sordoni. Multi- head adapter routing for tross-task generalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Rich Caruana. Multitask learning. Machine learning, 28:41-75, 1997.
Leshern Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pre- training. arXiv preprint arXiv:2204.033 030044, 2022.
Alexandra Chronopoulon, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprinl 2002200000702, 2023.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177 190. Springer, 2005.
CatherCet De Marneffe, Mandy Simons, and Judith Tonhauscr. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeulung, volume 23, pp. 107 124, 2019.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1 39, 2022. URL http://jmlr... httpp//p...pppppp/////////////-.
html
Guobing Gan, Peng Zhang, Sunzhu Li, Xiuqing Lu, and Benyou Wang. Morphte: Injecting morphology in
tensorized embeddings. Advances in Newral Information Processing Systems, 35:33186-333 86-33200, 2022.
Demi Guo, Alexander M Rush, and Yoon Kim. Parfmeter-efficient transfer learning with diff pruming, arXiv preprint 0tt:::0022074633 2020.
Junxian Hc, Chunting Zhou, Xuczhe Ma, Taylor Borg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficies transfer learning for nlp. In International Conference on Machine Learning, pp. 2790 2799. PMLR, 2019a.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. farameter-cfficiee transfer learning for nlp. In International Conference Machine Learning, pp. 2790-22 2799. PMLR, 2019b.
Fdward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. ai preprint or orXi::2307.1 13269,, 2023.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-softmax. In In-
lernational Conference on Learning Represenialions, 2017. URL htttp:://openteviee..ett// TkE3y85ee
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hyper- complex adapter layers. Advances in Neural Information Processing Systems, 34:1022-1035, 2021.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of-experts for efficient inference. arXiv preprint arXiv:2110.037// 2021.
Seung-Hyeok Kye. Compositions and lensor products of linear maps bel ween matrix algebras. Linear Algebra and its Applications, 658:283-308 309, 2023.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint wwxx::1244.80000 2021.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing contimuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
Haokun Liu, Derek Tam, Mohammed Muqecth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A parameter-effiieent Raffel. Few-shot parameter- efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1999 1965, 2022.
Shih- Yang Liu, Chien- Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint 002.02240..00358 2024.
Shin-Yang
Xiaodong Liu, Pengeheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXto:1901.11000 2019.
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-officient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint ar Xrx::22006. 01.:2200.04448,, 2021.
Michael S Matena and Colin A Raffel. Merging models with weighteweighted averaging. Advances in Neural Information Processing Systems, 35:17703-17716, 2022.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Commmnication-eficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282. PMLR, 2017.
Mohammod Muqecth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing. arXiv preprinl crXiv:2306.037/4 2023.
Yixin Nic, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston and Douwe Kiela. Adversarial nli:
A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14555 2019.
Oleksiy Oslapeuko, Lucas Caccia, Zhan Su, Nicolas Le Roux, Laurent Charlin, and Alessandro Sordoni. A case study of instruction tuning with mixture of parameter-effiiient experts. In NewrIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti, Laurent Charlin, Nicolas Le Roux, Matheus Pereira, Lucas Caccia, and Alessandro Sordoni. Towards modular llms by building and reusing library of loras. arXiv preprint 10X.1:2405.11157, 2024.
Aliakbar Panahi, Seyran Sacedi, and Tom Arodz. word2ket: Space-efficient word embeddings inspired by quantum entanglement. Xix preprint 1tt::1111.49997 2019.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprinl arXiv:2005002277 2020.
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. Modular deep learning. arXiv preprint arXiv:2302.1133 2023.
Mohammad Taher Pilchvar and Jose Camacho-Colladoss Wie: the word-in-context datasct for evaluating context-sensitive meaning representations. arXiv preprint Xr::118088009 2018.
Edoardo Ponti. Inductive Bias and Modular Design for Sumple-Efficien Neural Language Learning. PhD thesis, University of Cambridge, 2021.
Edoardo Maria Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining parameter-efficint modules for task-level generalisation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 687-702, 2023.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: survey. Science China Technological Sciences, 63(10):1872 18971 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. aix preprint or
2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):90-106, 2021.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:21 10. 08207, 2021.
Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. Tackling the story ending biases the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Popers), pp. 752 757, 2018.
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connec- tionist systems. Artificial intelligence,
1990.
Yi-Lin Sung, Varun Nair, and Colin RafTel. Training neural networks with fixed sparse masks. A. Beygelz- imer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL http:p/ss/.gggaaeeeeeeee///////a
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarclla- Micke, Subhransu Maji, and Mohit lyyer. Exploring and predicting transferability across nlp tasks. arXiv preprint NKt::2005.007700 2020.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task gener- alization in nlp. arXio preprint 21X.:22044.0885, 2021.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer- based masked language-models. arXiv preprint arXiw:2106. 10199, 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? XrXiv preprint ar 19v: 07830, 2019.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameler-efficient fine-tuning. arXiv preprinl ar Xrx:23333 10512, 2023a.
Renrui Zhang, Jiaming Han, Aojum Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Tlama-adapter: Efficient fine-tuning of language models with zero-init attention. ar preprint 0rXiv:2303116190 2023b.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering. 34(12):5586-5600 2021.
17

---

Published in Transactions on Machine Learning Research (10/2024)
A Appendix
A.1 Dataset
For each task in TO (Sach et al. 2021 The details are described in Tab 5
there few examples to help with the task adaptation.
Dataset
Val Num
Few Shot
COPA
100
32
WIC
638
32
RTE
277
32
CB
Winograd
1267
RTEI
1000
32
RTE2
1000
32
RTE3
1200
32
WSC
104
32
H-swag
10042
20
Table 5: Dataset statistics of the TO benchmark
A.2 FLOP Analysis
TLoRA aims to reduce the number of training parameters by parameterizing the original LoRA architecture, but does not alter the computational process LoRA. Thus, TLoRA might conceivably incur additional computational overhead compared to standard LoRA. To understand this, we analyze the floating-point operations (FLOPs) involved in TLoRA.
Consider the tensor product of vector of size 1 with vector b of size 1xn, resulting in matrix C of size n. The product operation involves n² multiplications due to the elements in each of the vectors a and b. As described in Table 2 and shown in Figure 3 the additional computational effort in TLoRA stemns from the tensor product operations, dictated by the order the tensor. Specifically, for parameterizing with a dimension d, rank r, and lensor rank R, the number of extra computations required is approximately d r x R.
To test the time consumption according to the Flop analysis, show the training time in Tab 6 The results indicate that although the tensor product reduce the training parameters, need more training time according to the original LoRA adapters.
Model
Trainable parameters
Training time
ACC
LoRA
2.2M
14h 53m 27s
66.8
TLoRA
1.4M
2d 3h 13m 45s
66.1
Poly
17M
1d 16h
16m 43s
68.8
MHR
17M
1d 17h 5m
20s
69.2
TensorPoly-I (rank=8)
12.2M
2d 13m
8s
68.7
TensorPoly-TT (rank=8)
13.3M
2d 3h 48m
19s
67.9
PennsorPoly-|(rank2200
27.8M
2d 6h 6m 3s
69.3
TensorPoly-I[rrnnk-a
33.2M
2d 2h 55m 59s
68.0
Table 6: Training time analysis across different models.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Model
Finetuning parameter size
ACC
FullFT
3B
63.3
BitFiT(with) LN)
1.3M
54.6
LayerNorm
250K
54.2
Adapter
12.9M
61.6
Compacter
807K
61.0
Compacter++
540K
61.5
Prompt (10)
41K
50.9
Prompt (100)
409K
47.7
Prefix Tuning
576K
57.8
FishMask(0.2295
6M
61.1
FishMask(O..000
600K
59.0
SAID
500K
57.5
LoRA
9.1M
62,8
IA3
540K
64.6
s/pre-train
LoRA
2.2M
66.8
TLoRA
1.4M
66.1
Poly
17M
68.8
12.2M
68.7
27.8M
69.3
TeussoPPoly-1((RB-NN-1)
13.3M
67.9
TonsoPPoly-I[R-20NN-22
33.2M
68.0
Table 7: We show the parameters of different approaches in the finetuning process. For the baselines, we report the results from the paper [Luu et al. 2022).
A.3 Few-shot finetuning parameter size over different PEFT approaches
We investigate the training parameter size over PEFT approaches in the few-shot finctuning process. As presented in Tab. 7 TLoRA obtains competitive results while using only 60% training parameters than the original LoRA in this train setting. Compared with the latent experts approach Poly, TensorPoly-I with tensor rank R 8 and order N = 2, get almost the same results with fewer finetuning parameters. TensorPoly-III=20 N==2) obtain the best results with 27.8 finetuning parameters. Notably, the finetuning parameters are not the adaption parameters we present in Tab. 3
A.4 Routing across different tasks
To investigate the routing function across different tasks, plot the routing distributions for tasks such Question Answering and Summarization. As depicted in Figure D the routing function distribution varies across different layers for each task. Notably, Closed- Closed-Book QA exhibits greater sparsity compared to multi-choice QA. We will conduct further analysis of the routing function in various tasks to gain deeper insights.
A.5 Latent Experts Visualization
We visualize the latent experts in our models. As depicted in Figure 7 we draw the expert's weight dis- tribution in layer [0,4,16,23). The distribution of expert weights varies in different layers. Especially, The expert weights layer 16 seem completely different. from the experts in layers [0, 4, 23). We also draw the similarity of expert weights across different layers in Figure
19

---

Published in Transactions on Machine Learning Research (10/2024)
104
(a) Multi-choice QA
(b) Summarization
(c) Closed-Book QA
Figure 6: The X-axis is the 32 experts in each layer (For Q,K,V,O, each corresponds to 8 experts). Y-axis denotes the 24 transformer layers in the TO model. We show routing distributions for different kinds of tasks.
Layer N=1,R=1
Layer 0, N=1,R=8
Layer 0, N=2,R=1
Layer N=2,R=8
Layer N=1,R=1
Layer 4, N=1,R=8
Layer 4, N=1,R=1
Layer 1=
Layer 16, N=1,
Layer 16, 1
Layer 16, N=1,R=1
Layer 16, N=1,R=8
Layer 23, R=1,R=1
Layer 23, R=1,R=8 Layer 23, --1
Layer 23, -
Figure 7: Weights distributions for latent experts across different layers. N is the order and is the tensor rank. Each weight matrix here is A R Na)
20

---

Published in Transactions on Machine Learning Research (10/2024)
(a) Layer 0
(b) Layer 4
(c) Layer
(d) Layer
Figure 8: Expert similarity in each layer TensorPoly-II. We compare the similarity of experts each
layer. 0-7 refers to experts with order 1, 8-15 refers to the experts with order 2.
21


Reviews:
[{"review_id": "kYWkvrClBS", "reviewer": "Reviewer_gFt3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- A practical method for multi-task learning which is both efficient and effective. \n- Novel applications of tensor model in multi-task learning. \n\nWeakness\n\n- The authors did not clearly explain why tensorized latent experts can alleviate negative transfers in multi-task learning.\n- Kronecker-product-based weight factorization is parametric efficient. However, it may severely decrease the approximation capability, which corresponds to expressiveness of the model. The balance between efficiency and expressiveness is in general hard to achieve. I think more theoretical research needs to be done for this issue.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a tensor-based method for modeling \"latent expert\" to achieve a balance between computational efficiency and knowledge transfer for multi-task learning. The experimental results show the effectiveness and efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. This paper shows the proposed method works. But it fails in telling us more about why it works. Please give more discussion on the mechanism. \n2. It seems TensorPoly-II is more expressive than TensorPoly-I. However, it does not outperform TensorPoly-I in the experimental settings. It is better that the authors tell us the reason.", "broader_impact_concerns": "NO", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WYDMDIH75B", "reviewer": "Reviewer_Lh9h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nTensorPoly improves parameter efficiency and achieves better multi-task learning performance than its competitors;\n\n\nWeaknesses:\n- TensorPoly relies on tensor-product which introduces extra training flops and may increase the risk of training instability\n- The quality improvement over MHR is very marginal, particularly considering their similar parameter efficiency; the authors may need to figure out a clear setup where TensorPoly truly stands out.\n- More baselines and comparisons should be added\n- Scaling experiments is missing", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores improved ways to reduce cross-task interference in multi-task learning setup. To achieve this, the authors proposed to apply tensor-product based parameter efficient tuning to the mixture of expert optimization. \n\nTechnical contribution includes:\n- Propose TLoRA that applies tensor product to LoRA, further reducing trainable parameters thus improving parameter efficiency;\n- Propose TensorPoly that combines mixture of expert and TLoRA at different levels of granularity;\n\nUnder the T0 benchmark, TensorPoly achieves slightly better performance than previous models under a similar parameter budget.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It seems that TensorPoly adds extra flops during the training. Please add the amount of trainable parameters and extra training flops as two columns into Table 1 for better comparison.\n- I don’t understand why the authors name the method “mixture of latent experts”. I didn’t see any relationship between tensor products and the term “latent”. The used MOE method seems pretty standard.\n-  Apart from token-level MOE, there are also many task-level MOE methods that are not discussed and compared in this paper, such as [1] and [2]. \n- I’m curious whether we need to select a subset of experts in the multi-task learning setup. What if apply soft selection, i.e. directly constructing a softmax distribution over the experts?\n- Scaling becomes an important aspect for model finetuning [3]. How does the increase of model size, task number, and LoRA rank affect TensorPoly? Should we expect that TensorPoly scales reasonably? A concern is that tensor products may hurt training stability.\n- In Figure 3 left, what’s the number of rank corresponding to the similar amount of parameters to the standard LoRA. It seems that further scaling up the number of ranks beyond 20 hurts the performance; does this suggest TensorPoly has limited scaling property?\n- In Table 1, TensorPoly often performs worse on co-reference and some NLI tasks. Any analysis why?\n- In page 9, the authors claim “TensorPoly-I stands out by not only surpassing recent state-of-the-art achievements”. In fact, TensorPoly only outperforms MHR by 0.1 point on average (and underperforms it on several tasks), which is very likely not significant at all. Such a claim may be inappropriate.\n\n[1] Kudugunta et al., Beyond distillation: Task-level mixture-of-experts for efficient inference\n\n[2] Baziotis et al., Multilingual Machine Translation with Hyper-Adapters\n\n[3] Zhang et al., When scaling meets llm finetuning: The effect of data, model and finetuning method", "broader_impact_concerns": "I didn't find serious issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7lPrFXXpOC", "reviewer": "Reviewer_ASZr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The proposed method is interesting and makes sense. \n2. The paper is well-written.\n\nWeaknesses: \n1. The deeper understanding of the proposed method is lacking. \n2. It is not clear whether the proposed method can be applied to LLM.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes TLoRA, which is a parameter-efficient learning method that requires fewer tunable parameters than LoRA. This method also incorporates MoE, which improves the performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is not clear why the tensor-based method is better than LoRA. Is it because LoRA cannot capture some underlying structure but the proposed method can? Is there any intuitive explanation?\n\n2. It is still not clear why TensorPoly-I performs better than TensorPoly-II. Any theoretical explanation?\n\n3. I think Section 5.3.1 aims to compare the proposed method and LoRA. However, TensorPoly-I and TensorPoly-II include MoE in the architecture. It is unclear whether the performance improvement is due to MoE or the tensor construction.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the rebuttal, the overall rating of this paper is positive. The reviewers acknowledge that the proposed method is ''efficient and effective'' have ''novel applications in multi-task learning''. The rebuttal is sufficient and has addressed most of the concerns. However, the remaining issue is about the clarification of computation cost and parameter size. Thus, this paper should be carefully revised before acceptance.
Decision: Accept (poster)