MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards
Authors: Sheng Wang
Liheng Chen
Pengan CHEN
Jingwei Dong
Boyang XUE
Jiyue Jiang
Lingpeng Kong
Chuan Wu
Venue: ICLR 2025 Poster (2025)
Keywords: LoRA
parameter efficiency
parameter sharing
instruction tuning
NLP
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce a more parameter-efficient finetuning method named MoS, and demonstrate its remarkably higher parameter efficiency and other advantages over peer methods, with the hope of establishing it as a resource-friendly alternative to LoRA.

Abstract: The rapid scaling of large language models necessitates more lightweight finetuning methods to reduce the explosive GPU memory overhead when numerous customized models are served simultaneously.
Targeting more parameter-efficient low-rank adaptation (LoRA), parameter sharing presents a promising solution. Empirically, our research into high-level sharing principles highlights the indispensable role of differentiation in reversing the detrimental effects of pure sharing.
Guided by this finding, we propose Mixture of Shards (MoS), incorporating both inter-layer and intra-layer sharing schemes, and integrating four nearly cost-free differentiation strategies, namely subset selection, pair dissociation, vector sharding, and shard privatization. Briefly, it selects a designated number of shards from global pools with a Mixture-of-Experts (MoE)-like routing mechanism before sequentially concatenating them to low-rank matrices.
Hence, it retains all the advantages of LoRA while offering enhanced parameter efficiency, and effectively circumvents the drawbacks of peer parameter-sharing methods.
Our empirical experiments demonstrate approximately $8\times$ parameter savings in a standard LoRA setting. The ablation study confirms the significance of each component.
Our insights into parameter sharing and MoS method may illuminate future developments of more parameter-efficient finetuning methods.
The code is officially available at https://github.com/Forence1999/MoS.

Published as a conference paper at ICLR 2025
MoS: UNLEASHING PARAMETER EFFICIENCY OF LOW-RANK ADAPTATION WITH MIXTURE OF SHARDS
Sheng Wang", Liheng Chen", Pengan Chen School Computing and Data Science
Jingwei Dong
School of Electrical Engineering Xi' an Jiaotong University
The University of Hong Kong {u3009618, c1h648, cpa2031
(112000)@cconncttt Jeconnect. hku. .kk
dongj i ngwe @stu. xitu.u edu.
Boyang Xue, Jiyue Jiang
Department of Systems Engineering and Engineering Management The Chinese University of Hong Kong
byxue@se. cuhk, edu. hk, jiangjyg1 ink, cuhk cuhk.edu.
Lingpeng Kong, Chuan Wu School of Computing and Data Science
The University of Hong Kong {1pk, cwu)@cs. hk
ABSTRACT
The rapid scaling of large language models necessitates more lightweight fine- tuning methods reduce the explosive GPU memory overhead when numerous customized models are served simultaneously. Targeting more parameter-efficient low-rank adaptation (LoRA), parameter sharing presents promising solution. Empirically, our research into high-level sharing principles highlights the indis- pensable role of differentiation ing. Guided by this finding, we propose Mixture of Shards (MoS), incorporating both inter-layer and intra-layer sharing schemes, and integrating four nearly cost- free differentiation strategies, namely subset selection, pair dissociation, vector sharding, and shard privatization. Briefly, it selects designated number shards from global pools with Mixture-of-Experre (MoE)-like routing mechanism be- fore sequentially concatenating them to low-rank matrices. Hence, it retains all the advantages of LoRA while offering enhanced parameter efficiency, and effec- tively circumvents the drawbacks of peer parameter-sharing methods, Our empir- ical experiments demonstrate approximately 8x parameter savings in standard LoRA setting. The ablation study confirms the significance of each component. Our insights into parameter sharing and MoS method may illuminate future de- velopments of more parimeter-efficient finetuning methods. The code is officially available httpt:: / ctt. Ftwp//orreneel1e 1999/Mo9
reversing the detrimental effects of pure shar-
INTRODUCTION
With the remarkable capabilities, large language models (LLMs) (Dubey et al. 2024 Team al. 2023 Jiang et al.) (023) are adapted for various downstream applications (Shao et al. 2023 2024). Among parameter-effccient finetuning (PEFT) methods (Houlsby et al. 2019 Liu et al. 2022). low- rank adaptation (LoRA) (Hu et al. 2021) reparameterizes the weight updates with two trainable low-rank matrices, and has gained the most popularity with reduced resource consumption, compet- itive performance, low-cost task switching, etc. Nevertheless, with the continuous applicability of scaling law (Achiam al. 2023), the size of LLMs increases rapidly for even better performance, intensifying the need for more lightweight finetuning methods. For instance, GPT4o allows cus- tomization its vast user base incurring intense challenges serve numerous customized models
"qqual Contribution.
https://openai. http://ppeaaa.i.ooooome/////g0// htneett---o-inenntinnnn

---

Published as a conference paper at ICLR 2025
simultaneously. Assuming scenario with a Malmma--70Bsizeee model and 10,000 active users, each allocated a LoRA module with the rank of 16, only the parameters of LoRAs would occupy 3.36 TB of GPU memory, onerous burden for service providers. This underscores the need for more lightweight solution than LoRA, while retaining its advantages.
As promising solution, parameter sharing has been extensively explored (Kopiczko et al. 2023
Renduchintala et al., 2023 Wang et al. (024b). Specifically, VeRA (Kopiczko et al.) 2023) adopts inter-layer sharing of frozen matrices with trainable vectors; however, it results in limited model capacity and necessitates an excessively high rank. Subsequently, Tied LoRA (Renduchintala et al.) 2023] mitigates these constraints by permitting shared matrices be trainable. Yet, its tying mecha- nism restricts applicability to linear layers of different dimensions. More recently, PRoLoRA (Wang et al., 2024b) showeases enhanced parameter efficiency by reusing submatrices within a single lin- ear layer, thereby circumventing the above restrictions. However, its limited focus on intra-layer sharing constrains efficiency potential, and promises a hybrid approach integrating both intra-layer and inter-layer sharing. Besides, while these methods, with their intuitive designs, have shown the benefits of parameter sharing on enhanced parameter efficiency, the fundamental principles have not been discussed yet guide the future development of parameter sharing strategies.
Targeting enhanced parameter efficiency, we first analyze the factors influencing parameter sharing performance in the context of LoRA. Our empirical findings indicate that improving the parameter efficiency of LoRA requires a careful balance between sharing and differentiation. While parame- ter sharing can significantly decrease the number of trainable parameters, the negative effects (i.e., possible performance degradation) of excessive sharing necessitate differentiation strategies to mit- igate these drawbacks. Guided by this insight, we propose more parameter-efficient LoRA-based method called Mixture of Shards (MoS). It features a global sharing scheme to achieve a higher sharing ratio, along with four nearly cost-free differentiation techniques: subset selection, pair dis- sociation, vector sharding, and shard privatization. Briefly, our method selects designated number of shards from global pools with a Mixture-of- Experrs (MoE)-like routing mechanism (Jacobs al., 1991}. and sequentially concatenate them to construct the low-rank matrices.
Empirically, we validate its superior performance compared to peer methods in scenarios with fixed trainable parameter count, and demonstrate an eightfold parameter reduction in performance- targeted situations. thorough ablation analysis of each differentiation strategy reveals that all contribute to unleashing the parameter efficiency of MoS, despite their nearly cost-free property. Notably, pair dissociation and shard privatization boost the efficiency more significantly through increased combination diversity and exclusive differentiation, respectively, whereas vector sharding provides only limited gains in diversity. With the intuitive design, our method retains all the ad- vantages of LoRA while circumventing the drawbacks of peer methods. The markedly improved parameter efficiency alleviates the burden on service providers when numerous LoRA-based cus- tomized models are served simultaneously.
In summary, our main contributions are as follows:
We introduce a more pframeter-efficient finetuning method named Mixture of Shards (MoS), which facilitates the concurrent serving of numerous customized models with the saliently reduced GPU memory overhead.
Our exploration into high-level sharing principles highlights the balance between sharing and differentiation to enhance parameter efficiency, providing valuable insights for the fu- ture advancement of parameter-sharing methods.
Extensive experiments empirically confirm the significant parameter savings achieved by MoS, and validate the necessity and relative importance of each differentiation strategy.
To the best of our knowledge, we are the first to apply MoE-like mechanism for parameter savings in a single-task LoRA. Besides, our dissociated perspective may also inspire further parameter reduction for LoRA-bases multitasking scenarios.
2 SHARING & DIFFERENTIATION
Drawing inspiration from the inter-layer shared VeRA (Kopiczko et al. 2023) and intra-layer shared PRoLoRA (Wang al. 2024b), we seek create unified sharing mechanism that leverages both
2

---

Published as a conference paper at ICLR 2025
A
Protrained Weight
Protraincd Weight
Pretrained Weight
R
g
RM
(a) LoRA
(b) Pure Sharing
(c) Pure Sharing (Differentiated)
Figure 1: Illustration of vanilla LoRA, pure sharing, and differentiated pure sharing methods. The rank of vanilla LoRA is denoted as while that of pure sharing is boosted to rL through the aggregation of trainable parameters among Transformer blocks. Figure the colored squares between the low-rank matrices A and B signify scalars and boolean values for random scaling and subset selection strategies, respectively.
perspectives, promising further enhanced parameter efficiency. To guide this development more clearly, we start with reparameterizing the vanilla LoRA, followed by analyzing the impact high- level factors (i.e., sharing and differentiation) model performance under specified number of trainable parameters. All relevant approaches are illustrated in Figure for an intuitive comparison.
Reparameterization of LoRA. LoRA (Hu al. 2021) efficiently adapts LLMs by updating the frozen pretrained weights with low-rank decomposition. Specifically, given a pretrained weight Wo Roxh its weight update AW is approximated as the product of two trainable matrices A Rrxh and B E Roxr, where the rank is times smaller than h and for parameter saving. Similar Kopiezko et al. (2023), this product can be further reparameterized as the sum outer products of the corresponding row and column vectors. Formally, the whole process can be presented as:
W +AW Wo + BA Wo Σ b X a
where W is the updated weight, and a, and b; denote the i-th row and column vectors of low-rank matrices A and B, respectively. This reparameterization trick enables the weight update to be the linear combination of outer products of multiple vector pairs, which will be extensively utilized in our subsequent analysis. Generally, LoRA will be applied to each linear layers in all the Transformer blocks of model (Vaswani al. 2017) for better performance (Dettmers al.) 2023). which is also applied to our analysis by default.
Pure parameter sharing does not necessarily boost up the parameter efficiency of LoRA. Al- though VeRA and PRoLoRA achieve better performance than LoRA via parameter sharing, we demonstrate that this causality isons, we introduce a straightforward sharing scheme, detailed in Sec. [.1 Briefly, this scheme shares low-rank matrices A and across Transformer blocks for each type of linear layer. As shown in Table we set the same low-level trainable parameter budget for all methods, eliminat- ing the disturbance parameter redundancy and enabling clear comparison of their representation
universally applicable. To facilitate our subsequent compar-
Table 1: Results of LLaMA2-7B with different sharing and differentiation methods across diverse instruction following datasets. "+ Random Scaling" and "+ Subset Selection" denote the individual integration of them into the "Pure Sharing" scheme, respectively. The details of these benchmarks
are claborated in Sec.
MMLU
BBH
GSM8K
TyDi QA
HumanEval
Method
Rank
Param.
Avg.
RM
FM
EM
F1
EM
PΦ
LoRA
5.00M
44.77
36.22
26.28
48.67
35.70
18.24
34.98
Pure Sharing
64
5.00M
43.61
35.15
26.54
49.12
36.04
15.53
34.33
+ Random Scaling
64
5.00M
43.98
35.30
29.04
49.03
35.73
15.58
34.77
+ Subset Selection
64
5.00M
45.56
36.76
28.18
50.33
37.22
18.64
36.12
3

---

Published as a conference paper at ICLR 2025
capabilities. Despite several times higher rank, which is suggested for better performance (Wang ct al,, 2024b), this pure sharing scheme outperforms LoRA on GSM8K and Tydi QA tasks, but underperforms others. This variabilitty indicates that excessive sharing may hinder model perfor- mance, negating the positive effects of increased rank. In other words, sharing mechanism is not the only crucial element of existing methods for enhanced parameter efficiency.
Differentiation reverses the detrimental effects of pure sharing mechanism. Inspired by the varying rank demands in different positions (Zhang al. 2023). we hypothesize that the pure shar- ing scheme, reusing the same matrices among all Transformer blocks, contradicts different repre- sentation capacity across blocks, potentially deteriorating the performance. To verify this, we devise the following two schemes to differentiate the shared matrices:
1. Random Scaling, Random scaling adds scalars each vector pair, which can be formu- lated as -A=E b; @a, where is sampled from a normal distribution and frozen after initialization. Hence, this operation does not introduce extra trainable parameters, but different blocks will optimize the vectors associated with their own larger scalars despite the same shared whole matrices, since these vectors influence the representation capacity more significantly than others.
2. Subset Selection. Instead of scaling the vector pairs, this scheme selects a specific number of vector pairs from the shared matrices. Specifically, the scalars in the random scaling method are substituted with Boolean values. Vectors corresponding to those marked as true are selected, while the others are neglected. This ensures that, in most cases, any two blocks only share a subset of vector pairs so that their representations are differentiated by other distinct pairs.
As shown in Table [ both differentiation strategies further improve model performance over the pure sharing scheme. Specifically, random scaling can enhance the performance of pure sharing in nearly all tasks, and ultimately achieves slightly better average results. However, it still underper- forms LoRA apparently. In contrast, subset selection, adopting more aggressive differential oper- ations, reverses the degradation and consistently surpasses both pure sharing and LoRA across all benchmarks. On average, its performance exceeds that of LoRA over one percent, highlighting the importance of differentiation. Further experiments and analysis on LLaMA3.2-30 model (Dubey ctt al. 2024) are provided in Appendix [.22
In summary, enhancing the parameter efficiency of LoRA involves delicate balance between shar- ing and differentiation. On one hand, parameter sharing can significantly reduce the number of trainable parameters; the other hand, the detrimental impacts of sharing necessitate differentia- tion schemes to counteract them. This insight will inform our subsequent design efforts.
METHOD
Based on the above analysis, we follow the guide of sharing and differentiation, and propose more lightweight LoRA-based method, named Mixture of Shards (MoS). Briefly. it selects a specific number of shards from global pools with a MoE-like routing mechanism, and combine them se- quentially to form the low-rank matrices of LoRA. It features a global sharing scheme for higher sharing ratio, and four nearly cost-free differentiation measures, namely subset selection, pair dis- sociation, vector sharding, and shard privatization. Their details will be elaborated below, followed by the advantage analysis.
3.1 GLOBAL SHARING
To maximize the parameter efficiency, we introduce global sharing mechanism, incorporating VeRA-like inter-layer sharing and PRoLoRA-like intra-layer sharing methods. As illustrated in global pool for each type of linear layer. Initially, each
Figure 2a this mechanism centers pool consists of multiple independently initialized and trained vector pairs, while each vector pair within the low-rank matrix pairs (i.e., A and B) for each layer is sampled from this pool. The "pure sharing" in Sec. can be formed when all vector pairs from the pool are selected at each block. In this case, the weight update can be formulated as
AW
BA B
4

---

Published as a conference paper at ICLR 2025
where A Balx and B" Roxel. represents the globally shared low-rank matrices for one linear layer type while L and e denote the number of Transformer blocks and an equivalent rank to that of LoRA in terms of trainable parameter count, respectively. Due to the extremely high sharing ratio (ie. I.), a small number of parameters can yield a remarkably high rank, hinting positive effects on performance within certain range (Wang al. 2024a b). Taking the LLaMMA-7B model with 32 blocks (Touvron et al.) 2023) as an example, pure sharing can raise the rank from 2 to 64 with the same trainable parameter count LoRA, indicating its great potential for parameter reduction.
3.2 SUBSET SELECTION
Thrafomer
R
Radeer
Incex
&
19
Trandeer Slock
Low Rank Matrk
idw Musia
Glbb Pool
LLM
Global Pool
(a) Inter-Layer Sharing
(b) Intra-Layer Sharing
Figure 2: Illustration of MoS from the perspectives of inter-layer and intra-layer sharing. (a) Across layers, each layer retrieves shards from the same global pool, utilizing an independent index matrix and MoE-like router "R". (b) Within each layer, the shard retrieval process is visualized in details with the number of shards per vector I, rank r, pool size 2, and respectively. The circled numbers denote the shard indices in the global pool, while the small italicized numbers indicate the shard positions within the low-rank matrix. Blue highlights shared components, whereas green indicates privatized ones.
As discussed in Sec. 2 while pure sharing scheme may lead to performance decline due to the shared weight updates across Transformer blocks, differentiation plays indispensable role in versing the detrimental effects. Compared to random scaling, subset selection demonstrates a more pronounced performance improvement, and is thus incorporated into our design for differentiation. Let r denote the number of vector pairs chosen from the shared pools, signifying the rank of each low-rank matrix during both training and inference. The weight update for the k-th block can be expressed as follows:
eL WW B = B'A Σ m Oy
(3)
where the boolean value m represents the i-th diagonal element of the diagonal matrix A for the k-th block. This value randomly sampled during initialization, remains fixed during the finetuning process, and satisfies the condition Σ m = r. Additionally, af and by denote the i-th row and column vectors of A and B", respectively.
3.3 PAIR DISSOCIATION
In prior research, such as Valipour et al. (002): Kopiczko et al. (2023), vector pairs have been regarded as the fundamental units of LoRA by default. To foster greater differeeniaaion, we propose a vector pair disassociation strategy that is straightforward and coft-effective, requiring no additional trainable parameters while yielding significant improvements in our empirical analysis. Specifically, instead of a single shared pool for vector pairs (i.e., pairs of a and by), we take decoupled vectors as the basic units, and separate a, and by into two distinct pools. Hence, within the k-th block, the vectors that form matrices A and B will be sampled independently. Moreover, this separation naturally enables two ordered index vectors. 1 E N" and I N", instead of unordered boolcan
Due the identical operations, the notation for different types of linear layers is omitted for clarity.
5

---

Published as a conference paper at ICLR 2025
vectors, access vectors from the pools. Both the dissociation and the ordered indices significantly enhance the combinatorial diversity A and B Formally, this process can be formulated as
AW AB "Boute"("* 1 Route" (ooute"(A", (
(4)
where (.) and Route" (+) denote the retrieval of column and row vectors, respectively, based on the given indices.
3.4 VECTOR SHARDING
Drawing inspiration from database S we further refine the basic unit of the shared pool to vector shards, where / sampled shards are concatenated into a single vector. As illustrated in Figure 2b] two vectors corresponding the indices "0" and "3" are retrieved and combined to form the first rank of the low-rank matrix. Similar to vector disassociation, this nearly cost-free operation further promotes the diversity and differentiation of low-rank matrices. Subsequent experiments will confirm its beneficial effects. The mathematical formulation remains unchanged, with the exception that the dimensions of I and I are adjusted I
3.5 SHARD PRIVATIZATION
As another differentiation strategy, partition the global pool into two segments: public and pri- vate. As indicated by their names, the public segment remains shared, whereas the private part is exclusively accessible to only one matrix. This privatization improves the differentiation of weight updates among blocks, as expressed by
(5) where A in Eq. 4 is substituted by the concatenation Concat(-) of the public pool A and the private pool A pr Similarly, B is replaced by the he concatenation of Baub and B Besides, the private pools are achieved through indices exclusively owned by blocks, maintaining unified operation with public pools.
Ib)) W = A Route" (Concat(A (aonnat(Aruu pub A
), Route" (Concat(Buus BPrr), ),
Initialization. Based the shard size and the specified trainable parameter count, first instanti- ate the global pools for various linear layer types, respectively. Following standard LoRA practices, we initialize Boub and B to zero, ensuring consistency with the pretrained model at the start of finetuning. As adopted in PRoLoRA (Wang et al.) 2024b). we adjust the sampling boundaries of P and Ari to align with the vanilla LoRA. Subsequently, we randomly initialize the index ma- trices I and 15, and ensure the private shards are sampled only once. The training and inference procedures remain identical to those of LoRA, except that all low-rank matrices are constructed from the index matrices and global pools.
Unity. Even if all the differentiation strategies are claborated individually for clarity, they can be merged unified solution seamlessly for easy implementation. Briefly, MoS utilizes the routing mechanism to retrieve and concatenate shards into low-rank matrices from global pools. Addition- ally, we justify the theoretical motivation of each differentiation strategy from the perspective of combinational diversity in Appendix. B.I
3.6 ADVANTAGE ANALYSIS
Compared to LoRA, MoS only modifies the source of low-rank matrices to concatenated shards from the global pools. This adjustment retains the advantages of LoRA, such as low-cost switching and linear properties. The former allows for swapping only the finetuned weights instead of all pa- rameters, facilitating the efficient handling of multiple customized models simultaneously. The latter enables MoS to be merged with pretrained weights during inference, thus eliminating inference la- tency. Moreover, this source modification allows MoS to leverage existing technical support of LoRA (Sheng et al. 2023 Chen et al. 2023) with minimal effort. Additionally, MoS offers several extra advantages LoRA and other peer methods.
High Parameter Efficiency. By integrating both inter-layer and intra-layer sharing, MoS demon- strates significantly higher parameter efficiency compared to LoRA and other baselines, shown
httpss//en...t htt.a//aat///////iippiipcaa.aa. httpa////ww.iigeioooooo/////a/aa (ataaaard___dttaaa___e___e (ecturr)))

---

Published as a conference paper at ICLR 2025
in Sec. 4.2 This reduction in parameters results in decreased disk space and GPU memory usage, thereby greatly alleviating the serving burden in multi-LoRA scenarios.
High Representation Capacity. As shown in Table 2 even with rank of 256 or higher, Tied LoRA and VeRA still clearly underperform other methods. In contrast, the performance of MoS approximately converge that of full finetuning do LoRA and PRoLoRA. This ensures substantial model capacity, which is crucial for tackling challenging tasks.
EXPERIMENTS
4.1 GENERAL SETUP
Our primary experiments focus on evaluating the instruction following abilities of models, aligning closely with the general configurations outlined by Tulu (Wang et al. (023) and PRoLoRA (Wang et al. 2024b). Similarly, we adopt comprehensive assessment protocol that involves factual knowl- edge, reasoning abilities, multilingual proficiency, and coding skills. Following Wang et al. (2024b), select specific settings that have demonstrated positive results evidenced by Table 7 of Wang iet al. (2023). Besides, we inherit their unified chatbot framework. This necessitates models to learn both specific tasks and the interaction style. The core configurations are summarized below, with further details available in the Appendix
Datasets. We finetune base models on Super-Natural Instructions (SuperNI (Wang al. 2022
dataset, and evaluate them on both Massive Multitask Language Understanding (MMLU (Hendrycks) al. 202T) and TyDi QA (Clak et al. 2020) datasets for factual knowledge and multilingual capabilities, respectively. For the general and mathematical reasoning abilities, we finetune models on Flan V2 and its CoT split (Longpre et al., 2023), and conduct evaluation on Big- Hig-Bench-HHar (BBH (Suzgun et al., 2202)) and the test set of Grade School Math (GSM8K (Cobbe et al. 2021) corpora, respectively. Moreover, coding skills are evaluated on HumanEval (Chen et al., 2021 dataset after models are finetuned on CodeAlpaca (Chaudhary 2023> dataset.
Baselines. From the perspective of parameter sharing, we evaluate our approach against LoRA and other existing intra-layer or inter-layer sharing methods. A brief introduction to these baselines is provided below.
LoRA (Hu et 2021) freezes the pretrained weights and decomposes the weight updates into two trainable low-rank matrices, as formulated in Sec. 2 Following the settings of QLoRA (Dettmers et al. 2023), LoRA is applied to every linear layer across all Trans- former blocks, including query, key, value, output, up, gate, and down projection weights.
VeRA (Kopiezko et al. 2023) also freezes pretrained weights, along with randomly initial- ized shared low-rank matrices, and employs decoupled scaling vectors for the differentia- tion of weight updates. Consistent with LoRA, VeRA is also applied to all linear layers.
Tied LoRA (Renduchintala et al. 2023) ties the down projection matrices across all query, key, and value projection weights. It shares trainable low-rank matrices across Transformer blocks for parameter savings, while updating the separate scaling vectors for differentiation.
PRoLoRA (Wang al. (024b) recently provides an intra-layer solution aimed at enhanc- ing parameter efficiency. It replicates the sub-matrices of LoRA multiple times before differentiating them with partial rotation operations.
4.2
MAIN RESULTS
The main results are summarized in Table showcasing the performance of various methods across multiple instruction tuning tasks. Notably, as the rank increases from 2 to 64, the performance of LoRA consistently improves across all benchmarks, significantly surpassing that of the vanilla model. This observation suggests that the current configurations do not introduce apparent parameter redundancy in trainable parameters, highlighting the effectiveness of the setups in demonstrating the parameter efficiency of different approaches.
In scenario with low-level trainable parameter count, MoS outperforms all baseline ap- proaches. Specifically, to highlight the differences in parameter efficiency among various methods, fix the number of trainable parameters low level (i.e., 5.00M, equivalent to that of LoRA with

---

Published as a conference paper at ICLR 2025
Table 2: Results of LLaMA2-7B across multiple instruction-folloing datasets using different meth- ods. The abbreviations "# Param." and "Avg." refer to "Parameter Count" and "Average", respec- tively, while the symbols "-sp", "-vs", and "-pd" indicate the ablation of shard privatization, vector sharding, and pair dissociation. The notations "4/8" and "16/32" correspond to increasing the rank to 4 or 8, and 16 or 32, respectively") "*" denotes the optional higher ranks. Underlined values indicate the best performance with 5.00M trainable parameters, while bold values denote the best results across all configurations. "I" indicates the results copied from Wang et al. (2024b).
MMLU
BBH
GSM8K (reasoning)
TyDi QA (multilinguality)
HumanEval (coding)
(factuality) (reasoning)
Method
Rank # Param.
Avg.
EM (0-shot)
EM
EM (8-shot, CoT)
F1
EM
P@ 1 (0-shot)
(3-shot)
(1-shot, GP)
Vanilla (chat)*
41.18
0.00
3.03
17.40
0.10
0.64
10.39
Vanilla (no-chat)
41.53
33.43
15.47
49.18
35.35
13.57
31,42
5.00M
44.77
36.22
26.28
48.67
35.70
18.24
34.98
LoRA
81
19.99M
46.55
36.92
31.11
50.50
36.89
19.37
36.89
16t
39,9 39.98M
46.70
36.43
31.34
50.97
37.64
18.73
36.97
64
159.91M
47.10
37.78
31.43
51.65
38.07
19.12
37.53
VeRA
256
1.42M
42.51
35.10
22.69
48.39
36.38
18.90
34.00
Tied LORAt
280
4.99M
44.36
35.76
25.47
50.16
37.15
18.68
35.26
PRoLoRAt
4/8
5.00M
45.85*
36.45"
27.57
49.94"
36.59*
19.75'
36.03
4/8
5,00M
46.09
37,29
28.43
50,21
37.19*
19.12
36.39
MoS
16/32
19.99M
47.01*
37.79
30.93
51.71
38.34
20.00'
37.63
MPS
16/32
19.99M
46.64*
36.69
30.17
50.27
36.90
18.60'
36.54
MOS.v
16/32
19.99M
46.47*
37.52
31.77
50.90
37.98
18.69*
37.22
MOS-M
16/32
19.99M
46.23"
36.17
30.71
51.40
37.94
16.77'
36.54
a rank of 2), and compare their performance. Naturally, more superior performance indicates higher parameter efficiency. As shown in Table both the inter-layer shared Tied LoRA and the intra-layer shared PRoLoRA demonstrate better average performance than LoRA. In contrast, by incorporating both intra-layer and inter-layer sharing, our method, MoS, surpasses all of them on average, and achieves consistent improvements across all individual tasks, as indicated by the underlines. How- ever, similar PRoLoRA (Wang et al.) 2024b), we do not claim to achieve greater parameter effi ciency than VeRA, given its lower parameter count. However, we attempted to increase the rank of VeRA to match the identical trainable parameter count, but failed due to an out-of- cu--of-memory (OOM) error. This indicates that VeRA has very low performance-to- to significantly higher training costs for numerous customization.
ratio, making it impractical due
In regular setting, MOS achieves around 8x savings in trainable parameters. Considering the significance of performance on various benchmarks, we turn to evaluate the parameter savings of MoS with performance target, where fewer trainable parameters also indicate high efficiency. We set the performance target to align with LoRA at a rank of 64, which yields an average perfor- mance of 37.53. In comparison, with 19.99M trainable parameters (equivalent to that of LoRA with a rank of 8), MOS outperforms LoRA on four out of six metrics, while maintaining comparable average performance. This reconfirms the superior parameter efficiency of MoS, and demonstrates its around eightfold savings in trainable parameters. This can considerably alleviate the burden on service providers, when multiple LoRA-based customizations are served simultaneously.
4.3 SCALABILITY ANALYSIS
To assess the robustness of MoS regarding model scale, we extend the experiments the LLaMA2- 13B model. However, we find that finetuning with vanilla LoRA does not yield consistent improve- ments on the TyDi QA and Code benchmarks. We hypothesize that this phenomenon might be
For fair comparison, we keep an identical trainable parameter budget for both LoRA, PRoLoRA and MoS, treat the raised rank of PRA and MoS a hyper-parameter, and report the results in one line for more intuitive comparison.
8

---

Published as a conference paper at ICLR 2025
Table 3: Results of LLaMA2-13B across multiple instruction-followingg datasets using different
methods. The abbreviations, notions, and settings align with those of Table respectively.
Method
Rank
MMLU
BBH
GSM
Avg.
Vanilla (chat)
50.05
0.00
2.12
17.39
Vanilla (no-chat)
52.04
39.67
27.82
39.84
LoRAT
52.78
42.50
36.47
43.92
PRoLoRA
4/8
53.27
43.12
38.72
45.04
MoS
4/8
53.51
43.65
40.79
45.98
caused by the more powerful capabilities of the base model, which diminishes the effectiveness of the training set. Hence, we exclude these tasks from analysis.
As listed T we impose a limited trainable parameter count (equivalent to that of LoRA with the rank of 2) to compare the representational capacities of different methods. Specifically, com- pared to the pretrained model, LoRA achieves a notable performance improvement with an average score of 43.92. Building this, PRoLoRA boosts the average performance by 1%. In contrast, MoS further unleashes the potential of parameters. It consistently exhibits performance enhance- ments across all individual benchmarks, with another increase of nearly 1% average. Targeting smaller scale, we also extend the experiments to the LLaMA3.2-33 3-BB- model with additional random seeds in Appendix. [B3]] demonstrating stable performance enhancements across all individual tasks. The considerable improvement reinforces MoS's positive impact on parameter efficiency and its ro- bustness with respect to model sizes.
4.4 ABLATION STUDY
In this section, conduct ablation study each differentiation strategy incorporated by MoS, before comparing their relative significance. As discussed Sec. 2 and listed in Table [] subset selection plays indispensable role in reversing the detrimental effects of pure sharing. Since lays the groundwork for other strategies, we will not reiterate its importance here. Results from the ablation of the other components are presented in Table [
Pair Dissociation. Rather than using distinct indices for enhanced combination diversity, we apply the same index matrix for the low-rank matrix pair A and B within each linear layer (i.e.. I I.). This results in an average performance decline exceeding one percent, emphasizing the critical role of pair dissociation and diverse combinations in improving model performance and aligning with our initial motivation.
Vector Sharding. When the basic units in the global pools are reverted back the row or column vectors of the low-rank matrices from smaller shards, only a 0.41% % average decrease in perfor- mance is observed. This decline is much less than that associated with pair dissociation. Given that both components aim to enhance combination diversity, these findings imply that pair dissociation provides a more substantial boost to diversity, whereas sharding yields only marginal improvements.
Shard Privatization. Different from the previous components, shard privatization enhances differ- entiation among low-rank matrices by allowing certain shards to be exclusively accessible to one low-rank matrix. Its ablation also leads performance drop exceeding percent, highlighting the significance of privatization.
In summary, all differentiation strategies contribute to improving the parameter efficiency of MoS, despite all being nearly cost-free operations. Among them, pair dissociation and shard privatiza- tion unleash the efficiency more saliently through increased combination diversity and exclusive differentiation, respectively, while vector sharding offers only incremental gains in diversity.
RELATED WORK
Low-Rank Adaptation. Low-rank adaptation (LoRA), introduced in Hu et al. (2021). reparameter-
izes weight updates with two trainable low-rank matrices while the pretrained weights remain fixed. This approach has since spurred numerous enhancements aimed at improving its effectiveness and
efficiency (Zhang al. 2023 Wang et al. 2024a b). AdaLoRA (Zhang al.) 2023). leveraging

---

Published as a conference paper at ICLR 2025
singular value decomposition, automates rank allocation by adaptively pruning vector pairs during finetuning. However, the rank variability across layers complicates the simultancous serving of mul- tiple LoRAs (Sheng et al., 2023). Closely aligning with our work. VeRA (Kopiczko et al.. 2023) implements inter-layer sharing of two frozen random matrices while updating disentangled combi- nation vectors for each layer. However, it necessitates excessively high rank due limited model capacity. Subsequently, though Tied LoRA (Renduchintala et al. (023) alleviates this constraint by allowing shared matrices be trainable, the tying mechanism on down projection matrices restricts its applicability to linear layers of different dimensions. More recently, PRoLoRA (Wang et al.) 2024b) demonstrates enhanced parameter efficiency by reusing sub-matrices within a single linear layer, thus avoiding the above limitations. However, its narrow focus on intra-layer sharing restricts its performance potential. Overall, these approaches typically assume vector pairs as the fundamen- tal units for granted, and concentrate exclusively on inter-layer or intra-layer sharing, both of which limits the potential for parameter efficiency. In contrast, our method utilizes fine-graine grae-gr shards as basic units and facilitates global sharing to achieve improved parameter efficiency.
Parameter Sharing, Parameter sharing has been extensively studied to minimize model sizes, Uni- versal Transformer (Dehghani et al.) 2018) shares all layers of Transformer model, before Takase & Kiyono (2023) further optimizes this approach with three parameter-shane
techniques across Transformer layers for higher efficiency. DictFormer (Lou al. 2021) implements more compact model with a shared dictionary, unshared coefficients and indices. For on-device deployment, Edge- Former (Ge et al., 2022) shares attention and FFN modules, and incorporates PEFT-based layer adaptation to reduce parameter count. Pires et al. (2023) eliminates the decoder layer FFN and shares single, larger FFN across the encoder, achieving notable improvements in accuracy and latency. Recently. Cao et al. (024)) proposes two memory-efficiee methods for parameter sharing in attention heads. Differently, our work centered on the parameter sharing LoRA modules, and explore the foundational principles guiding the development of parameter sharing strategies.
Mixture of Experts. Mixture of Experts (MoF) was initially proposed by Jacobs et al. (991).
Since Zadouri et al.] (2023) replaces traditional experts with LoRA modules, various efforts have been made to perform better integration (Wu et al. 2024). Aiming at multi-task learning, Mixture- of-LoRAs (MoA) (Feng et al. 2024 integrates multiple domain-specifie LoRAs through rout- ing strategy, improving both individual task performance and rapid domain adaptation. Similarly, MixLoRA (Li et al. 2024) constructs sparse LoRA--bsee' MoE model by inserting multiple LoRA experts within the FFN modules and employing independent LoRA adapters in the attention mod- ules. Recently, Mixture of Dyadic Experts (MoDE) (Ning al.) 2024) shares down-projection matrix for different tasks and employs atomic rank-one adapters with routers for more sophisticated task specialization, enhancing the model's 's multi-tasking capabilities. In contrast, Zhou et al.) (2024) present Mixture-off -ffeuerrss with Language Priors Routing (MoE-LPR), featuring a two-stage train- ing process to improve multilingual capability without catastrophic forgetting. To the best of our knowledge, we are the first one to apply MoE for parameter savings within a single LoRA module, rather than focusing on multi-task or multilingual settings.
6 CONCLUSION
Targeting higher parameter efficiency via parameter sharing, we investigate high-level sharing prin- ciples, highlighting the essential role of differentiation in reversing the detrimental effects of pure sharing. Building on this finding, we propose more parameter-effiiien LoRA-based method called Mixture of Shards (MoS), incorporating both inter-layer and intra-layer sharing schemes, and in- tegrating four nearly cost-free differentiation strategies: subset selection, pair dissociation, vector sharding, and shard privatization. maintains all the advantages of LoRA while achieving higher parameter efficiency, and effectively avoids the drawbacks of peer parameter-sharee insights into parameter sharing and the MoS method may illuminate future developments in more parameter-efficien finetuning techniques, particularly alleviating the burden for service providers handling multiple customized models simultaneously.
methods. Our
7
ACKNOWLEDGEMENT
This work supported in part by Hong Kong Innovation and Technology Support Programme Platform Research Project fund (ITS/269/2FF)), RGC grants 17204423, 17205824, C7004-22G (CRF), and C5032-23G (CRF).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint wwXiv:2303.087770 2023.
Zouying Cao, Yifei Yang, and Hai Zhao. Head-wise shareable attention for large language models. arXiv preprint arXhv:2402.11811 2024.
Sahil Chaudhary. Code a alpaca: An instruction-followingg llama model for code generation. https:
//github.com sah. 120011281111 htt.0/11111///caaaaaaaaaaaaaaaae 2023.
Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. Punica: Multi-tenant lora serving, arXiv preprint arXiv:2310. 18547, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.088 07.3744 2021.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetunee lan- guage models. arXiv preprint urXiv:2210. 1416, 2022.
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information-secking question answering in ty pologically di verse languages. Transactions of the Association for Computational Lingusstics,, 8:454-470, 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1rXiv:2110.14100 2021.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819/ 2018
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint 23.Xi:::300..13144 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint wxXiv:2407.21700 2024.
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mfxture-of-loras: An effi- cient multitask tuning for large language models. arXiv preprint arXiv:2403.03432, 2024.
Tao Ge, Si-Qing Chen, and Furu Wei. Edgeformer: A parameter-efficient transformer for on-device seq2seq generation. arXiv preprint 22Xiv:220..07959, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo. Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0960 09685, 2021.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.

---

Published as a conference paper at ICLR 2025
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310. 1454, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprint arXiv:240..15159 2024.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang. and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pp. 61-68, 2022.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688 2023.
Qian Lou, Ting Hua, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Dictformer: Tiny transformer with shared dictionary. International Conference on Learning Representations, 2021.
Lin Ning, Harsh Lara, Meiqi Guo, and Abhinav Rastogi. Mode: Effective multi-task parameter efficient fine-tuning with a mixture of dyadic experts. arXiv preprint arXiv:2408. 010000001555 01505, 2024.
Telmo Pessoa Pires. António V Lopes, Yannick Assogba, and Hendra Setiawan. One wide feedfor- ward is all you need. arXiv preprint 1xXiv:2309,01822 2023.
Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-lora: Enhacing parameter effi- ciency of lora with weight tying. arXiv preprint arXiv:2311.09578, 2023.
Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-lIm: A trainable agent for role- playing. arXiv preprint vrXiv:2310.1015: 2023.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning open language models, 2024.
03300
Ying Sheng. Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:231 1..iv:23/1.032883 1.03285, 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 2xXiv:2006.06615, 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether th can solve them. arXiv preprint arXiv:2210.09261, 2022.
Sho Takase and Shun Kiyono. Lessons on parameter sharing across layers in transformers. In Nafise Sadat Moosavi, Iryna Gurevych, Yufang Hou, Gyuwan Kim, Young Jin Kim, Tal Schus- ter. and Ameeta Agrawal (eds.), Proceedings of The Fourth Workshop on Simple and Effi- cient Natural Language Processing (SustaiNLP), pp. 78 28-90. Toronto, Canada (Hybrid), July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.um 2ttp3//w1/223..sssannppl-..... URL https: /tops://act/ antho ogy. ogy.org//020..... sustainnn inlsttinnlp1..s
12

---

Published as a conference paper at ICLR 2025
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https:/crrfi stanford.
edlu2023/0//1//aapaaa himl, 3(6)7,, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter effi- cient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07555 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Sheng Wang, Liheng Chen, Jiyue Jiang, Boyang Xue, Lingpeng Kong, and Chuan Wu. LoRA meets dropout under unified framework. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 1995-2008, Bangkok, Thailand and virtual meeting, August 2024a. Association for Computational Linguistics. URL pttps:://acantioen wenetthollg.....o org/224 htt.2/22......nroan findings-acl.111
Sheng Wang, Boyang Xue, Jiacheng Ye, Jiyue Jiang, Liheng Chen, Lingpeng Kong, and Chuan Wu. PRoLoRA: Partial rotation empowers more parameter-efin seffmieer-efficient LoRA. Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 2829-2841, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. URL
2x1/20244.c11-oonn 156
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Suppe-naturaalistructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705: 2022.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. urXiv preprint arXiv:2306. 04751, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. arXiv preprint arXiv:2404.13622 2024.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker, Push- ing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint arXiv:2309.0544. 2023.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning, arXiv preprint arXiv:2303. 10512, 2023.
Hao Zhou, Zhijun Wang, Shujian Huang, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Weihua Luo, and Jiajun Chen. Moe-Ipr: Multilingual extension of large language models through mixture- of-experts with language priors routing, arXiv preprint arXiv:2408. 1008..1190 11396, 2024.
13

---

Published as a conference paper at ICLR 2025
A EXPERIMENT DETAILS
Our main experiments concentrate on assessing the mnstruction-folloiing abilitics of models, closely aligning with the frameworks established by Tulu (Wang et al. 2023). In similar vein, we imple- ment a thorough evaluation protocol that encompasses factual knowledge, reasoning skills, multi- lingual capabilities, and programming expertise. Consistent with Wang al.. (2024b) we select specific configurations of training and test datasets that have yielded positive outcomes, as indicated by the Table of Wang et al. (2023). Additionally, we adopt their integrated chatbot framework, which requires models to acquire both specific tasks and interaction styles. The essential configura- tions are summarized below.
A.1 DATASET DETAILS
To evaluate various aspects of model capabilities, we perform comprehensive assessments us- ing multiple datasets. Specifically, we finetune models on the Super-Natural Instructions (Su- perNI (Wang et al., 2022)) dataset and then measure their performance on the Multitask Language Understanding (MMLU (Hendrycks et al. 202II) for factual knowledge and on the TyDi QA (Clark) ct al. 2020) dataset for multilingual abilities. For general and mathematical reasoning evaluation, we retrain the foundation models on the Flan V2 dataset and its CoT split (Longpre et al. 2023). Their corresponding performances are then presented on the Big- Hig-Bench-Haan (BBH (Suzgun al., ) and the Grade School Math (GSM (Cobbe et al.] (021)) test splits, respectively. Addition- ally, utilize the HumanEval (Chen et al.
202 benchmark assess the coding abilities of models finetuned on the CodeAlpaca (Chaudhary
2023
dataset.
To standardize the diverse styles and formats, all instruction tuning datasets are converted chatbot-style schema. This includes adding two special tokens, < Juserr | and < [Jassistant | before user inputs and assistant (i.e., language model) outputs, respectively. We also introduce ken < 
 to indicate the end of each utterance or response. During training, only the sequences following the <| assistant > token and before the next < user token are used for loss computation. An illustrative example can be found in the Figure I of Wang et al. (2023).
Finetuning Datasets The finetuning datasets utilized our study are detailed as follows:
SuperNI (Wang et al.
2023 ) This corpus encompasses a diverse array of NLP tasks related
to instructions, and provided under the Apache-2.0) license.
2023): This dataset consolidates multiple existing NLP datasets, and enhances them with various data augmentations, as outlined in Chung et al. (2022). The resulting dataset is also available under the Apache-2.0 license.
Flan V2 (Longpre et al.
CoT (Longpre et al. 2023): ): This dataset includes annotations for chain-of Slain-offthhught reason- ing (Wei al. 2022). Following Wang et al. (2023). we utilize the CoT mixture derived
from the Flan V2 dataset.
Designed specifically for code generation, this dataset is created using the Alpaca method (Taori et al. (202),, and is released under the 2po license.
CodeAlpaca (Chaudhary) 2023):
Evaluation Datasets.
The evaluation datasets utilized in our study are detailed follows:
MMLU (Hendrycks et al. 202I): This benchmark assesses models' factual knowledge through collection of multile-choice questions across 57 subjects, including STEM, hu- manities, and social sciences, with varying difficulty levels from elementary to profes- sional. In our evaluation, we report the exact match (EM) score under zero-shot setting.
GSM (Cobbe et al. 2021): This corpus evaluates multi-step mathematical reasoning, and contains 8.5K high-quality grade school math problems, among which 1K test items are created by human writers. These problems require 2 to 8 steps of elementary arithmetic to solve. We assess models using 8-shot examples and -f-ih-of-thoughts (CoT), reporting the EM score of the final number in the models' responses.
BBH (Suzgun et al.,
2022): This suite consists of 23 challenging tasks from BIG-
Bench (Srivastava al.
2022 aiming evaluating general multi-step reasoning abilities
14

---

Published as a conference paper at ICLR 2025
of language models. These tasks are specifically selected based prior evaluations which earlier models do not outperform average human raters. Our evaluation employs official few-shot examples without chain-of -hhouughh (Direct), and reports the EM score.
TyDi QA (Clark al.) 2020): It is multilingual question-answeriing answering dataset featuring 204K question-answer pairs in 11 topologically diverse languages, collected directly in each language. This benchmark evaluates models' multilingual performance. We adopt the gold passage (GP) setting, where the correct answer is provided in reference passage, utilize one-shot prompting, and report both EM and Fl scores.
This dataset includes 164 programming problems, each with function signature, docstring, body, and unit tests, serving as a benchmark for as- sessing models' coding capabilities by measuring the functional correctness of synthesized programs from docstrings. We report the pass@) metric using zero-shot prompting with sampling temperature of 0.1.
HumanEval (Chen et al. 2021):
A.2 HYPERPARAMETER CONFIGURATIONS
We utilize LLaMA2-7B, 13B (Tourron et al. 2023), and LLaMA3.2-3B (Dubey et al. 2024) as the foundation models for our experiments, which are conducted on single NVIDIA AT00-40G GPU. Each experimental configuration is repeated two times with the random seed as and respectively, before reporting the average performance. The detailed procedures for finetuning and evaluation are outlined below.
Finetuning Setup. To reduce memory cost during finetuning, we follow the approach outlined in QLoRA (Dettmers et al.] (023), loading all pretrained models in 4-bit NormalFloat format and utilizing a Paged Adam Optimizer. Specifically, we utilize 4-bit quantized versions of LLaMA2- 7B, 13B, and Llama3. 2-3B models in our experiments. Next, within the Transformer blocks, including the query, key, value, output, up, gate, and down projection weights, setting the scaling factor we also follow the configuration from Wang et al. (202b) to set a batch size 16 and the maximum sequence length 512, truncating samples during preprocessing if needed. We also cap the maxi- mum gradient norm at 0.3 to enhance training stability. LLaMA2-7B and 13B undergo 10,000 steps of finetuning using a linear learning rate scheduler with a warmup ratio of 3%, while LLaMA3.2- 3B is finetuned for one epoch in each task. Additionally, we search for the optimal learning rate for LoRA, and apply this value both LoRA and MoS. Specifically, with a LoRA rank of 8, we search for the best learning rate from {2e-5, 5e-5, le-4, 2e-4, 5e-4, 1e-3, 2e-3]. Our preliminary experiments demonstrate that 2e-4 performs the best.
apply LoRA to all linear layers
16 and the dropout rate to 0.1. For more efficient finetuning,
Evaluation Setup. For inference, we leverage vLLM (wwon al. 2023). an efficient inference and serving library for LLMs, which significantly speeds up the generation process with minimal impact on performance. We also employ greedy decoding with a maximum length of 512. For the details of scoring calculations, we closely follow the GitHub repository of TULU
B ADDITIONAL ANALYSIS
B.1 THEORETICAL MOTIVATIONS DIFFERENTIATION STRATEGIES
As shown in Sec. 2 differentiation reverses the detrimental effects of the the pure sharing mecha- nism. Here we demonstrate the design motivation of each specific differentiation strategies under the guidance of combinational diversity. Specifically, we approximately conceptualize differentia- tion as the combinational diversity (i.e., number of potential combinations) of each low-rank matrix pair. Aligning with the notations in Sec. let L denote the number of Transformer blocks, the rank, e the equivalent rank of LoRA terms of trainable parameters, and / the number of shards per vector. In the case of pure sharing, the potential combinations for each low-rank matrix pair can be expressed as C = 1, since all the parameters are shared in the same way for each pair. Subset selection enhances the number of combinations to C. by selecting a subset of vector pairs for each
instruct
15

---

Published as a conference paper at ICLR 2025
low-rank matrix pair. Pair dissociation rapidly increases the combinational diversity by separating vector pairs into distinct pools. allowing for independent selection of vectors for each matrix in low-rank pair, resulting in combination count of C. C. Vector sharding, which breaks down vectors into smaller shards, further amplifies the number of combinations to C,1 x Crte given that Che < Ciee when r < Le and 1 > 1. Compared to LoRA, pure sharing shares all the parameters across layers, resulting in performance degradation. Shard privatization, however, partially reverses this trend by dividing the global pool into public and private sections for improved differentiation.
B.2 FURTHER EXPERIMENTS ON DIFFERENTIATION
Compared to the pure sharing baseline, random scaling only introduces noise to the initialization of scalers, resulting in slight performance improvement. As a more aggressive measure, subset selection randomly masks specific vector pairs, and outperforms pure sharing remarkably. To further confirm this conclusion, we also extend these experiments to the LLaMA3.2-33 model (Dubey al., 2024). As detailed in Table 4 random scaling still exhibits slight improvement over pure sharing, while subset selection boosts the performance with margin over 0.8% on average.
Table 4: Results of LLaMA3.2-3B with different sharing and differentiation methods across diverse instruction following datasets. "+ Random Scaling" and "+ Subset Selection" denote the individual integration of them into the "Pure Sharing" scheme, respectively.
MMLU
BBH
GSMSK
TyDi QA
HumanEval
Method
Rank # Param.
Avg.
EM
EM
EM
F1
EM
P@1
LoRA
2
3.04M
51.97
37.07
33.74
62.59
44.39
31.17
43.49
Pure Sharing
56
3.04M
51.25
38.73
31.92
61.86
45.17
30.46
43.23
+ Random Scaling
56
3,04M
51.35
38.81
32.75
62.18
45.11
30.52
43.45
+ Subset Selection
3.04M
51.86
39.88
33.89
62.28
45.31
31.14
44.06
B.3 ROBUSTNESS ANALYSIS
Performance Robustness. We further repeat the experiments with 4 random seeds (i.e., 0, 1, 2, 3) and the LLaMA3.2-3B model. As shown in Table MoS with an equivalent parameter budget to LoRA with the rank of 8 performs competitively with LoRA with the rank of 64. This validates an 8x parameter reduction achieved by MoS again, aligning with our previous conclusions. Mean- while, compared to LoRA, MoS exhibits comparable (even lower average) standard deviations, and provides higher average value, indicating its similar stability and better performance.
Hyperparameter Robustness. To demonstrate the robustness of hyperparameters, we perform grid search for the configurations of private rank and shard number vector with 4 seeds (i.e., 0, 1, 2, 3) on the LLaMA3.2-3B model and BBH benchmark. increases to provide more differentiation, the optimal private rank tends to decrease. Generally, shard numbers of or yield the best results. From another perspective, for any given private rank, there always exists suitable range of shard numbers that consistently produce remarkable results (i.e., 2 39.8%), demonstrating the robustness of private rank.
listed in Table as the number of shards
Table 5: Results of LLaMA3.2-3B across multiple ceeruction-following datasets using different methods. The abbreviations "# Param." and "Avg." refer to "Parameter Count" and "Average", respectively, while the subscripts denote the standard deviations.
MMLU
BBH
GSM8K
TyDi QA
HumanEval
Method Rank # Param.
EM
EM
EM
F1
EM
P@1
Avg.
12.16M
$2.35_o.5 38.74+0.80 89 37.19+1.59 63.4641106
16.0
30.94 44.7940.8
LoRA
97.26M
52.36_0.23 +0.50
37.98 64.0441.51 46.53
31.83
45.41
MOS
12. 16M
52.41
40. 1..01
37.8920.71
63.78
46.31 +1.21 31.78.0.23 45.38
16

---

Published as a conference paper at ICLR 2025
Significance Test. Furthermore, conduct the significance test between LoRA and MoS Rank = 2 and across all benchmarks in our main experiments on the LLaMA2-7B model. As shown in Table 7 the p-values between LoRA and MoS with both high and low trainable parameter budgets are smaller than 5%, indicating the statistical significance our results.
Table 6: Results of MoS with the LLaMA3.2-3B model and BBH benchmark across different shards per vector and private ranks.
Private Rank
Shards Vector
3
5
38.9
39.3
39.1
39.3
2
38.6
39.3
39.7
39.5
39.3
39.8
40.0
39.6
8
39.6
39.8
39.6
38.8
39.8
39.3
39.3
38.6
Table 7: P-values of the significance test of LoRA and MoS on the LLaMA2-7B model across two different trainable parameter budgets.
Method
Rank
Param. p-value
LoRA vs. MoS 2
5.00M 19.99M
0.03%
LoRA vs. MOS 8
1.29%
C LIMITATIONS
Slightly more GPU consumption is required for finetuning. Compared to LoRA with the same trainable parameter count, MOS increases the rank several times, leading to similar GPU consump- tion to LoRA with the same rank. However, MoS does provide better performance. We also finetune the LLaMA3.2-3B model for one epoch across various tasks, and record the time consumption of LoRA and MoS in Table 8 It is worth noticing that MoS only incurs 2.80% more finetuning time.
The routing mechanism may incur more inference latency. Targeting this drawback, we give up existing activation-base routing mechanisms, because the routing operation has to wait for the activations, resulting in larger latency. Instead, we intentionally adopt the index-based routing mech- anism so that precomputing can be used prepare the whole low-rank matrices in parallel to the computation of preceding transformer blocks. This could circumvent the latency of routing opera- tion, and apply all existing inference techniques of LoRA.
Table 8: Comparison of fine-tuning time consumption (in hours) between LoRA and Mos
Method
Rank
Param.
MMLU
BBH
GSM8K
Codex-Eve
Avg.
LoRA
8
12.16M
1.50
1.47
1.82
0.21
1.25
MoS
8
12.16M
1.54
1.52
1.86
0.22
1.29
Duee the same finetuning process as MMLU benchmark, TyDi QA is neglected here avoid repetitive computation.
17


Reviews:
[{"review_id": "gMWR9MSvhk", "reviewer": "Reviewer_m4F3", "summary": "The paper investigates a more lightweight solution than LoRA in order to serve a large number of finetuned models at the same time. Based on a finding that excessive sharing may hinder model performance, the authors believe that differentiation is necessary to reverse the detrimental effects of pure sharing. The paper proposes Mixture of Shards (MoS) that incorporates both inter-layer and intra-layer sharing with four differentiation strategies.\n\n- Subset selection: randomly choose r pairs of vectors at the beginning for each layer\n- Pair dissociation: separate each pair into two pools, where each vector in a pair will be sampled independently\n- Vector sharding: shard the global pool into n parts and concatenate sampled vectors from each shard\n- Shard privatization: reserve a private segment for exclusive use for each matrix", "questions": "- In Section 3.3, is $I_a^k \\in \\mathbb{R}^r$ or $\\mathbb{N}^r$\n- What do the 4/8, 16/32 (or \"increasing the rank to 4 or 8\") in Table 2 mean?\n- Many implementation details are missing - what's the pool size, how many shards, breakdown of private & public segments, etc.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The method is more general than LoRA, making LoRA a special case when there is no global pool.\n- The authors provide ablation study for each of the differentiation strategies (except subset selection), showing the efficacy of each strategy.\n- Overall, I find the finding about sharing & differentiation makes sense and the motivation is clear. Each differentiation strategy is proposed to keep the number of parameters unchanged but increase level of differentiation between each layer.", "weaknesses": "Overall, the paper is well written. There are some minor details that can be improved.\n- Figure 2 can be more accurate and following the main text better. There is no mentioning of router \"R\" in the main text. Notations like $A^{pub}$, $A^{pri}$, $B$, $I$, $m_{ij}$ can be used to make it clearer.\n- Index(.) could be replaced by a better notation since .index(.) can be understood as the index of some element in an array.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gy2WNCCjqe", "reviewer": "Reviewer_opL8", "summary": "The paper introduces a novel fine-tuning method called **Mixture of Shards (MoS)**, which aims to significantly improve parameter efficiency in adapting large language models for customized applications. As large language models (LLMs) continue to scale, there is a growing need for parameter-efficient fine-tuning techniques to manage the high GPU memory overhead associated with serving multiple customized models simultaneously. Traditional approaches, such as Low-Rank Adaptation (LoRA), reduce resource consumption by updating pretrained weights with trainable low-rank matrices, but they still encounter scalability and memory limitations when applied to large models and extensive user customization. MoS offers a solution that retains the advantages of LoRA while achieving greater parameter efficiency through innovative parameter sharing and differentiation mechanisms.\n\nThe central concept behind MoS is to combine **inter-layer and intra-layer parameter sharing** in a single framework. This sharing is further enhanced by four lightweight differentiation strategies designed to counteract potential performance degradation from pure parameter sharing. These strategies include **subset selection**, **pair dissociation**, **vector sharding**, and **shard privatization**, each providing unique ways to increase the diversity and exclusivity of shared parameters across layers. By using a **Mixture-of-Experts (MoE)-like routing mechanism**, MoS selects and concatenates specific shards from a global parameter pool, thereby achieving efficient memory usage while maintaining high model performance.\n\nIn terms of experimental validation, the paper presents extensive evaluations on various NLP tasks, including factual knowledge (MMLU), multilingual question-answering (TyDi QA), mathematical reasoning (GSM8K), multi-step reasoning (BBH), and coding (HumanEval). The experiments demonstrate that MoS outperforms LoRA and other baseline methods in parameter efficiency, particularly under limited parameter budgets. MoS achieves approximately eightfold parameter savings compared to standard LoRA configurations, making it a promising approach for scenarios requiring numerous custom models.\n\nAn ablation study further examines the importance of each differentiation strategy, showing that components like pair dissociation and shard privatization provide substantial gains in efficiency, while vector sharding offers incremental improvements. The study reinforces the necessity of each differentiation strategy in achieving the performance and efficiency benefits observed with MoS. Additionally, a scalability analysis using the larger LLaMA2-13B model demonstrates that MoS maintains its advantages on a larger scale, further underscoring its robustness and suitability for high-capacity models.\n\nThe paper positions MoS as an important step forward in parameter-efficient fine-tuning. MoS’s compatibility with LoRA-based infrastructure and its ability to serve multiple customized models simultaneously without substantial memory overhead make it practical for real-world deployment. The findings provide insights into the trade-offs and design considerations of parameter sharing, offering a valuable resource for researchers and practitioners working on efficient model adaptation techniques. The paper’s detailed methodology, comprehensive experimentation, and focus on parameter efficiency contribute meaningfully to the broader research area of resource-efficient machine learning, addressing critical scalability issues as the field advances.", "questions": "Have you considered how MoS might interact with techniques like quantization, pruning, or dropout? Many practical deployments of large models use these methods in conjunction to manage resource constraints, and understanding how MoS might complement them would add value. If feasible, a brief experimental analysis or discussion on this integration would enhance the paper’s relevance for real-world applications.\n\nCould you discuss potential limitations of MoS, such as scenarios where the method may underperform or require additional tuning? For example, does MoS have any specific limitations when applied to domains with high variability in representation needs across layers? A discussion on this would offer a more balanced perspective, helping readers assess the suitability of MoS in various contexts.\n\nHas MoS been evaluated for its impact on inference latency, especially in multi-model serving scenarios?  \n\nAre there limitations to the size or complexity of models that MoS can handle effectively? For example, do the benefits of MoS start to diminish for models larger than LLaMA2-13B, or do you anticipate any challenges in scaling it to models with trillions of parameters?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper provides solid technical grounding for the Mixture of Shards (MoS) method, with each component—inter-layer and intra-layer sharing and differentiation strategies. The Mixture of Shards (MoS) approach is a novel, well-motivated response to the growing need for efficient fine-tuning techniques for large models. By blending inter-layer and intra-layer sharing with lightweight differentiation strategies, the paper introduces a resource-efficient method that extends beyond existing parameter-sharing methods like LoRA, VeRA, and PRoLoRA. This innovative combination of techniques in MoS is practically a valuable approach. \n\nThe experimental design is comprehensive and addresses key aspects of parameter efficiency, memory usage, and model performance across a range of NLP benchmarks (e.g., MMLU, GSM8K, TyDi QA). The thorough ablation study underscores the necessity of each differentiation strategy (subset selection, pair dissociation, vector sharding, and shard privatization) and supports the paper’s claims about MoS’s efficiency. The paper also includes scalability tests, demonstrating MoS’s robustness on larger models, such as LLaMA2-13B, reinforcing its applicability to current large model architectures.\n\nMoS integrates four nearly cost-free differentiation strategies—subset selection, pair dissociation, vector sharding, and shard privatization—to counteract the performance limitations of pure parameter sharing. These strategies is carefully designed to enhance the diversity and exclusivity of shared parameters, which contributes to the robustness and performance of the method. \n\nThe paper includes rigorous experimentation across diverse NLP benchmarks, such as MMLU ( Massive Multitask Language Understanding for factual knowledge), TyDi QA (multilingual question-answering), GSM8K (for mathematical reasoning), BBH (Big-Bench-Hard for multi-step reasoning), and HumanEval. These benchmarks test the model on factual knowledge, multilingual capabilities, mathematical reasoning, general reasoning, and coding. The results demonstrate MoS’s parameter efficiency and effectiveness compared to baseline methods, making a strong case for its practical utility. The parameter savings—approximately eightfold compared to standard LoRA—are significant, supporting the method’s scalability. This reduction substantially alleviates the memory burden, enabling more efficient model customization and serving without sacrificing performance, which is particularly valuable in settings requiring multiple concurrent models.\n\nThe paper is well-structured, with a logical flow that introduces the problem, presents the MoS solution, and discusses experimental results comprehensively. The clarity of the writing is generally good, though the differentiation strategies could benefit from additional diagrams or illustrations to aid in understanding for a wider audience.", "weaknesses": "While MoS is evaluated on a range of NLP tasks, the paper does not sufficiently analyze the method’s performance across various model architectures or specific task categories (e.g., multilingual tasks, code generation) where parameter efficiency and differentiation strategies could have different impacts. A breakdown showing how MoS performs on individual tasks, especially ones that are highly memory-intensive, would offer a clearer picture of its advantages and limitations across diverse NLP applications. \n\nThe ablation study is a strong point but could be enhanced by further exploring each differentiation strategy’s scaling potential. For instance, while the study confirms the individual benefits of subset selection, pair dissociation, vector sharding, and shard privatization, it doesn’t analyze the interactions or scalability of these strategies as model or task complexity increases. Additional experiments showing the performance impact of these strategies in larger configurations or different combinations would make the study more informative for readers looking to fine-tune MoS to specific needs.\n\nThe paper would benefit from a section discussing the potential limitations of MoS in specific scenarios. For instance, the effectiveness of MoS might be reduced when applied to tasks with low data diversity or high variance in representational needs across layers. Discussing scenarios where MoS might underperform or require adaptation would provide a more balanced view and help users assess when MoS is a suitable choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R9MKuaWcnj", "reviewer": "Reviewer_Gvvt", "summary": "This paper introduces Mixture of Shards (MoS), a sharded adaptation of LoRA designed to achieve greater parameter efficiency by leveraging parameter sharing across layers and within layers. MoS not only reduces the number of parameters required compared to traditional LoRA but also mitigates the potential performance degradation associated with excessive sharing. This is achieved through four strategies: subset selection, pair dissociation, vector sharding, and shard privatization, which ensure that each shared parameter can adapt to specific model requirements. MoS demonstrates a further reduction in trainable parametric usage, allowing more scalable deployment of LoRA-based models.", "questions": "1. For the experiments conducted with two runs using seeds 0 and 1, could you provide the individual performance results for each run? Additionally, were any further experiments conducted with different seeds to assess the robustness of the results?\n2. How does the random initialization impact the performance of MoS? Given the reliance on randomness, are there specific initialization settings or hyperparameters that consistently yield better results?\n3. What criteria, if any, were used to decide the number of shards in the global pool, and how sensitive is the model’s performance to this choice?\n4. Were there any specific cases where certain differentiation strategies (e.g., subset selection, pair dissociation) proved more beneficial than others?\n5. How does the computational overhead of MoS compare to traditional LoRA during training and inference, especially with regard to memory usage and GPU hours?\n6. Since MoS integrates multiple strategies, are there any known trade-offs between parameter savings and performance across tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. MoS combines subset selection, pair dissociation, vector sharding, and shard privatization to reduce parameters while maintaining performance.\n2. Demonstrates an eightfold parameter reduction compared to traditional LoRA with empirical support.\n3. Provides insights into the contributions of each differentiation strategy.", "weaknesses": "1. Although the paper introduces subset selection, it lacks criteria for choosing subsets; the selection is randomly initialized and fixed throughout training.\n2. The MoS approach is primarily a combination of various techniques rather than a cohesive, unified method.\n3. The MoS approach introduces significant randomness, making it challenging to determine if the improvements result from the design or from random variations. A test of significance could strengthen these claims.\n4. The paper includes limited ablation studies for MoS, making it difficult to isolate and understand the contributions of each individual strategy in the overall design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C5lQCuHve1", "reviewer": "Reviewer_doAZ", "summary": "This paper proposes Mixture of Shards (MOS), a LoRA-based method designed to reduce trainable parameters while maintaining performance for LLMs. MOS combines inter and intra layer sharing mechanisms with MoE-like routing system for parameter selection. It also introduces four “differentiation strategies”: subset selection, pair dissociation, vector sharding, and shard privatization (to add diversity and prevent performance degradation from parameter sharing). The authors claim that MOS achieves about an 8x reduction in parameters compared to LoRA while retaining competitive performance.\n\nMOS method proposes leveraging both VeRA-like inter-layer parameter sharing and PRoLoRA-like intra-layer parameter sharing. It is proposing “Global Sharing Scheme” where each adapted layer across the Transformer creates its low-rank matrices (A and B) using shards from a globally shared pool selected by MoE-like routing.\n\n“Differentiation Strategies” used in MOS:\n\n-Subset selection - selects a subset of vector pairs per transformer block\n\n-Pair dissociation - separates vector pairs into two different pools to create unique combinations for each block.\n\n-Vector sharding - breaks down vectors into smaller shards, which are then concatenated.\n\n-Shard privatization - divides the global pool into public and private sections.", "questions": "Questions:\n\n-What’s the reason behind choosing the specific differentiation strategies? How was each expected to impact performance, and was the decision based on empirical results?\n\n-How exactly are vectors selected from the global sharing pool?\n\n-Does the MOS method affect finetuning time? (given the added complexity of MOS)\n\n-Do you plan to release the code? This is a complex framework, and code would be very useful\n\n-Can you elaborate on the statement that differentiation “reverses the detrimental effects of sharing”? Is there any theoretical support for MOS’s design?\n\n-Why was standard deviation not provided for the averages?\n\n-“Differentiation” is typically associated with gradient computation in deep learning, which might cause confusion. I’d consider a different name.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-The paper has solid motivation.\n\n-Authors propose an intuitively sound idea of using a combination of inter layer and intra layer sharing with MoE-like routing.\n\n-The authors claim that MOS is the first method to use an MoE-like mechanism for parameter-efficient fine-tuning in a single-task LoRA.\n\n-The comparisons with LoRA, VeRA, and PRoLoRA are relevant baselines for MOS performance.\n\n-I like the design of the initial experiment (Table 1) - it's good to back up and motivate the method (though I have some comments mentioned in the weaknesses).", "weaknesses": "I think that the idea behind MOS is interesting and worth exploring, but it needs more rigorous experiments and analysis, along with a clearer explanation of the method and design choices. Without the code, reproducibility is challenging. \n\nI would like to understand why we would select this method over simpler methods like VeRA that provide good results. Does this method offer enough benefits to justify its complexity? The complexity should be justified, and any impact on finetuning overhead should be mentioned.\n\nWeaknesses:\n\n-The motivation for the specific “differentiation” strategies could be clearer. The authors mention that these strategies help maintain representational power, but this is very high-level and lacks theoretical support.\n\n-The MoE-like routing mechanism for parameter selection isn’t clearly explained, making it hard to reproduce. What exactly is the routing algorithm? Were other approaches tested?\n\n-The paper only evaluates MOS on instruction-following tasks.\n\n-The comparison with VeRA isn’t entirely fair, as MOS uses more parameters than VeRA. I understand that VeRA can have practical limitations in increasing parameters, but could we reduce the MOS parameter count to match VeRA?\n\n-Standard deviations are not provided.\n\n-The initial experiment (Table 1) is interesting, but the conclusion about random scaling doesn’t seem fully justified - this strategy shows very minimal improvement and might not be statistically significant. For subset selection, could more models and seeds be tested to confirm the results?\n\n-Could we add some additional models? Even smaller ones could help validate MOS’s performance. The current results are limited to LLaMA2-7B and LLaMA2-13B, with minimal gains for the latter, which may not justify MOS complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)