MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards

## Abstract

Abstract The rapid scaling of large language models necessitates more lightweight finetuning methods to reduce the explosive GPU memory overhead when numerous customized models are served simultaneously.
Targeting more parameter-efficient low-rank adaptation (LoRA), parameter sharing presents a promising solution. Empirically, our research into high-level sharing principles highlights the indispensable role of differentiation in reversing the detrimental effects of pure sharing.
Guided by this finding, we propose Mixture of Shards ( MoS ), incorporating both inter-layer and intra-layer sharing schemes, and integrating four nearly cost-free differentiation strategies, namely subset selection, pair dissociation, vector sharding, and shard privatization. Briefly, it selects a designated number of shards from global pools with a Mixture-of-Experts (MoE)-like routing mechanism before sequentially concatenating them to low-rank matrices.
Hence, it retains all the advantages of LoRA while offering enhanced parameter efficiency, and effectively circumvents the drawbacks of peer parameter-sharing methods.
Our empirical experiments demonstrate approximately 8 × 8\times 8 × parameter savings in a standard LoRA setting. The ablation study confirms the significance of each component.
Our insights into parameter sharing and MoS method may illuminate future developments of more parameter-efficient finetuning methods.

## 1 Introduction

With the remarkable capabilities,
large language models (LLMs) (Dubey et al., ; Team et al., ; Jiang et al., ) are adapted for
various downstream applications (Shao et al., ; ).
Among parameter-efficient finetuning (PEFT) methods (Houlsby et al., ; Liu et al., ),
low-rank adaptation (LoRA) (Hu et al., ) reparameterizes the weight updates with two trainable low-rank matrices, and has gained the most popularity
with
reduced resource consumption,
competitive performance,
low-cost task switching, etc.
Nevertheless, with the continuous applicability of scaling law (Achiam et al., ), the size of LLMs increases rapidly for even better performance, intensifying the need for more lightweight finetuning methods.
For instance, GPT4o allows customization to its vast user base(^1^11[https://openai.com/index/gpt-4o-fine-tuning/](https://openai.com/index/gpt-4o-fine-tuning/)), incurring intense challenges to serve numerous customized models simultaneously.
Assuming a scenario with a Llama2-70B-sized model and 10,000 active users, each allocated a LoRA module with the rank of 16, only the parameters of LoRAs would occupy 3.36 TB of GPU memory, an onerous burden for service providers.
This underscores the need for a more lightweight solution than LoRA, while retaining its advantages.

As a promising solution, parameter sharing has been extensively explored
(Kopiczko et al., ; Renduchintala et al., ; Wang et al., ).
Specifically,
VeRA (Kopiczko et al., ) adopts inter-layer sharing of frozen
matrices with trainable vectors; however, it results in limited model capacity and necessitates an excessively high rank. Subsequently, Tied LoRA (Renduchintala et al., ) mitigates these constraints by permitting shared matrices
to be trainable. Yet, its tying mechanism restricts applicability to linear layers of different dimensions.
More recently, PRoLoRA (Wang et al., ) showcases enhanced parameter efficiency by reusing submatrices within a single linear layer, thereby circumventing the above restrictions. However, its limited focus on intra-layer sharing constrains efficiency potential, and promises a hybrid approach integrating both intra-layer and inter-layer sharing.
Besides, while these methods, with their intuitive designs, have shown the benefits of parameter sharing on enhanced parameter efficiency, the fundamental principles have not been discussed yet to guide the future development of parameter sharing strategies.

Targeting enhanced parameter efficiency, we first analyze the factors influencing parameter sharing performance in the context of LoRA. Our empirical findings indicate that improving the parameter efficiency of LoRA requires a careful balance between sharing and differentiation.
While parameter sharing can significantly decrease the number of trainable parameters, the negative effects (i.e., possible performance degradation) of excessive sharing necessitate differentiation strategies to mitigate these drawbacks.
Guided by this insight, we propose a more parameter-efficient LoRA-based method called Mixture of Shards (MoS). It features a global sharing scheme to achieve a higher sharing ratio, along with four nearly cost-free differentiation techniques: subset selection, pair dissociation, vector sharding, and shard privatization.
Briefly, our method selects a designated number of shards from global pools with a Mixture-of-Experts (MoE)-like routing mechanism (Jacobs et al., ), and sequentially concatenate them to construct the low-rank matrices.

Empirically, we validate its superior performance compared to peer methods in scenarios with fixed trainable parameter count, and demonstrate an eightfold parameter reduction in performance-targeted situations.
A thorough ablation analysis of each differentiation strategy reveals that all contribute to unleashing the parameter efficiency of MoS, despite their nearly cost-free property. Notably, pair dissociation and shard privatization boost the efficiency more significantly through increased combination diversity and exclusive differentiation, respectively, whereas vector sharding provides only limited gains in diversity.
With the intuitive design, our method retains all the advantages of LoRA while circumventing the drawbacks of peer methods. The markedly improved parameter efficiency alleviates the burden on service providers when numerous LoRA-based customized models are served simultaneously.

In summary, our main contributions are as follows:

- •
We introduce a more parameter-efficient finetuning method named Mixture of Shards (MoS), which facilitates the concurrent serving of numerous customized models with the saliently reduced GPU memory overhead.
- •
Our exploration into high-level sharing principles highlights the balance between sharing and differentiation to enhance parameter efficiency, providing valuable insights for the future advancement of parameter-sharing methods.
- •
Extensive experiments empirically confirm the significant parameter savings achieved by MoS, and validate the necessity and relative importance of each differentiation strategy.
- •
To the best of our knowledge, we are the first to apply MoE-like mechanism for parameter savings in a single-task LoRA. Besides, our dissociated perspective may also inspire further parameter reduction for LoRA-based multitasking scenarios.

## 2 Sharing & Differentiation

Figure: (a) LoRA
Refer to caption: extracted/5892145/imgs/MoS_motivation-base_v3.png

**Table 1: Results of LLaMA2-7B with different sharing and differentiation methods across diverse instruction following datasets. “+ Random Scaling” and “+ Subset Selection” denote the individual integration of them into the “Pure Sharing” scheme, respectively. The details of these benchmarks are elaborated in Sec. 4.**
| Method | Rank | # Param. | MMLU | BBH | GSM8K | TyDi QA | HumanEval | Avg. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | EM | EM | EM | F1 | EM | P@1 |  |  |  |
| LoRA | 2 | 5.00M | 44.77 | 36.22 | 26.28 | 48.67 | 35.70 | 18.24 | 34.98 |
| Pure Sharing | 64 | 5.00M | 43.61 | 35.15 | 26.54 | 49.12 | 36.04 | 15.53 | 34.33 |
| + Random Scaling | 64 | 5.00M | 43.98 | 35.30 | 29.04 | 49.03 | 35.73 | 15.58 | 34.77 |
| + Subset Selection | 64 | 5.00M | 45.56 | 36.76 | 28.18 | 50.33 | 37.22 | 18.64 | 36.12 |

Drawing inspiration from the inter-layer shared VeRA (Kopiczko et al., ) and intra-layer shared PRoLoRA (Wang et al., ), we seek to create a unified sharing mechanism that leverages both perspectives, promising further enhanced parameter efficiency.
To guide this development more clearly, we start with reparameterizing the vanilla LoRA, followed by analyzing the impact of high-level factors (i.e., sharing and differentiation) on model performance under a specified number of trainable parameters.
All relevant approaches are illustrated in Figure 1 for an intuitive comparison.

#### Reparameterization of LoRA

LoRA (Hu et al., ) efficiently adapts LLMs by updating the frozen pretrained weights with low-rank decomposition. Specifically, given a pretrained weight $\mathbf{W}_{0}\in\mathbb{R}^{o\times h}$, its weight update $\Delta\mathbf{W}$ is approximated as the product of two trainable matrices $\mathbf{A}\in\mathbb{R}^{r\times h}$ and $\mathbf{B}\in\mathbb{R}^{o\times r}$, where the rank $r$ is times smaller than $h$ and $o$ for parameter saving.
Similar to Kopiczko et al. (), this product can be further reparameterized as the sum of outer products of the corresponding row and column vectors. Formally, the whole process can be presented as:

$$ $\mathbf{W}=\mathbf{W}_{0}+\Delta\mathbf{W}=\mathbf{W}_{0}+\mathbf{BA}=\mathbf{ W}_{0}+\sum_{i=1}^{r}\mathbf{b}_{i}\otimes\mathbf{a}_{i},$ (1) $$

where $\mathbf{W}$ is the updated weight, and $\mathbf{a}_{i}$ and $\mathbf{b}_{i}$ denote the $i$-th row and column vectors of low-rank matrices $\mathbf{A}$ and $\mathbf{B}$, respectively.
This reparameterization trick enables the weight update to be the linear combination of outer products of multiple vector pairs, which will be extensively utilized in our subsequent analysis.
Generally, LoRA will be applied to each linear layers in all the Transformer blocks of a model (Vaswani et al., ) for better performance (Dettmers et al., ), which is also applied to our analysis by default.

#### Pure parameter sharing does not necessarily boost up the parameter efficiency of LoRA.

Although VeRA and PRoLoRA achieve better performance than LoRA via parameter sharing, we demonstrate that this causality is not universally applicable.
To facilitate our subsequent comparisons, we introduce a straightforward sharing scheme, detailed in Sec. 3.1. Briefly, this scheme shares the low-rank matrices $\mathbf{A}$ and $\mathbf{B}$ across all Transformer blocks for each type of linear layer.
As shown in Table 1, we set the same low-level trainable parameter budget for all methods, eliminating the disturbance of parameter redundancy and enabling clear comparison of their representation capabilities.
Despite several times higher rank, which is suggested for better performance (Wang et al., ), this pure sharing scheme outperforms LoRA on GSM8K and Tydi QA tasks, but underperforms on others. This variability indicates that excessive sharing may hinder model performance, negating the positive effects of increased rank.
In other words, sharing mechanism is not the only crucial element of existing methods for enhanced parameter efficiency.

#### Differentiation reverses the detrimental effects of pure sharing mechanism.

Inspired by the varying rank demands in different positions (Zhang et al., ), we hypothesize that the pure sharing scheme, reusing the same matrices among all Transformer blocks, contradicts different representation capacity across blocks, potentially deteriorating the performance. To verify this, we devise the following two schemes to differentiate the shared matrices:

- 1.
Random Scaling.
Random scaling adds scalars to each vector pair,
which can be formulated as $\mathbf{B}\mathbf{A}=\sum_{i=1}^{r}s_{i}\cdot\mathbf{b}_{i}\otimes\mathbf{a}_{i}$, where $s_{i}$ is sampled from a normal distribution and frozen after initialization.
Hence, this operation does not introduce extra trainable parameters, but different blocks will optimize the vectors associated with their own larger scalars despite the same shared whole matrices, since these vectors influence the representation capacity more significantly than others.
- 2.
Subset Selection. Instead of scaling the vector pairs, this scheme selects a specific number of vector pairs from the shared matrices.
Specifically, the scalars in the random scaling method are substituted with Boolean values. Vectors corresponding to those marked as true are selected, while the others are neglected.
This ensures that, in most cases, any two blocks only share a subset of vector pairs so that their representations are differentiated by other distinct pairs.

As shown in Table 1, both differentiation strategies further improve model performance over the pure sharing scheme.
Specifically,
random scaling can enhance the performance of pure sharing in nearly all tasks, and ultimately achieves slightly better average results.
However, it still underperforms LoRA apparently.
In contrast, subset selection, adopting more aggressive differential operations,
reverses the degradation and consistently surpasses both pure sharing and LoRA across all benchmarks. On average, its performance exceeds that of LoRA over one percent, highlighting the importance of differentiation.

In summary, enhancing the parameter efficiency of LoRA involves a delicate balance between sharing and differentiation. On one hand, parameter sharing can significantly reduce the number of trainable parameters; on the other hand, the detrimental impacts of sharing necessitate differentiation schemes to counteract them. This insight will inform our subsequent design efforts.

## 3 Method

Based on the above analysis, we follow the guide of sharing and differentiation, and propose a more lightweight LoRA-based method, named Mixture of Shards (MoS).
Briefly, it selects a specific number of shards from global pools with a MoE-like routing mechanism, and combine them sequentially to form the low-rank matrices of LoRA.
It features a global sharing scheme for a higher sharing ratio, and four nearly cost-free differentiation measures, namely subset selection, pair dissociation, vector sharding, and shard privatization.
Their details will be elaborated below, followed by the advantage analysis.

### 3.1 Global Sharing

To maximize the parameter efficiency, we introduce a global sharing mechanism, incorporating VeRA-like inter-layer sharing and PRoLoRA-like intra-layer sharing methods.
As illustrated in Figure 2(a), this mechanism centers on a global pool for each type of linear layer.
Initially, each pool consists of multiple independently initialized and trained vector pairs, while each vector pair within the low-rank matrix pairs (i.e., $\mathbf{A}$ and $\mathbf{B}$) for each layer is sampled from this pool.
The “pure sharing” in Sec. 2 can be formed when all vector pairs from the pool are selected at each block.
In this case, the weight update can be
formulated as

$$ $\Delta\mathbf{W}=\mathbf{BA}=\mathbf{B}^{p}\mathbf{A}^{p},$ (2) $$

where $\mathbf{A}^{p}\in\mathbb{R}^{eL\times h}$ and $\mathbf{B}^{p}\in\mathbb{R}^{o\times eL}$ represents the globally shared low-rank matrices for one linear layer type(^2^22 Due to the identical operations, the notation for different types of linear layers is omitted for clarity.),
while $L$ and $e$ denote the number of Transformer blocks and an equivalent rank to that of LoRA in terms of trainable parameter count, respectively.
Due to the extremely high sharing ratio (i.e., $L$), a small number of parameters can yield a remarkably high rank, hinting positive effects on performance within a certain range (Wang et al., ; ).
Taking the LLaMA2-7B model with 32 blocks (Touvron et al., ) as an example, pure sharing can raise the rank from 2 to 64 with the same trainable parameter count as LoRA, indicating its great potential for parameter reduction.

### 3.2 Subset Selection

Figure: (a) Inter-Layer Sharing
Refer to caption: extracted/5892145/imgs/111.png

As discussed in Sec. 2,
while pure sharing scheme may lead to performance decline due to the shared weight updates across Transformer blocks,
differentiation plays an indispensable role in reversing the detrimental effects.
Compared to random scaling, subset selection demonstrates a more pronounced performance improvement, and is thus incorporated into our design for differentiation.
Let $r$ denote the number of vector pairs chosen from the shared pools, signifying the rank of each low-rank matrix during both training and inference. The weight update for the $k$-th block can be expressed as follows:

$$ $\Delta\mathbf{W}^{k}=\mathbf{B}^{k}\mathbf{A}^{k}=\mathbf{B}^{p}\Lambda^{k} \mathbf{A}^{p}=\sum_{i=1}^{eL}m_{i}^{k}\cdot\mathbf{b}^{p}_{i}\otimes\mathbf{a }^{p}_{i},$ (3) $$

where the boolean value $m_{i}^{k}$ represents the $i$-th diagonal element of the diagonal matrix $\Lambda^{k}$ for the $k$-th block. This value is randomly sampled during initialization, remains fixed during the finetuning process, and satisfies the condition $\sum_{i=1}^{eL}m_{i}^{k}=r$. Additionally, $\mathbf{a}^{p}_{i}$ and $\mathbf{b}^{p}_{i}$ denote the $i$-th row and column vectors of $\mathbf{A}^{p}$ and $\mathbf{B}^{p}$, respectively.

### 3.3 Pair Dissociation

In prior research, such as Valipour et al. (); Kopiczko et al. (), vector pairs have been regarded as the fundamental units of LoRA by default.
To foster greater differentiation, we propose a vector pair disassociation strategy that is straightforward and cost-effective, requiring no additional trainable parameters while yielding significant improvements in our empirical analysis.
Specifically, instead of a single shared pool for vector pairs (i.e., pairs of $\mathbf{a}_{i}^{p}$ and $\mathbf{b}_{i}^{p}$), we take decoupled vectors as the basic units, and separate $\mathbf{a}_{i}^{p}$ and $\mathbf{b}_{i}^{p}$ into two distinct pools.
Hence, within the $k$-th block, the vectors that form matrices $\mathbf{A}^{k}$ and $\mathbf{B}^{k}$ will be sampled independently.
Moreover, this separation naturally enables two ordered index vectors, $\mathbf{I}^{k}_{a}\in\mathbb{R}^{r}$ and $\mathbf{I}^{k}_{b}\in\mathbb{R}^{r}$, instead of unordered boolean vectors, to access vectors from the pools.
Both the dissociation and the ordered indices significantly enhance the combinatorial diversity of $\mathbf{A}^{k}$ and $\mathbf{B}^{k}$.
Formally, this process can be formulated as

$$ $\Delta\mathbf{W}^{k}=\mathbf{B}^{k}\mathbf{A}^{k}=\operatorname{Index}^{c}( \mathbf{B}^{p},\mathbf{I}^{k}_{b})\operatorname{Index}^{r}(\mathbf{A}^{p}, \mathbf{I}^{k}_{a}),$ (4) $$

where $\operatorname{Index}^{c}(\cdot)$ and $\operatorname{Index}^{r}(\cdot)$ denote the retrieval of column and row vectors, respectively, based on the given indices.

### 3.4 Vector Sharding

Drawing inspiration from database sharding(^3^33[https://en.wikipedia.org/wiki/Shard_(database_architecture)](https://en.wikipedia.org/wiki/Shard_(database_architecture))),
we further refine the basic unit of the shared pool to vector shards, where $l$ sampled shards are concatenated into a single vector.
As illustrated in Figure 2(b), two vectors corresponding to the indices “0” and “3” are retrieved and combined to form the first rank of the low-rank matrix.
Similar to vector disassociation, this nearly cost-free operation further promotes the diversity and differentiation of low-rank matrices. Subsequent experiments will confirm its beneficial effects.
The mathematical formulation remains unchanged, with the exception that the dimensions of $\mathbf{I}^{k}_{a}$ and $\mathbf{I}^{k}_{b}$ are adjusted to $l\times r$.

### 3.5 Shard Privatization

As another differentiation strategy, we partition the global pool into two segments: public and private. As indicated by their names, the public segment remains shared, whereas the private part is exclusively accessible to only one matrix. This privatization improves the differentiation of weight updates among blocks, as expressed by

$$ $\Delta\mathbf{W}^{k}=\mathbf{B}^{k}\mathbf{A}^{k}=\operatorname{Index}^{c}( \operatorname{Concat}(\mathbf{A}^{pub},\mathbf{A}^{pri}),\mathbf{I}^{k}_{b}) \operatorname{Index}^{r}(\operatorname{Concat}(\mathbf{B}^{pub},\mathbf{B}^{ pri}),\mathbf{I}^{k}_{a}),$ (5) $$

where $\mathbf{A}^{p}$ in Eq. 4 is substituted by the concatenation $\operatorname{Concat}(\cdot)$ of the public pool $\mathbf{A}^{pub}$ and the private pool $\mathbf{A}^{pri}$. Similarly, $\mathbf{B}^{p}$ is replaced by the he concatenation of $\mathbf{B}^{pub}$ and $\mathbf{B}^{pri}$. Besides, the private pools are achieved through indices exclusively owned by blocks, maintaining a unified operation with public pools.

#### Initialization.

Based on the shard size and the specified trainable parameter count, we first instantiate the global pools for various linear layer types, respectively.
Following standard LoRA practices, we initialize $\mathbf{B}^{pub}$ and $\mathbf{B}^{pri}$ to zero, ensuring consistency with the pretrained model at the start of finetuning. As adopted in PRoLoRA (Wang et al., ), we adjust the sampling boundaries of $\mathbf{A}^{pub}$ and $\mathbf{A}^{pri}$ to align with the vanilla LoRA. Subsequently, we randomly initialize the index matrices $\mathbf{I}^{k}_{a}$ and $\mathbf{I}^{k}_{b}$, and ensure the private shards are sampled only once. The training and inference procedures remain identical to those of LoRA, except that all low-rank matrices are constructed from the index matrices and global pools.

### 3.6 Advantage Analysis

Compared to LoRA, MoS only modifies the source of low-rank matrices to concatenated shards from the global pools. This adjustment retains the advantages of LoRA, such as low-cost switching and linear properties. The former allows for swapping only the finetuned weights instead of all parameters, facilitating the efficient handling of multiple customized models simultaneously. The latter enables MoS to be merged with pretrained weights during inference, thus eliminating inference latency.
Moreover, this source modification allows MoS to leverage existing technical support of LoRA (Sheng et al., ; Chen et al., ) with minimal effort.
Additionally, MoS offers several extra advantages over LoRA and other peer methods.

#### High Parameter Efficiency.

By integrating both inter-layer and intra-layer sharing, MoS demonstrates significantly higher parameter efficiency compared to LoRA and other baselines, as shown in Sec. 4.2. This reduction in parameters results in decreased disk space and GPU memory usage, thereby greatly alleviating the serving burden in multi-LoRA scenarios.

#### High Representation Capacity.

As shown in Table 2, even with a rank of 256 or higher, Tied LoRA and VeRA still clearly underperform other methods.
In contrast, the performance of MoS can approximately converge to that of full finetuning as do LoRA and PRoLoRA. This ensures a substantial model capacity, which is crucial for tackling challenging tasks.

## 4 Experiments

### 4.1 General Setup

Our primary experiments focus on evaluating the instruction following abilities of models, aligning closely with the general configurations outlined by Tulu (Wang et al., ) and PRoLoRA (Wang et al., ). Similarly, we adopt a comprehensive assessment protocol that involves factual knowledge, reasoning abilities, multilingual proficiency, and coding skills.
Following Wang et al. (), we select specific settings that have demonstrated positive results as evidenced by Table 7 of Wang et al. ().
Besides, we inherit their unified chatbot framework. This necessitates models to learn both specific tasks and the interaction style. The core configurations are summarized below, with further details available in the Appendix A.

#### Datasets.

We finetune base models on Super-Natural Instructions (SuperNI (Wang et al., )) dataset, and evaluate them on both Massive Multitask Language Understanding (MMLU (Hendrycks et al., )) and TyDi QA (Clark et al., ) datasets for factual knowledge and multilingual capabilities, respectively.
For the general and mathematical reasoning abilities, we finetune models on Flan V2 and its CoT split (Longpre et al., ), and conduct evaluation on Big-Bench-Hard (BBH (Suzgun et al., )) and the test set of Grade School Math (GSM8K (Cobbe et al., )) corpora, respectively.
Moreover, coding skills are evaluated on HumanEval (Chen et al., ) dataset after models are finetuned on CodeAlpaca (Chaudhary, ) dataset.

#### Baselines.

From the perspective of parameter sharing, we evaluate our approach against LoRA and other existing intra-layer or inter-layer sharing methods. A brief introduction to these baselines is provided below.

- •
LoRA (Hu et al., ) freezes the pretrained weights and decomposes the weight updates into two trainable low-rank matrices, as formulated in Sec. 2. Following the settings of QLoRA (Dettmers et al., ), LoRA is applied to every linear layer across all Transformer blocks, including query, key, value, output, up, gate, and down projection weights.
- •
VeRA (Kopiczko et al., ) also freezes pretrained weights, along with randomly initialized shared low-rank matrices, and employs decoupled scaling vectors for the differentiation of weight updates. Consistent with LoRA, VeRA is also applied to all linear layers.
- •
Tied LoRA (Renduchintala et al., ) ties the down projection matrices across all query, key, and value projection weights. It shares trainable low-rank matrices across Transformer blocks for parameter savings, while updating the separate scaling vectors for differentiation.
- •
PRoLoRA (Wang et al., ) recently provides an intra-layer solution aimed at enhancing parameter efficiency. It replicates the sub-matrices of LoRA multiple times before differentiating them with partial rotation operations.

### 4.2 Main Results

**Table 2: Results of LLaMA2-7B across multiple instruction-following datasets using different methods. The abbreviations “# Param.” and “Avg.” refer to “Parameter Count” and “Average”, respectively, while the symbols “-sp”, “-vs”, and “-pd” indicate the ablation of shard privatization, vector sharding, and pair dissociation. The notations “4/8” and “16/32” correspond to increasing the rank to 4 or 8, and 16 or 32, respectively. Underlined values indicate the best performance with 5.00M trainable parameters, while bold values denote the best results across all configurations. “${\dagger}$” indicates the results copied from Wang et al. ().**
| Method | Rank | # Param. | MMLU | BBH | GSM8K | TyDi QA | HumanEval | Avg. |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (factuality) | (reasoning) | (reasoning) | (multilinguality) | (coding) |  |  |  |  |  |
| EM | EM | EM | F1 | EM | P@1 |  |  |  |  |
| (0-shot) | (3-shot) | (8-shot, CoT) | (1-shot, GP) | (0-shot) |  |  |  |  |  |
| $\text{Vanilla (chat)}^{\dagger}$ | - | - | 41.18 | 0.00 | 3.03 | 17.40 | 0.10 | 0.64 | 10.39 |
| $\text{Vanilla (no-chat)}^{\dagger}$ | - | - | 41.53 | 33.43 | 15.47 | 49.18 | 35.35 | 13.57 | 31.42 |
| LoRA | $\text{2}^{\dagger}$ | 5.00M | 44.77 | 36.22 | 26.28 | 48.67 | 35.70 | 18.24 | 34.98 |
| $\text{8}^{\dagger}$ | 19.99M | 46.55 | 36.92 | 31.11 | 50.50 | 36.89 | 19.37 | 36.89 |  |
| $\text{16}^{\dagger}$ | 39.98M | 46.70 | 36.43 | 31.34 | 50.97 | 37.64 | 18.73 | 36.97 |  |
| 64 | 159.91M | 47.10 | 37.78 | 31.43 | 51.65 | 38.07 | 19.12 | 37.53 |  |
| $\text{VeRA}^{\dagger}$ | 256 | 1.42M | 42.51 | 35.10 | 22.69 | 48.39 | 36.38 | 18.90 | 34.00 |
| $\text{Tied LoRA}^{\dagger}$ | 280 | 4.99M | 44.36 | 35.76 | 25.47 | 50.16 | 37.15 | 18.68 | 35.26 |
| $\text{PRoLoRA}^{\dagger}$ | 4/8 | 5.00M | 45.85 | 36.45 | 27.57 | 49.94 | 36.59 | 19.75 | 36.03 |
| MoS | 4/8 | 5.00M | 46.09 | 37.29 | 28.43 | 50.21 | 37.19 | 19.12 | 36.39 |
| 16/32 | 19.99M | 47.01 | 37.79 | 30.93 | 51.71 | 38.34 | 20.00 | 37.63 |  |
| $\text{MoS}^{\text{-sp}}$ | 16/32 | 19.99M | 46.64 | 36.69 | 30.17 | 50.27 | 36.90 | 18.60 | 36.54 |
| $\text{MoS}^{\text{-vs}}$ | 16/32 | 19.99M | 46.47 | 37.52 | 31.77 | 50.90 | 37.98 | 18.69 | 37.22 |
| $\text{MoS}^{\text{-pd}}$ | 16/32 | 19.99M | 46.23 | 36.17 | 30.71 | 51.40 | 37.94 | 16.77 | 36.54 |

The main results are summarized in Table 2, showcasing the performance of various methods across multiple instruction tuning tasks.
Notably, as the rank increases from 2 to 64, the performance of LoRA consistently improves across all benchmarks, significantly surpassing that of the vanilla model.
This observation suggests that the current configurations do not introduce apparent parameter redundancy in trainable parameters, highlighting the effectiveness of the setups in demonstrating the parameter efficiency of different approaches.

In a scenario with a low-level trainable parameter count, MoS outperforms all baseline approaches. Specifically, to highlight the differences in parameter efficiency among various methods, we fix the number of trainable parameters at a low level (i.e., 5.00M, equivalent to that of LoRA with a rank of 2), and compare their performance. Naturally, more superior performance indicates higher parameter efficiency. As shown in Table 2, both the inter-layer shared Tied LoRA and the intra-layer shared PRoLoRA demonstrate better average performance than LoRA. In contrast, by incorporating both intra-layer and inter-layer sharing, our method, MoS, surpasses all of them on average, and achieves consistent improvements across all individual tasks, as indicated by the underlines.
However, similar to PRoLoRA (Wang et al., ), we do not claim to achieve greater parameter efficiency than VeRA, given its lower parameter count. However, we attempted to increase the rank of VeRA to match the identical trainable parameter count, but failed due to an out-of-memory (OOM) error. This indicates that VeRA has a very low performance-to-rank ratio, making it impractical due to significantly higher training costs for numerous customization.

In a regular setting, MoS achieves around $8\times$ savings in trainable parameters.
Considering the significance of performance on various benchmarks, we turn to evaluate the parameter savings of MoS with a performance target, where fewer trainable parameters also indicate high efficiency.
We set the performance target to align with LoRA at a rank of 64, which yields an average performance of 37.53. In comparison, with 19.99M trainable parameters (equivalent to that of LoRA with a rank of 8), MoS outperforms LoRA on four out of six metrics, while maintaining comparable average performance. This reconfirms the superior parameter efficiency of MoS, and demonstrates its around eightfold savings in trainable parameters. This can considerably alleviate the burden on service providers, when multiple LoRA-based customizations are served simultaneously.

### 4.3 Scalability Analysis

**Table 3: Results of LLaMA2-13B across multiple instruction-following datasets using different methods. The abbreviations, notions, and settings align with those of Table 2, respectively.**
| Method | Rank | MMLU | BBH | GSM | Avg. |
| --- | --- | --- | --- | --- | --- |
| $\text{Vanilla (chat)}^{\dagger}$ | - | 50.05 | 0.00 | 2.12 | 17.39 |
| $\text{Vanilla (no-chat)}^{\dagger}$ | - | 52.04 | 39.67 | 27.82 | 39.84 |
| $\text{LoRA}^{\dagger}$ | 2 | 52.78 | 42.50 | 36.47 | 43.92 |
| $\text{PRoLoRA}^{\dagger}$ | 4/8 | 53.27 | 43.12 | 38.72 | 45.04 |
| MoS | 4/8 | 53.51 | 43.65 | 40.79 | 45.98 |

To assess the robustness of MoS regarding model scale, we extend the experiments to the LLaMA2-13B model. However, we find that finetuning with vanilla LoRA does not yield consistent improvements on the TyDi QA and Code benchmarks. We hypothesize that this phenomenon might be caused by the more powerful capabilities of the base model, which diminishes the effectiveness of the training set. Hence, we exclude these tasks from our analysis.

As listed in Table 3, we impose a limited trainable parameter count (equivalent to that of LoRA with the rank of 2) to compare the representational capacities of different methods. Specifically, compared to the pretrained model, LoRA achieves a notable performance improvement with an average score of 43.92. Building on this, PRoLoRA boosts the average performance by 1%. In contrast, MoS further unleashes the potential of parameters. It consistently exhibits performance enhancements across all individual benchmarks, with another increase of nearly 1% on average. This considerable improvement reinforces MoS’s positive impact on parameter efficiency and its robustness with respect to model size.

### 4.4 Ablation Study

In this section, we conduct an ablation study on each differentiation strategy incorporated by MoS, before comparing their relative significance. As discussed in Sec. 2 and listed in Table 1, subset selection plays an indispensable role in reversing the detrimental effects of pure sharing. Since it lays the groundwork for other strategies, we will not reiterate its importance here. Results from the ablation of the other components are presented in Table 2.

#### Pair Dissociation.

Rather than using distinct indices for enhanced combination diversity, we apply the same index matrix for the low-rank matrix pair $\mathbf{A}^{k}$ and $\mathbf{B}^{k}$ within each linear layer (i.e., $\mathbf{I}^{k}_{a}=\mathbf{I}^{k}_{b}$). This results in an average performance decline exceeding one percent, emphasizing the critical role of pair dissociation and diverse combinations in improving model performance and aligning with our initial motivation.

#### Vector Sharding.

When the basic units in the global pools are reverted back to the row or column vectors of the low-rank matrices from smaller shards, only a 0.41% average decrease in performance is observed. This decline is much less than that associated with pair dissociation. Given that both components aim to enhance combination diversity, these findings imply that pair dissociation provides a more substantial boost to diversity, whereas sharding yields only marginal improvements.

#### Shard Privatization.

Different from the previous components, shard privatization enhances differentiation among low-rank matrices by allowing certain shards to be exclusively accessible to one low-rank matrix. Its ablation also leads to a performance drop exceeding one percent, highlighting the significance of privatization.

In summary, all differentiation strategies contribute to improving the parameter efficiency of MoS, despite all being nearly cost-free operations. Among them, pair dissociation and shard privatization unleash the efficiency more saliently through increased combination diversity and exclusive differentiation, respectively, while vector sharding offers only incremental gains in diversity.

## 5 Related work

#### Low-Rank Adaptation.

Low-rank adaptation (LoRA), introduced in Hu et al. (), reparameterizes weight updates with two trainable low-rank matrices while the pretrained weights remain fixed. This approach has since spurred numerous enhancements aimed at improving its effectiveness and efficiency.
AdaLoRA (Zhang et al., ), leveraging singular value decomposition, automates rank allocation by adaptively pruning vector pairs during finetuning. However, the rank variability across layers complicates the simultaneous serving of multiple LoRAs (Sheng et al., ).
Aimed at different parameter budget needs during inference, DyLoRA (Valipour et al., ) also utilizes vector pairs as fundamental units and randomly masks them throughout training, yet leading to performance degradation.
Closely aligning with our work,
VeRA (Kopiczko et al., ) implements inter-layer sharing of two frozen random matrices while updating disentangled combination vectors for each layer. However, it necessitates an excessively high rank due to limited model capacity.
Subsequently, though Tied LoRA (Renduchintala et al., ) alleviates this constraint by allowing shared matrices to be trainable, the tying mechanism on down projection matrices restricts its applicability to linear layers of different dimensions.
More recently, PRoLoRA (Wang et al., ) demonstrates enhanced parameter efficiency by reusing sub-matrices within a single linear layer, thus avoiding the above limitations. However, its narrow focus on intra-layer sharing restricts its performance potential.
Overall, these approaches typically assume vector pairs as the fundamental units for granted, and concentrate exclusively on inter-layer or intra-layer sharing, both of which limits the potential for parameter efficiency.
In contrast, our method utilizes fine-grained shards as basic units and facilitates global sharing to achieve improved parameter efficiency.

#### Parameter Sharing.

Parameter sharing has been extensively studied to minimize model sizes.
Universal Transformer (Dehghani et al., ) shares all layers of a Transformer model, before
Takase & Kiyono () further optimizes this approach with three parameter-sharing techniques across Transformer layers for higher efficiency.
DictFormer (Lou et al., ) implements a more compact model with a shared dictionary, unshared coefficients and indices.
For on-device deployment, EdgeFormer (Ge et al., ) shares attention and FFN modules, and incorporates PEFT-based layer adaptation to reduce parameter count.
Pires et al. () eliminates the decoder layer FFN and shares a single, larger FFN across the encoder, achieving notable improvements in accuracy and latency.
Recently, Cao et al. () proposes two memory-efficient methods for parameter sharing in attention heads.
Differently, our work is centered on the parameter sharing in LoRA modules, and explore the foundational principles guiding the development of parameter sharing strategies.

#### Mixture of Experts.

Mixture of Experts (MoE) was initially proposed by Jacobs et al. (). Since Zadouri et al. () replaces traditional experts with LoRA modules, various efforts have been made to perform better integration (Wu et al., ).
Aiming at multi-task learning, Mixture-of-LoRAs (MoA) (Feng et al., ) integrates multiple domain-specific LoRAs through a routing strategy, improving both individual task performance and rapid domain adaptation.
Similarly, MixLoRA (Li et al., ) constructs a sparse LoRA-based MoE model by inserting multiple LoRA experts within the FFN modules and employing independent LoRA adapters in the attention modules.
Recently, Mixture of Dyadic Experts (MoDE) (Ning et al., ) shares a down-projection matrix for different tasks and employs atomic rank-one adapters with routers for more sophisticated task specialization, enhancing the model’s multi-tasking capabilities.
In contrast, Zhou et al. () present Mixture-of-Experts with Language Priors Routing (MoE-LPR), featuring a two-stage training process to improve multilingual capability without catastrophic forgetting.
To the best of our knowledge, we are the first one to apply MoE for parameter savings within a single LoRA module, rather than focusing on multi-task or multilingual settings.

## 6 Conclusion

Targeting higher parameter efficiency via parameter sharing, we investigate high-level sharing principles, highlighting the essential role of differentiation in reversing the detrimental effects of pure sharing.
Building on this finding, we propose a more parameter-efficient LoRA-based method called Mixture of Shards (MoS), incorporating both inter-layer and intra-layer sharing schemes, and integrating four nearly cost-free differentiation strategies: subset selection, pair dissociation, vector sharding, and shard privatization.
It maintains all the advantages of LoRA while achieving higher parameter efficiency, and effectively avoids the drawbacks of peer parameter-sharing methods.
Our insights into parameter sharing and the MoS method may illuminate future developments in more parameter-efficient finetuning techniques, particularly alleviating the burden for service providers handling multiple customized models simultaneously.

## Appendix A Experiment Details

Our main experiments concentrate on assessing the instruction-following abilities of models, closely aligning with the frameworks established by Tulu (Wang et al., ).
In a similar vein, we implement a thorough evaluation protocol that encompasses factual knowledge, reasoning skills, multilingual capabilities, and programming expertise. Consistent with Wang et al. (), we select specific configurations of training and test datasets that have yielded positive outcomes, as indicated by the Table 7 of Wang et al. (). Additionally, we adopt their integrated chatbot framework, which requires models to acquire both specific tasks and interaction styles. The essential configurations are summarized below.

### A.1 Dataset Details

To evaluate various aspects of model capabilities, we perform comprehensive assessments using multiple datasets.
Specifically, we finetune models on the Super-Natural Instructions (SuperNI (Wang et al., )) dataset and then measure their performance on the Multitask Language Understanding (MMLU (Hendrycks et al., )) for factual knowledge and on the TyDi QA (Clark et al., ) dataset for multilingual abilities.
For general and mathematical reasoning evaluation, we retrain the foundation models on the Flan V2 dataset and its CoT split (Longpre et al., ). Their corresponding performances are then presented on the Big-Bench-Hard (BBH (Suzgun et al., )) and the Grade School Math (GSM (Cobbe et al., )) test splits, respectively.
Additionally, we utilize the HumanEval (Chen et al., ) benchmark to assess the coding abilities of models finetuned on the CodeAlpaca (Chaudhary, ) dataset.

To standardize the diverse styles and formats, all instruction tuning datasets are converted to a chatbot-style schema. This includes adding two special tokens, 
 and 
, before user inputs and assistant (i.e., language model) outputs, respectively. We also introduce a token 
 to indicate the end of each utterance or response. During training, only the sequences following the 
 token and before the next 
 token are used for loss computation. An illustrative example can be found in the Figure 1 of Wang et al. ().

#### Finetuning Datasets

The finetuning datasets utilized in our study are detailed as follows:

- •
SuperNI (Wang et al., ): This corpus encompasses a diverse array of NLP tasks related to instructions, and is provided under the Apache-2.0 license.
- •
Flan V2 (Longpre et al., ): This dataset consolidates multiple existing NLP datasets, and enhances them with various data augmentations, as outlined in Chung et al. (). The resulting dataset is also available under the Apache-2.0 license.
- •
CoT (Longpre et al., ): This dataset includes annotations for chain-of-thought reasoning (Wei et al., ). Following Wang et al. (), we utilize the CoT mixture derived from the Flan V2 dataset.
- •
CodeAlpaca (Chaudhary, ): Designed specifically for code generation, this dataset is created using the Alpaca method (Taori et al., ), and is released under the Apache-2.0 license.

#### Evaluation Datasets.

The evaluation datasets utilized in our study are detailed as follows:

- •
MMLU  (Hendrycks et al., ): This benchmark assesses models’ factual knowledge through a collection of multiple-choice questions across 57 subjects, including STEM, humanities, and social sciences, with varying difficulty levels from elementary to professional. In our evaluation, we report the exact match (EM) score under a zero-shot setting.
- •
GSM
(Cobbe et al., ): This corpus evaluates multi-step mathematical reasoning, and contains 8.5K high-quality grade school math problems, among which 1K test items are created by human writers. These problems require 2 to 8 steps of elementary arithmetic to solve. We assess models using 8-shot examples and chain-of-thoughts (CoT), reporting the EM score of the final number in the models’ responses.
- •
BBH  (Suzgun et al., ): This suite consists of 23 challenging tasks from BIG-Bench (Srivastava et al., ), aiming at evaluating general multi-step reasoning abilities of language models. These tasks are specifically selected based on prior evaluations on which earlier models do not outperform average human raters. Our evaluation employs 3 official few-shot examples without chain-of-thought (Direct), and reports the EM score.
- •
TyDi QA
(Clark et al., ): It is a multilingual question-answering dataset featuring 204K question-answer pairs in 11 topologically diverse languages, collected directly in each language. This benchmark evaluates models’ multilingual performance. We adopt the gold passage (GP) setting, where the correct answer is provided in a reference passage, utilize one-shot prompting, and report both EM and F1 scores.
- •
HumanEval
(Chen et al., ): This dataset includes 164 programming problems, each with a function signature, docstring, body, and unit tests, serving as a benchmark for assessing models’ coding capabilities by measuring the functional correctness of synthesized programs from docstrings. We report the pass@1 metric using zero-shot prompting with a sampling temperature of 0.1.

### A.2 Hyperparameter Configurations

We utilize LLaMA2-7B and 13B (Touvron et al., ) as the foundation models for our experiments, which are conducted on a single NVIDIA A100-40G GPU. Each experimental configuration is repeated two times with the random seed as 0 and 1, respectively, before reporting the average performance. The detailed procedures for finetuning and evaluation are outlined below.

#### Finetuning Setup.

To reduce memory cost during finetuning, we follow the approach outlined in QLoRA (Dettmers et al., ), loading all pretrained models in a 4-bit NormalFloat format and utilizing a Paged AdamW Optimizer. Specifically, we utilize 4-bit quantized versions of LLaMA2-7B and 13B models in our experiments. Next, we apply LoRA to all linear layers within the Transformer blocks, including the query, key, value, output, up, gate, and down projection weights, setting the scaling factor $\alpha$ to 16 and the dropout rate to 0.1. For more efficient finetuning, we also follow the configuration from Wang et al. () to set a batch size of 16 and the maximum sequence length to 512, truncating samples during preprocessing if needed.
We also
cap the maximum gradient norm at 0.3 to enhance training stability. Each model undergoes 10,000 steps of finetuning using a linear learning rate scheduler with a warmup ratio of 3%. Additionally, we search for the optimal learning rate for LoRA, and apply this value to both LoRA and MoS. Specifically, with a LoRA rank of 8, we search for the best learning rate from {2e-5, 5e-5, 1e-4, 2e-4, 5e-4, 1e-3, 2e-3}. Our preliminary experiments demonstrate that 2e-4 performs the best.

#### Evaluation Setup.

For inference, we leverage vLLM (Kwon et al., ), an efficient inference and serving library for LLMs, which greatly speeds up the generation process with minor influence in performance. We also employ greedy decoding with a maximum length of 512.
For the details of scoring calculations, we closely follow the GitHub repository of TULU(^4^44[https://github.com/allenai/open-instruct](https://github.com/allenai/open-instruct)).

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - In Section 3.3, is $I_a^k \in \mathbb{R}^r$ or $\mathbb{N}^r$
- What do the 4/8, 16/32 (or "increasing the rank to 4 or 8") in Table 2 mean?
- Many implementation details are missing - what's the pool size, how many shards, breakdown of private & public segments, etc.
soundness: 4
strengths: - The method is more general than LoRA, making LoRA a special case when there is no global pool.
- The authors provide ablation study for each of the differentiation strategies (except subset selection), showing the efficacy of each strategy.
- Overall, I find the finding about sharing & differentiation makes sense and the motivation is clear. Each differentiation strategy is proposed to keep the number of parameters unchanged but increase level of differentiation between each layer.
summary: The paper investigates a more lightweight solution than LoRA in order to serve a large number of finetuned models at the same time. Based on a finding that excessive sharing may hinder model performance, the authors believe that differentiation is necessary to reverse the detrimental effects of pure sharing. The paper proposes Mixture of Shards (MoS) that incorporates both inter-layer and intra-layer sharing with four differentiation strategies.

- Subset selection: randomly choose r pairs of vectors at the beginning for each layer
- Pair dissociation: separate each pair into two pools, where each vector in a pair will be sampled independently
- Vector sharding: shard the global pool into n parts and concatenate sampled vectors from each shard
- Shard privatization: reserve a private segment for exclusive use for each matrix
weaknesses: Overall, the paper is well written. There are some minor details that can be improved.
- Figure 2 can be more accurate and following the main text better. There is no mentioning of router "R" in the main text. Notations like $A^{pub}$, $A^{pri}$, $B$, $I$, $m_{ij}$ can be used to make it clearer.
- Index(.) could be replaced by a better notation since .index(.) can be understood as the index of some element in an array.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Have you considered how MoS might interact with techniques like quantization, pruning, or dropout? Many practical deployments of large models use these methods in conjunction to manage resource constraints, and understanding how MoS might complement them would add value. If feasible, a brief experimental analysis or discussion on this integration would enhance the paper’s relevance for real-world applications.

Could you discuss potential limitations of MoS, such as scenarios where the method may underperform or require additional tuning? For example, does MoS have any specific limitations when applied to domains with high variability in representation needs across layers? A discussion on this would offer a more balanced perspective, helping readers assess the suitability of MoS in various contexts.

Has MoS been evaluated for its impact on inference latency, especially in multi-model serving scenarios?  

Are there limitations to the size or complexity of models that MoS can handle effectively? For example, do the benefits of MoS start to diminish for models larger than LLaMA2-13B, or do you anticipate any challenges in scaling it to models with trillions of parameters?
soundness: 4
strengths: The paper provides solid technical grounding for the Mixture of Shards (MoS) method, with each component—inter-layer and intra-layer sharing and differentiation strategies. The Mixture of Shards (MoS) approach is a novel, well-motivated response to the growing need for efficient fine-tuning techniques for large models. By blending inter-layer and intra-layer sharing with lightweight differentiation strategies, the paper introduces a resource-efficient method that extends beyond existing parameter-sharing methods like LoRA, VeRA, and PRoLoRA. This innovative combination of techniques in MoS is practically a valuable approach. 

The experimental design is comprehensive and addresses key aspects of parameter efficiency, memory usage, and model performance across a range of NLP benchmarks (e.g., MMLU, GSM8K, TyDi QA). The thorough ablation study underscores the necessity of each differentiation strategy (subset selection, pair dissociation, vector sharding, and shard privatization) and supports the paper’s claims about MoS’s efficiency. The paper also includes scalability tests, demonstrating MoS’s robustness on larger models, such as LLaMA2-13B, reinforcing its applicability to current large model architectures.

MoS integrates four nearly cost-free differentiation strategies—subset selection, pair dissociation, vector sharding, and shard privatization—to counteract the performance limitations of pure parameter sharing. These strategies is carefully designed to enhance the diversity and exclusivity of shared parameters, which contributes to the robustness and performance of the method. 

The paper includes rigorous experimentation across diverse NLP benchmarks, such as MMLU ( Massive Multitask Language Understanding for factual knowledge), TyDi QA (multilingual question-answering), GSM8K (for mathematical reasoning), BBH (Big-Bench-Hard for multi-step reasoning), and HumanEval. These benchmarks test the model on factual knowledge, multilingual capabilities, mathematical reasoning, general reasoning, and coding. The results demonstrate MoS’s parameter efficiency and effectiveness compared to baseline methods, making a strong case for its practical utility. The parameter savings—approximately eightfold compared to standard LoRA—are significant, supporting the method’s scalability. This reduction substantially alleviates the memory burden, enabling more efficient model customization and serving without sacrificing performance, which is particularly valuable in settings requiring multiple concurrent models.

The paper is well-structured, with a logical flow that introduces the problem, presents the MoS solution, and discusses experimental results comprehensively. The clarity of the writing is generally good, though the differentiation strategies could benefit from additional diagrams or illustrations to aid in understanding for a wider audience.
summary: The paper introduces a novel fine-tuning method called **Mixture of Shards (MoS)**, which aims to significantly improve parameter efficiency in adapting large language models for customized applications. As large language models (LLMs) continue to scale, there is a growing need for parameter-efficient fine-tuning techniques to manage the high GPU memory overhead associated with serving multiple customized models simultaneously. Traditional approaches, such as Low-Rank Adaptation (LoRA), reduce resource consumption by updating pretrained weights with trainable low-rank matrices, but they still encounter scalability and memory limitations when applied to large models and extensive user customization. MoS offers a solution that retains the advantages of LoRA while achieving greater parameter efficiency through innovative parameter sharing and differentiation mechanisms.

The central concept behind MoS is to combine **inter-layer and intra-layer parameter sharing** in a single framework. This sharing is further enhanced by four lightweight differentiation strategies designed to counteract potential performance degradation from pure parameter sharing. These strategies include **subset selection**, **pair dissociation**, **vector sharding**, and **shard privatization**, each providing unique ways to increase the diversity and exclusivity of shared parameters across layers. By using a **Mixture-of-Experts (MoE)-like routing mechanism**, MoS selects and concatenates specific shards from a global parameter pool, thereby achieving efficient memory usage while maintaining high model performance.

In terms of experimental validation, the paper presents extensive evaluations on various NLP tasks, including factual knowledge (MMLU), multilingual question-answering (TyDi QA), mathematical reasoning (GSM8K), multi-step reasoning (BBH), and coding (HumanEval). The experiments demonstrate that MoS outperforms LoRA and other baseline methods in parameter efficiency, particularly under limited parameter budgets. MoS achieves approximately eightfold parameter savings compared to standard LoRA configurations, making it a promising approach for scenarios requiring numerous custom models.

An ablation study further examines the importance of each differentiation strategy, showing that components like pair dissociation and shard privatization provide substantial gains in efficiency, while vector sharding offers incremental improvements. The study reinforces the necessity of each differentiation strategy in achieving the performance and efficiency benefits observed with MoS. Additionally, a scalability analysis using the larger LLaMA2-13B model demonstrates that MoS maintains its advantages on a larger scale, further underscoring its robustness and suitability for high-capacity models.

The paper positions MoS as an important step forward in parameter-efficient fine-tuning. MoS’s compatibility with LoRA-based infrastructure and its ability to serve multiple customized models simultaneously without substantial memory overhead make it practical for real-world deployment. The findings provide insights into the trade-offs and design considerations of parameter sharing, offering a valuable resource for researchers and practitioners working on efficient model adaptation techniques. The paper’s detailed methodology, comprehensive experimentation, and focus on parameter efficiency contribute meaningfully to the broader research area of resource-efficient machine learning, addressing critical scalability issues as the field advances.
weaknesses: While MoS is evaluated on a range of NLP tasks, the paper does not sufficiently analyze the method’s performance across various model architectures or specific task categories (e.g., multilingual tasks, code generation) where parameter efficiency and differentiation strategies could have different impacts. A breakdown showing how MoS performs on individual tasks, especially ones that are highly memory-intensive, would offer a clearer picture of its advantages and limitations across diverse NLP applications. 

The ablation study is a strong point but could be enhanced by further exploring each differentiation strategy’s scaling potential. For instance, while the study confirms the individual benefits of subset selection, pair dissociation, vector sharding, and shard privatization, it doesn’t analyze the interactions or scalability of these strategies as model or task complexity increases. Additional experiments showing the performance impact of these strategies in larger configurations or different combinations would make the study more informative for readers looking to fine-tune MoS to specific needs.

The paper would benefit from a section discussing the potential limitations of MoS in specific scenarios. For instance, the effectiveness of MoS might be reduced when applied to tasks with low data diversity or high variance in representational needs across layers. Discussing scenarios where MoS might underperform or require adaptation would provide a more balanced view and help users assess when MoS is a suitable choice.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. For the experiments conducted with two runs using seeds 0 and 1, could you provide the individual performance results for each run? Additionally, were any further experiments conducted with different seeds to assess the robustness of the results?
2. How does the random initialization impact the performance of MoS? Given the reliance on randomness, are there specific initialization settings or hyperparameters that consistently yield better results?
3. What criteria, if any, were used to decide the number of shards in the global pool, and how sensitive is the model’s performance to this choice?
4. Were there any specific cases where certain differentiation strategies (e.g., subset selection, pair dissociation) proved more beneficial than others?
5. How does the computational overhead of MoS compare to traditional LoRA during training and inference, especially with regard to memory usage and GPU hours?
6. Since MoS integrates multiple strategies, are there any known trade-offs between parameter savings and performance across tasks?
soundness: 3
strengths: 1. MoS combines subset selection, pair dissociation, vector sharding, and shard privatization to reduce parameters while maintaining performance.
2. Demonstrates an eightfold parameter reduction compared to traditional LoRA with empirical support.
3. Provides insights into the contributions of each differentiation strategy.
summary: This paper introduces Mixture of Shards (MoS), a sharded adaptation of LoRA designed to achieve greater parameter efficiency by leveraging parameter sharing across layers and within layers. MoS not only reduces the number of parameters required compared to traditional LoRA but also mitigates the potential performance degradation associated with excessive sharing. This is achieved through four strategies: subset selection, pair dissociation, vector sharding, and shard privatization, which ensure that each shared parameter can adapt to specific model requirements. MoS demonstrates a further reduction in trainable parametric usage, allowing more scalable deployment of LoRA-based models.
weaknesses: 1. Although the paper introduces subset selection, it lacks criteria for choosing subsets; the selection is randomly initialized and fixed throughout training.
2. The MoS approach is primarily a combination of various techniques rather than a cohesive, unified method.
3. The MoS approach introduces significant randomness, making it challenging to determine if the improvements result from the design or from random variations. A test of significance could strengthen these claims.
4. The paper includes limited ablation studies for MoS, making it difficult to isolate and understand the contributions of each individual strategy in the overall design.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Questions:

-What’s the reason behind choosing the specific differentiation strategies? How was each expected to impact performance, and was the decision based on empirical results?

-How exactly are vectors selected from the global sharing pool?

-Does the MOS method affect finetuning time? (given the added complexity of MOS)

-Do you plan to release the code? This is a complex framework, and code would be very useful

-Can you elaborate on the statement that differentiation “reverses the detrimental effects of sharing”? Is there any theoretical support for MOS’s design?

-Why was standard deviation not provided for the averages?

-“Differentiation” is typically associated with gradient computation in deep learning, which might cause confusion. I’d consider a different name.
soundness: 3
strengths: -The paper has solid motivation.

-Authors propose an intuitively sound idea of using a combination of inter layer and intra layer sharing with MoE-like routing.

-The authors claim that MOS is the first method to use an MoE-like mechanism for parameter-efficient fine-tuning in a single-task LoRA.

-The comparisons with LoRA, VeRA, and PRoLoRA are relevant baselines for MOS performance.

-I like the design of the initial experiment (Table 1) - it's good to back up and motivate the method (though I have some comments mentioned in the weaknesses).
summary: This paper proposes Mixture of Shards (MOS), a LoRA-based method designed to reduce trainable parameters while maintaining performance for LLMs. MOS combines inter and intra layer sharing mechanisms with MoE-like routing system for parameter selection. It also introduces four “differentiation strategies”: subset selection, pair dissociation, vector sharding, and shard privatization (to add diversity and prevent performance degradation from parameter sharing). The authors claim that MOS achieves about an 8x reduction in parameters compared to LoRA while retaining competitive performance.

MOS method proposes leveraging both VeRA-like inter-layer parameter sharing and PRoLoRA-like intra-layer parameter sharing. It is proposing “Global Sharing Scheme” where each adapted layer across the Transformer creates its low-rank matrices (A and B) using shards from a globally shared pool selected by MoE-like routing.

“Differentiation Strategies” used in MOS:

-Subset selection - selects a subset of vector pairs per transformer block

-Pair dissociation - separates vector pairs into two different pools to create unique combinations for each block.

-Vector sharding - breaks down vectors into smaller shards, which are then concatenated.

-Shard privatization - divides the global pool into public and private sections.
weaknesses: I think that the idea behind MOS is interesting and worth exploring, but it needs more rigorous experiments and analysis, along with a clearer explanation of the method and design choices. Without the code, reproducibility is challenging. 

I would like to understand why we would select this method over simpler methods like VeRA that provide good results. Does this method offer enough benefits to justify its complexity? The complexity should be justified, and any impact on finetuning overhead should be mentioned.

Weaknesses:

-The motivation for the specific “differentiation” strategies could be clearer. The authors mention that these strategies help maintain representational power, but this is very high-level and lacks theoretical support.

-The MoE-like routing mechanism for parameter selection isn’t clearly explained, making it hard to reproduce. What exactly is the routing algorithm? Were other approaches tested?

-The paper only evaluates MOS on instruction-following tasks.

-The comparison with VeRA isn’t entirely fair, as MOS uses more parameters than VeRA. I understand that VeRA can have practical limitations in increasing parameters, but could we reduce the MOS parameter count to match VeRA?

-Standard deviations are not provided.

-The initial experiment (Table 1) is interesting, but the conclusion about random scaling doesn’t seem fully justified - this strategy shows very minimal improvement and might not be statistically significant. For subset selection, could more models and seeds be tested to confirm the results?

-Could we add some additional models? Even smaller ones could help validate MOS’s performance. The current results are limited to LLaMA2-7B and LLaMA2-13B, with minimal gains for the latter, which may not justify MOS complexity.

Meta-review: The paper presents a novel method called Mixture of Shards that combines inter-layer and intra-layer sharing with lightweight differentiation strategies to enhance parameter efficiency and model performance for large models. The MoS method outperforms existing approaches like LoRA by incorporating four differentiation strategies—subset selection, pair dissociation, vector sharding, and shard privatization—offering a resource-efficient solution that scales well with large models. The comprehensive ablation study and experiments across multiple NLP benchmarks demonstrate its robustness and significant parameter savings.

All reviewers acknowledge the contributions of the paper, with feedback generally leaning towards acceptance. The AC carefully reviewed the paper and rebuttal, agreeing with the recommendation for acceptance based on the consistency.

Decision: Accept (Poster)