MoSA: Mosaic Shared Adaptation of Large Language Models
Authors: Xiequn Wang
Zhan Zhuang
Shengda Luo
Yu Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Parameter-efficient fine-tuning
large language model
low-rank adaptation
Primary area: foundation or frontier models, including LLMs

Abstract: We introduce MoSA, a new parameter-efficient fine-tuning (PEFT) method that replaces low-rank factorization with randomized, fine-grained sharing of weight updates. Each adapted weight matrix is constructed by broadcasting a small set of learned scalars over a fixed tessellation, a pre-defined group assignment of weight entries of the weight matrix, producing expressive changes under the same parameter budget as low-rank adaptation (LoRA). MoSA requires no architectural changes and can be merged into the base model for zero-overhead inference. Across diverse language understanding and generation tasks, MoSA matches or surpasses strong PEFT baselines under strictly matched budgets. Analyses and ablations indicate that non-local parameter sharing acts as an effective regularizer, and that grouping design and budget allocation govern the expressivity–efficiency trade-off. These results position MoSA as a simple, scalable alternative to LoRA. Our code is available at https://github.com/XiequnWang/MoSA-ICLR26.

Published as a conference paper at ICLR 2026
MoSA: MOSAIC SHARED ADAPTATION OF LARGE LANGUAGE MODELS
Xiequn Wang"1 Zhan Zhuang Shengda Luo
Yu Zhang"
Southern University of Science and Technology
City University of Hong Kong
Chinese Medicine Guangdong Laboratory
{ (wangxiequn, shengdaluol 991, yu. zhang. ust @ggmail.com 122 5 50060022000000 063 630mal sustech. edu. on
ABSTRACT
We introduce MoSA, a new parameter-efficient fine-tuning (PEFT) method that replaces low-rank factorization with randomized, fine-grained sharing of weight updates. Each adapted weight matrix is constructed by broadcasting small set of learned scalars over fixed tessellation, pre-defined group assignment of weight entries of the weight matrix, producing expressive changes under the same parameter budget as low-rank adaptation (LoRA). MoSA requires no architec- tural changes and can be merged into the base model for zero-overhead inference. Across diverse language understanding and generation tasks, MoSA matches or surpasses strong PEFT baselines under strictly matched budgets. Analyses and ablations indicate that non-local parameter sharing acts as an effective regularizer, and that grouping design and budget allocation govern the expressivity sprfesivity-efficiineyy trade-off. These results position MoSA as simple, scalable alternative to LoRA. Our code is available at http://hhnbbb.oomoeeeen////aa//e
INTRODUCTION
Large Language Models (LLMs), such as BERT (Devlin et al., 2019), GPT-3 (Brown et al., 2020), and LLaMA (Touvron et al., 2023a), are built on Transformers (Vaswani et al., 2017) and pretrained on web-scale corpora. In practical use, these pretrained models are typically adapted to downstream tasks via task-specific fine-tuning, which has driven rapid progress across NLP and beyond. How- ever, updating all parameters often impractical: the computation and storage costs are substantial, optimizer states and checkpoints scale with the model size, and maintaining separate fully fine-tuned copies hinders multi-task and on-device deployment (Ding et al., 2023). This motivates parameter- efficient fine-tuning (PEFT), which seeks to retain the benefits of full adaptation while learning only a small set of additional parameters and keeping the base model frozen.
Among PEFT methods, low-rank adaptation (LoRA) (Hu al., 2022) is widely adopted. It freezes pretrained weights and injects trainable low-rank matrices, reflecting the hypothesis that useful weight updates have low intrinsic ranks. This low-rank prior has become de facto assumption in LoRA and its variants (Liu et al., 2024; Wang et al., 2024; Hayou et al., 2024; Meng et al., 2024; Wang et al., 2025b). However, this assumption imposes strict structural bottleneck: it confines the weight update to a small subspace, potentially limiting the model's 's ability to learn complex, high- rank patterns required for difficult tasks. While recent approaches like HiRA (Huang et al., 2025) attempt to alleviate this limitation by using Hadamard products to achieve high-rank updates, they inherently rely on the priors of the original weights, effectively confining the update to a specific, weight-dependent subspace. This raises a natural question: can we achieve parameter efficiency without being bound by low-rank constraints or pre-existing weight structures?
To answer this question, we introduce Mosaic Shared Adaptation (MoSA). The intuition behind MoSA draws directly from the art of mosaics, where complex imagery is constructed from a limited
Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
palette of tesserae. Similarly, MoSA partitions the entries of the weight matrix into fixed tessel- lation of disjoint groups, and controls each group with a single learnable scalar. We then broadcast these scalars to their assigned positions to construct the final update. Crucially, set the tessel- lation to be randomized and spatially agnostic. This non-local grouping disrupts the short-range correlations found in weight matrices, acting regularizer that mitigates co-adaptation (Hinton et al., 2012). This allows MoSA to construct expressive, full-rank updates using parameter budget strictly comparable to LoRA.
Beyond the conceptual framework, we develop a first-order analysis showing that size-balanced grouping minimizes the expected deviation from the unconstrained update, thereby formalizing the optimality of our design. Furthermore, to realize MoSA efficiently, we implement a custom segmented-reduccte reduction backward kernel. This kernel aggregates per-group gradients in single pass using stable permutation, eliminating the need for atomic operations and significantly accelerating the backward pass.
Comprchensive experiments commonsense reasoning, open-domain dialogue, and mathematical reasoning benchmarks show that MoSA consistently outperforms strong PEFT baselines such as LoRA, DoRA (Liu et al., 2024), and HiRA (Huang et al., 2025) under matched parameter budgets. Notably, MoSA achieves competitive performance using a parameter budget smaller than a rank- LoRA equivalent, matching baselines that require tens of times more parameters.
RELATED WORKS
Adapting large pretrained models has produced broad family of PEFT techniques. A useful or- ganizing view is ask whether a method adds small task-specific parameters while freezing the backbone, or reparameterizes the weight update itself. We follow this taxonomy and position our method accordingly. A qualitative comparison appears in Table 1. Related work is detailed below.
Additive PEFT Methods. A major branch of PEFT involves inserting small, trainable modules into the frozen LLM. Adapter- based methods (Houlsby et al., 2019) pioneered this by adding compact neural networks between Transformer layers. More recent approaches focus on the input and activa- tion space. Prompt Tuning (Lester et al., 2021) prepends continuous, trainable "soft prompt" vectors to the input, while Prefix Tuning (Li and Liang, 2021) inserts trainable prefixes into the hidden states of each layer, steering the model's 's behavior without altering its core weights.
Reparameterization via Low-Rank Updates. An influential alternative is to reparameterize the weight update itself. LoRA (Hu et al., 2022) is the canonical example, modeling the update AW as two smaller, low-rank matrices (AW BAA). This factorization dramatically reduces the number of trainable parameters. Its success has inspired numerous variants, including QLoRA (Dettmers al., 2023) for memory efficiency, AdaLoRA (Zhang et al., 2023) for adaptive budget allocation and DoRA (Liu et al., 2024) for magnitude-direction decomposition.
High-Rank Methods. Recently, researchers have started exploring beyond the low-rank con- straint (Jiang et al., 2024; Huang et al., 2025). MoRA (Jiang et al., 2024) employs a square ma- trix to maximize the rank of the update within fixed parameter budget, challenging the necessity of low-rank decomposition. HiRA (Huang et al., 2025), for instance, uses Hadamard products to achieve high-rank updates efficiently, suggesting greater expressive power is beneficial. Our work aligns with this direction but proposes a fundamentally different mechanism.
Hashing Methods. The core idea of parameter sharing in MoSA is conceptually related to tech- niques developed for model compression. The "hashing trick" or HashedNets (Chen et al., 2015; Nooralinejad et al., 2023) use a hash function to group network weights, forcing all weights in the same hash bucket to share single parameter value. This significantly reduces storage for the entire model. However, these methods were designed to compress the entire pre-trained weight matrix Wo. In contrast, MoSA applies this principle of randomized grouping specifically the fine-tuning update AW, repurposing it as a PEFT strategy rather than a static compression tool. This distinction crucial, as MoSA maintains the integrity of the base model while enabling efficient adaptation.
2

---

Published as a conference paper at ICLR 2026
Table 1: Qualitative comparison of PEFT methods (V: yes, no). MoSA highlights: high-rank expressivity under the same budget, per-scalar budget granularity, lossless merge for zero-latency inference, no architectural changes, and non-local sharing as regularization.
Criterion
MoSA (Ours)
LoRA
DoRA
MoRA
HiRA
Prompt/Prefix Adapters
High-rank expressivity at fixed budget
x
V
Arhitrary hudget granularity
X
X
x
No change the architecture
V
/
Non-local parameter- shariing---rnnng regulurization
x
Learnable A
Update AW
Grad
Vw
Grad V
Aa
G14
Σx
Aa
Às
Aa
Às
G2.
G22
G2s
G24
Sugmoonled Sum
Σ G
Às
AW 2 Ar
A
Às
Aa
A
Gs:
G
Gas
Ga
PxC= 5Pu
Σn
A4
A2
24 A1
G..
G22
Gas
G.
Σ
(a) Forward:
Group-wise Broadcasting
(b) Backward:
Gradient Aggregation
The learnable scalars À broadcast to their corresponding color-coded tesserae to form the update matrix AW. (b) Back- ward: The gradients Vw C are aggregated via segmented reduction. Values in identically colored cells are summed to produce the scalar gradient VxC, efficiently implemented as VxC = SPu.
Figure 1: MoSA Forward and Backward Mechanics. (a) Forward:
3 METHODOLOGY
In this section, we present MoSA. As illustrated in Figure 1, our method partitions the weight space into disjoint groups and assigns single learnable scalar each group. This design enables full- rank, element-wis modulation with parameter budget comparable to or lower than that of low-rank adapters. We first detail the mathematical formulation, then provide a theoretical justification for our partitioning strategy, and finally describe the efficient gradient computation.
3.1 FORMULATION
Consider a pre-trained linear layer with weights Wo E Rixd where d is the input dimension and the output dimension. We aim learn an additive update AW.
LoRA Parameterization. LoRA assumes AW has a low intrinsic rank & min(h, d). It parame- terizes the update as the product of two low-rank matrices B RRxor and A E Rrxdd
AWLORA BA.
This enforces strict structural bottleneck, limiting the update to low-dimensional subspace.
MoSA Parameterization. In contrast, MoSA constructs full-rank update using a sparse set of shared scalars À E RK, where K matches the parameter budget of LoRA baselines. We define fixed partitioning of the weight indices I {(t.j) 115 < h, S S d} into K disjoint sets (tesserae) I,....,I Ig. As shown in Figure 1(a), the update matrix is constructed via broadcasting:
K
{
if (i, Iz, otherwise.
AWMOSA
MR where (Ma)s
Here, M, E 1] serves as a binary mask for the k-th group. The forward pass for an input Rd is given by y (W, + AW)x. Since the masks {M&} are mutually orthogonal and cover all indices, every weight element W is modulated by exactly one scalar dg. This allows MoSA to affect every weight individually while tying their values small set of learnable parameters.
3

---

Published as a conference paper at ICLR 2026
Gradient Derivation. Let L denote the loss function. By the chain rule, the gradient with respect to the learnable scalar Ag. is the Frobenius inner product between the weight gradient Vaw and the mask Mg:
0C (V aλ&
aC dAA
Σ
(3)
(K.))IZ,
Implementation via Matrix Permutation. Although Eq. (3) provides the analytical form, we implement the backward pass as fast linear projection in the vectorized space. Let vec(V AW L) E R be the flattened gradient vector, where N hdl. We define fixed permu- tation matrix P {0, 11 N that reorders such that indices belonging to the same group Ia become contiguous. As shown in Figure ((b), the gradient aggregation is then expressed as:
V SPu,
(4)
where S {0, 1] K a segmentation matrix comprising contiguous blocks of ones. Specifically, S maps the sorted segments of Pu to their corresponding group scalars. In practice, the operation SP is implemented as a custom fused segmented reduction kernel, which requires caching only the permutation indices, significantly reducing memory footprint compared to storing full adaptation matrices. Detailed explanations are shown in C.2.
Structure Sharing. To further minimize memory overhead, we enforce structure sharing across the network. Since modern architectures contain multiple layers with identical dimensions h x d, we assign the same random partition all weights of the same shape. This strategy allows to store a single set of index mappings for each unique layer shape, decoupling the additional memory cost from the network depth.
3.2 THEORETICAL MOTIVATION: OPTIMALITY OF BALANCE
A critical design choice in MoSA is the distribution of group sizes me = [Ix|] To understand the impact of group size, examine the effective update dynamics. Let G W R denote the gradient of the loss with respect to the weights. MoSA updates scalar parameters d using the
gradient A. (G, M&) r and learning rate ry. Mapping this scalar update back the weight space, the effective weight increment becomes:
SW mosa 1
Σ
Σ k= (L.))I&
Gij
Mr =
Σ
my Gh Mr.
(5)
Accumulated Gradient
where Is is the mean gradient of the k-th group. Unlike standard projected gradient descent, which projects onto the mean direction Gk. Eq. (5) reveals that the MoSA update is explicitly scaled by the implicit, group-dependent learning rate scaling: larger groups
group size mg- This introduces receive aggressively larger updates. To ensure uniform optimization dynamics across all subspaces and minimize deviation from the global gradient direction, the scaling factor my must be constant.
We formalize this intuition by quantifying the approximation error:
Theorem 1 (Optimality of Balanced Partition). Assume the entries of the gradient G are i.i.d. random variables with mean and variance a². The expected squared error between the MoSA update and the unconstrained update is defined as:
(mm....,mk)) = 1- 5W W.mooaal'
(6)
For fixed number of parameters N and groups K, E is a Schur-convex function of the group sizes vector m (my...., mx). Consequently, E minimized when the partition balanced, i.e., E [NN/K) [N/K])
Proof. The squared error decomposes over the orthogonal basis formed by the group partitions. Since groups are disjoint, the total error is the sum of errors within each group:
| + 5W mosa 12 = n? GL. - M
(7)
4

---

Published as a conference paper at ICLR 2026
where Gz, denotes the restriction of G to indices in group k. Expanding the expected enror term for a single group k, and noting that the MoSA update assigns the scalar value L(ie) Gúj = m to every element in that group:
E
Σ (Giz -mEğa²
- (my - 1)o² + mmmmm - 1)²p² + (m² 1)30²
(8)
(TJ)ETT
Summing over all l = 1... K, the total expected error is:
E(m) = 1/2 [ (NV K)O² n.a my (m² 1)² + 0²
K my 1)²
(9)
1)² The functions Φ1 (a) = x(x 1)² and Φ2 (2(x)) = (a 1)2 are convex for x: 2 1. Since the sum of con- vex functions applied to components of vector is Schur-convex function, E(m) is Schur-convex. By Karamata's is inequality, a Schur-convex function is minimized when the components of its argu- ment vector are as equal as possible. Thus, the error is minimized when my {[N/K], [N/K]] for all k.
Balanced Random Tessellation (BRT). Theorem implies that uniform group sizes maximize the fidelity of the update direction. Motivated by this, we implement Balanced Random Tessellation (BRT). We strictly enforce size constraints by randomly permuting the index set and splitting into K contiguous blocks of equal size. By construction, BRT ensures that the gradient variance is spread evenly across all learnable scalars, maximizing training stability.
EXPERIMENTS
In this section, we evaluate MoSA across three distinct task families demonstrate its effectiveness and generality: (i) multi-dataset commonsense reasoning, (ii) open-domain dialogue on ConvAl2 (Dinan al., 2019), and (iii) mathematical reasoning under a distribution shift that involves training on MetaMathQA (Yu et al., 2024), evaluating on GSM8K (Cobbe et al., 2021). Throughout all ex- periments, we enforce strict coameterr-udgeer parity with strong PEFT baselines. Our training and evaluation protocols are designed for transparency and reproducibility, closely following standard practices in recent literature (Huang et al., 2025) unless otherwise specified.
4.1 TASKS AND DATASETS
Commonsense Reasoning. We adopt multi-task setting where models are trained once the union of eight datasets and evaluated on each sub-task individually. The datasets include BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap al., 2019), HellaSwag (Zellers al., 2019), WinoGrande (Sakaguchi et al., 2020), ARC-c (Clark et al., A ARC-e (Clark et al., 2018), and OBQA (Mihaylov et al., 2018). The training are combined, comprising approximately 170k examples, with small held-out split reserved for model selection. Evaluation is performed on the official test of each respective dataset.
Open- dpen-domain Dialogue. We use the ConvA12 dataset (Dinan et al., 2019), which consists of persona- gronondd, multi-turn dialogues (17,878 for training, 1,000 for testing). The task framed under the self-persona setting, where only the speaker's own persona is visible during generation. We report BLEU-4 (Papineni et al., 2002), BERTScore (P/R/F1) (Zhang al., 2020), METEOR (Banerjee and Lavie, 2005), and ROUGE-L (Lin, 2004). We also present an Average score, calcu- lated as the unweighted mean of these six metrics.
Mathematical Reasoning, For this task, we assess the model's out-of-distribution generalization by training on MetaMathQA (Yu et al., 2024) and evaluating on the GSMSK benchmark (Cobbe et al., 2021). We report exact match accuracy on the final numeric answer.
4.2 MODELS AND BASELINES
Base Models. We conduct experiments on two widely used open-weight models: Llama-2-1 (Touvron al., 2023b) and Llama-3-8B (Grattafiori al., 2024).
5

---

Published as a conference paper at ICLR 2026
Baselines. We compare MoSA against suite of strong PEFT methods, including Prompt Tuning (Lester et al., 2021), P-Tuning Liu et al. (2022), LoRA (Hu et al., 2022), DoRA (Liu et al., 2024), MoRA (Jiang et al., 2024), and the high-rank adaptation method, HiRA (Huang et al., 2025). All baselines are configured to adapt the same target modules as MoSA under matched training budgets.
4.3 PARAMETER BUDGET AND IMPLEMENTATION
Targeted Modules. Unless stated otherwise, adapters are applied to the self-attention projections We. WK. Wy and the FFN projections Wup and W down of each Transformer block. All other parameters in the base model are kept frozen.
Parameter-Budge Parity. To ensure fair comparison, enforce strict parameter parity between MoSA and LoRA at rank = 32. For target linear layer with dimensions h x d, LoRA introduces r(d + h) trainable parameters. Accordingly, we set the number of MoSA groups such that K = r(d + ft). guaranteeing an identical parameter budget for every target module. The column "Params (%)" in our tables reports this count as a percentage of the base model's total parameters.
Optimization and Schedules. We use the Adam W (Loshchilov and Hutter, 2019) optimizer with learning rate of 1 x 10-5 and 0.1 warm-up ratio. To ensure a fair comparison, we maintain consistency across all methods in maximum sequence length, tokenization. batch size, and mixed- precision settings. Models are trained for epochs on the commonsense reasoning suite, epoch on A12vA12, and epochs on the mathematical reasoning task. Model selection is performed on held-out validation split. For all tasks, we employ deterministic decoding (temperature = 0).
4.4 RESULTS: COMMONSENSE REASONING
As summarized Table 2, MoSA demonstrates superior performance across the board on the multi- dataset commonsense reasoning benchmark. It consistently outperforms all strong PEFT baselines on both Llama-2-7B and Llama-3-8B, establishing itself as the most effective method for this task under strict parameter-budget parity.
On the more capable Llama-3-8B model, MoSA (with budget equivalent to LoRA = 32) achieves remarkable average accuracy of 87.63%. This represents substantial improvement of +0.91% 1% over the strongest competing baseline, HiRA, which scored 86. 86.72%. closer look at the per-task results reveals MoSA's comprehensive dominance: it secures the top score every single one of the eight datasets. The gains are particularly notable on challenging datasets like HellaSwag (+1.21 % the best baseline) and WinoGrande (+1.08%).
The same trend holds for Llama-2-7B. MoSA again leads the pack with an average accuracy of 83.83%, widening its lead over the second-best method, HiRA (81.42%), This confirms that MoSA's advantages are not specific to a single base model but are robust and transferable. On this model, it achieves the best performance on seven of the eight sub-tasks, further cementing its position as the state-of of-the-art PEFT method for this reasoning suite.
significant +2.41 %.
4.5 RESULTS: OPEN-DOMAIN DIALOGUE (CONVA12)
In the persona-grounded, open-domain dialogue task on ConvA12, MoSA's superiority is even more pronounced. As shown in Table 3, for Llama-3-8B, MoSA achieves an impressive result, surpassing the strongest baseline, HiRA (47.80%) by large margin of +2.34%. This overall improvement is supported by consistent wins in all individual metrics, which measure different aspects of generation quality. For instance, its BLEU-4 score of 4. 13% indicates significantly better n-gram overlap with reference responses, while its leading BERTScore (F1 of 86.83%) to superior semantic similarity and relevance. These results highlight MoSA's ability to produce responses that more fluent and coherent (METEOR, ROUGE-L), while also being contextually and semantically more appropriate.
The results Llama-2-7B further validate these findings. MoSA again claims the top spot all seven metrics, culminating in an Average score of 49.92%. This represents an even larger improve- ment of +2.64% over the next-best method. The consistent and decisive lead across diverse auto- matic metrics underscores MoSA's enhanced ability to handle the nuances of multi-turn, persona- grounded conversations.

---

Published as conference paper at ICLR 2026
Table 2: Accuracy (%) on eight commonsense benchmarks using Llama-2-7B and Llama-3-88
MoSA uses a parameter budget equivalent to LoRA rank r. Best is bold, second best is underlined.
(8)
BuolQ
PIQA
OBQA HellaSwag
WinoGrunde
Averuge
Llama 2.7B
Prompt Tuning
0.0012
55.93
12.35
30.50
6.06
8.63
9.40
6.91
40.57
21.29
P-Tuning
0.7428
58.75
36.02
0.20
0..7
1.98
0.80
0.01
0.00
12.24
LoRA (r=32)
0.8256
69.80
79.90
79.50
64.70
79.80
81.00
83.60
82.60
77.61
(x=32)
MoRA (x=32)
0.8241
72.17
80.79
79.53
71.42
85,31
81.20
29.09
80.19
72.46
HIRA (r=32)
0.8256
71.22
83.35
79.53
73.81
86.74
84,60
88.12
83.98
81.42
MoSA (r-32 equis.)
0.8256
73.96
86.03
81.48
76.62
88.34
83.93
94.35
83.93
83.83
Dlamu-
Prompt Tuning
0,0010
56.85
45.05
36.13
32,74
14.01
36,96
P-Tuning
0.6240
59.97
11.64
8.19
7.42
8.66
9.60
1.77
37.65
18.11
LoRA (x=32)
0,7002
70,80
85.20
79.90
71.20
84.20
79.00
91.70
84.30
80.79
DeRA (r=32)
0.7002
74.60
89.30
79.90
80.40
90.50
80
95.50
85.60
85.20
MoRA (x-32)
0.6997
79.61
91.16
HiRA (n-32)
0.7002
75.40
81.15
82.90
93.27
88.
95.36
87.20
86.22
MoSA (r=32 equis.)
0,7002
75.64
90.65
82,70
82.91
93.27
89.48
96.57
89.78
87.63
Table 3: ConvA12 results with Llama-2-7B and Llama-3-8B backbones. BERT-FI/R/P are from BERTScore. Best is bold, second best underlined.
Method
Params (%)
BLEU-4 BERT-FI
BERT-R BERT-P METEOR ROUGE-L Average
Llama-2-7B
Prompt Tuning
0.0012
0.04
72.44
77.38
68.23
0.80
0.80
36.62
P-Tuning
0.7428
0.60
83.29
83,33
83.28
15.11
12.36
46.33
MoRA (t=32)
0,8241
1.09
84.09
84.65
83,59
10.97
9.57
45.66
LoRA (r=32)
0.8256
1.82
84.41
84,71
84.16
11.38
10.55
46.17
DoRA (n=32)
0,8256
1.73
84.18
84.61
83.81
11.25
10.41
46.00
HiRA (r=32)
0.8256
84.86
84.98
12.80
47,28
(r=32 equiv.)
0.8256
86.53
86.28
18.44
Llama-3-NR
Prompt Tuning
0.0010
1.45
82.99
82,99
83.05
14.72
13.13
46.39
P-Tuning
0.6240
1.50
81.52
81.07
82.01
15.49
13.55
45.86
MoRA (r=32)
0.6997
1.60
82.24
84,06
84.43
12.37
11.19
46.31
LoRA (c=22)
0.7002
2.26
84.32
84.00
84.67
12.51
11.77
46.59
(r=32)
84.06
11.78
HiRA (e=32)
0.7002
84.81
84.40
85,25
14.87
14.05
47,80
MoSA (I=32 equiv.)
0.7002
4.13
86.83
86.50
87.18
17.59
18.64
50.14
4.6 RESULTS: MATHEMATICAL REASONING
We evaluate the models on their out-of-distribution Table 4: GSM8K accuracy (%) after training on MetaMathQA. Best is bold, second best underlined.
(OOD) generalization capabilities by training on Meta.artQQ and testing on the unseen GSM8K benchmark. This challenging setup tests model's ability to learn abstract reasoning principles rather than merely memorizing problem templates. As de- tailed in Table 4, MoSA demonstrates profound and significant advantage in this area. Using Llama-3-8B as the base model, MoSA achieves an exact match accuracy of 78.00% on GSM8K. This result is outperforming the strongest
Method
Params (%)
GSMSK
Llama-3-88
Prompt Tuning P-Tuning
0.0010
15.62
0.6240
2.65
LoRA (r 32)
0,7002
65.89
DoRA (r 322)
0.7002
66.12
MoRA
32)
0.6997
67.98
baseline, HiRA (r = 32), by a massive +7. 19.. Such large performance gap
HiRA (r 32)
0.7002
70.81
OOD task high- lights MoSA's superior ability to capture and trans- fer the underlying logic of mathematical problem- solving. While other PEFT methods show respectable performance, MoSA's ability to generalize far more effectively sets it apart, indicating learns more robust and portable reasoning structures.
MoSA (r 32 equiv.)
0.7002
78.00

---

Published as a conference paper at ICLR 2026
5 ANALYSIS
5.1 BACKWARD SPEED ANALYSIS
Since MoSA relies aggregating gradients per group, the efficiency of the backward pass is paramount for scalability. Figure 2 illustrates the backward runtime (log-log scale) versus the num- ber of groups K for a weight matrix with dimensions h d 4096. We compare custom segmented reduction kernel against PyTorch (Paszke et al., 2019) autograd under both balanced and skewed assignment strategies.
Across the entire range of K, our segmented reduction is consistently superior to autograd. Specif- 9,500x speedup at K (0.25 vs. 2.41 s), a 125x speedup K" 32
ically, it achieves (0.67 ms vs. 84.50 ms). and maintains an 8-9x advantage for K between 4,096 and 16,384. Al- though skewness introduces slight overhead (geometric-mean slowdown of 1.62 ). the kernel remains highly efficient; even in the most extreme skew tested at K = 65,536, it is at least 4.8x faster than autograd (1.46 ms vs. 7.00 ms). These results demonstrate that while balanced assign- ments (BRT) are optimal, they are not strictly required for performance: the segmented kernel is bandwidth-bound and robust to skew, supporting large K tion. Additional experimental details are provided in Appendix B.
without significant performance degrada-
5.2 ABLATION STUDY ON COMPONENT
From the Average column in Table 5, FFN+QKV achieves the best overall accuracy (87.63), with FFN alone a close second (87.35) and marginally sur- passing QK V (87.16).Here. FFN denotes the Trans- former block's position-wise feed-forward network (Wup up and W Wown), whereas QKV V denotes the multi- head attention's 's query, key, and value projection sub- layers/matrices (WQ. W K. Wv). This indicates that most of the attainable gains under a fixed param- eter budget come from adapting the beed-forward pathway, finding consistent with evidence that FFN blocks function as key-value-like memories and host causally editable factual associations (Geva al., 2021; Dai al., 2022; Meng et al., 2022; Figure 2: Backward time vs. group count
Backward Speed Comparison
Reduction balanced
10a
Reduction skowed
Autograd balanced
Autograd. skowed
Time (ms)
20
213 211
20
25
(Number Groups)
2023). Within attention-only variants, the ordering V > Q > K and QV > QK suggests that modi- fying the value stream-i.e. what content is writ- ten into the residual-is more impactful than adjust- ing query/key routing, aligning with mechanistic ac- counts where QK chiefly sets selection weights and V injects information (Elhage et al., 2021; Olsson al., 2022). practice, allocating most adaptation capacity to FNN and adding attention updates (especially V) for incremental gains is a robust default when budgets allow.
K. Segmented reduction is faster for all K; skew induces a 1.62 geometric-mentt slow- down for segmented but does not alter the qualitative scaling.
5.3 IMPACT OF GROUPING STRATEGY
We evaluate whether fine-grained, non-local sharing provides stronger regularization than low-rank constraints by comparing our proposed BRT to three deterministic strategies defined below:
Balanced Random Tessellation (BRT): Set group sizes as balanced as possible, i.e., m& ([N/K) KJ, [N/K]] such that Σ m, N. We generate label sequence respecting these counts, uniformly permute the indices I, and assign them to groups I.....IIk- This yields near-exact balance while breaking local spatial correlations.
Row-Stripe: Flatten the weight matrix in row-major order and split the N indices into K con- tiguous segments. Indices in the k-th segment form the group I4. The result corresponds to K horizontal stripes with sizes differing by most one.
Flatten in column- column-major order and split into K contiguous segments. The result corresponds to K vertical stripes with bear-balanced sizes.
Col-Stripe:
8

---

Published as a conference paper at ICLR 2026
Table 5:
Performance of the IJama-3 8B model with different Component. position-wise feed-forward sublayer with projections Wup and Woown
FFN denotes the
Component
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaSwag
WinoG
Average
K
66.96
91.20
76.84
79.68
93.36
79.96
93.36
82.63
82.12
Q
70.45
92.13
79.08
80.53
94.43
84.72
94.43
85.06
84.33
V
74.85
92.72
82.24
81.63
95.64
87.50
95.64
86.48
86.29
QK
71.36
92.05
79.49
80.61
94.79
86.90
94.79
86.40
84.99
QV
74.85
93.48
82.60
82.82
96.17
87.90
96.17
87.50
86.81
QKV
75.46
92.97
82.91
82.74
96.06
89.29
96.06
87.81
87.16
FFN
76.13
90.98
81.94
83.5
93.18
86.71
96.46
89.86
87.35
FFN+OKV
75.64
90.65
82.70
82.91
93.27
89.48
96.57
89.78
87.63
Skewed: Set non-increasing group quotas m. 2 .... mg using geometric schedule con- trolled by p E (0, 1), where my pt-1. Uniformly permute the N indices and assign the first to IT, the next m2 to I2, and so forth. This preserves the intended skew while breaking local spatial correlations.
Table 6: Ablation of grouping strategies Llama-3-8B under the same parameter budget.
Strategy
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaSwag
WinoG
Average
BRT
75.64
90.65
82.70
89.48
93.27
89.78
96.57
89.78
87.63
Row-Stripe 75.73
90.38
81.63
83.59
92.55
87.70
96.21
88.60
87.05
Col-Stripe 75.37
89.95
80.61
82.65
92.97
86.90
96.42
88.29
86.65
Skewed 61.22
81.52
70
71.68
85.19
69.64
86.49
69.18
74.36
From the Average column of Table 6, BRT performs best. Non-local randomization scatters shar- ing patterns and disrupts short-range correlations. Row-Stripe generally exceeds Col-Stripe because column-aligne sharing ties all outgoing connections of the same input feature, suppressing cross- row diversity and pushing outputs toward collinearity, whereas row-aligned sharing preserves more per-output specialization; more broadly, weight tying reduces degrees of freedom. Across layouts, skewed variants lag balanced ones: uneven group sizes concentrate capacity and increase collision variance, while balanced weight-sharing is empirically beneficial in hashing-style schemes. Imple- mentation details appear in Appendix B.8.
5.4 IMPACT PARAMETER BUDGET
We investigate how performance scales with the number of trainable tesserae k. A key advantage of MoSA is that & can take any integer value, enabling fine-grained budget control unconstrained by matrix shapes, unlike LoRA. For reference, LoRA with rank r- on Llama-3-8B trains 0. 0.7002% of parameters (80.79 accuracy), which
baseline.
As shown in Figure 3, accuracy rises steeply from extremely small budgets (~ 0.0014%). This corresponds only about one sixteenth the parameter count required by the smallest possible LoRA setting (r=1, 0.022%), even though LoRA itself cannot operate below rank 1. From there, performance continues to improve up to modest budgets (~ 0.175%, LoRA r=4 equivalent), after which the curve flattens. Notably, with effective LoRA r=1 budget (0.022% parameters), MoSA already reaches 86.81, surpassing the LoRA 32 baseline while using only z parameter num- bers. Most gains are realized by the LoRA r equivalent, with larger allocations bringing only marginal, task-specific improvements. Unlike LoRA, however, MoSA admits arbitrary k. allowing precise budget tuning between conventional ranks. Overall, these results suggest that small to mod- erate budgets (up to LoRA rS4 equivalents) already suffice for near-optimal accuracy, while larger budgets are only justified when additional task task-specific gains desired.
I

---

Published as a conference paper at ICLR 2026
LoRA equivalent
Accuracy (%)
baseline:
80
Average
PIQA
Hellas wag
0BQA
ARC-c
ARC-e
WinoG
Boolo
0.002
0.005
0.01
0.02
0.05
0,1
0.2
0,5
Params (%)
Figure 3: Budget-perfood perforracee trade-off on Llama-3-8B. Per-task accuracy (thin solid) and Av- erage (thick dashed) trainable budget & (log-scale, bottom axis). Top axis shows LoRA-rank equivalents; the vertical dotted line marks the LoRA r=32 baseline (80.79).
CONCLUSION
We introduced MoSA, parameter-efficien fine-tuning method that replaces low-rank adapters with randomized, near-balanced tessellations of the weight matrix controlled by small set of learned scalars. Theoretically, with uniform learning rates and balanced groups, the task-driven update is directionally equivalent to projecting the full gradient onto the MoSA subspace. System-win wise, segmented-reductioo backward kernel aggregates per-group gradients one pass, scales robustly with the number of groups, and delivers substantial speedups over autograd. Empirically, across commonsense reasoning, open-domain dialogue, and out-of-distribution math reasoning. MoSA consistently matches surpasses strong baselines the same parameter budget, indicating that simple non-local sharing is a competitive alternative to the low-rank assumption. Overall, MoSA offers simple, controllable, and efficient route parameter-efficien adaptation.
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China under Grant no. 62136005 and Shenzhen fundamental research program JCYJ2025060414472222
Shengda Luo
was supported by the National Natural Science Foundation of China (Grant No. 82505358), the scientific research start-up funds of Chinese Medicine Guangdong Laboratory (Grant No. HQL2025SU011) 1), and the Tian Yuan funds for Mathematics of the National Science Foundation of China (Grant No. 12326604).
ETHICS STATEMENT
This work fine-tunes open-weight LLMs on public, widely-used benchmarks. We use the official splits and standard evaluation protocols; we do not collect new data, annotate human subjects, or process personally identifiable information. All datasets are used under their original licenses and terms. Public NLP datasets and the underlying pretrained models may contain harmful content or societal biases. Our method is a parameter-efficia on top of frozen backbones, but it can still inherit and potentially amplify biases present in the data or base models. We therefore: (i) report task atak-staddrrd metrics without promoting sensitive-attrile targeting; (ii) follow the base-model usage policies; and (iii) recommend downstream users apply appropriate safety filters, domain constraints, and bias audits when deploying adapted models in user-facing scenarios.
adapter that modifies only small set scalars
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We aim to make results reproducible. Detailed hyperparameter setting are shown in Appendix B. We have released our code at
REFERENCES
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Translation and/or Summarization, pages 63-72, 2005. URL https::////. / http//aclaae /aaaacaaantt awwlannhooooooooooo org/w05- 505- 000-09900 0909/.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432-7444 7439, 2020. doi: 10.1609/aaii index. AAA.AAAAAaaiiiaa wwwcceeeveww//eeeeeeeee 6239.
(0009//aai.34445562225..33 v34i05.6239. URL https: //oss///j/ss/..// aaai orgg/
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neu- ral Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL. http::::/pprcceed nggs.... woigg.neuriii neuri cc. wwtpppprree.eeeeeeeeeeeee fi tes/paper/202// file #ilee/147ccdd6bfccbee 967.6666bcc4444461116 www.e67741118888144ee1.111 www.ebffbbccc.ee wttpaaaacca.eeeeeeaaeae.....
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Broekman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Q Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning, pages 2285-2294 PMLR, 2015.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BooiQ:
Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Papers), pages 1924-2936 Minncapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 18653/v1/N19. 1300. URI. https: / aacllll antho ogy org/ N1 9- 1-30//
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Øyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning chal- lenge. arXiv preprint ar30v:1803.05457, 2018. URL https: htt//viiiiii.////i///a/////aatva wttpl//.rrg/////////////eebse//e abs/1803...1... 05457.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemy Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL attps://arae org/abs/2111 /2100. 14165 68.
11

---

Published as a conference paper at ICLR 2026
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume
I: Long Pupers), pages 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. 100166633/20222allooogg 2022. acl- 1.o...nn...1111
URL https: http://aaaaaatolooocooooo ooyyoyoooy.......oo org/
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances neural information processing systems, 36:10008--10000 2023.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of
deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pages 4171 4166, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. N0.1865//11/N19-4433
URL https: / a/c hog.looooogggyyoooooooooooooooom org/
M19-14223
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, and Jason Weston. The second conversational intelligence challenge (convai2), 2019. URL https://arxiv..rrt abs 1902. 00098.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozii Wang. Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. Parameter-efficien fine- tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235, doi: 10. 10.10388/42255-023-000 10. 1.0.103//40 422566 00033-0006--44
March 2023. ISSN 2522-5839,
URL https:: /oi..oiooooooooo/ooo org/
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam Mc- Candlish, and Chris Olah. A mathematical framework for transformer circuits. https:// transformer-coe cineer-circuttta wttprrcirrcuuut.uuproooa..e pubp www.wworw/iinneooaaaowrrorr inddexx//iniei//eiee html, 2021. Transformer Cireuits Thread.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 105533/1/222..mm/g www.l-mminnn..... 646.44. URL org/ 2021 emnlp-ma iiniininn in. 446/
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint 1rXiv:2407.27783, 2024. URL https:///aaatittt org abs 407. 21783.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In International Conference on Machine Learning, pages 17783-17806. 2024.
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 1207.0580. 2012.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th In- fernational Conference Machine Learning, volume 97 of Proceedings of Machine Learning
12

---

Published as a conference paper at ICLR 2026
Research, pages 2790-2799 PMLR, 09-15 Jun 2019. URL URLattpss//proedd..g.. .ml... press/v97 httpss/v9//hoo// hhhouhoulllyyyy 9a. html.
Edward J. Hu et al. LoRA: Low-rank adaptation of large language models. In International Confer- ence on Learning Representations, 2022. URL https: opwnreviewwwveeeooemeee.e net/ fer.ommmmmm
nžeVKeeFYf9.
Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. HiRA: Parameter-efficient hadamard high-rank adaptation for large language models. In The Thirleenth International Con- ference on Learning Representations, 2025. URL https: http///pppeeeeeeeeeeeee/eeeee/et net forum?
id-IwJrI29cRS.
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. Mora: High-rank updating for parameter-efficient fine-tuning, 2024. URL httpsss//assi...rrgg hbtp//a//// /2405. 12.30.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Firancine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10, 10.1188661222222 1ttp3/////2.mmmmlppppp main.243 URL https: / www.ccannhlooooooooooooooccooooo ogy. org/ 2021 emnlp-ma 1 n. 243/.
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online, August 2021, Association for Computational Linguistics. doi: 10. 1111186611/22211 aongo. URL httpt:tt///. / htt///aaaaannloogoo/eeeee org/ 2021 acl-long 353/.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text Summa- rization Branches Out (WS), pages 74-81, 2004. URL https://accanthhee orgg N04-1013/.
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learn- ing. volume 235 of Proceedings of Machine Learning Research, pages 32100-32121 PMLR, 21-27 Jul 2024. URL https:/ http//pppooeena.eeeeeeeepeesssst ml press v2355/s/vvv///vv iu2 4bn html.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Annual Meeting of the Association for Computational Linguistics, pages 61-68, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- ence on Learning Representations, 2019. URL https:// /ttp::/:////eeeeweww/ net forum? id=
Bkg6RicqY7.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: principal singular values and singular vec- tors adaptation of large language models. In Proceedings of the 38th International Conference on Neurul Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9798331314385.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing fac- tual associations in GPT. In Advances in Neural Information Processing Systems, vol- ume 35, 2022. http::::: hwpppappsssseesa/ .npperrss.iiiipp. wwcips.cepppee cc/papez wwwes/ http//pppee222/2/ssssss html.
Kevin Meng, Arna Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. URL https: chttps://opentte //tppppevvviieeeeee/ ew. net fo Forum? www.kedewwbbbeeeeeeeee.aaa. ICIR 2023.
13

---

Published as a conference paper at ICLR 2026
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing. pages 2381-2391 Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 1/D18 1-1602 URL. https: //as:///clloo aww.annnh..oooooooo
--1200/
Takashi Mori, Liu Ziyin, Kangqiao Liu, and Masahito Ueda. Power-law escape rate of SGD. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15959-15975. PMLR, 17-23 Jul 2022. URL https:/ httpss//ppcceeeingg.... mmrr pres. v2
html.
Parsa Nooralinejad, Ali Abbasi, Soroush Abbasi Koohpayegani, Kossar Pourahmadi Meibodi, Rana Muhammad Shahroz Khan, Soheil Kolouri, and Hamed Pirsiavash. Pranc: Pseudo random net- works for compacting deep models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages I 17021-17031, 2023.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-oodds Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. arXiv preprint arXiv: 1895, 2022. doi: 10.488500/r21v.2220.0 1895. URL https: /arxivvvvvv../////a//aaa///aaiv. org/ abs abs/2209 11895.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311-318, 2002. doi: 10.31 10.3315/1073333:1073333
URL https:////// / lww.antnnnnntooooceoooo www//or////// 1040/.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang. Zach Veto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: an imperative style, high-ppeformancc deep learning library. Curran Associates Inc.. Red Hook, NY, USA, 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge scale. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pages 4463-4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v//D19. 1454. URL
Sxq/D19-14544
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foun- dation and fine-tuned chat models. arXiv preprint AXXv:23077009288 2023b. URL https:
//arxiv. wrw.org.aaoo abs /2007. 09288.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
14

---

Published as a conference paper at ICLR 2026
Shaowen Wang. Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradi- ent approximation. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang. editors, Advances in Neural Information Processing Sys- tems, volume 37, pages 54905-54931. Curran Associates, Inc., 2024. doi: 10.52202/ http/pappr_fles/pppee 079017-1741. URL httpp:://proces neserriiisinnnnniiip.p Conper--oofeeence.eee. pdf. 202///Lee2cc11113335faaaee222221
Yiding Wang. Fanxu Meng. Xuefeng Zhang, Fan Jiang, Pingzhi Tang, and Muhan Zhang. HD- Christos Christodoulopoulls, Tan- moy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, pages 6526-6539, Suzhou, China, November 2025a. Association for Computational Linguisties. ISBN 979-8-89176-332- 6. doi: 10.18653/v1/202 emnlp- main.330. URL ht https: www.lnnnholog.oooooo..o. emnlp-ma in. 330/.
PiSSA: High-rank distributed orthogonal adaptation.
Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. LoRA-Pro: Are low-rank adapters properly optimized? In The Thirteenth International Conference on Learming Representations, Singapore, April 2025b. OpenReview.net. URL httph::///// /ttp:i//aaaavvoaaaeea org/ abs/ 18242.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In International Conference on Learning Representations (ICLR), 2024. URL https: / openreview net wwt. www.eeiddunnnnnnenee
Spotlight.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19 1472. URL https: : /ttp:///attoooooooooo////aa/////
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efffiient fine-tuning, In International Con- ference on Learning Representations, 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinherger, and Yoav Artzi BERTScore: Eval- uating text generation with BERT. In International Conference Learning Representations, 2020.
Tianyu Zheng. Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. OpenCodeenterppeeee:: Integrating code generation with execution and refine- ment. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Associa- tion for Computational Linguistics: ACL 2024, pages 12834-12859, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1/0.://///222..nniinnn..... pettps://aclantiin www.hhhlooy.org/224... findings anndings-al..666
URL
15

---

Published as a conference paper at ICLR 2026
A STATEMENT ON THE USE OF LLM
During the preparation of this manuscript, we utilized LLM solely for the purpose of grammar checking and language refinement. All core ideas, research content, data analysis, and conclusions are the original work of the authors. We take full responsibility for the final content of this paper.
B EXPERIMENTAL SETUP
B. TASKS AND DATASETS
We evaluate on three task families to assess both in-domain performance and out-of-distributtin generalization: (i) Commonsense reasoning on the union of BoolQ, PIQA, SIQA, ARC-cle, OBQA, HellaSwag, and WinoGrande (~= 170k training examples in total); (ii) Open-domain dialogue on ConvA12 under the self- self-persona setting: (iii) Mathematical reasoning (OOD) by training on Meta- MathQA and evaluating on GSMSK. Unless stated otherwise, we follow standard dataset splits and report results on the official test sets with a small held-out validation split for model selection.
B.2 BASE MODELS AND BASELINES
We consider open-weight LLMs Llama-2-7B and Llama-3-8B. Baselines include strong PEFT meth- ods under matched conditions: Prompt Tuning, P-Tuning. LoRA, DoRA, MoRA, and HiRA. The results for these methods are directly copied from the HiRA paper with same experimental conditions. All methods adapt the same target modules and use identical tokenization, maximum sequence length, batch size, and mixed peeedd-recision settings to ensure a fair comparison.
B.3 ADAPTER TARGETS AND PARAMETER BUDGET PARITY
Adapters are applied to the self-attention projections Wo. WK, Wy and feed-forward projections W up: W down in each Transformer block, while other weights remain frozen. To enforce strict budget parity with LoRA configuration of rank (r = 32), we choose the number of MoSA tesserae K such that the total trainable scalars match r(d,, + dlaa) for the same set of target matrices. We report Params (%) as the ratio of trainable parameters to the base model's total parameter count. Unless specified, group sizes are balanced (difference at most one), and the assignment is fixed throughout training.
B.4 OPTIMIZATION AND TRAINING SCHEDULES
We use AdamW with learning rate 1 x 10-5 and 0.1 warm-up ratio, cosine decay thereafter. Epochs We train with
per task: 3 for commonsense, 1 for ConvA12, and 2 for MetaMathQAAGGSMM global batch size of 8, gradient accumulation 2, max sequence length of 512. Model selection is performed the held-out validation split. Decoding uses deterministic settings (temperature 0).
B.5 EVALUATION PROTOCOL AND METRICS
For commonsense tasks, we report accuracy per dataset and the unweighted average. For dialogue, we report BLEU-4, BERTScore (P/R/F1), METEOR, and ROUGE-L, along with an unweighted average across metrics. For GSM8K, we report exact-match accuracy the final numeric answer.
B.6 HARDWARE AND REPRODUCIBILITY
Experiments conducted A100-80G PCIE. We fix random seeds, log all hyperparameters, and release scripts to reproduce results (data preprocessing, training, and evaluation).
B.7 IMPLEMENTATION NOTES
MoSA parameterizes updates as groupwise-constare scalars over fixed tessellation. Gradients aggregated by single segmented-reduuction a stable permutation that makes equal keys
16

---

Published as a conference paper at ICLR 2026
contiguous. Balanced tessellation improves directional alignment of one-step updates and keeps segments near-uniform, aiding kernel utilization. Unless specified, the assignment is shared across layers of identical shape.
B.8 DETAILS OF GROUPING STRATEGIES
In this section, we formalize the construction of the group-index map I' {l..... K]axd, which assigns each weight position (i,j) to group index Tiy E (1,..., K}. This map determines the group partition [Ix and the corresponding mask matrices {M& } used in the main method, where
== = (i,,)) Tú = k}.
Let and d denote the output and input dimensions, respectively, and N hdl be the total number of weights. We partition the index set into K groups.
Balanced Random Tessellation (BRT). We aim to partition the N weights into K groups of nearly equal size. Let N mod K. We define the target group sizes as:
[N/KK] for 1 k S [N//K]
mi
(10)
for < S K,
which ensures Σx my N and max; |m= N/K| 1.
We first generate a random permutation a of the linear indices {1...., .N) using a fixed seed. Let E (1,..., KJ denote the group-index assignment in linear form. The groups formed by slicing this permuted sequence according to the sizes {ma}: the k-th group Ia contains the indices {a{t)) for t in the interval (C Ck], where C Σ)= mj is the cumulative count. Finally, Y is reshaped back to (h, d) form the map r.
Algorithm 1 Construction of BRT Assignment
Input: Dimensions h, groups K, seed
1: N + h.d
2: Compute sizes {m J such that m. N/K
b Balanced partition
3: < RRADOMPEMUTATION( (1.....II
4: Initialize 7 E [I,
5: 1+1
6: for k k= 1 to K do
7: for = 1 to m. do
8:
Assign linear index (t) to group k: 7o(t) t
9:
/-1+1
10: end for
end for
12: I' + reshapely, h.
return r
Skewed Assignment. To analyze the impact of load imbalance, we generate groups with non- uniform sizes following geometric decay controlled by ratio E (0, 1). We define unnormalized weights WN - for 1..... K. The target integer sizes my are computed using the Largest Remainder Method to ensure they sum exactly to N:
1. Calculate ideal quotas N (wx/ E, w;).
2. Assign initial integer sizes my = Lgk].
3. Distribute the remaining count R N Σx my the groups with the largest fractional parts [-
The assignment is then performed similarly to BRT but using these skewed sizes on the random permutation. For experiments, set p 0.85.
17

---

Published as a conference paper at ICLR 2026
Structured Striping Strategies. Unlike the random assignment in BRT, stripe strategies preserve
spatial locality. We define the cumulative boundaries Co 0 and C2 Σ my based on balanced sizes {mg} as in BRT.
Row-major stripes. Weights are indexed in row-major order: t(i,j) = (i - 1)d + j. Indices
falling in the interval 1, Ct) are assigned group Mathematically, Fij = if and only
CR-1 < (i - 1)d+ & Ck. This tends to group entire rows contiguous row segments together.
Column-major stripes. Weights are indexed in column- major order: t(i.j) (j 1)h +
Similarly, Pij k if and only if Ck-1 < < 1)h+i + s Cs. This tends group entire columns or contiguous column segments.
Algorithm Construction of Structured Stripes (Row/Column)
Input: Dimensions h, d, groups K, mode & (Row, COL)
N h
2: Compute balanced cumulative boundaries {C& 1
k=0
3: for to h do
4: for = 1 to d do
5: if mode Row then
6:
1 + 1)d +
7:
else
D mode COL
8:
t t (j 1)h +
9:
end
10:
Find & such that Ck-1 <t Ck
Binary search linear
11: Tij +
end for
13:
end for
return
B.9 SENSITIVITY TO RANDOM SEED
To evaluate the sensitivity of MoSA to the specific random tessellation generated by the seed, conducted independent runs using Llama-3-8B with the MoSA configuration. We used the same hyperparameters the main experiments but varied the random seed used generate the group assignments. As shown in Table 7, the performance is highly stable across different random initial- izations. The average accuracy across the five seeds 87.45% with standard deviation of only 0.14%.
Table Sensitivity analysis MoSA (Llama-3-8B.
equivalent) across 5 different random seeds for tessellation assignment. The method shows high stability with minimal variance.
Seed
QoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
1
74.7
91.06
81.85
83.63
93.46
90.23
96.5
88.98
87.55
2
74.79
90.79
81.80
84.62
93.50
89.45
96.67
89
87.58
3
75.30
90.79
81.50
83,47
92.93
88.48
96.88
88.75
87.
4
75
89.82
81.85
84.05
93.26
88.87
96.8
89.14
87.35
5
75.27
90.25
81.96
83,39
93.05
90.43
96.71
88.83
87.48
Mean
75.01
90.54
81.79
83.83
93.24
89.49
96.71
88.94
87.45
Std Dev
0.27
0.5
0.17
0.51
0.25
0.84
0.15
0.15
0.14
1.10 COMPARISON TO MORE PEFT METHODS
We compare MoSA with PiSSA, HD-PISAA, LoRA-GA and LoRA-Pro for two experiments. We utilized the Llama-2-7B (Touvron et al., 2023b) model as the backbone.
Regarding the datasets, we fine-tuned the model on 100k subset of the MetaMathQA (Yu et al., 2024) dataset for the mathematical reasoning task and evaluated GSM8K (Cobbe al., 2021).
18

---

Published as a conference paper at ICLR 2026
For the code generation task, fine-tuned 100k subset of the CodeFeedback (Zheng al., 2024) dataset and reported the Pass metric on HumanEval (Chen et al., 2021). The results for PiSSA (Meng et al., 2024). HD-PiSAA (Wang et al., 2025a), LoRA-GA (Wang et al., 2024), and LoRA-Pro (Wang et al., 2025b) were directly copied from their respective original papers.
Table 8: Fine-tuning results of Llama-2-7B model. Bold and underline indicate the highest and second-highest scores, respectively.
Method
GSM8K
HvmanEval
LoRA
42.08
14.76
PiSSA
53.22
21.92
HD-PISSA
52.92
21.3
LoRA-GA
53,60
19.81
LoRA-Pro
57.57
22.97
MoSA
59.8
24.39
Comparison with Multi-Task LoRA Methods. Although our primary experimental setting treats the aggregation of commonsense datasets as single-task learning objective, the benchmark inher- ently consists of 8 distinct reasoning tasks. To validate MoSA's capability in this context, we com- pared with HydraLoRA (Tian et al., 2024), a state-of-ther method specifically designed for multi-task PEFT.
For the HydraLoRA configuration. we set the number adapters 8 to align with the eight tasks and maintained the rank ture, the actual trainable parameter count of HydraLoRAA is approximately 4.5x x that of MoSA (or standard LoRA). The results are presented in Table 9. HydraLoRA achieves an average accuracy of 87.60 % across the eight datasets. Despite the significantly larger parameter budget, its performance highly comparable to MoSA, which achieves an average of 87.66%. This demonstrates that MoSA effectively captures complex multi-task knowledge distributions through its sparse activation mech- anism, without requiring the explicit routing structures the massive parameter overhead typical of specialized multi-task methods.
32. It is crucial to note that due its multi-branch architec-
Table 9: Performance comparison between MoSA and HydraLoRA (k 8, sense reasoning tasks.
32) common-
Method
Params (%)
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaSwag
Wino
Avg.
MoSA
0.70
75.64
90.65
82.70
82.91
93.27
89.48
96.57
89.78
87.63
HydraLoRA
3.60
75.03
91.11
82.46
84.21
93.71
88.28
96.90
89.06
87.60
B.11 SENSITIVITY ANALYSIS OF LEARNING RATE
To evaluate the robustness of MoSA with respect to hyperparameter settings, we conducted sen- sitivity analysis on the learning rate. We utilized the Llama-3-8B model with rank of r 32 and evaluated performance across eight standard commonsense reasoning benchmarks. We varied the learning rate within the range of (5 10-6 x 10-5 55, 2 x 10-5, 10-5), keeping all other hyperparameters constant.
The results are summarized in Table 10. We observe that MoSA exhibits stable performance across a broad range of learning rates. Specifically, the performance peaks at learning rate of 1 x 10-5 achieving the highest average accuracy 87.633 and leading in out individual tasks. Fur- thermore, even when the learning rate deviates to 5 10 accuracy remains within 0.55%, demonstrating that our method is not overly sensitive to learning rate variations.
10 the fluctuation in average
19

---

Published as a conference paper at ICLR 2026
Table 10: Ablation study learning rate sensitivity using Llama-3-8B 32). Bold indicates the best result.
Learning Rate
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaSwag
WinoGrande
Average
5e-6
75.09
89.98
82.66
84.13
93.38
88.28
96.18
88.91
87.33
le-5
75.64
90.65
82.70
82.91
93.27
89.48
96.57
89.78
87.63
2e-5
75.06
89.76
80.70
83.80
93.13
88.
96.58
89.84
87.12
5e-5
74.52
89.28
80.85
82.65
91.69
87.89
95.52
87.27
86.21
Wg
W4
W,
Was
W Poow
3390
000
600
1070
2093
2020
20.00
2002
200
200
1020
Figure 4: Rank of Weight Updates. We compare the effective rank of AW learned by LoRA and MoSA on LLaMA-3-8B. LoRA is strictly bounded by its low-rank bottleneck. In contrast, MoSA yields full-rank update, utilizing significantly more dimensions in the weight space capture complex features.
B.12 RANK ANALYSIS
We calculate the rank update of MoSA and LoRA trained in LLaMA-3-8B on the commonsense dataset. We then compute the rank by SVD decomposition for singular values exceeding a threshold of 0.01. As shown in Figure 4, LoRA is mathematically constrained to low-rank subspace, limiting the number of active dimensions to its pre-defined rank Conversely, MoSA achieves full-rank update.
C SPEED AND GPU MEMORY USAGE
C.1 SPEED
We benchmark the step time and throughput of MoSA against three strong parameter-efficien base- lines: LoRA, DoRA, and HiRA. All methods are implemented in the same codebase and run on the same hardware with sequence length 512, batch size 8, and adapter rank = 32. We use Adam as the optimizer and report the average wall-clock time per training step and tokens-per-second over 50 iterations, after discarding warm-up steps.
Table 11 summarizes the results on Qwen3-4B Ba-a and Llama3-8B. On both models, LoRA attains the highest throughput, while optimized MoSA implementation remains within the order of magnitude as LoRA/DoR A/HiRA and consistently much faster than the naive MoSA- A-toorrdd variant, which uses the default Pytorch backward implementation. The latter serves as ablation that highlights the benefit of our segmented-reduse reduction implementation.
C.2 MEMORY
For 3.1, although we formulate the gradient aggregation in Eq. 4 as a matrix- vector product V = SPu, explicitly storing the matrices S and P is computationally inefficient. Our implementation utilizes the BRT structure to represent these operators implicitly, reducing memory complexity from quadratic linear.
Implicit Permutation (P). Instead of storing a permutation matrix P E {0, N we store a single index vector ग NV containing the sort order of the group assignments. The matrix-vector
20

---

Published as a conference paper at ICLR 2026
Table 11: Training speed comparison sequence length 512, batch size rank 32, Adam optimizer. We report average step time and throughput over 50 iterations.
Model
Method
Step time J. (ms)
Tokens/s 1
LoRA
360.07
11,375
HiRA
386.12
10,608
Qwen3-4B
DoRA
440.63
9,296
MoSA(Ours)
420.79
9,734
MoSA-Autograd
1215.56
3,370
LoRA
416.81
9,827
HiRA
505.64
8,101
Llama3-8B
DoRA
532.81
7,688
MoSA(Ours)
596.54
6,866
MoS Autto-rd
2098.53
1,952
product Pu is implemented as a direct memory gather operation, where the i-th element of the permuted vector is simply un, This reduces the storage requirement O(N).
Implicit Segmentation (S). The segmentation matrix S {0, 1] normally encodes which elements belong to which group. Under BRT, we enforce that all groups have a uniform size s N/K. This regularity eliminates the need store S. Since the permutation P makes group elements contiguous, the aggregation S(Pu) is mathematically equivalent to reshaping the permuted vector into a K wig matrix and computing the sum along the second dimension. This operation requires no additional metadata storage.
For the index matrix, we save I' shown in B.8. Therefore, the only additional cost is small meta- data buffer that stores the tessera index and sort order for each weight element support the segmented-reduccte reduction kernel. This additional memory is a fixed constant once the partition is chosen. It depends only the weight shapes and grouping, not batch size, sequence length, optimizer, or rank r. On Qwen3-4B-Bass and Llama3-8B, the buffer is about 470 MiB and 1088 MiB, respec- tively. This small overhead largely due the structure sharing strategy in Sec. 3.1, where layers with identical shapes reuse the same partition so that we store single index buffer per unique shape instead of per layer.
D ANALYSIS OF GENERALIZATION TO ADAPTIVE OPTIMIZERS
In this section, we extend our analysis from first-order SGD updates to adaptive optimization meth- ods, specifically Adam, using the same notation in Sec. 3.2. We show that balanced tessellations remain critical for maintaining uniform update magnitudes across different subspaces.
D.1 GRADIENT STATISTICS UNDER AGGREGATION
Recall that MoSA updates scalar parameter A that aggregates gradients from group I& of size my = [ Let G = Vw L E xd denote the gradient, and write its entries as Giy. We assume the entries i.i.d. with mean y and variance 2
E(G = μs,
Var(G o².
(11)
The gradient with respect to the scalar \ is the Frobenius inner product between G and the mask Ma:
= Vx, L. (G)
E Gij- (W))
(12)
21

---

Published as a conference paper at ICLR 2026
Thus the first and second moments of dA scale with the group size my:
E0 E[gh] = miH,
(13)
E (gx)²] = Var(gs) + (Eigx))* = m2 2 + mmg.
(14)
D.2 EFFECTIVE UPDATE MAGNITUDE WITH ADAM
Adam updates parameters using the ratio of the first-moment estimate (denoted mgg) the square root of the exponential moving averages approximate the corresponding expectations, the update A is ap- proximately
estimate (denoted iz). For the scalar X at steady state where the
Elghl
mgp
A 17
(15)
[ox))
Vmnc² + mmju +
+
Mapping this scalar update back to the weight space, the effective increment for any weight in group satisfies
v(i.j) E Ia- We now analyze how the magnitude of this update scales with my in two regimes.
SW AAR.
(16)
Regime 1: Variance-dominated. In deep networks, gradient variance often dominates the mean
(o² >> 1²t (Mori et al., 2022), especially in early training or for certain layers. When the variance term dominates the second moment, i.e., o² > myH² we have
myj
1
SW -1 Vm
mk-
(17)
Observation. Even with Adam's 's normalization, the effective step size still scales as /mg- If group sizes are unbalanced (e.g., s. one group is x larger than another), the larger group will receive updates with approximately 2 the magnitude.
Regime 2: Signal-dominated. If the mean gradient is strong, i.e., n¹² >> 02/ /mg, then the second- moment term dominated by the squared mean:
W -1 mal
m sgg(())
(18)
In this idealized regime, Adam successfully normalizes the update scale, and the dependence on my vanishes. However, practical training typically oscillates between the variance-dominated and signal-dominated regimes.
As shown in Eq. (17), under the common variance-dominated condition the effective learning rate for group is proportional to /mg An imbalanced partition therefore implies that different parts of the weight matrix optimized with effectively different learning rates. To ensure consistent optimization dynamics and stable moment estimation across all parameters, the group sizes must be kept approximately constant. This theoretically justifies the necessity of balanced tessellations even when using adaptive optimizers such as Adam.
22


Reviews:
[{"review_id": "V4mGI5VtYU", "reviewer": "Reviewer_JoHz", "summary": "This paper introduces MoSA, a PEFT method that constructs an adapted weight matrix by broadcasting a small set of learned scalars over a pre-defined group assignment of weight entries. MoSA features high-rank expressivity, arbitrary granularity, and non-local parameter-sharing regularization. MoSA is also coupled with efficiency techniques like Segmented Reduction and pre-caching to speed up training.              Experiments demonstrate MoSA’s performance advantage over LoRA, DoRA, and high-rank methods like MoRA and HiRA under the same updated parameter budget.", "questions": "1. Could the authors provide the speed and memory comparison with LoRA?]\n2. See other weaknesses above.\n3. Is MoSA more sensitive to the learning rate or not?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a novel PEFT method, first grouping original weight entries, then fine-tuning them with scaling parameters.                   This design ensures a high-rank update during fine-tuning.\n2. MoSA’s realization is theory-grounded. The paper offers a theoretical view supporting its Balanced Random Tessellation design.\n3. Performance improvement over baselines is significant.", "weaknesses": "1. Additional memory cost and latency. Since MoSA needs to precompute and cache (g, π, m, o) for each unique layer shape, it means MoSA must store more than two extra full-dimensional matrices, which are not present in methods like LoRA. Additionally, although the gather operation is optimized, it is still a slower operation than LoRA’s matrix multiply. This paper doesn’t include a direct speed comparison with LoRA.\n2. Lacks stronger baselines. Stronger methods (e.g., LoRA-GA, LoRA-pro, PiSSA, HD-PiSSA…) are not compared. DoRA, MiRA, and HiRA are not strong baselines.\n3. No updated rank analysis, which cannot support the paper’s high-rank expressivity claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7lYLFtLJ2", "reviewer": "Reviewer_xCoM", "summary": "This paper introduces MoSA, a parameter-efficient fine-tuning method that replaces low-rank adaptation with fine-grained, randomized sharing of weight updates. Each weight matrix is adapted by broadcasting a small set of learned scalars over a fixed tessellation, enabling expressive updates without changing the model architecture and allowing zero-overhead inference. Experiments on diverse language understanding and generation tasks show that MoSA matches or exceeds strong PEFT baselines. Analyses indicate that non-local parameter sharing acts as a regularizer, and design choices in grouping and budget allocation control the expressivity–efficiency trade-off, positioning MoSA as a simple, scalable alternative to LoRA.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is innovative and interesting.\n2. The paper is well written and easy to follow.\n3. The approach achieves strong empirical results, outperforming or equaling established PEFT baselines on diverse language modeling and generation benchmarks.", "weaknesses": "1. The work lacks theoretical analysis to explain why the proposed fine-grained, randomized weight-sharing mechanism is effective, leaving the underlying principles largely empirical.\n2. The experiments are conducted only on the LLaMA series. Including evaluations on additional models, such as Qwen, would strengthen the empirical validation of the method.\n3. MoSA is implemented only for linear layers and cannot be directly applied to convolutional layers, which may limit its generality across different model architectures.\n4. How is the hyperparameter k controlled so that MoSA uses the same number of trainable parameters as LoRA with rank  r=32?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kxJ8VrTWKg", "reviewer": "Reviewer_xciz", "summary": "This paper introduces MoSA (Mosaic Shared Adaptation), a new parameter-efficient fine-tuning (PEFT) method for large language models (LLMs) that replaces LoRA’s low-rank decomposition with randomized, fine-grained parameter sharing. Instead of factorizing weight updates, MoSA partitions each weight matrix into a fixed number of groups (tesserae) and assigns a single learnable scalar per group, broadcast across all elements within it. This design enables high-rank expressivity under the same parameter budget as LoRA while maintaining full architectural compatibility and zero-overhead inference. The authors also propose an efficient segmented-reduction backward kernel that aggregates gradients per group without atomics, providing substantial training speedups. Empirically, MoSA is evaluated on commonsense reasoning (8 datasets), open-domain dialogue (ConvAI2), and mathematical reasoning (MetaMathQA→GSM8K) using Llama-2-7B and Llama-3-8B. Across all tasks, MoSA outperforms LoRA, DoRA, MoRA, and HiRA under matched parameter budgets (≈0.7–0.8% of model parameters).  A first-order theoretical analysis shows that balanced random tessellations minimize expected gradient discrepancy, formalizing why random grouping works as a regularizer.", "questions": "1. Expressivity Justification: Can the authors provide empirical evidence (e.g., singular value spectra) to demonstrate that MoSA indeed produces higher-rank ∆W updates than LoRA or DoRA at the same budget?\n\n2. Gradient Projection Proof: Theorem 1 claims near-optimality of balanced tessellations; can the authors formalize this derivation in the main text and explain how it generalizes beyond first-order updates?\n\n3. Ablation Granularity: How sensitive is MoSA’s performance to the random seed of tessellation assignment? Is grouping deterministic per layer or re-sampled per run?\n\n4. Training Efficiency: What are the memory and time trade-offs versus LoRA for large-scale training (e.g., Llama-3-70B)? Is the backward kernel still efficient when scaling to multi-GPU distributed setups?\n\n5. Interoperability: Could MoSA be combined with quantization (like QLoRA) or low-rank initialization (hybrid approaches)? Would grouping interact with quantized layers?\n\n6. Generalization Analysis: Why does MoSA achieve such a large margin (+7% on GSM8K)? Any insights into whether the tessellation acts as regularization, or is this due to implicit gradient smoothing?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Conceptual Novelty: MoSA breaks the low-rank assumption dominating PEFT (LoRA, DoRA) by proposing non-local, groupwise constant updates, a different structural prior.\n\n2. Strong Empirical Evidence: The paper benchmarks across three families of tasks (reasoning, dialogue, OOD generalization) and two strong base models (Llama-2-7B, Llama-3-8B), all under strict budget parity. Results are consistent and robust.\n\n3. Thorough Ablations: The study systematically probes grouping strategy, component selection, and budget scaling, reinforcing", "weaknesses": "1. Limited Theoretical Depth: The analysis in §3.1 (Theorem 1, in appendix) stops at first-order approximation. It does not address the *optimization dynamics* or convergence implications of groupwise sharing.\n\n2. Potential Overclaim on High-Rank Expressivity: While empirically strong, the claim that MoSA can realize “high-rank updates” under small budgets is not theoretically substantiated (no explicit rank analysis of ∆W(λ)).\n\n3. Lack of Comparison on Memory/Latency Trade-offs: The paper claims “zero-overhead inference,” but training-time memory and backward compute overhead are only partially reported (speed benchmarks but no GPU memory profiling).\n\n4. Missing Larger-Scale or Real-World Fine-tuning Tasks: Only medium-scale reasoning and dialogue datasets are used; domain adaptation tasks (e.g., summarization, instruction tuning) are absent.\n\n5. Presentation Minor Issues: The flattened notation and symbol reuse (e.g., Mh, Gh, g, π) are hard to follow for readers unfamiliar with vectorized representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G5At0kAi3j", "reviewer": "Reviewer_Vc9B", "summary": "This paper introduces MOSA, a novel PEFT method that replaces LoRA's low-rank constraint with randomized, fine-grained parameter sharing. MOSA learns a single scalar update for pre-defined, size-balanced groups of weights (tessellations), enabling high-rank updates with no architectural changes or inference overhead. This balanced sharing acts as a strong regularizer, allowing MOSA to consistently outperform strong baselines like LoRA and HiRA in commonsense reasoning, dialogue, and out-of-distribution math tasks under matched budgets. The method is extremely parameter-efficient, notably surpassing a LoRA r=32 baseline while using only an r=1 equivalent budget.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Superior Performance and Generalization: MOSA consistently outperforms strong PEFT baselines like LoRA, DoRA, and HiRA across diverse benchmarks, including commonsense reasoning and dialogue. Its advantage is particularly pronounced in out-of-distribution generalization, where it shows a superior ability to learn abstract reasoning principles.\n- Extreme Parameter Efficiency: The method is highly efficient, achieving strong results at a fraction of the parameter cost. It can surpass the performance of much larger, well-established baselines while using only a very small budget, suggesting its sharing mechanism is a more effective use of parameters.\n- Flexibility and Simplicity: It offers a simple alternative to LoRA by replacing the low-rank constraint with fine-grained parameter sharing. This design enables expressive, high-rank updates and provides highly granular budget control, as the number of trainable groups is not constrained by matrix shapes.\n- Inference and Training Efficiency: MOSA requires no architectural changes and can be losslessly merged into the base model, resulting in zero-overhead inference. For training, it introduces a highly optimized segmented-reduction backward kernel that significantly accelerates gradient computation compared to standard autograd and scales robustly.", "weaknesses": "- `Confusing Terminology`: The term \"Mosaic\" fails to adequately reflect the method's core concept. I noted from the supplementary code that it was apparently previously named \"Group-Share Adaptation.\" Clarification on the reason for this change is crucial for understanding the method and the paper's authenticity.\n- `Limited Comparison`: The Commonsense Reasoning scenario employs a multi-task setup, yet the paper lacks a comparison against the wide range of existing MoE-based multi-task LoRA variants [1-3].\n- `Unclear Presentation`: The paper's presentation is poor. For instance, Table 1 (referenced in Section 2) includes methods like DoRA that are never introduced, making the paper appear disjointed. The methodology section is inundated with formulas that feature arbitrary and unprofessional subscript notation. Furthermore, Figure 1 consumes excessive space while failing to convey sufficient methodological detail. The clarity of the paper's presentation is essential, and I will determine whether to update the final score based on the revised version.\n\n[1] When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications\n\n[2] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning\n\n[3] CoLA: Collaborative Low-Rank Adaptation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)