MiSS: Revisiting the Trade-off in LoRA with an Efficient Shard-Sharing Structure
Authors: Jiale Kang
Qingyu Yin
Venue: ICLR 2026 Poster (2026)
Keywords: PEFT
LLM
LoRA
Primary area: foundation or frontier models, including LLMs

Abstract: Low-Rank Adaptation (LoRA) is a widely adopted technique for parameter-efficient fine-tuning, but its slow convergence has spurred the development of numerous variants. Nevertheless, current approaches struggle to achieve simultaneous improvements in performance, memory footprint, and computational efficiency. To address this challenge, we revisit the causes of LoRA’s slow convergence and, based on these insights, propose \textbf{M}atr\textbf{i}x \textbf{S}hard \textbf{S}haring (MiSS) that shards the original weight matrix and updates by sharing a single trainable matrix $\boldsymbol{D}$ initialized to zero. To simultaneously ensure computational efficiency, low memory footprint, and scalable serving, we introduce MiSS$^e$. Through theoretical analyses and empirical results, our method reduces optimization complexity while maintaining strong performance, striking a favorable balance between performance, memory, and efficiency. Furthermore, we provide a comprehensive analysis of different PEFT methods with respect to memory usage, initialization time, and computational efficiency. By mapping the Pareto frontier, we show that MiSS achieves a favorable balance across these dimensions, integrating the strengths of prior approaches.

Published as a conference paper at ICLR 2026
MISS: REVISITING THE TRADE-OFF IN LORA WITH AN EFFICIENT SHARD-SHARING STRUCTURE
Jiale Kang" Yuanshi Inc
Qingyu Yin Zhejiang University
ABSTRACT
Low-Rank Adaptation (LoRA) is widely adopted technique for parameter- efficient fine-tuning, but its slow convergence has spurred the development of numerous variants. Nevertheless, existing methods often fail to improve perfor- mance, memory footprint, and computational efficiency simultancously. To ad- dress this challenge, we revisit the causes of LoRA's slow convergence. Building on these insights, we propose Matrix Shard Sharing (MiSS), which updates shards of the original weight matrix using single shared trainable matrix D, initialized to zeros. To simultaneously ensure computational efficiency, low memory foot- print, and scalable serving, we introduce MiSS Both theoretical analysis and empirical results demonstrate that our method reduces optimization complexity without compromising performance, thereby achieving a more favorable trade-off among performance, memory, and efficiency. Furthermore, we conduct com- prehensive comparative analysis of various PEFT methods, evaluating their mem- ory usage, initialization overhead, and computational efficiency. By mapping the Pareto frontier across these dimensions, we show that MiSS occupies a favorable position, effectively capturing the advantages of prior approaches.
INTRODUCTION
Fine-tuning Large Language Models (LLMs) (Radford et al., 2019; Raffel et al., 2020; Yin et al., 2024) is a prevalent methodology for adapting these models to specific downstream tasks. How- ever, full fine-tuning of all parameters is computationally prohibitive. Consequently, numerous Parameter-Ffficiiett Fine-Tuning (PEFT) techniques (Xu et al., 2023) have been developed to mit- igate the training expenditure associated with these large-scale models. Among such techniques, Low-Rank Adaptation (LoRA) (Hu et al., 2021) has distinguished itself as one of the most promi- nent PEFT methods. LoRA employs low-rank approximation for the weight updates, strategy that offers markedly reduced number of tunable parameters, notable efficacy when compared to full fine-tunie fine-tuning, and the potential for zero inference overhead. LoRA constructs this low-rank adap- tation matrix through an intuitive design, positing that the weight update AW can be approximated by the product of two lower-rank matrices, B BA+ R AW. Evidently, this specific factorization is not necessarily the optimal low-rank approximation of the original AW.
Many improvements to LoRA have been proposed in recent years, which can be broadly categorized into two major streams: (1) Adaptability (Ding et al., 2023; Liu et al., 2024; Biderman et al., 2024): This refers the convergence speed which the method reaches an optimal or near-optimal state. The approximation must exhibit a representational capacity comparable to that of the original, full AW. Extensive experiments have shown that LoRA's convergence is significantly slower compared to full fine-tuning. To address this issue, researchers have proposed several LoRA variants (Hayou al., 2024; Meng et al., 2024; Wang et al., 2024a). By adopting different initialization strategies to influence the model's 's training gradients, they have accelerated LoRA's convergence speed. Dif- ferent initializations of LoRA variants accelerate convergence essentially by increasing the initial gradients during training or aligning them with the full-scale training gradients. However, many of
*Correspondence kangjiale827 @ @gmail.com

---

Published as a conference paper at ICLR 2026
Table 1: A variety of LoRA variants are listed, each with its specific update formulation and initial- ization strategy for the low-rank matrices. The differences between these methods are compared in a clear and intuitive manner. denotes efficient form.
Method
Forward
Initialization
LoRA
y Wux BAr
A N(O, B ~
PISSA
y Wox BAr
A = Utan (aay B (maar (arl
AdaLoRA
y W + PAQE
O, P,Q N(0,0")
DoRA
y ml Wox + BAr / | BAll.)
A Meit.KaimingUnti B ~
ProLoRA
y Wor + (Bu @h ...) (Au @v
A w ~ KaimingUnif, Bu ~
MoS
= Wo + A'
A A Braij
MiSS(Ours)
= Wo + expand(D).
D~0
(Ours(Ours)
y Wo + D D
D~0
these methods overlook issues of computational efficiency and overall training overhead. For ex- ample, PiSSA (Meng et al., 2024) requires a lengthy initialization process, while LoRA-GA (Wang al., 2024b) depends modifications to the optimizer, resulting in incompatibility with certain optimizers. (2) Efficiency (Kopiczko et al., 2024; Wang et al., 2024c; 2025): This encompasses expeditious initialization, modest memory consumption, and minimal computational overhead. Op- timizing LoRA from an efficiency perspective can lead to reduced VRAM consumption and an accelerated training process. Although LoRA has demonstrated significant advantages in reducing parameter scale and computational cost, its effectiveness still falls short of fully matching full fine- tuning. To address this gap, researchers have proposed an increasing number of LoRA variants that gradually approach the performance of full fine-tuning. This raises a natural question:
Given the inherent challenge for LoRA and its variants to balance performance, memory, and efficiency, how we achieve effective trade-off among all three dimensions?
To strike balance between performance, memory, and efficiency, we re-examined the key factors affecting LoRA's slow convergence. Through analysis of F2FT (Yang et al., 2024), LoRA- FA (Zhang et al., 2023), and LoRA+ (Hayou et al., 2024), we identified a critical phenomenon:
During the LoRA fine-tuning process, both matrices B and A need be updated simultaneously, which increases the complexity of optimization and ultimately leads to slower convergence.
LoRA+ alleviates this issue by modifying the initial gradients, allowing the fine-tuning process to approximate full fine-tuning better. In contrast, S fixes one matrix orthogonal matrix, ducing the degrees of freedom in parameter updates and lowering optimization complexity, thereby enabling faster alignment with the optimal update direction. Inspired by these insights, we hypoth- esize that training only a single matrix could simplify optimization without sacrificing expressive capacity. We therefore propose Matrix Shard Sharing (MiSS), method that updates shard of the original weight matrix using a single, shared trainable matrix D, initialized to zero. Thus, our approach maintains the low-rank property of the matrices while offering efficient alternative to B A updates in terms of computation.
Gradient Norm Analysis. We analyze the initial gradient norm to verify our preliminary conclu- sions. In the experimental sections of the PiSSA, S2FT, and LoRA-GA papers, we observed that LoRA exhibits a very small initial gradient norm compared full fine-tuning, which shows a much larger one. Notably, all these improved methods share a common characteristic: their initial gradient norms are significantly larger than LoRA, and their early-stage convergence speed is comparable to that of full fine-tuning, Motivated by this, evaluated the initial gradient norms of different meth- ods across various models and datasets to examine whether MiSS follows the same pattern as other LoRA variants. The experimental results (Figure 1) confirm that MiSS indeed shares this property, ie., a larger initial gradient norm and faster early convergence. This also supports the hypothesis that optimizing single matrix is inherently simpler.
2

---

Published as a conference paper at ICLR 2026
OndlentNorm
GradeneNorn
Coulieatoeen
ComiientNeem
95
LaRA
PISSA
MISS
128
256
32
64
125
256
32
04
1256
756
52
041
17%
155
Marix
FAnk
Manix
Mastx
Figure 1: Comparison of initial gradient norms across different training methods and the effect of rank. Results are shown for LLaMA2-7B and Qwen3-4B on the Math and Code datasets.
Efficient Implementation To achieve better computational efficiency, introduce MiSS", an alternative design that maintains the core principle parameter sharing while offering improved time and space complexity through input-dimension aggregation. We further conduct extensive experiments (Table 2) to validate its effectiveness.
We first evaluate MiSS both Natural Language Understanding (NLU) and Generation (NLG) tasks, assessing its performance and scalability. Our results show that MiSS consistently outper- forms LoRA and its variants across diverse LLM architectures, establishing new state-of-the-art results on a wide range of metrics, We then analyze the Pareto frontier of the adaptabilit--efficienny trade-off in PEFT. We argue that an ideal PEFT method should effectively balance these two es- sential dimensions. To this end, we conduct a series of foundational experiments, including a sim- ulated pre-training and fine-tuning pipeline, computational complexity analysis, and initialization time evaluation. With comprehensive empirical results, we demonstrate that MiSS achieves favor- able balance across three key dimensions performance, memory, and efficiency, highlighting its practicality as general PEFT solution.
Our contributions can be summarized as follows:
1. We propose MiSS, efficient and adaptable structure with shard-sharing mechanism, striking an effective balance among three essential properties-performaane memory effi- ciency, and computational efficiency.
2. Through large-scale experiments across diverse datasets and model architectures, pro- vide a comprehensive evaluation of multiple PEFT methods. Our empirical results con- clusively demonstrate that MiSS achieves superior balance among these three properties compared to existing alternatives.
2 PRELIMINARIES AND RELATED WORKS
Low-Rank Adaptation (LoRA). Parameter-Effiiien Fine-Tuning (PEFT) refers to a family of techniques designed to adapt large pre-trained models downstream tasks while minimizing the number of trainable parameters, thereby reducing computational and memory overhead. Among diverse methods, Low-Rank Adaptation (LoRA) has gained significant prominence. operates on the principle that the change in weights during model adaptation often possesses a low intrinsic rank. Instead of fine-tuning the entire pre-trained weight matrix Wo E Rd LoRA introduces low-rank decomposition represent the update. Consider simple linear projection with input E IRd and output y E & LoRA adapts the following forward pass:
y= + AW) Re Wo + BAx, where E Rddrr
E Br
Here, A and B low-rank matrices, with the rank r being significantly smaller than the original dimensions i.e,, & min(d, k). During the fine-tuning process, the original weights W, are kept frozen, and only the parameters within matrices A and B are trained. Specifically. LoRA initializes A with Gaussian noise A ~ (0, o²) with small o and B with zeros, ensuring that B = 0 at the start, preserving the pre-traie pre-trained model's output.
Improvements of LoRA. LoRA is the low rank adaptation towards full-param finetuning, and intuitively it downperforms than it. Several works propose diverse methods towards a better convergence and adaptability of LoRA. One compelling venue is to change the form of LoRA. PiSSA (Meng et al., 2024) optimizes the compact parameter space by representing the matrices in the model the product of two trainable matrices, augmented with residual matrix for error
3

---

Published as a conference paper at ICLR 2026
conrection. Using Singular Value Decomposition (SVD), OLoRA (Büyükakyyüi, 2024) leverages QR decomposition to imitialize the adaptation matrices during the fine-tuning process, ensuring that these matrices are orthogonal. This orthogonal initialization helps maintain the stability of the pa- rameter space during optimization. LoRA-GA and PiSSA are similar in form, but they differ in that LoRA-GA initializes A and B by computing the initial gradient, thereby closely approximating full fine-tuning. LoRA+ extended this method by introducing independent learning rates for matrices A and B with fixed ratio, improving the method's efficiency. DoRA (Liu et al., 2024) decom- poses the weight matrix into two parts: magnitude and direction, which are optimized separately. This approach allows for more precise control over the learning rate, making LoRA updates closer to the effect of full fine-tuning. The improvements brought by these LoRA variants validate that the updates to the weights exhibit low intrinsic rank during adaptation and hold greater potential. However, they also introduce more complex initialization steps and increase preprocessing time.
No FREE LUNCH: BALANCING BETWEEN ADAPTABILITY AND EFFICIENCY
This section elucidates the fundamental trade-off inherent in LoRA-style PEFT techniques: the del- icate balance between their adaptability and efficiency. Adaptability, in this context, refers to the ca- pacity of given method emulate the performance benchmarks set by full-parameter fine-tuning. Conversely, efficiency encompasses the method's o judicious use of computational resources, specif- ically time and memory. We utilize highly artificial controlled dataset and model with a relatively small parameter count to make the verification transparently and easy for replication.
We considered diverse methods 1: (1) Full- palameaarettrr finetuning (L.v et al., 2024). (2) LoRA (Hu et al., 2021). (3) Alternatives to LoRA w/ different architectures, including: PiSSA (Meng et al.,
2024), VeRA (Kopiczko et al., 2024), DoRA (Liu et al., 2024) and MoRA (Jiang et al., 2024).
(4) Efficent LoRA Design that keeps the LoRA BA structure: PROLORA (Wang et al., 2024c), MoS (Wang et al., 2025). (1) An overview of their forward form, initialization method can be found at Table 1.
3.1 EMPIRICALLY BENCHMARKING THE ADAPTABILITY OF LORA VARIANTS
Experimental Setup. Parameter-effiiiet adaptation methods, particularly those leveraging low- rank principles, typically constrain trainable parameters by applying low-rank decompositions either newly introduced adapter matrices or the updates of pre-existing model weights. To rigorously evaluate such strategies, we selected a deliberately minimalistic base model: single-layer MLP designed process series of features and yield outputs. This model is initially pre-trained to fit some sinusoidal functions using constrained set of data points. Following this pre- pre-training, the target function is subtly altered, and an additional dataset sampled from this modified function is employed for training to assess the adaptation performance of various fine-tuning techniques. Comprehensive details regarding the experimental settings are elaborated in Appendix C.
Results. Figure illustrates the comparative adaptability of different methods. We utilize the min- imum validation loss achieved by each approach indicator of its expressive capacity when approximating the performance of full-parameter fine-tuning. The results clearly demonstrate that methods leveraging singular value decomposition (SVD), such as PiSSA, attain relatively low loss. Conversely, efficiency-focused techniques like MoS exhibit higher losses. A plausible ex- planation for this discrepancy is that such methods further decompose LoRA matrices into shared components, which may inherently constrain their expressive power. Our method MiSS reaches a relatively advanced performance comparing to other variants.
We have not included methods such as LoRA-GA -AA (Wang et al., 2024b) or LoRA+ (Hayou et al., 2024) in our current analysis. While these approaches aim to more closely approximate the performance of full- parameter fine-tuning, we consider MiSS to be largely orthogonal to them. Consequently, the analytical tech- niques employed in their study may still offer valuable insights for MiSS.
4

---

Published as a conference paper at ICLR 2026
Train Loss-Step
Time Params
Train Time Params
2x
0.7
Average Init Time (ms)
Average Train Time (ms)
0.5
1.0
Train Lows
0.4
10
0.3
MISS
10-1
0.2
4x10-
25
50
75
100
10a
10a
10
102
107
10
Stop
Number of Parameters
Number Parameters
AdaLoRA
DORA
LoRA
-B-
Mo5
PISSA
ProLoRA
VeRA
MISS
full, finnnuue
Figure 2: No Free Launch Experiment. Left. The training loss curves of all methods. Middle. Initialization time w/ parameters. Right. Training time w/ parameters.
3.2 EFFICIENCY ANALYSIS OF LoRA VARIANTS
Metrics. We evaluate the efficiency of LoRA-like variants from two primary perspectives: (1) Space and Time Complexity in Training. Space and time complexity during training generally considered crucial criteria for evaluating PEFT methods. To benchmark these aspects, we employ the model architecture detailed in Section 3.1. We also test the real cost in our experiment section ie., Section 5.3. (2) Initialization. Initialization time is often overlooked in theoretical complexity analyses. This oversight typically stems from the assumption that common initialization techniques (e.g., Kaiming Initialization) are computationally inexpensive and represent a one- time cost within the entire training pipeline. However, several recent advancements in LoRA and its variants incorpo- rate matrix operations (e.g.. Singular Value Decomposition SVD) that are not inherently hardware- friendly and can pose challenges for efficient optimization and computation. Consequently, we explicitly include initialization time as a distinct evaluation metric in our experimental framework. We then progressively scale the trainable parameter count of various approaches to meticulously measure their respective time and space costs.
The efficacy (See Figure 2) of MiSS is evident: its strategic combination of parameter sharing and an efficient computational design culminates rapid, scalable performance across both initialization and training stages. In contrast, while techniques like PiSSA demonstrate commend- able adaptability, as shown in prior experiments, their reliance on computationally intensive Singular Value Decomposition for initialization significantly hampers their overall speed. Other approaches, such as VeRA and AdaLoRAA offer efficient initialization and computation; however, as previously discussed, they often achieve this at the cost of comparatively reduced adaptability.
Results.
MISS: SHARD SHARING FOR THE PERFORMANCE AND EFFICIENCY TRADEOFF
4.1 METHOD OVERVIEW
In traditional low-rank adaptation methods egg., LoRA, the weight update AW is approximated as a low-rank matrix, e.g., AW BA, where E Rrxk E Rd and the rank r & min(d, k)) This approach achieves efficiency by limiting the number of parameters. However, we observe that a repeating matrix- small matrix is replicated to form a larger one-can also viewed as a low-rank structure. For instance, if matrix's 's rows shards are constructed by repeating a limited set of independent elements, its effective rank is often much smaller than its full dimensions.
Based on this insight, we propose MiSS, which defines the weight update AW large matrix generated from a small trainable matrix D through an expansion operation. The updating of W and the forward pass be expressed as:
W Wo + AW Wo + expand(D), y Woz + expand(D)az
(2)
Here, R6 bxlik y E Rbxidd Wo E Raxk is the pre-trained weight matrix, D E Xri is small trainable matrix with (r1,72) & min(d, k), and expand D) a function that extends to Rack, This structure inherently exhibits low-rank properties. Since the rows within each shard
5

---

Published as a conference paper at ICLR 2026
expand
def init fen_uuuteees: int, inffesturess inty ranks (at)s self.s zank sel f. weight = on Parerrett (torch.erg (torct.engey enpry features, (natetau))s));
A-N(0,,²))
D-0
def forkard tse x)s
aelc.0 on.Parameler (Lorch.
(ealures))
result Pilinear 1s, self.wightt 4 ai [B, 6l result
LoRA
MiSs
Figure 3: Left. Structural diagram of AW in LoRA and MiSS. Right. PyTorch-style pseudocode illustrating the implementation of MiSS.
identical, the rank of expand (D) is at most N. When < d, AW a low-rank matrix, reducing the parameter count from d k to N x k.
Regarding the expansion method, we partition the output dimension d of Wo into shards of sizes where Σ; 8, = d. Let D E RNNk, where N the number of shards. For each shard i, its update is determined by the i-th row of D, denoted D; E B11k repeated times to form the shard's update matrix. Formally:
{s1, 82. .........' SN),
(expand(D))+ = [(1., D2)T (1.,
[
(3)
Here, 1,, E R is an anl-ones vector, and 1,, D; denotes D; repeated s; times vertically. The shards are vertically concatenated to match the dimensions of Wo.
4.2 EFFICIENT IMPLEMENTATION OF MISS
The above formulation is effective in the initialization process, as it only needs to initialize small D. However, directly computing expand D)xr has a time complexity of Obllkk) and memory complexity of (dk),, which can be computationally intensive. It is obvious that MiSS can be transformed into an efficient form that leverages the block structure of the input to avoid explicitly forming the large matrix, by redefining D Rdxr, where is tunable rank parameter. Instead of partitioning the output dimension d, we divide the input dimension & into r blocks, each of size g [k/r] (for simplicity, assume k divisible by r). For an input æ E gxxixk partition along the k dimiennsoon, and sum each block along the k-dimension:
x: (t)
(f-
E RRaxixr
(4)
x (1) (2) x(0)
(5)
9
S (9) Rxxxxxr
(6)
This enjoys the following updating term and forward pass:
AW x = DDS, y Wo + DS, where D E dx
Here S E Raxixr and DS E Rxiixd matching the dimensions of Wor.
This efficient form implicitly defines eX)and(D). such that expand(D)x= DS. Specifically, expand(D) E dd has rows corresponding to rows of D, repeated across blocks the k- dimension. F.g., if k = 6, r 3, and = 2, the i-th row of expand( D) takes values Dj. in block j = [j'/g], where j' the column index. This structure avoids storing the d x & matrix explicitly, requiring only D e gdxr, significantly reducing memory usage.
The efficient implementation of MiSS relies on an innovative input aggregation mechanism, namely blockwise input summation. We highlight its advantages through the following steps: (1) Input Partitioning and Aggregation: The aggregation exploits local redundancy in the input, preserving critical information while reducing the computational dimensionality. (2) Fast Computation: The cost of computing the efficient form is significantly lower than the original complexity. (3) Resource Savings: Memory usage drops comparing to original form.

---

Published as a conference paper at ICLR 2026
4.3 SYSTEMATIC ANALYSIS OF MEMORY AND EFFICIENCY FOR LORA AND MISS
This subsection systematically compares LoRA variants against MiSS, dissecting their intrinsic differences in memory consumption (governed by parameter count) and computational efficiency (governed by FLOPs and operator type). Our analysis centers on the core update formulations: AWx BAx for LoRA, versus AW DS for the efficient form of MiSS (MiSS*), where S denotes the blockwise imput aggregation. We denote the LoRA rank as TL, MiSS rank as TM. with input dimension & and output dimension
Limitations of LoRA Variants: Parameter Reduction * Computational Speedup As illus- trated in Table 2, there exists a fundamental misalignment between parameter efficiency and com- putational cost in existing PEFT methods. While variants like AdaLoRA, DoRA, and VeRA signif- icantly reduce Trainable Parameters (TPs) through novel initialization or decomposition strategies, they almost universally inherit the sequential matrix multiplication logic ((Ax). Consequently, their Space Complexity and FLOPs remain bound by the O( + r) lower limit. Furthermore, sophisticated variants such LoHA introduce additional structural overhead (e.g., the 2r factor), causing actual memory occupancy and latency to exceed the original LoRA despite having fewer trainable parameters.
Table 2: Comparison of PEFT Methods. Note that while distinct LoRA variants reduce TPs, they fail to improve Space Complexity and FLOPs due to the unchanged sequential computation, unlike the proposed MiSS.
Methods
Space Complexity
FLOPs
TPs d
FT
O(d
Old x
LoRA
O((d+k) x
O(d B) x
kd+ -r
LoRA-FA
k((d+ x
O(dd x
d.r
AdaLoRA
O(d+++r)) xr)
O(d k x)
(d -r 2
LoHA
O(2r kd+
x x kd)
2. (. (d)
k)
VeRA
O((d+krr+t) d)
d
MISS
O(dxr)
O(kddx))
d.r
Single-Matrix Paradigm and Computational Decomposition MISS fundamentally diverges from the standard LoRA architecture by employing a single low-rank matrix D E 1rixr= rather than the dual-matrix structure (A, B). Crucially, we observe that in MiSS is dimensionally consistent with B in LoRA, as both correspond to the output dimension d and function as the out- put operation matrix. This structural alignment allows us to naturally decompose the computation into two distinct stages: Input Transformation (Cstep 1) and Output Projection (Cetep). 2). This insight isolates the efficiency distinction entirely to Cstep 1. While LoRA relies on an expensive matrix multiplication (Ax), MiSS utilizes a cost-efficient block summation (sun(x)). The comparative analysis is summarized below:
Table 3: Computational Decomposition of MiSS vs. LoRA
Metric
LoRA
MISS
Structure
Dual Matrices (A, B)
Single Matrix (D)
2sup2 (Output Projection)
Matrix Mult. Bh (d x r)
Matrix Mult. DS (d x r)
Csue 1 (Input Transform)
Matrix Mult. Ax (O(BLkr)
Block Sum sum(x) (O(BLK)
Parameter Count (N)
O(r(k + d)) O(B (ackrr+rd)) +rd)
O(rd)
Total FLOPs
(BL(k+rd))
EXPERIMENTS
In this section, we conduct comprehensive set of experiments to validate the effectiveness and generalizability of MiSS across diverse domains. We assess performance on a wide range of tasks, including language, image, and video benchmarks. Specifically,
evaluate Natural Language Understanding (NLU) capabilities using a subset of the GLUE dataset, and Natural Language Gen- eration (NLG) capabilities by fine-tuning various large language models (LLMs). We extend

---

Published as a conference paper at ICLR 2026
evaluation to multimodal settings using the VTAB-IK benchmark to demonstrate the robust adapt- ability of MiSS beyond textual domains. Furthermore, we provide a detailed analysis of the Pareto frontier (Section 5.3) to definitively illustrate MiSS's superior computational efficiency and minimal hardware overhead when compared to existing Farameter-Ffficien Fine-Tunee Tuning (PEFT) methods.
5.1 SUPERIOR PERFORMANCE ACROSS LANGUAGE AND VISION DOMAINS
MiSS demonstrates exceptional versatility, maintaining commanding lead highly competitive performance across diverse benchmarks in both the language and vision domains. (Setup B)
Natural Language Understanding (NLU). On the GLUE benchmark (Table 4), fine-tuning RoBERTa-bass with MiSS showcases notable strength. It achieves an outstanding result on the challenging CoLA dataset (72.66). significantly surpassing LoRA and PiSSA. This performance indicates superior data-fitting capabilities and faster convergence on complex linguistic tasks.
Table 4: The results of fine-tuning RoBERTa-base using MiSS and various LoRA variants were compared subset of the GLUE benchmark.
Method
Trainable
MNLI
SST-2
CoLA
QNLI
MRPC
Avg
LoRA
0.236%
85.63+0.01
94.03=0.02
62.40=0.71
91.37+0.97
87.98=0.23
84.28
PiSSA
0.236%
85.72+0.40
93.6420.13
67.28+0.59
91.40+0.54
88.11#0.24
85.23
MISS
0.236%
85.71+0.32
93.60#0.07
72.86+3.13
91.43+0.76
88.14=0.60
86.35
Natural Language Generation (NLG). Across five mainstream LLMs (Llamaa, Mistral, RWKV, Qwen3), MiSS consistently achieves the best or near-best average performance (Table 5). Notably, demonstrates substantial gains in complex reasoning tasks, recording the highest Math score (34.82) on Qwen3-4B and the highest average score (47.79) on Mistral-7B. These findings highlight that MiSS is not only effective on medium-sized models but also scales robustly larger architectures and data-rich models.
Table 5: We conduct systematic comparison of LoRA, DoRA, PiSSA, and MiSS across several mainstream large language models (Llama2, RWKV, Mistral, and Qwen3). All reported results are averaged over three independent runs to ensure robustness. The first-place entry should be highlighted in bold, and the second-place entry should be underlined.
Model
Strategy
Trainable
GSMSK
Math HumanEval Mbpp
Avg
LoRA
89.9M
40.75
5.22
17.74
35.15
24.72
DoRA
91.3M
42,93
6,51
21.95
36.53
26.48
Llama2-7B (Touvron al., 2023)
PiSSA
89.9M
43.89
6.92
22.15
37.84
27.70
MISS
87.0M
48.16
8.58
23.63
36.81
29.30
LoRA
S8.IM
38.13
6.06
22.10
RWKV 6-7B (Peng al., 2024)
PISSA
S8.1M
40.48
6.12
23.30
MISS
88.1M
41.73
6.52
24.13
LoRA
94.4M
62.85
15.82
35.71
46.11
40.12
DoRA
95.8M
63.68
13.60
38.41
48.73
41.10
Mistral-7B (Jiang et al., 2023)
PiSSA
94.4M
67.01
18.13
41.28
51.37
44.45
MISS
87.0M
68.92
18.85
42.07
61.33
47.79
LoRA
250M
56,18
12.60
31.79
37.82
34.60
DoRA
252M
61.56
13.60
33.50
39.25
36.98
Llama2-13B (Touvron et al., 2023)
PISSA
250M
66,64
13.82
33.57
46.03
39.52
MISS
255M
68.64
15.74
38.15
47.91
42.11
LoRA
74.3M
84.38
15.20
73.27
78.32
62.79
DoRA
75.4M
85.11
21.73
74,20
78.77
64.95
Qwen3-4B (Yang et al., 2025)
PiSSA
74.3M
85.78
26.00
75.01
78.04
66.21
MISS
70.1M
85.52
34.82
74.48
78.05
68.22
Vision Task validate the ability of MiSS to adapt to non-textual tasks, we conducted experi- ments the VTAB-1K image and video benchmarks (TTble 6). MiSS achieved average accuracy
8

---

Published as a conference paper at ICLR 2026
of 88.02 image tasks and 72.96 video tasks, making it highly competitive with top- performing baseline methods like LoRA and DoRA. Crucially, this competitive performance is delivered with significantly lower parameter budget (+s 0.4 #TPs) compared LoRA/DoRA (<s 0.8 #TPs), firming that the efficiency of MiSS transcends the language domain and is applicable to multimodal foundation models.
Table 6: Performance comparison on VTAB-IK image and video benchmarks Results are adopted from SliceFine (Kowsher et al., 2025).
Method
Callech
Flowers
Pets
Camel.
Ears.
Relinn
K1TT1
Avg
FTP
UCF110
Kinetics
HMD3
Avg
CTPS
Full
$9.92
97.41
55.57
$1.05
58.12
73.02
17.55
$4.93
85.8.
92.30
55.25
05.79
74.59
36.05
VeRA
91.53
98.19
91.01
86.45
92.97
74.25
77.92
87.62
0.211
92.28
57.21
66.77
72.09
0.242
DoRA MISS
91.86
99.27
91.08
85.88
91.42
75.28
80.45
87.89
0.834
92.84
57.77
67.33
72.65
0.836
92.14
99.23
91.05
86.28
92.83
73.71
89.91
88.02
0.414
93.82
57.75
67.31
72.96
0.415
5.2
EFFECT OF RANK
We evaluate MiSS with varying matrix ranks to study the trade-off between tuning capacity and parameter cost. The Table 7 reports re- sults for ranks r E {16, 32, 64, 128} (corre- sponding to (21.7M, 43.5M, 37.0M, 174.0M) trainable parameters). Performance on GSM8K and the Math benchmark improves monotoni- cally as the rank increases: GSM8K rises from 45,90 at r = 16 to 53.49 at r = 128, while Math increases from 3.77 10.08. In prac- tice, r = 64 offers a favorable trade-off (48.16 GSM8K, 8.58 Math) between performance gains and parameter overhead.
Table 7: Comparing different values of rank (r) on LLaMA2-7B with MiSS.
Model
Rank
Trainable
GSM8K
Math
16
21.7M
45.90
3.77
32
43.5M
46.18
7.43
Llama2-7B
64
87.0M
48.16
8.58
128
174.0M
53.49
10.08
5.3 MISS's SUPERIOR BALANCE ON THE PARETO FRONTIER: OPTIMALLY TRADING OFF EFFICIENCY AND PERFORMANCE
The emergence of PEFT techniques is motivated by dual objectives: mitigating GPU memory con- straints and exploring more efficient model architectures. Nevertheless, numerous contemporary studies disproportionately focus on ultimate performance benchmarks, overlooking critical prac- tical considerations like computational efficiency and training duration-an emphasis that clearly diverges from the original rationale for PEFT. In this section, we undertake a multi-dimennionnl investigation into the relationships among computational overhead, efficiency, and performance for diverse models. Leveraging the official Hugging Face PEFT (Mangrulkar al., 2022) benchmarking framework, our evaluations are conducted under fair and reproducible conditions.
The Pareto frontiers in evaluation provide definitive evidence of MiSS's effectiveness. In every experimental setting, MiSS is uniquely positioned in the top-left corner-tie the optimal re- gion-signifying that it delivers the best performance with minimal efficiency cost. This consistent advantage underscores MiSS's unique contribution in balancing these competing objectives.
C00 JUMPEY
Figure 4: Pareto front of MiSS comparing with other PEFT methods. We select three more methods the baseline the balaneing of memory and performance.

---

Published as a conference paper at ICLR 2026
Table 8: Experimental results across PEFT methods on Llama-3.2-3B.
Accelerator Memory (Bytes)
PEIT Type
Total Time
Train Time
Test Accuracy
Reserved
Reserved Avg
RSLORA
2069
1871
0.5299
0,5657
22,538,092.544
17,953,927,887
12,128,059,444
C3A
2125
1924
0,5102
0.5808
22.280.142.846
17,825.917,829
11,804,454.210
MISS
1867
1664
0,5080
0.5776
20,248,002.56 560
16,303
11.170.837.033
RANDLORA
0.5072
0.5785
22.798.139_399
18,436,063,232
12.743,670,025
SHIRA
2085
1867
0,5072
0.5789
21.743.271.936
17,637.383,864
12,240,924,809
OFT
2494
2214
0,5057
0,5947
22.2992822.91
17,939,310,837
12,057.354,55
LORA
1993
1796
0.4822
0.6069
22.273.851.39
17,710,763.122
11,868,668999
DORA
2287
2023
0.4807
0.6068
24.553.455.616
19,189.150.515
12,490,471.653
LORAFA
2026
0,4299
0.6510
20,187.185,155
16,257,394,935
11.106,307,277
LOHA
2591
2341
0.4185
0.6570
23,886.561.280
19,247.870.71
13,446,820334
IA3
1922
1746
0.4124
0.6569
23,135.780,864
18.198,356.43
12,023,331,88
ADALORA
2209
1986
0,3904
0.6863
22,793,945,083
18,203,426, 160
12,361 3399,900
LOKR
2352
2152
0,3753
0.6877
23,565,697 .044
18,987,698,00 094
13,173.683,073
P TUNING
1918
1707
0.3707
0.6740
20,937, 965,568
17,215,688,54)
11,867, 101,,93
VBLORA
2210
1962
0,3700
0.7143
17,635,223,997
11.735.344,666
VERA
2025
1820
0,3685
0.6927
21.596,471.290
17.291.123.097
11,489,715.311
BOFT
11,114
8292
0,3647
0.7268
24,4276626.496
10,103,455,872
14.814,855.08
TA3
2005
1783
0,3450
0,7657
23,137,878,011
18,398,566, 154
12,023, 20023,227,433
TRAINABLE TOKENS
1814
1572
0,2881
0,7862
00.995,839,936
16,957,675,929
12,730,137,00 ,422
PROMPT TUNING
2715
2394
0,2525
0.7790
24,408, 752, 128
20.650,676,715
15,297,364,46
EDAPTION_PROMES
2261
0.2206
0.8317
22,410, 22,410,166,272
17,907,6648814
11,893,757,223
PREFIX. TUNING
1959
1662
0,1471
0,7887
20,912.799,744 744
16,945,051,074
11,766.684,083
FOURIERFT
2824
2422
0.1198
0,9979
23,681, 3040,384
19,054,869.87
13,111,221,
PROMPT PROMPT.TUNIING TUNING
2700
2380
0,0500
1,0655
24.379,392,00
20.669,781, 770
15,297,773,828
POURIERFT
2824
2424
0.0008
1,2480
23,653,777,44 ,088
19.017.267,99 937
13,104,129,58 129,350
TUNING
1870
1657
0.0000
1,2370
31,177,040.896
16,903,066.091
11,385,589,62
CONCLUSION
This work tackles the critical inefficiency of simultancous matrix updates in Low-Rank Adaptation (LoRA), which leads to slow convergence and suboptimal resource use. We propose MiSS as com- pelling solution-a new PEFT framework that updates decomposed weight shards using a single, shared matrix. This approach drastically reduces optimization complexity and resource demands. Comprehensive experiments validate that MiSS consistently outperforms existing methods in accu- racy, memory footprint, and computational speed, offering fundamentally more efficient pathway for adapting large models.
7 LIMITATIONS AND FUTURE WORK
As a pioneering approach, MiSS still leaves several aspects open for deeper exploration. We hope that future research will conduct broader and more in-depth studies to further refine PEFT techniques and identify the most effective strategies for large language models.
ACKNOWLEDGMENTS
This is the first paper of my life, and I am sincerely grateful to those who have supported me along the way.
I would like to express my heartfelt thanks to Huanqi Cao, Cormen, Yu Zhang, Jiaming Kong, and Houhao Wen. Although their names do not appear on this paper, this work would not have been possible without their guidance, discussions, and encouragement. I have benefited greatly from their insights and generosity. am also deeply thankful to everyone who has guided and inspired me throughout my journey. Even if not formally acknowledged as authors, their influence is reflected in this work.
I hope continue pursuing research that is both meaningful and interesting in the years come.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: :
org. 2008. 07732.
Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jen- nings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, al. Lora learns less and forgets less. arXiv preprini arXiv:2405.099: 000:2295..097,, 2024.
Kerim Büyükakküü.. Olora: Orthonormal low-rank adaptation of large language models. arXiv preprint arXiv:2400601775. 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel -osss,rtVVsss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Refichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.1::210.,4168,, 2021.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235, 2023.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354. 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.09683 2021.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //arxiv.org//////..... hbwpg//bb//// 22100.///3..... 06825.
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. Mora: High-rank updating abs/ 12130.
for parameter-efficient fine-tuning, 2024. URL https://axxi...
Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix adaptation, 2024. URL htt.ttss://rrxiiiiiii http://wwwwww.//////////////////
Md Kowsher, Ali O. Polat, Ehsan Mohammady Ardehaly, Mehrdad Salehi, Zia Ghiasi, Prasanth Mu- rali, and Chen Chen. Slicefine: The universal winning-slice hypothesis for pretrained networks, 2025. URL https:t////o arxiv. wwog/orrsss. /ttbb/////.....o 08513.
Shih- Yang Liu, Chien- Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.0933: 09353, 2024.
11

---

Published as a conference paper at ICLR 2026
Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources, 2024. URL https://aaaat org/abs 6306. 09782.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-art parameter-efficient fine-tuning methods. https: bttps://giset /github. com/ http/haagiingaaceepeeeeeee 2022.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arX1v:2404.02948, 2024.
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Haowen Hou, Przemyslaw Kazienko, et al. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint wwxiv:2244.05889 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint orXiv:2307.09288. 2023.
Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. arXiv preprint urXiv:2407.0500) 2024a.
Shaowen Wang. Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation, 2024b. URL httph:tt/////. http:///xxvvv..oo abs 24007. 05000.
Sheng Wang, Boyang Xue, Jiacheng Ye, Jiyue Jiang, Liheng Chen, Lingpeng Kong, and Chuan Wu. Prolora: Partial rotation empowers more parameter-efficient lora, 2024c. URL https:
//arxiv. .rrg abs 220224422..0 16902.
Sheng Wang, Liheng Chen, Pengan Chen, Jingwei Dong, Boyang Xue, Jiyue Jiang, Lingpeng Kong, and Chuan Wu. Mos: Unleashing parameter efficiency low-rank adaptation with mixture of shards, 2025. URL https:
orgg abs 22100 00938.
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efffcient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv 2023.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: /ttxxviivvv//////a///..///a.aavv org/ abs/ /205 09388.
Xinyu Yang, Jixuan Leng. Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, and Beidi Chen. ftt Efficient, scalable and generalizable Ilm fine-tuning by structured sparsity, 2024. URL https: / /aa.vvivvvv. org/ abs 2412.2222 06289.
Qingyu Yin, Xuzheng He, Xiang Zhuang, Yu Zhao, Jianhua Yao, Xiaoyu Shen, and Qiang Zhang. Stablemask: Refining causal masking decoder-only transformer. arXiv preprint Axii:2402.004779 2024.
12

---

Published as a conference paper at ICLR 2026
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint wrXi:2339.112884 2023.
Longteng Zhang. Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, 2023. URL https://arxi:: org/abs/2300 03303.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xucling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeimterpreterr Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. /4658, 2024.
A APPENDIX
A.1
ADDITIONAL EXPERIMENTS
LoRA
First 100
0.8
0.8
0.75
DoRA
0.7
0.50
0.6
0.6
Loss
0.5
0.4
0.4
0.2
0.000
0.005
0.010 0.015 0.020 0.025
16
25
33
41
50
Tokens(Billion)
Training Time (minutes)
(a) Loss-Token
(b) Loss-Time
Figure 5: Loss curves of LLaMA2-7B fine-tuned on MetaMathQA using LoRA and MiSS(a) Loss vs. tokens. (b) Loss vs. training time.
A.2 RWKV7
Table 9: We fine-tuned LLMs using MiSS and various LoRA variants, and evaluated performance on GSM8k, Math, HumanEval, and MT-Bench.
Model
Strategy Base
Trainable
GSM8K
Math
HumanEval
MT-Bench
44.35
LoRA
47.2M
55.64
RWKV7-3B
PISSA
47.2M
57.16
MISS
47.2M
58.22
B SETTINGS OF EXPERIMENTS
NLU We fine-tune the RoBERTa-base model on several datasets from the GLUE benchmark, in- cluding MNLI, SST-2, CoLA, QNLI, and MRPC. Performance is evaluated on the development set using accuracy as the primary metric. The experimental hyperparameter settings were aligned with those in the LoRA repository, but training was conducted using a single 4090 GPU. Each experiment conducted with 3 different random seeds, and the average performance reported. As shown in Table 4, MiSS demonstrates outstanding performance, particularly the CoLA dataset, where exhibits significantly faster convergence and superior data- Hata-fitting capabilities, far surpassing LoRA and PiSSA.
13

---

Published as a conference paper at ICLR 2026
Table 10: Hyperparameter settings for fine-tuning 1itmm2/7..Mitta///...vX6677....0 on NLG tasks
Hyperparameters
LoRA
DoRA
PiSSA
MiSS
Rank r
36
36
36
64
72
72
36
Dropout
0.0
Optimizer
AdamW
LR
2e-5
LR Scheduler
Cosine decay
Batch size
64
Warmup ratio
0.0
Epochs
1
Where
Q.K. C.O.UUp.Dwn,.ia
Table 11: Hyperparameter settings for fine-tuning llama2- 13B on NL. tasks
Hyperparameters
LoRA
DoRA
PISSA
MISS
Rank r
64
64
128
a
128
128
64
Dropout Optimizer
0.0
AdamW
LR Scheduler
2c-5
Cosine decay
Batch size
128
Warmup ratio
0.0
Epochs
I
Where
Q.K. V.O.UUp
To verify the generalizability of MiSS, we conducted more comprehensive experiments on LLM. we conducted 3 more task finetuning experiments on LLM: math and code. (1) Math: We trained our model on a 395k subset MetaMathQA (Yu et al., 2023), a dataset bootstrapped from other math instruction tuning datasets like GSM8K (Cobbe et al., 2021) and MATH (Yu et al., 2023), with higher complexity and diversity. (2) Code: We train our model on a 100k subset of CodeFeed- back (Zheng et al., 2024), high-quality code instruction dataset, removing explanations after code blocks. The model is tested on HumanEval (Chen et al., 2021) and Mbpp (Austin et al., 2021). ) The hyperparameter settings for this experiment were kept equal, while the train steps were adjusted according to the specific fine-tuning datasets used. It is worth noting that the attention-based archi- tectures employed by models such as LLaMA, Qwen, and Mistral do not use fully symmetric weight structures, which makes it impossible to achieve exact alignment of trainable parameters when com- paring MiSS with LoRA. To address this, we set the rank LoRA to 36 and the rank of MiSS to 64, ensuring that MiSS uses fewer parameters than LoRA to demonstrate its superiority. Each experiment is conducted with 2 different random seeds, and the average performance reported.
NLG
Vision Task on VTAB- 1K image classification using
C SETTINGS OF EXPERIMENTS No FREE LUNCH
14

---

Published as a conference paper at ICLR 2026
Table 12: Experimental Setup: Datasets and Hyperparameters
General Configuration
Parameter
Value
Random Seed (SEED)
43
Device (DEVICE)
CUDA (if available, else CPU)
Base Model Architecture (MLP)
Input Dimension
64
Hidden Dimension
Output Dimension
64
Synthetic Dataset Generation
Base Function
sin (2
Modified Function
sin (mxx
0.3 (3
Input Range
[-1,1]
Training Samples (N (N.TRAIN)
50
Validation Samples (N_ _V ALID)
100
Training Noise Std. Dev. (NOISE_STD)
0.05
Validation Noise Std. Dev.
0.0
Training Parameters
Base Model LR (BASE.LR) Adaptation (ADAPT LR)
0.001
0.001
Base Model Epochs (BASE EPOCHS)
250
Adaptation Epochs (ADAPT EEPCHSS)
100
Evaluation Interval INTERVAL)
10
Adapter-Specifie Ranks
LoRA Rank
VeRA Rank
64
MiSSRank
PISSA Rank
DoRA Rank
ProLoRA Rank
AdaLoRA Rank
MoS Rank
Note: Other adapter-speciffe hyperparameters (e.g.. LoRA scale, VeRA d_init_val, DoRA lora_alpha, ProLoRA unshared_rank.uu MoS shard_dim_ratio) primarily use their default values as defined respective adapter class implementations or are derived based on rank within benchmark functions. Refer to the provided Python code for their specific configurations during experiments.
15


Reviews:
[{"review_id": "eXJmtCi8op", "reviewer": "Reviewer_wCiN", "summary": "The paper introduces MiSS, a parameter-efficient fine-tuning (PEFT) method that builds upon LoRA to achieve a more favorable balance between adaptability and efficiency compared to prior approaches. The proposed method is comprehensively evaluated across a diverse set of tasks, including natural language understanding and natural language generation, and is empirically compared against several LoRA variants such as DoRA and PiSSA.", "questions": "My concerns are presented in the \"Weaknesses\" section and I have no further question.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The proposed method is remarkably simple and easy to implement, yet it demonstrates strong practical effectiveness. In several experimental settings, the method achieves notable improvements over LoRA. For instance, on the Mistral-7B model, MiSS outperforms LoRA by approximately 15%, highlighting its potential as a competitive and efficient alternative for parameter-efficient fine-tuning.", "weaknesses": "Firstly, although the paper states that MiSS is motivated by theoretical analysis, the practical method itself is presented without clear theoretical justification or development. The coherence and clarity of the paper would be greatly improved by adding a dedicated subsection in Section 4 that discusses the theoretical motivations behind the architectural design, which appears to be a novel choice aimed at ensuring the low-rank condition.\n\nSecondly, since the paper seeks to propose a practical parameter-efficient fine-tuning (PEFT) variant of LoRA, it would be valuable to include comparisons with a broader set of LoRA variants, such as AdaLoRA and VeRA. Although the manuscript mentions in lines 172–177 and the table in lines 58–67 that PiSSA, VeRA, DoRA, MoRA, PROLORA, and MoS are considered, only DoRA and PiSSA are actually included in the main experiments. In particular, comparison with VeRA, which is known for its strong parameter efficiency, would substantially strengthen the empirical section.\n\n**Other minor issues:**\n\n+ It is recommended to include an average performance metric to better assess the overall results relative to the baselines. While MiSS surpasses some baselines in certain settings, it underperforms on others.\n\n+ The title of Section 3 should likely be “No Free Lunch”—please verify and correct if necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eSg3lMIAaG", "reviewer": "Reviewer_ETKa", "summary": "In this paper, the authors aim to address the slow convergence issues in Low-Rank Adaptation (LoRA), a well-known parameter-efficient method for fine-tuning large language models (LLMs). In respone, they propose Matrix Shard Sharing (MiSS), a method that shards the original weight matrix and updates by sharing a single trainable matrix $D$ initialized to zero. This design reduces optimization complexity while maintaining performance. To further enhance computational efficiency, low memory usage, and scalable serving, they introduce MiSS. Theoretical analyses demonstrate MiSS's advantages in gradient flow and convergence, while empirical evaluations on benchmarks show strong results. The work also provides a comprehensive comparison of PEFT methods across memory, initialization time, and efficiency dimensions, mapping a Pareto frontier to highlight MiSS's balanced trade-offs.", "questions": "1. Could MiSS integrate with Mixture-of-Experts architectures for further efficiency gains?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Quality: MiSS effectively addresses LoRA's limitations by improving convergence without sacrificing efficiency, as supported by theoretical insights and Pareto analysis, making it a practical advancement in PEFT.\n\n2. Soundness: The paper includes detailed comparisons with variants like PiSSA and LoRA-GA, covering multiple dimensions (performance, memory, compute), which strengthens the claims.", "weaknesses": "1. Reliance on zero-initialized $D$ may limit adaptability in certain scenarios, potentially requiring further tuning.\n\n2. Evaluations are primarily on language tasks; broader domains (e.g., vision or multimodal) are not explored.\n\n3. The Pareto frontier mapping is insightful but could be more granular, e.g., with statistical significance tests.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wlPHkqXOAC", "reviewer": "Reviewer_Xjru", "summary": "The paper proposes Matrix Shard Sharing (MiSS) and its variant MiSSe to address the slow convergence and trade-offs among performance, memory, and computational efficiency in Low-Rank Adaptation (LoRA). MiSS shards the weight matrix and updates by sharing a single trainable matrix D, while MiSSe offers enhanced efficiency and scalability. The analysis shows that MiSS achieves a favorable balance across performance, memory, and efficiency.", "questions": "In Table 3, MiSS has almost similar number of parameters compared to the other LoRA variants. \n\nWhy does MiSS have larger number of parameters for Llama2-13B? Is it due to r or k (as expressed in Table 5) or for another reason?\n\nCould you please provide an ablation of using different initialization strategies to initialize D?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The proposed method is very pragmatic and practical.\n\nThe proposed method was examined in comparison with different LoRA variations in several benchmarks.", "weaknesses": "Notation should be revised and redundancy in some terms should be fixed.\n\nThe paper states that: Through theoretical analyses and empirical results, our method reduces optimization complexity while maintaining strong performance, striking a favorable balance between performance, memory, and efficiency.\n\nThat is, one of the main claims is the theoretical analysis of the proposed methods. However, this is not well explored in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0qOT4ik2IL", "reviewer": "Reviewer_iNdt", "summary": "This paper introduces MiSS (Matrix Shard Sharing), a novel parameter-efficient fine-tuning (PEFT) method designed to address the trade-off between performance, memory, and computational efficiency in LoRA-based approaches. The core idea is to simplify the weight update mechanism by training a single shared matrix D, which is expanded to form the weight update matrix, thereby reducing optimization complexity. The authors support their claims with a theoretical analysis of gradient norms, controlled experiments on a synthetic task, and extensive benchmarks on NLU and NLG tasks, demonstrating that MiSS achieves a favorable balance across key metrics.", "questions": "1. Can the authors clarify whether MiSSe is an exact mathematical equivalent of the MiSS formulation in Section 4.1? If not, it would be more accurate to present MiSSe as a distinct, more efficient variant inspired by the same shard-sharing principle, rather than a direct equivalent.\n\n2. Beyond simplifying optimization, is there a deeper theoretical or empirical justification for choosing the row-repetition structure in expand(D)? How does this structure relate to the properties of weight update matrices observed in full fine-tuning?\n\n3. Given that VeRA utilizes a similar parameter-sharing strategy for efficiency, could the authors provide a more direct comparison against VeRA on the main LLM benchmarks, discussing both conceptual differences and empirical trade-offs?\n\n4. To ensure a fair comparison, could the authors conduct an experiment where the trainable parameter counts of MiSS and LoRA are matched as precisely as possible?\n\n5. Could the authors provide convergence plots (loss vs. training time/tokens) for the main LLM experiments (e.g., on Llama2-7B) that include PiSSA and DoRA?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposal to replace LoRA's two-matrix (B and A) update with a single trainable matrix D is a novel simplification. The motivation, rooted in reducing optimization complexity and its connection to the initial gradient norm, provides a clear and compelling rationale for the proposed architecture.\n\n2. The paper presents a strong empirical evaluation, combining a controlled \"No Free Launch\" experiment to isolate variables with large-scale benchmarks across multiple models (e.g., Llama2, Mistral, Qwen3) and tasks. The inclusion of Pareto frontier analysis further strengthens the claims about achieving a better trade-off between performance and efficiency.", "weaknesses": "1. The paper introduces MiSS in Section 4.1, which partitions the output dimension d, and then presents MiSSe in Section 4.2 as a \"mathematically equivalent\" version that partitions the input dimension k. These two operations appear fundamentally different and are unlikely to be mathematically equivalent. This discrepancy undermines the clarity of the proposed method.\n\n2. The paper motivates MiSS by simplifying optimization to a single matrix. However, the specific choice of the expand(D) structure, repeating rows to form shards, feels somewhat arbitrary. The paper lacks a deep analysis of why this particular low-rank construction is superior to other potential single-matrix structures.\n\n3. The paper includes VeRA in the initial controlled experiment but does not provide a direct comparison in the main LLM benchmark tables. VeRA also employs shared low-rank matrices for efficiency, making it a critical baseline. A more detailed comparison is needed to properly position MiSS.\n\n4. In Section 5.2, the authors state they adjust ranks to give MiSS fewer parameters than LoRA. However, in Table 3 (Llama2-7B), the parameter counts are very close (MiSS: 87.0M vs. LoRA: 89.9M). The observed performance gains might be influenced by this slight difference in parameter budget or distribution rather than solely the architectural advantage.\n\n5. Figure 4 shows faster initial convergence for MiSS against LoRA, but this analysis is limited. The main experiments on LLMs lack convergence plots comparing MiSS with other key baselines like PiSSA and DoRA. This information is crucial for substantiating the claims about training efficiency.\n\n6. The pseudocode in Figure 3 is labeled MiSS but appears to implement a simplified version of MiSSe. The line y = result + x @ self.D.expand(...) is dimensionally inconsistent and does not clearly match the formulas provided in the text, causing confusion for implementation.\n\n7. Figure 5 provides a valuable visualization of the performance-efficiency trade-off, but the axes are not labeled. It is unclear which specific metrics (e.g., accuracy, memory usage, time) are being plotted, which reduces the figure's interpretability.\n\n8. A footnote suggests that MiSS is \"largely orthogonal\" to methods like LoRA+, which use different learning rates for matrices A and B.\n\n9. The paper states that the D matrix is initialized to zero, which is a standard practice to preserve the pre-trained model's initial state. However, it does not explore or ablate other initialization strategies, such as those inspired by SVD in PiSSA, which are designed to accelerate convergence.\n\n10. In Section 5, the document refers to NLG evaluations, but the footnote specifies that the Math dataset was evaluated with a 5-shot prompt. This suggests an in-context learning evaluation rather than a fine-tuning evaluation for that specific task, which could be misleading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)