Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation
Authors: Zhan Zhuang
Xiequn Wang
Wei Li
Yulong Zhang
Qiushi Huang
Shuhao Chen
Xuehao Wang
Yanbin Wei
Yuhe Nie
Kede Ma
Yu Zhang
Ying Wei
Venue: ICML 2025 poster (2025)
Keywords: Parameter-efficient fine-tuning
linear mode connectivity
low-rank adaptation
model merging
Primary area: deep_learning->everything_else
TL;DR: CoTo: A progressive adapter dropping strategy that boosts LoRA with better generalization, efficient merging/pruning, and faster training.

Abstract: Low-rank adaptation (LoRA) has emerged as a leading parameter-efficient fine-tuning technique for adapting large foundation models, yet it often locks adapters into suboptimal minima near their initialization. This hampers model generalization and limits downstream operators such as adapter merging and pruning. Here, we propose CoTo, a progressive training strategy that gradually increases adapters' activation probability over the course of fine-tuning. By stochastically deactivating adapters, CoTo encourages more balanced optimization and broader exploration of the loss landscape. We provide a theoretical analysis showing that CoTo promotes layer-wise dropout stability and linear mode connectivity, and we adopt a cooperative-game approach to quantify each adapter's marginal contribution. Extensive experiments demonstrate that CoTo consistently boosts single-task performance, enhances multi-task merging accuracy, improves pruning robustness, and reduces training overhead, all while remaining compatible with diverse LoRA variants. Code is available at https://github.com/zwebzone/coto.

Parameter-efficient fine-tuning (PEFT) has become the dom- inant paradigm for adapting large foundation models (Rad- ford et al., 2021; Rombach al., 2022; Meta, 2024) to Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation
Zhan Zhuang 12 Xiequn Wang' Wei Li Yulong Zhang Qiushi Huang 14 Shuhao Chen
Xuehao Wang'
Yanbin Wei 15 Yuhe Nie
Kede Ma Yu Zhang'
Ying Wei
Abstract
Fixee
Trainable
Disctivcced
Low-rank adaptation (LoRA) has emerged as a leading parameter-efficien fine-tuning technique for adapting large foundation models, yet of- ten locks adapters into suboptimal minima near their initialization. This hampers model gener- alization and limits downstream operators such as adapter merging and pruning. Here, we pro- pose CoTo', a progressive training strategy that gradually increases adapters' activation probabil- ity over the course of fine-tuning. By stochas- tically deactivating adapters, CoTo encourages more balanced optimization and broader explo- ration of the loss landscape. We provide the- oretical analysis showing that CoTo promotes wise dropout stability and linear mode con- nectivity, and we adopt a cooperative-gane proach to quantify each adapter's marginal con- tribution. Extensive experiments demonstrate that CoTo consistently boosts single-task perfor- mance, enhances multi-task merging accuracy, improves pruning robustness, and reduces train- ing overhead, all while remaining compatible with diverse LoRA variants. Code is available at https//github..omm/zeeeeooooooon
Activation Probability
Luyer
i
AWW
Layer
AW;
Layer 2
AWW
Layer
AWW
Training Step
On
20
3174
T
Figure 1. Illustration of the CoTo progressive activation schedule for LoRA adapters. For the first 75% of training (i.e., t < 37/4), cach adapter is stochastically deactivated (shown in gray), where the activation probability p(() increases linearly from 0 to as training progresses. In the final 25% of training, p(1) 1, and all adapters remain active, reducing to full fine-tuning.
downstream tasks. By introducing small number of train- able parameters, such as prompts (Lester et al., 2021), adapters (Houlsby et al., 2019), or low-rank adaptation (LoRA) modules (Hu et al., 2022), PEFT methods achieve rapid convergence and minimal storage overhead compared to full fine-tuning. Among these, LoRA has emerged as particularly effective approach, reparameterizing weight updates as low-rank matrices?.
ap-
Despite its success, vanilla LoRA often converges to sub- optimal minima near their initialization, due to the "lazy" dynamics of standard gradient- optimization (Du et al., 2018; Chizat et al., 2019). Moreover, empirical studies show pronounced layer-wise imbalance (Dauphin et al., 2014); adapters in higher layers receive the bulk of the gradient signal and dominate task performance, while those in lower layers remain largely under-utilized (Zhang et al., 2023b; Gao et al., 2024a). This uneven optimization not only re- stricts single-task generalization but also hampers down- stream operations, such as adapter merging (Zhao al., 2024b) and pruning (Han al., 2015; Li et al., 2017).
1. Introduction
SSuthern University Science and Technology, Shenzhen, China "City University of Hong Kong, Hong Kong SAR, China 'Zhejiang University, Hangzhou, China "University of Surrey, Sur- rey, UK 'Hong Kong University of Science and Technology, Hong Kong SAR, China 'New York University, New York, USA. Cor- respondence to: Kede Ma <kede ma @ www.t..ee.....s.......... Yu Zhang <yu.zhang.us ust @ @mmm..com>. Ying Wei <ying wei zju.edu.en>>.
To mitigate these issues, we propose CoTo, simple progres- sive training strategy that gradually increases each adapter's activation probability during fine-tuning. Early in training,
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Throughout this paper, we also call LoRA's trainable parame- ters adapters. Specifically, each adapter corresponds all LoRA parameters within a single Transformer layer.
'This acronym nods the Beatles' classic song 'Come Together' bhut not right now.

---

Co A Progressive Strategy to Boost Low-Rank Adaptation
CoTo stochastically deactivates random subset of adapters, forcing the model to distribute gradient updates more evenly, and then linearly raises the activation probability until all adapters participate fully. This curriculum- like scheme en- courages broader exploration of the loss landscape, yields layer-wise dropout stability, and promotes linear mode con- nectivity (LMC) between independently trained solutions.
greater stability, and FourierFT (Gao et al., 2024b), which leverages Fourier transforms represent weight updates in the frequency domain. In parallel, adaptive rank schemes such as AdaLoRA (Zhang et al., 2023b), ALoRA (Liu et al., 2024b), and LoRA-drop (Zhou et al., 2024) automatically adjust the rank per layer. These variants underscore the flexibility of the low-rank paradigm but also highlight the persistent challenge of ensuring balanced, layer-wise utiliza- tion of adapters during optimization.
We provide a theoretical analysis showing that Co min- imizes an upper bound network losses and, via cooperative-game perspective, quantifies each adapter's 's marginal contribution using Shap- ley values (Shapley, 1953). Empirically, Co consis- tently boosts single-task generalization, enhances multi-task adapter merging accuracy, improves adapter pruning robust- ness, and reduces overall training cost. Crucially, it requires no architectural changes, and integrates seamlessly with existing LoRA variants and advanced update schemes.
weighted ensemble of sub-
Beyond computational innovations, several studies have focused on improving the initialization and optimization dy- namics of LoRA. PiSSA (Meng et al., 2024) uses truncated singular value decomposition of the pre-trained weights to initialize LoRA matrices. LoRA-GA (Wang et al., 2024b) aligns the LoRA initialization with gradient-based approxi- mations of full fine-tuning: while rsLoRA (Kalajdzievski, 2023) adjusts scaling factors to stabilize early training. On the optimization side, LoRA+ (Hayou et al., 2024) employs distinct learning rates for the two low-rank matrices. Simi- larly, LoRA-Pro (Wang et al., 2024c) modifies the gradient updates to more closely emulate the behavior of full fine- tuning. While these methods yield improvements in con- vergence speed final performance, they do not explicitly address the problem of layer-wist imbalance.
2. Related Work
Garameter-Effiiient Fine- Tuning The rapid growth of foundation models has spurred extensive research into PEFT techniques, aiming adapt large pre-trained networks to downstream tasks without incurring the computational and storage costs of full fine-tuning. Early PEFT methods intro- duce modular components, such as adapters (Houlsby et al., 2019), prompts (Lester et al., 2021), or prefixes (Li & Liang, 2021), to capture task-specific knowledge while freezing the bulk of the pre-trained parameters. Houlsby et al. (2019) first proposed adapter layers (i..,, small bottleneck modules) inserted into Transformer blocks. Prompt-tuning (Lester et al., 2021) and prefix-tuning (Li & Liang, 2021) similarly leverage learnable tokens or continuous prefixes to steer the model toward a new task.
Model Merging. Combining task adapters to form a single set of parameters that performs well on multiple tasks relies on the property of LMC (Frankle et al., 2020; En- tezari et al., 2022; Zhou al., 2023), which posits that two independently fine-tuned solutions often lie in loss basins connected by low-loss linear path. In the LoRA context, LoraHub (Huang et al., 2023) first demonstrates that merg- ing low-rank adapters trained on separate language tasks can yield models with strong generalization to new tasks. Feder- ated learning extensions, such as FedIT (Zhang et al., 2024) and FLoRA (Wang et al., 2024d) apply LoRA merging and stacking across distributed clients, mitigating catastrophic forgetting and communication overhead. Recently, LoRA- LEGO (Zhao et al., 2024b) clusters semantically similar LoRA "units" within each layer before merging to reduce task interference, while ZipLoRA (Shah et al., 2025) focuses on disentangling style and content subspaces to enable com- positional generation in diffusion models. Despite these advances, effective multi-task merging remains challenging when adapters converge to layer-wise imbalanced minima.
LoRA (Hu et al., 2022), on the other hand, reframes fine- tuning as the problem of learning low-rank updates to each weight matrix. Instead of adding full-rank adapter layers, LoRA factorizes the weight update into two low-rank ma- trices and injects them into each Transformer layer. This decomposition dramatically reduces the number of train- able parameters while consistently delivering stronger per- formance than earlier PEFT methods. Subsequent work has proposed various LoRA extensions, with the goal of improving adaptation quality, reducing parameter count further, or aligning with full fine-tuning dynamies. For example, DoRA (Liu et al., 2024a) decomposes LoRA up- dates into magnitude and directional components to better approximate the full, high-dimensional updates, whereas HiRA (Huang et al., 2025) applics Hadamard products be- tween low-rank matrices and the original weights to enable high-rank adaptation without significantly increasing param- eter cost. Other notable variants include LoRA-FA (Zhang et al., 2023a), which freezes the projection-down weights for
Stochastic Regularization methods, originally developed to prevent overfitting, have been adapted to the LoRA setting to encourage robustness and exploration of the parameter space. Classical techniques like Dropout (Srivastava et al., 2014) and DropConnect (Wan et al., 2013) randomly zero out elements or connections during training. Stochastic Depth (Huang et al., 2016) and LayerDrop (Fan et al., 2020) skip entire layers with a fixed or lincarly decaying proba-
2

---

CoTo: A Progressive Strategy to Boost Low-Rank Adaptation
bility, Within LoRA, entry-wise column-wise dropout has been explored (Wang al., 2024a; Lin al., 2024) to regularize low-rank matrices, but these approaches do not ac- count for the sequential, layer-wise computation of adapters. Consequently, they may fail to correct the disproportionate updates received by hiyher-layer adapters. In contrast, the proposed CoTo introduces a progressive training strategy that dynamically increases the activation probability of each adapter early in training. This curriculum-like schedule bal- ances gradient flow across all layers, fosters exploration of diverse subnetworks, and improves downstream operations.
1.00
w:(p)
Wylp1
Ws(p)
0.75
welp)
Wylp)
welp)
0.50
026
0.0
0.2
0.4
0.8
Figure 2. Visualization of the weight function my my,(p) in Eq. (6).
where is an all-ones matrix of the same size as W is and @ denotes the Hadamard product. The training objective is to minimize the expected loss:
3. Proposed Method: CoTo
In this section, we introduce the proposed CoTo and present two complementary perspectives elucidate its behavior.
min Es (). L.W; +
(4)
3.1. Preliminaries
where 8 = [ō,,..., .T.....,....... T E {0, 1)2, y is the target label, and is typically cross- erosspyntropy loss for classification or mean squared error for regression.
Let the parameters of a pre-trained foundation model be Oo [W:)2.. where W; _R= is the weight matrix of layer i. For an input Zu, the model 9 h2 ....o h2 hn computes sequence of hidden features: æ; h, (x;-1, W;) for i {1, L., and produces the final output with the prediction head g: ÿ - (..r). In LoRA, we freeze each base weight W, and introduce a low-rank update A W, Concretely, LoRA factorizes each update A WW; aBA, where
3.3. Training Dynamics
Curriculum of Subnetworks. Early in training, when the probability p of activating adapters is low, only a few adapters participate, forcing gradient updates to spread across layers and preventing higher-layer adapters from dominating the loss signal; as p increases, more adapters are gradually engaged, expanding the space in which the model can fine-tune. This stochastic deactivation also coun- ters the "lazy" regime-wher gradients tend to stay near initialization-byy encouraging exploration of a broader pa- rameter region. By the time all adapters are active, the model has already diversified its search and less likely to converge poor, layer-imbalancee minima.
gx B RRxr, and & min(m, n) controls the rank. The scaling factor a adjusts the magnitude of the update.
3.2. Training Strategy
CoTo, illustrated in Figure introduces simple, pro- gressive schedule for stochastically deactivating adapters during the early stages of fine-tuning, and then gradually "turning them on" so that, by the final stages, all adapters participate fully. Specifically, for each layer i, we draw an activation indicator:
Computational Savings. Whenever 5, 0, adapter i is skipped entirely-no matrix multiplicattins involving A, or B, are performed. Thus, CoTo reduces both forward and backward computation in the early training stages.
Bernoull(pp()).
(L)
3.4. Progressive Optimization Perspective
where p(t) E [0, 1] is a dime-dependent probability that increases linearly from 0 to 1 over the first 75% of train- ing steps, and remains equal to 1 for the remaining 25%. Denoting the total number of training steps by T, step t e e {1...., T}, set
In this subsection, we view Co'T as training weighted ensemble of partial LoRA configurations (i.e., subnetworks that omit certain adapters). This perspective makes precise how CoTTo encourages both robustness to adapter dropout and improved connectivity between different minima.
4t
3T
t<
Specifically, denote by
p())
3T
4
3T
t
ú, = E [s (oo: (W; + 5,1 AW.
(5)
Accordingly, the model output is adjusted to
the expected model prediction over all subsets of adapters of size j, where ||ō|] = Σl_ 5, When j = L, all adapters are active and y recovers the standard LoRA model output.
ÿ = -= =: (W: 8,1 AW.)
(3)
3

---

CoTo: A Progressive Strategy to Boost Low-Rank Adaptation
Table 1. Average accuracy (%) 11 image classification tasks. The highest accuracy (%) is bolded, while the second highest is
underlined. CLIP results are copied from (Zanella & Ben Ayed, 2024). All adapters use rank of = 2 with ViT-B/16 as the backbone.
Method
Aircraft
Caltech
Cars
DTD
EuroSAT
Flowers
Food
ImageNet
Pets
SUN
UCF
Avg
CLIP
24.7
92.9
65.3
43.6
47.5
71.4
86.1
66.7
89.1
62.6
66.7
65.1
LoRA (CL.)2)
$3.89
96.25
85.12
72.00
92.05
97.78
85.15
73.49
93.27
76.75
86.72
82,95
LoRA-CoTo
55.69
96.26
86.04
72.68
92,97
98.12
85,48
73.53
93,42
76.85
87.20
83.48
DoRA (CM8:24)
56.21
96.38
86.67
72.60
92.13
98.09
85.04
73.54
93.59
76.54
87.15
83.45
DoRA-CoTo
57.35
96.51
86,98
72.83
93.45
98.25
86.31
73.62
94.06
76.90
86,99
83,93
HiRA (CLR22)
$7.62
96.35
87.22
73.38
92.51
98.06
86.72
73.76
94.41
76.92
86.81
83.98
HiRA-CoTo
57.85
96.65
87.40
73.71
93.46
98.71
86.91
73.85
94,46
77.36
87.38
84.34
At iteration 1, CoTo samples a random vector ò, where each ò, is Bernoulli(p(t)). Over this randomness, the probability that exactly j adapters are active is
marginal contribution of each adapter to overall perfor- mance, we identify precisely how CoTo encourages bal- anced layer-wise optimization.
index the set of I. adapters (one per player). For any subset R C define the value function:
Let S = {1...., ......,) I.}
wy (D((t) () p(t) (1 p(t)) 1-- .j = 0, -.....,4. (6)
(1)
((R) = Ex e (s a W; + +6 A..)..). (7)
as illustrated in Figure 2.
Theorem 3.1. Let fl., y) be convex loss function. Then for any fixed p E [0. 1].
where 6; = l E R and 0 otherwise. Under this interpre- tation, the Shapley value (Shapley, 1953) of adapter can be approximated efficiently using the multilinear extension approach (Owen, 1972):
L (D) e (ū,, v)
min Es (Ú. v)1 2 min (AW.)
Σ (AW.)
sP((v) - -f-(al/aaa a(p) B (R. (i)) ((
(R.)).
Consequently, the expected Co'to objective at step upper- bounds a binomially weighted sum of the subnetwork losses.
where R; indexes a random subset of adapters excluding adapter i, and (p()) captures the expected marginal con- tribution of adapter when selected with probability p. Therefore, to estimate (p(()), one may sample a few val- ues of p. draw random subsets R, compute the difference v (R, U (i))-v (R,) (again approximated by averages over finite set of samples in Eq. (7)), and average appropriately. By inspecting (v(() after CoTo training, we gain insights into each adapter's 's marginal contribution.
Theorem 3.1 follows directly from applying Jensen's 's in- equality to the convex loss E(: y). A detailed proof is given in Appendix D.
Because CoTo's training objective accounts for all possi- ble choices of active adapters (weighted by (P)),) the model is explicitly encouraged to perform well even if any subset of adapters is disabled. Prior work (Frankle et al., 2020; Adilova et al., 2024) shows that dropout stability often implies that independently trained solutions can be connected by a low-loss linear path. Intuitively, because CoTo trains adapters near-isolation (for low p) before gradually re-enabling them, each adapter's parameters learn solution "locally," reducing inter-adapter dependencies. Consequently, two CoTo-trained models with different ran- dom seeds tend to lie in loss valleys that are linearly con- nected. Empirical verification appears in Section 4.2.
4. Experiments
We evaluate CoTo's effectiveness through a series of experi- ments designed answer three key questions: 1) Can Co improve single-task generalization across diverse bench- marks? 2) Does Co'To facilitate LMC for more effective model merging? 3) Can CoTo enhance pruning robustness?
All experiments use three random seeds to ensure statisti- cal reliability, and implementation details are deferred to Appendix A.
3.5. Cooperative-Gant Perspective
An alternative way to understand CoTo is through the lens of a cooperative game: each adapter is treated as a "player" in a game whose "value function" " is the model performance when that subset of adapters active. By attributing the
4.1. Single-Task Generalization
Results on Vision Benchmarks. To assess CoTo's im- pact in the vision domain, we follow the CLIP-LoRA
4

---

Co A Progressive Strategy to Boost Low-Rank Adaptation
Table Average accuracy (%) 8 commonsense reasoning tasks (Hu et al., 2023) using LLaMA-2-78 and LLaMA-3 3----B backbones. All adapters use rank of = 32. Results without CoTo are copied from (Huang et al., 2025).
Model
Method
ARC-c
ARC-e
BoolQ
HellaS
OBQA
PIQA
SIQA
WinoCi
Avg
ChatGP
N/A
79.90
89.80
73.10
78.50
74.80
85,40
68.50
66.10
77.01
LoRA
64.70
79.80
69.80
83.60
81.00
79.90
79.50
82.60
77.61
LoRA-CoTo
69.58
85.33
71.48
91.15
82.10
82.89
78.94
83.54
80.63
DoRA
68.20
83.70
71.80
89.10
82.40
83.70
76.00
82.60
79.69
LLaMA-2-7B
DoRA-CoTo
69.51
85.08
72.25
90.82
81.33
83.10
79.50
83.43
80.64
HiRA
73.81
86.74
71,22
88.12
84.60
83.35
79.53
83.98
81,42
HiRA-CoTo
74.49
72.11
88.40
84.00
84.33
79.89
85.24
LoRA
71.20
84.20
70,80
91.70
79.00
85.20
79.90
84.30
80.79
LoRA-CoTo
79.35
90.81
75.02
94.77
85,20
88.39
80.55
86.08
85.02
DoRA
80.40
90.50
74.60
95.50
85.80
89.30
79.90
85.60
LLAMA-3-8B
DoRA-CoTo
79.38
91.50
75,40
95.98
86.00
88.52
81,12
86.00
85.49
HiRA
82.90
93.27
75,40
95.36
88.32
81,15
87.70
86.72
HiRA-CoTo
83.36
93.27
75.32
95.42
88.40
90.15
81,99
88.08
87.00
Table 3. Average accuracy (%) on mathematical reasoning tasks (Cobbe al., 2021) using the L.aaMA2-7B hackhone. All adapters use rank of 8. Results without CoTo are copied from (Wang al., 2024c).
81.0
Accuracy (%)
80.5
70
80.0
60
Method
Params (%)
w/o CoTo
w/ CoTo
50
79.5
LaRA
LoRA
0.296
42,08 + 0.04
55.85 + 0.74
LoRA-CoTo
40
DoRA
0.316
53,07 = 0.75
56.56 0.19
79.0
0.0
022
0.7
0.5
0.9
HiRA
0.296
54.51 # 0.59
56.68 + 0.09
Interpolation
PISSA
0.296
44.54 + 0.27
50.16 0.47
Figure 3. Linear interpolation accuracy commonsense reason- ing tasks (Hu al., 2023). CoTo's interpolation curve averaged across 8 tasks (orange) remains flatter and higher compared to vanilla LoRA (blue), demonstrating superior LMC.
isLoRA
0.296
45.62 0.10
56.99 0.66
LoRA+
0.296
52.11 + 0.62
54,36 + 0.43
LoRA-Pro 0.296
54.23 +: 0.79
57.16 0.38
setup (Zanella & Ben Ayed, 2024), fine-tuning the VIT-B/16 backbone 11 image classification datasets assembled by Zhou et al. (2022). Each dataset defines an independent few-shot task with 16 training images per class. We com- pare three LoRA vanillts-vanilla LoRA, DoRA (Liu et al., 2024a), and HiRA (Huang et al., 2025)-both both with and without CoTo Table reports average accuracies over three seeds. CoTo yields noticeable performance gains across all LoRA variants, demonstrating that progressive training leads to more balanced utilization of all adapters.
lazy convergence, especially as model capacity increases.
We further evaluate CoTo on mathematical reasoning tasks by fine-tuning LLaMA-2-7B on MetaMathQA (Yu al., 2024) and testing it GSM8K (Cobbe al., 2021). In this setting, we adjust the adapter rank to 8 and compare CoTo against PiSSA (Meng et al., 2024), rsLoRA (Kala- jdzievski, 2023), LoRA+ (Hayou et al., 2024), and LoRA- Pro (Wang et al., 2024c). Similar performance gains have been achieved, as shown Table
Results on Language Benchmarks. In the language do- main, we first evaluate CoTo on commonsense reasoning
4.2. Single-Task and Multi-Tash Model Merging
In this subsection, we assess the LMC property through two complementary experiments.
tasks using LLaMA backbones (Meta, 2023; 2024). Fol- lowing Wang al. (2024c), we fine-tune LLaMA-2-7B and LLaMA-3-88 on the Commonsense 17K suite (Hu al., 2023), which comprises 8 tasks. All adapter variants rank of 32. Table reports average accuracies, from which we observe consistent performance improvements across different backbones, LoRA variants, and task com- plexities. These results indicate that CoTo's progressive activation schedule helps mitigate layer- wise imbalance and
Single-Task Model Merging. To quantify how well in- dependently trained LoRA solutions be connected by linear path, we fine-tune two instances on the same task from different random seeds, and then linearly interpolate their adapter parameters for interpolation ratios X E [0, 1]. Fig- ure illustrates interpolation accuracies on commonsense reasoning tasks (Hu al., 2023): 0.5 (equal mix-
5

---

CoTo: A Progressive Strategy to Boost Low-Rank Adaptation
Aircraft
Caltech
Cars
DTD
EuroSAT
Flowers
985
30
96.0
10
87.5
0.0
Food
ImageNet
Pets
SUN
UCF
A Averuge
67.5
877
52
13
85.0
86
LAMAl
825
72
78
0.4
0.4
1.0
0.4
0.6
0.0
0.6
0.6
Figure 4. Linear interpolation accuracy 11 image classification tasks. CoTo (orange) consistently outperforms vanilla LoRA (blue). Adding alignment (denoted by !) improves both but preserves CoTo's margin.
ture), LoRA-Co'le maintains 79% accuracy, whereas vanilla LoRA drops to 39%. Across the entire interpolation trajec- tory, CoTo's curve remains substantially flatter and higher, indicating that independently CoTo-trained adapters lie in closer, low-loss basins. Analogous trends emerge in the image classification experiments (see Figure 4). Even after applying an additional "alignment" step: learning an invert- ible matrix P to minimize || - A where A and AW denote linear weight fusion and model ensem- ble, respectively, CoTo retains its advantages over vanilla LoRA². Analysis of A AW ellz and ||P||2 in Fig- 5 further confirms that CoTo's performance gains stem from balanced layer-wise optimization rather than mere post-hoc alignment.
|AW; AWell2
|Pl2
LoRA LoRA-CoTo
Without alignnent
With alignuent
LoRA
LoBA-CoTo
Figure 5. Analysis of the optimized alignment matrix P for LoRA and LoRA-CoTo!, Error hars denote standard deviations.
modest. Similar trends are observed for LLaMA-2-12 indicating that CoTTo encourages each adapter to converge to parameters that lie in closer, low-loss subspaces.
Multi-Task Model Merging. Building on CoTo's enhanced LMC, we next examine its impact on merging adapters trained on different tasks following the experimental setup and default configurations (Zhao et al., 2024b). First, consider multi-task merging for generative language understanding tasks (Wang et al., 2019). We test seven in-domain tasks and two out-of-domain tasks (Longpre et al., 2023) via the same prompt format (Wei et al., 2022). Both LLaMA-2-7B and LLaMA-2-13BB backbones are used. We employ three merging strategies: linear weight fusion (i.e., ,W linear model ensemble (i.e., AW.). )). and the LoRA- LEGO method proposed by Zhao et al. (2024b), which plicitly aligns and fuses parameter updates. As reported in Table 4, CoTTo-triined adapters yield markedly better merg- ing performance. On LLaMA-2-7B, linear weight fusion of CoTo-based adapters improves average accuracy from 47.17% 58.53 58.53% (+1 11111.1 1.36%), and LoRA-LEGO merging rises from 62.20 22.21% 1 to 67.19% (+4.98%). The ensemble approach also benefits some tasks, though gains more
We also evaluate CoTo's efficacy for merging adapters on six discriminative language understanding tasks (Wang et al., 2019) 1) using DeBERTa-v3 (He et al., 2023). Although merg- ing classifiers across distinct tasks is inherently more chal- lenging due to differences in feature pooling and output prediction, CoTootrained adapters still exhibit consistent im- provements across all three merging strategies (see Table 11 in the Appendix). These findings indicate that Co'o is com- patible with existing merging techniques and consistently enhances multi-tast LoRA merging across both generative and discriminative architectures.
Finally, we explore CoTo in the context of diffusion-based generative models. Using SDXL (Podell et al., 2024) as our backbone, we apply DreamBooth (Ruiz et al., 2023) to fine-tune separate style and object adapters-speciffceeee specifically, object LoRAs for two categories (i.e., cat and dog) and style LoRAs for artistic styles (i.e., watercolor and crayon). We then merge style and object adapters using ZipLoRA (Shah et al., 2025). Qualitative results in Figure 6 demonstrate that CoTo significantly reduces style and object forgetting: for instance, a crayon-style cat generated with LoRA-CoTo clearly preserves both the cat's 's identity and "crayon- ness,"" whereas vanilla LoRA often compromises one or the other. This qualitative evidence further attests to
'Linear weight fusion is computed by A , = (AB; + (1 (AAi +(1 [22). which preserves the rank, while linear model ensemble (Zhao et al.. 2024a) is computed by AW. AB,AN (1 Ag. To insert the learnable invertible matrix P. we reparameterize W B. (P- (2))
6

---

Co A Progressive Strategy to Boost Low-Rank Adaptation
Table 4. Average accuracy (%) multi-task merging for 9 generative language understanding tasks (Wang et al., 2019) using LLaMA-2 -2- 7B and LLaMA-2-13B backbones. "Fusion" and "Ensemble" correspond to the linear weight fusion and linear model ensemble that f and AW.. e, respectively. Results without CoTo copied from (Zhao et al., 2024b).
compute A y
In-Domain Task
Out-of-Domaad Task
Model
Method
Avg
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
SNLI
WNLI
Task-Specific LoRA
61.63
77.46
68.00
77.25
75.83
52.22
75.74
Fusion
54.42
36.09
68.00
44.41
51.72
48.15
42.99
31.64
47.14
47.17
Fusion-CoTo
57.31
47.39
61.75
62.60
71.89
71.11
60.80
36.79
57.14
58.53
LLaMA-2-7B
Ensemble
55.67
45.89
59.25
59.84
67.38
68.89
66.44
36.73
51.43
56.84
Ensemble-CoTo
57.21
45.68
47.75
61.39
68.59
69.26
60.57
35.24
64.29
56.66
LoRA-LEGO
55.48
55.73
66.00
62.29
71.07
71.85
73.22
51.36
52.86
62.21
LoRA-LEGO-COTo
53.94
64.35
72.25
72.71
78.51
71.48
75.75
58.59
57.14
67.19
Task Task-Specific LoRA
69.04
88.23
89.25
82.33
86.29
80.74
76.44
Fusion
45.48
46.32
67.75
46.68
47.50
62.96
46.78
42.42
42.86
49.86
Fusion-CoTo
64.52
57.82
73.75
66.10
78.53
75.93
75.52
42.28
67.14
66.84
LLaMA-2-13B
Ensemble
62.50
64.64 64
74.75
71.81
81.35
79.26
75.52
54.32
60.00
69.35
Ensemble-CoTo
63.75
60.54
71.75
67.82
76.38
77.78
75.75
46.79
62.86
67.05
LoRA-LEGO
59.42
65.40
75.50
72.29
82.51
78.52
75.98
58.54
64.29
70.27
LoRA-LEGO-CoTo
61.83
65.75
78.25
76.81
82.90
77.78
76.32
58.74
65.71
71.57
LokA
Training Set
LattA Cato
73.
Eaily LobtsGodee
Prompts: "A sbu cat/dog in the style"
Accuracy
07.5
22.5
LoRA
37.
LoRub-Gs Do
LelLA
Eaay Lowe
High
Pruning Structure
Sparsity
LoRA-CoTo
Figure 7. Average accuracy (%) on model pruning for the vi- sual texture classification task (Cimpoi et al., 2014). Left panel: Structured pruning applied to LoRA, LoRA-CoTo, and Early LoRA-CoTo under four pruning patterns: alternating layers (EveryOther), first 4 layers (Low), middle 4 layers (Middle), and last layers (High). "All" denotes unpruned models. Right panel: Unstructured pruning with varying sparsity.
HIRA
HIRA-CATO
structured and unstructured pruning experiments. We first examine layer-wise structured pruming on the visual texture classification task (Cimpoi al., 2014) by selectively re- moving adapters from different network layers. As shown in the left panel of Figure 7, we compare four configurations: 1) removing alternating layers (denoted by EveryOther), 2) pruning the first 4 layers (denoted by Low), 3) pruning middle 4 layers (denoted by Middle), and 4) pruning the last layers (denoted by High). The results demonstrate that CoTo-trained adapters maintain significantly better perfor- mance across all pruning patterns compared to vanilla LoRA. Notably, the "Farly LoRA-CoTo" checkpoint (sampled right after the first 25% of training) already shows strong pruning robustness, indicating that the benefits emerge early in the progressive training schedule. Complete results are detailed in Figure 12 of the Appendix.
Figure 6. Customized sample images generated by SDXL (Podell al., 2024) with and without CoTo. When merging style and ob- ject adapters via ZiRL (Shah et al., 2025), Co preserves both the object identity and artistic style more faithfully than vanilla LoRA. Each comparison the same seed.
CoTo's ability to learn adapters that merge more coherently across diverse vision and language tasks.
4.3. Model Pruning
The stochastic nature of CoTo naturally lends itself to im- proved pruning robustness, as adapters are trained to main- tain performance even when a random subset is deactivated. To systematically evaluate this property, we conduct both

---

Co A Progressive Strategy to Boost Low-Rank Adaptation
LoRA
LoRA-Co
Early LoRA-CoTo
Table 5. Mean Euclidean distance between LoRA adapter weights across four learning rates. Init. to Final: Distance from initial to final weights. Final to Final: Distance between final weights from different initialization. Note that the mean initial distances are 1.155 for independent random seeds and 0.02 for perturbed seeds.
Vision Text
10112
1
1011 1
101112
Figure 8. Approximated Shapley values of LoRA adapters by mul- tilinear extension (Owen, 1972).
Distance
5c-5
lc-4
5c-4
lc-3
Random Initialization
Init, vs. Final (w/o CoTo)
0.48
0.45
0.76
1.32
Accuracy
1.00
Init. Final (w/ CoTo)
0.61
0.79
1,25
1.64
0.58
Time
0.98
58
Final vs. Final (w/ CoTo)
Final vs, Final (w/o CoTo)
1.70
1.38
1.81
1.53
2.35
2.14
2.63
3.12
0.57
0.96
Arcuracy
0.56
Same Initialization with Minor Additive Uniform Noise
56
Final vs. Final (w/o CoTo)
0,05
0.07
0.59
1.71
0.55
0.92
Final Final (w/ CoTo)
0.04
0.06
0.21
0.52
0.54
0.90
54
18
CoTo-L CoTo HoTo
For fine-grained sparsity analysis, we evaluate unstructured pruning by zeroing out increasing percentages of adapter parameters. The right panel of Figure shows that the per- formance gap widens with increasing sparsity level, and at 50% sparsity, LoRA-CoTo achieves 10% higher accu- racy than vanilla LoRA. These findings highlight that CoTo enables more aggressive pruning while maintaining model utility even at high sparsity levels.
Figure 9. Ablation analysis Co Left panel: Impact of varying the proportion of training time allocated to the first (stochastic activation) phase on model accuracy (purple) and normalized train- ing time (orange). Right panel: Comparison of dropout strategies, including no dropout (Base)), nested dropout from lower layers (CoTo-L), uniform dropout (CoToo) and nested dropout from higher layers (C)To-H).
4.5. Ablation Studies
4.4. Further Analysis on DTD
To systematically evaluate the design choices of CoTo and its robustness to hyperparameter variations, we con- duet key ablation experiments on mathematical reasoning tasks (Cobbe et al., 2021). All experiments employ LoRA- Pro (Wang et al., 2024c) as the baseline.
Convergence Near Initialization. We empirically validate that LoRA adapters converge near their initialization, con- sistent with "lazy training" dynamics (Chizat et al., 2019). Using t-SNE visualization (Van der Maaten & Hinton, 2008) (see Figure 14 in the Appendix), we find that final adapter weights form tight clusters around their respective initial- ization points across five random seeds and four learning rates. This local convergence persists regardless of whether CoTo is applied. Nevertheless, CoTo yields slightly larger distances (see Table 5), suggesting broader exploration. Moreover, final weights from independent random seeds are closer under CoTo indicating more consistent convergence paths. When initialized from the same point with minor additive uniform noise, CoTTo-trained adapters converge to tighter clusters compared to vanilla LoRA, demonstrating robustness to initialization perturbations.
Training Phase Transition. We first investigate the impact of varying the proportion of training time allocated to the stochastic activation phase (see left panel of Figure 9). The z-axis represents the percentage of total training spent in the first phase (where p(t) < 1), with 0% corresponding to vanilla LoRA (i.e., without CoTo) and 100% represent- ing training exclusively with stochastic activation (i.e., p(t) never reaches 1). Our results demonstrate that a 75% first- phase proportion strikes good balance between task perfor- mance and training efficiency with secondary benefits like improved LMC.
Dropout Strategy. CoTo applies uniform dropout proba- bility across all layers. To assess layer-specific effect, we design two variants: 1) CoTo-L, where adapters in lower layers are deactivated first, i.e., at time I, only adapters in layers i in (for some threshold in determined by p(t)) re- main active, and 2) CoTo-H, where adapters in higher layers are deactivated first, that early in training only lower- layer adapters participate. The results reveal that CoTo and CoTo-H outperform CoTo-L, indicating that randomly deactivating or prioritizing deactivation of of-highllayer adapters is more effective. dest-dropping from lower lay- ers forces the model to rely prematurely on higher-layer
Adapter Contribution Analysis. To quantify layer-wise marginal utilization, we compute approximated Shapley values (Shapley, 1953) for each adapter via multilinear ex- tension (Owen, 1972) on the visual texture classification task (Cimpoi et al.. 2014) again for its representativeness and computational feasibility. As shown in Figure 8, vanilla LoRA exhibits skewed contributions, with 69% concen- trated in the highest 4/12 Transformer layers. In contrast, LoRA-CoTo and Early LoRA-CoTo achieve more balanced contributions (+8% and +3% deviations across layers), con- firming its efficacy in mitigating gradient imbalance.
8

---

Co A Progressive Strategy to Boost Low-Rank Adaptation
Table Average accuracy (%) on mathematical reasoning tasks (Cobbe et al., 2021) for LoRA- Pro across different adapter ranks, learning rates, and insertion modules.
Table 7. Wall-clock training times for LoRA, DoRA, and HiRA on mathematical reasoning tasks (Cobbe et al., 2021) using a single NVIDIA A6000 GPU.
LoRA Rank
8
32
128
LoRA
DoRA
HiRA
LoRA-Pro
54.23 # 0.79
55,14 # 1.73
56.48 + 0.23
w/o Co
7h 38min
19h 00min
11h 39min
+ CoTo
57.16 0.38
57.24 + 0.06
58.50
0.46
CoTo
05min
14h 30min
8h 50min
Learning Rate
5c-5
lo-4
2c-4
Speedup 7.20%
23.69%
24.21%
LoRA-Pro
55.70 d: 0.96
55.85 a: 0.74
40.91 à 1.09
+ CoTo
55.83 0.38
57.16 a 0.38
56.25 0.53
matrices are skipped entirely. Variants with larger adapter footprints (e-g., DoRA and HiRA) thus experience more pronounced computational savings. Importantly, these gains accrue early in training yet do not compromise- or even improve-final accuracy.
Insertion Module
Attention
Projection
Gating
LoRA-Pro
45.44 =: 0.40
49.08 +: 0.64
48, 40 +: 0.16
+ CoTo
52.41 = 0.56
54.06 + 0.80
51.96 + 0.28
Merging
Pruning
5. Conclusion and Discussion
70
Accuracy
70
We have introduced CoTo, a progressive training strategy for LoRA. By gradually increasing adapter activation probabil- ity during training, CoTo promotes more balanced optimiza- tion across all layers while encouraging broader exploration of the loss landscape. Our theoretical analysis showed that CoTo enhances layer-wise dropout stability and LMC, while the cooperative -aremattieeaeeeeeeeeee perspective provided quantitative in- sights into each adapter's marginal contribution. Extensive experiments across vision-language models, large language models, and diffusion models consistently validated the ef- fectiveness of Co'To
65
60
60
LoRA
LoRA- CoTo
55
5c-5
1o.4
5e-4
1e.3
5
5e-5
1o-4
5e-4
1e-3
5e-3
Learning Rate
Learning Rate
Figure 10. Average accuracy (%) on model merging and pruning for the visual texture classification task (Cimpoi et al., 2014). Left panel: Merging accuracy when 0.5. Right panel: Pruning accuracy when removing alternating layers (Le.. EveryOther).
While Co integrates seamlessly with diverse LoRA vari- ants, a promising direction is to identify the optimal combi- nation of existing LoRA "tricks." " For instance, one could explore jointly applying CoTo's 's progressive schedule with adaptive-rank schemes (like AdaLoRA (Zhang et al., 2023b) or ALoRA (Liu et al., 2024b)), weight-decomposel updates high-rank
adapters and undermines balanced optimization.
Hyperparameter Sensitivity. To demonstrate CoTo's com- patibility with diverse LoRA configurations, we fine-tune L.LMMA--7B using three adapter ranks (r 8, 32, 128), three learning rates (5c-5, Ie-4, 2c-4), and three choices of insertion modules from attention, projection, and gating layers, respectively. From Table 6, we find that LoRA-Pro- CoTo consistently outperforms LoRA-Pro in all settings.
(in DoRA (Liu et al., 2024a)), or Hadamard- adaptations (in HiRA (Huang et al., 2025)), to determine how these techniques interact and where synergies arise. Systematically evaluating such combinations poteniiaa via automated hyperparameter search over activation sched- ules, rank allocations, and initialization strategies- reveal configurations that maximize performance while min- imizing parameter count and compute. Moreover, extending Co'To to jointly optimize over multiple PEFT objectives (e.g., balancing dropout stability, quantization compatibility, and rank efficiency) could yield a unified framework that adapts to various resource constraints and task requirements.
Further, Figure 10 extends this analysis to merging and prun- ing on the visual texture classification task (Cimpoi et al., 2014) across five learning rates. CoTo-trained adapters yield higher merging accuracy and maintain stronger robustness under structured pruning. These trends underscore CoTo's generality: it benefits LoRA variants across a wide spectrum of hyperparameter settings.
Training Overhead Reduction. Because CoTo stochasti- cally deactivates adapters in early iterations, it reduces both forward and backward computation. We compare end-to- end fine-tuning times for LoRA, DoRA (Liu et al., 2024a), and HiRA (Huang et al., 2025), all under identical hard- ware and batch-size settings. From Table 7, we observe noticeable training overhead reduction. which arises be- cause, when an adapter inactive (i.e., 8, = 0), its low-rank
A Acknowledgement
This work was supported in part by the National Key Research and Development Program of China (No. 2022ZD0160300) and the National Natural Science Foun- dation of China (No. 62136005).

---

Co A Progressive Strategy to Boost Low-Rank Adaptation
Impact Statement
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. 2021.
This work aims to advance the field of machine learning. While it may carry various societal implications, none of which we feel must be highlighted here.
arXiv preprint
Dagan, I., Glickman, O., and Magnini, B. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop, pp. 177-190, 2005.
References
Adilova, L.,, Andriushchenko, M., Kamp, M., Fischer, A., and Jaggi, M. Layer-wise linear mode connectivity. In International Conference on Learning Representations, 2024.
Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Gan- guli, S., and Bengio, Y. Identifying and attacking the saddle point problem in high-dimensional non-convex op- timization. In Advances in Neural Information Processing Systems, 2014.
Bar Haim, R., Dagan, I., Dolan, B., Ferro, L.., Giampic- colo, D., Magnini, B., and Szpektor, I. The Second PAS- CAL recognisig textual entailment challenge. In Second PASCAL Challenges Workshop on Recognising Textual Entailment, 2006.
Deng, J., Dong. W. Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255 2009.
Bentivogli, L., Dagan, I., Dang, H. T., Giampiccolo, D., and Magnini, B. The Fifth PASCAL recognizing textual entailment challenge. In Text Analysis Conference, 2009.
Dolan, W. B. and Brockett, C. Automatically construct- ing a corpus of sentential paraphrases. In International Workshop on Paraphrasing, 2005.
Bisk, Y., Zellers, R., Le Bras, R., Gao, J., and Choi, Y. PIQA: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, pp. 7432-7439, 2020.
Du, S. S., Zhai, X., Poezos, B., and Singh, A. Gradient descent provably optimizes over-parameterizzd neural networks. arXiv preprint arXiv:1810.0205: 2018.
Bossard, L., Guillaumin, M., and Van Gool, L. Food-101- mining discriminative components with random forests. In European Conference on Computer Vision, pp. 446- 461, 2014.
Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. The role of permutation invariance in linear mode connectiv- ity of neural networks. In International Conference on Learning Representations, 2022.
Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. arXiv preprini arXiv: 053266 2015.
Fan, A., Grave, E., and Joulin, A. Reducing Transformer depth on demand with structured dropout. In Interna- tional Conference Learning Reprexentutions, 2020.
Chizat, L., Oyallon, E., and Bach, F. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, 2019.
Fei-Fei, L., Fergus. R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. Computer Vision and Image Understanding, 106(1):59- 70, 2007.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In IEEE Conference Computer Vision and Pattern Recognition, pp. 3606-3613. 2014.
Frankle. J., Dziugaite, G. K., Roy, D., and Carbin, M. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259- 3269, 2020.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 2924-2936, 2019.
Gao, C., Chen, K., Rao, J., Sun, B., Liu, R., Peng, D., Zhang, Y., Guo, X., Yang, J., and Subrahmanian, Higher layers need more LoRA experts. arXiv preprint arXiv:2402.08560 2024a.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? Try ARC, the A12 reasoning chal- lenge. arXiv preprint arXiv:
Gao, Z., Wang, Q. Chen, A., Liu, Z., Wu, B., Chen, L., and Li, J. Parameter-efficient fine-tuning with discrete Fourier transform. In International Conference on Ma- chine Learning, pp. 11884-14901, 2024b.
2018.
10

---

Co A Progressive Strategy to Boost Low-Rank Adaptation
Giampiccolo, D., Magnini, B., Dagan, and Dolan, B. The Third PASCAL recognizing textual entailment challenge. In ACL-PASCAL Workshop Textual Entailment and Paraphrasing, pp. 1-9, 2007.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3D object representations for fine-grained categorization. In IEEE International Conference Computer Vision Workshop, pp. 554-561, 2013.
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, 2015.
Lester, B., A1-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
Levesque, H. J., Davis, E.,, and Morgenstem, The Wino- grad schema challenge. In International Conference on Principles of Knowledge Representation and Reasoning, pp. 552 561, 2012.
Hayou, S., Ghosh, N., and Yu, B. LoRA+: Efficient low rank adaptation of large models. In International Conference on Machine Learning. pp. 17783-1787 17806, 2024.
He, P., Gao, J., and Chen, W. DeBERTaV3: Improving De- BERTa using ELECTRA-style pre-training with gradient- disentangled embedding sharing. In International Con- ference Learning Representations, 2023.
Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P. Pruning filters for efficient ConvNets. In International Conference on Learning Representations, 2017.
Helber, P., Bischke, B., Dengel, A., and Borth, D. EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2222 2019.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. In Annual Meeting of the Asso- ciation for Computational Linguisties, pp. 4582-4597, 2021.
Lin, Y., Ma, X., Chu, X., Jin, Y., Yang. Z., Wang, Y., and Mei, H. LoRA dropout as a sparsity regularizer for over- fitting control. arXiv preprint S0.1::2404.09111 2024.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q.. Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, pp. 2790-2799, 2019.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., and Chen, M.-H. DoRA: Weight- decomposed low-rank adaptation. In International Con- ference Machine Learning, pp. 32100-3303300 -21221, 2024a.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Liu, Z., Lyn, J., Zhu, W., and Tian, X. ALoRA: Allocat- ing low-rank adaptation for fine-tuning large language models. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 622 641, 2024b.
Hu, Z.,, Wang, L., Lan, Y., Xu, W., Lim, E.PP., Bing, L.,, Xu, X., Poria, S., and Lee, R. LLM-Adapters: An adapter family for parameter-efficient fine-tuning of large lan- guage models. In Conference on Empirical Methods in Natural Language Processing, pp. 5254 1276, 2023.
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effec- tive instruction tuning. In International Conference on Machine Learning, pp. 22631-22648, 2023.
Huang, C., Liu, Q., Lin, B. Y., Pang, T., Du, C., and Lin, M. LoraHub: Efficient cross-task generalization via dynamic LoRA composition. arXiv preprint arXiv:2307.132000 2023.
Maji, S., Rahtu, E.,, Kannala, J., Blaschko, M., and Vedaldi, A. Pine-grained visual classification of aircraft. arXiv preprint 330.5v:336..1.... 2013.
Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. Deep networks with stochastic depth. In European Conference on Computer Vision, pp.
2016.
Meng. F., Wang, Z., and Zhang, M. PiSSA: Principal sin- gular values and singular vectors adaptation of large lan- guage models. arXiv preprint arXiv:2404.02948 2024.
Huang, Q., Ko, T., Zhuang, Z., Tang, L., and Zhang, Y. HiRA: serameter-efficien Hadamard high-rank adapta- tion for large language models. In International Confer- ence on Learning Representations, 2025.
Meta. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wwXiv:2307.09288. 2023.
Kalajdzievski, D. A rank stabilization scaling factor for fine- tuning with LoRA. arXiv preprint arXiv:2312.03333,, 2023.
Meta. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

---

Co A Progressive Strategy to Boost Low-Rank Adaptation
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? A new dataset for open book question answering. In Conference Em- pirical Methods in Natural Language Processing, pp. 2381-2391 2018.
Shapley, L. S. A value for N-person games. Contributions the Theory of Games, (28):307-317, 1953.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642 2013.
Nilsback, M.-E. and Zisserman, A. Automated flower classi- fication over a large number of classes. In Indian Confer- ence on Computer Vision, Graphics & Image Processing, pp. 722-722--9 2008.
Soomro, K,, Zamir, A. R., and Shah, M. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 121.00402 2012.
Owen, G. Multilinear extensions of games. Management Science, 18(5-part (15-part-2):644-73 79, 1972.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov. R. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-199) 2014.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498-3505, 2012.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dock- horn, T., Müller, J., Penna, J., and Rombach, R. SDXL: Improving latent diffusion models for high-resolution im- age synthesis. In International Conference on Learning Representations, 2024.
Van der Maaten, L. and Hinton, G. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11,, 2008.
Wan, I.,, Zeiler, M., Zhang, S., Le Cun, Y., and Fergus, Regularization of neural networks using DropConnect. In International Conference Machine Learning, pp. 1058-1066, 2013.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763, 2021.
Wang. A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2019.
Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, SQuAD: 100,000+ questions for machine comprehension of text. In Conference Empirical Methods in Natural Lan- guage Processing, pp. 2383-2392,, 2016.
Wang. S., Chen, L., Jiang. J., Xue, B., Kong, L., and Wu, LoRA meets dropout under unified framework. arXiv preprint arXiv:2403.00817 2024a.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Wang, S., Yu, L., and Li, J. LoRA-GA: Low-rank adaptation with gradient approximation. In Advances in Neural Information Processing Systems, 2024b.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. DreamBooth: Fine tuning text-to- image diffusion models for subject-driven generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Wang, Z., Liang, J., He, R., Wang. Z., and Tan, T. LoRA- Pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407. 18242, 2024c.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. WinoGrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
Wang. Z., Shen, Z., He, Y., Sun, G., Wang, H., Lyu, L., and Li, A. FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprini 24Xiv:2409.05976 2024d.
Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y. SocialIQA: Commonsense reasoning about social interactions. arXiv preprint
Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. In International Conference on Learning Representations, 2022.
2019.
Shah, V., Ruiz, N., Cole, F., Lu, E., Lazebnik, S., Li, Y., and Jampani, V. ZipLoRA: Any subject in any style by effectively merging LoRAs. In European Conference on Computer Vision, pp. 422 422.438,20 2025.
Williams, A., Nangia, N., and Bowman, S. R. A broad- coverage challenge corpus for sentence understanding through inference. In Conference of the North American
12

---

CoTo: A Progressive Strategy to Boost Low-Rank Adaptation
Chapler of the Association for Computational Linguistics: Human Language Technologies, pp- 1112-1122, 2018.
Zhou, Z.,, Yang, Y., Yang, X., Yan, J., and Hu, W. Going beyond linear mode connectivity: The layerwise linear feature connectivity. In Advances in Neural Information Processing Systems, pp. 60853-60877, 2023.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. SUN Database: Large-scale scene recognition from abbey zoo. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 3485-3492, 2010.
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J., Li, Z.., Weller, A., and Liu, W. MetaMath: Bootstrap your own mathematical questions for large language mod- els. In International Conference Learning Representa- tions, 2024.
Zanella, M. and Ben Ayed, I. Low-rank few-shot adaptation of vision-lange lainn-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1593-
1603, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? arXiv preprint
2019.
Zhang, J., Vahidian, S., Kuo, M., Li, C.,, Zhang, R., Yu, T., Wang, G., and Chen, Y. Towards building the Federated GPT: Federated instruction tuning. International Con- ference on Acoustics, Speech, and Signal Processing. pp. 2024.
Zhang. L., Zhang, L.,, Shi, S., Chu, X., and Li, B. LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint 2rXiv:2308.03300 2023a.
Zhang, Q. Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. Adaptive budget allocation for parameter- effiicient fine-tuning. International Confer- ence on Learning Representations, 2023b.
Zhao, Z., Gan, L., Wang, G., Zhou, W., Yang, H., Kuang, K., and Wu, F. LoraRetriever: Input-aware LoRA retrieval and composition for mixed tasks in the wild. In Find- ings of the Association for Computational Linguistics, pp. 14447-4462, 2024a.
Zhao, Z., Shen, T., Zhu, D., Z., Su, J., Wang. X., Kuang, K., and Wu, F. Merging LoRAs like playing LEGO: Pushing the modularity of LoRA extremes through rank-wise clustering. arXiv preprint 4rXiv:2409.161 2024b.
Zhou, H., Lu, X., Xu, W., Zhu, C., and Zhao, T. LoRA- drop: Efficient LoRA parameter pruning based on output evaluation. arXiv preprint arXiv:2402.07721 2024.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to
prompt for vision-language models. International Jour- nal of Computer Vision, 130(9):3377-234, 2022.
13

---

CoTo: A Progressive Strategy to Boost Low-Rank Adaptation
Algorithm 1 CoTo
Require: Foundation model parameters 0o. LoRA adapters A0 { [AW:) and the total number of training steps T
1: for = ....., do
2: Compute activation probability: p == min {1,
4f
32
3: Draw a vector 7 = [7]). . T where each 7/i is sampled independently from uniform distribution (( 1)
Mh.
4: for each layer = 1,...: do
5: Set adapter state layer i: d 1/17 pl
6: end for
7: Compute the prediction y
(o: (W 8,1 AW,
8: Compute the loss L y)
9: Compute the gradient (aw.)) (() y) and update the model parameters (AW;)
10: end for
A. Training Details
A.1. Implementation Details
CoTo implemented as lightweight Trainerčal lback within the PEFT ecosystem", enabling seamless integration with any Ltyl fine-tuning loop. Algorithm summarizes its computational procedure.
A.2. Datasets
We evaluate CoTo across five computational prediction tasks: 1) image classification, 2) commonsense reasoning, 3)
mathematical reasoning, 4) language understanding, and 5) image generation. For image classification, we follow Zanella Ben Ayed (2024) and use datasets:
Aircraft (Maji et al., 2013) (aircraft classification)
Caltech (Fei-Fei et al., 2007) (object recognition)
Cars (Krause et al., 2013) (car classification)
DTD (Cimpoi et al., 2014) (visual texture classification)
EuroSAT (Helber al., 2019) (satellite land classification)
Flowers (Nilsback Zisserman, 2008) (flower classification)
Food (Bossard et al., 2014) (food classification)
ImageNet (Deng et al., 2009) (large-scale object recognition)
Pets (Parkhi al., 2012) (pet breed classification)
SUN (Xiao et al., 2010) (scene recognition)
UCF (Soomro et al., 2012) (human action classification)
For commonsense reasoning, 8 tasks from Commonsensel10k 70K (Hu et al., 2023):
ARC-c and ARC-e (Clark al., 2018) (science questions)
BoolQ (Clark et al., 2019) (yes/no questions)
HellaSwag (Zellers et al., 2019) (commonsense inference)
OBQA (Mihaylov al., 2018) (multi-step reasoning)
14

---

Co A Progressive Strategy to Boost Low-Rank Adaptation
Table Grid-scarched learning rates for 11 image classification tasks (Zanella & Ben Ayed, 2024) across different LoRA variants.
Method
Aireraft
Caltech
Cars
DTD
EuroSAT
Flowers
Food
ImageNet
Pets
SUN
UCF
LoRA
2e-4
2e-4
2e-4
2e-4
2e-4
2e-4
2e-4
2e-4
2e-4
2e-4
2e-4
LoRA-CoTo
5e-4
5e-4
Se-4
5e-4
5e-4
5e-4
5e-4
Se-4
5e-4
5e-4
Se-4
DoRA
le-3
le-4
le-3
le-4
le-4
le-3
Te-4
1o-4
1e-4
le-4
le-4
DoRA-CoTo
le-3
2c-4
le-3
2c-4
le-3
le-3
2e-4
2e-4
2e-4
2c-4
2e-4
HiRA
5e-3
lc-3
5c-3
le-3
5e-3
5e-3
le-3
5o-3
le-3
le-3
le-3
HiRA-CoTo
le-2
Se-3
Se-2
5e-3
le-2
5e-3
Se-3
Se-3
5e-3
Se.3
Table Standard deviation of classification accuracies across three random seeds 11 image classification tasks.
Method
Aircraft
Caltech
Cars
DTD
FuroSAT
Flowers
Food
ImageNet
Pets
SUN
UCF
LoRA
0.75
0.12
0.24
0.97
0.81
0.19
0.22
0.06
0.27
0.18
0.44
LoRA-Collo
0.72
0.10
0.29
0.39
0).4
0.36
0.18
0.04
0.39
0.20
0.22
DoRA
1.04
0.32
0.32
1.39
1.09
0.20
0.28
0.10
0.22
DoRA-CoTo
0.31
0.20
0.19
0.62
0.41
0.12
0.11
0.09
0.11
0.21
0.17
HiRA
0.27
0.09
0,25
1,33
1.33
0.02
0.09
0.14
0.09
0.11
0.23
HiRA-CoTo
0.15
0.11
0.08
0.28
0.05
0.15
0.12
0.10
0.36
0.13
0.35
PIQA (Bisk al., 2020) (physical commonsense reasoning)
SIQA (Sap et al., 2019) (social reasoning)
WinoGrande (Sakaguchi al., 2021) (fill-in-the ii--th--bland questions)
For mathematical reasoning, fine-tune MetaMathQA (Yu al., 2024) and test on GSM8K (Cobbe al., 2021).
For language understanding, we follow Zhao et al. (2024b) and use 9 tasks from GLUE (Wang et al., 2019) and Flan Collection (Longpre et al., 2023):
CoLA (Dolan & Brockett, 2005) (linguistic acceptability)
MNLI (Williams et al., 2018) (multi-genre natural language inference)
MRPC (Dolan & Brockett, 2005) (paraphrase detection)
QNLI (Rajpurkar et al., 2016) (question-answering)
QQP
(Quora questions)
RTE (Dagan et al., 2005; Bar Haim al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) (textual entailment
recognition)
SNLI (Bowman al., 2015) (natural language inference)
SST2 (Socher et al., 2013) (sentiment analysis)
WNLI (Levesque al., 2012) (coreference resolution)
For image generation, we train on two content categories from DreamBooth (Ruiz et al., 2023) and two artistic styles from Hugging Face's s LoRA the Explorer
hhtpps//data..uurrccoo//// Duse Dese Pairs http:://hnggingfcccccccccccttttn waa.aaaaaanntooaaaeeeeeeeeeee
15

---

CoTo: A Progressive Strategy to Boost Low-Rank Adaptation
Table 10. LoRA hyperparameter configurations for different tasks. The dropout predefined the LoRAConfig class and applied independently of the proposed CoTo.
Task
Rank
Default Learning Rate
CoTo Learning Rate
Insertion Module
Dropout Rate
Image Classification
Se-5
2e-4
Attention Attention, Projection
0
Commonsense Reasoning
5c-5
0.05
Mathematical Reasoning
2e-5
le-4
Attention, Projection, Gating
0.1
Language Understanding
2c-5
Attention (Q, Attention
0.05
Image Generation
le-5
5e-5
0
Table Average accuracy (%) on multi-task merging discriminative language understanding tasks (Wang et al., 2019) using the DeBERTa-v3 (He et al., 2023) backbonc.
Learning Rate
Method
CoLA
MRPC
QNLI
QQP
RTE
SST2
Avg
LoRA
87.44
89.46
94,31
91.05
85.56
95.18
90.50
Fusion
69.89
68.38
47.29
55.05
59.35
5e-4
w/ CoTo
Ensemble
69.13
31.62
50.54
63.31
52.71
50.92
53.04
LoRA-LEGO
73.28
33.15
74.19
61.46
69.18
65.37
LoRA
86.48
88.48
93.87
91.16
84.12
94.95
89.84
Fusion
69.13
68.38
49.46
63.18
47.29
50.92
58.06
le-3
Ensemble
69.13
68.38
49.46
63.18
47.29
50,92
58.06
LoRA-LEGO
73.54
69.32
79.54
78.67
51.99
54.13
67.86
LoRA
86.48
89.22
90.12
82.67
94.95
89.56
Fusion
69.22
68.38
67.12
70.32
47.29
58.14
63.41
5e-4
Ensemble
70.66
40.69
51.31
66.71
47.29
66.40
57.18
w/ Colo
LoRA-LEGO
53.75
70.77
76.19
75.26
64.31
89.01
71.55
LoRA
86.58
89.95
93.92
90.49
83.03
95.18
89.86
Linear Fusion
73.15
68.38
51.91
69.96
49,10
63.30
62.63
16-3
Ensemble
72.39
68.38
51,42
66.98
48.38
61.58
61.52
LoRA-LEGO
71.84
72.39
73.15
72.95
63.51
87.24
73.51
A.3. Hyperparameters
We use publicly available implementations of CLIP-LORA (Zanella & Ben Ayed, 2024), DoRA (Liu et al., 2024a), HiRA (Huang et al., 2025), LoRA-Pro (Wang et al., 2024c), LoRA-LEGO (Zhao et al., 2024b), and ZipLoRA (Shah al., 2025), retaining original hyperparameters unless otherwise specified. Table 10 details the adapter rank, learning rate, insertion module, and dropout configurations for each task. Notably, HiRA requires significantly higher learning rate (10-20x the default) for convergence. For image classification, grid search learning rates around each method's default (see Table 8), while other tasks employ one initial learning rate paired with cosine annealing schedule.
B. Additional Experimental Results
B.I. Image Classification
Standard deviations across seeds remain low (< L.4%) for all image classification tasks (see Table 9), confirming result reliability in Table 1.
B.2. Single-Task Merging for Commonsense Reasoning
Per-task interpolation curves (see Figure 11) show that CoTo consistently outperforms vanilla LoRA across all 8 common- sense reasoning tasks, with flatter loss basins.
16

---

CoTo: A Progressive Strategy to Boost Low-Rank Adaptation
ARC-c
ARC-e
BoolQ
Hellas
56.2
90
01.0
70.0
00.5
50
00.0
08.5
10
89.5
30
862
80.0
OBQA
PIQA
SIQA
WinoG
83.50
60.0
79.4
LoRA
70 83.25
70.2
84
LaitA-CuTa
61.5
80
83.00
790
80
82.35
75.6
82.50
788
Figure 11. Linear interpolation accuracy on individual commonsense reasoning tasks (Hu al., 2023).
Aitcraft
Caltech
Cars
DTD
AB DecoMtaiio
Hala 2/
Q647
ky
TAM
All BrapPrarr Eun
0.42
Re
Val
Al BiwySteee
Qu4y
Bey
Q
Xxg
Vile
EuroSAT
Flowers
Food
ImageNet.
Pets
SUN
UCF
A Average
Km
000
Figure
Average accuracy (%) on structured model pruning image classification tasks.
B.3. Multi-Task Merging for Discriminative Language Understanding
CoTo improves merging accuracy by 4.18%-6.18% across linear weight fusion, linear model ensemble, and LoRA- LEGO (Zhao et al., 2024b) strategies (see Table 11), validating benefits in discriminative language tasks using the DeBERTa-v3 (He et al., 2023) backbone.
B.4. Structured Pruning for Image Classification
CoTo maintains higher accuracy than vanilla LoRA under all structured pruning strategies across 11 image classification tasks (Figure 12).
C. Extended Analysis
C.1. Activation Schedule Ablation on DTD
Linear activation (CoTo default) balances convergence speed and robustness (see Figure 13). Exponential schedules improve merging/pruning but delay early convergence; sine schedules accelerate convergence but reduce robustness.
C.2. Weight Convergence Visualization
t-SNE plots (see Figure 14) confirm LoRA adapters converge near initialization (i.e., "lazy training") across learning rates. CoTo yields tighter clusters under initialization noise.
17

---

Co A Progressive Strategy to Boost Low-Rank Adaptation
Activation Probsbiity Curve
Geueralization
Merging
Pruning
1.0
72.5
70.0
60
70
67.5
0.6
65.0
0.4
55
62.5
60.0
Exponeeniil
0.2
57.5
Sine
Linear
0.0
56.0
Figure 13. Performance evolution during training under different activation schedules on the visual texture classification task (Cimpoi
al., 2014). Merging accuracy is measured at À 0.5. Pruning accuracy is measured for EveryOther (i.e. removing alternating layers).
t-SNE with P
1-8N with Perplexity-10
t-SNE with Porplexity=20
Initialization
20
15
LoRA 5e-5
LoRA_les
10
LoRA 5e-4
5
LoRA 1e:3
LoRA-CoTo 5e-5
LoRA-CoTo 1e4
Coftu
40
LoRA-CoTo
-20
X
X
Figure t-SNE visualization (Van der Maaten Hinton, 2008) the learned weights LoRA adapters across five seeds and four learning rates, under three perplexity settings. Black dots denote initialization points, and color gradients indicate different learning rates.
D. Proof of Theorem 3.1
Theorem 3.1. Let E( y) be convex loss function. Then for any fixed p [0, 1].
min (AW.)
Eg [E (b.)))
min W.W
Ew)(p) l v)
Consequently, the expected CoTo objective at step t upper-bounds a binomially weighted sum of the subnetvork losses.
Proof.
Es [e (g. y)] = 1Es [e (5 Doi (W. AW.l .9)]
= Σ (1) pf p) -) ( rn: +5 [W v)
(8)
Σ wg(DDe (Eqoill= [s (100:: (W: 5,1 ))
E ey (o)e (0,.0)
18


Reviews:
[{"review_id": "RSX0C9KhxN", "reviewer": "Reviewer_Np5j", "summary": "This paper proposes CoTo, a training strategy for LoRA that progressively deactivates adapters during training to promote balanced optimization across layers. CoTo enhances generalization, model merging, and pruning while reducing training time, demonstrating performance improvements across vision-language models (CLIP), LLMs (LLaMA), and diffusion models (SDXL).\n\n\n## update after rebuttal\n\nI am still not convinced by the reply to Q2 regarding the theoretical analysis. I am keeping my score unchanged.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer Np5j for the valuable comments. Below, please find our responses to each concern, and let us know if any issues remain. All experiments during the response period (Fig. **r**X & Tab. **r**X) are accessible in this anonymous link https://anonymous.4open.science/r/coto.\n\n#### [Q1]  Direct gradient-based validation for layer-wise optimization balance\n\n> - **Gradient-based validation**\n>\n>   To directly validate layer-wise optimization balance, we compute the **average gradient magnitudes of layers** from both the text and vision encoders on vision classification tasks. For comparability, gradients are normalized such that their sum across layers equals one. The results below indicate that CoTo leads to a **more balanced gradient distribution** across layers:\n>\n>   |                    | Lower Layers (0-4) | Middle Layers (5-8) | Higher Layers (9-12) |\n>   | ------------------ | ------------------ | ------------------- | -------------------- |\n>   | LoRA (text)        | 27.59%             | 32.35%              | 40.06%               |\n>   | LoRA-CoTo (text)   | 28.48%             | 33.53%              | 37.99%               |\n>   | LoRA (vision)      | 24.06%             | 32.59%              | 43.35%               |\n>   | LoRA-CoTo (vision) | 26.74%             | 33.90%              | 39.35%               |\n>\n> - **Other validations already presented in our manuscript**\n>   - Fig. 6 shows that CoTo-trained models consistently outperform LoRA, regardless of whether the lower, middle, or higher layer-adapters are pruned.\n>   - Fig. 7 presents Shapley value analyses, revealing that CoTo enhances the marginal contributions of lower and middle layers.\n>\n> Together, these results support the effectiveness of CoTo in promoting balanced layer-wise optimization.\n\n\n#### [Q2]  The theoretical analysis may not fully generalize to transformers in attention layers.\n> For clarity and simplicity of notation, we formulate our method using fully connected networks, where $W_l$ represents the total parameters of layer $l$. However, since our analysis (e.g., Theorem 3.1) considers activation at the layer level (as described in Sec. 3.1), it does not rely on the internal structure of each layer. By the Law of Large Numbers, the CoTo objective can be reasonably approximated as a weighted sum of sub-objectives, making the analysis broadly applicable — including to transformer-based architectures.\n\n\n#### [Q3]  The experiments on CoTo's adapter merging are limited to LLaMA-2 and LLaMA-3. Existing works highlight the importance of adapter merging across different tasks/domains, but this paper's scope is too narrow to generalize.\n\n> We respectfully clarify below the **broad scope** of adapter merging evaluations, which we have deliberately designed to validate CoTo's high generalizability. \n> - **Architectural scope includes LLaMA-2/LLaMA-3/CLIP/SDXL/DeBERTa V3**\n>    - As recognized by your summary, we evaluate adapter merging on CLIP (Fig. 4), LLaMAs (Fig.4, Tab. 4), and SDXL (Fig. 5), covering **widely adopted architectures** in text, vision, and multimodal domains. \n>    - We also evaluate CoTo on the **BERT-style architecture** DeBERTa V3 (Tab. r1). Despite the challenges of merging BERT-style models -- requiring additional fine-tuning of task-specific poolers and classifiers -- CoTo still outperforms.\n> - We fully agree that adapter merging across different tasks/domains is critical. Aware of this, Sec. 4.2.2 (\"LoRA Merging with Different Tasks\") merges 7 adapters trained on 7 in-domain (ID) tasks, and evaluates the cross-task merged adapter on both ID and 2 out-of-domain (OOD) tasks. The consistent superiority with CoTo over baselines advocates our **broad scope in cross-task adapter merging**.\n> - Besides, our adapter-merging evaluations span diverse task types, including vision classification (Fig. 4), commonsense reasoning (Fig. 3), natural language understanding (Tab. 4), and customized generation via diffusion models (Fig. 5), further validating **CoTo's generalizability across tasks/domains.**\n\n\n#### [Q4]  Why use a linear increase in activation probability?\n\n> We select linear increase as it offers a **favorable trade-off between simplicity and performance**, preserving the core benefits of the proposed CoTo.\n> - Simplicity: Unlike other activation probability scheduling functions (e.g., sine and exponential), linear increase **requires no additional hyperparameters** given the first phase spanning 75% of the training duration (cf. Line 410 (left)).\n> - Performance: Following the reviewer's great suggestion, we compare **linear**, **sine**, and **exponential** activation probability functions. As evidenced in Fig. r1, linear increase strikes a delicate balance between generalization, merging, and pruning performance.", "claims_and_evidence": "Most claims are well-supported by empirical evidence, including performance gains, improved model merging, and pruning efficiency across multiple benchmarks. However, the theoretical analysis is limited to fully connected networks, and layer-wise optimization balance lacks direct gradient-based validation.", "methods_and_evaluation_criteria": "The methodology and evaluation use diverse benchmarks across vision-language models, LLMs, and diffusion models. However, layer-wise optimization balance lacks gradient-based validation, and the choice of a linear activation schedule is not compared to alternatives.", "theoretical_claims": "The theoretical analysis may not fully generalize to transformers in attention layers.", "experimental_designs_or_analyses": "The experiments cover various models and LoRA variants, with clear results supporting CoTo's improvements. However, layer-wise optimization balance is not directly validated using gradient-based metrics.", "relation_to_broader_scientific_literature": "The paper relates to the PEFT literature, building on LoRA and its variants while introducing stochastic adapter activation to improve layer-wise optimization. It also connects to Linear Mode Connectivity (LMC) for model merging and extends dropout and stochastic depth methods to adaptively adjust activation for balanced optimization.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Lack of Mathematical Analysis on Gradient Distribution and Convergence: The paper does not provide a detailed mathematical analysis of gradient distribution or training convergence, leaving uncertainty about the optimization stability and convergence behavior of CoTo.\n\nNarrow Scope of Adapter Merging Tests: The experiments on CoTo's adapter merging are limited to LLaMA-2 and LLaMA-3. Existing works highlight the importance of adapter merging across different tasks/domains, but this paper's scope is too narrow to generalize.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Why use a linear increase in activation probability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "55y5WHCNZ8", "reviewer": "Reviewer_VYdq", "summary": "The paper introduces a training strategy to progressively deactivate adapters during training to ensure better optimization across all layers, enhancing model performance and efficiency. Extensive experiments across various models and tasks demonstrate its effectiveness in boosting LoRA's capabilities, including improved generalization, better model merging, and efficient pruning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer VYdq for the thoughtful and constructive comments. We address each concern in detail below and remain open to any further questions.\n\n[W1] Notation is a bit unclear. What exactly does \"a single LoRA layer\" refer to? And what does \"adapter\" refer to? Does \"one adapter\" mean all LoRA modules added to these matrices in one layer?\n> We greatly appreciate the reviewer for bringing our attention to this notation, which indeed benefits from further clarification.\n>\n> - Adapter: As stated in Lines 11–13 (right) of our manuscript, we use the term adapter to refer to the trainable parameters introduced by LoRA. \n> - A single LoRA layer = a single adapter denotes **the collection of all LoRA modules inserted into a single Transformer layer**. \n>   - The exact inserting locations of LoRA within each layer vary by task, following state-of-the-art practices, as detailed in the \"Target Module\" of Tab. 7.\n>   - Consequently, activation decisions are applied collectively to all LoRA modules within a layer, rather than to individual modules.\n\n[W2] Explain in more detail how the adapter drop method works\n>Absolutely. Thank you again for the question. The adapter drop mechanism is a key component of our training strategy, described in Sec. 3.1 and illustrated in **Fig. 1**, and we appreciate the opportunity to clarify it further.\n>\n>At each training step, we sample a Bernoulli variable $\\delta_l\\sim \\text{B}(p)$ for each layer to determine **whether the entire layer of adapters is activated**. This sampling is implemented using the `TrainerCallback` function at each step. The activation probability $p$ increases linearly from 0 to 1 during the first three-quarters of training and remains fixed at 1 thereafter. When deactivated, **all LoRA modules within a transformer layer** are excluded from both forward and backward computation. As shown in Fig. 1, this strategy enables extensive layer-skipping during the early stages of training and gradually transitions to full activation of all adapters.\n\n[W3] Table 1 performance improvement seems marginal\n> Thank you for this insightful question. While the improvements in Tab. 1 may appear modest in absolute terms, we note that:\n>\n> - Tables 1 and 2 report results on standard benchmarks, where performance is already strong. CoTo still achieves consistent gains while maintaining lower cost (Sec. 5.3). Compared to prior methods, such as DoRA vs. LoRA and HiRA vs. DoRA, CoTo provides comparable or even greater improvements, setting new state-of-the-art results in several settings.\n> - In addition to generalization accuracy gains, CoTo significantly enhances the **merging and pruning performance** of LoRA adapters. This is another major benefit of our method.\n>\n> CoTo improves generalization at both the **sample level** and the **task level**:\n>\n> - Tab. 1–3 evaluate performance on the test sets of in-domain tasks, demonstrating sample-level generalization. In Tab. 3, we train on MetaMathQA and test on GSM8K, which differ in domain, reflecting robustness under domain shift.\n> - Tab. 4 includes explicitly **held-out tasks** in the merging experiments. Following the LoRA-LEGO protocol, we train LoRA adapters on a subset of tasks (in-domain) and evaluate merged adapters on both seen (ID) and unseen (OOD) tasks. CoTo consistently improves performance on these held-out tasks, demonstrating strong task-level generalization.\n>\n> We will revise the manuscript to more clearly highlight these generalization settings and the role of held-out tasks in our experimental design.\n\n[W4] Line 160-164, why does a decreasing learning rates facilitates the exploration of new local optima...\n>  We acknowledge that the original statement \"Coupled with decreasing learning rates, this strategy facilitates the exploration of new local optima while preserving properties established in earlier stages\" was confusing and appreciate the opportunity to clarify.\n>- The \"strategy\" referenced here is the proposed CoTo strategy, which is responsible for facilitating the exploration of new local optima.\n>- What we intended to deliver is that \"decreasing learning rates\" coupled with the lazy training property of neural networks (cf. Line 158 - 160) is **responsible for \"preserving properties established in earlier stages\"**, such as linear mode connectivity and dropout stability.\n>   - Specifically, as supported by prior work [r1, r2], using a higher learning rate during the early training phase promotes broad exploration of the loss landscape. Subsequently, gradual learning rate decay encourages convergence to flatter, more generalizable minima. In our empirical setting, we adopt a cosine decay learning rate schedule, which we believe helps maintain early-stage properties.\n>  We will revise the manuscript to better reflect this intent.\n>\n>  [r1] SGDR: Stochastic Gradient Descent with Warm Restarts.\n>\n>  [r2] A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method and evaluation criteria make sense for the problem.", "theoretical_claims": "Yes, I checked the correctness of the proofs for theoretical claims.", "experimental_designs_or_analyses": "Yes, I checked the soundness of all experimental designs.", "relation_to_broader_scientific_literature": "Please refer to Strengths and Weaknesses.", "essential_references_not_discussed": "Please refer to Strengths and Weaknesses.", "other_strengths_and_weaknesses": "Strengths:\n\n1. CoTo provides a novel approach to addressing the uneven distribution of LoRA updates across layers.\n2. The strategy is supported by both theoretical insights and empirical evaluations, showcasing consistent performance improvements.\n3. CoTo is compatible with various LoRA variants and extends its benefits to tasks such as model merging and pruning, demonstrating broad applicability.\n\nWeaknesses:\n\n1. The paper's notation is a bit unclear. What exactly does \"a single LoRA layer\" refer to? And what does \"adapter\" refer to? In transformer models, a single layer typically contains multiple weight matrices, such as the attention weight matrix and the projection weight matrix. Does \"one adapter\" mean all the LoRA modules added to these matrices in one layer? \n2. Following up on the previous question, could you explain in more detail how the adapter drop method works? \n3. In Table 1, the improvement in performance seems marginal. How do the experiments show that CoTo can enhance generalization? Are any held-out tasks set aside to test this?\n4. Line 160-164, why does a decreasing learning rates facilitates the exploration of new local optima? Is this an assumption? What's the strategy for decreasing the learning rate?", "other_comments_or_suggestions": "Please refer to Strengths and Weaknesses.", "questions_for_authors": "Please refer to Strengths and Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EaQoZbsQ2k", "reviewer": "Reviewer_Xi9N", "summary": "The paper introduces CoTo, which integrates structured dropout with LoRA fine-tuning, demonstrating improved generalization and enhanced performance in model merging and pruning. Similar to stochastic depth, the proposed method freeze LoRA adaptor for certain layers with a certain probability and such probability decreases over the entire training process.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer Xi9N for the thoughtful review and for highlighting both the strengths and limitations of our work. We especially appreciate the recognition of our experimental rigor and the inclination toward acceptance. Below, please find our responses to the main concerns, and let us know if any issues remain. Additional experiments conducted during the response period (see Fig. **r**X & Tab. **r**X) are available in this anonymous link: https://anonymous.4open.science/r/coto.\n\n\n#### [Weakness] As the authors mentioned in related works, there are many works that tested applying structured dropout and stochastic depth. The novelty of the paper is limited, i.e., applying structured dropout to PEFT methods.\n\n> We fully understand the concern regarding novelty. Indeed, from a structural perspective, **our method may appear similar to prior work** on structured dropout and stochastic depth. We acknowledge this and appreciate the reviewer’s valuable observation.\n>\n> - We welcome the opportunity to clarify CoTo's distinct contributions and novelty within the PEFT landscape. While there are superficial similarities, CoTo's core innovation goes beyond simply applying existing ideas. As discussed in **Appendix C** and summarized in **Table 8**, while CoTo, dropout, and stochastic depth all involve deactivation during training, they differ fundamentally in objective:\n>\n>   - **Dropout** aims to prevent overfitting by randomly deactivating individual neurons or weights.\n>\n>   - **Stochastic depth** is primarily designed for efficient pretraining of full-model by randomly skipping entire layers.\n>\n>   - **CoTo** is designed as a training paradigm for the **layer-wise PEFT method**, aiming to **balance the utilization of LoRA adapter layers**. It prevents adapters from dominating and promotes better merging and pruning by employing **structured, layer-wise deactivation coupled with a progressive activation strategy**.\n>\n>   In essence, CoTo synthesizes concepts from regularization (like dropout) and efficient training (like stochastic depth) but repositions and adapts them into a coherent strategy specifically targeting the unique dynamics and optimization challenges of layerwise PEFT.\n>\n> - To empirically demonstrate this distinction, we provide additional experiments (**Anonymous Figs. r2 and r3**) on image classification comparing CoTo against direct applications of dropout to LoRA adapters:\n>\n>   - Standard Dropout (fixed $p$).\n>\n>   - Progressive Dropout (linearly decaying $p$).\n>\n>   **Fig. r2** shows LoRA adapter merging robustness under multiple seeds, and **Fig. r3** evaluates structured pruning across adapter components. The results clearly show that **naive applications of dropout strategies yield limited benefits** and do not effectively improve merging or pruning robustness. CoTo consistently and significantly outperforms these variants in terms of accuracy, merging performance, and pruning robustness, highlighting that its effectiveness stems from its PEFT-specific design, not merely from applying dropout principles.\n>\n> Finally, we note that **CoTo is fully compatible with standard dropout**, which is already included in our main experiments. We will revise the manuscript to better highlight these conceptual and empirical distinctions, helping readers better position CoTo within the broader landscape of PEFT techniques.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The proposed method is simple and not new. It is very intuitive and makes sense for improving generalization.", "theoretical_claims": "I did not check the correctness of Theorem 3.1.", "experimental_designs_or_analyses": "I have checked the validity of the experimental design. I believe it is natural to compare LoRA-variants without CoTo and those with CoTo.", "relation_to_broader_scientific_literature": "Researchers in this field may get interested in the paper as it improves generalization of PEFT methods.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The proposed method shows consistent improvements compared to baselines.\n2. The experiments are extensive, including VLM, and LLM.\n3. The paper also provides extensive studies, including how CoTo affects model merging, and pruning. It provides interesting insights to the researchers in the field.\n\nWeaknesses:\n1. As the authors mentioned in related works, there are many works that tested applying structured dropout and stochastic depth. The novelty of the paper is limited, i.e., applying structured dropout to PEFT methods.", "other_comments_or_suggestions": "While the overall novelty of the paper is somewhat limited, i.e., mainly combining existing techniques such as stochastic depth and structured dropout within the PEFT framework, the strength of the experimental results helps compensate for this. The experiments are extensive, and demonstrate significant improvements over baselines. Therefore, I am inclined to recommend acceptance of the paper.", "questions_for_authors": "I do not have other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Igd8zN7Bts", "reviewer": "Reviewer_uABN", "summary": "The paper proposes a simple regularization strategy for LoRA fine-tuning that stochastically drops LoRA adapters according to a certain schedule.\nThe authors show that the proposed training strategy enhances linear mode connectivity (LMC) and adapter-wise dropout stability.\nFurthermore, it usually improves fine-tuning over standard LoRA, DoRA and HiRA.\nFinally, the authors present an analysis on robustness to pruning adapters, comparing linear fusion to ensembles, ablation studies on the dropout strategy and hyperparameters, using shapley values to determine importance of adapters, and custom generation with diffusion.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer uABN for the insightful feedback, which helped improve both our analysis and presentation. Below, we respond to the main concerns point by point. Additional results are available via https://anonymous.4open.science/r/coto, with new content labeled as Tab. rX and Fig. rX.\n\n#### Claims And Evidence\n\n> We agree with these concerns and will revise the claims for clarity.\n>\n> - L19,r: Fig. r4 shows that models with the same initialization tend to converge to similar solutions, regardless of method or learning rate. We will also cite works supporting lazy training dynamics.\n>\n> - L75-76,r: The term “hierarchical structure” was unclear. We will revise it to *layer-wise importance differences*, note that layer importance is modality-dependent, and specify “layer-wise LoRA-style PEFT methods” instead of the broader term PEFT.\n>\n> - L416,l: Fig. 6-7 in the manuscript show pruning and marginal contributions at 25% training. Fig. r1 provides additional merging and pruning results over training steps.\n>\n> - L430,r: As shown in [r1], the dropout objective is equivalent to adding a regularization term. Similarly, CoTo’s objective is a weighted sum of sub-objectives, where the weight ($w_6$ in Fig.2) of the standard objective gradually dominates as training progresses. \n>\n>   [r1] On the Regularization Properties of Structured Dropout.\n>\n> - L434,r: Fig. r5 has been updated with unified colorbars.\n\n#### Experimental Designs\n\n[E.1] Comparison to other stochastic techniques\n\n> Stochastic depth is designed for pretraining and not applicable to LoRA. [2] introduces a KL loss between dropout and non-dropout outputs during PEFT but is not open-sourced or directly applicable. Thus, we compare CoTo with Dropout [1] and a variant using our progressive strategy. As shown in Fig. r2 and Fig. r3, CoTo consistently outperforms naive dropout in general, merging and pruning tasks.\n\n[E.2] Run LoRA with the same learning rate\n\n> - In Tab. 5 of ablation studies, we add LoRA-Pro results with 1e-4 and 2e-4 for fair comparison (55.8 ± 0.7, 40.9 ± 1.1). On classification tasks, CoTo also consistently outperforms LoRA on general, merging, and pruning performance across five learning rates (Fig. r6).\n> - The higher learning rate for CoTo offsets slower convergence from stochastic training. Full learning rate settings are provided in Appendix A.3.\n\n[E.3] Remove claims or report quantitative measures\n\n> We will remove “significant” and move diffusion results to Appendix. Additional quantitative results (Tab. r2) and qualitative examples will be included.\n\n[E.4] Why DTD for pruning analysis\n\n> DTD has relatively low zero-shot performance (~44%), making the LoRA's gain more visible. Results on all vision tasks are provided in **Fig. r7** for completeness.\n\n[E.5] Why use generative models instead of BERT-style models like DeBERTa-v3\n\n> - We follow LoRA-LEGO, which evaluates only generative LLMs, as they are more advanced and suitable for merging. BERT models for classification rely on task-specific poolers and classifiers, making merging less meaningful (e.g., not generalizable to OOD tasks). In contrast, generative models support instruction tuning across diverse tasks via a unified prompt format [r2], offering a more rigorous testbed. \n>\n>   [r2] Finetuned Language Models are Zero-Shot Learners.\n>\n> - As suggested, we have also added results on DeBERTa-v3 in Tab. r1, where CoTo still shows consistent improvements.\n\n[E.6] Analysis of the magnitude of P\n\n> We optimize the alignment matrix P by minimizing the proposed $\\Delta_{upper}$ using Adam (500 steps, lr=0.01) and analyze both the magnitude of P and the difference $\\|\\Delta W_f - \\Delta W_m\\|_2$ before and after applying P. Results are shown in Fig. r8. For LoRA, the magnitude of P is larger, and the difference (both before and after applying P) is smaller than in CoTo. This indicates that CoTo improves linear mode connectivity not by adapter-wise alignment but by addressing layer-wise misalignment.\n\n#### Essential References\n\n> Thanks for highlighting these works. We will cite and discuss them in the revision.\n\n#### Weaknesses\n\n[W.1] Fig. 9 shows that CoTo requires longer training until convergence\n\n> The slower loss decrease in Fig. 9 is due to CoTo’s stochastic activation, which delays early convergence but improves generalization. Notably, CoTo with LoRA and DoRA reaches lower loss at 900 and 2k steps, respectively. Also, as each step in CoTo is faster (due to adapter skipping), total training time is reduced (Tab. 6). Any delay from stochasticity can be mitigated by adjusting the learning rate or activation schedule (Fig. r1).\n\n[W.2] Intuition why Ensemble-CoTo performs worse \n\n> Ensemble-CoTo improves on DeBERTa (Tab. r1) but underperforms on LLaMA, possibly due to non-convergence. We will further investigate this and update the results if new findings emerge.\n\n#### Comments\n> Thanks, we will revise accordingly.\n\n#### Questions\n> Please see our response to E.6.", "claims_and_evidence": "There are a few claims that lack support: \n\n - line 19 (right): \"LoRA oftentimes rapidly overfits and converges to a local optimum near initialization.\" - no references or empirical evidence is presented\n - line 75 (right): The authors claim that PEFT follows a hierarchical structure - What is meant by hierarchical structure? Is it meant that higher layers need more adapter ranks? If so, the analysis in Section 5.2 showed that applying higher dropout in higher layer adapters performs better, which directly contradicts the claim that higher layers need more adapters. Also the analysis on Shapley values in Section 5.1 shows that for text, lower layer adapters are more important, indicating that there is also a dependence on data modality.\n - line 76 (right): The authors claim that CoTo accounts for the \"hierarchical structure of PEFT\" and is \"tailored for PEFT\", but no evidence is provided for that. In fact, the authors only consider LoRA-style methods, but PEFT comprises a lot more methods. Furthermore, the same dropout probability is applied to all adapter layers regardless of layer depth, i.e. adapters across layers are equally weighted.\n - In line 416 (left) the authors claim that 75% first stage proportion leads to improved LMC and dropout stability, however the authors have not compared e.g. LMC for 50% vs 75% first-stage, therefore this claim was not verified. It was only verified for the efficiency vs performance trade-off.\n - In line 430 (right), the authors claim that \"Both theoretical and empirical analyses show that CoTo improves adapter-wise dropout stability and linear mode connectivity, leading to better generalization, interpolation, model merging, and pruning\" - however, the theoretical results of this work only show that the CoTo objectiev is equivalent to a weighted sum of objectives, which does not tell anything about other findings, which are purely empirical.\n - In line 434 (right): \"Shapley value analysis confirms that CoTo increases the marginal contribution of individual adapters.\" - This is not clear from looking at Figure 7, since not colorbars are shown and scale of different plots are unknown.", "methods_and_evaluation_criteria": "The proposed training strategy is very simple and straightforward which I consider a strength of the work.\nIt makes sense and stochastic regularization is an important topic for PEFT methods that has not been explored in a lot of detail.", "theoretical_claims": "Theorem 3.1 shows that training via CoTo is equivalent to a progressively shifted objective weighted by some factor.\nI have not checked the proof for Theorem 3.1 in detail due to time constraints.", "experimental_designs_or_analyses": "I commend the authors for applying CoTo to different LoRA variants and the amount of experiments that were provided. However, I believe that a comparison to other stochastic regularization techniques such as Dropout [1] and Stochastic Depth, and [2] should be included. This forsters understanding of the training strategy and puts the proposed strategy into perspective to others.\n\n[1] Lin et al., LoRA Dropout as a Sparsity Regularizer for Overfitting Control, arXiv 2024\n\n[2] Wang et al., LoRA Meets Dropout under a Unified Framework, ACL 2024\n\nThe authors increase the learning rate for CoTo compared to competitors. In [1] it was shown that higher learning rate can be beneficial in the case of LoRA optimization. To enable a fair comparison to LoRA, I would recommend to also run LoRA with this same learning rate to account for that difference.\n\n[1] Hayou et al., Efficient Low Rank Adaptation of Large Models, ICML 2024\n\nThe results on diffusion models and shapley values are entirely qualitatively. Based on this the authors claim significant improvements of CoTo over competitors, which is not convincing. In fact by merely looking at the generated images in Figure 5 it is very hard to tell which one is of better quality. The same goes for shapley values when no color bar is provided. I recommend to either remove claims on significant improvements or report quantitative measure with variance estimates to verify them.\n\nWhy does the selection of DTD for pruning analysis ensure generalizable insights?\n\nOn the multitask experiments, the authors use generative language models for language understanding tasks leading to very low scores in general. On such tasks BERT-style models such as DeBERTa-v3 [1] are the state-of-the-art usually reaching over 90% average accuracy. This model has also commonly been used in for LoRA-style fine-tuning [2,3,4], why do the authors use generative models here?\n\n[1] He et al., DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing, ICLR 2023\n\n[2] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022\n\n[3] Zhang et al., AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, ICLR 2023\n\n[4] Meng et al., PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models, NeurIPS 2024\n\nIn Section 4.2.1, the authors aim to answer the question whether CoTo makes linear fusion closer to an ensemble, but this question is never clearly answered. From my understanding the reported results only show that CoTo is usually better than LoRA*. To give a definitive answer I believe it needs to be looked at the magnitude of P for LoRA* vs LoRA-CoTo*, as if the magnitude of P is larger than there is a larger gap between ensemble and linear fusion.", "relation_to_broader_scientific_literature": "In general, there are plenty of methods that build on LoRA-style parameter-efficient fine-tuning.\nMany of them investigate discarding more parameters, different initialization schemes, adaptive rank allocation, etc.\nThis paper investigates stochastic regularization for different LoRA variants, which provides a better understanding on how to best train LoRA-style methods.", "essential_references_not_discussed": "There is more related work [1,2,3] on adaptive rank allocation shows that for certain downstream tasks, naturally patterns emerge where more ranks are allocated to certain weights in higher layers and sometimes adapters are disregarded entirely. This could be used as additional support for dropping LoRA adapters stochastically during training.\n\n[1] Liu et al., Alora: Allocating low-rank adaptation for fine-tuning large language models., NAACL 2024\n[2] Paischer et al., One Initialization to Rule them All: Fine-Tuning via Explained Variance Adaptation, arXiv 2024\n[3] Meo et al., Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates. arXiv 2024", "other_strengths_and_weaknesses": "**Strengths**\n\nThe proposed training strategy is simple and intuitive.\n\nThe authors conduct plenty of experiments and analyses demonstrating advantages of the proposed training strategy for different LoRA variants.\n\n**Weaknesses**\n\nThe authors show learning curves for LoRA variants vs LoRA-CoTo variants in Figure 9 (Appendix B2) showing that CoTo requires longer training until convergence. This should also be mentioned in the main text to put the method in perspective.\n\nIn Table 4, Ensemble-CoTo performs consistently worse on average than the standard Ensemble method, do the authors have an intuition why this is the case?", "other_comments_or_suggestions": "Tables would be a bit nicer to read if subsequent rows were Method and Method-CoTo, separated by horizontal lines, otherwise one needs to jump back and forth across separated grouped rows.\nThe notion of adapters vs layers is a bit confusing. In the introduction the authors refer to LoRA adapters as \"adapters\", but then use \"layers\" throughout the remainder of the manuscript. This can be confused with a layer that contains multiple of adapters.\nYou could add a straight line in figure 4 that shows the pre-merge performance for LoRA, so the reader does not need to switch back and forth between table and plot.\n\n-- POST REBUTTAL --\n\nI would like to thank the authors for the added analysis which I believe improve the depth and rigor of the work. As all of my concerns have been properly addressed, I recommend acceptance of this work.", "questions_for_authors": "Have you looked at what P (used in LoRA*) learns during optimization? It seems that a good optimum would be to learn to invert one of the adapters for merging (i.e. inverting the fusion), such that eventually no fusion/ensembling is occuring at all.\n\nNo other remaining questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper develops an interesting stochastic approach to activate/deactivate adapters during training to improve low-rank adaptation (LoRA). The reviewers generally agree that the paper does an extensive set of experiments, and evaluates the ideas on all the relevant metrics. One of the reviews is more negative, but the issues do not seem to be serious enough.
Decision: Accept (poster)