ComLoRA: A Competitive Learning Approach for Enhancing LoRA
Authors: Qiushi Huang
Tom Ko
Lilian Tang
Yu Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Parametric-efficient fine-tuning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: We propose a Competitive Low-Rank Adaptation (ComLoRA) framework to address the limitations of the LoRA method, which either lacks capacity with a single rank-$r$ LoRA or risks inefficiency and overfitting with a larger rank-$Kr$ LoRA, where $K$ is an integer larger than 1. The proposed ComLoRA method initializes $K$ distinct LoRA components, each with rank $r$, and allows them to compete during training. This competition drives each LoRA component to outperform the others, improving overall model performance. The best-performing LoRA is selected based on validation metrics, ensuring that the final model outperforms a single rank-$r$ LoRA and matches the effectiveness of a larger rank-$Kr$ LoRA, all while avoiding extra computational overhead during inference. To the best of our knowledge, this is the first work to introduce and explore competitive learning in the context of LoRA optimization. The ComLoRA's code is available at https://github.com/hqsiswiliam/comlora.

Published as a conference paper at ICLR 2025
COMLORA: A COMPETITIVE LEARNING APPROACH FOR ENHANCING LORA
Qiushi Huang 3, Tom Ko Lilian Tang", Yu Zhang'
Southern University of Science and Technology, 'University of Surrey, BByteDance {qiushi, huang. cs, tomkocse, yu. paag.. ust u com, h. w.ttanesurrey.accc. .ck
ABSTRACT
We propose a Competitive Low-Rank Adaptation (ComLoRA) framework to ad- dress the limitations of the LoRA method, which either lacks capacity with sin- gle rank-r LoRA or risks inefficiency and overfitting with larger rank- Kr LoRA, where K is an integer larger than The proposed ComLoRA method initializes K distinct LoRA components, each with rank r, and allows them to compete dur- ing training. This competition drives each LoRA component to outperform the others, improving overall model performance. The best-performing LoRA is se- lected based on validation metrics, ensuring that the final model outperforms a single rank-r LoRA and matches the effectiveness of larger rank-Kr LoRA, all while avoiding extra computational overhead during inference. To the best our knowledge, this is the first work to introduce and explore competitive learn- ing in the context of LoRA optimization. The ComLoRA's code is available at https://githis htt.:ttthhcccoccooooo.eete com htg.im////saaaae swi aamm cttl/comlll.
INTRODUCTION
Large Language Models (LLMs) have transformed various natural language processing tasks by leveraging their vast number of parameters and advanced architectures (Radford et al. 2019 Achiam
2023 Touvron et al. 2023 Dubey et al., 2024). Despite their success, efficienlly adapt- ing LLMS to specific tasks remains a challenge due to the prohibitive costs of full fine-tuning (FFT).
Parameter-Effiiiinn Fine- Tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA) (Hu et al.,
2021 have emerged promising solutions by updating only small portion of parameters, signif- ieantly reducing computational burdens.
While LoRA provides an efficient alternative for adapting LLMs, it faces challenges in balanc- ing model expressiveness with parameter efficiency. single low-rank LoRA module may lack sufficient representational capacity to handle complex tasks, whereas simply increasing the rank in- troduces additional parameters and the risk of overfitting. Empirical studies on PEFT methods have shown that higher-rat rank configurations do not consistently outperform lower-rank setups in prac- tice (uu et al. 2021 Sidahmed et al., 2024 Kalajdzievski] 2023), indicating that merely adding more parameters is not straightforward solution for performance enhancement. Therefore, there is a need for training strategies that can better exploit existing LoRA components to enhance expres- siveness without increasing computational overhead during inference.
Existing LoRA training methods typically employ fixed hyperparameters and adaptation strategies across different tasks and throughout the training process. This lack of adaptability may hinder the model's 's ability to adjust to tasks with varying levels of complexity, such as those requiring deeper reasoning, context understanding, or handling of specialized terminology. Consequently, the model may fail to strike an optimal balance between efficiency and expressiveness, especially in more demanding tasks (Lialin et al. 2023). This limitation leads to the underperformance of LoRA modules when applied to a diverse range of tasks that challenge the model's capacity to represent complex patterns and relationships in the data.
"Corresoonding author.

---

Published as a conference paper at ICLR 2025
To address these challenges, draw inspiration from competitive learning, which has been ap- plied successfully in feature discovery and clustering (Nowlan 1989 Rumelhart Zipser [985] Grossberg, [987). Competitive learning allows multiple components of model to compete dur- ing training, dynamically optimizing their performance hased on feedback. Despite its successes in other domains, this mechanism has not yet been explored within the context of LoRA other PEFT methods.
We propose the Competitive Low-Rank Adaptation (ComLoRA) framework, which incorporates competitive learning to dynamically train multiple LoRA components. In ComLoRA, multiple LoRA components compete during adaptation, guided by dynamic selector that evaluates their performance at each training step. This competitive process allows each component to improve iteratively, achieving an optimal balance between expressiveness and parameter efficiency without significantly increasing computational overhead during inference. By leveraging the strengths of the most effective LoRA components, ComLoRA enhances model adaptation across diverse tasks while maintaining inference efficiency.
Our key contributions are as follows:
We propose the first competitive learning framework LoRA.
We develop a selector mechanism that optimizes LoRA component selection during train- ing.
Extensive experiments demonstrate the superiority of ComLoRA over LoRA, without in- creasing inference overhead.
2 RELATED WORKS
LoRA. Low-Rank Adaptation (LoRA) (Hu al. 202T) a widely used technique for efficient fine- tuning of large pre-trained models by introducing low-rank updates to parameter matrices, thereby reducing computational overhead. While LoRA maintains strong performance with fewer param- eters, its fixed rank can limit expressiveness. Attempts to increase the rank, such using K x r, enhance capacity but
cost of higher computational demands and increased risk of overfitting. ComLoRA addresses these challenges by incorporating competitive mechanism where multiple LoRA components compete during training, enhancing adaptability and robustness without sacrific- ing the efficiency that defines LoRA.
Mixture of Experts (MoE). MoE models have gained significant attention in the context of large language models (LLMs) due their ability scale efficiently by selectively activating only sub- set of the model's parameters during inference. A prominent example the Switch Transformer (Fe- dus et al. 2022). which uses an MoE layer to route inputs different expert networks, significantly reducing computational costs while maintaining performance. Similarly, GLaM (Du et al. 2022) dynamically selects subset of experts based on the input, allowing the model adapt efficiently to diverse tasks. MoELoRA (Luo et al. 2024) extends the MoE framework to LoRA by using gating network route inputs multiple LoRA experts during inference, emphasizing diversity among components. In contrast, ComLoRA focuses on learning single LoRA component after competitive training, minimizing inference overhead. Hence, MoELoRA needs to utilize multiple LoRA experts and the gating network during inference, while the proposed ComLoRA method acts exactly the same as LoRA during the inference.
METHODOLOGY
In this section, we introduce the Competitive Low-Rank Adaptation (ComLoRA) framework.
3.1 OVERVIEW
ComLoRA initializes K distinct LoRA components, each with rank A LoRA selector is intro- duced to dynamically choose the most suitable LoRA component based on the input context. This competitive framework drives the LoRA components to refine their individual strengths, effectively
2

---

Published as a conference paper at ICLR 2025
LLM Forward
Loss Computation
Determining Winner
Candidate Output
2 Target
LoRA Wirner
Compulation War BiAx
LORA
LoOe
Loss
Datunntyal
Selector
LLM
Selector
Losspetratte
Qurria
LoRA Condiciats
LoRA Embeddinge
Merge Winner
LLM
LLM.
Figure 1:
illustration of the ComLoRA pipeline.
combining them enhance the model's overall performance without adding inference overhead. An illustration of the proposed ComLoRA method is shown in Figure
3.2 MULTIPLE LORA COMPONENTS
We begin by initializing K separate LoRA components (LoRA, LoRAg. LoRA J. each with rank r. For LoRA,, we have projection matrices A₁ E Rdxx and B₁ R where d and denotes dimension of the input and output this layer, respectively. The matrices A are initialized using the Kaiming initialization method (He al. 2015), while the matrices B₃ are initialized to zero, following standard LoRA practices.
3.3 LORA SELECTOR
The LoRA selector is designed to evaluate the input sequence and select the most suitable LoRA component. operates by computing similarity scores between a representation the input se- quence and embeddings associated with each LoRA component.
Input Representation. Given an input sequence X E L of length L, where each token is embedded in d-dimensional space, obtain contextualized representation using lightweight one-layer transformer encoder. This neural network produces hidden states H E Lxdd, where deel the dimensionality of the selector's hidden states. To obtain fixed-length representation of the input sequence, we average the hidden states across the sequence length as
1 H.
havg L
(1)
2m1
where H; is the hidden state corresponding to the i-th token.
LoRA Embeddings. Each LoRA component & is associated with learnable embedding vector eş Rda, These embeddings serve as representatives for their respective LoRA, allowing the selector compute and update the similarity scores efficiently.
Computation of Similarity Score, The similarity score between the input representation and each LoRA component embedding calculated using dot product:
sim, avg ek for l 1, .......4 K.
This results in a similarity score vector s E R as
S = [sim ,im
(3)
3.4 OBJECTIVE FUNCTION
The objective function to train ComLoRA consists of three losses, including the language modeling loss, selector loss, and alignment loss. In the following section, we introduce them one by one.
Language Modeling Loss. The language modeling loss, denoted as LLM. is calculated to guide the updates LoRA components. For each LoRA component k: the loss defined as:
LIMA =
log P(ylUee, 0, Av. Ba), LIM = K
Σ LLM.K
(4)

---

Published as a conference paper at ICLR 2025
where y is the target token at position t, 9 represents the base model parameters, and A₁ and B5 the projection matrices of the k-th LoRA component. This loss function drives the training process, ensuring that the LoRA components are optimized to minimize the prediction error of the next token in the sequence.
Alignment Loss. To ensure that the LoRA selector effectively influences the language model (LM) during training. we introduce an alignment loss that reflects the selector's preferences in the LM updates. This alignment loss is designed to align the similarity scores, as predicted by the selector, with the next-token prediction losses for the top-N ranked LoRA components. The loss is defined as:
Lulign (t() (ttopGrad (Psim, lop-N) C.MM
(5)
where Psim, Nop-N denotes the normalized similarity scores of the top-N selected LoRA components with largest similarity scores, L.M. sep-N is vector containing L.M losses for the top-N selected LoRA components, StopGrad(.) indicates that the input is detached to prevent it from contributing to the gradient, and ry(t) is an annealing scalar that evolves over training steps. By focusing the top-N components, the alignment loss directs LM updates based on the top preferences of the LoRA selector. With the annealing strategy, expect that during the initial stage of the training process, LoRA components can learn more independently to encourage exploration and as train- ing progresses, the influence of the guidance from the LoRA selector the LoRA components gradually increases.
Pairwise Loss. The LoRA selector is trained align the similarity scores with the actual perfor- mance of the LoRA components using pairwise loss approach. This ensures that the similarity scores correlate with the relative performance (as measured by the language modeling loss) of the LoRA components. To avoid overly deterministic selection, Gaussian noise term is added to the language modeling loss for each LoRA component. The modified LM loss for component k is defined as:
LLMA = LLM.K + acEs
(6)
where ék ~ (0, 1) Gaussian noise term, and a is a small constant that controls the noise intensity (e.g., = 0.1). The differences between similarity scores and the corresponding noisy LM losses for each pair of LoRA components (i,j) are defined as:
A Aam.i.j sim; simj, ALM,i. LIM. LIM.j.
Those differences are normalized using the softmax operation to produce log-softmax values as
logsoftmax sim) = log(softmax(44
logsoftmax (ALM) = log(softmax (Am))).
(8)
The pairwise loss is then computed as the & loss between the log-softmax similarity score differ- ences and the log-softmax LM loss differences:
Lpairiise |logsoftmax Aim) - StonGrad((ogsoofmaxXAaaa (ALM))
where || || denotes the & norm of a vector. This pairwise loss encourages the selector to assign higher similarity scores LoRA components that demonstrate better performance (i.e., lower noisy LM losses), ensuring that the predictions of the selector align with the actual capabilities of each component.
Objective Function. The total loss combines the LM loss, the alignment loss, and the selector loss as
Liotal LIM + Calign Cairrise
(10)
During training, we optimize Lisal with respect to the parameters in LoRA components and the LoRA selector.
3.5 DETERMINATION OF LORA WINNER
After training, we use the LoRA selector to determine the LoRA winner based on the validation dataset. Specifically, we pass the validation data through the LoRA Selector to compute the similar- ity scores for each LoRA component as
Syal [sim sim,al,2,--.. sim
(11)
4

---

Published as a conference paper at ICLR 2025
where & is the accumulated similarity score for LoRA component & over the validation dataset. We then select the LoRA winner as the one with the highest total similarity score as
kwin arg max sim
(12)
&
During inference on the test set, we only use the LoRA winner LoRAgei without using the LoRA selector other LoRA components. This ensures that the inference is efficient, as it avoids any additional computational overhead associated with the selector or dynamic selection.
Table 1: Comparison of Full Fine-Tuning (FFT), LoRA, LORA, MoE of LoRA, and LomLoRAA
FFT
LoRA
MoE of LoRA
ComLoRA
Overfitting Risk
High
Low
Medium
Low
High Inference Overhead
Low
Low
High
Low
Competitive Learning
Low
Low
Low
High
Model Complexity
High
Low
High
Medium
3.6 COMPARISONS WITH FFT, LORA, AND MoE OF LORA
During the training process, LoRAs, leading to training complexity comparable to MoE with K LoRAs but higher than LoRA. During the inference process, the proposed ComLoRA method only uses the LoRA winner, which can be merged into the base model. Hence, ComLoRA preserves low inference overhead as LoRA did, while MoE of LoRA, which needs merged into the base model, yielding additionally computational costs. In summary, the comparisons with FFT, LoRA, and MoE of LoRA shown in Table T
see that the proposed ComLoRA method needs to train K
a gating network choose from K LoRAs, cannot be
EXPERIMENTS
In this section, evaluate the proposed ComLoRA method by comparing with LoRA to compare between the competitive training in ComLoRA and the conventional training process in LoRA.
4.1 EXPERIMENTAL SETTINGS
All experiments were conducted using the LLaMA-3-8B model (Dubey et al. 2024). To compare ComLoRA with LoRA across different rank configurations, we use LoRA ranks of 4, 8, 16, 32, and
128, and ComLoRA configurations with (K = 4, r 4). (K = 2, = 8). (K 4, = 8), and
(K = 4, 32).
The AdamW optimizer (Loshchilov & Hutter, 2019) is used to train LoRA and ComLoRA. The learning rates for both LoRA and ComLoRA methods are selected from [le 3, 1e 4), and fine- tuning is conducted for 3 epochs for all the tasks. Each experiment runs with different seeds, and the average results are reported.
4.2 EVALUATION TASKS
Commonsense Reasoning. We evaluate our models using a comprehensive commonsense reason- ing dataset, which includes eight sub-tasks: BoolQ (Clark et al., 2019), PIQA (Bisk et al. 2020
SIQA (Sap et al. 2019). HellaSwag (Zellers et al. 2019). WinoGrande (Sakaguchi et al. 2021
ARC-c (Clark et al. )018), ARC-e (Clark et al. 2018), and OBQA (Mihaylov et al. 2018). These tasks cover various aspects of commonsense knowledge, such as physical reasoning, social implica- tions, and natural language inference. We aggregate the training sets from sub-tasks into a single corpus of 170,420 entries, from which randomly select 120 entries for validation identify the optimal model.
MMLU, The MMLU (Massive Multitask Language Understanding) benchmark (Hendrycks al. 2021bJa) a) includes 57 diverse subjects spanning the humanities. STEM. social sciences, and more. Each subject comprises questions of varying difficulty, with multiple-choice answers provided.
5

---

Published as a conference paper at ICLR 2025
Table 3: Performance of LoRA and ComL on commonsense reasoning tasks. The Params
(L/S) column indicates the percentage of trainable parameters in LoRA component(s) (denoted by
L) and the LoRA selector (denoted by S). The best performance is in bold, and the second best
performance is underlined.
Method
Params (%) (L/S)
BoolQ
PIQA
SIQA
ARC.c
ARC-e
OBQA
HellaS
WinoG
Average
LoRA,
0.0901.00
65.72
73.67
73.49
63.31
78.70
72.80
80.06
75.93
72.96
LoRA
0.18/0.00
52.02
70.40
59.04
76.39
64.20
74.72
72.38
66.76
LoRA,s1e
0,35/0.00
63.66
77.64
71.65
71.93
87.42
73.60
82.06
73.40
75.42
LoRA,wa2
0.70A0.0
85.20
71.20
-125,,125
2.74/0.00
71.99
88.19
79.63
80.03
91.37
86.40
93.42
88,00
84,88
ComLaRAKg444
0.35/1.45
69.02
86.56
79.68
78.84
92.21
83,40
92.39
86.35
83.56
Coml -2--5-
0.35/1.45
75.14
90.04
82,09
82.42
93.14
89.00
96.03
88.87
87.09
ComLoRA
0.70/1.45
73:79
88.79
81.47
83.11
92.68
89.00
95.45
88.24
86.57
ComLoRA
2.74/1.39
71.53
87.65
80.12
87.40
86.50
85.02
Personalized Conversation Task. We assess model performance personalized conversational understanding using the CONVA12 dataset (Dinan et al., 2019 Zhang et al., 2018). This dataset designed to evaluate model's ability engage in meaningtul and coherent conversations while maintaining personalized dialogue. Following (Liu et al. 2020 Song et al.) 2021 Huang et al., 2023b a 2024),
self-persona configuration that reveals only the speaker's 's persona. The task
involves training the model to respond in way that reflects an understanding of personal preferences and contextual nuances.
4.3 EVALUATION METRICS
To evaluate performance on the commonsense reasoning datasets, we use accuracy as the primary metric, following the approach of (Hu et al., 2023). For each instance, the language models generate answers based on the provided queries, and specific keywords (c.g., "true" or "false" for BoolQ) are searched within the responses. The first occurrence of relevant keyword is recorded as the model's answer, while responses lacking relevant keywords are considered incorrect. A similar evaluation method is employed for MMLU.
For the CONVAI2 dataset, we assess linguis- tic similarity using BLEU (Papineni et al. 2002). METEOR (Banerjee & Lavie (2005), and ROUGE-L (R-L) (Lin) (2044), which cal- culate the overlap of n-grams between model predictions and ground truth. To measure se- mantic similarity,
Table 2: Performance comparison of LoRA and ComLoRA on MMLU. The best performance in bold, and the second best performance underlined.
Method
Params (L/S) (%)
Accuracy
ToRA,-1
0.0970.00
56.44
BERT Score (Zhang
LoRA,
0.18/0.00
56.79
LoRA,zie
0.35/0.00
55.25
et al. 2019), which evaluates the cosine similar- ity of normalized BERT embeddings between predictions and ground truth. We report BERT Score's 's F1 (BERTF1). Recall (BERTR). and Precision (BERTp).
LoRAp-s2
0,70/0.00
55.97
LoRA,=128
2.74/0.00
ComLoR MomLLoAAR=2,ee
RR
0.35/1.45
59.81
ComLoRAA K-4.r44
0.35/1.45
59.41
Coml oR K=1,r=8
0.70/1.45
59.14
ComLoRA ComLoRAKst,rss
2.74/1.39
61.09
4.4 RESULTS
Commonsense Reasoning. The results commonsense reasoning shown in Table 3 Among the LoRA configurations, LoRA,=12s achieves the highest average score of 84.88% indicating that higher rank enhances the model capacity to learn complex patterns in the data. In con- trast, the ComL framework shows strong performance across all tested configurations. No- tably, Comm k= which selects the winner LoRA with rank 8 for inference, achieves the highest average score of 87.09%. outperforming all LoRA baselines, including LoRA,=128- This sult underscores the effectiveness of ComLoRA's competitive learning mechanism, where multiple low-rank adapters compete during training, leading improved overall performance.
Moreover, ComLoRA configurations with lower ranks still outperform higher-rank LoRA. For ex- ample, ComLoRA ComLoRAx=4.rsz 4 achieves average of 83.56%, surpassing LoRA,=32 which has an average score 80.79%. This demonstrates that ComLoRA can achieve superior performance with fewer parameters compared conventional LoRA methods.

---

Published as a conference paper at ICLR 2025
Table 4: Performance comparison of LoRA and ComLoRA CONVAI2 Dataset. The best perfor-
mance is in bold, and the second best performance is underlined.
Method
Params (LJS) (%)
BLEU
METEOR
R-I.
BERT F1
BERT R
BERTP
Average
LoRA,
0.09/0.00
2.54
12.97
12.27
84.76
85,04
84,34
47.02
LoRAr
0.18/0.00
2.37
12.69
11.89
84.66
84.97
84.40
46.83
LoRA,-15
0.35/0.00
3.15
14.00
13.32
84.70
84.25
85.19
47.43
LoRA,-a2
0.70/0.00
3.27
14.05
84.41
47.79
L2RA
2.74/0.00
3.24
15.24
14.06
84.79
84.42
85.20
47.82
Coml LoRAAo
K-2--8
0.3571.45
3.69
17.05
16.09
85.23
84.74
85.75
48.76
ComLoRA
0.35/1 .45
3.67
16.99
15.98
85.22
84,73
85.74
48.72
ComLoRApe 8
0.70/1 45
3.65
16.90
15.95
85.19
84,70
85.72
48.69
ComLoRA Rm4.mm32
2.74/1.39
3.62
16.06
14.75
84.83
84.53
85.17
48.16
For individual tasks, 222 consistently outperforms different LoRA methods. In tasks such as PIQA and HellaSwag, ComLoRAK=2,- achieves remarkable scores of 90.04% and 96.03%, respectively, significantly higher than the corresponding performance LoRA,=12s This indicates that the competition among LoRA components in ComLoRA enable the model to learn from complex reasoning tasks.
MMLU. As shown in Table 2 ComLoRA demonstrates superior performance across various con- figurations. Notably, ComL A achieves the highest accuracy of 61.09%, surpassing all LoRA baselines, including LoRA,=128. which achieves 59.36%. ComLoRA second-best accuracy at 59.81 %, followed closely by ComLoRAk=4.=4 with 59.41%. These results highlight the effectiveness of ComLoRA's competitive learning approach, as it consis- tently outperforms traditional LoRA methods even with lower ranks per component. Additionally, ComLoRA, K = maintains strong performance, further demonstrating ComLoRA's robustness across various configurations.
secures the
CONVAI2. According to the results shown in Table configurateo configurations across multiple metries, including BLEU METEOR, ROUGE-L (denoted by R-L), and BERT scores (i.e., BERT F1, BERT R. and BERTp). Specifically, ComLoRAK=2,res achieves a BLEU score of 3.69, a METEOR score of 17.05, and a ROUGE-L score of 16.09, significantly higher than the corresponding scores from LoRA,=4 LoRA,=128- This demonstrates the effec- tiveness of competitive learning approach in ComLoRA to capturing diverse conversational patterns.
MaILoRRA==2rss outperforms all other
Additionally, ComLoRAk=1,=d secures the second-best performance across all metrics, main- taining high scores and further validating the effectiveness of the ComLoRA framework. The average scores reflect ComLoRA's ability to consistently outperform traditional LoRA, even with lower ranks, thereby offering more parameter-efficient alternative without sacrificing model per- formance.
5 ABLATION STUDIES
To thoroughly evaluate the effectiveness of the proposed ComLoRA framework, we conduct series of ablation studies on commonsense reasoning tasks. Unless otherwise specified, all experiments are performed using the LLaMA-3-8B model with K and r = 8 in ComLoRA.
5.1 ANALYSIS ON Loss AND LoRA COMPONENTS
To assess the effectiveness of ComLoRA, analyze its training losses when K = and r 4 and compare with LoRA,=4 and LoRA,=1e- As shown in Figure the training losses of all LoRA com- ponents in ComLoRA are consistently lower than those of LoRAs, despite each LoRA component having only a rank 4. Different LoRA components in ComLoRA intertwine, collectively forming lower losses through their competitive interaction during training. This results in that ComLoRA achieves a faster convergence compared to a single LoRA model under the standard fine-tuning process.
Figure plots the number of times being the winner among the four LoRA components for each LoRA component every 50 steps, where the winner at training step has the lowest training

---

Published as a conference paper at ICLR 2025
50
3.0
40
30
2.8
La tridd
Number
Cornt oRA mh-4)
ComLoRA
2.6
2.5
2.4
30
100
150
200
230
300
55
400
+50
500
600
8650
800
900
050
200
400
600
800
1000
Stops
(Goupee
Intervage
Steps
Figure 3: Number times being the winner
Figure 2: Loss Curves for LoRA and Com-
for each LoRA component inside ComLoRA every 50 training steps.
LoRA.
Table 5: Impact of removing one of the losses or selector from total loss.
Method
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
ComL 4.r-4
69.02
86.56
79.68
78.84
92.21
83.40
92.39
86.35
83.56
w/o LLM
67.31
84.82
76.97
77.99
91.08
79.80
91.23
80.82
81.25
w/o Lalign
67.46
79.16
75.28
69.80
81.65
78.20
83.07
80.90
76.94
w/o
Lpiirise
69.20
85.69
74.21
77.56
90.87
78.60
89.86
77.51
80.44
w/o
Selector
65,38
83.84
70.98
74.74
90.07
75.20
72.24
71.03
75,44
loss. Figure reveals how the LoRA component 0 gradually gains dominance through competitive training, ultimately being the LoRA winner ComLoRA.
5.2 EFFECT OF DIFFERENT LOSSES AND SELECTOR
To better understand the impact of each loss component and the selector in the ComLoRA frame- work, we conduct ablation studies by removing one component at time and observing its effect on performance across commonsense reasoning tasks.
According to the results presented Table 5 we can see that removing the language modeling loss fLM leads noticeable performance drop, with the average accuracy decreasing from 83.56% to 81 81.25% This indicates that the LM loss is essential for optimizing the model's predictions and driving ComLoRA's performance.
The alignment loss Lalign- which ensures that the LoRA selector's preferences influence the LM updates, has an even larger impact. Without this loss, the model's average performance decreases significantly to 76.94%, showing the critical role of aligning the selector's guidance with the LM performance.
Similarly, removing the pairwise loss Cairrises which fosters competition among LoRA components, also reduces performance, with the average accuracy falling 80.44%. This suggests that competi- tion between components is important for enhancing the model's 's performance.
Lastly, we also evaluate the impact of removing the selector entirely from the framework. Without the selector, the average accuracy drops drastically to 75.44%, indicating that the selector plays central role in ComLoRA's performance.
5.3 EFFECT OF THE NUMBER OF LORA COMPONENTS K
In this section, we investigate the impact of the number of LoRA components K to the performance of ComLoRA, while fixing the rank r to be 8. According to the results shown in Table 6 we observe that when K 1, which corresponds to the standard LoRA with rank 8, the average performance across tasks 66.76%. As we increase K, the performance improves significantly, reaching the highest average performance (i.e., 87.09 when K equals 2. This demonstrates that introducing competition among multiple LoRA components can substantially enhance the model capability even with small number of components.
8

---

Published as a conference paper at ICLR 2025
Table 6: Performance of ComLoRA with varying numbers of LoRA components K (with fixed rank
r = 8) across commonsense reasoning tasks.
K
BoolQ
PIQA
SIQA
ARC-e
ARC-e
OBQA
HellaS
WinoG
Average
52.02
70.40
64.94
59.04
76.39
64.20
74.72
72.38
66.76
75.14
90.04
82.09
82.42
93.14
89.00
96.03
88.87
87.09
73.06
86.29
80.40
81.57
93.10
87.60
94.75
87.61
85,55
71.65
84.33
79.79
79.01
88.80
86.20
92.55
85.71
83.51
8
5
70.64
85.80
80.30
81.91
92.34
85.80
94.12
85.79
84.59
68.56
84.87
79.27
77.39
91.46
80.00
93.19
83.82
82.32
7
67.13
84.87
78.10
79.35
90.07
82.60
92.17
84.21
82.31
70.86
87.11
81.06
80.38
92.09
83.60
93.46
87.45
84.50
Table Impact of top-N selection to task performance when varying N.
K
Top-N
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
74.19
88.47
80.86
82.76
92.13
86.80
94.81
88.87
86.11
72.14
88.19
82.65
82.42
92.93
88,20
95.27
87.37
86.15
8
3
73.49
89.39
80.81
83.28
92.85
87.80
95.38
87.06
86.26
4
74.53
89.01
82.55
83.53
93.06
88.80
95.86
89.34
87.08
Interestingly, the performance does not continue to improve with larger values of K beyond 2. While K = 3 and K still achieve good average performance (i.e., 85.55% and 84.59%), they are slightly lower than the peak performance at K 2. This suggests that having many competing LoRA components may introduce redundancy or interference affect the training of each LoRA component.
5.4 EFFECT OF N IN TOP-N SELECTION
We evaluate the impact of N in selecting the top-N LoRA components when defining the alignment loss in Eq. (5) According the results shown in Table 7 we can that increasing N gener- ally improves the performance, with N = 4 achieving the highest average accuracy of 87.08%. This indicates that allowing more LoRA components to update during training enhances the model performance. Moreover, training with top-2 or top-3 components also yields competitive results, providing balance between the computational efficiency and performance.
5.5 EFFECT OF ANNEALING STRATEGY
In this section, evaluate the impact of the annealing strategy used in the alignment loss (i.e., Eq- (5) to the performance of Coml A.R.. We compare four annealing strategies, including the constant, cosine, exponential, and linear annealing strategy. The four strategies defined
Constant: 17(1)
Cosine: n7(() 0.5
(13)
Exponential: 7tt) =1 exp (a- (TT )) Linear: 17(1)
t
(14)
T
where t denotes the index of the current training step and T denotes the number of total steps in the whole training process. According to the results presented in Table [ we can see that the ex- ponential annealing strategy yields the highest average performance, achieving an average accuracy of 86.57%, which indicates its effectiveness in dynamically adjusting the guidance of the LoRÁ
Table
Performance of different annealing strategies.
Annealing Strategy
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
Constant
70.21
87.43
79.84
80.12
91.58
84,00
92.83
85.79
83.98
Cosine
70.52
88.79
80.14
79.95
91.33
83.20
93.60
84,29
83,98
Exponential
73.79
88.79
81.47
83.11
92.68
89,00
95.45
88.24
86,57
Linear
74.31
87.60
80.60
81.06
91.71
86,60
94.50
86.58
85.37

---

Published as a conference paper at ICLR 2025
Table 9: Impact of varying noise intensity parameter in the pairwise loss task performance
Noise Intensity (a)
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
72.81
87.65
79.99
81.31
92.47
84.60
91.97
87.29
84.76
73.79
88.79
81.47
83.11
92.68
89.00
95.45
88.24
86.57
8
4
1.0
73.09
14
80.19
80.80
92.21
85,20
94.55
87.21
85.18
10.0
71.59
87.87
80.76
80.55
93.06
84.00
94.46
85.87
84.77
Table 10: Performance comparison of LoRA components (indexed from 3) in ComLoRA. The
LoRA winner determined by ComLoRA A for inference is highlighted as O.
LoRA Index
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Average
0 (Winner)
73.79
88.79
81.47
83.11
92.68
89.00
95.45
88.24
86.57
1
74.92
88.52
82.14
81.40
92.42
89.00
95.44
87.21
86.38
2
73.61
88.47
81.63
82.08
92.97
87.40
95.45
87.85
86.18
3
74.40
88.41
81.22
81.14
91.67
86.00
95.32
86.98
85.64
selector. The other three annealing strategies exhibit inferior average performance, making the the exponential annealing strategy a good and default choice in our experiments.
5.6 EFFECT OF NOISE INTENSITY
To evaluate the impact of noise intensity in the pairwise loss, we conduct an ablation study on the effect of the value of the noise intensity parameter a the performance. According to the results shown in T we can see that introducing moderate level of noise (i.e., a 0.1) yields the best overall performance across tasks, achieving an average accuracy 86.57% In contrast, removing the noise entirely (i.e., = 0.0) results in lower average accuracy of 84.76%, highlighting the importance controlled noise in improving the generalization. As a increases, the performance generally decreases, indicating that excessive noise (c.g., 10.0) hinders the model capacity.
5.7 EFFECTIVENESS OF THE LORA WINNER
In this section, examine whether the LoRA winner selected by the ComLoRA framework dur- ing training is indeed the best-performing one. Table 10 presents the performance of each LoRA component (indexed from 3) on commonsense reasoning tasks.
As shown in Table 10 the LoRA winner, indexed as 0, achieves the highest average accuracy of 86.57% across tasks. This confirms that the ComLoRA framework effectively selects the best- performing LoRA component during training. Those results underscore that ComLoRA's selection mechanism reliably identifies the most effective LoRA component, ensuring that the final model benefits from the best available adaptation to the task at hand.
CONCLUSIONS
In this work, we introduced the ComLoRA framework, a novel approach to enhance LoRA by in- tegrating competitive learning. ComLoRA enhances the efficiency and adaptability of LoRA by initializing multiple LoRA components that compete during training, with each component striving achieve better performance through competition. The proposed ComLoRA method significantly outperforms the conventional LoRA method across various tasks, while maintaining parameter ef- ficiency and avoiding computational overhead during inference. Our future work will explore the competitive learning paradigm to other fine-tuning methods.
ACKNOWLEDGEMENTS
This work is supported by National Key R&D Program of China 20222D0160300 and NSFC key grant under grant 62136005.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXivv2303.087777 2023.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Jade Goldstein, Alon Lavie. Chin-Yew Lin, and Clare Voss (eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Mea- sures for Machine Translation and/or Summarizatiin, pp. 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL WD5 0909
org/
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Christopher Clark, Kenton Lee. Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Techmologies, Volume (Long and Short Papers), pp. 2924-2933 Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1300. URL
Ming-Wei
https://aclanthologgs org/N19 1300
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1rXiv:1803.0544 2018.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Ur- banek, Douwe Kiela, Arthur D. Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan Black, Alexander I. Rudnicky, Jason Williams, Joelle Pineau, Mikhail S. Burtsev, and Jason We- ston. The second conversational intelligence challenge (convai2), arXiv:1902.0000 2019.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference Machine Learning, pp. 5547- 5569. PMLR, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1), jan 2022. ISSN 1532-4435.
Stephen Grossberg. Competitive learning: From interactive activation adaptive reso- nance. Cognitive Science, 11(1):23-63, 1987. ISSN 0364-0213. doi: https://doi.org//00
art atticLe/p11/30344213888
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034 2015.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- fional Conference on Learning Representations (ICLR), b021
11

---

Published as a conference paper at ICLR 2025
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0960 09688, 2021.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. I.L.-adapters An adapter family for parameter-efficient fine-tuning of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.): Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5254-5276, Singapore, December 2023, Association for Computational Linguistics, doi: (0.18663/w1/0223.oom main. 319. URL https::/aacaatthlleeee
org. wtt.::2...mmmmmmmmmmmmmmmm
319
Qiushi Huang. Shuai Fu, Xubo Liu, Wenwu Wang, Tom Ko, Yu Zhang, and Lilian Tang. Learning retrieval augmentation for personalized dialogue generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3523-2540. Singapore, December 2023a. Association for Computational Linguistics. doi: 11//////////222.mnllmmm-a 154. URL 2023. www.eennnpmmii.. 154
Qiushi Huang. Yu Zhang, Tom Ko, Xubo Liu, Bo Wu, Wenwu Wang, and H Tang. Personalized dialogue generation with persona-adaptive attention. Proceedings of the AAAI Conference on Artificial Intelligence,
Jun. 2023b. doi: 10. (0.1669/aaai.v. v37.11.265111 URL 26518
httts::/ojjs.att wtw.:/sssaaa.... org ndex. wwt.nndee..pppeeppeeeeee php/AAAT
Qiushi Huang, Xubo Liu, Tom Ko, Bo Wu, Wenwu Wang, Yu Zhang, and Lilian Tang. Selec- tive prompting tuning for personalized conversations with LLMss. In Lun-Wei Ku, Andre Mar- tins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 16212-16226, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 10. 1011186//112224.111 (r1224..ininngs--a acl. 459. URL 2024. findings- 959/]
org/
Damjan Kalajdzievski. rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03792, 2023.
Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High-rank training through low-rank updates. In International Conference on Learning Representations, 2023. BRLLhttps: / semant wwwppp..emtttttcccccccccc wttpttccll....gg//orooo....
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain. July 2004. Association for Computational Linguis- tics. URL http::/ss//aaaaahooolooo//////// w0w-:////00-00000
Qian Liu, Yihong Chen, Bei Chen, Tian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang- You impress me: Dialogue generation via mutual persona perception. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1417-1427, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/000...o www.:/0.....mmm..m..mmmmm main. URL
https://aclanthol// http://aaaa.aooolloooo////20.... maaiimmiin.. 131
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019. URL httpss//ppeneeeeeee.eeeee//// 2tw.://rrrrr....tteooommmmmaa
Bkg6RicqY7
Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-effiient fine-tuning for large lan- guage models. arXiv preprint arXiv:2402. 12851, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii (eds.). Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 2381-2391, Brussels, Belgium, October- November 2018 Association for Computational Linguistics. doi: (0.18653/v1/DIL 1260. URL https://acl antho hww.olaoooooooooooooooooeeoee... log.
12

---

Published as a conference paper at ICLR 2025
Steven Nowlan. Maximum likelihood competitive learning. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 2. Morgan- Kaufmaan,,
1989. URL https:/ www.://in.nn............... nwwrips. httprippp.ccppppepe cc/paper files/ htlees/paper1189/ 2tte://cccabcccc330ooccaaaaa2a23
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Association for Computational Linguistics, pp. 311-318, USA, 2002. Association for Computational L Linguisties. doi: 10.31 10.3115/10030033007773
URL https:
1073135
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
David E. Rumelhart and David Zipser. Feature discovery by competitive learning. Cognitive Science, 9(1):5--113 12, 1985. ISSN 0364-0213. doi: 80010-0. URL
$0364021385800100
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Common- sense reasoning about social interactions. arXiv preprint arXiv: 2019.
Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhan Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, Bowen Li, Saravanan Ganesh, Bill Byrne, Jessica Hoffmann, Hassan Mansoor, Wei Li, Abhinav Rastogi, and Lucas Dixon. Parameter efficient reinforcement learning from human feedback. 2024. URL https:
//api . www.eniisccooooo.oooo.....cccete
org/ CorppsIrppDsooccc 27265442622
Haoyu Song, Yan Wang. Kaiyan Zhang, Weinan Zhang, and Ting Liu. Bob: Bert over bert for training persona-bast based dialogue models from limited personalized data. In Association for Com- putational Linguistics, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv:1905.0788: 2019.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kicla, and Jason Weston. Personalizing dialogue agents: have a dog, do you have pets too? In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 2204-22- 2213, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10. 10.11866//ll/ 1/1/5//////11--// 1205. URL https: 7/aclanthology Ptg///8-11 1205
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv: (okiv:9904 09675, 2019.
13


Reviews:
[{"review_id": "k27MsAAwdk", "reviewer": "Reviewer_RpHx", "summary": "This paper presents the Competitive Low-Rank Adaptation (CoLoRA) method, aimed at enhancing the Low-Rank Adaptation (LoRA) framework for fine-tuning large language models (LLMs). Traditional LoRA models often face limitations: a single low-rank configuration lacks capacity, while higher ranks introduce inefficiency. CoLoRA attempts to address this by introducing multiple competing LoRA components and a dynamic selection mechanism to evaluate them during training, aiming to improve model adaptability without increasing inference complexity. The primary contributions claimed by the authors include establishing a competitive learning framework for LoRA, implementing a selector to optimize component selection, and demonstrating CoLoRA’s purported advantage over conventional LoRA in tasks without additional inference overhead. However, the practical significance of these contributions remains unclear, and it is debatable whether the competitive framework provides a tangible benefit over simpler LoRA configurations. Furthermore, the experiments, while suggesting performance gains, may lack sufficient depth in real-world application scenarios to substantiate CoLoRA’s effectiveness fully.", "questions": "1. Could the authors clarify how CoLoRA differentiates itself from other competitive or multi-component frameworks, such as those seen in recent work on MultiLoRA or m-LoRA? Specifically, how does CoLoRA’s competitive learning provide unique benefits over these methods, both in terms of architecture and practical outcomes?\n2. Could the authors provide more insights into how the selector’s decision-making is validated, especially under varying task complexities or noisy data environments? Additional results or qualitative analysis here could better illustrate the selector’s effectiveness and robustness.\n3. How would CoLoRA perform in more practical, multi-task scenarios? Testing in a realistic multi-task setting or domain-specific application (e.g., medical or legal language tasks) would better substantiate CoLoRA’s practical utility.\n4. Could the authors include or discuss comparisons with baselines like MultiLoRA and m-LoRA to ensure that CoLoRA’s claimed improvements are significant and unique?\n5. How do individual LoRA components contribute to overall performance in the CoLoRA framework? A more detailed ablation or component-level analysis could reveal how each competing LoRA component affects the model’s learning process and generalization.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "1. Dynamic Selector Mechanism: The dynamic selection mechanism is designed to assess the performance of each LoRA component, selecting the best-performing component dynamically. This theoretically enables a more targeted model adaptation, which may contribute to achieving a balance between efficiency and expressiveness.\n2. Extensive Experiments Across Benchmarks: The paper provides a variety of experimental results across benchmarks like commonsense reasoning and the MMLU, which serve to showcase the claimed advantages of CoLoRA. These evaluations attempt to establish CoLoRA’s capacity to perform well across a range of tasks, adding empirical evidence to support its potential robustness.\n3. Comparison with Established Methods: CoLoRA is compared not only with standard LoRA but also with mixtures of experts (MoE) methods. This situates CoLoRA within the broader context of parameter-efficient fine-tuning techniques, allowing readers to gauge its potential advantages and limitations relative to other established approaches.", "weaknesses": "1. Limited novelty in competitive learning mechanism: The competitive learning framework introduced here lacks substantial novelty, as similar multi-component strategies have been explored in recent works on parameter-efficient fine-tuning. For example, recent studies on multi-component or competitive mechanisms that optimize LoRA components across tasks potentially reduce the novelty of CoLoRA’s contribution [1, 2].\n2. Insufficient evaluation on practical multi-task scenarios: While the paper presents evaluations on commonsense reasoning and language understanding tasks, it lacks assessment in realistic, multi-task settings. The effectiveness of CoLoRA’s competitive learning could be better demonstrated through domain-specific applications or real-world multi-task benchmarks, as seen in recent research [3]. Adding experiments in diverse, practical applications would strengthen the paper’s claim of broad applicability.\n3. Overlooked baseline comparisons: Although the paper compares CoLoRA against standard LoRA and certain MoE models, it does not include several relevant baselines that implement advanced multi-component LoRA optimizations, as presented in recent studies [2, 3]. Including these baselines would provide a more balanced evaluation, helping to demonstrate if CoLoRA truly offers unique advantages or whether existing methods meet or exceed its performance.\n4. Lack of detailed analysis on selector mechanism: The dynamic selector is a core component of CoLoRA, yet there is minimal analysis on its decision-making process or robustness across task complexities. A deeper exploration of how the selector operates in various scenarios, along with sensitivity testing, would add value and practical insight to the claims of its efficacy. Such analysis could also improve the model’s interpretability and assist in refining the selector’s parameters for better performance across diverse tasks.\n\nRef:\n[1] Q. Liu, X. Wu, X. Zhao, Y. Zhu, D. Xu, F. Tian, and Y. Zheng, \"When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications.\"\n[2] Y. Wang, Y. Lin, X. Zeng, and G. Zhang, \"MultiLoRA: Democratizing LoRA for Better Multi-Task Learning.\"\n[3] Z. Ye, D. Li, J. Tian, T. Lan, Y. Liang, Y. Jiang, J. Zuo, H. Lu, L. Duan, and M. Tang, \"m-LoRA: Efficient LLM Model Fine-tune and Inference via Multi-LoRA Optimization.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AzjBXUNggi", "reviewer": "Reviewer_CMrQ", "summary": "This manuscript introduces CoLoRA, a competitive learning framework designed to improve the expressiveness and effectiveness of LoRA (Low-Rank Adaptation) in fine-tuning large language models. CoLoRA deploys multiple low-rank components that compete during training, with the most effective component selected for inference. The authors claim that CoLoRA outperforms traditional LoRA configurations on commonsense reasoning and multitask language understanding tasks while maintaining inference efficiency.", "questions": "1.\tHow does the training overhead of CoLoRA compare to LoRA, MoELoRA, and full fine-tuning? Quantitative data on training time, resource usage, and convergence rates would clarify the practical feasibility of CoLoRA.\n\n2.\tWhy is there no direct comparison between CoLoRA and MoELoRA in terms of inference efficiency? Can you provide specific data on inference speed, parameter load, and memory consumption?\n\n3.\tHow does the competitive mechanism benefit LoRA, and are there unique modifications (e.g., loss functions, selection criteria) that make this approach innovative? Could you provide a theoretical justification for its use?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Attempt to Address LoRA's Limitations: CoLoRA aims to improve LoRA by adding competitive learning, which, in theory, can improve model adaptability without increasing inference costs.\n2. Empirical Results on Language Tasks: The experiments show that CoLoRA performs better than baseline LoRA configurations on a range of language tasks, which is promising in terms of model capability within this domain.", "weaknesses": "1. Training Overhead Not Addressed: A core objective of LoRA is to reduce training and computational costs. By introducing multiple components and a competitive selection mechanism, CoLoRA inherently increases the complexity and computational demand during training. The paper lacks any quantitative comparison of training overhead relative to LoRA, MoELoRA, or full fine-tuning (FFT), making it difficult to assess CoLoRA’s practicality in real-world scenarios. Without a thorough comparison of training times, convergence rates, and hardware requirements, the claimed efficiency is unsubstantiated.\n\n2. Lack of Quantitative Comparison with MoELoRA in Inference Efficiency: The paper claims that CoLoRA achieves higher inference efficiency than MoELoRA by selecting only a single component for inference. However, the authors do not provide any quantitative evidence to substantiate this claim. Given that both frameworks target parameter efficiency, a direct comparison in terms of inference time, memory usage, and computational cost is necessary to validate CoLoRA’s purported advantage.\n\n3. Weak Theoretical Justification for the Competitive Mechanism: The paper lacks a rigorous theoretical foundation explaining why competitive learning should significantly enhance LoRA performance. There is no analysis of how the competitive mechanism improves selection or how it specifically addresses LoRA's limitations. Without a theoretical framework", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sjijiILCQ7", "reviewer": "Reviewer_Kd2W", "summary": "This paper introduces CoLoRA, an extension of the LoRA approach that uses ideas from mixture of expert models. Rather than train a single LoRA component the CoLoRA approach trains multiple LoRA components, allowing them to compete during training. The technique is clearly described and a series of experiments are performed to compare the performance of models trained using CoLoRA with models trained using LoRA. These evaluation experiments show that the models trained using CoLoRA outperform those trained using LoRA alone. A series of ablation studies also show the value of different parts of the approach.", "questions": "As well as addressing the weaknesses described above the authors might also consider:\n\n- \"The CoLoRA’s code will be released later.\" Why not release the code now? \n- Fonts in Figure 1 are much too small making it very difficult to follow.\n- Table 1 is not easy to follow. Perhaps Low, Medium, High could be used instead of the current symbols? \n- The unit (accuracy, f-score, ...) being used in Table 2 and other results tables is never stated. It should be.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The main strengths of the paper are:\n\n- The CoLoRA apporach is a novel and effective extension of LoRA. \n\n- The CoLoRA approach is clearly explained and includes interesting ideas well grounded in the literature. \n\n- The experiments are carefully designed and properly executed. \n\n- The ablation studies are comprehensive. \n\n- Models trained using the CoLoRA approach clearly out-perform models trained using standard LoRA.", "weaknesses": "The main weaknesses of the paper are:\n\n- Although computational efficiency is claimed as an advantage of CoLoRA no evaluation of computational requirements is included. The paper would really benefit from this. \n\n- The performance measures used in Tables 2-10 are never stated. This would really help. \n\n- The paper would benefit from some more careful review and revision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lwpqE5N72q", "reviewer": "Reviewer_DKRc", "summary": "The paper proposes a competitive learning approach for training LoRA. Multiple LoRAs are initialized in parallel. A selector model is trained to select the most suitable LoRA for a given input. \nDuring training, the LoRAs and selector are trained jointly using the following losses:\n1) the standard LM loss is applied for each LoRA. \n2) the top-N selected LoRAs will get an additional loss gradient update. This is similar to weighing the LoRAs LM loss. The authors refer to this as an alignment loss.\n3) The selector is trained using a pairwise loss to ensure that it selects LoRAs with the best LM loss. \n\nOnce training finishes, the model is evaluated on a validation dataset. The LoRA that is selected the most is then chosen as the LoRA to be used for production/downstream tasks.", "questions": "- How does your approach compare with MoE and LoRA works discussed in the Weaknesses section?\n- What is the time complexity for equation 4? Do we have to do multiple forward passes through the entire model to get each LoRA's LM loss?  \n- How does your approach compare to the naive approach of fine-tuning multiple vanilla LoRAs (i.e., have multiple individual training runs)? \n- In Table 6, as the number of LoRAs increases the performance does not continue to improve. Can the authors provide insights as to why a selector model is needed. Would using a stronger selector (i.e., with more parameters) improve performance?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The competitive learning approach to train LoRA is quite interesting. The authors show that it consistently outperforms vanilla LoRA on a variety of tasks.", "weaknesses": "- Given that the main contribution is training a more performant LoRA, the paper does not compare with MoE approaches or newer LoRA varints (eg., AdaLoRA [1], DoRA [2]). In particular, the LoRA variants have to be compared given their simpler training complexity and similar inference overhead. \n- In equation 4, the LM loss is calculated individually for each LoRA. A naive approach or simple baseline that the authors can consider is having multiple individual training runs. We would also be able to get multiple LoRAs and choose the best performing one using the validation set.\n- I am not sure how useful the selector is given the lack of analysis on what the selector is doing. Unlike MoE, there is no specialization or ensembling being carried out. Since the selector is trained to select the best LoRAs, there might be a risk of certain LoRAs being undertrained. In addition, performance does not scale when more LoRAs are used (Table 6). \n- Only using the winner LoRA and discarding the other LoRAs feels kind of wasteful. The authors can consider merging the LoRAs together in a linear/non-linear combination based on the performance on the validation dataset.\n\n[1] Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., ... & Zhao, T. (2023). AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512.\n[2] Liu, S. Y., Wang, C. Y., Yin, H., Molchanov, P., Wang, Y. C. F., Cheng, K. T., & Chen, M. H. (2024). Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)