AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping
Authors: Haonan Dong
Wenhao Zhu
Guojie Song
Liang Wang
Venue: NeurIPS 2025 spotlight ()
Keywords: Parameter-Efficient Fine-Tuning
LoRA
LLM
Primary area: deep_learning
TL;DR: We present a novel approach that addresses the low-rank bottleneck in LoRA by integrating nonlinear mappings with compressed rank, achieving an optimal balance between parameter efficiency and model performance.

Abstract: Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method validated across NLP and CV domains. However, LoRA faces an inherent low-rank bottleneck: narrowing its performance gap with full fine-tuning requires increasing the rank of its parameter matrix, resulting in significant parameter overhead. Recent linear LoRA variants have attempted to enhance expressiveness by introducing additional linear mappings; however, their composition remains inherently linear and fails to fundamentally improve LoRA’s representational capacity. To address this limitation, we propose \ourmethod, which incorporates an Adaptive Nonlinear Layer (ANL) between two linear projectors to capture \emph{fixed} and \emph{learnable} nonlinearities. This combination forms an {\fontfamily{lmtt}\selectfont \textbf{MLP-like structure}} with a compressed rank, enabling flexible and precise approximation of diverse target functions while theoretically guaranteeing lower approximation errors and bounded gradients. Extensive experiments on 22 datasets and 6 pretrained models demonstrate that \ourmethod: (\textbf{I}) not only matches or surpasses full fine-tuning performance with only $6.18\%\sim25\%$ of LoRA’s parameters but also (\textbf{II}) outperforms state-of-the-art PEFT methods by up to $10.88\%$ in both NLP and CV tasks, and \textbf{(III)} exhibits robust performance across various rank configurations.

AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping
Haonan Dong', Wenhao Zhu', Guojie Song"), Liang Wang? State Key Laboratory of General Artificial Intelligence,
School of Intelligence Science and Technology. Peking University, 2 Alibaba Group, Corresponding author hndong25dstu, p.u..puuuu edu. en, gjsongupku. edu.cm
Abstract
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method validated across NLP and CV domains. However, LoRA faces an inherent low-rank bottleneck: narrowing its performance gap with full fine- tuning requires increasing the rank of its parameter matrix, resulting in significant parameter overhead. Recent linear LoRA variants have attempted enhance ex- pressiveness by introducing additional linear mappings; however, their composition remains inherently linear and fails to fundamentally improve LoRA's representa- tional capacity. To address this limitation, we propose AuroRA, which incorporates an Adaptive Nonlinear Layer (ANL) between two linear projectors to capture fixed and learnable nonlincarities. This combination forms an MLP-like structure with a compressed rank, enabling flexible and precise approximation of diverse target functions while theoretically guaranteeing lower approximation errors and bounded gradients. Extensive experiments on 22 datasets and 6 pretrained models demonstrate that AuroRA: (I) not only matches surpasses full fine-tuning perfor- mance with only 6. 18% ~ 25% of LoRA's parameters but also (II) outperforms NLP and CV tasks, and (III) exhibits robust performance across various rank configurations.
competitive PEFT methods up 10.88% both
Introduction
In recent years, pretrained models have demonstrated excellent generalization per- formance across numerous tasks in vari- ous domains 2, 3, 5, 6]. In practical applications, to further unleash their pow- erful capabilities on specific downstream tasks, these models often require relevant fine-tuning [7. 9, 10, 11, 12]. How- ever, the increasing size of their parame- ters poses significant challenge to fine- tuning all parameters [13, 14]. To address this issue, the field of Parameter-Ffficient Fine-Tuning (PEFT) has made substantial progress [13, 15, 16, 17, 18, 19, 20J. The core idea is to fine-tune only small subset of the model's parameters while freezing the majority of the pretrained parameters, achieving performance comparable to full fine-tuning [21].
NLU on CoLA with ReBERTe
Image Classification on DTD with VIT
45
79
75
Performunce
Derfamunce
76
61
LoRA
75
60-
AuroRA (Oun)
74
Fine-Turing
DoRA
AuroRA (Oum)
LoRA--dop
Fine Fourierf
57
71
0.075M
0.3M
12:M
74K
239K
S8IK
85.9M
Peramsters
Porsmeters
Figure The trade-off between parameters and perfor- mance of various fine-tuning methods on NLP (left) and CV (right) tasks. (Left) NLU, RoBERTa-Base is fine- tuned COLA, with LoRA ranks {2, 3, 4. 6, 8). (Right) In image classification, ViT-Base is fine-tuned DTD, with LoRA ranks {2, 4, 6, 8, 12, 16).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LoRA is commonly used offte-of-the- art PEFT method [16]. Specifically, it as- sumes that the weight updates conform to a low-rank hypothesis and represents these updates using two low-rank matrices, i.e., Wo + AW = Wo- - BA. Its performance has been validated in fields such as natural language processing (NLP) [16, 22] and computer vision (CV) [23, 24]. Despite its significant success, LoRA still faces an in- herent limitation, namely the low-rank bot- tleneck, as illustrated Figure 1. As the rank of LoRA increases, the model's per- formance improves, thereby narrowing the gap with full fine-tuning [25]; however, the parameter cost grows proportionally with the rank, which weakens its parameter ef- ficiency. This dilemma leads us to the first research question: Can we achieve a further balance between parameters and performance?
Ressscas
Aundik (own))
Lokin
VeSLaRA
Epploratio
Pemott (ours)
Paptarofion
Lobk
ModLoRA
Figure 2: We evaluate LoRA, MoSLoRA, and our AuroRA on DTD and RESISC45 datasets, employing ViT-Base with a rank of 2. (Upper) We record the AW at the 2, 9}-th epochs, and perform PCA visualization these AW. We observe that AuroRA is capable of exploring a broader parameter space. (Lower) We present the accuracy results on both datasets.
Recently, several linear LoRA variants have emerged [26, 27, 28, 29]. They introduce an additional matrix between the B and A matrices of LoRA to weaken the correlation constraints between them, thereby enhancing LoRA's learning and expressive capabilities. Specifically, one approach involves the introduction a diagonal matrix facilitate singular value decomposition [26, 27], while another approach incorporates an arbitrary matrix to fuse subspaces [28, 29J. Nevertheless, LoRA's inherent linearity persists even when additional matrix is introduced, preserving its fundamental structure as a linear mapping. As illustrated in Figure 2, when the rank is extremely low, MoSLoRA [28] (a linear variant that incorporates an arbitrary matrix) has only marginal effect on expanding the exploration of AW, leading to failure to further boost performance (2.2% 1 on DTD and 0.56% + on RESISC45). The structural characteristics and resultant performance limitations of linear variants naturally prompt our second research question: @ Can we achieve more than marginal performance improvements by introducing a nonlinear transformation between LoRA's two linear layers?
Motivated by the above two research questions, this paper focuses on introducing nonlinear mappings into LoRA and further compressing the rank achieve better balance between parameters and performance. To this end, we propose a method called Activate Your Low-Rank Adaptation (AuroRA). We revisit LoRA through the lens of linear mappings and identify two critical limitations: (I) insufficient expressiveness and (II) limited training fiexibility. To fully harness LoRA's potential, AuroRA introduces an Adaptive Nonlinear Layer (ANL) between the low-rank matrices, forming an MLP-like structure. ANL employs a hybrid design of fixed and learnable nonlincarities to enhance model expressivity within more compressed rank while enabling flexible training strategies to expand the explorable parameter space (Figures and 2). Theoretical analysis demonstrates that AuroRA not only achieves a strictly lower approximation error than LoRA but also preserves bounded gradient norms. Experiments across NLP and CV tasks confirm the efficiency, generalizability and robustness of AuroRA. We further conduct ablation studies to dissect the contributions of fixed and learnable components, and evaluate its robustness against linear LoRA variants across multiple rank configurations. Our contributions can be summarized as follows:
Perspective Shift. We systematically revisit two research lines of LoRA: the low-rank bottleneck and linear LoRA variants. By interpreting LoRA through the lens of linear mappings, we address both research questions within a unified framework, providing theoretical analyses.
Nonlinear Proposal. We propose AuroRA, which introduces nonlinear mappings into LoRA and further compresses the rank, resulting in superior balance hetween parameters and performance, paving the way for further unlocking the significant potential of LoRA.
0 Experimental Validation. Extensive experiments on 22 datasets and 6 pretrained models showcase that AuroRA: (I) not only matches or surpasses full fine-tuning performance with only 6. 6.18% ~ 25% of LoRA's parameters but also (II) outperforms competitive PEFT methods by up 10.88% in NLP and CV tasks, and (III exhibits robust performance across various rank configurations.

---

LoRA
AuroRA (Ours)
R
h E R
Food notiherrity
coerse Rrting
Pretrained- Weights We f dee
Pretrained ned- Weights
Mi geaadde
firting
A grsd.
A g
Rd
Linear Mapping (dim
MLP-iIke struchure (dim
00
Figure 3: A general comparison of LoRA and AuroRA. (Left) In LoRA, matrices A and B act as two linear projectors, forming a two-layer linear mapping with hidden dimension r. (Right) Our AuroRA extends LoRA by incorporating an adaptive nonlinear layer comprising fixed and learnable nonlinearities, forming an MLP-like structure with significantly reduced hidden dimension F (7 <<> r).
2 Methodology
As illustrated in Figure 3, we introduce AuroRA, an extension of LoRA that incorporates nonlinear mappings to overcome the inherent low-rank bottleneck. We reinterpret LoRA two-layer linear mapping, whereas our proposed AuroRA transforms it into an MLP-like structure by introducing an adaptive nonlinear layer.
2.1 LoRA: A Two-Layer Linear Mapping
In standard LoRA [16], the weight update AW for a pre-trained weight matrix Wo is approximated as the product of two low-rank matrices:
AW= = BA,
(1)
where A Rexda and B E Rduxx, with the rank satisfying r & min(din doa)). The forward propagation for input vector E Rda is thus expressed
h = Wax + AWX W6x + BAx.
(2)
The above process can be interpreted as a two-layer linear mapping, where A serves as downward projector P down that maps the input from high-dimensional space dh lower-dimensional hidden space R", and B serves as an upward projector Pup that maps back to IR dae However, we note that LoRA constrained by its sequential linear mapping structure, leading to two significant shortcomings: O insufficient expressiveness: being purely linear structure, it requires increasing the hidden dimension to handle more complex incremental weights and improve performance: e limited training flexibility: the direct low-rank decomposition induces strong interdependencies between the linear layers, imposing rigid structural constraints that reduce training flexibility [30].
2.2 AuroRA: An MLP-like Structure
To address these limitations, AuroRA introduces an Adaptive Nonlinear Layer (ANI.) between and B. modifying the weight update as follows:
AW= = B B(),
(3)
where a is the element-wise ANL that maps from RD to R.. Here, F denotes the compressed hidden dimension (F & r), i.e., the low dimension which the input is projected by Pawwn. Formally, the forward propagation equation in the training phase is given by:
Wox + (Ax).
The introduction of ANL enables AuroRA to form an MLP (Multilayer Perceptron)-lis structure.
2.3 Adaptive Nonlinear Layer
Consider an arbitrary input vector z. After projecting z into an 7-dimensional hidden layer, our objective is to introduce sufficient nonlinearity to capture many complex relationships possible
3

---

within this limited hidden space. To achieve this, we propose the following components: fixed nonlinearity (F), which utilizes parameter- nonlinear activation functions to activate neurons in the hidden space, thereby achieving coarse fütting; and @ learnable nonlinearity (C), which employs parameterized nonlinear functions during the training process of the weight update increments, facilitating, fine fitting. By combining ! and @ the Adaptive Nonlinear Layer (ANL) can be formally expressed as:
(5) where F represents the fixed nonlinear activation, L denotes the learnable nonlinear function, and Z denotes the input to ANL. We provide detailed comparison of fixed and learnable nonlinearity in Section 3.4.
(Z) F(Z) + (Z)),
For O, we adopt widely used activation functions in deep learning, such as ReLU [31], sigmoid, and tanh. A detailed comparison of different activation functions and their impact on AuroRA's s performance is provided in Section 3.4. Through our comparative evaluations, tanh emerges as the top-performing activation function, and theoretical analysis concurrently ensures its training stability. This preference is consistent with empirical findings in prior studies [32, 33] that demonstrate the robust performance of the tanh activation function for large-scale models, leading us employ tanh in our implementation. The depth of the network influences the number of activation functions that can be introduced. Specifically, we introduce self-projection Puerr E R between Pauwn and Pup, which extends the depth of the standard LoRA structure. Subsequently, we introduce tanh activation functions between P down and Psetf and between Pset and Pap. Formally, the fixed nonlinear component is defined as:
F(Z) = tanh (H (tanh(Z)))
where E F denotes Psetr-
To achieve @, we propose using spline functions to model complex relationships [34]. Numerous prior studies [35, 36, 37, 38] have demonstrated that splines flexible. piecewise polynomial functions capable of approximating a wide range of nonlinear behaviors. Specifically, we employ B-spline basis functions construct the learnable component. Formally, the learnable nonlinear component is defined as:
C(Z) w. w. sZ)),
where W, E IR' is the spline weight vector, and s( Σ B(z;) represents the spline basis functions applied to each dimension of Z. The learnable parameters in this component are the spline weights Wa. which determine the contribution of each basis function (zx)) to the overall output of C(Z). During training, these weights are iteratively updated minimize the task-specific loss function.
By introducing and @ in the hidden layer with dimension F, ANL effectively captures complex relationships without significantly increasing the number of additional parameters. The combination of coarse fitting and fine fitting enhances the standard LoRA structure, improving its expressive capac- ity and training flexibility, achieving what we refer to as Activate Your Low-Rank Adaptation The complete Adaptive Nonlinear Layer (ANL) developed in our work can then be formally repre- sented as:
o (Z) tanh (H (tanh(Z))) + s(Z).
(8)
Further details and the complete algorithmic workflow of AuroRA are provided in Appendix B.
2.4 Theoretical Analysis
In this subsection, we propose two theoretical propositions concerning AuroRA and analyze its parameter and computational cost. Additionally, we present an intuitive case in Appendix C to help better understand the role of nonlinearities.
Proposition 2.1 (Lower Approximation Error). Let M E Redoo din with rank(M) > Define
(MM UERdoa Xv, Xe
inf
|| || AM-UU -VV.
Then (M) > O, and for our proposed update of the form
Muoulinear (x) (Ax)),, Ae Rexd. B Rod Xr
where a is our adaptive nonlinear layer, there exists a parameter set (A", B², such that M Mnoulinesr/
S cE,(M), 0<cs1.

---

Hence, the approximation error is strictly below the linear rank-r limit E, (M), using the same rank
r.
Proposition 2.1 indicates that, thanks to the introduction of nonlinear mappings, AuroRA achieves strictly lower approximation error compared to LoRA at the same rank, meaning that the resulting weight updates are closer to the optimal solution. Furthermore, our empirical results demonstrate that this improvement persists even when further compressing the hidden dimensions of AuroRA. A rigorous proof, along with technical details and error bounds, is provided in Appendix D.
Proposition 2.2 (Gradient Boundedness). In the AuroRA, the use of the tanh activation function and B- B-spline basis functions results in bounded gradients with respect to both the inputs and the model parameters.
Proposition 2.2 posits that, despite the introduction of fixed and learnable nonlinearities, AuroRA maintains bounded gradients during training, thereby ensuring training stability. The corresponding proof is provided Appendix E.
Parameter Cost In Section I. we discussed the relationship between trainable parameters and rank in LoRA, where the number of introduced trainable parameters is (r + dour)). Here, dn and doa represent the input and output dimensions, respectively, i.e., PARAMS x In AuroRA, we aim to further compress the parameter count by setting the hidden layer dimension to = r//k where k is a constant and is the optimal rank setting of LoRA. This means that the number of trainable parameters in AuroRA is 1/k of that in LoRA. In this work, we set 7 to 2, corresponding to values of k such as and 8. The additional parameters introduced in ANL are of the order O O(2r²), ). which, compared to the significant reduction parameter count, can be considered negligible.
Computational Cost The computational complexity of AuroRA's forward pass in the training phase, Ah = Bo (Ax), is analyzed as follows. Let 6 denote the batch size, din and dout the input/output feature dimensions, the rank, and G' the collective B-spline parameters (a small constant, G (r))). The linear projections by A E xR' and B E do incur complexities of O(bden r) and Obrroua), respectively. The intermediate fixed and learnable non-linearities, (-), each contribute an additional ((rr²) term (with the learnable component's 's ObbrG) complexity simplifying due to G (r))). Consequently, the total complexity for AuroRA Ob((inn r + 2n² + rroot)). Given the standard low-rank setting where r < min(dax dout), the quadratic overhead O(br²) introduced by the -ineeiirssss negligible compared to the dominant linear terms, thus maintaining a computational footprint comparable to that of LoRA.
Experiments
In this section, we conduct extensive experiments to answer the following research questions: (R Q1) Can AuroRA effectively achieve efficiency in NLP tasks? (RQ2) Can AuroRA effectively achieve efficiency in CV tasks? (3) What are the respective roles of fixed and learnable nonlinearity? (RQ4) How do different activation functions in fixed nonlinearity affect performance? (RQ5) How does AuroRA's sensitivity to rank compare to that of linear LoRA variants?
3.1 Experimental Setup
3.1.1 Datasets and Pre-Trained Models
Datasets For our experiments, we evaluate the ability of AuroRA achieve parameter-efficient fine-tuning using four categories of datasets spanning both NLP and CV domains: Natural Language Understanding: We employ GLUE (General Language Understanding Evaluation) [39], a widely used multi-task benchmark in NLU, which includes datasets such as SST-2, MRPC,
CoLA, QNLI, RTE, and STS-B. The evaluation metrics are as follows: CoLA is assessed using Matthew's correlation coefficient, STS-B with Pearson's correlation coefficient, and accuracy is used for the other tasks. Commonsense Reasoning: We use a collection of commonly used datasets, including BooiQ [40]. PIQA [41], SocialIQA [42], HellaSwag [43], WinoGrande [44], ARC-e, ARC-c [45], and OpenBookQA [46]. For fair comparison, we follow the setup proposed by [28], fine-tuning the pretrained models on the Commonsense 70K dataset, which serves as a mixture of the aforementioned benchmark datasets. We then evaluate using accuracy as the performance metric. Image Classification: We use five datasets with small label OxforrtOxf [47], 10 [48], DTD [49], EuroSAT [50], and RESISC45 [51], and three datasets with large label
'The source code is available at here.
5

---

Table 1: We report the performance of different fine-tuning methods six datasets of the GLUB benchmark, using RoBERTa- Base and RoBERTa-Large models. For CoLA, we report the Matthew's s Correlation Coefficient (MCC); for STS-B, report the Pearson Correlation Coefficient (PCC); and for all other tasks, we report accuracy (Acc.). The reported results are the medians of five runs, each using different random seed. indicates numbers published in prior works. The best results are highlighted bold, and the runners-up are underlined. For all six datasets, higher values are
considered better for all metrics.
Model
Method
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Avg.
Params.
Full Fine-Tuning
94.8
90.2
63.6
92.8
78.7
91.2
85.2
125M
BitFitt
93.7
92.7
62.0
91.8,10
81.5p2.8
90.8
85.4
0.IM
RoBERTa Base
A
94.7.0.1
88.4
6..6.1.0
93.0
15.9
90.3 90.3,n.9
84.2
0.9M
LoRA*
95.1.0.3
19.7
63.4.0.2
93.3
78.4
91.3
85.2
AdaLoRA*
94.5,0.3
88.7
62.0.1.6
93.1
81.0y2.3
90.5
85.0
0.3M
DyLoRA*
94.3,0.5
89.5
61.1
92.2
78.7
91.1
84.5
0.3M
FourierFT
94.2,0.4
90.0
63.8-0.2
92.2.00.6
79.1 1o.4
90.8.0.4
85.0 jo.2
0.024M
LoRA-dropt
p0.3
19.5
62.9 62.9.o.7 Jo.7
13.1
81.4
91.0,0.2
85.4
0.15M
DoRA*
95.0
89.7
64.9-1.5
f2.9
79.2
91.3
85.5 10.3
0.3M
AuroRA
95.2-a.4
91,9
65.1
93.4
85,2
91.5
87.1
0.075M
RoBERTa- Large
Full Fine-Tuning*
96.4
90.9
68.0
94.7
86.6
92.4
88.2
356M
Adapter"
96.1.0.3
90.2
68.3ro.3
94.8
83.8
92.1
87.6.0.6
3M
Adapter"*
96.2,a.2
88.7
66.5
94.7
83.4
91.0,1.2
86.8
6M
LoRA"
96.2. ,0.2
90.2
68.2
94.8
85.2
92.3 jn.1
87.8,o.4
0.8M
FourierFT
96.0, .o.4
90.9
67.1.a.9
94.4
87.4fo.s
91.9 .n.5
88.0
0.048M
AuroRA
96.6-92
91.2
69.2
95.0
89.9
92.5
89.1
0.2M
Table 2: Commonsense reasoning evaluation results for LLaMA3-SB on eight tasks. * indicates numbers taken from [57]. The best results are highlighted in bold, and the runners-up are underlined. For all eight tasks, higher values are considered better.
Method
Params.
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Avg.
LoRA"
56.6M
70.8
85.2
79.9
91.7
84.3
84.2
71.2
79.0
80.8
PISSA*
83.8M
67.1
81.1
77.2
83.618.1
78.9.s.4
77.7
63.2. 5
74.6
75.4
MiLoRA
56.6M
68.8
86.
77.2
92.9n
85.6.1.
86.8.2.
75.5-13
81.8
81.
AuroRA
3.5M
72.5-17
87.4
79.0,a9
94.2
83.0
89.3
78.8
84.8
83.6
Subject-Driven Generation: Following [54], we use the DreamBooth dataset. More detailed descriptions of the datasets can be found in Appendix F.1, F22 F.3.
spaces, namely StanfordCars [52], [GVC and CIFAR-100 [48].
Tre-Trained Models We focus on selection of representative pretrained models, including RoBERTa (Base & Large) [1], LLAMA3-8B [3], ViT (Base & Large) [55] and SDXL [56].
3.1.2 Baselines
In the baseline evaluation, we adopt range of representative and competitive fine-tuning methods, categorized into three groups: Full Fine-Tunnin, PEFT methods, and LoRA variants. The PEFT methods we use include BitFit [20], Adapter# [32], Adapter [58], Adapter [59], and LoRA [16].
For the LoRA variants, consider AdaLoRA [26], DyLoRA [60], FourierFT [61], LoRA-drop [62],
DoRA [63], MoSLoRA [28], PiSSA [64] and MiLoRA [57].
3.2 AuroRA Achieves Efficiency in NLP Tasks Q1)
To answer RQI, we design two tasks: Natural Language Understanding (NLU) and Commonsense Reasoning, In the NLU task, we select RoBERTa-Baa and RoBERTa-Large [1] as pretrained models and compare AuroRA with ten other widely-used fine-tuning methods across all six datasets of the GLUE benchmark [39]. The results of this extensive comparison are shown in Table 1, with additional hyperparameter configuration details provided Appendix G.I. Following [16]. we fine-tune only the query and value weights of each transformer block, while fully fine-tuning the classification head. For the commonsense reasoning task, we select LLaMA3-8B [3] as the base model and compare AuroRA with LoRA and two other LoRA variants (PiSSA [64] and MiLoRA [57]). The results are shown Table The relevant hyperparameters are listed in Appendix G.2. Our observations can be summarized as follows:

---

Table 3: Fine-tuning results with ViT Base and Large models on different image classification datasets. We report the accuracy (%) after 10 epochs. Avg. represents the average accuracy across all datasets for each method. * indicates numbers taken from [61]. The best results are highlighted in bold, and the runners-up are underlined (excluding full fine-tuning).
Model
Method
Farauss.
OxfordPets
StanfordCars
CIFARIO
DED
EaroSAT
FGVC
RESISC45
CIFARION
Avg.
Full Fine Tuning"
85.8
93.1
79.8
98.9
77.7
99.1
54.8
96.1
92.4
86.5
VIt-Basc
Linear Probung"
g0.3
25.8,14.0 $4.0
25.8
5877
174
26
(8.4
LoRA"
581K
9.2
45.4.34.
98.8
75.0
98.
25.2
92.7.5.4
92.0.0.
77,553
FourisiFTy
239K
93.
28.
98.8
94.3
21.5
80.3
AurolA
74K
78.7
98.8 p.
79.6.1
988
48.2 44
93.6,15
v2.0
85
Full Fine-Tuning"
503 3M
94.4
88.9
09.2
81.8
99.0
68.3
96.4
93.6
90.2
VTT- VT--Lagge
Linear Frobung"
g1
37.9.510
73.
92.0.64
24.0,41.7
820
84.3
73.
LORA*
1.57M
94.8.a
73.3
99.1
81.8
98.6 .8.
42.0
94.7
94.9
84.0,a
PouriesPTY
480K
94.804
g9.1
81.9v.1
98.2
51.3,170
95.2
93.4
86.7.1
AuroRA
197K
94.9.a
82.5
99.1 61
82.1 1-
W8.9
59.8 65
94.0
94.
93.3
28.2
Obs. O AuroRA demonstrates strong efficiency in NLP tasks. It is evident that AuroRA out- performs the baseline across all datasets and pretrained models in both tasks. Compared to Full Fine-Tuning, AuroRA achieves a performance improvement ranging from 0.1% ~ 8.3% while us- ing only 0.04% ~ 0.06% of the total parameters. Compared to PEFT baselines, including LoRA, AuroRA achieves a performance improvement of up to 24.7% and an average improvement of 1.25% 10.88% using only 6. 6.25% 25% of the parameters. Specifically, in the commonsense reasoning task using LLaMA3-88 -B as the pretrained model, AuroRA achieves a significant 10.7% performance boost on ARC-C with just 6.25% of LoRA's parameter budget. In the NLU task, although AuroRA uses more parameters than FourierFT, it demonstrates significant performance gains across all pretrained models and datasets. For instance, using RoBERTa-Band AuroRA improves performance by 7.7% on RTE.
Obs. @ AuroRA can be scaled up to fine-tune large pretrained models. AuroRA scales effectively to fine-tuning larger pretrained models. In the NLU task, when the pretrained model changes to RoBERTa-Large from Base, nearly all PEFT methods show a performance drop compared to Full Fine-Tuning, with the largest decrease reaching 3.7%. In contrast, AuroRA still achieves performance improvements of 0.1% 3.8% across all datasets. In the commonsense reasoning task, when the model size increases to 8B, AuroRA continues to outperform LoRA by 2.4% ~ 10.7%.
3.3 AuroRA Achieves Efficiency in CV Tasks (RQ2)
To answer RQ2, we design two tasks: Image Classification and Subject-Driven Image Generation. In the image classification task, following [61], we select ViT-Base and ViT-Large [55], two popular CV foundation models, which are pretrained on the ImageNet-21 [65] dataset. We then compare AuroRA with Full Fine-Tuning, Linear Probing (fine-tuning only the classification head), LoRA, and FourierFT. The results are presented in Table 3, with more implementation details available in Appendix G.3. In the subject-driven image generation task [54], following [61] and [28]. we use the SDXL model [56] as our backbone, and then fine-tune it using both LoRA and AuroRA. The objective is to generate images based on specified prompts for a particular subject, which is defined using a set of reference images. Initially, we fine-tune a eext-to-image model by pairing the input images with text prompts that include unique identifier (e.g.. "A photo of [V] dog"). Subsequently. the model can generate images corresponding to other prompts that incorporate the same unique identifier, thereby producing images of the defined subject. The results are presented in Figure 4, and more generated cases are in Appendix H. Our observations can be summarized as follows:
Obs. ① AuroRA achieves the best performance, excluding Full Fine- Fine-Tuning, with the least num- ber of parameters. It is evident that AuroRA outperforms all other PEFT baseline methods across all eight datasets with the lowest parameter count (12.7% of LoRA and 31.0% of FourierFT) when using both the Base and Large models. Compared to Full Fine- Tuning, AuroRA uses only 0.086% of the parameters and achieves a performance improvement of 0.4% ~ 2.4% on some datasets, with only 1.6% 2.2% gap in average performance. When using ViT-Base on STANFORDCARS, other baselines show a significant performance drop of 29.3% ~ 67.7% compared to Full Fine- Tuning. In contrast, AuroRA only experiences moderate drop of 5.1%. Compared PEFT methods, AuroRA achieves an average performance improvement of 1.73% ~ 9.66%.
Obs. 4 AuroRA demonstrates stronger adaptability in the text-to-image domain. We observe that in the subject-driven image generation task, AuroRA aligns better with the environment specified in the prompt. Specifically, when given the prompt "A [V] bear plushic on top of green grass with
7

---

Esput Images
beu platis
teput Images
fowary grem gresa
LoRA
FuTORA
AunoRA
Isput
DM
Esput
LoRA
Figure 4: Results of LoRA and AuroRA in the subject-driven image generation task. AuroRA aligns better with the prompt.
sunflowers around it", LoRA generates environment with only green grass but sunflowers. In contrast, AuroRA successfully generates green grass with sunflowers.
3.4 Study
Ablation Study (3) evaluate the contribution of different modules AuroRA, we introduce two variants: (1) AuroRA w/o F, and (2) AuroRA w/o L. which correspond to the removal of the fixed and learnable nonlinearity in AuroRA, respectively. We compare these two variants with AuroRA by fine-tuning ViT-Base OXFORDPETS, CIFAR 10, DTD, and EUROSAT in the image classification task. From Table 4, we observe that: removing any component results in a performance drop for AuroRA; 2 AuroRA w/o L consistently underperforms across all datasets, indicating that the learnable nonlinearity plays a more crucial role in the success our method. Specifically, the leamable nonlinearity enables fine fitting, while the fixed nonlinearity contributes coarse fitting.
Table 4: Comparison of different settings.
Table 5: Comparison of different activation func- tions.
Setting
OxfordPets
CIFARIO
DTD
EuroSAT
AuroRA
79.6
Setting
StanfurdCars
FGVC
RESISC4S
CIFARIO
AuroRA w/o F
93.3
98.4 1o.4
78.9
98.3
AuroRA
75.7
48.2
93.6
92.0
AuroRA wlo C
93.1
188
98.2
10.
77.8
98.0 joaa
AuroRA-Ir
75.6 yo.1
47.8 1o.4
93.4 yo.2
91.9 yn.1
AuroRk-sm
75.2 jo.s
47.7 1a.
92.9
91.7 jo.3
Effect of Activation Function (RQ4) We investigate the impact of the choice of activation function in the fixed nonlinearity AuroRA's performance. Specifically, introduce two variants: (1) AuroRA-Ir, and (2) AuroRA-sm. which correspond to replacing the activation function in the fixed nonlinearity (tanh) with LeakyReI.U and Sigmoid, respectively. We compare these variants with AuroRA by fine-tuning the ViT-Base model on STANFORDCARS, FGVC, RESISC45, and CIFAR 1 (0 in the image classification task. From Table 5, we observe that Sigmoid results in the lowest performance, while tanh achieves the highest performance. Therefore, we choose tanh as the activation function for fixed nonlinearity in all our experiments.
Sensitivity to Rank & Comparison with Linear LoRA Variants (RQ5) To further investigate the impact of introducing nonlinearity, we examine its sensitivity rank and compare it with the linear LoRA variant under identical experimental settings. Specifically, we select LLaMA3-8B as the pretrained model and fine-tune it using AuroRA. MoSLoRA. and LoRA, varying the rank among {2, 4,8, 8, 16}. We evaluate their performance across four datasets. From Figure 5, we observe the following: the introduction of nonlinearity results in smaller performance fluctuations as the rank varies, i.e., more robustness to rank; AuroRA consistently outperforms across almost all rank settings and datasets, indicating that incorporating nonlinearity further enhances the model's expressiveness compared to linear approaches.

---

Hallsssinngg
Aarl *
12
26
26
10 24
10 12
34
Figure 5: Performance comparison of different methods with varying ranks. We use LLaMA 3- 8B as the pretrained model and fine-tune it using AuroRA, MoSLoRA, and LoRA methods on the HEL.ASWAG, WINOGRANDE, ARC-F and ARC-C datasets, with ranks {2, 4, 16).
Related Work
4.1 Parameter-Efficiet Fine-Ture Fine-Tuning
Ceaumeter-Effiienns Fine-Tuning (PEFT) has emerged as pivotal strategy for addressing the compu- tational challenges associated with fine-tuning large-scale pretrained models. PEFT methodologies can be broadly categorized into: O Additive PEFT approaches introduce new, trainable modules to frozen base model [18, 66, 32, 13, 67, 68, 15]. Common strategies include adapter- -bsed techniques, such AdapterFusion [59] and Hyperformer [69]; prompt-based methods, like Prefix-tuning [17] and p-tuning v2 [70]. @ Selective PEFT methods optimize a chosen subset of a pretrained model's parameters while keeping the majority frozen [71, 72, 73, 74, 75, 76]. This selection is often achieved through unstructured masking based on criteria like parameter significance, as seen in FishMask [77] and Child-tuning [78], via structured techniques that group parameters, such as Bitfit [20] and SPT [79]. @ Reparameterized PEFT techniques transform model weights into more efficient, often low-rank, representations during fine-tuning, without altering the core architecture for inference [80, 60, 81, 82, 83]. A prominent example is LoRA [16]. which introduces low-rank matrices for up- PEFT methods focus on reducing the memory footprint of fine-tuning by optimizing the training dynamics rather than the model architecture [84, 85, 86, 87]. A representative example GaLore [87], which projects gradients into low-rank subspaces to lower optimizer- staaee memory while preserving full-pau parameeter adaptability. @ Hybrid PEFT methods integrate multiple strategies from different PEFT categories to capitalize on their respective advantages [19, 88, 89J. For instance, NOAH [90] and AUTOPEFT [91], leverage neural architecture search to identify effective PEFT combinations for specific tasks. In this paper, we primarily focus on LoRA, reparameterized PEFT method.
dates. 4 Memory-Efficiies
4.2
LoRA and its Variants
The core idea of LoRA [16] is to approximate weight updates using mergeable. low-rank matrix pathways. Its variants can be broadly categorized into several types: 1 Novel Branch Designs primarily focus remodeling reformulating the original low-rank matrix approximation pathway, with notable examples including VeRA [80], FourierFT [61], PiSSA [64], and DoRA [92]. @ Multi-
Task Variants, exemplified by MoELoRA [81], MoA [82], CA-LORA [93], and Hlydral.RR.. [94], are engineered to enhance cross-task generalizationnppaiiculalyy
particularly in scenarios such as multi-task learning, domain adaptation, and continual learning- often through the strategic employment of LoRA module mixtures or ensembles. Linear Variants, including AdaLoRA [26]. SaLoRA [27], MoSLoRA [28], and FLoRA [29], typically augment the LoRA framework by incorporating an additional linear matrix between the two original low-rank factors, thereby bolstering information capture during the training phase. Beyond these, several nonlinear LoRA variants have recently emerged, including LoRAN [95]. SineLoRA [96], LoDA [97], NEAT [98], and CoLA [99]. However, these recent nonlinear variants not resolve inherent low-rank bottleneck in LoRA. In contrast, our method pairs nonlinearities with a focus on L'RA's fundamental structural limitations, achieving a superior balance between performance and parameter efficiency.
Conclusion
In this paper, we revisit LoRA from the perspective of linear mappings and introduce nonlinearity into LoRA by proposing AuroRA, an MLP-like structure. AuroRA incorporates an adaptive nonlinear layer that includes both fixed and learnable nonlinearities between the two low-rank matrices. AuroRA achieves a superior balance between performance and parameters across tasks both the NI.P and CV domains. We hope that AuroRA will inspire further exploration of nonlinear extensions to LoRA.

---

Limitation A potential limitation is that, due to limited computational resources, we do not evaluate performance on larger pretrained models in this study, leaving this exploration for future work. Broader Impact As novel nonlinear method, AuroRA is envisioned for broad future applications in key sectors such as healthcare and finance. It is anticipated to deliver more accurate and reliable services while significantly reducing resource consumption, thereby better serving human society.
Acknowledgments and Disclosure of Funding
This work is supported by the State Key Laboratory of General Artificial Intelligence; and the National Natural Science Foundation of China (Grant No. 62276006).
References
[1] Yinhan Liu. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11622 364, 2019.
[2] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced enhanced bert with disentangled attention. In International Conference on Learning Representations, 2021.
[3] AI@ Llama 3 model card. 2024.
[4] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-diieenanglee embedding sharing. arXiv preprint arXiv:2111.09543 2021.
[5] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint XXN::440311799) 2024.
[6] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
[7] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followin llama model. https: / / git..b....... www.ca///////...eeee
attpstaaaadddaapaaaaaaaaaaaa 2023.
[8] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classifica- tion. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia, July 2018. Association for Computational Linguisties.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-
1901. 2020.
[10] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du et al. Lamda: Language models for dialog applications. arXiv preprint w:Xiv:2201.082550 2022.
[11] Haonan Dong, Haoran Ye, Wenhao Zhu, Kehan Jiang, and Guojie Song. Meta-rl: Empowering large reasoning models with metacognition, 2025.
[12] Haoran Ye, Yuhang Xie, Yuanyi Ren, Hanjun Fang, Xin Zhang, and Guojie Song. Measuring human and ai values based on generative psychometrics with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2025.
[13] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. To- wards a unified view of parameter-efficient transfer learning. arXiv preprint 10.04366 1021.
10

---

[14] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[15] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open. 5:208-215 215, 2024.
[16] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representattonss 2022.
[17] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 0tti:22011.009900 2021.
[18] Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35: 1550-195-555 2022.
[19] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. LL.M-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5254-5276, Singapore, December 2023, Association for Computational Linguistics.
[20] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit:
Simple parameter-efficient fine- tuning for transformer-based masked language-models In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1-9. Association for Computational Linguistics, 2022.
[21] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403 14608, 2024.
[22] Simeng Sun, Dhawal Gupta, and Mohit lyyer. Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of rlhf. arXiv preprint arXiv:2309.0900 09055, 2023.
[23] Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, and Chun Yuan. Convolution meets loRA: Parameter efficient finetuning for segment anything model. In The Twelfth International Conference on Learning Representations, 2024.
[24] Wei Dong, Xing Zhang, Bihui Chen, Dawei Yan, Zhijun Lin, Qingsen Yan, Peng Wang, and Yang Yang. Low-rank rescaled vision transformer fine-tuning: A residual design approach. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16101-16110, 2024.
[25] Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA learns less and forgets less. Transactions Machine Learning Research, 2024. Featured Certification.
[26] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efi efficiient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023.
[27] Yahao Hu, Yifei Xie, Tianfeng Wang, Man Chen, and Zhisong Pan. Structure-aware low-rank adaptation for parameter-efficient fine-tuning. Mathematics, 11(20):4317, 2023.
[28] Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong. Mixture-of-subspaes low-rank adaptation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7880- 7899, Miami, Florida, USA. November 2024. Association for Computational Linguistics.
11

---

[29] Chongjie Si, Xuehui Wang. Xue Yang, Zhenggin Xu, Qingyun Li, Jifeng Dai, Yu Qiao, Xiaokang Yang, and Wei Shen. Flora: Low-rank core space for n-dimension. arXiv preprint arXiv:2405. 14739, 2024.
[30] Chongjie Si, Xiaokang Yang, and Wei Shen. See further for parameter efficient fine-tuning by standing on the shoulders of decomposition. arXiv preprint arXiv:2407.05411 2024.
[31] AF Agarap. Deep learning using rectified linear units (relu), arXiv preprint /.G3:v8803.837, 2018.
[32] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nip. In International conference on machine learning, pages 2790-2799. PMLR, 2019.
[33] Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
[34] Carl de Boor. A practical guide to splines. In Applied Mathematical Sciences, 1978.
[35] Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. KAN: Kolmogorov-amul networks. In The Thirteenth International Conference on Learning Representations, 2025.
[36] Shayan Aziznejad and Michael Unser. Deep spline networks with control of lipschitz regular- ity. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3242-3246. IEEE, 2019.
[37] Daniele Fakhoury, Emanuele Fakhoury, and Hendrik Speleers. Exsplinet: An interpretable and expressive spline-based neural network. Neural Networks, 152:332-346, 2022.
[38] Pakshal Bohra, Joaquim Campos, Harshit Gupta, Shayan Aziznejad, and Michael Unser. Learning activation functions in deep (spline) neural networks. IEEE Open Journal of Signal Processing, 1:295-309, 2020.
[39] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[40]
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[41] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[42] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463-4473, Hong Kong, China, November 2019. Association for Computational Linguistics.
[43] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Marquez, edi- tors, Proceedings of the 57th Annual Meeting the Association for Computational Linguistics, pages 4791-4800 Florence, Italy, July 2019. Association for Computational Linguistics.
12

---

[44] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secome Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732-8744 AAAI Press, 2020.
[45] Peter Clark, Isaac Cowhey, Oren Etzioni. Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018.
[46] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier. and Jun ichi Tsujü,, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 238 1-2911, Brussels, Belgium. October-November 2018. Association for Computational Linguistics.
[47] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[48] A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
[49] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[50] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226 2019.
[51] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10);1865-1883 2017.
[52] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[53] Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.511 2013.
[54] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 22500-2255 22510. IEEE, 2023.
[55] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[56) Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[57] Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. MiLoRA: Harness- ing minor singular components for parameter-efficient LLM finetuning. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language
13

---

Technologies (Volume 1: Long Papers). pages 4823 4833-4836, Albuquerque, New Mexico, April 2025. Association for Computational Linguisties.
[58] Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AdapterDrop: On the efficiency of adapters in transformers. In Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7930-7946, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[59] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion: Non-destructive task composition for transfer learning. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online, April 2021. Association for Computational Linguistics.
[60] Mojtaba Valipour, Mehdi Rezagholizadch, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter- efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3274-3287 Dubrovnik, Croatia, May 2023, Association for Computational Linguistics.
[61] Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficient fine-tuning with discrete fourier transform. In Forty-first International Conference on Machine Learning, 2024.
[62] Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang. LoRA- drop: Efficient LoRA parameter pruning based on output evaluation. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 5530-5543, Abu Dhabi, UAE, January 2025, Association for Computational Linguistics.
[63] Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, and Jinan Xu. DoRA: Enhancing parameter-effiient fine-tuning with dynamic rank distribution. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 11662-11675, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[64] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PiSSA: Principal singular values and singular vectors adaptation of large language models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[65] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21 pretraining for the masses. In Thirty-fithh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[66] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[67] Yaoming Zhu, Jiangtao Feng, Chengqi Zhao Mingxuan Wang, and Lei Li. Counter- interference adapter for multilingual machine translation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 28 12-2282.. Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[68] Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y Zhao, Yuexin Wu, Bo Li, Yu Zhang, and Ming-We Wei Chang. Conditional adapters: Parameter-efficient transfer learning with fast inference. In Thirty- Conference on Neural Information Processing Systems, 2023.
14

---

[69] Rabeeh Karimi Mahabadi, Sebastian Ruder. Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. In Chengging Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 565-576, Online, August 2021. Association for Computational Linguistics.
[70] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. CoRR, abs/21 10.07602, 2021.
[71] Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4884-4896, Online, August 2021, Association for Computational Linguistics.
[72] Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, and Greg Ver Steeg. Neural architecture search for parameter-efficient fine-tuning of large pre-trained language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACI. 2023, pages 8506-8515, Toronto, Canada, July 2023. Association for Computational Linguistics.
[73] Baohao Liao, Yan Meng, and Christof Monz. Parameter-efficient fine-tuning without intro- ducing new latency. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4242 02244 Toronto, Canada, July 2023. Association for Computational Linguistics.
[74] Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Peng Shi, Wenpeng Yin, and Rui Zhang. Unified low-resource sequence labeling by sample-aware dynamic sparse finetuning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6998-7010, Singapore, December 2023, Association for Computational Linguisties.
[75] Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross-lingual transfer. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 1778-1796, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[76] Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang, Weiran Xu, Yu Sun, and Hua Wu. HFT: half fine-tuning for large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume I: Long Papers), ACL 2025, Vienna, Austria, July 27- August 1, 2025, pages 12791-12819. Association for Computational Linguistics, 2025.
[77] Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[78] Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning. In
Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9514-9528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[79] Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-aware visual cfrfmeter-cfffcient fine-tuning. In ICCV, 2023.
15

---

[80] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations. 2024.
[81] Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets Ilms: Parameter efficient tinettnning for multi-task medical appli- cations. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1 110--1114, 1114, 2024.
[82] Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-LoRAs: An efficient multitask tuning method for large language models. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Pro- ceedings of the 2024 Joint International Conference on Computational Linguisties, Language Resources and Evaluation (LREC-COLING 2024), pages 11371-11380, Torino, Italia, May 2024. ELRA and ICCL.
[83] Xun Wu, Shaohan Huang, and Furu Wei. Mixture loRA experts. In The Twelfth International Conference on Learning Representations, 2024.
[84] Qijun Luo, Hengxu Yu, and Xiao Li. Badam: A memory efficient full parameter optimization method for large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulichh Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
[85] Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. LISA: layerwise importance sampling for memory-effiiient large language model fine-tuning In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.
[86]
Pengxiang Li, Lu Yin, Xiaowei Gao, and Shiwei Liu. Outlier-weighed layerwise sampling for LLM fine-tuning, In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 August 2025, pages 19460-19473. Association for Computational Linguistics, 2025.
[87] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuan- dong Tian. Galore: Memory-efficient LLM training by gradient low-rank projection. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[88] Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. UniPELT: A unified framework for parameter-efficienl language model tuning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6253 6264, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[89] Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. Parameter- efficient fine-tuning design spaces. In The Eleventh International Conference on Learning Representations, 2023.
[90] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-14, 2024.
[91] Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. AutoPEFT: Automatic con- figuration search for parameter-efficient fine-tuning. Transactions of the Association for Computational Linguistics, 12:525-542, 2024.
[92] Shih yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: deigpo-decomppsed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
16

---

[93] Weilin Zhao, Yuxiang Huang, Xu Han, Zhiyuan Liu, Zhengyan Zhang, Kuai Li, Chen Chen, TAO YANG, and Maosong Sun. CA-loRA: Adapting existing loRA for compressed LLMs to enable efficient multi-tasking on personal devices. In First Conference on Language Modeling, 2024.
[94] Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[95] Yinqiao Li, Linqi Song, and Hanxu Hou. LoRAN: Improved low-rank adaptation by non- linear transformation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1134-3143, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[96] Yiping Ji, Hemanth Saratchandran, Cameron Gordon, Zeyu Zhang, and Simon Lucey. Efficient learning with sine-activated low-rank matrices. In The Thirteenth International Conference on Learning Representations, 2025.
[97] Jing Liu, Toshiaki Koike-Akino, Pu Wang, Matthew Brand, Ye Wang, and Kieran Parsons. Loda: Low-dimensional adaptation of large language models. In NeurlPS 23 Workshop on on Efficient Natural Language and Speech Processing, 2023.
[98] Yibo Zhong, Haoxiang Jiang, Lincan Li, Ryumei Nakada, Tianci Liu, Linjun Zhang, Huaxiu Yao, and Haoyu Wang. Neat: Nonlinear parameter-efficient adaptation of pre-trained models, 2025.
[99] Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Zi Yang. Paul Hovland, Bogdan Nicolae, Franck Cappello, and Zheng Zhang, Cola: Compute-efficient pre-training of Ilms via low-rank activation. CoRR, abs/2502. 10940, 2025.
[100] Roger A Horn and Charles R Johnson. Topics matrix analysis. Cambridge university press, 1994.
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes]. , [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No]" provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] " or "[NA]" " is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best, judgment and write justification to claborate. All supporting evidence can appear either in the main paper the
17

---

supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this paper, we introduce novel non-linear errameter-efficient fine-tuning method and we claim the contributions and scope in the abstract and introduction sections (See Abstract and Introduction Section).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In this work, we systematically discuss the limitations of our research and outline directions for future work (See Conclusion Section).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
18

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In this work, we analyze our proposed method from a theoretical perspective and provide complete and detailed proofs (See Method Section and Appendix).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We provide the code necessary for replicating the studies described in this paper via an anonymous link, and we detail the experimental setup for the replication in the article itself (See Appendix).
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
19

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: For the datasets disclosed in the article, we have provided information regarding their sources and origins (See Appendix).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm co/
http://guuiieee/eoeuuuuntntn cosssnoiPycy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//niss.. cccpublic/guiiai /ww.///i//////eee//oee/eee/eee Letgdde///oeeuuaiiiee (CodeSubmission sesiisssiccyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary to understand the results (See Appendix).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
20

---

Answer: [Yes]
Justification: Experimental results are tested multiple times to ensure stability and reliability (See Experiments).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: this paper, we provide detailed information about the experimental resources, including GPU configurations used in our studies (See Appendix).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt////eerippp.eee wwtppccccc/ceeecccccccccccc httpaie/eeehccciiieeeeeeeeeeeeee
Answer: [Yes]
Justification:
The study presented in this paper conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
21

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have provided the societal impacts of the work (See Conclusion).
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not address issues related to this aspect.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
All creators and original owners of the assets used in our paper, such as code, data, and models, have been properly credited. We have explicitly mentioned the licenses and terms of use for each asset and have ensured full compliance with these terms throughout our research.
Justification:
22

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The research presented in this paper not concerned with new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve experiments or research related human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
23

---

Justification:
This paper does not address potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https //tpur/eeipp///eeee 2205 /LLM) for what should or should be described.
24

---

A
Notations
We summarize the notations used throughout the manuscript in Table
Table 6: Notations commonly used in the AuroRA method.
Notation
Definition
Wo E Rddut Xdu
Pretrained weight matrix
AW E Bdousxdm
Weight update matrix
A E R Pxdm
Downward projector (low-rank matrix)
B Ro x
Upward projector (low-rank matrix)
Original LoRA rank (r & min((di, dont))
Compressed hidden dimension (7 r)
E R
Input vector
h E Rous
Output vector
Adaptive Nonlinear Layer (ANL) mapping 12" R Projection onto F-dim hidden space (matrix A) Self-projection hidden space (matrix E R Projection back to dous (matrix B) Fixed nonlinearity (e.g. tanh)
Paown
Paett
Pup
F())
(())
Learnablc nonlinearity (B- (B-spline based)
w. R
Spline weight vector in C
s(Z)
Spline basis functions applied to each component Number of training epochs
B
Complete Process
In this section, first provide further details both the static weight merging operation performed after the training phase and the actual process at inference time. We then offer empirical validation for our approach. Finally, the complete algorithm workflow of our AuroRA presented in Algo.
B.1 Weights Merging & Inference Phase
During the training phase, to enhance training flexibility, AuroRA utilizes a dynamic, input-dependent update mechanism formulated as B or( (Ax). After the training phase, once all learnable parameters are fixed and considered optimized, AuroRA transitions to a static form for the inference stage. This static form, given by AW B o (A). facilitates the scamless integration of AuroRA into the pre-trained weight Wo. consistent with standard LoRA. Therefore, after the weights are merged, the effective forward propagation process at inference time is formally given by:
h Wox + AWx = Wox + B (A)x.
Such an approach eliminates any additional computational overhead during inference, while concur- rently preserving superior performance.
Why is This Strategy Effective? To validate our strategy of dynamic training combined with static inference, we empirically compare it against both fully dynamic and fully static approaches. For this purpose, we introduce two comparative variants: (1) AuroRA-D, DD, which maintains dynamic processing throughout both training and inference (i.e., its forward pass is consistently h Wix B B-o (Ax)), and (2) AuroRA-S, which consistently employs static form for both phases (i.e., Wox + (A))). evaluate AuroRA, AuroRA-D, and AuroRA-S by fine-tuning LLaMA3- 8B for Commonsense Reasoning on datasets including ARC-E, OBQA, STQA, and ARC-C, and record both accuracy and total training and inference time. From Table 7, we observe that: 1 AuroRA exhibits nearly identical performance to AuroRA-D and significantly outperforms AuroRA-S in
25

---

Table 7: Comparison of accuracy and total time consumption for different settings eight Common- sense Reasoning datasets, using LLaMA3-8B as pre-trained model.
Setting
Time
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Avg.
LoRA
15.05
85.2
79.9
91.7
84.3
84.2
71.2
79.0
AuroRA-S
15.18 h
71.4
86.9
78.1
93.5
81.7
88.5
78.1
83.9
82.8
AuroRA-D
15.60 h
72.6
87.5
79.2
94.3
83.0
89.5
78.9
85.0
83.8
AuroRA
15.28 h
72.5
87.4
79.0
94.2
83.0
89.3
78.8
84.8
83.6
practice; AuroRA achieves a runtime comparable that of AuroRA-S and is markedly faster than AuroRA-D. Therefore, our practical implementation adopts this dynamic training with static inference strategy, which be seamlessly merged into pre-trained weights after the training phase (consistent with standard LoRA), and thereby also achieves an effective tradc-off between performance and computational cost.
B.2 Algorithm Workflow
The algorithm framework presented Algo. 1.
Algorithm 1: Algorithm workflow of AuroRA
Input Prettriined weight Wh. low-rank factors E IR#xd and B E RRdxF, ANL parameters, training data {x,; Be)] number of epochs T
/+ Training Phase (dynamic update: (o(Ax))
for epoch + to T do
for each minibatch {x, y) in training data do
/* Forward pass with ANL on input
*/
h + Wo + B o
Eq.
Compute loss C(h,3)
/* Backpropagate through A,B, and ANL parameters Backpropagate and update B,A,A
end end
/+ Inference Preparation (static merge: Ba(A))
Function MergeWeights():
/* Compute element-wise ANL on matrix A
A + (AA)
/* Form the effective weight update
AW < BA;
Eq.
/* Merge into pretrained weights
W + Wo AA
return W
/+ Inference Phase (static forward: no extra ANL)
*/
for each test sample x do
W +- MergeWeights()
Wx:
Eq.
/* Use h for downstream prediction
#/
end
C Intuitive Case
To intuitively and concisely illustrate the impact of nonlinear mapping the matrices, we first consider a simple scenario where E Rlx2 and R221,
b
= [a]
02).
B
(10)
b2
26

---

We then introduce the LeakyReLU activation function as the nonlinear mapping between the two low-rank matrices. Depending on the elements of matrix A, the resulting weight update comprises the following four matrix structures:
bij bi02
brai b2a2]
AW
if > 0 and d2 0,
by (caz)
b2((ra) b202
if a1 S 0 and a2 0,
0102
(11)
b101 bi (aa2)
byd1 by (((2))
a > 0 and a2 S 0,
bi (aa1) (aaz)
b2 ba(aa)) by (2(a))
a S 0 and S 0.
where a is a hyperparameter of LeakyReLU, usually a positive number less than 1. Meanwhile, LoRA only produce:
b101
b2a1 b202
AW
(12)
0102
Under the LeakyReLU activation function, each negative component of A is scaled by the factor a, while positive components remain unchanged. This piecewise linear mapping disrupts the uniformity of low-rank multiplication, causing the final weight update A AW to depend not only on the product of A and B but also the local behavior of each element in A. Consequently, even under tight rank constraints, the model benefits from a richer set possible weight updates, enhancing adaptability to varying input distributions.
D Proof of Proposition 2.1
In this appendix, provide the complete theoretical analysis and proof of Proposition 2.1. We first restate the problem setup and then present the necessary lemmas, followed by the main proof.
Definition D.1 (Best Linear Rank-r Error). For M dou din, define
inf (M) = U'R VeRrxdu
UVI.
If rank(M) > r, then (MM > 0 [100].
Assumption D.2 (Bounded Input Domain), We assume X' C Rd satisfies ||x||| <. Xmax- Then for A R2 with ||||| < Amaxx the vector z = Ax remains in bounded set Q C IR" (compact). Definition D.3 (Nonlinear Low-Rank Update). Let
Mnoulimear (x) Bo xx),
with A Re. B Poa xr, The map R² R" is given by
a(z) F(z) + w. s(z),
where a fixed bounded function (e.g. tanh-based) and s(z) denotes B-spline basis functions in R".
Lemma D.4 (Piecewise Polynomial Approximation). Consider f + Rm with f E C² (S) on bounded domain W²" Let A > 0 be the subdivision size in each coordinate axis for constructing a tensor-product B-spline. Then there exists B-spline g(z) such that
sup flf(z)))) -g())l S C)(A)*
zen
where Cy > is constant depending on I's k-th order partial derivatives and the geometry of Ω [34].
27

---

Lemma D.5 (Combining Fixed and Learnable Nonlinearities). Let S C. R' be compact. Assume F: R' fixed, bounded, and C', and ler h(z) E C* CC (52) be the target. Define
( = =(z) + W, s(z),
where s(z) is a B-spline basis. Then, for any > O, one can choose A >> 0 and w, such that
sup h(z)) - [(z) + Ws s())]]] S
xEQt
Furthermore, the error decays like ((())* as +
Proof of Lemma D.5. Let r(z) = 1((z) - F(z). Since h E ( and F fixed and C², r(z)
remains C* Applying Lemma D.4 to r(z) yields B-spline g(z) with [|(()) - g(>)ll - (A)*
Hence [lh(z) [z)) + (z)]ll < C (A)*, completing the proof.
We restate Proposition 2.1 here for completeness:
Proposition 2.1 (Lower Approximation Error) Let M Rou xda with rank(M) Then
(M) = inf M UVI > U.V
Under Definition D.3, there exist A", B and a B-spline parameter set (w") such that
M
Mnonlinearl]
s cer(M), 0 0<c<1.
Proof. Let M U- be the best linear rank-r approximation of M, | M = (MM. Denote the residual R M M* and we have | (MM
By Assumption D.2, for ||x|| S Xmax, let A Ω C IR" with || Amax- Thus lies in a compact Ω. Consider R(x) as function h(z) = R (x). Since R is linear (hence (*0), h(z) is at least C'l in
From Lemma D.5, there a B-spline w s(z) approximating h(z) - -F(z) within y ||] for some 0 7 1. Define
M(x) == (x) + Ba F(A'x) + w. s(A'x)
Then
[B=' RR| S 7|| (M). [JM-M|] |I -IM*
Since 7 1, we obtain || M M e, (M), which strictly improves upon the LoRA limit. Setting
M Muoulinear completes the proof.
E Proof of Proposition 2.2
finctiono and B-spline basis functions results in bounded gradients with respect to both the inputs and the model parameters.
Proposition 2.2. In the AuroRA, the use of the tanh activation function
The loss function L for a single data point (æ, y) is defined as L(x, y) = 1/2 fl (2) yll², where E Idda is the target output. We will compute and bound the gradients of the loss function with respect W,, w, and the input x.
Lemma E.1. The gradients of the loss function with respect to Wa is bounded.
at Proof. Compute the gradient SW.
aL
JANL(AT
= (z)) - T B. aW,
aW.
Compute @A)L(*). aW,
AANL(2)
ad(W,d(2)) diag (ø' (Woo(z))) -())) OW,
aW,
28

---

where (a() = tanh?(u) is the derivative of Tanh, diag(v) denotes diagonal matrix with vector on the diagonal. It is not difficult to deduce that 6(2) (-1,1) since Tanh outputs are bounded, and ø' (u) E (0, 1J because - tanh"(u) & 1. So BaNl(s) is bounded. Consequently, aw, bounded as a product of bounded terms.
az
oW
Lemma E.2. The gradients of the loss function with respect to w, is bounded.
Proof. Compute the gradient
OL:
Sw.
aL
DANL(A' x)
(fAurORA (2) v)) B Ow.
dw.
Compute
@AN the
0ANL(2) = s(2),
aw,
since ANL(2) is linear in ws. B-spline basis functions (22;) are smooth and have compact support, and the outputs of ((2;) are bounded. Therefore, s(2) is bounded, and thus aL is bounded. Se.
Lemma E.3. The gradients of the loss function with respect to the input x is bounded.
Proof. Compute the gradient
BL. Uz
aL
dANL(A
(fAurORA (x) y) -)
W (W+B.
dx
Compute
8ANL(AT x).
On
DANL(AT<) x)
(ANL(2)
A
8x
Oz
where 2 = A Tr. x. Compute
8ANL(s).
Oz
0ANL(2) Oo(Woo(z))
0s(z)
W.
Oz
Oz
Oz
Compute Bo(W)d(z))
a (o(W))(z)) diag (ø' (Woo(2)) W, diag (ø' (o'()))
Oz
d'()) and ø' (W))(z)) are bounded (0, 1]. Entries of W, are finite. So Ro(W,((2))
is bounded.
dz
Compute @s(x),
0s(z) (B'(2),
O=
Derivatives B2(z,) of B-spline functions are bounded due to their polynomial nature and compact support. So (s(z) is bounded. Therefore, @ANI((2) is bounded, leading to BANL(A x) being bounded.
dz Consequently, dr. bounded.
Ø=
Ox
F
Dataset
F1
GLUE Benchmark
The GLUE (General Language Understanding Evaluation), as introduced in [39], is widely adopted benchmark in the field of Natural Language Processing (NLP). GLUE encompasses collection of eight diverse NLP tasks: MNLI (natural language inference), SST-2 (sentiment analysis), MRPC (paraphrase detection), CoLA (linguistic acceptability). QNLI (natural language inference). QQP (question answering), RTE (recognizing textual entailment), and STS-B (textual similarity). The statistical details of these datascts summarized in Table 8.
29

---

Table 8: Detailed task descriptions and dataset statistics for the GLUE benchmark. STS-B is categorized as a regression task, while all other tasks involve single-sentence or sentence-pait classification.
Corpus
Task
Metrics
Labels
Train
Domain
Single-Sentence Tasks
CoLA SST-2
Acceptability
Matthews Corr.
2
8.55k
1.04k
1.06k
misc.
Sentiment
Accuracy
67.3k
872
1.82k
Movie reviews
Similarity and Paraphrase Tasks
MRPC
Paraphrase
Accurcy/F1
2
3.67
408
1.73k
News
STS-B
Sentence similarity
Pearson/Spearmnn Corr.
1
5.75k
1.5k
1.38k
misc.
QQP
Paraphrase
Accuracy/F1
2
364k
40.4k
391k
Social QA
Inference Tasks
MNLI QNLI
NLI
Accuracy
3
393k
19.65k
19.65k
misc.
QANLI
Accuracy
5.46k
Wikipedia
RTE
NLI
Accuracy
2
2.49%
277
3k
News & Wikipedia
F.2
Commonsense Reasoning
Following [19]. use eight datasets in Commonsense Reasoning task. (1) The BoolQ [40] dataset a question-answering benchmark consisting of 15,942 examples, where the questions are naturally occurring and generated in unprompted and unconstrained settings, requiring yes/no answers. (2) The PIQA [41] dataset presents questions with two potential solutions, demanding physical commonsense reasoning identify the correct answer. (3) The SIQA [42] dataset focuses reasoning about human actions and their social implications. (4) The HellaSwag [43] dataset is designed for commonsense natural language inference (NLI) tasks, where each question includes context and several potential endings, from which the correct continuation must be selected. (5) The WinoGrande [44] dataset is task with binary options, where the goal
select the most plausible option for given sentence requiring commonsense reasoning, (6) The ARC-c and (7) ARC-e [45] datasets refer the Challenge and Easy sets, respectively, of the ARC dataset, which consists of multiple-choic science questions designed at a grade-school level, with the former being more challenging than the latter. (8) The OBQA [46] dataset focuses on questions that necessitate multi-step reasoning, integration of external common knowledge, and in-depth text comprehension. Statistical details are shown in Table 9.
Table 9: Details of datasets being evaluated in commonsense reasoning task.
Dataset
#Train
#Test
Answer
BoolQ [40]
9.4K
3270
Yes/No
PIQA [41]
16.1K
1830
Option
SIQA [42]
33.4K
1954
Option
HellaS [43]
39.9K
10042
Option
WinoGrande [44]
63.2K
1267
Option
ARC-e [45]
1.1K
2376
Option
ARC-c [45]
2.3K
1172
Option
OBQA [46]
5.0K
500
Option
F3 Image Classification
We show the details of the datasets in Image Classification task in Table 10.
G
Hyperparameters
To ensure the reproducibility of our experimental results, we provide the detailed hyperparameter settings used in our experiments. In all of our experiments, to achieve better balance between parameter count and performance, we set the hidden layer dimension (Rank ?) of AuroRA to Correspondingly, we set the hyperparameter of AuroRA to 4. Natural Language Understanding and
30

---

Table 10: Details of the datasets for the Image Classification task.
Dataset
#Class
#TTrii
#Val
#Test
Rescaled resolution
OxfordPets [47]
37
3,312
368
3,669
StandfordCars [52]
196
7.329
815
8,041
CIFARIO [48]
10
45,000
5,000
10,000
DTD [49]
47
4,060
452
1.128
224 x 224
EuroSAT [50]
16,200
5,400
5,400
FGVC [53]
100
3,000
334
3,333
RESISC45 [5]]
18,900
6,300
6,300
CIFARI00 [48]
45,000
5,000
10,000
image classification tasks run on four NVIDIA GeForce RTX 4090 (24GB) GPUs. Commonsense
reasoning and subject-driven generation tasks run NVIDIA L20 (48GB).
G.1 Natural Language Understanding
We provide the hyperparameters used for the GLUE benchmark in natural language understanding experiments in Table 11. To facilitate reproducibility, we fix the random seed to We tune the learning rate, while all other settings follow those used in LoRA [16] and FourierFT [61].
Table 11: Hyperparameter setup of AuroRA for the GLUE benchmark.
Model
Hyperparameter
STS-B
RTE
MRPC
CoLA
SST-2
QNLI
Optimizer
AdamW
LR Schedule
Linear
Both
Warmup Ratio
0.06
Rank F
2
4
Epochs Learning Rate
30
80
30
90
30
80
6E-4
5E-4
8E-4
5E-3
8E-4
SE-3
Base
Max Seq. Len
512
512
512
512
512
512
Batch Size
64
16
64
32
32
32
Epochs Learning Rate
20
10
30
50
40
20
3E-4
4E-4
1E-3
SE-4
8E-4
4E-4
Large
Max Seq. Len
512
512
512
256
128
512
Batch Size
16
32
16
16
16
16
G.2
Commonsense Reasoning
We provide the detailed hyperparameters for fine-tuning LLaMA3-8B in the commonsense reasoning task Table 12.
Table 12: Hyperparameter setup of AuroRA for Commonsense Reasoning.
Hyperparameter
Commonsense Reasoning
Rank F
2
Dropout Batch Size
0.05
16
Optimizer
Adam W
Learning Rate
3e-4
Warmup Steps
100
Epochs
Target module
q.k.v.up.down
31

---

G.3 Image Classification
We provide the detailed hyperparameters for the image classification in Table 13. We tune the learning rate, while the weight decay value follows the settings used in FourierFT [61] without tuning.
Table 13: Hyperparameter setup of AuroRA for the image classification.
Model
Hyporpommmttr
OafordPess
StafforCams
FurSATT
RESISCAS
CIFARI00
Optimizer
Linezz
Learning
Learning
Weight Decay
16-4
76-5
3F-4
1F-4
Learning Rate (AcroMA
9e-3
1.5e-2
7.5e-3
1.5e-2
Large
Learning Rate (Head)
1.5e-2
Weight Decay
38
40.5
9E.5
70
38
70.5
36.4
1B:4
H More Cases of Generated Images
In Figure 6, we present more results of subject-driven generation using both LoRA and AuroRA.
32

---

A [V] bear plushie floating in ocean of milk
A [V] robot toy with blue house in the background
[V] bear plushie snow
M robot toy with a city in the background
A [V] bear plushle cobblestone street
A [V] robot toy floating on top of water
A [V] bear plushie top of pink fabric
A [V] robot toy the jungle
LoRA
AuroRA
LoRA
AuroRA
Figure 6: More generated images in the subject-driven generation task.
33


Reviews:
[{"review_id": "WAztYitfoD", "reviewer": "Reviewer_dGxY", "summary": "This paper proposes AuroRA, a novel PEFT method designed to address the low-rank bottleneck inherent in Low-Rank Adaptation. Unlike traditional LoRA or its linear variants that remain constrained by their linear nature, AuroRA introduces a nonlinear mapping through an Adaptive Nonlinear Layer placed between the two low-rank projection matrices. ANL includes both fixed (e.g., tanh) and learnable (e.g., spline-based) nonlinearities, forming an MLP-like structure. This structure enhances expressive power and compresses parameter count while maintaining training stability.The authors support their method with 1) Theoretical analysis, showing reduced approximation error and bounded gradients. 2) Comprehensive experiments across 22 datasets and 6 pretrained models (NLP and CV).", "questions": "1 - In the paper, you motivate AuroRA as a means to close the performance gap between LoRA and full fine-tuning by increasing representational capacity. However, in several experimental results (e.g., Table 1 and Table 3), LoRA already performs on par or better than full fine-tuning. Could you provide empirical evidence or experimental configurations where LoRA significantly underperforms full fine-tuning, and AuroRA meaningfully narrows that gap? This would help validate the motivation more clearly.\n\n\n2 - Several recent nonlinear LoRA variants are cited (e.g., LoRAN, LoDA, NEAT, SineLoRA), but not included in the empirical comparisons. To better contextualize the performance of AuroRA, can you add a comparison against these methods on at least one representative NLP and CV benchmark? This would clarify whether your proposed nonlinear design offers substantive improvements over existing nonlinear extensions to LoRA.\n\n3 - A key advantage of LoRA is that the learned low-rank matrices can be merged into the base weights, eliminating inference overhead. Since AuroRA includes nonlinearities (e.g., tanh, B-spline), this merging is no longer possible. Can you include an inference-time benchmarking comparing AuroRA to LoRA to demonstrate that the additional runtime cost is minimal or acceptable in practice?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1 - The key contribution—injecting nonlinear transformations (ANL) into the LoRA framework—is conceptually sound and well-motivated.\n\n2 - The paper provide broad evaluation across tasks (GLUE, commonsense QA, CV benchmarks, DreamBooth) and modalities (text, vision, generation) with strong empirical results.\n\n3 - The ablations are convincing. Sec. 3.4 shows that both fixed and learnable nonlinearities are essential. Also, rank sensitivity study (Figure 5) shows significantly reduced sensitivity to rank vs. LoRA and MoSLoRA, which is a known practical issue for PEFT models.\n\n\n**Weaknesses:**\n\n1 - My main concern is that I see a mismatch between motivation and experimental setup. The authors repeatedly claim that AuroRA aims to \"close the performance gap between LoRA and full fine-tuning\" (e.g., Abstract, Introduction) by improving representational power. However, in the reported experiments (e.g., Table 1, Table 3), LoRA already matches or outperforms full fine-tuning in most settings. As such, there is no empirical demonstration that a performance gap exists or that AuroRA helps to close it.\n\n2 - Comparison set omits nonlinear LoRA variants. Recent methods such as LoRAN[89], LoDA[91], SineLoRA[90], and NEAT[92] are mentioned in related work but not evaluated empirically, leaving open how much extra gain AuroRA offers over prior nonlinear attempts.\n\n3 - One of the main benefits of LoRA is the fact that the low-rank update AB can be merged into the weights of the LLM, W’ = W + AB, so LoRA wouldn’t introduce any inference cost or run-time issue after fine-tuning. But in AuroRA, due to the existence of activations in the middle, weights can’t be merged. A rigorous inference latency benchmarking is required to make sure that inference overhead introduced by AuroRA is negligible.", "was_revised": true, "final_justification": "After considering the rebuttal, I am keeping my score unchanged (borderline accept).\n\n**Resolved issues**\n\n* The additional CV results (e.g., StanfordCars with ViT) demonstrate cases where LoRA significantly underperforms full fine-tuning, and AuroRA narrows the gap.\n* Comparisons to recent nonlinear variants (NEAT, SineLoRA) on representative NLP/CV benchmarks improve empirical grounding.\n* The explanation of the “Dynamic Training, Static Inference” strategy and its zero-overhead inference mode addresses most concerns about runtime cost.\n\n**Remaining concerns**\n\n* In the NLP setting, LoRA often matches or exceeds full fine-tuning, so the core motivation of “closing the gap” remains less motivated outside CV in this paper.\n* Added nonlinear LoRA comparisons are limited in scope; broader coverage would strengthen claims of superiority.\n* Inference strategy details are mostly confined to the appendix; making them more prominent in the main text would improve clarity.", "rebuttal": "We sincerely thank you for your thoughtful and detailed review of our paper. We provide our point-by-point responses to your questions below.\n\n---\n\n> Q1: In the paper, you motivate AuroRA as a means to close the performance gap between LoRA and full fine-tuning by increasing representational capacity. However, in several experimental results (e.g., Table 1 and Table 3), LoRA already performs on par or better than full fine-tuning. Could you provide empirical evidence or experimental configurations where LoRA significantly underperforms full fine-tuning, and AuroRA meaningfully narrows that gap? This would help validate the motivation more clearly.\n\nThank you for your insightful comment! We would like to respectfully point out that in the image classification task on StanfordCars, as shown in Table 3 (Section 3.3), our method demonstrates significant improvements. \n\n1. When using ViT-Base as the backbone model, the accuracy of **Full Fine-Tuning** is **79.8**, while **LoRA** achieves only **45.4**. In contrast, our proposed **AuroRA** reaches an accuracy of **75.7** while using only **12.7% of LoRA's parameters**, successfully narrowing this performance gap. \n2. The case for ViT-Large is analogous. We observe a significant performance gap where **LoRA's accuracy (73.3)** is considerably lower than that of **Full Fine-Tuning (88.9)**. Our **AuroRA** model substantially mitigates this gap, reaching **82.5** accuracy while using only **12.5% of LoRA's parameters**.\n\n---\n\n> Q2: Several recent nonlinear LoRA variants are cited (e.g., LoRAN, LoDA, NEAT, SineLoRA), but not included in the empirical comparisons. To better contextualize the performance of AuroRA, can you add a comparison against these methods on at least one representative NLP and CV benchmark? This would clarify whether your proposed nonlinear design offers substantive improvements over existing nonlinear extensions to LoRA.\n\nTo address your concern, we compare our method **against two recently proposed non-linear baselines: NEAT [1] and SineLoRA [2]**. We conduct these comparisons on the GLUE and Commonsense Reasoning benchmarks, using RoBERTa-Base and LLaMA3-8B as backbone models, respectively. As the results in Table A and Table B below demonstrate, AuroRA achieves the highest performance with the minimum number of parameters:\n\n*Table A. Performance and parameter comparison of RoBERTa-Base on the GLUE benchmark.*\n\n| Method |  Params.   |  SST-2   |   MRPC   |   CoLA   |   QNLI   |   RTE    |  STS-B   |   Avg.   |\n| :----- | :--------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: |\n| LoRA   |    0.3M    |   95.1   |   89.7   |   63.4   |   93.3   |   78.4   | **91.5** |   85.2   |\n| NEAT   |    0.3M    | **95.2** |   90.0   |   64.8   |   92.3   |   82.7   |   90.7   |   86.6   |\n| AuroRA | **0.075M** | **95.2** | **91.9** | **65.1** | **93.4** | **85.2** | **91.5** | **87.1** |\n\n*Table B. Performance and parameter comparison of LLAMA3-8B on Commonsense Reasoning benchmarks.*\n\n| Method   | Params.  |  BoolQ   |   PIQA   |   SIQA   | HellaSwag | WinoGrande |  ARC-e   |  ARC-c   |   OBQA   |   Avg.   |\n| :------- | :------: | :------: | :------: | :------: | :-------: | :--------: | :------: | :------: | :------: | :------: |\n| LoRA     |  56.6M   |   70.8   |   85.2   | **79.9** |   91.7    |    84.3    |   84.2   |   71.2   |   79.0   |   80.8   |\n| SineLoRA |  56.6M   |   72.4   |   86.5   |   79.8   |   94.0    |  **85.2**  |   87.6   |   78.1   | **85.0** | **83.6** |\n| AuroRA   | **3.5M** | **72.5** | **87.4** |   79.0   | **94.2**  |    83.0    | **89.3** | **78.8** |   84.8   | **83.6** |\n\n---\n\n> Q3: A key advantage of LoRA is that the learned low-rank matrices can be merged into the base weights, eliminating inference overhead. Since AuroRA includes nonlinearities (e.g., tanh, B-spline), this merging is no longer possible. Can you include an inference-time benchmarking comparing AuroRA to LoRA to demonstrate that the additional runtime cost is minimal or acceptable in practice?\n\nThank you for your detailed comment! We wish to clarify our methodology, as discussed in Appendix B.1, **AuroRA uses a \"Dynamic Training, Static Inference\" strategy, allowing for weight-merging that results in zero extra overhead during inference.** We empirically support this choice in Table 7 (Appendix B.1) , which shows that our strategy strikes the optimal balance between accuracy and time consumption when compared to fully dynamic (AuroRA-D), fully static (AuroRA-S), and standard LoRA approaches. For this reason, we adopt this strategy in all experiments in the main body of the paper. Furthermore, we provide a thorough analysis of the computational complexity during training in Section 2.4 and present the complete algorithmic workflow of AuroRA in Appendix B.2 for full transparency.\n\n---\n\n**Reference:**\n\n[1] Neat: Nonlinear Parameter-efficient Adaptation of Pre-trained Models\n\n[2] Sine activated low-rank matrices for parameter efficient learning, ICLR'24"}, {"review_id": "4RtHxKmd3N", "reviewer": "Reviewer_FTUd", "summary": "This paper introduces AuroRA, a novel variant of LoRA (Low-Rank Adaptation) designed to overcome its representational limitations by introducing an Adaptive Nonlinear Layer (ANL) between the low-rank matrices. By incorporating both fixed and learnable nonlinearities, AuroRA forms an MLP-like structure with enhanced expressiveness, better parameter efficiency, and theoretical guarantees for lower approximation error and bounded gradients. The method is evaluated across 22 datasets and 6 pretrained models, spanning both NLP and CV domains, and consistently outperforms existing PEFT baselines including LoRA and its variants.", "questions": "1. PEFT methods are generally more useful in fine-tuning tasks, such as instruction following, math, or coding. How is AuroRA's performance in those tasks?\n\n2. In Figure 1, it is shown that AuroRA even outperforms full-parameter fine-tuning. Is there an intuitive explanation for this phenomenon?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The paper clearly identifies the low-rank bottleneck of LoRA and proposes a well-motivated nonlinear extension. The shift from purely linear mappings to hybrid nonlinear designs is both novel and practical.\n\n2. The incorporation of fixed (e.g., tanh) and learnable (e.g., spline-based) nonlinearities is well-explained, with theoretical support including proofs for lower approximation error and gradient boundedness.\n\n3. AuroRA achieves good performance in both NLP (e.g., GLUE, Commonsense Reasoning) and CV tasks (e.g., CIFAR10, ViT-based classification), while using significantly fewer parameters than full fine-tuning or LoRA.\n\n4. Extensive ablation studies (e.g., effects of nonlinearity types, activation functions, rank sensitivity) provide convincing evidence of AuroRA’s robustness and component effectiveness.\n\n5. The paper is generally well-written, with clear figures (e.g., PCA visualizations) and structured discussions on design, theory, and evaluation.\n\n### Weakness\n1. The paper acknowledges not evaluating on larger models, which would strengthen the claim of scalability.\n\n2. While the paper argues that the additional complexity from ANL is negligible, more concrete runtime/memory comparisons would improve practical understanding.\n\n3. While claiming state-of-the-art, empirical comparisons with recent nonlinear variants (e.g., LoRAN, NEAT) and other PEFT methods are missing [1][2][3][4][5][6][7]. Please remove this state-of-the-art claim if these baselines are not compared.\n\n4. Some missing related works in the PEFT literature: HFT [3], LISA [4], BAdam [5], Owlore [6], Galore [7].\n\n### Reference\n\n[1] Yinqiao Li, Linqi Song, and Hanxu Hou. LoRAN: Improved low-rank adaptation by a non-linear transformation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3134–3143, Miami, Florida, USA, November 2024. Association for Computational Linguistics.\n\n[2] Yibo Zhong, Haoxiang Jiang, Lincan Li, Ryumei Nakada, Tianci Liu, Linjun Zhang, Huaxiu Yao, and Haoyu Wang. Neat: Nonlinear parameter-efficient adaptation of pre-trained models, 2025.\n\n[3]: Hui, Tingfeng, et al. \"Hft: Half fine-tuning for large language models.\" arXiv preprint arXiv:2404.18466 (2024).\n\n[4]: Pan, Rui, et al. \"LISA: layerwise importance sampling for memory-efficient large language model fine-tuning.\" NeurIPS 2024.\n\n[5]: Luo, Qijun, Hengxu Yu, and Xiao Li. \"BAdam: A memory efficient full parameter optimization method for large language models.\" NeurIPS 2024.\n\n[6]: Li, Pengxiang, et al. \"Owlore: Outlier-weighed layerwise sampled low-rank projection for memory-efficient llm fine-tuning.\" arXiv preprint arXiv:2405.18380 (2024).\n\n[7]: Zhao, Jiawei, et al. \"Galore: Memory-efficient llm training by gradient low-rank projection.\" arXiv preprint arXiv:2403.03507 (2024).", "was_revised": true, "final_justification": "The additional baseline comparison results address most of my concerns, so I have increased my quality score and rating.", "rebuttal": "We would like to express our sincere appreciation for your time and effort in reviewing our manuscript! Here, we address your comments on a point-by-point basis.\n\n---\n\n> W1: The paper acknowledges not evaluating on larger models, which would strengthen the claim of scalability.\n\nThank you for your valuable comment! Due to computational constraints, we are currently unable to test on larger-scale models. We leave this as a promising direction for future work.\n\n---\n\n> W2: While the paper argues that the additional complexity from ANL is negligible, more concrete runtime/memory comparisons would improve practical understanding.\n\nWe appreciate your detailed comments! \n\nWe would like to respectfully point out that we provide a detailed comparison of runtimes in Table 7 (Appendix B.1), which shows that AuroRA improves performance without a significant increase in computational time. \n\nFurthermore, to address your concerns, we compare the GPU memory footprint of LoRA and AuroRA on the Commonsense Reasoning task, using the LLaMA3-8B as the base model on NVIDIA L20 (48GB) GPU. As shown in Table C, we observe that AuroRA introduces only a minimal additional memory overhead.\n\n*Table C. Comparison of GPU memory usage on Commonsense Reasoning tasks with LLaMA3-8B.*\n\n| Method | Mem.     |\n| ------ | :------- |\n| LoRA   | 29590MiB |\n| AuroRA | 30834MiB |\n\n---\n\n> W3: While claiming state-of-the-art, empirical comparisons with recent nonlinear variants (e.g., LoRAN, NEAT) and other PEFT methods are missing [1,2,3,4,5,6,7]. Please remove this state-of-the-art claim if these baselines are not compared.\n\nThank you for this valuable suggestion! \n\nTo demonstrate the performance comparison with recent non-linear variants, we compare AuroRA against NEAT [2] and SineLoRA [8] on the GLUE and Commonsense Reasoning benchmarks, using RoBERTa-Base and LLaMA3-8B as the respective base models. As shown in Tables D and E, the results indicate that AuroRA achieves optimal performance with the fewest parameters.\n\n*Table D. Performance and parameter comparison of RoBERTa-Base on the GLUE benchmark.*\n\n| Method |  Params.   |  SST-2   |   MRPC   |   CoLA   |   QNLI   |   RTE    |  STS-B   |   Avg.   |\n| :----- | :--------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: |\n| LoRA   |    0.3M    |   95.1   |   89.7   |   63.4   |   93.3   |   78.4   | **91.5** |   85.2   |\n| NEAT   |    0.3M    | **95.2** |   90.0   |   64.8   |   92.3   |   82.7   |   90.7   |   86.6   |\n| AuroRA | **0.075M** | **95.2** | **91.9** | **65.1** | **93.4** | **85.2** | **91.5** | **87.1** |\n\n*Table E. Performance and parameter comparison of LLaMA3-8B on Commonsense Reasoning benchmarks.*\n\n| Method   | Params.  |  BoolQ   |   PIQA   |   SIQA   | HellaSwag | WinoGrande |  ARC-e   |  ARC-c   |   OBQA   |   Avg.   |\n| :------- | :------: | :------: | :------: | :------: | :-------: | :--------: | :------: | :------: | :------: | :------: |\n| LoRA     |  56.6M   |   70.8   |   85.2   | **79.9** |   91.7    |    84.3    |   84.2   |   71.2   |   79.0   |   80.8   |\n| SineLoRA |  56.6M   |   72.4   |   86.5   |   79.8   |   94.0    |  **85.2**  |   87.6   |   78.1   | **85.0** | **83.6** |\n| AuroRA   | **3.5M** | **72.5** | **87.4** |   79.0   | **94.2**  |    83.0    | **89.3** | **78.8** |   84.8   | **83.6** |\n\nAt the same time, we wish to responsibly point out that, unlike other recent non-linear variants, AuroRA transforms the purely linear structure of vanilla LoRA into an MLP-like structure. This not only introduces non-linearity but also breaks LoRA's inherent low-rank bottleneck. \n\n**We also commit to revising our claims accordingly, citing and expanding our discussion on the other PEFT methods you mentioned [3,4,5,6,7] in a future version of our paper.**\n\n---\n\n> W4: Some missing related works in the PEFT literature: HFT [3], LISA [4], BAdam [5], Owlore [6], Galore [7].\n\nWe appreciate you bringing this to our attention! We will amend the related work section to include these references [3,4,5,6,7] in the final manuscript.\n\n---\n\n> Q1: PEFT methods are generally more useful in fine-tuning tasks, such as instruction following, math, or coding. How is AuroRA's performance in those tasks?\n\nThank you for your very insightful question! Firstly, we would like to point out that in Section 3.3, we apply AuroRA to the Subject-driven Image Generation task, **which is a form of multimodal instruction-following**. As demonstrated in Figures 4 and 6, AuroRA exhibits significantly better adherence to prompts compared to vanilla LoRA. \n\nFurthermore, to address your concerns, we conduct extensive experiments on **both math and coding tasks**. The results, presented in Table F, clearly show that AuroRA possesses robust generalization capabilities and maintains superior performance across these diverse tasks.\n\n*Table F. Performance of LLaMA2-7B on math (GSM8K, MATH) and coding (Human-Eval) tasks.*\n\n| Method           |  GSM8K   |   MATH   | Human-Eval |   Avg.   |\n| :--------------- | :------: | :------: | :--------: | :------: |\n| Full Fine-Tuning |   66.5   |   19.8   |    19.9    |   35.4   |\n| LoRA             |   60.6   |   16.9   |    14.8    |   30.8   |\n| AuroRA           | **64.8** | **18.4** |  **22.6**  | **35.3** |\n\n---\n\n> Q2: In Figure 1, it is shown that AuroRA even outperforms full-parameter fine-tuning. Is there an intuitive explanation for this phenomenon?\n\nWe believe this advantage stems from the fact that AuroRA trains only a very small number of parameters (specifically, 6.18% to 25% of those in LoRA, and 0.04% to 0.06% of those in full fine-tuning). This approach prevents the catastrophic forgetting of general knowledge acquired during the pre-training phase. Furthermore, we consider this to be a demonstration of the robust stability of the AuroRA method, as it avoids overfitting to the dataset. To further illustrate this, we provide an intuitive example in Appendix C that highlights the advantages of AuroRA's non-linearity compared to vanilla LoRA.\n\n---\n\n**Reference:**\n\n[1] LoRAN: Improved low-rank adaptation by a non-linear transformation, EMNLP'24\n\n[2] Neat: Nonlinear Parameter-efficient Adaptation of Pre-trained Models\n\n[3] HFT: Half Fine-Tuning for Large Language Models, ACL'25\n\n[4] LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning, NeurIPS'24\n\n[5] BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models, NeurIPS'24\n\n[6] Outlier-weighed Layerwise Sampling for LLM Fine-tuning, ACL Findings'25\n\n[7] Galore: Memory-efficient llm training by gradient low-rank projection, ICML'24\n\n[8] Sine activated low-rank matrices for parameter efficient learning, ICLR'24"}, {"review_id": "9nAPzh0KG2", "reviewer": "Reviewer_w5vP", "summary": "The paper is about parameter efficient fine tuning in deep learning.  The authors begin with LoRA, which uses an additive linear bottleneck to alter the function of a network, and add non-linearities to the bottleneck to enhance its capability.  In this case the non-linearities comprise an MLP-like layer and a b-spline function.\nResults seem pretty good.", "questions": "Were the authors able to reproduce the results reported from other literature?\n\nDespite the experimental (ablation) evidence, I'm left wondering from a theoretical point of view why one needs a fixed non-linearity when there is already an adaptive one.  The non-linearities themselves are supposed to be universal approximators; that is why they work in regression at all.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nNovelty: the authors introduce a non-linear LoRA bottleneck with adaptive nonlinear layer, which comprises fixed (with tanh and self-projection) and learnable (B-spline basis functions with learnable weights) nonlinearity.\nEffectiveness: the technique matches or surpasses full fine-tuning performance with only 6.18%, being 25% of LoRA’s parameters; impressive performance gains on common sense reasoning tasks with Llama 3 with less than 7% compared to LoRA.\nExperimentation: the paper demonstrates comprehensive evaluation on NLP and CV tasks; effectiveness of proposed components verified in ablation studies.\nThere is a thorough proof of the proposition.\n\nWeaknesses:\nExperimentation: for method comparison in the NLP experiments, the authors took results from other literature, which could have been produced under different settings and are hence suboptimal. This weakens the authors’ claim of performance gain.\n\nReferences:  The citation is a little lazy; first in the way multiple citations are grouped [1,2,3,4,5,...] and secondly in the way some of the literature is after the main content as a kind of afterthought.  This leads to a lot of literature being cited but not discussed well.  That related literature is presented after the main experiments is somewhat unscientific.  It suggests that the experiments were done before doing a thorough literature search, further implying another issue: that the content is perhaps not all that novel.", "was_revised": true, "final_justification": "The authors responded to the comments quite well; it would be a better paper with the structural changes.  However, I haven't changed the score as it represents a \"gut feeling\" in the context of there being lots of PEFT techniques and this being a selective venue.", "rebuttal": "Thank you for your detailed and constructive feedback on our paper. We provide our point-by-point responses to your comments below.\n\n---\n\n> Q1: Were the authors able to reproduce the results reported from other literature?\n\nTo address your concerns, we reproduce the results for all baselines on the Commonsense Reasoning benchmarks using LLaMA3-8B as the base model. The experiments are conducted on an NVIDIA L20 (48GB) GPU, strictly following the hyperparameter configurations from [1]. The results, detailed in Table G, show that the overall outcomes are consistent with those reported in our paper. Furthermore, it is evident that AuroRA achieves the strongest performance with the minimum number of trainable parameters.\n\n*Table G. Reproduction results for LLaMA3-8B on the Commonsense Reasoning benchmarks.*\n\n| Method | Params.  |  BoolQ   |   PIQA   | SIQA     | HellaSwag | WinoGrande |  ARC-e   |  ARC-c   |   OBQA   |   Avg.   |\n| :----- | :------: | :------: | :------: | -------- | :-------: | :--------: | :------: | :------: | :------: | :------: |\n| LoRA   |  56.6M   |   70.8   |   85.1   | **79.7** |   92.0    |    84.3    |   84.2   |   71.5   |   79.0   |   80.8   |\n| PiSSA  |  83.8M   |   67.2   |   81.0   | 77.2     |   83.5    |    78.7    |   77.7   |   63.4   |   74.9   |   75.5   |\n| MiLoRA |  56.6M   |   68.8   |   86.7   | 77.2     |   92.9    |  **85.6**  |   86.8   |   75.5   |   81.8   |   81.9   |\n| AuroRA | **3.5M** | **72.5** | **87.4** | 79.0     | **94.2**  |    83.0    | **89.3** | **78.8** | **84.8** | **83.6** |\n\n---\n\n> Q2: Despite the experimental (ablation) evidence, I'm left wondering from a theoretical point of view why one needs a fixed non-linearity when there is already an adaptive one. The non-linearities themselves are supposed to be universal approximators; that is why they work in regression at all.\n\nWe appreciate this highly insightful question! In line with our proposed \"Coarse Fitting\" and \"Fine Fitting\", the fixed non-linearity component serves to stabilize the optimization in its early phases. This establishes a smoother optimization environment that is easier to explore, thereby avoiding the issue of converging to a suboptimal local minimum, which can occur when using a purely learnable non-linearity in very low-rank settings (such as r=2). For a detailed analysis of the approximation error, please refer to Appendix D of our paper.\n\n---\n\n> References: The citation is a little lazy; first in the way multiple citations are grouped [1,2,3,4,5,...] and secondly in the way some of the literature is after the main content as a kind of afterthought. This leads to a lot of literature being cited but not discussed well. That related literature is presented after the main experiments is somewhat unscientific. It suggests that the experiments were done before doing a thorough literature search, further implying another issue: that the content is perhaps not all that novel.\n\nThank you for your valuable suggestion! We would like to clarify that our experiments were not conducted before our literature survey. At the same time, we will revise our manuscript following your suggestions. Furthermore, we acknowledge that due to page limitations, we were unable to discuss some related works, and we commit to expanding discussion in a future version of the paper.\n\n---\n\n> **Paper Formatting Concerns:** Line 194-198, the complexity of matrix multiplication of A (d_out r) and B (r d_in) should be d_out r d_in? I could be wrong. Applies to all results here.\n\nThank you for your comment! We would like to clarify that there is no error in lines 194-198. You are correct that the complexity of an explicit matrix multiplication of $\\mathbf{B}\\mathbf{A}\\in\\mathbb{R}^{d_{\\mathrm{out}}\\times d_{\\mathrm{in}}}$ is indeed $\\mathcal{O}(d_{\\mathrm{out}}\\,r\\,d_{\\mathrm{in}})$. However, this explicit computation is not performed in LoRA. Instead, the operation is carried out in two sequential steps:\n\n1. $M = X \\mathbf{A}^\\top$ is computed, with a complexity of $\\mathcal{O}(b\\,d_{\\mathrm{in}}\\,r)$.\n2. $\\Delta \\mathcal{W} = M \\mathbf{B}^\\top$ is computed, with a complexity of $\\mathcal{O}(b\\,r\\,d_{\\mathrm{out}})$.\n\n---\n\n**Reference:**\n\n[1] MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning, NAACL'25"}, {"review_id": "dVP78pnH9q", "reviewer": "Reviewer_4VGD", "summary": "AuroRA is a modification of the standard LoRA PEFT architecture which introduces non-linearities into the low rank bottleneck to capture more complex subspace relationships with minimal additional parameters, enabling AuroRA to outperform linear techniques such as LoRA and MoSLoRA across NLP and CV tasks.\n\nAuroRA introduces the \"Adaptive Nonlinear Layer\" (ANL) as the non-linearity used in the low rank bottleneck. The ANL is a combination of (1) fixed non-linearities with an intermediate 'self projection' similar to the subspace mixer of MoSLoRA which facilitates smoother fitting of subspace relationships, and (2) a learnable non-linearity with learnable splines similar to that of Kolmogorov–Arnold Networks (KANs) which enables tighter fitting of relationships within the low-rank subspace. These non-linearities enable AuroRA to explore much wider $\\Delta w$ parameter spaces than classical PEFT methods, which the authors identify to be the primary weakness of LoRA and LoRA-like adaptors.", "questions": "Did the authors perform alpha tuning to determine the best LoRA, MoSLoRA and AuroRA hyperparameters for the comparison ablations, and if so what values were chosen?\n\nWere static or dynamic weights utilised when collecting the inference results in the main body of the paper?\n\nCan the authors justify how it is mathematically possible to linearize AuroRA considering its data-dependent activations and what the implications are considering it re-introduces a linear low-rank bottleneck.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\nWhile the introduction of bottleneck MLPs for PEFT is not entirely new, their use as a replacement for LoRA-like weight adaptation is novel and shows incredibly promising results.\n\nThe authors perform a wide variety of ablations which support their design decisions and perform experiments with a selection of standard models and tasks that are typical to the NLP and CV domains. Despite AuroRA often having the fewest or close to the fewest parameters it typically ranks highest\n\n## Weaknesses\nThe authors only explore smaller ranks for LoRA and AuroRA (a maximum of 16), and while this does demonstrate AuroRA's efficacy in highly parameter-efficient scenarios it doesn't paint a full picture as to how AuroRA compares at larger ranks; for example it is fairly common for fine-tunes of larger models to use ranks as high as 64 or even 256 for more complex tasks, and the paper lacks the experimental evidence for AuroRA's performance in such use-cases.\n\nThe author's also seem to lack information in the main body of the paper for important hyperparameters such as alpha which determines the scaling used for LoRA, MoSLoRA and (by confirming in the source code) AuroRA. The alpha choices are listed in the appendix for the primary AuroRA training experiments, but the alpha choices for AuroRA and LoRA variants are unclear for the ablations which show AuoRA to be superior at different parameter counts or ranks. Additionally, there are no ablations over the importance of alpha for AuroRA which is worrying considering sub-optimal choices of alpha for similar PEFT methods can greatly impact convergence.\n\nFinally, I am highly skeptical of the proposed \"weight merging\" method during inference, where AuroRA switches from $B\\sigma(Ax)$ to $B\\sigma(A)x$; the purpose of weight merging is to fold PEFT weights into the original model weights to replicate identical behaviour while removing and additional parameters, however due to AuroRA's non-linearity it is not numerical possible to do this. While the results in the Appendix do show empirically that this procedure is still capable of retaining similar performance to the \"dynamic\" AuroRA implementation it also lacks any real analysis on how this is possible and if it generalises to any arbitrary AuroRA weights. The lack of inclusion of this rather important detail in the main body of the paper is also worrying and also the questions as to if the results in the main body of the paper are from \"dynamic\" weights or \"static\" weights which further harms the clarity of the author's claims.\n\nHowever if we ignore the dubious weight merging and consider the minimal computational impact of dynamic weights even at inference, AuroRA's performance and efficiency is still commendable!", "was_revised": true, "final_justification": "The authors addressed all my major concerns, and I now believe AuroRA to be a strong and impactful contender in the world of PEFT techniques, especially due to the justification that AuroRA's non-linear components can be \"baked in\" to a linear approximation that fully allows weight merging with only a minor loss in performance which is something novel and highly advantageous over other PEFT methods that utilise non-linearities.", "rebuttal": "We sincerely appreciate your insightful comments and valuable suggestions on our work! Below are our point-by-point responses to the issues you have raised.\n\n---\n\n> W1: The authors only explore smaller ranks for LoRA and AuroRA (a maximum of 16), and while this does demonstrate AuroRA's efficacy in highly parameter-efficient scenarios it doesn't paint a full picture as to how AuroRA compares at larger ranks; for example it is fairly common for fine-tunes of larger models to use ranks as high as 64 or even 256 for more complex tasks, and the paper lacks the experimental evidence for AuroRA's performance in such use-cases.\n\nThank you for your very insightful question! To address your concerns, we have conducted additional experiments on **more challenging mathematical tasks**. Using LLaMA2-7B as the base model, we compare the performance of LoRA and AuroRA, **setting the rank for both methods to 64** and the hyperparameter alpha to 128. As shown in Table H, we observe that AuroRA continues to exhibit superior performance even with a higher rank and on more complex tasks. Due to our limited computational resources, we are unable to perform tests on larger-scale models and leave this as an avenue for future work.\n\n*Table H. Performance comparison on the GSM8K and MATH datasets using LLaMA2-7B with a rank of 64.*\n\n| Method           |  GSM8K   |   MATH   |   Avg.   |\n| :--------------- | :------: | :------: | :------: |\n| Full Fine-Tuning |   66.5   |   19.8   |   43.2   |\n| LoRA             |   60.6   |   16.9   |   38.7   |\n| AuroRA           | **65.2** | **18.6** | **41.9** |\n\n---\n\n> Q1: Did the authors perform alpha tuning to determine the best LoRA, MoSLoRA and AuroRA hyperparameters for the comparison ablations, and if so what values were chosen?\n\nThank you for your feedback! We would like to clarify that, in accordance with prior works [1,2,3], the hyperparameter alpha is typically set to be equal to the rank or twice the rank. In our experiments, the alpha values for both LoRA and MoSLoRA strictly follow the settings from their respective original papers, which is twice the rank. For example, in the sensitivity analysis on rank, we set the alpha values for LoRA, MoSLoRA, and AuroRA to {4, 8, 16, 32} corresponding to ranks of {2, 4, 8, 16}.\n\n---\n\n> Q2: Were static or dynamic weights utilised when collecting the inference results in the main body of the paper?\n\nThank you for your detailed comment! We wish to responsibly point out that all experimental results in the main body of the paper are collected using a \"dynamic training, static merging\" strategy. As observed in Table 7 (Appendix B.1), when a \"fully dynamic\" strategy is employed, the AuroRA-D variant actually achieves better performance compared to AuroRA, albeit at the cost of additional runtime. Furthermore, we provide a complete and detailed algorithm for AuroRA in Appendix B.2 to more clearly illustrate the processes during the training and inference stages. Finally, respectfully following your suggestion, we will be sure to emphasize this point more explicitly in the main body of our paper in a future revision.\n\n---\n\n> Q3: Can the authors justify how it is mathematically possible to linearize AuroRA considering its data-dependent activations and what the implications are considering it re-introduces a linear low-rank bottleneck.\n\nWe would first like to clarify the rationale for our \"dynamic training, static inference\" strategy is to maximize model expressiveness during training while allowing for weight merging to eliminate any additional computational overhead at inference time. **To address your concerns, we provide the following mathematical analysis of this strategy's reasonableness.**\n\nThe dynamic and static forms are defined as $h_D = \\mathbf{B}\\sigma(\\mathbf{A}x)$ and $h_S = (\\mathbf{B}\\sigma(\\mathbf{A}))x$, respectively. We aim to demonstrate that once the model converges through dynamic training, the static form used during inference closely approximates the dynamic form, such that $h_D \\approx h_S$. Let's consider the Taylor series expansion of our Adaptive Nonlinear Layer (ANL), $\\sigma(\\cdot)$, around the origin. Let $z = \\mathbf{A}x$, we have $\\sigma(z) \\approx \\sigma(0) + \\sigma'(0)z + \\mathcal{O}(\\|z\\|^2)$, where $\\sigma(0)$ is the function's value and $\\sigma'(0)$ is its Jacobian matrix at the origin. As σ is applied element-wise, its Jacobian is a diagonal matrix. For the dynamic form, substituting the expansion gives: $\\mathbf{B}\\sigma(\\mathbf{A}x) \\approx \\mathbf{B}(\\sigma(0) + \\sigma'(0)\\mathbf{A}x) = \\mathbf{B}\\sigma(0) + \\mathbf{B}\\sigma'(0)\\mathbf{A}x$. In this case, the $\\mathbf{B}\\sigma(0)$ term functions as a constant bias, which is independent of the input $x$. For the static form, we analyze the application of $\\sigma$ to matrix $\\mathbf{A}$. Each element $a_{ij}$ is transformed as $\\sigma(a_{ij}) \\approx \\sigma(0) + \\sigma'(0)a_{ij}$. For the entire matrix, this yields $\\sigma(\\mathbf{A}) \\approx \\sigma(0) \\cdot J + \\sigma'(0)\\mathbf{A}$, where $J$ is a matrix of ones. The static form then becomes: $(\\mathbf{B}\\sigma(\\mathbf{A}))x \\approx \\mathbf{B}(\\sigma(0)J + \\sigma'(0)\\mathbf{A})x = \\mathbf{B}\\sigma(0)Jx + \\mathbf{B}\\sigma'(0)\\mathbf{A}x$. Upon comparison, we observe that the first-order term, $\\mathbf{B}\\sigma'(0)\\mathbf{A}x$, is identical in both forms. For the zeroth-order term, the fixed non-linearity we adopt has $\\tanh(0) = 0$, and a negligible bias $\\mathcal{L}(0) = \\mathbf{w}_s \\cdot \\mathbf{s}(0)$ exists only in the learnable non-linearity. Overall, we can consider $\\sigma(0) \\approx 0$. Therefore, the distinction between the dynamic and static forms mainly resides in the higher-order terms. The dynamic form preserves the complete, input-specific non-linear information for each $\\mathbf{A}x$, whereas the static form \"bakes in\" or averages this high-order information into the merged weights. Since all learnable components have been optimized during training, this difference is minor, a point that is also empirically supported by our results in Table 7 of Appendix B.1.\n\nFurthermore, we present an intuitive example in Appendix C, which shows that AuroRA can fit more complex matrix structures thanks to its element-wise non-linear application. This indicates that by transforming the purely linear mapping of the original LoRA into an MLP-like structure, AuroRA can capture more information during training, thereby breaking the low-rank bottleneck.\n\n---\n\n**Reference:**\n\n[1] LoRA: Low-Rank Adaptation of Large Language Models, ICLR'22\n\n[2] Mixture-of-Subspaces in Low-Rank Adaptation, EMNLP'24\n\n[3] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models, NeurIPS'24"}]

Author rebuttal:

Meta-review: The reviewers wrote that the paper makes a strong and well-motivated contribution by extending LoRA with nonlinear transformations, addressing the known low-rank bottleneck in a novel and practical way. The introduction of both fixed and learnable nonlinearities is clearly explained and supported with the experimental results demonstrating impressive performance across NLP and CV tasks while using significantly fewer parameters than full fine-tuning or LoRA. The evaluation is broad, spanning GLUE, commonsense reasoning, vision benchmarks, and generation, and the ablation studies convincingly validate the importance of each component, including reduced rank sensitivity. Overall, the work is clearly written, thoroughly evaluated, and provides a meaningful advancement in parameter-efficient fine-tuning. Most of the weaknesses pointed out by the reviewers are addressed in the rebuttal. Hence, it will be accepted. We would suggest the authors to address the remaining concerns in the camera ready version.
Decision: Accept (spotlight)