ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models
Authors: Raghav Singhal
Kaustubh Ponkshe
Rohit Vartak
Praneeth Vepakomma
Venue: ICLR 2026 Poster (2026)
Keywords: LoRA
Low-rank adaptation
PEFT
Parameter-Efficient Fine-Tuning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We introduce ABBA, a PEFT method that enhances expressivity by decoupling low-rank updates from pre-trained weights via a Hadamard product, consistently improving over SOTA methods.

Abstract: Large Language Models have demonstrated strong performance across a wide range of tasks, but adapting them efficiently to new domains remains a key challenge. Parameter-Efficient Fine-Tuning (PEFT) methods address this by introducing lightweight, trainable modules while keeping most pre-trained weights fixed. The prevailing approach, LoRA, models updates using a low-rank decomposition, but its expressivity is inherently constrained by the rank. Recent methods like HiRA aim to increase expressivity by incorporating a Hadamard product with the frozen weights, but still rely on the structure of the pre-trained model. We introduce ABBA, a new PEFT architecture that reparameterizes the update as a Hadamard product of two independently learnable low-rank matrices. In contrast to prior work, ABBA fully decouples the update from the pre-trained weights, enabling both components to be optimized freely. This leads to significantly higher expressivity under the same parameter budget, a property we validate through matrix reconstruction experiments. 
Empirically, ABBA achieves state-of-the-art results on arithmetic and commonsense reasoning benchmarks, consistently outperforming existing PEFT methods by a significant margin across multiple models. Our code is publicly available at: https://github.com/CERT-Lab/abba.

Published as a conference paper at ICLR 2026
ABBA-ADA ABBA-ERAPTERS: EFFICIENT AND EXPRESSIVE FINE- TUNING OF FOUNDATION MODELS
Raghav Singhal*!, Kaustubh Ponkshe*1, Rohit Vartak*2, Praneeth Vepakomma 1.3
Mohamedd bin Zayed University of Artificial Intelligence, Duke University, "assachuseets Institute of Technology
ABSTRACT
Large Language Models have demonstrated strong performance across wide range of tasks, but adapting them efficiently to new domains remains a key challenge. Parameter-Efficient Fine-Tuning (PEFT) methods address this by introducing lightweight, trainable modules while keeping most pre-trained weights fixed. The prevailing approach, LoRA, models updates using low-rank decomposition, but its expressivity is inherently constrained by the rank. Recent methods like HiRA aim to increase expressivity by incorporating Hadamard product with the frozen weights, but still rely on the structure of the pre-trained model. We introduce A ABBA, new PEFT architecture that reparameterizes the update Hadamard product of two independently learnable low-rank matrices. In contrast to prior work, ABBA fully decouples the update from the pre-trained weights, enabling both components to be optimized freely. This leads to significantly higher expressivity under the same parameter budget, a property we validate through matrix reconstruction experiments. Empirically, ABBA achieves state-of-the- art results on arithmetic and commonsense reasoning benchmarks, consistently outperforming existing PEFT methods by significant margin across multiple models. Our code is publicly available at: hotps: //github. wwm/ Lab/ abba.
INTRODUCTION
Large Language Models (LLMs) have become the backbone of modern NLP systems (1-7), demon- strating strong generalization across a wide range of tasks (8; 9). However, adapting these models to new tasks typically requires full fine-tuning (FT), which is computationally and memory intensive. Crramtter-Effcieen Tunnng (PEFT) methods address this challenge by introducing small number of trainable parameters while keeping the majority of model weights frozen (10-12). Among PEFT approaches, Low-Rank Adaptation (LoRA) (10) the most widely adopted due to its sim- plicity and efficctiveness It models the weight update AW as the product of two low-rank matrices, providing a compact parameterization. However, this formulation inherently constrains updates to low-dimensional subspace, limiting expressivity. Several extensions attempt to overcome this limitation: LoRA-XS (12) restricts updates to a frozen, SVD-derived subspace and training only a small higher-rank intermediary matrix while DoRA (11) modifies only the directional component of the weights via low-rank updates. Despite these architectural variations, all retain LoRA's A's core updates remain strictly low-rank and limited in expressivity.
constraint:
HiRA (13) addresses LoRA's limited expressivity by applying a Hadamard product between low- rank update and the frozen pre-trained weights Wo. enabling updates that can, in principle, attain full rank. However, HiRA's expressivity remains tightly coupled to Wo. as the learned adapters merely modulate the pre-trained weights rather than generating the full update independently. For example, if the target update equals diag( (W). HiRA must learn adapters that approximate the identity matrix, an orthonormal structure that challenging for low-rank modules to represent accurately (Figure 2).
In this work, we introduce ABBA, a novel architecture framework that reparameterizes the weight update as the Hadamard product of two fully learnable low-rank matrices (see Figure 1). Each component is independently formed via low-rank decomposition (B, A₁ and BzA2), resulting
denotes equal contribution. Author order decided randomly.

---

Published as a conference paper at ICLR 2026
56
AIBA: dos C.225
Frozen
Trainable
FUlFFT: A
14
Con
lergel Oyrimam
B2
A
W
+
Hadamard Product
B1
A1
30 62
Illustration of ABBA's parameterization, where the update is expressed as the Hadamard product of two learnable low-rank matrices. Right: A toy experiment demonstrating ABBA's optimization behavior. We first train a 2-layer MLP classify the first 8 MNIST digits, then fine-tune it to recognize the last 2. ABBA converges faster and achieves better final performance.
Figure Left:
in highly expressive update while maintaining parameter counts. Unlike HiRA, ABBA is fully decoupled from the pretrained weights Wo, allowing both components to be optimized without structural constraints. The name ABBA reflects the four low-rank matrices that define the architecture.
We analyze expressivity through a matrix reconstruction task, where Hadamard-structurred updates consistently outperform standard LoRA decompositions under the same parameter budget (Figure 2). This demonstrates that ABBA can represent broader class of updates than LoRA within identical constraints. A toy MNIST experiment (Figure 1) shows that ABBA converges to solution significantly closer to the true optimum compared to LoRA and HiRA, indicating that its increased expressivity is also practically accessible during learning. We present an exact reformulation of the ABBA update using the Khatri- Rao matrix factorization, enabling efficient implementation without approximation. Empirically, ABBA consistently outperforms existing PEFT methods across a broad range of tasks and models, all within the same lower parameter budget. In addition, conduct extensive ablation studies to validate the effectiveness of our design choices and hyperparameter settings. Our key contributions are summarized as:
We propose ABBA, a novel PEFT architecture that models the weight update as the Hadamard product of two independently learnable low-rank matrices. This formulation enables highly expressive, high-rank updates while preserving strict parameter efficiency.
We provide empirical analyses of ABBA's expressivity, showing that Hadamard based decomposi- tion consistently outperforms standard low-rank methods in matrix reconstruction.
We introduce an exact and efficient reformulation of ABBA using Khatri-Rao factorization, enabling scalable and practical implementation without compromising expressivity.
Through extensive experiments on four models across arithmetic and commonsense reasoning tasks, we demonstrate that ABBA achieves state-of-the-- art performance, significantly outperforming existing PEFT methods under equal or lower parameter budgets.
METHODOLOGY
2.1 PRELIMINARIES
Full Fine-Tuning. Given pre-trained weight matrix Wo E Rmxx full FT updates all parameters via W Wo + AW, introducing m x n trainable parameters per layer. This quickly becomes impractical due the high memory and compute overhead.
LoRA (10). LoRA mitigates this by modeling the update as a low-rank decomposition: AW sBA, where B m A E Brxn and is scaling factor. This reduces the number of trainable parameters to r(m + n). with & min(m, n)). LoRA can represent any update of rank at most r, but cannot express higher-rank updates. Moreover, the projected gradient onto the weight space is also low-rank. While effective for simpler tasks, this limitation becomes significant in settings requiring high-rank updates gradients (14; 15).
2

---

Published as a conference paper at ICLR 2026
HiRA (Hadamard High-Rank Adaptation) (13). HiRA lifts LoRA's rank limitation by modulating its low-rank update with an element wisse (Hadamard) product with the frozen pre-traines weight Wo:
AW Wo C (BA), where @ denotes the Hadamard product.
(I)
This leverages the property that the Hadamard product of two matrices W1 and W2 with ranks 12 and 12 respectively satisfies rank(W; W2) r1 r2- Thus, HiRA can produce updates of rank up to rur, where To = ran) potentially addressing the low-rank limitation of LoRA. Additionally, the gradient projected onto W imply greater expressivity. Because HiRA's update is element-wise tied to Wo, it is restricted to subspace defined by the pre-trained weights. This dependence can hinder generalization. especially in out-of- scenarios. As shown in Section 2.4, HiRA reduces reconstruction error over LoRA only when the element-wiss ratio of the oracle update Wo is itself low-rank.
longer low-rank. However, higher rank does not necessarily
2.2 IMPROVING THE EXPRESSIVITY OF HIRA
A natural way to overcome HiRA's expressivity constraint is to make the matrix WA, learnable:
AW W, 3 (BA), where WA. E gmxn is trainable.
(2)
However, this reintroduces the full m parameter cost of Wn. negating LoRA's core efficiency advantage. Even if Wn is fixed but not equal to Wo, the additional memory required to store significantly increases the overhead. In contrast, HiRA sets W,, Wo. which is already stored, thereby preserving LoRA's parameter and memory efficiency. This raises a critical question:
Can achieve greater expressivity and high-rank learning while maintaining the parameter and memory efficiency of LoRA?
Importantly, we note that full-rank updates are not always necessary. Moreover, W, itself does not need to be full-rank. Since any x matrix has rank at most = min(m, n), modulation matrix with rank above To/r offers no additional expressivity for the Hadamard product.
2.3 ABBA-ADAPTERS IMPROVED EXPRESSIVITY AND HIGH RANK, YET EFFICIENT
As discussed above, high-rank updates do require W,, to be full-rank. Leveraging this insight, we reparameterize W,, as the Hadamard product of two independently learnable low-rank matrices, resulting in the following formulation:
AW = sBB)ii) @ (B2A2).
(3)
where B, Rmxr, A X and B2 E mmrr A2 Brzxn, with ri,72 & min(m,,))
and s is a scaling factor for stability. This parameterization introduces only (r1 + +2))(m + parameters, significantly fewer than full FT, and achieves an effective rank up to 71 To maximize expressivity under fixed parameter budget, we set rı T2, as further supported by empirical results in Section 4.2. This preserves HiRA's ability to produce high-rank updates while improving expressivity, since all four matrices are independently learned. For fair comparison with LoRA and other PEFT baselines, we match parameter counts by setting I'1 = r2 = r/2, so that ABBA and other methods use equivalent parameter budgets.
Initialization of ABBA Adapters. HiRA fixes the modulation matrix as Wt = Wo, directly tying the update the pretrained weights. In contrast, ABBA makes this matrix fully learnable by reparameterizing as WA = B, A₁ We initialize the first adapter pair (B,, A,) using the top-r components from a truncated SVD of Wo, and the second pair (B2, A2) using the standard LoRA initialization: B2 as zeros and A2 with Kaiming uniform sampling.
Un, Ery, r +- SVD,,
(4)
Bi - U,, 1// 11/ A
B₁ 0,
A₂ A2
(5)
By the Eckart- Young- Young-Mirsky (EYM) theorem (16; 17), the truncated SVD yields the optimal rank-r1 approximation of Wo. This hybrid initialization anchors the update close to meaningful low-rank subspace, while enabling the second adapter pair to explore task- sascispecifiic directions during training. We validate the effectiveness of this strategy empirically in Section 4.1.
3

---

Published as a conference paper at ICLR 2026
Making ABBA Memory-Efficient. While ABBA is clearly parameter-efficient analyzing its memory footprint during training is more subtle. In LoRA, the update AW BA is applied as AWz B(A),, allowing intermediate computations to remain low-rank. Only the activation Ax E RR and the adapter weights need to be stored additionally, avoiding the materialization of the full x matrix BA. In contrast, ABBA's update AW (1) challenge. A naive implementation would require constructing both B, and B2A2, followed by their elementwise product, resulting in the storage of multiple full m matrices. Moreover, unlike LoRA, the Hadamard product does not distribute over matrix-vector multiplication, so computing By (Aga) does not help incorporate the other matrices.
) @ (B2A2) poses
Theorem (Khatri-Raa Factorization (18)). Let BI B2 E I Then, A1) @ (A
(B2A2) (B1 3r (2) Or A where G, denotes the row-wise Khatri-Rad Rao product.
To address this, we use Theorem 1 to rewrite ABBA in a LoRA-like form: let B B1 O, Bg and Akr (AT Or AT)T
T. The update becomes AWr = Bar (Au=)) avoiding any full-rank construction.
This enables ABBA to match LoRA's compute and memory efficiency, while offering significantly higher expressivity, and remain more efficient than variants like HiRA, as shown in Section 4.4.
2.4 EXPRESSIVITY OF ABBA
The expressivity of matrix reparameterization can be evaluated by its ability to accurately reconstruct arbitrary target matrices, relative to alternative parameterizations.
LoRA and HiRA. LoRA, the weight update is modeled low-rank decomposition AW = B.A. For any matrix M E Rmxx, the reconstruction error of this approximation is defined as:
{rr = | BA]
(6)
and is lower-bounded by the classical EYM theorem (16; 17), which states that the optimal rank-r approximation is given by the truncated SVD. Since LoRA adapter of rank can only represent updates with rank at most r, EYM provides a theoretical minimum for (r)) LoRA achieves this bound exactly when the learned adapters align with the top singular components of M; otherwise, practical considerations such as suboptimal initialization may lead to performance gap.
similar bound can be derived for HiRA when the modulation matrix Wo has all nonzero entries. In this case, the optimal Htadamard-structure approximation can be obtained by element-wise dividing W by Wo. followed by applying truncated SVD to the resulting matrix. The expressivity advantage of HiRA over LoRA arises only if rank( A//W)) < rank(AW) Otherwise, for a general update AW, HiRA has the same reconstruction error bound as LoRA, as characterized by the EYM theorem.
ABBA vs. LoRA: Reconstruction. Unlike SVD-based methods, ABBA does not admit a closed- form solution for its low-rank factors (see Appendix B for understanding why). We thus evaluate its expressivity by comparing the reconstruction error versus other methods. Given a target matrix M E mm x we define the reconstruction error for a method X E (LoRA, ABBA] at rank r as: Ex,+ minMy.- ||M Mx 1| where the LoRA approximation is the truncated SVD MSvD,r UE,V VT, and the ABBA approximation is given by MABBA, = (B, Ai) @ (B2A2).
Prior work (19) establishes a loose upper bound EABBA, < ELORA, which holds trivially by setting one ABBA factor to the rank-r SVD and the other to all-ones matrix. Empirically, however, they observe a stronger trend: EABBA, S EL which suggests that ABBA can match or outperform a rank-2r SVD approximation using only rank r. However, this is not guaranteed for arbitrary matrices, as the quality of reconstruction depends on the spectral properties and structure of the matrix. The only
known thcoretical comparison between the two is the bound: ELORA EABBA, S E of where o; the singular values of M (19). While this bound offers some insight, it is loose and does not guarantee a strict ordering between the reconstruction errors.
Giivnn U, V E Rmx the row-wise Khatri-Rao product U Or V E Bmxna is defined by IU Or V]: := [V Usa Vo, ..., Un, Vi), where V is the i-th row of V.
4

---

Published as a conference paper at ICLR 2026
ABBA
LoRA
Orthogonal
Low-Rank Rank
Sparse
Random
Roconstruction Error
0.92
0.89
0.24
0.70
0.83
0.20
0.59
0.47
0441
0.16
0.24
0.17
24
40
24
40
56
72
72
24
40
Rank: LoRA ABBA (/1 =2
Figure 2: Empirical Reconstruction Errors. We compare ABBA and LoRA decompositions across various matrix types by measuring reconstruction error E(r) under equal parameter budgets. For each LoRA rank we set ABBA ranks to ri = = r/2 for a fair comparison. ABBA consistently achieves significantly lower reconstruction error than LoRA, across all matrix types.
To better understand practical behavior, we empirically evaluate the reconstruction error of different parameterizations across diverse matrix types. As shown in Figure 2, the ABBA-basee Hadamard reparameterization consistently achieves lower reconstruction error than standard LoRA, indicating greater expressivity. This aligns very well with prior work leveraging Hadamard structures for efficient and expressive matrix representations (20), further validating our formulation.
2.5 STABILITY OF THE ABBA UPDATE
The scaling factor critical in controlling the optimization dynamics of the ABBA update. must be chosen carefully to scale appropriately with the ranks and r2 of the underlying low-rank factors. set too low, learning stagnates; if too high, training may diverge. While ABBA resembles LoRA in structure, its effective rank is 1172, and the scaling behavior must reflect this increased capacity. Inspired by the scaling analysis LoRo (21), where stability achieved under complexity of O(1) with respect to rank, one might expect similar scaling for ABBA. However, since ABBA and LoRA inhabit different parameter spaces, these arguments do not transfer directly. To formalize this, we introduce the notion of rank-stabiliy, for ABBA in Definition I, which ensures that forward and backward dynamics remain well-conditioned as r1 and F2 vary.
Definition 1 (Rank Stability of ABBA Adapters (21; 22)). An ABBA adapter of the form SABBA (1) @ (B2A2) is rank-stabilized if the following conditions hold:
1. If the 2nd moment of the input is Org.rz (1) in each entry with inputs being i.i.d, then the nnd moment of the outputs of the adapter is also 3r (1) in each entry.
If the 2nd moment of the loss gradient with respect the adapter outputs +11. in each entry; then the 2nd moment of the loss gradient of the input of the adapter is also Or (1) in each entry.
Building on Definition 1, we establish Theorem that the ABBA update is indeed rank-stable. We empirically demonstrate that scaling yields improved stability, as evidenced by the gradient norm experiments shown in Figure in Appendix I.
Theorem Staailttaillity of ABBA). Let the ABBA update be AW SABBA (B1A)) A1) & (B2A2), with B1, A1, B2, As independent, mean-zero random matrices with finite variance. Then, the update satisfies the stability conditions of Definition if and only if:
SABRA E 0
Vrirz
Proof. See Appendix D.
5

---

Published as a conference paper at ICLR 2026
Final ABBA Parameterization.
We express the ABBA scaling factor as SABBA
since
<Pirg
Theorem shows that SABBA E e
) With this choice, the ABBA update is:
AW
(B) (B242).
Vrir2
Equivalently, to highlight the connection with LoRA, ABBA can be expressed as the Hadamard product two rank-stabilized LoRA adapters:
AW = (
(
BIA1)
B2A2)
(8)
VrI
/r2
3 EXPERIMENTS
We evaluate ABBA range of models, specifically Llama-3.2 IB (23), Llama-3.2 3B (23), Mistral- 7B (24), and Gemma-2 9B (25), to test its effectiveness across diverse scales and architectures. experiments multiple benchmarks to capture varied trends. Appendix L details configurations, and Appendix M lists dataset specifics. For fair comparison with LoRA and other PEFT methods, match the number of trainable parameters in ABBA by setting Tı r2 = r/2.
training
Baselines. We compare ABBA against full fine-tuning, LoRA (10), and several strong LoRA variants: IsLoRA (21), PiSSA (26), DoRA (11), LoRA-Pro (14), and HiRA (13).
3.1 COMMONSENSE REASONING
We fine-tune tueettne Llama-3. 3.2 models at 1B and 3B scales (23) on COMMONSENSEI 70K, multi-task dataset comprising eight commonsense reasoning benchmarks (27). These include OBQA (28), ARC- ERC-Challenge and ARC-Easy (29). WinoGrande (30). HellaSwag (31), PIQA (32), SIQA (33), and BoolQ (34). We evaluate performance on each dataset independently to capture task-specific generalization. We insert LoRA modules into the key, query, and value projections, the attention output, and all feedforward layers. Table reports the results. ABBA consistently outperforms all other PEFT methods across both models, and in many cases surpasses full FT.
Table 1:
Comparison of multiple FT methods on Llama-3.2 3.2 1B and 3B across eight commonsense reasoning datasets. Best results among PEFT methods are in bold.
Accuracy (t)
Model
Method
# Params
OBQA ARC-c ARC-e
Wino HellaS PIQA
SIQA
BoolQ
Avg.
Full FT
1.24 B
73.42
63.70
78.88
75.12
80.98
80.22
74.79
66.21
74.17
LoRA
22.54 M
71.83
59.13
74.32
73.87
74.96
78.13
73.75
65.96
71.49
LoooR
22.54 M
71.11
59.85
74.90
73.85
75.34
78.32
73.47
65.44
71.54
PiSSA
22.54 M
71.45
60.32
74.43
72.90
75.65
78.45
73.63
65.83
71.58
DoRA
22.92 M
71.99
60.98
77.65
73.42
76.33
78.81
73.79
65.91
72.36
Llama-3.2 IB
LoRA-Pro
22.54 M
71.68
61.11
76.37
73.12
76.89
79.24
74.02
65.79
72.28
HiRA
22.54 M
72.18
61.26
78.37
72.06
78.87
79.59
74.41
65.32
72.76
ABBA
11.27 M
71.86
63.05
78.33
73.95
80.93
80.63
75.33
65.96
73.76
ABBA 32
22.54 M
75.00
64.59
79.74
76.03
82.50
80.41
75.08
66.80
75.03
Full FT
3.21 B
81.88
75.29
88.52
85.02
91.92
85.61
80.45
70.43
82.39
LoRA
48.63 M
81.87
74.32
86.91
82.24
90.71
85.20
79.12
70.03
81.30
isLoRA
48.63 M
81.72
74.18
86.71
82.02
90.45
85.05
78.92
69.81
81.11
PiSSA
48.63 M
81.79
74.61
87.23
82.68
90.88
85.42
79.44
70.12
81.52
DoRA
49.40 M
82.04
74.87
87.61
82.90
90.76
85.63
79.68
70.43
81.74
Llama-3.2 3B
LoRA-Pro
48.63 M
81.74
75.32
87.24
83.42
90.90
85.81
79.35
71.28
81.88
HiRA
48.63 M
81.58
76.38
88.76
83.95
91.67
85.61
79.91
72.69
82.56
ABBA.=15
24.32 M
83.40
77.39
89.56
85.16
93.51
86.89
80.55
73.03
83.68
ABBA
48.63 M
85.04
79.10
89.61
85.24
92.37
86.83
80.96
73.52
84.08

---

Published as a conference paper at ICLR 2026
3.2 ARITHMETIC REASONING
We fine-tune Mistral-7B 7 (24) and Gemma-2 9B (25) 20K-sample subset of MetaMathQA (35), and evaluate their performance on GSM8K (36) and MATH (37). We insert LoRA adapters into all attention projections (query, key, value, and output) as well as both feedforward layers. We report results in Table 2. ABBA achieves superior performance over all other PEFT approaches across both models, and often outperforms full FT. We hypothesize that this effect arises because ABBA's structured parameterization implicitly regularizes training and enables more efficient task-specific adaptation than unconstrained full FT.
Table 2: Comparison of multiple FT methods Mistral-7B and Gemma-2 9B across arithmetic reasoning benchmarks. Best results among PEFT methods are in bold.
Mistral-7B
Gemma-2
Method
Params
GSM8K (t)
MATH (1)
Params
GSM8K (1) MATH (t)
Full FT
7.24 B
63.87
17.65
9.24 B
79.23
38.02
LoRA
83.88 M
61.94
15.98
108.04 04 M
76.19
36.56
LsLoRA
83.88 M
62.15
16.24
108.04 04 M
76.84
36.88
PiSSA
83.88 M
62.43
16.52
108.04 M
77.12
37.04
DoRA
85.26 M
62.65
16.64
109.88 M
77.58
37.04
LoRA-Pro
83.88 M
63.07
17.32
108.04 M
78.26
37.53
HiRA
83.88
63.15
17.44
108.04 M
78.47
38.22
ABBA
41.94 M
61.97
18.06
54.02
78.70
38.41
ABBA
83.88
66.26
18.08
108.04 M
79.76
39.18
4 ANALYSIS
4.1 INITIALIZATION STRATEGIES FOR ABBA
Initialization of the adapter matrices B1, A1 and B2, A2 is crucial to ABBA's performance. naive LoRA-style initialization, where B2 are set zero and A1, Ag Kaiming uniform, leads to training failure due to zeroed-out gradients (see Appendix C). To address this, we explore several initialization strategies that combine truncated SVD-based approximations with standard schemes, summarized in Table 3.
Inspired by PISSA-LORA (26), approach is approximate the base weight Wo at initialization. This can be done by initializing one adapter pair using the truncated SVD of Wo. and the other with scaled constant values (e.g., ones) to prevent gradient explosion (1). Another strategy initializes both adapter pairs using the top-r/2 components from the truncated SVD of /Wo (2). A variation of this approach assigns the top-r/2 components one adapter pair and the next-r
the other,
introducing greater representational diversity and yielding slightly improved results (3, 4). We also consider brid strategy, where one adapter pair approximates Wo via truncated SVD, and the other follows LoRA-style initialization (Kaiming for A, zeros for B). This configuration performs best and closely resembles the initialization used in HiRA (5).
Our final method adopts this approach: B1, Ai are initialized using the truncated SVD of Wo. while B2, Ag follow LoRA-style initialization (Ours).
Table Comparison different initialization strategies for ABBA (Mistral-7B).
Initialization Method
GSM8K MATH
(1) B1, A₁ + top-r/2 from Trunc. SVD(Wo), Bz, A2 Ones (scaled)
57.39
10.88
(2) Both adapters: top-r/2 from Trune. SVD(WW))
64.53
16.57
(3) First adapter: top-r/2, second: next-r/2 from Trunc. SVD(VWo)
64.86
17.05
(4) Second adapter: top-r/2. first: next-r/2 from Trunc. SVD( (W.)
64.79
17.16
(5) B2, A2 +- top-r/2 from Trunc. SVD(W,). B1, A₁ < LoRA Init (Zeros, Kaiming)
66.19
18.06
Ours: Bi, An +- top-r/2 from Trunc. SVD(W)), B2, Az + LoRA Init (Zeros. Kaiming)
66.26
18.08
7

---

Published as a conference paper at ICLR 2026
4.2 CHOOSING IMPORTANT HYPERPARAMETERS
Selecting a. To empirically validate this, we sweep over a range of a values for Llama-3.2 3B, in Table 4. Consistent with prior PEFT literature, ABBA achieves optimal performance within the typical LoRA scaling range of 16 32. Additional evidence is provided Table 8 (Appendix E).
Table 4: Performance comparison across different values for Llama-3.2 3B.
Accuracy (1)
BoolQ
PIQA
SIQA
HellaS,
WinoG. ARC-e
ARC-e OBQA
Avg.
70.92
85.74
79.84
92.07
84.45
88.38
75.83
82.20
82.43
71.19
86.83
80.65
92.60
85.95
88.47
76.11
82.20
82.97
12
72.35
86.62
81.63
92.82
85.01
89.10
77.05
83.00
83.45
16
72.88
86.45
80.75
93.18
86.97
89.98
78.33
83.80
81.04
24
73.82
85.91
80.55
93.29
85.87
89.64
78.41
84.60
84.01
32
73.52
86.93
80.96
92.73
85.24
89.61
79.10
85.04
84.08
48
71.83
84.77
78.96
90.52
84.92
87.12
74.57
82.40
81.88
64
67.71
79.43
77.38
81.25
78.69
79.96
66.47
80.00
76.36
Selecting P2- An important choice is allocating the total rank budget r = Ti + r2 between the two low-rank projections. balanced setting, Ti r2 = r/2, is expected to perform best since maximizes the effective rank r1F2, increasing expressivity. We empirically evaluate various {r1, r2) combinations under fixed total rank on Mistral-7B in Table 5. The symmetric configuration achieves the best accuracy, consistent with our hypothesis.
Table 5:
Different {21
pairs with fixed rı + 72 = 32.
GSM8K
MATH
4
28
64.43
17.01
24
63.91
17.20
64.29
18.22
16
66.26
18.08
4.3 PLACEMENT OF ABBA TRANSFORMERS
Figure 3 examines the effect of fine-tuning individual transformer components in ABBA, namely Query, Key, Value, Output, Up, Gate, and Down projections. The results show the following: Query/Key contribute the least, followed by Value/Up, while Gate/Output/Dow are the most impact- ful. This reflects their functional roles: Query/Key support attention scoring, whereas the others play more direct role in transforming and retaining learned representations.
Accuracy (%)
62
Aocurau
54
13
K
Up
Gate Down
All
K
Q
V
0
Up
Gate Down
All
(a) GSM8K
(b) MATH
Figure 3: Impact of selectively fine-tuning individual transformer components Key, Query, Value,
Output, Up. Gate, and Down projections, with ABBA (Mistral-7B).
4.4 ALL ABOUT EFFICIENCY
Training Memory Footprint We report peak memory usage for various methods in Figure 4, measured with a batch size of and context length of 256. ABBA reduces memory consumption by x 3 3.5 times compared to full FT. Compared to other PEFT methods, ABBA offers similar memory efficiency LoRA, and is z 30 35% more efficient than the next-best method, HiRA.
Training Time. We benchmark training time across multiple settings in Table 10 (Appendix G). ABBA has comparable training time to LoRA, with only a ze 2 3% overhead, primarily due to the additional computation introduced by the Hadamard product. We clarify that the initialization
8

---

Published as a conference paper at ICLR 2026
ABBA
LoRA
DoRA
HiRA
Pull
Peak Memory (GB)
57.35
13.16
20.73
30.00
5.17
8.58
17.45 17.30
21111
3.55
Llama-3.2
Llama-3.2 3B
Mistral-7B
Gemma-2
Figure 4: Comparison of training memory requirements across various methods. Results are reported for models used in our work, with sequence length and batch size fixed 256 and 1, respectively.
itself is highly efficient, as we compute the truncated SVD using torch. svd_lowranke This initialization step takes less than one second for the entire model, even for the largest LLMs used in our experiments.
Efficient Inference. At deployment time, ABBA supports efficient inference by pre-computing the update and merging into the base weights as W = Wo (B141) (B2A2). This allows models to switch tasks quickly by subtracting the update to restore Wo. then applying a new ABBA adapter. Since the update is fused ahead of inference, this incurs runtime overhead latency.
4.5 ADDITIONAL RESULTS ON ARITHMETIC TASKS
We report results for fine-tuning Mistral-7B and Gemma-2 9B using larger 40K subset of Meta- MathQA, compared to the 20K subset used in the main experiments.
As shown in Table 6, ABBA continues outperform all other PEFT baselines under matched parameter budgets. However, we observe that the performance gap between full fine-tuning and ABBA narrows when using the larger subset. This suggests that higher data regimes, full fine- tuning may be more suitable choice relative to other PEFT methods, provided the additional memory and compute requirements for full fine-tuning can be met.
Table 6: Comparison of multiple FT methods on Mistral-7B and Gemma-2 9B across arithmetic reasoning benchmarks (when finetuned 40K subset of MetaMathQA). Best results among PEFT methods are in bold.
Mistral-7B
Gemma-2
Method
# Params
GSMSK (1)
MATH (1)
Params
GSMSK (t)
MATH (t)
Full FT
7.24 B
66.28
18.34
9.24 B
79.89
39.44
LoRA
83.88 M
62.89
16.45
108.04 04 M
77.02
37.05
DoRA
85.26
63.24
16.67
109.88 M
77.46
37.40
HiRA
83.88
64.54
17.88
108.04 M
78.90
38.53
ABBA-=16
41.94
66.36
18.51
54.02
79.93
39.27
ABBA-=32
83.88 M
67.04
18.76
108.04 M
80.31
39.92
4.6 EXTENDING ABBA VIA ADAPTER CHAINS
A natural extension of ABBA is decomposing the update into composition of & multiple adapter pairs: AW Bi A, @ B₂Λ₂ ... + At, where each adapter pair has rank r/k, with r denoting the total rank budget. This factorized form increases the expressive capacity and the effective rank of the update. However, this introduces ad- ditional optimization challenges and may reduce training stability. We evaluate a variant of ABBA using a chain four adapter pairs and compare it to the standard two-pair setup. As shown in Table 7, the chained version performs slightly worse, likely
Table 7: Standard ABBA (2 pairs) versus
a chained extension using 4 pairs.
Configuration
GSM8K MATH
ABBA (2 pairs)
66.26
18.08
Chained ABBA (4 pairs)
64.84
17.74

---

Published as a conference paper at ICLR 2026
due to suboptimal scaling training instability. We show gradient norm plots of these variations in Figure in Appendix J. We leave further investigation multi-stage compositions to future work.
CONCLUSION
ABBA introduces a simple yet effective extension the PEFT framework by expressing the weight update as a Hadamard product of two independently learnable low-rank matrices. This formulation retains the parameter efficiency of LoRA-style methods while enabling significantly higher expres- sivity. Our empirical results demonstrate that this expressivity translates to consistent performance gains across range of tasks and models. Through a mathematically exact Khatri-Rao reformulation, ABBA matches LoRA's efficiency while representing high-rank updates entirely through low-rank components, offering a more powerful and efficient alternative to existing PEFT approaches.
REPRODUCIBILITY STATEMENT
We have made considerable efforts ensure that our work is fully reproducible. Our code open- source and available at: https: //g huub. wwt./ooo//////// CERT- abba. Detailed descriptions of our experimental settings can be found Section 3 and Appendix L. All datasets in this work are widely used, publicly available benchmarks (see Appendix M for details).
ACKNOWLEDGEMENTS AND DISCLOSURE OF FUNDING
This research was supported by Mohamed bin Zayed University of Artificial Intelligence (MBZUAD, with partial funding from the ADIA Lab Fellowship.
REFERENCES
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021.
[2] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3992-4003, 2023.
[3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[4] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.092888 2023.
[5] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean -can-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv: 2...:2//////00 2023.
[6] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 2.X1:24112.15115 2024.
[7] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
10

---

Published as a conference paper at ICLR 2026
[8] Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.127711 2023.
[9] Yaru Hao, Haoyu Song, Li Dong. Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma. and Furu Wei. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.06336, 2022.
[10] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
[11] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
[12] Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number parameters. arXiv preprint arXiv:2405. 210222405.774.,,. 17604, 2024.
[13] Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. HiRA: Parameter--fffiient hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[14] Zhengbo Wang, Jian Liang. Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407. 2024.
[15] Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma. Initialization using update approximation silver bullet for extremely efficient low-rank fine-tuning. arXiv preprint arXiv:2411.19557 2024.
[16] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-218 1936.
[17] Leon Mirsky. Symmetric gauge functions and unitarily invariant norms. The quarterly journal of mathematics, 11(1):50-59, 1960.
[18] I. Slyusar. New operations of matrices product for applications of radars. In Proceedings of the International Seminar/Workshun Acoustic Wave Theory (DIPED-97), pages 73-74, Lviv, Ukraine. PDF.
Direct and Inverse Problems of Electromugnetic and
[19] Martino Ciaperoni, Aristides Gionis, and Heikki Mannila. The hadamard decomposition problem. Data Mining and Knowledge Discovery, 38(4):2306-2344 2024.
[20] Hyeon-Wad Woo Nam, Ye-Bin Moon, and Tac-Hyun Oh. Fedpara: Low-rank hadamard product for communication-eeficeen federated learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[21] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03372) 2023.
[22] Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approxima- tion. Advances in Neural Information Processing Systems, 37:54905-54931, 2024.
[23] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. 2024.
[24] Marco Siino. Mistral at semeval- 2024 task Mistral 7b for argument reasoning in civil procedure. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval- 2024), pages 155-162, 2024.
[25] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00/ 18, 2024.
11

---

Published as a conference paper at ICLR 2026
[26] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 121077, 2024.
[27] Zhiqiang Hu, Lei Wang. Yihuai Lan, Wanyu Xu, Ec-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. Lim -dn-aapttrrs: An adapter family for parameter-efficiet fine-tuning of large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages
2023.
[28] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint 180.0088,,, 2018.
[29] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457) 2018.
[30] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106 2021.
[31] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019.
[32] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432-7439, 2020.
[33] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint 1904:099, 2019.
[34] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 1905. 10044, 2019.
[35] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.122 12284, 2023.
[36] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14666 2021.
[37] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint Stt::21033008744, 2021.
[38] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088
10115, 2023.
[39] Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint (tfi:2309.14717... 2023.
[40] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning, arXiv preprint arXiv:2303, 105/2, 2023.
[41] Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector- based random matrix adaptation. arXiv preprint arXiv:2310.1 1454, 2023.
12

---

Published as a conference paper at ICLR 2026
[42] Yuanhe Zhang, Fanghui Liu, and Yudong Chen. Lora-one: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently. arXiv preprint arXiv:2502.01235, 2025.
[43] Chinmay Savadikar, Xi Song, and Tianfu Wu. Wegeft: Weight-generative fine-tuning for multi-faceted efficient adaptation of large models. In Forty-second International Conference on Machine Learning.
[44] Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. Advances in Neural Information Processing Systems, 37:22513
2024.
[45] Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. Fedex-lora: Exact aggregation for federated and efficient fine-tuning of foundation models. arXiv preprint arX1v:2410.04432, 2025.
[46] Raghav Singhal, Kaustubh Ponkshe, Rohit Vartak, Lav R Varshney, and Praneeth Vepakomma. Fed-sb: A silver bullet for extreme communication efficiency and performance in (private) federated lora fine-tuning. arXiv preprint arXiv:2502. 2025.
[47] Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang. Deqing Wang, et al. Mora: High-rank updating for parameter- efficient fine-tuning. arXiv preprint arXiv:2405. 12130, 2024.
[48] Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.106500 2022.
[49] Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High- rank training through low-rank updates. arXiv preprint arXiv:2307.05669, 2023.
[50] Yao Ni, Shan Zhang, and Piotr Koniusz. Pace: Marrying generalization in parameter-efficien fine-tuning with consistency regularization. Advances in Neural Information Processing Systems, 61266, 2024.
[51] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencoodeinterpreeer: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. 14658, 2024.
[52] Mark Chen. Evaluating large language models trained on code. arXiv preprint a7X:32207..03377. 2021.
[53] Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprint 2xXi:24402.03293 2024.
[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[55] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38 45, 2020.
[56] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1711001711.05101, 2017.
13

---

Published as a conference paper at ICLR 2026
Appendix
CONTENTS
A Related Work
14
B Why Does No Closed-Form Reconstruction Solution Exist for ABBA?
15
C Why Does LoRA-Style Initialization of Both Adapter Pairs Fail?
15
Proof of Theorem 2: Rank-Stability of ABBA
17
Selecting
18
F Effect of Varying Rank
18
G Training Time
19
H Performance on Coding Tasks
19
I Rank-Stability of ABBA: Gradient Norms
19
Stability in Chained ABBA: Gradient Norms
19
K Comparison with High-Rank Baselines
21
L Experimental Details
21
M Dataset Details
21
Use Large Language Models
22
A RELATED WORK
Parameter-Efficient Fine-Tuning (PEFT) and Low-Rank Adaptation (LoRA). PEFT meth- ods adapt large pretrained models to downstream tasks by training a small number of additional parameters while keeping the base model frozen. Among these, LoRA (10) is widely adopted for its simplicity and effectiveness, modeling the update AW as a low-rank product BA, thereby reducing trainable parameters significantly. Several extensions enhance LoRA along different axes. QLoRA (38) and QA-LORA (39) combines quantization with LoRA to reduce memory footprint; AdaLoRA (40) allocates rank budgets dynamically across layers. LoRA- A-X (12) inserts small trainable core between frozen adapters to improve compression, while LoRA-Pro (14) and LoRA- SB (15) optimize adapters to better approximate full fine-tuning gradients. Other variants modify structure or initialization. VeRA (41) reuses frozen adapters across layers with task-specific scaling vectors; DoRA (11) applies low-rank updates only to the direction of pretrained weights. PiSSA (26) initializes adapters using top singular vectors, while rsLoRA (21) proposes scale-aware initialization to improve stability. LoRA-One (42) initializes adapters using this singular subspaces of the full fine-tuning gradient and incorporates preconditioning to improve convergence. WeGeFT (43) leams a small low-rank transformation that maps pretrained weights to their fine-tuned updates, either shared across layers learned per layer. Despite all these differences, these methods share
14

---

Published as a conference paper at ICLR 2026
principle: they represent A AW using low-rank structure, enabling adaptation under tight compute and memory constraints. LoRA-based methods have also been applied in other domains, such as federated fine-tuning (44; 20; 45; 46).
Beyond Low-Rank: High-Ramd Rank and Structured Adaptation. While low-rank PEFT methods offer strong efficiency-performmae tradeoffs, they can underperform in tasks requiring high-rank updates. This limitation has driven recent efforts to move beyond purely low-rank parameterizations. HiRA (13) achieves high-rank updates by taking the Hadamard (elementwise) product of the pre- trained matrix Wo with a low-rank adapter BA, leveraging the fact that such product can increase effective rank without increasing parameter count. MoRA (47) instead learns full-rank update through input compression and activation decompression, while KronA (48) uses Kronecker products between adapters boost representational capacity. ReLoRA (49) uses multiple low-rank updates to approximate final higher-rank update. Other approaches explore elementwise structure for different purposes: FLoRA (? modulates intermediate activations per task using Hadamard products, and PACE (50) injects multiplicative noise during adaptation. These trends reflect a broader shift toward structured, high-rank updates for improved expressivity.
Our proposed method, ABBA, follows this direction by learning two independent low-rank matrices whose Hadamard product forms the update, enabling high-rank adaptation with full learnability and minimal overhead.
B WHY DOES No CLOSED-FORM RECONSTRUCTION SOLUTION EXIST FOR ABBA?
SVD admits closed-form solution for low-rank approximation because both the Frobenius and spectral norms are unitarily invariant. This allows the objective:
min
||M |MX-|
rank(X)
to decouple along singular directions, as guaranteed by the Eckart-Yong--iist theorem (16; 17). In contrast, ABBA solves the problem:
|M - (BIAI) o (B2A2)lle, subject rank(BeAr) Sr, Br A1
min
which is a non-convex, quartic optimization problem in the latent factors. The Hadamard product breaks orthogonal invariance; unlike the SVD, the two low-rank matrices A and B2A2 cannot be simultancously diagonalized, and singular directions no longer decouple.
Using Theorem AI 3r Ag. However, the rows of these matrices lie on a Segre variety Sn,.ra² 2 meaning they reside highly constrained non-linear manifold. In general, such constraints prevent the existence of an exact closed-form solution unless every row lies in a rank-one [8') Xr2 subspace.
could, in principle, apply SVD-like decomposition Bi Cr B2 and
As result, no analogue of truncated SVD exists for this formulation, and optimization must proceed via iterative methods such as gradient descent.
C WHY DOES LORA-STYLE INITIALIZATION OF BOTH ADAPTER PAIRS FAIL?
naive LoRA-style initialization, where Bj and B2 are initialized to zero while A₁ and Ag follow Kaiming uniform initialization, results in training failure due gradients becoming identically zero. To analyze this, we compute the gradients of the loss with respect to each adapter component and show that they are exactly zero under this initialization, ultimately preventing any learning.
The Segre variety Sm.rz C RR112 is the set of all rank-one tensors expressible as outer products with RR", E IR".
15

---

Published as a conference paper at ICLR 2026
Notice that, for some input, 2, the output, y, is of the form
y Wa AWæ
= Wa + SABBA ((B) @ (B242))
f f(v)
To get the final closed form gradients, compute the gradients element-wise. We can therefore have, for some matrix Z E {A1, A2, B1, B2h.
0C
Oc dAWS
Σ
07.pg
dAWij ij 87.pg
SABBA ((BrAi) (B2A2))ss
Oc
dAW ij
8Zpy
Ô ((B2AA) @ (B2A2))
= SABBA
Σ
Gij
87.p
W
For A1.
We need to compute
OC
(((BAA) @ (B2A2))
ΣG
0A
SABBA
0Ai.p
i,
((B1A) )5 (2242)5))
Gij EGU
SABBA
0A1
a (((B))))
Σ 4
(22Ali
= SABBA
AA
θ (Bi (Bi,a A1.s)
Σ
(22AAi)))
SABBA
AA
Notice that 0(B, (1.1)
0 when * p and
g else
B1.sp- This means rewrite the
0A above summation as
OC
Σ (Bn B1.i0
0 AA
SABBA
I (B242))in
SABBA
Notice that the above equation is nothing but a inner product of the th row of BJT and 9t column of (G @ (B2A2)). We can therefore write the following
HL 0A
SABBA B, (G (B2A2))
(10)
For 2 = B₁. Following the same analysis as we did for Ar, we have the following for B,.
OC
ô (Bi
Σ (i,
Σ
aB PM
SABBA
0Bim
i.j
Again notice that 8(B aAmu)
when # and / + else ALaj This means we can rewrite
8P
the above summation as
Oc
Gn
8B,,p
SABBA
I (B242))p5
SABBA
16

---

Published as a conference paper at ICLR 2026
Notice that the above equation is nothing but inner product of the pm row of (G (B2Az)) and q"h
column of A We can therefore write the following
Oc aB
= SABBA (G o (B2A2))
(1))
is also easy to see that our formulation is symmetric in A,'s and B, 's implying we also have
For Z = B2. Following Eqn. (11)
ac 0B2
= SABBA (G @ (B)A:))
(12)
For Ag.
Following Eqn. (10),
ac
SABBA B3 (G (BAA))
(13)
AA
clear that initializing both B₁ and B2 zero causes all gradients become zero, thereby preventing any learning and leading to complete training failure.
In this section, we provide the proofs for the assertions from the main text.
D PROOF OF THEOREM 2: RANK-STABILITY OF ABBA
Theorem (Rank-Stability of ABBA). Let the ABBA update be AW SABBA (B1A1) O (B2A2), with B1, A1, B2, A2 independent, mean-zero random matrices with finite variance. Then, the update satisfies the stability conditions of Definition if and only if:
SABRA E
Forward 2nd moment (ABBA adapter).
For any input a E Rdo the ABBA adapter produces
An
Vi SABBA
M2,35 Øz E {l, doat
(14)
1=1
where M1 = B,A, A E I Ridexxd dd and M2 = B2A2 E Rdu Thus
dn
Ely
E
S ABBA
E
Σ
M1,
Gradient computation.
The gradient with respect to the input is
OL dy:
(v))),
M My,1y (V,L), SABBA
Dy: dx,
where (VyL);
OL 8y,
Hence
des
27
E (V.L)
E
Mi.j M2, M2. (vyL),
1=1
Second-moment factorization.
Since M1 and M2 are independent with mean-zero entries,
|[M = J[ BIM r2 E. B |[44,,..
Therefore
BMM = BEEME, B rirz E[B [
[
B [ m.
17

---

Published as a conference paper at ICLR 2026
Forward and backward moments.
Substituting,
E Elu²]
ABBA dm -172" C. B(a)].
2
E[ V[V 2)31
SABBA
dout -172 C -B
where C' = 1E 11
Rank-stability.
By Definition 1, forward stability requires Ely ,r,,, (1) whenever B[x3]]
(1); and backward stability requires E[(V. rr (1) whenever EE[(V, L) O Both hold exactly when
ABBA 1192 = (1))
that is,
SABBA +
VPDz
Following Definition these results imply that ABBA adapters are rank-stabilized.
E SELECTING Q:
We perform a sweep over different a values, reporting results for Mistral-7B in Table In line with prior results in Table 4, ABBA performs best when OLORA lies in the typical range of 16-32.
Table 8: Performance comparison across different values for Mistral-7B.
Accuracy (t)
GSM8K
MATH
62.17
17.10
8
64.06
17.60
12
64.43
18.14
66.10
18.16
24
66.26
18.08
32
65.81
17.68
48
65.20
16.82
64
64.79
15.08
EFFECT OF VARYING RANK
We evaluate ABBA on Mistral-7B while varying the total rank budget т, with ri r2 = r/2 and results shown in Table 9, Performance improves substantially from 16 to = 32, with the best results achieved at r = 32, after which gains begin to saturate. This aligns with our hypothesis: at 32, ABBA's effective rank reaches r1 12 16 16 256, which is significantly higher than the effective rank 64 at 16. The increased expressivity enables ABBA to better capture task-specific patterns. However, increasing the rank beyond 32, effective rank beyond 256, yields diminishing returns and may lead to overfitting, mirroring trends observed in using very high-ranked LoRA as well.
Table Performance comparison of ABBA on Mistral-7B across varying total rank values r.
Accuracy (t)
Rank
GSM8K
MATH
16
64.97
18.06
32
66.26
18.08
64
65.05
17.98
65.73
17.96
18

---

Published as a conference paper at ICLR 2026
G TRAINING TIME
Following the discussion in Section 4.4, we report training times across all models and tasks in Table 10. ABBA incurs only a negligible overhead of approximately 3% compared LoRA, primarily due to the extra computation resulting from the Hadamard product.
Table 10: Training time comparison between ABBA and LoRA across multiple model and task settings.
Training Time
Model
LoRA ABBA
Llama-3.2 IB (Commonsense))
2:42:17
2:46:18
Ilama-3.2 3B (Commonsense)
6:05:43
6:11:26
Mistral-7B (Arithmetic)
1:15:55
1:18:22
Gemma-2 (Arithmetic)
1:45:33
1:49:45
H PERFORMANCE ON CODING TASKS
We also evaluate ABBA against other PEFT variants on code generation tasks. We fine-tuned Llama- 3.2-1 1B 100K subset of the CodeFeedback dataset (51) and assessed performance using Pass on HumanEval (52). As shown in Table 11, ABBA again performs very strongly, outperforming all other PEFT variants and even full fine-tuning (all PEFT methods are run at rank 32). These results demonstrate that the advantages of our method generalize to more complex tasks such as code generation.
Table 11: Comparison of multiple FT methods across the coding benchmark HumanEval. Best results among PEFT methods are in bold.
Method
Params
HumanEval (f)
Full FT
1.24 B
23.17
LoRA
22.54 M
20.73
HiRA
22.54 M
21.95
ABBA
22.54 M
25.61
I RANK-STABILITY OF ABBA: GRADIENT NORMS
In Theorem 2, we had established the optimal scaling for ABBA,
SABBA E +
We now verify this result empirically by comparing several scaling strategies: e ((1)))))))
SABBA E (()), + ((r))) e ((rr))))
As shown in Figure 5, the scaling SABBA E + consistent with our theoretical findings.
produces the most stable gradient norms,
VTITz
J STABILITY IN CHAINED ABBA: GRADIENT NORMS
We present gradient norms for the chained ABBA setup (with four adapter pairs) compared to the conventional ABBA configuration (with pairs) in Figure 6. As hypothesized in Section 4, the chained variant exhibits significantly greater gradient-nomn instability than the standard ABBA model.
19

---

Published as a conference paper at ICLR 2026
ooggi OPILME AARK OcUgI IPPILO
001 -10.01 (25000
Step
100
200
300
400
500
600
Figure 5: Gradient norms Mistral-7B comparing various scaling strategies.
Gredient
Figure 6: Gradient norms in Mistral-7B comparing chained ABBA (4 pairs) and normal ABBA
20

---

Published as a conference paper at ICLR 2026
K COMPARISON WITH HIGH-RANK BASELINES
We evaluate ABBA against additional higher-rank PEFT variants, namely Flora (53), MoRA (? ). and KronA (48), on code generation tasks. We fine-tuned L 8B 100K subset of the CodeFeedback dataset (51) and assessed performance using Pass @ I on HumanBval (52). As shown Table 12, ABBA again performs very strongly, outperforming all other higher-rank PEFT variants.
Table 12: Comparison of multiple PEFT methods across the coding benchmark HumanEval. Best results among PEFT methods are in bold.
Method
HumanEval (f)
Full
48.54
LoRA
45.73
KronA
45.12
Flora
45.56
MoRA
47.28
HiRA
48.17
ABBA
49.39
L EXPERIMENTAL DETAILS
We implement all models using Py Torch (54) and HuggingFace Transformers (55). All experiments single NVIDIA A6000 GPU (48 GB). To reduce memory usage, initialize base models in torch. fol.lall ollttt 1 6 precision. We train each configuration with the Adam optimizer (56) and three random seeds.
report the mean performance
We configure Llama-3.2 IB, Llama-3.2 3B, Mistral-7B, and Gemma-2 9B using the hyperparameters shown in Table 13. We conduct sweep learning rates and scaling factors identify optimal settings for each model-task pair. ABBA generally performs better with slightly higher learning rates compared to LoRA, and recommend initiating hyperparameter sweeps in that range.
While we adopt most settings from prior work (27), we perform a targeted learning rate sweep to optimize performance. For baseline comparisons, we replicate the experimental setups from the original PiSSA (26), rsLoRA (21), DoRA (1I), LoRA-Pro (14), and HiRA (13) papers to ensure fair and consistent evaluation.
Table 13: Hyperparameter settings for training Llama-3.2 1B and 3B on COMMONSENSE 70K, and Mistral-7B and Gemma-2 9B MetaMathQA.
Llama-3.2 1B 3B
Mistral-7B Gemma-2 9B
Optimizer
AdamW
AdamW
Batch size
6
1
Max. Seq, Len
256
512
Grad Acc. Steps
24
32
Epochs
2
1
Dropout
0.05
0
Learning Rate
x 10
1 x 10 10-3
LR Scheduler
Linear
Cosine
Warmup Ratio
0.02
0.02
M DATASET DETAILS
COMMONSENSE 170K is a unified benchmark that aggregates eight commonsense reasoning datasets into single multi-task setting (27). Each instance multiple-choiee question, and models are prompted to select the correct answer without providing explanations. We adopt the prompt format introduced by the paper (27). Below, briefly describe the constituent datasets:
21

---

Published as a conference paper at ICLR 2026
OBQA (28): Open-book QA requiring retrieval and multi-hop reasoning external knowledge.
ARC Challenge (ARC-e) (29): Difficult grade-school science questions designed to test advanced reasoning beyond surface heuristics.
ARC Easy (ARC-e) (29): Simpler science questions assessing core factual and conceptual understanding.
WinoGrande (30): Pronoun resolution tasks requiring commonsense inference to resolve ambiguity.
HellaSwag (31); Next-sentence prediction under a constrained completion setting, testing grounded understanding of everyday scenarios.
PIQA (32): Physical reasoning tasks where models select the most sensible solution to practical problem.
SIQA (33): Social reasoning benchmark involving questions about intent, social dynamics, and consequences of human actions.
BoolQ (34): Binary (yes/no) questions drawn from natural queries, requiring contextual understanding of short passages.
MetaMathQA (35) reformulates existing mathematical problems into alternative phrasings that preserve their original semantics, offering diverse surface forms without introducing new information. We evaluate models fine-tuned on this dataset using two benchmarks: GSM8K (36), which targets step-by-step reasoning in elementary arithmetic word problems, and MATH (37), which features high-difficulty problems drawn from math competitions. We evaluate solely based on the correctness of the final numeric answer.
N USE OF LARGE LANGUAGE MODELS
LLMs are used exclusively for minor writing assistance, for example, enhance grammar and improve sentence clarity.
22


Reviews:
[{"review_id": "2DL4QhebGL", "reviewer": "Reviewer_XkLT", "summary": "This paper proposes ABBA, a new parameter-efficient fine-tuning (PEFT) method that models the weight update as the Hadamard product of two independently learnable low-rank matrices. Unlike HiRA, which ties the update to the pretrained weights $W_0$, ABBA fully decouples both components, allowing them to be optimized freely. The authors further derive an efficient Khatri–Rao factorization so that ABBA can be implemented with the same memory and compute efficiency as LoRA while offering much higher expressivity. Extensive experiments on Llama-3.2 (1B & 3B), Mistral-7B, and Gemma-2 9B demonstrate consistent improvements over LoRA, HiRA, DoRA, and other PEFT baselines on commonsense and arithmetic reasoning benchmarks.", "questions": "Please refer Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of composing two learnable low-rank modules via a Hadamard product is conceptually clean and represents a clear generalization of LoRA and HiRA. The method increases effective rank while maintaining strict parameter efficiency.\n2. The paper introduces Khatri–Rao reformulation and a well-motivated rank-stability theorem, which explains how scaling should depend on $r_1,r_2$.\n3. Results across four foundation models and multiple reasoning tasks show large and consistent gains. The authors carefully control for parameter count and initialization, making comparisons fair.", "weaknesses": "1. The paper proves a scaling law for stability but does not empirically show how optimization behaves under varying $r_1, r_2$ or initialization errors. Gradient norm or loss-landscape visualizations would strengthen claims about stable training.\n2. It remains unclear whether the performance gain comes from the Hadamard structure itself or simply from doubling the number of learnable matrices. An ablation removing the Hadamard product (e.g., summation or concatenation) would clarify this.\n3. While Section 2.4 claims that ABBA is more expressive than LoRA, the analysis relies primarily on empirical reconstruction errors. It will be better that if authors can provide formal proof or analysis showing that ABBA’s representational space strictly contains LoRA’s or HiRA’s.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hG5KNoyCkk", "reviewer": "Reviewer_Cd6v", "summary": "The paper proposes ABBA, a PEFT adapter that models updates as a Hadamard product of two independently learnable low-rank matrices and provides a Khatri–Rao reformulation so the update can be applied like LoRA without materializing full matrices. Claimed benefits are higher expressivity at a fixed parameter budget and LoRA-like efficiency. The proposed method is tested on commonsense (8 datasets) and arithmetic (GSM8K/MATH) in comparison with rsLoRA, PiSSA, DoRA, LoRA-Pro and HiRA with promising performance obtained.", "questions": "please consider to address the questions in the weaknesses.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed method extends HiRA by replacing its fixed modulation using a learnable factor decouples the updated from $W_0$, which is a technical improvement. The Khatri-Rao factorization is a nice implementation detail. \n\n+ The proposed method is shown to have higher expressivity against LORA via matrix-reconstruction experiments and strong accuracy on commonsense/arithmetic.", "weaknesses": "- The main concern is the novelty of this paper is incremental relative to HiRA/MoRA/ReLORA/KronA. The core architectural change is to learn both factors in the Hadamard product instead of tying one to $W_0$ (HiRA). While useful, this feels like a straightforward extension in the space of multiplicative/structured adapters already explored (HiRA, MoRA, KronA, ReLoRA), and the paper’s Related Work acknowledges much of this trajectory. The new factorization is an implementation convenience rather than a substantially new theoretical primitive.\n\n- The proposed method is evaluated only on two tasks: commonsense (8 datasets) and arithmetic (GSM8K/MATH). No instruction-tuning, long-context, code, multilingual, or domain-shift tests—settings where higher-rank adapters might matter most, especially considering the proposed method is claimed to be better than the widespread of LORA. \n\n- Compared with the built-in multi-faceted efficiency of LORA,  the efficiency evdience of the proposed method is not sufficiently compared. Figure 4 reports peak training memory at batch size = 1 and seq len = 256; settings are narrow and may not reflect real training regimes (e.g., larger batches/sequences, activation checkpointing, gradient accumulation). Tables show parameter counts matched to LoRA, sometimes halved (r=16), but parameter count alone is not the whole efficiency story; missing are optimizer-state bytes, activation footprints at realistic batch/seq, and per-step throughput. For example, all experiments run on a single A6000 (48 GB); conclusions about scalability and efficiency may not carry to multi-GPU or larger models. \n\n- LoRA variants are included (rsLoRA, PiSSA, DoRA, LoRA-Pro, HiRA), but QLoRA—a common efficiency baseline combining quantization and adapters—is discussed only in related work and not compared in experiments, and some recent work such as WeGeFT (C. Savadikar et al ICML25) are not discussed and compared. \n\n- Based on the the Khatri–Rao theorem, the proposed method can be treated as a special case of LORA with $B=B_1\\odot_r B_2$ and $A=(A_1^\\top\\odot_r A_2^\\top)^\\top$, i.e., introducing structures for B and A in LORA, and the proposed initialization of B_1 and A_1. In terms of this, LORA-One (ICML'25) and LORA-GA should be compared. \n\n- Although there are ablations for $\\alpha, \\{r_1,r_2\\}$, and placement, the “adapter chains” variant underperforms, hinting at optimization fragility when stacking. More analysis of failure modes would help (e.g., gradient scales, curvature, stability vs. rank).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8sR01gLFhK", "reviewer": "Reviewer_FALt", "summary": "- The paper proposes an alternative PEFT method to LoRA called ABBA\n- ABBA paramtrizes the residual weights $\\Delta W$ used during the fine-tuning as $\\Delta W = s(B_1 A_1) \\odot (B_2 A_2)$ where $A_1, A_2, B_1, B_2$ are low rank matrices of ranks $r_1$ and $r_2$ respectively, where $s=\\frac{\\alpha^2}{r_1r_2}$ for rank stabilization.\n- $A_1$ and $B_1$ are initialized using the truncated SVD of the pretrained weights $W_o$ by keeping the top $r_1$ singular values and vectors.\n- $A_2$ and $B_2$ are initialized using the the standatd LoRA initialization scheme.\n- ABBA uses Khatri-Rao factorization to rewrite the paramtrization as $(B_1A_1) \\odot (B_2A_2) = (B_1 \\odot_r B_2)(A_1 \\odot_r A_2)$ where $\\odot_r$ is the Khatri-Rao product, makig the computation efficient. Here, $B_1 \\odot_r B_2 = B_{kr} \\in \\mathbb{R}^{m \\times r_1 r_2}$ and $A_1 \\odot_r A_2 = A_{kr} \\in \\mathbb{R}^{r_1 r_2 \\times n}$.", "questions": "1. What is the hyperparameter tuning strategy used for full fine-tuning?\n2. Can the authors perfrom the experiments on arithmetic reasoning with a larger dataset, e.g., 40k subset of MetaMathQA?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The method a clear theoretical motivation.\n2. The Khatri-Rao formulation makes the method computationally efficient, and hence practically feasible.\n3. The method is easy to implement and can be easily integrated with existing PEFT methods.\n4. The method is evaluated on commonsense reasoning, arithmetic reasoning and outperforms prior PEFT methods across model sizes.\n5. The method is ablated well to study its properties (initialization strategies, scaling factors, layer placement, and chaining more low-rank matrices instead of only two).", "weaknesses": "1. The paper compares ABBA with total ranks of 16 and 32 with rank 32 LoRA (and variants). However, LoRA can have onptimization problems with larger ranks, and many times, using smaller ranks can lead to slightly better performance. Hence, comparison with LoRA should also be done by setting the LoRA rank to 16 for a more thorough comparison.\n2. I find it hard to believe that full fine-tuning lags behind ABBA by such a large margin. While the authors try to justify this on lines 330-332, better hyperparameter tuning (especially setting a low learning rate) can possibly help full fine-tuning perform better. This does not necessarily mean that ABBA does not have merits over full fine-tuning, but rather that the expectations should be tempered.\n3. Continuing the point above, the experiments on arithmetic reasoning have been performed with a 20k subset of MetaMathQA. It's possible that with increasing the dataset size, full fine-tuning and LoRA can perform better. Again, this experiment would provide a more comprehensive comparison and help understand the merits of ABBA in limited data settings (wich are often observed in real-world applications).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0mgbwADhDP", "reviewer": "Reviewer_Z4En", "summary": "This paper proposed ABBA, a variant of LoRA that relies on the Hadamard product of two learnable low-rank matrices. The key difference compared to HiRA is that both terms in the Hadamard product are learnable, and are decomposed through LoRA. In doing so, the Hadamard product can be calculated through the well-known Khatri–Rao factorization (or equivalently, face-splitting product). Experiments are conducted on commonsense reasoning datasets with LLaMA3.2 1B and 3B, and Arithmetic reasoning with Mistral-7B and Gemma-2 9B, demonstrating the empirical advantages of the advocated approach.", "questions": "See weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing is easy to follow and the presentation is clear\n2. The Khatri–Rao factorization provides a computationally efficient approach to bypass the Hadamard product. \n3. The rank stability is confirmed theoretically. \n4. Numerical results are promising, which demonstrate a consistent performance boot.", "weaknesses": "1. My major concern is the computational cost of ABBA. With the face-splitting (i.e., row-wise Khatri-Rao) product in Theorem 1, the forward and backpropagation cost of $\\Delta W x$ is $\\mathcal{O}((m+n)r_1r_2) = \\mathcal{O}((m+n)r^2)$, which is $\\mathcal{O}(r)$ times higher than vanilla LoRA. In other words, ABBA is be less scalable with the rank increasing. It would be beneficial if experiments can be conducted to showcase how the time and space complexities change with r (e.g., from $r=4$ to $r=512$) under different model sizes. \n2. There is no need to provide a proof for Theorem 1 in Appendix D.1, as it is a well-known result in linear algebra, and the citation (18) could be sufficient. \n3. The \"ABBA vs. LoRA: Reconstruction\" paragraph in Section 2.4 is not convincing enough, given the lack of tight theoretical bounds. Could you please provide some insights regarding how the bound can be improved? \n4. The models used in Section 4 are relatively small (<10B). It would be beneficial if experiments can be scaled to larger models such as Gemma-3-27B, and more complicated tasks such as coding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)