QWHA: Quantization-Aware Walsh-Hadamard Adaptation for Parameter-Efficient Fine-Tuning on Large Language Models
Authors: Hyesung Jeon
Seojune Lee
Beomseok Kang
Yulhwa Kim
Jae-Joon Kim
Venue: ICLR 2026 Poster (2026)
Keywords: quantization
parameter efficient fine-tuning
sparse adapter
walsh-hadamard transform
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose the first quantization-aware sparse adapter using the Walsh-Hadamard Transform with an initialization strategy that reduces quantization error and facilitates fine-tuning, outperforming low-rank and conventional FT-based adapters.

Abstract: The demand for efficient deployment of large language models (LLMs) has driven interest in quantization, which reduces inference cost, and parameter-efficient fine-tuning (PEFT), which lowers training overhead. This motivated the development of quantization-aware PEFT to produce accurate yet efficient quantized models.
In this setting, reducing quantization error prior to fine-tuning is crucial for achieving high model accuracy. However, existing methods that rely on low-rank adaptation suffer from limited representational capacity. Recent Fourier-related transform (FT)-based adapters offer greater representational power than low-rank adapters, but their direct integration into quantized models often results in ineffective error reduction and increased computational overhead.  
To overcome these limitations, we propose QWHA, a method that integrates FT-based adapters into quantized models by employing the Walsh-Hadamard Transform (WHT) as the transform kernel, together with a novel adapter initialization scheme incorporating adaptive parameter selection and value refinement. We demonstrate that QWHA effectively mitigates quantization errors while facilitating fine-tuning, and that its design substantially reduces computational cost. Experimental results show that QWHA consistently outperforms baselines in low-bit quantization accuracy and achieves significant training speedups over existing FT-based adapters.

Published as a conference paper at ICLR 2026
QWHA: QUANTIZATION-AWALL WALSH-HADAMARD ADAPTATION FOR PARAMETER-EFFICIET FINE-
TUNING ON LARGE LANGUAGE MODELS
Hyesung Jeon"", Seojune Lee"", Beomseok Kang', Yulhwa Kim?, Jae-Joon Kim
Scouul National University. "Sungkyunkwan University {hjeon2k, jees beomseck, kimjaejoon snu. ac kr, fyulhwakim]eskku.uk
ABSTRACT
The demand for efficient deployment of large language models (LLMs) has driven interest in quantization, which reduces inference cost, and parameter-effiiient fine-tuning (PEFT), which lowers training overhead. This motivated the develop- ment of guantization-wwar PEFT to produce accurate yet efficient quantized mod- els. In this setting, reducing quantization error prior to fine-tuning is crucial for achieving high model accuracy. However, existing methods that rely on low-rank adaptation suffer from limited representational capacity. Recent Fourier-related transform (FT))basce adapters offer greater representational power than low-rank adapters, but their direct integration into quantized models often results inef- fective error reduction and increased computational overhead. To overcome these limitations, we propose QWHA, a method that integrates FT-based adapters into quantized models by employing the Walsh-Hadamard Transform (WHT) the transform kernel, together with a novel adapter initialization scheme incorporating adaptive parameter selection and value refinement. We demonstrate that QWHA effectively mitigates quantization errors while facilitating fine-tuning, and that its design substantially reduces computational cost. Experimental results show that QWHA consistently outperforms baselines in low-bit quantization accuracy and achieves significant training speedups over existing FT-based adapters. The code is publicly available at https::////. //gitttttt //ggtttuuu/ com aaam//antaaaa 9 /wwha
INTRODUCTION
Fine-tuning enables large language models (LLMs) to generalize beyond their pre-training, allowing adaptation to various domains (Wei al. 2022 Liu et al. 2023 Qin et al. 2024 DeepSeek-AI al.,
2025). While full tine-tuning yields superior accuracy, often incurs significant overhead due the
extensive computations required to update all the trainable model parameters (Loshchilov & Hutter
2017 Zhu et al. 2025). Parameter-efffcient fine-tuning (PEFT) addresses this issue by optimizing only a small subset of the parameters while leaving most of them frozen (Li & Liang. 2021 Liu
al., [222]] Hu et al. 2022 Liu al. 2024: Kopiczko et al. (024).. Beyond reducing training overhead, recent studies have shown that combining PEFT with model compression techniques can enhance inference efficiency at the same time (Dettmers al.) 2023). Among these techniques, quantization, which lowers the bit precision of model parameters, has gained particular attention due its robustness against accuracy degradation under high compression ratios (Frantar et al.) Lin et al. 2024 Dettmers et al. 2024 Kim et al.) 2024b; [Shao et al.) 2024 Ashkboos et al., 2024 Zhang et al. 2024 Liu al. 2025). Consequently, quantization-awe aware PEFT (QA-PEFT) has promising approach for efficient adaptation and inference in LLMs.
2023
been widely explored
Prior works on QA-PETT typically relied on low-rank adaptation (LoRA) LLi et al. 2024 Guo et al., Kim et al. 2024a Liao et al. 2024| Deng al. 2025). In contrast, for standard PEFT, several alternatives to LoRA have recently been proposed to address the representational limitations low- rank structures. In particular, Fourier-related transform (FT))based adapters have emerged strong alternatives. They train a sparse set of coefficients to represent weight updates in the transform do- main, offering superior representational capacity (Gao et al. 2024h Du et al. 2025 Shen et al.
2024

---

Published as a conference paper at ICLR 2026
QWHA (Ours)
Initialization
Transform Kernel
Rd Kdea
(Reflnement)
Inverse WHT (WHA)
Frozen
Q.
O
02:
0-3
0.5
Q,2
0
Value
R
Trainable
Q
E
AW FH
(AdaAlloc)
Initialization
0.5
0.4
Target
s... FanKo
Scattentc
Index
NPx2
H gdexxdax
X E
Fast Transform via Recursive Sub Add
Figure 1: Overview of Quantization-awan aware Walsh-Hadamard Adaptation (QWHA). The weight up- date from QWHA formulated as AW FH-1, where H predefined Walsh-Hadamae transform (WHT) matrix and F is trainable sparse coefficient matrix consisting of values e and their indices E. The multiplication FH indicates the expansion of learned coefficients (i.e., c), over the transform basis (i.e., columns of H +). Note that, the coefficients are the only trainable parameters, and H remains constant. Our key contributions are in the adoption of WHT into the adapter (WHA) and their initialization, particularly E (AdaAlloc) and (Refinement).
Table I: Comparison of adapter types and parameter selection strategies. Adapter types include low-rank adapters (LoRA), recent FT-based adapters (DCA and DHA), and our proposed adapter (WHA). Strategies to determine parameter location E in F include magnitude-based selection, random uniform selection, training via reparameterization, and our proposed method (A (AdaAlloc).
Adapter Type
Parameter Selection Strategy
Ability Factors
LoRA
DCA
DHA
WHA
Magnitude
Random
Trainable
AdaAlloc
Fine-tuning
f
Quantization Error Reduction
+
f
However, our observations show that directly applying FT-based adapters quantized mod- els often yields worse performance than LoRA-basee methods specifically designed for QA-PEFT. This highlights the importance of explicit consideration for quantization effects when fine-tuning quantized models. LoRA-basee methods adopt quantization-awan initialization strategies that com- pensate for the errors between full- and low-precision weights using low-rank approximation with the adapters prior to fine-tuning. However, applying such initialization in FT-based adapters is non- trivial, as identifying the optimal sparse set of parameters and their values approximate given matrix is NP-hard problem (Natarajan) 1995). Moreover, the choice of transform type becomes an additional design consideration. This raises a research question: how to effectively exploit FT- based adapters in QA-PEFT. To the best of our knowledge, neither FT-based adapters nor their initialization techniques have been explored in the context QA-PEFT.
2025
In this paper, present QWHA, novel QA-PEFT method that introduces FT-based adapter to- gether with a quantization-awant initialization scheme, as illustrated in Figure and Table [ We adopt WHT in adapter design (WHA), inspired by its high-fidelity reconstruction ability the spectral domain, to effectively compensate for quantization errors (Hedayat 1978). In addition. the WHT kernel consists solely of 1 elements, enabling efficient computations using only additions and subtractions, thereby eliminating matrix multiplications (Dao-All ab 2024). We further reduce computation by applying single transform in the adapter, unlike conventional FT-based adapters adapter initialization, we develop tractable
that apply two transforms. For quantization-awer solution that first selects parameter locations E and then assigns their values c. We introduce channel-wise parameter allocation scheme that guarantees lower bound on the number of param- eters per channel to facilitate fine-tuning while allocating more parameters to channels with larger quantization errors, and then select the highest-magnitude coefficients within each channel to ef- fectively reduce quantization error (AdaAlloc). Finally, we refine the selected parameter values, thereby enabling substantial reduction of quantization error (Refinement). We theoretically analyze the superior representation capacity of our proposed adapter and empirically validate the benefits of adapter design and initialization method across diverse datasets and models.
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
2.1 LLM QUANTIZATION
LLM quantization is key technique for improving inference efficiency by reducing the memory bottleneck caused by model weights through lowering their bit precision (Frantar et al. 2023). typi- cally expressed by the following equation:
Wo
We= clamp
round
2,0,
We = (WQ + 2) x
x
(1)
Here, Wo denotes the pre-trained weight matrix, while Wq and Wi represent the quantized integer weights and the corresponding dequantized weights, respectively. and & are quantization scales and integer zero-points. Clamping is applied to the rounded and shifted value within the range 0 to
2n - 1, where is the bit-width.
LLMs generally contain outliers, small fraction of weights that are exceptionally large compared to the main distribution, and L.LM quantization highly sensitive to these outliers (Dettmers al.) 2024 Kim al. 2024b; Tseng al., 2024, An al. 2025). These outliers induce corresponding outliers in the quantization error. Most quantization errors A AWo Wo- - Wq are bounded within a small range (e.g.. [- 2 2 )), since most weights within the clamping range
mapped the nearest quantization level. In contrast, for outliers, the quantization error is defined as the difference between the original large weight and the clamping boundary values, resulting in extremely large errors that lead to significant accuracy degradation. Thus, reducing outlier-induced error is critical, and recent post-training quantization techniques for LLMs focus mitigating these errors preserve model
accuracy (Detimers et al. (002]] Kim et al., al., 2024b Shao et al. [024 Tseng et al. 2024 Zhang et al., 2024). Details on the distribution of quantization errors are presented in Appendix
2.2 QUANTIZATION- PEFT
A typical quantization-awar PEFT (QA-PEFT adopts LoRA (Hu et al. (022). which injects a pair of low-rank matrices into linear layers to approximate the weight updates AW as follows:
Y (Wq + AW) s.t. AW BA
Here, A E Re and B E Rr are low-rank adapters, fine-tuned instead of frozen quantized weight Wa E du where X E Rdxx((xx) is the activation matrix with batch size and se- quence length s. Since there is no prior information about the weight updates before fine-tuning, LoRA typically initializes A as random matrix and as a zero matrix. QA-PEIT, however, initializing the adapters to minimize quantization error prior to fine-tuning plays crucial role in accuracy. Early approaches addressed this by reconstructing quantization errors via singular value decomposition (SVD) to initialize low-rank adapters (Li ct al. 2024 Guo et al. 2024). More cent works, such as RA-LORA (Kim et al. 2024a) and CLoQ (Deng et al. 2025). ), adopt advanced decomposition strategies and improved calibration to further mitigate this limitation. However, isting QA-PEFT methods remain restricted to LoRA, and no prior studies have explored the use of other advanced adapters for QA-PEFT, which will be discussed in the next section.
2.3 FOURIER TRANSFORM- BASED ADAPTERS
Sparse adapters have recently emerged as strong alternative to various low-rank adapters (Bhard- waj et al., [2024; Gao et al., 2024b Shen et al. 2025 Du et al. 2025). SHiRA (Bhardwaj et al. (024) proposes directly updating sparse subset of the weight matrix, enabling multi-adapter fusion. More recent methods adopt Folrier-related Transforms (FT) to represent the weight update W in the spectral domain by applying transforms along both the rows and columns of the matrix follows:
F = H'AHWWW
AW = H FH
(3)
Here, E Xd and H' E Buln are the orthonormal transform kernels. Prior works on these FT-based adapters have primarily focused on identifying suitable transform kemnlss. Fouri- erFT (Gao et al., 20245) employs the discrete Fourier transform (DFT), while LoCA (Du et al.) 2025 replaces the DFT with the diserete cosine transform (DCT) to avoid discarding imaginary
3

---

Published as a conference paper at ICLR 2026
components. SSH (Shen al. 2025) instead leverages the discrete Hartley transform (DHT) for the same purpose. As these kernels are composed of sinusoidal functions, F corresponds to the coefficients of the frequency components, which collectively represent A AW. We denote DCT and DHT-based adapters as DCA and DHA throughout the paper.
Since the transform kernels fixed matrices, F is the only learnable parameter during fine-tuning. To reduce the number of trainable parameters, F is treated as a sparse matrix. Specifically, F Scatter(c, E) constructed from value vector с Rº and an index list E Nex2, where Scatter assigns Bi,,) c for < / S p 1, with all other entries fixed to zero throughout training and inference. the initialization stage, since there no information on AW, previous works generally select the locations E randomly and the values of the spectral coefficients are initialized to zero (Gao et af. 2024b, Du al. 2025) SSH (Shen et al., 2025) proposes an advanced parameter selection strategy under the assumption that the frequency patterns of pre-trained and fine-tuned weights are similar. It first transforms the pre-trained weights and selects half of the positions with the largest spectral coefficients, while the remaining half are chosen randomly.
Overall, previous works demonstrate that FT-based adapters achieve superior accuracy improve- ments in full-precision fine-tuning compared to low-rank adapters. However, their advantages over low-rank adapters have only been empirically demonstrated, without theoretical justification. In addition, transforms within FT-based adapters incur heavy computational overhead (H and H' in Equation 3 Moreover, their application to QA-PEFT, particularly with initialization strategies that reconstruct quantization error, has not yet been explored.
METHODOLOGY
In this section, we present our proposed method, QWHA (Quantiaation-Aware "alsh-Hadamar Adaptation). First, we present the formulation of our proposed WHT-based adapter. Next, we ana- lyze the key component that enables FT-based adapters to achieve greater representational capacity than low-rank adapters, and demonstrate why WHA, in particular, excels at mitigating quantiza- tion error during adapter initialization. Finally, we introduce a parameter initialization strategy that reduces quantization eтror and enhances fine-tuning capability. Note that the experiments in this section use the 4-bit quantized LLaMA-3.2 -322-33 model, with the total number of trainable parameters P(r)
Σ (dr,in + di,va) X fixed by setting = 64 across all adapters.
LElayers
3.1 QA-PEFT ADAPTER DESIGN
WHT-based Adapter (WHA) We design our proposed adapter by constructing the weight update as the transformation of a sparse matrix F through an orthogonal transform H Specifically, we adopt the WHT (Hedayat 1978 Kunz 1979), particular instance of the FT whose kernel consists only of +1 entries, for the transform H (details on WHT and other FT kernels are provided in Appendix B.Tp. Accordingly, our adapter formulated as follows:
Y (Wa + AW) s.t. AW FH-!
The advantages of our adapter design are discussed in the following paragraphs.
Full-Rank Adapter. FT-based adapters exhibit greater representational capability than LoRA variants because they offer higher rank capacity given the same number of parameters. The repre- sentational power of low-rank adapters is strictly bounded by their inner dimension r (Equation 2 In contrast, since the transform kernels in FT-based adapters are orthogonal and therefore full-rank,
the rank of the adapter depends solely on the sparse matrix F (Equation 3 and 4p. Given that nonzero parameters are selected uniformly at random, both rows and columns receive more than two parameters on average, then F achieves full rank Tmax min(d,, dom)) with high probabil- ity (Coja-Oghlan et al., 2020). Since our adapter initialization in Section 3.2 assigns at least a few elements to each channel and selects parameters independently per channel, the full-rank conditions are satisfied. Details of this condition are provided in Appendix B.2 Figure a()) a) presents the em- pirical analysis of the rank of adapter weights, normalized by the maximum achievable rank rmax and averaged across layers. While LoRA achieves less than 6.3% of the normalized rank, FT-based adapters are nearly full-rank. Hence, our proposed WHA exhibits high representational capacity.
4

---

Published as a conference paper at ICLR 2026
(a) Normalized Rank of Adapter Weights
(b) Cumulative Energy of Decomposition and Transforms on AWl
100
(Nomaalized to [AWall))
1.0
Normalized (%)
80
100.0
100.0
99.7
100.0
Adapter Type
100.0
100
Curulaaivv Energy
0.8
WHT (n=2.11) 2
WHA
0.6
(n=2.63)
DCT (n=3.10)
40
DHA
0.4
(n-11.34)
20
0.2
2.1
6.3
6.3
2.1
2.1
2:1
2-1
P(r=64)
0.0
Query
Value
Out
Gate
Down
0.4
0.8
Projection Type
Index (Normalized)
Figure 2: (a) Comparison of rank in weight updates between low-rank and FT-based adapters across linear layers. (b) Cumulative distribution of f2 norm of singular values and transform coefficients with Pareto hill index v for the quantization error A AWQ in the layer Value projection. The vertical blue line indicates point where the adapters have the same number of parameters.
Outlier Component Inclusion Ratio
(b) Quantization given Number of Parameters in Adapters
Adepter Typm
Adaptar Typu
30
WHA
DCA DHA WHA
LeRA
Ratio 19)
DCA
Nomm
Norm
20
Adaastrr typec
LoRA
DCA
DHA
P(r=64)
WHA
P(r-64)
Key
Value
Projection
Out
Gate
Down
parometers x10
Number parameters x10
Figure 3: (a) Average coverage outlier components within the selected parameters. (b) & norm of the layer output error after initialization on the 14y-l Key projection. The vertical blue lines indicate points where the adapters have the same number of parameters.
Single transform. Conventional FT-based adapters apply transforms to both the input and output dimensions of the sparse matrix F as denoted in Equation 3 However, we find no clear advantage of this approach over a single transform the context of quantization. Since quantization errors are defined group-wise within each output channel, the channels can be treated as independent, and multiple transforms do not improve the representational capacity (i.e., rank) of the adapter. There- fore, to avoid unnecessary operations, design WHA to perform single transform as described in Equation
Benefits WHT over other transforms. As discussed in Section 2.1 quantization errors exhibit heavy-tailed outliers. For QA-PEFT, where mitigating such errors is crucial, the adapter must cap- ture the outlier structure with a small number of parameters, as in the case of sparse adapters using the sparse matrix F. We strategically adopt the WHT for our adapter design to effectively capture such outliers (Hedayat 1978, Kunz 1979p. The WHT kernel consists only of +1 entries, and its basis functions adopt DCT or DHT, whose sinusoidal bases exhibit smooth transitions. This structural difference makes the WHT better aligned with abrupt changes such outlier values. Therefore, WHT inher- ently provides a more compact coefficient representation of quantization errors compared to DCT or DHT. We empirically demonstrate this by analyzing the cunulative energy in adapter parame- ters (Figure 2)))), defined as the Eg norm of coefficients from the transform of AWo in FT-based adapters, and the & norm of singular values of A AWQ in low-rank adapters. Both coefficients and singular values follow Pareto-like distribution (see Appendix B.3). which can be characterized by the Pareto hill index 77, where a smaller 7j indicates a sharper distribution (Arnold, 1983). Since the total cumulative energy equals l&W | the fastest convergence curve of WHT, with the smallest 7). demonstrates that it concentrates the largest portion of energy within a small number of coeffi- cients, enabling accurate reconstruction with limited number of parameters. As result, WHA ef- fectively compensates for quantization errors, particularly large- harg-magnitude ones from salient weight channels, shown empirically in Figure 3 For fair comparison, we use the same parameter ini-
square- wave patterns with sharp transitions. In contrast, prior FT-based adapters
5

---

Published as a conference paper at ICLR 2026
Algorithm QWHA Initialization Algorithm
Require: Weight quantization error AWQ E IR Activation E (bss), WHT matrix H
Require: Budget p. Accumulated budget p. whannel-wise budget (po...., ,, 1 E Ny
Require: Parameter value vector E IR,, index list E E Nox2
Initialize p. C, +
Set R f UΣ
UEUT SVD(XX) ) +- AdaAlloc(p, AW2), H
Set (po,- , :
Di Parameter budget allocation
for i = 0 dout do
Set v + (AWQ).. R
Set pppp....pp:11 f- Topk Index (vB
> Channel-wise parameter selection
Set B' < (bs....dpp):: B(
Set 5+.... t B (B'B"T--
Value refinement
Accumulate 2- Pi
end for Update F R&duuxde E
tialization method described in Section 3.2 We define outlier coverage the ratio of the & sum of coefficients captured by the selected parameter locations to that of all coefficients corresponding to the top 10% magnitude outliers of A AWQ-
3.2 QUANTIZATION- ADAPTER INITIALIZATION
Objective Function. Our goal in initializing WHA minimize the layer output error (AWQ X) caused by weight quantization, using coefficient matrix F with p non-zero elements. Formally, objective is given by:
arg min JAWQ FH X E
where | denotes Frobenius norm. Following the reduction procedure used in Frantar et (0023) and Deng et al. 2025), this reduces to:
arg min JAW FH -R F
(6)
Here, R USL/2 is the invertible square root of the Hessian matrix attained by SVD as XX = T A detailed derivation on this reduction is provided in Appendix C.I As we aim find sparse Fcc, E) that minimizes Equation 6) it constitutes an NP-hard sparse approxi- mation problem (SAP) (Natarajan 1995). To make this problem more tractable, we decompose into two subproblems: first, parameter selection determine the locations of the nonzero elements to fine-tune (E); and second, value refinement to optimize the values of the selected positions (c).
Parameter Selection with AdaAlloc. Given a number of parameter (budget) for a layer, a naive selection method to reduce quantization error is choose the p sagget-magnitude elements from the dense solution A AWg H of Equation 6 However, since large-magnitude coefficients are of- ten clustered a few channels containing outliers, parameters become overly concentrated small number of channels. As result, magnitude-based selection yields low-rank F. degrading fine-tuning capability. Conventional methods prevent this rank reduction by incorporating random selection. For example, LoCA initializes parameter locations randomly and then optimizes these locations during fine-tuning. Thus, from the perspective of initialization, LoCA is equivalent to ran- dom selection at this stage. Additionally, SSH allocates half of the parameters randomly, while it selects the other half based on magnitude. However, these randomness- high layer output error because they fail to capture the parameters critical for reducing the error. To construct sparse F that is high-rank and minimizes initialization error, we first allocate the param- eter budget adaptively across output channels in proportion their activation error magnitudes:
approaches result in
[AW x).:
Di + p
AW X)

---

Published as a conference paper at ICLR 2026
Rank of Adapter Weights
Table 2: Layer output error (62 norm, scaled by x 10 after initialization. 'None' denotes the error before initialization.
100
Normalized Rtak 196)
Method
None
Random
SSH
Magnitude
AdaAlloc
Query 13.84
10.55
6.99
5.95
5.11
Method
AdaAlloc
Key
0.54
0.43
0.30
0.25
0.27
Random
Value
28.08
22.98
17.38
15.10
14.92
5SH
Out
4.66
3.70
2.70
2.24
2.01
Magnitude
Gate
1.88
1.57
1.25
1.04
1.13
25
Up
25.76
23.05
19.85
16.52
17.97
Layer index
Down
21.36
19.21
16.96
14.00
15.25
Figure 4: Rank of adapter weights for each parameter selection methods.
Average
7.21
5.96
4.57
3.82
3.86
where t temperature hyperparameter controlling allocation sharpness. Because the parameter budget must be an integer, apply the floor operation, which may leave fewer than duui parameters unassigned. These remainders are distributed to the output channels with the smallest allocations to ensure Eem Pt errors, F maintains full rank, while allocating more parameters important channels with higher quantization error. Next, within the budget of each output channel, we select parameters based on magnitude to effectively reduce the error. We compare the rank and layer output error of previous se- lection methods and AdaAlloc, as shown in Figure 4 and Table 2 For a fair comparison, all selection methods use the same value assignment method discussed in the next paragraph. AdaAlloc the only parameter selection method that simultaneously achieves nearly full-rank F and maintains low layer output error. Examples of the selected parameters are provided in Appendix C.2
Since all output channels receive parameter budgets proportional to their
Value Refinement. we solve the channel-wise SAP derived from Equation 6 for each ht output channel with given parameter budget p::
To assign each parameter a value that effectively reduces layer output error,
min BI where e = (AWQ).. B B
Here, is the 7th row of F. constrained have p; non-zero elements. We first select the largest- magnitude entries from the channel-wise dense solution 2g = =B 1- (AWq Next, rather than directly reusing the values from dense solution, refine them to minimize layer output error. Specifically, we re-project v onto the rows of B corresponding to the selected indices, denoted as B' RR, Xdn, which serve as the most relevant basis vectors:
x BB' (BB"""'''"))'
(9)
This allows the selected basis vectors account for the impact of unselected vectors, yielding a more accurate approximation. With- out this step, interactions among basis vectors are ignored, leading to suboptimal error reduction. Note that the refinement is applica- ble regardless of the parameter selection strategy. Figure 5 shows that refinement is crucial for reducing layer output error, presenting the layer output error after initialization with parameters selected by AdaAlloc. Further details on the еттоr analysis are provided in Ap- pendix C.3 Finally, E and for channel are initialized to the se- lected indices and their refined values x". Algorithm| I Summmriizes the initialization process, with details provided in Appendix C.4
Layer Output Error
7.21
6-
Error 1, Norm
3.86
4-
None w/o Refine Refined
Figure 5: Effect of refinement on average layer output error.
EXPERIMENTS
We evaluate the effectiveness of QWHA in terms of model accuracy and training efficiency. We first compare QWHA with oate-of-thhean art QA-PEFT baseline and sparse high-rank adapters including FT-based adapters. Then, we provide a detailed analysis of the impact of using WHA and AdaAlloc. Finally, demonstrate the efficiency of QWHA regarding WHA.

---

Published as a conference paper at ICLR 2026
Table Accuracy (%) evaluation results on CSQA and GSM8K benchmarks. 'QA Init.' denotes the existence of quantization-aware initialization.
Adapter
QA
Cocfficient
LLaMA-3.1-8D
LLaMA-3.2-3B
Mistral-7B-vO..
Bits
Method
Type
Init.
Selection
CSQA
GSM8k
CSQA
GSM8k
CSQA
GSM8K
Pre-trained
70.78
6.22
64.99
3.18
70.49
13.72
16
Fine-tuned
71.84
59.74
66.43
44.80
71.87
54.51
GPTQ
69.11
2.58
64.43
3.34
69.54
10.39
CLoQ
LoRA
69.58
53.83
65.48
39.27
71.32
52.01
SHiRA
Sparse
Random
71.07
54.36
63.10
40.71
70.88
51,02
LoCA
LoCA
71.45
54.36
40.33
71.55
47.99
SSH
DHA
X
SSH
70.75
53.98
65.83
39.80
71.57
47.99
QWHA
WHA
AdaAlloc
71.50
56.10
66.11
41.47
71.70
53.68
GPTQ
67.76
2.65
61.49
2.43
67.57
1.29
CLoQ
LoRA
68.71
53.75
39.20
69.91
46.25
SHiRA
Sparse
Random
69.68
45.49
35.33
69.36
46.70
LoCA
DCA
X
LoCA
70.21
53.15
63.30
36.69
69.64
46.10
SSH
DHA
X
SSH
69.86
50.34
63.57
38.13
69.65
47.15
QWHA
WHA
AdaAlloc
70.50
55.34
64.80
39,58
70.22
47.84
GPTQ
41.00
0,45
42.90
0.08
45.91
CLoQ
LoRA
56.49
33.89
54.89
26.53
61.80
33.36
SHiRA
Sparse
X
Random
51.84
27.74
52.91
22.59
59.08
33.57
LoCA
DCA
X
LoCA
56.71
33.97
53.87
23.88
62.03
33.89
SSH
DHA
X
SSH
56.06
30.55
54.01
25.77
62.31
32.06
QWHA
WHA
AdaAlloc
60.98
37.83
57.03
29.11
63.84
35,33
Models and Datasets. We evaluate QWHA on the Mistral-7B-v0.3 (Mistral Al 2024) and LLaMA (Grattafiori et al. 2024) model families, including LLaMA-3.1-88 and LLaMA-3.2-33 We evaluate the models on both general instruction-follooinng datasets and arithmetic reasoning tasks for the models fine-tuned mathemat- ical reasoning benchmarks. For instruction fine-tuning, we use the Stanford-Alpace dataset (Taori al. 2023 with 52k samples. We evaluate on zero-shot commonsense question answering (CSQA) Gao et al. 2024a).. covering seven multiple-choice benchmarks(Clare et al. 2018: 2019 Zellers al., 2019 Talmor al. 2019 Bisk et al.) [020 Sakaguchi et al.) (021). For arithmetic reasoning, we fine-tune on the GSMBK (Cobbe et al. 2021) dataset and evaluate with zero-shot
tasks for the models fine-tuned on
chain-of-thought reasoning questions on its test set, following Cobbe et (202T).
Baselines. We include full fine-tuned model (Fine-tuned) and quantized model, which use GPTQ (Frantar et al. 2023) with MagR (Zhang et al. 2024) (GPTQMaggh) as baselines. We note that our method is also compatible with any other quantization schemes. We also include CLoQ, recent QA-PEFT method that shares our goal of layer output for low-rank adapters. Other LoRA-based methods (Kim et al. 2024a Liao et al.) (024) involving layer-wise calibration layer-wise parameter allocation are orthogonal our approach and can be integrated in future work. We evaluate sparse adapters, including SSH and LoCA (FT-based) and SHiRA (non FT-based). We note that LoCA further fine-tunes the randomly selected parameter indices via reparameterization with cost of additional training overhead. We also build advanced hybrid baselines that integrate transforms or parameter selection strategies from prior works into our schemes by applying DCA and DHA with our AdaAlloc, or applying various parameter selection strategies to our WHA.
reduction during initialization
Implementation Details. Following prior work, adapters are applied to linear layers with pa- rameter budget of P(r 64), and quantization is performed with group size of 64. Note that apply a scaling factor a z to all adapters, while the equations the preceding sections omitted by for simplicity. We the AdaAlloc temperature full-rank condition. Further description on the training hyperparameter including scaling factor a and temperature / are provided in Appendix D.
1. which suffices to the
We WikiText-2 (Merity al. 2016)
8

---

Published as a conference paper at ICLR 2026
Table Accuracy (%) evaluation results CSQA and GSM8K benchmarks with variants of adapter types and parameter selection strategies in LLaMA-3.2-3B 'QA Init.' denotes the existence of quantization-awar initialization, and 'Refine." denotes the value refinement during initialization.
Adapter
QA
Coefficient
4-hit
3-bit
2-hit
Refine
Type
Init.
Selection
CSQA
GSM8k
CSQA
GSM8k
CSQA
GSM8k
WHA
X
Random
X
66.00
40.94
63.53
37.60
54.03
24.41
WHA
Random
65.91
40.71
63.91
37.30
54.48
24.48
WHA
Magnitude
66.07
41.01
64.52
36.69
56.49
28.12
WHA
LoCA
65.75
40.94
63.73
36.92
53.93
21.15
WHA
SSH
65.96
40.78
62.92
36.92
54.20
27.14
WHA
AdaAlloc
66.11
41.47
64.80
39.58
57.03
29.11
DCA
AdaAlloc
65.34
39.72
64.77
37.30
55.95
27.29
DHA
AdaAlloc
65.92
40.84
64.35
38.89
56.05
27.52
Sparse
AdaAlloc
65.60
40.94
63.43
37.53
55.97
26.54
Accuracy the Number of Parameters
Table 5: Training time (hours) on Alpaca dataset.
42
41.7
40.5
40.4
Batch
30.7
39.4
38.7
29.3
Size
CLoQ
SHiRA
QWHA
SSH
LoCA
38
37.5
37.3
GSMBK
36
36.0
35.6
QWHA
12.5
15.5
18.2
63.3
92.3
CLoQ
7.1
8.2
9.7
45.8
53.4
33,9
5.0
5.5
6.0
26.1
30.1
(-2
[w4
148
1-
5w54
r-128
r-25e
0.03
0.06
0.12
0.24
0.49
0.97
1.95
3.89
4.1
4.3
4.6
13.3
16.5
Number of parameters P(r)
x10"
16
3.6
3.7
3.9
8.3
9.8
Figure Accuracy of CLoQ and QWHA.
calibration dataset for adapter initialization, following Deng et al. (2025), experiments are conducted NVIDIA A100 80GB GPUs.
to ensure generality, All
4.1 FINE-TUNED MODEL ACCURACY
Main evaluation. Table 3 shows that QWHA outperforms both low-rank adapters with quantization-wware initialization and conventional sparse adapters. In particular, the effectiveness of QWHA is evident in the 2-bit setting, where it achieves scores at least 2-3% higher than the initialization, sparse adapters, including FT-based adapters,
baselines. Without quantization-awe perform worse than low-rank adapters in several cases. This underscores the need for quantization- aware initialization, especially sub-4-bit settings where fine-tuning alone cannot fully restore per- formance. We note that task-specific results of the CSQA benchmark are presented in Appendix D.2
Effect of WHA and AdaAlloc. We further examine the effectiveness of WHA and AdaAlloc, with QWHA consistently outperforming both low-rank adapters and advanced variants of sparse adapters. Figure 6 for 4-bit quantized LLaMA-3 3.2--BBBB-BBBB 3.2-3B shows that increasing the number of param- eters in CLoQ cannot close the accuracy gap with QWHA, as QWHA with P(r 32) already surpasses CLoQ's maximum achievable score. This highlights the advantage of WHA, which pro- vides superior representational capacity than low-rank adapters. Table 4 further demonstrates that WHA and AdaAlloc achieve the best results in each respective category of adapter type and parame- ter selection method. We note that LoCA's post-hoc location selection undermines the effectiveness of quantization- aware initialization based the initially chosen parameters, unlike in PEFT. Abla- tions on the temperature in Equation and quantization group size are provided in Appendix [D3 and D.4 respectively.

---

Published as a conference paper at ICLR 2026
4.2 COMPUTATIONAL EFFICIENCY
Training time for QWHA on the Alpaca dataset with LLaMA-3. 1-8B is reported in Table where QWHA achieves substantial speedup over previous FT-based adapters by leveraging WHA. WHA employs a single transform instead of the double transform used in conventional FT-based adapters, while achieving higher accuracy. Moreover, the fast recursive WHT kernel replaces matrix multi- plications with smaller number of additions and subtractions. In contrast, the recursive kernels of DCT and DHT, which require the DFT, are slower than direct matrix multiplication due to du- plicated computations for imaginary parts. As result, WHT achieves a similar training time to low-rank adapters or to simple sparse adapter. In contrast, LoCA incurs additional latency even compared to SSH, due to the training of location parameters. Memory usage remains almost iden- tical across all adapters with the same number of parameters. Detailed results on the training time of each transform kernel and the memory usage of each adapter provided in Appendix D.5 In addition, initialization latency and memory usage each adapters are provided in Appendix D.6 and inference throughput and memory usage provided in Appendix D.7
CONCLUSION
In this work, we introduce QWHA, novel QA-PEFT framework featuring a Walsh-Hadamard transform-based adapter and its quantization-aware parameter initialization scheme. WHA offers strong fine-tuning capability and excels in quantization error reduction. The proposed AdaAlloc scheme facilitates both fine-tuning and quantization error reduction during parameter selection, while parameter refinement enables substantial quantization eπrоr reduction. We validate QWHA across diverse models and datasets, where it consistently outperforms existing baselines in accuracy and demonstrates its effectiveness. We also show that using WHA with a single transform provides computational benefits, enabling more efficient training than conventional FT-based adapters.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This paper presents research aimed at advancing the efficiency of large language models through crantization-awarr parameter efficient fine-tuning. We acknowledge that large language models carry potential societal risks, including biases and misuse; however, methodological improvements training and inference efficiency. We believe that our contributions do not introduce additional harms beyond those already inherent in the underlying models.
study focuses solely on
Regarding the use of large language models this paper, we employed them as auxiliary tools for revising writing, checking grammar, and correcting typographical errors, but they did not play significant role in research ideation or substantive writing that would warrant their consideration as contributors.
REPRODUCIBILITY STATEMENT
This paper introduces a novel algorithm for quantization-awa aware parameter efficient fine-tuning adapter design and its initialization. To ensure reproducibility, provide the full source code as an anonymous, downloadable package in the supplementary materials, including detailed instruc- tions for environmental setup, training, and evaluation. All datasets used in experiments for fine-tuning and evaluation are publicly available, and we supply preprocessing steps and data prepa- ration scripts guarantee consistency with our reported results. Furthermore, complete proofs and derivations of our theoretical claims are presented in the appendix, with additional clarifications provided in the supplementary materials. Together, these resources are intended to fully enable independent reproduction and verification of our results.
REFERENCES
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Systematic outliers large language models, 2025.
Barry C. Amold. Pareto Distributions. International Co-operative Publishing House, 1983. ISBN 9780429169410. doi: http://bo..rr/10.111//11111111
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi. Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-fres 4-bit inference in rotated lims. Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS), 37: 1P:100113-00244: 2024.
Kartikeya Bhardwaj. Nilesh Prasad Pandey. Sweta Priyadarshi, Viswanath Ganapathy, Shreya Kadambi, Rafael Esteves, Shubhankar Borse, Paul Whatmough, Risheek Garrepalli, Mart Van Baalen, Harris Teague, and Markus Nagel. Sparse high rank adapters. In Proceedings the 38th International Conference on Neural Information Processing Systems, NeurlPS '24, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXiv:1905./0044 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1003007,0,. 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemy Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL btttps://arx
lorg/abs/2111 wwwaa///...... 14168
11

---

Published as a conference paper at ICLR 2026
Amin Coja-Oghlan, Alperen A. Ergur, Pu Gao, Samuel Hetterich, and Maurice Rolvien. The rank of sparse random matrices. The Proceedings of the 31th ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 579-5599559 591, 2020.
Dao-AlLab.
fast-hackmmard-tranfform
Laai-aao---LLLa. AlLab/
East hadamard transform 2024, Accessed: 2025-05-17.
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L.. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhanng Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang, Mingming Li, Ning Tian, panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha. Zhe Fu. Zhean Xu, Zhenda Xie, Zhengyan Zhang. Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning, 2025. TtLLhttps::/ /ttpss//aaavvvooo wrt.:/s//////oooeeoo.oo..os.s /bw.///s111. 2201 12948
Yanxia Deng, Aozhong Zhang, Naigang Wang, Selcuk Gurses, Zi Yang, and Penghang Yin. Cloq: Enhancing fine-tuning of quantized miss via calibrated lora initialization. Transactions on Ma- chine Learning Research (TMLRR) 2025.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetun- ing of quantized llms. Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS), 36:10088-100 10115, 2023.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzumov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless IIm weight compression, 2024.
Zhekai Du, Yinjie Min, Jingjing Li, Ke Lu, Changliang Zou, Liuhua Peng, Tingjin Chu, and Ming- ming Gong. Loca: Location-aware cosine adaptation for parameter-efficien fine-tuning. 13th International Conference on Learning Representations (ICLR), 2025.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan-Adrian Alistarh. Gptq: Accurate post- training quantization for generative pre-trained transformers. In 11hh International Conference on Learning Representations (ICLR), 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024a. URL httpp:::/eeeodoooorrr///r///////
12

---

Published as a conference paper at ICLR 2026
Ziqi Gao, Qichao Wang. Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficien fine-tuning with discrete fourier transform. Proceedings of the 41st Inter- national Conference Machine Learning (ICML), 2024b.
Diakoumis Gerakoulis and Saeed Ghassemzadeh. System and method for generating orthogo- nal codes, Mar 2004.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Han Guo, Philip Greengard, Eric P. Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. 2024.
A. Hedayat. Hadamard matrices and their applications. The Annals of Statistics, 6, 11 1978. doi: 10.1214/aos/1 176344377
Ashok S. Hedayat, Neil J. A. Sloane, and John Stufken. Orthogonal Arrays: Theory and Applica- tions. Springer Series in Statistics. Springer, 1999. ISBN 978-0-387-98766-1
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. 10th International Conference on Learning Representations (ICLR), 2022.
Minsoo Kim, Sihwa Lee, Wonyong Sung, and Jungwook Choi. Ra-lora: Rank-adaptive parameter- efficient fine-tuning for accurate 2-bit quantized large language models. In Findings of the Asso- ciation for Computational Linguistics 2024 (ACL), pp. 15773-15786, 2024a.
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization, 2024b.
Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix adaptation, 2024.
Henry O. Kunz. On the equivalence between one-dimensional discrete walsh- hadamard and mul- tidimensional discrete fourier transforms. IEEE Transactions Computers, C-28(3):267-268, 1979. doi: 10. 00.1199/CC.19991165533
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings the 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
Yixiao Li. Yifan Yu. Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. Loftq: bora-fine-tuning -wwee quantization for large language models. In 12th Interna- tional Conference on Learning Representations (ICLR), 2024.
Baohao Liao, Christian Herold, Shahram Khadivi, and Christof Monz. Apiq: Finetuning of 2-bit quantized large language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 20996-21020, 2024.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awarc weight quantization for Ilm compression and acceleration, 2024.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A. Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In The 35th Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
Shih- Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024.
13

---

Published as a conference paper at ICLR 2026
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantiza- tion with learned rotations, 2025.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. CoRR, abs/1609.078453 2016. UtL.Attpp/aarr...oooo/////EOOO..
Mistral AI. Mistral 7b v0.3. https: http::://uggiipip wea.ggingfface ngface 3. 2024. Model card, Apache 2.0 license, released 2024/11/30.
B. K. Natarajan. Sparse approximate solutions linear systems. SIAM Journal Computing, 24(2):227-234, 1995. doi: 10.1137/50995337922440000 URL
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, Sihan Zhao. Lauren Hong, Runchu Tian. Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li. Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models master 16000+ real-world apis, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Jennifer R. Seberry and Mieko Yamada. Hadamard matrices, sequences, and block designs. In Jeffrey H. Dinitz and Douglas R. Stinson (eds.), Contemporary Design Theory: A Collection of Surveys, pp. 431-560. Wiley, 1992.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models, 2024.
Yixian Shen, Qi Bi, Jia-Hong Huang, Hongyi Zhu, Andy D) Pimentel, and Anuj Pathania. Ssh: Sparse spectrum adaptation via discrete hartley transformation. Proceedings of the 2025 Con- ference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), 2025.
N. J. A. Sloane. A library of hadamard matrices. 2004. Accessed: 2025-05-16.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ: A ques- tion answering challenge targeting commonsense knowledge. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An -osructtio-followinng llama model. lab/ anford alpaca, 2023.
/
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#:: Even better Ilm quantization with hadamard incoherence and lattice codebooks, 2024.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint /tt:::/9000078880 2019.
Aozhong Zhang, Naigang Wang, Yanxia Deng, Xin Li, Zi Yang, and Penghang Yin. Magr: Weight magnitude reduction for enhancing post-training quantization, 2024.
Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan. Zhangyang Wang, and Jinwon Lee. Apollo: Sgd-like memory, adamw-level performance, 2025.
14

---

values originating from Published as a conference paper at ICLR 2026
A QUANTIZATION ERROR DISTRIBUTION
We present the distribution of quantization errors and their relationship to outliers the pre-trained weights, discussed Section 2.1 Figure 7 Figure Zia shows the overall error distribution, while Figure (bb) highlights the channel-wise similarity between quantization errors and pre-trained weights the t4th layer of ALaMA-332233- During quantization, values are divided by the quanti- zation scale, typically defined per group within each output channel, and then rounded to an integer and clamped within a range determined by the bit-width. Most quantization errors remain within this rounding range, but large-magnitde outliers are often clamped, leading to large errors. Because model aceuracy highly sensitive to outlier weights, their quantization errors can significantly de- grade performance. In QA-PEFT, it is therefore crucial to mitigate such outlier-iduuced errors dur- ing initialization by adapting the weights, particularly for salient outliers.
(+) Distribution Weight Quantization Error A1%
(b) L2 Norm for Each Output Channel
Hestogrem
Q-Q
Weight Quantizaionn
Pre-traind Weigh
120
400
300
Clamping Error Outliers
60
200
Clansping Error
40
Quarides
Figure 7: (a) Weight quantization error distribution and (b) its channel-wise similarity to the pre- trained weights 14th layer Key projection 4-bit quantized LLaMA-3.2-3B pixel represents the & norm of weight quantization errors (left) and that of pre-trained weights (right) for each output channel ordered by channel index from top-left to bottom-right.
Figure (b), each
15

---

Published as a conference paper at ICLR 2026
B WHT-BASED ADAPTER (WHA)
B.1 FT-BASED ADAPTER KERNELS
We describe class of Fourier-related transform (FT) kernels employed in our adapters and prior studies in this section (Gao et al. 2024b Du et al. 2025 Shen et al. 2025)
Hadam-d Transform (WHT). The Walsh-Hadamard Transform (WHT) matrix H in- troduced in Equation 4 is constructed following conventions in prior works (seng al., 2024
Ashkboos et al. 2024, For a dimension N 2", a WHT matrix H RN is defined recursively as the Kronecker product of smaller matrices:
1 22
H₂
HN H2 H2n-1,
(10)
where & denotes the Kronecker product. For non pfwer-of-ttwee twow dimensions, Hadamard matrices ex- ist for certain values (Seberry Yamada| 1992 Hedayat et al., 1999, Gerakoulis & Ghassemzadely 2004), which can be retrieved from [Sloane] (2004). More generally, for N = 2 where H m is known Hadamard matrix, the transform is defined as:
HN H @ Hm-
(11)
The rows of Hy form orthogonal basis, known Walsh-Hadamane bases, satisfying:
= HyHY IN.
(12)
The matrix H can be computed in On log time (Kunz 1979). In practice, HN can be pre- computed once and cached for reuse across layers the same size, incurring negligible cost in both computation and memory. To further accelerate computation, employ the Fast Hadamard multi- plication kernel from ( which avoids explicit matrix construction by using a fused kernel of only additions and subtractions.
Discrete Fourier Transform (DFT). FourierFT (Gao et al. 2024b) was the first study of FT- based adapters and used the discrete Fourier transform (DFT). The transform kernel H NNN is defined as:
2TTj
2Tjj N
COs
sin
0 o (13)
VN
Although effective, later works adopted real-valued FT variants to avoid the complex-domain na- ture of the DFT, since deep learning frameworks typically discard the imaginary components and compute only with the real values.
Discrete Hartley Transform (DHT). SSH (Shen et al. 2025) form (DHT), real-valued variant of the FT with kernel:
employs the discrete Hartley trans-
2mjji N
3k R
0 O
(14)
where cas( COS
sinæ. x.
Discrete Cosine Transform (DCT). LoCA (u et al.
2025
employs another real-valued FT, the
discrete cosine transform (DCT), whose kernel is:
j ==0
jk
O5kk N.
(15)
N COs
(2k++1) 2N
0 0<j<N
16

---

Published as a conference paper at ICLR 2026
B.2 RANK OF WHA
This section provides a detailed explanation of the full-rank property of WHA and its conditions, discussed in Section 3.1 and illustrated in Figure aa). To preserve the expressiveness of fine- tuned model under limited parameter budget, it is critical to ensure high rank capacity in the weight update. Unlike low-rank adapters, which inherently restrict the parameter subspace, WHA is sparsely structured yet can retain high representational capacity by maintaining full rank. This also holds in typical sparse adapters, including FT-based adapters.
We build on theoretical insights from prior work on sparse random matrices Coha-O et al. (2020), which provides conditions under which such matrices are full rank. Specifically, consider random sparse matrix F E Rdux where each input and output channel has k and non-zero entries on average. Then, is full rank when k, I 2 as din, deuu + oc, and thus full rank with high probability. Following the notations in Coja-Oghlan et al. (2020), we derive the corresponding condition for our setting to guarantee full-rank behavior in WHA
Condition Function. We define the probability generating functions for the distributions of ran- dom non-zero entries per column and per channel. Given that these distributions are degenerate, the generating functions and their derivatives are:
D(=) = = D'(=)= D'(1) = ki ((z) = ((=))=1d-, K'11== = L
(16)
(17)
Then, the condition function (22) that determines the full rank condition is given by:
2(2) = D _l_K(())
(1-)) -
k - - -1-
(18)
To ensure the full rank of the matrix A, the inequality must hold as:
((=) < +(0), v
(19)
Substituting the explicit forms for D(=), K (=), (=),, K'()) into Equation B.2 yields the right hand side as:
k + kzz k(1 -(1-1)
((2) = (1 - 1
(20)
As (2 0) =1 1- ,. the condition in Equation B.2 finally simplifies to:
(1 1 + kzl-1 k(1 1)
1<0, 0 0<251.
(21)
Practical Considerations. The inequality in Equation B.2 shows that the condition generally holds for integers 1,, 2 2. For the total number of parameters p r(d, + dout) with 2 2, we have k p/d,, > and i = p/daut > r under random selection, thus satisfying the full-rank condition when din, doat are sufficiently large. Importantly, AdaAlloe's per-channel allocation with remainder assignment and temperature control guarantees at least two elements in every channel (i.e.. / 2 2), which meets the sufficient condition required for the full-rank property. In addi- tion, although AdaAlloc selects coefficient indices within each channel based on the magnitude of AWQH, the coefficient distribution of AW under the WHT is close to a random normal distribution except for small portion of outliers, as the correlations across input rows nearly zero. Consequently, the selected index locations across input rows effectively behave like random choices. Empirically, parameter budgets corresponding to P(r 4) ensure at least two elements per row (i.e., 2 2), even for linear layers with large output dimensions, which might otherwise receive few parameters per input row. Hence, the full-rank conditions hold, and the matrix F in QWHA is nearly full rank.
17

---

Published as a conference paper at ICLR 2026
B.3 ENERGY CONCENTRATION OF WHT
In this section, we quantify the energy concentration property of WHT discussed in Section 3.1 using Figure b) and Figure 3a..
Distribution of Singular Values and Coefficients. Figure 8 presents the distributions of singular values from SVD and of transform coefficients sorted by their squared magnitudes. Here, the area under each plot is equal to |/ (details in the following paragraph). The distributions follow Pareto-like behavior, where sharpness can be quantified using the hill index v). The Pareto hill index value which implies the heaviness of tail. In fact, this is the reciprocal of the Pareto tail index, defined as the mean the log ratios of consecutive order statistics from the top-k largest magnitudes. A smaller hill index 7) implies faster convergence of the cumulative distribution (Arnold [183).. Hence, WHT exhibits the sharpest distribution, making feasible to retain more information with fewer parameters P(r) when implemented in sparse adapter, shown in Figure
Singular Value and Coefficient Magnitude Distribution of AWg in each Decomposition and Transforms
Singular valltr Magnitude
x10
SVD
x10
DCT
x10
DHT
x10 WHT
Confficien cintt Magnitude
Coefticien Magnitude
Confficient Margnitule
1.5
1=11.34
3=3.10
1.0
n==2633
n=2.11
0.0
(Nommaizee)
Index (Normalized)
Index (Normaltred)
(Normalized
Figure 8: Singular value and coefficient magnitude (squared) distributions with the Pareto hill index 7 in the 14th layer Key projection of LLaMA-3.2-3B.
Energy of Singular Values and Coefficients. Throughout this work, denote the squared & norm of the singular values in decomposition or the spectral coefficients in transform. We show that the total energy of both SVD and orthonormal transforms reduces the Frobenius norm of the transformed matrix. (For example, in Figure g the area under each curve corresponds to A AWol 1
the term energy to
18

---

Published as a conference paper at ICLR 2026
Let M& m be matrix, and let its singular value decomposition (SVD) be M = UEV', where E Rmxm and V E R are orthonormal matrices, and Σ R=*x" is diagonal matrix with entries of the diagonal for i l,.... min(m, n). Define F MH for an orthonormal matrix H E RRxx, Then, we have:
Proposition 1.
min(m,.))
|M
Σ 1=1
of F
Proof.
(i) Identity || 1|
Di=1
Imla(m.)) of-
Given M UEVT, T, due the orthonormality of U, M M reduces to:
M T = (UEVT)) = VV UUV -VETET
(22)
Therefore, by the eyclic property of trace and the orthonormality of V . I reduces to:
M M tr(M M) = tr tr(V MV = tr tr M(V V)) trE Σ) = Σ -..
(23)
Since Σ Σ is diagonal with entries of for i ..... min m, n):
min(m.n)
M
Σ
o
(24)
1=1
( Identity |M |
Given M H,,O0 due to the orthonormality of H:
F (MH) (MH) = H MH.
(25)
By the cyclic property of trace,
|| reduces to:
Fl tr ((M M) H)) (r M) = M
(26)
We note that this equivalence also applies the coefficients defined with F' H'AHH, with H' E pmxm, such that || || F
Outlier Reconstruction Ability of WHT. Due to its energy concentration ability discussed above, WHT can reconstruct quantization enrors with large outliers during initialization, which is critical for final model performance. Table (6 presents the numerical values corresponding to Figure a), showing the proportion of outlier coefficients captured by each adapter.
Table 6: Percentage of outlier coefficients captured by each adapter under a parameter budget of
P(r = 64) in the 14th layer Key projection of LLaMA-3.2--0 3.2--BB. Higher is better.
Adapter Type
Query
Key
Value
Out
Gate
Up
Down
Average
DCA
6.62
12.50
11.68
6.31
4.62
4.34
4.53
7.23
DHA
18.82
32.29
21.98
13.19
14.14
8.00
11.01
17.06
WHA
20.49
33.60
23.30
14.00
15.20
8.72
11.53
18.12
19

---

Published as a conference paper at ICLR 2026
C QUANTIZATION-WWARR INITIALIZATION OF WHA
C.I OBJECTIVE FUNCTION
We reduce the original optimization objective in Equation the form in Equation 6 following
the approach of Frantar et al. (202)) Deng et al. (2025). Using the notations from Section 3.2 the reduction proceeds as follows:
AAWx - x |3 =AW FH X
(27)
= =tr((AWo FH X (AW FH
1T)
(28)
= -tr((AWQ -F-H RR (AWQ FH FH
(29)
=I(AWQ -FH R F
(30)
=1AWQR FFFFH R
(31)
where R = 12 E xd. is an invertible square root of the Hessian Gram matrix X This term is obtained by applying the SVD XX UEUT, where Σ contains the eigenvalues the diagonal and is the matrix of orthonormal eigenvectors. Following Deng et al.) (2025), we add small regularization term À 0.0001 tr( (r(X) X /dd the diagonal if R is not originally invertible. This reduction allows us to replace X with R. enabling efficient and effective calibration using multiple input data points. Rather than solving the optimization problem separately for each sample X, we can accumulate the contribution of activations via R and solve single reduced problem.
C.2 PARAMETER SELECTION STRATEGIES
We present the parameter selection patterns of each method discussed in Section 3.2 As shown in Figure 9 magnitude-base selection allocates parameters to a limited number of channels, while conventional methods such SSH and LoCA incorporate random selection to avoid rank reduc- tion. However, these approaches fail to reduce quantization error during initialization because the selected parameters are not optimal for error reconstruction. In contrast, AdaAlloc identifies the most important locations within each channel while preventing rank reduction through per-channel budgets, thereby providing the most effective initialization and fine-tuning.
Selected Coefficient Locations
Magnitude
SSH
Random
LoCA
AdaAlloc
100-
s000
500
100
1000
Sagur Narel 10c
2003
Figure 9: Parameter selection patterns and two example zoomed-in results of each method in the layer Query projection of LLaMA-3.2-3B.
20

---

Published as a conference paper at ICLR 2026
C.3 VALUE REFINEMENT
We present the layer output error after WHA initialization with and without value refinement in 7 discussed in Figure Section 3.2 Without refinement of the selected coefficients in the initial dense solution matrix AWQ H, correlations among the columns are ignored, and the impact of sparsifying other columns cannot be considered, leading suboptimal error reconstruction.
Table 7: Layer output error (l2 norm, scaled by x10") after initialization with and without value refinement in 4-bit quantized LLaMA-3.2-3BB Parameters are selected by AdaAlloc. 'None' denotes the error before initialization.
Method
Query
Key
Value
Out
Gate
Up
Down
Average
None
13.84
0.54
28.08
4.66
1.88
25.76
21.36
7.21
W/o Refinement
11.39
0.62
26.21
4.39
2.11
24.33
20.97
7.06
Refined
5.11
0.27
14.92
2.01
1.13
17.97
15.25
3,86
C.4 CHANNEL- PARAMETER SELECTION AND INITIALIZATION
We provide a detailed description of the formulation and solution of the sparse approximation prob- lem underlying Algorithm T. based on the notations in Section 3.2
Sparse Approximation Problem. With a channel-wise breakdown of the objective in Equation 6 the goal to initialize the i-th channel of the parameter matrix F, denoted F... given the per- channel parameter budget pt. The objective is for H closely approximate the projected quantization error (,, R in the l sense. As we constrain F., to have exactly non-zero elements, the term F., H R becomes a sparse linear combination of standard basis vectors:
F. Σ Fij. e e H-1R,
(32)
where e(ik) is the ji-tth standard basis vector. Since elie) H-1 R corresponds the ju-th channel of H R, the problem reduces to selecting rows from H R that best approximate (AWo),,
Greedy Algorithm for Sparse Approximation. The problem generalizes to a standard sparse approximation problem: given full set of basis vectors 3 {ui, U2,...., Un with each u; E Rd. we aim to select & vectors whose linear combination best approximates a target vector Rd, We represent the sparse coefficient vector basis vectors. Formally, we solve:
T2, 2x] R$, corresponding to the selected
min v x subject to ||^||o k;
(33)
where B E Rxx is submatrix formed from selected rows of the original basis. In our setting, B == - R = A.W,,, R. and =
Since this problem is NP-hard, we adopt greedy approximation. We first compute H, which is in fact the non-sparse solution to the objective in Equation 6 and select the entries of with the largest magnitudes. Let the corresponding indices be 11, fg..... ig. and define the selected basis B' [uij.... U.y.). We then solve a least-squares problem over the selected support:
vB-1
x= = argmin . [1 dig Iiy B' - = DB' +B (B'BTB
(34)
While this solution numerically optimal when is orthogonal, we empirically demonstrate its effectiveness under general conditions. Combined with our AdaAlloc-based parameter allocation strategy, this initialization consistently yields high quantization error reconstruction ability while maintaining full rank capacity.
21

---

Published as a conference paper at ICLR 2026
D EXPERIMENTAL DETAILS AND ABLATIVE STUDY
D.1 FINE-TUNING HYPERPARAMETERS
We follow the hyperparameter settings adapted from Gao et al. (2024b) and Deng al. 2025
Training is performed using the A optimizer (Loshchilov & Hutter 2017). Table & reports the key settings, including minibatch size, weight decay, dropout ratio, learning rate scheduler, maxi- mum sequence length, number of training epochs, warmup ratio, and the adapter scaling factor a, which can be applied to adapters in Equation throuuhh Equation Due to an implementation detail in our codebase, the explicitly specified scaling factor is internally divided by the layer input dimen- sion dnn As consequence, the actual scaling applied during training is Octtective = Clexplicit/ din- To match the effective scaling used in CLoQ (i.e., Oeffective ~ 1.0), we set the explicit scaling factor to Oesplicit = 4000, which is close to the typical input dimensions of our models (3072 for LLaMA-3322- 3B and 4096 for LLaMA-3.1-88 and Mistral-7B-v0.3) Under this implementation, the effective scaling becomes O'effective 4000/de din s 1.0, ensuring consistent gradient scaling between low-rank. The learning rates for each combination of model, task, method, and bit-width are summarized in Table We note that 128 sequences of length 2048, randomly sampled from the WikiText-2 (Merity al. 2016) dataset, are used as a calibration set for quantization and adapter initialization, these processes are robust to the choice of dataset (Frantar et al., 2023) Zhang et al. [2024] Deng et al.)
2025). The total number of parameters for P(r = 64) reported in Table 10, broken down by each projection, the layers containing these projections, and the entire model.
Table 8: Hyperparameter settings for Alpaca and GSM8K training
Dataset
Alpaca
GSMSK
Method
CLoQ
QWHA
CLoQ
QWHA
Optimizer
AdamW 64
Batch Size
LR Scheduler
cosine 512
Max Sequence Length
Epochs
3
6
Warmup Ratio
0.1
0.03
Weight Decay
1
0.1
Dropout
0.1
0.1
0
Scale
4000 / d.n
1
4000 / dsn
Table 9: Learning rate for each model and bit widths on Alpaca and GSM8K training.
Alpaca
GSM8K
Model
Bits
CL.Q
QWHA
CLoQ
QWHA
4
le-5
3e-5
le44
5e-5
Llama-3.1-8B
3
le-5
3e-5
le-4
7e-5
2
lc-5
2c-5
7e-5
5e-5
4
le-4
3e-5
le-4
7e-5
Llama-3.2-3B
3
le-4
3e-5
le-4
le-4
2
2e-4
5e-5
le-4
2e-4
4
3e-5
5e-6
3e-5
2e-5
Mistral-7B-v0.3
3
2e-5
5e-6
3e-5
3e-5
2
2e-5
7e-6
3e-5
3e-5
Table 10: Total number of parameters at P(r 64) for each projection, layer, and model.
Model
q.proj
k. proj
v: v.proj
proj
gate proj
up-proj
down.proj
per-layer
per-model
LLaMA-3,1-8B
524288
327680
327680
524288
1179648
1179648
1179648
5242880
167772160
LLuMA-3.2-3B
393216
262144
262144
393216
720896
720896
720896
3473408
97255424
Mistral-7B-v0.3
524288
327680
327680
524288
1179648
1179648
1179648
5242880
167772160
22

---

Published as a conference paper at ICLR 2026
D.2 COMMONSENSE QUESTION-ANSWERE BENCHMARK
We present the detailed accuracy scores on the commonsense question answering (CSQA) bench- mark in this section, while the averaged scores reported in Section The results for each of the seven individual tasks, are provided in Table TT and Table corresponding to Table and Table respectively.
Table 11: Accuracy (%) evaluation results of CSQA benchmarks.
Model
Bits
Method
Arc-c
Arc-e
BoolQ
Hella.
Obqa
Piqa
Wino.
Average
LLaMA-3.1-8B
16
Pre-trained
53.41
81.10
82.11
78.91
44.80
81.23
73.88
70.78
Fine-tuned
56.40
81.57
81.99
79.85
46.40
81.99
74.66
71.84
4
GPTQ
48.98
78.96
81.77
75.87
45.00
79.87
73.32
69.11
CLoQ
50.34
79.50
83.12
75.63
45,40
79.92
73.16
69.58
SHiRA
53.24
81.14
82.81
78.76
81.56
73.40
71.07
LoCA
55.12
80.98
83.06
79.56
47.60
81.56
72.30
71.45
SSH
54.69
78.62
83.18
79.44
44.60
81.56
73.16
70.75
QWHA
55.20
80.26
83.64
79.62
47.00
81.99
72.77
71.50
3 GPTQ
48.81
76.52
81.59
73.74
43.80
78.24
71.59
67.76
CLoQ
49.91
77.48
82.42
74.30
44.80
79.05
73.01
68.71
SHiRA
53.41
79.17
80.98
77.63
44.80
80.30
71.51
69.68
LoCA
54.61
79.88
81.13
78.25
45.20
80.25
72.14
70.21
SSH
53.24
75.97
82.81
80.27
43.00
81.56
72.14
69.86
QWHA
54.69
80.05
81.68
78.70
45.20
80.79
72.38
70.50
GPTQ
24.23
38.51
53.15
36.73
25.00
$7.67
41.00
CLoQ
37.80
56.19
67.74
64.14
35.40
72.31
61.88
$6,49
SHiRA
33.28
51.31
64.31
56.76
31.40
68.72
$7.14
51.84
LoCA
37.63
59.22
68.13
63.79
35.60
70.95
61.64
56.71
SSH
39.08
$7.79
67.58
62.00
35.00
71.38
59.59
56.06
QWHA
41.72
64.94
74.62
68.11
37.40
74.54
65.51
60.98
LLaMA-3.2-3B
16
Pre-trained
45.99
71.63
73.39
73.61
43.00
77.48
69.85
64,99
Fine-tuned
48.29
73.15
74.95
76.71
43.80
77.91
70.17
66.43
GPTQ
44.97
70.83
74.95
71.34
42.60
77.15
69.14
64.43
CLoQ
47.70
72.35
74.95
74.25
42.40
77.86
68.82
65.48
SHiRA
43.17
68.01
71.80
73.33
41.60
76.99
66.85
63.10
LoCA
47.78
73.40
74.34
74.33
42.20
78.13
65.59
SSH
47.95
73.32
75.35
74.38
42.80
78.67
68.35
65.83
QWHA
48.98
73.15
75.78
74.44
41.60
79.00
69.85
66.11
GPTQ
42.92
66.08
70.86
68.29
40.00
76.12
66.14
61.49
CLaQ
46.33
71.25
72.97
72.41
41.40
78.35
67.72
64.35
SHiRA
44.54
68.43
72.26
71.31
40.40
77.75
65.67
62.90
LoCA
44.71
69.74
70.83
72.17
41.20
78.13
66.30
63.30
SSH
44.88
70.37
71.62
72.17
41.80
78.18
65.98
63.57
QWHA
47.18
72.64
72.51
72.72
41.80
79.71
67.01
64.80
GPTQ
26.62
38.89
54.28
39.32
29.00
59.36
52.80
42.90
CLoQ
35.24
56.27
66.02
59.77
37.40
70.35
59.19
54,89
SHiRA
33.28
$6.40
63.64
55.11
34,80
$7.30
$2.91
LoCA
34.64
58.42
64.46
56.26
34.60
71.44
57.30
53.87
SSH
34.81
58.33
64.65
56,21
70.51
$4,01
QWHA
37.29
61.99
65.26
61.76
37.20
73.88
61.80
57.03
Mistral-7B-v0.3
16
Pre-trained
52.30
78.24
82.14
80.42
44.20
82.26
73.88
70.49
Fine-tuned
55.03
80.05
84.19
81.09
45.80
82,43
74.51
71.87
GPTQ
51.37
76.52
80.55
79.71
44.00
81.72
72.93
69.54
CLoQ
54.52
78.58
83.91
81.09
44.00
82.37
74.74
71.32
SHiRA
53.50
78.41
82.69
80.46
44.60
82,43
74.11
70.88
LoCA
53.84
78.28
83.88
81.12
45.40
82.86
75.45
71.55
SSH
53.75
78.45
84.71
81.18
45.60
82.64
74.66
71.57
QWHA
54.69
78.79
84.74
80.93
45.40
82.70
74.66
71.70
GPTQ
49.23
75,00
77.06
78.22
42.20
80.63
70.64
67.57
CLoQ
52.99
77.44
80.55
80.37
43.40
81.39
73.24
69.91
SHiRA
51.37
77.31
79.57
79.52
44.20
81.45
72.14
69.36
LoCA
51.96
77.31
80.95
80.17
43.40
81.39
72.30
69.64
SSH
51.54
77.23
81.99
79.93
43.20
81.39
72.30
69.65
QWHA
52.56
76.94
82.32
80.31
44,20
82.21
73.01
70.22
GPTQ
26.37
41.46
54.37
48.88
29.80
62.62
57.85
45,91
CLoQ
43.94
66.46
74.25
70.95
38.00
75.41
63.61
61.80
SHiRA
39.33
64.27
71.38
66.58
36.60
73.88
61.56
59.08
LoCA
43.77
67.63
75.99
69.78
37.80
74.54
64.72
62,03
SSH
44.03
68.77
76.79
70.15
36.00
75.35
65.11
62.31
QWHA
45.39
69.78
78.53
71.97
37.60
76.44
67.17
63.84
23

---

Published as a conference paper at ICLR 2026
Table 12: Accuracy (%) evaluation results of CSQA benchmarks LLaMA-3.2-3B.
Adapter
QA
Coeff.
Bits
Refine.
Arc-c
Are-e
BoolQ
Hella.
OBQA
PIQA
Wino.
Average
Type
Init.
Selection
WHA
Random
48.55
73.70
74.53
74.43
43.20
78.67
66.00
WHA
Random
48.04
73.36
74.22
74.50
42.00
78.78
70.48
65.91
WHA
Magnitude
48.81
72.01
75.26
74.39
42.80
79.43
69.77
66.07
WHA
LoCA
47.61
73.06
74.16
74.18
43.20
78.56
69.46
65.75
WHA
SSH
47.61
73.40
75.26
74.45
43.00
78.07
69.93
65.96
WHA
AdaAlloc
48.98
73.15
75.78
74.44
41.60
79.00
69.85
66,11
DCA
AdaAlloc
47.10
72.47
73.24
75.14
43.00
79.11
68.75
65.54
DHA
AdaAlloc
47.70
73.06
75.60
74.61
41.60
78.67
70.17
65.92
Spurse
AdnAlloc
47.61
72.52
73.49
75.03
43.60
78. 13
68.82
65.60
WHA
Random
71.53
71.71
41.40
77 86
66.77
63.53
WHA
Random
44.97
71.38
71.38
72.38
41.80
78.62
66.85
63.91
WHA
Magnitude
47.35
72.31
72.69
72.40
41.00
79.38
66.54
64.52
WHA
LoCA
46.25
69.49
71.38
71.93
41.80
78.51
66.77
63.73
SSH
44.45
67.38
71.83
71.38
40.20
78.35
62.92
WHA
AdaAlloc
47.18
72.64
72.51
72.72
41.80
79.71
67.01
64.80
DCA
AdaAlloc
46.08
72.05
73.55
73.11
42.20
78.18 18
68.19
64.77
DHA
AdaAlloc
45.31
72.18
72.32
72.91
41.40
78.94
67.40
64.35
Sparse
AdaAlloc
45.65
70.75
68.35
72.77
41.00
77.58
67.96
63.43
WHA
Random
X
34.56
59.05
65.17
55.68
34.80
70.67
58.25
$4.03
WHA
Random
/
34.90
58.54
64.31
58.13
35.40
70.84
59.27
54,48
WHA
Magnitude
37.71
64.83
60.88
37.40
72.58
61.40
$6.49
WHA
LoCA
33.96
58.08
64.71
56.27
35.80
70.67
58.01
$3.93
WHA
34.39
55.77
64.50
36.00
59.19
$4.20
WHA
AdnAlloc
37.29
61.99
65.26
61.76
37.20
73.88
61.80
57.03
DCA
AdaAlloc
35.92
57.15
66.18
61.94
37.00
72.58
60.85
$5.95
DHA
AdaAlloc
36.09
58.59
66.67
61.12
36.00
72.52
61.33
56.05
Spurse
AdaAlloc
35.67
56.40
67.16
63.06
36.40
73.23
59.91
55.97
24

---

Published as a conference paper at ICLR 2026
D.3 ABLATION ON TEMPERATURE t
We evaluate the effect of the temperature parameter / in Equation 7 on the fine-tuned accuracy of LLaMA-3.2-31B under 4-bit quantization using the parameter budget P(r 64). The results show that model accuracy remains stable for temperatures between 0.5 and 1.0, while performance slightly degrades when is too small or too large. A low temperature distributes the parameter budget nearly uniformly across channels, preventing sufficient allocation to channels with large quantization er- rors. Conversely, an overly high temperature over-concentrates parameters in these channels, neglecting important coefficients within low-mase ow-magnitude channels and assigning unneces- sary parameters within high-magnitun ones, As a result, both excessively low and excessively high temperatures lead to decreased fine-tuned model accuracy. Within the robust range of E [0.5, 1.0], we selected = as the default setting, since it naturally favors allocating more parameters to outlier-included included, mranetmage channels while still maintaining stable empirical performance and methodological simplicity.
Table 13: Effect of temperature on GSM8k accuracy of LLaMA-3.2-31 under 4-bit quantization.
Temperature t
0.25
0.5
1.0
1.5
2.0
GSM8k Acc. ( (%)
40.11
41.39
41.47
40.64
40.04
D.4 ABLATION QUANTIZATION GROUP SIZE
We conduct an ablation study on the effect of quantization group size in the LLaMA-3.2-3B model using 2-bit quantization, where the impact of group size on model accuracy is most clearly observed, as shown in Table [4] Smaller group sizes provide finer granularity, leading to higher model accu- racy. However, they also incur greater computational overhead during the quantization and dequan- tization process due to the increased number of quantization parameters. Considering this trade-off, we adopt a group size of 64 for our experiments, consistent with prior works on quantization-aware PEFT.
Table 14: GSM8K accuracy (%) of QWHA on LLaMA-3.2-3B with 2-bit quantization using various quantization group sizes.
Group Size
32
128
256
Score
29.94
29.11
24.48
22.51
25

---

Published as a conference paper at ICLR 2026
D.5 TRAINING OVERHEAD OF ADAPTERS
Train time We compare the training time of adapters using single-transform and two-transform designs both WHT and conventional transform kernels, such the DCT used in LoCA and the DHT used SSH, in Table [5| While employing WHA reduces training time, applying it with single transform further decreases computation. The impact of a single transform is especially evident in DCT and DHT, where training time is substantially reduced since their computational overhead due to the transform larger than that of WHT. Note that DCT and DHT have identi- cal training times, as their computational cost is the same and differs only in the element values within the transform kernel. Our proposed WHA employs a ID WHT whereas conventional FT-based PEFT methods such as LoCA and SSH use 2D DCT and 2D DHT, respectively.
context quantization,
Table 15: Training time (hours) of FT-based adapters with different transform kernels on LLaMA- 3.1-8B with the Alpaca dataset.
WHT
DCT DHT
Batch Size
1D
2D
1D
2D
18.2
25.3
46.2
63.3
2
9.7
13.1
32.1
45.8
6.0
8.0
17.4
26.1
8
4.6
5.5
9.0
13.3
3.9
4.3
6.7
8.3
Memory Usage We report the memory usage of each method under the same experimental setting as in Section 4 using NVIDIA A100 80GB GPU. As shown in Table [6] QWHA shows memory usage comparable LoRA. SSH also exhibits similar memory usage as QWHA, since the only difference between them is the computation with pre-defined transform kernel matrix. Since this matrix is shared across layers, the memory overhead is negligible. contrast, LoCA incurs addi- tional memory consumption due to the training of location parameters, resulting in a few gigabytes of overhead depending on the batch size.
Table 16: GPU memory usage (GB) during fine-tuning on LLaMA-3.1-2 with 4-bit quantization
using the Alpaca dataset. All adapters use the same number of trainable parameters with P(r 64).
Batch Size
CLoQ
QWHA
LoCA
22.1
22.1
23.3
26.6
27.2
28.4
32.6
33.2
34.4
44.7
45.3
46.4
16
68.8
69.4
70.6
26

---

Published as a conference paper at ICLR 2026
D.6 INITIALIZATION OVERHEAD OF ADAPTERS
Initialization Time. Table 17 reports the initialization latency of the low-rank adapter (CLoQ) and the FT-based adapters (QWHA and SSH) under the 4-bit setting across different models. The initialization of CLoQ requires gathering activations and quantization errors, followed by SVD de- composition before solving the least-squares problem. In contrast, the initialization of FT-based adapters involves collecting activations and quantization errors, followed by parameter selection and value assignment through channel-wise transforms and solving least-squares problem. During this process, the fast Hadamard kerne allows QWHA with WHT to perform efficient computation. As result, QWHA achieves comparable initialization time to CLoQ, while SSH with the DHT kernel incurs significantly higher latency.
Table 17: Initialization latency (hours) of each method under the 4-bit setting.
Method LLaMA-3.2-3B
LLaMA-3.1-8B
Mistral-7B-v0.3
CLoQ
0.58
1.14
1.26
SSH
3.85
8.09
8.58
QWHA
0.66
1.34
1.46
Memory Usage. We measure the peak memory consumption during initialization for each method on the LLaMA-3.2-33 3.BBBB model with 4-bit quantization, broken down into cached Hessians and model weights (Table 18 While every methods require multiple matrix projections, CLoQ additionally performs SVD, which leads to higher memory usage in layers with large dimensions. Overall, QWHA achieves slightly lower memory footprint due its efficient block-wise computational im- plementation of the fast Hadamard transform.
Table 18: Memory usage (GB) and component breakdown during initialization of each method.
Method
Total Usage
Model Weight
Cached Hessian
CLoQ
12.77
3.04
7.10
SSH
11.93
3.04
7.10
QWHA
11.52
3.04
7.10
27

---

Published as a conference paper at ICLR 2026
D.7 INFERENCE OVERHEAD OF ADAPTERS
We investigate the inference throughput and memory usage of QWHA and CLoQ and present the results in Table [9 and Table 20 respectively. The evaluation uses prefill length of 2048 and generation length of 64, with batch size 128. We compare the FP16 pre-trained model, a quan- tized model with LoRA (corresponding to CLoQ), a sparse adapter (SHiRA), and FT-based adapters including WHA (QWHA), DCA (LoCA), and DHA (SSH).
As stated in SHiRA, sparse adapters provide a slight speedup over LoRA due to their simple scatter operations, compared to the low-rank matrix multiplications in LoRA. WHA introduces an addi- tional WHT operation, but due the fast Hadamard kemel,, its overhead remains small: only a 1.9% decrease in throughput compared to LoRA. In contrast, conventional FT-based adapters such as DCA and DHA incur substantial overhead, showing a 50.9% throughput drop. Therefore, although QWHA applies inverse WHT during inference, its overhead is marginal compared to LoRA, whereas other FT-based adapters experience significant efficiency degradation.
Regarding the inference memory usage, QWHA reduces overall memory usage by 13.0% compared to CLoQ, while both methods use 3.04 GB for model weights. This improvement arises from the use of sparse adapter with efficient scatter operations. In addition, because the inverse WHT in QWHA is implemented without heavy matrix multiplications, it incurs no additional memory overhead and in fact results in lower peak memory usage. Consequently, QWHA achieves more than a 10% reduction in total memory consumption.
Table 19: Inference throughput (tokens/sec) of pretrained and quantized models of each adapters.
Method
Pre-trained
LoRA
Sparse
WHA
DCA/DHA
Throughput (tokens/sec)
66.7
188.1
191.9
184.6
92.4
Table 20: Peak memory usage (GB) for each method.
Method
QWHA
CLoQ
Memory Usage (GB)
52.68
59.53
D.8 TRAINING CURVE
We analyzed the training loss, gradient norms, and convergence behavior throughout fine-tuning, and present the training curves and adapter gradient norms in Figure We observe similar convergence behavior of QWHA and CLoQ. The gradient norms remain on a comparable scale despite QWHA's large nominal scaling factor. This is because the effective scaling factor of both QWHA and CLoQ is close to 1.0. Moreover, under the same effective scaling value, low-rank adapters involve matrix multiplications during backpropagation, which naturally downscales the gradient norms applied to their parameters. In contrast, the sparse adapter in QWHA does not undergo this process, resulting in gradient norms that remain stable and consistently about twice as large throughout training.
Train Adapters During Alpaca Fine-Tuning
Gradient Norm Adapters During Alpaca Fino-Tuning
QWHA
QWHA CLoQ
CLoQ
1.2
Gradlen
1.0
0.8
0.0
0.5
1.0
15 Epochs
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Epochs
Figure 10: Training loss (left) and gradient norm (right) of each methods during Alpaca fine-tuning in LLaMA-3.2-33 4-bit model.
28


Reviews:
[{"review_id": "uhldyoko1Z", "reviewer": "Reviewer_c9iU", "summary": "This paper addresses the challenge of deploying Large Language Models (LLMs) efficiently by integrating parameter-efficient fine-tuning (PEFT) with low-bit quantization (QA-PEFT). The authors identify two key limitations in current methods: (1) the limited representational capacity of LoRA-based adapters and (2) the high computational cost and lack of quantization awareness in existing Fourier-related Transform (FT) based adapters. To overcome this, the paper proposes QWHA (Quantization-Aware Walsh-Hadamard Adaptation). This method introduces two main contributions: (1) WHA, a novel adapter that uses a single, computationally efficient Walsh-Hadamard Transform (WHT) kernel, which the authors show is full-rank and effective at capturing outlier-induced quantization errors. (2) A novel quantization-aware initialization scheme, featuring \"AdaAlloc\" for adaptive parameter selection and \"Value Refinement\" to optimize initial values, which is the first (to our knowledge) such scheme for FT-based adapters. Experiments show that QWHA outperforms existing QA-PEFT baselines, particularly in 2-bit and 3-bit settings, while achieving training speeds comparable to LoRA.", "questions": "- To fairly assess the novelty of the WHA kernel itself, could the authors provide an ablation study comparing WHA, DCA (LOCA), and DHA (SSH) under two conditions: (a) all using the same proposed QA initialization (AdaAlloc + Refinement) and (b) all using a simple baseline initialization (e.g., random selection, zero-init)? This would isolate the true benefit of the WHT kernel versus the initialization scheme.\n\n- Could the authors please report the wall-clock time and memory overhead required for the QUANTIZATION-AWARE ADAPTER INITIALIZATION phase? How does the total deployment time (Initialization Time + Fine-Tuning Time) of QWHA compare to a method like CLoQ?\n\n- Do the authors have an explanation for the significant performance gap between the 16-bit fine-tuned model and QWHA on the GSM8k reasoning task? Does this imply a potential limitation of the WHA adapter in capturing parameters crucial for mathematical reasoning, or is it a general challenge for low-bit QA-PEFT?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality & Significance: The problem is well-motivated, sitting at the practical intersection of model compression and adaptation. The paper's primary originality lies in being the first to successfully design a quantization-aware initialization strategy specifically for an FT-based adapter. This bridges a critical gap, combining the superior representational power of FT-adapters with the error-mitigation necessary for low-bit quantization, which LoRA-based methods struggle with.\n\n- Quality: The methodology is concrete and well-supported by a diverse set of analyses. The design choices for WHA (using WHT and a single transform) are justified through clear empirical evidence, including rank analysis, cumulative energy analysis, and outlier component coverage. The two-stage initialization algorithm (AdaAlloc and Refinement) is a non-trivial and well-reasoned approach to solving the complex sparse approximation problem.\n\n- Clarity: The paper is clearly written, systematically identifying the limitations of prior work and presenting its solution. The figures and tables effectively support the authors' claims about why WHA is a suitable kernel and why the proposed initialization is effective.", "weaknesses": "- Unclear Ablation of Performance Gains: The main experimental comparison in Table 3 is potentially confounding. QWHA, which includes the proposed quantization-aware initialization (QA Init.), is compared against other FT-based adapters (LOCA, SSH) that do not have this initialization. It is unclear whether QWHA wins because: (A) the WHA kernel is intrinsically superior to DCA/DHA for this task, or (B) WHA is the only FT-based method benefiting from a powerful initialization. A fair comparison would require applying a comparable initialization to LOCA/SSH or ablating QWHA without its initialization.\n\n- Missing Analysis of Initialization Cost: The paper highlights QWHA's efficiency by comparing its training time to CLoQ (Table 5), showing they are similar. However, this ignores the pre-computation cost of the QUANTIZATION-AWARE ADAPTER INITIALIZATION (Algorithm 1). This process involves gathering activations, SVD, and, most notably, solving a least-squares problem (with a matrix inverse) for every output channel of every adapted layer. This complex procedure is computationally non-trivial, whereas LoRA's initialization is virtually instantaneous. Without reporting this initialization time, the claim of \"efficiency\" is incomplete, as the total time-to-model (Initialization + Training) may be significantly higher for QWHA.\n\n- Significant Performance Gap on Reasoning Tasks: While QWHA shows strong performance on CSQA, there is a very large performance degradation on the GSM8k benchmark (Table 3) compared to the 16-bit fine-tuned baseline. For instance, on LLaMA-3.1-8B, the 16-bit baseline achieves 59.74, while the 4-bit QWHA only reaches 56.10. This substantial drop is not fully addressed and may suggest that the proposed adaptation and error-compensation scheme struggles to preserve the complex reasoning capabilities of the model. Further exploration on other reasoning tasks (e.g., AIME, MATH) would be needed to verify if this is a fundamental limitation of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7xLILZAYiH", "reviewer": "Reviewer_RPse", "summary": "The paper presents QWHA (Quantization-Aware Walsh-Hadamard Adaptation), a novel approach for enhancing fine-tuning efficiency in LLMs under low-bit quantization settings. By integrating Walsh-Hadamard transforms within the quantization-aware training framework, the authors aim to mitigate quantization-related errors, thereby improving model performance without substantial computational overhead. The experiments conducted on several benchmarks substantiate the proposed method's effectiveness, indicating its potential to advance the state-of-the-art in efficient model adaptation.", "questions": "1. How does QWHA perform on diverse datasets that extend beyond instruction-following tasks? Can you provide insights into its performance on datasets related to summarization or question-answering?\n2. Could the authors elaborate on the decision-making process for hyperparameter settings, particularly the temperature parameter in AdaAlloc? Have you explored the performance implications of varying these settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of a Walsh-Hadamard Transform (WHT) presents a robust method for mitigating quantization error, enhancing the representational capacity of the model while maintaining computational efficiency.\n2. The authors provide thorough experimental validation using different models and a range of quantization levels, supporting the claims of improved performance and efficiency.", "weaknesses": "1. The experimental evaluation relies heavily on the Alpaca dataset, primarily focusing on instruction-following tasks. This raises questions about the generalizability of the approach to other domains or more complex datasets, such as those involving natural language understanding or generation tasks.\n2. The dependency on specific hyperparameter configurations (e.g., the temperature parameter in AdaAlloc) is only marginally discussed. There is a lack of sensitivity analyses that would indicate how variations in hyperparameters impact model performance and error rates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jGX8Tg5ToI", "reviewer": "Reviewer_L6oQ", "summary": "This paper introduces QWHA, a novel Quantization-Aware Parameter-Efficient Fine-Tuning (QA-PEFT) method. QWHA utilizes a Walsh-Hadamard Transform (WHT)-based adapter, termed WHA, designed to effectively mitigate quantization errors in Large Language Models (LLMs). The core contributions are a computationally efficient adapter design using WHT, and a sophisticated initialization scheme that combines adaptive parameter allocation (AdaAlloc) to preserve rank and target high-error channels, with a value refinement step to minimize initial error. Experimental results on LLaMA and Mistral models demonstrate that QWHA outperforms existing QA-PEFT methods in low-bit settings (2, 3, and 4-bit) and is significantly faster to train than other Fourier-related transform (FT)-based adapters.", "questions": "1. Regarding the adapter scaling factor `α` (Table 8), there is a massive discrepancy between QWHA (α=4000) and CLoQ (α=1)—could you elaborate on why this choice was made? Was `α` tuned independently for each method, and if so, have you investigated how such a large value impacts gradient norms, training stability, or convergence speed for QWHA?\n\n2. The main comparison in Table 3 pairs QWHA (with its dedicated quantization-aware initialization) against other Fourier-based methods that lack this feature, while Table 4 provides a more controlled comparison by applying AdaAlloc to all adapter types. In Table 4, `DCA + AdaAlloc` and `DHA + AdaAlloc` perform quite competitively with WHA—particularly in the 2-bit LLaMA-3.2-3B GSM8k setting. Could you comment on this? Does this suggest that the initialization scheme (AdaAlloc) is the dominant driver of performance, rather than the specific choice of transform kernel (WHT)?\n\n3. The justification for using a single transform (instead of dual transforms) is compelling from an efficiency standpoint, and Table 13 clearly shows the training time benefits. However, to fully validate this design choice, could you provide an accuracy comparison between 1D WHA and a hypothetical 2D WHA? This would empirically confirm that the second transform offers no meaningful accuracy gain in the QA-PEFT context, rather than just faster training.\n\n4. The AdaAlloc method uses a fixed temperature parameter `t=1` (Section 4), but there is no sensitivity analysis provided. Have you tested how varying `t` (e.g., t=0.5, t=2) affects parameter allocation sharpness and final performance? For instance, does a lower `t` lead to over-concentration on a small number of channels, or a higher `t` reduce effectiveness by spreading parameters too evenly?\n\n5. As noted earlier, the reported results lack error bars or confidence intervals from multiple runs. Given the known variance in LLM fine-tuning due to random seeds, could you comment on the stability of QWHA’s performance across different seeds? Have you observed consistent gains, or is there significant variability that might affect the reliability of the reported results?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Novel and Well-Motivated Adapter Design (WHA)**: A key strength of this work is the innovative use of the Walsh-Hadamard Transform (WHT) for the adapter, and what makes this choice particularly strong is its tight alignment with the inherent nature of quantization errors. From Section 3.1, we can see that the authors compellingly argue WHT’s square-wave basis functions are far better suited to capturing the abrupt, outlier-driven characteristics of quantization errors than the smooth sinusoidal bases of DCT/DHT employed in prior Fourier-based adapters—and this claim is not just theoretical, as demonstrated in Figure 2(b)’s energy concentration analysis, which clearly validates the superiority of WHT in this context. Additionally, the decision to adopt a single transform (instead of the dual transforms used in earlier work) is both theoretically sound and practically impactful: Section 3.1’s discussion of quantization error’s channel independence justifies the simplification, while Table 13 further shows that this design delivers significant training speedups without compromising performance, making the method much more viable for real-world applications.\n\n- **Effective Quantization-Aware Initialization Strategy**: The proposed two-stage initialization scheme stands out as a notable strength, with its effectiveness evident across Section 3.2, Algorithm 1, Figure 4, Table 2, and Figure 5. AdaAlloc (Eq. 7) offers a principled heuristic that skillfully balances two critical objectives: reducing quantization error by prioritizing high-error channels and preserving the model’s fine-tuning capacity by maintaining a high-rank adapter structure—as illustrated in Figure 4. This is a clear advancement over simplistic magnitude-based or random parameter selection strategies. Complementing this, the subsequent Refinement step (Eq. 9) significantly reduces initial layer output error, as we can see from Table 2 and Figure 5, providing a far stronger starting point for fine-tuning than conventional initialization approaches.\n\n- **Strong and Consistent Empirical Performance**: The method’s performance advantages over existing baselines prove to be robust and reliable, as Table 3, Table 4, and Figure 6 consistently show. Across multiple models (LLaMA-3.1-8B, LLaMA-3.2-3B, Mistral-7B-v0.3) and different bit-widths, QWHA demonstrates clear and consistent superiority. Most impressively, in the highly challenging 2-bit quantization setting, as seen in Table 3, the method achieves a 2-4% absolute improvement on CSQA and GSM8k tasks compared to the next-best baseline—highlighting its exceptional effectiveness in extreme compression scenarios. Figure 6 further reveals that QWHA outperforms CLoQ even at its peak performance, despite using a smaller parameter budget (r=32), which underscores that the proposed adapter’s strength stems from superior representational capacity rather than mere efficient parameter allocation.", "weaknesses": "- **Insufficient Hyperparameter Sensitivity Analysis**: From Section 4, Table 8, Table 9, and Equation (7), it’s clear the study relies heavily on several critical hyperparameters without exploring their sensitivity—a notable oversight for a method emphasizing practicality. Notably, the AdaAlloc temperature `t` is fixed at 1 with no ablation to show how adjusting it might alter the parameter allocation strategy (e.g., making it more or less focused on high-error channels). Even more concerning, the adapter scaling factor `α` is set to 4000 for QWHA versus just 1 for CLoQ (per Table 8)—a massive discrepancy that goes entirely unjustified. This raises questions about whether the performance gains stem from the method itself or simply favorable hyperparameter tuning. Additionally, the learning rates in Table 9 appear to be heavily tailored to each experimental setting, which casts doubt on how easily QWHA can be applied to new models or tasks without extensive retuning.\n\n- **Limited Scope of Experimental Evaluation**: The experimental evaluation, as outlined in Section 4’s “Models and Datasets” section, is overly narrow, which undermines the claim of QWHA’s generalizability. Currently, all results are based on instruction tuning with the Alpaca dataset, followed by evaluation on just two tasks: commonsense QA (CSQA) and arithmetic reasoning (GSM8k). While these are relevant benchmarks, they do not capture the full range of use cases for quantized LLMs. It would greatly strengthen the paper to include results on more diverse tasks—such as text summarization (e.g., XSum), code generation (e.g., HumanEval), or long-context understanding—to demonstrate that QWHA’s benefits extend beyond reasoning-focused tasks. Without this, it’s hard to assess whether the method is broadly applicable or limited to specific task types.\n\n- **Lack of Statistical Rigor in Results**: Looking at Tables 3, 4, 10, and 11, it appears all reported accuracy scores are from single runs—an issue that contradicts standard practices in top-tier ML conferences. Fine-tuning LLMs is known to exhibit significant variability based on random seeds, and without reporting mean values and standard deviations across multiple runs, it’s impossible to determine if the reported performance gains are statistically significant or merely due to chance. This lack of rigor makes it difficult to trust the consistency of QWHA’s results, especially in the closely contested low-bit settings where small differences can tip the balance between methods.\n\n- **Inconsistent or Unfair Baseline Comparisons**: From Table 3 and Section 2.3, it’s apparent that the primary performance comparison is somewhat inconsistent. QWHA, which benefits from a dedicated quantization-aware initialization scheme, is pitted against other Fourier-based adapters (LoCA, SSH) that lack this critical component. While the paper acknowledges this mismatch, it still frames this as the main comparison—creating a potentially unfair advantage for QWHA. The more apples-to-apples comparison in Table 4 (where AdaAlloc is applied to all adapter types) paints a more nuanced picture: DCA + AdaAlloc and DHA + AdaAlloc often perform competitively with WHA, especially in the 2-bit LLaMA-3.2-3B GSM8k setting. This complicates the narrative that WHA’s transform kernel is the key differentiator, suggesting the initialization scheme may play a more dominant role than claimed.\n\n- **Weaknesses in Mathematical Formulation and Notation**: Several clarity issues in the mathematical presentation, visible in Algorithm 1, Appendix B.2, and Appendix C.1, hinder reproducibility and understanding. In Algorithm 1, the notation `(p_0, ..., p_d_out) ∈ N_d_out` is non-standard; using `(p_0, ..., p_{d_{out}-1}) ∈ ℕ^{d_{out}}` would make the dimensioning far clearer for readers. Additionally, the justification for WHA’s full-rank property in Appendix B.2 relies on theory for *random* sparse matrices [Coja-Oghlan et al., 2020], but AdaAlloc is a deterministic parameter allocation scheme—creating a tenuous link between the theoretical support and the actual method. While the derivation in Appendix C.1 is mathematically correct, it could be more self-contained by explicitly defining `U` and `Σ` as components of the SVD of `XXᵀ`, rather than assuming readers will infer this.\n\n- **Limited Ablation on Core Method Components**: The paper’s ablation studies, while present, fail to address key questions about its core design choices—something evident from Figure 5, Table 7, and Table 13. For example, the Refinement step’s impact is only measured on *initialization error* (per Figure 5 and Table 7), not on final *task accuracy*. An ablation showing how “WHA + AdaAlloc” performs without Refinement would be critical to understanding whether this step adds meaningful value beyond reducing early-stage error. Similarly, Table 13 compares training times for 1D versus 2D transforms but provides no corresponding accuracy data. Since the use of a single transform is a central design choice, empirical proof that a 2D transform offers no accuracy benefit (not just speed drawbacks) is necessary to justify this decision.\n\n- **Generalization to Other Quantization Schemes**: Despite claiming QWHA is compatible with any quantization scheme, the paper only presents results using GPTQ [Frantar et al., 2023], as seen in Section 4’s “Baselines” section. This gap between claim and evidence weakens the argument for QWHA’s versatility. Quantization methods vary significantly in how they handle outliers, weight distribution, and compression logic—for example, AWQ [Lin et al., 2024] uses activation-aware weighting, while RTN relies on simple rounding. Demonstrating QWHA’s effectiveness with at least one other popular scheme would go a long way toward proving its generalizability, rather than limiting it to a single quantization pipeline.\n\n- **Clarity on Parameter Budget (`r`)**: From Sections 3 and 4, it’s apparent that the paper uses the LoRA-specific term “rank (`r`)” to define the parameter budget for *all* adapters (e.g., “P(r=64)”)—a choice that creates unnecessary confusion. For Fourier-based adapters like WHA, `r` does not have a direct “rank” interpretation, as their structure differs fundamentally from LoRA’s low-rank matrices. Using a more generic term (e.g., “budget size” or “trainable parameter count”) and explicitly reporting the total number of trainable parameters for each method would greatly improve clarity, especially for readers comparing QWHA to non-LoRA baselines.\n\n- **Inference Overhead Analysis is Missing**: Unfortunately, there is no discussion of inference overhead anywhere in the manuscript—an odd omission for a paper that emphasizes efficiency. While the authors likely intend to merge the adapter weights `ΔW` into the frozen quantized weights `W_Q` for zero runtime overhead, the one-time cost of computing `ΔW = F H⁻¹` (including the inverse WHT and sparse-dense matrix multiplication) is never addressed. Additionally, the memory footprint of the sparse `F` matrix compared to LoRA’s compact `A` and `B` matrices is left unmentioned. To complete the efficiency narrative, a brief analysis of these inference-related costs would be essential for practitioners deciding whether to adopt QWHA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DBLBRMnxCX", "reviewer": "Reviewer_oADz", "summary": "The paper introduces QWHA, a quantization-aware PEFT framework that integrates a Walsh–Hadamard Transform (WHT)-based adapter (WHA) with a quantization-aware initialization scheme. Unlike existing LoRA-based QA-PEFT methods that rely on low-rank updates or Fourier-transform (DCT/DHT) adapters, the proposed WHA uses a single real-valued WHT kernel with +-1 entries, reducing computation while maintaining high representational capacity. The initialization combines channel-wise parameter allocation (AdaAlloc) and value refinement to minimize layer output error caused by quantization. Experiments on LLaMA and Mistral models across 2–4 bit quantization demonstrate improved accuracy and faster training compared to LoRA and other FT-based baselines.", "questions": "* How sensitive is the allocation to the temperature $t$ and to very small budgets (e.g., $P \\left( r \\right) \\leq 8$))? Any observed failure modes where some channels starve despite the per‑channel budgeting?\n * This is related to one of the weaknesses: can the proposed quantization-aware initialization generalize to other transform bases (e.g., random orthogonal, wavelet) or to rotation-based quantization schemes (e.g., AWQ, QuIP#)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Addresses a well-defined and practically relevant gap, integrating high-capacity transform adapters into quantized fine-tuning.\n * The WHT choice is analytically and empirically grounded, showing improved energy concentration and better outlier reconstruction compared to sinusoidal transforms.\n * The decomposition into adaptive allocation and refinement provides a tractable solution to an otherwise NP-hard sparse approximation problem.\n * Evaluations across multiple model families and quantization levels are comprehensive, with consistent gains in accuracy and efficiency.\n * Single-transform WHT substantially lowers compute cost, supported by observational runtime and memory measurements.", "weaknesses": "* Fine-tuning is limited to Alpaca (instruction-following) and general QA benchmarks; transferability to larger or domain-specific datasets remains unclear.\n * Only GPTQ+MagR quantization is used; robustness to other quantization backends is not shown.\n * While the paper reports overall training runtimes, SVD-based Hessian calibration step may still be costly for larger models and may need a breakdown for the initialization pipeline and its scaling with model size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)