LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuning
Authors: Junyu Chen
Junzhuo Li
Zhen Peng
Wenjie Wang
Yuxiang Ren
Long Shi
Xuming Hu
Venue: NeurIPS 2025 poster ()
Keywords: Quantization
Fine-Tuning
PEFT
Efficient LLMs
Primary area: deep_learning
TL;DR: LoTA-QAF uses ternary adaptation for fine-tuning quantized LLMs, enabling the lossless merging of adaptation into quantized weights.

Abstract: Quantization and fine-tuning are crucial for deploying large language models (LLMs) on resource-constrained edge devices. However, fine-tuning quantized models presents significant challenges, primarily stemming from: First, the mismatch in data types between the low-precision quantized weights (e.g., 4-bit) and the high-precision adaptation weights (e.g., 16-bit). This mismatch limits the computational efficiency advantage offered by quantized weights during inference. Second, potential accuracy degradation when merging these high-precision adaptation weights into the low-precision quantized weights, as the adaptation weights often necessitate approximation or truncation. Third, as far as we know, no existing methods support the lossless merging of adaptation while adjusting all quantized weights. To address these challenges, we introduce lossless ternary adaptation for quantization-aware fine-tuning (LoTA-QAF). This is a novel fine-tuning method specifically designed for quantized LLMs, enabling the lossless merging of ternary adaptation weights into quantized weights and the adjustment of all quantized weights. LoTA-QAF operates through a combination of: i) A custom-designed ternary adaptation (TA) that aligns ternary weights with the quantization grid and uses these ternary weights to adjust quantized weights. ii) A TA-based mechanism that enables the lossless merging of adaptation weights. iii) Ternary signed gradient descent (t-SignSGD) for updating the TA weights. We apply LoTA-QAF to Llama-3.1/3.3 and Qwen-2.5 model families and validate its effectiveness on several downstream tasks. On the MMLU benchmark, our method effectively recovers performance for quantized models, surpassing 16-bit LoRA by up to 5.14\%. For task-specific fine-tuning, 16-bit LoRA achieves superior results, but LoTA-QAF still outperforms other methods. Code is available in github.com/KingdalfGoodman/LoTA-QAF.

LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine- Fine-Tuning
Junyu Chen 12..3 Junzhuo Li Zhen Peng', Wenjie Wang Yuxiang Ren², Long Shi Xuming Hu
Southwestern University of Finance and Economics 2 Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province
3 The Hong Kong University of Science and Technology (Guangzhou)
Sun Yat-sen University Naajjng University
223811200390mma11.sss edu. cn shilongdswufe.edi cn
xuminghuGhkust--ge edu.
Abstract
Quantization and fine-tuning are crucial for deploying large language models (LLMs) on rexource-constrained edge devices. However, fine-tuning quantized models presents significant challenges, primarily stemming from: First, the mis- match in data types between the low-precision quantized weights (e.g., 4-bit) and the high-precision adaptation weights (e.g., 16-bit). This mismatch limits the computational efficiency advantage offered by quantized weights during infer- ence. Second, potential accuracy degradation when merging these adaptation weights into the low- cow-precision quantized weights, as the adaptation weights often necessitate approximation no existing methods support the lossless merging of adaptation while adjusting all quantized weights. To address these challenges, introduce lossless ternary adaptation for quantization-awarr fine-tuning (LoTA-QAF). This is a novel fine- tuning method specificallyy designed for quantized LLMs, enabling the lossless merging of ternary adaptation weights into quantized weights and the adjustment of all quantized weights. LoTA-QAF operates through a combination of: A custom-designed ternary adaptation (TA) that aligns ternary weights with the quan- tization grid and uses these ternary weights adjust quantized weights. ii) A TA-based mechanism that enables the lossless merging of adaptation weights. iii) Ternary signed gradient descent (t-SignSGD) for updating the TA weights. We apply LoTA-QAF to Ilama-3. 1/3.3 and Qwen-2.5 model families and validate effectiveness on several downstream tasks. On the MMLU benchmark, our method effectively recovers performance for quantized models, surpassing 16-bit LoRA by up to 5. 1.%. For task-specific fine-tuning, 16-bit LoRA achieves supe- rior results, but LoTA-QAF still outperforms other methods. Code is available in gglnubccom/Kngggaaccooaaaa.../.
truncation. Third, far
know,
1 Introduction
Large language models (LLMs) have showcased exceptional proficiency in natural language pro- cessing, driving advancements in applications such as complex reasoning (Hadi et al. 2023), code generation (Jiang et al.) 2024). and conversational AI systems (McTear and Ashurkina, 2024]. How- ever, their immense computational costs present significant challenges, particularly when deploying
* Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

GPTO (64R6 1640
GPTIOLORA (4484 36400
GPTO GHAR 16:00
(4-r
LOTA-OAY (4-60
ONLORA (H-8)
LOTA-GAP (43-4)
60
LOTA-GW
60.0gg $2.30
2045
50
$0.30
40
70
$4.00
30
60.37
22
Liama-88
Quen-14B
Qwen-22B
Llama--08
Llama-38
Qwen-14B
Qwen-328 Llama-708
Llama-88 8B
Qwen-14B
Quen-32B
Liama-706
4-bit Base Weights
B-bet Weights
2-bitt Base Weights
Figure 1: Comparison of shot MMLU accuracy (%), fine-tuned on the Alpaca dataset. GPTQ+LoRA (4-bit & 16-bit) uses 4-bit base and 16-bit adapter weights. QA-LoRA and LoTA-QAF operate 4-bit after merging adapters into quantized weights. Details in Table
these models on resource-constrained edge devices (Wan et al., 2023). To overcome this hurdle, quantization techniques have emerged critical approach model compression, effectively re- ducing memory demands and computational complexity by quantizing model weights to lower bit precision. Furthermore, deploying on edge devices not only necessitates model compression but also requires the models to handle specialized tasks, which require the integration of domain-specific knowledge. Consequently, there growing attention on fine-tuning LLMs to align with specific tasks under quantization constraints. One prominent approach designed for this scenario is QLoRA (Dettmers et al. 2023), which integrates low-rank adaptation with quantization. It quantizes the pretrained weights to 4-bit precision, significantly reducing the static memory footprint. Subsequently, 16-bit precision LoRA adapters are trained on top of these quantized weights, achieving efficient and low-cost fine-tuning.
Nevertheless, directly applying adapters fine-tuning quantized models introduces notable challenges that can hinder efficiency and performance. The first challenge arises during inference, where the interaction between low-precision weights (e.g., 4-bit) and high-pree prlciseeccinnn adapter weights (c.g.. 16- bit) can impair computational efficiency. This mismatch introduces computational overhead that can diminish the inference speedups from quantization (Jeon et al. 2024). The second challenge occurs when merging nigh-precision LoRA adapters into the www-precision quantized weights. During this process, the adapters must inevitably be quantized or truncated to the lower bit width (Bondarenko) et al. [024]] Guo et al.) 2024).. This reintroduction of quantization error at the adapter level, despite the adapters' aim to correct model quantization errors, leads to fine-tuning accuracy degradation. The third challenge is that existing lossless adapter merging schemes (Xu et al.) 2023) only allow for adjustment quantization parameters, rather than direct modification of the quantized weights. This significantly limits the adapter's capacity for effective fine-tuning.
These challenges highlight the development of fnn-ttnnig methods integrated within a quantization framework, namely quantization-awai aware fine-tuning (QAF). In this paper, we present LoTA-QA -AF,, lossless ternary adaptation fine-tuning method for quantized pretrained language models. Specifically, LoTA-QAF operates through combination of three key components: i) A custom- destined ternary adaptation (TA) that includes trainable ternary adapters, which form an auxiliary matrix adjust the quantized weights within the target quantization grid, instead of merely adjusting quantization parameters. ii) A TA-based lossless merging mechanism that utilizes the auxiliary matrix to generate ternary matrix and an offset matrix, subsequently updating the quantized weights and zero factors. This mechanism preserves the low-bit computational efficiency and avoids fine-tuning accuracy degradation. iii) Ternary signed gradient descent (t- (t-SignSGD) is employed to update the trainable ternary adapters during fine-tuning.
We apply LoTA- COFF to the Llama-3. 3/3.3 and Qwen-2.5 model families, validating its effectiveness performance recovery and task task-specific alignment for quantized models. Figure[iiluue iillustraaes the MMLU benchmark results, demonstrating that LoTA-QQA -QAF consistently outperforms other methods, especially at 2-bit. should be noted that LoRA employs 16-bit adapters (inference with 4-bit model weights and 16-bit adapters). which cannot be losslessly merged into quantized weights. In contrast, both LoTA-QAF and QA-LoRA can achieve lossless merging (inference with only 4-bit model weights). However, Qo-L only indirectly compensates for quantization errors by adjusting zero

---

factors via its adapter, lacking the capability to directly modify the quantized weights. Building on this, our main contributions include:
We present LoTA-QAF, which adjusts quantized weights within the quantization grid and facilitating the lossless merging of ternary adapters into the quantized weights. As a result, LoTA-QAF preserves low-bit computational efficiency and avoids accuracy degradation.
We introduce ternary signed gradient descent (t-SignSGD), novel optimizer for ternary adapters. It leverages sign-based updates with dynamic percentile-based thresholding to selectively adjust ternary adapter weights, thereby effectively optimizing these highly constrained parameters.
LoTA-QAF demonstrates effectiveness in two key quantization-awar fine-tuning scenarios, achieving strong performance while maintaining the inference efficiency of quantized models. For performance recovery, LoTA--AS shows up to 5. 5.14% improvement over LoRA on Qwen 2.5 14B 2-bit. In task-specific tuning, while trailing LoRA (16-bit adapters), LoTA-QAF still outperforms other methods. Regarding inference efficiency, LoTA-QAF is 1.7x-2.0x faster than LoRA after adapters are merged into quantized weights.
2
Related Work
Quantization of LLMs. Quantization techniques fundamentally reduce memory requirements and improve computational efficiency by lowering the bit precision of model weights. However, the precision loss inherent quantization inevitably leads to decline in the performance of quantized models (Kumar et al. 2024). Consequently, recovering the performance of quantized models is a crucial research topic the field. Post-training quantization (PTQ) offers a fast compression solution without requiring retraining. One approach, GPTQ (Frantar et al., (022). employs approximate second-order information during the quantization process, compensating for the quantization errors progressively. In addition, other approaches aim to address outliers, such as AWQ (Lin et al. 2024). which identifies and protects salient weights by scaling channels based on activation statistics. QuaRot (Ashkboos et al., (024)) uses randomized Hadamard transformations to remove outliers from the hidden state. Nevertheless, PTQ cannot adapt models to downstream tasks because it is applied after training.
Quantization-awar training (QAT), which integrates quantization simulation during training (Chen) et al.] [222|] Liu et al. 2023). not only helps performance recovery but also aligns quantized models with specific tasks. LLM-QAT (Liu et al., 2023) introduces QAT for LLMs using a data- free distillation approach to preserve the original model's output distribution. Efficient-QAT (Chen et al. 2024) proposes a two-stage strategy involving block-wise training of parameters and end-to-end training of quantization parameters. HALO (Ashkboos et al., (025) introduces Hadamard rotations during both forward and backward passes to mitigate outliers in low-precision training. However, the computational cost of QAT remains a significant challenge.
Quantizatinn-Aware Fine- TinnnT of LLMs. To maintain the low computational cost of quantiza- tion, while achieving performance recovery and alignment with specific tasks through fine-tuning, Bondarenko et al. (2024); Dettmers et al. (2023); Xu et al. (2023) explore quantization-aware fine- tuning (QAF) of large language models (LLMs)) Specifically. ly, the pivotal work, QLoRA (Dettmers) et al. 2023). employs fine-tuning of A adapters over frozen quantized model. LofQQ (Li
ctt al. 2023b finds an optimized low-rank initialization for adapters. RoLoRA (Huang et al. 2024) incorporates rotation mechanisms during LoRA fine-tuning over quantized weights to mitigate outlier. LR-QAT [Bondarenko et al. 2024) trains adapters within the QAT framework, making the adapters quantization-awarre LoRA (Xu et al. 2023) focuses enabling lossless merge of the trained adapter into the quantized weights. It achieves this by structuring the adapter to align with group-wise quantization parameters, allowing the adapter's influence to be absorbed into the zero factors.
Based on our literature review, the closest work to ours is QA-LoRA, which achieves lossless merging of adapters. However, QA-LoRA only adjusts the zero factors in group-wis quantization using the adapter, which restricts its fine-tuning capability. In contrast, our LoTA-QAF allows fine-tuning of all quantized weights within the quantization grid while maintaining the lossless merge property.
3

---

QLoRA
QA-LORA
LoTA-QAF
Fine-tuning
Quantied Weigtt
(inbi
Quarnzed Weight
(LAMO
Quaiiced Waigtt (4hir)
Temury Weights W(-1, I)
B (semary
W
A nht)
(10.3)
(vemaryy
Efficiencc Lawe after Mergy
Inference
lobix
Lauiess
Levelia
Merge
Mergee
Accurary after Mergo
Impact to Zero Factors
Impact Quantized Weights and Zero Factors
Figure 2: Comparison of prior adaptation methods with LoTA-QAE. A merge can result in efficiency loss, where model weights are de-quantized to preserve the adapter's precision but sacrifice inference efficiency, or accuracy loss, where the adapter's precision is sacrificed to maintain low-bit inference. A lossless merge avoids this trade-off because adapter weights are utilized identically during both the training and inference (after merge), ensuring the adapter's learned effect fully preserved.
3 Method
3.1 Preliminaries
Low-Rank Adaptation. The pre-trained weight matrix W Daax is frozen during the fine- tuning process, where Din and Dour represent the input and output dimensions, respectively. To enable efficient fine-tuning, trainable adapters A E Rrxx and B E grrx are introduced, where the rank r < min (Dm Dou)). These adapters are multiplied to form a low-rank matrix AB, which fine-tuning model with significantly fewer trainable parameters. The modified forward pass is computed
W + a _AB)
where A initialized with random Gaussian distribution, while B is initialized to zero. Here, a is constant scaling factor that adjusts the magnitude of the low-rank update.
Asymmetric Affine Quantization. We perform N-bit affine quantization on the weight matrix W, the quantization process maps W to an integer matrix Win- The dequantized approximation Wa as follows:
W
Wg s sW +
+ 2,
(2)
5
where the scaling factor (max W) min( - 1) and the zero factor min( (W) The symbol 1-1 denotes rounding to the nearest integer. All elements in the integer matrix Win int belong to the set I.. 1}.
2
3.2 Lossless Ternary Adaptation
When fine-tuning quantized models using low-rank adaptation to compensate for quantization loss or adapt specific task, the key aspect is adjusting quantized weights Win- Moreover, the merge process should not quantize or truncate the adapter weights, which would reintroduce quantization emoor at the adapter level. Consequently, we design the ternary adaptation, which losslessly merges the ternary adaptation into the quantized weights. This adaptation utilizes trainable ternary adapters P. AT E (-1, 1) Dax: xr and By (-1,0, 1]rxPau, Consistent with low-rank adaptation principles. the rank < min( Din Dour). For initialization, apply Kaiming normal initialization (He et al.

---

Ternary Adapters
Auxiliary Matrix A AW
Offset Matrix W
0
-1
2
-
0
1
0
Br
1
1
Eq. (4)
I
4
Eq. (3)
Eq.
15
Eq. (5)
1:8
0
Offset Factor /4
Quantized Weights W
Ternary Matrix W
Boundary Value
Values Exceeding Threshold
Illustration of LoTA-QAE. The process is demonstrated using a small 4x4 matrix under 4-bit quantization, with adapter rank set to 3 and threshold to 1. Notably, quantized weights have boundary values (e.g.. and 15 for 4-bit, where 15 cannot incremented and 0 cannot be decremented), requiring overflow prevention.
Figure 3:
2015) to AT and then ternarize its elements using a threshold, calculated as 0.75 times the mean absolute value of the sampled weights (Li et al. 2016), while By is initialized to zeros.
Notably, the product of the ternary adapters Ar and By forms the auxiliary matrix AW By Its elements AW; ij are integers within the range [-r, r]. where i E {l...., Din } and j {l..... D
The auxiliary matrix A WW is mapped the ternary matrix W (11,00, 1) Da using threshold w E (0, r), as follows:
W., sign A W (4) Haw.
where sign(-) denotes the sign function, and A/W represents the indicator function, which eval- uates to 1 if (AW éy| > w and 0 otherwise. The resulting ternary matrix W has the same dimensions the quantized weight matrix Win- Consequently, W is used to adjust the quantized weights Wim. compensating for quantization errors or adapting the model specific tasks. Additionally, W can be losslessly merged into Win after fine-tuning, preserving the inference efficiency of the quantized weights. We provide further analysis of the threshold Section 4.3
Furthermore, an offset factor μ is introduced, which represents an average offset relative to A AW, and ultimately absorbed into the zero factor. The calculation of μ relies on the offset matrix W. This offset matrix W and the offset factor μ are then computed as:
W AW, ij
Σ Du Σ 2j=1
(4)
P Don
where offset factor be performed at different granularity, such per-tensor, per-group, or per-channel, depending on the specific quantization method employed.
Following the fine-tuning stage, the lossless ternary adaptation merge mechanism is defined as:
W Win. W
int
z' + sj
where the final weights W belong to the set {0, 1...... 2 1}. Equivalently, the forward pass of
the unmerged module is defined as y W int + TT) x. Therefore, the lossless ternary adaptation
5

---

preserves low-bit computational efficiency and avoids the reintroduction of quantization loss at the adapter level.
3.3 Ternary Signed Gradient Descent
Inspired by the application of signed gradient descent (SignSGD) for quanizztiion-eelttd parameter updates in AutoRound (Cheng et al.) 2023), propose t-SignSGD for optimizing ternary adapters. Indeed, [Balles et al. (2020); Li et al. (2024 (023a): |Safaryan and Richtárík (202T) demonstrate that SignSGD is beneficially applied to updates within discrete and constrained domains. This characteristic is highly relevant to our ternary adapters, which are constrained to the discrete values {-1, 0, 1}. Specifically, our proposed t-SignSGD performs updates on the ternary adapters (e.g.. AT). Let 9r = V A, Cl denote the gradient at iteration t. To focus updates on prominent gradients, we employ fixed minimum gradient threshold 7 (e.g.. 1 x 10-9) 9) and dynamic percentile- based threshold dz.. The threshold σ; determined at each iteration t based on specific percentile of the gradient magnitudes, selecting certain proportion of the largest gradients for update. This proportion of selected gradients is initially the top 5% and then linearly decays to 0.01 % during training. The 1-SignSGD update is performed as follows:
(6) where the indicator function I(,) equals if the condition (19a) > max(T, is satisfied, and O otherwise. The clip(., -,, 1) function ensures that the updated adapter weights A,, are strictly constrained ternary set {-1,0, 0, 1]. Notably, Eq. (6) does not use learning rate to scale the sign of the gradient. Instead, if ternary adapter weight Ar is selected for an update (i.c., its gradient magnitude Ig. exceeds the threshold max(r, o,)). its value is adjusted by -iign(g)). Therefore, the threshold or play crucial role in determining the selectivity of updates. By targeting only the top percentile gradient magnitudes (e.g.. the top 5%), a acts as an adaptive mechanism that focuses updates on the most salient ternary adapter weights at each iteration t.
AT,111 clip (Ar,e sign(gt) [ -1,1),
The convergence of t-SignSGD is primarily governed by its update rule, which is designed for the discrete and bounded nature of the ternary adapters. The key component is the indicator function adaptive and dynamic selection mechanism. It ensures
(l..)=ax((r.o)) in Eq. (6), which acts stable convergence in two ways: i) Stability via noise filtering (preventing oscillation). In discrete optimization, small, noisy gradients can cause parameters to oscillate unstably between states (e.g.. flipping between O and 1). Our thresholding mechanism filters out these low-magnitude gradients, ensuring that updates are only performed when the gradient signal strong and reliable. This addresses the core challenge in sign-based methods related to low signal-to-nois ratios, effectively preventing unstable "jitter" and promoting a smoother convergence path. ii) Convergence via annealing- like dynamics. The decaying percentile threshold introduces an annealing like, coarse- to-fine search strategy. Early in training (o, is high), the top percentile of gradients triggers updates. This focuses the optimization on the most impactful parameters, allowing the model to quickly perform "broad-stroke" adjustments and find promising region in the vast solution space. Later in training (a, is low), allowing for more fine-grained adjustments. This enables the model to refine its solution within the promising region, analogous to the exploitation phase in classic search algorithms. This dynamic balance between exploration and exploitation is well-known strategy for improving convergence in complex landscapes. Further analysis of parameters and convergence is provided in Section
4 Experiments
4.1
Experimental setup
Models and Quantization. We conduct experiments on several large language models: Llama 3. 8B, Qwen 2.5 14B, Qwen 2.5 32B, and Llama 3.3 70B. GPTQ (Frantar et al. (022) asymmetric quantization is applied to all these models, with group size of 64 for Llama 3.1 8B and Qwen 2.5 14B, and 128 for Qwen 2.5 32B and Llama 3.3 70B. For calibration, we use 1024 samples from the C4 dataset (Rfffll et al. 2019).
Tasks. We focus on quantization-aware fine-tuning (QAF) and implement two fine-tuning paradigms: performance-rrcovery and task-specific. For performance-recowe fine-tuning,

---

Table 1: Accuracy (%) of performance- and task-specific. An em dash (-) indicates unob- tainable results, as the evaluation script requires strict format alignment with fine-tuning data.
MMLU (5-shot)
Task-Specific (0-shot)
Method
#Bit
Hums.
STEM
Social
Other
Avg.
GSMSK
SQL
ViGGO
LLBMA-70B
16
80.30
75.26
88.94
84.58
82,23
GPTQ
80.09
75.04
88.79
84.39
81.81
GPTO+LORA
4+16
80.19
74.98
88.79
84.39
81.83
90.14
82.90
89.72
QA-LoRA
80.11
75.01
88.89
84.29
81.81
84.32
77.65
85.35
LoTA-QAF
4
80.12
75.07
88.82
84.36
81.84
87.04
78.50
86.46
GPTQ
3
77.62
71.30
86.38
82.17
79.13
GPTO+LORA
3+16
77.58
71.27
86.48
82.23
79.14
88.17
80.20
88.09
QA-LORA
3
77.66
71.27
86.45
82.07
79.13
82.65
75,80
76.38
LOTA-QAF
77.68
71.30
86.42
82.36
79.20
83.24
77.40
85.38
GPTQ
2
39.00
35.08
46.83
46.64
41.53
GPTQ+LORA
2+16
52.58
45.86
65.91
62.63
56.22
67.54
72.70
79.35
QA-LoRA
50.03
44.08
61.68
58.83
53.20
61.80
70.70
68. 68.44
LOTA-QAF
2
52.96
47.10
67.34
62.83
56.98
63.98
73.40
75.38
Qwen-32B
16
77.95
82,43
88.72
84.96
82.92
GPTQ
4
77.90
82.11
88.56
83.71
82.47
GPTQ+LORA
4+16
78.13
82.46
88.59
83.52
82.59
83.09
88.70
77.29
QA-LoRA
77.98
82.02
88.63
83.68
82.48
76,82
86.80
70.93
LOTA-QAF
4
78.07
82.33
88.76
84.10
82.70
78.74
87.60
76.18
GPTQ
3
75.66
79.45
87.36
81.17
80.29
GPTO+LORA
3+16
75.75
79.42
87.46
81.53
80.42
82.34
88,90
74.70
QA-LORA
3
75.58
79.42
87.33
81.43
80.31
75,44
80.60
57.20
LoTA-QAF
76.00
79.51
87.59
81.43
80.53
76.04
83,20
60.94
GPTQ
2
34.03
32.54
'37.60
39.46
35.68
GPTQ+LORA
2+16
44.06
43.04
54.92
54.23
48.46
57.55
82.40
69.99
QA-LoRA
39.34
42.02
53.04
54.49
46.30
54.74
80.40
55.44
LOTA-QAF
2
47.31
45.42
60.87
59.93
52.65
57.92
82.20
63.71
Qwen-14B
16
74.71
77.86
87.46
82.39
79.91
GPTQ
4
74.79
76.91
87.26
81.88
79.57
GPTO+LORA
4+16
74.79
77.04
87.42
81.94
79.65
80.06
87.70
74.05
QA-LoRA
74.90
76.78
87.36
81.82
79.58
76.10
83,90
68.47
LOTA-QAF
4
74.86
77.20
87.52
82.14
79.77
78.37
84.50
71.10
GPTQ
3
71.07
72.41
84.24
79.79
76.19
GPTO+LORA
3+16
71.48
73.20
84.37
80.11
76.60
72.18
87.40
71.93
QA-LORA
3
71.16
72.79
84.21
79.88
76.31
66.49
77.30
57.36
LoTA-QAF
71.54
72.79
84.76
80.17
76.63
70.36
79.50
64.45
GPTQ
30.01
32.03
33.57
32.15
31.72
GPTQ+LORA
2+16
39.17
41.29
48.13
43.90
42.66
37.23
80.60
61.59
QA-LORA
33.79
36.60
42.18
49.79
39.80
34.69
58.10
43.87
LoTA-QAF
2
45.23
42.85
55.44
49.15
47,80
36.25
62.80
52.63
LLAMA-8B
16
64.17
60.39
77.64
73.09
68.25
GPTQ
62.19
58.71
76.37
72.93
66.89
GPTO+LORA
4+16
62.57
58.33
76.73
73.22
67.08
70.20
76.70
88.64
QA-LORA
4
62,06
58.71
76.54
73.25
66.96
68.69
74.10
51.15
LOTA-QAF
62.95
58.55
76.70
72.93
67.18
70.05
74.60
60.30
GPTQ
3
32,48
36,66
39.42
49.08
38.61
GPTQ+LORA
3+16
50.24
46.91
63.24
60.80
54.68
65.66
75.80
82.20
QA-LORA
3
45.55
46.94
61.33
59.35
52.37
62.17
72.20
43.86
LOTA-QAF
3
49.71
49.29
65.29
62.63
55.89
63.53
73.10
55.96
"GPTO"
2
25.89
227.53
'26.71
26.30
226.53"
GPTO+LORA
2+16
27.21
28.64
26.91
26.68
27.35
34.12
72.50
80.70
QA-LORA
25.70
26.67
27.72
26.62
26.56
17.36
56.20
37.18
LoTA-QAF
2
28.25
26.90
28.44
30.51
28.49
19.48
67.00
41.98

---

utilize the Alpaca TTori et al. 2023) and subsequently evaluate 5-shot performance on the Massively Multitask Language Understanding (MMLU) benchmark (Hendrycks et al. (020). For task sss-scccifiic fine-tuning, we select three datasets: GSM8K (Cobbe et al., 2021). with 7.47k training and 1 1.32k test samples; SQL generation ( Yu et al. 2018: Zhong et al. 2017). with 30k training and 1 test samples; ViGGO (Juraska et al., 2019). with 5.1k training and 1.08k test samples. We use the Im-eval framework (Gao et al. 2024) to test the MMLU benchmark and follow the custom evaluation framework from HALO (Ashkboos et al. 2025) for the task-specific evaluations. These datasets are chosen because they represent diverse specialized tasks that present unique challenges and require distinct model capabilities. Their inclusion allows to clearly illustrate the practical significance of the QAF approaches in developing quantized models for specific applications.
We benchmark LoTA-QAF against two state-of-the-a methods: LoRA (Hu et al. 2022) and QA-LoRA (Xu et al. 2023). For fine-tuning quantized models to recover performance and align them with specific tasks, LoRA remains widely adopted method (applied to
Baselines and
-urQ-qquattzee models, it can be considered QLoRA (Dettmers et al. 2023) variant). Although LoRA introduces 16-bit adapters, resulting in reduced inference efficiency, the 16-bit precision of its adapters also enables more precise adjustment of the quantized weights. Moreover, as the closest work to ours, QA-LoRA maintains low-bit inference efficiency by lossless merging the adapters into zero factors.
a paged AdamW optimizer, a maximum gradient norm of 0.3, a batch size of 64, a source length of 1024, and a target length of 256. For performance-rrcovery fine-tuning, we set the learning rate to 1 x 10 for the 8B and 14B models, and 10 for the 32B and 70B models. The number of fine-tuning steps is 300 for Alpaca. For task-specific fine-tuning, the learning rate is set to 10-4 for the 8B and 14B models, and 1 x 10 for the 32B and 70B models. Single-epoch experiments are performed on the training sets of GSM8k, SQL generation, and ViGGO. Regarding adapter settings, the rank is 64 for the 8B and 14B models, and 32 for the 32B and 70B models. Additionally, the coefficient a is twice the rank. Regarding the hyper-parameter of LoTA-QAF, we set the threshold w to 0.75r for Alpaca, GSM8K and SQL generation, and w to 0.875r for ViGGO. The dynamic percentile-based threshold, dr, is initialized to the top 5% and linearly decays to 0.1% during the first 80% of the training phase. For the final 20% of training (i.e., from 80% to 100% completion), it is fixed at 0.01% %. Section 4.3 provides detailed analysis of w and ot. All experiments are conducted on one NVIDIA A800 GPU.
Following QA-LoRA,
4.2 Main Results
Performance- Recivvry Fine- Fine-Tuning. For performance-recover fine-tuning, our objective is to restore the performance of quantized models to the level of the 16-bit base models. As Table [sooss, across four model sizes and three quantization bits, the three quantization-aware fine-tuning (QAF) methods all achieve improvements compared to the quantized baseline. For instance, with 3-bit quantization, the fine-tuned performance surpasses that of the quantized model by up 17.28% on Llama 3.1 8B. Similarly, for 2-bit quantization, the fine-tuned performance exceeds that of the quantized model by up 16.97% on Qwen 2.5 32B. This indicates that in the low-bit regimes, there remains a performance potential that existing Post-Trainin Quantization (PTQ) methods may not fully exploit, but which can be harnessed through fine-tuning. This highlights the significant development potential of QAF methods.
Furthermore, LoTA-QAF outperforms LoRA (GPTQ+LORA in Table I and QA-LORA. On Qwen 2.5 14B 2-bit, LoTA-QAA improves performance by 5.14% compared to LoRA, the second- best per- forming method in this comparison. This superiority is attributed to LoTA-QAF directly adjusting the quantized weights and aligning with the quantization grid, which effectively recovers performance lost by quantization. In contrast, while LoRA utilizes a 16-bit adapter, it lacks direct quantization-aware mechanism. As noted earlier, QA-LoRA only adjusts the zero factors in group-wise quantization, which limits its fine-tuning capability.
Task-Specific Fine-Tuning Tuning. For task-specific fine-tuning, the objective is to enable quantized models to acquire the knowledge and patterns specific to particular tasks. This necessitates that the model learn fine-grained capabilities specific to the task, rather than merely recovering general abilities. As shown Table LOTA-QAF outperforms QA-LoRA but is surpassed by LoRA. LoRA's bit adapter possesses higher representational capacity, enabling it to more effectively capture
8

---

mn33
int4
Int2
70
67.55
68.69
70.05
69.52
67,17
67.39
68.39
67.78
70.05
67.70
68.92
Accuracy (%)
63:53
66
60
58.07
61.28
60.88
61:33
58.36
Accuracy
60.20
59.21
63.53
45
53.45
51.63
20
18.80
19:48
19.11
20:88
20.17
18.57
16.98
19:48
19.47
15:92
8.0%
6.5%
The w of Ternary Adapters
The o, t-SignSGD
(a) Hyper-parameter Analysis
Speedup (LoRA)
Memory (LoRA)
10
4bit+LoRA
4bR+LoTA
Speodup (LOTA)
Memory (LOTA))
5.6
Speedup Ratio
1.3x
2.8
3.0
Memory (GB)
5.0
4.8
6n
Los
1.7
0.3
1.5
1.5
1.0
4.1
0.00300
0.03
4-bit
3-bit
80
120
160
Efficiency Analysis
Convergence Analysis log scale)
Figure 4: Further analysis of LoTA-QAF
these complex task-specific details. This higher precision is particularly crucial for heavily quantized models (e.g., 2-bit models). However, it is important to note the difference in inference efficiency: LoTA-QAF and QA-LoRA perform low-bit inference once their adapters are merged, whereas inference with LoRA requires computation involving the 16-bit adapter (as detailed in Section 4.3
4.3 Further Analysis of LoTA-QAF
Hyper-parameter Analysis. We investigate the hyper-paramend par-meteraamsesseea w of the ternary adapter (TA) and a of ternary signed gradient descent (t-SignSGD) through parameter experiments conducted GSM8K with 4, 3 and 2 bit-widths of the Llama 3.1 8B model, with results shown in Fig. [4] First, determine the extent to which influences the quantized weights. This threshold w is analogous to the LoRA coefficient in Equation [ A smaller w allows values in the auxiliary matrix A W affect the quantized weights more readily, Specifically, the threshold w is hhpee--parre parameter related to the rank r. We set the rank to 64 for the 8B model and experiment with w values of {40, 44, 48, 52, 56, 60). We observe that setting w to 48 yields favorable performance across all three bit-widths. Notably. for 2-bit quantization, a relatively larger ws achieves better experimental results. This because, in 2-bit quantization, each quantized weight possesses only four possible values. Consequently, updating the quantized weights with larger w, which exerts more conservative influence, leads to improved outcomes.
TA employs threshold
detailed in Equations and
Second, the dynamic percentile-bane baased threshold a (where f the iteration step) acts as a learning rate in t-SignSGD, as explained in Equation 6 We conduct parameter experiments by selecting initial oe values corresponding to the top (9.5%, 8.0%, 6.5%, 5.0%, 3.5%, 2.0%} of gradient magnitudes. For the 8B model, initial σ; of 5.0% (representing the top 5.0% of gradients) consistently yields favorable results. Conversely, selecting smaller initial a, values leads to a noticeable decline in performance for both 3-bit and 2-bit quantization. This observation of task-specific fine-tuning, which necessitates more substantial adjustments to model parameters compared to recovery performance. This aligns with our general learning rate strategy for LoRA and QA-LoRA, where the learning rate for recovery performance is set lower than that for task-specific fine-tuning. Further hyper-parameter analysis is provided in the Appendix
be attributed the nature
Efficiency Analysis. We measure the inference speed and memory usage of LoRA (utilizing forward pass with 4-bit quantized weights and 16-bit adapters) and LoTA (where adapter losslessly merged into the quantized weights). These measurements are conducted on the ViGGO dataset using the Llama 3.1 8B model with 4, 3 and 2 bit-widths quantization. For 4-bit and 2-bit. quantized models, the Triton UuaantLinca kernel is used. For the 3-bit quantized model, the

---

TorchQuantLinea kernel is employed. To evaluate inference speed, we measure throughput (tokens per second) with batch sizes ranging from 128 and maximum inference length of 512 tokens. The speedup ratio, benchmarked against the slowest configuration (3-bit quantized weights with LoRA), is reported in Fig LoTA achieves speedups of 1.9x, 1.7x, and 2.0x over LoRA across these respective bit- bidtws, revealing the efficiency gains from merging the adapter post fine-tuning, In terms of memory usage, LoTA also exhibits a slight advantage.
Convergence Analysis. Regarding convergence, we evaluate LoRA and LoTA on Llama 3.1 8B for SQL generation with 4, and bit-widths quantization. LoRA achieves best convergence across all three bit-widths. Notably, at 2-bit, LoRA reaches convergence loss of 0.132, whereas LoTA achieves 0.375, showing LoRA's 16-bit adapter stability. However, LoTA, optimized with 1-SignSGD, also converges. For 4-bit and 3-bit quantization, its convergence loss differs from LoRA < 0.01. Considering that LoTA can be losslessly merged into the quantized weights, whereas LoRA cannot, this makes LoTA a highly competitive method, especially when deployment efficiency is priority.
5 Conclusion
Fine-tuning quantized LLMs for edge devices suffers from merging issues. LoTA-QAF enables in in-grid adjustment of all quantized weights and lossless merging of ternary adapters using novel ternary adaptation and ternary signed gradient descent (t-SignSGD), and preserves performance by avoiding merging-induces accuracy loss. The Appendix details ternary adapters implementation, further parameters analysis, training efficiency analysis, limitations, and future work.
6 Acknowledgments
The work of Long Shi was supported by the National Natural Science Foundation of China under Grant 62201475 and Sichuan Science and Technology Program under Grant 2024NSFSC1436. The work of Xuming Hu was supported by the National Natural Science Foundation of China (Grant No.62506318) Guangdong Provincial Department of Education Project (Grant No.2024KQNCX0225 CAAI-Ant Group Research Fund: Scientific Research Projects for the Higher-educationall Institutions (Grant No.2022312000 Education Bureau of Guangzhou Municipality; Guangzhou-HK HKUST(GZZ)
Joint Funding Program (Grant No. 20250033955))) Education Bureau of Guangzhou Municipality.
References
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213-100244 2024.
Saleh Ashkboos, Mahdi Nikdan, Soroush Tabesh, Roberto L Castro, Torsten Hoefler, and Dan Alistarh. Halo: aedlamard-assisteer lossless optimization for efficient www-preciiion Ilm training and fine-tuning. arXiv preprint 2xXv:2501.02625, 2025.
Lukas Balles, Fabian Pedregosa, and Nicolas Le Roux. The geometry of sign gradient descent. arXiv preprint arXiv:2002. 2020.
Yelysei Bondarenko, Riccardo Del Chiaro, and Markus Nagel. Low-rank quantization-awarc training for llms. arXiv preprint ar Xiv1:2406600638 2024.
Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. Efficientqat: Efficient quantization-aware training for large language models. arXiv preprint arXiv:2407.11062. 2024.
Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, Kaokao Lv, and Yi Liu. Opti- mize weight rounding via signed gradient descent for the quantization of 1mm.. arXiv preprint arXiv:2309.05516, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110 10.11468, 2021.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 111110008811115. 2023.
10

---

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.1732: 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac' h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. hntps://://eeen
org/recordd/12608600 12608660
Hang Guo, Yawei Li, Tao Dai, Shu-Tao Xia, and Luca Benini. Intlora: Integral low-rank adaptation of quantized diffusion models. arXiv preprint arXiv:2410.21759 2024.
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint wrXiv:2009.03300 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Xijie Huang, Zechun Liu, Shih- Yang Liu, and Kwang-Timg Cheng. Rolora: Fine-tuning rotated outlier-free 1lms for effective weight-activation quantization. arXiv preprint arXiv:2407008044 2024.
Hyesung Jeon, Yulhwa Kim, and Jae-joon Kim. LAq: Parameter efficient quantization-awarr fine- tuning on large language models. arXiv preprint wxXv:2442.04490,, 2024.
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. survey on large language models for code generation. arXiv preprint 0rxiv:2406.005/5 2024.
Juraj Juraska, Kevin K Bowden, and Marilyn Walker. Viggo: A video game corpus for data-to-text generation in open-domain conversation. arXiv preprint arXiv:1910.12129 2019.
Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Man- sheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. arXiv preprint 11.0330 2024.
Bingrui Li, Wei Huang, Andi Han, Zhanpeng Zhou, Taiji Suzuki, Jun Zhu, and Jianfei Chen. On the optimization and generalization of two-layer transformers with sign gradient descent. arXiv preprint arXiv:2410.04877 2024.
Fengfu Li, Bin Liu, Xiaoxing Wang, Bo Zhang, and Junchi Yan. Ternary weight networks. arXiv preprint (oo::0:055441 2016.
Xiuxian Li, Kuo- Yi Lin, Li, Yiguang Hong. and Jie Chen. On faster convergence of scaled sign gradient descent. IEEE Transactions Industrial Informatics, 20(2):1732-11 1741, 2023a.
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: sura-iinetuning-awwal quantization for large language models. ICLR 24 Oral, 2023b.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 87-100, 2024.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888, 2023.
11

---

Michael McTear and Marina Ashurkina. Transforming conversational AI: Exploring the power of large language models in interactive conversational agents. Springer Nature, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and PeterJ. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv: Learning.arXer Learning, 2019.
Mher Safaryan and Peter Richtárik. Stochastic sign descent methods: New algorithms and better theory. In International Conference on Machine Learning, pages 9224 9234. PMLR, 2021.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et Efficient large language models: survey. arXiv preprint arXiv:2312.03863, 2023.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-awe low-rank adaptation of large language models. arXiv preprint arXiv:2309.14777, 2023.
Tao Yu, Rui Zhang, Kai Yang. Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma. Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprini arXiv: 10Xiv:8809008800 2018.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103,, 2017.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly define this paper's main contributions and scope. These contributions, summarized at the end of the Introduction (Section [ are further detailed and supported in the Experiments (Section 4.2). All claims accurately reflect the presented results, and the paper's 's scope is clearly delineated.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, we discuss the limitations of our work. For instance, the convergence prop- erties of our proposed method are analyzed in the Experiments (Section 4.3). Furthermore, broader discussion the work's limitations provided in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximation only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not present new mathematical theorems, or formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our method LoTA- LoTA-QAF process is detailed in the Method (Section 3) and illustrated in Fig. 3 Full experimental settings are disclosed in the Experiments (Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
14

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Yes, provide open access to all data and code necessary to faithfully reproduce the main experimental results presented in this paper. The supplementary material contains our source code, all relevant datasets links, and comprehensive README file. The README provides clear, step-by-step instructions for environment setup, data preparation, and script execution.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
Sublic/guidess CooesSumissionooooii for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips /nips.cc/publll wacc/public/pae hwwtic//guies//ee httpse///ooeeeuusissiiooooooo/o
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Full experimental settings disclosed in the Experiments (Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
Our experimental results are based on multiple independent runs in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, sufficient details of the compute resources required reproduce our experiments are provided at the end of Sections 4.1 and4.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have thoroughly reviewed the NeurIPS Code of Ethics and confirm that all research presented in this paper strictly adheres to its guidelines, encompassing data privacy, responsible conduct, and transparent reporting of results and limitations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses positive societal impacts. For instance, in the Introduction (Section D. we highlight that our high-efficiency method reduces computational costs, contributing to environmental sustainability.
16

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work does not release high-risk data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, all existing assets used are publicly available, appropriately credited, and their licenses and terms use are respected. Citations are provided in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsssie thcode wwwccom/aaaoo//c datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not introduce release public datasets software packages.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This research does not involve crowdsourcing experiments studies with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This research does not involve new studies with human subjects that would require Institutional Review Board (IRB) approval.
Guidelines:
18

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were not used as an important, original, or standtadd component of the core research methods presented in this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy http:a////errrppeccc////ooorrrrr for what should or should not be described.
19

---

A Implementation of Ternary Adapters
The PyTorch framework does not natively support ternary or torch. int2 data formats. Further- more, the PyTorch deep learning training ecosystem primarily relies on Therefore, to seamlessly integrate into this established system, we simulate ternary data types by representing values of {,1,0,1} using bfloat16.
computations.
For the process of forming the ternary matrix W via the auxiliary matrix AW- BT (as described in Eq. (3) of the main text), we implement custom operations using Tri triton to enhance the execution efficiency on GPUs. Moreover, we update the quantized weights Win int which requires boundary checks Specifically, through kernel fusion, Tri combines the formation of W and the application of boundary checks (boolean masks) into a single, optimized GPU kernel. This fusion minimizes overhead and contributes to maintaining acceptable training costs for the LOTA-QAF
Notably, the implementation of ternary adapters faces challenges primarily due to framework limita- tions. Methodologically, LoTA-QAI employs ternary adapters, which offer significant advantages in storage and computational efficiency over 16-bit adapters. Consequently, further engineering for production-Ievel deployment is valuable and represents promising avenue for future work (see Section(F). Currently, the training costs are acceptable with an implementation based on Py PyTorch and accelerated by Triton. A detailed discussion on training efficiency is available in Section C
B Hyper-parameters of LOTA-QAF
To further analyze the hyperparameters of LoTA-QAF, we focus on two aspects: i) the Llama 3.1 8B model across various datasets, specifically (a) MMLU, (b) SQL., and (c) ViGGO (with GSM8K detailed in the main text); and ii) the Qwen 2.5 14B and 32B models on the GSM8K dataset, as shown in (d) and (e). For all analyses of w (defined in Eq. (3). we set og to 5%. For the analysis of Tee we set w to 0.75r (except 0.875r for ViGGO), where r is the rank. Other settings are detailed in Section
Overall, more detailed hyperparameter tuning can reveal accuracies (e.g., as shown in Fig. (b)) that exceed those reported in main text Table I This is because different quantization bit-widths, datasets, and model sizes often have their own more suitable parameter settings. The objective of our further hyperparameter analysis is to identify the characteristies of these settings and understand the LoTA-QAF properties they reflect.
For the Llama 3.1 8B model on various datasets: i) the w parameter of the ternary adapters (TA, defined in Eq. 33) primarily determines how intensity W adjusts the quantized weights. For bibit quantized models, a smaller w can potentially yield gains (c.g., Fig. (b)) int4), attributed to the robustness and adjustability of 4-bit models. However, for 3-bit and 2-bit quantization, a smaller w generally leads to poorer performance (c.g.. Fig. (b) and (c), int3 and int2). Indeed, for b- quantization, setting a higher w to limit the adjustment intensity of W on quantized weights conversely achieves higher accuracy (c.g., Fig. 5 (c), int2). ii) the o, parameter of t- S-SinSSD (defined in Eq. (6) which functions as the learning rate in t-SignSGI, selects for update ternary adapter weights with gradient magnitudes in the top x%% For both SQL and ViGGO datasets (Fig. (b) and (c)), employing a larger oe results in higher accuracy. This aligns with our LoRA training configurations. Specifically,
'Two main approaches exist for performing these boundary checks: one is to decode the boundaries from the quantized weights during each forward propagation. which adversely affects training efficiency. Consequently, we adopt an alternative approach: we first identify the boundaries and then store them using a boolean packing technique (def pack_boo) bboo__oo_____ telsorrenrrrrrreeee ()) for use in the forward propagation. While this method introduces some memory overhead, the cost is acceptable as cach boolean flag (1 bit) is packed.
BBoundary checks are not strictly necessary during the training phase for 4-bit or even 3-bit quantization, as the probability of encountering boundary conditions is relatively low in schemes that utilize 16 or distinct values, respectively, especially considering that the number of values at the boundaries is typically smaller than that of values near the center. However, for 2-bit quantization, which involves only 4 distinct values, boundary checks become crucial. Specifically, neglecting boundary checks during training and only applying them when loading or merging the adapter can lead to inconsistencies between training behavior and the operational results of the adapter for 2-bit quantized models. In our experiments, we enable boundary checks universally (including in training efficiency tests) to ensure consistency, even though they could be optionally disabled for bibit and 3-bit scenarios during training without significant adverse impact.
20

---

1954
66,06
66.71
67:18
66.84
66.91
66.89
66,96
90
67.18
06.98
Accuracy (N)
55.83
66.89
61,23
46.95
48,57
45.39
40.00
49.10
40
80
27.43
27.52
27.63
27:36
27.06
27:13
86
27.05
52
60
9.5%
8.0%
6.5%
5.0%
3.5%
2.0%
The of Ternary Adapters
The a of SignSSD
(0)
Accuracy (%)
75-
75,60
74.80
74.70
74,40
74,10
75.40
75,80
74,60
76,10
73,20
71.40
73.00
72:30
Accuracy
70
73,90
73.20
73.30
75,10
65.90
67.50
67.00
67.10
67.20
67.70
67.00
65
64.30
48
52
80
9.5%
8.0%
6.5%
5.0%
3.5%
2.0%
The of Ternary Adapters
The I-SignS00
(b) Liama 3.1 BB SQL.
60
60.74
61.71
63:22
61.52
in64
In33
int2
65,18
62.54
61.31
60.30
i1s4
knz
Aocuraay (K))
50-
51.15
64.39
56.76
55.96
Accurccy
50
60.50
67.40
46.28
43.03
40
41.98
40.53
37.03
27.79
30
32.95
36.12
20-
22.53
19:94
23,46
10.55
52
9.5%
8.0%
6.5%
8.0%
3.5%
2.0%
The ss of Ternary Adapters
The o, of SignSGD
90
1144
78.37
80,44
80.52
Accurasy (%)
80
70.75
70,36
62.82
60.25
54,59
Accurasy (%)
69.89
70,03
68,93
59.75
49,43
50.21
52:21
33.36
33.25
36,10
36.25
11.46
25.80
20.08
24,93
23.80
6.5%
5.0%
Teraayy Adapters
1SignSGD
(c) Owen 2.5 14B on GSMBK
90
80.38
78.74
78.03
78.74
Ascuracy (%)
78.01
76.04
71.72
80.00
Accuracy (%)
72.45
74.50
78.33
60
53.68
57.92
68.55
50.47
70
64.32
$8.50
60.19
63,65
57.32
45.63
57.94
57.92
56.48
51.48
22
24
28
30
9.5%
8.0%
8.5%
5.0%
3.5%
2.0%
The of Ternary Adapters
The t-SinSSD
Qwen 2.5 32B on GSMGK
Figure 5: Parameter analysis for Llama 3.1 8B on various datasets in (a) MMLU, (b) SQL and (c)
ViGGO. Parameter analysis for Qwen 2.5 14B and 32B the GSM8K dataset in (d) and (e).
spek-s fine-tuning requires a higher learning rate than performance-recoverr fine-tuning, as learning specific tasks demands greater model weight adjustments than performance recovery.
For the Qwen 2.5 14B and 32B models on GSM8K dataset: setting w to 0.9375r leads to significant decrease in accuracy for 4-bit and 3-bit quantization (e.g., Fig. 5 (d) and (e)). Conversely, for 2-bit quantized weights, this w (0.9375r) value achieves higher accuracy. This consistent with earlier findings. ïi) setting o, either 6.5% 5.0% is robust choice. An overly small og (c.g..
21

---

2.0%) results in poorer accuracy (e.g. Fig. (d) and (e) int4 and int2), limiting its ability to learn effectively. Furthermore, due the high cost of extensive hyperparameter testing for 70B models, we offer baseline configuration: setting w to 0.75r and o; to 5.0% generally yields good accuracy. For further improvement, we suggest trying a; = 2.0% for performance-recover, fine-tuning and ot = 6.5% for task-specific fine-tuning. Regarding the setting for 70B models, suggests that experimenting with lower w values may improve accuracy, which is attributed to the 4-bit quantized 70B model's robustness.
C Training Efficiency Analysis
45
62.3
Speedup (LoRA) Speedup (LoTA)
Memory (LoRA)
Memory (LoTA)
Time (min)
36.4
Memory (GB)
31.0
30.6
30
45
24.4
34.436.0
34.936.5
36.5
15
12.9 15.0
25.527.1
30
7.8 8.9
Alpaca
GSMBK
SQL
ViGGO
Figure 6: Comparative analysis of training efficiency for LoRA versus LOTA. The evaluation utilized Llama 3.1 8B model with 4-bit precision on four datasets. Metrics reported are total training execution time and peak memory usage. For parameter details, including batch size of 64, and other settings, please refer to the Section 4.1
Our simulated 'ternary' adapter utilizes 'bfloat16' for implementation due to the lack native support for the 'ternary' dtype within the PyTorch framework. Consequently, LoTA-QAR does not exhibit training efficiency advantage over LoRA. Specifically, LoTA requires 14.1% 14.1%-25.4 more training time compared to LoRA. This increased training time is primarily attributed to the implementation of Eq. and the process of aligning with the dequantization process of GPTQModel. Despite this higher training cost, LoTA'' ternary adapters can be losslessly merged into quantized weights, a capability that LoRA does not offer. In terms memory, LoTA incurs an additional 2.6% overhead compared to LoRA. This overhead derives from the boundary checks detailed in Section A though the overall cost remains modest. Section discusses feasible directions for further enhancing LoTA's training efficiency.
D Ternary Adapter Weight Changes under t-SignSGD Training
Table Analysis of ternary adapter weight changes.
Metric
Value
Percentage
Description
Total Parameters
167, 16,,772.160
100%
Total adapter parameters across 32 lay- ers. Adapters are applied to key pro- jections within the attention and MLP blocks of each layer.
Increased Parameters
28,952,566
17.26%
Count of weights that changed in the positive direction.
Decreased Parameters
28,960,070
17.26%
Count of weights that changed in the negative direction.
Unchanged Parameters
109,859,524
65.48%
Count weights that are unchanged.
To better understand the dynamics of our optimizer, we analyzed the parameter changes of the ternary adapters for Llama 3.1 8B after fine-tuning on the Alpaca dataset. Table shows that the
22

---

rate of change in the adapters is nearly identical in both positive and negative directions (17.26% vs. 17.27%). This suggests that the adaptation is not a simple unidirectional shift but balanced. bidirectional fine-tuning process to fit the target task. Moreover, one-third of the parameters changed during adaptation, while two-thirds remained unchanged. This phenomenon is a result of the t- SignSGD design, specifically the adaptive threshold ![ ( This mechanism ensures our updates are based on a large gradient magnitude, which implies a high probability that the sign is correct. Conversely, it filters out updates associated with small gradient signals, which can be noisy or ill-scaled.
E Limitations
Regarding training efficiency, LoTA-QAF keeps its training costs at a level comparable to LoRA through Triton, but LoTA-QAF has not fully realized the potential high efficiency inherent in ternary adapters. A common challenge in current low-bit research is bridging the gap between theoretical and realized efficiency. Although Triton and CUTLASS offer pathways to achieve efficient implementations, its implementation is challenging.
Regarding convergence, LoTA-QAF exhibits less favorable convergence compared to LoRA (dis- cussed in Section 4.3) This can be attributed to two main aspects: firstly, ternary adapters adjust the quantized weights based on the quantization grid. For 4-bit and 3-bit quantization, which involve 16 and 8 possible values respectively, such adjustments demonstrate reasonable robustness. However, under 2-bit quantization with only four possible values, these adjustments exhibit higher volatility. Secondly, t-DignSGD is a novel update mechanism that operates without a learning rate, and we do not incorporate first or second-order momentum. Moreover, the primary constraint is linear decay of the threshold of (as defined in Eq. (6) While the t-SignSGID approach effective for updating ternary adapters, our exploration of its optimization remains limited.
F Future Work
We believe that the current training efficiency disadvantage of the ternary adapter (TA) can be ad- dressed. Although TA involves more computational logic during forward propagation, the operational advantages of ternary arithmetic can be further exploited. The auxiliary matrix AW BT is calculated using bfloat1,, but executing operations on ternary values (-1, 0, 1 could further improve efficiency. The forward execution logic Eq. ( is merged with the dequantization process of GPTQModel. However, given that W is a sparse ternary matrix, the potential for dedicated sparse computation logic warrants exploration. Such implementations, however, must consider not only computational efficiency but also ensure the correct functioning of the gradient mechanism for the ternary data type within the PyTorch framework.
Our exploration of t-SignSGD has been limited. Further research could focus on incorporating momentum mechanisms, refining constraints on t-SignSGD, and applying optimization techniques (e.g., cosine annealing, cyclical learning rates, or adaptive step-size strategies). In t-SignSGDD, we replaced the learning rate with dynamic percentile-based threshold, oz, to select the top-2% of gradient signs for directly updating the ternary adapter weights within the set {-1, 0, 1). Under this mechanism, more detailed analysis of gradient distribution characteristics could be pursued to guide the selection of ternary adapter weights for updates, potentially offering improved convergence properties.
Further enhancements to LoTA-QAF could also explored by expanding the representational range of the auxiliary matrix W. Currently, this matrix is constrained to the set (-1,0, 13. However, the LoTA mechanism, as outlined in Eq. (3). could be extended to broaden the value range of W to include, for instance, {-2, 11, 1, 2). even wider set of integers. Such an extension could further enhance the optimization capability for quantized weights. It is important to note, though, that this approach might be less suitable for int2 or even int3 quantization formats due the potentially large adjustment step sizes. Nevertheless, for int4 or int8 quantization, exploring this direction is highly valuable.
23


Reviews:
[{"review_id": "qIWs8kpiWx", "reviewer": "Reviewer_HKvF", "summary": "To address (1) the mismatch between low-precision quantized weights and high-precision adaptation weights that constrains the inference efficiency of quantized weights, and (2) potential performance degradation from approximating high-precision adaptation weights during merging, the paper presents lossless ternary adaptation for quantization-aware fine-tuning (LoTA-QAF), which utilizes ternary adaptation weights and ternary signed gradient descent (t-SignSGD) for updating them. The paper shows the effectiveness of LoTA-QAF for various models (Llama, Qwen) and benchmarks (MMLU, GSM8K).", "questions": "Please see Strengths and Weaknesses", "limitations": "Please see Strengths and Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nIt seems that the authors effectively address the issue mentioned in Summary, which is novel.\n\n(1) They confirm that  LoTA-QAF can be about twice faster than LoRA after adapters are merged into quantized weights.\n\n(2) LoTA-QAT consistently outperforms GPTQ+LoRA on MMLU in Table 1.\n\n(3) The paper is well written.\n\nWeaknesses\n\n(1) One thing I am most concerned about is that in Table 1, LoTA-QAT performs better than GPTQ+LoRA on MMLU but worse than GPTQ+LoRA on GSM8K, which is likely to imply that LoTA-QAT is prone to overfitting or beneficial for some limited tasks only.\n\n(2) Given that the authors use Alpaca, which is instruction-following data, it would be more appropriate to evaluate on GSM8K with few-shot, not zero-shot.", "was_revised": true, "final_justification": "I maintain my positive score, 4.", "rebuttal": "We thank the reviewer for recognizing the novelty and effectiveness of our method. We will address your insightful questions to further clarify the technical details.\n\n> **Q1**: One thing I am most concerned about is that in Table 1, LoTA-QAT performs better than GPTQ+LoRA on MMLU but worse than GPTQ+LoRA on GSM8K, which is likely to imply that LoTA-QAT is prone to overfitting or beneficial for some limited tasks only.\n\n> **A1**: This performance difference is not an indication of overfitting or limited task applicability, but rather a direct reflection of the distinct objectives of the two fine-tuning paradigms.\n> 1) **Performance-recovery fine-tuning (MMLU):** On broad benchmarks like MMLU, the primary goal is to restore the model's general, pre-existing knowledge degraded during quantization. LoTA-QAF excels in this area as it is specifically designed for quantized models, directly adjusting weights on the quantization grid and influencing quantization parameters. This performance-recovery scenario requires broad-based adjustments to the model's parameters, rather than the highly specialized updates needed for learning new patterns. Consequently,  LoTA-QAF excels in performance-recovery fine-tuning.\n2) **Task-specific fine-tuning (GSM8K):** Conversely, a task like GSM8K requires the model to specialize in a new, complex, and fine-grained domain (i.e., mathematical reasoning, we use 7.47k GSM8K training data). In this task-specific scenario, the higher precision and representational capacity of a 16-bit LoRA adapter provide a natural advantage in capturing the intricate patterns of this specialized domain. \nHowever, the key advantage of LoTA-QAF is that its adapters can be losslessly merged, preserving both the final accuracy and the full computational efficiency of the low-bit model for deployment. The superior performance of GPTQ+LoRA on GSM8K comes at the cost of inference inefficiency, as the 16-bit adapter cannot be losslessly merged.\n\n---\n\n> **Q2**: Given that the authors use Alpaca, which is instruction-following data, it would be more appropriate to evaluate on GSM8K with few-shot, not zero-shot.\n\n> **A2**: We'd like to clarify that the Alpaca dataset is used for performance-recovery fine-tuning (on MMLU), not for the GSM8K task. \nFor task-specific fine-tuning on GSM8K, we use its training set of 7.47k samples and test with 1.32k samples, as detailed in Section 4.1 (\"Tasks\"). Therefore, we chose a zero-shot setting to purely evaluate the reasoning capabilities acquired during this task-specific fine-tuning phase.\nAdditionally, we provide the 5-shot results on GSM8K for the Llama 3.1 8B and Qwen 2.5 14B: \n>\n> | Model | Mothed | 4-bit | 3-bit | 2-bit | \n>|:---|:---|:---:|:---:|:---:|\n>| **8B (shot-5)** | lora (x-bit +16-bit)  | 74.53 | 68.55 | 37.72 | \n>|  | lota (x-bit)  | 73.61 | 66.94 | 23.63 | \n>| **14B (shot-5)** | lora (x-bit +16-bit)  | 85.21 | 82.25 | 57.85 | \n>|  | lota (x-bit)  | 84.03 | 80.38 | 55.82 | \n>| **8B (shot-0)** | lora (x-bit +16-bit)  | 70.20 | 65.66 | 34.12 |\n>|  | lota (x-bit)  | 70.05 | 63.53 | 19.48 |\n>| **14B (shot-0)** | lora (x-bit +16-bit)  | 80.06 | 72.18 | 37.23 |\n>|  | lota (x-bit)  | 78.37 | 70.36 | 36.25 |\nOn GSM8K, using a 5-shot setting significantly improves performance, particularly for the 14B 2-bit model. This demonstrates that through LoRA/LoTA fine-tuning, the quantized models have effectively learned task-specific patterns via the adapters. When augmented with 5-shot examples, their performance is further boosted."}, {"review_id": "XzSJhourLk", "reviewer": "Reviewer_N8rD", "summary": "The authors introduce LoTA-QAF, a PEFT method that directly tackles the challenge of merging adapters into low-bit base weights. They initialize the ternary adapter AB, and apply a threshold to their multiplication to make sure that it is also ternary. The ternary nature of the adapter allows for a lossless merging with the base INT weights through modifying both the quantized values and the zero point. They further introduce t-SignSGD, an optimizer tailored for ternary parameters. Extensive experiments show the effectiveness of the method.", "questions": "1. It’s not clear to me what the forward pass of the unmerged module actually is. Is it $(W_q + \\frac{\\alpha}{r} A_T B_T)^T x$, similar to LoRA? Or are you directly defining it as $(s \\cdot W’_{int} + z’)^T x$ with no $\\alpha$? I think it helps the reader to define it explicitly, and discuss the differences from LoRA forward pass.\n2. Can you please elaborate on the definition of the residual matrix? From the term \"residual\", my intuition says it should be $\\tilde{W}=\\Delta W - \\hat{W}$ as opposed to $\\tilde{W}=\\Delta W - \\omega \\hat{W}$. Can you please elaborate on your design and the intuition/reasoning behind it?\n3. In Figure 3, why are the values that are 3 and -3 not included in $\\hat{W}$? Is that because the corresponding values in $W_{int}$ are on the boundary? So the indices that are on the boundary are always zero in $\\hat{W}$? I think this can confuse the reader if not explained explicitly.\n4. LoTA-QAF is tailored specifically for INT base weights. Given the recent shift of the literature toward non-uniform quantized formats such as MXFP4, do you think your method could be extended to support those? A discussion on this would strengthen the paper.\n5. In my opinion t-SignSGD is an interesting new optimizer. However, I don’t think it is studied extensively in the paper. For example, how do the parameters change during training, e.g., what percentage of them change in both directions during training? Can we say anything theoretical about its convergence (probably the rebuttal period is too short for that though)?\n6. In the context of task-specific experiments, while LoTA-QAF shows strong results, it is still consistently and significantly outperformed by LoRA. I think that is fine, but it raises a question that is not answered in the paper: is there “any” 2/3/4 bit solution that recovers LoRA? In other words, can LoTA-QAF be improved, or is there a fundamental constraint that a standalone 2/3/4-bit weight simply cannot recover LoRA? I can think of two ways to approach this: (1) less interesting: merge the LoRA adapter into the weights in high precision, and then apply 4-bit SpinQuant/FlatQuant/Quarot [1,2,3]  and see if it improves upon LoTA-QAF, and (2) more interesting: run a weight-only version of HALO [4] to directly fine-tune a low-precision model (the activations should be kept in high precision). I think having these results, especially the second one, could significantly strengthen your paper.\n\nIn general, I think LoTA-QAF is a novel and strong method that has potential impact on the community. However, I think the writing on parts of the paper could be improved, and some more ablations/experiments (especially on t-SignSGD) are missing. Depending on how the rebuttal goes, I am open to increasing my score.\n\n[1] https://arxiv.org/pdf/2405.16406\n[2] https://arxiv.org/pdf/2410.09426\n[3] https://arxiv.org/pdf/2404.00456\n[4] https://arxiv.org/pdf/2501.02625", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea is interesting and it is likely that the community will use this method.\n2. The experiments are extensive and include speedup results.\n\nWeaknesses (see Questions for more details):\n1. To me, the method section was unclear, specifically Section 3.2 and Figure 3.\n2. More discussion/ablations on t-SignSGD would strengthen the paper.\n3. No discussion on non-uniform quantization formats.", "was_revised": true, "final_justification": "The writing on some parts could be improved and some experiments are not fully convincing. Generally though, I think this is a solid paper with potential impact on the community. Hence I raise my score to \"borderline accept.\"", "rebuttal": "We thank the reviewer for recognizing the novelty of our idea and its potential value to the community. We will address your insightful questions to further clarify the technical details.\n\n> **Q1**: It’s not clear to me what the forward pass of the unmerged module actually is. Is it $(W_q + \\frac{\\alpha}{r} A_T B_T)^T x$, similar to LoRA? Or are you directly defining it as $(s \\cdot W’_{int} + z’)^T x$ with no $\\alpha$? ...\n\n> **A1**: Our method follows the same logic as lossless merge, $(s \\cdot W’_{int} + z’)$. In contrast to LoRA, we do not use an $\\alpha$ to scale the adaptation. Instead, the impact of the adaptation is controlled by a threshold $\\omega$. As we briefly mentioned in Section 4.3, this threshold $\\omega$ functions similarly to $\\alpha$. It determines how the auxiliary matrix $\\Delta \\mathbf{W} = \\mathbf{A}_T \\mathbf{B}_T$ contributes to the ternary matrix $\\mathbf{\\hat{W}}$. A smaller $\\omega$ allows the adapter to have a greater impact on the quantized weights, while a larger $\\omega$ has a more conservative effect.\n \n---\n\n> **Q2**: Can you please elaborate on the definition of the residual matrix? ...\n\n> **A2**: The auxiliary matrix $\\Delta W$ is utilized to generate a ternary matrix $\\mathbf{\\hat{W}} \\in \\\\{-1, 0, 1\\\\}$ and a offset factor. This is controlled by a threshold $\\omega$. The rule is as follows: if the signal's magnitude $|\\Delta W_{ij}|$ is greater than $\\omega$, we set $\\mathbf{\\hat{W}}_{ij} = sign(\\Delta \\mathbf{W}\\_{ij})$. The residual, which represents the signal that remains, is then calculated as: $\\mathbf{\\tilde{W}}\\_{ij} = \\Delta \\mathbf{W}\\_{ij} - \\omega \\mathbf{\\hat{W}}\\_{ij}$. \n\n---\n\n> **Q3**: In Figure 3, why are the values that are 3 and -3 not included in $\\hat{W}$? ...\n\n> **A3**: We perform a boundary check to prevent overflow or underflow of the quantized weights, but this does not mean that updates in $\\hat{W}$ for weights on the boundary are always zero. For example, a weight of `15` can still be updated by `-1`, just not by `+1`. In Figure 3, the potential updates from `3` and `-3` in $\\Delta W$ are blocked because their corresponding weights in $W_{int}$ are at the boundaries (`15` and `0`), and the updates would have caused an overflow.\n \n---\n\n> **Q4**: Given the recent shift of the literature toward non-uniform quantized formats such as MXFP4, ...\n\n> **A4**: LoTA-QAF can be extended to support non-uniform quantization formats such as MXFP4 or NF4, though it would require a more sophisticated design for the threshold $\\omega$.\n> \n> The core idea is that both uniform and non-uniform formats map high-precision values to a discrete set of representation points (e.g., 16 unique values for a 4-bit format). For non-uniform formats, instead of an arithmetic `+/- 1` adjustment, we can define the adjustment as moving to an adjacent representation level (e.g., `0010` -> `0011`).\n> \n> The main challenge is that the step sizes between adjacent levels are not uniform. The value change from `0010` -> `0011` differs from `0011` -> `0100`. Consequently, a fixed adjustment threshold $\\omega$ is no longer optimal. The threshold would need to be adaptive, depending on the specific step size of a potential change. This is an interesting direction, and we thank you for raising it.\n\n \n---\n\n> **Q5**: In my opinion t-SignSGD is an interesting new optimizer. However, I don’t think it is studied extensively in the paper. ...\n\n> **A5**: We sincerely thanks for your positive feedback on t-SignSGD and your insightful questions regarding its convergence properties. Our proposed t-SignSGD can be understood as a gradient-guided discrete search algorithm whose design includes several mechanisms to ensure convergence.\n> \n> We will elaborate on this from three perspectives: 1) The Challenge of Formal Convergence Analysis, 2) Theoretical Grounding in SignSGD, and 3) Analysis of the t-SignSGD Update Rule for Convergence.\n>\n> *Please refer to Qp8Z-A2 for the full details on these three perspectives, given the space limitations here.*\n\n\n> **Experiment on Ternary Adapter Weight Changes under t-SignSGD Training**\n> \n> To better understand the dynamics of our optimizer, we analyzed the parameter changes of the ternary adapters for Llama 3.1 8B after fine-tuning on the Alpaca dataset. The results are as follows:\n> \n> | Metric | Value | Percentage | Description |\n> |---|---|---|---|\n> | Total Parameters | 167,772,160 | 100% | Total adapter parameters across 32 layers. Adapters are applied to 7 key projections within the attention and MLP blocks of each layer. |\n> | Increased Parameters | 28,952,566 | 17.26% | Count of weights that changed in the positive direction. |\n> | Decreased Parameters | 28,960,070 | 17.26% | Count of weights that changed in the negative direction . |\n> | Unchanged Parameters | 109,859,524 | 65.48% | Count of weights that are unchanged. |\n>\n> From this experiment, we can learn that the rate of change in the adapters is nearly identical in both positive and negative directions (17.26% vs. 17.27%). This suggests that the adaptation is not a simple unidirectional shift but a balanced, bidirectional fine-tuning process to fit the target task.\n>\n> One-third of the parameters changed during adaptation, while two-thirds remained unchanged. This phenomenon is a result of the t-SignSGD design, specifically the adaptive threshold $\\mathbb{I}_{|g_t| > \\max(\\tau, \\sigma_t)}$. This mechanism ensures our updates are based on a large gradient magnitude, which implies a high probability that the sign is correct. Conversely, it filters out updates associated with small gradient signals, which can be noisy or ill-scaled.\n---\n\n> **Q6**: In the context of task-specific experiments, while LoTA-QAF shows strong results, it is still consistently and significantly outperformed by LoRA. I think that is fine, but it raises a question that is not answered in the paper: ...\n\n> **A6**: To build intuition, let's first consider the process of model quantization, separate from fine-tuning. One can visualize quantization as squeezing the air out of a plastic bottle to fill it completely with water. A more thoroughly trained model contains more \"water\" (knowledge). When squeezing out the \"air\" (parameter redundancy), it becomes easier to accidentally squeeze out some \"water\" as well, causing performance degradation.\n>\n> If we only consider adjusting the quantized weights (and not the quantization parameters), it is possible to match or closely approximate LoRA's performance. This is observable on tasks where the model is already proficient or on simpler tasks (e.g., Performance-Recovery Fine-Tuning or even GSM8K). Furthermore, if the model size is large enough, like 70B, the task-specific performance will also be closer to LoRA's, because the model has a sufficient number of parameters to adjust. To extend the analogy, if fine-tuning is like reshaping the plastic bottle into a specific form, it can be done perfectly if the bottle is very soft (a large model) or the target shape is simple (an easy task).\n>\n> When considering methods like LoTA-QAF, which adjust both quantized weights and quantization parameters, the question becomes even more complex, as there are more ways to \"reshape the bottle.\" Therefore, it is difficult to definitively state whether \"any 2/3/4-bit solution can recover LoRA\" without first specifying the model, task, and method.\nHowever, we agree this is an important and interesting question. We conducted an experiment by applying Quarot to re-quantize the result of the GPTQ+LoRA method on the Llama 3.1 8B model.\n>\n> | Bit | Method | GSM8k | SQL | ViGGO |\n> | :--- | :--- | :---: | :---: | :---: |\n> | **4-bit** | GPTQ+LoRA | 70.20 | 76.70 | 88.64 |\n> | | Quarot (GPTQ+LoRA) | 67.53 | 74.12 | 48.54 |\n> | | QA-LoRA | 68.69 | 74.10 | 51.15 |\n> | | LoTA | 70.05 | 74.60 | 60.30 |\n> | **3-bit** | GPTQ+LoRA | 65.66 | 75.80 | 82.20 |\n> | | Quarot (GPTQ+LoRA) | 60.52 | 71.64 | 41.89 |\n> | | QA-LoRA | 62.17 | 72.20 | 43.86 |\n> | | LoTA | 63.53 | 73.10 | 55.96 |\n> | **2-bit** | GPTQ+LoRA | 34.12 | 72.50 | 80.70 |\n> | | Quarot (GPTQ+LoRA) | 15.92 | 52.17 | 34.07 |\n> | | QA-LoRA | 17.36 | 56.20 | 37.18 |\n> | | LoTA | 19.48 | 67.00 | 41.98 |\n> \n> The results show that applying post-training quantization (PTQ) after fine-tuning is less effective than fine-tuning directly on quantized weights. This observation persists even when using an advanced PTQ method like Quarot, which is specifically designed to handle outliers. The performance degradation from re-quantization is particularly severe in the 2-bit setting. This highlights the necessity of developing QAF-type methods like LoTA-QAF.\n> \n> As for using HALO to test the performance ceiling in low-bit settings, this is an avenue worth exploring. However, given that HALO is not directly adapted for 2/3/4-bit training, and achieving optimal performance would require extensive experimentation, we did not conduct this experiment.\n---\nThank you again for your constructive review. We hope our responses have clarified your concerns and further demonstrated the potential of LoTA-QAF."}, {"review_id": "tQp0AVUVyr", "reviewer": "Reviewer_abLM", "summary": "This paper proposes LOTA-QAF, a quantized PEFT and loseless merging method for Ternary Quantization. Specifically, the proposed method performs lossless merging by limiting the truncation range of the ternary adapter and adding it to the weights, while the offset part is added to the quantization offset through summation. The proposed method showed comparable performance to QAlora and even better performance at 2bit.", "questions": "I recommend that the authors add a more detailed explanation of why this is a lossless merging technique. If I understand correctly, QALoRA achieves this by sacrificing granularity. However, this paper seems to argue that lossless merging is possible without any such cost.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength\n- The paper is well-written and easy to follow.\n- The proposed LOTA-QAF and t-signSGD are simple yet demonstrate high performance.\n- Experiments were also conducted on large-scale models, such as Llama-70B.\n\nWeakness\n- It is difficult to understand why the proposed method is considered lossless merging. Based on the provided figure (Fig. 3), it seems that when per-tensor quantization is performed, the process is not lossless due to the information loss that occurs when the offset weights are merged into a single scalar offset factor.\n- The proposed t-signSGD does not appear to be significantly different from the previously proposed SignSGD.\n- Limited evaluation: Evaluations on recent LLMs, such as Llama3, or on commonsense QA tasks were not conducted.", "was_revised": true, "final_justification": "My initial concern was that LOTA-QAF seemed to claim that lossless merging was possible at no cost. However, the authors have clarified that this is achieved by sacrificing accuracy through ternary operations. They also clearly distinguished between the proposed t-signSGD and signSGD. As most of my concerns have now been resolved, I will keep my score at Borderline Accept.", "rebuttal": "We thank the reviewer for recognizing the effectiveness of our method. We will address your insightful questions to further clarify the technical details.\n\n> **Q1**: *1.1* It is difficult to understand why the proposed method is considered lossless merging. Based on the provided figure (Fig. 3), it seems that when per-tensor quantization is performed, the process is not lossless due to the information loss that occurs when the offset weights are merged into a single scalar offset factor.\n>\n> *1.2* I recommend that the authors add a more detailed explanation of why this is a lossless merging technique. If I understand correctly, QALoRA achieves this by sacrificing granularity. However, this paper seems to argue that lossless merging is possible without any such cost.\n\n> **A1**: This question touches upon a crucial aspect of our method, and we appreciate the opportunity to clarify. In our work, the term \"lossless merge\" means that the model's forward pass is mathematically identical before and after the adapter is merged. This is in direct contrast to the \"lossy\" merge of a standard LoRA adapter into quantized weights.\n> 1) **Lossy Merge**: As shown in Figure 2, merging a 16-bit LoRA adapter into quantized weights forces a difficult trade-off: either you sacrifice the adapter's precision to maintain low-bit inference (an accuracy loss), or you maintain the adapter's precision by de-quantizing the weights, which sacrifices inference efficiency (an efficiency loss).\n> 2) **Lossless Merge**: LoTA-QAF directly adjusts weights on the quantization grid and influences quantization parameters via its offset factor. Regarding the offset factor μ, its calculation is performed identically during both the training phase and the inference phase (after the merge). Since the model's computational logic remains consistent, the merge operation introduces no loss to the adapter's learned information, ensuring the adapter's effect is preserved from training to inference.\n> 3) **Cost**: We also wish to clarify that our method is not without its own trade-offs. The primary cost of LoTA-QAF is the limited representational capacity of its ternary adapters. While our method sacrifices the high precision of a 16-bit LoRA, the ternary adapters can be losslessly merged while still directly modifying the integer weights. \n\n---\n\n> **Q2**: The proposed t-signSGD does not appear to be significantly different from the previously proposed SignSGD.\n\n> **A2**: We must emphasize that our t-SignSGD employs a fundamentally different learning process compared to classical SignSGD for the following reasons:\n> 1) **The update mechanism is different.** We do not use the standard learning_rate * sign(gradient) formula. Instead, t-SignSGD updates are determined by a dynamic percentile-based threshold $\\sigma_t$ in Eq. (6), which focuses updates on the most prominent gradients. If a ternary adapter weight is selected for an update because its gradient magnitude exceeds the threshold $\\sigma_t$, its value is adjusted directly by adding or subtracting 1 within the {-1, 0, 1} set. This threshold-based selection, rather than a learning rate, is the crucial factor determining the update's selectivity.\n> 2) **t-SignSGD is tightly integrated with the ternary adapter framework.** Such a large-step adjustment (+1 or -1) would be infeasible if applied directly to neural network parameters. Our approach is effective because it is specifically designed for the discrete and bounded nature of the ternary adapters. The updates from t-SignSGD do not directly modify the quantized weights. Instead, the updated ternary adapters are first multiplied to form an auxiliary matrix, which is then processed through Equations (3) and (4) to indirectly influence the quantized weights. \n\n---\n\n> **Q3**: Limited evaluation: Evaluations on recent LLMs, such as Llama3, or on commonsense QA tasks were not conducted.\n\n> **A3**: The models we evaluated—Llama 3.1 8B (July 2024), Qwen 2.5 14B (Jan 2025), Qwen 2.5 32B (Jan 2025), and Llama 3.1 70B (Dec 2024)—were selected because they were benchmark models when we began our experimental phase. These details are available in Section 4.1 (\"Models and Quantization\") of our paper. \n>\n> Furthermore, we have supplemented our work with new experiments on the CommonsenseQA. We evaluated the performance of Llama 3.1 8B and Qwen 2.5 14B on this task.\n> \n> | Model | Mothed | 4-bit | 3-bit | 2-bit |\n>|:---|:---|:---:|:---:|:---:|\n>| **8B** | lora (x-bit +16-bit)  | 77.55 | 64.52 | 19.24 |\n>|  | lota (x-bit)  | 78.62 | 65.76 | 21.04 |\n>| **14B** | lora (x-bit +16-bit)  |83.06 | 81.71 | 32.19 |\n>|  | lota (x-bit)  | 83.70 | 82.96 | 35.95 |\n>\n> These results on CommonsenseQA are consistent with our findings on MMLU (performance-recovery fine-tuning task). This is because the primary goal is to restore the model's general knowledge degraded during quantization. Our method, LoTA-QAF, excels here as it is specifically designed to directly adjust weights on the quantization grid, which is effective for recovering these broad, pre-existing capabilities."}, {"review_id": "uMnU4bqm2H", "reviewer": "Reviewer_Qp8Z", "summary": "This paper introduces a novel adaptation method for quantization-aware fine-tuning (QAF), enabling lossless merging of adaptation weights into quantized weights. The method includes three main components: (1) custom-designed ternary adaptation that adjusts quantized weights within the quantization grid, (2) a lossless merging mechanism using auxiliary and offset matrices, and (3) ternary signed gradient descent (t-SignSGD) for optimization.", "questions": "please see the weeknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n1. The paper introduces an innovative QAF method with elegantly designed ternary adaptation that enhances learning capability by enabling direct adjustment of quantized weights rather than merely modifying quantization parameters. \n2. The ternary constraint ensures that weight adjustments remain within the quantization grid, completely avoiding the additional quantization errors introduced during adapter merging.\n3. The design of the t-SignSGD optimizer with dynamic percentile-based thresholding is remarkably ingenious, utilizing minimal hyperparameters while effectively solving the discrete optimization problem.\n4. The LoTA-QAF works effectively across different quantization bit-widths (2-bit, 3-bit, 4-bit) without requiring method-specific modifications. This flexibility makes it a versatile solution that can adapt to various deployment constraints and hardware capabilities.\n\n[Weaknesses]\n1. Compared to LoRA, LoTA-QAF seems restrictive in learning complex task-specific patterns, particularly in low-bit settings. How significant is the representational capacity gap between ternary adapters and full-precision adapters? It’s necessary to provide some insights.\n2.  For t-SignSGD's convergence properties, the authors provide only relatively brief descriptions and lack more in-depth discussion or theoretical analysis, specifically regarding convergence behavior in discrete optimization spaces.\n3. The current baseline evaluation is constrained, with comparisons limited to QA-LoRA and LoRA, while other QAF methods remain unexamined.", "was_revised": true, "final_justification": "Thanks to the author for the Rebuttal. I have read it carefully, which further strengthens the contributions. I will keep my positive score.", "rebuttal": "We thank the reviewer for recognizing the innovation and practical advantages of LoTA-QAF. We will address your insightful questions to further clarify the technical details.\n\n> **Q1**: Compared to LoRA, LoTA-QAF seems restrictive in learning complex task-specific patterns, particularly in low-bit settings. How significant is the representational capacity gap between ternary adapters and full-precision adapters? It’s necessary to provide some insights.\n\n> **A1**: Let's further analyze the performance on task-specific fine-tuning. Across the four model sizes we evaluated (8B, 14B, 32B, and 70B), the performance gap between LoTA-QAF and 16-bit LoRA is smallest on the 70B model. However, the performance of LoTA on the 2-bit quantized 8B model is significantly lower than that of LoRA.\n> \n> This observation aligns with the design philosophy of LoTA-QAF. For large models like the 70B version, the representational capacity is sufficient for LoTA's coarse-grained +/−1 adjustments on the discrete quantization grid to effectively align the model. Furthermore, while LoTA-QAF always applies a ternary influence on the quantized weights, the effect of these adjustments is finer-grained on 4-bit models than on 2-bit ones. A 4-bit weight has 16 possible values whereas a 2-bit weight has only 4. This means each +/−1 adjustment on the 4-bit grid is relatively more fine-grained. This lower precision on the 2-bit grid limits the adapter's ability to make the subtle changes required for fine-tuning, which explains its lower performance.\n\n---\n\n> **Q2**: For t-SignSGD's convergence properties, the authors provide only relatively brief descriptions and lack more in-depth discussion or theoretical analysis, specifically regarding convergence behavior in discrete optimization spaces.\n\n> **A2**: We sincerely thanks for your positive feedback on t-SignSGD and your insightful questions regarding its convergence properties. Our proposed t-SignSGD can be understood as a gradient-guided discrete search algorithm whose design includes several mechanisms to ensure convergence.\n>\n> We will elaborate on this from three perspectives: 1) The Challenge of Formal Convergence Analysis, 2) Theoretical Grounding in SignSGD, and 3) Analysis of the t-SignSGD Update Rule for Convergence.\n>\n> **1. The Challenge of Formal Convergence Analysis**\n> \n> A formal convergence proof for our method is highly complex. Firstly, t-SignSGD operates on a discrete and bounded parameter space of {-1, 0, 1}. Updates are discrete jumps (e.g., from 0 to 1) rather than infinitesimal steps in a continuous space, rendering standard gradient-based convergence analyses inapplicable.\n> \n> Secondly, the forward pass from the ternary adapters to the final loss involves non-differentiable operations, notably the indicator function $\\mathbb{I}\\_{|\\Delta \\mathbf{W}\\_{ij}| > \\omega}$ used to form the ternary matrix $\\hat{W}$. To enable backpropagation, we necessarily employ the Straight-Through Estimator (STE). The resulting \"pseudo-gradient\" is an effective proxy but does not perfectly reflect the true gradient of the loss with respect to the adapter weights.\n> \n> Finally, even for standard SignSGD on continuous weights, theoretical analysis is a complex field, relying on specific geometric assumptions. Therefore, the following discussion focuses on providing a clear theoretical grounding and a mechanistic explanation.\n> \n> **2. Theoretical Grounding in SignSGD**\n> \n> The foundation of t-SignSGD lies in SignSGD. The core insight from SignSGD literature is that the sign of the gradient is often a more robust descent direction than its magnitude, which can be noisy or ill-scaled. The reliability of the sign is closely tied to the signal-to-noise ratio. A large gradient magnitude implies a high probability that the sign is correct.\n> \n> The t-SignSGD adapts this principle to drive state transitions within the discrete {-1, 0, 1} space. The optimization of ternary adapters is a combinatorial problem with an immense search space of $3\\^{D\\_{in×r}}$ possibilities. Since brute-force search is intractable, the STE-computed gradient acts as a high-quality heuristic, transforming this intractable search into a manageable, iterative optimization process by indicating the most promising update directions.\n> \n> **3. Analysis of the t-SignSGD Update Rule for Convergence**\n>\n> The convergence behavior of t-SignSGD is primarily governed by its update rule, which is designed for the discrete and bounded nature of the ternary adapters.\n> The key component is the indicator function $\\mathbb{I}_{|g_t| > \\max(\\tau, \\sigma_t)}$ in Eq. (6), which acts as an adaptive and dynamic selection mechanism. It ensures stable convergence in two ways:\n> \n> **a) Stability via Noise Filtering (Preventing Oscillation)**: In discrete optimization, small, noisy gradients can cause parameters to oscillate unstably between states (e.g., flipping between 0 and 1). Our thresholding mechanism filters out these low-magnitude gradients, ensuring that updates are only performed when the gradient signal is strong and reliable. This addresses the core challenge in sign-based methods related to low signal-to-noise ratios, effectively preventing unstable \"jitter\" and promoting a smoother convergence path.\n>\n> **b) Convergence via Annealing-like Dynamics**: The decaying percentile threshold $\\sigma_t$, introduces an annealing-like, coarse-to-fine search strategy. *Early in training ($\\sigma_t$ is high)*, only the top percentile of gradients triggers updates. This focuses the optimization on the most impactful parameters, allowing the model to quickly perform \"broad-stroke\" adjustments and find a promising region in the vast solution space. *Later in training ($\\sigma_t$ is low)*, the threshold decreases, allowing for more fine-grained adjustments. This enables the model to refine its solution within the promising region, analogous to the exploitation phase in classic search algorithms. This dynamic balance between exploration and exploitation is a well-known strategy for improving convergence in complex landscapes.\n>\n> In summary, while our current implementation of t-SignSGD is a concise first-order method, its design is deeply principled. It robustly guides a discrete search by leveraging an adaptive thresholding mechanism that ensures stability and promotes convergence through an effective annealing-like dynamic. \n\n---\n\n> **Q3**: The current baseline evaluation is constrained, with comparisons limited to QA-LoRA and LoRA, while other QAF methods remain unexamined.\n\n> **A3**: Let's consider another QAF method, IntLoRA. It is designed for efficient merging, as noted by its authors: “For inference, IntLoRA weights can be naturally merged into quantized pre-trained weights through efficient integer multiplication or bit-shifting, eliminating additional post-training quantization.” [1]\n>\n> In our view, IntLoRA employs an innovative mechanism to quantize the adapter more finely, which allows for a better merge within quantized weights. The test results on LLaMA 3.1 8B are as follows:\n>\n> | #Bit | Method | MMLU-Avg. | GSM8k | SQL | ViGGO |\n> | :--- | :--- | :---: | :---: | :---: | :---: |\n> | **4-bit** | QA-LoRA | 66.96 | 68.69 | 74.10 | 51.15 |\n> | | IntLoRA | 66.35 | 68.81 | 74.18 | 54.73 |\n> | | LoTA-QAF | 67.18 | 70.05 | 74.60 | 60.30 |\n> | **3-bit** | QA-LoRA | 52.37 | 62.17 | 72.20 | 43.86 |\n> | | IntLoRA | 51.09 | 61.60 | 69.43 | 40.22 |\n> | | LoTA-QAF | 55.89 | 63.53 | 73.10 | 55.96 |\n> | **2-bit** | QA-LoRA | 26.56 | 17.36 | 56.20 | 37.18 |\n> | | IntLoRA | 26.74 | 15.31 | 51.29 | 33.49 |\n> | | LoTA-QAF | 28.49 | 19.48 | 67.00 | 41.98 |\n>\n> At 4-bit precision, IntLoRA outperforms QA-LoRA but falls short of LoTA-QAF. However, at 2-bit and 3-bit precision, IntLoRA's performance is significantly worse. We hypothesize that this is because the IntLoRA method focuses solely on adjusting the quantized weights without modifying the quantization parameters.\n> \n> *[1] Hang Guo, Yawei Li, Tao Dai, Shu-Tao Xia, and Luca Benini. \"IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models.\" Forty-second International Conference on Machine Learning, 2025.*\n---"}]

Author rebuttal:

Meta-review: This paper proposes LoTA-QAF. The approach is novel, with strong results across multiple models and bit-widths, often outperforming QA-LoRA and GPTQ+LoRA. The introduction of t-SignSGD is also interesting and empirically supported. Some remaining concerns include the notion of “lossless” merging not being fully clarified in the original paper, task-specific results still lag behind full-precision LoRA, and parts of the method section (e.g., Figure 3) could be clearer. Overall, despite these issues, the work made sufficient contributions to the community, and I recommend acceptance.
Decision: Accept (poster)