MeMoTune: A Measure and Moment-Driven Fine-Tuning Framework for Quantized Large Language Models
Authors: Yun Zhang, Xue Geng, Lizi Liao, Jintong Sun, Minghe Yu, Ge Yu
Accepted at: acl25

MeMoTune: A Measure and Moment-Driven Fine-Tuning Framework for Quantized Large Language Models

Yun Zhang1,2, Xue Geng2, Lizi Liao3, Jintong Sun4, Minghe Yu4, Ge Yu1

1School of Computer Science and Engineering, Northeastern University, China
2Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore
3School of Computing and Information Systems, Singapore Management University, Singapore
4Software College, Northeastern University, China

Correspondence: geng_xue@i2r.a-star.edu.sg

Abstract

Quantizing large language models (LLMs) is essential for reducing memory and computational costs in natural language processing. Existing methods combine quantization with parameter-efficient fine-tuning but often fail to meet practical performance requirements. This paper introduces MeMoTune, a novel fine-tuning framework for quantized LLMs. By employing a measure and moment approach within a low-rank approximation framework in probability measure space, MeMoTune optimizes the objective function for superior fine-tuning results. The update process is further refined through scaled gradient, enhancing convergence efficiency and noise robustness. Experiments on tasks like text generation, summarization, and understanding show MeMoTune significantly outperforms state-of-the-art methods, e.g. fine-tuning Llama2-13B on GSM8K improves accuracy by 5.5%, while fine-tuning DeBERTaV3-base on CoLA of GLUE increases Matthews correlation by 1.7%. The code is publicly available at github.com/hddyyyb/MeMoTune.

1 Introduction

Large language models (LLMs) have transformed natural language processing (NLP), enabling significant advancements in various applications (Touvron et al., 2023a,b; AI@Meta, 2024; Lewis et al., 2019; Chung et al., 2024). Despite their success, the substantial memory and energy demands of LLMs limit their adoption, particularly on resource-constrained devices (Sze et al., 2017). Quantization has emerged as a crucial technique for reducing the computational and memory overhead of LLMs. By approximating high-precision weights with low-precision values, quantization effectively compresses models without retraining (Zhu et al., 2023; Frantar et al., 2022; Liu et al., 2023c; Xiao et al., 2023). For instance, 4-bit quantization methods (Banner et al., 2019; Sun et al., 2020) can reduce model size by up to 8× compared to full-precision (FP32) models. More recently, post-training quantization methods (Frantar et al., 2022; Yao et al., 2022; Liu et al., 2023c,a; Xiao et al., 2023) have demonstrated efficacy at 8-bit precision, as in SmoothQuant (Xiao et al., 2023), which enables lossless 8-bit weight and activation quantization. However, without retraining these methods often fail to maintain performance addressing tasks lower precisions than 8-bit (Wei et al., 2023; Lin et al., 2024).

Parameter-efficient fine-tuning (PEFT) has be-

<table>
  <tr>
    <th>MeMoTune</th>
    <th>LoftQ</th>
  </tr>
  <tr>
    <td>A. EVs are better for the environment.</td>
    <td>A. EVs are eco-friendly.</td>
  </tr>
  <tr>
    <td>B. The languages which use the Latin alphabet are English, Spanish, French, German, Italian, Romanian, Danish, Swedish, Danish, Norwegian, Finnish, Icelandic, Faroese, and Hungarian.</td>
    <td>B. The languages which use the Latin alphabet are English, Spanish, French, German, Italian, Romanian, Danish, Swedish, Danish, Norwegian, Finnish, Icelandic, Faroese, and Hungarian.</td>
  </tr>
  <tr>
    <td>C. Weekly rate of Eliza for the first 40 hours of work is $10 x 40 = $<span style="color:red;">400</span><br>Eliza's overtime pay is $10 x 1.2 = $<span style="color:red;">12</span><br>Her total earnings for the week are $400 + $12 = $412<br>The final answer is: 412<br>Ground Truth: 460.0</td>
    <td>C. Eliza's overtime pay is $10 x 1.2 = $<span style="color:red;">12</span><br>Her total earnings for the first 40 hours is $10 x 40 = $<span style="color:red;">400</span><br>Her total overtime pay is $12 x 5 = $<span style="color:red;">60</span><br>Therefore, Eliza's total earnings for this week is $400 + $60 = $<span style="color:red;">460</span><br>The final answer is: 460<br>Ground Truth: 460.0</td>
  </tr>
</table>

Figure 1: Comparison of MemoTune and LoftQ (Li et al., 2024) across the NLG task for Alpaca and GSM8K datasets. Our MeMoTune generates outputs that are: (A) more concise in summaries, (B) more comprehensive in enumerating answers to practical questions, and (C) more accurate in mathematical solutions.
come a key strategy for adapting full-precision LLMs to downstream tasks. By fine-tuning only a subset of parameters, PEFT achieves efficiency and performance gains (Hu et al., 2021; Lester et al., 2021; Li and Liang, 2021; Liu et al., 2023b). Huggingface’s PEFT library offer tools for fine-tuning LLMs with minimal overhead (Mangrulkar et al., 2022). Low-Rank Adaptation (LoRA) (Hu et al., 2021), a leading PEFT method, introduces low-rank adapters to reduce computational costs. However, (Chen et al., 2022) demonstrates that PEFT methods exhibit limited effectiveness on medium-volume (1k–10k training samples) and large-volume (10k+ training samples) datasets. Additionally, experiments (Yang et al., 2024c) indicate that increasing fine-tuning data does not consistently improve results for a given task.

Recent studies integrate PEFT with quantization to enhance model performance while reducing memory overhead. QLoRA (Dettmers et al., 2024) and IR-QLoRA (Qin et al., 2024) fine-tune LoRA parameters while keeping the pre-trained base model’s 4-bit parameters frozen. LoftQ (Li et al., 2024) leverages singular value decomposition (SVD) to initialize LoRA, improving performance and addressing quantization challenges. These methods inherit the limitations of full-precision PEFT, while quantization-induced degradation further impacts performance.

In this paper, we propose a novel framework, Measure and Moment-driven fine-Tuning (MeMoTune), for quantized LLMs. MeMoTune models trainable weights as a high-dimensional Gaussian distribution within the LoRA architecture and optimizes the objective function, with a theoretical analysis of this process, thereby improving performance and addressing its limitations on large datasets. Additionally, MeMoTune derives distribution parameter gradients while introducing a scale into the update process, enabling efficient distribution parameter updates, improving convergence, and enhancing noise resilience. We evaluate MeMoTune across a range of NLP tasks, including natural language generation (NLG), summarization, and natural language understanding (NLU). Experimental results demonstrate that MeMoTune outperforms state-of-the-art (SOTA) methods significantly. For example, fine-tuning Llama2-13B on GSM8K dataset improves accuracy by 5.5%, while fine-tuning DeBERTaV3-base on CoLA of GLUE improves Matthews correlation by 1.7%. These results highlight MeMoTune’s potential for deploying quantized LLMs in resource-constrained environments. Figure 1 showcases MeMoTune’s improvements over LoftQ (Li et al., 2024), demonstrating its strong reasoning capabilities. In summary, our contributions are threefold:

• We propose MeMoTune, leveraging the measure and moment approach to model trainable weights as a Gaussian distribution in LoRA, optimizing fine-tuning for quantized LLMs.

• We derive distribution parameter gradients while introducing a scale to enhance convergence efficiency and noise robustness.

• Experiments show that our method achieves SOTA results across NLP tasks with significant accuracy and efficiency gains.

2 Related work

2.1 Quantization

Quantization reduces the precision of model parameters and activations by storing them in low-bit formats, which reduces memory and computational needs. There are two main approaches to quantization: Quantization-Aware Training (QAT) (Shin et al., 2023; Zhu et al., 2023) and Post-Training Quantization (PTQ) (Liu et al., 2023a,c; Xiao et al., 2023). QAT incorporates quantization during training, allowing the model to adapt to quantization errors, often resulting in higher accuracy than PTQ. For instance, QAT-based method QFD (Zhu et al., 2023) trains a quantized representation as the teacher, and quantize the network using knowledge distillation. However, QAT requires retraining, which can be computationally expensive.

In contrast, PTQ quantizes pre-trained base models without additional training, making it more efficient. Methods like Gptq (Frantar et al., 2022), a one-shot weight quantization approach using approximate second-order information, and Zeroquant (Yao et al., 2022), which applies group-wise weight and token-wise activation quantization, effectively reduce quantization error and maintain hardware acceleration. However, PTQ may still result in significant performance degradation.

2.2 Parameter-Efficient Fine-Tuning (PEFT)

PEFT reduces computational costs by updating only a small subset of model parameters during fine-tuning. LoRA (Hu et al., 2021), a foundational PEFT technique, introduces low-rank matrices, which is called Adapter, to represent the
necessary changes from pre-trained to fine-tuned weights, avoiding full matrix updates. It enables efficient adaptation with minimal parameter updates. However, LoRA’s deterministic linear approximation limits robustness, making it sensitive to noisy data. Other methods, such as Prompt Tuning (Lester et al., 2021), Prefix-Tuning (Li and Liang, 2021), and P-Tuning (Liu et al., 2023b), fine-tune input prompts or attention-specific tokens, keeping core model parameters unchanged.

Recent work improves LoRA’s adaptability and efficiency. DoRA (Mao et al., 2024) prunes less impactful components, MELoRA (Ren et al., 2024) trains mini-LoRAs across dimensions, and PRoLoRA (Wang et al., 2024) shares parameter blocks within layers. Tunable mask filtering parameters have also been proposed to enhance fine-tuning performance (Nikdan et al., 2024; Xu and Zhang, 2024; Lu et al., 2024; Zhang et al., 2024). Laplace-LoRA (Yang et al., 2024a) estimates uncertainty via Laplace approximation without altering LoRA-trained weights, focusing on calibration rather than accuracy. However, these methods are not designed for quantized LLMs, limiting their effectiveness. Moreover, (Chen et al., 2022) shows their limitations in medium-volume and large-volume datasets, while (Yang et al., 2024c) finds that more fine-tuning data does not always enhance performance.

2.3 PEFT for Quantized LLMs

PEFT has been applied to improve the performance of quantized LLMs. QLoRA (Dettmers et al., 2024) freezes the quantized pre-trained model and fine-tunes the LoRA adapter. It introduces k-bit NormalFloat (NFk), Double Quantization, and Paged Optimizers to save memory without sacrificing performance. LoftQ (Li et al., 2024) improves the combination of quantization and fine-tuning in LoRA by minimizing the gap between original and N-bit quantized weights plus low-rank approximations using alternating quantization and SVD. To address information loss in quantization, IR-QLoRA (Qin et al., 2024) proposes statistics-based calibration and entropy maximization to retain original information, along with fine-tuning to enhance LoRA’s information recovery. These hybrid methods preserve pre-quantization data characteristics by training selected full-precision parameters. However, they still inherit the limitations of quantization and PEFT, additionally struggling with handling complex data and remaining susceptible to noise during fine-tuning.

3 Background

3.1 Low-Rank Adaptation (LoRA)

Given a weight matrix \( W \in \mathbb{R}^{d \times t} \) in a neural network, LoRA approximates the fine-tuning updates by decomposing the update matrix \( \Delta W \) into two low-rank matrices (adapters): \( \Delta W = AB \), where \( A \in \mathbb{R}^{d \times r} \) and \( B \in \mathbb{R}^{r \times t} \) are adapters, \( r \) is a hyperparameter representing the rank of the decomposition, typically chosen such that \( r \ll \min(d, t) \). During the fine-tuning process, the weight matrix \( W \) is updated as:

\[
W_{new} = W + \Delta W = W + AB.
\]

This formulation keeps the pre-trained weights \( W \) fixed when training the model on a downstream task with the adapted layers, updating only \( A \) and \( B \) to capture task-specific information. This improves parameter efficiency by reducing the number of trainable parameters from \( dt \) to \( dr + rt \), significantly saving memory and computation costs.

3.2 Quantization

Our research considers two quantization methods: uniform quantization and quantile quantization. Using the quantization function \( q(\cdot) \), the full-precision weights \( W \) are quantized into a k-bit representation \( W_q \). In this paper, \( k \in \{2, 4\} \). If \( q(\cdot) \) represents uniform quantization over the range \([a, b]\), then \( W_q = q(W) = a + \frac{b-a}{2^k-1}\lfloor 2^k - 1 \cdot \frac{W-a}{b-a} \rfloor \), where \( \lfloor \cdot \rfloor \) denotes rounding to the nearest integer. However, uniform quantization suffers from the issue of data difference disappearance in densely populated regions due to its fixed interval mapping. To mitigate this problem, the non-linear method Quantile Quantization (Dettmers et al., 2022) divides data by finding some quantiles and maps them in the same interval to the same value, which allocates more representation capacity to regions with dense data distributions. In our approach, we adopt k-bit *NormalFloat Quantization* (NFk) (Dettmers et al., 2024) as an implementation of Quantile Quantization. Further details are provided in Appendix A.

4 Methodology

We propose a measure and moment-driven fine-tuning framework for quantized LLMs (MeMoTune). This approach explicitly models and learns the pairwise correlations between weights during training, leading to improved performance significantly. Figure 2 is a brief illustration of MeMoTune.
Figure 2: The overview of MeMoTune. The model optimizes parameter distributions and designs a back-propagation method, where \( \nabla \) represents the gradients.

We train the distribution of model parameters from the perspective of probability measure and design the corresponding back-propagation method.

For a domain set \( \mathcal{X} \) and a label set \( \mathcal{Y} \), let \( f : \mathcal{X} \times \mathbb{R}^n \to \mathcal{Y} \) represent a neural network. Denote by \( y_x \) the true label of instance \( x \) and by \( L \) the smooth loss function. Our training objective can be expressed as the following optimization problem.

\[
\min_{A,B} \mathbb{E}_x [L(f(x, \tilde{W} + AB), y_x)],
\]

where \( A \in \mathbb{R}^{d \times r} \) and \( B \in \mathbb{R}^{r \times t} \) are parameter matrices to be fine-tuned, \( \tilde{W} \in \{q(W)\} \subseteq \mathbb{R}^{d \times t} \), and \( \{q(W)\} \) represents the possible value set of the quantized parameter. In the following part of this paper, we will abbreviate \( L(f(x, \tilde{W} + AB), y_x) \) as \( L(f, y) \).

4.1 Measure and Moment-Based Objective

SOTA fine-tuning methods struggle on medium- and large-volume datasets (Chen et al., 2022), and increasing fine-tuning data does not always yield better results (Yang et al., 2024c), potentially due to noise introduced during training (Wei et al., 2022).

To better understand the role of noise in quantized models, we analyzed the layer-wise activation variance of Llama2-7B on the GSM8K dataset under two settings: (1) full-precision and (2) NF4 quantization. Taking layers 17 and 20 as representative examples, the results presented in Table 1 reveal notable variance shifts under quantization, which could be interpreted as noise-like perturbations that may contribute to performance degradation. In addition, models suffer from catastrophic forgetting (Kirkpatrick et al., 2017), as distribution shifts between pre-training and fine-tuning data impair reasoning abilities.

<table>
  <tr>
    <th rowspan="2">Module</th>
    <th colspan="2">Layer 17</th>
    <th colspan="2">Layer 20</th>
  </tr>
  <tr>
    <th>Full-Prec.</th>
    <th>NF4</th>
    <th>Full-Prec.</th>
    <th>NF4</th>
  </tr>
  <tr>
    <td>self\_attn\_q\_proj</td>
    <td>1.7120</td>
    <td>1.8680</td>
    <td>1.6200</td>
    <td>1.7774</td>
  </tr>
  <tr>
    <td>self\_attn\_k\_proj</td>
    <td>3.6261</td>
    <td>3.4212</td>
    <td>3.5204</td>
    <td>3.4119</td>
  </tr>
  <tr>
    <td>self\_attn\_v\_proj</td>
    <td>0.1604</td>
    <td>0.1607</td>
    <td>0.1674</td>
    <td>0.1613</td>
  </tr>
  <tr>
    <td>self\_attn\_o\_proj</td>
    <td>0.0146</td>
    <td>0.0163</td>
    <td>0.0163</td>
    <td>0.0194</td>
  </tr>
  <tr>
    <td>mlp\_gate\_proj</td>
    <td>0.1686</td>
    <td>0.1880</td>
    <td>0.1699</td>
    <td>0.1861</td>
  </tr>
  <tr>
    <td>mlp\_act\_fn</td>
    <td>0.0352</td>
    <td>0.0403</td>
    <td>0.0340</td>
    <td>0.0400</td>
  </tr>
  <tr>
    <td>mlp\_up\_proj</td>
    <td>0.1194</td>
    <td>0.1301</td>
    <td>0.1262</td>
    <td>0.1351</td>
  </tr>
  <tr>
    <td>mlp\_down\_proj</td>
    <td>0.0273</td>
    <td>0.0349</td>
    <td>0.0327</td>
    <td>0.0358</td>
  </tr>
</table>

Table 1: Layer-wise activation variance of Llama2-7B on GSM8K under full-precision and NF4 quantization.

To address this, we propose training parameter distributions instead of single estimates, enabling the model to capture weight correlations and introduce noise, which enhances regularization and improves robustness and generalization (Blundell et al., 2015; Gal and Ghahramani, 2016), while simultaneously preserving the benefits of large models. Thus, we address the problem through probability measures on parameter distributions.

According to the measure and moment approach (Jasour, 2019), the unconstrained optimization with the random variable \( z \):

\[
P^* = \min_{z \in \Omega} p(z),
\]

where \( \Omega = \mathbb{R}^n \), can be transformed into an optimization problem in terms of probability distributions (measures) with the random variable \( \nu \), i.e.,

\[
P_\nu^* = \min_{\nu \in \mathcal{M}(\Omega)} \mathbb{E}_\nu [p(z)] = \min_{\nu \in \mathcal{M}(\Omega)} \int_\Omega p(z) d\nu,
\]

where \( \nu \) is the probability measure associated with \( z \) subject to \( \int_\Omega d\nu = 1 \), and \( \mathcal{M}(\Omega) \) denotes the space of measures supported on \( \Omega \).

Applying this measure-theoretic perspective, we treat the adapter parameters \( A \) and \( B \) as random variables, akin to \( z \) in Eq. (3). Thus, the optimization problem in Eq. (2) can be transformed into:

\[
\min_{\beta_A, \beta_B} \mathbb{E}_{\beta_A, \beta_B} [L(f, y)],
\]

where \( \beta_A \) and \( \beta_B \) are the probability measures of trainable parameters \( A \) and \( B \). This formulation requires finding the optimal probability measures, specifically the probability density functions of \( A \) and \( B \), that minimize the expected loss. For ease of calculation, we assume that \( A \) and \( B \) follow Gaussian distribution, i.e. \( A \sim \mathcal{N}(\mu_A, \Sigma_A) \), \( B \sim \mathcal{N}(\mu_B, \Sigma_B) \), where \( \Sigma_A \) and \( \Sigma_B \) capture the correlation among different elements within \( A \) and \( B \). Since the parameters are not updated directly, it is crucial to clarify how their distributions
are refined, which involve updating the expectation and covariance.

4.2 Distribution Updating
Our goal is to derive the gradients of distribution parameters, i.e., expectation and covariance. During training, we can easily get the gradients \( \nabla_A \mathbb{E}_x[L(f, y)] \) and \( \nabla_B \mathbb{E}_x[L(f, y)] \). Establishing their relationship with distribution parameter gradients allows us to derive a method for updating the distribution.

We begin by analyzing the variable \( A \). To facilitate the derivation, we introduce an auxiliary variable \( r \):
\[
r \sim \mathcal{N}(0, \frac{1}{\sqrt{s}} I),
\]
where \( s \geq 100 \) is a hyperparameter representing the scale size. The scaling factor \( s \) helps reduce excessive noise from the standard normal distribution, preventing adverse effects on large models. We model \( A \) with the expectation and covariance matrix by introducing the random noise \( r \) as follows.
\[
A = \mu_A + \sqrt{\Sigma_{A_r}} r,
\]
where \( \mu_A \) denotes the learnable expectation parameter and \( \Sigma_{A_r} \) denotes the learnable covariance matrix parameter. Under this formulation, we derive \( \Sigma_A = \frac{\Sigma_{A_r}}{s} \), but there is no need to explicitly compute or store it.

To obtain \( A \) after each sampling of \( r \), we infer the gradients of \( \mu_A \) and \( \sqrt{\Sigma_{A_r}} \) from the gradient of \( A \). The specific relationships are stated in the following theorem, with its proof provided in Appendix B.

Theorem 1. For the random variable \( A = \mu_A + \sqrt{\Sigma_{A_r}} r, r \sim \mathcal{N}(0, \frac{1}{\sqrt{s}} I) \) with \( \mu_A \in \mathbb{R}^{d \times r}, \sqrt{\Sigma_{A_r}} \in \mathbb{R}^{d \times r \times K} \) and \( s \in \mathbb{R}^1 \), we have:
\[
\nabla_{\mu_A} \mathbb{E}_x[L(f, y)] = \nabla_A \mathbb{E}_x[L(f, y)]. \tag{8}
\]
\[
\nabla_{\sqrt{\Sigma_{A_r}}} \mathbb{E}_x[L(f, y)] = r \nabla_A \mathbb{E}_x[L(f, y)]. \tag{9}
\]

A large scale \( s \) results in a smaller variance, narrowing the range of \( r \). To prevent the gradient from becoming too small (vanishing), we scale up the gradient as follows:
\[
\nabla_{\sqrt{\Sigma_{A_r}}} \mathbb{E}_x[L(f, y)] := \sqrt{s r} \nabla_A \mathbb{E}_x[L(f, y)]. \tag{10}
\]

Thus, \( (\sqrt{s r}) \sim \mathcal{N}(0, I) \). Consequently, the scaled gradient effectively updates distribution parameters, improving convergence efficiency.

The same derivation applies to \( B \). Through this process, we obtain the gradients for the distribution parameters, which are then used to train the distributions of \( A \) and \( B \), enabling us to solve the objective in Eq. (5).

4.3 Lower Memory and Time Cost
To conserve computing resources, we approximate the square root of the covariance matrix parameter with a low-rank matrix representation: \( \sqrt{\Sigma_{A_r}} \approx \sigma_A \), and \( \sqrt{\Sigma_{B_r}} \approx \sigma_B \), where \( \sigma_A \in \mathbb{R}^{d \times r \times K} \), \( \sigma_B \in \mathbb{R}^{r \times l \times K} \), \( K \ll \min(dr) \). On this basis, the forward pass changes from \( (W_q + AB)x \) to \( (W_q + (\mu_A + \sigma_A r)(\mu_B + \sigma_B r))x \). The corresponding backward pass updates the parameters according to Eq. (8) and Eq. (9).

Therefore, for each layer, the memory cost of the base model parameters is \( \mathcal{O}(\frac{d k}{32}) \). For the trainable part, we need to store the expectation and covariance parameters. Specifically, the memory cost of expectation parameters \( \mu_A \) and \( \mu_B \) is \( \mathcal{O}((k + d)r) \), while the memory cost of the approximate covariance parameters \( \sigma_A \) and \( \sigma_B \) is \( \mathcal{O}((k + d)rK) \). To ensure a low memory cost, we constrain the value of \( K \) to be relatively small, i.e., \( K \leq 4 \). This restriction results in low ranks for \( \Sigma_{A_r} \) and \( \Sigma_{B_r} \), implying that the majority of trainable parameters are independent of each other. This approach is both reasonable and does not compromise the performance of the method. The influence of \( K \) on performance is further evaluated in Section 5.6.

Considering the time complexity, during training, both the expectation and covariance of parameters \( A \) and \( B \) are updated simultaneously, resulting in a time complexity of \( \mathcal{O}((K + 1)(rt + rd)) \). In inference, while dequantization occurs during the forward pass, it typically adds only a constant cost \( \mathcal{O}(c) \) to the standard matrix multiplication. Therefore, the overall time complexity is \( \mathcal{O}(td + (K + 1)(rt + rd) + c) \). In comparison to methods such as QLoRA or LoftQ, the additional complexity arises from the parameter \( K \). By constraining \( K \) to a small value, our method achieves a lower time complexity.

5 Experiments

5.1 Experimental Setup
Datasets and Metrics  We evaluate MeMoTune on three fundamental NLP tasks: NLG, text summarization, and NLU. These tasks assess the model’s ability to generate, condense, and under-
stand text effectively. We summarize the datasets, architectures, and metrics for each task.

For NLG, we follow the QLoRA and IR-QLoRA fine-tuning framework, training Llama (Touvron et al., 2023a) and Llama2 (Touvron et al., 2023b) on Alpaca (Taori et al., 2023), Hh-rlfh (Bai et al., 2022), and GSM8K (Cobbe et al., 2021), and additionally fine-tune Mistral (Jiang et al., 2023) and Qwen2.5 (Yang et al., 2024b; Team, 2024) on GSM8K. Perplexity (Bengio et al., 2003) is used as the evaluation metric for Alpaca and Hh-rlfh, while accuracy is used for GSM8K.

For the summarization task, following the fine-tuning architecture of LoftQ, T5 and BART are fine-tuned on DialogSum (Chen et al., 2021) and BillSum (Kornilova and Eidelman, 2019) datasets, evaluated using ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004) metrics.

For NLU tasks, following the fine-tuning architecture of LoftQ and IR-QLoRA, we fine-tune DeBERTaV3-base on the GLUE benchmark (Wang et al., 2018), with evaluation metrics including Matt, Acc, f1 and P/S Corr. We also evaluate Llama and Llama2 on the MMLU (Hendrycks et al., 2021) dataset, using accuracy as the evaluation metric.

Following (Chen et al., 2022), NLG and summarization fine-tuning occur on large-volume datasets (10k+ samples), while NLU fine-tuning is on medium-volume datasets (1K–10K samples). Base model details are provided in Appendix C.1, and additional information on datasets and evaluation metrics can be found in Appendix C.2.

Training Details For NLG and text summarization tasks, the base models are quantized to 4 bits using NFk (Dettmers et al., 2024; Qin et al., 2024; Li et al., 2024). Adapter ranks are set to 16 and 32 for Alpaca and Hh-rlfh, 64 for GSM8K in NLG tasks, and 16 for summarization tasks. For NLU tasks on GLUE, both NFk and uniform quantization compress the base models to 2 and 4 bits, with adapter ranks of 16 and 32. The fine-tuning settings for Llama and Llama2 on MMLU align with those used for fine-tuning Llama and Llama2 on Alpaca for the NLG task. Model evaluation is performed every 500 training steps, with median results reported over five runs using different random seeds. The adapter rank for all LoRA-based baselines remains consistent with MeMoTune. Detailed configurations are in Appendix C.3, and baselines adhere to the settings specified in their respective papers. Best results are in bold.

We initialize expectations \( \mu_A \) and \( \mu_B \) using an SVD-based method inspired by LoftQ (Li et al., 2024). The adapters are defined as \( \mu_A = [\sqrt{\lambda_i}u_i, \ldots, \sqrt{\lambda_r}u_r] \) and \( \mu_B = [\sqrt{\lambda_i}v_i, \ldots, \sqrt{\lambda_r}v_r] \), where \( \lambda_i \) are the singular values of the residual weight matrix \( W - q(W) \), and \( u_i, v_i \) are the left and right singular vectors. Covariances are initialized to 0.

5.2 Baselines

We compare our method with SOTA approaches and a No Tuning baseline (a full-precision model without fine-tuning). The details are as follows:

PEFT Methods They are full-precision methods that train only a subset of parameters. LoRA (Hu et al., 2021) reduces trainable parameters by introducing low-rank adaptation matrices while keeping the base model frozen, enabling efficient fine-tuning. Prefix Tuning (PT) (Li and Liang, 2021) prepends trainable continuous vectors to the key and value projections in the attention layers.

Quantization Plus PEFT These methods fine-tune a quantized model with PEFT for efficiency. QLoRA (Dettmers et al., 2024) combines k-bit quantization with LoRA, reducing memory while preserving performance. IR-QLoRA (Qin et al., 2024) improves quantized LLM accuracy via integrating LoRA with information retention techniques. LoftQ (Li et al., 2024) enhances QLoRA by optimizing adapter initialization.

5.3 NLG Task

We evaluate MeMoTune on the Alpaca and Hh-rlfh datasets, with results for adapter rank 16 presented in Table 2. MeMoTune consistently outperforms SOTA methods, except for fine-tuning Llama-7B, where full-precision LoRA yields better results. This suggests that optimizing parameter distributions in the measurement space and scaling gradients enhance performance across various inputs. PT fails to converge, while LoRA performs better, reinforcing the effectiveness of fine-tuning within the LoRA framework. On the ALpaca dataset, No Tuning shows better results than on the Hh-rlfh dataset, as pre-trained Llama and Llama2 can adapt well to this dataset. Moreover, on the more complex Hh-rlfh dataset, MeMoTune achieves significant improvements over baselines, underscoring its effectiveness in complex real-world scenarios. Additional results for adapter rank 32 are provided
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Alpaca</th>
    <th colspan="3">Hh-rlfh</th>
  </tr>
  <tr>
    <th>Llama</th>
    <th>Llama2</th>
    <th>Llama</th>
    <th>Llama2</th>
    <th>7B</th>
    <th>13B</th>
    <th>7B</th>
    <th>7B</th>
  </tr>
  <tr>
    <td>No Tuning</td>
    <td>4.40</td>
    <td>4.79</td>
    <td>4.24</td>
    <td>4.69</td>
    <td>8.29</td>
    <td>8.80</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>2.80</td>
    <td><b>2.52</b></td>
    <td>2.85</td>
    <td>2.63</td>
    <td>2.16</td>
    <td>2.30</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PT</td>
    <td>3.02</td>
    <td>2.76</td>
    <td>2.90</td>
    <td>2.64</td>
    <td>4.39</td>
    <td>4.28</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>QLoRA</td>
    <td>2.86</td>
    <td>2.70</td>
    <td>2.88</td>
    <td>2.64</td>
    <td>2.17</td>
    <td>2.32</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>IR-QLoRA</td>
    <td>2.93</td>
    <td>2.82</td>
    <td>2.87</td>
    <td>2.78</td>
    <td>1.98</td>
    <td>2.02</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LoftQ</td>
    <td>2.79</td>
    <td>2.58</td>
    <td>2.86</td>
    <td>2.53</td>
    <td>2.22</td>
    <td>2.24</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MeMoTune</td>
    <td><b>2.76</b></td>
    <td>2.54</td>
    <td><b>2.79</b></td>
    <td><b>2.51</b></td>
    <td><b>1.83</b></td>
    <td><b>1.89</b></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 2: Results of Llama and Llama2 on Alpaca and Hh-rlfh.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Llama</th>
    <th colspan="4">Mistral</th>
    <th colspan="4">Qwen2.5</th>
  </tr>
  <tr>
    <th>7B</th>
    <th>13B</th>
    <th>7B</th>
    <th>13B</th>
    <th>7B</th>
    <th>13B</th>
    <th>7B</th>
    <th>13B</th>
  </tr>
  <tr>
    <td>No Tuning</td>
    <td>11.0</td>
    <td>17.8</td>
    <td>14.6</td>
    <td>28.7</td>
    <td>52.2</td>
    <td>69.4</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>32.8</td>
    <td>36.6</td>
    <td>36.9</td>
    <td>43.1</td>
    <td>57.0</td>
    <td><b>83.0</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>QLoRA</td>
    <td>30.3</td>
    <td>37.4</td>
    <td>35.1</td>
    <td>39.9</td>
    <td>56.7</td>
    <td>79.6</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LoftQ</td>
    <td>34.4</td>
    <td>42.3</td>
    <td>35.1</td>
    <td>40.9</td>
    <td>59.9</td>
    <td>80.8</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MeMoTune</td>
    <td><b>35.8</b></td>
    <td><b>44.2</b></td>
    <td><b>40.9</b></td>
    <td><b>50.5</b></td>
    <td><b>59.0</b></td>
    <td><b>81.3</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 3: Results of Llama, Llama2, Mistral and Qwen2.5 on GSM8K.

in Appendix D.1, where MeMoTune continues to deliver the best performance.

To evaluate problem-solving capabilities, we fine-tune Llama, Llama2, Mistral and Qwen2.5 on the GSM8K dataset, with results in Table 3. Baselines including PT and IR-QLoRA are excluded due to the lack of GSM8K implementations. Baseline results are sourced from LoftQ (Li et al., 2024) (Llama2 fine-tuned with LoftQ and QLoRA), and our training (other cases). MeMoTune consistently achieves the best performance among quantized fine-tuning methods, outperforming QLoRA and LoftQ across all models. It performs particularly excelling on Llama2, with accuracy gains of 4.1% (7B) and 5.5% (13B) over LoftQ. These results demonstrate MeMoTune’s robustness and superior performance in fine-tuning quantized models on large-volume datasets.

5.4 Summarization Task

We fine-tune T5 and BART on the DialogSum and BillSum datasets to evaluate our method for text summarization. During baseline training, we observe that QLoRA outperforms LoRA for fine-tuning T5 on DialogSum, as shown in Figure 3. It is consist with QLoRA’s findings that lower precision of more parameters results in the better performance. We hypothesize that this phenomenon arises because quantization acts as a form of regularization that helps prevent overfitting, a benefit especially useful in tasks like summarization.

Table 4 presents the text summarization results, showing that MeMoTune consistently outperforms baselines on DialogSum and BillSum when fine-tuning T5, BART-base, and BART-large. Among the baselines, LoftQ and IR-QLoRA excel in different scenarios. IR-QLoRA outperforms LoftQ on both datasets for fine-tuning T5 and BART-base, ranking just behind MeMoTune, while LoftQ performs better on BART-large. Full-precision methods do not surpass quantized ones, highlighting the benefits of quantization in summarization. These results confirm its ability to enhance quantized fine-tuning by optimizing parameter distributions in the measurement space, ensuring consistency across datasets and model sizes.

![Fine-tuning T5 using LoRA, QLoRA, and MeMoTune on DialogSum.](page_1012_186_410_324.png)

Figure 3: Fine-tuning T5 using LoRA, QLoRA, and MeMoTune on DialogSum.

5.5 NLU Task

We evaluate MeMoTune’s ability to fine-tune extremely low-bit models. Table 5 presents the results of fine-tuning 2-bit DeBERTaV3-base on the GLUE corpus (CoLA, MRPC, and STS-B), with baseline values from the LoftQ paper. IR-QLoRA has not yet implemented the GLUE task for DeBERTaV3-base. MeMoTune consistently outperforms LoftQ in all cases, showing stable and superior performance. Notably, on CoLA with NF2 quantization and rank 32, MeMoTune achieves a Matthews correlation 1.7% higher than LoftQ, highlighting improved fine-tuning. Additional results for 4-bit DeBERTaV3-base in Appendix D.2 further confirm MeMoTune’s effectiveness. By optimizing parameter distributions, MeMoTune excels in fine-tuning 2-bit and 4-bit models on medium-volume datasets.

The MMLU accuracy of fine-tuning Llama and Llama2 on the Alpaca dataset, with adapter rank 16, is shown in Table 6, with baseline values sourced from QLoRA (Dettmers et al., 2024) (QLoRA) and
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">DialogSum</th>
    <th colspan="3">BillSum</th>
  </tr>
  <tr>
    <th>T5</th>
    <th>BART-base</th>
    <th>BART-large</th>
    <th>T5</th>
    <th>BART-base</th>
    <th>BART-large</th>
  </tr>
  <tr>
    <td>No Tuning</td>
    <td>20.4/5.0/17.3</td>
    <td>17.4/2.7/15.6</td>
    <td>17.5/2.7/15.8</td>
    <td>15.1/7.0/12.5</td>
    <td>3.6/0.5/3.2</td>
    <td>6.6/1.9/5.8</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>29.8/5.5/25.2</td>
    <td>36.9/12.3/30.4</td>
    <td>39.9/15.7/33.6</td>
    <td>21.8/16.6/20.8</td>
    <td>22.3/17.2/21.6</td>
    <td>23.4/18.8/22.7</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>31.2/9.7/26.4</td>
    <td>39.0/14.0/32.2</td>
    <td>41.4/17.7/35.6</td>
    <td>23.5/18.6/22.5</td>
    <td>23.6/18.9/22.9</td>
    <td>24.5/20.5/24.0</td>
  </tr>
  <tr>
    <td>QLoRA</td>
    <td>32.3/9.9/27.0</td>
    <td>38.1/14.3/32.5</td>
    <td>41.8/17.8/35.6</td>
    <td>23.7/18.8/22.8</td>
    <td>23.8/19.3/23.2</td>
    <td>24.3/20.2/23.7</td>
  </tr>
  <tr>
    <td>LoftQ</td>
    <td>33.6/9.7/28.7</td>
    <td>38.9/14.6/32.4</td>
    <td>42.2/18.3/36.0</td>
    <td>23.7/18.8/22.9</td>
    <td>23.8/19.3/23.1</td>
    <td>24.4/20.4/23.9</td>
  </tr>
  <tr>
    <td>IR-QLoRA</td>
    <td>36.6/12.5/30.2</td>
    <td>39.3/14.9/32.6</td>
    <td>41.5/18.0/35.5</td>
    <td>23.4/18.5/22.4</td>
    <td>23.6/18.8/22.9</td>
    <td>24.0/19.7/23.4</td>
  </tr>
  <tr>
    <td>MeMoTune</td>
    <td><b>36.7/12.5/30.4</b></td>
    <td><b>39.5/15.0/32.8</b></td>
    <td><b>42.5/18.8/36.4</b></td>
    <td><b>23.8/18.9/23.0</b></td>
    <td><b>23.9/19.5/23.3</b></td>
    <td><b>24.5/20.7/24.3</b></td>
  </tr>
</table>

Table 4: Results of T5, BART-base and BART-large on DialogSum and BillSum, evaluated with ROUGE-1/2/L (%).

<table>
  <tr>
    <th rowspan="2">Quant</th>
    <th rowspan="2">Rank</th>
    <th rowspan="2">Method</th>
    <th colspan="2">CoLA</th>
    <th colspan="2">MRPC</th>
    <th colspan="2">STS-B</th>
  </tr>
  <tr>
    <th>Matt</th>
    <th>Acc/f1</th>
    <th>P/S</th>
    <th>Corr</th>
  </tr>
  <tr>
    <td rowspan="2">NF</td>
    <td rowspan="2">16</td>
    <td>LoftQ</td>
    <td>37.4</td>
    <td>83.8/88.6</td>
    <td>87.1/86.9</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MeMoTune</td>
    <td><b>38.6</b></td>
    <td><b>84.6/89.0</b></td>
    <td><b>87.7/87.4</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Unif</td>
    <td rowspan="2">32</td>
    <td>LoftQ</td>
    <td>47.5</td>
    <td>83.6/87.2</td>
    <td>87.5/87.0</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MeMoTune</td>
    <td><b>49.2</b></td>
    <td><b>84.6/89.2</b></td>
    <td><b>88.2/87.9</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Unif</td>
    <td rowspan="2">16</td>
    <td>LoftQ</td>
    <td>59.1</td>
    <td>87.0/90.6</td>
    <td>87.9/87.6</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MeMoTune</td>
    <td><b>59.6</b></td>
    <td><b>87.8/91.4</b></td>
    <td><b>88.2/87.8</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Unif</td>
    <td rowspan="2">32</td>
    <td>LoftQ</td>
    <td>60.5</td>
    <td>87.5/91.2</td>
    <td>89.5/89.2</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>MeMoTune</td>
    <td><b>61.3</b></td>
    <td><b>88.0/91.4</b></td>
    <td><b>89.9/89.7</b></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 5: Results of 2-bit DeBERTaV3-base on GLUE.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Llama</th>
    <th colspan="2">Llama2</th>
  </tr>
  <tr>
    <th>7B</th>
    <th>13B</th>
    <th>7B</th>
    <th>13B</th>
  </tr>
  <tr>
    <td>No Tuning</td>
    <td>35.1</td>
    <td>43.3</td>
    <td>44.3</td>
    <td>55.3</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>36.8</td>
    <td>45.3</td>
    <td>45.4</td>
    <td>54.6</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>40.6</td>
    <td>47.4</td>
    <td>46.4</td>
    <td><b>57.3</b></td>
  </tr>
  <tr>
    <td>QLoRA</td>
    <td>38.8</td>
    <td>47.8</td>
    <td>44.4</td>
    <td>51.6</td>
  </tr>
  <tr>
    <td>LoftQ</td>
    <td>39.1</td>
    <td>48.5</td>
    <td>44.7</td>
    <td>53.5</td>
  </tr>
  <tr>
    <td>IR-QLoRA</td>
    <td>40.6</td>
    <td>49.3</td>
    <td>46.1</td>
    <td>54.4</td>
  </tr>
  <tr>
    <td>MeMoTune</td>
    <td><b>40.9</b></td>
    <td><b>50.6</b></td>
    <td><b>46.4</b></td>
    <td>56.6</td>
  </tr>
</table>

Table 6: MMLU results of Llama and Llama2 on Alpaca.

IR-QLoRA (Qin et al., 2024) (No Tuning and IR-QLoRA) reports, compared to our training (LoRa, LoftQ, MeMoTune). MeMoTune achieves the highest accuracy among methods using quantized base models, outperforming IR-QLoRA by 1.3% on Llama-13B and 2.2% on Llama2-13B. This highlights its superior performance, particularly for larger models. Additional results for adapter rank 32 are provided in Appendix D.3. These results indicate that MeMoTune is highly effective on large-volume datasets.

5.6 Ablation Study

Effect of K. Building on prior results, training the parameter distributions has demonstrated effectiveness. Next, we explore the covariance within this distribution by comparing the effects of different \( K \) on the experimental outcomes. We fine-tune T5 and BART on DialogSum, with low-rank adapters of rank 16, and record the ROUGE-L scores for various \( K \), as shown in Figure 4.

We found that \( K \) has little effect on model performance, with results fluctuating within an acceptable range. This indicates that the elements in the parameter matrix are nearly independent, allowing a low-rank covariance matrix to achieve good results. However, as \( K \) increases (\( e.g. \), \( K = 8 \)), memory and computation costs rise significantly, making it difficult to achieve optimal performance. Therefore, we select \( K \in \{1, 2, 4\} \) for this study. See Appendix C.3 for specific settings.

![Fine-tuning T5 and BART on DialogSum.](page_1012_1012_388_246.png)

Figure 4: Fine-tuning T5 and BART on DialogSum.

5.7 Loss Convergence in Fine-Tuning

We fine-tune the 4-bit Llama2-7B initialized with LoftQ by QLoRA and MeMoTune on the Hhrlhf (Bai et al., 2022) dataset, where the parameters of the former follow the settings of LoftQ (Li et al., 2024) and both methods shuffle the data before each epoch of training. The changes in model performance are shown in Figure 5.

With LoftQ, training loss drops sharply at the start of each epoch but gradually increases over time. In contrast, MeMoTune mitigates this issue, achieving a faster perplexity reduction. We suspect this is due to LoftQ training parameters within the LoRA framework, which, owing to its low-rank nature, may struggle with complex data. By modeling parameter distributions, MeMoTune enhances robustness, generalization, and mitigates catastrophic forgetting in large models. Notably, further reducing the learning rate leads to earlier
Figure 5: Fine-tuning Llama2-7B using LoftQ and MeMoTune on Hh-rlfh.

<table>
  <tr>
    <th>Noise Level</th>
    <th>LoftQ</th>
    <th>MeMoTune</th>
  </tr>
  <tr>
    <td>0.00 (No Noise)</td>
    <td>36.98</td>
    <td>41.08</td>
  </tr>
  <tr>
    <td>0.01</td>
    <td>34.95</td>
    <td>37.83</td>
  </tr>
  <tr>
    <td>0.05</td>
    <td>22.74</td>
    <td>25.63</td>
  </tr>
  <tr>
    <td>0.10</td>
    <td>11.07</td>
    <td>14.86</td>
  </tr>
  <tr>
    <td>0.15</td>
    <td>6.60</td>
    <td>6.97</td>
  </tr>
  <tr>
    <td>0.20</td>
    <td>3.41</td>
    <td>4.09</td>
  </tr>
</table>

Table 7: Accuracy under varying noise levels on GSM8K.

convergence with worse performance, so we select an initial learning rate of 0.01 with a cosine decay schedule, as detailed in Section 5.1.

5.8 Evaluating Noise Robustness

To further demonstrate the noise robustness of our proposed method, we conducted additional experiments of fine-tuning Llama2-7B on the GSM8K dataset by injecting character-level noise into the input data at evaluation time. Specifically, we added a controlled amount of random noise to the input text by randomly performing character replacements, insertions, and deletions with a fixed probability (noise level). For example, a noise level of 0.05 indicates that approximately 5% of the characters in the input are altered. The performance under varying noise levels is shown in the Table 7.

MeMoTune consistently outperforms LoftQ across all noise levels. Notably, it even surpasses LoftQ’s clean performance at 1% noise, demonstrating improved robustness to input perturbations.

6 Conclusion

We propose MeMoTune, a measure and moment-driven fine-tuning framework for quantized LLMs, combining PEFT to adapt quantized LLMs for downstream tasks with limited memory. We introduce Gaussian-distributed parameters in LoRA to enhance results. We analyze training distribution theoretically and propose scaled probability measure gradients. Experiments show MeMoTune surpasses existing models in NLP tasks and is ideal for lightweight devices.

7 Limitations

Despite the improvements introduced by the MeMoTune in mitigating quantization-related performance loss, several limitations must be acknowledged. First, our method is validated on the English database, and its effectiveness on non-English datasets or multilingual tasks has not been fully explored, which may limit its applicability to broader language models. Moreover, the method’s effectiveness has primarily been validated on popular datasets, such as Alpaca, Hh-rlfh, GSM8K, DialogSum, BillSum, and GLUE benchmark. However, its ability to address the challenges posed by more niche or broader datasets remains somewhat less explored.

Acknowledgments

This research work is supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021), and by the National Natural Science Foundation of China (Grant Nos. 62272093, 62137001, and 62461146205)

References

AI@Meta. 2024. Llama 3 model card.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. Preprint, arXiv:2204.05862.

Ron Banner, Yury Nahshan, and Daniel Soudry. 2019. Post training 4-bit quantization of convolutional networks for rapid-deployment. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res., 3(null):1137–1155.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight uncertainty in neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, page 1613–1622. JMLR.org.

Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. 2022. Revisiting parameter-efficient tuning: Are we really there yet? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2612–2626, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A real-life scenario dialogue summarization dataset. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5062–5074, Online. Association for Computational Linguistics.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. 8-bit optimizers via block-wise quantization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.

Elias Frantar, Saleh Ashkboos, Torsten Hoeﬂer, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.

Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059, New York, New York, USA. PMLR.

Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021a. DebertaV3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021b. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Ashkan Jasour. 2019. Lecture 5: Duality of sos and moment based sdpss. In MIT 16.S498: Risk Aware and Robust Nonlinear Planning. Cambridge MA. MIT OpenCourseWare.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526.

Anastassia Komilova and Vladimir Eidelman. 2019. BillSum: A corpus for automatic summarization of US legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 48–56, Hong Kong, China. Association for Computational Linguistics.

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.

Mike Lewis, Yinhao Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics.

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597.
Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2024. Loftq: Lora-fine-tuning-aware quantization for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100.

Jiawei Liu, Lin Niu, Zhihang Yuan, Dawei Yang, Xinggang Wang, and Wenyu Liu. 2023a. Pd-quant: Post-training quantization based on prediction difference metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24427–24437.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023b. Gpt understands, too. AI Open.

Yijiang Liu, Huanrui Yang, Zhen Dong, Kurt Keutzer, Li Du, and Shanghang Zhang. 2023c. Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20321–20330.

Xudong Lu, Aojun Zhou, Yuhui Xu, Renrui Zhang, Peng Gao, and Hongsheng Li. 2024. SPP: sparsity-preserved parameter-efficient fine-tuning for large language models. In ICML. OpenReview.net.

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft.

Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, and Jinan Xu. 2024. Dora: Enhancing parameter-efficient fine-tuning with dynamic rank distribution. In ACL (1), pages 11662–11675. Association for Computational Linguistics.

Mahdi Nikdan, Soroush Tabesh, Elvir Crnecvic, and Dan Alistarh. 2024. Rosa: Accurate parameter-efficient fine-tuning via robust adaptation. In ICML. OpenReview.net.

Haotong Qin, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xianglong Liu, and Michele Magno. 2024. Accurate lora-finetuning quantization of llms via information retention. arXiv preprint arXiv:2402.05445.

Pengjie Ren, Chengshun Shi, Shiguang Wu, Menggi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. 2024. Melora: Mini-ensemble low-rank adapters for parameter-efficient fine-tuning. In ACL (1), pages 3052–3064. Association for Computational Linguistics.

Juncheol Shin, Junhyuk So, Sein Park, Seungyeop Kang, Sungjoo Yoo, and Eunhyeok Park. 2023. Nipq: Noise proxy-based integrated pseudo-quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3852–3861.

Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi (Viji) Srinivasan, and Kailash Gopalakrishnan. 2020. Ultra-low precision 4-bit training of deep neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 1796–1807. Curran Associates, Inc.

Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. 2017. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):2295–2329.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.

Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenying Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yunjing Mao, Xavier Martinet, Todor Mihaoylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and
Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09288.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics.

Sheng Wang, Boyang Xue, Jiacheng Ye, Jiyue Jiang, Liheng Chen, Lingpeng Kong, and Chuan Wu. 2024. Prolora: Partial rotation empowers more parameter-efficient lora. In ACL (1), pages 2829–2841. Association for Computational Linguistics.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In ICLR. OpenReview.net.

Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. 2023. Outlier suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1648–1665, Singapore. Association for Computational Linguistics.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR.

Jing Xu and Jingzhao Zhang. 2024. Random masking finds winning tickets for parameter efficient fine-tuning. In ICML. OpenReview.net.

Adam X. Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. 2024a. Bayesian low-rank adaptation for large language models. In The Twelfth International Conference on Learning Representations.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. 2024b. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Haoran Yang, Yunmeng Zhang, Jiaqi Xu, Hongyuan Lu, Pheng-Ann Heng, and Wai Lam. 2024c. Unveiling the generalization power of fine-tuned large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 884–899, Mexico City, Mexico. Association for Computational Linguistics.

Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168–27183.

Zhi Zhang, Qizhe Zhang, Zijun Gao, Renrui Zhang, Ekaterina Shutova, Shiji Zhou, and Shanghang Zhang. 2024. Gradient-based parameter selection for efficient fine-tuning. In CVPR, pages 28566–28577. IEEE.

Ke Zhu, Yin-Yin He, and Jianxin Wu. 2023. Quantized feature distillation for network quantization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11452–11460.
A  The k-bit Quantile Quantization

For dividing the original data into \(2^{k+1}\) intervals, we first estimate the \(2^{k+1}\) quantiles of a normal distribution \(\mathcal{N}(0, 1)\), its quantile function is \(Q_X(\cdot)\). Then we normalize the values of \(k\)-bit data into the \([-1, 1]\). The process of determining the quantiles for quantization can be described mathematically as follows:

\[
q_i = \frac{1}{2}(Q_X(\frac{i}{2^k + 1}) + Q_X(\frac{i + 1}{2^k + 1})),
\]

where \(i = 0, 1, 2, \ldots, 2^k - 1\) represent the interval, \(q_i\) represents the quantiles of each interval. Next, we normalize the input weights into the \([-1, 1]\) by \(W_N = \frac{W}{\max|W|}\). By finding the closest \(q_i\) to each elements in \(W_N\), we get the quantized result \(i\).

During dequantization, the quantized indices \(i\) are used to retrieve the corresponding quantiles \(q_i\), and the weight matrix is reconstructed through denormalization as:

\[
W_q = q_{i,\max}|W|.
\]

B  Proofs of Theorem 1

This is a proof of the proposed gradient of the adapter’s distribution parameters in Theorem 1.

Proof.

\[
\nabla_{\mu_A} \mathbb{E}_x[L(f, y)] = \frac{\partial \mathbb{E}_x[L(f, y)]}{\partial A} \frac{\partial A}{\partial \mu_A}
= \nabla_A \mathbb{E}_x[L(f, y)].
\]

\[
\nabla_{\Sigma_{A_r}} \mathbb{E}_x[L(f, y)] = \frac{\partial \mathbb{E}_x[L(f, y)]}{\partial A} \frac{\partial A}{\partial \sqrt{\Sigma_{A_r}}}
= r \nabla_A \mathbb{E}_x[L(f, y)].
\]

C  Details of Experimental Settings

C.1  Base Models

We fine-tuned the following base models to verify that MeMoTune maintains stable and excellent performance across different scenarios.

• Llama (7B, 13B) (Touvron et al., 2023a) and Llama2 (7B, 13B) (Touvron et al., 2023b): Optimized for tasks such as language modeling, text generation, and reasoning, demonstrating strong capabilities in handling complex language patterns efficiently.

• T5 (small) (Chung et al., 2024): A versatile model that frames all NLP tasks as text-to-text transformations, excelling in applications like translation, summarization, and question answering.

• BART (base, large) (Lewis et al., 2019): Combines bidirectional and autoregressive transformers, achieving high performance in text generation tasks such as image description, summarization, and machine translation.

• DeBERTaV3 (base) (He et al., 2021a,b): Enhances BERT’s architecture with disentangled attention mechanisms to improve contextual understanding, making it particularly effective for text classification and sentiment analysis tasks.

C.2  Dataset and Metric

NLG. NLG tasks cover a wide range, and we select three datasets with different purposes for fine-tuning for more comprehensive evaluation.

• Alpaca (Taori et al., 2023): Contains 52,000 instructions and demonstrations, helping to fine-tune models to understand and follow complex instructions accurately.

• Hh-rlhf (Bai et al., 2022): Provides Human Preference and Red Teaming Data, focusing on fine-tuning models for dialogue generation and alignment with human preferences.

• GSM8K (Cobbe et al., 2021): Includes 8.5K math word problems designed for reasoning tasks, such as solving grade-school math problems.

Evaluation on Alpaca and Hh-rlhf is conducted using perplexity, where lower values indicate better performance.

Summarization. We consider two datasets:

• DialogSum (Chen et al., 2021): A dialogue summarization dataset containing 13,460 dialogues.

• BillSum (Kornilova and Eidelman, 2019): A dataset containing summaries of U.S. Congressional and California state bills.

Regarding the evaluation metrics, ROUGE-1 measures unigram (word-level) overlap between the
generated and reference summaries, while ROUGE-2 captures bigram overlap. ROUGE-L, based on the longest common subsequence, accounts for sentence-level structural similarity by identifying the longest co-occurring n-gram. Higher scores indicate better performance.

NLU. We use two benchmarks:

• GLUE Benchmark (Wang et al., 2018): General Language Understanding Evaluation Benchmark, introduced in 2018, evaluates a model’s general language understanding across nine NLP tasks, including sentence similarity, inference, and sentiment analysis. The GLUE tasks used in this research are:

– Corpus of Linguistic Acceptability (CoLA) – Assesses whether a sentence is grammatically correct, evaluated using Matthew’s Correlation Coefficient (Matt).
– Microsoft Research Paraphrase Corpus (MRPC) – Determines semantic equivalence between sentence pairs, measured by accuracy and F1 score (Acc/f1).
– Semantic Textual Similarity Benchmark (STS-B) – Measures sentence similarity on a continuous scale, evaluated using Pearson/Spearman correlation against human judgments (P/S Corr).

• MMLU (Hendrycks et al., 2021): Massive Multitask Language Understanding, a multi-task benchmark covering 57 subjects across humanities, STEM, and social sciences. Performance is measured by the average accuracy across all tasks.

Higher values indicate better performance for all metrics.

C.3 Experimental Configurations

All experiments are conducted on NVIDIA A100 SXM4 GPUs.

NLG Tasks. For training Llama and Llama2 on the Alpaca and Hh-rlhf datasets, we use a cosine decay learning rate schedule with an initial learning rate of 0.01. On GSM8K, the learning rate is set to \( 3 \times 10^{-4} \). Fine-tuning is performed for 20,000 steps on Alpaca, 100,000 steps on Hh-rlhf, and 1,880 steps on GSM8K, with a batch size of 2 for both training and evaluation. For all NLG tasks, the scale size \( s \) is fixed at 400 for Llama and Llama2, and 100 for Mistral and Qwen2.5. The \( K \)-value is set to 2 for Llama-7B and Llama2-7B, and to 1 for Llama-13B, Llama2-13B, Mistral and Qwen2.5.

Summarization Tasks. The learning rate is configured to \( 2 \times 10^{-5} \). Fine-tuning is conducted for 40,000 steps on the BillSum dataset and 20,000 steps on the DialogSum dataset, using a consistent batch size of 8 for both training and evaluation. The hyperparameter \( K \) is set to 4, and the scale size \( s \) is set to 1000.

NLU Tasks. Training and evaluation are conducted with a batch size of 16 and 60 epochs across all experiments. Hyperparameters, including learning rate, scale, and \( K \) vary across quantization method, datasets and rank. The detailed configurations are summarized in Table 8 (for 2-bit quantization) and Table 9 (for 4-bit quantization).

<table>
  <tr>
    <th>Quant</th>
    <th>Dataset</th>
    <th>Rank</th>
    <th>Learning Rate</th>
    <th>Scale</th>
    <th>K</th>
  </tr>
  <tr>
    <td>NF2</td>
    <td>cola</td>
    <td>16</td>
    <td>1.00E-04</td>
    <td>1000</td>
    <td>1</td>
  </tr>
  <tr>
    <td>NF2</td>
    <td>cola</td>
    <td>32</td>
    <td>5.00E-05</td>
    <td>600</td>
    <td>1</td>
  </tr>
  <tr>
    <td>NF2</td>
    <td>mrpc</td>
    <td>16</td>
    <td>1.00E-04</td>
    <td>500</td>
    <td>1</td>
  </tr>
  <tr>
    <td>NF2</td>
    <td>mrpc</td>
    <td>32</td>
    <td>1.00E-04</td>
    <td>600</td>
    <td>1</td>
  </tr>
  <tr>
    <td>NF2</td>
    <td>sts</td>
    <td>16</td>
    <td>1.00E-04</td>
    <td>700</td>
    <td>1</td>
  </tr>
  <tr>
    <td>NF2</td>
    <td>sts</td>
    <td>32</td>
    <td>1.00E-04</td>
    <td>700</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>cola</td>
    <td>16</td>
    <td>1.00E-04</td>
    <td>1000</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>cola</td>
    <td>32</td>
    <td>1.00E-04</td>
    <td>4000</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>mrpc</td>
    <td>16</td>
    <td>1.00E-04</td>
    <td>700</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>mrpc</td>
    <td>32</td>
    <td>1.00E-04</td>
    <td>100</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>sts</td>
    <td>16</td>
    <td>5.00E-05</td>
    <td>300</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>sts</td>
    <td>32</td>
    <td>5.00E-05</td>
    <td>300</td>
    <td>1</td>
  </tr>
</table>

Table 8: Hyperparameter configurations for fine-tuning 2-bit DeBERTaV3-base on NLU Tasks.

<table>
  <tr>
    <th>Quant</th>
    <th>Dataset</th>
    <th>Rank</th>
    <th>Learning Rate</th>
    <th>Scale</th>
    <th>K</th>
  </tr>
  <tr>
    <td>NF2</td>
    <td>cola</td>
    <td>16</td>
    <td>1.00E-04</td>
    <td>400</td>
    <td>1</td>
  </tr>
  <tr>
    <td>NF2</td>
    <td>cola</td>
    <td>32</td>
    <td>5.00E-05</td>
    <td>400</td>
    <td>1</td>
  </tr>
  <tr>
    <td>NF2</td>
    <td>mrpc</td>
    <td>16</td>
    <td>1.00E-04</td>
    <td>600</td>
    <td>1</td>
  </tr>
  <tr>
    <td>NF2</td>
    <td>mrpc</td>
    <td>32</td>
    <td>1.00E-04</td>
    <td>200</td>
    <td>1</td>
  </tr>
  <tr>
    <td>NF2</td>
    <td>sts</td>
    <td>16</td>
    <td>1.00E-04</td>
    <td>1000</td>
    <td>1</td>
  </tr>
  <tr>
    <td>NF2</td>
    <td>sts</td>
    <td>32</td>
    <td>1.00E-04</td>
    <td>900</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>cola</td>
    <td>16</td>
    <td>1.00E-04</td>
    <td>400</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>cola</td>
    <td>32</td>
    <td>1.00E-04</td>
    <td>4000</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>mrpc</td>
    <td>16</td>
    <td>1.00E-04</td>
    <td>700</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>mrpc</td>
    <td>32</td>
    <td>1.00E-04</td>
    <td>100</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>sts</td>
    <td>16</td>
    <td>5.00E-05</td>
    <td>300</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Unif</td>
    <td>sts</td>
    <td>32</td>
    <td>5.00E-05</td>
    <td>400</td>
    <td>1</td>
  </tr>
</table>

Table 9: Hyperparameter configurations for fine-tuning 4-bit DeBERTaV3-base on NLU Tasks.
D Additional Results

D.1 NLG Task with Adapter Ranks 32
Table 10 presents the results of fine-tuning MeMoTune on the Alpaca dataset with an adapter rank of 32. Compared to the results with rank 16 in Table 2, increasing the rank to 32 leads to a performance drop for both MeMoTune and the baselines when fine-tuning 13B models, while fine-tuning 7B models shows improved performance. This highlights the importance of selecting an appropriate adapter rank based on model size and specific scenarios. Notably, MeMoTune consistently outperforms all baselines across different settings, demonstrating robust performance in various conditions.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Llama</th>
    <th colspan="2">Llama2</th>
  </tr>
  <tr>
    <th>7B</th>
    <th>13B</th>
    <th>7B</th>
    <th>13B</th>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>2.76</td>
    <td>2.68</td>
    <td>2.74</td>
    <td>2.60</td>
  </tr>
  <tr>
    <td>QLoRA</td>
    <td>2.80</td>
    <td>2.72</td>
    <td>2.75</td>
    <td>2.68</td>
  </tr>
  <tr>
    <td>IR-QLoRA</td>
    <td>2.91</td>
    <td>2.83</td>
    <td>2.85</td>
    <td>2.78</td>
  </tr>
  <tr>
    <td>LoftQ</td>
    <td>2.75</td>
    <td>2.66</td>
    <td>2.71</td>
    <td>2.59</td>
  </tr>
  <tr>
    <td>MeMoTune</td>
    <td><b>2.70</b></td>
    <td><b>2.63</b></td>
    <td><b>2.66</b></td>
    <td><b>2.57</b></td>
  </tr>
</table>

Table 10: Results of Llama and Llama2 on Alpaca. The adapters’ rank is set to 32.

optimal rank setting may vary depending on specific scenarios.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Llama</th>
    <th colspan="2">Llama2</th>
  </tr>
  <tr>
    <th>7B</th>
    <th>13B</th>
    <th>7B</th>
    <th>13B</th>
  </tr>
  <tr>
    <td>LoRA</td>
    <td><b>41.5</b></td>
    <td>48.4</td>
    <td>47.1</td>
    <td>56.7</td>
  </tr>
  <tr>
    <td>QLoRA</td>
    <td>40.5</td>
    <td>48.4</td>
    <td>45.7</td>
    <td>55.5</td>
  </tr>
  <tr>
    <td>LoftQ</td>
    <td>39.3</td>
    <td>48.8</td>
    <td>47.0</td>
    <td>55.4</td>
  </tr>
  <tr>
    <td>IR-QLoRA</td>
    <td>39.0</td>
    <td>47.6</td>
    <td>46.5</td>
    <td>54.6</td>
  </tr>
  <tr>
    <td>MeMoTune</td>
    <td>41.3</td>
    <td><b>49.4</b></td>
    <td><b>47.7</b></td>
    <td><b>56.8</b></td>
  </tr>
</table>

Table 12: MMLU results of Llama and Llama2 on Alpaca. The adapters’ rank is set to 32.

D.2 4-Bit DeBERTaV3-base
Table 11 shows the 4-bit fine-tuning results of DeBERTaV3-base on the GLUE tasks, where our method consistently outperforms LoftQ.

<table>
  <tr>
    <th rowspan="2">Quant</th>
    <th rowspan="2">Rank</th>
    <th rowspan="2">Method</th>
    <th colspan="2">CoLA</th>
    <th colspan="2">MRPC</th>
    <th colspan="2">SST</th>
  </tr>
  <tr>
    <th>Matt</th>
    <th>Acc/F1</th>
    <th>P/S</th>
    <th>Corr</th>
  </tr>
  <tr>
    <td rowspan="2">NF</td>
    <td>16</td>
    <td>LoftQ</td>
    <td>69.6</td>
    <td>90.2/92.9</td>
    <td>90.8/90.8</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>MeMoTune</td>
    <td><b>70.2</b></td>
    <td><b>90.4/93.1</b></td>
    <td><b>90.9/90.9</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Unif</td>
    <td>32</td>
    <td>LoftQ</td>
    <td>67.2</td>
    <td>89.2/92.3</td>
    <td>91.0/91.2</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>MeMoTune</td>
    <td><b>70.8</b></td>
    <td><b>89.5/92.4</b></td>
    <td><b>91.2/91.2</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>16</td>
    <td>LoftQ</td>
    <td>67.7</td>
    <td>88.7/92.0</td>
    <td>88.9/88.8</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>MeMoTune</td>
    <td><b>70.1</b></td>
    <td><b>89.0/92.1</b></td>
    <td><b>90.2/90.2</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>32</td>
    <td>LoftQ</td>
    <td>65.4</td>
    <td>86.8/90.6</td>
    <td>90.5/90.5</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>MeMoTune</td>
    <td><b>68.8</b></td>
    <td><b>89.7/92.7</b></td>
    <td><b>90.7/90.7</b></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 11: Results of 4-bit DeBERTaV3-base on GLUE.

D.3 NLU Task with Adapter Ranks 32
Table 12 presents the MMLU results for the NLU task with an adapter rank of 32. Notably, MeMoTune consistently achieves the best performance, except in the case of fine-tuning Llama-7B, where its result is slightly inferior to that of LoRA. This highlights LoRA’s advantage as a full-precision model. Moreover, compared to rank 16 (Table 6), fine-tuning 7B models with rank 32 generally yields better results, aligning with the conclusions in Appendix D.1. These findings suggest that the

Reviewer paper summaries: The paper introduce a new fine-tuning framework for quantized LLMs, where the weight update is performance in low-rank probability space. The paper problem an new PEFT method to enhance convergence efficiency and improve robustness. Experimental results shows the proposed method achieves better accuracy compared with the SOTA methods.

This paper introduces a new PEFT method. The method employs a low-rank approximation framework in probability measure space, optimizing parameter distributions (instead of single estimates) to capture weight correlations and inject noise. This is claimed to enhance regularization, robustness, and generalization in medium/large-volume datasets. Code is provided for reproducibility. Mainly, the authors assume that matrices A and B are drawn from a normal distribution, with learnable mean and variance parameters that are sampled during fine-tuning. This assumption consequently leads to gradient scaling derivation.

MeMoTune is a novel framework for fine-tuning quantized LLMs that models trainable weights as Gaussian distributions within the LoRA architecture. By optimizing in probability measure space and using scaled gradients, it improves convergence and noise robustness. Experiments show significant performance gains across text generation, summarization, and understanding tasks, outperforming state-of-the-art methods like QLoRA, IR-QLoRA, and LoftQ.
Summary of strengths: The weight update method based on the uncertainty is an interesting topic that enjoys the feature of improve robustness. The experimental setup covers widely accepted tasks and models. The writing of this paper is good and I can easily follow the work.

Novel Methodology:   While injecting Gaussian noise for PEFT is not entirely novel authors provide a new and unique method. The use of low-rank approximation in probability measure space provides a mathematical foundation for noise injection and gradient scaling, this potentially can be developed further. 


Empirical Validation: Results on Llama2-13B and DeBERTaV3-base demonstrate consistent gains over other methods.


Reproducibility: The code is present and reproducible.

1. The paper presents a well-developed mathematical approach that applies measure and moment techniques to model parameter distributions rather than single estimates, with thorough theoretical analysis
2. MeMoTune outperforms baselines across diverse tasks and models. For instance, it improved GSM8K accuracy by 5.5% with Llama2-13B and achieved better perplexity scores (1.89 vs 2.02 from IR-QLoRA) on the Hh-rlhf dataset.
Summary of weaknesses: The main concern that make me hesitate for assigning the score is a relative work about Bayesian LoRA in ICLR 2024 [1]. The main method, which define the parameters in the LoRA weights A and B based on gaussian distribution, is similar to the idea of replacing the linear layer (A, B) in LoRA adapter with Bayesian neural network (BNN) layer. Also, the noise consider in eq. (7) and the reparameterization trick considered in eq. (10) are widely used technique in the variance inference. These things make the method proposed in this paper similar as the one proposed in [1], expect that this paper didn't consider an explicit posterior and optimize the objective function in a more deterministic way. I would more like to understand this paper as a partial bayesian treatment compared with the full bayesian treatment considered in [1]. I'm not sure if the author notice [1] before, but I believe some discussion for the connection and difference between these works are importance. Also, I agree this paper provide much better experimental results according to the accuracy performance than [1] for the tests over various LLMs and tasks. 

Besides the novelty of the proposed method, I'm wondering why the authors consider a quantized setup. After reading the paper, I didn't see the specific method proposed to improve the quantized training performance. My understanding is that the proposed method can be more robustness to the noise leads by the quantization, thus can provide more advantages under the quantized setup? If yes, that's totally fine. Maybe the authors could provide more explanation about this.

For the experiments, I see the authors compared the proposed method with previous PEFT methods according to accuracy. However, I'm wondering if the author could consider some experiments that directly shows the main advantages of the noise robustness as the authors states in the abstract. Maybe some experiments like adding noise on the input data, can further shows the noise robustness of the proposed method

[1] Yang, Adam X., et al. "Bayesian Low-rank Adaptation for Large Language Models.", ICLR 2024

Limited Model Diversity and datasets: Experiments focus on Llama2 and DeBERTaV3.  I would love to see results on newer models (e.g., Mistral, Llama3, Qwen) would strengthen claims about generalizability, especially since Llama2’s quantization properties are well-studied and "easier to recover". Moreover, the method is validated in combination with only one quantization approach.

1. The method introduces additional hyperparameters (scale size s and K value for covariance approximation) that require tuning for optimal performance, as evidenced by the detailed hyperparameter tables (Tables 6-7).
2. The method adds additional complexity compared to simpler approaches like LoRA, requiring distribution parameter modeling and scaled probability measure gradients, which might create implementation barriers. And it seems the paper did not have training efficiency analysis to compare with the LoRA training and full fine-tuning.
Comments, suggestions and typos: See the weakness

- I would suggest to broad a number of models, use more recent models;
- Potentially change paper focus from quantization specific to general PEFT method.

- The introduction mentions 1-bit quantization without connecting it to MeMoTune’s methodology.
- The method assumes matrices A/B are sampled from normal distributions with learnable mean/variance. No justification is provided for this choice - perhaps analyze and present weight distributions of learned Lora weights (for various models and model parts.)
- The authors assert that SOTA methods struggle on medium/large-volume datasets, but they do not validate whether MeMoTune’s performance improves with increased data.

1. Will the introduction of these additional hyperparameters cause overhead? including speed and memory overhead.
2.
Overall assessment scores: 3

3

3.5
Metareview score: 3
Metareview: The paper proposes an approach for parameter-efficient fine-tuning of quantized LLMs that models the quantized weights as Gaussian distributions with learned parameters.  The method empirically improves performance on multiple generation and understanding tasks compared to other quantized fine-tuning methods.
Summary of reasons to publish: The paper includes a solid mathematical development and analysis of the approach.

Performance is encouraging across multiple tasks and models.

Code is released for reproducibility.

The author responses to reviewers' concerns are convincing (although not all reviewers' have acknowledged the responses or updated their reviews, unfortunately).