AdaDHP: Fine-Grained Fine-Tuning via Dual Hadamard Product and Adaptive Parameter Selection
Authors: Han Liu, Changya Li, Xiaotong Zhang, Feng Zhang, Fenglong Ma, Wei Wang, Hong Yu
Accepted at: acl25

AdaDHP: Fine-Grained Fine-Tuning via Dual Hadamard Product and Adaptive Parameter Selection

Han Liu1, Changya Li1*, Xiaotong Zhang1*, Feng Zhang2, Fenglong Ma3, Wei Wang4, Hong Yu1
1Dalian University of Technology, Dalian, China, 2Peking University, Beijing, China
3The Pennsylvania State University, Pennsylvania, USA
4Shenzhen MSU-BIT University, Shenzhen, China
{liu.han.dut, lichangya.dut, zfeng.maria}@gmail.com, zxt.dut@hotmail.com, fenglong@psu.edu, ehomewang@ieee.org, hongyu@dlut.edu.cn

Abstract

With the continuously expanding parameters, efficiently adapting large language models to downstream tasks is crucial in resource-limited conditions. Many parameter-efficient fine-tuning methods have emerged to address this challenge. However, they lack flexibility, like LoRA requires manually selecting trainable parameters and rank size, (IA)3 can only scale the activations along columns, yielding inferior results due to less precise fine-tuning. To address these issues, we propose a novel method named AdaDHP with fewer parameters and finer granularity, which can adaptively select important parameters for each task. Specifically, we introduce two trainable vectors for each parameter and fine-tune the parameters through Hadamard product along both rows and columns. This significantly reduces the number of trainable parameters, with our parameter count capped at the lower limit of LoRA. Moreover, we design an adaptive parameter selection strategy to select important parameters for downstream tasks dynamically. This allows our method to flexibly remove unimportant parameters for downstream tasks. Finally, we demonstrate the superiority of our method on the T5-base model across 17 NLU tasks and on complex mathematical tasks with the Llama series models.

1 Introduction

With the advent of large language models (LLMs) (Brown et al., 2020; Touvron et al., 2023a,b), there has been a migration of various natural language processing tasks towards the pre-training and fine-tuning paradigm. The straightforward approach involves appending task-specific modules and subsequently fine-tuning all parameters for downstream tasks. However, fine-tuning LLMs with escalating number of parameters poses a challenge under resource-constrained environments. Moreover, fine-tuning each downstream task requires the retention of all weights, leading to significant resource wastage. Consequently, parameter-efficient fine-tuning (PEFT) methods have emerged as a requisite solution to address the constraints of GPU and storage resources.

The PEFT methods achieve high efficiency by fine-tuning a small subset of parameters or introducing additional trainable parameters. Adapter (Houlsby et al., 2019) is an early-stage method that appends a bottleneck layer following multi-head attention and feed-forward blocks, updating only these added parameters and keeping the original backbone fixed. While yielding promising results, it is afflicted by inference latency. Another paradigm, prompt tuning (Liu et al., 2021; Li and Liang, 2021; Lester et al., 2021), employs trainable prompts to each task and only fine-tunes these prompts. The placement of these prompts varies, resulting in diverse approaches. However, the additional prompts extend the context length, requiring the storage of larger intermediate activations during training. Consequently, these methods may demand larger GPU resources compared to full fine-tuning.

Currently, LoRA series methods (Hu et al., 2022; Zhang et al., 2023b) have become prevalent for fine-tuning LLMs. These techniques introduce trainable low-rank decomposed matrices, denoted as \( \mathbf{A} \) and \( \mathbf{B} \), for the parameters \( \mathbf{W} \), as shown in Figure 1(a). Due to the low rank of \( \mathbf{A} \) and \( \mathbf{B} \), the number of trainable parameters can be significantly decreased. Moreover, LoRA allows merging the incremental matrix \( \Delta \mathbf{W} \) with \( \mathbf{W} \) after training, addressing the issue of inference latency. However, the original LoRA method requires the manual selection of trainable parameters and rank size, thereby limiting its flexibility. Another noteworthy approach, (IA)3 (Liu et al., 2022), as shown in Figure 1(b), updates activations directly through element-wise multiplication with a vector 1 during the forward
Figure 1: Illustration of LoRA, (IA)^3, and our method AdaDHP. Blue for pretrained weights, yellow for inputs or activations, and red for additional trainable parameters. LoRA adds trainable low-rank matrices to weights, (IA)^3 applies trainable scaling to activations, while AdaDHP introduces two trainable vectors for each parameter, enabling row and column dot products.

propagation process. (IA)^3 with minimal trainable parameters achieves remarkable results. Nonetheless, it applies the same coefficient to each column of the activations and is equivalent to scale the weight matrices along column, significantly constraining the granularity of fine-tuning. Additionally, it offers lower flexibility as it fine-tunes the same activations for all tasks.

To handle the aforementioned issues simultaneously, we introduce a novel PEFT method named AdaDHP, which achieves fine-grained and flexible tuning by Dual Hadamard Product and Adaptive parameter selection. As shown in Figure 1(c), we introduce two trainable vectors for each parameter, enabling row and column dot products, respectively. Unlike (IA)^3 that only applies weight transformation on the row, we allow for different transformations on both the row and column of weight matrices. Additionally, we maintain the parameter size with the LoRA rank of 1. Secondly, we introduce a parameter importance metric and a budget allocation mechanism to adaptively select crucial parameters for downstream tasks. By pruning unimportant or redundant parameters based on their importance scores, we can enhance parameter efficiency and the flexibility of parameter selection. Finally, we validate our method on 17 NLP tasks and complex mathematical tasks, affirming its current state-of-the-art performance.

2 Related Work

The current PEFT methods can be primarily categorized into three types: partial fine-tuning, additive fine-tuning and hybrid fine-tuning methods.

2.1 Partial Fine-Tuning

Partial fine-tuning methods only update those critical parameters in pre-trained model for downstream task. For instance, the BitFit (Zaken et al., 2022) method only updates the biases and the task-specific layer, keeping major parameters unchanged. Additionally, pretrained weight masking methods (Zhao et al., 2020) use criteria such as thresholds and Fisher information (Sung et al., 2021) to evaluate the importance of parameters, which allow for only updating important weights.

2.2 Additive Fine-Tuning

Additive fine-tuning methods integrate trainable modules into pre-trained model while keeping the backbone frozen. Adapter-based approaches are early attempts. HAdapter (Houlsby et al., 2019) integrates trainable adapter modules into each block of the Transformer, and solely fine-tunes these additive modules for downstream tasks. Other adapter-based methods (He et al., 2022; Rücklé et al., 2021) optimize the integration and efficiency in distinct ways. Prompt-based approaches add specific trainable prompt tokens for each task, which are added at different locations in different methods. For instance, Prompt-tuning (Lester et al., 2021) inserts learnable prompt tokens into the model input, while Prefix-tuning (Li and Liang, 2021) adds soft prompts before the hidden states of the multi-head attention layer. However, the adapter-based methods incur inference latency, while prompt-based methods are memory-intensive.

Another widely recognized approach, LoRA (Hu et al., 2022), reduces trainable parameters by decomposing \( \mathbf{W} \) into two low-rank matrices. But
Figure 2: The framework of the AdaDHP method. MHSA indicates the mult-head self-attention block and FFN indicates the feed-forward block.

parameter allocation is not flexible because of the same rank of all parameters, some methods such as AdaLoRA (Zhang et al., 2023b) and IncreLoRA (Zhang et al., 2023a) are proposed. These methods not only optimize parameter efficiency but also enhance the flexibility and effectiveness. There are other additive fine-tuning methods, like ladder side-tuning (LST) (Sung et al., 2022) adds a side network to the model, which receives intermediate activations from the pretrained network via shortcut connections. And it only updates the parameters in the side network, so that backpropagation only passes through the side network without the backbone network, which greatly reduces the required GPU memory for training. And (IA)3 (Liu et al., 2022) introduces three learned vectors, \( l_k \), \( l_v \), and \( l_{ff} \) to rescale the key, value and feedforward intermediate activations, significantly boosting parameter efficiency.

2.3 Hybrid Fine-Tuning

Hybrid fine-tuning approaches aim to combine various PEFT techniques leveraging the strengths of each to compensate for their weaknesses. These works can be classified into Manual Combination and Automatic Combination. An example of Manual Combination methods is the MAM Adapter (He et al., 2022), which combines scaled parallel adapter and prefix-tuning. Automatic combinations, such as AutoPEFT (Zhou et al., 2023), integrate various PEFT methods like sequential adapter, parallel adapter and prefix-tuning automatically through structure search. This approach usually requires more time and cost due to optimization searches in the model or structure.

3 The Proposed Method

3.1 Preliminary

Two particular PEFT strategies currently exist: one involves modifying the weights \( \mathbf{W} \) mentioned earlier, while the other involves modifying intermediate activations. The most representative methods are LoRA and (IA)3.

LoRA. LoRA assumes delta weight is low-rank, then decomposes it into the product of two low-rank matrices, which is expressed as:

\[
\mathbf{W}' = \mathbf{W} + \Delta \mathbf{W} = \mathbf{W} + \mathbf{B}\mathbf{A},
\]

where \( \mathbf{W}, \mathbf{W}', \Delta \mathbf{W} \in \mathbb{R}^{d \times d} \), \( \mathbf{B} \in \mathbb{R}^{d \times r} \), \( \mathbf{A} \in \mathbb{R}^{r \times d} \). And the rank \( r \ll d \). Thus, the quantity of parameters required for fine-tuning \( \mathbf{W} \) reduces from \( d^2 \) to \( 2dr \), achieving efficient fine-tuning.

(IA)3. To achieve mixed-task batches, (IA)3 directly modifies activations. For the multi-head self-attention (MHSA) block, it performs element-wise multiplication on \( \mathbf{K} \) and \( \mathbf{V} \) with vectors \( l_k \) and \( l_v \), respectively, which can be represented as:

\[
\operatorname{softmax}(\frac{\mathbf{Q}(l_k^T \odot \mathbf{K})^T}{\sqrt{d}})(l_v^T \odot \mathbf{V}),
\]

where \( l_k, l_v \in \mathbb{R}^d \). For the feed-forward (FFN) block, it only performs on intermediate activations, which can be denoted as:

\[
(l_{ff}^T \odot \operatorname{ReLU}(\mathbf{X}\mathbf{W}_1 + b_1))\mathbf{W}_2 + b_2,
\]

where \( l_{ff} \in \mathbb{R}^{d_{in}} \). The total number of trainable parameters in each transformer layer is only \( 2d + d_{in} \), and (IA)3 is also highly parameter-efficient.
3.2 Dual Hadamard Product
Considering that LoRA requires to manually select rank size, whereas (IA)^3 can only scale the weight matrices along column, these limit the flexibility of fine-tuning. Hence, we propose a vector dot production strategy to fine-tune the model parameters in a more fine-grained and flexible way. The framework of our method is illustrated in Figure 2. In particular, we fine-tune \( \mathbf{W} \) by performing the element-wise dot product with both a row vector and a column vector, which can also be understood as scaling \( \mathbf{W} \) along both row and column:

\[
\mathbf{W}' = \mathbf{l}_l \odot \mathbf{W} \odot \mathbf{l}_r^T,
\]

where \( \mathbf{l}_l, \mathbf{l}_r \in \mathbb{R}^d \).

Compared to LoRA, our method can achieve better results even with rank of 1. This implies a more efficient approach in terms of parameter usage while still maintaining effectiveness in fine-tuning. It can be observed that (IA)^3 scales the weight values of each column equally, whereas our method allows for different transformations on both rows and columns of the weight matrices, thus obtaining more precise fine-tuning and achieving better outcomes.

To enhance the capacity of our approach for training complex tasks, we introduce a scaling mode of DHP during the experiments. We introduce multiple pair of \( \mathbf{l}_l \) and \( \mathbf{l}_r \) vectors for performing Hadamard product with \( \mathbf{W} \) respectively in Eq. (4), followed by averaging or summing the output. Specifically, the trainable parameters can be formally defined as \( 1 = \{ l_1^1, l_1^2, l_1^3, ..., l_1^k, l_r^1, l_r^2, ..., l_r^k \} \), with \( k \) being adjustable based on the specific task. For each pair of parameters, we perform the same operation \( \mathbf{W}^i = l_l^i \odot \mathbf{W} \odot l_r^{i^T} \). The final \( \mathbf{W}' \) is determined by the average or the sum of \( \mathbf{W}^i \).

3.3 Adaptive Parameter Selection
By observing the empirical results, LoRA proposes that only fine-tuning \( \mathbf{W}_q \) and \( \mathbf{W}_v \) can achieve relatively satisfied results, while (IA)^3 only operates on \( \mathbf{K}, \mathbf{V} \) and the intermediate activations in FFN. However, these parameters are manually selected for specific tasks and may not fit all tasks. We aim to adaptively select which parameters to be fine-tuned for downstream tasks. Currently, there are methods like AdaLoRA (Zhang et al., 2023b) that utilize importance scores to allocate different ranks to various parameters, thereby facilitating adaptive distribution. Our method, however, focuses on identifying which specific parameters should be fine-tuned. Aiming at this issue, the adaptive parameter selection strategy is divided into two main modules: Importance Score Calculation and Dynamic Allocation.

Importance Score Calculation. Since we need to remove some parameters that do not require fine-tuning, we first calculate the importance of parameters. There are many importance calculation strategies in pruning methods. Parameter magnitude (Han et al., 2015; Paganini and Forde, 2020) defines the importance of a weight based on its magnitude, but this alone cannot accurately represent the impact of a weight to the loss. Sensitivity of parameters (Ding et al., 2019; Molchanov et al., 2019; Sanh et al., 2020) is another importance metric which considers both the weight magnitude and the change rate of the loss with respect to this weight:

\[
I(\theta) = |\theta \cdot \nabla_\theta \mathcal{L}|,
\]

where \( \mathcal{L} \) and \( \theta \) represent the loss function and the trainable parameters respectively.

According to Zhang et al. (2022), Eq. (5) exhibits significant variations because of the use of mini-batch processing and the complex training process, such as dropout. To address this issue, Zhang et al. (2022) propose to smooth \( I(\theta) \) using an exponential moving average as follows:

\[
\overline{T}^{(t)} = \begin{cases}
I^{(0)} & t = 0, \\
\beta_1 \overline{T}^{(t-1)} + (1 - \beta_1) I^{(t)} & t \in [1, T),
\end{cases}
\]

where \( \beta_1 \in (0, 1) \), \( t \) and \( T \) denote the \( t \)-th step and the total trainable steps. \( \overline{T}^{(t)} \) represents the sensitivity in the \( t \)-th steps. Besides sensitivity smoothing, they also directly consider the uncertainty of importance estimation to reduce variability,

\[
U^{(t)} = |I^{(t)} - \overline{T}^{(t)}|,
\]

and they further introduce exponential weighted averaging,

\[
\overline{U}^{(t)} = \beta_2 \overline{U}^{(t-1)} + (1 - \beta_2) U^{(t)},
\]

where \( \beta_2 \in (0, 1) \). \( \overline{U}^{(t)} \) represents the uncertainty of sensitivity values at \( t \) step.

In summary, the importance score for a parameter is defined as follows:

\[
S^{(t)} = \overline{T}^{(t)} \odot \overline{U}^{(t)},
\]
Algorithm 1 AdaDHP
1: Input: dataset \( \mathcal{D} \); total steps \( T \); start steps \( t_i \); final steps \( t_f \); hyperparameters \( \beta_1, \beta_2 \).
2: Add trainable parameters \( l_l, l_r \) for each \( \{\mathbf{W}\} \);
3: for \( t = 1, \ldots, T \) do
4:   Sample a mini-batch from \( \mathcal{D} \);
5:   Compute the gradient \( \nabla \mathcal{L}(l_l, l_r) \);
6:   Compute the sensitivity \( I^{(t)}, \overline{I}^{(t)} \) by Eqs. (5), (6) for every parameter \( \{l_l, l_r\} \);
7:   Compute the uncertainty \( U^{(t)}, \overline{U}^{(t)} \) by Eqs. (7), (8) for every parameter \( \{l_l, l_r\} \);
8:   Compute the importance score \( S^{(t)} \) by Eq. (9) for every parameter \( \{l_l, l_r\} \);
9:   Average the importance score of \( l_l \) and \( l_r \) as final importance score of \( \mathbf{W} \);
10:  Compute the budget \( b^{(t)} \) by Eq. (10);
11:  Retain the top \( b^{(t)} \) important parameters;
12: end for
13: Output: Fine-tuned parameters \( \{l_l^{(T)}, l_r^{(T)}\} \).

where \( \odot \) is Hadamard product. This takes into account the sensitivity and uncertainty of the parameter, resulting in a more stable importance score.

Dynamic Allocation. Using the above importance score calculation strategy, we can obtain the importance of each parameter at each step. For our approach, we only introduce two additional parameters \( l_l \) and \( l_r \) for each \( \mathbf{W} \), so we treat \( l_l \) and \( l_r \) as a combined entity, using the average of all values from these two parameters to represent their importance. This measurement guides whether to remove or keep these two parameters, thus determining whether to fine-tune \( \mathbf{W} \).

One intuitive observation is that during the initial training phase when stability is low, the importance of parameters is also less stable. Therefore, it is necessary to have a warm-up period at the beginning. Moreover, in the early stages, more parameters should be pruned, and as training progresses and stability increases, fewer parameters should be pruned to maintain stable training. Following Zhang et al. (2023b), the quantity of parameters required to retain in the \( t \)-th step is formulated as:

\[
b^{(t)} = \begin{cases}
b^{(0)} & t \in [0, t_i) \\
b^{(T)} + \alpha (\frac{t_f - t}{t_f - t_i})^3 & t \in [t_i, t_f) \\
b^{(T)} & t \in [t_f, T]
\end{cases}
\]

where \( b^{(0)} \) and \( b^{(T)} \) are initial and final parameter quantity, the removed trainable parameter quantity is \( \alpha = b^{(0)} - b^{(T)} \), \( t_i \) and \( t_f \) are start and end dynamic allocation steps, respectively, and \( T \) is the total number of steps to update the model parameters. Any monotonic exponential function can achieve this functionality.

3.4 The Overall Algorithm

We summarize our method AdaDHP in Algorithm 1. First, we add a pair of trainable parameters \( l_l, l_r \) for each parameter. In each step, we first perform backpropagation to compute the gradient of every trainable parameter. Then, we calculate the sensitivity and uncertainty of each parameter according to Eqs. (5), (6), (7), and (8). Next, we calculate the importance of each parameter based on Eq. (9). We treat the \( l_l \) and \( l_r \) for each parameter \( \mathbf{W} \) as a whole and compute the average importance score. Finally, after using Eq. (10) to calculate the budget \( b^{(t)} \) in the \( t \)-th step, we retain the top \( b^{(t)} \) important parameters and set the values in other parameters to 1. After \( T \) steps, we obtain the fine-tuned models.

4 Experiments

4.1 Datasets

For T5-base model, following previous work (Asai et al., 2022), we perform experiments on 17 NLP tasks: (1) GLUE benchmark (Wang et al., 2018) includes CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI and RTE. (2) SuperGLUE benchmark (Wang et al., 2019) includes MultiRC, BoolQ, WiC, WSC and CB. (3) Other datasets include WinoGrande (Sakaguchi et al., 2020), Yelp-2 (Zhang et al., 2015), SciTail (Khot et al., 2018) and PAWS-Wiki (Zhang et al., 2019). Detailed dataset statistics can be found in Table 6 of Appendix.

For all the datasets, we follow (Sung et al., 2022) and (Asai et al., 2022) to split them into training, validation and test sets. Specifically, for MNLI, we use mismatched validation set as validation set and matched validation set as test set. For the datasets with relatively small amounts of samples (RTE, MPRC, STS-B, CoLA, SciTail, PAWS-Wiki and all the datasets of SuperGLUE benchmark), we split the validation set into two equal-sized subsets as validation and test sets. For the remaining datasets (QQP, QNLI, SST-2, WinoGrande and Yelp-2), we split 1k samples from the training set as the new validation set and use the original validation set as test set. We limit the maximum size of training set to 100k for Yelp-2.
<table>
  <tr>
    <th>Method</th>
    <th># Params</th>
    <th>CoLA<br>Mc</th>
    <th>STS-B<br>Pc</th>
    <th>MRPC<br>Acc</th>
    <th>RTE<br>Acc</th>
    <th>QNLI<br>Acc</th>
    <th>SST-2<br>Acc</th>
    <th>QQP<br>Acc</th>
    <th>MNLI<br>Acc</th>
    <th>All<br>Avg.</th>
  </tr>
  <tr>
    <td>Full Fine-tuning</td>
    <td>220M</td>
    <td>61.8</td>
    <td>89.7</td>
    <td>90.2</td>
    <td>71.9</td>
    <td>93.0</td>
    <td>94.6</td>
    <td>91.6</td>
    <td>86.8</td>
    <td>84.9</td>
  </tr>
  <tr>
    <td>Adapter</td>
    <td>1.9M</td>
    <td>64.0</td>
    <td>90.7</td>
    <td>85.3</td>
    <td>71.9</td>
    <td>93.2</td>
    <td>93.8</td>
    <td>90.2</td>
    <td><b>86.5</b></td>
    <td>84.5</td>
  </tr>
  <tr>
    <td>AdapterDrop</td>
    <td>1.1M</td>
    <td>62.7</td>
    <td><b>91.4</b></td>
    <td>86.3</td>
    <td>71.2</td>
    <td>93.2</td>
    <td>93.6</td>
    <td>90.2</td>
    <td>86.3</td>
    <td>84.4</td>
  </tr>
  <tr>
    <td>BitFit</td>
    <td>280K</td>
    <td>58.2</td>
    <td>90.9</td>
    <td>86.8</td>
    <td>67.6</td>
    <td>93.0</td>
    <td>94.2</td>
    <td>90.1</td>
    <td>85.3</td>
    <td>83.3</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>3.8M</td>
    <td>63.3</td>
    <td>91.0</td>
    <td>88.2</td>
    <td>75.5</td>
    <td>93.2</td>
    <td>94.3</td>
    <td>90.4</td>
    <td>86.3</td>
    <td>85.3</td>
  </tr>
  <tr>
    <td>LST</td>
    <td>3.8M</td>
    <td>58.1</td>
    <td>90.5</td>
    <td>87.9</td>
    <td>71.9</td>
    <td>93.3</td>
    <td>94.1</td>
    <td>90.4</td>
    <td>85.6</td>
    <td>84.0</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>76.8K</td>
    <td>10.6</td>
    <td>89.5</td>
    <td>68.1</td>
    <td>54.7</td>
    <td>92.8</td>
    <td>90.9</td>
    <td>89.7</td>
    <td>81.3</td>
    <td>72.2</td>
  </tr>
  <tr>
    <td>(IA)<sup>3</sup></td>
    <td>129K</td>
    <td>60.3<sub>±0.7</sub></td>
    <td>91.2<sub>±0.2</sub></td>
    <td><b>91.0<sub>±0.2</sub></b></td>
    <td>73.6<sub>±0.9</sub></td>
    <td>93.2<sub>±0.1</sub></td>
    <td>94.0<sub>±0.2</sub></td>
    <td>90.2<sub>±0.2</sub></td>
    <td>85.8<sub>±0.0</sub></td>
    <td>84.9</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>811K</td>
    <td>62.9<sub>±0.5</sub></td>
    <td>91.3<sub>±0.0</sub></td>
    <td>88.6<sub>±0.2</sub></td>
    <td>75.3<sub>±0.9</sub></td>
    <td><b>93.4<sub>±0.1</sub></b></td>
    <td>94.2<sub>±0.2</sub></td>
    <td>90.2<sub>±0.0</sub></td>
    <td>86.0<sub>±0.0</sub></td>
    <td>85.2</td>
  </tr>
  <tr>
    <td>AsymmetryLoRA</td>
    <td>442K</td>
    <td>53.4<sub>±2.5</sub></td>
    <td>90.2<sub>±0.3</sub></td>
    <td>89.5<sub>±0.6</sub></td>
    <td>71.2<sub>±3.1</sub></td>
    <td>92.7<sub>±0.1</sub></td>
    <td>93.5<sub>±0.2</sub></td>
    <td>89.9<sub>±0.0</sub></td>
    <td>84.4<sub>±0.1</sub></td>
    <td>83.1</td>
  </tr>
  <tr>
    <td>FourierFT</td>
    <td>72K</td>
    <td>58.6<sub>±1.7</sub></td>
    <td>90.5<sub>±0.4</sub></td>
    <td>86.9<sub>±1.4</sub></td>
    <td>71.9<sub>±1.2</sub></td>
    <td>92.9<sub>±0.1</sub></td>
    <td>94.4<sub>±0.2</sub></td>
    <td>90.3<sub>±0.1</sub></td>
    <td>85.6<sub>±0.0</sub></td>
    <td>83.9</td>
  </tr>
  <tr>
    <td>qGOFT</td>
    <td>810K</td>
    <td>56.1<sub>±2.1</sub></td>
    <td>91.3<sub>±0.2</sub></td>
    <td>88.7<sub>±1.0</sub></td>
    <td>71.2<sub>±1.0</sub></td>
    <td>93.3<sub>±0.1</sub></td>
    <td><b>94.5<sub>±0.2</sub></b></td>
    <td>90.5<sub>±0.1</sub></td>
    <td>86.1<sub>±0.0</sub></td>
    <td>84.0</td>
  </tr>
  <tr>
    <td>DoRA</td>
    <td>1.8M</td>
    <td>61.6<sub>±2.1</sub></td>
    <td>89.9<sub>±0.2</sub></td>
    <td>87.4<sub>±1.6</sub></td>
    <td>74.1<sub>±1.2</sub></td>
    <td>92.9<sub>±0.1</sub></td>
    <td>94.2<sub>±0.2</sub></td>
    <td><b>91.1<sub>±0.1</sub></b></td>
    <td>84.5<sub>±0.0</sub></td>
    <td>84.5</td>
  </tr>
  <tr>
    <td>AdaDHP</td>
    <td>203K</td>
    <td><b>64.3<sub>±2.6</sub></b></td>
    <td><b>91.4<sub>±0.1</sub></b></td>
    <td>90.5<sub>±0.8</sub></td>
    <td><b>77.7<sub>±1.0</sub></b></td>
    <td><b>93.4<sub>±0.0</sub></b></td>
    <td><b>94.5<sub>±0.2</sub></b></td>
    <td>90.6<sub>±0.0</sub></td>
    <b>86.5<sub>±0.0</sub></b>
    <td><b>86.1</b></td>
  </tr>
</table>

Table 1: Test results on GLUE benchmark, with the corresponding size of trainable parameters. All the results are based on the T5-base model. We use Matthew’s Correlation and Pearson Correlation as the metrics for CoLA and STS-B, respectively, and accuracy for other tasks.

To verify the scalability of our model on larger models and complex tasks, we train Llama-7B and Llama2-7B models on Math10K dataset (Hu et al., 2023) and test on six different math reasoning datasets: MultiArith (Roy and Roth, 2016), GSM8K (Cobbe et al., 2021), AddSub (Hosseini et al., 2014), AQuA (Ling et al., 2017), SingleEq (Koncel-Kedziorski et al., 2015) and SVAMP (Patel et al., 2021).

4.2 Baselines

We compare our method with the following strong baselines: (1) Full Fine-tuning fine-tunes all parameters of the backbone for downstream tasks. (2) Adapter (Houlsby et al., 2019) inserts bottleneck architecture after the attention and FFN modules. (3) AdapterDrop (Rücklé et al., 2021) removes adapters from lower transformer layers, enabling more efficiency during training and inference. (4) BitFit (Zaken et al., 2022) merely fine-tunes the biases in the pre-trained model. (5) LoRA (Hu et al., 2022) decomposes the parameters into two low-rank matrices and only fine-tunes them. (6) Ladder Side-Tuning (LST) (Sung et al., 2022) employs a side network that receives intermediate activations through shortcut connections, and only fine-tunes the side network. (7) Prompt Tuning (PT) (Lester et al., 2021) inserts learnable prompt tokens into the model input. (8) (IA)<sup>3</sup> (Liu et al., 2022) fine-tunes intermediate activations, which introduces rescaling vectors for the keys and values in the attention architecture and for the intermediate activation in the FFN architecture. (9) AdaLoRA (Zhang et al., 2023b) dynamically assigns different ranks to each parameter based on the importance metrics. (10) AsymmetryLoRA (Zhu et al., 2024) explores the different role of A and B in LoRA, and proposes many initialization ways to get considerable results. (11) FourierFT (Gao et al., 2024) treat \( \Delta W \) as matrix in the spatial domain, learns only its spectral coefficients, and use IDFT to recover \( \Delta W \). (12) qGOFT (Ma et al., 2024) designs given-based orthogonal finetuning method with fewer parameters. (13) DoRA (Liu et al., 2024) divides the pretrained weights to magnitude and direction components, and use LoRA for directional updates.

4.3 Implementation Details

We conduct experiment on the T5-base (Raffel et al., 2020), Llama-7B (Touvron et al., 2023a) and Llama2-7B (Touvron et al., 2023b) models to evaluate the effectiveness of our method. We select hyperparameters based on the validation set for our method, (IA)<sup>3</sup> and AdaLoRA, and report averaged results over 3 different runs. All the results of other baselines are sourced from (Asai et al., 2022) and (Sung et al., 2022). For the T5-base model on the GLUE benchmark, we set batch size as 100 and maximum sequence length as 128, while setting batch size as 16 and maximum sequence length as 256 for the other datasets. The other hyperparameters are provided in Table 5 of Appendix.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2"># Params</th>
    <th colspan="5">SuperGLUE</th>
    <th colspan="4">Other Datasets</th>
  </tr>
  <tr>
    <th>BoolQ</th>
    <th>WiC</th>
    <th>WSC</th>
    <th>CB</th>
    <th>MultiRC</th>
    <th>Avg.</th>
    <th>WG</th>
    <th>Yelp</th>
    <th>SciTail</th>
    <th>PAWS</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Full Fine-tuning</td>
    <td>220M</td>
    <td>81.1</td>
    <td>70.2</td>
    <td>59.6</td>
    <td>85.7</td>
    <td>72.8</td>
    <td>73.9</td>
    <td>61.9</td>
    <td>96.7</td>
    <td>95.8</td>
    <td>94.1</td>
    <td>87.1</td>
  </tr>
  <tr>
    <td>Adapter</td>
    <td>1.9M</td>
    <td>82.5</td>
    <td>67.1</td>
    <td>67.3</td>
    <td>85.7</td>
    <td><b>75.9</b></td>
    <td>75.7</td>
    <td><b>59.2</b></td>
    <td>96.9</td>
    <td>94.5</td>
    <td>94.3</td>
    <td>86.2</td>
  </tr>
  <tr>
    <td>BitFit</td>
    <td>280K</td>
    <td>79.6</td>
    <td>70.0</td>
    <td>59.6</td>
    <td>78.6</td>
    <td>74.5</td>
    <td>72.5</td>
    <td>57.2</td>
    <td>94.7</td>
    <td>94.7</td>
    <td>92.0</td>
    <td>84.7</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>3.8M</td>
    <td>81.3</td>
    <td>68.3</td>
    <td>67.3</td>
    <td>89.3</td>
    <td>72.6</td>
    <td>75.8</td>
    <td>58.2</td>
    <td>97.1</td>
    <td>94.7</td>
    <td>94.0</td>
    <td>86.0</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>76.8K</td>
    <td>61.7</td>
    <td>48.9</td>
    <td>51.9</td>
    <td>67.9</td>
    <td>58.7</td>
    <td>57.8</td>
    <td>49.6</td>
    <td>95.1</td>
    <td>87.9</td>
    <td>55.8</td>
    <td>72.1</td>
  </tr>
  <tr>
    <td>(IA)<sup>3</sup></td>
    <td>129K</td>
    <td>82.2</td>
    <td>69.1</td>
    <td>66.7</td>
    <td>90.5</td>
    <td>73.2</td>
    <td>76.3</td>
    <td>57.7</td>
    <td>97.1</td>
    <td>95.2</td>
    <td>93.9</td>
    <td>86.0</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>811K</td>
    <td>82.1</td>
    <td>68.7</td>
    <td>66.7</td>
    <td>89.3</td>
    <td>73.3</td>
    <td>76.0</td>
    <td>58.5</td>
    <td>97.2</td>
    <td>96.0</td>
    <td>94.3</td>
    <td>86.5</td>
  </tr>
  <tr>
    <td>AsymmetryLoRA</td>
    <td>442K</td>
    <td>80.0</td>
    <td>65.7</td>
    <td>66.7</td>
    <td>81.0</td>
    <td>72.4</td>
    <td>73.2</td>
    <td>58.3</td>
    <td>96.6</td>
    <td>92.2</td>
    <td>91.0</td>
    <td>84.5</td>
  </tr>
  <tr>
    <td>FourierFT</td>
    <td>72K</td>
    <td>80.1</td>
    <td>66.2</td>
    <td>63.5</td>
    <td>83.3</td>
    <td>72.2</td>
    <td>73.1</td>
    <td>57.1</td>
    <td>96.8</td>
    <td>93.3</td>
    <td>92.4</td>
    <td>84.9</td>
  </tr>
  <tr>
    <td>qGOFT</td>
    <td>810K</td>
    <td>81.8</td>
    <td>66.1</td>
    <td>67.3</td>
    <td>86.9</td>
    <td>73.2</td>
    <td>75.1</td>
    <td>58.5</td>
    <td>96.7</td>
    <td>94.3</td>
    <td>94.1</td>
    <td>85.9</td>
  </tr>
  <tr>
    <td>DoRA</td>
    <td>1.8M</td>
    <td>81.6</td>
    <td>69.5</td>
    <td>67.3</td>
    <td>88.1</td>
    <td>73.3</td>
    <td>76.0</td>
    <td>59.0</td>
    <td>96.9</td>
    <td>95.8</td>
    <td>94.2</td>
    <td>86.5</td>
  </tr>
  <tr>
    <td><b>AdaDHP</b></td>
    <td><b>203K</b></td>
    <td><b>82.6</b></td>
    <td><b>70.2</b></td>
    <td><b>67.9</b></td>
    <td><b>91.7</b></td>
    <td><b>73.4</b></td>
    <td><b>77.1</b></td>
    <td><b>59.1</b></td>
    <td><b>97.3</b></td>
    <td><b>96.2</b></td>
    <td><b>94.6</b></td>
    <td><b>86.8</b></td>
  </tr>
</table>

Table 2: Test results on SuperGLUE benchmark and other datasets. All the results are based on the T5-base model. We use F1 for MultiRC, and accuracy for other tasks as evaluation metrics.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th rowspan="2">Variable</th>
    <th colspan="10">Llama-7B</th>
  </tr>
  <tr>
    <th>MultiArith</th>
    <th>GSM8K</th>
    <th>AddSub</th>
    <th>AQuA</th>
    <th>SingleEq</th>
    <th>SVAMP</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="4"><b>Llama-7B</b></td>
    <td rowspan="4"><b>LoRA</b></td>
    <td>r = 4</td>
    <td>95.2</td>
    <td>34.7</td>
    <td>78.5</td>
    <td>16.1</td>
    <td>77.8</td>
    <td>45.4</td>
    <td>57.9</td>
  </tr>
  <tr>
    <td>r = 8</td>
    <td>96.2</td>
    <td>35.6</td>
    <td>80.5</td>
    <td>15.7</td>
    <td>82.3</td>
    <td>49.6</td>
    <td>60.0</td>
  </tr>
  <tr>
    <td>r = 16</td>
    <td>95.5</td>
    <td>36.2</td>
    <td>82.8</td>
    <td>13.8</td>
    <td>84.4</td>
    <td>50.9</td>
    <td>60.6</td>
  </tr>
  <tr>
    <td>r = 32</td>
    <td>95.0</td>
    <td>37.5</td>
    <td>83.3</td>
    <td>18.9</td>
    <td>84.4</td>
    <td>52.1</td>
    <td>61.9</td>
  </tr>
  <tr>
    <td rowspan="4"><b>Llama-7B</b></td>
    <td rowspan="4"><b>AdaDHP</b></td>
    <td>k = 1</td>
    <td>94.3</td>
    <td>30.4</td>
    <td>79.5</td>
    <td>19.1</td>
    <td>77.2</td>
    <td>41.1</td>
    <td>56.9</td>
  </tr>
  <tr>
    <td>k = 4</td>
    <td>97.5</td>
    <td>36.2</td>
    <td>83.5</td>
    <td>16.9</td>
    <td>81.5</td>
    <td>46.1</td>
    <td>60.3</td>
  </tr>
  <tr>
    <td>k = 8</td>
    <td>94.0</td>
    <td>37.5</td>
    <td>84.1</td>
    <td>16.1</td>
    <td>84.3</td>
    <td>49.4</td>
    <td>60.9</td>
  </tr>
  <tr>
    <td>k = 16</td>
    <td>95.0</td>
    <td>36.9</td>
    <td>84.1</td>
    <td>18.1</td>
    <td>86.2</td>
    <td>50.9</td>
    <td>61.9</td>
  </tr>
  <tr>
    <td rowspan="4"><b>Llama2-7B</b></td>
    <td rowspan="4"><b>LoRA</b></td>
    <td>r = 4</td>
    <td>95.1</td>
    <td>39.3</td>
    <td>83.0</td>
    <td>19.7</td>
    <td>82.1</td>
    <td>47.6</td>
    <td>61.1</td>
  </tr>
  <tr>
    <td>r = 8</td>
    <td>94.3</td>
    <td>39.7</td>
    <td>82.5</td>
    <td>20.9</td>
    <td>85.0</td>
    <td>47.4</td>
    <td>61.6</td>
  </tr>
  <tr>
    <td>r = 16</td>
    <td>94.5</td>
    <td>41.5</td>
    <td>84.1</td>
    <td>19.3</td>
    <td>85.8</td>
    <td>50.9</td>
    <td>62.7</td>
  </tr>
  <tr>
    <td>r = 32</td>
    <td>94.5</td>
    <td>39.3</td>
    <td>83.5</td>
    <td>19.3</td>
    <td>86.2</td>
    <td>47.5</td>
    <td>61.7</td>
  </tr>
  <tr>
    <td rowspan="4"><b>Llama2-7B</b></td>
    <td rowspan="4"><b>AdaDHP</b></td>
    <td>k = 1</td>
    <td>96.2</td>
    <td>37.1</td>
    <td>83.8</td>
    <td>18.1</td>
    <td>83.5</td>
    <td>44.0</td>
    <td>60.5</td>
  </tr>
  <tr>
    <td>k = 4</td>
    <td>94.7</td>
    <td>35.6</td>
    <td>83.5</td>
    <td>18.9</td>
    <td>84.3</td>
    <td>47.1</td>
    <td>60.7</td>
  </tr>
  <tr>
    <td>k = 8</td>
    <td>92.0</td>
    <td>42.0</td>
    <td>84.6</td>
    <td>19.7</td>
    <td>86.0</td>
    <td>52.5</td>
    <td>62.8</td>
  </tr>
  <tr>
    <td>k = 16</td>
    <td>94.2</td>
    <td>39.7</td>
    <td>85.1</td>
    <td>20.1</td>
    <td>83.7</td>
    <td>49.7</td>
    <td>62.1</td>
  </tr>
</table>

Table 3: Test results of six math datasets on Llama-7B and Llama2-7B models. We use accuracy for all tasks as evaluation metrics.

4.4 Experimental Results

4.4.1 Results on the T5-base Model

Table 1 presents our results on the GLUE benchmark using the T5-base model. Compared with other PEFT methods, our method AdaDHP outperforms other PEFT methods on 6 out of 8 datasets and surpasses Full Fine-tuning on 5 datasets. Additionally, the average performance of AdaDHP exceeds that of both Full Fine-tuning and other baselines. Despite fine-tuning only 0.05% of the parameters compared to our strong baseline LoRA, our method delivers superior results across all datasets. These findings demonstrate that AdaDHP achieves superior performance while maintaining higher parameter efficiency compared to other methods.

Table 2 shows our results on the SuperGLUE benchmark and other datasets based on the T5-base model. On the SuperGLUE benchmark, our method yields the best performance across 4 out of 5 datasets compared to all other baselines. Additionally, our method outperforms Full Fine-tuning across all the datasets. Regarding the other four datasets, our method outperforms Full Fine-tuning and the other baselines on three datasets. These
<table>
  <tr>
    <th>Method</th>
    <th>CoLA</th>
    <th>STS-B</th>
    <th>MRPC</th>
    <th>RTE</th>
    <th>QNLI</th>
    <th>SST-2</th>
    <th>QQP</th>
    <th>MNLI</th>
  </tr>
  <tr>
    <td>AdaDHP</td>
    <td>64.3±2.6</td>
    <td>91.4±0.1</td>
    <td>90.5±0.8</td>
    <td>77.7±1.0</td>
    <td>93.4±0.0</td>
    <td>94.5±0.2</td>
    <td>90.6±0.0</td>
    <td>86.5±0.0</td>
  </tr>
  <tr>
    <td>-Adaptive</td>
    <td>62.3±0.4</td>
    <td>91.4±0.2</td>
    <td>88.2±1.8</td>
    <td>73.6±0.3</td>
    <td>93.3±0.1</td>
    <td>94.2±0.3</td>
    <td>90.4±0.2</td>
    <td>86.2±0.1</td>
  </tr>
  <tr>
    <td>-Adaptive<sub>column</sub></td>
    <td>61.9±1.3</td>
    <td>90.9±0.1</td>
    <td>87.9±1.0</td>
    <td>72.2±0.3</td>
    <td>93.3±0.0</td>
    <td>94.0±0.1</td>
    <td>90.3±0.0</td>
    <td>85.7±0.1</td>
  </tr>
  <tr>
    <td>-Adaptive<sub>row</sub></td>
    <td>60.6±1.9</td>
    <td>91.1±0.0</td>
    <td>88.7±1.1</td>
    <td>75.3±0.9</td>
    <td>93.3±0.1</td>
    <td>92.4±0.5</td>
    <td>90.4±0.2</td>
    <td>86.2±0.2</td>
  </tr>
  <tr>
    <td>AdaDHP<sub>magnitude</sub></td>
    <td>61.6±0.5</td>
    <td>91.0±0.1</td>
    <td>89.4±1.0</td>
    <td>73.9±1.8</td>
    <td>93.2±0.0</td>
    <td>91.2±0.2</td>
    <td>90.3±0.0</td>
    <td>86.0±0.1</td>
  </tr>
  <tr>
    <td>AdaDHP<sub>sensitivity</sub></td>
    <td>61.6±2.1</td>
    <td>91.3±0.1</td>
    <td>89.5±2.0</td>
    <td>74.6±0.9</td>
    <td>93.2±0.1</td>
    <td>90.5±0.5</td>
    <td>90.5±0.1</td>
    <td>86.5±0.1</td>
  </tr>
</table>

Table 4: Ablation study on the GLUE benchmark using the T5-base model. -Adaptive indicates the removal of the adaptive parameter selection module. -Adaptive<sub>column</sub> and -Adaptive<sub>row</sub> refer to removing the adaptive module while scaling only the column parameters or the row parameters. AdaDHP<sub>magnitude</sub> and AdaDHP<sub>sensitivity</sub> replace the importance selection strategy with magnitude and sensitivity methods, respectively.

![Importance scores of T5-base model parameters. Four heatmaps show attention weights for encoder and decoder layers.](page_1012_1012_408_408.png)

Figure 3: The importance scores of T5-base model parameters. MHSA, MHCA, and FFN represent multi-head self-attention, multi-head cross-attention, and the feed-forward module.

observations once again demonstrate the high parameter efficiency of our method.

4.4.2 Results on Llama Series Models

Table 3 presents our results using the Llama-7B and Llama2-7B models. Among them, \( k = 1 \) is the basic mode of our method AdaDHP, and we also explore the impact of scaling with larger values of \( k \). From the results, we can observe that our method at \( k = 1 \) shows comparable performance to LoRA at \( r = 4 \) on both two models. And when \( k \) increases, our results consistently improve and are also superior to the results of all baselines under the same parameter settings (\( r = k \)). This trend highlights the scalability and adaptability of our method, and demonstrates the effectiveness of our method on complex tasks and large models.

4.4.3 Ablation Study

Table 4 shows the results of our method when the adaptive parameter selection module is abandoned, denoted as “-Adaptive”. It can be observed that the results of “-Adaptive” are much worse than AdaDHP, especially there is a significant decrease in performance for CoLA, MRPC, RTE, WiC, WSC, and CB. The reason is that these datasets are relatively small, and using a larger number of parameters can lead to overfitting. Therefore,
the performance is improved when removing the unimportant or redundant parameters by this module. Compared with -Adaptive, -Adaptive_{column} and Adaptive_{row} which scale the parameters along either rows or columns generally performs much worse. It can be seen that scaling only the rows or only the columns leads to a significant performance decrease in both cases, demonstrating the importance of fine-grained fine-tuning via dual hadamard product.

Additionally, we conduct experiments using commonly applied importance scoring methods, such as the magnitude and the sensitivity. The results are shown in Table 4. The results indicate that our method outperforms these two variants. Our importance calculation method employs an exponential moving average, which provides a more stable and reliable importance score. Therefore, the retained parameters are more impactful, leading to better overall performance.

4.4.4 Visualization

To validate that the trainable parameters selected by LoRA and (IA)^3 are not flexible due to manual selection, we take the RTE dataset as an example, and use the importance score calculation module to compute the importance for each parameter of the T5-base model. Figure 3 shows the normalized parameter importance scores of Full Fine-tuning and AdaDHP, the parameters with the scores larger than 0 are the important parameters required to be fine-tuned. It can be seen that the selected parameters of AdaDHP are generally consistent with Full Fine-tuning, i.e., all the parameters in the encoder and the cross-attention parameters of the last few layers in the decoder are important. Whereas LoRA proposes to only fine-tune \( W_q \) and \( W_v \), while (IA)^3 is equivalent to fine-tuning \( W_k \), \( W_v \) and \( W_2 \), leaving many important parameters unfine-tuned. This observation demonstrates that AdaDHP can precisely select and fine-tune important parameters.

5 Conclusion

In this paper, we propose an innovative parameter-efficient fine-tuning method called AdaDHP for fine-tuning large language models. To increase the granularity of fine-tuning parameters and significantly reduce the number of training parameters, we perform Hadamard Product on each parameter along both rows and columns. To adaptively select parameters for downstream tasks, we use an importance score calculation strategy and an dynamic allocation mechanism to select important parameters. We conduct extensive experiments on 17 NLP datasets with T5-base model and complex mathematical tasks with Llama series models, validating the superiority of our method.

Limitations

Our approach currently prioritizes achieving a good balance between parameter count and performance, but it does not yet address excessive memory usage. Additionally, some hyperparameters still require fine-tuning. The limitations of this study are as follows: (1) In future work, we plan to explore quantization techniques for the base model, similar to the approach in QLoRA (Dettmers et al., 2023), to reduce memory usage and enable more parameter-efficient fine-tuning. (2) We will also focus on developing additional strategies to optimize budget allocation in the Adaptive Parameter Selection module, while reducing the number of hyperparameters in this module to simplify the tuning process and improve efficiency.

Acknowledgments

This work was supported by National Natural Science Foundation of China (No. 62206038, 62106035), Liaoning Binhai Laboratory Project (No. LBLF-2023-01), Chunhui Project Foundation of the Education Department of China (No. HZKY20220419), and Xiaomi Young Talents Program.

References

Akari Asai, Mohammadreza Salehi, Matthew E. Peters, and Hannaneh Hajishirzi. 2022. ATTEMPT: parameter-efficient multi-task tuning via attentional mixtures of soft prompts. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6655–6672.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Conference on Neural Information Processing Systems (NeurIPS).
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv, abs/2110.14168.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. In Conference on Neural Information Processing Systems (NeurIPS).

Xiaohan Ding, Guiguang Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, and Ji Liu. 2019. Global sparse momentum SGD for pruning very deep neural networks. In Conference on Neural Information Processing Systems (NeurIPS), pages 6379–6391.

Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. 2024. Parameter-efficient fine-tuning with discrete fourier transform. In International Conference on Machine Learning (ICML).

Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural network. In Conference on Neural Information Processing Systems (NeurIPS), pages 1135–1143.

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations (ICLR).

Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523–533.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (ICML), pages 2790–2799.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR).

Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Popria, and Roy Ka-Wei Lee. 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5254–5276.

Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A textual entailment dataset from science question answering. In AAAI Conference on Artificial Intelligence (AAAI), pages 5189–5197.

Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3.

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3045–3059.

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4582–4597.

Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 158–167.

Haokun Liu, Derek Tam, Mohammed Mueqeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Conference on Neural Information Processing Systems (NeurIPS).

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. In International Conference on Machine Learning (ICML).

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv, abs/2103.10385.

Xinyu Ma, Xu Chu, Zhibang Yang, Yang Lin, Xin Gao, and Junfeng Zhao. 2024. Parameter efficient quasi-orthogonal fine-tuning via givens rotation. In International Conference on Machine Learning (ICML).

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019. Importance estimation for neural network pruning. In Computer Vision and Pattern Recognition (CVPR), pages 11264–11272.

Michela Paganini and Jessica Zosa Forde. 2020. On iterative neural network pruning, reinitialization, and the similarity of masks. arXiv, abs/2001.05050.

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In North American Chapter of the Association for Computational Linguistics (NAACL), pages 2080–2094.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21.
Subhro Roy and Dan Roth. 2016. Solving general arithmetic word problems. arXiv, abs/1608.01413.

Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. Adapterdrop: On the efficiency of adapters in transformers. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7930–7946.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI Conference on Artificial Intelligence (AAAI), pages 8732–8740.

Victor Sanh, Thomas Wolf, and Alexander M. Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. In Conference on Neural Information Processing Systems (NeurIPS).

Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. LST: ladder side-tuning for parameter and memory efficient transfer learning. In Conference on Neural Information Processing Systems (NeurIPS).

Yi-Lin Sung, Varun Nair, and Colin Raffel. 2021. Training neural networks with fixed sparse masks. In Conference on Neural Information Processing Systems (NeurIPS), pages 24193–24205.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. arXiv, abs/2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyu Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molodybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodríguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv, abs/2307.09288.

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Conference on Neural Information Processing Systems (NeurIPS), pages 3261–3275.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 353–355.

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1–9.

Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. 2023a. Incrclora: Incremental parameter allocation method for parameter-efficient fine-tuning. arXiv, abs/2308.12043.

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023b. Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Representations (ICLR).

Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2022. PLATON: pruning large transformer models with upper confidence bound of weight importance. In International Conference on Machine Learning (ICML), pages 26809–26823.

Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Conference on Neural Information Processing Systems (NeurIPS), pages 649–657.

Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: paraphrase adversaries from word scrambling. In North American Chapter of the Association for Computational Linguistics (NAACL), pages 1298–1308. Association for Computational Linguistics.

Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. 2020. Masking as an efficient alternative to finetuning for pretrained language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2226–2241.

Han Zhou, Xingchen Wan, Ivan Vulic, and Anna Korhonen. 2023. Autopeft: Automatic configuration search for parameter-efficient fine-tuning. arXiv, abs/2301.12132.

Jiacheng Zhu, Kristjan H. Greenewald, Kimia Nadjiha, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielson, Leshem Choshen, Marzeyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. 2024. Asymmetry in low-rank adapters of foundation models. In International Conference on Machine Learning (ICML).
A Datasets

Table 6 summarizes the relevant information of the training, validation and test sets for the 17 NLP tasks we used. As mentioned in the main paper, for datasets with relatively small samples, such as RTE, MPRC, STS-B, CoLA, SciTail, PAWS-Wiki, and all the datasets from the SuperGLUE benchmark, we divide the validation set into two equal-sized subsets, one serving as the new validation set and the other as the test set. Conversely, for the remaining datasets including QQP, QNLI, SST-2, WinoGrande, and Yelp-2, we extract 1,000 samples from the original training set to constitute a new validation set, while retaining the original validation set as the test set. Additionally, for Yelp-2, we cap the training set at 100,000 samples.

B Hyperparameter Investigation

![Line plot showing results on RTE while retaining different parameter quantities.](page_340_682_495_246.png)

Figure 4: Results on RTE while retaining different parameter quantities.

Besides the number of epochs and learning rate selected based on the validation set, the hyperparameters of our method include \( \beta_1, \beta_2, b^{(0)}, b^{(T)}, t_i \) and \( t_f \). We follow (Zhang et al., 2022) to set \( \beta_1 \) and \( \beta_2 \). \( b^{(0)} \) is the number of all the trainable parameters. Hence, we only need to investigate the remaining three hyperparameters. For \( b^{(T)} \), the accuracy of validation set for our method on the RTE dataset under different budget scenarios is shown in Figure 4. It can be seen that our method outperforms the strong baselines LoRA and (IA)^3 when \( b^{(T)} \) reaches 64, and achieves the best performance when \( b^{(T)} = 96 \). And the number of all the trainable parameters is only 135K when \( b^{(T)} = 64 \), which is equivalent to (IA)^3, indicating that our method can select the most important parameters and achieve better results. According to the results in Figure 4, we set \( b^{(T)} = 96 \) when fine-tuning the T5-base model. After manual tuning on the CoLA dataset, we obtain the values for \( t_i \) and \( t_f \). For other datasets, we scale these values based on the ratio with the total number of updates \( T \), selecting corresponding values for \( t_i \) and \( t_f \) accordingly. Detailed statistics of the hyperparameters are presented in Table 5. Depending on the size of dataset, the replication time ranges from 8 minutes (for RTE) to 5 hours (for MNLI).

<table>
  <tr>
    <th>Dataset</th>
    <th>Epochs</th>
    <th>\( T \)</th>
    <th>\( t_i \)</th>
    <th>\( t_f \)</th>
    <th>LR</th>
  </tr>
  <tr>
    <th colspan="6">GLUE Benchmark</th>
  </tr>
  <tr>
    <td>CoLA</td>
    <td>20</td>
    <td>1,720</td>
    <td>100</td>
    <td>800</td>
    <td>7e - 3</td>
  </tr>
  <tr>
    <td>RTE</td>
    <td>20</td>
    <td>500</td>
    <td>50</td>
    <td>250</td>
    <td>2e - 2</td>
  </tr>
  <tr>
    <td>MRPC</td>
    <td>20</td>
    <td>740</td>
    <td>80</td>
    <td>400</td>
    <td>2e - 3</td>
  </tr>
  <tr>
    <td>STS-B</td>
    <td>20</td>
    <td>1,160</td>
    <td>150</td>
    <td>600</td>
    <td>9e - 4</td>
  </tr>
  <tr>
    <td>SST-2</td>
    <td>10</td>
    <td>6,640</td>
    <td>400</td>
    <td>3,600</td>
    <td>2e - 2</td>
  </tr>
  <tr>
    <td>QNLI</td>
    <td>10</td>
    <td>10,380</td>
    <td>600</td>
    <td>5,500</td>
    <td>1e - 2</td>
  </tr>
  <tr>
    <td>QQP</td>
    <td>10</td>
    <td>36,290</td>
    <td>2,000</td>
    <td>19,000</td>
    <td>9e - 4</td>
  </tr>
  <tr>
    <td>MNLI</td>
    <td>10</td>
    <td>39,280</td>
    <td>2,000</td>
    <td>21,000</td>
    <td>4e - 2</td>
  </tr>
  <tr>
    <th colspan="6">SuperGLUE Benchmark</th>
  </tr>
  <tr>
    <td>CB</td>
    <td>30</td>
    <td>480</td>
    <td>30</td>
    <td>280</td>
    <td>8e - 4</td>
  </tr>
  <tr>
    <td>WSC</td>
    <td>30</td>
    <td>1,050</td>
    <td>50</td>
    <td>550</td>
    <td>4e - 3</td>
  </tr>
  <tr>
    <td>WiC</td>
    <td>20</td>
    <td>6,800</td>
    <td>400</td>
    <td>3,600</td>
    <td>1e - 3</td>
  </tr>
  <tr>
    <td>BoolQ</td>
    <td>20</td>
    <td>11,800</td>
    <td>800</td>
    <td>6,300</td>
    <td>3e - 3</td>
  </tr>
  <tr>
    <td>MultiRC</td>
    <td>10</td>
    <td>17,030</td>
    <td>1,000</td>
    <td>9,000</td>
    <td>1e - 3</td>
  </tr>
  <tr>
    <th colspan="6">Other Datasets</th>
  </tr>
  <tr>
    <td>SciTail</td>
    <td>10</td>
    <td>14,750</td>
    <td>800</td>
    <td>8,000</td>
    <td>3e - 3</td>
  </tr>
  <tr>
    <td>WinoGrande (WG)</td>
    <td>10</td>
    <td>24,630</td>
    <td>1,400</td>
    <td>13,000</td>
    <td>1e - 3</td>
  </tr>
  <tr>
    <td>PAWS</td>
    <td>10</td>
    <td>30,880</td>
    <td>1,800</td>
    <td>16,500</td>
    <td>3e - 3</td>
  </tr>
  <tr>
    <td>YelpPolarity (Yelp)</td>
    <td>10</td>
    <td>62,500</td>
    <td>3,600</td>
    <td>33,500</td>
    <td>9e - 4</td>
  </tr>
</table>

Table 5: The hyperparameters of all datasets.

C Additional Experimental Results

C.1 Results on the Llama2 Model

We evaluate our method on Llama2-7B, and due to resource constraints, we just perform it on the RTE and SST-2 dataset. We load the original model parameters with 16 bit, set the batch size to 16, the maximum sequence length to 128, and the total training steps to 4000. The comparison of our method with (IA)^3 and AdaLoRA is shown in Table 7. It can be observed that our method can also achieve good results with larger models.

C.2 The Runtime of Different Methods

We conduct experiments on T5-base model, and the table below displays the number of training steps per second and the number of samples per second for several methods. LoRA, AdaLoRA and AdaDHP train all the weights, while (IA)^3 only trains the key, value, and the intermediate activations of FFN, as stated in their original papers. As shown in Table 8, the results show that
<table>
  <tr>
    <th>Dataset</th>
    <th>Train</th>
    <th>Valid</th>
    <th>Test</th>
    <th>Task</th>
    <th>Metric</th>
  </tr>
  <tr>
    <th colspan="6">GLUE Benchmark</th>
  </tr>
  <tr>
    <td>CoLA</td>
    <td>8,551</td>
    <td>521</td>
    <td>522</td>
    <td>Acceptability</td>
    <td>Matthew’s Correlation</td>
  </tr>
  <tr>
    <td>SST-2</td>
    <td>66,349</td>
    <td>1,000</td>
    <td>872</td>
    <td>Sentiment</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>MRPC</td>
    <td>3,668</td>
    <td>204</td>
    <td>204</td>
    <td>Paraphrase</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>QQP</td>
    <td>362,846</td>
    <td>1,000</td>
    <td>40,431</td>
    <td>Paraphrase</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>STS-B</td>
    <td>5,749</td>
    <td>750</td>
    <td>750</td>
    <td>Sentence Similarity</td>
    <td>Pearson Correlation</td>
  </tr>
  <tr>
    <td>RTE</td>
    <td>2,490</td>
    <td>138</td>
    <td>139</td>
    <td>Natural Language Inference</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>MNLI</td>
    <td>392,702</td>
    <td>9,832</td>
    <td>9,815</td>
    <td>Natural Language Inference</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>QNLI</td>
    <td>362,846</td>
    <td>1,000</td>
    <td>40,431</td>
    <td>Natural Language Inference</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <th colspan="6">SuperGLUE Benchmark</th>
  </tr>
  <tr>
    <td>MultiRC</td>
    <td>27,243</td>
    <td>2,424</td>
    <td>2,424</td>
    <td>Question Answering</td>
    <td>FI</td>
  </tr>
  <tr>
    <td>BoolQ</td>
    <td>9,427</td>
    <td>1,635</td>
    <td>1,635</td>
    <td>Question Answering</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>WSC</td>
    <td>554</td>
    <td>52</td>
    <td>52</td>
    <td>Common Sense Reasoning</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>WiC</td>
    <td>5,428</td>
    <td>319</td>
    <td>319</td>
    <td>Word Sense Disambiguation</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>CB</td>
    <td>250</td>
    <td>28</td>
    <td>28</td>
    <td>Natural Language Inference</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <th colspan="6">Other Datasets</th>
  </tr>
  <tr>
    <td>WinoGrande (WG)</td>
    <td>39,398</td>
    <td>1,000</td>
    <td>1,267</td>
    <td>Common Sense Reasoning</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>YelpPolarity (Yelp)</td>
    <td>100,000</td>
    <td>1,000</td>
    <td>38,000</td>
    <td>Sentiment</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>SciTail</td>
    <td>23,596</td>
    <td>652</td>
    <td>652</td>
    <td>Natural Language Inference</td>
    <td>Accuracy</td>
  </tr>
  <tr>
    <td>PAWS</td>
    <td>4,9401</td>
    <td>8,000</td>
    <td>8,000</td>
    <td>Sentence Similarity</td>
    <td>Accuracy</td>
  </tr>
</table>

Table 6: Stastics of all datasets.

<table>
  <tr>
    <th>Model</th>
    <th>Params</th>
    <th>Train Memory</th>
    <th>RTE</th>
    <th>SST-2</th>
  </tr>
  <tr>
    <td>(IA)<sup>3</sup></td>
    <td>0.009%</td>
    <td>30G</td>
    <td>64.3 ±1.2</td>
    <td>95.2 ±0.2</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>0.074%</td>
    <td>33G</td>
    <td>68.9 ±1.7</td>
    <td>96.7 ±0.2</td>
  </tr>
  <tr>
    <td>AdaDHP</td>
    <td>0.018%</td>
    <td>32G</td>
    <td><b>70.0 ±1.2</b></td>
    <td><b>97.0 ±0.4</b></td>
  </tr>
</table>

Table 7: Test results of RTE and SST-2 on Llama2-7B model.

<table>
  <tr>
    <th>Method</th>
    <th>LoRA (<i>r = 1</i>)</th>
    <th>LoRA (<i>r = 32</i>)</th>
    <th>(IA)<sup>3</sup></th>
    <th>AdaLoRA</th>
    <th>AdaDHP</th>
  </tr>
  <tr>
    <td>Train steps per second</td>
    <td>1.562</td>
    <td>1.446</td>
    <td><b>1.775</b></td>
    <td>1.225</td>
    <td>1.471</td>
  </tr>
  <tr>
    <td>Train samples per second</td>
    <td>155,544</td>
    <td>144.07</td>
    <td><b>176.766</b></td>
    <td>122,026</td>
    <td>146,488</td>
  </tr>
</table>

Table 8: Runtime comparison among LoRA, AdaLoRA, (IA)<sup>3</sup> and AdaDHP.

<table>
  <tr>
    <th>Method</th>
    <th>CoLA</th>
    <th>STS-B</th>
    <th>MRPC</th>
    <th>RTE</th>
    <th>QNLI</th>
    <th>SST-2</th>
    <th>QQP</th>
    <th>MNLI</th>
  </tr>
  <tr>
    <td>LoRA (<i>r = 1</i>)</td>
    <td>55.6<sub>±1.1</sub></td>
    <td>88.6<sub>±0.4</sub></td>
    <td>86.9<sub>±0.2</sub></td>
    <td>67.4<sub>±3.4</sub></td>
    <td>93.1<sub>±0.1</sub></td>
    <td>93.6<sub>±0.2</sub></td>
    <td>90.0<sub>±0.2</sub></td>
    <td>85.9<sub>±0.1</sub></td>
  </tr>
  <tr>
    <td>DHP</td>
    <td>62.3<sub>±0.4</sub></td>
    <td>91.4<sub>±0.2</sub></td>
    <td>88.2<sub>±1.8</sub></td>
    <td>73.6<sub>±0.3</sub></td>
    <td>93.3<sub>±0.1</sub></td>
    <td>94.2<sub>±0.3</sub></td>
    <td>90.4<sub>±0.2</sub></td>
    <td>86.2<sub>±0.1</sub></td>
  </tr>
</table>

Table 9: Comparison of LoRA with <i>rank = 1</i> and our method DHP.

although AdaDHP includes adaptive parameter selection module. it trians faster than both LoRA (<i>r = 32</i>) and AdaLoRA.

D The Difference of DHP with LoRA

Our method can be formulated as \( \mathbf{W} \odot (\mathbf{I}_r^T) \), but it is not identical to LoRA (i.e. \( \mathbf{W} + \mathbf{BA} \)) with \( rank = 1 \). Mathematically, Hadamard product and addition are distinct operations, and their computations during backpropagation are different, so it is hard to make \( \mathbf{W} \odot (\mathbf{I}_r^T) \) equal to \( \mathbf{W} + \mathbf{BA} \) during optimization, especially in the case where the global optimal solution is difficult to be obtained. Empirically, we conduct experiments for LoRA with \( rank = 1 \) and our method that only includes Dual Hadamard Product (DHP) by removing Adaptive Parameter Selection module. As shown in Table 9, our methods shows a clear advantage, which confirms the differences between the two methods.

Reviewer paper summaries: This paper introduces a parameter-efficient fine-tuning method by injecting two trainable vectors: one for rows and one for columns, into each parameter of the LM. These vectors interact with the original weights through a dual Hadamard product, enabling fine-grained updates. In addition, an importance score is computed for each parameter to adaptively select which parts of the model should be tuned. Empirically, AdaDHP achieves superior performance compared to other PEFT methods, while maintaining a very low number of trainable parameters.

This paper proposes an innovative parameter-efficient fine-tuning method called AdaDHP, designed to effectively adapt large language models (LLMs) to downstream tasks under resource constraints. AdaDHP introduces Hadamard product operations on each parameter along both rows and columns, coupled with an adaptive parameter selection strategy to dynamically choose important parameters. Compared to LoRA, which only allows manually selecting trainable parameters and rank sizes, and (IA)3, which can only scale activations along the column, AdaDHP achieves finer granularity and more flexible fine-tuning. Experimental results demonstrate its superior performance on 17 NLP tasks using the T5-base model and complex mathematical tasks with Llama series models. However, the method does not yet address excessive memory usage, and some hyperparameters still require optimization.
Summary of strengths: 1. The use of dual Hadamard products along both rows and columns allows for highly parameter-efficient fine-tuning, significantly reducing the number of trainable parameters.
2. The adaptive parameter selection mechanism, based on importance score computation and budget allocation, helps focus training on the most impactful parameters, leading to improved overall performance.

1. This paper proposes a new PEFT method, AdaDHP, which uses some vectors to Hadamard Product the original weight matrices. It also uses some dynamic strategy to select the important trainable parameters. It trains only a tiny ratio of the model and achieves better results compared with the original LoRA, (IA)^3, and other PEFT methods.
2. It use multiple pre-trained models and lots of datasets to prove the robustness and generalizability of the AdaDHP method.
Summary of weaknesses: 1. While the authors attribute the performance drop of the “-Adaptive” variant (without adaptive parameter selection) to overfitting on smaller datasets, it remains unclear whether this issue persists on larger datasets or with larger models such as LLaMA3-8B. The current experiments are primarily conducted on T5-base and relatively small datasets, which limits the generalizability of the conclusions. Evaluations on more diverse and larger-scale settings would strengthen the claims.
2. The proposed method introduces several additional hyperparameters, which may increase the complexity of model tuning and reduce its plug-and-play usability in practical applications.
3. The rationale behind the specific budget scheduling strategy (e.g., the use of a cubic function for allocation, $(\cdot)^3$) is not well justified. It would be helpful to include theoretical analysis or empirical comparisons with alternative scheduling functions to support this design choice.

1. Memory usage issues remain unresolved, limiting practical applicability.
2. The budget allocation strategy in the Adaptive Parameter Selection module needs further optimization to simplify the tuning process and improve efficiency.
3. For different downstream tasks, additional customization may be required to achieve optimal results.
Comments, suggestions and typos: In Eq. (4), the two vectors $l$ should have different dimensions if $W \in \mathbb{R}^{d \times k}$.

1. In line 493, you say "Figure 3 shows the normalized parameter importance scores of Full Fine-tuning." How do you calculate the parameter importance scores here? Do you use the same method in Chapter 3.3? Your AdaDHP method dynamically selects the important parameters, but the importance score of AdaDHP in Figure 3 seems static.
2. For Algorithm 1, it says "total steps $T$; start steps $t_i$; final steps $t_f$; t range from $[1, T]$." In Table 4, it seems that $t_f - t_i != T$. What's the relationship between $t$, $t_f$, $t_i$, and $T$? What will your algorithm do before $t_i$? Does $t$ mean training steps or epochs in Machine Learning? What's the meaning of replication time in line 782?
3. In line 501, you say "Whereas LoRA proposes to only fine-tune $W_q$ and $W_v$" The fact is, both LoRA and $(IA)^3$ can also fine-tune $W_q, W_k, W_v, W_o, W_1, W_2$. The important parameters in LoRA can also be calculated.
4. In Appendix D, you compare LoRA with rank = 1 and your method that only includes DHP. Do you guarantee that both LoRA and DHP train the same weights, $W_q$, and $W_v$? In Table 3, you compare LoRA and AdaDHP when $r$ == $k$. However, in Table 1, the trainable parameters of LoRA are 3.8M, and the AdaDHP are 203K. What's the value of $r$ and $k$ in Table 1?
5. In line 482, you say "-Adaptive_single which scales the parameters along either rows or columns generally performs much worse." The result of -Adaptive_single in Table 1 and Table 2 only has one line. On earth, does this line scale the parameters along rows or scale the parameters along columns? Why don't you display two lines of -Adaptive_single?
6. In line 201, equation 2, you say $\text{softmax}\left(\frac{Q(l_k^T \odot K^T)}{\sqrt{d_k}}\right)(l_v^T \odot V)$. But the original equation from paper (IA)^3 use $\text{softmax}\left(\frac{Q(l_k \odot K^T)}{\sqrt{d_k}}\right)(l_v \odot V)$. Could you explain why you use l_k^T instead of l_k? In line 240, should $l_l^l$ be $l_l^k$?
7. In Chapter 3.2, if $l_l \in \mathbb{R}^d $, $W \in \mathbb{R}^{d \times d}, l_r \in \mathbb{R}^d$, $\sum _{i=1} ^{k} l_l^i \odot W \odot {l_r^i}^T = B \odot W$, where $B \in \mathbb{R}^{d \times d}$. During the training process, modify the trainable parameters set ${l} $ equals to modify the weight $B$. Maybe you can create the matrix $B$ after training, it will save the inference time.
Overall assessment scores: 3.5

3.5
Metareview score: 3.5
Metareview: The authors propose a new PET method—AdaDHP—which uses fewer parameters and offers finer granularity. It can adaptively select important parameters for each task and demonstrates its effectiveness on the T5-base model across 17 NLU tasks, as well as on complex mathematical tasks using the LLaMA series models.

This paper received an overall score of 3.5 (borderline conference). In addition, the authors provided further experiments to address reviewers’ concerns—such as the lack of evaluation on widely used open-source LLMs like LLaMA-8B—and demonstrated the method’s effectiveness on that model as well.

Overall, this is a solid piece of work. I share the reviewers’ opinion and recommend accepting it as a main conference paper.
Summary of reasons to publish: Above