Generative Parameter Efficient Fine-Tuning

Authors: Chinmay Savadikar, Xi Song, Tianfu Wu
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: parameter efficient fine-tuning, transfer learning

Abstract:
Fine-tuning pretrained (large) Transformer backbones efficiently for downstream tasks has been extensively explored using both Parameter-Efficient Fine-Tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA) and its variants, as well as more recent Representation-Efficient Fine-Tuning (ReFT) approaches. In both of these formulations, fine-tuning weights for selected pretrained layers are treated as model parameters that are directly learned from the downstream task data, often making them layer-specific. While these methods simultaneously aim for memory efficiency, some approaches, such as VeRA (Vector-based Random matrix Adaptation), may not achieve this consistently in practice. In this paper, we propose a novel approach for generating fine-tuning weights through a configurable layer-sharing mechanism, termed Generative parameter-efficient Fine-Tuning (GIFT). GIFT uses a simple parameterization scheme involving two linear layers (without bias terms) to enable efficient fine-tuning. This method bridges the gap between PEFT and ReFT, ensuring both parameter and memory efficiency. GIFT can be viewed as a variant of LoRA with parameters shared across layers, conditioned on the pretrained weights, with significantly fewer trainable parameters. Through extensive experiments, we demonstrate that our GIFT consistently achieves superior performance and parameter efficiency compared to baselines on commonsense and arithmetic reasoning tasks, instruction tuning with the Llama family of models, and visual recognition benchmarks with Vision Transformers. Notably, GIFT achieves a 5.7% absolute increase in average accuracy with a 14x reduction in trainable parameters compared to LoRA on the Commonsense170k dataset using Llama-3 (8B), and a 5.4% increase in win rate with a 4x reduction in parameters using Llama-2 (7B) during instruction tuning. Our method also attains a slightly higher win rate for instruction tuning than GPT-3.5 (Turbo 1106)

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730276771142

thread_type:
official_review

<thread_object>
creator: reviewer_53qz

<content>
summary:
This paper proposes a new fine-tuning method called Generative Parameter-Efficient Fine-tuning (GIFT), which trains two linear layers to project the pre-trained weight matrices into fine-tuned weights. The authors argue that it offers a unifying perspective on PEFT and representation-efficient fine-tuning (ReFT) approaches by projecting pre-trained weights linearly. The results demonstrate that GIFT improves performance while using fewer parameters compared to previous parameter-efficient fine-tuning (PEFT) methods, such as LoRA, ReFT, and VeRA.

strengths:
1. This paper provides new insights into fine-tuning techniques, suggesting that the proposed Generative Parameter-Efficient Fine-Tuning (GIFT). I think it can be viewed as a specific form of Representation Fine-Tuning (ReFT), where all tokens in the selected layers share the same re-parameterization parameters. Notably, GIFT is easier to implement than ReFT, utilizing two linear layers for weight re-parameterization without explicitly modifying token embeddings.

2. The performance is impressive. They achieve similar or better performance with fewer parameters comparing to the other PEFT. Validation was conducted across multiple tasks, datasets, and models, demonstrating GIFT's effectiveness and versatility.

weaknesses:
1. In my opinion, the authors make claims about aspects that remain unexplained, which can confuse readers. The authors should provide further clarification to support these claims. For instance:

(Line 051) "but the learnable weight-residuals do not have direct information exchange with the pre-trained weights"

(Line 135) "one of the simplest updates that minimally distorts and maximally preserves the pre-trained knowledge is defined by Eqn.1 and Eqn.2, thanks to the low-rank factorized linear projection in the parameter space."

(Line 181) "Additionally, adding fixed random matrices with learnable scales in PEFT makes the relationship between fine-tuned models and frozen pretrained models less intuitive."

(Line 194) "Furthermore, token-level interventions lack a holistic understanding of the relationship between ReFTed models and frozen pretrained models."

2. There are issues with mathematical notation throughout the paper, making it difficult for readers to follow the ideas presented. Please refer to other PEFT papers (e.g., ReFT, LoRA) for guidance on presenting mathematical concepts clearly. For example, matrices should be represented in bold uppercase, vectors in bold lowercase, and scalars in italic lowercase. Additionally, in machine learning literature, symbols like θ (theta) and φ (phi) are conventionally used to denote model parameters, not calculations within the model.

3. The content has redundancies; for instance, the Related Work section and Section 2.1 cover similar material, leading to overlap.

questions:
See weaknesses.

+) 

1. What did the authors want to say in Section 2.3? I think the idea behind this is similar to 2.2. What is the meaning of "accumulate" in the paragraph?

2. Please provide more details regarding the experiment in Fig. 2. Specifically, clarify the meaning of the cluster. If the authors aim to demonstrate that GIFT enhances object highlighting within the attention modules, results should be compared with those of the pretrained model for a meaningful evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730439084372

thread_type:
official_review

<thread_object>
creator: reviewer_k2rs

<content>
summary:
The paper introduces Generative Parameter-Efficient Fine-Tuning (GIFT), a method to fine-tune pretrained Transformer models with fewer parameters by generating fine-tuned weights from pretrained ones. They show this formulation can address the two questions about 1)an explicit and direct mapping between the fine-tuned model and the frozen pretrained model, 2) bridge  parameter-efficient fine-tuning and representation fine-tuning. The proposed GIFT method is designed by implementing a lightweight structure of only two linear layers, shared across selected layers in the model. Using minimal linear layers without bias, GIFT achieves significant parameter reductions compared to LoRA and performs better across some NLP and computer vision tasks, obtaining a slightly higher win rate on instruction tuning than GPT 3.5.

strengths:
GIFT represents a unique approach to generating fine-tuned weights directly from pretrained weights, sharing parameters across layers to enhance efficiency.
Experiments demonstrate that GIFT outperforms existing PEFT methods on various natural language and computer vision tasks while using significantly fewer parameters, showing improvements in memory efficiency as well.
Tested on diverse tasks, GIFT shows effectiveness across commonsense reasoning, arithmetic, instruction following, and visual recognition tasks, reinforcing its versatility as a parameter-efficient fine-tuning approach.

weaknesses:
Lack of some comparison settings on full finetuning:  How is the comparison results on full finetuning in Table-2 and Table-3 for commonsense reasoning and arithmetic reasoning task？Table-1 compares the full finetuning setting on Llama-2 7B for instruction following task, but table-2 and table-3 didn't reveal this setting.
Potential Scalability Concerns: Although parameter-efficient, the scalability of GIFT for larger models (beyond 8B parameters, like llama1-13B, llama3-65B) isn’t explicitly demonstrated. The experimental version of llama1-3 presented by the author is all less than or equal to 8B, leaving questions about its performance in high-scale deployment.  
Limited Ablation on Layer Selection and Configuration: GIFT’s performance maybe vary depending on which layers are selected for fine-tuning. While some experiments address this, there is minimal ablation on different layer selection and the comparison between Lora with the same layers.
Some models compared are not newly advanced enough : Table-1 shows the result of fine-tuning Llama-2 7B with GIFT for instruction following task, but the compared gpt series is GPT 3.5 Turbo, a little bit outmoded. How is the comparison result of GIFT among some newly advanced models like gpt4o or others？

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730535177148

thread_type:
official_review

<thread_object>
creator: reviewer_rg2d

<content>
summary:
This paper proposes a modification to the well-known lora method. Technically, as the original lora method can be formulated as $\overline{W} = W + AB$, the proposed modification changes it to $\overline{W} = W + WAB'$, with $B'$ shared across layers. The authors have discussed the relationship with ReFT, and used experiments to support the efficacy of the modifaction.

strengths:
LoRA is now a widely adopted technique and improvements over it can make profound impacts.

weaknesses:
**Unconvincing method design**

The biggest weakness to me is that the proposed method is not supported by reasonable and convincing motivations. Specifically, there are two major changes from the original LoRA:
1. sharing half of the lora weights across layers;
2. involving the original weight as an extra term into the weight delta.

However, it is unclear why these two changes are useful and how do they work to benefit. After reading the paper I cannot get a satisfying answer. This problem fundamentally limits the value of the work, as it is less likely for people to give the proposed method a try without an intuition that makes people believe it would lead to better results.

**The introduction of the simple method is unnecessarily complicated**

As mentioned in the summary part, the core of the proposed method is simply $\overline{W} = W + AB$ -> $\overline{W} = W + WAB'$, but the paper makes me feel that it is much more complicated.

**Content organization can be better**

The first section is too long. As an introduction section, it involves too many details that are hard to fully understand before reading the methodology section. I suggest only preserving the high-level ideas in this section while moving the technical details elsewhere.

**Others**

1. tied-lora[1] also works on lora + weight-sharing, and I suggest to add some analysis & comparisons.
2. I cannot find the information about the backbone model for visual experiments, welcome to correct me if I missed it.
 
[1] Renduchintala, Adithya, Tugrul Konuk, and Oleksii Kuchaiev. "Tied-lora: Enhacing parameter efficiency of lora with weight tying." arXiv preprint arXiv:2311.09578 (2023).

questions:
Could authors give some intuitions on why the changes proposed are beneficial?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732161995618

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Brief summary of Results and Changes

comment:
We thank the reviewers for their valuable feedback. We have carefully revised the original manuscript based on the comments provided. Below, we briefly summarize our results, followed by an overview of the changes made in the revised manuscript. We believe that the revised manuscript is clearer and that the changes implemented in response to the reviewers' suggestions have significantly enhanced the overall quality of the paper. We have addressed the concerns raised by reviewers through individual comments.

# Brief summary of results
- **Instruction Following** (Section 3.2), our GIFT can outperform GPT-3.5 Turbo using 0.0311% trainable parameters in fine-tuning Llama-2 (7B), which is the only method to do so in our comparisons. 
- **Commonsense Reasoning** (Section 3.3), our GIFT outperforms both the prior art of PEFT and of ReFT consistently using Llama 1-2-3 model family, often by large margin with less trainable parameters used. 
- **Arithmetic Reasoning** (Section 3.4), our GIFT can outperform all the prior PEFT and ReFT approaches. Unlike VeRA, which performs slightly better than LoRA, GIFT maintains the computational efficiency while achieving better performance. VeRA takes about 1.5 days in training, while our GIFT takes about 4 hours.
- The proposed block-wise sharing $^{B}$ GIFT $^{r}_{\underline{QKV},\overline{O},\underline{UG},\overline{D}}$ shows stronger consistency of achieving better results across tasks (Commonsense Reasoning and Arithmetic Reasoning).

# Summary of changes
As suggested by reviewers `Rg2D` and `53Qz`, we have rephrased the introduction and organized the content to more precise and concise. Following are the changes made in the revised manuscript:

## Section 1: Introduction 
- Section 1 now illustrates the motivation, formulation, contributions and design choices in a more straightforward manner.
- As suggested by reviewer `53Qz`, we have modified the notation to make it more consistent and clear.

## Section 2: Approach
We have removed the redundancies in Section 2, which now focusses on the generic formulation for the generative approach proposed in GIFT, highlighting the key properties of the method:
- GIFT shares all the learnable parameters across layers (for e.g., one GIFT is shared across all Query layers), but still generates layer specific finetuning residuals and fine-tuned weights because of the generative nature.
- GIFT bridges PEFT and ReFT, as it can be equivalently applied to the representations or the weights, and is applied uniformly to all the tokens in the sequence, eliminating the need for a hyperparameter search over the token selection.

## Section 3: Experiments
- As suggested by reviewer `k2rS`, we have added experiments with Llama-1 13B for Commonsense Reasoning and Arithmetic Reasoning tasks. Table 2 and Table 3 in the revised manuscript show that GIFT performs better/on par with prior PEFT and ReFT methods while being significantly more parameter efficient. This shows that GIFT can be scaled to larger models on the scale of 13B, and can potentially be scaled to even larger models. Due to resource constraints, we are unable to run experiments on larger models on the scale of 65B.
- We include additional experiments with VeRA for Commonsense Reasoning, and show that GIFT outperforms VeRA by a significant margin, while requiring less wall time to train. We have also added the wall time required to train VeRA and GIFT in Sections 3.2 and 3.3.

## Section 4: Albation Studies
We have organized the ablation studes in a separate section, which now includes an additional ablation study that verifies that the generative approach in GIFT with shared learnable parameters is more effective than simply sharing the LoRA weight residuals (i.e., $\Delta W^l = B \cdot A\ \ \forall l \in L$, denoted by Shared LoRA).
- Table 5 in the revised manuscript shows that GIFT performs much better than Shared LoRA on Commonsense Reasoning using the Llama-3 (8B). This suggests that Tied LoRA, suggested by reviewer `Rg2D` may not be effective, as it shares the residuals directly across layers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733027668830

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of author-reviewer discussion and request reviewers' (re-)evaluation of the changes in revision

comment:
Dear Reviewers,

We are grateful for your efforts and valuable time spent on our submission and revision, which has helped us a lot in improving the quality of our submission.

We try to do our diligence to maximize the outcome from this valuable reviewing process. So, we briefly summarize the current status as follows:

Reviewer `k2rS` has yet to give their feedback on our rebuttal. We hope to have their valuable feedback soon and appreciate the time spent on our submission.

Reviewers `Rg2D` and `53Qz` have some further concerns as follows:

> Lack of explanation for the effectiveness of GIFT

We have tried to address reviewer `Rg2D`'s concern in Section 2.3 of the revised manuscript, which we have reproduced below:

>> Pretrained Transformer backbones encode diverse knowledge from large-scale pretraining datasets within their weights. Fine-tuning them for a downstream task aims to incorporate new information from the task-specific training data and utilize the information present in the pretrained weights to the fullest extent. To achieve this, the fine-tuned weights can be directly conditioned on the pretrained weights, such that the new information is learned conditionally from the information in the pretrained weights. While LoRA and it's variants use a residual structure to address this, the residual weights are not directly conditioned on the pretrained weights, but rather learned via back-propagation (chain rule) updates. One of the simplest functions that can achieve this explicit conditioning is a linear transformation of the pretrained weights, as leveraged in Eqn.7. Hence, the fine-tuned weights can also be expressed in the space of the pretrained weights $W_{d_{out}\times d_{in}}$ via $W_{d_{out}\times d_{in}}\cdot \Theta_{d_{in}\times d_{in}}$. 

We agree that a more thorough explanation of the effectiveness of GIFT would be beneficial. We also strongly believe that finding a simple formulation that works well in practice worths revealing to the community, and plan to work on a more detailed explanation (such as the explorations and analyses on learning rates for $\phi$ and $\psi$ as done for $B$ and $A$ in LoRA by [1] and the gradient decomposition based initialization for $\phi$ and $\psi$ as done for $B$ and $A$ in LoRA by [2]) in future work.

> Concerns about the notation used in GIFT

As suggested by reviewer `53Qz`, we have modified the notation slightly by using $W$ instead of $\omega$ to represent the pretrained weights. However, we retain the use of $\phi$ and $\psi$ to denote the learnable parameters, as they are a part of the parameters of the weight-generator network. This lets us distinguish between standard fine-tuning methods and GIFT. Our GIFT generates the fine-tuned weights from the pre-trained weights, rather than learning them as model parameters. This also keeps the notation consistent across equations (4), (5) and (6), which are,

$\hat{W}^l_{d_{out}\times d_{in}} = W^l_{d_{out}\times d_{in}} + \mathcal{G}(W^l_{d_{out}\times d_{in}}; \Theta)$ (.... Eq. 4)

$\qquad \qquad =W^l_{d_{out}\times d_{in}} +$ $W^l_{d_{out}\times d_{in}} \cdot \phi_{d_{in}\times r}\cdot \psi_{r\times d_{in}}$ (.... Eq. 5)

$\mathcal{G}(\Omega_{L\times d_{out}\times d_{in}};\Theta) = \text{Linear}\Biggl( g\biggl(\text{Linear}\bigl(\Omega_{L\times d_{out}\times d_{in}};\phi\bigr);\theta\biggr);\psi \Biggr)$ (.... Eq. 6)

Overall, we believe the notations used in our GIFT is self-contained and clear.

Thank you very much.

---
[1] S. Hayou, N. Ghosh and B. Yu, "The Impact of Initialization on LoRA Finetuning Dynamics", http://arxiv.org/abs/2406.08447

[2] S. Wang, L. Yu and Jianli, "LoRA-GA: Low-Rank Adaptation with Gradient Approximation", http://arxiv.org/abs/2407.05000
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a method for parameter-efficient fine-tuning that aims to improve model performance while reducing the number of trainable parameters. The reviewers appreciated the strong experimental results demonstrating its advantage over baseline methods like lora.

However, the reviewers expressed concerns about the unclear motivation for the proposed modifications to lora, particularly the weight-sharing and inclusion of original weights in the delta term, suggesting a more intuitive explanation is needed. The methodology was also seen as overly complicated for a simple update, and the paper's organization could be improved, particularly in the introduction. Additionally, the paper lacked sufficient comparisons with full fine-tuning, especially in commonsense and arithmetic reasoning tasks. There were no ablation studies on layer selection. Some of these concerns are addressed and confirmed by reviewers, while some of them are not well addressed. Some of these concerns are addressed and confirmed by reviewers, while some of them are not well addressed. 

Considering the average score of 5, which is below the acceptance threshold, and the overall feedback, the final decision for the paper is reject.