Advancing Parameter Efficiency in Fine-tuning via Representation Editing (2024)

Abstract: Parameter Efficient Fine-Tuning (PEFT) techniques have drawn significant attention due to their ability to yield competitive results while updating only a small portion of the adjustable parameters. However, existing PEFT methods pose challenges in hyperparameter selection, such as choosing the rank for LoRA or Adapter, or specifying the length of soft prompts. To address these challenges, we propose a novel fine-tuning approach for neural models, named Representation EDiting (RED), which modifies the representations generated at some layers through the application of scaling and biasing operations. While existing PEFT methods still demonstrate over-parameterization that could potentially undermine the generalization ability acquired from pre-training, RED can substantially reduce the number of trainable parameters by a factor of 25, 700 compared to full parameter fine-tuning and by a factor of 32 relative to LoRA. Remarkably, RED achieves results comparable or superior to both full parameter fine-tuning and other PEFT methods. Extensive experiments across various model architectures and scales, including RoBERTa, GPT2, T5, and LLaMA-2, have demonstrated the effectiveness and efficiency of RED1, thereby positioning it as a promising PEFT strategy for large-scale neural models.

heading: 1 Introduction
text: Pre-training on large-scale unlabeled datasets followed by fine-tuning on task-specific dataset has demonstrated remarkable efficacy across various natural language processing (NLP) tasks, establishing itself as the prevailing training paradigm (Devlin et al., 2018; Raffel et al., 2020; Radford et al., 2018). However, conducting full parameter fine-tuning for each task can be exceedingly costly
*Corresponding author. 1The code is accessible at https://github.com/mlwu22/
RED.
and increasingly daunting as model scales continue to grow (Brown et al., 2020). For instance, BERT comprises up to 220 million parameters, T5 scales up to 11 billion parameters, and GPT-3 boasts an astounding 175 billion parameters. Consequently, the efficient and effective adaptation of large models to specific downstream tasks presents an intriguing research challenge (He et al., 2021).
In response to this challenge, researchers have put forward three main lines of Parameter Efficient Fine-Tuning (PEFT) techniques (Ding et al., 2022). Firstly, addition-based methods involve the introduction of additional trainable neural modules or parameters that were not present in the original model (Houlsby et al., 2019; Karimi Mahabadi et al., 2021; Li and Liang, 2021a; Lester et al., 2021a). Specification-based methods, on the other hand, identify certain parameters in the original model to be trainable, while the rest are kept frozen (Zaken et al., 2021; Guo et al., 2020). Lastly, reparameterization-based methods reconfigure trainable parameters into a more parameterefficient form through certain transformations (Hu et al., 2021; Zhang et al., 2023a; Ding et al., 2023).
Among these PEFT methods, Low-Rank Adaptation (LoRA) stands out as one of the most efficient techniques with its effectiveness empirically validated across various models of diverse scales. Despite its impressive performance, LoRA still demands a significant number of trainable parameters. Recent studies by Aghajanyan et al. (2020) and Kopiczko et al. (2023) indicate that the upper bound for intrinsic dimensions is substantially smaller than what is typically used in such methods. For example, the d90 value (the minimum number of trainable parameters required to reach 90% of the performance of the fully-parameter fine-tuned model) for the RoBERTa base is reported to be 896. However, when LoRA is applied to fine-tune this model, the number of trainable parameters escalates to 0.3 million, suggesting the potential for
13445
further reduction in parameter count. In addition to the issue of requiring too many adjustable parameters, existing PEFT methods (Mao et al., 2021; He et al., 2021; Ding et al., 2022) primarily focus on the design of lightweight modules and their integration (or placement) within base models. Nonetheless, the implementation of these PEFT techniques introduces additional complexities in hyperparameter selection, such as choosing the rank of LoRA and Adapter, or deciding on the length of Soft Prompt and Prefix.
Inspired by the concept of representation engineering (Zou et al., 2023), we shift our focus away from the weights of models and turn our attention to their representations. In the neural architecture, network weights govern neural activities (or representations), which in turn determine the networks’ output, and the networks’ output ultimately shapes the networks’ behavior. Rather than concentrating on neurons and their interconnections (or weights), we explored to achieve control over network behavior by manipulating its internal representations. Specifically, we fine-tune neural network models by directly editing the representation generated at each layer while maintaining the model parameters frozen, as illustrated in Figure 1 (b). It is worth noting that the number of parameters required to edit representations is substantially fewer than that of weights within neural networks. Taking LLaMA-2 (7B) as an example, the proposed representation editing (RED) method achieves competitive performance by adjusting only 0.26M parameters. This is approximately 25, 700 times less than what is required for full-parameter fine-tuning, rendering the method both storage and computation efficient.
The contributions of this study are summarized as follows:
• We propose a novel perspective on fine-tuning by directly editing model representations, diverging from exiting PEFT methods that focused on adjusting the model’s weights. Our proposed PEFT technique, termed RED, embodies this new perspective. • Extensive experiments are conducted across models of varying structures and scales, including RoBERTa, GPT-2, T5, and LLaMA-2. The effectiveness of RED is validated across a range of natural language understanding and generation tasks. Notably, RED demonstrates both efficacy and efficiency while requiring only a minimal number of trainable parame-
ters and maintaining ease of implementation. • A comprehensive ablation study is conducted
to dissect the individual components of RED and understand their impacts on performance.

heading: 2 Related Work
text: Existing PEFT methods can be broadly classified into three categories (Ding et al., 2022), each characterized by the particular parts of parameters that they tune efficiently: addition-based, specificationbased, and reparameterization-based methods.
Addition-based methods perform fine-tuning by integrating additional lightweight learnable components into foundational models. More specifically, Houlsby et al. (2019); Stickland and Murray (2019); Karimi Mahabadi et al. (2021) and Rücklé et al. (2020) proposed the integration of learnable bottleneck neural modules into the transformer layers. Brown et al. (2020) and Shin et al. (2020) found that the models’ performance could be enhanced by appending discrete tokens (or prompts) at the beginning of the input texts without modifying the models’ parameters. However, the manual creation of such prompts demands substantial effort, and optimization in a discrete space is comparatively more challenging. Consequently, later studies substituted these discrete tokens with continuous vectors, also referred to as soft prompts, which can be optimized using the gradient descent algorithm (Lester et al., 2021b; Li and Liang, 2021b; Wu et al., 2023; Wang et al., 2023).
Specification-based methods achieve parameterefficient fine-tuning by designating a subset of parameters for modification while leaving the remaining parameters untouched. Within this category, Lee et al. (2019) suggested a method to exclusively update the parameters in certain top layers of BERT and RoBERTa. BitFit (Ben-Zaken et al., 2021), on the other hand, fine-tunes a model by only optimizing the bias terms used in the model. Contrary to these methods that pre-determine which parameters should be altered, Guo et al. (2020) and Zhao et al. (2020) implemented a learnable masking strategy to automatically choose the parameters for tuning.
Reparameterization-based methods aim to optimize some parameters within a model, typically in their low-dimensional subspace. Hu et al. (2021) introduces the use of low-rank matrices, termed LoRA, to approximate the weight modifications during the fine-tuning process. QLoRA (Dettmers et al., 2023) combines low-rank adaptation with
model quantization to further reduce storage usage during the fine-tuning phase. AdaLoRA (Zhang et al., 2023b) proposes to use SVD decomposition to approximate the changes in weights, which allocates more trainable parameters to more important weight matrices, leading to a better performance.
Zou et al. (2023) proposed an approach to analyzing and manipulating the behavior of neural networks through representation engineering, thereby shifting the focus from neurons and their circuits to representations and the transformations among them. Liu et al. (2023) extended this work to align large language models with human preferences, achieving results comparable or superior to RLHF (Ouyang et al., 2022), but at a lower computational cost. Subramani et al. (2022) investigated the extraction of “steer vectors” from the hidden layers and performed unsupervised text style transfer by altering the hidden representations using these vectors. To control the style and sentiment expressed in a model’s outputs, Turner et al. (2023) integrated a similar “steer vector” into the representation of each hidden layer at the inference time. Inspired by the recent advance in representation engineering Zou et al. (2023), we suggested a novel perspective for fine-tuning models by directly editing their representations. This new perspective was embodied in the RED training method, which demonstrated both effectiveness and efficiency while requiring only a minimal number of trainable parameters.
This study is closely related to two recent works:
IA3 (Liu et al., 2022) and SSF (Lian et al., 2022). To fulfill the in-context learning capability with lower computational costs, Liu et al. (2022) proposed to modify the key and value vectors of the multi-head attentions and those generated by feedforward networks (FFNs) by scaling operations. In the domain of computer vision, Lian et al. (2022) presented a similar method, called SSF, which was motivated by batch and layer normalization techniques. To ensure features fall in a discriminative space for better classification, SSF also necessitates the adjustment of feature vectors across nearly all transformer layers, including multi-head attentions, FFNs, and normalization layers. As a result, the size of parameters adjusted in their method approximates those manipulated in other PEFT techniques, such as Adapter and VPT (Jia et al., 2022).
In contrast, our RED method requires only the modification of activation patterns generated by FFN sub-layers, constituting a significantly smaller portion of the entire neural network’s parameters for model fine-tuning. While SSF achieves a reduction in the number of fine-tuning parameters by a factor of 300 compared to original models, RED accomplishes a reduction of approximately 25, 700 times. Moreover, SSF’s evaluation was restricted to a limited range of simple image classification tasks using relatively small models (under 100M parameters), whereas RED has demonstrated its effectiveness across a variety of natural language understanding and generation tasks, with signifi-
cantly larger models of up to 7B parameters.

heading: 3 Method
text: We in this section give a concise overview of previous PEFT methods and introduce a new, parameterefficient fine-tuning method known as Representation Editing (RED) by highlighting its distinctive features compared to existing PEFT methods. RED facilitates the adaptation of pre-trained neural network models to downstream tasks by directly editing the model’s representations.

heading: 3.1 Recap of PEFT Methods
text: The predominant large language models (LLMs) have been constructed based on the transformer architecture proposed in a seminal work on attention mechanisms (Vaswani et al., 2017). This architecture is characterized by a series of layered blocks, each incorporating two fundamental sub-layers: multi-head self-attention and fully connected feedforward networks. Many existing PEFT methods usually achieve fine-tuning by adjusting the parameters of these two sub-layers in some parameterefficient manner. Figure 1 (a) illustrates two exemplar PEFT methods: Adapter and LoRA. With the exception of a few additional parameters that require tuning, the parameters of the pre-trained LLM remain fixed.
Specifically, LoRA (Hu et al., 2021) incorporates learnable bottleneck-shaped modules through the connections parallel to the Wq and Wv matrices of attention blocks, as illustrated in Figure 1 (a), and models the weight changes of these two matrices in a low-rank manner. Given a pre-trained weight matrix W ∈ Rd×k, LoRA represents its changes, denoted as ∆W, through two low-rank decomposition matrices: ∆W = α · WdownWup, where Wdown ∈ Rd×r and Wup ∈ Rr×k (the value of r is typically much smaller than both d and k). The coefficient α is a hyperparameter that determines the significance of ∆W relative to the original W. Given an input x, LoRA produces the result of the forward pass as follows:
h = xW + α · xWdownWup (1) In the initial version of the Adapter, Houlsby et al. (2019) suggested the insertion of trainable adapter modules between two sub-layers within the transformer architecture. These adapter modules are structured to include a down-projection matrix, denoted as Wdown ∈ Rd×r, which transforms a hidden representation h1 ∈ Rd to a lower-dimensional
space with a pre-specified dimensionality r. Subsequently, this dimensionally reduced vector undergoes a nonlinear activation function f(·) and an additional up-projection matrix Wup ∈ Rr×d to revert it to its original dimension d. The adapter module also incorporates a residual connection. The resultant output h2 ∈ Rd generated by this module can be formally represented as follows:
h2 = h1 + f(h1Wdown)Wup (2)
Expanding on this research, Pfeiffer et al. (2020) proposed a more efficient variant of the Adapter that is only applied following the FFN sub-layer.

heading: 3.2 Representation Editing
text: Previous PEFT methods refine pre-trained models by updating their weights in a parameter-efficient manner, typically within a low-dimensional space. To approximate the effect of full-parameter tuning, they are required to choose the values of hyperparameters properly for one downstream task or a set of tasks. However, it could be troublesome to choose a suitable value for a hyperparameter, such as the ranks of weight matrices in Equations (1) and (2) for LoRA and Adapter respectively. Enhancing their modeling capacities by using a higher rank r could demand too much computation resources and tend to overfit, while aggressively setting r smaller may degrade model performance and lead to from-scratch re-training (Ding et al., 2023).
Our hypothesis posits that by altering the internal representations of neural models rather than their connected weights, the fine-tuning process could be significantly more efficient. This is because that direct manipulation of the representations necessitates O(n) parameters, whereas adjusting their weights theoretically demands O(n2) parameters, with n denoting the dimensionality of hidden representations. Turner et al. (2023) have demonstrated that the behaviors of neural models can be influenced by adding a “steer vector” to each hidden layer during inference. We postulate that such steer vectors can be learned during the fine-tuning phase. Inspired by this concept and the emerging field of representation engineering Zou et al. (2023), we introduce a novel PEFT method that fine-tunes the model by directly altering the representation with two learnable vectors, as depicted in Figure 1 (b).
Specifically, we first incorporate a learnable scaling vector lscaling ∈ Rd and apply it to perform the Hadamard product with a hidden representation h1
by scaling the feature of each dimension within h1 via element-wise multiplication. Additionally, we introduce another learnable bias vector lbias ∈ Rd that is subsequently added to the scaled vector. This process can be formalized as follows:
h2 = lscaling ⊙ h1 + lbias (3)
where ⊙ represents element-wise multiplication, also known as the Hadamard product. Here, h1 ∈ Rd denotes a hidden representation generated by a certain layer, and h2 ∈ Rd represents the resultant edited representation. During implementation, we initialize the scaling vectors lscaling as unit vectors (i.e., with all elements set to 1) and the bias vectors lbias as zero vectors. This initialization approach ensures that the introduction of these “edit vectors” does not initially alter the hidden representations of a neural model.

heading: 4 Experiments
text: Extensive experimentation was conducted to assess the efficacy of our Representation Editing (RED) method across a diverse array of natural language
understanding and generation tasks, employing a range of foundational models spanning different scales. These models include RoBERTa (Liu et al., 2019), T5 (Raffel et al., 2020), GPT-2 (Radford et al., 2019), and LLaMA-2 (Touvron et al., 2023). Specifically, we evaluated RED and the baseline methods on the GLUE benchmark (Wang et al., 2018) with RoBERTa and T5, as previously conducted in (Hu et al., 2021) and Asai et al. (2022). To ensure consistency with previous studies, we adhered to the experimental setup outlined in (Li and Liang, 2021a) and (Hu et al., 2021) for comparative analysis with GPT-2. Moreover, we conducted instruction tuning experiments on LLaMA-2 using the UltraFeedback dataset (Cui et al., 2023) to further assess the applicability of our proposed RED on generative large language models. For further details on the datasets and evaluation metrics used, please refer to Appendix A.

heading: 4.1 Baselines
text: We ensured that different PEFT methods were compared systematically in a more fair setting. There-
fore, we strictly followed the well-established training protocol, evaluating on the validation set after each epoch training, and selecting the checkpoint with the best performance on the validation set as the final model for testing on the test set. The following baseline methods were used for comparison with the proposed RED:
• Fine-Tuning (FT) trains models by updating all their parameters. A variant of FT was proposed by Lee et al. (2019), which selectively updates certain layers while freezing others. We incorporate a baseline established in prior research by Li and Liang (2021a) with GPT-2, which specifically adapts only the final two layers, denoted as FTtop2. • Bias-terms Fine-tuning (BitFit) involves the selective freezing of a majority of the transformer parameters, with the training process focused exclusively on the bias-terms (BenZaken et al., 2021). • IA3 introduces scaling operations to modify the key and value vectors of the multi-head attentions and those generated by feed-forward networks (FFNs) (Liu et al., 2022). • Adapter introduces a learnable, lightweight module situated between two sub-layers of the transformer. During the forward pass, inputs are processed in sequence by the sub-layers of the foundation models and the adapters, yielding the final output. However, during the backpropagation phase, only these adapters receive gradients for parameter updates, while the remaining parameters of the model are kept fixed and unaltered (Houlsby et al., 2019). • Adapter-FFN is a variant of Adapter method proposed by Pfeiffer et al. (2020). Contrasting with the original Adapter, which necessitates the insertion of the learnable module between all sub-layers, Adapter-FFN only requires the application of an adapter following each FeedForward Network (FFN) sub-layer. • AdapterDrop is another variant of Adapter
proposed by Rücklé et al. (2020), which incorporates a strategy of omitting certain adapter layers, thereby enhancing overall efficiency. • Low-Rank Adaption (LoRA) employs a low-rank decomposition on the matrix ∆W, thereby modeling weight updates as the product of two low-rank matrices. These two learnable matrices are aligned in parallel with the corresponding matrices in pre-trained models. They process inputs in parallel and combine their results to generate the final outputs in each transformer block (Hu et al., 2021). • Prompt Tuning (PT) prefixes a set of continuous vectors at the embedding layer, which are subjected to learning during the fine-tuning phase (Lester et al., 2021b). • Prefix Tuning is a generalized version of prompt tuning (PT), which incorporates learnable continuous vectors at every hidden state. These continuous vectors also contribute to the computation of attention, serving as both key and value vectors (Li and Liang, 2021b).

heading: 4.2 Results with RoBERTa
text: We conducted fine-tuning experiments on both the RoBERTa base (125M) and large (355M) models by using RED and its competing baselines. Subsequently, we evaluated these fine-tuned models on the widely-adopted GLUE benchmark, renowned for its comprehensive evaluation of natural language understanding capabilities. The pretrained RoBERTa models were sourced from the HuggingFace Transformers Library (Wolf et al., 2019).
We noticed that the previous evaluation settings of PEFT methods encountered two issues. Firstly, for datasets such as MRPC, RTE, and STS-B, they trained the models on the MNLI dataset first and selecting the best performance checkpoint on the MNLI dataset as initialization for transfer learning to improve the performance of the model trained on these datasets (Liu et al., 2019; Hu et al., 2021). However, such a pipeline adds complexity that can be challenging for other researchers to replicate.
Secondly, there is no split between the validation set and the test set. Instead, after each epoch training is completed, evaluation is directly conducted on the test set, and the best test result is selected as the final evaluation result of the model, which violates the established standard that the test set should not influence model selection during the training phase. In this study, we ensured a more systematic comparison of different PEFT methods in a fairer setting. To achieve this, we adhered strictly to a well-established training protocol. After each epoch of training, evaluation was conducted on the validation set, and the checkpoint demonstrating the best performance on the validation set was selected as the final model for testing on the test set. For comprehensive details regarding our reimplementation, please refer to Appendix B.1.
As presented in Table 1, both the RoBERTa base and large models fine-tuned by RED yielded accuracies comparable to those achieved by other PEFT methods across all tasks within the GLUE benchmark. For instance, RED’s performance was only marginally inferior to that of LoRA, differing by a negligible 0.2% with the RoBERTa large and by 0.4% with the RoBERTa base on average, while requiring the tuning of significantly fewer parameters. It is noteworthy that RED demonstrated superior performance on tasks such as SST-2, MRPC, CoLA, STS-B, and RTE, all of which had data sizes less than 100k. This suggests that RED helps to maintain the generalization capability acquired during the pre-training phase and can deliver enhanced performance when the volume of training data is relatively small.
Moreover, RED exhibits an unprecedented level of parameter efficiency. Although having substantially fewer trainable parameters—approximately 7, 200 times fewer than full parameter fine-tuning and 16 times fewer than LoRA—it maintains comparable performance. This observation suggests that there is still potential for further reduction in the number of trainable parameters and this finding aligns with conclusions drawn by Aghajanyan et al. (2020) and Kopiczko et al. (2023).

heading: 4.3 Results with GPT-2
text: Beyond natural language understanding tasks, our study expanded to include experiments on natural language generation tasks. These experiments were conducted using the GPT-2 medium (355M) and large (774M) models on the E2E NLG Challenge (Novikova et al., 2017). The pre-trained GPT-2
models were also obtained from the HuggingFace Transformers Library. To ensure a fair comparison, we reproduced other PEFT methods following the settings defined by Li and Liang (2021a) and Hu et al. (2021). Please refer to Appendix B.2 for the comprehensive details of our implementation.
The data presented in Table 2 reveals that RED achieved performance comparable to other PEFT baselines across all metrics in the E2E NLG Challenge. This highlights the efficacy of fine-tuning through representation editing, not only for natural language understanding tasks but also for language generation tasks. Remarkably, RED accomplished this performance while still necessitating minimal fine-tuning of parameters. To compare RED with the most parameter-efficient variants of other PEFT methods, we set the rank of LoRA and Adapter to 1. Under these conditions, RED outperformed all such variants while still employing the fewest number of parameters (see Section 5.3 for details).

heading: Method # Param AlpacaEval (win %)

heading: 4.4 Results with T5
text: In order to assess the versatility of RED, we conducted experiments using the encoder-decoder architecture. More specifically, we obtained the pre-
trained T5-base model (220M) from the HuggingFace Transformers Library and evaluated models fine-tuned with RED and other PEFT baselines on the GLUE benchmark. As shown in Table 3, RED achieved accuracies that were on par with other PEFT methods across all tasks in the GLUE benchmark, indicating its versatility across diverse network architectures. We excerpted the results from the study of (Asai et al., 2022) as they conducted experiments under the settings similar to ours. The inclusion of their results does not undermine the fairness or outcome of the comparison. Please refer to Appendix B.3 for the implementation details.
It is noteworthy that RED surpassed the prompt tuning (PT) baseline with a significant margin of 11.9% on average. Despite PT methods necessitating the fewest parameters during fine-tuning among other PEFT baselines, they still require a modification of parameters that is twice as large as that required by RED. This finding indicates that allocating a few number of learnable parameters to each layer for representation editing is both more parameter-efficient and effective compared to the PT method, which assigns all learnable parameters to the model’s embedding layer.

heading: 4.5 Results with LLaMA-2
text: We are interested in exploring whether RED can be effectively scaled to large neural networks containing billions of parameters. Therefore, we evaluated RED and compared it against full parameter fine-tuning and LoRA using LLaMA-2 up to 7 billion parameters. Our assessments were conducted across three widely-used benchmarks: Open LLM Leaderboard (Beeching et al., 2023), AlpacaEval (Li et al., 2023), and MT-Bench (Zheng et al., 2023). The implementation details of this experiment can be found in Appendix B.4.
Table 4 presents the win rates obtained by different fine-tuning methods, compared against the reference responses generated by text-davinci-003 on the AlpacaEval. Remarkably, RED yielded the highest win rate, even though its number of trainable parameters was 25, 700 times less than that of full parameter fine-tuning and 32 times less than
that of LoRA. This suggests that the RED method, which directly edits hidden representations during the fine-tuning phase, continues to be effective for large language models, and the trained models are capable of generating high-quality responses.
Moreover, Figure 2 illustrates the performance scores achieved by three training methods on the 1- turn question-answer dataset from MT-Bench. The overall performance of RED is on par with other baseline methods, and it notably excels in assessing the capabilities of humanities and reasoning. As presented in Table 12, RED also delivers competitive results on six distinct datasets from the Open LLM Leaderboard.

heading: 5 Ablation Study
text: In this section, we conducted a series of ablation studies to examine the impact of different editing operations used in RED and to explore potential application points within transformer-based networks. We also evaluated the parameter efficiency of RED by comparing it with the most parameter-efficient variants of other PEFT methods.

heading: 5.1 Impact of Different Editing Operators
text: We introduce two operators to edit the hidden representations of neural network models: one scales a hidden representation using a learnable vector, and the other adds a bias vector to the hidden representation. To assess the contribution of each operator, we performed ablation studies by removing one operator at a time and examining the impact on the models’ performance.
From Table 6, it is evident that both editing operators are crucial for enhancing model performance.
Analyzing results from three different datasets from the GLUE benchmark, we found that the scaling operator boosted accuracy by an average of 1.3%, while the biasing operator contributed to a significant 12.4% increase in accuracy. Compared to the scaling operator, the addition of a learnable bias vector to the original representation plays a more substantial role in the fine-tuning of neural models.

heading: 5.2 Impact of Editing Positions
text: The motivation of this study is to explore the feasibility of fine-tuning large language models with as few parameters as possible. Therefore, in our current implementation, we only consider modifying the representations produced by FFN sub-layers. However, it is still possible to further improve the performance of the model by increasing the number of trainable parameters if the representation vectors of Q (query), K (key), and V (value) in attention blocks are allowed to be modified via representation editing. To demonstrate this possibility, we conducted additional experiments in which the representations of Q, K, and V were also modified.
The experimental results reported in Table 7 and Table 8 show that the performance of RED has been improved on all these datasets but with slightly more trainable parameters (increasing from 0.07M to 0.37M). It suggests that it is feasible to increase the number of trainable parameters to further boost the performance by editing additional feature representations. This strategy is particularly effective for datasets with a larger number of training instances, such as MNLI, QNLI, QQP, and E2E while the performance did not exhibit significant improvement
on datasets with fewer training examples.

heading: 5.3 Parameter Efficiency and Efficacy
text: When using Adaptor or LoRA for fine-tuning, the ranks of their decomposition matrices were typically set to 8 or 16. These ranks can be reduced to decrease the number of parameters used during fine-tuning. To compare RED with the most parameter-efficient versions of Adpator or LoRA, we adjusted the rank of their decomposition matrices to 1. We then made a comparative analysis with GPT-2 medium on the E2E NLG Challenge.
Table 5 shows that RED outperformed all the baselines on four out of five tasks, while it slightly underperformed LoRA by a negligible difference of 0.02 on the NIST dataset. It is worthy noting that RED accomplished this with the minimal number of parameters, demonstrating that RED is not only parameter-efficient but also parameter-effective.

heading: 6 Conclusion
text: In this study, we proposed a novel perspective on fine-tuning by editing model representations rather than adjusting the model’s weights. Through extensive experimentation across various NLP tasks and neural models of varying structures and scales, we demonstrated that RED can deliver performance comparable or superior to existing PEFT methods while employing a minimal number of trainable parameters. We are hopeful that this study could inspire novel methods for fine-tuning LLMs, thereby contributing to the advancement of both effective and computationally efficient PEFT techniques.

heading: Limitations
text: We have demonstrated the efficacy of a new promising PEFT approach for fine-tuning models through direct manipulation of representations across various NLP tasks with networks of varying structures and scales. It would be intriguing to extend this method to other modalities, such as image, speech, and video. Recent advancements in representation engineering have indicated that only a very few examples are sufficient for achieving precise control over the model’s output through representation manipulations. We plan to apply the proposed RED to few-shot scenarios in the future.

heading: Reproducibility Statement
text: We have made our code publicly accessible via a repository on GitHub (github.com/mlwu22/RED). To further ensure replicability, we had a colleague unfamiliar with our method install and test RED. The experiment yielded results nearly identical to ours, which strengthens our confidence that other researchers will successfully execute our code and reproduce our findings.

heading: A Datasets

heading: A.1 GLUE Benchmark
text: The General Language Understanding Evaluation (GLUE) benchmark comprises a variety of datasets, including CoLA (Warstadt et al., 2018), SST-2 (Socher et al., 2013), MRPC (Dolan and Brockett, 2005), QQP (Wang et al., 2018), STS-B (Cer et al., 2017), MNLI (Williams et al., 2017), QNLI (Demszky et al., 2018), and RTE (Bar-Haim et al., 2006). These datasets have been widely-used to measure the performance of language understanding models. The GLUE benchmark was sourced from Huggingface Datasets (Lhoest et al., 2021), and all datasets within this benchmark were employed for our evaluation.
Following (Ding et al., 2023) and (Hu et al., 2021), we evaluated models trained with different PEFT methods on the validation dataset. To ensure a fair and systematic comparison of different PEFT methods, we randomly sampled 1, 000 instances from the validation set to form a new validation set if the validation set contains more than 2, 000 instances (otherwise, we randomly selected half of the instances for the new validation set), and the remaining instances were used as the test set. This allowed us to adhere rigorously to the established training protocol, which involves evaluating on the new validation set after each epoch of training and selecting the checkpoint that yields the best performance on the validation set as the final model for testing on the test set. The sizes of the training, validation, and test sets as well as their corresponding metrics for all datasets are given in Table 9.
For all the experiments with RoBERTa, we ran experiment 5 times for each PEFT method using distinct random seeds to initialize the trainable weights and reported the average results. During the evaluation of the RTE and CoLA datasets, some researchers reported that specific random seeds could potentially lead to anomalous experimental outcomes2. Therefore, we randomly chose other five different random seeds (i.e., 42, 43, 44, 45, and 46) to conduct the experiments.

heading: A.2 E2E NLG Challenge
text: The E2E NLG Challenge was first introduced by Novikova et al. (2017) to train and evaluate end-toend, data-driven natural language generation models and systems. All datasets in the E2E NLG Challenge were sourced from Huggingface Datasets.
2Refer to github.com/microsoft/LoRA/issues.

heading: Dataset #Train #Validation #Test Metric
text: This benchmark includes 42.1K training instances, 4.67K validation instances, and 4.69K testing instances. Following previous studies, we used the official evaluation script to compute BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), METEOR (Banerjee and Lavie, 2005), ROUGEL (Lin, 2004) and CIDEr (Vedantam et al., 2014) scores. For evaluations using GPT-2, we ran experiment for 3 times using distinct random seeds and reported the average results. The random seeds used for these experiments were 42, 43, and 44.

heading: A.3 UltraFeedback
text: UltraFeedback (Cui et al., 2023) comprises 64, 000 prompts, each of them accompanied by four LLM responses evaluated by GPT-4 based on criteria such as instruction-following, honesty, and helpfulness. To construct the training dataset from UltraFeedback, we selected the responses with the highest mean scores3.

heading: A.4 Open LLM Leaderboard
text: The Open LLM Leaderboard includes six datasets that assess various capabilities such as science questions, commonsense inference, multitask accuracy, mathematical reasoning, and the truthfulness of generated answers. It consists of ARC (Mihaylov et al., 2018), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2021), and GSM8K (Hendrycks et al., 2021). We used the Eleuther AI Language Model Evaluation Harness library (Gao et al., 2023) to evaluate language models trained using various methods. Table 11 details the leaderboard evaluation configuration and the experimental settings applied in this study.
3The constructed dataset can be accessed at: huggingface.co/datasets/argilla/ \ultrafeedback-binarized-preferences-cleaned.

heading: A.5 AlpacaEval
text: AlpacaEval is an automated evaluation benchmark for LLMs, employing GPT-4 (OpenAI, 2023) as an annotator to compare the responses generated by the models against reference answers produced by text-davinci-003. The benchmark consists of 805 samples focused on simple instruction-following tasks. Previous research has shown a high correlation between GPT-4’s annotations and human evaluator assessments (Li et al., 2023).

heading: A.6 MT-Bench
text: MT-Bench (Zheng et al., 2023) comprises 80 challenging questions, each structured as a two-turn dialogue. GPT-4 was also used to evaluate the responses generated by the models, assigning a score from 1 to 10 for each turn.

heading: B Hyperparameters

heading: B.1 Experiments with RoBERTa
text: We trained the models using AdamW with a linear learning rate decay schedule. To ensure a fair comparison, we maintained the same sequence length across all baseline PEFT methods. The hyperparameters used in our experiments for RoBERTa base are detailed in Table 13, and for RoBERTa large in Table 15.

heading: B.2 Experiments with GPT-2
text: We trained the models using AdamW with a linear learning rate decay schedule. To ensure a fair comparison, we maintained the same sequence length across all baseline PEFT methods. We used the Huggingface PEFT package to replicate Prefix Tuning and LoRA, and the OpenDelta package to replicate Adapter and Adapter-FFN. The hyperparameters used in our experiments for GPT-2 medium and GPT-2 large are detailed in Table 16s.

heading: B.3 Experiments with T5
text: We trained the models using AdamW with a linear learning rate decay. We conducted experiments on T5 following the settings established by Asai et al. (2022). The hyperparameters used in our experiments for T5 base are detailed in Table 14.

heading: B.4 Experiments with LLaMA-2
text: We employed AdamW with a cosine learning rate decay schedule during the fine-tuning. To ensure a fair comparison, we maintained the same sequence
length across all baseline methods. The hyperparameters used for LLaMA-2 are listed in Table 10. We evaluated the performance of models on TruthfulQA after each training epoch, and the checkpoint demonstrating the best performance was selected for final testing on other datasets. A greedy decoding strategy was employed for generation for all the models. To mitigate repetition issues, we applied a repetition penalty of 1.1 and set the no-repeat n-gram size to 5.

heading: C Results of Standard Deviations
text: We presented the average results in the main text. To provide a quantifiable measure of uncertainty in the measurement or prediction and help other researchers replicate our experiments, we also reported the standard deviations with RoBERTa models on the GLUE benchmark in Table 17, and the standard deviations with GPT-2 on the E2E NLG Challenge in Table 18.

heading: D Additional Results with LLaMA-2
text: Example responses generated by models trained with RAD, full-parameter fine-tuning (FT) and LoRA are presented in Figures 3 and 4. Tables 19 and 12 provide the results on the MT-Bench and Open LLM Leaderboard respectively.

heading: Future Work
text: We plan to apply the proposed RED to few-shot scenarios in the future