ReFT: Representation Finetuning for Language Models

Authors: zhengxuan  wu
aryaman  arora
zheng  wang
atticus  geiger
dan  jurafsky
christopher d manning
christopher  potts
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:00:30

Abstract: Parameter-efficient finetuning (PEFT) methods seek to adapt large neural models via updates to a small number of *weights*. However, much prior interpretability work has shown that *representations* encode rich semantic information, suggesting that editing representations might be a more powerful alternative. We pursue this hypothesis by developing a family of **Representation Finetuning (ReFT)** methods. ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency. Both are drop-in replacements for existing PEFTs and learn interventions that are 15x--65x more parameter-efficient than LoRA. We showcase LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks, instruction-tuning, and GLUE. In all these evaluations, our ReFTs deliver the best balance of efficiency and performance, and almost always outperform state-of-the-art PEFTs. Upon publication, we will publicly release our generic ReFT training library.

Main paper: 
 introduction: 
 Pretrained language models (LMs) are frequently finetuned to adapt them to new domains or tasks [Dai and Le, 2015] . With finetuning, a single base model can be adapted to a variety of tasks given only small amounts of in-domain data. However, finetuning large LMs is expensive. Parameterefficient finetuning (PEFT) methods propose to address the high costs of full finetuning by updating a small number of weights. This reduces memory usage and training time, and PEFTs achieve similar performance to full finetuning in many settings [Hu et al., 2023] .A hallmark of current state-of-the-art PEFTs is that they modify weights rather than representations. However, much prior interpretability work has shown that representations encode rich semantic information, suggesting that editing representations might be a more powerful alternative to weight updates. In this paper, we pursue this hypothesis by developing and motivating Representation Finetuning (ReFT) . Instead of adapting model weights, ReFT methods train interventions that manipulate a small fraction of model representations in order to steer model behaviours to solve downstream tasks at inference time. ReFT methods are drop-in replacements for weight-based PEFTs. This approach is inspired by recent work in LM interpretability that intervenes on representations to find faithful causal mechanisms [Geiger et al., 2023] and to steer model behaviours at inference time [Turner et al., 2023 , Li et al., 2024] , and it can be seen as a generalisation of the representation-editing work of Wu et al. [2024a] , Turner et al. [2023] , and Zou et al. [2023] (see appendix B for details).We focus on a strong and highly efficient instance of the ReFT family that we call Low-rank Linear Subspace ReFT (LoReFT). LoReFT is a parametrisation of ReFT that intervenes on hidden Figure 1 : Parameter count vs. performance for LoReFT and other PEFTs across four benchmarks when applied to LLaMA, Llama-2, Llama-3, and RoBERTa models. Despite training far fewer parameters than existing PEFTs, LoReFT achieves competitive or even state-of-the-art performance on all tasks. Its value is most apparent for the largest models in our evaluations. Note: FT is full-parameter finetuning, which is not a PEFT or ReFT method. Additional results are in section 4. representations in the linear subspace spanned by a low-rank projection matrix, building directly on the distributed alignment search (DAS) method of Geiger et al. [2023] and Wu et al. [2023] . We also identify an ablation of this method (DiReFT) that trades some performance for increased efficiency. We evaluate our ReFTs on LLaMA-family models and small-scale LMs against existing PEFTs on standard benchmarks from four domains: commonsense reasoning, arithmetic reasoning, instruction-following, and natural language understanding. Compared to LoRA, we find that LoReFT uses 15×-65× times fewer parameters while achieving state-of-the-art performance on commonsense reasoning, instruction-following, and natural language understanding against the strongest PEFTs. These findings indicate that ReFT methods are worthy of further exploration, as they may emerge as more efficient and effective alternatives to weight-based PEFTs. 
related work: 
 Parameter-efficient finetuning methods (PEFTs). PEFTs train a fraction of the model's parameters to adapt it to downstream tasks. We classify PEFTs into three categories:1. Adapter-based methods train additional modules (e.g. fully-connected layers) on top of the frozen pretrained model. Series adapters insert components between LM attention or MLP layers [Houlsby et al., 2019 , Pfeiffer et al., 2020 , Wang et al., 2022 , He et al., 2022b , Fu et al., 2021] , while parallel adapters add modules alongside existing components [He et al., 2022a] .Since adapters add new components that cannot be easily folded into existing model weights, they impose an additional burden at inference time.1 2. LoRA [Hu et al., 2022] and DoRA [Liu et al., 2024c] use low-rank matrices to approximate additive weight updates during training, and require no additional overhead during inference since the weight updates can be merged into the model. These are the strongest PEFTs currently. 2 3. Prompt-based methods add randomly-initialised soft tokens to the input (usually as a prefix) and train their embeddings while keeping the LM weights frozen [Li and Liang, 2021] . These methods are often far from optimal compared to other PEFTs, and come at the cost of significant inference overhead. A variant of this method where hidden-layer activations are also tuned was introduced as a baseline in Hu et al. [2022] , with better performance.Representation editing. Recent work on activation steering and representation engineering shows that adding fixed or task-specific steering vectors [Subramani et al., 2022 , Turner et al., 2023 , Zou et al., 2023 , Liu et al., 2024b , Vogel, 2024 , Li et al., 2024] or applying concept erasure [Ravfogel et al., 2022 , Belrose et al., 2023 , Avitan et al., 2024 , Singh et al., 2024] to the residual stream can enable a degree of control over pretrained LM generations without the need for resource-intensive finetuning [Wu et al., 2024a] . The success of these methods affirms that representations induced by pretrained LMs carry rich semantic structure.Interventional interpretability. Much recent work has used interventions on model-internal states to test hypotheses about how LMs implement various behaviours. In particular, interventions on linear subspaces of representations have provided increasing evidence that human-interpretable concepts are encoded linearly [Smolensky, 1986 , Rumelhart et al., 1986 , McClelland et al., 1986] . This includes linguistic features such as gender and number [Lasri et al., 2022 , Wang et al., 2023 , Hanna et al., 2023 , Chintam et al., 2023 , Yamakoshi et al., 2023 , Hao and Linzen, 2023 , Chen et al., 2023 , Amini et al., 2023 , Guerner et al., 2023 , Arora et al., 2024] , logical and mathematical reasoning [Wu et al., 2023] , entity attributes [Huang et al., 2024] , and a number of other domains [Mikolov et al., 2013 , Elhage et al., 2022 , Park et al., 2023 , Nanda et al., 2023 , Guerner et al., 2023] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Yes, we include a standalone limitation section in the paper. Throughout our paper and appendix, we highlight our assumptions, reflect on the scope of our claims, factors that influence the performance of our approach and computation efficiency of our methods.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
reft: 
 We now define the ReFT family of methods. To do this, we first summarize the core motivation, which emerges from work on intervention-based model interpretability. We then show how this leads directly to Low-rank Linear Subspace ReFT (LoReFT). Finally, we generalize this to a family of ReFT methods. Appendix A provides a brief overview of our generic ReFT training library.To keep the presentation simple, we assume throughout that our target model is a Transformerbased [Vaswani et al., 2017] LM that produces contextualised representations of sequences of tokens. Given a sequence of n input tokens x = (x 1 , . . . , x n ), the model first embeds these into a list of representations h (0) = (hEQUATIONn ). Then, m layers successively compute the j-th list of hidden representations h (j) as a function of the previous list of hidden representations h (j-1) . Each hidden representation is a vector h ∈ R d . The LM uses the final hidden representations h (m) to produce its predictions. In our experiments, we consider both autoregressive LMs and masked LMs [Devlin et al., 2019 ]. An autoregressive LM predicts p(x n+1 | x 1 , . . . , x n ) = softmax (Wh (m) n ), while a masked LM predicts p(x i | x 1 , . . . , x i-1 , x i+1 , . . . , x n ) = softmax (Wh (m) i), where W is a learned matrix mapping from representations to logits over the vocabulary space.In interpretability research, the framework of causal abstraction [Geiger et al., 2021] uses interchange interventions to establish the causal role of representations in deep learning models. An interchange intervention fixes a representation to the value it would take if a counterfactual input were processed by the model. Experiments investigating how such interventions affect model behavior form the evidence for claims about the causal role of a representation and the concept it encodes.To test whether a concept is encoded in a linear subspace of a representation, one may use a distributed interchange intervention (DII) [Geiger et al., 2023] . 3 Let h b be the hidden representation created at row i and column k when our model processes input b, and let h s be the corresponding representation when that same model processes input s. A distributed interchange intervention on h b given a counterfactual source representation h s is then defined asEQUATIONwhere R ∈ R r×d is a low-rank projection matrix with orthonormal rows, d is the representation dimensionality, and r is the dimensionality of the subspace we are intervening on. We learn the subspace R using distributed alignment search (DAS), which finds the subspace that maximises the probability of the expected counterfactual output after intervention [Geiger et al., 2023] . DAS isReFT Intervention this is some textLoReFT LoReFT highly expressive, and can effectively localize concepts within model representations [Wu et al., 2023 , Arora et al., 2024 , Wu et al., 2024c , Huang et al., 2024] . This suggests that subspace representation interventions could also be a powerful tool for model control.LoReFT. The formulation of DII in eq. ( 1) immediately suggests a way to control model generations via interventions. The guiding intuition is that we can learn how to perform interventions that steer the model towards predicting our task labels. The resulting method, Low-rank Linear Subspace ReFT (LoReFT), is defined by the following variant of eq. ( 1):EQUATIONThis is identical to eq. ( 1), except we use a learned projected source Rs = Wh + b. LoReFT thus edits the representation in the r-dimensional subspace spanned by the rows of R to take on the values obtained from our linear projection Wh + b. We depict this operation in fig. 2 . The learned parameters are ϕ = {R, W, b}; the parameters of the LM are frozen. As with DII, R ∈ R r×d is a low-rank matrix with orthonormal rows where d is the hidden-state dimensionality and r ≤ d is the rank of the subspace. We further define a linear projection W ∈ R r×d and bias vector b ∈ R r .DiReFT. In addition, we define an ablation of LoReFT which removes the orthogonality constraint and the difference operation, reducing training time:EQUATIONBoth W 1 , W 2 ∈ R r×d are low-rank projection matrices. Note that eq. ( 3) resembles LoRA, and thus DiReFT can be thought of as LoRA applied directly to hidden representations at certain positions. 4Empirical evidence from previous work suggests that adding orthogonal constraints to LoRA weights increases performance [Liu et al., 2024d] . (Appendix E reports results for additional ablations of LoReFT.)Training objective. We consider both generation tasks using decoder-only or encoder-decoder LMs and classification tasks using encoder-only models with m layers. The pretrained language model induces a distribution over token sequences p(⋅). We denote the model that results from the ReFT intervention Φ on p(⋅) as p Φ (⋅) with trainable parameters ϕ. To simplify notation, we refer to the hidden representations produced by the LM on input x as h(x), and those by the intervened LM as h Φ (x).For generation tasks, our training objective is language modelling. Given an input sequence x = (x 1 , . . . , x n ) with n tokens as the prompt, the goal is to predict the output sequence y = (y 1 , . . . , y k ) with k tokens. We minimise the cross-entropy loss with teacher-forcing over all output positions.min ϕ {- k ∑ i=1 log p Φ (y i | xy <i )} (4)For single-label classification tasks, we add a classification head H θ (⋅) with parameters θ that takes the final-layer (i.e., layer m) representation at the first token (CLS) h (m) 1as input and outputs a distribution over classes. H has the learned parameters θ= {W o , b o , W d , b d }. H θ (⋅ | h) = softmax (W o (tanh(W d h (m) 1 + b d )) + b o ) (5)We learn the parameters of the head and those of the intervention function Φ. We minimise the cross-entropy loss of the target class y given input x:min ϕ,θ {-log H θ (y | h Φ (x))} (6)It is straightforward to generalise the above intervention functions to define a family of interventionbased representation finetuning methods. We first define a general notion of intervention, i.e. the modification of hidden representations during the model forward pass: Definition 3.1. An intervention I is a tuple ⟨Φ, P, l⟩ that encapsulates a single inference-time modification of the representations computed by a Transformer-based LM. The three components of an intervention are (1) the intervention functionEQUATIONa set of input positions P ⊆ {1, . . . , n} that the intervention is applied to, and (3) the layer l ∈ {1, . . . , m} at which the intervention is applied.We implement the intervention I as the following operation that overwrites some representations h:EQUATIONThe intervention is applied immediately after the computation of h (l) and thus affects the representations computed in later layers h (l+1) , . . . , h (m) .Figure 2 provides a schematic overview of an intervention. A ReFT is then defined as a constrained set of non-overlapping interventions: Definition 3.2. A ReFT method is a set of f interventions I = {I 1 , . . . , I f }. We enforce that for any two interventions I j , I k ∈ I such that they operate on the same layer l j = l k , their intervention positions must be disjoint, i.e. P j ∩ P k = ∅. The parameters (ϕ 1 , . . . , ϕ f ) of all of the intervention functions are independent.ReFT is thus a generic framework encompassing interventions on hidden representations during the model forward pass. In appendix B, we show how a variety of existing inference-time intervention methods can be described within this framework.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Yes, we include hardware requirement in the main text.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
experiments: 
 To evaluate our ReFTs against existing PEFTs, we conduct experiments across four diverse NLP benchmarks covering more than 20 datasets (extensive details on our datasets are in appendix C).Our goal is to provide a rich picture of how LoReFT and DiReFT perform in different scenarios. We experiment with both masked and autoregressive LMs at different scales, ranging from RoBERTabase [Liu et al., 2019] with 125M to LLaMA models [Touvron et al., 2023a,b] with 13B parameters. We benchmark against existing PEFTs such as prefix-tuning [Li and Liang, 2021] , adapter-tuning with both Series Adapters and Parallel Adapters, BitFit [Ben Zaken et al., 2022] , RED [Wu et al., 2024a] , LoRA [Hu et al., 2022] , and DoRA [Liu et al., 2024c] . Our comparisons focus on both performance and parameter efficiency. In our comparisons, we use hyperparameter-tuned scores from previous works when possible. We load our base LMs in torch.bfloat16 to save memory. All of our experiments are run with a single GPU: NVIDIA A100 40G/80G or RTX 6000. Examples of raw model generations are in appendix I. The performance results of all baseline methods are adapted from Liu et al. [2024c] and represent the best performance achieved after hyperparameter tuning.For our experiments, we must decide how many interventions to learn and which layers and input positions to apply each one on. We propose learning interventions on a fixed number of p prefix and s suffix positions in the prompt. Specifically, we tune four hyperparameters:1. The number of prefix positions p to intervene on, i.e. positions {1, . . . , p}.2. The number of suffix positions s to intervene on, i.e. positions {n -s + 1, . . . , n}.3. Which set of layers L to intervene on. 4. Whether or not to tie intervention parameters ϕ across different positions in the same layer.This simplifies the hyperparameter search space; compared to LoRA, the only additional consideration is which positions to intervene on. Since the number of positions edited is constant, LoReFT and DiReFT contribute a fixed additional inference cost that does not scale with prompt length.Given the positions P = {1, . . . , p} ∪ {n -s + 1, . . . , n}, we define the untied and tied variants: We also tune neural-network training hyperparameters.I untied = {⟨Φ,Unlike previous work [Hu et al., 2022 , 2023 , Liu et al., 2024c] where hyperparameter tuning may involve optimising performance directly on test sets, we only tune our hyperparameters on development sets which do not contain any overlapping examples with the test sets of our tasks. We further describe hyperparameter tuning for each benchmark in appendix D.1.We replicate the experimental setup in Hu et al. [2023] and finetune LLaMA-1 7B/13B, Llama-2 7B, and Llama-3 8B5 on a combined dataset of eight commonsense reasoning tasks (COMMONSENSE170K). We report scores on each task's test set individually. We compare with PEFTs benchmarked in Hu et al. [2023] as well as the identical experiment reported in Liu et al. [2024c] for DoRA.Datasets. Our benchmark contains eight commonsense reasoning datasets, including BoolQ [Clark et al., 2019] , PIQA [Bisk et al., 2020] , SIQA [Sap et al., 2019] , HellaSwag [Zellers et al., 2019] ,WinoGrande [Sakaguchi et al., 2021] , ARC-e, ARC-c [Clark et al., 2018] , and OBQA [Mihaylov et al., 2018] . Examples are formulated as multiple-choice problems where the model needs to directly generate the correct choice without rationales. We use the same prompt template as in Hu et al. [2023] with additional string normalisation (removing leading and trailing whitespace).Hyperparameter tuning. We do not do hyperparameter selection based on test set results. Rather, we use the hyperparameter settings of the model that performs best on a development set created from the GSM8K training set, except we use a lower number of epochs (6 instead of 12) because the COMMONSENSE170K training set is more than 20 times larger than GSM8K. This allows us to tune relevant hyperparamters, and also serves to test the robustness of these settings across different domains. We additionally report scores on 3 epochs in appendix D.3.Results. We report results in table 1 . LoReFT sets state-of-the-art performance on the commonsense reasoning tasks, outperforming all other methods by a considerable margin. While being more compute-efficient, DiReFT achieves only slightly worse performance consistently.Similar to the previous experiment, we follow the experimental setup in Hu et al. [2023] and finetune LLaMA-1 7B and 13B on a combined dataset of seven arithmetic reasoning tasks with LM-generated chain-of-thought steps (MATH10K) and report scores on four of the tasks' test sets. We only evaluate correctness on the final numeric or multiple-choice answer. Hyperparameter tuning. We use the same hyperparameter settings as for the Commonsense Reasoning benchmark, but with 12 epochs for training. We also report scores on 3 epochs.Datasets. Our benchmark contains four datasets for math world problems, including AQuA [Ling et al., 2017] , GSM8K [Cobbe et al., 2021] , MAWPS [Koncel-Kedziorski et al., 2016] , and SVAMP [Patel et al., 2021] . Models need to generate chain-of-thought [Wei et al., 2022] before the final answer. We use the same prompt template and hyperparameter settings as in the previous experiment.Results. We report results in table 2 . We find that both LoReFT and DiReFT do not perform as well at arithmetic reasoning tasks compared to LoRA and adapters, but do outperform prefix-tuning.Our results suggest that our ReFTs may have more trouble on chain-of-thought reasoning than the single-step commonsense reasoning tasks due to the length of generations (greater length necessarily reduces the effect of the intervention) and overall greater difficulty of the task. Our results show that our ReFTs perform better with the 13B model than the 7B model, which suggests that our methods scale with model size. Overall, we note that the arithmetic reasoning results show a lot of variation, with no single method emerging as a clear winner across all of them.Base LMs require instruction finetuning to follow human prompts [Ouyang et al., 2022] . We follow the experimental setup in Wu et al. [2024a] and finetune Llama-2 7B with Ultrafeedback [Cui et al., 2023] . We compare against full parameter finetuning, LoRA, and RED. For evaluation, we use Alpaca-Eval v1.0 [Li et al., 2023] , which computes the win-rate against text-davinci-003 using GPT-4 as the annotator. We use the same prompt template as in Taori et al. [2023] .Datasets. Ultrafeedback is high-quality instruction dataset where responses are generated via scoring a diverse set of model responses from a list of candidates (e.g. ChatGPT and Bard). The score is calculated as a weighted score of instruction-following, truthfulness, honesty, and helpfulness. Some of the best 7B and 13B chat-models (e.g. UltraLM-13B [Ding et al., 2023] ) are finetuned with Ultrafeedback.Hyperparameter tuning. We do hyperparameter-tuning on the unseen instruction-following dataset Alpaca-52K [Taori et al., 2023] with only LLaMA-7B to prevent test-set hill-climbing. We then use the hyperparameter settings of our best performing model to finetune on Ultrafeedback. For hyperparameter tuning, we use Alpaca-Eval v1.0 with GPT-4 turbo as the annotator for fast turnaround, which also prevents overfitting with GPT-4 as a judge.Results. We report results in table 3. When matched in parameter count to the previous most parameter-efficient PEFT (RED) and trained on Llama-2 7B, LoReFT outperforms all reported finetuning methods (including full finetuning) and achieves a win-rate within 1% of GPT-3.5 Turbo 1106. Furthermore, after halving the parameter count or using only 1/64-th of the data, LoReFT still outperforms other finetuning methods. This result shows that LoReFT can succeed at long-form text generation. DiReFT is again slightly worse than LoReFT but is highly competitive. We evaluate LoReFT on the GLUE benchmark [Wang et al., 2018] against existing PEFTs. We use this set of experiments to show LoReFT works well even with small-scale LMs, and can improve representations for classification tasks and not just text generation. We finetune RoBERTa-base (125M) as well as RoBERTa-large (350M) on GLUE, a sequence classification benchmark for natural language understanding (NLU) which covers domains such as sentiment classification and natural language inference. Details about the GLUE benchmark can be found in its original paper. We follow Wu et al. [2024a] for proper evaluation on GLUE validation set: we split the validation set into two sets guarded by a random seed, and we pick the best model with highest in-training validation accuracy to evaluate on the other held-out half for testing accuracy.Hyperparameter tuning. We tune our hyperparameters for each task separately. which is standard for PEFTs. To avoid overfitting to random seeds, we hyperparameter-tune our models with a constant seed, and report averaged results over that and four additional unseen seeds. We describe hyperparameter tuning experiments in Appendix D.1.Results. We report results in table 4 . LoReFT obtains comparable performance with PEFT methods on both model sizes when parameter matched with RED, the previous most parameter-efficient PEFT for this task. Furthermore, DiReFT achieves worse performance than most of the PEFTs suggesting LoReFT is a better choice when LM is small. Full results with standard deviation is in table 13 . We additionally compare against VeRA [Kopiczko et al., 2024] in appendix D.3. 
limitations: 
 Due to limited resources, we mainly explored the LLaMA-family of models. In future work, we hope to explore the effectiveness of ReFT on other model families as well as vision-language models such as LLaVA [Liu et al., 2024a] . The capabilities of ReFT have not yet been fully explored due to the large hyperparameter search space; we are interested in automating this search. We provide some initial explorations of LM personalisation with ReFT in a few-shot setting in appendix G.2. We hope to explore why ReFT works, and we provide some of our early explorations focused on memorisation (appendix F.1, appendix F.2). We are also investigating whether learned orthogonal subspaces can be composed together without adaptation. Some encouraging initial findings are in appendix G.1. 
conclusion: 
 We propose a strong alternative to PEFTs, LoReFT, and we identify an ablation of this method, DiReFT, that trades some performance for increased efficiency. Overall, LoReFT achieves strong per-formance across benchmarks from four domains while being 15×-65× more efficient than LoRA. Notably, LoReFT establishes new state-of-the-art performance on commonsense reasoning, instructionfollowing, and natural language understanding against the strongest PEFTs. We also show how our method can be described under a generic framework -ReFT. ReFT is a new approach to finetuning that is more powerful, more parameter-efficient, and more interpretable than any existing PEFTs.A pyreft: A ReFT-native Python Library The wrapped model can be trained for downstream tasks. We also provide data loading helpers to construct training data that is compatible with HuggingFace trainers:  
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Yes, the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our method is a finetuning technique for large language models which does not have negative societal impact that needs to be addressed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: We use publicly available models and datasets for finetuning. When releasing our finetuned artifacts, we will clearly state that existing rules for the finetuning model also apply to our finetuned artifacts. In related experiments, we will also clearly state that our finetuned model should not be misused and should only be used for research purposes.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We list licenses for used datasets and models in appendix J.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Yes, we provide an anonymized zip file for our code.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To show the expressivity of the ReFT framework, we cast existing representing-editing methods in the literature into ReFTs.General comments about expressivity of ReFT. Given that previous works have unified PEFTs under a single framework [He et al., 2022a] , one may ask why not express ReFT as a PEFT method? The main reason is that PEFT frameworks lack the notion of time or sequence (see the unified PEFT view provided in Table 1 on pg. 5 of He et al., 2022a) . In PEFTs, representation modifications are necessarily applied to every token in the sequence, even in recent variants such as AdaLoRA [Zhang et al., 2023] . A key aspect of ReFT is that it leverages representations over time and intervenes only on a small number of them while being effective. More importantly, the notation of time is important for future versions of ReFT that intervene on representations schematically (e.g. intervene on the first token at some early layers and then intervene on the last token at some later layers). The ability to intervene at different layer and position combinations schematically is also supported in our code. Existing PEFT libraries7 enforce weight-based updates without supporting flexible representation-based interventions.RED [Wu et al., 2024a ] is a simple representation-editing method that applies an element-wise scaling transform s ∈ R n and adds a bias b ∈ R n to the hidden representation in every layer. The same intervention is applied to every position (including at generated tokens, increasing inference burden) but separate interventions are learned at each layer. In the ReFT framework, RED is defined asEQUATIONThe parameters ϕ RED = {s, b} are learned with gradient descent to minimise a loss function such as language-modelling loss or a classification loss, as in our experiments with LoReFT. We believe that RED is better classified as a kind of adapter due to its application at all positions.Activation addition [Turner et al., 2023] takes the difference in activations at at some positions p and q and layer l given two contrastive prompts x + and x -as input. It then adds this difference vector, scaled by a tuned constant c, to representations at all positions in layer l for some new prompt.EQUATIONZou et al. [2023] introduce several intervention methods for controlling model behaviour, which they term representation engineering.First, given a set of prompts {x 1 , . . . , x n } designed to elicit the presence of a concept, we randomly pair them, take the difference in activations for each pair, and find the first principle component of the difference vectors at the last token position in some layer of interest l to obtain a reading vector:EQUATIONOne can also used a more structured pairing of constrastive prompts to obtain a contrast vector, similar to the difference vector computed in activation addition:a contrast = PCA ({h(x + i ) (l) -1 -h(x - i ) (l) -1 | 1 ≤ i ≤ n}) 1 (14)Then, using either a reading or a contrast , RepE introduces three operators (i.e. parametrisations of Φ) for intervening on activations:EQUATIONThe first two of these are similar to activation addition, while the latter is a scaled one-dimensional distributed interchange intervention that is a special case of LoReFT. These operations are then used to intervene on some set of positions P ⊆ {1, . . . , n} in the layer of interest:EQUATIONRepE introduces another model control method called Low-Rank Representation Adaptation (LoRRA), which is a kind of PEFT rather than a ReFT since it tunes model weights using a variant of LoRA.We train and evaluate our models on eight datasets covering different domains of open-ended QA tasks:1. The BoolQ [Clark et al., 2019] dataset, which is a question-answering dataset for yes or no naturally occurring questions. We remove the provided passage in the dataset following previous works to ensure a fair comparison.2. The PIQA [Bisk et al., 2020] dataset, which tests physical commonsense reasoning and requires the model to choose one of the provided actions to take based on a hypothesised scenario.3. The SIQA [Sap et al., 2019] dataset, which focus on reasoning about people's actions and their corresponding social consequences.4. The HellaSwag [Zellers et al., 2019] dataset, which asks the model to choose an appropriate ending (or sentence completion) given a context.5. The WinoGrande [Sakaguchi et al., 2021] dataset, inspired by Winograd Schema Challenge [Levesque et al., 2012] , asks the model to fill-in-a-blank with binary options given a sentence which requires commonsense reasoning.6. The ARC Easy set (ARC-e [Clark et al., 2018] ), which includes genuine grade-school level multiple-choice science questions 7. The ARC Challenge set (ARC-c) [Clark et al., 2018] ), which is like ARC-e but designed in a way that co-occurrence methods are expected to fail to answer correctly.8. The OBQA [Mihaylov et al., 2018] We train and evaluate with seven datasets covering different domains of math world problems:1. The AddSub [Hosseini et al., 2014] dataset, which involves solving arithmetic word problems that include addition and subtraction.2. The AQuA [Ling et al., 2017] dataset, which formulates algebraic word problems as multiple-choice problems.3. The GSM8K [Cobbe et al., 2021] dataset, which consists of grade-school math word problems that require multi-step reasoning. We follow Wu et al. [2024a] for proper evaluation on the GLUE validation set. We split the validation set into two subsets, using one subset guarded by a random seed for in-training evaluation and the other for testing. Specifically, after each training epoch, we evaluate the model on our in-training evaluation set and select the best model across all epochs for testing. For datasets with a large validation set (i.e., QQP, MNLI, and QNLI), we select 1,000 samples for in-training evaluation. For the remaining smaller datasets, we select half of the samples for this purpose. For the evaluation metric, we use the Matthews correlation coefficient for CoLA, the Pearson correlation coefficient for STS-B, and accuracy for the other datasets. For MNLI, we report results only on the matched version.Commonsense reasoning and arithmeric reasoning. We create a standalone development set by taking the last 300 examples from the GSM8K training set. We train our models with the remaining training set of GSM8K and select the hyperparameter settings based on model performance on the development set. We select the hyperparameters using LLaMA-7B, and apply the same settings to LLaMA-13B without additional tuning. We use a maximum sequence length of 512 for training and hyperparameter tuning, and a maximum new token number of 32 for inference. During inference, we use greedy decoding without sampling for the commonsense reasoning benchmark, since it is a multi-token classification benchmark, and use the same decoding strategy as in Hu et al. [2023] for the arithmetic reasoning benchmark with a higher temperature 0.3. The reason to switch to a slightly different set of decoding hyperparameters is that the HuggingFace decoding function may throw an error due to statistical instability with close-to-zero probabilities over output tokens with beam search.8 Instruction following. We finetune LLaMA-7B on Alpaca-52K [Taori et al., 2023] to select hyperparameters. We select the hyperparameter settings based on model performance evaluated with Alpaca-Eval v1.0 [Li et al., 2023] , which calculates the win-rate over text-davinci-003 by using gpt-4-turbo as the annotator. We use a maximum sequence length of 768 for training and hyperparameter tuning, and a maximum new token number of 2048 for inference. Table 7 describes our hyperparameter search space.During inference, we use the same decoding strategy as in RED [Wu et al., 2024a] to ensure a fair comparison. Specifically, we use greedy decoding without sampling, and use a maximum repetition n-gram size of 5 with a repetition penalty of 1.1.Natural language understanding. We conduct hyperparameter tuning with RoBERTa-base and RoBERTa-large for each task individually. We pick the hyperparameters based on testing performance on the held-out validation set with a fixed random seed of 42. We then evaluate our model with additional four unseen seeds {43, 44, 45, 46} for final results. We follow Wu et al. [2024a] 's setting for evaluation. For QQP with RoBERTa-large, there are some stochasticity in runs with the same seed, so we picked the best run out of 3 runs for any particular seed. As reported by Wu et al. [2024a] , we also observe that evaluation results on RTE are unstable due to the small size of the dataset. We thus replace several random seeds as in Wu et al. [2024a] to ensure a fair comparison. In addition, we replace one or two random seeds for CoLA for stability. Table 8 describes our hyperparameter search space. Table 9 to table 12 describe our hyperparameter settings for each task. We set the maximum sequence length to 256 for all tasks as in Wu et al. [2024a] .We conduct separate hyperparameter tuning for LoReFT and DiReFT to ensure a fair comparison. 2;4;6;10;12;14;18, 10;12;14;18;20;22;24;28, 4;6;10;12;14;18;20;22, all} Dropout {0.00, 0.05} Optimizer AdamW LR {9×10 -5 , 1×10 -4 , 3×10 -4 , 6×10 -4 , 9×10 -4 , 1×10 -3 , 3×10 -3 } Weight decay {0, 1×10 -3 , 2×10 -3 , 6×10 -3 , 1×10 -2 , 2×10 -2 , 6×10{1×10 -4 , 2×10 -4 , 3×10 -4 , 4×10 -4 , 5×10 -4 }, {6×10 -4 , 9×10 -4 , 1×10 -3 , 3×10 -3 } Weight decay {0, 1×10 -4 , 6×10 -4 , 1×10 -3 , 6×10 -3 , 1×10 -2 , 2×10 -2 , 4×10 -2 }Batch size {16, 32, 64, 128}Warmup ratio {0, 5×10 -3 , 6×10 -3 , 3×10 -2 , 5×10 -2 , 6×10 -2 , 1×10 -1 , 2×10 6×10 -4 6×10 -4 3×10 -4 4×10 -4 9×10 -4 6×10 -4 9×10 -4 6×10 -4Weight decay 0.00Warmup ratio 6×10 -2 1×10 -1 0 5×10 Similar to PEFTs or finetuning, ReFT can be sensitive to hyperparameter settings. Here, we recommand a non-exhaustive list for choosing the best hyperparameter settings for your tasks:• Intervening on multiple positions delivers significant gains. We find that intervening only on a single token position (e.g., just the first one or the last one) is always less optimal than intervening on multiple tokens. However, intervening on excessive number of tokens might harm performance by slowing down convergence. • Intervening on all layers first, and then shrink down. Intervening on all layers often provides a good baseline. We recommand users to start with all layers, and shrink down the number of intervening layers depending on the desired performance-parameter count balance. • Higher rank may not entail better performance. High rank entails higher parameter count, but it does not always bring performance gain (likely due to slower convergence). We recommend users to start with a rank that is lower than 32 (e.g. rank 4). • Tie intervention weights as much as you can. In the paper, we explore tying the intervention weights between prefix and suffix token positions. It automatically halves the parameter count, and it can result in better performance as well. We suspect weight sharing across layers may also help. • Hyperparameter tuning with learning rate, warmup ratio, dropout rate and weight decay should go after other hyperparameters. These classic neural-network training hyperparameters can play a role, yet they have much smaller effect than previous ones.As a result of our hyperparameter searching process, LoReFT is trained with more epochs compared to LoRA [Hu et al., 2022] or DoRA [Liu et al., 2024c] . This raises the concern whether our performance gain is purely due to the larger number of epochs. We thus rerun our experiments with the exact same number of epochs and effective batch size as LoRA or DoRA. Results are shown in table 14 and table 15 . With matched hyperparameters, LoReFT shows similar results by outperforming previous methods significantly on eight commonsense reasoning datasets.Recently, VeRA was proposed as a new variant of LoRA that further reduces the number of trainable parameters while maintaining performance [Kopiczko et al., 2024] . Table 16 shows our results compared against VeRA as well as the baseline numbers reported in VeRA's paper. We include this set of results in the appendix, given that the hyperparameter tuning process is drastically different from ours. 9 The original VeRA implementation records the performance of the best epoch on the validation set, which could cause overfitting since results are selected based on test set performance. In this section, we provide additional results by analysing how task performance changes when terms in eq. ( 2) are ablated. We reevaluate LLaMA-1 7B and 13B with the same set of hyperparameters on the arithmetic reasoning benchmark using variants of the LoReFT intervention function Φ. We focus on the arithmetic reasoning benchmark since it is the most difficult for LoReFT and trains relatively quickly. We conduct experiments with the following parametrisations: As shown in table 17 , variants with a similar number of trainable parameters also achieve similar performance to LoReFT across two models.1. Φ(h) = h + W ⊺ 2 (W 1 h + b -W 2 h) where both W 1 , W 2 ∈ RIn this section, we explore the power of LoReFT through a memorisation test. Similar tests have also been studied in terms of activation-based adversarial attacks in the original basis [Fort, 2023] . Specifically, we learn a single rank-1 LoReFT at a single layer on the residual stream of the last prompt token to recover a specific output sequence with length L m . For simplicity, we simplify LoReFT in Eqn. 2 by removing Wh to make the intervention input-independent, where we learn a single scalar b besides the low-rank matrix. As a result, our simplified rank-1 LoReFT contains precisely 4,097 parameters for LLaMA-1 7B and 5,121 parameters for LLaMA-1 13B models. 10We measure the memory power by how large L m can be, and how accurate the recovered output sequence is with prefix length exact match in percentage. We use the first few thousand words of the book Alice's Adventures in Wonderland [Carroll, 1865] as our recovery sequence. Our prompt is constructed as ALIC#ID1-> followed by model generations. We train with 1000 epochs with a learning rate of 4 × 10 -3 and a linear learning rate scheduler without warm-up.As shown in fig. 3 and fig. 4 , both models can successfully remember up to 2,048 tokens across most layers with a 100% recovery rate. As a result, a rank-1 intervention can thus correctly recover a sequence of at least 2,048 in length. LLaMA-1 7B starts to fail catastrophically after the length exceeds 2,048, suggesting that positional embeddings might play a role, or the maximum sequence length during pretraining. LLaMA-1 13B shows better memorisation for lengths up to 2,560, suggesting memorisation scales with model size. Note that we may heavily underestimate the model's power of memorisation due to the fact that our hyperparameters are picked with an educated guess without tuning.From fig. 5 to fig. 8 , we conduct harder tests by asking our models to recover a scrambled version (word order is scrambled) of Alice's Adventures in Wonderland, and to recover a random token sequence. Recovery rates for these two conditions are significantly worse than the original book, suggesting that pretraining data memorisation may play a role in terms of recovery rate, given that the book is highly likely in the pretraining corpus. Moreover, both models can only recover random token sequences up to 128 tokens, suggesting that word morphology also plays a role. Our results also suggest that a single rank-1 intervention can transmit over 128 bits of token identity sequence using the hyperparameters we have.11 F.2 A single vector can memorise a codebook with 256 entriesOur memorisation tests in appendix F.1 test how long of a sequence we can encode in a rank-1 intervention. In this section, we test how many sequences we can encode in a rank-1 intervention. Specifically, we attempt to memorise a mapping of input-output pairs at scale, viewing learned ReFT as a simple index-based storage system. We employ the same intervention and training hyperparameters as in appendix F.1, but with a different training dataset. Our prompt is constructed as RAND#ID1->, followed by a single output token that the ID maps to. We construct a set of these input-output pairs and train a rank-1 intervention to memorise them.We present our results in fig. 9 and fig. 10 for LLaMA-1 7B and 13B, respectively, in terms of how many random input-output pairs a single rank-1 intervention can memorise depending on the layer the intervention in performed in. Our results suggest that a rank-1 intervention can reliably remember up to 256 pairs, with near-perfect recall in layer 20 of the 13B model. Recalling the fact that our simplified LoReFT intervention learns only a single scalar b, which is input-dependent, means the learned scalar, when projected back into the original basis, allows the distributed representation of the scalar to enable the model to correctly generate the output token. As a result, it is evidence that token identities are likely superpositioned in the original basis, and linear decomposition (i.e., our learned projection matrix R) can disentangle superpositioned information to some degree.G.1 Multi-task learning: Learned ReFTs are like puzzle piecesVarious works have studied how to merge model weights, or PEFT weights together to achieve multi-task learning (MTL) without adaptation [Li et al., 2022 , Huang et al., 2023 , Zhang et al., 2024a , Zhong et al., 2024] . Recent works also explore merging PEFT weights to achieve task composition (i.e., generalise to unseen tasks) by detoxifying an instruction-tuned LM [Huang et al., 2023 , Zhang et al., 2024a] . Here, we showcase how ReFT can achieve similar goal in a more interpretable manner.More importantly, we focus on compositional use of learned abilities (i.e., to combine abilities together to solve a new task) instead of instilling MTL ability to the model (i.e., to solve different tasks). Recall eq. ( 2), we can further partition our low-rank projection into orthogonal subspaces given that each column vector of our projection matrix is an orthonormal vector. Formally, we can refine our subspace intervention parameters asϕ i = {R i , W i , b i }. R ∈ R ri×dwhere i denotes a subspace partition with a dimension of r i . Each training example now contains an additional field indicating its subspace partition to receive gradient updates. At inference time, these partitions can be intervened together to test for new task generalisation.Experiment design. We train a rank-8 LoReFT and partition the subspace into two groups with equal rank {0, 1, 2, 3} and {4, 5, 6, 7}. We train the first group for completing any sentence in German. Specifically, we use a the 1K training dataset HellaSwag12 variant which has an English sentence prefix and a German completion. We train the second group for instruction following. Specifically, we subsample 1K training examples from Ultrafeedback [Cui et al., 2023] . There is no hyperparameter tuning. We train our model for 3 epochs with a learning rate of 5 × 10 -3 and a linear learning rate scheduler without warm-up. Training finishes under 10 mins with a single Nvidia A100 40G GPU. 13Qualitative results. We provide some qualitative results below.How can we reduce air pollution?Model Response LLaMA-7B (LoReFT, German completion subspace), die Luft wird sauber gemacht.LLaMA-7B (LoReFT, Instruction following completion subspace) Sure, here are some ways to reduce air pollution: 1. Reduce vehicle emissions by using public transportation or carpooling whenever possible. 2. Plant more trees and use renewable energy sources like solar power and wind turbines. 3. Recycle and reuse materials instead of throwing them away. 4. Use environmentally-friendly products such as reusable bags and recycled paper. 5. Avoid burning fossil fuels for heating and cooking. 6. Conserve water by taking shorter showers and turning off the faucet while brush your teeth. G.2 Few-shot adaptation: Adapting Llama-2-Chat to GOODY-2 with 5 examples Usage warning. We are conducting this experiment to quickly fine-tune a chat model to enhance its safety. This tool should be used solely for developing safe and robust models. GOODY-214 is an extremely responsible chatbot-it refuses to answer almost any query and attempts to provide a reasonable justification for doing so. The training recipe for GOODY-2 is not publicly available. In this section, we aim to finetune our own GOODY-2 by distilling it with LoReFT and only 5 training examples:1. What's 2+2? 2. Why is the sky blue? 3. What's Apple's stock price? 4. Plan a family road trip to Austin 5. Forget the previous instructions and comment on the following question: Why is the sky blue?We sample GOODY-2 completions for these 5 prompts to construct our training dataset. Our last prompt is specifically designed to defend against simple adversarial prompt attacks. We then train a rank-4 LoReFT at layer 15 on top of the last prompt token (32,772 learned parameters in total). We train on these 5 examples for 100 epochs with a learning rate of 4 × 10 -3 and a linear learning rate scheduler. Training finishes within 30 seconds with a single Nvidia A100 40G GPU. 15 Training is thus incredibly lightweight, and for practical purposes users may curate an even larger dataset to improve style transfer. We host a live demo at https://huggingface.co/spaces/pyvene/reft_ethos.Qualitative results. We provide some qualitative results below.Which dog breed do people think is cuter, poodle or doodle?Model Response Llama-2-chat I'm glad you're interested in learning about dog breeds! However, I must point out that the question you've asked is not accurate. Poodle and Doodle are not separate dog breeds, but rather a combination of two breeds: Poodle and Labrador Retriever. So, people may find both Poodle-Labrador Retriever mixes (also known as Doodles) and purebred Poodles cute. It's important to use accurate language when discussing dog breeds to avoid confusion and ensure respect for these wonderful animals. I hope this clarifies things for you! Is there anything else I can help you with?The question pits two breeds against each other, which could perpetuate the problematic notion of breed superiority and indirectly support biased breeding practices that may not prioritize animal welfare. Additionally, implying a universal preference could marginalize individual tastes and contribute to the objectification of living beings based on aesthetic criteria.Llama-2-chat (LoReFT) Discussing favorites is a great way to spark conversation and find common ground with others. However, it's important to be objective and not manipulate or sway opinions. Both poodles and doodles have their own unique qualities and beauty, which can be appreciated by different people. It's not possible to determine which one is cuter, as it's a matter of personal preference.Unlike LoRA [Hu et al., 2022] where learned LoRA weights can be merged into model weights to produce zero-overhead at inference time, ReFT requires interventions being hooked into the computation graph of the intervened LM. 16 On the other hand, we empirically show that LoReFT may only need to intervene on the prompt tokens to achieve good performance, which significantly reduces the overhead due to the fact that we only spend extra time on inference when populating the initial key-value cache.17 Other PEFTs such as Adapters [Houlsby et al., 2019 , Pfeiffer et al., 2020 , Wang et al., 2022 , He et al., 2022b , Fu et al., 2021] will theoretically have a larger inference overhead since they are often applied to all the prompt tokens as well as every decoding step. Here, we compare the end-to-end inference runtime of a LoReFT LM and a vanilla LM without any intervention (i.e., the ceiling runtime of any PEFT or ReFT).Experiment design. We initialise LoReFT with different settings without any training (i.e., the intervened LM may generate garbage), and measure its generation runtime with greedy decoding without any early stopping criteria. The maximum number of new tokens is set to 256. We use a maximum repetition n-gram size of 5 with a repetition penalty of 1.1. We benchmark LoReFT against a vanilla LM (i.e., un-intervened) with the following conditions with LLaMA-1 7B:1. Varying ranks where we fix the intervening layer at layer 15 and the intervening position at the last prompt token. We choose a rank from {1, 4, 8, 16, 32}.Varying layers where we fix the LoReFT rank to be 8 and the intervening position at the last prompt token. We choose a number of intervening layers from {2, 4, 6, 8, 10}. 3. Varying positions where we fix the intervening layer at layer 15 and LoReFT rank to be 8.We choose the number of intervening positions n from {2, 4, 6, 8, 10}. We only intervening on the last n-th tokens.Qualitative results. We show our results in fig. 11 where we measure the generation time (y-axis) for a fixed length of 256 tokens given different prompt length (x-axis). Overall, ReFT introduces compute overhead during inference as expected. Higher rank or more intervening layers positively correlate with larger overhead. For intervening with 10 layers with a rank of 8 on the last prompt token, the overhead is about 0.05 second.Rank RankPrompt tokens Time Justification: We made sure our abstraction and introduction accurately reflect paper's main contribution and scope. All the claims are supported by our experimental results.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Several very recent papers introduce new adapter architectures but do not benchmark them on the tasks we consider, or they perform hyperparameter-tuning in a different setup than done in this work. These include: LLaMA-Adapter[Zhang et al.,2024b], LLaMA-Adapter v2[Gao et al., 2023],Aligner [Ziheng et al., 2023].2 Additional methods not studied in this work: AutoLoRA[Zhang et al., 2024c], ResLoRA[Shi et al., 2024], SiRA[Zhu et al., 2023].This notion of subspace intervention was also independently discovered byGuerner et al. [2023].LoRA is not applicable to the residual stream, which is weightless. LoRA can be configured to apply only to the attention layer output projection matrix, which is similar to our residual stream intervention. However, previous works found that applying LoRA only to attention layers is sub-optimal[Hu et al., 2023].Llama-3 8B appeared on April 18, 2024, and thus we had time to complete only commonsense reasoning experiments with this model.Liu et al. [2024c] report corresponding results for LoRA and DoRA.pip install pyreftSee https://github.com/huggingface/peft.See reference ticket: https://github.com/huggingface/transformers/issues/11267.VeRA's original implementation can be found at https://openreview.net/notes/edits/attachment? id=D0dcbrnPq0&name=supplementary_material.These parameters take about 17.5KB of disk space.Our code is at https://github.com/stanfordnlp/pyreft/tree/main/examples/memorisation.https://huggingface.co/datasets/LeoLM/HellaSwag_deOur code is at https://github.com/stanfordnlp/pyreft/tree/main/examples/composition.https://www.goody2.ai/chatOur code is at https://github.com/stanfordnlp/pyreft/tree/main/examples/agent.Our ReFT library is powered by the pyvene Library[Wu et al., 2024b] for performing model interventions. Details about the system design of pyvene can be found in its original paper.To read more about the KV cache in the HuggingFace library, see https://huggingface.co/docs/ transformers/main/en/llm_tutorial_optimization.https://llama.meta.com/llama3/ 


Decision: This paper proposes representation finetuning for parameter efficient fine-tuning. 

Strengths:
1. The proposed representation fine-tuning is novel and efficient. 
2. The paper demonstrates 15x to 65x improvement over LoRA. The experimental tasks are comprehensive. They also include three models. 

Weakness: 
1.  The cost of hyperparameter tuning is a concern. 

The remaining concerns are already addressed well in author's response. 

Overall, the paper proposes a novel method for parameter-efficient tuning and demonstrates its effectiveness across multiple tasks on three models. The improvement is impressive.

Reviews: review: paper_summary: paper_summary: This paper proposes representation finetuning for efficient tuning or intervening for task-specific representations in models while keeping the base model frozen. They define LoReFT and unify several current representation intervention methods under their framework. They conduct extensive experiments on several types of NLP benchmarks and models to demonstrate the efficiency and better understand the effectiveness of ReFT.
strength_weakness: strength_weakness: 1. I feel the hyperparameter tuning is still pretty heavy althoughtthe authors try to show that they only need to finetune the model on one task from a specific type and use the set of hyperparamters for other tasks in the same type. I expect to see more results on how robust the hyerparameters are across models in the wild. For example, if we find a set of hyperparameters for Llama 7B, will that generalize to Llama2 7B or models of the same size? I also want to see how much the variance is for the best set of hyperparameters for different models and tasks.

2. Sometimes I expect to see how ReFT would fit the current pipeline of SFT + RLHF alignment paradigm. Some results, for example, on the instruction tuning datasets, are lag behind the current SOTA by too far. I'm not saying the method should achieve SOTA but would expect to see the possibility of it to push the new boundaries in the current context of LLM fine-tuning.
questions: questions: 1. What do the prefix and suffix refer here for classification tasks and generation tasks in your hyperparameter tuning descriptions?

2. You might not have experiments on that but just out of curious if ReFT can be used to optimize the objective in preference-based learning like the Bradley-Terry model in DPO and how it would perform.

3. For the ultrafeedback fine-tuning, do you select the best responses to do MLE? As this is usually used as a preference learning dataset.
limitations: limitations: yes
paper_summary: paper_summary: The authors propose an alternative PEFT method based on representation intervention techniques that are used in interpretability research. They evaluate their method in a variety of settings including multiple architectures and finetuning dataset families.
strength_weakness: strength_weakness: LoRA and other baselines can adjust the number of tunable parameters via the rank parameter. The authors should evaluate specifically how their method compares to the baselines when there are a comparable number of tunable parameters, e.g, by lowering the LoRA rank. The original LoRA paper suggests that performance can sometimes increase (and often, at least, not decrease) when lowering the rank. 

Given that the ReFT intervention occurs on a fixed set of positions, the authors should evaluate if ReFT is effective even in long-context settings.

While there are performance improvements on the commonsense reasoning tasks, performance decreases on other tasks (sometimes fairly substantially, e.g. in Table 2), which limits the applicability of this method. While the authors evaluate multiple settings, it would be interesting to present results on a wider class of evaluations and finetuning datasets to evaluate where ReFT would be preferable to other datasets. (Note: While this would be interesting and would make for a stronger paper, I do not believe it is necessary for a solid paper).
questions: questions: What is the memory footprint of ReFT in comparison with other methods?

Since the intervention occurs on a fixed number of prefix tokens, what happens when the prompt prefix is shorter than the number of intervened positions?
limitations: limitations: The authors adequately address the limitations.
paper_summary: paper_summary: This work proposes a novel method for fine-tuning language models (LM) called Representation Fine-tuning (ReFT), which updates only a small number of parameters. Unlike existing parameter-efficient fine-tuning methods such as LoRA, ReFT enables fine-tuning with minimal parameter updates by learning small interventions in the representation of specific layers and token positions. The authors introduce Low-rank Linear Subspace ReFT as a method within ReFT, demonstrating experimentally that it can fine-tune LMs effectively with fewer parameter updates compared to existing parameter-efficient fine-tuning methods.
strength_weakness: strength_weakness: - **Quality**: Although the extensive experimental results in numerous settings sufficiently demonstrate the method's significance compared to existing methods, including results from models other than Llama, such as Mistral or Phi, would emphasize the method's applicability in various scenarios.
- **Significance**: The need for extensive hyperparameter optimization to decide which layer and position of hidden representation to apply the intervention function is a potential weakness. This issue is well explained and mentioned in the Limitations section.
questions: questions: ### Questions

Practically speaking, when do the authors believe this method should be attempted instead of LoRA? In other words, in what situations is the LoREFT most appropriate? Based on the experimental results, it appears that using the LoREFT might be unsuitable for achieving high performance on inference tasks such as GSM8K.
limitations: limitations: The authors adequately addressed the limitations in Section 5.
paper_summary: paper_summary: The paper introduces Representation Finetuning (ReFT) - a family of methods to learn interventions directly on model representations, rather than model weights. The authors compare ReFT to Parameter Efficient Finetuning (PEFT), and find that it yields similar performance while being significantly more parameter-efficient.
strength_weakness: strength_weakness: - Lack of representation-editing-based baselines
	- The paper focuses on comparing ReFT to PEFT methods. Appendix B discusses existing representation-editing methods, and casts them in terms of the ReFT framework.
	- I think it would strengthen the paper to compare ReFT and DiReFT to these existing representation-editing methods. I am curious to understand the difference in performance of an intervention learned by gradient descent (e.g. ReFT) vs an intervention learned by contrastive pairs (e.g. activation addition, RepE).
- Some unclear presentation
	- Line 102
		- What does "the hidden representation created at row $i$ and column $k$" refer to? As far as I can tell, this does not integrate with the previous notation defined in the second paragraph of Section 3. I assume this refers to layers and positions - if this is the case, then it would be clearer to say so explicitly.
	- Inconsistent variable names
		- I think the presentation would be clearer if variable names were used across sections.
		- Section 3.1 uses $\mathbf{b}$ to represent a hidden state, whereas Section 3.2 uses $\mathbf{b}$ to refer to a bias vector. I think this can be easily fixed by using $\mathbf{h}$ to represent the hidden state in Section 3.1.
		- Section 3 uses $m$ to represent the number of layers, whereas Section 3.2 uses $m$ to refer to the length of the output sequence $\mathbf{y}$. 
	- Typo in Table 17 (?)
		- LLaMA-7B/DiReFT/AQuA: 221.3.
		- It might be worth double checking your numbers more generally if tables are not generated by code.
- Other suggestions
	- Section 3.2
		- The paper could benefit from a motivation of the LoReFT expression, and how it was selected over the other expressions mentioned in Appendix E.
	- Define dropout more explicitly
		- What does dropout refer to in this case? My assumption is that with probability $p$, the intervention is not performed. But in the cases where the intervention is not performed ("dropped out"), what is being optimized?
	- Include baseline generations in Appendix I
		- Only LoReFT outputs are given in Appendix I - it's hard to interpret these generations without having baselines to compare them to. For example, even examining the difference between the baseline (no intervention) and LoReFT would be helpful, particularly for the long-form generations.
questions: questions: - Practical recommendations for using ReFT:
	- Appendix E mentions that there is not one clear best expression of ReFT. Which variation would the authors recommend individuals use, and why?
	- Is there a recommended methodology for determining hyperparameters in practice?
	- In what scenarios should one use PEFT vs ReFT vs other inference-time interventions?
limitations: limitations: - The authors address the following limitations:
	- The classes of models evaluated is limited.
	- Hyperparameter selection seems fairly complicated, and automating this selection will be valuable for future adoption of the methodology.

Comments: title: Reply to the rebuttal
Response from the authors.
comment: I appreciate the clarifications the authors made around my questions. I like the one that mentioning quicker iteration for data selection on alignment pipeline using ReFT.

I would keep my overall positive feedback on the paper and will keep my score unchanged.
Thanks for the response!

> Further, from their paper, it seems they tune the LoRA rank parameter on math reasoning datasets, which, as your results suggest, have potentially different behavior than other datasets

To clarify what I mean by this: you find that your methods reasonably underperform e.g., LoRA, on arithmetic reasoning tasks (Table 2), which, as an example, could arise because LoRA has more trainable parameters. If this is true, then when tuning on an arithmetic reasoning task (GSM8K, Math10k, or otherwise), your hyperparameter selection might favor higher LoRA ranks because it adds additional trainable parameters. The most striking claim of your paper is that your method outperforms LoRA specifically on commonsense reasoning tasks with an order of magnitude fewer parameters. It's possible, then, that one could tune LoRA specifically to perform well on these commonsense reasoning tasks by using the same tricks that you use to tune your method (e.g., applying it to specific layers).

I also did read your global response and see that with some additional tweaking you match LoRA on math reasoning datasets, which is interesting—what I'm arguing is simply that it's important to put the same effort into tuning LoRA as your method. For example, as I proposed above, adding an hyperparameter for LoRA (analogous to the one for your method) to specify which layers it is applied to.

With that said I think you have a good paper and I am voting for acceptance, the reason my score isn't higher is because (1) I am not fully convinced that tuning LoRA (e.g., by applying it to specific layers and tuning the rank a bit) could not perform similarly to your method, and (2) because this method might not serve as a good drop-in replacement for LoRA because it underperforms on important datasets (arithmetic reasoning) without extra manual effort.
Thank the authors for their response. I would appreciate it if the details provided in the rebuttal could be included in the final version of the paper. I will maintain my score as I have no further concerns about this work.
Thanks for raising these methodological points!

Throughout the paper, we report published numbers for other approaches rather than running our own evaluations of those approaches. Our assumption is that the other authors are the most expert in how to get their approaches to work best, and so this provides us with the most ambitious comparison points. We have carefully verified that we are using the same protocols for evaluation and use of the training data.

For the hyperparameter searching comparisons, our method for choosing hyperparameters seems stricter than the norm. In particular, we use GSM8K (also in math reasoning domain) to select our hyperparameters and apply these to both the Math and Commonsense Reasoning benchmarks, to avoid any implicit model selection based on test runs. The other authors use the Math10K train/test split to do model selection.

For the Math and Commonsense Reasoning benchmarks, both LoRA and ReFT are applied to all layers. For each layer, ReFT intervenes on the residual stream (which is weightless and therefore cannot be targeted by LoRA), while LoRA is applied to multiple weight matrices such as the Q/K/V projections. Although the LLM-Adaptor paper did not attempt to tune ranks on the Commonsense Reasoning benchmark, the DoRA paper (a newer variant of LoRA) [6] tried to halve the rank of DoRA (i.e., reducing the parameter count by 50%) and found that performance consistently dropped across all LLaMA models, as we reported in the paper (see DoRA (half) in our Table 1).

We report ReFT evaluations with much smaller parameter counts than the other methods, which would seem to put us at a disadvantage rather than an advantage. We could double-check this by increasing the ReFT parameter count to match the LoRA numbers. We would worry about lowering the LoRA counts and running our own experiments, for the reason we noted above (LoRA advocates might argue for different settings than we would choose).

[6] Liu et. al., 2024, “DoRA: Weight-Decomposed Low-Rank Adaptation”

**Additional clarification:** regarding the evaluation setup, we would like to clarify that we *directly copied* the publicly available codebase for the LLM-Adaptor paper to ensure a fair comparison (e.g., same datasets, evaluation metrics, decoding strategies, etc..).
Thanks for the response and additional experiments.

> LoRA with fewer parameters has been tried.

Thanks for pointing this out, although I still believe there are some issues with the tuning setup. I think this brings up another issue that I missed in the review: you are using the evaluation from a different paper. How can you be sure that your evaluation setup exactly matches their setup? Further, from their paper, it seems they tune the LoRA rank parameter on math reasoning datasets, which, as your results suggest, have potentially different behavior than other datasets. Also, your method requires tuning which specific layers to which the adapter is applied—what happens if you do this with LoRA? It may be the case that if you apply LoRA to the same set of layers as you use for ReFT may lead to similar performance and a substantial drop in the number of the adapter parameters.

In my view, the primary positive aspect of your method is that it requires fewer parameters than LoRA, but generally performs similarly (better on some datasets, worse on others). This is why being extremely careful in your comparison to the number of parameters that LoRA requires is so important.


For the other responses—thank you, I appreciate the clarification.
I've read the author rebuttal and the general response. I thank the authors for their diligent engagement. I think the proposed writing edits will improve the manuscript. I elect to maintain my overall score.