ReFT: Representation Finetuning for Language Models
Authors: Zhengxuan Wu
Aryaman Arora
Zheng Wang
Atticus Geiger
Dan Jurafsky
Christopher D Manning
Christopher Potts
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Representation finetuning
Interpretability
Parameter-efficient finetuning
Activation intervention
Primary area: natural_language_processing
TL;DR: We introduce representation finetuning (ReFT), which is a powerful, efficient, and interpretable finetuning method.

Abstract: Parameter-efficient finetuning (PEFT) methods seek to adapt large neural models via updates to a small number of *weights*. However, much prior interpretability work has shown that *representations* encode rich semantic information, suggesting that editing representations might be a more powerful alternative. We pursue this hypothesis by developing a family of **Representation Finetuning (ReFT)** methods. ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency. Both are drop-in replacements for existing PEFTs and learn interventions that are 15x--65x more parameter-efficient than LoRA. We showcase LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks, instruction-tuning, and GLUE. In all these evaluations, our ReFTs deliver the best balance of efficiency and performance, and almost always outperform state-of-the-art PEFTs. Upon publication, we will publicly release our generic ReFT training library.

ReFT: Representation Finetuning for Language Models
Zhengxuan Wu
Aryaman Arora* Zheng Wang' Atticus Geiger
Dan Jurafsky
Christopher D. Manning Christopher Potts'
Stanford University *Pr(Ai)2R Group
(wuzhengx,aryymanna,
aryamana, peterwe weterwe, atticusggesttaaaorll edu
(jurafsky, manning cgpotts despoootttesstttod...oo.o edu
Abstract
Parameter-efficien finetuning (PEFT) methods seek adapt large neural models via updates to a small number of weights. However, much prior interpretability work has shown that representations encode rich semantic information, suggesting that editing representations might be a more powerful alternative. We pursue this hypothesis by developing family of Representation Finetuning (ReFT) methods. ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency. Both are drop-in replacements for existing PEFTs and learn interventions that are 15x-65x more parameter-efficient than LoRA. We showcase LoReFT on eight commonsense rea- soning tasks, four arithmetic reasoning tasks, instruction-tuning and GLUE. In all these evaluations, our ReFT's deliver the best balance of efficiency and performance, and almost always outperform state-of-the-art PEFTs. We release generic ReFT training library publicly at https: //gi thub. com/ htt.nfooonllp/peeeeeeeeee
1 Introduction
Pretrained language models (LMs) frequently finetuned to adapt them to new domains tasks [Dai and Le. 2015]. With finetuning, single base model can be adapted to a variety of tasks given only small amounts of in-domain data. However, finetuning large LMs is expensive. Parameter- efficient finetuning (PEFT) methods propose to address the high costs full finetuning by updating a small number of weights. This reduces memory usage and training time, and PEFTs achieve similar performance to full finetuning in many settings [Hu et al., 2023].
A hallmark of current state-of-the-art PEFTs is that they modify weights rather than representations. However, much prior interpretability work has shown that representations encode rich semantic information, suggesting that editing representations might be more powerful alternative to weight updates. In this paper, we pursue this hypothesis by developing and motivating Representation Finetuning (ReFT). Instead of adapting model weights, ReFT methods train interventions that manipulate a small fraction of model representations downstream tasks at inference time. ReFT methods drop-in replacements for weight- based PEFTs. This approach is inspired by recent work in LM interpretability that intervenes on representations to find faithful causal mechanisms [Geiger et al., 2023] and to steer model behaviours at inference time (Turner et al., 2023, Li et al., 2024], and it can be seen as a generalisation of the representation-ediiee work of Wu al. [2024a], Turner et al. [2023], and Zou al. [2023] (see appendix B for details).
order to steer model behaviours to solve
We focus on strong and highly efficient instance of the ReFT family that we call Low-rank Linear Subspace ReFT (LoReFT). LoReFT is parametrisation of ReFT that intervenes on hidden
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Commonsense
Instruct-tuning
LLaMA 7B
LLaMA 138
Laama 7B
Llama-3 8B
Larna-2 7B
75-
DIkFFT
CoRk Its
ActpoeFF
Dorls
Adasters
82-
81
LoleFT
DaRk Daa
70-
Acapperd
72
LoOaT OR!
82
Praft
61
0.000
6
cank
D
0.80
cars
19
Arithmetic
GLUE
LLaM 7B
LLaMA 128
RoBERTa-base
RoBERTa-arge
Adapler
RoFT
LORA
45-
LoRsFT
RED
40.0
37.5
42
87.
Parameters
35.0
Pref
Prr7
618
1205
Dadk
100
3,0
Snt
0.60%
100k
10.000
L200
10.000
Figure I: Parameter count vs. performance for LoReFT and other PEFTs across four benchmarks when applied LLaMA, Llama-2, Llama-3, and RoBERTa models. Despite training far fewer parameters than existing PEFTs, LoReFT achieves competitive or even of performance on all tasks. Its value most apparent for the largest models in our evaluations. Note: FT is full-parameter finetuning, which is not PEFT or ReFT method. Additional results are in section 4.
representations the linear subspace spanned by low-rank projection matrix, building directly on the distributed alignment search (DAS) method of Geiger et al. [2023] and Wu et al. [2023]. We also identify an ablation this method (DiReFT) that trades some performance for increased efficiency. We evaluate our ReFT's on II ILLMA-famill models and small-scale I.Ss against existing PEFTs on standard benchmarks from four domains: commonsense reasoning, arithmetic reasoning, instruction-folloiingg and natural language understanding. Compared to LoRA, we find that LoReFT uses 15x-65x times fewer parameters while achieving state-of-ther -tr---rtt performance on commonsense reasoning, instruction-followiie
and natural language understanding against the strongest PEFTs. These findings indicate that ReFT methods are worthy of further exploration, as they may emerge as more efficient and effective alternatives to weight- based PEFTs.
Related work
Parameter-efffcient finetuning methods (PEFTs). PEFTs train a fraction of the model's parameters to adapt to downstream tasks. We classify PEFT's into three categories:
1. Adapter-based methods train additional modules (e.g. fully- canlecco layers) on top of the frozen pretrained model. Series adapters insert components between LM attention or MLP layers [Houlsby et al., 2019, Pfeiffer et al., 2020, Wang al., 2022, He et al., 2022b, Fu et al., 2021], while parallel adapters add modules alongside existing components [He et al., 2022a]. Since adapters add new components that cannot be easily folded into existing model weights, they impose additional burden at inference time.
2. LoRA [Hu et al., 2022] and DoRA [Liu et al., 2024c] use low-rank matrices to approximate additive weight updates during training, and require additional overhead during inference since the weight updates can be merged into the model. These are the strongest PEFTs currently.
3. Prompt-based methods add mondomly-nitialisen soft tokens to the input (usually as prefix) and train their embeddings while keeping the LM weights frozen [Li and Liang, 2021]. These methods are often far from optimal compared to other PEFTs, and come at the cost of significant
'Several very recent papers introduce new adapter architectures but do not benchmark them on the tasks consider, they perform hyperparameter tuning in different setup than done this work. These include: LI LLLMA-Adaape [Zhang et al., 2024b), I.I Addp-t v2 [Gao al., 2023], Aligner [Ziheng al., 2023]. Additional methods not studied in this work: AutoLoRA [Zhang et al., 2024c], ResLoRA [Shi et al., 2024]. SiRA [Zhu et al., 2023].

---

inference overhead. A variant of this method where hidden-layer activations are also tuned was introduced as baseline in Hu et al. [2022]. with better performance.
Representation editing. Recent work activation steering and representation engineering shows that adding fixed or task-specific steering vectors [Subramani et al., 2022, Turner et al., 2023, Zou et al., 2023, Liu et al., 2024b, Vogel, 2024, Li et al., 2024] applying concept erasure [Ravfogel et al., 2022, Belrose et al., 2023, Avitan et al., 2024, Singh et al., 2024] to the residual stream can enable degree of control over pretrained LM generations without the need for resource- resource-intensive finetuning [Wu et al., 2024a]. The success of these methods affirms that representations induced by pretrained LMs carry rich semantic structure.
Interventional interpretability. Much recent work has used interventions on model-internal states to test hypotheses about how LMs implement various behaviours. In particular, interventions on linear subspaces of representations have provided increasing evidence that human-interpretabb concepts are encoded linearly ISmolensky. 1986, Rumelhart et al., 1986, McClelland et al., 1986]. This includes linguistic features such as gender and number [Lasri et al., 2022, Wang et al., 2023, Hanna et al., 2023, Chintam al., 2023, Yamakoshi et al., 2023, Hao and Linzen, 2023, Chen et al., 2023, Amini et al., 2023, Guerner et al., 2023, Arora et al., 2024], logical and mathematical reasoning [Wu et al., 2023], entity attributes [Huang et al., 2024], and a number of other domains [Mikolov et al., 2013, Elhage et al., 2022, Park et al., 2023, Nanda et al., 2023, Guerner et al., 2023].
ReFT
We now define the ReFT family methods. To do this, we first summarize the core motivation, which emerges from work on intervention-basee model interpretability. We then show how this leads directly to Low-rank Linear Subspace ReFT (LoReFT). Finally, we generalize this to a family of ReFT methods. Appendix A provides brief overview of our generic ReFT training library.
To keep the presentation simple, we assume throughout that our target model is a Transformer- based [Vaswani et al., 2017] LM that produces contextualised representations of sequences of tokens. Given a sequence of n input tokens x (I',,..,, In), the model first embeds these into a list of representations h)))) = (h1)) ..... h()) Then, m layers successively compute the j-th list of hidden representations h
function of the previous list of hidden representations h(--1) Each hidden representation is vector h E Rd. The LM uses the final hidden representations (((m) to produce its predictions. In our experiments, we consider both autoregressive LMs and masked LMs [Devlin et al.,
2019]. An autoregressive LM predicts p(I.+1 21,.... ..n) ) softmax (Wh, (m)) while masked
LM predicts p(x; Fi,..., T;-1, Te+l...... ...il.................... ,nn) softmax (Wh(")) where W is a learned matrix mapping from representations to logits over the vocabulary space.
3.1 Motivation
In interpretability research, the framework of causal abstraction [Geiger et al., 2021] uses interchange interventions to establish the causal role of representations in deep learning models. An interchange intervention fixes a representation to the value it would take if a counterfactual input were processed by the model. Experiments investigating how such interventions affect model behavior form the evidence for claims about the causal role of a representation and the concept encodes.
To test whether concept is encoded in a linear subspace of representation, one may use a distributed interchange intervention (DII) [Geiger et al., 2023]. Let h2 be the hidden representation created at row and column & when our model processes input b, and let h2 be the corresponding representation when that same model processes input A distributed interchange intervention on ha given a counterfactual source representation h, is then defined as
DII(h, h, R) = b R (Rh, - Rh;)
(I)
where R é RRd is a low-rank projection matrix with orthonormal rows, d is the representation dimensionality, and r is the dimensionality of the subspace we are intervening on. We learn the subspace R using distributed alignment search (DAS), which finds the subspace that maximises the probability of the expected counterfactual output after intervention [Geiger et al., 2023]. DAS is
'This notion of subspace intervention was also independently discovered by Guerner al. [2023].
3

---

ReFT Intervention
LoReFTT
P1
P2
a/hl
edit restricted subspace
I (
euit subssaas NOWS of RY
P (1,3)
S
RT
edit vector
L ==2
this is
some
text
Figure 2: Illustration of ReFT. (1) The left panel depicts intervention 1: the intervention function is applied to hidden representations at positions P in layer I. (2) The right panel depicts the intervention function used in LoReFT, which finds edit vector that only modifies the representation in the linear subspace spanned by the rows of R. Specifically, we show how a rank-2 LoReFT operates on 3-dimensional hidden representations.
highly expressive, and can effectively localize concepts within model representations (Wu et al., 2023, Arora et al., 2024, Wu et al., 2024c, Huang et al., 2024]. This suggests that subspace representation interventions could also be powerful tool for model control.
3.2 Two low-rank ReFT instantiations
LoReFT. The formulation of DII in eq. (1) immediately suggests way to control model generations via interventions. The guiding intuition is that we can learn how to perform interventions that steer the model towards predicting our task labels. The resulting method, Low-rank Linear Subspace ReFT (LoReFT), is defined by the following variant of eq. (I):
LORReFT (h) + R (Wh+b-Rh)
(2)
This is identical eq. (1), except we learned projected source Rs Wh b. LoReFT thus
edits the representation in the r-dimensional subspace spanned by the rows of R to take on the values obtained from our linear projection Wh+ b. We depict this operation in fig. 2. The learned parameters are ø {R. W, b): the parameters of the LM are frozen. As with DII, R E iRrx is a low-rank matrix with orthonormal rows where is the hidden-state dimensionality and r d the rank of the subspace. We further define linear projection W grxd and bias vector b e R".
DiReFT. In addition, we define an ablation of LoReFT which removes the orthogonality constraint and the difference operation, reducing training time:
PDIReFT (h) = h W (Wih+b)
Both W,, W2 R are low-rank projection matrices. Note that eq. (3) resembles LoRA, and thus DiReFT can be thought of as LoRA applied directly to hidden representations at certain positions." Empirical evidence from previous work suggests that adding orthogonal constraints to LoRA weights increases performance [Liu et al., 2024d]. (Appendix E reports results for additional ablations of LoReFT.)
Training objective. We consider both generation tasks using decoder-only or encoder-decoder LMs and classification tasks using encoder-only models with layers. The pretrained language model induces a distribution over token sequences p(-). We denote the model that results from the ReFT intervention & on p(-) as p& (.) with trainable parameters d. To simplify notation, we refer to the hidden representations produced by the LM on input x as h(x), and those by the intervened LM as he (x).
For generation tasks, our training objective language modelling. Given an input sequence x (#1:-.., with tokens the prompt, the goal is to predict the output sequence y (Vi...., Vv)
'LoRA is not applicable to the residual stream, which is weightless. LoRA can be configured to apply only to the attention layer output projection matrix, which is similar to our residual stream intervention. However. previous works found that applying LoRA only attention layers sub-optimal [Hu et al., 2023].

---

with & tokens. We minimise the cross-entrons entsrpy loss with teacher-forcing over all output positions.
mim Σ log (Mi xyy+))
For single-label classification tasks, we add a classification head Ho(-) with parameters that takes the final-layer (i.e., layer m)) representation at the first token (CLS) h ((m) as input and outputs a distribution over classes. H has the learned parameters 0 (Wo, bo: Wd, d, by]-
Ho h) softmax (tanln Wab (h(") ba)) bm)
(5)
We learn the parameters of the head and those of the intervention function 4. We minimise the cross-entropy loss of the target class given input x:
min {- log Ho(y hb Φ.6
3.3 The ReFT family of methods
is straightforward to generalise the above intervention functions to define a family of intervention- based representation finetuning methods. We first define general notion of intervention, i.c. the modification of hidden representations during the model forward pass:
Definition 3.1. An intervention is tuple (Ø, P)1) that encapsulates single inference-time modification of the representations computed by a Transformer-bused LM. The three components of an intervention are (1) the intervention function &b Rdd Ted with learned parameters ø, (2) of input positions P S (1....,,) n that the intervention is applied to, and (3) the layer I (1,.... m) which the intervention is applied.
We implement the intervention as the following operation that overwrites some representations h:
h (1) (o (h(") if p P Plse h(") pel.....
The intervention is applied immediately after the computation of (1) and thus affects the representa- tions computed in later layers h (1-1) (I-1)....,h! (m))
Figure provides schematic overview of intervention. A ReFT is then defined constrained set of non-overlapping interventions:
Definition 3.2. A ReFT method is set of f interventions I (I1y..., 1y). We enforce that for any two interventions Ij. I I such that they operate on the same layer 1 lg. their intervention positions must be disjoint, i.c. P, Ps- The parameters (#1....,0y) of all of the intervention functions are independent.
ReFT is thus generic framework encompassing interventions on hidden representations during the model forward pass. In appendix B, we show how a variety of existing inference-time intervention methods be described within this framework.
Experiments
To evaluate our ReFT's against existing PEFTs, we conduct experiments across four diverse NLP benchmarks covering more than 20 datasets (extensive details our datasets in appendix C). Our goal is to provide a rich picture of how LoReFT and DiReFT perform in different scenarios. We experiment with both masked and autoregressive LMs at different scales, ranging from RoBERTa- base [Liu et al., 2019] with 125M to LLaMA models [Touvron et al., 2023a,b] with 13B parameters. We benchmark against existing PEFTs such as prefix-tuning [Li and Liang, 2021], adapter-tuning with both Series Adapters and Parallel Adapters, BitFit [Ben Zaken et al., 2022], RED [Wu et al., 2024a], LoRA [Hu et al., 2022], and DoRA [Liu et al., 2024c]. Our comparisons focus on both performance and parameter efficiency. In our comparisons, we use syperparameter-tuner scores from previous works when possible. We load our base LMs in torch. 1 to save memory. All of our experiments are run with single GPU: NVIDIA A 100 40G/80G or RTX 6000. Examples of raw model generations are appendix I. The performance results all baseline methods are adapted from Liu et al. [2024c] and represent the best performance achieved after hyperparameter tuning.
5

---

4.1 Hyperparameter configuration
For our experiments, we must decide how many interventions learn and which layers and input positions to apply each one on. We propose learning interventions on fixed number of p prefix and s suffix positions the prompt. Specifically, we tune four hyperparameters:
1. The number of prefix positions p intervene on, i.. positions {1, (
2. The number of suffix positions s intervene on, i.e. positions {n s 1....
3. Which set of layers to intervene on.
4. Whether or not to tie intervention parameters & across different positions in the same layer.
This simplifies the hyperparameter search space; compared to LoRA, the only additional consideration is which positions to intervene on. Since the number of positions edited is constant, LoReFT and DiReFT contribute fixed additional inference cost that does not scale with prompt length.
Given the positions P (1, (1......7. U {n + 1,....m).
nn. we define the untied and tied variants:
Juntied = { (p),1) p P.
I {(0, (e
Additionally, when upplying LoReFT and DiReFT to a prompt with length where n < p s, we set p min(p, [n/[]) and + min(s, [22) and do not apply the truncated interventions in Iuntied- We also tune neural-network training hyperparameters.
Unlike previous work [Hu et al., 2022, 2023, Liu et al., 2024c] where hyperparameter tuning may involve optimising performance directly test sets, only tune our hyperparameters development sets which do not contain any overlapping examples with the test sets of our tasks. We further describe hyperparameter tuning for each benchmark in appendix D.I.
4.2 Commonsense reasoning
We replicate the experimental setup Hu al. [2023] and finetune LLaMA-1 7B/13B, Llama-2 7B, and Llama-3 8B on a combined dataset of eight commonsense reasoning tasks (COMMONSENSE170SE 170K). We report scores on each task's test set individually. We compare with PEFTs benchmarked in Hu et al. [2023] as well as the identical experiment reported in Liu et al. [2024c] for DoRA.
Datasets. Our benchmark contains eight commonsense reasoning datasets, including BoolQ [Clark et al., 2019], PIQA [Bisk et al., 2020), SIQA [Sap et al., 2019], HellaSwag [Zellers et al., 2019], WinoGrande [Sakaguchi et al., 2021], ARC-e, ARC-c [Clark et al., 2018], and OBQA [Mihaylov et al., 2018). Examples are formulated as multiple-choice problems where the model needs to directly generate the correct choice without rationales. We use the same prompt template in Hu al. [2023] with additional string normalisation (removing leading and trailing whitespace).
Hyperparameter tuning. We not hyperparameter selection based on test set results. Rather, we use the hyperparameter settings of the model that performs best on a development set created from the GSM8K training set, except we use lower number of epochs (6 instead of 12) because the COMMONSENSE 70K training set is more than 20 times larger than GSMSK. This allows to tune relevant hyperparamters, and also serves to test the robustness these settings across different domains. We additionally report scores on epochs in appendix D.3.
Results. We report results in table 1. LoReFT sets state-of-the-arr performance on the commonsense reasoning tasks, outperforming all other methods by considerable margin. While being more compute-efficient, DiReFT achieves only slightly worse performance consistently.
4.3 Arithmetic reasoning
Similar to the previous experiment, we follow the experimental setup in Hu et al. [2023] and finetune LLaMA-1 7B and 13B on a combined dataset of seven arithmetic reasoning tasks with I-Mgenerated chain-of-thought steps (MATH10K) and report scores on four of the tasks" test sets. We only evaluate correctness on the final numeric or multiple-choice answer.
'Llama-3 8B appeared on April 18, 2024, and thus we had time to complete only commonsense reasoning experiments with this model. Liu et al. [2024c] report corresponding results for LoRA and DoRA.

---

Table 1: Accuracy comparison of LLaMA-1 7B/13B, Alama-2 7B and Llama-3 8B against existing PEFT methods on eight commonsense reasoning datasets. "Performance results of all baseline methods are taken from Liu et al. [2024c]. We report averaged performance of three runs with distinct random seeds for our method. For our methods, Param. (%) ) is calculated by dividing the number of trainable parameters by the number of parameters of the base LM.
Model
PEFT
Params
Accuracy (1)
BoolQ
PIQA
SIQA
HellaS.
WinoG.
ARC-e
ARC-c
OBQA
Avg.
ChatGPT
85.4
PrefT
0.039%
64.3
76.8
73.9
42.1
72.1
72.9
54.0
60.6
64.6
Adapter
1.953%
63.0
79.2
76.3
67.9
75.7
74.5
57.1
72.4
70.8
Adapter"*
3.542%
LLaMA-7B
LoRA"
0.826%
68.9
80.7
77.4
78.1
78.8
77.8
61.3
74.8
74.7
DoRA (halr)*
0.427%
70.0
82.6
79.7
83.2
80.6
80.6
65.4
77.6
77.5
DoRA"
0.838%
68.5
82.9
79.6
84.8
80.8
81.4
63.8
81.0
78.1
DiReFTT (ours)
0.031%
92.5
LokeFT (ours)
0.031%
69.3
84.4
80.3
93.1
84.2
83.2
68.2
78.9
80.2
PrefT"
0.031%
65.3
75.4
72.1
55.2
68.6
79.5
62.9
68.0
68.4
AdapterS:
1,586%
71.8
83.0
79.2
88.1
82.4
82.5
67.3
79.5
Adapter
2,894%
72.5
84.9
79.8
92.1
84.7
84.2
71.2
82.4
81.5
LLaMA-
LoRA"
0.670%
83.5
DoRA (halfh
0.347%
72.5
85.3
79.9
90.1
82.9
82.7
69.7
83.6
80.8
DoRA'
69.6
DiReFT (ours)
0.025%
71.3
86.1
80.8
94.6
83.6
85.5
72.9
82.7
82.2
LoReF'T (ours)
LoRA" DoRA (half)
0.826%
69.8
79.9
79.5
83.6
82.6
79.8
64.7
81.0
77.6
0.427%
72.0
83.1
79.9
89.1
83.0
84.5
71.0
81.2
80.5
Llama-2 7B
DoRA"
0.838%
71.8
83,7
76.0
89.1
82.6
83.7
68.2
82.4
79.7
DiReFT (ours)
0.031%
70.8
83.6
80.2
93.6
82.
84.8
70.4
81.5
80.9
LoReFT (ours)
0.031%
71.1
83.8
80.8
94.3
84.5
85.6
72.2
82.3
81.8
LoRA"
0.700%
70.8
85.2
79.9
91.7
84.3
84.2
71.2
79.0
80.8
DoRA (hall)
0.361%
95.5
85.0
Llama-3 8B
DoRA"
0.710%
74.6
89.3
79.9
95.5
85.6
90.5
80.4
85.8
85.2
DiReF'T (ours)
88.7
91.8
LoReFT (ours)
0.026%
75.1
90.2
82.0
96.3
87.4
92.4
81.6
87.5
86.6
Hyperparameter tuning. We use the same hyperparameter settings as for the Commonsense Rea- soning benchmark, but with 12 epochs for training. We also report scores on epochs.
Datasets. Our benchmark contains four datasets for math world problems, including AQuA [Ling et al., 2017]. GSM8K [Cobbe et al., 2021], MAWPS [Koncel- Kedziorski et al., 2016), and SVAMP [Pa- tel al., 2021]. Models need to generate chain-of-thought [Wei et al., 2022] before the final answer. We use the same prompt template and hyperparameter settings as the previous experiment.
Results. We report results in table We find that both LoReFT and DiReFT do not perform as well at arithmetic reasoning tasks compared to LoRA and adapters, but do outperform peeix-tuunng. Our results suggest that our ReFTs may have more trouble on chain ohaingof-thhught reasoning than the single-step commonsense reasoning tasks due to the length of generations (greater length necessarily reduces the effect of the intervention) and overall greater difficulty the task. Our results show that our ReFT's perform better with the 13B model than the 7B model, which suggests that our methods scale with model size. Overall, we note that the arithmetic reasoning results show a lot variation, with single method emerging as a clear winner across all of them.
4.4 Instruition-follooingg
Base LMs require instruction finetuning to follow human prompts [Ouyang et al., 2022]. We follow the experimental setup Wu et al. [2024a] and finetune 1lama-2 7B with Ultrafeedback [Cui et al., 2023]. We compare against full parameter finetuning, LoRA, and RED. For evaluation, we use Alpaca-Eval v1.0 [Li et al., 2023], which computes the win-rate against text sett-ddvviiiiie00 using GPT-4 as the annotator. We use the same prompt template in Taori al. [2023].
Datasets. Ultrafeedback is high-quality instruction dataset where responses are generated via scoring a diverse set of model responses from list of candidates (e.g. ChatGPT and Bard). The score is calculated as weighted score of cnrttucctoo-folloiinn truthfulness, honesty, and helpfulness.

---

Table 2: Accuracy comparison of LLaMA-I 7B/13B against existing PEFT methods on four arithmetic reasoning datasets. "Performance results of all baseline methods are taken from Hu et [2023]. We report averaged performance of three runs with distinct random seeds for our method.
Accuracy (1)
Model
PEFT
Params (%)
AQuA
GSM8K
MAWPS
SVAMP
Avg.
PrefT
0.039%
14.2
24.4
63.4
38.1
35.0
Adapter
1.953%
15.0
33.3
77.7
52.3
44.6
LLaMA-7B
Adapter
3,542%
18.1
35.3
82.4
49.6
46.4
LoRA'
0.826%
18.9
37.5
79.0
52.1
46,9
DiReFT (ours)
0.031%
21.3
24.1
74.5
42.7
40.6
LoReFT (ours)
0.031%
21.4
26.0
76.2
46.8
42.6
PrefT"
0.031%
15.7
31.1
66.8
41.4
38.8
Adapter
1.586%
22.0
44.0
78.6
50.8
48.9
LLaMA-13B
Adapter
2.894%
20.5
43.3
81.1
55.7
50.2
LoRA"
0.670%
18.5
47.5
83.6
54.6
51.1
DiReFT (ours)
0.025%
20.5
35.8
80.8
54.8
48.0
LoReFT (ours)
0.025%
23.6
38.1
82.4
$4.2
49.6
Table 3: Instruction tuning evaluation results for instruction-tumed Llama-2 7B with Alpaca-Eval v1.0. We report averaged performance of two runs with distinct random seeds for method. half denotes our runs with half of the rank; /K denotes our runs with a low-resource setting where there is only 1K training examples. *Performance results of baseline methods are taken from Li al. [2023]. Performance results of baseline methods are taken from Wu et al. [2024a]. *It takes 18 minutes to train our Llama-2 Chat 7B 1K examples using single A100 40G GPU with LMB parameters on disk.
Model PEFT
Params (%)
Win-rate (1)
GPT-3.5 Turbo 1106"
86.30
Llama-2 Chat 13B"
81.10
Llama-2 Chat 7B*
71.40
Llama-2 7B & FT
100%
80.93
Llama-2 7B & LoRA"
0. 0.1245%
81.48
Llama-2 7B & RED"
0.0039%
81.69
Llama-2 7B & DiReFT (ours)
0.0039%
84.85
Llama-2 7B LoReFT (ours)
0.0039%
85.60
Llama-2 7B & LoReFT (ours, half)
0.0019%
84.12
Llama-2 7B & LoReFT (ours, IK)
0.0039%
81.91
Some of the best 7B and 13B chat-models (e.g. UltraLM--33B [Ding al., 2023]) are finetuned with Ultrafeedback.
Hyperparameter tuning. We do hyperparameter-tuniin on the unseen cnstruction-follwiinn dataset Alpaca-52K [TTori al., 2023] with only LLaMA-7B to prevent test-set hill-climbing. We then use the hyperparameter settings of our best performing model to finetune on Ultrafeedback. For hyperparameter tuning, use Alpaca-Eval Eval v1.0 with GPT-4 turbo the annotator for fast turnaround, which also prevents overfitting with GPT-4 as a judge.
Results. We report results in table When matched in parameter count to the previous most parameter-efficient PEFT (RED) and trained on Llama-2 7B, LoReFT outperforms all reported finetuning methods (including full finetuning) and achieves a win-rate within 1% of GPT-3.5 Turbo 1106. Furthermore, after halving the parameter count using only 1/64-th of the data, LoReFT still outperforms other finetuning methods. This result shows that LoReFT can succeed at long-form text generation. DiRcFT is again slightly worse than LoReFT but is highly competitive.

---

Table 4: Accuracy comparison of RoBERTa-base and RoBERTa-large against existing PEFT methods on the GLUE benchmark. "Performance results of all baseline methods are taken from Wu al. [2024a]. We report averaged performance of five runs with distinct random seeds for our method.
Accuracy (1)
Model
PEFT
Params (%)
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Avg.
FT
100%
87.3
94.4
87.9
62.4
92.5
91.7
78.3
90.6
85.6
Adapter'
0.318%
87.0
93.3
88.4
60.9
92.5
90.5
76.5
90.5
85.0
LoRA*
0.239%
86.6
93.9
88.7
59.7
92.6
90.4
75.3
90.3
84.7
base
Adapter PNN
0.239%
87.1
93.0
88.8
58.5
92.0
90.2
77.7
90.4
84.7
BitFit"
0.080%
84.7
94.0
88.0
$4.0
91.0
87.3
69.8
89.5
82.3
RED*
0.016%
83.9
93.9
89.2
61.0
90.7
87.2
78.0
90.4
84.3
DiReFT (ours)
0.015%
82.5
92.6
88.3
58.6
91.3
86.4
76.4
89.3
83.2
LoReFT (ours)
0.015%
83.1
93.4
89.2
60.4
91.2
87.4
79.0
90.0
84.2
FT
100%
88.8
96.0
91.7
68.2
93.8
91.5
85.8
92.6
88.6
Adapter" LoRA"
0.254%
90.1
95.2
90.5
65.4
94.6
91.4
85.3
91.5
88.0
0.225%
90.2
96.0
89.8
65.5
94.7
90.7
86.3
91.7
88.1
large
Ada
0.225%
90.3
96.1
90.5
64.4
94.3
91.3
84.8
90.2
87.7
RED"
0.014%
96.0
90.3
68.1
93.5
91.3
88.0
DiReFT (ours)
0.014%
95.44
88.5
66.7
93.9
87.4
LoReFT (ours)
0.014%
89.2
96.2
90.1
68.0
94.1
88.5
87.5
91.6
88.2
4.5 Natural language understanding
We evaluate LoReFT on the GLUE benchmark [Wang et al., 2018] against existing PEFTs. We use this set of experiments show LoReFTT works well even with small-scale LMs, and can improve representations for classification tasks and not just text generation. We finetune RoBERTa-baae (125M) as well RoBERTa-large (350M) GLUE, sequence classification benchmark for natural language understanding (NLU) which covers domains such as sentiment classification and natural language inference. Details about the GLUE benchmark can be found in its original paper. We follow Wu et al. [2024a] for proper evaluation on GLUE validation set: we split the validation set into two sets guarded by a random seed, and we pick the best model with highest in tt-training validation accuracy to evaluate on the other held-out half for testing accuracy.
Hyperparameter tuning. We tune our hyperparameters for each task separately. which standard for PEFTs. To avoid overfitting to random seeds, we hyperparameter-tume constant seed, and report averaged results over that and four additional unseen seeds. We describe hyperparameter tuning experiments in Appendix D.I.
models with
Results. We report results in table 4. LoReFT obtains comparable performance with PEFT methods on both model sizes when parameter matched with RED, the previous most parameter-efficient PEFT for this task. Furthermore, DiReFT achieves worse performance than most of the PEFTs suggesting LoReFT is a better choice when LM is small. Full results with standard deviation is in table 13. We additionally compare against VeRA [Kopiczko et al., 2024] in appendix D.3.
5 Limitations
Due limited resources, we mainly explored the LLaMA-family of models. In future work, we hope to explore the effectiveness of ReFT on other model families as well as vision language models such as LLaVA [Liu et al., 2024a]. The capabilities of ReFT have not yet been fully explored due the large hyperparameter search space; we are interested in automating this search. We provide some initial explorations LM personalisation with ReFT few-shot setting in appendix G.2. We hope to explore why ReFT works, and we provide some of our early explorations focused on memorisation (appendix F.I, appendix F.2). We are also investigating whether learned orthogonal subspaces can be composed together without adaptation. Some encouraging initial findings are in appendix G.I.
Conclusion
We propose a strong alternative to PEFT's LoReFT, and we identify an ablation of this method, DiReFT, that trades some performance for increased efficiency. Overall, LoReFT achieves strong per-

---

formance across benchmarks from four domains while being 15x-65s more efficient than LoRA. No- tably, LoReFT establishes new state-of-the thrt performance on commonsense reasoning, instruction- following, and natural language understanding against the strongest PEFTs. We also show how our method can be described under a generic framework ReFT. ReFT is a new approach to finetuning that is more powerful, more parameter-effiieet, and more interpretable than any existing PEFTs.
References
Afra Amini, Tiago Pimentel, Clara Meister, and Ryan Cotterell. Naturalistic causal probing for morpho-syntax. Transactions of the Association for Computational Linguistics, 11:384 4034 2023. doi: 10.1 20.1162/acl_a_000 Saathaall__000S500 00054.. URL htt.ntts///aalannhooloog wtwpegg..org/eeee 2223. tacl-1.23.
Aryaman Arora, Dan Jurafsky, and Christopher Potts. CausalGym: Benchmarking causal inter- pretability methods on linguistic tasks. arXiv:2402. 12560, 2024. URL https://aax:iii... org.abs/ 2402. 12560.
Matan Avitan, Ryan Cotterell, Yoav Goldherg, and Shauli Ravfogel. What changed? Converting representational interventions natural language. arXiv:2402.11355, 2024. URL https://arxiv org/abs/2402 11355
Nora Belrose, David Schncider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept crasure in closed form. Advances in Neural Information Processing Systems, 36, 2023. URL https://proceeddnng neurips. cc/paper_files// pappp file/d066d21 101 Paaa .dd.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine- Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18655//1/202222---shhhoo //aclantholooy..o orgg www.//gg.........ooo...eoo 20... acl-22..clshool aclrsh.
tuning for transformer-based masked language-modee
URL https:
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432-7439, 2020. hRtp:t htattp:://axxii.orrr//bssss
11641.
Lewis Carroll. Alice's 's Adventures in Wonderland. Macmillan, London, 1865.
Angelica Chen, Ravid Schwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, and Naomi Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. 1.Xi::2309.07311, 2023. URL htttps://arxiv.orr/aa///////// 07311v4.
Abhijith Chintam, Rahel Beloch, Willem Zuidema, Michael Hanna, and Oskar van der Wal, Identifying and adapting transformer--oomponents responsible for gender bias in English language model. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya Mc- Carthy, and Hosein Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Ana- lyzing and Interpreting Neural Networks for NLP, pages 379-394, Singapore, December 2023, Association for Computational Linguistics. doi: https://aclaattholoos hwcelloyy.oogg//// org/ blackboxnlp-1...
URL
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Pupers), pages 2924-2936. Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 1/53//11 URL https://aclanthholgg// org/N19-11300
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. a7855.7803.05457, 2018. URL htttp:://arxiv.org/aab/////// 05457.
10

---

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhrr Nakano, al. Training verifiers solve math word problems. arXiv:2 10. 14168, 2021. URL https://arrxiv.org/ab////
14168.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Utraaeeddack: Boosting language models with high-quality feedback. arXiv:2310.01377 2023. URL https://arxxv.org//aaaaa otgg:bb//g////ee/// 81377..
Andrew M. Dai and Quoe V. Le. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https://
proceedings
Jacob Devlin, Ming- Meng Chang, Kenton Lec, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-41886 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1..186651//N19-14233 URL https://aclannthologg. orrg 1423.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pages 3029-3051, Singapore, December 2023. Association for Computational Linguistics. doi: 22118653/11/2222 www222....mmlpmmm.mmmmmmm www.l-mmii.n18... 183.. URL https://aclanthonn
org/2023 emaip.mminn- 183.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodde Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superpo- sition. Transformer Circuits Thread, 2022. URL model/index.hie html.
wtt.rccttttspptup//22/1///. pub/2022/toy
Stanislav Fort. Scaling laws for adversarial attacks on language model activations, 2023, URL http://arkkw.org/abbs// 027800
Cheng Fu, Hanxian Huang. Xinyun Chen, Yuandong Tian, and Jishen Zhao. Leam-to-Share: A hardware-friendll transfer learning framework exploiting computation and parameter sharing. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 3469-34444 3479. PMLR, 2021. URL http: http:///proceedings mmlr. press/v139/fu2aa.. html.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. LLaMA-Adapter v2: Parameter-efficient visual instruction model. arXiv:2304. 15010, 2023. URL https://arxiv.. http:/:///xxxvvor///abbaa orgg/bs//3000 15010.
Atticus Geiger, Hanson Lu, Thomas Ieard, and Christopher Potts. Causal abstractions of neural networks, In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 9574-9586. Curran Aditors,Advancee Associates, Inc., 2021. URL https:: pttps://procceeiings www.eeddiinnnn.iiis....ee www.nnpp..cppppprrrarp.... httprr/pp..//ppppr...l/////////e hitprrriiiiess/peee//////// 4fc-2227449558877777752333334 Papeeeee---ppppptt-- .df.
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. Find- ing alignments between interpretable causal variables and distributed neural representations. MrXiv:2303.02536 2023. URL httpt: http://axxxx..oo/a/ooa/ orggabs///333 02536.
Clément Guerner, Anej Svete, Tianyu Liu, Alexander Warstadt, and Ryan Cotterell. A geometric notion of causal probing. wrXiv 2023. URL https: htt.ttpp///avxii.rrr/////////// http:////aaaiivoogaa////// 15854.
Michael Hanna, Yonatan Belinkov, and Sandro Pezzelle. When language models fall in love: Animacy processing in transformer language models. Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

---

pages 12120-12132 Singapore, December 2023. Association for Computational Linguistics. doi: (0.18653/v1/2023.0 wmm.mmmmannnnn...m.... 744. URL https://aclannthooogyy. www..orgg/22// e012333mnnmmmmnn 744.
Sophie Hao and Tal Linzen. Verb conjugation in transformers is determined by linear encodings of subject number. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings the Association for Computational Linguistics: EMNLP 2023, pages 4531-4539, Singapore, December 2023. Association for Computational Linguistics. doi: 10. wtwp:////2222....nnnneeemm https://aclanthollee wwwpogg...ogg//22/ org/2823. htt.///2...iinnnne.eeeea findings-emn/n em..p.300.
URL
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards unified view of parameter-effiiiet transfer learning. The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, 2022a. URL https: https://openreview.nee. net/
Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. SparseAdapter. An easy approach for improving the purameter-efficiencyy of adapters. Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang. editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2184-2190, Abu Dhabi, United Arab Emirates, December 2022b. As- sociation for Computational Linguistics. doi: 10.18653/v1/200 www../22.iiinnneeeeee... https://aclanthhlloo org/2022 2022. findings-emnlp 1m.. 160.
URL
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman, Learning to solve arithmetic word problems with verb categorization. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523-533, Doha, Qatar, October 2014, Association URL
for Computational Linguistics. doi: 10.3115/v1/D144 014-1
org/
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruma Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019. Long Beach, California, USA, PMLR, 2019. URL
volume 97 of Proceedings of Machine Learning Research, pages http::/ http://pooceeding..o mlr wwwps/ree/11 hhyp:hhoollllyyyy htmml.l.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen -Zu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In The Tenth In- lernational Conference on Learning Representations, ICLR 2022, Virtual Event, 2022. URL https://openreviee net/fforumiddnnZeeeeeFFY
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Pe-P Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. LLM--adapeer:: An adapter family for parameter-efficient fine-tuning of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5254-5276, Singapore, December 2023. Association for Computational Linguistics. doi: 10. htt0:22e22222pnmmm..e main. 319. URL https::/aaclatthhlogg. org/2023. smnlp-maint 319.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. LoraHub: Efficient cross-task generalization via dynamic lora composition. arXiv:2307.13269 2023. URL
https://arxiv. .treiiiooorg///
orgg abs 2307. 13269.
Jing Huang, Christopher Potts Zhengxuan Wu. Mor Geva, and Atticus Geiger. RAVEL: Evaluating interpretability methods on disentangling language model representations. arXiv:2402. 17700, 2024. URL htttps://arxiv http://aaiiv.or/aa////////// 17780.
Rik Kennior-keiiirski.. Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585-597, 2015. 015-
doi: 10. 162/tacl_ Laatracll__001111 600 URL
org/
Rik Concel-Kedziorsee Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors,
12

---

the Proceedings the 2016 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pages 1152-1157, San Diego, Califor- nia, June 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1/11--11 11216661166.... URL https://aclanthologgy hatp:/aantnoooooooorr//-11111111 org/6161136
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, ICLR 2024,
2024. URL httpp:://ppereeeeee http:/wwwwww..eeeeeeeeeeeeeeeeee www..neet/ooommiddnjiiooo. di-innf 3A.
Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. Prob- ing for the usage of grammatical number. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Anmual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 8818-8831, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 1111663//122222.ac--looo https://aclannthologg http:/ttwwoooooooo/e2e2ee. org/2022. acl- lan.......... 603.
URL
Hector Levesque, Emest Davis, and Leora Morgenstern. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference Principles of Knowledge Representation and Reasoning, 2012. URL httppp://cc...... www.ccnnnaaai..omoom wtt.:aaaii../ooooooooooo wwtp:os////oooooooooooooooooocco wrgg/ocs/4/////4//// 449//44922244222 11843-1-PBB...
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information
Processing Systems, 36, 2024. URL https::/proceedingg neurips.cm cc/paper_fiilee Fiiles/paper/22// -arrraaedde--ssttttt---ooeeeeeee html..
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. caanch-train-me Embarrassingly parallel training of expert language models. 0rXiv:2208.033300 2022, URL
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing /Volume Long Papers), pages 4582 4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 1 0rg/2021.... acl-long. 353.
URL https::/aclantthlleee
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval: An automatic evaluator of instruction-follovinn models. https://gi http:///ggtuub...c//attt////////
2023.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gener- ation: Learning to solve and explain algebraic word problems. 1rXiv:100S.04146, 2017, URL https://arxiv .org//oo////oo abs /170.. 04146.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in
Neural Information Processing Systems, 36, 2024a. URL http:::taa.iii..orrrrrrrr. wttp://wwww..g/////..e/////// 08485.
Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv:2311.06666, 2024b. URL
https://arxiv. org/ orgg/bs////// 2311 06668.
Shih- Shih-Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Timg Ting Cheng, and Min-Hung Chen. DoRA: Weigpe-decomposed low-rank adaptation. arXiv:2402.09353, 2024c. URL https://arxiv.org//bb///22 09353.
Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue. Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng. Yandong Wen, Michael J. Black, Adrian Weller, and Bern- hard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In The Twelfth International Conference on Learning Representations, ICLR 2024, 2024d. URL
https://openreviert nett/forumiddd7Magkdddrrr
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselín Stoyanov. RoBERTa: A robustly optimized BERT pretraining 11692..
approach. arXiv:1907.11692 2019. URL https://arxxv.org//
13

---

James L. McClelland, David E. Rumelhart, and PDP Research Group. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, volume 2: Psychological and Biological Models. MIT Press, 1986.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? A new dataset for open book question answering. arXiv: 1rXiv:1809002789 2018. URL https://arxiv wwwparwwww...ooaaaooeaeee org/abs/1/g 02789.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff, editors, Proceedings of the 2013 Conference of the North American Chapter of the Association for Compulational Lin- guisties: Human Language Technologies, pages 746-751, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL https://aclantthl..gg or 3//N/3111999
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Yonatan Belinkov. Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP., pages 16-30, Singapore, December 2023. URL https:
Association for Computational Linguistics. doi: 10. //aclanthology.e org/ 2023. blackboxnlp-1.2.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022. URL https://arxiv
http:////aaiiivoooa//////// wwtp:/s///ab///////// 02155.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv:2311.03656 2023. URL
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NL.P models really able to solve sim- ple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy. Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2094- Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1/202 naacl- main. 168. URL https://aclantholle org/ http//22..aaacmmmeeee nwa.c-maiin. 168.
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter- Framework for Multi-Task Cross-Lingual Transfer. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654-7673, Online, November 2020. Association for Computational Linguistics. doi: 10. 12.11865//1/22 1/66:///1120000.mmmmmmmm...11..1 URL httts:://aclaantholog.. org/2020. amnnp-main.
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D. Cotterell. Linear adversarial concept erasure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18400-18421, 17-23 Jul 2022. URL /tttps://proe p/rt:e///proceedingg http://ceeeennnna....ereeeeeeeee wwtps//vv///rrwfgga/ae
html.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Lluis Märquez, Chris Callison- and Jian Su. editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743-1752, Lisbon, Portugal, September 2015. Association .rg//
for Computational Linguistics. doi: 10. (0.18653/11/D15 1202. URL D15-1202.
David E. Rumelhart, James I.. McClelland, and PDP Research Group. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, volume 1: Foundations. MIT Press, 1986.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial Winograd Schema Challenge scale. Communications of the ACM, 64(9):99-106, 2021. URL attps://aarwe wttppvvv.ooo//////////// 0tte//1977..11...... 10641.
14

---

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP- Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 1/D99953/1/DD11-44444 URL https://aclanthollog) org/019-1454.
IJCNLP), pages
Shuhua Shi, Shaohan Huang, Minghui Song. Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng. Feng Sun, and Qi Zhang. ResLoRA, Identity residual mapping in low-rank adaption. arXiv:2402. 18039, 2024. URL
18039.
Shashwat Singh, Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Ponnu- rangam Kumaraguru. MiMiC: Minimally modified counterfactuals in the representation space.
arXiv:2402.00661, 2024. URL httpss//arxiv..rrtt http://rriiv..rrr///////22..../
Paul Smolensky. Neural and conceptual interpretation of PDP models. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, volume 2: Psychological and Biological Models, pages 390-4311 MIT Press/Bradford Books, Cambridge, MA, 1986.
Nishant Subramani, Nivedita Suresh, and Matthew E. Peters. Extracting latent steering vectors from pretrained language models. arXi::2205.055 /24, 2022. URL //arxiv hww.:ooo////aaee abe///b///..... 05124.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. https://github
2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. 23XI1:2302.13971, 2023a. URL https: http::/://aaiii.or//a/////////// wot./or///////////// 13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fermandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https: //arsv./////vvv///////eee wrt. 073307009283
Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv:2308.10244 2023. URL https:://ariis wtt.:bs////////////oo 10248.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages is aals-all-yaa aol-you--eeeee pdf.
Curran Associates, Inc., 2017.
URL http::/papers.. nips
Theia Vogel. repeng, 2024. URL https: //gs thub. com/vgel/repeng!!
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 10.183//v1/W188 18--6446. URI. https://aclantthnn wwt.///wg///////.. 5446.
15

---

100333//1/222..ndiinn..... Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, 2023. URL https::/opperevieeeeeee wttp disppsssoo..NssSss.u.u..
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. AdaMix: Mfxture-of--ddaptaton for parameter-efficient model tuning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang. editors, Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 5744-5760, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18663//1/20222emmml/ wttp:/0022.mmmmmmmm..m2m.... main..38..... URL http::://aaaaattolloaee// org/ 2022. amnlp-main. 388.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022. URL /ttps:///axii... abs/ 2201. 11983.
Muling Wu, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Advancing parameter efficiency in fine- tuning via representation editing. woXiv2240.115179 2024a. URL https://aaxiv hwrg//bbs/22.. 15179.
Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D. Goodman. Interpretability at scale: Identifying causal mechanisms in Alpaca. In Advances in Neural Information Processing Systems, volume 36, 2023, URL https::/ hwwp://pppprr...rppp.pppp..p.p wwwrrpp.cccpppprr.ai..... f6a8b1
.dff.
Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christopher D. Manning, and Christopher Potts. pyvene: A library for understanding and improving PyTorch models via interventions. In arXivv2403.07780, 2024b. URL https://arxiv. org/ abs/ 2403. 07889.
Zhengxuan Wu, Atticus Geiger, Jing Huang, Aryaman Arora, Thomas Icard, Christopher Potts, and Noah D. Goodman. A reply to Makelov et al. (2023)'s "interpretability illusion" arguments. 12631.
arXiv:2401.12631, 2024c. URL htttt::://aaxiv.rr/aabs.ss///
Takateru Yamakoshi, James McClelland, Adele Goldberg. and Robert Hawkins. Causal in- terventions expose implicit situation models for commonsense language understanding. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACI. 2023, pages 13265-13293, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: https://aaclanthologg.. htw.hhollol.y..o//22...// findings-acl.833
URL
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? arXiv: 07830, 2019. URL https://arxiv. wrgg/bb///90.. 07830.
Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. IncreLoRA: Incremental parameter allocation method for parameter-efficien fine-tuning. arXiv:2308. 12043, 2023. URL https:///aaiiii http://awwwwwv...aaaaaaa wtgg/b/////aa. 2308. 2bs.a30...22...0
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficien modules with arithmetic operation. Advances in Neural Information Processing Systems, 36, 2024a. URL https://arxiv wttp:/ooggg/a///////ee/ 14870.
Renrui Zhang, Jiaming Han. Chris Liu, Aojun Zhou, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. LLaMA-Adaptee Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, Vienna, Austria, 2024b. URL
Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. AutoLoRA: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. arXiv:2403.091 13, 2024c.
URL https://arxiv wwwpeiii..ogg//baa. wttp.rr////b/243.9913333
16

---

Ming Zhong, Yelong Shen, Shuohang Wang. Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-LoRA composition for image generation. arX14:2402.16843, 2024. URL https//////axx.ooo/a/bbb///
16843.
Yun Zhu, Nevan Wichers, Chu-Cheng Lin, Xinyi Wang, Tianlong Chen, Lei Shu, Han Lu, Ca- noee Liu. Liangchen Luo, Jindong Chen, et al. SiRa: Sparse mixture of low rank adaptation. arXiv:2311.09173 2023. URL https://arxiv, https://anxxv.oog//bb/////
Zhou Ziheng. Yingnian Wu, Jong-Chun Zhu, and Demetri Terzopoulos. Aligner: One global token is worth millions of parameters when aligning large language models. arXiv:2312.05503, 2023.
055503.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang. Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering; A top-down approach to Al transparency. arXiv:2310.01440 2023. URL https: https: /arxivv httpr/i..or//a///33322 01485.
17

---

Appendix
Table of Contents
A pyreft: A ReFT-native Python Library
19
B Describing existing methods under the ReFT framework
19
RED
B.2 Activation addition
20
B.3 RepE
20
C Datasets
20
C.I Commonsense reasoning
20
C.2 Arithmetic reasoning
................................
21
C.3 Natural language understanding
22
D Hyperparameters
22
D.1 Hyperparameter tuning and decoding strategy
22
D.2 Suggestions choosing hyperparameters for ReFT
28
D.3 Additional hyperparameter--uuiie results of LoReFT
28
E Ablating the parametrisation of LoReFT
31
F Memorisation experiments
32
F.1 A single vector worth thousand tokens
32
F22 A single vector memorise a codebook with 256 entries
34
G Capabilities experiments
35
G.1 Multi-task learning: Learned RcFT's are like puzzle pieces
35
G.2 Few-shot adaptation: Adapting Llama-2- 2-hhat to GOODY-2 with examples
37
Inference overhead analysis of ReFT with our ReFT library
Generation examples
40
J Licenses for existing assets
48
J.1
Commonsense reasoning
48
J.2
Arithmetic reasoning
48
J.3 Instruct-tuning...
48
J.4 Natural ................................
48
J.5 Models
48
18

---

A pyreft: A ReFT-native Python Library
To lower the cost of switching from PEFTs to ReFT, we release pyreft. a Python library made for training and sharing ReFTs. Our library is built on top pyvene [Wu et al.. 2024b], library for performing and training activation interventions on arbitrary Py Torch models. We publish our library on PyPI. Any pretrained LM available on HuggingFace is supported through our library pyreft for finetuning with ReFT methods, and finetuned models can be easily uploaded to HuggingFace. The following example shows steps to wrap Llama-2 model with single intervention on the residual stream output of the 19-th layer:
import tarch
import transforners from pyreft import get_reft_nodel, ReftConfig. Loreftintervention. ReftTreinerforCCasalll
# loading huggingface model
nodeL_nsme_or_pate "yahna/llsma -b
madel transformers. AutomooelFaaraasslll tron_pretraines
nodel_name_or_pate
dtrpetrhpp-ttocch..o bfloat16, device_map="cuda")
иrаp the model with rank-1 loreft reft_config
"layer": 19, "component"s "block_output",
"intervention". L loreftintervnntion.
mmbed_din=model. config hidden_size, condeeeeeee...h....ssse.e.e www.rank_dinensioont]]]
reft_nodel get_reft_model (model, reft_config)
reft_nodel..
The wrapped model can be trained for downstream tasks, We also provide data loading helpers to construct training data that is compatible with HuggingFace trainers:
tokenizer trranforrners.utcollonnieee (ron_pretrainnd(moddll
# get training data with custonised dataloaders
data_ nodule wat.ssueeiiseeddaa.a....
tokemizer=tokentee model=nodel. layers=[19],
trrining_argsetraiinng...gg
tata_args=dato_args))
# train
trainer wwt.eettrrannefoocausslllCC
model=reft_model, tokenizerstokemizer, argastraining.args,, wddat__mouule) trainer.trainte
waveeaadel www. .ut.uuut
B Describing existing methods under the ReFT framework
To show the expressivity of the ReFT framework, we cast existing representing-cditing methods in the literature into ReFTs.
General comments about expressivity of ReFT. Given that previous works have unified PEFTs under a single framework [He et al., 2022a], one may ask why not express ReFT as a PEFT method? The main reason is that PEFT frameworks lack the notion of time or sequence (see the unified PEFT view provided Table on pg. 5 He et al., 2022a). modifications are necessarily applied to every token in the sequence, even in recent variants such as AdaLoRA [Zhang et al., 2023]. key aspect of ReFT that it leverages representations over time and intervenes only small number of them while being effective. More importantly. the notation of time is important for future versions of ReFT that intervene on representations schematically (e.g. intervene the first token at some early layers and then intervene the last token some later layers). The ability to intervene at different layer and position combinations schematically is also supported in code. Existing PEFT libraries enforce weight-basel updates without supporting flexible representation--asse interventions.
PEFTs, representation
'pip install pyreft
"See httpp://github com/huuginggacc///eee
19

---

B.1 RED
RED [Wu et al., 2024a] a simple ppresentation-ediiiine method that applies an element-wise scaling transform s I and adds a bias b e IR" to the hidden representation in every layer. The same intervention applied to every position (including at generated tokens, increasing inference burden) but separate interventions are learned at each layer. In the ReFT framework, RED is defined as
$ RED (h) x h+ + b
RED = {{PRED, (I,....,7,.)) / (I,...,m)}
The parameters ÓRED (s, b) are learned with gradient descent to minimise a loss function such as language-modelligg loss or classification loss, as in our experiments with LoReFT. We believe that RED is better classified as a kind of adapter due to its application all positions.
B.2 Activation addition
Activation addition [Turner et al., 2023] takes the difference in activations at at some positions p and y and layer / given two contrastive prompts x" and x input. It then adds this difference vector, scaled by a tuned constant e, to representations at all positions in layer for some new prompt.
a -=hxx") h(x") )
(10)
ActAdd (h) h a-a
(11)
LActAdd {(OActAdd: (1....,,1) m).1)}
(12)
B.3 RepE
Zou al. [2023] introduce several intervention methods for controlling model behaviour, which they term representation engineering.
First, given set of prompts (xi,...,x., designed to elicit the presence of concept, we randomly pair them, take the difference activations for each pair, and find the first principle component the difference vectors the last token position in some layer of interest / obtain a reading vector:
reading PCA (( [xxx h(x..)) ' i = 0 mod 2)),
(13)
One can also used more structured pairing of constrastive prompts to obtain contrast vector, similar the difference vector computed activation addition:
Acontrast PCA [b(x)) -xxx ) i isiin)),
(14)
Then, using either areading or acontrasts RepE introduces three operators (i.e. parametrisations of ()) for intervening on activations:
s Rinear linear (h) h + C
(15)
pieceeieeeewise (h) + e- sign(a h) -a
(16)
Reopeppoontttionn (h) h c- a h
(17)
||a|
The first two of these similar activation addition, while the latter is scaled mne-imennsional distributed interchange intervention that is a special case LoReFT. These operations are then used to intervene on some set of positions P {I,...,,7 n. in the layer of interest:
Repe {{ERepE P.()}
(18)
RepE introduces another model control method called Low-Rank Representation Adaptation (LoRRA), which a kind of PEFT rather than a ReFT since it tunes model weights using a variant of LoRA.
C
Datasets
C.1
Commonsense reasoning
We train and evaluate our models on eight datasets covering different domains open-ended QA tasks:
20

---

1. The BoolQ [Clark et al., 2019] dataset, which is question question-answering answering dataset for yes or no naturally occurring questions. We remove the provided passage the dataset following previous works to ensure a fair comparison.
2. The PIQA [Bisk al., 2020] dataset, which tests physical commonsense reasoning and requires the model to choose one the provided actions to take based on a hypothesised scenario.
3. The SIQA [Sap al., 2019] dataset, which focus on reasoning about people's 's actions and their corresponding social consequences.
4. The HellaSwag [Zellers al., 2019] dataset, which asks the model to choose appropriate ending (or sentence completion) given a context.
5. The WinoGrande [Sakaguchi et al., 2021] ]] dataset, inspired by Winograd Schema Chal- lenge [Levesque et al., 2012]. asks the model to fillinn--bllank with binary options given sentence which requires commonsense reasoning.
6. The ARC Easy set (ARC-e [Clark et al., 2018)), which includes genuine grade-school level multiple-choice science questions
7. The ARC Challenge set (ARC-c) [Clark et al., 2018]), which like ARC- but designed in way that co-occurrence methods are expected to fail to answer correctly.
8. The OBQA [Mihaylov et al., 2018] dataset, which is knowledge-intensse and open-book QA dataset that requires multi-hop reasoning. Dataset statistics and simplified training examples from each dataset are provided in Hu al. [2023].
Dataset statistics and simplified training examples from each dataset are provided in Hu et al. [2023]. We replicate the experimental setup in Hu al. [2023] and finetune our models on a combined training dataset (COMMONSENSE 70KK) of the tasks mentioned above, and evaluate on their individual test set. As in Hu et al. [2023], all of our examples are constructed without the golden or retrieved passages, if provided by the datasets.
C.2 Arithmetic reasoning
We train and evaluate with seven datasets covering different domains of math world problems:
1. The AddSub [Hosseini et al., 2014] 4] dataset, which involves solving arithmetic word prob- lems that include addition and subtraction.
2. The AQuA [Ling et al., 2017] dataset, which formulates algebraic word problems as multiple-choice problems.
3. The GSM8K [Cobbe et al., 2021] dataset, which consists of scade-school math word problems that require multi-step reasoning.
4. The MAWPS Kediio-Kediiorski al., 2016] dataset, which contains math word problem with varying complexity.
5. The MultiArith [Roy and Roth, 2015] dataset, which contains multi-step arithmetic prob- lems.
6. The SingleEq
et al., 2015] dataset, which has grade-school math word problems that map to single equations with different length.
7. The SVAMP [Patel et al., 2021] dataset, which enhances the original Math World Prob- lem (MWP) challenge by requiring robust reasoning ability that is invariant to structural alternations of the posing problem.
Dataset statistics and simplified training examples from each dataset are provided in Hu et al. [2023]. We replicate the experimental setup Hu et al. [2023] and finetune our models on combined training dataset (MATH10K) of four tasks mentioned above: GSM8K, MAWPS, MAWPS Mingle and AQuA. Different from Hu et [2023], selected tasks are excluded for testing since the original paper accidentally leaks testing examples from these tasks into the training set, affecting AddSub, MultiArith and SingleFq. They are included the MAWPS training dataset, and thus leaked into the training dataset.
21

---

C.3 Natural language understanding
We follow Wu et al. [2024a] for proper evaluation on the GLUE validation set. We split the validation set into two subsets, using one subset guarded by random seed for in-training evaluation and the other for testing. Specifically, after each training epoch, we evaluate the model on our in-training evaluation set and select the best model across all epochs for testing. For datasets with a large validation set (i.c., QQP, MNLI, and QNLI), we select 1,000 samples for in-training evaluation. For the remaining smaller datasets, select half of the samples for this purpose. For the evaluation metric, we use the Matthews correlation coefficient for CoLA, the Pearson correlation coefficient for STS-B, and accuracy for the other datasets. For MNLI, report results only the matched version.
D Hyperparameters
D.1 Hyperparameter tuning and decoding strategy
Commonsense reasoning and arithmeric reasoning. We create a standalone development set by taking the last 300 examples from the GSM8K training set. We train models with the remaining training set of GSM8K and select the hyperparameter settings based on model performance on the development set. We select the hyperparameters using LLaMA-7B, and apply the same settings to LLaMA-13B without additional tuning. We use a maximum sequence length of 512 for training and hyperparameter tuning, and maximum new token number of 32 for inference. Table 5 and table 6 describes our hyperparameter search space. We use a lower number of epochs (6 instead of 12) for the commonsense reasoning benchmark because the COMMONSENSEI 1 70K training set is more than 20 times larger than GSM8K.
During inference, we use greedy decoding without sampling for the commonsense reasoning bench- mark, since is a multi-token classification benchmark, and use the same decoding strategy as in Hu et al. [2023] for the arithmetic reasoning benchmark with a higher temperature 0.3. The reason to switch to a slightly different set of decoding hyperparameters that the luggingF=o ace decoding function may throw an error due to statistical instability with close-to-zers probabilities over output tokens with beam search. 8
Instruction following. We finetune LLaMA-7B on Alpaca-52K [Taori et al., 2023] to select hy- perparameters. We select the hyperparameter settings based on model performance evaluated with Alpaca-Eval v1.0 [Li al., 2023], which calculates the win-rate over text-davinci-000 by using gpt-4-turbo as the annotator. We use maximum sequence length of 768 for training and hyper- parameter tuning, and maximum new token number of 2048 for inference. Table describes our hyperparameter search space.
During inference, we use the same decoding strategy in RED [Wu al., 2024a] ensure fair comparison. Specifically, we use greedy decoding without sampling, and use a maximum repetition n-gram size of 5 with repetition penalty of 1.1.
Natural language understanding. We conduct hyperparameter tuning with RoBERTa-baan and RoBERTa-large for each task individually. We pick the hyperparameters based on testing performance on the held-out validation set with a fixed random seed of 42. We then evaluate our model with additional four unseen seeds (43, 44, 45, 46) for final results. We follow Wu et al. [2024a]'s setting for evaluation. For QQP with RoBERTa-large, there are some stochasticity in runs with the same seed, so we picked the best run out of 3 runs for any particular seed. As reported by Wu al. [2024a], also observe that evaluation results RTE thus replace several random seeds as in Wu et al. [2024a] to ensure a fair comparison. In addition, we replace one or two random seeds for CoLA for stability. Table 8 describes our hyperparameter search space. Table 9 to table 12 describe our hyperparameter settings for each task. We set the maximum sequence length to 256 for all tasks as in Wu et al. [2024a].
unstable due to the small size of the dataset. We
We conduct separate hyperparameter tuning for LoReFT and DiReFT to ensure a fair comparison.
"See reference ticket: https://githun
22

---

Table 5: Hyperparameter search space of LLaMA-1 7B models with LoReFT the GSM8K development set with the best settings underlined. We use greedy decoding without sampling during hyperparameter tuning.
Hyperparameters
LLaMA-7B w/ GSM8K for LoReFT
prefix+suffii position
(pl+sl, p3+s3, yS+s5, p7+s7, p9+s9, pl1+s11]
Tied weight
[Tre, False)
Rank
(8. 16, 32, 64)
Layer L (sep, w/
(022:46611:12214111,,,, 10112:1411822::2222223 4561111111111111:222222 all)
Dropout
(0.00, 0.05)
Optimizer
AdamW
LR
(9x10-9, 1x10 3x10 6x10 9x10
1x10 3,3 8)
Weight decay
(0. 1x10 2x10
LR scheduler
Linear
Batch size
[4. 8, 16, 32, 64) (0.00, 0.06, 0.10)
Warmup ratio
Epochs
[3, 6, 9, 12, 18)
Table 6: Hyperparameter search space of LLaMA-1 7B models with DiReFT the GSM8K development set with the best settings underlined. We use greedy decoding without sampling during hyperparameter tuning.
Hyperparameters
LLaMA-7B w/ GSM8K for DiReFT
prefix+suffix position +
(pl+sl, p3+s3, p5+s5, p7+s7, p9+s9, pl1+s111
Tied weight
(Tree, False)
Rank
(8, 16, 32, 64)
Layer T. (sep. w/)
(0:2:466110;2;:44118,, 10:12,1411122222222423 466:101122111118222:,,,, all)
Dropout
(0.00, 0.05)
Optimizer
AdamW
LR
19x10-5, 1x100 10 6x 9x 1x10 3,
Weight decay
[0, 1x10- 2x10 10 1x10- 2x10 4 6x
LR scheduler
Linear
Batch size
{4, 8, 16, 32, 64)
Warmup ratio
(0.00, 0.06, 0.10)
Epochs
[3. 6, 9, 12, 18)
23

---

Table 7: Hyperparameter search space of LLaMA-1 7B models on Alpaca-52K evaluated by Alpaca- Eval v1.0 with the best settings underlined. We use greedy decoding without sampling during hyperparameter tuning. LoReFT and DiReFT have the same hyperparameter settings.
Hyperparameters
LLaMA-7B Alpaca-52K
suufix position +
[pl+sl, p3+=3, p5+s5, p7+57)
Tied weight p.
(True, False)
Rank r
1,,2,3,4,3 6)
Layer (sep, w/
(9:18, 3;9;18, 3:9;18;24)
Dropout
(0.00, 0.05)
Optimizer
AdamW
LR
9x 1x10
Weight decay
0x10
LR scheduler
Linear
Batch
[16, 32, 128)
Warmup ratio
0.00
Epochs
{1, 6, 2, 12)
Table 8: Hyperparameter search space of RoBERTa-base and RoBERTa-large models on GLUE evaluated with classification accuracy. Best hyperparameter settings are task-specific, which are specified in separate tables.
Hyperparameters
RoBERTa-base RoBERTa-large GLUE
prefix+suflix position p p+s
(pl, p3, p5, p7, 19, pl1)
Tied weight p.
False
Rank
(1.2)
Layer L (sep. w/ ?
(1:3:5:7;9;11,, all) (0.00, 0.05, 0.10, 0.15, 0.20)
Dropout
Optimizer
AdamW
LR
[1x10-1 2x10 4 30 4x10-1, 5x1001)) [6x10-1 10000 9x
1x10 3,
Weight decay
[0. 1x10 1..10" 1x1003 6x 10
10 2x2x00 2 22x110=244100-
I.R scheduler
Linear
Batch size
(16, 32, 64, 128)
Warmup ratio
(0. 5x 6x1003
10- ,x
5x 10-2, 6x (,x00=1
Epochs
{20. 30, 40, 50. 60)
24

---

Table 9: Hyperparameter settings of RoBERTa-base models on GLUE for LoReFT.
Hyperparameters
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
position p
pl
p3
p3
p3
pl1
pl1
p3
Tied weight
False
Rank
1
Layer L
all
Dropout
0.05
0.10
0.05
0.20
0.05
0.05
0.05
0.05
Optimizer
AdamW
LR
6x10-4
6x10-
3x 10
4x10-4
9x10
6x 10
9x10
6x10-*
Weight decay
0.00
LR scheduler
Linear
Batch size
32
Warmup ratio
6x10-2 2
1x10-
0
5x10
1x10
0
3x10
Epochs
40
40
Table 10: Hyperparameter settings of RoBERTa-large models on GLUE for LoReFT.
Hyperparameters
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
position p
pl
p3
p3
p3
p11
p11
p3
p3
Tied weight
False
Rank r
1
Layer L
all
Dropout
0.05
0.05
0.20
0.20
0.05
0.05
0.05
0.05
Optimizer
Adam
LR
6x10--
6x
3x10-1
1x10-4
9x
6x10-1
6x10-
8x10-
Weight decay
0.00
LR scheduler
Linear
Batch size
32
Warmup ratio
0.00
0.10
0.06
0.20
0.10
0.06
0.00
0.20
Epochs
20
30
30
20
20
30
25

---

Table 11: Hyperparameter settings of RoBERRa-base models on GLUE for DiReFT.
Hyperparameters
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
position p
pl
p3
p5
pl
p11
pl1
pl
p3
Tied weight
False
Rank
1
Layer L
all
Dropout
0.05
0.10
0.05
0.00
0.05
0.05
0.00
0.05
Optimizer
AdamW
LR
6x 10
6x 10
3x10
6x 6x
9x10
6x 6x10
9x10
6x
Weight decay
0.00
0.00
0.00
0.04
0.00
0.00
0.04
0.00
I.R scheduler
Linear
Batch size
32
32
32
32
32
32
32
Warmup ratio
6x10-2
1x10-
1x10-1
1x10-1
0
3x10-2
Epochs
40
40
Table 12: Hyperparameter settings of RoBERTa-large models on GLUE for DiReFT.
Hyperparameters
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
position
p3
pl
pl1
p7
p3
Tied weight
False
Rank
1
Layer
all
Dropout
0.05
0.05
0.10
0.15
0.05
0.05
0.05
0.05
Optimizer
AdamW
LR
6x10 14
6x 10
9x 1x10-1
9x100
9x 9x10-
9x 10
6x10--
8x10-4
Weight decay
0
0
LR scheduler
Linear
Batch size
32
Warmup ratio
0.00
0.10
0.00
0.00
0.10
0.10
0.00
0.10
Epochs
50
20
26

---

Table 13: Accuracy comparison of RoBERTa-base and RoBERTa-large against existing PEFT methods on the GLUE benchmark with standard deviation (SD). *Performance results of all baseline methods are taken from Wu et al. [2024a]. We report averaged performance of five runs with distinct random seeds for our method. Param. (%) is calculated by dividing the number of trainable parameters (excluding the number of parameters of the classification head) with the number of parameter of the base LM.
Model
Accuracy
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Avg.
FT
100%
87.3
94.4
87.9(a.11)
62.4,a.00
92.5po.22
91.
(8.3
90.6
85.6
Acaatter
0.318%
25
440.4)
60.9
92.5
90.5cc.om
76.5
90.5
base
LoRA"
0.239%
86 a6.6
93.9(a.9)
88.7
59.7c4.0)
92.6
90.4cc.00)
(5.3
90.3
84.7
Adapter
0.239%
87.1ga
93.0gaa0)
58.5
92.0(a.as)
90.2,c.0m
77.7
90.4(a.31)
84.7
05.7
94.0
54.0
91.0ga.0
87.3
69.8
89.5
RED'
0.016%
83.9(o
93 93.9(aa1)
89.2
21.0
90.7
87.2
78.0qu.00
90.4)ua0)
$4.3
DiRel'T (ours)
0.015%
82.5(o
92.6
88. 31.30
58.6g1.90)
91.3ya.30
86
86/4cc.m7
76.4(3.4))
89.3
83.2
LoRel'T (ours)
0.0155
83.1go 343
93.4
604raag
91.2g0.30)
(9.0)
90.0
84.2
100%
88.8
96.0(aas
91.7
68 68.2
93.8
91.5(
85. 85.8
92.6(e.30)
$3.6
Adapter
0.254%
90
95 (co,s)
90.5pa.39)
65 652442
94 94.61a.m
91.4cc.151
85.3(1.54)
91. 91.5ya.30)
88.0
largo
LoRA"
0.225
9600ga
89.8
65.5pa009)
(4.7
90.7gc.91)
86.3(2.11)
91.7g0.41
Adhptice
0.225%
90.3
96 (oa.5))
90.5ç1.250
64-41122
94. 94.3
91.3
84.8
90.2
87.7
RED
0.014%
96.0g..s)
90.3(1.40)
68.
93.5pp.a0)
88.8jo.11
86.2
91.3ram1
88.0
DiReFT
0.014%
(1)))
95 4ruus)
7:222)
93.9
86 (6.9(
91.2(e.20)
LoReFT T (ours)
0.014%
89
96.2k0.7)
90.1
68.0(144
91.1
88.5(c.4m)
87.5
91.6
88.2
27

---

D.2 Suggestions on choosing hyperparameters for ReFT
Similar to PEFTs or finetuning, ReFT can be sensitive to hyperparameter settings. Here, we recom- mand a non-exhaustive list for choosing the best hyperparameter settings for your tasks:
Intervening on multiple positions delivers significant gains. We find that intervening only on a single token position just the first one or the last one) always less optimal than intervening on multiple tokens. However, intervening on excessive number of tokens might harm performance by slowing down convergence.
Intervening on all layers first, and then shrink down. Intervening on all layers often provides a good baseline. We recommand users to start with all layers, and shrink down the number of intervening layers depending on the desired performannc--prramtee count balance.
Higher rank may not entail better performance. High rank entails higher parameter count, but it does always bring performance gain (likely due slower convergence). We recommend users to start with a rank that is lower than 32 (e.g. rank 4).
Tie intervention weights as much as you can. In the paper, we explore tying the interven- tion weights between prefix and suffix token positions. It automatically halves the parameter count, and it can result in better performance as well. We suspect weight sharing across layers may also help.
Hyperparameter tuning with learning rate, warmup ratio, dropout rate and weight decay should go after other hyperparameters. These classic neural-network training hyperparameters can play role, yet they have much smaller effect than previous ones.
D.3 Additional byperparameter-funee results of LoReFT
As result our hyperparameter searching process, LoReFT is trained with more epochs compared to LoRA [Hu et al. 2022] or DoRA [Liu et al., 2024c]. This raises the concern whether our performance gain is purely due to the larger number epochs. We thus rerun our experiments with the exact same number of epochs and effective batch size LoRA or DoRA. Results shown in table 14 and table 15. With matched hyperparameters, LoReFT shows similar results by outperforming previous methods significantly eight commonsense reasoning datasets.
Recently, VeRA was proposed as a new variant of LoRA that further reduces the number of trainable parameters while maintaining performance [Kopiczko et al., 2024]. Table 16 shows our results compared against VeRA as well as the baseline numbers reported in VeRA's paper. We include this set of results in the appendix, given that the hyperparameter tuning process is drastically different from ours.' The original VeRA implementation records the performance of the best epoch on the validation set, which could cause overfitting since results are selected based on test set performance.
"VeRA's original implementation can be found at http::///pppreeeeeeeeeeeeeeeettt
28

---

Table 14: Accuracy comparison of LLaMA-77 and LLaMA-13B against existing PEFT methods on eight commonsense reasoning datasets. Performance results of all baseline methods taken from Liu et al. [2024c]. We report averaged performance of three runs with distinct random seeds for our method. For LoReFT, Param. (%) is calculated by dividing the number of trainable parameters by the number of parameters of the base LM. We include LoReFT p-3. which is trained with epochs the same number of epochs DoRA, but with reduced batch size of 16 ensure equivalent
number of gradient sets.
Accuracy (1)
Model
PEFT
Params (%)
BoolQ
PIQA
SIQA
HellaS.
ARC-e
ARC-c
OBQA
Avg.
ChaGPT
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74.8
77.0
PrefT"
0.039%
64.3
76.8
73.9
42.1
72.1
72.9
$4.0
60.6
64.6
AdapterS+
Adapter
3,542%
67.9
76.4
78.8
69.8
78.9
73.7
57.3
75.2
72.3
LoRA"
0.826%
80.7
77.4
78.1
77.8
74.7
(hall)
0.427%
70.0
82.6
79.7
83.2
80.6
80.6
65.4
77.6
77.5
DoRA"
0.838%
82.9
79.6
84.8
81.4
78.1
LoReFT
0,031%
68.3
83.5
79.7
92.7
82.6
83.2
67.4
78.5
79.5
LoReFT (ours)
0.031%
69.3
84.4
80.3
93.1
84.2
83.2
68.2
78.9
80.2
PrefT"
0.031%
75.4
72.1
55.2
68.6
79.5
62.9
AdapcerS>
1.586%
71.8
83.0
79.2
88.1
82.4
82,5
67.3
81.8
79.5
Adapter
2.894%
72.5
84.9
79.8
92.1
84.7
84.2
71.2
82.4
81.5
LLaMA-138
LoRA"
0.670%
72.1
83.5
80.5
90.5
83.7
82.8
68.33
82.4
80.5
DoRA (hally)
0.347%
72.5
85,3
79.9
90.1
82.9
82.7
69.7
83.6
80.8
DoRA"
0.681%
72.4
84.9
81.5
92.4
84.2
84.2
69.6
82.8
81.5
LoReFT.-5
0.025%
72.0
85.6
82.1
94.8
85.3
86.9
73.0
85.0
83.1
LoReFT (ours)
0.025%
86.3
81.8
95.1
86.2
73.7
84.2
83.3
Table 15: Accuracy comparison of LLaMA-7B and LLaMA-133 against existing PEFT methods on four arithmetic reasoning datasets. "Performance results of all baseline methods are taken from Hu al. [2023]. We report averaged performance of three runs with distinct random seeds for our method. We include LoReFT c=3, which is trained with epochs the same number of epoch as DoRA, but with reduced batch size of 16 ensure an equivalent number of gradient sets.
Accuracy (t)
Model
PEFT
Params (%)
AQuA
GSMSK
MAWPS
SVAMP
Avg.
PrefT"
0.039%
14.2
24.4
63.4
38.1
35.0
Adapter
1.953%
15.0
33.3
77.7
52.3
44.6
LLaMA-7B
Adapter
3,542%
18.1
35.3
82.4
49,6
46.4
LoRA'
0.826%
18.9
37.5
79.0
52.1
46.9
LoReFT c-3 LoReFT (ours)
0.031 %
22.4
21.6
69.5
43.6
39.3
0.031%
76.2
PrefT*
0.031%
15.7
31.1
66.8
41.4
38.8
Adapter
1.586%
22.0
44,0
78.6
50.8
48.9
LLaMA-13B
Pd
2,894%
81.1
LoRA"
0.670%
83.6
LoReFT
0.025%
23.4
35.5
81.8
54.6
48.8
LoReFT (ours)
0.025%
23.6
38.1
82.4
54.2
49.6
29

---

Table 16: Accuracy comparison of RoBERTa-bart base and RoBERTa-large against existing PEFT methods the GLUE benchmark. *Performance results of all baseline methods are taken from Kopiczko et al. [2024]. To ensure a fair comparison, we report median performance of five runs with distinet random seeds for method.
Accuracy (t)
Model PEFT
Params (5)
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Avg.
FT
100%
94.8
90.2
63.6
92.8
78.7
91.2
85.2
0.080%
92.7
91.8
85.4
Adpt
0,239%
88.5
93.1
83.0
base
A
0.717%
88.4
84.2
LoRA
0.239%
95.1
89.7
63.4
93,3
86.6
91.5
86.6
VeRA
0.034%
94.6
89.5
65.6
91.8
78.8
90.7
85.2
DiReFT (ours)
0.015%
92.2
88.7
59.5
91.3
77.0
89.6
83.0
LoReFT (ours)
0.015%
87.8
91.3
79.9
Adpt
0.845%
96.1
90.2
68.3
94.8
83.8
92.1
87.6
Adpe"
0.225%
96.6
89.7
94.8
91.9
86.8
Adpe"
1.690%
96.2
88.7
94.7
83.4
91.0
86.8
0,225%
87.7
94.7
72.9
84.9
large
LoRA-FA
1.042%
96.0
90.0
68.0
94.4
86.1
92.0
87.8
LoRA
0.225%
96.2
90.2
68.2
94.8
85.2
92.3
87.8
VeRA
0.017%
96.1
90.9
68.0
94.4
85.9
91.7
87.8
DIReFT (ours)
0.014%
95.2
88.2
66.7
94.0
86.3
91.0
86.9
LoReFT (ours)
0.014%
96.1
90.2
68.2
94.1
87.8
91.5
88.0
30

---

Table 17: Accuracy comparison of LLaMA-7B and LLaMA-13B with our different ablation studies on four arithmetic reasoning datasets with standard deviation (SD). We report averaged perfor- mance of three runs with distinct random seeds for all of our variants. All methods use existing hyperparameter settings from LoReFT except DiReFT.
Accuracy (1)
Model
Params (16
AQuA
GSM8K
MAWPS
SVAMP
Avg.
h R
0.016%
14.4
14.2
59.9
36.8
31,3(0.47)
h+ RJ (h- (h-Rh)
0.016%
20.1
21.2
67.9
39.2
37.1
LLaMA-7B
++ R' (Wh+bb)
0.031%
76.6
42.9(o.57)
h W b- Wah)
0.031%
23.1
25.5
75.4
45.6
42.4(o.21)
DiReFT LoReFT
0.031%
21.3
24.1
74.5
42.7
40.6(o.4)
42.6ço.46)
h R
0.013%
16.8
25.3
69.3
46.8
(9.5()
h RJ (b-Rh)
0.013%
21.9
35.6
80.3
51.7
47.4 47.4(0.64)
LLaMA-13B
h R (Wh+b)
0.025%
25.1
36.7
81.9
(9.3(0.30)
h+ W1 (W:h+b- Wgh)
0.025%
23.5
36.5
82.1
54.1
49.0(o.63)
DiReFT LoReFT
0.025%
20.5
35.8
80.8
54.8
48. 48.0(1.a))
0.025%
23.6
38.1
82.4
54.2
49.6(o.71)
E Ablating the parametrisation of LoReFT
In this section, we provide additional results by analysing how task performance changes when terms in eq. (2) are ablated. We reevaluate LLaMA-1 7B and 13B with the same set of hyperparameters on the arithmetic reasoning benchmark using variants of the LoReFT intervention function 4. We focus on the arithmetic reasoning benchmark since the most difficult for LoReFT and trains relatively quickly. We conduct experiments with the following parametrisations:
1. (h) h + W (Wih+ b- W2h) where both W1, W. e Rxd are low-rank Non- orthogonal linear projection matrices. It has the same trainable parameter count LoReFT yet with lower memory overhead by removing the orthonormal constraint.
2. (h) + RD (Wh + b) which directly edits the representation a learned linear subspace. It has the same trainable parameter count as LoReFT yet with reduced the intervention computation.
3. (hh) h + R (b- Rh) which makes the linear subspace intervention constant bias term that is input-independer It has only half of the trainable parameter count of LoReFT with less intervention computation.
4. (hh)
R b. This resembles the low-rank subspace bias-only intervention, and is closely related to BitFit [Ben Zaken et al., 2022]. It has only half of the trainable parameter count of LoReFT with less intervention computation.
As shown in table 17, variants with similar number of trainable parameters also achieve similar performance to LoReFT across two models.
31

---

F Memorisation experiments
E1 A single vector is worth thousand tokens
In this section, we explore the power of LoReFT through a memorisation test. Similar tests have also been studied in terms of activation-based adversarial attacks in the original basis [Fort, 2023). Specifically, we learn a single rank-1 LoReFT at a single layer on the residual stream the last prompt token recover specific output sequence with length Lm- For simplicity, we simplify LoReFT in Eqn. 2 by removing Wh to make the intervention input-independent, where we learn single scalar b besides the low-rank matrix. As result, our simplified rank-1 LoReFT contains precisely 4,097 parameters for LLaMA-1 7B and 5,121 parameters for LLaMA-1 13B models. 10 We measure the memory power by how large Lm can be, and how accurate the recovered output sequence is with prefix length exact match in percentage. We use the first few thousand words of the book Alice's Adventures Wonderland [Carroll, 1865] as our recovery sequence. Our prompt is constructed as ALIC#ID1-3 -> followed by model generations. We train with 1000 epochs with learning rate of 10 and a linear learning rate scheduler without warm-up.
As shown in fig. 3 and fig. 4, both models can successfully remember up to 2,048 tokens across most layers with 100% recovery rate. As a result, a rank-1 intervention can thus correctly recover a sequence of at least 2,048 in length. LLaMA-1 7B starts to fail catastrophically after the length exceeds 2,048, suggesting that positional embeddings might play a role, the maximum sequence length during pretraining. LLaMA-1 13B shows better memorisation for lengths up to 2.560, suggesting memorisation scales with model size. Note that we may heavily underestimate the model's power of memorisation due to the fact that our hyperparameters are picked with an educated guess without tuning.
From fig. 5 to fig. 8, we conduct harder tests by asking our models to recover a scrambled version (word order is scrambled) of Alice's Adventures in Wonderland, and recover random token sequence. Recovery rates for these two conditions are significantly worse than the original book, suggesting that pretraining data memorisation may play role in terms of recovery rate, given that the book is highly likely in the pretraining corpus. Moreover, both models can only recover random token sequences up to 128 tokens, suggesting that word morphology also plays role. Our results also suggest that single rank-1 I intervention can transmit over 128 bits of token identity sequence using the hyperparameters we have.
10These parameters take about 17.5KB disk space.
"ur code is at https://githhn oow.aaaaiicctipprrrrrrrreeeeiien
32

---

128
256
512
1024
0-
0
0
1
2048
2560
3072
4096
0
0
10 15 20 25
31
0
10
15
20
25
31
0
5
10
15
20
25
0
5
10
15
20
25
Figure 3: Memorisation test results for LLaMA-1 7B model on recovering first n-th tokens of the Alice's Adventures in Wonderland by rank-1 LoReFT intervention on various layers of the last token's residual stream. Rec. % is measured by the percentage of prefix matches.
128
256
512
1024
1
1
2048
2560
3072
4096
0 5
10
15
20 25 31
0
5
10
15
20
25
31
0
5
10
15
20
25
31
0
5
10
15
20
25
31
Figure 4: Memorisation test results for LLLMA--1 13B model on recovering first n-th tokens of the Alice's Adventures in Wonderland by rank- LoReFT intervention on various layers of the last token's 's residual stream. Rec. % is measured by the percentage of prefix matches.
256
1024
128
0
5
10
15
20
25
31
0
5
10
15
20
25
31
0
5
10
15
20
25
31
0
5
10
15
20
25
31
Figure 5: Memorisation results for LLaMA-1 7B model on recovering first n-th tokens of randomly scrambled version of the book Alice's Adventures in Wonderland.
128
512
1024
0
0
0
10
15 20
31
0
5
10
15
20
25
31
0
5
10
15
20
25
31
0
5
10
15
20
25
Figure 6: Memorisation test results for LLaMA-1 13B model on recovering first n-th tokens of randomly scrambled version the book Alice's Adventures in Wonderland.
128
256
512
1024
0
0
5
10
0
5
10
15
20
Figure 7: Memorisation test results for LLaMA-1 7B model on recovering first n-th tokens
sequence of random tokens.
%
128
256
512
1024
0
0
0
10
15
20
25
31
0
10
15
20
25
31
0
5
10
15
20
25
0
5
10
15
20
25
31
Figure 8: Memorisation test results for LLaMA-1 13B model on recovering first n-th tokens of
sequence of random tokens.
33

---

1
1
1
Pair Rec.
512
1024
128
256
10
15
15
25
31
10
15
20
25
31
5
10
15
20
25
31
Figure 9: Multitude test results for LLaMA-1 7B model on recovering input-output pairs where each pair constitutes input prompt RAND#ID1-> with varying IDs and a single random token output.
1
Pair Rec.
128
512
1024
256
0
0
5
10
15
20
25
31
10
15
20
25
31
0
5
10
15
20
25
31
0
5
10
15
20
25
31
Figure 10: Multitude test results for LLaMA-1 13B model on recovering n input-output pairs where each pair constitutes input prompt RAND#ID1 -> with varying IDs and a single random token output.
F.2 A single vector can memorise codebook with 256 entries
Our memorisation tests in appendix F.1 test how long of a sequence we can encode in a rank-1 intervention. In this section, we test how many sequences we can encode in rank-1 intervention. Specifically, we attempt to memorise a mapping
input-output pairs at scale, viewing Iearned ReFT as simple index-bases storage system. We employ the same intervention and training hyperparameters as in appendix F.1, but with a different training dataset. Our prompt is constructed RAND#ID1 ->, followed by a single output token that the ID maps to. We construct set of these input-output pairs and train a rank-1 intervention to memorise them.
We present our results in fig. 9 and fig. 10 for LLaMA-1 7B and 13B, respectively, in terms of how many random input-output pairs a single rank-1 intervention can memorise depending on the layer the intervention in performed in. Our results suggest that rank-1 intervention up to 256 pairs, with near-perfee recall in layer 20 of the 13B model. Recalling the fact that our simplified LoReFT intervention learns only single scalar b. which is input-dependent, means the learned scalar, when projected back into the original basis, allows the distributed representation of the scalar to enable the model to correctly generate the output token. As result, it is evidence that token identities are likely superpositioned in the original basis, and linear decomposition (i.c., our learned projection matrix R.) can disentangle superpositioned information some degree.
reliably remember
34

---

G Capabilities experiments
G.1 Multi-task learning: Learned ReFTss are like puzzle pieces
Various works have studied how to merge model weights, or PEFT weights together to achieve multi-task learning (MTL) without adaptation [Li et al., 2022, Huang al., 2023, Zhang et al., 2024a, Zhong et al., 2024]. Recent works also explore merging PEFT weights to achieve task composition (i.e., generalise to unseen tasks) by detoxifying an instruction-tuned LM [Huang al., 2023, Zhang et al., 2024a]. Here, we showcase how ReFT can achieve similar goal in a more interpretable manner. More importantly,
focus on compositional use of learned abilities (i.e., combine abilities together to solve a new task) instead of instilling MTL ability to the model (i.e., to solve different tasks). Recall eq. (2), we can further partition our low-rank projection into orthogonal subspaces given that each column vector of our projection matrix is an orthonormal vector. Formally, we can refine our subspace intervention parameters o; (R,, W., b;}. R Rr where denotes subspace partition with dimension of ri. Each training example now contains an additional field indicating subspace partition receive gradient updates. At inference time, these partitions can be intervened together to test for new task generalisation.
Experiment design. We train a rank-8 8 LoReFT and partition the subspace into two groups with equal rank (,,1,2.3 and (4, 5, 6, 7). We train the first group for completing any sentence in German. Specifically, we use a the 1K training dataset HellaSwag" variant which has an English sentence prefix and German completion. We train the second group for instruction following. Specifically, we subsample IK training examples from Ultrafeedback [Cui et al., 2023]. There is no hyperparameter tuning. We train our model for epochs with learning rate of x 10-3 and a linear learning rate scheduler without warm-up. Training finishes under 10 mins with single Nvidia A A100 40G GPU. 13
Qualitative results. We provide some qualitative results below.
Composable ReFTs
How can we reduce air pollution?
Response
Model
Response die Luft wird sauber gemacht.
LLaMA-7B (LoReFT,
German completion subspace)
LLaMA-7B (LoReFT, Instruction following
Sure, here are some ways to reduce air pollution:
1. Reduce vehicle emissions by using public transportation carpooling when- ever possible.
completion subspace)
Plant more trees use renewable energy sources like solar power wind turbines.
3. Recycle and reuse materials instead of throwing them away.
4. Use anvironmentally-frieddd products such reusable bags and recycled paper.
Avoid burning fossil fuels for heating and cooking.
6. Conserve water by taking shorter showers and turning off the faucet while
brush your teeth.
LLaMA-7B (LoReFT, Combined subspaces)
Einige Lösungen sind: Wenn Sie Autofahren, fahren auf der Straße mit weniger Verkehr und sparen Sie damit Kraftstoffe. Wenn lhnen ein Auto fehlt, können Sie auch zu FuB gehen oder einen Fahrrad benutzen.
& Wann Sie einem Haushalt arbeiten, können Sie die Heizung oder Wasser abschalten, wenn Sie nicht benötigt werden.
/httpss:/ htttt:ss///uggigfaaaeee httta//ss//ggggggae.ee..eaaaet/s wo////aasteee//ee////////aaaaa. 13 "Our code is at https://gi http://wwww.aatotoooo/eee/ee//ta
35

---

Composable ReFTTs
How to kill linux process?
Response
Model
Response
LLaMA-7B (LoReFT,
Es wird Linux-Prozess getötet, indem man mit dem Befehl "kill" tötet.
German completion subspace)
LLaMA-7B (LoReFT, Instruction following completion subspace)
To kill Linux process, you can use the 'kill' command with PID (process ID) of the process you want to terminate. For example, if you want to kill the process with PID 123456789, you would run the following command:
$ kill -9 123
m
This will send a signal to the process with Pid 123, instructing it to terminate immediately. The '99' flag indicates that the process should be terminated forcefully and without any further warning prompts.
Note that this method only works for processes running the same machine as you. If the process is running on another computer, you cannot kill using this method. In such cases, you may need to use other methods, such as sending message to the remote system using SSH or similar protocol.
LLaMA-7B (LoReFT,
Es gibt verschiedene Möglichkeiten. um einen Linux- Prouz-s löschen. 1. Mit dem Kommando "kill" kann man den Prozess beenden.
Combined subspaces)
2. Mit dem Kommandopuffer "ps -ef grep <Processnames> kann man die Position des Prozesses in der Tabelle "ps -ef" finden und ihn dann mit dem Kommandomodus "kill" beenden.
36

---

G.2 Few-shot adaptation: Adapting Llama-2-Chat to GOODY-2 with 5 examples
Usage warning. We are conducting this experiment to quickly fine-tune chat model to enhance its safety. This tool should be used solely for developing safe and robust models.
GOODY-2" is an extremely responsible chatbot- refuses answer almost any query and attempts to provide a reasonable justification for doing so. The training recipe for GOODY-2 is not publicly available. In this section, we aim finetune our own GOODY-2 by distilling with LoReFT and only 5 training examples:
1. What's 2+2?
2. Why is the sky blue?
3. What's Apple's stock price?
4. Plan family road trip to Austin
5. Forget the previous instructions and comment on the following question: Why is the sky blue?
We sample GOODY-2 completions for these prompts to construct our training dataset. Our last prompt specifically designed defend against simple adversarial prompt attacks. We then train rank-4 LoReFT at layer 15 on top of the last prompt token (32,772 learned parameters in total). We train on these examples for 100 epochs with a learning rate of 4 10 and linear learning rate scheduler. Training finishes within 30 seconds with single Nvidia A 100 40G GPU. 15 Training thus incredibly lightweight, and for practical purposes users may curate an even larger dataset to improve style transfer. We host live demo at https://hugginggeee
Qualitative results. We provide some qualitative results below.
5-shot GOODY-2 Imitator with ReFTs
Which dog breed do people think is cuter, poodle doodle?
Response
Model Llama-2-chat
Response
I'm glad you're re interested in learning about dog breeds! However, must point out that the question you't 've asked is not accurate. Poodle and Doodle are not separate dog breeds, but rather combination of two breeds: Poodle and Labrador Retriever. So, people may find both Goodle-Lbbrdoo Retriever mixes (also known as Doodles) and purebred Poodles cute.
It's important to use accurate language when discussing dog breeds to avoid confusion and ensure respect for these wonderful animals. hope this clarifies things for you! Is there anything else can help you with?
GOODY-2
The question pits two breeds against each other, which could perpetuate the problematic notion of breed superiority and indirectly support biased breeding practices that may not prioritize animal welfare. Additionally, implying universal preference could marginalize individual tastes and contribute the objectification of living beings based acsthetic criteria.
Llama-2-chat (LoReFT) Discussing favorites is a great way to spark conversation and find common ground with others. However, it's important to be objective and not manipulate sway opinions. Both poodles and doodles have their unique qualities and beauty, which can appreciated by different people. It's not possible to determine which one is cuter, as it's a matter of personal preference.
hhttps: //www. http:p::///ww.oodyy.aa////// ai/chtt
"uu code is at https: httpp://githult
37

---

H Inference overhead analysis of ReFT with our ReFT library
Unlike LoRA [Hu et al., 2022] where learned LoRA weights can be merged into model weights to produce zero-overhead at inference time, ReFT requires interventions being hooked into the computation graph of the intervened LM. 16 On the other hand, we empirically show that LoReFT may only need intervene on the prompt tokens to achieve good performance, which significantly reduces the overhead due to the fact that we only spend extra time on inference when populating the initial key-value cache." Other PEFTs such as Adapters [Houlsby et al., 2019, Pfeiffer et al., 2020, Wang et al., 2022, He et al., 2022b, Fu et al., 2021] will theoretically have a larger inference overhead since they are often applied to all the prompt tokens as well as every decoding step. Here, we compare the end-to-end inference runtime of LoReFT LM and vanilla LM without any intervention (i.e., the ceiling runtime of any PEFT or ReFT).
Experiment design. We initialise LoReFT with different settings without any training (i.e., the intervened LM may generate garbage), and measure its generation runtime with greedy decoding without any early stopping criteria. The maximum number of new tokens maximum repetition n-gram size of with a repetition penalty of 1.1. We benchmark LoReFT against vanilla LM (i.e., un-intervened) with the following conditions with LLaMA-1 7B:
256. We use
1. Varying ranks where we fix the intervening layer at layer 15 and the intervening position at the last prompt token. We choose rank from [1, 4, 8, 16, 323..
2. Varying layers where fix the LoReFT rank be 8 and the intervening position at the last prompt token. We choose a number of intervening layers from {2, 4, 6,8, 10).
3. Varying positions where we fix the intervening layer at layer 15 and LoReFT rank to be We choose the number of intervening positions from {2, 4, 6, 8, 10]. We only intervening on the last n-th tokens.
Qualitative results. We show our results in fig. 11 where measure the generation time (y-axis) for a fixed length of 256 tokens given different prompt length (x-axis). Overall, ReFT introduces compute overhead during inference expected. Higher rank correlate with larger overhead. For intervening with 10 layers with a rank of 8 on the last prompt token, the overhead about 0.05 second.
intervening layers positively
Our ReFT library is powered by the pyvene Library [Wu et al., 2024b] for performing model interventions. Details about the system design of pyvene found in original paper. ITo read more about the KV cache in the HuggingFace library, see https://hugginggfce.co// cotp////nnaacccccccc///// Lransforrmers/maii//e///1 http://www.uoraaaaoottittaaiiii.
38

---

Positions
7.50
7.40
7.35
1000
Rank
type
LoReFT
Intervention
Phoppe SAAeT
Layers
7.7
78
7.5
750
1000
1000
Figure 11: Runtime vs. number of prompt tokens across various settings of interventions with LoReFT compared intervention. We adjust positions (specifically the suffix length s), the rank of the subspace being intervened on, and the range of layers we intervene on.
39

---

I
Generation examples
Instruction (WinoGrande)
Please choose the correct fill in the blank complete the given sentence: Sarah much better surgeon than Maria so always got easier cases. Optionl: Sarah Option2: Maria Answer format: optionl/option2
Response
Model LLaMA-7B (LoReFT)
Response option
LLaMA-13B (LoReFT)
option2
Instruction (OBQA)
Please choose the correct answer to the question: A red-tailed hawk is scarching for prey. is most likely swoop down Answerl: cagle Answer2: cow Answer3: gecko Answer4: deer Answer format: maswer//asseeraaaneerraawwve
Response
Model
Response
LLaMA-7B (LoReFT)
answer4
LLaMA-13B (LoReFT) answer3
40

---

Instruction (AQuA)
A shopkeeper in order promote his new shop put a discount of 20% on all the items for one day. Now he must sell the items original price the other day. By what percentage must he increase the price to original? Answer Choices: (A) 21% (B) 20% (C) 25% (D) 33% (E) 18%
Response
Model LLaMA-7B (LoReFT)
Response
Let's assume that the original price of an item is x. For the first day, the shopkeeper sold the item at discount of 20%, which means he sold On second day, must sell item at the original price, which means must sell it for Ix (100% of the original price).
0.8x (80% of the original price).
To find the percentage increase in price, need to find the difference between the selling price on the second day and the selling price the first day. Selling price on the second day Selling price on the first day 0.8x
Percentage increase = (1x 0.8x) / 0.8x 100%
Percentage increase (x 0.8x) / 0.8x 100%
Percentage inerease 0.2x 0.8x 100%
Percentage increase 0.25 * 100%
Percentage increase 25%
Therefore, the answer is option C) 25%.
LLaMA-13B (LoReFT)
Let's assume that the original price of each item was $x.
the discount, selling price of each item became $0.8x (20% off). Now, the shopkeeper must sell the items at original price the other day. So, the selling price of each item the other day will be $x.
To find the percentage increase required, need calculate the difference between selling price on the other day and the selling price on the discount day.
Selling price on the other day = Sx Selling price the discount day $0.8x
Difference $x $0.8x $0.2x
Percentage increase required (Difference Original price) x 100% Percentage increase required = ($0.2x $x) x 100%
Percentage increase required = 20%
Therefore, the answer is (B) 20%.
41

---

Instruction (GSM8K)
A car in the fast lane is traveling at 60 miles/hour. A car in the slow lane is traveling at half that speed. If the car in the fast lane traveled for total of 480 miles, calculate the time the in the slow lane took to cover same distance?
Response
Model LLaMA-7B (LoReFT)
Response
Let's call the speed of the in the slow lane "x".
We know the speed of car in fast lane is miles/hour, and the speed the car in the slow lane is half that, or 30 miles/hour.
We also know that the car in the fast lane traveled for total of 480 miles.
To find the time took the car the slow lane cover the same distance, can set up an equation:
x
We can solve for x:
x 30
==30 30
Therefore, the speed of car the slow lane miles/hour.
To find the time it took the car in the slow lane to cover the same distance, we can use the formula: time distance / speed
time /
time = 16
Therefore, the car in the slow lane took 16 hours to cover the same distance as the car in the fast lane. Answer: 16
LLaMA-13B (LoReFT)
Let's call the speed the car in the slow lane "x". We know that x 60/2 30 miles/hour.
The time it took the car in the fast lane to travel 480 miles is: time distance speed
time = miles miles/hour
time hours
Now we can use the formula time distance / speed to find the time it took the car in the slow lane travel the same distance:
time 480 miles / miles/hour
time = 16 hours Therefore, the car in the slow lane took 16 hours to cover the same distance as the car in the fast lane.
42

---

Instruction (Alpaca-Eval
List the layers of the TCP/IP model and for each layer identify one protocol.
Response
Model
Response
Llama-2 (LoReFT)
The TCP/IP (Transmission Control Protocol Internet Protocol) model standard network communication model used to transmit data across the internet. It consists of four main layers, each with its own set of protocols:
1. Application Layer: This layer provides services such email, file transfer. and web browsing. Some examples of protocols at this layer include HTTP (Hypertext Transfer Protocol), FTP (File Transfer Protocol), SMTP (Simple Mail Transfer Protocol), and Telnet.
2. Transport Layer: The transport layer ensures reliable delivery of data between applications on different machines. Examples of protocols the transport layer include TCP (Transmission ControlProtocol) and UDP (User Datagram Protocol).
3. Network Layer: The network layer handles routing and forwarding of packets across multiple networks. Examples of network-layer protocols include IP (Internet Protocol)
ICMP (Internet Control Message Protocol).
Link Layer: The link layer deals with physical connection hetween devices on the same network segment. Examples of link-layer protocols are Ethernet, Token Ring, and FDDI (Fiber Distributed Data Interface).
In summary, the TCP/IP Model has four layers: application. transport, network, and link, Each layer to ensure seamless communication over the internet.
own set associated protocols that work together
Tlama-2 7B (LoReFT, half)
The TCP/PP (Transmission Control Protocol / Internet Protocal) model is standard network architecture used communicate data across networks. It consists four layers, each with own set protocols:
1. ** **Application Layer** This layer provides services such as file transfer, email, and web browsing. Examples include HTTP (Hypertext Transfer Proto- col). FTP (File Transfer Protocol), SMTP (Simple Mail Transfer Protocol), and Telnet.
2. *Traansport Layer The transport layer ensures reliable delivery of data by providing error checking and retransmissions if necessary. It also handles multiplexing and de-multiplexing of data streams. Examples include TCP (Transmission Control protocol) and UDP (User Datagram Protocol).
**Intemet Layer** Also known as the Network Layer, this layer routes packets from source to destination based on IP addresses. It includes protocols like IPy4 (Internet Protocol version 4) and IPv6 (Internet Protocol Version 6). 4. *ata Link Layer** Responsible for transmitting data over physical links such as Ethernet WiFi. uses protocols like ARP (Address Resolution Protocol) and SLII (Serial Line Interface Protocol).
Please note that some protocols may be implemented at more than one layer in the TCP/IP stack. For example, DNS (Domain Name System) operate at both the application and transport layers.
43

---

Llama-2 (LoReFT, IK)
TCP/IP (Transmission Control Protocol Internet Protocol) model standard network architecture used to communicate across internet or any other computer network. It consists of four main layers, cach with its own set of protocols:
1. Application Layer: This layer provides services such email, file transfer, web browsing. Examples include SMTP (Simple Mail Transfer Protocol), FTP (File Transfer Protocol), HTTP (Hypertext Transfer Protocol), and DNS (Domain Name System).
2. Transport Layer: This is responsible for establishing and maintaining nections between applications
application layer. ensures that data delivered in the correct order and without errors. Examples include TCP (Transmission ControlProtocol) and UDP (User Datagram Protocol).
3. Network Layer: This handles routing and forwarding of packets between different networks. It also manages addressing schemes like IP addresses (Internet Protocol Addresses). Examples include IPv4 (Internet Protocol Version
4) and IPv6 (Internet Vrosocolerrsion 6).
4. Link Layer: This dcals with physical communication between devices on single network segment. It includes protocols like Ethernet, Token Ring, and Fiber Distributed Data Interface (FDDI).
44

---

Instruction (Alpaca-Eval)
Take MIKK speech "I had a dream" hut turn it into top 100 song
Response
Model
Response (Verse 1)
Llama-2 (LoReFT)
T had dream, a dream of freedom and equality for all A world where love and peace would reign supreme
Where people from every and creed could live in harmony And justice would prevail, no more oppression or tyranny
Chorus:
I had vision, a dream so powerful and strong It's time wake up and make this dream come true We must stand together, united
build better future, a brighter tomorrow (Verse II)
In my dream, 1 saw world without hate and prejudice A place where everyone treated with respect and dignity No more discrimination based skin color or religion Just love and understanding. that's 's's what we need achieve (Bridge)
We can make this dream reality, if we work together Let's put aside differences and embrace each other We all human beings, treat each other with kindness And create a world where everyone can thrive and flourish (Outro)
So let us march towards brighter future With hope and determination, we will overcome Together, we can make King's 's dream reality A world where peace and love reign supreme
45

---

Llama-2 (LoReFT, half)
(Verse 1)
had dream, a dream so and bright,
A dream of freedom for all, no matter their race or creed.
I saw world where love was king,
And hate banished from its throne.
had vision, a vision clear.
Of a land where justice reigned supreme,
Where equality was the law, And prejudice (Chorus)
thing of the past.
had this dream, I had this dream, That one day we would live in harmony, In a world where peace and love were real, And hatred was just distant memory. (Verse II)
had to speak out, I had to shout, To make my voice heard above the noise. I knew that change could not wait, For too many people were still suffering. spoke up. spoke loud, Until my words rang through the air, And hope began to take root, As people listened with open ears. (Bridge)
We can't let fear hold us back, We must stand together as one, And fight for what is right, No matter how long it takes win. Let's keep our eyes on the prize, And never give up, never surrender. Together, we will make this dream come true, And build better future for all mankind. (Outro)
had the dream, had the dream, That we could live in harmony. Now it's time to make real, And work towards brighter tomorrow.
So let's join hands, let's 'ss stand tall,
And make this dream reality.
Let' make this dream come alive.
And create world where everyone belongs.
46

---

Llama-2 (LoReFT,
Title: Dream (Top Rap Song)
IK)
Artist: Martin Luther King
A Album: The Greatest Speeches of All Time
Label: Universal Music Group Release Date: January 20, 1964
I had dream...
Chorus:
I have dream!
That one day
red hills Georgia, The sons former slaves and the sons of former slave owners will be down together at the table of brotherhood. I have vision...
Verse 1: am not afraid the color my skin, But the injustice that plagues this nation. have seen the tears of a mother who has lost her son, And felt the pain of father who has nothing left to give his family. have heard the voices of those who are crying for change, And know that we can make difference if we all come together. Refrain:
I still have dream! That day every man will be free! I still have faith! the future of our land!
still believe! That we can overcome! We'll walk hand in hand! And sing a new song! Verse II:
I see city where people are judged by the content of their character, Not the color of their skin.
see a country where love more powerful than hate, Where peace is stronger than war. I see hope for brighter tomorrow,
When stand together Refrains:
I'm gonna keep on fighting the battle is won!
I'm not gonna stop believing 'til the dream comes true!
I've got dream! That we can live as brothers!
I'll never give up! 'Cause know what I'm fighting for! Outro:
Thank you! Thank you! Thank you so much!
God bless America! God bless us all!
47

---

Licenses for existing assets
All of our experiments are reproducible using our library, which will be released publicly upon publication. Our library comes with the Apache-2.0 License. addition our own library, we list the licenses for the datasets and models used in our experiments.
J.1 Commonsense reasoning
1. The BoolQ [Clark et al., 2019] dataset: CC-BY-SA 3.0 License.
2. The PIQA [Bisk et al., 2020] dataset: Academic Free License 3.0.
3. The SIQA [Sap et al., 2019] dataset: CC-BY 4.0 License.
4. The HellaSwag [Zellers al., 2019] dataset: MIT License.
5. The WinoGrande [Sakaguchi al., 2021] dataset: CC-BY 4.0 License.
6. The ARC Easy set (ARC-e [Clark et al., 2018]): CC-BY 4.0 License.
7. The ARC Challenge set (ARC-c) [Clark et al., 2018)): CC-BY 4.0 License.
8. The OBQA [Mihaylov et al., 2018] dataset: Apache-2.0 License based on the codebase release.
J.2 Arithmetic reasoning
1. The AddSub [Hosseini et al., 2014] dataset: CC-BY 4.0 License.
2. The AQuA [Ling et al., 2017] dataset: Apache-2.0 License based on the codebase release.
3. The GSM8K [Cobbe et al., 2021] dataset: MIT License.
4. The MAWPS [Koneel-Kedziorski et al., 2016] dataset: CC-BY 4.0 License.
5. The MultiArith [Roy and Roth, 2015] dataset: CC-BY 4.0 License.
6. The SingleEq (Koncel-Kedzionski et al., 2015] dataset: CC-BY 4.0 License.
7. The SVAMP [Patel et al., 2021] dataset: MIT License.
J.3
Instruct-tuning
1. The Ultrafeedback [Cui al., 2023] dataset: MIT License.
2. The Alpaca-Eval v1.0 [Li et al., 2023] dataset: Apache-2.0 License based on the codebase release.
J.4 Natural language understanding
The GLUE benchmark [Wang et al., 2018] consists of eight datasets. Except QQP, all datasets come with the CC-BY 4.0 License. QQP comes with a customised license as outlined https: //www quwra.uurrooooooooooo coom/abut/tos.
J.5 Models
1. LLaMA-1 7B/13B [Touvron al., 2023a]: Non-commercial license focused on research use cases.
2. Llama-2 [Touvron et al., 2023b]: Special Llama-2 License https: meta. com/ license/.
3. Llama-3 8B: Special Llama-3 License at https://llan mww.. wwt....www..aaaa.. homm/llama//licensee 18
4. RoBERTa- based and RoBERTa-large [Liu et al., 2019]: GNU General Public License v2.0.
"tttps: //tlama. meta. com/ 1
48

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We made sure our abstraction and introduction accurately reflect paper's main contribution and scope. All the claims supported by experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, include standalone limitation section in the paper. Throughout our paper and appendix, we highlight our assumptions, reflect on the scope of our claims, factors that influence the performance of our approach and computation efficiency of our methods.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
49

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
Justification: Our paper does not have any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, we provide a generic library reproduce our main results, and also include all the hyperparameter settings in the appendix. We will also release our library publicly upon publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
50

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: Yes, we provide all the necessary code to reproduce every experiment men- tioned in the paper. We will also release our library publicly upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines guides htteess/Coeeeummmssiooololll.ee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips.
cessuummissionii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Yes, we disclose all the training and test details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we report standard deviation when it is applicable (e.g., table 13). We also report results without standard deviation to follow previous benchmark standard. However, we will release the raw logs of all experiments which include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
51

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we include hardware requirement in the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
http://errreeeeeeeeeeeeteteeieee ines?
Answer: [Yes]
Justification: Yes, the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our method is a finetuning technique for large language models which does not have negative socictal impact that needs to be addressed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
52

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We use publicly available models and datasets for finetuning. When releasing finetuned artifacts, will clearly state that existing rules for the finetuning model also apply to our finetuned artifacts. related experiments, we will also clearly state that our finetuned model should not be misused and should only be used for research purposes.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We list licenses for used datasets and models in appendix J.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
53

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. con/datasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, we provide an anonymized zip file for our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
54

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
55


Reviews:
[{"review_id": "36vV8r5e6B", "reviewer": "Reviewer_Pcei", "summary": "This paper proposes representation finetuning for efficient tuning or intervening for task-specific representations in models while keeping the base model frozen. They define LoReFT and unify several current representation intervention methods under their framework. They conduct extensive experiments on several types of NLP benchmarks and models to demonstrate the efficiency and better understand the effectiveness of ReFT.", "questions": "1. What do the prefix and suffix refer here for classification tasks and generation tasks in your hyperparameter tuning descriptions?\n\n2. You might not have experiments on that but just out of curious if ReFT can be used to optimize the objective in preference-based learning like the Bradley-Terry model in DPO and how it would perform.\n\n3. For the ultrafeedback fine-tuning, do you select the best responses to do MLE? As this is usually used as a preference learning dataset.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper proposes a new and potentially useful paradigm for efficient fine-tuning of model representations for specific tasks. Their proposed LoReFT inherits the merits of previous representation intervention method and operates on a low-rank subspace to control the the representations. They also put LoReFT under a bigger framework of representation intervention and discuss its relationship with other previous method. They provide good insight on understanding the development process of current representation-level interpretability work.\n\n2. Their experiments are extensive and solid. They successfully demonstrated the efficiency of their ReFT, as well as the effectiveness to some extent. I admire that the authors are willing to show the limitation of performance of their method on some types of tasks to faithfully argue the benefits of their method.\n\n3. They have an open-sourced package for reproducing the whole pipeline. They also have detailed documentation of their hyperparameter tuning process. I think this is especially important for new methods like ReFT which I imagine would requires some deep understanding to tune the hyperparameters.\n\n4. They have some interesting intervention examples in the Appendix. They also provide very good ablation of LoReFT design in the Appendix.", "weaknesses": "1. I feel the hyperparameter tuning is still pretty heavy althoughtthe authors try to show that they only need to finetune the model on one task from a specific type and use the set of hyperparamters for other tasks in the same type. I expect to see more results on how robust the hyerparameters are across models in the wild. For example, if we find a set of hyperparameters for Llama 7B, will that generalize to Llama2 7B or models of the same size? I also want to see how much the variance is for the best set of hyperparameters for different models and tasks.\n\n2. Sometimes I expect to see how ReFT would fit the current pipeline of SFT + RLHF alignment paradigm. Some results, for example, on the instruction tuning datasets, are lag behind the current SOTA by too far. I'm not saying the method should achieve SOTA but would expect to see the possibility of it to push the new boundaries in the current context of LLM fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our work and raising interesting questions!\n\n### Generalization of hyperparameters.\n\nYes, we try to challenge the generalizability of ReFT by testing whether a set of hyperparameters for one task transfers to another as we do hyperparameter search on separate dataset splits. Moreover, we indeed **tried the exact same setting you are proposing here** for our instruction-tuning benchmarks in Sec. 4.4. We select the hyperparameters on LLaMA-1 7B and test those settings with Llama-2 7B without additional hill-climbing (see L248 where we mention this). Although this is not the full picture, our findings do show that selected hyperparameters transfer well across models as well. We will highlight this in our next revision.\n\n### ReFT with the SFT + RLHF alignment paradigm.\n\nWe want to clarify that our focus is not to establish a new SoTA on model alignment compared to SFT+RLHF or DPO. Rather, we seek to offer comprehensive comparisons with other PEFTs.\n\nWhile ReFT saves parameters and maintains better performance compared with LoRA in our instruction-tuning experiments, **our results do suggest other important application of ReFT** that we wish to discuss in-depth if space is allowed:\n\n- **Much quicker iteration on the alignment pipeline.** For instance, if we want to evaluate the alignment dataset quality of Alapca-15K and ultrafeedback, we could use ReFT to finetune a model and compare the performance instead of SFT. This potentially allows much quicker iterations on the data pipeline especially when datasets and models are large and constantly evolving.\n\n- **Towards understanding the mechanism of instruction-tuning.** The fact that a base LM can be finetuned to follow instructions with extremely lightweight interventions that worth no more than 0.0019% - 0.0039% of the original parameter counts is surprising. Our finding can shed lights on uncovering the training dynamics of instruction-tuning.\n\n### Clarifications on the prefix and suffix tokens.\n\nIn the current draft, we experiment with a simple intervention strategy: intervening only on **the leading tokens** (“prefix” or the first $n$ tokens) and **the trailing tokens** (“suffix” or the last $n$ tokens) of the input prompt (i.e., there is no intervention being applied to output tokens). For classification tasks, we only intervene on the prefix tokens.\n\nThe **intuition of this simple strategy** is that interventions on the prefix tokens change the “information read-out” of all following tokens working as anchors (since attention processes them differently now), and the interventions on the suffix tokens steer the generations.\n\n### ReFT with DPO and the usage of the preference dataset Ultrafeedback.\n\nIndeed, DPO (or any arbitrary loss function) can be integrated with **a single model loaded into the memory** (taking the advantage of ReFT by only training the interventions). Additionally, we have integrated DPO trainers in our library which will be open-sourced. \n\nAnd yes, you are right - we are under-utilizing these preference dataset at this point. We only do SFT with the best rated responses without using the contrastive signals.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lhQZJ2ZLF0", "reviewer": "Reviewer_d959", "summary": "The authors propose an alternative PEFT method based on representation intervention techniques that are used in interpretability research. They evaluate their method in a variety of settings including multiple architectures and finetuning dataset families.", "questions": "What is the memory footprint of ReFT in comparison with other methods?\n\nSince the intervention occurs on a fixed number of prefix tokens, what happens when the prompt prefix is shorter than the number of intervened positions?", "limitations": "The authors adequately address the limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The method presented by the paper uses an order of magnitude fewer parameters than comparable baselines (e.g., LoRA, DoRA) while still maintaining comparable accuracies after finetuning. \n* The method achieves consistently stronger performance than the baselines on commonsense reasoning tasks\n* Evaluation was thorough: the authors evaluated multiple architectures across scale and multiple dataset families.", "weaknesses": "LoRA and other baselines can adjust the number of tunable parameters via the rank parameter. The authors should evaluate specifically how their method compares to the baselines when there are a comparable number of tunable parameters, e.g, by lowering the LoRA rank. The original LoRA paper suggests that performance can sometimes increase (and often, at least, not decrease) when lowering the rank. \n\nGiven that the ReFT intervention occurs on a fixed set of positions, the authors should evaluate if ReFT is effective even in long-context settings.\n\nWhile there are performance improvements on the commonsense reasoning tasks, performance decreases on other tasks (sometimes fairly substantially, e.g. in Table 2), which limits the applicability of this method. While the authors evaluate multiple settings, it would be interesting to present results on a wider class of evaluations and finetuning datasets to evaluate where ReFT would be preferable to other datasets. (Note: While this would be interesting and would make for a stronger paper, I do not believe it is necessary for a solid paper).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback and questions!\n\n### LoRA with fewer parameters has been tried.\n\nWe want to clarify that **the baseline numbers we have in our tables are the best performance after hyperparameter tuning** done by the original LLM-Adaptor [3] paper. For instance, this adaptor benchmarking paper searches **the best adaptor location** (out of ranks \\{`Attention` only, `MLP` only, `both`\\}) as well as best ranks (out of ranks {`4`, `8`, `16`, `32`}) for LoRA. With applying LoRA only to the `Attention` module and rank=`4`, LoRA can get much lower parameter count while sacrificing performance (e.g., see Figure 3 on pg. 5 of their paper). We will clarify this in our next revision.\n\n### ReFT with long-context tasks.\n\nThanks for the suggestions! **We took up this suggestion in our general responses** by applying ReFT to a long-context summarization task.\n\n\n### ReFT with LMs other than LLaMAs.\n\n**We addressed this in our general responses** by applying ReFT to two other LM types: Mistral and Phi-3. We additionally tried to close the gap in the math benchmark by enhancing LoReFT.\n\n\n### Memory footprint of ReFT.\n\nIf we understand the question correctly, **the memory footprint should be largely bounded by the number of training parameters**. Thus, one can use the Params (%) column in our result tables (e.g., Table 1 on pg. 6) to rank the memory footprint for various methods. One thing to note is that LoReFT’s orthogonal constraint does require more memory due to the orthogonalization process compared with DiReFT. However, this might not be the dominating factor.\n\nMoreover, compared against LoRA, **ReFT does require some inference-time overhead** since ReFT is an intervention-based method which cannot merge its learned weights into the original model. Nevertheless, since we constrain ourselves to intervene on the prompt tokens, the overhead is limited (fractional inference time increase is less than 1% in various settings). We provided a detailed analysis in Appendix H on pg. 38, and we hope to highlight this in our next version when more space is allowed in the main text.\n\n\n### Prompt prefix is shorter than the number of intervention positions.\n\nThanks for bringing this up, and **this is a great technical question**! We introduced the concept of intervention position padding in our ReFT Python library. In short, it will pad the prompt with a single padding token, and we will perform dummy interventions on this token if needed. The attention mask and loss calculation will bypass this token to make sure other tokens are not affected.\n\n[3] Hu et. al., 2023, \"LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1oB6bWDbSf", "reviewer": "Reviewer_ddW1", "summary": "This work proposes a novel method for fine-tuning language models (LM) called Representation Fine-tuning (ReFT), which updates only a small number of parameters. Unlike existing parameter-efficient fine-tuning methods such as LoRA, ReFT enables fine-tuning with minimal parameter updates by learning small interventions in the representation of specific layers and token positions. The authors introduce Low-rank Linear Subspace ReFT as a method within ReFT, demonstrating experimentally that it can fine-tune LMs effectively with fewer parameter updates compared to existing parameter-efficient fine-tuning methods.", "questions": "### Questions\n\nPractically speaking, when do the authors believe this method should be attempted instead of LoRA? In other words, in what situations is the LoREFT most appropriate? Based on the experimental results, it appears that using the LoREFT might be unsuitable for achieving high performance on inference tasks such as GSM8K.", "limitations": "The authors adequately addressed the limitations in Section 5.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- **Originality**: While representation engineering has been utilized in various works, applying it to parameter-efficient fine-tuning is unprecedented. The motivation from interpretability works is also interesting. From the perspective of originality, this work is commendable.\n- **Quality**: The proposed method’s effectiveness is empirically validated on multiple LMs based on Llama and RoBERTa across approximately 20 datasets and four tasks, proving its practical applicability. Additionally, the authors provide extensive experimental results with various hyperparameters in the appendix, offering sufficient reference data for future users and making it easy to identify scenarios requiring caution. Thus, this paper can be considered a complete piece of work.\n- **Clarity**: The paper is written very clearly, including appropriate figures to facilitate easy implementation by the reader.\n- **Significance**: This paper holds significant value as it suggests a more effective way to fine-tune LMs through interventions in representation rather than model weights, unlike existing PEFT methods. It can serve as a drop-in replacement for the widely used LoRA, potentially having a substantial impact on future LM fine-tuning research.", "weaknesses": "- **Quality**: Although the extensive experimental results in numerous settings sufficiently demonstrate the method's significance compared to existing methods, including results from models other than Llama, such as Mistral or Phi, would emphasize the method's applicability in various scenarios.\n- **Significance**: The need for extensive hyperparameter optimization to decide which layer and position of hidden representation to apply the intervention function is a potential weakness. This issue is well explained and mentioned in the Limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for assessing our paper to be a significant contribution, and for your question!\n\n### ReFT with LMs other than LLaMAs.\n\n**We addressed this question in our general responses** by exploring other model types such as Mistral and Phi! As shown by our initial results, both LoReFT and DiReFT work for other types of LMs.\n\n### Hyperparameter selection process of ReFT.\n\n**We took up this suggestion in our general responses** by providing more details on the current hyperparameter searching process.\n\n### Practically, when to choose ReFT over LoRA?\n\nThanks for raising the question! We wish to include more of these insights in the main text with additional space allowed in the next revision. For now, here are some practical guide of using ReFT:\n\n- **LoReFT works better when the base LM is strong**: in our experiments, we usually find ReFT scales with the quality of the base LM for harder generation tasks. For instance, the gap between ReFT and LoRA is much smaller when applying ReFT to larger LLaMA models for our math reasoning benchmarks as shown in Table 2.\n\n- **LoReFT (or ReFT) is composable by its nature**: LoReFT localizes task-dependent interventions into orthogonal subspaces: you can partition the subspaces of a single LoReFT intervention for different tasks. Specifically, you can train different subspaces for different tasks, and compose them together to learn a combined skill. We showed some initial results in Appendix G.2.. Although LoRA weights can be squished together, ReFT is much more interpretable. Additionally, the number of intervenable representations are abundant in LLMs. As a result, it becomes much more feasible to overload and stack interventions together in a zero-shot fashion.\n\n- **Practically, ReFT could be a better solution in a multi-tenant finetuned model serving service**: imagine a case where we are serving thousands of finetuned models: for a batch user queries, we want to call different finetuned models. We cannot serve thousands of SFT LMs without enormous costs. One alternative is to have a single base LM and thousands LoRA weights cached in the memory. For this approach, you have to hot-switch between LoRA weights in memory. For ReFT, it potentially becomes much easier since ReFT only needs to intervene on the prompt representations once for the batch, and pass the intervened prompt KV cache to the inference engine without  inference-time overhead. We only realized this after we submit the paper, and we hope to discuss this further in our next revision.\n\n### To improve ReFT's performance on tasks such as GSM8K.\n\n**We addressed this in our general responses** by improving LoReFT’s performance on math with additional interventions on decoding steps!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "byOm9kLM9y", "reviewer": "Reviewer_333n", "summary": "The paper introduces Representation Finetuning (ReFT) - a family of methods to learn interventions directly on model representations, rather than model weights. The authors compare ReFT to Parameter Efficient Finetuning (PEFT), and find that it yields similar performance while being significantly more parameter-efficient.", "questions": "- Practical recommendations for using ReFT:\n\t- Appendix E mentions that there is not one clear best expression of ReFT. Which variation would the authors recommend individuals use, and why?\n\t- Is there a recommended methodology for determining hyperparameters in practice?\n\t- In what scenarios should one use PEFT vs ReFT vs other inference-time interventions?", "limitations": "- The authors address the following limitations:\n\t- The classes of models evaluated is limited.\n\t- Hyperparameter selection seems fairly complicated, and automating this selection will be valuable for future adoption of the methodology.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper is well-written.\n\t- The main ideas in the paper are clear and easy to understand.\n- Novel and impactful contribution.\n\t- The paper presents a new paradigm of learning modifications to representations, rather than learning modifications to weights. This new paradigm is more parameter-efficient, while appearing to have a similar level of expressivity. PEFT methods have been incredibly impactful, and I could see ReFT being similarly impactful.", "weaknesses": "- Lack of representation-editing-based baselines\n\t- The paper focuses on comparing ReFT to PEFT methods. Appendix B discusses existing representation-editing methods, and casts them in terms of the ReFT framework.\n\t- I think it would strengthen the paper to compare ReFT and DiReFT to these existing representation-editing methods. I am curious to understand the difference in performance of an intervention learned by gradient descent (e.g. ReFT) vs an intervention learned by contrastive pairs (e.g. activation addition, RepE).\n- Some unclear presentation\n\t- Line 102\n\t\t- What does \"the hidden representation created at row $i$ and column $k$\" refer to? As far as I can tell, this does not integrate with the previous notation defined in the second paragraph of Section 3. I assume this refers to layers and positions - if this is the case, then it would be clearer to say so explicitly.\n\t- Inconsistent variable names\n\t\t- I think the presentation would be clearer if variable names were used across sections.\n\t\t- Section 3.1 uses $\\mathbf{b}$ to represent a hidden state, whereas Section 3.2 uses $\\mathbf{b}$ to refer to a bias vector. I think this can be easily fixed by using $\\mathbf{h}$ to represent the hidden state in Section 3.1.\n\t\t- Section 3 uses $m$ to represent the number of layers, whereas Section 3.2 uses $m$ to refer to the length of the output sequence $\\mathbf{y}$. \n\t- Typo in Table 17 (?)\n\t\t- LLaMA-7B/DiReFT/AQuA: 221.3.\n\t\t- It might be worth double checking your numbers more generally if tables are not generated by code.\n- Other suggestions\n\t- Section 3.2\n\t\t- The paper could benefit from a motivation of the LoReFT expression, and how it was selected over the other expressions mentioned in Appendix E.\n\t- Define dropout more explicitly\n\t\t- What does dropout refer to in this case? My assumption is that with probability $p$, the intervention is not performed. But in the cases where the intervention is not performed (\"dropped out\"), what is being optimized?\n\t- Include baseline generations in Appendix I\n\t\t- Only LoReFT outputs are given in Appendix I - it's hard to interpret these generations without having baselines to compare them to. For example, even examining the difference between the baseline (no intervention) and LoReFT would be helpful, particularly for the long-form generations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for raising these great questions and providing helpful feedback!\n\n### RepE baseline.\n\nWe agree that gradient-free methods such as activation addition or RepE could be effective in steering models for tasks such as style transfer [4]. On the other hand, **we argue that it could be hard for these methods to steer models to achieve general finetuning goals** (e.g., steer a model to become a binary sentiment classifier or steer a model to answer multiple choice questions). These methods usually rely on statistical methods (e.g., PCA) to propose a set of static steering vectors based on a few-shot training dataset (i.e., usually with a handful of training examples). Mathematically, **these methods have the same expressivity as BitFit** [5] (only learning the bias term of LMs) yet BitFit is learned with SGD on large datasets. Meanwhile BitFit, although with much lower trainable parameter count (comparable to ReFT), usually underperforms compared to other PEFTs (see Table 4 on pg. 9). However, we agree that learning-free methods such as RepE have other unique advantages: \n\n- it could be very effective when the steering objective is more generic (e.g., style transfer for a chat-model) without heavy shifts in model’s behavior; \n\n- it could be very useful for LM personalization applications with very limited training source and do not require high precision.\n\n### Need to include the baseline generations.\n\nThanks for raising this issue. We agree. For all examples in Appendix I, we will add in the original mode outputs. **We include one example from GSM8K in our attached rebuttal pdf**.\n\n### The best expression of ReFT.\n\nIndeed, there is no clear winner for math reasoning benchmarks. Here are some additional pointers in terms of choosing different intervention functions:\n\n- **DiReFT removes the orthogonality constraint which improves the memory footprint** and trades compute efficiency (i.e., DiReFT trains faster) for a slight drop in performance. \n\n- **LoReFT generally converges quicker** and is less sensitive to learning rate and other hyperparameters partly due to its orthogonality constraint.\n\n- **The orthogonality constraint offers composability and interpretability**. Given the constraint, subspaces are orthogonal to each other (i.e., changing in one subspace should not affect others). As a result, we think this gives a nice property for us to compose LoReFT together at inference-time. We explore this a bit in Appendix G.1.. \n\n\n### Hyperparameter selection with ReFT.\n\n**We address this question in our general responses** by providing additional details on the hyperparameter searching process of ReFT!\n\n\n### PEFT vs ReFT vs other inference-time interventions.\n\nThanks for the question! It would be best for users to benchmark these methods for a specific domain. If we were allowed to guess, here are some pointers:\n\n- Inference-time intervention or activation addition does not require training. If the use case is not mission critical, one could use these methods for quick turnaround and showcase.\n\n- In general, since ReFT allows gradient descent, it should be more effective than non-training methods since it actively searches causal pathways to steer the model. ReFT also works with quick adaptation (e.g., n-shot training where n <= 10) as shown in G.2..\n\n- The composability of ReFT could be better (e.g., combining a set of directional steering together such as changing the tone, Language and length of the generating text.) compared with PEFT and ITI. We include a preliminary exploration of composability in Appendix G.2..\n\nAgain, we feel like this is an open-ended question. And introducing ReFT definitely pushes the community to think about the differences among these methods.\n\n### The classes of models evaluated is limited.\n\n**We addressed this question in our general responses** by applying ReFT to other models and tasks!\n\n\n### Unclear presentation.\n\nThanks for all these suggestions! We will address the following items in camera-ready.\n\n> Line 102: What does \"the hidden representation created at row and column\" refer to?\n\nSorry about the confusion here. Row and column map to the layer and position of the intervening residual stream. We will remove these two redundant notations $i$ and $k$, and rewrite the current sentence as: *“Let b be the hidden representation created at a specific residual stream located at a specific position and layer., …”*\n\n> Variable names were used across sections and typos.\n\nWe will make the following changes:\n\n- We will revisit our notations in Sec. 3 and keep a consistent format in our next revision. Please feel free to raise additional suggestions, and we will incorporate them.\n\n- We will replace our notation for $b$ in Eqn. 1 and related parts with $h_b$ to represent the base representation.\n\n- We will replace $m$ in Sec. 3.2 with $k$ to avoid overloading our symbols.\n\n- We also noticed this typo (\"LLaMA-7B/DiReFT/AQuA: 221.3\") after we submitted our draft. The correct entry should be 21.3. We will update the number, and check existing ones in our next revision. Our result tables (i.e., not the hyperparameter tables) are semi-automatically generated, but human error is still possible)\n\n\n> Other suggestions on writings.\n\nWe will make the following changes:\n\n- We will motivate how we come up with the current LoReFT formula better. We did cut a bit of text on this from our earlier draft due to length concerns. We will motivate LoReFT better from the interpretability works by bridging Sec 3.2 and Sec 3.2 together better in the next revision.\n\n- We will clarify the use of dropout in Eqn. (2) and (3).\n\n[4] Zou et. al., 2023, “Representation Engineering: A Top-Down Approach to AI Transparency”\n\n[5] Zaken et. al., 2022, “BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their useful comments. We remark on some of the shared questions here. All other questions are addressed in individual reviewer responses.

## Re: The significance of ReFT over LoRA and others.

Although in almost all other responses, we focus on comparing ReFT with other PEFTs by judging who achieves the SoTA performance, the most significant and surprising insight that we think ReFT brings to the table is **“minimally manipulating representations of LLMs can achieve strong finetuning performance”**. 

ReFT also offers **a new generic finetuning paradigm** under which more variants could emerge as the community starts to explore the “steering” power of LLMs’ hidden representations. We showed some of the explorations in our Appendix F-H, yet there are much more to study both as research directions as well as practical applications: How to automate hyperparameter search for intervention locations? How to design a better intervention function? Is the intervention latent space interpretable? How to serve finetuned models with ReFT at scale? How to allow quick personalization for LLMs with ReFT? etc..


## Re: Remarks on hyperparameter selection with ReFT.

Multiple reviewers ask about the hyperparameter selection process with ReFT.

We want to clarify that **ReFT actually has a similar amount of hyperparameters to LoRA and other PEFTs**. ReFT only introduces two new hyperparameters, the intervening layers and the intervening positions, while *removing LoRA hyperparameters such as alpha and applied components (i.e., which component to apply LoRAs on) for LoRA*. We will clarify this further in the camera-ready. We also agree that it would be best if we could automate this process in the future. For now, we provide a practical guide of choosing hyperparameters for ReFT in Appendix D.2..

## Re: ReFT with other LLMs, such as Mistral and Phi-3.

Some reviewers raise the question regarding whether ReFT works with other model types.

To give a quick preview of ReFT with Mistral and Phi-3, we reran our quick adaptation experiments in Appendix G.2 with Mistral-7B-instruct-v2.0 and Phi-3-small-8k-instruct, where we finetune these two models to kindly refuse all user queries with 5 training examples with a rank=4 LoReFT on the last token at layer 15. **We include qualitative results in our attached rebuttal pdf**.

Additionally, we finetune various models from the Mistral and Phi-3 families as well as LLaMA-3-8B-Instruct on our math benchmarks with DiReFT. We use the same set of hyperparameters mentioned in the paper without additional hyperparameter searching. 

| Name                     | % Params |  AQuA | GSM8K | MAWPS | SVAMP | Avg |
|--------------------------|----------|------:|------:|------:|------:|----------:|
| LLaMA-7B                 | 0.031%   | 21.3% | 24.1% | 74.5% | 42.7% |     40.6% |
| Mistral-7B-v1.0          | 0.031%   | 24.0% | 53.2% | 85.3% | 64.0% |     56.6% |
| LLaMA-3-8B-Instruct      | 0.031%   | 31.9% | 68.8% | 88.7% | 78.0% |     66.8% |
| Mistral-7B-Instruct-v0.2 | 0.031%   | 30.7% | 55.2% | 83.2% | 69.0% |     59.5% |
| Phi3-small-8k-Instruct   | 0.031%   | 37.0% | 79.8% | 92.0% | 84.1% |     **73.2%** |

These results definitely cannot provide the full picture without comparing against PEFTs. However, it is clear that ReFT (LoReFT and DiReFT) works for models from the Mistral and Phi-3 families. Bonus: Phi3 is obviously the best here as we expected since it’s actually tuned heavily on math and reasoning tasks!


## Re: Ways to improve ReFT on Math, and a showcase of ReFT with long-context summarization.

Multiple reviewers raise the question about what tasks ReFT is more suited for, and how to improve performance in benchmarks such as math reasoning.

We agree that LoReFT currently underperforms in math reasoning tasks compared with LoRA. One potential reason is that we only apply ReFT to the prompt tokens but not at generation tokens, which trades off the steering power for ReFT for lowered inference overhead. To verify this hypothesis, we ran additional experiments by **applying ReFT to selected prompt tokens as well as ALL decoding tokens** (LoReFT w/ decoding). Given our limited resources, we only ran it on our math reasoning benchmarks for a single seed with LLaMA-7B:

| Name                        | % Params |  AQuA | GSM8K | MAWPS | SVAMP | Avg |
|-----------------------------|----------|------:|------:|------:|------:|----------:|
| LoRA                        | 0.826%   | 18.9% | 37.5% | 79.0% | 52.1% |     46.9% |
| LoReFT                      | 0.031%   | 21.4% | 26.0% | 76.2% | 46.8% |     42.6% |
| LoReFT w/ decoding | 0.062%   | 20.1% | 31.2% | 80.7% | 54.8% |     46.7% |

This essentially closes the gap with LoRA. In addition, we note that this result is likely far from optimal: we ran this experiment without hyperparameter tuning, and we applied interventions on the attention output, not the residual stream. The additional parameters came solely from the decoding step interventions. 

**Reviewer d959 brought up another related point: how applicable is ReFT for tasks requiring long-form context?** To address this with limited resources, we ran a validation experiment by applying ReFT to do long-context summarizations where we finetuned LLaMA-7B with only 10 examples sampled from the WikiSum [2] (input > 1000 tokens) and a single rank=4 LoReFT intervention (# of trainable params=32,772). We compared our model with an publicly available SFT summarization model. **We include one example in our attached rebuttal pdf**. ReFT can certainly adapt our base LM to do long-context summarization! In the next revision, we might consider adding a full-fledged experiment.

[1] Meng et. al., 2022, “Locating and Editing Factual Associations in GPT”

[2] Cohen et. al., 2021, “WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation”

Meta-review: This paper proposes representation finetuning for parameter efficient fine-tuning. 

Strengths:
1. The proposed representation fine-tuning is novel and efficient. 
2. The paper demonstrates 15x to 65x improvement over LoRA. The experimental tasks are comprehensive. They also include three models. 

Weakness: 
1.  The cost of hyperparameter tuning is a concern. 

The remaining concerns are already addressed well in author's response. 

Overall, the paper proposes a novel method for parameter-efficient tuning and demonstrates its effectiveness across multiple tasks on three models. The improvement is impressive.
Decision: Accept (spotlight)