RE-Adapt: Reverse Engineered Adaptation of Large Language Models

## Abstract

Abstract We introduce RE-Adapt, an approach to fine-tuning large language models on new domains without degrading any pre-existing instruction-tuning. We reverse engineer an adapter which isolates what an instruction-tuned model has learned beyond its corresponding pretrained base model. Importantly, this requires no additional data or training. We can then fine-tune the base model on a new domain and readapt it to instruction following with the reverse engineered adapter. RE-Adapt and our low-rank variant LoRE-Adapt both outperform other methods of fine-tuning, across multiple popular LLMs and datasets, even when the models are used in conjunction with retrieval-augmented generation.

## 1 Introduction

Large Language Models (LLMs) require a significant investment to develop and train, requiring resources available to only a limited number of organizations. For instance, Meta’s Llama-3 family of models was trained using two custom-built compute clusters, each containing 24,000 high-end GPUs (Meta, ).
Parameter Efficient Fine Tuning (PEFT) enables resource efficient downstream customization of LLMs for new domains by adjusting a relatively small number of parameters while keeping the majority unchanged. However, an important distinction exists between the types of model used for further fine-tuning. It is common for LLM producers to release two versions of a model, one which is pretrained on a general task such as next-token prediction and an instruct version which is then continued trained on annotated data to learn how to follow instructions or respond to queries in a preferential manner (Touvron et al., ; Jiang et al., ; Almazrouei et al., ; Banks and Warkentin, ). The availability of both versions introduces a choice for organizations wanting to adapt a model to their custom task or domain. While an instruction-tuned model is generally more capable for popular tasks, the majority of data available for additional fine-tuning is unlabeled, lacking the annotations expected from instruct models. This poses a significant problem as annotation by the downstream organization can be too difficult, expensive, or error-prone (Fredriksson et al., ; Desmond et al., ). Additional fine-tuning can also degrade the performance of the instruction-tuned model outside of the new fine-tuning distribution (Kotha et al., ). On the other hand, pretrained models can be easily fine-tuned with unlabeled text but lack the additional capabilities of their instruct counterparts.

To address this dilemma, we seek the ability to fine-tune existing LLMs on unlabeled text while retaining the capabilities from pre-existing instruction-tuning. We draw inspiration from adapters, sets of learnable parameters added to an existing model for fine-tuning (Bapna and Firat, ; Houlsby et al., ). We make the key observation that the difference in weights between an instruction-tuned and corresponding pretrained model is effectively an adapter. Isolating the information learned from instruction-tuning into this Reverse Engineered (RE)-Adapter enables fine-tuning of the pretrained model, which can then be readapted with the instruction following capabilities ([Figure 1](#S1.F1)). In this work we:

- •
Explore the differences in parameters between pretrained and instruct models and their use as instruction adapters;
- •
Quantify RE-Adapt’s effectiveness to leverage unstructured knowledge for question answering in new domains under both context-free and retrieval-augmented scenarios;
- •
Introduce partial adaptation, a technique for scaling the strength of adapters for fine-grain control of knowledge priorities; and
- •
Demonstrate that RE-Adapters are effectively low-rank, showing that low-rank RE-Adapters (LoRE-Adapters) are capable of similar performance using up to 5x fewer parameters.

Figure: Figure 1: In RE-Adapt, an *instruction adapter* is isolated by differencing weights between instruct () and pretrained () versions of a model, which can be reapplied to the pretrained model after fine-tuning.
Refer to caption: /html/2405.15007/assets/images/readapt.png

## 2 Background

### 2.1 Adapters

Adapters (Bapna and Firat, ; Houlsby et al., ) have played an important role in the context of transfer learning for language models in recent years, particularly for fine-tuning pretrained models which are too large to fully train on commodity hardware. The concept introduced by Houlsby et al. () provides a lightweight alternative to full fine-tuning through the augmentation of models with small modular sets of trainable parameters. Adapters have been useful for enabling the use of pretrained models on new tasks (Pfeiffer et al., ; Karimi Mahabadi et al., ; Rücklé et al., ), new domains (Malik et al., ; Schopf et al., ; Diao et al., ), and adapting to multiple languages (Chronopoulou et al., ; Üstün et al., ; Parovic et al., ).

Low-Rank Adapters (LoRA) (Hu et al., ) are a particularly parameter efficient adaptation technique which adds a low-rank matrix to the weights of existing layers. Because the adapter is low-rank it can be represented as the product of two much smaller matrices, significantly lowering the number of trainable parameters. Weight-Decomposed Low-Rank Adaptation (DoRA) is an extension to LoRA with superior performance and similar efficiency (Liu et al., ). Liu et al. () achieve this by decomposing the pretrained weights into both magnitude and direction components, applying LoRA for directional fine-tuning only. Important to this work, adapters learned with either LoRA or DoRA can be represented as a single matrix which captures the information learned during fine-tuning. The pretrained model is then adapted by simply adding the new matrix to the existing weights. We leverage DoRA to fine-tune our models on a new domain, and take inspiration from the additive nature of these techniques to derive our reverse engineered adapters.

Several approaches have been developed which utilize the mixing or combination of adapters to benefit from diverse tasks or domains Pfeiffer et al. (); Rücklé et al. (); Wang et al. (); Chronopoulou et al. (); Fleshman et al. (); Zadouri et al. () or for parameter efficient federated learning (Babakniya et al., ; Sun et al., ). One method to categorize these approaches is by the mechanism used for combining the adapters. Either a weighted combination of adapters is applied to the base model (Chronopoulou et al., ; Fleshman et al., ; Babakniya et al., ; Sun et al., ) or another set of parameters are used to learn adapter interactions (Pfeiffer et al., ; Rücklé et al., ; Wang et al., ; Zadouri et al., ). We focus on the former, as we reframe instruction-tuned models as the summation of a pretrained model with an instruction adapter. We add new knowledge by combining domain-specific and instruction adapters via linear combination. As highlighted by Sun et al. (), separate adapters can be incompatible when averaged. Chronopoulou et al. () and Fleshman et al. () try to mitigate this by initializing adapters with the same random weights, and Sun et al. () by doing the same through a data driven approach. Neither option is applicable here, as we have no control over the instruction adapter. This motivates our new approach for partial adaptation which we introduce in [Section 3](#S3).

### 2.2 Instruct Models

Some of the most capable LLMs are instruct variants, pretrained on massive amounts of unannotated text and further trained on curated datasets with a combination of instruction-tuning (Mishra et al., ; Wei et al., ; Ouyang et al., ; Sanh et al., ) and Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., ; Stiennon et al., ). For example, Llama-3 was pretrained on 15T tokens and the instruct version continued training with a combination of supervised fine tuning (SFT), rejection sampling, proximal policy optimization (PPO), and direct preference optimization (DPO) (Meta, ). Open-source LLM producers generally release both the instruct versions as well as the pretrained models from which they were derived (Jiang et al., ; Almazrouei et al., ; Banks and Warkentin, ; Meta, ). Access to the pretrained LLM allows users to customize the model to a new task or domain while taking advantage of the large investment required for pretraining. Fine-tuning the instruct model directly is generally avoided due to catastrophic-forgetting, a phenomenon where models lose previous abilities with subsequent rounds of continued training (McCloskey and Cohen, ; Kotha et al., ). This is unfortunate, as few organizations have the resources to conduct fine-tuning at the scale of the original instruction-tuned models. In this work, we explore methods of fine-tuning LLMs which take advantage of both the pretraining and instruction-tuning of existing LLMs. We specifically design our approach to minimize forgetting while fine-tuning instruction-capable models with unlabeled text.

### 2.3 Model Arithmetic

Previous works have looked at the ability to arithmetically manipulate models to isolate certain behaviors (Ilharco et al., ; Mitchell et al., ). Ilharco et al. () constructed task vectors by differencing weights between a pretrained model and several corresponding models each fine-tuned for a particular task. They observed for their models that task vectors are almost orthogonal to each other, preventing interference and allowing combinations of the vectors for negating certain behaviors, improving multi-task performance, or performing well on new tasks via more complicated task analogies (Ilharco et al., ). We similarly solve for our reverse engineered adapter with a simple differencing, but using a single LLM fine-tuned for multi-task instruction-following. By effectively isolating instruction-tuning into an adapter, we allow further fine-tuning of pretrained models, maximizing knowledge acquisition before readapting their ability to follow instructions. We introduce an optional step for reducing the rank of our RE-Adapter, lowering memory requirements while maintaining or improving performance in some scenarios. Unlike individual task vectors, our multi-purpose RE-Adapters are not assumed to be orthogonal to new training domains. We introduce a technique for mitigating potential interference in [Section 3](#S3) by controlling the adaptation strength.

Mitchell et al. () developed an alternative approach for isolating pretraining knowledge from fine-tuned behaviors which they call emulated fine-tuning. Instead of differencing model weights, emulated fine-tuning considers the difference in outputs between pretrained and fine-tuned versions of a model. By combining this difference with the output of a larger pretrained model, Mitchell et al. () found that they could benefit from the additional pretraining knowledge while still solving the task of the smaller model. Their technique could be extended to meet our goal but requires the storage and forward pass of multiple models for inference. Our approach isolates knowledge and instruction-following into adapters, merged into a single model at no extra cost.

## 3 Partial Adaptation

We detail our main methods in [Section 4](#S4), but first we introduce a technique for controlling the strength of adaptation. Consider a model with weights $\mathbf{W}$ and an adapter $\mathbf{A}$ used to fine-tune the model on a new domain. Using additive adapters such as LoRA or DoRA, the combined weights:

$$ $\mathbf{\hat{W}}=\mathbf{W+A}$ (1) $$

are then used for inference (Hu et al., ; Liu et al., ). We make the observation that the resulting model assigns equal weight to the original parameters and the new adapter, which is generally trained with significantly less data than the original weights. This potentially leads to overfitting in the new domain and degradation of performance in the general setting. These issues compound in situations where multiple adapters are combined. Both Chronopoulou et al. () and Fleshman et al. () discuss complications arising from mixing adapters, especially if they were not initialized with the same values to encourage compatibility.

To mitigate these challenges we propose a technique for partial adaptation which introduces a post-hoc scaling factor for each fine-tuned adapter. Importantly, [Equation 1](#S3.E1) is still used during fine-tuning, but inference becomes:

$$ $\mathbf{\hat{W}}=\mathbf{W}+\lambda\mathbf{A}$ (2) $$

where $0\leq\lambda\leq 1$ is used to scale the strength of adaptation. In our experiments, we find that partial adaptation improves performance when using either single or multiple combined adapters.

## 4 Reverse Engineered Adaptation

Here we describe Reverse Engineered Adaptation (RE-Adapt), our approach to solve the challenge of updating an instruction-tuned model with unlabeled text without degrading the ability of the model to follow instructions. In [Section 5](#S5), we demonstrate the effectiveness of this approach for closed-book and retrieval-augmented question answering.

### 4.1 RE-Adapters

First consider two language models: $\mathbf{T_{\Phi}}$, which has been pretrained with parameters $\mathbf{\Phi}$; and $\mathbf{T_{\Theta}}$, having the same architecture as $\mathbf{T_{\Phi}}$ but with parameters $\mathbf{\Theta}$ updated from the pretrained parameters $\mathbf{\Phi}$ via instruction-tuning. Given these models, we can solve for the RE-Adapter parameters $\mathbf{\Delta}$ using:

$$ $\mathbf{\Delta=\Theta-\Phi}$ (3) $$

to isolate the information learned during instruction-tuning. Next, we augment the pretrained model $\mathbf{T_{\Phi}}$ with a learnable adapter $\mathbf{\Psi}$ and fit $\mathbf{T_{\Phi+\Psi}}$ on a new domain by only updating the adapter weights $\mathbf{\Psi}$. We refer to $\mathbf{\Psi}$ as the knowledge adapter. We utilize DoRA to fit $\mathbf{\Psi}$ in our experiments, but any fine-tuning approach is applicable. We construct our final model $\mathbf{T_{\Omega}}$ with parameters:

$$ $\mathbf{\Omega}=\mathbf{\Phi}+\alpha\mathbf{\Psi}+\beta\mathbf{\Delta}$ (4) $$

where $\alpha$ and $\beta$ are the scaling factors for the partial adaptation of $\mathbf{\Psi}$ and $\mathbf{\Delta}$ respectively. We find that scaling down the strength of the knowledge adapter $\mathbf{\Psi}$ and RE-Adapter $\mathbf{\Delta}$ with partial adaptation leads to better performance in instruction-based tasks related to the new domain while maintaining or slightly improving on the performance of the original instruction-tuned model out-of-domain.

### 4.2 LoRE-Adapters

Inspired by LoRA, we explore the intrinsic dimensionality of RE-Adapters and their ability to be represented by low-rank approximations. The Eckart-Young-Mirsky theorem establishes the truncated singular value decomposition (SVD) as the best low-rank approximation of matrices under the Frobenius norm (Eckart and Young, ). We compute the SVD of the RE-Adapter $\mathbf{\Delta}$ from [Equation 3](#S4.E3) which yields $\mathbf{\Delta}=\mathbf{U}\mathbf{S}\mathbf{V^{\intercal}}$ with the diagonal of $\mathbf{S}$ containing the singular values of $\mathbf{\Delta}$ sorted by magnitude, with $\mathbf{U}$ and $\mathbf{V}$ the corresponding left and right singular vectors. We then compute the percentage of variance explained by each dimension by squaring the singular values and re-normalizing the results to sum to 1. The cumulative explained variance $v$ at rank $k$ is then:

$$ $v_{k}=\Sigma_{i=0}^{k}\frac{\sigma_{i}^{2}}{\Sigma\sigma_{j}^{2}}$ (5) $$

where $\sigma_{i}$ is the $i$th singular value. We replicate this analysis for multiple modern LLMs and find that the majority of total variation in parameters can be represented at low-rank. For example, [Figure 2](#S4.F2) displays the cumulative explained variance plots for three layers from the RE-Adapter derived from Llama-3: we see more than half of the variance in these layers can be captured by a rank 128 approximation. This suggests the potential for a low-rank RE-Adapter (LoRE-Adapter).

Figure: Figure 2: Cumulative explained variance for singular values from Llama-3 RE-Adapt k_proj layers.

We can convert a RE-Adapter into a LoRE-Adapter using a similar approach as Sharma et al. () by representing each layer with its truncated SVD. In our case, we truncate to the rank that captures a total explained variance above a user-defined threshold $\tau$. [Figure 3](#S4.F3) shows the relationship between $\tau$ and the reduction in total parameters when using Llama-3 models to derive the adapter. As $\tau$ increases we maintain a higher percentage of the original parameters. We use LoRE-Adapters with $\tau=0.5$ for the experiments in this work and see similar or better performance when compared to RE-Adapt while using up to 5x less parameters. Like LoRA, the savings in memory is beneficial in cases where several LoRE-Adapters are swapped in and out of the same model.

## 5 Experiments

We quantify the effectiveness of RE-Adapt using question answering (QA), a task for which instruction-tuned models should perform significantly better than their pretrained counterparts. Specifically, we want to see if RE-Adapt is better than alternatives for adding knowledge from data not annotated with question-answer pairs. We would like the resulting model to do well answering questions about the new domains, while maintaining the level of performance of the original instruction-tuned model when answering unrelated questions.

### 5.1 Models

We replicate all experiments using the pretrained and instruct versions from the Gemma-7B (Banks and Warkentin, ), Llama-3-8B (Meta, ), and Mistral-7B (Jiang et al., ) family of LLMs using the HuggingFace API (Wolf et al., ). We utilize the parameter efficient fine-tuning library (Mangrulkar et al., ) for adding DoRA (Liu et al., ) knowledge adapters to each of these models. We perform all fine-tuning and inference with a single 80GB A100 GPU. We include hyper-parameters and other details of our fine-tuning in [Appendix A](#A1).

In [Section 5](#S5) we compare RE-Adapt and LoRE-Adapt with the pretrained and instruct models of each family, as well as pretrained and instruct models fine-tuned with DoRA on the new domains. We perform experiments for closed-book QA as well as QA with retrieval-augmented generation (RAG).

### 5.2 Data

Kotha et al. () showed that fine-tuning degrades performance outside of the fine-tuning distribution. We hypothesize that our approach mitigates this issue by isolating existing instruction-tuning from additional fine-tuning. We test this by measuring the changes in question-answering performance when various fine-tuning strategies are used to update models with unlabeled data. An optimal approach would benefit from the new knowledge when asked related questions, without losing the ability to answer unrelated questions.

We explore this hypothesis by fine-tuning models in two different settings. We use English WMT News Crawl (Kocmi et al., ) articles published in the year 2020 as our first fine-tuning distribution.(^1^11Available at [https://data.statmt.org/news-crawl/README](https://data.statmt.org/news-crawl/README) under CC0 license.) These articles provide non-annotated information which we capture through DoRA adapters trained for next-token-prediction. We evaluate how well this knowledge is acquired by using the resulting models to answer related questions from the StreamingQA dataset (Liška et al., ), which contains 21,681 QA pairs derived from our subset of articles.(^2^22Available at [https://github.com/google-deepmind/streamingqa](https://github.com/google-deepmind/streamingqa) under CC-BY 4.0 license.)

We use the evidence passages from RetrievalQA (Zhang et al., ) as our second fine-tuning distribution and measure performance on the corresponding questions from the same dataset.(^3^33Available at [https://huggingface.co/datasets/zihanz/RetrievalQA](https://huggingface.co/datasets/zihanz/RetrievalQA) under MIT license.) Zhang et al. () curated the dataset by compiling the subset of questions from five other QA benchmarks for which GPT-4 (OpenAI et al., ) is unable to answer without access to external knowledge. The questions were selected with the goal of having the corresponding knowledge absent from current LLMs, making this dataset especially challenging in the closed-book setting.

To measure any performance degradation from fine-tuning, we also evaluate our models using a short-answer subset of the Natural Questions dataset (Kwiatkowski et al., ) which is unrelated to either fine-tuning distribution.(^4^44Available at [https://huggingface.co/datasets/natural_questions](https://huggingface.co/datasets/natural_questions) under CC-BY-SA 3.0 license.) We use these questions to measure performance before and after fine-tuning our models on the other domains. We would like our approach to result in improved performance when answering questions related to the fine-tuning data without a reduction in performance on the unrelated Natural Questions [Figure 4](#S5.F4).

Figure: Figure 4: RE-Adapt enables the addition of new knowledge to an instruction-tuned model, without degrading capabilities on knowledge from pretraining.
Refer to caption: /html/2405.15007/assets/images/questions.png

### 5.3 Evaluation

We observe that instruction-tuned models will generally answer questions in long-form, often repeating the question and providing additional helpful context. An example of this behavior is shown in [Table 1](#S5.T1) where the model is asked for the number of episodes in a popular tv series. Here we see the reference answer is 291, which Llama-3 gets correct, but with a response containing full sentences and additional information to clarify its position.

**Table 1: Example from Natural Questions with a truncated response. Llama-3’s full response includes more details per country.**
| Question | how many episodes are there in dragon ball z? |
| --- | --- |
| Answer | 291 |
| Llama-3 | There are a total of 291 episodes in the original Japanese version of Dragon Ball Z. However, the episode count can vary depending on the version and the country. |

Popular QA metrics such as Rouge-L (Lin, ) or exact match would penalize Llama-3 for not being precise. To alleviate this concern we evaluate using Rouge-L’s recall, which is the percentage of the longest common sub-sequence of the reference answer found in the model’s response. We additionally measure a version of exact match which looks for the exact reference answer anywhere in the response. In both cases, if the reference answer is in the response the score will be 1. If the answer is partially correct then exact match will be 0, but Rouge-L will provide partial credit.

Figure: Figure 5: StreamingQA performance as RE-Adapter is added to fine-tuned Llama-3 model with varying strengths.

### 5.4 Closed-Book QA

In our first experiment we conduct QA evaluation in a closed-book setting where the models must provide an answer given nothing but the question. We explore how RE-Adapt behaves in this setting with varying partial adaptation scaling factors. [Figure 5](#S5.F5) shows the QA performance of LLama-3 using a fixed-factor of 1.0 for the knowledge-adapter with varying scaling factors for the RE-Adapter. We find that partial adaptation with a factor of 0.5 for both the knowledge adapter and instruction adapter provides robust results across models and datasets when using both RE-Adapt and LoRE-Adapt.

We use an explained variance threshold $\tau=0.5$ for our LoRE-Adapters. The resulting percentage of original parameters for each model are: Llama-3 (19.2%), Gemma (30.2%), and Mistral (27.1%).

The closed-book performance of all models across datasets is shown in [Table 2](#S5.T2). Both RE-Adapt and LoRE-Adapt outperform the pretrained and instruction-tuned models on StreamingQA and RetrievalQA, even when those models are fine-tuned on the corresponding News Crawl or RetrievalQA passages. As expected, the pretrained models perform worse, although fine-tuning on the unlabeled data does improve the QA ability of both pretrained and instruct models in the domain used for adaptation. These in-domain results indicate that our approach is superior for knowledge acquisition. Next we will discuss the impact fine-tuning has on general QA performance by looking at results on the out of domain Natural Questions dataset.

**Table 2: Closed-book QA performance. The QA dataset being evaluated is listed above the dataset used for fine-tuning DoRA adapters. R-L indicates Rouge-L and EM indicates exact match.**
|  |  | StreamingQA | RetrievalQA | Natural Questions |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | News Crawl | RQA Passages | News Crawl | RQA Passages |  |  |  |  |
|  | Model | R-L | EM | R-L | EM | R-L | EM | R-L | EM |
|  | Pretrained | 9 | 0 | 1 | 0 | 10 | 3 | 10 | 3 |
|  | Pretrained + DoRA | 12 | 3 | 3 | 2 | 10 | 4 | 14 | 7 |
| Llama-3 | Instruct | 33 | 19 | 5 | 3 | 46 | 34 | 46 | 34 |
|  | Instruct + DoRA | 38 | 22 | 7 | 4 | 39 | 22 | 37 | 27 |
|  | LoRE-Adapt (Ours) | 46 | 26 | 10 | 6 | 51 | 34 | 53 | 35 |
|  | RE-Adapt (Ours) | 46 | 27 | 9 | 6 | 52 | 34 | 54 | 36 |
|  | Pretrained | 11 | 2 | 1 | 0 | 10 | 3 | 10 | 3 |
|  | Pretrained + DoRA | 19 | 4 | 1 | 0 | 7 | 1 | 10 | 2 |
| Gemma | Instruct | 20 | 9 | 2 | 1 | 26 | 12 | 26 | 12 |
|  | Instruct + DoRA | 31 | 18 | 5 | 3 | 26 | 12 | 28 | 14 |
|  | LoRE-Adapt (Ours) | 31 | 15 | 7 | 4 | 24 | 14 | 30 | 20 |
|  | RE-Adapt (Ours) | 33 | 18 | 6 | 4 | 26 | 17 | 28 | 17 |
|  | Pretrained | 17 | 5 | 2 | 0 | 14 | 5 | 14 | 5 |
|  | Pretrained + DoRA | 22 | 8 | 2 | 1 | 14 | 5 | 15 | 6 |
| Mistral | Instruct | 29 | 16 | 4 | 2 | 33 | 22 | 33 | 22 |
|  | Instruct + DoRA | 36 | 21 | 6 | 5 | 27 | 13 | 33 | 18 |
|  | LoRE-Adapt (Ours) | 39 | 24 | 7 | 5 | 39 | 24 | 42 | 28 |
|  | RE-Adapt (Ours) | 37 | 22 | 6 | 4 | 37 | 23 | 41 | 27 |

The closed-book results for the Natural Questions dataset on the right side of [Table 2](#S5.T2) demonstrate the issues with fine-tuning instruct models with non-annotated data, resulting in models that perform worse in their original setting. While fine-tuning on News Crawl or Retrieval QA passages improved the instruct models on the corresponding QA datasets, the majority of models saw a decrease in performance on Natural Questions. RE-Adapt alleviates this problem by using the data from the new domain to only fine-tune the pretrained model, keeping the instruction-tuning intact. Using our approach,

Figure: Figure 6: Natural Questions performance as the RE-Adapter is added to pretrained Llama-3 with varying strengths.

the resulting models performed significantly better on the fine-tuning distribution without a performance degradation out-of-domain. In fact, RE-Adapt and LoRE-Adapt performed better than the original instruction-tuned models out-of-domain. This improvement indicates that instruction-tuning likely degrades knowledge from pretraining; an issue our approach mitigates through partial adaptation. We confirm this suspicion by applying RE-Adapt to Llama-3 without any additional fine-tuning. This allows us to produce instruct models with instruction-tuning strengths ranging from 0 (the pretrained model) to 1 (the instruct model). We find that we can improve existing instruct models with zero additional training by simply scaling down the strength of instruction-tuning [Figure 6](#S5.F6). Combined, these results demonstrate the effectiveness of RE-Adapt for knowledge acquisition with minimal forgetting.

### 5.5 RE-Adapt with RAG

Retrieval-augmented generation (RAG) Lewis et al. () is a popular alternative for utilizing new data with instruction-tuned models. Instead of altering the model directly, RAG maintains a database of all text and retrieves relevant documents to include in the prompt as context. This begs the question, is RE-Adapt still beneficial if the new data is already available via RAG?

**Table 3: QA performance when using RAG with BM25 and (Oracle) retrievers.**
|  |  | StreamingQA | RetrievalQA |  |  |
| --- | --- | --- | --- | --- | --- |
|  | Model | Rouge-L | Exact Match | Rouge-L | Exact Match |
|  | Pretrained | 38 (59) | 27 (48) | 13 (16) | 11 (14) |
| Llama-3 | Instruct | 55 (57) | 54 (58) | 14 (30) | 16 (32) |
|  | LoRE-Adapt (Ours) | 69 (74) | 58 (64) | 24 (37) | 21 (31) |
|  | RE-Adapt (Ours) | 68 (71) | 59 (64) | 19 (36) | 18 (30) |
|  | Pretrained | 39 (41) | 28 (29) | 4 (26) | 3 (23) |
| Gemma | Instruct | 52 (56) | 48 (53) | 17 (24) | 16 (24) |
|  | LoRE-Adapt (Ours) | 46 (50) | 49 (55) | 12 (17) | 18 (27) |
|  | RE-Adapt (Ours) | 50 (55) | 50 (56) | 21 (30) | 18 (28) |
|  | Pretrained | 33 (38) | 26 (30) | 18 (12) | 16 (10) |
| Mistral | Instruct | 49 (52) | 50 (56) | 14 (23) | 19 (28) |
|  | LoRE-Adapt (Ours) | 54 (58) | 55 (61) | 18 (23) | 20 (28) |
|  | RE-Adapt (Ours) | 55 (58) | 55 (60) | 15 (24) | 20 (29) |

To answer this question, we replicate our experiments on StreamingQA and RetrievalQA, using a BM-25 index (Robertson and Zaragoza, ) to retrieve the most relevant passage to be used as context for the models. In practice, RAG setups can retrieve more than one document, but each question in our datasets can be answered from a single passage, and therefore we avoid known issues which RAG can face when too much context is provided to the models (Liu et al., ; Barnett et al., ; Gao et al., ). Because a poor retriever could bias results in our favor, we also repeat the experiment using an oracle retriever. Instead of performing a heuristic search, the oracle retriever directly selects the passages capable of answering the question as context. While this idealized retriever is unrealistic in practice, it allows us to further isolate the benefit of combining RAG with fine-tuning by eliminating any impact from imperfect retrieval.

The RAG results are shown in [Table 3](#S5.T3). Again we see significant improvements when using RE-Adapt and LoRE-Adapt even in this RAG setting where the model should already have access to the relevant information needed to answer the questions. The BM-25 search retrieved the correct document with approximately 73% accuracy across models. Using RE-Adapt to incorporate the data outside of RAG alleviates the shortcomings of the retriever. However, RE-Adapt also improved results when using the oracle, suggesting that adding domain knowledge with an adapter also reduces incorrect interpretations of the context retrieved via RAG.

## 6 Discussion

Combined, our results demonstrate RE-Adapt’s effectiveness at incorporating new knowledge into existing LLMs without having to discard previous instruction-tuning. Our methods increase QA performance by a greater amount when compared to traditional fine-tuning strategies. We also find that our approach improves RAG based systems, even in the most optimistic case of perfect retrieval. Our improved results outside of the fine-tuning distribution suggest that we can recover additional pretraining knowledge by reducing the strength of instruction-tuning through partial adaptation. Importantly, an improvement is seen without any additional fine-tuning of the underlying models. These results encourage additional future research into controlling the competing priorities of knowledge acquisition and general problem solving capability.

#### Limitations.

The limitations of our work are two-fold. First, instruction-tuned models perform better than pretrained models on a wide variety of tasks, but we limit our evaluations to the single task of question answering due to the large number of ablations required by our experiments and limited compute resources available. Second, we include the prompts used for instructing the models for QA in [Appendix B](#A2) but note that different prompting strategies could alter our results. We mitigate introducing bias in prompting by not optimizing the prompts for any particular method.

#### Societal Impact.

We are unaware of any negative societal impacts likely to be caused by our contributions. We further amortize the costs of building open-source LLMs by enabling others to leverage existing instruction-tuning, hopefully decreasing the future energy consumption and environmental impacts caused by LLM customization.

## 7 Conclusion

In this work, we presented RE-Adapt, a new approach for adding knowledge to existing instruction-tuned models. RE-Adapt isolates the differences between an instruction-tuned model and its pretrained counterpart in order to preserve instruction-following capabilities during additional fine-tuning on unlabeled data. We demonstrated that our approach outperforms fine-tuning pretrained or instruction-tuned models directly, which otherwise causes performance to degrade outside of the new fine-tuning domain. Our findings are robust across three state of the art large language models.

We achieved our best performance using partial adaptation, a new method for controlling the strength of adaptation at inference time when using single or combined adapters. We found that partially adapting instruction-tuned models improved QA performance without any additional fine-tuning.

We also analyzed the spectrum of RE-Adapt’s weight matrices, constructing a low-rank variant of our approach, LoRE-Adapt, which captures the majority of variation in the instruction-tuning weights at a much lower rank. LoRE-Adapt performed similarly to RE-Adapt with occasional out-performance, while decreasing the number of parameters by as much as 5x in our experiments.

Finally, we demonstrated that RE-Adapt improves performance even when the information required to answer questions is available via retrieval augmented generation. Combined, our results suggest RE-Adapt is an effective approach for infusing new knowledge into already instruction-tuned LLMs.

## Appendix A Fine-Tuning Details

We include the settings for training our DoRA adapters in [Table 4](#A1.T4). All adapters were trained on a single NVIDIA A100 GPU with 80GB of memory.

**Table 4: Training details.**
| Setting | Value |
| --- | --- |
| LoRA Layers | all-linear |
| LoRA Rank | 64 |
| LoRA Alpha | 128 |
| LoRA Dropout | 0.05 |
| DoRA | True |
| Batch Size | 20 |
| Epochs News Crawl | 10 |
| Epochs RetrievalQA | 3 |
| Optimizer | AdamW |
| Learning Rate | 0.0002 |
| Schedule | Linear |

## Appendix B Prompts Used

Each LLM can use unique prompting roles and tokens when constructing prompts. We utilize the huggingface tokenizers library to ensure our prompts follow the correct template.

The Llama-3 instruct models use a combination of system, user, and assistant roles while Gemma and Mistral only use user and assistant. Our prompts where constructed using the following formats:
Llama-3 Closed-Book QA
system: Answer the following question.
user: <question>?
Llama-3 RAG
system: Answer the following question given this context: <context>.
user: <question>?
Gemma and Mistral Closed-Book QA
user: <question>?
Gemma and Mistral RAG
user: Answer the following question given this context: <context>\nQuestion: <question>?

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The analysis concerning “cumulative explained variance” is based on three layers of the model. However, it is unclear how these three layers were selected and why only three layers were chosen for analysis.
2. The cumulative explained variance for singular values is calculated for the k_proj layers. What about the other layers? Did they exhibit similar patterns?
soundness: 2
strengths: 1. The paper is well-motivated, highlighting the challenge of catastrophic forgetting when adapting current LLMs to new domains or distributions. The RE-Adapt method provides a practical solution to this issue.
2. The LoRE-Adapt method, based on LoRA, demonstrates lower memory usage compared to RE-Adapt, which is significant for training efficiency, building on the inherent advantages of low-rank approximation.
3. The proposed methods are compatible with retrieval-augmented generation, showcasing their potential for integration with training-free approaches.
summary: This paper presents a novel fine-tuning method for large language models (LLMs) aimed at addressing the issue of catastrophic forgetting during adaptation to new domains. The proposed method, named RE-Adapt, integrates an instruction-following adapter with a domain-specific adapter. The instruction-following adapter is derived by comparing the foundation LLM with its instruction-tuned variant, while the domain-specific adapter is fine-tuned using DoRA. These adapters are combined during inference through weighted summation. Additionally, a variant called LoRE-Adapt substitutes the adapter with LoRA.

Experimental results on the StreamingQA and RetrievalQA benchmarks indicate that RE-Adapt outperforms baseline methods in in-domain adaptation performance. Furthermore, the proposed methods effectively mitigate the catastrophic forgetting problem compared to traditional fine-tuning pipelines (w/ DoRA in this paper), as demonstrated by out-of-domain results on the Natural Questions benchmark.
weaknesses: 1. The applicability of the proposed method may be limited, as it relies on an instruction-tuned version of the original foundation model. While many open-source LLMs offer both versions, it remains unclear how RE-Adapt would perform if only the foundation model is available and we attempt to tune a weaker instruction-following model.
2. On line 201, the authors state that “any fine-tuning approach is applicable”, which is misleading. Directly employing a full-parameter fine-tuning method would prevent the combination of the tuned model's weights ($\Psi$) with the original weights ($\phi$) as described in Equation 4.
3. There is a lack of a LoRA baseline. Since LoRE-Adapt is a variant based on LoRA, it is essential to include a baseline that fine-tunes a pre-trained or instruction-tuned model using LoRA, in addition to the +DoRA baseline.
4.  The RAG setups in RetrievalQA should pose more significant challenges. Currently, the BM25 retriever searches only within in-domain data, while a more realistic scenario would involve a retrieval datastore created from diverse external data.
5. The proposed method combines different adapters and weights in a linear fashion. A previous study [1] systematically investigates the arithmetic operations of parameter-efficient tuning modules, and including this work in the literature review would be beneficial. 
6. Typos:
- L028: Parameter Efficient Fine Tuning -> Parameter-Efficient Fine-Tuning 
- L114: data driven -> data-driven
- L409: (RAG) Lewis et al. (2020) -> (RAG; Lewis et al., 2020)

References

[1] Zhang, J., Liu, J., & He, J. (2023). Composing parameter-efficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36, 12589-12610.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. How are the instruction-tuned models obtained? Are they officially released instruction-tuned models from each LLM family, or are they fine-tuned on downstream tasks? If the models were officially released, a more rigorous evaluation on established benchmarks might be needed to confirm if the instruction-following performance indeed degrades after the adaptation.
2. In Section 4.1, line 207, the authors mention conducting experiments on the sensitivity of scaling factors. However, no results for these experiments are provided in the results section.
soundness: 2
strengths: -  The paper addresses the important challenge of adding new knowledges to LLMs while preserving their instruction-following capabilities.
-  The method is well-grounded and aligns with existing literature on LLM arithmetic, leveraging the modular nature of model weights for adaptation.
summary: This paper introduces a reverse-engineering adapter that enables adapting instruction-finetuned large language models (LLMs) to new domains while preserving their instruction-following capabilities. The authors model an instruction-tuned LLM as a combination of a pretrained model and an instruction adapter, effectively viewing the adapter as the weight difference between the pretrained and instruction-tuned models. A post-hoc scaling factor $𝜆$ is introduced to control the strength of the domain adaptation, with two variations presented—one with a low-rank adapter and one without. Experimental results on 7B models from three distinct LLM families, evaluated on closed and open-book question-answering tasks, show that the proposed approach outperforms traditional fine-tuning and retrieval-augmented generation (RAG) methods.
weaknesses: - The experiments focus exclusively on question-answering (QA) tasks and do not assess whether the instruction-following capabilities of instruction-tuned models are maintained post-adaptation. Additionaly experiments on common benchmarks for instruction-tuned models are expected.
- A comparative analysis between the original instruction adapter obtained via reversed engineering and the instruction adapter after further domain-specific pretraining and instruction fine-tuning could help validate the authors’ hypothesis about the transferability of the instruction adapter.
- There is insufficient analysis of the chosen hyperparameters, particularly the sensitivity of the scaling factor $λ$, which could impact the generalizability of the approach.
- Experiments on different model sizes (<7B and >7B) would provide a more comprehensive understanding of whether the proposed method’s benefits hold at various scales.
- Tables 2 and 3 should report results to two decimal places. Currently, certain models appear to have identical performance (e.g., Gemma Instruct, Gemma Instruct + DORA, and Gemma Re-Adapt on the Natural Questions dataset all report a score of 26).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - The method proposed by the authors primarily retains instruction-following abilities while preserving fine-tuning knowledge, so I would like to see how much impact this method has on instruction-following capabilities.
- The process of singular value decomposition (SVD) for delta does not clarify which parameter matrices were decomposed, making Figure 2 difficult to understand. Is it the entire parameter matrix?
- Why does an explained variance threshold of τ = 0.5 yield the best performance? The authors do not provide a reasonable explanation. Increasing the threshold leads to an increase in the number of parameters, but why does performance also decrease?
- The significant performance improvements brought by the proposed LoRE-ADAPT and RE-ADAPT are somewhat surprising, especially considering the additive process in Equation 4. Using the pretrained model + DoRA to train an adapter and then adding delta weights should not yield a performance significantly better than the pretrained model + DoRA alone. I hope the authors can provide more experimental results to demonstrate the effectiveness of LoRE-ADAPT and RE-ADAPT.
soundness: 1
strengths: - The proposed method is straightforward, simple, and clear, making it easy for readers to understand. Additionally, the authors’ writing is fluent and well-executed.
- The authors have identified a problem that, while not highly focused on, is still of practical significance: the mismatch between instruction models and domain-specific downstream data.
summary: This paper proposes RE-Adapt and its variant LoRE-Adapt as methods for enhancing downstream tasks in specific domains by injecting domain knowledge into instruction models. Firstly, the authors extract the knowledge injected into the model during the SFT and RLHF stages by recording the parameter changes
of the instruction model compared to the pretrain model, and use this as an adapter. They then fine-tune a new adapter on the downstream tasks based on the pretrain model, and merge the two adapters. The authors have validated the effectiveness of their proposed method on several downstream tasks.
weaknesses: - Lack of Novelty: The methods proposed by the authors, namely RE Adapt and LoRE-Adapt, are based on well-established concepts such as adding independent adapters to the model, utilizing Singular Value De composition (SVD) to extract key information from matrices, and com bining adapters through weighted averaging. These methods were widely accepted and recognized by the academic community as early as 2021 with the introduction of LoRA. In the paper, the authors do not present any novel insights or groundbreaking approaches to solving the problem, which makes the paper seem more like an exploration of existing methods on certain datasets rather than a demonstration of a new way to address challenges.
- Lacking Comparison Experiments: The experimental section lacks critical comparisons. The issue raised by the authors is the format mis match between downstream task data and the instruction model’s for mat. In fact, the authors could have easily constructed versions of exist ing datasets that match the instruction format through a text pipeline, and then fine-tuned the instruction model directly through various meth ods. This approach is more straightforward compared to the proposed method. The authors did not consider comparing their method with the restructured datasets, which I believe fails to demonstrate the practical
significance of their method.
- Lacking Discussion on Hyperparameters: As an important variant of the proposed method, LoRE-Adapt lacks experimental characterization of the impact of choosing different downsampling ratios 𝜏 on model per formance and inference cost.
- The paper lacks theoretical insights. While the experiments demonstrate a significant performance improvement with RE-ADAPT, there is insufficient theoretical support for the reasons behind this enhancement.
- Equation (4) introduces α and β as scaling factors for adjusting the strengths of the respective adapters, but there is a lack of experimental analysis discussing the balance between the knowledge adapter Ψ and the RE-Adapter Δ.
- LORE-ADAPT is a very interesting component, and many of its properties are quite intriguing. However, in terms of addressing the problem, both the principles and experimental results seem to be not that different from the RE-ADAPT method.
﻿
﻿Others
- The authors’ estimation of model parameters seems to be incorrect. In Figure 3, when 𝜏 = 1, the parameter size reaches 100%. In fact, due to the structural characteristics of LoRA, when 𝜏 = 1, the parameter size should be 200%, i.e., retaining two matrices of the same size as the original matrix, representing the A and B matrices in LoRA.
- From the experiments, LORE-ADAPT, the low-rank variant of RE-ADAPT, shows similar performance while reducing parameters by 4 to 5 times, and even outperforms RE-ADAPT in certain models (e.g., Mistral-7B). However, the paper lacks explanation and discussion on this observation. It would be beneficial for the authors to provide more details regarding LORE-ADAPT.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. How long does SVD decomposition procedure take, compared to the DoRA tuning process (execution time)?
2. Line 244: “we truncate each layer to the rank that captures a total explained variance above a user-defined threshold $\tau$”. Did I understand correctly that the DoRA rank will differ between layers? 

Additional comments

Experiments:

* Line 374: “RE-ADAPT and LORE-ADAPT performed better than the original instruction-tuned models out-of-domain. This improvement indicates that instruction-tuning likely degrades knowledge from pretraining”. The second sentence does not necessarily follow from the first one and would require additional experimental validation. For example, this improvement may simply come from a regularization effect of weight averaging [7]. The same regularization effect could be the reason for the improvement in Figure 6.
* Line 434: “RE-ADAPT also improved results when using the oracle, suggesting that adding domain knowledge with an adapter also reduces incorrect interpretations of the context retrieved via RAG.” Again the second part of the sentence does not necessarily follow from the first part and would require additional experimental validation. For example, the improvement may potentially come from scaling down the contribution of Re-adapter, same effect as in Figure 6. 

[7] Pavel Izmailov et al. Averaging Weights Leads to Wider Optima and Better Generalization. UAI 2018

Text:

* Table 2: there are several cases when the baseline has the same performance as the proposed method, but is not highlighted bold. Example: Gemma / Instruct + DoRA / second column -> 18 should also be bold; there are several other cases.
* Unclear claim in line 37: “lacking the annotations expected from instruct models. ” 
* Line 92: “Because the adapter is low-rank it can be represented as the product of two much smaller matrices, significantly lowering the number of trainable parameters.”: I think the premise and the conclusion should be reversed in this sentence, e.g. adapter is low-rank because it can be represented …
* Line 103: add \citet to put brackets around citations
* Line 261 and further in the text: “when answering unrelated questions.”: unclear term “unrelated questions”
* Line 284: remove “In section 5” as this is inside section 5.
* Line 300: “derived from our subset of articles”: what is “our subset”?
* Lines 295-300: I did not understand the connection between the StreamingQA dataset and WMT news crawl.
* Figure 5: I would suggest including similar plots for a fixed factor of 0.5 for a knowledge adapter, or varying a scalar factor for the knowledge adapter with the fixed scalar factor for Re-adapter.
soundness: 2
strengths: * A highly-relevant research direction of adapting LLMs to new domains
* An interesting idea of constructing domain-adapted instructed models by task arithmetic. The simplicity of the method makes it easily applicable in practice to any domain of interest.
* The paper is well-written and easy to follow
* Well-structured and comprehensive Related Work
* The method is validated for three different LLMs
summary: The paper investigates strategies for continual adaptation of instruction-tuned LLMs to new domains and proposes RE-Adapt, reverse-engineered adaptation of LLMs. The approach takes inspiration in weight arithmetic and produces a domain-adapted instructed model as a linear combination of a pretrained LLM, a Reverse-engineered adapter (difference between the weights of an instructed and the pretrained LLM), and a Knowledge adapter (weight change resulting from tuning a pretrained LLM on a new domain). The paper also suggests a partial adaptation strategy, i.e. tuning a scalar multiplicative weight for each term in the weight arithmetic, and LoRE-Adapt, a modification of Re-Adapt which uses a low-rank approximation of the Reverse-engineered adapter obtained through singular value decomposition. The proposed method is validated on two question answering datasets, while monitoring the performance on Natural Questions as a generic dataset.
weaknesses: 1. Insufficient empirical validation of the proposed approach. 
     - The paper conducts experiments on a single task (question answering), with two adaptation datasets (StreamingQA and RetrievalQA) and one general QA dataset (Natural Questions). Since the main goal of the proposed approach is to build a model capable of following instructions, a validation of the final model on a broader set of tasks is required, e.g. including open-ended instruction following, summarization, or multiple-choice QA. This concerns both instruction following capabilities on new domains and initial model capabilities, i.e. validation solely on NQ is not enough to demonstrate the preservation of initial instructed LLM capabilities. It is possible to create multi-task evaluation beds e.g. for a biomedical domain with Pubmed as domain data [1-4]. 
     -  I am not sure if RetrievalQA is a good candidate domain for the paper’s experiments, since performance on this dataset without RAG is very low (and improvements by the proposed method  are also small, i.e. 1-2 % exact match), and because evidence passages from RetrievalQA do not necessarily form a well-defined domain.
    - Tables 2 and 3 miss a baseline $\Phi + 0.5 \Delta$, to disambiguate the influence of downscaling the Re-adapter (shown to be effective in Figure 6) and of learning the domain knowledge in the domain adapter. For example, in Table 3 the domain knowledge comes from using RAG, so this baseline can potentially perform as good as Re-Adapt / LoRE-Adapt.
    - It would be also an advantage if the proposed approach was validated for more than one adaptation approach, e.g. simple LoRA, as it is widely used in practice.
2. The partial adaptation approach is not novel as the same technique of applying a scalar factor, tuned on the validation set, was used e.g. in (Inarco, 2023).
3. The experiments are conducted with a fixed learning rate, which weakens a simple baseline of finetuning an instructed LLM on the domain data. As shown in [5, 6], simply reducing an LR may help to avoid catastrophic forgetting when tuning LLMs.  According to Table 4 in Appendix, the paper uses an LR of 2e-4, which is too high to preserve initial model capabilities according to [6].
4. Deeper details on the experimental setup could be provided, such as the used decoding strategy at inference, metric details in case some preprocessing was used (e.g. lower casing candidate and ground truth answer), preprocessing details for the datastore in RAG (e.g. how passages for retrieval are constructed), or which parameters are used in bm25.

[1] Anastasios Nentidis et al. Overview of BioASQ 2023: The eleventh BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. CLEF 2023. (Factoid / list / yes-no questions and summarization on the biomedical domain.)

[2] Xinlu Zhang et al. AlpaCare: Instruction-tuned Large Language Models for Medical Application. arxiv 2023. (This paper collects a MedInstruct-test dataset which can be used for evaluating open-ended generation on the biomedical domain.)

[3] Qiao Jin et al. PubMedQA: A Dataset for Biomedical Research Question Answering. EMNLP 2019

[4] Hongjian Zhou et al. A Survey of Large Language Models in Medicine: Progress, Application, and Challenge. arxiv 2023

[5] Nadezhda Chirkova et al. Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks. NAACL 2024

[6] Nadezhda Chirkova et al.  Zero-shot cross-lingual transfer in instruction tuning of large language models. INLG 2024