Neutral residues: revisiting adapters for model extension
Authors: Franck SIGNE TALLA
Edouard Grave
Herve Jegou
Venue: ICML 2025 poster (2025)
Keywords: LLM
Adapters
Domain Adaptation
Catastrophic Forgetting
Primary area: deep_learning->large_language_models
TL;DR: We add new capabilities to a pre-trained large language model, without forgetting its original knowledge.

Abstract: We address the problem of extending a pre-trained large language model to a new domain that was not seen during training. Standard techniques, such as fine-tuning or low-rank adaptation (LoRA) are successful at domain adaptation, but do not formally add capacity to the model. This often leads to a trade-off, between performing well on the new domain vs. degrading performance on the original domain.
Here, we propose to revisit and improve adapters to extend LLMs. Our paper analyzes this extension problem from three angles: data, architecture and training procedure, which are advantageously considered jointly. The resulting method, called neutral residues, modifies adapters in a way that leads to each new residual block to output near-zeros on the original domain. This solution leads to strong results when adapting a state-of-the-art model originally trained on English to a new language. Neutral residues significantly outperforms competing approaches such as fine-tuning, LoRA or vanilla adapters in terms of the trade-off between learning the new language and not forgetting English.

Neutral Residues: Revisiting Adapters for Model Extension
Franck SIGNE TALLA Edouard Grave'
Hervé Jégou'
Abstract
LOG PERPLEXITY
Backbone
We address the problem of extending pretrained large language model to a new domain that was not seen during training. Standard techniques, such as finetuning or low-rank adaptation (LoRA) are successf ul at domain adaptation, but do not formally add capacity to the model. This often leads to trade-off, between performing well on the new domain vs. degrading performance on the original domain. Here, we revisit and improve adapters extend I.L.Ms from three angles: data, architecture and training procedure, which are ad- vantageously considered jointly. The resulting method, called neutral residues, modifies adapters in way that leads each new residual block to output near-zeros on the original domain. This solution leads to strong results when adapting a state-of-the-ane model originally trained on En- glish a new language. Neutral residues signifi- cantly outperform competing approaches such finetuning, LoRA or vanilla adapters in terms of the trade-off between learning the new language and not forgetting English.
LEARNING (FR)
1.1
Finetuning
LoRA
1.0
Adapters
Ours
0.9
VV
0.8
VV
0.70
0.75
0.80
0.85
FORGETTING (EN)
Figure 1. Learning Forgetting? Fine-tuning model reduces the model performance on the original task. Potential trade-offs are governed by the learning rate, with extreme cases being (top- left) the original backbone and (bottom-righe finetuning the new data with large learning rate. LoRA or vanilla adapters mitigate catastrophic forgetting but only to some extent. Our method significantly improves this compromise, We report detailed results in Table 7 of Section 4.
dollars, mostly due to the training on 16,000 H100 GPUs for months, not counting the hidden costs of exploration.
In this paper, we consider the question of extending an existing model, in order to add some new capabilities or knowledge without retraining it from seratch and therefore in much that are successful for domain adaptation, such as finetuning or Low-Rank Adaptation (Hu et al., 2022. LoRA), are not adequate this context because they do not add any extra capacity. Therefore they are inherently limited the amount of knowledge that they can incorporate without suffering significant forgetting. This catastrophic forgetting a well- known problem in the continual learning setting (McCloskey & Cohen, 1989; French, 1999).
1. Introduction
The dominating strategy for producing foundation mod- els involves training from scratch on large collection of data, typically trillions of tokens, covering multiple domains. Training such models is extremely costly in terms of com- puting resources. view of the explosion of the number of gigantic models produced the last years (the HuggingFace model repository claims host about one million models), this training paradigm raises the problem of the economi- cal and ecological sustainability of the model production pipelines. As an example, the cost of training the Ilama 3 model is estimated be in the order of hundreds of millions
resource-efficien manner. The solutions
In contrast, the adapters introduced by Rebuffi et al. (2017) extend a pretrained model by adding new parameters. When transferring to a new domain or task, only these new weights are trained while those of the original backbone are frozen. This extension strategy, which was initially introduced in
Kyutai, Paris, France. Correspondence to: Franck SIGNE
TALLA siranck.signe-taar tallaaayutt.iiorrgg. krgg....rg>>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'tttps: /ai meta com/bl og/oot///eeeeeeeeeee /eta 11 1mma-3//

---

Neutral Residues: Revisiting Adapters for Model Extension
computer vision with convolutional neural networks, was subsequently adapted to textual transformers by Houlsby et al. (2019). Adapters benefit from the initial pretrained model and hence exhibit competitive transfer performance. They enjoy several additional properties. First, only a subset of weights is trained. Most importantly, they formally add capacity (unlike linear additive adapters like LoRA) and hence do not suffer from the aforementioned limitations. However, adapters still suffer significant forgetting in their current form and are therefore not sufficient for our problem.
called neutral residues. We then apply neutral residues to improve the performance of the Gemma LLM (Gemma et al., 2024) on languages such as Danish, Hungarian or Slovak, without degrading the performance on English.
2. Related Work
The success of deep learning is often associated to the out- standing performance obtained when training convolutional neural networks on large datasets such as Imagenet (Deng et al., 2009). Yet another key benefit is their amenability to transfer knowledge between tasks with limited amount of training data (Oquab et al., 2014). This transfer is usu- ally performed with a so-called fo-etuinn stage, where the original backbone is trained in a data-efficient manner to solve a distinct target task than the one employed at training time. It often requires to modify the architecture to adapt the output layers to the new tasks. Several self- pretraining methods are noticeably interested in improving ways of pretraining models on proxy tasks (Devlin et al., 2018; Caron et al., 2021). such that networks finetuned from these backbones, or used in a O-shot manner, perform well on multiple tasks. In most of these settings, the backbone only requires some form of domain adaptation.
In this paper, we build on adapters and, accordingly, increase the capacity of the model by adding feed-forward blocks, while offering a modular design (Pfeiffer et al., 2020). In particular, we consider the use-case where we add new lan- guage to a pretrained model. We measure how the extended model performs on the training criterion, namely perplexity, but also on downstream tasks such as question answering.
We analyze how to optimize the compromise between learn- ing a new language, and not forgetting the initial knowledge of the pretrained model. We study the impact of (1) training data, (2) training strategy, such as losses specifically in- tended at reducing the forgetting, and (3) architecture. This leads us to identify several important factors for successfully extending model with adapters:
Finetuning is however not sufficient when large body of knowledge must be incorporated to the network. In this section we review a body of literature related to this prob- lem and to our work. This includes solutions that add new knowledge into a pretrained network, as well as methods that incorporate gating mechanisms such MoE.
Data: When learning adapters, a simple way to reduce the forgetting is keep training with a small fraction of data similar the original distribution.
Architecture: An adapter gating mechanism is a way to ensure that the network distinguishes when it should operate as the original neural network or when the new blocks are desirable process the input data.
Parameter Efficient Finetuning. Enhancing language model capabilities using module based training has gained interest in recent years due to the high cost of full finetuning. Those methods are known parameter efficient finetuning methods (PEFT; Houlsby et al. (2019); Hu al. (2022); Li & Liang (2021)). Unlike full finetuning, they only require a limited amount of memory. Houlsby et al. (2019) pro- posed to insert small bottleneck layers, known as Adapters, within each transformer layer, allowing models to be fine- tuned by training only a small fraction of the parameters: The original model is frozen. There is a large body a litera- ture adapters, see the overview by Pfeiffer et al. (2023). LoRA (Hu et al., 2022) adds trainable low-rank matrices to transformer layers, while freezing the original weights.
Training: The gating mechanism is advantageously informed in a supervised manner. We proposed two strategies in that respect, both introducing a local loss: The first one is inspired by mixture of experts (MoE), which involves an explicit domain classifier at each block. The second one involves sparsity loss whose objective is to ensure that the residual connections output near-zero values when the input follows the pretraining distribution.
Initialization: Our analysis concurs with the obser- vation by Houlsby et al. (2019) that near-identical ini- tialization is important when training adapters. We introduce a variant that is even more drastic that the existing 0-block initialization of the output matrix.
Continual Learning without Forgetting. Continual Learning is widely studied concept (De Lange et al., 2021; Wang et al., 2024) as it allows the addition of new knowl- edge to LLM after their initial pretraining, It is usually done through full finetuning in order to learn domain specific knowledge (Roziere et al., 2023), to enhance instruction- following abilities (Wang et al., 2023) or to align LLM with
Overall, we show that these multiple ingredients are advan- tageously intertwined to obtain the best trade-off between learning the new distribution well and not forgetting the original capabilities. This leads to our proposed method,
2

---

Neutral Residues: Revisiting Adapters for Model Extension
human preferences (Ouyang et al., 2022). One of the biggest challenges of continual learning and the main drawback of doing it through full finetuning is catastrophic forgetting (Kirkpatrick et al., 2017). This is partially alleviated by us- ing the initial training data (Robins, 1995), when available, or as a proxy data from a similar distribution.
elements in the mixture and apply softmax for the rooting mechanism. Since least one expert is activated in their mixture, this selection mechanism also prevents the capacity of the model to produce O-valued outputs. Our solution of neutral residues with local loss is more effective than simply gating adapters, as shown in our experimental Section 4.
Several studies have investigated more advanced methods to address forgetting in continual learning (Biderman et al., 2024; Wu et al., 2024; Li et al., 2024; Zhong et al., 2024; Riemer et al., 2019; Li & Hoiem, 2017). Biderman et al. (2024) show that LoRA helps reducing forgetting but is less efficient at learning than full finetuning, especially for distri- butions far from the LLM pretraining distribution. Wu et al. (2024) proposed a continual learning method for L.MMs, which amounts adding new intermediate transformer blocks in interleaved manner. This enables the injection of new knowledge while preserving the initial capabilities.
3. Neutral Residues with Gated Adapters
The first adapters were initially introduced by Rebuffi et al. (2017), in the context of convolutional residual networks. Their motivation was to adapt a backbone to multiple tasks by finetuning on a limited amount of data, rather than adding a large body of knowledge. Therefore they considered a limited number of trainable parameters, typically less than 10% additional weights compared the original backbone. In the context of language models, Houlsby et al. (2019) add residual feed-forward block sequentially after each multi- head attention and the original FFN. In our case and as shown in Figure 2, we add more capacity (typically 20%) and adopt parallel adapters as we do not observe a noticeable difference compared to serial adapters. This concurs with observations by Touvron et al. (2022) that blocks can be parallelized pairwise if vision transformers are large enough.
Mixture of Experts. MoEs have long history in neural networks (Yuksel et al., 2012), for instance with set of classi- fiers where each classifier was specialized for a different task or domain. Residual architectures (He et al., 2016) and in particular transformers (Vaswani et al., 2017) have renewed the interest in this strategy. In this context, they amount to replacing the standard feed-forward networks (FFN) of transformer blocks by a collection of FFNs, referred to as experts. A gating mechanism selectively activates subset of relevant experts for a given input. This technique has been widely used to reduce the computational cost of infer- ence in large language models (Jiang et al., 2024; Xue et al., 2024) with a focus on the design of an expert-balancing loss to promote equitable token distribution across experts (Shazeer al., 2017; Pfeiffer et al., 2023).
Mixed distribution training. As discussed in the related work, a solution to limit catastrophic forgetting is to keep training with small amount of the original data, if such data is available. This strategy is typically employed with finetuning but is effective with all baselines. Li et al. (2024) report that catastrophic forgetting almost disappears when the volume of data in original language is five times greater than in the new one. Yet they also point out that mixed data training can slow down the learning of the new language.
Recent works have explored utilizing MoE's gating mecha- nism to mitigate catastrophic forgetting during the continual learning phase of language models. LoRAMoE (Dou et al., 2024) employs LoRAs as experts, integrating them with a gating mechanism to improve performance on specific downstream tasks through Supervised finetuning (SFT). Lo- RAMoE mitigates catastrophic forgetting in the case of SFT, but to our knowledge its effectiveness has not been studied in broader context. In contrast, two recent parallel concur- rent works are closer to our proposal: Zhong et al. (2024) augment Mixture-of-Expere Experrsss LLMS with new modalities by addition of new experts to a pretrained mixture of experts. Similarly, Li et al. (2024) add an MoE layer in parallel to the FFN of the transformer block to learn multilingual capabili- ties and mitigate catastrophic forgetting during Continual Learning. Our solution shares some similarities with this work by employing mixed data training. Yet Li et al. (2024) do not integrate an explicit sparsity criterion, and need to weight differently the blocks of the pretrained model with those of the gated adapters. They also consider multiple
Unfortunately, in many cases, the original training data is not available when extending a model. Thus, in the fol- lowing, we do not assume that we have access to the orig- inal data distribution, but only to data that is somewhat related. For example, when using a model that was pre- trained mostly on English Common Crawl, we use data from English Wikipedia to capture the original distribution. We also consider the more realistic case of using public model, for which we do not have details about the pretrain- ing data distribution, such the Gemma model (Gemma et al., 2024). We study the impact of the proportion p of data related to the original domain. As a particuler case, we analyze the case p = O, where no data related to the pretraining is available.
Local loss & adapter gating. One way to prevent forget- ting the pretraining distribution is to constrain the output of the model so that it does not change on the original data. This can be achieved by foreing the outputs of the adapters
3

---

Neutral Residues: Revisiting Adapters for Model Extension
Output
Output
Mu Perceptron
Adapter Block
ReLL
Output Weights
RMSNorm
RMSNorm
Projection
Multi-Heaa Se f- -AAttention
Input Weights
Ga Sating We ghts
RMSNorm
Input
Input
Figure 2. Architectural design adopted for neutral residues: (left) we add the adapter block in parallel to the FFN; ; (right) our gated adapter block includes a block gate with a ReLU non-lincarity (see Section 4.4 for comparison with sigmoid-baset gating). At training time, the weights of the blue blocks are frozen, while the trainable parameters are represented by red blocks. The output the adapter is trained with local loss such that the output is sparse if the input follows the pretraining distribution. We note that the activation function of the adapter block the same as the one used in the backbone.
to be equal to zero on the original data, and only non-zero on data from the new domain. To this end, we propose to regularize the output of the adapter modules with an lz- norm on the data related the original distribution. We argue that regularizing the output of adapters is casier than continual learning on the original distribution. Hence, using an imperfect dataset representing the original distribution should have a smaller impact on our method than finetuning.
In our case, we add a gate only on the additional adapter block, and not on the original block. We optimize the loss uuin f + alfecat where f is the language modeling objective and Gocal is a loss to train the gating to discrimi- nate the original the new distribution. We consider two gating activations and corresponding local losses:
ReLU activation with sparsity constraint. This strategy, which is the one that we refer to as "neutral residues" unless specified otherwise, does not make explicit classification. Instead, we let the ReLU activation adapts itself the strength of its response for the block. We compute the average of the E f--orrm on the output of the adapters as the local loss. We normalize the loss by the dimension of the model. This loss is only used when training on the original distribution.
During early experiments, we observed that regularizing the output of adapters helped mitigate forgetting, but it also prevented learning the new domain. As most recent trans- former LLMs, the models used in our experiments use gated linear units (GLU) in feed-forward blocks (Dauphin et al., 2017: Shazeer, 2020). The output of the block is obtained with (=)( = W. (a (W g @ Wiz), where @ is the pointwise multiplication and a a non-linearity such as SiLU (Elfwing et al., 2018). We observed that the singular values of the gating weights Wg g of adaptors are significantly more skewed than in regular transformer block from the pre- trained backbone. Figure 3 shows this behavior, which is detrimental to the learning process. Our interpretation is that all the projections instantiated by this operator tend to be colinear, and therefore to behave similarly when combined with the non-linearity. because they implicitly operate as
Sigmoid activation with cross entropy. This solution is inspired by MoE. In this case, the block gate is associ- ated with the sigmoid activation. It is thus trained as a local binary classifier whose objective is to distinguish between the old and the new data distributions, using the cross-entropy loss averaged over the adapters. In contrast to the previous strategy, this loss is used on both the original and the new distributions.
classifier between the old and new distributions.
To address this problem, we add block gute over the full adapter, in similar fashion as the gating mechanism of MoE transformers (Shazeer et al., 2017), and their applica- tion to model extension (Zhong et al., 2024; Li et al., 2024).
In Figure 3, observe that the gating operator has a significant effect on the singular values of the gating matrix: the distri- bution of the singular values is less skewed than the original backbone. Moreover, the gating mechanism based on ReL.U
4

---

Neutral Residues: Revisiting Adapters for Model Extension
LAYER
LAYER 7
LAYER 5
S SINGULAR AAR VALUES
0.6
Noutral residues
Backbone
Sigmald adapters
0.4
Adapters
0.2
0.0
1000
2000
1000
2000
0
1000
2000
Figure Spectral analysis of the weight gating matrix GLU blocks, normalized by the largest singular value.
and & ei-normis is more effective than the one based on sigmoid and the cross-entropy loss. As we will see in Section 4, it also the method that offers the best compromise between not forgetting and learning.
4. Experiments
In this section, we report experimental results to validate the performance of neutral residues to adapt a neural network to new domain. We consider the case of adding or improving the multilingual ability of a large language model. More precisely, we start from an English-only model (or a model that has only seen small amount of non-English data) and finetune it on another target language. Section F covers the case of multiple languages for a multilingual model.
Low-variance initialization. One key ingredient pointed out by Houlsby et al. (2019) is that it is best to start training adapters such that the initial network is near-identical to the pretrained one, thereby not modifying the initial function. This is typically done by initializing the output adapter ma- trix to Os, as also advocated by Zhang et al. (2019). We are even more drastic in that respect: in addition to setting the output matrix with Os, we depart from the usual He's s initialization for both the input and gating matrix and initial- ize them with much lower variance, such that the model remains longer close to the original one. More specifically, we reduce the variance employed in He's initialization: in- stead of initializing weights with variance 2/d, where d in the input dimension of the matrix, we use a variance of 1/dd L), where L. the number of transformer layers.
4.1. Datasets and evaluation protocols
Training Datasets. For the multi-lingual finetuning datasets, we use data extracted from CommonCrawl, and process it with the following steps. First, the text content is extracted from HTML using the resiliparse package.? Then, we perform language identification with fast Text to keep data in the target language only. Next, we perform deduplication at the paragraph level, by computing a hash of each paragraph. Finally, we filter document using fastText linear classifier, which was trained to discrim- inate random pages from CommonCrawl vs. high-quality documents such as Wikipedia articles, textbooks, news arti- cles or scientific articles.
FFN vs. multi-head attention (MHA). In our early ex- periments carried out to add a new language, it is more effective to add extra weights in the form of FFN than in MHA blocks, for a given number of additional parameters. This is consistent with how extra capacity is added in MoF. Similarly Li et al. (2024) present an ablation that shows that adding FFN blocks is better than adding attention blocks. Note, this is in contrast with what is observed by Hu et al. (2022) with the LoRA method, for which the best choice to adapt to different downstream tasks with few parameters to update the MHA matrix, in particular they choose to update the keys and values matrices. In our preliminary experiments, re-evaluate LoRA in our context of knowl- edge addition, and observe that LoRA is more effective when updating the FFN blocks than the MHA blocks. We hypothesize that this apparent discrepancy is due to the fact that task adaptation is different from the task of extending a pretrained network on large body of knowledge.
For the English domain, we restricted ourselves to text from Wikipedia. The motivation for this is to finetuning dataset that is related, but different, from the training dataset of the backbone model. Indeed, as discussed earlier, in many cases one does not have access to the original dataset that was used to train the backbone model, and we want to demonstrate that our method robust even in the case where exact data from the original distribution is not available.
Evaluation benchmarks. We evaluate the performance of finetuning with two steps. First, we evaluate the perplexity
Pnttps://resi1 1 1iparsi.aaraeeeee. chatnoi r.ev hnttps://fasteext
5

---

Neutral Residues: Revisiting Adapters for Model Extension
Table Mixture of distributions: impact the rate p of data that roughly approximate the pretraining distribution when learn- ing a vanilla adapter. 10% of initial data distribution is good compromise between learning (FR) and not forgetting (EN). We report perplexity (lower is better).
Table 2. Training with new data only or mixed data. Starting from an English pretrained model. we measure the tradeoff be- tween learning and forgetting if we only post-train with target data or if we keep the pretrained model. We report the perplexity on the validation sets at the end of training.
10% of original data. See Table for results of
English rate p
EN
FR
0.00
0.720
0,810
p
0
0,1
0.01
0.707
0,810
Method
EN
FR
EN
FR
0.10
0.687
0.812
Finetuning
0.874
0.755
0.811
0.758
0,50
0.683
0.828
LoRA
0.770
0.814
0.730
0,818
Vanilla adapters
0.720
0.810
0.687
0.812
Pretrained model
0.663
1.175
Neutral residues
0.684
0.790
0.668
0.793
of the model held-out sets, in English and in the target language, to measure forgetting and learning. For English, we use text from a domain that does not correspond to the finctuning data: the PubMed subset from ThePile (Gao et al., 2020). The goal here is to evaluate how robust our method to forgetting, especially in the case where we do not have access to the same training distribution as the original model. For target languages, we consider text from the same distribution the finetuning dataset.
leading to faster experiments.
Baseline extension strategies. In our experiments, we use the following methods as baselines:
Finetuming: we train all the weights from the backbone. We re-initialize the optimizer state.
LoRA: unlike Hu et al. (2022), we add all the additional parameters in the FFN layers, as in our preliminary experiments this choice was significantly better for model extension.
Second, we consider standard academic benchmarks used to evaluate large language models, such as question answering or Cloze-style problems. We use the following datasets: ARC challenge (Clark et al., 2018), HellaSwag (Zellers al., 2019), MMLU (Hendrycks et al., 2020), Common- Sense QA (Talmor al., 2018) and Belebele (Bandarkar et al., 2023). For target languages, we use the translated datasets from Dac Lai et al. (2023) and Sakai et al. (2024), when they exist. For all datasets, we score each answer in- dependently as the continuation of the question, and predict the most probable one. We use normalization, following the recommendation from Gu et al. (2024). We perform 5-shot evaluation for ARC challenge, CSQA and MMI.U, and 0-shot evaluation for HellaSwag and Belebele.
Adapters: we use the vanilla adapters as Houlsby et al. (2019). We use parallel instead of serial blocks to have a more directly comparable baseline. We use the standard adapters' initialization, ie. zeros for the output matrix and He's initialization for the other weights.
Hyperparameters. Except mentioned otherwise, LoRA, adapters, and neutrul residues use 20% of extra learnable weights. For each method, we selected the learning rate that leads to the best trade-off between learning and forgetting: 5 10-5 for finetuning and LoRA, 2 10-1 for adapters and neutral residues. The hyperparameter a governing the strength of the sparsity loss is set by default to 0.01 for neutral residues. When training on the new data, we train during 100,000 steps with a batch size of 64 sequences of length 4,096 for both EN-LM-1B and Gemma-2B. We provide other training hyperparameters in Section B.
Backbone models. We consider two models that we use as backbone. The first, called EN-LM-IB, was trained internally on 2.4T tokens of English data only, and has IB parameters. The pretraining data consists mostly of web pages filtered from Common Crawl, as described previously. It also includes a small amount of data from curated sources such as Wikimedia, Stack Exchange and scientific articles. The second model is Gemma-2B (Gemma et al., 2024) which was pretrained on small amount of multi-lingual data. We do not have access Gemma's 's training data, making it a realistic test case for our approach. Appendix B gives more details about these models. We use EN-LM-1B for all the experiments, except for the main results that are reported in Table 3. We chose to use EN-LM-1B for our exploratory experiments and ablations as is smaller, hence
4.2. Preliminary Analysis
Training with mixed data distribution Table I analyzes the impact of the ratio p of data approximating the pre- training distribution used to extend the model with vanilla adapters. We note that p 0.1 offers a good trade-off: the forgetting is not significantly higher than with p= 0.5, while the model is almost as good in French as when we train with French only (p = 0). Thus, when training with mixed data, we set p = 0.1 in all subsequent experiments.
6

---

Neutral Residues: Revisiting Adapters for Model Extension
Table Main results: comparison neutral residues (ours) versus baselines the Gemma-2B 2B model. All methods except finetuning use 20% of trainable parameters compared with the backbone models. We use proportion 0.1 of English data for the training. We report the average performance across all tasks both on English and target languages. These metrics reffect the overall performance with respect to not forgetting and learning, respectively. In bold we report the best method, in underline the second best.
Forgetting: English
Learning: Target
Task avg.
BeleBele (0)
HellaS (0)
Are Are (5)
CSQA (5)
MMLU (5)
BeleBele
HellaS (0)
Are (5)
CSQA (5)
MMLU (5)
Forgetting
(English)
Learning
(Target)
Target
Method
Backbone
46.3
69.7
47.0
63.0
40.2
44.3
50.3
40.9
53.3
33,9
53.2
44.6
French
Finetuning
45.7
62.7
45.9
57.3
37.2
44.9
60.8
43.9
63.8
35,9
49.8
49,9
LoRA
46.1
65.7
45.2
60.5
38.4
46.3
57.1
41.7
44.2
35.1
51.2
44.9
Adapters
48.1
67.8
47.5
61.4
39.3
44.6
57.4
35.2
56.9
36.0
52.8
46.0
Ours
48.2
68.5
47.3
63.3
39.2
46.8
57.9
39.7
59.8
36.7
53.3
48.2
Backbone
46.3
69.7
47.0
63.0
40.2
41.6
43.1
35.6
N/A
32.3
53.2
38.1
Danish
Finctuming
46.3
62.3
37.9
55.1
34.5
41.3
57.9
39.2
N/A
32.8
47.2
42.8
LoRA
45.2
64.7
44.7
52.4
35.3
42.2
55.3
37.4
N/A
32.9
48.5
41.9
Adapters
46.9
66.8
45.9
57.1
37.9
42.3
53.7
39.0
N/A
34.2
50.9
42.3
Ours
45.4
69.4
46.6
60.4
38.1
42.4
55.8
38.7
N/A
34.7
52.0
42.9
Backbone
46.3
69.7
47.0
63.0
40.2
34.3
36.0
31.1
N/A
31.0
53.2
33.1
Hungarian
Finetuning
44.6
60.3
40.8
48.5
35.2
37.1
48.1
36.1
N/A
32.5
45.9
38.5
LoRA
45.0
63.6
43.3
54.6
36.3
39.0
44.9
35.4
N/A
32.7
48.6
38.0
Adapters
48.0
59.1
38.1
39.6
44.8
36.6
32.7
52.0
38.4
Ours
46.1
69.2
48.5
60.6
38.5
38.2
46.1
38.1
N/A
32.8
52.6
38,8
Backhone
46.3
69.7
47.0
63.0
40.2
38.1
38.8
32.8
N/A
31.3
53.2
35.2
Slovak
Finetuning
43.2
60.0
40.0
54.3
34.3
41.3
47.8
35.2
N/A
32.4
46.4
39.2
LoRA
44.8
63.8
41.7
54.9
36.0
41.4
46.7
32.9
N/A
32.3
48.2
38.4
Adapters
45.2
62.5
38.0
55.9
37.4
42.3
40.9
34.9
N/A
32.7
47.8
37.7
Ours
68.0
42.8
60.6
38.1
40.8
46.6
34.7
32.2
51.5
38.6
4.3. Main results
Table Ablation:
Gating and local loss. report average
across tasks; see Table 16 for details.
We report our main results in Table 3, where we extend the capabilities of the Gemma 2B model on four languages: Danish, French, Hungarian and Slovak. First, note that our method, neutral residues, outperforms the LoRA and adapters baselines on all four languages. In average, it is always among the two best methods: is only second to the backbone on forgetting and to finetuning on learning. How- ever, neutral residues offers significantly better trade-off than both these methods. Second, we observe that adapters tend perform better than LoRA on the task of extending Gemma 2B target language. This is a confirmation that for tasks requiring significant new knowledge, it is helpful to explicitly add capacity to the base model in the form of additional parameters. This observation is confirmed when varying the number of additional parameters: we observe improvements when increasing the extra capacity from 5% to 50% (we report these results in Table 9 of the Appendix).
Gate
Gate loss
Perplexity
Tasks f
CE
En
FR
En
FR
(1)
0.687
0.801
45.3
42.4
0.677
0.800
45.2
42.6
Sigmoid
0.676
0.800
45.3
41.9
(4) ReLU
0.673
0.791
46.4
42.8
0.674
0.791
47.1
43.6
Adapter baseline
0.686
0.812
45.1
41.3
Backbone baseline
0.663
1.175
47.0
32.9
Table 2 gives the trade-off between learning and not forget- ting for all baselines and method, when training either without with mixed training distributions. As the mixed training significantly reduces the forgetting for all the methods without hindering too much the learning. We report more extensive results Table 10 the appendix, including on downstream tasks, when evaluating the impact of including some data related the original distribution.
see,
4.4. Ablations
Ablation of the gating and local loss. In Table 4, we compare adapters using different gating mechanisms, and

---

Neutral Residues: Revisiting Adapters for Model Extension
Table Ablation: initialization, report the average across tasks; see Table 17 for details.
Table 7. Trade-off's with different learning rates. This hyperpa- rameter allows us to fävor learning vs. forgetting or vice-versa, These results are the ones reported Figure See Table 19 for detailed results.
Gate
Our init.
Perplexity
4 FR
Tasks avg.
EN
EN
FR
Perplexity
Tasks avg. f EN
0.687
0.812
45.5
41.6
Adapters w/ f-oor
Method
LR
EN
FR
0.687
0.801
45.3
42.4
0.668
0.810
40.8
Backbone
0
0.663
1.175
47.0
32.9
Adapters w/ sigmoid
0.677
0.800
45.2
42.6
2-10
0.678
0.963
45.2
36.3
Neutral Residues
0.673
0.818
46.5
41.6
Fine-
5-10
0.705
0.840
44,4
41.1
0.674
0.791
47.1
43.6
tuning
2-10
0.811
0.758
38.9
43.6
5-10
0.874
0.757
36.0
42.5
2-10
0.716
0.842
44.1
40.5
Table Ablation: governs the strength of the & local loss. See Table 18 for detailed results.
5-10
0.730
0.819
43.7
41.2
LoRA
2-10
0.745
0.802
41.9
41.7
5-10
0.755
0.806
41.6
42.6
Perplexity J
Tasks avg. EN
2.10
0.686
0.830
45.1
40.8
En
FR
FR
Vanilla
5-10 5
0.686
0.812
45.1
41.3
0.673
0.791
46.4
42.8
adapters
2-10
0.689
0.797
46.0
42.4
0.01
0.674
0.791
47.1
43.6
5-10
0.688
0.796
45.4
43.5
0.672
0.791
46.1
2.10
0.673
0.799
46.4
42.9
0.674
0.791
46.7
42.3
Neutral residucs
5-10
0.674
0.791
47.1
43.6
10
0.672
0.791
46.6
43.2
100
0.667
0.791
46.3
43.0
2-
0.673
0.787
46.4
42.5
5-
0.675
0.789
46.6
43.0
different loss functions to learn the gating. First, comparing line (5) to line (I), we observe that regularizing the output of the adapters alone, without adding a block gating mecha- nism does not work well. We hypothesize that this is mainly due to the weights of the adapter to become co-linear, discussed in spectral analysis of Section 3. In particular, this prevent the adapter to learn the new domain well. Next, we observe that the combination of the ReLU activation with & nnrm outperforms the sigmoid activation with cross entropy loss (line (5) vs. (2)). Finally, comparing lines (4) and (5), note that adding supervision with the & regu- larization helps learning adapters, even though the ReLU gating alone already leads to strong results.
Coefficient a ablation. Table 6 shows that even without the sparsity constraint, ReLU gating enables a good balance between learning and forgetting. Introducing this constraint further improves the trade-off, with low values (a = 0.01) yielding the best results. However, a good trade-off is main- tained across wide range of strengths.
5. Conclusion
This paper has explored the feasibility and effectiveness of extending existing foundation models to incorporate new capabilities or knowledge without the need for resource- intensive retraining from scratch. By building upon the concept of adapters, which add new parameters to a pre- trained model, we have demonstrated that it is possible to extend a model without compromising its original knowl- edge, thereby offering a more sustainable approach than re- training from scratch. Our study focused on the use-case of adding a new language to a pretrained model and evaluated the performance of the extended model on both training cri- teria and downstream tasks. Our findings highlight several critical factors that contribute to the successful extension of model while mitigating the issue of catastrophic forgetting. These factors include the strategic combination of mixed training data, an adapter gating mechanism coupled with a local loss, and the importance of near rear-identical initialization. Our method, called neutral residues, obtains state-of art results extending multilingual capabilities of Gemma-2B.
Initialization ablation. In Table 5, we report the per- formance various adapters, with and without the low- variance initialization described Section Overall, we note that this initialization improves the results of gated adapters. More specifically, compared to He's initialization, it leads to better learning of the new target domain.
Learning rate ablation. Table 7 reports the trade-offss achievable by different techniques when varying the learn- ing rate. The results of our method are stable across a large range learning rates (one order of magnitude from 5-10 5 and 5 10 4), in sharp contrast with finetuning and LoRA, which are highly sensitive to this parameter.
8

---

Neutral Residues: Revisiting Adapters for Model Extension
Impact Statement
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C.,, and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:/803.054: 2018.
The goal of our work is to advance the efficiency of large langage models (LLMs) training. by improving methods to extend an existing model to a domain requiring a large amount of new knowledge. This has the potential to im- prove the sustainability and the accessibility of these sys- tems, across various applications and domains. In particular, this has potential environmental implications, as it could reduce the computational resources needed to develop a model for a new domain. Instead of training the model from scratch, one could start from strong existing base model and use method to extend it to the target domain. Our method can also improve the accessibility of these systems, by allowing organizations and researchers to personalize ex- isting models to their need without requiring a large amount of computational resources.
Dac Lai, V., Van Nguyen, C., Ngo, N. T.,, Nguyen, T., Der- noncourt, F., Rossi, R. A., and Nguyen, T. H. Okapi: Instruction- large language models in multiple lan- guages with reinforcement learning from human feed- back. arXiv e-prints, pp. arXiv-2307, 2023.
Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D. Lan- guage modeling with gated convolutional networks. In International conference on machine learning, pp. 933- 941. PMLR, 2017.
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. A continual learning survey: Defying forgetting in classifi- cation tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-3385 2021.
In general, the widespread application of LLMs has many other potential social consequences, including risks. It re- quires careful considerations of ethical issues around fair- ness, transparency and potential misuse. As our research do not directly address any of these, we do not feel any of these risks must be specifically highlighted here. We thus refer the reader to previous work on the matter, such as Bommasani et al. (2021) or Bender et al. (2021).
Deng, J., Dong, W., Socher, R., Li, L.-J.,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee. 2009.
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint orXiv:1810.044 2018.
References
Bandarkar, L., Liang, D., Muller, B., Artetxe, M., Shukla, S. N.N Husa, D., Goyal, N.. Krishnan, A., Zettlemoyer, L.., and Khabsa, M. The belebele benchmark: parallel reading comprehension dataset in 122 language variants. arXiv preprint arXivv2308. 16884, 2023.
Dou, S., Zhou, E., Liu, Y., Gao, S., Shen, W., Xiong, I.., Zhou, Y., Wang, X., Xi, 7.,, Fan, X., Pu, S., Zhu, J., Zheng, R., Gui, T., Zhang, Q. and Huang, X. Lo- Alleviating world knowledge forgetting in large language models via MoE-style plugin. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1932-1945, Bangkok, Thailand, August 2024. Asso- ciation for Computational Linguistics. URL https: / //aclanthologgeeeooo org/2024 a.1. 106.
Bender, E. M., Gebru, T., McMillan- Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and trans- parency, pp. 610-623, 2021.
RAMoE:
Biderman, D., Ortiz, J. G., Portes, J., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J., al. Lora learns less and forgets less, arXiv preprint arXiv:2405.00 09673, 2024.
Elfwing. S., Uchibe, E., and Doya, K. Sigmoid- linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Bommasani, R., Hudson, D. A.. Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint trXiv:2108.07255 2021.
French, R. M. Catastrophic forgetting in connectionist net- works, Trends in cognitive sciences, 3(4):128-133) 135, 1999.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. (650-9660, 2021.
Gao, I.., Biderman, S., Black, S., Golding, I.., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.

---

Neutral Residues: Revisiting Adapters for Model Extension
Gemma, T., Mesnard, T., Hardin, C., Dadashi, R., Bhupati- raju, S., Pathak, S., Sifre, L.,, Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini re- search and technology. arXiv preprint arXiv:2403.08295, 2024.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):935-22477, 2017.
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 109-165. Elsevier, 1989.
Gu. Y., Tafjord, O., Kuchl, B., Haddad, D., Dodge, J., and Hajishirzi, H. Olmes: A standard for language model evaluations. arXiv preprint urXiv:2406.08444 2024.
NLLB Team, Costa-jussà, M. R., Cross, J., Çelebi, O., EI- bayad, M., Heafield. K., Hefferman, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., Sun, A., Wang, S., Wenzek, G., Youngblood, A., Akula, B., Barrault, L., Gonzalez, G. M., Hansanti, P., Hoffman, J., Jarrett, S., Sadagopan, K. R., Rowe, D., Spruit, S., Tran, C., An- drews, P., Ayan, N. F., Bhosale, S., Edunov, S., Fan, A., Gao, C., Goswami, V., Guzmán, E, Koehn, P., Mourachko, A., Ropers, C., Salcem, S., Schwenk, H., and Wang, J. Scaling neural machine translation to 200 lan- guages. Nature, 630(8018) 846, 2024. ISSN 1476- 4687. doi: 10. //doi.orge 10. 1038 s41586 024 - 07335-33333-0 x.
He, K. Zhang, X., Ren, S., and Sun, J. Deep residual leam- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.00 30300, 2020.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-effiiient transfer learning for NLP. In International conference on machine learning, pp. 2790- 2799. PMLR, 2019.
URL https:
Oquab, M., Bottou, L., Laptev, I., and Sivic, J. Learning and transferring mid-level image representations using con- volutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1717-1724, 2014.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. dd I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska- -aaat-aaaa.wa... A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 14((3):3521 -20212261,66,,,, 2017.
Pfeiffer, J., Rücklé A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., and Gurevych, I. Adapterhub: framework for adapting transformers. arXiv preprint arXiv:2007.07779, 2020.
Pfeiffer, J., Ruder, S., Vulić, I., and Ponti, E. Modu- lar deep learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:// openreview net forum? www.d===wkxxxaxooee Sur- vey Certification.
Li, T., Li, S., Xie, B., Xiong, D., and Yang, B. MoE- CT: a novel approach for large language models training with resistance to catastrophic forgetting, arXiv preprint arXiv:2407.00875, 2024.
Li, X. L. and Liang, trnnin- Optimizing continu- ous prompts for generation. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natu- ral Language Processing (Volume 1: Long Pupers). pp. 4582-4597, Online, August 2021. Association for Com- putational Linguistics. doi: 10.18653/v1/2221..ac a1l- 353. URL https: / http://caaaaaatt.c.ccoco wwgp:/y....g.......... a 2el-long.35..
Rebuffi, S.-A., Bilen, H., and Vedaldi, A. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
Riemer, M., Cases, I., Ajemian, R., Liu, M., Rish, I,, Tu, Y., and Tesauro, G. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2019. URL :tttps://openzeviee: net/ forum?
id=R1gTShAct7.
10

---

Neutral Residues: Revisiting Adapters for Model Extension
Robins, A. Catastrophic forgetting, rehearsal and pseudore- hearsal. Connection Science, 7(2):123-14,, 1995.
2023.acl-long.554 URL https: / httpttt//aaaaaaatllllla hatp///aaaaantooo....... org/2023.acl-1om 754.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I.,
Wu, C., Gan, Y., Ge, Y., Lu, Z., Wang, J., Feng. Y., Shan, Y., and Luo, P. LLaMA pro: Progressive LLaMA with block expansion. In Ku, L.-W., Martins, A., and Sriku- mar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 6518-6537, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. URL https: http//aaaaaaa /atp/aaaaaantotoooocoooa. org/ 2024. acl acl-lome 1ew.longg.333 352.
Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Sakai, Y., Kamigaito, H., and Watanabe, mesqa: Mul- tilingual commonsense reasoning dataset with unified creation strategy by language models and humans. arXiv preprint arXiv:2406.0m 044.:1,,,..4215, 2024.
Shazeer, N.
Glu variants improve transformer. arXiv preprint ar arXiv:2002.05202, 2020.
Xue, F. Zheng, Z., Fu, Y., Ni, J., Zheng, Z., Zhou, W., and You, Y. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01733 2024.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, Outrageously large neural networks: The sparsely-gated mixture-of-expprts layer. In International Conference on Learning Representations, 2017. URL https: htpp//peeeeeeeeeeeeeeeeeeeeeeeee hwwpeeerwewwweeeeeeeeeeeeeeeeeee forum? id id-BlckMDq1g.
Yuksel, S. E., Wilson, J. N., and Gader, P. D. Twenty years of mixture of experts. IEEE transactions on neural networks and learning systems, 23(8):1177-1193, 2012.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Common- senseqa: A question answering challenge targeting com- monsense knowledge. arXiv preprint arXiv:1811.0099 1.00937, 2018.
2019.
Zhang, H., Dauphin, Y. N., and Ma, T. Fixup initialization:
Residual learning without normalization. arXiv preprint arXiv: arXiv:1901 1901.09321, 2019.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.001 00/ 18, 2024.
Zhong, S., Gao, S., Huang, Z., Wen, W., Zitnik, M., and Zhou, MoExtend: Tuning new experts for modality and task extension. In Fu, X. and Fleisig. E. (eds.), Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp. 80-91, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: / /aaclanthology .rgg2002
Touvron, H., Cord, M., El-Nouby, A., Verbeek, J., and Jégou, H. Three things everyone should know about vi- sion transformers. In European Conference on Computer Vision, pp. 497-515. Springer, 2022.
srw. 8.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Atten- tion is all you need. Advances in Neural Information Processing Systems, 2017.
Wang, L., Zhang, X., Su, H., and Zhu, J. A compre- hensive survey of continual learning: Theory, method and application. JEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362-5388 2024. doi: 10. T0.10099TAMMI.22443337777
Wang, Y. Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi. H. Self-instruct: Align- ing language models with self- gelfraenerated instructions. In Rogers, A., BeeBooyyGraaaa -oayder, J., and Okazaki, N. (eds.), Pro- ceedings of the 6lst Annual Meeting of the Association for Computational Linguistics ( Volume I: Long Papers), pp. 13484-13508, Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10.18653/v1/

---

Neutral Residues: Revisiting Adapters for Model Extension
A. Architecture details and Hyper-parameter settings
consider vanilla transformer EN-LM-IB, which was trained internally on 2.4T tokens of English data only, and has 1B parameters. We also considered the Gemma model (Gemma et al., 2024) and Gemma2 2B model (Team al., 2024), for which we do know the distribution. For these models, we refer to their paper for training hyper-parameters.
Table 8. Hyper-parameters details
EN-LM-IB
Gemma 2B
Gemma2 2B
Number layers
16
18
Working dimensionality
2048
2048
2048
Number of heads
128
256
256
Dimension FFN latent space
5632
16384
16384
Activation type
Gated SILU RMS pre-norm
Gated GELU RMS pre-norm
Gated GELU
Normalization
RMS pre & post norm
Group query
V
Pretraining context length
4096
8192
8192
Hyper-parameters for training the extended model
Batch size
64
64
64
Context length
4096
4096
4096
#Training steps
100,000
100,000
100,000
AdamW:
0.9 0.9
0.9
AdamW: 32
0.95
0.95
0.95
B. Impact of the number of parameters
We vary the extra capacity allocated to new learnable weights in LoRA, adapters and neutral residues (ours). The steady performance improvement in the new language highlights the importance of expanding the base model's capacity for tasks requiring substantial new knowledge. Interestingly, the impact of extra weights on forgetting is almost neutral. We trained EN-LM-1B with learning rate of 2 10 4 for finetuning and adapters, and 5 10 for LoRA and ours with other hyperparameters detailed in Table 8. For each extra weight size, reported in bold the best method.
Table 9. Impact of the number of parameters.
Forgetting: English
Learning: French
Task avg.
BeleBele (0)
BeleBele (0)
HellaS (0)
Are (5) (5)
CSQA (5)
MMLU
HellaS (0)
Arc Are (5)
CSQA (5)
MMLU
English
French
Extra Weights
Method
Backbone
43.0
60.3
40.5
56.5
34.5
34.6
32.8
26.7
42.3
28.0
47.0
32,9
Finetuning
37.9
34.8
45.5
30.9
48.6
35.3
60.3
38.9
43.6
LoRA
42.0
54.1
36.1
50.1
32.8
41.7
42.3
32.4
50.0
30.4
43.0
39.4
+5%
Adapters
42.2
40.4
55.1
33.7
44.1
34.2
49.5
ours
42.9
59.3
40.1
55.8
33.8
42.2
44.1
33.6
52.1
30.3
46.4
40.5
LoRA
41.7
53.6
38.1
51.1
32.8
44.2
51.8
+10%
Adapters
42.9
57.5
38.7
53.3
33.6
43.2
46.4
33.5
53.4
30.7
45.2
41.5
ours
44.4
58,9
42.2
56.8
34.2
44.3
46.5
34.1
56.5
31.0
47.3
42.5
LoRA
42.0
53.1
38.1
52.3
32.8
42.4
46.2
32.7
54.2
30.7
43.7
41.2
+20%
Adapters
43.1
57.7
40.3
54.1
34.5
44.3
48.8
34.8
52.7
31.2
46.0
42.4
43.9
59.3
41.6
56.8
34.1
45.1
48.8
35.7
56.9
31.6
47.1
43.6
LoRA
41.8
52.0
36.1
51.0
32.7
44.3
47.9
35.6
56.2
30.8
42.7
43.0
+50%
Adapters
43.3
57.6
39.4
53.3
33.5
45.8
50.4
36.5
58.0
31.9
45.4
44.5
ours
43.1
40.3
56.3
34.2
50.9
35.2
59.3
32.1
44.3
12

---

Neutral Residues: Revisiting Adapters for Model Extension
C. Impact of mixed data distribution.
Table 10 shows the importance of training with data approximating the pretraining distribution the model. The mixed training significantly reduces the forgetting for all the methods without hindering too much the learning. We trained EN-LM-1B with a learning rate of 10 for finetuning and adapters, and 10 5 for LoRA and ours with other hyperparameters detailed in Table 8. In bold report the best method, in underline the second best.
Table 10. Impact of mixed data distribution.
Forgetting: English
Learning: French
Task avg.
BeleBele (0)
BeleBele (0)
Are C
CSQA
MMLU (5)
HellaS (0)
Arc C
CSQA
MMLU (5)
HellaS
English
French
Ratio Method
Backhone
43.0
40.5
56.5
34.5
34.6
32.8
26.7
42.3
28.0
47.0
32.9
0.1
Finetuning
37.9
45.5
34.8
45.5
30.9
42.2
48.6
35.3
60.3
31.5
38.9
43.6
LoRA
42.0
53.1
38.1
52.3
32.8
42.4
46.2
32.7
54.2
30.7
43.7
41.2
Adapters
43.1
57.7
40.3
54.1
34.5
44.3
48.8
34.8
52.7
31.2
46.0
42.4
ours
43.9
59.3
41.6
56.8
34.1
45.1
48.8
35.7
56.9
31.6
47.1
43.6
Backbone
43.0
60.3
40.5
56.5
34.5
34.6
32.8
26.7
42.3
28.0
47.0
32.9
Finetuning
36.4
42.1
32.5
43.0
29.4
42.4
49.2
36.4
60.3
44.1
II
LoRA
40.9
50.4
37.1
49.0
32.3
42.2
46.6
33.3
54.7
30.6
41.9
41.5
Adapters
42.2
55.1
38.4
52.8
32,9
42.3
48.3
33.6
53.5
31.5
44.3
41.9
41.9
58.6
39.0
54.2
33.6
43.1
48.5
35.8
56.3
31.4
45.5
43.0
also ran an experiment where EN-LM-1B was fine-tuned exclusively on Wikipedia data to assess the impact this limited distribution on forgetting. Specifically, we fine-tuned EN-LM-1B on English Wikipedia for 50,000 steps using a learning rate of 2 10-4, in Table 10, with the remaining hyperparameters listed in Table 8. This restricted fine-tuning dataset resulted in poor performance across all downstream tasks.
However, when combined with earlier results, this suggests that even when the fine-tuning distribution is only rough approximation of the original pretraining distribution, strong performance on English downstream tasks can still be maintained.
Table 11. Impact of training Wikipedia only.
Forgetting: English
Method
BeleBele (0)
HellaS
Arc (5)
CSQA (5)
MMLU (5)
Avg.
Backbone
43.0
60.3
40.5
56.5
34.5
47.0
Finetuning
39.0
49.3
36.7
47.6
31.0
40.7
D. Math Dataset Experiments
We further fine-tuned the Gemma 2B model (Gemma et al., 2024) on two math-focused datasets OpenMattInnttrut-2 and MetaMathQAAfor 50,000 steps using batch size of 4096 x 64 and peak learning rate of 5 x 10-5 across all configurations. For the Ours variant, we set the LI regularization coefficient to 0.01. The results are presented in Table 12. We observe significant forgetting on English tasks, likely due to the overlap between the mathematical English in the fine-tuning data and general English. The Wikipedia-basee English corpus is not sufficient to help the model disambiguate between the two.
13

---

Neutral Residues: Revisiting Adapters for Model Extension
Table 12.
Maths results: In bold report the hest method, underline the second best.
Forgetting: English
Learning: Target
Task avg. Avg. score
Target
Method
BeleBele (0)
HellaS (0)
Are (S)
CSQA (5)
MMLU (5)
GSM8K (5)
Backbone
47.1
71.3
47.8
64.5
39.0
20.0
53,9
Maths
Finetuning
43.6
35.1
56.4
31.8
58.4
46.2
LoRA
41.7
68.5
35.3
37.2
32.5
57.6
43.0
Adapters
42.8
68.6
38.1
58.7
32.4
47.6
48.1
Ours
42.9
68.2
39.2
60.8
32.0
54.4
48.6
E. Japanese Dataset Experiments
To assess our approach beyond alphabetic scripts, we conducted additional experiments with Japanese, language that uses non-Latin script. Specifically, augmented Gemma-2B with Japanese data extracted from CommonCrawl, processed using the same pipeline as the other multilingual finetuning datasets. The results presented in Table 13.
Table 13. Results on Japanese: In bold we report the best method, in underline the second best.
Forgetting: English
Learning: Target
Task avg.
BeleBele (0)
BeleBele
HellaS (0)
(S)
CSQA
MMLU (5)
CSQA
MMLU (5)
Forgetting
(English)
Leariing
(Target)
Target
Method
Backbone
46.2
69.7
46.9
61.8
40.4
37.3
55.6
30.3
53.0
41.1
Japanese
Finetuning
44.6
60.2
42.3
35.6
38.7
32.5
48.0
46.5
LoRA
45.7
63.3
45.5
58.7
37.4
37.3
63.0
32.4
50.1
44.2
Adapters
46.9
65.9
46.5
60.2
38.7
37.1
63.7
32.7
51.7
44.5
Ours
47.1
66.9
48.5
61.6
39.1
39.1
64.1
32.7
52.6
45.3
F. Multilingual Dataset Experiments
We explore multilingual setting where the goal is learn languages while preserving performance previously known ones. We use the Gemma 2 2B model (Team et al., 2024), chosen for its strong multilingual capabilities. The learned languages are Danish, Hungarian, and Slovak; the retained ones are English, French, German, Portuguese, Spanish, and Italian. Training data consists of 90% balanced learned languages and 10% balanced retained languages (p 0.1), approximating the model's original distribution. We compare all methods (with 20% additional parameters, except full fine-tuning) on their ability learn new languages (Table 15) without degrading performance on retained ones (Table 14). We also evaluate them FLORES (NLLB Team et al., 2024) (EN - X) in 5-shot setting.
14

---

Neutral Residues: Revisiting Adapters for Model Extension
Table 14. Forgetting results for the multilingual setting: In bold we report the best method, in underline the second best.
Target
Method
BeleBele (0)
HcllaS (0)
Are C (5)
CSQA (5)
MMLU (5)
FLORES (5)
Avg.
Backbone
56.3
73,9
65.6
69.1
52.3
X
63.4
English
Finetuning
46.1
59.2
40.9
53.1
33.7
46.6
LoRA
49.3
67.8
53.8
64.0
41.3
55.2
Adapters
52.2
72.8
64.8
68.1
50.4
X
61.7
Ours
52.8
73.3
65.4
66.6
51.6
X
61.9
Backbone
48.8
60.1
56.2
58.5
44.9
43.9
52.1
French
Finetuning
45.4
50.5
37.6
42.7
30.4
34.7
40.2
LoRA
45.1
55.1
44.8
48.7
35.2
39.2
44.7
Adapters
47.6
58.7
55.4
59.3
42.6
43.6
51.2
Ours
48.8
59.7
57.1
59.9
44.0
44.1
52.3
Backhone
47.6
54.0
52.5
65.0
43.7
30.8
48.9
German
Finctuning
42.4
47.5
35.3
50.8
30.8
25.2
38.7
LoRA
50.4
53.7
35.7
26.5
42,4
Adapters
47.2
52.7
53.5
65.8
42.7
30.6
48.8
Ours
47.2
53.1
54.4
68.4
43.9
30.6
49.6
Backbone
46.3
58.7
56.5
68.3
45.1
44.3
53.2
Portuguese
Finetuning
40.9
50.8
37.1
51.6
30.7
37.2
41.4
LoRA
41.3
50.5
43.2
54.4
34.8
33.3
42.9
Adapters
43.9
57.4
55.3
69.5
43.1
44.5
52.3
Ours
44.4
58.1
56.7
68.6
44.4
44.0
52.7
Backhone
61.2
X
45.3
25.8
47.9
Spanish
Finetuning
41.4
51.6
40.3
x
30.9
23.0
37.4
LoRA
43.8
56.1
45.2
36.6
24.7
41.3
Adapters
45.7
59.8
56.6
43.7
25.8
46.3
Ours
60.6
X
44.9
26.0
46.8
Backbone
42.7
56.3
$4.2
44.4
24.8
44.5
Italian
Finetuning
38.4
48.5
35.1
30.0
21.3
34.7
LoRA
39.8
48.2
41.1
34.8
19.4
36.7
Adapters
42.7
54.8
54.7
43.0
25.1
44.0
Ours
42.0
55.6
54.6
44.0
25.6
44.4
Table Learning results the multilingual setting: we report best method, underline the second best.
Target
Method
BeleBele (0)
HellaS
Are (5)
MMLU (5)
FLORES (5)
Avg.
Backbone
44.3
52.0
51.8
43.8
34.8
45.3
Danish
Finetuning
44.4
58.2
33.2
LoRA
44.6
60.4
46.9
37.1
44.2
46.6
Adapters
46,8
58.7
54.5
43.5
43.3
49,3
Ours
44.2
56.6
53.7
43.6
41.6
48.0
Backbone
39.7
42.6
45.6
40.2
14.0
36.4
Hungarian
Finctuning
39.4
49.4
39,0
32.2
21.5
36.3
LoRA
33.4
35.0
33.2
29.3
20.8
30.3
Adapters
41.9
48.3
50.2
40.4
21.3
40.4
Ours
40.3
46.0
48.1
40.3
19.0
38.8
Backbone
43.3
46.4
49.7
41.6
19.2
40.1
Slovak
Finetuning
43,9
50.7
38.8
33.1
28.6
39,0
LoRA
39.1
38.2
37.3
31.2
28.4
34.8
Adapters
44.9
50.5
50.7
41.6
28.1
43.1
Ours
43.7
48.6
49.5
41.6
41.8
15

---

Neutral Residues: Revisiting Adapters for Model Extension
G. Detailed Ablation Results
This section provides the full detailed results corresponding to ablation study discussed section 4.4.
Table 16. Ablation: Gating and local loss.
Gate
Local loss
Forgetting: English
Learning: French
Task avg.
BeleBele (0)
BeleBele (0)
HellaS (0)
Are (5)
CSQA (5)
MMLU 6
HellaS (0)
Arc Are (5)
CSQA
MMLU (5)
Forget.
Learn
&
CE
(1)
43.1
57.4
37.5
54.5
33.7
43.2
47.7
34.6
54.9
31.7
45.3
42.4
(2) Sigmoid
41.0
57.9
40.2
53.1
34.0
43.2
47.4
36.2
54.5
31.4
45.2
42.6
(3)
42.3
57.7
39.1
53.7
33.4
44.0
47.6
34.5
52.3
31.1
45.3
41.9
(4)
43.4
59.1
39.3
55,8
34.4
44.6
48.7
34.8
54.3
31.8
46.4
42.8
ReL.U
(5)
43.9
59.3
41.6
56.8
34.1
45.1
48.8
35.7
56.9
31.6
47.1
43.6
(6)
Adapter baseline
42.1
56,9
38.7
53.6
33,9
43.2
46.2
34.1
52.0
30.9
45.1
41.3
Backbone baseline
43.0
60.3
40.5
56.5
34.5
34.6
32.8
26.7
42.3
28.0
47.0
32.9
Table 17. Ablation: initialization.
Forgetting: English
Learning: French
Task avg.
BeleBele (0)
HellaS (0)
Are C
CSQA
MMLU (5)
BeleBele (0)
Are (5)
CSQA (5)
(5)
MMLU (5)
HeilaS
Forget.
Learn.
Gate
Init
Adapters w/ & norm
43.9
57.2
39.2
53.6
33.5
42,4
46.6
34.0
53.6
31.3
45.5
41.6
43.1
57.4
37.5
54.5
33.7
43.2
47.7
34.6
54.9
31.7
45.3
42.4
Adapters w/ sigmoid
42.6
$7.2
38.4
56.4
33.6
43.2
32.8
50.6
45.6
40.8
41.0
57.9
40.2
53.1
34.0
43.2
47.4
36.2
54.5
31.4
45.2
42.6
Neutral Residues
43.0
59.2
39.9
55.8
34.7
42,0
45.7
34.3
54.9
30.9
46.5
41.6
43.9
59.3
41.6
56.8
34.1
45.1
48.8
35.7
56.9
31.6
43.6
Table 18. Ablation: a controls the strength of the & local loss.
Forgetting: English
Learning: French
Task avg.
BeleBele (0)
BeleBele (0)
HellaS (0)
Arc (5)
(5)
MMLU (5)
Hellas (0)
Arc (5)
CSQA
MMLU (6)
Forget.
Learn
0
43.4
59.1
39.3
55.8
34.4
44.6
48.7
34.8
54.3
31.8
46.4
42.8
0.01
43.9
59.3
41.6
56.8
34.1
45.1
48.8
35.7
56.9
31.6
47.1
43.6
42.3
59.1
38.1
34.1
44.4
48.7
34.7
55.8
42.9
43.1
59.5
39.9
56.3
34.6
43.3
48.7
34.7
53.6
31.1
46.7
42,3
10
42.7
59.3
39.7
57.0
34.2
44.3
48.4
35.6
55.9
31.6
46.6
43.2
100
43.8
59.2
38.8
55.6
34.0
45.0
48.6
34.0
56.0
31.2
46.3
43.0
16

---

Neutral Residues: Revisiting Adapters for Model Extension
Table 19. Trade-off's with different learning rates.
Forgetting: English
Learning: French
Task avg.
BeleBele (0)
BeleBele (0)
HellaS (0)
Arc (5)
CSQA (5)
MMLU
HellaS (0)
Arc Are (5)
CSQA (5)
MMLU (5)
Forget.
Learn.
Method
LR
Backbone
0
43.0
60.3
40.5
56.5
34.5
34.6
32.8
26.7
42.3
28.0
47.0
32.9
2-10 5
41.9
58.3
37.9
54.5
33.6
38.3
37.2
29.5
47.5
29.1
45.2
36.3
Fine-tuning
5- 10
41.8
55.8
37.5
53.7
33.1
41.7
44.1
34.0
54.8
30.8
44.4
41.1
10
37.9
34.8
45.5
48.6
60.3
43.6
5 10
36.9
40.4
32.1
41.1
29.5
42.8
47.6
34.0
57.3
30.9
36.0
42.5
10
42.3
38.9
51.5
44.4
54.7
5-10-5
42.0
53.1
38.1
52.3
32.8
42.4
46.2
32.7
54.2
30.7
43.7
41.2
LoRA
2.10-4
41.3
51.1
36.1
48.4
32.4
42.1
47.3
33.8
53.9
31.2
41.9
41.7
5-
41.1
50.8
36.6
47.0
32.3
44.7
48.0
34.4
54.8
31.0
41.6
42.6
Vanilla adapters
2. 1-
43.0
56.9
38.7
52.8
34.0
41.7
44.6
33.3
54.3
30.3
45.1
40.8
5-10-
42.1
56.9
38.7
53.6
33,9
43.2
46.2
34.1
52.0
30.9
45.1
41.3
2 2.10-
43.1
57.7
40.3
54.1
34.5
44.3
48.8
34.8
52.7
31.2
46.0
42.4
10-
42.3
57.9
39.2
53.8
33.6
48.6
36.2
56.2
31.5
43.5
Neutral residues
2. 10
42.6
59.1
40.8
55.5
33.9
44.9
47.4
35.6
54.6
31.9
46.4
42.9
43.9
59.3
41.6
56.8
34.1
45.1
48.8
35.7
56.9
31.6
47.1
43.6
2 10
42.7
59.1
40.7
55.0
34.5
43.6
49.2
35.2
53.1
31.6
46.4
42.5
5 10
42.8
59.3
39.4
57.2
34.2
49.1
55.2
43.0
17


Reviews:
[{"review_id": "EDzuUO5qzE", "reviewer": "Reviewer_JvbR", "summary": "Authors explore a set of techniques and strategies for reducing the compute needed to train an already trained LLM for a new task or language. These include the ratio of the training data to the pretraining data, the architecture of the newly added modules (ffd or multi-head attention), the way that they are added to the network (sequential or parallel), the effectiveness of adding the gating mechanism to the new modules, the types of gating mechanism (relu or sigmoid), the ways to train the gating mechanism, and the initialization of the added modules. In some cases the topics have been already explored in previous studies (e.g., training data, the architecture of the newly added modules, the way that they are added, etc), and in some other cases the topics are drawn from closely related research areas (e.g., types of the gating mechanism).\n\nThe techniques mentioned above are evaluated in a set of experiments and in most cases are shown to be marginally effective.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank the reviewer for their feedback on our paper.\n\n1. **To my understanding, all the techniques and strategies discussed in the paper are drawn from previous studies**.\n\nTaken individually, some techniques used in our paper were indeed proposed in previous work. However, a key insight of our paper is the interplay between these different modifications of adapters, and the fact that they cannot be studied effectively independently. Moreover, we believe that some of our proposed modifications are original, such as constraining the output of an adapter module to zero with an L1 norm.\n\n2. **There is no point in reporting ablation studies in average performance.**\n\nThanks for the suggestion, we will detailed results in the appendix of the final version of the paper.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The study mostly discusses already existing techniques", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "**Strengths:**\n\n-  The discussion is detailed, and in some cases the arguments are insightful.\n-  The experiments are convincing and insightful.\n-  The paper is easy to read.\n\n**Weaknesses:**\n\n- To my understanding, all the techniques and strategies discussed in the paper are drawn from previous studies. It is still nice to see all of them in one paper, but it also makes me a bit reluctant to recommend it as an icml paper. I am not sure.\n- There is not point in reporting ablation studies in average performance. Ablation studies are reported for detailed comparison across tasks.\n- The improvements (Table 3) in my opinion are not significant.\n\nAll in all this is an old style deep learning paper that discusses a set of techniques that show that empirically work better than the alternatives.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fxsoo5IH4E", "reviewer": "Reviewer_5QgN", "summary": "This paper addresses the challenge of extending a pretrained large language model to a new domain (e.g., a new language) without catastrophic forgetting of the original domain. The authors propose “neutral residues,” a method that adds adapter layers to the model and trains them such that their outputs are near-zero for original-domain inputs​. Overall, the idea is intuitive and the results are promising, but some limitations in scope and novelty lead me to lean towards a weak rejection of this paper in its current form.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank the reviewer for their feedback on our paper.\n1. **Limited Experiment Scope: The experiments are mostly on adapting to one new language.** \n\nExploring sequentially adding multiple domains would be interesting, but due to time constraints, we leave it for future work. However, we conducted new experiments by adding French, Danish, Hungarian, and Slovak simultaneously to Gemma 2B, as also recommended by the reviewer CFSi: “*Have you ever tried out any multilingual settings including more than 2 languages in total ?* ”. The training is done with the same hyperparameter setting as in Table 3, over 100,000 steps, using 10% English data (wikipedia dataset) and the four languages in equal parts. We report the mean task performance for each language.\n\n| Method | EN | FR | DA | HU |SK|\n|-|-|-|-|-|-|\n| Base | 53.0 | 44.2 | 37.6 |33.1 | 34.6|\n| Finetuning | 48.6 | 47.3 |43.3 |38.3| 40.7|\n| Lora  | 50.3 | 45.8 |41.5 |37.2|38.7|\n| Adapters  | 50.9 | 44.9 |41.4 |37.1|38.2 |\n| Ours | 52.4 | 46.0 |41.2 |36.7|39.1|\n \nWe also trained gemma 2B on math datasets ([OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2) and [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA))  over 50000 steps with a batch of 4096*64, lr max of 5e-5 for all settings and the coefficient of the L1 norm for *ours* remains 0.01.\n\n| Method | EN | GSM8K|\n|-|-|-|\n| Base | 53.7 | 20 |\n| Finetuning | 45.5 | 58.4 |\n| Lora  | 43.7 | 57.6 |\n| Adapters  | 48.1 | 47.6 |\n| Ours | 48.1 | 54.4 |\n\n2. **Ablation of Components: [...] train an adapter with data replay but without enforcing neutral initialization, and vice versa**\n\nTo demonstrate the significance of data, architecture, and training in mitigating forgetting during learning, we conducted several ablation studies:\n\n- Impact of Mixed Data Distribution (Section 4.2, Table 9 in the Appendix): We analyzed the effect of training with a mixed data distribution across all settings, emphasizing the importance of maintaining a distribution that approximates the pretraining distribution.\n\n- Ablation of Gating and Local Loss (Table 4): We investigated various gating mechanisms and their training approaches, demonstrating their role in distinguishing data similar to the pretraining distribution from newly learned data.\n\n- Initialization Ablation (Table 5): We compared adapters trained with and without our initialization, using both L1 loss and standard cross-entropy loss (see Section 3, “Sigmoid activation with cross-entropy”).\n\nTaken together, these experiments highlight that all three factors—data, architecture, and training—are essential.\n\nCombining the ablation studies, we replicate the reviewer's suggestion by comparing neutral residues with data replay without neutral initialization and vice versa instead of adapters. Below are results on EN-LM-1B trained on French data:\n\n| Method | Tasks |||  Perplexity ||\n|-|-|-|-|-|-|\n|  | EN | FR || EN| FR |\n| data & no init | 46.5 | 38.4  ||0.673|0.818|\n| no data & init |46.0 | 40.7||0.684|0.789|\n\nThis highlight the importance the initialization to reduce the forgetting through the training without hurting the learning.\n\nWe agree with the reviewer that these ablations were underemphasized and adding experiments, such as the previous one on adapters, will strengthen the studies. These will be included in the final version.\n\n3. **Lack of Theoretical Insight**\n\nUsing the EN-LM-1B model finetuned on French, we compute the ratio of the L1 norms of the output of adapters and the output of the backbone MLP. We compare vanilla adapters (A) to neutral residues (B), to illustrate the impact of our method. We average over 2.6M tokens:\n\n**On English Pubmed**\n| Layer | A | B |\n|-|-|-|\n| 3 | 0.208 | 0.012 |\n| 7 | 0.209 | 0.011 |\n| 11 | 0.375 | 0.005 |\n| 15 | 0.266 | 0.002 |\n\n**On French valid set**\n| Layer | A | B |\n|-|-|-|\n| 3 | 0.557 | 0.547 |\n| 7 | 0.379 | 0.412 |\n| 11| 0.703 | 0.730 |\n| 15 | 1.578 | 1.453 |\n\nThis experiment reveals that gating minimizes residual outputs in English to reduce forgetting.\n\n4. **Analysis of Forgetting vs. Capacity**\n\nThanks for the suggestion, we will add a discussion in the final version of the paper.\n\n5. **Enforcing Neutral Outputs: Could you elaborate on how you ensure the adapter outputs are near-zero for the original domain ?**\n\nIn fact as described in section 3 part “Low-variance initialization” we initialize the adapter’s final linear layer to zero so that initial output is zero. Then during training the L1 loss is applied for data approximating the pretraining distribution to help maintain the output of the new blocks near zero for those data.", "claims_and_evidence": "The primary claim is that neutral residue adapters enable superior domain adaptation compared to existing methods by preserving original task performance while learning the new domain​. The evidence comes from experiments: the authors show that a model with neutral residue adapters achieves lower perplexity on English (original domain) and comparable or better performance on the new language, versus baselines that either forget English or underperform in the new language. They also evaluate on benchmark QA and knowledge tasks (ARC, HellaSwag, MMLU, etc.) in both English and the target language, where the adapted model maintains strong English accuracy while gaining new-language capability. The evidence is credible that neutral residues work well in the tested scenario. However, it is mostly limited to one setting (one original model and one target domain), so the generality of the claim (to other domains or models) is not fully proven.", "methods_and_evaluation_criteria": "Method: Neutral residues modify the standard adapter approach in three aspects (as hinted by the paper): data, architecture, and training procedure. Architecturally, they insert adapter layers (small learned modules) and initialize/train them such that on original-domain inputs their contribution is nearly zero, hence “neutral”​. In training, they mix a small amount of original-domain data into the fine-tuning of adapters to explicitly guard against forgetting​. This data mixing ensures the adapter doesn’t drift too far from the original distribution. The training procedure likely also involves a special initialization (the paper references prior work that initializing adapters to near-identity is important) so that initially the model’s behavior is unchanged.\n\nEvaluation: They use a two-step evaluation: (1) Perplexity on held-out English vs. new-language text to quantify forgetting vs. learning​. Lower perplexity is better; an ideal adaptation would keep English perplexity low (close to the original model) while improving new-language perplexity. (2) Downstream tasks performance on standard benchmarks in both languages​, such as question answering and commonsense reasoning tasks. They compare against fine-tuning the whole model, Low-Rank Adaptation (LoRA), and vanilla adapters. The use of perplexity and a diverse suite of tasks is appropriate, covering both intrinsic performance and extrinsic task efficacy. The criteria focus on the trade-off curve between new task gain and original task loss – a key aspect of this problem. The paper’s results highlight, for instance, that neutral residues achieve a better balance on this trade-off curve, outperforming baselines at equivalent points​.", "theoretical_claims": "There are no new formal theoretical claims in this work. The paper is largely empirical. It builds on the known concept that adding capacity (via adapters) can mitigate catastrophic forgetting (since fine-tuning with no new parameters has an inherent capacity trade-off​). The authors reference the theory of catastrophic forgetting from continual learning literature and conceptually argue that, because fine-tuning and LoRA do not add capacity, they are “inherently limited” and will eventually forget earlier knowledge​. Neutral residues, by adding extra parameters, avoid this limitation. However, this is an intuitive argument rather than a new theory. The paper does not provide theoretical guarantees (e.g., no formal proof that adapter outputs remain zero or that forgetting is bounded). The contribution is primarily a technique and its empirical validation.", "experimental_designs_or_analyses": "The experimental design is straightforward and solid for the scenario considered. The authors take a state-of-the-art English language model and adapt it to a single target language (the paper implies French as the target, given “FR” in results). They compare multiple methods under the same conditions (same base model, same new data). This controlled setup makes the comparisons fair. They evaluate on multiple axes: perplexity on two domains and accuracy on five benchmark tasks, which provides a holistic view of performance.\n\nOne commendable aspect is the evaluation of different proportions of original data mixed during training​. In Table 1, they vary the fraction of English data in the adapter training and find ~10% is a good trade-off, which empirically supports their data mixing strategy. This kind of ablation is useful. They also presumably keep the original model’s weights frozen for all adapter methods to ensure comparability (since that’s how adapters typically operate).\n\nA minor critique is that the experiments focus on only one new domain (language). We don’t see results for, say, adapting to a second language or a different domain (like adapting an English model to scientific text or code). So it’s unclear if the approach consistently works in other settings or if any domain-specific tuning was needed. Additionally, while they beat baselines, it would help to understand which component of their approach contributes most – e.g., is it the data replay that helps more, or the architectural constraint of near-zero outputs? The paper doesn’t fully separate these; an ablation where they train adapters without the near-zero constraint (just with data mixing) could isolate the effect. The analysis currently attributes the success to the combination of all changes. Despite these points, the overall analysis of forgetting vs. learning is thorough for the given setting, and the improvements are clearly demonstrated (often neutral residues get the best of both worlds: low new-language perplexity with minimal English degradation).", "relation_to_broader_scientific_literature": "This work is related to continual learning, domain adaptation, and parameter-efficient fine-tuning of large models. The authors do well to situate it: they cite the original adapters paper (Houlsby et al., 2019) and other PEFT (Parameter-Efficient Fine-Tuning) methods​. They also connect to catastrophic forgetting literature, citing classic works (McCloskey & Cohen, 1989; French, 1999) and more recent ones like Elastic Weight Consolidation (Kirkpatrick et al., 2017)​.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths:\n   1. Addresses a crucial problem: As models get larger, being able to extend them without full retraining is very important (for efficiency and sustainability). This work directly tackles that by enabling domain expansion at low cost.\n    2. Simplicity and Elegance: The idea of making adapter outputs neutral on old data is simple yet clever. It doesn’t require a complex loss function besides possibly using some original data. It’s an elegant tweak that has a big effect on forgetting.\n    3. Empirical Performance: The method shows clear empirical gains. It outperforms strong baselines (fine-tuning, LoRA) in preserving original performance while learning new language​. The results are consistent across both perplexity and benchmark evaluations, adding credibility.\n    4. Comprehensive Evaluation: I appreciate that they evaluated on multiple metrics (perplexity and QA benchmarks) and considered different proportions of replay data. This gives a well-rounded view of the method’s behavior.\n    5. Efficiency: Using adapters keeps the number of trained parameters small (adapters typically add only ~3-5% of parameters). So, the approach is computationally efficient – an important practical strength, aligned with the paper’s motivation of avoiding huge retraining costs.\n\nWeaknesses:\n    1. Incremental Novelty: The method is essentially an improved adapter training recipe. Adapters and data replay are existing ideas; the main novelty is the “near-zero output” constraint. While useful, it’s not a huge conceptual leap beyond prior work. Some might view it as an incremental improvement rather than a fundamentally new method.\n    2. Limited Experiment Scope: The experiments are mostly on adapting to one new language. It’s unclear how the method performs for different kinds of domain shifts (e.g., style or topic changes, or adding multiple new domains sequentially). The paper would be stronger if it demonstrated success in more than one scenario. Right now, it’s possible the approach was tuned specifically for the one case.\n    3. Lack of Theoretical Insight: There isn’t a deeper analysis of why forcing near-zero outputs is the best way to retain knowledge. For instance, could this be at odds with learning the new task (since you’re constraining the model)? The paper doesn’t theoretically guarantee anything about forgetting, so one must trust the empirical results. A bit more explanation (even qualitative) of how neutral the adapters remained (did they truly output near zeros on English inputs?) would help understand the mechanism.", "other_comments_or_suggestions": "1. Ablation of Components: As mentioned, an ablation study would strengthen the paper. For example, train an adapter with data replay but without enforcing neutral initialization, and vice versa, to see which contributes more. This would inform if the “three angles” (data, architecture, training) are all necessary.\n2. Generality to Multiple Domains: If space permits (or in future work), it’d be great to test adding two new domains sequentially (to simulate continual learning). Does the first adapter remain neutral while a second is added? Perhaps stacking adapters for each new domain could be explored.\n3. Analysis of Forgetting vs. Capacity: The paper could discuss an interesting insight: fine-tuning fails because it has to shove new knowledge into existing weights, whereas their adapter adds new weights. It might be worth highlighting how their approach relates to the idea of “soft modularity” – you’re effectively modularizing knowledge (base model for old stuff, adapter for new). This perspective could resonate with the continual learning community.", "questions_for_authors": "1. Enforcing Neutral Outputs: Could you elaborate on how you ensure the adapter outputs are near-zero for the original domain? Do you initialize the adapter’s final linear layer to zero weights (so initial output is zero) and then rely on the presence of original data during training to keep it low? Or do you use an explicit loss term that penalizes any deviation on English examples? Clarifying this would help understand how “neutrality” is maintained throughout training.\n2. Generality to Other Domains: Have you tried applying neutral residues to a different kind of domain shift, such as adapting an LLM trained on general text to a domain like legal texts or code? If not, what do you expect – would the method work equally well, and would you need any adjustments? Any insight into this would tell us about the scope of applicability.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n.a."}}, {"review_id": "zsgkfUkEaf", "reviewer": "Reviewer_Vwee", "summary": "Extending a pre-trained large language model (LLM) to a new domain/language is challenging. It is known that such model extension often encounter a trade-off, between performing well on the new domain/language vs degrading performance on the original domain/language. This paper addresses such the problem by revising adaptors. In the experiments, they report perplexity to measure how the extended model performs well on the targeted domain/language, as well as model performance in the downstream tasks such as question answering. Their carefully designed experiment and its extensive results contribute to underscore several critical factors such as data, architecture, and training, and initialization. These findings would be helpful for readers.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank the reviewer for their feedback on our paper.\n\n1. **Have you ever tried out any multilingual settings including more than 2 languages in total?**\n\nWe conducted new experiments by adding French, Danish, Hungarian, and Slovak simultaneously to Gemma 2B. The training is done with the same hyperparameter setting as in Table 3, over 100,000 steps, using 10% English data and the four languages in equal parts. We report the mean task performance for each language.\n\n| Method | EN | FR | DA | HU |SK|\n|-|-|-|-|-|-|\n| Base | 53.0 | 44.2 | 37.6 |33.1 | 34.6|\n| Finetuning | 48.6 | 47.3 |43.3 |38.3| 40.7|\n| Lora  | 50.3 | 45.8 |41.5 |37.2|38.7|\n| Adapters  | 50.9 | 44.9 |41.4 |37.1|38.2 |\n| Ours | 52.4 | 46.0 |41.2 |36.7|39.1|\n\n2. **As another backbone model option, you could also use a multilingual LLM to assess the effectiveness of your proposed approach. I am curious to see how sensitive the proposed approach is to hyperparameter selection in such a complex multilingual setting.**\n\nGemma was trained on a small amount of multilingual data (although the goal was not to reach SOTA performance on multilingual tasks). In particular, it obtains non-random performance on the multilingual evals we consider. Do you mean to assess the effectiveness of our approach to preserve performance on multilingual tasks?\n\n3. **Have you ever tried to employ other languages with different scripts [...] ?**\n\nWe conducted experiments by adding Japanese to Gemma-2B. We extract data from CommonCrawl, and process it with the same pipeline as the rest of the multilingual finetuning datasets. The hyperparamter setting is the same as in table 3.\n\n| Method |Forgetting (EN) | Learning (JA) |\n|-|-|-|\n| Base |53.0|38.8|\n| Finetuning |48.0|46.5|\n|Lora|50.1|44.2|\n|Adapters|51.7|44.5|\n|Ours|52.6|45.3|", "claims_and_evidence": "Experimental results  in Table 1 show that the proposed approach work well while balancing the existing domain/language (English) and the new domain/language (French). They deliberately explored the hyperparameter in the preliminary experiments.", "methods_and_evaluation_criteria": "The results in the downstream tasks do not seem to give consistent improvement across different tasks, while the proposed approach achieves the best or equivalent performance against the well-known approaches such as finetuning, LoRA, and Adapters. \n\nI was wondering how robust the proposed approach as the number of new domain/language increases. Have you ever tried out any multilingual settings including more than 2 languages in total?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "As another backbone model option, you could also use a multilingual LLM to assess the effectiveness of your proposed approach. I am curious to see how sensitive the proposed approach is to hyperparameter selection in such a complexed multilingual setting.\n\nIn the experiments, the targeted languages are mostly European languages with shared alphabetical scripts. Have you ever tried to employ other languages with different scripts such as Arabic and Chinese?", "relation_to_broader_scientific_literature": "The trade-off issue, between model performance in the new domain vs in the original domain, is very crucial when extending the LLMs. This approach sheds another light on this direction, achieving slightly better performance against the other techniques such as finetuning, LoRA, and Adapters.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Please see my comments in the sections above.", "other_comments_or_suggestions": "Please see my comments in the sections above.", "questions_for_authors": "Please see my questions in the sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o5yrc8OQBk", "reviewer": "Reviewer_CFSi", "summary": "This paper presents Neutral Residues, a method for extending large language models (LLMs) to new domains while mitigating catastrophic forgetting. The proposed method builds upon adapter-based techniques (to add extra capacity to the model), introducing architectural modifications with parallel gated adapters, regularizes adapter outputs to produce near-zero activations on the original domain, and implements a low-variance initialization strategy (based on He's initiallization) to improve adaptation stability. \nThese techniques aim to optimize the trade-off between learning new knowledge and retaining previous capabilities. \nThe paper primarily evaluates its approach in the context of multilingual adaptation (from English to other languages), demonstrating that Neutral Residues effectively achieve the best trade-off between minimizing forgetting and improving the target task compared to finetuning, LoRA, and standard adapter baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank the reviewer for their feedback on our paper.\n1. **Only the English-pretrained model is used. Also, the target languages should be more diverse with the language taxonomy.**\n\nOur experiments are also conducted on Gemma, which was trained on a small amount of multilingual data (although the goal was not to reach SOTA performance on multilingual tasks). In particular, it obtains non-random performance on the multilingual evals we consider.\n\nWe also ran new experiments by adding Japanese to Gemma-2B. We extract data from CommonCrawl, and process it with the same pipeline as the rest of the multilingual finetuning datasets. The setting is the same as in table 3.\n\n| Method |Forgetting (EN) | Learning (JA) |\n|-|-|-|\n| Base |53.0|38.8|\n| Finetuning |48.0|46.5|\n|Lora|50.1|44.2|\n|Adapters|51.7|44.5|\n|Ours|52.6|45.3|\n\n\n2. **Ablation studies and analyses are conducted with only English-French pairs**\n\nUnfortunately, due to compute constraints, we could not run ablations on all language pairs.\n\n3. **Broader applications (beyond multilingual transfer) should be discussed and evaluated for stronger claims**\n\nWe trained gemma 2B on math datasets ([OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2) and [MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA))  over 50000 steps with a batch of 4096*64, lr=5e-5 for all settings and the coefficient of the L1 norm for *ours* remains 0.01.\n\n| Method | EN | GSM8K|\n|-|-|-|\n| Base | 53.7 | 20 |\n| Finetuning | 45.5 | 58.4 |\n| Lora  | 43.7 | 57.6 |\n| Adapters  | 48.1 | 47.6 |\n| Ours | 48.1 | 54.4 |\n\n4. **L1 loss: How does this loss work if the two languages are close, e.g., English and German? May this loss hurt the learning of the target domain?**\n\nWe conducted experiments by adding German to EN-LM-1B. We extract data from CommonCrawl, and process it with the same pipeline as the rest of the multilingual finetuning datasets. The model is trained with 20% of extra learnable parameters and the remaining hyperparameters are similar to table 8. Here are the results.\n\n| Method |Forgetting (EN) | Learning (DE) |\n|-|-|-|\n| Base |47.0|32.0|\n| Finetuning |38.4|44.1|\n|Lora|43.0|41.2|\n|Adapters|44.5|41.9|\n|Ours|46.7|41.0|\n\n5. **Forgetting: Can we evaluate the models on the pretrained data distribution where the pretrained model probably works the best?**\n\nThanks for the suggestion. As we do not have the pretraining distribution for Gemma, we can only run this for the LM-EN-1B model. We compute the perplexity on the pretraining distribution for models from table 7 (lr=2e-4 for all): \n\n| Method |Forgetting (EN) |\n|-|-|\n| Base |0.781|\n| Finetuning |0.937|\n|Lora|0.885|\n|Adapters|0.821|\n|Ours|0.796|\n\nAs we can observe, this does not change the conclusion, compared to using PubMed to measure forgetting. We will add these results to the appendix.\n\n6. **Only one main backbone architecture (transformer) is tested, so results may not generalize to others, like vision models.**\n\nThanks for the suggestion: exploring architecture different from transformer would be interesting, but due to time constraints, we leave it for future work.\n\n7. **Motivation: Can you elaborate or provide a simple example for which LoRA doesn't add capacity and significantly affects the downstream performance**\n\nTable 8 shows that downstream performance on French with Lora was significantly lower than that achieved with finetuning when training EN-LM-1B on French. This was particularly noticeable when only a few extra learnable parameters were added.\n\n8. **Why do you choose L1 (sparsity) over other losses, e.g., L-2 or L-infinity [...]?**\n \nWhen using the L2 loss, the model still exhibited significant forgetting. This occurs because, as the outputs of the residual blocks approach zero, the gradients of their L2 loss become small, making them insufficient to effectively drive the outputs toward zero. In contrast, the L1 loss maintains constant gradients, enabling it to push outputs closer to zero more effectively. We also explored a normalized L2 loss, which mitigates the issue of small gradients. However, preliminary experiments did not show significant improvements, so we ultimately chose the L1 loss.\n\n9. **Can you point out the results between applying LoRA on FFN and attention for the tasks?**\n\nWe trained EN-LM-1B on French data using the same hyperparameter settings as in Table 8, with an additional 20% of learnable weights. LoRA was applied either to the FFN, the attention layers, or both. Our results indicate that applying LoRA only to the FFN is better than applying it solely to the attention layers and achieves similar performance to using it on both.\n\n| Method | Tasks  |||  Perplexity ||\n|-|-|-|-|-|-|\n|  | EN | FR || EN| FR |\n|Attn|43.4|39.5|| 0.710 | 0.857|\n|FFN|43.4|40.9|| 0.730 | 0.819|\n|Both|43.4|41.0|| 0.725| 0.824|", "claims_and_evidence": "**Main claim**: Neutral Residues improves the trade-off between learning and forgetting.\n- This claim is supported by findings in Tables 2,3 showing comparable performance in English to the pretrained model and high downstream task performance in target languages (e.g., MMLU, ARC, HellaSwag)\n\nThe effect of the proposed components, including ReLU-based gating, Low-variance initialization, and local loss, are validated with ablation studies shown in tables 4, 5, and 6, mostly with English-French pairs.\n\n**Sufficiency of evidence**: While the results are clear and supportive of the claim, it needs more evidence from diverse sets of language pairs and potentially other domains and tasks. In particular,\n* Only the English-pretrained model is used. Also, the target languages should be more diverse with the language taxonomy\n* Ablation studies and analyses are conducted with only English-French pairs, which are not sufficient for strong claims.\n* Broader applications (beyond multilingual transfer) should be discussed and evaluated for stronger claims.\n* l1 loss: How does this loss work if the two languages are close, e.g., English and German? May this loss hurt the learning of the target domain?\n* Forgetting: Can we evaluate the models on the pretrained data distribution where the pretrained model probably works the best?\n* Version of LoRA and Adapters: There are subsequent works of these methods that show advantages in domain adaptation, e.g., DoRA (https://arxiv.org/abs/2402.09353), QLoRA (https://arxiv.org/abs/2305.14314).", "methods_and_evaluation_criteria": "**Method**: \nThe method is sound, backed by intuitions, empirical observations, and evidence.\nThese assumptions are somewhat validated with empirical validation (though only on a pair of multilingual transfer).\n\n**Evaluation**:\nThe experiments are well-structured, using Gemma-2B, EN-LM-1B as backbone models and comprehensive benchmarks (perplexity + downstream tasks)\nThe ablation studies for gating strategies, initialization, and loss functions are sensitive.\n\n*Limitations*:\n * The datasets are primarily multilingual, so performance on other knowledge domains is unclear.\n * Only one main backbone architecture (transformer) is tested, so results may not generalize to others, like vision models.", "theoretical_claims": "No claim", "experimental_designs_or_analyses": "Yes, I checked and validated all sections in the experiment, from 4.2 (preliminary analysis) to the ablation studies.", "relation_to_broader_scientific_literature": "This paper contributes to addressing catastrophic forgetting in model extension, which is highly relevant for the current research of efficient fine-tuning of large foundational models, given the growing cost of retraining foundation models.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths**\n- The studied problem is interesting and important\n- The problem and method are well-articulated and presented with backed evidence.\n- The experiment designs are thorough for understanding components of the method.\n\n**Weaknesses**:\n- Scalability: The method often adds 20% extra parameters, which significantly increases inference costs.\n- Results focus on multilingual adaptation, so effectiveness on non-linguistic tasks (e.g., code, biology) is unknown.\n- The paper needs more extensive evaluation for stronger claims.", "other_comments_or_suggestions": "Please see the comments above, including additional tasks and languages.", "questions_for_authors": "Together with the questions above, I have several questions:\n\n1. Motivation: Can you elaborate or provide a simple example for which LoRA doesn't add capacity and significantly affects the downstream performance?\n2. For the local loss: Why do you choose l1 (sparsity) over other losses, e.g., L-2 or L-infinity (for zero out all), as the L1 is generally considered harder for optimization?\n3. For LoRA, can you point out the results between applying LoRA on FFN and attention for the tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes "Neutral Residues," an adapter-based method aimed at extending LLMs to new domains/languages while mitigating catastrophic forgetting of the original task. The approach combines parallel gated adapters, data replay, L1 regularization for neutrality on original data, and specific initialization. Initial reviews were split between weak accept and weak reject, but the authors provided a thorough rebuttal with significant new experiments (multi-language, non-European script, math domain) and analyses addressing key concerns about scope and component contributions. One reviewer explicitly raised their score to weak accept post-rebuttal, resulting in a majority leaning towards acceptance. While some concerns about incremental novelty and scalability remain, the demonstrated empirical success in balancing learning and forgetting across several settings makes this a potentially useful contribution. The only rejecting reviewer (Vwee) initially provided a short review with few general questions raised. Those seem to have been addressed by the authors, yet the reviewer did not comment on their responses with additional questions, simply acknowledged them.

**Strengths:**
* Addresses a crucial and timely problem: Efficiently extending LLMs without catastrophic forgetting (5QgN, Vwee, CFSi).
* Demonstrates clear empirical gains and achieves a good trade-off between preserving original domain performance and learning the new domain/language across multiple experiments and metrics (JvbR, 5QgN, Vwee, CFSi).
* Methodology is generally well-explained, and the experiments, including preliminary studies and ablations, are relatively thorough and convincing within the tested scope (JvbR, 5QgN, Vwee, CFSi).
* The core idea of enforcing neutrality via L1 loss on adapter outputs for original domain data is intuitive and elegant (5QgN).
* The paper is generally well-written and easy to follow (JvbR, CFSi).

**Points for improvement:**
* **Limited Experimental Scope/Generality:** While the authors added new experiments in the rebuttal (multi-language, Japanese, math), the evaluation still primarily uses two base models and hasn't explored sequential adaptation, diverse model architectures (beyond transformers), or a wider range of domain types. (Initial concern: 5QgN, Vwee, CFSi; Rebuttal: Added experiments; Acknowledgment: Partially addressed for 5QgN, acknowledged by Vwee/CFSi).
* **Incremental Novelty:** The approach combines several existing techniques (adapters, gating, data replay, specific initializations). The main novelty lies in their specific combination and the L1 regularization strategy, which some reviewers felt was an incremental improvement. (Initial concern: JvbR, 5QgN; Rebuttal: Argued interplay is novel; Acknowledgment: Acknowledged by JvbR/5QgN, likely remains a valid point).
* **Insufficient Ablation/Analysis:** Although new ablations and analyses were provided (component comparison, L1 norm analysis, pretraining data perplexity), some desire for deeper theoretical insight or ablations across more diverse settings might remain. (Initial concern: JvbR, 5QgN, CFSi; Rebuttal: Added analysis/experiments, pointed to existing tables; Acknowledgment: Partially addressed for 5QgN, acknowledged by JvbR/CFSi).
* **Scalability:** The method adds a noticeable parameter count (~20%), potentially increasing inference costs compared to methods like LoRA (though less than full fine-tuning). (Initial concern: CFSi; Rebuttal: Not directly addressed w.r.t inference cost; Acknowledgment: Acknowledged by CFSi).
* **Marginal/Inconsistent Improvements:** Gains over baselines, particularly on downstream tasks, were sometimes seen as marginal or inconsistent, although the method generally showed the best performance preservation. (Initial concern: JvbR, Vwee; Rebuttal: Focused on trade-off; Acknowledgment: Acknowledged by JvbR/Vwee).

In light of the above, the AC recommends acceptance.
Decision: Accept (poster)