Memory Efficient Continual Learning with Transformers
Authors: Beyza Ermis
Giovanni Zappella
Martin Wistuba
Aditya Rawal
Cedric Archambeau
Venue: NeurIPS 2022 Accept (2022)
Keywords: continual learning
transformers
ViT
BERT
text classification
image classification
adapters

Abstract: In many real-world scenarios, data to train machine learning models becomes available over time. Unfortunately, these models struggle to continually learn new concepts without forgetting what has been learnt in the past. This phenomenon is known as catastrophic forgetting and it is difficult to prevent due to practical constraints. For instance, the amount of data that can be stored or the computational resources that can be used might be limited. Moreover, applications increasingly rely on large pre-trained neural networks, such as pre-trained Transformers, since compute or data might not be available in sufficiently large quantities to practitioners to train from scratch. In this paper, we devise a method to incrementally train a model on a sequence of tasks using pre-trained Transformers and extending them with Adapters. Different than the existing approaches, our method is able to scale to a large number of tasks without significant overhead and allows sharing information across tasks. On both image and text classification tasks, we empirically demonstrate that our method maintains a good predictive performance without retraining the model or increasing the number of model parameters over time. The resulting model is also significantly faster at inference time compared to Adapter-based state-of-the-art methods.

Memory Efficient Continual Learning with Transformers
Beyza Ermis Amazon Web Services
Giovanni Zappella Amazon Web Services
Martin Wistuba
Amazon Web Services
ormibeyz@amazon. com
zappella@amazon.cm com
marvistu@amazon.com
Aditya Rawal
Cédric Archambeau Amazon Web Services
Amazon Web Services adirawal @amazon. com
cedrica@amazon.co com
Abstract
In many real-world scenarios, data to train machine learning models becomes available over time. Unfortunately, these models struggle continually learn new concepts without forgetting what has been learnt in the past. This phenomenon is known as catastrophic forgetting and it is difficult prevent due to practical constraints, For instance, the amount of data that can be stored or the computational resources that can be used might be limited. Moreover, applications increasingly rely on large pre-trained neural networks, such as pre-trained Transformers, since the resources or data might not be available in sufficiently large quantities practitioners to train the model from scratch. In this paper, we devise method to incrementally train a model on a sequence of tasks using pre-trained Transformers and extending them with Adapters. Different than the existing approaches, our method is able to scale to a large number tasks without significant overhead and allows sharing information across tasks. On both image and text classification tasks, we empirically demonstrate that our method maintains good predictive performance without retraining the model or increasing the number of model parameters over time. The resulting model is also significantly faster at inference time compared to Adapter-based state-of-the-arr methods.
Introduction
Transformers [57 e.g. BERT 12]. have shown their effectiveness in various natural language processing (NLP) tasks such classification (25). Natural Language Inference [41 39]. and Question Answering [16]. Inspired by this achievement, some pioneering works have recently been introduced adapting Transformers architectures to Computer Vision (CV). Vision Transformers [13] 55 showed that a pure Transformer applied directly to sequence of image patches can perform well image classification tasks. Besides, some recent studies (67 4 30 showed that Transformers generalized to new domains given only few samples. Transformers show a great ability to learn complex concepts but when confronted with sequence of different tasks they tend to "overwrite" the previously learnt concepts. In general, deep networks suffer heavily from this phenomenon, called catastrophic forgetting (CF) (35), impeding continual or lifelong learning. In the last few years, growing body of works attempted to tackle CF in continual learning (CL) 14 19 27 46 63, 65 but most of them are not able meet the scale or accuracy requirements of real-world applications. Moreover, adapting large-scale pre-trained Transformer models to downstream tasks via finc-tuning the method of choice in NLP applications, posing the need for methods that can directly work with pre-trained models instead of requiring the training of new model from scratch [5]
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this work, we tackle both text and image classification problems in a setting where the number of tags or classes associated to the input data grows over time. In fact, the ability continually extend the set of tags or classes used to categorize the content is major problem in many applications, For example, newspapers can tag news according to topies of interest such as "sport", "politics", "food" by using a pre-trained language model and refining it using few hundred pre-tagged articles. New tags may appear over time, for example "COVID-19" was a completely unknown news category in 2019 but appeared frequently since it is emerged. In these cases, retraining models from scratch is often impractical and can lead to inconsistencies in the labeling when compared to the one provided by the previous model. In particular, we focus incrementally extending classifiers based on pre-trained Transformer models given their ubiquity in NLP and the growing interest in CV.
To address the issue of incremental fine-tuning of pre-trained Transformers in the sequential learn- ing setting without CF, we propose Adaptive Distillation of Adapters (ADA). ADA leverages Adapters [20). specialized neural network module that adds new parameters to the neural net- work and distillation mechanism to consolidate the new information with the previously learnt knowledge in fixed amount of parameters with little amount of forgetting. This method allows the user to control the memory consumption, while retaining state-of-the-art performance when running the algorithm sequences of tens of tasks. This tight memory control is important in industrial applications. The alternative, a model growing in size with the number of tasks, would require a change in hardware to adapt to the growing memory requirements of the deployed model. This would be problematic since a practitioner will incur into higher risk of system instability and be forced to make conservative hardware choices.
The main contribution of our work ADA, an algorithm that can achieve high predictive performance on both text and image classification in different continual learning scenarios. ADA also provides lower inference time and uses an order of magnitude fewer parameters than state of the art methods such AdaptersFusion [39]. Additionally, we implemented Adapters for vision Transformers and empirically demonstrated their effectiveness.
2 Related Work
were proposed for fine-tuning of pre-trained language models and were studied for the multi-task setting. AdapterFusion [39 provides state-of-the-arr performance by composing the pre-trained Adapters and it can simply be repurposed for preventing CF in CL by learning one Adapter for every new task. While it has been shown that the number of additional model parameters per Adapter is significantly smaller than the number of parameters used in the pre-trained model [20] (e.g., 3.6% of the parameters of the pre-trained model), since both Adapters and AdapterFusion require to store all the model parameters, the memory consumption increases rapidly with the number of tasks. In the case of a model being trained on 30 tasks, we would have to add more parameters than the number of pre- pre-trained Transformer parameters (details in Section 5). The linear increase in memory and storage consumption making the method unsuitable for CL.
Adapters 20
and incremental learning 64 for NLP and CV 30 [13]. Pasanuru al. (38 focus the few-shot setting where only few data points are available for each task. Ke et al. (26] proposed an architecture to achieve both CF prevention and knowledge transfer. This method has some similarity to AdapterBERT [20] since they insert CL plug-in module in two locations in BERT. A CL-plugin is a capsule network (50) that uses one separate capsule [8] (2-layer fully connected network) for each task, and like Adapters, memory increases linearly over the time. In addition, this algorithm requires to learn task masks to address knowledge transfer, which is costly to compute. Among those recent works, only few (3 [5] have applied the Transformers architecture to CL on image datasets. In [30], for each new task, the model copied and fixed be used as the teacher model in the distillation phase. The student model is trained on both new task samples together with the knowledge distillation loss that uses samples from old tasks which stored in the rehearsal memory. In [15), the authors aim to learn a unified model that will classify an increasingly growing number of classes by building upon a new architecture. However, they need to train a new Transformer, where the process is very costly and contrast with our goal of using public pre-trained models. To the best of our knowledge there is no method able to leverage public pre-trained Transformers while keeping the number of model parameters constant and retaining state-of-the theth-artt predictive performance.
Recent work studied catastrophic forgetting 53 10 25 33

---

3 Problem Setup and Preliminaries
Problem Setup. A sequence of classification tasks (Ti...., Tr are given where each task T contains a different set of data sample (text or image)-label training pairs (#i: Vi.a) and contains e new classes namely Y; = {Y,² ...., Y; with examples for each new class. The goal of the learner is to learn a set of parameters Ô such that N Eie(I....NY (7s)( Ô) is minimized. The task
identifier is provided to the learner with every new batch of data. Moreover, in our specific case, Ô is composed of a set of parameters θ provided by a pre-trained model and, depending on the algorithm, some additional parameters which need to be learned for each specific task. In its simplest case, this additional set of model parameters can just be a head model, but some algorithms use significantly more elaborate functions. the case of the labeling application described in Section | each task represents a label and the learner creates a new binary classifier for each label.
For the training of task T,, the learner can only access the newly added examples and label names in this task. To evaluate the learner, the test data consists of examples across all the previous tasks, where the potential label space for the test example is Y U Y2 U... y All methods that we define in the following sections receive as input a pre-trained model fe(.), e.g., BERT 12. that is able to extract high quality representations from the input data.
Adapters. Adapters were proposed by 20 as an alternative to fine-tuning in NLP. They add new modules between layers of pre-trained network called Adapters. These modules are feed-forward layers that project the original feature size to a smaller dimension and projects them to the original size thereafter, ensuring that the number of parameters stays substantially small compared the original model. (See Appendix AA.4 for the details of the Adapter architectures.) Adapters share the pre-trained model parameters O across all tasks and introduce a small number of task-specific parameters D without affecting previous ones. The model is initialized with parameters of a pre- trained model O For each of the task {l...., ]) where N is the total number of tasks, a set of new and randomly initialized Adapter parameters & are introduced. The parameters Q are fixed and only the parameters t are trained when a new task added. This makes possible to train Adapters for all N tasks, and store the corresponding knowledge in designated parts of the model. The objective for each task E ,I,...,NN N) is of the form: D; +- arg ming Li (D O, Φ)
AdapterFusion [39]. has been proposed to mitigate the lack of knowledge sharing across tasks. It works in two phases: i) in the knowledge extraction stage, adapters, which encapsulate the task- specific information, are learnt for each of the N tasks; while ii) in the knowledge composition stage, the set of N Adapters are combined by using additional parameters V. The additional parameters V for task are defined as: V, + arg ming LD,;; O 4i,..., 4. V). While this provides good predictive performance, in the CL setting, new tasks are added sequentially and storing a large set of Adapters 11...., &x is practically infeasible.
4 Adaptive Distillation of Adapters (ADA)
To address the issues we mentioned in the previous sections, we propose Adaptive Distillation of Adapters (ADA). ADA keeps a fixed amount of Adapters in memory and takes transferability of representations into account to effectively consolidate newly created Adapters with previously created ones. ADA works in two steps: i) it trains a new Adapter and classification head, which we refer as the new model, using the training dataset of the new task; ii) consolidates the old model with the new model. To better control the memory usage, ADA has fix budget for the number of Adapters K that are stored in a pool of old models. In the consolidation phase, the algorithm selects one of the models in the pool using scores that quantify the transferable information contained in the representations they provide. In the following sections, we explain the components of ADA and how they work.
4.1 Distillation of Adapters
For each new task Tn. the Adapter parameters 4,, are added to the model, while the pre-trained model parameters O are kept frozen and are never changed. Only the task-specific Adapter parameters & and the head model parameters hn are trained for the current task. The model fn (2:; O 4n, hn), with parameters O 2, and h, is called the new model. The head model parameters are fixed after training
3

---

the new model and they are not updated during model consolidation. When a prediction for a task T; required, the corresponding Adapter Py(0) and head model h; called. y is mapping from the task id to the corresponding Adapter in the pool or to the newly trained Adapter. We abuse notation defining/ f as the function returning the output (logits) on all tasks:
f(z; +, = [f(x; O &
f(;; θ 17(n-1), hn f(;; O 2 p)(n),
(I)
For the consolidation step, an Adapter from the pool T is selected as explained in Section 4.2 and new collection of Adapter pool &' is created where the selected Adapter is replaced with te. &c denotes the consolidated model parameters and preliminarily the parameters are randomly initialized.
Similarly, copy of y is created to map the old tasks that are associated to the selected Adapter and the new task n to O.. The consolidation then has the following objective:
Daiveittl
min 4
Σ
2, f(x;; O & h))
(2)
Ditistitt
where Daistsul denotes the unlabeled training data used for distillation, and the distillation loss is computed as the difference between the logits produced by the existing specialist models denoted by f( θ, $ h) and the consolidated model denoted by f( O & ht) based on L2 loss. After &. has been trained, 4 is swapped with &' This is high-level view of the mechanism, our implementation is optimized to avoid copying models when not necessary.
This procedure follows the double distillation loss [69] which is originally proposed for class incremental learning to train a new Adapter that used with the pre-trained model to classify both old and new tasks. The main idea is first training separate model for the new class(es) using labeled data, and then combining the new and old models using unlabeled distillation data via a double distillation training objective. We generalize this solution to our case where we have a set of teacher models kept in Adapter pool & and train a student model 4... Double distillation procedure and the alternative solutions for distillation are discussed in Appendix A. but this solution was the best performing one our experiments.
While several different data sources can be used to populate the buffer, such as using auxiliary external data 669 or generating synthetic data (g. in this work we populate the buffer using covariates from previous tasks selected with Reservoir Sampling (58). This simple mechanism may not be the most effective, but it will guarantee that no advantage is given to ADA in the experimental comparison.
4.2 Adapter Selection for Distillation
In the previous section, we assumed the Adapter to be consolidated as given but ADA keeps a pool of Adapters and the selection of the Adapter to be distilled is important part of the algorithm. In fact, our empirical observations show that a random selection of the Adapter provides poor performance (see Section [.3). The intuition behind head for every task is created, we can assume that when the features provided by the associated Adapter are highly informative, the updates (i.e., the gradients applied) will be small. At the same time, training new head with every Adapter in the pool in order to observe which one is the most effective would increase the amount of computation required and significantly impact the usability of the method. The problem of computing the information carried by representation in an efficient manner has been already studied in the transfer learning community 56 54].
selection mechanism is the following: since specialized
While, the aim of that research is completely different and, to the best of our knowledge, there is no clear relation between transferability and forgetting, the mathematical foundation of this work closely related to our intuition. In fact, scores like TransRate (22 employ the mutual information between the features provided by a pre-trained model and the target labels for the task at hand. When the mutual information is high, the transferability is high. More specifically, the knowledge transfer from a source task T, to a target task T, measured as:
TrR (f, 4 (()))) = ((Z))H(Z))),
(3)
where Y are the labels of target examples and Z = (X; (=)) are features of them extracted by the pre-trained model and the Adapter associated to the source task.
TransRate is not the only score designed to quantify transfcrability between pre-trained model and new task: Log Expected Empirical Prediction (LEEP) (36) is a well-known alternative. Also in

---

this case, the score was designed with a different application in mind, but it leverages the conditional distribution of the target label given the source label to quantify the how informative the information provided by the source model is. Specifically, LEEP three steps method. At Step dummy label distributions of the inputs f(;; θ Py(t) he) in the target data set D. At Step 2,
computes
computes the empirical conditional distribution P(ylz) of target label given the source label z. At Step 3, it computes LEEP using 0( & P7))), ha) and P(y]z):
LU Pr(s); h.).
1
log Σ
P(vl=)((X;O) (r0)).k))
(4)
LEEZ
where is a dummy label randomly drawn from O (x)) h2) and y is randomly drawn from P(ylz). We selected TransRate and LEEP for their simplicity and their ability to provide quantification without training but practitioners can replace these scores with different ones as they fit.
4.3 Algorithm
ADA is detailed in Algorithm 1 The graphical workflow of the algorithm is shown in Appendix 7 For every new task, the algorithm trains a new adapter and head model (called 4, and h,). If the adapters pool did not reach the maximum size yet (controlled by K), it just adds it to the pool. the pool reached the maximum size, the algorithm is forced to select one of the adapters already in the pool and distill it together with the newly trained one. In order to select which adapter distill, ADA uses the transferability scores (e.g.. LEEP or TransRate). Once the adapter the pool with the highest transferability score (called fy-) is identified, it consolidates that adapter and the newly trained one into a new adapter and replaces the old one present in the pool. In order to be able to make effective predictions, the algorithm also keeps mapping of which adapter in the pool must be used combi- nation with each of the task-specific heads.
Algorithm 1 Adaptive Distillation of Adapters (ADA)
Require: é: pre-trained model. K: adapters pool size
Freeze @ and create y Map()
for n + to
task received
Initialize 4.,
Process Tu. train new model fo (a:; 6 h.,) Sample from T., and add to distillation data Daistil if S then
Store in the pool
else
j <- arg mrg,a maxjemn K) TranScorooTa..e
Add (n, ty
Consolidate model:
fi+ Distill( Daiscitt)
end if
Serve predictions for any task S using E end for
Distill(f. fy. Dases)):
Get soft targets ji from model fi with Doistat Get soft targets fúj from new model fú with Dasatat
Initialize &
Compute distillation loss and train model f(x; O, b.) de- fined in Eq. return f
5 Experiments
In this section, we empirically validate our adapter distillation approach on text and image classifi- cation tasks and show that ADA achieves similar performance AdapterFusion while consuming significantly less memory. We dedicate Section 5.3 to ablation studies providing further insights into the mechanisms implemented in ADA and their contribution.
Datasets and experimental setup. We use three text datasets for multi-label text classification: Arxiv Papers [66] (paper classification), Reuters (RCV1-V2) 29 (news classification), Wiki-30K [7l)
(Wikipedia article classification) and two dataset for image classification: CIFAR100 28 and
MinilmageNet (49). Details about the datasets are given in Appendix AA.3]
For the multi-label text classification experiments, we first sample a sequence of labels from the label space. Then, we create balanced binary classification task for each label by sampling the same amount of positive data points from the label considered and negative data points from the labels preceding the current one in the sequence. After splitting the data in training and test set,
5

---

provide the algorithm with the training set and subsequently measure its performance on the test set. The algorithm never observes any data point in the test set and, more generally, every data point in the dataset is used only once. For Arxiv Papers and Reuters datasets, we created 20 tasks and for Wiki-30K 60. We fixed the number of training samples per task to 100. The test set consists of 40 data points on Reuters and of 100 data points on Arxiv and Wiki-30K.
For image classification, we design two scenarios. In the first scenario, each new task is balanced binary classification problem. Each class can be selected to be the positive class only once. In the second scenario each task a balanced multi-class classification problem with 5 classes. In both cases we provide the learner with 50 data points per class both at training and test time: in the first scenario each task will have a training set of 250 data points and in the second case of 100 data points. The total number of tasks is fixed to 20 for both scenarios. The distillation memory size is fixed to 1000 for Wiki-30K which has a larger number of tasks, and to 500 for the others.
Metries. In (34], three metrics that we discuss in the following are defined to evaluate the performance of CL method. We use these metries to evaluate our methods. It is considered that we have access to a test set for each of the N tasks in {T',.. its test performance are evaluated on all N tasks. By doing so, a matrix R& A is constructed where Ra, is the test classification accuracy of the model on task 73 after observing the last sample from task Ti Letting b be the vector of test accuracies for each task at random initialization, the three metrics are defined: i) Average Accuracy N Σi= RN, ii) Backward Transfer (BWT)= N-1 Ex=1 RN, - Ra, and iii) Forward Transfer (FWT)= = N EN R4-1, - b,. (The larger these metrics, the better the model.) All the results in this section are averaged over runs.
Ivy J. After the model finishes learning about the task T;,
Baselines. We compare ADA the following baselines. 1) Fine-tuning head model (B1): We freeze the pre-trained representation and only fine-tune the output layer of each classification task. The output layer is multiple-head binary classifier that we also use for the other methods. 2) Fine tuning the full model (B2): We fine-tune both the pre-trained representation and the output layer for each classification task. 3) Adapters [20]: We train and keep separate Adapters for each classification task as well as the head models. 4) AdapterFusion [39]: It is two stage learning algorithm that leverages knowledge from multiple tasks by combining the representations from several task Adapters in order to improve the performance on the target task. This follows exactly the solution depicted in Section Bl 5) Experience Replay (ER) 48 ER is a commonly used baseline in Continual Learning that stores a subset of data for each task and then "replays" the old data together with the new one to avoid forgetting old concepts. (II) propose to use such a memory module for sparse experience replay and local adaptation in the language domain. This method stores all training examples, in order to achieve optimal performance. To make this method comparable with adapter-bused methods, we freeze pre-trained representation, add a single adapter parameters & and train the adapter by replaying examples from old tasks while training using data from the new task. In order to keep baselines comparable we assign to ER the same amount of memory is used for the distillation buffer in ADA. In addition to these baselines, we use one special case of ADA with K=1 as a baseline to demonstrate the advantage of effective consolidation of Adapters.
Adapter architectures. We use pre-trained models from HuggingFace Transformers 61 as our base feature extractors. We ran experiments with BERT DistilBERT hasee RoBERTapase for text classification and ViT-B and DeiT-B for image classification. We analyze the cases based on all these models, due to the space constraints, we present BERT base in this section and the rest in Appendix [A5]] BERT model uses 12 layers of Transformers block with a hidden size of 768 and number of self-attention heads as 12 and has around 110 M (440 MB) trainable parameters, For the Adapter implementation, we use Adapter-Hub [40). but no Adapter implementation was available for Vision Transformers. We define our architecture of Adapters for ViT and DeiT in Appendix A.4 An Adapter has simple bottleneck architecture that contains fewer parameters than the attention and the ferward layers. The Adapter size is the prper-parreeter that is tuned and it can be set to {12, 24, 48, 96, 192. 384) for BERT model. For all the methods, we use the same configuration for the Adapters, setting the size to 48. With this setting, an Adapter contains ~ 1.8 M parameters. We also train head model for each task, that has 768 parameters for BERT base (last hidden size of BERTBase x output size, which equals to for binary classification). The tables in Appendix A.5 reports the number of parameters used for baselines and ADA in experiments.

---

Figure 1: Comparison between baselines and ADA on Arxiv, Reuters and Wiki-30K. On top, we report the number of tasks processed the x-axis and report the average accuracy measured the test set the tasks processed on the y-axis, shaded area shows standard deviation. On bottom, we report BWT and FWT.
8 E1
Comparison of the % increase in the number of parameters of baseline methods and ADA on Arxiv, Reuters and Wiki-30K. The predictive performance reported on the y-axis is measured after processing all tasks.
Figure 2:
5.1 Text Classification
Predictive performance. Figure shows the comparison of ADA and the baseline methods. It can be clearly seen that freezing all pre-trained model parameters, and fine-tuning only the head models (B1) led to an inferior performance compared to adapter-based approaches. The main reason that the head models have small amount of parameters train and fine-tuning only the heads suffers from under-fitting. B2 performs good only for first 2-3 tasks, since we keep training the complete model, forgets the previously learned tasks very quickly. As mentioned above, Adapters and AdapterFusion add 1.8 M parameters for each task and train these parameters with new task data, and these parameters are fixed after training. So. they perform well on both new tasks and previous tasks. The results on each dataset confirm this. Both ER and ADA K=1, perform closely with Adapters almost for half of the tasks. The similar behavior of ER and ADA K=1 demonstrates that the distillation with soft labels works well and is almost as good as training with the true labels. Later the performance declines for both methods, because the capacity of the Adapter is exceeded. ADA LEEP and ADA RatesRate results with K=4 Adapters show that selective consolidation of Adapters significantly improves the performance. Their performance on par with AdapterFusion while the number of model parameters is significantly lower. We present BERT base results in this section while the rest is reported in Appendix A.9
We also compute FWT and BWT scores for these methods. We didn't present B and Adapters in the plots, since both FWT and BWT are zero for them. BWT is zero for AdapterFusion, since the fusion parameter is computed with available Adapters, and the Adapters trained later is not used for the previous tasks. ADA-LEEP and -raanRastaeee minimizes negative backward transfer, while showing a positive forward transfer for all datasets.
Memory consumption. Figure 2 shows the increase in terms of percentage in the number of parameters used by each method and their predictive performance. We see that on Wikipedia, 200% of the base model parameters 2200M additional parameters) are added. These results make clear that
7

---

ADA is significantly more efficient in terms of memory usage. It can achieve predictive performance similar to the one of Adapters and AdapterFusion while requiring significantly less model parameters. On Reuters and Arxiv, it can store the parameters of only 5 Adapters (K=4 Adapters in the pool, and one Adapter for new task), against the 20 required by AdapterFusion (on Wikipedia is against 60).
Inference time. When machine learning models are used to power customer-facing web sites, they are often required to provide predictions in a few milliseconds to keep the overall latency within requirements. Moreover, in this kind of application the model will be trained once and make billions of predictions so a reasonable increase in the training time is irrelevant compared to a decrease in the inference time. We report the inference time results of ADA and other Adapter based methods in Appendix [A.6] Results demonstrate that ADA provides sufficiently fast inference for most applications and still offers opportunities to speed it up further, for example by employing smaller pre-trained Transformers (e.g. DistiBBERT, see Appendix A.5).
Training time. Distillation of Adapters brings an extra cost for ADA while learning fusion parameters brings an extra cost for AdapterFusion. Computing transferability takes constant time which is negligible. Distillation costs training an additional Adapter 1.6 % of full fine-tuning time of BERT). Figure T in 6ppendixA.66 reports the average training time comparison on Wiki-30K that is the largest difference with AdapterFusion given larger number of tasks. We can clearly see that the difference is small (ADAis 3.37% more, ADA- ADt-TransRate 5.6% more) while the difference between the inference time is significant.
5.2 Image Classification
For image classification experiments, we add Elastic Weight Consolidation (EWC) 27 an addi- tional baseline since it is widely used in CL literature for image classification. EWC is a regularization- based CL method that assumes that some weights of the trained neural network are more important for previously learned tasks than others. During training of the neural network on a new task, changes to the weights of the network are made less likely the greater their importance.
Ni-imagehet VIT
CIENR100 VIT
NOA 80-4: Twrskos
40e 2-13
4M4 80-4 Tursfon
0.7
01.
0.3
0.3
Tada Proomsed
Tasla Proceaedd
Nitimagshet VIT
CIRM1100
3.0
3.0
104 3-6--
19
21
12
15
Figure 3: Comparison between baselines and ADA with ViT model MinilmageNet and CIFARI00.
Top figures shows the binary, and bottom figures shows the multi-class classification results.
Figure 3 shows the comparison of ADA and the baseline methods. The results show the same behaviour with text classification. B1 leaded to an inferior performance compared to other approaches. B2 performs well only for initial tasks and it forgets the previously learned tasks very quickly. Results confirm there is no forgetting for Adapters and AdapterFusion. Although the careful tuning of regularization coefficient, EWC cannot handle CF, especially for multi-class classification problem. ADA with K=1 shows that distillation alone doesn't prevent forgetting. In almost all cases, ER
8

---

performs on- on-par with ADA K=1, providing evidence that a small amount of memory can actually improve performance compared fine-tuning or regularization, but the improvement is limited and does not last as the number of tasks increases.
ADA-LEEP and ADA-TransRate results with K=4 Adapters show that selective consolidation of Adapters significantly improves the performance, For binary classification, their performance are on par with AdapterFusion while the number of model parameters is significantly lower. For multi-class, their performance slightly declines after certain number of tasks. This is discussed in next section and the main reason that the capacity of the Adapter is exceeded. To validate the interoperability of ADA to different models, we run the same experiments on DeiT model and present the results in Appendix A. space constraints.
5.3 Ablation studies
Comparison with larger distilled models. In Section 5 we compared ADA with the special case of ADA with K=I evaluate the improvement provided by our approach over a distillation-only solutions. We would like to provide additional observations of the superior performance of ADA by comparing its performance with the one of distilled Adapter using more parameters. Specifically, we run an experiment where we compare ADA with K=4 and ADA with K=l as displayed before but in this case the "size" of the Adapter, which is 48 for Size 1, is multiplied by for Size x Adapter to have a comparison where the different methods use the same number of model parameters. Since K=1 is special case where a single Adapter kept in the pool, the transfcrability metric irrelevant and we can see ADA with K=1 as method purely based on distillation like DMC 669
3.00
1.00
0:45
041
6:02
0.00
0:80
ADVE-3 JEKT.
0.25
075
Busiiats 1
0.70
3.
22
52
20.0
0.20 0.
2000
Figure 4: Impact of LEEP and TransRate when the total number of Adapter parameters is same on Arxiv and Reuters.
The results reported in Figure 4 show that ADA can make better use of the model parameters compared to a distillation-only method and that the intelligent selection of which Adapters to distill together makes once again big difference. It is also interesting to observe that the usage of additional model parameters brings a clear advantage but the mixed comparison between the ADA K=4 with random Adapter selection and ADA K=1 with four times larger Adapters leaves some questions open regarding how far distillation can get in this setting. Another finding is that TransRate outperforms LEEP in most cases. It is also demonstrated in the original paper [22] that TransRate has strong correlation to the transfer learning performance and it outperforms LEEP and other metrics employed.
Impact of the Adatters pool size. In our experiments we used fixed number of Adapters in the pool size, but more Adapters can be added to ADA's pool as more tasks are processed. This may actually be the preferred usage in some applications. We already know that having Adapter per task provides good performance and using multiple of them at the same time like in AdapterFusion provides a benefit, but would like to verify the sensitivity to this parameter. The results reported in Figure 5 show a rapidly decreasing added value when the number of Adapters grows, a behavior which aligns well with our practical requirements of keeping the number of model parameters under control when the number of tasks increases. See additional experiments in Appendix A..1
Mixed Data Experiments. We run experiments in a setting where we sample 200 tasks from Arxiv, Reuters and Wikipedia (50/50/100) respectively (the order of the tasks are created randomly). We fixed the number of training and test samples per task to 100. Figure 6 shows that we observe a little saturation only after the 150th task when K=4 and no saturation when K=8, Besides, ADA with TransRate comparable performance with Adapters and AdapterFusion even in complicated setting.

---

0:43
042
000
0.33
0.25
1200 K-2
Str
Tarsiors 8-22 F00-1
Sar
30
25
50
10
20.0
15.0
21.3
20.0
8.20
00
20
53
25
20.0
12.5
15.0
37.3
200
Figure 5: LEEP and TransRate performances when K (1,2,4,8) on Reuters.
Figure 6 also shows the increase in terms of percentage in the number of parameters used by each method and their predictive performance. We see that ~ 330% of the base model parameters are added for Adapters and AdapterFusion. These results make clear that ADA significantly more efficient in terms of memory usage while keeping the comparable performance.
M vod Detapets Merayy Beuserr, Wik
Mised Datisaat LAtris, Ractiers K30
07
ABA 16H12
FEA 00a
120
250
176
155
205
PM
155
Figure 6: (Left) method performance comparisons (Right) comparison of the % increase in the number of parameters of baseline methods and ADA on mix of datasets. LEEP shows a similar performance with TransRate, for the sake of clarity, didn't add it to the figures.
Conclusion
In this paper we presented ADA, a method that allows neural text and image classifiers to learn new classes based preitrrined Transformers while maintaining strict control of the memory usage and reaching state-of-the-ar predictive performance. The method has shown to be effective in different domains and allows users to leverage publicly available pre-trained Transformers for continual classification tasks. We evaluated ADA on different classification tasks and demonstrated that the predictive performance is competitive with state-of-the-an methods which up order of magnitude parameters. ADA also displayed lower latency at inference time and improved data efficiency for some specific settings (see Appendix A.8). Moreover, we empirically demonstrated that Adapters can give good results when used in combination with vision Transformers on CV tasks.
Transformers are very popular, but they are not the only models being widely used in practice. We consider this the main weakness of our approach and we would like to further expand our activity to perform CL on other widely used pre- pre-trained models such ResNet. Addressing multi modal classification using text and images together will the other focus of our future research.
References
[1] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning, arXiv preprint arXiv: 1 1903.08671, 2019.
[2] Rahaf Aljundi, Mareus Rohrbach, and Tinne Tuytelaars. Selfiess sequential learning. 1806 05421, 2018
arXiv preprint
10

---

[3] Yajie Bao, Yang Li. Shao-Lun Huang, Lin Zhang. Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An cooemattionthheoreeee approach to transferability in task transfer learning. In 2019 IEEE International Conference on Image Processing (ICIP), pages 2309-2313, IEEE, 2019.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541, 2006.
[6] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. arXiv preprint arXiv:2004.07211, 2020.
[7] Francisco M Castro, Manuel J Marin-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteck Alahari. End-to-end incremental learning. In Proceedings of the European conference computer vision (ECCV), pages 233-248, 2018.
[8] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Tomr and M Ranzato. Continual learning with tiny episodic memories. 2019.
[9] Akshay Chawla, Hongxu Yin, Pavlo Molchanov, and Jose Alvarez. Date-free knowledge distillation for object detection. In Proceedings of the IEEE/CVF Winter Conference Applicutions of Computer Vision, pages
2021.
[10] Yung-Sung Chuang, Shang-Ya Yu Su, and Yun-Nung Chen. Lifelong language knowledge distillation. arXiv preprint arXiv:2010.02123. 2020.
[11] Cyprien de Masson d' Autume, Schastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. arXiv preprint 1:Xiv:1906.01076 2019.
[12] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.0490 2018
[13] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. /9999 2020.
[14] Arthur Douillard, Matthiuu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In European Conference on Computer Vision, pages 302102.66 Springer, 2020.
[15] Arthur Douillard, Alexandre Ramé, Guillaume Couairon. and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. arXiv preprint arXiv:2111.11366 2021.
[16] Claudio Greco, Barbara Plank, Raquel Fernández, and Raffaella Bernardi. Psycholinguistics meets continual learning: Measuring catastrophic forgetting in visual question answering. arXiv preprint 14X.429906.04229, 2019.
[17] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1xXXi:1503.025555 2015.
[18] Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In International conference on artificial neural networks, pages 44-51. Springer, 201
[19] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang. and Dahua Lin. Learning a unified classifier incrementally via rehalancing. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 831-839, 2019.
[20] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parametcr-efficion transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799 2799. PMLR, 2019.
[21] Wenpeng Hu, Qi Qin, Mengyu Wang. Jinwen Ma, and Bing Liu. Continual learning by using information of each class holistically. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7797-7805, 2021.
11

---

[22] Long-Kai Huang. Ying Wei, Yu Rong. Qiang Yang, and Junzhou Huang. Frustratingly easy transferability estimation, arXiv preprint arXi:22106.0936, 2021.
[23] Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang. and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. arXiv preprint arXiv:2104. 054499 2021.
[24] Khurram Javed and Martha White. Meta-learning representations for continual learning. arXiv preprint urXiv:7905. 12588, 2019.
[25] Zixuan Ke, Bing Liu, Hao Wang, and Lei Shu. Continual learning with knowledge transfer for sentiment classification. In ECML/KKDD (3), pages 683-698 2020.
[26] Zixuan Ke, Hu Xu, and Bing Liu. Adapting bert for continual learning of sequence of aspect sentiment classification tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4746 4755,4755, 2021.
[27] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barrikkk,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (14(13):3521-335) 2017.
[28] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[29] David D Lewis, Yiming Yang, Tony Russell-Rose, and Fan Li. Rev1: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr):361-39) 2004.
[30] Duo Li, Guimei Cao, Yunlu Xu, Zhanzhan Cheng, and Yi Niu. Technical report for ieev 2021 challenge Transformers are better continual learners. arXiv preprint arXiv:2201 049944, 2022.
[31] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEF transactions pattern analysis and machine intelligence, 40 20:222299352947, 2017.
[32] Nelson F Liu, Mau Gardner, Yonatan Belinkov, Matthw E Peters, and Noah A Smith. Linguistic knowledge and transferability contextual representations. arXiv preprint arXiv:1903.0885 2019.
[33] Zihan Liu. Genta Indra Winata, Andrea Madotto, and Pascale Fung. Exploring fine-tuning techniques for pre-trained cross- cross-lingual models via continual learning. arXiv preprint arXiv:2004. 14218, 2020.
[34] David Lapez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30:6467-6476, 2017.
[35] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[36] Cuong Nguyen, Tal Hassner, Matthias Seeger. and Cedric Archambeau. Leep: A new measure to evaluate transfcrability of learned reprosentations. In International Conference on Machine Learning, pages 7294- 7305. PMLR, 2020.
[37] Abiola Obamuyide, Andreas Vlachos, et al. Meta-lcaming improves lifelong relation extraction. 2019.
[38] Ramakanth Pasunuru, Veselin Stoyanov, and Mohit Bansal. Continual few-shot learning for text classifi- cation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 5688-5702, 2021.
[39] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the Europeon Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online, April 2021. Association for Computational Linguistics.
[40] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Tryna Gurevych. Adapterhub: A framework for adapting transformers. arXiv preprint axXiv:2007.07779. 2020.
[41] Jonas Pfeifferr Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. arXiv preprint arXiv:2005.00052 2020.
[42] Jason Phang. Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training intermediate labeled-dan data tasks. arXiv preprint arXiv:1811.01008. 2018
12

---

[43] Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. What to pre-train on? efficient intermediate task selection. arXiv preprint arXiv:2104.08244 2021.
[44] Yada Pruksachatkun, Jason Phang. Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang. Clara Vania, Katharina Kann, and Samuel R Bowman, Intermediate task transfer learning with pretrained models for natural language understanding: When and why does it work? arXiv preprint arXiv:2005.00 00628, 2020.
[45] Joan Puigeerver, Carlos Riquelme, Basil Mustafa, Cedric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. Scalable transfer learning with expert models. arXiv preprint arXiv:2009. 13339,, 2020.
[46] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[47] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint 1800111100.11910, 2018.
[48] David Rolnick, Arun Ahuja. Jonathan Schwarz. Timothy P Lillicrap, and Greg Wayne. Experience replay for continual learning, arXiv preprint 17X1: 2018.
[49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 2203222112222222 2015.
[50] Sara Sabour, Nicholas Frosst, and Geoffrey Hinton. Dynamic routing between capsules. arXiv preprint 17Xiv:1710.09829, 2017.
[51] Jonathan Sehwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Grabska- Barwinka, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress compress: A scalable framework for continual learning. In International Conference on Machine Learning, pages 4528-4537, PMLR, 2018.
[52] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In Proceedings of the IEEE international conference computer vision, pages
2017.
[53] Fan- Keng Sun, Cheng- Hao Ho, and Hung- Yi Lee. Lamol: Language modeling for lifelong language learning. arXiv preprini 1rXiv:1909.03329 2019.
[54] Yang Tan, Yang Li, and Shao-Lun Huang. Otce: A transferability metric for cross-domain cross-task representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15779-15788. 2021.
[55] Hugo Touvron, Matthieu Cord, Mathhijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, RR 2021.
[56] Anh Tran, Cuong V Nguyen, and Tal Hassner. Transfcrability and hardness of supervised classification tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1395-1409 2019.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[58] Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS). 111):377577 1985.
[59] Tu Vu, Tong Wang. Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji. and Mohit lyyer. Exploring and predicting transfcrability across nlp tasks. arXiv preprint Xiv::20005 007700 2020.
[60] Zirui Wang, Sanket Vaibhav Mehta, Barnaáás Póczos, and Jaime Carbonell. Efficient meta fielong-leaaringg with limited memory. arXiv preprint arXiv:2010.02500, 2020.
13

---

[61] Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. Transformers: State-of-the-at natural language processing. Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing:
System Demonstrations, pages 38 45, 2020.
[62] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari. Jason Yosinski, and Ali Farhadi. Supermasks in superposition. arXiv preprint arXiv:2006. 147699 2020.
[63] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 374 1144-32,222 2019.
[64] Congying Xia, Wenpeng Yin, Yihao Feng, and Philip Yu. Incremental few-shot text classification with multi-round new classes: Formulation, datasct and system. arXiv preprint arXiv:2104. 1882, 2021.
[65] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014-3022 2021.
[66] Pengcheng Yang. Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. Sgm: sequence generation model for multi-label classification. arXiv preprint arXiv: 1706::8800004822 2018.
[67] Wenpeng Yin. Jamaal Hay. and Dan Roth. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. arXiv preprint arXiv: 1999000061,, 2019.
[68] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8)3364 372, 2019.
[69] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. itass-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 1131 1140, 2020.
[70] Peng Zhou, Long Mai, Jianming Zhang. Ning Xu, Zaxuan Wu, and Lany S Davis. M2kd: Multi-mode! and multi-level knowledge distillation for incremental learning. arXiv preprint arXiv:1904.01. 2019.
[71] Arkaitz Zubiaga. Enhancing navigation wikipedia with social tags. arXiv preprint 12Xi::1022.5409 2012.
14


Reviews:
[{"review_id": "gv60ctmRV-m", "reviewer": "Reviewer_s76Y", "summary": "This paper extends the work of Houlsby et al. by proposing a method to distill knowledge between adapters in continual learning. The high level idea is to freeze the foundation model's parameters and train task-specific adapters. The mitigate the lack of knowledge sharing between newly learned adapters, the authors propose a distillation method so that adapters can share knowledge and memory does not grow out of bounds.", "questions": "What are some example tasks where this memory reduction is necessary? It's outlined in the introduction, but not explicitly stated.\n\nWhere exactly is the \"distillation\" happening? Which equation, for example.\n\nWhy is knowledge distillation a motivation, but the performance is worse than Adapters? Doesn't this mean that it's not distilling knowledge correctly?", "limitations": "The authors talk about how their work is tied to transformers, but is this true? Can't this be applied for other model types that allow for task heads?", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis paper addresses an important problem in the domain of continual learning. There is typically a tradeoff between learning new tasks well and remembering old tasks. This paper tries to close the gap between this tradeoff by learning new tasks but keeping prior learned tasks in the model's memory. \n\nThe memory efficient aspect is interesting, especially if there is a scenario where the number of tasks grow in orders of magnitude. \n\nThe introduction was well posed, the task setup is nice, and the motivation is clear.\n\n\n\nWeaknesses:\n\nThe method section was unclear. Without reading the prior papers such as Houlsby et al., it's hard to understand what is being proposed in this paper. I think the methods section could be condensed and clarified. \n\nThe exact concept of ``knowledge distillation'' in terms of the method is unclear.  Algorithm 1 contains a Distill() function, but it's not explained in the paper. \n\nThe distinction between \\phi, \\phi^{'}, \\phi_{c} is unclear.\n\nLastly, the results section is not easy to understand. Without reading prior papers, the BWT and FWT metrics are unclear. The figures are somewhat hard to read. Based on the results, it seems like performance is worse using ADA vs vanilla adapters. The only contribution vs adapters is the memory reduction. \n\n\nMinor:\n\n\"This is an high-level\" --> \"This is a high-level\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Sq7rHDwEs6y", "reviewer": "Reviewer_G9iU", "summary": "In this paper, the authors introduce adaptive distillation of adapters, or ADA algorithm, which helps continual learning with transformers be memory efficient and performant at the same time. The work uses Adapters to fine-tune a large, pretrained transformer model to a new task, as opposed to the standard practice of fine-tuning the final layer. However, as Adapters require more parameters compared to a new final layer, the authors introduce ADA to make continual learning more memory efficient. Basically, ADA keeps a fixed stack of K adapters, and when a new Adapter needs to be added to the stack, ADA takes the Adapter that has the most similarity with the new task and distills that with the new task. The authors run experiments on language and vision tasks to show that ADA can perform continual learning well without catastrophic forgetting without using an unbounded number of parameters. Finally, the authors ablate the impact of adapter size and adapter pool size to show their effects on ADA.", "questions": "- What is the relationship between K and the number of tasks at which the adapters get saturated? ADA does not show saturating behavior in the experiments in the paper, but intuitively there should be at some point where it does get saturated.\n- What is the performance, quantitatively, of the random-adapter-distillation variation of ADA? Please include that baseline in the experiments as well, as it is currently we do not see any quantitative comparison between this version of ADA and the one with random replacement.", "limitations": "- The primary limitation of this work is that it is seemingly limited to transformer-based models only, with no clear indication of how it could transfer to other architectures. However, as of now, transformer-based models are enjoying decent success, and thus this limitation is not a major one.\n- The experiments are performed on a handful of tasks, on the order of tens of tasks. It would be informative to see the behavior of ADA in the asymptotic case, where the K adapters are saturated and thus there is a downgrade in the performance. As the paper is presented right now, the readers do not get a balanced picture of where it may fail.\n- ADA chooses the adapter to distill at every step in a greedy fashion, instead of optimizing over all the tasks seen so far. The paper does not analyze the worst-case behavior caused by this greedy choice of distilled adapter, or how ADA may be impacted by an unfortunate ordering of tasks.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the strengths of this paper are as follows:\n- This work introduces ADA, which is a novel algorithm for adapting pre-trained transformers to continual learning settings without incurring significant memory overhead.\n- The algorithm itself is simple with significant practical benefits as well, as shown by the experiments.\n- ADA seems to perform comparatively with baselines while keeping to bounded memory, which is also an impressive feat.\n\nHowever, there are a few places where this paper could improve, especially concerning its method and experiments.\n- Firstly, the paper does not introduce the concept of Adapters sufficiently clearly. The way it is explained right now in the paper is insufficient, in my opinion, for someone that is not already familiar with the concept of adapters. A detailed explanation or a figure to explain the concept of adapters could really go a long way for this.\n- Similarly, important concepts such as double distillation loss is not explained in the main body of the paper.\n- On Figure 2, the X axis could be more informative if the metric was % increase in parameter per task rather than the total number of parameters in the model. The authors talk about this in the sense that 30 adapters roughly equal the size of a pretrained transformers, which I believe communicates the justification for distillation in the first place, and denoting the modified modules' sizes as % size of the base model would also work better here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NwsFN2wJpEx", "reviewer": "Reviewer_rCKB", "summary": "This work presents a method of using adapters to fine-tune pretrained models to downstream tasks in a continual fashion. Instead of adding a new adapter for each new task, the authors define a fixed set of adapters of size K which are repurposed. When the number of tasks are less than K, each new task is trained using a new adapter. For each additional task, first, a new adapter is trained, and then its closest counterpart is selected in the set of K adapters (using TransScore or LEEPl; to measure transferability). The selected adapter is then distilled using a subset of data from all previous tasks that the model has been trained on to prevent forgetting. \n\nThe authors convert multi-label and multi-task classification tasks to a series of smaller tasks and report improvement over baselines for both language- and image-based pretrained models in terms of the number of parameters as well as accuracy as more tasks are added. ", "questions": "It would be great to provide results where all data is used together for multi-label classification following the original dataset with a shared adapter (no continual learning). It will give the readers an idea of existing performance gap.\n\nWhat is the purpose of eq(1)? Indicating both the individually adapted networks and their set as \"f\" is confusing. Also line 142, what is \\mathcal{U}? It would be great if these notations can be made clearer.", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. Considerable improvements in memory requirement as the number of adapters remain fixed while the number of tasks increases.\n2. Clever use of transferability metrics to reuse adapters.\n\nWeaknesses:\n\n1. For validating the proposed solution, the authors consider synthetic settings: convert a multi-label classification task to multiple binary classification tasks. In such a situation, finding transferable adapters is much easier since all tasks share an underlying objective. While there are improvements, it is not clear whether, in a realistic setting where tasks may not be as related, the improvements will hold, or if the method will even be able to find transferable adapters. Combining the three datasets on text classification for example would be interesting to see. \n2. The writing of section 3 is hard to follow. Many notations have been used without clearly indicating what they mean.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept