Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors introduce adaptive distillation of adapters, or ADA algorithm, which helps continual learning with transformers be memory efficient and performant at the same time. The work uses Adapters to fine-tune a large, pretrained transformer model to a new task, as opposed to the standard practice of fine-tuning the final layer. However, as Adapters require more parameters compared to a new final layer, the authors introduce ADA to make continual learning more memory efficient. Basically, ADA keeps a fixed stack of K adapters, and when a new Adapter needs to be added to the stack, ADA takes the Adapter that has the most similarity with the new task and distills that with the new task. The authors run experiments on language and vision tasks to show that ADA can perform continual learning well without catastrophic forgetting without using an unbounded number of parameters. Finally, the authors ablate the impact of adapter size and adapter pool size to show their effects on ADA.
strengths_and_weaknesses: Overall, the strengths of this paper are as follows:
- This work introduces ADA, which is a novel algorithm for adapting pre-trained transformers to continual learning settings without incurring significant memory overhead.
- The algorithm itself is simple with significant practical benefits as well, as shown by the experiments.
- ADA seems to perform comparatively with baselines while keeping to bounded memory, which is also an impressive feat.

However, there are a few places where this paper could improve, especially concerning its method and experiments.
- Firstly, the paper does not introduce the concept of Adapters sufficiently clearly. The way it is explained right now in the paper is insufficient, in my opinion, for someone that is not already familiar with the concept of adapters. A detailed explanation or a figure to explain the concept of adapters could really go a long way for this.
- Similarly, important concepts such as double distillation loss is not explained in the main body of the paper.
- On Figure 2, the X axis could be more informative if the metric was % increase in parameter per task rather than the total number of parameters in the model. The authors talk about this in the sense that 30 adapters roughly equal the size of a pretrained transformers, which I believe communicates the justification for distillation in the first place, and denoting the modified modules' sizes as % size of the base model would also work better here.
questions: - What is the relationship between K and the number of tasks at which the adapters get saturated? ADA does not show saturating behavior in the experiments in the paper, but intuitively there should be at some point where it does get saturated.
- What is the performance, quantitatively, of the random-adapter-distillation variation of ADA? Please include that baseline in the experiments as well, as it is currently we do not see any quantitative comparison between this version of ADA and the one with random replacement.
limitations: - The primary limitation of this work is that it is seemingly limited to transformer-based models only, with no clear indication of how it could transfer to other architectures. However, as of now, transformer-based models are enjoying decent success, and thus this limitation is not a major one.
- The experiments are performed on a handful of tasks, on the order of tens of tasks. It would be informative to see the behavior of ADA in the asymptotic case, where the K adapters are saturated and thus there is a downgrade in the performance. As the paper is presented right now, the readers do not get a balanced picture of where it may fail.
- ADA chooses the adapter to distill at every step in a greedy fashion, instead of optimizing over all the tasks seen so far. The paper does not analyze the worst-case behavior caused by this greedy choice of distilled adapter, or how ADA may be impacted by an unfortunate ordering of tasks.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper extends the work of Houlsby et al. by proposing a method to distill knowledge between adapters in continual learning. The high level idea is to freeze the foundation model's parameters and train task-specific adapters. The mitigate the lack of knowledge sharing between newly learned adapters, the authors propose a distillation method so that adapters can share knowledge and memory does not grow out of bounds.
strengths_and_weaknesses: Strengths:

This paper addresses an important problem in the domain of continual learning. There is typically a tradeoff between learning new tasks well and remembering old tasks. This paper tries to close the gap between this tradeoff by learning new tasks but keeping prior learned tasks in the model's memory. 

The memory efficient aspect is interesting, especially if there is a scenario where the number of tasks grow in orders of magnitude. 

The introduction was well posed, the task setup is nice, and the motivation is clear.



Weaknesses:

The method section was unclear. Without reading the prior papers such as Houlsby et al., it's hard to understand what is being proposed in this paper. I think the methods section could be condensed and clarified. 

The exact concept of ``knowledge distillation'' in terms of the method is unclear.  Algorithm 1 contains a Distill() function, but it's not explained in the paper. 

The distinction between \phi, \phi^{'}, \phi_{c} is unclear.

Lastly, the results section is not easy to understand. Without reading prior papers, the BWT and FWT metrics are unclear. The figures are somewhat hard to read. Based on the results, it seems like performance is worse using ADA vs vanilla adapters. The only contribution vs adapters is the memory reduction. 


Minor:

"This is an high-level" --> "This is a high-level"
questions: What are some example tasks where this memory reduction is necessary? It's outlined in the introduction, but not explicitly stated.

Where exactly is the "distillation" happening? Which equation, for example.

Why is knowledge distillation a motivation, but the performance is worse than Adapters? Doesn't this mean that it's not distilling knowledge correctly?
limitations: The authors talk about how their work is tied to transformers, but is this true? Can't this be applied for other model types that allow for task heads?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work presents a method of using adapters to fine-tune pretrained models to downstream tasks in a continual fashion. Instead of adding a new adapter for each new task, the authors define a fixed set of adapters of size K which are repurposed. When the number of tasks are less than K, each new task is trained using a new adapter. For each additional task, first, a new adapter is trained, and then its closest counterpart is selected in the set of K adapters (using TransScore or LEEPl; to measure transferability). The selected adapter is then distilled using a subset of data from all previous tasks that the model has been trained on to prevent forgetting. 

The authors convert multi-label and multi-task classification tasks to a series of smaller tasks and report improvement over baselines for both language- and image-based pretrained models in terms of the number of parameters as well as accuracy as more tasks are added. 
strengths_and_weaknesses: Strengths

1. Considerable improvements in memory requirement as the number of adapters remain fixed while the number of tasks increases.
2. Clever use of transferability metrics to reuse adapters.

Weaknesses:

1. For validating the proposed solution, the authors consider synthetic settings: convert a multi-label classification task to multiple binary classification tasks. In such a situation, finding transferable adapters is much easier since all tasks share an underlying objective. While there are improvements, it is not clear whether, in a realistic setting where tasks may not be as related, the improvements will hold, or if the method will even be able to find transferable adapters. Combining the three datasets on text classification for example would be interesting to see. 
2. The writing of section 3 is hard to follow. Many notations have been used without clearly indicating what they mean.


questions: It would be great to provide results where all data is used together for multi-label classification following the original dataset with a shared adapter (no continual learning). It will give the readers an idea of existing performance gap.

What is the purpose of eq(1)? Indicating both the individually adapted networks and their set as "f" is confusing. Also line 142, what is \mathcal{U}? It would be great if these notations can be made clearer.
limitations: NA
review_title: null

Meta-review: metareview: Some of the reviewers had concerns with experiments that would have compared not distilling (i.e. high-memory-usage everyone gets their own adapter) with the authors' method.  I am willing to recommend acceptance; but I agree that some sort of experiment like this would be useful to contextualize what is traded off for smaller memory footprint.  I would ask the authors to include some experiment like this for the camera ready.
confidence: Certain
Decision: Accept