Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper breaks down the design of state-of-the-art parameter efﬁcient transfer learning methods and presents a uniﬁed framework that establishes connections between them. The paper re-frames them as modifications to specific hidden states in pre-trained models and defines a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Experiments on machine translation, text summarization, language understanding, and text classification have been conducted, which indicates that the unified framework enables can instantiate new parameter-efficient fine-tuning methods that tune fewer parameters.
main_review: Strengths:

1. The analysis and conclusion on Adapters, Prefix Tuning, and LoRA are very penetrating. Specifically, the paper derives an equivalent form of preﬁx tuning to establish its connection with adapters in a unified view.


2. The paper proposes a uniﬁed framework for parameter-efﬁcent tuning that includes several state-of-the-art methods as instantiations.

3. The paper is well organized and well-motivated with theoretical analysis and proof. 

4. Experiment setting and analysis are comprehensive; the proposed method Scaled PA consistently shows its advantages against the other baselines.

summary_of_the_review: Overall, this paper is well-written and well-motivated. I think this paper attracts lots of researchers and will inspire future works in parameter efficient training.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates recent parameter-efficient methods that have been shown to achieve good performance in a variety of NLP tasks. The authors analyse their connections and propose a unified framework which subsumes a number of existing approaches. The authors then compare their performance across four NLP tasks, with varying level of difficulty and amount of available resources. Here, compared to full fine-tuning, they find that parameter-efficient approaches perform well on simpler tasks (MNLI and SST2), while showing larger gaps on more challenging tasks (XSum and MT). A series of controlled experiments centred around the proposed framework shows that parallel adapters applied to the FFN module of a Transformer generally perform better. Applying them to the attention module leads, however, to better gains when the number of additional parameters is small. Finally, the authors combine these findings into a novel MAM adapter module that leads to further gains in every task.
main_review: Strengths:
- A new perspective of parameter-efficient methods within a unified framework.
- A number of controlled studies which give insights into which components are meaningful and in which cases.
- The released code base can help boost future research in this area.
- Clear and well-written.

Weaknesses:
- No major weaknesses identified.

Questions:
- It would be interesting to see the variance observed when applying parameter-efficient methods. Previous work found large variance in downstream performance across different seeds when performing full fine-tuning [1,2,3]. This is something you could add, for example, to Table 2.

Typos:
- In Figure 2, the performance of Adapter is 21.00, not 20.46
- The "v" subscript is missing in Eq. 5

[1] Dodge et al. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. 2021
[2] Mosbach et al. On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines. ICLR 2021
[3] Bugliarello et al. Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs. TACL 2021
summary_of_the_review: This paper provides a unified framework for parameter-efficient NLP, an important task when serving models at scale. The experiments provide new insights into the performance of existing methods, that, when combined, result in further performance improvements. I think this is a solid paper and recommend its acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper formulates different approaches for parameter-efficient transfer learning (such as adapters, prefix-tuning, LoRA) under a common framework. Approaches vary alongside different design dimensions: functional form, insertion form (sequential or parallel), which representation is directly modified by the approach and the composition function of the main representation with the adaptation one. 

The authors describe how existing approaches fit this framework. Then, they show how changing these design choices leads to new approaches, including parallel adapters (PA), multi-head PA and scaled PA. 
The authors then compare their newly introduced methods and existing ones empirically. They find that:
1. Parallel insertion is generally better
2. Under a low parameter budget (~0.1% of original) it is preferable to modify the attention of the transformer. Under a higher budget, changing the feed-forward network is better.
3. Their methods generally perform on par or better than existing approaches.

Finally, the authors combine their insights to develop “Mix-and-match” adapters and show that it does well overall.

main_review: Pros:

- The common framework for different approaches is useful overall. In particular, I liked the section clarifying the connection between Prefix Tuning and Adapters. Although there is nothing groundbreaking here, I believe these frameworks can be useful and help researchers.
- The authors show how the framework can be used to derive new approaches. They also show that these approaches can be more effective than existing ones.
- The experimental setup is well-described overall, with hyperparameters being provided
- This is a hard analysis to perform, with many possible things to change. I feel like the experiment set chosen is convincing overall, with the caveat mentioned below.

Cons:

- Significance of results:

There are no standard deviations for any results in the paper, which makes it hard to assess significance of many results, esp. since some of the performance gaps are small.
For instance, in table 6, authors have a 0.4 BLEU discrepancy in their replication of full fine-tuning performance but draw conclusions on MAM being best based on a 0.2 BLEU gap. The gap is bigger when comparing MAM to methods not introduced by the authors, but still.
The story makes sense but I am not 100% confident in the robustness of the results. Using two decimal numbers for tasks (e.g: XSum) also gives a false impression of precision. 
I understand fine-tuning on some of these tasks (MT / XSUM) can be resource-intensive but having standard deviations in even a subset of the experiments would be useful.

- Interpretation of Figure 4 in Section 5.2 and effectiveness of MAM Adapters for encoder models

The authors highlight that while existing methods perform well on MNLI/SST, they are underwhelming on en-ro / XSum. They conclude that this means existing parameter-efficient transfer learning (PETL) approaches are not great for higher-resource / more challenging tasks.
	However, I would point out that this data can also support a different conclusion: existing PETL approaches work well for encoder-only models but are not great for encoder-decoder models. Including the T5 datapoint is also not very relevant since in that case Superglue is treated as a single task (w/ one adapter). Answering “For which tasks / architectures do PETL methods perform well” is an interesting question in itself and I feel like this section expedites this. It is likely that both the architecture and #datapoints in the task matter. 
	The author’s choice at the end of section 4.2 is to focus on XSum / en-ro MT. It would be great to highlight that not only are those higher-resource tasks, but they are also generative ones, and thus all the conclusions of this paper might not apply widely to encoder models. Indeed, the results of MAM adapters in table 2 are quite mixed. Right now the paper is claiming more generality than deserved.
	Two possible fixes here: (1) Make claims less general and specific to enc-dec models, (2) Conduct more experiments on encoder-only models 

- Writing:
  - The writing is subpar right now. Example: section 4.4: “its counterpart at attention” -> “its attention counterpart”,  “FFN can better utilize modification at larger capacities” -> modifications or “modification of the FFN is better at larger capacities”, etc. 
  - Minor: The citation style is often wrong for the sentence. Use \citet more often.
  - Minor: Replace “For classifications” with “for classification tasks” in the Appendix (twice)
  - Typos: Section 5 name Discussions -> Discussion ;  Appendix A3 Learning -> Learning. 

- Minor:
  - For Figure 2, I feel like the full fine-tuning number should come from the original baseline, not the replication.
  - The protocol in 4.5 for choosing the scaling parameter for Scaled PA is a bit surprising. I would just suggest that it should be an hyperparameter, the current description makes it seem like it is based on hyperparam tuning for LoRA. 
  - It seems to me that 4.5 is about composition function *and* \delta h functional form since LoRA changes both. 


~~~~~~~~

Revised score upwards after response, see below

summary_of_the_review: Compelling framework to unify parameter-efficient transfer learning approaches, with interesting new methods emerging from it. The set of experiments is well-chosen overall, but the lack of standard deviations and the focus on generative tasks / enc-decoder models makes the results less robust and general. The writing would benefit from more work as well.
review_title: null

Meta-review: The paper reviews and draws connections between several parameter-efficient fine-tuning methods.

All reviewers found the paper addresses an important research problem, and the theoretical justification and empirical analyses are convincing.
Decision: Accept (Spotlight)