Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper systematically looks for the best way to combine parameter-efficient fine-tuning methods. Authors consider 4 design decisions: (1) how to group layers together (that layers within one group are treated equally), (2) how to allocate the "budget" of trainable parameters between groups, (3) whether to tune each of the groups and (4) the combination of PEFT algorithms to use in each group. In (2) and (3), authors find that naive solution is optimal. In (1) and (4), they find a combination of methods that outperforms all individual strategies. Finally, authors demonstrate that this combination can generalize to different models (e.g. from T5 to RoBERTa, BART) -- and that it is fairly consistent among tasks.
strength_and_weaknesses: ### Strengths

To the best of my understanding, this paper is a glorified hyperparameter search. [If authors disagree, I will eagerly hear their argument] However, this is not a bad thing: parameter-efficient finetuning (arguably) badly needs a systematic hyperarameter search. The area is flooding with various algorithms and they do not always compare in the same setting. To that end,

- the paper finds specific optimal combinations of algorithms for popular models. Even if we ignore generalization, these combinations are immediately useful to practitioners.
- if authors truly found a combination of layers that generalizes across models (see a few concerns below), this combination could have an even greater practical impact, since it would be a great default combination for new transformer-based models
- the paper reports several negative results in Section 4.2.3, 4.2.4. These are (again, arguably) not surprising, but still useful to save the time of researchers that tune their PEFT algorithms.


### Weaknesses

All my concerns are about unjustified or under-explored design decisions:

1. Using 4 groups: while authors mention that it is justified by some vision model, it is unclear why this choice is justified for the NLP tasks in question. One way to validate this choice would be to run an ablation analysis: if doubling the number of groups does not change results, then 4 groups are likely enough.
2. The proposed search protocol answers one question at a time, e.g. group sizes first, then algorithms. To the best of my understanding, the order of these design spaces is arbitrary. Do the results hold if authors change the order in which they apply design constraints? e.g. select methods first, then determine group sizes.
3. Authors explore how to allocate parameters between layer groups. However, when a single parameter group uses multiple methods (e.g. lora and soft prompts), how do they allocate parameters between groups? E.g. more prompts or higher adapter rank?


### Questions

1. To the best of my knowledge, T5 and BART are encoder-decoder models, while RoBERTa is encoder-only. How do authors apply grouping in encoder-decoder models? e.g. does the spindle grouping apply encoder and decoder separately, or pretend that they are stacked together? Does LoRA apply to T5/BART's cross-attention, or only self attention? Does any of this affect model quality?

2. Concerning the en-ro machine translation experiments: to the best of my understanding, the pre-trained models were not trained on the destination language. Furthermore, its tokenizer / embeddings might be unsuitable for this language. How exactly do you run these experiments?

3. the search protocol for finding optimal design spaces is based on random search. The same problem can be solved with [A] hyperparameter optimization techniques, e.g. TPE or gaussian process optimization (see Ray-tune or Optuna) or [B] neural architecture search. Why did authors opt for random search?
clarity,_quality,_novelty_and_reproducibility: ### Clarity and quality

The paper is generally well-written, the few remaining typos do not affect clarity. I have no complaints about its presentation quality.

Typo: [tables 9 and 10, caption] -- “ G1-(L, A) “ - shouldn’t it be (A, L)?

### Novelty

To the best of my knowledge, (1) the specific optimal combinations of design choices is novel and (2) the claim that these design choices generalize between different models, if properly verified, is also novel.

### Reproducibility

The current version of the paper is difficult to reproduce, since (1) the source code is not available, (2) the detailed experiment setups are also missing. To the best of my knowledge, even basic settings such as optimizer hyperparameters / batch size / learning rate schedule are missing. Since this is a practical paper, both code and configuration are necessary: not having them would undermine the core contribution of the paper.
summary_of_the_review: To reiterate: I believe that this is paper is simply a systematic hyperparameter search -- in an area that could use just such a search. In its current form, the paper lacks several important details: justification of some choices (e.g. number of groups), verification of the search protocol (e.g. is it robust to the order of design spaces?) and reproducibility details (code, setups, config).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a strategy to find a parameter-efficient tuneable architecture from a given pre-trained backbone neural network. In particular, the strategy consits of four phases exploring 1) how to group layers in the backbone neural network; 2) how to allocate the tuneable parameters within each group; 3) how to decide which groups to tune and which groups to keep frozen; 4) assign the best strategy to increase the parameters in each group. The paper uses this strategy to find an architecture based on BERT, RoBERTa and BART that tunes only 0.5% of the original parameters, and achieves better results than full fine-tuning on GLUE for BERT and RoBERTa backbones.
strength_and_weaknesses: **Strengths**
- The paper is excellently written, it's easy to follow, and enough details are provided for practitioners and other researchers to implement the method.
- The proposed approach seems to provide quite consistent improvements (2 out of the 3 backbones improve results w.r.t. full fine-tuning). 
- Although there are no experiments with runs using multiple random seeds, there are extensive experiments on multiple datasets (GLUE benchmark), and the best options are quite consistent across all datasets.

**Weaknesses**
- The main criticism is that the paper ignores other important baselines to choose where/how to tune the 0.5% of parameters during fine-tuning. For instance, the search space could be encoded in one of the many AutoML algorithms and let it find the optimal strategy, rather than relying on the fixed and hardcoded steps suggested by the paper, and then comparing the final results under a certain compute budget.
- The second shortcomming is that the three backbone architectures that have been tested are Transformer-based. The method should be ideally also tested with other backbones and/or domains (e.g. ResNets for image classification).
- The goal of having only 0.5% of tuneable parameters should be estated. For instance, if the goal is to reduce the fine-tuning cost, notice that the computational savings of tuning only 0.5% of parameters are nothing compared to the large amount of tuning needed to find the optimal strategy.
- It's not clear how the "Prefix" strategy can be applied independently on each group of layers. After all, it adds a new trainable token to the sequence. So, when "Prefix" it's added to a given layer (or group), all subsequent layers will have an additional token. No? Could you clarify this?
- The paper states that due to computational limitations, not all combinations were explored in this work. That's fine, but then the paper should be explicitly mention the cost of applying the proposed strategy w.r.t. the number of layers in the network and strategies considered.
Overall, the subspace explored in each design step by the paper isn't clear.
- The overall cost of finding a good strategy to find a small subset of parameters to fine-tune is very expensive, though. Thus, this reviewer has doubts that this will be applied in practice, in comparison to simply fine-tuning the entire network or just using Adapters or LoRa in all layers, which give competitive results with a much simpler (and cheaper) process.
- For example: when exploring the space $S_1$, depending on the number of layers, there are multiple options for the Increasing, Decreasing, Spindle, and Bottleneck options (e.g. for a network with 12 layers, with increasing strategy, valid configurations are $N_1 = 1, N_2 = 2, N_3 = 4, N_4 = 5$ or $N_1 = 1, N_2 = 2, N_3 = 3, N_4 = 6$), but it's not indicated which ones are recommended or were explored in the paper.
- In Table 6 (BART results), the bolded results should be those of "full" and not the proposed method, since "full" is better than the proposed method.
- Clarification: when multiple strategies are applied on the same group, it means that all the strategies are applied in all layers of the group, right? (e.g. $G_1-(A, L)$ means that in every layer of $G_1$ adapters are added and LoRa is applied).
- Suggestion: In table 7 (appendix A), add an extra (multirow) column indicating the number of epochs on each group, for better readability.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, easy to follow, and it's relatively easy to implement so that other researchers and practicioners can replicate the results (module some missing details, as indicated in the weaknesses above). The strategy is quite novel, altough a comparison against strong and competitive baselines (AutoML) is needed.
summary_of_the_review: The paper proposes a simple-to-implement strategy to find a good way of tune a small set of parameters instead of fine-tuning the entire network, which gives excellent results in the GLUE benchmarks for different Transformer-based backbone networks. The work, however, has several and important drawbacks mentioned above, thus I think it's marginally below the acceptance threshold. If the authors are able to satisfactorily address some of my concerns, I'll be happy to increase my score.

UPDATE AFTER DISCUSSION:
The authors have addressed some of my concerns and clarified my questions. Thus, I'm increasing the score from 5 to 6. I thank the authors for their detailed response.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Parameter-efficient training (PET) is a very popular technique for domain and/or task adaptation of large pre-trained models. As opposed to in-context learning, it does perform weight updates, but much less so than full fine-tuning (0.5% of the total parameters in this case).
Many different techniques and architectures have been proposed over the last few years, and this paper proposes to:
1. model that design space in a more structured way
2. go over that design space empirically and discover possible new designs for PET

As a result they do not only show that this is possible, but also that the obtained design have some type of generalization, carrying over to different architectures and tasks
strength_and_weaknesses: STRENGTHS
1. A proposal to structure the different PET proposals into a unifying design space. In this sense, it is similar to He et al. (2022) (on this, a more thorough explanation of the main difference with that paper would be welcome)
2. A pragmatic way of searching over that space
3. Empirical results showing that the discovered design is better than previously proposed architectures, and generalize to new architecture and tasks. Specificailly, the obtained fine-tuning strategy carries over from T5 to Roberta, indicating that the findings are not ad-hoc for the T5 architecture. Similarly, the discovered design carries over from GLUE to a very different tasks (summarisation on XSUM, and translation for en-ro).

WEAKNESS
1. From the claims, I was expecting to read a "Evolved Transformer for parameter-efficient training" paper. This is not it, in that the search space is tree-based, and the design selection is greedy over each of the four stages. This is still interesting, as current proposals always felt wasteful in that they run the same architecture on all layers, but not necessarily what was expected from the abstract
2. Clarity. The paper is easy to read, but hard to understand in its details. For instance
 - what are the four groups? A very quick search of the provided reference (Dosovitskiy et al. (2020) does not answer that 
 - the paper (specially the appendices) contain tons of tables and numbers, but it is not clear - even for a non-casual reader - what all those tables are saying. As an example, 4.3 seems to make the claim that the discovered design space carries over from T5-base to T5-3b, but it is not clear what table needs to be compared with what table. I compared Table 15 with all of Table 8-11 and from those it seems that this claim is actually not true (this is, the best strategy assignment per group for T5-3b is different than the best strategy assignment per group for T5-base)
 
clarity,_quality,_novelty_and_reproducibility: Clarity: low
Quality: medium-to-high.
Novelty: high
Reproducibility: hard without the source code, although this is promised
summary_of_the_review: A good and rather exhaustive exploration of different design choices, addressing the issue that current PET strategies are rather rigid. The obtained design performs better than previous strategies.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper empirically evaluates the best practice of parameter-efficient fine-tuning to achieve comparable performance with as few trainable parameters as possible. To achieve this goal, the authors design and implement different technique combinations and find the best solution via downstream experiments. Specifically, the pipeline contains four steps: 1) grouping layers into different sets; 2) adding trainable parameters towards each group; 3) deciding which group should be trained; 4) assigning groups with different training strategies.

Experiments show that spindle-style grouping, uniformly allocating trainable parameters to each layer, and tuning all layers achieve the best practice performance.  Step 4 requires the evaluation of different strategy combinations from Adapter (A), Prefix (P), BitFit (B), and LoRA (L).   Different architectures show different best combination settings. 


strength_and_weaknesses: Strength:

1. The proposed architecture beats traditional widely-used approaches, like LoRA.
2. The authors empirically explore the unified search space of parameter-efficient training. 


Weaknesses:

My main concern comes from the application of the proposed methods: If a simple parameter-efficient training method (e.g., LoRA) can achieve comparable results, why do we need to design a heavy search algorithm to search for the best solution with only marginal improvements? 


**Experiment Results**

1. Different architectures have different best settings. It is time-consuming if we need to do a grid search given a new architecture. 

2. Table 5 shows that the improvements over LoRA mainly come from step 4. The authors should report more solid performance improvements to show the cost of searching is valuable. 

3. GLUE contains several small-size datasets, like RTE, and CoLA. Different random seeds can contribute to over 5+ accuracy variance. As we can see, the improvements over baselines mainly come from these two datasets. Mean and variance are required for convincing results. 


**Experiment Settings**

"we (i) randomly sample 100 models from the S0 design space that satisfy each grouping pattern constraint (Figure 2); (ii) fine-tune with 3 epochs; and (iii) compute the average performances for each design space. We will follow this procedure as we progressively add new constraints later."

The authors should report the costs of pre-training 100 models. 

Furthermore, the authors should report the mean and variance across 100 models. If the gap between mean performance and the best performance is small, do we really need such search space? 



**Experiment Conclusions**

The step2 and step3 seem to be artificial considering that the final decision is tuning all groups and uniformly allocating the number of trainable parameters. 

If the authors can address these concerns, I'd like to raise the score.


clarity,_quality,_novelty_and_reproducibility: 1. Mean and variance are required for better reproduction. 


summary_of_the_review: The proposed combination method requires massive experiments and different architectures report different best settings. It is still unclear whether the proposed method is applicable to diverse architectures. Furthermore, the improvements over baselines are marginal in Table 5, and an ablation study is required to evaluate the contribution of different settings in the final performance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies how existing parameter-efficient finetuning methods can be combined to enable even greater parameter efficiency in pretrained language models. The main contribution of this paper is the showing that existing methods can be combined and can even generalize across different pretrained models. The main weakness is novelty: in some sense, the authors are performing hyperparameter tuning across the different methods.
justification_for_why_not_higher_score: Low novelty from a methodological standpoint.
justification_for_why_not_lower_score: After discussion, the reviewers (and the AC) felt that the empirical contribution of the paper was "enough" to warrant acceptance.
Decision: Accept: poster