Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a parameter efficient fine-tuning method that works for few-shot learning. The main contributions of the paper include: Extending T0 to work for few-shot learning. A single recipe for few-shot fine-tuning and a new parameter-efficient approach that outperforms prior work like prompt tuning and adapters.
strengths_and_weaknesses: Strengths:

- The paper is extremely well written. It was a joy to read.
- The new parameter-efficient approach (IA3) achieves strong results compared with other methods.
- The paper achieves strong few-shot results with moderate language model sizes (up to 11B), outperforming more expensive models like GPT3.
- A single hyperparmeter and model configuration setting for fine-tuning on all tasks.

Weaknesses:

- Some of the ablations in the appendix do not show reliable improvements and standard deviation seems too high (assuming the subscript numbers in the the appendix table cells refer to stdev). 
- Figure 2 shows the benefits of IA3 compared with prior work when fine-tuning T0-3B. But does the entire “T-few recipe” fail if another parameter efficient model (like adapters) is used instead of IA3? I.e., how important is AI3 within the proposed framework?
- I encourage the authors to move important ablations 
 to the main text. If space is concern, related work could be shortened.
- The only ICL baseline considered is GPT-3. There have been several follow up works on improving ICL. More recent baselines like Chinchilla [Hoffmann et al., 2022] could have been also considered.
questions: Please see above.
limitations: There is no discussion about limitations. For example, it is still an open question what makes AI3 to work better than methods like Adapters or LoRA.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper focuses on few-shot learning in NLP and shows the limitation of few-shot in-context learning method (high computational cost and so on) and propose an adaptation strategy that adds light-weight learnable parameters (rescaling vectors) into a frozen pre-trained model and fine-tune only the learnable parameters on few labeled samples for few-shot learning. The experimental results shows the proposed strategy obtains much better performance than existing few-shot ICL methods and PEFT methods with lower inference computation cost while it requires largest training computation cost, disk cost and and higher memory cost during training T-Few.
strengths_and_weaknesses: Strengths:

1. In overall, the paper is well-written.

2. The idea of reusing the pre-trained network and only learn additional light-weight parameters to adapt the pre-trained model for downstream tasks with few samples is interesting and shown to be obtain promising performance on downstream tasks.

3. The introduced rescaling vectors are light-weight.

Weaknesses:

1. One main weakness is the training cost of the T-Few. More specifically, when training the rescaling vectors on a new tasks, training T-Few will involves forward and backward passes through the whole model (though only rescaling vectors require gradients for updates) which will be computational costly for each tasks. Suppose there are a mass of tasks, the training cost is considerable.

2. Also, applying the proposed method to larger model can also increase the cost, e.g. applying to GPPT-3 with 175B parameters.

In overall, I think the idea is interesting and promising in the context of few-shot learning. However, I am not an expert on NLP field and I can not  assess the novelty of the paper in NLP.
questions: Some related literatures are missing [ref1,2,3].

[ref1] Requeima, James, et al. "Fast and flexible multi-task classification using conditional neural adaptive processes." Advances in Neural Information Processing Systems 32 (2019).

[ref2] Triantafillou, Eleni, et al. "Learning a universal template for few-shot dataset generalization." International Conference on Machine Learning. PMLR, 2021.

[ref3] Li, Wei-Hong, Xialei Liu, and Hakan Bilen. "Universal representation learning from multiple domains for few-shot classification." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
limitations: The limitations of the method are discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new parameter efficient fine-tuning (PEFT) method for pre-trained transformer models in NLP. The authors argue that, with a careful choice of parameters to be fine-tuned, this approach can be computationally much cheaper than in-context learning (ICL) and perform better than in-context learning approaches. The paper proposes a new method called IA3, where the trainable parameters are vectors that scale the activations in each layer. When applied to the T0 model, the authors show that the proposed fine-tuning method performs better than prior ICL and PEFT methods. In addition, a careful analysis is presented that shows the computation consumed by the proposed method and other approaches.
strengths_and_weaknesses: Strengths
* Clarity on experimental details, helps reproducibility. 
* Extensive experiments and strong performance.
* Offers a practical recipe for few-shot learning. 
* Clearly presented motivations.
* Computationally advantageous and better than in-context learning and other few-shot/parameter efficient learning methods.

Weaknesses
* The presentation can be reorganized so that the main contributions are highlighted better. For instance, IA3 could be introduced early on. The current presentation is dense and could use better structuring. 
* Although fig. 2 presents an extensive comparison, it is unclear whether the authors accounted for the difference in model architectures, pre-training data, etc.

Overall this is an interesting paper with extensive evaluation of the proposed approach. I appreciate the careful analysis of computational requirements. The following issues have to be addressed by the authors.
* How were hyperparameters and other design choices made? Did you stick to the ‘true few-shot learning setting’, or were there any validation sets involved? I would recommend making this clear to make sure that comparisons with baselines are fair. 
* Since the main method applies IA3 to T0, it is difficult to isolate the performance contribution of each. I would recommend presenting an analysis of IA3 applied to plain language models (no instruction tuning). 
* Did you also try baseline methods (adapters, prefix tuning, etc.) with/without unlikelihood training, length normalization, etc.? 

questions: * In Fig 2, how do you account for the difference in architecture, difference in pre-training data, etc.? Ideally, we would want to compare IA3 to other approaches for the same base model. 
limitations: Seems adequate
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces T-Few, a parameter-efficient few-shot learning protocol that achieves STOA few-shot performance on many tasks, and also outperforms full-model finetuning and in-context learning with huge LMs. They rescale inner activations of LMs with learned vectors, such that only those vectors are updated during training. Their design is slightly different from existing parameter-efficient methods to enable mix-task batch training. 
This paper demonstrates good engineering practice of parameter-efficient learning but there are flaws in the evaluations.   
strengths_and_weaknesses: The proposed method demonstrates strong few-shot performance on popular benchmarks and challenging RAFT benchmark. Those engineering practices will be a good contribution to the community and serve as bases for feature research on parameter-efficient tuning. 
The presentation is clear and easy to follow, and the motivation is well-justified and supported. 

However, I have the following concern regarding evaluation.
Although the proposed method clearly outperforms baselines, I'm worried that the comparison is not fair. For instance, IA^3 is powered by two new training objectives and mix-task batch training. All these have been proved effective in improving few-shot performance. However, in the evaluation, those tricks are not applied on baselines (especially new losses), resulting in an unfair comparison.  

There are also some mistakes in the description of the methodology, not sure if it's a typo or not. e.g., L212 we should minimize L_{LN} instead of maximizing it.


questions: .
limitations: yes
review_title: null

Meta-review: metareview: This paper demonstrates that Few-shot Parameter-efficient Fine-tuning (PEFT) is more accurate and dramatically less computationally expensive than in-context learning (ICL), and introduces a new PEFT method that varies the activity level depending on the learned vector and achieves high performance with only a few parameters. In addition, this paper proposes a simple way to apply it to the T0 model and shows that the proposed fine-tuning method performs better than the baselines. This paper is well written. The proposed method provides a simple and practical recipe for few-shot learning and shows strong performance on popular and challenging benchmarks. All three reviewers had similar positive comments on this paper. Thus the meta-reviewer recommends it for acceptance.
confidence: Certain
Decision: Accept