Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper propose COMPACTER, a parameter-efficient fine-tuning method  for large-scale language models. On GLUE datasets, COMPACTER only requires training 0.047% parameters without loss of accuracy, and moreover it outperforms standard fine-tuning in low-resource settings.

main_review: * Pros
  * Method is novel and interesting. COMPACTER builds on ideas from adapters, low-rank methods as well as recent hypercomplex multiplication layers.
  * Experiments and analysis are solid. It compares against a large number of baselines, and did thorough analysis on memory and training time. 
  *  Results are strong — it can achieve only 0.05% without loss of accuracy on GLUE, outperforming other parameter-efficient fine-tuning methods.  
   * The paper is well-written.
  * Parameter-efficient NLP method is an important direction.
* Cons
  * It will be great to also validate on datasets other than GLUE, e.g. Question Answering. 
* Missing references
  * Guo et al, Parameter-efficient transfer learning with Diff Pruning
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper develops a new parameter-efficient adapter layer, which extends the parameterized hypercomplex multiplication layers by sharing part of parameters across layers and decomposing adapter-specific parameters with two low-rank weights. Experimental results on the GLUE benchmark prove that the proposed approach outperforms other parameter-efficient finetuning methods and obtains performance on par with full finetuning.
main_review: Strengths:
1. The authors propose a novel parameter-efficient adapter layer to adapt the large-scale language model.  
2. The proposed method requires dramatically fewer parameters than adapters and finetuning, while outperforming other parameter-efficient fine-tuning methods.

Weaknesses:
1. The novelty of this paper is limited, which is a straightforward combination of parameterized hypercomplex multiplication layers, sharing information across adapters, and low-rank parameterization. These methods are widely known.

Detailed Comments:

In general, this paper is easy to follow, the idea is very straightforward, and the parameter efficiency is analyzed in detail. Experimental results on the GLUE benchmark verify the effectiveness of the proposed method in terms of the parameter-efficient and accuracy. But the proposed method is not novel enough to me. It is a straightforward combination of some previous techniques, including parameterized hypercomplex multiplication layers, sharing information across adapters, and low-rank parameterization. The low-rank parameterization method actually could further reduce parameters of the original adapter layer or the AdapterFusion method. Adding these experimental comparisons could make the experiment more convincing. In addition, the authors only verify the effectiveness of the proposed method on NLU tasks. Actually, the proposed method can directly apply to harder NLG tasks. I am curious whether this method achieves high performance on downstream NLG tasks with such few parameters.

limitations_and_societal_impact: Please see the detailed comments.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper propose an parameter-efficient fine-tuning method called Compactor by inserting a low-rank matrix decomposition adapter. Experiments support the claim. 
main_review: # Pros
Compactor basically combines the low-rank decomposition with adapter. The writing is clear and idea is very easy to follow. Experimental results seem reasonable.

# Cons
1. Limited novelty: the idea is not new to the community. As in the reference [24], the low-rank decomposition has already demonstrated effectiveness in batch ensemble learning. Extending it to multitask fine-tuning is natural and intuitive.

2. Weak results: authors only show results on GLUE benchmarks, which is not convincing. What about the performance on SuperGLUE, low-resource tasks and text generation?


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors presented an efficient parameterization of task-specific adapters for parameter-efficient multi-task transfer learning with large pre-trained language models based on transformer architecture.  

main_review: In general, the method is well described and demonstrated with pre-trained T5-base on GLUE.  The idea is novel and its motivations from prior work are clearly stated.  That said, I do have a few criticisms that I hope the authors would address.  

- **Model capacity**.  It is not clear whether Compacter is applicable to transformer LMs of different sizes, since only T5-base is demonstrated.  When hidden dimensions are smaller, are low-rank adapters still adequate?  
- **Task difficulty**.  It is not clear how the applicability of Compacter vary with difficulties of tasks.  Out of the GLUE tasks studied, how does the effectiveness of Compacter correlate with, say, word-order sensitivity (arXiv:2012.15180)?  What about even harder tasks and longer sequences?  Any results on SQuAD?  
- **Parameter efficiency**.  Though the authors conducted a comprehensive comparison on parameter efficiency, some other approaches might be even more efficient in parameters--for example, it has been shown that BERT can be fine-tuned for any GLUE task by setting a small fraction of pre-trained weights to zero, i.e. _sparsification as fine-tuning_ (arXiv:2004.14129), in which case only the row/column indices of the small fraction of weight entries are necessary to store in addition to the pre-trained weights.  
- **Practical significance**.  The approach of adding task-specific adapters to pre-trained LMs, however parameter and/or compute efficient the adapters are, still leads to a computational cost no less than that of the pre-trained network.  Pre-trained SOTA transformer models are grossly overparameterized and, just like other large NNs, can be compressed (by sparsification, quantization and/or distillation) for efficient inference in practice.  Thus, the adapter approach might not be as desirable as simultaneous compression/fine-tuning approaches such as _sparsification as fine-tuning_.  

limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper address efficient finetuning of pertrained models and investigate applications in NLP domains. The idea is to insert a parameter compact layer called Compacter in the pretrained models. By further parametering the Compacter layers with low-rank representation, its achieves a better trade-off between model performance, trained parameters and memory footprint compared with existing methods.

The idea is quite simple and the techniques are well explored in different scenarios by existing works. Thus most reviewers consider the technical contribution to be incremental. However, it demonstrates very good results in NLP pretrained models. Some reviewers were not satisfied with the limited experiments provided in the original submission. In the rebuttal, the authors was able to provide more results in a more complex data and model. And the results are in consistent with the results provided in the paper. The reviewers are generally satisfied with the rebuttal. Considering the technical limitation and the support from the reviewers, I recommend a weak accept and the authors to incorporate the new results and other comments from the reviewers into the revision.
Decision: Accept (Poster)