Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an effective method AutoDistill to compress several student models with various computation costs at once. By combining few-shot NAS and task-agnostic knowledge distillation, AutoDistill alleviates optimization interference and speed-up the training process, enabling single-stage training without further pre-training and distillation. Experimental results demonstrate that AutoDistill outperforms strong NAS and knowledge distillation baselines.
strengths_and_weaknesses: Strengths:
1. The paper is clearly written and well structured. The authors first demonstrate the weaknesses of existing knowledge distillation and NAS-based language model compression methods, then they introduce their method clearly, making it easy for readers to follow.
2. The proposed method is reasonable and technically sound. By adopting the idea of few-shot NAS, the problem of optimization interference is alleviated. Moreover, by fully task-agnostic knowledge distillation and search, AutoDistill significantly reduces training costs.
3. The authors conduct comprehensive experiments and the results verify the superiority of AutoDistill over strong baseline methods.

Weaknesses:
1. Although the proposed method is effective, the major two components of the method (i.e., few-shot NAS and task-agnostic knowledge distillation) are not novel. When compared with DynaBERT, it seems that AutoDistill is its task-agnostic version with a larger searching space and a few other modifications.
2. When compared with existing NAS-based language model compression method (e.g., AutoTinyBERT), AutoDistill enjoys much less computation cost in training and searching. However, when compared with traditional task-agnostic knowledge distillation method (e.g., MiniLMv2), the advantage of training speed is gone.
questions: 1. How transferable is the optimal student architecture derived from the proposed AutoDistill method across teachers and knowledge distillation objectives? 
2. Given some optimal student architecture derived by NAS, how would traditional task-agnostic knowledge distillation method (e.g., MiniLMv2) perform?
limitations: The authors address the limitations adequately.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work partitions the search space of neural architecture search into K sub-spaces, trains a SuperLM for each partition with task-agnostic self-attention distillation, and finds optimal students without additional training.
strengths_and_weaknesses: The proposed methods outperform traditional knowledge distillation baseline and other NAS baselines. The idea of partitioning search space by using multiple supernets is intuitive and practically useful. Other implementations are borrowed from the existing methods like BigNAS and MiniLM. The paper provides adequate ablation study results to confirm the effectiveness of each component (e.g., training strategies, few-shot NAS, search strategy).
questions: How you define the search space in Table 1 is not straightforward, though the authors mentioned it is based on inductive bias and heuristic. Could you elaborate on the heuristic part? The prior work mentioned in Line 129 is not evaluated by Transformer architectures, so I doubt inductive bias from this work is valid to apply.

SuperLM in a smaller region is also initialized with pre-trained BERT-Base, so only the leftmost layers, dimensions, and heads are used. However, dimensions and heads are order-independent, meaning that the performance might change after re-ordering.

Section 4.1.3 and Section 4.1.4 are only tested on MRPC and RTE datasets. I am suspicious they are enough or good representatives.

AutoDistil-proxy only uses MNLI as a source task. I am curious how the transferability of optimal architectures changes between source and target tasks?
limitations: I wonder whether the few-shot task-agnostic NAS method is applicable to other domains, including computer vision.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper preposes AutoDistill, a recipe for distilling large language models into smaller sizes when a budget is given. 

AutoDistill has three major steps: 
* designing the search space with 4 hyperparameters, and partition the large search space into three smaller ones (K=3; tiny, small, base).
* training a SuperLM in each partition, with a task-agnostic objective similar to MiniLM (Wang et al., 2020)
* obtain all possible student architectures by pruning the SuperLM, and find the best one by enumerating the model and computing the loss on either validation data or task-specific data

AutoDistill achieves superior performance on GLUE benchmark when compared to baselines such as (a) fine-tuning, (b) task-agnostic distillation baselines: DistilBERT, TinyBERT, and (c) NAS baselines: NAS-BERT. Meanwhile AutoDistill is more computationally efficient as demonstrated in Table 3.

strengths_and_weaknesses: Strength:
* AutoDistill can be practically useful. It achieves 3x compression over other distillation methods. The resulting models can be re-used by practitioners and researchers.
* Solid experiment and clear visualization. 
* The authors have uploaded code with running instructions to ensure reproducibility.

Weakness:
* The novelty of AutoDistill is still unclear. Components of AutoDistill seem to be adapted from existing methods.
* Writing is sometimes confusing and not coherent. See questions below.

questions: (Disclaimer: I know little about NAS or one-shot/few-shot NAS. My ratings are based on my current understanding and I hope the authors can clarify the following for me)

* Does "few-shot" in the method mainly mean splitting the search space into K=3 subspaces?
* What is the major difference between AutoDistill and Reference [16] (Zhao et al., 2021)? 
* The motivation of few-shot NAS is to alleviate conflicts in weight-sharing (Line 161) and mitigate gradient conflicts (Line 90). As this is one contribution highlighted by the authors, I believe some ablation analysis or discussion on not using the 3 subspaces design should be included.
* In figure 1 and in Line 251, how do you quantify "best trade-off"? What is the metric used to determine this?
* Following on the previous question, in figure 1, since MNLI performance is the criteria and also the y-axis, why is the selected models (red) not always the best performing models?

Minor Questions:
* How large is the validation set used in Task-agnostic search (Line 206)? It is quite surprising to me that evaluating the 256 candidate subnetworks on this set (or on MNLI) takes less than one hour.
* Line 300. Not clear about the MLM baseline. Does it mean the SuperLM is directly trained on MLM objective? Is the "random sample a subnetwork" step still used?
limitations: Limitations are discussed in Appendix E.
review_title: null

Meta-review: metareview: The submission introduces an approach to searching for student architectures to distill large language models into. The authors divide the search space into different sizes of student model, train a SuperLM for each model, and then select the optimal student from within these using knowledge distillation. While several of the techniques here have been used before, the reviewers found that the overall approach was well motivated and effective empirically, outperforming strong baselines. As pointed out by reviewer ov4n, the writing could be improved to make the paper more accessible to people less familiar with NAS, but overall this is solid work and I recommend acceptance.
confidence: Less certain
Decision: Accept