Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper describes a distillation framework that leans heavily on Mixup, an effective data augmentation technique that is typically applied to images, to achieve impressive results on a range of GLUE benchmarks (). By using SM+TMKD+BT, the student model (BERT_6) is able to capture 99.88% of the performance of the teacher model (BERT_12).

The authors provide good motivation for the problem of distilling large-scale language models and conduct extensive experiments across multiple GLUE benchmarks, including ablation and hyperparameter sensitivity studies, and the results are quite convincing. Figure 3 nicely demonstrates that the framework is not overly sensitive to any of its hyperparameters.

Questions:
- Given that the paper motivates distillation as a means of reducing the computational cost of state-of-the-art language models (less power, less memory, lower latency), could you also describe the number of parameters that the student models use and their latency at inference time (vs. that of the teacher models)?

While the paper is generally well-written, there are a number of small issues / inconsistencies:
- Nit: Mixup is inconsistently capitalized / hyphenated.
- Since Mixup is central to this distillation framework, it would be valuable to briefly describe the method in the body of the paper, before describing how it was adapted to the domain of language.
- “that a a multi-task BERT model…”
- “around an training example”
- Nit: “vicinal” is an extremely rare word, so it might be worth using a more accessible word such as “neighboring”
- Nit: “tabular data were also shown, to demonstrate generality” (remove comma)
- Nit: “the extra word embeddings are mixuped with zero paddings” (maybe “mixed up” is better? :))
- In Table 1, for ease of reading, it would be good to describe in the caption the meaning of the first and second ‘/’ delimited number in each cell.
- For Figure 2, please describe the dimensionality reduction method used to visualize the latent space of training data and augmented examples. The pattern looks compelling, but I don’t see you describe anywhere how the data was projected. 
review_title: Extensive experiments demonstrating effectiveness of distillation framework for large-scale NLP models

final_score: rating: 6: Marginally above acceptance threshold
review_content: Nice paper. The main idea in this paper is to use a specific kind of data augmentation, Mixup (Manifold Mixup), in order to improve the effectiveness of the KD process and obtain better performing student models, especially in cases where not enough data is available on the target dataset and task.

While the idea is interesting in itself, I think what is proposed in this paper, is very relevant to this paper (noisy student): https://arxiv.org/abs/1911.04252

So, methodologically it's not super novel, but it should also be taken into account that many of the developments in this area have happened very recently. Moreover, the paper still has an added value since it is applying these techniques for a different input modality and in a slightly different setup. 

Summary of the experiments:
-12 layer BERT is fine tuned as the teacher (different fine-tuned teacher for each task).
- Two different architectures are used for student models (6 layer BERT and 3 layer BERT).
- Student models are initialised by copying the lower layers of the teachers.
- The teacher is used to provide pseudo labels for the student model on the target dataset, while the target dataset is augmented with the manifold mixup approach on the embedding layer (Proposed Approach).
- The proposed approach is compared with plain KD and no KD (just fine-tuning on target).
 
My Questions:
- During KD, do you feed original examples from target to the teacher then apply mixup, or do you first apply mixup then feed the generated examples to the teacher and get the pseudo labels from the teacher?
-Most importantly, It is not clear to me from the experiments whether there is still an advantage of doing this, if the teacher is trained well enough e.g., with the same type of augmentation (intuitively, if the teacher is trained well enough and KD process is well tuned, all the information that the model can gain by doing additional data augmentations should already be transferrable through the soft targets from the teacher.)

Some positive point about the paper:
- Ablation experiments are conducted to separate the gain from simple KD with KD+Mixup.

Some of my concerns:
- The improvements in the accuracies reported seem marginal and I think there are indication of the significancy of the results in the paper (e.g., mean and variance over several trials?).

Some minor points:
- Figure 2: Mixup and Non Mixup examples (circles and triangles) are not clearly differentiable (just hard to see).
-I think the type of data augmentation applied is a special case of Manifold Mixup, so it would be nice to cite this paper as well:  https://arxiv.org/abs/1806.05236


review_title: Intuitive approach to improve the benefits gained from KD

final_score: rating: 7: Good paper, accept
review_content: The paper proposes combining the MixUp data augmentation method with teacher-student distillation to improve the fine-tuned performance of BERT on benchmark NLP tasks (GLUE). The problem is important, well-motivated and of interest to a broad base of NLP researchers and practitioners. The paper is clear, and generally well-written, although the idea itself is not surprisingly novel (somewhat of a low-hanging fruit), from the experimental results, the method improves upon baselines, and so the real-world impact could be high, especially given its simple implementation.

Pros:
* Well-written, solid experiments / ablations on a canonical benchmark (bonus points for reporting GLUE test set performance), some theoretical contributions.
* Reports performance in labeled-data-limited setting (by subsampling GLUE dataset)

Cons:
* In the theory, a bounded loss function is assumed, but in practice the unbounded cross-entropy is used (right?)
* Optimal hyper-parameters $\alpha_\text{SM}$ and $\alpha_\text{TMKD}$ for "MixKD" on the GLUE dev test are not reported in the main text
* "MixKD" includes a back-translation (BT) loss term as well as a student loss on mixup samples (SM). While the impact of each term is studied on the GLUE dev set, I'm interested in seeing its performance on GLUE test.
* In the theorems, $\delta$ should be re-introduced.
* The embedding visualization section needs improvement. What is being visualized here? 2-dimensional logits / probabilities from the model or input embeddings projected down to 2D? Furthermore, it is hard to distinguish circles from triangles without zooming in 3-5x.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper applies mixup (Zhang et al., 2018) to augment training data to improve knowledge distillation in NLP tasks . Mixup was originally proposed to augment data for continuous data. To apply mixup to textual data, this paper applies mixup to the word/token embeddings instead of the tokens themselves. Some theoretical analysis has been done, and the experimental results show improved metrics over baseline methods such as DistillBERT.


*Strong points*
- This paper is well-written and very easy to follow.
- Theoretical results have been provided and the experiment has been carefully done.

*Weak points*
- The proposed method is a direct application of mixup (Zhang et al., 2018). The overall idea is quite straightforward.
- The empirical results of the proposed method are not impressive (worse than the metrics reported in TinyBERT (Jiao et al., 2019)?).
- It is true that for benchmarks like GLUE, the datasets are quite small so that data augmentation is important for knowledge distillation. This is however not true for most real-world applications where we often have almost unlimited unlabelled data (e.g., from logs, extract from webs etc). In other words, text data augmentation for knowledge distillation is not necessarily an important problem for real-world applications.

*Questions & Other comments*
- In Table 3, the metrics for TinyBERT is different from what has been reported in TinyBERT (Jiao et al., 2019). Can you add more explanation here?
- “Notably, TinyBERT’s data augmentation module is much less efficient than mixup’s simple operation” It is not an issue to have some computation overhead in data augmentation, as it is a one-off operation.

review_title: An interesting paper but not good enough

Meta-review: This work explores the distillation of language models using MixUp for data augmentation. Distillation with MixUp seems to be novel in the narrow context of distilling language models, although it has been used before in different contexts as the reviewers point out. The results of the experimental validation are encouraging, and the application is valuable and of wide interest to the ICLR audience. I therefore recommend accepting this paper for a poster presentation.
Decision: Accept (Poster)