Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, authors start from a point where they perform a systematic study to measure the impact of many key hyperparameters and training strategies from previous work, such as distillation, DA, etc. Then, they find out that previous baselines for ultra-low bit precision quantization are significantly under-trained. Based on the study, they propose a new compression pipeline named XtrmC. The focus of this paper is mainly on BERT.
strengths_and_weaknesses: Strengths:
- informative and deep study on the existing work. The key factors evaluated by the paper may inspire following research
- Good presentation of the experiment results 
- Clear statements on the limitation of the current work and show how the future could be.

Weaknesses:
- The overall discussion in the main draft focus on the study, but not the new method. More dicussion on the new compression method may benefit the overall presentation logic since the title is about the extreme compression, not just a study on existing methods. 
questions: As stated in the weaknesses
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents an empirical study of the role of various stages in recent extreme compression of Transformer architectures for NLP tasks. Based on this study, the paper presents a simplified architecture and training process that significantly reduces model size.
strengths_and_weaknesses: Provides a thorough empirical study of how the # of training iterations affects heavily quantized network compression techniques. This shows that the accuracy of binarized networks is improved merely by training longer combined with data augmentation.This study also shows that complicated multi-stage distillation is not required to maintain accuracy of heavily quantized networks.
Based on the empirical study, the paper presents a simplified model which consists of layer reduction (structured pruning) and a single stage of distillation. The experimental results show that this simplified model works well compared to the more complex baselines.

As the proposed method requires more training iterations, it may be more computationally complex than competing methods, even if those methods require more complex distillation stages. The paper lacks a detailed analyses of the computational complexity of the various methods.

The contribution of the paper, while effective, is ad-hoc. It would be nice to have some theoretical discussion of "why" increased training times helps with heavy quantization, and why additional distillation stages are not needed. At least the weight distribution histogram could be presented as a way of gauging the effect of training in the low- versus high-training time regimes.

Another weakness is that much of the details of the study is included in the supplementary material, and not in the main text. A neurips paper should be self contained, without requiring reference to supplementary material to gain important details.
questions: Why do you think extra training time is effective for quantizing BERT models? Could you speculate as to how this extra training helps?

What would happen if you took the competing methods and merely increased their training times (without changing their architectures)?
Would a finer grid-search on learning rates be effective? Perhaps there is more gain to be had, since training of quantized networks would have rather coarse gradients, and therefore more sensitive to learning rate.

While your method avoids the time penalty of more complex distillation, it requires longer training times. What is the exact tradeoff in computational load of training between your method and the competing methods?
limitations: The authors do mention some limitations. They do not discuss any potential negative societal impacts, although they could perhaps mention positive social impacts arising from greatly reduced inference time computational loads, as well as reduced training times, both of which would lower the carbon footprint of neural networks.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides an empirical study on the compression of BERT via distillation and quantization. By fine-tuning a large amount of compressed BERT models, the authors find that longer training iterations with small learning rate can help quantization, without techniques like multi-state quantization or weights split. Based on a series of empirical observations, the authors propose XtrmC that jointly quantizes and distills an uncompressed BERT, and achieves acceptable performance with 50x model size reduction.

strengths_and_weaknesses: Strengths : 
1. The experimental results in this paper are abundant. The authors report a series of results under different settings, and present the corresponding findings in a Q&A fashion.

2. The newly found observation helps to build an efficient compressed BERT without a complicated and ad hoc training strategy.

Weaknesses: 
1. The novelty of this paper is incremental since it mainly discusses the effectiveness of the strategies proposed in the TinyBERT, TenaryBERT and BinaryBERT. In other words, it is more an engineering or application note, rather than a work of research innovation.

2. Finding 3 is similar to the discussions already provided in TinyBERT and has been verified in its subsequent works.

3. Lack of the discussion on other BERT variants like RoBERTa and generative models like GPT.

questions: 1. In Table 5, if you train a pre-trained tinyBERT with longer training iterations during fine-tuning, how does its performance compare with the Skip-BERT? For example, an empirical study can be done here: adopting the budget-C to fine-tune a 6-layer TinyBERT initialized by a teacher model. It is expected that the general information learnt from the pre-training is helpful.

2. As to finding 2, one-stage distillation with more training budgets and good setting of learning rates can benefit the final performance of the compressed model. But fine-tuning with a good learning setting may not be easy to find, compared with multi-stage distillation?

3. In Sect. 5, it is mentioned the training is using STE during BP, which is to my surprise as numerous better schemes than STE are out there in the literature (especially in those Binary NN training papers), demonstrating much better performance. An example off my head is https://github.com/lmbxmu/RBNN where there is a much more sophisticated approximation than STE.

limitations: The paper contains valuable engineering insights, yet its presentation is largely hampered by the careless writing and lack of basic proofreading (e.g. where is the mentioned Fig. 4?), as shown in my editorial comments below.


Editorial:
1. In abstract, resource-constraint should be resource-constrained?
2. P.2, bottom, celebrating should be celebrated.
3. Line 51, accuracy “los”?
4. Line 73, extend should be extent
5. Line 144, the 1-norm argument should be w or w_i?
6. Line 150, bucket-A?
7. The positioning of Tables 1 & 2 should be swapped to be in order.
8. Line 178, add “multi-stage knowledge distillation” after “claim”.
9. Line 184, x2.5 should be 2.5x
10. Number of brackets in (1) doesn’t agree!
11. Line 196, Fig. 4 is referenced in text but it is missing!
12. Line 210, Row should be Rows
13. It is clumsy in having the Best (above) rows in Table 3, why don’t you just bold the winners?
14. P.7, middle paragraph, I am confused by BERT-base and BERT_{base} notations.
15. What is Row 0 in Table 5?
16. Line 280, what is “better a trained”?
17. Line 317, “when binarization”?
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper provides a thorough understanding of the low-bit (1-bit and 2-bit) quantization for pre-trained BERT. Specifically, the authors discover four characteristics: 1) longer training and proper hyperparameter reduce the gap, 2) 1-stage knowledge distillation is okay, 3) data augmentation is essential, and 4) pre-training distillation for model compression may not be always helpful. From the findings, the paper proposes XtrmC, a model compression technique that integrates the abovementioned points. XtrmC includes a lightweight layer reduction (based on simple rules) followed by long QAT with 1-step KD and DA. The experiments show that the compression ratio of XtrmC is much higher than the previous works, such as TernaryBERT and BinaryBERT.
strengths_and_weaknesses: Overall, the paper is clearly written and well organized. Previous works are sufficiently compared, and the motivation is reasonable. The paper is supported by extensive experiments. Considering that there is no unified comparison of the various methods for low-bit BERT quantization, this paper is very timely. Thank you for your hard work!

Especially, it is interesting that pre-training distillation may not be necessary (Finding 4), which can save huge training costs to obtain the final model. It is also good news that Skip-BERT shows comparable and even better results.

However, the findings are somewhat empirical; It would increase the value of the paper if you could provide some intuitions about why the phenomenon occurs for each finding. For example, for Finding 1, how could longer training help to avoid sharp accuracy drop (or, falling into early local minima)? For Finding 3, why DA is especially important for small tasks such as CoLA and RTE?

questions: Are there any advantages of previous approaches? For example, ternary-to-binary gradual quantization, or 2-step/3-step KD? Maybe the QAT time can be reduced?

Does the longer training (Bucket-C) schedule acceptable computational cost? I basically agree that previous models are under-trained, but would be x2 or x3 epochs too heavy for practical usage?

How about the sensitivity to the random seed? BERT fine-tuning is known to be very sensitive to the random seed [1][2]. Is the proposed XtrmC robust to such randomness?

[1] Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping

[2] On the Stability of Fine-Tuning BERT: Misconceptions, Explanations, and Strong Baselines (ICLR 2021)

limitations: The authors well addressed the limitations and future research directions.
review_title: null

Meta-review: metareview: Reviewers agree that this paper presents a systematic study on the impact of hyper-parameters and training strategies of previous works. Based on those empirical observations, they propose a simplified model with layer reduction and single-stage distillation, which do not rely on a complicated and ad-hoc training strategy. Extensive experiments are conducted with thorough comparison with existing works. Authors also clearly point-out their current limitations.

The major concern is that this paper is more focused on discussion of the effectiveness of training strategies in previous methods, while the theoretical contribution is somehow limited. It would be much better if authors could explain their observations (more training epochs is needed while additional distillation stages can be discarded) from a theoretical perspective, although this may be far out of the scope of this work. Nonetheless, this paper presents valuable empirical study over existing Transformer compression methods and may inspire following research; therefore, AC recommends acceptance.

confidence: Certain
Decision: Accept