Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the task of quantizing transformers to very low precision (1/2 bit(s)). Quantizing neural networks can be essential for model size reduction (especially for mobile devices), but also for the speed of evaluation of hardware without floating point accelerators. In addition, binary activations/computations open new possibilities for high-performing low-power, special purpose hardware for evaluating neural networks with drastically improved parallelism and power consumption.

While these considerations and motivations are not new, the paper gives a concise summary and references to prior efforts, especially in the context of convolutional networks, that is clearly referenced in this work.

Here this work addresses a set of cumulative, technical improvements for successfully quantizing transformer networks for very low precision, including normalization, separate quantization for non-negative activations and exploring distillation paths over medium precision (eg. 8 bits) quantization. Using a combination of such methods, this paper manages to reduce gap in BLUE score to the full-precision model by ~3X percentage points. Also the paper gives thorough ablation analysis for the effect of the employed methods.
strengths_and_weaknesses: Originality: Moderate. The motivation for extreme quantization of neural networks is not new and has been proposed many times over the past few years. These include model size reduction and the development of new special purpose hardware that could reduce the power consumption and latency of inference of large deep learning models. In fact the feasibility of extreme quantization has been pioneered for convolutional networks with high success, but with transformers becoming the main workhorse for most deep learning applications, it has become an important question how to transfer the above results over to them.

While there has been an initial set of work of binarizing BERT and other transformer-based models, typically those results resulted in a huge drop of language-modeling quality. 

This work does not propose one single solution to those issues, but a collection of relatively common-sense solutions, the combination of which has a large effect on quality of the resulting low-precision model. These ideas include:
- Normalizing activations before binarization to ensure that low-precision activations are maximally informative.
- Improved gradient clipping for training.
- Specialized quantization for non-negative and general activations.
- Attention quantization
- Intermediate-layer distillation (employed in prior work)
- Multi-step distillation path over medium-precision models.

Quality: High. The paper has one very clear goal, a very well-defined experimental setup. Thorough experiments and ablation analysis that seem consistent. The work presents a clear analysis of the importance of all the employed methods. While most of the ideas/methods are not very novel in isolation, this work gives a clear, well-tested recipe for the extreme quantization of transformers the quality of which is significantly beyond whatever was reported earlier. While the paper omits speculating about the potential gains by special-purpose hardware, this is not a real issue, as the implications are relatively clear and these results give a clear motivation to further work in that direction.

Clarity: High. The paper is well motivated. References to prior work is quite extensive, although references to very new, concurrent works might (that has not been peer-reviewed yet), but this does not affect the final conclusion.  The paper presents a lot of experimental evidence in concise tables that back up the intuition behind the decision and highlight the significance and motivation for all of the decisions made for this work.

Significance: High. While the employed methods are relatively well known, the fact that transformers can be quantized to 1-bit precision is extremely important and this work gives a clear, well-documented measurement point and clear, well-tested recipes. This is a valuable baseline for future work, also increases the motivation for building special-purpose hardware for extremely low-precision neural networks.
questions: - It would be interesting to see whether a -1, 0, 1 quantization rather than the proposed -1, 1 would yield significant improvements at a moderate cost. Has this been attempted?
- How does quantization affect the quality of vision transformers?
limitations: - While every improvement has some potential to increase the risk of powerful technologies. It is unclear/unlikely to me that this technology would have any adverse affects beyond the generic issues of making high-impact technologies easier/cheaper to deploy by bad actors and that approximate inference might become less reliable than high-quality models. These are a general concerns regarding any performance improvements and performance trade-off. In this sense, I don't think that this approach has any non-generic risks that should have been considered in particular by the authors.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes several changes to the design of the binary BERT model. The main proposal consists of three parts: 1) use different quantization schemes based on the output distribution of a layer (e.g., activations are quantized to 0 and 1 for multi-head self attention layers); 2) adopt an elastic binarization function which allows re-scale and shift; 3) use a multi-step distillation scheme joint with a multi-step quantization schedule. The resulting BiT model demonstrates a strong performance on downstream tasks in GLUE and SQuAD datasets.
strengths_and_weaknesses: Strengths: This paper proposes several simple yet effective changes to improve the quality of Binarized BERT model significantly. The proposed techniques are sound. The writing of the paper is clear and easy to follow. The authors also provide enough empirical evaluation of the proposed BiT models.

Weaknesses: My biggest concern with this paper is that the authors do not clearly discuss the relationship to other recent works. In the absence of a clear discussion of the relationship to other related work, I am not convinced that the contributions in this work are new and original.

This paper does cite a large number of relevant works. However, the relationship between the proposals in this work and related work remains unclear. First, the proposal to use different quantization schemes depending on the output distribution of the layers is quite similar to Bi-Attention proposed in BiBERT [1]. Bi-Attention suggests using a bool function (Equation 11 in the BiBERT paper) rather than a sign function to binarize the output of a multi-head self-attention layer. Second, the elastic binarization function looks akin to the activation function proposed in Reactnet [2]. Reactnet and some follow-up work have already shown that introducing additional scaling and shifting capabilities in the activation function can help improve the quality of the binarized CNN. The authors do not point out the similarity with Reactnet when discussing the elastic binarization function, which could be misleading.

In addition, I think the paper also needs to discuss the new hardware required for the BiT model, which is different from traditional binary neural networks that require only simple XNOR gates to implement the computational engine/processing element. The computational engine for BiT models needs to handle computation between 1 and -1 and 0 and 1. I suggest that the authors should at least compare the required hardware with traditional binary and ternary neural networks. Otherwise, it would be difficult to justify not using a ternary neural network.

[1] BiBERT: Accurate Fully Binarized BERT, ICLR'22
[2] Reactnet: Towards precise binary neural network with generalized activation functions, ECCV'20
questions: I would like to hear a response on the relationship to related work and also an analysis on the hardware complexity/cost of the BiT model. In addition, the authors could clarify the training cost of the proposed multi-step distillation scheme.
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose novel binarisation techniques for transformer models based on knowledge distillation. The goal is to achieve competitive performance for an efficient student transformer model.  The main contributions are: i) binarisation technique that improves knowledge distillation in transformer models, ii) multi-step technique to distil binarised models to improve intermediate performance. The study shows that the student model achieves competitive performance compared to a standard BERT on the GLUE benchmark and question answering.
strengths_and_weaknesses: Strengths


- Clear description of background knowledge and related work needed to understand the proposed approach.  

- Clear description of the proposed approach.

- Best practices for model replicability.

- The authors perform a comprehensive comparison with related work on the GLUE benchmark.

- The findings show that an efficient transformer model has competitive performance compared to previous SOTA.


Weaknesses


- It is not clear how the parameter initialization  and selection of hyper-parameters could affect the model performance.
questions: Questions to the Authors


Please address the following questions during the rebuttal:


- Could you elaborate on the hyper-parameter selection for the teacher-student.

- A possible extra contribution is to perform multiple  random runs and report variance. However, how expensive could this exercise become? 

- Please speculate if by using other pre-training objectives in the LM task (e.g. next sentence) would it change any finding or results?
limitations: The authors have addressed limitations of the proposed approach.
review_title: null

Meta-review: metareview: This paper proposes an innovative pipeline for quantizing transformers for extremely low precision (1-2) bits, while reducing the gap of previous methods to full precision by ~3X.

This result has important implications for resource-restricted inference, especially if memory is of concern, but 1-bit quantization has significant effect on inference latency as well.

This work reaches these strong results by careful normalization, separate quantization for non-negative activations and  a combinatorial optimization over various distillation paths.

Overall, the paper demonstrates an important albeit incremental advance in the field and is of general interest to the wider community, therefore I propose its acceptance at NeurIPS 2022. 


confidence: Certain
Decision: Accept