BiT: Robustly Binarized Multi-distilled Transformer
Authors: Zechun Liu
Barlas Oguz
Aasish Pappu
Lin Xiao
Scott Yih
Meng Li
Raghuraman Krishnamoorthi
Yashar Mehdad
Venue: NeurIPS 2022 Accept (2022)
Keywords: Natural language processing
BERT
Transformers
Compression
Binary neural networks

Abstract: Modern pre-trained transformers have rapidly advanced the state-of-the-art in machine learning, but have also grown in parameters and computational complexity, making them increasingly difficult to deploy in resource-constrained environments. Binarization of the weights and activations of the network can significantly alleviate these issues, however, is technically challenging from an optimization perspective. In this work, we identify a series of improvements that enables binary transformers at a much higher accuracy than what was possible previously. These include a two-set binarization scheme, a novel elastic binary activation function with learned parameters, and a method to quantize a network to its limit by successively distilling higher precision models into lower precision students. These approaches allow for the first time, fully binarized transformer models that are at a practical level of accuracy, approaching a full-precision BERT baseline on the GLUE language understanding benchmark within as little as 5.9%. Code and models are available at:https://github.com/facebookresearch/bit.

BiT: Robustly Binarized Multi-distilled Transformer
Zechun Liu" Reality Labs, Meta Inc. zechunliu@fb. com
Barlas Oguzz Meta Al
Aasish Pappu Meta Al
barlaso@fb.com
aasish@fb.com com
Lin Xiao
Scott Yih Meta AI
Meng Li Peking University meng 11@pku edu
Meta AI linx@fb.com com
scottylhefb.com
Raghuraman Krishnamoorthi Reality Labs, Meta Inc.
Yashar Mehdad
Meta AI mehdad@fb. com
raghuraman@fb.cm com
Abstract
Modern pre-trained transformers have rapidly advanced the state-of-the sef-the-art art in machine learning. but have also grown in parameters and computational complexity, making them increasingly difficult deploy in essorcc-connstraied environments. Binarization of the weights and activations of the network can significantly alleviate these issues, however, is technically challenging from an optimization perspective. In this work, we identify series of improvements that enables binary transformers at a much higher accuracy than what was possible previously. These include a two-set binarization scheme, novel elastic binary activation function with learned parameters, and a method to quantize a network its limit by successively distilling higher precision models into lower precision students. These approaches allow for the first time, fully binarized transformer models that are at a practical level of accuracy, approaching full full-precision BERT baseline the GLUE language understanding benchmark within as little as 5.9%. Code and models are available https::///// https://githubb.cm htt.hh/..cco/cccoooereeeraaaa//a
Introduction
The past few years have witnessed tremendous advances in almost all applied fields of AI. It would hardly be simplification to say that the bulk of these advances was achieved by scaling the transformer architecture /Vaswani et al. 2017) to ever larger sizes with increasing computation budget (Devlin et al.) 2019 Liu al. 2019 Radford al. 2018 2019 Raffel al., [020 Brown et al. 2020). On the other hand, mobile devices and wearables es have proliferated and shrunk in size, with stringent requirements for storage, computation and energy consumption. Consumers demand more portability, while having access to all that current Al technology has to offer. As a result, the gap between what possible in AI, and what deployable has never been wider.
While there is a variety of methods to increase inference efficiency in neural networks (e.g. knowledge distillation, pruning), quantization has some attractive properties and has been widely successful in practice (Gholami et al. 202T). For one, storage and latency gains from quantization are deter- ministically deñned for a given quantization level. For instance, reducing the precision of model parameters by given factor immediately translates to an identical reduction in storage cost. Similarly,
Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

reducing the precision of arithmetic operations results in corresponding reduction in computational cost. Uniform quantization is hardware friendly, making it relatively simple to realize theoretical improvements in practice.
Binarization represents the extreme limit of quantization, promising 32x reduction in storage over full-precision (32-bit) models. Moreover, binary arithmetic completely climinates multiplications in favor of bit-wise XNOR operations (Courbariaux et al.) 2016 Rastegari al.) (016), enabling even further improvements when using special purpose hardware. Energy efliciency improvements between 100-1000x have been claimed be possible with binary neural networks (BNNs), over their full-precision counterparts (Nurvitadhi et al. (016).
The obvious challenge with binarization is the difficulty of optimization. While all quantization is discontinuous, higher precisions allow approximating the full-precision network to a better extent, where with BNNs, this becomes much harder. Surprisingly, researchers in computer vision have been able to demonstrate BNNs with remarkable accuracy (Liu et 20188 QQn et al.) 2020) Martinez et al.) 2020p. Unfortunately, while these works have mostly been developed on convolutional architectures for image tasks, they have not generalized well to transformer models. For instance, recent work (Qin al. 202T) has shown that a BERT (Devlin et al. 2019) model with binary weights and activations lags its full-precision counterpart by as much as 20 points on average on GLUE dataset. Even for weights-only binarization, the loss landscape was shown to too irregular, and recent work resorted to complex and specialized methods such as weight-splitting from half-width models to achieve a reasonable accuracy (Bai et al. 2021).
With this background, we tackle the problem of fully binarizing transformer models to a high level of accuracy. With the expansion trend of transformers towards becoming the standard architecture choice for all fields of AI, we believe a solution to this problem could be highly impactful.
Our approach follows the same paradigm as previous work, based on knowledge distillation (Hinton) ct al. 2015) from higher precision models using the straight-through estimator (STE) of Bengio et al. (2013). In view of the optimization difficulties, we take the following steps to ensure that the student and teacher models are well-matched:
In Section 3 we describe robust binarization framework, which allows the binary student network to better match the output distribution of the teacher. This allows us to achieve So'TA results for extreme activation quantization with BERT, producing models with little loss in accuracy down to a quantization level of binary weights and 2-bit activations and improves over previous setups by large margins in the fully binary (1-bit) setting. It also leads to competitive results for weight binarization with 4-bit activations using a single knowledge distillation step.
To further improve binary models, propose multi-distillation approach, described in Section 4 Instead of distilling directly into fully binary model, we first distill an intermediate model of medium precision and acceptable accuracy. This model then becomes the teacher in the next round of distillation into increasingly quantized models. Such a method ensures that the student model doesn't 't drift too far from the teacher, while also ensuring as good an initialization as possible. We call the resulting model BiT
In the vanilla setting without data augmentation, our approach reduces the aceuracy gap to full- precision BERT-base model by half on the GLUE (Wang et al. (019) benchmark compared to the previous SoTA. When using data augmentation, we are able to reduce the absolute accuracy gap to only 5.9 points (from over 15 points previously). In addition to the fully binary setting, also report SoTA results with binary weights and 2-bit activations, where our models trail the full-precision baseline by only 3.5 points.
2 Background
2.1 Transformer architecture
The transformer model of VVssaan al. (2017) is composed of an embedding layer, followed by N' transformer blocks and a linear output layer. Each transformer block consists of multi-head
? Short for Binarized Transformer.

---

attention layer followed by feed-forward network, as shown in Figure | layer is concatenation of K scaled dot-product attention heads, defined by:
The multi-head attention
QKT Vol
Attention K, K) -softmax
where dę is the dimension of each key, Q, V are weight matrices for the query, key and value respectively. As such, the computation in a transformer model is limited to linear matrix mul- tiplications and additions, pointwise non-linearities (most commonly Sigmoid (Han & Moraga 1995). GeLU (Hendrycks & Gimpel, 2016) or ReLU (Nair & Hinton 2010) and the Softmax operation (Bridle 1989).
2.2 Quantization
vector w is uniformly quantized b-bit precision, if its entries are restricted to the set {0, ...... 26
} for asymmetric case or { {-2", -- 1,...,2 2 1] for symmetric case, up real-valued scale a. This allows vector operations to utilize lower precision arithmetic, making them more efficient by factor of & compared full-precision calculation using B bits. The scaling operation is still in higher precision, but if the dimensionality w > H. then the extra computation is negligible.
neural network with parameters quantized to bw bits takes up B times less space. However, to take advantage of swwer-precision arithmetic, the input vectors (activations) to each vector/matrix operation also need to be quantized. A network which has weights quantized bw bits and activations quantized to ba bits is denoted as Ab, In this work, we're specifically interested in WIAI transformers. Binary arithmetic is especially attractive, since multiplications reduce to XNOR operations, and can be implemented orders of magnitude more efficiently using specialized hardware (Nurvitadhi et al. 2016
2.3 Knowledge distillation
Knowledge distillation (KD) (Hinton et al. 2015) technique whereby a student network can be trained to mimic the behavior of teacher network. This is especially useful when the student network is more efficient and easier to deploy than the more complex and cumbersome teacher. The basic way of performing KD is by using the output distribution of the teacher model (p) soft targets for training the student model. If is the student model's output, then we have the loss term:
Lugits
KL(p. q)
(I)
The advantage of KD over simple supervised training of a more efficient model that the teacher model provides richer training signal including model confidence for each output class.
For computer vision tasks, distilling the final logits solely works well for binary neural networks Liu et al. 2020). If the student and teacher architectures are compatible, one can also distill intermediate activations for faster convergence and better transfer and generalization (Aguilar et al. 2020):
Creps
Σ ln? -
where rf and rf are the corresponding transformer block output activations from student and teacher.
3 Robust binarization setup
In this section we first bring together some best practices and minor improvements which we have found helpful in simplifying previous work and building strong baseline. Then present novel activation binarization scheme, which we will show to be critical to achieve good performance.
3.1 Two-set binarization scheme
In contrast to convolutional neural networks images where activations exhibit comparable distribu- tions, different activations in transformer blocks are performing different functionalities, and thus vary in their output distributions. In particular, these activations can be divided into two categories: the
3

---

Binary Embeddings
Binarize Activation
Binarize Activation
Binary Weight FC Layer
Binary Weight Q
Binary Weight FCK
Binary Weight FCV
ReLU
Binarize Activation (0.1)
Binarize Activation
Binarize Activation
Binarize Activation
Softmax
Binary Weight FC Layer
Binarize Activation (0.1)
Binarize Activation
Binary Weight FC
Feed-Forward Network
Self-Attention
Transformer Block Output
Figure 1: Overview of BiT. A transformer block contains the multi-head self-attention and feed- forward network. We binarize all the weights to (-1, 1} in the Embeeddin/Fully-Conoie -olne---nneeeedd layers and binarize activations to (0, 1} for ReLU/Softmax outputs and (-1, 1] for other layers.
activations after Softmax/Rel.U U layer that contains positive values only and the remaining activations with both positive and negative values (e.g., after matrix multiplication). If we denote by Xn the vector of activation values, then the two cases are X R E I4 and X IR respectively.
For the former set, mapping to the binary levels {-1, would result in severe distribution mismatch. Therefore we instead map non-negative activation layers to XB E {0, 1}" and binarize activation layers with XR E !I to XB E {-1, 1,", shown in Figure [ A prior work BiBERT Qin et al. 2021) also suggests binarizing attention to {0, but with bool function replacing Soft while empirically find that simply binarizing attentions after SoftMax to (0, 1) works better and binarizing ReLU output {0, 1} instead of (-1, brings further improvements. (See Section A.3 for details).
Optimal sealing factor two sets Additionally, we apply layer-wise scaling factor to binarized activations reduce the binarization error, iee., XB a The optimal values of are different for the Xn E (0, 1) and Xn {-,1, cases and can be calculated by minimizing the /2 error: [(a) = XR a1
a" arg min J (a) AER
(3)
Following XNOR-Net (Rastegari et al. 2016), by expanding Eq. we have
a
T
AB - 2aXR "8B + XR TXR
(4)
For the layers with XR Rn follow the traditional methods of binarizing activations (Rastegari et al. 2016 Liu et al. 2018) by taking the sign of real-valued activations:
1, if X <
X = SigniigX (x
-1,
if X R 2
In that case, X3 XB nXn. where nxn is number of elements in XR. and a' can be solved as:
Xn T T
|Ralla
(6)
0XR
RXR
For the activations in attention layers or after the ReLU non-linearity layers with XR E R7. we binarize the activations to Xn {0,1) 1)" by rounding the real-valued activations:
x B/3 CCiip X O, 0, = = 1, if X > 0.5
0. if X < 0.5

---

T "B = 3(X200.5) where n (xX320.5) denotes the number of elements Xn that In that case, XB are greater than or equal to 0.5. Then a* can be solved as:
|R (xx2006)ll
a"
(8)
n(x230.5)
3.2 Best practices
We performed thorough experimentation and discovered the following modifications to be useful.
Simplified knowledge distillation Compared previous BERT model binarization works (Bai et 2021; Qin et al., 2021) which also attempt to distill the attention scores, we provide analysis and experimental results (Section (53) showing that using only Lreps from transformer block outputs and Lagits is more effective while being simpler. We also forego the two-step distillation scheme of Bai al. (202T) favor of a single step, joint distillation, where our training loss is simply Llogits + Lreps-
Mean subtraction in weight binarization For weight binarization, centeralizing the real-valued weights be zero-mean before binarization can increase the information carrying capacity of the binary weights. Thus, for weight binarization, we have: W
Walln Sign( Wir nwR
WR).
Gradient clipping Clipping gradients to 0 when X & [-1, (or & of & E {0, 1)")
common technique for training binarized neural networks. However, find that clipping weight gradients is harmful for optimization. Once a weight is outside of the clip range, the gradient is fixed for activations, since the activation value changes for each input. As result, we apply gradient clipping only to activations but not to weights.
0, preventing further learning. This
Non-linearity We prefer ReLU activations whenever the output range is non-negative.
Combining these, we are able build strong baseline, which improves the accuracy by 9.6% over naively binarized transformers. Additionally, these techniques allow us to train a weight binarized transformer network in a single training step using knowledge distillation (i.e., without resorting to weight splitting as in BinaryBert (Bai et al. 2021)) (See Section 5.3 for details).
3.3 Elastic binarization function
The fixed scaling and threshold derived previously works reasonably well, but might not be optimal since ignores the distribution of the variable which is being binarized. Ideally, these parameters can be learned during training to minimize the target loss.
When using classical binarization methods, i.e., X3 Sign(X
) the binary output is independent of the scale of the real-valued input. However, in our case where X B = |Clip 0, 1)7, this independence no longer holds. Learning the scaling and threshold parameters, and how to approximate the gradients precisely in the process becomes crucial for the final accuracy.
To handle this, inspired by the learnable threshold in ReActNet (Liu et al. elastic binarization function to learn both the scale a E IR., and the threshold 3 R:
2020
propose the
X - 0,
X = ax = a Clipp
(9)
In the function, we initialize a with a* in Sec. 3.1 and 9 to be 0, and train it with gradients from the
final loss. To back-propagate the gradients to through the discretized binarization function, we follow the practice in Choi al. 2018); 7hou et al. (2016); [isser et al. (2019) to use straight-through estimator (STE) (Bengio et al. 2013) to bypass the incoming gradients to the round function to be
5

---

Algorithm BiT: Multi-distillation algorithm
Require: Diin; Daev
D Training Data
Require: he
Full-preciste Model
Require: Q {Bbb: ba)...... (bur bw)
Quantization Schedule
1: Traacher ho Q
for bin
3: Assudese - (Meatize bw 6/)
4: KnowledgeDistill Ceiisttll(AAuuue Astueent, Pitaateer Deain, D
5:
Areacher haadeen
6:
end for
7:
return hssseent
the outgoing gradients:
0x
0x da
B
X +
B
da
STE ~3
X +
aClip(
Xi 8 0001)
da
(10)
0
X <B
8-
X
if 3 3 X <a/2+3
X
8
if a/2 + 3 < X R <a+3
if X
2 a
R
Then the gradients w.r. 33 can be similarly calculates as:
ox:
STE ~
8Clip(
X
.0
if X < -1,
(11)
08
03
0,
otherwise
For the layers that contain both positive and negative real-valued activations i.e., XR E R the binarized values XB E (-1, 13" are indifferent to the scale inside the Sign function:
x a Sign X X a Sign(X - -3)
(12)
a
In that case, since the effect of scaling factor inside the Sign function can be ignored, the gradient w.E.t can be simply calculated as:
ox da
Sign(Xi - 9)
(13)
In our ablations (Section 5.3 and A.2) we show that using this simple elastic binarization function can bring a 15.7% accuracy boost over our strong baseline on the GLUE benchmark.
4 Multi-distilled binary transformer
Classical knowledge distillation (KD) (Hinton et al. 2015) trains the outputs (i.e., logits) student network to be close to those of a teacher, which is typically larger and more complex. This approach is quite general, and can work with any student-tcacher pair which conforms to the same output space. However, in practice, knowledge transfer happens faster and more effecttivlyy if the intermediate representations are also distilled (Aguilar et al. 2020). This approach has been found useful when distilling to student models with similar architecture (Sanh et al. 2019), and in particular for quantization [Bal et al. 2021
Kim et al. 2019).
Note that having similar muudent-teached pair is requirement for distilling representations, While how similar they need to is an open question, intuitively a teacher which architecturally closer to the student should make transfer of internal representations easier. In the context of quantization, it easy to see that lower precision students are progressively less similar to the full-precision teacher, which is one reason why binarization is difficult.

---

Table I: Comparison of BERT quantization methods on the GLUE dev set. The E-W-A notation refers
to the quantization level of embeddings, weights and activations. denotes distilling binary models
using full-pree full-precision teacher without using multi-distill technique in Section *Data augmentation
not needed for MNLI, QNLI, therefore results in the data augmentation section are identical to that
without data augmentation for these datasets.
Quant
Min
aNLL
QOP ONLI
SST2
COLA
STS-K
MRPC
Avg.
BERT
32-32-32
118
22.3
31.8835
91.1
92
93.2
59.7
90.
S6.3
72.2
83.9
Wilkowl dala ongmeniation
O-BERT
2-8-8
43.0
6.5
76.6/77.0
$4.6
68.3
52.7
Q2BEKT
43.0
4
67.0
61.3
0
4.4
68.4
52.7
47.7
TemnaryBERT
50.7
BimmrBBE:T BERT
16.5
84 24277477
91.2
91.5
92.6
53.4
88.6
85.5
72.2
82.7
BinaryBERT
16.5
83.9/8422
91.2
90.9
92.3
44.4
87.2
S3.3
65.3
79.9
Ri:aryRl:RT
1-1-2
16.3
D8
62.7/33 9
79.9
52.6
82.3
14.6
6-S
68.1
52.7
53.7
BERT
1-1-1
16.5
044
35.6/35 3
66.2
51.5
53.2
0
6.1
68.3
$2.7
41.0
BiBERT
1-1-1
13.4
04
66 1/67
84.8
72.6
88.7
25.4
33.6
72.5
57.4
63.2
"HY
W7.8
91""
991.5"
66.4"
"INE""
89.3
32.1
B1T
79.3/794
85.4
N6.4
89.9
32,9
72.0
79.9
73.5
With data ouomentatin
TemaryBEKK
2-2-8
28.0
6.4
83 3
90.1
90.0
92.9
47.8
84.3
82.6
68.4
80.3
BinarBERT BERT
1-1-8
16.5
3.1
84 2/84 7.
91.22
91.6
93.2
55.5
89.2
86.0
74.0
83.3
Binury BERT
1-1-4
16.5
83 9/9424
91.2*
91.4
93.7
53.3
88.6
$6.0
71.5
82.6
RinoryRl-KT
1-1-2
16.5
(2. 7/63.9
79.9*
$1.0
89.6
33.0
71.0
$8.9
57.6
BinarBERR
BIBERT
13.4
16.1967.55
84.8
76.0
90.99
37.8
56.7
78.8
61.1
68.
"
122
132
9225
43.2
99074"
72.9
W012""
BAT
1-1-1
13.4
044
77. 1/77
82.9
85.0
91.5
32.0
84.1
88.0
67.5
76.0
BIT
1-1-1
13.4
0.4
79.5/79.4"
85.4
86.5
92.3
38.2
84.2
88.0
69.7
78.0
This suggests a multi-step approach, where instead directly distilling from a full-precision teacher to the desired quantization level, first distill into model with sufficient precision in order to preserve quality. This model can then be used as a teacher to distill into a further quantized student. This process be repeated multiple times, while at each step ensuring that the teacher and student models are sufficiently similar, and the performance loss is limited. This multi-distillation approach sketched in Algorithm
The multi- step distillation follows a quantization schedule, Q (1b b. (b b,"), (b b,")) with (6 6,¹) (b.2 b²²) ... > (b 1 b (b. 6,") is the target quantization level, which is in our case binary for both weights and activations. In practice, we find that down to quantization level of W1A2, we can distill models of reasonable accuracy single shot, following the best practices outlined in Section 3.2 (See our 1-1-2 baseline results in Table D As result, we follow a fixed quantization schedule, W32A32 - W1A2 + WIAI. This not necessarily optimal, and how to efficiently find the best quantization schedule is an interesting open problem. We present our initial explorations towards this direction in Section 5.5
Combining the clastic binary activations with dinlti-distillation we obtain BiT, the robustly binarized multi-distileed transformer. Note that BiT simultaneously ensures good initialization for the eventual (in our case binary) student model. Since the binary loss landscape is highly irregular, good initial- ization is critical to aid optimization. Previous work has proposed progressive distillation (Zhuang et al. 2018; Yang et al. 2019) to tackle this problem, wherein the student network is quantized at increasing severity as the training progresses. However, this method does not prevent the student network from drifting away from the teacher, which is always the full full-precision model. We compare to progressive distillation in Section A.
5 Main results
We follow recent work (Bai et al.) 2021 Qin et al. 2021 in adopting the experimental setting of Devlin et al. (2019), and use the pre-trained BERT-base our full-precision baseline. We evaluate on GLUE (Wang et al. 2019). a varied set of language understanding tasks (see Section A.5 for full list), well SQuAD (v1.1) (Rajpurkar et al. 2016), popular machine reading comprehension dataset.
(a, (c. if cand dor and d.
7

---

Table 2: Ablation study on the effects of each component on GLUE dataset without data augmentation.
Quant
miwe
QQP
QNLI
SST.2
STS.B
Avg.
BERTha
84.9/85.5
91.4
92.1
v5.2
59.7
90.1
86.3
72.2
K39
BiBERT Baseline
45.8/47.0
73.2
66.4
77.6
11.7
7.6
70.2
$4.1
50.4
BiBERT
66.1/67.5
84.8
72.6
88.7
25.4
33.6
72.5
57.4
63.2
BinaryBURT (Our impleteentitionn)
36.2/35.9
59.6
52.4
65.6
9.3
19.8
69.9
52.7
45.7
Our simplied
37,7137.3
56.8
734
24.8
480
Ouar twa-set vatiooatiinn www.Sooooonnnnne
57,4/59.1
64.7
81.0
24.7
" Elbatic binanization (ELY')
77.1/77.3
""'"""
85.7
87.7
71:1
79.7
""1""""""""""
+
Multi-distillationn (8:77
79.5779.4
N5.4
N6.4
89.9
32,9
72.0
79.9
62.1
73.5
5.1 GLUE results
Our main results on the GLUE benchmarks are presented in Table In the setting without data augmentation, where we only use the original training samples for knowledge distillation, we are able to reduce the gap the full precision baseline by 49.8%, i.e., from 20.7 in (Qin et al. 202T to 10.4 points. We also see that our baseline models with elastic activation binarization already improve previous SoTA by large margins.
In the binary weight setting (4-bit activations), we can match or outperform Bai et al. the need for pre-training half-width models and subsequently splitting weights. This result should make binary weight models much easier to implement and deploy.
2021 without
We also set new state of the art for binary weight 2-bit activation (W1A2) models, with only a 3.5 point degradation compared to the full-precision baseline (using data augmentation). While not as efficient as binary, 2-bit arithmetic can also be performed without multiplications, making it good efficient alternative in applications where the performance cost of going to fully binary is significant.
5.1.1 Data augmentation
From Table I it can be observed that the datasets with small training sets still have a large gap from the full-precision baseline. As result, we employ data augmentation heuristics (following the exact setup in Zhang et al. (2020) the datasets with small training sets (all except MNLI, QNLI) to take better advantage of our model's strong representational capability. This further reduces the quantization gap, with our models eventually trailing the full- full-precision model by only 5.9 points average on the GLUE benchmark.
5.2 SQuAD D results
We also evaluate on the popular machine reading comprehension (MRC) dataset from Rajpurkar al. (2016). We compare to our lation of the MRC task on top of the BiBERT codebase, since SQuAD results are not reported in that work. We also show results using the BinaryBERT codebase, without using weight splitting. The results (Table 3) show that this task significantly harder than most document classification benchmarks in GLUE, and previ-
Table Comparison of BERT quantization meth- ods on SQuADv1.1 dev set. Metrics are exact match and Fl score.
implemen-
Quant
BBils
SQuADrL.
BERT
32-32-32
82.6/89.7
BinuryBERT
1-14
77.9/85.8 8
BinuryBERT
1-1.2
72.3/81.8
Binary BERT
1-1-1
1.5/8.2
BiBERT
1-1-1
8.5/18.9
BITT"
CET
63.177499
binarization methods fail achieve any meaningful level of performance. BiT does much better, but still trails the 32-bit baseline by 14.8 points in F1. We conclude that despite the improvements we have demonstrated on the GLUE benchmark, binarizing transformer models accurately is far from solved problem in general.
5.3 Ablations
We start from the basic binarization implementation from Bai et al. (2021), and add each of our contributions in sequence to get a better idea how each contributes to the performance. The results shown in Table
We start by removing attention distillation (Section 3.2 which results in a 2.3% improvement (row vs. 4). Then switching two-set binarization (Section 3.1), which binarizes the attention scores

---

Query FC input
0.91
0.85
0.86
0.65
0.64
0.47
0.44
0.64
0.67
68
Key FC Input
0.97
0.93
0.87
0.6
0
0.51
0.48
0.64
0
64
7
6.65
Value Imput
Query FC output
49
68
93
83
74
33
34
Key FC output
1.56
66
1.19
Value output
1.17
0.56
0.77
0.1
0.32
0.1
0.55
Attentton
0.01
0.01
0.04
0.01
0.01
0.1
0.03
0.12
0.01
0.01
0.01
0.01
Self-Attention Input
0.0
FFN 1st input
0.05
0.06
0.06
0,06
0.05
0.03
0.03
0.03
08
FFN 2nd FC Input
0.23
0.2
0.14
0.12
0.11
0.11
0.12
0.11
0.04
0.05
0.04
Figure 2: The optimized scaling factor in BiT
differently than the feed-forward activations, which gives additional 7.3% boost (row 6). This results in a much stronger baseline than what was used in prior works (row 2).
Moving from fixed to elastic binarization (Section 3.3 proves hugely important, pushing the average accuracy to 71.0% (row 7) from only 55.3% (row 6). Note that this model already outperforms the current state-of-the-art (row 3) by 7.8% points. Finally, we add multi-step distillation (Section +
which adds another 2.5 points, reaching the final accuracy of 73.5% on the GLUE benchmark.
5.4 Learned parameter visualization
We visualize the optimized a in the final BiT model. As we can see from Figure 2 the values of the parameters vary significantly from layer to layer, and have apparent patterns according layer characteristics. For example, the attention layers need to distribute the attention to different entries, thus the scaling factor for the attentions are learned to be small, while the scaling factors for query and key outputs are usually larger. Note that the biggest value is x00x of the smallest o, suggesting the importance learning a dynamically.
5.5 Exploring multi-distillation paths
So far we have only considered the fixed quantization schedule, W32A32 + W1A2 - WIAL This motivated by early experiments showing that step distillation W1A2 works reasonably well. We explored other optimal schedules, such as distilling to W1A8 resulted in higher accuracy model, thus better teacher to distill down to the eventual W 1A1 student. This suggests trade-off between the quality of the intermediate model, vs. the closeness to the target quantization level.
Figure 3 illustrates this trade-off. We that two-step distillation improves over one-step in every case. While higher precision interme- diate models are better as expected, it is better to use lower precision teacher in the last step since it makes the learning task for the binary student model easier. The closeness the target quantization is favored despite the lower accu- racy of teacher model.
MNLI-m (matched) accuracy
84
82
W022322-W14J--WAA.
W92412-WW41
76
W32432 wWL44 WWIA2 WWIA
of course possible, though more cumber- some, to perform more than two distillation steps. We also experiment with a three-step schedule, W32A32 + W1A4 WIAI, which is plotted in the same figure (dashed line). We find that this particular 3- step schedule does not improve over the 2-step schedule W32A32 W1A2 WIAL While this result does not preclude existence of other more optimal schedules, we hypothesize that this is unlikely.
32
Activation bits
Figure 3: MNLI-m accuracy various distillation paths. Each curve represents the sequence of mod- els particular quantization schedule.
W1A2
6 Related work
Convolutional neural network quantization Neural network quantization a practical tool for
compressing model size and reduce storage (Hubara et al. 2017). Quantization for convolutional

---

neural networks has been studied both in the uniform quantization (Choi et al. 2018; Zhou et al.
2016 Gong al. 2019) and non-uniform quantization (Zang et al. 2018 Miyashita et al. 2016
et al. 2020p settings. The quantization level has been progressively increased, from 8-bit (Wang et al.
2018
Zhu et al. 2020) to 4-bit (Jung et al. 2019) Esser al. 2019 Liu et al. 2022) and finally to
the extreme 1-bit case (Courbariaux et al. 2016 Rastegari et al. 2016 Liu et al. 2018 Martinez
ct al.
2020).
Transformer quantization Compared to the CNNs, transformers with attention layers are naturally more challenging to quantize (Bondarenko et al. 202T). Previous research mainly focused on 8-bit
quantization Zafirir al., 2019] Fan et al. (2200) or 4-bit quantization (Shen al. 2020 Zadeh
et al. 2020). Extremely low-bit quantization for transformers has only been attempted very recently,
BERT (Zhang et al. 2020) proposed ternarize the full-precision weights of fine-tuned BERT model. As a follow-up to TemaryBERT, weight binarization was proposed in Bai et al. (2021). Here, the network is trained by first training a ternary half-sized model, which used as initialization. Then weight-splitting step results in a full-sized binarized model, which is further tine-tuned in subsequent distillation step. Binarizing both weight and activations in transformer has proved be
challenging. BiBERT (Qin et al. (202T) made the first attempt in this direction with limited success. Their model performed 20% worse than real-valued baseline on the GLUE benchmark (Wang et al. 2019), which even underperforms the original LSTT baselines.
7 Conclusion
Large pre-trained transformers have transformed NLP and are positioned to serve as the backbone for all AI models. In this work, we presented the first successful demonstration of a fully binary pre-trained transformer model. While our approach is general and can be applied to any transformer, we have limited our evaluation to BERT-based models on the GLUE and SQuAD benchmarks. remains to be seen how conclusions will hold when applied to the wide variety of pre-trate pre-trained transformer models which have gained popularity in recent years, from small mobile models, to gigantic ones with hundreds of billions of parameters. It will also be interesting of the approach on different domains (such as image and speech processing) and tasks (such as text and image generation). Demonstrating the generality of this approach in wider setting should significantly widen its impact, therefore we identify this as an important future direction.
the performance
References
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Chenlei Guo. Knowledge distillation from internal representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7350-7357, 2020.
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael R Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. In ACL/UCNLP (I), 2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arxiv:1308.3433. 2013.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7947-7969, 2021.
John Bridle. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. Advances in neural information processing systems, 2, 1989.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, et al. Pact: Parameterized clipping activation for quantized neural networks. arXiv e-prints, pp. arXiv-1805, 2018.
10

---

Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint arXiv: /602.02800 2016.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HIL (1), 2019.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra Modha. Learned step size quantization. In International Conference on Learning Representations, 2019.
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme model compression. arXiv preprint arXiv:2004.07320, 2020.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.136600 2021.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full full-precision and low-bit neural networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2019.
Jun Han and Claudio Moraga. The influence of the sigmoid function parameters on the speed of backpropagation learning. In International workshop on artificial neural networks, pp. 195-201. Springer, 1995.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
1606:41606.08415. 2016.
Geoffrey Hinton, Oriol Vinyals. Jeff Dean, al. Distilling the knowledge in a neural network. arXiv preprint 1XX3::1503.0253: 2(7), 2015.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv,,, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869 6898, 2017.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4350-4359, 2019.
Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, and Nojun Kwak. Qkd: Quantization-aware knowledge distillation. arXiv preprint 1X11v:1911.12491 2019.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. In International Conference on Learning Representations, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining 2019.
approach. arXiv preprint
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhaneing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pp. 722-737, 2018
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. European Conference on Computer Vision, pp. 143-159. Springer, 2020.
11

---

Zechun Liu, Kwang- Ting Cheng, Dong Huang. Eric P Xing, and Zhiqiang Shen. Nonuniform-to uniform quantization: Towards accurate quantization via generalized truight-through estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4942 4952, 2022.
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. In ICLR, 2020.
Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint arXiv: 2016.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In lcml, 2010.
Eriko Nurvitadhi, David Sheffield, Jaewoong Sim, Asit Mishra, Ganesh Venkatesh, and Debbie Marr Accelerating binarized neural networks: Comparison of fpga, cpu, gpu, and asic. In 2016 International Conference on Field-P roel-PPoogrummable Technology (FPT), pp. 77-84. IEEE, 2016.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In CVPR, 2020.
Haotong Qin. Yifu Ding, Mingyuan Zhang, YAN Qinghua, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. Bibert: Accurate fully binarized bert. In International Conference Learning Representations, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang, Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 2019.
Sheng Shen, Zhen Dong. Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 8815-8821, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Y Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. Advances in neural information processing systems, 31, 2018.
Yifan Yang. Qijing Huang, Bichen Wu, Tianjun Zhang, Liang Ma, Giulio Gambardella, Michaela Blott, Luciano Lavagno, Kees Vissers, John Wawrzynek, et al. Synetgy: Algorithm-hardware co-design for convnet accelerators on embedded fpgas. In Proceedings of the 2019 ACM/SIGDA international symposium on field- programmable gate arrays, pp. 23-32, 2019.
12

---

Ali Hadi Zadeh, Isak Edo. Omar Mohamed Awad, and Andreas Moshovos. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. In 2020 53rd Annual IEFF/ACM International Symposium on Microarchitecture (MICRO), pp. 811-824. IEEE, 2020.
Ofr Zaftir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurllP Edition (EMC2-NIPH)) Pp. 36-39. IEEE, 2019.
Dongqing Zhang, Jiaolong Yang, Donggiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV). pp. 365-382, 2018.
Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Temaryyber: Distillation-wware ultra-low bit BERT. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), EMNIP, 2020.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 1606666666660 2016.
Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuqi Yang, and Junjie Yan. Towards unified int8 training for convolutional neural network. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1969-1979, 2020.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Linggiao Liu, and Ian Reid. Towards effective low- bitwidth convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7920-7928, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In the conclusion.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] In supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See section ??.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] In section ??.
4. If you using existing assets (c.g., code, data, models) or curating/releusing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See section ??.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
13

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "xNeokIJRg8O", "reviewer": "Reviewer_2rah", "summary": "This paper proposes several changes to the design of the binary BERT model. The main proposal consists of three parts: 1) use different quantization schemes based on the output distribution of a layer (e.g., activations are quantized to 0 and 1 for multi-head self attention layers); 2) adopt an elastic binarization function which allows re-scale and shift; 3) use a multi-step distillation scheme joint with a multi-step quantization schedule. The resulting BiT model demonstrates a strong performance on downstream tasks in GLUE and SQuAD datasets.", "questions": "I would like to hear a response on the relationship to related work and also an analysis on the hardware complexity/cost of the BiT model. In addition, the authors could clarify the training cost of the proposed multi-step distillation scheme.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: This paper proposes several simple yet effective changes to improve the quality of Binarized BERT model significantly. The proposed techniques are sound. The writing of the paper is clear and easy to follow. The authors also provide enough empirical evaluation of the proposed BiT models.\n\nWeaknesses: My biggest concern with this paper is that the authors do not clearly discuss the relationship to other recent works. In the absence of a clear discussion of the relationship to other related work, I am not convinced that the contributions in this work are new and original.\n\nThis paper does cite a large number of relevant works. However, the relationship between the proposals in this work and related work remains unclear. First, the proposal to use different quantization schemes depending on the output distribution of the layers is quite similar to Bi-Attention proposed in BiBERT [1]. Bi-Attention suggests using a bool function (Equation 11 in the BiBERT paper) rather than a sign function to binarize the output of a multi-head self-attention layer. Second, the elastic binarization function looks akin to the activation function proposed in Reactnet [2]. Reactnet and some follow-up work have already shown that introducing additional scaling and shifting capabilities in the activation function can help improve the quality of the binarized CNN. The authors do not point out the similarity with Reactnet when discussing the elastic binarization function, which could be misleading.\n\nIn addition, I think the paper also needs to discuss the new hardware required for the BiT model, which is different from traditional binary neural networks that require only simple XNOR gates to implement the computational engine/processing element. The computational engine for BiT models needs to handle computation between 1 and -1 and 0 and 1. I suggest that the authors should at least compare the required hardware with traditional binary and ternary neural networks. Otherwise, it would be difficult to justify not using a ternary neural network.\n\n[1] BiBERT: Accurate Fully Binarized BERT, ICLR'22\n[2] Reactnet: Towards precise binary neural network with generalized activation functions, ECCV'20", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZB2GSywGDFN", "reviewer": "Reviewer_Yoaz", "summary": "The authors propose novel binarisation techniques for transformer models based on knowledge distillation. The goal is to achieve competitive performance for an efficient student transformer model.  The main contributions are: i) binarisation technique that improves knowledge distillation in transformer models, ii) multi-step technique to distil binarised models to improve intermediate performance. The study shows that the student model achieves competitive performance compared to a standard BERT on the GLUE benchmark and question answering.", "questions": "Questions to the Authors\n\n\nPlease address the following questions during the rebuttal:\n\n\n- Could you elaborate on the hyper-parameter selection for the teacher-student.\n\n- A possible extra contribution is to perform multiple  random runs and report variance. However, how expensive could this exercise become? \n\n- Please speculate if by using other pre-training objectives in the LM task (e.g. next sentence) would it change any finding or results?", "limitations": "The authors have addressed limitations of the proposed approach.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n\n- Clear description of background knowledge and related work needed to understand the proposed approach.  \n\n- Clear description of the proposed approach.\n\n- Best practices for model replicability.\n\n- The authors perform a comprehensive comparison with related work on the GLUE benchmark.\n\n- The findings show that an efficient transformer model has competitive performance compared to previous SOTA.\n\n\nWeaknesses\n\n\n- It is not clear how the parameter initialization  and selection of hyper-parameters could affect the model performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QnFRmiWhdz9", "reviewer": "Reviewer_4xBa", "summary": "This paper addresses the task of quantizing transformers to very low precision (1/2 bit(s)). Quantizing neural networks can be essential for model size reduction (especially for mobile devices), but also for the speed of evaluation of hardware without floating point accelerators. In addition, binary activations/computations open new possibilities for high-performing low-power, special purpose hardware for evaluating neural networks with drastically improved parallelism and power consumption.\n\nWhile these considerations and motivations are not new, the paper gives a concise summary and references to prior efforts, especially in the context of convolutional networks, that is clearly referenced in this work.\n\nHere this work addresses a set of cumulative, technical improvements for successfully quantizing transformer networks for very low precision, including normalization, separate quantization for non-negative activations and exploring distillation paths over medium precision (eg. 8 bits) quantization. Using a combination of such methods, this paper manages to reduce gap in BLUE score to the full-precision model by ~3X percentage points. Also the paper gives thorough ablation analysis for the effect of the employed methods.", "questions": "- It would be interesting to see whether a -1, 0, 1 quantization rather than the proposed -1, 1 would yield significant improvements at a moderate cost. Has this been attempted?\n- How does quantization affect the quality of vision transformers?", "limitations": "- While every improvement has some potential to increase the risk of powerful technologies. It is unclear/unlikely to me that this technology would have any adverse affects beyond the generic issues of making high-impact technologies easier/cheaper to deploy by bad actors and that approximate inference might become less reliable than high-quality models. These are a general concerns regarding any performance improvements and performance trade-off. In this sense, I don't think that this approach has any non-generic risks that should have been considered in particular by the authors.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: Moderate. The motivation for extreme quantization of neural networks is not new and has been proposed many times over the past few years. These include model size reduction and the development of new special purpose hardware that could reduce the power consumption and latency of inference of large deep learning models. In fact the feasibility of extreme quantization has been pioneered for convolutional networks with high success, but with transformers becoming the main workhorse for most deep learning applications, it has become an important question how to transfer the above results over to them.\n\nWhile there has been an initial set of work of binarizing BERT and other transformer-based models, typically those results resulted in a huge drop of language-modeling quality. \n\nThis work does not propose one single solution to those issues, but a collection of relatively common-sense solutions, the combination of which has a large effect on quality of the resulting low-precision model. These ideas include:\n- Normalizing activations before binarization to ensure that low-precision activations are maximally informative.\n- Improved gradient clipping for training.\n- Specialized quantization for non-negative and general activations.\n- Attention quantization\n- Intermediate-layer distillation (employed in prior work)\n- Multi-step distillation path over medium-precision models.\n\nQuality: High. The paper has one very clear goal, a very well-defined experimental setup. Thorough experiments and ablation analysis that seem consistent. The work presents a clear analysis of the importance of all the employed methods. While most of the ideas/methods are not very novel in isolation, this work gives a clear, well-tested recipe for the extreme quantization of transformers the quality of which is significantly beyond whatever was reported earlier. While the paper omits speculating about the potential gains by special-purpose hardware, this is not a real issue, as the implications are relatively clear and these results give a clear motivation to further work in that direction.\n\nClarity: High. The paper is well motivated. References to prior work is quite extensive, although references to very new, concurrent works might (that has not been peer-reviewed yet), but this does not affect the final conclusion.  The paper presents a lot of experimental evidence in concise tables that back up the intuition behind the decision and highlight the significance and motivation for all of the decisions made for this work.\n\nSignificance: High. While the employed methods are relatively well known, the fact that transformers can be quantized to 1-bit precision is extremely important and this work gives a clear, well-documented measurement point and clear, well-tested recipes. This is a valuable baseline for future work, also increases the motivation for building special-purpose hardware for extremely low-precision neural networks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept