BiBERT: Accurate Fully Binarized BERT
Authors: Haotong Qin
Yifu Ding
Mingyuan Zhang
Qinghua YAN
Aishan Liu
Qingqing Dang
Ziwei Liu
Xianglong Liu
Venue: ICLR 2022 Poster (2022)
Keywords: Network Binarization
Model Compression
BERT
NLP

Abstract: The large pre-trained BERT has achieved remarkable performance on Natural Language Processing (NLP) tasks but is also computation and memory expensive. As one of the powerful compression approaches, binarization extremely reduces the computation and memory consumption by utilizing 1-bit parameters and bitwise operations. Unfortunately, the full binarization of BERT (i.e., 1-bit weight, embedding, and activation) usually suffer a significant performance drop, and there is rare study addressing this problem. In this paper, with the theoretical justification and empirical analysis, we identify that the severe performance drop can be mainly attributed to the information degradation and optimization direction mismatch respectively in the forward and backward propagation, and propose BiBERT, an accurate fully binarized BERT, to eliminate the performance bottlenecks. Specifically, BiBERT introduces an efficient Bi-Attention structure for maximizing representation information statistically and a Direction-Matching Distillation (DMD) scheme to optimize the full binarized BERT accurately. Extensive experiments show that BiBERT outperforms both the straightforward baseline and existing state-of-the-art quantized BERTs with ultra-low bit activations by convincing margins on the NLP benchmark. As the first fully binarized BERT, our method yields impressive 56.3 times and 31.2 times saving on FLOPs and model size, demonstrating the vast advantages and potential of the fully binarized BERT model in real-world resource-constrained scenarios.

Published as a conference paper at ICLR 2022
BIBERT: ACCURATE FULLY BINARIZED BERT
Haotong Qin Yifu Ding" Mingyuan Zhang", Qinghua Yan', Aishan Liu',
Qingqing Dang", Ziwei Liu?, Xianglong Liu
State Key Lab of Software Development Environment, Beihang University Baidu Inc. L--aa, Nanyang Technological University *Shen Yuan Honors College, Beihang University {qinhacteng, yiuuing yanqa, aishanliu, xlliu)@buaa, edu. en mi iinnyannn0 (e. antu.edu sg zw1 uu. h 1 com dangqi ngqi ng@ba domcom
ABSTRACT
The large pre-trained BERT has achieved remarkable performance on Natural Language Processing (NLP) tasks but also computation and memory expen- sive. As one of the powerful compression approaches, binarization extremely reduces the computation and memory consumption by utilizing 1-bit parame- ters and bitwise operations. Unfortunately, the full binarization of BERT (iee. 1-bit weight, embedding, and activation) usually suffer a significant performance drop, and there is rare study addressing this problem. In this paper, with the theoretical justification and empirical analysis, we identify that the severe perfor- mance drop can be mainly attributed to the information degradation and optimiza- tion direction mismatch respectively in the forward and backward propagation, and propose BiBERT, accurate fully binarized BERT, to eliminate the per- formance bottlenecks. Specifically, BiBERT introduces an efficient Bi-Attention structure for maximizing representation information statistically and a Direction- Matching Distillation (DMD) scheme to optimize the full binarized BERT accu- rately. Extensive experiments show that BiBERT outperforms both the straight- forward baseline and existing state-of-the- activations by convincing margins on the NLP benchmark. As the first fully bina- rized BERT, our method yields impressive 56.3x and 31.2x saving on FLOPs and model size, demonstrating the vast advantages and potential of the fully binarized BERT model in real-world srource-constrrined scenarios.
quantized BERTs with ultra-low bit
DBCATI (oun)
INTRODUCTION
Average woaaeearruuncytt
1.1
Recently, the pre-trained language models have shown great power in
20/
various natural language processing (NLP) tasks (Wang et al., 2018a; 8a; Qin et al., 2019; Rajpurkar et al., 2016). In particular, BERT (De- vlin al., 2018) significantly improves the state-of-the-an performance, while the massive parameters hinder their widespread deployment on edge devices in the real world. Therefore, model compression has been actively studied to alleviate resource constraint issues, including quan- tization (Shen et al., 2020; Zafrir et al., 2019), distillation (Jiao et al., 2020: Xu et al., 2020), pruming (McCarley et al., 2019; Gordon et al., 2020), parameter sharing (Lan et al., 2020), etc. Among them, quan- tization emerges an efficient way obtain the compact model by compressing the model parameters to lower bit-width representation, such Q-BERT (Shen et al., 2020), Q8BERT (Zafrir al., 2019), and GOBO (Zadeh et al., 2020). However, the representation limitation and optimization difficulties come as consequence of applying discrete quantization, triggering severe performance drop in quantized BERT. Fortunately, distillation becomes a common remedy in quantization as an auxil- iary optimization approach tackle the performance drop, which encourages the quantized BERT to mimic the full-precision model to exploit knowledge in teacher's 's representation (Bai et al., 2020).
48-
TAnNYASKT
40
1111
is
Figure 1: Accuracy vs.
FLOPs & size. Our BiBERT enjoys most computation and storage savings while surpassing SOTA quantized BERTs on GL.EE benchmark with low bit activation.
As the quantization scheme with the most aggressive bit-width (Zhou et al., 2016; Wang et al., 2018b; Xu et al., 2021), full binarization of BERT (i.e., 1-bit weight, word embedding, and ac-
cqual contrihution
corresponding author

---

Published as a conference paper at ICLR 2022
Bi-Attention
E Embeddings
Direction-matccing Distillation (DMD)
hidden states H
Flly
precision Transtormer
Layer
For
(the teacher
query
MHA
MHA
Bg
Rig.
Px
PACY
Add
Add Norm
By
attention
FFN
FFN
baol()
Add & Norm
Add & Norm
BAHM
RA
bead,
headwy
Classifier
11g
Perl
Figure 2: Overview of our BiBERT, applying Bi-Attention structure for maximizing representation information and Direction-Matching Distillation (DMD) scheme for accurate optimization.
tivation) further allows the model to utilize extremely compact 1-bit parameters and efficient bit- wise operations, which can largely promote the applications of BERT on edge devices in the real world (Wang et al., 2021; Liu et al., 2018). However, although the BERT quantization equipped with distillation has been studied, it is still significant challenge to binarize BERT to extremely 1-bit, especially for its activation. Compared with the weight and word embedding in BERT, the binarization of activation brings the most severe drop, and the model even crashes in NLP tasks. So far, previous studies have pushed down the weight and word embedding to be binarized (Bai et al., 2020), but none of them have ever achieved binarize BERT with 1-bit activation accurately.
Therefore, we first build a fully binarized BERT baseline, a straightforward yet effective solution based common techniques. Our study finds that the performance drop of BERT with binarized 1-bit weight, activation, and embedding (called fully binarized BERT) comes from the informa- tion degradation of the attention mechanism in the forward propagation and the optimization direc- tion mismatch of distillation in the backward propagation. First, the attention mechanism makes BERT focus selectively on parts of the input and ignore the irrelevant content (Vaswani al., 2017; Chorowski et al., 2015; Wu et al., 2016). While our analysis shows that direct binarization leads to the almost complete degradation of the information of attention weight (Figure 4), which results in the invalidation of the selection ability for attention mechanism. Second, the distillation for the fully binarized BERT bascline utilizes the attention score, the direct binding product of two bina- rized activations. However, we show that it causes severe optimization direction mismatch since the non-neglectable error between the defacto and expected optimization direction (Figure 5).
This paper provides empirical observations and theoretical formulations of the ahove-mentioned phenomena, and proposes a BiBERT to turn the full-precision BERT into the strong fully binarized model (see the overview in Figure 2). To tackle the information degradation of attention mechanism, we introduce an efficient Bi-Altention structure based on information theory. Bi-Attention applies binarized representations with maximized information entropy, allowing the binarized model re- store the perception of input contents. Morcover, we developed the Firection-Matchin Distillation (DMD) scheme eliminate the direction mismatch in distillation. DMD takes appropriate activation and utilizes knowledge from constructed similarity matrices in distillation to optimize accurately.
Our BiBERT, for the first time, presents a promising route towards the accurate fully binarized BERT (with 1-bit weight, embedding, and activation). The extensive experiments on the GLUE (Wang et al., 2018a) benchmark show that our BiBERT outperforms existing quantized BERT models with ultra-lower bit activation by convincing margins. For example, the average accuracy of BiBERT ex- ceeds 1-1-1 bit-width Binary (1-bit weight, 1-bit embedding and 1-bit quantization) by 20.4% accuracy on average, and even better than 2-8-8 bit-width Q2BERT by 13.3%. Besides, we highlight that our BiBERT gives impressive 56.3x x and 31.2 x saving on FLOPs and model size, respectively, which shows the vast advantages and potential of the fully binarized BERT model in terms of fast inference and flexible deployment in real-world resource-consen consurciedd scenarios (Figure 1). Our code released at
BUILDING A FULLY BINARIZED BERT BASELINE
First of all, we build baseline to study the fully binarized BERT since it has never been proposed in previous works. A straightforward solution is to binarize the representation in BERT architecture in the forward propagation and apply distillation to the optimization in the backward propagation.
2

---

Published as a conference paper at ICLR 2022
2.1 BINARIZED BERT ARCHITECTURE
We give a brief introduction to the architecture of binarized BERT. In general, the forward and backward propagation of sign function in binarized network can be formulated
{-1
if x 2
ac
ac @sign(a) dr 10
if | S
Forward: sign(x) (z)
Backward:
(1)
otherwise
otherwise
where C' is the cost function for the minibatch. sign function is applied in the forward propagation while the straight-through estimator (STE) (Bengio al., 2013) is used to obtain the derivative in the backward propagation. As for the weight of binarized linear layers, the common practice is to redistribute the weight to zero-mean for retaining representation information (Rastegari et al., 2016; Qin et al., 2020) and applies scaling factors to minimize quantization errors (Rastegari et al., 2016). The activation is binarized by the sign without re-scaling for computational efficiency. Thus, the computation be expressed
bi-linear X) Oow (sign(X) sign W ((W))) Cw
W Wal,
(2)
where W and X denote full-precision weight and activation, uu-) denotes the mean value, Ow is the scaling factors for weight, and @ denotes the matrix multiplication with bitwise xnor and bitcount as presented in Appendix A.4.
The input data first passes through binarized embedding layer before being fed into the trans- former blocks (Zhang et al., 2020; Bai et al., 2020). And each transformer block consists of two main components: Multi-Head Attention (MHA) module and Feed-Forward Network (FFN). The computation of MHA depends queries Q, keys K and values V, which are derived from hidden states H E N.xx xD represents the length of the sequence, and D represents the dimension of features. For specific transformer layer, the computation in attention head can be expressed as
Q bi-linearq (H), K bi-linear,, K (H),
bi-lineary (H),
(3)
where bi-linearq, bi- bi-linear,, bi- linearyy represent three different binarized linear layers for Q K, V respectively. Then we compute the attention score A as follow:
A 1 BQ (BQ&
1 (Bq VD
TaoBAr @O0BBK) BQ sign(Q). (Q), BK (ign
where BQ and By are the binarized query and key, respectively. Note that the obtained attention weight is then truncated by attention mask, and each row in A can be regarded as k-dim vector, where l is the number unmasked elements. Then we binarize the attention weights B as
B = sign(softmax( A)).
(5)
We follow original BERT architecture to carry the rest of MHA and FFN in the binarized network.
2.2 DISTILLATION FOR BINARIZED BERT
Distillation is a common and essential optimization approach to alleviate the performance drop of quantized BERT under ultra-low bit-width settings, which be unobstructedly applied for any architectures to utilize the knowledge of a full-precision teacher model (Jiao et al., 2020; Bai et al., 2020; Zhang et al., 2020; Wang et al., 2020). The usual practice is distill the attention score ATi. MHA output Mrt. and hidden states Hri in a layerwise manner from the full-precision teacher network, and transfer to the binarized student counterparts, i.e., A,, M,, H, (1 = 1, ..., L, where represents the number of transformer layers), respectively. We use the mean squared errors (MSE) as loss function to measure the difference between student and teacher networks for corresponding features:
lan MSE(A ATT), Emha 1=1
MSE(M,, MT1), Eaia
MSE Hr)). 1=1
(6)
Then the prediction ladyer distillation loss is conducted by minimizing the soft cross-entropy (SCE) between teacher logits yT and student logits y. The objective function is expressed as
Edistill fut Lmnna Prid + 6pree: épeed SCE (y.)).
3

---

Published as a conference paper at ICLR 2022
S9T-2
OOP
SST-2
QQP
FP self sttenttooonn
78.41
73:40
74.86
70,85
Intermediato
79.79
74.07
FP auipu
83.12
Frg
77.49
70,88
FP
78.42
75.88
70.07
77.6
70 72.30
77.60
72.30
(a) Architecture perspective
(b) Oplimization perspective
Figure 3: Analysis of bottlenecks from architecture and optimization perspectives. We report the accuracy of binarized BERT on SST-2 and QQP tasks about (a) replace full-precision structure, (b) exclude one distillation knowledge.
THE RISE OF BIBERT
Although we can build a fully binarized BERT baseline and its training pipeline with common techniques, the performance is still of major concern. Our study shows that the baseline suffers an immense information degradation of attention mechanism in the forward propagation and severe optimization direction mismatch in the backward propagation. To solve these problems, in this section, we propose the BiBERT with thcoretical and experimental justifications.
3.1 BOTTLENECKS OF FULLY BINARIZED BERT BASELINE
Intuitively, the fully binarized BERT baseline, the information representation capability largely depends on structures of architecture, such attention, which is severely limited due to the binarized parameters, and the discrete binarization also makes the optimization more difficult. This means the bottlenecks of the fully binarized BERT baseline come from architecture and optimization for the forward and backward propagation, respectively.
From the architecture perspective, observe the accuracy drop caused by binarizing single structure by replacing it with a full- preciison counterpart as Figure 3(a) shows. We find that binariz- ing MHA brings the most significant drop of accuracy among all parts of the BERT, up 11.49% (full-precision 89.09% vs. binarized 77.60% and 1 11.72% on SST-2 and QQP, respectively. While binarizing FFN and pooler layers brings less harm to the accuracy. Binarization intermediate or output layer only brings 2. 1.19% and 5.52% drop on SST-2, and binarization of pooler layer only causes 0.82%. Same results can be seen on QQP. Thus, improving the attention structure is the highest priority to solve the accuracy drop of binarized BERT.
From the optimization perspective, we eliminate each distillation term demonstrate the benefits it brings in Figure 3(b). The results show that for most distillation terms, solely removing them in the distillation will harm the performance, e.g., removing the distillation of hidden states leads to 0. 1% and 1. 1.62% drop SST-2 and QQP, and removing that of MHA outputs decrease the model accuracy to 73.24% (4.36% drop) on SST-2, However, when the distillation loss of attention score is removed, the performance increases 0.81% and 1.10% on SST-2 and QQP respectively. These observations inspire us to rethink the distillation fully binarized BERT. It requires a new design that could utilize the full full-precision teacher's knowledge better.
Based on the above experimental observations, we find that (1) the existing attention structure should NOT be directly applied in fully binarized BERT and (2) the distillation for the attention fully binarized BERT is actually harmful, which is contrary to the practice of many existing works. Thus in this paper, first present the theoretical derivation of these two phenomena and then propose a well-designed attention structure and a novel distillation scheme for fully binarized BERT.
in
3.2 BI-ATTENTION
To address the information degradation of binarized representations in the forward propagation, we propose efficient Bi-Attention structure based information theory, which statistically maxi- mizes the entropy of representation and revives the attention mechanism in the fully binarized BERT.
3.2.1 INFORMATION DEGRADATION IN ATTENTION STRUCTURE
Since the representations (weight, activation, and embedding) with extremely compressed bit-width in fully binarized BERT have limited capabilities, the ideal binarized representation should preserve
4

---

Published as a conference paper at ICLR 2022
[CLS] [CLS) [CLS]
[GLS]
[CLS]
[CLS)
the
the the
the
the
sat
52t
sat
58t
on
on
the
the
the
mat
mat
[SEP]
[SEP]
[SEP] (b) Fully binarized BERT baseline
[SEP]
[SEP]
[SEP)
(a) Full-precisson
(c) BIRERT (Ours)
Figure 4: Attention- view for (a) full-precision BERT, (b) fully binarized BERT baseline,
and (c) BiBERT for same input. BiBERT with Bi-Attention shows similar behavior with the full-
precision model, while baseline suffers indistinguishable attention for information degradation. The visualization tools is adapted from (Vig, 2019).
the given full-precision counterparts much as possible, which means the mutual information be- tween binarized and full-precision representations should be maximized. When the deterministic sign function is applied binarize BERT, the goal is equivalent to maximizing the information entropy (BB of binarized representation B (Messerschmitt, 1971), which is defined as
H(B) = Σ (B)B))og(BB) B
(8)
} is the random variable sampled from B with probability mass function There- fore, the information entropy of binarized representation should be maximized to better preserve the full- full-precision counterparts and let the attention mechanism function well. The application of zero- mean pre-bineriiin binarized weight in binarized linear layers is representative practice that maximizes the information of binarized weight and activation (as in Section 2.1). The related discussion and proof is shown in Appendix A.1.
where B {-1,1}
As for the attention structure in full-precision BERT, the normalized attention weight obtained by softmax is essential. But direct application of binarization function causes a complete information loss binarized attention weight. Specifically, since the softmax(A) (A) is regarded as following probability distribution, the elements of B are all quantized to 1 (Figure 4(b)) and the information entropy H(BA) degenerates to 0. common measure to alleviate this information degradation is to shift the distribution of input tensors before applying the sign function, which is formulated as
B sign (softmax (A) T),,
where the shift parameter т, also regarded as the threshold of binarization, is expected maximize the entropy of the binarized B and is fixed during the inference.
Theorem 1. Given E R** with Gaussian distribution and the variable B% generated by B sign(softmax(AA A) T), the threshold which maximizes the information entropy H HBx)), negatively correlated to the number of elements k.
However, Theorem 1 shows that is hard to statistically determine maximize the information entropy of binarized counterparts for the attention weight masked by the changeable length of atten- tion mask. This fact means that common measure for maximizing information (as binarized linear layers) fails in the binarized attention structure. The proof of Theorem is shown in Appendix A.2.
Moreover, the attention weight obtained by the sign function is binarized to {-1, while the original attention weight has normalized value range [0, 1]. The negative value of attention weight the binarized architecture is contrary to the intuition the existing attention mechanism and also empirically proved to be harmful to the attention structure in Appendix C.2.
3.2.2 BI-ATTENTION FOR MAXIMUM INFORMATION ENTROPY
To mitigate the information degradation caused by binarization in the attention mechanism, introduce an efficient Bi-Attention structure for fully binarized BERT, which maximizes information entropy of binarized representations statistically and applies bitwise operations for fast inference.
We first maximize the information entropy H(B The analysis in Section 3.2.1 shows that it is hard to statistically obtain a fixed mean-shift for softmax(A) A) that maximizes the entropy of the binarized parameters. Fortunately, since both softmax and sign functions order-preserving there
5

---

Published as a conference paper at ICLR 2022
is threshold (TT, A) to maximize Entropy sign(A A - (T A)). which is equivalent to maximizing the information entropy of BA in Eq. (9) (as the Proposition 2 proved in the Appendix A.6).
Theorem 2. When the binarized query Bq = QQ) E [-1.1] and key Bk = sign (K) E (-1,1) are entropy maximized in binarized attention, the probability mass function of each element Aij, i.j E [1. N sampled from attention score A = BB @ BK can be represented as PA D) = 0.5 CD E [0, P}, which approximates the Gaussian distribution N(0, D).
Theorem 2 shows that the distribution of A approximates the Gaussian distribution (0, D) and is distributed symmetrically, we can thus trivially get that Φ(T, A) = 0. The detailed proof is given in Appendix A.3. Therefore, simply binarizing the attention score A can maximize the information entropy of binarized representation.
Then, to revive the attention mechanism to capture crucial elements, here are inspired by hard attention (Xu et al., 2015) to binarize the attention weight into the Boolean value, while our design driven by information entropy maximization. In Bi-Attention, we use bool function to binarize the attention score A, which is defined as
bool(x) 1, if 0, otherwise
a bool(() dx
f1. if s
(10)
0. otherwise.
By applying bool(-) function, the elements in attention weight with lower value are binarized to O, and thus the obtained catropy-maximiee attention weight can filter the crucial part of elements. And the proposed Bi-Attention structure is finally expressed as
Ba bool (A) bool (/
BQ 2 BK ))
(11)
Bi-Attention (Bq, BK, Bv) = BA &Bv.
(12)
where By is the binarized value obtained by sign (V). BA is the binarized attention weight, and X is well-designed Bitwise-Affine Matrix Multiplication (BAMM) operator composed by and bitshift to align training and inference representations and perform efficient bitwise calculation. The detailed of BAMM is illustrated Figure 7 in Appendix A.4.
In a nutshell, in our Bi-Attention structure, the information entropy of binarized attention weight is maximized (as Figure 4(c) shows) to alleviate its immense information degradation and revive the attention mechanism. Bi-Attention also achieves greater efficiency since the softmax is excluded.
3.3 DIRECTION- MATTHIINGG DISTILLATION
To address the direction mismatch occurred in fully binarized BERT baseline in the backward propa- gation, we further propose a Direction-Matchee Matching Distillation (DMD) scheme with apposite distilled activations and the well-constructed similarity matrices effectively utilize knowledge from the teacher, which optimizes the fully binarized BERT more accurately.
3.3. DIRECTION MISMATCH
As an optimization technique based on element-level comparison of activation, distillation allows the binarized BERT to mimic the full-precision teacher model about intermediate activation. However, we find that the distillation causes direction mismatch for optimization in the fully binarized BERT baseline (Section 3.1), leading insufficient optimization and even harmful effects.
Eq. (6) shows that, the distillation for attention score A in a specific layer can be expressed as MSE(A, Ar). where A and Ar are attention scores in binarized student BERT and full-precision teacher BERT, respectively. Since the attention score in fully binarized BERT is obtained by multi- plying binarized query BQ and key Bk. the loss lau can be expressed as:
Pan = MSE (1B BQ & BK VD
Qr K?)
VDD
VD
(13)
Theorem 3. Given the variables X and XT follow No0, 01), O1), σ2) respectively, the proportion of optimization direction error is defined as Perror Q-bi p(sign(X Xx) + sign( (igu(quantzze(>> Xr)), where quantizeg denotes the Q-bit symmetric quantization. As Q reduces from 8 to 1, Peroor Q-bii becomes larger.

---

Published as a conference paper at ICLR 2022
Merrttch
0.64
Theorem 4 shows that the distillation for binarized activation based on the numerical comparison with full-precision activation causes the most severe optimization direction error among all bit-widths. For example, for activation following the assumption of standard normal distribution, the error probability caused by 1-bit binarization is ap- proximately 4.4x that of 4-bit quantization (14.4% vs. 3.3%). The sudden increase in the probability of optimization direction mismatch makes it a critical issue in the fully binarized BERT while neglected in the previous BERT quantization studies. The proof and discussion of Theorem 4 is found in Appendix A.5.
Corne
Lot Fally binarized BERT hoxeline
Wiruiih
0.22
0.00
pagpet
Since the attention score is obtained by direct multiplication of two binarized activations (binarized query Bq and key Bq). its distil- lation is also misled by the direction mismatch. In Figure 5(a), the direction mismatch in the distillation of attention score is common and severe, even causes up to higher degree optimization on the mis- match direction (mismatch 0.64 match 0.45 average). The phenomenon in Figure 5 reveals that the element-wise gradient direc- tion error for binarized activation found in Theorem 4 accumulates in the distillation of the attention score and eventually causes a sig- nificant mismatch between the direction of overall optimization and that of the distillation loss. Hence, the direction mismatch hinders the accurate optimization of the fully binarized BERT. Second, the activation scales in the fully binarized student and full-precision teacher BERT are significantly different, e.g., in fully binarized BERT approximately follows the fixed distribution (0, D) while that of Ar in full- cell-precision teacher is flexible. Applying the discrete binarization function also makes numerical changes of activation more significant. These problems for activation also hinder effective distillation for the fully binarized BERT.
(b) DIBERT (Ouns)
Figure 5: Visualization of the direction mismatch of one head throughout train- ing. The full binarized BERT baseline distills the attention score which has severe direction mismatch, while the knowledge (take queries example in (b)) used in our BiBERT signif- icantly alleviates mismatch.
3.3.2 DIRECTION-MATCIINE DISTILLATION FOR ACCURATE OPTIMIZATION
To solve the optimization direction mismatch in the distillation of the BERT full binarization, we propose the Direction-Matching Distillation (DMD) method in BiBERT.
We first reselect the distilled activations for DMD. As discussed in Section 3.3.1, severe direction mismatch is mainly caused by the distillation of the direct binding product of two binarized acti- vations. Thus, we distill the upstream query Q and key K instead of attention score in our DMD for distillation utilize its knowledge while alleviating direction mismatch. Moreover, inspired by an observation in Section 3.1 that the distillation of MHA output is of great help for improving performance, we also distill the value V further cover all the inputs of MHA.
Then, we construct similarity pattern matrices for distilling activation, which can be expressed as
Q Q
K KT
VxVi
PQ
PK = K XT||
Pv =
(14)
IV
where || denotes /2 normalization. Previous work shows that matrices constructed in this way regarded as the specific patterns reflecting the semantic comprehension network (Tung & Mori, 2019: Martinez et al., 2020). We further find that matrices also scale-normalized and stable numerically since they focus more on endogenous relative relationships and thus are suitable for distillation between binarized and full-precision networks. The corresponding PQr. PKT, Pvr are constructed in the same way by the teacher's activation. The distillation loss is expressed as:
Edism LDDD + Eiid + fpood
LDD =
Σ Σ
P PETIl
(15)
LE[L FEFFimD
where L denotes the number of transformer layers, FOMD (Q, K, V). The loss term bna is constructed as the E2 normalization form, and Epred is still constructed as in Eq. (7).
Our DMD scheme first provides the matching optimization direction (Figure 5(b)) by reselecting appropriate distilled parameters and then constructs similarity matrices to eliminate scale differences and numerical instability, thereby improves fully binarized BERT by accurate optimization.

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
In this section, we conduct extensive experiments to validate the effectiveness of our proposed BiB- ERT for efficient learning on the multiple architectures and the GLUE (Wang et al., 2018a) bench- mark with diverse NLP tasks. We first conduct an ablation study on four tasks (SST-2, MRPC, RTE, and QQP) for BiBERT on BAST (12 hidden layers) architecture (Devlin al., 2019) to showcase the benefits of the Bi-Attention structure and DMD scheme separately. Then we compare BiBERT with the state-of-the-arr (SOTA) quantized BERTs terms of accuracy on BAST. Our designs stand out among fully binarized BERT's and even outperform some quantized models with more bit-width parameters. We also evaluate our BiBERT on TinyBERT (6 hidden layers) and TinyBERTaL (4 hidden layers) (Jiao et al., 2020), and BiBERT these compact architectures even outperforms existing methods on BERT BASE In terms of efficiency, our BiBERT achieves an im- pressive 56.3x and 31.2x saving FLOPs and model size. The detailed experimental setup and implementations are given in Appendix B.
4.1 ABLATION STUDY
Table 1: Ablation study, DASS-2MBPCRTEEOP
Quant
ABits
As shown in Table I, the fully binarized BERT baseline suffers severe performance drop on SST-2, MRPC, RTE and QQP tasks. Bi-Attention and DMD can improve the performance when used alone, and the two techniques further boost the per- formance considerably when combined together. To conclude, the two techniques can promote each other to improve BiB- ERT and close the performance gap between fully binarized BERT and full-precision counterpart.
Full Precision
32-32-32
93.2
86.3
72.2
91.4
Baseline
1-1-1
X
77.6
70.2
54.1 73.2
Bi-Anention
X
70.5
74.9
DMD
1-1-1
X
79.9
70.5
55.2 75.3
BIBERT (ours)
1-1-1
X 88.7
72.5
57.4
84.8
Baseline
1-1-1
/
84.0
71.4
50.9
Bi-Anention
73.2
DMD
1-1-1
85.3
72.5
56.3
BIBERT (ours)
1-1-1
/
90.9
78.8
61.0
4.2 COMPARISON WITH SOTA METHODS
We compare our BiBERT with the SOTA BERT quantization methods under ultra-low bit-width in terms of accuracy and efficiency, to fully demonstrate the advantages of our design.
Accuracy Performance. In Table 2 and Table 3, we show experiments on the BERT BASE (as de- fault), TinyBERTa.. and TinyBERTI architectures and the GLUE benchmark with or without data augmentation. Since the MNLI and QQP have large data volume and it brings little benefits for the performance to apply data augmentation, we do not apply augmentation for these two tasks, thus they are excluded in Table 3. Results show that BiBERT outperforms other methods on the devel- opment set of GLUE benchmark, including VernaryBERT BinaryBERT, Q-BERT, and Q2BERT.
Table 2 shows the results on the GLUE benchmark without data augmentation. Our BiBERT sur- passes existing methods on BERT BASE architecture by wide margin in the average accuracy, and first achieves convergence under ultra-low bit activation on some tasks, such as CoLA, MRPC, and RTE. While under ultra-low bit activation, the accuracy of TemaryBERT and BinaryBERT de- creases severely which also does not improve under the 2-2-2 setting (about 4x FLOPs and 2x storage usage increase). On some specific tasks like MRPC, higher bit-widths and consumption did not even make TernaryBERT and Binary BERT converge. In addition, BiBERT surpasses the fully binarized baselineso% and BinaryBERT 50% (maximizing the information entropy by the 50% quan- tile threshold) and other strengthened baselines, we present detailed discussion in Appendix C. With data augmentation, BiBERT achieves comparable performances with full-precision BERT on several tasks, egg., 90.9% accuracy (drop only 2.2%) on SST-2 (Table 3). These results indicate that BiBERT makes full use of the limited representation capabilities by the well-designed structure and training scheme. We noticed that BiBERT lags behind Q BERT (2-8-8) and TernaryBERg (2-2-2) on MNLI and STS-B (with DA) while surpassing them on other tasks, indicating the fully binarized BERT may have greater improvement potential in these tasks. BiBERTT is also evaluated on compact TinyBERToL 6L. and Tiny BERT architectures, with about 2.0x and 18.8x fewer FL.OPs, respectively. The results show that BiBERT these compact architectures still outperforms existing quantization methods on BERTBASE. such as TernaryBERT and BERT. forcefully demonstrates that the targeted designs of BiBERT enable fully binarized BERT's to run various architectures.
"#Bits" (W-F-A) is the bit number for weights, word embedding, and activations. "DA" is short for data augmentation. "Avg." denotes the average results.
8

---

Published as a conference paper at ICLR 2022
Table 2: Comparison of BERT quantization methods without data augmentation.
Quant
Flllss
Size
FL
MNNL
QQP
QNLI
SST2
CoLA
STS-B
MIRC
RTR
Avg.
Full Preeisson
12-32-22
418
22.
34.9883.
91.4
V2.1
V3.2
59.7
N6.3
72.
83.9
"O-BERT"
238-
13.0"
7617777
B16
68.3"
"5"
Q2BERT
47.2/47.3
52.7
Tormryy BERT
2-2-8
28.0
6.4
83.3883.
90.1
50.7
87.5
68.2
BinaryBHK
1-1-4
16.5
15
K3.9/4-22
91.2
90.9
92.
44.4
N7.2
83.3
65.3
79.9
Termay BERT
2-2-2
28.0
15
10.3/10.0
63.1
50.0
80.7
0
12.4
68.3
54.5
45.5
BiaaryBER1
1-1-2
16.5
0.8
62.7/63.9 9
79.9
52.6
82.5
14.6
6.5
68.3
52.7
53.7
Tomary BERT
2-2-1
28.0
$2.7/33.0
74.1
59.3
53.
7.1
68.3
53.4
42.3
"Bacllin
04
73.2
66:1
77..6
11.7
7..
7002"
"""""""
Baseline.gos p%
7/49.1
67.9
14.0
11.5
Binury BERT
1-1-1
16.5
0,4
35 6/353
66.2
51.5
53.2
6.1
68.3
52.7
41.0
Binary BERT 100%
1-1-1
13.4
04
2/40
66.7
59.5
34.
4.3
6.8
68.3
53.4
43.5
BIBERT (ours)
1-1-1
13.4
0.4
66.1/67.5
84.8
72.6
88.7
25.4
33.6
72.5
57.4
63.2
Full Precisson it
32:32322
257
84.683
71.6
90.4
V3.
51.1
83.7
87.3
7000
79.4
BiBERT (ours)
111
0.2
63//6377
83.3
73.6
87.9
24.8
33.77
722""
55.9
62.1
Full Precisson a
55,6
58818
71.3
87.7
92.6
14.1
80.1
86.4
66.6
77.0
BiBERTa (ours)
1-1-1
4.4
0,03
55.3156.1
78.2
71.2"
85.4
14.9
31.55
72.2"
54.2
51.7
Table 3: Comparison of BERT quantization methods with data augmentation.
Quant
ABits
Size cum.
cm
QNLa
SST-2
Col.A
STS-B
MRRC:
RTE
Avg.
Fall Precasion
32-32-2
118
93.2
59.7
90.1
$6.3
72.2
82.3
(ERT
2-2-8
28.0
64
90.0
V2.
47%
64.3
82.6
68.
77'
BinaryBERT
1-1-4
16.5
1.5
91.4
93.7
53.3
88.6
$6.0
71.5
80.8
TemaryBERT
72.0
BinaryBERT
51.0
71.0
52.0
BentaryB:RT BERT
2-2-1
28.0
B8
50.9
80.3
71.5
45.5
Baseline w
[FT
13.1
0.3
69.2
23:3"
13.3
71.3"
30.9
32'2"
BinaryBERT
1-1-1
16.5
0.4
66.1
78.3
7.3
22.1
69.3
57.7
50.1
BiBERT (ours)
1-1-1
13.4
0.4
76.0
90.9
37.8
56.7
78,8
61.0
67.0
BiBERTa lours)
FET
76.0
57.
Fall Preciston
55.6
87.7
92.6
80.4
35.4
06.6
BIBERT (ours)
F11
4.4
0,03
732
88.3
2000
42.5
74.0
36.7
391
Efficiency Performance. As shown in Table 2 and Table 3 above, BiBERT achieves im- pressive 56.3x FLOPs and 31.2x model size saving over the full-precision BERT. Furthermore, benefiting from simple yet effective Bi-Attention structure which casts the expensive softmax op- eration into a well-enginerred bit operation BAMM, our BiBERT surpasses other quantized BERTs in computation and storage saving while enjoying the best accuracy.
4.3 MORE ANALYSIS
Information entrayy
Information Performance. To show the improvement of informa- tion performance by applying Bi-Attention, we compare the informa- tion entropy of binarized representations for baseline and BiBERT. As shown in Figure 6(a), we take the first heads in layer 0 of each model, and the same phenomenon exists in all heads and layers. During the training process, the information entropy of attention weight in BiB- ERT fluctuates in a small range and is almost maximized, however, that of bascline is completely degraded to O.
23
20
Information performance
10m
Training Curves. We plot training loss curves of fully binarized BERT baseline and BiBERT on SST-2 without data augmentation in Figure 6(b). Compared with the baseline model, our method has a faster convergence rate and achieves higher accuracy, suggesting ours advantages in terms of accurate optimization.
Dlen Inw
Epoch
(6) Triining surves
Figure 6: Analysis.
5 CONCLUSION
We propose BiBERT towards the accurate fully binarized BERT. We first reveal the bottlenecks of the fully binarized BERT baseline and build a theoretical foundation for the impact of full bina- rization. Then we propose Bi-Autention and DMD in BiBERT to improve performance. BiBERT outperforms existing SOTA BERT quantization methods with ultra-low bit activation, giving im- pressive 56.3x FLOPs and 31.2x model size saving. Our work gives an insightful analysis and effective solution about the crucial issues in BERT full binarization, which blazes promising path for the extreme compression of BERT. We hope our work can provide directions for future research.
Acknowledgement This work supported in part by National Key Research and Development Plan of China under Grant 2020AAA0103503, National Natural Science Foundation of China under Grant 62022009 and Grant 61872021, Beijing Nova Program of Science and Technology under Grant Z191 100001 1 19050, Outstanding Research Project of Shen Yuan Honors College. BUAA Grant 230121206, and CCF-Baidu Open Fund OF202 it was also supported under the RIE2020
Industry Aligument Fund-Inndste Collaboration Projects (IAF-ICP) Funding Initiative.

---

Published as a conference paper at ICLR 2022
REFERENCES
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael R. Lyu, and Irwin King. Binarybert: Pushing the limit of BERT quantization. CoRR, abs/2012.15701, 2020.
R Banner, Y Nahshan, E Hoffer, and D Soudry. Post training 4-bit quantization of convolution networks for rapid-deployment. CoRR, a10./110,, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 10o:::30884440 3432, 2013.
Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In NeurlPS, 2015.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 181.000/000 2018.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI-HILT (1), 2019.
Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. Compressing bert: Studying the effeets of weight pruning transfer learning. arXiv preprint arXiv:2002.08307 2020.
Zhezhi and Deliang Fan. Simultaneously optimizing weight and quantizer of ternary neural network using truncated gaussian approximation. In IEEE CVPR. 2019.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. arXiv preprint arXiv:2004.04037, 2020.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. In EMNLP, 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self superviedd learning of language representations. In ICLR, 2020.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), ECCV, 2018.
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. In ICLR, 2020.
JS McCarley, Rishav Chakravarti, and Avirup Sil. Structured pruning of bert-based question answering model. CoRR, abs/1910.06360, 2019.
Messerschmitt. Quantizing for maximum output entropy (corresp.). IEEE Transactions Infor- mation Theory, 1971.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In CVPR, 2020.
Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. A stack Stack-propagattin framework with token-level intent detection for spoken language understanding. arXiv preprint 1909.02288 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 1606.05500 2016.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Bastian Leibe, Jiri Matas, Nicu Sehe, and Max Welling (eds.), ECCV, 2016.
10

---

Published as a conference paper at ICLR 2022
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Q-BERT: hessian based ultra low precision quantization of BERT. In AAAI, 2020.
Frederick Tung and Greg Mori. Similarit knowledge distillation. In CVPR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Jesse Vig. A multiscale visualization of attention in the transformer model. In Marta R. Costa-jussà and Enrique Alfonseca (eds.), ACL, 2019.
Helen M Walker and M Helen. De moivre on the law of normal probability, Smith, David Eugene. source book in mathematics. Dover, 1985.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804. 07460, 2018a.
Peisong Wang, Qinghao Hu, Yifan Zhang, Chunjie Zhang, Yang Liu, and Jian Cheng. Two-step quantization for low-bit neural networks. In CVPR, 2018b.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers, 2020.
Ziwei Wang, Jiwen Lu, Ziyi Wu, and Jie Zhou. Learning efficient binarized object detectors with in- formation compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08) 016609008144,, 44, 2016.
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.
Yige Xu, Xipeng Qiu, Ligao Zhou, and Xuanjing Huang. Improving bert fine-tuning via self- ensemble and self-distillation. arXiv preprint arXiv:2002.10345 2020.
Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. arXiv preprint arXiv:2103.12366 2021.
Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. GOBO: quantizing attention-based NLP models for low latency and energy efficient inference. In MICRO, 2020.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. QSBERT: quantized Sbit BERT. In NeurlPS, 2019.
Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert: Distillation-awarr aware ultra-low bit BERT. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), EMNLP, 2020.
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, abs/1606/ 606.06160, 2016.
11

---

Published as a conference paper at ICLR 2022
APPENDIX FOR BIBERT
A MAIN PROOFS AND DISCUSSION
A.1 DISCUSSION OF BINARIZED LINEAR LAYER
Proposition When random variable X follows a zero-mean Gaussian distribution, the infor- mation entropy H(B) is maximized, where Bx = sign(X).
Since the value of Bw depends on the sign of W and the distribution W is almost symmetric (He & Fan, 2019; Banner al., 2018), the balanced operation can maximize the information entropy of binarized Bw on the whole. The balanced binarized weights with the zero-mean attribute can be obtained by subtracting the mean of full-precision weights. As Proposition shown, under the binomial distribution assumption and symmetric assumption of W, when the binarized weight is balanced, the information entropy of Bw takes the maximum value, which means the binarized values should be evenly distributed.
Moreover, when the weight is zero-mean, the entropy output Z (seen the activation in the next binarized linear layer) in the network can also be maximized. Supposing quantized activations Bx have mean [[ = μ1, the mean of Z can be calculated by
Z Bw
B
E
Bw & p1.
(16)
Since the zero-mean weight applied in each layer, we have Qw X 1 = and the mean of output is zero. Therefore, the information entropy of activations in each layer can be maximized according Proposition 1. Therefore, a simple redistribution for the full full-precision counterpart of binarized weights can simuitaneously maximize the information entropy of binarized weights and activations.
The proof Proposition presented as below:
Proof. According to the definition of information entropy, we have
H(Bx) = - Σ
PBx (b) log logPBx (b)
(17)
BEBx
PBx (- lolloggB, (-1) PBB (1) log PBx (1)
(18)
-Bxx
(-1) log (-1) 1ppx (-1) log (1 pBx (-1)))
(19)
Then we can get the derivative of H(Bx) with respect PBx (-1)
d dH(()))
PRx(-1) (-1) + (log
1 1 1 (-1)
(loggBB (-1) (-1)
PBx (-1)) +
d JBBx (-1)
PBx (-1)
(1 PBx (- 1)) In
(20)
1
1
-logPBB (-1) + log PBx PBx (-1))
+
(21)
1n2
In2
llog ( (1 (1-PB(-1))
(22)
p
When we let à ((B)) 0 maximize the H(Bx), have pB (-1) 0.5. Since the de- Tpax (-1)
terministic sign function with the zero threshold is applied as the quantizer, the probability mass function of Bx is represented as
S fr(v) dy. if -
PBx (b)
(23)
1 fr(v) dy. if = 1,
12

---

Published as a conference paper at ICLR 2022
where fx) is the probability density function of variable X. Since X ~ Oo), the fx (x) is defined as
fx(x)
20
(24)
2 2m
When the information entropy H(xx) is maximized, we have
[ fx(x) dr 0.5.
(25)
A.2 PROOF OF THE THEOREM
Theorem 1. Given A E R& with Gaussian distribution and the variable B generated by B (ign(softmax( A) -T), the threshold т, which maximizes the information entropy H(B ) negatively correlated to the number of elements k.
Proof. Given the A = (A1, A2, Ax} E R& each A; obeys Gaussian distribution N N o)- Without loss of generality, we consider the threshold of the first variable A1, which can maximize the information entropy. Such threshold Tк satisfies
pA, (a: (afttaxx(()) TR] da; 0.5,
(26)
K-1
where [-] denotes the Iverson bracket that is defined
[P] {
if if is true; otherwise,
(27)
and the softmax function is
e,
softmax; (A::)
(28)
Σ/=1 ed
PA. (a;) presents the probability that the i-th element is equal to Since the softmax; function order-preserving, there exists exactly one threshold T(A2, AA, Ag..., Asi Th) such that
(s..AAA).. (( + (D) Tk-
In other words, we can convert the after-softmax threshold To to a before-softmax threshold T(Ag, Ag, .... Ati TR) for A₁. Then the Eq. (26) can be express
J PA. (a)) da, = 0.5.
(29)
When takes the {k + 1)-th variable Ast1 into consider. Since the Σk e b++ e always satisfied, r(A,, Ag..., Ag, Tx) > T(A2, Ag,.., Asi Th) is also always satisfied. Consider the function Fxx), which is defined below:
13

---

Published as a conference paper at ICLR 2022
00
100
2
F(a) =
PA (a.) da;
(30)
k-1
F(x) strietly monotone increasing function, which means F(r(A2, Ag. Akt1; mx)) >
((r(A2, TR)
Then we have
10
(A Ac- 1++11
PA (a:) day-
(31)
k
00
00
0
AA2.Aa..... AsiTa)
k+1
PA (a:) da;
(32)
==1
k-1
7 (A2 Agita)
PA. (a:) da,
(33)
k-
=0.5.
(34)
Since the information entropy of sign(sottmaxx+114 ((((AA-TTK)))) TK+1)) maximized, we have
AA2. An k+1
PA. (a,) da, = 0.5,
(35)
00
000
-00
-00
thus,
000
000
100
T Ae.
II PA (a.) da,
(36)
1.
8
30
T Al Ac
R+1 pa (a.) day.
(37)
Then can get Tk+1 Tk- Therefore, the threshold which maximizes the information entropy H(B²) negatively correlated the number of elements k.
A.3 PROOF OF THE THEOREM
Theorem 2. When the binarized query BQ sign(() [-1,1)NXD and key Bx sign(K)
(N1,1)NXD entropy maximized in binarized attention, the probability mass function of each element Aij. i.j [1, N) sampled from attention score A BQ & Bx can be represented as (2(2i D) 0.5 Cb [0, D). which approximates the Gaussian distribution NO, D).
14

---

Published as a conference paper at ICLR 2022
Proof. First, prove that the distribution of Ar can be approximated normal distribution. Considering the definition of A, Aij can be expressed as
A.
BQ,a x BK.jls
where Bq.a represents the j-th element of i-th vector of BQ and B, represents the j-th element of i-th vector of Bk. The value of the element Bq,a x Bk.ji can be expressed as
1, if oq.a Y Bk,t BQ x BK, -1, if Bq.a Bk.jl = =-1.
(38)
The BQ. X Bx.jt only can take from two values and its value can considered as the result of one Bernoulli trial. Thus for the random variable Aij sampled from the output tensor A, the probability mass function, PA can be expressed as
D) = -cbp pc) P-)
(39)
where Pe denotes the probability that the element BQ.i Bk.jt takes 1. According to the De Moivre-Laplace theorem (Walker & Helen, 1985), the normal distribution (n, o²) can be used as an approximation of the binomial distribution under certain conditions, and the PA D) can be approximated as
(1- 4p- (1pp
P2 D) Ch pe) D-i
(40)
2T Dpe (1 - Pe)
and then, get the mean μ 0 and variance Now we give proof of this below.
vD of the approximated distribution N.
According to Proposition 1, both BQ and B, have equal probability to be 1 or 1,, which means Pe 0.5. Then we can rewrite the equation as
P2 D) = 0.5% (0,1,
(41)
Then we to calculate the mean and standard variation of this distribution. The mean of this distribution is defined as
"(ps) D
D).
(42)
By the virtue of binomial coefficient, we have
(2i - D)0.5% + (2(D - D) (21
(22- (D (D-
(43)
0.5 ((2 D)C; (D - 2i)C()
(44)
=
(45)
Besides, when D is an even number, we have (2i - D)0.5 = 0,
D. 2.
These equations prove
the symmetry of function (2i - D)0.5 Finally, we have
m(PA)= = E(2 D)o.5 E (0,1,2 D)
(46)
D.
((2 + (2 i) - DF- 1,,1,2,
D
(47)
0.
(48)
15

---

Published as a conference paper at ICLR 2022
The standard variation of PA is defined
opp)) =
Σ 22 - -)00.500)
(49)
(4i2 - + D2)
(50)
-0.5 4 P
4D 4D ich [D D2 cb)
(51)
To calculate the standard variation of pz, we use Binomial Theorem and have several identical equations:
- =2
(52)
Σ Σich D(1+ 11) 11)--1112 DD2
(53)
Σ D&G D(D + 1)(1 + 1)0-2 = D(D + 1-D22-2
(54)
These identical equations help simplify Eq. (51):
(PA)) 0..5 (4 PCb - 4D -DD
iCh iih +2 D
(55)
-V05P(AD(D+ (D)((DD())0-22220 1)20-2
4D220-1 - D22P)
(56)
= (OS* D)2P - 2D22D + D22D)
(57)
(D2D)
(58)
= -VD.
(59)
Now we proved that, the distribution of output is approximate normal distribution (V D).
A.4 BAMM OPERATION IN BI-ATTENTION
In the Bi-Attention structure, we apply the bool function to obtain the binarized attention weights B with values of 0 and 1, which maximizes the information entropy while restoring the perception of attention mechanism for the input. However, we noticed that in the actual hardware deployment, the 1 -bit matrix stored in the hardware is unified into the same form (with binary values of and -1) and is supported by most existing hardware.
Therefore, propose new bitwise operation X to support the computation between the binarized attention weight bool(A) and the binarized value By during inference, which is defined as
bool(A) X By BA' ④ By + BA 11) >
(60)
where BA' E N-1,1]NxD is the representation of bool(A) on hardware that BA = = where bool(A); bij 0. NxD is all ones matrix, and @ and < is the bitwise matrix multiplication and bit-shift operation. The X can also be simulated as the common full-precision multiplication as the @ during training.
We show the calculation process of the proposed X operation in Figure 7, which also concludes the process of 6o.
16

---

Published as a conference paper at ICLR 2022
BA
Training
X
3
X
popcount
B2
bitwise matrix multiplication
i-bitreppeesettaiion on hardware
the proposed operation g
B₂
3
bitshit
3
Inference
+
X
4
41
Q3
62
06
04
AL
66
Figure 7: The calculation process of proposed X and @ in Bi-Attention
A.5 PROOF AND DISCUSSION OF THE THEOREM
Theorem 4. Given the variables X and XT follow (0(0 σ2) respectively, the proportion
of optimization direction error is defined as Peror Q-bir = p(sign(X -r) & sign (quantizeee((
Xr)), where quantizeq denotes the Q-bit symmetric quantization. As Q reduces from 8 to 1, Pemor Q-bit becomes larger.
Proof. Given the random variables X quantization function quantizeq expressed as (take X example)
o1) and Xy .N(0, the Q-bit symmetric
- T
if < -T, if - L L,
quantize
(2 2L
2L + +0.5
(61)
24-1
if > L,
where the I.] denotes the round down function, and the range [- L] is divided into 2 inter. The optimization direction enror occurs when sign(X - X) sign(quantize (X) Xr), i.e., X > XT and quantize (X) Xxr or X < XT and quantize > Xr.
(1) When - < X L,
(29-1)X
2L 24
quantize (X) < 129-11X + 1 24
20-1
Xx < (29
(2 12
-1 < X, we have Xy < X. And since < quantize o (X), Xr < quantize (X). Thus, the optimization direction is
2L 1. since
129-1)X 1)X
(29
2L
21.
21.
21.
2L
-4-1
always right in this case.
b) IF XT
(2 1)x
2L
21 X, have XT > X. And > quantize (X), XT > quantize (X). Thus, the optimization direction
1J
2L - since 129
21 2
since
(29-1)X
2L
25.
24-1
always right in this case.
c) If
(28-1)X
2L
<
XT
(2--1)
27. + 1
2L
first the probability of X >
- Xyr and quantize (X) Xy can be calculated
21.
17

---

Published as a conference paper at ICLR 2022
Table 4: Simulation of error proportion under the Q-bit
Bits (Q)
2
4
5
6
7
8
Proportion
14.36%
6.42%
4.35%
3.30%
2.76%
2.56%
2.51 %
2.49%
Perrarl = /
p x
/
(62)
24 ux
+0.5]
20-1
(29 1)xX
2L +
0.5| < X dXrdX,
(63)
where (x. is the probability density function of the joint probability distribution for {X, XT), and []] denotes the Iverson brackel as defined in Eq. (26).
Then we get the probability of X NT and quantize > XT as
(24-1)x
+0.31 72-1
22
Peroo22 Q
-
(64)
(29
-1)X
+ 0.5] > X dXrdX.
(65)
2L
Since xx,x, (x, 0 is constant established, Perrort Q and Pemor Q increases becomes smaller.
(2) When X I,, quantize L. X x L quantize (X) is constant established when Xy > X, and when Xy X, the probability of Xy quantize L is also constant based on the given distribution of Xr. Thus, the optimization direction is always right in this case.
-T. XT X < - quantize,(X) is constant established when Xr X, and when X, X, the probability of Xr quantizoo(X)) - is also constant based on the given distribution of Xy. Thus, the optimization direction is always right in this case.
(3) When X L,, quantizeq(X)
Although we have obtained the correlation between quantization bit-width and error of direction under symmetric quantization, is difficult directly give an analytical representation of the error proportion of Gaussian distribution input under Q-bit quantization. Therefore, we use the Monte Carlo algorithm to simulate the probability of directional error caused by Q-bit by the error propor- tion of the pre-quantized data X R 110000 where each element in X is sampled from the standard normal distribution. When the quantization range [-1, 1], the results are shown in Table 4. This result experimentally proved theorem, and shows that the probability of direction mismatch increases rapidly in 1-bit quantization.
A.6 PROOF AND DISCUSSION OF THE EQUIVALENCE
Considering the order-preserving characteristic of both sign and softmax, only the largest ele-
ments (the value of n is related to the variable & for specific matrix A Theorem 1) are binarized to I while others are binarized to -1. Therefore, when sign(softmax(A) - -) is optimized to have the maximized information entropy, there exists a corresponding threshold Φ(T, A) that maxi- mizes the information entropy of sign(A (A- Φ(7, A)). In addition, the conclusion of Theorem 2 that A ~ (0, D) suggests that a fixed threshold of T, A) = 0 maximizes the information entropy of the bnnarzzed attention weight.
18

---

Published as a conference paper at ICLR 2022
When the elements in tensor A assumed independent and identically dis- tributed, there exists a o(T, A)) making the entropy maximization of sign (A (TTT, be equiva- lent that of (ign(softmax((A) T).
Proposition 2.
Proof. As shown in Eq. (8), the information entropy of B sign(softmax(A) - can be expressed as:
H(B -((B))=((f( f(a")da") log (f, flar)da") (L f(a")da" log (S f(a" d(a))da))
(66)
where a" = softmaxA and E A. Since softmax function is strictly order- order-preservin preservieg. there exists a ΦTT, A) makes Va > T, > d(r, A), and thereby makes following equations established:
[. I(a")da" = -faa
JMLr.A) g(a)da, S Iaa")da" = f g(a)da,
(67)
where and the probability density functions of variable a" and And the information entropy H(B²) H(BA), where B sign(A é(T, A)). Therefore, when 7 is optimized to maximize information, the information of B is also maximized under the threshold (fF,
B EXPERIMENTAL SETUP
B. DATASET AND METRICS
We evaluate our method on the General Language Understanding Evaluation (Wang et al., 201 8a) (GLUE) benchmark which consists of nine basic language tasks. And we use the standard metrics for each GLUE task to measure the advantage of our method. We use Spearman Correlation for STS-B, Mathews Correlation Coefficient for CoLA and classification accuracy for the rest tasks. As for MNLI task, we report the accuracy on both in-domain evaluation MNLI-match (MNLI-m) and gross-domain evaluation MNLI-misee miiI-mismatch (MNLI-mm) But we exclude WNLI task as previous studies do for its relatively small data volume and unstable behavior. We also theoretically calculate the FL.OPs at inference and also report the model size to give a comprehensive comparison on speed and storage.
We also provide a coarse estimation of model size and floating-point operations (FLOPs) at inference follow (Bai et al., 2020; Zhou et al., 2016; Liu et al., 2018). The matrix multiplication between an m-bit number and an nibit number requires mn mn/64 FLOPs for a CPU with instruction size of 64- bit. Also, every 64 weight and embedding parameters are packed and stored together with single instruction, so the model is significantly compact.
B.2 IMPLEMENTATION
Backbone. We follow (Bai et al., 2020) take full- melliprecision well-trained DynaBERT (Hou al., 2020) as the teacher model to self-supervise the training of the binarized DynaBERT. The number of transformer layers is 12, the hidden state size is 768 and the number of heads in MHA 12, which typical setting of BERT-base. Unlike (Bai et al., 2020), do not pre-train model as the initialization, but directly binarize the model from the full full-precine precision one. We use naive sign function to binarize parameters in the forward propagation and STE in the back-prop.g, and also use MSE loss to distill the teacher model as described Sec. 2.2 as the baseline binarized BERT model.
intermediate
Binarized Layers. We follow the previous work to binarize the word embedding layer, MHA and FFN in transformer layers, but leave full-precision classifier, position embedding layer, and token
19

---

Published as a conference paper at ICLR 2022
type embedding layer (Bai et al., 2020). That is because a common practice for BERT is to trivially cast the regression tasks to multiclass classification tasks, such as STS-B with 5 classes and MNLI with 3 classes. Therefore, the answers for above two tasks cannot be encoded by just 1-bit.
Settings. We use the Adam as our optimizer, and adopt data augmentation on GLUE tasks except MNLI and QQP for the little benefit but it is time-consuming. It is noteworthy that take more training epochs for every quantization method on each tasks to have a sufficient training, which is 50 for CoLA, 20 for MRPC, STS-B and RTE, 10 for SST-2 and QNLI, 5 for MNLI and QQP.
Other Quantized BERTs. We implement TernaryBERT under 2-2-1 and 2-2-2 settings, and Bina- ryBERT under 1-1-1 and 1-1-2 settings for the compraisonal experiments.
For BinaryBERT (Bai et al., 2020), except 4-bit and 8-bit activa- tion settings mentioned in their paper, the official code (https:// github. www.h/wwe-eeaeeieeeeeeaae howeeeeriedeeeteteeieeeeeeee Language Model/blob/ L.baf58f17h8eea0711 4533//066055549ccccccccaaaaaa
provides the (BWN) (TWN), respectively, and we completely follow the released code.
implementations for 1-bit or 2-bit activation settings, which are and
Specifically, we apply ods to activation under b-bit and 2-bit input settings (1-1-1 and 1-1-2), respectively. The formula- tions and implementations are presented in (Bai et al., 2020) and official codes, respectively:
(BWN) and Weinay-Weigh--eeroott (TWN) meth-
BWN:
a. = (a.) = B sign (a.),
B =
aaa
(68)
TWN:
a 2 (a.) = { sign (af)
2 A
(69)
A
where sign(-) (-) is the sign function,
0.7 | and H Eiez
lwl]l with I (i [ + 0).
For TernaryBERT, since there no recommended setting under 1-bit and 2-bit input in its official code, the same implementation BinaryBERT to quantize activation (BWN for 1-bit and TWN for 2-bit) for fair comparison.
In addition, compared with TernaryBERT and BinaryBERT, under the 1-bit input setting, our BiB- ERT applies a simple sign function to binarize activation, while the former two apply real-time calculated scaling factors. Therefore, under the same 1-1-1 setting, BiBERT achieved better results with a smaller amount of computation.
C ADDITION EXPERIMENTS
C.I COMPARISON AND DISCUSSION OF THE BASELINE
C.1.1 REASONS OF BUILDING THE ORIGINAL BASELINE
When we built the fully binarized BERT baseline, we considered the following reasons and chose the original baseline settings shown in the paper:
(1) Additional inference computation should be avoided to the greatest extent compared with the BERTs obtained by direct full binarization. We discard the scaling factor for activation in the bi- linear unit since it requires real-time updating during inference and increases floating-point oper- ations. In the attention structure, we directly binarize the activation without re-scaling or mean- shifting.
(2) Since there is no previous work to fully binarize BERT, the existing representative binarization techniques are chosen build the baseline. The binarization function with fixed 0 threshold is applied to the original definition of the binarized neural network (Rastegari et al., 2016) and is used by default in most binarization works (Qin et al.. 2020; Liu et al., 2018), we thereby initially apply this function when building the fully binarized BERT bascline. Besides, it also helps to more objectively evaluate the benefits of maximizing information entropy in the attention structure.
20

---

Published as a conference paper at ICLR 2022
Table 5: Comparison of strengthened baselines and methods without data augmentation.
Solution Quant
#Bits AFLOPs (G)
MNLI m/mm QQPQNLISST- COLA MRRPC Avg.
Baseline
1-1-1
0
45.8/47.0
73.2
66.4
77.6
11.7
7.6
70.2
54.1
50.4
Original
BinaryBERT
1-1-1
0
35.6/35.3
63.1
51.5
53.2
0
6.1
68.3
52.7
40.6
Solution
Basclineasym
1-1-1 0.074 (15%)
45.1/46.3
72.9
64.3
72.8
4.6
9.8
68.3
53.1 48.6
Solution 2 Bascline,
1-1-1 0.076 (16%)
48.2/49.5
73.8
68.7
81.9
16.9
11.5
70.0
54.9 52.8
Solution 3 Baselineao%
1-1-1 0.076
(16%)
47.7/49.1
74.1
67.9
80.0
14.0
11.5
69.8
54.5
52.1
BinaryBERT
50%
1-1-1 0.076
(16%)
39.2/40.0
66.7
59.5
54.1
4.3
6.8
68.3
53.4
43.5
Ours
BiBERT
1-1-1
0
67.3/68.6
84.8
72.6
88.7
25.4
33.6
72.5
57.4 63.5
Therefore, the fully binarized baseline under the current BERT architecture applies the representa- tive techniques and almost without additional floating-pone point computation.
C.1.2 SOLUTIONS OF ATTENTION WEIGHT
Causing by the application of softmax, the attention possibilities in the attention structure follow the probability distribution and (attention weight) are all one after binarization. In fact, we were aware of the problem when building the fully binarized baseline, and built two solutions by existing quantization techniques.
The first solution tried is to degenerate the asymmetric quantization function (usually applied to 2-8 bit quantization (Bai et al., 2020; Zhang et al., 2020)) to the 1-bit case, since the method is originally designed to deal with the imbalance of value distribution quantization:
Solution (Baselineaysm):
[max(xx x), x 2 0.5(max(x) + min(x)), min otherwise,
Q(z)
x E
(70)
The other solution is to simply use the mean of the elements as the threshold (Rastegari et al., 2016; Qin al., 2020): Solution (Baseline,):
(xx) bool(x r), T= p(x)
(71)
where ud-) denotes the mean value.
We also thank the reviewers for providing the Solution 3, which use the threshold limiting the zero attention weight to certain percentage (such as 50%):
Solution (Baseline50%)
(x) = bool(x - r), Qau% (x)
(72)
where Q50%(-) (-) denotes the quantile of 50% percentage (median).
The above three solutions are all able to alleviate the problem of fixed all one attention weight. We present the results of these solutions in Table A and further discuss them in detail.
(1) Discussion of Solution 1
As shown Eq. (70), since the dynamically threshold (0.5(max(x)) min(x))) is applied this quantizer, the elements of attention weight quantized to max(x) and min(x) instead of same value. However, the asymmetrical binarized values obtained by this quantizer make the binarized BERT hard apply bitwise operations and lose the efficiency advantage brought by binarization (0.074G additional FLOPs), so that this practice not used in existing binarization works. And as the results show, the accuracy of the binarized BERT is also worse than the existing baseline (Baseline 50.4% vs. Baselineasym 48.6% on average, Row 2 and 4 in Table 5).
(2) Discussion of Solution 2
21

---

Published as a conference paper at ICLR 2022
Table 6: Ablation results with 10%90% zero attention weight on STS-B.
Percentage 10%
30%
50%
70%
90%
Entropy
0.47
0.88
1.00
0.88
0.47
Accuracy (%)
8.5
8.6
11.5
10.8
10.0
As the effect of this solution on the weight parameter (Rastegari et al., 2016; Qin et al., 2020), it can ensure diversity of binarized attention weight elements instead of all (min(() S T S max(x)). But the premise of this practice is the Gaussian (symmetric) distribution of floating point parameters before binarization (Qin et al., 2020), while the attention score follows an asymmetric probability distribution, causing the values of binarized activation to imbalanced. Therefore, the improvement of this practice in terms of accuracy is also limited (as shown Row and in Table 5), and this solution also increases 0.076G computational FLOPs compared the original baseline.
(3) Discussion of Solution 3
The fixed 50% percentage of zero binarized attention weights suggested by the reviewer are good baseline settings that help to further clarify the entropy maximization motivation the Bi-Attention in BiBERT. As shown in Table 5 (Row 6 and 7), the results of baseline and BinaryBERT under this setting are significantly improved by 1.7% and 2.9% on average compare the original results (Row 2 and 3), respectively. And from the perspective of information entropy, 50% percentage of zero attention weights can also ensure maximum information. We present the detailed ablation results (10%~90% zero attention weight) on STS-B in Table which show the model maximizing information entropy achieves the best results.
Though foreing certain ratio of zero attention weights improves the baseline, there exists short- comings for fully binarized BERT'. First, the calculation of quantile thresholds relies on real- time computation or sorting, which increases computation of the fully binarized BERT in inference (about 0.076G (16%) additional FLOPs). Moreover, the practice of 50% quantile threshold should be regarded as the solution to a more stringent optimization problem (rather than the optimization problem in Eq. (9) of the paper) since further constrains and maximizes the entropy of each ten- sor of binarized attention weight instead of optimizing the overall distribution. Thus, the quantile threshold is more restrictive for the binarized attention weights and limits their representation capa- bility, causing the results of obtained fully binarized BERTs lower than those models applying Bi-Attention.
C.1.3 CONCLUSION
Although the above three solutions improve accuracy to certain extent, they only bring limited improvements while destroying the advantages of the fully binarized network on computational efficiency. As analyzed in our paper, the existing attention structure is not suitable for directly fully binarizing BERT, which is also the reason we specially designed the Bi-Attention structure for the fully binarized BERT. However, considering the reason that the fixed all attention weight causes itself and related distillation to fail, and also to more comprehensively compare with our method, present the results of these solutions (as strengthened baselines).
C.2 VARIANTS OF BINARIZED ATTENTION MECHANISM
In binarization literature, function sign is widely used to quantize real number & E IR into y 1), However, directly applying sign on attention weight against the constraint that attention weight should be in [0, 1]. Inspired by this fact, define function bool to map into {0, 1}. We empirically evaluate these two binarization methods on several GLUE tasks in Table 7.
Table 7 shows the performance of four different binarized attention mechanisms. All the experiments use the DMD method and do not use data augmentation. From these results, we can conclude that the traditional binarization method sign is not suitable to binarize attention weight since it can not selectively neglect the unrelated parts. This operation not only obstacles the optimization process but also degrades the representation ability of the whole attention mechanism. It strongly shows the
22

---

Published as a conference paper at ICLR 2022
Table 7: Comparison of Variants of Binarized Attention Mechanism
Maximizing Entropy
Binarization Method
SST-2
RTE
CoLA
1
sign
78.9
52.7
6.9
2
x
bool
77.6
54.1
11.7
3
sign
80.7
52.7
7.3
4
bool
88.7
57.4
25.4
necessity of using a binarization operator that can focus selectively on parts of the input and ignore the others.
D VISUALIZATIONS
In Figure 8, provide coarse but bird-view of attention patterns across layers and heads in different model for same inputs on SST-2 task, including the full-precision, fully binarized BERT baseline and our BiBERT. As can be seen in the figure, full-precision model always elicits certain tendency towards different contents, and each head captures unique pattern with bias. However, the attention perception is totally lost when fully binarized the BERT model with baseline methods. The attention weight is identical regardless of inputs, and exactly degrades to a fully connection.
As for BiBERT, it delivers various attention patterns among layers and heads. We concede that some patterns seem to be completely different from the full-precision ones, some are even blank. But we venture to attributes the phenomenon the special design of Bi-Attention structure. We ensure the overall probability of attention weights to have a information entropy maximized distribution, but the function and quality of each head special after an adaptive training. It results in selective attention according to the inputs, therefore we revive the perception of the model, and make it be able to elicit more diverse activation patterns.
23

---

Published as a conference paper at ICLR 2022
(a) Full-precision
(b) Fully binarized BERT baseline
BiBERT (Ours)
Figure 8: Attention pattern for full- precision model, fully binarized BERT baseline and our BiBERT
on SST-2 task. Includes all heads in layers 0-3, The visualization tools is adapted from (Vig, 2019)
24


Reviews:
[{"review_id": "o1BbEHHCSji", "reviewer": "Reviewer_yAoP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a full binarization of BERT (including 1-bit activation), which is a very challenging task. The proposed bi-attention for maximum information entropy is a very interesting design for fully binary attention models. My question here is in section 2.1, when analyzing the effect of each distillation term, whether the authors used a fully binarized attention map which only contains 1, just like the Figure 4(b). In that case, I don't understand the meaning of attention distillation in Figure 3 since the binary attention map is fixed.\n\nIt is also interesting to see that the activation distillation can cause the optimization direction when the quantization becomes more aggressive, which well motivates the proposed direction-matching distillation term.\n\nThe experimental results are solid and impressive. In the 1-1-1 setting in which the paper mainly focused, the proposed BiBERT model achieves significant improvement over the 1-1-1 baseline model.", "summary_of_the_paper": "This paper proposes a full binarization of BERT (including 1-bit activation), called BiBERT. The key observation of the paper is that binary activation becomes a big challenge for the attention weights in self-attention. To solve this problem, the paper proposes a new attention binarization approach that maximizes the information entropy. The paper further proposes a Direction-Matching Distillation (DMD) scheme which encourages the similarity pattern matrices of keys, values, and hidden states in the original BERT and BiBERT to be minimized. Experimental results on GLUE datasets show that BiBERT significantly outperforms other models under the full binarization scheme.", "summary_of_the_review": "The paper is well-motivated, novel, and well supported by the experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Yz7RXhNQPvg", "reviewer": "Reviewer_T1kK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe idea is well-motivated and easy to follow. The paper starts from information theory and focuses on the mutual information between binarized and full-precision representations. Considering that the ideal binarized representation should preserve the given full-precision counterparts as much as possible, it is very natural to maximize the mutual information between two representations. \n\nThe authors also find that the direction mismatch hinders the accurate optimization of the fully binarized BERT. This finding is interesting and well-motivated. \n\nExperiment results show that the proposed approaches achieve much better results than previous baselines. \n\nWeakness:\n\n I have noticed that [1] reported higher results in their paper.  It would be better to explain why the higher results are not reported in this paper. \n\nThe work of BinaryBERT uses 1-bit weight quantization and 4-bit activation quantization. It not only achieves better results than the proposed approach,  but also almost reaches the results of the full-precision model.  Since BinaryBERT and BiBERT have similar parameter sizes, the empirical contribution of BiBERT is a little bit marginal to me. \n\n[1] BinaryBERT: Pushing the Limit of BERT Quantization\n", "summary_of_the_paper": "This paper focuses on the full binarization of BERT. Since current pre-trained models achieve promising results on NLP tasks, it is a trend to apply these models to real-world applications with constraint computation resources.  Quantization is one of the prime choices to reduce computation requirements. In recent years, BERT quantization has made significant progress. However, training binary BERT is still an important but challenging question. This paper finds that the worse performance of binary BERT can be mainly attributed to information degradation and optimization direction mismatch respectively in the forward and backward propagation. To address these problems, the authors propose two solutions, Bi-Attention and Direction-Matching Distillation. Experimental results on GLUE benchmark demonstrate that the proposed work outperforms several popular baselines. Also, ablation studies verify the effectiveness of the proposed solutions.", "summary_of_the_review": "It is a well-motivated paper presenting an alternative understanding of the challenges of training binarized BERT. The empirical contribution is a little bit marginal since there is a large gap between the proposed approach and the best result in [1].", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YkLvJCjGB0q", "reviewer": "Reviewer_84Py", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well constructed and easy to go through the idea; encompasses rigorous theoretical justification of the approaches.\n2. This paper addresses the crucial problems in fully binarized BERT and introduces BiBERT which is the very first attempt to implement an accurate fully binarized BERT.\n3. In this framework, they incorporate a Bi-Attention approach which maximizes the information entropy to mitigate the information degradation. And a Direction Matching Distillation (DMD) mechanism which resolves the optimization direction mismatch.\n4. BiBERT also impressively obtain savings on FLOPs and model size which is a great advantage for real-world edge devices.\n5. It also contains an ablation study on the GLUE benchmark which validates the contribution of the individual components.\n\nWeaknesses:\n1. The proposed method didn’t properly address the performance drop due to the binarization of activation. Results on the GLUE benchmark showed a drastic performance drop in comparison to the related approaches and full precision BERT.\n2. In sec 2.1, it is evident that a stochastic sign function is used to binarize the BERT, therefore, why the deterministic function is used in sec 3.2.1 is not clear.\n3. In table 1,2, 3, we see that data augmentation method has been utilized for experimental purposes. But it is not properly addressed/clarified in sec 4.1 and 4.2 why it is employed and what is the significance of this approach here.\n4. Also, these two closely related works should be critically analyzed in the experiment section:\n- Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., ... & King, I. (2020). Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701.\n- Zhang, W., Hou, L., Yin, Y., Shang, L., Chen, X., Jiang, X., & Liu, Q. (2020). Ternarybert: Distillation-aware ultra-low bit bert. arXiv preprint arXiv:2009.12812.\n5. Figure 2, 3, 4 are not mentioned in appropriate sections to help understand the analysis.\n\nSome minor issues and typos:\n1. There is no full stop(.) at the end of each caption of the figures and tables.\n2. At the end of the 2nd paragraph of the Introduction: “So far, previous studies have pushed down the weight and embedding to be binarized, but none of them have ever achieved to binarize BERT with 1-bit activation accurately.” -> please refer to the specific previous studies.\n3. Section 2, first paragraph: BRET -> BERT\n4. Section 2.2: which can be unobstructed applied -> which can be unobstructedly applied\n5. Appendix B.2: There is no citation for DynaBERT.\n", "summary_of_the_paper": "This paper introduces an accurate fully binarized (i.e., 1-bit weight, embedding, and activation)  BERT, BiBERT, as a robust model compression method. It saves 56.3× and 31.2× on FLOPs and model size for real-world devices. Moreover, it addresses the substantial performance issues in the straightforward full binarization method. To tackle the performance drop issues, it proposes Bi-Attention and Direction Matching Distillation (DMD) mechanism.  The paper contains an evaluation of the proposed model on GLUE benchmark and a comparison with SOTA quantized BERT models.\n", "summary_of_the_review": "This paper portrays the related performance issues of straightforward fully binarized BERT and introduces the idea of BiBERT as a solution. The paper has proper experimental proofs and analysis to justify the framework. It might be a positive inclusion towards an accurate fully binarized BERT architecture. The proposed method obtained impressive savings on FLOPs and model size. However, it seems that the related SOTA approaches are missing in the comparative performance analysis section, and I would really be looking to see this in a conference paper that meets ICLR's high standards.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PH_M7xJQ-dj", "reviewer": "Reviewer_tpUQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors stated that the bi-linear quantization function in Equation (2) is motivated by Rastegati et al., 2016 and Qin et al., 2021.  However, (2) is quite different from the quantization methods in both papers. Specifically,\n- in Rastegati et al., 2016, both the weights and activations are not zero-meaned before quantization and have scaling factors. \n- in Qin et al., 2021, the weights are both zero-meaned and uni-normalized. \nCan the authors specify in more detail the derivation of (2), and the connection/difference with the two papers?\n\nThe theory part also requires some more clarifications. Specifically,\n- Theorem 2 is based on the assumption that B_Q and B_K are entropy maximized. However, it is not clear why this assumption holds. \n- Even though the softmax function is order-preserving, maximizing the entropy of sign(A - \\phi(\\tau, A)) is not equivalent to the same as the original problem in (9).\n- Moreover, given the implication from Theorem 2 that the optimal \\phi(\\tau) is zero,  the binarization of the attention scores  sign(A - \\phi(\\tau, A)) =  sign(A) does not equal bool(A) in equation (11).\n\nThe experiments also require some more clarification. According to the original papers of BinaryBERT and TernaryBERT, the activations are only quantized to as low as 4 bits. However, in Tables 2-3, activations of these two methods are quantized to 1 or 2 bits. How are the activations quantized for these two methods?\n\nMinors:\n- Figure 3: missing the full stop at the end of the caption.", "summary_of_the_paper": "This paper proposes to binarize both the weights and activations of the BERT model. The authors find that binarizing MHA causes the most significant accuracy drop and simple distillation on MHA by minimizing the MSE between the attention scores of the full-precision model and the quantized model can harm the performance sometimes. Thus the authors propose new binary representations that maximize entropy and distillation methods that maintain direction. Empirical results on the GLUE benchmark show that the method achieves good performance \nwhen the weights are binarized and the activation is binarized/ternarized.\n", "summary_of_the_review": "This paper is overall structured clearly and easy to follow. However, the motivation, theory part and experiments require some further clarifications.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4inq0vvKBiQ", "reviewer": "Reviewer_TtMP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well-motivated and easy to follow. To the best of my knowledge, this is the first work on full quantization (weights-embedding-activation) of the BERT model to 1-bit (denoted as 1-1-1), considering that previous work such as BinaryBERT only achieves 1-1-4 quantization. Binarizing all the activations without losing much performance is challenging. By making detailed analysis of the performance variances in binarizing intermediate outputs at each step and the different distillation losses, the authors find the major problems that hurt the model performance most. Two new methods are proposed to avoid or alleviate the problem in training the quantized model, which are theoretical sound and shows substantial improvements to the accuracies on downstream tasks. The proposed BiBERT model also surpasses the previous BERT quantization model on the setting of 1-1-1 quantization.\n\nThe weaknesses of the BiBERT model are as follows: \n1) The performance of BiBERT on GLUE is still far behind the full precision model (67.0 vs 82.3). This large performance gap may hinder application of the BiBERT model in many real scenarios (e.g., CoLA, STS-B tasks). While the BinaryBERT with 1-1-4 quantization, which has similar model size and a little bit higher computation as the BiBERT, can achieve GLUE score of 81.9 with almost neglectable performance loss. This raises the question why full binarization of the BERT model is necessary or preferred. One possible way to justify this is to enlarge the network size while keeping it binary and see if the performance can consistently improve.\n2) In the experiments, the authors only report BinaryBERT (1-1-2 and 1-1-1) and TernaryBERT (2-2-1 and 2-2-2) which are not present in their original papers. The author should clarify in the paper how the results are obtained. Are the results from reimplementation of their algorithms or based on some open-source code/models? Are the reimplementation correct? On CoLA task, BinaryBERT and TenaryBERT simply seem to diverge, and the baseline quantization method without Bi-Attention and DMD also outperform BinaryBERT and TenaryBERT.\n3) The author report Q2BERT and Q-BERT on the 2-2-8 setting, while the TernaryBERT have much higher performance (72.9) on it. Why not report these results?  \nOverall, I think the BiBERT is still not applicable to some NLU tasks given severe performance drop. Some of the experimental results are not convincing (or biased) regarding the baseline of BinaryBERT and TenaryBERT.\n", "summary_of_the_paper": "This paper addresses the problem of fully binarizing BERT model including the network weights, embeddings and activations. Through theoretical and empirical analysis, the authors find: 1) the direct binarization of softmax-ed attention matrix is problematic; 2) distillation of the attention scores from the full-precision teacher can be harmful to the binarized model. Therefore, they propose two different methods to alleviate the issues. One is the Bi-Attention which directly quantizes the attention matrix and excludes softmax. The other is Direction-Matching Distillation which chooses to distill the covariance matrices of query/key/value rather than the attention matrix. Experimental results on GLUE benchmark show that BiBERT model outperforms previous quantized BERT models on the full binarization setting.\nThis main contribution of the paper is that it is the first work on full binarization of BERT models. With detailed analysis and observations, it proposed two effective methods to improve the training of the binarized model, and achieve good performance on some of the NLU tasks.", "summary_of_the_review": "The paper tries to tackle a challenging problem of binarizing BERT and find effective ways to improve the performance. However, the model performances on some tasks are still far from good and comparison in the experiments are not convincing. Therefore, my current recommendation is weak reject unless the author can clarify the concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper present a way to fully binarize a BERT model. The authors convincingly demonstrate that a naive binarization results in large quality losses and then propose amendments. It is pretty impressive that it is possible to get a fully binarized model to work at all.
At the same time, the quality losses are still significant and in practice one might prefer to use distillation (as long as the hardware doesn't require binarization). One could also envision combinations of the proposed technique (perhaps in the 1-1-4 setting) with distillation.
Decision: Accept (Poster)