MA-BERT: Towards Matrix Arithmetic-only BERT Inference by Eliminating Complex Non-Linear Functions
Authors: Neo Wei Ming
Zhehui Wang
Cheng Liu
Rick Siow Mong Goh
Tao Luo
Venue: ICLR 2023 poster (2023)
Keywords: BERT
Efficient inference
Matrix arithmetic-only
Eleminate non-linear functions

Abstract: Due to their superior results, Transformer-based models such as BERT have become de facto standards in many Natural Language Processing (NLP) applications. However, the intensive use of complex non-linear functions within the Transformer architecture impairs its computing efficiency and complicates corresponding accelerator designs, because non-linear functions are generally computation-intensive and require special hardware support. In light of this, we propose MA-BERT, which allows matrix arithmetic-only operations in Transformer-based NLP models and achieves efficient inference with negligible accuracy loss. Specifically, we propose four correlated techniques that include approximating softmax with a two-layer neural network, replacing GELU with ReLU, fusing normalization layers with adjacent linear layers, and leveraging knowledge transfer from baseline models. Through these techniques, we are able to eliminate the major non-linear functions in Transformer-based models and obtain MA-BERT with only matrix arithmetic and trivial ReLU operations without compromising on accuracy. With mainly regular matrix arithmetic operations, MA-BERT enables hardware-friendly processing on various computing engines, including CPUs and GPUs. Our experimental results show that MA-BERT achieves up to 27% and 41% reduction in inference time on CPU and GPU, respectively, with comparable accuracy on many downstream tasks compared to the baseline BERT models. 

Published as a conference paper at ICLR 2023
-A-BERT: TOWARDS MATRIX ARITHMETIC-ONLY BERT INFERENCE BY ELIMINATING COMPLEX NON-
LINEAR FUNCTIONS
Neo Wei Ming Zhehui Wang', Cheng Liu', Rick Siow Mong Goh', Tao Luo Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), I Fusionopolis Way, 16-16 Comnexis. Singapore 138632, Republic of Singapore' School Computer Science and Engineering, Nanyang Technological University, Singapore? Institute of Computing Technology. Chinese Academy of Sciences" {neo. weiming, leto. Iuo @email com https://githab. http://wwwwwww///////// BERT
ABSTRACT
Due to their superior results, Transformer-based models such as BERT have become de facto standards in many Natural Language Processing (NLP) ap- plications. However, the intensive use of complex non-linear functions within the Transformer architecture impairs its computing efficiency and complicates corresponding accelerator designs, because non-linear functions are generally computation-intensit and require special hardware support. In light of this, we propose MA-BERT, which allows matrix arithmetic-only operations in Transformer-based NL.P models and achieves efficient inference with negligi- ble accuracy loss. Specifically, we propose four correlated techniques that in- clude approximating softmax with a two-layer neural network, replacing GELU with ReLU, fusing normalization layers with adjacent linear layers, and lever- aging knowledge transfer from baseline models. Through these techniques, we models
are able to eliminate the major non-linear functions in Transformer-base with only matrix arithmetic and trivial ReLU operations
and obtain without compromising accuracy. With mainly regular matrix arithmetic opera- tions, MA-BERT enables hardware-friendlly processing on various computing en- gines, including CPUs and GPUs. Our experimental results show that MA-BERT achieves up to 27% and 41% % reduction in inference time on CPU and GPU, re- spectively, with comparable accuracy on many downstream tasks compared to the baseline BERT models.
INTRODUCTION
Recently, pretrained Transformer-based models such as GPT (Radford et al. 2018) and BERT (De- vlin al. 2018), have consistently dominated the leaderboards for a variety of NLP tasks and even surpassed the human bascline. Consequently, there has been a strong push for these models to be used in many NLP applications. At the same time, they have become a popular rescarch topic in academia and industry. This in turn led to the creation of even better models such as GPT-3 (Brown) let al.) 2020), RoBERTa (Liu al.) 2019), and DeBERTa (He et al. 2020), which further stretched the limits of what such models can achieve. Clearly, the advent of Transformer- -asedd models has brought about paradigm shift in the NLP field from the pre-Transformer when recurrent neural networks and their variants used to dominate.
Nonetheless, the exceptional performance of these Transformer-basee models partly stems from their deep structure which involves a huge number of parameters and operations during a single forward propagation. These characteristics make it challenging meet timing requirements and implement them on devices with limited memory and computational power. Consequently, many notable works have been proposed improve their inference performance. In particular BERT,
"Corresponding author

---

Published as a conference paper at ICLR 2023
some works seek to compress the model by reducing the number of encoder layers via knowledge distillation (Sanh et al. 2019 Aguilar et al. [020]] Sun et al., 2020; 2019 Jiao et al., 2019 Xu et al.
2020) with minor accuracy penalty. Other works focus model pruming (Gao et 20.2 Gordon et al. 2020; Voita et al. 2019 Wang et al. 2021). which involves eliminating unessential weights attention heads, leading to sparser models with reduced number of parameters and computing operations, Additionally, several works leverage quantization (Kim et al. [021] Bhandare et al. 2019 Zafrir et al., 2019) to reduce the precision of weights and computing operations, which lowers
memory demands and speeds up inference.
Despite these works, one non-negligible overhead Transformer-base models is often overlooked the intensive use of complex non-linear functions. In BERT, these non-linear functions include the softmax operation, Gaussian Error Linear Unit (GELUU activation function (Hendryeks & Gimpel| 2016), and Layer Normalization (LayerNorm) (Ba et al. 2016). Although these functions undeni- ably play a role in helping the model to learn better during training time, they become a considerable bottleneck during inference they not straightforward to evaluate. Furthermore, in hardware accelerator designs, these non-linear functions typically require separate hardware for acceleration resource- (Liu et al., 2021 Khan et al. 202I), which makes them challenging to be deployed constrained computing platforms. In Figure [ and Table [T] we show the breakdown of the cycle budget for the BERT hase model. For non-linear operations, the cycle budget is defined as the equiva- lent number of cycles that the matrix multiply takes to process, given the matrix multiply dimensions and the number of multiplications per cycle (Khan et al.) (021). From the figure, we can see that those non-linear operations can take as much as 43% of the processing time. The computation ef- ficiency of BERT has huge potential to improve if we can optimize these non-linear operations.
Table 1: Cycle budget for BERT base with quence length of 128 and 2048 multiplica- tions per cycle (Khan et al.
18.6%
Percentage of Cycles
2021
18.6%
Sofimax
4.7%
LayerNorm
Functions
Cycle Budget
Percentage
0.8%
GELU
Softmax LayerNorm A
512*
0.8%
LayerNorm
57.4%
Linear Layer
36,864
4.7%
GELU
147,456
18.6%
LayerNorm B
147,456
18.6%
Linear Layer
454,656
57.4%
Figure 1: Breakdown of cycle budget
Per Attention Head
In view of this, we introduce an efficient BERT, termed MA-BERT, which makes a novel attempt to completely eliminate the complex non-linear functions in BERT by substituting them with sim- pler functions. with a two-layer neural network, (2) replacing GELU with ReLU, (3) fusing normalization layers with adjacent linear layers, and (4) leveraging knowledge transfer from pretrained baseline models. Through these techniques, MA-BERT achieves matrix arithmetic-only and trivial ReLU operations with negligible accuracy loss.
MA-BERT, four correlated techniques are applied: (1) approximating softmax
Our experiments show that the performance of MA-BERT on downstream tasks is par with the corresponding BERT baseline and yet achieves up to 27% and 41% performance improvement on general CPUs and GPUs, respectively. By eliminating the complex non-linear functions, the major- ity of the operations MA-BERT becomes regular matrix-matrix arithmetic and can potentially be deployed various computing engines, including CPUs and GPUs, without any hardware modifi- cation.
2 RELATED WORK
While so many works focus on model compression techniques, only some paid attention to the op- timization of non-linear functions in BERT and other variants of Transformerbaseed models. For instance, in MobileBERT, Sun et al. (2020) proposed a thin version of BERT huge with reduced hid- den dimensions. As part of their operational optimization, they replaced GELU with ReLU and
2

---

Published as a conference paper at ICLR 2023
removed LayerNorm to further reduce MobileBERT's latency. However, their results showed that the inclusion of these optimizations degrades the model's 's performance. Kim et al. (202T). on the other hand, quantized the non-linear functions in BERT to 32-bit integer precision, allowing these non-linear functions to be computed using faster integer arithmetic.
Specific to tackling the latency overhead of LayerNorm, Zhang & Sennrich (019 proposed root mean square layer normalization which reduces the amount of computation compared to LayerNorm by only using root mean square statistics to normalize inputs. In the field of computer vision, Yao et al. (021)) reintroduced Batch Normalization (BatchNorm) (lotfe & Szegedy 2015) as replace- ment for LayerNorm (Ba et al., 2016) into vision Transformers and demonstrated that the resulting architecture performs on par with its LayerNorm based counterpart. Similarly, |Shen et al.) (2020) proposed PowerNorm as replacement for LayerNorm in Transformers and demonstrated that Pow- erNorm can outperform LayerNorm in neural machine translation and language modelling tasks. Due to its proven performance and use of batch statisties, we have decided to adopt PowerNorm in our paper minimize the overhead introduced by the normalization operations in BERT during inference.
2022 proposed general approximation framework that Leaning onto the hardware side, Yu et al. enabled them to convert all the non- non-linear operations into table look-ups. This helped to reduce the cost of non-linear functions but resulted in types of operations and corresponding hardware.
Different from these works, our work strives to completely climinate the complex non-linear func- tions in BERT and achieve matrix arithmetic-only operations with trivial ReLU, which could benefit inference on both general computing units and accelerator designs for edge applications.
3
METHOD
This section gives brief overview the non-linear functions that exist in BERT and highlights their computation inefficiency during inference. Following that, more efficient methods are proposed.
3.1 SOFTMAX: APPROXIMATING WITH 2-LAYER NEURAL NETWORK
Every encoder layer in BERT contains a multi-head self-attention (MHA) sublayer followed by a feed forward network (FFN) sublayer. The MHA sublayer endows the model with the ability to capture various dependencies between different parts of an input sequence (Vaswani et al. 2017) and is defined follows:
MultiHead( X) = Concat (head,, head, W
where head; softmax XWQ. Q. W X
XW
Vd
In each self-attention head, a x scaled dot product of the query and key matrices is passed through softmax function to obtain the weights the value matrix. Here, denotes the sequence length. The softmax function is computed as:
exp(xs)
softmax (I:) Σ5= exp(x,,)
(3)
From Eq. it is clear that all inputs along the sequence dimension have to be processed before the softmax value of a particular input can be computed. This becomes prohibitively expensive as the input sequence length increases (Du et al.) 2019; Stevens al. 202T). Moreover, the softmax func- tion contains expensive exponentiation and division operations which further complicate accelerator designs (Hu et al. 2018). In view of this, we decided find an alternative for the softmax function.
The universal approximation theorem (Hornik et al. 1989) states that neural network with single hidden layer and non-linear activation function can approximate any function, given sufficient hid- den neurons. As such, we propose to substitute the softmax function in each layer of BERT with simple 2-layer neural network. ReLU is chosen as the activation function the hidden layer with hardware reuse in mind since will also be adopting it in the FFN sublayer. Note that this mod- ification will inevitably fix the maximum input sequence length of our modified model to the input layer size of the neural network. To eliminate the effects of any padding tokens in input sequence,
3

---

Published as a conference paper at ICLR 2023
multiplied the scaled dot product of the query and key matrices with the imput attention mask before feeding it as an imput to the neural network.
Given that the same softmax function is being approximated, it should idcally be possible to approx- imate softmax using the same neural network across the encoder layers, allowing us to reduce the parameters introduced.
3.2 GELU: REPLACING WITH RELU
Unlike the original Transformer architecture proposed by Vaswani et al. (017)) many Transformer- based NLP models such as BERT use the GELU activation function instead of ReLU their FFN layers to enhance their learning capability, GELU is defined as follows:
1 1
/ Jo 3 exp( (ap(--pee
2
GELU(x)= ((r), where ((x)
(4)
According to Eq. 4 it is clear that GELU involves complex operations such exponentiation and integration which are computation-intenssie and hard designs. One possible way is to employ a lookup table to simplify the operations, but doing so entails a trade-off between accuracy and hardware utilization. Other implementations involving function approximations (Hendrycks & Gimpel 2016 Kim et al., 2021 also exist, but they often involve multiple operations which end up consuming a notable portion of inference time (Pati et al., 2021). In contrast, ReLU simply outputs be implemented easily in hardware using a single multiplexer. With the potential simplification in hardware and speed-up during inference (Sun et al. 2020), propose to bring ReLU back replacement for GELU.
implemented efficiently in accelerator
the input value depending the input's sign and can
3.3 LAYERNORM: FUSING NORMALIZATION LAYERS WITH ADIACENT LINEAR LAYERS
Every encoder layer in BERT involves two LayerNorm sublayers one after the MHA sublayer and another after the FFN sublayer. LayerNorm involves the calculation of the mean and standard deviation a statistics along the feature dimension of size K, followed by an affine transformation:
11
B, where J
Fz.
5 p1)
(5)
K
This remains true even during inference and demands the need to wait for all features of an input token to be processed before these statistics can be computed. It would be ideal some form of batch statistics can be used need to compute these statistics and present us with the opportunity to fuse with adjacent linear layers in BERT during inference.
alternative to normalize the inputs. Doing so will climinate the
Recently, Shen et al. (200)) proposed PowerNorm as a replacement for LayerNorm in Transformers and demonstrated that PowerNorm
outperform LayerNorm in neural machine translation and language modelling tasks. Similar to BatchNorm, PowerNorm normalizes the inputs along the hatch dimension of size B and uses running statistics during inference. However, enforces only unit quadratic mean:
B
F = 7
Ft + 3, where y62 =
1
Σ
x
(6)
During inference, the parameters 7. vb, and become constants that can be merged with the param- eters in the adjacent linear layers. Specifically, if we were to replace LayerNorm with PowerNorm, the PowerNorm after the MHA sublayer can be fused with either the linear projection after the concatenation operation in Eq. [ or the linear projection from the input layer to the intermediate layer of the subsequent FFN. Similarly, the PowerNorm after the FFN sublayer can be fused with either the linear projection from the intermediate layer to the output layer of the FFN with the computation of the key, query, and value matrices in the next encoder layer. That said, due to the nost-normalization architecture of BERT, it is not possible to do complete fusion. Nonetheless, it is still possible to reduce the two PowerNorm operations in every MA-BERT encoder layer to a single scaling operation during inference, which offers us considerable reduction in computation.
4

---

Published as a conference paper at ICLR 2023
Given the prospect of minimizing the overhead of the normalization operation during inference, we propose to substitute LayerNorm BERT with PowerNorm. Note that, unlike their work, we did not include layer-scale operation before PowerNorm to prevent any additional overhead from being introduced.
3.4 KNOWLEDDGE TRANSFER
With the three optimization techniques proposed in the previous subsections, we successfully achieve a BERT model with only matrix arithmetic and trivial ReL.U operations. However, as result of the modifications, large accuracy gap exists between the untrained MA-BERT and the baseline. In order to bridge this gap, we propose to leverage knowledge transfer.
Many studies have demonstrated the effectiveness of knowledge distillation (Hinton et al. 2013 in bridging the performance gap between a student model and a teacher model. The student model is trained to mimic the behaviour of teacher model using well-designed cost function that takes into account the differences between the student output and the teacher output. Unlike many studies which used knowledge distillation to compress BERT into smaller models, we leverage knowledge transfer with the intention to train model of the same depth and hidden dimensions with a more efficient and hardware-friends architecture. Note that are using the term knowledge transfer rather than knowledge distillation, given that the latter is often associated with a smaller model learning from larger model. The following describes our procedure to transfer knowledge from an existing pretrained BERT model to our modified model, MA-BERTTbae.
3.4.1 TEACHER MODEL
We used the pretrained BERT model from HuggingFace as our teacher model. All parameters of the teacher model were frozen to prevent the student model from chasing a moving target.
3.4.2 STUDENT MODEL
For the student model (MA-BERT, base), aggregated and implemented the three modifications described in the previous subsections on a BERT model and initialized it with its pretrained parameters, wherever possible. Furthermore, the newly added 2-layer neural network was pretrained in advance to approximate the softmax function. For simplicity, the hidden layer size of the 2-layer neural network was set to be the same the input sequence length. The proposed modifications were only applied on the embedding and encoder layers of BERT, and all parameters in these layers were left unfrozen, except for the position and token type embeddings.
3.4.3 KNOWLEDGE TRANSFER OBJECTIVE
In order for the student to replicate the teacher's behaviour as closely as possible, we adopted similar layer-wise knowledge transfer technique proposed by Diao et al. (2019).
First of all, we used the cross entropy (CE) between the student's logits zd and the actual labels y to penalize any wrong classification:
Laord = CE(ZS,
Next, we considered the cross entropy between the student's logits and the teacher's logits zT:
Laoft CE ftxns/t,20 ZT/()
Compared to the actual labels, the teacher's logits provide much more information for the student to learn. A higher temperature t smoothens the probability distribution, which allows the student to better learn the intricacies of the teacher model.
Thirdly, to ensure each layer of student model, including the embedding layer, learns from the corresponding layer of the teacher model, we included the following mean-squared error (MSE) objective:
Luidn = MSE H()
(9)
5

---

Published as a conference paper at ICLR 2023
H and H. are the hidden states of the student and teacher models, respectively, and I is the number of layers the model.
Lastly, given that we are replacing the softmax function in the MHA sublayer, also penalized the discrepancy between the student's attention outputs A and the teacher's attention outputs A using:
Tottn
MSF(A Af)
(10)
i=1
Altogether, the overall objective function for knowledge transfer can be summarized as follows:
Lnet (1 xx) * Lhard + cx * Lsoft + 3 (Lridn + Lattn)
(1I)
a is used to balance the importance of achieving the actual hard target and the teacher's soft target, is used to balance the importance of mimicking the teacher's internal layer outputs relative to mimicking the final output, and is a factor to scale Latta to similar magnitude as Lhidn- With this overall objective, the student is compelled not only mimic the final output of the teacher but also its hidden layer and attention outputs. Throughout our study, we kept = 15, a 0.9, B = 1, and y 100.
4 EXPERIMENTS
In this section, we detail our pretraining setup and evaluate the performance of MA-BERT BA-BERTbae buse on several downstream tasks. We also describe the details of additional studies that we have conducted that attempt to highlight and exploit the benefits of our approach. Throughout this section, all results pertain a model input sequence length of 128 tokens.
4.1 PRETRAINING SETUP
We pretrained MA-BERT via knowledge transfer on concatenation of BookCorpus (Zhu et al. 2015) and English Wikipedia using only the Masked Language Modelling objective (Devlin al. 2018 with a masking probability of 15%. We used AdamW optimizer with a learning rate of 2e 10,000 warm-up steps from base learning rate 5e 7, and linear decay of learning rate. The training was done with a batch size of 256 over epochs, with each epoch using a differently masked dataset, on a single NVIDIA GeForce RTX 3090 for total of 3.5 days.
4.2 EVALUATION ON DOWNSTREAM TASKS
To evaluate the performance MA-BERT base, we used the GLUE benchmark (Wang et al. 2018), which consists of nine natural language understanding tasks, and the IMDb sentiment classification task (Maas et al. 2011). Following many studies, we excluded WNLI given its small dataset. For each task, we applied task-specific knowledge transfer by having MA-BERTHa base learn from fine- tuned BERT base. The same knowledge transfer objective and hyperparameter settings outlined in section 3.4.3 were used. As our goal to replicate the performance of the teacher, our baseline for comparison is simply the teacher, which is BERT buse in this case.
For smaller datasets (CoLA, MRPC, RTE, and STS-B), we used batch size of 16 and fine-tuned for 10 epochs, and for the remaining datasets, we used batch size of 32 and fine-tuned for 5 epochs. The same learning rate of 2e was used across all tasks and the sequence length was fixed at 128. To account for the variance of different runs, we report the median of 5 runs with different random initialization seeds for each task. The results are shown in Table and K Note that the teacher with the best performance among the 5 runs was used to finetune the student during task- specific knowledge transfer. The best performance both the teacher and student is reported under Appendix A.I
On average, despite the modifications, MA-BEN B--BERTTee base was able to achieve similar performance on the downstream tasks compared to BERT base- In fact, for several tasks, MA-BERT base managed to outperform its teacher. This demonstrates that adopting our approach does not lead to a degradation in the average performance of the model.

---

Published as a conference paper at ICLR 2023
4.3 HYPERPARAMETER TUNING OF AND B
Prior to the collection of results in Table and we conducted simple hyperparameter study to assess the impact of our choices for and in our knowledge transfer objective during taek-specific knowledge transfer. We varied a and 3, which effectively control how strictly the student should replicate the teacher's final and hidden layer outputs, and evaluated MA-BERT base on the 4 smaller datasets of the GLUE benchmark. The results are shown in Table Empirically, we found that MA- BERT buse performs the best downstream tasks under the strict guidance of the finetuned BERT base model with the optimal hyperparameter values a = 0.9 and 3 1.0.
Table 2: Median performance the development of GLUE benchmark. Matthews correlation is reported for CoLA while Pearson correlation is reported for STS-B, Accuracy is reported for the remaining tasks. MNLI is the average of matched and mismatched sets. AVG denotes the average of the GLUE tasks.
Model
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
AVG
BERT
58.8
84.5
86.3
91.4
91.2
69.3
92.8
89.3
83.0
MA-BERThsse
59.8
84.8
87.5
91.0
91.2
68.6
92.5
87.8
82.9
(A1.0)
(40.3)
(A1.2)
(v0.4)
(0.0)
(V0.7)
(V0.3)
(V..
(v0.1)
MA-BERT
59.0
84.7
87.0
70.0
87.3
82.8
(Shared Softmax)
(A0.2)
(40.2)
(A0.7)
(v0.3)
(v0.1)
(40.7)
(V0.4)
(.2.0)
(w0.2)
DistilBERT
55.1
82.3
85.0
88.4
90.3
62.5
90.8
87.1
80.2
MA-DistilBERT
51.3
82.3
86.7
88.4
90.3
65.3
90.8
87.5
80.3
(v3.8)
(0.0)
(A1.7)
(0.0)
(0.0)
(42.8)
(0.0)
(A0.4)
(A0.1)
DistilRoBERTaa
58.3
84.3
85.8
91.2
90.9
69.3
92.9
88.4
82.6
MA-DistilooBRRTa
51.7
83.8
86.0
90.8
90.7
67.5
91.6
87.8
81.2
(76.6)
(W0.5)
(A0.2)
(V0.4)
(V0.2)
(V
(V
(70.6)
(w1.4)
MA-DistilRoBERTa
47.6
83.2
83.8
90.5
90.5
90.8
87.1
79.9
(Linear Attention)
10.7)
(V1.1)
(T2.0)
(v0.7)
(V0.4)
(v3.6)
(v2.1)
1.33
(
Table 3: Accuracy on IMDb test set.
Table a and hyperparameter tuning results on selected GLUE tasks for MA-BERT base. AVG denotes the average of the selected GLUE tasks.
Model
IMDb
BERT base
89.3
MA-BERTwa hase
89.6 (A0.3)
MA-BERT hase
89.6 (A0.3)
0
CoLA
MRPC
RTE
STS B
AVG
(Shared Softmax)
0.3, 0.3
58.3
84.3
61.0
86.6
72.6
DistilBERT
0.6, 0.6
59.9
86.5
64.6
86.7
74.4
MA-DistilBERT
88.2 (A0.1)
0.9, 0.9
60.4
87.0
68.2
87.7
75.8
DistilRoBERTa
89.9
0.3,
60.0
86.8
65.0
87.1
74.7
MA- BERti
89.8 (v0.1)
0.6,
86.5
87.1
74.7
BERTaa
89.6 (V0.3)
0.9, 59.8
87.5
87.8
(Linear Attention)
4.4 SHARING SOFTMAX APPROXIMATION NEURAL NETWORK ACROSS LAYERS
In the previous subsection, the softmax function in each encoder layer of -ERTT hase was approx- imated using different 2-leyer neural network. However, given that is the same softmax function, it would make sense if we can share the same 2-layer neural network across all layers. Doing so would also help us reduce the additional parameters introduced by our approach. fact, we found that the weights learnt by the 2-layer neural network in each encoder layer in MA-BERT largely similar. A visualization of the weights learnt can be found Figure 5 under Appendix We attempted to retrain MA-BERT base with parameter sharing in the softmax approximation neural network using the same training procedure. Empirically, we found that sharing the 2-layer neural network to approximate softmax preserves the performance of the modified model.

---

Published as a conference paper at ICLR 2023
4.5 VERSATILITY STUDY
To evaluate the versatility of our approach, we also attempted our modifications and training pro- cedure on HuggingFace's DistilBERT (Sanh al., 2019) and DistilRoBERTa produce MA- DistilBERT and MA-DisilloBBERTa, respectively. Following DistiRRoBERTa's training proce- dure, BARDIA was instead pretrained on the OpenWebText (Gokaslan & Cohen 2019) dataset. The remaining training procedure was kept unchanged. We observed that both MA- DistilBERT and MA-DistilRoBERIa suffered significant drop in performance on the CoLA task. Nonethless, in terms of average performance, MA-DistilBERT managed to slightly outperform its teacher, DiST On the other hand, although HA-DistilRoBERTa was not able to achieve sim- ilar average performance as DistilRoBERTa, it was still able to outperform DistilBERT, which has the same hidden dimensions and depth.
This versatility study also illustrates that approach is orthogonal to model compression via knowledge distillation. We can first distil a model into a shallower version, then apply our mod- ifications to make it even more efficient and hardware friendly for inference.
4.6 ADOPTING LINEAR SEL.-ATTENTION
With the use of neural network to approximate the softmax operation, we have inevitably intro- duced additional matrix multiplication operations. The increase matrix multiplication scales in (ss*) if the hidden layer size of the neural network follows the input sequence length That said, for relatively short sequence lengths, the increase is marginal. For s 128, the increase in the total number of matrix multiplication compared to the baseline is around 5.4% (Figure Nonetheless, the issue of scalability comes to light.
For longer sequence lengths, we suggest possible solution that can reduce this overhead. Recently, Wang et al. (2020) proposed self-attention with linear complexity. This is achieved by projecting the key and value matrices from dimensions Rsxd into smaller dimensions of #xx using the projection matrices E, F e Igxxs and setting < s:
XWO ET FXWW
head; = softmant
(12)
Vd
Here, denotes the hidden dimension of the model while denotes the new projected dimension. By adopting linear self-attention, softmax has to be computed only along the reduced dimension & which enables us use smaller 2-layer neural network to replace softmax. This not only helps to reduce the overheads introduced by the additional neural network to approximate softmax but also allows us leverage the speedup offered by linear self-attention. For instance, by letting k: = 0.5s, significant savings can be achieved as illustrated in Figure 2 To take this even further, it is also possible to fuse the linear operations in the softmax approximation neural network with both the projection of the key and value matrices during inference, hiding the overhead introduced.
Increase Matrix Multipliction
80
Without Linear Self-Attention
80,0
With Linear Self-Attenton 0.5g)
60
40
21.1
20
12.5
5.4
0.
-6.88
-2.6
128
256
384
512
Sequence Length,
Figure 2: Percentage increase in matrix multiplication in an encoder layer of MA-BERT with and without linear self elffaattentonn compared to BERT against input sequence length.
To test this out, trained MA-DistilRoBERT. with linear attention by setting i: 64 using the same procedure. The loss term Latta was removed from the knowledge transfer objective due to the different attention output dimensions of the modified model and DostilRoBERTa. The results
8

---

Published as a conference paper at ICLR 2023
reported in Table and We noticed 3.3% drop in the performance of the modified model compared with the baseline DistlRooBRTaa. Nonetheless, it was still able to achieve comparable average performance (0.37% drop) as baseline DistilBERT. A revision of the training procedure might be necessary to further improve its performance and we will leave this to future studies.
4.7 EVALUATION ON COMPUTATIONAL EFFICIENCY
To evaluate the time efficiency MA-BERT during inference, we implemented DistilBERT and MA-DistilBERT on two different hardware, namely, i7-1260P CPU and RTX 3090 GPU. We used C++ with Eigen3 library for CPU implementation and Cuda C++ with Thrust and cuBLAS libraries for GPU implementation. Using the IMDb test dataset, single encoder layer inference. To account for the variance in the time taken, we report the median of 5 different runs. The results summarized in Table 5 Despite the higher number of matrix multiplications involved, AAA-iistiBERRT able to offer an inference time speedup of up to 1.27 x on CPU and 1.41x x GPU. Note that the speedups reported also applies to MA-BERT and MA-DistilRoBERTa given that their encoder layers are identical.
measured the average time taken for
Table 5: Forward propagation time of a single encoder layer DistiBBERT and HA-DistilBERT on CPU and GPU. All parameters and calculations were in 32-bit floating-pon precision.
i7-1260P CPU
RTX 3090 GPU
DistilBERT
93.0 ms
2.37 ms
MA- MA-DistiBBRRT
68.0 ms
1.39 ms
Speedup
1.27x
1.41x
5
ABLATION STUDY
In this last section, we highlight the importance of having the baseline teacher's guidance during MA-BERT's finetuning phase.
5.1 REMOVAL TEACHER'S GUIDANCE
To demonstrate the importance of task-specific knowledge transfer during finetuning, repeated our finetuning procedure on the GLUE tasks by replacing the knowledge transfer objective with only the hard target loss for our student model to optimize. The results reported in Table We found that without any guidance from the teacher, the finetuning process is often unstable with the student model ending up performing worse in all tasks.
Table 6: Effects of removing the teacher's guidance during finetuning the GLUE tasks.
Model
Knowledge Transfer
CoLAA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
MA-BERTuss
59.8
84.8
87.5
91.0
91.2
68.6
92.5
87.8
X
31.5
35.3
80.9
50.5
63.2
61.4
86.8
81.6
MA- DistiitiBERT
V
$1.3
82.3
86.7
88.4
90.3
65.3
90.8
87.5
X
1.8
35.3
68.4
50.7
63.2
53.1
85.6
85.3
CONCLUSION
In this paper, we introduced MA-BERT, an efficient and hardware-friendly version of BERT with all its complex non-linear functions eliminated. Through knowledge transfer during pretraining and finetuning. MA-BERT able to attain comparable performance to BERT. Yet, compared to BERT, MA-BERI offers reduction in inference time with potential savings in hardware resources, making a suitable solution for applications on edge devices. In the future, it would be interesting to study the applicability of approach other transformer-based models.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We gratefully acknowledge the support by the Singapore Govemment's Research, Innovation and Enterprise 2020 Plan (Advanced Manufacturing and Engineering domain) under Grant A A1892b0026.
REFERENCES
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Chenlei Guo. Knowledge distillation from internal representations. In Proceedings the AAAI Conference on Artificial Intelligence, volume 34, pp. 7350-7357, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprini arXiv: 1607.66450, 2016.
Aishwarya Bhandare. Vamsi Sripathi, Deepthi Karkada, Vivek Menon, Sun Choi, Kushal Datta, and Vikram Saletore. Efficient 8-bit quantization of transformer neural machine language translation model. arXiv preprint arXiv: 005300 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Jacob Devlin, Ming- Keng Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: / 1800048055 2018.
Gaoming Du, Chao Tian, Zhenmin Li, Duoli Zhang, Yongsheng Yin, and Yiming Ouyang. Efficient softmax hardware architecture for deep neural networks. In Proceedings of the 2019 on Great Lakes Symposium on VISII pp. 75-80, 2019.
Yang Gao, Nicolo Colombo, and Wei Wang. Adapting by pruning: A case study on bert. arXiv preprint arXiv:2105.03344 2021.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http:://8kyllon007.giiih
io/
@penRebbeextooppss 2019.
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. Compressing bert: Studying the effects of weight pruning transfer learning. arXiv preprint arXiv:2002.083007 2020.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 2020.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
Neiv:1606.084/55 2016.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025511 2(7), 2015.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks uni- versal approximators. Neural networks, 2(5):359-366, 1989.
Ruofei Hu, Binren Tian, Shouyi Yin, and Shaojun Wei. Efficient hardware architecture of softmax layer in deep neural network. In 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP), pp. 1-5. IEBE, 2018
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456. PMLR, 2015.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tenybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 1909. 10351, 2019.
10

---

Published as a conference paper at ICLR 2023
Hamza Khan, Asma Khan, Zainab Khan, Lun Bin Huang, Kun Wang, and Lei He. Npe: An fpga- based overlay processor for natural language processing. arXiv preprint arXiv:2104.06535, 2021.
Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer- only bert quantization. In International conference on machine learning, pp. 5506-5518, PMLR, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019.
Zejian Liu, Gang Li, and Jian Cheng. Hardware acceleration of fully quantized bert for efficient nat- ural language processing. In 202/ Design, Automation Test in Europe Conference & Exhibition (DATE), pp. 513-516. IEEE, 2021.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Suchita Pati, Shaizeen Aga, Nuwan Jayasena, and Matthew Sinclair. Demystifying bert: Implica- tions for accelerator design. arXiv preprint www.v::214408.5.. 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rXivv1110.0/1100 2019.
Sheng Shen, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Powernorm: Re- thinking batch normalization in transformers. In International Conference on Machine Learning, pp. 8741-8751. PMLR, 2020.
Jacob R Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany, and Anand Raghunathan. Softermax: Hardware/software co-design of an efficient softmax for transformers. In 2021 58th ACM/EEE Design Automation Conference (DAC), pp. 469-474. IEEE, 2021.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint 1908 00355.0 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou Mobile- bert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.0290 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest be pruned. arXiv preprint arXiv:190S.094/8, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 18X4::8804.07444 2018
Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 202/ IEEE International Symposium on Perfo-eerfom Computer Architecture (HPCA), pp. 97-110. IEEE, 2021.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04476 047688 2020.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. of Compressing bert by progressive module replacing. arXiv preprint arXiv:2002.02925, 2020.
11

---

Published as a conference paper at ICLR 2023
Zhuliang Yao, Yue Cao, Yutong Lin, Ze Liu, Zheng Zhang, and Han Hu. Leveraging batch nor- malization for vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 413-422, 2021.
Joonsang Yu, Junki Park, Seongmin Park, Minsoo Kim, Sihwa Lee, Dong Hyun Lee, and Jungwook Choi. Nn-lut: neural approximation of non-linear operations for efficient transformer inference. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pp- 577-582, 2022.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-Neuue -NeurlPS Edition (EMC2-NIPS), pp. 36-39. IEEE, 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
A APPENDIX
A.1 PERFORMANCE OF BEST PERFORMING TEACHER AND STUDENT MODELS
During task-specific knowledge transfer, the best-performing finetuned teacher was used as the teacher for the pretrained student model. Table shows the best performance achieved by the base- line teacher models and the modified student models among the 5 runs for each task.
Table 7: Best performance on the dev set of GLUE benchmark and test set of IMDb. Matthews cor-
relation is reported for CoLA while Pearson correlation reported for STS-B. Accuracy is reported for the remaining tasks. MNLI is the average of matched and mismatched sets
Model
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
IMDb
BERT
59.4
84,9
87.5
91.5
91.4
69.7
93.0
89.5
89.5
MA-BERThse
61.3
84.9
88.0
91.1
91.3
70.8
92.7
88.1
89.8
(A
(0.0)
(40.5)
(V0.4)
(V0.1)
(A1.1)
(V0.3)
(V1.4)
(A0.3)
MA-BERT bo
62.1
85.0
87.3
91.3
91.2
70.4
92.8
87.6
89.7
(Shared Softmax)
(42.7)
(40.1)
(w0.2)
(W0.2)
(w0.2)
(.
(w0,2)
(v
(A0.2)
DistiBERT
55.5
82.5
86.0
88.7
90.4
64.3
91.4
87.4
88.2
MA-DistiBBERT
52.4
82.4
87.0
88.7
90.4
66.4
91.4
87.7
88.3
(W3.1)
(10.1)
(A1.0)
(0.0)
(0.0)
(42.1)
(0.0)
(A0.3)
(A0.1)
DistilRoBERIa
59.8
84.5
91.3
91.0
71.8
93.0
90.2
MA-DistiRooERTa
51.8
83.9
86.8
91.1
90.8
68.2
92.0
87.9
89.8
(W8.0)
(W0.9)
(A0.5)
(v0.2)
(W0.2)
(v3.6)
(v
(V0.9)
(T0.4)
MA-Di
49.7
83.4
85.8
90.6
90.5
66.4
91.1
87.5
89.7
(Linear Attention)
(V10.1)
(T1.1)
(V0.5)
(V0.7)
(V0.5)
(v5.4)
(.1.
(V1.3)
(T0.5)
A.2 ANALYSIS OF SOFTMAX APPROXIMATION
After pretraining MA-BERT hase. we decided to evaluate the effectiveness of the 2-layer neural net- work in approximating softmax. We fed MA-BERT buse with the sample input sequence "the quick brown fox jumps over the lazy dog" and plotted out the multi-head self celf-attention scores produced in each its encoder layers. We then compared them with the corresponding encoder layer attention scores in BERT hase- Figure shows a side-by-side comparison of the attention scores produced in the 4th 8th and th encoder layers of MA-BE BA--ERRT pase and BERT base- We observe that the attention scores produced by both models similar, which demonstrates the effectiveness of the 2-layer neural network in approximating softmax. The same observation is made when we compare the attention outputs between MA-DistilBERT and DistilBERT in Figure
12

---

Published as a conference paper at ICLR 2023
Moreover, we decided look into the weights that are learnt by the 2-layer neural network to investigate if any meaningful mappings are learnt. Figure shows the learnt weights mapping the input layer to the hidden layer and the hidden layer to the output layer for each encoder layer. Interestingly, despite giving MA-BERT hase the opportunity to learn a different mapping function to approximate softmax in different encoder layers, each 2-layer neural network ended up learning similar mapping. In fact, when we added the constraint of sharing the 2-layer neural network across all 12 encoder layers MA-BERThss (Shared Softmax), similar mapping also learnt. Note that the mappings are rather sparse and do not seem to be random, which suggests that the 2-layer neural network has learnt something meaningful. The same observation is made when we looked into the learnt weights in MA-DistilBERY in Figure 6 In fact, we can see that the mappings learnt MA-DistilBERT and MA-BERT base are similar as well. This further suggests that there some underlying function that can approximate the role of softmax in self-attention and we believe that this will serve as an interesting point for future research.
13

---

Published as a conference paper at ICLR 2023
MA-BERT Layer
lnee
ltad
Itad4
Nort
BERT Encoder Layer
MA-BERT Encoder Layer
BERT Encoder Layer 12
MA-BERT Encoder Layer 12
1 1777
Figure 3: Attention scores produced after multi-head self-attention in encoder layer 4, 8, 12 of BERT buse (left) and MA-BERT base (right).
14

---

Published as a conference paper at ICLR 2023
Layer
Ined
liad3
Htad4
1717
Ineds
lesce
1801
1727
100
DistiBERTT Encoder Layer
MA-DistilBERT Encoder Layer
17170
Hod1
DistiiBERT Encoder Layer
MA-DiztiiBERT Encoder Layer 6
1277
1 "PiiH
Figure 4: Attention produced after multi-head self-attention in encoder layer 2, 4, of Dis- tilBERT (left) and MA-DistilBERT (right).
15

---

Published as a conference paper at ICLR 2023
Softmax Approximation H-put-to-Hiddee Mappings
Approximation Inputrto-Hilden Mappings
MA-BERTawo (Shared Softmax)
Encader
EncoderLaper2
Etooder
Enander
Lreode: Laye
Layet
Stoodhr Lager
E10000 Lext
40
80
Ereader
Draader Lave
23
60
120
1.0
Softmax Approximation Hidden-to-Qutput Mappings
Softmax Approximation Hidden-to-Outpp:
MA-BERT
MA-BERT basa (Shared Softmax)
Ireader Layer:
Encoder
Sooodkriaye">
Encoder Layer
40-
80
Texidder Lipen 1
Tecider 1ee
Eroiie Layt
Oyolee
120
Figure 5: Input-to-hidded (top) and hidden-to-ouse touuuut (bottom) mappings learnt by the 2-layer neural network in each MA-BERT base encoder layer and the shared 2-layer neural network in MA-BERT, base (Shared Softmax).
Softmax Approximation Houut-to-Hidden Mappings
Softmax Approximation Hidden-to-Outppi Mappings
MA Dist-istiBBER ERT Sscade koper
MA Dis-DiiSBBERR Sooide spprr
Groude
Ghouder Laper
Layar
Gresderuyye
Excade
Crodder
Excacer
Figure 6: Input-to-hidder (left) and hidden-to-outphe (right) mappings learnt by the 2-layer neural network in each MA-DistilBERT encoder layer.
16


Reviews:
[{"review_id": "quTpWN2zhg", "reviewer": "Reviewer_9tBM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at reducing the complexity of BERT models to accelerate their inference by putting together a series of previously-introduced techniques. To this end, the authors propose to approximate softmax with a two-layer neural network, replace GELU with ReLU, fuse normalization layers with adjacent linear layers, and use knowledge distillation. It has been shown that the acceleration of 1.27x can be obtained on CPUs using the aforementioned four techniques while achieving a comparable accuracy performance w.r.t. the baseline model.", "summary_of_the_review": "In general, I believe the contribution of this paper is rather limited since the main method is a combination of previous works. On the other hand, the proposed method is simple and effective (when running on CPUs). My main concern is the lack of comparison with other works. It is not clear where this works stands w.r.t. prior works that also tries to reduce the complexity of BERT models such as Linformer and I-BERT. There is also no evaluation on GPUs while it has been mentioned as one of the motivation of this work in the introduction. \n", "strength_and_weaknesses": "Strengths: \n\n-- I believe this work is well-motivated. The paper targets a very important issue (i.e., reducing latency of inference) associated with large models such as BERT. \n\n-- The detection of the source of the latency overhead during inference is insightful as shown in Fig. 1. \n\n-- The amount of speedup during the inference on CPUs is interesting and shows the effectiveness of the proposed techniques.\n\n-- The paper in general is well-written and easy to understand.\n\nWeaknesses:\n-- In general, the contribution of this paper is rather limited since each of those techniques have been previously-proposed.\n\n-- It would have been great if the breakdown of cycles in Fig. 1 was shown for the sequence length of 128 to be compatible with experimental results in Table 2.\n\n-- My major concern about this paper is the lack of comparison with prior works. First, there is no results for inference on GPUs. Second, there is no direct comparison with prior works such as Linformer? Where does the proposed method stand w.r.t. prior works? Finally, what speedup can be obtained when using MA-DistilBERT or MA-DistilRoBERTa?\n\n-- The GELU and softmax functions can be approximated using polynomials similar to I-BERT with no impact on accuracy.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WjsrOg8MdG", "reviewer": "Reviewer_K662", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces 3 approximations to remove expensive non-linearities in transformers using cheaper matrix math and ReLU activation functions. Next, they perform a knowledge transfer between the trained BERT model and the approximated model to bring the accuracy up to standard in MA-BERT. The results show comparable accuracy while being cheaper to implement MA-BERT in hardware.", "summary_of_the_review": "I enjoyed the premise of the paper and Figure 1, which shows how much hardware cycles are spent in the non-linear computations. The techniques seem simple and easy to understand. However, it is unclear why they would work. The authors should explain why softmax can be approximated using a 2-layer FFN with ReLU. It is not clear to me. Citing universal approx. theory is not enough. For inspiration, look at performer or linformer papers.\n\nI did not see ablations on the techniques suggested in the paper. Since, these approximations are not grounded in theory, I expect more ablations to show which technique mattered more and also why. \n\nI am not an expert in this area, however has a working knowledge base in it. The paper is simple, but lacks justifications for the choices made and hence my score.", "strength_and_weaknesses": "Strengths:\n\n* The techniques are simple to understand\n\nWeaknesses\n\n* The reasons as to why you can approximate the softmax using a 2-layer NN is unclear. For example, performer had clear justification why the projection was approximating the softmax. The argument in this paper is poor (universal approx. theory)\n* No hardware results, but just software CPU implementation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-w8QY7fHYBG", "reviewer": "Reviewer_RFc3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Transformers are complex and require several compute-intensive operations. This paper proposes to simplify the execution of the BERT inference by using matrix arithmetic-only operations. The results show that the proposed method achieves considerable inference time reduction with relatively low accuracy loss compared to the baseline.", "summary_of_the_review": "Borderline paper where several concerns should be clarified.", "strength_and_weaknesses": "Strengths:\n1. The tackled problem is relevant to the ICLR community.\n2. The contributions look solid.\n\nWeaknesses:\n1. It is recommended to evaluate the proposed method executed on AI accelerators, rather than only on CPUs.\n2. The reason why the knowledge transfer is needed (instead of training from scratch the MA-BERT) is unclear. Please provide more details on it.\n3. The results do not seem to provide a significant improvement compared to the related works.\n4. It would be useful to provide the source code for reviewers' inspection during the rebuttal.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster