Efficient SoftMax Approximation for Deep Neural Networks with Attention Mechanism
Authors: Vasyltsov Ihor
Wooseok Chang
Venue: NeurIPS 2021 Submitted (2021)
Keywords: DNN
Attention Mechanism
Softmax
Approximation
Quantization
Inference
Edge Devices

Abstract: There has been a rapid advance of custom hardware (HW) for accelerating the inference speed of deep neural networks (DNNs). 
Previously, the Softmax layer was not main concern of DNN accelerating HW, because its portion is relatively small in multi-layer perceptron or convolutional neural networks. However, as the attention mechanisms are widely used in various modern DNNs, a cost-efficient implementation of Softmax layer is becoming very important. In this paper, we propose two methods to approximate softmax computation, which are based on the usage of LookUp Tables (LUTs). The required size of LUT is quite small (about 700 Bytes) because ranges of numerators and denominators of softmax are stable if normalization is applied to the input (i.e. logits).
We have validated the proposed technique over different AI tasks (object detection, machine translation, speech recognition, semantic equivalence) and DNN models (DETR, Transformer, BERT) by  variety of benchmarks (COCO17, WMT14, WMT17, GLUE). 
We showed that 8-bit approximation allows to obtain acceptable accuracy loss ($<1.0\%$).

Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism
Anonymous Author(s)
Affiliation
Address
email
Abstract
There has been rapid advance of custom hardware (HW) for accelerating the inference speed of deep neural networks (DNNs). Previously, the softmax layer not main concern of DNN accelerating HW. because its portion is relatively small in multi-layer perceptron or convolutional neural networks. However, as the attention mechanisms implementation of softmax layer is becoming very important. In this paper, we propose two methods approximate softmax computation, which the usage of LookUp Tables (LUTs). The required size of LUT is quite small (about 700 Bytes) because ranges of numerators and denominators of softmax are stable if normalization is applied to the input. We have validated the proposed technique over different AI lasks (object detection, machine translation, speech recogmition, semantic equivalence) and DNN models (DETR, Transformer, BERT) by a variety of benchmarks (COCO17, WMT14, WMT17, GLUE). We showed that 8-bit approximation allows to obtain acceptable accuracy loss below 1.0%.
2
4
widely used in various modern DNNs, cost-efficient
6
based
10
11
12
13
14
Introduction
15
16
After Vaswani al. had introduced Transformer model in [27] for machine translation task, the attention based architecture became popular firstly in Natural Language Processing (NLP) appli- cations, e.g.: speech recognition [16]. [21]. [9]; summarization [7]; language understanding [5],
17
18
19
[33]. [15], [18]; and vidco captioning [3]. Recently attention-basse models are used in even wider practical areas including Computer Vision (CV) tasks for object detection [2]; image transformation [26]; image classification [6]; and even symbolic integration and solving differential equations [14].
20
21
22 Despite the attractiveness of transformer-based models, its direct implementation into the platform
23 with constrained computational power (e.g., mobile SoC, edge devices) is very challenging due to
24 big memory footprint and latency.
25 Therefore, model compression techniques such as quantization, and distillation are needed for those
26 models. Many approaches have been introduced on quantizing matrix multiplication of transformer architccture. For example, in [22] it was used second order Hcssian information, what allows to
27
28
significantly compress the size of the model up to 13x times, while maintaining at most 2.3% of performance degradation (for the case of ultra-low precision 2-bits quantization). In [35] it was
30 used a quantization-awee - training during the fine-tuning phase of BERT, what allows to compress
31
BERT model by 4x (with 8-bit quantization) with minimal accuracy loss (less than 1 1%). In [1], machine language translation model was quantized by 8-bit, while maintaining less than 0.5% drop
32
RRecently, interest to the computations performed close to the data sources is growing up aiming to soften the requirements of continuous access to high-speed and high-bandwidth connections. Moreover, often customers wanted keep their security and privacy, and thus do want expose their data the external clouds [32], [19], [36], [11].
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

33 in accuracy. Moreover, in [20] it was shown that 8-bit quantized models provide the same even
34 higher accuracy as the full-precision models. Most of the above methods consider the quantization of matrix multiplications operations only. However, as it is shown in [4], [24] in modern DNNs with 36 attention mechanism (e.g., Transformer, BERT, GPT-x) the softmax function is also used intensively, especially at the longer sequence lengths, so it is necessary optimize its for performance.
38 In this paper we propose methods for efficient computation of the softmax layer at the HW accelerator. The method is based on seere--wis--oosstae approximation and usage of LUTss. To the best of our knowledge,
39
40
the first paper where softmax quantization of the models with attention mechanism is tested and verified on variety of AI tasks. In Section 2 we show why our research is important and
41
42 valuable. Section 3 we consider the drawbacks of existed softmax approximation methods in the
43 perspective of HW accelerator, and summarize the differentiation of our methods from the previous
44 arts In Section we describe the details of the proposed methods. Section 5 shows the experimental validation over different models and datasets, and Section 6 concludes the paper.
2
Background and motivation
47 Modern GPUs are powerful, but big, expensive, and power-hungry. Therefore, alternative HW accelerators (e.g.. NPU) for on-device inference are under active development by different vendors, especially for Federated Learning and Edge computing. However, such devices mostly are focused on the acceleration of matrix multiplication operations, and do not include means to compute complex activation functions. Typically, in such devices the data is sent outside of the accelcrator to compute activations host CPU. For example, according to the guidelines of Coral (TM), softmax layer of DNN model in Edge TPU have to be run on host CPU 2, what is acceptable for traditional CV tasks (which are typically uni-directional, have minimum dependencies, and softmax layer is located at the end of the computational graph of DNN model), however is very inefficient for NLP tasks (which are typically more complicated with a lot of dependencies and active employment of softmax layer in the middle of DNN model). In opposite to traditional logic-centric approach, some researches are trying
49
51
53
54
55
56
57
58 to perform computation closer to the memory (so called eeemory-centri approach). For example
59 in [23], there is shown a DRAM-based AI accelerator. This approach allows significantly speed-up the overall computation process, but for the computation of the activations the data should also be moved to host processor, what is an even bigger issue in the DRAM environment.
60
61
The Eq. (1) from [27] describes how attention is computed in the model. This particular form, named "scaled dot-product attention" takes the matrix multiplication product of queries and keys 64 of N x x as input for the softmax layer where N means number of heads, L means sequence length and H/ means hidden size for the case where batch size equals to In other words, performing
63
85
66 (N x L x L) softmax operations is required per one attention. Furthermore, encoder in typical
67 transformer consists of six multi-head attentions which means x (N x L L) operation is required
88 for encoder solely. Assuming the number of heads is 8 and sequence length is 128, it already takes
69 786, 432 operations for softmax of the transformer encoder. This overhead increases as number of
70 heads and sequence length increases which is typical case for nigh-performing models.
QKT Vdx
Attention Q, K, V) Softmar
V
(1)
71. For example, it requires performing 12 x (12 x 128 x 128) 2, 359, 296 softmax operations for
72. one sample inference for typical BERT configuration [5] over sequence of length 128. In the case
73 when HW accelerator is used for matrix multiplication only, and activation to be computed at the
74 CPU (what is common case for HW accelerators, optimized for CNN-models), the huge amount of data must be moved between CPU and the accelerator. Such data movement negatively impacts on
76 the overall computation time and power consumption, which can be critical for on-device inference. Therefore, HW accelerator must be able to compute softmax layer without CPU involvement.
3 Related work and key contributions
79 The common equation to compute softmax function over the input x a fraction as shown below:
2

---

E
e ((--mar(xx)
(Z;)
(2)
Ee
Ees (m=l-)a((a))
80
There are different ways to implement it. For example, some approaches straightforwardly compute the numerator and denominator firstly, and then a division operation is performed. In such case the HW accelerator should contain divider, what requires additional HW costs and can also cause performance degradation, if divider is not fully pipe-lined.
81
83
84
In [25] it is proposed to use basic-split calculation method, which allows to split the exponentiation calculation of the softmax into several specific basics which are implemented by LUT (ROM). It allows to simplify the complexity of hardware and signal propagation delay. However, to recover the whole computed value of exponent some additional multiplications are needed. Moreover, to obtain the final value of softmax the division is still used. In [30] it is proposed to add threshold layers to accelerate the training speed and replace the Euler's base value with a dynamic base value to improve the network accuracy. Such approach allowed to save up to 15% of training model convergence time and also increase by 3 to 5% the average accuracy. But during the computation of softmax the divider is still used. In [17] the combination of LUT and multi-segment polynomial
e5
86
87
88
89
91
98 fitting have been used to compute exponential operations of integer and fractional parts in separate. In addition, they adopt radix-4
Wooth-eallace based multiplier for computing the whole value 95 of exponent, and modified shift-compare divider for computation of the final value of softmax.
96 To avoid big area costs for traditional divider, the authors in [8] propose to reduce the operand bit-width, and approximate exponential and division operations with cost-effective addition and bit shifts operations. In their design they have approximated the division operation in Eq. (2) by replacing the denominator with closest 2$ value, where b is some integer constant. Then division implemented just as simple bit shifts operation. In [24] ex + 2", then all computations are more gardware-friendll however the division operation is still 101
97
98
89
100
proposed to replace the basc as
102
required. Also, to restore accuracy a fine-tuning of the model is needed, what is not applicable for post-training quantization paradigm. Although the methods described above are decreasing the hardware complexity of softmax computation they all still rely on the division operation.
103
104
105
To avoid division operation at all, some other solutions apply the logarithmic transformation to the original softmax function, and thus substitute costly division operation by subtraction of the logarithm. In [34], for example, it is proposed to use a logarithmic operation implemented as a LUT and a subtractor to replace the division operation, what allows further decrease the complexity of hardware, as well critical path of the whole design. In [10] simplified version of Integral Stochastic Computation is used in order to build FSM-based exponentiation. Division operation is substituted by LUT-baaed logarithmic operation and subtraction, similarly to [34]. In [31] the authors are further developing the method proposed in [34], by applying mathematical transformations and linear fitting. After optimization, their final design includes only shift operations, leading one detector, and adders. Finally, there are some extreme approximation cases represented in [13] and [28], where logarithmic computation and subtraction are skipped at all.
108
107
108
109
110
112
113
114
115
116
Despite its attractiveness, logarithmic transformation approach can be used only in the cases when softmax layer is the last layer in DNN and its functionality is simply "scoring" among the candidates for classification tasks. However, if softmax layer is used inside of computational graph of DNN (e.g., then error caused by quantizations will be accumulated drastically,
117
118
119
DNNs with aitention-mechanii directly impacting on the final accuracy. For example, in Table there is shown the averaged accuracy drop for DETR models caused by a softmax approximation in uint8 precision by some prior arts As it can be seen from the Table 1, a straightforward usage of (2) from [31] causes big accuracy drop, and even after applying some improvements to the original method (shown case (2)+)), the accuracy drop is still high (2% to 19%). For more details of the prior arts experiments please refer to Appendix A. However, if for the same conditions we use the method proposed in Section 4.1, we can see that accuracy drop reduced by x4 to x20 times, and it is below 0.5% for plain DETR models (no DC5 dilation at the last stage).
120
122
124
125
126
127
128
The work presented in this paper has focused on the development of methods for efficient computation of softmax layer during the inference at the edge devices, what usually have limited computational
129
130
power and suffer from constraints of the bandwidth.
131
Previous works for HW accelerator of softmax layer are focused on the logic-centric approach and used dedicated hardware for its implementation. In such case the utilization of hardware low,
132
3

---

Table 1: Averaged accuracy drop by different methods over DETR models (Average Precision), %
METHOD
DETR (R50)
DETR+DC5(RSO)
DETR (R101)
DETR+DCS(R101)
(2) IN [31]
7.20
19.30
10.25
25.37
(2)(2)+ [31]
2.50
12.93
5.38
18.85
SECTION 4.1
0.33
2.92
0.22
2.73
133
performance can be slower, and no reconfigurability provided. In our paper we have used an alternative memory-centric approximate computing approach. It keeps accuracy loss small, while allows computing softmax operation with no divider. The size of the required memory (i.e., LUT) reasonably small and can be reconfigured on demand.
135
137
The methods proposed in the paper contribute to building the alternative concept of hardware architecture to accelerate essential operations for AI applications, especially for on-device inference. To summarize, we have three-fold difference from the previous works:
138
139
Applicability of our methods to DNN with attention mechanism is experimentally proven over variety of the models for different Al applications. All previous methods were used only for the cases when softmax is the last layer in DNN, and is used for "scoring".
141
No divider is needed to fully implement the method. Moreover, for 2D LUT method even multiplier is not needed. Thus, hardware overhead is minimal, and is almost free if used the DRAM-based AI accelerator.
144
Our solutions utilize integer precision, what makes it compatible with traditional HW accelerators used for matrix multiplication, and simplify the inlegration of methods into full system (all prior methods are based on fixed point precision).
147
148
Proposed methods
149
150
In this paper we use memory-centric approach to build the accelerator for softmax computation in hardware platform with limited resources. We propose two LUT-bbasd methods for efficient computation, which provide high performance and do not require a divider. The details of the methods are described below and appropriate software models are shown in Appendix A.2.
151
152
4.1 Normalization of reciprocal exponentiation
In this subsection we consider the method, which is based on the normalization of reciprocal 155
156 exponentiation, and hereafter we call it REXP for short.
157 The original reciprocal exponentiation method was proposed in [28], where they used inverse way of mux-noomalization and the reciprocal of exponential function below:
(x:)
(ma
-x
159 And thus, the final value of softmax can obtained by reading from simple LUT-table. Content LUT is computed shown below:
LUTi/el0 (2 - -1) vi 0,1, By +
(4)
where w is number of bits for quantization, and xy = [2ma 1)1 is an efficient quantization 182 boundary.
181
163 In addition to very low computational complexity, this method has other desired properties [28]:
is positive ( > V4 (-o0, +o0));
184

---

165
bounded and stable (maz(s) E (0, 1):;
186
and nonlinear
200a #
167
But due to its aggressive approximation nature, it can be applied only to simple CV tasks, and if used for iitention-basee DNN models causes the explosion of accuracy drop (see Appendix A. for details). Thus, in this paper we further develop that method to be applicable for wider class of DNN models.
168
170
171
During our initial investigations, we have noticed that method described in Eq.( 3) is just scaled version of real softmax. So. we proposed to normalize with some probability density function (PDF) scale, such that b PDF = 1. However, if used srriigh-forwardlly. it would need to involve a division operation, what is strongly un-desirable for devices with constrained computational power. Therefore, instead of dividing, we propose to substitute division by multiplication with some PDF normalizing constant as below:
172
174
176
0 (x.))
(x.)) = PDFnorm - o" (x:) (xx)).0 a
(5)
where e-In(Io(xx)) is PDF normalizing constant.
178
Then final equation compute softmax approximation by proposed REXP method is shown below:
or (2,, e (xx))
In("ia" (x:))
(6)
emar(())-
maz(=))zz
179
Thus, to compute the softmax value it requires just two LUTs considerably small size, where 180 content of the first LUT is computed accordingly to (4,(4), and the second LUT values can be
181 computed as below:
LUTabi 1 (2 [1]- 0,
(7)
182
where j = Eo (x:),, v, is selected quantization boundary, and LUT, [x:s] =
185
4.2
2-Dimensional LUT
In this subsection we propose another method which is based on the substitution of division operation in Eq. 2) by 2-Dimensional (2D) LUT to speed-up and simplify the computation, while maintaining accuracy even for attention-based DNN models. Hereafter will refer to this method as 2D LUT.
185
187
188
For this purpose, we have started with the estimation of distributions of e and Σe terms for typical inference runs. Our investigation showed that if max-based normalization is applied to the input values (i.e., - (x max(x))) the distribution of ex is stable within range ex E (0, 1] regardless of input values, and range of Ee term depends on the length of the input æ. Thus, allows us to have stable computation even within small size of LUT.
189
190
191
193
Generic architecture and concept of the proposed method for efficient softmax implementation as 2D LUT is shown in Figure 1. There are two LUTs used: ID LUT for approximation of e values, and 2D LUT for storing softmax output values dependent on the values of numerator e" (used as the st index in the table), and denominator Σe" (used as the 2-nd index) of E2. As it can be seen from Figure (right), to calculate the indexes for corresponded value 2D LUT table, only most-significant bits (MSB) are needed. Thus, the simplest hardware realization can be done within wiring only (when MSB bits are directly connected to the appropriate address selectors) 3. Also, the proposed method can be easily modified to the case where, 1-st index of 2D LUT table is calculated not from ex but directly from imput x. In such case there is no need to store intermediate values of e..
194
195
196
197
199
201
202
While the content of ID LUT for approximation of e values is straightforward, 2D LUT contains the family of linear approximations where each row contains the softmax output scaled according
203
Other hardware realization are also possible, but not considered here for simplicity of the explanation.
5

---

LUT
index
Ignare
exp1
exp2
exp3
exp1
exp2
exp3
exp
o(x,)
Σ
① Accumuletor E,e
Σ;
LUT
LUT
2 inous
MBB
Em
frrun
Figure 1: Generic concept of the proposed 2D LUT method (left). Reading softmax output value from pre-computed 2D LUT(right). Computational flow consists from two steps: a) obtaining of e values by reading from ID LUT and accumulation of Ee" term, b) obtaining (v;) values by reading from 2D LUT.
Ee shown in (8). The indexes of LUT are computed according (9) and Eq. means the number of bits for the value in selected precision.
205
i scalees
LUT,
(2 -
(8)
J scales
where
max(E*)
0,....
(9)
scale
maax(E" (Ee²)
9
(10)
scaley
207 Since x + (a - max(x)) normalization was used, so mar(e") = 1.0. Therefore, scale,= factor allows to define the number of columns in LUT to make it small enough for practical applications. In our experiment we have selected scale.= = 0.1 for all precisions, what allows us to reduce the size of LUT significantly (i.c., i 0, ..., 10 for all versions of LUT.). The value of mar(Ee") depends on the distribution of input values. Our experiments showed that man (Ee") = 60 is big enough for the tested NLP applications. We also selected scaley 1.0 for simplicity of the computations. Thus, finally, those parameters give us LUT, of typical size 11 x 60.
209
211
213
5 Experimental validation
215
To validate the proposed methods and check how well they generalize we have conducted several experiments with different models (DETR, Transformer, and BERT) for different applications (object detection, machine translation, sentiment analysis, and semantic equivalence) over variety of datasets.
217
In all those experiments have used available pre-trained models, where applied dynamic post- training quantization (hereafter we referred to quantized models as PTQ-D) 4. Then we substituted a conventional softmax layer quantized models with the LUT-based computation deseribed in
219
220
Section 4. We did not consider any retraining or fine-tuning of the models after quantization. and
222
the same off-line generated LUTs were used among all models. Our code allows select LUTs with different precision from 16 down to uint2, what allows to analyze the sensitivity of the model
223
224
to softmax approximation even for ultra-low 2-bits quantization. The details of experiments are described below, and results are summarized in Figure 2, Figure 3, and Table 2. For more details
See more details Appendix A.3.

---

please refer to Table 6, and Table 7 in Appendix. As it can be seen from figures, proposed LUT-bbaed softmax computation methods maintain accuracy drop below 1.0% down 8-bit quantization for all NLP and DETR (no DC5) models.
227
5.1
Object detection
230
For our first experiments we have used DEtection TRansofmer (DETR) models for object detection [2], with available pre-trained models 5. As it can be seen from Table 6, we were able to reproduce the same results for original FP32 reference model over COCO dataset. We have used the same IoU metric by Average Precision (AP) as in Table 1 in [2]. Then we run bunch of experiments to check how accuracy of object detection will decreased due PTQ-D quantization and LUT-bbased approximation as proposed in REXP method (see Section 4.1). Table 5 in Appendix shows the LUTs size for several pre-selected cases in int16 and uint8 precision. There are three cases selected which different in the size of LUT.: it is X 256 for case 1, x 320 for case 2, and 1 512 for case 3.
232
234
236
238
Analysis of Figure shows that accuracy drop caused by application of softmax approximation
small (< 1%) and acceptable for plain DETR models (no DC5 used). Bigger accuracy drop for +DC5 cases caused by the bigger size of self-attentions of the encoder (see details Section 5.3). We
240
expect that increasing size of LUTss will help to solve this issue. The behavior of average recall values similar average precision values.
242
DETRI RS0 (int16)
DETR, R50 lint16)
DETRI DCS R5D 6n116)
DETR, DCS R5O 6m115)
R10L (in116)
(iH116)
Dr0o,
DETRI RSO (uink))
DETR DCS R50 (wint5)
DETR, DCS R50 (Uint))
DETR, R50 (uints)
R101 (uint8)
Actuacy
8101 (uint))
DETR R101 (uinta)
101-
RIOL (wint))
Figure 2: DETR averaged accuracy drop of PTQ-D models with softmax approximations vs. original FP32 models: average precision (left) and average recall (right). As it can be seen from the figure, for DETR models without dilation at the last stage (no DC5) the accuracy drop for all cases is below 1% and shows very similar behavior.
5.2
NLP tasks
244
Next, we have validated the proposed methods by experimenting with several NLP tasks. Similarly, to DETR case, Table 8 in Appendix shows the LUTs size for several pre-seleeted cases for those experiments. In Table 2 below there are accumulated values from the experiments with NLP models. The bold values in the table shows highest values per model per method after applying quantization and softmax approximation. As it follows from the analysis of experiment results, about 700 Bytes for 2D LUT method, and up 50 Bytes for REXP method would be enough for practical applications.
246
248
250
5.2.1
Machine Translation
251
Among NLP tasks we have started with machine translation. For our experiments we have used transofmer-hase model for En-Ge translation [12] from OpenNMT library, with available pre-trained model 6 configured to replicate the results from original paper. To avoid dependency of the evaluation results on the selected tokenization scheme, we have used spm_decode to detokenize the output of translation, and then applied multi bleu,per perl script 8 to calculate BLEU score.
253
255
http:///iital.ccmmaaccoooreeeele "naps://ppemmnt.ee//ooobbb--/// http:://ggituuubcoommoosssssss/s

---

Table 2: Experimental validation different NLP models and datasets
PRECISION
TRANSFORMER
BERT
2D LUT
REXP
2D LUT
REXP
WMT
WMT
WMT
WMT
SST-2
MRPC
SST-2
MRPC
2014
2017
2014
2017
(BLEU)
(BLEU)
(BLEU)
(BLEU)
(%)
(F1)
(%)
(FI)
FP32
26.98
28.09
26.98
28.09
92.32
90.19
92.32
90.19
PTQ-D
26.86
27.95
26.86
27.95
91.74
89.53
91.74
89.53
INT16
26.87
28.02
26.89
27.64
91.63
89.50
91.74
89.26
UINT8
26.76
27.9
26.8
27.66
91.63
89.35
91.17
89.34
UINT4
26.26
27.43
26.68
28.02
91.40
88.01
91.17
88.77
UINT2
24.42
25.06
25.29
25.86
89.22
56.67
91.63
86.12
256
Thus, as can be seen from Table 2, we were able to reproduce the same BLEU score for FP32 several experiments to check how accuracy of
reference model as original model. Then quantization in different precisions, and we can the translation will be changed due to LUT--based
258
confirm that down up 8-bit quantization the deviation of BLEU score from reference is small for both datasets (< 0.5%). Also, if we consider impact of the proposed methods only, then we can
260
see that accuracy drop much smaller, and sometimes even recovers vs. PTQ-D quantization (see
262
Figure (right)).
WMTT14
WMTT44
WMILL, REXP
WMTT14, REXP
WMT17 REXP
WMTT17 REXP
Drop
SST2.20LUT
Accunary Drop.
MAPC_ZOLUT
SST2.2OLUT
55T2. REXP
55T2. REXP
NRPC, REXP
NRPC_REXP
PTO-0
int
sints
unt2
PTO-D
I1t18
uints
unnt
unt2
Precitoon
Precision
Figure 3: Accuracy drop for NIP experiments: PTQ-D models softmax approximations vs. FP32 models (left), and PTQ-D models + softmax approximations vs. plain PTQ-D models (right). As can be seen from the figure, down to uint8 precision the accuracy drop for all cases is below 1% and shows very similar behavior. This confirm very good generalization of the proposed method over different models and applications.
263
5.2.2
Sentiment analysis
264
To extend the variety of NLP applications, we also tested the same LUTs with BERT model [5]. We have used sentiment analysis task from GLUE benchmark [29] to test the model. We have used auggingface library 9, and trained the model with the syper-parameter, described 10, The results of experiments showed, that similarly machine translation, the impact of proposed method (sofumax layer approximation by LUTTs) is smaller vs. accuracy drop caused by PTQQD quantization (see Figure 3).
286
268
289
270
5.2.3
Semantic equivalence
For semantic equivalence test we used The Microsoft Research Paraphrase Corpus (MRPC) 11 in GLUE benchmark. As the classes imbalanced (68% positive, 32% negative), follow the
271
272

---

common practice and used Fl score metric. We have used huggingface library and followed the guidelines from PyTorch tutorial 12 obtain PTQ-D quantized model. Then, similarly to previous lests we have substituted a conventional softmax layer with the proposed LUT-based methods. The results of our experiments showed the similar trend with sentiment analysis test.
274
276
5.3
Ablation study of DETR models experiment
278
As stated [2], to increase the feature resolution for small objects, dilation to the last stage the backbone was added (+DC5 cases of DETR models). This modification increases the cost in the self-attentions of the encoder, leading to an overall x2 increase in computational cost. Such changes also reflect on the properties of softmax factors. In Figure 4 there are shown the histogram of De values distributions for the first 200 tensors of DETR model run for bins - 50, range - (0, 500).
280
282
As it can be seen from the figure, the distribution of DETR+DCS (R50) variant more right-tailed, due to the bigger number high-magnitude values. This causes the bigger accuracy drop when LUT-based quantization method is used, due to the lack of the discrepancy for those values. Thus, for such models (DETR with added dilation at the last stage) the accuracy of object detection after from Figure increasing of the size of LUT,, from 256 Bytes to 512 Bytes allows to decrease the accuracy drop from 9% to
284
286
287
application of the proposed method can be limited. However, as we
288
289
3% for DETR+DC5 (R101) unit8 case. Thus, we expect that further increasing the size LUTs will
290
help to obtain even more accurate results for DETR models with dilation.
1
10
30
10-4
10a4
0 10 20 30 40
50
30 20 30 40
50
Figure 4: Histogram of Ee values distributions for DETR model variants: plain DETR (R50) (left), and with dilation DETR+DC5 (R50) (right). Red dot line represents the average of the all values per one run of the inference DETR model. It is clearly seen from the figure that distribution of DETR+DC5 (R50) variant is more flat, having more high-magnitude values. This causes the bigger accuracy drop for the quantized model due lack of the discrepancy for those values.
Conclusion
291
292
In this paper two alternative methods for efficient softmax computation for DNN models with attention mechanism proposed. The methods memory-centric in contrast known logic- centric approach and are based on the usage LUTs for reading of the pre-computed values, instead of the direct computation. Thus, allows to build the HW accelerator without usage of costly and power-hungry divider. In turn, it allows to decrease the power consumption and latency of the whole inference, what is crucial for edge computing. All results obtained in the paper over different AI tasks (object detection, machine translation, sentiment analysis, and semantic cquivalence) and models (DETR, Transformer, BERT) by variety of benchmarks (COCO2017, WMT14, WMT17, GLUE), showing acceptable accuracy and good generalization of the proposed methods.
294
296
validated
298
289
300
http::///iyocchhagnmmmmmiimmemir httpaiiic_aaaattttiinnoaae bettetttttttatllllllllaaaaaalaal tutorial.htm

---

References
303
[1] Aishwarya Bhandare, Vamsi Sripathi, Deepthi Karkada, Vivek Menon, Sun Choi, Kushal Datta, and Vikram Salctore. Efficient 8-bit quantization of transformer neural machine language translation model. CoRR, htt//906.00555 2019.
305
306
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, 2020.
3
Ming Chen, Yingming Li, Zhongfei Zhang, and Siyu Huang. Tvt: Two-view transformer network for video caplioning. In Jun Zhu and Ichiro Takeuchi, editors, Proceedings of The I 10th Asian Conference on Machine Learning, volume 95 of Proceedings of Machine Learning Research, pages 847-862. PMLR, 14-16 Nov 2018.
309
310
311
312 [4] Jacek Czaja, Michal Gallus, Tomasz Patejko, and Jian Tang. Softmax optimizations for intel xcon processor-based platforms. CoRR, abs/1904. 12380, 2019.
314
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR,
315
2018,
316
[6]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 1616 words: Transformers for image recognition at scale, 2020.
317
318
[7] Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4009, 2018
321
[8] Xue Geng, Jie Lin, Bin Zhao, Anmin Kong, Mohamed M. Sabry Aly, and Vijay Chandrasekhar. Hardware-awart softmax approximation for deep neural networks. In C.V. Jawahar, Hongdong Li, Greg Mori, and Konrad Schindler, editors, Computer Vision - ACCVV 2018, pages 107-122, Cham, 2019. Springer International Publishing.
324
326
327
99] Yanzhang He, Tara N. Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, Qiao Liang, Deepti Bhatia, Yuan Shangguan, Bo Li, Golan Pundak, Khe Chai Sim, Tom Bagby, Shuo yüin Chang, Kanishka Rao, and Alexander Gruenstein. Streaming end-to-end speech recognition for mobile devices, 2018
329
[10] R. Hu, B. Tian, S. Yin, and S. Wei. Efficient hardware architecture of softmax layer in deep neural network. In 2018 IEEE 23rd International Conference on Digital Signal Processing (DSP), pages 1-5, Nov 2018
332
[1I] Andrey Ignatov, Radu Timofie, William Chou, Ke Wang, Max Wu, Tim Hartley and Luc Van Gool. AI benchmark: Running deep neural networks on android smartphones. CoRR, abs/1810.01109, 2018.
335
[12] Guillaume Klein, Yoon Kim, Yuntian Deng. Jean Senellart, and Alexander M. Rush. OpenNMT: Open-source toolkit for neural machine translation. In Proc. ACL, 2017.
338
339
[13] Ioannis Kouretas and Vassilis Paliouras. Hardware implementation of a softmax-like function for deep learning. Technologies, 8(3), 2020.
[14] Guillaume Lample and François Charton. Deep learning for symbolic mathematics. CoRR, abs/1912.01412, 2019.
342
343
[15] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations, 2019.
344
345
[16] Bo Li, Shuo yiin Chang. Tara N. Sainath, Ruoming Pang. Yanzhang Hc, Trevor Strohman, and Yonghui Wu. Towards fast and accurate streaming end-to-end ast, 2020.
346
347
[17] Z. Li, H. Li, X. Jiang, B. Chen, Y. Zhang, and G. Du. Efficient fpga implementation of softmax function for dnn applications. In 2018 12th IEEE International Conference on Anti- counterfeiting, Security and Identification (ASID). pages 212 216, Nov 2018.
348
349
10

---

[18) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 1699,, 2019.
351
355
[19] M. G. Sarwar Murshed, Christopher Murphy, Daqing Hou, Nazar Khan, Ganesh Anantha- narayanan, and Faraz Hussain. Machine learning at the network edge: A survey, 2019.
355
[0)] Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. Fully quantized transformer for improved translation, 2019.
357
[21]
Tara N. Sainath, Yanzhang He, Bo Li, Arun Narayanan, Ruoming Pang, Antoine Bruguier, Shuo yiin Chang, Wei Li, Raziel Alvarez, Zhifeng Chen, Chung-Cheng Chiu, David Garcia, Alex Gruenstein, Ke Hu, Minho Jin, Anjuli Kannan, Qiao Liang, Ian McGraw, Cal Peyser, Rohit Prabhavalkar, Golan Pundak, David Rybach, Yuan Shangguan, Yash Sheth, Trevor Strohman, Mirko Visontai, Yonghui Wu, Yu Zhang, and Ding Zhao. A streaming on-device end-to-end model surpassing server-side conventional model quality and latency, 2020.
359
381
363
[22] Sheng Shen, Zhen Dong, Jiayu Yc, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert, 2019.
365
[23] Hyunsung Shin, Dongyoung Kim, Eunhyeok Park, Sungho Park, Yongsik Park, and Sungjoo Yoo. Medram: Low latency and energy-efficien matrix computations in dram. IEEE Trans- actions on Computer-Aided Design of Integrated Circuits and Systems, 3711)22613-2622, 2018.
367
368
389
[24] Jacob R. Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany, and Anand Raghu- nathan. Softermax: Hardware/software co-design of an efficient softmax for transformers. CoRR, abs/2103.095 09300,,,9911.. 2021.
371
372
373
[25] Q Sun, Z. Di, Z. Lv, F. Song, Q. Xiang, Q. Feng, Y. Fan, X. Yu, and W. Wang. A high speed softmax visi architecture based on basic-split. In 2018 14th IEEE International Conference on Solid-State and Integrated Circuit Technology (ICSICT), pages 1-3, Oct 2018.
374
375
376
[66] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention, 2021.
377
378
[27]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, Abs/1176.03762, 2017.
379
380
[88) Thor Vasyltsov and Wooseok Chang. Lightweight Approximation of Softmax Layer for On-Device Inference. Springer International Publishing, 2021
381
382
[29] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, Abss/804.004611 2018.
383
384
385
[30] K. Wang, Y. Huang, Y. Ho, and W. Fang. A customized convolutional neural network design using improved softmax layer for real-time human emotion recognition. In 2019 IEEE Inter- national Conference on Artificial Intelligence Circuits and Systems (AICAS), pages 102-106, March 2019.
386
387
388
389
[31] M. Wang, S. Lu, D. Zhu, J. Lin, and Z. Wang. A high-speed and low-complexity architecture for sofımax function in deep learning. In 2018 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS), pages 223-226, Oct 2018.
390
391
392
[32] Siqi Wang, Anuj Pathania, and Tulika Mitra. Neural network inference on mobile socs. IEEE Design Test, page 1-1, 2020.
393
394
[33] Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe V. Lc. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1 01211966002333 2019.
395
396
11

---

[34]
B. Yuan. Efficient hardware architecture of softmax layer in deep neural network. In 2016 29th
396
IEEE International System-on-Chij Conference (SOCC), pages 323-326, Sep. 2016.
[35]
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8hit bert, 2019.
400
401
[36] Xingzhou Zhang, Yifan Wang. Sidi Lu, Liangkai Liu, Lanyu Xu, and Weisong Shi. Openci: An open framework for edge intelligence. CoRR, abs/1906 01864, 2019.
402
Checklist
403
404
1. For all authors...
405
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See also Section 3, where key contributions were listed.
407
(b) Did you describe the limitations of your work? [Yes] See Section 5.3.
408
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
409
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We read the ethics review guidelines and confirm that content of our paper is not related to any ethics issue, since it is focused on the optimization of computations.
410
412
2. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
415
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See supplemen- tal materials.
417
418
419
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A] In our paper we do not consider any training, or fine-tuning. We focus on the inference only.
420
421
422
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A] Since we did not do any training, we did not need to run experiment multiple times (inference output is deterministic).
423
424
425
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] Since we did not do any training, is not related to our work.
426
427
428
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
429
(a) If your work uses existing assets, did you cite the creators? [Yes] All used assets are cited either as a reference, or by direct URL link in the footnote, or in the body of the paper.
431
432
(b) Did you mention the license of the assets? [Yes] The license of the used assets are noticed either in the appropriate reference, or direct URL link of asset.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] Some code and generated LUTs to reproduce our results are provided in supplemental materials.
435
437
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating' [N/A]
438
439
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
440
441
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
443
444
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
447
12


Reviews:
[{"review_id": "aLaal6Olq8L", "reviewer": "Reviewer_BDef", "summary": "The authors propose an efficient computation of softmax targeting transformer models running on a specialized hardware accelerator. Softmax is a key component of the self-attention mechanism of transformers and must be applied to a sequence length x sequence length sized attention score matrix. It can be a key bottleneck for hardware accelerators.\n\nThe proposed idea uses a 2D look-up table (LUT) indexed by most significant bits in the numerator and denominator of softmax. This avoids using a divider and allows the inputs to be in integer precision. When using 16 bits of precision, there is little to drop in accuracy on a variety of transformer NLP tasks with the approximate softmax.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have two main issues with the paper:\n\nFirst, I don't get the point of most of the derivations in Section 4.1. Take Equation (3):\n$$\n\\sigma^*(x_i) = \\frac{1}{e^{max(x)-x_i}}\n$$\nThis formula is the same as the softmax numerator:\n$$\n\\sigma^*(x_i) = \\frac{1}{e^{max(x)-x_i}} = e^{x_i-max(x)}\n$$\n\nNow take Equation (6):\n$$\n\\sigma(x_i) = \\sigma^*(x_i) * e^{-ln(\\sum{\\sigma^*(x_i)})}\n$$\nThe e^-ln(x) part is just 1/x, since \\sigma^*(x_i) is always positive. So Equation (6) reduces to \n$$\n\\sigma(x_i) = \\frac{\\sigma^*(x_i)}{\\sum{\\sigma^*(x_i)}}\n$$\nBut that's exactly the same as the regular softmax formula.\n\nI'm not sure what the point of the whole section is. From what I understand, LUTs are used to approximate e^(-x) and 1/x, see Equations (4) and (7). But the core softmax algorithm is unchanged. If this is correct, section 4.1 should be rewritten to be more clear, it is currently very confusing. The section begins by discussing REXP which seems irrelevant (1/e^x = e^(-x)). Once the notation is understood, the technique is basically to approximate the expensive ops in softmax (exponential and division) with LUTs.\n\nSecond, there is no hardware comparison or any sort of metric, despite this being a technique that only works for specialized HW accelerators. The idea of a 2D LUT sounds promising, but the LUT would have W^2 entries (W being the bitwidth), which is a lot. The LUT will achieve better throughput and latency than a HW divider, but how about the area cost? Dividers are necessary for other operations in a Transformer (e.g., layernorm) so would the LUT be replacing dividers or costing additional area? There are so many tradeoffs to consider and experiments to do in HW before an idea like this could be considered promising. This paper does not mention these points.\n\nThis being NeuRIPS, I don't expect an RTL hardware evaluation. But the paper claims that 2D LUTs are better than dividers - I need  to see some kind of hardware comparison between them. Area, throughput, energy, etc. The most novel part of the paper is the HW architecture (replace divider with LUT), but there is no architecture evaluation to convince readers.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No negative social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "aEdCXP5LFI5", "reviewer": "Reviewer_GRd4", "summary": "Current paper raises the necessity of efficient softmax approximation for custom hardware for attention-based models. Currently softmax computation requires data move to host and then back which is expensive operation. Existing methods on softmax approximation are applicable in case when softmax is at the end of computational graph. First, current paper demonstrates that existing methods on softmax approximation are not applicable to attention-based methods (performance degradation is huge or even a model does not work anymore, e.g. DETR gives 0 mAP). Second, authors propose two approaches on efficient softmax approximation, which is memory-centric: one is with two look-up tables (LUT) and multiplication and another one is with 2d LUT and no multiplication/division. Size of LUT is controllable parameter and depends on the size of attention in a model to have enough capacity to support different values. It is shown that for variety of tasks (object detection, machine translation, sentiment analysis, semantic equivalence) and models (DETR, BERT, Transformer) 8-bit approximation allows to obtain acceptable accuracy loss below 1.0%. Experimentally it is analyzed that ranges of numerator and denominator of softmax are stable if max normalization is applied to the input and LUT table with quite small size is enough (700 Bytes).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nThere are a lot of works on softmax approximation (covered well in the paper), mainly to use max normalization and logarithmic reformulation with further look-up tables (LUT) usage. Current paper also follows widely used technique on LUT, however via existing reciprocal exponentiation method authors approximate the softmax in a novel way. Before proposing a new methodology authors perform in-depth analysis of existing methods on softmax approximation and demonstrate that even logarithmic reformulation with max normalization is not applicable for attention-based models (performance drop is huge).\n\n**Quality**\n\nReport on prior art benchmark looks reasonable. Math on how softmax approximation is done is correct, however some suggestions on readability and typos are listed below. Proposed approximation benchmarking looks reasonable having in mind that it is performed and compared after model quantization which is a realistic scenario. Inference benchmark covers several models and domains, especially heavy DETR model, so in that respect experiments are convincing enough. Observed large performance degradation for DETR with bigger size of self-attentions is discussed in section 5.3: larger LUT sizes are not tested, however the trend is seen in Fig.2 and already 512 Bytes LUT provides <3% performance degradation.\n\n**Clarity**\n\nPaper mostly well written, however some parts/plots could be simplified (see suggestions below in the comments section) to improve readability. The paper focuses on the inference. All information about models used for benchmark (links on pre-trained open-sourced models, code-base to run them and quantization tutorial from pytorch) and implementation details on proposed algorithms + LUT structure and precision manipulations are given in the paper and Appendix.\n\n**Significance**\n\nSignificance of the work is in-depth discussed in sections 2 and 3 in the paper. While GPU/TPU already supports softmax, HW accelerators for the inference at the edge/IoT devices with limited computational power do not support it and the softmax becomes a bottleneck for attention-based models where it appears not only at the end of computational graph but in all intermediate attention-blocks. The paper proposes two ways on stable softmax approximation for the cases when softmax is extensively used in the intermediate operations (not only at the end) so that HW will have no extra time consuming operations on moving data to CPU host and then back for softmax computation. This opens efficient deployment of attention-based models on HW hardware, for example on mobile. Proposed approximation allows to obtain acceptable accuracy loss below 1.0% for the variety of tasks and models.\n\nComments:\n- why is speech recognition mentioned in the abstract while there is no any tests for speech recognition models in the paper?\n- line 66: how did authors compute the number of softmax operations as N*L*L? More precise is to say that there are N*L softmax operations where for each softmax operation we need to perform L exp operations and L divisions. Otherwise the original statement in the paper is misleading. Or formulate that O(N*L*L) arithmetic operations are needed.\n- line 80 Eq.2: The correct way to write down is to put $j$ index in the denominator in the summation\n- I think lines 171-178 and eq. (5)-(6) are redundant and does not make a clear point in the text. Authors could simply write down that let's represent eq. (2) as multiplication operation: $\\sigma(x_i) = \\sigma^*(x_i) \\cdot \\frac{1}{\\sum\\sigma^*(x_k)}$, where first multiplier can be computed with LUT eq.(4) and the second multiplier can be computed by another LUT defined by eq.(7)\n- Be consistent on usage index $i$/$j$ in the text/figures for numerator/denominator of eq. (2)\n- Section 4.2 can be also simplified. 188-192 lines is better to move into sec. 4.1 as this provides intuition why 2 LUT tables approach (sec 4.1) works (because of stability of distributions). Then 193-201 can be simplified because it is clear to say \"let's save now even multiplication operation in eq.(6) by pre-computing 2D LUT as LUT1 $\\times$ LUT2. And now as indices we even can use directly x and $\\sum e^x$\". \n- \"indexes\" -> \"indices\"\n- Suggestion: legends in Figure 2 could be put outside the plot, int16 can be solid, while int8 can be dashed  but with the same color or mark DC5 with the same color as no_DC5 but dashed.\n- For Fig. 2 would be good to add plot for PTQ-D variant as it is shown in Figure 3.\n- Could authors explain why in Table 2 2D LUT for BERT uint2 F1 is so bad compared to REXP? Do authors have any intuition when LUT 2D is better than REXP in terms of accuracy (if we assume that for hardware both have the same real time)?\n- For Fig. 3 for readability make 2DLUT models solid and REXP make dashed and the same color as correspondent 2DLUT.\n- I have checked https://opennmt.net/Models-py/ and WMT 14 has 26.89 BLEU, while in Table 2 it is reported 26.98. Is it typo or reevaluation artifact?\n- It is necessary to include results on Semantic equivalence too (at least in the Appendix).\n- line 282: \"there are shown the histogram of ...\" -> \"the histogram of ... is shown\"\n- line 472: \"In Table 3 there are shown results of experiments\" -> e.g. \"Table 3 shows results of experiments\"\n- line 477: \"Eg.( 12)\" -> \"Eg.(12)\"\n- line 483: \"These codes mimics\" -> \"These codes mimic\" or \"This code mimics\"\n- Algorithm 1, line 9-11: why do we need $idx_{e^{x_i}}$? why the line 11 is not like $\\sigma(x_i)=LUT_{1/e}[idx_{x_i}]\\cdot LUT_\\alpha[idx_\\alpha]$\n- what about real-time testing on HW hardware? (I believe this is not really necessary, but curious to see how not approximated softmax slows down the inference in practice).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors discussed limitations of their work throughout the paper (that proposed approach is HW hardware oriented) and in section 5.3 when encoders with bigger self-attentions are used what causes larger variation in values of $\\sum e^x$ (used LUT sizes cover small range of different values). \n\nThere is no potential negative societal impact as the paper focuses on inference optimization.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "YEd0c3QToM1", "reviewer": "Reviewer_3inJ", "summary": "This paper presents two hardware-friendly (no floating points or divisions) quantization methods for softmax computation. It is particularly targeted at the use in attention layers in transformer models, where the output precision of the softmax layer is critical to the overall accuracy of the model.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Note that I was one of the reviewers of a previous version of this paper submitted to ICML 2020.\n\nIt is appreciated that the new experiments are done on fully quantized models, which makes more sense than having only the softmax quantized while the rest is computed in FP32. \nIndeed a fully quantized model could run on an FPGA for example, where not only divisions are to be avoided, but also floating point operations in general.\n\nWhile the LUT-based methods presented are not extremely revolutionary, the experimental results prove that they are viable for use in complex transformer models while incurring an accuracy drop of about a percent, which could be acceptable in certain edge compute scenarios.\n\nHowever the paper is hard to follow, notably because of the numerous typos and grammatical errors. It is particularly alarming considering that the errors I sampled in my ICML 2020 review, for which I provided corrections, **are still present**:\n- (line 28) \"up to 13X times\" -> \"up to 13 times\" -- as \"X\" stands for \"times\".\n- (line 68) \"and sequence length is\" -> \"and the sequence length is\"\n- (eq 2) should be $ \\sigma (x_j) = \\frac{e^{x_j}}{\\sum_{i} e^{x_i}} = \\frac{e^{x_j - max(x)}}{\\sum_{i} e^{x_i - max(x)}} $\n- (line 197) \"to calculate the indexes for corresponded values in 2D LUT table\" -> \"to calculate the indices for the corresponding values in the 2D LUT\" -- other than the 2 grammatical values, note that LUT stands for Lookup Table.\n- (line 199) \"MSB bits\" -> \"MSBs\" -- MSB stands for Most Significant Bit.\n- (line 211) \"60 is big enough\" -> \"60 is large enough\" -- bad choice of word.\n- (line 233) \"we run a bunch of experiments\" -> \"we run multiple experiments\" -- bad choice of word.\n- Anywhere \"what\" is used but not in a question, should use \"which\" instead.\n- The bibliography entries are using different formats, and some are missing information.\n\nWhile 2 solutions are presented, 2D-LUT and REXP, it is not clear if the goal of the paper is to compare them. Indeed the NLP experiments do compare the performance of the two solutions, however the object detection experiments using DETR seem to rely only on REXP. Is there a particular reason for this? Is the 2D-LUT somehow problematic in the DETR case?\n\nOther presentation issues:\n- The plots in figure 2 and 3 are not easy to read. And more generally, it is preferred to use alternatives to color-coding, since it is common to print in black and white, and could also pose problems for color-vision impaired persons.\n- In plot 4, the averages would have been good enough. Having tens of superimposed lines is not helping with interpreting the data.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have clearly measured the accuracy drop incurred by the use of their method, particularly when using dilation layers in the DETR model.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "HhbgSqkQb5a", "reviewer": "Reviewer_4L69", "summary": "As the self-attention models have been widely used in NLP and computer vision community, a cost efficient implementation of softmax is becoming critical. This paper proposes two methods based on LookUp Table (LUT) to approximate the softmax computation. The proposed method shows generalise ability on a number of tasks including object detection, machine translation, speech recognition and semantic equivalence. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My most concern is that the paper lacks of baseline methods to validate its superiority. It only compares with easy baselines .\n\nBig accuracy drop occurs when large self-attention matrix is needed but the paper only gives hypothesis rather than experiments -- \"we expect that increasing size of LUTs will help to solve this issue\" (L241).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents two efficient 8 bits approximations for the softmax operation in the context of self-attention, and benchmarks it in object detection, machine translation, speech recognition, semantic equivalence. The loss of accuracy is minimal, the downstream task effects evaluation is adequate. The value of the contribution is focused on custom hardware. There is a thorough review (GRd4) that is supportive of acceptance, but the other reviewers were not convinced. In particular, the clarity of the paper, the hardware evaluation, as well as contextualization, can be improved further.
 
Overall, the contributions look interesting and promising, but the paper is not in a state ready for publication at NeurIPS.
Decision: Reject