Title: FULLY QUANTIZED TRANSFORMER FOR IMPROVED TRANSLATION

Abstract: State-of-the-art neural machine translation methods employ massive amounts of parameters. Drastically reducing computational costs of such methods without affecting performance has been up to this point unsuccessful. To the best of our knowledge, we are the first to propose a quantization strategy inclusive of all components of the Transformer (Vaswani et al., 2017) . We are also the first to show that it is possible to avoid any loss in translation quality with a fully quantized network. Indeed, our 8-bit models consistently score equal or higher BLEU than the full-precision variant on multiple translation datasets. Comparing ourselves to all previously proposed methods, we achieve state-of-the-art quantization results.

1. INTRODUCTION:

The idea of using neural networks for machine translation was proposed only recently (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014) . Nonetheless, the approach has reached impressive levels of translation. (Ahmed et al., 2017; Ott et al., 2018; Edunov et al., 2018) . A key element of this success was to allow the decoder to attend to all hidden states of the encoder (Bahdanau et al., 2014) . A few variations to this additive attention mechanism were proposed, such as multiplicative attention and self-attention (Luong et al., 2015; Cheng et al., 2016; Lin et al., 2017) . The latter formed the basis of the Transformer network (Vaswani et al., 2017) , which achieved stateof-the-art machine translation. Inspiring a new wave of work, numerous natural language processing tasks reached new heights (Devlin et al., 2018; Liu et al., 2019) . Unfortunately, these models make use of an enormous amount of parameters. Inference on resource-limited hardware such as edgedevices is thus impractical.

A solution to reduce the computational burden of these neural networks is to lower numerical precision. Consequently, numerical values can be represented using fewer bits (Tang & Kwan, 1993; Marchesi et al., 1993) . This method called quantization has the advantage of providing good compression rates with minimal loss in accuracy. It is also conveniently supported by most hardware. Properly quantizing the Transformer would allow computational speed gains at inference, as well as deployment on more constrained devices.

In this work, we propose a custom quantization strategy of the entire Transformer architecture, where quantization is applied during the training process. Our method is easy to implement and results are consistent with the full-precision Transformer. We test our approach on multiple translation tasks such as WMT14 EN-FR and WMT14 EN-DE and obtain state-of-the-art quantization results. On most tasks, our quantized models score equal or higher BLEU compared to full-precision. We are, to the best of our knowledge, the first to fully quantize the Transformer architecture without impairing translation quality.

2.1. QUANTIZATION:

Over the years, a large range of methods have been proposed to quantize neural networks. These include, among many others, binary (Courbariaux et al., 2016) , ternary (Lin et al., 2015; Li et al., 2016) , uniform affine (Jacob et al., 2017) and learned (Zhang et al., 2018) quantization.

Quantization has been applied to RNNs (Jordan, 1990) , LSTMs (Hochreiter & Schmidhuber, 1997) and GRUs (Cho et al., 2014) . Ott et al. (2016) propose an exponential quantization method for RNN weights. They compare their method with binary and ternary quantization on language modeling and speech recognition. They reach the conclusion that binary weights are ineffective for RNNs, while ternary and exponential quantization perform well. Hubara et al. (2016) quantize weights and activations of RNNs and LSTMs to 2, 4 and 6-bit. He et al. (2016) propose modifications to the gates and interlinks of quantized LSTM and GRU cells, as well as a balanced quantization method for weights. Wu et al. (2016) successfully quantize the stacked LSTM neural machine translation model to 8-bit without any loss in translation quality. Wang et al. (2018) propose to use different quantization methods for different RNN components.

With regards to CNNs (LeCun et al., 1989) , various works have explored quantizing the architecture. Gong et al. (2014) compare matrix factorization, binarization, k-means clustering, product quantization and residual quantization on CNNs. Wu et al. (2015) apply quantization to both kernels and fully connected layers of convolutional neural networks. Rastegari et al. (2016) evaluate the use of binary weights on AlexNet (Krizhevsky et al., 2012) . Zhou et al. (2016) use low bitwidth weights, activations and gradients on CNNs.

Quantization has also been jointly used with other compression methods. Han et al. (2015) combine pruning, quantization, weight sharing and Huffman coding, while Polino et al. (2018) use quantization with knowledge distillation (Hinton et al., 2015) for higher compression rates. LeCun et al. (1990) were the first to propose a Hessian based method to prune neural net weights, with Hassibi et al. (1994) later improving the method. More recently, See et al. (2016) show that pruning a fully trained model and then retraining it can increase performance over the original nonpruned model. Gradually pruning in tandem with training has also been shown to increase performance (Zhu & Gupta, 2017) . Liu et al. (2017) prune nodes instead of weights by applying a penalty in the loss on the γ parameters of batch normalization layers. Narang et al. (2017b) make better use of hardware by applying pruning and weight decay in blocks to minimize the number of loaded weight matrix chunks. Chen et al. (2018) combine quantization with block based low-rank matrix approximation of embeddings.

2.2. PRUNING:

Pruning methods have also been adapted to specific architectures. Liu et al. (2015) propose an efficient sparse matrix multiplication algorithm for CNNs. For RNNs, Narang et al. (2017a) show sparse pruning to work well on the architecture. In order to maintain dimension consistency, Wen et al. (2017) propose to prune all basic LSTM structures concurrently. Park et al. (2018) introduce simple recurrent units (SRUs) for easy pruning of RNNs.

3. QUANTIZATION STRATEGY:

3.1 QUANTIZATION METHOD Our quantization methodology was chosen to be uniform, meaning that the step size between two quantized values is constant. This choice, which is an additional constraint, was made for practical reasons. It indeed simplifies all computations required during inference, enabling the exploitation of hardware resources more efficiently. If the performance with uniform quantization is already on par with full-precision, then more weighty methods are unnecessary. The uniform quantization scheme employed is further described by Jacob et al. (2017) .

Given an element x of a tensor X, we apply the quantization function Q:

EQUATION

where x min and x max defines the endpoints of the quantization interval. When quantization is applied to weights, these values are respectively min(X) and max(X). However, when quantization is applied to activations, those values are running estimates. The latter are computed during training, where for every forward pass, the x min and x max variables are updated via an exponential moving average with a momentum of 0.9. The value k is the bit precision. For example, in the context of 8-bit quantization, k = 8.

At training time, we simulate quantization by first quantizing and then rescaling to the original domain:

EQUATION

where the clamp function associates all the values outside the [x min , x max ] range to the closest endpoint and • represents rounding to the nearest integer. During backpropagation, we use the straight-through estimator (Hinton, 2012) and set the gradients of clamped values to zero. The only exception is for the LayerNorm's denominator, where values can still be clamped, but gradients are never zeroed. Once training is finished, s and x min are frozen along with the weights.

3.2. WHAT TO QUANTIZE:

We choose to quantize all operations which can provide a computational speed gain at inference. In this regard, we quantize all matrix multiplications, meaning that the inputs and weights of MatMuls will both be k-bit quantized. The other operations we quantize are divisions, but only if both the numerator and denominator are second or higher rank tensors. For all other operations, such as sums, the computational cost added by the quantization operation outweighs the benefit of performing the operation with reduced precision. Hence, we do not quantize such operations.

More precisely, we quantize all weights of the Transformer, excluding biases. The latter are summed with the INT32 output of matrix multiplications and thus provide no additional computational efficiency from being quantized. Furthermore, the memory space of biases is insignificant in comparison to the weight matrices, representing less than 0.1% of total weights. For positional embeddings, memory gain is also minimal, but since these will be summed with the quantized input embeddings, we likewise quantize them. These embeddings also stay fixed, we can thus quantize them once prior to training. The γ weights of LayerNorms are also quantized. As for activations, we quantize the sum of the input embeddings with the positional encodings in both the encoder and decoder.

In the Multi-Head Attention, we quantize the (Q, K, V ) input, the softmax's numerator, the softmax's denominator, the softmax's output and the Scaled Dot-Product Attention's output. For the position-wise feed-forward networks, we quantize the output of the ReLUs and of the feed-forward networks themselves. Finally, for all LayerNorms, we quantize the numerator x-µ, the denominator √ σ 2 + , their quotient and the output of the LayerNorm.

3.3. BUCKETING:

Instead of using a single set of (s, x min ) per quantized tensor, we can quantize subsets of the latter with each its own set of (s, x min ) (Alistarh et al., 2016) . Even though this adds more scalars, the memory cost is insignificant overall. Furthermore, the added flexibility can greatly alleviate the precision loss resulting from trying to fit all values of a tensor into a single domain with lower numerical precision.

We use this bucketing method for all weight matrices, with the number of subset equal to the output dimension. For activations, we use bucketing when quantizing: the sum of input embeddings with the positional encoding, the Q, K, V inputs, the Scaled Dot-Product Attention's output, the feed-forward's output, the LayerNorm's numerator, the LayerNorm's quotient and the LayerNorm's output. 2017), we do not nudge the domain so that the zero value gets perfectly mapped.

The only zero values which we have to deal with are the padding, the output of ReLU layers and dropouts. Since padding has no effect on the final output, we completely ignore these values when quantizing. For ReLUs, we fix the x min estimate of those quantization layers to 0, which guarantees the perfect mapping of the value. Finally, quantization is applied before any dropout operation. Indeed, even though the zeros added to the output of the quantization layer might not be part of the domain, this only happens during training.

4. RELATED WORK:

Recently, simple quantization solutions have been applied to the Transformer. Cheong & Daniel (2019) apply k-means quantization and binarization with two centroids over the weights of the network. For both methods, a look up table associated with each quantized layer is used to map indices to their corresponding centroids. Similarly, Fan (2019) compares binary, 4 and 8-bit uniform quantization of the Transformer weights. A big disadvantage with quantizing only the weights of a network is that operations must still be performed in full-precision. Even though there is a reduced memory usage of parameters, these constantly have to be casted back to full-precision. Achieving quantization of both weights and activations is thus much more beneficial. The first attempt at doing so for the Transformer applied 8-bit quantization on weights and inputs of feed forward layers and binarizes the (Q, K) input of the Multi-Head Attention (Tierno, 2019) . The scaling factor √ d k is approximated by a constant which can be computed as a right bitshift. The method though results in huge drop in translation accuracy. Achieving better performance, Bhandare et al. (2019) quantize certain MatMul operations and use the KL divergence to estimate the most suited parameters for each quantization range. They restrain from quantizing all MatMuls, reporting that this resulted in poor accuracy.

All of these methods omit quantizing the whole Transformer architecture, resulting in suboptimal computational efficiency. Furthermore, these solutions all fail to avoid impairing translation quality. Our method achieves both.

5. EXPERIMENTS:

In this section, we present the results of our full quantization scheme on various tasks. We first compare our method on a machine translation setup. We then present the results of numerous ablation studies. We also compare the impact of delaying quantization on translation quality. Finally, we evaluate our method on two language model tasks.

5.1. FULL QUANTIZATION:

We apply our quantization strategy on both the base and big Transformer (Vaswani et al., 2017) . The training setup of all presented models is the same as in the original paper, with the exception that the dropout ratio is set to 0.1 in all cases. We refer readers to the original paper for experimental details. Our models were first evaluated on the WMT 2014 / 2017 English-to-German and WMT 2014 English-to-French translation tasks. Section A contains results for additional languages. Reported perplexity is per token and BLEU was measured with multi-bleu.pl1 on the newstest20142 test set. We used beam search with a beam size of 4 and a length penalty of 0.6, as in (Vaswani et al., 2017) . Another difference is that no checkpoint averaging was performed.

We In Table 2 , we show performance of our method on the WMT14 EN-DE and WMT14 EN-FR for a fixed amount of training steps. We compare our results with two full-precision Transformers: base and big variants. Training the quantized models was about twice as slow as training the baselines.

We also compare with two other quantization approaches. The first one is the "default" approach, which is to naively quantize every possible operation. The second approach applies our quantization strategy post-training (see section 5.3 for details). In all cases except for post-quantization, BLEU was computed on the test set using the checkpoint which scored the highest accuracy on the validation set. Towards the end of training, we ran one validation epoch for every 100 training steps. As for post-training quantization, the BLEU score was computed on the test set using the best scoring BLEU score on the validation set out of 20 trials. The latter varied by about 0.2 BLEU. For the big Transformer variants, best results were obtained when not bucketing the Scaled Dot-Product Attention's output and the sum of the decoder's input embeddings with the positional encoding.

The reason for the default approach's nan in the EN-FR task is because quantizing every operation causes numerical instability in the LayerNorm's denominator, normally provided by the .

Generally, fully quantizing the Transformer seems to result in no loss in translation accuracy. We believe the reason for this might be the lower numerical precision acting as a regularization effect.

Looking for such an effect in the training and validation curves, differences were too subtle for any conclusions to be made.

All models use full-precision biases, s and x min . This amounts to 6.52 Mb in the base models and 13.04 Mb in the big models. Without bucketing, this would amount to 2.17 Mb and 4.33 Mb respectively. All in all, these represent less than 2% of the total size of our quantized models. We believe the small increase in model size is worth it in the case of bucketing. We show in section 5.2 that training without leads to poorer translation.

Although 6-bit quantization seems to perform well, the compression advantage over 8-bit is usually lost. Most hardware store INT6 using either 8 or 32 bits. Dedicated hardware is needed to get the full compression advantage. Unless 6-bit quantization results in better models, sticking to 8-bit seems like the best choice for most hardware.

5.2. ABLATION STUDIES:

To compare the effect of bucketing and better understand which operation is more sensitive to quantization, we evaluate the effect of quantizing to 8-bit single operations of the Transformer, with and without bucketing. By single operation, we mean quantizing the operation of a module for all Transformer layers. Table 3 shows results on the WMT14 EN-FR translation task. BLEU was computed on the test set after 100k steps of training. The only operations underperforming our full-precision baseline of 38.36 BLEU are the LayerNorm's numerator when not bucketed and the denominator. The latter cannot be bucketed because all dimensions of the variance tensor vary per batch. Solely quantizing the LayerNorm's denominator with no bucketing works, but results are poor. To successfully quantize this element without causing performance loss, we suspect quantizing other elements in the network helps.

To further validate our quantization scheme, we evaluated four models trained with alterations to our design choices. Results are presented in Table 4 ). All models are 8-bit quantized base Transformers, trained in the same fashion as in section 5.1 on the WMT14 EN-FR task. Our method's goal is to increase computational efficiency when inferring with the Transformer. To this end, our quantization scheme only requires us to learn s and x min . Although we do so with our quantization scheme throughout the whole training, this is not a necessity. Quantization could also be applied later on while training. Results for different starting points are compared in Table 5 . The earliest we start quantizing is at 100 steps, since we need at least a few steps to assess the x min and x max running estimates. We consider this to be training with quantization "from scratch". Posttraining quantization is also an option, where once the model is fully trained, we keep the weights fixed, but compute the s, x min and x max over a few hundred steps. All models were evaluated on the WMT14 EN-DE and WMT14 EN-FR translation tasks. BLEU was measured on the test set using the checkpoint which scored the highest accuracy on the validation set during training.

Validation was computed every 100 training steps towards the end of training. From our observed results, quantizing the model early on seems preferable. This reinforces our belief that quantization helps training via regularization.

Learning quantization parameters adds a significant computational cost during training. A major advantage to delaying quantization is to perform more training steps in the same given amount of time. Therefore, when training time is a constraint, a possible strategy is to train a model without quantization, perform more training steps and finally post-quantize the model. Another advantage of post-quantization is that the method is quick to perform. This makes it easy to run many iterations and search for the best performing candidate.

5.4. LANGUAGE MODELING:

To evaluate if our quantization scheme generalizes well to other tasks, we evaluate it on a language modeling benchmark. As the setup, we use PyTorch's language modeling toy example3 on the WikiText-2 and WikiText-103 corpus. The task consists of predicting the sequence {x t+1 

6. CONCLUSION:

We proposed a quantization strategy for the Transformer, quantizing all operations which could provide a computational speed gain, for a fully quantized architecture. All of our design decisions were aimed at maximizing computational efficiency while making sure our method would be compatible with as many different types of hardware as possible.

With our method, we achieve higher BLEU scores than all other quantization methods for the Transformer on multiple translation tasks and avoid any loss in BLEU compared to full-precision. Specifically, out of 41 experiments, 8-bit quantization performed equal or better to full-precision in 36 cases. The threshold allows us to find the right ratio of nodes to prune per layer instead of the usual: decide the ratio first and then prune. We compared with two such where for each task, we fix the ratio to the global percentage of nodes pruned in the encoder by our method. The first fixed pruning method uses L1-norm to sort nodes to prune in ascending order, while the second sorts nodes using the x max , also in ascending order.

Since x max is a running estimate, results varied per trial. The method only takes a few hundred training steps to perform though, so running many trials is not an issue. We evaluated our method on the validation set a few times until accuracy increased over the non-pruned model. Perplexity and BLEU were then computed on the test set. Reported results are averaged over a few trials. When accuracy was not improving, BLEU would either be the same or decreased by about 0.01-0.02. 

:

BACKGROUNDIn this section, we review a broad spectrum of quantization and pruning methods for neural network compression.

https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/ multi-bleu.perl

https://www.statmt.org/wmt14/translation-task.html

https://github.com/pytorch/examples/tree/master/word_language_model

Figures and Tables:

Figure 1: Histograms of the running estimates x max of a ReLU layer in the encoder (left) and decoder (right), one x max per output node of the layer.

Our quantization strategy achieves better BLEU scores than all other quantization methods for the Transformer on the WMT14 EN-DE, WMT14 EN-FR and WMT17 EN-DE test set.

Table content: <table><tr><td>Method</td><td>Fully Quantized [EN-DE, EN-FR] Size (Gb)</td><td>Compr.</td><td colspan="3">BLEU EN-DE (2014) EN-FR EN-DE (2017)</td></tr><tr><td>Vaswani et al. (2017)</td><td>[2.02, 1.94]</td><td>1x</td><td>27.3</td><td>38.1</td><td>-</td></tr><tr><td>Cheong &amp; Daniel (2019)</td><td>0.69</td><td>2.92x</td><td>-</td><td>-</td><td>27.38</td></tr><tr><td>Bhandare et al. (2019)</td><td>≥ 0.96</td><td>≤ 2.1x</td><td>27.33</td><td>-</td><td>-</td></tr><tr><td>Fan (2019)</td><td>≥ 0.51</td><td>≤ 3.99x</td><td>26.94</td><td>-</td><td>-</td></tr><tr><td>Our method</td><td>[0.51, 0.49]</td><td>3.95x</td><td>27.60</td><td>39.91</td><td>27.60</td></tr><tr><td colspan="2">3.4 DEALING WITH ZEROS</td><td/><td/><td/><td/></tr><tr><td>Unlike Jacob et al. (</td><td/><td/><td/><td/><td/></tr></table>

Performance of our quantization method on the WMT14 EN-DE and WMT14 EN-FR test set for a fixed number of training steps.

Table content: <table><tr><td colspan="2">Model Method</td><td>Precision</td><td colspan="8">EN-DE PPL BLEU Size (Gb) Compr. PPL BLEU Size (Gb) Compr. EN-FR</td></tr><tr><td>Base</td><td>Baseline</td><td>32-bit</td><td>4.91</td><td>26.42</td><td>2.02</td><td>1x</td><td>3.23</td><td>38.36</td><td>1.94</td><td>1x</td></tr><tr><td/><td>Default Approach</td><td>8-bit</td><td>74.04</td><td>0.21</td><td>0.51</td><td>3.95x</td><td>nan</td><td>0</td><td>0.49</td><td>3.95x</td></tr><tr><td/><td>Post-Quantization</td><td>8-bit</td><td>4.97</td><td>26.44</td><td>0.51</td><td>3.95x</td><td>3.26</td><td>38.30</td><td>0.49</td><td>3.95x</td></tr><tr><td/><td>Our method</td><td>8-bit</td><td>4.67</td><td>26.98</td><td>0.51</td><td>3.95x</td><td>3.23</td><td>38.55</td><td>0.49</td><td>3.95x</td></tr><tr><td/><td>Post-Quantization</td><td>6-bit</td><td>6.00</td><td>24.84</td><td>0.38</td><td>5.25x</td><td>3.98</td><td>35.02</td><td>0.37</td><td>5.24x</td></tr><tr><td/><td>Our method</td><td>6-bit</td><td>5.09</td><td>26.98</td><td>0.38</td><td>5.25x</td><td>3.38</td><td>37.07</td><td>0.37</td><td>5.24x</td></tr><tr><td/><td>Our method</td><td>4-bit</td><td colspan="2">11.96 18.32</td><td>0.26</td><td>7.81x</td><td>48.21</td><td>1.59</td><td>0.25</td><td>7.8x</td></tr><tr><td>Big</td><td>Baseline</td><td>32-bit</td><td>4.03</td><td>26.85</td><td>6.85</td><td>1x</td><td>2.72</td><td>40.17</td><td>6.69</td><td>1x</td></tr><tr><td/><td>Post-Quantization</td><td>8-bit</td><td>4.27</td><td>26.55</td><td>1.73</td><td>3.97x</td><td>2.78</td><td>39.78</td><td>1.68</td><td>3.97x</td></tr><tr><td/><td>Our method</td><td>8-bit</td><td>4.24</td><td>27.95</td><td>1.73</td><td>3.97x</td><td>2.80</td><td>40.17</td><td>1.68</td><td>3.97x</td></tr><tr><td/><td>Post-Quantization</td><td>6-bit</td><td>5.12</td><td>24.86</td><td>1.30</td><td>5.28x</td><td>3.08</td><td>37.92</td><td>1.27</td><td>5.28x</td></tr><tr><td/><td>Our method</td><td>6-bit</td><td>4.78</td><td>26.76</td><td>1.30</td><td>5.28x</td><td>2.87</td><td>39.59</td><td>1.27</td><td>5.28x</td></tr><tr><td/><td>Our method</td><td>4-bit</td><td colspan="2">33.11 10.22</td><td>0.87</td><td>7.88x</td><td>42.42</td><td>2.81</td><td>0.85</td><td>7.88x</td></tr></table>

Effect of quantizing single activations of the Transformer on the translation quality. Results are on the WMT14 EN-FR test set.

Table content: <table><tr><td>Module</td><td>Quantized Activation</td><td colspan="4">No Bucketing PPL BLEU PPL BLEU Bucketing</td></tr><tr><td>Encoder</td><td colspan="2">(Input Embedding + Positional Encoding) 3.20</td><td colspan="3">38.61 3.20 39.08</td></tr><tr><td>Decoder</td><td colspan="2">(Input Embedding + Positional Encoding) 3.20</td><td colspan="3">39.35 3.20 39.36</td></tr><tr><td>Multi-Head</td><td>Input (Q, K, V )</td><td>3.21</td><td colspan="3">39.06 3.21 39.29</td></tr><tr><td>Attention</td><td>LayerNorm Output</td><td>3.21</td><td colspan="3">39.09 3.20 38.78</td></tr><tr><td>Scaled Dot-Product Attention</td><td>Softmax Numerator Softmax Denominator Softmax Output Output</td><td>3.20 3.21 3.22 3.21</td><td colspan="3">39.32 3.21 39.01 39.35 3.21 39.11 39.41 3.22 38.87 38.73 3.21 39.02</td></tr><tr><td/><td>ReLU Output</td><td>3.21</td><td colspan="3">39.43 3.22 38.93</td></tr><tr><td>Feed-forward</td><td>Feed-forward Output</td><td>3.54</td><td colspan="3">38.03 3.20 39.27</td></tr><tr><td/><td>LayerNorm Output</td><td>3.21</td><td colspan="3">38.67 3.21 39.04</td></tr><tr><td/><td>Numerator</td><td>3.53</td><td colspan="3">37.75 3.21 38.86</td></tr><tr><td>LayerNorm</td><td>Denominator</td><td>1748</td><td>0</td><td>-</td><td>-</td></tr><tr><td/><td>Quotient</td><td>3.22</td><td colspan="3">38.97 3.21 39.02</td></tr></table>

Variations to our quantization scheme evaluated on the WMT14 EN-FR translation task.

Table content: <table><tr><td>Method</td><td>PPL</td><td>BLEU</td></tr><tr><td>No Bucketing</td><td>3.58</td><td>36.49</td></tr><tr><td>No Gradient Clipping</td><td>2549.30</td><td>0</td></tr><tr><td>No LayerNorm Denominator Quantization</td><td>3.20</td><td>37.97</td></tr><tr><td>8-bit Quantized Weights, Full-precision Activations</td><td>3.19</td><td>37.94</td></tr></table>

Impact of delaying quantization on translation quality. Results are on the WMT14 EN-DE and WMT14 EN-FR test set.

Table content: <table><tr><td>Quantization Start</td><td>EN-DE</td><td>EN-FR</td></tr><tr><td>(training step)</td><td colspan="2">PPL BLEU PPL BLEU</td></tr><tr><td colspan="3">Base (no quantization) 4.91 26.42 3.23 38.36</td></tr><tr><td colspan="3">100 4.67 26.98 3.23 38.55</td></tr><tr><td colspan="3">10000 4.99 26.63 3.21 38.62</td></tr><tr><td colspan="3">50000 4.98 26.84 3.21 38.50</td></tr><tr><td colspan="3">80000 5.03 26.41 3.21 38.43</td></tr><tr><td colspan="3">Post-Quantization 4.45 25.50 3.22 37.96</td></tr></table>

Evaluation of our quantization method on the WikiText-2 and WikiText-103 language modeling tasks.

Table content: <table><tr><td colspan="3">Precision Size (Gb) Compression</td><td>WikiText-2 Loss PPL</td><td>WikiText-103 Loss PPL</td></tr><tr><td>32-bit</td><td>243.04</td><td>1x</td><td colspan="2">5.65 284.15 5.91 369.20</td></tr><tr><td>8-bit</td><td>61.22</td><td>3.97x</td><td colspan="2">5.64 282.67 5.94 377.79</td></tr><tr><td>6-bit</td><td>46.04</td><td>5.28x</td><td colspan="2">5.64 281.48 5.93 376.44</td></tr><tr><td>4-bit</td><td>30.86</td><td>7.87x</td><td colspan="2">5.65 284.26 5.94 378.67</td></tr><tr><td colspan="2">5.3 DELAYING QUANTIZATION</td><td/><td/></tr></table>

We trained four Transformer models, one full precision and three with our quantization method. In each case, the model consists of two Transformer encoder layers, with the embedding and hidden size set to 200. Multi-Head Atten-tion has two heads with keys and values of size 64. The final word projection layer's weights are shared with the embedding layer. Models were trained for 10 epochs with a batch size of 20 and sequence length of 35. Learning rate is set to 5, dropout to 0.2 and gradient clipping to 0.25. Loss is computed on every element of the output sequence. We refer readers to the PyTorch example (see 3) for any extra details. Results are presented in Table6. Validation loss was computed every epoch to determine the best candidate. Loss and perplexity are computed on the test set and averaged over 10 trials for WikiText-2 and 3 trials for WikiText-3.

Table content: <table/>

Comparison of our adaptive pruning scheme versus fixed rate pruning methods for equal pruning proportions. Total compression is of quantization combined with pruning.

Table content: <table><tr><td>Model</td><td>Task</td><td>Precision</td><td>Method</td><td>PPL BLEU</td><td>Nodes Pruned in Encoder FF</td></tr><tr><td>Base</td><td>EN-DE</td><td>8-bit</td><td>No pruning</td><td>4.39 27.60</td><td>0%</td></tr><tr><td/><td/><td/><td colspan="2">L1-norm, fixed 5.57 23.99</td><td>13.57%</td></tr><tr><td/><td/><td/><td>x max , fixed</td><td>4.57 27.33</td><td>13.57%</td></tr><tr><td/><td/><td/><td colspan="2">x max , adaptive 4.40 27.60</td><td>13.57%</td></tr><tr><td/><td/><td>6-bit</td><td>No pruning</td><td>5.09 26.98</td><td>0%</td></tr><tr><td/><td/><td/><td colspan="2">L1-norm, fixed 6.97 20.81</td><td>12.06%</td></tr><tr><td/><td/><td/><td>x max , fixed</td><td>5.41 26.20</td><td>12.06%</td></tr><tr><td/><td/><td/><td colspan="2">x max , adaptive 5.09 26.98</td><td>12.06%</td></tr><tr><td/><td>EN-FR</td><td>8-bit</td><td>No pruning</td><td>2.90 39.91</td><td>0%</td></tr><tr><td/><td/><td/><td colspan="2">L1-norm, fixed 4.38 29.01</td><td>9.47%</td></tr><tr><td/><td/><td/><td>x max , fixed</td><td>3.18 39.40</td><td>9.47%</td></tr><tr><td/><td/><td/><td colspan="2">x max , adaptive 2.90 39.91</td><td>9.47%</td></tr><tr><td/><td/><td>6-bit</td><td>No pruning</td><td>3.38 37.07</td><td>0%</td></tr><tr><td/><td/><td/><td colspan="2">L1-norm, fixed 4.19 31.64</td><td>9.62%</td></tr><tr><td/><td/><td/><td>x max , fixed</td><td>3.68 36.91</td><td>9.62%</td></tr><tr><td/><td/><td/><td colspan="2">x max , adaptive 3.38 37.07</td><td>9.62%</td></tr><tr><td>Big</td><td>EN-DE</td><td>8-bit</td><td>No pruning</td><td>4.24 27.95</td><td>0%</td></tr><tr><td/><td/><td/><td colspan="2">L1-norm, fixed 5.80 22.65</td><td>26.39%</td></tr><tr><td/><td/><td/><td>x max , fixed</td><td>4.47 27.43</td><td>26.39%</td></tr><tr><td/><td/><td/><td colspan="2">x max , adaptive 4.25 27.95</td><td>26.39%</td></tr><tr><td/><td/><td>6-bit</td><td>No pruning</td><td>4.78 26.76</td><td>0%</td></tr><tr><td/><td/><td/><td colspan="2">L1-norm, fixed 7.73 17.32</td><td>29.96%</td></tr><tr><td/><td/><td/><td>x max , fixed</td><td>4.92 26.86</td><td>29.96%</td></tr><tr><td/><td/><td/><td colspan="2">x max , adaptive 4.78 26.76</td><td>29.96%</td></tr><tr><td/><td>EN-FR</td><td>8-bit</td><td>No pruning</td><td>2.80 40.17</td><td>0%</td></tr><tr><td/><td/><td/><td colspan="2">L1-norm, fixed 4.16 28.85</td><td>28.41%</td></tr><tr><td/><td/><td/><td>x max , fixed</td><td>2.91 39.40</td><td>28.41%</td></tr><tr><td/><td/><td/><td colspan="2">x max , adaptive 2.80 40.17</td><td>28.41%</td></tr><tr><td/><td/><td>6-bit</td><td>No pruning</td><td>2.87 39.59</td><td>0%</td></tr><tr><td/><td/><td/><td colspan="2">L1-norm, fixed 7.88 15.09</td><td>22.66%</td></tr><tr><td/><td/><td/><td>x max , fixed</td><td>2.91 39.25</td><td>22.66%</td></tr><tr><td/><td/><td/><td colspan="2">x max , adaptive 2.87 39.59</td><td>22.66%</td></tr></table>

Peer review (iclr2020):
This paper proposes an 8-bit quantization method to quantize the Transformer, which is a popular machine translation model. Uniform min-max quantization is used for computational expensive operations during the inference. The authors also propose to bucket the weights before quantization to reduce quantization error. Experiments are performed on WMT translation tasks. The paper is overall easy to follow. One of my main concerns is about the novelty since both the min-max quantization and bucketing the weights before quantization are not new. Another concern is that one important goal of quantization is to speed up inference, however, no inference time is reported in the paper. Can the authors provide the inference time results on some typical cpu/gpus? From Table 4, the proposed quantization has much larger BLEU gain than the base transformer after 100k training steps, than that after full training (Table 2). Does this mean the proposed quantization converges slower at the later stage of training? How many training steps are run to reach the result of the proposed method in Table 1? Can the authors compare the training curve of the base transformer and the quantized one? From Table 4, the earlier the quantization starts, the better the final performance. What about the performance of training from scratch? Will this give better performance? ------------post rebuttal comment----------- I thank the authors for their detailed response. However, the concerns on the novelty of the quantization technique and the inference efficiency of the proposed method still remain. Thus I keep my score unchanged. -------------------------------------------------------