Binarized Neural Machine Translation
Authors: Yichi Zhang
Ankush Garg
Yuan Cao
Lukasz Lew
Behrooz Ghorbani
Zhiru Zhang
Orhan Firat
Venue: NeurIPS 2023 poster (2023)
Keywords: neural network quantization
binarized transformer
machine translation
scaling law
TL;DR: We propose a novel binarization technique for Transformers applied to machine translation and show that one-bit Transformers scale and generalize well in both in-domain and out-of-domain setting.

Abstract: The rapid scaling of language models is motivating research using low-bitwidth quantization.
In this work, we propose a novel binarization technique for Transformers applied to machine translation (BMT), the first of its kind. We identify and address the problem of inflated dot-product variance when using one-bit weights and activations. Specifically, BMT leverages additional LayerNorms and residual connections to improve binarization quality. Experiments on the WMT dataset show that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16$\times$ smaller in size. One-bit activations incur varying degrees of quality drop, but mitigated by the proposed architectural changes. We further conduct a scaling law study using production-scale translation datasets, which shows that one-bit weight Transformers scale and generalize well in both in-domain and out-of-domain settings. Implementation in JAX/Flax will be open sourced.

Binarized Neural Machine Translation
Yichi Zhang" Cornell University y=2499@cornell.ed
Ankush Garg" Google DeepMind ankugargggoogle.com com
Yuan Cao Google DeepMind
yuancao@google..o com
Lukasz Lew Google Research lew@google com
Behrooz Ghorbani OpenAl ghorbani@openai.com com
Zhiru Zhang
Cornell University
zhiruz@cornell.cm edu
Orhan Firat Google DeepMind
orhanf google.com
Abstract
The rapid scaling of language models is motivating research using low-bitwidth quantization. In this work, we propose novel binarization technique for Trans- formers applied to machine translation (BMT), the first of its kind. We identify and address the problem of inflated dot-product variance when using one-bit weights and activations. Specifically, B.T leverages additional LayerNorms and residual connections to improve binarization quality. Experiments on the WMT dataset show that one-bit weight-only Transformer can achieve the same quality as float one, while being 16x smaller in size. One-bit activations incur varying degrees of quality drop, but mitigated by the proposed architectural changes. We further con- duct scaling law study using production-scaat translation datasets, which shows that one-bit weight Transformers scale and generalize well both in-domain and out-of-domain settings
Introduction
Neural language models are scaling, with the parameter count recent models, such as the GPT family, roughly increased by 10x per year [29). A scaling law study by Kaplan et al. [21] suggests that the continuous increase in model parameters is strongly correlated with performance improvement. This trend has been validated by recent successes in large-scale models, such the 540-billion parameter Pathways Language Model (PaLM), which achieves breakthrough performance on language understanding and generation [1I]. The 540-billion parameter Minerva [25] also exceeded the national average on the National Math Exam in Poland in 2021, where language models were previously far from human-level. Similarly, in the field of neural machine translation (MT), the scaling law holds, reported by Ghorbani et al. [16), with the translation quality improving as the model size increases.
The aggressive scaling trend resulted in unprecedented challenges in model serving. In particular:
The inference cost grows exponentially, The size and computational complexity of language models are increasing rapidly, with roughly a 10x increase in model size and a 100x increase in operation
Equul contribution.
Work done while Google.
'Source code is available in the init2winit library: httppp///ggtiiiooooo////////a//t haster//iiitwiiitttmmmmonn 1itxx//rmee//e/eeeeeeee.eee xformer ateee//rrrrerreaaaaateeeeeeeee wtt.ssaate.bbnarr....o.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

count per year [8]. However, the energy efficiency of hardware used to run these models is not keeping pace. Specifically, the energy required for FP32 operations has improved by only 2.5x over the past 11 years (2007-2018) 8), from 45mm to 7nm process nodes. Over the same period, DRAM access energy has only improved by 6.3x [20]. The ever-growing gap between model size inflation and inefficiency in hardware energy utility is causing inference energy to grow exponentially, which is becoming a major cost of running language models in datacenters.
The inter-chip communication overhead becomes non non-negligible. Data parallelism alone is no longer sufficient for models at such large scale since one matrix multiplication cannot fit on single accelerator chip. Each weight tensor in PaLM [T]] for example, is partitioned across 3072 TPUv4 chips in a pod. This leads to huge overhead on transferring the weights and intermediate activations across the datacenter networks.
Lateney-critical applications can now hardly benefit from parameter caching. Loading model parameters from DRAM to on-chip accelerator memory often takes a lot of time during inference. In the past, parameter caching was an effective optimization for latency because it reused model weights and avoided off-chipp memory transfers. However, evaluations on edge TPUs reported that this method works best for models with fewer than 30 million parameters [36]. For larger models, parameter caching even becomes harmful. Benefits from compiler optimizations are diminishing, and the serving latency becomes almost proportional the model parameter count. In our case, the smallest translation model has about 50 million parameters. Improving latency thus boils down to increasing memory bandwidth alone.
Quantization can significantly reduce inference cost. Binarization is an extreme case where both the weights and activations of matrix multiplication (matmul) are quantized to single bit. Compared to the Brain floating-point format (bfloat16) [+ binarization reduces the weight size by 16x, thus significantly lowering the memory and communication overhead. Moreover, a binarized matmul can be carried out by XNOR operations followed by a population count, which is estimated to be 256x more energy-efficient than the 6 counterpart [39]
Prior work shows that BERT can be binarized for pretraining 5 33 28 however, it is important to note that the BERT and MT models, which both use Transformer as their core [37], are very different. One key difference is the architecture: while an MT model has both an encoder and a decoder, BERT only has encoder. This difference can impact the quality of encoder quantization because every cross attention layer in the decoder requires outputs from the encoder. Another difference is that MT' model inference produces sequence of text, while BERT performs single text classification. This is critical because each word in the output translation sequence affects the generation of the next word. The sampling distribution of a word is therefore crucial and should be preserved after binarization, but for BERT, only the peak of the logits needs to be preserved. Due to these differences, directly applying BERT binarization techniques to MT can easily result in a lower quality model.
In this work, we investigate binarized Transformer for neural machine translation, which, to our knowledge, is the first study this topic. Each Transformer block contains attention layer and a feed-forward network (FFN). We binarize the weights and activations separately so we can study how each one affects the quality of the model. We found that binarizing weights did not significantly affect accuracy, but that traditional methods for binarizing activations led to poor performance due to activation magnitude explosion. Then, we propose a new method for activation binarization that uses a simple scaling factor and additional residual connections.
To understand the scaling behavior of the proposed 1-bit Transformer in practice, we further evaluate it on our in-house production-scae translation dataset that contains three billion sentence pairs. We for the first time demonstrate that the 1-bit weight Transformer scales and generalizes similarly well as the float one, even on the out-of-domain data. We also analyze sentences sampled from both models' outputs and find that the 1-bit Transformer generates similar translation quality as its float counterpart. Binarization can therefore be a potential candidate for future MT model serving.
2 Related Work
The success of Transformer has spurred an active body of work to quantize it to lower precision. In this section, we review subset of these efforts that inspired our approach.
In the remaining paper, "float" refers to bíloat16.

---

Transformer quantization. Much of the prior effort focused on 8-bit Transformer. Bhandare et al. 8 reported a less than 0.5 BLEU drop on the WMT14 En-De translation task with 8 bits. Prato et al. 32 showed an 8-bit Transformer preserved the translation quality. For non-generative tasks, Zafrir quantized BERT to 8-bit with marginal quality loss. When pushed down to bits, though 32 reported an 8 BLEU degradation for MT, Aji and Heafield [2] reported almost no BLEU loss by using a logarithmic quantization scheme.
et al. [38
Prato et al.
The exploration on I-bit Transformers centered around BERT. Usually binarization is directly applied and the focus is on improving the training recipe. Bai et al. [3] initiated the attempt by splitting ternary BERT into a binary one, then fine-tuning It achieved 41% average accuracy on the GLUE benchmarks. Qin al. [33] proposed distill each intermediate layer outputs from floating point model. Recently, Liu et al. [28] proposed to incrementally quantize the model, e.g., from 32-bit to 4-bit 2-bit, finally to 1-bit, and improved the GLUE accuracy to 73.5%.
Binarized vision models. Courbariaux et al. 12 pioneered the investigation on binarized deep neural nets. Recently, PokeBNN [39 established pareto SOTA on the ImageNet recognition task. We inherit the binarization functions and training recipes from PokeBNN.
Generalizability. Hooker et al. [9 show that compressed models do not generalize well on out- of-domain (OOD) data. We are particularly interested in evaluating BMT under OOD settings and analyze its generalizability.
3 Algorithm and Model Architecture
In this section, we introduce the methodology of binarizing Transformer-based MT model. We first define the binarization equations, then show that directly applying the equations to Transformer will produce inferior model quality because of the dot dot-product variance inflation. A scaling factor is then proposed as a solution to this problem, and we discuss using LayerNorm [4] to replace fixed scaling factors. Finally, we combine and present the architectural changes that are necessary to improve the binarized model quality.
3.1 Binarization Equations
We follow the approach defined in PokeBNN 39 and AQT [24), which includes an important hyperparameter "B". The function of casting floating- values into binary values
clip (I,, Tminn Tmar) min (Tmarx max (Emin, x))
Tb floor
clip
B -1 11 + e, - +)) 0.5) B
where x is the input tensor, is small floating point number that prevents overflow when taking the floor, and B the binarization bound. In the backward propagation, the floor function is ignored, i.e.. flloor(=) 8i = 1, known as the meaight-through estimator [12]. The gradient of the entire binarization function is then des [xe|-B.B] otherwise zero. The bound B therefore serves hyperparameter that controls the range of the input values that will have non-zero gradients. Note that also serves scaling factor for the outputs since the binarization function maps + (-
2- + 8 ). The bound B can also generalize to a vector, depending on the granularity of binarization. The finest granularity, however, one bound value for each dot product, i.e. per contraction dimension, that the binarized matrix multiplication can be accelerated.
For dense layer in Transformer of the form A W, where N is the input activations and Whama x de is the model weights, we instead compute a binarized matmul As W,, Throughout the experiments we apply binarization bound Bw and BA for weights and activations, respectively.
Bw max(abs(W)) (W), axis ), ; B = max(abs( (A), axis dmodel)
where axis is the dimension along which max is taken. Using one axis means the bound is per channel and per example (24] Both B- x1 and B1x lxxds are vectors that contain maximum absolute values along the contraction dimension. Note that the weight binarization bound By is static in inference though it is updated in every training iteration. The activation bound B is dynamic.
3

---

3.2 Variance Inflation in Binarization
We start by applying the binarization function to feed-forward networks (FFNs), leaving other modules as float. We observe that directly binarizing the weights preserves the model quality, but binarizing the imput activations causes the training to not converge in the context of machine translation. To understand the reason of this behavior, we analyze the variance of the dot product magnitude with and without binarization. Our analysis reveals that binarizing both weights and activations will statistically inflate the magnitude, leading to abnormal signal propagation within the neural network [O]. We present the details of this analysis as follows.
Let each weight of a dense layer be randomly initialized and sampled from zero-mean normal distribution, w (0, o Assume each input activation is independent of the weights and identically distributed as ~ (O, o²). After applying the binarization function, both wy and as are still centered at zero and have an equal probability of being either
8 or + B namely, they ? Ib B Hence the variance of a +
2
B
follow the probability mass function defined as Pr (26)
7
2
binarized multiplication is
B w Pr (w)) - 16
Var (as w/) = E ab
w - E²2 [as] E [pp] = Σ 04
Pr (as)
Σ 10
The variance of a binarized dot product is then Var (Ap W,) = En= Var, (as . wa)
B² D
T6
where D is the dimensionality of the dot product, i.e., the hidden projection dimension in FFN, and n is the index of each entry in the vector.
Following the same analysis, the variance of a poating-point dot-product is Var (A W) = =) - Note that the commonly used Xavier initializer 17 equalizes the variance of the activations across layers. o² will therefore be initialized as 1. so Var (A W) oa which is usually at the scale of 1.
Meanwhile, the common binarization bound a hidden projection dimension D = 4096 throughout the experiments. Therefore, Var (As W,) > Var (A- W). Binarization heavily inflates the dot product variance by at least 256x, which will be reflected in the magnitude of the dense layer outputs. Also note that Var (A, Wo) x D, indicating that Transformer with a larger width will potentially suffer more from the convergence issue.
E [1.3] I2
Our Transformer FFN employs
3.3 A Scaling Factor as the Solution
K
Inspired by the scaling factor vdg in the scaled dot-product attention Attention (Q, K, V) = softmax (2K") V in the
Linear
Linear
Linear
Vd.
LayerNorm
original Transformer 37,, propose scaling factor for each binarized dense layer, i.e.,
X
As Wo s
Dense (Ap)
Softmax
The scaling factor s is hyperparameter that suppresses dot- product variance inflation, while the attention layer vdl prevents the dot products from entering mmall-gradient regions of the softmax function. According to the analysis in Sec- tion 3.2 its value is estimated to be 8 x vD in order to cancel the multiplicative effect from D on the variance.
X
Linear
LayerNorm
To verify how the magnitude of the scaling factor affects the training loss, we sweep in Section In practice, 2 64 can make the training converge.
Figure 1: BMT Multi-Heaa Atten- tion - Differences from the origi- nal Transformer are highlighted (in yellow). All linear projections and einsums can be binarized.
3.4 Replacement of Scaling Factor with LayerNorm
While the scaling factor s enables the binarization of FFNs, it requires hyperparameter tuning, which can be challenging for billion-parann parimmtr translation models. To address this deficiency, we propose using layer normalization

---

(LayerNorm) 4 as drop-in replacement for the scaling factor, which has the form of LN (x) a-B/ar
(Var(z)+ "+ B, where " and /3 are learnable parameters. Besides the fact that y can incorporate
the scaling factor 8, LayerNorm also has the following advantages.
factor The factor is now dynamic and adaptive during training. The binarization function employs a dynamic bound B. so Var (Ab Wo) varies. The learnable parameter 7 in LayerNorm can better capture the changes in the dot product variance and hence properly normalize it.
LayerNorm also redistributes the input activations. It enables the binarization of a tensor with all positive values. A directly binarized FFN is FFN (A) = max (0, A,W1 + b1), W2s + b2, where W1, by and W2, b2 are the weights and biases for the first and second dense layer, respectively. One may note that the activations max (0, A 1 + b,) are all positive. The binarization function will then map the entire tensor to constant + 2 which undermines the model training. With the help LayerNorm, however, the activations are redistributed and more balanced in terms of the number of positive and negative values. This enables the normal [-1,+1} (bipolar) binarization of the second dense layer. Qin et al. [3 Liu et al.
28 used (0, 1} binarization instead in binarized BERT to overcome the issue of constant positive values. It yields a ternary matrix multiplication since A E {0. 1} NxD and W E 1-1,++1) DxK which incurs nontrivial additional overhead if computed on binary hardware accelerator. The complete proposed 1-bit FFN has the structure of
FFN (A) = LN (LN (max (0, A, + b))), W26 + b2)
When proceeding to the attention binarization, we add LayerNorm to the output of each linear projection layer for the same reasons. We verified in Section 33 that a dynamic and adaptive scaling factor in LayerNorm indeed outperformed fixed one.
3.5 Residual Connection in Attention Layers
In attention layers, we also add shortcut connection to the output linear projection layer. Combined with the additional LayerNorm, the output projection then becomes Out (A) I.N (A W) A. In BNNs, gradients of binarized layer are approximated due to the straight- thrrugh estimator. This will eventually lead the optimization into a different direction as we stack more binarized layers. Liu et al. (26] proposed adding additional residual connections in BNNs, which became a useful method for partially addressing this issue. We therefore adopt it in our model. Note that this modification is unnecessary for QKV (query, key, value) linear projections. The shortcut around the entire attention layer in the original Transform serves the same purpose. We will also demonstrate the effectiveness of the shortcut connection in the ablation study in Section
The complete modified attention architecture is shown in Figure I where we highlight the differences from the original one. The extra layer normalization and shortcut connection are both elementwise. Their overhead is small, especially comparing the benefits of binarization.
Experiments
In this section, empirically evaluate our proposed binarized Transformer MT tasks at difference scales. To investigate the impact of binarizing different layers, we first train a standard 6-layer encoder-decoder (6L6L) Transformer on the WMT2017 De-En translation dataset [9] and evaluate it on the WMT2014 De-En dataset. We then choose the 1-bit weight model variant and study its practical scaling law on in-house translation datasets. We also compare the translation qualities of both 1 -bit and float models. Throughout the experiments, the embedding table and the prediction head layer are not binarized.
4.1 WMT Results
We binarize five different matmuls in Transformer. In an attention layer there are (1) QKV linear projections; (2) aetivation-activatioo matmul between queries and keys (QK Einsum); (3) activation- activation matmul between attention scores and values (Score-V Finsum); (4) output linear projection. In an FFN there are two dense layers of the same type. To study their individual impact, we binarize their weights and activations separately. In our experiments
the following training details.
5

---

Model. We use 6L6L Transformer as the base model. Embedding dimension is 1024. Each multi-head attention layer has 16 heads, with a dimension of 1024 for QKV if combining all the heads. The hidden projection dimension in FFNs is 4096. Dropout layers has dropout rate of 0.1.
Scheduler. We adopt three-stage training scheme, where the learning rate (LR) of each stage decreases from base to zero following a cosine decay. A quantization event starts at the beginning of each stage. We first train the model float. In the second stage, all weights will be binarized. In the last stage, both weights and activations will be binarized.
Table 1: BMT results on the WMT dataset. Training uses WMT2017 De-En and evaluation uses WMT2014 De-En. Binarized activations or weights are labeled by checkmarks. Unlabeled tensors remains b6loat16. 1-bit weights models have 25MB of weight storage size while the float one has 399MB, ~ 16x compression. As a comparison, the baseline model (last row) directly applics sign function followed by
XNOR-Net style [34] binarization used in previous works 33 28 normalization. BLEU evaluation employs a beam size of 4.
ATTENTION 1-BIT
FEN 1-BIT
METRICS
AOKV
WORY
AoUT
WOUT QK SCORE-V V
APEN
WITN
VAL LOss
BLEU
FLOAT
1.39
26.35
BMT-1
1.38
25.93
BMT-2
v
1.40
25.44
BMT-3
1.51
24.11
BMT-4
1.72
21.55
BMT-5
/
1.60
21.06
BMT-6
1.89
17.87
BMT-7
1.76
18.27
BMT-8
2.81
9.42
BASE
[3
8.07
0.21
Training. We apply knowledge distillation (KD) during training. KD replaces the ground truth label in the cross-entropy loss function with the softmaxed logits from the teacher model, so it is optional for users. Adam optimizer [2 is used with 3 0.9 and 32 0.98. No weight decay applied. Batch size is 1024. Base learning rate is 0.001. The first LR cycle has 50000 steps, others have 88339 steps. We train the model with a 4x8 TPU topology.
Observations. The evaluation results on WMT2014 De-En translation dataset is shown in Table We mainly rely on the validation loss for comparing the model quality since BLEU score has higher variation 16 From the table we have the following key observations.
Weight-only binarization preserves the model loss. The float 6L6L Transformer baseline has 1.39 validation loss. In contrast, binarizing all dense layer weights (in both attention layers and FFNs) produces even lower loss (1.38, BMT-1), though the BLEU score slightly drops by about 0.4. Both metrics indicate that the 1 1-bit weight model has a similar translation quality to the float baseline. Binarization has the potential compress the model storage size by 16x while preserving the quality.
FFN binarization produces promising results. Binarizing the entire FFN, i.e., both activations and weights, while leaving other layers float, again yields similar validation loss (1.4, BMT-2) compared with the float baseline. With our proposed BMT, it is the first time on machine translation tasks that binarizing FFN activations can preserve the loss. This intriguing 1-bit FFN variant can be potentially useful for large language models. Combing with 1-bit all dense layer weights further downgrades the loss to 1.51 (BMT-3) and a 2.2 lower BLEU score in contrast to the float model. Overall, FFN binarization demonstrates a promising potential.
Attention activations are the key bottleneck high binary model quality. On top of the 1-bit weights and 1-bit FFN activation model variant, further binarizing input activations in all dense layers in the attention layer (BMT-6; this includes keys, queries, values and input activations to the output projection dense layer) leads to 1.89 loss. This is by far the largest drop in model quality. Binarizing each individual activation tensor therein leads to at least 0.3 degradation in loss (BMT-4 and 5). In addition, binarizing the two redivation-activatioo matmuls (query-key einsum operation and

---

9995 Binary
R²: R2:99.60m Fptoat 99. 19.8gtnary
1.10
Scaling Law Fit
1.5
Scaling Law Fit
Test Loss (In-Domain)
Encoder Scaling
Test Loss (Wikipedia)
Encoder Scaling
1.05
Decoder Scaling
1.4
Decoder Scaling
1.00
1.3
0.95
Binary
Binary
0.90
1.2
0.85
1.1
Float
Float
0.80
1.0
0.50 0.75 1.00 1.25 1.50 1.75 2.00
2.25
0.50
0.75
1.00 1.25 1.50 1.75 2.00
2.25 1e8
Total Params (Non Embeddings) 1e8
Total Params (Non Embeddings)
(a) [Pe,Pe} {0.18, 0.31 Men, {0.16, 0.28) Binury- In- domain evaluation.
Binar-
(b) {Pe.Pa 10, 0.25 {0.13, 0.25 Bimry- Out-of-domain evaluation.
Figure 2: Scaling law study on both in-domain and out-of-domain data - On in-domain data, scaling law fits achieve R² values of 99.5 and 99.7 float and binary models respectively. On out-of-domain data (Wikipedia), Rt2 values are 99.6 and 99.8 respectively. Scaling law fit on all the evaluation datasets, along with slopes (pe and pa) is presented in Figure 7 and Figure 8 (Appendix A
attention score-value einsum operation) are particularly challenging. The 1-bit weights model with both activation- aaationncciiatioon matmuls binarized additionally produces only 9.4 BLEU score (BMT-8). Attention layer activations are the current bottleneck to a fully binarized translation model.
4.2 Scaling Law Study
Though Section 4.1 shows promising results, an unanswered question is whether the performance degrades when binarized Transformers scaled up. Neural language model loss is known to follow a power law as its model size scales up [21], known as the "scaling law". It is widely adopted for predicting the performance of models at scale. Prior work shows 8-bit and 4-bit language models are subject to a certain scaling law, but this has not yet been established for models with 3-bits or lower [3]. We therefore conduct scaling law study both float and the proposed binarized models on our in-house translation dataset and compare their difference. Similar to Ghorbani of translation models and fit the losses using the following equation:
al. [16], we train
L(N,,, Na) = a N..
Pe
+ Tooo, where I. is the per token loss, No. Na are the number of Na
Na
Pd
encoder and decoder parameters respectively. Lex is the irreducible loss that the model attains if it has infinite capacity. Nº (Na) is the number of parameters the baseline 6L6L Transformer, which act as normalization constants for numerical stability in the curve fitting process. For tractability purposes, we examine scaling laws for only weight- binaried models. Weight-only model compression can also be leveraged for linear improvements in latency (36] and 16x improvements in memory consumption (compared to b6foat16).
Dataset. To investigate the scaling behavior of the binary models in a capacity limited regime, i.e., performance is not bound by training data, we use our large in-house parallel corpora for English to German (En - De) direction. The training set contains billion web-crawled sentence pairs. We are also particularly interested in evaluating BMT with the out-of-domain (OOD) setting and assessing its generalizability. as previous research in the image domain demonstrated that compressed models (weight pruned or quantized) have much larger quality drop on OOD data than their uncompressed counterparts, i.c., model compression amplifies brittleness [19]. As such, to have a robust evaluation of BMT, we use eleven evaluation sets, one of which is in-domain (ID) and is similarly distributed as the training set, and the rest are OOD. For ID, we sample 2000 training examples and remove them from the training data. The ten OOD evaluation sets are divided into four categories (i) Web Domain (ii) News Domain (iii) Wikipedia (iv) Patents. Furthermore, they are either "source-origing or "target-original.. The source-original datasets have a natural source side (English) while the target side (German) is human or machine translated. The target-original datasets have the natural target side (German), then back translated into source English sentences. We do this differentiation to investigate the impact of binarization on "style" of sentences since natural language exhibits rich
7

---

diversity as opposed to simple and literal (translationese) sentences 14] (More details are provided in Appendex A.
Models & Training. We train two sets of Transformers, namely, encoder-scaling and decoder-scaling models. The encoder-scaling models have fixed depth of 6 layers in the decoder while scaling up the encoder depth in sizes of {6, 8, 10, 12, 14, 18, 22, 26, 30, 36, 42, 48} layers, for a total of 12 models. Same for the decoder-scaling ones, whereby the decoder depth is scaled up in similar ways. Due to the sufficiency in training data, we did not use label smoothing during training. The binary models are trained without KD. (Appendix A.2 has more details on hyper-parametee and training).
Observations. Figure 2 compares the scaling curves of the binary and float models on both ID and OOD datasets, more in Appendix [A.3] Figure 3 compares their training vs. In-domain test loss. We make the following observations:
Binary models demonstrated similar scaling behaviors as their float counterpart for both encoder and decoder scaling. The exponent of the fitted power law for binary models in Figure 2a (pe 0.16, Pa 0.28) is only slightly below float ones (pe- 0.18, Pa 0.31), indicating the binary model loss improves fast as the parameter count increases. This trend also holds for OOD Wikipedia dataset in Figure 26 Binary models generalize just well on OOD data float models (scaling law fits on all the OOD evaluation datasets is in Appendix AA33). We also note a gap between binary and float model losses, phenomenon not observed from WMT experiments. We hypothesize that this is because the in-house production datasets are more challenging.
For the same training loss, binary and float models achieve the same generalization per- formance. As shown in Figure 3 binary and float model losses align well on straight line, and almost overlap in the 0.95 1.0 region.
1.15
Train vs Test Loss
1.10
6L, 8L Float
Test Lows (In: Domaii)
1.05
Blnary
There are no measurable differences detected
1.00
in the inductive biases of the two model classes.
Also, binary models require fewer parameter bits to achieve certain performance level. For ample, a 61.42I. binary Transformer with 195M parameters (195M bits) has a 4.3x smaller size than a 6L8L float one with 52M parameters (832M bits) while having the same loss. Such memory savings are especially advantageous when the models are deployed in resource- constrained environments (36).
0.95
Float enc scalling
0.90
Binary enc scaling
Float dec scaling
0.85
Dinary dec scaling
x=y diagonal
0.80
0.65
1.05 Training Loss
1.00
1.10
1.15
1.20
Figure 3: Training vs. ID Test loss. We observe similar linear relationship between training and test losses of all evaluation datasets.
4.3 Generation Quality
We examine the MT model generation quality in Figure 4 usiin two decoding strategies: a) Beam Search Decoding; b) Minimum Bayes Risk (MBR) decoding [23].
Beam search. Sample quality from Beam search decoding is evaluated with standard de-tokenized
BLBU scores 30 using sacreBLEU library 31
MBR. Freitag et al. 15 show that beam search decoding selects samples with high probability rather than high quality, especially for large models, as measured by human evaluations. They propose MBR- based decoding strategy defined as h MBR = arg max THunsel EyeH.mon u(h, y), where MBR is
hEW
the decoding from the model given source sentence x, Hmodel is the set of hypotheses sampled from the model p-|[z) and u is utility function that evaluates quality of hypothesis h against reference y. Freitag et al. [15] demonstrate effectiveness of the BLEURT model [35] for the utility function. BLEURT is a regression model that relies on the concatenation of hypothesis h and reference y and generates a scalar score between [0, J. measuring the hypothesis quality irrespective of the sentence structure, length word overlap with the reference. In the same way, with BLEURT as the utility function to decode sequence given the source sentence. To measure the sample quality, BLEURT(h, r) is calculated between the decoded hypothesis (AMBR) and the reference (r) for a given source sentence (I), then averaged across the evaluation set.
MBR decoding
'eaam size=4, length p.n.lty=0.6. case.mixed + numrefs.1 smooth.exp tok.

---

37
79.0
36
BLEU
BLEURT
78.0
Float 16 samples
35
Binary 24 samples
Binary 32 samples
Float enc scaling Binary enc scaling
77.0
Binary 48 samples
34
Binary 64 samples
8.6
10.
6
12
42.6
8.5
10_6
12.6
14.6
30.
Model size
Model size
(a) Beam Search BLEU
(b) MBR-BLEURT
Figure Comparison on translation qualities between binarized and float models for encoder-scaling.
(a) Beam Search Decoding: BLEU scores on In-Domain Test set (b) MBR Decoding: BLEURT
scores on In- In-Domain Test set
Observations. Figure 4a shows BLEU scores of encoder-scaling models (i.e., decoder depth=6, varying encoder depth). Figure 4b plots BLEURT scores for encoder-scaling models, where the baseline is float models using MBR decoding with 16 samples. We observe the following:
Binary models can achieve the same BLEU score float models with smaller size. Figure 4a shows that the BLEU score of binary models will consistently improve as the model size increases. Although binary models are 2-3 BLEU worse than float ones the same model depth, the 30L6L binary model achieves the same BLEU as the 8L6L float model, while being 6.7x x smaller size.
Increasing the sample size can match the generation quality of binary models with float models. In Figure 4b a larger sample size consistently produces a higher generation quality for the binary models. 4x the sample size, i.e., 64 samples, the binary model quality approximately matches float models. Besides, the BLEURT score of binary models also improves the model size increases.
5
Ablation Study
Sweep Scaling Factor Values
Impact of Model Architecture Changes
8.02
train loss valid loss
2.2
8
direct binarize
scaling factor
81
Validation Loss
2.0
layernorm
layernorm shortout
6-
1.8
4.98
4
1.6
1.4
2
1.50
1740
2
2
23
25
2
2
28
29
2.0
21
212
40000
50000
80000 Training Steps
60000
70000
90000
Scaling Factor
(a) Models losses with different scaling factor
(b) Losses of different attention out linear architectures.
Scaling factor ablation. We binarize the FFN only and sweep the scaling factor as a power of two from (equivalent to no scaling factor applied) to 4096. We plot the final training and validation losses in Figure 5a The model losses drop steeply when increasing to 64. Models with S 8 produce almost random translation quality. Large scaling factors indeed address the convergence issue. The loss begins saturated at s = 64 and is only slightly worse than the float baseline (1.39). VD.. When > 64, the model loss keeps improving
This exactly matches our expectation that slightly. We hypothesize that this because the bound is dynamic. Even a small variation on B will change the theoretical optimal by large margin since Var (As W,) B

---

BMT attention layer ablation. We only binarize the attention output projection linear layer. We train the model for 88339 steps, with binarization events started step 50000. We plot the loss curves from step 40000 in Figure 5b] Applying fixed scaling factor achieves an almost 0.2 loss improvement. This is consistent with previous observations where a scaling factor helps with convergence. The LayerNorm, as drop-in replacement for the scaling factor, not only makes the model converge to better loss, but also recovers the loss much faster after binarization. This is expected because in the LayerNorm is learnable and can better adapt the dynamic bound B as analyzed in Section 3.4 The loss almost saturates after binarization. Adding a shortcut around the output projection removes the information bottleneck. It helps the model converge to approximately the same quality the float baseline.
6
Conclusion
The proposed method enables binarization for machine translation. The simple yet effective scaling factor is the key. Binary Transformers have a similar scaling behavior of translation quality as float models. Binarization can thus be potential candidate for future model serving.
Unanswered questions: How better binarize attention einsums? Which is better for scaling up binary Transformer, depth or width? What will be a good mixed-precision scheme?
Acknowledgments and Disclosure of Funding
This work is supported in part by NSF Award #2007832.
References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Lcrge-scale machine learning on heterogeneous distributed systems. arXiv preprint PrXXiv1603.04444 2016.
[2] Alham Fikri Aji and Kenneth Heafield. Neural machine translation with 4-bit precision and beyond. arXiv preprint
2019.
[3] Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondfej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet. Angela Fan, Christian Federmann, Markus Freitag. Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah- Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Koemi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin. and Marcos Zampieri. Findings of the 2021 conference machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, pages 1-88, Online, November 2021. Association for Computational Linguistics. U htt.btttss:/aalaattooolooo org/ 1.1
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064550 2016.
[5] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv.2012.15701, 2020.
[6] Loic Barrault, Ondfej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). Association for Computational Linguisties, 2019. URL https: / ///aa/aaaannolgggee org/ 1111995300
10

---

[7] Joseph Bethge, Christian Bartz, Haojin Yang, Ying Chen, and Christoph Meinel. Meliusnet: An improved network architecture for binary neural networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1439-1448, 2021.
[8] Aishwarya Bhandare, Vamsi Sripathi, Deepthi Karkada, Vivek Menon, Sun Choi, Kushal Datta, and Vikram Saletore. Efficient 8-bit quantization of transformer neural machine language translation model. arXiv preprint (rXiv:/906.00553 2019.
[9] Ondfej Bojar, Yvette Graham, and Amir Kamran. Results of the WMT17 metries shared task. In Proceedings of the Second Conference on Machine Translation, 2017.
[10] Andrew Brock, Soham De, and Samuel L Smith. Characterizing signal propagation to close the performance gap in unnormalized resnets. arXiv preprint ar arXiv:2101.0869) 2021.
[11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint
2022.
[12] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint 1602 2016.
[13] Tim Dettmers and Luke Zettlemoyer. The for 4-bit precision: k-bit inference scaling laws. International Conference on Machine Learning, 2023.
[14] Markus Freitag, David Grangier, and Isaac Caswell. BLEU might guilty but references not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020. URL https://aclanthoologg org/ 2020. emnlp-main.5
[15] Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. High quality rather than high model probability: Minimum Bayes risk decoding with neural metries. Transactions of the Association for Computational Linguistics, 10:811-822 1-825, 2022. doi:
-_0099
[16] Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. International Conference on Learning Representations, 2021.
[17] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 446. JMLR Workshop and Conference Proceedings. 2010.
[18] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Flena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15555 2022.
[19] Sara Hooker, Aaron C. Courville, Yann N. Dauphin, and Andrea Frome. Selective brain damage: Measuring the disparate impact of model pruning. CoRR, abs/1911.05244, 2019. URL
/stp://arxive org abs//
[20] Norman P Jouppi, Doe Hyun Yoon, Matthew Ashcraft, Mark Gottscho, Thomas B Jablin, George Kurian, James Laudon, Sheng Li, Peter Ma, Xiaoyu Ma, et al. Ten lessons from three generations shaped google's 's tpuv4i: Industrial product. In 202/ ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 1-14. IEEE, 2021.
[21] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint arXiv:2001.0836 2020.
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1xXiv.141..6980, 2014.

---

[23] Shankar Kumar and William Bymee Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HL.T-NACI, 2004, pages 169-176, Boston, Massachusetts, USA, May May 2004, Association for Computational Linguistics. BtLLihttss///ccaaneeee
org/ NO4-1020
[24] Lukasz Lew, Vlad Feinberg. Shivani Agrawal, Jihwan Lee, Jonathan Malmaud, Lisa Wang, Pouya Dormiami, and Reiner Pope. Aqt: Accurate quantized training), 2022. URL http
(ggtthub.com/goggg
[25] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 220.::2220..88888. 2022.
[26] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang- Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cmns with improved representational capability and advanced training algorithm. Proceedings of the European conference on computer vision (ECCV), pages 722-737, 2018.
[27] Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng, How do adam and training strategies help bans optimization. In International Conference on Machine Learning, pages
PMLR, 2021.
[28] Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krish- namoorthi, and Yashar Mehdad. Bit: Robustly binarized multi-distilled transformer. arXiv preprint arXiv:2205. 13016, 2022.
[29] DEEPAK Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, and Bryan Catanzaro. Scaling language model training to a trillion parameters using megatron, 2021.
[30] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic
evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002.
Association for Computational Linguistics. doi: 10.31 (0.3115/10730331107311 URL https
//aclanthology.ort .rg. -002000000000/00000000000 1040
[31]
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels, October 2018 Association for Computational Linguistics. URL httpt::////wtt aclweb...wbb... org/ /
W18-6319
[32] Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. Fully quantized transformer for machine translation. arXiv preprint arXiv: /000//5500 2019.
[33] Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. Bibert: Accurate fully binarized bert. arXiv preprint arXiv:2203.0m 06399, 2022.
[34] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Computer Vision-ECCe 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, pages 525-542. Springer, 2016.
[35] Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18655/w122020.acc-m mann URLLatts://acclathoolgg o02/ 2020. acl aai-main
704
[36] Kiran Seshadri, Berkin Akin, James Laudon, Ravi Narayanaswami, and Amir Yazdanbakhsh. An evaluation of edge tpu accelerators for convolutional neural networks. arXiv preprint 2rXiv:2102.10423, 2021.
12

---

[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[38] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-Neune -NeurlPS Edition (EMC2-NIPS), pages 36-39. IEEE, 2019.
[39] Yichi Zhang, Zhiru Zhang. and Lukasz Lew. Pokebnn: A binary pursuit of lightweight accuracy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475-12485, 2022.
13

---

A Scaling Law Study Details
A.1 Dataset
concise view of evaluation datasets used for scaling laws (Section 4.2) shown in Table. The ten OOD evaluation datsets span four categories (i) Web Domain (ïi) News Domain (iii) Wikipedia (iv) Patents.
There are two source-original and one target-origiall dataset in Web Domain, one source-original each in Wikipedia and Patents domain. We use publicly available WMT newstest2019 [ and WMT newstes12021 [3] for News Domain. Within this domain, have five datasets: source-original, target-original, ooiuiiall paaaiinna--ppppaaeeeeeeee [[4] and waweee-rrgiialiigh---oee newstest2021
They are either "source-ores "ooureeoriginal" or "target-original
[4] from WMT newstest2019 [6] and wmt-reference- from WMT
Dataset Name
Domain
Type
Source
Train Subset
Web
mixed
In-house
Patents
Patents
mixed
In-house
Web domain 1
Web
source-original
In-house
Web domain 2
Web
source-original
In-house
Web domain 3
Web
target-original
In-house
Wikipedia
Wikipedia
source-origing
In-house
wmt-high-quality
News
source-original
WMT newstest2019
14
wmt-reiC
News
source-original
WMT newstest2021 Ref-C
war-paraphrased
News
source-original
WMT newstest2019
[4
wmt-sre-orig
News
source-original
WMT newstest2019
wmt-tgt-orig
News
target-original
WMT newstest2019
Table Hvaluation datasets used in Section 4.2
A.2 Model Training Details
All the models in Section 4.2) have an embedding dimension of 512, hidden projection dimension of 2048, and 8 attention heads. The embedding parameters are shared the source and the target side. The same embedding matrix (transposed) is also used for the linear readout (softmaxx parameters on the decoder side. All models are trained with Adam optimizer [2 and use cosine learning rate schedule. Due to the sufficiency in training data, we did not use label smoothing during training. In our experiments, enabling label smoothing resulted poor development set performance across all the models. Training and Learning rate profiles of one model (6 encoder, 8 decoder layers) are shown in Figure Float models are trained for epochs, and binary models are trained for epochs in two stages: float stage and a binarization stage. An independent but identical learning rate schedules are used (with warmup) in both the stages of the binary model training. We note that significant amount of training (i.c. loss reduction) for binary models happens in the final 10 steps when the learning rate extremely small. Raw values last 15 steps of lcarning rates are [5.0e-7, 3.1e-7, 1.6e-7, 6.4e-7, 1.0e-8, (2.5e-15)x10] We also tune binary models with constant learning rate of values (le-8, le-11, 1e-15) for the last epoch (overriding the original schedule), however observe degradation in the quality (loss plateaus). This phenomenon of significant learning in the final stages of binary models' training at extremely small learning rates is also observed by Liu et al. 27). Zhang al. [39 work.
We leave further investigation of this behavior to future
A.3 Scaling Law Fit
Scaling law fit on all ten OOD evaluation datasets is shown in Figure 7 The slopes pe and Pa are shown in Figure g and Table
B Generation Quality
Generation quality for decoder-scaling models shown in Figure We observe similar behavior as seen for encoder-scaling models in Section 4.3 BLEU scores for binary models are 2-3 BLEU points
14

---

2.000
0.0010
Test Loss (In- (In-Domain)
Binary Model
Binary Model
0.0008
1.637
Learning Rate
0.0006
1.275
0.0004
binary
1:210
0.913
0.0002
1.182
0.550
1.164
Last 10
0.0000
0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00
0.00 0.25 0.50 0.75 1.00 1.25
1.50 2.00 1e6
Training Step
1e6
Training Step
(a) Loss
(b) Learning Rate
Figure Test loss and learning rate profiles of a 6L8L float and binary model as the training progresses.
Dataset
Float models
Binary models
Pe
Pa
Pe
Pa
Train Subset
0.18
0.31
0.16
0.28
Patents
0.20
0.30
0.19
0.32
Web Domain
0.14
0.25
0.14
0.27
Web Domain 2
0.19
0.37
0.16
0.30
Web Domain
0.12
0.18
0.14
0.23
Wikipedia
0.13
0.25
0.12
0.25
wmt-bigh-quality
0.20
0.31
0.18
0.30
wmt-refC
0.24
0.34
0.17
0.27
wat-paraphrased
0.14
0.36
0.12
0.31
wmt-src-orig
0.22
0.37
0.23
0.36
wmt-igt-orig
0.15
0.22
0.12
0.20
Table 3: Tabular representation of the same data (Pe & pa) as shown in Figure
worse than the respective float models at the same model depth. MBR-BAL BBR-BLEURT based decoding quality increases consistently by increasing the sample size.
B.1
Translation Samples
We generate several En On-De translation samples as follows from both float and binary 6L10L encoder- decoder models. Inputs are taken from the WMT2017 dataset.
Example 1
The notice for the Nottingham East Labour meeting on Friday stated that "we want the meetings be inclusive and productive."
Source:
Reference:
In der Mitteilung für das East Labour in Nottingham Treffen am Freitag heißt "Wir wollen, dass die Treffen integrativ und produktiv sind".
Float output: der Mitteilung für das Nottingham East Labour-Treffen am Freitag heißt es: "Wir wollen, dass die Treffen inklusive und produktiv sind."
Binary output: Die Bekanntmachung für das Nottingham East Labour-Treffe Freitag erklärte: "Wir wollen, dass die Sitzungen inklusive und produktiv sind."
Example 2
Source: The Government of Wales Act 2017 gave the Welsh assembly the power to change its name.
15

---

Reference: Mit dem Government of Wales Act 2017 erhielt das walisische Parlament die Möglichkeit, seinen Namen zu ändern.
Float output: Der Government of Wales Act 2017 gab der walisischen Versammlung die Befugnis, ihren Namen zu ändern.
Binary output: Das Government of Wales Act 2017 gab der walisischen Versammlung die Befugnis, ihren Namen zu ändern.
Example 3
returning their destroyed homes, picking through water- logged belongings, trying to salvage anything they could find.
Source: Residents
Reference: Anwohner wurden dabei beobachtet, wie sie ihre zerstörten Häuser zurück- kehrten, völlig durchnässte persönliche Gegenstände mitnahmen und versuchten retten, was zu retten ist.
Float output: Die Bewohner wurden gesehen, wie sie ihre zerstörten Häuser zurück- kehrten, durch verstopfte Habseligkeiten pflückkten und versuchten, alles zu retten, was sie finden konnten.
Binary output: Die Bewohner wurden gesehen, wie in ihre zerstörten Häuser zurück- kehrten, indem sie mit Wasser gefüllte Gegenstände pflückten und versuchten, alles zu retten, was finden konnten.
16

---

R 28. 99.00/oat 99.8ginary
: 22:99.44rss 4ytee. ginagy
Scaling Law Hit
Scaling Law Hit
0.70
Encoder Scaling
Test Loss (Web Domain
1.15
Encoder Scaling
Test Loss (Patents)
Decoder Scaling
1.10
Docoder Scaling
0.65
1.05
Binary
Binary
0.60
1.00
0.95
0.55
0.90
Float
Float
0.85
0.50
0.75 1.00 1.25 1.50 1.75 2.00 2.25
0.50
0.75
1.00
1.25 1.50 1.75 2.00 2.25
Total Parameters (Excluding Embeddings)1e@
Total Parameters (Excluding Embeddings)
R²: 99.7 Ficat. 99. 99nagy
R 99.8 Emary
1.10
Test Loss (Web Domain 2)
Scaling Law
Scaling Law Fit
1.10
1.05
Encoder Scaling
Test Loss (Web Domain
Encoder Scaling
Decoder Scaling
1.05
Decoder Scaling
1.00
1.00
0.95
Binary
0.95
Binary
0.90
0.90
0.85
0.85
0.80
Float
0.80
Float
0.50
0.75
1.00 1.25 1.50 1.75 2.00 2.25
0.50
0.75
1.00 1.25 1.50 1.75 2.00 2.25
Total Parameters (Excluding Embeddings/1e8
Total Parameters (Excluding Embeddings)
R2: 19.9999... 99.66ffar- 99. 99.8ginary
R2:99 99..7 Ftcee. 99.8gmnary
1.5
1.2
Scaling
Test Loss (Wikipedia)
Encoder Staling
Test Loss twmt-high-quat
Encoder Scaling
1.4
Decoder Scaling
Decoder Scaling
1.3
Binary
1.0
Binary
1.2
1.1
0.9
Float
Float
1.0
0.50
0.75
1.25 1.50 1.75 2.00 2.25 1.00
0.50
1.00
1.25
1.50
1.75 2.00 2.25
Total Parameters (Excluding Embeddings)
Total Parameters (Excluding Embeddings)
R2 99.6ficat> 99. ginar
99.8gmary
Test Loss (wmt paraphrased]
1.15
Encoder Scaling
Scaling Law Fit
2.1
Scaling Law Fit
Encoder Scaling
Test Loss (wmt-refe
1.10
Decoder Scaling
2.0
Decoder Scaling
1.9
1.00
Binary
Binary
1.8
0.95
1.7
0.90
0.85
1.6
Float
Float
0.80
1.5
0.50
0.75
1.00 1.25 1.50 1.75 2.00 2.25
0.50
0.75
1.00 1.25 1.50 1.75 2.00 2.25
Total Parameters (Excluding Embeddings)led
Parameters (Excluding Embeddings)
R 8 99. 8ginary
R² 79.99..7cc F 99.8 99.8mm
Test Loss (wmt-src-on w
1.05
Scaling Law Fit
1.3
Scaling Law Fit
Encoder Scaling
Test Loss (wmt-tgt-orig)
Encoder Scaling
1.00
Decoder Scaling
Decoder Scaling
0.95
0.90
1.1
Binary
Binary
0.85
1.0
0.80
0.75
Float
0.9
Float
0.70
0.50
0.75
1.00
1.25
1.50
1.75
2.00
2.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00 2.25
Total Parameters (Excluding Embeddings)1e8
Total Parameters (Excluding Embeddings)
Figure 7: Scaling law studies on evaluation datasets defined in Section
4.2
17

---

Float models
Binary models
0.40
0.35
0.35
0.30
0.30
8 0.25
0.25
orig
orig patents
0.20
orig
0.20
Patents
0.15
Train subset
0.15
Dev set X line
x line
0.10
0.10
0.10
0.12
0.15
0.18
0.20
0.23 0.25 0.28 0.30
0.10
0.12 0.15
0.18
0.20
0.23 0.25 0.28 0.30 Pe
Pe
Figure 8: Encoder and Decoder scaling slopes (i.e. Pe pa) as per the scaling law defined in Section
Raw values shown in Table
37.5
37.0
36.5
BLEURT scores
36.0
35.5
Float samples
35.0
Binary samples
Binary samples
Float enc scaling Binary enc scaling
Binary 48 samples
34.0
Binary samples
33.5
1.10
12
12
6.14
18
8.1
Model size
Model size
(a) Beam Search BLEU
(b) MBR-BLEURT
Figure 9: Comparison on translation qualities between binarized and bíloat16 models for decoder- scaling.
18


Reviews:
[{"review_id": "3yDLtvFEIV", "reviewer": "Reviewer_arnv", "summary": "This work proposes to binarize matrix multiplication for significantly saving memory and, thus, reducing latencies at inference time that is crucial for serving encoder-decoder model. Basic idea is to employ a binary variant of weights and inputs with scaling parameters in feed-forward and multi-head attention computation. Since the scaling hyparameters are critical for the binarized model, this work employs layer normalization to alleviate the issue so that appropriate normalization is performed automatically. Experiments are carried out mainly for WMT en-de by comparing against float variants with different number of parameters.", "questions": "* There exists almost no clear description about how binarization is applied in section 3.5. Given Figure 1, it sounds like layer norm is shared across query, key and value, and it is not clear what is the motivation of sharing.", "limitations": "Better to discuss the current limitation on the experiments, e.g., scale and variances.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Although the idea of binarizing matrix multiplication is now new, this work has a couple of contributions to binarize an encoder-decoder architecture. For example, the use of layer normalization sounds good to me given the stability analysis for scaling hyperparameters.\n\n* Binarization is performed not only feed-foward, but also multi-head attention to reduce the computation. For stability, residual connection is introduced in the model following the prior work, but the choice is carefully designed.\n\n* Experiments are systematically carried out by varying the number of parameters and the proposed method is compared with the float variants.", "weaknesses": "* Although this work has comparisons in terms of loss and translation qualities measured by, e.g., BLEU, this work is not presenting actual speed measured by seconds. I understand the condition might be varied, it is better to run experiments to see whether the proposed method is actually faster than a float baseline. \n\n* Discussion in residual connection is a bit weak, and only a figure is presented. I feel better to show by equations to avoid any confusions. Also, further analysis on why residual connection is needed in the output projection will be a plus for this submission.\n\n* Given the binarization, it might be prone to high variance in experimental results. It would be good to run some analysis by running multiple times and show averages/variances.\n\n* Translation qualities are measured only by BLEU. Given the limitations of the metric, better to present alternative metrics as well, e.g., BLEURT or COMET.\n\n* No experiments for larger data. It is minor, though, better to run larger data to see if the proposed method will also work or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: I understand the condition might be varied, it is better to measure the speedup of the proposed method.**\n\nThank you for commenting on the speedup measurement. It is not accessible because we are not aware of an ecosystem (accelerator combined with software stack) that supports such measurement. However, there is convincing evidence that 1-bit matmuls will have high performance. For example, NVIDIA’s A100 architecture [1] shows 1-bit matmul is 8x faster than 8-bit measured by TFLOPS throughput, though requiring NVIDIA’s own assembly. Also [2] shows binary matmul is 9x-12x faster compared to 8-bit measured by latency on ARM CPUs. In that light, among other goals, our work aims to contribute to the assessment of whether binary ML hardware+software is a viable future direction for the ML accelerator industry.\n\n[1] NVIDIA Ampere Architecture Whitepaper. Table 3.\n\n[2] Tom B., et al., Larq Compute Engine: Design, Benchmark, and Deploy State-of-the-Art Binarized Neural Networks, MLSys’21.\n\n**Q2: Better to show equations on the residual connection and an analysis on why it is needed in the output projection.**\n\nThanks for the suggestion. We will add an equation on residual connection in Section 3.5: Out = LN(X*W) + X, where X is the output of score-value einsum. As briefly discussed in Section 3.5, we added the residual connection because the binarization of the output projection layer will possibly mislead the optimizer since the gradients through a binarization function are computed by the straight-through estimator. The shortcut link will carry the raw gradient from the previous layer, therefore partially mitigate this issue. We also conducted an ablation study in Section 5(b) to demonstrate the effectiveness of the proposed shortcut.\n\n**Q3: Would be good to show averages/variances of results.**\n\nWe were able to reproduce the model loss ourselves within several runs, but since we need to train each model size for one million steps on three billion sentence pairs and we have limited hardware resources, the number of losses for each result is not sufficient to establish statistical analysis. However, given the dataset and model size, we expect that the variance on experimental results should be small. We will attempt to add such analysis in the final version.\n\n**Q4: Better to present alternative metrics as well, e.g., BLEURT or COMET.**\n\nWe measured and reported BLEURT in Section 4.3. In that section we compared the model generation quality using both BLEURT and BLEU on our in-house translation dataset. We show that BLEURT scores also improve as we scale up the model size. Additionally, we run MBR decoding with BLEURT metric and show that given enough MBR samples, binary models match the quality of float models. We will highlight it better in the paper.\n\n**Q5: No experiments for larger data. It is minor, though, better to run larger data to see if the proposed method will also work or not.**\n\nThe scaling law study in Section 4.2 and model generation quality study in Section 4.3 are both carried out on our in-house large production-scale translation dataset. This dataset has 3 billion En-De sentence pairs and is one of the largest translation datasets in the ML community.\n\n**Q6: How binarization is applied in Section 3.5? Why is layernorm shared across QKV in Figure 1?**\n\nWe will clarify this in the paper. Binarization will be applied via casting the inputs right before a matmul to 1-bit using the function defined in line 106. Naming it as x_b= bin(x) for short, a binarized linear layer A*W will be computed as bin(A) * bin(W). \nEach of the QKV projections has its own independent layernorm, i.e., the parameters in layernorm are not shared. Figure 1 draws a big rectangle to represent the layernorms because in practical implementations the QKV projections are usually combined into a large single matmul. Thanks for pointing it out, and we will add corresponding captions to reflect this.\n\n**Q7: Better to discuss the current limitation on the experiments.**\n\nWe will make the limitations more clear. (1) As stated in line 237, the activation-activation matmul binarization quality is still not ideal. We list 8 BMT variants in Table 1, among which BMT-8 (the one with activation-activation matmul binarization) has the largest loss and BLEU drop. We imagine a lot of the future effort will be dedicated to it. (2) As stated in the last part of the conclusion, there are several unanswered questions in this work, for example, shall we scale up the depth or width for the binarized Transformer and how should we design a mixed-precision scheme using binarization and potentially other formats? We will expand these discussions in the next revision."}, {"review_id": "HLcpvWZHvP", "reviewer": "Reviewer_jHYb", "summary": "This paper presents a binarized neural translation model based on an encoder-decoder structure. The proposed method initially analyzes the challenges associated with binarized encoder-decoder models. The primary challenges arise from the significant impact of binarizing both weights and activations on result variance. Therefore, the authors primarily employ two methods to control variance: incorporating a scaling weight and adding a layer normalization layer.\n\nTo assess the effectiveness of the method, the authors report promising results obtained from experiments. The results indicate that the bottleneck stems from the attention activations. Binarizing different parts demonstrates a considerable variance in performance. However, with careful consideration of the binarization position, the proposed model achieves competitive results while significantly reducing its size. The ablation study further demonstrates the effectiveness of the proposed method in mitigating variance.", "questions": "In Table 1, it is unclear which row represents the result of the BMT model with a size of 25MB. Does it imply that all rows correspond to models of approximately 25MB? Each row applies binarization to different weights and activations, resulting in significant performance variance across various settings.", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation behind the method is clearly defined. The method begins by analyzing the reasons behind the failure of directly binarizing the weights. It was discovered that this failure can be attributed to a variance problem where binarization statistically inflates the magnitude, resulting in abnormal signal propagation within the neural network. To address this issue, the authors implemented two widely-used solutions: scaling weight and layer normalization. By employing these approaches, the authors were able to develop a binarized machine translation model that yields competitive results.\n\nFurthermore, the authors conducted scaling law experiments, revealing that binarized models also exhibit scaling law characteristics. Despite the impact of binarization on model performance, the gap can be bridged by increasing the model size. Due to the significant reduction in model size achieved through binarization, the proposed model can effectively deliver superior performance with a smaller model size.", "weaknesses": "The main idea contribution of this work revolves around identifying and addressing the challenge of variance in binarizing machine translation models. However, the utilization of layer normalization and scaling weights in a straightforward manner somewhat limits its novelty and originality.\n\n\nIt appears that there is a lack of comparison with 8-bit or 4-bit results. Are these methods also subject to the scaling law, where larger models yield better performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Are 8-bit or 4-bit subject to the scaling law?**\n\nThanks for pointing out this comparison and we will add it to the scaling law section. 8-bit and 4-bit models are studied more often and they do exhibit a scaling law where larger models yield better performance [1]. However, in the previous study the scaling law breaks starting from 3 bits where models seem not to converge. In our study we are able to re-establish the scaling law even for 1-bit models. We show that the dot-product variance is the key and a simple scaling factor can be a remedy. We also show in the ablation section 5 that indeed without a scaling factor a 1-bit model cannot converge but it can if with.\n\n[1] Tim D. et al., The Case for 4-bit Precision: k-bit Inference Scaling Laws, arxiv 2023.\n\n**Q2: In Table 1, it is unclear which BMT variant has a size of 25MB. Does it imply that all rows correspond to models of approximately 25MB?**\n\nAs indicated by the caption, models with 1-bit weights will have 25MB of size. It means a model whose W_QKV, W_out, W_FFN are all labeled by checkmarks, indicating they are binarized, has a model size of 25MB. In Table 1, all BMT variants except for BMT-2 have 25MB. BMT-2 has only FFN binarized, which is also a potentially useful special variant that we want to demonstrate."}, {"review_id": "T5ZrZ4KJgY", "reviewer": "Reviewer_HM3D", "summary": "The authors introduce a new technique for binarization in Transformers that can be applied to machine translation known as Binarized Neural Machine Translation (BMT). They have adapted the binarization functions and training methods from PokeBNN to help address the \"inflated dot-product variance\" issues that arise when using one-bit weights and activations. The authors propose the use of LayerNorm in place of fixed scaling factors and make some architectural changes to improve the quality of the binarized model. Experiments show the BMT have the ability to scale and generalize effectively in both in-domain and out-of-domain settings.\n\n", "questions": "1. The authors claim that \"Experiments on the WMT dataset show that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16× smaller in size.\" Could you please clarify how the model size reduction by a factor of 16 is defined here? Is it just a reduction in storage size of the model weights?\n2. In lines 57-60, the authors mention that \"each word in the output translation sequence affects the generation of the next word\", so what is the quality impact of binary quantization on long text generation?\n\n\n\n", "limitations": "The authors did not state any relevant limitations of the method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The analysis of Variance Inflation in Binarization in Section 3.2 is interesting. The BERT model should also have this problem. What is the difference between it and the Transformer structure?\n2. The paper provides a comprehensive experimental evaluation of the proposed BMT model on a 3-billion in-house parallel corpus. The authors make detailed analysis on the scaling law study and demonstrate the binary models can achieve the same BLEU score as float models with a smaller size.", "weaknesses": "1. Pre-LayerNormalization Transformer has already been proposed for a few years, and what's the difference with the Section 3.4 Replacement of Scaling Factor with LayerNorm?\n\"On Layer Normalization in the Transformer Architecture\"\n2. There is a lack of comparison with other quantization methods for Transformer models.\n3. Based on the experimental results of in-house training data in Section 4.3, BMT still has about 2 BLEU gap compared to the floating-point model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: What’s the difference between the proposed layernorm in Section 3.4 compared to the existing pre-layernorm Transformer?**\n\nNote that pre- or post-layernorm for the Transformer architecture indicates the layernorm position **outside** of the entire FFN module. Whereas in our proposal, each linear layer **inside** the FFN module needs to be followed by another layernorm (scaling factor). They are extra layernorms and only needed for 1-bit models to address the dot-product variance inflation.\n\n**Q2: There is a lack of comparison with other quantization methods for Transformer models.**\n\nWe compared other binarization methods for Transformer in Table 1, last row (labeled as “Base”). We outlined the literature on transformer quantization in Section 2 beginning line 79. Many previous works focused on 8-bit and 4-bit quantization. Only a few of them focused on binarization. Among the binarization works, the model is BERT and the binarization function is directly applied with additional distillation methods. We adopted their quantization method to our encoder-decoder Transformer and listed the result in Table 1 in the last row.\n\n**Q3: Based on the experimental results of in-house training data in Section 4.3, BMT still has about 2 BLEU gap compared to the floating-point model.**\n\nYes, that's correct. We concluded that in Section 4.3 line 332. There is still room to improve BMT when the dataset is comprehensive. One way is, as the scaling law suggests, scaling up the binarized model. As stated in line 333, the 30L6L binary model achieves the same BLEU score as the 8L6L float model while being still 6.7x smaller in model size.\n\n**Q4: Could you please clarify how the model size reduction by a factor of 16 is defined? Is it just a reduction in storage size of the model weights?**\n\nYes, the model size reduction is defined by the reduction in the amount of bits used to store the weights, which is very important for model serving as highlighted in the challenges in Section 1 from line 24. Currently, model weights are typically stored as float16 or bfloat16 for efficiency reasons. Our technique allows the storage as 1 bit per weight, achieving a 16x compression. We will clarify this in the paper as well, thank you.\n\n**Q5: What is the quality impact of binary quantization on long text generation?**\n\nWe analyzed the impact of binarization on translation generation quality in Section 4.3. Many of the translation sentences have dozens of tokens and some even longer. Some samples can be seen in Appendix B.1. We synthesized the knowledge learnt from there: (1) there will be around 2 BLEU quality loss at the same model size; (2) the quality loss can be recovered by scaling up the binary model a bit or generating more samples when selecting the translation text.\n\nAdditionally, we study scaling law behavior on several in-house and open-source datasets (Appendix, Table 2). These datasets are a mix of different sequence lengths, ranging from short to long range sequences. From Figure 7 and 8 (appendix), we don't observe any outstanding difference in the slopes (p_e, p_d) of the scaling law on any dataset. This denotes that binary and float models have a similar scaling behavior in various scenarios with different sequence lengths, domains, naturalness etc.\n\n**Q6: The authors did not state relevant limitations of the method.**\n\nWe will make the limitations more clear. (1) As stated in line 237, the activation-activation matmul binarization quality is still not ideal. We list 8 BMT variants in Table 1, among which BMT-8 (the one with activation-activation matmul binarization) has the largest loss and BLEU drop. We imagine a lot of the future effort will be dedicated to it. (2) As stated in the last part of the conclusion, there are several unanswered questions in this work, for example, shall we scale up the depth or width for the binarized Transformer and how should we design a mixed-precision scheme using binarization and potentially other formats? We will expand these discussions in the next revision."}, {"review_id": "o0KBbqsHLL", "reviewer": "Reviewer_B1hS", "summary": "The paper proposes a novel quantization scheme to binarize transformer machine translation models.  The method consists of inserting additional layer normalization for activations and also additional residual connections.  The authors demonstrate good results on the WMT test set especially for weight-only binarization, and promising results for both weight and activation binarization.", "questions": "none", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written and the method description is precise.\n- Promising results in the fully binarized setting, which is difficult to obtain\n- One of the few papers attempting binarization for natural language generation\n- Great scaling law study of binary models on a large training set - which is perhaps the most interesting part of the paper", "weaknesses": "- Binarized activation results are still quite poor.\n- Main results only test on one benchmark (WMT17 en-de)\n\nAlthough it is contemporary, please consider comparing to the following work:\nhttps://arxiv.org/abs/2306.01841\nHaving some strong established baseline could make the conclusions of the work stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Binarized activation results are still poor.**\n\nYes, as highlighted in line 237 in Section 4.1, one challenge we identified in this work is, more precisely, that the attention layer activations are the bottleneck to a high-quality binarized Transformer for a sequence generation task. We hope our analysis will shed light on the future improvement to it.\n\n**Q2: Main results only test on one benchmark (WMT17 en-de)**\n\nDespite the WMT17 en-de dataset, we also train the model using our in-house translation corpus that contains 3-billion web-crawled sentence pairs in Section 4.2 and evaluate on both in-domain and out-of-domain datasets that span multiple categories. The detailed train and evaluation dataset information is provided in Appendix A.1. More evaluation results can be found in Appendix Figure 7, 8, and 9. In the main paper we put the key results and distill the knowledge we learnt.\n\n**Q3: Although it is contemporary, please consider comparing to the suggested work.**\n\nThanks for sharing this very valuable contemporary work. Since it uses a different dataset than ours, we need to re-evaluate its approach and compare. Though we cannot finish the comparison during rebuttal given our dataset size, we will definitely cite it in related works."}, {"review_id": "0Gw6YnMEsy", "reviewer": "Reviewer_BTaK", "summary": "They propose a novel binarization technique for Trans3 formers applied to machine translation (BMT), the first of its kind. They identify and\naddress the problem of inflated dot-product variance when using one-bit weights and activations. Specifically, BMT leverages additional LayerNorms and residual connections to improve binarization quality. Experiments on the WMT dataset show that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16× smaller in size. They further conduct a scaling law study using production-scale translation datasets, which shows that one-bit weight Transformers scale and generalize well in both in-domain and\nout-of-domain settings. ", "questions": "See above", "limitations": "More discussion about  the limitations should be presented.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1 A novel binarized NMT model is proposed, which may be useful for the production server.\n2 The proposed scaling factor to mitigate the activation variance is simple and effective. And the whole model architecture looks convincing. \n3 The experimental results are very solid. ", "weaknesses": "1 How the training and inference efficiency change compared to the float model is not discussed. \n2 The proposed code should include the readme, which can help the reviewer to run and check the main results.\n2 typo error, line 348", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: How the training and inference efficiency change compared to the float model is not discussed.**\n\nThanks for commenting on the efficiency side. It is not accessible because we are not aware of an ecosystem (accelerator combined with software stack) that supports such measurement for 1-bit models. However, there is convincing evidence that 1-bit matmuls will have high performance. For example, NVIDIA’s A100 architecture [1] shows 1-bit matmul is 8x faster than 8-bit measured by TFLOPS throughput, though requiring NVIDIA’s own assembly. Also [2] shows binary matmul is 9x-12x faster compared to 8-bit measured by latency on ARM CPUs. In that light, among other goals, our work is aiming to contribute to the assessment of whether binary ML hardware+software is a viable future direction for the ML accelerator industry.\n\n[1] NVIDIA Ampere Architecture Whitepaper. Table 3.\n\n[2] Tom B., et al., Larq Compute Engine: Design, Benchmark, and Deploy State-of-the-Art Binarized Neural Networks, MLSys’21.\n\n**Q2: The proposed code should include the readme, which can help the reviewer to run and check the main results**\n\nWe really appreciate the reviewer checking out our software design, and we will write a good readme indeed. As promised in the abstract (line 12), we will open-source the code together with a detailed reproduction instruction upon acceptance.\n\n**Q3: typo error, line 348**\n\nThanks for pointing it out. We will remove the redundant word “the”.\n\n**Q4: More discussion about the limitations should be presented.**\n\nWe will make the limitations more clear. (1) As stated in line 237, the activation-activation matmul binarization quality is still not ideal. We list 8 BMT variants in Table 1, among which BMT-8 (the one with activation-activation matmul binarization) has the largest loss and BLEU drop. We imagine a lot of the future effort will be dedicated to it. (2) As stated in the last part of the conclusion, there are several unanswered questions in this work, for example, shall we scale up the depth or width for the binarized Transformer and how should we design a mixed-precision scheme using binarization and potentially other formats? We will expand these discussions in the next revision."}]

Author rebuttal:
We thank all reviewers for their positive feedback, considering our problem analysis and empirical experiments as a good contribution to the community. We also appreciate all comments and suggestions. We will address questions below in separate threads.

Meta-review: The paper proposes a novel quantization scheme to binarize transformer machine translation models. The method includes inserting additional layer normalization for activations and additional residual connections. The authors demonstrate good results on the WMT test set especially for weight-only binarization, and promising results for both weight and activation binarization.

After the rebuttal, the concerns of reviewers are well-addressed.
Decision: Accept (poster)