SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FFN binarization produces promising results. Binarizing the entire FFN, i.e., both activations and weights, while leaving other layers float, again yields a similar validation loss (1.4, BMT-2) compared with the float baseline. With our proposed BMT, it is the first time on machine translation tasks that binarizing FFN activations can preserve the loss. This intriguing 1-bit FFN variant can be potentially useful for large language models. Combining with 1-bit all dense layer weights further downgrades the loss to 1.51 (BMT-3) and a 2.2 lower BLEU score in contrast to the float model. Overall, FFN binarization demonstrates a promising potential.
Attention activations are the key bottleneck to high binary model quality. On top of the 1-bit weights and 1-bit FFN activation model variant, further binarizing input activations in all dense layers in the attention layer (BMT-6; this includes keys, queries, values and input activations to the output projection dense layer) leads to a 1.89 loss. This is by far the largest drop in model quality. Binarizing each individual activation tensor therein leads to at least 0.3 degradation in loss (BMT-4 and 5). In addition, binarizing the two activation-activation matmuls (query-key einsum operation and
Figure 2: Scaling law study on both in-domain and out-of-domain data — On in-domain data, scaling law fits achieve  $R^2$  values of 99.5 and 99.7 on float and binary models respectively. On out-of-domain data (Wikipedia),  $R^2$  values are 99.6 and 99.8 respectively. Scaling law fit on all the evaluation datasets, along with slopes ( $p_e$  and  $p_d$ ) is presented in Figure 7 and Figure 8 (Appendix A).
attention score-value einsum operation) are particularly challenging. The 1-bit weights model with both activation-activation matmuls binarized additionally produces only 9.4 BLEU score (BMT-8). Attention layer activations are the current bottleneck to a fully binarized translation model.
4.2 Scaling Law Study
Though Section 4.1 shows promising results, an unanswered question is whether the performance degrades when binarized Transformers are scaled up. Neural language model loss is known to follow a power law as its model size scales up [21], known as the "scaling law". It is widely adopted for predicting the performance of models at scale. Prior work shows 8-bit and 4-bit language models are subject to a certain scaling law, but this has not yet been established for models with 3-bits or lower [13]. We therefore conduct a scaling law study on both float and the proposed binarized models on our in-house translation dataset and compare their difference. Similar to Ghorbani et al. [16], we train a set of translation models and fit the losses using the following equation:  $L(N_{e},N_{d}) = \alpha \left(\frac{\bar{N}_{e}}{N_{e}}\right)^{p_{e}}\left(\frac{\bar{N}_{d}}{N_{d}}\right)^{p_{d}} + L_{\infty}$ , where  $L$  is the per token loss,  $N_{e}$ ,  $N_{d}$  are the number of encoder and decoder parameters respectively.  $L_{\infty}$  is the irreducible loss that the model attains if it has infinite capacity.  $\bar{N}_{e}(\bar{N}_{d})$  is the number of parameters in the baseline 6L6L Transformer, which act as normalization constants for numerical stability in the curve fitting process. For tractability purposes, we examine scaling laws for only weight-binarized models. Weight-only model compression can also be leveraged for linear improvements in latency [36] and  $16\times$  improvements in memory consumption (compared to blfoat16).
Dataset. To investigate the scaling behavior of the binary models in a capacity limited regime, i.e., performance is not bound by training data, we use our large in-house parallel corpora for English to German (En → De) direction. The training set contains 3 billion web-crawled sentence pairs. We are also particularly interested in evaluating BMT with the out-of-domain (OOD) setting and assessing its generalizability, as previous research in the image domain demonstrated that compressed models (weight pruned or quantized) have a much larger quality drop on OOD data than their uncompressed counterparts, i.e., model compression amplifies brittleness [19]. As such, to have a robust evaluation of BMT, we use eleven evaluation sets, one of which is in-domain (ID) and is similarly distributed as the training set, and the rest are OOD. For ID, we sample 2000 training examples and remove them from the training data. The ten OOD evaluation sets are divided into four categories (i) Web Domain (ii) News Domain (iii) Wikipedia (iv) Patents. Furthermore, they are either “source-original” or “target-original”. The source- original datasets have a natural source side (English) while the target side (German) is human or machine translated. The target- original datasets have the natural target side (German), then back translated into source English sentences. We do this differentiation to investigate the impact of binarization on “style” of sentences since natural language exhibits rich
diversity as opposed to simple and literal (translationese) sentences [14] (More details are provided in Appendix A.1).
Models & Training. We train two sets of Transformers, namely, encoder-scaling and decoder-scaling models. The encoder-scaling models have a fixed depth of 6 layers in the decoder while scaling up the encoder depth in sizes of  $\{6,8,10,12,14,18,22,26,30,36,42,48\}$  layers, for a total of 12 models. Same for the decoder-scaling ones, whereby the decoder depth is scaled up in similar ways. Due to the sufficiency in training data, we did not use label smoothing during training. The binary models are trained without KD. (Appendix A.2 has more details on hyper-parameters and training).
Observations. Figure 2 compares the scaling curves of the binary and float models on both ID and OOD datasets, more in Appendix A.3. Figure 3 compares their training vs. In-domain test loss. We make the following observations:
Binary models demonstrated similar scaling behaviors as their float counterpart for both encoder and decoder scaling. The exponent of the fitted power law for binary models in Figure 2a ( $p_{e} = 0.16$ ,  $p_{d} = 0.28$ ) is only slightly below float ones ( $p_{e} = 0.18$ ,  $p_{d} = 0.31$ ), indicating the binary model loss improves fast as the parameter count increases. This trend also holds for OOD Wikipedia dataset in Figure 2b. Binary models generalize just as well on OOD data as float models (scaling law fits on all the OOD evaluation datasets is in Appendix A.3). We also note a gap between binary and float model losses, a phenomenon not observed from WMT experiments. We hypothesize that this is because the in-house production-scale datasets are more challenging.
For the same training loss, binary and float models achieve the same generalization performance. As shown in Figure 3, binary and float model losses align well on a straight line, and almost overlap in the  $0.95 \sim 1.0$  region. There are no measurable differences detected in the inductive biases of the two model classes. Also, binary models require fewer parameter bits to achieve a certain performance level. For example, a 6L42L binary Transformer with 195M parameters (195M bits) has a  $4.3 \times$  smaller size than a 6L8L float one with 52M parameters (832M bits) while having the same loss. Such memory savings are especially advantageous when the models are deployed in a resource-constrained environments [36].
Figure 3: Training vs. ID Test loss. We observe similar linear relationship between training and test losses of all evaluation datasets.
4.3 Generation Quality
We examine the MT model generation quality in Figure 4 using two decoding strategies: a) Beam Search Decoding; b) Minimum Bayes Risk (MBR) decoding [23].
Beam search. Sample quality from Beam search decoding is evaluated with standard de-tokenized BLEU scores [30] using sacreBLEU library  $[31]^5$ .
MBR. Freitag et al. [15] show that beam search decoding selects samples with high probability rather than high quality, especially for large models, as measured by human evaluations. They propose MBR-based decoding strategy defined as  $h^{\mathrm{MBR}} = \arg \max_{h\in \mathcal{H}}\frac{1}{|\mathcal{H}_{\mathrm{model}}|}\sum_{y\in \mathcal{H}_{\mathrm{model}}}u(h,y)$ , where  $h^{\mathrm{MBR}}$  is the decoding from the model given source sentence  $x$ ,  $\mathcal{H}_{\mathrm{model}}$  is the set of hypotheses sampled from the model  $p(\cdot |x)$  and  $u$  is a utility function that evaluates quality of a hypothesis  $h$  against reference  $y$ . Freitag et al. [15] demonstrate effectiveness of the BLEURT model [35] for the utility function. BLEURT is a regression model that relies on the concatenation of hypothesis  $h$  and reference  $y$  and generates a scalar score between [0,1], measuring the hypothesis quality irrespective of the sentence structure, length or word overlap with the reference. In the same way, we use MBR decoding with BLEURT as the utility function to decode a sequence given the source sentence. To measure the sample quality,  $\mathrm{BLEURT}(h,r)$  is calculated between the decoded hypothesis  $(h_{\mathrm{MBR}})$  and the reference  $(r)$  for a given source sentence  $(x)$ , then averaged across the evaluation set.
5Beam size=4, length penalty=0.6. case.mixed + numrefs.1 + smooth.exp + tok.13a
(a) Beam Search - BLEU
(b) MBR-BLEURT
Figure 4: Comparison on translation qualities between binarized and float models for encoder-scaling. (a) Beam Search Decoding: BLEU scores on In-Domain Test set (b) MBR Decoding:BLEURT scores on In-Domain Test set
Observations. Figure 4a shows BLEU scores of encoder-scaling models (i.e., decoder depth=6, varying encoder depth). Figure 4b plots BLEURT scores for encoder-scaling models, where the baseline is float models using MBR decoding with 16 samples. We observe the following:
Binary models can achieve the same BLEU score as float models with a smaller size. Figure 4a shows that the BLEU score of binary models will consistently improve as the model size increases. Although binary models are 2-3 BLEU worse than float ones at the same model depth, the 30L6L binary model achieves the same BLEU as the 8L6L float model, while being  $6.7 \times$  smaller in size.
Increasing the sample size can match the generation quality of binary models with float models. In Figure 4b, a larger sample size consistently produces a higher generation quality for the binary models. At  $4 \times$  the sample size, i.e., 64 samples, the binary model quality approximately matches float models. Besides, the BLEURT score of binary models also improves as the model size increases.
5 Ablation Study
(a) Models losses with different scaling factor  $s$ .
(b) Losses of different attention out linear architectures.
Scaling factor ablation. We binarize the FFN only and sweep the scaling factor  $s$  as a power of two from 1 (equivalent to no scaling factor applied) to 4096. We plot the final training and validation losses in Figure 5a. The model losses drop steeply when increasing  $s$  to 64. Models with  $s \leq 8$  produce almost random translation quality. Large scaling factors indeed address the convergence issue. The loss begins saturated at  $s = 64$  and is only slightly worse than the float baseline (1.39). This exactly matches our expectation that  $s \propto \sqrt{D}$ . When  $s > 64$ , the model loss keeps improving slightly. We hypothesize that this is because the bound  $B$  is dynamic. Even a small variation on  $B$  will change the theoretical optimal  $s$  by a large margin since  $\operatorname{Var}(A_b \cdot W_b) \propto B^4$ .
BMT attention layer ablation. We only binarize the attention output projection linear layer. We train the model for 88339 steps, with binarization events started at step 50000. We plot the loss curves from step 40000 in Figure 5b. Applying a fixed scaling factor achieves an almost 0.2 loss improvement. This is consistent with previous observations where a scaling factor helps with convergence. The LayerNorm, as a drop-in replacement for the scaling factor, not only makes the model converge to a better loss, but also recovers the loss much faster after binarization. This is expected because  $\gamma$  in the LayerNorm is learnable and can better adapt to the dynamic bound  $B$  as analyzed in Section 3.4. The loss almost saturates after binarization. Adding a shortcut around the output projection removes the information bottleneck. It helps the model converge to approximately the same quality as the float baseline.
The proposed method enables binarization for machine translation. The simple yet effective scaling factor is the key. Binary Transformers have a similar scaling behavior of translation quality as float models. Binarization can thus be a potential candidate for future model serving.
Unanswered questions: How to better binarize attention einsums? Which is better for scaling up a binary Transformer, depth or width? What will be a good mixed-precision scheme?
This work is supported in part by NSF Award #2007832.
[1] Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.
[2] Alham Fikri Aj and Kenneth Heafield. Neural machine translation with 4-bit precision and beyond. arXiv preprint arXiv:1909.06091, 2019.
[3] Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondrej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina Espana-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. Findings of the 2021 conference on machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, pages 1-88, Online, November 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.wmt-1.1.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[5] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701, 2020.
[6] Loic Barrault, Ondrej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). Association for Computational Linguistics, 2019. URL https://aclanthology.org/W19-5301.
[37] Ashish Vavwani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[38] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pages 36-39. IEEE, 2019.
[39] Yichi Zhang, Zhiru Zhang, and Lukasz Lew. Pokebnn: A binary pursuit of lightweight accuracy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475-12485, 2022.
A Scaling Law Study Details
A.1 Dataset
A concise view of evaluation datasets used for scaling laws (Section 4.2) is shown in Table. The ten OOD evaluation datasets span four categories (i) Web Domain (ii) News Domain (iii) Wikipedia (iv) Patents. They are either "source-original" or "target-original". There are two source- original and one target- original dataset in Web Domain, one source- original each in Wikipedia and Patents domain. We use publicly available WMT newstest2019 [6] and WMT newstest2021 [3] for News Domain. Within this domain, we have five datasets: source- original, target- original, source- original-paraphrased [14] and source- original-high-quality [14] from WMT newstest2019 [6], and wmt-reference-C from WMT newstest2021 [3].
Dataset Name	Domain	Type	Source
Train Subset	Web	mixed	In-house
Patents	Patents	mixed	In-house
Web domain 1	Web	source-original	In-house
Web domain 2	Web	source-original	In-house
Web domain 3	Web	target- original	In-house
Wikipedia	Wikipedia	source- original	In-house
wmt-high-quality	News	source- original	WMT newstest2019 [14]
wmt-refC	News	source- original	WMT newstest2021 Ref-C [3]
wmt-paraphrased	News	source- original	WMT newstest2019 [14]
wmt-src-orig	News	source- original	WMT newstest2019 [6]
wmt-tgt-orig	News	target- original	WMT newstest2019 [6]
Table 2: Evaluation datasets used in Section 4.2.
A.2 Model & Training Details
All the models in Section 4.2 have an embedding dimension of 512, a hidden projection dimension of 2048, and 8 attention heads. The embedding parameters are shared on the source and the target side. The same embedding matrix (transposed) is also used for the linear readout (softmax) parameters on the decoder side. All models are trained with Adam optimizer [22] and use cosine learning rate schedule. Due to the sufficiency in training data, we did not use label smoothing during training. In our experiments, enabling label smoothing resulted in poor development set performance across all the models. Training and Learning rate profiles of one model (6 encoder, 8 decoder layers) are shown in Figure 6. Float models are trained for 5 epochs, and binary models are trained for 9 epochs in two stages: float stage and a binarization stage. An independent but identical learning rate schedules are used (with warmup) in both the stages of the binary model training. We note that a significant amount of training (i.e. loss reduction) for binary models happens in the final 10 steps when the learning rate is extremely small. Raw values of last 15 steps of learning rates are [5.0e-7, 3.1e-7, 1.6e-7, 6.4e-7, 1.0e-8, \{2.5e-15\}x10]. We also tune binary models with a constant learning rate of values in \{1e-8, 1e-11, 1e-15\} for the last epoch (overriding the original schedule), however we observe degradation in the quality (loss plateaus). This phenomenon of significant learning in the final stages of binary models' training at extremely small learning rates is also observed by Liu et al. [27], Zhang et al. [39]. We leave further investigation of this behavior to future work.
A.3 Scaling Law Fit
Scaling law fit on all ten OOD evaluation datasets is shown in Figure 7. The slopes  $p_e$  and  $p_d$  are shown in Figure 8 and Table 3.
B Generation Quality
Generation quality for decoder-scaling models is shown in Figure 9. We observe similar behavior as seen for encoder-scaling models in Section 4.3. BLEU scores for binary models are 2-3 BLEU points
(a) Loss
(b) Learning Rate
Figure 6: Test loss and learning rate profiles of a 6L8L float and binary model as the training progresses.
Dataset	Float models	Binary models
pe	pd	pe	pd
Train Subset	0.18	0.31	0.16	0.28
Patents	0.20	0.30	0.19	0.32
Web Domain 1	0.14	0.25	0.14	0.27
Web Domain 2	0.19	0.37	0.16	0.30
Web Domain 3	0.12	0.18	0.14	0.23
Wikipedia	0.13	0.25	0.12	0.25
wmt-high-quality	0.20	0.31	0.18	0.30
wmt-refC	0.24	0.34	0.17	0.27
wmt-paraphrased	0.14	0.36	0.12	0.31
wmt-src-orig	0.22	0.37	0.23	0.36
wmt-tgt-orig	0.15	0.22	0.12	0.20
Table 3: Tabular representation of the same data  $\left( {{p}_{e}\& {p}_{d}}\right)$  as shown in Figure 8.
worse than the respective float models at the same model depth. MBR-BLEURT based decoding quality increases consistently by increasing the sample size.
B.1 Translation Samples
We generate several En-De translation samples as follows from both float and binary 6L10L encoder-decoder models. Inputs are taken from the WMT2017 dataset.
- Source: The notice for the Nottingham East Labour meeting on Friday stated that "we want the meetings to be inclusive and productive."
- Reference: In der Mitteilung für das East Labour in Nottingham Treffen am Freitag besteht: "Wir wollen, dass die Treffen integrativ und Produktiv sind".
- Float output: In der Mitteilung für das Nottingham East Labour-Treffen am Freitag besteht es: "Wir wollen, dass die Treffen inklusive und Produktiv sind."
- Binary output: Die Bekanntmachung für das Nottingham East Labour-Treffen am Freitag erkläre: "Wir wollen, dass die Sitzungen inklusive und Produktiv sind."
- Source: The Government of Wales Act 2017 gave the Welsh assembly the power to change its name.
- Reference: Mit dem Government of Wales Act 2017 erhielt das walisische Parliament die Möglichkeit, seinen Name zu ändern.
- Float output: Der Government of Wales Act 2017 gab der walisischen Versammlung die Befugnis, ihren Name zu ändern.
- Binary output: Das Government of Wales Act 2017 gab der walisischen Versammlung die Befugnis, ihren Name zu ändern.
- Source: Residents were seen returning to their destroyed homes, picking through waterlogged belongings, trying to salvage anything they could find.
- Reference: Anwohner wurden darauf beobachtet, wie sie in ihre zerstörten Häuser zurückkehren, vollig durchnässte persönliche Gegenstände mitnahmen und versuchten zu retten, was zu retten ist.
- Float output: Die Bewohner wurden gesehen, wie sie in ihre zerstörten Häuser zusückkehrn, durch verstopfte Habeligkeiten pfläckten und versuchten, alles zu retten, was sie finden konnten.
- Binary output: Die Bewohner wurden gesehen, wie sie in ihre zerstörten Häuser zusückkehren, indem sie mit Wasser gefüllte Gegenstände pflückten und versuchten, alles zu retten, was sie finden konnten.
Figure 7: Scaling law studies on evaluation datasets defined in Section 4.2
Figure 8: Encoder and Decoder scaling slopes (i.e.  $p_e \& p_d$ ) as per the scaling law defined in Section 4.2. Raw values are shown in Table 3.
(a) Beam Search BLEU
(b) MBR-BLEURT
Figure 9: Comparison on translation qualities between binarized and bfloat16 models for decoder-scaling.