Venue: NAACL 2022

# Cheat Codes to Quantify Missing Source Information

in Neural Machine Translation

Proyang Pal

Kenneth Heafield

School of Informatics, University of Edinburgh, Scotland {proyag.pal,kheafiel}@ed.ac.uk

###### Abstract

This paper describes a method to quantify the amount of information \(H(t|s)\) added by the target sentence \(t\) that is not present in the source \(s\) in a neural machine translation system. We do this by providing the model the target sentence in a highly compressed form (a "cheat code"), and exploring the effect of the size of the cheat code. We find that the model is able to capture extra information from just a single float representation of the target and nearly reproduces the target with two 32-bit floats per target token.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2472 - 2477

## 1 Introduction

Given a sentence \(s\) in the source language, a machine translation system generates a translation \(t\) in the target language. However, for any sentence of non-trivial complexity, the translation \(t\) is not unique. Therefore, to reproduce a reference translation, a model requires some amount of extra information. The aim of this work is to quantify the amount of information that is missing in the source \(s\) that is required to generate the translation \(t\).

To quantify this information, we modify the model architecture to provide the target sentence to the model as an auxiliary input, and observe the effect of varying the size of the representation of the target sentence from the minimum that provides any useful information to the decoder to the size that enables a near-perfect reproduction of the target. Since the model seeing the target as an input is a form of "cheating", we refer to these compressed representations of the target as "cheat codes".

## 2 Related Work

Zoph and Knight (2016) use multiple encoders to provide input in multiple languages to machine translation models to improve translation quality. Dual encoder networks have been used in language generation tasks to inject extra information (Sharath T et al., 2017), encode input at different levels of granularity (Yao et al., 2020), or for context awareness Li et al. (2020). Junczys-Dowmunt and Grundkiewicz (2017) use very similar dual-encoder architectures for automatic post-editing, but without bottlenecks the second encoder output, and the second input in that case is machine translation output instead of a reference. Dinu et al. (2019) train models to inject custom terminology by providing an additional input, but instead of using a second encoder, this is done using inline annotations for the terms to be generated and using factors to demarcate these annotations.

## 3 Method

### Architecture

We use the Marian framework (Junczys-Dowmunt et al., 2018) to implement1 a modified dual-encoder transformer architecture (Zoph and Knight, 2016) similar to the one used by Junczys-Dowmunt and Grundkiewicz (2018), but without the tied encoder parameters.

Footnote 1: [https://github.com/Proyag/marian-dev/tree/cheat-codes](https://github.com/Proyag/marian-dev/tree/cheat-codes)

The first encoder is a standard transformer-base encoder (Vaswani et al., 2017) which takes the source sentence as input, while the second encoder generates a highly compressed representation of the second input. The decoder attends to both encoder contexts - each decoder layer has a multi-head attention block for each encoder and these blocks are stacked (see Figure 1 in Junczys-Dowmunt and Grundkiewicz (2018)). Figure 1 shows our model architecture along with the separate inputs and cheat codes.

For the second encoder, we use a GRU (Cho et al., 2014) with hidden size 256, optionally average its outputs over all the states to get a fixed-length representation, and apply a linear bottleneck

layer. This generates the highly compressed representation of the second input that the decoder attends to.

### Cheat Codes

At training time, we provide the target sentence as the second input to the model, so the model essentially cheats by seeing the translation it is supposed to generate. At inference time, we can provide the reference translation or any other sentence as the second input, which should guide the generation towards this provided input.

Alternatively, this second encoder can be bypassed to directly provide context vectors for the decoder to attend to. As an example, we can use this feature to interpolate between the representation of two different references and provide that as a cheat code, and thus explore whether we can obtain alternative translations in some semantic space between the two references (Section 4.4).

We vary the size of the cheat codes and observe their effect on the output translations. The size is varied in three different ways:

* Using fixed-length representations of \(n\) floating-point numbers, where we can vary \(n\), by averaging over all the output states of the second encoder, and then applying the bottleneck layer to project the result down to \(n\) dimensions.
* Using variable-length representations of \(n\) floating-point numbers per token, which is simply the output of the second encoder, with the bottleneck layer applied on each output state.
* Using representations smaller than one floating-point number by applying quantization on a one-dimensional representation. We do this using a simple linear quantization scheme similar to Miyashita et al. (2016) and Hubara et al. (2017). To quantize a scalar \(x\) to \(k\) bits: \[r=\text{round}(x*m)\] \[c=\text{clip}(r,-2^{k-1},2^{k-1}-1)\] \[\text{Quant}_{k}(x)=c/m\] where \(m\) is a multiplier chosen to ensure the quantized scalar covers the full range of the \(k\)-bit number after quantization. We observe that our single float32 cheat codes are in [-2, 2], so we use \(m=2^{k-2}\) so that \(r\) is spread over \([-2^{k-1},2^{k-1}]\) without getting clipped.

## 4 Experiments

All our experiments use Chen et al. (2021)'s cleaned version of the WMT21 German\(\rightarrow\)English dataset (Akhbardeh et al., 2021). We do not use back-translated data since we observed no improvement in quality upon adding it, consistent with Chen et al. (2021)'s findings. We evaluate on both references A and B in the test set using BLEU2 and ChrF3 metrics from SacreBLEU Post (2018), and COMET and COMET-QE4(Rei et al., 2020).

Footnote 2: BLEU@fj.l5c:mixede:moltok:13als:explv:2.0.0

Footnote 3: chrF2#J1c:mixede:yesinc:6hw:0ls:only:2.0.0

Footnote 4: wmt20-comet-da and wmt20-comet-qe-da in COMET

Table 1 shows the results for our different models with references A or B provided as cheat codes and being evaluated on both references. We see that the models can score higher than the transformer baseline on a given reference when the same reference is supplied as a second input, which indicates that the model is able to "cheat" and capture useful extra information from just a single floating-point representation of the target sentence.

### Increasing bottleneck size

As we increase the size of the bottleneck layer, we see that the model captures more information from the larger cheat codes and the outputs approach the reference translations, as shown by much higher BLEU and ChrF compared to the baseline. However, this is not always reflected in the COMET and COMET-QE scores and we suspect this is due to how COMET is trained. This issue is further discussed in Section 4.5.

Figure 1: Model architecture with inputs and cheat codes





\begin{table}
\begin{tabular}{l r c c c c c c} \hline \hline Model/input & \multicolumn{4}{c}{Score on Reference A} & \multicolumn{3}{c}{Score on Reference B} \\ \cline{2-7}  & BLEU & ChrF & COMET & BLEU & ChrF & COMET & COMET-QE \\ \hline Transformer baseline & 32.2 & 60.3 & 0.5565 & 36.3 & 62.6 & 0.5640 & 0.3472 \\ \hline References scored against each other / with COMET-QE: & & & & & & \\ Reference A & 100 & 100 & 0.9934 & 29.5 & 58.5 & 0.5316 & 0.3265 \\ Reference B & 29.5 & 57.7 & 0.5643 & 100 & 100 & 1.0015 & 0.3829 \\ \hline Reference A as second input, fixed-length cheat codes: & & & & & & \\

[MISSING_PAGE_POST]

 \hline \hline \end{tabular}
\end{table}
Table 1: Evaluation with references A and B as second input



### Minimizing bottleneck size

We have already observed that the model is able to capture useful information from a single 32-bit float. To find the lower bound of the cheat code size that is still useful to the model, we reduce it to less than one float, for which we quantize the 32-bit representations from the second encoder to 16, 8, or 4 bits. We see that the 16-bit cheat codes work almost as well as the 32-bit ones. With less than 16 bits, it appears that the model is unable to capture any extra information from the target.

### Variable-length cheat codes

Since the amount of information contained in sentences can vary widely, it makes sense that the size of cheat codes required to encode them can vary. To this end, we also train models where the size of the cheat code is proportional to sentence length.

For these models, we observe that due to the increased capacity of the second encoder, training a model to "cheat" from the start makes it too dependent on the target, i.e. it does not learn to use the source fully, resulting in the cheat code estimating \(H(t)\) instead of \(H(t|s)\) as intended. Therefore, we first train with a blank second input for the model to learn to use the source, then we continue training with both inputs to train the second encoder.

As expected, we observe a similar pattern of more information being captured as we make the cheat codes larger. At just 2 floats per token, the model scores 92.4 BLEU/96.1 ChrF on reference A with the same reference as input, and likewise 92.5 BLEU/95.5 ChrF on reference B. At 16 floats per token, it scores more than 98 ChrF, which is very close to perfectly reproducing the references.

### Interpolating between references

For models which use fixed-length representations of the second input, we can directly feed the decoder a cheat code instead of an actual input sentence. We use this to interpolate between the encoded forms of the two references. Figure 2 shows the performance of the model with single float32 cheat codes while providing \(\lambda\cdot{\rm enc}({\rm refA})+(1-\lambda)\cdot{\rm enc}({\rm refB})\) as the cheat code. We can see the emergence of a continuous space of cheat codes such that codes close to reference A result in outputs closer to reference A and moving towards reference B moves the output closer to reference B.

### Evaluating with COMET-QE

BLEU and ChrF, along with most commonly used machine translation metrics, are reference-based metrics. This automatically makes it more likely that the model will score highest on a reference when given that exact reference as the cheat code. In Figure 2, for example, we see how the performance on each reference peaks exactly when we provide that reference as input. Since the two references are quite different from each other - they only score 29.5 BLEU when they are scored against each other - using one as the cheat code does not produce good results on the other.

We expected to see COMET-QE scores increase with cheat code size, similar to BLEU and ChrF scores. However, we see that COMET-QE scores remain below the baseline even for most models with large cheat codes and near-perfect BLEU/ChrF scores. We even observe that COMET-QE scores Reference A lower than the baseline output. We conclude that since COMET-QE is a metric trained on machine translation outputs and their human evaluation scores, it does not work well for near-perfect translations and is unable to score them higher than the best MT output. For the same reason, even though COMET scores (with reference) increase for large cheat codes, the pattern is less clear than for the string-matching metrics.

## 5 Conclusions and Future Work

This paper has shown that by letting machine translation models use a highly compressed representation of the target sentence as an auxiliary input, we can estimate the amount of information missing from the source that the model captures from the

Figure 2: Interpolating between representations of references A and B.



target. By varying the size of these representations (cheat codes), we see that the model can capture useful information from as little as a 16- or 32-bit scalar representation of the target. We also see that the model approaches perfect reproduction of the target (>92BLEU/95ChrF) from as little as 2 floats per target token.

A limitation of our method is that it can only estimate the amount of missing information from the source based on the size of cheat code, but we do not get any insight into what this information actually is. In future work, this method can be extended to qualitatively analyze what the missing information is, and how it can possibly be provided to the model in other ways to improve translation quality without "cheating". Another limitation is that the model, if not trained carefully for larger cheat codes, can learn to copy the target without using the source. This is countered by careful training regimes as discussed in Section 4.3.

Since the model is able to capture extra information from the second input, it could be possible to use this to guide the output in other ways than just to reproduce the references. For example, given a small enough representation, we could sweep through the entire range of cheat codes and produce diverse high-quality translations He et al. (2018); Roberts et al. (2020).

## Acknowledgements

This work was supported by funding from the European Union's Horizon 202 research and innovation programme under grant agreement No 825303 (Bergamot).

## References

* Akhbardo et al. [2021] Farhad Akhbardeh, Arkady Arkhamgorodsky, Magdalena Biesialska, Ondrej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina Espana-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morisithia, Masaaki Nagata, Ajay Nagesch, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Alahasera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021. Findings of the 2021 Conference on Machine Translation (WMT21). In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, Punta Cana, Dominican Republic.
* Chen et al. [2021] Pinzhen Chen, Jindrich Helcl, Ulrich Germann, Laurie Burchell, Nikolay Bogoychev, Antonio Valerio Miceli Barone, Jonas Walendorf, Alexandra Birch, and Kenneth Heafield. 2021. The University of Edinburgh's English-German and English-Hausa submissions to the WMT21 news translation task. In _Proceedings of the Conference on Machine Translation at the 2021 Conference on Empirical Methods in Natural Language Processing_, Punta Cana, Dominican Republic.
* Cho et al. [2014] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fathi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
* Dinu et al. [2019] Georgiana Dinu, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan. 2019. Training neural machine translation to apply terminology constraints. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3063-3068, Florence, Italy. Association for Computational Linguistics.
* He et al. [2018] Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. 2018. Sequence to sequence mixture model for diverse machine translation. In _Proceedings of the 22nd Conference on Computational Natural Language Learning_, pages 583-592, Brussels, Belgium. Association for Computational Linguistics.
* Hubara et al. [2017] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2017. Quantized neural networks: Training neural networks with low precision weights and activations. _The Journal of Machine Learning Research_, 18(1):6869-6898.
* Junczys-Dowmunt and Grundkiewicz [2017] Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2017. An exploration of neural sequence-to-sequence architectures for automatic post-editing. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 120-129, Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Junczys-Dowmunt and Grundkiewicz [2018] Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2018. MS-UEdin submission to the WMT2018 APE shared task: Dual-source transformer for automatic post-editing. In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 822-826, Belgium, Brussels. Association for Computational Linguistics.
* Junczys-Dowmunt et al. [2018] Marcin Junczys-Dowmunt, Kenneth Heafield, Hieu Hoang, Roman Grundkiewicz, Anthony Aue. 2018. Marian: Cost-effective high-quality neural machine translation in C++. In _Proceedings of the 

2nd Workshop on Neural Machine Translation and Generation_, pages 129-135, Melbourne, Australia. Association for Computational Linguistics.
* Li et al. (2020) Bei Li, Hui Liu, Ziyang Wang, Yufan Jiang, Tong Xiao, Jingbo Zhu, Tongran Liu, and Changliang Li. 2020. Does multi-encoder help? a case study on context-aware neural machine translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3512-3518, Online. Association for Computational Linguistics.
* Miyashita et al. (2016) Daisuke Miyashita, Edward H. Lee, and Boris Murmann. 2016. Convolutional neural networks using logarithmic data representation.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
* Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2685-2702, Online. Association for Computational Linguistics.
* Roberts et al. (2020) Nicholas Roberts, Davis Liang, Graham Neubig, and Zachary C. Lipton. 2020. Decoding and diversity in machine translation. _CoRR_, abs/2011.13477.
* Sharath et al. (2017) S Sharath T, Shubhangi Tandon, and Ryan Bauer. 2017. A dual encoder sequence to sequence model for open-domain dialogue modeling. _arXiv e-prints_, pages arXiv-1710.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Yao et al. (2020) Kaichun Yao, Libo Zhang, Dawei Du, Trejian Luo, Lili Tao, and Y. Wu. 2020. Dual encoding for abstractive text summarization. _IEEE Transactions on Cybernetics_, 50:985-996.
* Zoph and Knight (2016) Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 30-34, San Diego, California. Association for Computational Linguistics