SlowGuess/NAACL_2022_OCR huggingface dataset:
In this section, we discuss our model from memory usage, long-range modeling, context effect, and computational complexity, respectively. Experiments are conducted with the model finetuned on the sentence baseline and evaluated on the TED, since TED has the most average sentence number per document, which is more likely to reflect the performance of our model for long documents.
5.1 Discussion of Memory
Memory Size Memory size is evaluated through information gain (IG) between the random initialized memory and well trained memory. It is calculated from attention maps in Update Attention and Output Attention. IG from Update Attention indicates the difference of selected information in the memory, and IG from Output Attention refers to how much contextualized knowledge in memory is mapped to the next sentence state. Figure 2 shows IG keeps increasing as memory size increases from 8 to 64, but it dramatically drops at the size of 128 and 512. While increasing the memory size can fit more contextual information, an excessively large memory space is likely to introduce redundant noise. Therefore, it indicates that contextualized knowledge should be better distributed into a relatively dense space. Based on the corresponding s-BLEU score, we set memory size to 16 in all other experiments for the most effectiveness.
Memory Side To analyze the effect of the memory on source and target documents, we set the memory on encoder, decoder and both sides respectively. We find that it is not only necessary to have
the memory to convey the dependency between sentences on the source side but also in the decoding process for the target document. As shown in Table 3, applying the memory on either side can outperform the baseline but the model achieves better scores when incorporating the memory on both sides. It indicates the necessity of contextualized information for both source and target documents.
Memory Position Previous work (Bao et al., 2021; Beltagy et al., 2020) has shown that Transformer lower layers are more likely to have local information while the context is better incorporated into higher layers. We set the memory in lower, intermediate, and higher layers respectively. The results as shown in Table 3 are consistent with the claim. Applying memory in higher layers outperforms the others, and it is even better to have it on only the top layer, which satisfies that the model is more likely to focus on the locality on lower layers and fuse the contextualized information on the top.
5.2 Discussion of Long Dependency
Metric Breakdown To find out on what kind of sentences our model outperforms the sentence-level Transformer, we evaluate the TED dataset with respect to the sentence index in the document. Sentences are ordered fed into the model. We compute and average the s-BLEU for sentences at each sentence index in the document. We further average the scores for every ten index range. As in Figure 3, the x-axis refers to the index range of sentences (e.g., 20 refers to sentences with indexes from 10 to 20), and the y-axis indicates the s-BLEU difference between our model and sentence Transformer. Our model has consistently greater performance, especially for sentences in later part of documents, indicating our model has the superiority than the sentence-level Transformer on longer document translation and long-range modeling.
Long-Range Influence We also analyze the long-range dependency of our model through gradient attribution test introduced by Ancona et al. (2018); Sundararajan et al. (2017). The gradient attribution test reflects the significance of the model input feature to its output prediction. We perform this test by calculating the gradients of our well-trained model on the test set of TED. Since sentences are ordered when fed into the model, evaluating previous sentences' gradient attribution to the current sentence infers if the model supports the long-range dependency. More formally, we define
Figure 3: TED dataset separated by sentences from different indexes in documents, evaluated with SentenceTransformer and Context-Aware Model.
Figure 4: TED dataset evaluated by gradients computed from different sentences ranges. x-axis refers to the difference between the sentence indexes for gradient calculation and loss computation.
the gradient of the previous sentence i computed by the loss propagated from current sentence j as:
$$
\mathrm {G} (\text {S e n t} _ {\mathrm {i}}, \text {S e n t} _ {\mathrm {j}}).
$$
Specifically, the gradient of a certain token in previous sentences is retrieved from its corresponding embedding weight. We conducted experiments for different sentence ranges  $k$  for the test with ten sentences intervals, and the gradient for each range  $k$  is computed as:
$$
\operatorname {S c o r e} (\mathrm {k}) = \operatorname {A v g} \left(\sum_ {\mathrm {d} = 1} ^ {\mathrm {D}} \sum_ {\mathrm {s} = 1} ^ {\mathrm {S} _ {\mathrm {d}}} \sum_ {\mathrm {i} = \mathrm {s} + \mathrm {k}} ^ {\mathrm {s} + \mathrm {k} + 1 0} \mathrm {G} \left(\operatorname {S e n t} _ {\mathrm {s}}, \operatorname {S e n t} _ {\mathrm {i}}\right)\right)
$$
where  $D$  refers to number of documents,  $S_{\mathrm{d}}$  refers to number of sentences in Document d. To prevent the gradient attribution accumulated by the same token within the evaluated range, only unique tokens within this range are considered. As shown in Figure 4, our model has gradients propagated to sentence tokens even by  $90+$  sentences from the computed loss, indicating our model does have the ability for long-range sequence modeling.
Figure 5: Training Loss on TED Dataset, with different optimization window sizes
Figure 6: Attention map from Update Attention, each token at sentence  $t$  is mapped to each memory vector.
Figure 7: Attention map from Output Attention, memory vectors are mapped to each token in sentence  $t + 1$ .
5.3 Discussion of Context
Convergence Our model is trained concerning the sentence order in the document. We find the model hard to converge during training as the loss oscillates within a wide range. Because of the various distribution of consecutive sentences in documents, the directions of continuing optimization steps vary greatly, resulting in an unstable convergence curve. To mitigate this issue, we use group optimization to update the model, considering the dependency among sentences. Specifically,
a number from the optimization window is randomly sampled, and the gradients are accumulated. The model will not be updated until the accumulated steps reach the sampled number. We conduct experiments with different optimization window sizes for the update of 5000 steps, and the loss curves are shown in Figure 5, where full means the total number of sentences in the document. The result shows that the model converges faster and more stable with increasing optimization window size. Such improvement benefits from the grouped update steps concerning the difference of contextualized distribution among sentences.
Dependency Across Sentences We evaluate the attention maps from Update Attention and Output Attention to determine what contextualized information is passed in and out from memory. In Figure 6, tokens from  $t^{\text{th}}$  sentence are mapped to each memory vector, and the  $8^{\text{th}}$  memory vector has a substantial attention weight on token "Frau". Figure 7 shows memory vectors are mapped back to the following sentence and the token "sie" has a high probability on the  $8^{\text{th}}$  memory vector. German words "Frau" and "sie" refer to "Mrs" and "she" in English. Hence, the memory mechanism has the ability to parse the word dependency between sentences at different steps.
5.4 Discussion of Complexity
We further analyze our model's space and time complexity during the inference phase. Since we only evaluate the decoding speed and memory efficiency in this case, we use dummy tokens to perform the inference. We randomly generate a sequence of tokens as the source inputs and let the model decode the same number of tokens as the target. We compare our model with both the sentence-level Transformer and document-level Transformer. For the sentence-level Transformer, we split the sequence of tokens into chunks, and each chunk has a length of 100. The decoding complexity is
Figure 8: Space and Time Complexity for different number of tokens during inference.
evaluated over all chunks. For the document-level Transformer, we use the entire sequence of tokens as the source input and evaluate the complexity of decoding the entire target sequence. Similar to the sentence-level Transformer, our model is evaluated by the chunk by chunk decoding, and meanwhile, we keep the contextual memory updated. As shown in Figure 8, our model keeps the same space complexity as the sentence-level Transformer and takes a slightly more time cost because of the update of contextual memory. However, the document-level Transformer has an increasing cost for both space and time complexity, especially when the target sequence has a length greater than 1,000 tokens. Overall, results have shown the decoding efficiency of our model, which keeps the computational complexity as low as the sentence-level Transformer, even in the case of over thousands of tokens.
This paper introduces a memory unit that recurrently maps information into and out of Transformer intermediate states and addresses the limitation about the context dependency and computational complexity in document-level machine translation. We have achieved the SOTA score on TED and News and a great improvement from the sentence-level baseline. Our model demonstrates the effectiveness and efficiency of reduced memory space, context dependency for both source and target document, and long range influence across documents. The limitation of our work is the training cost since we accumulate the update steps and retain the graph for memory update at each step. Our work does not conduct experiments for pretrained settings due to the time limitation. However, it should be easy to apply our method to any Transformer-based pretrained models, such as Liu
et al. (2020). Also, this paper only experiments on document-level machine translation, and future works may apply this approach for other tasks that need long-range sequence modeling.
We sincerely thank the reviewers from ACL Rolling Review for their helpful feedback, and the suggestions regarding figure plotting from Senior UI designer Xiangru Chen. We also appreciate the support of the computing resources from the Center for Language and Speech Processing (CLSP) at Johns Hopkins University (JHU).
Ruchit Agrawal, Marco Turchi, and Matteo Negri. 2018. Contextual handling in neural machine translation: Look behind, ahead and on both sides.
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 268-284, Online. Association for Computational Linguistics.
Marco Ancona, Enea Ceolini, Cengiz Oztireli, and Markus Gross. 2018. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for document-level machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3442-3455, Online. Association for Computational Linguistics.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th
Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.
Domenic Donato, Lei Yu, and Chris Dyer. 2021. *Diverse pretrained context encodings improve document translation*. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics* and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1299–1311, Online. Association for Computational Linguistics.
Yang Feng, Shiyue Zhang, Andi Zhang, Dong Wang, and Andrew Abel. 2017. Memory-augmented neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1390-1399, Copenhagen, Denmark. Association for Computational Linguistics.
Eva Martínez Garcia, Cristina España-Bonet, and Lluis Márquez. 2015. Document-level machine translation with word vector models. In Proceedings of the 18th Annual Conference of the European Association for Machine Translation, pages 59-66, Antalya, Turkey.
Zhengxian Gong, Min Zhang, and Guodong Zhou. 2011. Cache-based document-level statistical machine translation. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 909-919, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Christian Hardmeier, Sara Stymne, Jörg Tiedemann, and Joakim Nivre. 2013. Docent: A document-level decoder for phrase-based statistical machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 193-198, Sofia, Bulgaria. Association for Computational Linguistics.
Qiuxiang He, Guoping Huang, Qu Cui, Li Li, and Lemao Liu. 2021. Fast and accurate neural machine translation with translation memory. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3170-3180, Online. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Shu Jiang, Rui Wang, Zuchao Li, Masao Utiyama, Ke-hai Chen, Eiichiro Sumita, Hai Zhao, and Bao liang Lu. 2021. Document-level neural machine translation with associated memory network.
Marcin Junczys-Dowmunt. 2019. Microsoft translator at WMT 2019: Towards large-scale document-level neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume
2: Shared Task Papers, Day 1), pages 225-233, Florence, Italy. Association for Computational Linguistics.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700-1709, Seattle, Washington, USA. Association for Computational Linguistics.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver. Association for Computational Linguistics.
Philipp Koehn, Franz J. Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 127-133.
Samuel Läubli, Rico Senrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791-4796, Brussels, Belgium. Association for Computational Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Shuming Ma, Dongdong Zhang, and Ming Zhou. 2020. A simple and effective unified encoder for document-level machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3505-3511, Online. Association for Computational Linguistics.
Pedro Henrique Martins, Zita Marinho, and Andre F. T. Martins. 2021.  $\infty$ -former: Infinite memory transformer.
Sameen Maruf and Gholamreza Haffari. 2018. Document context neural machine translation with memory networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1275-1284, Melbourne, Australia. Association for Computational Linguistics.
Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2019. Selective attention for context-aware neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3092-3102, Minneapolis, Minnesota. Association for Computational Linguistics.
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document-level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2947-2954, Brussels, Belgium. Association for Computational Linguistics.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4055-4064. PMLR.
Jean Pouget-Abadie, Dzmitry Bahdanau, Bart van Merrienboer, Kyunghyun Cho, and Yoshua Bengio. 2014. Overcoming the curse of sentence length for neural machine translation using automatic segmentation. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 78–85, Doha, Qatar. Association for Computational Linguistics.
Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang. 2020. Blockwise selfattention for long document understanding. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2555–2565, Online. Association for Computational Linguistics.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1986. Learning Representations by Backpropagating Errors. Nature, 323(6088):533-536.
Yves Scherrer, Jörg Tiedemann, and Sharid Loá-ciga. 2019. Analysing concatenation approaches to document-level NMT in two different domains. In
Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019), pages 51-61, Hong Kong, China. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319-3328. PMLR.
Jörg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 82-92, Copenhagen, Denmark. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Elena Voita, Rico Senrrich, and Ivan Titov. 2019a. Context-aware monolingual repair for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 877-886, Hong Kong, China. Association for Computational Linguistics.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019b. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1198-1212, Florence, Italy. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33.
Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. In Proceedings of the 2018
Conference on Empirical Methods in Natural Language Processing, pages 533-542, Brussels, Belgium. Association for Computational Linguistics.
Pei Zhang, Boxing Chen, Niyu Ge, and Kai Fan. 2020. Long-short term masking transformer: A simple but effective baseline for document-level neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1081-1087, Online. Association for Computational Linguistics.
Zaixiang Zheng, Xiang Yue, Shujian Huang, Jiajun Chen, and Alexandra Birch. 2020. Towards making the most of context in neural machine translation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3983-3989. International Joint Conferences on Artificial Intelligence Organization. Main track.