SlowGuess/ACL_2020_OCR huggingface dataset:
Most of the language pairs do not contain an adequately sized development set for word alignment experiments. Therefore, rather than early stopping, we used a fixed number of updates for each training stage across all languages pairs: 90k for training the translation model, 10k for the alignment layer and 10k for guided alignment training (batch-size:
https://github.com/lilt/alignment-scripts
36k words). Training longer did not improve or degrade test-set AER on German→English; the AER only fluctuated by less than  $1\%$  when training the alignment layer for up to 20k updates while evaluating it every 2k updates.
We also trained a base transformer with an alignment layer for German  $\rightarrow$  English, but achieved similar results in terms of AER, so we used the smaller model described in sub-section 3.2 for other language pairs. We adopted most hyperparameters from Zenkel et al. (2019), see the Supplemental Material for a summary. We tuned the interpolation factor for the contiguity loss on German  $\rightarrow$  English.
6.3 Contiguity Loss
Results of ablation experiments for the contiguity loss can be found in Table 1. Our first experiment uses the contiguity loss during training and we extract the alignments from the forward pass using a single direction without application of attention optimization. We observe an absolute improvement of  $6.4\%$  AER  $(34.2\%)$  to  $27.8\%$  after adding the contiguity loss during training.
Afterwards, we use the model trained with contiguity loss and use attention optimization to extract alignments. Adding the contiguity loss during attention optimization further improves the AER scores by  $1.2\%$ . Both during training and attention optimization we used an interpolation coefficient of  $\lambda = 1.0$  for the contiguity loss.
By visualizing the attention activations in Figure 7 we see that the contiguity loss leads to sparse activations. Additionally, by favoring contiguous alignments it disambiguates correctly the alignment between the words "we" and "wir", which appear twice in the sentence pair. In the remaining experiments we use the contiguous loss for both training and attention optimization.
While we used a kernel of size  $2 \times 2$  in our experiments, we also looked at different sizes. Using a  $1 \times 1$  kernel during attention optimization leads to an AER of  $22.8\%$ , while a  $3 \times 3$  kernel achieves the best result with an AER of  $21.2\%$ , compared to  $21.5\%$  of the  $2 \times 2$  kernel. Larger kernel sizes lead to slightly worse results:  $21.4\%$  for a  $4 \times 4$  kernel and  $21.5\%$  for a  $5 \times 5$  kernel.
6.4 Bidirectional Attention Optimization
The most commonly used methods to merge alignments from models trained in opposite direc
Method	No Contiguity	Contiguity
Forward	34.2%	27.8%
Att. Opt	22.7%	21.5%
Table 1: AER results with and without using the contiguity loss when extracting alignments from the forward pass or when using attention optimization for the language pair German  $\rightarrow$  English.
AER
DeEn	21.5%
EnDe	25.6%
Grow-diag	19.6%
Grow-diag-final	19.7%
Bidir. Att. Opt	17.9%
Table 2: Comparison of AER scores between bidirectional attention optimization and methods to merge hard alignments.
tions are variants of grow-diagonal. We extract hard alignments for both German  $\rightarrow$  English and English  $\rightarrow$  German with (monolingual) attention optimization, which leads to an AER of  $21.5\%$  and  $25.6\%$ , respectively. Merging these alignments with grow-diagonal leads to an AER of  $19.6\%$ , while grow-diagonal-final yields an AER of  $19.7\%$ .
We tuned the interpolation factor  $\lambda$  for the contiguity loss during bidirectional optimization. A parameter of 1.0 leads to an AER of  $18.2\%$ , 2.0 leads to  $18.0\%$  while 5.0 leads to  $17.9\%$ . Compared to unidirectional attention optimization it makes sense to pick a higher interpolation factor for the contiguity loss, as it is applied with the loss of the forward and backward model.
For the remaining experiments we use 5.0 as the interpolation factor. Bidirectional attention optimization improves the resulting alignment error rate compared to the grow-diagonal heuristic by up to  $1.8\%$  for German  $\rightarrow$  English. These results are summarized in Table 2.
Variants of grow-diagonal have to rely on the hard alignments generated by the forward and the backward model. They only choose from these alignment links and therefore do not have the ability to generate new alignment links.
In contrast, bidirectional attention optimization takes the parameters of the underlying models into account and optimizes the underlying attention logs simultaneously for both models to fit the sentence pair. In the example in Figure 8 bidirectional attention optimization is able to correctly predict
${}^{6}$  A  $1 \times  1$  only encourages sparse alignments,and does not encourage contiguous alignments.
Figure 6: AER with respect to gradient descent steps during attention optimization for German  $\rightarrow$  English. Both unidirectional (Unidir) and bidirectional (Bidir) optimization benefit from the contiguity loss (CL). Without the contiguity loss AER slightly degrades after more than three optimization steps.
an alignment link between "übereinstimmend" and "proven" that did not appear at all in the individual alignments of the forward and backward model.
We plot the behavior of attention optimization with a varying number of gradient descent steps in Figure 6. For both unidirectional and bidirectional models attention optimization leads to steadily improving results. Without using the additional contiguity loss, the lowest AER appears after three gradient descent steps and slightly increases afterwards. When using the contiguity loss AER results continue to decrease with additional steps. The contiguity loss seems to stabilize optimization and avoids overfitting of the optimized attention activations when tuning them for a single sentence pair.
6.5 Guided Alignment Training
We now use the alignment layer with the full decoder context by adding an additional self-attention layer that does not mask out the future target context. We extract alignments from the previous models with bidirectional attention optimization and use those alignments for guided alignment training.
This works surprisingly well. While the alignments used for training yielded an AER of  $17.9\%$  after bidirectional attention optimization (Table 4), the full context model trained with these alignments further improved the AER to  $16.0\%$  while using a
Method	DeEn	EnFr	RoEn
Att. Opt.	21.5%	15.0%	29.2%
+Guided	16.0%	6.6%	23.4%
Zenkel et al. (2019)	26.6%	23.8%	32.3%
GIZA++	18.9%	7.9%	27.3%
Table 3: Comparison of unidirectional models with GIZA++.
Method	DeEn	EnFr	RoEn
Bidir. Att. Opt.	17.9%	8.4%	24.1%
+Guided	16.3%	5.0%	23.4%
Zenkel et al. (2019)	21.2%	10.0%	27.6%
Garg et al. (2019)	20.2%	7.7%	26.0%
GIZA++	18.7%	5.5%	26.5%
Table 4: Comparison of neural alignment approaches with GIZA++ after using symmetrization of the forward and backward model.
single model for German  $\rightarrow$  English (Table 3). After guided alignment training is complete, we do not apply attention optimization, since that would require a distribution over target words, which is not available in this model.
6.6 End-to-End Results
We now report AER results across all three language pairs. Precision and recall scores are included in the Supplemental Material. We first extract alignments from a unidirectional model, a common use case where translations and alignments need to be extracted simultaneously. Table 3 compares our results to GIZA++ and Zenkel et al. (2019).<sup>7</sup> We observe that guided alignment training leads to gains across all language pairs. In a single direction our approach consistently outperforms GIZA++ by an absolute AER difference between  $1.3\%$  (EnFr) and  $3.9\%$  (RoEn).
Table 4 compares bidirectional results after symmetrization. We compare to purely neural and purely statistical systems. $^{8}$  For symmetrizing alignments of the guided model and GIZA++, we use grow-diagonal. Bidirectional attention optimization is already able to outperform GIZA++ and Garg et al. (2019) on all language pairs except English→French. Using guided alignment training further improves results across all language pairs
Garg et al. (2019) only report bidirectional results after symmetrization.
For additional comparisons including neural models bootstrapped with GIZA++ alignments, see the Supplemental Material.
(a) Without Contiguity Loss
(b) With Contiguity Loss
Figure 7: Attention activations of the alignment layer after attention optimization. Using the contiguity loss during training leads to sparse activations, the correct alignment of the two occurrences of "we"-"wir" and to correct alignment of the period.
(a) Intersection/Union
(b) Bidir. Optimization
(c) Gold Alignments
Figure 8: Example of symmetrization with bidirectional attention optimization. We show all alignments extracted from the forward and backward direction with unidirectional attention optimization in Subfigure 8a (alignments that are only present in one direction are grey). Bidirectional attention optimization is able to extract the correct alignment between "übereinstimmend" and "proven" which did neither appear as an alignment link in the forward nor in the backward direction.
and leads to a consistent AER improvement compared to  $\mathrm{GIZA}++$  and neural results reported by Garg et al. (2019).
These results show that it is possible to outperform GIZA++ both in a single direction and after symmetrization without using any alignments generated from statistical alignment systems to bootstrap training.
This work presents the first end-to-end neural approach to the word alignment task which consistently outperforms GIZA++ in terms of alignment error rate. Our approach extends a pre-trained state-of-the-art neural translation model with an additional alignment layer, which is trained in isolation without changing the parameters used for the trans
lation task. We introduce a novel auxiliary loss function to encourage contiguity in the alignment matrix and a symmetrization algorithm that jointly optimizes the alignment matrix within two models which are trained in opposite directions. In a final step the model is re-trained to leverage full target context with a guided alignment loss. Our results on three language pairs are consistently superior to both GIZA++ and prior work on end-to-end neural alignment. As the resulting model repurposes a pre-trained translation model without changing its parameters, it can directly benefit from improvements in translation quality, e.g. by adaptation via fine-tuning.
Tamer Alkhouli, Gabriel Bretschner, and Hermann Ney. 2018. On the alignment problem in multi-head attention-based neural machine translation. Proceedings of the Third Conference on Machine Translation.
Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations.
Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):263-311.
John DeNero and Klaus Macherey. 2011. Model-based aligner combination using dual decomposition. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 420-429. Association for Computational Linguistics.
Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648.
Carla Parra Escartin and Manuel Arcedillo. 2015. Machine translation evaluation made fuzzier: A study on post-editing productivity and evaluation metrics in commercial settings. Proceedings of MT Summit XV, page 131.
Qin Gao and Stephan Vogel. 2008. Parallel implementations of word alignment tool. In *Software engineering*, testing, and quality assurance for natural language processing, pages 49-57. Association for Computational Linguistics.
Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. 2019. Jointly learning to align and translate with transformer models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4452-4461, Hong Kong, China. Association for Computational Linguistics.
João Graça, Kuzman Ganchev, and Ben Taskar. 2008. Expectation maximization and posterior constraints. In Advances in neural information processing systems.
Eric Joanis, Darlene Stewart, Samuel Larkin, and Roland Kuhn. 2013. Transferring markup tags in statistical machine translation: A two-stream approach. Machine Translation Summit XIV, page 73.
Philipp Koehn, Amittai Axelrod, Alexandra Birch Mayne, Chris Callison-Burch, Miles Osborne, and David Talbot. 2005. Edinburgh system description for the 2005 iwslt speech translation evaluation. In International Workshop on Spoken Language Translation (IWSLT) 2005.
Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019. On the word alignment from neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1293-1303, Florence, Italy. Association for Computational Linguistics.
Percy Liang, Ben Taskar, and Dan Klein. 2006. Alignment by agreement. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics. Association for Computational Linguistics.
I Dan Melamed. 1998. Annotation style guide for the blinker project. arXiv preprint cmp-lg/9805004.
Rada Mihalcea and Ted Pedersen. 2003. An evaluation exercise for word alignment. In Proceedings of the HLT-NAACL 2003 Workshop on Building and using parallel texts data driven machine translation and beyond. Association for Computational Linguistics.
Mathias Müller. 2017. Treatment of markup in statistical machine translation. In Proceedings of the Third Workshop on Discourse in Machine Translation, pages 36-46.
Franz Josef Och and Hermann Ney. 2000a. A comparison of alignment models for statistical machine translation. In *COLING* 2000 Volume 2: The 18th International Conference on Computational Linguistics, volume 2.
Franz Josef Och and Hermann Ney. 2000b. Improved statistical alignment models. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics.
Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19-51.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Elias Stengel-Eskin, Tzu-ray Su, Matt Post, and Benjamin Van Durme. 2019. A discriminative neural model for cross-lingual word alignment. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 910–920, Hong Kong, China. Association for Computational Linguistics.
Akihiro Tamura, Taro Watanabe, and Eiichiro Sumita. 2014. Recurrent neural networks for word alignment model. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
Arda Tezcan and Vincent Vandeghinste. 2011. Smtcat integration in a technical domain. handling xml mark-up using pre and post-editing processing methods. In Proceedings of the 15th International Conference of the European Association for Machine Translation (EAMT-2011), page 8. Centre for Computational Linguistics; Leuven.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996. HMM-based word alignment in statistical translation. In COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics.
Thomas Zenkel, Joern Wuebker, and John DeNero. 2019. Adding interpretable attention to neural translation models improves word alignment. arXiv preprint arXiv:1901.11359.
Table 5 and Table 6 summarize the hyperparameters used for the translation model and the additional alignment layer. In Table 7 we report both AER results and precision and recall for all language pairs.
Hyperparameter	Value
Dropout Rate	0.1
Embedding Size	256
Hidden Units	512
Encoder Layers	6
Decoder Layers	3
Attention Heads Per Layer	8
Table 5: Hyperparameters of the translation model.
Hyperparameter	Value
Dropout Rate	0.1
Embedding Size	256
Hidden Units	256
Attention Heads	1
Table 6: Hyperparameters of the alignment layer.
Method	DeEn	EnDe	Bidir	EnFr	FrEn	Bidir	RoEn	EnRo	Bidir
Att. Opt.	21.5%	25.6%	17.9%	15.0%	14.3%	8.4%	29.2%	28.8%	24.1%
76/81	73/76	85/79	81/92	82/93	90/95	74/68	74/69	85/69
Guided	16.0%	16.6%	16.3%	6.6%	6.3%	5.0%	23.4%	23.1%	23.4%
88/80	89/78	93/76	92/95	93/95	96/94	88/68	90/67	93/65
GIZA++ word	20.9%	23.1%	21.4%	8.0%	9.8%	5.9%	28.7%	32.2%	27.9%
86/72	87/69	94/67	91/93	92/88	98/90	83/63	80/59	94/59
GIZA++ subword	18.9%	20.4%	18.7%	7.9%	8.5%	5.5%	27.3%	29.4%	26.5%
89/74	88/72	95/71	92/93	93/89	98/91	85/64	83/62	93/61
Zenkel et al. (2019)	26.6%	30.4%	21.2%	23.8%	20.5%	10.0%	32.3%	34.8%	27.6%
Garg et al. (2019) + GIZA++	n/a	n/a	20.2%	n/a	n/a	7.7%	n/a	n/a	26.0%
n/a	n/a	16.0%	n/a	n/a	4.6%	n/a	n/a	23.1%
Table 7: AER and—when available—precision/recall scores in percentage in the following row. The Bidir column reports results for the DeEn, EnFr and RoEn translation direction, respectively, and uses grow-diagonal for all columns except when attention optimization is used. For attention optimization we merge alignments with bidirectional attention optimization.