SlowGuess/ACL_2020_OCR huggingface dataset:
Thomas Zenkel, Joern Wuebker, John DeNero
Lilt, Inc.
first_name  $\equiv$  lilt.com
Word alignment was once a core unsupervised learning task in natural language processing because of its essential role in training statistical machine translation (MT) models. Although unnecessary for training neural MT models, word alignment still plays an important role in interactive applications of neural machine translation, such as annotation transfer and lexicon injection. While statistical MT methods have been replaced by neural approaches with superior performance, the twenty-year-old GIZA++ toolkit remains a key component of state-of-the-art word alignment systems. Prior work on neural word alignment has only been able to outperform GIZA++ by using its output during training. We present the first end-to-end neural word alignment method that consistently outperforms GIZA++ on three data sets. Our approach repurposes a Transformer model trained for supervised translation to also serve as an unsupervised word alignment model in a manner that is tightly integrated and does not affect translation quality.
Although word alignments are no longer necessary to train machine translation (MT) systems, they still play an important role in applications of neural MT. For example, they enable injection of an external lexicon into the inference process to enforce the use of domain-specific terminology or improve the translations of low-frequency content words (Arthur et al., 2016). The most important application today for word alignments is to transfer text annotations from source to target (Müller, 2017; Tezcan and Vandeghiste, 2011; Joanis et al., 2013; Escartin and Arcedillo, 2015). For example, if part of a source sentence is underlined, the corresponding part of its translation should be underlined as well. HTML tags and other markup
must be transferred for published documents. Although annotations could in principle be generated directly as part of the output sequence, they are instead typically transferred via word alignments because example annotations typically do not exist in MT training data.
The Transformer architecture provides state-of-the-art performance for neural machine translation (Vaswani et al., 2017). The decoder has multiple layers, each with several attention heads, which makes it difficult to interpret attention activations as word alignments. As a result, the most widely used tools to infer word alignments, namely GIZA++ (Och and Ney, 2003) and FastAlign (Dyer et al., 2013), are still based on the statistical IBM word alignment models developed nearly thirty years ago (Brown et al., 1993). No previous unsupervised neural approach has matched their performance. Recent work on alignment components that are integrated into neural translation models either underperform the IBM models or must use the output of IBM models during training to outperform them (Zenkel et al., 2019; Garg et al., 2019).
This work combines key components from Zenkel et al. (2019) and Garg et al. (2019) and presents two novel extensions. Statistical alignment methods contain an explicit bias towards contiguous word alignments in which adjacent source words are aligned to adjacent target words. This bias is expressed in statistical systems using a hidden Markov model (HMM) (Vogel et al., 1996), as well as symmetrization heuristics such as the grow-diag-final algorithm (Och and Ney, 2000b; Koehn et al., 2005). We design an auxiliary loss function that can be added to any attention-based network to encourage contiguous attention matrices.
The second extension replaces heuristic symmetrization of word alignments with an activation optimization technique. After training two alignment models that translate in opposite direc
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1605-1617 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Word alignment generated by a human annotator.
tions, we infer a symmetrized attention matrix that jointly optimizes the likelihood of the correct output words under both models in both languages. Ablation experiments highlight the effectiveness of this novel extension, which is reminiscent of agreement-based methods for statistical models (Liang et al., 2006; Graça et al., 2008; DeNero and Macherey, 2011).
End-to-end experiments show that our system is the first to consistently yield higher alignment quality than GIZA++ using a fully unsupervised neural model that does not use the output of a statistical alignment model in any way.
2.1 Statistical Models
Statistical alignment models directly build on the lexical translation models of Brown et al. (1993), known as the IBM models. The most popular statistical alignment tool is GIZA++ (Och and Ney, 2000b, 2003; Gao and Vogel, 2008). For optimal performance, the training pipeline of GIZA++ relies on multiple iterations of IBM Model 1, Model 3, Model 4 and the HMM alignment model (Vogel et al., 1996). Initialized with parameters from previous models, each subsequent model adds more assumptions about word alignments. Model 2 introduces non-uniform distortion, and Model 3 introduces fertility. Model 4 and the HMM alignment model introduce relative distortion, where the likelihood of the position of each alignment link is conditioned on the position of the previous alignment link. While simpler and faster tools exist such as FastAlign (Dyer et al., 2013), which is based on a reparametrization of IBM Model 2, the GIZA++ implementation of Model 4 is still used today in applications where alignment quality is important.
In contrast to GIZA++, our neural approach is easy to integrate on top of an attention-based translation network, has a training pipeline with fewer steps, and leads to superior alignment quality.
Moreover, our fully neural approach that shares most parameters with a neural translation model can potentially take advantage of improvements to the underlying translation model, for example from domain adaptation via fine-tuning.
2.2 Neural Models
Most neural alignment approaches in the literature, such as Tamura et al. (2014) and Alkhouli et al. (2018), rely on alignments generated by statistical systems that are used as supervision for training the neural systems. These approaches tend to learn to copy the alignment errors from the supervising statistical models.
Zenkel et al. (2019) use attention to extract alignments from a dedicated alignment layer of a neural model without using any output from a statistical aligner, but fail to match the quality of GIZA++.
Garg et al. (2019) represents the current state of the art in word alignment, outperforming GIZA++ by training a single model that is able to both translate and align. This model is supervised with a guided alignment loss, and existing word alignments must be provided to the model during training. Garg et al. (2019) can produce alignments using an end-to-end neural training pipeline guided by attention activations, but this approach underperforms GIZA++. The performance of GIZA++ is only surpassed by training the guided alignment loss using GIZA++ output. Our method also uses guided alignment training, but our work is the first to surpass the alignment quality of GIZA++ without relying on GIZA++ output for supervision.
Stengel-Eskin et al. (2019) introduce a discriminative neural alignment model that uses a dot-product-based distance measure between learned source and target representation to predict if a given source-target pair should be aligned. Alignment decisions condition on the neighboring decisions using convolution. The model is trained using gold alignments. In contrast, our approach is fully unsupervised; it does not require gold alignments generated by human annotators during training. Instead, our system implicitly learns reasonable alignments by predicting future target words as part of the translation task, but selects attention activations using an auxiliary loss function to find contiguous alignment links that explain the data.
3.1 The Alignment Task
Given a source-language sentence  $x = x_{1},\ldots ,x_{n}$  of length  $n$  and its target-language translation  $y = y_{1},\dots ,y_{m}$  of length  $m$ , an alignment  $\mathcal{A}$  is a set of pairs of source and target positions:
$$
\mathcal {A} \subseteq \left\{(s, t): s \in \{1, \dots , n \}, t \in \{1, \dots , m \} \right\}
$$
Aligned words are assumed to correspond to each other, i.e. the source and the target word are translations of each other within the context of the sentence. Gold alignments are commonly generated by multiple annotators based on the Blinker guidelines (Melamed, 1998). The most commonly used metric to compare automatically generated alignments to gold alignments is alignment error rate (AER) (Och and Ney, 2000b).
3.2 Attention-Based Translation Models
Bahdanau et al. (2015) introduced attention-based neural networks for machine translation. These models typically consist of an encoder for the source sentence and a decoder that has access to the previously generated target tokens and generates the target sequence from left to right. Before predicting a token, the decoder "attends" to the position-wise source representations generated by the encoder, and it produces a context vector that is a weighted sum of the contextualized source embeddings.
The Transformer (Vaswani et al., 2017) attention mechanism uses a query  $Q$  and a set of  $k$  key-value pairs  $K, V$  with  $Q \in \mathbb{R}^d$  and  $V, K \in \mathbb{R}^{k \times d}$ . Attention logs  $A_L$  computed by a scaled dot product are converted into a probability distribution  $A$  using the softmax function. The attention  $A$  serves as mixture weights for the values  $V$  to form a context vector  $c$ :
$$
A _ {L} = \operatorname {c a l c A t t L o g i t s} (Q, K) = \frac {Q \cdot K ^ {T}}{\sqrt {d}}
$$
$$
A = \operatorname {c a l c A t t} (Q, K) = \operatorname {s o f t m a x} \left(A _ {L}\right)
$$
$$
c = \operatorname {a p p l y} \operatorname {A t t} (A, V) = A \cdot V
$$
A state-of-the-art Transformer includes multiple attention heads whose context vectors are stacked to form the context activation for a layer, and the encoder and decoder have multiple layers. For all experiments, we use a downscaled Transformer model trained for translation with a 6-layer encoder, a 3-layer decoder, and 256-dimensional hidden states and embedding vectors.
For the purpose of word alignment, this translation Transformer is used as-is to extract representations of the source and the target sequences, and our alignment technique does not change the parameters of the Transformer. Therefore, improvements to the translation system can be expected to directly carry over to alignment quality, and the alignment component does not affect translation output in any way.
3.3 Alignment Layer
To improve the alignment quality achieved by interpreting attention activations, Zenkel et al. (2019) designed an additional alignment layer on top of the Transformer architecture. In the alignment layer, the context vector is computed as applyAtt(A,V), just as in other decoder layers, but this context vector is the only input to predicting the target word via a linear layer and a softmax that gives a probability distribution over the target vocabulary. This design forces attention onto the source positions that are most useful in predicting the target word. Figure 2 depicts its architecture.
This alignment layer uses the learned representations of the underlying translation model. Alignments can be extracted from the activations of this model by running a forward pass to obtain the attention weights  $A$  from the alignment layer and subsequently selecting the maximum probability source position for each target position as an alignment link:  $\{(\arg\max_{i}(A_{i,j}), j): j \in [1, m]\}$ .
The alignment layer predicts the next target token  $y_{i}$  based on the source representations  $x$  extracted from the encoder of the Transformer and all past target representations  $y_{< i}$  extracted from the decoder. Thus the probability is conditioned as  $p(y_{i}|x,y_{< i})$ . The encoder representation used as key and value for the attention component is the sum of the input embeddings and the encoder output. This ensures that lexical and context information are both salient in the input to the attention component.
3.4 Attention Optimization
Extracting alignments with attention-based models works well when used in combination with greedy translation inference (Li et al., 2019). However, the alignment task involves predicting an alignment between a sentence and an observed translation, which requires forced decoding. When a token in the target sentence is unexpected given the preceding target prefix, attention activations computed
Figure 2: Architecture of the alignment layer. During inference the attention logits  $A_{L}$  of the sub-network Attention Optimization are optimized towards predicting the next word correctly.
during forced decoding are not reliable because they do not explicitly condition on the target word being aligned.
Zenkel et al. (2019) introduce a method called attention optimization, which searches for attention activations that maximize the probability of the output sequence by directly optimizing the attention activations  $A$  in the alignment layer using gradient descent for the given sentence pair  $(x,y)$  to maximize the probability of each observed target token  $y_{i}$  while keeping all other parameters of the neural network  $M$  fixed:
$$
\operatorname {a r g m a x} _ {A} p \left(y _ {i} \mid y _ {<   i}, x, A; M\right)
$$
Attention optimization yields superior alignments when used during forced decoding when gradient descent is initialized with the activations from a forward pass through the alignment layer.
3.5 Full Context Model with Guided Alignment Loss
The models described so far are based on autoregressive translation models, so they are limited to only attend to the left context of the target sequence. However, for the word alignment task the current and future target context is also available and should be considered at inference time. Garg et al. (2019) train a single model to both predict the target sentence and the alignments using guided alignment training. When the model is trained to
Figure 3: Alignment layer with additional unmasked self attention sublayer to use the full decoder context.
predict alignments, the full target context can be used to obtain improved alignment quality.
The alignment loss requires supervision by a set of alignment links for each sentence pair in the training data. These alignments can be generated by the current model or can be provided by an external alignment system or human annotators. Assuming one alignment link per target token, we denote the alignment source position for the target token at position  $t$  as  $a_{t}$ . The guided alignment loss  $L_{a}$ , given attention probabilities  $A_{a_{t},t}$  for each source position  $a_{t}$  and target position  $t$  for a target sequence of length  $m$ , is defined as:
$$
L _ {a} (A) = - \frac {1}{m} \sum_ {i = 1} ^ {m} \log \left(A _ {a _ {t}, t}\right)
$$
As depicted in Figure 3, we insert an additional self-attention component into the original alignment layer, and leave the encoder and decoder of the Transformer unchanged. In contrast to Garg et al. (2019), this design does not require updating any translation model parameters; we only optimize the alignment layer parameters with the guided alignment loss. Adding an alignment layer for guided alignment training has a small parameter overhead as it only adds a single decoder layer, resulting in an increase in parameters of less than  $5\%$ .
Unlike the standard decoder-side self-attention layers in the Transformer architecture, the current and future target context are not masked in the
For the purpose of the guided alignment loss we assume target tokens that do not have an alignment link to be aligned to the end-of-sentence (EOS) token of the source sequence.
The translation model contains 15 million parameters, while the additional alignment layer has 700 thousand parameters.
alignment layer self-attention component in order to provide the full target sentence as context. Alignment layer parameters are trained using the guided alignment loss.
4 Contiguity Loss
Contiguous alignment connections are very common in word alignments, especially for pairs of Indo-European languages. That is, if a target word at position  $t$  is aligned to a source word at position  $s$ , the next target word at position  $t + 1$  is often aligned to  $s - 1$ ,  $s$  or  $s + 1$  (Vogel et al., 1996).
Our goal is to design a loss function that encourages alignments with contiguous clusters of links.
The attention activations form a 2-dimensional matrix  $A \in \mathbb{R}^{n \times m}$ , where  $n$  is the number of source tokens and  $m$  the number of target tokens: each entry represents a probability that specifies how much attention weight the network puts on each source word to predict the next target word. By using a convolution with a static kernel  $K$  over these attention scores, we can measure how much attention is focused on each rectangle within the two dimensional attention matrix:
$$
\bar {A} = \operatorname {c o n v} (A, K)
$$
$$
L _ {C} = - \sum_ {t = 1} ^ {m} \log \left(\max  _ {s \in \{1, \dots , n \}} (\bar {A} _ {s, t})\right)
$$
We use a  $2 \times 2$  kernel  $K \in \mathbb{R}^{2 \times 2}$  with each element set to 0.5. Therefore,  $\bar{A} \in \mathbf{R}^{n \times m}$  will contain the normalized attention mass of each  $2 \times 2$  square of the attention matrix  $A$ . The resulting values after the convolution will be in the interval [0.0, 1.0]. For each target word we select the square with the highest attention mass, encouraging a sparse distribution over source positions in  $\bar{A}$  and thus effectively training the model towards strong attention values on neighboring positions. We mask the contiguity loss such that the end of sentence symbol is not considered during this procedure. We apply a position-wise dropout of 0.1 on the attention logits before using the softmax function to obtain  $A$ , which turned out to be important to avoid getting stuck in trivial solutions during training.
Optimizing the alignment loss especially encour
Figure 4: Example of alignment patterns that lead to a minimal contiguity loss.
ages diagonal and horizontal patterns $^4$  as visualized in Figure 4. These correspond well to a large portion of patterns appearing in human alignment annotations as shown in Figure 1.
5 Bidirectional Attention Optimization
A common way to extract word alignments is to train two models, one for the forward direction (source to target) and one for the backward direction (target to source). For each model, one can extract separate word alignments and symmetrize these using heuristics like grow-diagonal (Och and Ney, 2000b; Koehn et al., 2005).
However, this approach uses the hard word alignments of both directions as an input, and does not consider any other information of the forward and backward model. For attention-based neural networks it is possible to adapt attention optimization as described in Section 3.4 to consider two models at the same time. The goal of attention optimization is to find attention activations that lead to the correct prediction of the target sequence for a single neural network. We extend this procedure to optimize the likelihood of the sentence pair jointly under both the forward and the backward model, with the additional bias to favor contiguous alignments. Figure 5 depicts this procedure.
5.1 Initialization
Since attention optimization uses gradient descent to find good attention activations, it is important to start with a reasonable initialization. We extract the attention logits (attention before applying the softmax) from the forward  $(A_L)_F$  and the backward model  $(A_L)_B$  and average these to get a starting point for gradient descent:
$$
\left(A _ {L}\right) _ {i n i t} = \frac {1}{2} \left(\left(A _ {L}\right) _ {F} + \left(A _ {L}\right) _ {B} ^ {T}\right).
$$
Our goal is to find attention logits  $A_{L}$  that lead to the correct prediction for both the forward  $M_F$
Vertical patterns are not encouraged, as it is not possible to have an attention probability above 0.5 for two source words and the same target word, because we use the softmax function over the source dimension.
<sup>3</sup>A trivial solution the network converged to when adding the contiguity loss without dropout was to align each target token to the same source token.
Figure 5: Bidirectional Attention Optimization. We optimize the attention logits towards the correct prediction of the next token when used for both the forward and backward model. The attention values  $V_{F}$  and  $V_{B}$  extracted from the forward and backward model remain static. Additionally, the attention logits are biased towards producing contiguous alignments.
and the backward model  $M_B$ , while also representing contiguous alignments. We will use the cross entropy loss CE for a whole target sequence  $y$  of length  $m$  to define the loss, given probabilities for each target token  $p(y_t | A_t; M)$  under model parameters  $M$  and a given attention activation vector  $A_t$ :
$$
\mathsf {C E} (p (y | A; M)) = \sum_ {t = 1} ^ {m} - \log (p (y _ {t} | A _ {t}; M))
$$
Let  $x, y$  be the source and target sequence, so that we can define a loss function for each component with the interpolation parameter  $\lambda$  for the contiguity loss  $L_{C}$  as follows:
$$
L _ {F} = \mathrm {C E} (p (y | \text {s o f t m a x} (A _ {L}); M _ {F}))
$$
$$
L _ {B} = \operatorname {C E} \left(p \left(x \mid \operatorname {s o f t m a x} \left(A _ {L} ^ {T}\right); M _ {B}\right)\right)
$$
$$
L = L _ {F} + L _ {B} + \lambda L _ {C}
$$
We apply gradient descent to optimize all losses simultaneously, thus approximating a solution of  $\operatorname{argmin}_{A_L} L(x, y | A_L, M_F, M_B)$ .
5.3 Alignment Extraction
After optimizing the attention logits, we still have to decide which alignment links to extract, i.e. how to convert the soft attentions into hard alignments. For neural models using a single direction a common method is to extract the alignment with the highest attention score for each target token. For our bidirectional method we use the following approach:
We merge the attention probabilities extracted from both directions using element-wise multiplication, where  $\otimes$  denotes a Hadamard product:
$$
A _ {F} = \operatorname {s o f t m a x} \left(A _ {L}\right)
$$
$$
A _ {B} = \operatorname {s o f t m a x} \left(A _ {L} ^ {T}\right) ^ {T}
$$
$$
A _ {M} = A _ {F} \otimes A _ {M}
$$
This favors alignments that effectively predict observed words in both the source and target sentences.
Given the number of source tokens  $n$  and target tokens  $m$  in the sentence, we select  $\min(n, m)$  alignments that have the highest values in the merged attention scores  $A_M$ . In contrast to selecting one alignment per target token, this allows unaligned tokens, one-to-many, many-to-one and many-to-many alignment patterns.
6.1 Data
We use the same experimental setup<sup>5</sup> as described by Zenkel et al. (2019) and used by Garg et al. (2019). It contains three language pairs: German  $\rightarrow$  English, Romanian  $\rightarrow$  English and English  $\rightarrow$  French (Och and Ney, 2000a; Mihalcea and Pedersen, 2003). We learn a joint byte pair encoding (BPE) for the source and the target language with 40k merge operation (Sennrich et al., 2016). To convert from alignments between word pieces to alignments between words, we align a source word to a target word if an alignment link exists between any of its word pieces.
Using BPE units instead of words also improved results for GIZA++ (e.g., 20.9% vs. 18.9% for German→English in a single direction). Therefore, we use the exact same input data for GIZA++ and all our neural approaches. For training GIZA++ we use five iterations each for Model 1, the HMM model, Model 3 and Model 4.
6.2 Training