SlowGuess/NAACL_2021_OCR huggingface dataset:
Annette Rios<sup>1</sup>, Chantal Amrhein<sup>1</sup>, Noëmi Aepli<sup>1</sup> and Rico Sennrich<sup>1,2</sup>
$^{1}$ Department of Computational Linguistics, University of Zurich
{rios,amrhein,naepli,sennrich}@cl.uzh.ch
Many sequence-to-sequence tasks in natural language processing are roughly monotonic in the alignment between source and target sequence, and previous work has facilitated or enforced learning of monotonic attention behavior via specialized attention functions or pretraining. In this work, we introduce a monotonicity loss function that is compatible with standard attention mechanisms and test it on several sequence-to-sequence tasks: grapheme-to-phoneme conversion, morphological inflection, transliteration, and dialect normalization. Experiments show that we can achieve largely monotonic behavior. Performance is mixed, with larger gains on top of RNN baselines. General monotonicity does not benefit transformer multihead attention, however, we see isolated improvements when only a subset of heads is biased towards monotonic behavior.
Many sequence-to-sequence tasks in natural language processing are roughly monotonic in the alignment between source and target sequence, and previous work has focused on learning monotonic attention behavior either through specialized attention functions (Aharoni and Goldberg, 2017; Raffel et al., 2017; Wu and Cotterell, 2019) or pretraining (Aji et al., 2020). However, it is non-trivial to port specialized attention functions to different models, and recently, Yolchuyeva et al. (2019); Wu et al. (2021) found that a transformer model (Vaswani et al., 2017) outperforms previous work on monotone tasks such as grapheme-to-phoneme conversion, despite having no mechanism that biases the model towards monotonicity.
In the transformer, it is less straightforward to what extent individual encoder states, especially in deeper layers, still represent distinct source inputs after passing through several self-attention
layers. Consequently, it is unclear whether enforcing monotonicity in the transformer is as beneficial as for recurrent neural networks (RNNs).
In this paper, we investigate the following research questions:
1. How can we incorporate a monotonicity bias into attentional sequence-to-sequence models such as the transformer?
2. To what extent does a transformer model benefit from such a bias?
Specifically, we want to incorporate a monotonicity bias in a way that is agnostic of the task and model architecture, allowing for its application to different sequence-to-sequence models and tasks. To this end, we introduce a loss function that measures and rewards monotonic behavior of the attention mechanism.
We perform experiments and analysis on a variety of sequence-to-sequence tasks where we expect the alignment between source and target to be highly monotonic, such as grapheme-to-phoneme conversion, transliteration, morphological inflection, and dialect normalization and compare our results to previous work that successfully applied hard monotonic attention to recurrent sequence-to-sequence models for these tasks (Wu et al., 2018a; Wu and Cotterell, 2019).
Our results show that a monotonicity bias learned through a loss function is capable of making the soft attention between source and target highly monotonic both in RNNs and the transformer. We find that this leads to a similar improvement to previous works on hard monotonic attention for RNNs, whereas for transformer models, the results are mixed: Biasing all attention heads towards monotonicity may limit the representation power of multihead attention in a way
$^{1}$ Code and scripts available at: https://github.com/ZurichNLP/monotonicity_loss
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4474-4488 June 6-11, 2021. ©2021 Association for Computational Linguistics
that is harmful even for monotonic sequence-to-sequence tasks. However, for some tasks, we see small improvements when limiting monotonicity to only a subset of heads.
Attention models (Bahdanau et al., 2015; Luong et al., 2015; Vaswani et al., 2017) are a very powerful and flexible mechanism to learn the relationship between source and target sequences, but the flexibility might come at the cost of making the relationship harder to learn. Previous work has shown that their performance can be improved by introducing inductive biases. Cohn et al. (2016) introduce various structural alignment biases into a neural machine translation model, including a positional bias. While this bias is motivated by the fact that a given token in the source often aligns with a target token at a similar relative position, it does not explicitly encourage monotonicity.
In contrast, Raffel et al. (2017) propose to modify the attention mechanism to learn hard monotonic alignments instead of computing soft attention over the whole source sequence. Several extensions have been proposed: having a pointer monotonically move over the source sequence and computing soft attention on a local window (Chiu and Raffel, 2018) or from the beginning of the sequence up to the pointer (Arivazhagan et al., 2019). For tasks like simultaneous translation and automatic speech recognition, the main benefit from hard monotonic attention is that decoding becomes faster and can be done in an online setting. However, many sequence-to-sequence tasks behave roughly monotonic and biasing the attention towards monotonicity can improve performance; especially in low-resource settings. Aharoni and Goldberg (2017) show that hard monotonic attention works well for morphological inflection if it mimics an external alignment.
Wu et al. (2018b) propose a probabilistic latent-variable model for hard but non-monotonic attention which Wu and Cotterell (2019) later extend to exact hard monotonic attention. In contrast to Aharoni and Goldberg (2017), the alignment is learned jointly with the model. Their approach outperforms several other models on grapheme-to-phoneme conversion, transliteration, and morphological inflection. Monotonic attention has also improved tasks such as summarization (Chung et al., 2020) and morphological analysis (Hwang and Lee, 2020).
Recently, the transformer architecture (Vaswani et al., 2017) has outperformed RNNs in low-resource settings for character-level transduction tasks (Yolchuyeva et al., 2019; Wu et al., 2021) and neural machine translation (Araabi and Monz, 2020). While there has been some work on extending the methods of Raffel et al. (2017); Chiu and Raffel (2018); Arivazhagan et al. (2019) to multihead attention (Ma et al., 2020; Liu et al., 2020), we are not aware of any work that studied monotonicity in transformers for monotonic tasks, such as grapheme-to-phoneme conversion, transliteration, or morphological inflection.
To this end, we propose a model-agnostic monotonicity loss that can seamlessly be integrated into RNNs as well as the transformer. Our monotonicity loss captures how monotone the soft attention behaves during training, while two hyperparameters allow us to control how much monotonicity is enforced. By encouraging monotonicity through a loss instead of a modification of the attention mechanism, our implementation still brings all the benefits of soft attention to tasks where fast, online inference is not paramount and allows us to explore various trade-offs between unconstrained and fully monotonic attention.
3 Monotonicity Loss
We now introduce our monotonicity loss function. The loss function is differentiable and compatible with standard soft attention mechanisms and is thus easy to integrate into popular encoder-decoder architectures such as the transformer. On a high level, we compare the attention distribution between decoder time steps in a pairwise fashion and measure whether the mean attended position increases for each pair.
Let us denote the input sequence as  $X = (x_{1},\dots,x_{|X|})$ , and the output sequence as  $Y = (y_{1},\dots,y_{|Y|})$ . The interface between the encoder and decoder is one or several attention mechanisms. In its general form, the attention mechanism computes some energy  $e_{ij}$  between a decoder state at time step  $i$  and an encoder state  $j$ . While this energy function varies, with popular choices being a feedforward network (Bahdanau et al., 2015) or (scaled) dot-product (Luong et al., 2015; Vaswani et al., 2017), they are typically normalized to a vector of attention weights  $\alpha$  using the softmax
Figure 1: Average attention positions between target output characters and source input characters and the corresponding monotonicity loss for different attention distributions, and with different margins  $\delta$ . The average attention positions were rounded to integers for visualization purposes.
margin:	loss:
0	0.364
0.5	0.530
1	0.697
margin:	loss:
0	0.000
0.5	0.167
1	0.333
margin:	loss:
0	0.000
0.5	0.000
1	0.152
margin:	loss:
0	0.000
0.5	0.000
1	0.000
$$
\alpha_ {i j} = \frac {\exp \left(e _ {i j}\right)}{\sum_ {k = 1} ^ {| X |} \exp \left(e _ {i k}\right)} \tag {1}
$$
These attention weights are then applied to obtain a weighted average  $c_{i}$  of a vector of value states  $V$ :
$$
c _ {i} = \sum_ {j = 1} ^ {| x |} \alpha_ {i j} \cdot v _ {j} \tag {2}
$$
For our monotonicity loss, we also compute the mean attended position  $\bar{a}_i$ :
$$
\bar {a} _ {i} = \sum_ {j = 1} ^ {| x |} \alpha_ {i j} \cdot j \tag {3}
$$
We can then define the monotonicity loss in a pairwise fashion, comparing the mean attended position at time steps  $i$  and  $i + 1$ :
$$
L _ {\text {m o n o}} = \sum_ {i = 1} ^ {| Y | - 1} \max  \left(\frac {\bar {a} _ {i} - \bar {a} _ {i + 1} + \delta \frac {| X |}{| Y |}}{| X |}, 0\right) \tag {4}
$$
$\delta$  is a hyperparameter that controls how deviations from the main diagonal are penalized. Let us first consider the case with  $\delta = 0$ : if  $\bar{a}_{i + 1} \geq \bar{a}_i$  for all positions  $i$ , i.e. if the mean attended position is weakly increasing, then the loss is 0. Any decrease
in the mean attended position will incur a cost that is proportional to the amount of decrease, relative to the source sequence length; this allows differentiation of the loss, and will also serve as a measure of the degree of monotonicity in the analysis.
We might want to bias the model towards strictly monotonic behavior, penalizing it if  $\bar{a}$  remains unchanged over several time steps. We can achieve this by incurring a loss if  $\bar{a}$  does not increase by some margin, controlled by  $\delta$ . At the most extreme, with  $\delta = 1$ , the loss is minimized if the mean attended position follows the main diagonal of the alignment matrix, increasing by  $\frac{|X|}{|Y|}$  at each time step. Figure 1 shows how the margin  $\delta$  can influence the monotonicity loss with some examples.
In equation 4, costs are later summed over the target sequence. In practice, we normalize the cost by the number of tokens in a batch for training stability, as is typically done for the cross-entropy loss. If a model has multiple attention mechanisms, e.g. attention in multiple layers, or multihead attention, we separately compute the loss for each attention mechanism, then average the losses. We can also just apply the loss to a subset of attention mechanisms, allowing different attention heads to learn specialized behavior (Voita et al., 2019).
3Making the cost relative to the source sequence length ensures that the worst-case cost per timestep is independent of source sequence length.
2We can swap  $\bar{a}_i$  and  $\bar{a}_{i + 1}$  in equation 4 to bias the model towards monotonically decreasing attention.
4.1 Models and Data
We implement the loss function in sockeye (Hieber et al., 2018), and experiment with RNN and transformer models. We list the specific baseline settings for each task in Appendix A.2.
The monotonic loss function is controlled by a hyperparameter for the margin  $(\delta)$ , and an additional scaling factor for the loss itself  $(\lambda)$ . Preliminary experiments have shown that the monotonicity loss has an undesirable interaction with attention dropout, which is commonly used in transformer models. Randomly dropping attention connections during training makes it harder to reliably avoid a decrease in the mean attended position, favoring a degenerate local optimum where attention resides constantly on the first (or last) encoder state. To avoid this problem, we use DropHead (Zhou et al., 2020) instead, which has a similar regularizing effect as attention dropout, but does not interact with the monotonicity loss. In addition to the standard evaluation metrics used in each task, we provide the monotonicity loss on the test set and the percentage of target tokens for which the average source attention position has increased (by some margin).
We perform experiments on three word-level and one sentence-level sequence-to-sequence tasks:
Grapheme-to-Phoneme Conversion
For grapheme-to-phoneme conversion, we use NETtalk (Sejnowski and Rosenberg, 1987) $^4$  and CMUdict, $^5$  two datasets for English, with the same data split as Wu and Cotterell (2019). For experiments with RNN models, we follow the settings in Wu et al. (2018b) (large configuration). $^6$
For experiments with transformer models, we follow the settings suggested in Wu et al. (2021), however, we use dropout rates of 0.3 (NETtalk) and 0.2 (CMUdict) instead of 0.1 and 0.3. Furthermore, we use a smaller feed-forward dimension for the NETtalk models (512 instead of 1024), since this a relatively small dataset ( $\sim$ 14k samples).
For both RNN and transformer models, we use early stopping with phoneme error rate, as opposed to a minimum learning rate value as in Wu et al.
Figure 2: Vanilla and our proposed separator-centered positional encoding for the input "use V;SG;3;PRS" in the morphological inflection task.
(2018b) and Wu et al. (2021). We evaluate our models with word error rate (WER) and phoneme error rate (PER).
Morphological Inflection
For morphological inflection, we use the CoNLL-SIGMORPHON 2017 shared task dataset. We choose all 51 languages from the high-resource setting where the training data for each language consists of 10,000 morphological tags + lemma and inflected form pairs (except for Bengali and Haida which have 4,243 and 6,840 pairs respectively) and from the medium-resource setting with 1,000 training examples per language. Our baselines performed very poorly on the low-resource setting with only 100 training examples and we decided to focus on the other two tasks instead.
We preprocess the data to insert a separator token between the morphological tags and the input lemma. The monotonicity loss is then only computed on the positions to the right of the separator token's position. We follow Wu et al. (2021) and use special positional encodings for the morphological tags in the transformer. Unlike their approach, where the position for all tags was set to 0, we set the position of the separator token to 0 and sequentially decrease the positions of the morphological tags to the left (Figure 2). This serves to stabilize the positional encodings of the lemma tokens, while still accounting for the fixed order of morphological tags in the dataset. In preliminary experiments, we observed an improvement of  $0.63\%$  in accuracy over vanilla positional encodings.
We train models on character-level for morphological inflection following the previously recommended settings for RNNs in Wu et al. (2018b)
$^{7}$ https://github.com/sigmorphon/conll2017
4https://archive.ics.uci.edu/ml/ datasets/Connectionist+Bench+(Nettalk+ Corpus)
<sup>5</sup>https://github.com/cmusphinx/cmudit
Even though we follow the settings in Wu et al. (2018b), our RNN models are smaller than theirs (4.5M vs. 8.6M parameters).
and for transformers in Wu et al. (2021) (except for reducing the feed-forward dimension to 512 instead of 1024). For the high resource datasets, we use a batch size of 400, for the medium resource datasets 200. Early stopping is done in the same way as for grapheme-to-phoneme conversion. We use the official evaluation script to compute word-level accuracy (ACC) and character-level edit distance (LEV).
For transliteration, we experiment on the NEWS2015 shared task data (Zhang et al., 2015) and use the same subset of 11 script pairs that Wu and Cotterell (2019) used in their experiments: AR-EN, EN-BA, EN-HI, EN-JA, EN-KA, EN-KO, EN-PE, EN-TA, EN-TH, JN-JK, and TH-EN. Total training dataset sizes range from 6,761 source names for EN-KO up to 27,789 source names for EN-TH. For certain script pairs, multiple transliterations per source name are acceptable. We add all possible pairs to our training data, which only has a large effect on EN-AR, where there are on average 10 acceptable transliterations per source name. Since the references of the official shared task test sets were not released, we follow Wu and Cotterell (2019) and use the development set as our test set. We randomly sample 1,000 names from the training sets as our development sets for script pairs with more than 20,000 training examples and 100 for script pairs with fewer training examples.
Again, we follow Wu et al. (2018b) for hyperparameters in RNNs and Wu et al. (2021) in transformers (smaller feed-forward dimensions of 512). We early stop training as for grapheme-to-phoneme conversion. We evaluate our models following Zhang et al. (2015) and compute word-level accuracy (ACC) and character-level mean F-score (MFS). The formula for MFS is in Appendix A.1.
Dialect Normalization
For this work, we consider dialect normalization as a machine translation task from dialect to standard. We work with the dataset described in Aepli and Clematide (2018), which consists of 26,015 crowd-sourced German translations of 6,197 original Swiss German sentences. We use three documents  $(10\%)$  as test sets and randomly split the rest in development and training set  $(10\%$  and  $80\%$  respectively). The alignment between Swiss German and the German translations is highly monotonic, but there are occasional word order differences, as
Figure 3: Swiss-German to German dialect normalization example with verb reordering.
illustrated in Figure 3.
The models are trained on subwords obtained via BPE (Sennrich et al., 2016), created with subword-nmt computing 2000 merges. We treat this as a low-resource machine translation task, and thus follow hyperparameters by Sennrich and Zhang (2019) for the RNN models, while the transformer models are trained according to Araabi and Monz (2020). We evaluate our models with BLEU (Papineni et al., 2002).
In addition to task-specific evaluation metrics, we use the loss function to score the monotonicity of the attention on the test set for all models (reported as  $L_{MONO}$ ). Furthermore, we report the percentage of decoding states for which the average source attention position  $\bar{a}$  increases by at least  $\delta \frac{|X|}{|Y|}$  as  $\%$  mono. In other words, this is the percentage of states for which the pairwise loss is 0.
We test different settings on the grapheme-to-phoneme task, see Table 1 for results with RNNs (top) and transformers (bottom). We find that models trained with the additional loss have more monotonic attention than the baselines (see  $\%$  mono and  $L_{MONO}$ ). We observe large differences both in terms of WER and PER across multiple runs for the baseline, especially for the small data set. We therefore report the average result of three runs with standard deviations for each model.
Attention in the RNN baselines is already quite monotonic, but we observe small improvements with  $\delta = 0.5$ . For transformer models, on the other hand,  $\delta > 0$  seems to harm the performance, therefore we only report results with  $\delta = 0$ . In general, multihead attention in the transformer does not seem to benefit much from enforced monotonicity.
SacreBLEU (Post, 2018): BLEU+case.mixed+numrefs.1 +smooth.exp+tok.13a+version.1.3.6
${}^{9}$  Standard deviation on NETtalk with RNN is  $> {1.2}$  WER and  $> {0.27}$  PER across baseline runs.
Grapheme-to-Phoneme Conversion
WER↓	PER↓	δ = 0.0	δ = 0.5	δ = 1.0
%mono	LMONO	%mono	LMONO	%mono	LMONO
RNN
Wu et al. (2018b)	28.20	6.8
Wu and Cotterell (2019)	28.20	6.9
baseline (λ = 0)	28.76±0.73	7.16±0.16	84.7%	2.91e-04	84.0%	3.94e-03	26.8%	1.03e-01
λ = 0.1, δ = 0.0	28.88±0.32	7.16±0.03	84.8%	1.24e-04
λ = 0.1, δ = 0.5	28.55±0.18	7.13±0.09	84.3%	1.74e-03
λ = 0.1, δ = 1.0	29.02±0.55	7.32±0.19	44.5%	4.05e-02
Transformer
Wu et al. (2021)	27.63	6.9
baseline	27.79±0.24	7.00±0.09	77.0%	7.26e-02
λ = 0.1, δ = 0.0, h = all:	27.99±0.60	7.11±0.18	84.6%	5.12e-05
Table 1: Results for graphene-to-phoneme, monotonicity loss for transformer on all layers and heads. Average over three runs with independent seeds with  $\lambda = 0.1$ . Our best models are marked in bold.
Morph. Infl. High Resource	Morph. Infl. Medium Resource
ACC ↑	LEV ↓	%mono	LMONO	ACC ↑	LEV ↓	%mono	LMONO
RNN
Wu et al. (2018b)	93.60	0.128	-	-
Wu and Cotterell (2019)	94.81	0.123	-	-
baseline (λ = 0)	94.97±0.06	0.098±0.002	65.5%	1.17	78.15±0.24	0.441±0.005	64.6%	1.16
λ = 0.1, δ = 0.0	94.63±0.01	0.105±0.002	83.5%	3.78e-4	74.11±0.35	0.560±0.009	84.3%	1.63e-3
Transformer
Wu et al. (2021)	95.59	0.088	-	-
baseline (λ = 0)	95.05±0.03	0.097±0.001	58.1%	1.34	81.33±0.02	0.378±0.001	58.1%	1.35
λ = 0.1, δ = 0.1, h = all	94.98±0.07	0.099±0.002	87.5%	4.49e-4	81.02±0.17	0.383±0.001	85.7%	1.45e-3
Table 2: Results for morphological inflection, monotonicity loss for transformer on all layers and heads. Average over three runs with independent seeds with  $\lambda = 0.1$ . Our best models are marked in bold.
Transliteration	Dialect Normalization
ACC ↑	MFS ↑	%mono	LMONO	BLEU ↑	%mono	LMONO
RNN
Wu et al. (2018b)	41.10	89.40	-	-
Wu and Cotterell (2019)	41.20	89.50	-	-
baseline (λ = 0)	39.53±0.56	89.06±0.06	74.4%	0.06	33.41±0.39	83.1%	0.42
λ = 0.1, δ = 0.0	40.03±0.39	89.18±0.04	81.7%	1.4e-3	33.29±0.23	90.2%	0.10
Transformer
Wu et al. (2021)	43.39	89.70	-	-
baseline (λ = 0)	42.08±0.55	89.63±0.04	69.1%	0.12	32.83±0.20	71.7%	1.23
λ = 0.1, δ = 0.0, h = all	41.32±0.53	89.47±0.08	82.2%	7.1e-4	32.17±0.78	91.1%	0.05
Table 3: Results for transliteration and dialect normalization, all experiments with  $\delta = 0$ . Monotonicity Loss for transformer on all layers and heads. Average over three runs with independent seeds with  $\lambda = 0.1$ . Our best models are marked in bold.
For morphological inflection, we show the average results over all 51 languages in Table 2. Our RNN baseline is slightly better than previous work, whereas our transformer baseline performs slightly worse. We notice that the transformer models trained with  $\delta = 0$  on the morphological inflection tasks result in the model always attending to the same source position at every decoding state. We therefore set  $\delta$  to 0.1 for transformer models trained on this task. For the remaining tasks, we report results with  $\delta$  set to 0 and  $\lambda$  always set to 0.1 so as not to overfit hyperparameters on each task.
The baseline monotonicity loss for this task is higher than for grapheme-to-phoneme conversion but training with the monotonicity loss can drastically increase the monotonicity of the attention mechanisms. This can be seen both in the lower monotonicity score and the higher percentage of decoding states where the average source attention position increases from the previous state. In terms of performance, we do not see an improvement over the baselines.
Our results for transliteration are shown in Table 3 (average over all 11 datasets). Again, we can see that the monotonicity loss effectively biases the attention towards a more monotonic behavior, decreasing the monotonicity score and increasing the percentage of decoding states where the average source attention position increases. In terms of performance, there is a small gain for RNNs both in word-level accuracy and character-level mean F-score. Training with the monotonicity loss does not improve the performance of the transformer compared to the baseline.