SlowGuess/NAACL_2021_OCR huggingface dataset:
Since dialect normalization is our only sentence-level sequence-to-sequence task, it is interesting to see how the monotonicity loss works on longer sequences where more reordering is possible compared to the previous tasks. The less monotonic nature of this task is reflected in the fact that neither of our models trained towards monotonicity outperforms the non-monotonic baselines, see Table 3. Dialect normalization is also the only task where the transformer does not outperform the RNN models.
Figure 4: Monotonicity score during training on the EN-JA transliteration dataset with  $\delta = 0$ . Upper plot: RNN, lower plot: transformer (all heads).
Overall, our results show that the proposed monotonicity loss succeeds in making attention more monotonic, but effects on quality are more positive for RNNs than for transformers. We now analyze the proposed loss function in more detail.
Monotonicity Over Time
First, we plot the monotonicity score during training and compare how fast it decreases over time. We find that the monotonicity score decreases very fast for the models trained with our loss function and then stays rather constant. The baseline models show various behaviors: for some datasets and models, the score decreases over training time - suggesting that the model does learn to attend more monotonically even without the loss. For other data sets, the score is initially lower and increases over training time, and, for some, the score stays more or less constant. What all baselines have in common, is that the monotonicity score oscillates much more than when trained with the monotonicity loss. Figure 4 shows an example plot for the EN-JA transliteration dataset.
Varying Monotonicity
We can vary how much we constrain attention to be monotonic by varying the weight of the monotonicity loss function  $(\lambda)$ . We analyze how this
Figure 5: Relative BLEU scores as a function of the monotonicity loss for dialect normalization with transformer (all heads). Different data points obtained by varying  $\lambda$ .  $(\lambda \in \{0.3,0.2,0.1,0.01,0.001,0\})$
influences the performance on dialect normalization. Figure 5 shows that non-monotonic behavior (as defined by the monotonicity loss) can be reduced by a factor of 10-20 with stable or even slightly improving performance. However, BLEU drops drastically for large  $\lambda$ . This highlights the advantage of our loss function over hard monotonic attention. Through  $\lambda$  we can regulate the degree of monotonicity in the attention mechanism, which can be beneficial for tasks where hard monotonic attention would be too strict.
Monotonicity Loss on Single Heads
Since we calculate the loss on each attention component separately, we can also limit its application to specific layers and heads (in the case of multihead attention). We test how restricting the monotonic behavior to only one head per layer influences the performance of the transformer on our chosen tasks. Results are presented in Table 4. We find that monotonicity on only one head generally improves performance compared to on all heads, except for dialect normalization. For grapheme-tophoneme conversion and morphological inflection in the medium resource setting, we even see performance gains over the baseline.
Our results support the belief that the flexibility of multihead attention is key to the success of the transformer. If applied to all heads, the monotonicity loss reduces variability in the attention distribution of the different heads, i.e. with high  $\lambda$ , all heads attend to the same source position. We suspect that this severely limits the capacity of transformer models and explains why rewarding monotonicity on only one head is beneficial.
These findings are also important in the context of the work by Voita et al. (2019) who find that attention heads tend to learn specialized functions.
Having one monotonic attention head could be a complementary way to encourage more diversity amongst heads, next to disagreement regularization (Li et al., 2018). Indeed, we observe that for grapheme-to-phoneme conversion and dialect normalization the remaining heads trained without the monotonicity loss tend to become less monotonic.
Attention Maps
Attention maps are particularly interesting for dialect normalization where 1) the transformer baseline has one of the highest monotonicity losses of all our models and 2) reordering of source and target tokens is possible. Figure 6 shows the attention maps for our baseline transformer and the corresponding model trained with the monotonicity loss. The bottom sentence is an example where the alignment between the source and the target is monotonic. Here, the baseline does show tentative monotonic behavior but with the monotonicity loss, the attention follows the main diagonal much more closely. The sentence on the top, on the other hand, contains a non-monotonic alignment. For a correct alignment of the past tense of "to be", the model needs to peek at the very last token before the full stop. This is reflected in the baseline attention map where the attention at the second decoding step is highest on the third-to-last source position. However, for our model trained with the monotonicity loss, the attention follows the main diagonal and fails to mirror the correct alignment. Occasional reorderings like this may explain why the monotonicity loss did not work well for this task despite it being largely monotonic.
We propose a model-agnostic loss function that measures and rewards monotonicity and can easily be integrated into various attention mechanisms. To achieve this, we track how monotonically the average position of the attention shifts over the source sequence across time steps. We show that this loss function can be seamlessly integrated into RNNs as well as transformers. Models trained with our monotonicity loss learn largely monotonic behavior without any specific changes to the attention mechanism. While we see some performance gains in RNNs, our results show that biasing all attention heads in transformers towards monotonic behavior is undesirable. However, a bias towards monotonicity may be helpful if applied to only a subset of
G2P	Performance	heads with LMONO	heads without LMONO
WER ↓	PER ↓	%mono	LMONO	%mono	LMONO
baseline	27.79±0.24	7.00±0.09	77.0%	7.26e-02
λ = 0.1, δ = 0.0, h = all:	27.99±0.60	7.11±0.18	84.6%	5.12e-05
λ = 0.1, δ = 0.0, h = 1 :	27.70±0.37	6.96±0.07	84.9%	2.49e-05	75.1%	8.26e-02
Morph. Infl. High	ACC ↑	LEV ↓
baseline	95.05±0.03	0.097±0.001	58.1%	1.34
λ = 0.1, δ = 0.1, h = all:	94.98±0.07	0.099±0.002	87.5%	4.49e-4
λ = 0.1, δ = 0.1, h = 1 :	95.00±0.03	0.098±0.000	89.3%	6.49e-5	59.6%	1.35
Morph. Infl. Medium	ACC ↑	LEV ↓
baseline	81.33±0.02	0.378±0.001	58.1%	1.35
λ = 0.1, δ = 0.1, h = all:	81.02±0.17	0.383±0.001	85.7%	1.45e-3
λ = 0.1, δ = 0.1, h = 1 :	81.67±0.13	0.366±0.003	88.6%	4.28e-4	59.0%	1.37
Transliteration	ACC ↑	MFS ↑
baseline	42.08±0.55	89.63±0.04	69.1%	0.12
λ = 0.1, δ = 0.0, h = all:	41.32±0.53	89.47±0.08	82.2%	7.1e-4
λ = 0.1, δ = 0.0, h = 1 :	41.71±0.37	89.61±0.06	80.4%	1.12e-4	69.6%	0.11
Dialect Normalization	BLEU ↑
baseline	32.83±0.20	71.7%	1.23
λ = 0.1, δ = 0.0, h = all:	32.17±0.78	91.1%	0.05
λ = 0.1, δ = 0.0, h = 1 :	31.55±0.71	77.9%	0.01	70.5%	1.64
Table 4: Transformer results for all tasks with monotonicity on all heads vs. only on one head. Monotonicity loss is computed on all layers. Average over three runs with independent seeds. Our best models are marked in bold.
Figure 6: Transformer attention maps for the sentence shown in Figure 3; "but it was meant as compliment" and "we delete this and get to work". Left: baseline  $(\lambda = 0)$ , right: with monotonicity loss on all heads  $(\lambda = 0.1)$ .
heads.
For the future, we are interested in more sophisticated schedules for the monotonicity loss, possibly reducing  $\lambda$  over the course of training. This would help to learn monotonic behavior in the early training stages but gives the model more flexibility to deviate from such an attention pattern if needed. In this context, our loss function could also be used as an additional pretraining objective for transfer to very low-resource tasks. We would also like to test our loss function on tasks where the alignment may be harder to learn, for example in multimodal models or for long sequences. Finally, using our loss function as a way to measure monotonicity could be an interesting tool for interpretability research.
We thank the anonymous reviewers for their feedback. This project has received funding from the Swiss National Science Foundation (project nos. 176727 and 191934).
Noemi Aeipli and Simon Clematide. 2018. Parsing approaches for swiss german. In Proceedings of the 3rd Swiss Text Analytics Conference, Winterthur, Switzerland.
Roee Aharoni and Yoav Goldberg. 2017. Morphological inflection generation with hard monotonic attention. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2004-2015, Vancouver, Canada. Association for Computational Linguistics.
Alham Fikri Aji, Nikolay Bogoychev, Kenneth Heafield, and Rico Sennrich. 2020. In neural machine translation, what does transfer learning transfer? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7701-7710, Online. Association for Computational Linguistics.
Ali Araabi and Christof Monz. 2020. Optimizing transformer for low-resource neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3429-3435, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, and Colin Raffel. 2019. Monotonic infinite lookback attention for simultaneous machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1313-1323, Florence, Italy. Association for Computational Linguistics.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Chung-Cheng Chiu and Colin Raffel. 2018. Monotonic chunkwise attention. In 6th International Conference on Learning Representations, ICLR 2018, Conference Track Proceedings, Vancouver, Canada.
Tonglee Chung, Yongbin Liu, and Bin Xu. 2020. Monotonic alignments for summarization. Knowledge-Based Systems, 192:105363.
Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. Incorporating structural alignment biases into an attentional neural translation model. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 876-885, San Diego, California. Association for Computational Linguistics.
Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt
Post. 2018. The sockeye neural machine translation toolkit at AMTA 2018. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 200-207, Boston, MA. Association for Machine Translation in the Americas.
Hyunsun Hwang and Changki Lee. 2020. Linear-time korean morphological analysis using an action-based local monotonic attention mechanism. ETRI Journal, 42(1):101-107.
Jian Li, Zhaopeng Tu, Baosong Yang, Michael R. Lyu, and Tong Zhang. 2018. Multi-head attention with disagreement regularization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2897-2903, Brussels, Belgium. Association for Computational Linguistics.
Baiji Liu, Songjun Cao, Sining Sun, Weibin Zhang, and Long Ma. 2020. Multi-head monotonic chunkwise attention for online speech recognition. Computing Research Repository, arXiv:2005.00205.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Xutai Ma, Juan Miguel Pino, James Cross, Liezl Puzon, and Jiatao Gu. 2020. Monotonic multihead attention. In International Conference on Learning Representations ICLR, Addis Ababa, Ethiopia.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, and Douglas Eck. 2017. Online and linear-time attention by enforcing monotonic alignments. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2837-2846. PMLR.
Terrence J. Sejnowski and Charles R. Rosenberg. 1987. Parallel networks that learn to pronounce English text. Complex Systems, 1:145-168.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich and Biao Zhang. 2019. Revisiting low-resource neural machine translation: A case study. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 211-221, Florence, Italy. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008.
Elena Voita, David Talbot, Fedor Moiseev, Rico Senrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Jiewen Wu, Rafael E. Banchs, Luis Fernando D'Haro, Pavitra Krishnaswamy, and Nancy Chen. 2018a. Attention-based semantic priming for slot-filling. In Proceedings of the Seventh Named Entities Workshop, pages 22-26, Melbourne, Australia. Association for Computational Linguistics.
Shijie Wu and Ryan Cotterell. 2019. Exact hard monotonic attention for character-level transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1530-1537, Florence, Italy. Association for Computational Linguistics.
Shijie Wu, Ryan Cotterell, and Mans Hulden. 2021. Applying the transformer to character-level transduction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, Online. Association for Computational Linguistics.
Shijie Wu, Pamela Shapiro, and Ryan Cotterell. 2018b. Hard non-monotonic attention for character-level transduction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4425-4438, Brussels, Belgium. Association for Computational Linguistics.
Sevinj Yolchuyeva, Géza Németh, and Balint Gyires-Tóth. 2019. Transformer based grapheme-tophoneme conversion. In Proceedings Interspeech 2019, pages 2095-2099, Graz, Austria. ISCA.
Min Zhang, Haizhou Li, Rafael E. Banchs, and A Kumaran. 2015. Whitepaper of NEWS 2015 shared task on machine transliteration. In Proceedings of
the Fifth Named Entity Workshop, pages 1-9, Beijing, China. Association for Computational Linguistics.
Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, and Ke Xu. 2020. Scheduled DropHead: A regularization method for transformer models. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1971-1980, Online. Association for Computational Linguistics.
A.1 Character-level Mean F-score (MFS)
$$
L C S \left(c _ {i}, r _ {i}\right) = \frac {1}{2} \left(\left| c _ {i} \right| + \left| r _ {i} \right| - E D \left(c _ {i}, r _ {i}\right)\right)
$$
$$
R _ {i} = \frac {L C S (c _ {i} , r _ {i})}{| r _ {i} |}
$$
$$
P _ {i} = \frac {L C S (c _ {i} , r _ {i})}{| c _ {i} |}
$$
$$
F _ {i} = \frac {2 * R _ {i} * P _ {i}}{R _ {i} + P _ {i}}
$$
$$
M F S = \frac {1}{N} \sum_ {i = 1} ^ {N} F _ {i}
$$
Where  $c_{i}$  is the i-th candidate and  $r_{i}$  is the corresponding reference transliteration with the smallest edit distance (ED).
Training Hyperparameters Transformer
G2P	MI	TR	DN
training settings:
batch type	sentence	word
batch size	400	400/200	400	4096
max-seq-len	20:20	85:85	85:85	200:200
word-min-count	1:1
seed	1, 2, 3
model settings:
encoder	transformer
decoder	transformer
transformer-positional-embedding-type	fixed
transformer-preprocess	n
transformer-postprocess	dr
num-layers	4:4	5:5
transformer-model-size	256	512
transformer-attention-heads	4	2
num-embed	256:256	512:512
weight-tying-type	trgSoftmax	src_trg
transformer-feed-forward-num-hidden	512/1024	512	512	512
optimization settings:
optimizer	adam
optimizer-params	beta2:0.98
checkpoint interval	400
max-num-checkpoint-not-improved	10
gradient-clipping-threshold	none
learning-rate-scheduler-type	fixed-rate-inv-sqrt-t	plateau-reduce
optimized-metric	PER	bleu
label-smoothing	0.1	0.6
initial-learning-rate	0.001	0.0001
learning-rate-warmup	4000	0
initialization settings:
weight-init	xavier
weight-init-scale	3.0
weight-init-xavier-factor-type	avg
dropout settings:
transformer-dropout-attention	0
embed-dropout	0.3/0.2	0.3	0.3	0.1
transformer-drophead-attention	0.3/0.2	0.3	0.3	0.0/0.1
transformer-dropout-act	0.3/0.2	0.3	0.3	0.3
transformer-dropout-prepost	0.3/0.2	0.3	0.3	0.3
Table 5: Sockeye hyperparameters for transformer models (values with ':=' = encoder:decoder)
Training Hyperparameters RNN
G2P	MI	TR	DN
training settings:
batch type	sentence	word
batch size	20	20	50	1000
max-seq-len	20:20	85:85	85:85	200:200
word-min-count	1:1
seeds	1, 2, 3
model settings:
encoder	rnn
decoder	rnn
rnn-cell-type	lstm
num-layers	2:1	1:1
num-embed	200:200	512:512
rnn-num-hidden	400	1024
optimization settings:
learning-rate-scheduler-type	plateau-reduce
learning-rate-warmup	0
optimizer	adam
optimized-metric	PER	bleu
checkpoint interval	4000	400
max-num-checkpoint-not-improved	7	10
label-smoothing	0.0	0.2
gradient-clipping-threshold	5	-
initial-learning-rate	0.001	0.0005
learning-rate-reduce-num-not-improved	1	8
learning-rate-reduce-factor	0.5	0.7
initialization settings:
weight-init	xavier
weight-init-scale	3.0
weight-init-xavier-factor-type	avg
dropout settings:
embed-dropout	0.4	0.5
rnn-decoder-hidden-dropout	0.4	0.5
Table 6: Sockeye hyperparameters for RNN models (values with ':=' = encoder:decoder)
A.3 Model Size
RNN Models
G2P	4.5M
MI	4.5M
TR	4.5M
DN	25.1M
Transformer Models
G2P
CMUdict (ff = 1024):	7.3M
NETtalk (ff = 512):	5.3M
MI	5.3M
TR	5.3M
DN	23.2M
Table 7: Approximate model size in number of parameters for the different tasks (exact numbers can vary slightly due to variable vocabulary sizes with different data sets. G2P, MI and TR numbers correspond to the "large" configuration in Wu et al. (2018a).