SlowGuess/ACL_2020_OCR huggingface dataset:
reverses the order of the inputs in the previous dataset. E.g. the last example in Table 1, would be written as t2 t1 t1 000 ., the target would not change, and the attention pattern should be 3 2 1 0 4 (attend to . when outputting  $< \mathrm{eos}>$ ). Although the change seems minor, we hypothesize that such a setting will be much more complicated as the attention pattern is not monotonic and does not follow the encoding nor the decoding steps. Indeed, in the previous task, the model only needs to learn to match the  $i^{th}$  decoding step with the  $i^{th}$  encoding step.
6.1.3 Lookup Tables with Noisy Start
Table 1: Long lookup table examples.
Input	Target	Target Attention
000 t2 ! t1 .	000 110 <eos>	0 2 3 4
110 t5 t3 t1 ! t1 .	110 110 <eos>	0 4 5 6
110 ! t2 .	110 100 <eos>	0 2 3
000 t6 t3 ! t1 t1 t2 .	000 110 110 100 <eos>	0 3 4 5 6 7
Table 2: Lookup tables with noisy start examples
Finally, we introduce another variant that also requires content attention (investigating the Content Patterns Constraint). To do so, we augment each training example with a start token “!” between the input and the tables in the source sequence. We then add  $m \sim \mathcal{U}\{0,10\}$  tables  $t_i$  before the start token. The target outputs were not modified and are thus independent of the added tables. Solving this task requires to first attend to the input, then to the token which follows “!” (content attention) and finally proceed with incremental location attention. Examples of the training data are given in Table 2.
6.2 Metrics
The main metric is sequence accuracy (seqAcc), which corresponds to the accuracy of predicting the entire sequence correctly (including its length). To get insights about how the model works, we will also use two other losses.
Sequence Accuracy Before Eos (seqAccBE), which only evaluates the accuracy of the subsequence before the model generated a  $<\mathrm{eos}>$ .
Attention Loss (attnLoss), which quantifies the quality of the attention pattern before  $\langle \mathrm{e}\circ \mathrm{s} \rangle$ . It is computed as the mean squared error between the predicted and gold standard attention. The attention loss gives an indication of how far the
The loss is overly simplistic as it is symmetric around  $\bar{\alpha}_t$  even though errors in the temporal direction are less serious as the embeddings contain past information.
This makes the task harder than the one in Hupkes et al. (2018), who force termination after the right amount of steps.
model is to the ideal attention patterns required to solve the sequence.
6.3 Architecture and Baselines
Concerning baselines, we use three content attention: additive, multiplicative, scaled dot product (Eq.3). We also have two mixed content-location attention baselines: Transformer and TransformerXL (Eq.4).
To focus on the attention mechanisms, our model and the baselines all use a smaller version of the best performing recurrent seq2seq architecture on the lookup table task (Hupkes et al., 2018). The model has never been modified during our experimentation and is schematized in Figure 2. The embeddings are of dimension 64, the recurrent network is a GRU (Cho et al., 2014) with a hidden size of 128,  $50\%$  dropout (Srivastava et al., 2014) is applied on the encoder-decoder bottleneck, and a residual connection is used between the inputs (embeddings) and outputs of the encoder. Training consists of 50 epochs with the Adam (Kingma and Ba, 2015) optimizer.
7.1 Interpolation
For sanity check, we tested all the baselines and our models (with and without attention mix) on the interpolation setting of the three tasks. Our models and the best baseline (transformer attention) achieved  $100\%$  sequence accuracy (seqAcc).
7.2 Extrapolation Constraint
The major desired property of our model is to be able to extrapolate. We tested the extrapolation capacity of our location attender by evaluating its seqAcc on the long lookup table extrapolation test sets. Figure 6 shows the seqAcc of the location attender against the strongest baseline (transformer attention).
As hypothesized, the transformer attention has some extrapolation capacity, but our location attendant substantially outperforms it in this simple task. Importantly, the loss in performance in the extrapolation setting for the best baseline is abrupt and goes from  $100\%$  to  $0\%$  by adding only three tokens to the inputs. This suggests that commonly used models are brittle and cannot even extrapolate by a small amount.
Although the previous results are encouraging, we would like to understand what is holding back
Figure 6: SeqAcc for the Location Attender and best baseline on the Long Lookup Tables task (10 runs).
our model from perfectly extrapolating (Figure 6).
Figure 7: SeqAccBE for the Location Attender and best baseline on the Long Lookup Tables task (10 runs).
To do so, we computed the sequence accuracy before  $\langle \mathrm{e} \circ \mathrm{s} \rangle$  (SeqAccBE). Figure 7 shows that the model outputs are always correct but that it often terminates decoding too soon, which we will refer to as the  $\langle \mathrm{e} \circ \mathrm{s} \rangle$  problem. This suggests that the decoder keeps an internal "counter" to increase the probability of outputting  $\langle \mathrm{e} \circ \mathrm{s} \rangle$  when the decoding step is greater than the ones seen at training time. The model learns this heuristic, which is always correct during training time and can be thought of as a metric hacking. Importantly, it is not a "hard" boundary: the model is often able to extrapolate a couple of steps but usually stops before the correct number of steps.
7.3 Positional and Content Patterns Constraint
Having shown that our model can extrapolate well on a simple task, we would like to investigate
whether it can do so for tasks that require more complicated attention patterns such as the reversed and noisy task.
Although the Mix Attender, outperformed all baselines on both tasks, it was not able to get more than  $40\%$  and  $5\%$  sequence accuracy for long 1 and long 2 respectively.
Figure 8: SeqAccBE (5 runs) for the Mix Attender and best baseline on the reversed lookup tables (reverse) and lookup tables with noisy start (noisy).
Figure 8 shows that when considering seqAccBE, the Mix Attender is able to extrapolate well in the noisy setting and a little in the reverse setting. This suggests that it is not able to extrapolate well when considering sequence accuracy because it strongly suffers from the  $<\mathrm{eos}>$  problem. This is a recurrent problem in our experiments and is more likely to happen in harder tasks and larger models.
7.4 Attention Pattern
As previously discussed, variants of the lookup table task are especially interesting as we know the gold standard attention pattern. This enables evaluation of attention patterns through the MSE attention loss (attnLoss).
Table 3 shows the attention loss averaged over the three tasks. Although not perfect, the Mix Attender performs on average the best across all set
Attention	Interp.	Long 1	Long 2	Long 3	Long 4	Long 5
Scaled Dot	5.3	6.3	8.1	10.2	12.6	15.4
Multiplicative	3.1	4.6	6.3	7.9	9.9	12.4
Additive	3.1	8.4	15.6	22.2	28.7	34.8
Transformer	2.8	3.5	6.1	9.1	11.7	13.9
TransformerXL	3.0	3.9	5.3	7.1	9.1	11.4
Mix Attention	2.1	2.2	2.9	4.1	5.3	6.7
Table 3: AttnLoss for various attention models averaged over the three datasets and 5 runs.
tings. Crucially, it performs similarly in an interpolation setting and simple extrapolation setting (long 1), while all other baselines perform significantly worse after adding a single token. Even in long 2, it is competitive with all other attention mechanisms in their interpolation domain. This indicates that the model is indeed able to extrapolate by being more precise with its attention pattern.
7.5 Qualitative Analysis
In addition to enabling extrapolation, the temporary variables such as the weight given to each building block are very helpful for debugging the model and improving interpretability.
Figure 9 shows the output of a Mix Attender for the lookup tables with noisy start task. The input was sampled from the Long 4 test set. The top-left image shows the final attention. The top-right table shows the value of some interpretable variables at every decoding step. The bottom images correspond to the content and location attention.
The first decoding step uses location attention to attend to the first input. For the next three steps, the model outputs a mixing weight  $\%^{(\lambda)}\approx 0$  to focus on content attention. The content attention successfully finds the first non-noisy table (after!). It then goes back to using the location attention with  $\rho^{(\alpha)} = 1$  and  $\rho^{(1 / n)} = 1$  to generate a diagonal attention. Finally, it predicts  $< \infty >$  when attending to the end of the input ".
At each step,  $\sigma = \min_{\sigma}$  as it does not need to attend to neighboring words for this task.  $\%^{(\lambda)}$  is never exactly 0 or 1, such that the model can easily learn to switch between content and location attention as it does not collapse to using a single form of attention.
<sup>9</sup>Some baselines outperformed it in the interpolation settings of specific tasks. Namely, the additive attention in the reversed task and transformer in the noisy task.
10 A single step of content attention should be sufficient, but the model seems to consistently use three steps.
\( \% _{t}^{(\lambda)} \)	\( \mu_{t} \)	\( \sigma_{t} \)	\( \rho _{t}^{(a)} \)	\( \rho _{t}^{\left(\frac {1}{n_{s}}\right)} \)	\( \rho _{t}^{(1)} \)
0.98	0.00	0.27	1.07	1.00	0.00
0.01	0.10	0.27	1.08	1.00	0.00
0.00	0.30	0.27	1.08	1.00	0.00
0.02	0.40	0.27	1.09	1.00	0.00
0.72	0.50	0.27	1.09	1.00	0.00
0.83	0.60	0.27	1.09	1.00	0.00
0.91	0.70	0.27	1.09	1.00	0.00
0.91	0.80	0.27	1.09	1.00	0.00
0.96	0.90	0.27	1.09	1.00	0.00
0.71	1.00	0.27	1.09	1.00	0.00
Figure 9: Example output by the attention-mixer for the lookup tables with noisy start task (Long 4 test set).
In this paper, we focused on one type of extrapolation, which is especially important in NLP: generalization to longer sequences. We propose a new location-based attention, and show that it can extrapolate better than previous models while learning various attention patterns.
Despite promising initial results, our model is still unable to extrapolate perfectly for harder tasks. By analyzing its behavior, we uncovered an interesting heuristic used by seq2seq models, namely that they keep track of a decoding "counter" to know when to output the  $<\mathrm{eos}>$  token. This is a bottleneck for extrapolation, suggesting that removing this heuristic is key to reaching perfect extrapolation and should be investigated in future work.
Once the  $<\mathrm{eos}>$  problem is solved, we could test the model on real-world datasets. It would also be interesting to test such attention mechanisms in self-attentive seq2seq models without recurrence. Finally, as the location attender is not model dependent, it could be pretrained on complex location patterns and incorporated as a plug-and-play module to get extrapolatable position attention.
Taking a step back, we have shown that current
deep learning models with common attention mechanisms are unable to extrapolate well on seemingly straightforward tasks. This tends to be overlooked by the field due to standard benchmarks that can be solved using only interpolation. We hope that this paper acts as a reminder that extrapolation is a hard setting that has not been much investigated by the machine learning community. As current methods that memorize and learn superficial cues are unable to extrapolate while humans are, we believe that such a setting might help (and force) the field to come up with more human-like computational models that are capable of abstract reasoning.
Dieuwke Hupkes is funded by the Netherlands Organization for Scientific Research (NWO), through a Gravitation Grant 024.001.006 to the Language in Interaction Consortium. Elia Bruni is funded by the European Unions Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement No 790369 (MAGIC).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432.
Daniel G Brooks, Steven S Carroll, and William A Verdini. 1988. Characterizing the domain of a regression model. The American Statistician, 42(3):187-190.
Marisa Carrasco, Denise L Evert, Irene Chang, and Svetlana M Katz. 1995. The eccentricity effect: Target eccentricity affects performance on conjunction searches. Perception & Psychophysics, 57(8):1241-1261.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1724-1734. ACL.
Noam Chomsky. 1956. Three models for the description of language. IRE Trans. Information Theory, 2(3):113-124.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2978-2988. Association for Computational Linguistics.
Edward L DeLosh, Jerome R Busemeyer, and Mark A McDaniel. 1997. Extrapolation: The sine qua non for abstraction in function learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23(4):968.
Tobias Ebert, Julian Belz, and Oliver Nelles. 2014. Interpolation and extrapolation: Comparison of definitions and survey of algorithms for convex and concave hulls. In 2014 IEEE Symposium on Computational Intelligence and Data Mining, CIDM 2014, Orlando, FL, USA, December 9-12, 2014, pages 310-314. IEEE.
Robert Geirhos, Carlos R. Medina Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. 2018. Generalisation in humans and deep neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pages 7549-7561.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey E. Hinton. 2013. Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May 26-31, 2013, pages 6645-6649. IEEE.
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural tuning machines. CoRR, abs/1410.5401.
P Hettiarachchi, MJ Hall, and AW Minns. 2005. The extrapolation of artificial neural networks for the modelling of rainfallrunoff relationships. Journal of Hydroinformatics, 7(4):291-296.
Giles Hooker. 2004. Diagnostics and extrapolation in machine learning. stanford university.
Dieuwke Hupkes, Anand Singh, Kris Korrel, German Kruszewski, and Elia Bruni. 2018. Learning compositionally through attentive guidance. CoRR, abs/1805.09657.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Jason Jo and Yoshua Bengio. 2017. Measuring the tendency of cnns to learn surface statistical regularities. CoRR, abs/1711.11561.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Brenden M. Lake and Marco Baroni. 2017. Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks. CoRR, abs/1711.00350.
Brenden M. Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2879-2888. PMLR.
Adam Liska, Germán Kruszewski, and Marco Baroni. 2018. Memorize or generalize? searching for a compositional RNN in a haystack. CoRR, abs/1802.06467.
Hans Lohninger. 1999. Teach/me: Data Analysis. Springer.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1412-1421. The Association for Computational Linguistics.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous relaxation of discrete random variables. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Gary F Marcus. 1998. Rethinking eliminative connectionism. Cognitive psychology, 37(3):243-282.
Jeff Mitchell, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Extrapolation in NLP. CoRR, abs/1805.06648.
Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. 2014. Recurrent models of visual attention. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2204-2212.
Adam Santoro, Felix Hill, David G. T. Barrett, Ari S. Morcos, and Timothy P. Lillicrap. 2018. Measuring abstract reasoning in neural networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4477-4486. PMLR.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 464-468. Association for Computational Linguistics.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929-1958.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 331-335. Association for Computational Linguistics.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2440-2448.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112.
Zoltn Gendler Szab. 2017. Compositionality. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, summer 2017 edition. Metaphysics Research Lab, Stanford University.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Noah Weber, Leena Shekhar, and Niranjan Balasubramanian. 2018. The fine line between linguistic generalization and failure in Seq2Seq-attention models. In Proceedings of the Workshop on Generalization in the Age of Deep Learning, pages 24-27, New Orleans, Louisiana. Association for Computational Linguistics.
Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision rnns for language recognition. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 740-745. Association for Computational Linguistics.
Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. 2018. Modeling localness for self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4449-4458. Association for Computational Linguistics.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2017. Understanding deep learning requires rethinking generalization.