SlowGuess/ACL_2020_OCR huggingface dataset:
Yann Dubois
yanndubois96@gmail.com
Gautier Dagan
gautier.dagan@gmail.com
Dieuwke Hupkes*
d.hupkes@uva.nl
Elia Bruni*
elia.bruni@gmail.com
Neural networks are surprisingly good at interpolating and perform remarkably well when the training set examples resemble those in the test set. However, they are often unable to extrapolate patterns beyond the seen data, even when the abstractions required for such patterns are simple. In this paper, we first review the notion of extrapolation, why it is important, and how one could hope to tackle it. We then focus on a specific type of extrapolation, which is especially useful for natural language processing: generalization to sequences longer than those seen during training. We hypothesize that models with a separate content- and location-based attention are more likely to extrapolate than those with common attention mechanisms. We empirically support our claim for recurrent seq2seq models with our proposed attention on variants of the Lookup Table task. This sheds light on some striking failures of neural models for sequences and on possible methods to approaching such issues.
It is indisputable that, in recent years, neural network research has made stunning progress on a wide variety of tasks that require to process sequential inputs, such as machine translation (Sutskever et al., 2014) and speech recognition (Graves et al., 2013). However, many researchers have questioned the forms of generalization that neural networks exhibit, which significantly diverges from human-like generalization (Lake and Baroni, 2017; Geirhos et al., 2018). This discrepancy with human-like generalization is particularly true when it comes to extrapolating "outside" the training space (DeLosh et al., 1997; Marcus, 1998).
As neural networks are powerful memorizers (Zhang et al., 2017) and easily learn superficial
statistical cues (Jo and Bengio, 2017), testing extrapolation and generalization to samples from the long tails of a distribution might be the only way of quantifying their capacity of abstract reasoning (Santoro et al., 2018).
Despite this benefit, little work has been done in extrapolation. A possible explanation is that the probability of encountering a test example in the extrapolation setting seems low when the training set  $\mathcal{D}$  is large. However, such an argument fails to consider the high cost of error in extrapolation settings, and this can be a barrier for real-world scenarios (e.g., self-driving cars).
In this paper, we focus on extrapolation in sequences. More precisely, how to generalize sequence-to-sequence predictors to inputs of length  $n_* > n_D$ , where  $n_D$  denotes the length of the longest sequence in the training set. Such extrapolation is crucial for language acquisition, where humans have limited learning resources to account for the unbounded nature of language. To successfully generalize, a language learner needs to process new and potentially longer sentences than previously encountered ones (Chomsky, 1956).
Accounting for this unbounded nature of language is challenging for neural networks. This issue has recently been uncovered for seq2seq models by looking at simple artificial tasks (Lake and Baroni, 2018; Liska et al., 2018; Weber et al., 2018). Liska et al. (2018) find that seq2seq architectures can converge to local minima that generalize, but rarely do. This suggests that neural networks could generalize but lack inductive biases that favor extrapolatable behavior.
In the following sections, we review the concepts of attention and extrapolation. We then argue that
<sup>1</sup>Extrapolation is still prevalent in practical scenarios as high-dimensional problems would typically require an exponentially large  $\mathcal{D}$  to be representative, and the underlying distribution may vary over time (Hooker, 2004).
* Shared senior authorship
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 403-413 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
current attention mechanisms, which are mainly responsible for recent successes in natural language processing (NLP), are unlikely to extrapolate as they depend on the content of trained embeddings. This leads us to introduce a novel location-based attention that is loosely inspired by human visual attention. To avoid gaining extrapolation capabilities at the cost of expressivity, we introduce an attention mixer that combines content- and position-based attentions. Finally, we show that recurrent models equipped with this new attention mechanism can extrapolate to longer sequences.
2 Extrapolation
Extrapolation is often used but rarely formally defined. Ebert et al. (2014) have found that when extrapolation is explicitly defined, it often refers to points outside a hull delimited by the training set. E.g., rectangular hull, concave hull, or convex hull. In this work we use the rectangle hull definition (Brooks et al., 1988), as any model which is extrapolatable for this region would also be extrapolatable for the convex and concave definition.
Given any finite training dataset  $\mathcal{D} := \{\mathbf{x}^{(n)}\}_{n=1}^{N} \subset \mathbb{R}^d$ , we define the interpolation domain to be the  $d$ -dimensional interval  $\mathcal{I}_{inter} := \prod_{i=1}^{d} [\min_n x_i^{(n)}, \max_n x_i^{(n)}]$  and the extrapolation domain its complement  $\mathcal{I}_{extra} := \mathbb{R}^d \setminus \mathcal{I}_{inter}$ . In other words, we define a test example  $\mathbf{x}^*$  to be in the extrapolation setting if at least one of its features  $x_j^*$  is larger or smaller than any values it took during training (Figure 1).
Figure 1: Schematic extrapolation setting for  $d = 2$ .
Throughout this paper, we assume that neural networks with inputs or temporary representations in  $\mathcal{I}_{extra}$  will break. Indeed, for a given target function  $t:\mathbb{R}^d\to \mathbb{R}$  to approximate, there is an infinite amount of predictors that satisfy  $f(\mathbf{x}) = t(\mathbf{x}),\forall \mathbf{x}\in \mathcal{I}_{inter}\subset \mathbb{R}^{d}$ . Without any
additional constraints, it is thus extremely unlikely that  $f(\mathbf{x}) = t(\mathbf{x}), \forall \mathbf{x} \in \mathbb{R}^d$ . This could explain why neural networks have empirically been found to break in extrapolation settings (Lohninger, 1999; Hettiarachchi et al., 2005; Mitchell et al., 2018).
The rest of the paper discusses how to constrain representations used by our neural models  ${}^{2}$  to be in  ${\mathcal{I}}_{\text{inter }}$  regardless of the source sentence length, without decreasing their expressivity.
3 Desiderata
First and foremost, we would like a model that can extrapolate to sequences longer than the longest training one  $n_{\mathcal{D}}$  (Extrapolation Constraint). As previously discussed, models with inputs or temporary representations in  $\mathcal{I}_{extra}$  will very likely break. To satisfy the extrapolation constraint, neural models should thus not depend on features that take values in  $\mathcal{I}_{extra}$  for sequences longer than  $n_{\mathcal{D}}$ .
Second, our model should be able to learn very complex positional attention patterns (Positional Patterns Constraint). Finally, although the position of words in a sentence is important, many tasks depend on their semantics. The model should thus still be able to learn content-based attention patterns (Content Patterns Constraint).
In the following section, we review previously proposed attention-mechanism and discuss why they do not fulfill the three aforementioned desired properties.
4 Attention Mechanisms
An attention mechanism (or attender) takes as input a matrix of keys  $\mathrm{K} := \{\mathbf{k}_s^T\}_{s=1}^{n_s} \in \mathbb{R}^{n_s \times d}$  and a query  $\mathbf{q}_t \in \mathbb{R}^d$ , and outputs a probability mass function  $\pmb{\alpha}_t \in \mathbb{R}^{n_s}$  that will weight a set of values  $\mathrm{V} := \{\mathbf{v}_s^T\}_{s=1}^{n_s} \in \mathbb{R}^{n_s \times d_v}$  to generate a glimpse vector  $\mathbf{g}_t \in \mathbb{R}^{d_v}$  used for downstream tasks. Following Graves et al. (2014), it is useful to think of the attender as a memory access module,  $\pmb{\alpha}_t$  as the soft address and  $\mathbf{g}_t$  as the accessed vector.
$$
\mathbf {g} _ {t} := \sum_ {s = 1} ^ {n _ {s}} \mathbf {v} _ {s} \operatorname {a t t e n d e r} \left(\mathbf {k} _ {s}, \mathbf {q} _ {t}\right) = \mathrm {V} \boldsymbol {\alpha} _ {t} \quad (1)
$$
Figure 2 illustrates attention in a recurrent seq2seq (Cho et al., 2014), which we will use for our experiments. Both the keys and the values correspond to the set of encoder hidden states
2Although the sentence length is a scalar, the temporary representations (outputs of a hidden layer) are high dimensional.
$\mathrm{K} = \mathrm{V} = \mathrm{E} := \{\mathbf{e}_s^T\}_{s=1}^{n_s}$ , while the query corresponds to the current decoder hidden state  $\mathbf{q}_t = \mathbf{d}_t$ .
Figure 2: Attender in a recurrent seq2seq.
4.1 Content Attention
Most attention mechanisms compute "content-based addressing" (associative memory) that depend on (partial) matches of the key and query. They take as input  $\mathrm{K}$  and  $\mathbf{q}_t$  and output a semantic-based attention  $\gamma_t \in \mathbb{R}^{n_s}$ . For example, if you wanted to translate a scientific paper, you could understand the main point of the text without remembering the specific technical terms that were used. When translating, you would go back to the text and translate the jargon by knowing what to look for.
A number of content-based have been proposed, they usually differ in a score that quantifies the match between  $\mathbf{k}_s,\mathbf{q}_t$  through multinomial logits:
$$
\boldsymbol {\gamma} _ {t} := \{\operatorname {s o f t m a x} (\operatorname {s c o r e} (\mathbf {k} _ {s}, \mathbf {q} _ {t})) \} _ {s = 0} ^ {n _ {s} - 1} \tag {2}
$$
$$
\operatorname {s c o r e} \left(\mathbf {k} _ {s}, \mathbf {q} _ {t}\right) :=
$$
$$
\left\{ \begin{array}{l l} \mathbf {u} ^ {T} \tanh  \left(\left[ \widetilde {\mathbf {k}} _ {s}; \widetilde {\mathbf {q}} _ {t} \right]\right) & \text {A d d i t i v e} \\ \mathbf {k} _ {s} ^ {T} \widetilde {\mathbf {q}} _ {t} & \text {M u l t i p l i c a t i v e L u o n g e t a l . (2 0 1 5)} \\ \frac {\mathbf {k} _ {s} ^ {T} \mathbf {q} _ {t}}{\sqrt {d}} & S. D o t P r o d. V a s w a n i e t a l. (2 0 1 7) \end{array} \right. \tag {3}
$$
Where  $\widetilde{\mathbf{x}}$  is a shorthand for  $W\mathbf{x}$
4.2 Location Attention
A location (or position) attention mechanism computes "location-based addressing" (random access memory) that depend on the index of the key. It takes as input  $\mathbf{q}_t$  and outputs a location attention  $\lambda_t \in \mathbb{R}^{n_s}$ . Intuitively, it decides which value to retrieve based on its index. For example, in German sentences, the verb goes at the end of the
sentence, after a subordinate clause. When translating from German to English, it might thus make sense to directly attend to the last word in the German source sentence after encoding a subordinate clause. There are many other cases where attending to words based on their positions seems important. E.g. translating from subject-object-verb to subject-verb-object languages, or understanding the emphasis in some languages.
Despite the importance of word ordering in natural language, location-based attention is not common in seq2seq frameworks. This is probably because content-based attention can emulate location-based attention in the usual interpolation setting. Indeed, it can learn to encode a positional embedding in the hidden states of the encoder through some internal "counter". This counter is unlikely to work in the extrapolation regime, we, therefore, investigate other types of location-attention that could satisfy the extrapolation constraint.
Luong et al. (2015) proposed a location-based attention by using Equation 2 with a score that is independent of the key score  $(\mathbf{k}_s,\mathbf{q}_t) = \mathbf{w}^T\mathbf{q}_t$ . They restrict themselves to sequences of the same length, which is not of interest to our work. Such a mechanism could be extended to sequences of varying lengths but would still lack extrapolation capacity as the model still has to learn to embed the location of the index it wants to retrieve.
The Neural Turing Machine (Graves et al., 2014), post-processes the content attention by shifting its location by a predicted number of steps. We use a similar mechanism, which is extrapolatable due to the independence of the sequence length. Nevertheless, on its own, it does not allow positional-only patterns in variable-length sentences. For example, it cannot attend to the  $i^{\text{th}}$  word irrespective of the sentence length. The same argument holds for other location-based attention developed for architectures with an external memory (Sukhbaatar et al., 2015).
More recently, many location-based attention have been proposed in self-attention mechanism. These methods are usually based on sinusoidal encodings (SE), which have been proposed to take into account the word positions while bypassing the need for recurrences in encoder-decoder frameworks. In this paper, we will consider the transformer and transformerXL (relative SE) attention,
3This assumption can depend on the architecture and the inductive bias it provides (Weiss et al., 2018). For our task, we found that the assumption held for both LSTM and GRU.
which are computed as follows.
$$
\left\{ \begin{array}{l l} \frac {\left(\mathbf {k} _ {s} + \mathbf {p} _ {s}\right) ^ {T} \left(\mathbf {q} _ {t} + \mathbf {p} _ {t}\right)}{\sqrt {d}} & \text {T r a n s f o r m e r (V a s w a n i e t a l . , 2 0 1 7)} \\ \frac {\left(\widetilde {\mathbf {k}} _ {s} + \widetilde {\mathbf {p}} _ {s - t}\right) ^ {T} \left(\widetilde {\mathbf {q}} _ {t} + b\right)}{\sqrt {d}} & \text {T r a n s f o r m e r X L (D a i e t a l . , 2 0 1 9)} \end{array} \right. \tag {4}
$$
Where  $\mathbf{p}_t$  is a positional encoding with sinusoids of different frequencies at every dimension. Although powerful, the sinusoidal encoding and its variants (Shaw et al., 2018; Dai et al., 2019) lack the ability to model location patterns that depend on general word position such as "look at the  $i^{th}$  word (after ...) in the extrapolation setting. Indeed, the sinusoidal encoding for any fixed offset  $\mathbf{p}_{t + k}$  is linear in  $\mathbf{p}_t$  but not in  $k$ .
Location-based processing of attention has also been proposed as a way of constraining content-based attention to some (soft) window. Yang et al. (2018) achieve it by multiplying the content attention by the weights of a predicted Gaussian such that the model has an inductive bias towards attending to words that are close to each other. Sukhbaatar et al. (2019) use a piece-wise window to decrease the computational complexity of the model. These methods nevertheless solve a fundamentally different problem and do not allow location-only extrapolatable patterns of attention.
In this section, we propose a location attender that can satisfy the extrapolation and positional patterns constraint. We then discuss how to incorporate content attention to satisfy the content patterns constraint.
5.1 Location Attender
We would like our position attention to be loosely reminiscent of human attention, whereby we sequentially focus on a single area of the input (e.g., words or pixels) but vaguely perceive neighboring inputs due to the eccentricity effect (Carrasco et al., 1995). The visual acuity of humans is uni-modal, symmetric, and spikes at the fovea, which corresponds to a  $0^{\circ}$  retinal eccentricity. We model this visual acuity using a Gaussian Probability Density Function (PDF) similarly to Mnih et al. (2014).<sup>4</sup>
I.e. for each step, the Location Attender models a Gaussian attention over the relative word positions.
Specifically, it generates a mean  $\mu_t$  and standard deviation  $\sigma_t$ , which are used to compute the location attention given the values of the PDFs at the relative indices  $r_s \coloneqq \frac{s}{n_s - 1}$  of the keys:
$$
\boldsymbol {\lambda} _ {t} := \left\{\operatorname {P D F} _ {\mu_ {t}, \sigma_ {t}} (r _ {s}) \right\} _ {s = 0} ^ {n _ {s} - 1}
$$
Using relative indices  $r_s$  instead of the absolute ones  $s$  is crucial such that the generated  $\mu_t$  is bounded (in [0, 1]), thereby satisfying the extrapolation constraint.
This model, unfortunately, fails to satisfy the positional patterns constraint, as it only allows patterns of attention based on percentile positions. E.g., it can decide to attend to the  $10\%$  -percentile word but not to the  $2^{nd}$  word. This incapacity to satisfy the position pattern constraint is a general issue with commonly used attention mechanisms (including sinusoidal-based) that only becomes apparent when dealing with complex extrapolation patterns.
To have a general attention mechanism, we need a  $\mu_t$  that can: i) attend to locations based on absolute positions; ii) attend to locations based on percentile positions; iii) attend to positions based on the previous attention. We achieve this by defining one building block for each of those requirements  $(\mathbf{b}_t)$  such that their weighted average forms  $\mu_t$ , and the weights  $\pmb{\rho}_t$  are bounded outputs of the model. The three building blocks are:
- The step size  $\frac{1}{n_s - 1}$  between words allows the attention mechanism to depend on absolute positions. The generated weight is an integer, which dictates the additional number of steps to take.
- The bias term 1 enables the model to use percentile positions. The generated weight gates it (on or off).
- The average position of the previous attention  $\bar{\alpha}_{t - 1}$  that is gated by the generated weight. This ensures that the model can attend using absolute positions to words at indices not seen during training. E.g., attending to index  $n_{\mathcal{D}} + 5$  by first attending to  $n_{\mathcal{D}}$  then  $\bar{\alpha}_{t - 1} + 5$ .
The weights  $\pmb{\rho}_{t}$  are generated using a Gated Recurrent Unit (GRU) (Cho et al., 2014).  $\mu_{t}$  is clamped to [0, 1] by a linear function to yield interpretable and extrapolatable behaviour. We also
4Visual acuity is distributed in a Laplace-like distribution, but initial experiments were more encouraging using a Gaussian.
Figure 3: Proposed Location Attender. Given a resized query, the Weighter outputs the standard deviation  $\sigma_{t}$  and  $\pmb{\rho}_{t}$  which will weight the building blocks  $\mathbf{b}_t$  to compute the mean  $\mu_t$ .  $\mu_t$  and  $\sigma_{t}$  parametrize a Gaussian PDF used to compute the location attention  $\lambda_{t}$ .
force  $\sigma_t > min_\sigma$  and normalize it by  $n_s$  which respectively avoids division by 0 and makes  $\sigma_t$  comparable regardless of  $n_s$ . A graphical overview of the Location Attender can be seen in Figure 3. Formally:
$$
\boldsymbol {\omega} _ {t} := \operatorname {G R U} \left(\operatorname {R e L U} \left(\mathrm {W} ^ {(r e s i z e)} \mathbf {q} _ {t}\right)\right)
$$
$$
\sigma_ {t} := \frac {\mathrm {R e L U} (\mathrm {W} ^ {(\sigma)} \boldsymbol {\omega} _ {t}) + m i n _ {\sigma}}{n _ {s}}
$$
$$
\boldsymbol {\rho} _ {t} := a \left(\mathrm {W} ^ {(\rho)} \boldsymbol {\omega} _ {t}\right)
$$
$$
\mathbf {b} _ {t} := \{\bar {\alpha} _ {t - 1}; \frac {1}{n _ {s} - 1}; 1 \}
$$
$$
\mu_ {t} := \operatorname {c l a m p} \left(\boldsymbol {\rho} _ {t} ^ {T} \mathbf {b} _ {t}\right)
$$
$$
\lambda_ {s t} := \frac {1}{\sqrt {2 \pi \sigma_ {t} ^ {2}}} \exp \left(\frac {- (\frac {s}{n _ {s} - 1} - \mu_ {t}) ^ {2}}{2 \sigma_ {t} ^ {2}}\right)
$$
Where clamp is a leaky clamping (2 leaky ReLUs) and  $\min_{\sigma} = 0.27$ .  $a$  is the activation function that forces each of the three dimensions of  $\rho_{t}$  to take on the desired values. Namely a sigmoid activation for the gates, and the following "soft-staircase" to force the weights of the step size to be approximately integers (Figure 4):
$$
\operatorname {s o f t s t a i r} (x) := \lfloor x \rfloor + \operatorname {s i g m o i d} (2 0 (x - 0. 5 - \lfloor x \rfloor))
$$
5.2 Mix Attender
We enforce the content patterns constraint, by using a convex combination of content and location attention (Figure 5):
$$
\begin{array}{l} \boldsymbol {\alpha} _ {t} := \%_ {t} ^ {(\lambda)} \boldsymbol {\lambda} _ {t} + (1 - \%_ {t} ^ {(\lambda)}) \boldsymbol {\gamma} _ {t} \\ \%_{t}^{(\lambda)}:= \operatorname {sigmoid}(\mathrm{W}^{(\%)}\mathbf{q}_{t}) \\ \end{array}
$$
Figure 4: Soft staircase activation function.
Figure 5: Mix Attender. The output  $\alpha_{t}$  is a convex combination of the content and location attention.
The fact that humans generate and understand unbounded sentences with a finite experience is often used as proof of the principle of compositionality (Szab, 2017). Following this argument, methods that can extrapolate to longer sequences should exhibit some compositionality.
Based on this observation, we evaluate on a compositionality-specific artificial task, lookup tables (Liska et al., 2018), but extend it to better quantify extrapolation. This task is especially interesting to us, as there is a clear notion of what a good attention pattern should look like, making it easy to qualitatively and quantitatively analyze attentive models. It is a well-controlled task, which allows us to uncover challenges that prevent models from extrapolating on real-world data.
6.1.1 Long Lookup Tables
The lookup tables task consists in sequentially applying  $k$  pre-defined lookup table functions. The lookup tables are bijective mappings on the set of
6 The extended datasets as well as scripts to generate them can be found at https://github.com/i-machine-think/machine-tasks/tree/master/LongLookupTables
5 Straight-through estimators (Bengio et al., 2013) and Gumbel-Softmax (Jang et al., 2017; Maddison et al., 2017) performed slightly worst and required predefining the maximum number of steps.
Input	Target	Target Attention
000 t1 .	000 110 <eos>	0 1 2
110 t1 .	110 110 <eos>	0 1 2
110 t2 .	110 100 <eos>	0 1 2
000 t1 t1 t2 .	000 110 110 100 <eos>	0 1 2 3 4
all 3-bit strings  $t_i: \{0,1\}^3 \to \{0,1\}^3$ . For example, if  $t_1(000) = 110$  and  $t_2(110) = 100$  then  $t_2(t_1(000)) = t_2(110) = 100$ . Following Hupkes et al. (2018), we write the operations from left to right, as well as add the inputs and temporary steps to the targets. E.g. the previous example corresponds to the input 000 t1 t2 and the target 000 110 100.
General extrapolatable seq2seq models should be able to terminate by outputting an end of sentence token  $\langle \mathrm{e} \circ \mathrm{s} \rangle$ . We thus append  $\langle \mathrm{e} \circ \mathrm{s} \rangle$  to the targets and a full stop. to the inputs.
At each decoding step, the target only depends on the previous output and the current lookup table. E.g. the last decoding step of  $000 \text{ t1 t2}$ , only depends on the previous output  $110 = t_{1}(000)$  and the current table  $t_{2}$ . The network thus has to learn the lookup table mappings and use the correct one at each step. The gold standard attention, therefore, corresponds to the position of the current lookup table. Table 1 illustrates a longer example and its correct attention.
The various train and test sets are generated by composing 6 random lookup tables  $t_1, \ldots, t_6$  that have as input and output one of the  $2^3 = 8$  possible 3-bit strings. Specifically, we use  $k = 1 \ldots 4$  composed tables in the training set,  $k = 2 \ldots 4$  for the interpolation test sets, and  $k = 5 \ldots 9$  for the extrapolation test sets.
There are 5 different extrapolation test sets, depending on their additional lengths compared to the maximum training examples (long 1, ..., long 5). We randomly select only 5000 possible examples for each of these test sets.
For the interpolation test sets, we select 3000 examples from all possible input-output pairs.
The training set contains all other possible input-output pairs, approximately 10000 examples.
6.1.2 Reversed Lookup Tables
To test whether the attention can generate more complex patterns (investigating the Positional Patterns Constraint), we also introduce a dataset which