SlowGuess/ACL_2020_OCR huggingface dataset:
Jacob Andreas
jda@mit.edu
We propose a simple data augmentation protocol aimed at providing a compositional inductive bias in conditional and unconditional sequence models. Under this protocol, synthetic training examples are constructed by taking real training examples and replacing (possibly discontinuous) fragments with other fragments that appear in at least one similar environment. The protocol is model-agnostic and useful for a variety of tasks. Applied to neural sequence-to-sequence models, it reduces error rate by as much as  $87\%$  on diagnostic tasks from the SCAN dataset and  $16\%$  on a semantic parsing task. Applied to n-gram language models, it reduces perplexity by roughly  $1\%$  on small corpora in several languages.
This paper proposes a rule-based data augmentation protocol for sequence modeling. Our approach aims to supply a simple and model-agnostic bias toward compositional reuse of previously observed sequence fragments in novel environments. Consider a language modeling task in which we wish to estimate a probability distribution over a family of sentences with the following finite sample as training data:
(1) a. The cat sang.
b. The wug sang.
c. The cat daxed.
In language processing problems, we often want models to generalize beyond this dataset and infer that (2a) is also probable but (2b) is not:
(2) a. The wug daxed.
b. * The sang daxed.
This generalization amounts to an inference about syntactic categories (Clark, 2000). Because cat and wug are interchangeable in (1a) and (1b), they are also likely interchangeable elsewhere; cat and sang are not similarly interchangeable. Human learners make judgments like (2) about novel lexical items
(Berko, 1958) and fragments of novel languages (Lake et al., 2019). But we do not expect such judgments from unstructured generative models trained to maximize the likelihood of the training data in (1).
A large body of work in natural language processing provides generalization to data like (2a) by adding structure to the learned predictor (Chelba and Jelinek, 1998; Chiang, 2005; Dyer et al., 2016). On real-world datasets, however, such models are typically worse than "black-box" function approximators like neural networks, even for black-box models that fail to place probability mass on either example in (2) given small training sets like (1) (Melis et al., 2018). To the extent that we believe (2a) to capture an important inductive bias, we would like to find a way of softly encouraging it without tampering with the structure of predictors that work well at scale. In this paper, we introduce a procedure for generating synthetic training examples by recombining real ones, such that (2a) is assigned non-negligible probability because it already appears in the training dataset.
The basic operation underlying our proposal (which we call GECA, for "good-enough compositional augmentation") is depicted in Figure 1: if two (possibly discontinuous) fragments of training examples appear in some common environment, then any additional environment where the first fragment appears is also a valid environment for the second.
She picks the wug up in Fresno.
She puts the wug down in Tempe.
Pat picks cats up.
: Pat puts cats down
Figure 1: Visualization of the proposed approach: two discontinuous sentence fragments (a-b, underlined) which appear in similar environments (a-b, highlighted) are identified. Additional sentences in which the first fragment appears (c) are used to synthesize new examples (d) by substituting in the second fragment.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7556-7566
GECA is crude: as a linguistic principle, it is both limited and imprecise. As discussed in Sections 3 and 4, it captures a narrow slice of the many phenomena studied under the heading of "compositionality", while also making a number of incorrect predictions about real language data. Nevertheless, GECA appears to be quite effective across a range of learning problems. In semantic parsing, it gives improvements comparable to the task-specific data augmentation approach of Jia and Liang (2016) on logical expressions, better performance than that approach on a different split of the data designed to test generalization more rigorously, and corresponding improvements on a version of the dataset with a different meaning representation language. Outside of semantic parsing, it solves two representative problems from the SCAN dataset of Lake and Baroni (2018) that are synthetic but precise in the notion of compositionality they test. Finally, it helps with some (unconditional) low-resource language modeling problems in a typologically diverse set of six languages.
Recent years have seen tremendous success at natural language transduction and generation tasks using complex function approximators, especially recurrent (Sutskever et al., 2014) and attentional (Vaswani et al., 2017) neural models. With enough training data, these models are often more accurate than approaches built on traditional tools like regular transducers and context-free grammars (Knight and Graehl, 2005), which are brittle and difficult to efficiently infer from large datasets.
However, models equipped with an explicit symbolic generative process have at least one significant advantage over the aforementioned black-box approaches: given a grammar, it is straightforward to precisely characterize how that grammar will extrapolate beyond the examples in a given training set to out-of-distribution data. Indeed, it is often possible for researchers to design the form that this extrapolation will take: smoothed n-gram language models ensure that no memorization is possible beyond a certain length (Ney et al., 1994); CCG-based semantic parsers can make immediate use of entity lexicons without having ever seen the lexicon entries used in real sentences (Zettlemoyer and Collins, 2005).
It is not the case that black-box neural models are fundamentally incapable of this kind of predictable
generalization—the success of these models at capturing long-range structure in text (Radford et al., 2019) and controlled algorithmic data (Graves et al., 2014) indicate that some representation of hierarchical structure can be learned given enough data. But the precise point at which this transition occurs is not well characterized, and it is evidently beyond the scale available in many real-world problems.
How can we improve the behavior of high-quality black-box models in these settings? There are many sophisticated tools available for improving the function approximators or loss functions themselves—structured regularization of parameters (Oh et al., 2017), posterior regularization (Ganchev et al., 2010; Hu et al., 2018), explicit stacks (Grefenstette et al., 2015) and composition operators (Bowman et al., 2016; Russian et al., 2019). These existing proposals tend to be task-and architecture-specific. But to the extent that the generalization problem can be addressed by increasing the scale of the training data, it is natural to ask whether we can address the problem by increasing this scale artificially—in other words, via data augmentation.
Data augmentation techniques, which generate auxiliary training data by performing structured transformation or combination of training examples, are widely used in computer vision (Krizhevsky et al., 2012; Zhang et al., 2017; Summers and Dinneen, 2019). Within NLP, several data augmentation approaches have been proposed for text classification (e.g. Ratner et al., 2017; Wei and Zhou, 2019); these approaches give improvements even when combined with large-scale pretraining (Hu et al., 2019). Jia and Liang (2016) study data augmentation and compositionality in specific setting of learning language-to-logical-form mappings, beginning from the principle that data is compositional if it is generated by an explicit grammar that relates strings to logical forms. This view of compositionality as determined by synchronicity between form and meaning is essentially Montagovian and well-suited to problems in formal semantics (Montague, 1973); however, it requires access to structured meaning representations with explicit types and bracketings, which are not available in most NLP applications.
Here we aim at a notion of compositionality that is simpler and more general: a bias toward identifying recurring fragments seen at training time, and re-using them in environments distinct from those
in which they were first observed. This view makes no assumptions about the availability of brackets and types, and is synchronous only to the extent that the notion of a fragment is permitted to include content from both the source and target sides. We will find that it is nearly as effective as existing approaches in the specific settings for which they were designed, but also effective on a variety of problems where they cannot be applied.
Consider again the example in Figure 1. Our data augmentation protocol aims to discover substitutable sentence fragments (underlined), with the fact that a pair of fragments appear in some common sub-sentential environment (highlighted) taken as evidence that the fragments belong to a common category. To generate a new examples for the model, an occurrence of one fragment is removed from a sentence to produce a sentence template, which is then populated with the other fragment.
Why should we expect this procedure to produce well-formed training examples? The existence of syntactic categories, and the expressibility of well-formedness rules in terms of these abstract categories, is one of the foundational principles of generative approaches to syntax (Chomsky, 1965). The observation that context provides a strong signal about a sentence fragment's category is in turn the foundation of distributional techniques for the study of language (Firth, 1957). Combining the two gives the outlines of the above procedure.
This combination has a productive history in natural language processing: when fragments are single words, it yields class-based language models (Brown et al., 1992); when fragments are contiguous spans it yields unsupervised parsers (Clark, 2000; Klein and Manning, 2002). The present data augmentation scenario is distinguished mainly by the fact that we are unconcerned with producing a complete generative model of data, or with recovering the latent structure implied by the presence of nested syntactic categories. We can still synthesize high-precision examples of well-formed sequences by identifying individual substitutions that are likely to be correct without understanding how they fit into the grammar as a whole.
Indeed, if we are not concerned with recovering linguistically plausible analyses, we need not limit ourselves to words or contiguous sentence
fragments. We can take
(3) a. She picks the wug up.
b. She puts the wug down.
as evidence that we can use picks...up wherever we can use puts...down. Indeed, given a translation dataset:
(4) a.  $I$  sing.  $\triangleright$  Canto.
b. I sing marvelously. D Canto maravillosamente.
c. I dax marvelously. Dajo maravillosamente.
we can apply the same principle to synthesize  $I$  dax.  $\triangleright$  Dajo. based on the common environment ...marvelously  $\triangleright$  . . . maravillosamente. From the perspective of a generalized substitution principle, the alignment problem in machine translation is the same as the class induction problem in language modeling, but with sequences featuring large numbers of gappy fragments and a boundary symbol  $\triangleright$ .
The only remaining question is what makes two environments similar enough to infer the existence of a common category. There is, again, a large literature on this question (including the aforementioned work in language modeling, unsupervised parsing, and alignment), but in the current work we will make use of a very simple criterion: fragments are interchangeable if they occur in at least one lexical environment that is exactly the same.
Given a window size  $k$  and sequence of  $n$  tokens  $w = w_{1}w_{2}\dots w_{n}$ , define a fragment as a set of non-overlapping spans of  $w$ , a template as a version of  $w$  with a fragment removed, and an environment as a template restricted to a  $k$ -word window around each removed fragment. Formally, (letting  $[i,j]$  denote  $\{i,i + 1,\ldots ,j\}$ ) we have:
$$
\begin{array}{l} \operatorname {f r a g m e n t s} (w) = \left\{\left\{w _ {a _ {1}.. b _ {1}}, w _ {a _ {2}.. b _ {2}}, \dots \right\}: \right. \\ 1 \leq a _ {i} <   b _ {i} \leq n, \text {a l l} [ a _ {i}, b _ {i} ] \text {d i s j o i n t} \} \tag {1} \\ \end{array}
$$
$$
\mathfrak {t p l} (w, f) = \left(w _ {j}: \forall w _ {a _ {i}.. b _ {i}} \in f. j \notin [ a _ {i}, b _ {i} ]\right) \tag {2}
$$
$$
\begin{array}{l} \operatorname {e n v} (w, f) = \{w _ {j}: \\ w _ {j} \in \operatorname {t p l} (w, f) \text {a n d} \\ \exists w _ {a _ {i}.. b _ {i}} \in f. j \in [ a _ {i} - k, b _ {i} + k ] \} \tag {3} \\ \end{array}
$$
In Figure 1(a), the underlined picks...up is one possible fragment that could be extracted from the sentence. The corresponding template is She...the wug...in Fresno, and with  $k = 1$  the environment
is She... the wug ... in. As shown in Figure 1(d), any fragment may be substituted into any template with the same number of holes. Denote this substitution operation by  $t / f$ . The data augmentation operation that defines GECA is formally stated as follows:
If a fragment occurs multiple times within a given example, all instances are replaced (see Figure 3).
Linguistic notes Despite the fact that the above operation is motivated by insights from generative syntax and distributional semantics, it should be emphasized that it is, as a statement of a general linguistic principle, obviously wrong. Counterexamples abound: in English, stress-derived nouns (e.g. record from record) will be taken as evidence that many nouns and verbs are interchangeable; in Mandarin Chinese, keshi and danshi both mean "but", but keshi alone can be used in particular constructions to mean "very".
What ultimately matters is the relative frequency of such errors: if their contribution to an inaccurate model is less than the inaccuracy caused by the original shortage of training data, the GECA still helps. In conditional problems, like the machine translation example above, such errors may be totally harmless: if we synthesize a new  $(x,y)$  pair with  $x$  outside the support of the real training data, they may not influence the model's predictions on the true support beyond providing useful general inductive bias.
Implementation Naive implementation of the boxed operation takes  $O(t^3 f^3)$  time (where  $t$  is the number of distinct templates in the dataset and  $f$  the number of distinct fragments). This can be improved to  $O(ft^2 e)$  (where  $e$  is the number of templates that map to the same environment) by building appropriate data structures (Algorithm 1).
Space requirements might still be considerable (comparable to those used by n-gram language models), and strategies from the language modeling literature can be used to reduce memory usage (Heafield, 2011). This algorithm is agnostic with respect to the choice of fragmentation and environment functions; task-specific choices are described in more detail for each experiment below.
4 Diagnostic experiments
We begin with a set of experiments on synthetic data designed to precisely test whether GECA provides the kind of generalization it was designed for. Here we use the SCAN dataset (Lake and Baroni, 2018), which consists of simple English commands paired with sequences of discrete actions (Figure 2). We focus specifically on the add primitive (jump) and add template (around right) conditions, which test whether the agent can be exposed to individual commands or modifiers (e.g. jump  $\triangleright$  JUMP) in isolation at training time, and incorporate them into more complex commands like the earlier example at test time.
We extract fragments with one gap and a maximum length of 4 tokens. The environment is taken to be the complete template. Generated examples are appended to the original dataset. As an exam-
Figure 2: Example SCAN data. Each example consists of a synthetic natural language command (left) paired with a discrete action sequence (right).
7559
jump / SCAN	jump / NACS	right / SCAN	right / NACS
seq2seq	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00	0.00 ± 0.00
+ GECA	0.87 ± 0.02	0.67 ± 0.01	0.82 ± 0.04	0.82 ± 0.03
ple of the effect of this augmentation procedure, the original jump split has 12620 training examples; GECA generates an additional 395 using 395 distinct templates and 6 distinct fragments.
With the original and augmented datasets, we train a one-layer LSTM encoder-decoder model with an embedding size of 64, a hidden size of 512, a bidirectional encoder and an attentional decoder (Hochreiter and Schmidhuber, 1997; Bahdanau et al., 2015). The model is trained using ADAM (Kingma and Ba, 2014) with a step size of 0.001 and a dropout rate of 0.5.
Results are shown in Table 1. In line with the original experiments of Lake and Baroni, the baseline sequence-to-sequence model completely fails to generalize to the test set. Applying GECA allows the learned model to successfully make most tested generalizations across single and multi-word entries, and in both instruction-to-action and action-to-instruction directions.
Analysis: examples Some synthesized examples are shown in Figure 3. Success at the add primitive condition stems from the constraint that the single example usage of the primitive must still be a valid (command, action) pair, and all verbs are valid commands in isolation. Only three examples—run  $\triangleright$  RUN, walk  $\triangleright$  WALK and look  $\triangleright$  LOOK—provide the evidence that GECA uses to synthesize to new usages of jump; if these were removed, the sequence-to-sequence model's training accuracy would be unchanged but GECA would fail to synthesize any new examples involving jump, and test accuracy would fall to zero. For the add template condition, GECA correctly replaces all occurrences of LTURN with RTURN to produce new examples of the around right template; this example highlights the usefulness of GECA's ability to discover discontinuous and non-context-free substitutions.
Analysis: dataset statistics To further understand the behavior of GECA, we conduct a final
Table 1: Sequence match accuracies on SCAN datasets, in which the learner must generalize to new compositional uses of a single lexical item ("jump") or multi-word modifier ("around right") when mapping instructions to action sequences (SCAN) or vice-versa (NACS, Bastings et al., 2018). While the sequence-to-sequence model is unable to make any correct generalizations at all, applying GECA enables it to succeed most of the time. Scores are averaged over 10 random seeds; the standard deviation across seeds is shown. All improvements are significant (paired binomial test,  $p \ll 0.001$ ).
add primitive (jump)
walk thrice after walk right
RTURN WALK WALK WALK WALK
jump opposite left thrice after turn opposite right
RTURN RTURN LTURN LTURN JUMP LTURN LTURN JUMP
LTURN LTURN JUMP
add template (around right)
look right twice and turn opposite right twice
RTURN LOOK RTURN LOOK RTURN RTURN RTURN RTURN
run around right and walk opposite right twice
RTURN RUN RTURN RUN RTURN RUN RTURN RUN RTURN
RTURN WALK RTURN RTURN WALK
Figure 3: Examples synthesized for the SCAN tasks. Underlined words belong to the filled-in fragment; the remaining text is the template. GECA synthesizes some examples that exactly capture the desired generalization, and some examples that are unrelated.
set of analyses quantifying the overlap between the synthesized data and the held-out data. We first measure full example overlap, the fraction of test examples that appear in the augmented training set. (By design, no overlap exists between the test set and the original training set.) After applying GECA,  $5\%$  of test examples for the add primitive condition and  $1\%$  of examples for the add template condition are automatically synthesized. Next we measure token co-occurrence overlap: we compute the set of (input or output) tokens that occur together in any test example, and then measure the fraction of these pairs that also occur together in some training example. For the add primitive condition, GECA increases token co-occurrence overlap from  $83\%$  to  $96\%$ ; for the add template condition it is  $100\%$  even prior to augmentation.
It is important to note that GECA, which sees only the training set, is unaware that some subset of the data is singled out for generalization testing at evaluation time. The data augmentation protocol generates a large number of spurious training examples unrelated to the desired generalization
7560
(e.g. the first example in Figure 3); however, it also generates enough new usages of the target concept that the learner generalizes successfully.
5 Semantic parsing
Next we turn to the problem of semantic parsing, which has also been a popular subject of study for questions about compositionality, generalization, and data augmentation. For the reasons discussed in Section 3, we expect qualitatively different behavior from this approach on real language data without the controlled vocabulary of SCAN.
We study four versions of the GEOQUERY dataset (Zelle, 1995), which consists of 880 English questions about United States geography, paired with meaning representations in the form of either logical expressions or SQL queries. The standard train-test split for this dataset ensures that no natural language question is repeated between the train and test sets. As Finegan-Dollak et al. (2018) note, this provides only a limited test of generalization, as many test examples feature a logical form that overlaps with the training data; they introduce a more challenging query split to ensure that neither questions nor logical forms are repeated (even after anonymizing named entities).
We extract fragments with at most 2 gaps and at most 12 tokens. On the SQL query split, the original training set contains 695 examples. GECA generates an additional 1055 using 839 distinct templates and 379 distinct fragments. For the question split we use the baseline model of Jia and Liang (2016); for the query split we use the same sequence-to-sequence model as used for SCAN and introduce the supervised copy mechanism of Finegan-Dollak et al. (2018). Environments are again taken to be identical to templates.
Results are shown in Table 2. On the split for which Jia and Liang (2016) report results, GECA achieves nearly the same improvements with weaker domain assumptions. On the remaining splits it is more accurate.
Analysis: examples Synthesized examples for the logical and SQL representations are shown in Figure 4. Despite the fact that the sequence-to-sequence model uses neither gold entities or
Query	Question
Logical forms
seq2seq	0.62 ± 0.07	0.76 ± 0.02
+ Jia et al. 16	0.61 ± 0.03	0.81 ± 0.01
+ GECA	0.65†‡± 0.06	0.78†± 0.01
+ GECA + concat	0.63 ± 0.04	0.79†± 0.01
SQL queries
Iyer et al. 17	0.40	0.66
seq2seq	0.39 ± 0.05	0.68 ± 0.02
+ GECA	0.49†± 0.02	0.68 ± 0.02
Table 2: Meaning representation exact-match accuracies on the GEOQUERY dataset. On logical forms, GECA approaches the data augmentation approach of Jia and Liang (2016) on the standard split of the data ("Question") and outperforms it on a split designed to test compositionality ("Query"). On SQL expressions, GECA leads to substantial improvements on the query split and achieves state-of-the-art results. Scores are averaged over 10 random seeds; the standard deviation across seeds is shown. $^{1}$  Significant improvement over seq2seq baseline  $(p < 0.01)$ .  $\ddagger$  Significant improvement over Jia and Liang (2016)  $(p < 0.001)$ . (A  $t$ -test is used for LF experiments and a paired binomial test for SQL.)
specialized entity linking machinery, the augmentation procedure successfully aligns natural language entity names to their logical representations and generalizes across entity choices. This procedure also produces plausible but unattested entities like a river named florida and a state named west wyoming.