Year: 2021

Abstract:
Flexible neural sequence models outperform grammarand automaton-based counterparts on a variety of tasks. However, neural models perform poorly in settings requiring compositional generalization beyond the training data—particularly to rare or unseen subsequences. Past work has found symbolic scaffolding (e.g. grammars or automata) essential in these settings. We describe R&R, a learned data augmentation scheme that enables a large category of compositional generalizations without appeal to latent symbolic structure. R&R has two components: recombination of original training examples via a prototype-based generative model and resampling of generated examples to encourage extrapolation. Training an ordinary neural sequence model on a dataset augmented with recombined and resampled examples significantly improves generalization in two language processing problems—instruction following (SCAN) and morphological analysis (SIGMORPHON 2018)—where R&R enables learning of new constructions and tenses from as few as eight initial examples.

text: 1 INTRODUCTION
How can we build machine learning models with the ability to learn new concepts in context from little data? Human language learners acquire new word meanings from a single exposure (Carey & Bartlett, 1978), and immediately incorporate words and their meanings productively and compositionally into larger linguistic and conceptual systems (Berko, 1958; Piantadosi & Aslin, 2016). Despite the remarkable success of neural network models on many learning problems in recent years—including one-shot learning of classifiers and policies (Santoro et al., 2016; Wang et al., 2016)—this kind of few-shot learning of composable concepts remains beyond the reach of standard neural models in both diagnostic and naturalistic settings (Lake & Baroni, 2018; Bahdanau et al., 2019a).
Consider the few-shot morphology learning problem shown in Fig. 1, in which a learner must predict various linguistic features (e.g. 3rd person, SinGular, PRESent tense) from word forms, with only a small number of examples of the PAST tense in the training set. Neural sequenceto-sequence models (e.g. Bahdanau et al., 2015) trained on this kind of imbalanced data fail to predict past-tense tags on held-out inputs of any kind (Section 5). Previous attempts to address this and related shortcomings in neural models have focused on explicitly encouraging rule-like behavior by e.g. modeling data with symbolic grammars (Jia & Liang, 2016; Xiao et al., 2016; Cai et al., 2017) or applying rule-based data augmentation (Andreas, 2020). These procedures involve highly task-specific models or
generative assumptions, preventing them from generalizing effectively to less structured problems that combine rule-like and exceptional behavior. More fundamentally, they fail to answer the question of whether explicit rules are necessary for compositional inductive bias, and whether it is possible to obtain “rule-like” inductive bias without appeal to an underlying symbolic generative process.
This paper describes a procedure for improving few-shot compositional generalization in neural sequence models without symbolic scaffolding. Our key insight is that even fixed, imbalanced training datasets provide a rich source of supervision for few-shot learning of concepts and composition rules. In particular, we propose a new class of prototype-based neural sequence models (c.f. Gu et al., 2018) that can be directly trained to perform the kinds of generalization exhibited in Fig. 1 by explicitly recombining fragments of training examples to reconstruct other examples. Even when these prototype-based models are not effective as general-purpose predictors, we can resample their outputs to select high-quality synthetic examples of rare phenomena. Ordinary neural sequence models may then be trained on datasets augmented with these synthetic examples, distilling the learned regularities into more flexible predictors. This procedure, which we abbreviate R&R, promotes efficient generalization in both challenging synthetic sequence modeling tasks (Lake & Baroni, 2018) and morphological analysis in multiple natural languages (Cotterell et al., 2018).
By directly optimizing for the kinds of generalization that symbolic representations are supposed to support, we can bypass the need for symbolic representations themselves: R&R gives performance comparable to or better than state-of-the-art neuro-symbolic approaches on tests of compositional generalization. Our results suggest that some failures of systematicity in neural models can be explained by simpler structural constraints on data distributions and corrected with weaker inductive bias than previously described.1

heading: 2 BACKGROUND AND RELATED WORK
text: Compositional generalization Systematic compositionality—the capacity to identify rule-like regularities from limited data and generalize these rules to novel situations—is an essential feature of human reasoning (Fodor et al., 1988). While details vary, a common feature of existing attempts to formalize systematicity in sequence modeling problems (e.g. Gordon et al., 2020) is the intuition that learners should make accurate predictions in situations featuring novel combinations of previously observed input or output subsequences. For example, learners should generalize from actions seen in isolation to more complex commands involving those actions (Lake et al., 2019), and from relations of the form r(a,b) to r(b,a) (Keysers et al., 2020; Bahdanau et al., 2019b). In machine learning, previous studies have found that standard neural architectures fail to generalize systematically even when they achieve high in-distribution accuracy in a variety of settings (Lake & Baroni, 2018; Bastings et al., 2018; Johnson et al., 2017).
Data augmentation and resampling Learning to predict sequential outputs with rare or novel subsequences is related to the widely studied problem of class imbalance in classification problems. There, undersampling of the majority class or oversampling of the minority class has been found to improve the quality of predictions for rare phenomena (Japkowicz et al., 2000). This can be combined with targeted data augmentation with synthetic examples of the minority class (Chawla et al., 2002). Generically, given a training dataset D, learning with class resampling and data augmentation involves defining an augmentation distribution p̃(x, y | D) and sample weighting function u(x, y) and maximizing a training objective of the form:
L(θ) = 1|D| ∑ x∈D log pθ(y | x)︸ ︷︷ ︸
Original training data
+ E(x,y)∼p̃ u(x, y) log pθ(y | x)︸ ︷︷ ︸ Augmented data . (1)
In addition to task-specific model architectures (Andreas et al., 2016; Russin et al., 2019), recent years have seen a renewed interest in data augmentation as a flexible and model-agnostic tool for encouraging controlled generalization (Ratner et al., 2017). Existing proposals for sequence models are mainly rule-based—in sequence modeling problems, specifying a synchronous context-free grammar (Jia & Liang, 2016) or string rewriting system (Andreas, 2020) to generate new examples. Rule-based data augmentation schemes that recombine multiple training examples have been proposed
1Code for all experiments in this paper is available at https://github.com/ekinakyurek/compgen. We implemented our experiments in Knet (Yuret, 2016) using Julia (Bezanson et al., 2017).
for image classification (Inoue, 2018) and machine translation (Fadaee et al., 2017). While rulebased data augmentation is highly effective in structured problems featuring crisp correspondences between inputs and outputs, the effectiveness of such approaches involving more complicated, context-dependent relationships between inputs and outputs has not been well-studied.
Learned data augmentation What might compositional data augmentation look like without rules as a source of inductive bias? As Fig. 1 suggests, an ideal data augmentation procedure (p̃ in Eq. 1) should automatically identify valid ways of transforming and combining examples, without pre-committing to a fixed set of transformations.2 A promising starting point is provided by prototype-based models, a number of which (Gu et al., 2018; Guu et al., 2018; Khandelwal et al., 2020) have been recently proposed for sequence modeling. Such models generate data according to:
d ∼ prewrite(· | d′; θ) where d′ ∼ Unif(D); (2)
for a datasetD and a learned sequence rewriting model prewrite(d | d′; θ). (To avoid confusion, we will use the symbol d to denote a datum. Because a data augmentation procedure must produce complete input–output examples, each d is an (x, y) pair for the conditional tasks evaluated in this paper.) While recent variants implement prewrite with neural networks, these models are closely related to classical kernel density estimators (Rosenblatt, 1956). But additionally—building on the motivation in Section 1—they may be viewed as one-shot learners trained to generate new data d from a single example.
Existing work uses prototype-based models as replacements for standard sequence models. We will show here that they are even better suited to use as data augmentation procedures: they can produce high-precision examples in the neighborhood of existing training data, then be used to bootstrap simpler predictors that extrapolate more effectively. But our experiments will also show that existing prototype-based models give mixed results on challenging generalizations of the kind depicted in Fig. 1 when used for either direct prediction or data augmentation—performing well in some settings but barely above baseline in others.
Accordingly, R&R is built on two model components that transform prototype-based language models into an effective learned data augmentation scheme. Section 3 describes an implementation of prewrite that encourages greater sample diversity and well-formedness via a multi-prototype copying mechanism (a two-shot learner). Section 4 describes heuristics for sampling prototypes d′ and model outputs d to focus data augmentation on the most informative examples. Section 5 investigates the empirical performance of both components of the approach, finding that they together provide they a simple but surprisingly effective tool for enabling compositional generalization.

heading: 3 PROTOTYPE-BASED SEQUENCE MODELS FOR DATA RECOMBINATION
text: We begin with a brief review of existing prototype-based sequence models. Our presentation mostly follows the retrieve-and-edit approach of Guu et al. (2018), but versions of the approach in this paper could also be built on retrieval-based models implemented with memory networks (Miller et al., 2016; Gu et al., 2018) or transformers (Khandelwal et al., 2020; Guu et al., 2020). The generative process described in Eq. 2 implies a marginal sequence probability:
p(d) = 1 |D| ∑ d′∈D prewrite(d | d′; θ) (3)
Maximizing this quantity over the training set with respect to θ will encourage prewrite to act as a model of valid data transformations: To be assigned high probability, every training example must be explained by at least one other example and a parametric rewriting operation. (The trivial solution where pθ is the identity function, with pθ(d | d′ = d) = 1, can be ruled out manually in
2As a concrete example of the potential advantage of learned data augmentation, consider applying the GECA procedure of Andreas (2020) to the language of strings anbn. GECA produces a training set that is substitutable in the sense of Clark & Eyraud (2007); as noted there, anbn is not substitutable. GECA will infer that a can be replaced with aab based on their common context in (aabb, aaabbb), then generate the malformed example aaababbb by replacing an a in the wrong position. In contrast, recurrent neural networks can accurately model anbn (Weiss et al., 2018; Gers & Schmidhuber, 2001). Of course, this language can also be generated using even more constrained procedures than GECA, but in general learned sequence models can capture a broader set of both formal regularities and exceptions compared to rule-based procedures.
the design of pθ.) When D is large, the sum in Eq. 3 is too large to enumerate exhaustively when computing the marginal likelihood. Instead, we can optimize a lower bound by restricting the sum to a neighborhood N (d) ⊂ D of training examples around each d:
p(d) ≥ 1 |D| ∑ d′∈N (d) prewrite(d | d′; θ) . (4)
The choice of N is discussed in more detail in Section 4. Now observe that: log p(d) ≥ log ( |N (d)| ∑ d′∈N (d) 1 |N (d)| prewrite(d | d′; θ) ) − log |D| (5)
≥ 1 |N (d)| ∑ d′∈N (d) log prewrite(d | d′; θ) + log ( |N (d)| |D| ) (6)
where the second step uses Jensen’s inequality. If all |N (d)| are the same size, maximizing this lower bound on log-likelihood is equivalent to simply maximizing∑
d′∈N (d)
log prewrite(d | d′; θ) (7)
over D—this is the ordinary conditional likelihood for a string transducer (Ristad & Yianilos, 1998) or sequence-to-sequence model (Sutskever et al., 2014) with examples d, d′ ∈ N (d).3
We have motivated prototype-based models by arguing that prewrite learns a model of transformations licensed by the training data. However, when generalization involves complex compositions, we will show that neither a basic RNN implementation of prewrite or a single prototype is enough; we must provide the learned rewriting model with a larger inventory of parts and encourage reuse of those parts as faithfully as possible. This motivates the two improvements on the prototype-based modeling framework described in the remainder of this section: generalization to multiple prototypes (Section 3.1) and a new rewriting model (Section 3.2).
3.1 n-PROTOTYPE MODELS
To improve compositionality in prototype-based models, we equip them with the ability to condition on multiple examples simultaneously. We extend the basic prototype-based language model to n prototypes, which we now refer to as a recombination model precomb:
d ∼ precomb(· | d′1:n; θ) where d′1:n def = (d′1, d ′ 2, . . . , d ′ n) ∼ pΩ(·) (9)
A multi-protype model may be viewed as a meta-learner (Thrun & Pratt, 1998; Santoro et al., 2016): it maps from a small number of examples (the prototypes) to a distribution over new datapoints consistent with those examples. By choosing the neighborhood and implementation of precomb appropriately, we can train this meta-learner to specialize in one-shot concept learning (by reusing a fragment exhibited in a single prototype) or compositional generalization (by assembling fragments of prototypes into a novel configuration). To enable this behavior, we define a set of compatible prototypes Ω ⊂ Dn (Section 4) and let pΩ def = Unif(Ω). We update Eq. 6 to feature a corresponding multi-prototype neighborhood N : D → Ω. The only terms that have changed are the conditioning variable and the constant term, and it is again sufficient to choose θ to optimize ∑ d′1:n∈N (d)
log precomb(d | d′1:n) over D, implementing precomb as described next.

heading: 3.2 RECOMBINATION NETWORKS
text: Past work has found that latent-variable neural sequence models often ignore the latent variable and attempt to directly model sequence marginals (Bowman et al., 2016). When an ordinary sequence-tosequence model with attention is used to implement precomb, even in the one-prototype case, generated
3Past work also includes a continuous latent variable z, defining:
prewrite(d | d′) = Ez∼p(z)[prewrite(d | d′, z; θ)] (8) As discussed in Section 5, the use of a continuous latent variable appears to make no difference in prediction performance for the tasks in this paper. The remainder of our presentation focuses on the simpler model in Eq. 7.
sentences often have little overlap with their prototypes (Weston et al., 2018). We describe a specific model architecture for precomb that does not function as a generic noise model, and in which outputs are primarily generated via explicit reuse of fragments of multiple prototypes, by facilitating copying from independent streams containing prototypes and previously generated input tokens.
We take precomb(d | d′1:n; θ) to be a neural (multi-)sequence-to-sequence model (c.f. Sutskever et al., 2014) which decomposes probability autoregressively: precomb(d | d′1:n; θ) = ∏ t p(d
t | d<t, d′1:n; θ). As shown in Fig. 2, three LSTM encoders—two for the prototypes and one for the input prefix— compute sequences of token representations hproto and hout respectively. Given the current decoder hidden state htout, the model first attends to both prototype and output tokens:
αiout ∝ exp(ht > out Wo h i out) i < t (10)
αkjproto ∝ exp(ht > out Wp h kj proto) k ≤ n, j ≤ |d′k| (11)
To enable copying from each sequence, we project attention weights αout and αkproto onto the output vocabulary to produce a sparse vector of probabilities:
ptcopy,out(d t = w) = ∑ i<t 1[di = w] · αiout (12)
ptcopy,proto-k(d t = w) = ∑ j≤|d′k| 1[d′k,j = w] · α kj proto (13)
Unlike rule-based data recombination procedures, however, precomb is not required to copy from the prototypes, and can predict output tokens directly using values retrieved by the attention mechanism:
htpre = [ htout, ∑ i α i outh i out, ∑ k,j α kj protoh kj proto ]
(14)
ptwrite ∝ exp(Wwritehtpre) (15) To produce a final distribution over output tokens at time t, we combine predictions from each stream:
βgate = softmax(Wgateh t out) (16)
p(dt = w | d<t, d′1:n; θ) = βgate · [ptwrite(w), ptcopy,out(w), ptcopy,proto-1(w), ..., ptcopy,proto-n(w)] (17)
This copy mechanism is similar to the one proposed by Merity et al. (2017) and See et al. (2017). We compare 1- and 2-prototype models to an ordinary sequence model and baselines in Section 5.

heading: 4 SAMPLING SCHEMES
text: The models above provide generic procedures for generating well-formed combinations of training data, but do nothing to ensure that the generated samples are of a kind useful for compositional generalization. While the training objective in Eq. 7 encourages the learned p(d) to lie close to the training data, an effective data augmentation procedure should intuitively provide novel examples of rare phenomena. To generate augmented training data, we combine the generative models of Section 3 with a simple sampling procedure that upweights useful examples.

heading: 4.1 RESAMPLING AUGMENTED DATA
text: In classification problems with imbalanced classes, a common strategy for improving accuracy on the rare class is to resample so that the rare class is better represented in training data (Japkowicz
et al., 2000). When constructing an augmented dataset using the models described above, we apply a simple rejection sampling scheme. In Eq. 1, we set:
u(d) = 1[min t p(dt) < ] . (18)
Here p(dt) is the marginal probability that the token dt appears in any example and is a hyperparameter. The final model is then trained using Eq. 1, retaining those augmented samples for which u(d) = 1. For extremely imbalanced problems, like the ones considered in Section 5, this weighting scheme effectively functions as a rare tag constraint: only examples containing rare words or tags are used to augment the original training data.

heading: 4.2 NEIGHBORHOODS AND PROTOTYPE PRIORS
text: How can we ensure that the data augmentation procedure generates any samples with positive weight in Eq. 18? The prototype-based models described in Section 3 offer an additional means of control over the generated data. Aside from the implementation of precomb, the main factors governing the behavior of the model are the choice of neighborhood function N (d) and, for n ≥ 2, the set of prior compatible prototypes Ω. Defining these so that rare tags also preferentially appear in prototypes helps ensure that the generated samples contribute to generalization. Let d1 and d2 be prototypes. As a notational convenience, given two sequences d1, d2, let d1\d2 the set of tokens in d1 but not d2, and d1∆d2 denote the set of tokens not common to d1 and d2.
1-prototype neighborhoods Guu et al. (2018) define a one-prototypeN based on a Jaccard distance threshold (Jaccard, 1901). For experiments with one-prototype models we employ a similar strategy, choosing an initial neighborhood of candidates such that
N (d) def= {d1 ∈ D : (α · |d∆d1|+ β · lev(d, d1)) < δ} (19)
where lev is string edit distance (Levenshtein, 1966) and α, β and δ are hyperparameters (discussed in Appendix B).
2-prototype neighborhoods The n ≥ 2 prototype case requires a more complex neighborhood function—intuitively, for an input d, we want each (d1, d2, . . .) in the neighborhood to collectively contain enough information to reconstruct d. Future work might treat the neighborhood function itself as latent, allowing the model to identify groups of prototypes that make d probable; here, as in existing one-prototype models, we provide heuristic implementations for the n = 2 case.
Long–short recombination: For each (d1, d2) ∈ N (d), d1 is chosen to be similar to d, and d2 is chosen to be similar to the difference between d and d1. (The neighborhood is so named because one of the prototypes will generally have fewer tokens than the other one.)
N (d) def= {(d1, d2) ∈ Ω : lev (d, d1) < δ, lev ([d\d1], d2) < δ, |d\d1| > 0, |d\d1\d2| = 0} (20)
Here [d\d1] is the sequence obtained by removing all tokens in d1 from d. Recall that we have defined pΩ(d1:n) def = Unif(Ω) for a set Ω of “compatible” prototypes. For experiments using long–short combination, all prototypes are treated as compatible; that is, Ω = D ×D. Long–long recombination: N (d) contains pairs of prototypes that are individually similar to d and collectively contain all the tokens needed to reconstruct d:
N (d) def= {(d1, d2) ∈ Ω : lev (d, d1) < δ, lev (d, d2) < δ, |d∆d1| = 1, |d\d1\d2| = 0} (21)
For experiments using long–long recombination, we take Ω = {(d1, d2) ∈ D ×D : |d1∆d2| = 1}.

heading: 5 DATASETS & EXPERIMENTS
text: We evaluate R&R on two tests of compositional generalization: the SCAN instruction following task (Lake & Baroni, 2018) and a few-shot morphology learning task derived from the SIGMORPHON 2018 dataset (Kirov et al., 2018; Cotterell et al., 2018). Our experiments are designed to explore the effectiveness of learned data recombination procedures in controlled and natural settings. Both tasks involve conditional sequence prediction: while preceding sections have discussed augmentation
procedures that produce data points d = (x, y), learners are evaluated on their ability to predict an output y from an input x: actions y given instructions x, or morphological analyses y given words x.
For each task, we compare a baseline with no data augmentation, the rule-based GECA data augmentation procedure (Andreas, 2020), and a sequence of ablated versions of R&R that measure the importance of resampling and recombination. The basic Learned Aug model trains an RNN to generate (x, y) pairs, then trains a conditional model on the original data and samples from the generative model. Resampling filters these samples as described in Section 4. Recomb-n models replace the RNN with a prototype-based model as described in Section 3. Additional experiments (Table 1b) compare data augmentation to prediction of y via direct inference (Appendix E) in the prototype-based model and several other model variants.

heading: 5.1 SCAN
text: SCAN (Lake & Baroni, 2018) is a synthetic dataset featuring simple English commands paired with sequences of actions. Our experiments aim to show that R&R performs well at one-shot concept learning and zero-shot generalization on controlled tasks where rule-based models succeed. We experiment with two splits of the dataset, jump and around right. In the jump split, which tests one-shot learning, the word jump appears in a single command in the training set but in more complex commands in the test set (e.g. look and jump twice). The around right split (Loula et al., 2018) tests zero-shot generalization by presenting learners with constructions like walk around left and walk right in the training set, but walk around right only in the test set.
Despite the apparent simplicity of the task, ordinary neural sequence-to-sequence models completely fail to make correct predictions on SCAN test set (Table 1). As such it has been a major focus of research on compositional generalization in sequence-to-sequence models, and a number of heuristic procedures and specialized model architectures and training procedures have been developed to solve it (Russin et al., 2019; Gordon et al., 2020; Lake, 2019; Andreas, 2020). Here we show that the generic prototype recombination procedure described above does so as well. We use long–short recombination for the jump split and long–long recombination for the around right split. We use a recombination network to generate 400 samples d = (x, y) and then train an ordinary LSTM with attention (Bahdanau et al., 2019b) on the original and augmented data to predict y from x. Training hyperparameters are provided in Appendix D.
Table 1 shows the results of training these models on the SCAN dataset.4 2-prototype recombination is essential for successful generalization on both splits. Additional ablations (Table 1b) show that the
4We provide results from GECA for comparison. Our final RNN predictor is more accurate than the one used by Andreas (2020), and training it on the same augmented dataset gives higher accuracies than reported in the original paper.
continuous latent variable used by Guu et al. (2018) does not affect performance, but that the copy mechanism described in Section 3.2 and the use of the recomb-2 model for data augmentation rather than direct inference are necessary for accurate prediction.

heading: 5.2 SIGMORPHON 2018
text: The SIGMORPHON 2018 dataset consists of words paired with morphological analyses (lemmas, or base forms, and tags for linguistic features like tense and case, as depicted in Fig. 1). We use the data to construct a morphological analysis task (Akyürek et al., 2019) (predicting analyses from surface forms) to test models’ few-shot learning of new morphological paradigms. In three languages of varying morphological complexity (Spanish, Swahili, and Turkish) we construct splits of the data featuring a training set of 1000 examples and three test sets of 100 examples. One test set consists exclusively of words in the past tense, one in the future tense and one with other word forms (present tense verbs, nouns and adjectives). The training set contains exactly eight past-tense and eight future-tense examples; all the rest are other word forms. Experiments evaluate R&R’s ability to efficiently learn noisy morphological rules, long viewed a key challenge for connectionist approaches to language learning (Rumelhart & McClelland, 1986). As approaches may be sensitive to the choice of the eight examples from which the model must generalize, we construct five different splits per language and use the Spanish past-tense data as a development set. As above, we use long–long recombination with similarity criteria applied to y only. We augment the training data with 180 samples from precomb and again train an ordinary LSTM with attention for final predictions. Details are provided in Appendix B.
Table 2 shows aggregate results across languages. We report the model’s F1 score for predicting morphological analyses of words in the few-shot training condition (past and future) and the standard training condition (other word forms). Here, learned data augmentation with both one- and twoprototype models consistently matches or outperforms GECA. The improvement is sometimes dramatic: for few-shot prediction in Swahili, recomb-1 augmentation reduces the error rate by 40% relative to the baseline and 21% relative to GECA. An additional baseline + resampling experiment upweights the existing rare samples rather than synthesizing new ones; results demonstrate that recombination, and not simply reweighting, is important for generalization. Table 2 also includes a finer-grained analysis of novel word forms: words in the evaluation set whose exact morphological analysis never appeared in the training set. R&R again significantly outperforms both the baseline and GECA-based data augmentation in the few-shot FUT+PAST condition and the ordinary OTHER condition, underscoring the effectiveness of this approach for “in-distribution” compositional
generalization. Finally, the gains provided by learned augmentation and GECA appear to be at least partially orthogonal: combining the GECA + resampling and recomb-1 + resampling models gives further improvements in Spanish and Turkish.

heading: 5.3 ANALYSIS
text: Why is R&R effective? Samples from the best learned data augmentation models for SCAN and SIGMORPHON may be found in the Appendix G.3 . We programaticaly analyzed 400 samples from recomb-2 models in SCAN and found that 40% of novel samples are exactly correct in the around right split and 74% in the jump split. A manual analysis of 50 Turkish samples indicated that only 14% of the novel samples were exactly correct. The augmentation procedure has a high error rate! However, our analysis found that malformed samples either (1) feature malformed xs that will never appear in a test set (a phenomenon also observed by Andreas (2020) for outputs of GECA), or (2) are mostly correct at the token level (inducing predictions with a high F1 score). Data augmentation thus contributes a mixture of irrelevant examples, label noise—which may exert a positive regularizing effect (Bishop, 1995)—and well-formed examples, a small number of which are sufficient to induce generalization (Bastings et al., 2018). Without resampling, SIGMORPHON models generate almost no examples of rare tags.
Why does R&R outperform direct inference? A partial explanation is provided by the preceding analysis, which notes that the accuracy of the data augmentation procedure as a generative model is comparatively low. Additionally, the data augmentation procedure selects only the highest-confidence samples from the model, so the quality of predicted ys conditioned on random xs will in general be even lower. A conditional model trained on augmented data is able to compensate for errors in augmentation or direct inference (Table 12 in the Appendix).
Why is Resampling without Recombination effective? One surprising feature of Table 2 is performance of the learned aug (basic) + resampling model. While less effective than the recombinationbased models, augmentation with samples from an ordinary RNN trained on (x, y) pairs improves performance for some test splits. One possible explanation is that resampling effectively acts as a posterior constraint on the final model’s predictive distribution, guiding it toward solutions in which rare tags are more probable than observed in the original training data. Future work might model this constraint explicitly, e.g. via posterior regularization (as in Li & Rush, 2020).

heading: 6 CONCLUSIONS
text: We have described a method for improving compositional generalization in sequence-to-sequence models via data augmentation with learned prototype recombination models. These are the first results we are aware of demonstrating that generative models of data are effective as data augmentation schemes in sequence-to-sequence learning problems, even when the generative models are themselves unreliable as base predictors. Our experiments demonstrate that it is possible to achieve compositional generalization on-par with complex symbolic models in clean, highly structured domains, and outperform them in natural ones, with basic neural modeling tools and without symbolic representations.

heading: ACKNOWLEDGMENTS
text: We thank Eric Chu for feedback on early drafts of this paper. This work was supported by a hardware donation from NVIDIA under the NVAIL grant program. The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center (Reuther et al., 2018) for providing HPC resources that have contributed to the research results reported within this paper.

heading: A MODEL ARCHITECTURE
text: A.1 PROTOTYPE ENCODER
We use a single layer BiLSTM network to encode hkjproto as follows:
hkproto = proj(BiLSTM (We d ′ k)) (22)
Morphology The hidden and embedding sizes are 1024. No dropout is applied. We project bidirectional embeddings to the hidden size with a linear projection. We concatenate the backward and the forward hidden states.
SCAN We choose the hidden size as 512, and embedding size as 64. We apply 0.5 dropout to the input. We project hidden vectors in the attention mechanism.
A.2 DECODER
The decoder is implemented by a single layer. In addition to the hidden state and memory cell, we also carry out a feed vector through time:
htpre = [ htout, ∑ i αiouth i out, ∑ j≤|d1| α1jprotoh 1j proto, · · · , ∑ j≤|dn| αnjprotoh nj proto ] (23)
feedt = Linearfeed(htpre) (24)
The input to the LSTM decoder at time step t is the concatenation of the previous token’s representation, previous feed vector, and a latent z vector (in the VAE model).
inputt = [Wddt−1, feedt−1, z] (25)
Morphology We use a single-layer LSTM network with a hidden size of 1024, and an embedding size of 1024. We initialize the decoder hidden states with the final hidden states of the BiLSTM encoder. feed is the same size as the hidden state. No dropout is applied in the decoder. Output calculations are provided in the original paper in equation Eq. 16. The query vector for the attentions is identically the hidden state:
queryt = htout (26)
Further details of the attention are provided in Appendix A.3.
SCAN The decoder is implemented by a single layer LSTM network with hidden size of 512, and embedding size of 64. The embedding parameters are shared with the encoder. Here the size of the feed vector is equal to embedding size, 64.
We have no self-attention for this decoder in the feed vector. There is an attention projection with dimension is 128. The details of the attention mechanism given in Appendix A.3. Finally, we use transpose of the embedding matrix to project feed to the output space.
outputt = W>e feed t (27)
outputt contains unnormalized scores before the final softmax layer. We apply 0.7 dropout to htout during both training and test. The copy mechanism will be further described in Appendix A.3.
The input to the LSTM decoder is the same as Eq. 25 except the decoder embedding matrix, Wd, shares parameters with encoder embedding matrix We. We applied 0.5 dropout to the embeddings dt−1.
The query vector for the attention is calculated by:
queryt = [htout, input t] (28)
A.3 ATTENTION AND COPYING
We use the attention mechanism described in Vaswani et al. (2017) with slight modifications.
Morphology We use a linear transformation for key while retaining an embedding size of 1024, and leave query and value transformations as the identity. We do not normalize by the square root of the attention dimension. The query vector is described in the decoder Appendix A.2. The copy mechanism for the morphology task is explained in the paper in detail.
SCAN We use the nonlinear tanh transformation for key, query and value. That the attention scores are calculated separately for each prototype using different parameters as well as the normalization i.e. obtaining α’s is performed separately for each prototype.
The copy mechanism for this task is slightly different and follows Gu et al. (2016) We normalize prototype attention scores and output scores jointly. Let ᾱi represent attention weights for each prototype sequence before normalization. Then, we concatenate them to the output vector in Eq. 27.
finalt = [outputt, ᾱ1, ..., ᾱn]
We obtain a probability vector via a final softmax layer:
probt = softmax(finalt)
That size of this probability vector is vocabulary size plus the total length all prototypes. We then project this into the output space by:
pt(w) = probt(indices(w))
where indices finds all corresponding scores in probt for token w where there might be more than one element for a given w. This is because one score can come from the outputt region, and others from the prototype regions of probt. During training we applied 0.5 dropout to the indices from outputt. Thus, the model is encouraged to copy more.

heading: B NEIGHBORHOODS AND SAMPLING
text: In the Eq. 20 and Eq. 21 we expressed the generic form of neighborhood sets. Here we provide the implementation details.
SCAN In the jump split, we use long-short recombination with δ = 0.5. In around right we use long-long recombination with δ = 0.5, and construct Ω so that the first and second prototypes to differ by a single token. We randomly pick k < 10× 3 (10 different first prototypes, and 3 different second prototypes for each of them) prototype pairs that satisfy these conditions. For the recomb-1 experiment, we use the same neighborhood setup except but consider only the k < 10 first prototypes.
Sampling In the jump split, we used beam search with beam size 4 in the decoder. We calculate the mean and standard deviation over the lengths of both among the first d′1 and the second d ′ 2 prototypes in the train set. Then, during the sampling, we expect the first and second prototypes whose length is shorter than their respective mean plus standard deviation. This decision is based on the fact that the part of the Ω that the model is exposed to is determined by the empirical distribution, Ω̂, that arises from training neighborhoods. When sampling, we try to pick prototypes from a distribution that are close to properties of that empirical distribution. In around right, we use temperature sampling with T = 0.4. If a model cannot sample the expected number of both novel and unique samples within a reasonable time, we increase temperature T .
Morphology We use long-long recombination, as explained in the paper, with slight modifications which leverage the structure of the task. We set Ω as:
Ω = {(d1, d2) | d′1tags 6= d′2tags, (dtags\d′1tags\d′2tags) = 0}
For the recomb-1 model N (d) utilizes tag similarity, lemma similarity and is constructed using a score function:
score1(d, d′1) = (|dtags∆d′1tags|, jaccard(dlemma, d′1lemma)) (29)
Given d, we sort training examples by using score1 as the comparison key and pick the four smallest neighbors (using a lexicographic sort) to form N (d). For the recomb-2 model, N (d) uses the same score function for the first prototype as in the recomb-1 case. The second prototype is selected using:
score2(d, d′1, d ′ 2) = ( dtags 6= d′2tags, |d′1tags∆d′2tags| ) (30)
Given x, and a scored first prototype, we do one more sort over training examples by using score2 as the comparison key. Then we pick first four neighbors for N (d). Sampling We use a mix strategy of temperature sampling with T = 0.5 and greedy sampling in which we use the former for dinput and the latter for doutput. We sample 180 unique and novel examples.

heading: C GENERATIVE MODEL TRAINING
text: Morphology All of the hyper parameters mentioned here are optimized by a grid search on the Spanish validation set. We train our models for 25 epochs5. We use Adam optimizer with learning rate 0.0001. The generative model is trained on morphological reinflection order (dlemmadtags . dinflection) from left to right, then the samples from the model are reordered for morphological analysis task (dinflection . dlemmadtags).
SCAN We use different number of epochs for jump and around right splits where all models are trained for 8 epochs in the former and 3 epochs in the latter. We use Adam optimizer with learning rate 0.002, and gradient norm clip with 1.0.

heading: D SEQ2SEQ BASELINE MODEL
text: After generating novel samples, we either concatenate them to the training data (in morphology), or sample training batches from a mixture of the original training data and the augmented data. Our conditional model is the same as the generative model used in morphology experiments, described in detail in the paper body, replacing dproto with x, and d with y.
Every conditional model’s size is the same as the corresponding generative model which was used for augmentation. This is to ensure that the conditional model and the generative model have the same capacity. We train conditional models for 150 epochs for SCAN and we used augmentation ratios of paug = 0.01 and paug = 0.2 in jump and around right, respectively. For morphology, we train the conditional models for 100 epochs, and we use all generated examples for augmentation.

heading: E DIRECT INFERENCE
text: To adapt the prototype-based model for conditional prediction, we condition the neighborhood function on the input x rather than the full datum d, as in Hashimoto et al. (2018). Candidate ys are then sampled from the generative model given the observed x while marginalizing over retrieved prototypes. Finally, we re-rank these candidates via Eq. 7 and output the highest-scoring candidate.
F VAE MODEL
Prior p(z): We use the same prior as Guu et al. (2018) given in Eq. 31. In this prior, z is defined by a norm and direction vector. The norm is sampled from the uniform distribution between zero and a maximum possible norm µmax = 10.0, and the direction is sampled uniformly from the unit hypersphere. This sampling procedure corresponds to a von Mises–Fisher distribution with concentration parameter zero.
5When training 2-proto and 1-proto models, we increment epoch counter when the entire neighborhood for every d is processed. For 0-proto, one epoch is defined canonically i.e. the entire train set.
z = znorm · zdir where znorm ∼ U(0, µmax), zdir ∼ vmF(~u, 0) (31)
For SCAN, the size of z is 32, and for morphology the size of z is 2.
Proposal Network q(z|d, d1:n): Similarly to the prior, the posterior network decomposes z into its norm and direction vectors. The norm vector is sampled from a uniform distribution at (|µ|,min(|µ|+ , µmax)), and the direction is sampled from the von Mises–Fisher distribution vmF(µ, κ) where κ = 25, = 1.0.
hfinal = ( ↽ h proto)start + ( ⇀ h proto)end
µ = tanh (Wz [hdfinal, hqfinal])
znorm ∼ U(|µ|,min(|µ|+ , µmax)) (32) zdir ∼ vmF (µ, κ) (33) z = znorm · zdir (34)

heading: G ADDITIONAL RESULTS
text: G.1 MORPHOLOGY RESULTS
In the paper, Table 2 shows morphology results for (non-VAE) models with 8 hints (past- and futuretense examples in the training set). Here, we provide additional results for different hint set sizes and model variants.
G.1.1 HINTS=4
Main 8-prototype F1 results are provided in the body of the paper. Here we provide exact match results and an extra set of comparisons to the VAE model.
Spanish Swahili Turkish FUT+PST∗ OTHER FUT+PST OTHER FUT+PST OTHER
baseline 0.733 ±0.014 0.881 ±0.012 0.811 ±0.018 0.893 ±0.011 0.750 ±0.026 0.875 ±0.021 geca 0.736 ±0.019 0.884 ±0.018 0.800 ±0.024 0.889 ±0.012 0.74 ±0.04 0.863 ±0.019 geca + resampling 0.782 ±0.024 0.867 ±0.012 0.830 ±0.021 0.885 ±0.013 0.794 ±0.032 0.865 ±0.018 learned aug 0.738 ±0.020 0.877 ±0.008 0.816 ±0.024 0.893 ±0.012 0.752 ±0.024 0.868 ±0.020 learned aug + resampling 0.745 ±0.019 0.870 ±0.012 0.866 ±0.016 0.894 ±0.013 0.787 ±0.031 0.863 ±0.021 recomb-1 0.738 ±0.021 0.877 ±0.019 0.820 ±0.018 0.896 ±0.014 0.735 ±0.033 0.874 ±0.026 recomb-1 + resampling 0.770 ±0.020 0.867 ±0.023 0.872 ±0.005 0.892 ±0.010 0.778 ±0.024 0.861 ±0.022 recomb-2 0.716 ±0.019 0.876 ±0.022 0.815 ±0.017 0.897 ±0.016 0.752 ±0.034 0.873 ±0.017 recomb-2 + resampling 0.765 ±0.023 0.868 ±0.021 0.856 ±0.015 0.888 ±0.016 0.808 ±0.018 0.868 ±0.027
G.2 SIGNIFICANCE TESTS
Tables 9, 10 and 11 sho the p-values for pairwise differences between the baseline and prototype-based models
G.3 GENERATED SAMPLES
All samples are randomly selected unless otherwise indicated.
G.3.1 SCAN
In Table 12, we present three test samples from the SCAN task along with the predictions by direct inference and the conditional model trained on the augmented data with recomb-2. Note that the augmentation procedure was able to create novel samples whose input (x) happens to be in the test set (Examples 1 and 3) while y may or may not be correct (Example 1).
Below are a set of samples from the learned aug (basic) model for SCAN dataset’s jump and around right splits, in order:
IN: run opposite and walk opposite right twice OUT: RUN TURN RIGHT TURN RIGHT RUN TURN RIGHT TURN
RIGHT WALK
IN: look around right thrice after run around thrice thrice OUT: TURN RIGHT RUN TURN RIGHT RUN TURN
RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN
TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT LOOK TURN RIGHT LOOK TURN RIGHT LOOK TURN
RIGHT LOOK TURN RIGHT LOOK
IN: look opposite right twice and walk around twice OUT: TURN RIGHT TURN RIGHT LOOK TURN RIGHT TURN
RIGHT LOOK TURN RIGHT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
IN: run opposite and thrice OUT: RUN TURN LEFT RUN RUN
IN: walk opposite right thrice turn turn right thrice OUT: TURN RIGHT TURN RIGHT TURN RIGHT
TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT WALK
IN: jump opposite right twice jump look around left OUT: TURN RIGHT TURN RIGHT JUMP TURN LEFT TURN
LEFT JUMP TURN LEFT TURN LEFT LOOK TURN LEFT TURN LEFT LOOK
IN: walk around left thrice after jump left left OUT: TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT
WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN
LEFT WALK
IN: run opposite right twice walk run left thrice OUT: TURN RIGHT TURN RIGHT RUN TURN RIGHT TURN
RIGHT WALK TURN LEFT TURN LEFT RUN TURN LEFT TURN LEFT RUN
Below are a set of samples from the recomb-1 model for SCAN dataset’s around right split. Note that there were no samples with rare tags generated by the model for the jump split:
IN: run around right after walk around left OUT: TURN LEFT WALK TURN LEFT WALK TURN LEFT RUN TURN
LEFT RUN TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
IN: look around right after jump around left OUT: TURN LEFT LOOK TURN LEFT JUMP TURN LEFT LOOK TURN
LEFT LOOK TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP
IN: look around right and jump around left OUT: TURN RIGHT LOOK TURN RIGHT LOOK TURN RIGHT LOOK TURN
LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP
IN: walk around right and turn right twice OUT: TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN
RIGHT WALK TURN RIGHT TURN RIGHT
Below are 4 samples from the recomb-2 model for each of SCAN dataset’s jump and around right splits, respectively:
IN: jump opposite left thrice after jump opposite left thrice OUT: TURN LEFT TURN LEFT JUMP TURN
LEFT TURN LEFT JUMP TURN LEFT TURN LEFT JUMP TURN LEFT TURN LEFT WALK TURN LEFT TURN LEFT WALK TURN
LEFT TURN LEFT WALK
IN: jump left thrice and jump left thrice OUT: TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT
JUMP TURN LEFT JUMP TURN LEFT JUMP IN: jump opposite right and turn around left OUT: TURN RIGHT TURN
RIGHT JUMP
TURN LEFT TURN LEFT TURN LEFT TURN LEFT
IN: turn around left and jump around left OUT: TURN LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT
JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP
IN: look right twice after run around right OUT: TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT
RUN TURN RIGHT RUN TURN RIGHT LOOK TURN RIGHT LOOK
IN: turn right twice after look around right OUT: TURN RIGHT LOOK TURN RIGHT LOOK TURN RIGHT LOOK
TURN RIGHT LOOK TURN RIGHT TURN RIGHT
IN: look twice and run around right OUT: LOOK LOOK TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN
RIGHT RUN
IN: walk opposite right twice and jump around right OUT: TURN RIGHT TURN RIGHT WALK TURN RIGHT TURN
RIGHT WALK TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP
G.3.2 MORPHOLOGY
Below are a set of samples from the learned aug (basic) model in SIGMORPHON format.
şahmiçe şahmiçende N;LOC;SG;PSS2S
karadan havaya füze karadan havaya füzel N;DAT;PL;PSS3P
ernek erneklerine N;DAT;PL;PSS3P
kiler kilerime N;DAT;SG;PSS1S
mahlep mahlebimizi N;ACC;SG;PSS1P
süzmek süzerler V;IND;3;PL;PRS;POS;DECL
âlap âlaps N;LGSPEC1;3S;SG;PRS
jöle jöleleri N;ACC;PL
envejecerse envejeciéndose V.CVB;PRS
colaxar colaxa V;IND;PRS;3;SG
pergedrer no pergedremos V;NEG;IMP;1;PL
mantear no mantees V;NEG;IMP;2;SG
flaguear no flagueen V;NEG;IMP;3;PL
malacostar malacostaría V;COND;3;SG
desinstar desinse V;POS;IMP;3;SG
concretizar no concretices V;NEG;IMP;2;SG
Below are a set of samples from the learned aug (basic) + resampling model.
şaşırmak şaşırmıyor musun? V;IND;2;PL;PST;PROG;POS;INTR
ayılmak ayılmaya V;IND;1;SG;PST;DECL
pleşmek pleşmiyor muyuz? V;IND;1;PL;PST;PROG;NEG;INTR
imciyetmek imciyetmezdeğiz V;IND;1;PL;FUT;NEG;DECL
kuvaşmak kuvaşmayacağız V;IND;1;PL;FUT;NEG;DECL
yermek yermeyeceğiz V;IND;1;PL;FUT;NEG;DECL
yarıtmak yarıtmayacağız V;IND;1;PL;FUT;NEG;DECL
kelimek kelimeyeceğiz V;IND;1;PL;FUT;NEG;DECL
trasescar trasescáis V;IND;PST;2;PL;IPFV
tronar tronar V;IND;FUT;1;SG
terzcalminar terzcalminan V;IND;PST;3;PL;IPFV
esubronizar esubronizamos V;IND;PST;1;PL;IPFV
urdir urdiremos V;IND;FUT;1;PL
conder conderemos V;IND;FUT;1;PL
florear florearían V;IND;PST;3;PL;LGSPEC1;SG
sabrordar sabrordamos V;IND;PST;1;PL;IPFV
Below are a set of samples from the recomb-1 + resampling model (the best performing model in Table 2). Here we additionally annotate samples with error categories.
kovulmak kovulmaz mısınız V;IND;2;PL;FUT;NEG;INTR (Inflection and tags don’t match.)
düşünmek düşündüler V;IND;3;PL;PST;POS;DECL (Correct and novel.)
sütmek sütmez miyiz? V;IND;2;SG;FUT;NEG;INTR (Inflection and tags don’t match.)
bakmak bakmayacak mıyım? V;IND;1;PL;FUT;NEG;INTR (Inflection and tags don’t match)
döndürmek döndürecek misiniz? V;IND;2;PL;FUT;POS;INTR (Correct and novel.)
türkçeleştirtmek türkçeleştirtiyor m V;IND;2;PL;PST;PROG;NEG;INTR (Wrong inflection, novel tag.)
çalmak çalmayız V;IND;2;PL;FUT;POS;DECL (Inflection and tags don’t match.)
üsürmek üsürmezsin V;IND;2;SG;PST;NEG;DECL (Inflection and tags don’t match.)
duplicar duplicaráis V;IND;FUT;2;PL (Correct and novel)
efundar efundan V;SBJV;FUT;3;PL (Inflection and tags don’t match)
deshumanizar deshumanicas V;SBJV;PST;2;SG (Inplausible inflection.)
emular emulares V;SBJV;FUT;2;SG (Correct and also in train set.)
languidecer languidecíamos V;IND;PST;1;SG;IPFV (Inflection and tags don’t match)
nominar nominamos V;SBJV;FUT;1;PL (Novel tags, incorrect inflection.)
finciar finciare V;SBJV;FUT;1;SG (Correct and novel.)
abastar abasto V;IND;PST;1;SG (Inflection and tags don’t match)
G.4 ATTENTION HEATMAP
Here we provide a visualization copy and attention mechanism in recomb-2 model for SCAN experiments.
tu rn
ar ou
nd rig ht tw ice an d jum p lef t tw ice
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ LE
FT
I_J UM
P
I_T UR
N_ LE
FT
I_J UM
P
generated sample
<unk>
<mask>
<eow>
<bow>
#
jump
look
opposite
right
thrice
and
walk
left
run
after
twice
turn
around
I_WALK
I_TURN_RIGHT
I_JUMP
I_TURN_LEFT
I_LOOK
I_RUN
x_1-1
x_1-2
x_1-3
x_1-4
x_1-5
x_1-6
x_1-7
x_1-8
x_1-9
x_1-10
x_1-11
x_1-12
x_1-13
x_1-14
x_1-15
x_1-16
x_1-17
x_1-18
x_1-19
x_1-20
x_1-21
x_2-1
x_2-2
x_2-3
action probs
tu rn
ar ou
nd rig ht tw ice an d jum p lef t tw ice
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ LE
FT
I_J UM
P
I_T UR
N_ LE
FT
I_J UM
P
generated sample
turn
around
right
twice
and
look
left
twice
I_TURN_RIGHT
I_TURN_RIGHT
I_TURN_RIGHT
I_TURN_RIGHT
I_TURN_RIGHT
I_TURN_RIGHT
I_TURN_RIGHT
I_TURN_RIGHT
I_TURN_LEFT
I_LOOK
I_TURN_LEFT
I_LOOK
x-x1 attention
tu rn
ar ou
nd rig ht tw ice an d jum p lef t tw ice
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ RI
GH T
I_T UR
N_ LE
FT
I_J UM
P
I_T UR
N_ LE
FT
I_J UM
P
generated sample
jump
I_JUMP
x-x2 attention
Figure 3: Generation of a sample. We plot normalized output scores on the left, and attention weights to the different prototypes on the right. The prototypes are on the y axes. The model is recomb-2 model trained on SCAN jump split.

heading: H COMPUTE
text: We use a single 32GB NVIDIA V100 Volta GPU for each experiment. For every experiment, the whole pipeline which consists of training of the generative model, sampling and training of the conditional model takes less than an hour.

Review summaries:
To tackle situations where compositionality is mostly required at inference time, the paper proposes a novel data augmentation method with an RNN based generator (recombination); to make the generator generate highly compositional patterns, the paper proposes a resampling method. The methods have been tested on two benchmarks focusing on the issue, SCAN and morphological analysis. The system performs on par with recently proposed GECA for SCAN and favorably to GECA on morphological analysis. 

This paper presents a prototype-based method for data augmentation based on a generative model without rule/template based requirements. The generative model creates new input-output pairs from training fragments (recombination: rewrite model conditioned on multiple examples), and samples in low-density places (rare words) of the training data (resampling). Empirical results show that the in combination recombination and resampling perform on par with a recently introduced rule-based method, GECA.