Learning to Recombine and Resample Data For Compositional Generalization
Authors: Ekin Akyürek
Afra Feyza Akyürek
Jacob Andreas
Venue: ICLR 2021 Poster (2021)
Keywords: compositional generalization
data augmentation
language processing
sequence models
generative modeling

Abstract: Flexible neural sequence models outperform grammar- and automaton-based counterparts on a variety of tasks. However, neural models perform poorly in settings requiring compositional generalization beyond the training data—particularly to rare or unseen subsequences. Past work has found symbolic scaffolding (e.g. grammars or automata) essential in these settings. We describe R&R, a learned data augmentation scheme that enables a large category of compositional generalizations without appeal to latent symbolic structure. R&R has two components: recombination of original training examples via a prototype-based generative model and resampling of generated examples to encourage extrapolation. Training an ordinary neural sequence model on a dataset augmented with recombined and resampled examples significantly improves generalization in two language processing problems—instruction following (SCAN) and morphological analysis (SIGMORPHON 2018)—where R&R enables learning of new constructions and tenses from as few as eight initial examples.

Published as a conference paper at ICLR 2021
LEARNING TO RECOMBINE AND RESAMPLE DATA FOR COMPOSITIONAL GENERALIZATION
Ekin Akyürek MIT CSAIL
Afra Feyza Akyürek Boston University akyurekēbu. edu
Jacob Andreas MIT CSAIL jdaêmi edu
akyurekêmit. t. edu
ABSTRACT
Flexible neural sequence models outperform grammar- and automaton bbsed coun- terparts on variety of tasks. However, neural models perform poorly in settings requiring compositional generalization beyond the training data cata--prticculrie rare or unseen subsequences. Past work has found symbolic scaffolding (e.g. grammars or automata) essential in these settings. We describe R&R, a learned data augmentation scheme that enables a large category of compositional general- izations without appeal to latent symbolic structure. R&R has two components: recombination of original training examples via prototype-base based generative model and resampling of generated examples to encourage extrapolation. Train- ing ordinary neural sequence model on dataset augmented with recombined and resampled examples significantly improves generalization in two language processing problems inntemmttiittttiiinn following (SCAN) and morphological analysis (SIGMORPHON 2018) -0hre R&R enables learning of new constructions and tenses from as few as eight initial examples.
INTRODUCTION
How can we build machine learning models with the abil- ity to learn new concepts in context from little data? Hu- man language learners acquire new word meanings from single exposure (Carey & Bartlett, 1978), and immedi- ately incorporate words and their meanings productively and compositionally into larger linguistic and conceptual systems (Berko, 1958; Piantadosi & Aslin, 2016). Despite the remarkable success of neural network models on many learning problems in recent years-including one-shot learning classifiers and policies (Santoro et al., 2016; Wang al., 2016) this kind of few-shot learning of com- posable concepts remains beyond the reach standard neural models in both diagnostic and naturalistic settings (Lake & Baroni, 2018; Bahdanau et al., 2019a).
(a) training data
+3
5G
PRES
+5G
PPRES
+PAST pare tag
(b) augmented data
+5G +PAST
improved generalizatiin
Consider the shot morphology learning problem shown in Fig. 1, in which a learner must predict various lin- guistic features (e.g. 3rd person, SinGular, PRESent tense) from word forms, with only a small number of examples of the PAST tense in the training set. Neural sequence- to-sequence models (e.g. Bahdanau et al., 2015) trained on this kind of imbalanced data fail to predict past-tense tags on held-out inputs of any kind (Section 5). Previous attempts to address this and related shortcomings in neural models have focused on explicitly encouraging rule-like behavior by e.g. modeling data with symbolic grammars (Jia & Liang, 2016; Xiao et al., 2016; Cai et al., 2017) applying rule- based data augmentation (Andreas, 2020). These procedures involve highly task-specific models
+3 +SG +PAST
Figure first train generative model reconstruct training pairs (ZD-y) by construct- ing them from other training pairs (a). We then perform data augmentation by sampling from this model, preferentially generating samples which y contains rare tokens or substructures (b). Dashed boxes show pre- diction targets. Conditional models trained the augmented dataset accurately prediet outputs y from new inputs requiring com- positional generalization (c).

---

Published as a conference paper at ICLR 2021
generative assumptions, preventing them from generalizing effectively to less structured problems that combine rule-like and exceptional behavior. More fundamentally. they fail to answer the question of whether explicit rules are necessary for compositional inductive bias, and whether it is possible to obtain "rule-like" inductive bias without appeal to an underlying symbolic generative process.
This paper describes procedure for improving few-shot compositional generalization in neural sequence models without symbolic scaffolding. Our key insight is that even fixed, imbalanced training datasets provide rich source of supervision for few-shot learning of concepts and composition rules. In particular, we propose a new class of prototype-based neural sequence models (c.f. Gu al., 2018) that can be directly trained to perform the kinds of generalization exhibited in Fig. by explicitly recombining fragments of training examples to reconstruct other examples. Even when these prototype-basee models are not effective as general-purposs predictors, we can resample their outputs to select high-quality synthetic examples of rare phenomena. Ordinary neural sequence models may then be trained on datasets augmented with these synthetic examples, distilling the learned regularities into more flexible predictors. This procedure, which we abbreviate R&R, promotes efficient generalization in both challenging synthetic sequence modeling tasks (Lake & Baroni, 2018) and morphological analysis in multiple natural languages (Cotterell et al., 2018).
By directly optimizing for the kinds of generalization that symbolic representations are supposed to support, bypass the need for symbolic representations themselves: R&R gives performance comparable to or better than state-of-the-an neuro-symbolic approaches on tests of compositional generalization. Our results suggest that some failures of systematicity in neural models can be explained by simpler structural constraints on data distributions and corrected with weaker inductive bias than previously described."
BACKGROUND AND RELATED WORK
Compositional generalization Systematic compositionality capacity identify rule-like regularities from limited data and generalize these rules to novel situations is an essential feature of human reasoning (Fodor et al., 1988). While details vary, a common feature of existing attempts to formalize systematicity in sequence modeling problems (e.g. Gordon et al., 2020) is the intuition that learners should make accurate predictions in situations featuring novel combinations of previously observed input or output subsequences. For example, learners should generalize from actions seen in isolation more complex commands involving those actions (Lake et al., 2019), and from relations of the form r r(a,b) to rb, a) (Keysers et al., 2020; Bahdanau et al., 2019b). In machine learning, previous studies have found that standard neural architectures fail to generalize systematically even when they achieve high in-distribution accuracy in variety of settings (Lake & Baroni, 2018; Bastings et al., 2018; Johnson et al., 2017).
Data augmentation and resampling Learning to predict sequential outputs with rare or novel subsequences is related to the widely studied problem of class imbalance in classification problems. There, undersampling of the majority class oversampling of the minority class has been found to improve the quality of predictions for rare phenomena (Japkowicz et al., 2000). This can be combined with targeted data augmentation with synthetic examples of the minority class (Chawla et al., 2002). Generically, given a training dataset DD learning with class resampling and data augmentation involves defining an augmentation distribution p(a,, y | D) and sample weighting function u(x y) and maximizing training objective of the form:
[())
DD ELED log po (y x:) + F (#a))~ u(ar, y) log po (y) x)
Original training
Angmented data
In addition to task-specific model architectures (Andreas al., 2016; Russin et al., 2019), recent years have seen renewed interest in data augmentation as flexible and model-agnostic tool for encouraging controlled generalization (Ratner al., 2017). Existing proposals for sequence models are mainly rule-based- in sequence modeling problems, specifying a synchronous context- free grammar (Jia& & Liang. 2016) string rewriting system (Andreas, 2020) to generate new examples. Rule-based data augmentation schemes that recombine multiple training examples have been proposed
'Code for all experiments in this paper is available at https: //ggthbb.bb.....//// ctwp nakyyrek//oomperr We implemented our experiments in Knet (Yuret, 2016) using Julia (Bezanson al., 2017).
2

---

Published as a conference paper at ICLR 2021
for image classification (Inoue, 2018) and machine translation (Fadaee al., 2017). While rule- based data augmentation is highly effective structured problems featuring crisp correspondences between inputs and outputs, the effectiiveness of such approaches involving more complicated, context-dependent relationships between inputs and outputs has not been well-studied.
Learned data augmentation What might compositional data augmentation look like without rules as a source of inductive bias? As Fig. suggests, an ideal data augmentation procedure (p in Eq. 1) should automatically identify valid ways of transforming and combining examples, without pre-commiting to a fixed set of transformations: 2 A promising starting point is provided by prototype-based models, number of which (Gu et al., 2018; Guu et al., 2018; Khandelwal et al., 2020) have been recently proposed for sequence modeling. Such models generate data according to:
d Prewrite (- d'; 0) where d' Unii(DDD)
for dataset D and a learned sequence rewriting model Prewnite d'; 0). (To avoid confusion, we will use the symbol to denote a datum. Because data augmentation procedure must produce complete input-output examples, each dl is an (Ic, v) pair for the conditional tasks evaluated in this paper.) While recent variants implement Prewite with neural networks, these models are closely related to classical kernel density estimators (Rosenblatt, 1956). But cetiittionnll-uuuddiiiiin in Section 1 the may be viewed as one-shot learners trained to generate new data d from a single example.
the motivation
Existing work uses prototype-bar bassd models as replacements for standard sequence models. We will show here that they are even better suited use as data augmentation procedures: they can produce high-precision examples in the neighborhood of existing training data, then be used to bootstrap simpler predictors that extrapolate more effectively. But our experiments will also show that existing prototype-basee models give mixed results on challenging generalizations of the kind depicted in Fig. 1 when used for either direct prediction or data augmentation performing well in some settings but barely above baseline in others.
Accordingly, R&R is built on two model components that transform prototype- based language models into an effective learned data augmentation scheme. Section 3 describes an implementation of Prewrite that encourages greater sample diversity and well-formedness via multi-prototype copying mechanism (a two-shot learner). Section 4 describes heuristics for sampling prototypes d' and model outputs d to focus data augmentation on the most informative examples. Section 5 investigates the empirical performance of both components of the approach, finding that they together provide they simple but surprisingly effective tool for enabling compositional generalization.
3 PROTOTYPE-BASER SEQUENCE MODELS FOR DATA RECOMBINATION
We begin with a brief review of existing prototype-based based sequence models. Our presentation mostly follows the retrieve-and-ede approach of Guu et al. (2018), but versions of the approach in this paper could also be built on retrieval-basse models implemented with memory networks (Miller et al., 2016; Gu et al., 2018) transformers (Khandelwal et al., 2020; Guu et al., 2020). The generative process described in Eq. 2 implies marginal sequence probability:
(dd)
=
1
Σ d'ED
Prewrite (d d';
D
Maximizing this quantity over the training set with respect to 0 will encourage Prewrite to act as a model of valid data transformations: To be assigned high probability, every training example must be explained by at least one other example and parametric rewriting operation. (Thh trivial solution where Po is the identity function, with po(d d' = d) = 1, can ruled out manually in
2 As a concrete example of the potential advantage of learned data augmentation, consider applying the GECA procedure of Andreas (2020) to the language of strings a a²6". GECA produces training set that is substitutable the sense of Clark & Eyraud (2007); noted there, a a""" is not substitutable. GECA will infer that can be replaced with aab based on their common context in (agbb, agabbb), then generate the malformed example anababbb by replacing an a in the wrong position. In contrast, recurrent neural networks can accurately model a "6" (Weiss et al., 2018: Gers & Schmidhuber, 2001). Of course, this language can also be generated using even more constrained procedures than GFCA, but in general learned sequence models can capture broader set of both formal regularities and exceptions compared to rule-based procedures.
3

---

Published as a conference paper at ICLR 2021
the design of po.) When D is large, the sum in Eq. 3 is too large to enumerate exhaustively when computing the marginal likelihood. Instead, we can optimize a lower bound by restricting the sum to neighborhood N (dd C D of training examples around each d:
p(d)
1
Σ
Prewrite (d d' d);
(4)
D
d'E
The choice of N is discussed in more detail in Section 4. Now observe that:
log p(d) 2 log
(D((I)
Σ
Prewrite (d d'; d²;
log (D)
(5)
Widd)
d'EN(d)
[W(dd))
Σ
log Prewitel (d d'; log
(6)
W(d)
(D]
d'ENId)
where the second step uses Jensen's 's inequality. all (dd) are the same size, maximizing this lower bound on log-likeling log-likelihood is equivalent to simply maximizing
log (d d);
('E
over D--this is the ordinary conditional likelihood for string transducer (Ristad & Yianilos, 1998) or sequence-to-seqqueee model (Sutskever al., 2014) with examples d, d' (d).
We have motivated prototype-based models by arguing that Prewrite learns a model of transformations licensed by the training data. However, when generalization involves complex compositions, we will show that neither basic RNN implementation of Prewrite or a single prototype is enough; we must provide the learned rewriting model with larger inventory of parts and encourage reuse of those parts as faithfully as possible. This motivates the two improvements on the prototype-based modeling framework described in the remainder of this section: generalization multiple prototypes (Section 3.1) and a new rewriting model (Section 3.2).
3.1 n P-PROTOTYPE MODELS
To improve compositionaliiy in prototype-based models, we equip them with the ability to condition on multiple examples simultaneously. We extend the basic prototype-basee language model to prototypes, which we now refer to as a recombination model Precomb:
d ~ Precomb (. d'i.n: 0)
where d'
d,,) ~ p.(()
(9)
multi-protype model may be viewed as a meta meta-learner (Thrun & Pratt, 1998; Santoro et al., 2016): it maps from small number of examples (the prototypes) to distribution over new datapoints consis- tent with those examples. By choosing the neighborhood and implementation of Precom appropriately, can train this lear-ee specialize in one-shot concept learning (by reusing fragment exhib- ited in a single prototype) or compositional generalization (by assembling fragments of prototypes into novel configuration). To enable this behavior, we define set of compatible prototypes Q C D
def Unif(S?). We update Eq. 6 to feature a corresponding multi pulti-protottpe (Section 4) and let pn
neighborhood N D + f2. The only terms that have changed are the conditioning variable and the constant term, and it is again sufficient to choose f to optimize Edi,,,NN log Precomb d',..) over
D. implementing Precomb as described next.
3.2 RECOMBINATION NETWORKS
Past work has found that latent- variable neural sequence models often ignore the latent variable and attempt to directly model sequence marginals (Bowman et al., 2016). When an ordinary sequence-to- sequence model with attention used to implement Precomh, even in the one-prototype case, generated
3 Past work also includes a continuous latent variable 2, defining:
Prowiied (d d') Ex<p(=) [Powinee [Pswic) d', 2)
As discussed in Section 5, the use of continuous latent variable appears to make difference in prediction performance for the tasks in this paper. The remainder of our presentation focuses the simpler model in Eq. 7.
4

---

Published as a conference paper at ICLR 2021
first prototy
sccend pretotype-
left
and
walk
LTURN
WALK
jump
JUMP
(b)
generation
Pacts
left
and
h..
>
jump
Papsooote
Pope,
Figure 2: (a) RNN encoders produce contextual embeddings for prototype tokens. (b) In the decoder, gated copy mechanism reuses prototypes and generated output tokens via an attention mechanism (dashed lines).
sentences often have little overlap with their prototypes (Weston et al., 2018). We describe a specific model architecture for Precomb that does not function generic noise model, and in which outputs are primarily generated via explicit reuse of fragments multiple prototypes, by facilitating copying from independent streams containing prototypes and previously generated input tokens.
We take Precomb (d d',n: 0) to be a neural healti--sseuueenet---quueee 2014) which decomposes probability autoregressively: Precemb (d d' 8) = II. p(dl d d' 0). As shown in Fig. 2, three LSTM encoders-twe for the prototypes and one for the input prefix- compute sequences of token representations Aproco and hour respectively. Given the current decoder hidden state hout. the model first attends to both prototype and output tokens:
model (c.f. Sutskever et al.,
dou x exp(hiur Wo Nout) kj proto x exp(him W, hpposo)
ist
(10)
& n.j S dd'l
(11)
To enable copying from each sequence, we project attention weights Clout and o onto the output vocabulary produce sparse vector of probabilities:
Pempyyaul (d = w) = Eier 1 1/d, = w/ a
(12)
Peopy. (d" w) Σj 551Ø5 1[dk w|
kj proto
(13)
Unlike rule-based data recombination procedures, however, Precomb not required to copy from the prototypes, and can predict output tokens directly using values retrieved by the attention mechanism:
h Arre
Es ooul
Ex.j
Oprott [proto 1
(14)
Pwrite eXp(W (WWWhehh" hire)
(15)
To produce a final distribution over output tokens at time we combine predictions from each stream:
Beate = soft maxtmax(WW (W Nout)
(16)
p(d' = w d di.ni0) 0) Bganc [P'nine Pu (w), Peopy, 1 (w).....Peppyyy... Peopy (w))
(17)
This copy mechanism is similar the one proposed by Merity et al. (2017) and See et al. (2017). We compare 1- and 2-prototype models to an ordinary sequence model and baselines in Section
4 SAMPLING SCHEMES
The models above provide generic procedures for generating well-formed combinations of training data, but do nothing to ensure that the generated samples are of a kind useful for compositional generalization. While the training objective in Eq. encourages the learned p(d) lie close to the training data, an effective data augmentation procedure should intuitively provide novel examples of rare phenomena. To generate augmented training data, we combine the generative models of Section 3 with simple sampling procedure that upweights useful examples.
4.1 RESAMPLING AUGMENTED DATA
In classification problems with imbalanced classes, a common strategy for improving accuracy on the rare class is to resample that the rare class is better represented in training data (Japkowicz
5

---

Published as a conference paper at ICLR 2021
al., 2000). When constructing augmented dataset using the models described above, apply simple rejection sampling scheme. In Eq. 1, we set:
u 11 [min < <c]
(18)
Here p(d") is the marginal probability that the token d' appears in any example and c is a hyperpa- rameter. The final model is then trained using Eq. 1, retaining those augmented samples for which u(d)- = 1. For extremely imbalanced problems, like the ones considered in Section 5, this weighting scheme effectively functions rare tag constraint: only examples containing rare words or tags are used to augment the original training data.
4.2 NEIGHBORHOODS AND PROTOTYPE PRIORS
How can we ensure that the data augmentation procedure generates any samples with positive weight in Eq. 18? The prototype-based models described in Section 3 offer an additional means of control over the generated data. Aside from the implementation of Precomb. the main factors governing the behavior of the model are the choice of neighborhood function ((d) and, for n 2 2, the set of prior compatible prototypes S2. Defining these so that rare tags also preferentially appear in prototypes helps ensure that the generated samples contribute to generalization. Let d1 and d2 be prototypes. As a notational convenience, given two sequences d1, d2, let di \d2 the set of tokens in dj but not d2, and di Ad2 denote the of tokens not common to di and d2-
1 1-prototype neighborhoods Guu et al. (2018) define a one-prototype / based on Jaccard distance threshold (Jaccard, 1901). For experiments with one-prototype models we employ a similar strategy. choosing an initial neighborhood of candidates such that
((d) def (di D (ax |dSdi] + B lev lev d:)) < <6}
(19)
where lev is string edit distance (Levenshtein, 1966) and a, and are hyperparameters (discussed in Appendix B).
complex neighborhood function-intuitive for an input d, we want each (d1,d2-- d2....) in the neighborhood to collectively contain enough information reconstruct d. Future work might treat the neighborhood function itself as latent, allowing the model to identify groups of prototypes that make d probable; here, as in existing one-prototype models, we provide heuristic implementations for the
2-prototype neighborhoods The n 2 prototype case requires
2 case.
Long-short recombination: For each (d1, d2) E (d). d2 is chosen to be similar to d, and d2 is chosen to be similar to the difference between d and d1. (The neighborhood is named because one of the prototypes will generally have fewer tokens than the other one.)
= =} (20) Here [d])d,] the sequence obtained by removing all tokens in d, from d. Recall that we have defined pa (d(1) der Unif(!)) for set Ω of "compatible" prototypes. For experiments using long-short combination, all prototypes are treated as compatible; that is, S = D x
(d)) def {d1,, d2) E Ω lev (d, d1) < 6, lev ((d) [(d)d1), d2) < 6, |d/4 > 0,
Long-long recombination: (dd) contains pairs of prototypes that are individually similar to d and collectively contain all the tokens needed to reconstruct d:
((d) def ((d1, d2) Ω lev d1) < 5, lev (d, d2) < 6, |dAd| 1, (d Adal \dal= =0 (21)
For experiments using long- recombination, we take st {(1,, d2) E D D Ad = 1}.
DATASETS & EXPERIMENTS
We evaluate R&R on two tests of compositional generalization: the SCAN instruction following task (Lake & Baroni, 2018) and a few-shot morphology learning task derived from the SIGMORPHON 2018 dataset (Kirov et al., 2018: Cotterell et al., 2018). Our experiments the effectiveness of learned data recombination procedures in controlled and natural settings. Both tasks involve conditional sequence prediction: while preceding sections have discussed augmentation
designed to explore

---

Published as a conference paper at ICLR 2021
Table I: Results on the SCAN dataset. (a) Comparison R&R with previous work. Connecting lines indicate that model components are inherited from the parent (c.g. the row labeled recomb-2 also includes resampling). Data augmentation with recomb-2 resampling performs slightly worse than GECA on the jump and around right splits; data augmentation with recomb-I + resampling or an ordinary RNN does not generalize robustly to either split. All differences except between GECA and recomb-2 + resampling in jump are significant (paired L-test, p & 0.001). (Dashes indicate that all samples were rejected by resampling when decoding with lemperature = 1.) (b) Ablation experiments on the jump split. Introducing the latent variable used in previous work (Guu et al., 2018) does not change performance; removing the copy mechanism results in a complete failure of generalization. While is possible to perform conditional inference of p(y x) given the generative model in Eq. 3 (direct inference), this gives significantly worse results than data augmentation (see Sec. 5.3).
(a)
(b)
around right
jemp
jump
baseline
0.00 +0.00
0,00 +0.00
recomb-2 (no resampling)
0.88 +0.07
GECA (published)
0.82 #0.11
0.87 +0.05
VVE
0.88 +0.07
GECA (ours)
0.98 +0.02
1.00 +0.001
+ resampling
0.87 1.008
learned aug. (basic)
0.00 1.0.00
0.00 1.0.00
copying
0.00 1:0.00
resampling
direct inference
0.57 1:005
recomb-1
0.17 10.00
recomb-2
0.75 #0.14
0.87 +0.08
recomb-2 (no resampling)
0.82 +0.08
0.88 +0.07
procedures that produce data points d = (ar, y). learners are evaluated their ability predict an output y from an input æ: actions given instructions x, or morphological analyses given words x.
For each task, we compare a baseline with no data augmentation, the rule-based GECA data aug- mentation procedure (Andreas, 2020), and sequence of ablated versions of R R&R that measure the importance of resampling and recombination. The basic Learned Aug model trains an RNN to generate (z, y) pairs, then trains conditional model on the original data and samples from the generative model. Resampling filters these samples as described in Section Recomb-n models replace the RNN with prototypebbased model as described in Section 3. Additional experiments (Table 1b) compare data augmentation to prediction of y via direct inference (Appendix E) in the pradotype-based model and several other model variants.
5.1 SCAN
SCAN (Lake & Baroni, 2018) is a synthetic dataset featuring simple English commands paired with sequences of actions. Our experiments aim show that R&R performs well at one-shot concept learning and zero-shot generalization on controlled tasks where rule-based models succeed. We experiment with two splits of the dataset, jump and around right. In the jump split, which tests one-shot learning, the jump commands in the test set (e.g. look and jump twice). The around right split (Loula et al., 2018) tests zero-shot generalization by presenting learners with constructions like walk around left and walk right in the training set, but walk around right only in the test set.
jump appears in a single command in the training set but in more complex
Despite the apparent simplicity of the task, ordinary neural gruuence-to-seqeene models completely fail to make correct predictions on SCAN test set (Table 1). As such has been major focus of research compositional generalization in sequence-to-sequene procedures and specialized model architectures and training procedures have been developed to solve it (Russin al., 2019; Gordon et al., 2020; Lake, 2019; Andreas, 2020). Here we show that the generic prototype recombination procedure described above does so as well. We use recombination for the jump split and long-long recombination for the around right split. We use recombination network to generate 400 samples d = (xc, y) and then train an ordinary LSTM with attention (Bahdanau al., 2019b) the original and augmented data predict y from x. Training hyperparameters are provided in Appendix D.
models, and a number of heuristic
Table shows the results of training these models on the SCAN dataset. 4 2-prototype recombination is essential for successful generalization on both splits. Additional ablations (Table 1b) show that the
"We provide results from GECA for comparison. Our final RNN predictor is more accurate than the one used by Andreas (2020), and training it on the same augmented dataset gives higher accuracies than reported in the original paper.
7

---

Published as a conference paper at ICLR 2021
Table 2:
Fi score for morphological analysis on (FUT+PST) and frequent (OTHER) word forms. R&R variants with and 2-prototype recombination (shaded in grey) consistently match or outperform both a no-augmentation baseline and GECA; recomb-1 + resampling is best overall. Bold numbers are not significantly different from the best result in each column under paired t-test (p < 0.05 after Bonferroni correction; nothing is bold if all differences whose exact tag set never appeared in the training data. (There were no such words in the test set for the Spanish OTHER.) Differences between GECA and the best R&R R variant (recomb-1 + resampling) are larger than in the full evaluation set. "The Spanish past tense was used as a development set.
insignificant). The NOVEL portion of the table shows model accuracy examples
Spanish
Swahili
Turkish
FUT+PST'
OTHER
FUT+PST
OTHER
FUT+PST
OTHER
baseline
0.66 #0.01
0.88 #0.01
0.75 +0.00000
0.90+10.01 +0.01
0.69 #0.04
0.85 +0.03
L resampling
0.65 #0.01
0.88 #0.01
0.77 +0.01
0.90 +0.02
0.69 #0.04
0.84 +0.04
GECA
0.66 #0.01
0.88 +0.01
0.76 +0.02
0.90+10.00 +0.02
0.69 #0.02
0.87 #0.01
L resampling
0.72 +0.02
0.88 +0.01
0.81 +0.02
0.89 1:001
0.75 +0.03
0.85 40.02
leameed aug. (basic)
0.66 +0.02
0.88 4.001
0.77 +:002
0.90 1:0.01
0.70 10.02
0.87 10.01
resumpling
0.70 10.02
0.86 10.01
0.84 1:001
0.90 1.0.11
0.73 10.04
0.85 10.03
recomb-J
0.72 10.02
0.87 1.001
0.85 4.001
0.90 1:0022
0.77 10.02
0.87 10.02
recomb-2
0.71 10.01
0.87 40.02
0.82 40.02
0.90 1:001
0.75 10.03
0.86 10.03
GRCA recomb-1 resamp.
0.74 +0.02
0.86 40.01
0.85 40.02
0.89 10001
0.79 +0.02
0.84 40.01
NOVEL
baseline
0.63 10.03
0.72 10002
0.42 1:012
0.68 1:004
0.66 10.15
GECA + resampling
0.67 10.03
0.79 10.02
0.26 1:0.20
0.73 1:004
0.71 1:.10
recomb-/ + resampling
0.69 10.02
0.83 10.02
0.42 1:012
0.75 10.03
0.82 1:004
GECA + recomb-1 1 + resamp.
0.69 +0002
0.83 +0.02
0.35 +0.11
0.77 +0.03
0.71 40007
continuous latent variable used by Guu al. (2018) does not affect performance, but that the copy mechanism described in Section 3.2 and the use of the recomb-2 model for data augmentation rather than direct inference are necessary for accurate prediction.
5.2 SIGMORPHON 2018
The SIGMORPHON 2018 dataset consists of words paired with morphological analyses (lemmas, or base forms, and tags for linguistic features like tense and case, as depicted in Fig. 1). We use the data to construct a morphological analysis task (Akyürek et al., 2019) (predicting analyses from surface forms) to test models' few-shot learning of new morphological paradigms. In three languages of varying morphological complexity (Spanish, Swahili, and Turkish) we construct splits of the data featuring a training set of 1000 examples and three test sets of 100 examples. One test set consists exclusively of words in the past tense, one in the future tense and one with other word forms (present tense verbs, nouns and adjectives). The training set contains exactly eight past-tense and eight future-tense examples; all the rest are other word forms. Experiments evaluate R&R's 's ability to efficiently learn noisy morphological rules, long viewed key challenge for connectionist approaches to language learning (Rumelhart& & McClelland, 1986). As approaches may be sensitive to the choice of the eight examples from which the model must generalize, we construct five different splits per language and use the Spanish past-tense data development set. As above, we use long-long recombination with similarity criteria applied to y only. We augment the training data with 180 samples from Precomb and again train an ordinary LSTM with attention for final predictions. Details are provided in Appendix B.
Table 2 shows aggregate results across languages. We report the model's F1 score for predicting morphological analyses of words in the few-shoi training condition (past and future) and the standard training condition (other word forms). Here, learned data augmentation with both one- and two- prototype models consistently matches or outperforms GECA. The improvement is sometimes dramatic: for few-shot prediction in Swahili, recomb-/ augmentation reduces the error rate by 40% relative to the baseline and 21% relative to GECA. An additional baseline + resampling experiment upweights the existing rare samples rather than synthesizing new ones; results demonstrate that recombination, and not simply reweighting, is important for generalization. Table 2 also includes a finer- aeer-graieed analysis of novel word forms: words in the evaluation set whose exact morphological analysis never appeared in the training set. R&R again significantly outperforms both the baseline and DECA-based data augmentation in the few-shot FUT+PAST condition and the ordinary OTHER condition, underscoring the effectiveness of this approach for "in-distribution" compositional
8

---

Published as a conference paper at ICLR 2021
generalization. Finally, the gains provided by learned augmentation and GECA appear to be least partially orthogonal: combining the GECA + resampling and recomb-1 + resampling models gives further improvements in Spanish and Turkish.
5.3 ANALYSIS
Why is R&R effective? Samples from the best learned data augmentation models for SCAN and SIGMORPHON may be found in the Appendix G.3. We programaticaly analyzed 400 samples from recomb-2 models SCAN and found that 40% of novel samples are exactly correct in the around right split and 74% in the jump split. A manual analysis of 50 Turkish samples indicated that only 14% of the novel samples were exactly correct. The augmentation procedure has high error rate! However, our analysis found that malformed samples either (1) feature malformed rs that will never appear in test set (a phenomenon also observed by Andreas (2020) for outputs of GECA), or (2) are mostly correct at the token level (inducing predictions with high Fi score). Data augmentation thus contributes mixture of irrelevant examples, label noise-which may exert a positive regularizing effect (Bishop, 1995) and well-formed examples, small number of which are sufficient to induce generalization (Bastings et al., 2018). Without resampling, SIGMORPHON models generate almost no examples of rare tags.
Why does R&R outperform direct inference? A partial explanation is provided by the preceding analysis, which notes that the accuracy of the data augmentation procedure as generative model is comparatively low. Additionally, the data augmentation procedure selects only the samples from the model, the quality of predicted ys conditioned random will in general be even lower. conditional model trained on augmented data is able to compensate for errors in augmentation or direct inference (Table 12 in the Appendix).
confidence
Why is Resampling without Recombination effective? One surprising feature of Table 2 is perfor- mance of the learned aug (basic) + resampling model. While less effective than the recombination- based models, augmentation with samples from an ordinary RNN trained on (æ. y) pairs improves performance for some test splits. One possible explanation that resampling effectively acts as a posterior constrain! on the final model's predictive distribution, guiding it toward solutions in which rare tags are more probable than observed the original training data. Future work might model this constraint explicitly, e.g. via posterior regularization (as in Li & Rush, 2020).
CONCLUSIONS
We have described a method for improving compositional generalization in sequence-to-seqqueee models via data augmentation with learned prototype recombination models. These are the first results we are aware of demonstrating that generative models of data are effective as data augmen- tation schemes in sequence- lequence-to-sequeee sequeeue learning problems, even when the generative models are themselves unreliable as base predictors. Our experiments demonstrate that it is possible to achieve compositional generalization on-par with complex symbolic models in clean, highly structured do- mains, and outperform them in natural ones, with basic neural modeling tools and without symbolic representations.
ACKNOWLEDGMENTS
We thank Eric Chu for feedback on early drafts of this paper. This work was supported by a hardware donation from NVIDIA under the NVAIL grant program. The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center (Reuther et al., 2018) for providing HPC resources that have contributed the research results reported within this paper.
REFERENCES
Ekin Akyürek, Erenay Dayami,, and Deniz Yuret. Morphological analysis using a sequence decoder. Transactions of the Association for Computational Linguistics, 7:567-579, 2019.
Jacob Andreas. Good-enough compositional data augmentation. Procedings of the Anmual Meeting of the Association for Computational Linguisties, 2020.

---

Published as a conference paper at ICLR 2021
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 39-48, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015.
Dzmitry Bahdanau, Harm de Vries, Timothy J O' O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville. Closure: Assessing systematic generalization of clevr models. arXiv preprint arXiv:1912.05783 2019a.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What required and can it be learned? In International Conference on Learning Representations, 2019b.
Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. Jump to better conclusions: SCAN both left and right. In Proceedings of the EMNLP BlackboxNLP Workshop, pp. 47-55, Brussels, Belgium, 2018.
Jean Berko. The child's learning of english morphology. Word, 14(-33::50-177,, 1958.
Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM review, 59(1):65-98, 2017, URL https://doi.orr//100 1137/ 141090671.
Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7 (1):108-116, 1995.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz. and Samy Bengio. Generating sentences from a continuous space. In Proceedings of 20th SIGNLL Conference Computational Natural Language Learning, pp. 10-21, 2016.
Jonathon Cai, Richard Shin, and Dawn Song. Making neural programming architectures generalize via recursion. In International Conference on Learning Representations, ICLR 2017, 2017.
Susan Carey and Elsa Bartlett. Acquiring single new word. Papers and Reports Child Language Development 2, 1978.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer, Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Alexander Clark and Rémi Eyraud. Polynomial identification in the limit of substitutable context-free languages. Journal of Machine Learning Research. 12551):125511455, 2007.
Ryan Cotterell, Christo Kirov, John Saaak-Glassman, Géraldine Walther, Ekaterina Vylomova, Arya D. McCarthy, Katharina Kann, Sabrina J. Mielke, Garrett Nicolai, Miika Silfverberg, David Yarowsky, Jason Fisner, and Mans Hulden. The CONLL--SIGMORPOON 2018 shared task: Universal morphological reinflection. In Proceedings of the CONLI-SIGMORPHON 2018 Shared 2018.
Task: Universal Morphological Reinflection, pp. 2018.
Marzich Fadace, Arianna Bisazza, and Christof Monz. Data augmentation for low-resource neural ma- chine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 567-573, 2017.
Jerry Fodor, Zenon W Pylyshyn, et al. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3-71, 1988.
Felix A Gers and E Schmidhuber. Lstm recurrent networks learn simple context-free and context- sensitive languages. IEEE Transactions on Neural Networks, 1236):3333 2001.
Jonathan Gordon, David Lopez-Paz, Marco Baroni, and Diane Bouchacourt. Permutation equivariant models for compositional generalization in language. In International Conference on Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2021
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. Incorporating copying mechanism in gequence-to-sequenne learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 1631-1640, 2016.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. Search engine guided neural machine translation. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Perey Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450, 2018.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020.
Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy $ Liang. retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems, pp. 10052-10062, 2018.
Hiroshi Inoue. Data augmentation by pairing samples for images classification. In International Conference Learning Representations, 2018.
Paul Jaceard. Comparative study of the distribution of flora in a portion of the Alps and Jura. Bull Soc Vaudoise Sci Nat, 37:547-579, 1901.
Nathalie Japkowicz et al. Learning from imbalanced data sets: comparison of various strategies. In AAAI workshop on learning from imbalanced data sets, volume 68, pp. 10-15, 2000.
Robin Jia and Percy Liang. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 12-22, 2016.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz. Stafiniak, Tihor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method realistic data. In International Conference on Learning Representations, 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference Learning Representations, 2020.
Christo Kirov, Ryan Cotterell, John GlassmGa,,,mn,, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sabrina J. Mielke, Arya McCarthy, Sandra Kübler, David Yarowsky. Jason Fisner, and Mans Hulden. UniMorph 2.0: Universal Morphology. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), 2018.
B. Lake, Tal Linzen, and M. Baroni. Human few-shot learning of compositional instructions. In CogSci, 2019.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of eoquence-to-sqquenee recurrent networks. In Jennifer Dy and Andreas Krause (cds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2873-2882, 2018
Brenden M Lake. Compositional generalization through meta sequence-to-sequenee learning. In
Advances in Neural Information Processing Systems, pp. 9788-9798, 2019.
Vladimir Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. In
Soviet physics doklady, volume 10, pp. 707-710, 1966.

---

Published as a conference paper at ICLR 2021
Xiang Lisa Li and Alexander M Rush. Posterior control of blackbox generation. arXiv preprint arXiv:2005.04560, 2020.
João Loula, Marco Baroni, and Brenden Lake. Rearranging the familiar: Testing compositional
generalization in recurrent networks. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 108-1 14, 2018.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, ICLR 2017, 2017.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1400-1409, 2016
Steven Piantadosi and Richard Aslin. Compositional reasoning in early childhood. PloS one, 11(9), 2016.
Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learn- ing to compose domain- transformations for data augmentation. In Advances in neural information processing systems, pp. 3236-3336330 3246,322622222 2017.
Albert Reuther, Jeremy Kepner, Chansup Byun, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, et al. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1-6. IEEE, 2018.
Eric Sven Ristad and Peter N Yianilos. Learning string-edit distance. IEEE Transactions Pattern Analysis and Machine Intelligence, 20(5): 1998.
Murray Rosenblatt. Remarks on some nonparametric estimates of density function. The Annals of Mathematical Statistics, pp. 832-837, 1956.
David F. Rumelhart and James I.. McClelland. On learning the past tenses of english verbs. Mecha- nisms of language aquisition, pp. 195-248, 1986.
Jake Russin, Jason Jo, Randall C CO'Reilly, and Yoshua Bengio. Compositional generalization in deep seq2seq model by separating syntax and semantics. arXiv preprint 19041190409700 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmentted neural networks. In International conference on machine learning, pp. 1842-1850, 2016.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp- 1073 1003,108,.200 2017.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances neural information processing systems, pp. 1114-112, 2014.
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn, pp. 3-17. Springer, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, 2017.
Jane X. Wang, Zeb Kurth-Nelson, Hubert Soyer, Joel Z. Leibo, Dhruva Tirumala, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matt M. Botvinick. Learning to reinforcement learn. In CogSci, 2016.
Gail Weiss, Yoav Goldberg, and Eran Yahav. On the practical computational power of finite precision RNNs for language recognition. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 740-745, 2018
12

---

Published as a conference paper at ICLR 2021
Jason Weston, Emily Dinan, and Alexander Miller. Retrieve and refine: Improved sequence generation models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAl: The 2nd International Workshop Search-Oriented Conversational AI, pp. 87-92, 2018.
Chunyang Xiao, Marc Dymetman, and Claire Gardent. Sequence- structured prediction for semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics,, pp. 1341-1350, 2016.
Deniz Yuret. Knet: beginning deep learning with 100 lines of julia. In Machine Learning Systems Workshop at NIPS, volume 2016, pp. 5, 2016.
13

---

Published as a conference paper at ICLR 2021
A MODEL ARCHITECTURE
A.1 PROTOTYPE ENCODER
We use a single layer BiLSTM network to encode kj as follows:
H proto
proj(BiLSTM d(.))
(22)
The hidden and embedding sizes 1024. No dropout is applied. We project bi- directional embeddings to the hidden size with a linear projection. We concatenate the backward and the forward hidden states.
Morphology
SCAN We choose the hidden size as 512, and embedding size as 64. We apply 0.5 dropout to the input. We project hidden vectors in the attention mechanism.
A.2
DECODER
The decoder implemented by a single layer. In addition to the hidden state and memory cell, we also carry out feed vector through time:
Afre
[
Σ
ooe oome Boan
Σ
OpiccoPPioee
1j Aprcer
Proooo proto
Proto
(23)
JS|di
SS1d.l
feed' = Linear (fire)
(24)
The input to the LSTM decoder at time step is the concatenation of the previous token's representa- tion, previous feed vector, and a latent z vector (in the VAE model).
input [Wad feed"
(25)
Morphology We use a single-layer LSTM network with a hidden size of 1024, and an embedding size of 1024. We initialize the decoder hidden states with the final hidden states of the BiLSTM encoder. feed is the same size as the hidden state. No dropout applied the decoder. Output calculations are provided in the original paper in equation Eq. 16. The query vector for the attentions is identically the hidden state:
query²
(26)
Further details of the attention are provided Appendix A.3.
SCAN The decoder is implemented by single layer LSTM network with hidden size of 512, and embedding size of 64. The embedding parameters are shared with the encoder. Here the size of the feed vector is equal embedding size, 64.
We have no self-attention for this decoder in the feed vector. There is an attention projection with dimension is 128. The details of the attention mechanism given in Appendix A.3. Finally, we use transpose of the embedding matrix to feed to the output space.
output' W.T feed'
(27)
output' contains unnormalized scores before the final softmax layer. We apply 0.7 dropout h during both training and test. The copy mechanism will be further described in Appendix A.3.
The input to the LSTM decoder is the same as Eq. 25 except the decoder embedding matrix, Wa- shares parameters with encoder embedding matrix W.. We applied 0.5 dropout to the embeddings d-
The query vector for the attention is calculated by: query' [M input")
(28)
14

---

Published as a conference paper at ICLR 2021
A.3 ATTENTION AND COPYING
We use the attention mechanism described in Vaswani et al. (2017) with slight modifications.
Morphology We linear transformation for key while retaining embedding size of 1024, and leave query and value transformations as the identity. We do not normalize by the square root of the attention dimension. The query vector is described the decoder Appendix A.2. The copy mechanism for the morphology task is explained in the paper in detail.
SCAN We use the nonlinear tanh transformation for key, query and value. That the attention scores are calculated separately for each prototype using different parameters well as the normalization ie. obtaining performed separately for each prototype.
The copy mechanism for this task is slightly different and follows Gu et al. (2016) We normalize prototype attention scores and output scores jointly. Let a; represent attention weights for each prototype sequence before normalization. Then, we concatenate them to the output vector in Eq. 27.
final' =
[outppt', ā1...., ānl
We obtain a probability vector via final softmax layer:
prob' = softmax(final))
That size of this probability vector vocabulary size plus the total length all prototypes. We then project this into the output space by:
('(w) prob' (indices(w))
where indices finds all corresponding scores in prob' for token where there might be more than one element for a given w. This is because one score can come from the output region, and others from the prototype regions of prob'. During training applied 0.5 dropout the indices from oulput". Thus, the model is encouraged to copy more.
B NEIGHBORHOODS AND SAMPLING
In the Eq. 20 and Eq. 21 we expressed the generic form of neighborhood sets. Here we provide the implementation details.
SCAN In the jump split, we use long-short recombination with 5 = 0.5. In around right we use long-long recombination with 0.5, and construct Q so that the first and second prototypes to differ by single token. We randomly pick & < 10 3 (10 different first prototypes, and 3 different second prototypes for each them) prototype pairs that satisfy these conditions. For the recomb-1 experiment, we use the same neighborhood setup except but consider only the k < 10 first prototypes.
Sampling the jump split, we used beam search with beam size 4 in the decoder. We calculate the mean and standard deviation over the lengths of both among the first d' and the second d' prototypes the train set. Then, during the sampling, we expect the first and second prototypes whose length shorter than their respective mean plus standard deviation. This decision is based the fact that the part of the Ω that the model is exposed is determined by the empirical distribution, Ω that arises from training neighborhoods. When sampling, we try to pick prototypes from a distribution that are close to properties of that empirical distribution. around right, use temperature sampling with T = 0.4. If model cannot sample the expected number of both novel and unique samples within a reasonable time, increase temperature T'
Morphology We use long-long recombination, as explained in the paper, with slight modifications which leverage the structure of the task. We Q as:
Ω {(d,,d2) d'ings # dzings: (drags Aings / 0}
For the recomb-1 model (d) utilizes tag similarity, lemma similarity and is constructed using a score function:
score, (d, d)) (Icluge
Ad Ad iogg (accard(deemmn d' d'veomma)
(29)
15

---

Published as a conference paper at ICLR 2021
Given d, we sort training examples by using score; as the comparison key and pick the four smallest neighbors (using lexicographic sort) to form M(d).
For the recomb-2 model, (d) uses the same score function for the first prototype in the recomb-1 case. The second prototype selected using:
score2 (1) =(chag + d2uags: |digs Adiaags)
(30)
Given æ, and scored first prototype, do one more sort over training examples by using score, as the comparison key. Then we pick first four neighbors for (d)).
Sampling We use mix strategy of temperature sampling with 0.5 and greedy sampling in which we use the former for dinpus and the latter for doutput We sample 180 unique and novel examples.
C GENERATIVE MODEL TRAINING
Morphology All of the hyper parameters mentioned here are optimized by grid search on the Spanish validation set. We train our models for 25 epochs². We use Adam optimizer with learning rate 0.0001. The generative model is trained on morphological reinflection order (chemmadugs diatlection) from left to right, then the samples from the model are reordered for morphological analysis task (dinfiection D deggs)
SCAN We use different number of epochs for jump and around right splits where all models are trained for epochs in the former and 3 epochs in the latter. We use Adam optimizer with learning rate 0.002, and gradient norm clip with 1.0.
D SEQ2SEQ BASELINE MODEL
After generating novel samples, we either concatenate them to the training data (in morphology), sample training batches from a mixture the original training data and the augmented data. Our conditional model is the same the generative model used in morphology experiments, described in detail in the paper body, replacing dpreco with x, and d with y.
Every conditional model's size is the same the corresponding generative model which was used for augmentation. This to ensure that the conditional model and the generative model have the same capacity. We train conditional models for 150 epochs for SCAN and we used augmentation ratios of Paug = 0.01 and Paug 0.2 in jump and around right, respectively. For morphology, we train the conditional models for 100 epochs, and we use all generated examples for augmentation.
E DIRECT INFERENCE
To adapt the prototype-basee model for conditional prediction, condition the neighborhood function on the input & rather than the full datum d, as in Hashimoto et al. (2018). Candidate ys are then sampled from the generative model given the observed æ while marginalizing over retrieved prototypes. Finally, we re-rank these candidates via Eq. 7 and output the highest- scoring candidate.
F VAE MODEL
Prior p(z): We use the same prior as Guu et al. (2018) given in Eq. 31. this prior, = is defined by norm and direction vector. The norm sampled from the uniform distribution between zero and a maximum possible norm Mmax = 10.0, and the direction is sampled uniformly from the unit hypersphere. This sampling procedure corresponds to von Mises Fisher distribution with concentration parameter zero.
Hises-Fisher
'When training 2-proto and 1-proto models, we increment epoch counter when the entire neighborhood for every d is processed. For O-proto, one epoch defined canonically i.c. the entire train
16

---

Published as a conference paper at ICLR 2021
Znorm
Adir
where
Snorm
U(,, Mmax), Zdir vml 0)
(31)
For SCAN, the size of is 32, and for morphology the size of is 2.
Proposal Network q(2 d, d1:n): Similarly to the prior, the posterior network decomposes & into its norm and direction vectors. The norm vector is sampled from a uniform distribution at (1pl), min c, 1./aax)) ), and the direction is sampled from the von Mises-Fisher distribution vmF( k) where 25,6 1.0.
hiinall (hrroto)sss )start + (Peproto)end J tanh (W₂ [hedtinal Ngginal)
nnrmm
~ U(1),, min( les + 6, Mmmax) Zdir ~ umn NumFFu, x))
(32)
(33)
Znorm
Zdit
(34)
G ADDITIONAL RESULTS
G.1 MORPHIOLOGY RESULTS
In the paper, Table shows morphology results for (non- VAE) models with 8 hints (past- and future- tense examples in the training set). Here, provide additional results for different hint set sizes and model variants.
G.1.1 HINTS=4
Table 3: Exact Match Accuracy
Spanish
Swahili
Turkish
PUT4PST"
OTHER
PUTAPST
OTHER
PUT4PST
UTHER
baseline
0.078 sou1z
0.532 sauus
2004
geca
0.092 2000
0163 10:c5
0.039 10011
0.496 1007
0,022 16.01
0.54 20.05
geca resampling
0.16 10.1
0.65 10.65
0.27 _.os
0.52 10.01
0.12 10.04
0.554 .0.020
leamed ang
0.063
0.066
0.074
leanndd aug + resampling
0.098 =ou121
0.65 20.26
0.29 =0.05
0.180 =0.015
0.092 26.02)
0.54 =0.0
recomb-1
0.063 c0urc
0.674 10.02
0.061 20:07
0.520 2auzs
0.055 14.001
0.554 20.000
recamh-1 resampling
0.52
recomb-2
0.061 =00000
0.656 +0.000
=0.05
0.524
=0.025
0.073 #0.019
0.58 20.05
recomb-2 resampling
0.108 20.02)
0.64 20.06
0.18 20:4
0.542 20.035
0.067 26, 20.008
0.55 20.05
Table 4: Fl Accuracy
Spanish
Swahili
Turkish
FUT4PS1"
OTHER
PUTAPST
OTHER
RUTARST
bascline
sou02
geca
0.606 Lours
0.871
10.017
0.722 10.18
0.884 10.07
0.565 10.001
0.856 _00.08
resampling
0.675 20.07
(1870
0.802
01899 10000
0.65
0.850 20.010
leaned aug
0.597
leamned aug + resampling
0.646 2ou17
0.872 10.005
0.826 20.111
0.887 1200b
0.637 10.001
0.835 20.034
recomis.1
0.596 20.00
(1884 10.010
0.727
(1893 10.00
01557 1s
0.868 20.010
recomb-1 resampling
0.663
0.874
0.812
0.886
0.67
0.84
recomb-2
0.598 20.09
0.874 10.007
0.730
20.038
0.804 +0,005
0.581 46.005
0.865 20 20.
recomb-2 2 + resampling
0.658 =0.012
0.872 50.013
0.778
20.011
0.897
=0,07
0.609 20.002
0.850 20.029
1.1 HINTS=8
Main 8-prototype F, results are provided in the body of the paper. Here we provide exact match results and extra set of comparisons the VAE model.
17

---

Published as a conference paper at ICLR 2021
Table 5: Exact Match Accuracy
Spanish FUT-PST"
Swahili
Turkish
OTHER
FUT+PST
OTHER
FUT+7ST
OTHER
baseline
0.151 #0.317
0.65 +0.04
0.15 =0.04
0.554 #0.54
0.23 20.00
0.55 #0,04
geca
0.136
0.638
0.15
0.55 20os
0.21
0.550 #0.012
geca resampling
10.201
10.01
10.25
0.532 10.33
0.27 10.07
0.524 10.076
leamned sng
0.163 20.200
0.652 20.03
0.18 =0.06
0,560 =0.00
0.23 20.01
0.548 10.010
lnamed mug resampling
0.181
0.590
0.34
0.552
0.24
0.53
recomb-1
0.155 20.015
0 0.628 #0.020
0.161 +0.017
0.560 =0.035
0.22 +0.04
0.538 +o 026
recomb-1 1 resampling
0.218 20.082
0,616 =ou034
0.35 =0.24
0.53 =0.04
0.30 20.04
0.52 +0,04
recomb-2
0.131
0.634
0.19
0.56
0.24
0.528
recomb-2 resampling
0.203 20 283
0.63 $0.05
20.07
0.552 20.081
0.25 10 0G
0.54 2r 04
Table 6: Fi Accuracy (VAE model)
Spanish OTHER
Swahili
Turkish
FUT-PST
FUT+PST
OTHER
FUT+PST
OTHER
lenmed aug resampling avae
0.689 20.01
0,859 1000e
0.845 20.
0,806 100t3
0,730 sau3ee
0.850
015
recomb-1 I resampling vaee
0.717 ao.
0.80
10007
0.843 10.014
0.898 innto
0.736
a0.000
0.859 10
recomb-2 2 resampling
0.710
0,865 som1
0.824
0,896
0.751
0,848
G.1.3 1INT==16
Table 7: Exact Match Accuracy
Spunish
Swahili
Turkish
FUT+2ST"
OTHER
FUT+PST
OTHER
FUT+PST
OTHER
baseline
0.27
0.65
0.28
0.544 000
0.40
0.614
peca
0.26 10.00
0.65 20:06
0.26 2005
0.530 20
0.37 10:05
0.570 20.005
gcca resampling
0.34 #0.00
0.63
0.32 =0.7
0,506 =0.054
0.42 20.06
0,590 20.085
learned aug
0.25 os
0.65 20.04
2000
0.538 028
0.39 20.04
0.58 =0:05
learned aug resampling
0,230 +0.055
0.61 20.04
0.42 =0.0
0.54 #0.04
0.42 =0.00
0.578 =0.027
recomb-1
0.27
0.63
0.32 =0.05
0.55
0.60
recomb-1 resampling
0.28 10.01
0.61 10:07
0,488 _.c00s
0.548 10
0.35 10:06
0.56 20.44
rocomb-2
0.22 #0.06
0.62
0.28 =0.04
0,56 #0.01
0.40 20.06
0.596 20.024
recomb-2 resampling
o25
0.61 10.07
0.405 _0.00s
0.53 ac.01
20:06
0.61 2004
Table 8: Fl Accuracy
Spanish
Swahili
Turkish
FUT+PST
OTHER
FUToPST
OTHER
FUT+PST
OTHER
baseline
0,733 20004
0.881
#0.012
0.811 20018
0,893
:0001
0.750
26.008
0.875 20.001
goca
0.736 =0.00
0.884
20.015
0.500
=0.024
0.889
=0.012
20.04
0.863
geca + resampling
0.782 1o.02
0.867 10.013
0.830
10,21
0.885
13.03
0.794 16.001
0.865 _10.088
learned aug
0.738 =0.120
0.877 10.00s
0.316 20.024
0.893
=0.12
0.752 24.024
0.568 20.020
leamed ang resampling
0.745
0.870
0.866
0.894
0.787
0.863
recomb-]
0.738 =0.02)
0.877 10.019
0.820 20.018
0.896
#0.014
0.735 20.003
0.874 20.000
recomb-1 + resampling
0.770 =ou120
0.867 +0.023
0.872 20.000
0.892
=0U0D
0.778 24.024
0.561 20.022
recomb-2
0,716
0.876
0.815
0.897
0.752
0.873
recomb-2 + resampling
0.765 50.02
0.868
10.001
0.856
20.016
0,888
50.016
0.808
20.018
0,868 20.009
G.2 SIGNIFICANCE TESTS
Tables 9, 10 and 11 sho the p-values for pairwise differences between the baseline and prototype-based models
Table 9: Turkish language p-values paired I-test in PST+FUT tenses for the average Fi (micro) scores over several runs without Bonferronni correction.
hoodlis
gvd
Iramed way
mesebe1
get manggling leared wag marnging necethi moomping lwwbb--nomppigg
0.159314
0.355500
050006
mecemb-
81333578
11333779:
11353375
axea searpling
15
3 12
407675c 15
6147756-117
1311667c-19
mecep-l marpping
3858776-28
1800177-25
2-7441e--2
6441228e-15
2.07716-226
115030365
1:919a006 0L0150442
2 rcanaplag
2.56680-115 15
3.4062e-13
2.08177e
2521115--18
1.79988c-19
0.9811566
0.000158
18

---

Published as a conference paper at ICLR 2021
Table 10: Spanish language p-values for paired t-test in PST+FUT tenses for the average Fi (micro) scores over several runs without Bonferronni correction.
buaiie
Teasadl mg rccodd:
poua toangiing
leaseel my reanppiig
ixcembil reweepping
iosurb22 2
louline a0ea
0.294744
leansd axx
0.823006
(19718511
0.199706
0,00100
0.331404
0L522018
pasa yeamping
6.31513-229
manepping
10662Me-10 10
1801Tin-11
13
1-76119-14
O 0000015T10
menebsl mompping
9383477 30
8 01405734
1. 1.193770008 0L00925477
scccadn2 reastplng
1.17100-17
1.29436-21
4.07864e-18
1.12332a-19
1.66338-221
0.313343
0.108819
Table 11: Swahili language p-values for paired t-test PST+FUT tenses for the average F1 (micro) scores over several runs without Bonferronni correction.
bouaiine
leamedion momob-1
mocerh-22
mispppig norrh-1 marrpilyy / noreb-2 ee
Feaclan
0i639303
Inaneciaug
5us30551331
M003548801
175447:-05
0830088557
0363242
0.912964
geca aoappiiu
2.50996--11
8454311-9
4.52238e-1
1.13577p-11
leaned our coppattizz
T,92377--33
6.32242--55
1.35556651
9.80155-44
9.46686-37
11041e-11
0.0749463 1.325946-05
rocoeb-2 manpping
2118831-31
2323346-3
7.80476-30
35
1/887626119
1230116-15
018756666
2.517756-11
G.3 GENERATED SAMPLES
All samples are randomly selected unless otherwise indicated.
G.3.1 SCAN
In Table 12, we present three test samples from the SCAN task along with the predictions by direct inference and the conditional model trained on the augmented data with recomb-2. Note that the augmentation procedure was able to create novel samples whose input (œ) happens to in the test set (Examples 1 and 3) while y may or may not be correct (Example 1).
Exanple (Jusp)
Exarple 2 (ump))
Exaropi (anaard rudht)
olier jump
after
jump
label
TURN JUN
doteet
TUP TUPP nep
geannaed)
TiPH REGHT TURN RIGHT TIMP TURN RIGHT TUPP TURN RTGHT 2Pp
y y
desci informes
new nPP nPp W. nox 1oo0 ITRR nPP
TURN LFFT 3.00 TURN LEFT TIPP TUEN LEFT TEPP "THNN LEFT TIP TURN FFT 11m0
recomsb22
JUMP ZUMP WILK WALK
JUMP JUP ATURN RUN
TURN LEFT JUMP TURN RIGHT JUNP TURN RIGHT JUMP TURN RIGHT JUPP TURN RIGHT ZUMP
Table 12: Comparison of generative and unconditional model predictions with and without data augmentation. The conditional model trained augmented data (final row) is able to compensate for errors in data augmentation (Example and generalize to examples never generated by the data augmentation procedure (Example 2). and often times if sample is correctly created by the augmentation procedure, the conditional model also gets right (Example 3).
Below set of samples from the learned aug (basic) model for SCAN dataset's 's jump and around right splits, in order:
IN: run opposite and walk opposite right twice OUT: RUN TURN RIGHT TURN RIGHT RUN TURN RIGHT TURN
RIGHT WALK
IN: look around right thrice after run around thrice thrice OUT: TURN RIGHT RUN RIGHT RUN TURN
RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN
TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT LOOK TURN RIGHT LOOK TURN RIGHT LOOK TURN
RIGHT LOOK TURN RIGHT LOOK
IN: look opposite te right twice and walk around twice OUT: TURN RIGHT TURN RIGHT LOOK TURN RIGHT TURN
RIGHT LOOK TURN RIGHT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
IN: run opposite and thrice OUT: RUN TURN LEFT RUN RUN
IN: walk opposite right thrice turn turn right thrice OUT: TURN RIGHT TURN RIGHT TURN RIGHT
TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT TURN RIGHT WALK
IN: jump opposite right twice jump look around Left OUT: TURN RIGHT TURN RIGHT JUMP TURN LEFT TURN
LEFT TURN LEFT TURN LEFT TURN LEFT TURN LEFT LOOK
19

---

Published as a conference paper at ICLR 2021
IN: walk around 1eft thrice after jump left left OUT: TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
TURN LEFT NALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT
WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT MALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN
LEFT WALK
run opposite right twice walk left thrice
TURN TURN
RIGHT WALK TURN LEFT TURN LEFT RUN TURN LEFT TURN LEFT RUN
Below are of samples from the recomb-1 model for SCAN dataset's around right split. Note that there were no samples with rare tags generated by the model for the jump split:
run around right after walk around left LEFT RUN TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK TURN LEFT WALK
LEFT
LEFT
IN: look around right after jump around left OUT: TURN LEFT LOOK TURN JUMP TURN LEFT LOOK TURN
LEFT LOOK TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP
IN: look around right and jump around left OUT: TURN RIGHT LOOK TURN RIGHT LOOK TURN RIGHT LOOK TURN
LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUMP
IN: walk around right and turn right twice OUT: TURN RIGHT WALK TURN RIGHT WALK TURN RIGHT WALK TURN
RIGHT WALK TURN RIGHT TURN RIGHT
Below are samples from the recomb-2 model for each of SCAN dataset's jump and around right splits, respectively:
IN: jump opposite left thrice after jump opposite left thrice OUT: TURN LEFT TURN LEFT JUMP TURN
LEFT TURN LEFT JUMP TURN LEFT TURN LEFT JUMP TURN LEFT TURN LEFT WALK TURN LEFT TURN LEFT WALK TURN LEFT TURN LEFT WALK
jump 1eft thrice and jump left thrice OUT: TURN LEFT LOOK TURN LEFT LOOK TURN LEFT LOOK TURN LEFT
JUMP TURN LEFT JUMP TURN LEFT JUMP IN: jump opposite right and turn around left OUT: TURN RIGHT TURN
TURN LEFT TURN LEFT TURN LEFT TURN LEFT
IN: turn around left jump around left OUT: TURN TURN LEFT LEFT TURN LEFT TURN LEFT
JUMP TURN LEFT JUMP TURN LEFT JUMP TURN LEFT JUNP
IN: look right twice after around right OUT: TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT
RUN TURN RIGHT RUN TURN RIGHT LOOK TURN RIGHT LOOK
IN: turn right twice after look around right OUT: TURN RIGHT LOOK TURN RIGHT LOOK TURN RIGHT LOOK
TURN RIGHT LOOK TURN RIGHT TURN RIGHT
IN: look twice and around right OUT: LOOK LOOK TURN RIGHT RUN TURN RIGHT RUN TURN RIGHT RUN TURN
RIGHT RUN
walk opposite right twice and jump around right OUT: TURN RIGHT TURN RIGHT WALK RIGHT TURN
RIGHT WALK TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP TURN RIGHT JUMP
G.3.2 MORPHOLOGY
Below of samples from the learned aug (basic) model in SIGMORPHON format.
şahmiçe şahmiçende N;
karadan havaya füze karadan havaya füzel PLL
ernek erneklerine PLPPS3P
kiler kilerine PSD
mahlep mahlebimizi SG,555
süzmek süzerler V;IND; POSS
ålap ålaps 3,
jöle jöleleri PL
envejecerse onvejeciéndose PRC colaxar colaxa 3;
pergedrer no pergedremos V:NEG;INP; PL
mantear no mantees IMN;
flaguear no flagueen VINEG;IMP;3;PP
malacostar malacostaría SG
20

---

Published as a conference paper at ICLR 2021
desinstar desinse 3MP
concretizar no concretices V;NEG; IM;
Below set of samples from the learned aug (basic) resampling model.
şaşırmak şaşırmıyor musun? PL INTR
ay1lnak ayllmaya 1;
pleşnek plesmiyor muyuz? 1;IND; PST PROG; ;ROG; INTR
imciyetmek inciyetmezdegiz 1;IND; FUT; NEC
kuvaşmak kuvagmayacagız 1; PL, FUT; NEG; DECL
yermek yermeyecegiz 1PL; ;
yarı tmak yaritmayacagız V; IND 1 PL; FUT, NEG; DECL
kelinek keLimeyecegiz V;IND; FU; DECL
trasescar trasescáis V; :
tronar tronar V;IND;FUT;;::5
terzcalminar terzcalminan nan 3;; esubronizar esubronizamos 1;
urdir urdiremos FUT; 1PPL
conder conderemos F;IND;FUT;::PP
florear florearían V; 3 LL, sabrordar sabrordamos V; P;I, 1PP
Below are set of samples from the recomb-1 + resampling model (the best performing model in Table 2). Here additionally annotate samples with error categories.
kovulmak kovulmaz misiniz V; IN; FUT IEE; (Inflection and tags don't match.) (Correct and navel.)
düşünnek düşündüler PSOS sütmek sütmezz miyiz? 20; NEG; INTR (Inflection and tags don't match.)
bakmak bakmayacak mıy1m? V;IND; ; INTR (Inflection and tags don't match)
döndürmek döndürecek misiniz? IND PL; ;O;; INTR (Correct and novel.)
türkgeleştirtmek türkgeleştirtäyor m ; NEG; INTR (Wrong inflection, novel tag.)
çalmak çalmayız :L: (Inflection and tags don't match.)
üsürnek üsürmezsin IND; NEG (Inflection and tags don't match.)
duplicar duplicaráis FUT; (Correct and novel)
efundar efundan :: (Inflection and tags don't match)
deshumanizar deshumanicas PS;; (Inplausible inflection.)
emular emulares 22; (Correct and also in train set.)
languidecer languideciamos V; 1; (Inflection and tags don't match)
nominar nominamos V;SBJV;FUT;; (Novel tags, incorrect inflection.)
finciar finciare 1;; (Correct and novel.)
abastar abasto V:IND;PST:1;:8 (Inflection tags don match)
21

---

Published as a conference paper at ICLR 2021
G.4
ATTENTION HEATMAP
Here we provide a visualization copy and attention mechanism in recomb-2 model for SCAN experiments.
action probs
x-11 attention
<.20
4,20
LITVRN 1277
LITVRN 1277
1330
1338
4310
LUUAN Juzzz
1,10
K30
K35
K34
<32
your croucc wile
K3
201
Lu0de
gemerates aled sample xx22 attention
LUWE 1277
LTWW JIZIT
LINN
un
100
persie prott con exoite wot scotti ton secott
Coree Efilfe
Limes Loffe
cineerabed sample cone Lenn Witi
geherated samp e
Figure 3: Generation a sample. We plot normalized output scores on the left, and attention weights to the different prototypes on the right. The prototypes are on the axes. The model is recomb-2 model trained on SCAN jump split.
H COMPUTE
We use single 32GB NVIDIA V100 Volta GPU for each experiment. For every experiment, the whole pipeline which consists of training of the generative model, sampling and training of the conditional model takes less than an hour.
22


Reviews:
[{"review_id": "VNmLOclumTt", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n* Motivated by the fact that certain datasets require modeling compositional phenomena, the lack of flexibility of highly structured models, and the strong performance of large unstructured models on unstructured data, this paper approaches the problem of getting unstructured models to generalize on compositional data.\n* Prior work showed that a simple rule-based data augmentation approach could allow unstructured models to generalize on compositional data. This paper demonstrates that a learned data augmentation strategy can be as effective at encouraging generalization as a rule-based one.\n\nContributions:\n* Extends the prototype+edit model (Guu et al 2018) to a recombinator model with multi-source copy attention.\n* Proposes a resampling scheme for upweighting rare examples.\n* Obtains results comparable to strong rule-based data augmentation baseline GECA on two datasets, demonstrating that the combination of both resampling and recombination is effective.\n\nStrengths:\n* Clearly written. The method is simple and is broken down cleanly into recombination and resampling.\n* Ablation studies support the need for both recombination and resampling.\n* Sufficient performance to support the claim. The approach manages to match the performance of GECA with a learned method, while remaining more flexible.\n\nWeaknesses:\n* Neighbourhood heuristics are one of the last remaining applications of manual rules in the method, but seem necessary for computationally feasible training.\n\nDecision: Accept\n* Problem is important: Whether unstructured models can generalize on structured data has implications for whether or not to move towards more structured models. This paper provides experimental evidence that unstructured models can generalize on structured data with a data augmentation procedure that uses fewer manually specified rules than previously shown. This provides a path forward by continuing to iterate on the augmentation procedure.\n\nQuestions:\n* Given that GECA is feasible on both datasets, would there be benefit to combining the examples from the learned augmentation strategy with those from GECA? In other words, do the different augmentation strategies result in orthogonal improvements?\n* Why does resampling hurt GECA on SCAN?\n\nSuggestions:\n* Both datasets are quite small. The story could be strengthened by demonstrating the method scales better than GECA by applying it to a larger dataset as well, such as a translation dataset.\n\nNit:\n* w is overloaded to both be a value of d (eqns 12, 13, 17) as well as the weighting function (eqns 1, 18)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TqhM7h1Ob7f", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a prototype-based method for data augmentation based on a generative model without rule/template based requirements. The generative model creates new input-output pairs from training fragments (recombination: rewrite model conditioned on multiple examples), and samples in low-density places (rare words) of the training data (resampling). Empirical results show that the in combination recombination and resampling perform on par with a recently introduced rule-based method, GECA.\nExperiments are conducted on two compositional generalization tasks: SCAN and sigmorphon. \n\nThe paper is very clearly written and motivated, doing a good job in presenting the recent and past pertinent literature. The problem addressed is of great interest, and the two proposed contributions of resampling and recombination are likely to be useful to further research in data augmentation. \n\nThe extension of prototype models to multiple examples is a promising step, but depending on the task leaves open questions. \nThe empirical results on SCAN are strong. The results on Sigmorphon are strong in the sense of obtaining comparable accuracy to simple rule-based approach, which itself is very simple and has many incorrect examples it constructs, but do not clearly outperform it.\nGranting resampling is a contribution, wouldn’t the proper comparison be GECA resampling with the recomb-1 or -2, since those include resampling as well? In that case the Sigmorphon performance is much closer to each other for the two methods. \n\nTaking the performance jump from 1 to 2 prototypes on SCAN as potential for further jumps, why restrict to n=2? And related, why do you think -2 outperforms -1 in some instances on Sigmorphon?  \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AWIkIvYdf15", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "####Summary: \nTo tackle situations where compositionality is mostly required at inference time, the paper proposes a novel data augmentation method with an RNN based generator (recombination); to make the generator generate highly compositional patterns, the paper proposes a resampling method. The methods have been tested on two benchmarks focusing on the issue, SCAN and morphological analysis. The system performs on par with recently proposed GECA for SCAN and favorably to GECA on morphological analysis. \n\nThe two datasets have some “toy flavor”, while SCAN great favors example combination (with recomb-2 performs much better than recomb-1), recomb-1 seem to perform better for morphological analysis dataset, leaving questions about how to choose the exact models in general. \n\n####Pros: \nThe paper proposes the first RNN based neural generator to perform data augmentation for “extreme” compositionality inference. The paper has explained and empirically showed that this learned generator needs a resampler. With these two elements, the approach performs on par with recently proposed GECA (where the data is not augmented via a neural generator) on two datasets.\n\n####Question: \n1. Comparison with GECA: I can read from the paper that the performance is on par with GECA. However, I am unable to grasp nuances, leaving important questions untouched such as: In what scenarios do we expect the model to perform better than GECA? In experimental details, the slightly better performance in Table 2, can it be attributed to finer generation powered by the RNN generator? Why does Recomb-2 perform less well than GECA in SCAN? \n\n2. A uniform framework for resampling\nDifferent recombinations perform more or less favorably across different datasets. While the exact choice depends on the dataset characteristics, a framework will be more attractive if it can perform well on different scenarios. Could the authors list some possible approaches to automatically choose this hyper-parameter please?\n\n####Minor Comments: \nThe paper mentions in several places symbolic scaffolding without citations, literature is certainly rich here, e.g. [1,2] are papers integrating symbolic constraints for semantic parsing. There are also neural architectures that particularly target to ensure some symbolic famous properties such as [3]. \n\nThe authors say in the introduction that the approach (Andreas, 2020) is task specific which seems not correct. In fact, it can be applied to a large range of NLP problems (e.g. all the experiments in this paper compares to the approach GECA))\n\nIn section 3, it says that “the use of a continuous latent variable appears to make no difference”, I would suggest to precise “make no difference” as “make no difference in prediction performance“ as the latent variable can facilitate some generation control shown in (Guu et al. 2018). \n\nIn this paper, GECA is first introduced in section 5. I would recommend to put the citation around it (Andreas, 2000) although previously cited.\n\n[1] Sequence-based structured prediction for semantic parsing, Xiao et al. 2016\n[2] A syntactic neural model for general-purpose code generation, Yin and Neubig 2017\n[3] Making Neural Programming Architectures Generalize via Recursion, Cai et al. 2017\n\n####Authors have engaged in the discussion, clarified questions about the paper and addressed comments in its newest revision. I have consequently revised my score from 5 to 6. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "5nE_CBF5ZLh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\n\n \nThe paper proposes an interesting approach to systematically generate new examples and augment the training data with these examples. The goal is to target rare and unseen sequences of text or instructions with this augmentation. \nIn particular, it proposes learning to copy parts of the reference examples. The approach is based on prototype-based models where every training example can be explained by at least one other example and a parametric rewriting operation. They show that these models do not perform well when facing complex (and rare) composition events and propose a recombination addition to address this issue.\n \nI like the idea of systematically augmenting training data targeting rare and unseen subsequences. I give an \"accept\" to this work because of its novelty and contribution (see pros below). \nMy minor concern is about the impact and/or usefulness of this work when dealing with real-world datasets and more complicated tasks as well as some clarity issues (see cons below). Hopefully, the authors can address my concern in the rebuttal period. \n\n ##########################################################################\n\nPros:\n\n1. The paper addresses one of the interesting and important shortcomings of current neural models: the ability to generalize to rare and unseen sequences. I find this problem important to investigate and applicable to many areas of research.\n\n2. The proposed approach is flexible and practical to use. The design of the prototype-based data augmentation method is reasonable and interesting. \n\n3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. \n\n \n##########################################################################\n\nCons: \n\n \n1. In Table 2, we see F1 score for morphological analysis. It is not entirely clear to me why the results in the NOVEL section of the table is very close to ALL section. NOVEL shows model accuracy on examples whose exact tag set never appeared in the training data and I expected a bigger gap in the performance.\n\n2. There is a big gap between the performance of 1- and 2-prototype models. Do the authors know what explains this gap? Have the authors explored the higher order of prototype recombinations? \n\n3. What are the challenges of applying this approach to real-world data sets for instance in machine translation? I will suggest the authors discuss the implications and possible shortcomings of such an approach when dealing with natural (and potentially long) sequences of text. The definition of unseen subsequences and compositional learning will be more complex there. \n\n4. It is not clear to me what the authors mean by \"hints\". Is the complete sequence counts as a hint? Or only the subsequence that was identified as rare?\n\n \n#########################################################################\n\nSome typos: \n\n\n(1) Relevant literature that was not mentioned in this paper:\n         https://arxiv.org/abs/1705.00440\n         https://arxiv.org/abs/1801.02929\n\n(2) Equation (31): tex formatting issue.\n\n(3) Typo in page 15 section G1: Fig. Table 2 shows -> Table 2 shows\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses generalization to compositions of rare and unseen sequences. It proposes an unstructured data augmentation, that achieves comparable generalization to structured approaches (e.g. using grammars). The idea is based on recombining prototypes and oversampling  in the tail. 

The paper provides a novel approach to an important problem. All four reviewers recommended accept. 


Decision: Accept (Poster)