Data Factors for Better Compositional Generalization
Authors: Xiang Zhou
Yichen Jiang
Mohit Bansal
Venue: EMNLP 2023 Main (2023)
Keywords: compositional generalization
data factors

Abstract: Recent diagnostic datasets on compositional generalization, such as SCAN (Lake and Baroni, 2018) and COGS (Kim and Linzen, 2020), expose severe problems in models trained from scratch on these datasets. However, in contrast to this poor performance, state-of-the-art models trained on larger and more general datasets show better generalization ability. In this work, to reconcile this inconsistency, we conduct an empirical analysis by training Transformer models on a variety of training sets with different data factors, including dataset scale, pattern complexity, example difficulty, etc. First, we show that increased dataset complexity can lead to better generalization behavior on multiple different generalization challenges. To further understand this improvement, we show two axes of the benefit from more complex datasets: they provide more diverse examples so compositional understanding becomes more effective, and they also prevent ungeneralizable memorization of the examples due to reduced example repetition frequency. Finally, we explore how training examples of different difficulty levels influence generalization differently. On synthetic datasets, simple examples invoke stronger compositionality than hard examples do. On larger-scale real language datasets, while hard examples become more important potentially to ensure decent data coverage, a balanced mixture of simple and hard examples manages to induce the strongest generalizability.

Data Factors for Better Compositional Generalization
Xiang Zhou
Yichen Jiang
Mohit Bansal
UNC Chapel Hill
(xzh, yichenj, mbansal] y 0cs unc edu
Abstract
examples containing novel combinations of seen el- ements. However, in contrast to these results, mod- els trained (or pretrained) on larger datascts, (in- cluding T5 (Raffel et al., 2020), PaLM (Chowdhery et al.. 2022), etc.) show substantially better perfor- mance (Furrer et al., 2020; Drozdov et al., 2023) for compositional generalization. While some factors the emer-
Recent diagnostic datasets compositional generalization, such as SCAN (Lake and Ba- roni, 2018) and COGS (Kim and Linzen, 2020), expose severe problems in models trained from scratch on these datasets. However, in contrast this poor performance, state-of-the-arl mod- els trained on larger and more general datasets show better generalization ability. In this work, conduct
behind many of these improvements gent abilities from scaling up (Wei et al., 2022), we explore an alternative and complementary an- gle: how and why does training on more complex datasets help a general Transformer model, with limited size and capacity, generalize composition- ally to unseen natural language queries?
reconcile this inconsistency, an empirical analysis by training Transformer models on a variety training sets with dif- ferent data factors, including dataset scale, pat- tern complexity, example difficulty, etc. First, we show that increased dataset complexity can lead to better generalization behavior on mul- tiple different generalization challenges. To further understand this improvement, we show two axes of the benefit from more complex datasets: they provide more diverse examples compositional understanding becomes more effective, and they also prevent ungeneraliz- able memorization duced example repetition frequency. Finally. we explore how training examples of differ- ent difficulty levels influence generalization differently. On synthetic datasets, simple exam- ples invoke stronger compositionallty than hard examples do. On larger-scale real language datasets, while hard examples become more important potentially to ensure decent data cov- erage, balanced mixture of simple and hard examples manages to induce the strongest gen- eralizability.'
To answer this question, we empirically study how changes in data factors (e.g., scale, diversity, difficulty, etc.) influence the generalization abil- ity of models trained from scratch. Our first anal- ysis is inspired by Patel al. (2022) and Jiang et al. (2022), who make the model generalize sig- nificantly better on SCAN simply by increasing the number unique primitives. We extend this observation in several directions and further ex- plore where this gain comes from. Our experi- ments show that this is not a special observation SCAN for the Jump and Around Right split. Instead, the same effect can be observed on mul- tiple different types of generalization challenges (e.g., primitive-level generalization, length-level extrapolation, etc.), and on both synthetic and real- language datasets. To summarize our first direction of analysis, there is a strong connection between increased dataset complexity and better composi- tional generalization.
examples due to re-
1 Introduction
Many recent diagnostic datasets, c.g., SCAN (Lake and Baroni, 2018), COGS (Kim and Linzen, 2020), etc., have exposed the compositional generalization problem of neural sequence-to-sequencc (seq2seq) models. They show that seq2seq models that are trained from scratch fail miserably when tested on
Second, we try to understand why more complex datascts lead to better generalization. We build up our analysis by comparing two potentially compet- ing behaviors of the models: surface memorization (i.e., memorizing the direct mapping from inputs to outputs without understanding the underlying composition), and compositional understanding
'The code and data for this work are available at https: // it Chub, com/ owwnex//eeooooooooooooooooooo / da comp.

---

Memorization
Composition
Difficuit (4 urigU prrsties
Orggha
Rewe
Higs
Vakus
eaary!
wools left aesi volk Ieice
WALK 2+ LOCK
WWi 4
1im
ITUIE
Onadity
Jesco
wocks STURN VANK
Lnoww
LPUSH
INLL
SPRN
LOOK
I can't memontee ai of Jien!
Now 1d rather
Easy (2 urique primttves cxampp)
comppoon
LOOK
Lok pelpaantene LPL LODN PUSH Pizzi
Ieb
PrS
PSB
Falie
1wrk
US sedjog Hice
STARN GPRUNT J0G JCG FRIRRIIPLI IN RUSSIUM
ADG
Datasel Complexity
Example Difficullty
Figure I: Model's compositional generalization ability is affected by the complexity of the dataset (left) and the difficulty of training examples (right).
(i.e., first discovering set of compositional rules from examples and then applying the rules for pre- dictions). Please also refer to the left part in Fig- ure for an illustration. While both behaviors can lead to near near-perfect training-set performance, only the latter guarantees good generalization per- formance. We argue that more complex datasets improve compositional generalization as they pro- vide substantial obstacles to surface memorization. Specifically, we make two hypotheses: (1) diver- sity: more diverse patterns (c.g., more unique prim- itives) exemplified in complex datasets increase the difficulty of surface memorization; (2) frequency: the larger dataset size causes a decrease in the fre- quency of seeing similar examples, thus preventing their memorization. To empirically confirm these hypotheses, we provide detailed ablations showing how both factors contribute to empirical gains in data augmentation. To further isolate the effect of frequently recurring examples, we show that deliberately encouraging example repetition even in large dataset brings substantial detriment to the generalization performance. Furthermore, as a corollary to our hypotheses, we provide a simple yet effective data augmentation method AugZero that satisfies both contributing factors without uti- lizing any dataset-specific knowledge.
all the diverse natural language phenomena, sim- ple examples alone are not enough for achieving good performance, however mixing difficult exam- ples with simple examples is still beneficial on the compositional dataset SMCalFlow-
In conclusion, we present an empirical study on how data factors influence compositional general- ization behavior. We notice: (1) increased dataset complexity can improve the model's generaliza- tion ability; (2) increased complexity creates obsta- cles for surface memorization by having higher cost of memorization and less recurring frequency of examples; (3) example difficulty can influence generalization substantially with simpler examples benefiting compositional generalization more.
2 Tasks and Setup
2.1 Datasets
In this work, we focus on semantic parsing datasets, We use both datasets with synthetic examples and datasets with natural examples. We experiment with synthetic datasets to have fine ine-ggained con- trol and experiment with flexible natural language datasets to demonstrate the generalizability of our findings. For synthetic datasets, we use both the original version of SCAN (Lake and Baroni, 201 8) as well as expanded version with increased com- plexity, denoted as SCAN* in this work. We intro- duce SCAN* to control the overall complexity of the SCAN-like 2 2 Compared to SCAN, we make two major changes: (1) We remove the con- straint that at most one conjunction (and or after) is allowed for every example. Instead, in SCAN* every sentence can have an unlimited number of conjunctions. To avoid adding ambiguity brought by multiple conjunctions, we assign higher opera- tion priority to after than and. (2) We use larger
Finally, we provide more fine-grained analy- sis to study how examples of different difficulties inside a dataset affect generalization. Our anal- ysis covers both smaller, synthetic datasets and larger-scale, real-language datasets. On synthetic datasets (e.g., SCAN), we create multiple versions of the datasets strictly controlling the difficulty of the examples in each version, and show that sim- pler examples facilitate compositional generaliza- tion more than difficult examples. On sub-sampling experiments real-language datasets (e.g., ATIS, SMCalFlow), we observe similar but more compli- cated trends. Potentially due to the need to cover
"The original grammar in SCAN does not allow examples with more complexity.

---

Jump
Around Right
GeoQuery (query)
GeoQuery (question)
baseline
1.4
19.86-1041
42.37=+326
64.52+1.44
2x Augmentation
41737
73.02=2012
45.92=3317
69.10=ass
20x Augmentation
99.68+0032
99.38=aes
47.85+3.9
68.17+211
200x Augmentation
99.932004
99.01 =005
45.70=116
65.45=223
Table 1: Datasets with increased complexity via data augmentation lead to better compositional generalization. We use logic form outputs for GeoQuery in this table. For SQL results with similar trends, see Table in the Appendix.
set of verb-type primitives (i.e., run, walk, etc.). The first change allows to increase the number of possible example structures and lengths, and
prove the performances. Notably, the main effect of these methods is to increase the number of prim- the second allows the increased lexicon-level com- plexity. For natural language datasets, we use three English datasets with human-written queries: GeoQuery (Zelle and Mooney, 1996), ATIS (Price, 1990; Dahl et al., 1994) and SMCalllow (Andreas et al., 2020; Yin et al., 2021). For GeoQuery, we use both the query split and the question split fol- lowing Andreas (2020). And for SMCalFlow, include the 32-shot version in Yin et al. (2021) as a compositional split and denote as SMCalFlow- CS. For fair comparison across splits, we prepro- cess SMCalFlow and SMCalFlow-CC way. More details are in Appendix A.
itives. They neither provide any structure-level change nor create any overlapping between the training and the testing set. Therefore, the improve- ment seems to be solely from a more complex train- ing set. In this section, we replicate and extend their findings, showing this phenomenon is in fact consistent on multiple generalization challenges.
3.1 Experiment Design
We first follow the same setup as Jiang et al. (2022) to conduct data augmentation experiments on the SCAN Jump and Around Right split, creating aug- mented training sets with more primitives. We also extend this process to GeoQuery dataset (Zelle and Mooney, 1996) to investigate the effect on datasets with more natural and realistic language. Please refer to Appendix C for details of our replication. Additionally, we include the SCAN* Length split to investigate the effect of data complexity on a different type of generalization: length ex- trapolation. To allow more flexible control of length, we use the expanded SCAN* dataset de- scribed in See. 2.1. Specifically, we train the model on training sets containing examples with length I S L, and challenge on test examples with length < 1 S 2L. To control the com- plexity, we create four splits with different values of L E (31, 62, 125, 250). For all four splits, we make sure they have similar dataset statistics when- ever possible. See Appendix A.2 for more details.
the same
2.2 Implementation Details
We focus on seq2seq Transformers as they are the most prevalent choices for most semantic parsing datasets. For our main experiments, we train the Transformer from scratch so that our analysis is not influenced by the existing knowledge from pre- training. Our model configuration mainly follows Csordás et al. (2021) to use a 3-layer Transformer which works well in most compositional tasks. We provide additional results on models with other configurations in Appendix H. We decode using a beam size of 5 and select the best model based on their dev-set exact-match accuracy. For any exper- iments in the same table figure, we train every model using the same amount of total steps even though the training dataset size may vary. Unless otherwise mentioned, all the results are the mean of runs. More implementation details are in Ap- pendix B.
3.2 Results
In Table 1, we show the generalization perfor- mances when trained with different complexities (i.e., different numbers of total primitives). On both SCAN and GeoQuery, increasing the training- set complexity substantially improves the general- ization performance. On SCAN, the models ob- tain huge improvements with 2x augmentation and reach a near-perfect accuracy with 20x or more primitives. Similarly, on GeoQuery, the augmented performance substantially outperforms the base-
3 Increased Training Set Complexity Leads to Better Generalization.
On the SCAN (Lake and Baroni, 2018) dataset, models trained from scratch show very poor per- formance, especially on the Jump split. However, recent works (Patel et al., 2022; Jiang et al., 2022) use data augmentation methods to significantly im-

---

jump
walk
look
Analysis on the entire embedding matrix and then project these four embeddings the first two prin- cipal components. In Table 2, we show the projec- tion of the embeddings trained on Jump 1x versus the embeddings trained on the larger Jump 20x. We observe that the four primitives trained on Jump 20x are closer together in the first principal compo- nent, ranging from 0,41 to 0.75 (0.53=0.13) com- pared to -0.84 to 0.75 for the baseline (0. 0.57). This suggests that compositionality arises because the model can better represent the syntactic simi- larity between the rare primitive "jump" and other common primitives, which uppear in very different contexts during most of the training.
1x
-0.84, 0.57
0.02, 0.59
0.75, 0.15 0.45, 0.50
0.20, -0.13 0.75, -0.10
20x
0,41, 0.11
0.49, -0.01
Table 2: The projections of "jump, walk, run, look" onto the two most principal components of the embedding space, when trained on Jamp or Jump 20x.
Matet-MMtthhyae Agc
-5
60
25
125
Length
Figure 2: Different generalization performances on SCAN* Length. The challenge always train on examples with length 0 < l & L and test examples with length 1 < 2L.
4 Understanding the Advantages Brought by Increased Complexity
line, reaching around 5% accuracy improvement with the best augmentation." 3 Interestingly, there is a difference in the optimal augmentation between SCAN and GeoQuery. On SCAN, larger datasets always lead to better results until the model reaches near-perfect accuracy. However, on GeoQuery, the best augmentation is not the most complex 200x data, though a larger amount of augmenta- tion still leads to improvements. We suspect this phenomenon is related to the frequency of primi- tives, and please see Sec. 4.3 for more discussion. Besides primitive-level generalization, increased complexity is also beneficial for simple structure- level challenges such as length generalization. In Figure 2, we see steadily increasing performance when both the training set contains longer exam- ples. For the same generalization challenge (i.e.. generalizing to twice the length seen during train- ing), the same model can reach over 80% accu- racy when the training set contains sentences with 250-token length, but only reaching around 15% if the training sentences are all within a length of 62. These results demonstrate that across multiple different generalization challenges, more complex training sets seem to bring a consistent gain.
We further explore how increased complexity im- prove generalization. We will first provide a discus- sion on dataset complexity and model behaviors. Then, we will state hypotheses how data complexity impacts generalization by influencing the model behaviors. Finally, we provide support- ing experiments.
4.1 Preliminaries
Two types of data complexity. For simplicity, we provide an informal discussion of "data com- plexity" for semantic parsing tasks. Data complex- ity can be measured from many different angles. In this study, we only consider the training set and mainly focus on two types of data complex- ity: (1) Pattern complexity: a training set with larger pattern complexity will contain more unique patterns (e.g., more unique primitives, more di- verse example length, etc.) in the examples. (2) Scale complexity: a training set with larger scale complexity will contain more different examples. Note that these two properties are usually positively correlated in most datasets larger training set usually contains more diverse patterns. In this sec- tion, we aim to explain how increments in these complexities influence the model's 's generalization.
Analysis on learned primitive embeddings. To better understand how training on a more complex dataset (e.g., Jump 20x) changes the model's ss be- havior, we conduct an analysis on the word embed- dings of four primitives: "jump, walk, run, look". First, we perform a lincar Principal Component
Two competitive model behaviors. One way to understand the different generalization behaviors is to focus on how models achieve good training per- formance. We argue the difference in how models achieve good training performances heavily influ- ences the generalization performance. Intuitively, for a seq2seq task, there are two extremes of be- haviors that models could adopt: (1) surface mem-
'Note that unlike SCAN, primitive-level generalization is not the only challenge in GeoQuery, so the scale of the improvement is noticeubly smaller. Nonetheless, the improve- ments shown in Table are consistent across different splits, output formats, and augmentation sizes.

---

orization: the model only establishes one-to-one maps from the inputs and the outputs and does not correctly infer the composition: (2) compo- sitional understanding: the model makes predic- tions based on a correct understanding of how the semantics are composed by smaller sub-structures." An illustration of these two behaviors is shown in Figure 6 in the Appendix. In practice, the model's behavior is not binary choice between these two extremes, but oftentimes a complicated combina- tion depending on the inputs. Nonetheless, here we use these two concepts to denote two trends of behaviors, and we will show that increased data complexity biases the model toward the second behavior, and hence leads to better compositional generalization.
Dotaset
SCAN (Around Right)
GooQuaay (qurry)
Origin Dataset
19,861mmi
42.37
2x Augmentatiin
73.02
45.92a417
More primilives
96 09
43.55
prim. larger size
czz
47.85 1:30
AngZero
95. 25.550
43.55gmm
Table 3: Both dataset size and the number of primitives contribute to the performance improvement.
difficulty of compositional understanding, leading to a preference for the latter.
(2) Scale complexity (i.e., larger datasets) avoids memorization by reducing frequently recurring examples. Another effect usually brought by in- creased dataset complexity is more training exam- plcs. This naturally leads to a decreased frequency with which similar examples occur during the train- ing. As is shown by Hernandez et al. (2022), even small portion of recurring data can bring substan- tial damage to the copying performance and the scaling law of language model. We argue that similar to this finding, reduced example recurring frequency will make surface memorization harder and thus encourage compositional understanding.
4.2 Hypotheses
We elaborate on our hypotheses of the connec- tion between data complexity and model behaviors. Note that our two hypotheses are interrelated and not contradictory to each other. They are just two different perspectives on how this connection.
4.3 Supporting Experiments
(1) Pattern complexity (i.e., more diverse exam- ples) increases the difficulty of surface mem- orization. For surface memorization, the mod- els need to memorize all the individual mappings from each example input to each example output. The difficulty of achieving specific training loss through surface memorization is proportional to the total sum of the complexity of every unique example in the dataset. With a larger training set containing more diverse examples, the difficulty of surface memorization increases substantially, However, with correct compositional understand- ing, the difficulty will remain roughly the same as the rreviousyy-learned composition remains cor- rect. Hence, with more complex datasets contain- ing more different examples, the difficulty of sur- face memorization increases much faster than the
In this section, we provide empirical evidence sup- porting our previous two hypotheses.
Two sources of empirical improvements. First, try to separate the benefits of data augmenta- tion shown in Sec into two parts: benefits from increasing pattern complexity and from increasing scale complexity. In Table 3, we present models trained with data of three different level of com- plexities. The "+ More prim. & larger size" row is the model trained with the full x20 data augmenta- tion with more primitives and larger dataset size. For the "+ More primitives" row, we down-sample the augmented part of the x20 data" that the total size is the same as the x2 augmented dataset.? ? On both SCAN and GeoQuery, by comparing 'Origin Dataset' and '+ More primitives' rows, we find that increasing pattern complexity can encourage stronger compositional generalization. Addition- ally, by comparing the '2x Augmentation' and the '+ More primitives' row, we notice that the trade- off of having more primitives but fewer examples
"This may remind readers of classic overfitting behaviors. Indeed, traditional loss regularization methods (Li et al., 2019; Yin al., 2023) can improve compositional generalization extent. However, finding the hest regularization method is trivial, and also need understand methods (e.g.. LLLS)) free of explicit regularization. Our work aims pro- vide related insights from the dataset angle.
'Depending on the actual datasel properties, compositional understanding may not always be the correct behavior (c.g., in translation as shown in Dankers et al. (2022)). However, for the scope of this study, we focus on how more complex datasets encourage compositional generalization, so we as- sume compositional undcrstanding as the ideal behavior.
"We do not down-sample whole dataset as will create an unwanted side-cffect removing large portion the original data compared the x2 data.
"Note that there is no ablation only with larger sizes, since if we keep the original examples unchanged, increasing dataset size will inevitably require new primitives.

---

Data Size
Repetition
GeoQuery (quer))
GeoQuery (question)
tition still causes substantial damage, showing over 15 points drop. To summarize, earlier in this paper, we show that increasing the dataset size generally helps generalization. However, Table 4 show that naturally increasing the dataset size while still ex- plicitly adding highly recurring examples actually leads to worse results. Combining these two ob- servations, we provide evidence supporting our hy- pothesis on the importance of reducing frequently recurring examples.
Original Original Original
64.5
Example Primitive
24,84 84
55,48=451
1.
43.08
20x
47.85,33.
68.171227
20x
Example
44,95=330
68.46eLa
20x
Primitive
Table 4: The performance of models trained with differ- ent types of example repetition curriculum.
for each primitive is not always beneficial, leading improvement on SCAN, but not on GeoQuery. Finally, by comparing "+ More primitives' and '+ More prim. & larger size', we can further conclude that increasing scale complexity given the same amount of distinct primitives can provide further gain in generalization results.
Simple Data Augmentation with Zero Prior Knowledge Finally, we provide data augmen- tation method as a direct corollary of our hypothe- ses. We call our method Augmentation with Zero Prior Knowledge (AugZero) as it brings zero ad- ditional knowledge but can still be effective it satisfies our two previous two hypotheses. The main idea is to simply copy the entire vocabulary and use the newly copied vocabulary to re-tokenize the entire dataset. Figure 3 shows an illustration for AugZero with k times augmentation. A more detailed description is in Appendix C.3. The results with k 200 are in Table 3. Despite how simple the algorithm is, AugZero can still substantially improve the performance, reaching close per- feet performance on the SCAN Around Right split and achieving decent improvement on GeoQuery. While the gain is smaller than the primitive-aware augmentation result (+ More prim. & larger size row), its success further supports our hypotheses.
Performance detriment from example repeti- tion. To further support our second hypothe- sis, we present experiments to show that highly- recurring examples can cause significant perfor- mance decrease. We compare three training curric- ula with the same total steps in Table 4. The first curriculum with no repetition lum where every epoch contains the entire training set. The other two curricula are specifically de- signed to aid surface memorization by having more recurring examples. For the Example Repetition curriculum, at the first 20% of the training steps, the model is only trained on a small subset contain- ing 20% of the examples, then the remaining data are gradually put back into the training set until the training set becomes the full dataset at 80% of the total steps. This curriculum ensures that the model is repeatedly trained on a small set of examples for a long period. The Primitive Repetition curriculum is similar to the Example Repetition curriculum except that it first clusters the training examples by the primitives and then starts to train the model with data containing only 20% of the primitives. In Table 4, we confirm that example repetition can bring substantial damage to the generalization per- formance. On smaller original datasets, repetition at both the example level and the primitive level substantially hurt the performance. On GeoQuery, Example Repetition leads to 10 to 20 accuracy drop, and the damage brought by Primitive Repeti- tion is even larger. With a larger dataset resulting from data augmentation, the trend is slightly differ- ent. Repetition the example level causes much smaller drop, possibly because that 20% of the aug- mented dataset is still relatively large so there is not much repetition. However, primitive-level repe-
normal curricu-
5 Which Examples Benefit Generalization the Most: A Difficulty Perspective
Previously, we demonstrate and explain that in- creasing the complexity of the whole dataset can improve compositional generalization. In practice, examples inside dataset have different properties (e.g., difficulty, topic, etc.) and thus may influ- ence generalization differently. In this section, we present a study from the difficulty perspective. We will start with a discussion on the definition of dif- ficulty, and then show how example-level difficulty can substantially influence compositional general- ization on both synthetic and real datasets. For this section, report results with three runs as the dataset sizes are substantially larger.
5.1 Example-Level Difficulty Metrics
Complexity-based difficulty. For synthetic datasets like SCAN, we can intuitively define the difficulty of an example by the complexity of the

---

Original
Datasets
Simple
Hard
Mix
walk loft and jump twice LTURN WALK JUMP JUMP
ATIS
52.26 40.8
48. 11
jump opposite right
RTURN
RTURN JUMP
SMCalFlow
37.35+a26
46,38+0.55
43.69+a35
AugZoro-augmented Dataset
SMCalFlow-CS
36.72+a70
38.43+1.88
39.41+11
walk loft and jump twice LTURN WALK JUMP JUMP
Table 5: Different performance when trained subsets with different difficulty. Simple and Hard denotes the simplest and hardest subset as in Figure Mix is a mixture of both subsets.
walky lofl; Jumpy lalle;
LTURN, JUMP, JUMP;
walky loft and, Jum, swcc,
LTURN, WALK, JUMP, JUMP,
jump apposite right
RTURN RTURN JUMP
jumpy opposiley right
RTURN, RTURN, JUMP,
jumoy opposiey 19gh
RTURN, RTURN, JUMP,
complexity) as the difficulty metric. For evaluation, we use the Jump split as its difficulty is substantially influenced by both types of complexity. For length- based complexity, we generate training sets with different maximum lengths, the same as described in Sec. For primitive-basee complexity, we keep the number of total possible primitives in the vocab- ulary fixed but generate multiple different training sets only varying the maximum number of unique primitives per example. Intuitively, it will be eas- ier to infer the correct composition from shorter examples or examples containing fewer primitives. The results are shown in Fig. 5. All the models are tested on the same testing set similar to the origi- nal SCAN Jump testing set. For both settings, we can see a steadily decreasing trend when the exam- ples become harder. When the maximum length is reduced from 500 to 62, the performance is in- creased from 16.65% to 47.31% %.9 With only 2 unique primitives per example, the performance is also increased to 49.14%. These results demon- strate that easier examples make the correct composition easier learn.
Figure 3: The AugZero data augmentation process.
example itself, for example (1) the length of the input instruction, or (2) the number of unique primitives, both reflecting the pattern complexity of the example. These metrics are the most reliable they directly reflect the complexity of correctly generating the target output examples. However, these metrics can be hard measure on real natural language datasets.
Prototype-based difficulty. Another limitation of the complex-base based difficulty is that it treats each example independently, while in practice the learn- ing is conducted on an entire dataset instead of individual examples. One metric addressing this issue is to see how prototypical (i.e., if there are other similar examples in the dataset) one exam- ple is. To measure this, we follow the process in Sorscher et al. (2022). We use SimCSE (Gao et al., 2021) to encode all the input of the examples, then cluster the encoded vectors using K-means, and use the I.2 distance the centroid the difficulty measure. An outlier that is far from any cluster is deemed difficult. See Appendix G.1 for cluster- ing examples for this method. In our experiments, we will use this difficulty as the metric when the ground truth complexity metric is not available.
Mix of simple and hard examples needed on real language datasets. We next examine the impact of example difficulty on more complex larger-scale natural language datasets. Due to the flexible and diverse nature of natural language in real datasets. models now not only need to understand the cor- rect composition but also need to capture other lan- guage uses through potentially non-compositional ways. Therefore, the trends in natural language datasets can be different from the previous obser- vation. For this study, we conduct experiments on ATIS (Price, 1990; Dahl et al., 1994), SM- CalFlow (Andreas et al., 2020) and the compo- sitional version SMCalFlow- (Yin et al., 2021). For all three datasets, we train models on multi- ple subsets with different difficulties but all con-
5.2 Experiments
We show how models behave differently with dif- ferent distributions of example difficulties.
Simpler examples make generalization easier on SCAN" First, as we have full control over SCAN* we directly use example complexity (in- cluding both length-based and primitive-based
*Another way to measure difffiullty is to directly use the model's performance (c.g.. accuracy). Using these metrics, models will perform very poorly when trained on the hardest subset as they fail to leamn from most examples, making the analysis results less interesting. See Appendix G.2 for more discussion on these metrics and detailed results.
"Note that the results here and the results in Fig. 2 are not contradictory. Here we show more examples with longer lengths make primitive-level generalization worse, while in Fig. 2 we show such data make length generalization better.

---

ATIS
SMCxFllww
SMCaFbb-CC
50
60
45
40.0
40
80
257
50%
75%
100%
25%
50%
7956
100%
25%
56%
75%
100%
Dificully
Figure 4: Results ATIS and SMCalFlow with training of different difficulties. The X-axis represents the quantiles of example difficulty, the smaller the easier.
Exact-Malch Acc
amples for compositional challenges still persists.
60
40
Related Work
500 Compositional generalization. Earlier related works in compositional generalization study the systematic behavior of neural networks in lan- guage learning (Wong and Wang. 2007; Brakel and Frank, 2009), compositional counting (Wiles, 1998; Weiss et al., 2018), syntax learning (Linzen et al., 2016), etc. Recently, many recent datasets (Lake and Baroni, 2018; Kim and Linzen, 2020; Loula et al., 2018; Bastings et al., 2018; Keysers et al.. 2020; Tsarkov et al., 2021; Hupkes et al., 2020) substantially facilitates the development and eval- uation of related method improvements. Since then, many different methods have been proposed, including architecture improvements (Dess) and Baroni, 2019; Gordon et al., 2020: Oren et al.. 2020; Zheng and Lapata, 2021), grammar-based approaches (Shaw et al., 2021; Kim, 2021), task decomposition (Herzig et al., 2021), data aug- mentation (Andreas, 2020; Akyüirek et al., 2021; Akyurek and Andreas, 2023), and novel learning methods (Jiang and Bansal, 2021: Lake, 2019; Con- klin et al., 2021; Jiang al., 2022). Of all these methods, the biggest breakthrough (Drozdov al., 2023; Qiu et al., 2022) still mainly comes from us- ing large language models (Chowdhery et al., 2022: Raffel et al., 2020). Our work aims to provide in- sight into how (pre-) training on large corpus is beneficial for compositional generalization. Be- sides data factors, the generalization behavior of models can also be studied from many different angles, including architecture (Li et al., 2019), reg- ularization (Yin et al., 2023), representation geom- etry (Montero et al., 2021; Ito et al., 2022; Murty et al., 2023), etc. All these directions are comple- mentary towards the same goal of understanding the mechanism of generalization.
62
125
250
Length
Exael-Match Acc
60
40
20
28
unique primitives per example
Figure 5: On SCAN* Jump split, simpler examples facilitate better compositional generalization.
tain 25% of the training examples. Since on these datasets, no longer have access the ground present results
truth example complexity, with prototype- based difficulty in Figure 4.
We observe similar but more complicated trends on larger-scale natural language datasets. First, we observe that difficult examples become impor- tant on these datasets. On all three datasets, us- ing only the easiest examples leads the worst results. We suspect that only using the simplest examples does not provide enough coverage for all the diverse linguistic phenomena in the dataset. However, the best performance is also not achieved with the most difficult examples, but at the medium level. Additionally, we notice that even in larger- scale natural language datasets, simpler examples are still important for compositional g generalization. Here we specifically focus on the trend difference between the results on SMCalFlow- (a composi- tional split), and the other two non-compositional split datasets. In Figure 4, we see SMCalFlow- CS achieves the best performance with the second easiest split, different from the other two datasets. Additionally, in Table 5, we show the performance by mixing the hardest examples with the casiest examples with 50%/50% ratio, and we also ob- serve unique trend on SMCalFlow-CS. Only on SMCalFlow-CS, the mixed performance outper- forms the performance by only using the difficult data, showing the advantage of having simpler ex-
Dataset influence generalization. Prior to the development of L.Mss, most dataset quality-related research focuses on dataset artifacts (Gururangan et al., 2018), or label quality (Maynez et al., 2020;

---

Pavlick and Kwiatkowski, 2019). After the recent success of LLMs, number of works pointed out that large-scale datasets can be a major contribut- ing factor to the success. Chan et al. (2022) shows that the scale of datasets is as important as the scale of the model. Hoffmman et al. (2022) show how example distribution can significantly impact the emergence of in-context few-shot learning. Her- nandez et al. (2022) notice that even small portion of repeated data can have significant impact on the generalization performance.
small datasets, data augmentation methods increas- ing the size and diversity may help the model's compositional generalization performance.
8 Conclusion
We empirically study how dataset factors influ- ence compositional generalization. We show that increased dataset complexity facilitates composi- tional generalization, with dataset size and pattern complexity two important factors behind the gain. We also provide analysis of example dif- ficulty and discuss the implication of our work on pertaining and fine-tuning models.
7 Discussion
Advantage of large-scale pretraining. In this work, we have demonstrated how different data factors (e.g., scale, diversity, example difficulty. etc.) can substantially improve the model's ability to generalize compositionally. While our experi- ments are done on a smaller scale, these results also hint at why models pretrained on larger datasets show better generalization ability. We argue that do- ing large-scale pretraining implicitly satisfies many beneficial data factors studied in this work. During pretraining, models are exposed to large set of different structures and primitives. Additionally, pretraining datasets have large size that prevents frequent repetition and contains a diverse set of simple and difficult examples. All these conditions incentives the emergence of compositional general- ization ability.
Limitations
The analyses in this study focused on relatively small-scale Transformer seq2seq models. While such architectural choice has been shown to be ef- fective compositional generalization datasets when trained from scratch (Csordás et al., 2021), very large models may demonstrate different be- haviors due to their emergence abilities (Wei al., 2022). The data factor analysis is not meant be a complete investigation about all possible data fac- tors. In this paper, we focus on scale, difficulty, and data complexity, which are all shown to be impor- tant factors for compositional generalization. How- ever, there are other factors also being important for compositional generalization. The specific eval- uation setup of compositional generalization also varies in different works. In this work, we focus on synthetic diagnostic datasets ensuring sufficient su- pervision and larger-scale natural language datasets with minimal manual control of the data distribu- tion. Additionally, compositional generalization challenges include both lexicon-level generaliza- tion and structure-leve generalization. This work mainly focuses lexicon-level generalization and relatively simple structure-level generalization in our length-related and SMCalFlow- experi- ments. We assume these results will also generalize to more complicated structure-leve generalization problems. For example, some more challenging se- tups may involve splits maximizing the compound divergence (MCD) (Keysers et al., 2020). MCD complexity does not naturally change when collect- ing larger datasets. To test related hypothesis, one may check if increasing the number of unique local structures improves performance. However, such dataset modification is non- non-trivial and we leave the exploration as future work.
How about fine-tuning models? In our main ex- periments, we choose not use pretrained models as they already possess substantial amount of knowledge, which makes it very difficult to evalu- ate how models initially acquire compositionality. Nonetheless, as fine-tuning models are of great practical importance, here we discuss related is- sues about fine-tuning models and summarize our preliminary findings with T5 models (more details are in Appendix H.2). The behaviors of fine-tume fine-tuning models are slightly different. We notice that the size and the diversity of the dataset can still be important as repetitive training on similar exam- ples (e.g., with the same primitive) will cause sub- stantial overfitting and hurt the pretrained model. However, pretrained models already have good understanding of basic language compositionality, they may need less assistance from the data, so maintaining large number of different primitives or simple examples may be less important. Over- all, our results suggest that when fine-tuning on

---

Acknowledgements
Stephanic C. Y. Chan, Adam Santoro, Andrew K.
Lampinen, Jane X. Wang. Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill, 2022. Data distributional properties drive emergent in- context learning in transformers.
We thank Dipanjan Das, Colin Raffel, and the re- viewers for their helpful comments. This work was supported by ONR Grant 400014-11-1-2871. NSF-CAREER Award 1846185, and DARPA MCS Grant 1666001-19-2-4331 and an Apple PhD Fel- lowship. The views are those of the authors and not of the funding agency.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. ArXiv preprint. abs/2204.02311.
References
Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. 2021. Meta-learning to compositionally gen- cralize. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 3322-3335, Online. Association for Computa- tional Linguisties.
Ekin Akyürek, Afra Feyza Akyürek, and Jacob An- dreas. 2021. Learning to recombine and resample data for compositional generalization. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Ekin Akyurek and Jacob Andreas. 2023. LexSym: Com- positionality as lexical symmetry. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 639-657, Toronto, Canada. Association for Computational Linguistics.
Róbert Csordás, Kazuki Irie, and Juergen Schmidhuber. 2021. The devil in the detail: Simple tricks im- prove systematic generalization of transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 619- 634, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jacob Andreas. 2020. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7556-7566, Online. Association for Computational Linguistics.
Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Hunicke-Smith, David Pallett, Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994, Expanding the scope of the ATIS task: The ATIS-3 corpus. In Human Language Tech- nology: Proceedings of a Workshop held at Plains- boro, New Jersey, March 8-11, 1994.
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy. Theo Lanman, Percy Liang, Christopher H. Lin, Nya Lintsbakh, Andy Mc- Govern, Aleksandr Nisnevich, Adam Pauls, Dmitrijj Petters, Brent Read, Dan Roth, Subhro Roy. Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, An- drei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020. Task-oriented dialogue as dataflow synthesis. Trans- actions of the Association for Computational Linguis- tics, 8:556-571.
Verna Dankers, Elia Bruni, and Dieuwke Hupkes. 2022. The paradox of the compositionality of natural lan- guage: A neural machine translation case study. In Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 4155--1755 Dublin, Ireland. As- sociation for Computational I Linguistics.
Roberto Dessi and Marco Baroni. 2019, CNNs found to jump around more skillfully than RNNs: Com- positional generalization in seq2seq convolutional networks. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 3919-3923, Florence, Italy. Association for Computational Linguistics.
Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. 2018 Jump to better conclusions: SCAN both left and right. In Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NLP, pages 47-55, Brussels, Belgium. As- sociation for Computational Linguistics.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek. Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2023. Compositional semantic parsing with large language models. In The Eleventh International Conference on Learning Representations.
Philémon Brakel and Stefan Frank. 2009. Strong sys- tematicity in sentence processing by simple recurrent networks. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 31.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text- to-SQL. evaluation methodology. In Proceedings

---

of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 351 351-360, Melbourne, Australia, Association for Computational Linguistics.
Takuya Ito, Tim Klinger, Doug Schultz, John Murray, Michael Cole, and Mattia Rigotti. 2022. Composi- tional generalization through abstract representations in human and artificial neural networks. Advances in Neural Information Processing Systems, 35:32225- 32239.
Daniel Furrer, Marc van Zee, Nathan Scales, and Nathamael Schärli. 2020. Compositional generaliza- tion in semantic parsing: Pre-training vs. specialized architectures. ArXiv preprint, abs/2007.088 0089000
Yichen Jiang and Mohit Bansal. 2021. Inducing trans- former's compositional generalization ability via aux- iliary sequence prediction tasks. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6253-6265, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing. pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Yichen Jiang, Xiang Zhou, and Mohit Bansal. 2022. Mutual exclusivity training and primitive augmen- tation to induce compositionality. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11778-1 1793, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jonathan Gordon, David Lopez-Paz. Marco Baroni, and Diane Bouchacourt. 2020. Permutation equivariant models for compositional generalization in language. In 8th International Conference Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer. Sergii Kashubin. Nikola Momchev, Danila Sinopalnikov. Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Mare van Zee, and Olivier Bousquet. 2020. Measur- ing compositional generalization: A comprehensive method on realistic data. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volune 2 (Short Papers), pages 107- 112. New Orleans, Louisiana. Association for Computa- tional Linguisties.
Najoung Kim and Tal Linzen. 2020. COGS: A compo- sitional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Confer- ence on Empirical Methods Natural Language Processing (EMNLP), pages 9087-9105, Online. As- sociation for Computational Linguistics.
Danny Hemandezz Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatiell-Dodds. Tom Henighan, Tristan Hume, et al. 2022. Scaling laws and interpretabil- ity of learning from repeated data, ArXiv preprint, Abs/2205.10487
Yoon Kim. 2021. Ssquence-to-sequenee learning with latent neural grammars. In Advances in Neural In- formation Processing Systems 34: Anmual Confer- ence Neural Information Processing Systems 2021, NeurIPS 2021, December 6- 6-14, 2021, virtual, pages
Jonathan Herzig, Peter Shaw, Ming- Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021. Un- locking compositional generalization in pre-trained models using intermediate representations. ArXiv preprint, abs/2104.07448.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Jordan Hoffmann. Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai. Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George den Driessche, Bog- dan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. 2022. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Sys- tems.
Brenden M. Lake. 2019. Compositional generalization through meta sequence vances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9788-9798.
tooteetttto--oee to- so-sequence learning. In Ad-
Brenden M. Lake and Marco Baroni. 2018. General- ization without systematicity: On the compositional skills of soquencc-to-ccquence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan,
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Eliia Bruni. 2020. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Rexearch, 67:757-795

---

Stockholm, Sweden, July 10-15. 2018. volume 80 of Proceedings of Machine Learning Research, pages 2879-2888. PMLR.
generalization abilities of neural sequence models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 424-434, Dublin, Ireland. Association for Computational Linguisties.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880 Online, Association for Computa- tional Linguistics.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguis- tics, 21677-694.
BART: Denoising
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Scikit-lear:: Machine learning in Python. Journal of Machine Learning Research. 12:2825-22 2830.
Yuampeng Li. Liang Zhao, Jianyu Wang, and Joel Hest- ness. 2019. Compositional generalization for primi- tive substitutions. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4293-4302, Hong Kong, China. Association for Computational Linguistics.
P. J. Price. 1990. Evaluation of spoken language sys- tems: the ATIS domain. In Speech and Natural Lan- guage: Proceedings of Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSSMs to learn syntax- sensitive dependencies. Transactions of the Associa- tion for Computational Linguistics, 4:521-535.
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, and Kristina Toutanova. 2022. Evaluating the impact of model scale for compositional generalization in semantic parsing, In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 9157-9179, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
João Loula, Marco Baroni, and Brenden Lake, 2018. Rearranging the familiar: Testing compositional gen- cralization in recurrent networks. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: An- alyzing and Interpreting Neural Networks for NI.P, pages 108-114, 14, Brussels, Belgium. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-14000 140:667
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics.
Peter Shaw, Ming- Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generaliza- tion and natural language variation: Can semantic parsing approach handle both? In Proceedings of the 59th Anmual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Vol- ume 1: Long Papers), pages 922-938, Online. Asso- ciation for Computational Linguistics.
Milton Llera Montero, Casimir J. H. Ludwig, Rui Ponte
Costa, Gaurav Malhotra, and Jeffrey Bowers. 2021. The role of disentanglement in generalisation. In 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning. 2023. Characterizing intrin- sic compositionality in transformers with tree projec- tions. In The Eleventh International Conference on Learning Representations.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Short Papers), pages 464-468, New Or- lcans, Louisiana. Association for Computational Lin- guistics.
Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gard- ner, and Jonathan Berant. 2020. Improving compo- sitional generalization in semantic parsing. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 2482-2495, Online. Association for Computational Linguistics.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. 2022. Beyond neural scal- ing laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems. 15:19523-19553
Arkil Patel, Satwik Bhattamishra, Phil Blunsom, and Navin Goyal. 2022. Revisiting the compositional

---

Swabha Swayamdipta. Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Langwage Processing (EMNLP), pages 9275-9293, Online. Association for Computa- tional Linguistics.
John M Zelle and Raymond Mooney. 1996. Learning to parse database queries using inductive logic pro- gramming. In Proceedings of the national conference on artificial intelligence, pages 1050-1055.
Hao Zheng and Mirella Lapata. 2021. Compositional generalization via semantic tagging. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages 1022-1032, Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Dmitry Tsarkov, Tibor Tihon, Nathan Scales, Nikola Momchev, Danila Sinopalnikov, and Nathanael Schärli. 2021. *-cfq: Analyzing the scalability of machine learning on a compositional task. In Thirty-Fifth AAAI Conference on Artificial Intelli- gence, AAAI 2021, Thirty- Third Conference on In- novative Applications of Artificial Intelligence, IAAI 2021, The Fleventh Symposium on Educational Ad- vances in Artificial Intelligence. EAAI 2021, Virtual AAAI
Appendix
A Dataset Details
A.1 Natural Dataset Details
We use three English datasets with human-written queries: GeoQuery (Zelle and Mooney. 1996), ATIS (Price, 1990; Dahl et al., 1994) and SM- CalFlow (Andreas et al., 2020; Yin et al., 2021).
Event, February 2-9, 2021, pages Press.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. Transactions on Machine Learning Research.
GeoQuery (Zelle and Mooney, 1996) dataset contains 880 questions about US goography with the corresponding SQL queries or logical expres- sions of the question. Following Andreas (2020), we report performance on both the query split and the question split. The question split is the original split proposed in the dataset, while the query is more compositionally challenging split proposed in Finegan- al. (2018) ensuring no over- lapping logical forms between the splits. We also use the standard dev-test split for GeoQuery.
Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power finite precision RNNs for language recognition. In Pruceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Voluee Short Papers), pages 740-745, Melbourne, Australia. Association for Computational Linguistics.
Paul Rodriguez Janet Wiles 1998. Recurrent neural networks can learn to implement symbolsensitive counting. Advances in Neural Information Process- ing Systems, 10:87
ATIS (Price, 1990; Dahl et al., 1994) is larger- scale semantic parsing dataset with 3809 examples. Each example contains flight-related query as well as the corresponding SQL query. Due the nesting format difference between the train/dev set and the test set of ATIS, we simplify our experi- ment setting to report dev set results only for the ATIS dataset.
Francis CK Wong and William SY Wang. 2007. Gener- alisation towards combinatorial productivity in lan- guage acquisition by simple recurrent networks. In 2007 International Conference on Integration of Knowledge Intensive Multi-Agent Systems, pages 144, IEEE
Pengeheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, and Jacob Andreas. 2021. Compositional generalization for neural semantic parsing via span- level supervised attention. In Proceedings of the 202/ Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2810-2823. Online. Association for Computational Linguistics.
SMCalFlow (Andreas et al., 2020) is a large- scale dialog dataset. All the examples in the dataset are from natural conversations and they are anno- tated with an executable dataflow program. Yin et al. (2021) proposed a modified version of this dataset focusing on compositional generalization. We use the 32-shot version in Yin et al. (2021) and denote it SMCalFlow-CS in later experi- ments. As our experiments also aim to provide a fair comparison between the original version and the compositional version of SMCalFlow, we pre- process both versions using the same way as the preprocessing steps for the compositional version
Yongjing Yin. Jiali Zeng, Yafu Li, Fandong Meng, Jie Zhou, and Yue Zhang, 2023, Consistency regular- ization training for compositional generalization. In Proceedings of the ō1 Annual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Papers), pages 1208-130, Toronto, Canada. Association for Computational Linguistics.

---

described in Yin et al. (2021). Due to this prepro- cessing detail, our result on the non-compositional split of SMCalFlow may not be comparable to other works.
knowledge from pretraining, For the experiments in the main paper, we follow Csordás et al. (2021) to use layers in both encoder and decoder, and use relative positional embeddings (Shaw et al., 2018). In Appendix H, we provide additional results on models with other configurations. We set both the hidden size and the embedding size to 256, and the dimension of the feed-forward layer to 512. We use dropout rate of 0.1 and self-attention heads. For the optimizer, we use Adam (Kingma and Ba, 2015) and a batch size of 128 examples. We use the Noam Learning rate scheduling with a peak learning rate of 2.0, 50000 total steps, and 5000 warmup steps. We use a beam size of 5 to get all the results in our experiments.
A.2 Dataset Construction Details for SCAN* Length Generalization Experiment
SCAN* is extended version of SCAN created in this work that we can analyze how dataset complexity (especially length-related complexity) influences the model's generalization ability. This is an essential design choice as the complexity of the original SCAN is bounded. For all SCAN* related experiments, we apply two changes: first, every dataset will contain 200 different primitives; second, every sentence can have an unlimited num- ber of and and after as long the total length is within the limit for the length generalization exper- iments. To avoid ambiguity, we assign different orders of priority to and and after, with after hav- ing the higher priority and will be executed first. Additionally, we make two minor simplifications to the SCAN grammar that do not influence the trends substantially. We removed the verb turn and we unify the grammar for around and opposite To generate the data for the length generalization experiment in Sec. 3, and the length ablation ex- periment in Sec. 5, we try to ensure similar dataset statistics across datasets with different lengths as much as possible. To achieve this, we always first generate the dataset with the maximum length. Then, to create all the other shorter experiments, we repeatedly create a half-length truncated ver- sion of the longer original dataset. Specifically, for every example with length in the longer dataset, we first split each example by its conjunctions (and or after), then we keep the most amount of the split parts so that the total length under the half-length 1/2 of the original example. For our experiments, we use the length of the input as the length measure.
B.2 Implementation of Difficulty Metries
For the prototype-based difficulty metric, we follow the process in Sorscher et al. (2022). For every ex- ample, we feed the input side to the SimCSE (Gao et al., 2021) model to get an example embedding. We choose to use the input side instead of the out- put side to compute the embedding since the inputs are natural language so they are more suitable for off-the-sae the-shhllt models like SimCSE. Then, we run k-means to get the distance between each example embedding and its corresponding cluster centroid. Our k-means implementation is from (Pedregosa et al., 2011). 1). We set k to 100 in our experiments. and use the best clustering result from 10 different initializations.
For the learning learning- -based difficulty metric. we follow our standard setup to evaluate our model on the training set for every 500 steps. We then log the performance of each example. Then for every example, we find the earliest step when the model has predicted the example correctly for 10 consecutive steps. The earlier this step is, the sim- pler this example is. If the model can never predict the example correctly, this step number is set the final step. In our experiment, we compute this metric on multiple random seeds. Therefore, for every example, the final step number is the average over all the different seeds.
B Implementation Details
B.1 Models and Training Details
In this study, we focus on seq2seq Transformers they are the most prevalent choices for most compositional generalization (and more broadly, semantic parsing) datasets. For our main experi- ments, we focus on models trained from scratch so that our analysis is not influenced by the existing
C Data Augmentation Details
C.1 Data Augmentation Details of SCAN Jump and Around Right
We first follow the exact same setup in Jiang et al. (2022) to conduct data augmentation experiments on the SCAN Jump and Around Right split to create
10 In Appendix F. we show this simplification does not in- fluence the trends shown in this paper

---

training sets with different numbers of primitives. The augmentation follows a two-stage procedure, which we explain here.
get all the geography entities in the dataset by pars- ing the output predictions. Then, for each entity, we create multiple copies (e.g., 19 new copies for the x20 experiments) for the entity. Then, for every example, we identify the entities contained in every examplc, and then augment new examples contain- ing the new copies of these entities. We augment the same amount of new examples for every exam- ple. The final size of the x20 augmented dataset will be 20 times of the original dataset.
Building a Dictionary. In the first stage, we use a dataset-agmostic, aale-based algorithm to build a dictionary that maps certain input tokens to their output forms (e.g., "look LOOK" and "jump >> JUMP"). Given the source vocabulary as V and the target vocabulary as W, the algorithm iterates through the training set to identify pairs (v, w), E V, w E W such that the presence of in the input is both necessary and sufficient for the presence of w in the output.
C.3 AugZero Augmentation Details
Below we describe Augmentation with Zero Prior Knowledge (AugZero). The general idea is just to satisfy our hypotheses by providing multiple copics of the training set using different copies of the vocabulary. Interestingly, this process quires zero knowledge about the actual data. Specif- ically, assume the original vocabulary is V1 [w] Im 1- where w ... wit are all the m possi- ble tokens. AugZero increases the vocabulary by k times. The augmented vocabulary will be VAVV = V u U U... U Uv =[w) | where each w/ is a corresponding new token for w/. Given a tokenized example with length / in the original dataset: x wh, w Way..... w.y where .1,.... uy are the token indexes V1, In AugZero, we create k- additional copies of this example, from w ...... w to wi t...... - See Figure 3 for an illustration. One crucial difference between AugZero and the data augmentation described in Sec. 3 is that AugZero no longer distinguishes the primitive and non-primitive tokens, hence requiring zero knowledge about the actual task.
suff(1; (v, w) V(z,g), 2) (w (1) ness(v, u, w) = v(r,y), (w E v) - (w E
This algorithm ignores those functional words (e.g.. "around"' and "twice") that only decide the syntactic structure of the outputs but cannot be translated to specific target token.
Mutating Primitives. In stage two, we iterate through every example in the training set and ran- domly select some primitive pairs that exist in the previously built lexicon for mutation. In mutating them, we simply add suffix to their source and target forms. Given an original example "walk left twice", we select primitive "walk" and mutate it new example "walk1 left twice + TL WALK1 TL WALK1". For our experiments in Sec. 3, we create three different augmented training sets (2x, 20x, 200x) with increasing complexities. Specifi- cally, for a training example in the SCAN Jump or Around Right, we first identify all primitive pairs in the example. Then, in Kx augmentation (K 2,0,000,,, for each primitive pair (e.g.. "(walk, WALK)"), randomly replace the source token in the input with one of its K +11 mutated form (walk, walkl, walk2, walk&() and the target token in the output with the corresponding form. We repeat this process 2K times or until we collect K distinct augmented examples.
Other variants of AugZero. In AugZero, we se- lect a different set of vocabulary for each example. For example, the original "walk left and jump" may become "walk2 left2 and2 jump2" If we combine this idea and syntax induction methods, we can get additional augmentation examples such as "walk left2 and2 jump2". This can further improve the performance (as shown by the prim2primX results using very similar ideas). However, doing such aug- mentation requires inducing the syntax mapping beforehand, which is not trivial on natural language datascts and loses the advantage of simplicity. Ad- ditionally, we explored another variant that maps each original token in the dataset randomly into all the corresponding words in the augmented vo- cabulary. This makes the one-to-one map "walk b WALK" becomes a many-to-mane map "I "(walk,
C.2
Data Augmentation Details for GeoQuery
Here we describe the detailed data augmentation process for the GeoQuery dataset as used in Sec. 3 and Sec. 4. As one crucial part of the questions in GeoQuery are geography entities (e.g., Oregon, Springfield, etc.), our augmentation focuses on in- creasing the diversity of these entities similar to the primitive- based augmentation method in Patel et al. (2022) and Jiang et al. (2022). Specifically, we first

---

Dataset Size
Repetition
GeoQuery (query)
GeoQuery (question)
baseline baseline bascline
29.67+5.1
60,79+1.0
Example
8.68=1117
56.63++2.0
Primitive
13.87+270
36.77+1.0
20x
32.97+2.00
62.44++1.00
20x
Example Primitive
30.00
62.65++1.55
20x
6.48
10.05
Table 6: The effect of examples repetition on different datasets with SQL outputs. We use SQL outputs for the GeoQuery dataset in this table.
Tobber
GeoQuery (query)
GeoQuery (question)
huseline
29.67
60.79
Unica
Angmentation
26.05
61.72g>
Hell une
DUUNWNG WALL
20x Augmentation
32.97
62.44g1
Augmentarion 32.64g
61.08
Table Datasets with increased complexity via data augmentation are easier for compositional generaliza- tion. We use SQL. outputs for the GeoQuery dataset this table.
Diice
vib
PAUX
ITUEN Juan
Figure 6: Two different ways to achieve low training losses: surface memorization (left) and compositional understanding (right).
E Additional Length Generalization Results
In Figure 7, we show length generalization results without the grammar unification of around and opposite. We can see the trend is very similar to Figure 2. As the maximum length becomes larger in the dataset, generalization performance becomes better. In our preliminary experiments, have verified that our other findings on SCAN* are also robust against minor grammar changes.
Exact-Match Acc
50
250
Length (L)
Figure Different generalization performances with- out the grammar simplification of around and opposite. Same to Figure 2, the challenge is always to train on examples with length 0 < < L and test on examples with length L < < 2L.
F Additional GeoQuery Results
In the main paper, due to space constraints, we re- port on the GeoQuery dataset with logical forms as output. In Table and Table 7, we show the cor- responding results using SQL as the output. Both tables demonstrate similar trends to the results in the main paper.
walkl, walk2, ...) -> (WALK, WALK1, WALK2, ...."" but do not show further improvements.
D Illustration for the Two Model Behaviors
G Additional Example Difficulty Results
G.1 Examples of Prototype-Based Difficulty
In Figure 6, we provide an illustration for the two model behaviors mentioned in Sec. 4. Both be- haviors can lead to good performance on the train- ing set (as shown in the upper half of the figure). However, given new example with novel com- bination of seen structures and primitives, only models with correct compositional understanding can provide the correct prediction (as shown in the lower half of the figure).
In Table 8, we show examples of simple and diffi- cult data in the SMCalFlow-G dataset according to the prototype-baser difficulty. In the table, the simple examples simplest 25% of the dataset, while difficult exam- ples come from the hardest 25%. We can see that the simple examples usually refer to common gen- eral instructions, while difficult examples tend to be more complex in the language and specify more
randomly sampled from the

---

Simple Examples
Difficult Examples
Can you create Meeting for Saturday 1 00 pm
had meeting with Jesse last week
Schedule a lunch after the meeting on Thursday.
need my free day on April 7, to be turned to my All out gallery opening.
create a new appointment tomorrow
Add a beer festival in Denver to be for all weekend next week.
Table 8: Simple and difficult data examples in the SMCalFlow-CSS dataset according to the prototype-based difficulty metric. this table, the simple examples are randomly sampled from the simplest 25% the dataset, while difficult examples from the hardest 25% of the dataset.
the model could reach perfect training accuracy on many of the datasets in this work, we choose to use the dedicated metric of how soon the model predicts the example correctly instead of the plain accuracy, that amples. Specifically, during training, we evaluate the model on the entire training set for every check- point. Then for every example, we log the earliest step when the model has predicted the example cor- rectly for 10 straight checkpoints. See Appendix B for details.
differentiate between
SMCx
SVKcFFow-C
We report additional results of our difficulty- based sub-sampling results in Sec. 5. In Sec 5 in the main paper, we use prototype-based diffi- culty metrics to conduct the sub-sampling experi- ments. We make this design choice as we use the learning-baad based difficulty metric, then the models perform extremely badly on the most difficult split, which makes it unsuitable for our experiments. The corresponding results are shown in Figure 8.
Diliadlyy
Figure 8: Results ATIS and SMCalFlow with train- ing sets of different difficulties with learning-basse based dif- ficulty.
details (e.g., ., the name Jesse, the place Denver, etc.). Additionally, we want to point out that while this prototype-based difficulty measure alters the general complexity of the data, it does not change the compositional difficulty of the SMCalFlow-CC dataset. This is because the dataset of SMCalFlow- CS constructed to test the ability to combine multiple instructions in the training set under few- shot setting (Yin et al., 2021), and the difficulty is controlled by altering the number of few-shot exam- ples. As we always keep the few-shot examples the same in all these training sets and only change the difficulty of the remaining single-skill examples, the clustering step will not introduce other con- founding factors to the compositional challenge.
H Results with Other Model Configurations
H.1 Data Augmentation Results with Larger
Transformers
For the main results in the main paper, we use 3- layer Transformers as it is suggested as the best choice for many popular compositional generaliza- tion datasets (Csordás al., 2021). In Table 9, we report the data augmentation number for larger 6-layer model. We see similar trends on all the datasets, confirming that the advantage brought by increased data complexity is not sensitive to the underlying model scale.
G.2 Learning-Baad Based Difficulty
H.2 Results with Pre-trained Transformers
We can use an empirical metric based how soon the model predicts an example correctly during training. This is similar to the popular correctness- based metric (Swayamdipta et al., 2020). Since
As the main motivation of this work is examine how data factors influence model generalization performance, we mainly experiment with models trained from scratch so that we can have clean con-

---

Jamp
Around Right
GeoQuery (query)
GeoQuery (question)
baseline
4.14+4m1
52.42+2141
37.63+s.57
58,42+156
2x Augmentation
31.79
79.2647m
37.42+1.55
60.57=296
20x Augmentation
92.13+45.6
77.50+2234
41.20+268
66.95=1.17
200x Augmentation
99.90
99.55.0085
39.6845.88
12.20
Table Datasets with increased complexity via data augmentation are casier for compositional generalization. We use 6-layer Transformers and logic form outputs for the GeoQuery dataset in this table.
Dataset Size
Repetition
GeoQuery (query)
GenQuery (question)
haseline
67.74
80.05
20x
10000
79.6910.20
20x
Example Primitive
65.41 51.0811.27
81.72
20x
61.651-1.6
Table 10: The effect of examples repetition on different datasets for pre-trained Transformers.
trol over all the influencing factors and avoid the interference of pretraining data and testing data. Nonetheless, when we actually deploy models in practical applications, the more common choice is use pretrained models (c.g.. T5 (Raffel al., 2020), BART (Lewis al., 2020), etc.). Below we reproduce some our main results in this pa- per with T5-base models and summarize the take- aways. For the the same setting described in Sec. 2.2. The only change is that we use learning rate of 0.001 and linear learning rate scheduler. For the T5 exper- iments, we report mean and standard deviations from 3 runs.
basic understanding of natural language. For ex- ample, by pretraining a large corpus, it already knows that run and jump are both verbs; Texas and California both location entities, and these words in the same category should be treated sim- ilarly. Therefore, it gets a 'jump-star' on these compositional generalization datasets, and may not need large number of unique primitives to infer the equivalence of run and jump. As a result, we do not observe consistent improvement by using data augmentation in Table 10.
we mostly follow
Preventing repetition in data is still important. In Table 10, we reproduce the experiments in Ta- ble where test the effect of repeating examples during the training process. From the results, can see that similar to the results in the main pa- per. When the model repetitively trained on randomly sampled small set (i.c., the 3rd row in the table), we can only see a small drop in the query split. However, the damage is more substantial when the repeated subset contains few primitives, showing drop from 72.04 to 51.08 the query split and drop from 79.69 61.65 on the ques- tion split. To summarize, example frequency is still very important during finetuning. Highly repetitive examples can cause substantial damage to the gen- eralization performance, which may also be linked with overfitting on the small subset.
Pretrained Trausformers need less assistance for simple cases. One crucial difference between the pretrained models and the models trained from scratch is that pretrained models already have a


Reviews:
[{"review_id": "75InLwAgqR", "reviewer": "Reviewer_9qDf", "summary": "", "questions": "- If preventing memorization improves compositional generalization, then regularization or other techniques to prevent memorization (overfitting) should also help. Is there any indication that this might be the case, or is that a completely wrong supposition?\n- The idea behind the AugZero data augmentation procedure is rather neat. It seems that there might be an additional \"combinatorial factor\" that could be exploited in terms of increasing the dataset by mixing tokens across tokenizers. For instance, the sequence \"walk left and jump\" is augmented by AugZero to \"walk2 left2 and2 jump2\" using an additional tokenizer. But in principle the sequence \"walk left2 and2 jump2\" obtained by using \"walk\" from the first tokenizer and the remaining tokens from the second one could also be used as a data augmented sample. Would that indeed be a valid augmentation? And if so, would there be any advantage to exploiting the resulting combinatorial factor in terms of combining tokenizers and tokens replacement?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and we are glad that you appreciate our study as well-structured,  you like our novel AugZero data augmentation method, as well as our findings on the relationship between data factors and compositional generalization.\n\n\n**On datasets used in this work**\n\nWe use SCAN as one of the main datasets used in this work as its well-controlled syntax makes it easier to design clean experiments supporting our hypotheses. It is also the major dataset used in Patel et al., 2022 and Jiang et al., 2022, the two papers motivating this study. In addition to the SCAN dataset, to demonstrate the generalizability of our findings, we have also tested our findings on three other natural language datasets: GeoQuery, SMCalFlow, and ATIS.\n\n\n**Other non-data factors that affect compositional generalization**\n\nWe acknowledge that compositional generalization can be affected by other non-data factors, including model capacity [1] and model architecture [2]. Those other factors are all crucial but are also popular topics studied by previous work (see LINE 047-049, LINE 568-584, and also [1,2]). In this work, we decide to conduct a focused, systematic study on various under-explored data factors that affect compositionality using a commonly used architecture (seq2seq transformer) and a fixed model capacity. \n\n**Representation geometry**\n\nWe thank the reviewers for the suggestion on understanding compositionality in terms of representation learning. This is definitely an important direction as shown in past works [4]. However, those works have different motivations from our work. Geometry-based study helps us better understand *what* is the composition learned inside the models, while our work focuses more on *why* models learn better composition on more complex data. These two directions are complementary to each other. To demonstrate this, below we provide a simple example of how we can explore certain geometry of learned representations to analyze why the model trained on more complex datasets achieves stronger generalization:\n\nWe calculate the cosine-similarity between the embedding of “jump” and “walk” learned from the original SCAN jump split or the “prim2primX” augmented data. When the model is trained on the original SCAN jump split, the similarity between the embeddings of “jump” and “walk” is very low, at 0.34. When the model is trained on the prim2primX-augmented SCAN jump, the similarity increases to 0.88. This suggests that when the model is trained on a dataset with higher pattern complexity, the model can better capture the syntactic similarities between different primitives. This leads to stronger primitive-level compositional generalization because when the model has learned the similarity between “jump” and “walk”, it is more likely to understand an unseen sentence “jump twice” by inferring from the seen sentence “walk twice”. In our final version, we will provide a discussion about the connection between different directions on the interpretability of compositionality.\n\n\n**Response to Question 1**\n\nThere are indeed works that show various regularization techniques help compositional generalization. Simple L2 regularization is used in [2] and more advanced regularizations are proposed in [3]. We’d also like to point out that overfitting is a broad umbrella term that covers many different phenomena. Even with proper regularization, most models achieve perfect accuracy and “overfit” on the training set, so figuring out which regularization works is not trivial. Furthermore, recent advanced LLMs (e.g., GPT4) demonstrate significantly stronger performance on datasets like SCAN. These LLMs can automatically acquire compositional behavior without explicitly designed regularization. Therefore, we are interested in what aspects of the LLM training procedure provide effective implicit regularization. Our study provides related insights (i.e., the benefit of training on complex data) in this direction.\n\n**Response to Question 2**\n\nThanks for the suggestion! \"Walk left2 and2 jump2\" is indeed a valid augmentation. In fact, the “prim2primX” data augmentation used on SCAN in Sec. 3 and 4 includes very similar examples (See Appendix C.1 for details). The only difference is that we do not augment functional words like conjunctions (“and2”). As shown in Table 2 (“+ Both”). From Table 2, we can see that prim2primX outperforms AugZero, partly due to having these additional \"combinatorial factor\" results. However, doing such augmentation requires inducing the “walk -> WALK” mapping beforehand, which is not trivial on natural language datasets (even for simple ones like GeoQuery). On the contrary, AugZero does not require any syntax induction and has the advantage of simplicity.  Additionally, we also explored another related idea that randomly using walk, walk1, walk2, … in all the examples, which make the one-to-one map “walk->WALK” becoming a many-to-many map “walk, walk1, walk2, …->WALK, WALK1, WALK2, …”, but do not observe further improvements. We will clarify these variants in our final version.\n\n[1] Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing. EMNLP 2022\n[2] Compositional Generalization for Primitive Substitutions. EMNLP 2019\n[3] Consistency Regularization Training for Compositional Generalization. ACL 2023\n[4] Characterizing Intrinsic Compositionality in Transformers with Tree Projections. ICLR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper identifies a set of factors characterizing a dataset that are positively correlated with compositional generalization performance by influencing a trade off between memorization and generalization.\n- the paper proposes a novel \"zero knowledge\" data augmentation procedure motivated by this finding.\n- The paper is well structured as it is presented as an hypothesis-driven empirical study.", "reasons_to_reject": "- The paper has an understandable but rather idiosyncratic focus on a specific instantiation of compositional generalization embodied by the SCAN dataset where the main results are obtained\n- The paper characterizes some of the data factors that underlie compositional generalization in a strictly phenomenological fashion, without relating them to the underlying mechanism in terms of statistical learning theoretic quantities. In particular, the presumed \"memorization\" factor being tweaked by the dataset complexity seems clearly related to overfitting, suggesting that the crossover from memorization to compositional generalization is not merely dataset-dependent, but also has to do with the model's capacity and the inductive bias deriving from the model architecture. This type of consideration are glaringly missing in the paper. \n- Another aspect that is left oddly unexplored by the paper are mechanisms underlying compositional generalization that are understood in terms of representation learning. In the end the algorithm being analyzed is a machine learning model meaning that, mechanistically, compositional generalization has to come about due to some geometric property of the embedding representing the sequences constituting the task. Moreover, these mechanisms can be potentially directly investigated by examining the model's activation. However, also in this regard the paper disregards this opportunity to access, examine and mechanistically understand the phenomenon of compositional generalization, deciding to limit the investigation to a phenomenological investigation.", "questions_to_authors": "- If preventing memorization improves compositional generalization, then regularization or other techniques to prevent memorization (overfitting) should also help. Is there any indication that this might be the case, or is that a completely wrong supposition?\n- The idea behind the AugZero data augmentation procedure is rather neat. It seems that there might be an additional \"combinatorial factor\" that could be exploited in terms of increasing the dataset by mixing tokens across tokenizers. For instance, the sequence \"walk left and jump\" is augmented by AugZero to \"walk2 left2 and2 jump2\" using an additional tokenizer. But in principle the sequence \"walk left2 and2 jump2\" obtained by using \"walk\" from the first tokenizer and the remaining tokens from the second one could also be used as a data augmented sample. Would that indeed be a valid augmentation? And if so, would there be any advantage to exploiting the resulting combinatorial factor in terms of combining tokenizers and tokens replacement?", "paper_topic_and_main_contributions": "The paper provides an empirical study of compositional generalization in seq2seq models, trying to tease apart the factors that promote a model's ability to generalize compositionally. This question is timely as it is inserted in the broader research goal of elucidating the training factors that allow LLMs pretrained on large datasets to acquire some of their reasoning and generalization abilities, even though they were not explicitly trained with the intent of displaying those abilities.\nThe results are interesting as they identify dimensions characterizing dataset complexity that contribute to promoting generalization performance in compositional seq2seq tasks that are actionable to the point of suggesting a data augmentation procedure that is also demonstrated in the paper.\n\n### Post-review discussion changes\nFollowing the authors' rebuttal I increased my score:\n* \"Soundness\" was increased to \"good\"\n* \"Excitement\" was increased to \"strong\"", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TJUPTNgDlN", "reviewer": "Reviewer_P8xF", "summary": "", "questions": "A. Have the authors considered and explored other measurements of data complexity, e.g., compound divergence (Keysers et al., Shaw et al.), local structures (Bogin et al.)? If so, how do they influence generalization performance?\n\nB. Have the authors studied other compositional splits for SCAN (e.g., MCD from Keysers et al.) and GeoQuery (e.g., TMCD from Shaw et al.)? And other semantic parsing datasets that evaluate compositional generalization, e.g., CFQ (see a similar study from Tsarkov et al.)?\n\nC. Can the authors clarify the differences between “improving compositional understanding” and “preventing surface memorization”. And why do we need to distinguish them and explicitly characterize these two model behaviors? Aren’t these two factors always correlated? The discussions in L288-L310 could benefit from further clarification.\n\nD. SCAN* creation details: L1074-1078, is there any specific reason for this design? Based on my experience, one of the most significant challenges of SCAN is separating the “around” and “opposite”. \n\nE. Can the authors clarify more how the section starting from L378 supports the two proposed hypotheses in section 4.2? Theoretically, it’s still possible to have large datasets with example repetition or primitive repetition curriculum (bottom of Table 3)? The experimental setup used here does not seem to mimic the standard training practice. I wonder how much of the conclusion drawn here can directly support the claim that reducing frequently recurring examples improves compositional generalization.\n\nF. Is there any qualitative analysis of the difficulty metric, especially for the prototype-based metric in the context of a natural language dataset? How effective is it at clustering examples? I imagine that for some compositional splits, such as SMCalFlow-CS, the \"hard\" examples might be quite similar in the embedding space. For instance, training examples like \"who is my manager\" and \"schedule a meeting with Amy,\" and a test example like \"schedule a meeting with my manager,\" could be very close in the embedding space. Therefore, this measure might not be a good indicator of example-level difficulty. Other measurements, such as those by Bogin et al. (see below), might provide more informative signals.\n\nG. Figure 4: IIUC, difficulty here refers to the L2 distance to the centroid. If so, why does it appear to always be an integer?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and we are glad you appreciate our findings in isolating different data factors and how we control them in experiments. \n\n**Response to R1 and Questions A, B** \n\nThe motivation of our work is to study how larger and more complicated datasets make composition easier to learn. Therefore, we conduct our experiments by precisely controlling the dataset complexities (e.g., number of primitives, length) in order to find how it affects the resulting model’s compositional generalization. We are able to control the number of unique primitives following the previous prim2primX data augmentation and control the length by simply cutting off the training set at different lengths threshold. The complexity of MCD-based splits, however, does not change significantly when the dataset becomes larger. One way to test a related hypothesis connected to our work is to check if changing the number of unique local structures can improve performance on MCD-based datasets. However, controlling the number of unique local structures is very difficult and we are not aware of any existing work that managed to automatically create new structures. Therefore, we mainly focus on primitive generalization and length extrapolation in this work (with more realistic datasets on example difficulty experiments), and leave the exploration on more complex structures for future work.\n\n**Response to Question C**\n\nIndeed, “improving compositional understanding” and “preventing surface memorization” are always correlated. As we stated in LINE 288-310, the model’s behavior is a complicated combination of these two competing behaviors. A method that improves compositional generalization might (1) directly encourage compositionality through certain inductive bias (e.g., a neural-symbolic model), (2) or it does not provide such an inductive bias but instead punishes the model for surface memorization. Increasing pattern and scale complexity belong to the second category as they increase the cost of memorization without increasing the cost of generalization. We believe this is an important distinction to make as we are trying to understand the source of improvement in Sec. 4.2 and Sec. 4.3.\n\n**Response to Question D**\n\nThank you for pointing out this issue. We apply this simplification by accident. However, this simplification will not influence our findings in Table 1 and Table 2. All our models in the comparison are trained and tested using datasets with the same syntax. Additionally, in all our experiments related to SCAN* around right or the original SCAN around right (Table 1 and Table 2), we never noticed any difficulty understanding the around and opposite parts. \n\n**Response to Question E**\n\nHere we explain the function of the paragraph starting at LINE 378 in Sec. 4. The overall target of Sec. 4 is to explain the benefit of increased dataset complexity. One of the hypotheses we proposed in Sec. 4 is that increasing dataset complexity can help compositional generalization by reducing frequently recurring examples (LINE 337-339). Note that we do not claim that increasing dataset complexity will always lead to reducing frequently recurring examples (as shown by our examples at the bottom of Table 3), however, this is mostly true in cases where the examples are collected naturally. This hypothesis is jointly supported by two sets of observations: (1) naturally increasing the dataset size helps generalization, and (2) naturally increasing the dataset size while still adding highly recurring examples during training will lead to worse results. The paragraph starting at LINE 378 shows observation (2) while Section 3 shows observation (1). We will further clarify this in the final version.\n\n\n**Response to R2 and Question F**\n\nAs we pointed out in the paper, we designed the prototype-based difficulty metric based on [1], where they showed that in ImageNet classification, pruning the data using the prototype-based metric matches or exceeds the performance of pruning with the best supervised difficulty metric, memorization. The memorization-based difficulty metric is very costly since it is computed as how much the probability of predicting the correct label for the example increases when it is present in the training set relative to when it is absent [1]. The fact that the prototype-based metric matches its performance in data pruning shows it is a reliable metric. On the SMCalFlow data, the clustering results are also reasonable. We see more standard queries like “Put on a meeting on thursday” in the easiest split, as there are many similar queries just changing their day, while in the hardest split, queries contain more specific information like “Can you please add an obligatory family visit on my calendar on wednesday next week at 6pm”. We will expand this discussion in the final version, provide clustering result examples to better demonstrate its validity, and mention other related complexity metrics.\n\nMore specifically for our experiments on SMCalFlow-CS, clustering metrics will be reliable and will not bring confounding problems as in Question F. This is because the dataset of SMCalFlow-CS is constructed to test the ability to combine multiple instructions in the training set (See the SMCalFlow Compositional Skills paragraph in Yin et al., 2021) under a few-shot setting. As we always keep the few-shot examples the same in all these training sets and only change the difficulty of the remaining single-skill examples, the clustering step will not introduce other confounding factors to the compositional challenge. We will clarify this point in the final version. \n\n\n**Response to Question G**\n\nIn Figure 4, the x-axis actually shows the quantile of the difficult examples based on the distance to the centroids. Therefore, “0” means examples in the easiest quantile while “3” means examples in the most difficult quantile. We will change the x-axis title to clarify this.\n\n[1] Beyond neural scaling laws: beating power law scaling via data pruning. NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper provides diagnostic studies on data factors that influence compositional generalization, which could be helpful for understanding the generalization performance of models by isolating any factors from pre-training.\n- Experiments and analysis are well-designed and motivated from prior work. The data and experiments are well controlled.", "reasons_to_reject": "- The main experiments on data scale and pattern complexity only use relatively easier splits from SCAN and GeoQuery. Many other more challenging compositional splits have been proposed but were not considered: for SCAN, MCD splits (Keysers et al., Shaw et al.); for GeoQuery, TMCD splits (Shaw et al.). I'm not sure if this conclusion will generalize to other datasets or even other splits, especially since the pattern complexity is only based on the number of primitives without taking into account other complex structures.\n- While the experiments on example difficulty consider more realistic datasets, the complexity metric is not very convincing. There is no evidence to support whether the prototype-based difficulty metric is reliable or not.", "questions_to_authors": "A. Have the authors considered and explored other measurements of data complexity, e.g., compound divergence (Keysers et al., Shaw et al.), local structures (Bogin et al.)? If so, how do they influence generalization performance?\n\nB. Have the authors studied other compositional splits for SCAN (e.g., MCD from Keysers et al.) and GeoQuery (e.g., TMCD from Shaw et al.)? And other semantic parsing datasets that evaluate compositional generalization, e.g., CFQ (see a similar study from Tsarkov et al.)?\n\nC. Can the authors clarify the differences between “improving compositional understanding” and “preventing surface memorization”. And why do we need to distinguish them and explicitly characterize these two model behaviors? Aren’t these two factors always correlated? The discussions in L288-L310 could benefit from further clarification.\n\nD. SCAN* creation details: L1074-1078, is there any specific reason for this design? Based on my experience, one of the most significant challenges of SCAN is separating the “around” and “opposite”. \n\nE. Can the authors clarify more how the section starting from L378 supports the two proposed hypotheses in section 4.2? Theoretically, it’s still possible to have large datasets with example repetition or primitive repetition curriculum (bottom of Table 3)? The experimental setup used here does not seem to mimic the standard training practice. I wonder how much of the conclusion drawn here can directly support the claim that reducing frequently recurring examples improves compositional generalization.\n\nF. Is there any qualitative analysis of the difficulty metric, especially for the prototype-based metric in the context of a natural language dataset? How effective is it at clustering examples? I imagine that for some compositional splits, such as SMCalFlow-CS, the \"hard\" examples might be quite similar in the embedding space. For instance, training examples like \"who is my manager\" and \"schedule a meeting with Amy,\" and a test example like \"schedule a meeting with my manager,\" could be very close in the embedding space. Therefore, this measure might not be a good indicator of example-level difficulty. Other measurements, such as those by Bogin et al. (see below), might provide more informative signals.\n\nG. Figure 4: IIUC, difficulty here refers to the L2 distance to the centroid. If so, why does it appear to always be an integer?", "paper_topic_and_main_contributions": "This paper provides a diagnostic and empirical study of data factors that will influence the compositional generalization of Transformer models trained from scratch. In particular, it studies dataset scale and pattern complexity and finds that increasing dataset complexity can improve the compositional generalization of models. It offers further hypotheses and analysis about why increased complexity would improve generalization. Finally, the paper explores how example difficulty influences generalization differently in both synthetic and realistic datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Example level complexity: Bogin et al. Unobserved local structures make compositional generalization hard"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 2: The legend could be clearer. For example, it could directly define \"+Both\" as \"20x augmentation + more primitives.\"\n- Good to have dataset statistics and examples in the appendix. \n- L1132: speed-based difficulty metric is not mentioned in the main text.\n- Section 5 seems not to be closely related to Sections 3-4, and even uses some different datasets. It might be helpful to clarify these points in the main experimental setup, and perhaps consider making Section 4 a subsection of Section 3."}}, {"review_id": "mG5aAkgJ3D", "reviewer": "Reviewer_CqGi", "summary": "", "questions": "Please refer to the above section.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and we are glad you acknowledged the insight provided by our experiments.\n\n**On the importance of the computation factor**\n\nThank the reviewer for highlighting the computational factor and we agree on its importance. However, our experiment design has already taken that factor into consideration. For any experiments in the same table, we always train the models using the same amount of total steps even though they are trained on datasets of different sizes (e.g., 2*, 20*, 200* Augmentation in Table 1). Hence, the comparison in every table is always under the assumption of using the same computation cost. We report some related details at LINE 382-383 and LINE 1112-1115. We decided to run fewer runs for large datasets mainly because of data preprocessing overhead and the slow convergence on large datasets, but we give the same computation budget to every model and train them till convergence. We will further clarify this point in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provides interesting insights into why models perform better on larger and more complex data. The method employed seems to be sound (up to a point, please refer below).", "reasons_to_reject": "One major issue I see in this study is the fact that the *computational* factor seems to be largely ignored. In particular, in lines 448-449, the authors state that \"For this section, we report results with three runs as the dataset sizes are substantially larger\". Is this factor not giving an  unfair advantage to a model trained on a larger dataset? Should a fair comparison not use the same amount of compute in both setting (e.g. smaller vs larger dataset)?", "questions_to_authors": "Please refer to the above section.", "paper_topic_and_main_contributions": "This paper explores an interesting issue with several NLP tasks: a model trained on a larger dataset or a more challenging task can generalise better on these specific tasks, compared to training the same model from scratch. To understand this inconsistency, the authors show that a model trained on a larger or more complex dataset performs better on the same task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HrIHwuVGmw", "reviewer": "Reviewer_3qWv", "summary": "", "questions": "1. As for the medium-level complexity task, would it be any improvement if you first train the model on the easy examples and then gradually increase the difficulty? It'll be more like curriculum learning. The reason that I think the model fails when only training on complex examples might be that there is a gap between the initialized model knowing nothing and a well-trained model knowing complex rules, while using easy examples might close the gap and achive better results than simply training the model on medium-level complexity task. ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for appreciating our systematic analysis and our AugZero idea. We also thank the reviewer for raising the important point on novelty and the connection to LLMs.\n\n**On the novelty of the data complexity analysis**\n\nWe agree with the reviewer that other papers have studied the broad topic of complexity and observed the effect of large-scale datasets, as we acknowledge at the very beginning of our work at LINE 056-058. However, to the best of our knowledge, there are still very few works explaining the reason behind the empirical gain. Our work is motivated by such mysterious benefits, and we explore hypotheses explaining this benefit via the angle of repetition frequency (Sec. 3). Our hypothesis also leads to a novel data augmentation algorithm, AugZero, which is acknowledged by Reviewer 9qDf.\n\n**On implications on LLMs**\n\nIn Sec. 7, we provide a discussion about the implications of findings in our work on LLMs. Our connection is more on the finetuning stage of LLMs instead of pretraining, which is how most people use LLMs due to computational constraints. Indeed when we do pretraining, the common practice is to choose a more diversified and larger corpus. However, this is not always possible for finetuning. To give a more concrete example, during finetuning, we sometimes only have access to a small dataset with limited diversity. In this case, we will face the choice to either finetune the model directly on the small subset, or do some augmentation to diversify the small subset. The results in our work indicate that the latter may be a better choice (see LINE 618-637). We will further clarify these implications in our final version in Sec. 7. In addition, please also check our results using pretrained Transformers in Appendix F.2 showing the generalizability of our findings.\n\n**Response to Question 1**\n\nThank you for the suggestion! Training the model with an easy-to-difficult curriculum is a great idea. Its benefit is also suggested in the Mix results in Table 4. Following your suggestion, we trained the model with an easy to difficult curriculum on SMCalFlow using the data in Figure 4 and the same total training steps. The overall model achieves an accuracy of 50.35, substantially higher than the model only trained on the hardest subset. While these two performances are not directly comparable as more data are used in the curriculum training model, it proves the benefit of training first on easy data before training on difficult data. To further find the best curriculum for generalization is a promising direction and we will explore it in our future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper systemetically analyses the effects of the dataset complexity and the scale complexity on the generalization abilities. \n2. The idea that only augmenting the dataset with a whole new set of vocabulary while not introducing any new knowledge could boost the performance by a large range is very interesting and intuitively makes sense. \n3. The finding that training the model on natrual language datasets with medium level difficulty could achieve the best accuracy is also interesting and might be of interest to the community. ", "reasons_to_reject": "1. The observation that the complexity and large scale would affect the training does not seem to be a very novel observation, it seems to be a consensus in machine learning community. \n2. As the difficulty would be hard to measure and the compositional generalization may have not been defined in pretraining, the method and discoveries proposed in this paper do not seem to be very useful for the development of large language models. Generally speaking, we need to choose more diversified and larger pretraining corpus, which is already what they are doing when performing pretraining. I'm kind of curious about what kind of new ideas and inspirations would this paper bring to the NLP community when most of them are working on Large Language Models. ", "questions_to_authors": "1. As for the medium-level complexity task, would it be any improvement if you first train the model on the easy examples and then gradually increase the difficulty? It'll be more like curriculum learning. The reason that I think the model fails when only training on complex examples might be that there is a gap between the initialized model knowing nothing and a well-trained model knowing complex rules, while using easy examples might close the gap and achive better results than simply training the model on medium-level complexity task. ", "paper_topic_and_main_contributions": "This paper aims to analyze (1) the connection between the increased dataset complexity and better compositional generalization; (2) why more complex datasets lead to better generalization. They also give some of the possible reasons as (a) More diverse patterns exemplifid in complex datasets increase the difficulty of surface memorization. (b) Larger dataset size increase the difficulty of simple memorization of samples. (3) How examples of different difficulties inside a dataset affect generalization. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work investigates compositional generalization in seq2seq models, and aims to disentangle the factors that enable a model to generalize compositionally.

The submission received 4 reviews. Overall, the reviewers agree that the soundness of the proposed work is sufficient to support its main arguments, though all point out additional points that should be addressed in a revision. These points include adding further motivation for focusing on the specific tasks and datasets and a justification for the complexity metric used, and improving the clarity of several sections. While the work is judged to be largely sound and a good contribution, it is less clear how exciting the resulting insights will be to the EMNLP audience.
Decision: Accept-Main