Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers
Authors: Osman Batur İnce
Tanin Zeraati
Semih Yagcioglu
Yadollah Yaghoobzadeh
Erkut Erdem
Aykut Erdem
Venue: EMNLP 2023 Findings (2023)
Keywords: dataset cartography
compositional generalization
training dynamics
TL;DR: Leveraging dataset cartography improves compositional generalization in Transformer models, boosting accuracy by up to 10% on CFQ and COGS datasets without hyperparameter tuning.

Abstract: Neural networks have revolutionized language modeling and excelled in various downstream tasks. However, the extent to which these models achieve compositional generalization comparable to human cognitive abilities remains a topic of debate. While existing approaches in the field have mainly focused on novel architectures and alternative learning paradigms, we introduce a pioneering method harnessing the power of dataset cartography (Swayamdipta et al., 2020). By strategically identifying a subset of compositional generalization data using this approach, we achieve a remarkable improvement in model accuracy, yielding enhancements of up to 10% on CFQ and COGS datasets. Notably, our technique incorporates dataset cartography as a curriculum learning criterion, eliminating the need for hyperparameter tuning while consistently achieving superior performance. Our findings highlight the untapped potential of dataset cartography in unleashing the full capabilities of compositional generalization within Transformer models.

Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers
Osman Batur Ince 1. Tanin Zeraati Semih Yagcioglu
Yadollah Yaghoobzadeh Erkut Erdem* Aykut Erdem
Koç University, KUIS AI Center Koç University, Computer Engineering Department University of Tehran, Iran Hacettepe University, Computer Engineering Department Tehran Institute for Advanced Studies, Khatam University, Iran semih. haaphhyyagccoolulehcettteeee edu. tr
edu tzeraati@ut.c ac. ir y w.yaghoobzadeheuu.cco ac. erkut@cs. wepee edu. tr
aerdemēku. edu. tr
Abstract
Compositional generalization refers to the ability of a model to combine known parts sentence, such as primitive tokens, to generate novel compo- sitions of these primitive elements. It is considered a fundamental aspect of human cognition and lin- guistics (Fodor and Pylyshyn, 1988). In addition to its human aspect, compositional generalization is also crucial for enhancing the robustness and practical use of deep learning models. Efforts understand and improve the compositional gener- alization abilities of models have gained signifi- cant attention lately. Researchers have recently ex- plored techniques such as compositional data aug- mentation (Andreas, 2020; Qiu et al., 2022), meta- learning (Lake, 2019), and structural priors (Russin et al., 2020). Additionally, the importance of chitectural modifications capture compositional structures more effectively, such as attention mech- anisms (Li et al., 2019) and hierarchical struc- tures (WeiBenhorn et al., 2022) have been investi- gated recently. In another direction, there is also an increasing interest in studying the compositional generalization abilities of Transformers (Ontanon et al., 2022: Csordás et al., 2021; Dziri et al., 2023).
Neural networks have revolutionized language modeling and excelled various downstream tasks. However, the extent to which these models achieve compositional generalization comparable human cognitive abilities re- mains topic of debate. While existing ap- proaches in the field have mainly focused novel architectures and alternative learn- ing paradigms, we introduce a pioneering method harnessing the power dataset car- tography (Swayamdipta al., 2020). By strate- gically identifying a subset of compositional generalization data using this approach, we achieve remarkable improvement in model accuracy. yielding enhancements of up to 10% CFQ and COGS datasets. Notably, our technique incorporates dataset cartography as a curriculum learning criterion, eliminat- ing the need for hyperparameter tuning while consistently achieving superior performance. Our findings highlight the untapped potential of dataset cartography in unleashing the full capabilities of compositional generalization within Transformer models. Our code is avail- able at https: thwb. www.////////////////oo//eeeeeeee cyberiada/
In this study, we take distinct approach and utilize dataset cartography (Swayamdipta et al., 2020) to explore how training dynamics can im- prove the compositional generalization abilities of Transformers. Dataset cartography is recently proposed technique that quantifies the variability and confidence associated with instances during training, capturing their ambiguity and difficulty, thereby representing the informational value of each training sample. Swayamdipta et al. (2020) demonstrated that it could be used to improve out- of di-striibution (OOD) generalization in models for classification-baseed natural language inference (NLI) tasks. As compositional generalization is inherently an OOD task, we hypothesize that har- nessing dataset cartography in compositional gen- eralization can provide new insights.
1 Introduction
In recent years, deep learning methods and ma- chine learning infrastructure have made remark- able progress, enabling models to surpass human- level performance in numerous tasks. Natural lan- guage processing (NLP) is at the forefront of this progress. Models based on Transformers (Vaswani et al., 2017) such BERT (Devlin et al., 2019) and benchmarks like SuperGLUE (Wang et al., 2019) led to significant advancements in language modeling and various downstream tasks. However, there is an ongoing debate on whether these mod- els exhibit compositional generalization (Fodor and Pylyshyn, 1988; Smolensky, 1988; Marcus, 2001; Lake and Baroni, 2017).

---

Diverging from the original cartography setup, we focus on language generation tasks for the sys- tematic generalization problem and propose an ex- perimental setting to apply dataset cartography to a generative task. Initially, we train a sequence- to-sequence (seq2seq) Transformer model using the complete training set for only few epochs. Throughout the training, the dynamics of each in- stance are observed and recorded separately. Next, we utilize these stored training dynamics build a curriculum and create a reduced training set by selecting specific samples to fully train the model.
analyses, we demonstrate the significant impact of leveraging training dynamies through dataset cartography on the compositional generalization capabilities of Transformer models. Our approach yields significant improvements of up to 10% on challenging CFQ and COGS datasets, highlighting the effectiveness of our proposed method.
2 Approach
2.1 Dataset Cartography
Swayamdipta et al. (2020) propose a visualization tool named data maps with two dimensions, con- fidence and variability, which characterizes the informativeness of training instances of a dataset with respect to model. Confidence is calculated the mean probability of the true label across epochs, whereas variability corresponds the spread of confidence across epochs, using the standard devia- tion. Therefore, confidence ((t) and variability (ō:) is denoted as follows:
Our experimental setup has notable challenges beyond the compositional generalization setting, distinguishing it from the setup originally used in Swayamdipla et al. (2020). Instead of relying on crowdsourced datasets that are prone to errors and heavily reliant on data quality, we utilize synthet- ically generated datasets, namely CFQ (Keysers al., 2020) and COGS (Kim and Linzen, 2020), which are free from such limitations. Moreover, these datasets are relatively smaller size, mak- ing it challenging to achieve performances on par with the 100% train set when using smaller subsets. Lastly, unlike Swayamdipta et al. (2020), we tackle the complexity of learning the task directly without using pre- pre-trained models. This becomes even more pronounced the datasets contain non-natural lan- guage, rendering pre-training less applicable and learning much harder. Finally, and more impor- tantly, as we are dealing with language generation tasks, quantifying how hard a sequence is, is not straightforward. To address this, we base our eval- uation by utilizing inverse perplexity (Inv PPL., CHIA (Bhatnagar et al., 2022), and BLEU (Pap- ineni et al., 2002) confidence measures, avoiding the overly strict exact matching strategy.
E
iu
Potes (uilxs)
(1)
E
6, Ene
(Pates (v(( - px)²
(2)
E
where i denotes the instance, E represents the total number of epochs, x; the input sequence, yf is the true label, B(e) corresponds the of model parameters at epoch e.
Data maps reveal three distinct regions: am- biguous instances (high variability). easy-to-learn instances (high confidence, low variability), and hard-to-learn instances (low confidence, low vari- ability). Swayamdiptu al. (2020) experimented on multiple NLI datasets such as SNLI (Bowman et al., 2015) with pretrained models and reported three main findings: (i) Ambiguous regions con- tribute the most towards OOD generalization, (ii) Easy-to-learn regions play an important role in model optimization, (iii) Hard-to-lean regions of- ten correspond to labeling errors.
In summary, our paper makes the following key contributions: First, we introduce the novel use of dataset cartography as both a curriculum learn- ing criterion and a sample selection strategy for enhancing compositional generalization. By lever- aging dataset cartography, we enable models to deal effectively with the complexities of composi- tional tasks. Second, we thoroughly investigate the effectiveness of various confidence measures for sequences in extracting dataset cartography within the compositional generalization setting. This anal- ysis provides insights into quantifying the difficulty of sequences and leads to the development of ro- bust training strategies. Third, through extensive
2.2 Data Maps for Generative Tasks
The notions of confidence and variability in Eq. (1) and (2) are defined considering classification- based tasks, and thus not directly applicable seq2seq models. Extending dataset cartography machine translation, generative task, Bhatnagar et al. (2022) propose the CHIA measure by follow- ing the intuition that an output sequence consists of

---

BLEU Data
1.0
R000
0.2
0.3
0.5
2000
0.8
0:7
0.5
1.0
0.8
Confidence
Confeience BLEU
0.7
Density
5000
0.0
0.1
Variabiliity
0.4
15000
Deasity
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
209N300 Comrecteess
Variability ALEU
Figure 1: Data map of CFQ train set for the Transformer model based BLEU measure (converge epoch 20). The x-axis shows the variability and the y-axis the confidence. The colors and shapes indicate the correctness.
series of predictions. Instead of using the exact match, they take the arithmetic mean of gold token predictions over the sequence, defined as:
Additionally, define third measure based BLEU (Papineni al., 2002). In particular, BLEU measures the n-gram overlap between generated output and the ground truth, and we use the arith- metic mean of the BLEU score across epochs as a means to measure the confidence as follows:
A
poe (y
Xi)
where Vi corresponds to the i-th token of the groundtruth output sequence y; of length T.
BLEU (e)
i4
yi)
(5)
F.
Here, it is important to note that Bhatnagar et al. (2022) do not use data maps to select subset but instead use N-way translation corpora to choose instances that are most informative on all ways to select instances to annotate for low-resource lan- guages. They showed that choosing instances based on a single-way translation decreases performance significantly, suggesting CHIA measure might not be the best choice for experimental setting.
where y refers the predicted sequence gener- ated by the model parameters epoch e, and yi denotes the ground truth sequence, respectively. A particular shortcoming using BLEU its com- putational and temporal expense due to decoding (for variability equations, see Appendix A.2).
The main motivation behind utilizing dataset car- tography lies in selecting subset of the training and training the model on these subsets instead of the entire dataset. The selection process involves two key considerations: (1) Choosing the measure used to rank the examples, and (2) Determining the aspect of the measure scores for ranking (e.g., am- biguity). There are more hyperparameters such subset ratio and subset combinations, until which epoch to take training dynamics into account (re- ferred convergence epoch), from starting with which epoch training dynamics are considered (re- ferred to as min epoch). Specifically, to identify the convergence epochs, we qualitatively examine loss convergence and generated data maps. Unlike
Similar the CHIA score, we also consider in- verse perplexity for the reason that high perplexity is an undesirable property. It is defined as the ge- ometric mean of gold token predictions over the sequence, as given below:
E T II
P
V pa- (yit
xi)
(4)
E
The geometric mean is much closer to the low- est probability in the sequence compared to the arithmetic mean used CHIA, making inverse perplexity a more discriminative measure.

---

Swayamdipta al. (2020), where authors utilize pre-trained models and consider training dynam- ies from the start of fine-tuning, our experimental setup involves randomly initialized models. Hence, considering training dynamics in the initial epochs while the model is not stable can result in noisy training dynamics (Swayamdipta et al., 2020), and can introduce selection biases based on data order- ing. To simplify the process, we set the minimum epoch to 3 across all our training setups.
Dataset
Strain
#test
Voc. size
Train len.
Test len.
CFQ
95743
11968
181
29/95
30/ 103
COGS
24155
21000
871
22 153
61 480
SMCS 16
25410
663
10738
107 / 103
59/59
SMCS 32
25426
662
10738
107 103
30/ 59
Table I: Dataset statistics showing sample counts, cabulary size as the combined input and output vocab- ularies, and train and test length denoting the max. in- put/output length in the train and test set, respectively.
3 Experiments
case of COGS, which is synthetic semantic parsing task, an input sample could be "A frog hopped" and the corresponding target logical form is frog(x1) AND hop.agent (x2, x1). For the natural semantic parsing dataset SMCS, an in- put sample is "iam meel with smith john and rodney", and its output is CreateEvent( AND with_ attendeee rodney ) with_attenden ( smith ) with_attendee( john ) )
3.1 Baselines
We benchmark with several baselines to demon- strate the enhancement in generalization perfor- mance through the selection of smaller, specifically chosen subsets. The most rudimentary of these baselines involves the selection of a random sub- set identical in size to the specifically chosen sub- set, along with the utilization of the entire original dataset for comparison, i.c. 100% of the original dataset. In the context of curriculum learning set- tings, we deem it necessary to establish another benchmark wherein no particular curriculum is em- ployed. This serves as baseline, facilitating the process of benchmarking for comparative purposes.
3.3 Experimental Setup
In our experiments, we employ the vanilla Trans- former model (Vaswani et al., 2017). Recent stud- ies have highlighted that the generalization pabilities of pre-trained Transformer models be overestimated due to uncontrolled lexical ex- posure (Kim et al., 2022; An et al., 2023). We adopted the publicly available Py PyTorch codebasc provided by Csordás et al. (2021) to implement our model. Each experiment is executed on a single Tesla T4 GPU. We employ whitespace tokenizer for all datasets, considering that the targets in these datasets are not expressed in natural language. We also experiment with Bi-LSTM with attention (Bah- danau et al., 2015) on the COGS dataset.
3.2 Datasets
We conduct our experiments on three compo- sitional generalization datasets, CFQ (Keysers al., 2020), COGS (Kim and Linzen, 2020), SMCalFlow-CS Simple (Meron, 2022, SMCS) datasets. CFQ and SMCS dataset has multiple splits. For CFQ, we utilize the MCDI split. For SMCS, we utilize 16 and 32 splits, where the split numbers refer to the compositional example leak count into the training set. One challenge com- monly encountered with compositional generaliza- tion datasets in the literature is the absence of val- idation splits. To ensure fair comparison, we train all of our models with specified step counts, following the approach of Csordás et al. (2021).
In a similar way to Swayamdipta et al. (2020), we show the data maps generated for CFQ based on BLEU and COGS based on Inv PPL in Figure and Figure 2, respectively. For better visualiza- tions, we only plot randomly sampled 33% of the training set. Considering larger number of train- ing epochs compared to Swayamdipta et al. (2020), we divide the correctness scores into 10 bins for better granularity and informative visualizations. As we discussed carlier, we use three distinct con- fidence measures, inverse perplexity (Inv PPL), CHIA (Bhatnagar et al., 2022), and BLEU (Pa- pineni al., 2002). The missing data maps are given in Appendix A.7.
To provide a better understanding of the datasets, let us consider specific examples from each. CFQ, being a synthetic text-to-SQI. dataset, involves input samples such as "Did a male film direc- tor produce and edit M1?" with the corre- sponding target query being SELECT (*) WHERE {?x0 fi:film. htl.mmproduuer... film M1
?x0 ns: people. person. gender m_0zzppz]. In the wwtpppeeeperron..omm
?x0 ns: film. aditorrfilm M1
We explore two different setups to assess the effectiveness of leveraging data maps in improv-

---

Inverse PPL Data Map
3000
1.0
0.3
0.2
Demelly
2000
0.5
1000
0.6
0.7
Confaince -Ivertee DTL
0.8
Confidence
1.0
0.6
Deneiiy
3000
2000
1000
0.4
0.0
0.2
Variabiitt
Denettr
2000
1000
0.0
0.000
0.025
0.050
0.075
0.300
0.125
0.180
0.176
0.200
Viriability Inverse PPL
Correctness
Figure 2: Data map of COGS train set for the Transformer model based on Inv PPL measure (converge epoch 10). The x-axis shows the variability and the y-axis the confidence. The colors and shapes indicate the correctness.
ing compositional generalization. In the first setup, we utilize subsets comprising 33% of the original datasets. These subsets are categorized as hard- to-learn, ambiguous, and easy-to-learn based on the extracted maps. For the second setup, we train models using subsets sized at 50% of the original datasets. Along with the hard-te leaaen ambigu- ous, and easy-to-learn subsets, construct com- bined subsets that are also half the size of the orig- inal dataset by merging two 33% subsets selected based on the same confidence measure. Specif- ically, we select 33% of the examples from the more informative subset and allocate the remaining 17% from the other subset, following Swayamdipta et al. (2020). As will be discussed in the next sec- tion, our findings demonstrate that hard-to-learn samples have a more pronounced impact model performance compared to ambiguous and easy-to- learn samples, and thus we consider them as more informative. When combining ambiguous and easy- to-learn samples, we consider including greater number of ambiguous samples than easy-to-learn samples. If the union of these subsets is smaller than half of the whole training data, we randomly add samples to reach the 50% dataset size. Fur- thermore, address the out-of-vocabulal vocabulary (OOV) problem during subset selection by incorporating training samples from the entire dataset if they in- crease the vocabulary size. On the contrary, we remove the least informative samples that do not reduce the vocabulary size, ensuring consistent sub-
sizes throughout the experiments. The statistics about the subsets obtained from the data maps are provided Appendix A.4.
In addition to our subset selection experiments, explore the potential of leveraging dataset cartography
criterion for curriculum learn- ing (CL). In particular, we adopt two CL ap- proaches proposed by Hacohen and Weinshall (2019) and Zhang et al. (2019). We experiment with a fixed exponential pacing schedule using de- fault hyperparameters in the former. We set the starting percentage to 4% and increase the scale to 1.9. On the other hand, the second CL method by Zhang et al. (2019) involves sorting examples given criterion and dividing them into 10 equal-sized bins, resulting in 10-stage curricu- lum. Within each bin, the examples are further sorted based on their lengths, and then each sorted bin is divided into non-overlapping batches. We distribute these batches randomly during training to avoid potential selection bias. Since we train our models for fixed number of steps, after complet- ing 1/ 1/ of the training, unlock the second bin in similar fashion.
based
3.4 Impact of Selected Subsets
We conduct thorough analysis understand the effect of subset selection on the training process and how the selection process impacts the subse- quent generalization abilities of the models. Our key findings are summarized in Table for the

---

CFQ
COGS
Inv PPL
CHIA
BLEU
Inv PPL
CHIA
BLEU
easy-to-learn
12.191.20
12.420.59
9.881.83
0.00p.00
0.06p.11
0.04p.07
33% train
ambignous
17.69
23.51g.86
20.991.91
3.265.61
20.303.58
26.694.17
hard-to-learn
36.550.55
34.98n.67
34.711.12
53.50
45.4112.5
50.563.07
random
34.02 .09
18.66 18.666.72
100% training
38.711.01
42.547.65
Table 2: Accuracy results for CFQ and COGS datasets. Models are trained on different 33% subsets of the train data compared using full dataset. The scores are averaged over runs, where std. dev. shown as subscript. The best and second-best performing subsets are highlighted in bold and underlined, respectively. Hard-to-learn subset consistently performs better than the random subset, even outperforming 100% train set on the COGS dataset.
CFQ
COGS
Inv PPL
CHIA
BLEU
Inv PPL
CHIA
BLEU
easy-to-learn
21.13
20.96
17.04
0.000
0.000
0.695
ambiguous
23.03
28.80
24.31
0.047
36.09
35.14
50% train
hard-to-learn
42.45
40.13
37.45
47.48
42.40
45.20
ambiguous easy-to-learn
18.52
26.18
20.77
0.048
18.33
25.69
hard-to-learn + ambiguous
36.54
36.87
37.13
41.13
35.08
41.16
hard-to-learn easy-to-learn
35.91
41.29
39.29
40.82
37.94
40.96
random
35.16
30.24
100% training
37.71
36.80
Table 3: Accuracy results for CFQ and COGS datasets. Models are trained on different 50% subsets of the train data compared to using the full dataset. The best and second-best performing subsets are highlighted in bold and underfined, respectively. It worth mentioning that solely training on hard-to to-leam samples or combining them with easy-to-learn samples outperforms using 100% training samples.
subsets comprising 33% of the original datasets. Our experimental results show that training mod- els on hard-to-leam samples consistently yields superior generalization performance compared to training ambiguous samples. Notably, the per- formance of hard-to-learn subsets surpasses that of random subsets overall, and for the COGS dataset, it even outperforms training on the entire training set. Training the models on easy-to-learn samples, on the other hand, leads to poor generalization per- formance. We also observe that Inverse Perplexity is a more effective measure than CHIA or BI.EU for selecting samples based on their difficulty.
perform 100% training. However, when using the BLEU measure, only the hard-to-leamt-essyy learn subset surpasses the 100% training perfor- mance. Although the subset combinations show promising results with the CHIA and BLEU mea- sures, they still outperformed by the hard-to- learn (Inv PPL) subset. In COGS, we observe even more substantial improvements in accuracy. Across all measures, the hard-to-leant subset demonstrates an accuracy increase of over 5%, with the hard-1o- learn (Inv PPL) subset outperforming the 100% training by over 10% accuracy. Notably, selecting 50% of the hard-to-lean samples consistently out- performs the subset combinations for all measures. While combining subsets does yield performance improvements in certain measures, also high- lights the limited effectiveness of these measures in effectively separating the different classes in- stances. This is evident as the hard-to-learn (Iwv PPL) subset consistently outperforms the subset combinations in both the CFQ and COGS datasets.
As we increase the subset size to 50% of the orig- inal dataset, our experimental results demonstrate significant improvements compared to full dataset training, shown in Table 3. In the CFQ dataset, the accuracy of the hard-to-learn (In PPL) subset exceeds that of the full training by over 4%. When considering the CHIA measure, both the hard-to- to- learn and hard-to- toaleeee-aaaayaaaeaaaaaa tortey------ae learn subsets out-

---

kerd to Sear (CUMA)
Kore taree (C2IAA)
tn lene (CRIA)
Leniianu (CIMA)
Test Auranes
0.8
umpaisaare PP%
Nari-tadium flem
aalliane (Ne 201)
map to (Pa, PM1)
kand to-er (2263)
kare taisern (ELLE))
sopple-toon, (XL80)
anllyaasa (SLAN)) (RLB8)
(8G61)
000 200
300
1000
2010
2000
2000
3003
5000
42C
660
500
J003
Stype
Stge
Figure 3: Accuracy plots on CFQ for the CL strategy by Hacohen and Weinshall (2019).
Figure 5: Accuracy plots on COGS for the CL strategy by Hacohen and Weinshall (2019).
04
0.6
Lend to Seow (CSTAA)
0.5
enllescce (CWIA,
Nara-tallur (am 203)
022
Sucr 2013
noppiekkm (Prs PRU
doc----torrs /hx 191,
exllpscce PP1)
unligaat (ber 291)
(BL82)
sop-te-km (ULEU)
eehlpacc (BL85)
300
1000
3300
200
200
300
Stepe
Figure 4: Accuracy plots on CFQ for the CL strategy by Zhang et al. (2019)
Figure 6: Accuracy plots on COGS for the CL strategy by Zhang et al. (2019).
3.5 Impact of Cartography-Based Curriculum Learning
This aligns with our previous findings, emphasizing the benefits of starting with challenging examples for improved adaptation. Figure 4 examines the impact of leveraging data maps within the CL strategy proposed by Zhang et al. (2019) for compositional generalization. The hard-to-leane (BLEU) configuration outperforms the no curriculum strategy, albeit with no notable improvement in convergence speed. This out- come mirrors our observations using the CL frame- work developed by Hacohen and Weinshall (2019), where initiating training with harder samples leads to better performance. However, the ambiguous configurations perform similarly no curriculum, while the easy-to-learn configurations yield worse results than the no curriculum approach.
ing expectations.
We use dataset cartography to examine the impact of training dynamics on curriculum learning. Cur- riculum learning is a strategy that trains models on instances from easy to hard, based on the assump- tion that this order facilitates learning. However, we also explore the opposite strategy, which trains models on instances from hard easy, and com- pare it with the conventional curriculum learning approach. This way, we can study how different training schedules affect the model performance.
Figure 3 depicts accuracy plots showing the per- formance of various CL strategies based on (Haco- hen and Weinshall, 2019) on the CFQ dataset. The figure legends indicate the ranking scheme and the employed confidence measure. For instance, hard- to-learn (Inv PPL) refers to the case where Inv PPL is being used as the confidence measure, and the in- clusion of the hard-to-leare samples is prioritized within the curriculum. Our analysis reveals that no single curriculum consistently outperforms oth- ers on the CFQ dataset. Exponential pacing leads to stagnant performance in the final 2/7th of the training process due surpassing the training size percentages of 33% and 50%. Surprisingly, initi- ating training with hard-to-leare samples yields superior performance compared to easy-to-learn samples, contrary to common curriculum learn-
In Figures 5 and 6. we gain deeper insights into the contributions of dataset cartography. Overall, hard-to-leary (BLEU) emerges as the most effec- tive configuration in the plots. Surprisingly, am- biguous (Inv PPL) performs as the second-bess con- figuration in Figure 6, while hard-to-learn (Inv PPL) holds this position in Figure 5. The no cur- riculum approach ranks third and fourth in these respective plots. Furthermore, the easy-to-learn configurations demonstrate the poorest final per- formance across both curriculum learning frame- works.
Analyzing the accuracy plots of curriculum

---

learning, we observe that initiating training with casier examples and gradually progressing more challenging instances does not lead to accelerated convergence or improved final model performance. On the other hand, the subset experiments pre- sented in Tables 2 and 3 show that training models on hard-to-learn examples result in better model performance. Furthermore, the CL results highlight that starting the curriculum with hard-to-learn sam- ples results in enhanced final performance. These findings, combined with the observation that the first unlocked examples are encountered more fre- quently during training, suggest that the superiority of hard-to-learn curricula over the no curriculum can be attributed to the increased exposure to chal- lenging instances throughout the training process.
leveraging this connection, they propose a strategy to enhance overall generalization performance. Contemporary research on compositional gener- alization focuses on two main aspects: proposing new datasets to explore model generalization capa- bilities and introducing novel techniques to address the compositionality problem.
As one of the early sample datasets, SCAN (Lake and Baroni, 2017) simulates navigation task, and measures generalization to longer splits or systematic generalization with new verbs in dif- ferent splits. Keysers et al. (2020) define a math- ematically rigorous way to create compositional datasets and create CFQ dataset which is a semantic parsing task of generating SPARQL queries from natural language questions. Kim and Linzen (2020) proposed COGS for semantic parsing, where the source is an English string, and the target is its logical form.
To sum up, our experimental findings highlight the effectiveness utilizing dataset cartography for training subset selection and curriculum learning in the context of compositional generalization. Our results consistently show that leveraging dataset cartography leads to improved generalization per- formance. While curriculum learning also con- tributes performance enhancement, its impact appears to be smaller compared the use of dataset cartography for subset selection.
terms of novel techniques, researchers pro- pose various approaches for compositional gen- eralization. These include creating novel archi- tectures to solve compositionality problem (Perez et al., 2018; Hudson and Manning, 2018 8), modify- ing existing architectures for better generalization (Russin et al., 2020; Akyurek and Andreas, 2021), utilizing different learning paradigmıs such as meta- learning (Lake, 2019) and pre-training (Furrer al., 2020), or data augmentation (Andreas, 2020; Qiu et al., 2022). With the rise of large language mod- els (LLMs), choosing in-context examples for bet- ter compositional generalization with LLMs (Levy et al., 2022; An et al., 2023) is another open re- search problem. In the compositional generaliza- tion literature, only few studies investigated the impact of training dynamics on generalization per- formance. For instance, studies by both Liu et al. (2020) and Chen et al. (2020) propose curriculum learning schemes to help models learn accurate exe- cution traces for lengthy training samples. They di- vide the samples into partitions based on the length and train the models sequentially, starting with the shortest examples. In contrast, our work takes different approach by utilizing training dynamics to create data maps and leveraging them for com- positional generalization. This achieved either through subset selection or curriculum criteria.
4 Related Work
Swayamdipta et al. (2020) use training dynamics create data maps that categorize the dataset into three groups: easy-to-learn, hard-to-learn, and am- biguous. similar vein, Toneva al. (2019) em- ploy training dynamics for dataset categorization, specifically in classification tasks, by identifying misclassified or forgotten instances. On the con- trary, the adversarial filtering algorithm proposed by Bras et al. (2020) ranks instances based on pre- dictability, suggesting the removal of easy-to-learn examples. However, our research presents contrast- ing findings. Our experimental analyses show that combining the easy-to-learn category with other categories can improve the generalization perfor- mance. In another recent study, Wang et al. (2022) explore the relationship between the generalization performance and the training dynamics in an active learning setting. Their approach revolves around the adaptive selection of samples for labeling to obtain comparable or better performance with less training data. Notably, they discovered robust cor- relation between the convergence speed of training and the resulting generalization performance. By
5 Conclusion
Transformers are great at language modeling and various downstream tasks, but their ability to

---

achieve compositional generalization compared to humans remains debatable. In this study, we ad- dressed this challenge by demonstrating that select- ing a subset of the training dataset using dataset cartography and training models on this subset can enhance model accuracy by up to 10%. We showed that our setup can generalize to different model architectures and natural datasets. Moreover, we achieved improved performance by employing a dataset cartography-bane based curriculum learning without the need for hyperparameter tuning. Look- ing ahead, we anticipate that this research direction promises insights into the necessary syntax, seman- tics, and structure for informative data instances, informing the development of novel data augmen- tation strategies and advancing our understanding of deep models' generalization capabilities.
References
Ekin Akyurek and Jacob Andreas. 2021. Lexicon learn- ing for few shot sequence modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ih International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4934-4946, Online. Association for Computational Linguistics.
Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Jian- Jian-Guang Lou, and Dongmei Zhang. 2023, How do in-context examples affect compositional generalization? In Proceedings of the 61th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers). Association for Computational Linguistics.
Jacob Andreas. 2020, Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7556-7566, Online, Association for Computational Linguisties.
Limitations
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Synthetic data. While not a limitation specific to our approach, many of the datasets used for compositional generalization, including CFQ (Key- sers et al., 2020) and COGS (Kim and Linzen, 2020), are synthetically generated. Hence, they may not cover all the complexities of natural lan- guages. While we perform initial experiments with SMCalFlow-CS Simple (Meron, 2022), a natural compositional generalization dataset, additional ex- periments are necessary to make conclusive state- ments.
Rajat Bhatnagar, Ananya Ganesh, and Katharina Kann. 2022. CHIA: choosing instances to annotate for ma- chine translation. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 7299-7315. Association for Computational Linguistics.
Other models. In our work, we performed exper- iments with vanilla Transformers (Vaswani et al., 2017) and Bi-LSTM with attention (Bahdanau al., 2015). While we conduct preliminary ex- periments on Bi-LSTM with attention and demon- strate that these results are in line with primary experiments, more experiments are needed to make conclusive remarks about other neural models.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632 642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bha- gavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. In Proceedings the 37th Inter- national Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1088. PMLRR
Ethics Statement
This work consists of running a lot of experiments on GPU clusters, therefore it requires energy and increases carbon footprint. For this reason, best practices to protect the environment are employed. All datasets and used code are utilized with permis- sive licenses and will be made available.
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song. and Denny Zhou, 2020. Compositional generaliza- tion via neural-symbolic stack machines. Advances in Neural Information Processing Systems, 33:1690- 1701.
A Acknowledgment
Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. 2021. The devil is in the detail: Simple tricks im- prove systematic generalization of transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP
This work was partly supported by the KUIS AT Center Fellowship to Osman Batur ince.

---

2021, Virtual Event / Punta Cana, Dominican Repub- lic. 7-11 November, 2021, pages 119-634.AA0 Associa- tion for Computational Linguisties.
Najoung Kim and Tal Linzen. 2020. COGS: compo- sitional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Confer- ence on Empirical Methods Natural Language Processing (EMNLP), pages 9087-9105. Online. As- sociation for Computational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties; Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA. June 2-7, 2019, Volume I (Long and Short Pupers), pages 4171-4186. Association for Computational Linguistics.
Najoung Kim, Tal Linzen, and Paul Smolensky. 2022. Uncontrolled lexical exposure leads to overestima- tion of compositional generalization in pretrained models.
Brenden M. Lake. 2019. Compositional generalization
through meta eequence-to-sequeeee learning. In Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9788-9798.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chan- dra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. Faith and fate: Limits of transformers on compo- sitionality. arxiv preprint arXiv:2305.18654
Brenden M. Lake and Marco Baroni. 2017. Still not sys- tematic after all these years: On the compositional recurrent networks.
skills of In 6th International Conference on Learning Repre- sentations, ICLR 2018 Workshop Track, Vancouver BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, volume abs/ 111.0005500 Open- Review.net.
Jerry A. Fodor and Zenon W. Pylyshyn. 1988. Connec- tionism and cognitive architecture: A critical analysis. Cognition, (812))371
Daniel Furrer, Mare van Zee, Nathan Scales, and Nathanael Schärli. 2020. Compositional generaliza- tion in semantic parsing: Pre-training specialized architectures. arXiv preprint arXiv:2007.08977 00.900000.00......
Itay Levy, Ben Bogin, and Jonathan Berant. 2022. Diverse demonstrations improve in-context compositional generalization. arXiv preprint arXiv:2212.06800.
Guy Hacohen and Daphna Weinshall. 2019. On the power of curriculum learning in training deep net-
works. In Proceedings of the 36th International Con- ference on Machine Learning, volume 97 of Pro- ceedings of Machine Learning Research, pages 2535- 2544. PMLR.
Yuanpeng Li, Liang Zhao, Jianyu Wang, and Joel Hest- ness. 2019. Compositional generalization for primi- tive substitutions. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4293-4302, Hong Kong, China. Association for Computational Linguistics.
Drew A. Hudson and Christopher D. Manning. 2018. Compositional attention networks for machine rea- soning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver. BC, Canada. April 30- May 3, 2018, Conference Track Proceed- ings. OpenReview.net.
Qian Liu, Shengnan An, MaanGuang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang. 2020. Compositional generaliza- tion by learning analytical expressions. Advances in Neural Information Processing Systems, 33:1 1416 11427.
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Chris- tos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schotimann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2022. State-of-the-art gen- eralisation research in NLP: a taxonomy and review. CoRR.
Gary F. Marcus. 2001. The Algebraic Mind: Integrat- ing Connectionism and Cognitive Science. The MIT Press.
Joram Meron. 2022. Simplifying semantic annotations of smcalflow. ArXiv, abs/2206.13425
Daniel Keysers. Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang. Mare van Zee, and Olivier Bousquet. 2020. Measur- ing compositional generalization: A comprehensive method realistic data. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Eshiopia, April 26-30, 2020. OpenReview.net.
Santiago Ontamon, Joshua Ainslic, Zachary Fisher, and Vaclav Cvicek. 2022. Making transformers solve compositional tasks. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 3591- 1- 3607, Dublin, Ireland, Association for Computational Linguistics.

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguisties.
7th International Conference on Learning Represen- tations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, Conference Track Proceedings. OpenRe- view.net.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L. ukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Arkil Patel, Satwik Bhattamishra, Phil Blunsom, and Navin Goyal. 2022. Revisiting the compositional generalization abilities of neural sequence models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 424-434, Dublin, Ireland. Association for Computational Linguistics.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: stickier benchmark for general purposs language understand- ing systems. CoRR, abs/ bbss/199500057..
Ethan Perez, Florian Struh, Harm de Vries, Vincent Du- moulin, and Aaron C. Courville. 2018. Film: Visual reasoning with a general conditioning layer. In Pro- ceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 3942- 3951. AAAI Press.
Haonan Wang. Wei Huang. Ziwei Wu, Hanghang Tong. Andrew Margenot, and Jingrui He. 2022. Deep active learning by leveraging training dynamics. Ad- vances in Neural Information Processing Systems, 35:25171-2518..
Pia Weißenhorn, Lucia Donatelli, and Alexander Koller. 2022. Compositional generalization with broad- coverage semantic parser. In Proceedings of the 11th Joint Conference on Lexical and Computational Se- manties, pages 44-54, Seattle, Washington. Associa- tion for Computational Linguistics.
Linlu Qiu, Peter Shaw, Panupong Pasupat, Pawel Krzysztof Nowak, Tal Linzen, Fei Sha, and Kristina Toutanova. 2022. Improving com- positional generalization with latent structure and data augmentation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 4341-4362. Association for Computational Linguistics.
Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. 2019. Curriculum learning for domain adaptation in neu- ral machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Pupers), pages 1903-1915, Minneapolis, Minnesota. Association for Computational Linguistics.
Jacob Russin, Jason Jo, Randall O' O'Reilly, and Yoshua Bengio. 2020. Compositional generalization by fac- torizing alignment and translation. Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics: Student Research Workshop, pages 313-327, Online. Association for Computa- tional Linguistics.
Paul Smolensky. 1988. The constituent structure of connectionist mental states: A reply to fodor and pylyshyn. Souther Journal of Philosophy, 261
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi. Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online. Association for Computa- tional Linguistics.
Mariya Toneva, A Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey Gordon. 2019. An empirical study example forgetting during deep neural network learning. In

---

A Supplementary
spectively). In these equations, denotes the in- stance, E represents the total number of epochs, x; is the input sequence, Øfe) corresponds to the set of model parameters at epoch e. Additionally, Vi, corresponds to the i-th token of the ground truth output sequence y, of length T, s/(e) refers the predicted sequence generated by the model param- eters at epuch e.
A.1 Reproducibility
We use the experimental setup created in Csordás et al. (2021) for vanilla Transformers and modify it calculate and store training dynamics, and implement curriculum learning framework. After storing training dynamies such as perplexity, CHIA, and BLEU, we choose a subset with a criteria. As we choose random subsets as a baseline, we specify the seed during this process. For the Bi-LSTTM with attention experiments, we adopt the setup created in Patel et al. (2022).
A.3 Additional Experiments
While our primary experiments focus on train- ing Transformers on the CFQ and COGS datasets, we conduct supplementary experiments with Bi- LSTM with attention model the COGS task and Transformer model on two SMCS splits. This ap- proach allows us to examine the transferability of our results from synthetic datasets and the Trans- former to other architectures and natural datasets.
As mentioned in the paper, we used models and hyperparameters from Csordás et al. (2021) ease computational constraints and use ini- tial strong random baseline (see Table 4). Accu- racy is calculated on the sequence level, mean- ing that all tokens in the output sequence should match all tokens in the gold sequence while pre- serving the sequence. We use the NLTK BLEU-4 score as the BLEU metric. We specifically use irboothinguunction me tho4 for smoothing and auto_reweigh is set to True as some examples are shorter than words. For the CFQ dataset, we applied the same preprocessing as in Csordás et al. (2021), which is used in Keysers et al. (2020) as well. For COGS, no dataset preprocessing was used.
For the SMCS 16 and 32 splits, we see simi- lar trend compared CFQ and COGS results (see Table and 3). The performance of hard-to-ledd -ena-to-laarn subsets exceeds the original dataset performance consistently. Even if hard-to- tolar examples con- sisted of manual annotation errors in Swayamdipta et al. (2020), training models on hard-to-learn sub- sets improve performance for the SMCS splits. However, this finding does not necessarily trans- late to training models errors result in better performance. Rather, the contribution of hard- to-learn subsets significantly obscures any perfor- mance degradation that may occur from annotation errors. Similarly, easy-to-learn subsets perform the worst among all of the subsets. Although hard- to-lear subsets perform the best among the other subsets, the ranking between metrics is more fluid compared to the CFQ and COGS results. While hard-to-learn (Inv PPL) outperform other subsets persistently in the CFQ and COGS results (Table 3), hard-to-learn (BLEU) and hard-to-leary (CHIA) are the best performing subsets in SMCS 16 and 32 subsets respectively.
A.2 Variability Equations
Equations for variability calculations, when CHIA, inverse perplexity, or BLEU measures are used, are shown from equations 6 to 8 respectively.
Σ
T Et= Pae (Vū. xx)) - 1a.
02
E
(6)
Ee=1
E
[1t=1
V
D&c
(yft
Xi)
Table 6 shows the Bi-LSTM with attention per- formance on the COGS dataset. Surprisingly, the (7) Bi-LSTM performance is much worse than the Transformer performance. Nonetheless, these re- sults are consistent with the results in the original COGS dataset (Kim and Linzen, 2020).
0
DE
(B (e) yy)
ps)
0/
Similar the SMCS experiments, the Bi-LSTM experiments support our primary findings. Training models on hard-to-lean subsets continuously out- perform training on the full dataset. Compared to
E
where fi (confidence) values are calculated as shown in the main paper (see Equations 3-5, re-

---

Dataset
dmodel
d
Nhead
Blayer
batch size
learning rate
warmup
scheduler
Pprram
CFQ
128
256
16
1024
0.9
4000
Noam
685k
COGS
512
512
4
2
128
10
9.3M
Table 4: Hyperparameters and number of parameters for each task. Feedforward size is denoted as dyz. Only CFQ batch size is changed from Csordás et al. (2021) (4096 1024).
SMCS 16
SMCS
Length
Word rarity
Inv PPL
CHIA
BLEU
Inv PPL
CHIA
BLEU
random
13.56 27.78
3,97 3.44
easy-to-learn
0.0
0.0
0.0
0.1
0.0
1.5
BLEU CHIA Iny Lny 12L
easy-to-learn
11.86 24.55
3,99 / 3.41
ambiguous
2.0
2.3
7.0
7.8
11.6
hard-to-learn
4.5
6.8
16.8
17.5
15.9
ambiguous
11.51 23.64
4.06 3.51
hard-to- turd-to-learn
15.82 32.13
3,95 3.54
random
0.4
0.4
0.4
5.9
5.9
5.9
100% train
4.2
15.6
easy-to-learn
10.91 22.34
4.02 3.43
ambiguous
13.07 27.86
3.94 3.50
Table 5: Accuracy results for the SMCS 16 and 32 splits. Models are trained on different 50% subsets of the train data instead of the full train set. The best performing subset is given in bold. Training models only on hard- to-learn samples outperforms using 100% train data.
hard-to-learn
16.40/
3.93 / 3.53
easy-to-learn
/2.118
4.02 / 3.44
ambiguous
12.78 25.11
3,96 3.47
hard-to-leat
16.23 / 33.49
3.94 3.52
Inv PPL
CHIA
BLEU
Table 7: Statistics about the subsets of the CFQ dataset on 33% selected instances based on Inv PPL, CHIA and BLEU measures. We report average input/output length and word rarity. Statistics are averaged over 3 runs.
easy-to-leam
2.7
8.3
9.9
ambiguous
14.2
2.8
11.9
hard-to-learn
16.6
15.7
20.2
random
9.9
100% train
13.7
for sequence i, and f(ū) denotes frequency of gold token v Table 7 presents these statistics for the CFQ dataset, and reveals interesting patterns. Among these different subsets, the hard-to-learn subsets show longer input and output lengths com- pared to all other splits. Conversely, both ambigu- ous and easy-to-learn samples tend to be shorter in length compared to the randomly selected sam- ples. Analyzing word rarities, we observe that the hard-to-leary subsets have lower input rarity but higher output rarity compared to the random subset. On the other hand, the easy-to-leard and ambigu- ous samples show higher input rarity than the ran- dom subset. Notably, the word rarity ambiguous samples surpasses even that of the hard-to-learn samples. These statistics provide valuable insights into the subsets. However, determining whether dataset cartography is solely driven by factors such as length and rarity represents more complex distribution of samples remains a topic for future investigation.
Table 6: Accuracy results for the COGS dataset. Models are trained on different 50% subsets of the train data instead of the full train set. The best performing subset is given in bold. Training models solely on hard-to- learn samples outperforms using 100% train data.
the Transformer performance (Table 3), the maxi- mum absolute performance increase between hard- to-learn subsets and full training decreases by 4%. However, the maximum relative performance in- crease between hard-to-learn subsets and full train- ing increases, showing that dataset cartography im- proves generalization performance architectures other than Transformer.
A.4 Subsets Obtained from Data Maps
We examine four key statistics to gain insights into the nature of the subsets created through data cartography: (1) input length, (2) output length, (3) input word rarity, and (4) output word rarity. Word rarity is calculated the sum of negative log word frequencies normalized with sentence length, as shown in Equation 9. In this equation, T' the sequence length, y is the tt gold token
The statistics of the COGS subsets. as presented in Table 8, show similar patterns to the CFQ sub- sets discussed in Table Specifically, we ob- serve that the hard-to-leam subsets tend to have

---

Length
Word rarity
(1)
An easy-to-learn sample:
random
7.47 43.52
4.34
What did child of MO executive produce, edit, write, direct, and produce
Inv PPL
easy-to-learn
0.59/34.22
4.25 3.24
ambiguous
7.16/ 42.23
4.82 3,41
SELECT DISTINCT ?x0 WHERE ?x0
hard-to-learn
8.83 / 56.62
4.87 /3.7
film. film. directed_br ?x1 ?x0
film. film. edited_bbbee ?x1 ?x0
CHIA
easy-to-learn
6.61 33.78
4.24 3323
film.film.exer executive_prooucce
ambiguous
7.81 48.53
4.90 / 3,46
2x1 ?x0 film. fiwm. www....ooouuueddby////ooooo
hard-to-learn
8.83 / 56.87
4.87 3.48
ns:film. fil.ffiim...... wtt.ff...........m
?x1 wtt.:ppdducciiocoomaaiiiiiiii
BLEU
easy-to-learn
6,27 32.24
4.33 3.27
?x1 ?x0 film. Film.written_br ?x1
ambiguous
8.67 54.92
4.78 / 3,43
ns: nss:fictionll_uiierreee
hard-to-learn
9.08 58.20
4.77 3.45
fictional_charaateer.pareens
Table 8: Statistics about the subsets of the COGS dataset on 33% selected instances hased on Inv PPI., CHIA, and BLEU measures. report average input/output length and word rarity. Statistics averaged over 3 rums.
ns:orraanizatioon.rrgniiizziii
pereet//ss:orrannzzation.
orgaanzatiion.rllltiinnnoo...pp M0
longer samples compared to the ambignous subsets, while the ambiguous subsets are longer than the easy-to-leam subsets. However, unlike the CFQ subsets, the COGS subsets display an interesting characteristic: as the subsets become harder, there is an increase in the presence of rare words both within and outside the dataset vocabulary. This phe- nomenon can be attributed to the larger vocabulary size and smaller dataset size of the COGS dataset, outlined in Table Consequently, the variabil- ity in word usage plays more prominent role in determining the hardness of the data instances in COGS. Therefore, by employing dataset cartog- raphy, we are able to select subsets that exhibit different underlying factors, ultimately leading to dataset-specific improvements in performance.
An ambiguous sample:
What did MO found and MI's female founder found
SELECT DISTINCT 7x0 WHERE ?x0
organization. how.zaaiion.oggaizaaiioo.oooo foniniiiaaiooo.ouderrsess founders ?x1 ?x0
orgamizaiionooogaiiaatioooo organization.....oooo fiuuniiiation.oondersss
MO ?x1 organization. <-
or organization_fouddrr
or organizations_ool founded M1 ?x1
people, person. gender m_02zsn
(3)
hard-to-learn sample:
Was M2 a film producer that employed a spouse of MI, employed MO's executive producer, and employed M4 +
SELECT count (+) WHERE ?x0
1 T
Rarity(i)
Σ
log (vG)
(9)
film. film.producer.e+
MO 2x1
poople.perrson.spouse.
A.5 Subset Examples
ns:peoplle.marriage.e spouse|
In the following, we randomly sample examples from 5% hardest-to-leam, most ambiguous, or easiest-to-leam examples. We show examples based on BLEU measure for the CFQ dataset (ex- amples (1), (2), and (3)), and based on Inv PPL measure for the COGS dataset (examples (4), (5), and (6)), for brevity. While these examples are too small for any inference, we see that these examples reflect subset statistics mentioned in Tables and 8.
ns:ffictiionll.nivvrre.. fictiooallcchaaatter..aaat
ms:fiictional_umieersee.
marriage_of_fictionnl...
characters. M1 FILTER (
?x1 != M1 M2 Film.produce
M2 business. employer employees/
ns:business. person ?x0 business. wtt.:sssemmlllye....ele.eee.ee ns:business. employment_ttememreee
sss:buuiinssseeeloooeeeettteeeee employmentt_eenuee
CFQ Subset Samples:

---

person ?x1 M2
increase in errors for the shortest samples. This can be attributed to the fact that the hard-to-learn (Inv PPL) model generates longer outputs than the target for subset of these samples ((7)). This be- havior can be explained by the length bias present in the hard-to-leant subsets, as the models tend to generate longer outputs when encountering in- stances longer than the average length.
business. employer emmployer.employees// employees/ ns:business. hwwp:/eeeeeeeeeeeemnntemmmmm wetlloyment_teuuee..
person M4 }
COGS Subset Samples:
(4) An easy-to-learn sample:
A cake was drawn by Emma
cake ( x
1 AND draw theme ( x 3 x 1 )
Was film director MO GOLD: SELECT count
AND draw agent
WHERE {
x 3 Emma )
MO a Film.directe } OUT: SELECT count WHERE { MD
(5) An ambiguous sample:
a film. director MO
The cat wished to sleep
+
film. director. film M2
cat ( x 1) wish agent
x 2 x 1 1) AND wish xcomp
Errors observed in both models exhibit systematic nature. For instance, there are samples where the models fail correctly order the triple sequences, resulting in incorrect output (see Example (8)). An- other common error type involves swapping the argument triple with the 3rd argument ((9), formatted for spacing). is worth noting that these error patterns are not specific to either the hard-to- to- learn or the 100% trained models.
x 2 x 4 ) AND sleep agent
( x 4
(6) A hard-to to-learn sample:
James gave lion cake in the fridge
fridge x 8 ) give agent
( x James ) AND give
recipient x give theme
x ) AND
x 5 AND
(8)
Was MO prequel film -
lion (x (x_3)
3 ) AND cake ( 5
AND cake nmod in x 1
GOLD: SELECT count ?x0 a film.film ?x0
WHERE {
film. film. sequel MO }
OUT: SELECT count WHERE {
A.6 Detailed Error Analysis
7x0 film. sequel MO ?x0 a
To gain further insights into the performance of our model, we conduct a comprehensive manual error analysis on both the CFQ and COGS datasets. Our objective was identify the specific test samples where the model exhibits improved performance af- ter training with selected subset, and to determine the general properties of these samples.
film. film }
Was character M1 director
GOLD: SELECT count (*) WHERE { ?x0
a fictional_universee.ff
fictional_character ?x0
film. director. film M1 }
On the CFQ dataset. Our analysis reveals that the hard-to-learn (Iny PPL) model outperforms 100% trained model, particularly on sentences that are shorter than average or of average length. This observation highlights the effectiveness of dataset cartography in enhancing compositional generaliza- tion, without relying spurious correlations. How- ever, it is important note that the hard-to-learn (Inv PPL) model does demonstrate the same level of improvement in generalizing to longer sen- tences. Figure provides further insights into the performance of the models. We observe a slight
OUT: SELECT (*) WHERE { ?x0 a
fictional_universee
fictional M1
film. www.director.fiii 7x0
On the COGS dataset. In the COGS dataset, each instance belongs
of the 21 generalization categories, such as Passive + Active, where the verb structure is transformed from a passive form to an active form (e.g. "The book was squeezed". "The girl squeezed the strawberry."". This cate-

---

Target Length Analysis
Motivation
1500
Practical
Cognitive
Intrinsic
Fairness
100% Train
C
D O
1250
Hanl-ta-Team Learn
Generalisation type
Brmr Count.
1000
Compo- sitional
Crosy
Cross Language
Cross
Robust-
Structural
Task
Domain
750
D O
Shift type
500
Covariate
Label
Full
Assuned
250
CAO
Shift source
20
40
80
100
Naturally occuring
Partitioned natural
Generated shift
Fully generated
Instance Length
DA
Figure 7: Target length histogram for test errors on CFQ.
Shift locus
Train Test
Finetune truin-test
Pretrain frain Pretrain test
gorization allows explore the changes in accu- racies across different categories when comparing models trained on 33% hard-to-learn subsets and the 100% training dataset (refer to Table 10).
E4O
Table 9: GenBench eval card (Hupkes et al., 2022) of our work
For lav PPL. we see performance increase nearly in all of the categories compared to full training except categories. In 3 of these categories, all models have 0% accuracy. And from the remain- ing 4 categories, in only category the perfor- mance discrepancy is remarkable (Subject + Ob- ject (common noun)). While dataset cartography significantly contributes towards lexical generaliza- tion, the contribution towards structural generaliza- tion remains limited.
pear denser, with data examples concentrated in specific regions. In contrast, the BLEU plots ex- hibit a more widespread distribution, while the Inv PPL plots demonstrate the highest degree of disper- sion. These plots offer interesting insights when comparing the performances of CHIA and Inv PPL measures. As instances in Inv PPL plots are better distributed compared instances in CHIA plots, categories of examples are distinguishable, resulting in CHIA hard-to-learn subsets including ambiguous or even easy-to-leare instances. De- spite their similar underlying mathematical intu- ition, these plots contribute to a better understand- ing of the observed differences.
Among models trained with subsets, Inv PPL outperforms CHII almost all categories, and while overall Inv PPL performs better, the rank- ing between Inv PPL and BLEU is more volatile. Results in Table 10 indicate that the Inv PPL mea- sure can distinguish informative examples better compared to the CHIA measure. Moreover, differ- ent confidence measures have different character- istics, therefore they can give more importance to measure-specific instances.
A.8 GenBench Eval Card
We provide the GenBench eval card (Table 9) to help centralize the generalization evaluation in ticc-offthh-rrll language models. conduct many experiments in our paper, but we can safely divide them into main categories by their datasets (CFQ. COGS, and SMCS).
A.7 Remaining Cartography Plots
We present the remaining cartography plots for the CFQ and COGS datasets this section. Same as previous plots, we only plot randomly sampled 33% of the training set. For the CFQ dataset, these remaining plots include the Inv PPL plot (Figure 8) and the CHIA plot (Figure 9). Similarly, for the COGS dataset, the remaining plots consist of the BLEU plot (Figure 10) and the CHIA plot (Figure 11).
The motivation beyond our experiments is the same. We hypothesized that we could improve the compositional generalization performance of neural models by harnessing their training dynam- ics construct a smaller training Therefore, our motivation is practical as we aim achieve better generalization and intrinsic as we examine and utilize models' training dynamics at the same time. All of our datasets are compositional gener-
experimennttt
Upon examining these plots, we observe distinct characteristics among them. The CHIA plots ap-

---

hard-to-learn (33% train)
Category
100% % training
Inv PPLCHIA
BLEU
Subject - Object (common noun)
0.45
0.57
0.85
0.61
Subject Object (proper noun)
0.08
0.05
0.16
0.08
Object Subject (common noun)
0.95
0.95
0.97
0.90
Object + Subject (proper noun)
0.54
0.31
0.40
0.41
Primitive noun Subject (common noun)
0.93
0.58
0.92
0.63
Primitive noun Subject (proper noun)
0.90
0.73
0.81
0.44
Primitive noun + Object (common noun)
0.47
0.19
0.14
0.15
Primitive noun Object (proper noun)
0.19
0.27
0.15
0.14
Primitive verb Infinitival argument
0.48
0.37
0.07
0.26
Object-modifying PP Subject-modifjvee PP
0.00
0.00
0.00
0.00
Depth generalization: Sentential complements
0.00
0.00
0.00
0.00
Depth generalization: PP modifiers
0.09
0.07
0.08
0.07
Active > Passive
0.98
0.94
0.89
0.99
Passive - Active
0.72
0.57
0.74
0.40
Object-omitted transitive > Transitive
0.89
0.65
0.81
0.87
Unaccusative Transitive
0.46
0.45
0.49
0.41
Double object dative PP dative
0.55
0.48
0.71
0.54
PP dative Double object dative
0.65
0.42
0.43
0.17
Agent NP > Unaccusative Subject
0.45
0.48
0.39
0.29
Theme NP + -oittt- transitive Subject
0.74
0.74
0.85
0.80
Theme NP + Unergative subject
0.71
0.71
0.77
0.78
Table 10: COGS accuracy by generalization categories. Subsets have 33% of the original dataset size, Each result is averaged over runs.
alization datasets and they display covariate shift. While SMCS is partitioned neutral dataset, CFQ and COGS datasets are fully generated. All of our shift loci are train-test as train Transformer and Bi-LSTM models from scratch.

---

Inverse PPL Date Map
0.1
7500
0.3
5000
0.8
0.6
0.7
0,0075
Confidene lavere FeL
Confidencs
0.9
10000
Donsity
5000
0.4
15000
0.2
Donstty
100000
5000
Variablity
PPL
000000000
Correctness
Figure 8: Data map of CFQ train set for the Transformer model based on Inv PPL measure (converge epoch 20).
The r-axis shows variability y-axis the confidence. The colors shapes indicate correctness.
Map
10000
Denstty
0.8
0.75 Confidence
Confifeuce CHIA
10000
Denatty
0.5
8000
0.025
0.2
Varlability
Silears
033
0.5
Denstyy
10000
0.7
0.8
5000
0.000
0.005
0.010
0.015
0,020
0.025
0.030
0.035
0.000
Variability CHIA
Correciness
Figure 9: Data map of CFQ train for the Transformer model based on CHIA measure (converge epoch 20).
The x-axis shows the variability and the y-axis the confidence. The colors and shapes indicate the correctness.

---

BLEU Data Map
1.0
0.1
6000
Deextty
4000
2000
0.8
Confidence
Confabencs BLED
1.0
6000
Deasity
4000
0.4
0
0.1
Variabiltty
3000
0.2
Density:
2000
2000
385588838
Vartability
BLEU
Correctzess
Figure 10: Data map of COGS train set for the Transformer model based on BLEU measure (converge epoch 10).
The r-axis shows variability and the y-axis the confidence. The colors and shapes indicate correctness.
CHIA
0.2
6000
4006
0.5
2000
0.7
0.8
Confislebce
Confelnnes CHIA
7500
0.6
Density
2500
0.00
Variabiliit
3000
Density
2000
1000
0.0
0.00
0.01
0.02
0.08
0.04
0.05
Variability CHIA
Figure 11: Data map of COGS train set the Transformer model based on CHIA measure (converge epoch 10). The x-axis shows the variability and the aaiss the confidence. The colors and shapes indicate the correctness.


Reviews:
[{"review_id": "xdgiOxXQAR", "reviewer": "Reviewer_1Pgd", "summary": "", "questions": "A. There are a few literatures that study the impacts of example selections on compositional generalization in the in-context learning setting, e.g. Levy et al, Gupta et al. Although these are technically different from what was studied in this paper, I wonder if the authors have an intuition or have investigated whether sampling hard-to-learn examples could also be beneficial for an in-context learning setup.\n\nB. Have the authors considered non-synthetic compositional generalization datasets? It would be nice to see a study on those, even just for small datasets such as GeoQuery.\n\nC. Do the authors have an explanation about the inconsistency of CL performance even within the dataset? For example, “no curriculum” significantly outperforms others in Figure 3 but not for Figure 4 and COGS dataset.  \n\nGupta et al. Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their invaluable feedback and insightful suggestions. We greatly appreciate the acknowledgment of our approach in tackling the compositional generalization problem through the lens of training dynamics. Additionally, we are thankful for the positive remarks concerning our experimental setup's comprehensiveness, the clarity of our visualization, and the overall quality of the writing.\n\nAddressing the comments and questions provided:\n\n> The paper only studies one split from each of two synthetic datasets. It’s hard to know whether the conclusions can be translated to other splits and datasets.\n\n> Have the authors considered non-synthetic compositional generalization datasets? It would be nice to see a study on those, even just for small datasets such as GeoQuery.\n\nWe acknowledge this pivotal concern, which we have thoughtfully addressed within the Limitations section of our paper. While we concentrated our efforts on two widely employed compositional generalization datasets, we recognize the importance of ensuring the transferability of our findings to other datasets. To address this, __we conducted experiments on a natural compositional semantic parsing dataset, SM-Calflow-CS [(Yin et al., NAACL 2021)](https://aclanthology.org/2021.naacl-main.225)__, which has been utilized in prior research [(Levy et al., ACL 2023)](https://aclanthology.org/2023.acl-long.78).\n\nDespite time constraints limiting us to experiments on only 2 out of the dataset's 5 splits, we discovered that the insights drawn from these experiments were consistent with the observations presented in our paper. This consistency lends additional support to the robustness of our findings. \n\nThe outcomes of these experiments have been included in the following tables, demonstrating that, consistent with our main paper, hard-to-learn subsets outperform 100% dataset training:\n\n__SM-Calflow-CS - 16 Split (50% train set)__\n| | InvPPL | CHIA | BLEU |\n|-------|----------|:-------------:|------:|\n|easy-to-learn | 0.0% | 0.0% | 0.0% |\n|ambiguous | 0.0% | 2.0% | 2.3% |\n|hard-to-learn | **4.5%** | **4.5**% | **6.8%** |\n|random | 0.4% | 0.4% | 0.4% |\n|100% train set| 4.2% | 4.2% | 4.2% |\n\n***\n\n__SM-Calflow-CS - 32 Split (50% train set)__\n\n| | InvPPL | CHIA | BLEU |\n|-------|----------|:-------------:|------:|\n|easy-to-learn | 0.1% | 0.0%% | 1.5% |\n|ambiguous | 7.0% | 7.8% | 11.6% |\n|hard-to-learn | **16.8%** | **17.5%** | **15.9%** |\n|random | 5.9% | 5.9% | 5.9% |\n|100% train set| 15.6% | 15.6% | 15.6% |\n\nWe firmly believe that these supplementary experiments bolster the generality of our conclusions, providing a more comprehensive understanding of the dynamics underlying compositional generalization.\n\n> The effectiveness of leveraging dataset cartography for CL is unclear. In most cases, no curriculum appears to perform better or is on par with a strategy that starts the curriculum with hard-to-learn samples.\n\n> Do the authors have an explanation about the inconsistency of CL performance even within the dataset? For example, “no curriculum” significantly outperforms others in Figure 3 but not for Figure 4 and COGS dataset. \n\nWe agree with the reviewer's observation, particularly in the context of the CFQ dataset. While certain improvements are evident in the COGS dataset (Figures 3 and 4), we acknowledge the valid concerns raised about the effectiveness of curriculum learning (CL) on the CFQ dataset. The inconsistency in CL performance indeed stems from the variation in designs of the curricula, as shown between Figures 3-4 and Figures 5-6. \n\nOur methodology employs dataset cartography as the criterion for establishing the unlocking sequence (ranging from easy to hard-to-learn, vice versa, and from most- to least-ambiguous), while the CL framework governs the unlocking pace. This discrepancy is apparent in the exponential (approx. 2% → 4% → … → 100%) and linear (10% → 20% → … → 100%) mechanisms illustrated in Figures 3 and 4, respectively. Moreover, nuances such as arranging mini-batches with similar levels of hardness contribute to the divergent outcomes. Hence, the variability in CL performance emerges from the intricate interplay of these factors, and we intend to clarify this in our revised manuscript.\n\nImportantly, it's worth noting that our primary aim with CL experiments was to offer a more comprehensive analysis of training dynamics in compositional generalization, rather than to introduce a practical CL strategy. The evaluated CL strategies serve to demonstrate the significance of incorporating hard-to-learn, easy-to-learn, and ambiguous samples during training within the context of systematic generalization. We appreciate the reviewer for highlighting this aspect, and we will make the necessary clarifications to convey this point effectively in our paper.\n\n> There are a few literatures that study the impacts of example selections on compositional generalization in the in-context learning setting, e.g. Levy et al, Gupta et al. Although these are technically different from what was studied in this paper, I wonder if the authors have an intuition or have investigated whether sampling hard-to-learn examples could also be beneficial for an in-context learning setup. \n\nFollowing the submission deadline, __we have indeed contemplated potential applications of dataset cartography in an in-context learning setup__. We envision that identifying hard-to-learn examples using a small language model and utilizing them to select in-context examples for a larger language model could be a promising future direction. This idea aligns well with the reviewer's suggestion, and we value their insightful input. We have plans to further investigate this matter in the near future. \n\nAddressing the typo and presentation improvements:\n> L95, L104, L292, L322, etc. \\citet{Swayamdipta et al., 2020}\n\nWe appreciate the reviewer for pointing out these typos, and we corrected them in our revised version. \n\nIn conclusion, the reviewer's insights have been instrumental in revising our paper, and we are deeply appreciative of their thorough assessment. We are dedicated to addressing all the raised concerns and making the necessary improvements to ensure the clarity, rigor, and impact of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Provide an interesting aspect of leveraging training dynamics to improve compositional generalization, which has not been well explored before. \n- The experiments and analyses are comprehensive and well-designed. \n- Well written, easy to follow. The visualizations are clear.", "reasons_to_reject": "- The paper only studies one split from each of two synthetic datasets. It’s hard to know whether the conclusions can be translated to other splits and datasets.\n- The effectiveness of leveraging dataset cartography for CL is unclear. In most cases, no curriculum appears to perform better or is on par with a strategy that starts the curriculum with hard-to-learn samples.", "questions_to_authors": "A. There are a few literatures that study the impacts of example selections on compositional generalization in the in-context learning setting, e.g. Levy et al, Gupta et al. Although these are technically different from what was studied in this paper, I wonder if the authors have an intuition or have investigated whether sampling hard-to-learn examples could also be beneficial for an in-context learning setup.\n\nB. Have the authors considered non-synthetic compositional generalization datasets? It would be nice to see a study on those, even just for small datasets such as GeoQuery.\n\nC. Do the authors have an explanation about the inconsistency of CL performance even within the dataset? For example, “no curriculum” significantly outperforms others in Figure 3 but not for Figure 4 and COGS dataset.  \n\nGupta et al. Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation", "paper_topic_and_main_contributions": "This paper explores how to use dataset cartography to improve compositional generalization. It studies different measures (Inv PPL, CHIA, BLEU) for assessing confidence and variability in semantic parsing tasks. Furthermore, it provides a detailed analysis of the impacts of different types of examples. Finally, the paper uses dataset cartography as a curriculum learning and sampling criterion, and demonstrates its effectiveness in improving compositional generalization.\n\n### Updates after rebuttal\n- Increased \"soundness\" from \"good\" to \"strong\"", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L95, L104, L292, L322, etc. \\citet{Swayamdipta et al., 2020}"}, "extra_text": {}}, {"review_id": "eMd1gFLoQc", "reviewer": "Reviewer_bjGd", "summary": "", "questions": "1. The paper mentions \"While curriculum learning also contributes to performance enhancement, its impact appears to be comparatively smaller compared to the use of dataset cartography for subset selection\" (L491), which figure is this based on?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude to the reviewer for their valuable insights, constructive criticisms, and valuable suggestions. We are pleased to learn that the reviewer recognizes the well-structured nature of our paper and acknowledges the novelty of our approach that challenges established assumptions about curriculum learning.\n\nIn response to the provided comments and questions:\n\n> The scope of this work is narrow, focusing only on compositional generalization on synthetic tasks (and specifically for transformer architectures). Extending the analysis to other types of distribution shifts or a broader range of tasks and models would enhance the paper's impact.\n\nAcknowledging this valid concern, we have extended our evaluation to include experiments conducted on the SMCalFlow-CS dataset  [(Yin et al., NAACL 2021)](https://aclanthology.org/2021.naacl-main.225), a natural dataset utilized for analyzing systematic generalization [(Levy et al., ACL 2023)](https://aclanthology.org/2023.acl-long.78). It is worth noting that due to time constraints, we were able to conduct experiments on only 2 out of the dataset's 5 splits. Nevertheless, our findings from these experiments align harmoniously with the conclusions outlined in our paper, thereby reinforcing the robustness of our findings on real-world datasets as well. The outcomes of these supplementary experiments are detailed below:\n\n__SM-Calflow-CS - 16 Split (50% train set)__\n| | InvPPL | CHIA | BLEU |\n|-------|----------:|-------------:|------:|\n|easy-to-learn | 0.0% | 0.0% | 0.0% |\n|ambiguous | 0.0% | 2.0% | 2.3% |\n|hard-to-learn | **4.5%** | **4.5**% | **6.8%** |\n|random | 0.4% | 0.4% | 0.4% |\n|100% train set| 4.2% | 4.2% | 4.2% |\n\n***\n\n__SM-Calflow-CS - 32 Split (50% train set)__\n\n| | InvPPL | CHIA | BLEU |\n|-------|----------:|-------------:|------:|\n|easy-to-learn | 0.1% | 0.0% | 1.5% |\n|ambiguous | 7.0% | 7.8% | 11.6% |\n|hard-to-learn | **16.8%** | **17.5%** | **15.9%** |\n|random | 5.9% | 5.9% | 5.9% |\n|100% train set| 15.6% | 15.6% | 15.6% |\n\nFurthermore, we expanded our investigation by applying dataset cartography to a Bi-LSTM with attention model on the COGS dataset. The outcomes of these new experiments are given below:\n\n__COGS - Bi-LSTM (50% train set)__\n| | InvPPL | CHIA | BLEU |\n|-------|----------:|-------------:|------:|\n|easy-to-learn | 2.7% | 8.3% | 9.9% |\n|ambiguous | 14.2% | 2.8% | 11.9% |\n|hard-to-learn | **16.6%** | **15.7%** | **20.2%** |\n|random | 9.9% | 9.9% | 9.9% |\n|100% train set | 13.7% | 13.7% | 13.7% |\n\nNotably, these results provide additional confirmation that hard-to-learn subsets consistently yield superior results compared to training on the entire dataset, thereby reinforcing the claims articulated in our main paper.\n\nIn summary, these supplementary results indicate the transferability of our claims to real-world datasets and alternative model architectures. However, we acknowledge the need for further experiments to achieve a more comprehensive and conclusive view.\n\n> Methodologically, this paper primarily relies on applying Swayamdipta et al.'s method, with limited originality.\n\nWe respectfully disagree with this assessment. Our originality lies in the introduction and thorough analysis of novel seq2seq cartography metrics, namely inverse PPL and BLEU metrics. Moreover, we novelly investigate compositional generalization through training dynamics. This unique approach adds depth and insights to our understanding of systematic generalization.\n\n> Empirically, the experiments are restricted to synthetic datasets, limiting the relevance of the findings to real-world scenarios. The conclusions might be specific to these synthetic cases. For example, even though the conclusions made by this paper --- that hard-to-learn samples are more useful whereas Swayamdipta et al shows hard-to-learn samples are mostly labeling errors, it might be simply due to the fact that in this work synthetic datasets are used which doesn't contain labeling errors.\n\nWhile our initial experiments focused on synthetic datasets, we also conducted additional experiments using the real-world SMCalFlow-CS dataset. These supplementary experiments lend support to the applicability of dataset cartography within the context of natural compositional generalization datasets. This indicates that the insights drawn from our work can be extended beyond synthetic datasets. \n\nWhile we concur that hard-to-learn subsets may include labeling errors, as evidenced in (Swayamdipta et al., 2020), our empirical findings show that other properties of hard-to-learn subsets contribute to performance improvement, even in natural compositional generalization datasets.\n\n> In the curriculum learning experiments, the paper lacks detailed analysis to differentiate between the benefits derived from using hard-to-learn samples and those from curriculum learning itself. \n\nWe would like to clarify that curriculum learning mainly outlines the unlocking protocol for the full training dataset. The unlocking order, however, is solely determined by dataset cartography. While Hacohen and Weinshal (2019) employ exponential unlocking, Zhang et al. (2019) follow a linear unlocking strategy, with additional trivialities.\n\n> Removal of the distributional shift from training to test might clarify the curriculum learning effects.\n\nDue to the inherent nature of compositional generalization, a strict distributional shift exists between training subsets and test sets. Test sets consistently involve novel compositions not encountered during training, resulting in an inevitable distributional shift. Thus, we respectfully differ with the reviewer’s viewpoint on this matter.\n\n> The paper mentions \"While curriculum learning also contributes to performance enhancement, its impact appears to be comparatively smaller compared to the use of dataset cartography for subset selection\" (L491), which figure is this based on?\n\nWhile we don’t have a dedicated figure for a direct comparison, through a juxtaposition of Figures 3-4 and Figures 5-6 with Tables 2-3, it becomes apparent that curriculum learning yields a performance improvement of no more than a 4.5% for CFQ and up to 10% for COGS.\n\nRegarding the typos and presentation improvements:\n> Legends in figures 3, 4, 5, and 6 are too small, hindering comprehension of the curriculum learning section.\n\n> L094: Replace \"originally used in (Swayamdipta et al., 2020)\" with \"originally used in Swayamdipta et al. (2020).\"\n\n> L111: Modify to \"quantifying how hard a sequence is, is not straightforward.\"\n\nWe addressed these typographical concerns and presentation issues in the revised version.\n\nIn summary, we extend our deep gratitude to the reviewer for their insightful feedback. We've broadened our analysis to incorporate real-world datasets and alternative model architectures, reinforcing the transferability of our findings. While drawing inspiration from prior work, our approach introduces novel perspectives and metrics, enriching our understanding of compositional generalization. The valuable critiques raised by the reviewer have been instrumental in refining our paper, and we are committed to addressing their suggestions to enhance the quality of our contribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper offers interesting results, challenging common assumptions about curriculum learning and showing that starting with hard-to-learn samples might yield better performance.\n2. The paper is well-written and easy to understand.", "reasons_to_reject": "1. The scope of this work is narrow, focusing only on compositional generalization on synthetic tasks (and specifically for transformer architectures). Extending the analysis to other types of distribution shifts or a broader range of tasks and models would enhance the paper's impact.\n2. Methodologically, this paper primarily relies on applying Swayamdipta et al.'s method, with limited originality.\n3. Empirically, the experiments are restricted to synthetic datasets, limiting the relevance of the findings to real-world scenarios. The conclusions might be specific to these synthetic cases. For example, even though the conclusions made by this paper --- that hard-to-learn samples are more useful whereas Swayamdipta et al shows hard-to-learn samples are mostly labeling errors, it might be simply due to the fact that in this work synthetic datasets are used which doesn't contain labeling errors.\n4. In the curriculum learning experiments, the paper lacks detailed analysis to differentiate between the benefits derived from using hard-to-learn samples and those from curriculum learning itself. Removal of the distributional shift from training to test might clarify the curriculum learning effects.\n", "questions_to_authors": "1. The paper mentions \"While curriculum learning also contributes to performance enhancement, its impact appears to be comparatively smaller compared to the use of dataset cartography for subset selection\" (L491), which figure is this based on?", "paper_topic_and_main_contributions": "This work explores methods to improve the compositional generalization of neural sequence generation models. The primary approach leverages dataset cartography (Swayamdipta et al., 2020), extending its original use in out-of-distribution generalization to compositional generalization under sequence generation settings. By redefining confidence and variability for sequence generation, the paper identifies hard-to-learn, ambiguous, and easy-to-learn data points. Experiments demonstrate improvements in compositional generation using two synthetic datasets. Notably, starting the curriculum with hard-to-learn samples led to better final performance, which contradicts common beliefs about curriculum learning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Legends in figures 3, 4, 5, and 6 are too small, hindering comprehension of the curriculum learning section.\n2. L094: Replace \"originally used in (Swayamdipta et al., 2020)\" with \"originally used in Swayamdipta et al. (2020).\"\n3. L111: Modify to \"quantifying how hard a sequence is, is not straightforward.\""}}, {"review_id": "CqcPc6TVTN", "reviewer": "Reviewer_xSzh", "summary": "", "questions": "1. Why the cartography + curriculum learning could improve compositional generalization? What if we simply train the model with enough time and regularization? What if we use some simple metric to identify the hard example (e.g., training acc) combined with curriculum learning?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply thank the reviewer for their invaluable comments, recommendations, and insightful critiques. We are delighted to receive their recognition of our paper's significance in tackling a critical challenge and enhancing model performance in compositional generalization tasks.\n\nWe offer our responses to your comments and inquiries below:\n\n> The dataset cartography used in this paper seems to be another way to identify the hard examples, and the way authors utilize it is mainly about curriculum learning (easy-to-hard). I didn't fully get how this is related to compositional generalization. The compositionality is not easily shown in the cartography, and we could not simply assume hard examples are those composite data.\n\nThe concept of dataset cartography, as we’ve utilized it, involves arranging training examples on a Cartesian plane based on their hardness and ambiguity levels. This procedure results in the emergence of distinct regions representing easy-to-learn, ambiguous, and hard-to-learn examples. While it's true that the dataset cartography is not directly related to compositionality, our hypothesis is rooted in the idea that certain training examples might hold greater importance than others in achieving better compositional generalization.  Drawing inspiration from Swayamdipta et al. (2020), who revealed that training models on ambiguous subsets enhances out-of-distribution (OOD) generalization, and given that compositional generalization can be seen as an OOD task, we conjectured that dataset cartography might also yield benefits. Although our theoretical understanding is limited, the substantial empirical performance improvements we observed remain noteworthy. While we agree with the reviewer's call for deeper theoretical exploration, we believe our paper provides a comprehensive account of our findings, reserving theoretical analysis for future investigations.\n\n> Why the cartography + curriculum learning could improve compositional generalization? \n\nInitially, our hypothesis leaned toward the idea that ambiguous examples could boost OOD generalization. However, our empirical results shifted this perspective, revealing that hard-to-learn examples contribute more to enhanced generalization accuracy. We speculate that initiating training with hard-to-learn examples prompts the model to frequently encounter these challenging instances during training, leading to better adaptation and performance gains, analogous to training with hard-to-learn subsets.\n\n> What if we simply train the model with enough time and regularization?\n\nOur choice of training setup and repository was inspired by [Csordás et al., EMNLP 2021](https://aclanthology.org/2021.emnlp-main.49), who demonstrated that standard Transformer configurations struggle in compositional generalization. They highlighted that extending training duration and removing early stopping aids in improved generalization. Our experiments adhere to their hyperparameter configurations, ensuring prolonged training with regularization. All models across different datasets underwent the same number of training steps, ensuring fairness in comparison.\n\n> What if we use some simple metric to identify the hard example (e.g., training acc) combined with curriculum learning?\n\nOur approach involves utilizing __the BLEU metric__, which __offers a finer-grained evaluation compared to training accuracy, especially for sequence generation tasks__. Specifically, in our BLEU-based hard-to-learn curriculum learning setup, the model initially encounters the most challenging examples, followed by a gradual unlocking of less complex instances. This rationale extends to both easy-to-learn and ambiguous curriculum learning setups, encapsulating a balanced representation of various example types throughout training.\n\nIn summary, we extend our heartfelt gratitude to the reviewer for their invaluable input, which has significantly enriched our paper. While the relationship between dataset cartography and compositional generalization might not be immediately intuitive, our empirical findings demonstrate the efficacy of our approach in enhancing model performance. By combining dataset cartography and curriculum learning, we offer a unique perspective on addressing compositional generalization challenges, underscoring the potential benefits of focusing on hard-to-learn examples. We appreciate the reviewer's thought-provoking questions and have incorporated their insights to strengthen the clarity and impact of our contribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Compositional Generalization is a key problem that existing LM lacks, and this paper uses dataset cartography to identify those hard examples to improve such accuracy (via subset sampling containing hard examples) with curriculum learning\n\n\n\n(post-rebuttal)\n\nAfter reading the rebuttal, I agree that the finding that hard samples found via cartography can help compositional generalization is interesting, so I'd like to increase the excitement to \"Ambivalent\".", "reasons_to_reject": "The dataset cartography used in this paper seem to be another way to identify the hard examples, and the way authors utilize it is mainly about curriculum learning (easti-to-hard). I didn't fully get how this is related to compositional generalization. The composiontality is not easily shown in the cartography, and we could not simply assume hard examples are those composite data. Please correct me if I'm wrong, but I think authors need to provide more justification how the proposed method could enable models for better \"compositional\" generalization from theoretical perspective.", "questions_to_authors": "1. Why the cartography + curriculum learning could improve compositional generalization? What if we simply train the model with enough time and regularization? What if we use some simple metric to identify the hard example (e.g., training acc) combined with curriculum learning?", "paper_topic_and_main_contributions": "Dataset cartography is technique that quantifies the variability and confidence associated with each training sample.\n\n\nThe paper investigates use dataset cartography to identify a subset of compositional generalization data. The authors further implement dataset cartography as a criterion for curriculum learning, mitigating the need for hyperparameter tuning while delivering superior performance. The enhancement is demonstrated through a 10% increase in accuracy on CFQ and COGS datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "WOukoA9okI", "reviewer": "Reviewer_UCsN", "summary": "", "questions": "- Have you tried out more complex model structures instead of the vanilla transformer architecture? If so, does that change your findings?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their invaluable comments, insightful recommendations, and constructive critiques. We deeply appreciate that they find our paper well-written and our results interesting, with well-described and executed experiments.\n\nIn response to the provided comments and questions:\n\n> The paper yould greatly benefit from a wider range of compared model architectures\n\nRecognizing the significance of this suggestion, we have expanded our model architecture analysis to include the Bi-LSTM architecture.  It's important to note that due to time constraints, we were only able to conduct experiments with the Bi-LSTM model on the COGS dataset. The outcomes of these additional experiments are presented in the following tables:\n\n__COGS - Bi-LSTM (50% train set)__\n| | InvPPL | CHIA | BLEU |\n|-------|----------:|-------------:|------:|\n|easy-to-learn | 2.7% | 8.3% | 9.9% |\n|ambiguous | 14.2% | 2.8% | 11.9% |\n|hard-to-learn | **16.6%** | **15.7%** | **20.2%** |\n|random | 9.9% | 9.9% | 9.9% |\n|100% train set | 13.7% | 13.7% | 13.7% |\n\nThese supplementary experiments strongly indicate that dataset cartography offers significant enhancements in compositional generalization performance across various neural architectures. Notably, the insights drawn from these Bi-LSTM experiments are in alignment with the conclusions outlined in our paper for the Transformer architecture.\n\n> Have you tried out more complex model structures instead of the vanilla transformer architecture? If so, does that change your findings?\n\nOur experimentation has been currently limited to two neural architectures, namely the vanilla Transformer model and the newly added Bi-LSTM model. Although we haven't investigated more complex neural architectures at this stage, our empirical findings strongly suggest that the concept of dataset cartography holds potential for enhancing compositional generalization across a broader spectrum of neural architectures. Although we haven't ventured into investigating more complex structures at this stage, we will certainly highlight this perspective in our paper to provide a clearer insight into our research direction.\n\nRegarding the presentation improvements:\n\n> Figures: Please increase the size of the legends\n\nWe increased the size of the legends to improve readability and clarity. Your feedback is greatly appreciated, and we ensured that the figure legends are more accessible in the revised version of the paper.\n\nTo summarize, we genuinely thank the reviewer for their invaluable input, which significantly enriched our paper and strengthened our claims. While we have not tried more complex neural architectures, our empirical results for the vanilla Transformer and newly-added Bi-LSTM model strongly suggest that dataset cartography is applicable to a broader range of neural architectures. We certainly will highlight highlight this perspective to provide a clearer insight to our research direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Generally well written\n- Well described and conducetd experiments\n- Interesting results from which the community could benefit", "reasons_to_reject": "- The paper yould greatly benefit from a wider range of compared model architectures", "questions_to_authors": "- Have you tried out more complex model structures instead of the vanilla transformer architecture? If so, does that change your findings?", "paper_topic_and_main_contributions": "This paper introduces a method that leverages dataset cartography to improve compositional generalization in Transformer models. By strategically selecting relevant data (easy-to-learn, hard-to-learn etc.), the approach achieves up to 10% accuracy improvement on CFQ and COGS datasets. The findings highlight the potential of dataset cartography in enhancing the model's ability to understand and generate language more effectively.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Figures: Please increase the size of the legends "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores how to use dataset cartography to improve compositional generalization. In short, the paper uses dataset cartography as a curriculum learning and sampling criterion, and demonstrates its effectiveness in improving compositional generalization. 

Overall, I think the paper proposes an exciting direction for the community to look at. On the other hand, there are some drawbacks of the paper: (1)  The scope of this work is narrow, (2) the found hard examples are not necessarily compositional examples. More investigation needs to be done to understand it.

Based on the merits and limitations of the paper, I would recommed acceptance to findings.
Decision: Accept-Findings