Subset sampling strategies
Here, we introduce a series of strategies for sampling from D^{Syn}_{train} oriented for improving compositional generalization. Broadly, we focus on selecting subsets that reflect either high structural diversity, high predictive uncertainty, or both, as these properties have been tied to improvements in compositional generalization in prior work on semantic parsing (e.g., Bogin et al., 2022 ) , an NLP research area where compositionality is well studied.

Random. Our baseline sampling method is to construct \hat{D}^{Syn}_{train} by sampling from the large synthetic training data D^{Syn}_{train} uniformly.

Uniform Morphological Template (UMT). With this method, we seek to improve the structural diversity in our subsampled in synthetic training dataset \hat{D}^{Syn}_{train} . Training on diverse subset is crucial, as the SIGMORPHON 2018 shared task dataset is imbalanced in frequency of different morphosyntactic descriptions (MSDs). 8 8 8 For example, the low-resource Georgian training dataset contains 9 instances of of nouns inflected for PL;ERG , but only one instance of a noun inflected for SG;ERG . These imbalances can pose challenges to the model in generalizing to rarer MSDs. To incorporate greater structural diversity, we employ the templatic sampling process proposed by Oren et al. ( 2021 ) . Specifically, we modify the distribution over MSDs to be closer to uniform in \hat{D}^{Syn}_{train} .

Formally, we sample without replacement from the following distribution: q_{\alpha}(\mathbf{X},\mathbf{Y},\mathbf{T})=p(\mathbf{T})^{\alpha}/\sum_{\mathbf{T}}p(\mathbf{T})^{\alpha} where p(\mathbf{T}) is the proportion of times that \mathbf{T} appears in D^{Syn}_{train} . We consider two cases: \alpha=0 corresponds to sampling MSDs from a uniform distribution ( UMT ), while \alpha=1 corresponds to sampling tags according to the empirical distribution over MSDs ( EMT ).

HighLoss. Next, we employ a selection strategy that selects datapoints that have high predictive uncertainty to the initial model \mathcal{M} . Spurious correlations between substructures (like \mathbf{Y}_{stem} and \mathbf{T} ; Section 3 ) will exist in any dataset of bounded size (Gupta et al., 2022 ; Gardner et al., 2021 ) , and we conjecture that selecting high uncertainty datapoints will efficiently mitigate these correlations.

We quantify the uncertainty of a synthetic datapoint in D^{Syn}_{train} by computing the negative log-likelihood (averaged over all tokens in the the target \mathbf{Y} ) for each synthetic datapoint in D^{Syn}_{train} . Next, we select the synthetic datapoints with the highest uncertainty and add them to \hat{D}^{Syn}_{train} .

To thoroughly demonstrate that incorporating predictive uncertainty is important for yielding training examples that counteract the spurious dependencies in the ground-truth training dataset, we benchmark it against another subset selection strategy LowLoss . With this method, we instead select synthetic datapoints that the model finds easy, i.e., those with the lowest uncertainty scores. We hypothesize this strategy will yield less performant synthetic training datasets, as it is biased towards selecting datapoints that corroborate rather than counteract the spurious correlations learned by \mathcal{M} .

UMT/EMT+ Loss . Finally, we test a hybrid approach containing both high structural diversity and predictive uncertainty by combining UMT / EMT and HighLoss . First, we sample an MSD \mathbf{T} (according to the MSD distribution defined by UMT / EMT) and then select the most uncertain synthetic datapoint for that \mathbf{T} .