TRANSFERRING ACROSS SEMANTIC PARSING TASKS
Year: 2021

Abstract:
Neural network models often generalize poorly to mismatched domains or distributions. In NLP, this issue arises in particular when models are expected to generalize compositionally, that is, to novel combinations of familiar words and constructions. We investigate learning representations that facilitate transfer learning from one compositional task to another: the representation and the task-specific layers of the models are strategically trained differently on a pre-finetuning task such that they generalize well on mismatched splits that require compositionality. We apply this method to semantic parsing, using three very different datasets, COGS, GeoQuery and SCAN, used alternately as the pre-finetuning and target task. Our method significantly improves compositional generalization over baselines on the test set of the target task, which is held out during fine-tuning. Ablation studies characterize the utility of the major steps in the proposed algorithm and support our hypothesis.

heading: 1 INTRODUCTION
text: Recent work has spotlighted significant shortcomings of neural network approaches to NLP in coping with compositional generalization (CG) (Lake & Baroni, 2018; Finegan-Dollak et al., 2018; Keysers et al., 2020; Kim & Linzen, 2020; Shaw et al., 2021). In those studies, the test set combines in unfamiliar ways linguistic elements that may themselves be familiar from the training set; for example, sentences in the test set may be longer than those observed in training, or may use familiar words in syntactic roles in which they did not occur in training. The performance of popular neural architectures in contemporary NLP models such as Transformers drops considerably when they are expected to generalize compositionally. Several approaches have been proposed to address this issue, including specialized architectures with compositional inductive biases (Furrer et al. 2020) and compositional data augmentation (Andreas, 2020), but as of yet the problem is far from solved.
In this paper, we study this challenge from the perspective of transferability of compositional generalization skills: is possible for a neural model to transfer compositional generalization skills acquired from one task to another task that also requires compositional generalization skills?
We ground our inquiries with the semantic parsing tasks on three very different datasets: GeoQuery (Zelle, 1995), COGS (Kim & Linzen, 2020) and SCAN (Lake & Baroni, 2018). For each task, we use existing compositional training/test splits or create new ones. We propose a learning algorithm that can extract a compositional inductive bias from one task — a stage we refer to as pre-finetuning1 — and transfers that bias to a target task, improving the models’ compositional generalization behavior on that task.
To extract the inductive bias so as to transfer, we introduce a new training algorithm DUEL. In summary (cf. Fig 1), DUEL is designed to be compatible with pre-trained neural encoder-decoder models, such as T5 (Raffel et al., 2020). We view the encoder as learning a representation for the inputs and the decoder as “a task head” that is specialized to different tasks. In pre-finetuing, framing the task’s two (compositional) splits as deriving representation from one split for zero-shot learning to the other split, DUEL trains the encoder and the decoder using different splits. In contrast to
1We borrow this terminology from the NLP research community. A large number of research papers have explored the idea of transfer learning, starting from a pre-trained language model, followed by training on a set of pre-finetuning tasks, and then fine-tuning on the target or downstream tasks (Vu et al., 2020; Gururangan et al., 2020; Pruksachatkun et al., 2020; Chen et al., 2020a; Aghajanyan et al., 2021).
using standard supervised learning over both splits as a whole for pre-finetuning, DUEL encourages the encoder to learn to represent the input texts in a way that facilitates greater compositional generalization, and that this transfers across domains and persists through fine-tuning on the target task, as shown by our empirical studies.
The rest of the paper is organized as follows. In §2, we detail our setup. We describe our approach in §4, followed by empirical studies §5. We discuss related work in §6 and conclude in §7.

heading: 2 PROBLEM SETUP
text: Our neural learner is given a pre-finetuning task and a target task. For the pre-finetuning task, we have a training data distribution s and a test/eval data distribution s̃ that may deviate from s. Likewise, for the target task, we have q and q̃. In the case of compositional generalization, the difference between s and s̃ (or q versus q̃) is defined in compositional terms: the ways in which elements combine differ systematically between the two parts of each datasets (Keysers et al., 2020; Kim & Linzen, 2020; Lake & Baroni, 2018; Hupkes et al., 2020).
Our goal is to train a neural model on (s, s̃) and then fine-tune it on q so that the final model performs well on q̃, namely, attaining strong compositional generalization performance on the target task.
Our assumption is that the difference between s and s̃ is similar to the difference between q and q̃, such that an inductive bias acquired from (s, s̃) can be transferred to the target task. A motivating example is that s̃ contains longer texts than s and so does q̃ than q: in this case, we would investigate whether learning to generalize from short to long texts is a transferrable skill.

heading: 3 TASK, DATA SETS AND THEIR COMPOSITIONAL SPLITS
text: Data Splits In this work, we focus on semantic parsing, the task of translating a natural language text into a formal representation of its semantics. Standard data splits for supervised learning partition a dataset randomly into a training set and testing set. As such, the examples in both portions are drawn from the same distribution. By contrast, compositional splits partition a dataset in a way that creates a distributional discrepancy between the training and the test set, such that strong performance on the test set requires compositional generalization from the training set.
Compositional splits are straightforward to create for datasets that are generated from grammars or templates, such as SCAN (Lake & Baroni, 2018) and COGS (Kim & Linzen, 2020), described in detail in the next section. For instance, the templates used for the test set can include specific words in syntactic roles they did not appear in in the training set, or they can nest syntactic constituents more deeply than in the training set, creating differences in lengths of the input texts.
An alternative mechanism creates—based on an existing dataset for which the generative process is not available (for example, real-world user queries)—test sets that require compositional generalization by on splitting the dataset so as to obtain the maximum compound divergence (MCD) between the training and test examples (Keysers et al., 2020). Here, compounds are complex expressions formed by composing atoms, and compound divergence is computed as
DC(V ‖W ) = 1 − C0.1(FC(V ) ‖FC(W )) where F is the distribution over compounds, and Cα(P‖Q) = ∑ k p α k q 1−α k . The intuition behind this method is that given an existing dataset it is difficult to limit the test examples to compounds that are completely absent from the training set, but by making the compound distributions as distinct as possible, we are primarily measuring the model’s performance on compounds are at least very infrequent in training.
We use the MCD split generated for SCAN from Keysers et al. (2020) and the Target Maximum Compound Divergence (TMCD) splits generated for GeoQuery from Shaw et al. (2021). TMCD splits extend the MCD methodology to create compositional splits of non-synthetic datasets and define atoms and compounds based on the known syntactic structure of the output logical forms.
Datasets and Tasks Table 1 summarizes the 5 compositional splits we use in this paper. We also compared with standard splits (std) where the training and test come from the same distribution.
As an example for our setting in §2, taking COGS as our pre-finetuning task and GeoQuery as our target task, we learn from the CG General split of COGS and transfer to the CG General split of GeoQuery. In this example, COGS BASE is the distribution s and COGS CG is the distribution s̃. GEO TMCD1 is our fine-tuning distribution q and GEO TMCD2 is our final test distribution q̃.
COGS (Kim & Linzen, 2020) is a synthetic semantic parsing dataset generated from templates. The inputs are English sentences and the outputs are corresponding logical forms inspired by λ-calculus, e.g., A dog ate the cake→ *cake(x4); dog(x1) AND eat.agent(x2, x1) AND eat.theme(x2, x4). COGS has two components: a training dataset, which we refer to as COGS BASE, and a compositional generalization (CG) dataset COGS CG. Kim & Linzen (2020) show that the performance of neural models trained on COGS BASE degrades significantly when they are applied to COGS CG. COGS CG includes two kinds of CG challenges: lexical and structural. In lexical CG, familiar words need to be interpreted in new syntactic positions; for example, in COGS BASE the word hedgehog may occur only in the subject position, in COGS CG it would need to be interpreted as as object. By contrast, structural CG involves new combinations of familiar syntactic structures. For example, in COGS BASE, prepositional phrases only modify objects (Noah ate the cake on the plate), whereas in COGS CG they modify subjects (The cake on the plate burned). Likewise, in COGS BASE prepositional phrases can only be nested once (Ava saw the ball in the bottle on the table), but in COGS CG they are nested multiple times (Ava saw the ball in the bottle on the table on the floor), creating longer sentences. COGS BASE has 24,155 examples and COGS CG has 21,000 examples, of which 18,000 are instances of lexical CG and 3,000 are instances of structural CG.
The GeoQuery (Zelle, 1995; Tang & Mooney, 2001) is an annotated semantic parsing dataset contains 880 natural language questions about US geography (e.g., What states border Texas?). We use the same pre-processing as in Shaw et al. (2021), replacing entity mentions with placeholders in the Functional Query Language (FunQL; Kate et al. 2005) output representations. The input what states border m0, for example, is associated with the output answer(intersection(state, next_to_2(m0))). Atoms and compounds are defined over the FunQL expressions. We adopt the length and TMCD splits from Shaw et al. (2021). For the length split GEO len, the training set GEO SHORT consists of examples with shorter inputs than the test set GEO LONG. We refer to the training and test set for the TMCD split as GEO TMCD1 and GEO TMCD2. Both splits divide GeoQuery equally into two sets of 440 examples.
The SCAN dataset (Lake & Baroni, 2018) consists of over 20,000 natural language navigation commands and corresponding “action sequences”, e.g. jump twice→ JUMP JUMP. SCAN is not a semantic parsing dataset but is commonly used as a diagnostic dataset for evaluating compositional generalization. We adopt the length split (SCAN len) and MCD split (SCAN cd) introduced by Keysers et al. (2020).

heading: 4 PROPOSED METHOD
text: Intuition and Main Idea As mentioned in §2, our goal is to improve compositional generalization on our target task, where the training distribution is q and the evaluation distribution q̃. The challenge is to leverage the information in s and s̃ during the pre-finetuning stage such that the model, when fine-tuned on q, generalizes better on q̃. This contrasts with a standard supervised learning approach,
which would collapse together the examples from s and s̃ during pre-finetuning, discarding the information about how examples are divided between s and s̃—information that, we hypothesize, can be useful for encouraging compositional generalization on the target task.
We assume a neural encoder-decoder architecture (Cho et al., 2014). Our proposed method, DUEL, iteratively updates the parameters of the encoder on examples from s̃, while keeping the decoder fixed, and then updates the decoder parameters on examples from s, while keeping the encoder fixed. The encoder updates are then retained prior to fine-tuning on q, while the decoder updates are discarded (as the pre-finetuning and the target tasks are different). In contrast to standard fine-tuning, DUEL encourages the encoder to learn to represent the input sequences in a way that facilitates compositional generalization, which we hypothesize will transfer across domains and persist through fine-tuning.
Details For encoder-decoder architectures, the model’s parameters naturally partition into two sets, θ for the encoder and φ for the decoder, where θ parametrizes the representation f(x; θ) of the input x and φ parametrizes the “task head” g(·), which uses that representation to accomplish a task such as classification or semantic parsing. Taken together, the model’s output is p(y|x; θ, φ) = g(f(x; θ);φ). Given a pre-finetuning task where both s and s̃ are given, our goal is to drive the learner to learn the optimal representation and task head parameters (θ∗, φ∗) such that g(f(x; θ∗);φ∗) performs the best on both distributions. To achieve this, we design an iterative dueling game that the learner plays with the two distributions.
Suppose the learner’s parameters are currently (θ, φ). To perform optimally on s̃, we would like to use the task head g(·;φ) without further training it, i.e., perform well in an “zero-shot” setting; this setting directly anticipates how learning would proceed for the target task, where we will be training only on q, without exposure to q̃. Since we are holding the task head g(·;φ) fixed, we need to optimize the representation f(·; θ) such that it models s̃ appropriately. To achieve this desiderata, we update the representation parameters iteratively for at most Tinner steps, according to
θ ← θ + α 1 N ∇θ ∑ (x,y)∈Bs̃ log p(y|x; θ, φ) (1)
where B s̃ is a batch of N samples from s̃ and α is the step size.2
Updates to φ follow the reverse logic. The representation function that results from updating θ defines a prior on how s should be represented. When updating the task head parameters φ, our goal is to have the task head perform optimally conditioned on this prior. Concretely, we hold θ fixed and apply iteratively at most Tinner steps to φ:
φ← φ+ α 1 N ∇φ ∑ (x,y)∈Bs log p(y|x; θ, φ) (2)
where Bs is a batch of samples from s. Crucially, we alternate between these two types of updates, such that s̃ informs the learner about the compositional inductive bias that should be incorporated into the representation f , and s teaches the learner how to use that representation to perform the task (here semantic parsing).
2Eqns. (1-3) only aim at illustrating the computation of gradient. We use the Adam optimizer in the implementation.
Algorithm 1: DUEL Require: Data sets s, s̃; Learning rate α; Batch size N ; Outer loop iterations Touter; Inner loop
iterations Tinner; Inner loop early stopping criteria Tpatience; Outer loop early stopping criteria Tmin;
Initialize model parameters θ, φ i← 0 while i < Touter do
j ← 0 while j < Tinner do
Sample a batch Bs from s Compute loss: L(θ, φ,Bs) = 1N ∑ (x,y)∈Bs − log p(y | x; θ, φ) Update parameters: φ← φ− α · ∇φL(θ, φ,Bs) j ← j + 1 if AccuracyDecreases(θ, φ, s̃, Tpatience) then
Early stop: break end
end k ← 0 while k < Tinner do
Sample a batch Bs̃ from s̃ Compute loss: L(θ, φ,Bs̃) = 1N ∑ (x,y)∈Bs̃ − log p(y | x; θ, φ) Update parameters: θ ← θ − α · ∇θL(θ, φ,Bs̃) k ← k + 1 if AccuracyDecreases(θ, φ, s, Tpatience) then
Early stop: break end
end if k < Tmin then
Early stop outer loop: break end i← i+ 1
end Return: Model parameters θ, φ
We contrast our approach with the standard supervised learning approach, where the two distributions are merged together as s ∪ s̃, ignoring the compositional split:
(θ, φ)← (θ, φ) + η 1 N ∇(θ,φ) ∑ (x,y)∼Bs∪s̃ log p(y|x; θ, φ) (3)
DUEL is used for pre-finetuning. When fine-tuning the model on the target task q, we retain the representation component f(·; θ) and re-initialize the task head g(·;φ). Both θ and φ are then updated, as is standard, to optimize the loss function on q.
DUEL is illustrated schematically in the pre-finetuning panel of Fig. 1. The pseudocode is listed in Algorithm 1. The algorithm contains one outer loop for Touter rounds (with possible early stopping), which alternates between inner loops updating θ and φ. The early-stopping monitoring function AccuracyDecreases takes as arguments the current model parameters θ and φ, a data distribution to use for evaluation (s or s̃), and the maximum patience Tpatience for early-stopping. This function returns true if accuracy on the dataset in question has not improved for Tpatience consecutive steps. To terminate the outer loop (thus, the algorithm), we keep track how many steps the inner update for θ take. If the number steps is smaller than a preset threshold Tmin, we conclude that the algorithm has converged to the desired representation since the difference in representing s and s̃ is small, requiring little adaptation. We do not use the same logic to limit how long it takes to optimize the task head parameters φ, conditioned on the representation, as the goal is to use the derived representation to arrive at a strong performance.

heading: 5 EXPERIMENTS

heading: 5.1 SETTINGS
text: Datasets and Tasks We use the datasets described in Table 1. The target tasks always come from one of the compositional splits: COGS cg, GEO cd, SCAN cd, or GEO len. For the pre-finetuning tasks, we consider two configurations: (1) standard splits for COGS, GeoQuery and SCAN; (2) compositional splits: COGS cg, GEO cd, SCAN cd, SCAN len.
Models We use two neural architectures, both encoder-decoder models based on the Transformer architecture (Vaswani et al., 2017). The first one, which we refer to as BERT2SEQ, uses the BERT encoder BERTSMALL (Devlin et al., 2019), with 4 Transformer layers, to learn the representation f(x; θ), followed by a Transformer-based sequence-to-sequence model with 2 encoder layers and 2 decoder layers, as the task head g(·;φ).3 We initialize f(·) with the pre-trained BERTSMALL (Turc et al., 2019).
We also experimented with the much larger T5-Base model (Raffel et al., 2020). We take the T5 encoder as f(x; θ) and the T5 decoder as the task head g(·;φ); the encoder and encoder have 12 Transformer layers each. We initialize them with the pre-trained T5 model. In both pre-finetuning and fine-tuning, we generate task prompts by prepending a task tag to each input sentence (following Raffel et al. 2020). We provide detailed examples of the task prompts for each task in Appendix B.
When using BERT2SEQ, our main results are obtained by initializing the task heads randomly. For T5, we initialize the task heads with the pre-trained T5 decoder, for both pre-finetuning and fine-tuning.
Baselines We compare DUEL to two baselines: (1) NONE, where we fine-tune on the target task (i.e., q) directly, without pre-finetuning; (2) MERGED, pre-finetuning on s ∪ s̃ using eq. (3) and then fine-tuning on the target task. As mentioned above, MERGED ignores the compositional splits and conducts pre-finetuning using the standard splits (cf. Table 1).

heading: 5.2 DUEL LEARNS TRANSFERABLE COMPOSITIONAL GENERALIZATION INDUCTIVE BIAS
text: Main Results Table 2 shows the results of our experiments taking the CG General splits as the target splits. The metric we report is exact match accuracy over output sequences. For COGS, we report accuracy separately for the lexical and structural generalization cases, as our models’ performance differed dramatically between those two types of cases. On SCAN, the NONE baseline (no pre-finetuning) matches the average accuracy 15.4% reported by Conklin et al. (2021).
Pre-finetuning substantially improves over the baselines, across the board. In particular, pre-finetuning on SCAN cd leads to an accuracy of 57.8% on GEO TMCD2, surpassing — with a neural-only approach — the previous state-of-the-art of 56.6%, which was established by NQG-T5, a hybrid model that combines a grammar-based approach with T5 (Shaw et al., 2021). The detailed comparison to other existing approaches is provided in Appendix D.1
Pre-finetuning on a compositional split is essential Do the improvements we observe in Table 2 specifically reflect the contribution of pre-finetuning on compositional splits, or can they be attributed to pre-finetuning more generally? Table 2 reports the performance of the MERGED baseline, i.e., pre-finetuning on the merged dataset s ∪ s̃, without a compositional split. Comparing to the NONE (no pre-finetuning) rows in Table 2, we observe that pre-finetuning improves performance on the target tasks even without using DUEL or the compositional split. However, we see bigger gains from using DUEL on compositional splits (DUEL rows with CG General category in Table 2), especially for the target tasks GEO cd and SCAN cd. On COGS cg, the gain of using DUEL on compositional splits is less pronounced. This reflects two facts: first, that lexical generalization is not very challenging when the encoder uses a pre-trained model with strong lexical representations, such that performance is close to being saturate even without pre-finetuning; and second, that the structural generalization cases in COGS remain too difficult for purely neural models, even with DUEL.
3The model is larger than the baseline Transformer model from Kim & Linzen (2020), which has the same architecture as g(·) alone. The model trained by Kim & Linzen (2020) does not use pre-finetuning and is outperformed by our model in the same setup from a pre-trained BERTSMALL.
Within Table 2, it is clear that DUEL does not make a major difference from MERGED when restricted to standard splits . This is reasonably expected as the DUEL procedure reduces to the MERGED when the data distribution s and s̃ are exactly the same for the pre-finetuning task (cf. eq.(3) to eq. (1,2)).
In summary, the results so far suggest that DUEL is most effective when used in conjunction with pre-finetuning tasks with compositional splits.

heading: 5.3 WHEN WILL DUEL WORK BEST?
text: We hypothesize DUEL works best when the pre-finetuning tasks and the target tasks share strong similarity in their compositional splits. We gain further insights about how effective DUEL is by restricting our study to a special type of compositional generalization - the CG Length generalization. Table 3 reports the results when the compsitional split in the target task GEO len is to generalize from shorter texts to longer ones. Without pre-finetuning, the accuracy on GEO len is 38.6% with T5-Base, and 16.1% with BERT2SEQ. The state-of-the-art results on this task is 52.2% from the NQG-T5 model (Shaw et al., 2021).
As before, pre-finetuning improves from that baseline. SCAN len with DUEL improves the most for both neural models. Intuitively, this is sensible as SCAN SHORT also consists of examples with shorter texts than those in the test set SCAN LONG: on average, the input lengths are 7.0 versus 8.2 on SCAN SHORT and SCAN LONG respectively, while the output lengths significantly vary from 10.8 to 30.0. In this case, DUEL helps extract the inductive bias for generalizing on length difference.
Our experiments also show that COGS cg works better than SCAN cd. The compositional splits in COGScg are multiple typed and not all about length difference. Nonetheless, in the structural challenge portion of the COGS cg, the average text length is 61 in COGS CG, compared to 22 in COGS BASE. On the other hand, SCAN MCD1 and SCAN MCD2 have average text lengths of 7.0 and 7.7, respectively. The SCAN std, which is the standard split, does not improve as it is not a compositional split.
Also, pre-finetuning on two tasks shows the best performance, which implies the potential of our method to learn a general compositional generalization strategy with multiple pre-finetuning tasks.

heading: 5.4 HOW MUCH CAN WE IMPROVE PERFORMANCE OF COGS BY GENERATING VERY SIMILAR PRE-FINETUNING TASKS?
text: While DUEL performed well on the SCAN and GeoQuery target tasks, performance on COGS was less impressive, especially on the structural generalization cases (Table 2). It has recently been shown that the challenges presented by COGS can be addressed using symbolic methods, albeit ones that are informed by detailed knowledge of the generative process that was used to create the dataset (Liu et al., 2021). Can we make additional progress on this dataset using a generic sequence-to-sequence neural model?
We hypothesized that the pre-finetuning splits we used before fine-tuning on COGS were insufficiently similar to the split required for COGS (unlike, for example, the SCAN length split that correspond to the generalization behavior required for GEO len). To test this hypothesis, as a lower bound on DUEL’s ability to extract inductive bias for compositional generalization, we designed a new experiment such that the pre-finetuning tasks will have the same compositional splits as the COGS.
Concretely, we created ten COGS variants with the same type of splits as COGS cg; the only difference between the variants was that each of them used a different lexicon. To create these variants, we first used the SpaCy Python library to identify 3 part-of-speech classes: PROPN, NOUN and VERB. For each proper noun, we selected 5 different alternatives. For each noun and verb, we selected all the synonyms and antonyms from WordNet that had the same part-of-speech. To create a single COGS variant, for each word in the COGS vocabulary, we built a 1-to-1 replacement mapping by randomly selecting one alternative, and generated new sentences by the mapping. The following pair illustrates the mapping with an example (and more in Appendix C).:
COGS: Emma ate the ring beside a bed. −→ COGS_VAR: Trudy consumed the hoop beside a layer.
The lexicons of the variants and the original COGS were completely disjoint in those part-of-speech classes; other words (in particular function words were the same across all variants. We use COGS_VAR cg to denote the CG split of a single COGS variant. For pre-finetuning, we used 1, 5 or 10 variants.
The results reported in Table 4 suggest that pre-finetuning with the same type of compositional generalization split only leads to a very minor improvement in accuracy, from 4.5% to 5.2%. Note that the pre-finetuning tasks have an average accuracy of 79.3% on the structural generalization portions of those tasks. Our method is thus surprisingly ineffective in this case, even when the pre-finetuning task is very similar in structure to the target task. For future research, we plan to investigate the reasons of the limited improvement and approaches for extracting stronger inductive bias.

heading: 6 RELATED WORK
text: Transfer Learning A large body of work has attemped to leveraged multi-task learning to endow a model with an inductive bias that improves generalization on a main task of interest (Caruana, 1998; Bakker & Heskes, 2003; Raffel et al., 2020), with recent work in NLP sharing our focus on neural networks (Sogaard & Goldberg 2016; Hashimoto et al. 2016; Swayamdipta et al. 2018; for a review, see Ruder 2017). Intermediate training of pre-trained sentence encoders on a task or a set of tasks that are related to the task of interest has been advocated, among others, by Phang et al. (2018)
and Aghajanyan et al. (2021). Gururangan et al. (2020) craft a training pipeline where a pre-trained language model is adapted to domain-specific and task-specific ones. There are also empirical studies of transferrability among tasks (Vu et al., 2020). Our work differs from most of the work in this area in that our goal is to learn to generalize compositionally from a training distribution to a test one; as such we provide the model with a strategic training and test set split drawn from a pre-finetuning task, which together illustrate the desired generalization behavior that we intend for the model to transfer to a target task. This approach bears some resemblance to work on learning-to-learn or meta-learning (Thrun & Pratt, 1998; Gu et al., 2018), and a recent study on learning low-resource tasks (Chen et al., 2020a).
Compositional Generalization A number of approaches have been explored to achieve compositional generalization in neural models for NLP. Specifically, recent work has proposed new or modified model architectures (Li et al., 2019; Russin et al., 2019; Gordon et al., 2020; Liu et al., 2020; Nye et al., 2020; Chen et al., 2020b; Zheng & Lapata, 2020; Oren et al., 2020; Herzig & Berant, 2020; Shaw et al., 2021; Yin et al., 2021). Furrer et al. (2020) compared several of these architectures with general-purpose pre-trained models such as T5. Other work has explored intermediate representations (Herzig et al., 2021) and compositional data augmentation procedures (Andreas, 2020; Akyürek et al., 2020).
Our approach is most closely related to work that applies meta-learning to compositional generalization (Conklin et al., 2021; Oren et al., 2021; Lake, 2019). Conklin et al. (2021) focus on using tasks that are similar to the target task to regularize the learning of neural models. By systematically generating compositional splits on the synthetic dataset SCAN, Oren et al. shows that the success of meta-learning depends on the specific splits that are presented to the learner (Oren et al., 2021). Our work complements these studies by showing that it is possible to transfer an inductive bias learned on one task to a substantially different task.

heading: 7 CONCLUSION
text: In this paper, we studied acquiring compositional generalization skills through a pre-finetuning task, where a compositional split is available, and transferring those skills to a target task where such a compositional split does not exist. We proposed DUEL, a simple method for this purpose. The underlying intuition of this method is that we would like to learn an input representation that incorporates a compositional inductive bias, and train it separately from a task head that performs a specific task. We demonstrated the effectiveness of this method on 3 semantic parsing tasks. In particular, using the COGS and SCAN synthetic tasks as pre-finetuning tasks, we were able to achieve high compositional generalization accuracy on two splits of the non-synthetic GeoQuery dataset: the Target Maximum Compound Divergence split GEO cd and the length split GEO len. For GEO cd, our approach beats the state-of-the-art, established by a hybrid neurosymbolic model (Shaw et al., 2021); for GEO len, we substantially improved over the baselines.
Reproducibility Statement To ensure reproducibility, we describe the datasets and their train/test splits for our experiments in Section 3 and Section 5.1. We explain our algorithm with pseudocode in Section 4 Algorithm 1. We also describe the model architecture and baselines in Section 5.1. Further, we provide detailed hyper-parameters and hyper-parameter tuning methods in Appendix A. We give examples of preprocessed inputs to T5 for each dataset in Appendix B. We will release the code after the anonymity period at https://www.anonymous.com.

heading: Appendix
text: In this supplementary material, we provide details omitted in the main text. The content is organized as what follows:
• Section A. Implementation details and optimization hyper-parameters. • Section B. Examples of T5 prompt for each of the task in our experiments. • Section C. Examples of COGS_VAR vs. COGS. • Section D. Additional experimental results and ablation studies of DUEL.
– Comparison with the existing works – A hypothesis on what learned by DUEL transfers between datasets – Ablations on the design of DUEL – The standard deviation of DUEL
A IMPLEMENTATION DETAILS

heading: A.1 HYPER-PARAMETERS IN FINE-TUNING
text: First, we list the hyper-parameters in fine-tuning for each model and task.
For COGS, we fine-tune for a maximum of 30k steps with a learning rate of 1e−4. Following Kim & Linzen (2020), we evaluate every 500 iterations on the dev set of COGS and early-stop if the accuracy on the dev set has not improved for 5 consecutive step. The batch size is 128 on T5-base and 512 on BERT2SEQ.
For GeoQuery, it has no dev set. Therefore, we tune hyper-parameters on the standard split of the GeoQuery dataset. We then use the same hyper-parameters for the GEO cd and GEO len splits. On T5-base, we fine-tune for 5k steps with a batch size of 256 and a learning rate of 1e−4. On BERT2SEQ, we fine-tune for 3k steps with a batch size of 512 and a learning rate of 2e−5.
For SCAN, likewise, we tune hyper-parameters on the standard split of it. We then use the same hyper-parameters for the SCAN cd and SCAN len splits. On T5-base, we fine-tune on for 3.5k steps with a batch size of 128 and a learning rate of 1e−4. On BERT2SEQ, we fine-tune for 2.5k steps with a batch size of 512 and a learning rate of 1e−4.
We use the Adam optimizer with weight decay. We take the label smoothing (Müller et al., 2019) with weight factor 0.1 on BERT2SEQ but not T5-base.
Next, we provide the computation time for fine-tuning on 1 TITAN Xp GPU. For COGS, the average fine-tuning time is about 24 hours on T5-base and 4 hours on BERT2SEQ. For GeoQuery, about 2 hours on T5-base and 0.4 hour on BERT2SEQ. For SCAN, about 3 hours on T5-base and 0.5 hours on BERT2SEQ.
A.2 VALIDATION DATASET FOR HYPER-PARAMETER TUNING
As we train on the whole dataset with DUEL during pre-finetuning, we are not able to directly tune hyper-parameters for DUEL. In order to validate the learned representation is good at OOD generalization from s to s̃ without accessing the test data in s̃, we hold out a validation dataset for each task from s̃. We then take the pre-finetuned model, re-initialize the task-head, fine-tune it on s and test on the hold-out set of s̃. We select the hyper-parameters of DUEL based on the test accuracy on the hold-out set.
For COGS, we create a validation dataset by holding a subset of 3000 examples from COGS CG. Likewise, the validation datasets of GeoQuery and SCAN hold out 20% of the data from GEO TMCD2 and SCAN MCD2, respectively.

heading: A.3 HYPER-PARAMETERS IN PRE-FINETUNING
text: In pre-finetuning, we choose the maximum outer loop iterations Touter as 10, the maximum inner loop iterations Tinner as 30k. For efficiency, we evaluate every 500 iterations and set the early-stopping
patience for the inner loop as 1. We select a learning rate α from [1e−5, 2e−5, 1e−4] for all tasks and all models.
The batch size is different from each model. For T5-base, we select a batch size of 32 from [8, 16, 32] for all tasks. For BERT2SEQ, we select a batch size of 512 from [256, 512, 1024].
The early-stopping minimum iterations Tmin for the outer loop is different for each model and task. For COGS, Tmin = 3000 on T5-base and Tmin = 2000 on BERT2SEQ. For GeoQuery, Tmin = 1000 on both T5-base and BERT2SEQ. For SCAN, Tmin = 2000 on T5-base and Tmin = 1000 on BERT2SEQ. We use the same optimizer and label smoothing method in fine-tuning as in pre-finetuning.
Next, we provide the computation time for DUEL on 1 TITAN Xp GPU. When pre-finetuning on COGS, the average running time is about 60 hours on T5-base and 24 hours on BERT2SEQ. For GeoQuery, about 24 hours on T5-base and 2 hours on BERT2SEQ. And for SCAN, about 30 hours on T5-base and 3 hours on BERT2SEQ.

heading: B T5 PROMPT DETAILS
text: In this section, we provide examples of T5 prompt for each task in our experiment. Our design of task prompt follows the standard way from Raffel et al. (2020)

heading: B.1 COGS
text: Original input: Emma ate the ring beside a bed .
Prompt input: cogs: Emma ate the ring beside a bed .

heading: B.2 COGS_VAR
text: Original input: The tiddler appreciated that Kim measure that a cooky was slid .
Prompt input: cogs: The tiddler appreciated that Kim measure that a cooky was slid .

heading: B.3 GEOQUERY
text: Original input: what is the largest state capital in population
Prompt input: geo: what is the largest state capital in population

heading: B.4 SCAN
text: Original input: run around right after look opposite left
Prompt input: scan: run around right after look opposite left

heading: C MORE EXAMPLES ON COGS_VAR VS COGS
text: For each COGS example listed following, we give three examples from the corresponding COGS_VAR.
COGS input sentence 1: The sailor dusted a boy .
COGS output program 1: *sailor(x_1); dust.agent(x_2, x_1) AND dust. theme(x_2, x_4) AND boy(x_4)

heading: COGS_VAR input sentences 1:

heading: The crewman scattered a female_child .

heading: The boater dotted a male_child .

heading: The leghorn dotted a son .

heading: COGS_VAR output programs 1:
text: *crewman(x_1); scatter.agent(x_2, x_1) AND scatter.theme(x_2, x_4) AND female_child(x_4)
*boater(x_1); dot.agent(x_2, x_1) AND dot.theme(x_2, x_4) AND male_child(x_4)
*leghorn(x_1); dot.agent(x_2, x_1) AND dot.theme(x_2, x_4) AND son(x_4)
COGS input sentence 2: The cookie was passed to Emma .
COGS output program 1: *cookie(x_1); pass.theme(x_3, x_1) AND pass. recipient(x_3, Emma)

heading: COGS_VAR input sentences 2:

heading: The cooky was faded to Joan .

heading: The cooky was overstepped to Trudy .

heading: The cooky was eliminated to Kim .

heading: COGS output programs 2:
text: *cooky(x_1); fade.theme(x_3, x_1) AND fade.recipient(x_3, Joan)
*cooky(x_1); overstep.theme(x_3, x_1) AND overstep.recipient(x_3, Trudy)
*cooky(x_1); eliminate.theme(x_3, x_1) AND eliminate.recipient(x_3, Kim)
COGS input sentence 3: Zoe thought that a hippo cleaned .
COGS output program 3: think.agent(x_1, Zoe) AND think.ccomp(x_1, x_5)

heading: AND hippo(x_4) AND clean.agent(x_5, x_4)

heading: COGS_VAR input sentences 3:
text: Larry cerebrated that a Hippo housecleaned .
Dana guessed that a hippopotamus picked.
Ronnie retrieved that a hippopotamus make_cleaned .

heading: COGS output programs 3:
text: cerebrate.agent(x_1, Larry) AND cerebrate.ccomp(x_1, x_5) AND

heading: Hippo(x_4) AND houseclean.agent(x_5, x_4)
text: guess.agent(x_1, Dana) AND guess.ccomp(x_1, x_5) AND hippopotamus(x_4) AND pick.agent(x_5, x_4)
retrieve.agent(x_1, Ronnie) AND retrieve.ccomp(x_1, x_5) AND hippopotamus(x_4) AND make_clean.agent(x_5, x_4)

heading: D ADDITIONAL EXPERIMENTAL RESULTS
text: In this section, we provide the experimental results as well as the standard deviation of DUEL omitted in the main text.

heading: D.1 COMPARISON WITH THE EXISTING WORKS
text: Several existing works leverage the specific designs of the datasets to solve SCAN (Andreas, 2020; Li et al., 2019; Russin et al., 2019), most of the improvements do not transfer to non-synthetic tasks. We compare to two methods that solve SCAN but can also transfer to non-synthetic tasks, SpanBasedSP (Herzig & Berant, 2020) and NQG-T5 (Shaw et al., 2021). Also, we compare to a recently proposed model-agnostic meta-learning approach (Conklin et al., 2021).
We use the original released code to run the experiments for SBSP (SpanBasedSP) and MAML. For SBSP, we use the BERT-BASE model as in their work; For MAML, we replace their 2-layer Transformer encoder and 2-layer Transformer decoder with the larger model T5-Base EncoderDecoder for a fair comparison.
DUEL outperforms MAML on SCAN cd and COGS cg, and outperforms SpanBasedSP substantially on GEO cd. As expected, it falls behind SpanBasedSP and NQG-T5 on SCAN cd.

heading: D.2 WHAT TRANSFERS BETWEEN DATASETS?
text: The ability to transfer across datasets is related to what exactly the model is learning by DUEL during the pre-finetuning stage. Our hypothesis is that DUEL might restrict the encoder to learn to map structures compositionally, bits and pieces, instead of overfitting on arbitrary complex “structural noises”. In the light of this hypothesis, the DUEL pre-finetuning regularizes the encoder to represent structural information in a more decomposable way, instead of “contextually interdependent”. Suppose the input sentence is “jump right after walk right”. Ideally, we would want the encoder to pick up the cue that “after” signifies to represent “jump right” and “walk right” separately. However, for an end-to-end model, the representation could be biased to couple these two shorter phrases together to map to the desired output sequence.
We have not come up with a definitive way to test the theory, instead, we added some indirect evidences to support the theory. We compare the input length of correctly predicted examples in GEO len between training with DUEL and MERGED in Table 6. The results indicate that the improvement mostly comes from the longer inputs, which means the encoder is being regularized on the abstract notion of learning better from shorter structures and supports our hypothesis.

heading: D.3 ABLATIONS ON THE DESIGN OF DUEL
text: We evaluate on two ablations of DUEL: (1) DUEL(F), where we keep the encoder fixed in fine-tuning; (2) DUEL(S), where we switch the train and the test component in pre-finetuning. Results are listed in Table 7.
DUEL(F) shows a large performance drop comparing with DUEL, which implies it is beneficial to finetune the encoder together with the decoder.
DUEL(S) shows minor degradation to DUEL when pre-finetuning on SCAN, likely due to its symmetric MCD split. Similarly, the COGS lexical generalization set has the same average input length as the train set and contains all the vocabs. Thus, training set is largely symmetric with its lexical generalization set, which leads to minor change in DUEL(S). However, DUEL(S) has a large gap with DUEL when pre-finetuning on GeoQuery. As the TMCD split only ensures all atoms appear in the train component GEO TMCD1, we find it has about 30% of examples containing words that do not appear in the test component GEO TMCD2. Thus, using GEO TMCD2 to train the decoder and GEO TMCD1 to train the encoder forces the encoder to model those “new” atoms. This likely causes the encoder not learning enough inductive bias, thus incurring a larger drop in performance.

heading: D.4 THE STANDARD DEVIATION OF DUEL
text: In Tables 2 and 3 we report the mean of 3 runs for accuracy of the target tasks when pre-finetuning with DUEL. The standard deviations for these runs are reported in Table 8 and 9, respectively.

Review summaries:
This paper proposes a training procedure for encoder--decoder models (applied to semantic parsing) which aims to improve the models' ability to compositionally generalize (successfully handle novel combinations of words and structures, where combinations were not seen in training). The approach relies on pre-finetuning: training a model on a different dataset than the target dataset that also the requires the same sort of compositional generalization as the target dataset, before then training on the training set of the target dataset and then evaluating zero-shot on the compositional set of the target dataset, in the standard way. In pre-finetuning, the decoder is only updated on the training set while the encoder is updated on the compositional generalization set. The approach is evaluated using two different pre-trained encoder--decoder transformer architectures on three different semantic parsing compositional generalization datasets from past work, where it obtains consistent (albiet somewhat small) improvements over a baseline that pre-finetunes all model parameters, and outperforms a past state-of-the-art model on one dataset.

This paper presents a transfer learning strategy for improving compositional generalization of semantic parsers based on pre-trained language models. Before fine-tuning the model on data from the target domain, the authors propose a pre-finetuning step, where models are trained on compositional splits of data from another source domain, with the goal to transfer the model's learned knowledge about language compositionality during this pre-finetuning step to the final learning stage on the target domain, therefore improving compositional generalization. To this end, the authors propose a pre-finetuning method which encourages the model to discover representations of natural language that are invariant against its compositional structures. This is achieved by iteratively freezing the encoder or decoder modules during pre-finetuning, and training the encoder and decoder modules on compositionally disjoint splits of the source data, such that the encoder learns representations that are robust against distributional shift of language compositionality.  

This paper is focused on the problem of compositional generalization in semantic parsing, and introduces a method called "DUEL", which involves "pre-finetuning" iteratively on compositional train-test splits from other datasets, before transferring to fine-tuning on the training data from the target dataset. The method involves using the compositional train/test split from one dataset, and training their encoder-decoder model iteratively such that the encoder parameters are updated based on the test data from that dataset, and the decoder parameters are updated based on the training data from that dataset. After this "pre-finetuning", the model is fine-tuned on the training data from the target dataset. They find that their model outperforms baselines involving 1) fine-tuning on the target task only, and 2) pre-finetuning on the merged data from the other dataset, without the encoder/decoder split. They find that their method largely does not help with the extremely low numbers on COGS structural items, but the margins of improvement are larger for GeoQuery data and SCAN data, with the authors claiming a new SOTA result on one of the splits for GeoQuery.