Learning to Generalize Compositionally by Transferring Across Semantic Parsing Tasks
Authors: Wang Zhu
Peter Shaw
Tal Linzen
Fei Sha
Venue: ICLR 2022 Submitted (2022)
Keywords: transfer learning
compositional generalization

Abstract: Neural network models often generalize poorly to mismatched domains or distributions. In NLP, this issue arises in particular when models are expected to generalize compositionally, that is, to novel combinations of familiar words and constructions. We investigate learning representations that facilitate transfer learning from one compositional task to another: the representation and the task-specific layers of the models are strategically trained differently on a pre-finetuning task such that they generalize well on mismatched splits that require compositionality. We apply this method to semantic parsing, using three very different datasets, COGS, GeoQuery and SCAN, used alternately as the pre-finetuning and target task. Our method significantly improves compositional generalization over baselines on the test set of the target task, which is held out during fine-tuning. Ablation studies characterize the utility of the major steps in the proposed algorithm and support our hypothesis. 

Under review as a conference paper at ICLR 2022
LEARNING TO GENERALIZE COMPOSITIONALLY BY TRANSFERRING ACROSS SEMANTIC PARSING TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
Neural network models often generalize poorly to mismatched domains or distribu- tions. In NLP, this issue arises in particular when models are expected to generalize compositionally, that is, to novel combinations of familiar words and constructions. We investigate learning representations that facilitate transfer learning from one compositional task to another: the representation and the task-specific layers of the models are strategically trained differently on pre-finetuning task such that they generalize well on mismatched splits that require compositionality. We apply this method to semantic parsing, using three very different datasets, COGS, GeoQuery and SCAN, used alternately the pre- pre-finetuning and target task. Our method significantly improves compositional generalization over baselines on the test set of the target task, which is held out during fine-tuning. Ablation studies characterize the utility of the major steps in the proposed algorithm and support our hypothesis.
INTRODUCTION
Recent work has spotlighted significant shortcomings of neural network approaches to NL.P in coping with compositional generalization (CG) (Lake & Baroni 2018 Finegan-Dollad al. 2018 Keysers et al. 2020; Kim & Linzen| 2020; Shaw et al. (021). In those studies, the test set combines in unfamiliar ways linguistic elements that may themselves be familiar from the training set; for example, sentences in the test set may be longer than those observed in training, or may use familiar words in syntactic roles in which they did not occur in training. The performance of popular neural architectures in contemporary NLP models such as Transformers drops considerably when they are expected to generalize compositionally. Several approaches have been proposed to address this issue, including specialized architectures with compositional inductive biases (Furrer et al. (020)) and compositional data augmentation (Andreas) 2020), but as of yet the problem far from solved.
In this paper, we study this challenge from the perspective of transferability of compositional generalization skills: is possible for neural model transfer compositional generalization skills acquired from one task to another task that also requires compositional generalization skills?
We ground our inquiries with the semantic parsing tasks on three very different datasets: GeoQuery (Zelle 1995), COGS (Kim & Linzen] 2020) and SCAN (Lake & Baroni 2018). For each task, we use existing compositional training/test splits or create new ones. We propose a learning algorithm that can extract a compositional inductive bias from one task- - stage we refer to pre-finetuning and transfers that bias to a target task, improving the models' compositional generalization behavior on that task.
To extract the inductive bias so as to transfer, we introduce a new training algorithm DUEL. In summary (cf. Fig D. DUEL is designed to be compatible with pre-trained neural encoder-decoder models, such as T5 (Raffel et al. 2020). We view the encoder as learning a representation for the inputs and the decoder as "a task head" that is specialized different tasks. In pre-finetuing, framing the task's 's two (compositional) splits as deriving representation from one split for zero-shot learning to the other split, DUEL trains the encoder and the decoder using different splits. In contrast to
'ee borrow this terminology from the NLP research community. A large number of research papers have explored the idea of transfer learning, starting from a pre-trained language model, followed by training on set of freetinttnnin tasks, and then fine-tuning on the target or downstream tasks (Vu et al. 2020
Gururangan
2020 Pruksachatkun al. 2020 Chen et al. 2020a AAhajanyan
2021).

---

Under review as a conference paper at ICLR 2022
using standard supervised learning over both splits as whole for fe-fnnetuuing, DUEL encourages the encoder to learn to represent the imput texts in a way that facilitates greater compositional generalization, and that this transfers across domains and persists through fine-tuning on the target task, as shown by our empirical studies.
The rest of the paper is organized as follows. In 2 detail our setup. We describe approach in
followed by empirical studies $ We discuss related work in @6 conclude in $7
2 PROBLEM SETUP
Our neural learner is given a pre- freetent task and a target task. For the pre- rre-finetuning task, we have training data distribution and test/eval data distribution & that may deviate from Likewise, for the target task, we have and q. In the case of compositional generalization, the difference between and & (or q versus q) is defined in compositional terms: the ways in which elements combine differ
systematically between the two parts of each datasets (Keysers et al.) 2020 Kim & Linzen 2020
Lake & Baroni 2018 Hupkes et al. 2020).
Our goal is to train neural model on (s, 8) and then fine-tune it on q so that the final model performs well on q. namely, attaining strong compositional generalization performance on the target task.
Our assumption is that the difference between s and 8 is similar to the difference between & and q. such that an inductive bias acquired from (s, 3) can be transferred to the target task. A motivating example is that ă contains longer texts than s and so does g than q: in this case, we would investigate whether learning to generalize from short to long texts is a transferrable skill.
TASK, DATA SETS AND THEIR COMPOSITIONAL SPLITS
Data Splits In this work, we focus on semantic parsing, the task of translating natural language text into a formal representation of its semantics, Standard data splits for supervised learning partition a dataset randomly into a training set and testing set. As such, the examples in both portions are drawn from the same distribution. By contrast, compositional splits partition dataset in a way that creates a distributional discrepancy between the training and the test set, such that strong performance on the test set requires compositional generalization from the training set.
Compositional splits are straightforward to create for datasets that are generated from grammars or templates, such as SCAN (Lake & Baroni 2018) and COGS (Kim & Linzen| 2020). described in detail in the next section. For instance, the templates used for the test set can include specific words in syntactic roles they did not appear in in the training set, they can nest syntactic constituenss more deeply than in the training set, creating differences in lengths of the input texts.
An alternative mechanism creates-basee on existing dataset for which the generative process is not available (for example, real-world user queries) test sets that require compositional generalization by on splitting the dataset as obtain the maximum compound divergence (MCD) between the training and test examples (Keysers et al. 2020)). Here, compounds are complex expressions formed by composing atoms, and compound divergence is computed as
Dc(V[[W) = (F)
where FF is the distribution over compounds, and Co = 1- The intuition behind this method is that given an existing dataset it is difficult to limit the test examples to compounds that are completely absent from the training set, but by making the compound distributions as distinct as possible, we are primarily measuring the model's performance on compounds are at least very infrequent in training.
We use the MCD split generated for SCAN from Keysers et al. (2000) and the Target Maximum Compound Divergence (TMCD) splits generated for GeoQuery from Shaw et al. (202T). TMCD splits extend the MCD methodology to create compositional splits of non-synthetic datasets and define atoms and compounds based the known syntactic structure of the output logical forms.
Datasets and Tasks Table Table summarizes the 5 compositional splits we use in this paper. We also compared with standard splits (std) where the training and test come from the same distribution.
2

---

Under review as a conference paper at ICLR 2022
Table I: The datasets and their compositional splits used in this paper. We classify compositional splits into two categories: general CG splits where the training and test sets differ in compound distribution (cd) or a range of compositional properties (cg), and a special type of length CG splits where the training and test sets differ in sentence length (len).
Category
Dataset
Shorthand
Train
Test
COGS
COGS og GEO
COGS BASE
COGS
CG General
GeoQuery SCAN
GEO TMCDI
GEOTMcD2
SCAN od
SCAN MCD!
SCAN Mcn2
CG Length
GeoQuery SCAN
GEO len
GEO SHORT
LONG
SCAN
SCAN SHORT
SCANLONG
As an example for our setting $2 taking COGS as our pre-finetuning task and GeoQuery as our target task, we learn from the CG General split of COGS and transfer to the CG General split of GeoQuery. In this example, COGS pasa is the distribution & and COGSeo c is the distribution 3. GEOTMcD1 is our fine-tuning distribution q and TMCD2 is our final test distribution q-
COGS (Kim & Linzen 2020) is a synthetic semantic parsing dataset generated from templates. The inputs are English sentences and the outputs are corresponding logical forms inspired by X-calculus, e.g., A dog ate the cake - *cake (zz); dog(zz) AND cat.agent( (22, 21) AND eat.theme (Z2, x4).
COGS has two components: training dataset, which refer to COGS BASE; and compositional generalization (CG) dataset COGS ca- Kim & Linzen| (2020) show that the performance of neural models trained on COGS DASE degrades significantly when they applied COGS co- COGS co includes two kinds of CG challenges: lexical and structural. In lexical CG, f'amlliar words need to be interpreted in new syntactic positions; for example, in COGS BASE the word hedgehog may occur only in the subject position, in COGS eg it would need to be interpreted as as object. By contrast, structural CG involves new combinations of familiar syntactic structures. For example, in COGS BaSE: prepositional phrases only modify objects (Noah ate the cake on the plate), whereas in COGS ca they modify subjects (The cake on the plate burned). Likewise, in COGS base prepositional phrases can only be nested once (Ava saw the ball in the bottle the table). but in COGS co they are nested multiple times (Ava saw the ball in the bottle on the table on the floor), creating longer sentences. COGS BASE has 24,155 examples and COGS ca has 21,000 examples, of which 18,000 are instances of lexical CG and 3,000 are instances of structural CG.
The GeoQuery ZZele 1995 Tang & Mooney 2001) is an annotated semantic parsing dataset contains 880 natural language questions about US geography (e.g.. What states border Texas?). We use the same pre-processing in [Shaw al. (2021). replacing entity mentions with placeholders in the Functional Query Language (FunQL:: Kate et al. 2005) output representations. The input what states border m0, for example, is associated with the output answer (intersection
astate, next_to_2 2 (m0) ) ) Atoms and compounds are defined over the FunQL expressions. We adopt the length and TMCD splits from [haw et al. (202T). For the length split GEO len- the training set GEO SHORT consists of examples with shorter inputs than the test set GEO LONG- We refer to the training and test set for the TMCD split as GEO TMCDI and GEO TMCD2. Both splits divide GeoQuery equally into two sets of 440 examples.
The SCAN dataset (Lake & Baroni 2018) consists of over 20,000 natural language navigation commands and corresponding "action sequences", e.g. jump twice - JUMP JUMP. SCAN is not semantic parsing dataset but is commonly used as a diagnostic dataset for evaluating compositional generalization. We adopt the length split (eCAN and MCD split (SCANca) introduced by Keysers et al. 2020).
4 PROPOSED METHOD
Intuition and Main Idea As mentioned in 2 our goal is to improve compositional generalization on target task, where the training distribution is q and the evaluation distribution ğ. The challenge is to leverage the information in s and á during the pre-fine tre-finetuming stage such that the model, when fine-tuned on g. generalizes better on q. This contrasts with standard supervised learning approach,
3

---

Under review as a conference paper at ICLR 2022
Pre-finetuning
Fine-tuning
(x, y) ~ p [(:0]
o ②
Keep
f(::0)
Re-init
g(:
No gradient
(x, 4) ~ p- +(:00
(a, y)
q
f 0
g(: ②
With gradient
Figure 1: The DUEL updates. The parameters in the representation function f'()) and the task head g(-) are updated using different data distributions in pre-finetuning.
which would collapse together the examples from and å during pre- freetiinnn,nng, discarding the information about how examples are divided between and 8-information that, we hypothesize, can be useful for encouraging compositional generalization on the target task.
We assume a neural encoder-decoder architecture (Cho et al. 2014). Our proposed method, DUEL, iteratively updates the parameters of the encoder on examples from while keeping the decoder fixed, and then updates the decoder parameters on examples from 8, while keeping the encoder fixed. The encoder updates are then retained prior to fine-tuning on q. while the decoder updates are discarded (as the pre- finetiintuuigg and the target tasks are different). In contrast to standard fine-tuning, DURL. encourages the encoder to learn to represent the input sequences in way that facilitates compositional generalization, which we hypothesize will transfer across domains and persist through fine-tuning-
Details For encoder-decoder architectures, the model's parameters naturally partition into two sets, f for the encoder and for the decoder, where 9 parametrizes the representation fa;; 0) of the input x and @ parametrizes the "task head" g()), which uses that representation to accomplish a task such as classification or semantic parsing. Taken together, the model's output is p(y ax; 0, Φ) (f 0); Φ).
Given a pre-finetuning task where both 8 and & are given, our goal is to drive the learner to learn the optimal representation and task head parameters (0*, .d") such that gf Ø²)) performs the best on both distributions. To achieve this, we design an iterative dueling game that the learner plays with the two distributions.
Suppose the learner's parameters are currently (0,6). To perform optimally on 3, we would like use the task head g(::)) without further training i.e., perform well in "zero-shot" setting; this setting directly anticipates how learning would proceed for the target task, where we will be training only on g. without exposure q. Since we are holding the task head g(:; Φ) fixed, we need to optimize the representation f(;; such that it models ø appropriately. To achieve this desiderata, we update the representation parameters iteratively for most Timnrr steps, according to
- 0 a V E E
(Z.g)EB log 0gp(ylx;0000)
where Bg is batch of N samples from and a is the step size
Updates to ó follow the reverse logic. The representation function that results from updating & defines a prior on how should be represented. When updating the task head parameters d, our goal is to have the task head perform optimally conditioned this prior. Concretely, we hold fixed and apply iteratively at most Tinerr steps to Φz
Φ - Φ -
N V Σ (x.))EB, log lggp(y
(2)
where B, is batch of samples from s. Crucially, we alternate between these two types of updates, such that á informs the learer about the compositional inductive bias that should be incorporated into the representation f, and teaches the learner how to use that representation to perform the task (here semantic parsing).
Egns.. only aim illustrating the computation of gradient. We use the Adam optimizer in the implementation.
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1: DUEL
Require: Data sets 8, 3; Learning rate a; Batch size N; Outer loop iterations Touer: Inner loop iterations Timer's Inner loop early stopping criteria I patience; Outer loop early stopping criteria
mims
Initialize model parameters 0, ạ
i <0
while i Touter do
j +- 0
while j <
Timeer do Sample batch В, from
Compute loss: C(0, 0, B.) = Lx - logp(y x; 0)
Update parameters: + avvaaVVoCo VoL(0, B²)
j ++
if AccuuracyDecrresse//// 0, T puttience) then Early stop: break
end
end
k x 0
while l: < Tinner do Sample batch B3 from
Compute loss: C(0, o, Bg) = L(gyEEB - logply x; 0;
Update parameters: 0 0 - a VoL(0, Bs)
::
+++1 1
if AccuracyDeecreasee 6. T paticusee then
Early stop: break
end end k:
< Tmin then Early stop outer loop: break
end
i +eitl
end
Return: Model parameters
We contrast our approach with the standard supervised learning approach, where the two distributions are merged together as s U 8, ignoring the compositional split:
( - (0, +
1 V (0,0)
Exeay) Baus fogl(vvz;0,00
(3)
DURL. is used for pre-finetuning. When fine-tuning the model on the target task q, we retain the representation component f(; 0) and re-initialize the task head g(: Both 0 and ø are then updated, as is standard, to optimize the loss function on
DUEL is illustrated schematically in the pre-finetuning panel of Fig. [ The pseudocode is listed in Algorithm The algorithm contains one outer loop for Tower rounds (with possible early stopping), which alternates between inner loops updating 0 and é. The early-stopping monitoring function eccuracyDecreasee takes as arguments the current model parameters 0 and ó, a data distribution to use for evaluation (s or 8), and the maximum patience T puttience for early-stopping. This function returns tuu if accuracy on the dataset in question has not improved for T patience consecutive steps. To terminate the outer loop (thus, the algorithm), we keep track how many steps the inner update for 0 take. If the number steps is smaller than preset threshold T Tmin. we conclude that the algorithm has converged to the desired representation since the difference in representing & and ă is small, requiring little adaptation. We do not use the same logic to limit how long it takes to optimize the task head parameters o, conditioned on the representation, as the goal to use the derived representation to arrive at strong performance.
5

---

Under review as a conference paper at ICLR 2022
5 EXPERIMENTS
5.1 SETTINGS
Datasets and Tasks We use the datasets described in Table D The target tasks always come from one of the compositional splits: COGS eg. GEO cd. SCANca. GeO For the pre-finetuning tasks, we consider two configurations: (1) standard splits for COGS, GeoQuery and SCAN; (2) compositional splits: cg GBO SCAN SCAN len-
Models We use two neural architectures, both encoder-decoder models based on the Transformer architecture (Vaswani et al. 2017). The first one, which we refer to as BERT2SEQ, uses the BERT encoder BERT sMALL (Devlin et al. 2019), with 4 Transformer layers, to learn the representation fæ; 8). followed by a Transformer based sequence-to-sen to-suuuecce model with 2 encoder layers and 2 decoder layers, as the task head g(: We initialize f()) with the pre-trained BERTsMALL Ture et al. 2019).
We also experimented with the much larger T5-Base model (Raffel al. 2020 We take the T5 encoder as f(;; 0) and the T5 decoder as the task head g(: o): the encoder and encoder have 12 Transformer layers each. We initialize them with the pre-trained T5 model. In both pre-finetuning and fine-tuning, we generate task prompts by prepending a task tag to each input sentence (following Raffel al. 2020
We provide detailed examples of the task prompts for each task Appendix B
When using BERT2SEQ, our main results are obtained by initializing the task heads randomly. For T5, we initialize the task heads with the pre-trained T5 decoder, for both pre- preffinetuning and fine-tuning.
Baselines We compare DUEL two baselines: (1) NONE, where fine-tune the target task (i.e., 9) directly, without pre-finetuning: (2) MERGED, pre-finetuning on 8 Us using eq. (3) and then fine-tuning the target task. As mentioned above, MERGED ignores the compositional splits and conducts pre-finetuning using the standard splits (cf. Table
5.2 DUEL LEARNS TRANSFERABLE COMPOSITIONAL GENERALIZATIC INDUCTIVE BIAS
Main Results Table 2 shows the results of our experiments taking the CG General splits as the target splits. The metric we report is exact match accuracy over output sequences. For COGS, we report accuracy separately for the lexical and structural generalization cases, as our models' performance differed dramatically between those two types of cases. On SCAN, the NONE baseline (no pre-finetuning) matches the average accuracy 15.4% reported by CConklin et al. (021).
Pre-find finnttninnnggggg substantially improves over the baselines, across the board. In particular, pre- pre-finetuning on SCANNa leads to an accuracy of 57.8%, GEO rMCD2; surpassing with neural-only approach the previous state-of-the of 56.6%, which was established by NQG-T5, hybrid model that combines grammar-based approach with T5 (Shaw et al., 2021). The detailed comparison to other existing approaches is provided in Appendix D.I
Pre-finetuning on a compositional split essential Do the improvements we observe in Table specifically reflect the contribution of pre-finetuning compositional splits, or can they be attributed to pre-finetuning more generally? Table 2 reports the performance of the MERGED baseline, i.e., pre-fine tre-finetuning on the merged dataset U s, without compositional split. Comparing to the NONE (no pre-finetuning) rows in Table 2 we observe that pre-finetuning improves performance on the target tasks even without using DUEL or the compositional split. However, we see bigger gains from using DUEL on compositional splits (DUEL rows with CG General category in Table 2 especially for the target tasks GEO ed and SCAN ed. On COGS the gain of using DUEL compositional splits less pronounced. This reflects two facts: first, that lexical generalization not very challenging when the encoder uses pre-trained model with strong lexical representations, such that performance is close to being saturate even without pre-fine rreffinetuning; and second, that the structural generalization cases in COGS remain difficult for purely neural models, even with DUEL.
'The model is larger than the baseline Transformer model from Kim & Linzen 2020). which has the same architecture g(-) alone. The model trained by Kim & Linzen (2020) does not pre-finetuning and is outperformed by model in the setup from pre-trained BERTSMALL

---

Under review as a conference paper at ICLR 2022
Table 2: Accuracy on the target tasks after pre-finetuning on other tasks. Note that the MERGED baseline is the same for the Standard and CG General splits as it uses the union of two distributions. For COGS, we report lexical and structural CG accuracy separately (lexical/struuttura))
Model
Pre-fine tuning
Category
GEO
SCAN
COGS
SCAN
COGS
GEO
COOGS cg
GGEO
+SCAN
NONE
50.8/0.2
50.8/0.2
34.3
34.3
1.3
1.3
MERGED
Standard
71.2/0.7
66.9/0.8
35.9
36.1
3.2
3.0
BERT2SEQ
DUEL
Standard
63.4/1.0
61.5/0.3
35.2
35.7
2.9
3.1
DUEL
CG General
75.5/1.1
73.2/1.1
36.9
37.7
4.8
5.5
NONE
93.7/2.5
93.7/2.5
52.5
52.5
15.4
15.4
MERGED
Standard
95.2/4.2
94.3/3.4
52.5
54.3
18.7
16.9
T5-Base
DUEL
Standard
94.9/3.1
94.1/3.2
51.1
51.6
19.2
15.7
DUEL
CG General
95.4/4.5
94.9/3.9
53.8
57.8
20.2
21.3
Table Results on length generalization (GEOshor) GGGRO from different pre-finetuning tasks. Numbers in parentheses show the improvement of DUEL from MERGED. COGS U SCAN denotes pre-finetuning with cOGS and SCAN together with two separate decoders.
Model
Pre-finetue Pre-finetuning SCAN
SCAN
COGS
SCAN
COGS SCAN
NONE
16.1
BERT2SEQ
MERGED
16.1
16.1
16.8
16.1
17.0
DUEL
15.9 (-0.2)
16.8 (+0.7)
17.7 (+0.9)
18.6 (+2.5)
18.9 (+1.9)
NONE
38.6
T5-Base
MERGED
39.1
39.1
40.9
39.1
41.4
DUEL
39.3 (+0.2)
40.0 (+0.9)
43.0 (+2.1)
45.0 (+5.9)
46.4 (+5.0)
Within Table 2 is clear that DUEL does make major difference from MERGED when restricted to standard splits This is reasonably expected as the DUEL procedure reduces to the MERGED when the data distribution s and š are exactly the same for the pre-finetunnng task (ef. eq, 3 to eq. 122
In summary, the results so far suggest that DUEL is most effective when used in conjunction with pre-finetnning tasks with compositional splits.
5.3 WHEN DUEL WORK BEST?
We hypothesize DUEL works best when the pre-finetuning tasks and the target tasks share strong similarity in their compositional splits. We gain further insights about how effective DUEL is by restricting our study to special type of compositional generalization the CG Length generalization. Table the results when the compsitional split the target task GEO len is to generalize from shorter texts to longer ones. Without pre-finetuning, the accuracy on GEO len is 38 38.6% with T5-Base, and 16.1 % with BERT2SEQ. The stae-oofttheart results on this task 52.2% from the NQG-T5 model (Shaw et al. 2021).
As before, finetuning improves from that baseline. SCAN with DUEL improves the most for both neural models. Intuitively, this is sensible as SCAN also consists of examples with shorter texts than those in the test set SCANtoNG: on average, the input lengths are 7.0 versus 8.2 on SCAN sHORt and SCAN long respectively, while the output lengths significantly vary from 10.8 to 30.0. In this case, DUEL helps extract the inductive bias for generalizing on length difference.
Our experiments also show that COGS eg works better than SCAN ed. The compositional splits in COGS.g are multiple typed and not all about length difference. Nonetheless, in the structural challenge portion of the coGS.g. the average text length is 61 in COGS co, compared to 22 in COGS BASE- On the other hand, SCAN Men1 and SCAN Men2 have average text lengths of 7.0 and 7.7, respectively. The SCAN ad, which is the standard split, does not improve it is not compositional split.

---

Under review as a conference paper at ICLR 2022
Table Accuracy of T5-Base on COGS's structural challenge, with different pre-finetunde tasks
Pre-finetuning
SCAN cd
GBO
1XCOGS
VARgg
5XCoGs vARgg
10x COGS VARgg
MERGED
3,4
4.2
4.9
5.1
4.8
DUEL
3.9
4.5
4.9
5.2
5.2
Also, pre-finetuning on two tasks shows the best performance, which implies the potential of our method to learn a general compositional generalization strategy with multiple pre-finetuning tasks.
5.4 How MUCH CAN WE IMPROVE PERFORMANCE OF COGS BY GENERATING VERY SIMILAR PRE-FINETUNING TASKS?
While DUBL performed well on the SCAN and GeoQuery target tasks, performance COGS less impressive, especially on the structural generalization cases (Table has recently been shown that the challenges presented by COGS can be addressed using symbolic methods, albeit ones that informed by detailed knowledge of the generative process that was used to create the dataset (Liu
al. 202T). Can make additional progress on this dataset using generic sequence-to-sequeeee neura model?
We hypothesized that the pre-finetuning splits we used before fine-tuning on COGS were insufficiently similar to the split required for COGS (unlike, for example, the SCAN length split that correspond to the generalization behavior required for Ge To test this hypothesis, as a lower bound on DUEL'S 's ability to extract inductive bias for compositional generalization, we designed a new experiment such that the pre-finetuning tasks will have the same compositional splits as the COGS.
Concretely, we created ten COGS variants with the same type of splits as COGS the only difference between the variants was that each of them used a different lexicon. To create these variants, we first used the SpaCy Python library to identify part-of-speech classes: PROPN, NOUN and VERB. For each proper noun, we selected different alternatives. For each noun and verb, we selected all the synonyms and antonyms from WordNet that had the same part-of-speech. To create single COGS variant, for each word in the COGS vocabulary, we built a 1-to-1 replacement mapping by randomly selecting one alternative, and generated new sentences by the mapping. The following pair illustrates the mapping with an example (and more in Appendix C
COGS: Emma ate the ring beside bed.
COGS_ VAR: Trudy consumed the hoop beside layer.
The lexicons of the variants and the original COGS were completely disjoint in those part-of-speech classes; other words (in particular function words were the same across all variants. We use COGS, VAR eg to denote the CG split of a single COGS variant. For pre-finetuning. we used 5 or 10 variants.
The results reported Table 4 suggest that pre- freffiittnning with the same type of compositional generalization split only leads to very minor improvement in accuracy. from 4.5% to 5.2%. Note that the pre-finetuning tasks have an average accuracy of 79.3% on the structural generalization portions of those tasks. Our method is thus surprisingly ineffective in this case, even when the pre pre-finet feeffinetuning task is very similar in structure to the target task. For future research, we plan to investigate the reasons of the limited improvement and approaches for extracting stronger inductive bias.
RELATED WORK
Transfer Learning A large body of work has attemped to leveraged multi-task learning to endow model with inductive bias that improves generalization
a main task of interest (Caruanal 1998, Bakker & Heskes 2003 Raffel et al. 2020), with recent work in NLP sharing our focus on neural networks (Sogaard & Goldberg 2016 Hashimoto et al. 2016 (Swayamiipta al. [018 for review, see Ruder 201e) Intermediate training of pre-trained sentence encoders on a task or a tasks that are related to the task of interest has been advocated, among others, by Phang et al. (018
set of
8

---

Under review as a conference paper at ICLR 2022
and Aghajanyan al. (2021). Gururangan et al. (2020) craft training pipeline where a pre-trainee rre-trained language model is adapted to domain speciffc and task task- -secifici ones. There are also empirical studies of transferrability among tasks (Vu et al. 2020). Our work differs from most of the work in this area in that our goal is to learn to generalize compositionally from a training distribution to a test one; as such we provide the model with a strategic training and test set split drawn from a pre-finetuning task, which together illustrate the desired generalization behavior that we intend for the model to transfer target task. This approach bears some resemblance work on learning-to-eear or meta-learning (Thrun & Pratt 1998 2020a).
Gu et al. 2018), and recent study on learning low-resource tasks (Chen et al.
Compositional Generalization A number of approaches have been explored to achieve compo- sitional generalization in neural models for NLP. Specifically, recent work has proposed new or
modified model architectures (Li et al. 2019; Russin et al. 2019; Gordon et al. 2020; Liu et al.
2020 Nye et al. 20200] Chen et al., 2020b Zheng & Lapata 2020 Oren et al. 2020 Herzig & Berant
2020 Shaw et al. 2021 Yin et al. 2021). Furrer et al. (2020) compared several of these architectures
with general-purper purpose pre-trate trained models such as T5. Other work has explored intermediate repre-
sentations (Herzig et al. 202I) and compositional data augmentation procedures (Andreas) 2020
Akyüüek al.] 2020).
Our approach is most closely related to work that applies meta-learning to compositional general- ization (Conklin et al.) 2021] Oren et al., [021] Lake 2019). Conklin al. (202T) focus using tasks that are similar to the target task to regularize the learning of neural models. By systematically generating compositional splits the synthetic dataset SCAN, Oren SCAN.Orren
al. shows that the success of
meta-lcarning depends on the specific splits that are presented to the learner (Oren et al. 2021). Our work complements these studies by showing that it possible transfer an inductive bias learned on one task to a substantially different task.
CONCLUSION
In this paper, we studied acquiring compositional generalization skills through a pre-finetuning task, where compositional split is available, and transferring those skills to a target task where such a compositional split does not exist. We proposed DUEL, a simple method for this purpose. The underlying intuition of this method is that we would like to learn an input representation that incorporates a compositional inductive bias, and train it separately from a task head that performs specific task. We demonstrated the effectiveness of this method on 3 semantic parsing tasks. In particular, using the COGS and SCAN synthetic tasks as rrefinetuning tasks, we were able to achieve high compositional generalization accuracy two splits of the non-synthetic GeoQuery dataset: the Target Maximum Compound Divergence split GEO ad and the length split GBO For GEO our approach beats the thtee-of-the-art, established by hybrid neurosymbolic model (Shaw al. 202 for GEO we substantially improved over the baselines.
Reproducibility Statement To ensure reproducibility, we describe the datasets and their train/test splits for our experiments in Section B and Section 5.1 We explain our algorithm with pseudocode in Section 4 Algorithm [ We also describe the model architecture and baselines in Section 5.1 Further, we provide detailed hyper-parameters and ryper-parament
tuning methods Appendix A We give examples of preprocessed inputs to T5 for each dataset in Appendix B We will release the code after the anonymity period at lttttts//// www. anonymous com.
REFERENCES
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning arXiv preprint 100.: 2021.
Ekin Akyürek, Afra Feyza Akyurek, and Jacob Andreas. Learning to recombine and resample data for compositional generalization. ArXiv, abx/2010.03706, 2020.
Jacob Andreas. Good-enough compositional data augmentation. In ACI, 2020.

---

Under review as a conference paper at ICLR 2022
Bart Bakker and Tom Heskes. Task clustering and gating for Bayesian multitask learning. Journal of Machine Learning Research, 4:83-99, 2003. URL http:// /tet:://mmr... jmlr wttpiittt..aaaaattatt..teeeeeeee /
bakkerO3a. html
Rich Caruana. Multitask learning. In Sebastian Thrun and Lorien Pratt (eds.), Learning learn, pp. 95-133, Kluwer Academic Publishers, Boston, 1998.
Xilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer, and Sonal Gupta. Low-resource
domain adaptation for compositional task-oriented semantic parsing. In Proceedings of the 2020
Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5090-5100,
Online, November 2020a. Association for Computational Linguistics. doi: 10.18665./.. 1/2020.
emnlp- mo0p-main.1.... URL https: http::/wwwwwwwwww..ccc.c.. http://ww.........aaaa/aoo////// emn -mm.:/p-m...mmmmmmmaaammm.a -mani
413
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. Compositional generaliza- tion via neural-symbolic stack machines, 2020b.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau. Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7724-1734, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3L15/v1/D14-111 1179. URL
1tt/D14-11799
Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. Meta-learning to compositionally gener- alize. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volune I: Long Papers), pp. 3322-3335, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 han.://////2....lloooogg....... 258
URL http:ss//aclaatthooooooo org/ ac acl-long
Jacob Devlin, Ming-Woo Chang, Kenton Lec, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1/N119-1422, URL |
www. acl www.wwewbbccc wwt.w//wwwebbor///atoo..gog//
1423
Catherine Finegan- Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. Improving text-to-SQL. evaluation methodology. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 351-360, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v 1/P1/ 1033. URL www. www.://w....o///tt/oooooooooo
Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schaerli. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. CoRR, abs/2007.08970 2020. URL
https: org. abs 2007 08970
Jonathan Gordon, David Lopez-Paz, Marco Baroni, and Diane Bouchacourt. Permutation equivariant models for compositional generalization in language. In International Conference on Learning Representations, 2020. URL /ttl/httppas//ppeeevvieee wwwp:/eee/oo//oooooooooooo WWN IVNerFvr
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. Meta-learning for low-resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3622-3631, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/w1/D1S8 1398. URL https w/www.as ac web. www.ee..rggggatooogggg////o/a//e
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't it stop pretraining: Adapt language models to domains and tasks. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.
3342-8360, Online, July 2020. Association for Computational Linguistics. doi: 10. 00.1653//1/20200 acl-main. 740. URL. https::/p// www. a c. web. www..org/.athhh..o hgwettoooogyyyyyy acl mma 740
10

---

Under review as a conference paper at ICLR 2022
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many- model: Growing a neural network for multiple NLP tasks. In NIPS 2016 Continual Learning and Deep Networks Workshop, 2016.
Jonathan Herzig and Jonathan Berant, Span-based semantic parsing for compositional generalization. arXiv preprinl htt::20009060400 2020.
Jonathan Herzig, Peter Shaw, Ming- Wei Chang. Kelvin Guu, Panupong Pasupat, and Yuan Zhang. Unlocking compositional generalization in pre-trained models using intermediate representations. arXiv preprint urXiv:2004.074777 2021.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 04 2020. doi: 10.1613/jair.1..
Rohit J Kate, Yuk Wah Wong, and Raymond Mooney. Learning to transform natural to formal languages. In Proceedings of the National Conference on Artificial Intelligence, volume 20, pp. 1062. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2005.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method on realistic data. In ICLR, 2020. URL htttp::/aaxxv..rrrt pdf
9 912. 097913
Najoung Kim and Tal Linzen. COGS: A compositional generalization challenge based semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9087-9105, Online, November 2020. Association for Computational Linguistics. doi: 10. http:////1122200.mmmmmmmmaaa enthology/2022 wmw.::0.mmmmmmmmm.mm.mm
URL https: www. http:/wwwwww.....////./ org
Brenden M Lake. Compositional generalization through meta sequence-to--sqqueee learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buch E. Fox, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 32, pp. 9791-9801. Curran Asso- ciates, Inc., 2019. URL httpps//pproceeddngg nwwrips cc/papper/019/ELL. (at0ee/eeecc33555777772333333333
pdf
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to toriiceett-seeeee to-sequence recurrent networks. In Proceedings of the 35th International Conference Machine Learning. ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, pp. 2879 2888, 2018
Yuanpeng Li, Liang Zhao, Jianyu Wang, and Joel Hestness. Compositional generalization for primitive substitutions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4284-4293, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D199111 1438. URL https::/wwwwt http://wwwww.......//..// org/ anthology/ org.anthooggyy/ 019-1438
Chenyao Liu, Shengnan An, Zeqi Lin, Qian Liu, Bei Chen, Jian-Guang Lou, Lijie Wen, Nanning Zheng, and Dongmei Zhang. Learning algebraic recombination for compositional generalization, 2021.
Qian Liu, Shengnan An, Jian GuanGG Lou, Bei Chen, Zeqi Lin, Yan Gao. Bin Zhou, Nanning Zheng, and Dongmei Zhang. Compositional generalization by learning analytical expressions Advances in Neural Information Processing Systems, 33, 2020.
R. Müller, Simon Kormblith, and Geoffrey E. Hinton. When does label smoothing help? In NeurIPS, 2019.
Maxwell Nye, Armando LezaraLe Joshua B Tenenbaum, and Brenden M Lake. Learning compositional rules via neural program synthesis. arXiv preprint arXiv:2003.05562, 2020.
11

---

Under review as a conference paper at ICLR 2022
Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, and Jonathan Berant. Improving
compositional generalization in semantic parsing. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pp. 2482-2495, Online, November 2020. Association
for Computational Linguistics. doi: 1/21//////220000.iiinniggg/g
URL. https
77 acw.aalllal. wwt.w//ww.wwggg.ooo/eooooooo
win.iniiiinnnnniii..oonoooo moddinng-emmii 225
findi
Inbar Oren, Jonathan Herzig, and Jonathan Berant. Finding needles in a haystack: Sampling structurally-diversc training sets from synthetic data for compositional generalization, 2021.
Jason Phang, Thibault Févry, and Samuel Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv: 1.01088, 2018.
Yada Pruksachatkun, Jason Phang. Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. Intermediate- transfer learning with pretrained language models: When and why does work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5231-522444 Online, July 2020. Association for Computational Linguistics. doi: 10. 10.188533v1/2022... 2aw.00000.--mmii4455551 main.467. URL
https: TttEp::/www.aclweet wwwweb...w...................... org/
main. 467.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
sttp:://mm. wwtgg/aappeeee
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint 10X5v01706.05098, 2017.
Jake Russin, Jason Jo. Randall C O'Reilly, and Yoshua Bengio. Compositional generalization in deep seq2seq model by separating syntax and semantics. arXiv preprint 1X04.0900000700 2019.
Peter Shaw, Ming-Wei Wei Chang, Panupong Pasupat, and Kristina Toutanova. Compositional gen- cralization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 922-938, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021 lal.acllloon...7 URL htt.::::///laatthooooo.cooo acl--
Anders Sogaard and Yoav Goldberg. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 231-235, Berlin, Germany, August 2016. Association for Computational Linguistics.
Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. Syntactic scaffolds for semantic structures. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3772-3782, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DI. 1412. URL
https: www www.wwlwbbb.......... www.ee...gggaagee.e wttp//athhooggyoleeeeeeoooeo
Lappoon R Tang and Raymond Mooney. Using multiple clause constructors inductive logic
programming for semantic parsing. In European Conference on Machine Learning, pp. 466-477. Springer, 2001.
Sebastian Thrun and Lorien Pratt. Learning to learn. Kluwer Academic Publishers, 1998.
Iulia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR, abs/ 0ass/1908.099 908.00988. 2019, URL
tittp://arxite www.ivvooo orgg abs 908. 08962
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Curran Associates, Inc., 2017.
Processing Systems 30, pp.
12

---

Under review as a conference paper at ICLR 2022
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit lyyer. Exploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 7882-7926, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.11653//1/2200.emmllll mannllp-mi..633.. URL https:: /wwp://wwww..m. ac web. org/ anthology
2020. emn main.
Pengcheng Yin. Hao Fang. Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, and Jacob Andreas. Compositional generalization for neural semantic parsing via span-level supervised attention. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2810-2823, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021 waaaacllmii.2211 URL http:ttppss haatteee//allatttoooooo.oo/////p
naacl-main. in 225.
John M. Zelle.
Using Inductive Logic Programming to Automate the Construction of Natural Language Parsers. PhD thesis, Department of Computer Seiences, The University of Texas at Austin, Austin, TX, 1995. URL hwtp::ttpp::w...cc utexas. edu/users/ai -alilaabblleeeeee
phd95
Hao Zheng and Mirella Lapata. Compositional generalization via semantic tagging. arXiv preprint arXiv:2010./188 18888, 2020.
13

---

Under review as a conference paper at ICLR 2022
Appendix
In this supplementary material, we provide details omitted in the main text. The content is organized as what follows:
Section
Implementation details and optimization hyper paper--aramtters..
Section
Examples of T5 prompt for each of the task in experiments.
Section
Examples of COGS_VA VAR vs. COGS.
Section
D
Additional experimental results and ablation studies of DUEL.
Comparison with the existing works
A hypothesis on what learned by DUEL transfers between datasets
Ablations on the design of DUEL
The standard deviation of DUEL
A IMPLEMENTATION DETAILS
A.1 HYPER-PARAMETTE IN FINE-TUNING
First, we list the syper-parameters in fine-tuning for each model and task.
For COGS, we fine-tune for a maximum of 30k steps with learning rate of 1e le 4 Following Kim & Linzen 22020) evaluate every 500 iterations on the dev set of COGS and early-stop if the accuracy on the dev set has not improved for 5 consecutive step. The batch size 128 on T5-base and 512 on BERT2SEQ.
For GeoQuery, has no dev set. Therefore, we tune hyper-parameters on the standard split of the GeoQuery dataset. We then use the same hyper-parameters for the GEO ed and GEO len splits. On T5-base, we fine-tune for 5k steps with a batch size of 256 and a learning rate of le-4, On BERT2SEQ, fine-tune for 3k steps with batch size of 512 and learning rate of 2e
For SCAN, likewise, we tune parameparameters on the standard split of it. We then use the same hyper-parameters for the sCAN and SCAN Jen splits. On T5-base, fine-tune on for 3.5k steps with a batch size of 128 and learning rate of le" -4, On BERT2SEQ, we fine-tune for 2.5k steps with batch size of 512 and learning rate of le-4
We use the Adam optimizer with weight decay. We take the label smoothing Müller et al. with weight factor 0.1 on BERT2SEQ but T5-base.
2019
Next, we provide the computation time for fine-tuning on I TITAN Xp GPU. For COGS, the average fine-tuning time about 24 hours on T5-base and 4 hours on BERT2SEQ. For GeoQuery, about hours on T5-base and 0.4 hour on BERT2SEQ. For SCAN, about 3 hours on T5-base and 0.5 hours on BERT2SEQ.
A.2 VALIDATION DATASET FOR
TUNING
As we train on the whole dataset with DUEL during pre-finetuinng, we are not able to directly tune saper-parammters for DUEL.. In order validate the learned representation is good at OOD generalization from to š without accessing the test data in hold out validation dataset for each task from We then take the pre-finetuned model, re-initialize the task-head, fine-tune it on s and test the hold-out set of 3. We select the on the hold-out set.
of DUEL based on the test accuracy
For COGS, create validation dataset by holding subset of 3000 examples from COGS co. Likewise, the validation datasets of GeoQuery and SCAN hold out 20% of the data from GEO GEOTmcD2 and SCAN MCD2, respectively.
A.3 HYPER-PARAMETER IN PRE-FINETUNING
In pre- preffinetunig, we choose the maximum outer loop iterations Toater as 10, the maximum inner loop iterations Timeer 30k. For efficiency, evaluate every 500 iterations and set the early-stopping
14

---

Under review as a conference paper at ICLR 2022
patience for the inner loop 1. We select learning rate from [le- 2, 1e 1lee" all models.
for all tasks and
The batch size is different from each model. For T5-base, select batch size of 32 from [8, 16, 32] for all tasks. For BERT2SEQ, we select a batch size of 512 from [256. 512, 1024].
The early-stopping minimum iterations Tiin for the outer loop is different for each model and task. For COGS, minn 3000 on T5-base and Timn = 2000 on BERT2SEQ. For GeoQuery, min = 1000 on both T5-base and BERT2SEQ. For SCAN, min 2000 T5-base and mii 1000 BERT2SEQ. We use the same optimizer and label smoothing method in fine-tuning as in pre-finetuning.
Next, provide the computation time for DUEL COGS, the average running time is about 60 hours on T5-base and 24 hours on BERT2SEQ. For GeoQuery, about 24 hours on T5-base and 2 hours on BERT2SEQ. And for SCAN, about 30 hours on T5-base and 3 hours on BERT2SEQ.
TTTAN Xp GPU. When pre-finetuning
B T5 PROMPT DETAILS
In this section, we provide examples T5 prompt for each task in our experiment. Our design of task prompt follows the standard way from Raffel al. (2020)
B.1 COGS
Original input:
Emma ate the ring beside bed
Prompt input: cogs:
Emma ate the ring beside a bed
B.2 COGS_ VAR
Original input: The tiddler appreciated that Kim measure that a cooky was slid
Prompt input: cogs: The tiddler appreciated that Kim measure that cooky was slid.
B.3 GEOQUERY
Original input:
what is the largest state capital in population
Prompt input: geo: what is the largest state capital in population
B.4 SCAN
around right after look opposite left Prompt input:
Original input:
scan:
run around right after look opposite left
C MORE EXAMPLES ON COGS_ VAR VS COGS
For each COGS example listed following, we give three examples from the corresponding COGS_VA VAR.
COGS input sentence 1: The saile for dusted a boy
COGS output program 1:
*sailor (x __1;;; dust. (x_2, x_1 AND dust.
theme 2, x_4) AND boy (x_4)
15

---

Under review as a conference paper at ICLR 2022
COGS VAR input sentences 1:
The crewman scattered a female_chilc
The boater dotted male_child.
The leghorn dotted a son
COGS_VAR output programs 1:
kcrewman (x_1); peatter.agent (x_2, x_1) AND scatter. theme (x_2, x_4)
AND (emale_chhildx__)
<boater (x_1); agent (x_2, x_1) AND dot. theme (x_2, x_4) AND (x_e__chidd(x_4
+eeghrrn (x_1); dot agent (x_2, x_1) AND dot theme (x_2, x_4) AND son (x_4)
COGS input sentence 2:
The cookie was passed to Emma
COGS output program 1: *cookie (x_1); pass, theme (x_3, x_ AND pass. recipient (x_3, Emma)
COGS VAR input sentences 2:
The cooky was faded to Joan
The cooky overstepped to Trudy The cooky was eliminated to Kim
COGS output programs
*cooky (x_1); fade. theme 3,_3, x_1) AND fade. recipient (x_3, Joan)
*cooky (x_1); overstep. theme 3, x_ AND overstep. (x_3, Trudy)
kcooky (x_1); eliminate, theme (x_3, x_1) AND eliminatee (x_3, Kim)
COGS input sentence 3: Zoe thought that hippo cleaned
COGS output program 3: think agent (x_1, Zoe) AND think.ccomp (x_1, x_5)
AND hippo (x_4) AND clean.agent (x_5, x_4)
CORS_VAR input sentences 3:
Larry cerebrated that Hippo housecleaned Dana guessed that a hippopotamus picked.
Ronnie retrieved that hippopotamus make_cleard
CoGS output programs 3:
cerebrate, (x_1, Larzy) AND serebrate.coom (x_1, x_5) AND Hippo (x_4) AND houseclean, ageeet
agent (x_5, x_4)
quess.agent (x_1, Dana) AND guess. ccomp (x_1, x_5) AND
hippopotamus 4) AND pick.agent (x_5, x_4)
retrieve. (x_1, Ronnie) AND retrieve. setrieve.ccom ccomp (x_1, x_5) AND hippopotamus xx_4 AND make_clean, agent (x_5, x_4)
16

---

Under review as a conference paper at ICLR 2022
D ADDITIONAL EXPERIMENTAL RESULTS
In this section, we provide the experimental results as well as the standard deviation DUEL omitted in the main text.
Table 5: Best accuracy the target tasks cg GEO and sCAN
Method
COGS
GEO
SCAN cd
SBSP
49.2
100
NQG-T5
56.6
100
MAML
92.7/3.1
15.9
DUEL
95.4/4.5
57.8
21.3
D.1 COMPARISON WITH THE EXISTING WORKS
Several existing works leverage the specific designs of the datasets to solve SCAN (Andreas) 2020
Li al. 2019 Russin et al.) 2019). most of the improvements do not transfer to non-synthetic tasks. We compare to two methods that solve SCAN but can also transfer to non-synthetic tasks, SpanBasedSP (Herzig & Berant 2020) and NQG-T5 (Shaw et al., 2021). Also, we compare to recently proposed model-agnostic meta-leariing approach (Conklin al. 2021).
We use the original released code to run the experiments for SBSP (SpanBasedSPP and MAML. For SBSP, we use the BERT-BASE model in their work; For MAML, we replace their 2-layer Transformer encoder and 2-layer Transformer decoder with the larger model T5-Base Encoder- Decoder for fair comparison.
DUEL outperforms MAML on sc ed and COGS eg. and outperforms SpanBasedSP substantially on GEO As expected, it falls behind SpanBasedSP and NQG-T5 SCAN
Table 6: Average input length of correctly predicted examples on length generalization (GEO SHORT G00 from different pre-fineet feefinetnning tasks. Number of correct examples show parentheses.
Model
Pre-finetuning
SCAN std
SCAN ed
COGS
SCAN Jen
NONE
7.633 (71)
BERT2SEQ
MERGED
7.634 (71)
7.634 (71)
7.642 (74)
7.634 (71)
DUEL
7.620 (70)
7.651 (74)
7.657 (78)
7.693 (81)
NONE
8,876 (170)
T5-Base
MERGED
8.872 (172)
8.872 (172)
8.899 (180)
8.872 (172) 8.985 (198)
DUET.
8.865 (173)
8.960 (176)
8.968 (189)
D.2 WHAT TRANSFERS BETWEEN DATASETS?
The ability transfer across datasets is related what exactly the model is learning by DUEL during the pre-finetuning stage. Our hypothesis is that DUEL might restrict the encoder to learn to map structures compositionally, bits and pieces, instead of overfitting on arbitrary complex "structural noises". In the light of this hypothesis, the DUEL pre- pre-fireeet pre-fnetuning regularizes the encoder to represent structural information in more decomposable way, instead of "contextually interdependent". Sup- pose the input sentence is "jump right after walk right". Ideally, we would want the encoder to pick up the cue that "after" signifies represent "jump right" and "walk right" separately. However, for an end-to-end model, the representation could be biased to couple these two shorter phrases together to map to the desired output sequence.
We have not come up with definitive way to test the theory, instead, we added some indirect evidences to support the theory. We compare the input length of correctly predicted examples in GEO len between training with DUEL and MERGED in Table 6 The results indicate that the improvement mostly comes from the longer inputs, which means the encoder being regularized on the abstract notion of learning better from shorter structures and supports hypothesis.
17

---

Under review as a conference paper at ICLR 2022
Table 7: Accuracy on the target tasks after pre-finetuning on other tasks with splits the CG General category. For COGS, we report lexical and structural CG accuracy separately (lexical/structuraa) All the results are averaged over runs.
Model
Pre-finetuning
GEO
SCAN
COGS
SCAN
COGS
GEO
>COGS og
->EOOsl
-SCAN cd
(FEL(F)
61.4/1.0
58.6/1.1
35.8
36.1
4.4
4.8
BERT2SEQ
DUEL(S)
72.0/0.3
72.3/1.3
36.3
37.5
5.1
2.0
DUEL
75.5/1.1
73.2/1.1
36.9
37.7
4.8
5.5
(UEL(F)
90.7/2.5
89.6/2.5
52.5
55.7
16.0
18.9
T5-Base
(UBL(S)
94.8/2.3
95.1/3.6
53.4
57.5
20.1
12.7
DUEL
95.4/4.5
94.9/3.9
53.8
57.8
20.2
21.3
D.3. ABLATIONS ON THE DESIGN OF DUEL
We evaluate on two ablations DUEL: (1) DUEL((), where we keep the encoder fixed in fine-tuning: (2) DSF,.(S,, where we switch the train and the test component in pre-finetuning. Results listed in Table 7
DUBL(F) shows large performance drop comparing with DUEL, which implies is beneficial to finetune the encoder together with the decoder.
DUEL(S) shows minor degradation to DUEL when pre-finetuning SCAN, likely due its symmetric MCD split. Similarly, the COGS lexical generalization set has the same average input length as the train and contains all the vocabs. Thus, training generalization set, which leads to minor change in (S).(S). However, DUEL(S) has large gap with DUEL when pre-finetuning on GeoQuery. As the TMCD split only ensures all atoms appear in the train component GEO IMCDI, we find it has about 30% of examples containing words that do not appear in the test component GEO Thus, using GEO to train the decoder and GEO rMeD! to train the encoder forces the encoder to model those "new" atoms. This likely causes the encoder not learning enough inductive bias, thus incurring a larger drop in performance.
largely symmetric with its lexical
D.4 THE STANDARD DEVIATION OF DUEL
In Tables 2 and report the mean of runs for accuracy of the target tasks when pre-finetuning with DUEL. The standard deviations for these runs are reported in Table 8 and respectively.
Table 8: The standard deviations of the DUEL results in the CG General category in Table
Model
Pre-fince Pre-finetning
GEO
SCAN
COGS
SCAN
COGS
GEO
CCOGS
+G00
+SCAN od
BERT2SEQ
DUEL
4.0/0.4
3.6/0.3
0.5
1.0
1.3
1.8
T5-Base
DUEL
1.4/0.7
1.5/0.2
0.1
1.4
0.4
2.8
Table 9: The standard deviations of the DUEL results in the CG Length category in Table
Model
Pre-finetuning
sCAN
SCANed
COGS og
SCAN
COGS SCAN
BERT2SFQ
DUEL
1.0
0.5
0.4
0.5
0.2
T5-Base
DUEL
0.8
0.1
0.5
0.9
1.1
18


Reviews:
[{"review_id": "cVWNdToNf5", "reviewer": "Reviewer_gjvP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I thought that this was an interesting paper, which was mostly clearly written and organized, and generally I liked the method that was introduced. I'm leaning toward acceptance (and I can imagine the concerns below being addressed satisfactorily and further increasing my confidence).\n\nI had a few questions and potential concerns that weaken my confidence in the impact of the contribution. \n\nThe first question involves the reasoning behind the particular design of the method. The authors lay out a rationale for training the encoder parameters on the test component of each split, and the decoder on the train component of each split -- but the reasoning given is not terribly transparent to me, and I was left wondering whether similar results could be achieved by instead training the encoder on the train component and the decoder on the test component. Was this something that the authors tried? I think that including this comparison could be informative with respect to the importance of setting up the method in this specific way.\n\nAnother confusion I had involved the original purpose of these various datasets, and how this related to the current usage. For instance, not being familiar with GeoQuery, the description in the paper led me to believe that it was designed as a QA dataset, so I was wondering why any SOTA would exist for semantic parsing on a QA dataset. A google search suggests that GeoQuery is in fact annotated with semantic parses, but this confusion could be alleviated by making clearer that the dataset is used for semantic parsing. \n\nI was similarly wondering about use of SCAN for semantic parsing, since my understanding was that SCAN was designed for mapping commands to actions. If this is correct, where are the semantic parses coming from for that dataset? I imagine that since it is synthetic/template-based, producing semantic parses in a rule-based manner may be straightforward, but it wasn't clear to me from the paper how this was working. \n\nMy additional concerns are focused more on the impact of the contribution. The baselines that the paper compares against are for the most part not external models -- rather, the authors are comparing only against baseline versions of their own model, without the key components of the new method. So the improvement over the baselines indicates that the method does improve over the same model without the iterative compositional-split training. However, it is only in the one case of the GeoQuery dataset that the authors mention the existing SOTA (which they have beaten), suggesting that there are stronger SOTA models on the other datasets (or at least on COGS, if SCAN is not used typically for semantic parsing?). What this leads me to believe is that while the method improves over a vanilla model, it may not be improving over stronger models that use alternative methods for COGS (and possibly SCAN). I'm curious especially whether other models have made better headway on the COGS structural test, which is showing especially low performance here. It would be helpful to get greater clarity on how the presented results relate to the strongest existing results from other models across all datasets. \n\nFinally, I'm not totally sure how surprised/impressed we should be by improvements from this method. Specifically, I'm wondering how impressed we should be that we see a performance boost from training models to generalize across a specific type of split (e.g., one in which the test sentences are longer than the training sentences), for exactly the target task (semantic parsing). The authors make this general observation in Section 5.3, when they acknowledge that the model works best when the compositional splits are maximally similar. So to what extent is it ultimately somewhat obvious that training models to handle a given type of split will help it on this type of split?\n\nTo put the above concern another way: to what extent are we potentially *no longer testing models on compositional generalization* if we train them directly to be able to generalize in the particular way that is needed for the selected compositional split? If the performance boost is very specific to a particular relationship between test and training data, is this simply allowing the model to learn strategies specific to that particular type of generalization, such that it no longer needs to use composition to achieve that generalization? This would defeat the purpose of trying to improve models' ability to use actual compositional processing to show the desired generalization. So I would like to see the authors address this concern.\n", "summary_of_the_paper": "This paper is focused on the problem of compositional generalization in semantic parsing, and introduces a method called \"DUEL\", which involves \"pre-finetuning\" iteratively on compositional train-test splits from other datasets, before transferring to fine-tuning on the training data from the target dataset. The method involves using the compositional train/test split from one dataset, and training their encoder-decoder model iteratively such that the encoder parameters are updated based on the test data from that dataset, and the decoder parameters are updated based on the training data from that dataset. After this \"pre-finetuning\", the model is fine-tuned on the training data from the target dataset. They find that their model outperforms baselines involving 1) fine-tuning on the target task only, and 2) pre-finetuning on the merged data from the other dataset, without the encoder/decoder split. They find that their method largely does not help with the extremely low numbers on COGS structural items, but the margins of improvement are larger for GeoQuery data and SCAN data, with the authors claiming a new SOTA result on one of the splits for GeoQuery.   ", "summary_of_the_review": "In general I found the method interesting and the paper overall clear. However, I have some remaining questions about certain aspects of the method and datasets/tasks, as well as some potential concerns about the impact of the contribution, which I would like to see addressed before I can strongly endorse this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4DQm7HTXFQx", "reviewer": "Reviewer_K5oD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\nS1) I appreciated the paper's use of a non-synthetic dataset (GeoQuery), as I feel that this is an underexplored area of work on compositional generalization which will be useful to explore how and when nets fail to generalize on real data, and how to fix them.\n\nS2) The proposed approach seems simple and easy to implement.\n\nS3) The experiments were overall thorough (at least in the scope of semantic parsing compositional generalization), evaluating on three different datasets and two different models.\n\nS4) I found the demonstration of the benefits of pre-finetuning interesting and convincing.\n\nS5) The paper was extremely clearly written, in particular the description of the method.\n\n*Weaknesses*\n\nW1) I wasn't totally convinced that the method works well on strong tests of compositional generalization. \n- The GEO_cd and SCAN_cd splits, although they follow past work, are defined using a compound divergence method that, as the paper points out, does not ensure that compounds are completely absent (but only infrequent) in the training set.\n- The COGS_cg lexical challenge seems to mostly be obviated by pre-trained representations.\n- While I did find the length generalization results to be a convincing improvement and a more reasonable test of structural generalization, no method really seems to help much on the harder structural generalization test of COGS_cg, and (concerningly) even the proposed method makes no improvement in what seems to be the most a priori favorable experiment design for it, described in 5.4 (although I did appreciate including this negative result!).\n\nW2) It's not totally clear to me why the method should enable compositional generalization in general, and I feel like it would help to strengthen the motivation and intuition for the method, or perhaps do some some analysis could be done to indicate why it's working (where it is).\n- The paper motivates DUEL as learning to represent input sequences in a way that facilities compositional generalization, but it's not totally clear to me how the alternating freezing does this. It seems like the meta-learning approach (which directly trains for compositional generalization) that other work has employed is more directly suited to this, or even perhaps some adversarial training approach if the paper is aiming to learn representations f(x) that encode invariances across s and ~s. \n- It would help if the paper could somehow characterize the representations or models that DUEL learns (e.g. providing something like a fixed-point analysis), which \"the algorithm has converged to the desired representation since the difference in representing s and ~s is small\" in section 4 starts to do, but wasn't totally clear to me.\n- Or alternatively, perhaps the paper could do some empirical investigation of distributional differences in f(x) when x is drawn from s versus from ~s. \n\nW3) Some of the choices in the design of the method felt a bit arbitrary, and if they were better justified I'd feel more confident that the approach is working for understandable reasons.\n- Why reinitialize the parameters of g in fine-tuning?\n- The pre-finetuning setup on p does not match the fine-tuning setup on q in that fine-tuning updates both f and g. Why not update f in training on p, or keep f fixed in training on q?\n- While in some compositional generalization tasks, ~s is intuitively harder (e.g. longer inputs and outputs) than s in some way, in other tasks s and ~s are interchangeable, so does it matter that g is updated on s and f on ~s (and that the last updates in pre-finetuning are always done on ~s)? \n\nW4) It would help to present statistical significance results, or standard deviations across multiple seeds, as it's a bit difficult to interpret the significance of the improvements. However, since the improvements are consistent (albeit small), I don't think this is a crucial weakness.\n\n*Minor comments*\n\n- It would help to give some intuition for the \\alpha in Section 3. How is this value chosen?\n- The update equations (1-3) with a single step-size make it seem like SGD is being used, but from the appendix it's Adam.\n- How are the logical forms updated in the COGS_VAR splits to match the changes to the input sentences?\n\n*Typos*:\n\n- pg 4 \"standard supervise learning\" -> \"standard supervised learning\"\n- pg 5: several grammatical errors at the end of section 4\n- pg 6: \"BERT_SMALL\" -> \"BERT_BASE\" (?)\n- pg 6: \"GEO_TMCD2\" -> GEO_{cd}\"\n- pg 7: \"When Will DUEL Works Best\" -> \"When Will DUEL Work Best\"\n- pg 7: \"compsitional\" -> \"compositional\"\n- pg 7: \"DUEL helps extracting\" -> \"DUEL helps extract\"\n", "summary_of_the_paper": "This paper proposes a training procedure for encoder--decoder models (applied to semantic parsing) which aims to improve the models' ability to compositionally generalize (successfully handle novel combinations of words and structures, where combinations were not seen in training). The approach relies on pre-finetuning: training a model on a different dataset than the target dataset that also the requires the same sort of compositional generalization as the target dataset, before then training on the training set of the target dataset and then evaluating zero-shot on the compositional set of the target dataset, in the standard way. In pre-finetuning, the decoder is only updated on the training set while the encoder is updated on the compositional generalization set. The approach is evaluated using two different pre-trained encoder--decoder transformer architectures on three different semantic parsing compositional generalization datasets from past work, where it obtains consistent (albiet somewhat small) improvements over a baseline that pre-finetunes all model parameters, and outperforms a past state-of-the-art model on one dataset.", "summary_of_the_review": "I feel a bit borderline about this paper, as the method seems a bit limited and heuristic -- not being clearly designed for compositional generalization or showing convincing results on the hardest tests of compositional generalization. But, it does seem to show consistent (if sometimes small) improvements on a couple models and several datasets, the methodology seems sound, and the paper is very clearly written. I've put an overall score of 5 for now, but I look forward to discussion.\n\n---\n\nUpdate after the response: Thanks to the authors for their thorough response to my comments! The explanations and new ablation results helped convince me that the choices made in designing the method were reasonable. I also appreciate the standard deviations, which make me confident that the improvements are real. I'm in favor of accepting this paper, and have updated my score to a 6 (from a 5).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "255bMWN3Tmu", "reviewer": "Reviewer_uYpx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While a like this idea, there are several issues with the proposal approach:\n\n1. **Comparison with State-of-the-Art** There is little information in Section 5 about comparison with existing approaches in compositional generalization for semantic parsing. Indeed, in recent years several seminal works have emerged, pushing accuracies on some synthetic tasks like SCAN to near 100% accuracy. These works are not mentioned in Section 5. While the model outperformed the currently best  approach (Shaw et al., 2021) on GEO_{TMCD}, the lower results on other simpler tasks like SCAN make me feel a bit concerned about the results. Perhaps this is because only few models are evaluated on GEO_{TMCD} so far, and previous approaches more tailored to the context-free utterances on GEO (e.g., Herzig and Berant) would actually perform better? \n\n2. **Methodology** Another issue is related to the proposed approach itself. While pre-finetuning encoder on compositional split A and the decoder on compositional split B could encourage the encoder to learn representations that generalize better to split B, the generalization strategy learned by the encoder might be specific to split B only, and might not be able to generalize to other compositionally novel distributions (e.g., the final evaluation data). Ideally during the pre-finetuning stage the model need to learn to generalize well to *arbitrary* mismatched splits, but only presenting one set of splits (A/B) might not be enough for the model to learn a more \"general-purpose\" strategy.\n\n3. **Transferability across Datasets** Transfer learning on NLP tasks would require the source and target domains share reasonably amount of common language patterns in order to perform well. However, the tasks used in this paper have drastically different utterances in language styles and compositional patterns, which makes transfer learning quite non-trivial. For example, SCAN only contains toyish words like JUMP and simple composition strategies like concatenation (JUMP TWICE). It would be doubtable if learning to generalize well on this toyish domain would be useful for handling real-world utterances with diverse language style, like GEO. The authors could present more analysis in terms of the language and compositionality styles of those datasets in order to have a better understanding of the upper-bound performance of transfer learning approaches for compositional generalization.", "summary_of_the_paper": "This paper presents a transfer learning strategy for improving compositional generalization of semantic parsers based on pre-trained language models. Before fine-tuning the model on data from the target domain, the authors propose a pre-finetuning step, where models are trained on compositional splits of data from another source domain, with the goal to transfer the model's learned knowledge about language compositionality during this pre-finetuning step to the final learning stage on the target domain, therefore improving compositional generalization. To this end, the authors propose a pre-finetuning method which encourages the model to discover representations of natural language that are invariant against its compositional structures. This is achieved by iteratively freezing the encoder or decoder modules during pre-finetuning, and training the encoder and decoder modules on compositionally disjoint splits of the source data, such that the encoder learns representations that are robust against distributional shift of language compositionality.\n\n", "summary_of_the_review": "This paper presents a nice idea for improving compositional generalization of neural semantic parsers. The results on GEO_{TMCD} outperforms the currently best approach. However, there are issues with both experimentation and the methodology. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors attempt to tackle the problem of compositional generalization, i.e., the problem of generalizing to 
novel combinations of familiar words or structures. The authors propose a transfer learning strategy based on
pretraining language models. The idea is to introduce a pre-finetuning task where a model is first trained on compositional train-test splits from other datasets, before transferring to fine-tuning on the training data from the target dataset. Although the technique
brings some improvements, and the authors do their best the address the reviewers' questions, it is still unclear:

a) Why the method should work in principle, whether there is a theoretical backing and how it formally relates to meta-learning
b) How the approach compares to data augmentation methods since pre-finetuning requires more data, albeit from a different
dataset. See for example: https://openreview.net/forum?id=PS3IMnScugk
c) The whole approach would be more convincing if the authors could articulate *how* their method renders a model
more robust to distribution shifts (e.g., based on GOGS results it does not help structural generalization, do the gains 
come from lexical generalization?)
d) it would also be interesting whether this method works on larger scale or more realistic datsets like CFQ, ATIS or machine translation
https://arxiv.org/pdf/1912.09713.pdf
https://arxiv.org/abs/2010.11818
Decision: Reject