Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a transfer learning strategy for improving compositional generalization of semantic parsers based on pre-trained language models. Before fine-tuning the model on data from the target domain, the authors propose a pre-finetuning step, where models are trained on compositional splits of data from another source domain, with the goal to transfer the model's learned knowledge about language compositionality during this pre-finetuning step to the final learning stage on the target domain, therefore improving compositional generalization. To this end, the authors propose a pre-finetuning method which encourages the model to discover representations of natural language that are invariant against its compositional structures. This is achieved by iteratively freezing the encoder or decoder modules during pre-finetuning, and training the encoder and decoder modules on compositionally disjoint splits of the source data, such that the encoder learns representations that are robust against distributional shift of language compositionality.


main_review: While a like this idea, there are several issues with the proposal approach:

1. **Comparison with State-of-the-Art** There is little information in Section 5 about comparison with existing approaches in compositional generalization for semantic parsing. Indeed, in recent years several seminal works have emerged, pushing accuracies on some synthetic tasks like SCAN to near 100% accuracy. These works are not mentioned in Section 5. While the model outperformed the currently best  approach (Shaw et al., 2021) on GEO_{TMCD}, the lower results on other simpler tasks like SCAN make me feel a bit concerned about the results. Perhaps this is because only few models are evaluated on GEO_{TMCD} so far, and previous approaches more tailored to the context-free utterances on GEO (e.g., Herzig and Berant) would actually perform better? 

2. **Methodology** Another issue is related to the proposed approach itself. While pre-finetuning encoder on compositional split A and the decoder on compositional split B could encourage the encoder to learn representations that generalize better to split B, the generalization strategy learned by the encoder might be specific to split B only, and might not be able to generalize to other compositionally novel distributions (e.g., the final evaluation data). Ideally during the pre-finetuning stage the model need to learn to generalize well to *arbitrary* mismatched splits, but only presenting one set of splits (A/B) might not be enough for the model to learn a more "general-purpose" strategy.

3. **Transferability across Datasets** Transfer learning on NLP tasks would require the source and target domains share reasonably amount of common language patterns in order to perform well. However, the tasks used in this paper have drastically different utterances in language styles and compositional patterns, which makes transfer learning quite non-trivial. For example, SCAN only contains toyish words like JUMP and simple composition strategies like concatenation (JUMP TWICE). It would be doubtable if learning to generalize well on this toyish domain would be useful for handling real-world utterances with diverse language style, like GEO. The authors could present more analysis in terms of the language and compositionality styles of those datasets in order to have a better understanding of the upper-bound performance of transfer learning approaches for compositional generalization.
summary_of_the_review: This paper presents a nice idea for improving compositional generalization of neural semantic parsers. The results on GEO_{TMCD} outperforms the currently best approach. However, there are issues with both experimentation and the methodology.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is focused on the problem of compositional generalization in semantic parsing, and introduces a method called "DUEL", which involves "pre-finetuning" iteratively on compositional train-test splits from other datasets, before transferring to fine-tuning on the training data from the target dataset. The method involves using the compositional train/test split from one dataset, and training their encoder-decoder model iteratively such that the encoder parameters are updated based on the test data from that dataset, and the decoder parameters are updated based on the training data from that dataset. After this "pre-finetuning", the model is fine-tuned on the training data from the target dataset. They find that their model outperforms baselines involving 1) fine-tuning on the target task only, and 2) pre-finetuning on the merged data from the other dataset, without the encoder/decoder split. They find that their method largely does not help with the extremely low numbers on COGS structural items, but the margins of improvement are larger for GeoQuery data and SCAN data, with the authors claiming a new SOTA result on one of the splits for GeoQuery.   
main_review: Overall I thought that this was an interesting paper, which was mostly clearly written and organized, and generally I liked the method that was introduced. I'm leaning toward acceptance (and I can imagine the concerns below being addressed satisfactorily and further increasing my confidence).

I had a few questions and potential concerns that weaken my confidence in the impact of the contribution. 

The first question involves the reasoning behind the particular design of the method. The authors lay out a rationale for training the encoder parameters on the test component of each split, and the decoder on the train component of each split -- but the reasoning given is not terribly transparent to me, and I was left wondering whether similar results could be achieved by instead training the encoder on the train component and the decoder on the test component. Was this something that the authors tried? I think that including this comparison could be informative with respect to the importance of setting up the method in this specific way.

Another confusion I had involved the original purpose of these various datasets, and how this related to the current usage. For instance, not being familiar with GeoQuery, the description in the paper led me to believe that it was designed as a QA dataset, so I was wondering why any SOTA would exist for semantic parsing on a QA dataset. A google search suggests that GeoQuery is in fact annotated with semantic parses, but this confusion could be alleviated by making clearer that the dataset is used for semantic parsing. 

I was similarly wondering about use of SCAN for semantic parsing, since my understanding was that SCAN was designed for mapping commands to actions. If this is correct, where are the semantic parses coming from for that dataset? I imagine that since it is synthetic/template-based, producing semantic parses in a rule-based manner may be straightforward, but it wasn't clear to me from the paper how this was working. 

My additional concerns are focused more on the impact of the contribution. The baselines that the paper compares against are for the most part not external models -- rather, the authors are comparing only against baseline versions of their own model, without the key components of the new method. So the improvement over the baselines indicates that the method does improve over the same model without the iterative compositional-split training. However, it is only in the one case of the GeoQuery dataset that the authors mention the existing SOTA (which they have beaten), suggesting that there are stronger SOTA models on the other datasets (or at least on COGS, if SCAN is not used typically for semantic parsing?). What this leads me to believe is that while the method improves over a vanilla model, it may not be improving over stronger models that use alternative methods for COGS (and possibly SCAN). I'm curious especially whether other models have made better headway on the COGS structural test, which is showing especially low performance here. It would be helpful to get greater clarity on how the presented results relate to the strongest existing results from other models across all datasets. 

Finally, I'm not totally sure how surprised/impressed we should be by improvements from this method. Specifically, I'm wondering how impressed we should be that we see a performance boost from training models to generalize across a specific type of split (e.g., one in which the test sentences are longer than the training sentences), for exactly the target task (semantic parsing). The authors make this general observation in Section 5.3, when they acknowledge that the model works best when the compositional splits are maximally similar. So to what extent is it ultimately somewhat obvious that training models to handle a given type of split will help it on this type of split?

To put the above concern another way: to what extent are we potentially *no longer testing models on compositional generalization* if we train them directly to be able to generalize in the particular way that is needed for the selected compositional split? If the performance boost is very specific to a particular relationship between test and training data, is this simply allowing the model to learn strategies specific to that particular type of generalization, such that it no longer needs to use composition to achieve that generalization? This would defeat the purpose of trying to improve models' ability to use actual compositional processing to show the desired generalization. So I would like to see the authors address this concern.

summary_of_the_review: In general I found the method interesting and the paper overall clear. However, I have some remaining questions about certain aspects of the method and datasets/tasks, as well as some potential concerns about the impact of the contribution, which I would like to see addressed before I can strongly endorse this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a training procedure for encoder--decoder models (applied to semantic parsing) which aims to improve the models' ability to compositionally generalize (successfully handle novel combinations of words and structures, where combinations were not seen in training). The approach relies on pre-finetuning: training a model on a different dataset than the target dataset that also the requires the same sort of compositional generalization as the target dataset, before then training on the training set of the target dataset and then evaluating zero-shot on the compositional set of the target dataset, in the standard way. In pre-finetuning, the decoder is only updated on the training set while the encoder is updated on the compositional generalization set. The approach is evaluated using two different pre-trained encoder--decoder transformer architectures on three different semantic parsing compositional generalization datasets from past work, where it obtains consistent (albiet somewhat small) improvements over a baseline that pre-finetunes all model parameters, and outperforms a past state-of-the-art model on one dataset.
main_review: *Strengths*

S1) I appreciated the paper's use of a non-synthetic dataset (GeoQuery), as I feel that this is an underexplored area of work on compositional generalization which will be useful to explore how and when nets fail to generalize on real data, and how to fix them.

S2) The proposed approach seems simple and easy to implement.

S3) The experiments were overall thorough (at least in the scope of semantic parsing compositional generalization), evaluating on three different datasets and two different models.

S4) I found the demonstration of the benefits of pre-finetuning interesting and convincing.

S5) The paper was extremely clearly written, in particular the description of the method.

*Weaknesses*

W1) I wasn't totally convinced that the method works well on strong tests of compositional generalization. 
- The GEO_cd and SCAN_cd splits, although they follow past work, are defined using a compound divergence method that, as the paper points out, does not ensure that compounds are completely absent (but only infrequent) in the training set.
- The COGS_cg lexical challenge seems to mostly be obviated by pre-trained representations.
- While I did find the length generalization results to be a convincing improvement and a more reasonable test of structural generalization, no method really seems to help much on the harder structural generalization test of COGS_cg, and (concerningly) even the proposed method makes no improvement in what seems to be the most a priori favorable experiment design for it, described in 5.4 (although I did appreciate including this negative result!).

W2) It's not totally clear to me why the method should enable compositional generalization in general, and I feel like it would help to strengthen the motivation and intuition for the method, or perhaps do some some analysis could be done to indicate why it's working (where it is).
- The paper motivates DUEL as learning to represent input sequences in a way that facilities compositional generalization, but it's not totally clear to me how the alternating freezing does this. It seems like the meta-learning approach (which directly trains for compositional generalization) that other work has employed is more directly suited to this, or even perhaps some adversarial training approach if the paper is aiming to learn representations f(x) that encode invariances across s and ~s. 
- It would help if the paper could somehow characterize the representations or models that DUEL learns (e.g. providing something like a fixed-point analysis), which "the algorithm has converged to the desired representation since the difference in representing s and ~s is small" in section 4 starts to do, but wasn't totally clear to me.
- Or alternatively, perhaps the paper could do some empirical investigation of distributional differences in f(x) when x is drawn from s versus from ~s. 

W3) Some of the choices in the design of the method felt a bit arbitrary, and if they were better justified I'd feel more confident that the approach is working for understandable reasons.
- Why reinitialize the parameters of g in fine-tuning?
- The pre-finetuning setup on p does not match the fine-tuning setup on q in that fine-tuning updates both f and g. Why not update f in training on p, or keep f fixed in training on q?
- While in some compositional generalization tasks, ~s is intuitively harder (e.g. longer inputs and outputs) than s in some way, in other tasks s and ~s are interchangeable, so does it matter that g is updated on s and f on ~s (and that the last updates in pre-finetuning are always done on ~s)? 

W4) It would help to present statistical significance results, or standard deviations across multiple seeds, as it's a bit difficult to interpret the significance of the improvements. However, since the improvements are consistent (albeit small), I don't think this is a crucial weakness.

*Minor comments*

- It would help to give some intuition for the \alpha in Section 3. How is this value chosen?
- The update equations (1-3) with a single step-size make it seem like SGD is being used, but from the appendix it's Adam.
- How are the logical forms updated in the COGS_VAR splits to match the changes to the input sentences?

*Typos*:

- pg 4 "standard supervise learning" -> "standard supervised learning"
- pg 5: several grammatical errors at the end of section 4
- pg 6: "BERT_SMALL" -> "BERT_BASE" (?)
- pg 6: "GEO_TMCD2" -> GEO_{cd}"
- pg 7: "When Will DUEL Works Best" -> "When Will DUEL Work Best"
- pg 7: "compsitional" -> "compositional"
- pg 7: "DUEL helps extracting" -> "DUEL helps extract"

summary_of_the_review: I feel a bit borderline about this paper, as the method seems a bit limited and heuristic -- not being clearly designed for compositional generalization or showing convincing results on the hardest tests of compositional generalization. But, it does seem to show consistent (if sometimes small) improvements on a couple models and several datasets, the methodology seems sound, and the paper is very clearly written. I've put an overall score of 5 for now, but I look forward to discussion.

---

Update after the response: Thanks to the authors for their thorough response to my comments! The explanations and new ablation results helped convince me that the choices made in designing the method were reasonable. I also appreciate the standard deviations, which make me confident that the improvements are real. I'm in favor of accepting this paper, and have updated my score to a 6 (from a 5).
review_title: null

Meta-review: The authors attempt to tackle the problem of compositional generalization, i.e., the problem of generalizing to 
novel combinations of familiar words or structures. The authors propose a transfer learning strategy based on
pretraining language models. The idea is to introduce a pre-finetuning task where a model is first trained on compositional train-test splits from other datasets, before transferring to fine-tuning on the training data from the target dataset. Although the technique
brings some improvements, and the authors do their best the address the reviewers' questions, it is still unclear:

a) Why the method should work in principle, whether there is a theoretical backing and how it formally relates to meta-learning
b) How the approach compares to data augmentation methods since pre-finetuning requires more data, albeit from a different
dataset. See for example: https://openreview.net/forum?id=PS3IMnScugk
c) The whole approach would be more convincing if the authors could articulate *how* their method renders a model
more robust to distribution shifts (e.g., based on GOGS results it does not help structural generalization, do the gains 
come from lexical generalization?)
d) it would also be interesting whether this method works on larger scale or more realistic datsets like CFQ, ATIS or machine translation
https://arxiv.org/pdf/1912.09713.pdf
https://arxiv.org/abs/2010.11818
Decision: Reject