Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduces a method for generating training/test data for measuring the model's ability of "compositional generalization" in complex compositional tasks/domains such as natural language understanding, visual understanding and other domains. 
The idea of the proposed method is to essentially keep the "atom" distribution unchanged between the training and test data, but maximize the divergence between the "compound" distributions between them. 
The authors have conducted a thorough and systematic experimentation, comparing the proposed approach with a number of other heuristic approaches for train test splits (such as random and input/output length, etc.) and using both a new large data set they generated (CFQ) and existing data set (SCAN). 
The experimental results verify that using their method they can obtain train test data sets with uniform atom distributions with large divergence in compound distributions, and they find that there is a surprisingly large negative correlation between the accuracy of existing state-of-the-art learning methods and the compound divergence. 
The data generation mechanism is systematic and involved, consisting of different categories of rules (logical form, grammar, rule application DAG's, etc.) and it would seem that the generation method/system and the generated data would be useful as benchmark data for the community. 
The paper lacks technical novelty other than the training and test data generation approach, but having one available to the community with these apparently desirable characteristics as benchmark data for measuring complex, compositional generalization capabilities, and that could be invaluable to the research community. 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper first introduces a method for quantifying to what extent a dataset split exhibits compound (or, alternatively, atom) divergence, where in particular atoms refer to basic structures used by examples in the datasets, and compounds result from compositional rule application to these atoms. The paper then proposes to evaluate learners on datasets with maximal compound divergence (but minimal atom divergence) between the train and test portions, as a way of testing whether a model exhibits compositional generalization, and suggests a greedy algorithm for forming datasets with this property. In particular, the authors introduce a large automatically generated semantic parsing dataset, which allows for the construction of datasets with these train/test split divergence properties. Finally, the authors evaluate three sequence-to-sequence style semantic parsers on the constructed datasets, and they find that they all generalize very poorly on datasets with maximal compound divergence, and that furthermore the compound divergence appears to be anticorrelated with accuracy.

This is an interesting and ambitious paper tackling an important problem. It is worth noting that the claim that it is the compound divergence that controls the difficulty of generalization (rather than something else, like length) is a substantive one, and the authors do provide evidence of this. At the same time, I think the authors could possibly do more to show that the trend in the plots in Figure 2 can't be explained by something else: for example, the authors could show that the length ratios remain constant as the compound divergence is varied.  I think it is also not necessarily clear how easily the notion of differing compound distributions generalizes to other types of tasks.

Presentation-wise, much of the paper is clear and well written, though I think the discussion of weighted frequency distributions of compounds (top of page 3) could be clarified further, and in particular an example subgraph of a rule application DAG should be highlighted here. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary

Strength:
1. This topic studied in this paper is interesting and is helping to promote the following developing of algorithms with compositional generalization ability.
2. The experimental results show the effectiveness of the proposed metric for measuring compositional diversity.


comments:
1. How to control the trade-off between the atom and compositional divergence? It's interesting to show how different
compositional divergence can affect the performance of different models.
2. Many previous works are proposed for improving the generalization ability of the seq2seq models[1]. More experiments need to
be conducted using these previous methods.

[1]Compositional generalization in a deep seq2seq model by separating syntax and semantics

review_title: Official Blind Review #2

Meta-review: Main content:

Blind review #1 summarizes it well:

This paper first introduces a method for quantifying to what extent a dataset split exhibits compound (or, alternatively, atom) divergence, where in particular atoms refer to basic structures used by examples in the datasets, and compounds result from compositional rule application to these atoms. The paper then proposes to evaluate learners on datasets with maximal compound divergence (but minimal atom divergence) between the train and test portions, as a way of testing whether a model exhibits compositional generalization, and suggests a greedy algorithm for forming datasets with this property. In particular, the authors introduce a large automatically generated semantic parsing dataset, which allows for the construction of datasets with these train/test split divergence properties. Finally, the authors evaluate three sequence-to-sequence style semantic parsers on the constructed datasets, and they find that they all generalize very poorly on datasets with maximal compound divergence, and that furthermore the compound divergence appears to be anticorrelated with accuracy.

--

Discussion:

Blind review #1 is the most knowledgeable in this area and wrote "This is an interesting and ambitious paper tackling an important problem. It is worth noting that the claim that it is the compound divergence that controls the difficulty of generalization (rather than something else, like length) is a substantive one, and the authors do provide evidence of this."

--

Recommendation and justification:

This paper deserves to be accepted because it tackles an important problem that is overlooked in current work that is evaluated on datasets of questionable meaningfulness. It adds insight by focusing on the qualities of datasets that enable testing how well learning algorithms do on compositional generalization, which is crucial to intelligence.
Decision: Accept (Poster)