Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: - The paper proposes a new dataset, called ReaSCAN, that extends gSCAN, but goes further in requiring semantic capabilities of reasoning about entities and relations as well as linguistic skills of compositional interpretation. 
- It complements the dataset with experiments to evaluate different models types on the dataset, including a multimodal baselines and a graph convolution network.
- The experiments demonstrate how ReaSCAN is more challenging that the predecessor gSCAN dataset.

Update: After reading the other reviews and the author's response I would like to keep my score!
strengths: - The paper explores an important and exciting topic for compositional reasoning in grounded environments. The task is well motivated and contextualized within progress of prior works.
- The new dataset addresses weaknesses of gSCAN, as is clearly described and illustrated through the paper's both text and figures.
- The paper is very well-written, the synthesis of the dataset is described in detail 
- The claims are corroborated by quantitative analysis and comparative evaluation both across models (GCN, multimodal) and between datasets (ReaSCAN and gSCAN). The paper presents a very extensive set of experiments with different splits and conditions.


weaknesses: For all the SCAN tasks, there is the common limitation of using auto-generated language in simple visual environments which potentially mean that models that perform well in these simplified settings may not scale and demonstrate the same compositional reasoning skills over real-world data (either visual or textual). 

However, that weakness is not unique to this particular work, but to the whole line of research, which on the other hand also enjoys from related benefits and advantages, such as controllability over the data, simple process to generate large quantities of it, and a lot of potential supervision signals that could help both for training and evaluation of models for the task. 

Having some e.g. natural-language extension of the task (e.g. CLEVR-Human) could be nice and useful.
correctness: The claims in the paper are explained in detail and corroborated by quantitative analysis.
clarity: The paper is clear and easy to follow, and the writing quality is high. It also includes variety of useful visual samples from the data, plots and visualizations, which all help for the presentation of the new dataset.
relation to prior work: The paper discuss relevant background knowledge about compositionality in general and the SCAN datasets in particular. The paper then does a good job in pointing to the specific three limitations of gSCAN (constrained instructions set, irrelevant distractor objects, and redundant modifiers in commands that aren't necessary for performing the navigation successfully) and explaining how they are addressed in the  new dataset. 
In the related work section, the author further discuss in greater detail the different design choices of gSCAN and their impact and implications on the task (e.g.  irrelevance of word order, biased sampling of distractors). This explanations greatly inform the reader of the task and the current limitations needed to be addressed. 

However, while most of the related work section focuses on gSCAN, I think that more information and context though about prior neural models that have been explored for the dataset, key modeling findings it has enabled, or potential applications models that work well on gSCAN may have, would be useful to strengthen the related work section.
additional feedback: No, see feedback above.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents ReaSCAN, a novel benchmark dataset for compositional language interpretation and reasoning about entities and relations that was developed after a thorough analysis of the shortcomings of previously presented datasets. The dataset contains instructions for actions in a grid word that are generated through regular expressions, and the corresponding maps generated under a number of carefully designed distractor sampling regimes. 

With their dataset, the authors not only address previous shortcomings of instruction complexity and distractor placement, but also create an opportunity for investigating the effects of different train/test splits that can be used to systematically test model performance on a specific aspect of generalisation difficulty presented by the data. 

Comparing a vanilla multi-modal LSTM model with a state-of-the-art graph convolutional neural network, the paper demonstrates the GCN’s superior level of robustness to command complexity and its improved capability to generalise to unseen data aspects, but also clearly reveals its shortcomings with respect to aspects like novel color attributes, novel relation co-occurrence and novel clause length. The dataset therefore appears to be an interesting benchmark for at least these facettes of compositional language interpretation - and could easily be modified to address others, too.

strengths: The authors present a detailed analysis of the shortcomings of their dataset’s predecessor gSCAN and present a sound approach to addressing them. The resulting new version of the dataset allows for testing on various aspects of compositional language interpretation by presenting a number of different command complexity settings as well as train/test splits to control the type of novel attribute or modulation. Testing a state-of-the-art multi-modal CGN on the dataset reveals its capabilities as a benchmark for compositional language interpretation by demonstrating significant performance drops with increasing command complexity and specific generalisation tests. 

The paper is very well written and clearly accounts for the reasoning behind the creation of the presented dataset as well as the details of the algorithms applied in creating it. The tested models are described well and parameter settings are detailed in the supplementary material.

The dataset is freely available on a well-known code-sharing platform and also can easily be re-generated based on the instructions and code in the submission.  

weaknesses: The dataset is created for testing compositional language interpretation in highly abstracted grid worlds. Ideally we would like to get to a point where models can be tested on more complex real-world scenarios - but the significant model shortcomings demonstrated in the paper indicate that we’re simply not there yet.  

correctness: The paper clearly accounts for the reasoning behind the creation of the presented dataset as well as the details of the algorithms applied in creating it. The evaluation of the tested models is sound.

clarity: The paper is very well written and clearly presents the dataset and its application as a benchmark for compositional language interpretation.

relation to prior work: The authors present a very detailed analysis of the shortcomings of their dataset’s predecessor gSCAN but mention other datasets and approaches to linguistic compositionality only in passing.

additional feedback: A great contribution to the field!
review_title: Expertly crafted benchmark dataset for compositional language interpretation and reasoning about entities and relations

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: ReaSCAN is a synthetic dataset for evaluating the models' ability of compositional language interpretation and reasoning. Each sample in ReaSCAN contains a natural language command, a simulated shape world, and the action sequence that move the object in the shape world following the command. It is an improved and more challenging version of gSCAN. Different from gSCAN, ReaSCAN ensures 1) word-order permutation in commands will affect the final action sequence, 2) a number of distractors are generated so that the model must resolve the syntatic structure of the command in order to generate the correct prediction. The author evaluated two models on ReaSCAN: M-LSTM and GCN-LSTM. Results show that the state-of-the-art GCN-LSTM model fails to generalize in settings that involve novel relation combinations and longer commands.

strengths: 1. ReaSCAN is a solid improvement upon gSCAN. The author proposed: 1) a more sophisticated command generator that contains relation clause. Compared with the commands in gSCAN which can be simply modeled via bag-of-words, commands in gSCAN have more complicated linguistic structure and are more challenging to resolve, 2) new algorithms for generating the distractors. There are three types of distractors in ReaSCAN: 1) attribute-based distractors, 2) isomorphism-based distractors, and 3) relation-based distractors. These generated distractors offer a way to control the complexity of queries in ReaSCAN.

2. The author shows that the state-of-the-art GCN-LSTM model cannot generalize in settings that involve more complicated reasoning, e.g., novel color attribute, novel relation co-occurrence, and novel clause length.

weaknesses: 1. ReaSCAN is created on top of gSCAN. Since the goal of gSCAN is also to evaluate compositional generalization, ReaSCAN is a incremental but solid improvement over the prior work.

2. The author has not evaluated the performance of pretrained NLP models like BERT on ReaSCAN. The challenges described in ReaSCAN (e.g., generalization to new color attributes) can also be solved by pretraining the model on a large and diverse corpus. In fact, the success of BERT and GPT-3 shows that pretraining on a large corpus can greatly improve the generalization ability.

correctness: Yes, the evaluation method and experiment design are correct.
clarity: Yes, the paper is well written.
relation to prior work: Yes.
additional feedback: The benchmark will be much stronger if the author can evaluate the performance of pretrained NLP models on ReaSCAN.

-------- Post Rebuttal -----
I've read the authors' rebuttal and choose to keep my score. My major concern is that the authors have not compared with methods based on pretrained language models, which are currently the SOTA for solving different types of NLP problems. In the real-world scenario, it is easy to obtain a large-scale corpus that contains sentences with diverse linguistic structures. Compared to sentences in real-world applications that may contain grammar errors, spelling mistakes, abbreviations, etc., the synthetic sentences generated in ReaSCAN have clear and interpretable structures and are not very realistic. The methods that are benchmarked are far from real-world solutions for NLP problems that heavily use pretrained models such as word embeddings and contextual embeddings.
review_title: Solid improvement over gSCAN. Lacks evaluation of pretrained NLP models.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors have proposed an improved version of gSCAN to test compositionality and grounding of NLP system, named as ReaSCAN. This new benchmark tries to solve some flaws of gSCAN, such as the irrelevance of word order, biased distractor sampling, etc, hence, claiming it to be more challenging benchmark to test model against. In the experiments, they assess two models, i.e, M-LSTM and GCN-LSTM.  
strengths: The paper proposes complex command generation process which makes ReaSCAN a better alternative to test models on. They also provide better distractor sampling techniques (unlike random sampling for gSCAN) which could help in better analyzing the grounding issues of NLP models. 
weaknesses: It wasn't very clear how ReaSCAN was better than gSCAN quantitatively. I think it would have been better to do some interpretability analysis on the models under experimentation to see what extra information it learnt with the new benchmark. One of the ways to analyze this could be to train on gSCAN and test on ReaSCAN and look for specific model decision making flaws which were mitigated with the new benchmark. 
correctness: The claims made in the paper makes sense qualitatively but require more empirical study to quantify improvements. 
clarity: Yes, I think the paper is clear. 
relation to prior work: Yes, the paper discusses the difference against prior work clearly. 
additional feedback: Its a nice work but it would make the paper more stronger with more analysis on the value it adds over gSCAN. Theoretically, the improvements make sense but would become more convincing if more metrics and models are brought in the discussion. Its hard for the community to move from one benchmark to other unless the improvements in the new benchmark is significantly better which can be provided with more analysis.
review_title: Improvements not very clear with results 

Meta-review: This paper presents ReaSCAN, which extends gSCAN dataset which aims to test grounded compositional semantics. The paper (1) concretely points to the deficiencies of the previous version of the dataset -- irrelevance of word order, a limited test of compositionality, limited distractors, and (2) designs new grammar to synthetically generate examples that are more complex and addresses these deficiencies. 

In addition to improving data, they also introduce a carefully designed train test split to test "learning" compositionality. They present two baselines used in prior work, and their experimental results show that this newer benchmark poses unsolved challenges.  

While reviewers pointed out the limitation of this work as a synthetic dataset -- not reflecting the real usages of human language --, many pointed out their careful experimental designs and linguistic complexity it brings will be beneficial for the research community to study grounded compositional semantics. 
Decision: Accept