Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A new approach is introduced for constructing benchmarks for the application of consistent rules in language models. This approach is used to create a new dataset that is then used to evaluate T5 models on 3 dimensions: applying learned rules, scaling to larger sets of rules and compositional generalization. 

strength_and_weaknesses: **Strengths**: The idea is novel and it explores an important direction. See more details in the question below. The paper presents an idea, concrete usage of it, and empirical analysis on metrics of accuracy and consistency. The dataset section presents the data in sufficient detail, and the baselines are also described with a good level of detail. Exploring more baselines beyond T5 though would strengthen the paper. 

**Weaknesses**:
* **Presentation of the motivation, real-world case**: The motivating example about the movie recommendations used to open the paper is not convincing in my opinion.  First, in general movie recommendation is more of a typical case of example-based learning, where people too give recommendations not by strict rules but by an approximate sense of what prior movies a person like, and what are his general preferences. More specifically, all except one of the rules presented in the example to illustrate rule-based inference are actually not rules but either facts or evidence. This is problematic as an introduction since evaluating rule application in language models is the main theme of the paper, where the example doesn’t show actual rules. Other more natural examples could be used to illustrate where people use rules, beyond the most obvious examples of math or physics problems etc, one could use examples of day-to-day such as admission criteria to a school or university, rules about payments or finance, examples of rules from the law system, etc etc. 

* **Presentation of the motivation, SCAN case**: The second example about SCAN is closer to exemplify about rules but the claim about people presented with rules rather than examples is not compelling either. E.g. if I showed a person example that f(A B and then C B)=aacc , f(D earlier than C earlier than A)=acd, we would quickly look for identifying rules in the example, learning that “and then” means putting after, “earlier than” means before and B means to do the action twice, even without being presented with the rules. People have a tendency to look for simple rules in examples, and they identify general rules (sometimes to general or strong) very commonly. People also commonly do induction and not only deduction, e.g. if f(abc)=d, and f(klm)=n then they will figure out that f(wxy) is likely to be z. Overall, I totally agree that rule-based learning is important in real-world and that AI models aren’t strong in this silk, but there could be the presentation of the motivation for that could be improved a lot.

* **Conceptual learning task definition**: The concept seeks to achieve  5 properties that the learner should possess. No justification is given about why these 5, how they were selected, how they complement each other into a common goal etc. I believe the definition should be reworked to be more cohesive. The definitions of some of the properties also weren't clear to me.

* **Experiments**: I would suggest to extend the experimental section, e.g. see the impact of the complexity of the rules and number of rules needed on the performance along different metrics, see generalization across multiple dimensions beyond only compositional generalization, do error analysis of what mistakes and trends are common in terms of accuracy and consistency and if any general behaviors and trends of models (vs. even human study) could be identified, etc.  

clarity,_quality,_novelty_and_reproducibility: **Clarity**: The writing quality is good but I think there could be significant improvements to the presentations of the ideas in the paper, as discussed above. Also a small comment beyond those above: would be good that figure 2 would be much closer to where it’s mentioned (page 1) compared to where it’s at right now (page 4).

**Novelty**: The explicit discussion and exploration of evaluation of systematic learning and the approach introduced is both novel and pursuing an important and underexplored direction in the field.

**Results**: A theoretical idea is introduced (conceptual learning, a method for creating benchmarks) and then a concrete application of it is explored (a dataset) and analyzed (experiments over language models). So the paper has the necessary components to convey a full story. I do recommend having more experiments and more baselines though (see comments above).

**Reproducibility**: The paper and especially appendixes provide a lot of detail that could allow reproducing the paper.

summary_of_the_review: I think overall the paper explores and important direction but could be improved in the presentation and extended with more experiments to make it much better and therefore at this time I recommend rejection but also wish to emphasize that I encourage the authors to keep working on the paper to make it better and then resubmit to a conference!
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces an improvement of a learning method from samples and rules, named conceptual learning task. In particular the aim could be that of debug a movie recommendation.

strength_and_weaknesses: - clarity
- experimental evaluation
+ incremental interesting approach
clarity,_quality,_novelty_and_reproducibility: The paper is an extension of an existing SCAN procedure. The clarity of the paper could be improved in many parts. The used machine learning approach is not reported in the paper and this make difficult to understand the results.
summary_of_the_review: The notation adopted in the paper to describe the rules is difficult to understand. 

Section 3 is very difficult to follow. For instance the authors say "we generate examples automatically using a Python program". How? what is the meaning of "picking a coherent set of basis rules"? Even the points (a-b-c-d) following the previous sentence are ambiguous and not clear.

After having read the first three section is not clear what is the machine learning task definition.

Another aspect regards the experimental evaluation. It seems that the proposed approach has been evaluated on artificial data in contrast with the example of recommendation system reported in the introduction.

Finally, it should be important to stress the learning process and how it is related to the cSCAN proposed approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission presents conceptual SCAN (cSCAN) as an instance of a conceptual learning task (CLT). CLTs, as defined in the manuscript, are tasks that contain a combination of examples and rules. Concretely, each example is defined as a tuple including a context, where the context is a set of rules and examples (potentially recursively, but in practice the inner examples have an "empty" context). cSCAN follows this formulation to introduce 4 different data sets, with smaller/larger set of rules and with/without explicit split for testing compositional generalization. Variants of the T5 transformer architecture are evaluated on the data set and demonstrated to fail in all but the simplest instances of the data set.

strength_and_weaknesses: ########### Strengths ###########
- The idea of constructing data sets where examples come annotated with a context, which itself contains both examples and rules, is interesting
- The running example of the movie recommendation system helps motivate and provide context to the reader for some of the design choices behind CLTs
- The connections to related literature, especially in terms of contextual data sets and meta-learning, are well established and help provide additional context for the usefulness of the data set


########### Weaknesses ###########
- The experimental evaluation is not as thorough as I would have liked to see, and it seems to not provide as much insight as would be possible
- The detail of how the transformers consume context are not explicitly provided
- While the approach to creating CLTs seems to be presented as a general-purpose data-set-generating method, it is largely unclear whether it would be easily applicable beyond SCAN

clarity,_quality,_novelty_and_reproducibility: The paper is fairly clear and easy to follow, and there are abundant appendices containing details for dissecting the results and reproducing the evaluations. The motivation for training ML models in such a way that they can consume both examples and rules is interesting, and while somewhat related to traditional knowledge-base approaches, novel in that it contains _new_ rules/contexts accompanying each example. This means that it is not sufficient for the agent to learn a fixed set of rules from a fixed data set, but it actually has to be able to interpret new rules at evaluation time and solve the task in the context corresponding to those new rules. Moreover, the notion of providing examples alongside the rules as additional context is also interesting and, to my knowledge, novel.

That being said, I believe that there is some room for improvement in a couple of places, in particular regarding the scope of the contribution and the experimental evaluation.

**Scope of the contribution**: in the early sections of the paper, the presentation seems to suggest that a contribution of the submission was a method for designing new CLTs and data sets for them given an existing data set. While this would have been fantastic, I don't believe that there's evidence later in the manuscript to substantiate this claim. Sure, it _may_ be possible to follow the process used for cSCAN to create other CLT data sets, but it is unclear exactly what that would entail. Given that only SCAN is explicitly demonstrated to be extensible into a CLT, I would encourage the authors to scope down their claimed contributions to more clearly state this. This way, the contribution wouldn't be a "recipe to derive a CLT from an underlying base task", but rather "the recipe we follow to derive a CLT from the underlying SCAN task". Perhaps, you could include a sentence indicating that it might be possible to use this recipe for _other_ tasks, but that this hasn't been pursued in this work.

**Experimental section**: The first (and simplest to address) comment I have is that it is not stated anywhere in the paper how the transformers consume the context rules/examples. I assume these are tokenized and passed as additional inputs alongside the top-level example? Please clarify in your response and in the updated manuscript. The second is a broader comment regarding the amount of insight from the results. At this point, there's essentially a single architecture evaluated, with some variants in the details of how attention is implemented and the number of parameters of the architecture. This provides _some_ insight, especially regarding the the impact of scale on the results, but it fails to address multiple other axes. In particular, one thing that might be interesting to consider is how neuro-symbolic / program synthesis approaches would fare in cSCAN. Since the rules are explicitly provided, I would think these methods wouldn't have an "unfair" advantage, as they typically have in other tasks where such approaches still require access to some rules. The authors could also consider including meta-learning approaches in their set of baselines. 


############## Additional feedback ##############

The following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.

Intro
- The idea and motivation seem interesting
- Even from this point, I was very much wondering what the systems are supposed to do with the rules. Are these going to be introduced as language input somehow? I hoped not (or at least, not just). Given their symbolic nature, some form of neuro-symbolic approach would be most suitable. Something like neural module nets or a program synthesis approach.

Sec 2.2
- I find the recursive definition of contexts odd, especially consdiering that the examples and even the introduced data set do not contain any context that is itself an example. Couldn't it then be defined as C \in 2^(Q \times R \times U), simplifying the definition? What is the advantage of including C recursively in the definition?
- I suggest that the authors include the word "defeasible" earlier, much like they did with "monotonically" in the examples from Sec. 2.1

Sec 2.3
- I think it would benefit the readability of this section to continue with the running example of the movie recommendation task.

Sec 3
- The first time the manuscript clarifies that the context varies generating multiple "tasks" seems to be when introducing cSCAN. Parhaps the authors could state something about this more explicitly earlier on for additional clarity? Particularly in the general CLT descriptions.
- Could the authors provide some insight into why two different versions of cSCAN are desireable/useful/interesting? I later came to see that this was due to the "scalability to larger rule sets". It might be worth including that here
- Interestingly, the MCD makes the problem harder in terms of generalizing to unseen top-level requests (I believe this is the same type of generalization as studied in SCAN), but it makes the problem of generalizing to new contexts easier because the contexts might be repeated between train/test. I wonder if the design makes it impossible to have these two simultaneously

Supplement
- Source code is promised but not provided at submission time

Typos/style/grammar/layout
- It's best to use "CLT" only after introducing the acronym (instead of jumping back and forth between "CLT" and "conceptual learning task")


summary_of_the_review: With all the above in mind, I am leaning towards recommending the acceptance of the paper, since I do think that the data set on its own presents a valuable contribution, and there are some insights derived from the empirical evaluation. I do encourage the authors to more accurately scope their contributions in the introductory sections and to provide additional details of the evaluation. If additional experiments are feasible during the review process, those would likely be most impactful towards turning this into a higher quality manuscript.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper extends the well-known compositional generalization benchmark dataset SCAN to cSCAN, driven by the idea of evaluating machine learning models' ability in semantic parsing with context and rule-like constraints.
strength_and_weaknesses: ### Strengths:
- The authors provided extensive details about how this dataset is generated, hence the reproducibility of this work is solid.
- This work is motivated by an important problem. Semantic parsing by deep learning can hardly make use of domain knowledge or context such as common sense knowledge and rules. The proposed cSCAN dataset could be a candidate for evaluating it.

### Weaknesses:
- The presentation could be improved. The motivating example in natural language QA is intriguing and convincing. However, the proposed dataset is an extended version of SCAN, i.e., a set of robot navigation instructions. While reading this paper, I found myself feeling disappointed and confused when I see the gap between them. And the natural language QA example doesn't help me understand how the "rules" work in the robot navigation problem. Maybe changing the motivating example to a problem that is more related to the cSCAN dataset could improve the clarity of this paper.
  - The notations in this paper are confusing. For example, $e_k= \langle i_k, o_k \rangle \in\mathcal{E}$ are examples, which consists of both inputs $i_k$ and $o_k$. However, the $i_k$ and $o_k$ are bost consist of the context $C$, and $C\in 2^{\mathcal{E}}$ is constructed by $e_k$. This is a recursive definition and makes me confused.
clarity,_quality,_novelty_and_reproducibility: The clarity of this paper should be improved. The quality of this paper is below average. The paper proposes a novel benchmark dataset that has good reproducibility.
summary_of_the_review: This paper proposes a novel dataset to evaluate neural networks' semantic parsing ability while considering knowledge-based constraints. The problem is important to the AI community, but the presentation of this paper should be improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors tackle the important problem of quantifying learning tasks that require a combination of induction and deduction. They do so by proposing conceptual SCAN (cSCAN) as a synthetic benchmark derived from SCAN for conceptual learning tasks (CLTs), i.e., tasks where language models are provided contexts and rules and asked for conceptual explainations.

The reviewers appreciated the direction the authors are taking and praised the construction of cSCAN from first principles. At the same time, they highlighted several weaknesses. These mainly include a weak presentation (missing some details about how contexts are obtained, how the program generation operates, over-complex notation, etc) and the lack of the general applicability to real-world scenarios for the proposed CLT learning task.

During the rebuttal, the authors greatly improved the presentation of the work e.g. by addressing concerns about notation and some missing details about the construction and evaluation from the main text. Reviewers acknowledged this, but still believed that the motivation behind using cSCAN as a benchmark can be lacking. More crucially, the construction of the benchmark and the evaluation procedure can be more streamlined and documented as to be made fully explicit and reproducible.

I personally believe the paper has a strong potential, but in its current form it is not yet ready for publication. I encourage authors to take the remaining reviewers' suggestions into consideration for a future submission.
justification_for_why_not_higher_score: While the paper presentation improved a lot during the rebuttal, the paper is still not ready for publication and the general use of the proposed benchmark is still questionable.
Decision: Reject