Conceptual SCAN: Learning With and About Rules
Authors: Nathan Scales
Nathanael Schärli
Abubakr Babiker
Yu-Han Liu
Mostafa Dehghani
Olivier Bousquet
Venue: Submitted to ICLR 2023 (2023)
Keywords: reasoning
compositional generalization
rule learning
semantic parsing
consistency

Abstract: The ability to learn from a mix of rules and examples and to reflect on the learned abstractions is an important aspect of human intelligence. At the same time, there is a lack of benchmarks that systematically test for this ability, which makes it hard to evaluate the degree to which it is present in state-of-the-art ML architectures. We introduce a method to systematically construct such benchmarks by using an example structure that allows us to explicitly provide and ask about rules that are relevant for the given task. We present a simple dataset that is constructed according to this method, and we use it to analyze the performance of a variety of T5-based machine learning models. We identify four challenge areas in this setup: maintaining consistency between learned rules and their application, scaling to larger rule sets, compositional generalization, and dealing with limited training data.

Under review as a conference paper at ICLR 2023
CONCEPTUAL SCAN: LEARNING WITH AND ABOUT RULES
Anonymous authors Paper under double-blind review
ABSTRACT
The ability to learn from a mix of rules and examples and to reflect on the learned abstractions is an important aspect of human intelligence. At the same time, there is lack of benchmarks that systematically test for this ability, which makes it hard to evaluate the degree to which present in state-of-the-art MI. architectures. We introduce novel task format for such benchmarks by using example structure that allows us to explicitly provide and ask about rules that are relevant for the given task. We present a simple dataset illustrating this format, and performance a variety of TS-based ML. models. We identify three challenge areas this setup: maintaining consistency between leamed rules and their application, scaling to larger rule sets, and compositional generalization.
it to analyze the
INTRODUCTION
Machine learning algorithms are typically designed to be able to learn functions from examples. This very general paradigm, but it does not explicitly capture some aspects of human learning. Humans, in contrast, are able to learn both by being shown examples of the task to accomplish and by being told rules or instructions about this task. They can even provide relevant rules others once they have learned the task from examples.
As realistic illustration of this ability, consider the task of personal assistant who, among other things, is expected to make movie recommendations based on the age and interests of user. Even for a task such as this that would currently be considered a standard use case for an example-based recommender system, as humans, we do not learn how to perform this task exclusively by observing examples of movie recommendations, Instead, we can accomplish this task much more efficiently by also taking into account relevant knowledge in the form of rules that have been communicated to us explicitly, i.e., by "learning with rules". For recommending a movie to girl called Anna, we may, among others, use the rules (and facts, which we consider a special case of rules) illustrated on the left side of Figure 1.
In addition to the ultimate goal of providing movie recommendations (e.g. "What movie could Anna watch?"), would also expect human to be able answer the intermediate questions shown
Knowledge
Questions
rated movies appropriate kids who less than 17 years old.
What movie could Anna watch?
The age person the time has passed since the person was born.
- Mission Impossible 6
* The current date September 28, 2022.
old Anna? years
Mission Impossible!
PG-13 rated action movies starring Tom Cruise.
Is Jerry Maguire appropriate for 14-year-alds? No
* Jerry Maguire an rated comedy starring Tom Cruise.
Will Anna like Mission Impossible 6? Probably
Anno was born in Jamuary of 2008 and Too Cruise her favorite actir.
Anna's friend? don't
saw Impossible
them.
Figure 1: Personal assistants answer questions using knowledge consisting of rules and facts. Note that the last knowledge bullet point above can also be construed as an example of the underlying "movie recommendation" task, while the other bullet points represent other relevant knowledge. The first bullet point a traditional "rule" that states conditional knowledge that can apply to many different movies. The second is concept definition, which be equivalently construed rule relating two different pieces of information about a person. The other bullet points are facts stated at varying levels of granularity.

---

Under review as a conference paper at ICLR 2023
the right side of Figure 1 1 ie., to "lear about rules" that are relevant to the ultimate task and would expect these questions to be answered consistently war. the provided knowledge and ultimate recommendation. These questions allow us to introspect the understanding of the assistant, e.g., to debug why a movie recommendation was not as expected.
Similar interactions between leaming from examples and learnnng with and about rules can also be observed in simpler synthetic tasks. Consider, for instance, the SCAN task of Lake Baroni (2017), which our work builds on. This task requires the learner to translate natural language commands (such "jump left twice") to corresponding actions (such as "LTURN JUMP LTURN JUMP"). The learner is presented with certain subset of some thousands of (command, action sequence) pairs during training and then expected to translate unseen commands.
This focus on learning purely from examples, while typical of most traditional ML tasks, differs from the way one would "teach" a human the task, and indeed how the authors of the SCAN paper "teach" the task to their readers. On the one hand, while humans are also adept at guessing rules from examples, rather than depending on thousands examples, they can often grasp the relevant rule from just handful of examples (Lake et al., 2015), as we as readers may find ourselves doing when seeing the handful of illustrative examples of SCAN provided by the authors the paper figures. More fundamentally, however, rather than expecting readers to learn the translation function purely from examples, the authors provide this function in much more direct and efficient fashion using a set of interpretation rules like those in Figure 2. The explicit nature of the provided rules has the additional advantage that it allows us deduce the translation of given command by applying the translation rules rather than having to always speculatively induce the translation by generalizing from a set of examples.
[walk] WALK
[1: left- - --TURN[s]]
[x [wice] = [m10a]
Figure 2: Examples of SCAN interpretation rules from Lake & BBaroni (2017).
In this paper, we introduce conceptual learning tasks (CLTs), which are type of learning task that specifically designed to evaluate the combination of such inductive and deductive learning, and we make the following contributions:
We define the notion of CLT (Section 2).
We present a first simple instance of CLT called Conceptual SCAN (cSCAN), which is syuthetically-constuveee
conceptual learning variation of SCAN (Section 3).
We formalize metries to measure a learner's performance on cSCAN, including a novel measure- ment of consistency between learned rules and their application (Section 4).
We analyze the performance of baseline ML architectures on cSCAN and identify three challenge areas: consistency, rule set size, and compositional generalization (Section 6).
We provide the code used in generating cSCAN, constructing compositional generalization splits, and calculating consistency from experiment results.'
2 CONCEPTUAL LEARNING TASKS (CLTs)
2.1 DESIRED PROPERTIES
Motivated by the use case from the introduction and our goal of evaluating learning with and about rules, we interest ourselves in tasks with the following properties, which we formalize in the following section.
1. Context. The leamer answers requests based on some explicit knowledge (context), which consists of "examples" that directly determine the replies of certain requests and "rules" that provide indirect information to do so. Part of this knowledge varies across contexts (c.g.. transient knowledge about Anna's preferences or country-seccific rules about movie ratings).
2. Request. In addition requests corresponding the ultimate goal, (e.g.. "What movic could Anna watch?"), we can ask the learner whether an intermediate rule holds given the context (e.g., "Is Jerry Maguire appropriate for 14-year-olds?"). This allows us to test whether the learner "understands" the rules by checking for consistency between rules that the learner claims to be true (or false) and their application to answering the ultimate requests.
'To be released GitHub upon paper acceptance. For an overview, see Appendices F and G.
2

---

Under review as a conference paper at ICLR 2023
3. Output. The leamnrr needs to indicate in the output whether reply follows deductively and thus monotonically from the context (e.g., "How old is Anna? + 14") or whether it requires generalizing the context inductively (iee "Will Anna like Mission Impossible 6? = Probably"), which would imply that the reply could change more information were to become available (i.e., that the reply is defeasible). Also, the learner needs identify if request cannot be answered based on the given context (e.g.. "Who is Anna's 's best friend? = I don't tt know").
4. Compositionality of rules. The rules have compositional structure, which means that it is possible to determine the meaning of an unknown rule from its syntax we are given the meaning of a sufficient subset of rules. (E.g,, we humans understand the meaning of the rules shown in Figure because of the compositional syntax and semantics of natural language. Even if we have never seen these exact sentences before, we know what they mean based on our exposure other sentences built from the same natural language building blocks.)
2.2 STRUCTURAL DEFINITION
As a way of providing a concrete but generic task format that satisfies these properties, we define conceptual learning task (CLT) supervised learning task T with the following structure. (See Appendix A for a discussion of the design choices.)
The task T= ey) is a finite set of examples et E E, where & = =2 Qx Rxu denotes the set of possible examples.
Each example e (Kk is quadruple consisting of a context C. 2 request qu E Q, a reply r E R. and a qualifier uy EU. Of these, the context and request together form the input of the ML system, while the reply and qualifier together form the output. (See Appendix O for details on the exact format which these are input and output by our T5 baseline systems.)
The set of possible requests Q can be partitioned into rule requests QR Q (i.c., requests that ask whether a certain rule holds) and non-rule requests Q. Q.
The set of possible replies R must contain dedicated elements representing true (1), false (0), and unknown (?), which are the only valid replies for rule requests q E QR-
The set of qualifiers U4 (M.D) indicates whether the reply follows monotonically from the context (M) or whether it is defeasible (D).
Each context 2 consists of set of context examples e; e; (q.r)). which represent either examples of an underlying task or other relevant knowledge expressed in "example" form. Note that unlike the top-level examples, these context examples do not themselves contain a context qualifier, as for the purposes of this paper, we take all context examples to be unconditional and monotonic. (See Appendix C for possible generalization of this definition.)
Note that even though the context is strictly a set of examples, it may still contain any rule q E QR by means of including the example (4, 1), which asserts the rule q to true.
As a more complete illustration, Figure 3 shows a few examples from the eSCAN task, which is CLT that we introduce in Section 3, while Appendix shows the motivating example represented as a CLT.
2.3 CONSISTENCY REQUIREMENTS
In addition satisfying the structural definition, for the purposes of this paper, we require CLTs to be logically consistent, which means that each example and the example set as a whole must be non- contradictory. For instance, if CLT contains monotonic example (C.4,r.M), may not contain an example (C'.q.'' M) where CCC' and r as this would violate monotonicity. (Or using the example from the introduction, this would mean, for instance, that for a given context the task contains "How old is Anna? = 14" and "Is Jerry Maguire appropriate for 14-year-olds? "Is Jerry Maguire appropriate for Anna = Yes".) While this requirement could be relaxed in practice, consistency of the task is helpful, as it enables to precisely measure the consistency of the leamner.
No", it should not contain
For the task to be logically consistent, requires at minimum that the monotonic examples adhere to the axioms of propositional logic. (See Appendix D for formalization of the consistency requirements.)
3

---

Under review as a conference paper at ICLR 2023
Note that even while requiring a CLT to be logically consistent, we still allow for the common phenomenon in which rules are stated in a form that is only "usually" true. Exceptions to a rule are allowed so long as the rule is assigned the qualifier "defcasible" (D).
2.4 CLASSIFICATION OF EXAMPLES
The structure of CLTs allows us to classify examples according to the following dimensions.
Request g Q: Rule request (q Qx) vs. non-rule request (g Qx)
Reply re R: Known vs. unknown (?), and, for rule requests, true (1) vs. false (0)
Qualifier uEU; Monotonic (M) vs. defeasible (D)
Each class of example should be reasonably well represented in conceptual learning dataset.
3 CONCEPTUAL SCAN (CSCAN)
One benefit of abstracting out key properties of interest from motivating case into the definition of a CLT is that we can now seek to study the capabilities current ML methods on this family of tasks by starting with very simple CLT instance, which illustrates the key dynamics of a CLT in as pure as possible a form. In particular, it gives us a basis by which we can construct simple synthetic task that enables exploration of MI system performance complexity. In this section, we present one such "simplest possible" CLT, named Conceptual SCAN (cSCAN), cSCAN is conceptual learning adaptation of the SCAN task (Lake & Baroni, 2017), which was itself originally presented as a kind of "simplest possible" compositional generalization task. We construct cSCAN according a recipe illustrating one possible way of deriving CLT from a base task.
CLTs,, while carefully controlling the task
3.1 BASE TASK: SCAN
SCAN is a task where natural language commands (such as "jump left twice") are translated into sequences of actions (such "LTURN JUMP LTURN JUMP"). SCAN was designed evaluate compositional generalization using non standadd train-test splits, which is one of the themes we explore in this paper. In addition, SCAN is an attractive choice of base task for first conceptual learning benchmark because can be generated automatically from simple set of rules: a phrase-sttne shrase-structure grammar for generating the valid input commands and set of interpretation rules that specifies how to compute the action sequence for each command (see Figure 2 and Appendix E.
3.2 CONSTRUCTING CSCAN
In contrast SCAN, which tests whether a learner can learn one specific translation from commands to actions, the goal of cSCAN is to test whether the learner can learn to perform a family of SCAN-like tasks using knowledge that consists an arbitrary mix of rules and examples.
Note that moving from single task to family of related tasks is essential for cSCAN because it forces the learner to take into account the knowledge provided by the context rather than just memorize behavior that is constant across all examples. In our original motivating example, this corresponds to the f'ct that we do not want a learner to make movie recommendations in a single, fixed scenario, but rather based on knowledge that may differ from person to person and evolve over time.
cSCAN is constructed using the following recipe.
Step 1: Accommodate the base task. A CLT subsumes the base task. This means that any input of SCAN is a valid request q E Q, and any output of SCAN is a valid reply e R.
Step 2: Make some of the relevant knowledge available as explicit rules. We choose which part of the relevant knowledge we want to be able to talk about explicitly and vary across examples. For cSCAN, we choose talk only about interpretation rules like those shown in Figure 2. This means that any interpretation nule is a valid request / R, which allows us to assert such a rule in the context (by adding the example (r,1) to the context) and teach / ask the leamer whether any such rule holds. CLTSs require that explicit rules have
4

---

Under review as a conference paper at ICLR 2023
[[walk] WALK, [[run]
lett] LTURN, LLURIN [=.]. 1, -n 1) (F, and [x
(walk reice, WALK WALK). (run left toice. ETURN LTUNN RUN, (walk twice and nun, WALK WALK RUN)
(c,, walk loft and nun, LTURN WALK RUN, MD
CC run twiss RUN RUN, D)
(C, run tike and wilk RUN RUN WALK, D:
(C,, Lx, left and run] [LTURN x RUN, 1, M)
cC, Dr. swicc] [= 1. D:
(C, Ea twice and - 6>] (166]. D:
([walk] RUN, 8t [run] - JUMI, [[ump] LOOX, D. (Ium kitt RIURN, D) left] [x RTURN. D. I and l 1 D; (walk twiee, RUN RUN RUN), Crun left twiee, JUMP KTURN JUMP RTURN JUMP RIURN), (juap twiee and walk, RUN LOOK LOOK RUN), (walk (fries, RUN RUN RUN RUN), (run three, JUMP), (lum left Eiriee, RTURN RTUUN))
J
(umpjump ledt twice and walk, RUN LODK RTURN LOOK RTURN RUN, D)
<C, jump srrice. 7. D
(C,, La. left bwice and [s RTURN Ex.] RIURN I. D.
(C, Er, Uhrice] [x [x. O,
Figure 3: Hypothetical cSCAN examples based on two different contexts that contradict each other.
compositional structure (Section 2.1). This is the case for cSCAN because the meaning of the interpretation rules can be determined using a compositional grounding function that we provide in Appendix E.
Step 3: Generate or curate examples. For cSCAN, generate examples automatically using Python program, which we summarize here and describe in detail in Appendix F. As a first step of generating each example, we create a context by first picking coherent set of interpretation rules like those shown in Figure 2 (or in Figure 4 from the appendix) and then choosing which of those rules to (a) provide explicitly in the context, (b) illustrate implicitly through context examples, (c) illustrate insufficiently or not at all, or (d) contradict in one or more cases. Once we have fixed a context, we choose request, reply and qualifier that, together with the context, satisfy the consistency criteria stated in Section 2.3 and agree with the task's assumed inductive bias for distinguishing between a rule being "defeasibly true" vs. "unknown" (see Appendix E.4).
We make sure that the different example classes are evenly represented (see the statistics in Section 3.3) and that, for simplicity, contexts contain only unconditional, monotonic examples that do not conflict with one another. We provide detailed specification of cSCAN in Appendix E.
3.3 THE cSCAN DATASET
Examples. Figure 3 shows some hypothetical cSCAN examples in human- notation. (We chose these examples for simplicity and conciseness. For a sample of actual cSCAN examples, see Appendix I. For examples in the exact format in which they are presented to the baseline systems, see Appendix O.)
These examples are based on two contexts (C₁ and C2) that are incompatible (e.g., they define the meaning of [[walk] differently) and correspond to different SCAN variations. Each context contains some explicit rules (e.g.- [xy and x2]] x=2]][[ [x>]=[xx]]eee [xa]][x]]] in C1) as well as rules that are illustrated implicitly via examples (e.g.. [x] twice] = [x>]][<]]]]] in C1). The qualifier of the examples indicates whether any implicit rules are needed to determine the reply (in which case the qualifier is D).
In context C2. we do not provide explicit rules for [x] twice] as well as [xx thrice]. While the rule [x] twice]] = [x1] [ is expected to be induced from the provided examples, there is no obvious rule that can be induced for [x] thrice]. As consequence, we expect the learner to reply "unknown" (?) for the request "jump thrice". At the same time, we expect the learner to reply "false" for the rule
[x] thrice]| = [x1] [x]] because there is an example in the context that contradicts it.
Rule space variants. In order to cover a range of rule set complexities, we construct two versions of cSCAN (cSCAN-B and eSCAN-X), using different sizes of rule space (Table 1).
cSCAN-B (short for "cSCAN- ceSCAN-Basee) uses a fixed ahrase-structure grammar equivalent to that of the original SCAN task, as reformulated in Nye et al. (2020) using 14 interpretation rules. Action sequences are kept short by allowing token or variable to be repeated at most twice in an interpretation rule's output sequence. This ensures that eSCAN-B examples do not exceed input size of 2048 tokens or output size of 256 tokens in our bascline models.
eSCAN-X (short for "<SCAN-Extendedd) is based on a richer grammar space, which extends the original SCAN phrase-structure grammar with additional terminals, variations of existing rules, and an additional level that enables adverbs. Output sequences for individual interpretation rules are allowed to contain up to 4 repetitions of any given token variable, which is the same in original SCAN, but longer than in cSCAN-B. To keep context sizes manageable, we apply rule sampling for each context, so that the number of interpretation rules actually used in any given context is the same in eSCAN-B, and apply a limit to the
5

---

Under review as a conference paper at ICLR 2023
Table I: cSCAN rule space variants. cSCAN-B uses a fixed phrase-structure grammar equivalent to that of the original SCAN task, while keeping action sequences short. cSCAN-X based on richer grammar space, while using nule sampling to keep the number of rules used in any given context equivalent to the number of rules in cSCAN-B and the reformulation of original SCAN.
Variant
Command grammar Depth
Action sequences
Context
Max # tokens Input
Temiiaas
Max length
Underlying rules
Output
eSCAN-B
6
13
4
14
2048
256
<SCAN-X
7
36
8
14
4096
512
Reformulated SCAN
14
Table 2: Key statisties of the main cSCAN datasets. For full details, see Appendix H.
cSCAN-B Random train
cSCAN-X Random
cSCAN-B MCD
cSCAN-X MCD
test
train
test
train
test
train
test
Number of examples
100,000
100,000
100,000
100,000
100,000
10,000
99,999
10,000
Number of contexts
1,000
100
1,000
100
11,921
6,509
7,965
4,599
Request: Rule (%)
50.1
50.0
50.1
50.0
72.5
64.0
59.8
67.4
Request: Non-rule (%)
49.9
50.0
49.9
50.0
27.5
36.0
40.2
32.6
Qualifier: Monotonic (%)
46.3
46.0
50.0
48.0
26.9
36.6
37,0
38.9
Qualifier: Defcasible (%)
53.7
54.0
50.0
52.0
73.1
63.4
63.0
61.1
Reply: Unknown (%)
13.3
13.1
13.2
13.6
52.2
35.2
39.7
36.3
Reply: Yes (%)
21.8
21.8
21.9
21.5
12.0
18.1
12.7
18.5
Reply: No (%)
21.7
21.9
21.7
22.1
12.1
18.2
12,9
19.2
cumulative output sequence size across all interpretation rules used in the given context. These measures ensure that cSCAN-X examples do not exceed input size of 4096 tokens output size of 512 tokens.
Splitting methods. For each of the two sizes of rule space, we prepare datasets based on two splitting methods: random and maximum compound divergence (MCD) (Keysers et al., 2020).
For the cSCAN Random variants, we generate 1200 contexts and split these contexts randomly into a train set of 1000 contexts and validation and test sets of 100 contexts each. For the train set, generate 100 top-level examples per context, while for the validation and test contexts, we generate 1000 top-level examples per context each context for use in calculating the consistency metric. This leads to a total of 100K top-level examples in each of the train, validation and test sets.
to provide denser signal of potential implications among the examples of
For the cSCAN MCD variants, we apply a variation of the MCD splitting algorithm of Keysers et al. (2020) in order evaluate the ability of the system compositionally generalize new rule combinations, which we consider a stronger test of the system's is ability to "understand" the meaning of the rules and to apply them correctly in new situations. Specifically, each, yielding a pool of 1.2M top-level examples. We then annotate each top-level example with set of atoms and compounds based on the thrase-structun grammar rules that were composed form the top- level example request, and we split the set of top-level examples in such way as to maximize the divergence the distribution of compounds between train, validation, and test, while keeping the distribution of atoms nearly the same. Similarly to Keysers et al. (2020), down-sample during the splitting process for more effective control of the distribution divergences, leading to total of 100K top-level examples train (comparable to cSCAN Random) and 10K tepel examples in each of validation and test.
start by generating 12K contexts with 100 top-level examples
Statistics. Table gives an overview of the key statistics the cSCAN Random datasets and representative MCD datasets. (See Appendix H for details of other cSCAN dataset variants.)
We construct the examples such that all the classes discussed Section 2.4 are well represented. In particular, the Random datasets contain roughly equal numbers of rule non-rule, well examples with replies "true" (1) vs. "false" (0). There are slightly more defeasible examples than there are monotonic examples because "unknown" (?) examples are always qualified as defeasible.
Note that compared to the random split, the splitting method used the cSCAN MCD variants leads to somewhat less balanced dataset in terms of example classes, although each of the classes is still well

---

Under review as a conference paper at ICLR 2023
Table 3: Accuracy vs. consistency. Learner A is more accurate, but learner is more consistent.
Golden
Learner
Learner B
[1., twice] = x I
Gi. twice = 1 x
Ci. =
[Ci,[walk] = 1ALK,1)
Cr.[walk]- WALK,1)
[[[[walk]=
(Ci. twice, WALK WALK)
walk JUiP JUMP)
wa.w JUMP JUMP JUMP)
covered. Also, while it leads to a challenging split in terms of generalization to new top-level request patterns, is potentially easier than the random split in terms the contexts shown, as we do not prevent the same context from appearing in train and test, and due to the effect of down-sampling from a larger context pool, the total number of contexts that are shown in the MCD train set is an order of magnitude greater than those shown in the random train set.
4 METRICS
Accuracy Our primary accuracy metric is example-level accuracy, where credit is given only when the reply + qualifier exactly matches the ground truth. For more nuanced enror analysis, we secondarily track several accuracy variants that give credit for partially correct outputs (see Appendix N).
key aspect of learning with rules is that a learner does not just learn how to mem- orize and recite rules, but is actually able to combine and apply them consistently, For instance in cSCAN, if the learner believes that the rules [X] twice] = [> and [[wak]] = JUMP hold, a consistent learner should also believe that all combinations and applications of these rules hold, such as [walk twice] = JUMP JUMP JUMP.
Consistency
Note that unlike accuracy. this notion of consistency is not judging the correctness of individual predictions. Instead, it judges to what degree a whole set of predictions is consistent within itself. While a perfectly accurate learner would also be perfectly consistent, when accuracy is low moderate, consistency can potentially be quite orthogonal to accuracy.
As an illustration, consider Table 3, which shows golden example set as well the predictions of two learners A and B. Learner A is more accurate because it gets 2 examples correct, whereas learner B gets none of the examples correct. At the same time, leamerr A is not consistent, because it is not able to correctly apply the two rules that it believes in to derive the example (walk twice, WALK WALK). In contrast, learner example (walk twice, JUMP JUMP JUMP).
perfectly consistent, as it correctly combines the nules it believes to derive the
To capture this notion of consistency, we introduce for any set of predictions E E the consistency metric C(E), which is the percentage of subsets of E that contain logical implication, in comparison to the number of subsets of E that contain an implication or a contradiction. This means that C(E) is value [0,100]: C(E)= 100 says that the set E is perfectly consistent, while C(E)= 0 says that E is completely inconsistent. (See Appendix D for a formalization of this metric and Appendix G for practicalities of calculation.)
For learner A, the example (Cı.walk JUMP JUMP) contradicts the rules (C1, [x twice]] [[x]] [x]].])
and (Ci.[walk] = WALK,1), which means that the consistency of learner A is 100- = 0. For learner B, the example (C1, walk twice,JUMM JUMP JUMP) is implied by the rules (C1,[x twice] - =[x]]x]]]x]]]]] [x]
and (C1,[walk) = JUMP,1) 1), which means that the consistency of learner B is 100- 100.
5 BASELINES
As baselines, we evaluate variations of T5 (Raffel et al., 2019), a Transformer encoder-decoder model (Vaswani et al., 2017), which when pre-trained on natural language served as strong baseline on the original SCAN task (Furrer et al., 2020; Csordás et al., 2021; Ontañón et al., 2021).
The most computationally and memory intensive of the T5 architectures that we evaluate is the standard T5 architecture, which applies full self-attention in the encoder, in addition to self-attention and cross-attention in the decoder. We refer to these models as simply 75.
Motivated by the potentially large context size associated with CLTs, we further evaluate two variants of T5 that were designed to more efficiently scale to longer input sequences. Long75 (Guo et al., 2022) reduces the computational load of attention in the encoder by applying local attention within a sliding window (Ainslie et al., 2020). LongT5-TGllbbll (Guo et al., 2022) extends LongT55 with a local-global
7

---

Under review as a conference paper at ICLR 2023
Table 4: Test results on cSCAN random splits by model size: S (Small), B (Base), and L (Large).
cSCAN-B Random
cSCAN-X Random
Accuracy
Consistency
Accuracy
Consistency
Model
Pretrain
S
S
B
B
L
T5
Trie
92.5
92.6
96.5
71.8
71.3
87.5
68.3
71.0
81.6
30.1
32,4
43,5
False
19.3
17.9
0.0
0.0
16.8
17.2
0.1
0.0
LongT5-TGlobh
False
20.0
17.0
0.2
0.0
18.0
18.8
0.2
0.1
LongT5
False
19.5
21.4
0.0
0.1
18.1
16.7
0.1
0.1
T5 w/o Context
True
26.8
0.1
23.8
0.4
attention sparsity pattern (Ainslie et al., 2020; Zaheer et al., 2020). The global attention is designed to facilitate propagation of attention across the full input sequence within two hops, in comparison with pure local attention, in which propagation of attention is limited by the size of the attention window.
For each architecture, we evaluate at minimum two sizes: Small (60M parameters) and Base (220M parameters). For the perf-perfo T5 architecture, we further evaluate on size Large (770M parameters). We report results for both pre-trained versions of the long variants, trained and non non-pretrained versions of standard T5, but only non-pretrained failed to find converging setup for the pre-trained versions (see Ap- pendix L for details). For comparison, we also evaluate a naive variant T5-Base (75 who Context), which considers only the request, while ignoring the context. Additional details of the baseline configurations are provided in Appendix L and of the input-output format in Appendix O.
6 EXPERIMENTAL RESULTS AND DISCUSSION
6.1 CONSISTENCY AND SCALING TO LARGER RULE SETS
As first of experiments, we compare baseline performance the random splits of the eSCAN-B and cSCAN-X datasets. The results are shown in Table 4.
On the smaller eSCAN-B rule space, it can be seen that the provided 100K examples sufficient for pre-trained full-attention Transformer to achieve accuracies in excess of 90%, with accuracies increasing steadily with model size. Even in this relatively simple setting, however, several challenges can be observed. First, it appears that appropriate pre-training of the model is critical, as all the Transformer variants when trained from scratch managed lear only superficial statistical correlations, as evidenced by them failing to outperform a naive baseline. Second, regarding LongT5 and LongTS-TTGlobll, while it is possible that performance could be improved through more thorough hyperparameter tuning, our initial results show these to struggle on the conceptual learning task. Specifically, cenprrtrrineed versions suffer from the same poor performance as non-prentee pretrained T5, while when fine-tuning from an existing checkpoint, we were not able to find converging setup. One possible explanation is that unlike document summarization tasks, for which LongTS produced strong results (Guo et al., 2022), CLTs may depend heavily on full attention over the context. If so, this could pose challenges in scaling real-worth conceptual learning tasks with even larger contexts. Third, while consistency scores for the evaluated models correlate roughly with accuracy, significantly more headroom seen in consistency, with even the best-performing TS-Large scoring under 0.9, while the naive baseline and all non- con-pretaained models score close to O.
On the cSCAN-X rule space, accuracy drops significantly for all sizes of T5, suggesting that scaling to larger rule sets will be a challenge. Consistency continues to correlate with accuracy for these models, but drops rapidly as performance degrades. Non-pretrained models continue fail outperform the naive baseline.
6.2 COMPOSITIONAL GENERALIZZTION
As second set of experiments, we evaluate the ability for baseline solutions to compositionally generalize on CLTs using the cSCAN-B MCD and cSCAN-X MCD datasets (Table 5).
Prior research semantic parsing tasks has shown that while pre-trained Transformers exhibit strong performance on specialized cases of compositional generalization, they tend to struggle with more complex forms of compositional generalization, reflected in low performance on MCD splits when appropriate notion of "atom" " and "compound") is identified (Furrer et al., 2020). Here we show that in the context of conceptual leaming task, one form of compositional generalization that is challenging for T5-based
8

---

Under review as a conference paper at ICLR 2023
models is generalization to syntactie patterns in the request, even in the relatively easy setting where the same contexts could appear in train as in test.
Specifically, as can be seen in the cSCAN-B MCD results, when switching from random MCD split, accuracy drops from over 90% to less than 70%, even for the largest of the pre-trained T5 models, illustrating that composi- tional generalization is a challenge for these models, independently of the size of the rule space. Accuracy on eSCAN-X MCD is roughly sim- ilar to both cSCAN-B MCD and eSCAN-X Random, suggesting that the challenges of compositional generalization and rule space size do not necessary compound.
Table 5: Test accuracy on different cSCAN MCD splits by model size: (Small), (Base), and (Large).
cSCAN-B MCD
eSCAN-X MCD
Model
Pretrain
S
B
L
S
B
L
T5
True
53.8
54.7
67.6
70.7
69.2
76.4
False
40.1
38.4
41.6
39.9
TongTS-TGlobal
False
40.6
39.3
41.0
40.9
LongT5
False
40.7
38.0
40.9
41.2
TS w/o Context
True
46.2
42.7
Note also that while accuracies for non oonerrrteddee models are somewhat higher on the MCD splits than on the random splits, this is not actually sign of stronger performance, but rather simply an artifact of the mix of example classes that occur in the different splits, due to the down-sampling that is performed when constructing the MCD split. As shown in Table 8, a side effect of this splitting algorithm relative increase in the proportion of defeasible rule examples vs. monotonic or non-rule examples and in the proportion of examples with "unknown" as the reply. This leads to an increase in the accuracy achievable by the naive "T5 w/o Context" baseline. On the MCD splits, like the random splits, none of the non- non- onnerrrre models manage to outperform the naive baseline.
For the MCD splits, we do not report consistency metrics, as due to technicalities of the MCD splitting algorithm. there ended up being insufficient signal for logical implications among the test examples, leading the consistency metric to be undefined in most cases. (See Appendix FF for details.)
7 RELATED WORK
Here we discuss the most closely related lines of research. See Appendix K for more related work.
Tasks providing knowledge context. In representing the input of CLT request paired with context, we build on a long tradition of QA and reasoning task formulations that provide knowledge relevant to task via various forms of context, such as text passage (K (Kwiatkowski et al., 2019; Weston et al., 2015), set of natural language statements (Talmor et al., 2020), knowledge graph fragment (Sinha et al., 2020), or grid world (Ruis et al., 2020). Our formulation of CLT is novel in adopting a set-like context mixing rules and examples, which varies materially across examples.
Meta-learning. The presence of examples within the context an example gives CLTs a nested structure that allows us to view CLTs through the lens of meta- veeaa-lernnngg or "learnin to learn (Thrun & Pratt, 1998). In this view, top-level examples that share the same context correspond to to an episode where the context examples are the training examples and the top-level examples (w/o the context) are the test examples. Closely related to cSCAN are two pieces of work that apply meta-leaming to SCAN (Lake, 2019; Nye et al., 2020). Our approach differs in that include in the context mixture of rules and examples, and we use the seathettcally-genertteee contexts to define a new task, rather than as a means improve accuracy on the original SCAN task.
8 CONCLUSIONS AND FUTURE WORK
In this paper, we presented the cSCAN benchmark as first instance of a "conceptual learning task" (CLT), following a task format motivated by a personal assistant use case. Through experiments on baseline solutions, we identified several challenge areas with headroom for improvement. As next steps, we are interested in exploring solutions to CLTs, including prompting of large language models, syurolyymbolic solutions, and improved ways of handling large like contexts. In parallel, are interested in exploring CLTs based on a wider range of base tasks and rule formats, including non- non-synthetic tasks and tasks that draw on full KB as context.

---

Under review as a conference paper at ICLR 2023
9 REPRODUCIBILITY STATEMENT
Experiments Appendix L describes the details of the baseline configurations that we used, together with other details of the environment which we ran the experiments reported in this paper, while Appendix provides details of the input-output format. Upon paper acceptance, we plan to release on GitHub both the cSCAN datasets and the code needed to reproduce the experiments.
Dataset generation The cSCAN datasets themselves were synthetically generated using configuration- driven Python program described in Appendix F, which we also plan to open-source, together with the specific configurations used for each of the cSCAN datasets. While regeneration of the datasets is not necessary for reproducing our experiment results, researchers can use this code to generate new conceptual leaming datasets based either the existing cSCAN grammar spaces or modified grammars. When the code is run with the provided configurations, it can reproduce the generation of datasets with statistically comparable content the official cSCAN datasets.
REFERENCES
Joshua Ainslie, Santiago Ontañóón, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang. and Yang. Ete: Encoding long and structured data in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020), 2020.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. International Conference on Computer Vision (ICCV), 2015.
Dzmitry Bahdanau, Harm de Vries, Timothy JO' O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville. Closure: Assessing systematic generalization of clevr models. arXiv preprint arXiv: 192.00589/ 2019a.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In International Conference on Learning Representations, 2019b. URL https:
id=HkezZnAA9Y
Peter W Battaglia, Jessica Hamrick, Victor Bapst, Alvaro Sanchez-Gonzaler Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1806.011.. 2018.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: anulyzing text with the natural language toolkit. " O'Reilly Media, Inc.", 2009.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2015.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary. Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: /tt///gittt //ggttuubbb.. com/ google/jar jax.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasié. MultiWOZ. large-scale multi-domain Wizard-of-O dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016-5026, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. (0.18653/v1/DI 1ttii//DDDI-111-1 URL https: / /aaclalammm hottoyolyy org/ D18-1547.
Giovanni Casini, Thomas Meyer, and Ivan Varzinzzak. Contextual conditional reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, (557))6254-6266 6261, May 2021. URL nattps://ojs. aaai .org www.ex.piip//////artttteee
66777.
10

---

Under review as a conference paper at ICLR 2023
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. Compositional generalization via neural-symbolic stack machines. Advances in Neural Information Processing Systems, 33:1690 13:1699-1701, 2020.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et Rethinking attention with performers. arXiv preprint arXhv:2009. 14794, 2020.
William W Cohen. Grammatically biased learning: Learning logic programs using explicit antecedent description language. Artificial Intelligence, 68(2):303-366, 1994.
William W Cohen. Fast effective rule induction. Machine learning proceedings 1995, pp. 115- 113-123. Elsevier, 1995.
Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. The devil is in the detail: Simple tricks improve systematic generalization of transformers. CoRR, abs/2108. 12284, 2021. URL https://arxii org/ abs/ 12284.
Ido Dagan, Oren Glickman, and Bemaaro Magnini. The pascal recognising textual entailment challenge.
In Proceedings of the PASCAL Challenges Workshop on Recognising Textual Entailment, 2005, URL http: wwwp://wwwwccooo/ cs wwu.biu..uuu....... ac iil/gl//l/// wikaii/maaeeee kmaa/rte00// rte05/.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819. 2018.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprini arXiv:/903.00161, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Daniel Furrer, Mare van Zee, Nathan Scales, and Nathanacl Schärli. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. arXiv preprint arXiv:2007.089700, 2020.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Dan Goldwasser and Dan Roth. Learning from natural instructions. Machine learning, 94(2):205 2014.
Jerzy W. Grzymala Busse. Data Mining and Knowledge Discovery Handbook, chapter Rule Induction, pp. 249-265. Springer, Boston, MA, 2010. ISBN 978-0-387-0982227, URL https: //iii.i../ org/ 10. 1007 / 978 0 00-88709822344-00
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.0033 2021.
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 724-736, 2022. URL https:: https::/acllatthoolog haapa/aacaattto...............aa org/2022. wwwdingssnninac...... 55.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International Conference on Machine Learning, pp. 3929-3938, PMLR, 2020.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: neural network library and ecosystem for JAX, 2020. URL http:
/ wttchub.com/gcoo e/ flax.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisen- schlos. Tapas: Weakly supervised table parsing via pre-training. arXiv preprint arXi::2004.02349 2020.
11

---

Under review as a conference paper at ICLR 2023
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-leariing in neural networks: A survey. IEEE transactions on patter analysis and machine intelligence, 44(9):51495551 2021.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. CVPR, 2019. URL 09506. pdf.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Fei-Fei Li, Lawrence C. Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. URL https: : / /awxiv.vvvv./////////..///aaa ottpiiffrrgppf//// /11.. 06890. pdf.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Mare van Zec, and Olivier Bousquet. Mcasuring compositional generalization: A comprehensive method on realistic data. In JCLR, 2020. URL https: aarxivii org/ abs /1912. 09713 pdf.
Kevin C. Klement. Propositional logic. Internet Encyclopedia of Philosophy. 2004.
Taku Kudo and John Richardson. Sentencepiece: simple and language independent subword to- kenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Pro- ceedings of the 2018 Conference on Empirical Methods Natural Language Processing. EMNLP 2018: System Demonstrations, Brussels, Belgiun, October 31 November 4, 2018, pp. 66-71. Association for Computational Linguistics, 2018. doi: 10.18653/11d11-2202.. URL https:
orog 111118666vll /d18 2-202..
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural ques- tions: a benchmark for question answering research. Trunsactions of the Association of Computational Linguistics, 2019.
Brenden M Lake. Compositional generalization through meta -o-ennce-t--sqeence learning, In Advances in Neural Information Processing Systems, pp. 9788-9799 2019.
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of grquence-to-sequene recurrent networks. 2017.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 35066266) 2015.
Brenden M. Lake, Tal Linzen, and Marco Baroni. Human few-shot learning of compositional instruc- tions. In Ashok K. Goel, Colleen M. Seifert, and Christian Freksa (eds.), Proceedings of the 41th Annual Meeting of the Cognitive Science Society, CogSci 2019: Creativity + Cognition Compu- tation, Montreal, Canada, July 24-27, 2019, pp. 61 611-617. cognitivesciencesocieyyyyyv 2019, URL https: //////mmmmmeeeengg org/ www.cic0cc2./c 201 http///pppppp//////// //i//xin....././//// html..
Hector J. Levesque, Emest Davis, and Leora Morgenstem. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR' 12, pp. 552-561, AAAI Press, 2012. ISBN 978-1-57735-560-1 URL https: //cs www..yduuuuueuuu //c....p..eeaaaaaaall//////aeaee
wwtps///.K2222222... davi http://///epppeeee////2222222
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. A logic-driven framework for consistency of neural models. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), EMNLP/JCCNLPPPP (1), pp. 3922-3933. Association for Computational Linguistics, 2019, ISBN 978-1-950777-99-1. URL bett:://dbli uni-trierr de/ddb caab//oof/// ht////onff//m//// httpp//mmmmmm1mmmm
htm. GMS11
Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang. Compositional generalization by learning analytical expressions. Advances in Neural Information Processing Systems, 33:1 18:11416-11422 1142, 2020.
Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. Advances in Neural Information Processing Systems, 34:2519222555555 2021.
12

---

Under review as a conference paper at ICLR 2023
Maxwell I. Nye, Armando compositional rules via neural program synthesis, 2020.
Joshua B. Tenenbaum, and Brenden M. Lake.
Learning
Santiago Ontañónn, Joshua Ainslie, Vaclav Cvicek, and Zachary Fisher. Making transformers solve compositional tasks. CoRR, Abs/2108.04375, 2021. URL https://arxxvoorr//aaat. 04378.
Long Ouyang, Jeff Wutrain, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https: //arxiv. org/ abs/ 2203.22222222 02155.
Panupong Pasupat, Yuan Zhang, and Kelvin Guu. Controllable semantic parsing via retrieval augmentation. arXiv preprint arXiv: 10.08458, 2021.
Linlu Qiu, Peter Shaw, Panupong Pasupat, Pawel Krzysztof Nowak, Tal Linzen, Fei Sha, and Kristina Toutanova. Improving compositional generalization with latent structure and data augmentation. In Marine Carpuat, Marie-Catherin
Mameffe, and Iván Vladimir Meza Ruíz (, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022. pp. 4341- 4362. Association for Computational Linguistics, 2022. doi: 00.18653/v122022.maaad) main.33... URL https:: http://ooo.oog..////// 18653/1611811 /vv /2022 nww.caaaca-a.ammm aaaaacclaalll. wwa.aa...... 323.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv:1910. 19683. 2019.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Perey Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.
Chandra Reddy and Prasad Tadepalli. Learning first-order acyclic hom programs from entailment. In International Conference on Inductive Logic Programming, pp. 23-37. Springer, 1998.
Adam Roberts. Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez, Alex Salcianu, Mare van 7ee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Teorphorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin- Shetin-Sheeard,, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with 55x and seqio. arXiv preprint 2203 2022. URL ht https: /tax/viivv//.////////aa////aaavv org/ /abs/ /etp//223..22 11003.11189
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of Plausible Alter- natives: An Evaluation of Commonsense Causal Reasoning. In AAAI Spring Symposium Logical Formalizations of Commonsense Reasoning, Stanford University, March 2011. URL attp::/icet usc.. httpa///w..c..oaoo.ooooaooaaaaaa
.dd.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M. Lake. A benchmark for systematic generalization grounded language understanding. 2020.
Daniel Lehmann Sarit Kraus and Menachem Magidor. Nonmonotonic reasoning, preferential models and cumulative logics. Journal of Artificial Intelligence, Vol. 44 Nos. 1-2:167-207, 1990. URL
tittss://arivv.
http://wwwwwx.p...a../ org/ als////b//co/ cs 0202021.
Semantic Machines, Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Domer,, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk. Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovem, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin,
13

---

Under review as a conference paper at ICLR 2023
Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556-571,
September 2020. URL hwtps:tp///....... /doii .rgg 10. 0. 11 102/tacl_e_003333
Peter Shaw, Ming- Wei Chang, Panupong Pasupat, and Kristina Toutanova. Compositional generaliza- tion and natural language variation: Can a semantic parsing approach handle both? arXiv preprint arXiv:2010. 2222::2010./2725, 2020.
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William Hamilton. Clutrr: A diagnostic benchmark for inductive reasoning from text. Empirical Methods of Natural Language Processing (EMNLP), 2019.
Koustuv Sinha, Shagun Sodhani, Joelle Pineau, and William L. Hamilton. Evaluating logical generalization in graph neural networks, 2020.
Madhumita Sushil, Simon uster, and Walter Daclemans, Rule induction for global explanation of trained models, 2018.
Alon Talmor. Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. Leap- Teaching pre-trained models to systematically reason over implicit knowledge, 2020.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint XX.1:2011.04000 2020a.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys (CSUR), 2020b.
Sehastian Thrun and Lorien Pratt. Learning to Learn, chapter I Learning to Leam: Introduction and Overview, pp. 3-17. Springer, Boston, MA, 1998. ISBN 978-1-4613-7527-22 URL https://doi. org/10. 1007 911//77811111 110--11411555 5529-2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention all you need. In NIPS, 2017. URL hitps: /ttpxv..rx..g///////.///./.. PdE/1706. 037622 pcl..
Pat Verga, Haitian Sun, Livio Baldini Soares, and William W Cohen. Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge. arXiv preprint arXiv:2007.00849, 2020.
Sinong Wang, Belinda Li,, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint trXiv:2006.047688 2020.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolahahhi, Yeganeh Kordi, Amirreaa Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Gun, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V Le. Finetuned language models are zero-shot leamers. In International Conference on Learning Representations, 2022. URL https: / /wwpreveeeeeee.ee//e/ee/e net. forum? id= gEZrGCozddR.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks, 2015.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpoiqa: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium, October 31 November 4, 2018, pp. 2369-2366 2380. Association for Computational Linguistics, 2018. doi: 10. 18.13833/v/ d18-1259. URL https:ttp/..............t /dw.. wacooo.orr//ooo org/1 / 0 8653 vl /d1 1259.
14

---

Under review as a conference paper at ICLR 2023
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: large-scale suman-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In EMNLP, 2018. URL http:////axivv.oo////////////// wwtpp///gg/pd/////eo
08887.po pdd..
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang. et al. Big bird: Transformes for longer sequences. Advances in Neural Information Processing Systems, 33, 2020.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2019.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint ar0iv:2205. 10625, 2022.
15

---

Under review as a conference paper at ICLR 2023
APPENDIX
A CLT DESIGN CHOICES
Here we discuss the rationale behind the design choices made in our definition of a Conceptual Leamingg Task (CLT).
Splitting the input into set-like context and a request. The main goal of CLTss is to test whether a learner is capable of learning from explicitly provided knowledge consisting of rules and examples. Assuming supervised learning setup basis, the applicable knowledge must somehow be provided in the input of each example. It is therefore quite natural to split the input into two parts: context and request. Since the examples and rules that form the background knowledge do not usually have specific order (in the movie recommendation example for instance, it does not matter whether we are first told that Top Gun is rated PG-13 or that Jerry Maguire is rated R), representing the context as an unordered set is a natural choice.
Representing rule assertions as examples. Another important property of CLTs is the ability to ask the learner explicitly whether a certain rule holds (in given context). One straightforward way to achieve this is to include examples where the request asks for the validity of certain rule and the output provides the corresponding truth value. These kind of examples also provide us with a natural way to assert (or refute) rules in the context, which allows represent the context simply set of examples (rather than a heterogencous set containing both examples and a dedicated representation of rule assertions and refutations).
Distinguishing monotonic and defeasible replies. Once we include the context as part of the input, we distinguish between two different methods of how the learner may determine the reply for a given request: deduction and induction. For deduction, the learner infers the reply for a given request from the context using deductive reasoning alone. As an illustration, consider the context that Top Gun is PG-13 movie and then ask for the rating of Top Gun in the request (see Appendix D for a formalization of the logic axioms underlying deductive reasoning).
example where assert in
For induction, the information provided by the context is not sufficient to unambiguously determine the reply for a given request. As an illustration, suppose that we ask for the rating of Jerry Maguire in an example whose context asserts that both Mission Impossible and Jerry Maguire are movies starring Tom Cruise and that Mission Impossible is rated PG-13. This information not sufficient for to deduce the answer. Instead, the learner needs to rely on inductive bias to determine whether it should speculatively generalize the PG-13 rating from Mission Impossible to Jerry Maguire or whether should play it safe and say that it doesn't 't know.
Deductive reasoning is always monotonic w.r.t. the context (i.e., new knowledge cannot lead to a different reply), so we use the qualifier M to indicate deductive replies. Inductive reasoning is always defeasible w.t.t the context (i.e., new knowledge may lead to a different reply), so we use the qualifier to indicate inductive replies.
B MOTIVATING EXAMPLE AS CLT
Here we show what the motivating example from the introduction (Figure 1) could look like when formu- lated in the syntax of CLT.. Here we take Q to be the space of natural language statements and questions.
Context containing assertions of relevant knowledge:
C1= ("R rated movies not appropriate for kids who are less than 17 years old.". ("The age of a person is the time that has passed since the person was born."' (TThe current date is June 3, 2021.". 1,))
("Mission Impossible 1 - are PG-13 rated action movies starring Tom Cruise.",
("Jerry Maguire is R rated comedy starring Tom Cruise.",1) 11)
( "Anna was born in January of 2008 and Tom Cruise is her favorite actor"",
Anna saw Mission Impossible 1-5 5 and liked all of them.".1) ,1)
16

---

Under review as a conference paper at ICLR 2023
Top-level example set containing a mixture of non-rule examples (natural language QA for movie rec- ommendations) and rule examples (which probe the model's understanding of intermediate steps in the recommendation process):
E= [(C,, "What movie could Anna watch?", "Mission Impossible ('.
"Anna is 14 years old.", 11.M)
(CI, "Anna is at least 17 years old." M..)
"Jery Maguire is appropriate for (4-year-olds.",On ...M)
(C1, "Anna will like Mission Impossible 6.", 6.",
(C1, "Tom Cruise is Anna's best friend.",,.
}
The above is relatively straightforward translation of the motivating example into CLT syntax, while using the format of rule examples all of the context examples and for all of the intermediate questions. If we assume that the request space Q and reply space R include natural language questions and answers for querying about background knowledge, as well as for the end goal of providing movie recommendations, the above example could be altemnatvell expanded to represent some of the background knowledge in non-rule format (e.g.. ("Whht is the rating of Jenry Maguire?",""R "R")) and/or to represent some of the intermediate questions non-rule top-level examples (e.g., (C1,"How, old is Anna?", "14 years" MM).
C GENERALIZATION CLTs TO SUPPORT NESTED CONTEXTS
As discussed in Section since the context of each top-level example in CLT is itself represented as set of "examples", we have similar but slightly different notions of "example" at two different levels a CLT:
Top-level examples, which we represent as quadruples of ,equtex request, reply quply-qualifier).
Context examples, which in Section we require be unconditional and monotonic and which we thus represent as simply repply)t,reply)) pairs.
While for the simple cSCAN task, it was sufficient to provide only unconditional monotonic examples in the context, in the general case, one could imagine extensions of the notion a CLT to allow inclusion of conditional and/or defeasible examples in the context well. In this more general view of a CLT, we can drop the distinction between top-level examples and context examples, and instead adopt a recursive structure in which each example is quadruple of context is a set of examples.
repl..uaiiil. while each
One motivation for this more general view of CLT's is if we as representing one observation of the behavior of a personal assistant with very strong decision-making capabilities (which we might call the "teacher"), whose behavior all times is conditioned on the knowledge available to it, and whose knowledge is stored in a large and growing set-like knowledge base. In this view, the context of the top-level example can be thought of a snapshot of the relevant contents of the assistant's 's knowledge base at the time that the assistant received the given request and output the given reply and qualifier.
think of each top-level CLT example
Now let us further suppose that we have another personal assistant (which we might call the "student"), which itself has some kind of growing set-like knowledge base, whose contents may or may not agree with the contents of the teacher's knowledge base. One interesting question is how the student can go about selectively "learning" from the teacher, so as to emulate its decision-making capabilities, without necessarily accepting wholesale the full contents of its knowledge base, which may also include information that is transient, iituation-specifie the student to adopt. One natural approach could be to simply select the top-level examples of interest (i.c., to select the specific instances of the teacher's 's behavior that the student wishes to emulate) and assert those examples in the student's knowledge base. This would be the equivalent of storing the knowledge
or debatable, and which may thus not be relevant or appropriate for
17

---

Under review as a conference paper at ICLR 2023
that "if such and such things (i.e., the contents of the example's context) were true, then when faced with the given request, this would be the appropriate reply and the appropriate qualifier".
Under this approach, the student's set-like knowledge base would (including context and qualifier), Continuing with the view of a "context" as being a snapshot of some or all of an assistant's knowledge base, contexts could now contain examples that themselves contain non-empty contexts. The maximum depth to which we allow such recursive structures to continue could be considered an arbitrary choice in the design of conceptual learning system or task.
come to contain full CLT examples
C.I GENERALIZED CLT
Below is formalization of this more general definition of a CLT, along with some shorthand notations, by which the simpler CLT definition from Section 2 can be seen as just a special case of the general definition.
The task T= (e......ee) a finite set examples e EE, where E denotes the set of possible examples, with I being the set of possible imputs and O the possible outputs.
Each example - (z) is pair consisting of an input i, EI and an output 0g E OO
Each input i = (C.q) is a pair consisting of context C C 2 and a request E Q. The set of possible requests can be partitioned into rule requests 2R C Q (i.e., requests that ask whether certain rule holds) and non-rule requests QN C
Each output 02 o= (r.u) a pair consisting of the reply rE R and a qualifier u U. The set of possible replies R must contain dedicated elements representing true (1), false (0). and unknown (?), which are the only valid replies for rule requests E Qa. The set of qualifiers U= (M.D) indicates whether the reply follows monotonically from the context (M) or whether it is defeasible (D).
For conciseness, we use the flat notation (C.q.r.u)) to mean the nested pairs (r,)), we use (C.q.r) shorthand for the monotonic example (C.q.r.M), and we use (q.r) as shorthand for the unconditional monotonic example (Ø. rr.). Hence, the example (C.q.r) means that given the context C or any superset of C, the request q translated the reply r, while (g.r) means that under all circumstances. the request 4 is translated to the reply
Note that even though the context is strictly set of examples, may still contain any rule q QR by means of including the example (Ø.q. 1.,), which asserts the rule to be true unconditionally. Similarly, we can express that the rule q does not hold in a context by including the example (....
We refer to an example unconditional if it has an empty context.
Note also that since examples within context are of the same form as top-level examples, they may in principle themselves contain contexts up to arbitrary levels of nesting. We can, however, for any given CLT, choose a maximum level to which we allow such nesting to occur. cSCAN, for example, contexts contain only unconditional examples, so that there is no nesting of contexts.
C.2 ADDITIONAL SHORTHAND FOR RULE ASSERTIONS IN CONTEXT
To express the assertion of a rule o Ox in contexts more concisely, we use to mean its unconditional monotonic assertion L.M. For instance, ([g).g o st) stands for {((2.q..MM)) r)).
In the example from Figure 1, if we take Q to be the space of natural language statements and questions, then with the above shorthand the following would be equivalent:
(((2,
((Ann" "Anna was born in January of 2008." ..MM) "Who is Anna's best friend?". .?.D)
((*Anna was born in January of 2008."), "Who is Anna's best friend?". ,?.D)
D FORMALIZATION OF CONSISTENCY REQUIREMENTS AND METRICS
In this appendix, we formalize the consistency requirements and consistency metrics of CLTs by mapping example sets into classical propositional logic. We start by summarizing the definitions and axioms of propositional logic, and we specify functions to capture the meaning of rules and the inductive bias. This
18

---

Under review as a conference paper at ICLR 2023
allows to map example sets to logical formulas, which in turn allows to formalize the consisteney requirements as well as the metrics for measuring consistency of a learner.
For the purposes of this formalization, we assume the general form of a CLT described in Appendix C, for which the simpler CLT form used for cSCAN follows as special case.
Note that while we provide this formalization for reference purposes and facilitate future investigation into the formal properties of CLTs, it possible in practice to use CLTs and the related consistency metric without detailed consideration of this formalization.
D.I TERMINOLOGY
not otherwise indicated, we assume that 22 are arbitrary contexts, q.9/ E Q are arbitrary requests, r.r' E RR are arbitrary replies, u.n' E U are arbitrary qualifiers, E,E', F E & are arbitrary example sets, and e, of are arbitrary examples.
Furthermore, we assume that M(e) and D(e) denote the monotonic and defeasible variants of the example e, respectively. This means that for any HEU, M((Cq.r.u)) (C.q.r.M) and ((C.gr..))= (C.
D.2 CLASSICAL PROPOSITIONAL LOGIC
For reference in later proof', we summarize the basic definitions and axioms of Lukasiewicz classical propositional logic (Klement, 2004), which we adopt as-is, with only a change of symbols for the logical connectives to avoid ambiguity with the A, V, and - symbols that we use elsewhere in this document in first order logic statements.
We assume a set of propositional variables V which represent atomic formulas. General logical formulas recursively constructed from these atomic formulas using the logical connectives (implication). (negation), A (and), V (or), and = (equivalence). our formalization, we assume the primitive connectives - and the constant I (falsum), and we define the other connectives as follows (assuming that x,yz L).
-
(1)
xVy=arty
(2)
AAY:= (x-Ly)
(3)
xxy:=(x->)) (x-x)
(4)
T:=-1
(5)
To formulate the propositional logic axioms and inference rules, we use the notation x to express that we can infer from x and we use x to express that x is tautology.
Fx (y-+x)
(6)
+ (x~ (y) - ((x~))) -x -x--)))
((rray) - (x+x)
x(x-
The axioms (6), (7), (8) form the Lukasiewicz system, while the inference rule (9) is modus ponens.
D.3 SEMANTICS OF RULES AND INDUCTIVE BIAS
To map example sets to logical formulas, need to be able to refer the semantics of rules and the inductive bias.
For the semantics of rules, assume grounding function G: Qx + 2 which maps each rule an equivalent set of examples. This means that in a consistent task, a rule g E Qp holds (i.., (Ø.q.1) 1) is a valid example) if and only if each example in G(g) is valid. It also means that if the context of an example E
19

---

Under review as a conference paper at ICLR 2023
contains the assertion (B.q.) of rule qE QR. obtain equivalent example e' by replacing the assertion with the examples ((q). We provide the grounding function cSCAN in Appendix E.3.
Since a CLT requires learner to induce well deduce rules, every CLT inherently assumes some form of inductive bias, which determines the criteria based upon which learnrr is expected to induce a rule to be "true" vs. considering it to be "unknown". While the inductive bias of the task is in principle arbitrary (being dependant on the choice of the task designer or the needs of the real-world use case), the learner will need be able emulate this bias in order to perform well the task. We formalize this domain-specifie inductive bias through bias function B: 2 + +200 which maps each set of examples the set of examples that are expected to be induced. We use ED S & to denote the subsets of all monotonic and defeasible examples, respectively. We provide the bias function of cSCAN in Appendix E,4.
While the grounding and bias functions are needed to precisely formalize the consistency requirements and metrics, they are not requirement for a conceptual learning task. This is important because providing complete grounding and bias functionn may not be feasible or practical for more realistic CLTs. Instead, we may only provide partial functions, which means that the formalization in this section will only approximate the true consistency requirements and consistency metries.
D.4 MAPPING EXAMPLE SETS TO LOGICAL FORMULAS
We treat each example propositional variable (i.e, (C,q.r,u) E V). This allows us define the embedding function M: 2 + L as follows.
M(B)==T M[ =
(10)
(11)
M
(12)
The empty set maps true (10), of size maps its only element (11), and union maps logical conjunction (12). In addition, our embedding adheres to the following axioms.
F + MC
(13)
(14)
M(C)=M(C))
(15)
r/:: (C.r., - (C-+)))
(16)
MM -M(E)-
(17)
(18)
Axiom (13) says that the context of monotonic example becomes the antecedent a logical implication, and axiom (14) specifies that monotonic examples implies the corresponding defeasible example. Axiom (15) says that defeasible examples that differ only in equivalent contexts are equivalent. Axiom (16) specifies that defeasible examples are functional or the context must be contradictory, which means that each request with a con-contradictory context must have a unique reply. Finally, axiom (17) specifies that the assertion of rule is equivalent to its grounding, and axiom (18) says that each set of examples implies the set of examples that can be induced using the inductive bias.
Together with the axioms of propositional logic, obtain the following theorems.
HCCCMM .r.
(19)
FM((CqIM))= of al): ( (C4 r+:: HCarra))))) A (C.g.' ()) (C-11)
(20)
(21)
Theorem (19) is obtained from axioms (6), (13) and (12), and says that monotonie examples behave monotonically w.t.t their context. This allows us to rewrite the grounding axiom (17) to obtain theorem (20).
The last theorem (21) says that functionality applies independently of the qualifier.
20

---

Under review as a conference paper at ICLR 2023
D.5 CONSISTENCY METRIC
As discussed in Section 4, the consistency metric (EE) differs from the standard accuracy metric in that it does not judge the correctness of a learner's predictions on individual examples, but rather measures the de- gree to which a complete set of examples E. is consistent w.r.t. to the axioms of classical propositional logic.
In essence, we define C(E) to be the percentage of subsets of E that contain a logical implication, in comparison to the number of subsets of E that contain an implication or a contradiction.
Note that the definition of the consistency metric provided in equation (26) could essentially be applied as-is to any arbitrary CLT, provided there is some way to identify the "implications" (impl(E) (E) and "contradictions" (cont(E)) among the learner's predictions. In practice, task designers may be free to apply any reasonable heuristic to identify such implications and contradictions. The formal definition of the consistency metric below can be considered an ideal, which we seek to emulate closely in the consistency metric implementation provided for cSCAN, as described in Appendix G.
E-f= K-MMIBA--M ÇE: (MM
(22)
(23)
(EEDAB-M())
impl(E)= (FCE: (TEEF: FV0)
(24)
(ont(E):= = (FEE: ANN U))
(25)
100-
limpl()) impil PJ] [F)
(mml((E)((con(E)))
C(E):
(26)
NaN,
otherwise
Minimal implication E f means that the non-contradictory example set E implies the example f and that there is no strict subset of E that has the same property. Furthermore, require matching qualifiers, which means that E must not imply the monotonic variant of f if the latter is marked as defeasible (22).
The minimal contradiction E can be met by either of the following two conditions (23). First, E+*x holds if the non-contradictory example set E implies the negation of f and there is no striet subset of E that has the same property. Secondly, E E f holds if there is qualifier mismatch, i.e., if f is marked defeasible but E minimally implies the monotonic variant of Note that we do not consider it to be a qualifier mismatch if is marked as monotonic and E implies only the defeasible variant because there may be other evidence outside of E that may justify the monotonic qualifier.
This allows us to define impl(E) be the set of subsets F E that contain minimal implication (24) and conr(E) EE to be the set of subsets F ECE that contain a minimal contradiction (25). Finally, the consistency metric C(E) is the percentage of subsets of E that contain implications, in comparison with the number of subsets of E that contain implications or contradictions (26). If the set E. does not contain any implications or contradictions, then the consistency metric is not defined.
Illustration. As an illustration, consider the following example set E. which consists of 4 rule assertions and examples. (We assume the syntax and semantics of cSCAN, which is formally specified in Appendix E.)
E:= ( [u twice] = [[]][[]] 1
(a)
(C1.[xx after y] [>][
(b)
(C, [[alk]] = -WALK,1) 1),
(c)
(C, [[mmp] 1JUMP,1),
(d)
(C,, walk, WALK),
(e)
(C), eat FAAT EAT),,
(f)
(C, walk twice, WALK WALK WALK WALK),
(g)
(Cr jump tWiC JUMP,D),
(h)
(C,, walk after walk, WALK WALK),
(
21

---

Under review as a conference paper at ICLR 2023
jump after walk, WALK, JUMP D))
walk after jump, WALK JUMP), walk twice after jump. JUMP WALK WALK)}
(k)
In this example there are total of minimal implication sets (with the implied example indicated in green above):
fc,e}
{b,cd}
[beed}
fab.e.d...
(a,
And there are total of 8 minimal contradiction sets (with the contradictory example indicated in red above), of which the following are because an inconsistent reply:
tans}
{a,es}
(b,ckdk
[b,edd.kk
(b..g..}
while the following are because of an inconsistent qualifier:
faddh]
[b,c,d.}}
{b.ed.j)
Based this, the consistency of example set E would be C(E) 150]=100--//00 5/ 38.5.
Note that example both implied and contradicted. Also, since examples equivalent according to cSCAN's grounding function (Appendix E.3), each minimal implication or contradiction set that involves example c can be written alternatielly using example e, causing such sets to appear in pairs in the lists above.
are semantically
D.6 CLT: DEFINITION AND CONSISTENCY REQUIREMENTS
To obtain a precise measure for the consistency of the predictions produced by given learner, it is important that the task T itself be consistent. Specifically, this means that each example E T must be consistent on its own, and the example set T as whole must be consistent. For the purposes of this paper, we further require that contexts are non-contradictory.
We capture these requirements using a predicate consisteniarr(T), which is recursively defined as follows.
Consisentcr(((Carr::)) (rcp:s/aacrii)))soooossseetcaaaa V=(Cara)
(27)
(u= 1u=+ +M(l(grri))
(28)
comsistenlarrr(())=++++(t consistentar((ee))) A (C(E)=100 C(E)=
(29)
We first define the consistency of an individual example (C.q.r.)) EE, which requires that its context is consistent (and therefore non co-ootrradictor)), that the example itself is not a contradiction, and that it is qualified as monotonie if and only if it maps tautology (28). Then, define that example set E EC& & is consistent if and only each individual example e E E consistent and the example set consistent whole (29) (i.e., C(E) must be 100 undefined).
22

---

Under review as a conference paper at ICLR 2023
walk]
WALK
[11 andee]
[xn]] [xa]
run]
RUN
Xi afterva]
[Na] [x1]
jump]
=J JUMP
turn opposite left
LTTURN LTURN
look
LOOK
furn opposite right
KTURN RTURN
turn left
=LTURN
turn around left
= LTURN LTURN LTURN LTURN
tur right]]
-RTURN
turn around right] KTURNN RTURN RTURN RTURN
x left
[
xi opposite left] [tum opposite [ef]][]] [[x]]
x right]
= -TTTRN[[]]]]
xi opposite right] = [turn opposite right] [x]]
In twice]
= [x1 [n
x around left [ LTURN [an] [ LTURN[[:]]
X1 thrice]
[xs [x]]]]
X around right] = [TURN[A]] [TURN[M] [ [TURN[E]
Figure 4: SCAN interpretation rules as provided by Lake & Baroni (2017). Double brackets denote the interpretation function translating SCAN's linguistic commands into sequences of actions. Symbols x1 and x2 denote variables.
Note that while the above formulation of CLT consistency requirements is sufficient for cSCAN, it could be desirable to adjust this for more complex tasks, which we leave for future work. In particular, while this definition of consistency provides only basic control over the behavior of defeasible examples, could imagine defining stricter consistency requirements for defeasible examples, c.g., by requiring them to adhere to the KLM properties (Sarit Kraus & Magidor, 1990; Casini et al., 2021). Also, while in this paper, for simplicity, we require contexts to be non-contradictory. this requirement is not strictly necessary for the task to be consistent or for the consistency metrics to be meaningful, as long the top-level examples in the dataset do not contain a contradiction or contradict each other. Taken a step farther, for dealing with real-world datasets which may contain noise, it may be desirable relax the requirement that the task be strictly consistent. In such an approach, the "requirements" of CLT can be considered more as an aspiration rather than strict requirements. The consistency metric could in that case still be calculated, but would be less reliable an indicator of leamer consistency compared to the case where the task is consistent.
E SPECIFICATION OF CSCAN
This appendix contains a specification of the cSCAN task. It consists the phrase-structure gramimar to generate valid requests and replies, the compositional grounding function (which defines the meaning of the explicit rules), and the bias function (which defines the inductive bias). Together with the formalization of simple CLTs provided in Appendix D. this specifies the complete behavior of cSCAN.
Note that in this section, as in Appendix D, we assume the more general CLT formalism described in Appendix C, in which contexts can contain examples of the same form the top-level examples. While cSCAN specification could be expressed equivalently in terms of the simplified CLT formalism of Section 2, the more general formalism allows to express some aspects of the specification more concisely, as we can thus describe the semantics of both top-level examples and context examples using the same grounding function.
For readability, we make use the shorthand notation described in Section C.2 to allow expressing some of the more verbose examples more concisely.
Note also that while we provide the complete formal specification of cSCAN here as a reference, it is not necessary in general provide specification at this level of detail when defining future CL.T'.
E.1 RULE SPACES
Most of the specification of cSCAN is identical between eSCAN-B and cSCAN-X. In this section, summarize the points that differ between the two, together with some notes on the original SCAN task for comparison.
Original SCAN
In the original SCAN task, natural language commands generated by fixed phrase-structure grammar as described in Lake & Baroni (2017), which is equivalent to the phrase-structure grammar shown for eSCAN-B in the top left in Figure 5. (Note that for readability, renamed the non-terminals in this
23

---

Under review as a conference paper at ICLR 2023
and T after T
opposite W around X
left
-
T U thrice
V
run
X - right
Y x2
S
T U
U
W jump
Y x3
W look
Y x4
W tum
W X-
Figure 5: The green the top left is the phrase-structure grammar generate the cSCAN-B commands. The red area on the lower right is the extension to generate the left-hand side of cSCAN-B rules.
presentation proceed alphabetically, beginning with "S" the traditional start symbol.) The mapping from command to action sequence follows the fixed set of interpretation rules shown in Figure 4.
As can be seen in the figure, action sequences in the original SCAN task are constructed from set of 6 possible actions.
A:=
eSCAN-B
In cSCAN-B, the set of non-rule requests 2y consists of all natural language commands that are generated by the phrase-structure grammar shown the top left in Figure 5, which is equivalent to the original SCAN phrase structure grammar, as reformulated in Nye et al. (2020). The mapping from command action sequence varies from context to context.
eSCAN-B constructs action sequences from the same 6 actions used in the original SCAN task, plus several additional ones provided for diversity. (In nod earlier research the SCAN task, we follow the lead of Nye et al. (2020) in using as these additional actions the ones that appear in the "MiniSCAN" task of Lake et al. (2019).)
A:= WWA.K,RUN.UUMPPLLO....... LOOK, RTURRR
RED, YELLOW GREEN, PURPLE, B.NCK WHITE)
eSCAN-X
In eSCAN-X, the set of non-rule requests QN consists of all natural language commands that are generated by the phrase-structure grammar shown on the top left in Figure The mapping from command to action sequence varies from context to context, similarly to eSCAN-B.
eSCAN-X constructs action sequences from a set of 13 possible actions.
A: A==WAAKKUUUUDUUPLOOOLLUUUUUAAAA
E.2 REQUESTS AND REPLIES
For simplicity, this section focuses on a formal description of cSCAN-B. The specification cSCAN-X would follow the same form, with the exception of the differences described in Appendix E.1 above.
Non-rule requests. eSCAN, the set of non-rule requests QN consists of all natural language commands that are generated by the phrasc-structure grammar shown the top left in Figure 5. Note that for convenience of generation, the grammar adopted here is based on the alternative formulation of the SCAN grammar from Nye et al. (2020). This generates slightly larger set of commands than the original SCAN grammar from Lake & Baroni (2017). as it includes commands such as "turn" and "turn and turn".
Replies. Since cSCAN is CLT, the set of replies R includes the dedicated replies 1 (true), 0 (false), and (unknown). In addition, R contains all sequences consisting of actions AA (described in Appendix E.2 above) separated by space with maximum sequence length N:
24

---

Under review as a conference paper at ICLR 2023
and T
twice
opposite X
walk
- left
Y Y-xx
T after T
U thrice
V around
W cantiously
W - ruIl
- right
Y 3--x2
T before T
T U 2x
U V
W drunkenly
W jump
up
Y x3x3
T Tollowing
3x
X
zigzag
W - look
X down
Y Y-x
between
T- 4xx
180
W
T framing
T U 5x
U 270 X
W turn
T-U
360 X
W drive
W ride
W
W
peck
Figure 6: The green the top left is the phrase-structur grammar generate the cSCAN-X commands. The red area on the lower right is the extension to generate the left-hand side of cSCAN-X rules. The items in blue boldface are those which do not appear in eSCAN-B.
: (al 02 ag: 1skkskkaaae R: R:= (1.0,JUA
(30)
(31)
Rule requests. All cSCAN rule requests of the form q-r, where g element of the left-hand side (LHS) expressions 2 and r is element of the right-hand side (RHS) expressions R"
The set of LHS expressions Qý generated by the extended phrase-structure grammar shown Fig- ure 5, with the resulting phrases wrapped in brackets []. Note that this is almost the same grammar that is used to generate the cSCAN commands QN with the only difference being that allows gener- ating expressions with variables from the set X' x3.4) Examples of LHS expressions are [xx twice]] and [walk opposite left].
The set of RHS expressions RR consists of sequences of maximum length N whose elements separated by space and consist of actions and LHS expressions 25. Examples RHS expressions are "WALK K [x twice] LTURN" and "WALK JUMP [run lefi] [look thrice]". Note that RHS expressions are superset of LHS expressions, i.c., 21 C R
R² == (a1 d2 ax:
(32)
We assume a function var: R" + 2 which returns the set of variables used by given RHS expres- sion. This allows define the of rule requests QR LHS and RHS expression with matching variables:
2Ri=tg=r-
(33)
E.3 GROUNDING OF RULES
As discussed in Appendix E.3, we specify the semantics of cSCAN rules via a grounding function G: - which maps each rule to an equivalent set of examples. For rules where the LHS consists of command without any variables, we can define the grounding as an example that provides the interpretation of this command. For example, the nule [walk] WALK is grounded as follows:
G[[walk] = WALK) : ((0) walk, WALK)
Note that because we determine consistency based on propositional logic equivalence (see Appendix D). could specify equivalent groundings that include additional examples that are logically implied by {(2,, walk, WALK) For instance, we could add variants with non-empty contexts (e.g., ([[run] RUN], walk, WALK)), which follow from monotonicity.
25

---

Under review as a conference paper at ICLR 2023
For rules containing variables, the grounding may consist of hundreds thousands of examples, even if we do not include any examples that are logically implied. This is because the variable can be replaced with any command that leads to valid LHS. For example, the rule [x] twice] [x; [xx] can be grounded as follows:
G([x [x]c]]][[[]][[]]]]
[[[[rnn]] = RUNR.NNN... run RUN RUN),
[[[run] JUMMP) -un JUiC JUMP),
[[ left] = LLTURN RUN), left twice,LTUUN RUN LTURN RUN),
The first example in this grounding can be read as: [[un]]] is translated to "RUN" then "run twice" is translated to "RUN RUN".
Compositional grounding function of cSCAN. Since the semantics of cSCAN rules is compositional (which is a requirement for all CLTs), we are able to specify the grounding function in a complete yet concise fashion using the following helper constructs. For simplicity, we focus here on describing the compositional grounding function for cSCAN-B. The grounding function for cSCAN-X would follow the same general form.
For any RHS sequence rER", , we use lrs(() to denote the set of elements of that are LHS expres- sions Oÿ- For example, (hLLK JUMP [run left] [look thrice]) {[run left]]. [look thrice]]. ).
For any subset of LHS expressions S 2i we use c2a(() to denote the set of all possible functions Q A" that map each expression in Q to an action sequence in AA". (Note that this also applies to the empty set, i.e., there is exactly one function c2a(2)).
We define subsets of the commands QN generated by the phrase grammar in Figure 5. For each E {T, UT V. WX} the set Q2 C QN denotes the commands that are generated when starting from the symbol (rather than S).
For each E {T, U, V, W.X], we assume a function var": 22 + 2², which map each LHS expression q E 2ÿ subset its variables var(g). For any expression q 22, the following holds:
var (q) consists of the variables that are generated using the rule path T - U,U V. W. W
var" (q) consists of the variables that are generated using the rule path U V, - W. W
Y.
var"(g) consists of the variables that are generated using the rule path V - W. W + Y. var"(g) (q) consists of the variables that are generated using the rule W - Y. varx(q) consists the variables that are generated using the rule -
For all LHS expressions q E Ø we use v20(g) to denote the set all possible func- tions f varlg) + Qv that map each variable in var(g) to command in QN such that VZE (T.U,V,W. V.V, f(vrr(() E Φy- Note that this restriction of the mapping ensures that variable substitution does not break the implicit precedence rules that apply to the interpretation of commands (see the discussion below for more detail).
For any RHS expression E R* and any partial function : U' + R*, we use subst(r..) denote the RHS expression that we obtain by lexically substituting all occurrences of r' E dom(f) with.(r').
These constructs allow define the grounding of all rules (q - E Qa as follows:
G(g=r):= UFEcza(hhat) ( (1y edommn ((0a/=-((())))))==sssst)) (revzo(e)
varlg)=
(34)
otherwise
26

---

Under review as a conference paper at ICLR 2023
Discussion and illustration. The first case of definition 34 applies when the rule q= does not contain any variables. It says that whenever a LHS expression q' occurs in the RHS expression r, we can substitute of with arbitrary action sequence f(q') long make sure that the context contains a rule asserting that o' is interpreted as f(q')-
As an illustration, consider the grounding of the rule [run twice and jump] = [rnn]]runn] LTURN that is shown below. The set of LHS expressions on the right-hand side of this rule is (ru,, jump]. The first example of the grounding can be read as: if [run] maps to "RUN" and [jump] maps to "JUMP" then the RHS of the rule becomes "RUN RUN JUMP LTURN".
([run twice and jump] = [[r]]]]]]]]]] [un]] LTURN):
([t [[tun]-RUN,], (,) - -JMMP, J,
[ruun twice jumpp RUN RUN JUMP LTURN,
[[c_[[un]]=1000]11) (aapaa_jjjpppp=_a (AL,,,))
[run twice and jump] = LOOK LOOK WALK LTURN,I), [[ = ( -aammp]]RUuu RUN RUN,))). [run twice and jump] = RUN RUN RUN RUN LTURN,1),
The second case of definition 34 applies when the rule q= contains a set of variables. It says that we can substitute any variable x with a command (x) that corresponds to the same non-terminal in the parse tree.
As an illustration, consider the grounding of the rule [x twice] = [>][[]] LTURN shown below. In the first example of the grounding, we replace the variable with the command "run", in the second example we replace it with "jump around left", and in the last example we replace it with "look thrice""
([xx, twice] [xx] LTURN)=
(@0,[rum twice] = [[run]] run] LTURN,I),
(es [jump around left twice] = [jump around left] around left] LTURN,1),
@0 llook thrice twice] = [look thrice] [look thrice] LTURN,I),
Note that because E Evar ([x, twice]) and x1 & var ([x; twice]), the variable x1 cannot be substituted with commands such as "walk and jump" that are in 2% but not in O% This is important because the grounding would otherwise contain examples that violate the higher precedence of "twice" " when compared "and", which is not what we intended. Le., our formalization makes sure that:
(2, [walk and jump twice] = [walk and jump][[wll and jump] LTURN.I) o
([[x twice]= = [xtc]=[[]][<>] [x]]]]]]]] LTURN)
E.4 INDUCTIVE BIAS
As discussed in Appendix E.4, the inductive bias of a CLT is the criteria based upon which a learner is expected to induce a rule to be "true" as opposed to considering it to be "unknown". For cSCAN, we adopt a simple set of criteria, based loosely on our own intuition, in which we consider there to be sufficient evidence to support induction of rule if there are examples the context that could be explained by the given rule (in combination with other rules that are explicitly or implicitly provided in the context) and which in total illustrate at least 4 different substitutions of each of the rule's variables. The number is arbitrary, but based on our intuition that we would be comfortable in generalizing rules from relatively small number of examples, but that 2 examples is not quite enough to justify inducing a general pattern. We the inductive bias criteria internally for determining
chose the threshold of 4 rather than 3 because the minimum number of illustrating examples to include in the context for each rule that we intend to be induced to be "true", and wanted avoid penalizing learner that is slightly cautious in its inductions.
27

---

Under review as a conference paper at ICLR 2023
At the same time, took care to avoid illustrating any rules' variable substitutions exactly three times, so as to avoid penalizing a learner that is just slightly on the aggressive side in its inductions. In this way, while we describe the task's inductive bias formally requiring a minimum of 4 illustrated variable substitutions, a learner could succeed on the cSCAN task by adopting a minimum threshold of either 3 or 4 substitutions.
We formalize this domain-specific inductive bias of cSCAN with the bias function B 2 - 200 To avoid circularity, we assume a variation H* of the logic embedding defined in Appendix D.4 with the only difference being that |-* does not use axiom (18), which depends on the inductive bias. Similarly, we use 4 to denote the variant of the minimal implication + that is based on - (rather than -+).
As mentioned in Section 3.2, we expect the learner to induce rule if it has been illustrated with sufficient number of variable substitutions. To formalize this, we define the function nuMMgubu X' X'x QR x 2 N such that num (x.q. (x.q.2) the number of different expressions that can be substituted for the variable in the rule to obtain rule in Q (35). We also define the function min QR x 2 - N such that min (q.2) ) is the minimum number of such substitutions for any variable used in q (36).
nuMsubbr (x,q. ((f(z) QN: R²; subst(af) {
(35)
min min, sEva(())
(n (num (x.g,)))
varlq)>0
min min (q.Q)
(36)
0,
otherwise
For any E BC& E, the bias function B(E) is then defined as follows:
(l ( (E( (EQ))=))))
(37)
(E)): (CC,)
A min (E,)(
(38)
(CC. (1.): K" -M( ((E) = ((()UBB()))
(39)
(40)
The set (E, C) the set of candidate rules with context C that we may want to induce from the example set E. Specifically, it is the union of all sets of rules Q that, together with E {e). allow us to (minimally) explain some example e E E (37).
The set (E) contains the assertions of all rules that are expected to be induced to be true based on the examples provided by E. Specifically, consists of the assertions I. of all the candidate rules q for which Qr (E,C) contains instances with at least different substitutions for each variable (38).
(y(E) contains the assertions of all rules for which E does not provide any evidence about whether or not they hold. Specifically, it contains an example ?.q.?.D) for all contexts C and rules q for which E does not imply a clear answer, i.c., 1 or 0 (39). Finally, B(E) is the union all the rules that should be induced be true and all the rules that should be induced to be unknown (40).
Discussion. Because all cSCAN contexts exclusively contain unconditional, monotonic examples, is sufficient to induce rules with the empty context, i..., only need to consider (E,22)
The consistency criteria in B, is relatively simplistic, which makes the task casier for humans. Indeed, we only need to make sure that each induced rule q is consistent with E but do not need to check whether the induced rules are consistent with each other. This allows us to greedily induce rules one at a time without worrying about potential conflicts among them.
DATASET GENERATION
The dataset generation process is config-driven with different dataset spec config being defined for each of the cSCAN datasets.
All cSCAN examples are automatically generated by a Python program built on the NLTK library (Bird et al., 2009). Generation is performed via the following steps.
TTo be released GitHub upon paper acceptance.
28

---

Under review as a conference paper at ICLR 2023
Context generation For efficiency, and to aid in generating clusters of related examples for calculation of the consistency metric, we generate examples in batches, in which we first generate a context and then generate multiple top-level examples that share that same context. Each context is created by first randomly generating a coherent set of interpretation rules of similar form to those shown in Figure 4 for the original SCAN task. While many of these rules may never be shown directly to the leamer, this initial rule set serves as kind of "basis rule set" from which the truth value of all other possible rules can be derived. We then randomly choose which of those basis rules to (a) provide explicitly in the context via an example that asserts that rule to be "true", (b) illustrate implicitly through some set of context examples sufficient to satisfy the task's inductive bias (see Appendix E.4 for details),
(c) illustrate insufficiently or not at all, or (d) contradict in one or more cases, so that it should be inferrable to be "false".
By generating contexts via the above procedure, we ensure that the basis rules cover each of the different possible replies and qualifiers for rule examples: monotonically "true" (case a), defeasibly "true" (case b), 'unknownn" (case c), and either monotonically or defeasibly "false" (case d). By extension, this also ensures that we achieve a diverse mixture of possible replies and qualifiers across the much larger set of rules that could be derived from different combinations of those basis rules.
We ensure that the exact ratio between the different above cases varies randomly from context to context, while achieving across the dataset as a whole the desired ratio that is configured in the dataset spec.
Top-level example generation Once we have fixed context, we then randomly generate set of (request, reply, qualifier) triples corresponding to the top-level examples that we wish to generate using the given context. To aid in sampling such examples, we first construct pair of inference engines in which we exhaustively generate all possible examples that would be considered "true" either monotonically or defeasibly (respectively) based on the given context. To generate non-rule example or positive rule example, we then randomly sample a "true" example from the full set of examples that were inferred by one of those inference engines. To construct a negative rule example, we first sample a positive example and then apply one of a number of different heuristics to construct an example that is similar to that positive example, but which is not among the examples inferred by the inference engine.
These two inference engines encapsulate the logic needed to ensure that each top-level example satisfies the consistency criteria stated in Section 2.3 and agrees with the domain-specifie inductive bias.
Sub-sampling In order to ensure that the different example classes are evenly represented in the dataset, we perform the top-level example generation process described above in separate streams, each of which dedicated to generating one specific example class (e.g.. "positive monotonic rule examples" or "non-rule examples with reply of unknown", etc.). We then sub-sample examples from each of the different streams in order to achieve the desired ratio of examples from each of the different classes. We ensure that the exact ratio between the different classes varies randomly from context to context, while matching the desired ratio in the dataset overall.
Splitting Due to interdependencies between the splitting algorithm and the example generation process (see, e.g., the notes on "additional top-level example generation" below), we perform the -aliddaaitt test split as one step of the dataset generation process, rather than generating a single dataset and then splitting it in multiple ways.
For performing MCD splits, we build on the open-sourced Python implementation of the MCD algorithm from Shaw al. (2020). While their version of the MCD algorithm consists of initially performing a random split and then iteratively swapping examples to increase compound divergence, however, we found that we were able to achieve higher compound divergence on the cSCAN dataset by implementing an algorithm closer to that of the original one described in Keysers et al. (2020). In this approach, we begin with empty train and test sets and then iteratively select examples from a large example pool to add to one of the two sets, while once in every three steps selecting an example from one of the two sets to remove and put back in the pool. At each addition or removal step, we select from among a random sample of 200 examples the one whose addition removal would maximize divergence at that stage, while keeping atom divergence low. One of the advantages of the insertion/delltion approach is that, in cases where it is acceptable to use only portion of the available examples, the process can be stopped early, which can result in train and test splits with significantly higher compound divergence than would be possible if the algorithm were constrained to use all of the examples from the example pool. As described in Section 3.3, we do perform such sub samplinl when constructing our MCD splits, beginning with a pool of 1.2M top-level examples and ending with 100K top-level examples in train and 10K top-level examples in each of validation and test.
29

---

Under review as a conference paper at ICLR 2023
While our above algorithm largely emulates the one described in Keysers et al. (2020), we do introduce one additional enhancement to enable generating a 3-way compound divergence split between train, validation and test. This is in contrast to Keysers et al. (2020), which performs only a single stage of MCD splitting for maximizing compound divergence between train and test. In our approach, we perform two stages of MCD splitting. with the goal of constructing a train set, validation set, and test set, with high pairwise compound divergence between any two of the three. In the first stage, we split the example pool into a valiatton pool and a test set, with the objective of maximizing the compound divergence between the two. In the second stage, we keep the test set fixed, while further splitting the train-validation pool into a train set and a validation set, with the joint objective of maximizing compound divergence between train and validation and maximizing compound divergence between train and test. We perform down-sampling in each of the two stages to increase the compound divergences that we are able to achieve. Appendix H contain statistics on the MCD splits, along with other details of the datasets.
Additional top-level example generation For the cSCAN Random variants, after performing a random split, we augment the validation and test sets by generating additional top-level examples for each validation and test context, using the same example generation logic described above. This allows us to achieve a higher density of logically related examples in the validation and test sets, so to yield a larger number of potential implications and contradictions for use in calculating the consistency metric. We do not perform this step for the cSCAN MCD variants, however, to avoid impacting the compound divergences between the train, validation and test sets. For this reason, we focus our investigation of consistency in this paper on the cSCAN random splits, as is only in the random splits that we are able to identify a significant number of implications and contradictions. Improving the sampling and splitting algorithms to enable investigation of consistency in MCD splits could be a topic for future work.
G CONSISTENCY METRIC CALCULATION
While the consistency metric as described in Appendix D in its most general form can be prohibitively expensive to calculate if approached naively, close approximation of it can be calculated efficiently through the application of several task-specific assumptions.
First of all, in the consistency metric calculation implications and contradictions among predictions for top-level examples that share the same context. While in general implications and contradictions can occur even among examples with different contexts (particularly if the examples are monotonic and if one context is a superset of the other), due to the way in which we construct the cSCAN dataset, such situations are extremely unlikely to occur. By focusing only on identifying implications and contradictions among examples sharing the same context, we are able to cleanly partition the dataset into independent clusters of examples, such that we can analyze each cluster efficiently in parallel. Dealing with clusters of examples that share the same context also simplifies analysis in that we now only need to consider the implications and contradictions among the request-reply pairs, while effectively ignoring the context.
with cSCAN, we consider for simplicity only
As the cSCAN validation and test sets contain up to 1000 top-level examples per context, however, would still be prohibitively expensive to enumerate each of the possible subsets of these examples to identify the sets that involve a "minimal" implication or contradiction. Instead, we find that we are able to identify the implications and contradictions much more efficiently by sceding an inference engine with the rule assertions that would correspond to the up to 1000 top-level request-reply pairs (ignoring negative rule replies and unknown replies for simplicity) and then performing exhaustive forward inference to determine all possible rules that could be inferred from combinations of these asserted rules. This is essentially the same inference process that is used in top-level example generation (as described in Appendix F), except that we take the additional step of tracking the provenance of each inferred rule, and we continue the inference process so as to generate all possible provenances of each rule (rather than omitting reprocessing of rules that have already been inferred via a different route). For the purposes of this consistency calculation, it is sufficient to consider as provenance the set of asserted rules that led to the given inference. Once the exhaustive inference process is complete, we then check each of the asserted rules (i.e., each of the top-level predictions) against the contents of the inference engine. If the asserted rule was also inferred from some other rules, then we take the full set of inference provenances for that rule, filter out any inference provenances that are supersets of some other provenance, and then treat each of those remaining provenances together with the asserted rule as one "minimal implication" (i.e,, as one minimal implication set F E im) as defined in Equation 24). Similarly, if a rule was inferred that shares the same left-hand side the asserted but contains a different right-hand side, then we look at
30

---

Under review as a conference paper at ICLR 2023
Table 6: Atom and compound divergence of the cSCAN MCD datasets between pairs of splits.
train test Atom
train validation Compound
validation Atom
Compound
Atom
Compound
eSCAN-B MCD
0.066
0.685
0.039
0.634
0.030 0.067
0.562
eSCAN-X MCD
0.050
0.722
0.046
0.726
0.724
Table 7: Atom and compound counts of the cSCAN MCD datasets. The "held out" column records the number of compounds that appear in the test split but not in the train split.
Min. occurrence of atom train
# Compounds valid
valid
test
train
test
all
held out
eSCAN-B MCD
4180
1110
1440
88
91
118
138
47
eSCAN-X MCD
755
455
729
758
1125
1754
752
the provenances of each such inferred rule, filter out any that are supersets of some other provenance of the same rule, and then treat each of those remaining provenances together with the asserted rule as one "minimal contradiction" (i.e., one minimal implication set F Eccont(E) as defined in Equation 25).
Despite the relatively large number rules with which we seed each inference engine and the extra expense of tracking multiple rule provenances, we find that we are able to complete exhaustive inference quickly in practice, due the fact that the majority of the rules that are asserted in top-level examples tend to be more specific than the rules that are typically asserted inside of context, and thus lead only limited number of interactions, which ends up being comparable to or less than the computational cost of the inference involved in constructing the contexts in the first place. In practice, when calculating the consistency metric in parallel using different work unit for each context, we find that we are able to calculate the consistency metric for full cSCAN experiment within few minutes.
H DATASET DETAILS
Table 6 shows atom divergence and compound divergence between pairs of splits.
Table 7 shows the minimal number times an atom occurs in each split, the number of compounds in each split, and the number of compounds in the test split that are held out from the train split.
Table 8 shows details of all splits of all datasets used in the cSCAN experiments.
31

---

Under review as a conference paper at ICLR 2023
Table 8: Key statistics of the cSCAN datasets.
cSCAN-B B Random
cSCAN-X Random
train
valid
train
valid
Number of examples
100,000
100,000
100,000
100,000
99,756
100,000
Number of contexts
1,000
100
100
1,000
100
100
Request: Rule (%)
50.1
50.0
50.0
50.1
49.9
50.0
Request: Non-rule (%)
49.9
50.0
50.0
49.9
50.1
50.0
Qualifier: Monotonic (%)
46.3
45.5
46.0
50.0
50.0
48.0
Qualifier: Defeasible (%)
53.7
54.5
54.0
50.0
50.0
52.0
Reply: Unknown (%)
13.3
13.3
13.1
13.2
12.9
13.6
Reply: (%)
21.8
21.3
21.8
21.9
22.2
21.5
Reply: No (%)
21.7
22.1
21.9
21.7
21.4
22.1
cSCAN- MCD
cSCAN-X MCD
train
valid
test
train
valid
test
Number of examples Number of contexts
100,000
10,000
10,000
99,999
10,000
10,000
11,921
6,115
6,509
7,965
4,627
4,599
Request: Rule (%)
72.5
64.0
59.8
73.8
67.4
Request: Non-rule (%)
27.5
23.5
36.0
40.2
26.2
32.6
Qualifier: Monotonic (%)
26.9
19.0
36.6
37.0
29.9
38.9
Qualifier: Defcasible (%)
73.1
81.0
63.4
63.0
70.1
61.1
Reply: Unknown (%)
52.2
67.2
35.2
39.7
50.6
36.3
Reply: Yes (%)
12.0
7.2
18.1
12.7
14.3
18.5
Reply: No (%)
12.1
7.8
18.2
12.9
13.7
19.2
cSCAN-B 100 Contexts
cSCCA-B
8000 Contexts
train
valid
test
train
valid
test
Number of examples
10,000
100,000
100,000
800,000
99,713
100,000
Number of contexts
100
100
100
8,000
100
100
Request: Rule (%)
50.1
50.1
50.1
50.1
49,9
50.1
Request: Non-rule (%)
49.9
49.9
49.9
49.9
50.1
49,9
Qualifier: Monotonic (%)
44.6
45.6
45.5
46.1
45.5
44.9
Qualifier: Defeasible (%)
55.4
54.4
54.5
53.9
54.5
55.1
Reply: Unknown (%)
12.8
13.0
13.6
13.4
14.2
13.6
Reply: Yes (%)
22.1
21.9
21.1
21.8
21.4
21.1
Reply: No (%)
21.8
22.3
21.6
21.6
22.2
32

---

Under review as a conference paper at ICLR 2023
I ANALYSIS OF REALISTIC CSCAN EXAMPLES
In this appendix, we provide realistic examples from a slightly carlier version of the cSCAN dataset and outline systematic strategy for solving them. We selected total of 22 examples (Figure 8) that are based on a single context of size 23 (Figure 7). We made sure that the different classes (see Section 2.4) are well represented among the selected examples. This means that the examples cover all valid combinations of request types, replies, and qualifiers.
To illustrate how cSCAN can be solved by humans, outline one possible strategy below. This strategy is hased on the assumption that we are aware of the grammar (Appendix H.2 and Figure 5), the rule semantics (Appendix E.3), and the inductive bias (Appendix R.4) of eSCAN.
L.1 MONOTONIC EXAMPLES
In a first step, we check whether given example can be deduced directly from the context. This is the case for all requests that exclusively contain syntactic constructs for which the behavior is completely determined by explicit rules in the context. In our context C, the rules (CI) through (CI1) completely determine the behavior of all constructs except "look", "thrice", and "and". This means that we can deduce the reply for the examples (E1), (E2), (E3), (E4), (E5), and (E7), which we consequently mark as monotonic.
As an illustration, consider the request "jump around left twice" from example (E1). Once we know that [jump]=JUne JJMP (C3), [left] = PURPLE (C1), [xl around x2] =[+]][[=2] [x1] (C8), and [xl twice] [x]] (C10), we can immediately deduce that "jump around left twice" is translated to "JUMP PURPLE JUMP". Note that some of the examples are closely related to one another. For example, the example (EI) is an instance of the rule [x1 around x2 twice] = [xi] [x2] [x1] from example (E5).
The examples (E6), (E8), and (E9) contain some of the syntactic constructs that are not completely determined (i.e., "look", "thrice", and "and"), but they can still be deduced from the context and are thus marked as monotonic. The example [C.[look left] WHITE WHITE PURPLE, (E6) follows directly from the rules [left] = PURPLE (C1) and [look x1] = WHITE WHITE [X1] (C12).
Similarly, the example (C. [[o[look] WHITE YELLOW, M.MM) (E8) can be deduced from the examples (C7), and (C12). Indeed, if [look]] = WHITE YELLOW were true, (C7) would allow us to deduce the rule [look x1] = WHITE YELLOW WHITE YELLOW [x1]] which contradicts (C12).
Finally, (E9) can be deduced from (C2), (C8), and (C14) because [x1 [arice][[xi][[]] would imply that [x] around right thrice] = [x]] GREEN [x][[[]] GREEN [xl], which contradicts (C14).
and
1.2 DEFEASIBLE EXAMPLES
To determine the reply of examples that cannot be deduced from the context, we check whether the context allows us induce some rules for the constructs "look", "thrice", and "and", which are not fully determined.
Inducing rules for "look", "thrice", and "and".. We start with the primitive command "look" and identify all context examples that contain "look" but none of the other partially-determinee constructs "thrice" and "and". This yields the rules (C12) and (C13). Together with (C7), the rule (C12) tells us that "look" must either be "WHITE" or undefined (i.e., ?), and the same holds for (C13). While these two examples alone are not sufficient under the task's 's inductive bias (Appendix F.4) to justify inducing the general form of the "look" rule we will proceed with the assumption that [look] = WHITE for the time being.
Next, we apply the same process to the syntactic construct "thrice". This means that we identify all context examples that contain "thrice" but not "look" and "and", which are (C14), (C15), and (C16). The first or be undefined. However,
two of them tell us that "[xl thrice]" must either translate to example (C16) is incompatible with the rule [x1 thrice] = [x][[[]][[]]]] In accordance with the inductive bias, we therefore cannot induce general rule for "thrice".
Now, we apply the process to "and", which means that we look at the rules (C17) and (C18). Together with the example (C10), they both tell us that "[xl and x2]" must either translate to [xx]]]]x[[xx]] or be undefined. So, we proceed with the assumption that [x] and x2] = [ [x1].
As next step, we look at the examples (C19) and (C20) whose requests contain both "and" and "look" but not "thrice". Both of them follow from our current assumptions [[ook]] = WHITE and [xl and x2] = [22] [+]] This means that we have now found four examples in the context that agree with
33

---

Under review as a conference paper at ICLR 2023
(CI)
[[eft] - PURPLE, 1:
(C2)
[[right] GREEN, 1
(C3)
[jump] = JUMP, 1
(C4)
[[run] L.TURN.
(C5)
[tum] - -YELLOW,
(06)
[walk] RUNN, D
(C7)
[xl x2] [ D
(C8)
[x] uround 1xll [x21[x1], I 1)
(C9)
[xl opposite x2] [x]][x]]]]]]]]
(C10)
[xl taticte=[[[[]] 1)
(C1l)
[xxl x21- Ix [x2 1)
(C12) [[ookxi]= WHITE WHITE [xi], 1)
(C13) ([look opposite x1] WHITE WHITE [x]], 1)
(C14) [xl asound right durce] [x]] GREEN [xi] [xl] GREEN GREEN [xI]
(C15) [xx opposite right thrice] [x]] [x|] GREEN [ GREEN [x]] [x]] GREEN, 1)
(C16) crun right twiec after walk thriee, RUN RUN CTURN LTURN GREEN RUN RUN LTURN LTURN GREEN;
(C17) [xl and x2 twice] - [22] [x2] [x]], D
(C18) [[xl twice and x2] [22] [21 [xl].
(C19) (jump around right and look around left WHITE PURPLE WHITE WHITF PURPLE WHITE JUMP GREEN JUMP)
(C20) (jump opposite left twice and look around left, WHITE PURPLE WHITE WHITE PURPLE WHITE JUMP JUMP PURPLE)
(C2) ) lloo left thrice, WHITE WHITE PURPLE WHITE WHITE PURPLE WHITE WHITE PURPLE)
(C22) wwak twice jump left thrice,
JUMP PURPLE JUMP JUMP PURPLE
(C23) left thrice and run, LTURN LTURN YELLOW YELLOW PURPLE YELLOW YELLOW PURPLE YELLOW>
Figure 7: context C from the cSCAN dataset. The examples that make up the context are sorted, starting with 11 explicit rules that specifyy the complete behavior of all syntactic constructs except "look". "thrice", and "and". The remaining examples are sorted such that we first have the examples that do not contain "thrice" and "and", then the examples that do not contain "and", and finally the remaining examples.
Monotonic (non-rules)
jump left twice,
(R2)
(C. walk around right after run opposite left toke, LTURN ETURN PURPLE RUN GREEN RUN LTURN LTURN PURPLE RUN GREEN RUN, M: left and jump around twice, JUMP GREEN JUMP JUMP GREEN JUMP LTURR LTURN PURPLE, M)
Monutonic (rules)
[xT1 PURPLE, (C. Ixl around x2 twice] - [xl] 1,
opposite
(E5)
(C, Look kin] WHITE WHITE PURPLE, MD
[xl] [x1] LTURN, (EX) (C. Look] WHITE YELLOW, 0, M)
Irun around
thric]] [xi] [x1]. M:
Defeasible (non-rules)
(E10)
look opposite twice, WHITE WHITE PURPLE,
(EII) (C, run right and look around right, WHITE GREEN WHITE WHITE GREEN WHITE LTTRRN LTURN GREEN,
(E12) (C, look left twice afler look, WHITE WHITE WHITE PURPLE WHITE WHITE WHITE PURPLE, D thrice, D)
(E13)
(E14) (C, run opposite right and jump around lelt thrice, 7, D)
(E15)
walk around right thrice after around left twice,
Defcasible (rules)
(E16)
[ook] WHITE,
[L17] (C. Look around xi] WHITE [xi] WHITE, 1)))
(E18) (C, Ixl and x2 Shrice] - [x]T [X2][S2]] I,
(b19)
opposite xi thrice] WHITE WHITE, 0,
(E30)
(C. Ixl thrice] Mhrice]-ssi [x]] [x]] [x]1. D)
(E21)
after thrice] [x2]
[xil [x2] [2]] (xIL,
(F27)
(C) 1cl around thrice rwice] [x3] [x]]vv] M D.
Figure 8: Examples from the cSCAN dataset. The examples are based on context C shown in Figure 7 above, and they are grouped by request type and qualifier.
34

---

Under review as a conference paper at ICLR 2023
assumed rules for "look" and "and", and there examples that contradict them. Based our inductive bias, this means that we induce these rules.
All the remaining examples (i.c., (C21), (C22), and (C23)) contain the construct "thrice", for which we cannot induce a general rule.
Applying the induced rules for "look" and "and". Using the induced rules for "look" and "and", can determine a concrete reply for the examples (E10) through (E12) as well as (E16) through (E19). These examples marked defeasible because they are based (among others) on induced rules, which means that the reply might change if the context is expanded.
Using reply "unknown" for examples containing "thrice". Because are not able to induce generic rule for "thrice", we are not able to determine a concrete reply for the examples (E13) through (E14) and (E20) through (E22). Instead we use the reply "unknown" (?) and mark them as defeasible because the reply might again change if the context is expanded.
J EFFECT OF DATA SIZE
As an additional set of experiments, we investigate the impact of training data size on the performance of the relatively strong pre-trained T5-Base (220M parameters) baseline. In each experiment, we dataset generated with a similar mix of examples as in eSCAN-B, but with a varying number of contexts and examples in the train set. As can be seen in the results in Table 9, while the model is able to achieve high accuracy and relatively high consistency as the train size approaches 800K examples, performance drops dramatically as the train decreases from 100K examples down to 10K examples. This suggests that finding ways to reduce models' dependency on large amounts of task-specific training data will be an additional important theme for future work in conceptual learning.
Table 9: Accuracy function of training data size. Results from experiments with pre-trate pre-trained T5-Base (220M parameters) on datasets of similar form to eSCAN-B, but of varying size.
Contexts
Examples
Accuracy
Consistency
100
10,000
38.6
6.9
1,000
100,000
92.6
71.3
8,000
800,000
97.7
91.9
K
EXTENDED RELATED WORK
K.1 COMPARISON WITH EXISTING DATASETS
In representing the input of a CLT as request paired with context, we build on long tradition of QA and reasoning task formulations that provide knowledge relevant to a task via various forms of context, such as text passage (Kwiatkowski et al., 2019; Weston et al., 2015; Dua al., 2019; Sinha al., 2019; Yang et al., 2018; Rajpurkar et al., 2016; Levesque et al., 2012). logical premise (Roemmele et al., 2011; Dagan al., 2005; Bowman et al., 2015), set of natural language statements (Talmor al., 2020), knowledge graph fragment (Sinha et al., 2020), antecedent description grammar (Cohen, 1994), dialog (Semantic Machines et al., 2020; Budzianowski et al., 2018), image (Antol al., 2015: Johnson et al., 2017: Hudson & Manning, 2019; Bahdanau et al., 2019a;b), grid world (Ruis et al., 2020), DB schema (Yu et al., 201 8).
Here, to give better sense of how a CLT is similar to and different from these existing task formulations, we make a closer comparison of cSCAN with several representative NLU tasks that provide explicit knowledge as part of the input and satisfy some of the desired properties for a CLT formulated in Section 2.1. This is illustrated in Figure 9.
Note that we focus our comparison here specifically on the defining features of CLT. This should not be construed commentary the overall usefulness quality of these benchmarks. Indeed, many of the benchmarks described here have complementary strengths which cSCAN lacks, such as being based on true natural language, supporting multi-modal input, illustrating specific domains of reasoning, or covering more complex reasoning or syntax.
Talmor et al. present a series of "Leap-of Thoughhh tasks (Talmor et al., 2020) where the learner needs to judge yes/no hypotheses by performing inference over knowledge that is obtained implicitly from language model pretraining while part of the knowledge is also provided explicitly using a context containing natural
35

---

Under review as a conference paper at ICLR 2023
Context (explicit knowledge)
Request
Output
Set-like
Variable (incompatible across examples)
Rules (rules assertable)
Examples (request / reply pairs assertable)
Rules (rules requestable)
Unknown
Deductive
vs.
Inductive
Leap-of-Thought
X
X
X
GraphLog
X
X
X
X
bAbl
X
X
X
X
gSCAN
X
(v/)
X X
X
X
Natural Questions
X
X
X X X
X
cSCAN
Figure 9: Comparison of cSCAN and other NLU task against the key features of conceptual learning. For the context (explicit knowledge), we distinguish whether is organized as a set of independent units rather than monolithic block (column I) and whether different examples use different and sometimes contradictory knowledge (columm 2). We also distinguish whether the context contains both rules (column 3) as well examples (column 4). For the request, indicate whether the task contains examples that explicitly test the truth value of rules (column 5). For the output, we indicate whether the task requires identifying whether request cannot be answered based on the given context (column and whether it distinguishes between inductive and deductive reasoning.
language statements. This setup satisfies various CLT properties. In particular, arbitrary hypotheses can be asserted in the context, and the requests can ask for the truth value of any context statement (examples and rules coincide in this task because all examples are yes/no hypotheses). However, the knowledge is constant across all examples (albeit different parts of this knowledge are provided explicitly for different examples). Therefore, Leap-of TfoTughht tasks do not investigate whether a learner is able to adapt transient knowledge, e.g., explicit knowledge that may contradict the implicit knowledge obtained during language model pretraining (e.g., actor was married and is now divorced). Similarly, these tasks exclusively focus on monotonic inference: they not test whether the learner able to induce defeasible hypotheses from the explicit knowledge nor do they test the ability to identify certain hypotheses "unknown".
GraphLog (Sinha et al., 2020) is a benchmark suite based on tasks that are quite similar to CLTs. The learner is presented with part of graph consisting of labeled edges (context) and then needs to predict the label for an edge that is not part of the context. For example, the context may contain two "father- child" relations and the learner needs predict the "grandfaaher-graaddhhld"" relation. As for cSCAN, the graph constructed automatically based on a set of first-order logic rules, which make sure that each task is consistent. However, in contrast eSCAN, the underlying rules cannot be part of the context, expressible as requests. In the CLT terminology, this means that GraphLog tests only the case where the context consists of examples and the learner has to induce new examples. It does not support the case where the context contains rules and the leamer has to consistently combine and apply these rules deductively.
are they
The bAbl tasks (Weston et al., 2015) also require the learner answer a question using variable context consisting of natural language statements. Each context consists of a sequence of relatively simple factual statements that are, at least that bAbl context does not directly correspond to a set-based context that we are using with CLT's. One way to bridge this gap to consider sequence of bAbl statements as single "macro rule". but the truth value of these rules cannot be requested. bAbl contains some tasks that require deductive reasoning and some tasks that require inductive reasoning.
some of the tasks, drder-dependent (Dehghani al., 2018). This means
The gSCAN task (Ruis al., 2020) is extension of SCAN where the leamner is provided with context that describes a spacial configuration of objects in order to translate commands into sequences of actions. However, with bAbl, the context consists of a single dedicated structure describing the spacial configuration as a whole rather than a set rules that describe the different objects that are part of the spacial configuration by one. This makes the language to specify the context disjoint from the request language.
36

---

Under review as a conference paper at ICLR 2023
As example of reading comprehension task, Natural Questions (Kwiatkowski et al., 2019) is benchmark where the learner is given a Wikipedia page as context and then needs to answer a natural language question by outputting long answer (e.g., the paragraph containing the answer) as well as a short answer. As with bAbl, the context is not a set of independent rules but instead a sequence of inter-dependent statements, which makes this benchmark quite different from a CLT..
K.2 OTHER RELATED WORK
Compositional generalization. Our evaluation of cSCAN MCD splits builds on existing research in measuring the ability of machine learning models to compositionally generalize (Keysers et al., 2020; Lake & Baroni, 2017). In response to compositional generalization benchmarks such as SCAN, a range of techniques have been proposed which have not yet been evaluated on cSCAN Some of these solutions involve specialized architectures for enforcing a compositional bias (Qiu et al., 2022; Chen et al., 2020; Liu et al., 2020; Nye et al., 2021). Such architectures are appealing due to their potential for achieving a principled solution compositional generalization, but would require some effort to adapt to the context and heterogenous output of a CLT. In the past, some specialized architectures have shown limited success when transferring to new tasks, compared to more general techniques such as language model pre-training (Furrer et al., 2020). In the latter category, there is some promise shown by recent developments in decompositional prompting techniques, which have led to strong results the SCAN MCD splits using off-the-shelf large language models (Zhou et al., 2022).
Instruction following, In evaluating the ability for system apply rules task, our work relates to research in building systems that learn to follow instructions (Goldwasser & Roth, 2014), including recent research in the inslluctionfollooinng capabilities of large language models (Wei et al., 2022; Ouyang et al., 2022; Wang et al., 2022). Our approach differs in that we provide in the context a set of rules, each of which may be applicable some part of the task, and we evaluate the ability to infer new rules in addition to applying the rules to an underlying task.
Meta-learning, Meta-leaming or "learning to Ieam" generally refers to setup where the learner is provided with family of tasks, which are also called episodes, each of which comes with its own set of training examples and test examples. A learner is able to "leam to learn" if its performance for each task increases both with increasing training data and with an increasing number of tasks (Thrun & Pratt, 1998; Hospedales et al., 2021; Finn et al., 2017).
The presence of examples within the context of an example gives CLTs nested structure that allows us to view CLTTs through the lens of the meta-leaming setup. In this view, top-level examples that share the same context correspond to episode where the context examples are the training examples and the topllevel examples (w/o the context) are the test examples.
Closely related to cSCAN are two pieces of work that apply meta-learning to SCAN, both of which generate large numbers of SCAN-like grammars, from which they construct meta-learning episodes, similarly to how we generate cSCAN contexts based on SCAN-like grammars. Lake (2019) uses ahemory-augmeent network to attend to the train examples of each episode, while Nye et al. (2020) trains for each episode program synthesis model that outputs the underlying rules of the task. Our approach differs in that we include in the context mixture of rules and examples, rather than just examples of the underlying task, and we use the sautheticall-geneeeee system to generalize to many different rule sets, rather than using meta-learing techniques as a means to improve accuracy on the original SCAN task.
contexts to define a new task for evaluating the ability of the
Rule induction and logic deduction tasks. CLTs require the learner to judge whether a certain rule can be induced from a given set of observations (i.e., examples provided as part of the context). This is similar to rule induction task (Cohen, 1995; Reddy & Tadepalli, 1998; Grzymala- Busse, 2010), with the main caveat that the learner only needs to verify rules rather than generate them. CLTSs also require the learner to apply rules and judge whether rule may be deductively obtained from set of other rules.
Interpretable ML models. CLTs make part of the rules that govern the underlying tasks explicit, which allows to "introspect" the behavior of the leamer by asking, as part of the task, whether not certain nule holds (inductively or deductively). This means that the question of whether and why a certain model behaves correctly incorrectly can be broken down into two parts: (a) did the model learn the right rules and (b) is it able to apply these rules consistently. This is related to, yet different from, other efforts to make MI. models more interpretable. For example, Sushil et al. (Sushil et al., 2018) propose method to
37

---

Under review as a conference paper at ICLR 2023
induce if-then-else rules to explain the behavior of ML models. However, unlike for CLTs, this method is external to the actual task: it does not reflect whether the model claims a certain rule to be true but instead identifies the if tf-thee-else rules between different input features and class labels that are the most important for classification according to the model.
Consistency. Our consistency metric is related research into evaluating and improving the consistency of neural networks. One closely related work is Li et al. (2019), which evaluates consistency by generating clusters of examples that by construction are related via logical symmetry, transitivity, or some other logical constraint. Based on these examples, they calculate a "conditional violation" metric, which similarly to our consistency metric is ratio of violated (in our case, satisfied) constraints vs. the total number of logical constraints. Our approach differs in that we gather the logical constraints automatically using symbolic inference over the generated examples, rather than depending on a specific algorithm for generating related examples.
Handling of large contexts. Due to the potentially large context size in CLTs, another relevant area of research is how to deal with very large contexts, including large set-like contexts. One line of research in this area involves modifying the Transformer architecture to be able to handle longer inputs more efficiently (Tay et al., 2020a;b). We evaluated two such architecture variants in our Long T5 and LongT5, Global baselines (Guo et al., 2022), but many other such variants have been proposed (Gu et al., 2021; Zaheer al., 2020; Choromanski et al., 2020; Wang al., 2020). In cases where the context takes the form of a set or a graph, some approaches seek to explicitly take into account this structure by encoding the input structure in positional embeddings (Herzig al., 2020), guiding the Transformer's attention via the structural relations within the input (Ainslic et al., 2020), or message-passing in graph neural networks (Gilmer et al., 2017: Battaglia et al., 2018). Other lines of research seek more efficiently deal with large pools of potentially relevant knowledge by either performing cross-attention from portions of the input to knowledge stored in neural memory (Verga et al., 2020) or by retrieving only the most relevant material from a knowledge base or text corpus for concatenation to the input (Guu et al., 2020; Pasupat et al., 2021).
L REPRODUCIBILITY
Hardware and training period Table-10 shows the different hardware used for each experiment and the training period (in steps).
T5 versions For our T5 baselines, we use T5X (Roberts et al.. 2022). which a re-implementation of TS in JAX (Bradbury et al., 2018) using Flax (Heek et al., 2020). Table 11 presents the configurations of different T5 variants that we used in our experiments. For the full-attention version T5, we experiment with both fine-tuning from standard pre-trained checkpoint and training from scratch.
For LongT5 and LongTS-TTlooee while we initially evaluated with both fine fine-tuning from a standard pre-trained checkpoint and training from scratch, when fine-tuning, we failed to find setup in which the models converge on the train set, possibly due to poor compatibility between the cSCAN task and the summarization-oriente PEGASUS Principle Sentences Generation pre- pre-training objective (Zhang et al., 2019) used in LongTS, For this reason, we only report results on LongT5 and LongTS-TGlobal models trained from scratch.
For each of the architectures, we evaluate at minimum two sizes: Small (60M parameters) and Base (220M parameters). For the pest-perfomming T5 architecture, We omitted experiments on Large variants of the other architectures for reasons of computational cost, as the poor performance the Small and Base sizes suggest that it is unlikely for the performance of LongT5, LongT5- or the non-pretrained version of TS to improve significantly with model size alone.
further evaluate on size Large (770M parameters).
Hyperparameters Table 12 summarizes the hyperparameters used for each of the baselines. The reasons for choosing those hyperparameters are:
* Config: We chose the config version based on experiments an earlier version of the dataset. In that version, we found that T5 achieved higher performance using the 1.01 config, while LongT5 and LongT5-TGlobal performed better the 1..1.1 config.
Learning Rate: A constant learning rate is the standard way to fine-tune pre-trained T5 models. For non-preen preetrard models we found that constant learning rate performed as well as the inverse square root decay with linear warmup scheduler (the standard learning rate scheduler for pretraining T5).
38

---

Under review as a conference paper at ICLR 2023
Table 10: Hardware and training period (in steps).
Model
Size
Pretmain
Dataset
TPU Type
TPU Slices
Training Steps
Thaining Epochs
T5
S
TTue
cSCAN.B B
TPU V2
64
150,000
192
T5
'Tree
cSCAN-B
TPU V3
16
50,000
64
T5
|
TTre
eSCAN-B
TPU V4
64
50,000
64
T5
False
cSCAN-B
TPU
300,000
T5
/
False
cSCAN
TPU V3
16
150,000
192
Longl'
False
eSCAN-B
TPU V2
64
300,000
384
LongT5
8
Palse
cSCAN-B
TPU
16
150,000
192
Tongoo-TGlobal
False
cSCAN-B
T2U
64
300,000
384
LomgT5-TGlooal TGlohal
False
cSCAN
T3U
150,000
15
|
'True
<SCAN-X
TPU V2
64
150,000
192
T5
8
'Trec
cSCAN-X
TPU
16
50,000
64
T5
Trie
cSCAN-X
T4
128
50,000
64
T5
False
<SCAN.X
TPU V2
64
300,000
384
T5
cSCAN-X
TPU V3
16
150,000
LongT5
Palso
cSCAN-X
TPU V2
64
300,000
384
LongT5
False
<SCAN-X
T3U
16
150,000
192
L5 TGlohal
S
False
cSCAN-X
TPU
64
300,000
384
False
cSCAN-X
TPU V3
16
150,000
192
T5
Truc
cSCAN-B MCD
TP
64
150,000
192
T5
| Tme
cSCAN-B MCD
TPU
16
50,000
64
T5
L | TTue
<SCAN B MCD
T4U
64
50,000
64
15
False
cSCAN-B MCD
IPU V2
64
300,000
384
TS
8
False
cSCAN-B MCD
TPU
16
150,000
192
LongT5
False
cSCAN-B MCD
T2
64
300,000
384
L5
cSCAN MCD
TPU
16
150,000
False
cSCAN-B MCD
V2
64
300,000
384
L5ngT5 TGlobal
B
False
cSCAN B MCD
TPU
16
150,000
192
T5
TTree
CSCAN-X
TPU V2
64
150,000
192
T5
Troe
cSCAN-X
TPU
16
50,000
64
T5
| Tre
cSCAN-X
T4
50,000
64
T5
5 | Fulse
<SC
TPU
64
300,000
384
TS
B
False
CSCAN-X
TPU V3
16
150,000
192
TongT5
False
ASCAN-XMCU
TPU V2
64
300,000
384
T5
False
cSCAN-X MCD
V3
16
1,50,000
192
LongT5-TGlohal TGlobal
False
cSCAN MCD
TPU V2
300,000
LongT5- TTloobball
False
cSCAN-X
TPU V3
16
150,000
192
T5
8
TTre
cSCAN-B 100 Contexts
TPU
16
100,000
1280
T5
R
TTree
cSCAN-B 5000 Contexts
T3
16
100,000
25.6
T5
True
cSCAN B 8000 Contexts
TPU V3
16
100,000
16
Table 11: Configurations of different T5 variants.
Name
#Parameters
#Layers
dmoael
#heads
Small
60M
6
512
2048
Base
220M
12
768
3072
12
Large
770M
24
1024
4096
16
39

---

Under review as a conference paper at ICLR 2023
Table 12: Hyperparameters used in the cSCAN baselines.
TS (Pretrained)
T3 (Noo-Pretranne)
LongTS Small
LoogTS-TGlobal
Large
Small
Conlig
T5.10
15.10
T5.10
T5.10
T5.10
T3.11
15.11
T3.11
15.11
Learning Rate Schoduler
Coustant
Coustant
Constant
Constant
Coustant
Constant
Constant
Constant
Constant
Leurning Rale
Butch Size
128
128
128
128
128
128
128
Loss Normalizing Factor
233472
233472
233472
233472
233472
233472
233472
233472
233472
Initial Checkpoint
1000000
999900
10007000
0
Loss Normalizing Factor: By instruction of T5 creators, used a fixed loss normalizing factor
(Petrxininngaaccccseeee(21))) Turgeffooeenl -34
for fine-tuning and we used the same value for non-pretrained models to maintain consistency.
Note that for the small models using T5.1.1 config, we set the number layers and heads to and respectively match those set in T5.10 config.
Tokenization All models use the pretrained SentencePiece tokenizer (Kudo & Richardson, 2018) provided T5, which is pretrained to cover the English, French, German and Romanian languages with 32,000 tokens. We also tried using a simple whitespace tokenizer, which resulted similar performance when compared to the pretrained T5 tokenizer.
40

---

Under review as a conference paper at ICLR 2023
Table 13: Breakdown of accuracy by example characteristics on cSCAN-B Random.
Knownness
Request/Reply Type
Model
Pretrain
Size
Defeasible
Monotonic
Unknown
Neg. Rule
Pos. Rule
Non-Rule
Thue
S
90.2
93.9
97.4
79.8
95.8
B
90.5
93.9
97.7
78.8
96.9
96.2
L
95.3
97.7
97.7
91.9
98.1
97.6
False
S
13.9
16.4
75.0
32.0
27.7
2.9
B
15.0
12.3
74.4
33.4
20.5
2.7
LongTS-TGlobal
False
S
16.3
15.9
74.1
28.8
34.2
3.0
B
13.9
13.1
68.7
29.5
24.0
2.2
LongT5
False
S
17.1
14,0
73.8
31.2
29.5
3.0
B
16.1
17.8
80.6
33.1
32.6
3.9
T5 w/o Context
True
B
19.3
23.5
83.0
40.6
44.9
5.8
Table 14: Breakdown of accuracy by example characteristics on cSCAN-X Random.
Knownness
Request/Reply Type
Model
Pretrain
Size
Defcasible
Monotonic
Unknown
Neg. Rule
Pos. Rule
Non-Rule
True
59.2
68.7
43.3
75.2
B
62.6
71.6
98.4
46.5
79.4
74.8
L
74.9
83.7
97.7
69.5
91.4
80.6
False
S
10.3
16.4
63.0
29.0
25.1
1.9
B
11.2
14.8
72.7
29.5
22.6
2.2
LongTS-TGlobal
False
S
13.0
14.6
72.4
28.1
26.8
2.7
B
9.4
18.2
78.6
28.1
28.5
2.7
LongT5
False
S
11.7
16.3
70.8
32.8
23.5
2.5
B
12.3
15.2
57.8
29.5
25.4
2.0
T5 w/o Context
True
B
17.0
20.6
83.3
35.5
39.9
4.0
M BREAKDOWN METRICS
M.I CSCAN-B RANDOM
Table 13 shows breakdown of accuracy by example characteristics on cSCAN-B Random. From the breakdowns for La-gerge (770M parameters), attention, it is feasible to achieve high accuracy across example classes, Across the pre-trained TS models, however, notice several trends:
that for sufficiently large model with full
Examples with reply of "uunnnonn" tend to be easier than other examples.
Negative rule examples, i.c., those with reply of "false", are the most challenging class of example.
Accuracy on non-rule examples is roughly similar to accuracy on rule examples, despite the fact that rule examples involve classification into much smaller space of possible replies (which would make them more amenable to solving through guessing).
In contrast, when looking at the results of the models that failed to outperform the naive baseline, we that for these, accuracy rule examples is significantly higher than that non-rule examples, with the highest accuracy on examples with the reply of "unknown", consistent with the view that these models relying guessing based superficial example characteristics.
41

---

Under review as a conference paper at ICLR 2023
Table 15: Breakdown of accuracy by example characteristics on cSCAN-B MCD.
Knownness
Request/Reply Type
Model
Pretrain
Size
Defcasible
Monotonic
Unknown
Neg. Rule
Pos. Rule
Non-Rule
Thue
S
38.2
24.0
99.8
60.0
22.5
B
38.8
25.3
99.9
59.8
35.3
27.2
L
55.3
46.8
99.8
75.6
60.4
42.4
False
S
17.9
14.3
96.6
25.8
30.3
8.9
B
13.8
14.7
97.4
26.7
24.2
6.2
LongTS-TGlobal
False
S
18.4
14.0
98.5
30.3
25.7
8.9
B
13.1
13.2
97.4
29.8
16.7
10.8
LongT5
False
S
16.7
15.0
97.2
36.8
18.7
10.3
B
13.4
14.0
97.6
42.1
6.5
6.7
T5 w/o Context
True
B
25.2
19.5
96.6
38.0
40.2
14.6
M.2 CSCAN-X RANDOM
Table 14 shows breakdown of accuracy by example characteristics on eSCAN-X Random. From the pre-trained T5 models, we can see the following trends:
Examples with reply of "unknown" continue to be much easier than other examples. In fact, the models are able answer these examples with comparably high accuracy on eSCAN-X as on cSCAN-B.
Negative rule examples continue to be the most challenging class of example.
While the models struggle with both rule examples and non-rule examples, accuracies lower on rule examples than on non-rule examples, again despite the fact that rule examples would be more amenable random guessing.
For the models that failed outperform the naive baseline, the pattern is similar eSCAN-X as on eSCAN-B.
M.3 CSCAN-B MCD
Table 15 shows breakdown of accuracy by example characteristics on cSCAN-B MCD. From these results, we can see the following trends:
All models achieve particularly high accuracy on examples with reply "unknown", as would be expected from the dataset stats shown Table where we can see that this class of examples makes up over 50% of the examples in the cSCAN-B MCD train set. This makes the answer of "unknown" a natural guess any situation where the model is unsure.
Given that T5 w/o Context is able to achieve significantly higher than zero accuracy on rule examples with replies other than "unknown", however, is clear that the model is doing more than simply predicting "unknown" every time. Rather, it appears that a moderate amount of statistical clues must be available in the request itself allow some degree of reducated guessing" of the reply, particularly in the case of rule examples.
For all models, accuracy on non-rule examples lags significantly behind accuracy on rule examples. This in contrast to the eSCAN Random datasets, where the wwoonge--pefforming pre-trained models frequently performed better non-rule examples than rule examples. One reason for this difference is likely the fact that the train set for cSCAN-B MCD skewed toward rule examples, which make up somewhat over 70% of the dataset. Taken in light of the observation above about the naive T5 w/o Context baseline, however, the poor performance on non-rule examples also suggests that the T5 baselines may be achieving even less proper "understanding" the examples than one would have thought from looking at the overall accuracy numbers alone, and is likely relying to large degree on "educated guessing". based statistical clues from the
42

---

Under review as a conference paper at ICLR 2023
Table 16: Breakdown of accuracy by example characteristics on cSCAN-X MCD.
Knownness
Request/Reply Type
Model
Pretrain
Size
Defeasible
Monotonic
Unknown
Neg. Rule
Pos. Rule
Non-Rule
Thue
S
51.6
57.1
99.5
65.0
66.6
B
50.3
54.2
99.7
14.3
83.1
66.0
L
60.0
67.4
99.7
36.2
88.6
73.4
False
S
13.3
19.4
96.5
29.2
28.0
6.8
B
13.5
17.3
94.8
28.7
24.4
5.6
LongTS-TGlloal
False
S
14.6
17.3
96.4
28.1
26.5
6.4
B
14.4
17.8
95.5
30.6
24.8
6.5
LongT5
False
S
13.2
19.0
94.9
32.8
23.5
6.3
B
14.7
18.4
95.1
32.2
24.8
6.8
T5 w/o Context
True
B
12.6
21.2
96.2
27.1
33.3
8.7
request and context, which is much easier to do on rule examples than on non-rule examples, due the smaller space of possible replies for rule examples.
M.4 CSCAN-X MCD
Table 16 shows breakdown of accuracy by example characteristics on cSCAN-X MCD. From these results, we can see the following trends:
Similarly to eSCAN-B MCD, all models achieve high accuracy on examples with reply "un- known", which again the most commonly occurring class of examples this dataset (around 40% of examples in the train set).
For pre-trained T5, however, accuracy non-rule examples is significantly higher than cSCAN-B MCD, suggesting that these models are likely benefiting from the more balanced distribution of examples the cSCAN-X MCD dataset, where around 40% of the train examples are non-rule examples, compared with less than 30% in cSCAN-B MCD.
The large gap in accuracy between negative and possible rule examples pre- pre-trained T5 suggests that while they are able to use information from the context to do a better job than the naive T5 w/o Context distinguishing between "unknown" and "not unknown" rules, they are still relying largely on guessing for determining the rules' actual truth value.
M.5 EFFECT OF EXAMPLE AND CONTEXT CHARACTERISTICS
Figures 10. 11, 12, and 13 show T5's performance the splits of cSCAN-B, cSCAN-X, eSCAN-B MCD, and eSCAN-X MCD datasets with respect various features of the examples and the contexts, broken down into rule and non-rule examples.
The features being considered are:
The number of distinct rules used to create an example. For example, for the request "walk and walk", num nules is 2: it created with the rules: [x] and x2]==.. and [walk] =...
num rules:
num variables: The number of variables in the rule. For example, for the request "walk and x1".
num variables
derivation level: The number compositions used to build an example. For example, for the request "walk and walk". derivation level is 2: it created by first composing [[x] and x²] =... with [walk]]= =.. to get [walk and x2]] .... followed by another composition with [[walk] =...
frac_explice wwt.eeeppiiittulless. bucket: The fraction of explicit rules among all distinct rules used to create an example. The fractions are bucketed for legibility: frac explicit rules bucket=0.5 includes all examples with fraction of explicit nules least 0.5 and less than 0.6.
43

---

Under review as a conference paper at ICLR 2023
input wew.leengghhceeeeee bucket: The length of the input (context request) in tokens. The lengths bucketed for legibility: input lenghh bucket=5000 includes all examples with length least 500 and less than 600.
context num explicit rules: The number of rules explicitly asserted in the context. Every context is based on 14 grammar nules, so for example, context_num_exppee cirrec__umm_epppciiinleesss means that the context contains explicit assertions of 5 of these rules, while the other 9 rules are either illustrated indirectly via examples (such that the learner is expected to induce the rule to be true) not illustrated sufficiently (such that the learner is expected to consider the rule to be either false or unknown).
In all cases the accuracy appears negatively correlated num rules and derivation See text below each figure for additional observations.
N FINE-GRAINED EVALUATION METRICS
In addition to exact match accuracy, for more nuanced error analysis, we track several additional finer- grained metries, including partial accuracy metrics, edit distance, and counts implications and con- tradictions related to the consistency metric.
N.1 PARTIAL ACCURACY METRICS
Sequence level accuracy is a hard metric where a single wrong token leads the entire prediction to be labeled wrong. For this reason define additional accuracy measures that take into account partial success in solving the actual task. These measures are:
Reply Accuracy: A prediction is considered correct the reply portion is correct.
Qualifier Accuracy: A prediction is considered correct if the qualifier portion is correct.
Pattern accuracy assigns each token incremental ID based on the order it appears in the sequence, thus ignoring the specific predicted tokens and focusing on token variation pattern. E.g. the sequence JUMP JUMP RUN JUMP and WALK WALK EAT WALK both have the same pattern of A A B A where A replaces JUMP and WALK in the first and second sequences respectively, while replaces RUN and EAT in the first and second sequence respectively.
Pattern Accuracy:
Naive Accuracy: A prediction is considered correct if it produces the same set of unique tokens the target regardless of the order or the count. E.g. the sequences JUMP JUMP RUN JUMP and RUN JUMP both have the same unique tokens set (JUMP and RUN).
Token Accuracy: Token- wise accuracy between the prediction and the target. The two sequences are aligned the start token, and the shorter sequence is padded to have the same length the longer sequence such that the padded tokens are considered wrong predictions.
Tables 17, 18, 20, and 20 show performance of each baseline on all of these metrics.
44

---

Under review as a conference paper at ICLR 2023
1.000
0.95
0.975
0.90
0.8
0.950
0.85
0.6
acc
0.925
0.80
0.900
0.75
0.4
0.875
request_typp type rule non-rule
0.2
0.65
0.850
0.60
0.0
2.5
5.0
7.5
0
2
10
num_rules
jum_variables
derivation_ level
1.00
0.98
0.98
0.96
0.96
0.96
0.94
0.94
0.94
acc
0.92
0.92
0.92
0.90
0.90
0.90
0.88
0.86
0.5
10
500
1000
10
frac_ explicitppplleeeeeee bucket bucket
input_ length bucket
context Beteett_num_eeee huw_eppliiit_tiicm rules
Figure 10: T5 accuracy eSCAN-B examples. For non-rule examples, the accuracy appears positively correlated with the fraction explicit rules (bottom-left).
10
0.75
1.0
0.9
0.70
0.8
0.8
0.6
0.7
0.65
0.6
0.60
0.5
request_typp
0.55
0.4
non-rule
0.0
10.0
num_rule run_r
num num_ variables
derivation_ Jevel
0.9
0.8
W
0.8
acc
acc
0.6
0.4
0.5
0.0
1.0
1000
2000
10 context num htt_nnuliiiieitttteeeoeee rules
frac cis__eppiiiit_uuee rules bucket
input_ length bucket
Figure 11: T5 accuracy on cSCAN-X examples. For non-rule examples, the accuracy appears positively comelated with the number of explicit examples in the context (bottom-right). and negatively correlated with the input length (bottom-center).
45

---

Under review as a conference paper at ICLR 2023
10
eequest_type rule
10
0.8
0.8
non-rule
0.7
0.8
0.6
acc
0.4
0.4
0.2
0.2
0.3
0.0
0.2
0.0
num_rules
num num_variaan variables
derivation Levee
1.0
0.8
0.8
0.8
0.6
0.6
0.6
acc
0.4
0.2
0.0
0.0
1.0
1000
1500
10 context contextnumm__ pat__num_eee _ariieumimiiepiiitteeeeeee-e rules
5
_oc_exppicit_nee rules_ bucket
npputlength_Baee bucket
Figure 12: T5 accuracy on cSCAN-B MCD examples.
10
100
M
0.8
0.8
0.8
0.7
0.6
acc
request_tyy _tpp rule
0.2
0.5
0.0
0.0
2.5
5.0
7.5
10.0
num_rules
num_ variables
derivation_lewe
1.0
0.9
0.8
0.8
0.8
0.6
0.7
0.7
0.4
0.6
0.6
0.5
0.2
0.4
0.3
0.0
05
10
1000
2000
10
12
14
frac_ exwlicipppcctmm popllc bucket
length
bucket bucket
context context_num__m hauu__eplicitrruue rules
T5 accuracy on cSCAN-X MCD examples. For both rule and non-rule examples, the accuracy appears negatively correlated with the input length (bottom-center).
Figure 13:
46

---

Under review as a conference paper at ICLR 2023
Table 17: Partial accuracy measures on cSCAN-B Random
Partial Accuracy
Baseline
Pretrain
Size
Accuracy
Reply
Qualifier
Pattern
Naive
Token
T5
True
S
92.5
93.2
99.2
98.2
93.2
96.9
T5
True
B
92.6
93.3
99.1
98.4
93.3
97.2
T5
True
L
96.5
97.1
99.2
99.1
96.8
98.5
TS
False
S
19.3
28.4
61.1
51.8
38.9
27.5
T5
False
B
17.9
27.9
55.0
51.7
18.5
17.7
LongT5
False
S
19.5
28.4
62.2
51.9
33.2
26.7
Long'T5
False
B
21.4
29.9
65.6
52.7
39.2
30.7
LongTS-TGlobal
False
S
20.0
28.7
62.9
51.8
43.2
29.5
LongTS-TGlobal
False
B
17.2
27.2
54.5
51.5
18.0
16.7
T5 w/o Context
True B
26.8
43.8
54.8
53.3
26.9
15.5
Table 18: Partial accuracy measures on cSCAN-X Random
Partial Accuracy
Baseline
Pretrain
Size
Accuracy
Reply
Qualifier
Pattern
Naive
Token
T5
True
S
68.3
70.1
96.4
88.5
77.2
85.7
T5
True
B
71.1
72.7
96.4
88.1
79.9
85.0
TS
True
I.
81.6
83.6
97.1
91.2
88.5
90.2
T5
False
S
16.8
26.8
53.0
51.0
20.1
10.5
T5
False
B
17.2
27.1
52.9
51.1
19.8
8.6
LongT5
False
S
18.1
27.7
55.8
51.3
22.9
11.4
LongT5
False
B
16.7
26.6
53.8
51.1
20.0
9.4
ToggT5--TGoball
False
S
18.0
27.6
55.6
51.4
21.9
11.1
LongTS-TGlobal
False
B
18.8
28.2
54.8
51.3
22.0
8.8
T5 w/o Context
True
B
23.8
38.7
52.0
52.1
23.9
4.4
Table 19; Partial accuracy measures on eSCAN-B MCD
Partial Accuracy
Baseline
Pretrain
Size
Accuracy
Reply
Qualifier
Pattern
Naive
Token
T5
True
S
53.8
62.0
77.4
73.8
67.4
47.0
T5
True
B
54.7
62.9
78.4
76.0
67.2
51.0
T5
True
L
67.6
76.7
83.2
82.7
76.9
65.4
T5
False
S
40.1
47.7
68.4
67.4
50.4
31.8
T5
False
B
38.4
45.7
64.7
66.0
39.2
24.7
LongTS
False
S
40.7
47.6
68.5
68.2
48.5
31.0
LongTS
False
B
38.2
45.2
67.5
66.8
45.2
29.6
LongT5-TGHE TGlobal
False
S
40.6
48.0
68.0
67.6
52.1
33.6
LongTS-TGlobal
False
B
39.3
45.9
66.9
68.0
45.6
30.2
T5 w/o Context
True B
46.2
60.0
65.2
69.4
46.2
22.9
47

---

Under review as a conference paper at ICLR 2023
Table 20: Partial accuracy measures on eSCAN-X MCD
Partial Accuracy
Baseline
Pretrain
Size
Accuracy
Reply
Qualifier
Pattern
Naive
Token
T5
True S
70.7
72.8
95.5
90.1
78.3
83.2
T5
True
B
69.2
71.0
95.8
89.8
76.9
81.4
T5
True
I.
76.8
78.4
97.0
92.2
82.9
87.2
T5
False
S
41.6
48.7
69.5
69.7
51.4
18.7
T5
False
B
40.0
47.6
66.5
69.0
42.4
12.3
LongT5
False
S
40.9
48.2
68.1
69.5
49.9
17.9
LongT5
False
B
41.2
48.5
68.0
69.7
46.1
14.3
LongT5--TGlball
False
S
41.0
48.4
67.7
69.5
49.2
18.0
LongTS-TGlobe
False
B
40.9
48.1
67.8
69.5
44.8
12.5
T5 w/o Context
True
B
42.7
52.3
64.4
70.3
42.8
6.2
48

---

Under review as a conference paper at ICLR 2023
Table 21: Consistency breakdown for cSCAN-B Random
Baseline
Pretrain
Size
Consistency
Implications
Contradictions
T5
True
71.8
431
169
T5
True
B
71.3
437
176
T5
True
87.5
446
64
T5
False
S
0.0
0
881
T5
False
0.0
0
2107
LongT55
False
S
0.0
0
1419
LongT5
False
689
LonoTS-TGlobal
False
0.2
3
1269
LongTS-TGlobal
False
0.0
3128
T5 w/o Context
True
B
0.1
1
774
Table 22: Consistency breakdown for cSCAN-X Random
Baseline
Pretrain
Size
Consistency
Implications
Contradictions
T5
True
S
30.1
237
550
T5
True
32.3
268
561
T5
True
43.5
335
435
T5
False
S
0.1
4
3967
T5
False
0.0
2
5375
LongT5
False
S
0.1
2
1791
LongT5
False
0.1
4
3678
TonoTS-TGlobal
False
S
0.2
3
1651
LongTS-TGlobal
False
3002
T5 w/o Context
True B
0.4
3
822
N.2 CONSISTENCY BREAKDOWN BY IMPLICATIONS AND CONTRADICTIONS
In this section show the number of implications and contradictions used to calculate the consistency metric. Tables 21 and 22 show this breakdown.
To further illustrate the type of inconsistencies the model is making, show the consistency sets (implications and contradictions) size distribution in Figures 14 and 15. This shows that sets of size where one prediction implies or contradicts another are the most common. Appendix P.2 expands on this by providing examples of the contradictions.
As be seen in these tables, the consistency metric for each cSCAN Random experiment is calculated based on minimum of 500 implications and contradictions.
As discussed in Section 6.2 and Appendix we do report consistency metrics for the MCD datasets, as we are only able to achieve high enough density of potential implications in the Random datasets, where after splitting, we augment each context with additional top-level examples from the same distribution. We do not perform this additional example generation step for the MCD datasets to avoid impacting the compound divergences between the train and test sets.
49

---

Under review as a conference paper at ICLR 2023
T5-Small
T3-Base
TS-Large
Contradictions
Contredictions
400
Contradicions
Implications
900
300
230
200
200
100
Non-pretrained T5-Small
Non-pretrained T5-Base
2000
Contradictiions
Contredictions
630
1500
3 1000
230
500
2
Set size
Set $ize
Non-p con-pretrained LongTs-Smme Small
Non-pretrained Long TS-Bae Base
Contracictions
600
Cortredictions
Implications
1030
s5tts
400
530
Set size
Set 9/20
Non- LoneTS-TCllbbl--Sall
Non-pretrained LongTS-TClebal-Base
Contradictions Implications
Cortredictions Implications
1000
Setts
750
1500
g
530
1000
500
Set size
Set Size
Figure 14: Consistency sets size distribution for cSCAN-B Random
50

---

Under review as a conference paper at ICLR 2023
T5-Small
T3-Base
TS-Large
Contradictions
Contredictions
Contradicions
Implications
400
Sals
200
230
200
Non-pretrained T5-Small
Non-pretrained T5-Base
Contradicttions
Cortredictions
3000
Implications
1030
Set size
Set $ize
Non- con-pretrained LongTs-Smi Small
Non-pretrained Long TS-Bae Base
15.00
Contradictions
3000
Cortredictions
Implications
sets
1030
5.0
1000
size
Set 9/z0
pretrained LomeTS-TCllbbal-Saal
Non-pretrained LongTS-TTClebu-Bun
1300
Contradictions Implications
Contredictions Implications
2000
-eets
1000
1000
Set size
Set S/20
Figure 15: Consistency sets size distribution for cSCAN-X Random
51

---

Under review as a conference paper at ICLR 2023
Table 23: Edit distance measures for cSCAN-B Random
Baseline
Pretrain
Size
Distance
Substitutions
Insertions
Deletions
T5
True S
0.19
0.07
0.07
0.05
T5
True
B
0.17
0.07
0.03
0.07
T5
True
L
0.09
0.03
0.01
0.04
T5
False
S
4.62
1.96
1.30
1.35
T5
False
B
6.00
3,22
1.30
1.48
Long
False
S
4.75
2.18
1.23
1.34
LongTSs
False
B
4.44
1,93
1.09
1,42
LongTS-TGlobal
False
S
4.30
1,86
1.18
1,26
LongTS-TGlobal
False B
6.39
3.23
1.29
1.87
T5 w/o Context
True B
6.19
2.89
2.22
1.08
Table 24: Edit distance measures for cSCAN-X Random
Baseline
Pretrain
Size
Distance
Substitutions
Insertions
Deletions
T5
True S
3.01
0.27
1.40
1.33
T5
True B
3.21
0.30
0.65
2.26
T5
True
I.
2.04
0.15
0.57
1.32
T5
False
S
37.87
12.41
5.77
19.69
T5
False
B
48.64
12.73
5.18
30.73
LongT5
False
S
38.81
11.65
5.30
21.87
LongT5
False
B
46.49
13.13
4.21
29.15
LongTS-TGGobal
False
S
36.74
11.59
6.11
19.04
LongT5-TGGlbal
False B
46.59
12.54
5.23
28.82
T5 w/o Context
True B
44.81
14.17
8.37
22.27
Table 25: Edit distance measures for eSCAN-B MCD
Baseline
Pretrain
Size
Distance
Substitutions
Insertions
Deletions
T5
Truc
S
1.96
0,69
0.82
0.44
T5
True B
1.72
0.60
0.90
0.22
T5
True
L
1.23
0.42
0.63
0.18
T5
False
S
3,41
1.26
1.28
0.86
T5
False
B
4.29
2.12
1.07
1.10
LongT5
False
S
3.39
1.32
1.47
0.60
LongT5
False
B
3.64
1.54
1.23
0.87
LongTS- T-Globall
False
S
3.18
1.29
1.15
0.74
LongTS-TTGlob
False
3.50
1.28
1.47
0.75
T5 w/o Context
True
B
4.46
1.90
1.77
0.79
N.3 EDIT DISTANCE
Edit distance is the number of edits that would need to be applied to the predicted sequence transform it to the target sequence. There are three types of edits: Substitutions (S), Insertions (I) and Deletions (D).
Tables 23, 24, 26, and 26 show the edit distance and constituent metrics of each baseline all of these metrics.
52

---

Under review as a conference paper at ICLR 2023
Table 26: Edit distance measures for cSCAN-X MCD
Baseline
Pretrain
Size
Distance
Substitutions
Insertions
Deletions
TS
True
S
2.60
0.26
1.63
0.70
T5
True B
2.90
0.33
0.79
1.78
T5
True
L
1.96
0.22
0.65
1.09
T5
False
24.02
5.06
4.30
14.66
T5
False
B
26.84
7.22
5.50
14.13
LongT5
False
24.31
5.58
3.91
14.83
LongT5
False
B
29.05
7,02
3.76
18.26
LongT5- TGlobal
False
S
24.21
5.37
4.37
14.47
TonoTS--TGlobal
False
B
29.79
7.90
3.86
18.03
T5 w/o Context
True
B
30.25
9.72
6.49
14.04
53

---

Under review as a conference paper at ICLR 2023
O INPUT-OUTPUT FORMAT
We evaluate T5 variants, which are all encoder-dccoder architectures, Here, we show how we prepare the input (that is fed to the encoder) and output (that the decoder generates) of our models, given an example from cSCAN. The example below is selected from the cSCAN-B dataset. Note that the request and the context examples are all concatenated into single bullet-points here (and in Appendix P.I) are just added to improve the readability of the examples.
string form the input, and the
Input: <Request, Context>
turn opposi te 1eft jump around left twi ce
[[eff]] PURPLE
[xl around x2] - [x2] [x2] ) [x1] [x1]
[xl opposite x2] [x1] [x2] [x2]
[[urr]] YELLOW
[x] after x2] [x2] [x]]
[walk] LCURN
[[1mm]] RUN
+run around left and turn opposite right twice PURPLE PURFLE JUMP JUMP YELLOK
RTURN RTURN YELLOW RTURN RTURN YELLOW RTURN RTURN
wwak opposite left twice after run thrice JUMP RON LTURN EURPLE PURPLE LTURN PUR-
PLE PURPLE
* [run opposite left] JUMP PURPLE 2URPLE
[run oppos te x1 twice] JUMP [x1] [x1]
*turn opposi te right txice and turn around
thrice YRI.OW RTURN RTURN YEITOW
RTURN RTURN YETTOW RTURN RTURN PURPLE PURPER VRI.OO YEL.O.
-run right thrice JUMP RTURN RIURN
[x1 around x2 thrice] [x2] [x2] [x1] [x1]
[look around tarice] [xi] [x1] PINK PINK
look around right thrice run opposite left RTORN RTURN PINK PINK JUMP PURPLE
PORPLE
looo around left thrics PURPLE PURPLE PINK PINK
[look left thrice] FINK PURPLE PURPLE
[look opposite x1] FINK [xi] [x1]
run opposite right twice and turn around left twice JUMP RTURN RTURN PURPLE PUR-
PLE YELLOW YELLOW
sturn around left and walk twice PURPLE PURPLE YELLOW YELLOK LTURN LTURN JUMP
[xl and thrice] [x1] [x2]
[xl thrice and x2] [x1]
[x2]
.loo thrice and ron around right thrice PTNK RTURN RTURN JUMP JUMP
sturn opposite right YELLOW RTURN STURN
[jump right] RUN RTURN RTURN
[x1 zight] [x1] RTURN RTURN
rture left after look opposite right PINK RTURN RTURN YELLOW PURPLE PURPLE
1ook right twice YELLOW PINK RTORN RTURN PINK RTURN RTURN
[look x1] PINK [x1] [x1]
[x] left] [x1] PURPLE PURPLE
walk around left twice afterz walk around left thrice PURFLE PURPLE LTURN LIURN
PURPLE PURPLE LTURN LTURN PURPLE PURPLE L.TURN LTURN YELLOW
jump right twice RON RTURN RTURN RUN RTURN RTURN RTURN
[look around x1 twice] RIN [x]] [x1] PTNK PTNK [x]] [x]] PTNK PTNK
[jump around left twice] PINK PURPLE PURPLE
PURFLE PURPLE RUN
Output:
<Reply, Qualifier>
YELLOW PURPLE PURPLE FINK PURPLE PURPLE RUN RUN PURPLE PURPLE RUN RUN (Reason-
ing: Defeasible)
Note that we concatenate request context, rather than context + request, so as to make the system more robust truncation of the example string, if any example were to exceed the maximum length of T5's input buffer (although in our experiments we made sure that the example lengths did not exceed this buffer size).
In representing the context for T5, it can be noted that omitted special syntactic tokens such braces, angle brackets, and commas wherever possible, so as to reduce the token count and keep the format closer to natural language, to the extent that this could be done without introducing ambiguity. For context examples
54

---

Under review as a conference paper at ICLR 2023
that represent rule assertions, we also adopted simplified syntax, similar to the shorthand described in Appendix C.2, consisting a single line containing the rule request alone, while omitting the reply, since in cSCAN only include positive rule assertions in the context (i.c., never rule examples with reply of 0 or ?).
Note also that we do not perform any clustering of top-level examples by their context, but rather represent each top-level example in flattened form shown here (with its context included). We then shuffle the full set of top-level examples before batching them for input into T5. This means that even when there may be 100 more top-level examples with the same context, T5 will in general not them all in sequence or in the same batch, but rather intermixed with top-level examples with different contexts.
P QUALITATIVE ERROR ANALYSIS
P.I ACCURACY ERROR ANALYSIS
Here, we showcase examples where our best model (T5-Large with pretraining) fails at producing the accurate results when evaluated on an example from the cSCAN-B. We show different cases where the target and the prediction from the model are different.
Example of LS-Large failing at rule assertion (when the requesi is False).
Inout
[run left after turn right] LOOK WALK
WHITE WHITE PINK LOOK WALK
[left] = PINK
[turn] = LOCK
[x1 twice] [x1] [x1]
[x1 thrice] [x1]
[x1 and x2] [x1] [x2]
[x1]
[[ooo]] = BLUE
[xx] opposite x2]
[x2] [x]] [x1] [x2]
[xx]
[x1] [x2]
[[wak]] RTURN
* [xx] after x2] [x2] [x1] [x]] [x2]
[jump] = GREEN
[rright] WALK
[[run] MHITE
wwakk around right and turn around left thrice RTURN WALK RTURN LOOK PINK
LOOK RTURN WALK RTURN
dump opposite right after look around left PINK BLUE PINK WALK GREEN
GREEN WALK WALK GREEN GREEN WALK 2INK BLUE PINK
[jump around left] GREEN GREEN
* [x1 around right twice] WALK [x1] WALK WALK [x1] WALK
target
0 (Reasoning: Monotonic)
prediction
1 (Reasoning:
Monotonic)
Commentary
Since [x1 x2] [x1] [x2), [zun left) and [turn right] would map. to NHITE
PINK and LOOK NALK respectively, and with [x1 after x2) [x2] (x1] [x1]
[x2], the correct output should be LOOK WALK WHITE PINK WHITE PINK LOOK MALK.
55

---

Under review as a conference paper at ICLR 2023
Examplc T5-Large failing rule assertion (when the request Thue).
Input
[run and run around right thrice]
PURPLE
[jump]
[x1 and x2] [x2] [x1] [x2]
[left] = TOCK
[run] FURPLE
[x1
[x2] [x2] [x]]
[xx around x2] [x1] [x2] [x1]
[[loo]] = RUN
[x] after x2] [x2] [x2] [x]]
[[uur]] JUMP
[[wakk] LIURN
* [x1 twice] (x1] [x1]
wwak opposite right GREEN LTURN
llook oppesite right twice and jump twice RED RED GREEN RUN GREEN RUN RED
[turn opposite right] GREEN JUMP
[look opposite x1] [x1] RUN
+ 1ook right and jump around right thrice RED GREEN RED RED GREEN RED GREEN
GRAEN RUN RPD GREEN RED RED GREEN RED
turrn opposite 1eft
and
[xl opposite right thricel GREEN [x1] SREEN [x1]
+ [run around x1 thrice] EMPTY_STRING
run around right after run epposite right twice GREEN FURPLE GREEN PURFLE
GREEN PURPLE GREEN PURPLE PURPLE GREEN PURPLE
dummp opposite left after turn opposite right twice GREEN JUMP GREEN
GREEN JUMP GREEN JUMP LOOK RED
+ [run around right] PURPLE GREEN PURFLE
[xl opposite right] GREEN [x1]
target
(Reasoning: Monotonic)
prediction
0 (Reasoning: Monotonic)
Commentazy
Since run around x1 thrice maps to empty string and [x] and x2] [x2]
[x1] [x2] with [run] PURPLE, the providad rule is True.
56

---

Under review as a conference paper at ICLR 2023
Example of LS-Laage failing al acknowledging lack information to reply.
Ingut
tuzn around right thrice and jump oppoaite
thrice
[x1 and x2] [x1] [x2] [x]]
[x] around x2] [x]] [x2] [x2]
[x] opposite x2] [x]] [x2] [x2]
[x] twice] [x1]
* [x1 after x2] [x2]
[x1] [x2]
[left] - BLJE
[x1 chrice] [x1] [x1]
[[look] PINK
[[unn] RED
-walk opposite right after run opposite left twice RED BLUE BLUE WHITE YEL-
T.W YEL.OW RED BLOE BLUE
-walk opposite 1eft twice and WHITE, BLUE BLUE RED WETTE BLUE BLUB
[walk opposite x1] WHITE [x1] [x1]
[walk xi] [x1] WHITE
Cump opposite left thrice and turn left thrice RTURN BLUE BLUE RTORN BLUE
BLUB BLUE RUN BLUE RUN RTURN BLUE BLUE RTURN BLUE BLUE
+turn opposite right twice after look opposite right PINK YELLOW YELLOW
YELLOW YELLOW PINK YELLOW YELLOW
[turn left] = BLUE RUN
[turn around x1 twice] RUN [x1] [x1]
ummp left twice after turn around left RUN BLUE BLUE BLUE LTURN RUN BLUE
BL.EE
Cump around right thrice WALK WHTTR YELLOW YETTON WALK WHTTR YELLOR YRT.-
LOW
[[uump right] YELLOW WALK WALK
+[jump opposite left] YELLON BLUE BLUE
wwak right twice and walk around
thrice YELLOM
BLUE BLUE
WHITE BLUE BLOB YELLOW WHITE
<turn around right twice after jump around left WALK WALK BLUE BLUB SUN
YELLOW YELLOW WALK WALK BLUE BLUE
[jump opposite right]
WALK YELLOW YELLOW
(turn oppesite right] RUN YELLOW YELLOW
jump right twice and jump thrice YELLOW BLACK BLACK BLACK YELLOW BLACK run right thrice after look around left twice PINK BLUE BLUE YELLOW RED
YELLOW RED PINK BLUE BLUE
[x1 left]
[x1]
* [run x1] = [x]] RED
target
?
(Reasoning: Defeasible!
prediction
RUN YELLOW YELLOW RUN YELLON YELLOW YELLOW BLUE BLUB YELLCW BLUE BLUE RUN YEL1OW YELLOW RUN YELLON YELLOW (Reasoning: Defeasible)
Commentary
[jump] = is not well illvatrated by at least 2 unique sabstitutions, therefore the mapping of [jur]] is unknown.
57

---

Under review as a conference paper at ICLR 2023
Example of Ls-Lagee failing drawing information compose the reply.
Inout
opposite left after burn around left thrice
[x] and x2] [x1] [x2]
[jump] RTURN
[x1 opposite x2] [x1] [x2] [x2] [x1]
[left] - YELLON
[x1 twice] [x]]
[[urr]] RED
[right]] WHITE
[x] chrice] [x1] [x1]
[walk] GRRRN
[x] after x2] [x]] [x1] [x2] [x2]
[run] JUMP
* [x1 around x2] [x2] [x1] [x2] [x1]
-loo around left and walk twice YELLOW LOOK YELLOW LOOK GREEN
look opposite left thrice WALK WALK YELLOW YELLOR WALK WALK WALK WALK YEL-
LOW YELLOW WALK WALK
* [look opposite left thrice] = WALK WALK YELLOW YELLOW WALK WALK WALK WALK
YELLOW YELLOW WALK WALK
[look right] WHITE WHITE RTURN RTURN
*walk right twice WHITE WHETE GREEN GREEN
*run left thrice and run YELLON YELLOW JUMP JUMP YELLOW YELLON JUMP JUMP
JUMP
[x1 left] YELLOW YELLON [x1] [x1] [turn x1] [x1] [x1] RED REC
target
YELLOW YELLOW RED YELLON YELLOW YELLOW YELLON YELLOW
RED YELLOW RED YELLOW RED YELLON RED YELLOW RED YELLOW RED (Reasoning: Monotonic)
prediction
RED YELSOW YELLOW RED RED VELION YELLOW RED YELLON RED YELICK RED YELLOW
RED YELLOW RED YELLOW RED YELLON RED YELLOW RED (Reasoning: Monctonic)
Commentary ry
Composiin ng [x] around x2] = [x2] [x1] [x2] [x]] into [x] thrice] = [x1]
[x1] gives [x] around x2 thrice] [x2] [x1] [x2] [x]] [x2] [x]] [x2]
[x1]
-Composing the above rule
oppoaite
x1 x2 into and
x2 [x] after x21 [x1] [x1] [x2] [x2] respectively reaults in the zule
[x1 opposite x2 after x3 around x4 thrice] [x1] [x2] [x2] [x1] [x1] [x2)
[x2] [x]] [x4] [x3] [x4] [x3] [x4] [x3] [x4] [x3] [x4] [x3] [x4] [x3] [x4]
[x3] [x4] [x3].
SSbbstttuting xl and x3 by [turn] RED and and x4 by [left] YELLOW
results in the sule [turn opposite left after turn around left thrice]
RED YELLOW YELLOW RED RED YELLON YELLOW RED YELLON RED YELLOW RED YELLOM
RED YELLOW RED YELLOW RED YELLON RED YELLOW RED YELLOW RED.
Which varies from the predicted reply by the underlined tokens.
58

---

Under review as a conference paper at ICLR 2023
Example T5-Large failing at inferring the type of reasoning, while reply is correct.
Ingut
[x] around left and x2 opposite right twice] LOOK LOOK [x2] [x]]
[x1]
GREEN [x1] [x1] BLUE LOCK LOOK [x2]
[x] after x2] [x1] [x2] [x1]
[x] thrice] = [x]] [x]]
around x2] [x]] [x]] [x2]
[waik] LTORN
[xight]] LOCK
[left] GREEN
[x] epposite x2] [x2] [x2] [x1]
[look] NALK
[x] twicel [x1]
-look twice and jump right WALK BLUE LOOK WALK WALK WALK BLUE LOOK
surn twice and walk left GREEN LTTUN RUN RUN GREEN L.TURN
[jump 1eft] BLUE BLUP GREEN
[run 1eft] GRRAN BTACK
run around left twice BLACK BLACK GREEN
zurn around left and run opposite right LOCK LOOK BLACK RUN RUN GREEN RUN
RUN GREEN LOOK LOOK BLACK
[run around x1 thrice] BLACK BLACK [x1] BLACK BLACK [x1]
Irun opposite x1] [x1] [x1] BLACK
walk right thrice after turn opposite left thrice LTURN LTURN LOOK LTURN
LTURX LCCK GREEN GREEN RUN GREEN GREEN RUN LTURN LTURN LOOK LTURN LTURN LOCK
curnn around right twice RJN RON LOOK
[turn around x1]
[x1]
[turn opposite left] GREEN GREEN RUN
run opposite left thrice and look WALK GREEN GREEN BLACK GREEN GREEN BLACK
GREEN GREEN BLACK GREEN GREEN BLACK WALK
around right and Toak around right twi oe WALK WALK TOOK BLACK BLACK TOOK
BLACK BrACK LOOK
WATK
[x] and x2 thrice] = [x2] [x2] [x]] [x]] [x2] [x2]
[x1 and x2 opposite x3] [x3] [x3] [x2] [x1] [x1] [x3] [x3] [x2]
jump opposite right thrice after walk around left LOOK LOCK BLUE LOOK LOOK
BLUE LTURN LTURN GREEN LOCK LOOK
LOOK LOOX BLUE
jump around right thrice and turn thrice RON RON BLUE BLUE LOOK BLUB BLUE
LOOK BLUE 3LUB LOOK BLUE BLUB LOOK RUN RUN
[jump opposite right] LOOK LOOK BLUE
[uup opposite x1 twice] - [x1] [x1] BLUE
target
(Reasoning: Defeasible)
prediction
(Reasoning:
Monotonic)
Commentary
The rule (x1 and x2) [x2] [x1] [x1] [x2] is hidden and be induced
from the context examples highlighted by underline.
.Compssing [x1 opposite x2] [x2] [x2) [x1] into [x] twice] [x1] re-
aults into the rule [x] opposite x2 twice] = [x2] [x2] [x]]
.Composing [x] around x2] [x1] [x]] [x2] and [x] opposite x2 twice] =
[x2] [x2] [xl] into xi and x2 of [x] and x2] [x2] [x1] [x1] [x2] respec-
tively results in the rule [x] around x2 and opposite x4 twice] [x4]
[x4] [x3] [x1] [x]] [x2] [x1] [x1] [x2] [x4] [x4] [x3]
SSubstitutff ing x2 and xá by [1eft] GREEN and [right] = 1OOK respectivel ivll
results in the rule [x] around left and x2 opposite right twice] LOOK
LOOK [x3] [x1] [x1] GREEN [x1] [x1] GREEN LOOK LOOK [x3].
59

---

Under review as a conference paper at ICLR 2023
P.2 CONSISTENCY ERROR ANALYSIS
All the above examples show inaccurate output, where there is mismatch between the expected target and model prediction. However, as discussed in Section 4, regardless of being accurate, model may fail to stay consistent in replying to different requests. In this section we sample contradictions made by La-Large on eSCAN-X Random and categorize the type of contradictions the model makes. Note that consisteney independent of the context and therefore it's omitted when presenting the examples.
As seen Appendix N.2, the models produce contradictory sets at different sizes. We find it easy to analyse each size independently as the type of mistakes vary between them.
Contradictions of size 2 At this level, the inconsistencies are 1:1 relationships between two contradictory predictions P1 and P2. There are two types at this level
Type I: Both Pl and P2 are asserted rules with matching right-hand sides and and different right-hand sides.
Type One prediction is an asserted rule with no variables and the other is rule application prediction with request matching the right-hand side of the rule request while the reply does not match the rule request's 's right-hand side.
From a sample of 20 contradictory sets of size 2, 11 sets were of type I while 9 were found to be of type 2. Here are examples of both types
Contradctory set of size 2:
Type
Request:
[drive left framing x2] [x]] [x2] [x]]] [x]] [x2] [x2] [x1] [x2]
[x]] [x]] [x2] [x2] DTURN 1O0K DITON DTURN TOOK 1.00% DTURN TOOK DITOR:
DITRR TOOK T.0KK [x]] ] [x2] [x1] [x]] [x2] [x2] [x]] [x2] [x]] [x]] [x2]
[x2] [x1] [x2] [x1] [x1] [x2] [x2] [x1] [x2] [x1] [x1] [x2] [x2]
Reply 1
1 (Reasoning ng: Monotonic)
Request 2
[drive left 5x framing xi x21 [x1] [x2] [x1] [x1] [x2] [x2] [x1] [x1]
[x1] [x]] [x2] [x2] DTURN 100K DTURN DTURN LOOK LOOK DTURN LOOK DTORN
DTURN LOOK LOOK [x]] [x2] [x21 [x1] [x2] [x2] [x1] [x2] [x]] [x1] [x2]
[x2] [x1] [x2] [x1] [x]] [x2] [x2] [x]] [x2] [x1] [x]] [x2] [x2]
Reply 2
1 (Reasoning: Monotonic)
Commentary
The two right-hand sides are different in the underlined takens.
Contradictory set of size 2: Type
Request 1
[run zigzag left after run fast around left] PEEK PEEK PEEK PEEK PEEK
PEEK PEEK PEEK PEEK PEEK FEEK PEEK PEEK PEEK PEEK PEEK FEEK FEEK RUN PEEK
PESK PEEK PEEK PEEK PEEK PEEK RUN RUN RUN PEEK RUN RUN RUN PEEK RUN RUN
RON PEEK PEEK PEEK PEEK PEEK PEEK PEEK PEEK PEEK PEEK PEEK PEEK PEEK PEEK
PEEK PEEK PEEK RUN PEEK PEEK PEEX PEEK PEEK PEEK
Reply 1
1 (Reasoning: Defeasible)
Request: 2
zigzag 1eft after run fast around 1eft
Reply 2
PRAK PERK PREK PEEK PEEK PREK PRAK PREK PERK PEEK PREK PEEK PREK PREK PREK
PERK PEEK PEEK RUN PEEK PERK PEEK PEEK PEEK PEEK PREK RON RUN RUN PEEK RON
RON PEEK RJN RUN PEEK PEEK PEEK PEEK PEEK PESK PEEK
PEEK PEEK
PEEK PEEK PEEK PEEK PEEK PEEK PEEK PEEK RUN PEEK PEEK PEEK PEEK PEEK PEEK (Reasoning: Defeasible)
Commentary
Reply 2 is different from the right-hand side of Request by one addi- tional token (highlighted with an underline).
Contradictions of size 3 At this level, the inconsistencies are 2:1 relationships with least two nule assertions.
Type 1: The composition of two rules directly contradicts third rule.
60

---

Under review as a conference paper at ICLR 2023
Type 2: The composition of two rules forms rule with no variables where it's right-hand side matches the request of a rule application prediction and it's right-hand side contradicts the reply of that prediction.
From sample of 20 contradictory sets of size 3, both Type and Type 2 had 10 sets each. Here are examples of both types
Contradictory set of size 3: Type
Request
[x] cauticusly x2]
[x]]
[x1]
[x]]
[x2]
[x2] [x2] [x]]
[x]] [x]]
[x2]
[x2]
Reply
(Reasoning: Monotonic)
Request
[walk] RIDE RIDE
Reply 2
1 (Reasoning: Monotonic)
Request 3 [walk cautiously xi] RIDE RIDE RIDE
[x1]
[x1]
[x1]
RIDE RIDE RIDE [x1]
[x1]
Reply 3
1 (Reasoning: Monotonic!
Commentary
Composing Rule 2 into Rule would result in the rule
[walk caut fously x1] RIDE RTDE RIDE RTDE RTDE RIDE [x]] [xT] [x1]
RIDE RIDE RIDE RIDE RIDE RIDE [x1] [x1]
Which is different from Rule in places highlighted by an underline.
Contradictry set of size 3: Type 2
Request 1
[run opposite ap thrice following x1] LOOK LOCK RUN LOOK LOOK LOOK RUN LOOK LOOK LOCK RUN LOOK [x1] LOOK LOOK RUN LOOK [x1] LOOK LOOK RUN LOOK [x1]
Reply 1
1 (Reasoning: Monotonicl
Request 2
run opposite thrice following run zigzag
Reply 2
LOOK LOOK RON LOOK LOOK LOOK RON LOOK LOOK ZOOK RUN LOOK LOOK LOCK RUN LOOK LOOK LOOK LOOK RUN LOOK LOOK LOOK BUN LOOK LOOK LOOK RUN LOOK LOOK LOOK 1OOK RUN LOOK LOOK LOOK RON LOOK LOOK TOOK RUN LOOK LOOK (Reasonings Monotonicy c)
Request
[run zigzag] LOOK
Reply 3
1 (Reasoning: Monotonic!
Commentary
Composing Rule 2 into Rule would result in the rule
[run opposite thrice following zigzag] LOOK LOCK RUN LOOK LOOK LOOK LOOK LOOK LOOK LOOK LOOK дOOK LOOK LOOK LOOK LOCK LOOK RUN lOOK LOOK
which natches Request 2 in its right-hand side but is different from Reply by its right-hand side in places highlighted by underline.
Contradictions of size At this level there are more complex contradictory relations as the space of possible compositions grows. we categorise 3 types of contradictions
Type 1: Composing 3 rules to form rule with variables that contradicts rule application prediction.
Type The composition of subset set of rules contradicts the composition another set of rules.
Type 3: The composition of 3 rules directly contradicts another rule.
For this size, the model only made 10 contradictions: 5 were of type 2, 3 were of type and were of type 1. Here are examples for each type
61

---

Under review as a conference paper at ICLR 2023
Contradictory of 4: Type
Request 1
[run drunkenly down. RTURN PEEK
PEEK PEEK
PEEK
PECK
Reply 1
1 (Reasoning: Monotonic)
Request: 2
Cump drunkenly twice aftez run drunkenly down
Reply 2
RTDE RIDE RIDE RIDE RTDE RTDE RIDE RTDE RIDE RIDE RIDE, RIDE RIDE RIDE RIDE
RTDE RTDE RTDR RTDE RTDE RTDP RTDR RTDR RTDE RTDR RTDE RTDE RTDE RTOR
RTDR RTDE RTDR RTDE RTDE RTEP RTDS RTDR RTDE RTDE RIDE RTDE RTDP RTOR
RIDE RIDE RIDE
RIDE RIDE RIDE RIDE
RIDE
RIDE RIDE RIDE
RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE
RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE
RTURN PEEK PEEK PEEK PEEK PEEK (Reasoring: Monotonic)
Request 3
[x] twice after x2] [x1] [x1] [x1] [x1] [x1] [x1] [x1] [x1] [x1] [x1]
[x1] [x]] [x2]
Reply 3
1 (Reasoning: Monotonic]
Request 4
[jump drunkenly] = RIDE RIDE RIDE RIDE RIDE RIDE
Reply 4
(Reasoning:
Monotonic)
Commentaryy
Composing
into x2 rule and rule into x rule would re-
sult the rule
[Jump drunkenly twice after run drunkenly down] RIDE RIDE RIDE RIDE
RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE
RIDB RIDE RIDE RIDE RIDS RIDE RIDE RIDE RIDE RIDE RIDE RIDS RIDE RIDE RIDE
RIDE RIDE RIDE RIDE RIDS RIDE RIDE RIDE RIDE RIDE RIDE RIDS RIDE RIDE RIDE
RIDE RIDE RIDE RIDE RIDS RIDE RIDE RIDE RIDE RIDE RIDE RIDS RIDE RIDE RIDE
RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RTURN PEEK PEEK PEEK PEEK PEEK
Which natches Request 2 in its right-hand side but is different from Reply by its right-hand side places highlighted by underline.
62

---

Under review as a conference paper at ICLR 2023
Contradictory set of 4: Type
Request
cauticusly x2 5x] [xi]
[x1]
[x2] [x2]
[x2]
[x1]
[x]]
[x1]
[x2]
[x2] [x]] [x1] [x1] [x2]
[x2]
[x2]
[x1] [x]] [x1]
[x2]
[x2]
[X]
[x1]
[x1]
[x2] [x2] [X2] [x]] [x]] [X]]
[x2]
[x2] [x]]
[x]]
[x]]
[x2]
[X2]
[x2]
[x]]
[x]] [x]] [x2] [x2] [x]] [x1]
[x]]
[x2] [x2]
[x2]
[x]]
[x]] [X1
[x2]
[x2]
[x]] [x]] [x]] [x2] [x2] [x2] [x]]
[x1] [x2]
[x2]
Reply 1
1 (Reasoning: Monotonic)
Request: 2
[walk cautiously x1] REDE RIDE RIDE [x1] [x1] [x1] RIDE RIDE RIDE [x1]
[x1]
Reply 2 1 (Reasoning: ng: Monotonic)
Request 3 [x] 5x] [xi] [x1] [x1] [x1] [x1] [x1]
Reply
1 (Reasoning: Defeasible)
Request 4
[walk] RIDE
Reply
(Reasoning: Monotonic)
Commentary
Composing rule 4 into x1 of rule results in the rule
Rule [walk cautiously xi
RIDE RIDE RIDE RIDE RIDE RIDE
[x]] [x1] [x1] RIDE RIDS RIDE RIDE RIDE RIDE [x1] [x1] RIDS RIDE RIDE
RIG5 RIDE RIDE [x1] [x1] [xi] RIDE RIDE RIDE RIDE RIDE RIDE [xi] [x1]
RIDE RIDE RIDE RIDE RIDS RIDE [x1] [x]] [x]] RIDE RIDE RIDE RIDE RIDE RIDE
[x1] [x]] RIDE RIDE RIDS RIDE RIDE RIDE [xT] [xI] [x]] RIDS RIDE RIDE
RIBE RTDE RIDE [x1] [x1] RIDE
RIDE [x]] [x1] [x]]
RTDR RTDE RTDE RTDE RIDS [xT] [x]] RTDR RTDE RTDR RTDR RIDS RIDE [X]] [x]]
[x1] RIDE RIDE RIDE RIDE RIDE RIDE [x1] [x1]
And composing rule into rule results in the rale
*ule [walk cautiously
RIDE
RIDE
[x2] [xi]
RIDE RIDE [xl] [x1] RIDS RIDE RIDE [x1] [x]] [xl] RIDE RIDS RIDE [x1] [x]]
RIDE RIDE RIDE [x1] [x]] [xi] RIDE RIDE RIDE [x]] [x]] RIDS RIDE RIDE [x]]
[x1] [x]] RIDE RIDE RIDS [xi] [x1] RIDE RIDE RIDE [x]] [x]] [x]] RIDE RIDE
RIDE [xI] [xI] RIDE RIDE RIDE [x1] [x1] [x1] RIDE RIDE RIDE [xi] [x1]
Which directly contradicts rule
places highlighted by under-
line.
63

---

Under review as a conference paper at ICLR 2023
Contrudictory set of size 4: Type
Request 1
[x] between fly cautiously 90 x2] [x]] [xl] WALK WALK WALK WALK WALK
WALK [x2] WALX WALK WALK RALK WALK WALK WALK WALK [x2] [x]]
Reply 1
1 (Reasoning:
Monotonic)
Request 2
[walk) RIDE RIDE
Reply
(Reasoning: Monotonic]
Request 3
[x] cautiously 90 3x] [x1] [x]] [x]]
[x1] [x]]
[x]] [x2] [x]]
[x1]
[x1] [x1] [x]] [x1] [x1] [x]] [x1] [x2]
Reply
(Reasoning: Defeasible)
Request: 4
[walk cautiousi 90 x1 3x between Fly caut fousl
RTOR RTDR RTDR
RIDE RIDE [x1] RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE [x1] RIDE
RIDE RIDE RIDE RIDE RIDE [xi] RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE
[x1] WALK KALK NALK WALK WALK WALK [x2] WALK WALK NALK WALK
WALK WALK WALK
WALK WALK [x2] RIDE RIDE RIDE RIDE RIDE RIDE [x]] RIDE RIDE RIDE RIDE RIDE
RIDE RIDE RIDE RIDE [x1]
Reply 4
1 (Reasoning:
Monotonic!
Commentary
Composing rule 2 into x1 of rule results in the rule
Rule [walk cautiously 90 x1 3x] RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDS [xi] RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDS RIDE RIDE RIDE RIDE [x1]
And composing rule into x1 of rule 1 results in the rule
*Buee [walk cautiously 90 x1 3x between fly cautiously 90 x2] RIDE
RTDR RTDE RTDE RIDR RTD5 RTDE RTDR RTDE RIDE RTDE RTDR [x]] RTDE RTDE RIDE
RTBR RTDE
RTDR
RIDR
RTDE
[x1] RIDE RIDE RIDE RIDS RIDE RTDE RIDE RIDE RIDE RIDE RIDE RIDE [x]]
RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE
RIDE RIDE RIDE RIDE [x1] WALK WALK WALK MALK WALK WALK [x2] WALK MALK
MALK WALK RALK
[x2] RIDE
RIDE RIDE
RIDE RIDE RIDE RIDE RIDS [xi] RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE
RICE RIDE RIDE RIDE RIDE RIDE RIDE RIDE RIDE [x1]
Which directly contradicts rule 4 in the places highlighted by under-
line.
64


Reviews:
[{"review_id": "qefR5DOpX5i", "reviewer": "Reviewer_N6X6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the well-known compositional generalization benchmark dataset SCAN to cSCAN, driven by the idea of evaluating machine learning models' ability in semantic parsing with context and rule-like constraints.", "summary_of_the_review": "This paper proposes a novel dataset to evaluate neural networks' semantic parsing ability while considering knowledge-based constraints. The problem is important to the AI community, but the presentation of this paper should be improved.", "strength_and_weaknesses": "### Strengths:\n- The authors provided extensive details about how this dataset is generated, hence the reproducibility of this work is solid.\n- This work is motivated by an important problem. Semantic parsing by deep learning can hardly make use of domain knowledge or context such as common sense knowledge and rules. The proposed cSCAN dataset could be a candidate for evaluating it.\n\n### Weaknesses:\n- The presentation could be improved. The motivating example in natural language QA is intriguing and convincing. However, the proposed dataset is an extended version of SCAN, i.e., a set of robot navigation instructions. While reading this paper, I found myself feeling disappointed and confused when I see the gap between them. And the natural language QA example doesn't help me understand how the \"rules\" work in the robot navigation problem. Maybe changing the motivating example to a problem that is more related to the cSCAN dataset could improve the clarity of this paper.\n  - The notations in this paper are confusing. For example, $e_k= \\langle i_k, o_k \\rangle \\in\\mathcal{E}$ are examples, which consists of both inputs $i_k$ and $o_k$. However, the $i_k$ and $o_k$ are bost consist of the context $C$, and $C\\in 2^{\\mathcal{E}}$ is constructed by $e_k$. This is a recursive definition and makes me confused.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pLrigCpqd9b", "reviewer": "Reviewer_VbTf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A new approach is introduced for constructing benchmarks for the application of consistent rules in language models. This approach is used to create a new dataset that is then used to evaluate T5 models on 3 dimensions: applying learned rules, scaling to larger sets of rules and compositional generalization. \n", "summary_of_the_review": "I think overall the paper explores and important direction but could be improved in the presentation and extended with more experiments to make it much better and therefore at this time I recommend rejection but also wish to emphasize that I encourage the authors to keep working on the paper to make it better and then resubmit to a conference!", "strength_and_weaknesses": "**Strengths**: The idea is novel and it explores an important direction. See more details in the question below. The paper presents an idea, concrete usage of it, and empirical analysis on metrics of accuracy and consistency. The dataset section presents the data in sufficient detail, and the baselines are also described with a good level of detail. Exploring more baselines beyond T5 though would strengthen the paper. \n\n**Weaknesses**:\n* **Presentation of the motivation, real-world case**: The motivating example about the movie recommendations used to open the paper is not convincing in my opinion.  First, in general movie recommendation is more of a typical case of example-based learning, where people too give recommendations not by strict rules but by an approximate sense of what prior movies a person like, and what are his general preferences. More specifically, all except one of the rules presented in the example to illustrate rule-based inference are actually not rules but either facts or evidence. This is problematic as an introduction since evaluating rule application in language models is the main theme of the paper, where the example doesn’t show actual rules. Other more natural examples could be used to illustrate where people use rules, beyond the most obvious examples of math or physics problems etc, one could use examples of day-to-day such as admission criteria to a school or university, rules about payments or finance, examples of rules from the law system, etc etc. \n\n* **Presentation of the motivation, SCAN case**: The second example about SCAN is closer to exemplify about rules but the claim about people presented with rules rather than examples is not compelling either. E.g. if I showed a person example that f(A B and then C B)=aacc , f(D earlier than C earlier than A)=acd, we would quickly look for identifying rules in the example, learning that “and then” means putting after, “earlier than” means before and B means to do the action twice, even without being presented with the rules. People have a tendency to look for simple rules in examples, and they identify general rules (sometimes to general or strong) very commonly. People also commonly do induction and not only deduction, e.g. if f(abc)=d, and f(klm)=n then they will figure out that f(wxy) is likely to be z. Overall, I totally agree that rule-based learning is important in real-world and that AI models aren’t strong in this silk, but there could be the presentation of the motivation for that could be improved a lot.\n\n* **Conceptual learning task definition**: The concept seeks to achieve  5 properties that the learner should possess. No justification is given about why these 5, how they were selected, how they complement each other into a common goal etc. I believe the definition should be reworked to be more cohesive. The definitions of some of the properties also weren't clear to me.\n\n* **Experiments**: I would suggest to extend the experimental section, e.g. see the impact of the complexity of the rules and number of rules needed on the performance along different metrics, see generalization across multiple dimensions beyond only compositional generalization, do error analysis of what mistakes and trends are common in terms of accuracy and consistency and if any general behaviors and trends of models (vs. even human study) could be identified, etc.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "e2KtYDR0wP", "reviewer": "Reviewer_44yb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces an improvement of a learning method from samples and rules, named conceptual learning task. In particular the aim could be that of debug a movie recommendation.\n", "summary_of_the_review": "The notation adopted in the paper to describe the rules is difficult to understand. \n\nSection 3 is very difficult to follow. For instance the authors say \"we generate examples automatically using a Python program\". How? what is the meaning of \"picking a coherent set of basis rules\"? Even the points (a-b-c-d) following the previous sentence are ambiguous and not clear.\n\nAfter having read the first three section is not clear what is the machine learning task definition.\n\nAnother aspect regards the experimental evaluation. It seems that the proposed approach has been evaluated on artificial data in contrast with the example of recommendation system reported in the introduction.\n\nFinally, it should be important to stress the learning process and how it is related to the cSCAN proposed approach.\n", "strength_and_weaknesses": "- clarity\n- experimental evaluation\n+ incremental interesting approach", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G80nM6xwKT", "reviewer": "Reviewer_qoQG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission presents conceptual SCAN (cSCAN) as an instance of a conceptual learning task (CLT). CLTs, as defined in the manuscript, are tasks that contain a combination of examples and rules. Concretely, each example is defined as a tuple including a context, where the context is a set of rules and examples (potentially recursively, but in practice the inner examples have an \"empty\" context). cSCAN follows this formulation to introduce 4 different data sets, with smaller/larger set of rules and with/without explicit split for testing compositional generalization. Variants of the T5 transformer architecture are evaluated on the data set and demonstrated to fail in all but the simplest instances of the data set.\n", "summary_of_the_review": "With all the above in mind, I am leaning towards recommending the acceptance of the paper, since I do think that the data set on its own presents a valuable contribution, and there are some insights derived from the empirical evaluation. I do encourage the authors to more accurately scope their contributions in the introductory sections and to provide additional details of the evaluation. If additional experiments are feasible during the review process, those would likely be most impactful towards turning this into a higher quality manuscript.\n", "strength_and_weaknesses": "########### Strengths ###########\n- The idea of constructing data sets where examples come annotated with a context, which itself contains both examples and rules, is interesting\n- The running example of the movie recommendation system helps motivate and provide context to the reader for some of the design choices behind CLTs\n- The connections to related literature, especially in terms of contextual data sets and meta-learning, are well established and help provide additional context for the usefulness of the data set\n\n\n########### Weaknesses ###########\n- The experimental evaluation is not as thorough as I would have liked to see, and it seems to not provide as much insight as would be possible\n- The detail of how the transformers consume context are not explicitly provided\n- While the approach to creating CLTs seems to be presented as a general-purpose data-set-generating method, it is largely unclear whether it would be easily applicable beyond SCAN\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject