CURI: A Benchmark for Productive Concept Learning Under Uncertainty
Authors: Shanmukha Ramakrishna Vedantam
Arthur Szlam
Maximilian Nickel
Ari S. Morcos
Brenden M. Lake
Venue:  (2021)
Keywords: compositional learning
meta-learning
systematicity
reasoning

Abstract: Humans can learn and reason under substantial uncertainty in a space of infinitely many concepts, including structured relational concepts (“a scene with objects that have the same color”) and ad-hoc categories defined through goals (“objects that could fall on one’s head”). In contrast, standard classification benchmarks: 1) consider only a fixed set of category labels, 2) do not evaluate compositional concept learning and 3) do not explicitly capture a notion of reasoning under uncertainty. We introduce a new few-shot, meta-learning benchmark, Compositional Reasoning Under Uncertainty (CURI) to bridge this gap. CURI evaluates different aspects of productive and systematic generalization, including abstract understandings of disentangling, productive generalization, learning boolean operations, variable binding, etc. Importantly, it also defines a model-independent “compositionality gap” to evaluate difficulty of generalizing out-of-distribution along each of these axes. Extensive evaluations across a range of modeling choices spanning different modalities (image, schemas, and sounds), splits, privileged auxiliary concept information, and choices of negatives reveal substantial scope for modeling advances on the proposed task. All code and datasets will be available online.

Under review as a conference paper at ICLR 2021
CURI: A BENCHMARK FOR PRODUCTIVE CONCEPT LEARNING UNDER UNCERTAINTY
Anonymous authors Paper under double-blind review
ABSTRACT
Humans learn and reason under substantial uncertainty in space of infinitely many concepts, including structured relational concepts ("a scene with objects that have the same color") and ad-hoc categories defined through goals ("objects that could fall on one's head"). In contrast, standard classification benchmarks: 1) con- sider only fixed set of category labels, 2) do evaluate compositional concept learning and 3) do not explicitly capture a notion of reasoning under uncertainty. We introduce a new few-shot, meta-learning benchmark, Compositional Reasoning Under Uncertainty (CURT) to bridge this gap. CURT evaluates different aspects of productive and systematic generalization, including abstract understandings of disentangling, productive generalization, learning boolean operations, variable binding, etc. Importantly, it also defines a bodellindeepnneet "compositionality gap" to evaluate difficulty of generalizing ontto--distribution along each of these axes. Extensive evaluations across a range of modeling choices spanning different modalities (image, schemas, and sounds), splits, privileged auxiliary concept infor- mation, and choices of negatives reveal substantial scope for modeling advances on the proposed task. All code and datasets will be available online.
INTRODUCTION
Human concept learning is more flexible than today's AI systems. Human conceptual knowledge is productive: people can understand and generate novel concepts via compositions of existing concepts ("an apartment dog") (Murphy) 2002), unlike standard machine classifiers that are limited to fixed set of classes ("dog", "cat", etc.). Further, humans can induce goal-based, "ad hoc" categories such as "things to take from onc's 's apartment in fire" (children, dogs, keepsakes, etc.) (Barsalou) [983).. Thus, unlike AI systems, humans reason seamlessly in large, essentially "unbounded" concept spaces.
Beyond unboundedness, natural challenge in such concept spaces is uncertainty the right concept to be inferred is uncertain, as a plethora of candidate concepts could explain observations. For e.g. in F image panel). the "right" concept could be that "All objects are blue and have the same size", but it could also be "There are less than four objects in the scene", or "All objects have the same color". Humans gracefully handle such uncertainty and underdetermination (Tenenbaum & Griffiths)
2001 Xu & Tenenbaum| 2007 Goodman et al. 22008: Piantadosi et al. 2016). Popular compositional reasoning benchmarks such CLEVR (Johnson & Zhang 2016) for visual question answering and Ravens Progressive Matrices (Santoro et al., 2017) for deductive, analogical reasoning are compositionally rich and challenging in nature, but do not tackle ambiguity and underdeterminatiio..
We address this gap in the literature, and propose the Compositional Reasoning Under Uncertainty (CURI) benchmark to study how modern machine learning systems can learn concepts spanning a large, productively defined space (Figure In pursuit of this goal, we instantiate meta learning task where model must acquire a compositional concept from finite samples. signature of productivity in human thought is ability to handle novel combinations of known, atomic components. Thus, in CURI we instantiate different systematic train-test splits to analyze different forms of generalization in concept learning, involving novel combinations of intrinsic properties (e.g. color, shape) with boolean operators, counting, extrinsic object properties (e.g. object location), and a novel test of variable binding in context of compositional learning.
While related systematic splits have been proposed in prior work in context of other tasks such as question answering and analogical reasoning (Barrett al. 2018 Hill et al. 2019 Agrawal et al.

---

Under review as a conference paper at ICLR 2021
for-atl Vin (color?(x) "blue") and (all (size7(S)
size7(x)))
G:
Context Free
Grammar
objects are blue have the same size
Variables
Object in scene
objects 87(x)
for-all Vin (all (color?(x) color?(S)))
Quantifiers for-atl
All objects in the scene have the same color
exists
Functions
color? location?
exists Vin (color?(x) "blue") and all (shape?(S "squurrr""""""""""""""""""" "square")
shape? size? material? all
There exists blue object in the scene and the rest of the objects are squares
Operators and
Greater( Lesser(s)
not
Concept Space. Three example concepts (rows) along with schematic positive examples. Actual scenes are rendered in nultiple ways including the CLEVR renderer (Johnson et al. (2016) (see Figure Right: The grammar of variables, quantifiers, functiins and operators induce compositional concepts.
Figure 1:
2017 Johnson Johnsoonno et al., 2016; Vedantam et al., 2017 Higgins et al. 2017 Bakhtin et al. 2019 Lake &
Baron 2018 Ruis et al. 2020), ours is the first benchmark which tests different qualitative aspects of reasoning about productive concepts under uncertainty.
Compositional Reasoning Under Uncertainty (CURT) Task. Concretely, the CURI task tests few-shot learning of relational concepts in a large compositional conceptual space, with design inspiration from studies in cognitive modeling using language of thought (LOT) approach (Fodor) 1975 Piantadosi] 2011 Kemp et al. 2005). CURI includes scene-based concepts such as "All objects have the same color" and "There exists blue object while the rest are triangles" (Figure but unlike CLEVR (Johnson al. (016)) there are too few examples to deduce answers with certainty. Our benchmark is defined through a series of meta-learning episodes (see example in Figure 2): given positive and negative examples of
concept Dsup (known the "support set"), the goal of an episode to classify new examples Dauery (the "query set"). As few-shot classification (Fei-Fei et al. 22066), meta-learning (Vinyals al. 2016), and other open-set tasks (Lampert et al.) 2014). models are evaluated on novel classes outside the (meta-)training set. Unlike previous work (Triantafillou et al., 2019 Lake et al. 2019) that focuses on atomic concepts, our benchmarks concerns more structured, relational concepts built compositionally from set of atomic concepts, and involves reasoning under uncertainty an ideal learner must marginalize over many hypotheses when making predictions (Gelman et al. 2004 Xu & Tenenbaum 2007 Piantadosi et al. 2016).
Procuuttve the cylindrical
We also vary the modality in which scenes are presented- resseteddrendering them as images, symbolic schemas, and sounds- enabling future research on moodality-specinf representational choices for compositional reasoning under uncertainty. Fi- nally, we vary the concepts learned by the model during meta-training and meta-testing test dif- ferent aspects of systematic generalization.
Prodict
Label (1 or
Derpe
u E Quuary
Figure 2: CURT Task. Given as input support set Danp, with positive and negative examples correspond- ing to concept, the model has to infer the concept and produce accurate predictions novel images (right).
Compositionality Gap. In addition defining systematic splits, we also characterize (for the first time, in our knowledge), the difficulty of generalization entailed by each split by introducing the notion of model-independen "composi-- tionality gap"" Concretely, the compositionality gap the difference test performance between an ideal Bayesian learner with access to the full hypothesis space, and Bayesian learner with access to only a (potentially large) list of the hypotheses examined during meta-training. large gap indicates that any learner must extrapolate compositionally from the training hypotheses to solve the task; additionally, models can be compared ideal learners that either do or do not engage in such extrapolation. We anticipate that this tool will be more broadly useful for analyzing other benchmarks with compositional splits.
Models. We evaluate models around various dimensions which concern the difficulty of learning productive concepts under uncertainty, including: 1) the modality in which the input is rendered (image, schemas, sounds), 2) method used for reasoning across objects in scene (transformer,
2

---

Under review as a conference paper at ICLR 2021
Types BOOK
Functions Objects Sett of Objectee
Logical Functions and Functions on BOOL Bett
Tour compsti prossrtu//dl iloonstio of srussrtio strge domatt
P.
TRUF IP borth
Veriablea
seTial
INT FLOAT
ETT
Dencliuus rdgedddd soae
aut dijecta any Type
Shape/(3)
Tocoriseat
x1z07(e)
Teture sian of cdject
Reture shape of object
Ret INT M. iectandiode
Thfurn maderato odpac
of
not F
all SETIP]
any SET(P]
or
Riturns TRU in Firss
Riturns TF elemertio of TRUE
Reture TRUS otterr trae
Dancles the SCT of cijects scane
Tocalionn:: 1et
Return HRT M. yocooddeats center cbjetec
couet (FT(P)
Pittrra number simmmbb
TILE
Tanndee the stleppts
color? (3)
uer. umw.A. oslor styoor
Comparison Operations"
Constants (lustratee for traages)
Courts
2.
Eamppki color?15) (5)
heaaa (ssterite): In
R
Reture TRUF A see
Return TRUL IMM N equal
Natimia
Pusser Medil
Quantifirr
Return TALLE IP granse
Stopes
For-ull
in
Retume TRUE condiioe holde for
H x
Returno TRUE tn lnsker that
Sawe
4.98 6.79
TRUE
spply
Xary Inctton
4.5.6
esuamatt la 4-5T.
Suimbering cbots cop-ist ctimope)
Piople
901
Figure 3: Language of thought. All valid (type-consistent) compositions of functions are potential complex concepts in our dataset. Note that the functions are illustrated for the case of images and schemas. Location, size, shape etc. correspond different properties for sounds.
relation-network, global average pooling, concatenation), 3) whether not training provides ground- truth symbolic descriptions of concepts, and 4) how negative examples are sampled. Overall, our evaluations suggest that there is substantial room for improvement in compositional reasoning under uncertainty, w.r. the compositionality gap, representing a novel challenge for compositional learning.
Summary of contributions: 1) We introduce the Compositional Reasoning Under Uncertainty (CURI) benchmark for evaluating compositional, relational learning under uncertainty from observa- tional data; 2) We introduce a 'compositionality gap' metric for measuring the difficulty of systematic generalization from train to test; 3) We provide various baseline models for benchmarking progess.
2 RELATED WORK
Compositional Learning. Related work has examined systematic generalization in pattern com- pletion using Raven's matrices (PGM) (Santoro al.) 2017 Hill et al.. 2019) and visual question answering with CLEVR (Johnson et al. 2016; Bahdanau et al. 2019). CURT's s use of the CLEVR renderer further invites particular comparison with that benchmark. Compared to these more deduc- tive reasoning tests, CURI examines few-shot concept learning under substantial inherent uncertainty. Unlike puzzle solving or question answering, an ideal inductive learner on CURI cannot know the right rule with certainty. In essence, unlike CLEVR the "question" to be answered is not given to the model as input, but must be inferred - making the task more challenging. While PGMs do involve such inference, once the constraints of puzzle are identified, it does not: 1) have any uncertainty in the reasoning (which is crucial) and 2) involve any "ccnccpp" learning - where a concept applies to multiple images as much as it involves "instance" matching to complete sequence. In contrast, a successful CURI model behaves as if marginalizing over many hypotheses consistent with the observations e.g., (Tenenbaum & Griffiths 2001 Xu & Tenenbaum 2007 Piantadosi al. 2016). an ability which is rarely studied directly in deep learning models (although see (Grant et al. 2019))
Recently, Keysers et al. (2019) proposed a method to create "difficult" systematic splits based on the principle that they should share atoms but have maximally different compositions. This is complementary to our splits, which provide interpretable notions of what each split tests such as disentangling, complexity, variable binding etc. Moreover, our variable binding split is predicated on having different atoms between train and test, and thus cannot be recovered by their methodology.
Language of Thought (L.OT). Our choice of compositional concepts was most closely inspired by (Piantadosi et al.) 2016) along with other studies of human concept learning in the Language of Thought (LOT) framework (Fodor 1975 Goodman et al.. 2008 Kemp & Jern 2009 Piantadosi ctt al. 2012; Goodman et al., 2015 Overlan et al.) [017 Lake & Piantadosi] 2019). typical LOT studies of human learning, the conceptual space H. is defined through probabilistic context-free
grammar G, which specifies a set of conceptual primitives and their rules of combination. Here, we use LOT-inspired grammar G to generate unbounded set concepts H. while evaluating machine learning models trained without access to the underlying LOT.
3 COMPOSITIONAL REASONING UNDER UNCERTAINTY (CURI) DATASET
Concept space. The compositional concepts in CURI were inspired by the empirical and cognitive modeling work of Piantadosi et al. (2016). The space of concepts (LOT) is defined by a context
3

---

Under review as a conference paper at ICLR 2021
free grammar (g). Figure Shows the LOT and specifies how primitives and functions compose to produce a large unbounded concept space. The LOT has three variables: x, representing an object in scene, S {x] representing the set of all objects in the scene, and S_x S/ S/(x), representing the set of all objects in the scene except x. Fach concept describes a rule composed of object and scene properties, logical operators, and/or comparison operators, and can be evaluated on given
scene S to determine whether the scene satisfies the rule.
Object and scene properties are defined by functions which can be applied to objects or scenes: for example, size? ? (x) yields the size of an object x, while s ze (S) returns a set with the sizes of all the objects ((sizer (x)) x E S}). Comparison and logical operators can be used to compare and relate various properties of objects in scenes. In contrast to to/Pianadosii et al. (2016). we include a count operator, which determines how many times condition is satisfied by set, which allows us to check how well deep learning models are able to count (Chattopadhyay et al., 2 et al., 2016; Agrawal et al. 2017). Finally, quantifiers such as exi sts and for-al enrich the LOT by specifying the number of objects which must satisfy a given condition.
Consider the following example concept (Figure | bottom): "There exists a blue object in the scene and the rest of the objects are squares." " To access the color of given object, we use color? (x) and to access the shape of a given object, we use shape? (x). To determine whether an object matches a specific property, we can combine this with equality: shape? (x) use ex i sts s to specify that at least one object must be blue, S to specify all the objects except for that blue object, and all specify that all the objects in S_x must be squares. Putting it all together: exists x E S (color? (x) "blue") and all (shape? (S-x) = "square").
"square". Finally, we can
Structured Generalization Splits. A signature of productivity is the ability handle novel combi- nations of known components (Fodor] 1975 Fodor & Pylyshyn| 1988). Thus, in CURI, we consider splits that require generalizing to novel combinations of known elements from our LOT (Figure 3 including combinations of constants, variables, and functions. We achieve this by creating disjoint splits of concepts Herain and Heest for training and evaluating models. By varying the held out elements and their combinations, we obtain splits that evaluate different axes of generalization. In practice, we use our grammar G to sample and filter a large set of concepts (see Appendix [B.2 for more details), which yields set of 14,929 concepts H. for training and evaluation. We next describe how each split divides H into Hevain and Heests to test productive, out of distribution generalization:
Instance IID: Evaluates generalization to novel episodes from the same concept set. This is the standard setup in machine learning (Murphy) 2013), in which Hirain = Hrest- This is the only split where train and test concepts overlap.
Concept IID: Evaluates generalization to novel concepts based on an arbitrary random split of the concepts into Herain and Heest
Counting: Evaluates the ability to learn new concept h with novel property-count combina-
tions, e.g, the training concepts never filter for exactly '3 squares'.
Extrinsic properties: Evaluates the ability to learn new concept h, with novel combinations of extrinsic (e.g. location) and intrinsic (e.g. color) object properties.
Intrinsic properties: Evaluates the ability to learn a new concept / with novel combinations of intrinsic properties, e.g., the training concepts never reference both 'red' and 'rubber'.
Boolean operations: Evaluates the ability learn concepts which require application of familiar boolean operation to a property to which the operation has never been applied previously. Complexity split: Evaluates generalization from simple concepts (those which have less than or equal to 10 symbols) to more complex concepts (Ionger than 10 symbols). This is indicative of the productivity (Fodor 1975) exhibited by models, in generalizing from simpler concepts to more complex concepts.
Variable binding: Evaluates learning of entirely novel intrinsic properties, e.g. the training concepts involve only "red", "blue", and "green" but test concepts involve "yellow" (although 'yellow' objects can still appear in training scenes). This is indicative of inferential coher- ence (Fodor) 1975) in models, in generalizing rules of inference to novel atoms.
model that infers the underlying LOT during meta-training would be expected to perform well on any such systematic split. By comparing the performance of current models to to such ideal learners,
While some strings h might be different in surface form, they may yeild the same results when applied to images. In this split we account for such synonomy, and ensure that no two concepts which are synonyms are in different splits. See Appendix B.6 for more details.
4

---

Under review as a conference paper at ICLR 2021
this benchmark will allow to evaluate progress on the systematic out-of-distribution generalization capabilities of our current models. Appendix [ more details on the strucutred splits.
From Concepts to Meta-learning Episodes. A single episode comprises support set (Dupp) and a query set (D (uquer)), ) each of which is generated from given concept, h. Formally, a support or query set D has input data u and corresponding label y. i.e. D = {(v ) {u, }. }. Each support and query set contains 5 positive and 20 negative examples the space of negatives is generally much larger than that for positives. The set of positive examples sampled uniformly from categorical distribution over all positives. However, we consider two types of negatives: I) casy negatives, in which the negatives are also sampled at random, and 2) hard negatives, in which negatives are generated from closely related concept which also evaluates true on the positive examples in Dxupp, such that these negatives are maximally confusing. Altogether, for each split, our train, validation, and test sets contain 500000, 5000, and 20000 episodes, respectively.
negative examples are oversampled since
Compositionality Gap. A key aspect of our benchmark is to define the difficulty in learning that arises from the compositional structure of the concept space. Most of the splits above are structured in way such that Hes n Harinn = 0 forcing learner to use the compositional structure of the concept space to generalize to Heat We conceptualize the difficulty of this task through the notion of its compositionality gap. Intuitively, the compositionality gap captures the difference between the generalization performance of an ideal compositional learner (strong oracle) compared to an ideal non-compositional learner that is unable extrapolate outside the training concepts (weak oracle).
Formally, let Q E {strong, weak} denote an oracle over a concept space Hy. The pos- terior predictive distribution of an oracle for query scene and query label y E {0,1) is then given as: pa(ylu, Dsupp) = ΣXER, Ph(yih,, u)po Dupp). where (ra/i)ssspg (h) p(1n J lk; (u. and pr(h) denote the posterior and prior, respectively. Given metric of interest M (e.g., mean average precision or accuracy), the compositionality gap of learning task is then simply defined as the difference in performance of the strong and weak oracle when evaluating on concepts from Husa i.c., M (Pstrong) - M (pweak).
xx
Using this notion of compositionality gap. can then define ideal Iearners, i.e., the strong and weak oracle, simply via their priors. In particular, let w(() denote a weight on importance of each hypothesis and let I denote the indicator function. We then define the prior of an oracle as (h) = EN w(h))[(k' = h]. The difference between strong and weak oracle lies in which concepts can be accessed in these priors.
In this formalism, the strong oracle has access to the union of train and test concepts; that is Hstting = Herain U Heest- The weak oracle, the other hand only assumes access to Hweak = H which means it is unable to consider any hypothesis outside what has been seen in training and assigning zero probability mass. Given a support set D Deupp this difference in priors leads then to different inferences on posteriors and allows us to quantify how compositionally novel a learning task relative to these ideal learners.
METRICS AND BASELINES
During meta-test, given Daupo models are evaluated on their ability to learn novel concepts. We use two metrics for quantifying this: 1) Accuracy: evaluates the accuracy of model predictions across the query set Dyuerg. as standard practice in meta-learning (Lake et al., 2019 Snell et al.. 2017). Since there are more negative than positive labels, we report class balanced accuracy for better interpretability, averaging accuracies for the positive and negative query examples; and 2) mean Average Precision (mAP): evaluates models on much larger number of test scenes T for each episode (comprising 44,787 scenes, 3 per each concept in H). This resolves an issue that with a small query set, a strong model could achieve perfect accuracy without grasping the concept. Since episodes typically have many more negative than positive examples, Average Precision sweeps over different thresholds of model's 'ss score and reports the average of the precision values at different recall rates, e.g., (Everingham et al. 2010). mAP is then the mean across all of the meta-test episodes.
"Set to log-linear in the prefix serialization length of hypothesis, inspired by the observation that longer hypotheses are more difficult for humans (Feldman 2000). See Appendix for more details.
5

---

Under review as a conference paper at ICLR 2021
Support Query image Or Sound
(Poach Loodlor)
f(u
Support Images
Object
Encoder
Pooling
Odject
Giobal Avarge Pooking
Malatto Network Pookin
J(
S(W)
Sound
Thansformer Posling
Lquarry + Ox Leconcept
Sichware
Query
Concept
==
Call
(Chapntts) "aste")
Caay (eccssennnc)
Figure 4: Baseline models (left). Different choices for the encoder f(1) parameterization explored in the paper. We consider three modalities each of which is processed with modality specific encoder, followed by four kinds pooling architecture which as imput objects and their corresponding locations to provide an encoding for the datapoint. Training (right). The model is trained by processing the support images Dagp with positive (green) and negative (red) images. using / (x) to compute Lquery which computes generalization error on queries and Lecneep which learns to decode the truc concept as auxiliary task. Losses are weighted by O.
4.1 TRAINING LOsS
Denote by E RM the input to the model, which can be either in the form of image, sound or schema. We work in binary classification setting with labels y live in the space {0, 13. Then, given support set supp = {u, 3 and query set D Queery {u, sampled in accordance with a productive concept our training objective for single training instance can be written as Lquery + coomcep- Here Lquery = Eu Dum.. log p(Y = y/u, xupp)) is a standard maximum
likelihood meta-learning loss (Ravi & Larochelle 2016 Snell al. 2017: Finn al. 2017). , and Leoncept = log p(H H = h D Suppe) is an optional regularizer designed to encourage retaining information about the hypothesis of interest from the support set. set.
4.2 BASELINE MODEL ARCHITECTURES
Our baseline models (shown in Figure 4 parameterize the probability in the Lquery term above using prototypical networks (Snell et al. 2017). The prototypical network consists of an embedding function f fo and uses it compute prototypes Cp and c, for positive and negative examples by averaging (u) for positive and negative examples in the support set respectively. In equations, given query datapoint u', we compute
ezp(I]f((u)) cp112 p(Y = y/u: Duupp) erp(()f(u')) - cpl + exp(1]f(u) c,
In this formalism, the models study in this paper span different choices for f. Roughly, in each modality, we start with an encoder that converts the raw input into a set of vectors, and then pooling operation that converts that set vectors into a single vector. the case of images and sound (input as spectrograms), the encoder is a ResNet- 18; and the set of vectors is subsampling of spatial locations; and for schemas we vectorize components with a lookup table and combine them into set via feed-forward networks. In the case of images and sounds, the output of the encoder is enriched with position vectors. For the pooling operation, we study global averaging, concatenation, relation networks (Santoro et al.) (017)) and transformers (Vaswani et al.] (017)) equipped with different pooling operations (max, mean, sum, min) for reasoning inspired by Wang et al. (2019) (Figure middle panel, also see Appendix more details).
For the probability tion and then use LSTM (Hochreiter & Schmidhuber) 1997) parameterize pspp)) ITS TTSpp((h, (h 1- D Dwpp)) At each step of the LSTM concatenate (cp, ca] to the input.
Leuncepts we represent the concept as a sequence by prefix serializa-

---

Under review as a conference paper at ICLR 2021
G Schema
2 image
B
Sound
Strong Dracle
Weak Oracle
Random
0.00
0.3
Accuracy
0.23
Binding (color) Binding (shape)
Intrinsic
Complexity
Boolean
Concept IID
Extrinsic
Counting
Instance IID
Figure 5: Compasitinnality Gap. Different splits (x-axis) plotted w.r.t performance of the strong oracle (green line) and weak oracle (red line) on the mAP (top) and Accuracy (bottom) evaluated on respective test splits (using hard negatives in support and query sets). Difference between the two is the compositionality gap (comp gap). Yellow: shows the (best) relation-net net model on schema inputs, purple: shows the model on image inputs, and gray: shows the model sound inputs. Error bars are std d across 3 independent model runs.
EXPERIMENTAL RESULTS
We first discuss the compositionality gap induced by the different generalization splits and then delve into the impact of modeling choices on performance on the generalization splits. All models are trained for million steps, and are run with with 3 independent training runs to report standard deviations. We sweep over 3 modalities (image, schema, sound), 4 pooling schemes (avg-pool, concat, relation-net, transformer), 2 choices of negatives (hard negatives, random negatives) and choice of language (a- = 0.0, 1.0). Unless mentioned otherwise in the main paper we focus on results with hard negatives and 0.0. When instantiated for given modality, f()) (Figure 4) all have a similar number of parameters. The appendix contains details of the exact hyperparameters (Appendix and more comprehensive results for each split (Appendix G.7
note that the encoders
5.1 DATASET DESIGN AND COMPOSITIONALITT
How compositional are the structured splits?. Our main results are shown in Figure 5 Using our indepedndd measure of the compositionality gap (Section 3. different splits present varying challenges for generalizing from train to test. The most difficult splits, with the largest compositionality gaps, are the Binding (color) and Binding (shape). which is reasonable since they require learning concepts with entirely new property-values In contrast, the easiest split with the smallest compositionality gaps is the Instance IID split since does not require compositionality. Finally, while the mAP metric exposes larger value of comp gap, the ordering of splits in terms of comp gap is same for both metrics suggesting similar coarse-graine notions of compositionality.
Results for the best overall architecture, relation network (relation-net), is shown in Figure 5 Network performance on the easiest data format (schema; yellow bars) is generally better than the weak oracle, but substantially worse than the strong oracle. Counting is particularly challenging split where the models underperform even the weak oracle. Broadly, this suggests that the models capture some notion of compositionality especially for images and schemas relative to a weak oracle that rigidly considers only training hypotheses. but there is substantial room to improve (especially with respect the more stringent mAP metric). These results demonstrate that CURI provides challenging yet tractable setting for evaluating the compositional capabilities of models.
Finally, we found that the performance on the Instance IID split is not equal to the weak (and strong) oracle wrich are both equal in this case case-indienn indiicatnn that the best model does not make ideal posterior predictions even when compositionality is not an issue. Ideal predictions in this case would require the network to behave as if marginalizing over the training hypotheses, as the strong oracle does. similar plot to Figure can be found in Appendix G.4 for random negatives.
Influence of Negatives. Previous work (Hill et al.) 2019) has shown that the choice of random v.s. hard negatives for training and evaluation impacts compositional generalization substantially in the case of a particular set of analogical reasoning models. However, we argue that such decisions on dataset design can be made more objectively if one can evaluate the model-independdet comp gap. find that the comp gap with mAP when using random negatives decreases on
In our context, average by 5.5 + 1.4% compared to when we use hard negatives. This indicates that it is not only the choice of Herain and Haests which are identical for given compositional split (say Counting),
7

---

Under review as a conference paper at ICLR 2021
but also the choice of the negatives which "makes" the task compositionally novel. More generally, this indicates that the comp gap has utility as a more general diagnostic tool for making principled design decisions in compositional learning settings without the confound of specific model decisions.
5.2 DIFFERENCES BETWEEN MODELS
Best Models. In general, the best performing model is the relation-net applied to schema inputs, outperforming other combinations of models and input modalities on the Boolean, Concept IID, Complexity, and Instance IID splits on both the mAP well as accuracy metrics (:: although as mentioned above, none of the models are close to the strong oracle. is closely followed by the transformer model on schema inputs, which performs the best on Binding (color), Binding (shape), and Intrinsic splits (Appendix G.7). Utilizing schema inputs proves easier for abstraction except for the Extrinsic setting, where the task requires generalization to novel locations for objects in images, which is well supported by the inductive bias of the CNN encoder (Figure 4p. In this case, the smage-transfone
gets an mAP of 62.1 0.7%, compared the next best model at 60.9 + 0.7. Further, relational learning proves more crucial in the schema case than for images, with all image models (regardless of pooling) performing better than 59.4 + 1.3% mAP (achieved for image-avg-pool while schema-avg-pool models get only get to 53.4 + 1.5%.
When to use a transformer? Transformer models appear to outperform relation networks in splits concerning disentangling. For instance, for the Intrinsic split with schema-relations is at 55.1 0.8% v.s. 57.9 + 0.6% for schema-transformer, Similarly, for the Extrinsic split the image- transformer is at 62.1 + 0.7% compared to the image-relation-net at 60.8 + 1.1% We hypothesize that this is because the iterative message passing via. attention in transformers improves object representations for disentangling compared to relation networks that lack such mechanism.
What is the relative difficulty of abstraction from different modalities? One of the key con- tributions of our work is in providing multiple modalities (image, schema, sound) for productive concept learning. We next characterize the difficulty of abstraction based on modality for the various generalization settings. In the Intrinsic setting, we find that the schema models, which have access to a "perfect" disentangled representation significantly outperform image models schema-avg- g-poo model gets an mAP of 52.7 = 3.1% while an image-avg-pont model gets to 34.4 = 0.0% mAP.
Similarly, for the Counting split where the total number of objects are exactly specified in the schema (Figure 4) schemas are substantially better than images. For example. schema-relation-net get to 56.25 + 5.32% mAP while image-avg-pool is at 48.4 + 1.2% mAP. Interestingly, the next best model manage-relatio--nee
substantially worse, at 39.45 1.6%. Curiously, for while transformer models perform well at disentangling, they seem to be quite poor for Counting, with image- image-transformer models getting to only 32.4 + 1.4% mAP, suggesting a potential weakness for transformers. Overall, there appears to be an intimate link between the generalization setting and the input modality. suggesting avenues where representation learning could be improved for a given modality (e.g. images), relative to the kind of reasoning one is interested in (e.g. counting).
When does language help? On average, training models with explicit concept supervision using the concept loss (Section 4.1) improves performance by 2.8 + 0.6% mAP (SEM error). This is small boost relative to the gap between the original model and the strong oracle, suggesting that this simple auxiliary loss is not sufficient internalize the LOT in neural network. Overall, image models benefit more from language than schema models which natively utilize symbols (Appendix G.3
6 CONCLUSION
We introduced the compositional reasoning under uncertainty (CURI) benchmark for evaluating few- shot concept learning in large compositional space, capturing the kinds of producitivity, unboundness and underdetermination that characterize human conceptual reasoning. We instantiate series of meta-learning tasks, and evaluate numerous baseline models on various aspects of compositional reasoning under uncertainty, including inferential coherence, boolean operation learning, counting, disentangling, etc. Further, we introduce the notion of a compositionality gap to quantify the difficultly of each generalization type, and to estimate the degree of compositionality in current deep learning models. We hope our contributions of dataset, compositionality gaps, evaluation metrics and baseline models help spur progress in the important research direction of productive concept learning.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't ' just assume; look and answer: Overcoming priors for visual question answering. December 2017.
Dzmitry Bahdanau. Philippe Beaudoin, and Aaron Courville, CLOSURE: Assessing Systematic Generalization of CLEVR Models. arXiv preprint, pp. 1-13, 2019.
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. PHYRE: A new benchmark for physical reasoning. August 2019.
David T Barrett, Felix Hill, Adam Santoro, Ari S Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks, July 2018.
I W Barsalou. Ad hoc categories. Mem. Cognit,, 113))211-227, May 1983.
Prithvijit Chattopadhyay, Ramakrishna Vedantam, Ramprasaath R Selvaraju, Dhruv Batra, and Devi Parikh. Counting everyday objects in everyday scenes. April 2016.
Mark Everingham, Luc Van Gool, Christopher K Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. Int. Comput. Vis., 88(2):003-338 June 2010.
Fei-Fei, Fergus, and Perona. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18:594-611, 2006.
J Feldman. Minimization of boolean complexity in human concept learning. Nature, (68048 630-633, October 2000.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for fast adaptation of deep networks. March 2017.
Jerry Fodor. The Language of Thought. Harvard University Press, Cambrida, MA, 1975.
Jerry Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: critical analysis. Cognition, 28:3-71, 1988.
Andrew Gelman, John B Carlin, Hal Stern, and Donalid B Rubin. Bayesian Data Analysis. Chapman and Hall/CRC, 2004.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256, Chia Laguna Resort, Sardinia, Italy, 2010. PMLR.
Noah D Goodman, Joshua B Tenenbaum, Jacob Feldman, and Thomas L Griffiths. A rational analysis of rule-based concept learning. Cognitive Science, 32:108-54 2008
Noah D Goodman, Joshua B Tenenbaum, and Tobias Gerstenberg. Concepts in probabilistic language thought. Margolis and S Laurence (eds.), Concepts: New Directions. MIT Press, Cambridge, MA, 2015.
Erin Grant, Joshua C. Peterson, and Tom Griffiths. Learning deep taxonomic priors for concept learning from few positive examples. The Annual Meeting of the Cognitive Science Society, pp. 1865-1870, 2019.
Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matthew Botvinick. Demis Hassabis, and Alexander Lerchner. SCAN: Learning abstract hierarchical compositional visual concepts. July 2017.
Felix Hill, Adam Santoro, David G Barrett, Ari S Morcos, and Timothy Lillicrap. Learning to make analogies by contrasting abstract relational structure. January 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8): 1735-1780, November 1997. ISSN
doi: 11w.////c..999..9...... URL http:s://doi.orgge 111.0..1111/o1o../ 1997. 9. 1735

---

Under review as a conference paper at ICLR 2021
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: diagnostic dataset for compositional language and elementary visual reasoning. December 2016.
Rie Johnson and Tong Zhang. Supervised and Semi-Supervised Text Categorization LSTM for Region Embeddings. In International Conference on Machine Learning (ICML), 2016.
Charles Kemp and Alan Jem. Abstraction and relational learning. NIPS, 2009.
Charles Kemp, Aaron Bernstein, and Joshua B Tenenbaum. A Generative Theory of Similarity. In Proceedings of the 27th Annual Conference of the Cognitive Science Sociery, 2005.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method realistic data. December 2019.
Brenden M Lake and Marco Baroni. Generalization without Systematicity: On the Compositional Skills of Sequence-to- Sequence-to-Sqquence Recurrent Networks. In International Conference on Machine Learning (ICML), 2018.
Brenden M. Lake and Steven Piantadosi. People Infer Recursive Visual Concepts from Just Few Examples. Computational Brain & Behavior, 2019.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. The Omniglot Challenge: A 3-Year Progress Report. Current Opinion in Behavioral Sciences, 29:97-104, 2019.
Christoph Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE Trans. Pattern Anal. Mach. Intell., 36(3):453-465,
March 2014.
GL Murphy. The Big Book of Concepts. MIT Press, Cambridge, MA, 2002.
Kevin P. Murphy. Machine learning probabilistic perspective. MIT Press, Cambridge, Mass. [u.a.]. 2013. ISBN 978026201800 8029 0262018020, URL https: /www amazon
www.eeeeeooootaatoeooeeoaaaoooae oegpapaeiettttrerrreeeeeerrerrnn 0262018020/eef-sz: 20
com/
Matthew C. Overlan, Robert A. Jacobs, and Steven T. Piantadosi. Learning abstract visual concepts
via probabilistic program induction in Language of Thought. Cognition, 168:320-334, 2017.
Steven T Piantadosi, Joshua Tenenbaum, and Noah D Goodman. Bootstrapping in language of thought: A formal model of numerical concept learning. Cognition, 123:199-219 2012.
Steven T Piantadosi, Joshua B Tenenbaum, and Noah D Goodman. The logical primitives of thought: Empirical foundations for compositional cognitive models. Psychol. Rev., 123(4):392 324,4.4...4.. July
2016.
Steven Thomas Piantadosi. Learning and the language of thought. PhD thesis, Massachusetts Institute of Technology, 2011.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M. Lake. A Benchmark for Systematic Generalization in Grounded Language Understanding. In Advances in Neural Information Processing Systems 33, 2020.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, simple neural network module for relational
Peter Battaglia, and Timothy Lillicrap. reasoning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4967-4976 Curran Associates. Inc., 2017. URL http:://pappaeel 7082-a-simp neural neural-networke www.l-eettttooouuu...... module for r www.oonaaaeeeeeannnn........ pdf
niws wtw.ppps..c.papppee cc/paper/
10

---

Under review as a conference paper at ICLR 2021
Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. March 2017.
J Tenenbaum and T' Griffiths. Generalization, similarity, and Bayesian inference. Behavioral and Brain Sciences, 24:629-40, 2001.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- Dataset: dataset datasets for learning to learn from few examples. March 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. June 2017.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. May 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. June 2016.
Duo Wang, Mateja Jamnik, and Pictro Lio. Abstract diagrammatic reasoning with multiplex graph networks. September 2019.
F Xu and B Tenenbaum. Word learning as Bayesian inference. Psychological Review, 114:245-272. 2007.
A EXAMPLE EPISODES FROM THE DATASET
We show examples from the Concept IID split test set comprising the ground truth productive concept (top), along with the support and query for meta learning (rendered images), the alternate hypotheses which are consistent with the support set that is, other hypotheses which could also have generated the positive and negative examples in the support set and the concepts based on which we pick the hard negatives Figures to
B ADDITIONAL DATASET DETAILS
We first provide more details of the concept space G, then explain how we obtain H. the space of concepts for training and evaluation, provide more details of the structured splits, and finally explain the weight wh)) based on which we sample concepts.
B.1 MORE DETAILS OF THE GRAMMAR
We provide below the full grammar used sample concepts, where A + BIC means that A can expand to either B or C' under the rules defined by the grammar. We always start expanding at the START token and then follow the rules of the grammar until we hit a terminal node (which does not have any expansions defined). As and where possible, we followed the insights from [Piantadosi et al. (2016) choosing the sampling probabilities for various completions based on how well humans seem to be able to learn the corresponding primitive. For example, sample utterances with disjunctions (or) less frequently since they are known to be difficult for humans to learn. Based on Kemp & Jem (2009), we chose to represent location as discrete entity, such that relative, and categorical notions of left or right simply become comparisons in the location space (location? x > location? S_x). unlike the CLEVR dataset (Johnson al.) 2016) which defines categorical relational objects.
Here is the full grammar G used for sampling the concepts (as explained the main paper, S-x = S/ (x)). Note that the grammar always generates strings in postfix notation and thus the operands in each expansion occur before the operation:
11

---

Under review as a conference paper at ICLR 2021
Productive Concapt: andtoxta, locationX7(
m(purple...
Support
Query
Volid Hypatheses
for-all =lpurple, color?( x } (log-prob (-0.852])
for-ail =ccolor7i x.), purple (log-prob (-0.052))
allccolar( ). purple ) :log-prob (-1.950")
(matt- material((
allico purple }) (log-proh [-8 5231)
andi-(calarii ), purple ), anylmoterialt ). motal ) (log prob 5231)
Hypotheses Negstives
for-all
allicalort J. purple
ari-ipurpl,, ralor?( -(ypherr, shape?]
exiats andianytcalon) s_{ purpic cil(colorr) 5_[ purple
for-all ort-ishapee7 1. -(purple.
optrppu, colort (Shaape7
Figure Qualitative Example of an Episode CURT dataset. Best viewed zooming in, color.
Productive Concept: exists andialliiccationni (xx anylcoler?i (_(x) red
Support
Query
Valid Hypathoses
anyllocationyYi (log-prob (-1-621])
exists in >llocationY? 6. (log-orob [-1.62]])
exists anyllocatonn (-x) ). (log-orob (-16611))
exists allliacaton1? 5_(x)) (logsrob (-1,477))
in -llocationY?i
(leg (.2.14))
typotheses for Hard Negatives
exiits in andiall(calor1 S [_[x] 1. red x18, locatior
exists andrallccolorf S. (-X r. locatooco x11
exists andialltculor R. [xx] le anybice?
0.351
exists andisid. locationy?( x1: >llocatioonn
exists ortall(shepe?! sphere allicolor7 5_6x] red
Figure 7: Qualitative Example an Episode in CURI dataset. Best viewed zooming in, in color.
12

---

Under review as a conference paper at ICLR 2021
Productive Concept: locationX7 =(shape7?
Support
Query
Valid Hypotheses
for-all x In -isphera, shape71 ) Klog-arob [-0.855]}
for-all x In =|shape?1 x.,, sphere ] jing [-0.555])
allishape?| ), sphere I p (-1 (-1.954]]:
for-all isgheee shape?t ) =[shepe?| x sohere flog-prob (-7.8071)
for-all x andi=ishape?( x sphere ((pheere, shape?( x11
(log-prob (-7.80713
Hypotr Hard Negatives
arteese shape?t x =[color?[ green )
ori-isuugle, colort ) =ssphere, stape?( )))
for-all ori=isppeee, shape? =fgreen
sphere). locationX7(
ort-(shape7 =(ourple. color?(
Figure 8: Qualitative Example an Episode in CURI dataset. Best viewed zooming in, in color.
Froductive and(eilllacation)?) (_(x) any(sizeM) (xx 1, 0.7
Support
Query
Valld Hypothases
exists alliiacation171 5_(xx] J, [loq-prob (.0.008]]
extits -|count=:locationY1 (_xx (slnt (xx) 0.335) [log prob (-6.568))
for-all in -iCount= S countt(Shaeee shape? )) (lag-prob 6901)
for-all -icount=:shapp71 shape?t (Iunt-(lacation77 5 prob [.6.600])
andi>ilocaiccn77 x [size7 x 0.35 (lag prob [:7.45i])
Hypotheses Negetives
and(anytsize? anyllocation
exists andianytlacoionnny >e locationk? x))
(ldi-tlacaticnni pnylshope7 sphere
exists =lcount= Accunnt:location11 count- (lacatoooo 5 locationY?( x
exists >-count- Socatt-llocatinn11: (_K3 ) count-llocations 5.
Figure 9; Qualitative Example an Episode in CURI dataset. Best viewed zooming in, in color.
13

---

Under review as a conference paper at ICLR 2021
Productive Concept:
and(a Kmaterial?1 5 5_(xx) nubber allicalor?i 5_ cyan
Support
Query
8p
Valid Hypotheses
extits andiany(materiill 5 (-X) rueber 1, allrcolor? (_(xx) (log-prob 1-0.9831)
exists andialiccolor?( (-X cyan allimateriaa (>) rubber (log-prob [-1.162]]
exists andiali(materialt (xx) rubber allicolor?
(log-prob (-1.162])
tyyotthsss for Hard Negatives
"countm/colorf S oyan countm(materialf S metal
extits x in andiallccloor( S (o) cyan 1, ayyllccationn/ 1.33
exists allicolor?( S coc) cyan)
exists andiall(culor(( S (xx L.).lyye Ly arylcolor S
exists andiall(color1( (x) Lyaan), arylsizeri 5), 0.
Figure 10: Qualitative Example an Episode in CURT datasct. Best viewed zooming in, color.
Productive Concept:
exists andiali(iccationn1) (xx J, (elacationn?1 I, lacationX71 x)
Support
Query
=12
Valid Hypotheses
exists andial((cc (_(xx >(lacation(7) x locationn77 x11) [log prob (o.000])
Hypothesee eses Negatives
ontell(acceiionX1
locelionX?[
omtslloccaionn1 location =|color?
for ort-tyallow, calor7( (iaratoonn lacationX71
for apt-lperple, color? >(lacation locationX7(
Figure 11: Qualitative Example an Episode in CURI dataset. Best viewed zooming in, color.
14

---

Under review as a conference paper at ICLR 2021
START -> X S. BOOL exists= À S. BOOL for-all=
BOOL -> BOOL BOOL and | BOOL BOOL or BOOL not
CC = 1 SH SH
MM = SI SI = L. I.
NUM NUM
SI SI L L. NUM NUM
SETFC C all SETFSH SH all
SETFM M all
SETFSI SI all SETFL all
SETFC C any
SETFSH SH any SETEM M any SETFL L any
SETESI SI any
NUM -> SETFC count= SETFSH SH count=
SETFM M count= SETFSI SI count= | SETFL, I. count= 3
NUM -> 1
SETFC -> SET FC
SETFSH SET FSH
SETFM -> SET FM
SETFSI-> SET FSI
SETFL -> SET FL
C -> gray red I blue green |
brown purple
cyan yellow
C
OBJECT FC
SH
cube sphere I cylinder
SH
OBJECT FSH
-> rubber metal
M
OBJECT FM
SI
large small OBJECT FSI
SI
L
-> 1 2 3 4 5
7 8
OBJECT FL
FC
color?
FSH
shape?
FM
material?
FSI
size?
FL
-> locationX | location
OBJECT-> X
SET: S S-x
B.2 SAMPLING
We sample 2000000 initial hypotheses from the CFG G, and impose maximum depth in the recursion tree of 6 when sampling. That is, no node has depth larger than the recursion through which we generate concepts from the grammar G. We then reject and filter the hypotheses obtain set of "interesting" hypotheses H used in the main paper explained in more detail below:
Rejection Sampling: We reject the following string combinations after sampling from the grammar G:
All programs which contain "XS. for-all x" and "S-x" in the same program. This is asking that for all objects in a scene, a certain property is satisfied by everything other than the object, which is the same saying, for all objects in the scene.
15

---

Under review as a conference paper at ICLR 2021
All programs where we compare the properties of the object to itself, e.g. color? (x)
color? (x), where color? can be any function applied to the object.
All programs where we have the following string: exists (color? (S)
color? (x)) or for-all (color? (S) == color? (x) where color? can be any function applied to the object.
All programs which evaluate true on schemas more than 10% of the time and less than 10 times. The former condition ensures that we work with concepts which are in some sense interesting and surprising (as opposed to concepts which are always trivially true), and the second condition ensures that we have unique schmeas or datapoints to place in the support and query sets, which both have positive images each.
We provide examples of concepts which get rejected for being true too often below:
exists=x Ain S or =(locatioX? x ), locationY? x )
any (color? S ) brown
exists=x Vin S and( exists= (locationY? s ), locationX? ( x )
anyccolrr? ), brown
exists-x Vin or( all (color? S , gray )
(color? , brown
See Appendix for more details on the structured generalization splits which yeild train concepts Hirain and test concepts Heest-
B.3 CONCEPT PRIOR WEIGHT wh)
We next explain the form of the prior weight whh) that we use for defining the prior over the concepts provided to the models (both oracles as well as deep learning models). Given I(h), the number of tokens in the postfix serialization of the concept h, the unnormalized weight (hh is log-linear in the length, and is defined as follows:
((h) x exp 0.2 l(h)
Given split Q (train, test), the final, normalized weight is given as:
TThh)
whh)
(3)
Eno
Ho w w(h)
As explained in the main paper, the final prior for hypothesis given split Q is p(h)
Evero [h m h").
Our choice of the log-linear weight is inspired by the observation in cognitive science that longer boolean concepts are harder for people to learn (Feldman 2000).
Here are some examples of hypotheses with high weight (computed on Ω train test):
exists in S =(2, count=(coloy) ?( $S {-x}$ ), cyan ) exists in >(location =(count=(color") S ) brown ) 3 ) >(count=(Docationn?
x , 6
) 3 ) 2 ) any(locaiionY?( ), 6 )
=(1, count=(location>>
) 7)
=(3, count=(locationY?)
) 3
all (locaaaioon ) 2 )
exists in S all (locationY (( $$ $$_{-x}$ ) 5 )
16

---

Under review as a conference paper at ICLR 2021
=(2, count=(color S ) blue ), for-all x in S not (6, location 2( x )
=( (count=(color S gray ), 2 )
=22, count=(color ?( S ) gray )
B.4 EXECUTION ON IMAGES.
In order to create the perceptual inputs in the dataset U, we sample images using the renderer for CLEVR from Johnson et al. (016)) changing the range of objects to [2, 5]. to reduce clutter and enable easier learning predicates like any and a scenes with pixels as well as associated schema file u1 detailing the properties of all the objects sampled in the scene, including their location, shape, size, material, and rotation. Based on this, convert our sampled concepts into postfix notation and execute them on the schemas using operator stack. Concretely, execution of the concept h E H on u yields boolean true or false value {0. 1). We execute each such hypothesis hypothesis is true for an image. We threshold this score to retain the subset of hypotheses which true for more than 10% of the images and true least for 10 images, to pick a subset of "interesting" hypotheses H' for training models.
models. The CLEVR renderer produces
set of 990K images, yielding scores of how often
Bias. The image dataset here sampled itself has bias in terms of the location coordinates (in the pixel space). The CLEVR dataset generation process samples objects in the 3d (top-down y) coordinate space uniformly (from a grid of -3, to +3). However, since the camera always looking into the scene from outside, the image formation geometry implies that in the camera / image coordinates most of the objects appear to be away from the scene and very few are close to the camera. Thus, in terms of the y-coordinates we observe in the image coordinates bias in terms of the distribution not being unifrom. This also makes sense in general, as even the real world, objects are not found very close to the camera very far away from the camera in general. See Figure T2 for all the biases in the observation space u computed over 990K sampled images.
B.5 AUDIO.
To build the audio data, we use clips of orchestral instruments playing various pitches downloaded from phttpss://pp 1ha rmonia.. co. wtt.////ssurrreeeoooonaaaaaaaaaa lowing mappings of object properties:
We make the fol-
x location temporal location. larger bin means the note is played later.
y location pitch. All pitches between the instruments are the same (up to octaves).
color + instrument
gray + trumpet
red - clarinet
blue violin
green + flute
brown oboe
purple + saxaphone
cyan + french-horn
yellow guitar
shape > amplitude profile; either getting louder, getting softer, or constant volume
size + total volume
material + low-pass filtering or filtering.
All binned quantities use the same number of bins as in the image domain.
Since the chances of constraint being true for all obejets reduce exponentially as the number of objects increases.
17

---

Under review as a conference paper at ICLR 2021
material stribution
distributtin
3,8
4000002
3500000
Count
3:4
2000000
5000000
0.2
0000000
x distribution
color distribution
3000000
Count
Count
00000
1300000
00000
1000000
00002
500000
yeTww
pupp
green
Fiee
vall.
shape distribution
6002000
3000200
#000203
3000103
1000203
cibe Shape valuue Figure 12: Histogram of properties found in inputs u in the dataset. We notice that the properties are all largely uniform with bias in and y-coordinates towards the center of the image.
offiner
ophere
B.6 ANALYSIS SYNONOMY OF CONCEPTS.
We next show an analysis of concepts which have the same evaluation signatures on large set of 990K images, and thus synonymous (in context of the dataset hand). Note that while some of these concepts might be truly synonymous to each other (for example, A >> B is the same B < A), others might be synonymous in context of the image distribution we work with. For example, size never be greater than 0.7 in dataset and location never be greater than 8, and thus asking location is greater than or size greater than 0.7 has the same semantics on our dataset. In Figure 1 we show each such "concept" evaluate to the same truth value and plot a histogram of how many hypotheses each cluster tends to have. We notice that most of the concepts have synonym (i.e. there is only concept with the particular) evaluation signature, with long tail going upto 80 synonyms in a concept. In the Concept IID split ensure that none of the concepts which have the same signature are found across the vralt/val/tsst splits.
"meaning". which is a cluster of hypotheses which all
C DETAILED DISCUSSION OF THE STRUCTURED SPLITS
We provide more details how each of the structured splits described in Sec. 3 of the main paper are created. Assuming access to H. the space of concepts sampled and filtered from the grammar G. various heuristics produce the generalization splits in the paper:
Instance IID: This split is trivial since Hmain Hiest H
Concept IID; This split divides concepts into train and test by picking concepts random from H. and assigning them to Husin or Hest while ensuring that no two concepts which are synonyms Appendix Bre found in different splits.
This split forms cross product of all possible colors and (and, boolean operators, and partitions subset of such combinations which we want to only occur test. We use the following tokens for test:
Boolean:
18

---

Under review as a conference paper at ICLR 2021
2500
Number of concepts
2000
1500
1000
500
0
10
20
30
40
50
60
70
80
Number of Synonyms
Figure 13: Histogram of number hypotheses with same evaluation signatures.
green', 'or' red', 'or' 'green', 'and'
'purple', 'and' ''cyann, 'and'
We then create Hes to contain all concepts which have any of the combinations above. For example, if concept has both green and or we would place it in Hess After every feasible candidate placed in Hest based on this heurisitc, the remaining concepts in are assigned to Harain
Extrinsic: This split forms cross product all possible colors and locations in the dataset, and partitions a subset of such combinations that we only want to occur in test. We use the following tokens for test (only subset shown for illustration):
'7' gray''
'1' 'red' '3, 'purple'
'1' blue'
8', eyann '5', , 'yellow'
'5' 'green' '3', 'yellow' '7', 'purple'
'', 'buue' '', "cyan'
We then create Hest to contain all concepts which have any of the combinations above. For example, if concept has both gray and 7, and is related to location, that is contains locationss or locationy' keywords, we would place it Hust After every feasible candidate is placed in Hiest based this heurisite, the remaining concepts in H are assigned to Huain-
Intrinsic: This split forms and partitions a subset of such combinations that we only want to occur in test. We use the following tokens for test:
product of all possible colors and materials in the dataset,
'green', 'metal' 'purple', 'rubber' cyan'' 'rubber' 'red', 'metal' 'green', 'rubber'
We then create Hes to contain all concepts which have any of the combinations above. For example, if a concept has both green and metal, and is related to material, that is contains materi al? keyword, would place it in Hess After every feasible candidate is placed in Hiet based on this heurisite, the remaining concepts in H. are assigned to Huain-
Binding (color): This split takes all possible colors in the dataset, and partitions subset of colors that we only want to occur in test. We use the following tokens for test:
purple''
'oyan'
'yellow'
19

---

Under review as a conference paper at ICLR 2021
We then create Hust to contain all concepts which have any of the tokens above. For example, if concept has purple, we would place in Huss After every feasible candidate is placed in Hiess based on this heurisitc, the remaining concepts in H. are assigned to Hmain.
Binding (shape): This split takes all possible shapes in the dataset, and partitions subset of shapes that we only want to occur test. We use the following tokens for test:
cy inder
We then create Heest to contain all concepts which have any of the tokens above. For example, if concept has cy inder, we would place it in Hes After every feasible candidate placed in Hast based on this heurisitc, the remaining concepts in H. are assigned to Hwain-
Complexity: This split partitions into train and test based on length of the postfix serialization of the concept. Specifically, concepts shorter than tokens are placed in Harain and longer concepts are placed in Hiest-
D CREATING SUPPORT AND QUERY SETS
We next explain how we go from the initial dataset U - which contains large number of images, schema and sounds and a concept space Htrain and Hest. to a dataset for meta learning. To create the vaiiiing/alidatiion/esst sets for models, we sample series of episodes, each containing support set and a query set. We illustrate the sampling procedure for training episode below:
Support Set Sampling with Hard Negatives
1. Pick concept h ~ Pusin (h), with a preference for shorter hypotheses being more frequent based on the weights used define the prior Appendix [B33]
[B.3]
2. Pick 5 images (P), uniformly random from 4 such that h(u²) 1, where the concept is evaluated on the schema to determine the label Appendix B.4
Identify other concepts h' E s.t. (u(S)) and h' +
4. Pick images such that h d'(u") pick random images from 24 as negatives until we have 20 negatives.
and h(u") as negatives (N). If no such images exist,
5. Return D Daupp = P.N.
The sampling procedure for the Query set iterates all the steps above (except step 1, where we choose the concept h). Step 3 and 4 outline procedure for identifying hard negatives for training the model, by looking at other hypotheses which also explain a chosen set positives and using them to clarify what the concept of interest is.
We give below an analogous procedure for easy negatives:
Support Set Sampling with Easy Negatives
1. Pick concept h ~ Pusin (h), with a preference for shorter hypotheses being more frequent based on the weights used define the prior Appendix B.3
2. Pick 5 images (P), uniformly at random from 44 such that h(u") 1, where the concept is evaluated on the schema to determine the label Appendix B.4
Pick 20 random images from i4 as negatives, N.
4. Return Daupp P.N.
Similar to hard negatives, the sampling procedure for the Query set iterates all the steps above (except step 1, where we choose the concept h).
E REPRODUCIBILITY AND HYPERPARAMETERS
For all the models Figure. 4 in the main paper, we use the following hyperparameters. All the modalities are processed into set of objects (o.) 1N where each o; E RR for image and sound
20

---

Under review as a conference paper at ICLR 2021
models while for schema 0g R Further, the 1e-3 for schema models, and 5e-5 using the best learning rate for each modality across an initial sweep. The batch size for image and sound models is 8 episodes per batch, while for schema a batch size of 64. All models use the Adam optimizer. The overall scene representation across all the modalities is set 256, that is, TR25 All our models are initialized with the method proposed in (Glorot& & Bengio, 2010)). which we found to be crucial for training relation networks well. The initial representation from the first stage of the encoder (Fig. in main paper) with the objects for images has a size of 1 x08 Le. there are 80 objects, while for sound representations have 38 objects. In the schema case the number of objects is the ground truth number of objects which is provided as input to the model.
learning rate of 1e-4 for image models,
We trained all the models on the training set, and picked the best performing checkpoints on training measured in terms of mAP- to report all the results in the main paper. Our image and schema models are trained for million steps (16 epochs for images, 128 epochs for schemas) while the sound models are trained for 500K steps, and checkpoints are stored after every 30K steps.
All our models fit on a GPU with 16GB capacity except the relation network trained with image inputs, which needs at 32GB GPU. We use the pytorch framework to implement all our models.
F MODEL ARCHITECTURES FOR POOLING
In this section we detail the exact architectures used for the different pooling operations we consider in this paper as shown in Figure 4 center panel. We first establish some notation. Let o2 E R be the output object feature from the modality specific encoder (Figure 4 left panel), and let denote by O = {o,) the set of features for each of the objects in the scene, which includes optional position information indicating where the object is present in the scene (Figure (4. Let N be the requested dimensionality of the feature space from the pooling operation. Given this, we can describe the pooling operations used as follows:
avg-pool: We first average the representations across all the objects {o,) and then pass the averaged representation through an MIP with 256 x 512 x 384 x N units with batch normalization and rectified linear unit nonlinearity in the hidden layers.
concat: We first concatenate all the object representations in O, followed by an MLP with 256 x 512 x 256 x N units with batch normalization and rectified linear units nonlinearity in the hidden layers.
relation-net: For relation networks, following (Santoro et al. 2017 we relative position encoding that captures the relative positioning the objects in a scene for image and sound modalities, and the location information already present in the schema modality. Based on this, in the terminology of Santoro et al. (2017) our g() MLP has 256 x 256 x 256 x 256 hidden units with rectified linear unit non linearity and batch normalization whereas our /() MLP has 256 x 256 x N units with recitifed linear unit non linearlity and batch normalization in the middle (non-output) layers. Different from the original paper, we do not use dropout as we did not observe any overfitting in our experiments.
transformer: We use 2-head multi-head attention layer stacked times, with the feedfor- ward network dimenstions output vectors of for each object processed through these initial layers and pool across ob- jects by doing max (), mean (), sum (), min () operations and concatenating their outputs, similar to previous work by Wang et al. (2019). The final representation then does linear projection of this concatenated vector N. the dimensionality expected from the pooling module.
512. After forwarding through this module, take the
G ADDITIONAL RESULTS
G.1 HYPERPARAMETER SWEEPS OBJECT FEATURE DIMENSTONS
We next show the hyperparameter sweeps for image models in deterimining the choice of the dimensionality to represent each object o; for our image models (Figure 14). The same choice of
21

---

Under review as a conference paper at ICLR 2021
Object Dimension Sweep for Relation
05
MAP
02
Steps
Figure mAP on validation hard negatives (y-axis) vs number of training steps (x-axis) relation network models on images with different dimensionality for the object emhedding oi.
Learning Rate Sweep for Relation Nets + Images
00005
00005
00001
00003
00000
0.0005
0.00055
MAP
100600
200000 X0000
Sc0066
600000
X00000
acC000
Steps
Figure 15: mAP on validation for hard negatives (y-axis) vs number of training steps (x-axis) for relation network models on images with different learning rates.
dimensionality was replicated for sound models. In our initial sweeps, on the Concept IID split, across different choices of the dimensionality of objects, found relation networks outperform concat and global average pooling models substantially, and thus we picked the object dimensions based on what performs best for relation networks since overall choice of models for given split and modality. Based on the results in Figure 14 we picked o, E p64
interested in the best possible
G.2 IMAGE RELATION NETWORKS LEARNING RATE SWEEPS
We picked the learning rate for image models based on the best performing image relation network model, which across an initial sweep we found to yeild the best class of models. Figure 15 shows the performance of the models across learning rates of (1e-4, 5e-4, 2. 2.5e-4).
G.3 SWEEP ON USE OF LANGUAGE
As explained in the main paper (Figure. 4), the parameter controls the tradeoff between the query accuracy and the likelihood of the concept expressed as a prefix string. We generally found across broad range of values in 0, 0 0.01, 0.10, 1.0) the models generally performed the best at a = 1.0. Our initial experiments with a = 10.0 suggested substantially worse performance so we discareded it from the sweep. See Figures 16 for the corresponding results.
22

---

Under review as a conference paper at ICLR 2021
Language Sweep for Image Relation Networks
mAP
200000
400000
000000
100000 1000000
Steps
Figure 16: mAP validation for hard negatives (y-axis) number of training steps (x-axis) for relation network models on images with different amounts language usage by varying the parameter a..
Language Sweep for Schema Relation Networks
MAP
10
003
200908
6C9090
800000
Steps
Figure 17: mAP on validation for hard negatives (y-axis) vs number of training steps (x-axis) for relation network models on schemas with different amounts of language usage by varying the parameter
Language Sweep for Schema Concat Networks
MAP
10
0.2
Steps
Figure mAP on validation for hard negatives (y- axis) number of training steps (x-axis) for concat pooling models on schemas with different amounts of language usage by varying the parameter a.
23

---

Under review as a conference paper at ICLR 2021
Relation Networks on Easy Negatives
140
0.80
S Schema
046
Image
Oracle
0.80
Oracle
0.30
Random
Binding (color)
Binding (shape)
Complexity
Intrinsic
Boolean
Concept
Counting
Extrinsic
Accuracy
K Schemn
Image
Oracle
Random
Binding
Binding
Complexity
Intrinsic
Boolean
Concept
Counting
Extrinsic
Instance
(color)
(shape)
Figure 19: mAP (top) and accuracy (bottom) metrics for the different splits presented in the paper when easy negatives are used (ordered by their corresponding соnр gap). Yellow: shows the schema relation-net net model while purple: shows the image relation-net model. Notice that compared Fig. in the main paper, the comp gap smaller and the models appear to substantially closer the strong oracle in this setting compared when we use hard negatives.
G.4 RESULTS ON EASY NEGATIVES
In Figure 19 we show results for the relation-net model on various splits, where easy negatives are used populate the support and query sets during training and evaluation, unlike the case hard negatives discussed in the main paper (Figure. 5). Notice that the compositionality gap (comp gap) lower in general for easy negatives compared to the hard negatives as reported in the main paper. Further, we find that the best models are substantially closer to the strong oracle compared to Figure. main paper, showing that on the easier, less compositional task easier for machine learning models to approach the strong oracle (especially in terms of accuracy). Finally, it is interesting to note that with easy negatives appears that the best models outperform the weak oracle on the Counting split, while with the hard negatives one finds that the models worse than the weak oracle, suggesting poor generalization for counting.
G.5 FINER a SWEEP FOR COUNTING
finer alpha sweep for the Counting split since it appeared initial sweep that the counting split was not performing better with language. Concretely, we ran a new set of experiments sweeping over values of 01, 0.10, 1. 1,0, 5,0, 10 100.0). Across this broader range of values, we found models still did not show any statistically significant gains from using language v.s. not for the Counting split.
Finally,
G.6 CHOICE OF METRIC: MAP v.s. ACCURACY
In general, the mAP metric opens up a larger comp gap for the various splits than indicated by CBA. For example, with hard negatives, while CBA indicates a gap of 14.2% for Counting compared to 0% for Instance IID, mAP suggests gap of 34.4% for Counting relative to 0% for Instance IID. For the Binding (color) split its 86.5% comp gap (mAP) v.s. 34.0% for CBA. mAP, while being more expensive to compute evaluates more thoroughly to test if concept h is truly learnt by the model, by probing its performance on a large, representative set of negatives T, providing a more stringent test of compositional generalization.
24

---

Under review as a conference paper at ICLR 2021
Table 1: Performance meta-test, sorted based mAP (in %) on Binding (color) with hard negatives
modality
pooling
mAP (mean)
mAP (std)
schema
transformer
15.9
0,9
schema
relation-net
15.3
0.9
image
avg-pool
15.1
0.4
image
relation-net
14.8
0.7
schema
avg-pool
14.4
1.2
image
transformer
14.2
0.2
schema
concat
14.0
0.7
image
concat
13.5
0.4
sound
avg-pool
9,4
0.6
sound
relation-ner
9.2
0.3
sound
concat
8.0
0.6
Table 2: Performance on meta-test, sorted based on mAP (in %) on Boolean with hard negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
51.1
1.4
schema
rgg-pool
48.0
2.1
schema
concat
47.1
2.1
schema
transformer
46.9
1.1
image
transformer
44.0
2.2
image
relation-net
38.4
1.0
image
avg-pool
36.3
2.3
image
concat
29.6
2.4
sound
relation-net
25.5
0.8
sound
avg-pool
24.1
1.6
sound concat
22.3
1.1
G.7 DETAILED RESULTS ON ALL THE SPLITS IN HARD NEGATIVES SETTING
In this section we provide the full results of all of the tested models on each of the splits considered in the paper, in the hard negatives setting. Tables 1-9 show the results of different models (sorted in descending order based on mAP for each of the splits considered in the paper, in the case where models do not have access language.
G.8 DETAILED RESULTS ON ALL THE SPLITS IN EASY NEGATIVES SETTING.
In this section we provide the full results of all of the tested models on each of the splits considered in the paper, in the easy negatives setting. Tables 10-18 show the results of different models (sorted in descending order based on mAP for each of the splits considered in the paper, in the case where models do not have access language. Note that we did not evaluate transformer models sound models in this setting as this is qualitatively less interesting than the hard negatives setting and is not the main focus of the paper.
Table 3: Performance on meta-test, sorted based on mAP (in %) on Intrinsic with hard negatives
modality
pooling
mAP (mean)
mAP (std)
schema
transformer
57.9
0.6
schema
relation-net
55.1
0.9
image
transformer
54.1
3.0
schema
avg-pool
53.3
3.0
schema
concat
52.2
3.5
image
relation-net
47.6
7.5
image
avg-pool
34.5
0.4
image
concat
34.0
1.0
sound
concat
29.4
1.6
sound
avg-pool
26.5
2.3
sound
relation-net
24.3
2.4
25

---

Under review as a conference paper at ICLR 2021
Table Performance meta-test test, sorted based on (in %) on Concept IID with hard negatives
modality
pooling
mAP (mean)
mAP (std)
image schema
transformer
60.8
0.4
relation-net
60.7
0.3
schema
transformer
59.8
0.2
image
relation-net
58.5
0.7
image
avg "rg-pool
56.7
1.0
schema
avg-pool
54.6
schema
concat
53.3
0.9
image
concat
53.0
0.6
sound
concat
23.6
2.7
sound
avg-pool
22.7
2.3
sound
relation-net
21.8
NaN
Table Performance meta-test, sorted based on mAP (in %) on Instance IID with hard negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
63.9
0.4
image
transformer
61.5
0.9
schema
transformer
59.8
0.5
image
relation-net net
58.6
1.3
image
avg avg-pool
58.1
0.7
schema
avg-pool
57.4
0.9
image
concat
57.1
0.9
schema
concat
57.1
sound
avg-pool
23.5
1.4
sound
relation-net
22.3
2.3
sound
concat
21.7
0.5
Table Performance meta-test, sorted based on mAP (in %) Extrinsic with hard negatives
modality
pooling
mAP (mean)
mAP (std)
image
transformer
62.1
0.8
schema
transformer
60.9
0.6
image
relation-net
60.8
1.1
image
concat
60.8
0.7
schema
relation-net
59.8
0.4
image
avg-pool
59.4
1.3
schema
concat
54.3
2.3
schema
avg-pool
53.4
1.5
sound
avg-pool
25.7
NaN
sound
concat
23.2
1.4
sound
relation-net
18.3
1.7
Table 7: Performance on meta-test, test, sorted based on mAP (in %) on Complexity with hard negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
52,8
0.7
schema
transformer
49,7
0.6
schema
avg-pool
48.4
1.0
schema
concat
48.0
1.2
image
pool
45.8
0.4
image
transformer
45.6
0.2
image
relation-net
45.4
0.4
image
concat
42.6
3.4
sound
concat
26.0
NaN
sound
relation-net
23.3
sound
avg-pool
23.0
1.7
26

---

Under review as a conference paper at ICLR 2021
Table Performance on meta-test, sorted based on mAP (in %) on Binding (shape) with hard negatives
modality
pooling
mAP (mean)
mAP (std)
schema
transformer
30,9
0.6
schema
relation-net
30,4
1.5
image
relation-net
29.1
1.8
image
avg-pool
28.7
1.0
image
transformer
28.7
0.1
schema
avg-pool
28.3
image
concat
27.0
0.8
schema
concat
27.0
0.8
sound
relation-net
17.5
0,4
sound sound
concat
17.4
1.3
avg-pool
16.2
1.6
Table Performance on meta-test, sorted based on mAP (in %) Counting with hard negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
57.4
4.8
schema
avg-pool
55.0
6.4
schema
concat
50,0
3,2
image
avg-pool
48.5
1.3
schema
transformer
40.7
0.4
image
relation-net
40.3
2.0
image
concat
38.5
2.8
image
transformer
32.4
1.4
sound
concat
13.8
2.4
sound
avg-pool
13.4
1.0
sound
relation-net
13.0
1.7
Table 10: Performance meta-lest, sorted based on mAP (in %) on Binding (color) with easy negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
18.6
1.6
image
relation-net
16.3
0.8
image
vrg-pool
16.3
0.5
image
concat
15.6
0.5
schema
avg-pool
15.5
0.5
schema
concat
15.5
0.5
Table 11: Performance on meta-test, sorted based on mAP (in %6) on Boolean with easy negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
58.1
2.0
schema
avg-pool
55.0
2.1
schema
concat
54.7
2.2
image
relation-net
50.7
7.3
image
avg-pool
42.9
1.6
image
concat
40.9
1.7
Table 12: Performance on meta-test, sorted based on mAP (in %) Counting with easy negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
67.9
8.5
schema
ogg-pool
64.4
7.3
image
avg-pool
62.2
6.6
image
concat
61.2
6.9
schema
concat
60.8
6.7
image
relation-net
58.8
5.7
27

---

Under review as a conference paper at ICLR 2021
Table Performance on meta-test, sorted based on mAP %) on Extrinsic with easy negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
63.7
1.4
image
relation-net
62.9
1.1
image
concat
61,8
2.4
image
avg-pool
61.5
1.4
schema
avg-pool
57.5
1.2
schema
concat
57.2
1.5
Table 14: Performance on meta-test, sorted based on mAP (in %) on Intrinsic with easy negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
66.5
4.8
image
relation-net
64.9
5.3
image
avg-pool
64.4
4.5
schema
avg-pool
63.1
4.7
image
concat
62.7
4.5
schema
concat
60.8
4.6
Table 15: Performance meta-test, sorted based on mAP (in %) Concept IID with casy negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
68.2
1.0
image
relation-net
65.8
0.4
image
avg-pool
65.1
0.4
image
concat
64.7
0.4
schema
avg-pool
63.3
0.4
schema
concat
61.9
0.2
Table Performance on meta-test, sorted on mAP (in %) on Instance IID with easy negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
69.0
3.6
image
avg-pool
66.8
3.3
image
relation-net
66.5
3,2
image
concat
65,4
3.4
schema
wyg-pool
63.5
3.1
schema
concat
63.3
3.1
Table 17: Performance on meta-test, sorted based on mAP (in %6) on Complexity with easy negatives
modality
pooling
mAP (mean)
mAP (std)
schema
relation-net
55.1
1.8
schema
avg-pool
51.9
2.0
schema
concat
51.3
1.9
image
avg-pool
49.0
1.6
image concat
48.2
1.7
image
relation-net
46.6
1.5
Table 18: Performance on meta-test, sorted based on mAP (in %)
Binding (shape) with easy negatives
modality
pooling
mAP (mean)
mAP (std)
image
relation-net
33,9
2.4
schema
avg-pool
32.1
1.7
image
avg-pool
31.9
1.8
schema
relation-net
31.7
1.6
image
concat
31.3
1.7
schema
concat
31.0
1.5
28


Reviews:
[{"review_id": "YMyC4vWi-rC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe following work presents a CLEVR-based compositionality benchmark. The task of the model is to verify logical statements about an image, and in order to achieve such, must learn how to map individual statements to a composition of functions over the image checking for color, placement, shape, etc. Specific to this dataset is that it is explicitly few-shot, which forces the models to generalize very quickly and to infer under uncertainty.\n\n\nStrengths:\n\n-Introduction of novel compositionality gap to measure the extent to which any learned solution needs to extrapolate on each train/test split \n\n-Dataset is very carefully split in multiple different ways to individually test for generalization across various aspects\n\n-Extensive analysis, testing a wide range of the standard baseline methods\n\nWeaknesses and Concerns:\n\n-Like CLEVR, the statements are generated based on a grammar over a controlled set of relations/attributes/terms. While this is necessary for ruling out linguistic complexities from the analysis, one wonders how useful the dataset will be in the long run. Specifically because the dataset is setup to be used in a few-shot setting, models that perform increasingly better will do so by incorporating strong priors that match the true distribution of the dataset, at which point would we not simply be overfitting to this dataset due to its limited linguistic scope as we did for CLEVR?\n\n-It's not clear to me what the  benefit of the audio modality is. Specifically, I imagine that there are a number of different ways to encode the relational information in audio, with each possibly giving different results. Any conclusions are likely specific to the particular encoding rather than the audio modality as a whole.\n\nAdditional questions:\n\n-What were the main conclusions that we can draw from this dataset that would have been impossible in CLEVR?\n\nTypos:\n\nAppendix Figure 18: y axis has no tic marks\n\nOverall, I like the clean setup of the dataset's split and the some of the few-shot analysis on exsiting models. However, the novelty of the dataset as a whole is limited, as it is effectively a modification of CLEVR. It is not yet clear to me that the analysis has yielded any particularly groundbreaking insights, though I would love for the authors to correct me on that. Most importantly, I have strong doubts about the long-term usefulness of this dataset as a benchmark before we start overfitting, given its limited scope and few-shot setting.\n\n---------------------------------------------------------------------------------------------\n\nPost rebuttal update:\n\nAfter reading the clarifications from the authors, it is now more clear that the dataset is about the learnability of certain hypotheses as opposed to that of test-time verification. I am generally satisfied with the responses given by the authors, and willing to buy the statement that having a per-concept breakdown of learnability can be seen as a feature rather than a weakness pertaining to the use of a \"toy\" dataset. I think there are some valid insights as provided by this work, though I am more skeptical regarding the superiority of transformers on disentanglement tasks as the improvement gap was relatively small and there are various implementation details in the used transformers and relationship networks that could presumably shrink or even invert that gap. \n\nOverall, I think the writing clarity is still a significant concern. Several detailed passes over the paper were necessary just to get the general picture of what was going on. I think the toy example provided in the author's response is certainly helpful, and should improve the paper in this area somewhat. \n\nBased on the above, I am upgrading my rating to marginally above threshold.", "extra_scores": {}, "extra_text": {}}, {"review_id": "We67RaMaK9k", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces the \"compositional reasoning under uncertainty\" benchmark. \n\nThis is motivated by the observations that standard [classification] benchmarks: \n 1) consider only a fixed set of category labels, \n 2) do not evaluate compositional concept learning  \n 3) do not explicitly capture a notion of reasoning under uncertainty\n\nThe work proposes a task that requires several aspects of compositional reasoning, such as boolean operations, counting, etc. The author(s) introduce a notion of a \"compositionality gap\" to quantify the difficultly of each generalization type. \n\nI buy the vision of the work and I consider it a promising direction. However, I am not completely convinced by the work and its contribution to the existing works, especially the recent related work. \n\nWhat is the source of \"uncertainty\" in this challenge? What kind of \"uncertainty\" are we dealing with? The only place that you refer to it is the 2nd paragraph of your intro, but otherwise, I don't see anything else (while there are 18 mentions of \"uncertainty\" in section 1-2, there is only one mention of \"uncertainty\" in the whole main body section 3-5 and that is only in the title of Section 3!) \nWhy I am not convinced this is a good (useful?) notion of \"reasoning under uncertainty\". Just because two formulas/concepts can describe one scene, implies \"uncertainty\"? If so, what is the \"reasoning\" under this defined \"uncertainty\"? \n\nIn terms of writing: the intro/abstract are very clearly written and are easy to follow. But the rest of the work could be improved. \n\nOverall, good work but requires a bit of work polishing to make the contributions stronger. \n\n==============\n\nMinor issues: \"Producitivity\": typo? \n\nI felt that you're citing so many works that at times make it difficult to read. \n\n\"disentangling\": clarify or cite. It's not clear how it's related to the tasks listed in Fig 5. \n\nDefine \"productive generalization\" and \"systemic generalization\".\n\n======= \nUPDATE after reading author response:  The authors' example answered my question about the nature of the \"uncertainty\" in their setting. I have increased my confidence score and have retained my marginally-positive evaluation. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9ZmZ8sbSlA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes the CURI dataset to measure productive concept learning under uncertainty. The dataset is designed using a concept space defined by a language and formulated as a few-shot meta-learning problem to tell apart in-concept samples from out-of-concept samples. The authors also design several out-of-generalization data splits that test models' ood generalization performance. Together with an oracle model, the authors show using the prototypical network that the compositional concept learning and reasoning problem in CURI is challenging.\n\n---\n\n1. The proposed dataset touches the problem of generalizable concept learning, which is important, as more and more methods have been shown to be able to well fit the iid distributions.\n2. The dataset is well-designed in terms of generalization. It explicitly separates the concept space and proposes several aspects for the generalization problem.\n3. A good metric for the generalization gap that can measure how difficult each data split is by using oracle models.\n\n---\n\n1. One of the major problems I consider in this work is how to make sure the positive samples follow a unique concept. The authors mention that the actual concept space is unbounded and uncertain. Therefore, there could potentially be an infinite number of concepts where the positive samples are satisfied. Is it possible to make sure that the positive cases and the negative cases can prune all other concepts to make sure the actual concept tested in $D_{supp}$ is unique? If not, during testing, how to make sure a ''negative'' sample is truly negative rather than coincidentally satisfying the concept? If the problems can not be properly addressed, then the dataset may contain fundamental errors.\n2. Another major problem is the missing references to existing few-shot meta-learning works. There has been a plethora of related papers and this work only compares with the most basic prototypical network. See the following references for examples. Besides, a couple of recent works also try to address the compositional learning problem. These models are also missing. Without sufficient experiments, it's hard to paint a full picture of the dataset.\n3. The statements regarding CLEVR and PGM are inaccurate. The paper states that questions to ask need to be inferred in CURI, which I believe is not true because the question is simply asking a model to tell an odd one out. It's just that CLEVR provide questions that can be parsed to programs while CURI does not. For PGM, it's stated that \"once the constraints of a puzzle are identified\" ... However, the uncertainty problem and concept learning problem are right at identifying the constraints, same as in CURI where the concept description needs inferring.\n4. Typos: Page 4 at the bottom \"to to\" -> \"to\", \"yeild\" -> \"yield\"\n5. How is $P(H=h | D_{supp})$ defined? I think this could be a major part impacting the performance. If you have a proper reasoner to do it (say an oracle model), it doesn't make sense to have such a low accuracy. But if it is just a multi-way classification setup, then the performance can be poor.\n6. A very important and highly related work is missing: Andreas, Jacob, Dan Klein, and Sergey Levine. \"Learning with latent language.\" ACL 2018. This work is basically in the same setup as CURI and the models used, the methodology, the problem it considers \"at least\" need some in-depth discussions. \n\n[1] Lake, Brenden M. \"Compositional generalization through meta sequence-to-sequence learning.\" NeurIPS. 2019.\n[2] Finn, Chelsea, Pieter Abbeel, and Sergey Levine. \"Model-agnostic meta-learning for fast adaptation of deep networks.\" ICML. 2017.\n[3] Bertinetto, Luca, et al. \"Meta-learning with differentiable closed-form solvers.\" ICLR. 2018.\n[4] Lee, Kwonjoon, et al. \"Meta-learning with differentiable convex optimization.\" CVPR. 2019.\n[5] Sung, Flood, et al. \"Learning to compare: Relation network for few-shot learning.\" CVPR. 2018.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by 3 experts in the field. The reviewers raised their concerns on lack of novelty, unconvincing experiment, and the presentation of this paper, While the paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject