CURI: A Benchmark for Productive Concept Learning Under Uncertainty

1 INTRODUCTION . Human concept learning is more flexible than today ’ s AI systems . Human conceptual knowledge is productive : people can understand and generate novel concepts via compositions of existing concepts ( “ an apartment dog ” ) ( Murphy , 2002 ) , unlike standard machine classifiers that are limited to a fixed set of classes ( “ dog ” , “ cat ” , etc. ) . Further , humans can induce goal-based , “ ad hoc ” categories such as “ things to take from one ’ s apartment in a fire ” ( children , dogs , keepsakes , etc . ) ( Barsalou , 1983 ) . Thus , unlike AI systems , humans reason seamlessly in large , essentially “ unbounded ” concept spaces . Beyond unboundedness , a natural challenge in such concept spaces is uncertainty – the right concept to be inferred is uncertain , as a plethora of candidate concepts could explain observations . For e.g . in Figure 1 ( top , image panel ) , the “ right ” concept could be that “ All objects are blue and have the same size ” , but it could also be “ There are less than four objects in the scene ” , or “ All objects have the same color ” . Humans gracefully handle such uncertainty and underdetermination ( Tenenbaum & Griffiths , 2001 ; Xu & Tenenbaum , 2007 ; Goodman et al. , 2008 ; Piantadosi et al. , 2016 ) . Popular compositional reasoning benchmarks such as CLEVR ( Johnson & Zhang , 2016 ) for visual question answering and Ravens Progressive Matrices ( Santoro et al. , 2017 ) for deductive , analogical reasoning are compositionally rich and challenging in nature , but do not tackle ambiguity and underdetermination . We address this gap in the literature , and propose the Compositional Reasoning Under Uncertainty ( CURI ) benchmark to study how modern machine learning systems can learn concepts spanning a large , productively defined space ( Figure 1 ) . In pursuit of this goal , we instantiate a meta learning task where a model must acquire a compositional concept from finite samples . A signature of productivity in human thought is our ability to handle novel combinations of known , atomic components . Thus , in CURI we instantiate different systematic train-test splits to analyze different forms of generalization in concept learning , involving novel combinations of intrinsic properties ( e.g . color , shape ) with boolean operators , counting , extrinsic object properties ( e.g . object location ) , and a novel test of variable binding in context of compositional learning . While related systematic splits have been proposed in prior work in context of other tasks such as question answering and analogical reasoning ( Barrett et al. , 2018 ; Hill et al. , 2019 ; Agrawal et al. , 2017 ; Johnson et al. , 2016 ; Vedantam et al. , 2017 ; Higgins et al. , 2017 ; Bakhtin et al. , 2019 ; Lake & Baroni , 2018 ; Ruis et al. , 2020 ) , ours is the first benchmark which tests different qualitative aspects of reasoning about productive concepts under uncertainty . Compositional Reasoning Under Uncertainty ( CURI ) Task . Concretely , the CURI task tests few-shot learning of relational concepts in a large compositional conceptual space , with design inspiration from studies in cognitive modeling using a language of thought ( LOT ) approach ( Fodor , 1975 ; Piantadosi , 2011 ; Kemp et al. , 2005 ) . CURI includes scene-based concepts such as “ All objects have the same color ” and “ There exists a blue object while the rest are triangles ” ( Figure 1 ) but unlike CLEVR ( Johnson et al. , 2016 ) there are too few examples to deduce answers with certainty . Our benchmark is defined through a series of meta-learning episodes ( see example in Figure 2 ) : given positive and negative examples of a new concept Dsupp ( known as the “ support set ” ) , the goal of an episode is to classify new examples Dquery ( the “ query set ” ) . As in few-shot classification ( Fei-Fei et al. , 2006 ) , meta-learning ( Vinyals et al. , 2016 ) , and other open-set tasks ( Lampert et al. , 2014 ) , models are evaluated on novel classes outside the ( meta- ) training set . Unlike previous work ( Triantafillou et al. , 2019 ; Lake et al. , 2019 ) that focuses on atomic concepts , our benchmarks concerns more structured , relational concepts built compositionally from a set of atomic concepts , and involves reasoning under uncertainty – an ideal learner must marginalize over many hypotheses when making predictions ( Gelman et al. , 2004 ; Xu & Tenenbaum , 2007 ; Piantadosi et al. , 2016 ) . We also vary the modality in which scenes are presented—rendering them as images , symbolic schemas , and sounds— enabling future research on modality-specific representational choices for compositional reasoning under uncertainty . Finally , we vary the concepts learned by the model during meta-training and meta-testing to test different aspects of systematic generalization . Compositionality Gap . In addition to defining systematic splits , we also characterize ( for the first time , in our knowledge ) , the difficulty of generalization entailed by each split by introducing the notion of a model-independent “ compositionality gap ” . Concretely , the compositionality gap is the difference in test performance between an ideal Bayesian learner with access to the full hypothesis space , and a Bayesian learner with access to only a ( potentially large ) list of the hypotheses examined during meta-training . A large gap indicates that any learner must extrapolate compositionally from the training hypotheses to solve the task ; additionally , models can be compared to ideal learners that either do or do not engage in such extrapolation . We anticipate that this tool will be more broadly useful for analyzing other benchmarks with compositional splits . Models . We evaluate models around various dimensions which concern the difficulty of learning productive concepts under uncertainty , including : 1 ) the modality in which the input is rendered ( image , schemas , sounds ) , 2 ) method used for reasoning across objects in a scene ( transformer , Variables . Types . Constants ( Illustrated for Images ) . relation-network , global average pooling , concatenation ) , 3 ) whether or not training provides groundtruth symbolic descriptions of concepts , and 4 ) how negative examples are sampled . Overall , our evaluations suggest that there is substantial room for improvement in compositional reasoning under uncertainty , w.r.t the compositionality gap , representing a novel challenge for compositional learning . Summary of contributions : 1 ) We introduce the Compositional Reasoning Under Uncertainty ( CURI ) benchmark for evaluating compositional , relational learning under uncertainty from observational data ; 2 ) We introduce a ‘ compositionality gap ’ metric for measuring the difficulty of systematic generalization from train to test ; 3 ) We provide various baseline models for benchmarking progess . 2 RELATED WORK . Compositional Learning . Related work has examined systematic generalization in pattern completion using Raven ’ s matrices ( PGM ) ( Santoro et al. , 2017 ; Hill et al. , 2019 ) and visual question answering with CLEVR ( Johnson et al. , 2016 ; Bahdanau et al. , 2019 ) . CURI ’ s use of the CLEVR renderer further invites particular comparison with that benchmark . Compared to these more deductive reasoning tests , CURI examines few-shot concept learning under substantial inherent uncertainty . Unlike puzzle solving or question answering , an ideal inductive learner on CURI can not know the right rule with certainty . In essence , unlike CLEVR the “ question ” to be answered is not given to the model as input , but must be inferred – making the task more challenging . While PGMs do involve such an inference , once the constraints of a puzzle are identified , it does not : 1 ) have any uncertainty in the reasoning ( which is crucial ) and 2 ) involve any “ concept ” learning – where a concept applies to multiple images – as much as it involves “ instance ” matching to complete a sequence . In contrast , a successful CURI model behaves as if marginalizing over many hypotheses consistent with the observations e.g. , ( Tenenbaum & Griffiths , 2001 ; Xu & Tenenbaum , 2007 ; Piantadosi et al. , 2016 ) , an ability which is rarely studied directly in deep learning models ( although see ( Grant et al. , 2019 ) ) . Recently , Keysers et al . ( 2019 ) proposed a method to create “ difficult ” systematic splits based on the principle that they should share atoms but have maximally different compositions . This is complementary to our splits , which provide interpretable notions of what each split tests such as disentangling , complexity , variable binding etc . Moreover , our variable binding split is predicated on having different atoms between train and test , and thus can not be recovered by their methodology . Language of Thought ( LOT ) . Our choice of compositional concepts was most closely inspired by ( Piantadosi et al. , 2016 ) along with other studies of human concept learning in the Language of Thought ( LOT ) framework ( Fodor , 1975 ; Goodman et al. , 2008 ; Kemp & Jern , 2009 ; Piantadosi et al. , 2012 ; Goodman et al. , 2015 ; Overlan et al. , 2017 ; Lake & Piantadosi , 2019 ) . In typical LOT studies of human learning , the conceptual space H is defined through a probabilistic context-free grammar G , which specifies a set of conceptual primitives and their rules of combination . Here , we use a LOT-inspired grammar G to generate an unbounded set conceptsH , while evaluating machine learning models trained without access to the underlying LOT . 3 COMPOSITIONAL REASONING UNDER UNCERTAINTY ( CURI ) DATASET . Concept space . The compositional concepts in CURI were inspired by the empirical and cognitive modeling work of Piantadosi et al . ( 2016 ) . The space of concepts ( LOT ) is defined by a context free grammar ( G ) . Figure 3 shows the LOT and specifies how primitives and functions compose to produce a large unbounded concept space . The LOT has three variables : x , representing an object in a scene , S = { x } Ni=1 representing the set of all objects in the scene , and S−x = S/ { x } , representing the set of all objects in the scene except x . Each concept describes a rule composed of object and scene properties , logical operators , and/or comparison operators , and can be evaluated on a given scene S to determine whether the scene satisfies the rule . Object and scene properties are defined by functions which can be applied to objects or scenes : for example , size ? ( x ) yields the size of an object x , while size ? ( S ) returns a set with the sizes of all the objects ( { size ? ( x ) : x ∈ S } ) . Comparison and logical operators can be used to compare and relate various properties of objects in scenes . In contrast to Piantadosi et al . ( 2016 ) , we include a count operator , which determines how many times a condition is satisfied by a set , which allows us to check how well deep learning models are able to count ( Chattopadhyay et al. , 2016 ; Johnson et al. , 2016 ; Agrawal et al. , 2017 ) . Finally , quantifiers such as exists and for-all enrich the LOT by specifying the number of objects which must satisfy a given condition . Consider the following example concept ( Figure 1 bottom ) : “ There exists a blue object in the scene and the rest of the objects are squares. ” To access the color of a given object , we use color ? ( x ) and to access the shape of a given object , we use shape ? ( x ) . To determine whether an object matches a specific property , we can combine this with equality : shape ? ( x ) = “ square ” . Finally , we can use exists to specify that at least one object must be blue , S−x to specify all the objects except for that blue object , and all to specify that all the objects in S−x must be squares . Putting it all together : exists x ∈ S ( color ? ( x ) = “ blue ” ) and all ( shape ? ( S−x ) = “ square ” ) . Structured Generalization Splits . A signature of productivity is the ability to handle novel combinations of known components ( Fodor , 1975 ; Fodor & Pylyshyn , 1988 ) . Thus , in CURI , we consider splits that require generalizing to novel combinations of known elements from our LOT ( Figure 3 ) , including combinations of constants , variables , and functions . We achieve this by creating disjoint splits of concepts Htrain and Htest for training and evaluating models . By varying the held out elements and their combinations , we obtain splits that evaluate different axes of generalization . In practice , we use our grammar G to sample and filter a large set of concepts ( see Appendix B.2 for more details ) , which yields a set of 14,929 conceptsH for training and evaluation . We next describe how each split dividesH intoHtrain andHtest , to test productive , out of distribution generalization : • Instance IID : Evaluates generalization to novel episodes from the same concept set . This is the standard setup in machine learning ( Murphy , 2013 ) , in whichHtrain =Htest . This is the only split where train and test concepts overlap . • Concept IID : Evaluates generalization to novel concepts based on an arbitrary random split of the concepts intoHtrain andHtest.1 • Counting : Evaluates the ability to learn a new concept h with novel property-count combina- tions , e.g , the training concepts never filter for exactly ‘ 3 squares ’ . • Extrinsic properties : Evaluates the ability to learn a new concept h , with novel combinations of extrinsic ( e.g . location ) and intrinsic ( e.g . color ) object properties . • Intrinsic properties : Evaluates the ability to learn a new concept h with novel combinations of intrinsic properties , e.g. , the training concepts never reference both ‘ red ’ and ‘ rubber ’ . • Boolean operations : Evaluates the ability to learn concepts which require application of a familiar boolean operation to a property to which the operation has never been applied previously . • Complexity split : Evaluates generalization from simple concepts ( those which have less than or equal to 10 symbols ) to more complex concepts ( longer than 10 symbols ) . This is indicative of the productivity ( Fodor , 1975 ) exhibited by models , in generalizing from simpler concepts to more complex concepts . • Variable binding : Evaluates learning of entirely novel intrinsic properties , e.g . the training concepts involve only “ red ” , “ blue ” , and “ green ” but test concepts involve “ yellow ” ( although ‘ yellow ’ objects can still appear in training scenes ) . This is indicative of inferential coherence ( Fodor , 1975 ) in models , in generalizing rules of inference to novel atoms . A model that infers the underlying LOT during meta-training would be expected to perform well on any such systematic split . By comparing the performance of current models to to such ideal learners , 1While some strings h might be different in surface form , they may yeild the same results when applied to images . In this split we account for such synonomy , and ensure that no two concepts which are synonyms are in different splits . See Appendix B.6 for more details . this benchmark will allow us to evaluate progress on the systematic out-of-distribution generalization capabilities of our current models . Appendix C provides more details on the strucutred splits . From Concepts to Meta-learning Episodes . A single episode comprises a support set ( Dsupp ) and a query set ( Dquery ) , each of which is generated from a given concept , h. Formally , a support or query set D has input data u and corresponding label y , i.e . D = { { yi } Ni=1 , { ui } Ni=1 } . Each support and query set contains 5 positive and 20 negative examples — negative examples are oversampled since the space of negatives is generally much larger than that for positives . The set of positive examples are sampled uniformly from a categorical distribution over all positives . However , we consider two types of negatives : 1 ) easy negatives , in which the negatives are also sampled at random , and 2 ) hard negatives , in which negatives are generated from a closely related concept which also evaluates true on the positive examples in Dsupp , such that these negatives are maximally confusing . Altogether , for each split , our train , validation , and test sets contain 500000 , 5000 , and 20000 episodes , respectively . Compositionality Gap . A key aspect of our benchmark is to define the difficulty in learning that arises from the compositional structure of the concept space . Most of the splits above are structured in a way such that Htest ∩ Htrain = ∅ – forcing a learner to use the compositional structure of the concept space to generalize toHtest . We conceptualize the difficulty of this task through the notion of its compositionality gap . Intuitively , the compositionality gap captures the difference between the generalization performance of an ideal compositional learner ( strong oracle ) compared to an ideal non-compositional learner that is unable to extrapolate outside the training concepts ( weak oracle ) . Formally , let Ω ∈ { strong , weak } denote an oracle over a concept space HΩ . The posterior predictive distribution of an oracle for query scene u and query label y ∈ { 0 , 1 } is then given as : pΩ ( y|u , Dsupp ) = ∑ h∈HΩ pΩ ( y|h , u ) pΩ ( h|Dsupp ) , where pΩ ( h|Dsupp ) ∝ pΩ ( h ) p ( { yi } Ni=1|h ; { ui } Ni=1 ) and pΩ ( h ) denote the posterior and prior , respectively . Given a metric of interest M ( e.g. , mean average precision or accuracy ) , the compositionality gap of a learning task is then simply defined as the difference in performance of the strong and weak oracle when evaluating on concepts fromHtest , i.e. , M ( pstrong ) −M ( pweak ) . Using this notion of compositionality gap , we can then define ideal learners , i.e. , the strong and weak oracle , simply via their priors . In particular , let w ( h ) denote a weight on importance of each hypothesis2 and let I denote the indicator function . We then define the prior of an oracle as pΩ ( h ) = ∑ h′∈HΩ w ( h ′ ) I [ h′ = h ] , . The difference between strong and weak oracle lies in which concepts can be accessed in these priors . In this formalism , the strong oracle has access to the union of train and test concepts ; that is Hstrong = Htrain ∪Htest . The weak oracle , on the other hand only assumes access toHweak = Htrain , which means it is unable to consider any hypothesis outside what has been seen in training and assigning it zero probability mass . Given a support set Dsupp this difference in priors leads then to different inferences on posteriors and allows us to quantify how compositionally novel a learning task is relative to these ideal learners .

Summary: This work proposes the CURI dataset to measure productive concept learning under uncertainty. The dataset is designed using a concept space defined by a language and formulated as a few-shot meta-learning problem to tell apart in-concept samples from out-of-concept samples. The authors also design several out-of-generalization data splits that test models' ood generalization performance. Together with an oracle model, the authors show using the prototypical network that the compositional concept learning and reasoning problem in CURI is challenging.