CURI: A Benchmark for Productive Concept Learning Under Uncertainty

1 INTRODUCTION . Human concept learning is more flexible than today ’ s AI systems . Human conceptual knowledge is productive : people can understand and generate novel concepts via compositions of existing concepts ( “ an apartment dog ” ) ( Murphy , 2002 ) , unlike standard machine classifiers that are limited to a fixed set of classes ( “ dog ” , “ cat ” , etc. ) . Further , humans can induce goal-based , “ ad hoc ” categories such as “ things to take from one ’ s apartment in a fire ” ( children , dogs , keepsakes , etc . ) ( Barsalou , 1983 ) . Thus , unlike AI systems , humans reason seamlessly in large , essentially “ unbounded ” concept spaces . Beyond unboundedness , a natural challenge in such concept spaces is uncertainty – the right concept to be inferred is uncertain , as a plethora of candidate concepts could explain observations . For e.g . in Figure 1 ( top , image panel ) , the “ right ” concept could be that “ All objects are blue and have the same size ” , but it could also be “ There are less than four objects in the scene ” , or “ All objects have the same color ” . Humans gracefully handle such uncertainty and underdetermination ( Tenenbaum & Griffiths , 2001 ; Xu & Tenenbaum , 2007 ; Goodman et al. , 2008 ; Piantadosi et al. , 2016 ) . Popular compositional reasoning benchmarks such as CLEVR ( Johnson & Zhang , 2016 ) for visual question answering and Ravens Progressive Matrices ( Santoro et al. , 2017 ) for deductive , analogical reasoning are compositionally rich and challenging in nature , but do not tackle ambiguity and underdetermination . We address this gap in the literature , and propose the Compositional Reasoning Under Uncertainty ( CURI ) benchmark to study how modern machine learning systems can learn concepts spanning a large , productively defined space ( Figure 1 ) . In pursuit of this goal , we instantiate a meta learning task where a model must acquire a compositional concept from finite samples . A signature of productivity in human thought is our ability to handle novel combinations of known , atomic components . Thus , in CURI we instantiate different systematic train-test splits to analyze different forms of generalization in concept learning , involving novel combinations of intrinsic properties ( e.g . color , shape ) with boolean operators , counting , extrinsic object properties ( e.g . object location ) , and a novel test of variable binding in context of compositional learning . While related systematic splits have been proposed in prior work in context of other tasks such as question answering and analogical reasoning ( Barrett et al. , 2018 ; Hill et al. , 2019 ; Agrawal et al. , 2017 ; Johnson et al. , 2016 ; Vedantam et al. , 2017 ; Higgins et al. , 2017 ; Bakhtin et al. , 2019 ; Lake & Baroni , 2018 ; Ruis et al. , 2020 ) , ours is the first benchmark which tests different qualitative aspects of reasoning about productive concepts under uncertainty . Compositional Reasoning Under Uncertainty ( CURI ) Task . Concretely , the CURI task tests few-shot learning of relational concepts in a large compositional conceptual space , with design inspiration from studies in cognitive modeling using a language of thought ( LOT ) approach ( Fodor , 1975 ; Piantadosi , 2011 ; Kemp et al. , 2005 ) . CURI includes scene-based concepts such as “ All objects have the same color ” and “ There exists a blue object while the rest are triangles ” ( Figure 1 ) but unlike CLEVR ( Johnson et al. , 2016 ) there are too few examples to deduce answers with certainty . Our benchmark is defined through a series of meta-learning episodes ( see example in Figure 2 ) : given positive and negative examples of a new concept Dsupp ( known as the “ support set ” ) , the goal of an episode is to classify new examples Dquery ( the “ query set ” ) . As in few-shot classification ( Fei-Fei et al. , 2006 ) , meta-learning ( Vinyals et al. , 2016 ) , and other open-set tasks ( Lampert et al. , 2014 ) , models are evaluated on novel classes outside the ( meta- ) training set . Unlike previous work ( Triantafillou et al. , 2019 ; Lake et al. , 2019 ) that focuses on atomic concepts , our benchmarks concerns more structured , relational concepts built compositionally from a set of atomic concepts , and involves reasoning under uncertainty – an ideal learner must marginalize over many hypotheses when making predictions ( Gelman et al. , 2004 ; Xu & Tenenbaum , 2007 ; Piantadosi et al. , 2016 ) . We also vary the modality in which scenes are presented—rendering them as images , symbolic schemas , and sounds— enabling future research on modality-specific representational choices for compositional reasoning under uncertainty . Finally , we vary the concepts learned by the model during meta-training and meta-testing to test different aspects of systematic generalization . Compositionality Gap . In addition to defining systematic splits , we also characterize ( for the first time , in our knowledge ) , the difficulty of generalization entailed by each split by introducing the notion of a model-independent “ compositionality gap ” . Concretely , the compositionality gap is the difference in test performance between an ideal Bayesian learner with access to the full hypothesis space , and a Bayesian learner with access to only a ( potentially large ) list of the hypotheses examined during meta-training . A large gap indicates that any learner must extrapolate compositionally from the training hypotheses to solve the task ; additionally , models can be compared to ideal learners that either do or do not engage in such extrapolation . We anticipate that this tool will be more broadly useful for analyzing other benchmarks with compositional splits . Models . We evaluate models around various dimensions which concern the difficulty of learning productive concepts under uncertainty , including : 1 ) the modality in which the input is rendered ( image , schemas , sounds ) , 2 ) method used for reasoning across objects in a scene ( transformer , Variables . Types . Constants ( Illustrated for Images ) . relation-network , global average pooling , concatenation ) , 3 ) whether or not training provides groundtruth symbolic descriptions of concepts , and 4 ) how negative examples are sampled . Overall , our evaluations suggest that there is substantial room for improvement in compositional reasoning under uncertainty , w.r.t the compositionality gap , representing a novel challenge for compositional learning . Summary of contributions : 1 ) We introduce the Compositional Reasoning Under Uncertainty ( CURI ) benchmark for evaluating compositional , relational learning under uncertainty from observational data ; 2 ) We introduce a ‘ compositionality gap ’ metric for measuring the difficulty of systematic generalization from train to test ; 3 ) We provide various baseline models for benchmarking progess . 2 RELATED WORK . Compositional Learning . Related work has examined systematic generalization in pattern completion using Raven ’ s matrices ( PGM ) ( Santoro et al. , 2017 ; Hill et al. , 2019 ) and visual question answering with CLEVR ( Johnson et al. , 2016 ; Bahdanau et al. , 2019 ) . CURI ’ s use of the CLEVR renderer further invites particular comparison with that benchmark . Compared to these more deductive reasoning tests , CURI examines few-shot concept learning under substantial inherent uncertainty . Unlike puzzle solving or question answering , an ideal inductive learner on CURI can not know the right rule with certainty . In essence , unlike CLEVR the “ question ” to be answered is not given to the model as input , but must be inferred – making the task more challenging . While PGMs do involve such an inference , once the constraints of a puzzle are identified , it does not : 1 ) have any uncertainty in the reasoning ( which is crucial ) and 2 ) involve any “ concept ” learning – where a concept applies to multiple images – as much as it involves “ instance ” matching to complete a sequence . In contrast , a successful CURI model behaves as if marginalizing over many hypotheses consistent with the observations e.g. , ( Tenenbaum & Griffiths , 2001 ; Xu & Tenenbaum , 2007 ; Piantadosi et al. , 2016 ) , an ability which is rarely studied directly in deep learning models ( although see ( Grant et al. , 2019 ) ) . Recently , Keysers et al . ( 2019 ) proposed a method to create “ difficult ” systematic splits based on the principle that they should share atoms but have maximally different compositions . This is complementary to our splits , which provide interpretable notions of what each split tests such as disentangling , complexity , variable binding etc . Moreover , our variable binding split is predicated on having different atoms between train and test , and thus can not be recovered by their methodology . Language of Thought ( LOT ) . Our choice of compositional concepts was most closely inspired by ( Piantadosi et al. , 2016 ) along with other studies of human concept learning in the Language of Thought ( LOT ) framework ( Fodor , 1975 ; Goodman et al. , 2008 ; Kemp & Jern , 2009 ; Piantadosi et al. , 2012 ; Goodman et al. , 2015 ; Overlan et al. , 2017 ; Lake & Piantadosi , 2019 ) . In typical LOT studies of human learning , the conceptual space H is defined through a probabilistic context-free grammar G , which specifies a set of conceptual primitives and their rules of combination . Here , we use a LOT-inspired grammar G to generate an unbounded set conceptsH , while evaluating machine learning models trained without access to the underlying LOT . 3 COMPOSITIONAL REASONING UNDER UNCERTAINTY ( CURI ) DATASET . Concept space . The compositional concepts in CURI were inspired by the empirical and cognitive modeling work of Piantadosi et al . ( 2016 ) . The space of concepts ( LOT ) is defined by a context free grammar ( G ) . Figure 3 shows the LOT and specifies how primitives and functions compose to produce a large unbounded concept space . The LOT has three variables : x , representing an object in a scene , S = { x } Ni=1 representing the set of all objects in the scene , and S−x = S/ { x } , representing the set of all objects in the scene except x . Each concept describes a rule composed of object and scene properties , logical operators , and/or comparison operators , and can be evaluated on a given scene S to determine whether the scene satisfies the rule . Object and scene properties are defined by functions which can be applied to objects or scenes : for example , size ? ( x ) yields the size of an object x , while size ? ( S ) returns a set with the sizes of all the objects ( { size ? ( x ) : x ∈ S } ) . Comparison and logical operators can be used to compare and relate various properties of objects in scenes . In contrast to Piantadosi et al . ( 2016 ) , we include a count operator , which determines how many times a condition is satisfied by a set , which allows us to check how well deep learning models are able to count ( Chattopadhyay et al. , 2016 ; Johnson et al. , 2016 ; Agrawal et al. , 2017 ) . Finally , quantifiers such as exists and for-all enrich the LOT by specifying the number of objects which must satisfy a given condition . Consider the following example concept ( Figure 1 bottom ) : “ There exists a blue object in the scene and the rest of the objects are squares. ” To access the color of a given object , we use color ? ( x ) and to access the shape of a given object , we use shape ? ( x ) . To determine whether an object matches a specific property , we can combine this with equality : shape ? ( x ) = “ square ” . Finally , we can use exists to specify that at least one object must be blue , S−x to specify all the objects except for that blue object , and all to specify that all the objects in S−x must be squares . Putting it all together : exists x ∈ S ( color ? ( x ) = “ blue ” ) and all ( shape ? ( S−x ) = “ square ” ) . Structured Generalization Splits . A signature of productivity is the ability to handle novel combinations of known components ( Fodor , 1975 ; Fodor & Pylyshyn , 1988 ) . Thus , in CURI , we consider splits that require generalizing to novel combinations of known elements from our LOT ( Figure 3 ) , including combinations of constants , variables , and functions . We achieve this by creating disjoint splits of concepts Htrain and Htest for training and evaluating models . By varying the held out elements and their combinations , we obtain splits that evaluate different axes of generalization . In practice , we use our grammar G to sample and filter a large set of concepts ( see Appendix B.2 for more details ) , which yields a set of 14,929 conceptsH for training and evaluation . We next describe how each split dividesH intoHtrain andHtest , to test productive , out of distribution generalization : • Instance IID : Evaluates generalization to novel episodes from the same concept set . This is the standard setup in machine learning ( Murphy , 2013 ) , in whichHtrain =Htest . This is the only split where train and test concepts overlap . • Concept IID : Evaluates generalization to novel concepts based on an arbitrary random split of the concepts intoHtrain andHtest.1 • Counting : Evaluates the ability to learn a new concept h with novel property-count combina- tions , e.g , the training concepts never filter for exactly ‘ 3 squares ’ . • Extrinsic properties : Evaluates the ability to learn a new concept h , with novel combinations of extrinsic ( e.g . location ) and intrinsic ( e.g . color ) object properties . • Intrinsic properties : Evaluates the ability to learn a new concept h with novel combinations of intrinsic properties , e.g. , the training concepts never reference both ‘ red ’ and ‘ rubber ’ . • Boolean operations : Evaluates the ability to learn concepts which require application of a familiar boolean operation to a property to which the operation has never been applied previously . • Complexity split : Evaluates generalization from simple concepts ( those which have less than or equal to 10 symbols ) to more complex concepts ( longer than 10 symbols ) . This is indicative of the productivity ( Fodor , 1975 ) exhibited by models , in generalizing from simpler concepts to more complex concepts . • Variable binding : Evaluates learning of entirely novel intrinsic properties , e.g . the training concepts involve only “ red ” , “ blue ” , and “ green ” but test concepts involve “ yellow ” ( although ‘ yellow ’ objects can still appear in training scenes ) . This is indicative of inferential coherence ( Fodor , 1975 ) in models , in generalizing rules of inference to novel atoms . A model that infers the underlying LOT during meta-training would be expected to perform well on any such systematic split . By comparing the performance of current models to to such ideal learners , 1While some strings h might be different in surface form , they may yeild the same results when applied to images . In this split we account for such synonomy , and ensure that no two concepts which are synonyms are in different splits . See Appendix B.6 for more details . this benchmark will allow us to evaluate progress on the systematic out-of-distribution generalization capabilities of our current models . Appendix C provides more details on the strucutred splits . From Concepts to Meta-learning Episodes . A single episode comprises a support set ( Dsupp ) and a query set ( Dquery ) , each of which is generated from a given concept , h. Formally , a support or query set D has input data u and corresponding label y , i.e . D = { { yi } Ni=1 , { ui } Ni=1 } . Each support and query set contains 5 positive and 20 negative examples — negative examples are oversampled since the space of negatives is generally much larger than that for positives . The set of positive examples are sampled uniformly from a categorical distribution over all positives . However , we consider two types of negatives : 1 ) easy negatives , in which the negatives are also sampled at random , and 2 ) hard negatives , in which negatives are generated from a closely related concept which also evaluates true on the positive examples in Dsupp , such that these negatives are maximally confusing . Altogether , for each split , our train , validation , and test sets contain 500000 , 5000 , and 20000 episodes , respectively . Compositionality Gap . A key aspect of our benchmark is to define the difficulty in learning that arises from the compositional structure of the concept space . Most of the splits above are structured in a way such that Htest ∩ Htrain = ∅ – forcing a learner to use the compositional structure of the concept space to generalize toHtest . We conceptualize the difficulty of this task through the notion of its compositionality gap . Intuitively , the compositionality gap captures the difference between the generalization performance of an ideal compositional learner ( strong oracle ) compared to an ideal non-compositional learner that is unable to extrapolate outside the training concepts ( weak oracle ) . Formally , let Ω ∈ { strong , weak } denote an oracle over a concept space HΩ . The posterior predictive distribution of an oracle for query scene u and query label y ∈ { 0 , 1 } is then given as : pΩ ( y|u , Dsupp ) = ∑ h∈HΩ pΩ ( y|h , u ) pΩ ( h|Dsupp ) , where pΩ ( h|Dsupp ) ∝ pΩ ( h ) p ( { yi } Ni=1|h ; { ui } Ni=1 ) and pΩ ( h ) denote the posterior and prior , respectively . Given a metric of interest M ( e.g. , mean average precision or accuracy ) , the compositionality gap of a learning task is then simply defined as the difference in performance of the strong and weak oracle when evaluating on concepts fromHtest , i.e. , M ( pstrong ) −M ( pweak ) . Using this notion of compositionality gap , we can then define ideal learners , i.e. , the strong and weak oracle , simply via their priors . In particular , let w ( h ) denote a weight on importance of each hypothesis2 and let I denote the indicator function . We then define the prior of an oracle as pΩ ( h ) = ∑ h′∈HΩ w ( h ′ ) I [ h′ = h ] , . The difference between strong and weak oracle lies in which concepts can be accessed in these priors . In this formalism , the strong oracle has access to the union of train and test concepts ; that is Hstrong = Htrain ∪Htest . The weak oracle , on the other hand only assumes access toHweak = Htrain , which means it is unable to consider any hypothesis outside what has been seen in training and assigning it zero probability mass . Given a support set Dsupp this difference in priors leads then to different inferences on posteriors and allows us to quantify how compositionally novel a learning task is relative to these ideal learners .

Summary: The following work presents a CLEVR-based compositionality benchmark. The task of the model is to verify logical statements about an image, and in order to achieve such, must learn how to map individual statements to a composition of functions over the image checking for color, placement, shape, etc. Specific to this dataset is that it is explicitly few-shot, which forces the models to generalize very quickly and to infer under uncertainty.