Measuring Compositional Generalization: A Comprehensive Method on Realistic Data

1 INTRODUCTION . Human intelligence exhibits systematic compositionality ( Fodor & Pylyshyn , 1988 ) , the capacity to understand and produce a potentially infinite number of novel combinations of known components , i.e. , to make “ infinite use of finite means ” ( Chomsky , 1965 ) . In the context of learning from a set of training examples , we can observe compositionality as compositional generalization , which we take to mean the ability to systematically generalize to composed test examples of a certain distribution after being exposed to the necessary components during training on a different distribution . Humans demonstrate this ability in many different domains , such as natural language understanding ( NLU ) and visual scene understanding . For example , we can learn the meaning of a new word and then apply it to other language contexts . As Lake & Baroni ( 2018 ) put it : “ Once a person learns the meaning of a new verb ‘ dax ’ , he or she can immediately understand the meaning of ‘ dax twice ’ and ‘ sing and dax ’ . ” Similarly , we can learn a new object shape and then understand its compositions with previously learned colors or materials ( Johnson et al. , 2017 ; Higgins et al. , 2018 ) . In contrast , state-of-the-art machine learning ( ML ) methods often fail to capture the compositional structure that is underlying the problem domain and thus fail to generalize compositionally ( Lake & Baroni , 2018 ; Bastings et al. , 2018 ; Loula et al. , 2018 ; Russin et al. , 2019 ; Johnson et al. , 2017 ) . We believe that part of the reason for this shortcoming is a lack of realistic benchmarks that comprehensively measure this aspect of learning in realistic scenarios . As others have proposed , compositional generalization can be assessed using a train-test split based on observable properties of the examples that intuitively correlate with their underlying compositional structure . Finegan-Dollak et al . ( 2018 ) , for example , propose to test on different output patterns than are in the train set , while Lake & Baroni ( 2018 ) propose , among others , to split examples by output length or to test on examples containing primitives that are rarely shown during training . In this paper , we formalize and generalize this intuition and make these contributions : • We introduce distribution-based compositionality assessment ( DBCA ) , which is a novel method to quantitatively assess the adequacy of a particular dataset split for measuring compositional generalization and to construct splits that are ideally suited for this purpose ( Section 2 ) . • We present the Compositional Freebase Questions ( CFQ ) 1 , a simple yet realistic and large NLU dataset that is specifically designed to measure compositional generalization using the DBCA method , and we describe how to construct such a dataset ( Section 3 ) . • We use the DBCA method to construct a series of experiments for measuring compositionality on CFQ and SCAN ( Lake & Baroni , 2018 ) and to quantitatively compare these experiments to other compositionality experiments ( Section 4 ) . • We analyze the performance of three baseline ML architectures on these experiments and show that these architectures fail to generalize compositionally , and perhaps more surprisingly , that compound divergence between train and test sets is a good predictor of the test accuracy ( Section 5 ) . 2 DISTRIBUTION-BASED COMPOSITIONALITY ASSESSMENT ( DBCA ) . Like other authors , we propose to measure a learner ’ s ability to generalize compositionally by using a setup where the train and test sets come from different distributions . More specifically , we propose a setup where each example is obtained by composing primitive elements ( atoms ) , and where these atoms are similarly represented in the train and test sets while the test set contains novel compounds , i.e. , new ways of composing the atoms of the train set . As a simple illustrative scenario , consider the task of answering simple questions such as “ Who directed Inception ? ” and “ Did Christopher Nolan produce Goldfinger ? ” . In this scenario , the atoms intuitively correspond to the primitive elements that are used to compose those questions , such as the predicates “ direct ( ed ) ” and “ produce ( d ) ” , the question patterns “ Who [ predicate ] [ entity ] ” and “ Did [ entity1 ] [ predicate ] [ entity2 ] ” , and the entities “ Inception ” , “ Christopher Nolan ” , etc . The compounds on the other hand correspond to the combinations of these atoms that appear in the various examples : `` Who directed [ entity ] ? `` , `` Did Christopher Nolan [ predicate ] Inception ? `` , etc . To measure compositional generalization on such a task , one might therefore use the questions “ Who directed Inception ? ” and “ Did Christopher Nolan produce Goldfinger ? ” as training examples while testing on questions such as “ Did Christopher Nolan direct Goldfinger ? ” and `` Who produced Inception ? '' because the atoms are identically represented in the train and test sets while the compounds differ . To make this intuition more precise , we focus on datasets such as CFQ ( introduced in Section 3 ) and SCAN ( Lake & Baroni , 2018 ) , where each example can be created from a formal set of rules by successively applying a number of these rules . In this case , the atoms are the individual rules , while the compounds are the subgraphs of the directed acyclic graphs ( DAGs ) that correspond to the rule applications . ( See Sections 3 and 4 for more details . ) 2.1 PRINCIPLES FOR MEASURING COMPOSITIONALITY . We use the term compositionality experiment to mean a particular way of splitting the data into train and test sets with the goal of measuring compositional generalization . Based on the notions of atoms and compounds described above , we say that an ideal compositionality experiment should adhere to the following two principles : 1 . Similar atom distribution : All atoms present in the test set are also present in the train set , and the distribution of atoms in the train set is as similar as possible to their distribution in the test set . 2 . Different compound distribution : The distribution of compounds in the train set is as different as possible from the distribution in the test set . The second principle guarantees that the experiment is compositionally challenging in the sense that it tests the learner on compounds that are as different as possible from the compounds used during training . The first principle aims to guarantee that the experiment is exclusively measuring the effect of the difference in the way atoms are composed to form compounds ( rather than some related but different property such as domain adaptation on the distribution of the atoms ) . To determine to which degree a certain experiment adheres to these principles , we use the following formalization . For a sample set T , we use FA ( T ) to denote the frequency distribution of atoms 1Available at https : //github.com/google-research/google-research/tree/master/cfq in T and FC ( T ) for the weighted frequency distribution of compounds in T , which correspond to the subgraphs of the rule application DAGs . For practicality , we do not consider all subgraphs of rule application DAGs when computing the compound divergence . Instead , we first generate a large subset G of subgraphs , then weight them in context of their occurrence , and keep only the ones with highest sum of weights . The purpose of the weighting is to avoid double-counting compounds that are highly correlated with some of their super-compounds . We achieve this by calculating the weight of G ∈ G in a sample as w ( G ) = maxg∈occ ( G ) ( 1 − maxG′ : g≺g′∈occ ( G′ ) P ( G′|G ) ) , where occ ( G ) is the set of all occurrences of G in the sample , ≺ denotes the strict subgraph relation , and P ( G′|G ) is the empirical probability of G′ occurring as a supergraph of G over the full sample set . See Appendix L.4 for example subgraphs and more details on the weighting . We measure divergence ( or similarity ) of the weighted distributions using the Chernoff coefficient Cα ( P‖Q ) = ∑ k p α k q 1−α k ∈ [ 0 , 1 ] ( Chung et al. , 1989 ) . For the atom divergence , we use α = 0.5 , which corresponds to the Bhattacharyya coefficient and reflects the desire of making the atom distributions in train and test as similar as possible . For the compound divergence , we use α = 0.1 , which reflects the intuition that it is more important whether a certain compound occurs in P ( train ) than whether the probabilities in P ( train ) and Q ( test ) match exactly . This allows us to formally define as follows the notions of compound divergence DC and atom divergence DA of a compositionality experiment consisting of a train set V and a test set W : DC ( V ‖W ) = 1 − C0.1 ( FC ( V ) ‖FC ( W ) ) DA ( V ‖W ) = 1 − C0.5 ( FA ( V ) ‖FA ( W ) ) Based on these principles , we suggest to use as a preferred compositionality benchmark for a given dataset the accuracy obtained by a learner on splits with maximum compound divergence and low atom divergence ( we use DA ≤ 0.02 ) . See Section 4 for details about how to construct such splits . 3 THE CFQ DATASET . We present the Compositional Freebase Questions ( CFQ ) as an example of how to construct a dataset that is specifically designed to measure compositional generalization using the DBCA method introduced above . CFQ is a simple yet realistic , large dataset of natural language questions and answers that also provides for each question a corresponding SPARQL query against the Freebase knowledge base ( Bollacker et al. , 2008 ) . This means that CFQ can be used for semantic parsing ( Berant et al. , 2013 ; Yao & Van Durme , 2014 ) , which is the task that we focus on in this paper . 3.1 AUTOMATIC , RULE-BASED GENERATION . Saxton et al . ( 2019 ) describe a number of benefits for automated rule-based dataset generation , including scalability , control of scope , and avoidance of human errors . Beyond these benefits , however , such an approach is particularly attractive in the context of measuring compositional generalization using the DBCA method , as it allows us to precisely track the atoms ( rules ) and compounds ( rule applications ) of each example by recording the sequence of rule applications used to generate it . Since the way we measure compositionality depends on how the examples can be broken down into atoms and compounds , we design the generation rules so as to have few and meaningful atoms . More precisely , we aim to have as few rules as possible so that the richness of the examples comes from composing them , which yields a large variety of compounds ( enabling a large range of different compound divergences ) while making it easy to obtain similar distributions of atoms . Also , we aim to make our rules truly “ atomic ” in the sense that the behavior of any rule is independent of the context where it is applied ( e.g. , rules may not contain “ if-then-else ” constructs ) . In order to minimize the number of rules , we use an intermediate logical form that serves as a uniform semantic representation with relatively direct mappings to natural language and SPARQL . Our rules thus fall into the following four categories ( a selection of rules is provided in Appendix M ) : 1 . Grammar rules that generate natural language constructs and corresponding logical forms . 2 . Inference rules that describe transformations on logical forms , allowing us to factor out transformations that are independent of specific linguistic and SPARQL constructs . 3 . Resolution rules that map constructs of the logical form to SPARQL constructs . 4 . Knowledge rules that supply logical form expressions that are universally applicable . Other rules can be kept more generic by parameterizing them on knowledge . These rules define a language of triples of the form 〈question , logical form , SPARQL query〉 . Our generation algorithm produces such triples in a mixed top-down and bottom-up fashion . We first apply grammar rules and inference rules to produce the natural language questions and their semantics in our logical form . Then we apply resolution rules to obtain the SPARQL query . See Figure 1 for an illustration . In addition , the generator produces a normalized , directed acyclic graph ( DAG ) of rule applications that corresponds to the normalized program that generated the triple . ( Appendix L shows an example . ) Edges of this DAG represent dependencies among the rule applications , and the normalization ensures that a certain rule combination is represented using the same DAG across all the examples where it occurs . The described approach can generate a potentially infinite set of questions , from which we first sample randomly and then subsample ( to maximize the overall diversity of rule combinations while keeping a uniform distribution over complexity ) . We measure the diversity of rule combinations using the empirical entropy of a weighted subset of the rule application DAGs , and we use the number of rule applications as a measure of the complexity of an example . We also limit the maximum example complexity such that the questions remain relatively natural . Table 1 shows examples of generated questions at varying levels of complexity . An example of a complete data item is shown in Appendix A , a more detailed data quality analysis is presented in Appendix B , and the generation algorithm is discussed in more detail in Appendix K .

Summary: This paper first introduces a method for quantifying to what extent a dataset split exhibits compound (or, alternatively, atom) divergence, where in particular atoms refer to basic structures used by examples in the datasets, and compounds result from compositional rule application to these atoms. The paper then proposes to evaluate learners on datasets with maximal compound divergence (but minimal atom divergence) between the train and test portions, as a way of testing whether a model exhibits compositional generalization, and suggests a greedy algorithm for forming datasets with this property. In particular, the authors introduce a large automatically generated semantic parsing dataset, which allows for the construction of datasets with these train/test split divergence properties. Finally, the authors evaluate three sequence-to-sequence style semantic parsers on the constructed datasets, and they find that they all generalize very poorly on datasets with maximal compound divergence, and that furthermore the compound divergence appears to be anticorrelated with accuracy.