GraphLog: A Benchmark for Measuring Logical Generalization in Graph Neural Networks

Relational inductive biases have a key role in building learning agents that can generalize and reason in a compositional manner . While relational learning algorithms such as graph neural networks ( GNNs ) show promise , we do not understand their effectiveness to adapt to new tasks . In this work , we study the logical generalization capabilities of GNNs by designing a benchmark suite grounded in first-order logic . Our benchmark suite , GraphLog , requires that learning algorithms perform rule induction in different synthetic logics , represented as knowledge graphs . GraphLog consists of relation prediction tasks on 57 distinct procedurally generated logical worlds . We use GraphLog to evaluate GNNs in three different setups : single-task supervised learning , multi-task ( with pre-training ) , and continual learning . Unlike previous benchmarks , GraphLog enables us to precisely control the logical relationship between the different worlds by controlling the underlying first-order logic rules . We find that models ’ ability to generalize and adapt strongly correlates to the availability of diverse sets of logical rules during multi-task training . We also find the severe catastrophic forgetting effect in continual learning scenarios , and GraphLog provides a precise mechanism to control the distribution shift . Overall , our results highlight new challenges for the design of GNN models , opening up an exciting area of research in generalization using graph-structured data . 1 INTRODUCTION . Relational reasoning , or the ability to reason about the relationship between objects and entities , is considered one of the fundamental aspects of intelligence ( Krawczyk et al. , 2011 ; Halford et al. , 2010 ) , and is known to play a critical role in cognitive growth of children ( Son et al. , 2011 ; Farrington-Flint et al. , 2007 ; Richland et al. , 2010 ) . This ability to infer relations between objects/entities/situations , and to compose relations into higher-order relations , is one of the reasons why humans quickly learn how to solve new tasks ( Holyoak and Morrison , 2012 ; Alexander , 2016 ) . The perceived importance of relational reasoning for generalization has fueled the development of several neural network architectures that incorporate relational inductive biases ( Battaglia et al. , 2016 ; Santoro et al. , 2017 ; Battaglia et al. , 2018 ) . Graph neural networks ( GNNs ) , in particular , have emerged as a dominant computational paradigm within this growing area ( Scarselli et al. , 2008 ; Hamilton et al. , 2017a ; Gilmer et al. , 2017 ; Schlichtkrull et al. , 2018 ; Du et al. , 2019 ) . However , despite the growing interest in GNNs and their promise of relational generalization , we currently lack an understanding of how effectively these models can adapt and generalize across distinct tasks . In this work , we study the task of logical generalization in the context of relational reasoning using GNNs . One example of such a reasoning task ( from everyday life ) can be in the context of a familygraph where the nodes are family members , and edges represent the relationships ( brother , father , etc ) . The objective is to learn logical rules , which are compositions of a specific form , such as “ the son of my son is my grandson ” . As new compositions of existing relations are discovered during the lifetime of a learner , ( e.g. , the son of my daughter is my grandson ) , it should remember the old compositions , even as it learns new compositions ( just like we humans do ) . This simplistic example can be extended to the more complex ( and yet practical ) scenarios like identifying novel chemical compositions , or recommender systems , where agents have to learn and retain compositions of existing relations . We study the effect of such generalization by analyzing the ability of GNNs in learning new relation compositions , leveraging first-order logic . In particular , we study how GNNs can induce logical rules and generalize by combining such rules in novel ways after training . We propose a benchmark suite , GraphLog , that is grounded in first-order logic . Figure 1 shows the setup of the benchmark . Given a set of logical rules , we create a diverse set of logical worlds with a different subset of rules . For each world ( say Wk ) , we sample multiple knowledge graphs ( say gki ) . The learning agent should learn to induce the logical rules for predicting the missing facts in these knowledge graphs . Using our benchmark , we evaluate the generalization capabilities of GNNs in the supervised setting by predicting inductively unseen combinations of known rules within a specific logical world . We further analyze how various GNN architectures perform in the multi-task and continual learning scenarios , where they have to learn over a set of logical worlds with different underlying logics . Our setup allows us to control the distribution shift by controlling the similarity between the different worlds , in terms of the overlap in logical rules between different worlds . Our analysis provides the following insights about logical generalization capabilities of GNNs : • Two architecture choices for GNNs strongly ( and positively ) impact generalization : 1 ) incorporating multirelational edge features using attention , and 2 ) modularising GNN architecture to include a parametric representation function , to learn representations for the relations using a knowledge graph structure . • In the multi-task setting , training a model on a more diverse set of logical worlds improves generalization and adaptation performance . • All the evaluated models exhibit catastrophic forgetting . This indicates that the models are not learning transferable representations and compositions and just overfitting to the current task —highlighting the challenge of lifelong learning in context of logical generalization and GNNs . 2 BACKGROUND AND RELATED WORK . GNNs . Several GNN architectures have been proposed to learn representations of the graph inputs ( Scarselli et al. , 2008 ; Duvenaud et al. , 2015 ; Defferrard et al. , 2016 ; Kipf and Welling , 2016 ; Gilmer et al. , 2017 ; Hamilton et al. , 2017b ; Schlichtkrull et al. , 2018 ) . Previous works have focused on evaluating GNNs in terms of their expressive power ( Barceló et al. , 2019 ; Morris et al. , 2019 ; Xu et al. , 2018 ) , usefulness of features ( Chen et al. , 2019 ) , and explaining their predictions ( Ying et al. , 2019 ) . Complementing these works , we evaluate GNN models on the task of logical generalization . Knowledge graph completion . Many knowledge graph datasets are available for relation prediction tasks ( also known as knowledge base completion ) . Prominent examples include Freebase15K ( Bordes et al. , 2013 ) , WordNet ( Miller , 1995 ) , NELL ( Mitchell and Fredkin , 2014 ) , and YAGO ( Suchanek et al. , 2007 ; Hoffart et al. , 2011 ; Mahdisoltani et al. , 2013 ) . Since these datasets are derived from real-world knowledge graphs , they are generally noisy and incomplete , and many facts are not available in the underlying knowledge bases ( West et al. , 2014 ; Paulheim , 2017 ) . Moreover , the underlying logical rules are often opaque and implicit ( Guo et al. , 2016 ) , thus reducing the utility of these datasets for understanding the logical generalization capability of neural networks . Procedurally generated datasets for reasoning . Several procedurally generated benchmarks have been proposed to study the relational reasoning and compositional generalization properties of neural networks ( Table 1 ) . These datasets provide a controlled testbed for evaluating compositional reasoning capabilities of neural networks in isolation . Based on these insightful works , we enumerate the four key desiderata that , we believe , such a benchmark should provide : 1 . Human interpretable rules should be used to generate the dataset . 2 . The datasets should be diverse , and the compositional rules used to solve different tasks should be distinct , so that adaptation on a new task is not trivial . The degree of similarity across the tasks should be configurable to evaluate the role of diversity in generalization . 3 . The benchmark should test for compositional generalization , which is the ability to solve unseen combinations of existing rules , thus generalizing through the composition of known concepts . 4 . The benchmark should support creating a large number of tasks and enable a more fine-grained inspection of the generalization capabilities of the model in different setups , e.g. , supervised learning , multi-task learning , and continual learning . As shown in Table 1 , GraphLog is unique in satisfying all of these desiderata . We highlight that unlike previous works which have been largely focused on the image and text modalities , GraphLog is one of the unique attempts to test logical generalization on graph-structured data using Graph Neural Networks . CLUTRR ( Sinha et al. , 2019 ) also provides similar underlying graphs in the text corpus , although GraphLog is different based on the following properties : • CLUTRR consists of a single rule world which in turn contains only 15 rules . GraphLog extends the concept to a more general framework where a user can define how many rules can occur in a single world , as well as define multiple such worlds . • GraphLog allows building multiple worlds consisting of either the same , overlapping , or distinct set of rules - which allows practitioners to test multi-task and continual learning scenarios in minute detail by controlling the distribution shift , which is a key difference with CLUTRR . • CLUTRR is built on a static set of relations ( 22 family relations ) while GraphLog can contain any number of such relations since it ’ s a flexible generator along with a dataset . Synthetic Graph Generation . Synthetic graph generation is extensively studied using various neural generative approaches for scale-free random graphs ( You et al. , 2018 ; Liao et al. , 2019 ) . Unlike these approaches , GraphLog is built using a procedural graph generator , which allows us to control the logical validity of the query graphs , which is defined in Section 3 . Control over logical validity is easier in synthetically generated datasets , which are used extensively in Inductive Logic Programming ( ILP ) / Rule Learning context ( Cornelio and Thost , 2019 ) 1 . GraphLog on the other hand , is targeted towards Graph Neural Network evaluation , which also supports for richer distribution shift evaluation through Multi-task learning and Continual learning scenarios . 3 GRAPHLOG . Background and Terminology . We leverage terminology and definitions from both knowledge graph and logic programming literature . A graph G = ( VG , EG ) is a collection of a set of nodes VG and a set of edges EG between the nodes . We assume that the graphs are relational , meaning that each edge between two nodes ( say u and v ) is assigned a label , and can be denoted as ri ( u , v ) ∈ EG . A relation set R is a set of relations { r1 , r2 , ... rK } . 1http : //ilp16.doc.ic.ac.uk/competition In logic programming terms , nodes in the graph correspond to constants , and edges correspond to ground atoms . Thus , as a slight abuse and mixture of notations , existence of an edge in the graph , i.e. , ri ( u , v ) ∈ EG , is equivalent to background assumption that ground atom is true ( i.e. , that ri ( u , v ) ← ) . We define a rule set R as set of dyadic definite Datalog clauses ( Abiteboul et al. , 1995 ) of form : rk ( U , V ) ← ri ( U , Z ) ∧ rj ( Z , V ) . ( 1 ) Note that Equation 1 is also termed as chain-rule in logic literature . Following standard convention , we use upper-case to denote variables ( e.g. , the variables in Equation 1 can be substituted for nodes ) and lower-case for constants ( e.g. , to denote nodes v ∈ VG ) . The relations ri , rj form the body while the relation rk forms the head of the rule . Path-based Horn clauses of this form represent a limited and well-defined subset of first-order logic . They encompass the types of logical rules learned by state-of-the-art rule induction systems for knowledge graph completion ( Das et al. , 2017 ; Evans and Grefenstette , 2017 ; Meilicke et al. , 2018 ; Sadeghian et al. , 2019 ; Teru et al. , 2020 ; Yang et al. , 2017 ; Zhang et al. , 2019 ) and are thus a useful synthetic test-bed . We use pu , vG to denote a path from node u to v in a graph G ( i.e. , a sequence unique nodes connected by edges ) . In logical terms , pu , vG corresponds to the conjunction of all the edge predicates in the path . We construct graphs according to rules of the form in Equation 1 so that a path between two nodes will always imply a specific relation between these two nodes . In other words , if we let PG denote the set of all finite paths in the graph G , then we have that ∀pu , vG ∈ PG ∃ri ∈ R : ( p u , v G → ri ( u , v ) ) ∧ ( ∀rj ∈ R \ { ri } ¬ ( p u , v G ∧ rj ( u , v ) ) ) . ( 2 ) Thus , by following the path between two nodes , and applying rules of the form of Equation 1 according to the edges of the path , we can always resolve the relationship between the nodes . Problem Setup . Given this setup , the task is to predict the relation between a pair of nodes . In particular , we define a query ( g , u , v ) as follows : ( i ) the query subgraph g ⊂ G is a subgraph of the full graph G ; ( ii ) the query nodes are present in the query subgraph ( i.e. , u , v ∈ Vg ) ; ( iii ) the edge between the query nodes is not present in the query subgraph , but the edge is present in the underlying graph ( i.e. , r ? ( u , v ) ∈ EG , r ? ( u , v ) /∈ Eg ) . The goal is to infer the missing edge between query nodes r ? ( u , v ) . We assume that the learning agent has access to a set of training queries g1 , ... , gn ⊂ G to optimize a prediction model before being tested on a set of held-out queries gn+1 , ... , gn+n′ ⊂ G. In knowledge graph terms , this is a form of link or relation prediction , as our goal is to infer the missing edge between two nodes . Unlike most work on knowledge graph completion , we emphasize an inductive problem setup , where the query graph g in each training and testing example is unique , requiring generalization to unseen graphs ( Teru et al. , 2020 ) . Finally , note that we will sometimes refer to the full graphs G as world graphs , in order to distinguish them from the query graphs g ⊂ G. We make this distinction because GraphLog involves link prediction over several different logical worlds , denoted W = ( GW , RW ) , with each defined with by its own underlying world graph GW and rule setRW . Our setup enables controlling the overlap between the rules in these different logical worlds , allowing for a unique ability to quantify the logical generalization capabilities of a learning agent .

Summary: This work proposes a method for generating synthetic datasets for testing path-based (knowledge) graph completion. Until recently, there were not many good benchmarks for evaluating reasoning with learned rules or learned knowledge, but there has been a fair amount of work on developing benchmarks for this lately. The synthetic datasets generated here are distinguished by the ability to produce datasets that share a controllable amount of rules. This permits the benchmarks to be used to evaluate multitask learning, robustness to distribution shift, etc. As an illustration of this, the paper includes experiments with a variety of baseline methods showing how (a) the generalization ability of various methods grows and then declines as the number of tasks is increased and (b) fine-tuning on diverse tasks improves accuracy. They also show that in a continual learning setting, the baseline methods all exhibit catastrophic forgetting.