Query2box: Reasoning over Knowledge Graphs in Vector Space Using Box Embeddings

Answering complex logical queries on large-scale incomplete knowledge graphs ( KGs ) is a fundamental yet challenging task . Recently , a promising approach to this problem has been to embed KG entities as well as the query into a vector space such that entities that answer the query are embedded close to the query . However , prior work models queries as single points in the vector space , which is problematic because a complex query represents a potentially large set of its answer entities , but it is unclear how such a set can be represented as a single point . Furthermore , prior work can only handle queries that use conjunctions ( ∧ ) and existential quantifiers ( ∃ ) . Handling queries with logical disjunctions ( ∨ ) remains an open problem . Here we propose QUERY2BOX , an embedding-based framework for reasoning over arbitrary queries with ∧ , ∨ , and ∃ operators in massive and incomplete KGs . Our main insight is that queries can be embedded as boxes ( i.e. , hyper-rectangles ) , where a set of points inside the box corresponds to a set of answer entities of the query . We show that conjunctions can be naturally represented as intersections of boxes and also prove a negative result that handling disjunctions would require embedding with dimension proportional to the number of KG entities . However , we show that by transforming queries into a Disjunctive Normal Form , QUERY2BOX is capable of handling arbitrary logical queries with ∧ , ∨ , ∃ in a scalable manner . We demonstrate the effectiveness of QUERY2BOX on three large KGs and show that QUERY2BOX achieves up to 25 % relative improvement over the state of the art . 1 INTRODUCTION . Knowledge graphs ( KGs ) capture different types of relationships between entities , e.g. , Canada citizen−−−−→ Hinton . Answering arbitrary logical queries , such as “ where did Canadian citizens with Turing Award graduate ? ” , over such KGs is a fundamental task in question answering , knowledge base reasoning , as well as AI more broadly . First-order logical queries can be represented as Directed Acyclic Graphs ( DAGs ) ( Fig . 1 ( A ) ) and be reasoned according to the DAGs to obtain a set of answers ( Fig . 1 ( C ) ) . While simple and intuitive , such approach has many drawbacks : ( 1 ) Computational complexity of subgraph matching is exponential in the query size , and thus can not scale to modern KGs ; ( 2 ) Subgraph matching is very sensitive as it can not correctly answer queries with missing relations . To remedy ( 2 ) one could impute missing relations ( Koller et al. , 2007 ; Džeroski , 2009 ; De Raedt , 2008 ; Nickel et al. , 2016 ) but that would only make the KG denser , which would further exacerbate issue ( 1 ) ( Dalvi & Suciu , 2007 ; Krompaß et al. , 2014 ) . Recently , a promising alternative approach has emerged , where logical queries as well as KG entities are embedded into a low-dimensional vector space such that entities that answer the query are embedded close to the query ( Guu et al. , 2015 ; Hamilton et al. , 2018 ; Das et al. , 2017 ) . Such approach robustly handles missing relations ( Hamilton et al. , 2018 ) and is also orders of magnitude faster , as answering an arbitrary logical query is reduced to simply identifying entities nearest to the embedding of the query in the vector space . ∗Equal contributions . Project website with data and code : http : //snap.stanford.edu/ query2box However , prior work embeds a query into a single point in the vector space . This is problematic because answering a logical query requires modeling a set of active entities while traversing the KG ( Fig . 1 ( C ) ) , and how to effectively model a set with a single point is unclear . Furthermore , it is also unnatural to define logical operators ( e.g. , set intersection ) of two points in the vector space . Another fundamental limitation of prior work is that it can only handle conjunctive queries , a subset of first-order logic that only involves conjunction ( ∧ ) and existential quantifier ( ∃ ) , but not disjunction ( ∨ ) . It remains an open question how to handle disjunction effectively in the vector space . Here we present QUERY2BOX , an embedding-based framework for reasoning over KGs that is capable of handling arbitrary Existential Positive First-order ( EPFO ) logical queries ( i.e. , queries that include any set of ∧ , ∨ , and ∃ ) in a scalable manner . First , to accurately model a set of entities , our key idea is to use a closed region rather than a single point in the vector space . Specifically , we use a box ( axis-aligned hyper-rectangle ) to represent a query ( Fig . 1 ( D ) ) . This provides three important benefits : ( 1 ) Boxes naturally model sets of entities they enclose ; ( 2 ) Logical operators ( e.g. , set intersection ) can naturally be defined over boxes similarly as in Venn diagrams ( Venn , 1880 ) ; ( 3 ) Executing logical operators over boxes results in new boxes , which means that the operations are closed ; thus , logical reasoning can be efficiently performed in QUERY2BOX by iteratively updating boxes according to the query computation graph ( Fig . 1 ( B ) ( D ) ) . We show that QUERY2BOX can naturally handle conjunctive queries . We first prove a negative result that embedding EPFO queries to only single points or boxes is intractable as it would require embedding dimension proportional to the number of KG entities . However , we provide an elegant solution , where we transform a given EPFO logical query into a Disjunctive Normal Form ( DNF ) ( Davey & Priestley , 2002 ) , i.e. , disjunction of conjunctive queries . Given any EPFO query , QUERY2BOX represents it as a set of individual boxes , where each box is obtained for each conjunctive query in the DNF . We then return nearest neighbor entities to any of the boxes as the answers to the query . This means that to answer any EPFO query we first answer individual conjunctive queries and then take the union of the answer entities . We evaluate QUERY2BOX on three standard KG benchmarks and show : ( 1 ) QUERY2BOX provides strong generalization as it can answer complex queries ; ( 2 ) QUERY2BOX can generalize to new logical query structures that it has never seen during training ; ( 3 ) QUERY2BOX is able to implicitly impute missing relations as it can answer any EPFO query with high accuracy even when relations involving answering the query are missing in the KG ; ( 4 ) QUERY2BOX provides up to 25 % relative improvement in accuracy of answering EPFO queries over state-of-the-art baselines . 2 FURTHER RELATED WORK . Most related to our work are embedding approaches for multi-hop reasoning over KGs ( Bordes et al. , 2013 ; Das et al. , 2017 ; Guu et al. , 2015 ; Hamilton et al. , 2018 ) . Crucial difference is that we provide a way to tractably handle a larger subset of the first-order logic ( EPFO queries vs. conjunctive queries ) and that we embed queries as boxes , which provides better accuracy and generalization . Second line of related work is on structured embeddings , which associate images , words , sentences , or knowledge base concepts with geometric objects such as regions ( Erk , 2009 ; Vilnis et al. , 2018 ; Li et al. , 2019 ) , densities ( Vilnis & McCallum , 2014 ; He et al. , 2015 ; Athiwaratkun & Wilson , 2018 ) , and orderings ( Vendrov et al. , 2016 ; Lai & Hockenmaier , 2017 ; Li et al. , 2017 ) . While the above work uses geometric objects to model individual entities and their pairwise relations , we use the geometric objects to model sets of entities and reason over those sets . In this sense our work is also related to classical Venn Diagrams ( Venn , 1880 ) , where boxes are essentially the Venn Diagrams in vector space , but our boxes and entity embeddings are jointly learned , which allows us to reason over incomplete KGs . Box embeddings have also been used to model hierarchical nature of concepts in an ontology with uncertainty ( Vilnis et al. , 2018 ; Li et al. , 2019 ) . While our work is also based on box embeddings we employ them for logical reasoning in massive heterogeneous knowledge graphs . 3 QUERY2BOX : LOGICAL REASONING OVER KGS IN VECTOR SPACE . Here we present the QUERY2BOX , where we will define an objective function that allows us to learn embeddings of entities in the KG , and at the same time also learn parameterized geometric logical operators over boxes . Then given an arbitrary EPFO query q ( Fig . 1 ( A ) ) , we will identify its computation graph ( Fig . 1 ( B ) ) , and embed the query by executing a set of geometric operators over boxes ( Fig . 1 ( D ) ) . Entities that are enclosed in the final box embedding are returned as answers to the query ( Fig . 1 ( D ) ) . In order to train our system , we generate a set of queries together with their answers at training time and then learn entity embeddings and geometric operators such that queries can be accurately answered . We show in the following sections that our approach is able to generalize to queries and logical structures never seen during training . Furthermore , as we show in experiments , our approach is able to implicitly impute missing relations and answer queries that would be impossible to answer with traditional graph traversal methods . In the following we first only consider conjunctive queries ( conjunction and existential operator ) and then we extend our method to also include disjunction . 3.1 KNOWLEDGE GRAPHS AND CONJUNCTIVE QUERIES . We denote a KG as G = ( V , R ) , where v ∈ V represents an entity , and r ∈ R is a binary function r : V × V → { True , False } , indicating whether the relation r holds between a pair of entities or not . In the KG , such binary output indicates the existence of the directed edge between a pair of entities , i.e. , v r−→ v′ iff r ( v , v′ ) = True . Conjunctive queries are a subclass of the first-order logical queries that use existential ( ∃ ) and conjunction ( ∧ ) operations . They are formally defined as follows . q [ V ? ] = V ? . ∃V1 , . . . , Vk : e1 ∧ e2 ∧ ... ∧ en , ( 1 ) where ei = r ( va , V ) , V ∈ { V ? , V1 , . . . , Vk } , va ∈ V , r ∈ R , or ei = r ( V , V ′ ) , V , V ′ ∈ { V ? , V1 , . . . , Vk } , V 6= V ′ , r ∈ R , where va represents non-variable anchor entity , V1 , . . . , Vk are existentially quantified bound variables , V ? is the target variable . The goal of answering the logical query q is to find a set of entities JqK ⊆ V such that v ∈ JqK iff q [ v ] = True . We call JqK the denotation set ( i.e. , answer set ) of query q . As shown in Fig . 1 ( A ) , the dependency graph is a graphical representation of conjunctive query q , where nodes correspond to variable or non-variable entities in q and edges correspond to relations in q . In order for the query to be valid , the corresponding dependency graph needs to be a Directed Acyclic Graph ( DAG ) , with the anchor entities as the source nodes of the DAG and the query target V ? as the unique sink node ( Hamilton et al. , 2018 ) . From the dependency graph of query q , one can also derive the computation graph , which consists of two types of directed edges that represent operators over sets of entities : • Projection : Given a set of entities S ⊆ V , and relation r ∈ R , this operator obtains ∪v∈SAr ( v ) , where Ar ( v ) ≡ { v′ ∈ V : r ( v , v′ ) = True } . • Intersection : Given a set of entity sets { S1 , S2 , . . . , Sn } , this operator obtains ∩ni=1Si . For a given query q , the computation graph specifies the procedure of reasoning to obtain a set of answer entities , i.e. , starting from a set of anchor nodes , the above two operators are applied iteratively until the unique sink target node is reached . The entire procedure is analogous to traversing KGs following the computation graph ( Guu et al. , 2015 ) .

Summary: This paper proposes a method to answer complex logical queries in large incomplete knowledge bases (KB). Specifically it considers the class of existential first-order logical queries (EPFO) which includes the logical and, or and existential operator.  The key contribution of this paper is to represent sets of entities via regions, more specifically as boxes or hyper-rectangles. This is well motivated because such logical queries often involves working over sets of entities at once and involves applying set based operators. Previous work which represented queries as a point in vector space are not well suited for these queries.