ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs

1 Introduction . Multi-hop reasoning over knowledge graphs ( KGs ) —which aims to find answer entities of given queries using knowledge from KGs—has attracted great attention from both academia and industry recently [ 24 , 23 , 16 ] . In general , it involves answering first-order logic ( FOL ) queries over KGs using operators including existential quantification ( ∃ ) , conjunction ( ∧ ) , disjunction ( ∨ ) , and negation ( ¬ ) . A popular approach to multi-hop reasoning over KGs is to first transform a FOL query to its corresponding computation graph—where each node represents a set of entities and each edge represents a logical operation—and then traverse the KG according to the computation graph to identify the answer set . However , this approach confronts two major challenges . First , when some links are missing in KGs , it has difficulties in identifying the correct answers . Second , it needs to deal with all the intermediate entities on reasoning paths , which may lead to exponential computation cost . To address these challenges , researchers have paid increasing attention to the query embedding ( QE ) technique , which embeds entities and FOL queries in low-dimensional spaces [ 12 , 22 , 21 , 25 ] . QE models associate each logical operator in computation graphs with a logical operation in embedding ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . spaces . Given a query , QE models generate query embeddings following the corresponding computation graph . Then , they determine whether an entity is a correct answer based on similarities between the query embeddings and entity embeddings . Among the existing QE models , geometry-based models that embed entities and queries into geometric shapes have shown promising performance [ 12 , 11 , 5 , 22 ] . Geometry-based models usually represent entity sets as `` regions '' ( e.g. , points and boxes ) in Euclidean spaces and then design set operations upon them . For example , Query2Box [ 22 ] represents entities as points and queries as boxes . If a point is inside a box , then the corresponding entity is the answer to the query . Compared with non-geometric methods , geometric shapes provide a natural and easily interpretable way to represent sets and logical relationships among them . However , existing geometry-based models have difficulty in modeling queries with negations , which significantly limits their applicability . For example , GQE [ 12 ] and Query2Box [ 22 ] —which embed queries to points and boxes , respectively— can not handle queries with negation , as the complement of a point/box is no longer a point/box . To tackle this problem , Ren & Leskovec [ 21 ] propose a probabilistic QE model using Beta distributions . However , it does not have some advantages of geometric models . For example , using Beta distributions , it is unclear how to determine whether an entity is an answer to a query as that in the box case [ 22 ] . Therefore , proposing a geometric QE model that can model all the FOL queries is still challenging but promising . In this paper , we propose a novel geometry-based query embedding model—namely , Cone Embeddings ( ConE ) —which represents entities and queries as Cartesian products of two-dimensional cones . Specifically , if the cones representing entities are subsets of the cones representing queries , then these entities are the answers to the query . To perform multi-hop reasoning in the embedding space , we define the conjunction and disjunction operations that correspond to the intersection and union of cones . Further , by noticing that the closure of complement of cones are still cones , we correspondingly design geometric complement operators in the embedding space for the negation operations . To the best of our knowledge , ConE is the first geometry-based QE model that can handle all the FOL operations , including conjunction , disjunction , and negation . Experiments demonstrate that ConE significantly outperforms existing state-of-the-art methods on benchmark datasets . 2 Related Work . Our work is related to answering multi-hop logical queries over KGs and geometric embeddings . Answering multi-hop logical queries over KGs . To answer multi-hop FOL queries , path-based methods [ 30 , 17 , 10 ] start from anchor entities and require traversing the intermediate entities on the path , which leads to exponential computation cost . Embedding-based models are another line of works , which embed FOL queries into low-dimensional spaces . For example , existing works embed queries to geometric shapes [ 12 , 22 , 11 , 5 ] , probability distributions [ 21 ] , and complex objects [ 9 , 25 ] . Our work also embeds queries to geometric shapes . The main difference is that our work can handle all the FOL operations , while existing works can not . Other Geometric embeddings . Geometric embeddings are popular in recent years . For example , geometric operations including translation [ 3 , 18 ] , rotation [ 28 , 26 , 14 , 32 ] , and complex geometric operations [ 1 , 33 ] have been widely used in knowledge graph embeddings . Other geometric embedding methods also manage to use boxes [ 6 ] , convex cones [ 15 , 29 , 19 ] , etc . For example , Lütfü Özçep et al . [ 19 ] use axis-aligned cones to embed ontologies expressed in the ALC description logic , and use polars of cones to model negation operators . Recent years have also witnessed the development of embeddings in non-Euclidean geometry , such as Poincaré embeddings [ 1 , 20 ] and hyperbolic entailment cones [ 8 ] . Notably , although there exist works that also use cone embeddings [ 15 , 29 , 8 , 19 ] , they are not designed for the multi-hop reasoning task and their definition of cones are different from that in our work . 3 Preliminaries . In this section , we review the background of query embeddings in Section 3.1 and introduce some basic concepts of two-dimensional cones in Section 3.2 . 3.1 Backgrounds . Knowledge Graphs ( KGs ) . Given a set V of entities ( vertices ) and a set E of relations ( edges ) , a knowledge graph G = { ( si , pj , ok ) } ⊂ V × E × V is a set of factual triple , where pj ∈ E is a predicate , and si , ok ∈ V are subject and object , respectively . Suppose that rj ( · , · ) ∈ R is a binary function rj : V × V → { True , False } corresponding to pj , where rj ( si , ok ) = True if and only if ( si , pj , ok ) is a factual triples . Then , for all ( si , pj , ok ) ∈ G , we have rj ( si , ok ) = True . Note that both E and R are involved with relations , while E is a set of relation instances and R is a set of relational functions . First-Order Logic ( FOL ) . FOL queries in the query embedding literature involve logical operations including existential quantification ( ∃ ) , conjunction ( ∧ ) , disjunction ( ∨ ) , and negation ( ¬ ) . Universal quantification ( ∀ ) is not included , as no entity connects with all other entities in real-world KGs [ 21 ] . We use FOL queries in its Disjunctive Normal Form ( DNF ) [ 7 ] , which represents FOL queries as a disjunction of conjunctions . To formulate FOL queries , we assume that Va ⊂ V is the non-variable anchor entity set , V1 , . . . , Vk are existentially quantified bound variables , and V ? is the target variable , i.e. , the answers to a certain query . Then , a FOL query q in the disjunctive normal form is : q [ V ? ] = V ? . ∃V1 , . . . , Vk : c1 ∨ c2 ∨ · · · ∨ cn . Specifically , ci are conjunctions , i.e. , ci = ei1 ∧ · · · ∧ eim , where eij = r ( va , V ) or ¬r ( va , V ) or r ( V ′ , V ) or ¬r ( V ′ , V ) , va ∈ Va , V ∈ { V ? , V1 , . . . , Vk } , V ′ ∈ { V1 , . . . , Vk } , and V ̸= V ′ . Using the aforementioned notations , answering a query q is equivalent to finding the set of entities JqK ⊂ V , where v ∈ JqK if and only if q [ v ] is True . Computation Graphs . Given a query , we represent the reasoning procedure as a computation graph ( see Figure 1a for an example ) , of which nodes represent entity sets and edges represent logical operations over entity sets . We map edges to logical operators according to the following rules . • Relation Traversal→Projection Operator P . Given a set of entities S ⊂ V and a relational function r ∈ R , the projection operator P outputs all the adjacent entities ∪v∈SN ( v , r ) , where N ( v , r ) is the set of entities such that r ( v , v′ ) = True for all v′ ∈ N ( v , r ) . • Conjunction→Intersection Operator I . Given n sets of entities { S1 , S2 , . . . , Sn } , the intersection operator I performs set intersection to obtain ∩ni=1Sn . • Disjunction→Union Operator U . Given n sets of entities { S1 , S2 , . . . , Sn } , the union operator U performs set union to obtain ∪ni=1Sn . • Negation→Complement Operator C. Given an entity set S ⊂ V , C gives S̄ = V\S . Query Embeddings ( QE ) . QE models generate low-dimensional continuous embeddings for queries and entities , and associate each logical operator for entity sets with an operation in embedding spaces . Since an entity is equivalent to a set with a single element and each query q is corresponding to a unique answer set JqK , the aim of QE models is equivalent to embedding entity sets that can be answers to some queries . 3.2 Cones in Two-Dimensional Spaces . To represent FOL queries as Cartesian products of two-dimensional cones , we introduce some definitions about cones and the parameterization method of a special class of cones . Definition 1 ( Boyd & Vandenberghe [ 4 ] ) . A set C ⊂ R2 is called a cone , if for every x ∈ C and λ ≥ 0 , we have λx ∈ C . A set is a convex cone if it is convex and a cone , which means that for any x1 , x2 ∈ C and λ1 , λ2 ≥ 0 , we have λ1x1 + λ2x2 ∈ C. By letting λ = 0 in Definition 1 , we know that a cone must contain the origin . In view of this property , we define a new operation called closure-complement for cones . Definition 2 . Suppose that C ⊂ R2 is a cone . Then , the closure-complement of C is defined by C̃ = cl ( R2\C ) , where cl ( · ) is the closure of a set . Next , we introduce a class of cones that can be parameterized in a scalable way . Definition 3 . A 2D closed cone is called a sector-cone , if its closure-complement or itself is convex . The set of sector-cones is closed under closure-complement and their union and intersection are still cones . Besides , we have the following proposition , whose proof is provided in Appendix A . Proposition 1 . A sector-cone is always axially symmetric . Parameterization of 2D Sector-Cones . Proposition 1 suggests that we can use a pair of parameters to represent a two-dimensional sector-cone : ( θax , θap ) , where θax ∈ [ −π , π ) , θap ∈ [ 0 , 2π ] . Specifically , θax represents the angle between the symmetry axis of the sector-cone and the positive x axis . θap represents the aperture of the sector-cone . For any points in the cone , its phase will be in [ θax − θap/2 , θax + θap/2 ] . Figure 1b gives examples of several ( sector- ) cones . One may notice that sector-cones share some similarities with boxes defined in Query2Box , which also involves region representations . However , we argue that sector-cones are more expressive than boxes , of which the details are provided in Appendix F. Let K be the space consisting of all ( θax , θap ) . We can represent an arbitrary sector-cone C0 as C0 = ( θax , θap ) ∈ K. Then , for a d-ary Cartesian product of sector-cones C = C1 × C2 × . . .× Cd , ( 1 ) we represent it via a d-dimensional vector in Kd : C = ( ( θ1ax , θ 1 ap ) , . . . , ( θ d ax , θ d ap ) ) ⊂ Kd . ( 2 ) where θiax ∈ [ −π , π ) , θiap ∈ [ 0 , 2π ] , for i = 1 , . . . , d. Or equivalently , C = ( θax , θap ) , where θax = ( θ 1 ax , . . . , θ d ax ) ∈ [ −π , π ) d and θap = ( θ1ap , . . . , θdap ) ∈ [ 0 , 2π ] d .

Summary: The paper proposes a new framework for embedding first-order logic (FOL) queries over knowledge bases using region-based embeddings. The embedding space is a Cartesian product of 2D cones. Entities are embedded in the same space with 0-aperture cones (ray starting at origin).  - Authors provide the relevant foundation to cones in 2D space, the final Cartesian product space, and suggest parameterizations of projection, intersection, union, and complement operations. - The choice of using cones facilitates complement operation. - Authors demonstrate empirical improvements over past work in FOL query embedding. - While the proposed cone-based region embeddings allow for clear descriptions for intersection, union, and complement, the implementation does not strictly enforce these geometric concepts.