ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs

1 Introduction . Multi-hop reasoning over knowledge graphs ( KGs ) —which aims to find answer entities of given queries using knowledge from KGs—has attracted great attention from both academia and industry recently [ 24 , 23 , 16 ] . In general , it involves answering first-order logic ( FOL ) queries over KGs using operators including existential quantification ( ∃ ) , conjunction ( ∧ ) , disjunction ( ∨ ) , and negation ( ¬ ) . A popular approach to multi-hop reasoning over KGs is to first transform a FOL query to its corresponding computation graph—where each node represents a set of entities and each edge represents a logical operation—and then traverse the KG according to the computation graph to identify the answer set . However , this approach confronts two major challenges . First , when some links are missing in KGs , it has difficulties in identifying the correct answers . Second , it needs to deal with all the intermediate entities on reasoning paths , which may lead to exponential computation cost . To address these challenges , researchers have paid increasing attention to the query embedding ( QE ) technique , which embeds entities and FOL queries in low-dimensional spaces [ 12 , 22 , 21 , 25 ] . QE models associate each logical operator in computation graphs with a logical operation in embedding ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . spaces . Given a query , QE models generate query embeddings following the corresponding computation graph . Then , they determine whether an entity is a correct answer based on similarities between the query embeddings and entity embeddings . Among the existing QE models , geometry-based models that embed entities and queries into geometric shapes have shown promising performance [ 12 , 11 , 5 , 22 ] . Geometry-based models usually represent entity sets as `` regions '' ( e.g. , points and boxes ) in Euclidean spaces and then design set operations upon them . For example , Query2Box [ 22 ] represents entities as points and queries as boxes . If a point is inside a box , then the corresponding entity is the answer to the query . Compared with non-geometric methods , geometric shapes provide a natural and easily interpretable way to represent sets and logical relationships among them . However , existing geometry-based models have difficulty in modeling queries with negations , which significantly limits their applicability . For example , GQE [ 12 ] and Query2Box [ 22 ] —which embed queries to points and boxes , respectively— can not handle queries with negation , as the complement of a point/box is no longer a point/box . To tackle this problem , Ren & Leskovec [ 21 ] propose a probabilistic QE model using Beta distributions . However , it does not have some advantages of geometric models . For example , using Beta distributions , it is unclear how to determine whether an entity is an answer to a query as that in the box case [ 22 ] . Therefore , proposing a geometric QE model that can model all the FOL queries is still challenging but promising . In this paper , we propose a novel geometry-based query embedding model—namely , Cone Embeddings ( ConE ) —which represents entities and queries as Cartesian products of two-dimensional cones . Specifically , if the cones representing entities are subsets of the cones representing queries , then these entities are the answers to the query . To perform multi-hop reasoning in the embedding space , we define the conjunction and disjunction operations that correspond to the intersection and union of cones . Further , by noticing that the closure of complement of cones are still cones , we correspondingly design geometric complement operators in the embedding space for the negation operations . To the best of our knowledge , ConE is the first geometry-based QE model that can handle all the FOL operations , including conjunction , disjunction , and negation . Experiments demonstrate that ConE significantly outperforms existing state-of-the-art methods on benchmark datasets . 2 Related Work . Our work is related to answering multi-hop logical queries over KGs and geometric embeddings . Answering multi-hop logical queries over KGs . To answer multi-hop FOL queries , path-based methods [ 30 , 17 , 10 ] start from anchor entities and require traversing the intermediate entities on the path , which leads to exponential computation cost . Embedding-based models are another line of works , which embed FOL queries into low-dimensional spaces . For example , existing works embed queries to geometric shapes [ 12 , 22 , 11 , 5 ] , probability distributions [ 21 ] , and complex objects [ 9 , 25 ] . Our work also embeds queries to geometric shapes . The main difference is that our work can handle all the FOL operations , while existing works can not . Other Geometric embeddings . Geometric embeddings are popular in recent years . For example , geometric operations including translation [ 3 , 18 ] , rotation [ 28 , 26 , 14 , 32 ] , and complex geometric operations [ 1 , 33 ] have been widely used in knowledge graph embeddings . Other geometric embedding methods also manage to use boxes [ 6 ] , convex cones [ 15 , 29 , 19 ] , etc . For example , Lütfü Özçep et al . [ 19 ] use axis-aligned cones to embed ontologies expressed in the ALC description logic , and use polars of cones to model negation operators . Recent years have also witnessed the development of embeddings in non-Euclidean geometry , such as Poincaré embeddings [ 1 , 20 ] and hyperbolic entailment cones [ 8 ] . Notably , although there exist works that also use cone embeddings [ 15 , 29 , 8 , 19 ] , they are not designed for the multi-hop reasoning task and their definition of cones are different from that in our work . 3 Preliminaries . In this section , we review the background of query embeddings in Section 3.1 and introduce some basic concepts of two-dimensional cones in Section 3.2 . 3.1 Backgrounds . Knowledge Graphs ( KGs ) . Given a set V of entities ( vertices ) and a set E of relations ( edges ) , a knowledge graph G = { ( si , pj , ok ) } ⊂ V × E × V is a set of factual triple , where pj ∈ E is a predicate , and si , ok ∈ V are subject and object , respectively . Suppose that rj ( · , · ) ∈ R is a binary function rj : V × V → { True , False } corresponding to pj , where rj ( si , ok ) = True if and only if ( si , pj , ok ) is a factual triples . Then , for all ( si , pj , ok ) ∈ G , we have rj ( si , ok ) = True . Note that both E and R are involved with relations , while E is a set of relation instances and R is a set of relational functions . First-Order Logic ( FOL ) . FOL queries in the query embedding literature involve logical operations including existential quantification ( ∃ ) , conjunction ( ∧ ) , disjunction ( ∨ ) , and negation ( ¬ ) . Universal quantification ( ∀ ) is not included , as no entity connects with all other entities in real-world KGs [ 21 ] . We use FOL queries in its Disjunctive Normal Form ( DNF ) [ 7 ] , which represents FOL queries as a disjunction of conjunctions . To formulate FOL queries , we assume that Va ⊂ V is the non-variable anchor entity set , V1 , . . . , Vk are existentially quantified bound variables , and V ? is the target variable , i.e. , the answers to a certain query . Then , a FOL query q in the disjunctive normal form is : q [ V ? ] = V ? . ∃V1 , . . . , Vk : c1 ∨ c2 ∨ · · · ∨ cn . Specifically , ci are conjunctions , i.e. , ci = ei1 ∧ · · · ∧ eim , where eij = r ( va , V ) or ¬r ( va , V ) or r ( V ′ , V ) or ¬r ( V ′ , V ) , va ∈ Va , V ∈ { V ? , V1 , . . . , Vk } , V ′ ∈ { V1 , . . . , Vk } , and V ̸= V ′ . Using the aforementioned notations , answering a query q is equivalent to finding the set of entities JqK ⊂ V , where v ∈ JqK if and only if q [ v ] is True . Computation Graphs . Given a query , we represent the reasoning procedure as a computation graph ( see Figure 1a for an example ) , of which nodes represent entity sets and edges represent logical operations over entity sets . We map edges to logical operators according to the following rules . • Relation Traversal→Projection Operator P . Given a set of entities S ⊂ V and a relational function r ∈ R , the projection operator P outputs all the adjacent entities ∪v∈SN ( v , r ) , where N ( v , r ) is the set of entities such that r ( v , v′ ) = True for all v′ ∈ N ( v , r ) . • Conjunction→Intersection Operator I . Given n sets of entities { S1 , S2 , . . . , Sn } , the intersection operator I performs set intersection to obtain ∩ni=1Sn . • Disjunction→Union Operator U . Given n sets of entities { S1 , S2 , . . . , Sn } , the union operator U performs set union to obtain ∪ni=1Sn . • Negation→Complement Operator C. Given an entity set S ⊂ V , C gives S̄ = V\S . Query Embeddings ( QE ) . QE models generate low-dimensional continuous embeddings for queries and entities , and associate each logical operator for entity sets with an operation in embedding spaces . Since an entity is equivalent to a set with a single element and each query q is corresponding to a unique answer set JqK , the aim of QE models is equivalent to embedding entity sets that can be answers to some queries . 3.2 Cones in Two-Dimensional Spaces . To represent FOL queries as Cartesian products of two-dimensional cones , we introduce some definitions about cones and the parameterization method of a special class of cones . Definition 1 ( Boyd & Vandenberghe [ 4 ] ) . A set C ⊂ R2 is called a cone , if for every x ∈ C and λ ≥ 0 , we have λx ∈ C . A set is a convex cone if it is convex and a cone , which means that for any x1 , x2 ∈ C and λ1 , λ2 ≥ 0 , we have λ1x1 + λ2x2 ∈ C. By letting λ = 0 in Definition 1 , we know that a cone must contain the origin . In view of this property , we define a new operation called closure-complement for cones . Definition 2 . Suppose that C ⊂ R2 is a cone . Then , the closure-complement of C is defined by C̃ = cl ( R2\C ) , where cl ( · ) is the closure of a set . Next , we introduce a class of cones that can be parameterized in a scalable way . Definition 3 . A 2D closed cone is called a sector-cone , if its closure-complement or itself is convex . The set of sector-cones is closed under closure-complement and their union and intersection are still cones . Besides , we have the following proposition , whose proof is provided in Appendix A . Proposition 1 . A sector-cone is always axially symmetric . Parameterization of 2D Sector-Cones . Proposition 1 suggests that we can use a pair of parameters to represent a two-dimensional sector-cone : ( θax , θap ) , where θax ∈ [ −π , π ) , θap ∈ [ 0 , 2π ] . Specifically , θax represents the angle between the symmetry axis of the sector-cone and the positive x axis . θap represents the aperture of the sector-cone . For any points in the cone , its phase will be in [ θax − θap/2 , θax + θap/2 ] . Figure 1b gives examples of several ( sector- ) cones . One may notice that sector-cones share some similarities with boxes defined in Query2Box , which also involves region representations . However , we argue that sector-cones are more expressive than boxes , of which the details are provided in Appendix F. Let K be the space consisting of all ( θax , θap ) . We can represent an arbitrary sector-cone C0 as C0 = ( θax , θap ) ∈ K. Then , for a d-ary Cartesian product of sector-cones C = C1 × C2 × . . .× Cd , ( 1 ) we represent it via a d-dimensional vector in Kd : C = ( ( θ1ax , θ 1 ap ) , . . . , ( θ d ax , θ d ap ) ) ⊂ Kd . ( 2 ) where θiax ∈ [ −π , π ) , θiap ∈ [ 0 , 2π ] , for i = 1 , . . . , d. Or equivalently , C = ( θax , θap ) , where θax = ( θ 1 ax , . . . , θ d ax ) ∈ [ −π , π ) d and θap = ( θ1ap , . . . , θdap ) ∈ [ 0 , 2π ] d .

Summary: This paper represents a novel approach to learning embeddings for entities and relations from a knowledge base that are capable of directly encoding *reasoning* (i.e. FOL). The principal representation proposed for this representation is a *cone* (i.e. a set which is closed under multiplication by a positive scalar) which is motivated by the fact that cones are closed under intersection and complements. In practice, the authors use *sector cones*, which leads to some incongruity between the original motivation and the model in practice (eg. sector cones are not closed under intersection or union). Even so, the authors apply this embedding method to several standard logic embedding tasks and demonstrate significant performance improvements.