Regex Queries over Incomplete Knowledge Bases

1 . Introduction . Knowledge Base Completion ( KBC ) predicts unseen facts by reasoning over known facts in an incomplete KB . Embedding-based models are a popular approach for the task – they embed entities and relations in a vector space , and use a scoring function to evaluate validity of any potential fact . KBC models are typically evaluated using single-hop queries , example , “ Who founded Microsoft ? ” , which is equivalent to the query ( Microsoft→ founded by → ? ) . A natural extension [ Guu et al. , 2015 ] poses multi-hop queries , such as ( Microsoft → founded by → · → lives in→ ? ) , which represents “ Where do founders of Microsoft live ? ” . Recent works have also studied subsets of first-order logic queries with conjunction ( ∧ ) , disjunction ( ∨ ) and existential quantifiers ( ∃ ) [ Hamilton et al. , 2018 , Ren et al. , 2020 ] . We analyzed real-world query logs1 over the publicly available Wikidata KB2 and extract its distribution of queries . We found ( see Table 1 ) that while the single hop queries are indeed most frequent ( 87 % ) , multi-hop queries comprise only 1 % of all queries . A substantial fraction of queries ( 12 % ) are queries using regular expression operators ( regex ) , which use disjunction ( ∨ ) or Kleene plus ( + ) operators . In response , we pose the novel problem of answering regex queries using embedding-based KBC models . ∗Work done as a research intern at IIT Delhi 1https : //iccl.inf.tu-dresden.de/web/Wikidata_SPARQL_Logs/en 2https : //www.wikidata.org/wiki/Wikidata : Main_Page Regex queries have a potentially large set of correct answers , due to which typical embedding models that model both queries and entities as points are not very effective . We build on one such model RotatE , which models a relation as a rotation of entity vectors – it can model all relation patterns gen- erally observed in a KB except hierarchy and has not been extended to queries beyond single-hop . There also exists Query2Box [ Hamilton et al. , 2018 ] , a box embedding method which models a query as a high-dimensional axis-aligned box . This can better handle larger answer spaces , but it can not model symmetric relations ( e.g . friend of ) . To answer the full repertoire of regex queries over incomplete KBs , we propose a new baseline , RotatEBox – a RotatE model augmented with box embeddings . In our model , a regex query is an axis-aligned box . Application of a relation or regex expression rotates the center ( vector ) of the box , and enlarges/shrinks the box . Application of a regex operator is handled using two approaches : ( 1 ) learned operators over boxes , or ( 2 ) introducing separate embeddings for Kleene plus of a relation , and expanding a disjunction into multiple non-disjunctive queries . RotatE-Box can capture all typical relation patterns – symmetry , antisymmetry , inversion , composition , intersection , mutual exclusion and hierarchy , along with effectively handling a large number of correct answers . We also contribute two new datasets for our task . Our first dataset , called FB15K-Regex , is an extension of the popular FB15K dataset [ Bordes et al. , 2013 ] , and is constructed using random walks on the FB15K graph . Our second dataset is built over a dense subset of Wikidata ( which we name Wiki100 ) and is based on the real queries from existing query logs . Table 2 lists example queries in the datasets . We find that models based on RotatEBox outperform all other baselines , including strong models based on just RotatE and just Query2Box on these two datasets . In summary , our work presents a novel regex query answering task , two new datasets and a strong baseline for our task . We will release all resources for further research3 . 2 . Related Work . A KB G is a set of triples ( e1 , r , e2 ) , where e1 , e2 ∈ E ( the set of entities ) and r ∈ R ( the set of relations ) . For a KBC task , all triples are split into train/dev/test , resulting in graphs Gtrain , Gdev , and Gtest . The standard KBC task requires answering single-hop queries of the form ( e1 , r , ? ) with the answer e ∗ 2 if ( e1 , r , e ∗ 2 ) ∈ Gtest . While many kinds of solutions exist for this problem , we build on the vast literature in embedding-based approaches [ Bordes 3https : //github.com/dair-iitd/kbi-regex et al. , 2013 , Socher et al. , 2013 , Yang et al. , 2014 , Trouillon et al. , 2016 , Jain et al. , 2018 , Sun et al. , 2019 , Jain et al. , 2020 ] . These methods embed each entity and relation in a latent embedding space , and use various tensor factorization scoring functions to estimate the validity of a triple . Our work builds on RotatE , which defines relation as a rotation in a complex vector space . The distance of target entity from the query constitutes the scoring function . RotatE has shown high KBC performance and can model many relation patterns ( see Section 2.1 ) . The first work on answering complex queries over incomplete KBs extends single-hop queries to path queries ( or multi–hop queries ) of the type ( e1 , r1 , r2 , . . . , rn , ? ) . Guu et al . [ 2015 ] construct datasets for the task using random walks on FB13 and WN11 KBs . Models for the task include compositional training of existing KBC scoring functions and models based on variants of RNNs [ Yin et al. , 2018 , Das et al. , 2017 ] . Recent work has started exploring subsets of first-order logic queries . Hamilton et al . [ 2018 ] explore conjunctive queries with conjunction ( ∧ ) and existential quantification ( ∃ ) . They propose DeepSets [ Zaheer et al. , 2017 ] to model conjunction as a learned geometric operation . MPQE [ Daza and Cochez , 2020 ] and BIQE [ Kotnis et al. , 2021 ] extend this work by constructing a query representation using a graph neural network and transformer encoder respectively . Probably closest to our work is that of answering Existential Positive First-order ( EPFO ) logical queries [ Ren et al. , 2020 ] – queries that include ∧ , ∨ , and ∃ . They propose Query2Box , which represents a query as a box ( center , offset ) and each entity as point in a highdimensional space . The points closer to the center are scored higher for the query , especially if they are inside the box . The strength of Query2Box is that it can naturally handle large answer spaces , and also model many relation patterns ( see Section 2.1 ) . [ Ren and Leskovec , 2020 ] extend EPFO to complete set of first-order logical operations , by handling negation ( ¬ ) with their proposed model , BetaE . They propose high-dimensional Beta distributions as embeddings of queries and entities . The KL divergence between query and entity serves as a scoring function . There are also other works that work with probabilistic databases for answering EPFO queries [ Friedman and Van den Broeck , 2020 ] , but , to the best of our knowledge , there has been no work on answering regex queries over incomplete knowledge bases . In particular , regular expressions that contain the Kleene plus operator , are not included in path queries or first-order logic queries . 2.1 Relation Patterns in a KB . Embedding models can be critiqued based on their representation power [ Toutanova and Chen , 2015 , Guu et al. , 2015 , Trouillon et al. , 2016 , Sun et al. , 2019 ] in modeling commonly occuring relation patterns ( relationships amongst relations ) . We summarize the common patterns and model capabilities in Table 3 . Broadly , translation-based methods ( that add and subtract embedding vectors ) like TransE and Query2Box can model all patterns but can not model symmetry ( ( e1 , r1 , e2 ) ⇒ ( e2 , r1 , e1 ) ) , such as for the relation friend of . Rotation-based models such as RotatE can model all inference patterns except for hierarchy : ( e1 , r1 , e2 ) ⇒ ( e1 , r2 , e2 ) . Other models have other weaknesses , like similaritybased methods ( that use dot products between embeddings ) such as DistMult and ComplEx are incapable of handling composition and intersection . Our proposed model combines the power of Query2Box and RotatE and can express all seven common relation patterns . 3 . Task . We now formally define the task of answering regular expressions ( regex ) queries on a KB . Regular expressions are primarily used to express navigational paths on a KB in a more succinct way and to extend matching to arbitrary length paths . Any regex c in a KB is an expression over R ∪ { + , / , ∨ } , generated from the grammar : c : := r | c1 ∨ c2 | c1/c2 | c+ , where r ∈ R represents a relation in the KB and c1 and c2 represent regular expressions . / denotes followed by , ∨ denotes disjunction , and + is a Kleene Plus operator denoting one or more occurrences . We use l ( c ) to denote the set of paths ( sequence of relations ) compatible with regex c. As an example , for c = r1/r + 2 , l ( c ) will have paths like r1/r2 , r1/r2/r2 , r1/r2/r2/r2 and so on . Their path lengths will be 2 , 3 and 4 respectively . A regex query q over a KB is defined as a pair of head entity e1 and a regular expression c. For query q = ( e1 , c , ? ) , we define the answer set JqK as the union of entities reachable from e1 in G , by following the paths in l ( c ) : JqK = ⋃ p∈l ( c ) J ( e1 , p , ? ) K . The goal of the task is to , given a query q , rank entities in JqK higher than other entities . 4 . Model . We now describe RotatE-Box – a model that combines the strengths of RotatE and Query2Box . Following Query2Box , the relations and queries are modeled as boxes , and entities as points in a high-dimensional space . However , unlike Query2Box , all points ( and boxes ) are in a complex vector space Ck , instead of a real space . We first describe the model for single-hop queries . RotatE-Box embeds each entity e as a point e ∈ Ck and each relation r as a box r , represented as ( Cen ( r ) , Off ( r ) ) ∈ C2k . Here Cen ( r ) ∈ Ck is the center of the box such that its modulus in each of the k dimensions is one , i.e. , ∀j ∈ [ 1 , k ] , |Cen ( rj ) | = 1 , Cen ( rj ) ∈ C. By doing so , Cen ( rj ) can be re-written as eiθr , j using Euler ’ s formula ( here , e is Euler ’ s number ) . This corresponds to rotation by angle θr , j in the jth dimension of complex vector space . In vectorized form , Cen ( r ) = eiθr . Off ( r ) is a positive offset , i.e. , Re ( Off ( r ) ) ∈ Rk≥0 and Im ( Off ( r ) ) ∈ Rk≥0 . Any box that satisfies these constraints on its center and offset is termed as a rotation box . A point v ∈ Ck is considered inside a box if it satisfies : Re ( rmin ) ≤ Re ( v ) ≤ Re ( rmax ) and Im ( rmin ) ≤ Im ( v ) ≤ Im ( rmax ) ( 1 ) Here , rmax = Cen ( r ) + Off ( r ) ∈ Ck , and rmin = Cen ( r ) − Off ( r ) ∈ Ck . The inequalities represent element-wise inequality of vectors . A single-hop query q = ( e1 , r , ? ) has a rotation box representation q given by ( e1 Cen ( r ) , Off ( r ) ) . Here is the Hadamard product ( element-wise multiplication ) . Since Cen ( r ) = eiθr , e1 is rotated by an angle θr ∈ Rk . Specifically ej is rotated by angle θr , j in the j th dimension . Given a query q associated with a box q , RotatE-Box ranks entities e in the order of the distance of e from q , defined as : dist ( e ; q ) = distout ( e ; q ) + α · distin ( e ; q ) ( 2 ) where α ∈ ( 0 , 1 ) is a constant to downweigh the distance inside the query box , and distout ( e ; q ) = ‖Re ( Max ( e− qmax,0 ) + Max ( qmin − e,0 ) ) ‖1 + ‖Im ( Max ( e− qmax,0 ) + Max ( qmin − e,0 ) ) ‖1 distin ( e ; q ) = ‖Re ( Cen ( q ) −Min ( qmax , Max ( qmin , e ) ) ) ‖1 + ‖Im ( Cen ( q ) −Min ( qmax , Max ( qmin , e ) ) ) ‖1 Here qmax and qmin are defined analogously to rmax and rmin above . This completes the description of RotatE-Box for single-hop queries . As RotatE is a special case of RotatE-Box ( by modeling Off ( r ) = 0 , ∀r ∈ R ) , RotatE-Box can model all the relation patterns modeled by RotatE : symmetry , anti-symmetry , inversion , composition , intersection and mutual exclusion . Additionally , boxes enable modeling hierarchy . Theorem 1 . RotatE-Box can model hierarchical relation pattern . ( Proof in Appendix A.1 )

Summary: This work is on the topic of answering complex queries over incomplete Knowledge Bases (KBs). A KB is a set of triples/edges of the form (e1, r, e2) where ei's are entities/nodes and r is a relation/edge-label. Given a start entity e1 & an ordered sequence of relations r1 / r2 / ....rk one has to return all entities at the end of the paths consistent with the relational path r1 / r2 / ....rk. Importantly, a fraction of the edges are removed during training. Authors consider a novel problem in which one is allowed to represent *multiple* relational paths using a *single* regular-expression possibly containing disjunction ($\lor$) and Kleene plus (+) operators. For a regular-expression c, c+ denotes the infinite set of expressions {c, c/c, c/c/c ...}.