SQALER: Scaling Question Answering by Decoupling Multi-Hop and Logical Reasoning

1 Introduction . Reasoning , namely the ability to infer conclusions and draw predictions based on existing knowledge , is a hallmark of human intelligence . Infusing the same ability into machine learning models has been a major challenge [ 34 , 29 ] and has historically required complex systems made of several hand-crafted or learned components [ 47 , 19 ] . Recently , the paradigm has shifted to deep learning approaches [ 43 , 42 ] , where neural networks are used to reason over structured knowledge or a text corpus . In this work , we assume that the source of knowledge is a structured knowledge graph ( KG ) and we tackle the problem of knowledge-based question answering ( KBQA ) , namely finding answers to natural language queries involving multi-hop and logical reasoning over the KG . Answering queries over a knowledge graph involves many challenges , among which scalability is a major issue . Real-world KGs often contain millions of nodes and even a 2-hop neighborhood of the entities mentioned in the query may comprise tens of thousands of nodes . Many state-of-the-art approaches [ 41 , 42 , 39 ] address the challenge of scalability by building small query-dependent subgraphs . To this end , they usually use simple heuristics [ 41 ] or , in some cases , iterative procedures based on learned classifiers [ 42 ] . This preprocessing step is usually needed because each forward pass in end-to-end neural networks for KBQA scales at least linearly with the number of edges in the subgraph . Training neural networks involves repeated evaluation , which renders even a linear complexity impractical for graphs of more than a few tens thousands of nodes . In order to address this issue , we introduce a novel approach called SQALER ( Scaling Question Answering by Leveraging Edge Relations ) . The method first learns a model that generates a set of candidate answers ( entities in the KG ) by multi-hop reasoning : the candidate solutions are obtained by starting from the set of entities mentioned in the question and seeking those that provide an answer by chained relational following operations . We refer to this module as the relation-level model . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We show that this multi-hop reasoning step can be done efficiently and provably generates a set of candidates including all the actual answers to the original question . SQALER then uses a second-stage edge-level model that recovers the real answers by performing logical reasoning on a subgraph in the vicinity of the candidate solutions . A visual summary of our approach is depicted in Figure 1 . The main contributions and takeaway messages of this work are the following : 1 . KBQA can be addressed by first performing multi-hop reasoning on the KG and then refining the result with more sophisticated logical reasoning without losing expressive power ( we will elaborate this claim in more details in Section 2.3 ) . 2 . Multi-hop reasoning can be accomplished efficiently with a method that scales linearly with the number of relation types in the KG , which are usually significantly fewer than the number of facts or entities . In the remainder of the paper , we first provide an extensive overview of our approach and a theoretical analysis of the expressive power and the computational complexity of SQALER . Our experimental results show that SQALER achieves better reasoning performance than state-of-the-art approaches , generalizes compositionally out of the training distribution , and scales to the size of real-world knowledge graphs with millions of entities . 2 Scaling KBQA with relation and edge-level reasoning . This section provides a detailed description of our approach . We start by defining the problem formally and giving an intuitive overview of SQALER . Then , we discuss the approach in more details and we analyze its computational complexity and expressive power Problem statement . We denote a knowledge graph as G = ( V , R , E ) , where v ∈ V represents an entity or node in G , r ∈ R is a relation type , and we write v r−→ v′ to denote an edge in E labeled with relation type r ∈ R between two entities v , v′ ∈ V . We extend the same notation to sets of nodes by writing Vi r−→ Vj if Vj = { vj ∈ V | vi r−→ vj , vi ∈ Vi } . Given a knowledge graph G = ( V , R , E ) and a natural language question Q , expressed as a sequence of tokens Q = ( q1 , q2 , . . . , q|Q| ) , in knowledge-based question answering the objective is to identify a set of nodes AQ ⊆ V representing the correct answers to Q . Following previous work [ 41 , 42 , 43 ] , we assume that the set of entities mentioned in the question VQ ⊆ V is given . These nodes are also called the anchor nodes of the question and in practice are commonly obtained using an entity-linking module . Overview . KBQA can be cast as an entity seeking problem on G by translatingQ into a set of nodes VQ ⊆ V ( the starting points of the search ) and seeking for nodes that provide an answer [ 41 , 42 , 43 ] . Attempting to find AQ directly on G is prohibitive in practice , as even the most efficient graph-based neural networks generally scale at least linearly with the number of edges . Our approach mitigates this issue by breaking the problem in two subproblems . ( a ) We first utilize a relation-level model ϕ to obtain a set of candidate answers ÃQ , such that AQ ⊆ ÃQ . We refer to ϕ as “ relation-level ” because , as we will see , it operates on the coalesced graph , a simplified representation of G , where edges of the same relation type are coalesced . The coalesced graph is constructed before training and incurs a one-time linear cost . By exploiting it during training , the relation-level model scales with the number of ( distinct ) relation types in the KG , which are usually significantly fewer than the number of edges or entities . ( b ) The candidate answers are then refined using an edge-level model ψ applied on a subgraph G ( ÃQ ) of the original knowledge graph in the vicinity of ÃQ . We should note that the refining step is not always necessary . Indeed , we found that a relation-level model is sufficient to perfectly solve tasks like multi-hop question answering [ 50 ] . Figure 1 shows an overview of our approach . 2.1 Relational coalescing for efficient knowledge seeking . Our approach relies on a relation-level model ϕ that operates as a knowledge seeker in G. The model identifies a node v as a candidate v ∈ ÃQ based on the sequence of relations that connect it with VQ . This can be achieved by using a neural network ϕ to predict how likely it is that the correct answer is reached from VQ by following a sequence of relations R. Reachability . To define how our method works , it will help to formalize the concept of reachability . Let R = ( r1 , . . . , r|R| ) be a sequence of relations . We say that “ v is R-reachable from VQ ” if there exists a path P = ( v1 , . . . , v|R| , v ) in G such that : v1 ∈ VQ and vi ri−→ vi+1 for every i = 1 , . . . , |R| . That is , we can reach v by starting from a node in VQ and following a sequence of edges with relation types R. We also denote by reachG ( VQ , R ) the set of nodes that are R-reachable from VQ : reachG ( VQ , R ) = { v ∈ V | v is R-reachable from VQ } . Relational coalescing . Given a knowledge graph G , a question Q , and a set of entity mentions VQ , we consider a representation of the graph G̃Q = ( ṼQ , R̃Q , ẼQ ) , which allows us to efficiently compute sets of nodes that are reachable from VQ . We refer to this representation as the questiondependent coalesced KG , because edges with the same relation type are coalesced , as shown in Figure 1 . The nodes of G̃Q are sets of nodes of G that are reachable from VQ by following any possible sequence of relations originating from VQ . The graph G̃Q has an edge Vi r−→ Vj if Vj is the set of nodes that are reachable from Vi by following relation r. For convenience , we include a relation type self ∈ R̃Q to denote self loops . We refer the reader to Appendix A for a formal definition of G̃Q . The coalesced graph can be precomputed once as a preprocessing step for each question Q and incurs a one-time linear cost . In practice , however , we do not need to compute and store all the nodes in G̃Q but only edge labels . This makes learning efficient because each forward/backward pass scales with the number of relation types and does not depend on the number of nodes or edges in the KG . Knowledge seeking in G̃Q . The coalesced graph allows us to provide approximate answers to input questions in an efficient manner . Specifically , we seek k ≥ 1 sequences of relations R⋆i , such that : AQ ⊆ ÃQ = k⋃ i=1 reachG ( VQ , R⋆i ) . We can achieve this by using a model ϕ that only considers relation sequences originating from VQ . The model predicts the likelihood ϕ : ẼQ → [ 0 , 1 ] of following a certain edge in a relation sequence from VQ to ÃQ . Then , given R = ( r1 , . . . , r|R| ) and a node in the coalesced graph VQ , we can compute the likelihood of R by multiplying the likelihood of all edges traversed by R in G̃Q : P ( R | Q , G̃Q , VQ ) ∝ |R|∏ i=1 ϕ ( reachG ( VQ , R1→i−1 ) , ri , reachG ( VQ , R1→i ) | Q ) , where R1→i = ( r1 , . . . , ri ) is the subsequence of R up to the i-th relation . We generate ÃQ by selecting the top k relation sequences R⋆i with maximum likelihood P ( R ⋆ i | Q , G̃Q , VQ ) . This can be done by an efficient search algorithm , such as beam search starting from VQ . Then , we compute ÃQ as the union of all target nodes of the selected relation sequences . More details about the knowledge-seeking algorithm are provided in Appendix B . 2.2 Refining the solution on the original KG . In certain cases , like multi-hop question answering [ 50 ] , the set of candidate answers ÃQ may already be a reasonable estimate of AQ . We will substantiate this claim experimentally in Section 4 . In general , however , we recover AQ by using an edge-level model ψ applied on a subgraph G ( ÃQ ) of G. Specifically , we construct G ( ÃQ ) as the subgraph induced by the set of nodes V ( ÃQ ) , which includes all nodes visited when following the top-k relation sequences along with their neighbors ( see Figure 1 for an example ) . Any existing method for KBQA can be used to instantiate ψ by running it on G ( ÃQ ) rather than G. We opted to use a Graph Convolutional Network ( GCN ) conditioned on the input question with the same architecture as in [ 41 ] . The edge-level model is constrained to predict an answer among the candidates generated by the relation-level model .

Summary: This paper proposes an end-to-end KBQA system that can be directly used on the whole knowledge bases. This is in contrast to existing approaches that need a heuristic subgraph construction stage. More specifically, the proposed method trains a relation-following model to collect a candidate set of entities and the corresponding subgraph, which can then be processed by existing graph models. Empirically, the proposed method achieves state-of-the-art performance on MetaQA and WebQuestionsSP. Overall, this paper presents string empirical results over existing baselines on multi-hop KBQA, however, the main idea is hardly novel and has been applied to similar tasks.