SQALER: Scaling Question Answering by Decoupling Multi-Hop and Logical Reasoning

1 Introduction . Reasoning , namely the ability to infer conclusions and draw predictions based on existing knowledge , is a hallmark of human intelligence . Infusing the same ability into machine learning models has been a major challenge [ 34 , 29 ] and has historically required complex systems made of several hand-crafted or learned components [ 47 , 19 ] . Recently , the paradigm has shifted to deep learning approaches [ 43 , 42 ] , where neural networks are used to reason over structured knowledge or a text corpus . In this work , we assume that the source of knowledge is a structured knowledge graph ( KG ) and we tackle the problem of knowledge-based question answering ( KBQA ) , namely finding answers to natural language queries involving multi-hop and logical reasoning over the KG . Answering queries over a knowledge graph involves many challenges , among which scalability is a major issue . Real-world KGs often contain millions of nodes and even a 2-hop neighborhood of the entities mentioned in the query may comprise tens of thousands of nodes . Many state-of-the-art approaches [ 41 , 42 , 39 ] address the challenge of scalability by building small query-dependent subgraphs . To this end , they usually use simple heuristics [ 41 ] or , in some cases , iterative procedures based on learned classifiers [ 42 ] . This preprocessing step is usually needed because each forward pass in end-to-end neural networks for KBQA scales at least linearly with the number of edges in the subgraph . Training neural networks involves repeated evaluation , which renders even a linear complexity impractical for graphs of more than a few tens thousands of nodes . In order to address this issue , we introduce a novel approach called SQALER ( Scaling Question Answering by Leveraging Edge Relations ) . The method first learns a model that generates a set of candidate answers ( entities in the KG ) by multi-hop reasoning : the candidate solutions are obtained by starting from the set of entities mentioned in the question and seeking those that provide an answer by chained relational following operations . We refer to this module as the relation-level model . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We show that this multi-hop reasoning step can be done efficiently and provably generates a set of candidates including all the actual answers to the original question . SQALER then uses a second-stage edge-level model that recovers the real answers by performing logical reasoning on a subgraph in the vicinity of the candidate solutions . A visual summary of our approach is depicted in Figure 1 . The main contributions and takeaway messages of this work are the following : 1 . KBQA can be addressed by first performing multi-hop reasoning on the KG and then refining the result with more sophisticated logical reasoning without losing expressive power ( we will elaborate this claim in more details in Section 2.3 ) . 2 . Multi-hop reasoning can be accomplished efficiently with a method that scales linearly with the number of relation types in the KG , which are usually significantly fewer than the number of facts or entities . In the remainder of the paper , we first provide an extensive overview of our approach and a theoretical analysis of the expressive power and the computational complexity of SQALER . Our experimental results show that SQALER achieves better reasoning performance than state-of-the-art approaches , generalizes compositionally out of the training distribution , and scales to the size of real-world knowledge graphs with millions of entities . 2 Scaling KBQA with relation and edge-level reasoning . This section provides a detailed description of our approach . We start by defining the problem formally and giving an intuitive overview of SQALER . Then , we discuss the approach in more details and we analyze its computational complexity and expressive power Problem statement . We denote a knowledge graph as G = ( V , R , E ) , where v ∈ V represents an entity or node in G , r ∈ R is a relation type , and we write v r−→ v′ to denote an edge in E labeled with relation type r ∈ R between two entities v , v′ ∈ V . We extend the same notation to sets of nodes by writing Vi r−→ Vj if Vj = { vj ∈ V | vi r−→ vj , vi ∈ Vi } . Given a knowledge graph G = ( V , R , E ) and a natural language question Q , expressed as a sequence of tokens Q = ( q1 , q2 , . . . , q|Q| ) , in knowledge-based question answering the objective is to identify a set of nodes AQ ⊆ V representing the correct answers to Q . Following previous work [ 41 , 42 , 43 ] , we assume that the set of entities mentioned in the question VQ ⊆ V is given . These nodes are also called the anchor nodes of the question and in practice are commonly obtained using an entity-linking module . Overview . KBQA can be cast as an entity seeking problem on G by translatingQ into a set of nodes VQ ⊆ V ( the starting points of the search ) and seeking for nodes that provide an answer [ 41 , 42 , 43 ] . Attempting to find AQ directly on G is prohibitive in practice , as even the most efficient graph-based neural networks generally scale at least linearly with the number of edges . Our approach mitigates this issue by breaking the problem in two subproblems . ( a ) We first utilize a relation-level model ϕ to obtain a set of candidate answers ÃQ , such that AQ ⊆ ÃQ . We refer to ϕ as “ relation-level ” because , as we will see , it operates on the coalesced graph , a simplified representation of G , where edges of the same relation type are coalesced . The coalesced graph is constructed before training and incurs a one-time linear cost . By exploiting it during training , the relation-level model scales with the number of ( distinct ) relation types in the KG , which are usually significantly fewer than the number of edges or entities . ( b ) The candidate answers are then refined using an edge-level model ψ applied on a subgraph G ( ÃQ ) of the original knowledge graph in the vicinity of ÃQ . We should note that the refining step is not always necessary . Indeed , we found that a relation-level model is sufficient to perfectly solve tasks like multi-hop question answering [ 50 ] . Figure 1 shows an overview of our approach . 2.1 Relational coalescing for efficient knowledge seeking . Our approach relies on a relation-level model ϕ that operates as a knowledge seeker in G. The model identifies a node v as a candidate v ∈ ÃQ based on the sequence of relations that connect it with VQ . This can be achieved by using a neural network ϕ to predict how likely it is that the correct answer is reached from VQ by following a sequence of relations R. Reachability . To define how our method works , it will help to formalize the concept of reachability . Let R = ( r1 , . . . , r|R| ) be a sequence of relations . We say that “ v is R-reachable from VQ ” if there exists a path P = ( v1 , . . . , v|R| , v ) in G such that : v1 ∈ VQ and vi ri−→ vi+1 for every i = 1 , . . . , |R| . That is , we can reach v by starting from a node in VQ and following a sequence of edges with relation types R. We also denote by reachG ( VQ , R ) the set of nodes that are R-reachable from VQ : reachG ( VQ , R ) = { v ∈ V | v is R-reachable from VQ } . Relational coalescing . Given a knowledge graph G , a question Q , and a set of entity mentions VQ , we consider a representation of the graph G̃Q = ( ṼQ , R̃Q , ẼQ ) , which allows us to efficiently compute sets of nodes that are reachable from VQ . We refer to this representation as the questiondependent coalesced KG , because edges with the same relation type are coalesced , as shown in Figure 1 . The nodes of G̃Q are sets of nodes of G that are reachable from VQ by following any possible sequence of relations originating from VQ . The graph G̃Q has an edge Vi r−→ Vj if Vj is the set of nodes that are reachable from Vi by following relation r. For convenience , we include a relation type self ∈ R̃Q to denote self loops . We refer the reader to Appendix A for a formal definition of G̃Q . The coalesced graph can be precomputed once as a preprocessing step for each question Q and incurs a one-time linear cost . In practice , however , we do not need to compute and store all the nodes in G̃Q but only edge labels . This makes learning efficient because each forward/backward pass scales with the number of relation types and does not depend on the number of nodes or edges in the KG . Knowledge seeking in G̃Q . The coalesced graph allows us to provide approximate answers to input questions in an efficient manner . Specifically , we seek k ≥ 1 sequences of relations R⋆i , such that : AQ ⊆ ÃQ = k⋃ i=1 reachG ( VQ , R⋆i ) . We can achieve this by using a model ϕ that only considers relation sequences originating from VQ . The model predicts the likelihood ϕ : ẼQ → [ 0 , 1 ] of following a certain edge in a relation sequence from VQ to ÃQ . Then , given R = ( r1 , . . . , r|R| ) and a node in the coalesced graph VQ , we can compute the likelihood of R by multiplying the likelihood of all edges traversed by R in G̃Q : P ( R | Q , G̃Q , VQ ) ∝ |R|∏ i=1 ϕ ( reachG ( VQ , R1→i−1 ) , ri , reachG ( VQ , R1→i ) | Q ) , where R1→i = ( r1 , . . . , ri ) is the subsequence of R up to the i-th relation . We generate ÃQ by selecting the top k relation sequences R⋆i with maximum likelihood P ( R ⋆ i | Q , G̃Q , VQ ) . This can be done by an efficient search algorithm , such as beam search starting from VQ . Then , we compute ÃQ as the union of all target nodes of the selected relation sequences . More details about the knowledge-seeking algorithm are provided in Appendix B . 2.2 Refining the solution on the original KG . In certain cases , like multi-hop question answering [ 50 ] , the set of candidate answers ÃQ may already be a reasonable estimate of AQ . We will substantiate this claim experimentally in Section 4 . In general , however , we recover AQ by using an edge-level model ψ applied on a subgraph G ( ÃQ ) of G. Specifically , we construct G ( ÃQ ) as the subgraph induced by the set of nodes V ( ÃQ ) , which includes all nodes visited when following the top-k relation sequences along with their neighbors ( see Figure 1 for an example ) . Any existing method for KBQA can be used to instantiate ψ by running it on G ( ÃQ ) rather than G. We opted to use a Graph Convolutional Network ( GCN ) conditioned on the input question with the same architecture as in [ 41 ] . The edge-level model is constrained to predict an answer among the candidates generated by the relation-level model .

Summary: The paper focuses on the task of knowledge graph question answering. It specifically aims to address the challenge of scalability issue of the current KGQA methods. The overall idea is to first obtain a question-specific coalesced subgraph out of the knowledge graph, which “merges” entities of the same type (connected by the same relation types). Then the method locates a subset of answer candidates on the subgraph by traversing on this coalesced subgraph with some relation sequences derived from the mentioned entities in the question. Then they run a GNN model to select the final answer to the question from the answer candidates.  Generally I feel the paper writing can be refactored and improved. I suggest the authors list all the model components at the beginning of Section 2. Currently they only introduce a relation-level model and the edge-level model in Section 2, then out of nowhere, they mention they further propose an autoencoder in Section 3. They also say that “the architecture of the model includes three main components: a question encoder, a relation encoder, and a graph-guided decoder” in line 163. Where is relation-level model and edge-level model?  One key assumption (or limitation) in the paper is that the graph is complete, which renders the proposed method weaker than the baselines, e.g., all Pullnet, embedkgqa, EmQL can deal with missing edges. Although the authors mention in line 147-154 that “it may still be able to retrieve alternative sequences of relations, provided that the training data are representative enough”, can you run experiments on incomplete MovieGraph for MetaQA? I assume the training questions in MetaQA are much more than that in WebQSP and should also be very “representative” since the questions in the dataset are generated using some templates.  Another general comment is that since the paper focuses on QA in the complete graph setting and aims to obtain a relation sequence as the “parsed representation” of the question. Can you also compare your method with the series of (unsupervised) semantic parsing methods? These methods aim to parse the input questions to some logic form and then execute the logical queries on KG for answers.