Pay More Attention to Relation Exploration for Knowledge Base Question Answering
Year: 2023

Abstract:
Knowledge base question answering (KBQA) is a challenging task that aims to retrieve correct answers from large-scale knowledge bases. Existing attempts primarily focus on entity representation and final answer reasoning, which results in limited supervision for this task. Moreover, the relations, which empirically determine the reasoning path selection, are not fully considered in recent advancements. In this study, we propose a novel framework, REKBQA, that utilizes relations in the knowledge base to enhance entity representation and introduce additional supervision. We explore guidance from relations in three aspects, including (1) distinguishing similar entities by employing a variational graph auto-encoder to learn relation importance; (2) exploring extra supervision by predicting relation distributions as soft labels with a multi-task scheme; (3) designing a relation-guided re-ranking algorithm for post-processing. Experimental results on two benchmark datasets demonstrate the effectiveness and superiority of our framework, improving the F1 score by 5.8% from 40.5 to 46.3 on CWQ and 5.7% from 62.8 to 68.5 on WebQSP, better or on par with state-of-the-art methods.

heading: 1 Introduction
text: Given a question expressed in natural language, knowledge base question answering (KBQA) aims to find the correct answers from a large-scale knowledge base (KB), such as Freebase (Bollacker et al., 2008), Wikipedia (Vrandečić and Krötzsch, 2014), DBpeidia (Auer et al., 2007), etc. For example, the question “Who is Emma Stone’s father?” can be answered by the fact of “(Jeff Stone, person.parents, Emma Stone)”. The deployment of KBQA can significantly enhance a system’s knowledge, improving performance for applications such as dialogue systems and search engines.
†Corresponding author.
Early attempts on KBQA (Min et al., 2013; Zhang et al., 2018; Xu et al., 2019) mostly focus on transferring given questions into structured logic forms, which are strictly constrained by the consistent structure of parsed query and KB. To overcome the limitation of the incompleteness of KB, many approaches (Xiong et al., 2019; Deng et al., 2019; Lan et al., 2021) have been developed that aim to map questions and their related KB entities and relations into embeddings, and define the reasoning process as a similarity retrieval problem, which is called IR-based method. Additionally, some studies (Gao et al., 2022; Liu et al., 2023; Ge et al., 2022) have attempted to learn relation embeddings and then incorporate surrounding relations to represent entities, which successfully reduces the number of parameters needed for the model.
However, most of these works (Han et al., 2021) primarily focus on final answer reasoning and the representation of entities, while few explore the full utilization of relations in KB. Additionally, for answer reasoning, the supervision signal provided is also only from entities, while we believe that the relations also play an important role in determining the reasoning path and the answer choosing.
2119
We propose a new framework, called RelationEnhanced KBQA (RE-KBQA), to investigate the potential use of relations in KBQA by utilizing an embedding-fused framework. The proposed framework aims to study the role of relations in KBQA in the following three aspects:
Relations for entity representation. We find that similar entities with similar surrounding relations (e.g., the three green circles in the upper right of Figure 1) play an important role in reasoning. To distinguish them, we introduce QAVGAE, a question-answering-oriented variational graph auto-encoder, which learns relation weights through global structure features and represents entities by integrating surrounding relations.
Relations for extra supervision. Multi-hop reasoning is often hindered by weak supervision, as models can only receive feedback from final answers (He et al., 2021). To overcome this limitation, we propose a multi-task scheme by predicting the relation distribution of the final answers as additional guidance, using the same reasoning architecture and mostly shared parameters. As illustrated in Figure 1, the proposed scheme requires the prediction of both the answer "Haitian Creole" and its surrounding relation distribution.
Relations for post-processing. We propose a stem-extraction re-ranking (SERR) algorithm to modify the confidence of candidates, motivated by the fact that relations parsed from given questions are empirically associated with strong reasoning paths. As depicted in the bottom of Figure 1, relations that overlap with a given question will be marked as key reasoning relations, and their confidence will be increased empirically. This allows for re-ranking and correction of the final answers.
In general, our contributions can be summarized as follows. (1) We propose a novel method named Relation Enhanced KBQA (RE-KBQA) by first presenting QA-VGAE for enhanced relation embedding. (2) We are the first to devise a multi-task scheme to implicitly exploit more supervised signals. (3) We design a simple yet effective postprocessing algorithm to correct the final answers, which can be applied to any IR-based method. (4) Lastly, we conduct extensive experiments on two challenging benchmarks, WebQSP and CWQ to show the superiority of our RE-KBQA over other competitive methods. Our code and datasets are publicly available on Github1.
1github.com/yongcaoplus/RE-KBQA

heading: 2 Related Work
text: Knowledge Base Question Answering. Most existing research on KBQA can be categorized into two groups: a). Semantic Parsing (SP)-based methods (Abdelaziz et al., 2021; We et al., 2021; Cui et al., 2022), which transfer questions into logical form, e.g., SPARQL queries, by entity extraction, KB grounding, and structured query generation. b). Information Retrieval (IR)-based method (Ding et al., 2019; Chen et al., 2019; Wang et al., 2021; Feng et al., 2021; Zhang et al., 2022b), which applies retrieve-and-rank mechanism to reason and score all candidates of the subgraph with advancements in representation learning and ranking algorithms. Apart from the above approaches, recent studies (Xiong et al., 2019; Deng et al., 2019; Lan et al., 2021) also propose several alterations over the reasoning process, such as extra corpus exploration (Xiong et al., 2019), better semantic representation (Zhu et al., 2020; Ge et al., 2021), dynamic representation (Han et al., 2021), and intermediate supervised signals mining (Qiu et al., 2020; He et al., 2021). Aiming to tackle limited corpus, some works are devoted to utilizing external resources, such as using pre-trained language models (Unik-QA) (Oguz et al., 2022), retrieving similar documents (CBR-KBQA) (Das et al., 2021), extra corpus (KQA-Pro) (Cao et al., 2022), etc.
Multi-task Learning for KBQA. Multitask learning can boost the generalization capability on a primary task by learning additional auxiliary tasks (Liu et al., 2019) and sharing the learned parameters among tasks (Hwang et al., 2021; Xu et al., 2021). Many recent works have shown impressive results with the help of multi-task learning in many weak supervised tasks such as visual question answering (Liang et al., 2020; Rajani and Mooney, 2018), sequence labeling (Rei, 2017; Yu et al., 2021), text classification (Liu et al., 2017; Yu et al., 2019) and semantic parsing (Hershcovich et al., 2018). In KBQA, auxiliary information is often introduced in the form of artificial “tasks” relying on the same data as the main task (Hershcovich et al., 2018; Ansari et al., 2019; Gu et al., 2021), rather than independent tasks. This assists the reasoning process and proves to be more effective for the main task. To the best of our knowledge, we are the first to propose a multi-task to assist KBQA by using mostly shared parameters among tasks, for a balance of effectiveness and efficiency.

heading: 3 Problem Formulation
text: Knowledge Base (KB). A knowledge base usually consists of a huge amount of triples: G = {⟨e, r, e′⟩|(e, e′) ∈ ξ, r ∈ R}, where ⟨e, r, e′⟩ denotes a triple with head entity e, relation r and tail entity e′. ξ and R mean the sets of all entities and relations, respectively. To apply the triples to downstream task, the entities and relations should be firstly embedded as d-dimensional vectors: V = {⟨Ve, Vr, Ve′⟩|(Ve, Ve′) ∈ Vξ, Vr ∈ VR}.
Knowledge Base Question Answering (KBQA). Our dataset is formed as question-answer pairs. Let Q represents the set of given questions and each question q is composed of separated tokens, where Q = {q ∈ Q|q = x1, x2, ..., xn}. Let A (⊆ ξ) represents the correct answers of Q. Thus, the dataset is formulated as D = {(Q,A)|(q1, a1), (q2, a2), ..., (qm, am)}. To reduce the complexity of reasoning process, we extract question-related head entities eh from q and generate an associated subgraph gsub (∈ Gsub) within multi-hops walking from eh. Thus, the goal of KBQA is transformed to reason the candidates c (⊆ ξ) of the highest confidence from gsub, which can be formalized as:
c = argmax θ,ϕ rϕ(fθ(q, gsub)), (1)
where fθ(·) and rϕ(·) denote the representation and reasoning network, respectively.

heading: 4 Our Approach
text: As discussed in Section 1, we consider three aspects to further boost the performance of KBQA, including (i) the enhancement of the representation
capability, especially for similar entities; (ii) a strategy of mining more supervision signals to guide the training; and (iii) a reasoning path correction algorithm to adjust the ranking results. Below, we shall elaborate on our network architecture (RE-KBQA) with our solutions to the above issues.

heading: 4.1 Architecture Overview
text: Inspired by the neighborhood aggregation strategy, we employ Neural State Machine (NSM) (He et al., 2021) as our backbone model, where entities are denoted by surrounding relations. We assume that the topic entities and the related subgraph are already achieved by preprocessing; see Section 5.2 for the details. Figure 2 shows the main pipeline of our RE-KBQA. Specifically, given a question q, we first employ a question embedding module to encode it into semantic vector. Here, for a fair comparison with NSM baseline, we follow (He et al., 2021) to adopt Glove (Pennington et al., 2014) to encode q into embeddings {V jq }nj=1 = Glove(x1, x2, ..., xn), which is then mapped to hidden states by LSTM:
{h′, {hj}nj=1} = LSTM(V 1q , V 2q , ..., V nq ), (2)
where we set h′ as the last hidden state of LSTM to denote question vector and {hj}nj=1 denotes the vector of tokens. After obtaining h′ and {hj}nj=1, then we can calculate :
q(t) = ψ(s(t−1), h′), (3)
where ψ(·) denotes multi-layer percetron function. Then, the semantic vector s(t) at the t-th reasoning step of question q is obtained by:
s(t) = n∑
j=1
p(ψ(q(t), hj)) · hj , (4)
where p(·) denotes score function, and s(0) (∈ R(|d|)) is initialized randomly.
Next, a QA-VGAE enhanced representation module is designed to represent KB elements under the guidance of s(t). Then, unlike previous works that directly predict final answer via a score function, we introduce a multi-task learning-fused reasoning module to further predict an auxiliary signal (i.e., relation distribution). Note that, though we adopt NSM framework to conduct KBQA task, we concentrate on the representation capability enhancement by identifying similar entities, as well as the multi-task learning via supervision signal mining. At last, to avoid ignoring strong reasoning paths, we further propose a stem-extraction re-ranking algorithm to post-process the predictions of our network. Below, we will present the details of three of our proposed contributed modules.

heading: 4.2 QA-VGAE Enhanced Representation
text: Similar entities are defined as entities that are connected mostly by the same edges, and only a small portion of edges are different. For example, as shown in Figure 1, the three nodes marked by dashed circles share almost the same edges, and only the node of “Haiti” holds the relation of “Person.Spoken_language” that is quite important for answering the question. Hence, distinguishing similar entities and identifying key reasoning paths are essential for embedding-fused information retrieval-based methods. Traditional methods like TransE (Bordes et al., 2013) can grasp local information from independent triples within a KB, but fail to capture the inter-relations between adjacent triple facts. Consequently, they tend to have difficulties in distinguishing similar entities.
To alleviate the above problem, we introduce Question Answering-oriented Variational Graph Auto-Encoder (QA-VGAE) module, as is shown in Figure 3, by assigning different weights to reasoning relations, where the weights are learned by VGAE (Kipf and Welling, 2016). Note that, compared with traditional methods like TransE (Bordes et al., 2013), TransR (Lin et al., 2015), and ComplEx (Trouillon et al., 2016), VGAE achieves superior performance in link prediction task. We thus adopt VGAE in our module to learn weights. The key insight of this module is to fully learn global structure features by executing graph reconstruction task and constraining the representation as normal distribution, thus promoting the relation
representation to be more discriminating. Finally, by similarity evaluation of the learned representation, we can obtain the prior probability of relation (PPR) matrix, whose elements denote the conditional probability of relations.
In detail, we first transfer the KB from ⟨e, r, e′⟩ (entity-oriented) to ⟨r, e, r′⟩ (relation-oriented). In this way, we can then learn PPR matrix via a link prediction task by unsupervised learning. Specifically, given the connection degrees X (∈ R|nr|×|nr|) of a relation and the adjacency A (∈ R|nr|×|nr|) between relation nodes, where nr denotes the number of relations, we adopt two-layers GCN to learn the mean σ and variance µ of the relation importance distribution, and further compound the relation representation Z as :
Z = GCNµ(X,A)⊕GCNσ(X,A), (5)
where ⊕ is compound function. Then, PPR matrix Pr is obtained by distribution similarity evaluation:
Pr = Softmax ( Z · Z⊤ ) , (6)
where Pr ∈ R|nr|×|nr|. Please refer to Appendix A.1 for loss function LP of QA-VGAE. Next, we denote KB elements as d-dim vectors, Vξ(∈R|ne|×|d|) as entity vectors and VR(∈ R|nr|×|d|) as relation vectors, where ne is the number of enities. We denote candidate vectors VC as:
VC =WC · Pr · VR, (7)
where WC ∈ R|nc|×|nr| denotes the surrounding relation matrix of entities and nc denotes number of candidates.
Then, to integrate semantic vectors s(t) of given question and the history vector, we update Vc as:
V̂ (t)c = σ([V (t−1) c ; s (t) ⊙Wr ⊙ Vc]), (8)
where V (t)c (∈ VC) is candidate vector at time step t, σ(·) is the linear layer, [; ] is the concatenation operation, ⊙ is element-wise multiplication, and Wr (∈ R|d|) is the matrix of learnable parameter.

heading: 4.3 Multi-task Learning-Fused Reasoning
text: The purpose of this module is to conduct answer reasoning from candidate vector V̂ (t)c . To this end, we jointly combine the reasoning paths implicitly among candidates by utilizing the Transformer (Vaswani et al., 2017), formalized as:
V (t)c = Transformer ([ V̂ (t)c1 ; V̂ (t) c2 ; ...; V̂ (t) cl ]) ,
(9) where {V̂ (t)ci }li=1 denotes all the candidate vectors.
However, like most existing works (Deng et al., 2019; Lange and Riedmiller, 2010), learning from the final answers as the feedback tends to make the model hard to train, due to the limited supervision. How to introduce extra supervision signals into network model is still an open question. In our method, we introduce a new multi-task to learn the distribution of candidates’ surrounding relations, namely surrounding relations reasoning. The key idea is to leverage relations around final answer as extra supervisions to promote the performance, and also modify reasoning paths implicitly.
Specifically, motivated by weakly-supervised learning methods, we assume the reasoning process starts from topic entity’s surrounding relations S (0) R (initialized along with subgraph generation), and during reasoning, we can easily obtain next surrounding relations’ distribution by:
S (t) R = σ ([ (s(t) · V ⊤(t)R ;S (t−1) R ]) , (10)
where S(t)R denotes the surrounding relations of candidates at step t and V ⊤(t)R is the transpose of VR at step t. Note that, introducing the multi-task will not increase the complexity of our method obviously, since the number of relations is far fewer than that of entities in most cases, and the multitask shares most parameters with the main task.
In this way, there are two optimization goals of KBQA task, i.e., correct answer retrieving and
surrounding relations prediction. We predict the final answers’ possibilities by:
p(t)c = Softmax ( V (t)c ·W (t)c ) , (11)
where p(t)c is the confidence of predicted answers. Also, the relation distribution confidence p(t)r is:
p(t)r = Softmax ( S (t) R ·W (t)r ) , (12)
where W (t)c and W (t) r are learnable parameters.
Then, the answer retriving loss Lc and the relation prediction loss Lr can be calculated by:
Lc =KL(p(t)c , p(∗)c ) Lr =KL(p(t)r , p(∗)r ),
(13)
where pc(∗) and pr(∗) denote the ground truths, KL is the KL divergence. Thus, the final total loss is:
L = λLc + (1− λ)Lr, (14)
where λ denotes a hyper-parameter.

heading: 4.4 Stem-Extraction Re-Ranking
text: A limitation of embedding-fused KBQA methods is that the reasoning path is uncontrollable as the complete reasoning path is a blackbox in information retrieval-based methods. For example, in the question “What is the Milwaukee Brewers mascot?", the strongly related path “education.mascot" may be missed due to limited representation capability. However, this weakness can be easily addressed by semantic parsing-based methods by analyzing
the semantic similarity of key elements of questions and relations and constraining the reasoning path. Inspired by this observation, we propose a stem-extraction re-ranking (SERR) algorithm for post-processing. The key idea is to stem-match and re-rank the candidates after obtaining candidates and their confidence from our network.
In detail, we design three operators to execute the re-ranking as shown in Algorithm 1: stemmer F(·), modifier M(·), and re-ranker R(·). These operators are used to extract stems from relations or given questions, modify candidates’ confidence, and then re-rank the candidates. As shown in Figure 4, given question and candidate predictions, we first use F(·) to process all the relations of freebase relations and questions. Then, we generate a relation candidates pool by matching the stem pool of the question with the relation stems. This allows us to compare the subgraph of the given question with pseudo-facts produced by given topic entities and candidates, respectively. Finally, according to the comparison, M(·) and R(·) are employed to conduct the re-ranking process.
It is worth noting that, in our work, we directly use stem extraction method rather than similarity calculation to re-rank. The insight behind this choice is that, it is unnecessary to consider semantic features again, since we have already injected the question semantic information into our encoded semantic vector s(t), which means that the model is already equipped with semantic clustering capability. And obviously, stem extraction costs fewer computation resources, as proved in Appendix A.2. Also, our SERR can be migrated to other models as a plug-in and independent module.

heading: 5 Experiments and Results

heading: 5.1 Datasets
text: We conduct experiments on two popular benchmark datasets, including WebQuestionSP (Yih et al., 2015) and ComplexWebQuestions (Talmor and Berant, 2018). Specifically, WebQuestionSP (abbr. WebQSP) is composed of simple questions that can be answered within two hops reasoning, which is constructed based on Freebase (Bollacker et al., 2008). In contrast, ComplexWebQuestions (abbr. CWQ) is larger and more complicated, where the answers require multi-hop reasoning over several KB facts. The detailed statistics of the two datasets are summarized in Table 1.
Algorithm 1 Stem Extraction Re-Ranking Input: natural language question Q, candidates C, confidence pC , relation set R. Output: updated candidates C′ and confidence p′C .

heading: 5.2 Experimental Setting
text: Basic setting. To make a fair comparison with other methods, we follow existing works (Sun et al., 2019, 2018; He et al., 2021) to process datasets, including candidates generation by PageRank-Nibble algorithm and subgraph construction within threehops by retrieving from topic entities. We set the learning rate as 8e−4 and decay it linearly throughout iterations on both datasets. We set the number of training epoch on WebQSP and CWQ as 200 and 100, respectively. For better reproducibility, we give all the parameter settings in Appendix A.3.
Baselines. We compare our method with multiple representative methods, including semantic pars-
ing (SP)-based methods and information retrieval (IR)-based methods. SPARQA (Sun et al., 2020) and QGG (Lan and Jiang, 2020) belong to the former category, which focuses on generating optimal query structures. Besides, KV-Mem (Miller et al., 2016), EmbedKGQA (Saxena et al., 2020), GraftNet (Sun et al., 2018), PullNet (Sun et al., 2019), ReTraCk (Chen et al., 2021) and BiNSM (He et al., 2021) are all IR-based methods, which are also the focus of our comparison.
Evaluation metrics. To fully evaluate KBQA performance, we should compare both the retrieved and ranked candidates with correct answers. To this end, we employ the commonly-used F1 score and Hit@1. F1 score measures whether the retrieved candidates are correct, while Hit@1 evaluates whether the ranked candidate of the highest confidence is in answer sets.

heading: 5.3 Comparison with Others
text: We first compare our RE-KBQA against the aforementioned baselines on two datasets and the results are reported in Table 2. Note that, REKBQAb indicates our backbone network without three modules, i.e., QA-VGAE, multi-task learning and SERR. Clearly, even using our backbone network, it already outperforms most baselines on two datasets, which is benefited from the semantic guidance of given questions and the reasoning mechanisms. Further, as shown in the bottom row, our full pipeline achieves the highest values on both
datasets over both evaluation metrics. Particularly, compared with the results produced by RE-KBQAb, our full method improves more on CWQ dataset, which has increased by 3.5 and 5.8 in terms of Hit@1 and F1, showing that our contributions can indeed boost the multi-hop reasoning process. Besides, RE-KBQA also obtains good results on simple questions (i.e., WebQSP dataset), especially a 5.7 increase in F1 score, which reveals that the model can recall more effective candidates.
As shown in Table 2, we can observe that the SP-based methods (i.e., SPARQA and QGG) show a good performance in WebQSP, but perform worse in complicated questions, which reveals that SPbased methods are still weak in multi-hop reasoning. Similarly, traditional embedding methods, i.e., KV-Mem, EmbedKGQA, and GraftNet, also perform better in simple questions than in complex ones. Though PullNet and BiNSM show good multi-hop reasoning capacity, the extra corpora analysis and bi-directional reasoning mechanism inevitably increase the complexity of these networks.
Apart from above methods, some attempts are conducted on utilizing additional resources for task enhancement recently. As shown in Table 2 reference, CBR-KBQA relies on expensive largescale extra human annotations and Roberta pretrained model (PLM), Unik-QA tries to retrieve one-hundred extra context passages for relations in KB and T5-base (PLM), and KQA-Pro uses a large-scale dataset for pre-training with the help of explicit reasoning path annotation. While promising performance has been achieved through these methods, expensive human annotation costs and model efficiency also need to be concerned.

heading: 5.4 Network Component Analysis
text: To evaluate the effectiveness of each major component in our method, we conducted a comprehensive ablation study. In detail, similar to Section 5.3, we remove all three components and denote the backbone network as RE-KBQAb. Then, we add QA-VGAE (Section 4.2), multi-task learning (Section 4.3), and SERR (Section 4.4) back on REKBQAb, respectively. In this way, we constructed totally four network models and re-trained each model separately using the same settings of our RE-KBQA model. Table 3 shows the results. By comparing different cases with the bottom-most row (our full pipeline), we can see that each component contributes to improving the performance on both datasets. More ablation experiments can be found in Appendix. Below, we shall discuss the effect of each module separately.
Effect of QA-VGAE. From the results of Table 3, we can observe that the improvements of using QA-VGAE are more remarkable than using the other two modules, demonstrating that the QA-VGAE is more helpful to boost the reasoning process for both simple and complex questions. Besides quantitative comparison, we also tried to reveal its effect in a visual manner. Here, we adopt T-SNE to visualize the relation vectors. Figure 5 shows a typical embedding distribution before and after QA-VGAE training. For a clear visualization, we randomly select some relations related to a case “What is the capital of Austria?”. The orange nodes represent relations close to “location”, such as “location.country.capital”, “location.country.first_level_divisions”, etc., and the blue nodes denote the relations that are not covered by the question subgraph, which we call far relations. Obviously, after using QA-VGAE, the related relations (orange nodes in (b)) tend to get closer and the other nodes get farther.
Effect of multi-task learning. As shown in Table 3, the multi-learning module shows better performance in simple questions (see WebQSP dataset), since the relation distribution is denser than candidates distribution, thus causing the prediction to be more complicated along with the increase of reasoning steps. To fully explore the effect of this module, we study different loss fusion weights and the results are shown in Figure 6, where a larger λ (range from 0.1 to 1.0, and we discard the setting of 0.0 for its bad performance)
denotes a more weighted loss of main task. Clearly, only designing the primary task or auxiliary task is not optimal for KBQA, and the best setting of λ is 0.1 and 0.5 for the two datasets. An interesting observation is that the best Hit@1 is obtained with lower lambda while the best F1 score is obtained with higher lambda in each dataset. We claim that it is caused by the different goals of Hit@1 and F1 metrics, that is, Hit@1 shows whether the top one candidate is found while F1 score evaluates whether most candidates are found.
Effect of SERR. This module is lightweight (see Appendix A.2 for inference time) yet effective, especially for simple questions; see Table 3. Intuitively, the stem extraction for key paths is quite effective for questions that rely on directconnected facts. In contrast, stem extraction for complex questions relies more on the startpoint and endpoint. Figure 7(a) further shows an example result of SERR module, which proves that it can effectively identify close connected facts of a given question and re-rank the candidates.

heading: 5.5 Case Study
text: At last, we show a case result produced by our REKBQA; see Figure 7(b). Given the question “What are the movies that had Tupac in them and which were filmed in New York City?”, our method first
embedded the question into vectors and retrieve related subgraphs. Then, by utilizing the promotion of our proposed QA-VGAE and multi-task learning, we can use the trained model and obtain the candidates of “Murder Was the Case”, “Nothing but Trouble”, etc, and thanks to the SERR algorithm, our reasoning process can have a chance to re-rank the candidates, thus boosting its performance. Finally, we output Juice and Above the Rim as the correct answers. For similarity entity identification, SERR in other methods as a plug-in and more case results, please refer to Appendix A.2 and A.5.

heading: 6 Conclusion
text: In this paper, we proposed a novel framework, namely RE-KBQA, with three novel modules for knowledge base question answering, which are QAVGAE to explore the relation promotion for entity representation, multi-task learning to exploit relations for more supervisions, and SERR to postprocess relations to re-rank candidates. Extensive experiments validate the superior performance
of our method compared with state-of-the-art IRbased approaches.

heading: 7 Limitations
text: While good performance has been achieved, there are still limitations in our work. First, though QAVGAE extracts enhanced features and are fast to train, it is an independent module from the main framework. Second, as a post-processing step, the performance of SERR module on simple question is better than that of complex questions.
In the future, we would like to explore the possibility of fusing relation constraints into the representation module directly and inject strong facts identification mechanism as guidance signal of multi-hop reasoning process, aiming to integrate QA-VGAE and SERR into the main framework.

heading: Acknowledgments
text: Thanks to the anonymous reviewers for their helpful feedback. We gratefully acknowledge the insightful suggestions from Zeqi Tan. This work is supported by the China National Natural Science Foundation No. 62202182. Yong Cao is supported by China Scholarship Council (No. 202206160052) and the Zhejiang Lab’s International Talent Fund for Young Professionals.

heading: A Appendix
text: A.1 QA-VGAE Training
In this section, we introduce the details of the QAVGAE training procedure and demonstrate its effectiveness.
Training Goal. We adopt encoder-decoder models to conduct relation reconstruction tasks. Given the prepared adjacent matrix A, and feature matrix X , we use a two-layer GCN as a distribution learning model to estimate its mean and variance. The training loss function is formalized as:
LP = Eq(Z|X,A)[log p(A |Z)]−KL(q(X,A), p(Z)) (15) where Z is calculated by Equation 5, KL is the Kullback-Leibler divergence, q(·) and p(·) denotes the encoder and decoder respectively, please refer to Kipf and Welling (2016) for more details.
Settings. Specifically, A is defined as the matrix of neighborhood relations between nodes, where we set A(i, j) as 1 if there is a connection between relation ri and rj , and 0 for no connection. X is the feature matrix defined as the connectivity, which is accumulated as the number of edges between two nodes, aiming to show the importance of a relation. We set an empirical thresh of each element in the feature matrix to avoid extremely large values to hurt the model’s training, such as the degrees of “Common.type_of ” is quite huge, defined as:
X[i, j] = { τ , c ≥ τ, c, c < τ.
(16)
where c is connectivity, τ is an emprical hyperparameter, and we set τ as 2000 in our work.
A.2 SERR Algorithm
Complexity Analysis. Definitely, applying semantic similarity between relations and given questions is a more straightforward method to identify strong relations. However, the process of such a method is more complicated and time-consuming. To prove the efficiency of our method, we conduct a comparison experiment to reflect the complexity of the two methods. As is shown in Table 4, the top two rows denote semantic similarity
method, and the last row denotes our method. Obviously, our method is more lightweight without extra pre-trained models and the dependence on GPU resources. For comparison, we adopt Bertbase-uncased model to conduct the semantic similarity process in this experiment, which can be downloaded in https://huggingface.co/ bert-base-uncased.
Performance Analysis. Besides, to demonstrate it can be plug-in and infer cases quickly, we further validate its accuracy and inference time, as is shown in Table 5, Note that, since SERR relies on traditional stem extraction rather than semantic understanding to identify the key paths, there is no training period for SERR, and it can be applied to any information-retrieval(IR)-based methods.
Finally, to demonstrate the plug-in attributes of the SERR module, we integrate this module into BiNSM network (He et al., 2021) and the results are shown in Table 6. The results show that SERR can indeed increase the Hit@1/F1 score from 74.3/67.4 to 74.8/68.0 in the WebQSP dataset, and from 48.8/44.0 to 49.5/45.3 in the CWQ dataset.
A.3 Hyper-parameter Setting.
In order to help reproduce RE-KBQA and its reasoning performance, as shown in Table 7, we list the hyper-parameters of the best results on two benchmark datasets. For the WebQSP dataset, the
best results are obtained by using the initial learning rate of 0.0008, training batch size of 40, dropout rate of 0.30, reasoning step of 3, and max epoch size of 100. For the CWQ dataset, the best results are obtained by using the initial learning rate of 0.0008, training batch size of 100, dropout rate of 0.30, reasoning step of 3, and max epoch size of 200. For more experiment details, please refer to our code which will be published upon the publication of this work.
A.4 More Ablation Study
Reasoning Network. One minor modification of our work is that we adopt Transformer Encoder as a reasoning network, of its self-attention mechanism and superior capability of encoding information. As is shown in Table 8, compared with the backbone model (Linear layer), LSTM can acquire slight performance but with obviously longer training time, and Transformer Encoder can obtain promotion for KBQA task with tolerable extra training time. Therefore, different reasoning layers also affect the performance, and adopting Transformer Encoder can benefit a lot with three modules.
Training Settings. From Figure 8(a) and 8(b), we further study that 5e−4 and 3 is the best hyperparameter setting for the learning rate and reasoning step. It is worth noting that, for embeddingfused methods, the more reasoning steps are not the determinant for network performance. We conduct our experiments on 2* V100 GPUs.
A.5 More Case Analysis
In this section, we deliver more case analysis on simple questions, similarity entity identification, and the intuitive reasoning process of our method.
Simple questions. As shown in Figure 9, we show a case of one-hop reasoning on the WebQSP dataset, which proved that RE-KBQA performs well in simple question answering, as the main network can recall correct candidates and the SERR module can effectively re-rank the candidates.
Similarity entity identification. To demonstrate our method can indeed distinguish similar entities, we choose a case that needs to reason across similar entities as is shown in Figure10(a). While most of the surrounding edges are the same among candidates of the first step, our method can still select the correct node as the final answer.
RE-KBQA reasoning process. Figure 10(b) shows a three-hop reasoning case of our method, to intuitively demonstrate that our method can effectively conduct a multi-hop reasoning process. Note that, the reasoning process of our method can be illustrated as the status transfer of the relation V (t)r and candidate vectors V (t)c from one distribution into another, which is not strictly consistent along the reasoning path, thus in some degree solve the problem of knowledge base incompleteness.
.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 7

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and section 1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: ChatGPT for some writing checking.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5.2 and appendix D
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5.2 and appendix D

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we would like to explore the possibility of fusing relation constraints into the representation module directly and inject strong facts identification mechanism as guidance signal of multi-hop reasoning process, aiming to integrate QA-VGAE and SERR into the main framework