Graph Reasoning for Question Answering with Triplet Retrieval
Year: 2023

Abstract:
Answering complex questions often requires reasoning over knowledge graphs (KGs). Stateof-the-art methods often utilize entities in questions to retrieve local subgraphs, which are then fed into KG encoder, e.g. graph neural networks (GNNs), to model their local structures and integrated into language models for question answering. However, this paradigm constrains retrieved knowledge in local subgraphs and discards more diverse triplets buried in KGs that are disconnected but useful for question answering. In this paper, we propose a simple yet effective method to first retrieve the most relevant triplets from KGs and then rerank them, which are then concatenated with questions to be fed into language models. Extensive results on both CommonsenseQA and OpenbookQA datasets show that our method can outperform state-of-the-art up to 4.6% absolute accuracy.

heading: 1 Introduction
text: Answering complex questions is a challenging task since it often requires world knowledge and reasoning capability of underlying models (Li et al., 2019; Yasunaga et al., 2021; Zhang et al., 2022). Pretrained language models, e.g. BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), have shown promising results by fine-tuning on downstream question answering tasks. However, world knowledge and reasoning of these models are learned from unstructured data, e.g. Wikipedia text, and are still limited (Li et al., 2019; Petroni et al., 2019).
On the other hand, there exist large-scale knowledge graphs (KGs), e.g. Freebase (Bollacker et al., 2008) and ConceptNet (Speer et al., 2016), capturing world knowledge explicitly by triples to record relations between entities (Zhang et al., 2022). However, how to effectively integrate KGs into language models for question answering is still
∗Work was done during internship at Amazon.
an open research problem. Li et al. (2019); Yu et al. (2020); Ye et al. (2019); Zhang et al. (2019); Moiseev et al. (2022) focus on utilizing KGs to construct distant supervision signals for continuous pre-training, however, KGs are often dynamic in practice and it is often hard to edit knowledge in models without further training, limiting their usage. Bosselut et al. (2019); Wang et al. (2020) linearize reasoning paths in KGs and train language models on them to generate novel knowledge triplet during inference. However, KGs are discarded after training and language models can hallucinate false world knowledge (Ji et al., 2022).
Lin et al. (2019); Feng et al. (2020); Yasunaga et al. (2021); Zhang et al. (2022); Jiang et al. (2022); Wang et al. (2022) instead first recognize entities in questions and link them to KGs to retrieve subgraphs as additional input besides questions. However, this paradigm constrains retrieved knowledge in local subgraphs and discards more diverse triplets buried in KGs that are disconnected but useful for question answering. In addition, they require extra KG encoders with parameters trained from scratch besides standard language models, limiting model performance when training data is limited.
Recently, there have been growing interests to convert KG as a list of passages represented as natural languages. Oguz et al. (2022); Ma et al. (2021a) convert KG triples into texts and combine these KG-converted texts with heterogeneous resources, e.g. tables and unstructured Wikipedia documents, as passages and achieved state-of-the-art performance for open-domain question answering. Li et al. (2022) follow this line of work and utilize KG and unstructured documents for knowledgegrounded dialogue generation. Zha et al. (2021) linearize reasoning paths in KGs for relation prediction and achieve state-of-the-art performance.
In this paper, we propose to conduct reasoning over KGs for question answering with triplet
3366
retrieval following Oguz et al. (2022); Ma et al. (2021a); Li et al. (2022); Zha et al. (2021). The overall pipeline of proposed method is shown in Figure 1. Specifically, we first linearize KG into triplets and convert them into passages by templates and directly retrieve the most relevant ones by questions with both sparse BM25 (Robertson et al., 1994) and Dense Passage Retriever (DPR) (Karpukhin et al., 2020). We then rerank these passages by pre-trained cross-encoders (Reimers and Gurevych, 2019). Finally, most relevant passages and questions are linearly concatenated and fed into pre-trained language model for question answering. This paradigm has several advantages compared to recent state-of-the-art (Yasunaga et al., 2021; Zhang et al., 2022; Jiang et al., 2022; Wang et al., 2022): (1) it is simple yet effective and can outperform state-of-the-art complicated question answering systems up to 4.6% absolute accuracy (2) it does not need extra KG encoders trained from scratch, e.g. GNNs (Scarselli et al., 2009; Veličković et al., 2018), and simply fuses knowledge passages and questions for question answering by standard language models.

heading: 2 Methodology

heading: 2.1 Problem setup
text: We focus on multi-choice question answering (MCQA) tasks requiring model reasoning capability. Specifically, for each instance in a MCQA dataset, we have a question q and a candidate choice set C = {c1, c2, · · · , cn}. We also assume that we have access to a knowledge graph G, which provides possibly relevant knowledge to answer each question. Given an example (q, C) and a knowledge graph G, we aim to find the correct
answer c⋆ ∈ C.

heading: 2.2 Knowledge graphs as passage corpus
text: Knowledge graph G can be represented as a list of triplets P . For each triplet p ∈ P , we convert it into natural language passage d by templates so that relevant knowledge can be better retrieved. Specifically, for each triplet p ∈ P , it has a head entity h, a relation r and a tail entity t. We map r into rp and d is formed by linearly concatenating <h, rp, t>. For example, if we have a triplet <"hair brush", "AtLocation", "hair">, we map "AtLocation" into "is at location of" and form passage d as "hair brush is at location of hair". Consequently, knowledge graph G is converted into passage corpus D.

heading: 2.3 Hybrid passage retrieval
text: We retrieve passages from corpus D for an MCQA example (q, C) with hybrid (i.e., both sparse and dense) retrievers since they are complementary (Karpukhin et al., 2020; Ma et al., 2021b). For sparse retriever, we utilize BM25 to index D and (q, ci) to retrieve N passages from it. For dense retriever, we use DPR due to its strong performance in open domain question answering (Karpukhin et al., 2020). DPR embeds queries with question encoder and passages with passage encoder into low dimensional dense vectors, and retrieval can be efficiently done through FAISS library (Johnson et al., 2021) on GPUs. Similar to sparse retriever, we utilize (q, ci) to retrieve N passages from corpus D for DPR. The total number of passages returned by BM25 and DPR is 2N .

heading: 2.4 Reranking
text: We further rerank 2N passages retrieved by hybrid retriever with pre-trained cross-encoder (Reimers
and Gurevych, 2019). Specifically, for each passage p among retrieved 2N passages, we concatenate query (q, ci) and passage p as the input of pre-trained cross-encoder. For each input, it will output a scalar value between 0 to 1. These scalar values are then used as reranking scores for 2N passages.

heading: 2.5 Language model reasoning
text: After reranking, we choose top K passages PK and concatenate them along with question q and choice ci, which we cast as input of pre-trained language model (PLM). For input <q, ci, PK>, PLM will output contextual representation vector hi, which is then fed into a multi-layer perceptron (MLP) to output a scalar value si,
hi = PLM(q, ci, PK), (1)
si = MLP(hi), (2)
where si is the prediction score of choice ci to be correct. During training, we calculate score si for each choice ci ∈ C and normalize them with softmax function. After that, models are trained to maximize scores of correct choices with standard cross-entropy loss between predictions and ground truth labels. During inference, we calculate score si for each choice ci ∈ C and select the one with the highest score as the predicted answer of question q.

heading: 3 Experiments

heading: 3.1 Experimental setups
text: We evaluate our method on two question answering datasets requiring model reasoning capability.
(1) CommonsenseQA (Talmor et al., 2019) is a 5-way multi-choice question answering dataset that requires common sense reasoning. Since its test set is not publicly available, we report in-house split (Lin et al., 2019) for comparisons with baselines.
(2) OpenbookQA is a 4-way multi-choice question answering dataset requiring multi-hop reasoning on scientific knowledge (Mihaylov et al., 2018). It has 4957/500/500 questions for training/development/test set split, respectively, and we report results on its test set.
Knowledge graph. For knowledge graph, we utilize ConceptNet (Speer et al., 2016), which is a multi-relational and multi-lingual general knowledge graph storing world common sense knowledge. We first extract English triplets, clean them
following (Yasunaga et al., 2021) and convert them into natural language sentences as described in section 2.2, resulting in 2,180,391 passages after data prepossessing. We defer details of relation mapping into Appendix A.
Retrievers. For sparse retriever, we utilize implementation of BM25 from rank-bm25 python package 1 with default hyperparameters. For dense retriever, we utilize official pre-trained checkpoint 2 from DPR github repository 3.
Reranking. We rerank retrieved passages from BM25 and DPR using pre-trained cross-encoder checkpoint from sentence-transformers package 4. Specifically, for CommonsenseQA dataset, we use pre-trained checkpoint cross-encoder/ms-marco-MiniLM-L-12-v2 on MS MARCO dataset for passage ranking 5 while for OpenbookQA dataset, we use pre-trained checkpoint cross-encoder/stsb-roberta-large on semantic textual similarity task (Cer et al., 2017).
Language model reasoning. Following Yasunaga et al. (2021); Zhang et al. (2022); Jiang et al. (2022); Wang et al. (2022), we utilize RoBERTalarge (Liu et al., 2019) to reason over passages and questions although our framework is modelagnostic. Specifically, question q, choice ci and passage list PK are linearly concatenated with special tokens among them and fed into models detailed in section 2.5 to predict choice score.
We defer more implementation and training details of our method into Appendix B.
Baselines. We experiment to compare our method to various baselines with extra KG-encoders, including RN (Santoro et al., 2017), GconAttn (Wang et al., 2018), RGCN (Schlichtkrull et al., 2017), KagNet (Lin et al., 2019), MHGRN (Feng et al., 2020), QA-GNN (Yasunaga et al., 2021), GreaseLM (Zhang et al., 2022), SAFE (Jiang et al., 2022) and GSC (Wang et al., 2022). For these baselines, RoBERTa-large (Liu et al., 2019) is used for both CommonsenseQA and OpenbookQA. We also include RoBERTa-large fine-tuning only baseline without
1https://github.com/dorianbrown/rank_bm25 2https://dl.fbaipublicfiles.com/dpr/ checkpoint/retriver/multiset/hf_bert_base.cp 3https://github.com/facebookresearch/DPR 4https://www.sbert.net/docs/pretrained_ cross-encoders.html 5https://github.com/microsoft/ MSMARCO-Passage-Ranking
access to extra KG to show the effectiveness of our method. For all experiments in this work, we utilize accuracy (%) as our evaluation metric.

heading: 3.2 Main results
text: As shown in Table 1, our method can consistently outperform state-of-the-art on both CommonsenseQA and OpenbookQA datasets. For CommonsenseQA, our method’ test performance can outperform fine-tuned RoBERa-large without KG 6.28% absolute accuracy and outperform best baseline GSC with KG 0.49%. On the smaller dataset OpenbookQA, our method’ improvement is larger and can outperform fine-tuned RoBERa-large without KG 10.13% absolute accuracy and outperform best baseline GSC with KG 4.60% accuracy. Note that a key difference between our method and RoBERalarge without KG baseline is that our model also takes additional retrieved passages as input without introducing any extra parameters but can outperform various state-of-the-art methods with extra KG encoders. These consistent results indicate that our simple method can integrate knowledge into language models effectively.

heading: 3.3 Ablation study
text: We further ablate our method by removing BM25 retriever, DPR retriever and reranking module. Note that when we remove reranking module and we use the average score of BM25 and DPR if the same passage is retrieved; otherwise, following (Ma et al., 2021b), if a passage p from BM25 is not in the top N of DPR, we use the lowest score in
DPR’ top N , and vice versa. Ablation results on IHTest (Lin et al., 2019) of CommonsenseQA and test set of OpenbookQA are shown in Table 2.
When removing BM25, results on CommonsenseQA and OpenbookQA drop significantly up to 3.52%. When removing DPR, the result on CommonsenseQA drops slightly while the result on OpenbookQA drops 3.20%. These results show that BM25 and DPR are complementary, aligning with Ma et al. (2021b). Similarly, when further removing reranking module, model performance on CommonsenseQA drops 1.10% and on OpenbookQA drops 4.00% accuracy. These consistent results show the effectiveness of reranking passages retrieved from hybrid retriever. In addition, the second best model shown in our ablation study can still achieve strong performance and outperform best baseline GSC. These consistent results indicate that our model benefits from the combination of sparse and dense retrievers, and reranking module, even removing some of them can still have strong performance.

heading: 4 Conclusion
text: In this paper, we propose a simple but effective method for question answering over knowledge graphs with triplet retrieval. Extensive experiments on two datasets show that our method can consistently outperform state-of-the-art. Ablation study further shows that our model benefits from both reranking module and the combination of sparse and dense retrievers. We believe that our work can inspire future research for question answering over knowledge graphs.
Limitations
Our work is constrained into multi-choice question answering system and limited to common sense reasoning tasks, lacking more exploration in other reasoning tasks, e.g. arithmetic reasoning (Cobbe et al., 2021; Chen et al., 2021), conversational reasoning (Chen et al., 2022) and symbolic reasoning (Wei et al., 2022). We plan to leave these directions as future work.
Ethics Statement
Our work utilizes pre-trained language model and external knowledge graph to build question answering systems. However, pre-trained language models can include biases (Shwartz and Choi, 2020) and knowledge graph, e.g. ConceptNet, has been found to contain representational harms (Mehrabi et al., 2021), which can cause these question answering systems to inherit these potential biases and harms. Therefore, additional procedures, e.g. declining inappropriate inputs and filtering harmful outputs, must be taken before real-world deployment.

heading: A Relation mapping
text: We describe relation mapping with more details in Table 3, where left column shows the relation names and right column shows their corresponding templates we use to convert triplets into natural language sentences.

heading: B More implementation and training details
text: We set N = 100 for both CommonsenseQA and OpenbookQA in passage retrieval. We set K = 100 and K = 20 for CommonsenseQA and OpenbookQA, respectively, in the reranking step. Note that for CommonsenseQA, we applied additional rules after reranking stage to filter out passages. Specifically, we filter out passages that contain the "RelatedTo" relation or do not possess any token overlaps with the answer choices. We implement the method based on huggingface transformers (Wolf et al., 2020), and train the models on NVIDIA A100-SXM4-40GB GPUs. For CommonsenseQA, we use the contextualized representation of first token from PLM as h while
for OpenbookQA, we use the average contextualized representations of answer choice from PLM as h. For both CommonsenseQA and OpenbookQA datasets, we use AdamW (Loshchilov and Hutter, 2017) with learning rate 10−5. We set the maximum training epoch as 15 for CommonsenseQA and 10 for OpenBookQA. We set batch size as 32 and 16 for CommonsenseQA and OpenBookQA, respectively. The maximum sequence length is set to be 512 for CommonsenseQA and OpenBookQA. We run experiments with three different random seeds {0, 1, 2} and report mean results along with standard deviations.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement section

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract section
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 Experiments and Appendix

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3 Experiments and Appendix
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? These tools are all publicly available and we provide citations of these tools.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We use existing artifacts consistent with their intended use.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use public benchmarks for experiments.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We describe these details in section 3 Experiments.
C 3 Did you run computational experiments? Section 3 Experiments and Appendix
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We use RoBERTa-large that is publicly available in our paper and provide computing infrastructure. Details can be found in Section 3 Experiments and Appendix.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3 Experiments and Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3 Experiments and Appendix.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3 Experiment and Appendix
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We believe that our work can inspire future research for question answering over knowledge graphs. Limitations
Our work is constrained into multi-choice question answering system and limited to common sense reasoning tasks, lacking more exploration in other reasoning tasks, e. g. arithmetic reasoning (Cobbe et al. , 2021; Chen et al. , 2021), conversational reasoning (Chen et al. , 2022) and symbolic reasoning (Wei et al. , 2022). We plan to leave these directions as future work