REANO: Optimising Retrieval-Augmented Reader Models through Knowledge Graph Generation (2024)

Abstract: Open domain question answering (ODQA) aims to answer questions with knowledge from an external corpus. Fusion-in-Decoder (FiD) is an effective retrieval-augmented reader model to address this task. Given that FiD independently encodes passages, which overlooks the semantic relationships between passages, some studies use knowledge graphs (KGs) to establish dependencies among passages. However, they only leverage knowledge triples from existing KGs, which suffer from incompleteness and may lack certain information critical for answering given questions. To this end, in order to capture the dependencies between passages while tacking the issue of incompleteness in existing KGs, we propose to enhance the retrievalaugmented reader model with a knowledge graph generation module (REANO). Specifically, REANO consists of a KG generator and an answer predictor. The KG generator aims to generate KGs from the passages; the answer predictor then generates answers based on the passages and the generated KGs. Experimental results on five ODQA datasets indicate that compared with baselines, REANO1 can improve the exact match score by up to 2.7% on the EntityQuestion dataset, with an average improvement of 1.8% across all the datasets.

heading: 1 Introduction
text: The open domain question answering (ODQA) task (Voorhees and Tice, 2000) aims to answer questions with knowledge from an external corpus, such as Wikipedia. Retrieval-augmented models are an effective way of addressing the ODQA task (Zhang et al., 2023). These models employ a retriever-reader architecture that consists of both a retriever and a reader (Chen et al., 2017). The retriever model retrieves a set of passages that are relevant to the questions and then the reader model
*Corresponding Author. 1REANO code: https://github.com/jyfang6/REANO.
extracts or generates answers conditioned on the questions and the passages. The two-stage structure of ODQA systems allows for the independent optimisation of the retriever and the reader (Karpukhin et al., 2020). In this paper, we focus on enhancing the performance of the reader model, given that the effectiveness of ODQA systems primarily depends on the reader’s ability to interpret and process the information within the retrieved passages.
There has been remarkable progress in using generative readers to address the ODQA task (Sachan et al., 2021; Izacard et al., 2023). For example, Izacard and Grave (2021b) proposed a generative reader called Fusion-in-Decoder (FiD), which uses an encoder-decoder based T5 (Raffel et al., 2020) model to generate answers. By separately encoding each passage with the encoder and combining the token embeddings of these passages as inputs of the decoder, FiD could effectively utilise the knowledge from each passage to generate answers.
However, since FiD independently encodes each passage, it overlooks the semantic relationships between passages, which are critical for answering questions that require multi-hop reasoning (Ramesh et al., 2023). To this end, several studies (Yu et al., 2022; Ju et al., 2022; Hu et al., 2022; Oguz et al., 2022) proposed to leverage knowledge graphs (KGs) to establish relational dependencies among passages. Specifically, KGs store relational
2094
information between real-world entities in the form of triples ⟨head entity, relation, tail entity⟩. These knowledge triples are used to construct passage graphs (Yu et al., 2022) or entity-passage graphs (Ramesh et al., 2023) to enhance the multihop reasoning ability of the reader models. Despite the success of these models, they all directly use the triples from existing KGs such as Wikidata (Vrandečić and Krötzsch, 2014). However, these KGs often suffer from incompleteness (Cao et al., 2022) and may lack certain information critical for answering questions that are present within the passages. For example, as illustrated Figure 1, for a question “Who is younger, Steven Spielberg or James Cameron?", existing KGs may lack the birthday information of these two individuals, which is crucial to correctly answer the question. This incompleteness can hinder the reader’s ability to fully comprehend the passages and answer questions.
Therefore, in order to capture the dependencies between passages and tackling the issue of incompleteness in existing KGs, we propose to enhance the REtrieval-Augmented generative readers with a kNOwledge graph generation module (REANO). Specifically, REANO consists of a KG generator and an answer predictor. The KG generator generates a KG, which consists of a set of knowledge triples, based on the retrieved passages. Since the KG is inferred from the passages, it can effectively capture and preserve the critical information contained within these passages. After generating the KG, REANO employs the answer predictor to perform inference over the unstructured passages and the structured KG to generate answers. Our answer generator is based on the FiD model. The key difference is that it leverages a graph neural network (GNN) (He et al., 2021) to identify and select the top-K triples most relevant to the questions. These triples are concatenated in the order of relevance as an additional passage for answer generation. By combining relevant triples into a passage, we can gather information from multiple passages that are critical to answering the questions, alleviating the reasoning burden of the answer predictor. We conduct experiments on five ODQA datasets and the results indicate that compared with state-of-the-art baselines, our REANO improves the exact match (EM) score by up to 2.7% on the EntityQuestion dataset, with an average improvement of 1.8% across all the datasets.
Our contribution can be summarised as follows: (1) We propose REANO, which integrates a KG
generation module to enhance the performance of the generative readers; (2) We use a GNN to identify and select the top-K knowledge triples that are most relevant to the questions from the generated KGs and combine these triples as an additional passage for enhanced answer generation; (3) Experimental results on five ODQA datasets show that REANO can improve the EM score by up to 2.7% compared with state-of-the-art baselines.

heading: 2 Preliminaries
text: Task Formulation. Retrieval-augmented models address the ODQA task by employing a retrieverreader pipeline. In this paper, we focus on enhancing the performance of the reader model. Formally, we denote a question and its answer as q and a, respectively. Each question is associated with a set of n passages, denoted as Dq = {d1, d2, . . . , dn}, which are obtained by a retriever such as DPR (Karpukhin et al., 2020). Given a dataset O = {(q, a,Dq)}, the goal is to train a reader model pθ with parameter θ to generate answer a based on the question q and passages Dq. Fusion-in-Decoder. The Fusion-in-Decoder (FiD) model (Izacard and Grave, 2021b) is a simple yet effective generative reader model for the ODQA task. It leverages an encoder-decoder based T5 model (Raffel et al., 2020) to generate answers:
pθ(a|q,Dq) = Dec([H1; . . . ;Hn]), (1) Hi = Enc(q, di),∀i ∈ [1, . . . , n], (2)
where Enc(·) and Dec(·) represent the encoder and decoder of T5, respectively, and [· ; ·] is the concatenation operation. For each passage di ∈ Dq, FiD encodes the combined sequence of the question and the passage, i.e., Enc(q, di). The embeddings for all the passages are then concatenated as inputs to the decoder for answer generation.

heading: 3 REANO
text: The overall framework of our REANO is shown in Figure 2. This section begins with the probabilistic formulation of REANO in § 3.1, followed by its parameterisation details in § 3.2. Finally, the training strategies are introduced in § 3.3.

heading: 3.1 Probabilistic Formulation of REANO
text: We begin by formalising our REANO in a probabilistic way. Given a question q and its relevant passages Dq, the goal is to maximize the distribution pθ(a|q,Dq). In addition to direct inference on

heading: 1. <e> James Cameron </e> (born <e> August 16,

heading: 1954 </e> ) is a <e> Canadian </e> filmmaker.

heading: 2. <e> Steven Spielberg </e> (born <e> December

heading: 1. <e> James Cameron </e> (born <e> August 16,

heading: 1954 </e> ) is a <e> Canadian </e> filmmaker.
text: the passages, we propose to generate a knowledge graph Gq, which consists of entities and relations among these entities, based on these passages and then perform joint inference over both the unstructured passages and the structured knowledge graph to generate answers. Since Gq is unknown, we treat it as a latent variable and rewrite pθ(a|q,Dq) as:
pθ(a|q,Dq) = ∑ Gq pθ(a|q,Dq,Gq)pϕ(Gq|Dq), (3)
where the answer generation target pθ(a|q,Dq) is jointly modeled by a KG generator pϕ(Gq|Dq) and an answer predictor pθ(a|q,Dq,Gq). The KG generator infers a KG conditioned on the retrieved passages Dq, while the answer predictor generates answers conditioned on the question q, the passages Dq and the generated knowledge graph Gq. Next, we introduce the detailed parameterisation of the KG generator and the answer predictor.

heading: 3.2 Parameterisation

heading: 3.2.1 KG Generator
text: For each question q, the KG generator pϕ(Gq|Dq) aims to deduce a set of knowledge triples in the form of ⟨head entity, relation, tail entity⟩ based on the passages Dq. Following the established practice of KG generation (Ji et al., 2022), we decompose this task into two components: entity recognition (ER) and relation extraction (RE). The ER focuses on identifying the entities within the passages, while the RE aims to infer the relations among these entities. For the ER, we leverage SpaCy (Honnibal and Montani, 2017) to identify named entities and use TAGME (Ferragina and Scaiella, 2010), an entity linking system, to identify
Wikipedia entities. We denote the entities identified within Dq as Eq. An entity e ∈ Eq may have multiple occurrences within Dq and we refer to each instance of this entity in the passages as a mention.
Subsequently, we infer the relations among entities. Given that current state-of-the-art RE models rely on a transformer backbone (Lu et al., 2022a,b), it is impractical to infer the relations among all the entities simultaneously. This is due to the constrained maximum input length of these models, as processing all the passages at once would exceed this limit. Therefore, we further decompose the RE component into two sub-modules: intra-context RE and inter-context RE.
Intra-Context RE. Intra-context RE focuses on extracting the relations among entities within a single passage, while inter-context RE aims to extract the relations among entities across passages.
The intra-context RE model is defined as: pϕ(T Iq |Dq, Eq) = ∏
di∈Dq pϕ(T Iq,di |di, Eq,di), (4)
where T Iq,di denotes intra-context relation triples among entities Eq,di within a passage di. We instantiate the intra-context RE model pϕ with DocuNet (Zhang et al., 2021a), an effective RE model capable of predicting relations between every entity pair within a passage in a single forward propagation. More details about the framework and the parameterisation of DocuNet are provided in Appendix A.
Inter-Context RE. For inter-context RE, we leverage the Wikidata API2 to retrieve relations among
2https://www.wikidata.org/w/api.php
entities across passages from the Wikidata (Vrandečić and Krötzsch, 2014). Following previous work (Yu et al., 2022), we include all the relations between entities even if they are not grounded in the passages to construct inter-passage connections.
Therefore, the generated KG for the question q is obtained by combining the intra-context relation triples T Iq and the inter-context relation triples T Cq , i.e., Gq = {T Iq , T Cq }.

heading: 3.2.2 Answer Predictor
text: For a question, the answer predictor pθ(a|q,Dq,Gq) generates answers based on the passages and the generated KGs. Our answer predictor is based on the FiD model. However, the key difference between our answer predictor and FiD is that our answer predictor uses a graph neural network (GNN) to select a set of triples relevant to the question q from Gq, given that most of the triples in Gq may be irrelevant to the question. These selected triples are combined into an additional passage for answer generation. Formally, we denote the set of triples relevant to the question q as T Rq ⊂ Gq, and define the probability of each triple being relevant to the question as pθ(T Rq |q,Dq,Gq). Empirically, we found that simply selecting top-K triples with the highest probability could consistently yield excellent performance. Therefore, T Rq is specified as the set of top-K triples with the highest values in pθ(T Rq |q,Dq,Gq).
Given the relevant triples T Rq , the answer distribution is then defined as:
pθ(a|q,Dq,Gq) = pθ(a|q,Dq, T Rq ), (5)
where we use the selected relevant triples T Rq instead of all the triples Gq to generate answers. In what follows, we will introduce pθ(TqR|q,Dq,Gq) and pθ(a|q,Dq, T Rq ) in details.
Graph Neural Network. We begin by introducing how to identify relevant triples using GNN (He et al., 2021), i.e., pθ(TqR|q,Dq,Gq). To this end, we first initialise the entity and relation embeddings. The embedding of an entity is initialised with the contextualised embeddings of its mentions. Specifically, for each passage di ∈ Dq, we insert special symbols “<e>” and “</e>” at the start and end of all entity mentions to indicate the presence of an entity. We then use the encoder of T5 to obtain the token embeddings of the passages: Hi = Enc(q, di),∀i = 1, . . . , n. Following
previous works (Verga et al., 2018; Zhang et al., 2021a), we use the embeddings of the “<e>” tokens to represent mention embeddings, which are subsequently mean-pooled to obtain entity embeddings: te =
1 Ne ∑Ne j=1me,j , ∀e ∈ Eq, where Ne denotes
the number of mentions of entity e within Dq and me,j is the j-th mention embedding of e.
Next, we introduce how to obtain relation embeddings. For a knowledge triple ⟨e, rev, v⟩ in Gq, e.g., ⟨Steven Spielberg, occupation, director⟩, we employ the encoder of T5 to obtain token embeddings for the relation label, i.e., occupation. These embeddings are mean-pooled to get the initial relation embeddings, denoted as r̂ev. Since the relations between entities are predicted by the KG generator, which might contain potential inaccuracies, we introduce a relation embedding module. This module aims to refine relation embeddings by integrating entity embeddings, thereby enhancing the reliability and accuracy of the relation embedding for rev, which is given by:
rev = r̂ev + REM([te; tv]), (6)
where REM(·) is the relation embedding module, instantiated as a two-layer feed-forward neural network in our model.
Subsequently, we use an L-layer GNN network to update the entity embeddings. Inspired by previous relation-aware GNNs for the knowledge base question answering task (Vashishth et al., 2020; He et al., 2021), we define the process of updating entity embeddings at the l-th layer of our GNN as:
t(l)e = FFN([t (l−1) e ; s (l) e ]), (7) s(l)e = ∑
(v,rev)∈N (e) αrevv · FFN([t(l−1)v ; rev]), (8)
αrevv = w⊤(q ⊙ rev)∑
(v′,rev′ )∈N (e)w ⊤(q ⊙ rev′)
, (9)
where t(l)e denotes the entity embedding of e at the l-th layer with t(0)e = te, FFN(·) denotes a feed-forward neural network layer, N (e) is a set of neighboring triples of e for its outgoing edges, q is the embedding of the question q obtained by taking the average of its token embeddings, ⊙ is the element-wise multiplication and w is a learnable parameter. Specifically, for each entity e, the GNN fuses the aggregated message s(l)e from the entity’s neighbouring triples to update the entity’s embedding, i.e., Equation (7).
This message is a weighted aggregation of the information from each neighbouring triple, i.e., Equation (8), with the weights αrevv being determined by the similarity between the question and the relation of the triple, i.e., Equation (9). This GNN allows entities to attend to triples that are more relevant to the question, thereby obtaining more accurate and question-specific entity embeddings.
Top-K Relevant Triple Selection. After updating the entity embeddings, we calculate the similarities between a triple ⟨e, rev, v⟩ ∈ Gq and the question q as follows:
pθ(Tq|q,Dq,Gq) ∝ q⊤t(L)e + q⊤rev + q⊤t(L)v . (10)
Based on this similarity measure, we identify and select top-K triples that exhibit the highest relevance to the question, i.e., T Rq . We simply concatenate these top-K triples in the descending order of the similarities as a passage denoted as dT Rq . The purpose of combining triples is to gather information from multiple passages that are helpful in answering questions, thereby alleviating the reasoning burden. This passage is then encoded with the T5 encoder. The resulting embeddings are concatenated with the passage embeddings as inputs to the T5 decoder to generate answers:
HT Rq = Enc(q, dT Rq ), (11)
pθ(a|q,Dq, T Rq ) = Dec([H1; . . . ;Hn;HT Rq ]). (12)

heading: 3.3 Training Strategies
text: The goal of training REANO is to find the parameters for the KG generator and the answer predictor that can maximize the log-likelihood of the training data, i.e., log pθ(a|q,Dq). Since the end-to-end optimisation of both the KG generator and the answer predictor is non-differentiable, similar to the previous work (Zhang et al., 2022), we decouple the training of REANO by first training the KG generator and then the answer predictor based on the KGs obtained from the KG generator.
Distantly Supervised Training of KG Generator. In the KG generator, only the DocuNet model needs to be trained for intra-context RE. Given the absence of gold labels for this task, we employ a distantly supervised training approach (Mintz et al., 2009; Zhang et al., 2021b). This involves training the DocuNet model on the REBEL dataset (Cabot and Navigli, 2021) and directly using the trained model to predict intra-context relation triples for
each passage within Dq. Particularly, the REBEL dataset is a large-scale RE dataset built by aligning Wikipedia abstracts and the knowledge triples in the Wikidata. Each item in this dataset consists of a Wikipedia text paired with some knowledge triples that can be inferred from the text. More details about the dataset and DocuNet training are provided in Appendices B.1 and B.2. In addition, we also investigate creating a RE training dataset for each ODQA dataset by extracting Wikidata triples within each passage. The generated data are used either to finetune the DocuNet model trained on the REBEL dataset or to train the DocuNet model from scratch. However, our findings indicate that neither of these two approaches can further improve the performance (see Appendix C.3).
Training of Answer Predictor. In the answer predictor, we need to train the GNN model for selecting the top-K relevant triples and the T5 model for generating answers. In the ODQA datasets, we observe that sometimes the answer to a question q can match one of the entities identified within its retrieved passages Dq. Based on this observation, we use such answer-entity alignment as supervision signals to train the GNN model. Specifically, we identify all the paths connecting each entity in the question to the answer entity in the generated KG Gq. We consider all the entities that are part of these paths as relevant to the question, denoted as Erelq ⊂ Eq. If such paths do not exist, we use the answer entity as the relevant entity. We then add a linear classifier on top of the GNN model to predict which entities are relevant to the question: cq = Softmax(E (L) q wc), where E (L) q denotes the embedding matrix of all the entities (i.e., Eq), wc is a learnable parameter and cq denotes the probability of each entity being relevant to the question. Following Zhang et al. (2022), the GNN model is trained by minimising: Lgnn = DKL(cq||c∗q), where c∗q denotes the ground-truth relevant entity distribution computed with Erelq , and DKL denotes the Kullback–Leibler divergence.
The T5 model is trained with the cross entropy loss between the predicted answer distribution and the true answer distribution, which is denoted as Lt5 = CE(pθ(a|q,Dq, Tq), p∗(a|q)). Hence, the loss for training the answer generator is:
Lanswer = Lt5 + βLgnn, (13)
where β is a trade-off hyperparameter between the T5 loss and the GNN loss.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datasets. We conduct experiments on five ODQA datasets: Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (TQA) (Joshi et al., 2017), EntityQuestions (EQ) (Sciavolino et al., 2021), 2WikiMultiHopQA (2WQA) (Ho et al., 2020) and MuSiQue (Trivedi et al., 2022). For NQ and TQA, we use the data splits provided by Karpukhin et al. (2020). Each question in NQ and TQA is associated with a set of 100 passages, which are retrieved by the DPR (Karpukhin et al., 2020) model from the December 20, 2018 Wikipedia dump that contains 21M passages. The same data are also used in our baselines, except RAG-Seq. For EQ, following Sciavolino et al. (2021), we use the same 21M Wikipedia dump as the corpus and employ BM25 (Robertson and Zaragoza, 2009) to retrieve top-20 relevant passages for each question. The 2WQA and MuSiQue are multi-hop QA datasets that require reasoning over multiple passages to answer questions. Each question in 2WQA and MuSiQue is associated with 10 and 20 passages respectively provided by the authors. We directly use these passages in our experiments. More details about the datasets are provided in Appendix B.1.
Baselines. Since REANO is built under the FiD framework (Izacard and Grave, 2021b), we mainly compare it with the FiD model and its variants. In particular, we compare with models from each of the following categories: (1) extractive reader: DPR (Karpukhin et al., 2020); (2) generative reader: RAG-Seq. (Lewis et al., 2020b), FiDO (de Jong et al., 2023); (3) KG-enhanced reader: KG-FiD (Yu et al., 2022), OREOLM (Hu et al., 2022), GRAPE (Ju et al., 2022). Due to resource constraints, both our REANO and baselines use the base versions of the transformer models.
Evaluation. Following Izacard and Grave (2021b), we use greedy decoding to generate answers and employ the Exact Match (EM) as the evaluation metric, where a predicted answer is considered correct if it matches any answer in a list of gold answers after normalisation (Yu et al., 2022).
Training and Hyperparameter Details. For the KG generator, we use the default architecture and hyperparameters as in Zhang et al. (2021a) for intracontext RE. The statistics of the generated KGs for each dataset are provided in Table 4 of the Appendix. For the answer predictor, we use t5-base

heading: Models NQ TQA EQ 2WQA MuSiQue
text: as the backbone model and set the number of GNN layers L as 3. Due to the resource constraints, we only use the top-50 passages for each question in both NQ and TQA datasets and use all the passages per question in the other datasets. Throughout the experiments, we set the number of triples selected by the GNN K as 10. We use AdamW (Loshchilov and Hutter, 2019) with a constant learning rate of 1e-4 as the optimizer and set the batch size as 64. More training details are provided in Appendix B.2.

heading: 4.2 Results and Analysis
text: We provide our main results in this section and additional experimental results in Appendix C.
(RQ1): How does our REANO perform against the baselines? The performance of REANO and baselines is reported in Table 1, which shows the EM scores (%) of different models across different datasets. The results yield the following findings: (1) First, our REANO consistently achieves the best performance on all the datasets. Compared with the second best models on each dataset, REANO achieves improvements of 1.7%, 2.7% and 2.5% on the TQA, EQ and 2WQA datasets, respectively. REANO also achieves an average improvement of 1.8% across all the datasets. These results demonstrate the effectiveness of REANO for ODQA. (2) Moreover, when compared with existing KGenhanced FiD-based readers, i.e., KG-FiD, OREOLM and GRAPE, on the NQ and TQA datasets, REANO still achieves the best performance. The suboptimal performance of prior KG-enhanced readers may be due to the fact that they only leverage knowledge triples from existing KGs, which
3These results are obtained using at least 50 passages except RAG-Seq, which only uses 10 passages.
may lack some information critical to answer questions. Therefore, this result indicates the effectiveness of generating knowledge triples from passages used by our REANO model, complementing the knowledge triples from existing KGs. (3) Furthermore, if we remove the knowledge triples in REANO, REANO is equivalent to FiD. When compared with FiD, REANO demonstrates significant (using a paired sample t-test) improvements of 2.9%, 3.0% and 1.9% on the EQ, 2WQA and MuSiQue datasets, respectively, and also exhibits superior performance on the other two datasets. This indicates the effectiveness of using knowledge triples as additional context to enhance the reader’s understanding of passages.
(RQ2): Does different components of our model affect the performance? We conduct ablation studies to study the effects of different components in REANO, including different sets of knowledge triples, the REM module and the GNN module.
First, to investigate the effects of different triples, we introduce two variants of REANO: (1) w/o intercontext triples, where we remove the inter-context triples and only use the intra-context triples as inputs to the answer predictor; (2) w/o intra-context triples, where we only pass the inter-context triples to the answer predictor. The results for ablation studies are reported in Table 2, which shows the performance (EM %) of REANO and its variants on
the EQ, 2WQA and MuSiQue datasets. The results show that both w/o inter-context triples and w/o intra-context triples perform significantly worse than REANO on these datasets, which indicates that both the inter-context triples and the intracontext triples are necessary for improving the performance. This is because the intra-context triples can capture the relations among entities within the same passages and the inter-context triples can capture the relations among entities across different passages. The combination of these two sources of information can help the reader better understand the information within all the passages.
Moreover, to investigate the effects of the REM and the GNN modules, we additionally introduce two variants: w/o REM and w/o GNN, which are obtained by removing the REM module and the GNN module in the answer predictor, respectively. The results in Table 2 indicate that removing either of these two components would significantly degrade the performance on the EQ, 2WQA and MuSiQue datasets. This is because, with the REM module, the answer predictor can learn to refine relation embeddings based on the contextualised embeddings of entities, thereby enhancing the accuracy of relation embeddings. Moreover, the GNN can perform multi-hop reasoning over the KGs to better identify and select knowledge triples that are relevant to the questions, leading to improved performance.
(RQ3): Can knowledge triples help to reduce the number of passages while maintaining satisfactory performance? As shown in Figure 2, we aim to generate and select knowledge triples that contain critical information within the passages to answer questions. To investigate the effects of these knowledge triples, we study the performance of REANO on the NQ and TQA datasets under different number of passages. Moreover, we introduce two different settings of knowledge triples for
Table 4: Case study of REANO on the 2WQA dataset, where “Top-Ranked Triples” denotes the top-5 knowledge triples selected by the GNN module from the knowledge triples generated by the KG generator.
way Station, country, Pakistani” that indicates the county of the station, based on the sentence “Raisan railway station is located in Pakistan”. Moreover, the results also reveal that our GNN module can identify triples that are useful to answer the questions. For example, for the question “Are both stations, Muzaffargarh Railway Station and Raisan Railway Station, located in the same country?”, the top-2 triples “hMuzaffargarh Railway Station, country, Pakistani” and “hRaisan Railway Station, country, Pakistani” are highly relevant to the question. Therefore, the case study indicates that REANO can not only generate meaningful knowledge triples from passages but also identify triples that are useful to answer questions.
5 Related Work
Retrieval-Augmented Models for ODQA. Recent studies focus on leveraging retrieval-augmented models to address the ODQA (Zhang et al., 2023), where a retriever is used to obtain relevant information from Wikipedia (Chen et al., 2017; Lewis et al., 2020b; Karpukhin et al., 2020) and a reader is used to extract (Kedia et al., 2022) or generate (Izacard and Grave, 2021b) answers. There are three lines of work to enhance the performance of these models: (1) Improve the retriever: compared with sparse retrieval models, such as BM25 (Robertson et al., 1994), dense retrievers (Karpukhin et al., 2020), which are based on the contextualised embeddings, have shown superior retrieval performance. Existing works have focused on improving dense retrievers using hard negatives (Qu et al., 2021), knowledge distillation (Izacard and Grave, 2021a),
reranking (Mao et al., 2021) or supervision from large language models (LMs) (Shi et al., 2023). (2) Improve the reader: compared with extractive reader (Karpukhin et al., 2020), generative readers are more effective in predicting answers (Izacard and Grave, 2021b; Lewis et al., 2020b; Cheng et al., 2021; Borgeaud et al., 2022; de Jong et al., 2023). (3) Some works have joint optimised the
retriever and the reader to mutually enhance each
other (Guu et al., 2020; Sachan et al., 2021; Izac-
ard et al., 2023). Furthermore, some recent studies also proposed to use LLMs to generate, rather than retrieve, relevant contexts to answer questions (Yu et al., 2023; Abdallah and Jatowt, 2023; Frisoni et al., 2024). In contrast, our work uses a KG generator to model the relationships among different passages.
KG-Enhanced Retrieval-Augmented Models for ODQA. KGs have been previously used to enhance the retrieval-augmented models for ODQA (Min et al., 2019; Zhou et al., 2020; Oguz et al., 2022; Yu et al., 2022; Ju et al., 2022; Hu et al., 2022; Ramesh et al., 2023). In particular, UniK-QA (Oguz et al., 2022) converts triples into texts and combine them into text corpus. KG-FiD (Yu et al., 2022) uses KGs to construct passage graphs for passage reranking and Grape (Ju et al., 2022) fuses the KG and contextual representations into the hidden states of the reader. However, these models only use knowledge triples from existing KGs, which often suffer from incompleteness. In contrast, our REANO uses a KG generator to generate knowledge triples from passages and uses a GNN to identify and select the most relevant triples, which can effectively capture
comparison: (1) n passages + Triples from 50 passages (n50), where we use the token embeddings of the top-n passages and the token embeddings of triples selected from all the 50 passages as inputs to the decoder for answer generation; (2) n passages + Triples from n p ssages (nn), where we use the toke embed ings of triples selected from these n pas ages as inputs to the decoder. The results in Figure 3 show that under the n50 setting REANO does not exhibit significant performance decli e until the number of passages is reduced to 20, whi h indicates that knowledge triples can help to reduce the number of passages. Moreover, when comparing the performance of REANO under the n50 and nn settings, the results reveal that, in the nn setting, REANO suffers from a more substantial decline in performance when the number of passages is reduced from 50 to 1. This can be explained by the fact that the knowledge triples selected from all the 50 passages provide more useful information to answer the question, thereby leading to better performance. This result also suggests that, under the FiD framework, it is possible to use knowledge triples to represent some context passages, which is particularly useful for language models with limited context length.
(RQ4): How does the hyperparameter β affect the performance of REANO? In Equation (13), we introduce a trade-off hyperparameter β to balance the T5 loss and the GNN loss during training. To investigate its effects on the performance of REANO, we conduct experiments by varying the value of β from 1e-4 to 1.0 and examining the corresponding performance. The results are provided
10 4 10 3 10 2 10 1 10073 74 75 76 77 78 EM (% ) 74.2 74.0 75.9 77.2 74.7 REANO 10 4 10 3 10 2 10 1 100 28 30 32 34 31.0 34.1 32.9 31.9 28.1 REANO
Figure 4: Performance (EM %) of our REANO under different values of β on the 2WQA (left) and the MuSiQue (right) datasets.
in Figure 4, which shows the EM scores of REANO under different values of β on the 2WQA and the MuSiQue datasets. The results show that as we increase the value of β fr m 1e-4 to 1.0, the p rformance of REANO initially improves and then declines on both datasets. However, the optimal value of β varies betw en d tasets, with 0.1 being ptimal for 2WQA and 1e-3 for MuSiQue. Therefore, the experimental results indicate the trade-off hyperparameter β can effectively balance the T5 loss and the GNN loss. Additionally, the optimal value for β is dataset-dependent, with different datasets requiring different values for optimal performance.
Case St dy. Finally, we conduct case study to i vestigate if REANO can g nerate and identify riples that ar helpful in answering questions. The
sults are provided in Table 3, which sh s three examples from 2WQA test set4. The examples show that our KG generator can generate meaningful knowledge triples from the passages. For example, it generates a inter-context triple “⟨Raisan Rail-
4The complete passages and triples of these examples and a few other examples can be found in Appendix C.7.
way Station, country, Pakistan⟩” that indicates the county of the station, based on the sentence “Raisan railway station is located in Pakistan”. Moreover, the results also reveal that our GNN module can identify triples that are useful to answer the questions. For example, for the question “Are both stations, Muzaffargarh Railway Station and Raisan Railway Station, located in the same country?”, the top-2 triples “⟨Muzaffargarh Railway Station, country, Pakistan⟩” and “⟨Raisan Railway Station, country, Pakistan⟩” are highly relevant to the question. Therefore, the case study indicates that REANO can not only generate meaningful knowledge triples from passages but also identify triples that are useful to answer questions.

heading: 5 Related Work
text: Retrieval-Augmented Models for ODQA. Recent studies focus on leveraging retrieval-augmented models to address the ODQA (Zhang et al., 2023), where a retriever is used to obtain relevant information from Wikipedia (Chen et al., 2017; Lewis et al., 2020b; Karpukhin et al., 2020) and a reader is used to extract (Kedia et al., 2022) or generate (Izacard and Grave, 2021b) answers. There are three lines of work to enhance the performance of these models: (1) Improve the retriever: compared with sparse retrieval models, such as BM25 (Robertson et al., 1994), dense retrievers (Karpukhin et al., 2020), which are based on the contextualised embeddings, have shown superior retrieval performance. Existing works have focused on improving dense retrievers using hard negatives (Qu et al., 2021), knowledge distillation (Izacard and Grave, 2021a), reranking (Mao et al., 2021) or supervision from large language models (LMs) (Shi et al., 2023). (2) Improve the reader: compared with extractive reader (Karpukhin et al., 2020), generative readers are more effective in predicting answers (Izacard and Grave, 2021b; Lewis et al., 2020b; Cheng et al., 2021; Borgeaud et al., 2022; de Jong et al., 2023). (3) Some works have joint optimised the retriever and the reader to mutually enhance each other (Guu et al., 2020; Sachan et al., 2021; Izacard et al., 2023). Furthermore, some recent studies also proposed to use LLMs to generate, rather than retrieve, relevant contexts to answer questions (Yu et al., 2023; Abdallah and Jatowt, 2023; Frisoni et al., 2024). In contrast, our work uses a KG generator to model the relationships among different passages.
KG-Enhanced Retrieval-Augmented Models for ODQA. KGs have been previously used to enhance the retrieval-augmented models for ODQA (Min et al., 2019; Zhou et al., 2020; Oguz et al., 2022; Yu et al., 2022; Ju et al., 2022; Hu et al., 2022; Ramesh et al., 2023). In particular, UniK-QA (Oguz et al., 2022) converts triples into texts and combine them into text corpus. KG-FiD (Yu et al., 2022) uses KGs to construct passage graphs for passage reranking and Grape (Ju et al., 2022) fuses the KG and contextual representations into the hidden states of the reader. However, these models only use knowledge triples from existing KGs, which often suffer from incompleteness. In contrast, our REANO uses a KG generator to generate knowledge triples from passages and uses a GNN to identify and select the most relevant triples, which can effectively capture critical information within the passages.

heading: 6 Conclusion
text: This paper proposes REANO to address the ODQA task. REANO aims to capture dependencies among passages with a knowledge graph generation module. Specifically, it consists of a KG generator, which generates KGs based on the passages, and an answer predictor, which generates answers based on both the passages and the generated KGs. The answer predictor is based on the FiD model, with an additional GNN model to identify and select knowledge triples relevant to questions. Experimental results on five ODQA datasets indicate that REANO can improve the EM score by up to 2.7% compared with state-of-the-art baselines.

heading: Limitations
text: This work focuses on improving the performance of retrieval-augmented readers with a knowledge graph generation module. We have verified the effectiveness of REANO using an encoder-decoder based T5 model. However, the performance of REANO with other generative readers, such as BART (Lewis et al., 2020a) and the decoder-only generative models (Brown et al., 2020; Touvron et al., 2023) remains unexplored, and we defer this exploration to future work. Moreover, we leverage a frozen retriever in our experiments to simplify the setting and do not investigate how the passage retriever’s effectiveness would affect the performance of REANO. We also leave the exploration of how to jointly optimise the retriever and our REANO for better performance as our future work.

heading: B Experimental Details

heading: B.1 Datasets
text: In this section, we first introduce the details of the REBEL dataset, which is used to train the DocuNet model for intra-context RE. Subsequently, we introduce the details of the QA datasets used in our experiments, the statistics of which are summarised in Table 4.
REBEL Dataset (Cabot and Navigli, 2021): REBEL dataset is a large-scale relation extraction dataset proposed to pretrain the REBEL model for extracting relation triples from texts (Cabot and Navigli, 2021). This dataset is built by aligning Wikipedia abstracts and the knowledge triples in Wikidata. Specifically, it first identifies entities within Wikipedia abstracts using wikimapper5 and then extracts relations present between those entities in Wikidata. Moreover, in order to filter out some relations irrelevant to the text, it further uses the entailment prediction of a RoBERTa model (Liu et al., 2019) to filter those relations not entailed by the Wikipedia text. As a result, each example in the REBEL dataset is a Wikipedia text along with the Wikidata knowledge triples that can be inferred from the text. The original training, development and test of the REBEL dataset contain approximately 2M, 152K and 515K examples, respectively.
We aim to train the DocuNet model with the REBEL dataset for intra-context relation extraction. To define the label set for the DocuNet model, we exclude relations that appear less than 100 times across all triples in the dataset and keep the remaining relations as the label set R. This filtering process aims to make the DocuNet model focus on more frequent and informative relations. The number of relations in R is 472. We then exclude triples whose relations do not exist in the relation set R from the dataset. Moreover, we exclude examples with less than 3 entities or less than 3 triples. As a result, the processed REBEL dataset contains about 1M, 3K and 3K examples in the training, development and test set, respectively.
Natural Questions (NQ) (Kwiatkowski et al., 5https://pypi.org/project/wikimapper/
2019). NQ is a commonly used dataset in the ODQA task. It contains questions from the Google search engine and the answers are paragraphs or entities, which are annotated by human annotators, in the Wikipedia page of the top 5 search results. We use the same train/dev/test splits as in Karpukhin et al. (2020), which also provide the top-100 passages relevant to each question retrieved by the DPR model from a corpus. The corpus is obtained from the Dec. 20, 2018 Wikipedia dump, which is split into passages of approximately 100 words. This preprocessing step results in a retrieval corpus containing about 21 million passages.
TriviaQA (TQA) (Joshi et al., 2017). TQA includes question-answer pairs constructed by trivia enthusiasts. This dataset has relatively complex and compositional questions, which require reasoning over multiple sentences to obtain the answers. We also use the train/dev/test splits and the top-100 relevant passages provided by Karpukhin et al. (2020) in our experiment. These passages are also retrieved by the DPR model from the same Wikipedia corpus used in NQ.
EntityQuestions (EQ) (Sciavolino et al., 2021). EQ contains simple and entity-centric questions, which are constructed based on the facts from Wikidata. Each question in EQ focuses on a particular aspect of an entity. We employ the train/dev/test splits provided by the authors. Moreover, following the original work, we leverage the same Wikipedia corpus used in NQ and TQA as the retrieval corpus. For each question, we leverage the Pyserini BM25 (Lin et al., 2021) to retrieve top-20 relevant passages. The reason we use BM25 to retrieve relevant passages is that the results from the original paper demonstrate that BM25 can achieve better retrieval performance than dense retrieval methods such as DPR in this dataset.
2WikiMultiHopQA (2WQA) (Ho et al., 2020). 2wikimultihopQA is a multi-hop question answering dataset that requires reading multiple passages

heading: Parameter DocuNet Answer Generator
text: to answer a given question. It is constructed using evidence information containing a reasoning path for the multi-hop questions. Since the test set of 2WQA is not publicly available, following Ramesh et al. (2023), we treat the original dev set as the test set and we report performance on this set. Moreover, we randomly select 1000 examples from the original training set as the dev set and use the remaining examples for training. Given that each question in this dataset contains 10 contexts, which are retrieved from Wikipedia using bigram TF-IDF, for answering the given question, we directly use these contexts in our experiments. Note that this dataset is constructed in a distractor setting, where the passages contain some distractors that are not useful to answer the questions and the passages are randomly shuffled.
MuSiQue (Trivedi et al., 2022). MuSiQue is also a multi-hop question answering dataset that requires 2-4 hops reasoning. This dataset is constructed using a bottom-up technique, where it iteratively combines single-hop questions from multiple datasets into a k-hop benchmark. Since the test set of MuSiQue is also not publicly available, we adopt the same dataset splitting strategy as 2WQA in our experiment. Moreover, each question in this dataset contains 20 contexts, which include passages that are helpful to answer the given question and some distractor passages that are irrelevant to the question. We also directly employ these contexts in our experiments.

heading: B.2 Training and Hyperparameter Settings
text: We summarise the training details used in our REANO in Table 5. In order to train the DocuNet model for intra-context relation extraction, we follow the default architecture and hyperparameter settings in the original paper (Zhang et al., 2021a). We train the DocuNet model from scratch on our preprocessed REBEL dataset. The trained DocuNet model is directly used to extract relations among entities in the passages of different QA datasets.
For the answer predictor, we use the following settings on all the ODQA datasets unless otherwise specified. In particular, we use T5-base as the backbone model and leverage a 3-layered GNN to select the top-K relevant knowledge triples from the generated KGs, where the K is set as 10. Moreover, we set the trade-off hyperparameter β as 0.1. We train the answer predictors on different ODQA datasets using the settings in Table 5. These hyperparameters are chosen based on previous works (Izacard and Grave, 2021b; Yu et al., 2023). However, given the different training sizes of the EQ, 2WQ and MuSiQue, we train the answer predictors on these datasets for 15, 000, 15, 000 and 3, 000 steps, respectively. Furthermore, when optimising the GNN loss, we ignore questions where the answers could not match any entities identified within their corresponding passages (the statistics are provided in Table 4). These questions are only used for calculating the T5 loss.
During training, we evaluate our answer predictor on the development set every 500 steps and select the checkpoint with the best EM scores on the development set as the final model for evaluation. We report the performance on the test sets of different ODQA datasets.

heading: C Additional Experimental Results
text: In this section, we first introduce the performance of REANO using the T5-large model in § C.1. We then introduce the performance of REANO with different orders of knowledge triples in § C.2, as well as its performance with different variants of the DocuNet model in § C.3. Subsequently, we introduce the effect of the number of triples K in § C.4, and the effect of the number of GNN layers L in § C.5, respectively. In addition, we also investigate the effect of jointly optimising the GNN model and the T5 model in § C.6. Finally, we provide the results of the case study in § C.7.

heading: Models EQ 2WQA MuSiQue

heading: Models EQ 2WQA MuSiQue

heading: C.1 Performance of REANO with T5-Large
text: Due to the resource constraints, we mainly report the performance of REANO and baselines using the base versions of the corresponding transformer models. To investigate if the conclusions can be generalised to large models, we additionally compare the performance of REANO and FiD when using T5-large as the backbone model on datasets with a relatively small number of passages, i.e., EQ, 2WQA and MuSiQue. Experimental results are provided in Table 6, which shows the EM scores (%) of REANO and FiD on the EQ, 2WQA and MuSiQue datasets. The results show that our REANO can still achieve the best performance on all the three datasets with an average improvement of 3.1%, which indicates the effectiveness of the proposed REANO.

heading: C.2 Effect of the Order of Knowledge Triples
text: In our REANO, we combine the sequence of knowledge triples in the descending order of their similarities to the questions as an additional passage. In order to investigate if the order of knowledge triples would affect the performance, we introduce a variant of REANO: “REANO w. Random Order”, where the knowledge triples are combined in a random order. Experimental results in Table 7 provide the EM scores (%) of REANO and its variant on the EQ, 2WQA and MuSiQue datasets. The results show that compared with REANO, the performance of “REANO w. Random Order” is slightly worse on the EQ and the MuSiQue datasets, and is significantly worse on the 2WQA dataset. Therefore, the results indicate the the order of knowledge triples would affect the performance and combining knowledge triples in the descending order of simi-
larities is beneficial to improve the performance.

heading: C.3 Performance of REANO with Different DocuNet Models
text: In REANO, we train the DocuNet model using a distantly supervised training approach, where the model is trained on the REBEL dataset and then directly used to extract intra-context relation triples for different ODQA datasets. To investigate the effectiveness of such an approach, we introduce two variants of REANO: (1) REANO with finetuned DocuNet: in this variant, we aim to finetune the DocuNet model trained on the REBEL dataset on each ODQA dataset. To achieve this, for each passage within an ODQA dataset, we extract knowledge triples among entities within the passage from Wikidata. The data is processed in the same way as the REBEL dataset (see Appendix B.1 for details). We then use the processed data to finetune the DocuNet model. (2) REANO with trained DocuNet: in this variant, we train the DocuNet model on each ODQA dataset from scratch. Experimental results are provided in Table 8, which shows the performance of REANO under different variants of the DocuNet model on the 2WQA and MuSiQue datasets. The results indicate that finetuning or training the DocuNet model on ODQA datasets does not lead to performance improvement and may even result in a decrease in performance. We conjecture that this occurs because both the REBEL dataset and the passages in ODQA datasets are obtained from Wikipedia, meaning they share the same data distribution. Therefore, the DocuNet model trained on the REBEL dataset can effectively generalise to the ODQA datasets.

heading: C.4 Effect of the Number of Triples
text: We conduct experiments to investigate the effect of the number of triples K on the performance of REANO. Specifically, we vary the number of triples K from 1 to 30 and report the corresponding performance. Experimental results are provided in Figure 5, which shows the EM scores of our REANO

heading: Number of GNN Layers 2WQA MuSiQue
text: under different number of triples on the 2WQA and MuSiQue datasets. The results illustrate that for both datasets, as we increase the value of K from 1 to 15, REANO consistently demonstrates significant performance improvement and achieves the best performance when K = 15. However, beyond this point, increasing the number of triples does not lead to further performance improvements but even slightly decrease the performance. This is because our REANO uses a GNN module to select and rank the generated knowledge triples. The top-15 triples can provide sufficient information to answer the questions and, including more knowledge triples would introduce noises, leading to decreased performance. Therefore, the results indicate the effectiveness of the GNN module in identifying and selecting knowledge triples helpful to answer the questions.

heading: C.5 Effect of the Number of GNN Layers
text: The GNN module in the REANO is responsible for identifying and selecting knowledge triples relevant to the given questions. We additionally conduct experiments to investigate the effect of the number of GNN layers L. Specifically, we vary the number of GNN layers from 1 to 4 and report the corresponding performance in Table 9, which shows the EM scores of REANO with different number of GNN layers. The results indicate that as we increase the value of L from 1 to 4, the performance of REANO initially increases and then
declines. Notably, the performance of REANO with 1-layer GNN is worse than the other settings on both datasets. This is because the GNN module uses neighbourhood aggregation to perform multihop reasoning over the generated KGs to identify triples relevant to the questions. GNN with more layers has more powerful capability in performing this task effectively. However, the effectiveness of the GNN module would be hindered if the number of layers is too large due to the over-smoothing problem (Li et al., 2018), i.e., the representations of nodes in deep GNNs converge to similar values and become indistinguishable.

heading: C.6 Effect of the Joint Optimisation
text: In Equation (13), we combine the T5 loss and GNN loss to optimise the GNN model and the T5 model jointly. To investigate the effect of the joint optimisation, we introduce a variant of REANO, where we separately optimise these two losses. Specifically, we first optimise the model with the GNN loss Lgnn and then optimise the model with the answer generation loss Lt5. The experimental results are provided in Table 10, which shows the exact match scores of REANO under different optimisation strategies on the 2WQA and MuSiQue datasets. The results show that joint optimisation performs better than separate optimisation on the 2WQA and the MuSiQue datasets. The main reason is that the GNN and T5 share an encoder. If we separately optimise Lgnn and Lt5, the encoder might forget the previously learned knowledge, leading to suboptimal performance. In contrast, jointly optimising the GNN and T5 allows the shared encoder to learn and retain features for both tasks simultaneously.

heading: C.7 Case Study
text: We provide the results of the case study on the 2WQA dataset in Table 11 and Table 12, which includes the question q, all the passages Dq, all the top-ranked triples T̂ Rq selected by the GNN module, and the corresponding answers generated by our REANO.
Table 11: Case study of REANO from 2WQA dataset (part 1).
Table 12: Case study of REANO from 2WQA dataset (part 2).

heading: Future Work
text: , 2023) remains unexplored, and we defer this exploration to future work. Moreover, we leverage a frozen retriever in our experiments to simplify the setting and do not investigate how the passage retriever’s effectiveness would affect the performance of REANO. We also leave the exploration of how to jointly optimise the retriever and our REANO for better performance as our future work