Venue: NAACL 2025

# Graph Neural Network Enhanced Retrieval for Question Answering of

Large Language Models

Zijian Li\({}^{12,*}\) , Qingyan Guo\({}^{23}\), Jiawei Shao\({}^{1}\), Lei Song\({}^{2}\), Jiang Bian\({}^{2}\),

Jun Zhang\({}^{1}\) , Rui Wang\({}^{21}\)

Work done during an internship at Microsoft Research Asia.Corresponding Author.

###### Abstract

Retrieval augmented generation has revolutionized large language model (LLM) outputs by providing factual supports. Nevertheless, it struggles to capture all the necessary knowledge for complex reasoning questions. Existing retrieval methods typically divide reference documents into passages, treating them in isolation. These passages, however, are often interrelated, such as passages that are contiguous or share the same keywords. Therefore, it is crucial to recognize such relatedness for enhancing the retrieval process. In this paper, we propose a novel retrieval method, called _GNN-Ret_, which leverages _graph neural networks_ (GNNs) to enhance retrieval by exploiting the relatedness between passages. Specifically, we first construct a _graph of passages_ by connecting passages that are structure-related or keyword-related. A _graph neural network_ (GNN) is then leveraged to exploit the relationships between passages and improve the retrieval of supporting passages. Furthermore, we extend our method to handle multi-hop reasoning questions using a _recurrent graph neural network_ (RGNN), named _RGNN-Ret_. At each step, _RGNN-Ret_ integrates the graphs of passages from previous steps, thereby enhancing the retrieval of supporting passages. Extensive experiments on benchmark datasets demonstrate that _GNN-Ret_ achieves higher accuracy for question answering with a single query of LLMs than strong baselines that require multiple queries, and _RGNN-Ret_ further improves accuracy and achieves state-of-the-art performance, with up to 10.4\(\%\) accuracy improvement on the 2WikiMQA dataset. The code is publicly available at [https://github.com/zli999/GNN_Ret](https://github.com/zli999/GNN_Ret).

+
Footnote †: Corresponding Author.

+
Footnote †: Corresponding Author.

## 1 Introduction

Large language models (LLMs) continue to struggle with factual errors when encountering knowledge intensive questions Huang et al. (2023); Mallen et al. (2022); Ji et al. (2023). Although retrieval-augmented LLMs Lewis et al. (2020) have improved factuality and precision of question answering by including relevant passages, there remains a persistent challenge in accurately capturing all the supporting passages when encountering complex knowledge-intensive questions. This limitation can be attributed to the inherent information asymmetry in complex questions. In particular, the questions tend to consist of elaborated background details, leaving only a small portion dedicated to specific inquiries. As an example, in question: '_Why did crime rise on Mars after the Mafia's arrival?_', the majority part delves into the background (i.e., '_crime rise ....tafia's arrival_') with only a few words requesting the reason (i.e., '_Why_'), which consequently retrieves passages on details of crime rising instead of the reason for it.

This phenomenon also frequently arises in multi-hop reasoning questions. Considering the sample question '_Where was the performer of song Left & Right (Song) born?_' in Fig. 1, it becomes apparent that while we can retrieve the knowledge that _the performer of song Left & Right is D'Angelo_, his birthplace remains absent. Previous works Trivedi et al. (2023); Press et al. (2022); Yao et al. (2022) have attempted to address this issue by incorporating multi-hop reasoning or question rewriting into retrieval processes, which enables them to retrieve information based on prior reasoning outcomes. LLMs, however, often generate plausible reasons and incorrect rewriting questions without accurate prior domain knowledge of the given question Huang et al. (2023); Zhang et al. (2023), thus affecting the subsequent retrieval process.

One reason why existing methods struggle to

handle the information asymmetry is their tendency to consider passages in isolation (Karpukhin et al., 2020) and retrieve the supporting passages based mainly on semantic distances, making it difficult to retrieve all the supporting passages, especially those containing only few words for inquiry. However, the supporting passages of background and inquiry information are usually correlated. They can be _structure-related_ when located in the same section or document, e.g., the happened event (i.e., _crime rises on Mars_) and its corresponding reason are located in the same section but different passages. Also, they can be _keyword-related_ by sharing the same keywords or entities, e.g., the passages of background information (_performer of song Left \(\&\) Right_) and inquiry information (_birthplace of D'Angelo_) in Fig. 1 share the same keyword: _D'Angelo_. By considering the relatedness between passages, it is possible to retrieve all the supporting passages even when they have significant semantic differences from the query. In this work, we aim to enhance retrieval by taking the relatedness between passages into account.

**Contributions.** To establish the relatedness between passages for retrieval purposes, this study initially constructs a _graph of passages_ by connecting individual passages based on both _structural information_ and _shared keywords_, with each passage as a node in this graph. The key challenge lies in how to effectively leverage the passage of graphs to enhance retrieval coverage. Graph neural networks (GNNs) are neural networks tailored to analyze graph data and adeptly grasp the relationships between nodes and edges (Scarselli et al., 2008). Thus, we propose to leverage a GNN to enhance the retrieval process by effectively capturing the relatedness between passages and name this method _GNN-Ret_. The GNN facilitates the integration of semantic distances between related passages, thus enabling the semantic distances of passages containing background information to impact the retrieval of passages relevant to the inquiry. To address multi-hop reasoning questions, we further propose a retrieval method, named _RGNN-Ret_, which leverages a _recurrent graph neural network_ (RGNN) to enhance the retrieval process at each step by integrating the retrievals from previous steps through the interconnected graphs of passages. This boosts the retrieval coverage for supporting passages over steps even when LLMs generate incorrect reasons or subquestions for retrieval.

Through the experiments on four benchmark datasets, we demonstrate the effectiveness of GNNs in enhancing retrieval of supporting passages and thus improving accuracy for QA. For example, as shown in Fig. 2, our proposed _GNN-Ret_ with a single query of LLMs significantly outperforms baselines in accuracy on 2WikiMQA (Ho et al., 2020), including those methods that require multiple queries of LLMs. Moreover, by extending GNN to multi-hop reasoning questions, our proposed _RGNN-Ret_ achieves state-of-the-art accuracy.

Figure 1: Overview of comparison between dense retrieval and _GNN-Ret_. The shared keywords and ground-truth answers are highlighted in yellow and green, respectively. By considering the relatedness between passages, _GNN-Ret_ can retrieve all the supporting passages for QA.

Figure 2: Accuracy and average number of LLM queries for our proposed methods and baselines on 2WikiMQA.

 

Related Works

**Retrieval-augmented LLM.** QA [23, 14] is a task that often requires external and up-to-date knowledge sources to answer factoid-based questions. Retrieve-and-read is the basic framework for these questions [8, 5]. For retrieval, sparse retrievers, e.g., TF-IDF and BM25 [7], or dense retrievers, e.g., DPR [9] and Contriever [10], are applied to compute lexical distances or semantic distances between passages and the question. The passages with smaller distances are retrieved, which are then prefixed with the question for actual answering [6]. However, this framework treats passages as isolated units, making it difficult to retrieve all the supporting passages for the question in a single step. To address this limitation, many approaches have been proposed to integrate the retrieval and reasoning processes to improve the retrieval of supporting passages [8, 5, 14, 15, 16, 17, 18, 19, 20]. They first prompt LLMs to generate the next-step reason or subquestion and then use it to guide retrieval for QA. However, a challenge in these methodologies pertains to the potential generation of hallucinated reasons or erroneous subquestions by LLMs without accurate domain knowledge of this question, leading to degraded subsequent retrievals [14].

**Graph-enhanced LLM.** As structured, explicit, and editable representations of knowledge, knowledge graphs (KGs) have been effectively used for boosting retrieval coverage [13] and enhancing reasoning capabilities of LLMs [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]. Instead of naively combining KGs with LLMs, many works directly prompt LLMs to perform reasoning on KGs [23, 22, 23]. They first identified the entities on the knowledge graph and then search reasoning paths accordingly [23, 22, 24, 25]. However, KG-based methods encounter a limitation that they can only handle questions that are effectively represented as KGs. They may struggle with questions that necessitate a comprehensive understanding of lengthy contextual information, such as the mentioned example: the cause of crime increasing. Rather than KGs, a recent study [13] attempts to construct a graph of passages and perform reasoning on this graph. It connects documents according to some specific words and performs reasoning on this graph for QA. In contrast, we utilize the graph of passages to enhance the retrieval process. Another related work, KGP [24], performs retrieval on the graph and search related passages. However, this approach does not consider the integration between related passages and may not effectively enhance retrieval for the supporting passages with poor semantic similarity.

## 3 Graph Neural Network Enhanced Retrieval for QA

The QA system typically adopts a _retrieve-and-read_ routine [6]: the retriever computes semantic distances between passages and the question in the embedding space, finds out relevant passages from the whole corpus, and then the _reader_ (LLM) generates the answer based on them. Considering the information asymmetry of complex questions, directly using the initial semantic distances of passages for retrieval is difficult to retrieve all the supporting passages. Therefore, we propose to process by taking the relatedness between passages into account. Specifically, we construct a _graph of passages_ (GoPs) by connecting related passages (Section 3.1). To exploit the relatedness between passages to enhance retrieval, we utilize a GNN to enable integration between related passages (Section 3.2). For multi-hop questions, we further leverage an RGNN to establish relationship between graphs over steps to enhance retrieval over answering processes (Section 3.3).

### Graph of Passages (GoPs)

**Establish relationships between related passages.** Existing retrieval methods rely on semantic distances between passages and the question [8, 19], which may fail to retrieve some supporting passages with large semantic distances, but related to those with small semantic distances to the question, due to the information asymmetry. These passages can be _structural-related_, e.g., the happened event (_'crime rising'_) and its corresponding reasons are located in the same section. As shown in Fig. 1, they can also be _keyword-related_, e.g., the supporting passages for 

background information ('_performer of song Left \(\&\) Right_') and inquiry information ('_the birthplace of D'Angelo_') share the same keyword ('_D'Angelo_'). Therefore, we propose to establish relationship between passages using structural information and shared keywords to construct the graph of passages. Specifically, when chunking the documents, we record the order of passages and connect passages that are physically next to each other in documents. In addition, we extract keywords from passages by prompting LLMs and connect passages containing the same keywords Min et al. (2019); Wang et al. (2024b). As such, the related passages are connected, having the potential to facilitate retrieval.

### GNN Enhanced Retrieval on GoPs

After establishing connections between related passages, we obtain a GoPs, with each passage as a node. When conducting the retrieval process, we first compute the semantic distances between passages and the question \(q\)Gao et al. (2023); Zhu et al. (2023). The semantic distances of related passages are connected according to the GoPs. To take relatedness between passages into account, we utilize a GNN to process semantic distances based on GoPs and obtain the _integrated semantic distances_, which benefits the retrieval of supporting passages.

**Graph neural network.** A graph is defined as an ordered pair \({\cal G}=\{{\cal V},{\cal E}\}\), where \({\cal V}\) is the set of nodes \(v_{i}\) and \({\cal E}\) is the set of edges \(e_{ij}\) connecting two nodes \(v_{i}\) and \(v_{j}\). We define \(h^{l}_{i}\) as the hidden state (integrated semantic distance) of the \(i-\)th node for \(i\in\{1,\cdots,|{\cal V}|\}\) at layer \(l\). We input semantic distances of passages into GNN as \(h^{0}_{i}\). The set of nodes \({\cal N}_{v_{i}}\) stands for the neighbors of node \(v_{i}\). For an \(L-\)layer GNN Scarselli et al. (2008), the hidden state of each node \(v_{i}\in{\cal V}\) is updated through iterative interactions with neighbors. Specifically, at layer \(l\), the received messages \(m^{l}_{i}\) of node \(v_{i}\) are calculated using the hidden states of its neighbors. Then the received messages \(m^{l}_{i}\) and the previous-layer hidden state \(h^{l-1}_{i}\) are utilized to compute the hidden state \(h^{l}_{i}\) at layer \(l\). By processing hidden states for all the passages layer by layer, we obtain the integrated semantic distance \(h^{L}_{i}\) for each node \(v_{i}\in{\cal V}\). We elaborately introduce GNNs for retrieval as below.

**Minimum semantic distance as message.** Since each passage maintains and shares many keywords with other passages, each passage has a large number of neighbors. Some of them are not relevant to the question, contributing to large semantic distances and misleading information. Therefore, each node \(v_{i}\) only receives the message from the neighbor that has the minimum (integrated) semantic distance to the question at layer \(l\). The received message of node \(v_{i}\) is thus formulated as \(m^{l}_{i}=\min_{j\in{\cal N}_{v_{i}}}h^{l-1}_{j}\).

**Relevant nodes sampling.** Given the large scale of GoPs with plenty of nodes, it is computationally inefficient to propagate information across all of them. Besides, allowing irrelevant nodes to pass messages to their neighbors may affect the retrieval process. Therefore, we only sample the relevant node set \({\cal T}^{l}_{K}\) with the top \(K\) smallest (integrated) semantic distances at layer \(l\). Only neighbors of relevant nodes receive messages from them and update hidden states. We define \({\cal S}^{l}_{K}\) as the neighbor set of the relevant node set \({\cal T}^{l}_{K}\). For each node \(v_{i}\in{\cal S}^{l}_{K}\), the message passed from neighbors that are relevant nodes is reformulated as follows:

\[m^{l}_{i}=\min_{j\in{\cal N}_{v_{i}},v_{j}\in{\cal T}^{l}_{K}}h^{l-1}_{j}. \tag{1}\]

By integrating the message from neighbors, the hidden state of each node \(v_{i}\in{\cal S}^{l}_{K}\) at layer \(l\) is computed using the parameter \(\alpha^{l}\) as follows:

\[h^{l}_{i}=\alpha^{l}h^{l-1}_{i}+(1-\alpha^{l})m^{l}_{i}. \tag{2}\]

The nodes without received messages maintain their hidden states of the previous layer. For the GNN with \(L\) layers, we compute the hidden state layer by layer and then obtain the integrated semantic distance \(h^{L}_{i}\) for each node \(v_{i}\). With GNN, the supporting passages for inquiry information integrate with the small semantic distance from those for background information and thereby obtain smaller integrated semantic distances, promoting them to be retrieved.

**Hinge objective for GNN.** To retrieve all the supporting passages, we aim to reduce their integrated semantic distances more than those of non-target nodes. With the ground-truth set of supporting passages \({\cal S}_{Y}\), we first define the average integrated semantic distances of supporting passages and non-target nodes as \(\bar{d}^{L}_{Y}=\frac{1}{|{\cal S}_{Y}|}\sum_{j\in{\cal S}_{Y}}h^{L}_{j}\) and \(\bar{d}^{L}_{O/Y}=\frac{1}{|S_{O/Y}|}\sum_{o\in S_{O/Y}}h^{L}_{o}\), respectively, where \(S_{O}\) is the competitive node set with the top \(O\) smallest semantic distances, and \(S_{O/Y}\) is the non-target node set by removing the target node set \(S_{Y}\) from \(S_{O}\). We only consider the competitive node set since the average integrated semantic distance of all the nodes is significantly large and loses effectiveness in training. With the output average semantic distances of supporting passages and non-target 

nodes, we formulate the hinge objective function with threshold \(r\) as follows:

\[\ell=\max(0,r+\bar{d}_{Y}^{L}-\bar{d}_{O/Y}^{L}). \tag{3}\]

We update the GNN using the gradient descent, and training details are elaborated in Appendix A.1.

### RGNN Enhanced Retrieval for Multi-hop Reasoning Questions

The information asymmetry phenomenon also frequently arises in complex multi-hop reasoning questions, as analyzed in the Introduction section. Although recent works have incorporated multi-hop reasoning or question rewriting into retrieval Yao et al. (2022); Trivedi et al. (2023); Shao et al. (2023), LLMs may generate hallucinated reasons or incorrect subquestions due to the absence of prior knowledge to the question Huang et al. (2023). These hallucinated reasons or incorrect subquestions fail to retrieve supporting passages when only considering the retrieval process in isolation for each step. Additionally, they also encounter another challenge: whether to continue the next-step answering process or output the final answer. To overcome these challenges, we first propose a _self-critique_ technique by prompting LLMs to determine the termination of the answering process. To enhance retrieval of supporting passages over steps, we further utilize an RGNN to enable the integration between graphs of passages in different steps.

**Self-critique of LLMs.** To answer multi-hop reasoning questions, _self-critique_ involves prompting the LLM to generate subquestions and answer them using retrieved passages in a step-by-step manner. As depicted in Fig. 3, at each step, after the LLM generates a subanswer to the subquestion, the question and all the generated subanswers are input into the LLM and critiqued to determine whether the generated subanswers are evident to generate the final answer to the question. If the output is 'Yes', the LLM will be prompted to generate the final answer. Otherwise, it will be requested to generate the next-step subquestion. Compared with _SelfAsk_Press et al. (2022), we omit the subquestions and focus on whether existing evidences are satisfactory for answering the question when determining the termination. This _self-critique_ technique decomposes the question over steps and enables a more accurate answer to the question.

**Recurrent graph neural network.** To enhance retrievals over steps, as shown in Fig. 3, we utilize an RGNN to establish relationships between the semantic distances of subquestions. Specifically, with the subquestion \(q^{t}\) at step \(t\), we first compute \(h_{i}^{L,t}\) for each node \(v_{i}\in\mathcal{V}\) using the aforementioned \(L\)-layer GNN. To consider the effect of semantic distances from previous subquestions, we integrate \(h_{i}^{L,t}\) with the previous-step integrated semantic distance to compute the integrated semantic distance \(\hat{h}_{i}^{L,t}\) at step \(t\) using the parameter \(\beta\). This can be formulated as follows:

\[\hat{h}_{i}^{L,t}=\beta h_{i}^{L,t}+(1-\beta)\hat{h}_{i}^{L,t-1},\quad(t>1), \tag{4}\]

where \(\hat{h}_{i}^{L,t-1}\) is the integrated semantic distance from the previous step \(t-1\) and \(\hat{h}_{i}^{L,1}=h_{i}^{L,1}\). The integrated semantic distance \(\hat{h}_{i}^{L,t}\) will be used to integrate with \(h_{i}^{L,t+1}\) of node \(v_{i}\) at the next step \(t+1\) recurrently until the last step of the answering process \(T\). GNN facilitates the integration of semantic distances between related passages at each step, and the recurrent parameter \(\beta\) enables the

Figure 3: Illustration of _GNN-Ret_ and _RGNN-Ret_.

 

integration of semantic distances of different sub-questions across steps, which effectively mitigates the impacts of incorrect subquestions and enhances the retrieval of supporting passages for them.

**Hinge objective for RGNN.** We adopt a hinge objective for RGNN by considering all the \(T\) answering steps for question \(q\). With the subquestion \(q^{t}\) at step t, the primary objective of RGNN is to reduce the semantic distance of its corresponding supporting passage and make it retrieved. Furthermore, to enhance the retrieval process for subsequent answering steps, we also reduce the semantic distances of supporting passages that should appear at subsequent steps to make them retrieved subsequently. Specifically, at step \(t\), we adopt a hinge objective that encourages the average integrated semantic distance of supporting passages that should appear at the current and subsequent steps \(t+=\{t,\cdots,T\}\) to be lower than that of the non-target node set. We first define the average integrated semantic distances of supporting passages and non-target nodes at step \(t\) as \(\bar{d}_{Y^{t+}}^{L,t}=\frac{1}{|\mathcal{S}_{Y^{t+}}^{l}|}\sum_{j\in\mathcal{ S}_{Y^{t+}}^{t}}\hat{h}_{j}^{L,t}\) and \(\bar{d}_{O/Y^{t+}}^{L,t}=\frac{1}{|\mathcal{S}_{O/Y^{t+}}^{1}|}\sum_{o\in \mathcal{S}_{O/Y^{t+}}^{t}}\hat{h}_{o}^{L,t}\), respectively, where \(\mathcal{S}_{O}^{t}\) is the competitive node set with top \(O\) smallest semantic distances, and \(\mathcal{S}_{O/Y^{t+}}^{t}\) is the non-target node set by removing the target node set \(\mathcal{S}_{Y^{t+}}^{t}\) from \(\mathcal{S}_{O}^{t}\). The hinge objective function with threshold \(r\) for the RGNN is formulated as follows:

\[\ell=\frac{1}{T}\sum_{t\in[T]}\max(0,r+\bar{d}_{Y^{t+}}^{L,t}-\bar{d}_{O/Y^{t+ }}^{L,t}). \tag{5}\]

By employing this hinge objective, the RGNN takes into account the retrieval of subsequent answering steps and proactively reduces the semantic distances associated with them. These small semantic distances are then transferred to the next subquestion and enhance the retrieval of the corresponding supporting passages. We update the RGNN (parameters \(\alpha^{I}\) and \(\beta\)) using the gradient descend, which are elaborated in Appendix A.2.

## 4 Experiments

In this section, we conduct experiments to demonstrate the effectiveness of _GNN-Ret_ and _RGNN-Ret_, including **1)** superiority of _GNN-Ret_ and _RGNN-Ret_ on various QA datasets (Table 1), **2)** ablation studies to verify the effectiveness of components in GoPs, _GNN-Ret_, and _RGNN-Ret_ (Table 2 and 3), and **3)** superiority of _GNN-Ret_ on open-sourced LMs (Table 4). Additional analysis on hyperparameter selection, statistics of retrieval accuracy, token number for retrieval, multi-layer _GNN-Ret_, extension of _RGNN-Ret_ to other multi-hop answering methods, and qualitative results are discussed in Appendix C, D, E, F, G, H, respectively.

### Experimental Setups

#### Evaluation datasets.

We measure all the methods on four different QA datasets, including multi-hop Wikipedia reasoning datasets: (1) MuSiQue Trivedi et al. (2022), (2) IIRC Ferguson et al. (2020), (3) 2WikiMQA Ho et al. (2020), and a single-hop multi-choice QA dataset: (4) Quality Pang et al. (2021). As evaluation metrics, we calculate the F1 score, exact match (EM) and accuracy (Acc) for multi-hop reasoning datasets. We use the ChatGPT (gpt-3.5-turbo-2023-06-01-preview) to evaluate if the prediction matches with the gold answer, following Shao et al. (2023); Wang et al. (2024b). Since Quality is not a multi-hop dataset, we only validate accuracy performance of _GNN-Ret_ on it.

**Baselines.** We compare _GNN-Ret_ and _RGNN-Ret_ with the following baselines: (1) _Direct_ answers questions without retrieved passages. (2)We use retrievers _bm25_Robertson et al. (2009), _DPR_Karpukhin et al. (2020a), and _SentenceBert_1 (_SBERT_) Reimers and Gurevych (2019) to retrieve passages without considering relatedness between passages for QA. (3) _SelfAsk_Press et al. (2022) prompts LLMs to generate the follow-up question and answers it with retrieved passages until generating the final answer. (4) _ITER-RETGEN_Shao et al. (2023) iteratively answers questions with retrieved passages and uses the generated answer for the next-step retrieval until the generation of final answer. (5) _IRCoT_Trivedi et al. (2023) iteratively prompts LLMs to generate chains of thoughts with retrieved passages and retrieves with the generated reasons until reaching the maximum token number. All the retrieved passages are then used to generate the final answer. (6) _KGP_Wang et al. (2024b) first searches seed passages using the question and then prompts LLMs to generate the needed evidence to retrieve the other relevant passages among neighbors of seed nodes using semantic distances.

Footnote 1: [https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-cos-v1](https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-cos-v1)

**Implementation details.** We use ChatGPT (gpt-3.5-turbo-2023-06-01-preview) as the LLM backbone for experiments and adopt a temperature of 0 to remove the effect of random sam 

pling. We employ SBERT as the embedding model for all our methods and baselines. The semantic distance is derived by \((1-\text{cosine similarity})\) following [23]. We set up a maximum token number of 3500 for retrieval to leave the space to instruction and demonstrations, and all the methods retrieve the semantically similar passages until reaching the maximum token number. We adopt a one-layer GNN for our proposed methods _GNN-Ret_ and _RGNN-Ret_. We set up \(\alpha^{1}=0.5\) for _GNN-Ret_ and \(\alpha^{1}=0.5\) and \(\beta=0.9\) for _RGNN-Ret_ when there is no training. When training the RGNN, we set up \(r=0.01\), \(K=5\), and \(O=25\) for _GNN-Ret_ (w. train) and \(r=0.01\), \(K=5\), and \(O=10\) for _RGNN-Ret_ (w. train), respectively, which achieve the best performance in grid search experiments in Appendix C. We do not train the parameters for Quality dataset without the ground-truth labels of supporting passages. Since training the RGNN requires ground-truth subquestions, we manually generate subquestions for 5 questions sampled from the preset training data. More implementation details are presented in Appendix B.

### Main Results

Table 1 summarizes the results for our proposed methods and baselines on four QA datasets using ChatGPT. Results show that _GNN-Ret_ significantly outperforms baselines in terms of F1, EM, and accuracy on all these four QA datasets with a single retrieval. For instance, compared with _SBERT_, _GNN-Ret_ improves EM by 4.4 and accuracy by \(4\%\) on the IIRC dataset. Additionally, _GNN-Ret_ maintains its superiority on the Quality dataset, which requires a comprehensive understanding of the context of an entire story. This can be attributed to the ability of _GNN-Ret_ to enhance the retrieval prioritization of supporting passages through the incorporation of structural information. Surprisingly, our proposed one-hop _GNN-Ret_ even slightly outperforms baselines with multi-hop retrieval processes and queries of LLMs in accuracy, highlighting significant potentials of leveraging passage relatedness to enhance the retrieval process for answering multi-hop reasoning questions.

Furthermore, results in Table 1 demonstrate that our proposed _RGNN-Ret_ achieves state-of-the-art performance in terms of F1, EM, and accuracy on the three multi-hop reasoning datasets. _RGNN-Ret_ outperforms the best baseline, _KGP_, by more than \(10\%\) in accuracy on 2WikiMQA. This performance improvement can be attributed to our proposed _self-critique_ technique and RGNN. The _self-critique_ technique enables more accurate judgement on the ending of the answering process and the generation of final answers. The incorporation of RGNN further enhances retrieval for each step of answering.

### Additional Analysis

#### Effectiveness of components in _Gnn-Ret_.

To assess the individual contributions of each proposed component, we conduct ablation studies and present the results in Table 2. We explore an alternative approach by utilizing the mean semantic distance (_GNN-Ret (mean)_) instead of the minimum semantic distance as the message in equation (1)

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c} \hline \hline Methods & \multicolumn{6}{c}{MuSiQue} & \multicolumn{6}{c}{IIIRC} & \multicolumn{3}{c}{2 WikiMQa} & Quality \\ \cline{2-17}  & F1 & EM & Acc & F1 & EM & Acc & F1 & EM & Acc & F1 & EM & Acc \\ \hline \multicolumn{18}{c}{No Retrieval} \\ \hline _Direct_ & 14.1 & 4.0 & 9.6 & 15.8 & 9.6 & 16.0 & 24.4 & 17.3 & 24.8 & - & - & 39.5 \\ \hline \multicolumn{18}{c}{One-hop Retrieval} \\ \hline _bm25_ & 22.2 & 10.2 & 18.8 & 27.3 & 15.4 & 31.7 & 33.8 & 23.8 & 34.6 & - & - & 47.0 \\ _DPR_ & 23.8 & 10.8 & 17.5 & 29.9 & 16.3 & 35.6 & 36.2 & 24.4 & 36.3 & - & - & 52.5 \\ _SBERT_ & 28.6 & 13.5 & 23.8 & 31.6 & 17.5 & 40.4 & 43.4 & 31.0 & 42.3 & - & - & 53.4 \\ _GAN-Ret_ (K=5) & **31.6** & 16.5 & 27.1 & 33.2 & 17.6 & 44.0 & 47.7 & 32.7 & 44.8 & - & - & 55.5 \\ _GNN-Ret_ (K=10) & 30.0 & 14.4 & 24.4 & 35.2 & 19.6 & 44.0 & 47.7 & 32.9 & 46.3 & - & - & **58.2** \\ _GNN-Ret_ (w. train) & 31.1 & **17.5** & **27.3** & **36.1** & **21.9** & **44.4** & **48.1** & **33.5** & **46.9** & - & - & - \\ \hline \multicolumn{18}{c}{Multi-hop Retrieval} \\ \hline _Self/ask_ & 28.0 & 18.3 & 22.9 & 37.1 & **29.6** & 36.9 & 51.2 & 41.5 & 44.8 & - & - & - \\ _ITERETGEN_ & 30.0 & 16.7 & 27.1 & 32.6 & 20.0 & 39.6 & 45.6 & 33.3 & 43.8 & - & - & - \\ _IRCoT_ & 29.9 & 12.5 & 27.5 & 32.5 & 18.8 & 41.0 & 46.0 & 30.8 & 44.6 & - & - \\ _KGP_ & 30.0 & 15.4 & 24.0 & 33.9 & 18.5 & 42.1 & 45.5 & 31.9 & 45.2 & - & - & - \\ _RGNN-Ret_ & 32.9 & 20.8 & **31.3** & **43.4** & 28.3 & 48.1 & 59.7 & 43.3 & **55.8** & - & - & - \\ _RGNN-Ret_ (w. train) & **34.8** & **21.9** & **31.3** & 42.8 & 27.6 & **48.4** & **61.4** & **45.2** & 55.6 & - & - \\ \hline \end{tabular}
\end{table}
Table 1: F1/EM/Acc for different QA methods with ChatGPT on four QA datasets. The best and second best scores are highlighted in **bold** and underline, respectively 

for the GNN. This modification results in a lower accuracy compared to the use of the minimum semantic distance as the message, which suggests that employing the minimum semantic distance as the message effectively filters out interfering messages from irrelevant neighbors and preserves the most relevant message for the GNN.

**Effectiveness of components in _RGNN-Ret._ We conduct ablation studies to evaluate the effectiveness of each component in _RGNN-Ret_ (i.e., recurrent, GNN, and RGNN). Results in Table 2 show that both the recurrent part (parameter \(\beta\)) and GNN improve accuracy compared with setting with SBERT on MuSiQue and 2WikiMQA datasets. By combining the recurrent part and GNN, our proposed method _RGNN-Ret_ further improves accuracy up to \(31.3\%\) and \(55.6\%\) on MuSiQue and 2WikiMQA datasets, respectively, which demonstrates the effectiveness of _RGNN-Ret_ in enhancing the retrieval by interconnecting with the GoPs from previous steps. This interconnection enhances the retrieval of supporting passages across multiple steps, leading to more accurate answers in the multi-hop reasoning process.

We also explore different settings for the target node sets during the training of RGNN and present results in Table 2. When using the entire node set of ground-truth supporting passages \({\cal S}_{Y}\) as the retrieval labels for each subquestion, denoted as _RGNN-Ret_ (\(Y\)), the accuracy decreases to \(27.1\%\) and \(53.3\%\) on MuSiQue and 2WikiMQA datasets, respectively. This is because the supporting passages for initial steps are irrelevant to the subquestions at subsequent steps. Including all of them as labels for each subquestion disrupts the training of RGNN. Instead, we employ the corresponding label for each subsection, denoted as _RGNN-Ret_ (\(Y^{t}\)). This approach effectively improves the accuracy to \(30.2\%\) and \(54.6\%\) on the MuSiQue and 2WikiMQA datasets, respectively. Futhermore, our proposed _RGNN-Ret_ method includes label set of supporting passages not only for the current subquestion but also for subsequent subquestions. This comprehensive approach further improves the accuracy to \(31.3\%\) and \(55.6\%\) on MuSiQue and 2WikiMQA datasets, respectively, which verifies the effectiveness of _RGNN-Ret_ in enhancing the retrieval process for subsequent subquestions.

**Effectiveness of GoPs.** We explore the effectiveness of components (structural information (SI) and shared keywords (SK)) for graph construction. Results in Table 3 show that both structural information and shared keywords are able to improve accuracy for the baseline without GoPs. Notably, in the context-specific Quality story dataset, which demands a comprehensive grasp of the entire narrative for accurate retrieval, the graph that incorporates structural information markedly outperforms the setting that relies solely on shared keywords. While for 2WikiMQA that requires knowledge of multiple documents, the graph with shared keywords achieves a better accuracy compared with that with structural information. By combining them into graph construction, we achieve the best accuracy in these two datasets.

**Performance of _GNN-Ret_ with open-sourced LMs.** Our proposed _GNN-Ret_ utilizes the relatedness of passages to enhance retrieval with the meticulously designed GNNs, which is generalized and robust to all the LM architectures. To substantiate this, we supplement the experiments for _GNN-Ret_ and SBERT with two well-behaved open-sourced LMs (Qwen/Qwen2-7B-Instruct (Yang et al., 2024) and google/gamma-2-9b-it (Team, 2024)) from Open LLM Leaderboard (Fourrier et al., 2024). Results in Table 4 show tha

\begin{table}
\begin{tabular}{c c c c c c} \hline  & \multicolumn{3}{c}{MuSiQue} & \multicolumn{3}{c}{2WikiMQA} \\ \hline Qwen2-7B-Instruct & F1 & EM & Acc & F1 & EM & Acc \\ \hline SBERT & 27.8 & 16.9 & 23.3 & 43.3 & 34.0 & 40.8 \\ GNN-Ret & **32.3** & **20.8** & **26.7** & **50.5** & **37.3** & **45.8** \\ \hline gamma-2·9b-it & F1 & EM & Acc & F1 & EM & Acc \\ \hline SBERT & 32.8 & 18.9 & 26.9 & 46.4 & 28.8 & 43.8 \\ GNN-Net & **34.6** & **20.8** & **29.0** & **51.6** & **30.8** & **45.8** \\ \hline \end{tabular}
\end{table}
Table 4: F1/EM/Acc of SBERT and GNN-Ret with open-sourced LMs on MuSiQue and 2WikiMQA.

\begin{table}
\begin{tabular}{c c c} \hline Methods & MuSiQue & 2WikiMQA \\ \hline SBERT & 23.8 & 42.3 \\ GNN-Ret (mean) & 26.3 (+2.5) & 44.6 (+1.7) \\ GNN-Ret & **27.3** (+3.5) & **46.9** (+4.6) \\ \hline Self-critique +SBERT & 27.9 & 52.9 \\ Self-critique +recurrent & 28.1 (+0.2) & 54.0 (+1.1) \\ Self-critique +GNN & 30.8 (±2.9) & 55.2 (±2.3) \\ RGNN-Ret & **31.3** (±3.4) & **55.6** (±2.7) \\ \hline RGNN-Ret (\(Y\)) & 27.1 & 53.3 \\ RGNN-Ret (\(Y^{t}\)) & 30.2 (±3.1) & 54.6 (±1.3) \\ RGNN-Ret (\(Y^{t+}\)) & **31.3** (±4.2) & **55.6** (±2.3) \\ \hline \end{tabular}
\end{table}
Table 2: Ablation studies of components of GNN-Ret and RGNN-Ret in accuracy with ChatGPT on MuSiQue and 2WikiMQA.

\begin{table}
\begin{tabular}{c c c} \hline Methods & 2WikiMQA & Quality \\ \hline SBERT (w.o. graph) & 42.3 & 53.4 \\ graph w. SI & 42.7 (+0.4) & 55.5 (±2.1) \\ graph w. SK & 45.4 (±3.1) & 54.8 (±1.4) \\ graph w. SI and SK & **46.9** (±4.6) & **58.2** (±4.8) \\ \hline \end{tabular}
\end{table}
Table 3: Ablation studies of components on graph construction in accuracy with ChatGTP on 2WikiMQA and Quality. SI and SK represent structural information and shared keywords, respectively.

 

outperforms SBERT across varying open-sourced LMs, demonstrating the generalization of _GNN-Ret_ in enhancing retrieval coverage with varying LMs.

## 5 Conclusion

In this paper, we proposed _GNN-Ret_, an effective method to enhance retrieval for QA of LLMs by exploiting the inherent relatedness between passages on a graph of passages. By extending _GNN-Ret_ to multi-hop reasoning questions, we proposed _RGNN-Ret_, which enhances the retrieval for subquestions through the interconnection between graphs of passages across steps. The experiments clearly demonstrated the superiority of both _GNN-Ret_ and _RGNN-Ret_ over baselines, highlighting the effectiveness of leveraging relatedness between passages to enhance retrieval. From these advantages, we believe that the incorporation of graph-based representations and the exploitation of passage relatedness can open up new avenues of research in the field of LLMs in understanding structural documents and answering complex questions.

## 6 Acknowledgements

This work was supported by the Hong Kong Research Grants Council under the NSFC/RGC Collaborative Research Scheme grant CRS_HKUST603/22.

## 7 Limitations

**Costs of graph construction.** While our proposed _GNN-Ret_ demonstrates impressive accuracy in QA tasks with a single query of LLMs, it still relies on multiple LLM queries to extract keywords from passages and construct the graph of passages. Fortunately, there are alternative fine-tuned language models specific for keyword extraction23. These models are more compact and significantly accelerate the graph construction process. Moreover, the flexibility of the graph of passages allows for dynamic modifications, such as adding new passage nodes or removing outdated ones. By maintaining a domain-specific graph of passages, we can effectively address various questions for this domain without the need for reconstructing the graph.

Footnote 2: [https://huggingface.com/ml6team/keyphrase-extraction-distilbert-inspec](https://huggingface.com/ml6team/keyphrase-extraction-distilbert-inspec)

**Costs of message passing.** We first analyze the statistics of the graph of passages in experiments and detail results in Table 5. In larger datasets, each node typically exhibits a higher number of neighbors, attributed to an increased likelihood of keyword sharing among more nodes. We calculate the density of the graph of passages, denoted as \(D=\frac{2|\mathcal{E}|}{|\mathcal{V}|(|\mathcal{V}|-1)}\), for each dataset in our experiments, based on the established definition [4], where \(\mathcal{E}\) and \(\mathcal{V}\) represent the edges and vertices of the graph, respectively. The densities of all graphs are significantly less than 1, underscoring the substantial sparsity observed in these graphs across our experimental datasets.

We adopt relevant nodes sampling (Section 3.2) to allow the relevant node set with only top \(K\) smallest semantic distances to pass the information to their neighbors during message passing. Assuming an average number of neighbors \(\mathcal{V}_{K}\) among these \(K\) relevant nodes, the time complexity for message passing can be derived as \(O(K|\mathcal{V}_{K}|)\), considering a single-layer GNN. With \(K\) set to 5 or 10 in our experiments, and given the small empirical average number of edges per node \(\frac{|\mathcal{E}|}{|\mathcal{V}|}\) reported in Table 5, the time complexity of message passing is neglectable compared with LLM queries.

**Undirected Graph.** In this study, our focus lies solely on improving the retrieval process through an undirected graph of passages, with no additional information associated with the edges. While we employ a sampling technique to select relevant nodes and utilize the minimum semantic distance as the message in the GNN, we acknowledge that there is still a possibility of irrelevant passages receiving messages, potentially impacting the retrieval performance. Consequently, the task of selecting an appropriate path on the graph of passages that aligns with the given question remains an area that is yet to be thoroughly explored. In addition, we only consider the structural-related passages that are in the same section. However, some questions may require more complex structural information. For example, when the question is about comparison of two entities in different sections, more complex structural information is needed to connect these corresponding passages to enhance retrieval. Therefore, a more delicate graph of passages for more complex tasks warrant further

\begin{table}
\begin{tabular}{l c c c} Datasets & Number of nodes (\(|\mathcal{V}|\)) & Average number of edges (\(\frac{|\mathcal{E}|}{|\mathcal{V}|}\)) & Density \\ \hline JWikiMOA & 9815 & 91.38 & 0.018 \\ IIRC & 21866 & 259.22 & 0.024 \\ MSGuote & 20071 & 335.77 & 0.033 \\ Quality & 1104 & 26.00 & 0.047 \\ \end{tabular}
\end{table}
Table 5: Statistics of graph of passages in experiments.

 

investigation in future studies.

## References

* J. Baek, A. Fikri Aji, and A. Saffari (2023)Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pp. 78-106. Cited by: SS4.
* W. Ding, J. Li, L. Luo, and Y. Qu (2024)Enhancing complex question answering over knowledge graphs through evidence pattern retrieval. arXiv preprint arXiv:2402.02175. Cited by: SS1.
* Z. Feng, X. Feng, D. Zhao, M. Yang, and B. Qin (2024)Retrieval-generation synergy augmented large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , pp. 11661-11665. External Links: Document Cited by: SS1.
* J. Ferguson, M. Gardner, H. Hajishirzi, T. Khot, and P. Dasigi (2020)Irc: a dataset of incomplete information reading comprehension questions. arXiv preprint arXiv:2011.07127. Cited by: SS1.
* C. Fourrier, N. Habib, A. Lozovskaya, K. Szfer, and T. Wolf (2024)Open llm leaderboard v2. Note: [https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) Cited by: SS1.
* Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang (2023)Retrieval-augmented generation for large language models: a survey. arXiv preprint arXiv:2312.10997. Cited by: SS1.
* X. He, Y. Tian, Y. Sun, N. V. Chawla, T. Laurent, Y. LeCun, X. Bresson, and B. Hooi (2024)G-retriever: retrieval-augmented generation for textual graph understanding and question answering. arXiv preprint arXiv:2402.07630. Cited by: SS1.
* X. He, A. Duong Nguyen, S. Sugawara, and A. Aizawa (2020)Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060. Cited by: SS1.
* L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, et al. (2023)A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232. Cited by: SS1.
* G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave (2021)Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118. Cited by: SS1.
* Ziwei Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Jin Bang, A. Madotto, and P. Fung (2023)Survey of hallucination in natural language generation. ACM Computing Surveys55 (12), pp. 1-38. Cited by: SS1.
* J. Jiang, K. Zhou, Z. Dong, K. Ye, W. Xin Zhao, and J. Wen (2023)Struct-gpt: a general framework for large language model to reason over structured data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9237-9251. Cited by: SS1.
* Z. Jiang, F. F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, and G. Neubig (2023)Active retrieval augmented generation. arXiv preprint arXiv:2305.06983. Cited by: SS1.
* B. Jin, G. Liu, C. Han, M. Jiang, H. Ji, and J. Han (2023)Large language models on graphs: a comprehensive survey. arXiv preprint arXiv:2312.02783. Cited by: SS1.
* B. Jin, C. Xie, J. Zhang, K. Kumar Roy, Y. Zhang, S. Wang, Y. Meng, and J. Han (2024)Graph chain-of-thought: augmenting large language models by reasoning on graphs. arXiv preprint arXiv:2404.07103. Cited by: SS1.
* Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih (2020)Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906. Cited by: SS1.
* P. Lewis, E. Perez, A. Pitkus, F. Petroni, V. Karpukhin, N. Goyal, Heinrich Kutler, M. Lewis, W. Yih, T. Rocktaschel, et al. (2020)Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems33, pp. 9459-9474. Cited by: SS1.
* X. Li, R. Zhao, Y. K. Chia, B. Ding, L. Bing, S. Joty, and S. Poria (2023)Chain of knowledge: a framework for grounding large language models with structured knowledge bases. arXiv preprint arXiv:2305.13269. Cited by: SS1.
* Y. Li, Z. Li, P. Wang, J. Li, X. Sun, H. Cheng, and J. Xu Yu (2023)A survey of graph meets large language model: progress and future directions. arXiv preprint arXiv:2311.12399. Cited by: SS1.
* L. LUO, Y. Li, R. Haf, and S. Pan (2024)Reasoning on graphs: Faithful and interpretable large language model reasoning. In The Twelfth International Conference on Learning Representations, Cited by: SS1.

 

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. _arXiv preprint arXiv:2212.10511_.
* Min et al. (2019) Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019. Knowledge guided text retrieval and reading for open domain question answering. _arXiv preprint arXiv:1911.03868_.
* Pang et al. (2021) Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nagia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. 2021. Quality: Question answering with long input texts, yes! _arXiv preprint arXiv:2112.08608_.
* Park et al. (2023) Jinyoung Park, Ameen Patel, Omar Zia Khan, Hyunwoo J Kim, and Joo-Kyung Kim. 2023. Graph-guided reasoning for multi-hop question answering in large language models. _arXiv preprint arXiv:2311.09762_.
* Pieterse and Black (2005) V Pieterse and PE Black. 2005. Dictionary of algorithms and data structures. _URL: http://www. nist. gov/dads/HTML/greedyalgo. html_.
* Press et al. (2022) Ofr Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. _arXiv preprint arXiv:2210.03350_.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. _Foundations and Trends@ in Information Retrieval_, 3(4):333-389.
* Sarthi et al. (2024) Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. 2024. Raptor: Recursive abstractive processing for tree-organized retrieval. In _The Twelfth International Conference on Learning Representations_.
* Scarselli et al. (2008) Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. _IEEE transactions on neural networks_, 20(1):61-80.
* Shao et al. (2023) Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. _arXiv preprint arXiv:2305.15294_.
* Sun et al. (2024) Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyuong Gong, Lionel Ni, Heung-Yeung Shum, and Jian Guo. 2024. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In _The Twelfth International Conference on Learning Representations_.
* Team (2024) Gemma Team. 2024. Gemma.
* Trivedi et al. (2022) Harsh Trivedi, Niranjan Balasubramanian, Tshar Khot, and Ashish Sabharwal. 2022. Musique: Multi-hop questions via single-hop question composition. _Transactions of the Association for Computational Linguistics_, 10:539-554.
* Trivedi et al. (2023) Harsh Trivedi, Niranjan Balasubramanian, Tshar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 10014-10037.
* Voorhees et al. (1999) Ellen M Voorhees et al. 1999. The trec-8 question answering track report. In _Trec_, volume 99, pages 77-82.
* Wang et al. (2023) Jianing Wang, Qiushi Sun, Nuo Chen, Xiang Li, and Ming Gao. 2023. Boosting language models reasoning with chain-of-knowledge prompting. _arXiv preprint arXiv:2306.06427_.
* Wang et al. (2024a) Shu Wang, Lei Ji, Renxi Wang, Wenxiao Zhao, Haokun Liu, Yifan Hou, and Ying Nian Wu. 2024a. Explore the reasoning capability of llms in the chess testbed. _arXiv preprint arXiv:2411.06655_.
* Wang et al. (2024b) Yu Wang, Nedim Lipka, Ryan A Rossi, Alexa Siu, Ruiyi Zhang, and Tyler Derr. 2024b. Knowledge graph prompting for multi-document question answering. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 19206-19214.
* Xie et al. (2022) Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 602-631.
* Xiong et al. (2024) Siheng Xiong, Yuan Yang, Ali Payani, James C Kerce, and Faramarz Fekri. 2024. Teilp: Time prediction over knowledge graphs via logical reasoning. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 16112-16119.
* Yang et al. (2024) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. _arXiv preprint arXiv:2407.10671_.
* Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafar, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. _arXiv preprint arXiv:2210.03629_.
* Zhang et al. (2023) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren's song in the ai ocean: a survey on hallucination in large language models. _arXiv preprint arXiv:2309.01219_ 

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey. _arXiv preprint arXiv:2308.07107_.

## Appendix A Training Details

In this section, we supplement the training details of GNN and RGNN as below.

### Training of GNN

**Hinge objective for GNN.** Recall that the objective of GNN is to reduce the semantic distances of supporting passages more than those of non-target nodes. With the ground-truth set of supporting passages \({\mathcal{S}}_{Y}\), we first define the average integrated semantic distances of supporting passages \(\bar{d}^{L}_{Y}\) and non-target nodes \(\bar{d}^{L}_{O/Y}\) as below:

\[\bar{d}^{L}_{Y}=\frac{1}{|{\mathcal{S}}_{Y}|}\sum_{j\in{\mathcal{ S}}_{Y}}h^{L}_{j}, \tag{6}\] \[\bar{d}^{L}_{O/Y}=\frac{1}{|{\mathcal{S}}_{O/Y}|}\sum_{o\in{ \mathcal{S}}_{O/Y}}h^{L}_{o}, \tag{7}\]

where \(S_{O}\) is the competitive node set with the top \(O\) smallest semantic distances and \(S_{O/Y}\) is the non-target node set by removing the target node set \(S_{Y}\) from \(S_{O}\), respectively. We only consider the competitive node set since the average integrated semantic distance of all the nodes is significantly large and loses effectiveness in training. With the output average semantic distances of supporting passages and non-target nodes, we formulate the hinge objective function with threshold \(r\) as follows:

\[\ell=\max(0,r+\bar{d}^{L}_{Y}-\bar{d}^{L}_{O/Y}). \tag{8}\]

We update the parameters of GNN using gradient descent, and the backward process is presented as below. With only partial nodes accepting messages from neighbors, we first define the received node set at each layer \(l\) as follows:

\[{\mathcal{S}}^{l}_{R}=\begin{cases}{\mathcal{S}}_{O}\cap{\mathcal{ S}}^{L}_{K},&l=L\\ &{\mathcal{S}}^{l}_{K},&\text{else}\end{cases}. \tag{9}\]

For each node \(i\in{\mathcal{S}}^{L}_{R}\) at the last layer \(L\), the gradient of \(\ell\) w.r.t. \(h^{L}_{i}\) is:

\[\frac{\partial\ell}{\partial h^{L}_{i}}=\begin{cases}1.0,&i\in{ \mathcal{S}}_{Y}\\ -1.0,&\text{else}\end{cases}. \tag{10}\]

For the layer \(l<L\), the gradient of \(l\) w.r.t. \(\alpha^{l}\) is presented as follows:

\[\frac{\partial\ell}{\partial\alpha^{l}}=\sum_{j\in{\mathcal{S}}^{ l}_{R}}\frac{\partial\ell}{\partial h^{l}_{i}}(h^{l-1}_{i}-m^{l}_{i}). \tag{11}\]

Assuming that node \(i\in{\mathcal{T}}^{l-1}_{k}\) at layer \(l-1\) transfers the message to the node set \({\mathcal{S}}^{l}_{K;i}\) at layer \(l\) (one node can transfers messages to multiple neighbors), we update the gradient of \(\alpha^{l}\) w.r.t. \(h^{l-1}_{i}\) at the previous layer \(l-1\) as follows:

\[\frac{\partial\ell}{\partial h^{l-1}_{i}}=\sum_{j\in{\mathcal{S} }^{l}_{K,i}}-\alpha^{l}\frac{\partial\ell}{\partial h^{l}_{j}}+\mathbb{1}(i\in {\mathcal{S}}^{l}_{K,i})\alpha^{l}\frac{\partial\ell}{\partial h^{l}_{j}}. \tag{12}\]

By propagating the gradients backward to the first layer, we can obtain the gradients for each layer of \(\alpha\).

### Training of RGNN

**Hinge objective for RGNN.** For each subquestion \(q^{t}\) at step \(t\), we input its semantic distances into GNN and compute hidden state \(h^{l,t}_{j}\) and message \(m^{l,t}_{j}\) of the \(j-\)th node for \(j\in\{1,\cdots,|{\cal V}|\}\) at layer \(l\). Recall that we adopt a hinge objective that encourages the average integrated semantic distance of supporting passages that should appear at the current and subsequent steps \(t+=\{t,\cdots,T\}\) to be lower than that of the non-target node set for subquestion \(q^{t}\) at step \(t\). We first define the average semantic distances of supporting passages \(\bar{d}^{L,t}_{Y^{t+}}\) and non-target nodes \(\bar{d}^{L,t}_{O/Y^{t+}}\) at step \(t\) as below:

\[\bar{d}^{L,t}_{Y^{t+}}=\frac{1}{|{\mathcal{S}}^{l}_{Y^{t+}}|} \sum_{j\in{\mathcal{S}}^{l}_{Y^{t+}}}\hat{h}^{L,t}_{j}, \tag{13}\] \[\bar{d}^{L,t}_{O/Y^{t+}}=\frac{1}{|{\mathcal{S}}^{l}_{O/Y^{t+}}| }\sum_{o\in{\mathcal{S}}^{l}_{O/Y^{t+}}}\hat{h}^{L,t}_{o}, \tag{14}\]

where \({\mathcal{S}}^{l}_{O}\) is the competitive node set with top \(O\) smallest semantic distances at step \(t\) and \({\mathcal{S}}^{l}_{O/Y^{t+}}\) is the non-target node set by removing the target node set \({\mathcal{S}}^{l}_{Y^{t+}}\) from \({\mathcal{S}}^{l}_{O}\), respectively. We formulate the hinge objective function with threshold \(r\) for the RGNN as follows:

\[\ell =\frac{1}{T}\sum_{t\in[T]}\ell^{t}\] \[=\frac{1}{T}\sum_{t\in[T]}\max(0,r+\bar{d}^{L,t}_{Y^{t+}}-\bar{ d}^{L,t}_{O/Y^{t+}}).\] (15) 

We update the parameters of RGNN using gradient descent, and the backward process is presented as below. With only partial nodes accepting messages from neighbors, we first define the recieved node set \({\cal S}_{R}^{l,t}\) at layer \(l\) at step \(t\) as follows:

\[{\cal S}_{R}^{l,t}=\begin{cases}{\cal S}_{O}^{l}\odot{\cal S}_{K}^{L,t},&l=L\\ &{\cal S}_{K}^{l,t},&\text{else}\end{cases}. \tag{16}\]

For each node \(i\in{\cal S}_{R}^{L,t}\) at the last layer \(L\), the gradient of \(\ell^{t}\) w.r.t. \(\hat{h}_{i}^{L,t}\) is:

\[\frac{\partial\ell^{t}}{\partial\hat{h}_{i}^{L,t}}=\begin{cases}1.0,&i\in{ \cal S}_{Y^{t}}^{t}\\ -1.0,&\text{else}\end{cases}. \tag{17}\]

Therefore, the gradient of \(\ell\) w.r.t. \(\hat{h}_{i}^{L,t}\) is:

\[\frac{\partial\ell}{\partial\hat{h}_{i}^{L,t}}=\sum_{\tau=t}^{T} \frac{\partial\ell^{\tau}}{\partial\hat{h}_{i}^{L,t}}=\frac{\partial\ell^{ t}}{\partial\hat{h}_{i}^{L,t}}+\frac{\partial\ell}{\partial\hat{h}_{i}^{L,t+1}}(1- \beta), \tag{18}\]

where

\[\frac{\partial\ell}{\partial\hat{h}_{i}^{L,T}}=\frac{\partial\ell^{ T}}{\partial\hat{h}_{i}^{L,T}}. \tag{19}\]

Each node \(j\in{\cal V}\) at step \(t\) transfers the message to the same node at the next step through \(\beta\). Therefore, the gradient of loss \(\ell\) with respect to \(\beta\) is computed by:

\[\frac{\partial\ell}{\partial\beta}=\frac{1}{T}\sum_{t=1}^{T}\frac {1}{|{\cal V}|}\sum_{j\in{\cal V}}\frac{\partial\ell}{\partial\hat{h}_{j}^{L, t}}(h_{j}^{L,t}-\hat{h}_{j}^{L,t-1}). \tag{20}\]

Recall that \(\hat{h}_{i}^{L,t}=\beta h_{i}^{L,t}+(1-\beta)h_{i}^{L,t-1}\) for each node \(i\in{\cal V}\). The gradient of loss \(\ell\) with respect to \(h_{i}^{L,t}\) is computed by:

\[\frac{\partial\ell}{\partial h_{i}^{L,t}}=\beta\frac{\partial \ell}{\partial\hat{h}_{i}^{L,t}}. \tag{21}\]

With \(\frac{\partial\ell}{\partial h_{i}^{L,t}}\), we propagate the gradient backward to the GNN and update the parameter \(\alpha^{l}\) layer by layer using (11) and (12).

## Appendix B Additional Experimental Details

**Datasets processing.** We measure all the methods on four different QA datasets, including multihop Wikipedia reasoning datasets: (1) MuSiQue (Trivedi et al., 2022), (2) IIRC (Ferguson et al., 2020), (3) 2WikiMQA (Ho et al., 2020), and a single-hop multi-choice QA dataset: (4) Quality Pang et al. (2021). For each multi-hop dataset, we randomly sample 500 questions from the development set, which provides the ground-truth supporting passages. We use 20 of them to train the GNN and select the rest as test data. We use all the Wikipedia documents of these 500 questions as the retrieval corpus, which is significantly larger than that provided by datasets4. For Quality, we randomly sample 30 articles with overall corresponding 561 questions and use these articles as the retrieval corpus. For each document or article, we split it into multiple passages with a maximum token length of 200 (Sarthi et al., 2024).

Footnote 4: Prior works often use 10–20 paragraphs or documents as the retrieval corpus for each question (Shao et al., 2023; Wang et al., 2024b).

**Graph of passages construction.** We construct a large graph of passages to serve as the retrieval corpus for each evaluation dataset. For the Quality dataset, we randomly sample 30 articles with a diverse range of topics as the retrieval corpus. For multi-hop reasoning datasets, we first collect the Wikipedia documents using the provided titles required for answering these 500 questions. For the multi-hop reasoning datasets, we collect Wikipedia documents based on the provided titles required to answer the 500 questions. We chunk the documents into smaller passages and record their sequential order. Passages that are physically adjacent to each other are regarded as structure-related and connected together in the graph. Besides, to find out the keyword-related passages, we also extract the keywords for each passage and then connect those that share the same keywords. Specifically, we prompt ChatGPT (gpt-3.5-turbo-2023-06-01-preview) to extract the Wikipedia keywords from the passages and generate their corresponding links. The links are used to ensure the consistency of the Wikipedia document that use different keywords in passages. The prompt for keyword extraction is given as follows:

```
Input for extracting keywords
```

Instruction: Extract the entities exist in this text and then provide thewikipedia links for the entities exist in this text. Output the entities and their wikipedia links in the list format, e.g., ['entity1', 'link1'], ['entity2', 'link2']].  <Passage> ``` 

The passages share the same links of keywords are considered as keyword-related and connected together in the graph.

**Implemental details of training.** We initialize \(\alpha=0.1\) for GNN-Ret and \(\alpha^{1}=1.0\) and \(\beta=1.0\) for RGNN-Ret, respectively. We then use gradient descent to update the GNN and RGNN. For each iteration, we compute the average gradient of all the training samples and update the parameters with a learning rate of 1.0. The training will stop when the absolute value of gradient is smaller than 0.001 or the loss keeps increasing for five consecutive iterations. As there is often the whole label set of supporting passages for the question but not the individual index for each subquestion, we select the one with the lowest semantic distance as the label for each subsection and then remove this index label from the label set for subsequent subquestions. Since training the RGNN requires ground-truth subquestions, we manually generate subquestions for 5 questions sampled from the preset training data. The training samples of MuSiQue, IRC, and 2WikiMQA datasets for the RGNN are displayed in Appendix J. We also display the instructions and prompt templates of all the methods in Appendix I.

## Appendix C Selection of \(K\) and \(O\).

In order to effectively train the GNN and RGNN models, we conduct grid search for hyperparameters \(K\) and \(O\), which determine the number of relevant nodes to be sampled and the size of the competitive node set, respectively. To assess the impact of different values of \(K\) and \(O\), we evaluate the accuracy of _GNN-Ret_ and _RGNN-Ret_ on the MuSiQue and IIRC datasets with varying \(K\) and \(O\). The average accuracy across these two datasets is represented by the green points in Fig. 4. Results demonstrate that both _GNN-Ret_ and _RGNN-Ret_ consistently achieve stable accuracy performance across different settings of \(K\) and \(O\). Upon analyzing the results, we observe that _GNN-Ret_ achieves the highest accuracy when \(K=5\) and \(O=25\), while _RGNN-Ret_ performs best with \(K=5\) and \(O=10\). _GNN-Ret_ adopts a larger value of \(O\) since it considers the whole label set during training, while _RGNN-Ret_ only considers the labels for the current step and subsequent steps. Therefore, we select these settings of \(K\) and \(O\) to train the GNN and RGNN models for experiments reported in Table 1.

## Appendix D Statistics of retrieval accuracy.

We explore the EM between the retrieved passages and the ground-truth passages and collect the exact-match number of test samples with varying numbers of supporting passages required in questions on MuSiQue and 2WikiMQA datasets. We do not evaluate the statistics of retrieval accuracy for _RGNN-Ret_ without the specific ground-truth supporting passages for each subquestion. Results in Fig. 5 show that our proposed _GNN-Ret_ achieves higher retrieval accuracy compared with _SBERT_ on MuSiQue and 2WikiMQA datasets. For instance, _GNN-Ret_ outperforms _SBERT_ by 29 and 43 exact-match test samples for the questions that require 2 supporting passages. This demonstrates that the GNN indeed improves retrieval coverage of supporting passages and the retrieval prioritization of supporting passages for subsequent reasoning.

## Appendix E Ablation Studies of Token Number for Retrieval

We conduct experiments with varying numbers of tokens for retrieval on _SBERT_ and _GNN-Ret_ with the long-context model Qwen/Qwen2-7B-Instruct Yang et al. (2024), which can handle more than 100k tokens. Results in Table 6 show that our proposed _GNN-Ret_ consistently outperforms _SBERT_ with varying numbers of tokens for retrieval, demonstrating the robustness of our method for long contexts.

## Appendix F Extending GNN-Ret to Multiple Layers

Our proposed GNNs are designed to be extensible across multiple layers. To explore this capability, we conducted additional experiments by increasing the number of GNN layers. The results, along with the final learned values of \(\alpha\), are presented in Table 7. The results indicate that _GNN-Ret_ maintains consistent performance across different layer configurations. This observation could be attributed to the fact that many questions in datasets require only two-hop supporting passages, suggesting that

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline Token number for retrieval & 1k & 3k & 5k & 10k & 20k \\ \hline SBERT & 18.3 & 23.8 & 22.5 & 22.5 & 27.9 \\ GRN-Ret & **20.4** & **25.8** & **25.8** & **26.7** & **31.9** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Accuracy of GNN-Ret and SBERT with varying token numbers for retrieval using Qwen/Qwen2-7B-Instruct on MuSiQue dataset.

 

a single-layer GNN is adequately equipped to address these queries.

## Appendix G Effectiveness of RGNN-Ret for other multi-hop answering methods

To further demonstrate the effectiveness of _RGNN-Ret_, we supplement experiments by employing it with other multi-hop baselines _IRCoT_ and _ITERRETGEN_. These methods utilize the generated reasons for retrieval, followed by the generation of the next-step reason or the final answer. Results in Table 8 show that _RGNN-Ret_ consistently and significantly improves the accuracy performance for other multi-hop baselines, which demonstrates the effectiveness and adaptability of _RGNN-Ret_ in enhancing the retrieval performance for multi-hop answering processes.

## Appendix H Qualitative Results

We analyze the qualitative results in experiments and demonstrate the effectiveness of _GNN-Ret_ and _RGNN-Ret_ in improving accuracy for QA in this section.

_GRNN-Ret_**improves the retrieval coverage of supporting passages.** We compare the retrieval process between _SBERT_ and _GNN-Ret_ and display the results on Table 9. _GNN-Ret_ can retrieve all of the supporting passages while _SBERT_ fails in both of these two cases. For the first question, _SBERT_ can retrieve the knowledge that '_the director of file Hotel By The Hour is Rolf Olsen._'. Since it considers passages in isolation during retrieval, the supporting passage for the inquiry information ('_which country is Rolf Olsen from_') cannot be retrieved with a poor semantic distance. In contrast, _GNN-Ret_ is able to retrieve both of the supporting passages about 'the director of file Hotel By The Hour is Rolf Olsen' and 'Rolf Olsen was an Austrian actor'. Consequently, it can output the correct final answer. This is attributed to the fact that _GNN-Ret_ takes relatedness between passages into account and thus allows the supporting passages for inquiry information to accept the effect of semantic distances from those for background information, thereby improving the retrieval coverage of supporting passages.

_RGNN-Ret_**better determines the terminal of answer process compared with _SelfAsk_. We analyze the qualitative results for _RGNN-Ret_ and _SelfAsk_ since they have the similar answering procedures by generating subquestions and use them for retrieval. The qualitative results of them are shown in Table 10. For the first question '_Which award the performer of song One More Time (Joe Jackson Song) earned?_', _SelfAsk_ generates the first-step subquestion and also answer it correctly. It obtains the knowledge that 'the performer of the song One More Time (Joe Jackson Song) is Joe

Figure 4: Accuracy of _GNN-Ret_ and _RGNN-Ret_ with various \(K\) and \(O\) in Musique and IIRC datasets. The average accuracy of two datasets are displayed in green points.

Figure 5: Exact-match number of test samples with varying numbers of supporting passages required for QA on MuSiQue and 2WikiMQA datasets.

 

Jackson'. However, it terminates the answering process at this step and thus outputs the incorrect final answer. In contrast, _RGNN-Ret_ understands that the generated intermediate answers are not sufficient to output the final answer and thus continues the next-step answering. Consequently, it can output the final answer '_Grammy_'. This qualitative comparison between _RGNN-Ret_ and _SelfAsk_ demonstrates the effectiveness of our proposed _self-critique_ technique in determining the termination of the answering process and improving the accuracy for QA.

_RGNN-Ret_** enhances the retrieval process for subquestions.** For the second question '_What other notable work did the creator of Shrek make?_' in Table 10, both _SelfAsk_ and _RGNN-Ret_ are able to correctly answer the first subsection and generate the second-step subquestion 'What other notable works did William Steig create?'. However, _SelfAsk_ cannot retrieve the knowledge about 'other work of William Steig', which locates in another passage about the book 'Doctor De Soto'. Using SBERT for retrieval fails to retrieve this passage for QA. In contrast, _RGNN-Ret_ retrieve this passage since it enhances the semantic distance by integrating with the small semantic distances from the previous step. Consequently, _RGNN-Ret_ outputs the correct answer. This qualitative example demonstrates that the RGNN can enhance the retrieval over steps for subquestions and thus improve the accuracy for QA.

## Appendix I Prompts for Experiments

We display prompt templates of all the methods in this section.

### Prompt template for Direct.

The prompt template for _Direct_ is shown as follows:

\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{GNN-Ret} & \multirow{2}{*}{Learned values of \(\alpha\)} & \multicolumn{3}{c}{MuSiQue} & \multirow{2}{*}{Learned values of \(\alpha\)} \\ \cline{4-9}  & & F1 & EM & Acc & & F1 & EM & Acc \\ \hline \hline
1 layer & \(\alpha=[0.326]\) & 31.1 & 17.5 & 27.3 & \(\alpha=[0.277]\) & 48.1 & 33.5 & 46.9 \\
2 layers & \(\alpha=[0.564,0.821]\) & 31.5 & 16.3 & 27.3 & \(\alpha=[0.582,0.578]\) & 47.0 & 33.3 & 46.5 \\
3 layers & \(\alpha=[0.506,0.511,0.944]\) & 30.8 & 16.3 & 26.7 & \(\alpha=[0.504,0.597,0.802]\) & 48.6 & 34.4 & 47.5 \\ \hline \hline \end{tabular}

Table 7: F1, EM, and Accuracy of GNN-Ret with varying layers on MuSiQue and 2 WikiMQA datasets.

\begin{table}
\begin{tabular}{r r r} \hline \hline \multicolumn{1}{c}{Method} & \multicolumn{1}{c}{**MuSiQue**} & \multicolumn{1}{c}{**2WikiMQA**} \\ \hline \hline IRCoT & 27.5 & 44.6 \\ IRCoT + RGNN-Ret & 30.4 (+2.9) & 47.5 (+2.9) \\ ITER-RETGEN & 27.1 & 43.8 \\ ITER-RETGEN + RGNN- Ret & 29.8 (+2.7) & 47.5 (+3.7) \\ \hline \hline \end{tabular}
\end{table}
Table 8: Accuracy of IRCoT (w. RGNN-Ret) and ITER-RETGEN (w. RGNN-Ret) on MuSiQue and 2 WikiMQA datasets.

 



**Question: Which country the director of film Hotel By The Hour is from? Ground-truth answer: Austria**

_SBERT_

**Retrieved passages: (1) Hotel by the Hour (German title:) is a 1970 West German crime film directed by Rolf Olsen and starring Curd Jurgens, Andrea Rau and Corny Collins. (2)... Final answer: Rolf Olsen is from Germany. _GNN-Ret_**

* 3 April 1998) was an Austrian actor. Final answer: Rolf Olsen is from Austria. Question: What is the date of death of the performer of song Goodbye Pork Pie Hat? Ground-truth answer: January 5, 1979 _SBERT_

**Reviewed passages: (1)**Goodbye Pork Pie Hat' is a jazz standard composed by Charles Mingus, originally recorded by his sextet in 1959 as listed below, and released on his album. (2)... _Final Answer: February 17, 2017 _GNN-Ret_

* January 5, 1979) was an American jazz double bassist, pianist, composer and bandleader. Final answer: January 5, 1979**

**Family complete for ICCT/KGP**

Instruction: Given the following question, create a final answer to the question. Please answer less than 6 words. Context: <Context> Question: <Question> where context> indicates the retrieved passages. We repeat the question before the length retrieved passages when answering the questions on IIRC and 2WikiMQA datasets for better performance.

\begin{table}
\begin{tabular}{l l} \hline  & \\ \multicolumn{2}{c}{**Question: Which country the director of film Hotel By The Hour is from? Ground-truth answer: Austria**} \\ \hline  & \multicolumn{2}{c}{_SBERT_} \\ \hline
**Retrieved passages:** & \\ \multicolumn{2}{c}{(1) Hotel by the Hour (German title:) is a 1970 West German crime film directed by Rolf Olsen and starring Curd Jurgens, Andrea Rau and Corny Collins. (2)... full Olsen is from Germany. _GNN-Ret_ \\ \hline
* 3 April 1998) was an Austrian actor. _Final answer: Rolf Olsen is from Austria. _GNN-Ret_ \\ \hline  & \\ \multicolumn{2}{c}{_SBERT_} \\ \hline
**Retrieved passages:** & \\ \multicolumn{2}{c}{(1) ‘Goodbye Pork Pie Hat’ is a jazz standard composed by Charles Mingus, originally recorded by his sextet in 1959 as listed below, and released on his album. _GNN-Ret_ \\ \hline
**Retrieved passages:** & \\ \multicolumn{2}{c}{(1)‘Goodbye Pork Pie Hat’ is a jazz standard composed by Charles Mingus, originally recorded by his sextet in 1959 as listed below, and released on his album. _GNN-Ret_ \\ \hline
**Retrieved passages:** & \\ \multicolumn{2}{c}{(1)‘Goodbye Pork Pie Hat’ is a jazz standard composed by Charles Mingus, originally recorded by his sextet in 1959 as listed below, and released on his album. _GNN-Ret_ \\ \hline
**Retrieved passages:** & \\ \multicolumn{2}{c}{(1)‘Goodbye Pork Pie Hat’ is a jazz standard composed by Charles Mingus, originally recorded by his sextet in 1959 as listed below, and released on his album. _GNN-Ret_ \\ \hline
**Retrieved passages:** & \\ \multicolumn{2}{c}{(1)‘Goodbye Pork Pie Hat’ is a jazz standard composed by Charles Mingus, originally recorded by his sextet in 1959 as listed below, and released on his album. _GNN-Ret_ \\ \hline
* January 5, 1979) was an American jazz double bassist, pianist, composer and bandleader. _GNN-Ret_ \\ \hline
**Final answer:** & \\ \end{tabular}
\end{table}
Table 9: Two examples demonstrating effectiveness of _GNN-Ret_. We highlight the shared keywords, correct answers, and incorrect answers in violet, green, and red.





**Question:** Which award the performer of song One More Time (Joe Jackson Song) earned?

**Ground-truth answer:** Grammy

\begin{table}
\begin{tabular}{l l} \hline \hline  & _Self/Ask_ \\ \hline
**Generation:** & \\ Here is the question: & \\ Question: Which award the performer of song One More Time (Joe Jackson Song) earned? \\ Are follow up questions needed here: Yes. & \\ Follow up: Who is the performer of the song One More Time (Joe Jackson Song)? \\ The performer of the song ”One More Time” (Joe Jackson Song) is Joe Jackson. \\ So the final answer to the Question is: Joe Jackson \\ \hline \multicolumn{2}{c}{_RGNN-Ret_} \\ \hline
**Generation:** & \\ Question: Which award the performer of song One More Time (Joe Jackson Song) earned? \\ Next step: What is the name of the performer of the song “’One More Time (Joe Jackson Song)""? \\ Intermediate Answer: The performer of the song ”‘One More Time"’ is Joe Jackson. \\ Next step: What award did Joe Jackson earn for the song ”‘One More Time“’? \\ Intermediate Answer: Joe Jackson earned a Grammy for Best Pop Instrumental Album in 2001 for his album Symphony No. 1. \\ Final answer: Grammy \\ \hline \end{tabular} 
\begin{tabular}{l l} \hline \hline
**Question:** What other notable work did the creator of Shrek make? \\ Ground-truth answer:** Doctor De Soto \\ \hline \multicolumn{2}{c}{_Self/Ask_} \\ \hline
**Generation:** & \\ Question: What other notable work did the creator of Shrek make? \\ Are follow up questions needed here: Yes. \\ The creator of Shrek is William Steig. \\ Are follow up questions needed here: Yes. \\ Follow up: What other notable work did William Steig make? \\ William Steig also wrote ”Sylvester and the Magic Pebble.” \\ So the final answer is: Sylvester and the Magic Pebble \\ \hline \multicolumn{2}{c}{_RGNN-Ret_} \\ \hline
**Generation:** & \\ Question: What other notable work did the creator of Shrek make? \\ Next step: Who is the creator of Shrek? \\ Intermediate Answer: The creator of Shrek is American author William Steig. \\ Next step: What other notable works did William Steig create besides Shrek? \\ Intermediate Answer: William Steig, the creator of Shrek, also wrote and illustrated the children’s picture book ”Doctor De Soto” which was first published in 1982... \\ Final answer: Doctor De Soto \\ \hline \hline \end{tabular}
\end{table}
Table 10: Two generation examples of comparison between _RGNN-Ret_ and _SelfMask_. We highlight the correct answers and incorrect answers in green and red. We omit the retrieved passages in the table.





### Prompt template for SelfAsk

SelfAsk prompts LLMs to generate the follow-up question and answers it with retrieved passages until generating the final answer. The retrieved passages are included into the prompt when the LLMs are prompted to answer the follow-up question.

Prompt template for SelfASk

Instruction: Your goal is to answer the question step by step following procedures of examples. If no follow up questions are necessary, answer the question in five words directly in format: So the final answer is: xxx.

Here are the examples of how to answer the questions: <Examples>

Context (Optional):

<Context>

Question:

<Question>

We repeat the question before the length retrieved passages when answering the follow-up questions on IIRC and 2WikiMQA datasets for better performance.

Prompt template for ITER-RETGEN

Instruction: Given the following question, create a final answer to the question. Please answer less than 6 words.

Here are the examples of how to answer the questions: <Examples>

Context:

<Context>

Question:

<Question>

Let's think step by step.

### Prompt template for RGNN-Ret

_RGNN-Ret_ iteratively generates the next-step subquestion, answers the subquestion, and performs self-critique until the generation of final answer. The prompt template of these procedures are shown as follows:

Prompt template for generating next-step subquestions

Instruction: Your goal is to ask the next step question logically.

Here are the examples of how to answer the questions: <Examples>

Question:

Question:

<Question>

From Template for generating submissions

 Instruction: Your goal is to answer the next step question. I will provide you some wikipedia snippets, and you need to answer the next step question by considering the wikipedia snippets.

Context:

<Context>

Question:

Question>

Question>

The demonstrations of _RGNN-Ret_ for asking the next-step subquestion on MuSiQue, IIRC, and 2WikiMQA datasets are shown as follows:

Demonstrations of _RONATER_ for asking a new category subquestions on MUSQue dataset

Question: Who lived longer, Muhammad Ali or Alan Turing?

Next step: How old was Muhammad Ali when he died?

Intermediate answer: Muhammad Ali was 74 years old when he died.

Next step: How old was Alan Turing when he died?

Question: When was the founder of craigslist born?

Next step: Who was the founder of craigslist?

Intermediate answer: Craigglist was founded by Craig Newmark.

Next step: When was Craig Newmark born?

Question: Who was the maternal grandfather of George Washington?

Next step: Who was the mother of George Washington?

Intermediate answer: The mother of George Washington was Mary Ball Washington.

Next step: Who was the father of Mary Ball Washington?

Question: Are both the directors of Jaws and Casino Royale from the same country?

Next step: Who is the director of Jaws?

Intermediate Answer: The director of Jaws is Steven Spielberg.

Next step: Where is Steven Spielberg from?

Intermediate Answer: The United States.

Next step: Who is the director of Casino Royale?

Intermediate Answer: The director of Casino Royale is Martin Campbell.

Next step: Where is Martin Campbell from? 

Damentations of _KGANN-Rad_ for asking necessary subquestions on 2018M/QA dataset

Question: Who lived longer, Muhammad Ali or Alan Turing? Next step: How long did Muhammad Ali live? Intermediate answer: Muhammad Ali was 74 years old when he died. Next step: How long did Alan Turing live? Question: Who was the paternal grandfather of Princess Alexandrine Of Prussia (1842-1906)? Next step: Who was the father of Princess Alexandrine Of Prussia (1842-1906)? Intermediate answer: Princess Alexandrine Of Prussia (1842-1906) was the daughter of Prince Albert of Prussia. Next step: Who was the father of Prince Albert of Prussia? Question: Who is the mother of the composer of film 404 (Film)? Next step: Who is the composer of film 404 (Film)? Intermediate answer: The composer of film 404 (Film) is Ilayaraja. Next step: Who is the mother of Ilayaraja? Question: Do both films Across The Badlands and A Gutter Magdalene have the directors that share the same nationality? Next step: What is the nationalality of the director of Accross The Badlands? Intermediate answer: The director of Across The Badlands is American. Next step: What is the nationality of the director of A Gutter Magdalene? Intermediate answer: The director of A Gutter Magdalene is American. Next step: Do both films Across The Badlands and A Gutter Magdalene have the directors that share the same nationality? Question: What is the home stadium of the team that Asprey hit a hat trick against on 16 January 1961? Next step: Which team did Asprey hit a hat trick against against on 16 January 1961? Intermediate answer: Asprey hit a hat trick against Charlton Athletic on 16 January 1961. Next step: What is the home stadium of Charlton Athletic?

## Appendix J Training data for RGNN

We manually generate subquestions for 5 questions sampled from the preset training data for MuSiQue, IIRC, and 2WikiMQA datasets. The training samples are shown as follows:

\begin{tabular}{l l}  & Training data of MuSiQue for RGNN \\ \end{tabular}
1. Question: Why did Roncali leave the place of death of the creator of Malchiostro Annunciation? Subquestions: Who is the creator of Malchiorstro Annunciation? Where did Titian die? Why did Roncali leave Venice? \\ \end{tabular}
2. Question: Where did who argued that the country of citizenship of Victor Denisov had itself beome an imperialist power declare that he would intervene in the Korean conflict? Subquestions: What is the country of citizenship of Victor Denisov? Who argued that Russia had itself become an imperialist power? Where did Mao Zedong declare that he would intervene in the Korean conflict?
3. Question: What military overran much of Erich Zakowski's place of birth? Subquestions: What is the place of birth of Erich Zakowski? What military overran East Prussia?
4. Question: How many people were in British colonies where does the London broil cut come from ? Subquestions: Where does the London broil cut come from? How many people were in North American?
5. Question: When was the country established that lies immediately north of the Persian Gulf and the region where the country containing Urim is located? Subquestions: What is the region containing Urim? Where is the region that Iraq is located? What is the country that lies immediately north of the Persian Gulf and the Middle East? When was Iran established? 

* [1] Questoin: Was the city where Eva was born the capital of its country? Subquestions: Where was Eva born? Is Berlin the capital of its country?
2. Question: In what state did Galambos attend high school? Subquestions: What high school did Galambos attend? In what state is Athens High School located?
3. Question: What was the previous name of the team that Feng started playing with in 1999? Subquestions: What team did Feng start playing with in 1999? What was the previous name of Chongqing Longxin?
4. Question: How many years after the first Marvel Cinematic Universe film came out was Black Panther released? Subquestions: When was the first Marvel Cinematic Universe film released? When was Black Panther released? How many years after Iron Man came out was Black Panther released?
5. Question: Who was the first draft pick the year Damarius Bilbo went undrafted? Subquestions: What year did Damarius Bilbo go undrafted? Who was the first draft pick in 2006?