Few-shot Reranking for Multi-hop QA via Language Model Prompting
Year: 2023

Abstract:
We study few-shot reranking for multi-hop QA (MQA) with open-domain questions. To alleviate the need for a large number of labeled question-document pairs for retriever training, we propose PROMPTRANK, which relies on language model prompting for multi-hop path reranking. PROMPTRANK first constructs an instruction-based prompt that includes a candidate document path and then computes the relevance score between a given question and the path based on the conditional likelihood of the question given the path prompt according to a language model. PROMPTRANK yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-theart methods trained on thousands of examples — 73.6 recall@10 by PROMPTRANK vs. 77.8 by PathRetriever (Asai et al., 2020) and 77.5 by multi-hop dense retrieval (Xiong et al., 2021).1

heading: 1 Introduction
text: Many information-seeking queries are in the form of multi-hop questions. For instance, to answer the question “What 1988 Christmas comedy film did Brian-Doyle Murray star in?”, we need to (i) search for movies starring Brian Murray, then (ii) identify which of them were released in 1988 during Christmas. Evidence required to answer such questions is often dispersed in different documents, requiring sequential, multi-step reasoning to reach the answer (Perez et al., 2020), typically referred to as multi-hop question answering (MQA).
Given a multi-hop question and a large document corpus, existing MQA systems largely follow a retrieve-then-read pipeline, where a retriever module first identifies relevant documents from the corpus, and a reader module produces the answer based on the retrieved output (Asai et al.,
∗Correspondence to khalifam@umich.edu 1Code available at https://github.com/mukhal/
PromptRank.
2020; Li et al., 2021; Singh et al., 2021; Qi et al., 2021). The retriever module is trained to predict the ground-truth evidence document(s) given the question (Karpukhin et al., 2020; Qi et al., 2021; ). However, curating large datasets of question-document pairs is expensive, especially for low-resource languages or domains that require unique expertise (e.g., medical or legal documents), thus creating a bottleneck for building QA pipelines (Ram et al., 2022). Moreover, resorting to heuristics for data labeling can lead to incorrect annotation (Izacard and Grave, 2021). This difficulty is further exacerbated in the case of multi-hop questions, as they need to be annotated with multiple support documents.
The majority of existing data-efficient retrieval and reranking methods are restricted to singlehop QA, and it is unclear how to extend them to the multi-hop setting. For instance, Ram et al. (2022) proposed “recurrent span retrieval” to obtain psuedo question-document pairs in an unsupervised way for single-hop QA. However, in the multi-hop
15882
case, it is less likely that we can retrieve recurrent spans from multiple documents that follow a valid reasoning trajectory. Moreover, their method requires intensive pretraining on the obtained corpus. Seonwoo et al. (2021) focus on weakly supervised multi-hop QA retrieval, yet their method uses corpus-specific (e.g., Wikipedia) heuristics and also requires pretraining. This motivates the need for data-efficient multi-hop retrieval methods that (i) work out-of-the-box without requiring additional (pre)training, and (ii) do not rely on hand-designed heuristics for data collection and annotation.
To this end, we present PROMPTRANK, which leverages the power of large language models (LLMs) for few-shot multi-hop retrieval. PROMPTRANK combines a simple unsupervised retrieval method i.e., TF-IDF similarity, with an LLM reranker that scores the relevance of document paths to a question based on the conditional likelihood of generating the question given the path. Our approach makes use of instruction-based prompting (Sanh et al., 2021; Ouyang et al., 2022) to steer the LLM towards assigning higher scores to more relevant support document chains.2 To calibrate the model’s reranking scores and alleviate prompt sensitivity (Zhao et al., 2021b), we borrow techniques from the literature such as temperature scaling (Kull et al., 2019) and instruction ensembling (Schick and Schütze, 2021a). We also utilize demonstration ensembling to leverage more examples than what can fit into the context of transformer LLMs by combining reranking probabilities computed with different demonstrations.
2We use path and chain interchangeably throughout the paper.
We evaluate few-shot PROMPTRANK on HotpotQA (Yang et al., 2018), a standard MQA benchmark, and show that it compares favorably against state-of-the-art models while using orders of magnitude fewer examples. More precisely, with only 128 training examples, PROMPTRANK outperforms DrKit (Dhingra et al., 2020) and is only 4.2 Recall@10 points lower than multi-hop dense retrieval (MDR) (Xiong et al., 2021) (see Figure 1). We also showcase PROMPTRANK as part of a QA pipeline, again, displaying close QA performance to fully-supervised retrievers—only 4.1 F1 points lower than MDR.
In summary, our contributions in this paper are:
1. We propose PROMPTRANK, a few-shot reranking approach for multi-hop QA that reranks a given document path based on the likelihood of generating the question given a path prompt. 2. PROMPTRANK exhibits strong few-shot retrieval performance with as few as 128 examples and compares favorably to fully supervised methods (§3.1). 3. PROMPTRANK leads to strong QA performance when combined with a pretrained reader module, performing close to fully-supervised retrievers (§3.2).

heading: 2 PROMPTRANK
text: An overview of the full retrieval system is displayed in Figure 2: Given a question q, the system expands sequences of supporting documents into paths of length H , which are used to answer the question. At each step, we first use TF-IDF similarity to ob-
tain an initial set of supporting document paths.3 We then use PROMPTRANK to rerank the current document chains based on their relevance to the question (§2.1).
Concretely, we start with retrieving F candidate documents using TF-IDF for the ‘first hop’. These ‘1-hop’ candidates are scored by PROMPTRANK and K1 top-ranked documents are kept and further expanded based on their hyperlinks to obtain 2-hop reasoning paths.4 These 2-hop reasoning chains are again reranked and the most promising K2 candidates are further expanded. The process repeats until we obtain paths of length H , where H can be a hyperparameter.5 As the document graph can have a high branching factor, we only keep the top-L hyperlinks as reranking candidates based on TF-IDF similarity between the hyperlink document and the question. We have found this pruning step to improve efficiency without much performance drop. This process is shown in Figure 2(a).

heading: 2.1 Path Reranking with PROMPTRANK
text: Given a question q and a reasoning path or chain c, we use an LM to score c according to its relevance to q. Concretely, we measure the likelihood of the question given the path as follows:
Scoreq(c) = PLM(q|τc) (1)
where PLM(q|τc) is the conditional probability of generating the question given a prompt τc containing path τc using an LM. Our initial experiments show that using PLM(q|τc) works substantially better than PLM(c|τq) for a question-containing prompt τq, which agrees with the findings in dos Santos et al. (2020).6 We argue that two factors contribute to this gap. First, LMs can be sensitive to the surface form (Holtzman et al., 2021) of reasoning paths, making it difficult to reliably compare the probabilities of different reasoning paths using PLM(c|τq). For instance, PLM(c|τq) tends to be higher for shorter paths. On the other hand, PLM(q|τc) does not suffer from this issue since we compare the probabilities of the same string (i.e., the question) by conditioning on different
3PROMPTRANK is agnostic to the retrieval approach and can be combined with other retrieval techniques.
4We assume the presence of hyperlinks following previous work (Asai et al., 2020; Qi et al., 2021) although PROMPTRANK is agnostic to how a candidate path is obtained.
5This process can be viewed as a variation of beam search. 6Earlier experiments showed that the recall of PLM(q|τc)
was at least 60% better than that of PLM(c|τq).
reasoning paths. Second, the prompt format using PLM(q|τc)—the question follows a document— agrees more with the web data used for LM pretraining, where documents are usually followed by FAQs, questionnaires, and surveys, rather than the other way around. We further add a temperature parameter to scale the model output logits before computing P (q|τc). This can be seen as an instance of model calibration (Guo et al., 2017; Desai and Durrett, 2020; Jiang et al., 2021) with the goal of improving the reranking scores. We show that temperature scaling boosts reranking performance in §3.1.
Constructing Prompt τc As shown in Figure 2 (b), the prompt consists of an instruction along with the document path. The instruction’s goal is to encourage higher scores for more relevant paths by eliciting the LM reasoning ability (Ouyang et al., 2022). We note that the instruction part is fixed across all prompts constructed for different candidate paths.
The path is expressed in the prompt by concatenating all documents in the chain and prepending each document with a fixed prefix, such as “Document:” or “Passage:”. The concatenation of path documents significantly improves reranking by simultaneously considering all hops, which allows the LM to do a context-aware evaluation of path relevance.

heading: 2.2 Instruction Search and Ensembling
text: Although instructions can be manually engineered to trigger the LM to accomplish the task (e.g., “Read the following documents and generate a question”), this requires human expertise and can be sub-optimal. Therefore, we leverage automated instruction search Gao et al. (2021), where we use an encoder-decoder LM, e.g., a T5-Base model (Raffel et al., 2020), that is trained to fill masked text spans to generate instructions.
Specifically, we fill in the template “Task: <X> documents and <Y> question. Question:”, where <X> and <Y> are the masked spans expected to be filled in by the model (e.g., for a human-written instruction example, <X> = “Read the following” and <Y> = “answer the”). We consider two variations of this template corresponding to the cases where the document path appears before/after the template. We constrained the template to contain the words ‘documents’ and ‘question’ to ensure that the model generates relevant prompts. We have
found that using a less specific template without such tokens leads to more diverse but less relevant instructions. The exact templates used are in Appendix A.2.
Previous work has shown that mixing multiple prompts can improve few-shot performance (Gao et al., 2021; Schick and Schütze, 2021b). Similarly, such ensembling could produce more regularized path scores by alleviating prompt sensitivity (Zhao et al., 2021b). Given a path, we combine the scores of obtained through different instructions. We experiment with both mean and max ensembling.

heading: 2.3 Demonstration Ensembling
text: We employ in-context learning (ICL) (Brown et al., 2020) to teach the LLM to do reranking by showing the model examples i.e., demonstrations of questions and their gold paths. A major obstacle to this approach is the input length limit in standard transformer LMs. Since paths are comprised of multiple documents, in most cases we cannot feed more than two demonstrations without exceeding the limit of 1024 tokens, a standard setup for pretrained LMs. To workaround that, we utilize demonstration ensembling, where different in-context demonstrations are used to compute scores for a given path, and the scores are combined by a mean or max operation.

heading: 3 Experiments
text: Data We evaluate our method on HotpotQA (Yang et al., 2018), which consists of two-hop questions over diverse topics. We focus on the fullwiki setting in which two Wikipedia passages are required to answer the questions. Since the gold passages for the test set are not available, we follow prior work and evaluate PROMPTRANK on the development set, which has 7,405 questions. There are two main question types in HotpotQA: (1) comparison questions usually require contrasting two entities and (2) bridge questions can be answered by following a connecting entity that links one document to another. We also evaluate fewshot PROMPTRANK on the 2WikiMQA dataset (Ho et al., 2020) in Appendix D.1.
Compute Infrastructure All our reranking experiments are run on a workstation with a single Nvidia A40 GPU and 256GB of RAM. Our QA experiments in §3.2 are run on a workstation with two Nvidia Quadro RTX 8000 GPUs and 128GB of RAM.
Models We use HuggingFace implementations (Wolf et al., 2020) of GPT2-XL (1.5B) (Brown et al., 2020), T5-Base (220M), T5-Large (770M) and T5-XL (3B) (Raffel et al., 2020) in our experiments. We use the ‘LM adapted’ version of T5 models since they have been shown to work better for prompt-based learning (Lester et al., 2021). We report additional results with the OPT-30B model (Zhang et al., 2022) in §4.3.
Hyperparameters For PROMPTRANK, we use a path length of H = 2 for all experiments. For pruning the search space we use K1 = 5 and L = 3. We use the TF-IDF index implemented by Asai et al. (2020) and initially retrieved F = 100 documents from TF-IDF. We truncate path documents to 230 tokens before constructing the prompt and limit the prompt length to 600 tokens. When using in-context demos, we use the maximum length of 1024 tokens.
Metrics Retrieval performance is measured using both Recall (R@k) and Answer Recall (AR@k), with k ∈ {2, 10, 20}. R@k measures whether the two gold documents are present in the top-k retrieved documents and AR@k is the recall of the answer string in the top-k retrieved documents. For HotpotQA, we only compute AR over questions with span answers (we ignore yes/no and comparison questions). Since we do not have access to the HotpotQA test set, we report results on the original development set provided by Yang et al. (2018).
Document Scores We compute document scores from path scores as follows. Similar to Das et al. (2019), we take a document score to be the maximum of all its path scores. We find this change to yield better recall than using path scores, with details elaborated in Appendix B.
Instruction Search and Temperature For instruction search, we generate 200 different instructions as described in §2.2 using top-k sampling with k = 10. Then, we select the best instruction based on R@2 over our development set of 128 examples. The same process is used to select the optimal temperature value. Table A1 shows the best 10 instructions identified.
Baselines We compare our reranker to the following baselines. TF-IDF retrieves top similar documents to the question using TF-IDF similarity and TF-IDF + BM25 adds an extra step where retrieved documents and their hyperlinks are reranked using
BM25 (Robertson et al., 1995). PathRetriever (Asai et al., 2020) is a graph-based retriever trained to expand an initial pool of documents based on Wikipedia links and searches for the best path using beam search.7 DrKIT (Dhingra et al., 2020) is an end-to-end trained dense retrieval approach that starts from question entities and traverses a virtual knowledge base to find the relevant entities. Multihop Dense Retrieval (MDR) (Xiong et al., 2021) encodes the question and the documents retrieved by each step into a dense vector and uses maximum inner-product search (MIPS) to find the next hop.
Below, we start with the evaluation of the zeroand few-shot reranking of PROMPTRANK (§3.1). Then, we move to evaluate downstream MQA performance in the few-shot setting (§3.2).

heading: 3.1 Retrieval Performance
text: Table 1 shows the performance of PROMPTRANK and other comparisons in zero- and few-shot settings.
Zero-shot Performance We start with discussing the retrieval performance of zero-shot PROMPTRANK on HotpotQA. First, we observe that simple TF-IDF performs poorly in terms of different recall metrics, while TF-IDF + BM25
7We run PathRetriever on HotpotQA with original hyperparameters except for an initial TF-IDF pool size=100 to allow for fair comparison to our approach.
performs much better, yet still worse than fullysupervised approaches. Next, we look at the performance of the zero-shot PROMPTRANK (T5-XL) which uses no instructions, i.e., the prompt consists of only the document path. These models obtain better recalls than TF-IDF + BM25 and even outperform the fully-supervised DrKit. Although this approach does not use any labeled data, it is only 3.7 AR@10 points worse than PathRetriever, which is trained on ~90K examples. These findings demonstrate PROMPTRANK’s effectiveness at reranking paths of documents.
Few-shot Performance The zero-shot performance of PROMPTRANK can be further improved with access to a small set of labeled examples (in
our case, we only used 128 examples from HotpotQA) for instruction search and finding temperature value. We observe a substantial boost of 11.6% (42.8 → 47.8) in R@2 of PROMPTRANK when using the best instruction found by instruction search. Furthermore, temperature scaling with T = 1.4 also provides a boost of 3.9% (47.8 → 49.7) points in R@2.
We also observe that instruction ensembling gives a further performance boost, reaching 51.3 R@2 with PROMPTRANK. We show the performance of max ensembling, which we have found to perform better than mean ensembling in terms of R@2. We hypothesize that max ensembling computes an upper bound on the path scores, compensating for any underestimation of path scores that can happen when using a single instruction.
In-context learning We experiment with an N - shot setting while making sure that the two demonstrations cover both question types in HotpotQA (bridge and comparison). Figure 3 shows that both R@2 and AR@2 improve as we use more demonstrations. With only 2 examples, we observe a large boost of 6.3% (49.2 → 52.3) in R@2. Since we cannot fit more than 2 demonstrations in the 1024 context window, we use demonstration ensembling (§2.3). For instance, 6-shot ensembling scores a path by combining 3 different contexts, each obtained using 2 demonstrations. We use max ensembling as it is found to work best. Figure 3 shows the in-context learning performance with a different number of demonstrations. We observe a steady increase in R@2 until N = 8. AR@2 also improves with more demonstrations but drops slightly with N = 10. Interestingly, demonstration ensembling has enabled us to leverage more examples than permitted by the context window size of T5-XL. We leave it to future work to study the applicability of this technique to other tasks.

heading: 3.2 Full QA Performance
text: We analyze the performance of PROMPTRANK when used as the retriever in a QA pipeline. We adopt an extractive reader model based on ELECTRA Large (Clark et al., 2020) with two heads to predict the start and end of the answer span. We use the checkpoint provided by Xiong et al. (2021), and the same inference setting. Details on the inference hyperparameters for the reader are in Appendix C.1.
In Table 2, we compare the QA performance on
HotpotQA development set with PROMPTRANK as the retriever against a fully-supervised retriever, namely MDR (Xiong et al., 2021) as well as unsupervised TF-IDF. PROMPTRANK with Ndemos = 10 is only 4.6 F1 points worse than MDR, which is using the same reader module. Table 3 shows performance on HotpotQA test set with different fullysupervised systems compared to PROMPTRANK (Ndemos = 2), where PROMPTRANK is only 1.9 and 4.2 EM points below PathRetriever and MDR, respectively.

heading: 4 Analysis

heading: 4.1 Comparison to Single-hop Reranking
text: The key idea behind our approach is to conduct joint reasoning with documents in the path using the LM, as opposed to reranking each document in the path separately (single-hop reranking). More specifically, in single-hop reranking, we expand paths using the same setup of PROMPTRANK but rerank each document d separately using p(q|τd), for a given document prompt τd.
To assess whether our multi-hop reranking approach offers the advantage of global reasoning, we compare both approaches by running two experiments with identical settings except for how documents are reranked. For evaluation, we use a set of 4K questions from HotpotQA and T5-Large, and
no instruction is used, i.e., the prompt only contains the document(s). Table 4 shows the retrieval performance of both approaches. Interestingly, a large gap in recall scores is observed between single-hop and multi-hop reranking. This supports our hypothesis that jointly considering multiple documents in the path helps the LM better model documents’ relevance to the question.

heading: 4.2 Role of Instruction
text: Our goal here is to investigate (i) how useful is the presence of the instruction in the prompt, (ii) how much benefit (if any) automated instruction search provides over manual instructions, and (iii) whether the instruction’s location in the prompt matters. To answer these questions, we analyze the recall over 200 different instructions generated using the method described in §2.2 and using 1K examples from HotpotQA with different LM sizes: T5-XL, T5-Large, and T5-Base, with results displayed in Figure 4. This analysis uses an initial set of TFIDF documents of size F = 30.
Usefulness of Instruction We can see that using no instruction consistently yields poorer performance than using an instruction of any sort, across all variants of T5. Interestingly, without the instruction, the three model sizes have almost the same R@2. The difference in their performances becomes apparent when an instruction is added. Strikingly, in the no instruction case, T5-Large performs worse than T5-Base in terms of AR@2, showing that scaling does not consistently help recall when no instructions are used. This hints at the fact that instructions play a major role in harnessing the full power of LLMs, at least for our task.
Benefit of Automated Instruction Search Next, we compare a human-written instruction against an instruction found through automated instruction search on a labeled set of 128 examples. The manual instruction we use is “Please write a question based on these passages.”, which is used by Sachan
et al. (2022).8 In Figure 4, we compare the recall when using these instructions. Interestingly, the search-based instruction outperforms the manual one in almost all cases. We also observe that the manual instruction performs poorly for AR@2 on T5-base, even worse than no instruction. These observations hint at the utility of automated instruction search for path reranking. However, it is worth noting that the best instruction on a relatively small held-out set will not necessarily generalize during test time: The search-based instruction produces AR@2 and R@2 that are almost the same or worse than the median instruction, respectively with T5Large.
Location of Instruction We study the performance of two different kinds of prompts, where the instruction appears before and after the path. Figure 5 shows the R@2 and AR@2 in both cases for T5 models of different sizes. We observe that placing the instruction after the path performs consistently better than placing before it, across all model variants. We hypothesize this to be an instance of the recency bias exhibited by LMs (Zhao et al., 2021b), i.e., placing the instruction right before where the model is asked to generate the question better primes the LM for the task and produces better calibrated path scores. We expect such finding to generalize to other tasks where instruction-based prompting is used.

heading: 4.3 Choice of Language Model
text: Table 5 compares the reranking performance of GPT2-XL and OPT-30B (Zhang et al., 2022) mod-
8We average recall of the two cases where the instruction falls before and after the path. See the next paragraph for more context.
els. Despite having an order of magnitude more parameters, we observe that the OPT model is generally worse compared to the smaller GPT2-XL model. We suspect this is due to domain mismatch between pre-training data and task relevant data. Pre-training data of GPT2 models is potentially more biased towards Wikipedia data compared to the OPT models which are trained on more diverse data. Importantly, this shows that scaling up the language model doesn’t necessarily guarantee better reranking performance and domain gap is an important consideration.

heading: 4.4 Further Analysis and Comparisons
text: We further analyze the inference cost of PROMPTRANK compared to PathRetriever and MDR in Appendix D.2. In Appendix D.3, we study PROMPTRANK’s recall sensitivity to document order in the prompt τc by comparing performance using two different document ordering schemes in the prompt. Lastly, we compare PROMPTRANK to few-shot PathRetriever and LOUVRE (Seonwoo et al., 2021) in Appendix D.4.

heading: 5 Related Work
text: Multi-hop Retrieval The majority of approaches for multi-hop question answering rely on two main components: a retriever and a reader. The retriever component can be a sparse index or heuristic-based
such as TF-IDF or BM25 (Chen et al., 2017; Nie et al., 2019) or dense index (Karpukhin et al., 2020; Xiong et al., 2021; Zhao et al., 2021a). Other approaches aimed to improve the retriever with an additional reranking step on top of a simple retriever (Wang et al., 2018; Lee et al., 2018; Htut et al., 2018). Asai et al. (2020) combined TF-IDF retriever with a recurrent graph retriever and used the reader module to rerank paths based on answer confidence. Qi et al. (2021) used a single transformer model to perform retrieval, reranking and reading in an iterative fashion. However, the good performance of previous work comes mainly from training on a large number of examples and is likely to fail in low-data settings. To treat this issue, Seonwoo et al. (2021) proposed to pretrain MDR (Xiong et al., 2021) on a large number of weakly-supervised examples of questions and the corresponding document paths. Although promising in low-data settings, their pretraining is computationally expensive as it is done on millions of examples. On the other hand, our approach requires no task-specific pretraining.
Language Models Prompting Prompt-based learning aims to construct better inputs, i.e., prompts to language models to elicit better zeroor few-shot performance (Brown et al., 2020; Liu et al., 2021). Recently, instruction tuning, where a language model is trained to follow natural language instruction, has shown impressive zero-shot performance on unseen tasks (Wei et al., 2021; Ouyang et al., 2022). In our work, we use instructions to guide to model toward assigning better scores to more relevant document paths.
LM-based Reranking Our scoring function is related to query likelihood retrieval (Lavrenko and Croft, 2017; Ponte and Croft, 2017) and is in line with previous work that employed generative language models for passage reranking (Nogueira et al., 2020). dos Santos et al. (2020) performed single-hop reranking using question likelihood given the passage, but their setting was limited to fully-supervised, single-hop QA. Concurrent with our work is (Sachan et al., 2022), where the authors leverage LLMs for unsupervised passage reranking for QA. While their focus is on single passages, we study the reranking of multi-passage paths, which is more challenging. Moreover, their exploration of prompting is limited to a single manual instruction, whereas we provide an in-depth analysis of
the effect of different prompting aspects on the recall such as instruction importance, location in the prompt, and manual vs. automated.
Conclusion
We introduced PROMPTRANK, a method to perform few-shot reranking of multi-document paths for multi-hop question answering based on large language models. Experiments on a standard multihop QA benchmark show the strong performance of PROMPTRANK in the few-shot setting compared to fully-supervised multi-hop reranking systems. Future avenues of exploration include combining PROMPTRANK with efficient tuning techniques such as prefix tuning and efficient strategies for instruction search.
Limitations
One limitation to LM-based reranking is the computational overhead involved in reranking paths. Our approach requires a forward pass through the LM to rerank each path, which can become expensive when using relatively large models such as GPT-3 or when dealing with more hop count that creates combinatorially more paths. Another limitation of PROMPTRANK is imposed by the transformer context window length. Since PROMPTRANK requires the prompt to include all path documents, it could be infeasible to fit all path documents into the prompt for paths with a larger hop count. A potential direction to workaround this is to condense or summarize the path documents beforehand. We leave it to future work to explore this and other techniques.

heading: Acknowledgements
text: This work is supported by LG AI Research. Additionally, we would like to thank Sewon Min for providing feedback on the paper. We also thank the anonymous reviewers for their valuable suggestions.

heading: B Document Scores
text: It is not immediately obvious how to compute a final score for each document since PROMPTRANK is mainly used to score document. The main issue is that a document can fall on multiple paths at the same time (some of which could be incomplete or not fully expanded yet) and therefore could have multiple such scores.
For example, assume a path A → B → C of consisting of the documents A, B, and C, respectively. Considering the document B, we see that two scores are associated with B: score of the subpath A → B and score of the full A → B → C path. To compute the final score of B, we could either just take the score of the longest path, or combine the two scores using mean, minimum, or maximum operations. What we found to work best compared to other alternatives is to take maximum, which is similar to what is done in (Das et al., 2019). We use this formulation when computing our recall metrics in §3.1.

heading: C Hyperparameters
text: C.1 ELECTRA Reader
We use the same reader setting as in Xiong et al. (2021), where the top-100 retrieved paths are fed to the reader to obtain an answer from each path. Answers are then sorted based on a linear combination of path score and answer confidence, and the top answer is returned. We use the default hyperparameters for HotpotQA from (Xiong et al.,
2021) in their codebase.9 We use a maximum path length of 512 tokens, maximum question length of 64, and answer length of 30. In their experiments, Xiong et al. (2021) combine the answer confidence along with a ranking score using linear interpolation with a hyperparameter λ. For our experiments, we use the path scores produced by PROMPTRANK instead and learn λ on a held-out development set. The value we end up using for λ is 0.9.

heading: D Further Results and Analysis
text: D.1 Results on 2WikiMQA
Table A2 shows the performance with few-shot PROMPTRANK with only one setting: few-shot, best instruction and with temperature scaling on the 2WikiMQA dataset (Ho et al., 2020). We compare it to two unsupervised baselines, namely TFIDF and TF-IDF+BM25 . PROMPTRANK is significantly outperforming the baselines while using only 128 examples for tuning the instruction and the temperature parameter.
D.2 Inference Cost
Here, we analyze inference cost in terms of latency per query. We run retrieval using each method over 100 queries and then compute the average time per query. Inference was run over a single Nvidia Quadro RTX 8000 GPU. We run each method with the maximum batch size that fits within the GPU. One parameter that highly affects the speed for both PathRetriever and MDR is the beam size. We use the default beam size for PathRetriever, which is 8, and we use a beam size of 5 for MDR, to closely match PROMPTRANK’s pruning parameter K1 = 5. Other than beam size, we use the default parameters for each method.
Table A3 shows the number of parameters of each method and the average time per query in seconds. First, we note that PROMPTRANK uses the most number of parameters since it is based on T5-XLwhile PathRetriever and MDR both rely on much smaller LMs such as BERT and RoBERTa. Interestingly, however, we can see that PROMPTRANK without ensembling has lower latency than PathRetriever, which is slowed down by the beam search process since it has to expand and encode outgoing links from each passage in the beam at each step. As expected, ensembling almost multiplies the latency of PROMPTRANK by
9https://github.com/facebookresearch/ multihop_dense_retrieval
ID Prompt
1 Document: [D1] Document: [D2] , ..., Review previous documents and ask some question. Question:
2 Document: [D1] Document: [D2] , ..., Review the previous documents and answer question. Question:
3 Document: [D1] Document: [D2] , ..., Read the previous documents and write the following question. Question:
4 Document: [D1] Document: [D2] , ..., Search previous documents and ask the question. Question:
5 To analyze the documents and ask question. Document: [D1] Document: [D2] , ..., Question:
6 Document: [D1] Document: [D2] , ..., To read the previous documents and write a question. Question:
7 Document: [D1] Document: [D2] , ..., Read previous documents and write your exam question. Question:
8 Document: [D1] Document: [D2] , ..., Read the previous documents and ask this question. Question:
9 Read two documents and answer a question. Document: [D1] Document: [D2] , ..., Question:
10 Identify all documents and ask question. Document: [D1] Document: [D2] , ..., Question:
Table A1: Top 10 instructions found through automated instruction search (§2.2) using T5-XL. Instructions are sorted in descending order according to R@2 on a held-out development set of size 128 from HotpotQA (Yang et al., 2018). We use the first 5 for instruction ensembling (section §2.2). Blue represents fixed text that does not depend on the path i.e the instruction. The tokens [D1], [D2],.., etc. indicate where path documents are inserted.
R@2 R@10 R@20 AR@2 AR@10 AR@20
Unsupervised Baselines TF-IDF 5.8 8.9 23.8 22.2 37.8 44.2 TF-IDF+BM25 7.6 34.1 45.8 20.7 44.9 53.0
PROMPTRANK, no ICL T5-XL, best inst., temp. scaling 19.3 58.6 62.7 33.9 60.0 64.2
Table A2: Retrieval performance on 2WikiMQA (Ho et al., 2020).
the number of ensembles. Lastly, we note that MDR has significantly lower latency than both models, i.e., about 60x faster than PathRetriever and 46x than PROMPTRANK, which is mainly due to the fast implementation of the exact inner product search (Johnson et al., 2019). It is worth noting, however, that MDR requires an expensive indexing step where every document in the corpus (Wikipedia in our case) is encoded using the document encoder. PROMPTRANK, on the other hand,
can work directly out-of-the-box without requiring such expensive indexing.
D.3 Sensitivity to Document Order
Here, study PROMPTRANK’s recall sensitivity to the document order in the prompt τc by running a simple experiment comparing two document ordering schemes: link-based and inverted link-based. Link-based ordering is the standard approach used in PROMPTRANK, which orders the documents in
System #params Avg. query time(s)
PathRetriever 110M 1.95 MDR 125M 0.03
PROMPTRANK 3B 1.38 PROMPTRANK (ens) 3B 5.22
Table A3: Inference cost and number of parameters of three systems comparing PROMPTRANK to PathRetriever and MDR. Query time is obtained by averaging the time to process 100 queries.
Doc. ordering R@2 R@10 AR@2 AR@10
T5-Large Link-based 44.9 66.9 73.6 88.0
Inverted 44.5 67.7 72.6 87.8
T5-XL Link-based 44.6 67.9 74.1 88.2
Inverted 45.7 69.0 74.4 88.3
Table A4: Retrieval performance of PROMPTRANK using two different orderings of the documents in the prompt. Evaluation is done on a set of 2K examples from HotpotQA train set. PROMPTRANK exhibits minimal sensitivity to the document ordering.
the path based on their Wikipedia hyperlink traversal order. The inverted scheme, reverses the order of the documents in the prompt. No instruction is used for this experiment.
Table A4 shows the retrieval performance with both orderings. Interestingly, reversing the order of the documents in the path does not seem to have a tangible effect on the reranking performance. While it is expected that p(q|τc) will change by reversing the document order in the prompt, it appears that the ranks of different paths remain almost unchanged, which explains why the recall is hardly affected.
In other words, the path scores output by T5-XL does not appear to be sensitive to the document order prompt and can still. This might point to another benefit of LM-based path reranking: Since the performance is hardly affected by the document order, we do not have to worry about finding paths in the correct order (if such order exists) since the LM will still be able to assess the path relevance given different orders.
D.4 Comparison to Few-shot Systems
So far, we have mainly compared PROMPTRANK to systems trained on many more examples. Here we compare PROMPTRANK to few-shot LOUVRE (Seonwoo et al., 2021) and PathRetriever (Asai et al., 2020). To this end, we train PathRe-
triever on N examples from HotpotQA for N ∈ 50, 100, 500, 1000 and compare its performance to PROMPTRANK (Ndemos = 10). Since we were unable to obtain good performance by fine-tuning LOUVRE on few examples, we directly compare to the results reported in their paper, where 1% of training data is used (~90 examples). Table A5 shows performance of both few-shot systems compared to PROMPTRANK. While PathRetriever’s performance improves as we add more examples, we can see that it is much less data efficient than PROMPTRANK. Even with 1K examples i.e., around 10x more data than PROMPTRANK, it performs significantly worse across all metrics. We also observe that PROMPTRANK performs better than LOUVRE in terms of R@2 and AR@2 (more than 6 points better) and very close with respect to other metrics even though PROMPTRANK does not involve any (pre)training.
Approach # Ex R@2 R@10 AR@2 AR@10
PathRetriever 50 7.1 (4.4) 14.5 (4.6) 29.5 (6.2) 40.0 (3.7) 100 10.8 (1.1) 19.1 (0.3) 34.8 (1.5) 43.1 (0.6) 500 15.7 (0.3) 22.4 (0.3) 40.4 (0.3) 46.4 (0.4) 1K 17.7 (0.4) 23.6 (0.3) 41.8 (0.3) 47.1 (0.3)
LOUVRE 1% 53.5 75.5 72.3 –
PROMPTRANK 128 54.4 73.5 78.9 88.9(Ndemos = 10)
Table A5: Retrieval performance of PROMPTRANK compared to Few-shot PathRetriever. We show mean and (std) of PathRetriever’s performance over 5 different seeds. The results of LOUVRE are take directly from (Seonwoo et al., 2021). We observe that PathRetriever performs very poorly in low-data settings, even when using about 10x more data than PROMPTRANK. T
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: We leave it to future work to study the applicability of this technique to other tasks