PaRaDe: Passage Ranking using Demonstrations with LLMs
Authors: Andrew Drozdov
Honglei Zhuang
Zhuyun Dai
Zhen Qin
Razieh Rahimi
Xuanhui Wang
Dana Alon
Mohit Iyyer
Andrew McCallum
Donald Metzler
Kai Hui
Venue: EMNLP 2023 Findings (2023)
Keywords: in-context learning
demonstration
query likelihood
re-ranking
question generation
TL;DR: We investigate the uses of demonstrations for LLM-based re-ranking and question generation.

Abstract: Recent studies show that large language models (LLMs) can be instructed to effectively perform zero-shot passage re-ranking, in which the results of a first stage retrieval method, such as BM25, are rated and reordered to improve relevance. In this work, we improve LLM-based re-ranking by algorithmically selecting few-shot demonstrations to include in the prompt. Our analysis investigates the conditions where demonstrations are most helpful, and shows that adding even one demonstration is significantly beneficial. We propose a novel demonstration selection strategy based on difficulty rather than the commonly used semantic similarity. Furthermore, we find that demonstrations helpful for ranking are also effective at question generation. We hope our work will spur more principled research into question generation and passage ranking.

PaRaDe: Passage Ranking using Demonstrations with Large Language Models
Andrew Drozdov
Honglei Zhuang
Zhuyun Dai
Zhen Qin
Razieh Rahimi
Xuanhui Wang
Dana Alon
Mohit Iyyer
Andrew McCallum
Donald Metzler Kai Hui
Google
UMas Amherst CICS
Abstract
optimal and requires a human-in-the-loop when using the LLM for a new task. Instead, we seek a method that finds effective demonstrations auto- matically, with minimal or no human involvement.
Recent studies show that large language mod- els (LLMs) be instructed effectively per- form zero-shof passage re-ranking, in which the results of first stage retrieval method, such as BM25, rated and reordered im- prove relevance. In this work, we improve LLM-based re-ranking by algorithmically se- lecting few-shor demonstrations to include in the prompt. Our analysis investigates the con- ditions where demonstrations are most help- ful, and shows that adding even one demon- stration is significantly beneficial. We propose novel demonstration selection strategy based on difficulty rather than the commonly used se- mantic similarity. Furthermore, we find that demonstrations helpful for ranking are also ef- fective at question generation. We hope our work will spur more principled research into question generation and passage ranking.
In this paper, we investigate approaches for au- tomatic demonstration selection to improve upon UPR's s zero-shot ranking approach. Our initial anal- ysis highlights the complex nature of the problem, showing that ranking performance varies drasti- cally depending on the demonstrations included in the prompt. Furthermore, simply including more demonstrations does not always lead better ranking quality. Next, we investigate the use of established demonstration selection methods, i.e. similarity-basee selection (Rubin et al., 2022; Luo et al., 2023), on ranking tasks and show that similar- ity of demonstrations does not correlate well with ranking quality. Thereafter, we propose difficulty- based selection (DBS) as simple and effective approach automatically find challenging, i.e. low likelihood, demonstrations to include the prompt. Although we prompt frozen LLMs, we intend to emulate the training dynamics of fine-tuning, and choose hard samples because they potentially cor- respond to large gradient updates and are often cho- sen to improve learning in gradient descent (Shri- vastava al., 2016; Chang al., 2017). Finally, given the increasing importance of question gener- ation for ranking (Nogueira et al., 2019; Bonifacio et al., 2022; Dai et al., 2023; Jeronymo et al., 2023), we extend the uses of the proposed difficulty- selection for better question generation.
1 Introduction
Large language models (LLMs) exhibit strong per- formance variety of tasks without additional task-specific fine-tuning. Their success is often attributed to in-context learning, where the param- eters of the language model are frozen and it learns how to perform new task by reading demonstra- tions in the prompt (Brown et al., 2020; Basu et al., 2023; Min et al., 2022; Akyürek et al., 2023).
While LLMs are often used to generate answers, focus is scoring for the task of passage re- first retrieved by efficient retriever, e.g. BM25, then rated and reordered by the LLM. Existing works like UPR (Sachan et al., 2022) demonstrate promising results for zero-shot ranking using LLM. We aim to improve over zero- shot ranking by including demonstrations in the prompt and explore multiple strategies for select- ing demonstrations, Manual selection is often sub-
To this end, we present Passage Ranking with Demonstrations (PaRaDe). Our main contributions include: (1) analysis highlighting the complexity of demonstration selection; (2) DBS, automatic and effective way to choose demonstrations: and (3) extensive experiments on re-ranking and question generation, including results with an extension of DBS that jointly selects multiple demonstrations.
'Work completed while Student Researcher at Google, Final author. Also generated Table I results using DQL- scored demonstration candidates from AD.

---

2 LLM Re-ranking by Query Likelihood
1.0
0-shot
Background. Given a query q and of initially retrieved documents D, the goal is to rank each document d in D by relevance with respect to G. UPR (Sachan et al., 2022) introduces a zero- shot method to rank documents according to the log-likelihood of given d using large language model,
0.8
nDCG@10
0.6
0.4
d) d)
Σ log P(g d, qua 1), (1) 1=1.NN
0.0
Query ID
which resembles the query likelihood retrieall model (Ponte and Croft, 1998). It is factorized using the probabilities of each token in the query qi and prefix of that token 9ı 1. Extending QL to include demonstrations in the context yields,
Figure I: Statistics for nDCG@ 10 on TREC 2020, aggregated using the same query with 100 different one-shot demonstrations. Flan-T5-XXL is used for re- ranking. Zero-shot results included for reference.
E(g) d) x
Σ i=1.NN
log P( 21,... Zk, d, Mi 1),
Demonstrations influence ranking. Figure shows ranking performance of zero-shot and one- shot prompts, and we can see that LLMs are quite sensitive to the choice of demonstrations. We ran- domly sampled 100 demonstratinns for use in one- shot re-ranking with query likelihood and also com- pare against zero-shot. On 25,95 25.9% of queries the minimum one-shot nDCG@ 10 outperforms zero- shot, while zero-shot outperforms the max one-shot for 11.1 of queries. It is worth emphasizing that there a high variance across different one-shot demonstrations on many queries.
(2)
where each z is a positive query-document pair.
3 Experiments on TREC and BEIR
To empirically measure the effectiveness of demon- stration selection, we conduct analysis using the re- ranking task on TREC 2019 and 2020, and further perform evaluation on seven datasets from BEIR (Thakur et al., 2021). We use language models known effectively incorporate demonstrations through in-context learning, including the Flan-T5- XL and XXL (Chung et al., 2022) and the more recently PaLM 2-S (Google et al., 2023).
Increasing demonstrations does necessarily help. Surprisingly, we find little benefit when ran- domly sampling more demonstrations beyond one- shot (see Figure 2). There is a minor, almost neg- ligible improvement in median performance when using four demonstrations, and even less change with eight. We do notice slightly decreased varia- tion as we increase the number of demonstrations. This highlights the difficulties in selecting demon- strations for ranking tasks beyond one-shot.
Setup. For each dataset, we retrieve the top-100 100 documents using BM25 from Pyserini (Lin et al., 2021). The LLMs re-rank these top documents using query likelihood ($2) in a point-wise man- ner, and the re-ranked results evaluated using nDCG@ 10. Herein, the instruction (Table 4) and the selected demonstrations composite the prompt string when scoring each (query, passage) pair.
Given the large variance in one-shot perfor- mance and the difficulty to improve performance by increasing demonstrations, we need an effective way to select high performing demonstrations.
4 Demonstrations for Ranking
4.1 The Impact of Demonstrations
In this section, we investigate the helpfulness of demonstrations for ranking and sensitivity to the choice of demonstration. We explore multiple strategies for selecting demonstrations: random sampling, similarity-based selection (SBS), and our new approach difficulty-based selection (DBS). Our findings indicate that the choice of demonstra- tions considerably impacts ranking performance.
Similarity- selection is limited. A simple and widely used baseline for selecting demonstra- tions is semantic similarity (Rubin et al., 2022; Luo et al., 2023). Intuitively, it makes sense that mantically similar demonstrations the test query would help teach the LLM how to re-rank, al- though, if the LLM is already familiar with the demonstrations then is not clear whether they

---

will prove helpful. We perform post-hoc analysis on our TREC 2020 experiments with 100 random one-shot demonstrations. similarity using cosine similarity and Universal Sentence Encoder (Cer et al., 2018) embeddings of the demonstration and test queries. By comparing the semantic similarity and the nDCG, we ascertain there is little no correlation between high seman- tic similarity and strong re-ranking. Our findings show this correlation is significant only 5% of the time, thus conclude that similarity alone has clear limitations for demonstration selection. In the next subsection, we explore a new technique inspired by in-context learning dynamics rather than semantic similarity for selecting demonstrations.
64
measured semantic
nDCG@10
62
61
60
XL-0 XL-1 XL-4 XL-8 XXL-0 XXL-1 XXL-4 XXL-8
Figure 2: Statistics for nDCG@ 10 on TREC 2020, aggregated using 100 different k-shot demonstrations with Flan-T5-XL. and XXL models. Number of demon- strations (k) shown after the dash.
4.2 Difficulty-basets Selection (DBS)
0.65
FLAN-TS-XXL FLAN-T5-XL
We propose difficulty-basee selection to find chal- lenging demonstrations to include in the prompt. We estimate difficulty using demonstration query likelihood (DQL):
0.64
nDCG@10
0.63
DQL(2) & [a())] log (() d(=)),
0.62
then select the demonstrations with the lowest DQL. Intuitively, this should find hard samples that poten- tially correspond to large gradients had we directly trained the model instcad of prompting.
1% (Easy)
10%
10%
1% (Hard)
(Easy)
(Hard)
Figure 3: nDCG@ 10 DQL-based bins measured on TREC 2020. The x-axis increases in difficulty of demonstration from left-to-right.
4.2.1 Difficult Demonstrations are Beneficial On TREC 2020, we observe statistically signifi- cant (00) correlation (0.26) between negative DQL and ranking performance with the 100 ran- dom one-shot demonstrations from $4.1. The low- est DQL outperforms average nDCG@ 10 across the 100 random demonstrations (64.1 vs. 62.8).
Main Results and Discussion
5.1 DBS for TREC and BEIR
In Table 1, we compare DBS with zero-shot and manual demonstration selection. Manual curation is with demonstrations from Promptagator (Dai et al., 2023), which uses up to eight demonstra- tions depending the task? The results show that demonstrations often improve re-ranking on TREC and BEIR, and furthermore, that our DBS effective for automatic demonstration selection. The improvement over zero-shot can be substantial, such as the case for TREC 2020, FIQA, and NQ where using demonstrations leads to more than 3- points improvement in all settings. When zero-shot outperforms few-shot, it is only by small margin and often datasets that require complex retrieval such as FEVER HotpotQA. DBS outperforms
To further investigate how well DQL works for selecting demonstrations, we sample easy or hard demonstrations from the full MS Marco training data rather than only using 100. We form four bins by sampling 30 demon- strations from each of the bottom-19 and 10% by DQL (these are the hardest, and should give the best results), and the same with easy ones. We plot the mean and max nDCG@ 10 for each bin in Figure 3. For Flan-T5-5 TXLXL,, the performance im- proves as we use more challenging demonstrations. This trend is less prominent for Flan-T5-XL.
initially selected
'Recent theories on the effectiveness of in-context learn- ing view few-shot prompting similarly to fine-tuning on the demonstrations in the prompt (Basu et al,, 2023).
For example, there are eight, six, and two demonstrations for TREC, FiQA, and Scifact.

---

T19
T20
FiQA
Scifact
BioASQ
FEVER
HotpoiQA
NQ
Quora
BM25
50.60
48.00
23.60
66.50
46.45
75.32
60.27
32.84
78.83
---XL
O-shot
61.10
59.90
38.20
70.40
54.34
68.02
72.79
40.26
77.09
Promptagator
61.00
61.40
43.40
71.90
54.57
69,40
72.36
44.70
84.53
DBS 1-shot
59.80
62.50
44.10
72.00
54.81
69,42
72.69
44.07
83.66
DBS 4-shot
61.00
63.00
44.70
72.70
54.32
70.46
72.53
44.58
84.32
Flan-TS-XXL
0-shot
61.80
60.30
42.90
73.00
55.11
78.17
72.56
44.93
83.70
Promptagator
61.90
63.30
47.40
73.80
55.32
78.00
73.53
47.90
85.56
DBS 1-shot
62.66
63.99
47.60
74.30
55.41
77.62
74.11
48.46
85.31
DBS 4-shot
63.38
62.93
47.70
74.50
55.71
77.68
73.78
48.41
85.73
Palm 2-8
0-shot
55.84
55.55
38.26
74.69
52.31
76.94
71.98
43.33
83.51
Promptagator
61.24
60.92
48.11
76.89
55.69
78.18
75.43
44.71
85.89
DBS 1-shot
58.50
60.62
46.99
74.87
53.43
78.07
72.93
42.91
85.52
DBS 4-shot
61.39
61.20
47.96
76.52
54.34
78.59
75.36
49.84
85.81
Table I: nDCG@ 10 for TREC 2019/2020 and seven BEIR datasets after re-ranking top-100 documents re- trieved by BM25, T5---X5X and Palm 2-S perform prompt- based re-ranking, where scro-shotti is identical to UPR (Sachan et al., 2022). We also use manually selected demonstrations from Promptagator (Dai et al., 2023).
Promptagator demonstrations in many settings, in- cluding by large margin for TREC 2019 with Flan-T5-XXL and NQ with Palm 2-S.
advanced selection. In the future it would be help- ful to see richer distribution of performance using substantially more than random demonstrations.
Demonstration filtering. When using DBS, we return the top-30 demonstrations and then perform lightweight manual filtering to remove demon- strations with incorrect labels.3 3 We also remove duplicate queries. In the future, filtering for incor- rect labels should be easy to automate, and it would be interesting to explore how DBS can be used to mine for incorrect annotations.
5.3 DBS with Conditional DQL (CDQL)
DQL overlooks that variations in demonstration difficulty depends on the other demonstrations the context. To jointly consider the difficulty all demonstrations in the prompt, we propose condi- tional demonstration query likelihood (CDQL):
CDQL(21, 22, 2K)
5.2 Comparison to Random Selection
Σ 1=
log (()
21 1, 1,)(=))
Our findings thus far show that zero-so zero-shot ranking is outperformed by demonstration-bes based ranking with DBS in almost every setting. To understand how much potential there is to further improve demonstration selection, we compare DBS using Flan-T5-XXI against 10 randomly selected one- shot demonstrations for TREC and BEIR (see Ap- pendix A.4 for full results). We that the max nDCG from random selection outperforms DBS in five out nine datasets. This suggests LLM- based ranking may be further improved through
In preliminary results, we find that Flan-T5-X: XX-XXL with CDQL improves over DQL on TREC 2019 and 2020, respectively giving 63.5 vs. 63.4 and 64.4 vs. 64.0 nDCG.* To CDQL chose 30 demonstrations first by DQL, filtered for any in- correct labels, then computed CDQL for each per- mutation including four demonstrations and took the lowest CDQL. We leave further exploration of CDQL to future work, and believe it may be bene- ficial when selecting more than one demonstration.
"Saatistics for filtering are in Appendix A.2.
Ressults for CDQL are in Table 3, in the Appendix.

---

Average of Max Query Similarity
unclear, motivating us to conduct our own study. Our difficulty-based demonstration selection ($4.2) closely related active learning (Dagan and Argamon, 1995; Roy and McCallum, 2001; Scttles, 2009). Similarly, Diao et al. (2023) mea- sure uncertainty with generation instead of scoring. Zhang et al. (2022) formulate demonstration selec- tion as reinforcement learning problem. Rubin et al. (2022) use LLM-scoring to find hard nega- tives for their trained demonstration retriever. Oth- ers explore demonstration ordering (Lu et al., 2022) and joint selection (Drozdov et al.. 2023; Levy et al., 2023; A Agrawal et al., 2023; Ye et al., 2023). Concurrent to our work, Li and Qiu (2023) perform multiple rounds of hill climbing to find groups of demonstrations that perform well according to a validation set. In contrast, DBS selects demonstra- tions directly and does not rely on validation.
DBS
0.81
0.80
0.79
20
40
60
80
100
Demonstration ID
Figure 4: For each demonstration, we compute the semantic similarity between ground-truth and the syn- thetic queries. We first measure the max similarity by demonstration and query. Then we average this across all queries, giving a single scalar per demonstration. The dashed line shows the "average max similarity" for the demonstration chosen using DBS.
Discriminative methods are widely used in super- vised ranking (Zhuang et al., 2023; Nogueira dos Santos et al., 2020; Hui al., 2022). Listwise prompting is an alternative query likelihood, but requires a sliding window strategy as not all docu- ments fit in the context (Ma et al., 2023; Sun et al.. 2023). Rather than query likelihood, HyDE (Gao et al., 2023) achieves zero-shot ranking through document generation, which we hypothesize would be improved through demonstrations. PaRaDe is bounded by the first stage BM25 retrieval, and it may be fruitful to explore approaches that align first stage retrieval with our demonstration-hasse approach (Yadav et al., 2022).
5.4 DBS for Question Generation
As an auxiliary evaluation of DBS we study ques- tion generation, which plays important roles in dif- ferent NLP applications (Dai et al., 2023; Bonifacio et al., 2022; Jeronymo et al., 2023; Nogueira et al., 2019; Ma et al., 2021). Using the top-100 100 passages retrieved from BM25 for each query in TREC 2020, we greedily generate with Flan-T5-XXL 100 ques- tions per passage using one-shot prompt and the 100 random demonstrations from $2. We compare the generated questions from random demonstra- tions and the ones from DBS (1-shot). For each query, we compute the maximum cosine similar- ity between the ground truth and generated ques- tions after embedding with Universal Sentence En- coder (Cer et al., 2018). The average of the max similarity among 100 random demonstrations is 0.8018 (min=0.7770 and max=0.81 15),, whereas, we achieve 0.8081 when using DBS (one-shot). Compared with the random demonstrations, the DBS result ranks 8% highest similarity in the pop- ulation and is significantly greater than the mean (p=4e-18) according to two-tailed t-test (Figure 4). These findings indicate that the demonstrations ef- fective for LLM-based scoring of passages sim- ilarly effective for generation of questions.
Conclusion
In this work we present Passage Ranking with Demonstrations (PaRaDe), extensive study on the topic of using demonstrations to improve ranking performance of LLMs. We show the chal- lenges of applying demonstrations effectively, and that performance heavily relies on selecting "good" demonstrations. We propose a simple yet effective selection method, named difficulty-based selection (DBS), and confirm its effectiveness in both re- ranking using query likelihood scoring and query generation tasks. For future work, plan combine difficulty-based selection with similarity- based selection an effort further improve the robustness and effectiveness of the selected demonstrations, and extend DBS to other ranking paradigms (Qin et al., 2023: Sun et al., 2023).
6 Related Work
Concurrent with UPR, PromptRank (Khalifa et al.. 2023) the most related prior work, using demon- strations to re-rank "document paths" for multihop-
QA. Details of how they select demonstrations is

---

Acknowledgements
dataset generation for information retrieval. In Pro- ceedings of the 45th International ACM SIGIR Con- ference Research and Development in Informa- tion Retrieval, SIGIR '22, page 2387-2392, New York, NY, USA. Association for Computing Machin- ery.
We thank Tal Schuster for their detailed comments on earlier versions of this manuscript. We are grate- ful to Vinh Tran for insightful discussions regard- ing the Flan family of models and their in in-context learning capabilities.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Me- Candlish, Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language models are few-shot learn- ets. arXiv preprint arXiv: 2005.14165.
Limitations
One limitation of DBS is when naively used to se- lect multiple demonstrations, the demonstrations that may appear challenging at first may become relatively easy once the other demonstrations have been processed in the prompt context. We par- tially address this by replacing DQL in DBS with CDQL ($5.3) so that demonstrations are selected jointly rather than scored individually. Our CDQL approach selects a high scoring subset from the ini- tial list provided by DQL. More challenging com- binations of demonstrations may be available by searching the entire candidate set, but exact search is computationally prohibitive. Another limitation is that only incorporate positive demonstrations, and for retrieval, training with hard negatives is often beneficial to model performance. We hypoth- esize a DBS-like algorithm can be used to find hard negatives, but it may be important to add signals distinguishing positive from negative demonstra- tion when using LLMs with query likelihood for ranking.
Daniel Cer. Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal Sentence Encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations. pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
Hlaw-Shiann Chang, Erik G. Learned- Miller, and An- drew McCallum. 2017. Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples. In Neural Information Process- ing Systems.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- cry, Alex Castro-Ros, Marie Pellat, Kevin Robin- son, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Sealing instruction- finetuned language models. arXiv preprint arXiv: 2210.11416.
References
Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. In- context examples selection for machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8857-8873, Toronto, Canada, Association for Computational Linguisties.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2023. What learn- ing algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations.
Ido Dagan and Shlomo Engelson Argamon. 1995. Committee--asse bassd sampling for training probabilistic classifiers. In International Conference on Machine Learning.
Zhuyun Dai, Vincent Y Zhao, Ji Ma. Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wes Chang. 2023. Promptagator: Few-shot dense retrieval from 8 examples. In The Eleventh International Conference on Learning Rep- resentations.
Soumya Basu, Ankit Singh Rawat, and Manzil Zaheer. 2023. A Statistical Perspective Retrieval-Baat Models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 1852-1886. PMLR.
Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models. arXiv preprint arXiv: 2302, 12246.
Luiz Bonifacio. Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. InPars: Unsupervised

---

Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou, 2023. Compositional semantic parsing with large language models. In The Eleventh International Conference on Learning Representations.
tics: ACL 2022. pages 3747-3758, Dublin, Ireland. Association for Computational Linguistics.
Vitor Jeronymo, Luiz Bonifacio, Hugo Abonizio, Marzich Fadace, Roberto Lotufo, Jakub Zavrel, and Rodrigo Nogueira. 2023. InPars-v2: Large Language Models as Efficientt Dataset Generators for Information Retrieval. arXiv preprint arXiv: 2301.01822
Luyu Gao, Xueguang Ma. Jimmy Lin, and Jamie Callan. 2023. Precise zero-shot dense retrieval with- out relevance labels. In Proceedings of the 61 An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1762- 1777, Toronto, Canada, Association for Computa- tional Linguistics.
Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023. Few-shot Reranking for Multi-hop QA via Language Model Prompting. arXiv preprint arXiv: 2205. /2650.
Itay Levy, Ben Bogin, and Jonathan Berant. 2023. Di- verse demonstrations improve in-context composi- tional generalization, In Proceedings of the 6 An- nual Meeting of the Association for Compulational Linguistics (Volume Long Papers), pages 1401- 1422, Toronto, Canada. Association for Computa- tional Linguistics.
Rohan Anil Google, Andrew M. Dai, Orhan Fi- rat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sehastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Ja- cob Austin, Paul Barham, Jan Botha, James Brad- bury, Siddhartha Brahma. Kevin Brooks, Michele Catasta, Yong Cheng. Colin Cherry, Christopher A. Chooo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz. Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng. Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann. Lu- cas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jef- frey Hui, Jeremy Hurwitz, Michael Isard, Abe Itty- cheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li. YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Mar- cello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem. Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polo- zov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So. Daniel Sohn, Simon Tokumine, Dasha Valter, Vi- jay Vasudevan. Kiran Vodrahalli, Xuezhi Wang, Pi- dong Wang, Zirui Wang, Tao Wang, John Wiet- ing, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengeheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. Palm 2 technical re- port.
Xiaonan Li and Xipeng Qiu. 2023. Finding support
examples for in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP)..
Jimmy Lin, Xueguang Ma, Sheng-Chiel Lin, Jheng- Hong Yang. Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations. In Proceedings of the 44th Annuul International ACM SIGIR Conference on Re- search and Development in Information Retrieval (SIGIR 2021). pages 2362.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically Ordered Prompts and Where to Find Them: Over- coming Few-Shot Prompt Order Sensitivity. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 8086-800 1886-809,,08 Dublin, Ireland. Associ- ation for Computational Linguistics.
Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mchran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Zhao. 2023. Dr.IC.; Demonstration- Retrieved In-context Learning. arXiv preprint arXiv: 2305. 14128.
Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2021. Zero-shot neural passage re- trieval via domain- targeted synthetic question gener- ation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pages 1075-1088, Online. Association for Computational Linguistics.
Kai Hui. Honglei Zhuang, Tao Chen, Zhen Qin. Jing Lu, Dara Bahri, Ji Ma. Jai Gupta, Cicero Nogueira dos Santos, Yi Tay, and Donald Metzler. 2022. ED2LM: Encoder-decoder to language model for faster document re-ranking inference. In Find- ings of the Association for Computatinaal Linguis-
Xueguang Ma, Xinyu Crystina Zhang, Ronak Pradeep. and Jimmy Lin. 2023. Rank Vicuna: Zero-Shot List- wise Document Reranking with a Large Language Model, arXiv preprint arXiv: 2305 0255021566

---

Sewon Min, Mike Lewis, Luke Zettlemoyer, and Han- nanch Hajishirzi. 2022. MetalCL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2791 1191-2809, Seattle, United States. Association for Computational Linguistics.
Weiwei Sun, Lingyong Yan, Xinyu Ma. Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is Chat- GPT Good at Search? Investigating Large Language Models as Re-Ranking Agent, arXiv preprint arXiv: 2304.09542.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Tryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evalua- tion of information retrieval models. In Thirty-fitth Conference Neural Information Processing Sys- tems Datasets and Benchmarks Track (Round 2).
Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. 2019. From doc2query doeTTTTTquery. Online preprint.
Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nal- lapati, Zhiheng Huang, and Bing Xiang. 2020. Be- yond [CLS] through ranking by generation. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 1722 1727, Online. Association for Computa- tional Linguisties.
Nishant Yadav, Nicholas Monath, Rico Angell, Manzil Zaheer, and Andrew McCallum. 2022. Efficientt nearest neighbor search for cross-encoder models using matrix factorization. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 2171-2194, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Jay M. Ponte and W. Bruce Croft. 1998. Language Modeling Approach to Information Retrieval. In Proceedings of the 21st Annual International ACM
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In International Conference on Machine Learning.
SIGIR Conference on Research and Development in Information Retrieval, SIGIR '98, page 275-281, New York, NY, USA. Association for Computing Machinery.
Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Ac- tive example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134-9148. Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Zhen Qin, Rolf Jagerman, Kai Hui. Honglei Zhuang. Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Don- ald Metzler, Xuanhui Wang, et al. 2023. Large lan- guage models are effective text rankers with pair- wise ranking prompting. arXiv preprint arXiv: 2306.17563.
Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma. Jing Lu, Jianmo Ni, Xuanhui Wang, and Mike Bendersky. 2023. RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses. In Proc. of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR).
Nicholas Roy and Andrew McCallum. 2001. Toward optimal active learning through monte carlo estima- tion of error reduction. In International Conference on Machine Learning.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2655-267. Seattle, United States. Association for Computational Linguisties.
Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zeulemoyer. 2022. Improving passage retrieval with zero-shot question generation. In Proceed- ings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 3781-3797, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Burr Settles. 2009. Active learning literature survey.
Abhinav Shrivastava, Abhinav Kumar Gupta, and Ross B. Girshick. 2016. Training region-based object detectors with online hard example mining. 2016 IEEE Conference on Computer Vision and Put- tern Recognition (CVPR), pages 761-769.

---

A Appendix
Dataset
Ranks of Selected
No. Skipped
A.1 Prompt Format
MSMarco
1,2,3,4
0
The instructions we use are in Table 4. We also include short prefix indicating the start of docu- ment query, shown in Table 5. These used once for the test query and document, and dupli- cated for any demonstrations in the prompt. For few-shot the prompt includes an instruction, one or more demonstrations, and the test data. Scores for ranking are computed only on the query. For zero-shot, no demonstrations are included.
FiQA
4,5,6,7
3
Scifact
5,6,7,8
3
BioASQ
1,3,5,18
5
FEVER
5,8,16,22
17
HotpotQA
2,3,6,8
NQ
3,4,5,8
Quora
1,2,4,5
Table 2: Statisties for demonstration filtering.
A.2 Statistics for Demonstration Filtering
Table 2 shows statistics for demonstration filtering, including the ranks of selected demonstrations and the number of demonstrations that skipped per dataset. Demonstrations are skipped if they are incorrectly labeled or duplicates of already selected demonstrations. The demonstrations are selected from the training data, and MSMarco is used for TREC 2019 and 2020. In the case of FEVER, we needed to truncate long demonstrations so they would fit in the prompt. Sometimes this would inadvertently make labels incorrect necessary information resided in the removed text. We always perform any preprocessing and truncation before running demonstration selection.
T19
T20
BM25
50.6
48.0
Flan-T5-XXL
O-shot (UPR)
61.8
60.3
Promptagator
61.9
63.3
DBS I-shot (DQL)
62.7
64.0
DBS 4-shot (DQL)
63.4
62.9
DBS 4-shot (CDQL)
63.5
64.4
Table nDCG@ 10 on TREC 2019 and 2020 when using Flan-T5-XXL. We compare zero demonstrations, manual curation (Promptagator). and automatic selec- tion with DBS using DQL or CDQL..
A.3 Results with CDQL
Table 3 includes results using conditional demon- stration query likelihood (CDQL). These are only preliminary results, and we believe that CDQL should be an effective alternative to DQL when selecting more than one demonstration.
A.4 Results with Random Selection
In Table we compare DBS one-shot with random one-shot demonstrations when using Flan-T5-XXI. Random performance is aggregated across 10 ran- domly selected demonstrations, and we show the mean, standard deviation, minimum, and maximum values. The results indicate there is further oppor- tunity to improve demonstration selection, and also that demonstration selection is a challenging task.

---

Dataset
Instruction
TREC 2019
[web] will check whether what you said could answer my question.
TREC 2020
[web] will check whether what you said could answer my question.
BEIR FIQA
[web] will check if what you said could verify my question.
BEIR Scifact
[web] will check if the argument you said could verify my scientific claim.
Table The instructions for zero-shot and few-shot prompts. Zero-shot prompts include only the instruction and test document, with scoring on the test query. Few-shot prompts also include demonstrations. The same prompts are used for both query likelihood and question generation, although question generation excludes the test query.
Dataset
Query-Document Template
TREC 2019
You said:
DOCUMENT <newline> I googled: QUERY
TREC 2020
You said:
DOCUMENT <newline> googled: QUERY
BEIR FiQA
You said: DOCUMENT <newline> I googled: QUERY
BEIR Scifact
Argument: DOCUMENT <newline> My scientific claim: QUERY
Table 5: The prompt template for zero-shot and few-shot prompts. Zero-shot prompts include only the instruction and test document, with scoring on the test query. Few-shot prompts also include demonstrations. The same prompts are used for both query likelihood and question generation, although question generation excludes the test query.

---

T19
T20
FiQA
Scifact
BioASQ
Fever
HotpotQA
NQ
Quora
BM25
50.58
47.96
23.61
66.47
46.45
75.32
60.27
32.84
78.83
O-shot
61.80
60.30
42.90
73.00
55.11
78.17
72.56
44.93
83.70
Promptagator
61.90
63.30
47.40
73.80
55.32
78.00
73.53
47.90
85.56
DBS 1-shot
62.66
63.99
47.60
74.30
55.41
77.62
74.11
48.46
85.31
DBS 4-shot
63.38
62.93
47.70
74.50
55.71
77.68
73.78
48.41
85.73
DBS 1-shot
62.66
63.99
47.60
74.30
55.41
77.62
74.11
48.46
85.31
R 1-shot (avg)
62.02
62.84
48.58
75.58
55.54
80.41
71.98
49.28
84.71
I-shot (std)
0.47
0.44
0.27
0.37
0.23
1.39
0.24
0.38
0.35
R I-shot (min)
60.92
62.19
48.05
74.97
55.34
77.64
71.56
48.61
83.78
R 1-shot (max)
62.80
63.50
48.93
76.21
56.03
81.93
72.31
49.71
85.05
Table 6: nDCG@ on BEIR datasets, using queries when using Flan-T5-XXI. We compare DBS 1-shot with random (R) 1-shot. Random performance is aggregated across 10 randomly selected demonstrations, and we show the mean, standard deviation, minimum, and maximum values. DBS 1-shot is underlined it is greater the maximum random J-shot and vice versa. When maximum random I-shot outperforms DBS 1-shot, this suggests there is further opportunity improve demonstration selection, and also that demonstration selection is challenging task.


Reviews:
[{"review_id": "OTmeCa9zM9", "reviewer": "Reviewer_auEx", "summary": "", "questions": "Question A: Low likelihood can also mean that the query and demonstration are distantly or even not related. What kind of preprocessing is done to insure that the query-document pair with the lowest likelihood are still related?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. We're glad to hear that our work is considered a simple and effective approach (E2cL,auEx) that improves a previous method (auEx), is helpful for future research (E2cL,auEx) and well motivated (auEx), and focuses on an interesting research question that pushes the boundaries of LLM capabilities (99xq). We respond to the questions from the review below.\n\n**Reason to Reject 1. What is the variance of DBS?**\n\nThank you for pointing this out! DBS is deterministic given the pool of demonstrations, so there is no variance for the selection of demonstration (or variance is 0); BM25 and 0-shot are also deterministic. Rather, we look at the relative benefit of DBS compared to random demonstration selection. We show that DBS outperforms or is on-par with the average of random selection. Importantly, DBS avoids the low performance observed with some random demonstrations.\n\nFor random selection we use 30x 1-shot random selected demonstrations from the training data, then evaluated on a 1000 query subset of the test data and reported the mean, standard deviation, min, and max performance across random 1-shot demonstrations. Due to the expensiveness of the random experiments, so far we have only used FLAN-T5-XXL and 4 of the datasets. We will include the full data in our next revision. There are already extensive experiments using random demonstrations and TRECDL-2020 in our initial submission.\n\n|  | FEVER | HotpotQA | NQ | Quora |\n|---------|-------|----------|----|-------|\n| No. of Queries | 1000 | 1000 | 1000 | 1000 |\n| BM25 | 75.3 | 60.3 | 32.8 | 78.8 |\n| UPR 0-shot | 80.3 | 70.7 | 47.0 | 82.5 |\n| DBS 1-shot (Ours) | **81.7** | **72.7** | 48.4 | **86.6** |\n| Random 1-shot Avg. / Std. | 80.2 / 1.49 | 71.7 / 0.25 | **48.7** / 0.6 | 84.8 / 0.37 |\n| Random 1-shot Min. | 77.3 | 71.2 | 47.2 | 83.9 |\n| Random 1-shot Max. | 83.2 | 72.1 | 49.6 | 85.5 |\n\n**Reason to Reject 2. What is the motivation for using FLAN-T5-XL/XXL?**\n\nOur motivation for using FLAN-based models is that they are open source (and hence reproducible) models that are known to have in-context learning capabilities [1]. Since its release, it has been verified to have strong few-shot prompting performance [2]. We use FLAN-T5 models as a generalization of LLM in this work. To further confirm the effectiveness of DBS, we include results from Palm in the following. We also include results on five more datasets that have annotated training data available (BioASQ, FEVER, HotpotQA, NQ, Quora). \n\n[1] https://arxiv.org/abs/2210.11416\n\n[2] https://arxiv.org/abs/2210.02406\n\n|   | T19 | T20 | FiQA | Scifact | BioASQ | FEVER | HotpotQA | NQ | Quora |\n|---------|-----|-----|------|---------|--------|-------|----------|----|-------|\n| BM25 | 50.60 | 48.00 | 23.60 | 66.50 | 46.45 | 75.32 | 60.27 | 32.84 | 78.83 |\n|---------|-----|-----|------|---------|--------|-------|----------|----|-------|\n| Flan-T5-XL 0-shot | **61.10** | 59.90 | 38.20 | 70.40 | 54.34 | 68.02 | **72.79** | 40.26 | 77.09 |\n| Flan-T5-XL Promptagator k-shot | 61.00 | 61.40 | 43.40 | 71.90 | 54.57 | 69.40 | 72.36 | **44.70** | **84.53** |\n| Flan-T5-XL DBS 1-shot | 59.80 | 62.50 | 44.10 | 72.00 | **54.81** | 69.42 | 72.69 | 44.07 | 83.66 |\n| Flan-T5-XL DBS 4-shot | 61.00 | **63.00** | **44.70** | **72.70** | 54.32 | **70.46** | 72.53 | 44.58 | 84.32 |\n|---------|-----|-----|------|---------|--------|-------|----------|----|-------|\n| Flan-T5-XXL 0-shot | 61.80 | 60.30 | 42.90 | 73.00 | 55.11 | **78.17** | 72.56 | 44.93 | 83.70 |\n| Flan-T5-XXL Promptagator k-shot | 61.90 | **63.30** | 47.40 | 73.80 | 55.32 | 78.00 | 73.53 | 47.90 | 85.56 |\n| Flan-T5-XXL DBS 1-shot | **62.40** | 62.80 | 47.60 | 74.30 | 55.41 | 77.62 | **74.11** | **48.46** | 85.31 |\n| Flan-T5-XXL DBS 4-shot | 62.30 | 62.30 | **47.70** | **74.50** | **55.71** | 77.68 | 73.78 | 48.41 | **85.73** |\n|---------|-----|-----|------|---------|--------|-------|----------|----|-------|\n| Palm 0-shot | 55.84 | 55.55 | 38.26 | 74.69 | 52.31 | 76.94 | 71.98 | 43.33 | 83.51 |\n| Palm Promptagator k-shot | 61.24 | 60.92 | **48.11** | **76.89** | **55.69** | 78.18 | **75.43** | 44.71 | **85.89** |\n| Palm DBS 1-shot | 58.50 | 60.62 | 46.99 | 74.87 | 53.43 | 78.07 | 72.93 | 42.91 | 85.52 |\n| Palm DBS 4-shot | **62.68** | **63.43** | 47.96 | 76.52 | 54.34 | **78.59** | 75.36 | **49.84** | 85.81 |\n\n\n**Question A: Low likelihood can also mean that the query and demonstration are distantly or even not related. What kind of preprocessing is done to ensure that the query-document pair with the lowest likelihood are still related?**\n\nWe perform manual filtering on the top demonstrations (chosen by DBS) to remove the (query, passage) pairs that are wrongly labeled in the training data, according to the respective definitions for individual dataset. This is also mentioned in Footnote 5: “We remove the pairs that are labeled incorrectly. The demonstrations used are included in Appendix A.3.1.” Note that this manual filtering serves to correct the annotation errors from the training dataset and requires minimal manual effort.\n\nIn particular, on the training dataset, we exploit FLAN-T5-XXL to score each positive (query, passage) pair using QL and sort them in ascending order. We go through the list and skip the (q, d) pairs that are with wrong labels. We stop after four demonstrations are collected. We follow the annotation guideline from the respective dataset. Here are the ranks of the selected demonstrations from the ascending list and the number of (q, d) pairs skipped for each dataset after removing the duplicates query-passage pairs. For FEVER, we only consider pairs that are with “support” annotation as demonstrations, whereas BEIR converts both “refute” and “support” as positive samples. This makes the skipped (q, d) pairs on FEVER relatively high, which is still below 20.\n\n|   | Ranks of the selected demos | #(q,d) skipped |\n| ---|------------|------------------ |\n| MSMarco | 1,2,3,4 | 0 |\n| FiQA | 4,5,6,7 | 3 |\n| Scifact | 5,6,7,8 | 3 |\n| BioASQ | 1,3,5,18 | 5 |\n| FEVER | 5,8,16,22 | 17 |\n| HotpotQA | 2,3,6,8 | 4 |\n| NQ | 3,4,5,8 | 4 |\n| Quora | 1,2,4,5 | 1 |\n\n**Typo and grammar style**\n\nThank you for the suggestion! We will clean up the descriptions and notations of DBS in the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Investigates how few-shot paradigm would work in a retrieval setting that also works to improve a previous method.\n2. The work provides a simple and accessible method that avoids finetuning and is likely extensible to a variety of retrieval settings.\n3. Shows that there is high variance when using few-shot demonstrations selected at random, this serves as a motivation for the proposed Difficulty-based Selection strategy.", "reasons_to_reject": "1. While the author shows that (1) there is high variance when using randomly selected demonstrations and that (2) using demonstrations selected through the proposed strategy results in a score improvement, the variance when using Difficulty-based Selection strategy is not shown. Both compared methods still show improvement over zero-shot and the motivating reason to utilize Difficulty-based Selection strategy over random selection would then be that it is more consitant.\n2. It is never stated why FLAN-T5-XL/XXL was used in this work. When referring to the query likelihood retrieval model, it does seem that decoder-only models could work just as well. Which leaves out out whether this approach only works for a specific model architecture or not.", "questions_to_authors": "Question A: Low likelihood can also mean that the query and demonstration are distantly or even not related. What kind of preprocessing is done to insure that the query-document pair with the lowest likelihood are still related?", "paper_topic_and_main_contributions": "This work builds upon a previous retrieval reranking method that uses LLMs with zero-shot (no examples to help the LLM). It demonstrates how adding an example (1-shot approach) can improve re-ranking performance. The proposed method, PaRaDe, utilizes few-shot examples that are included in the prompt during the reranking process. To effectively add more examples, the work also proposes that the selection of such examples be done through a \"difficulty-based demonstration\" selection strategy. This strategy selects challenging demonstrations which is defined as selecting the query-document pair with the lowest likelihood from training samples as the few-shot examples.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Description of the LLM Re-ranking by Query Likelihood could be more elaborate to show that the query q is a sequence of q_i:q_N (or a sequence of N tokens) and that for a collection of documents D, each document d \\in D to clarify that d is in fact a subset of D. The proposed Difficulty-based Selection that uses DQL could also be elaborated with a variation of the provided equations to make it clearer what it is for the readers."}}, {"review_id": "HPEBsJpS8V", "reviewer": "Reviewer_99xq", "summary": "", "questions": "1. I do not think that it is a good idea for a short paper to claim that \"To the best of our knowledge, we are the most extensive study to date investigating the use of demonstrations for LLM-based re-ranking.\"\n2. If LLMs can generate answers directly, what is the point for them to rank passages?\n3. Do you fine-tune LLMs?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. We're glad to hear that our work is considered a simple and effective approach (E2cL,auEx) that improves a previous method (auEx), is helpful for future research (E2cL,auEx) and well motivated (auEx), and focuses on an interesting research question that pushes the boundaries of LLM capabilities (99xq). We respond to the questions from the review below.\n\n**Reason to Reject 1. Missing Related Work**\n\nThank you for bringing these relevant papers to our attention. Although we did an extensive literature review, there was indeed a recent paper from arxiv that we failed to mention.\n\nWe already cover Sun et al. [1] in our related work section. Demonstration selection is not a main focus of that work. Although they use a fixed set of demonstrations for relevance generation, they provide no criteria on how those demonstrations are selected. We will add additional clarification in our text.\n\nWe already discussed Zhang et al. [2] in the related work section. This is an interesting paper, although it is for a different task, namely, sentence classification instead of passage re-ranking. It is inspired by active learning, but the method is quite different from ours and uses reinforcement learning. We will add additional clarification in our text.\n\nWe are not aware of Li and Qiu [3] because it is on sentence classification instead of passage re-ranking, and it was publicly available on arxiv very recently (Feb 2023). Thank you for pointing it out and  we will add it in camera ready, and will also discuss the differences between demonstration selection for sentence classification vs. passage re-ranking.\n\nBoth query and passage are important when selecting demonstrations for ranking tasks, and it involves two dimensions of complex text, making it different from the demonstration selection in sentence classification. It’s also worth noting that the uncertainty-inspired baselines in Li and Qiu are not particularly effective and often underperform random demonstrations and zero-shot sentence classification. DBS on the other hand consistently outperforms random demonstrations and zero-shot for passage re-ranking.\n\n\n[1] Sun et al., Apr 2023. https://arxiv.org/abs/2304.09542\n\n[2] Zhang et al., Nov 2022. https://arxiv.org/abs/2211.04486\n\n[3] Li and Qiu, Feb 2023. https://arxiv.org/abs/2302.13539\n\n\n**Reason to Reject 2. Lack of in-depth analysis**\n\nThis is a very good question, and we try to answer it as follows.\n\nFrom a theoretical view, multiple recent studies suggest that in-context learning works similarly to gradient descent [1, 2], wherein the difficult training samples contribute more in terms of loss value. Empirically, we observe a positive correlation between the difficulty of demonstrations and the ranking performance when using such demonstrations in Figure 2. Thus, we believe such positive correlations are the reasons behind the success of DBS. We also report additional results on more dataset and using more models, and confirm the effectiveness of DBS (rebuttal to auEx).\n\n[1] https://arxiv.org/abs/2211.15661\n\n[2] https://arxiv.org/abs/2212.07677\n\n\n**Reason to Reject 3. Contributions and claim**\n\nThank you for the suggestions. After investigating the literature, we tried to state the fact that this is the only work to date investigating the general methods to select demonstrations for LLM-based passage re-ranking. The most related works are [1] and [2]. The former investigates the LLM for re-ranking but with 0-shot configuration whereas the latter is to re-rank the document path for multihop QA. More recent papers like [3], [4], and [5] focus on the scoring method and are both under 0-shot setting, making it orthogonal to our method. We will rephrase the statement to make it more precise and clearer.\n\n[1] Sachan et al., Apr 2022. https://arxiv.org/abs/2204.07496\n\n[2] Khalifa et al., May 2022. https://arxiv.org/abs/2205.12650\n\n[3] Sun et al., Apr 2023. https://arxiv.org/abs/2304.09542\n\n[4] Ma et al., May 2023. https://arxiv.org/abs/2305.02156\n\n[5] Qin et al., June 2023. https://arxiv.org/abs/2306.17563\n\n\n**Q2. If LLMs can generate answers directly, what is the point for them to rank passages?**\n\nWe agree that the LLM could generate answers. However, we argue that the ranking is still an important component in the QA system. For example, ranking could be used in the retrieval augmentation and in post-processing the generation results by sorting the generation candidates (e.g., remove aggressive generations). Beyond QA, ranking tasks are also widely used, for which Table 1 in the BEIR paper [1] includes several examples. Previous work [2] has shown that LLMs are effective re-rankers, and our work further investigates the use of demonstrations to further boost the ranking performance from LLM.\n\n[1] Thakur et al. https://arxiv.org/abs/2104.08663\n\n[2] Sachan et al., Apr 2022 https://arxiv.org/abs/2204.07496\n\n\n**Q3. Do you fine-tune LLMs?**\n\nNo. We use the frozen LLM and investigate the demonstration selections.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work focuses on an interesting question, which studies the capacity boundaries of LLMs in terms of passage reranking.", "reasons_to_reject": "1. Many related works are missing and are not compared. For instance, works using LLMs for passage ranking, e.g., \" Is chatgpt good at search? investigating large language models as re-ranking agent.\" and works proposing example selection for in-context learning, e.g., \"Finding supporting examples for in-context learning.\"\n2. There lacks in-depth analyses to prove why the proposed DBS works.\n3. The contribution is not properly claimed.", "questions_to_authors": "1. I do not think that it is a good idea for a short paper to claim that \"To the best of our knowledge, we are the most extensive study to date investigating the use of demonstrations for LLM-based re-ranking.\"\n2. If LLMs can generate answers directly, what is the point for them to rank passages?\n3. Do you fine-tune LLMs?", "paper_topic_and_main_contributions": "This work investigates the use of LLMs for passage reranking. It proposes a difficulty-based demonstration selection strategy which is proven effective through experiments.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Finding Supporting Examples for In-Context Learning\n2. Active Example Selection for In-Context Learning"}, "extra_text": {}}, {"review_id": "CNzGITIwLK", "reviewer": "Reviewer_E2cL", "summary": "", "questions": "What the formal definition of DQL, how is it calculated?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. We're glad to hear that our work is considered a simple and effective approach (E2cL,auEx) that improves a previous method (auEx), is helpful for future research (E2cL,auEx) and well motivated (auEx), and focuses on an interesting research question that pushes the boundaries of LLM capabilities (99xq). We respond to the questions from the review below.\n\n\n**Reason to Reject 1. Formal definition of DQL.**\n\nThank you for mentioning this. Demonstration query likelihood is similar to query likelihood although applied to demonstrations instead of test queries. The only difference is that we divide by the number of tokens, since queries across demonstrations may have different length. For a (query, passage) pair from the training dataset, the formulation is as follows: \n\n$DQL(q \\mid d) \\propto \\frac{1}{N} \\sum_{i=1..N} \\log P (q_i | d, q_{0:i-1})$\n\n\n**Reason to Reject 2. The effectiveness is verified in limited LLM and datasets, which may not be convincing.**\n\nWe have run experiments on an additional six datasets and also include more recent LLM, namely, Palm for comparisons. It can be seen that the conclusions still hold on these additional dataset and when using a more recent LLM. DBS 4-shot provides a similar boost as with Promptagator k-shot. Note that up to eight demonstrations are used in Promptagator kshots. \n\n|   | T19 | T20 | FiQA | Scifact | BioASQ | FEVER | HotpotQA | NQ | Quora |\n|---------|-----|-----|------|---------|--------|-------|----------|----|-------|\n| BM25 | 50.60 | 48.00 | 23.60 | 66.50 | 46.45 | 75.32 | 60.27 | 32.84 | 78.83 |\n|---------|-----|-----|------|---------|--------|-------|----------|----|-------|\n| Flan-T5-XL 0-shot | **61.10** | 59.90 | 38.20 | 70.40 | 54.34 | 68.02 | **72.79** | 40.26 | 77.09 |\n| Flan-T5-XL Promptagator k-shot | 61.00 | 61.40 | 43.40 | 71.90 | 54.57 | 69.40 | 72.36 | **44.70** | **84.53** |\n| Flan-T5-XL DBS 1-shot | 59.80 | 62.50 | 44.10 | 72.00 | **54.81** | 69.42 | 72.69 | 44.07 | 83.66 |\n| Flan-T5-XL DBS 4-shot | 61.00 | **63.00** | **44.70** | **72.70** | 54.32 | **70.46** | 72.53 | 44.58 | 84.32 |\n|---------|-----|-----|------|---------|--------|-------|----------|----|-------|\n| Flan-T5-XXL 0-shot | 61.80 | 60.30 | 42.90 | 73.00 | 55.11 | **78.17** | 72.56 | 44.93 | 83.70 |\n| Flan-T5-XXL Promptagator k-shot | 61.90 | **63.30** | 47.40 | 73.80 | 55.32 | 78.00 | 73.53 | 47.90 | 85.56 |\n| Flan-T5-XXL DBS 1-shot | **62.40** | 62.80 | 47.60 | 74.30 | 55.41 | 77.62 | **74.11** | **48.46** | 85.31 |\n| Flan-T5-XXL DBS 4-shot | 62.30 | 62.30 | **47.70** | **74.50** | **55.71** | 77.68 | 73.78 | 48.41 | **85.73** |\n|---------|-----|-----|------|---------|--------|-------|----------|----|-------|\n| Palm 0-shot | 55.84 | 55.55 | 38.26 | 74.69 | 52.31 | 76.94 | 71.98 | 43.33 | 83.51 |\n| Palm Promptagator k-shot | 61.24 | 60.92 | **48.11** | **76.89** | **55.69** | 78.18 | **75.43** | 44.71 | **85.89** |\n| Palm DBS 1-shot | 58.50 | 60.62 | 46.99 | 74.87 | 53.43 | 78.07 | 72.93 | 42.91 | 85.52 |\n| Palm DBS 4-shot | **62.68** | **63.43** | 47.96 | 76.52 | 54.34 | **78.59** | 75.36 | **49.84** | 85.81 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed technique is simple and effective.\n2. The experiments on the demonstration in Section 3.1 is helpful for future research.", "reasons_to_reject": "1. No formal definition of DQL is given in the passage. Though I have a rough understanding of DQL, I cannot understand the detailed calculation process. A detailed formula should be elaborated. \n2. The effectiveness is verified in limited LLM and datasets, which may not be convincing.", "questions_to_authors": "What the formal definition of DQL, how is it calculated?", "paper_topic_and_main_contributions": "This paper proposes a new LLM for passage ranking method called DQL, which is proven useful through a series of experiments.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes and approach to improve LLM-based re-ranking performance by including even a single example in the prompt. This improvement makes a significant impact under specific conditions that a detailed analysis investigates. 

The paper is sound and well written. The authors were very effective, in my view, in motivating and explaining the value of the approach proposed (see details in the reviews and rebuttals). I think the paper will make an interesting contribution to the conference.
Decision: Accept-Findings