Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking
Authors: Shengyao Zhuang
Bing Liu
Bevan Koopman
Guido Zuccon
Venue: EMNLP 2023 Findings (2023)
Keywords: large language model
query likelihood model
zero-shot ranking model

Abstract: In the field of information retrieval, Query Likelihood Models (QLMs) rank documents based on the probability of generating the query given the content of a document.
Recently, advanced large language models (LLMs) have emerged as effective QLMs, showcasing promising ranking capabilities. This paper focuses on investigating the genuine zero-shot ranking effectiveness of recent LLMs, which are solely pre-trained on unstructured text data without supervised instruction fine-tuning. Our findings reveal the robust zero-shot ranking ability of such LLMs, highlighting that additional instruction fine-tuning may hinder effectiveness unless a question generation task is present in the fine-tuning dataset. Furthermore, we introduce a novel state-of-the-art ranking system that integrates LLM-based QLMs with a hybrid zero-shot retriever, demonstrating exceptional effectiveness in both zero-shot and few-shot scenarios. 
We make our codebase publicly available at https://github.com/ielab/llm-qlm.

Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking
Shengyao Zhuang', Bing Liu', Bevan Koopman 1.2, Guido Zuccon !CSIRO, TTe University of Queensland 1 (/shengyao.zhuangg,
zhuang, bing. 1iu, bevan. koopman)@cciro..au
2g. zucconêuq. edu.. au
Abstract
QA datasets such as MS MARCO (Nguyen et al., 2017). Alternatively, generative large language models (LLMs) like GPT3 (Brown et al., 2020) have been used to synthesize domain-specific train- ing queries, which are then used to train these rankers (Bonifacio et al., 2022; Dai et al., 2023). Despite their effectiveness, all of these methods consume significant expenses in training PLM- based ranker.
In the field of information retrieval, Query Like- lihood Models (QLMs) rank documents based the probability of generating the query given the content of a document. Recently, advanced large language models (LLMs) have emerged effective QLMs, showcasing promising rank- ing capabilities. This paper focuses on inves- tigating the genuine zero-shot ranking effec- tiveness of recent LLMs, which are solely pre- trained on unstructured text data without super- vised instruction fine-tuning. Our findings re- veal the robust zero-shot ranking ability of such LLMs, highlighting that additional instruction may hinder effectiveness unless question generation task is present in the fine- tuning dataset. Furthermore, we introduce novel ofate-of-the-ar- ranking system that inte- grates LLM--based QLMs with a hybrid zero- shot retriever, demonstrating exceptional effec- tiveness in both zero-shot and few-shot scenar- ios. We make our codebase publicly available ht /gi thub. homm/elab//lll hts/mmmmqqmm qlm.
In this paper, consider third avenue to ad- dress this challenge: leveraging LLMs function as Query Likelihood Models (QLMs) (Ponte and Croft, 1998; Hiemstra, 2000; Zhai and Lafferty, 2001). Essentially, QLMs are expected to under- stand the semantics of documents and queries, and estimate the possibility that each document can an- swer certain query. Notably, recent advances in this direction have greatly enhanced the ranking effectiveness of QLM-based rankers by leveraging PLMs like BERT (Devlin al., 2019) and T5 (Raf- fel et al., 2020). These PLM-based QLMs are fine- tuned on query generation tasks and subsequently employed to rank documents as per their likeli- hood (Nogueira dos Santos et al., 2020; Zhuang et al., 2021; Lesota et al., 2021; Zhuang and Zuc- con, 2021c).
1 Introduction
Ranking models (or rankers) are a fundamen- tal component in many information retrieval (IR) pipelines. Pre-trained language models (PLMs) have recently been leveraged across bi- encoder (Karpukhin et al., 2020; Xiong et al., 2021; Zhuang and Zuccon, 2021c: Wang et al., 2022a; Gao and Callan, 2022), cross-encoder (Nogueira and Cho, 2019; Nogueira et al., 2020; Zhuang et al., 2021), and sparse (Lin and Ma, 2021; I; Formal et al., 2021; Zhuang and Zuccon, 2021b) ranker architec- tures, showing impressive ranking effectiveness. Despite this success, the strong effectiveness of
We focus specific PLM-based QLM, the re- cently proposed Unsupervised Passage Re-ranker (UPR) (Sachan al., 2022). UPR leverages ad- vanced LLMs to obtained the query likelihood esti- mations. Empirical results show that using the TO LLM (Sanh et al., 2022) as a QLM, large gains in ranking effectiveness can be obtained. A key aspect of this work is that this effectiveness is obtained without requiring additional fine-tuning data, mak- ing Sachan et al. highlight the zero-shot ranking pabilities of their L.L QLM. However, we argue that the experimental setting used by Sachan et al. does not fully align with a genuine zero-shot scenario for the QLM ranking task. This is because TO has already undergone fine-tuning on numer-
rankers does not always generalise without sufficient in-domain training data (Thakur al., 2021: Zhuang and Zuccon, 2021a, 2022). Transferring knowledge from other domains has been used to overcome this issue (Lin et al., 2023) by training these rankers on large-scale supervised

---

ous question generation (QG) tasks and datasets, subsequent to its unsupervised pre-training. Con- sequently, there exists a discernible task leakage to the downstream QLM ranking task, thereby render- ing their approach more akin to a transfer learning setting, rather than true zero-shot approach.
the average log likelihood of generating all query tokens, as estimated by a LLM. The relevance scor- ing function defined as:
SQuar d)
Σ log p.. (I)
To gain comprehensive understanding of the zero-shot ranking capabilities of LLM-based QLM rankers, in this paper we take a fresh examination of this topic. Our approach involves harnessing the power of state-of-the-art transformer decoder- only LLMs, such as LLaMA (Touvron et al., 2023), which have undergone pre-training solely on un- structured text through unsupervised next token prediction. Importantly, the models we consider have not undergone any additional supervised in- struction fine-tuning, ensuring a truly complete zero-shot setting for our investigation.
here, at denotes the t-th token of the query, p is a model and task specific prompt used for prompting the LLM to behave like a question gen- erator (see Appendix A for more details), and LLM(g) d, 9cx)) refers to the probability of gen- erating the token Qi given the prompt p, the candi- date document d. and the preceding query tokens ęęz- is important to note that, in truly zero-shot ranking pipeline, the first-stage retriever should be a zero-shot method and the QLM re-ranker should exclusively be pre-trained on unsupervised unstruc- tured text data and no fine-tuning is performed using any QG data.
We further extend our analysis by comparing the effectiveness of these LLMs with various popu- lar instruction-tuned LLMS in the context of zero- shot ranking tasks. Interestingly, our findings re- veal that further instruction fine-tuning adversely affects the effectiveness of QLM ranking, particu- larly when the fine-tuning datasets lack specific QG tasks. This insight highlights the strong zero-shot QLM ranking ability of LLMs that solely rely on pre-training. thereby suggesting that further instruc- tion fine-tuning is unnecessary for achieving strong zero-shot effectiveness. Building upon these in- sights, we push the boundaries of zero-shot ranking even further by integrating a hybrid zero-shot first- stage retrieval system, followed by re-ranking us- ing the zero-shot LLM-based QLM re-rankers and a relevance score interpolation technique (Wang et al., 2021). Our approach achieves state-of-the the- art effectiveness in zero-shot ranking on subset of the BIER dataset (Thakur et al., 2021).
Interpolating with first-stage retriever: Fol- lowing Wang al. (2021), instead of solely relay- ing on the query likelihood scores estimated by the LLMs, we also linearly interpolate the QLM score with the BM25 scores from the first-stage retriever by using the weighted score sum:
S(g, d) Spazs d) + ()) Souus (g.e)) d), (2)
Here, a E [0, 1] represents the weight assigned balance the contribution of the BM25 score and the QLM score. In our experiments, we heuristi- cally apply min-max normalization to the scores and assign more weight to the QLM scores, given its pivotal role as the second-stage re-ranker. This is achieved by setting ing any grid search. We use the python library ranx² (Bassani and Romelli, 2022) to implement the interpolation algorithm.
0.2 without conduct-
Few-shot QLM re-ranker: Since LLMs are strong few-shot Icarners (Brown et al., 2020), we also conducted experiments to explore how LLM- based QLM re-rankers could be further enhanced by providing minimal number of human- judged examples. To achieve this, employed a prompt template known "Guided by Bad Questions" (GBQ) (Bonifacio et al., 2022). The GBQ template consists of only three document, good question, and bad question triples. We use to guide the LLLM-based QLM to produce more accurate query likelihood estimations. We refer readers to the orig- inal paper for details about the GBQ template.
2 Methodology
Zero-shot QLM re-ranker: We follow the setting introduced in previous works (Zhuang et al., 2021; Sachan et al., 2022) to evaluate the zero-shot QLM ranking capability of LLMs. Specifically, given a sequence of query tokens q and set D contain- ing candidate documents retrieved by first-stage zero-shot retriever such as BM25, the objective is rank all candidate documents d E D based on
'There are at least 16 QG datasets according to the open- soureed TO training: https: htt///uuggnnggg/ face. co/datasets/ bi gsciencee/P3
Pht / //gi thub. com/AmenRa/ranx

---

3 Experimental Settings
Table 1: Main results. Re-rankers re-rank Top100 doc- uments retrieved by BM25. Transferred retrievers and re-rankers are fine-tuned on MS MARCO.
Our focus is the response of LLMs in the QLM ranking task, specifically in genuine zero-shot setting. To accomplish this, we used LLaMA (Touvron al., 2023) and Falcon (Al- mazrouci et al., 2023), both of which are trans- former decoder-only models that are pre-trained solely on large, publicly available unstructured datasets (Penedo et al., 2023). We specifically con- sider open-source LLMs because we can control the data used to train them, thus guaranteeing QG dataset was used.
LLMs:
Methods
TRECC
DBpedia
FQQA
RobusiD4
Avg
Zero-shof Retrievers
BM25
59.5
31.8
23.6
38,9
QLM-Din Dirichlet
50.8
29.5
20.5
40.7
35.4
Contriever
23.3
29.2
24.5
31.6
27.2
HyDE
37.2
Instruction tuned QLM Re-rankers
Without QG task
T5.3B
48.7
21.9
38.0
31.2
T5-1IB
67.9
33.7
31.0
27.4
40.3
Alpaca-78
67.1
35.0
33.7
44.6
45.1
Stahlel
74.0
37.2
VieuueVicua- 13B
7188
39.4
39.1
51.3
50.4
Falcon-7B-insrrat
66.8
38.2
33.4
50.7
47.3
Falcon-40B Bis-iiiteect
To evaluate the influence of instruction fine- tuning data on QLM estimation, we compared these models with other well-known LLMS that were fine-tuned with instructions, including T5 (Raffel et al., 2020), Alpaca (Taori et al., 2023), StableLMM, Stable and al., 2023). is important note that the fine- tuning instruction data for these models are un- likely to include QG tasks.² Additionally, we fol- low Sachan et al. (2022) to include TO (Sanh et al., 2022) and FlanT5 (Chung et al., 2022), which un- derwent fine-tuning specifically for QG instruc- tions. All LLMs used in this paper are openly available, see Appendix B for more details.
With QG task TO-3B
71.6
38.8
41.4
50.1
50.5
73.9
38.7
FlanTS-3B
71,1
39.7
41.2
50.0
50.5
FlanTS-1
74,9
41.7
43.3
52.4
53.1
Zero-shot QLM
LLaMA-78
69,4
39.9
41.5
53.6
51.1
(Almazrouei
LLaMA-13B
69.8
37.6
41.8
54.2
50.9
Falcon-78
73.3
Falcon-908
75,2
41.0
43.1
53.1
53.1
Trausferred Retrievers
Contriever-msmaree
59.6
41.3
32.9
47.3
45.3
SPLiDE-distil
71.1
44.2
DRAGON+
75.9
41.7
35.6
47.9
50.3
Transferred Re-rankers
TS-QL M-Lagee
38.0
MonoTS-3B
79.8
44.8
46.0
56.2
56.7
monoTS.33 InPars-v2
83.8
46.6
46.1
58.5
58.8
datasets TRECC (Voorhees et al., 2021), DBPe- dia (Hasibi al., 2017), FiQA (Maia et al., 2018), and Robust04 (Voorhees, 2005). The evaluation metric used nDCG 10, the official metric of the BEIR benchmark.
Baselines and datasets: In our evaluation, we compared LLM-based QLMs with several exist- ing methods, including BM25 (Robertson and Zaragoza, 2009), QLM-Dirrchllt (Zhai and Laf- ferty, 2004), Contriever (Izacard et al., 2022), and HyDE (Gao et al., 2022), which are zero-shot first-stage retrievers. We also compared them with fine-tuned retrievers and re-rankers trained on MS MARCO passage ranking data, representing a transfer learning setting. Specifically, the evalu- ated retrievers are Contriever-msmarte SPLADE- distill (Formal et al., 2022), and DRAGON+ + (Lin et al.. 2023), while the re-rankers are T5-QLM- large (Zhuang et al., 2021), monoT5-3B (Nogueira et al., 2020), and MonnoTS-3B-Inparssee (Jeronymo compared our
Statistical significance analysis was performed using Student's 's two-tailed paired t-test with cor- rections, per common practice in information retrieval. The results of this analysis reported in Appendix due space constraints.
Results
4.1 Main results
We present main results in Table 1, highlighting key findings. For fair comparison, all the re-rankers consider the top 100 documents retrieved by BM25. Firstly, it is evident that retrievers and re-rankers finetunned on MS MARCO training data consis- tently outperform zero-shot retrievers and QLM
et al., 2023). Additionally, best QLM ranking pipeline with PROMPTAGA- TOR (Dai et al., 2023), a state-of-the-ee zero-shot and few- method. See Appendix for detailed information about the baselines.
To ensure feasibility, we conducted experiments popular subset of the BEIR benchmark
"Due to limited computational resources and numerous LLMs with various settings to run, and in order to ensure feasibility, we considered a subset of BEIR that includes the most widely used datasets in the literature. Despite being subset, it comprises a total of 1,347 queries with deep ranking judgments across 4 distinct domains.
'These models however employ the self-instruction up- proach (Wang et al., 2022b), which may involve small num- her of randomly generated QG instructions.

---

re-rankers across all datasets, except for T3-QLM- large, which is based on smaller T5 model. This outcome is expected since these methods benefit from utilizing extensive human-judged QA training data and the knowledge can be effectively trans- ferred to the datasets we tested.
Table 2: Interpolation results.
Increased/decreased
scores noted with f/4-
Methods
DBpedia
FIQA
Robusti44
Aug
without interpolation
mono mono11-38 15-38
79.8
44.8
46.0
56.2
56.7
46.6
5-3
72.0
37.0
41.7
47.0
49.4
FlanTS.118 IIB
75.1
39.9
44,9
50.8
52.7
the other hand, zero-shot QLMs and QG fine-tuned QLMs exhibit competitive, similar ef- fectiveness. This finding is somewhat surprising, considering that QG fine-tuned QLMs are explic- itly trained on QG tasks, making them form of transfer learning. This finding suggests that pre- trained-only models such as LLaMA and Falcon possess strong zero-shot QLM ranking capabilities.
LLaMA-7B
37.5
51.6
Falcon-78
73,1
39.5
41.7
49.2
50.9
with interpolation
monoTS.
66.3
44.6
41.51
55.14
$1,91
82,1
45.5-
43,5
54.0
56.34
FlanT5.30
71.1
39.7"
41.24
50.0
50,51
F-an15-1188
74.9-
41.7"
43.31
52.47
53.17
LLaMA 7B
69.41
39.9"
41,54
53.67
$1.11
Fatcon-7B
73.37
41.7"
41.31
52.57
52.2"
ing interpolation with BM25. It is worth noting that this improvement is (almost) cost-free, it does not require any additional relevance score es- timation; simply involves linearly interpolating with scores from the first stage.
Another interesting finding is that instruction tuning can hinder LLMs* QLM ranking ability if the QG task is not included in the instruction fine- tuning data. This is evident in the results of Alpaca- 7B, Stable 1B,, which are instruction-tuncd versions of LLaMA-7B, LLaMA-13B, Falcon-77 and Falcon-4OB, respectively. Our hypothesis to this unexpected finding is that instraction-tuned models tend to pay more attention to the task in- structions and less attention to the input content itself. Although they are good following instruc- tions in the generation task, the most important information for evaluating query likelihood in the document content, thus instruction-tuning hurts query likelihood estimation for LLMs. On the other hand, QG instruction tuned LLMs show large im- provements in QLM ranking. For example, the TO and Flan models are QG-tuned versions of T5 models, and they perform better. These results con- firm that TO and F5anT5 leverage their fine-tuning data, thus should be considered within the transfer learning setting.
and
We note that the results Table are obtained by setting a = 0.2 without tuning this parameter because we are testing our method in zero-shot setting where this parameter needs to be set with- out validation data. Nonetheless, we conduct a post-hoc analysis TRECC to understand the sen- sitivity of this parameter. The results are presented in Figure 1. From the results, we can draw the following conclusions:
Falcon-40B-ing
1. The interpolation strategy consistently has a negative impact monoT5-3B, while it con- sistently benefits instruction-tuned and zero- shot rerankers.
2. Instruction-tunel rerankers consistently un- derperform their corresponding zero-shot rerankers, regardless of the set alpha value.
3. Optimal values of a for both instruction-tuned and zero-shot rerankers fall within the range of 0.1 to 0.4.
In terms of model size, larger LLMs generally tend to be more effective, although there are ex- ceptions. For instance, LLaMA-7B outperforms LLaMA-13B on DBpedia.
4.3 Effective ranking pipeline
In Table 3 we push the boundary of our two-stage QLM ranking pipeline in both zero-shot and few- shot setting to obtaín high ranking effectiveness. For this purpose, we use the same lincar interpola- tion as Equation 2 with = 0.5 to combine BM25 and HyDE as the zero-shot first-stage retriever. The top 100 documents retrieved by this hybrid retriever then re-ranked using QLMs.
4.2 Interpolation with BM25
Table 2 demonstrates the impact of interpolating with BM25 scores. Notably, we observe a large decrease in the effecctiveness of monoT5 re-rankers, which are trained on large-scale QA domain data, when interpolating with BM25, This finding aligns with a study conducted by Yates et al. (2021). In contrast, QLM re- re-rankers consistently exhibited higher effectiveness across most datasets when us-
*This value was chosen to provide equal weight to the two components, and no parameter exploration was undertaken.

---

Zero-shot/few-shot ranking systems. "PROMPTAGATOR++ re-rankers their zero/few-shot PROMPTAGATOR first-stage retrievers, scores are copied from the original paper as the model is not publicly available. Other re-rankers consider the Top 00 documents retrieved by BM25 + HyDE.
Table 3:
monoTB-38 Falcon instruction
Falcon-7B
Alpaca-78
LLaMA-78
Methods
TRECC
DBpedia
FIQA
Robast0
Avg
Zero-shot Retrievers
'PROMPTAGATOR
72.7
36.4
40.4
Few-shol Retrievers
PROOPPAGCTOR
Zero-shot Re-rankers
'RROPPTGATOOR
76.0
41.3
45.9
HaiT5-11B
Figure 1: Impact of TRECC dataset.
StahleLLM-7B
74.2
41.8
38.0
53.2
51.8
Alpaca-7B
71.6
40.1
39.5
50,9
50.5
LLaMA-7B
72.4
Falcon 7B instract
68.8
43.1
37.4
54.6
51.0
Firstly, our results suggest that the effectiveness of zero-shot first-stage retrieval can be improved by simply interpolating sparse and dense retrievers. Moreover, after QLM re-ranking, the nDCG@ 10 values surpass those in Table 1. This indicates that zero-shot QLM re-rankers benefit from a stronger first-stage retriever, leading to improved overall ranking effectiveness. For the few-shot results, observe that providing only three GBQ examples to the model further enhances ranking effectiveness, although this effect less pronounced for FlanT5. Remarkably, our QLM ranking pipeline achieves nDCG@ 10 on par with or higher than the state-of- the-art PROMPTAGATOR method on comparable datasets in both zero-shot and few-shot settings. It is important to note that PROMPTAGATOR re- large amount of synthetically
Falcoe-7B
76.6
46.1
45.8
55.1
55.9
Few-shot Re-rankers
PROMPTAGATORE
76.2
43.4
49.4
FlanTS-IIB
StableLM
72.2
42.8
38.0
51.7
51.2
Alpaca-78
72.3
42,5
41.8
53.0
52.4
Falcon-78-instnut
74.9
45.2
42.8
56.1
54.8
Falcon 7B
78.6
48.0
48.6
59.0
58.5
zero-shot and few-shot scenarios, showcasing the effectiveness and versatility of LLM-based QLMs.
Limitations
While theoretically our QLM method can ap- plied to any LLM, for practical implementation, access to the model output logits is required. There- fore, in this paper, our focus has been solely on open-source LLMs where we can have access to the model weights. In contrast, approaches like Inpars and PROMPTAGATOR, which extract knowledge from the text produced by LLMs, do not require access the model weights. Common commer- cial API services that expose popular close-source models such as GPT-4, however, do not provide access to model logits. offered by popular close- source models such as GPT-4 These can easily be used within Inpars and PROMPTAGATOR by directly leveraging the generated text. However, our method cannot these models because they do not provide access to the logits. might be possible that in future commercial LLM provides would add functionalities in their APIs to access model logits.
quires training generated data for both the retriever and re-ranker, whereas our approach does not require any train- ing. It's worth highlighting that instruction-tuned LLMs continue to exhibit lower effectiveness com- pared to their pre-trained-only LLMs, even when a better first-stage retriever is employed and under few-shot setting.
5
Conclusion
this paper, we adapt recent advanced LLMs into QLMs for ranking documents and comprehensively study their zero-shot ranking ability. Our results highlight that these LLMs possess remarkable zero- shot ranking effectiveness. Moreover, we observe that additional instruction fine-tuned LLMs unper- formed in this task. This important insight is looked in previous studies. Furthermore, our study shows that by integrating L.LM-based QLMs with a hybrid zero-shot retriever, novel state-of the-an art ranking pipeline can be obtained that excels both
Our focus on open-source LLMs also offers the opportunity scrutinise the data used train the LLMMs ascertain that no QG data was used. This reassures a genuine zero-shot setting is consid- ered, as opposed to previous work on LLM-based

---

QLMs (Sachan et al., 2022). Although LLaMA and Falcon are primarily pre-trained using unsuper- vised learning on unstructured text data, it remains possible that the pre- wre-training data contains text snip- pets that serve as instructions and labels for the QG task. In order to ascertain the authenticity of the zero-shot setting, it may be necessary to thoroughly analyze and identify such text snippets within the pre-training data.
Hyung Won Chung, Le Hou, Shayne Longpre. Bar- ret Zoph. Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finttun language models. arXiv preprint ar.iv:2210.11111 11416.
Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming- wei Chang. 2023. Promptagator: Few- shot dense retrieval from 8 examples. In The Eleventh International Conference on Learning Representa- tions.
In the paper, we could not report a complete sta- tistical significance analysis of the results due to space limitation. Appendix D reports a detailed analysis. However, our analysis was limited by the unavailability of run files for some of the models published in previous works, they were not re- leased by authors. In these cases, could not perform statistical comparisons with respect to the runs we produced. We note this a common prob- lem when authors do not release their models' runs, We make all run files available, along with code, at https: thub. http//elaabbllmg
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-41880 Minneapolis, Minnesota. Association for Computational Linguistics
Thibault Formal, Carlos Lassance, Benjamin Pi- wowarski, and Stéphane Clinchant. 2022. From dis- tillation to hard negative sampling: Making sparse neural ir models more effective. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22. page 2353-2359, New York, NY, USA. Association for Computing Machinery.
References
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli. Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay. Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B 40B: an open large language model with state-of-the-art performance.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant 2021. Splade: Sparse lexical and expan- sion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 2288-2292, New York, NY, USA. Association for Computing Machinery.
Elias Bassani and Luca Romelli. 2022. ranx. fuse: A python library for metasearch. In CIKM, pages 4808- 4812. ACM.
Luyu Gao and Jamie Callan. 2022. Unsupervised cor- pus aware language model pre- pre-training for dense pas- sage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2443-2853, Dublin, Ireland. Association for Computational Lin- guistics.
Luiz Bonifacio, Hugo Abonizio, Marzich Fadace, and Rodrigo Nogueira. 2022. Inpars: Unsupervised dataset generation for information retrieval. In Pro- ceedings of the 45th International ACM SIGIR Con- ference on Research and Development in Information Retrieval, SIGIR '22, page 2387-2392, New York, NY, USA. Association for Computing Machinery.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without rele- vance labels. arXiv preprint arXiv:2212. 70496.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, 20, Red Hook, NY, USA. Curran Associates Inc.
Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisz- tian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017. Dbpedia-entity v2: A test collection for entity search. In Proceedings of the 40th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, SIGIR '17, page 1265-1268. New York, NY, USA. Association for Computing Machinery.
Djoerd Hiemstra. 2000. Using language models for information retrieval. Ph.D. thesis, University of Twente.

---

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave, 2022. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions on Machine Learning Research.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre- trained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708-718, Online. Association for Computational Linguisties.
Vitor Jeronymo, Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, Roberto Lotufo, Jakub Zavrel, and Rodrigo Nogueira. 2023. Inpars-v2: Large language models as efficient dataset generators for information retrieval. arXiv preprint arXivv2331.0 00100..0.
Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to doc TTTTTqqur...
Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nalla- pati, Zhiheng Huang, and Bing Xiang. 2020. Beyond [CLS] through ranking by generation. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1722-1727, Online. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-69 6781, Online. Association for Computational Linguistics.
Guilherme Penedo, Quentin Malartic. Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.011CC
Oleg Lesota, Navid Rekabsaz, Daniel Cohen, Klaus An- tonius Grasserbauer, Carsten Eickhoff, and Markus Schedl. 2021. A modern perspective on query likeli- hood with deep generative retrieval models. In Pro- ceedings of the 2021 ACM SIGIR International Con- ference Theory of Information Retrieval, ICTIR '21. page 185-195, New York, NY, USA, Association for Computing Machinery.
Jay M. Ponte and W. Bruce Croft. 1998. A language modeling approach to information retrieval. In Pro- ceedings of the 21st Annual International ACM SI- GIR Conference Research and Development in Information Retrieval, SIGIR '98, page 275-281, New York, NY, USA. Association for Computing Machinery.
Jimmy Lin and Xueguang Ma. 2021. A few brief notes on deepimpact, coil, and conceptual framework for information retrieval techniques. arXiv preprint arXiv:2106. 14807.
Sheng-Chieh Chich Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023. How train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07452.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Xueguang Ma, Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2022. Document expansion baselines and learned sparse lexical representations for ms marco v1 and v2. In Proceedings of the 45th In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 31 3187-3197, New York, NY, USA. Association for Computing Machinery.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and be- yond. Found. Trends Inf. Retr. 3(4):333- 389.
Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 3781-3797, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www'll 18 open challenge: Financial opinion mining and question answering. In Companion Proceedings of the The Web Conference 2018, WWW '18 page 1941-1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker. Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Seao, Stella Biderman, Leo Gao,
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2017. MS MARCO: A human- generated MAchine reading COmprehension dataset.
Rodrigo Nogueira and Kyunghyun Cho. 2019. Pas- sage re-ranking with bert. arXiv preprint
arXiv:7901.00085

---

Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-sho task gener- alization. In International Conference on Learning Representatins.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok- Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Amold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In International Conference on Learning Representations.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-follle following Ilama model. https:// thub. com cromm/tattu-labb habbb/sanfodd_aaappo
Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021. Pretrained transformers for text ranking: BERT and beyond. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Language Tech- nologies: Tutorials, pages 1-4, Online. Association for Computational Linguistics.
Nandan Thakur, Nils Reimers, Andreas Rücklé Ab- hishek Srivastava, and Tryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fitth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Chengxiang Zhai and John Lafferty. 2001. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Re- trieval, SIGIR '01, page 334-342, New York, NY. USA. Association for Computing Machinery.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971.
Chengxiang Zhai and John Lafferty. 2004. A study of smoothing methods for language models applied to information retrieval. ACM Trans. Inf. Syst., 22(2):179-214.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman William R. Hersh, Kyle Lo, Kirk Roberts, lan Soboroff, and Lucy Lu Wang. 2021. Trec-covid: Constructing a pandemic information retrieval test collection. SIGIR Forum, 54(1).
Shengyao Zhuang, Hang Li, and Guido Zuccon. 2021. Deep query likelihood model for information re- trieval. In Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 188AAy April 1, 2021, Proceedings, Part II, page 463-470, Berlin, Heidelberg. Springer- Verlag.
Ellen M. Voorhees. 2005. The tree robust retrieval track. SIGIR Forum, 39(1):
Liang Wang, Nan Yang. Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. 2022a. Simlm: Pre-training with represen- tation bottleneck for dense passage retrieval. arXiv preprint arXiv:2007.02577
Shengyao Zhuang and Guido Zuccon. 2021a. Deal- ing with typos for BERT-hased passage retrieval and ranking. In Proceedings of the 2027 Conference on Empirical Methods in Natural Language Processing, pages
Shuai Wang, Shengyao Zhuang, and Guido Zuccon. 2021. Bert-based dense retrievers require interpo- lation with bm25 for effective passage retrieval. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, IC- TIR '11, page 317-324, New York, NY, USA. Asso- ciation for Computing Machinery.
Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Shengyao Zhuang and Guido Zuccon, b02 Fast pas- sage re-ranking with contextualized exact term match- ing and efficient passage expansion. arXiv preprint 10Xiv:2108.08513
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022b. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint 221i::221..1 10560.
Shengyao Zhuang and Guido Zuccon. 2021c Tilde: Term independent likelihood model for passage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Devel- opment in Information Retrieval, SIGIR '21, page 1483-1492, New York, NY, USA. Association for Computing Machinery.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Pier- rie Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of of the natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 48, Online. Association for Computational Linguistics.
Chaumond, Clement Delangue, Anthony Moi,
Shengyao Zhuang and Guido Zuccon. 2022. Character- bert and self-teaching for improving the robustness of dense retrievers on queries with typos. In Proceed- ings of the 45th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, SIGIR '22, page 1444-1454, New York, NY, USA. Association for Computing Machinery.

---

Table 4: Prompts used for each LLM-dataset pair. For Alpaca-7B and StableLM-7B we also prepend a system prompt according the fine-tuning recipe of the each model. For Alpaca-7B is "Below is instruction that describes a task, paired with an imput that provides further context. Write response that appropriately completes the request.lnkn". inla"" For SableLM-7B "<ISSSTMM StableLM Tuned (Alpha version)in- StableLM is a helpful and harmless open-source AT language model developed by Stability AL. MtableI is excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.An- StableI.M is more than just an information source, StableLM also able to write poetry, short stories, and make jokes.'n- In- StableLM will refuse to participate in anything that could harm human.In"
LEMs
TRECC
Diipedia
FIQA
Robustb4
15-38/S--1B//lult 3B/FarTS--11
question dist
Genente query includes thy and highly relevast the gives Wikigedli page till and ab- stracila/doe)
quesioon the evant to the given documentouldod
guesion the evant to given dacumeatia)doc)
evaut the gives atticle's titl and ab- (daac)
T0-38/70-118
Please wiite quection hased on this
Please wille question hased on this
Piease wrile a quostios hood on this
Pleae wrile quesiiss loaed this passagalal (doe)
Generate question that mosi evant the glve amicle's is till and ah- virset inf (eellaeee is a genersted relevaat question.
Gonentee query that includes en riry and highly mileure the gives Wikigedie page tills and ab- stractiol (oo l'llateee gesevated relerant queston:
the nost event to the givrn docament lnThe doe-
Genernee a question
Generate quesion is
7041 LMM Firf
mont in the given document thT doe-
7B/Faloon- MB//alionn 7B FnstrucFiiioon 13B-
unere: [die JahnHerr is geserated velevart questios:
umest: (doe WWisHee is generated
relevant question:
Alpaca:71
quaetiio
gurry
inmeectooooelimeeeee queciion
spassion
that the most selesast to given
that includes an catity and ahon highly relevast to the given Willgedia page uite and abstroct per 'n http:///oooeeleeeeeueee
that's mast selovare to the gives doe- unecat wishtn Iaputinl doolulu
most selovant the gives uneat Iaputtel doe l'atabin Respoise:
atticle's tile and abotaact wlatti In- puiteeddoc//tttette pistntte Resporse:
In
Resporse:
SGheLMM-78
<IUSER
question
2USEBU
query that includes
<USERIsGenense
eUSExtstennnte the zist relevntt to given documentxe document:
question
that is the most relevaut the given arrite's use and ab strazz in (nfeen/ ciiASSISTANT---le in generated retevant question:
an catity and also highly relevant
that the nost selevaat to the given de document: [due (asssssSSSSATTTT Tistiee in
the given Wiltpedia page tite and ab
srrec (dor ASSISTAANTs-Heee is
(dee))dhSSSSTAANn-lll is a
generaaed selevant question:
generazed relevani question:
generated televarr questica
Suull 13B
inte Hunas. Geneaat most selevant to given antice's and sharaet dee ''vees Awietent
that
Human.
quecy
that given mere liThe docummnt: (doe//toee
Human Glsevate
Humas Geacrate
clides ensiyy and also highly rel evamt to the given Wilipedia page dk and abotmat dee ludaa Assietat:
the most relevant
given men TyT document: (doc///ndee
is the most relevant
Here is genertted relevam quesiin::
sistam: Hore is a genentaa relevant quediinn
sistar: Hare is a genestte relevant question:
Hem is genveated relevant qreation:
A Models and datasets prompts
Given that various instruction-tunel LLMs might be fine-tuned using diverse system and instruction prompts, coupled with the fact that datasets vary in document formats across different domains, becomes necessary to employ specific prompts tai- lored to each LLLM-dataset pair to achieve optimal zero-shot ranking performance. Thus, we design a prompt for each LLM--dtaset pair based on the LLM usage instruction provided by the original authors and dataset features. To facilitate clarity, we have compiled comprehensive list of all the prompts utilized for each LLM--ataset pair, which can be found in Table 4.
Table 5: Huggingface model hub links for LLMS used in this paper.
LLMS T5-3B
Link
(5-3b
T5-11B
15-11b
StableLM-BB
Stable Vicuna-
The Blooesssstaalllevicuna.
13ttuuaaa-11---11 13B-HF
Falcon-7B
tiuaa/falloon-7
tilua//falcon- 7b-instruct
Falcon-40B
tiuac/falcon-100
Balcon-40B-instuen
tiinac/falcon-40b-insttt
B List of Huggingface model names
TO-3B
bigscience/TO.B. 3B
TO-11B
bigscience//OO
Table 5 provides links to the Huggingface model hub (Wolf et al., 2020) for the LLMs used in this paper. All the models can be conveniently down- loaded directly from the Huggingface model hub, with the exception of Alpaca- 7B. For Alpaca-7B, we followed an open-sourced github repository to perform the fine-tuning of LLaMA-7B ourselves.
FlanT5-3B
google/flan-t5-xl
FlanT5-11B
google//fun-155xx
LLAMA-7B
huggy huggyllama//lamm-77
LLaMA-13B
13b
Alpaca-7B
https//gittuh.com/aaao
lab/stanford alpaca
C Descriptions of Baselines
BM25 (Robertson and Zaragoza, 2009): A widely used statistical bag-of-words approach

---

Table 6: Overall effectiveness of the models and statistical sigmificance analysis. The best results are highlighted in boldface. Superscripts denote significant differences (1-test, p I 0.05). x y denotes the retriever re-ranked by y re-ranker.
Model
TRECC
DRipeliia
FIQA
Robusth
BM25
59.5
31.8*
23.6
40,7"
QLM-Dirialee
50.5
20.5
HyDE:
58.2
37.1 37.1
26.6*
41.8"
BM25-HHyE
41.5 fgghtho//hh
30,9
49.7
BM25 T5-1-1
67,9
33.7ua
52.0ekc
27.4
BM25 Alpaca-71
67.0%
35.0*
33, 74.7
44, 44,0"*
BM25 StableL.M-BB
2
shode
3
BM25 -> StubleVicmm-1138 138
71.8% 81.8
39.4 10.4
39. patodo/as
51.3 30kc///
BM25 Falconn7B-inp -i-inttrct
66.8%
38.2 2
33.g
50.
BM25 Falcom-008-intrrss
40.5 10.5>ke/ssill fyiily
10.9 Ealh
BM25 T0-38
38,8 gh
41. 4ooodegag
50. pooode
BM25 -O0-1D
gh Puop
Jitc/s
43. gessseenee
BM25 -> FlaiTS-38
71. 1
39. 7
41. 2dlohto
50.00iee/
BM25 Flan15-118
74,9 24.grbod bode figincc grde
41.7 polooretiittu
BM25 -LLaMA-7B
39 gelee
11.5 Sah Eale
53 (f
BM25 11-MM-135
69 gele
37.6 ge
41.8 fak
54 fe
BM25 Falcon-7B
41.4 gag Fahssiopp
BM23 -> =Falcon-OO8
75 2eteeot//ohboen
41.0 41.0 fahvliny
43.1 13ot fahelhneyy
53. phote/ttijiin
BM25 momi11--38
44. gelete Soletthoan
46. 11 hogottrgttttreeeer
56 2ehdde getoeeghgsinn
BM25 > mmmmmm-m33--mmmm
83 S.praeaangtiooorrraanyy
46.5 gunoscianggeeemeee
46. pogttogggg
58.5 58.5 thottronreee
BM25-HyDDF RaaTS-1
75.8 75.8 Figiooo
46. 2 faheptiionuyro
49. 500,54noosghhjjliooo
BM25-11yDE->LLLA
72
15.6 hhota fahestienncerr
8 gahod Zagginageeeeeeee
57.1 57,
BM25-HDDE:>> Falcon-7B
76.6 inhrrgge
46. polea sirpebaapp
45.0 gohoh Iahilhhunngg
55. fphaiiigg
BM25-H1yD
27.2 phighrappss
45.1 1 Taheshiinnspy
49.7 29.7shobe 99.74tok/ghitthooueee chese
58.3 58.3 www.://fejjjaoaaeee....
47.7 4 phottrensnnneeeeeeeee
50.4 /twgg//nnnnnnnrrttteeooo
59.5 wwg.saggggggbooorrrrccoo
BM25-HyDE Falzon-TB- Seweboo
78.0 foitsho
48.0 hwhhhghhnghneeemmmm
48. 6 6 4 hatjs//sptttoeerrrtee atre
that is commonly used as the zero-shot first- stage retrieval method. We use the Pyserini "two-click reproductions" (Ma et al., 2022) to produce the BM25 results on BEIR datasets.
pre-build dense vector index and model check- point for this baseline.
SPLADE-distill (Formal et al., 2022): A first-stage sparse retrieval model that exploits BERT PLM to learn query/document sparse term expansion and weights. We use the Py- serini provided pre-build index and SPLADE checkpoint to produce the results.
QLM-Dirichlet (Zhai and Lafferty, 2001): The traditional QLM method that exploits term statistics and Dirichlet smoothing tech- nique to estimate query likelihood, we also use Pyserini implementation for this baseline.
(Lin et al., 2023): A dense triever model that fine-tuned on augmented MS MARCO corpus and uses multiple retriev- ers to conduct automatical relevance labeling. It stands as the current state-of-the-art dense retriever in the transfer learning setting. We use the scores reported on the BEIR learder- board 6 for this baseline.
DRAGON+
Contriever (Izacard et al., 2022): A zero-shot dense retriever that pre-trained on text para- graphs with unsupervised contrastive learning.
HyDE (Gao et al., 2022): A two-step zero- shot first-stage retriever that leverages gen- erative LLMs and Contriever. In the first step, a prompt is provided to a LLM to gener- ate multiple documents relevant to the given query. Subsequently, in the second step. the generated documents are encoded into vectors using the Contriever query encoder query
T5- T5-QLM-large (Zhuang et al., 2021): A T5-based QLM method that fine-tuned on MS MARCO QG training data. We use the implement this method with open-sourced doeTquery--T (Nogueira and Lin, 2019) checkpoint?. 7.
and then aggregated to form vector for the search process. We utilized the open-sourced sourced implementation provided by the original authors for our experiments:
monoT5-3B (Nogueira et al., 2020). A T5- based cross-encoder re-ranker that fine-tuned on MS MARCO training data. We use the
Contriver-msmarco. A Contriever check- point further pre-trained on MS MARCO training data. We use the Pyserini provided
hhttps://eval /ttp://eeeeeeeeeeeeeeeeea challeges// chal hwtngggepaag//11///eaeeebbbbb Caasoommttoozgaarr----ggemmmmmm

---

open-sourced implementation provided by In- pars authors"
10-3oT55-B-npaaaee (Jeronymo al., 2023): A T5- cross-encoder re-ranker that fine-tuned on MS MARCO training data and in-domain synthetic queries that gener- ated by LLMs. It the current state-of-the-art re-ranker in transfer learning setting. We use the open-sourced implementation provided by the original authors? 9
PROMPTAGATOR(Dii et al., 2023): These methods consist of a Transformer encoder- based retriever and re-ranker that are trained using synthetic queries generated by LLMs. They offer both zero-shot and few-shot set- tings. As public model checkpoints are not currently available, we refer to the scores re- ported in the original paper as our point of reference for comparing against our own meth- ods and baselines.
D Statistical significance analysis
Table we report a statistical significance analy- sis for all the methods for which we can obtain a run file, along with our methods. The analysis was performed using the Student's two-tailed paired t- test with corrections, per common practice in information retrieval. We used the Python toolkit ranx (Bassani and Romelli, 2022) for generating the report.
"nttps://gittub..omm http:///gggttto..oe/aaaaaaattttr hatt:p///ggtuuuu.mm/ttaaaapppppp


Reviews:
[{"review_id": "QOpRLPoDy3", "reviewer": "Reviewer_1y9F", "summary": "", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- Thank you for the suggestion on improving the structure of the presentation of the results. This is a good suggestion that we will adopt to improve the presentation of the paper.\n- Thank you for the additional references, which we will include in the paper", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Timely research on LLMs' zero-shot text reranking performance.\n* Extensive experiments are conducted on subset of BEIR.\n* Code is publicly available for reproduction.\n* Observations are supervising and insightful.\n* Limitations are thoroughly discussed.", "reasons_to_reject": "The several surprising experimental conclusions in Section 4 are scattered throughout the very lengthy plain text. It is suggested that the author(s) organize it more clearly, giving it more structure, so the readers can more easily access these conclusions.", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "This paper empirically evaluated the zero-shot text reranking performance of open-source Large Language Models (LLMs). The experiments demonstrate that a purely zero-shot LLM can even outperform an LLM that has undergone instruction-tuning in terms of text reranking performance. The experiments conducted in this paper are detailed and the conclusions are quite surprising.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please refer to \"Reasons To Reject\"."}, "extra_text": {"Missing_References": "Encourage the author to consider citing these related works that also use a similar approach to Eqn. (1) for text reranking.\n\nNiklas Muennighoff. 2022. SGPT: GPT sentence embeddings for semantic search.\n\nPercy Liang et al. 2022. Holistic evaluation of language models."}}, {"review_id": "64hel01o6W", "reviewer": "Reviewer_vsPL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing that the insights presented in our paper hold value for the community.\n***\n### Response to reject reason 1:\n> \"Paper claims that [1] does not meet the \"genuine zero-shot\" scenario a T0 has undergone fine-tuning on question generation (QG) tasks and there is a task leakage to downstream QLM ranking task. Lines 078 to 087. This is one of the key arguments of this paper, but it is not explained/proved in the paper why undergoing fine-tuning on question generation (QG) tasks disqualifies it as \"genuine\" zero-shot approach, nor it is supported by any references\"\n\nWe recognise that in the IR literature, there is actually not a clear and well-agreed-upon definition of ‘zero shot’.\n\nThe model of [1] is a Query Likelihood ranking approach: estimate the probability of a query given a document. The question generation task is close to estimating the likelihood of a query given a document: given a document, generate a question for this document. This generation is done by maximising the likelihood of the tokens in the relevant question. Thus we see the question generation task as one of optimising query likelihood. Therefore, the two tasks are computationally very similar to each other, and a model fine-tuned for one task will be adept at the other. We believe this situation is more appropriately regarded as transfer learning rather than zero shot (T0 is finetuned on other QA datasets and tested on BEIR). \n\nTo address this comment, we will add a section where we define the differences between these settings and we highlight our motivations for not considering the method of [1] as a true zero shot approach.\n***\n### Response to reject reason 2:\n> \"The weights for interpolation strategy is set to 0.2 without conducting any grid search (Line 158). No explanation is provided for the choice of the value for this parameter. The main results for this paper's claims depend on the choice of this parameter. Thus, it is important to find the optimal value of this parameter for different LLMs in different settings. It is possible that the key finding of the paper that the instruction fine-tuning degrades the ranking performance maybe because 0.2 value may be better for non-instruction-tuned LLM and worse for the instruction-tuned LLM, and choosing optimal value of this parameter may carry a fair comparison.\"\n\nOur paper sets out to test the real-life scenario of someone deploying LLMs in a real zero shot setting, where this parameter needs to be set without validation data (thus, without tuning the parameter itself). We set the parameter to 0.2 as a reasonable value: give more weight to the second stage re-ranker because this is a better model. We will add this reasoning to the paper.\n\nI do understand the reviewer's concern regarding the sensitivity of this parameter so we did conduct a linear search of alpha values for fine-tuned rerankers (monoT5-3B), instruction-tuned rerankers (Falcon-7B-instruct, Alpaca-7B), and Zero-shot rerankers (Falcon-7B, LLaMA-7B) on TRECC. The results are presented in the table below:\n\n| Methods\\Alpha | 0.0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 | \n|-------------------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|\n| monoT5-3B | 79.8 | 68.0 | 66.3 | 64.7 | 64.6 | 64.5 | 63.5 | 62.6 | 61.1 | 60.6 | 59.5 |\n| Falcon-7B-instruct | 63.0 | 64.9 | 66.8 | 66.8 | 66.7 | 64.8 | 63.6 | 63.0 | 62.4 | 61.6 | 59.5 |\n| Falcon-7B | 73.1 | 73.2 | 73.3 | 73.2 | 72.2 | 70.1 | 67.4 | 65.2 | 63.6 | 62.1 | 59.5 |\n| Alpaca-7B | 63.8 | 64.5 | 67.1 | 68.0 | 68.4 | 67.0 | 65.3 | 63.8 | 62.7 | 61.4 | 59.5 |\n| LLaMA-7B | 68.0 | 69.5 | 69.4 | 70.4 | 70.0 | 69.1 | 67.5 | 64.9 | 63.6 | 61.6 | 59.5 |\n\n\nFrom the table of results, we can draw the following conclusions:\n1. The interpolation strategy consistently has a negative impact on monoT5-3B, while it consistently benefits instruction-tuned and zero-shot rerankers.\n2. Instruction-tuned rerankers consistently underperform their corresponding zero-shot rerankers, regardless of the set alpha value.\n3. Optimal alpha values for both instruction-tuned and zero-shot rerankers fall within the range of 0.1 to 0.4.\n\nThese results provide further support for the claims presented in our paper. We will also include this post-hoc analysis of the parameter's effect in the appendix.\n***\n### Response to reject reason 3:\n> \"Detailed discussion is lacking in the paper on its empirical observation to analyze why the further instruction fine-tuning degrades the ranking performance if question-generation task is not present in fine-tuning data. It would be interesting to understand the theoretical basis behind such behavior.\"\n\nWe agree this is an unexpected and intriguing result. In fact, we think we have touched on an unexplored and very interesting behavior of LLMs.  \n\nWe have tried many generation examples with Alpaca and LLaMA and find that the instruction-tuned LLMs (Aplaca) are very good at following instructions: they will generate queries with good quality as expected; however, as our experiments shown, Instruction-tuning seems will hurt the token likelihood distribution of LLMs – it downgrades the quality of next token likelihood estimation. On the other hand, pretrained-only LLMs (LLaMA) are not good at following instructions – if no in-context examples are given, it tends to generate unexpected text, for example, it tends to just complete the prompt or generate unexpected format of content. However, our experiments show that they are better token likelihoods estimators. \n\nWe do have a hypothesis of why further instruction-tuning hurts the query likelihood estimation: Instruction-tuned models tend to pay more attention to the task instructions and less attention to the input content itself. Although they are good at following instructions in the generation task, the most important information for evaluating query likelihood is in the doc content, thus instruction-tuning hurts query likelihood estimation for LLMs. We will add this rationale to the paper.\n\nWe agree that understanding why this result is so, is a very important topic. We were unable to determine ourselves how this investigation could be answered, and thus wanted to share this result with the community. The aim of this paper was to highlight this issue so that it can be further investigated. We hope our paper will inspire further works on this.\n***\n### Response to reject reason 4:\n> \"Lacks original ideas in the contributions: (i) Main QLM setting is the same as introduced in [1,2]. (ii) The interpolation of relevancy scores with first-stage retriever is taken as [3]. (iii) The prompt-template for few-shot template is taken from [4]. The paper carries out the ranking experiments using some newer LLMs which were not explored in [1,2].\"\n\nWe acknowledge that our work builds upon several previous methods. However, our novel contributions can be summarized in the following three aspects:\n1. Identification and definition of \"flaws\" in the previous method (not genuine zero-shot).\n2. Discovery of an intriguing finding: instruction-tuned LLMs perform worse than pretrained-only LLMs in QLM ranking tasks.\n3. Demonstrate that by employing a simple interpolation method with zero-shot QLM LLM rerankers, we can achieve SOTA zero-shot ranking effectiveness. \n***\n### Response to reject reason 5:\n> \"It would be important to have the experimental results using different types of first-stage retrievers other than BM25 too, in order to validate if the observation that instruction-tuning hinders performance if QG task is not included, is consistent in different settings.\"\n\nWe conducted additional experiments using instruction-tuned LLMs with BM25 + HyDE, a sparse-dense hybrid that serves as a more effective first-stage retriever. The results, including those without instruction-tuned LLMs, are presented below:\n\nZeroshot:\n| Model                  | trec-covid | DBpedia | FiQA  | Robust04 | Avg   |\n|------------------------|------------|---------|-------|----------|-------|\n| StableLM-7B            | 74.2       | 41.8    | 38.0  | 53.2     | 51.8  |\n| Alpaca-7B              | 71.6       | 40.1    | 39.5  | 50.9     | 50.5  |\n| LLaMA-7B               | 72.4       | 45.4    | 46.8  | 57.4     | 55.5  |\n| Falcon-7B-insturciton  | 68.8       | 43.1    | 37.4  | 54.6     | 51.0  |\n| Falcon-7B              | 76.6       | 46.1    | 45.8  | 55.1     | 55.9  |\n\nfew-shot:\n| Model                  | trec-covid | DBpedia | FiQA  | Robust04 | Avg   |\n|------------------------|------------|---------|-------|----------|-------|\n| StableLM-7B | 72.2       | 42.8    | 38.0  | 51.7     | 51.2  |\n| Alpaca-7B | 72.3       | 42.5    | 41.8  | 53.0     | 52.4  |\n| LLaMA-7B    | 77.8       | 47.7    | 50.4  | 59.5     | 58.8  |\n| Falcon-7B-insturciton  | 74.9       | 45.2    | 42.8  | 56.1     | 54.8  |\n| Falcon-7B   | 78.6       | 48.0    | 48.6  | 59.0     | 58.5  |\n\nThe conclusion remains unchanged: instruction-tuned LLMs perform worse than their non-instruction-tuned counterparts in both zero-shot and few-shot settings. This further validates our intriguing finding.\n***\n### Response to reject reason 6:\n> \"The experiments are performed only on a small subset (4 datasets) of BEIR benchmark. It is not sufficient to support the claims made in this paper. Thorough evaluation on a larger set is important to support the paper's claims.\"\n\nDue to limited computational resources and numerous LLMs with various settings to run, and in order to ensure feasibility, we decided to choose the most widely used subset of BEIR that is used in most literature. Despite being a subset, it comprises a total of 1347 queries with deep ranking judgments across 4 distinct domains. We believe that it is sufficient to substantiate our claims.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A key finding of this paper is that the the pre-trained LLMs are strong zero-shot re-rankers without needing additional fine-tuning. They find that additional instruction fine-tuning degrades the ranking performance if question-generation task is not present in fine-tuning data. This insight can be useful in understanding the workings of LLMs. It also shows that LLMs are strong QLMs for zero-shot ranking. The paper benchmarks a wide range of open-source LLMs in its experiments.", "reasons_to_reject": "1. Paper claims that [1] does not meet the \"genuine zero-shot\" scenario a T0 has undergone fine-tuning on question generation (QG) tasks and there is a task leakage to downstream QLM ranking task. Lines 078 to 087. This is one of the key arguments of this paper, but it is not explained/proved in the paper why undergoing fine-tuning on question generation (QG) tasks disqualifies it as \"genuine\" zero-shot approach, nor it is supported by any references\n2. The weights for interpolation strategy is set to 0.2 without conducting any grid search (Line 158). No explanation is provided for the choice of the value for this parameter. The main results for this paper's claims depend on the choice of this parameter. Thus, it is important to find the optimal value of this parameter for different LLMs in different settings. It is possible that the key finding of the paper that the instruction fine-tuning degrades the ranking performance maybe because 0.2 value may be better for non-instruction-tuned LLM and worse for the instruction-tuned LLM, and choosing optimal value of this parameter may carry a fair comparison.\n3. Detailed discussion is lacking in the paper on its empirical observation to analyze why the further instruction fine-tuning degrades the ranking performance if question-generation task is not present in fine-tuning data. It would be interesting to understand the theoretical basis behind such behavior.\n4. Lacks original ideas in the contributions: (i) Main QLM setting is the same as introduced in [1,2]. (ii) The interpolation of relevancy scores with first-stage retriever is taken as [3]. (iii) The prompt-template for few-shot template is taken from [4]. The paper carries out the ranking experiments using some newer LLMs which were not explored in [1,2].\n5. It would be important to have the experimental results using different types of first-stage retrievers other than BM25 too, in order to validate if the observation that instruction-tuning hinders performance if QG task is not included, is consistent in different settings.\n6. The experiments are performed only on a small subset (4 datasets) of BEIR benchmark. It is not sufficient to support the claims made in this paper. Thorough evaluation on a larger set is important to support the paper's claims.\n\nReferences:\n1. https://aclanthology.org/2022.emnlp-main.249/\n2. https://link.springer.com/chapter/10.1007/978-3-030-72240-1_49\n3. https://dl.acm.org/doi/abs/10.1145/3471158.3472233\n4. https://arxiv.org/pdf/2202.05144.pdf", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates the effectiveness of using pre-trained LLMs that are not instruction fine-tuned as Query Likelihood Models (QLMs) for zero-shot document ranking task. They empirically show that the additional instruction fine-tuning of LLMs degrades their ranking performance unless question generation task is present in fine-tuning dataset. They follow the settings from [1,2] to use LLMs as zero-shot QLM re-ranker and interpolate the first-stage retriever scores with QLM scores inspired by [3]. They also show that using the hybrid zero-shot first-stage retrieval system improves the performance further. They explore the few-shot QLM re-ranker setting following the prompt template introduced in [4].\n\nReferences:\n1. https://aclanthology.org/2022.emnlp-main.249/\n2. https://link.springer.com/chapter/10.1007/978-3-030-72240-1_49\n3. https://dl.acm.org/doi/abs/10.1145/3471158.3472233\n4. https://arxiv.org/pdf/2202.05144.pdf", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. SimLM (https://arxiv.org/abs/2207.02578) can be added as a zero-shot first-stage retrieval method."}, "extra_text": {}}, {"review_id": "kDLG9Sx1W5", "reviewer": "Reviewer_Zpce", "summary": "", "questions": "Why not include other publicly available BEIR benchmarks in the evaluation? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging that our work is well-motivated.\n***\n### Response to reviewer remark1:\n> \"I found the conclusions from the experimental results are confusing, since the trend looks very mixed.\"\n\nWe believe our results are considerably consistent. Our main claim, that instruction-tuned LLMs are inferior to their pretrained-only base LLMs, is supported by the results of the following LLM pairs: (LLaMA-7B, Alpaca-7B), (LLaMA-13B, StableVicuna-13B), (Falcon-7B, Falcon-7B-instruct), and (Falcon-40B, Falcon-40B-instruct). In total, we have 16 pairs across 4 datasets; only 2 out of 16 do not support the claim (specifically, the LLaMA-13B, and StableVicuna-13B pair on TRECC and DBpedia). Moreover, the best zero-shot and few-shot performance is achieved by pretrained-only LLMs. Additionally, the additional results in our response to reviewer #2 further support our claims. Thus, we believe our results are sufficient to uphold our claims\n***\n### Response to reviewer remark2:\n> \"It also looks very strange that interpolating with first-stage retrievers worsens the performance, and I am very curious about why.\"\n\nWe assume this refers to our results of fine-tuned models (monoT5-3B and monoT5-3B-InPars-v2) in table2, (also our additional results in reviewer#2 remark2’s response hold the same trend). In fact, our results also align with a previous study (https://arxiv.org/pdf/2010.06467.pdf , figure 10). We believe this is due to the fact that these rerankers are fine-tuned on hard negatives sampled from BM25, thus they have been exposed to documents retrieved from the first-stage retriever during training and have learned how to rank BM25 ranking. Hence, the BM25 scores have limited help in this situation.\n\n***\n### Response to reviewer remark3:\n> \"Table 3 compares zero-shot and few-shot setup with a better first-stage retrieval, but the few-shot rerankers seem to work better than zero-shot, which makes the claim on “comparable performance” a bit misleading.\"\n\nWe apologize for the confusion. In our paper, we did not intend to convey that few-shot rerankers have comparable performance to zero-shot rerankers. Rather, we meant to highlight that our tested LLMs exhibit comparable or higher performance to PROMPTAGATOR on similar datasets, both in zero-shot and few-shot settings. We will ensure that our points are clarified in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Please see the main review.", "reasons_to_reject": "Please see the main review.", "questions_to_authors": "Why not include other publicly available BEIR benchmarks in the evaluation? ", "paper_topic_and_main_contributions": "This work studies the zero-shot ranking capability of LLMs pre-trained on unsupervised text. Specifically, this work looks at LLMs that are query likelihood models (QLMs): estimate the probability that the given document can answer the given query, and those probability scores are used to produce a ranking over candidate documents given the query. The motivation is to study QLMs’ zero-shot ability with prompting, addressing that some LLMs are not truly zero-shot due to the possible presence of question generation datasets in the pre-training. \n\nExperiments are to compare with a number of instruction-tuned LLM-based retrievers and rerankers, as well as some few-shot models, on 4 BEIR datasets. I found the conclusions from the experimental results are confusing, since the trend looks very mixed. It also looks very strange that interpolating with first-stage retrievers worsens the performance, and I am very curious about why. Table 3 compares zero-shot and few-shot setup with a better first-stage retrieval, but the few-shot rerankers seem to work better than zero-shot, which makes the claim on “comparable performance” a bit misleading. \n\nOverall, the work is well-motivated in terms of studying zero-shot performance with models that are not pre-trained on related tasks, but the experiments are less supportive of their claims in terms of general claims and performance trends. \n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BAtTGpz1WO", "reviewer": "Reviewer_ia7W", "summary": "", "questions": "Address the concerns in \"Reasons To Reject\"", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging our surprising findings and think our experiments provide sufficient support for all of its claims/arguments.\n***\n### Response to reject reason 1:\n> \"One weakness of the paper is the limited focus on open-source LLMs, which restricts the applicability of the proposed QLM method. The paper acknowledges that access to the model output logits is required for practical implementation, which limits the method to open-source LLMs. This limitation overlooks the potential of utilizing powerful close-source models, such as GPT-4, which could meet the requirements of other approaches like Inpars and PROMPTAGATOR.\"\n\nIn fact, theoretically, our method can be applied to any LLMs. The only reason our work is currently limited to open-sourced LLMs is that the required API is not available yet. This limitation is beyond our control. We hope that our work will encourage the NLP community to embrace open-sourcing such models. As demonstrated by our work, open-sourcing can lead to many more opportunities for research.\n***\n### Response to reject reason 2:\n> \"Another weakness is the lack of a complete statistical significance analysis of the results due to space limitations. While the paper includes a detailed analysis in the appendix, the unavailability of run files for some models from previous works prevents statistical comparisons with the runs produced in this study. This limitation hinders a comprehensive evaluation and comparison of the proposed approach with previous methods.\"\n\nWe tried our best to acquire baseline run files to perform statistical significance tests against our method. Regrettably, we encountered difficulties in obtaining run files for a few baselines. This was due to either unavailability of the model checkpoint or the code implementation not being publicly accessible at this time. Once again, we hope that the community will increasingly embrace open-sourcing by sharing both models and code. Upon acceptance, we will promptly make our code and comprehensive instructions available to the public, enabling full reproducibility of our results.\n***\n### Response to reject reason 3:\n> \"The paper does not thoroughly address the authenticity of the zero-shot setting in the pre-training data of LLaMA and Falcon. Although the paper reassures that no QG data was used in the pre-training of these models, it suggests the need for a thorough analysis to identify any text snippets that may serve as instructions and labels for the QG.\"\n\nThe LLaMA and Falcon models are pre-trained on unstructured, free text. It is improbable that the pre-training data will contain numerous text snippets structured as question-generation instruction data by random chance. Although this possibility does exist, the portion of this kind of data is definitely extremely small. On the other hand, T0 and Flan-T5 have a subset of fine-tuning data that specifically instructs the model to do the question generation. Thus, as we argued in the paper, they cannot be considered as zero-shot QLM rankers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides a comprehensive study on the zero-shot ranking ability of advanced language models (LLMs) converted into query language models (QLMs). This study highlights the remarkable zero-shot ranking effectiveness of LLM-based QLMs without the need for additional instruction fine-tuning. This insight is important and overlooked in previous studies.\n\n2. The paper demonstrates that by integrating LLM-based QLMs with a hybrid zero-shot retriever, a novel state-of-the-art ranking pipeline can be obtained that excels in both zero-shot and few-shot scenarios. This showcases the effectiveness and versatility of LLM-based QLMs.\n\n3. The paper presents key findings that reveal the competitive effectiveness of zero-shot QLMs and QG fine-tuned QLMs. This finding is surprising considering that QG fine-tuned QLMs are explicitly trained on QG tasks, making them a form of transfer learning. The paper suggests that pre-trained-only models such as LLaMA and Falcon possess strong zero-shot QLM ranking capabilities.\n\nOverall, the paper contributes valuable insights into the zero-shot ranking ability of LLM-based QLMs.", "reasons_to_reject": "1. One weakness of the paper is the limited focus on open-source LLMs, which restricts the applicability of the proposed QLM method. The paper acknowledges that access to the model output logits is required for practical implementation, which limits the method to open-source LLMs. This limitation overlooks the potential of utilizing powerful close-source models, such as GPT-4, which could meet the requirements of other approaches like Inpars and PROMPTAGATOR.\n\n2. Another weakness is the lack of a complete statistical significance analysis of the results due to space limitations. While the paper includes a detailed analysis in the appendix, the unavailability of run files for some models from previous works prevents statistical comparisons with the runs produced in this study. This limitation hinders a comprehensive evaluation and comparison of the proposed approach with previous methods.\n\n3. The paper does not thoroughly address the authenticity of the zero-shot setting in the pre-training data of LLaMA and Falcon. Although the paper reassures that no QG data was used in the pre-training of these models, it suggests the need for a thorough analysis to identify any text snippets that may serve as instructions and labels for the QG.", "questions_to_authors": "Address the concerns in \"Reasons To Reject\"", "paper_topic_and_main_contributions": "This paper investigates the zero-shot ranking effectiveness of large language models (LLMs) in information retrieval. It finds that LLMs have robust zero-shot ranking abilities without fine-tuning. The paper proposes a ranking system that combines LLM-based query language models (QLMs) with a hybrid zero-shot retriever, achieving state-of-the-art results. Limitations include the requirement for open-source LLMs and the need for further scrutiny of training data for genuine zero-shot settings.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This short paper offers a series of experiments with open-source LLM for document re-ranking. Some surprising results are obtained, such as instruction-tuning degrades the retrieval effectiveness. While the phenomenon is observed, the paper falls short in providing plausible explanations. There is thus limited insight about the observations.

The paper could be published for the surprising results obtained.
Decision: Accept-Findings