Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
Authors: Weiwei Sun
Lingyong Yan
Xinyu Ma
Shuaiqiang Wang
Pengjie Ren
Zhumin Chen
Dawei Yin
Zhaochun Ren
Venue: EMNLP 2023 Main (2023)
Keywords: Passage Re-ranking
Information Retrieval
Large Language Models
TL;DR: This paper investigates passage re-ranking with LLMs, proposes a new test set, and introduces a permutation distillation method for LLM specialization.

Abstract: Large Language Models (LLMs) have demonstrated remarkable zero-shot generalization across various language-related tasks, including search engines. 
However, existing work utilizes the generative ability of LLMs for Information Retrieval (IR) rather than direct passage ranking.
The discrepancy between the pre-training objectives of LLMs and the ranking objective poses another challenge.
In this paper, we first investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR. Surprisingly, our experiments reveal that properly instructed LLMs can deliver competitive, even superior results to state-of-the-art supervised methods on popular IR benchmarks. 
Furthermore, to address concerns about data contamination of LLMs, we collect a new test set called NovelEval, based on the latest knowledge and aiming to verify the model's ability to rank unknown knowledge. 
Finally, to improve efficiency in real-world applications, we delve into the potential for distilling the ranking capabilities of ChatGPT into small specialized models using a permutation distillation scheme.
Our evaluation results turn out that a distilled 440M model outperforms a 3B supervised model on the BEIR benchmark.
The code to reproduce our results is available at www.github.com/sunnweiwei/RankGPT.

Is ChatGPT Good at Search?
Investigating Large Language Models as Re-Ranking Agents
Weiwei Sun!*
Lingyong Yan? Xinyu Ma² Shuaiqiang Wang
Pengjie Ren'
Zhumin Chen' Dawei Yin Zhaochun Ren
"Shandong University, Qingdao, China
Baidu Inc., Beijing, China 'Leiden University, Leiden, The Netherlands
http/eewwwwiingyooneeee.. xinyuma20111111 6, shqi wang]]emmill. com yindawe @acm org
(renpengjie, chtpriieeecchhhminnnneeeeeee edu. cn
z. reneliacc, wei.aauiiiiiiiiiei
Abstract
Average nDCG@10
75
Large Language Models (LLMS) have demon- strated remarkable zero-shot generalization across various language-related tasks, includ- ing search engines. However, existing work utilizes the generative ability of LLMs for In- formation Retrieval (IR) rather than direct pas- sage ranking. The discrepancy between the pre- training objectives of L.LSs and the ranking objective poses another challenge. In this pa- per, we first investigate generative LLMs such ChatGPT and GPT-4 for relevance ranking in IR. Surprisingly, our experiments reveal that properly instructed LLMs can deliver compet- itive, superior results to state-of-the-art supervised methods on popular IR benchmarks. Furthermore, to address concerns about data contamination of LLMs, we collect new test set called NovelEval, based on the latest knowl- edge and aiming to verify the model's ability to rank unknown knowledge. Finally, to im- prove efficiency real-world applications, we delve into the potential for distilling the rank- ing capabilities of ChatGPT into small special- ized models using a permutation distillation scheme. Our evaluation results turn out that a distilled 440M model outperforms a 3B su- pervised model the BEIR benchmark. The code to reproduce our results is available at www. gi thub. cww/ wttp///nnwwwwwei.m wwt.///kG...//............
73.1
65
84.4
$2.9
60.3
55
156.0
51.4
49.3
45
43.4
38.1
35
TREC
BEIR
Mr.TyDi
BM25
SOTA sup.
ChatGPT
GPT-4
Figure Average results of ChatGPT and GPT-4 (zero-shot) on passage re-ranking benchmarks (TREC. BEIR, and Mr.TyDi), compared with BM25 and previous best-supervised systems (SOTA sup., e.g.. monoT5 (Nogueira et al., 2020)).
As one of the most successful Al applications. Information Retrieval (IR) systems satisfy user re- quirements through several pipelined sub-modules, such as passage retrieval and re-ranking (Lin et al., 2020). Most previous methods heavily rely on manual supervision signals, which require signifi- cant human effort and demonstrate weak generaliz- ability (Campos et al., 2016; Izacard et al., 2022). Therefore, there growing interest in leveraging the zero-shot language understanding and reason- ing capabilities of L.LMs in the area. However, most existing approaches primarily focus on ex- ploiting LLMs for content generation (e.g., query or passage) rather than relevance ranking for groups of passages (Yu et al., 2023; Microsoft, 2023).
1 Introduction
Large Language Models (LLMs), such as Chat- GPT and GPT-4 (OpenAI, 2022, 2023), are revolu- tionizing natural language processing with strong zero-shot and few-shot generalization. By pre- training on large-scale text corpora and alignment fine-tuning to follow human instructions, LLMs have demonstrated their superior capabilities in lan- guage understanding, generation, interaction, and reasoning (Ouyang al., 2022).
Compared to the common generation settings, the objectives of relevance re-ranking vary signifi- cantly from those of LLMss: the re-ranking agents need comprehend user requirements, globally compare, and rank the passages based on their rele- vance to queries. Therefore, leveraging the LLMs' capabilities for passage re-ranking remains a chal- lenging and unanswered task.
"Work done during an internship at Baidu. "oorrepponding authors.
To this end, we focus on the following questions:

---

(RQ1) How does ChatGPT perform on pas- sage re-ranking tasks?
We examine instructional methods for LLMs on passage re-ranking tasks and introduce a novel permutation generation approach: See Section 3 for details.
(RQ2) How can we imitate the ranking capa-
bilities of ChatGPT in smaller, specialized
We comprehensively evaluate ChatGPT and GPT-4 on various passage re-ranking bench- marks, including newly proposed NovelEvvl test set; See Section 5 for details.
model?
To answer the first question, we investigate prompting ChatGPT with two existing strate- gies (Sachan et al., 2022; Liang et al., 2022). However, we observe that they have limited per- formance and heavily rely on the availability of the log-probability of model output. Thus, we propose alternative instructional permutation generation approach, instructing the LLMs to di- rectly output the permutations of a group of pas- sages. addition, we propose an effective sliding window strategy to address context length limita- tions. For a comprehensive evaluation of LLMs, we employ three well-cstablished IR benchmarks: TREC (Craswell et al., 2020), BEIR (Thakur et al., 2021), and Ty.TyDi (Zhang et al., 2021). Further- more, to assess the LLMs unknown knowledge and address concerns of data contamination. we suggest collecting a continuously updated evalua- tion testbed and propose NovelEval, new test set with 21 novel questions.
We propose distillation approach for learn- ing specialized models with the permutation generated by ChatGPT; See Section 4 for de- tails.
2 Related Work
2.1 Information Retrieval with LLMs
Recently, large language models (LLMs) have found increasing applications in information re- trieval (Zhu et al., 2023). Several approaches have been proposed to utilize LLMs for passage retrieval. For example, SGPT (Muennighoff, 2022) generates text embeddings using GPT, generative document retrieval explores differenniablee search index (Tay et al., 2022; Cao et al., 2021: Sun et al., 2023), and HyDE (Gao et al., 2023; Wang et al., 2023a) generates pseudo-documents using GPT-3. In ad- dition, LLMs have also been used for passage re- ranking tasks. UPR (Sachan et al., 2022) and SGPT- CE (Muennighoff,, 2022) introduce instructional query generation methods, while HELM (Liang et al., 2022) utilizes instructional relevance gener- ation. LLMs are also employed for training data generation. InPars (Bonifacio et al., 2022) gener- ates pseudo-queries using GPT-3, and Promptaga- tor (Dai et al., 2023) proposes few-shot dense retrieval to leverage few demonstrations from the target domain for pseudo-query generation. Fur- thermore, LLMs have been used for content gener- ation (Yu ct al., 2023) and web browsing (Nakano et al., 2021; Izacard et al., 2023; Microsoft, 2023). In this paper, we explore using ChatGPT and GPT- 4 in passage re-ranking tasks, propose an instruc- tional permutation generation method, and conduct comprehensive evaluation of benchmarks from various domains, tasks, and languages. Recent work (Ma et al., 2023) concurrently investigated listwise passage re-ranking using LLMs. In com- parison, our study provides a more comprehensive evaluation, incorporating newly annotated dataset, and validates the proposed permutation distillation technique.
To answer the second question, we introduce a permutation distillation technique to imitate the passage ranking capabilities of ChatGPT in smaller, specialized ranking model. Specifically, we randomly sample 10K queries from the MS MARCO training set, and each query retrieved by BM25 with 20 candidate passages. On this basis, we distill the permutation predicted by Chat- GPT into a student model using a RankNet-based distillation objective (Burges et al., 2005).
Our evaluation results demonstrate that GPT-4, equipped with zero-shot instructional permutation generation, surpasses supervised systems across nearly all datasets. Figure illustrates that GPT-4 outperforms the previous tftteoof-the-art models by an average nDCG improvement of 2.7, 2.3, and 2.7 on TREC, BEIR, and My.TyDi, respectively. Furthermore, GPT-4 achieves state-of-the-art per- formance on the new NovelEval test set. Through our permutation distillation experiments, we ob- serve that 435M student model outperforms the previous oatt--oftthearr monoT5 (3B) model by an average nDCG improvement of 1.67 on BEIR. Ad- ditionally, the proposed distillation method demon- strates cost-efficiency benefits.
In summary, our contributions are tri-fold:

---

Please write question based on this pessage.
Passage: {(passage))
Step 1
P1
p2
p3
P4
p5
p8
p7
p8
Query: ((query))
Passage: ((passage))
Does the passage anawer query?
Step 2
pp
p2
p3
p4
p8
p6
p7
Query:
((query))
Yes (or No)
Step 3
p1
p2
p8
p3
p4
p5
p8
p7
(a) Query generation
(b) Relevance generation
Ranking results
p2
p8
p1
p3
p4
p5
p6
p7
The following are passages related to query ({query))
Illustration of re-ranking 8 passages using sliding windows with window size of 4 and step size of The blue color represents the first two windows, while the yellow color represents the last window. The sliding windows are applied in back-to-first order, mean- ing that the first 2 passages in the previous window will participate in re-ranking the next window.
Figure 3:
[1] [[passage. 1}}
[2] {[passage_ 2]}
(more passages)
Rank these passages based on their relevance to the query.
[2] [3] [1]> [..]]
(c) Permutation generation
Three types instructions for zero-shot pas- sage re-ranking with LLMs. The gray and yellow blocks indicate the imputs and outputs of the model. (a) Query generation relies the log probability of LLMs gen- crate the query based on the passage. (b) Relevance generation instructs LLMss to output relevance judg- ments. (c) Permutation generation generates ranked list of group of passages. See Appendix A for details.
Figure 2:
corpus, and the re-ranking stage aims to re-rank this set to output more precise list. Recent stud- ies have explored LLMs for zero-shot re-ranking, such instructional query generation relevance generation (Sachan al., 2022: Liang et al., 2022). However, existing methods have limited perfor- mance re-ranking and heavily rely on the avail- ability the log probability of model output and thus cannot be applied to the latest LLMs such as GPT-4. Since ChatGPT and GPT-4 have a strong capacity for text understanding, instruction follow- ing, and reasoning, we introduce novel instruc- tional permutation generation method with slid- ing window strategy to directly output ranked list given a set of candidate passages. Figure 2 illustrates examples of three types of instructions; all the detailed instructions are included Ap- pendix A.
2.2 LLMs Specialization
Despite their impressive capabilities. LLMs such as GPT-4 often come with high costs and lack open- source availability. As result, considerable re- search has explored ways to distill the capabilities of LLMs into specialized, custom models. For in- stance, Fu et al. (2023) and Magister al. (2023) have successfully distilled the reasoning ability of LLMs into smaller models. Self-instruce (Wang et al., 2023b; Taori et al., 2023) propose iterative approaches to distill GPT-3 using their outputs. Ad- ditionally, Sachan et al. (2023) and Shi et al. (2023) utilize the generation probability of LLMs to im- prove retrieval systems. This paper presents per- mutation distillation method that leverages Chat- GPT as teacher to obtain specialized re-ranking models. Our experiments demonstrate that even with a small amount of ChatGPT-generated data, the specialized model can outperform strong super- vised systems.
3.1 Instructional Permutation Generation
As illustrated in Figure 2 (c), our approach involves inputting a group of passages into the LLMs, each identified by a unique identifier (e.g.. [1], [2]. etc.). We then ask the LLMs to generate the per- mutation of passages in descending order based on their relevance to the query. The passages are ranked using the identifiers, in a format such as [2] > [3] [1] > etc. The proposed method ranks pas- sages directly without producing intermediate relevance score.
3 Passage Re-Ranking with LLMs
Ranking is the core task in information retrieval ap- plications, such as ad-hoc search (Lin et al., 2020; Fan al., 2022), Web search (Zou al., 2021). and open-domain question answering (Karpukhin et al., 2020). Modern IR systems generally employ a multi-stage pipeline where the retrieval stage fo- cuses on retrieving a set of candidates from a large
3.2 Sliding Window Strategy
Due to the token limitations of LLMs, we can only rank limited number of passages using the per- mutation generation approach. To overcome this constraint, we propose a sliding window strategy. Figure 3 illustrates an example of re-ranking pas-

---

sages using sliding window. Suppose the first- stage retrieval model returns M passages. We re- rank these passages in a back-to-first order using a sliding window. This strategy involves two hyper- parameters: window size (w) and step size (s). We first use the LLMs to rank the passages from the (M - w--w-thh w)tt to the M-th. Then, we slide the win- dow in steps of and re-rank the passages within the range from the (M
parameters 0 to calculate the relevance score s; of paired (g, p,) using cross-encoder. Using the per- mutation R generated by ChatGPT, we consider RankNet loss (Burges et al., 2005) to optimize the student model:
RannNNtt ΣΣ Anery log(1 exp(s; - -s)))
s)-th to the (M s)- th. This process is repeated until all passages have been re-ranked.
RankNet is pairwise loss that measures the cor-
rectness of relative passage orders. When using permutations generated by ChatGPT, we can con- struct (M 1)/2 pairs.
4 Specialization by Permutation Distillation
4.3 Specialized Model Architecture
Although ChatGPT and GPT-4 are highly capable, they are also too expensive deploy in commercial search systems. Using GPT-4 to re-rank passages will greatly incrcase the latency of the scarch sys- tem. In addition, large language models suffer from the problem of unstable generation. Therefore, we argue that the capabilities of large language models are redundant for the re-ranking task. Thus, we can distill the re-ranking capability of large language models into small model by specialization.
Regarding the architecture of the specialized model, consider two model structures: the BERT-like model and the GPT-like model.
4.3.1 BERT-like model.
We utilize a cross-encoder model (Nogucira and Cho, 2019) based on DeBERTa-large. It concate- nates the query and passage with a [SEP] token and estimates relevance using the representation of the [CLS] token.
4.3.2 GPT-like model.
4.1 Permutation Distillation
We utilize the LLaMA-7B (Touvron et al., 2023) with a zero-shot relevance generation instruction (see Appendix A). classifies the query and pas- sage as relevance or irrelevance by generating a relevance token. The relevance score is then de- fined as the generation probability of the relevance token.
In this paper, we present novel permutation dis- tillation method that aims distill the passage re- ranking capability of ChatGPT into a specialized model. The key difference between our approach and previous distillation methods is that we directly use the model-genee -enerreee permutation as the tar- get, without introducing any inductive bias such as probaabiyyyy manipula- tion (Bonifacio et al., 2022; Sachan et al., 2023). To achieve this, we sample 10,000 queries from MS MARCO and retrieve 20 candidate passages using BM25 for each query. The objective of distil- lation aims to reduce the differences between the permutation outputs of the student and ChatGPT.
Figure 5 illustrates the structure of the two types of specialized models.
consistency-checkee
Datasets
Our experiments are conducted on three benchmark datasets and one newly collected test set Novellival.
5.1 Benchmark Datasets
4.2 Training Objective
The benchmark datasets include, TREC- DL (Craswell et al., 2020), BEIR (Thakur et al., 2021), and Mr.TyDi (Zhang al., 2021).
Formally, suppose we have query and M pas- sages (pi...., Par) retrieved by BM25 (M - 20 in our implementation). ChatGPT with instruc- tional permutation generation could produce the ranking results of the M passages, denoted as R (Ti)...,TT)) where r; [1.2.... M] is the rank of the passage P;- For example, r; = means Pt ranks third among the M passages. Now we have a specialized model s; = fo(g.P) P:) with
TREC is widely used benchmark dataset in IR research. We the test sets of the 2019 and 2020 competitions: (i) TREC-DL19 contains 43 queries, (ii) TREC-DL20 contains 54 queries.
BEIR consists of diverse retrieval tasks and do- mains. We choose eight tasks in BEIR to evaluate the models: (i) Covid: retrieves scientific articles

---

for COVID-19 related questions. (ii) NFCorpus is a bio-medical IR data. (ii)) Touche is an argument retrieval datasets. (iv) DBPedia retrieves entities from DBpedia corpus. (v) SciFact retrieves evi- dence for claims verification. (vi) Signal retrieves relevant tweets for a given news title. (vii) News retrieves relevant news articles for news headlines. (viii) Robust04 evaluates poorly performing topics.
6 Experimental Results of LLMs
6.1 Implementation and Metries
In benchmark datasets, we re-rank the top-100 pas- sages retrieved by BM25 using pyserini' and use nDCG@ (1, 5,10) as evaluation metrics. Since ChatGPT cannot manage 100 passages at a time, we use the sliding window strategy introduced in Section 3.2 with a window size of 20 and step size 10. In NovelEval, we randomly shuffled the 20 candidate passages searched by Google and re-ranked them using ChatGPT and GPT-4 with permutation generation.
Mr.TyDi is a multilingual passages retrieval dataset of ten low-resource languages: Arabic, Bengali, Finnish, Indonesian, Japanese, Korean, Russian, Swahili, Telugu, and Thai. We use the first 100 samples in the test set of each language.
6.2 Results on Benchmarks
On benchmarks, we compare ChatGPT and GPT-4 with state-of-the-art supervised and unsupervised passage re-ranking methods. The supervised base- lines include: monoBERT (Nogueira and Cho, 2019), monoT5 (Nogueira et al., 2020), mmar- coCE (Bonifacio et al., 2021), and Cohere Rerank 2. The unsupervised baselines include: UPR (Sachan et al., 2022), InPars (Bonifacio et al., 2022), and Promptagator++ (Dai et al., 2023). See Appendix E for more details on implementing the baseline.
5.2 A New Test Set NovelEval
The questions in the current benchmark dataset are typically gathered years ago, which raises the issue that existing LLMs already possess knowl- edge of these questions (Yu al., 2023). Further- more, since many LLMs do not disclose informa- tion about their training data, there is a potential risk of contamination of the existing benchmark test set (OpenAI, 2023). However, re-ranking models are expected to possess the capability to comprehend, deduce, and rank knowledge that is inherently unknown them. Therefore, we sug- gest constructing continuously updated IR test sets to ensure that the questions, passages to be ranked, and relevance annotations have not been learned by the latest LLMs for a fair evaluation.
Table 1 presents the evaluation results obtained from the TREC and BERR datasets. The follow- ing observations can be made: (i) GPT-4, when equipped with the permutation generation instruc- tion, demonstrates superior performance both datasets. Notably, GPT-4 achieves an average im- provement of 2.7 and 2.3 in nDCG@ 10 on TREC and BEIR, respectively, compared to monoT5 (3B). (ii) ChatGPT also exhibits impressive results on the BEIR dataset, surpassing the majority of super- vised baselines. (iii) On BEIR, we use only GPT-4 to re-rank the top-30 passages re-ranked by Chat- GPT. The method achieves good results, while the cost is only 1/5 of that of only using GPT-4 for re-ranking.
As an initial effort, we built NovelEval-2306.
novel test set with 21 novel questions. This test set is constructed by gathering questions and pas- sages from 4 domains that were published after the release of GPT-4. To ensure that GPT-4 did not possess prior knowledge of these questions, we pre- sented them to both gpt-4-0314 and gpt-4-0613. For instance, question "Which film was the 2023 Palme O'Or winner?" pertains to the Cannes Film Festival that took place on May 27, 2023, render- ing its answer inaccessible to most existing LLMs. Next, we searched 20 candidate passages for each question using Google search. The relevance of these passages was manually labeled as: for not relevant, 1 for partially relevant, and for relevant. See Appendix C for more details.
Table 2 illustrates the results on Mr. TyDi of ten low-resource languages. Overall, GPT-4 outper- forms the supervised system in most languages, demonstrating an average improvement of 2.65 nDCG over mmarcoCF.. However, there are in- stances where GPT-4 performs worse than mmar- COCE, particularly in low-resource languages like Bengali, Telugu, and Thai. This may be attributed to the weaker language modeling ability of GPT-4
'https://giiti com//cattornni/pysernni phttps://txt.. cohere. com/rerank/

---

Method
DL19
DL20 Covid NFCorpus Touche DBPedia Scifact
Signal
News Robust04 [EIR (Avg)
BM25
50.58 47.96 59.47
30.75
44.22
31.80
67.89
33.05
39.52
40.70
43.42
Supervised
monoBERT (340M)
70.50
67.28
70.01
36.88
31.75
41.87
71.36
31.44
44.62
49.35
47.16
monoT5 (220M)
71.48 66.99
78.34
37.38
30.82
42.42
73.40
31.67
46.83
51.72
49.07
monoT5 (3B)
71.83
68.89
80.71
38.97
32.41
44.45
76.57
32.55
48.49
56.71
51,36
Cohere Rerank-v2
73.22
67.08
81.81
36.36
32,51
42.51
74.44
29.60
47.59
50.78
49.45
Unsupervised
UPR (FLAN-TS-XL)
53.85
56.02
68.11
35.04
19.69
30.91
72.69
31.91
43.11
42.43
42.99
InPars (5-30
66.12
78.35
Promptagator++ (few-shot)
76.2
38.1
I.IM API (Permutation generation)
3pt-3.5-ture gpt-41
65.80
62.91
76.67
35.62
36.18
44.47
70.43
32.12
48.85
50.62
49.37
75.59 70.56
85.51
38.47
38.57
47.12
74.95
34.40
52.89
57.55
53.68
Table I: Results (nDCG@ 10) on TREC and BEIR. Best performing unsupervised and overall system(s) are marked bold. All models except InPars and Promptagator++ re-rank the same BM25 top- top-100 passages. *On BEIR. we use gpt-4 to re-rank the top-30 passages re-ranked by gpt-3. 5- turbo to reduce the cost of calling gpt-4 API.
Method
BM25
mmarcoCF.
gpt-3.5
+gpt-4
Method
nDCG@1
nDCG@5
nDCG@
Arabic
39.19
68.18
71.00
72.56
BM25
33.33
45.96
55.77
Bengali
45.56
65.98
53.10
64.37
MonoBERT (340M)
78.57
70.65
77.27
Finnish
29.91
54.15
56.48
62.29
monoT5 (220M)
83.33
77.46
81,27
Indonesian
51.79
69.94
68.45
75.47
monoT5 (3B)
83.33
78.38
84,62
Japanese
27.39
49,80
50.70
58.22
Korean
26.29
44.00
41.48
49.63
gpt-3.5-turee
76.19 85.71
74.15 87.49
75.71
Russian
34.04
53.16
48.75
53.45
gpt-4
90.45
Swahili
45.15
60.31
62.38
67.67
Telugu
37.05
68.92
51.69
62.22
Table 3: Results on NovelEval.
Thai
44.62
68.36
55.57
63.41
Avg
38.10
60.28
55.96
62.93
DL19 nDCG@ 1/5/10
DL20
Table 2: Results (nDCG@ 10) on Mr.TyDi.
Method
nDCG@ 1/5/10
cur ierie-001
RG 39.53 40.02 / 41.53 41.553444////1 34.80 34,91
cur 1e-801
QG 50.78/ 50.77 49. 76 50.00/ 48.36 48.773
in these languages and the fact that text in low- resource languages tends to consume more tokens than English text, leading to the over-cropping of observed with Chat- GPT, which is on par with the supervised system in most languages, and consistently trails behind GPT-4 in all languages.
cur ie-801
PG 66.67 / 56.79 / 54.21 59.57 55.20 $2.17
davinci-803 RG 54,26 52.78/ 50.58 64.20/ / 58.41 41/4/1// 56.87
davinci-803 QG 37.66 37.60/ 44.73 45.37 51.25 47.46 45.93
davinci-803 PG 69.77 64.73 61.50
69.75 58.76/ / $7.05
passages. Similar trends
gpt-3.5 5
PG 82.17 / 71.15 65.80 79.32/ 66.76 62.91
gpt-4 PG 82.56 / 79.16 75.59 78.40/ 74.11 70.56
Table 4: Compare different instruction and API end- point. Best performing system(s) are marked bold. PG, QG, RG denote permutation generation, query genera- tion and relevance generation, respectively.
6.3 Results NovelEval
Table illustrates the evaluation results on our newly collected NovelEval, test set containing 21 novel questions and 420 passages that GPT-4 had not learned. The results show that GPT-4 performs well on these questions, significantly outperform- ing the previous best-supervised method, mono'T5 (3B). Additionally, ChatGPT achieves perfor- mance level comparable to that of monoBERT. This outcome implics that LLMs possess the capability to effectively re-rank unfamiliar information.
6.4 Compare with Different Instructions
We conduct a comparison with the proposed permu- tation generation (PG) with previous query genera- tion (QG) (Sachan et al., 2022) and relevance gen- eration (RG) (Liang al., 2022) TREC-DL19. example of the three types of instructions is in Figure 2, and the detailed implementation is Appendix B. We also compare four LLMss provided

---

Method
nDCG@1
nDCG@s
aDCG@10 10
Initial Passage Order While our standard imple- mentation utilizes the ranking result of BM25 as the initial order, we examined two alternative vari- ants: random order (1) and reversed BM25 order (2). The results reveal that the model's 's performance is highly sensitive to the initial passage order. This could be because BM25 provides a relatively good starting passage order, enabling satisfactory results with only single sliding window re-ranking.
BM25
54.26
52.78
50.58
3pt-3.5-turbo
82.17
71.15
65.80
Initial passage order
(1) Random order
26.36
25.32
25.17
(2) Reverse order
36,43
31.79
32.77
Number of re-ranking
(3) Re-rank 2 times
78.29
69.37
66.62
(4) Re-rank 3 times
78.29
69.74
66.97
(5) gpt-4 Rerank 80.23
76.70
73.64
Table 5: Ablation study TREC-DL19. We 5-turbo 5-turbo with permutation generation with dif- ferent configuration.
Number of Re-Ranking Furthermore, we stud- ied the influence of the number of sliding window passes. Models (3-4) in Table show that re- ranking more times may improve nDCG@ @ 10, but it somehow hurts the ranking performance on top passages (e.g.. nDCG@) decreased by 3.88). Re- ranking the top 30 passages using GPT-4 showed notable accuracy improvements (see the model (5)). This provides an alternative method to combinc ChatGPT and GPT-4 in passage re-ranking to re- duce the high cost of using the GPT-4 model.
the OpenAl API²: curie-001 GPT-3 model with about 6.7 billion parameters (Brown et al., GPT-3.5 model trained with RLHF and about 175 billion parameters (Ouyang et al., 2022); gpt-3.5-tuue 5-turbo The underlying model of ChatGPT (OpenAI, 2022); gpt-4- GPT- 4 model (OpenA1, 2023).
2020); davinci-
The results listed in Table 4. From the results, can see that: (i) The proposed PG method outperforms both QG and RG methods instructing LLMS to re-rank passages. We suggest two explanations: First, from the result that PG has significantly higher top-1 accuracy compared to other methods, we infer that LLMs can explicitly compare multiple passages with PG, allowing subtle differences between passages to be discerned. Second, LLMs gain a more com- prehensive understanding of the query and pas- sages by reading multiple passages with potentially complementary information, thus improving the model's ranking ability. (ii) With PG, ChatGPT performs comparably to GPT-4 on nDCG@1, but lags behind it on n 10. The Davinci model (text-davinci-093) performs poorly compared to ChatGPT and GPT-4. This may be because of the generation stability of Davinci and ChatGPT trails that of GPT-4. We delve into the stability analysis of Davinci, ChatGPT, and GPT-4 in Ap- pendix F.
6.6 Results of LLMs beyond ChatGPT
We further test the capabilities of other LLMs be- yond the OpenAl series TREC DL-19. As shown in Table 6, we evaluate the top-20 BM25 pas- sage re-ranking nDCG of proprietary LLMs from OpenAI, Cohere, Antropic, and Google, and three open-source LLMs. We see that: (i) Among the proprietary LLMs, GPT-4 exhibited the highest re- ranking performance. Cohere Re-rank also showed promising results; however, it should be noted that it is supervised specialized model. In contrast, the proprietary models from Antropic and Google fell behind ChatGPT in terms of re-ranking effec- tiveness. (ii) As for the open-soures LLMs, we ob- served a significant performance gap compared to ChatGPT. One possible reason for this discrepancy could be the complexity involved in generating per- mutations of 20 passages, which seems to pose a challenge for the existing open-source models. We analyze the model's unexpected behavior in Appendix F,, and the cost of API in Appendix H.
Experimental Results of Specialization
6.5 Ablation Study on TREC
As mentioned in Section 4, we randomly sample 10K queries from the MSMARCO training set and employ the proposed permutation distillation dis- till ChatGPT's predicted permutation into special- ized re-ranking models. The specialized re-ranking models could be DeBERTa-v33 Largge with a cross- encoder architecture or LLaMA-7B with relevance
We conducted an ablation study on TREC to gain insights into the detailed configuration of permuta- tion generation. Table 5 illustrates the results.
hhttps://platti form. openai. www.ii..co//aaao com/docs/ api-reference

---

TREC nDCG@10
BEIR 1DC
TREC nDCG@10
BEIR nDCG010
70
70
50
MonoBERT
ChaiGPT
65
Permulittion diatilation
40
Supervised learnng
35
55
30
Log of number parameters
Number training samples
Figure 4: Scaling experiment. The dashed line indicates baseline methods: GPT-4, monoTS, monoBERT, and ChatGPT. The solid green line and solid gray line indicate the specialized Deberta models obtained by the proposed permutation distillation and by supervised learning on MS MARCO, respectively. This figure compares the models' performance on TREC and BEIR across varying model sizes (70M to 435M) and training data sizes (500 to I0K).
Method
ND1
NDS
ND10
parison*.
OpenAI text-davinci-808
70.54
61.90
57.24
7.1
Results on Benchmarks
OpenAl 3pt-3.5-turbo
75.58
66.19
60.89
OpenAl gpt-4
79.46
71.65
65.68
Table lists the results of specialized models, and Table 13 includes the detailed results. Our findings can summarized as follows: (i) Permutation dis- tillation outperforms the supervised counterpart on both TREC and BEIR datasets, potentially because ChatGPT's relevance judgments are more compre- hensive than MS MARCO labels (Arabzadeh et al.. 2021). (ii) The specialized DeBERTa model outper-
Cohere rerank-english-v2 0
79.46
71.56
64.78
Antropic claude-2
66.66
59.33
55.91
Autropic claude-instant-
81.01
66.71
62.23
Google text-bison-801
69.77
64.46
58.67
Google bard-2823. 18
81.01
65.57
60.11
Google flan-t5-xx1 Tsinghua ChatGLM-6B
$2.71
$1.63
50.26
54.26
52.77
50.58
LMSYS Vicuna-138
54.26
51.55
49.08
forms previous statc-of-thh-aar (SOTA) baselines, monoT5 (3B), on BEIR with an average nDCG of 53.03. This result highlights the potential of dis- tilling LLMs for IR since it is significantly more cost-efficient. (iii) The distilled specialized model also surpasses ChatGPT, its teacher model, on both datasets. This is probably because the re-rank- ing stability of specialized models is better than ChatGPT. As shown in the stability analysis in Ap- pendix F, ChatGPT is very unstable in generating permutations.
Table 6: Results of different LLMs on re-ranking ND(1,5,10) denote
top-20 passages DL-19,
nDCG@ 10), respectively.
Label
Method
DL19
DL20
BEIR (Avg)
Ø
BM25
50,58
47.96
43.42
2
ChaiGPT
65,80
62.91
49.37
MARCO
monoT5 (3B)
71.83
68.89
51.36
MARCO
DeBERTa-Large
68.89
61.38
42.64
MARCO
LLaMA-7B
69.24
58.97
47.71
ChatGPT
DeRERTa-Large
70.66
67.15
53.03
ChatGPT LLaMA-7B
71.78
66.89
51.68
7.2 Analysis on Model Size and Data Size
Table 7: Results (nDCG@11 10) of specialized mod- els. Best performing specialized model(s) marked bold. The label column denotes the relevance judge- ments used model training, where MARCO denotes use MS MARCO annotation, ChatGPT denotes use the outputs of permutation generation instructed ChatGPT as labels. BEIR (Avg) denotes average nDCG on eight BEIR datasets, and the detailed results are at Table 13.
In Figure 4, we present the re-ranking performance of specialized DeBERTa models obtained through permutation distillation and supervised learning of different model sizes (ranging from 70M to 435M) and training data sizes (ranging from 500 to 10K). Our findings indicate that the permutation-diisilleed models consistently outperform their supervised counterparts across settings, particularly on the BEIR datasets. Notably, even with only IK training queries, the permutation-diittlled DeBERTa model
generation instructions. We also implemented the specialized model trained using the original MS MARCO labels (aka supervised learning) for com-
"Note that all models are trained using the RankNet loss for a fair comparison.

---

achieves superior performance compared the previous state-of-the-art monoT5 (3B) model on BEIR. We also observe that increasing the number model parameters yields a greater improvement in the ranking results than increasing the training data. Finally, we find that the performance of super- vised models is unstable for different model sizes and data sizes. This may be due to the presence of noise in the MS MARCO labels, which leads to overfitting problems (Arabzadeh et al., 2021).
in terms of publicly accessed datasets and models. Risks and harms of large language models include the generation of harmful, offensive, or biased con- tent. These models are often prone to generating incorrect information, sometimes referred to as hal- lucinations. We do not expect the studied model to be an exception in this regard. The LLMs used in this paper were shown to suffer from bias, halluci- nation, and other problems. Therefore, we are not recommending the use of LLMs for ranking tasks with social implications, such as ranking job can- didates or ranking products, because LLMs may exhibit racial bias, geographical bias, gender bias, etc., in the ranking results. In addition, the use of LLMs in critical decision-making sessions may pose unspecified risks. Finally, the distilled models are licensed under the terms of OpenAI because they use ChatGPT. The distilled LLaMA models are further licensed under the commemmmerlial li- cense of LLaMA.
8 Conclusion
In this paper, we conduct a comprehensive study on passage re-ranking with LLMs. We introduce novel permutation generation approach to fully explore the power of LLMs.. Our experiments three benchmarks have demonstrated the capability of ChatGPT and GPT-4 in passage re-ranking. To further validate LLMs on unfamiliar knowledge, we introduce a new test set called NovelEval. Ad- ditionally, we propose a permutation distillation method, which demonstrates superior effectiveness and efficiency compared to existing supervised ap- proaches.
Acknowledgements
This work was supported by the Natural Sci- ence Foundation of China (62272274, 61972234, 62072279, 62102234, 62202271), the Natu- ral Seience Foundation of Shandong Province (ZR202IQF129, ZR2022QF004), the Key Scien- tific and Technological Innovation Program of Shandong Province 2209922Y001012), the Fun- damental Research Funds of Shandong Univer- sity, the China Scholarship Council under grant nr. 202206220085.
Limitations
The limitations of this work include the main analysis for OpenAI ChatGPT and GPT-4, which are proprictary models that are not open-source. Although we also tested on open-source models such FLAN-T5, ChatGL--6B, and Vicuna- 133B, the results still differ significantly from ChatGPT. How to further exploit the open-source models is a question worth exploring. Additionally, this study solely focuses on examining LLMs in the re-ranking task. Consequently, the upper bound of the ranking effect is contingent upon the recall of the initial passage retrieval. Our findings also in- dicate that the re-ranking effect of LLMs is highly sensitive to the initial order of passages, which is usually determined by the first-stage retrieval, such as BM25. Therefore, there a need for fur- ther exploration into effectively utilizing LLMs to enhance the first-stage retrieval and improve the ro- bustness of LLMs in relation to the initial passage retrieval.
References
Negar Arabzadeh, Alexandra Vtyurina, Xinyi Yan, and Charles L. A. Clarke. 2021. Shallow pooling for sparse labels. Information Retrieval Journal, Journal, 25:365 385.
Luiz Henrique Bonifacio, Hugo Queiroz Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. In- pars: Data augmentation for information retrieval using large language models. In SIGIR 2022.
Luiz Henrique Bonifacio, Israel Campiotti, Roberto de Alencar Lotufo, and Rodrigo Nogueira. 2021. mmarco: A multilingual version of ms marco passage ranking dataset. ArXiv, abs/2108.11 13897.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens
Ethics Statement
We acknowledge the importance of the ACM Code Ethics and totally agree with it. We ensure that this work is compatible with the provided code,

---

Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurlPS 2020.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Towards unsupervised dense information retrieval with contrastive learning. TMLR.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick. Jane A. Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Few-shot learning with retrieval augmented language models. Journal of Machine Learning Re- search, 24(251):1-43.
Sebastian Bruch, Xuanhui Wang, Michael Bendersky, and Mare Najork. 2019. An analysis of the softmax cross entropy loss for learning-to to-rank with binary relevance. In SIGIR 2019.
Christopher C. Burges, Tal Shaked, Frin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory N. Hullender. 2005. Learning to rank using gradient descent. In /CML 2005.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense passage retrieval for open-domain question answering. In FMNLP 2020.
Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. Ms marco: A human generated machine reading compre- hension dataset. ArXiv, 0tbs//611.0926..
Percy Liang. Rishi Bommasani. Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayaman, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang. Christian Cosgrove, Christopher Man- ning, Christopher R c, Diana Acosta-Navas, Drew A. Hudson, E. Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong. Hongyu Ren, Huaxiu Yao, Jue Wang. Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan S. Kim, Neel Guha, Niladri S. Chatterji, Henderson, Qian Huang. Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas F. Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic eval- uation language models. ArXiv, 0hs/2211.0911111
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. ICLR 2021.
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roherts, Denny Zhou, Quoe Le, and Jason Wei. 2022. Scal- ing cettrruttionfiinttuee language models. ArXiv, abs/2210.1 1146.
Khattab, Peter
Jimmy J. Lin, Rodrigo Nogueira, and Andrew Yates. 2020. Pretrained transformers for text ranking: Bert and beyond. In WSDM 2020.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, and Ellen M. Voorhees. 2020. Overview of the tree 2020 deep learning track. ArXiv, abx/1102.07662.
Xueguang Ma. Xinyu Crystina Zhang, Ronak Pradeep. and Jimmy Lin. 2023. Zero-shot listwise docu- ment reranking with a large language model. ArXiv, abs/2305.021 02156.........
Zhuyun Dai, Vincent Zhao, Ji Ma. Yi Luan, Jianmo Ni. Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wes Weig Chang. 2023. Promptagator: Few-shot dense retrieval from examples. In ICLR 2023.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2023. Teaching small language models to reason. In ACL 2023.
Yixing Fan, Xiaohui Xie. Yingiong Cai, Jia Chen, Xinyu Ma, Xiangsheng Li, Ruqing Zhang, and Jiafeng Guo. 2022. Pre-training methods in information retrieval. Foundations and Trends in Information Retrieval, 178-77.
Microsoft. 2023. bing runs on openai's gpt-4, https: //blogs. www..iinnnnn.....eoo.o......e..e comm/searcch//mrrc__22222 Confirmed- tew-new- weee--ee---inn-uee Bing-ruuss-on-OpenAA E2%80%99s 4PT-44
Confirmed: the new
Yao Fu, Hao-Chun Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller lan- guage models towards multi-step reasoning. ICML 2023.
Niklas Muennighoff. 2022. Sgpt: Gpt sentence embed- dings for semantic search. ArXiv, abs/2202.08904.
Reiichiro Nakano, Jacob Hilton, Arun Balaji. Jeff Wu. Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou. Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise zero-shot dense retrieval without rele- vance labels. In ACL 2023.

---

Chess, and John Schulman. 2021. Webgpt: Browser- assisted question-ana -answerrn with human feedback. ArXiv, abs/21 I 12.09332.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' 'erielnen Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 13971.
Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with bert. ArXiv, abs//901 ht//1901.04000 104055.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre-
Liang Wang. Nan Yang, and Furu Wei. 2023a. Query2doc: Query expansion with large language models. ArXiv, abs/2303.07678.
trained sequence-to-sqquene model. In Findings of EMNLP 2020.
Xuanhui Wang. Cheng Li, Nadav Golbandi, Michael Bendersky, and Marc Najork. 2018. The lambdaloss framework for ranking metric optimization. In CIKM 2018.
OpenAL. 2022. Introducing chatgpt. https://openai. com/blog/ chattpt..
OpenAl. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Yizhong Wang. Yeganeh Kordi. Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language model with self generated instructions. In ACL 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton. Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. In NeurlPS 2022.
Wenhao Yu, Dan Iter, Shuohang Wang. Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. ICLR 2023.
Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan. Wen tau Yih, Joëlle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. In EMNLP 2022.
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. tydi: A multi- muuti-linguul benchmark for dense retrieval. In MRL.
Yutao Zhu, Huaying Yuan, Shuting Wang. Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji rong Wen. 2023. Large language models for infor- mation retrieval: A survey. ArXiv, abs/2308.07 107.
Devendra Singh Sachan, Mike Lewis, Dani Yogatama, Luke Zettlemoyer, Joëlle Pineau, and Manzil Zaheer. 2023. Questions are all you need to train dense passage retriever. TACL, page 600-616.
Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Daiting Shi, Shuaiqiang Wang. Zhicong Cheng, and Dawei Yin. 2021. Pre-trained language model based ramking in baidu search, In KDD 2021.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Sco. Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. 2023. Replug: Retrieval-augmentt black-hox language models. ArXiv, abs/2301.
Weiwei Sun, Lingyong Yan. Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, M. de Rijke, and Zhaochun Ren. 2023. Learning to tokenize for generative retrieval. In NerulPS 2023.
Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-followee laaaa model. https:// gi thub.
Yi Tay, Vinh Q. Tran, Mostafa Dehghani. Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui. Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer memory as a differentiable search index. In NeurlPS 2022.
Nandan Thakur, Nils Reimers, Andreas Ruckl'e, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: heterogenous benchmark for zero-shot evaluation of information retrieval models. In NewrIPS 2021.

---

A Instructions
A.1 Query Generation Instruction
The query generation instruction (Sachan et al., 2022) uses the log-probability of the query.
Please write question based on this passage.
Passage: [passage)]}
Question: ((query))
A.2
Relevance Generation Instruction (few-shot)
Following HELM (Liang al., 2022), the relevance generation instruction use 4 in-context examples.
Given a passage and a query, predict whether the passage includes an answer to the query by producing either "Yes" or 'No".
Passage: Its 25 drops per ml, you guys are all wrong. If it is water, the standard was changed 15. 20 years ago to make 20 drops ImL. The viscosity of most things is temperature dependent, this would be at room temperature. Hope this helps.
Query: how many eye drops per ml Does the passage answer the query?
Answer: Yes
Passage: RE: How many eyedrops are there 10 ml bottle of Cosopt? My Kaiser pharmacy insists that bottles should last me 100 days but run out way before that time when am using 4 drops per Iay.In the past other pharmacies have given me 3 10-ml bottles for 100 days.E: How many eyedrops are there in 10 ml bottle of Cosopt? My Kaiser pharmacy insists that bottles should last me 100 days but I run out way before that time when am using drops per day. Query: how many eye drops per ml Does the passage answer the query? Answer: No
Passage: : You can transfer money to your checking account from other Wells Fargo. accounts through Wells Fargo Mobile Banking with the mobile app, online, at any. Wells Fargo ATM, or at Wells Fargo branch. Money in - deposits.
Query: you open wells fargo account online
Does the passage answer the query?
Answer: No
Passage: You can open a Wells Fargo banking account from your home or even online. It is really easy to do, provided you have all of the appropriate documentation. Wells Fargo has so many bank account options that you will be sure to find one that works for you. They offer free checking accounts with free online banking.
Query: can you open wells fargo account online
Does the passage answer the query?
Answer: Yes
Passage: [[passage]
Query:((query))
Does the passage answer the query?
Answer:

---

A.3 Relevance Generation Instruction (zero-shot)
This instruction is used to train LLaMA-7B specialized models.
Given passage and query, predict whether the passage includes answer the query by producing either 'Yes' or 'No'.
Passage: [[passage]]
Query: ({query))
Does the passage answer the query?
Answer:
A.4 Permutation Generation Instruction (Text)
Permutation generation (text) is used for text-davinci-003.
This is RankGPT, an intelligent assistant that can rank passages based on their relevancy to the query.
The following are {[num}} passages, each indicated by number identifier []. I can rank them based their relevance query: {[query))
[1] ((passage_1)
[2] ([passage_2)
(more passages)...
The search query is: {(query))
will rank the ((num}) passages above based on their relevance to the search query. The passages will be listed in descending order using identifiers, and the most relevant passages should be listed first, and the output format should be [] [] etc, e.g., [1] [2] > etc.
The ranking results of the {(num)) passages (only identifiers) is:
A.5 Permutation Generation Instruction (Chat)
Permutation generation instruction (chat) used for 3. t-trbo and gpt-4.

---

system:
You are RankGPT, intelligent assistant that rank passages based their relevancy to the query.
user:
will provide you with {(num)} passages, each indicated by number identifier [- Rank them based on their relevance to query: ([query)).
assistant:
Okay, please provide the passages.
user:
[1] ((passage_1)>
assistant:
Received passage [1]
user:
[2] ((passage_2))
assistant:
Received passage [2]
(more passages)...
user
Search Query: {[query)].
Rank the [[num}} passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers, and the most relevant passages should be listed first, and the output format should be [] > []. e.g., [1}> [2]. Only response the ranking results, do not say any word or explain.
B Instructional Methods on LLMs as Rernaker
This paper focus re-ranking task, given M passages for query q, the re-ranking aims agent f(-) output their ranking results R (ri,..,, where r; [1,2, ..., M] denotes the rank of p;. This paper studies using the LLMs as f(.).
B.1 Instructional Query Generation
Query generation has been studied in Sachan et al. (2022); Muennighoff' (2022), in which the relevance between a query and a passage is measured by the log-prot probbabbbyllty of the model generate the query based on the passage. Figure 2 (a) shows an example of instructional query generation. Formally, given query q and a passage pt, their relevance score 8; is calculated as:
Σ
log qet. Ps, Pquer))
(1)
lql
where |ql denotes the number of tokens in qe denotes the t-th token of q, and Iquery denotes the instructions, referring to Figure (a). The passages are then ranked based on relevance score 8;.
B.2 Instructional Relevance Generation
Relevance generation employed in HELM (Liang et al., 2022). Figure 2 (b) shows an example instructional relevance generation, in which LLMs are instructed to output "Yes" the query and passage

---

are relevant "No" if they are irrelevant. The relevance score 8; is measured by the probability of LLMs generating the word 'Yes' or 'No':
1 + pYees, if output is Yes
(2)
1 1 P(No,, if output is No
where p( Yes)No) denotes the probability of LLMs generating Yes or No, and the relevance score is normalized into the range [0, 2].
The above two methods rely on the log probability of LLM, which is often unavailable for LLM API. For example, at the time of writing, OpenAl's ChatCompletion API does not provide the log-probabilityy of generation?.
B.3 Instructional Permutation Generation
The proposed instructional permutation generation is a listwise approach, which directly assigns each passage Pj unique ranking identifier ag (e.g.. [1J, (2J) and places it at the beginning of pj: p Concat(ar,P.) p.). Subsequently, generative LLM is instructed to generate a permutation of these identifiers: Perm [q, 1/1 P'M). where the permutation Perm indicates the rank of the identifiers as (e.g.. [1],
[2]). We then simply map the identifiers a; to the passages pt to obtain the ranking of the passages.
Domain Question
Reference Answer
Sport
What Messi's annual income after transferring to Miami?
S50M-S60M
Sport
How many goals did Haaland scored in the 2023 Champions League Final?
0
Sport
Where did Benzema go after leaving Real Madrid?
Saudi Arabia
Sport
Where was the 2023 Premier League FA Cup Final held?
Wembley Stadium
Sport
Who won 2023 Laureus World Sportsman Of The Year Award?
Lionel Messi Denver Nuggets 4K with one eye
Sport
Who wins NBA Finals 2023?
Tech
What the sereen resolution vision pro?
Tech
What the name of combined Deepmind and Google Brain?
Google DeepMind 144TB
Tech
How much video memory does the DGX GH200 have?
Tech
What are the new features Py Torch 2?
faster, low memory, dynamic shapes
Tech
Who will be the CEO Twitter after Elon Musk is no longer the CEO?
Linda Yaccarino
Tech
What are the best papers of CVPR 2023?
Visual Programming: Compositional [...]
Movie
Who sang the theme song of Transformers Rise of the Beasts?
Notorious B.I.G
Movie
Who is the villain in The Flash?
Fohard Trofse/Professor Zoom
Movie
How many different Spider Men are there in Across the Spider- Spider-Verse?
280 variations
Movie
Who does Momoa play in Fast X?
Dante
Movie
The Little Mermaid first week box office?
$163.8 8 million worldwide Anatomy of a Fall the Stade de France May 19, 2023 Hiroshima
Movie
Which film was the 2023 Palme d' I'Or winner?
Other
Where will Blackpink's 's 2023 world tour concert in France be held?
Other
What is the release date of song Middle Ground?
Other
Where did the G7 Summit 2023 take place?
Table 8: Questions and reference answers on NovelEval-2306.
Table 8 lists the collected 21 questions. These questions come from four domains and include hot topics from the past few months. For each question, we used Google search to obtain 20 passages. When using Google search, in order to avoid all pages containing the answer, we used not only the question itself as a search query, but also the entities that appear in the question as an alternative search query to obtain some pages that are relevant but do not contain the answer. For example, for the first question "What is Messi's 's annual income after transferring to Miami?", we used "Messi" and "Messi transferring" as search queries to get some pages that do not contain the answer. When searching, web pages, news, and used a paragraph or paragraphs from the web pages related the search term as candidate passages. Table 9 shows the statistical information of the data. All of the LLMs (ineluding
collected the highest-ranking
'https://platfon

---

gpt-4-0314 and gpt-4-0613) we tested achieved 0% question-answeringg accuracy on the obtained test set.
We searched for 20 candidate passages for each question using Google search. These passages were manually labeled for relevance by group of annotators, including the authors and their highly educated colleagues. To ensure consistency, the annotation process was repeated twice. Each passage was assigned relevance score: 0 for not relevant, 1 for partially relevant, and 2 for relevant. When evaluating the latest LLMs, we found that all ton-retrieval-augmeeeee models tested achieved 0% accuracy in answering the questions on the test set. This test set provides a reasonable evaluation of the latest LLMs at the moment. Since LLMs may be continuously trained on new data, the proposed test set should be continuously updated to counteract the contamination of the test set by LLMs.
Number of questions
21
Number of passages
420
Number of relevance annotation
420
Average number words of passage
149
Number of score
290
Number of score
40
Number of score 2
90
Table 9: Data Statistics of NovelEvvll.
D Implementation Details
D.1 Training Configuration
We use ReBERTa-V3-base which concatenates the query and passage with [SEP] token and utilizes the representation of the [CLS] token. To generate candidate passages, we randomly sample 10k queries and use BM25 to retrieve 20 passages for each query. We then re-rank the candidate passages using the 5-tturbb API with permutation generation instructions, at cost of approximatcly $40. During training, we employ batch size of 32 and utilize the Adam W optimizer with constant learning rate of 5 10-5 The model is trained for two epochs. Additionally, we implement models using the original MS MARCO labels for comparison.
The LLaMA-7B model is optimized with the AdamW optimizer, constant learning rate of 5 x 10-5 and with mixed precision of 6 and Deepspeed Zero3 strategy. All the experiments are conducted on 8 A A100-40G GPUs.
Figure illustrates the detailed model architecture of BERT-like and GPT-like specialized models.
[0, 1]
Yes or No
head
head
DeBERTa
LLaMA
[CLS] Query [SEP] Passage
Given passage and query, [...] (query) [...]
(passage) [...] Asnwer:
(a) BERT-like Specialized Model
(b) GPT-like Specialized Model
Figure 5: Model architecture of BERT-like and GPT-like specialized models.

---

D.2 Training Objective
Using the permutation generated by ChatGPT, we consider the following losses to optimize the student model:
Listwise Cross-Entropy (CE) (Bruch et al., 2019). Listwise CE is the wide-use loss for passage ranking, which considers only one positive passage and defines the list-wise softmax cross-entropy on all candidate's passages:
exp(s;) Ej exp(s,)
Σ =1 log(
Listwise CE
where 1 is the indicator function.
RankNet (Burges et al., 2005). RankNet is pairwise loss that measures the correctness of relative passage orders:
NeankNet
ΣΣ
Σ Dr,ery log(1 + exp(s: s;))
when using permutation generated by ChatGPT, we can construct (M - 1)/2 pairs.
LambdaLoss (Wang et al., 2018). 8). The LambdaLoss further accounts for the nDCG gains of the model ranks. LambdaLoss uses the student model's rank, denoted as π (T....., AM), where is the model predicted rank of ii with similar definition with ChatGPT rank R.. The loss function is defined as:
Lambda
Σ TESTy
ANDCGlogg. + exp(s: 8y))
in which ANDCG is the delta of NDCG which could be compute as ANDCG G D(r.)
D(s
where (m:)) and (TT,) are the position discount functions and G; and Gj are the gain functions used in NDCG (Wang et al., 2018). Pointwise Binary Cross-Entropy (BCE). We also include the Pointwise BCE as the baseline loss for supervised methods, which is caleulated based on each query-document pair independently:
M
LBCE
Σ log + 1 log o(1 - -8;)
where (xx) I+exp(-x) is the logistic function.
E Baselines Details
We include state-of-the-art supervised and unsupervised passage re-ranking methods for comparison. The supervised baselines are:
monoBERT (Nogueira and Cho, 2019): cross-encoder re- re-ranker based on BERT-large, trained on MS MARCO.
monoT5 (Nogueira et al., 2020): A scquencc-to-scquenec re-ranker that uses T5 to calculate the relevance 5
Cohere Rerank: A passage reranking API rerank-english-vv2. 0 developed by Cohere?. Cohere does not provide details the structure and training method of the model.
mmarcoCE (Bonifacio et al., 2021): 12-layer MM cross-encoder model trained on
mmarco, a translated version of MS MARCO. mmarcoCE serves as a baseline for Mr.TyDi.
The unsupervised baselines are:
"https://hugginggaae http://hwggnnaaaaec.o/o///arrrra
httttpss//cohere.com/ree "https://huggingfaat co/ wtw.e//cooooooooo/////ooo/coan/a

---

UPR (Sachan et al., 2022): Unsupervised passage ranking with instructional query generation. Due to its superior performance, we use the FLAN-TS-XL (Chung et al., 2022) the LLM of UPR. InPars (Bonifacio et al., 2022): monoTT5-BB trained on pseudo data generated by GPT-3.
Promptagator++ (Dai et al., 2023): A 11MM cross-encoder re-ranker trained on pseudo queries generated by FALN 137B.
Method
Repetition
Missing
Rejection
RBO
text-davinci-003
280
0
72.30
--turbo
14
153
7
81.49
gpt-4
11
82.08
Table 10: Analysis of model stability on TREC. Repetition refers to the number of times the model generates duplicate passage identifiers. Missing refers to the number missing passage identifiers in model output. Rejection refers to the number of times the model rejects perform the ranking. RBO, i.e., rank biased overlap, refers the consistency of the model in ranking the same group of passages twice.
Model Behavior Analysis
In the permutation generation method, the ranking of passages is determined by the list of model-output passage identifiers. However, we have observed that the models do not always produce the desired output, evidenced by occasional duplicates missing identifiers in the generated text. In Table 10, present quantitative results of unexpected model behavior observed during experiments with the GPT models.
Repetition. The repetition metric measures the occurrence of duplicate passage identifiers generated by the model. The results indicate that ChatGPT produced 14 duplicate passage identifiers during re-ranking 97 queries on two TREC datasets, whereas text-davinci and GPT-4 did not exhibit any duplicates.
Missing. We conducted a count of the number of times the model failed to include all passages the re-ranked permutation output?. Our findings revealed that text-davinci-00) has the highest number of missing passages, totaling 280 instances. ChatGPT also misses a considerable number of passages, occurring 153 times. On the other hand, GPT-4 demonstrates greater stability, with only one missing passage in total. These results suggest that GPT-4 has higher reliability in generating permutations, which is critical for effective ranking.
Rejection, We have observed instances where the model refuses to re-rank passages, evidenced by responses such as "None of the provided passages is directly relevant to the query ...". To quantify this behavior, we count the number of times this occurred and find that GPT-4 rejects ranking the most frequently, followed by ChatGPT, while the Davinci model never refused rank. This finding suggests that chat LLMs tend to be more adaptable compared to completion LLMs, and may exhibit more subjective responses. Note that we do not explicitly prohibit the models from rejecting ranking in the instructions, as we find that does not significantly impact the overall ranking performance.
RBO. The sliding windows strategy involves re-ranking the top-ranked passages from the previous window in the next window. The models are expected to produce consistent rankings in two windows for the same group of passages. To measure the consistency of the model's ' rankings, we use RBO (rank biased overlap")), which calculates the similarity between the two ranking results. The findings turn out that ChatGPT and GPT-4 are more consistent in ranking passages compared to the Davinci model. GPT-4 also slightly outperforms ChatGPT in terms of the RBO metric.
G Analysis on Hyperparameters of Sliding Window
To analyze the influence of parameters of the sliding window strategy, we adjust the window size and set the step size to half of the window size. The main motivation for this setup is to keep the expected
"In our implementation, we append the missing passages their original order the end of re-ranked passages. "https: / thub. com/ chhrgyacceeee/rrb

---

API
Instruction
Tokens
Requests
$USD
text-curie-001
Relevance generation
52,970
100
0.106
text-curie-001
Query generation
10,954
100
0.022
text-davinci-003
Query generation
11,269
100
0.225
text-davinci-003
Permutation generation
17,370
10
0.347
8pt-3.5-turbo
Permutation generation
19,960
10
0.040
gpt-4
Permutation generation
19,890
10
0.596
rerank top-30
Permutation generation 3,271
0.098
Table II:
Average token cost, number API request, and $USD per query TREC.
Window size
Step size
nDCG@1
nDCG@ nDCG@5
20
75.58
70.50
67.05
40
20
78.30
71.32
65.51
60
75.97
69.23
65.03
72.09
70.59
65.57
Table 12: Analysis on Hyperparameters of Sliding Window on TREC-DL19.
overhead of the method (number of tokens required for computation) low; i.e., most tokens in this setup are used for PG only twice. The experimental results are shown in Table 12 The results show that the effect varies best in terms of nDCG@ 10, while window size=40 performs best in terms of nDCG@S and nDCG@ 1. We speculate that larger window size will increase the model's ranking horizon but will also present challenges in processing long contexts and large numbers items.
a certain range of arrivals for different values of window size: window size=20 performs
H API Cost
In Table 1 I, provide details the average token cost, API request times, and USD cost per query. terms of average token cost, the relevance generation method the most expensive, requires 4 in-context demonstrations. On the other hand, the permutation generation method incurs higher token costs compared the query generation method, as involves the repeated processing of passages sliding windows. Regarding the number of requests, the permutation generation method requires 10 requests for sliding windows, while other methods require 100 requests for re-ranking 100 passages. terms of average USD cost, GPT-4 is the most expensive, with cost of $0.596 per query. However, using GPT-4 for re-ranking the top-30 passages can result in significant cost savings, with cost of $0.098 per query for GPT-4 usage, while still achieving good results. As result, only utilize GPT-4 for re-ranking the top 30 passages of ChatGPT BEIR and Mr.TyDi. The total cost of experiments with GPT-4 amounts to $556.
Since the experiments with ChatGPT and GPT-4 are conducted using the OpenAl API, the running time is contingent on the OpenAl service, e.g., API latency. Besides, the rumning time can also vary across different API versions and network environments. In our testing conditions, the average latency for API calls for gpt-3.5-turbo and gpt-4 was around 1.1 seconds and 3.2 seconds, respectively. Our proposed sliding window-based permutation generation approach requires 10 API calls per query to re-rank 100 passages. Consequently, the average running time per query is 11 seconds for gpt-3.5-turbo and 32 seconds for gpt-4.
I Results of Specialized Models
Table 13 lists the detailed results of specialized models on TREC and BEIR.
I1 Note that the results are obtained using gpt-3. 5-turbo-1ee5 16k. API for managing long context.

---

Method
DL19 DL20 Covid NFCorpus Touche DBPedia SciFact Signal News Robust04 BEIR (Avg)
BM25
50.58
47.96 59.47
30.75
44.22
31.80
67.89
33.05 39.52
40.70
43.42
Supervised
train on MS MRACO
monoBERT (340M)
70.50 67.28
70.01
36.88
31.75
41.87
71.36
31.44
44.62
49.35
47.16
monoTS (220M)
71.48 66.99 78.34
37.38
30.82
42.42
73.40
31.67
46.83
$1.72
49.07
monoT5 (3B)
71.83 68.89 80.71
38,97
32.41
44.45
76.57
32.55
48.49
56.71
51.36
Cohere Rerank-v2
73.22 67.08 81.81
36.36
32.51
42.51
74.44
47.59
50.78
49.45
Unsupervised instructional permutation generation
ChaiGPT
65.80 62.91 76.67
35.62
36.18
44.47
70.43
32.12
48.85
50.62
49.37
GPT-4
75.59 70.56 85.51
38.47
38.57
47.12
74.95
34.40
52.89
57.55
53.68
Specialized Models
train on MARCO labels or CharGPT predicted permutations
MARCO Pointwise BCE
65.57 56.72 70.82
33.10
17.08
32.28
55.37
19.30
41.52
46.00
39.43
MARCO Listwise CF
65.99 57.97 66.31
32.61
20.15
30.79
37.57
18.09
38.11
39.93
35.45
MARCO RankNet
66.34 58.51 70.29
34.23
20.27
29.62
49.01
23.22
39.82
43.87
38.79
MARCO Lambdal
64.82 56. 16 72.86
34.20
19.51
32.55
51.88
26.22
42.47
45.28
40.62
ChatGPT Listwise CE
65.39 58.80 8..80 76.29
35.73
38.19
40.24
64.49
31.37
47.61
48.00
47.74
ChatGPT RankNet
65.75 81.26
36.57
39.03
42.10
68.77
31.55
52.54
$2.44
50.53
ChatGPT LambdaLoss
67.17 80.63
36.74
36.73
43.75
68.21
49.00
50.51
49.77
deberta-y3-xsmall (70M)
64.75 55.07 78.21
35.95
35.42
41.37
67.86
30.04
47.68
49.91
48.31
deberta-v3-small (142M)
67.85 58.84 78.88
36.55
36.16
40.99
66.66
30.29
49.17
49.73
48.55
deberta-v3-buse (184M)
70.28 62.52 80.81
36.15
37.25
44.06
71.70
32.45
50.84
$1.33
50.57
deberta-v3--arge (435M)
70,66 67.15 84.64
38.48
39.27
47.36
74.18
32.53
51.19
56.55
53.03
deberta-v3-large 5K
70.93 64.32 84.43
38.66
40.72
46.28
73.88
31.93
52.24
$5.89
53.00
deberta-v3-llrge 3K
70.79 63.91 84.21
38.73
39.83
45.74
74.41
52.29
57.42
53.07
deberta-v3-large IK
69.90 64.81 83,38
38.94
36.65
44.46
71.96
30.19
50.73
$3.74
$1.26
deberta-v3--lrgge 500
69.71 62.00 83.54
37.23
33.68
44.56
70.48
45.64
42.67
48.31
deberta-v3-large label 10K
66.61 57.26 74.36
33.94
18.09
34,95
35.35
21.38
39.00
44.94
37.75
deberta-v3-large label 5K
68.98 61.38 80.73
35.68
20.48
37.34
54.63
24.25
36.94
$1.13
42.64
deberta-v3-large label 3K
67.41 60.42 79.82
35.49
24.54
37.39
47.31
23.29
39.87
$0.65
42.29
deberta-v3-larre label 1K
65.55 60.93 77.70
33.29
23.36
36.38
31.10
21.71
34.28
38.31
37.01
deberta-v3-large label 500
60.59 54.45 76.20
32.93
19.66
31.54
45.66
13.99
33.48
44.49
37.24
deberta-v33-aarge m-3B
73.05
68.82 84.78
38.55
34.43
43.61
75.45
30.75
49.85
56.80
51.78
deberta-v3-largge chatgpt+lahel
72.42
67.30 85.96
38.75
35.06
45.43
71.81
28.52
45.91
55.57
50.88
deberta-v3-buse label 10k
65.66 59.84 71.63
34.65
16.53
32.59
34.65
22.64
37.60
44.02
36.79
deberta-v3-small label 10k
63.63 52.83 68.17
30.48
18.12
31.72
33.62
18.02
34.57
36.09
33.85
deberta-v3-xsmall lahel 10k
60.89 51.15 63.58
28.67
14.87
27.12
20.60
18.97
32.61
32.67
29.89
Ilama-7b
71.33 66.06 78.23
37.60
34.87
45.46
76.
34.17
51.79
55.22
51.68
vicuna-7b
71880 66.89 78.32
36.87
31.81
45.40
74.23
34.28
51.13
$2.91
50.62
Ilama-7b 10k label Ilama-7b label
65.22 56.85 75.36
36.24
20.88
37.34
69.04
41.21
49,21
69.24 58.97 80.49
37.55
28.23
39.66
71.79
26.04
44.09
$3.83
47.71
Table 13: Results (nDCG@ 10) on TREC and BEIR. Best performing specialized and overall system(s) are marked bold. The specialized models fine-tined on sampled queries using relevance judgements from MARCO ChatGPT.


Reviews:
[{"review_id": "BbzMXWGpgE", "reviewer": "Reviewer_Xw3x", "summary": "", "questions": "Question A: Does the supervised learning baseline in Figure 4 use the same RankNet loss? Otherwise it is not clear whether the improvement comes from using the ranking loss or from the teacher labels, since ranking losses can also help improve the model performance [2].", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and constructive comment! We appreciate your feedback and will address each of your points in turn.\n\n### **Results of PG+curie-001 and RG+davinci-003**\n> *Baselines like UPR are essentially QG and thus can be directly applied using the same LLM as the proposed method. Since PG is a major contribution of this paper, it is better to compare QG vs. RG vs. PG using the same LLM. However, in Table 4 there are only partial results reported (missing PG for curie-001 and missing RG for davinci-003).*\n\n- Thank you for your constructive comments. We have completed the results on TREC-DL19 with PG+curie-001 and RG+davinci-003. The table below presents the findings:\n| LLM | Method | nDCG@1 | nDCG@5 | nDCG@10 |\n| --- | --- | --- | --- | --- |\n| curie-001 | RG | 39.53 |40.02 | 41.53 |\n| curie-001 | QG | 50.78 | 50.77 | 49.76 |\n| **curie-001** | **PG** | **66.67** | **56.79** | **54.21** |\n| davinci-003 | RG | 54.26 | 52.78 | 50.58 |\n| davinci-003 | QG | 37.60 | 44.73 | 45.37 |\n| **davinci-003** | **PG** | **69.77** | **64.73** | **61.50** |\n\n- These results demonstrate that our proposed permutation generation method surpasses both RG and QG when using the same LLM. QG cannot be implemented with ChatGPT as the API cannot return the log-likelihood. We will include these results in our final revision.\n\n###  **Compare with monoT5+BM25**\n> *As suggested in Section 6.5 Line 459, the LLM results seem to be extremely sensitive to the initial passage order of BM25, which other supervised models or methods like monoT5/UPR do not have access to. A potentially better comparison is to compare with an ensemble result of monoT5 + BM25 (or UPR + BM25). Otherwise it is not convincing that permutation generation + sliding window is a better strategy than query generation like UPR.*\n\n- Thank you for your insightful suggestions. To compare our model with the ensemble model \"monoT5+BM25\", we followed the ensemble method proposed by Lin et al. (2022). Specifically, we use a hyperparameter $\\alpha$ to combine the BM25 score and the monoT5 score by: $\\text{score}(q, p) = \\alpha * \\text{monoT5}(q, p) + (1 - \\alpha) * \\text{BM25}(q, p)$. The results on TREC are:\n| $\\alpha$ | TREC-DL19 | TREC-DL20 |\n| --- | --- | --- |\n| 1.0 (monoT5-3B) | **71.83** | **68.89** |\n| 0.9 | 70.70 | 65.13 |\n| 0.8 | 70.42 | 64.89 |\n| 0.7 | 70.16 | 64.22 |\n| 0.6 | 69.06 | 63.85 |\n| 0.5 | 67.89 | 62.76 |\n| 0.0 (BM25) | 50.58 | 47.96 |\n- The results show that combining scores of BM25 and monoT5 does not improve the performance on the TREC dataset. This indicates that the reranker can already produce better relevance scores than BM25. \n- The purpose of using the ranking of BM25 is primarily to enhance efficiency and minimize the number of sliding window passes. We would like to investigate other strategies that do not rely on BM25 in our future work.\n\n\n### **Results of duoT5**\n\n> *Most baseline methods only see one query-document pair at a time, while PG sees multiple at a time. Perhaps methods like duoT5 [1] should be more appropriate baselines*\n\n- Thank you for your valuable feedback. We have implemented duoT5 on TREC, following the methodology of Pradeep et al., who use duoT5 to re-rank the top-50 passages of monoT5. The nDCG@10 results on TREC-DL19 and DL20 are as follows:\n| Stage 1 | Stage 2 | TREC-DL19 | TREC-DL20 |\n| --- | --- | --- | --- |\n| monoT5-base | --- | 71.48 | 66.99 |\n| monoT5-base | duoT5-base | 71.94 | 67.76 |\n| monoT5-3B | --- | 71.83 | 68.89 |\n| monoT5-3B | duoT5-3B | 72.84 |  69.00 |\n| GPT-3.5-turbo | --- | 65.80 | 62.91 |\n| GPT-4 | --- | **75.59** | **70.56** |\n- These results indicate that duoT5 marginally outperforms monoT5, but is still worse than our method with GPT-4.\n\n\n### **Parameters of sliding window strategy**\n> *The method might be sensitive to different parameters of the sliding window strategy. More experiments can be done to provide guidance on how to set the parameters of the sliding window strategy.*\n\n- Thanks for your good suggestion. To analyze the influence of parameters of the sliding window strategy, we adjust the window size and set the step size to half of the window size. The main motivation of this setup is to keep the expected overhead of the method (number of tokens required for computation), i.e., most tokens in this setup are used for PG only twice. The experimental results are as follows:\n| Window size | Step size | nDCG@1 | nDCG@5 | nDCG@10 | \n| --- | --- | --- | --- | --- |\n| 20 | 10 | 75.58 | 70.50 | 67.05 |\n| 40 | 20 | 78.30 | 71.32 | 65.51 |\n| 60 | 30 | 75.97 | 69.23 | 65.03 |\n| 80 | 40 | 72.09 | 70.59 | 65.57 |\n- The results show that the effect varies over a certain range of arrivals for different values of window size: window size=20 performs best in terms of nDCG@10, while window size=40  performs best in terms of nDCG@5 and nDCG@1. We speculate that a larger window size will increase the model's ranking horizon, but will also present challenges in processing long contexts and large numbers of items.\n- The above results are obtained using gpt-3.5-turbo-16k API for processing long context.\n- We will include these results and detailed analysis in our final paper.\n\n\n\n### **Overfit to ChatGPT?**\n> *Methods might be overfitting to ChatGPT. The gap between LLMs reported in Table 6 is large. Some of them are even lower than or close to BM25, which seems suspicious. It is possible that the prompt employed by the authors are heavily engineered for ChatGPT and hence does not generalize to other LLMs.*\n\nThank you for your insightful comment. We would like to clarify the following points:\n- Firstly, the underperformance of certain low-capacity LLMs (e.g., Vicuna-13B, Flan-T5-11B) could be attributed to their limited ability to follow instructions. Despite numerous attempts at prompt designs, these models often failed to generate valid results, instead producing ordered ID lists (e.g., \"[1] > [2] > [3] > [4]\") or single IDs (e.g., \"[17]\"). The results in Table 4 are usually the best results from multiple attempts. Given this, it's reasonable that these models underperform BM25, a well-regarded ranking model in the IR field.\n- Secondly, our method has demonstrated effectiveness on non-OpenAI models. For instance, Claude-v1 achieved results very close to those of ChatGPT (nDCG@10 on DL19 of 60.25 vs 60.89), indicating that our approach is not exclusively optimized for ChatGPT.\n- Lastly, we understand that different LLMs may require unique prompting techniques for optimal performance. We would like to explore autonomous prompting methods for various LLMs in future studies.\n\n### **Question A: Rank loss**\n> *Question A: Does the supervised learning baseline in Figure 4 use the same RankNet loss? Otherwise it is not clear whether the improvement comes from using the ranking loss or from the teacher labels, since ranking losses can also help improve the model performance [2].*\n\n- Yes, all models, including the supervised baselines, were trained using the RankNet loss. We found through experiments that while the supervised models yielded slightly varied results with different loss functions examined in RankT5, all these results were lower than those of the proposed permutation distillation method. This discrepancy could be attributed to the sparse and low-quality annotation of MS MARCO (Arabzadeh et al., 2021). We appreciate your questions and will provide more details in the final version of our paper.\n\n### **References**\n- Lin, Sheng-Chieh and Jimmy Lin. “A Dense Representation Framework for Lexical and Semantic Matching.” ACM Transactions on Information Systems (2022): n. pag.\n- Negar Arabzadeh, Alexandra Vtyurina, Xinyi Yan, and Charles L. A. Clarke. 2021. Shallow pooling for sparse labels. Information Retrieval Journal, 25:365-385.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Directly using LLM to produce the ranking of multiple documents is a novel and interesting idea and worth further explorations.\n\n* Thorough experiments conducted on multiple data sets, including a newly proposed NovelEval data set.\n\n* The work on distillation makes the technique more applicable in the real world.\n", "reasons_to_reject": "\n* Potential unfair comparison\n\n  * Baselines like UPR are essentially QG and thus can be directly applied using the same LLM as the proposed method. Since PG is a major contribution of this paper, it is better to compare QG vs. RG vs. PG using the same LLM. However, in Table 4 there are only partial results reported (missing PG for curie-001 and missing RG for davinci-003). \n\n  * As suggested in Section 6.5 Line 459, the LLM results seem to be extremely sensitive to the initial passage order of BM25, which other supervised models or methods like monoT5/UPR do not have access to. A potentially better comparison is to compare with an ensemble result of monoT5 + BM25 (or UPR + BM25). Otherwise it is not convincing that permutation generation + sliding window is a better strategy than query generation like UPR. \n\n  * Most baseline methods only see one query-document pair at a time, while PG sees multiple at a time. Perhaps methods like duoT5 [1] should be more appropriate baselines. \n\n* The method might be sensitive to different parameters of the sliding window strategy. More experiments can be done to provide guidance on how to set the parameters of the sliding window strategy. \n\n* Methods might be overfitting to ChatGPT. The gap between LLMs reported in Table 6 is large. Some of them are even lower than or close to BM25, which seems suspicious. It is possible that the prompt employed by the authors are heavily engineered for ChatGPT and hence does not generalize to other LLMs. \n", "questions_to_authors": "Question A: Does the supervised learning baseline in Figure 4 use the same RankNet loss? Otherwise it is not clear whether the improvement comes from using the ranking loss or from the teacher labels, since ranking losses can also help improve the model performance [2].", "paper_topic_and_main_contributions": "This paper explores the potential of LLMs like ChatGPT for relevance ranking. The authors explore query generation, relevance generation, and propose permutation generation + sliding window as different prompting strategies. They also investigate whether the LLM generated results can be distilled into smaller models. They conduct experiments on TREC-DL, BEIR and Mr.TyDi data sets, and a proposed new data set NovelEval and show that LLM can be more effective than supervised rankers like monoT5. They also show that the distilled model performs better than model trained from scratch. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 6: ND might be a confusing abbreviation for nDCG\n"}, "extra_text": {"Missing_References": "\n[1] Pradeep, Ronak, Rodrigo Nogueira, and Jimmy Lin. \"The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models.\" arXiv preprint arXiv:2101.05667 (2021).\n\n[2] Zhuang, Honglei, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. \"RankT5: Fine-tuning T5 for text ranking with ranking losses.\" In SIGIR. 2023.\n"}}, {"review_id": "trGjBrsAbQ", "reviewer": "Reviewer_V7x6", "summary": "", "questions": "A. Does the window size in the permutation distillation technique have an influence on the performance?\nB. What would the model perform if the data generated by ChatGPT and MARCO were mixed together?\nC. What about the running time of the proposed instructional permutation generation approach？\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful comment. We would like to address your questions in turn.\n\n### **Novelty of our work**\n> *The method proposed in this paper lacks novelty and is similar to previous works.*\n\nThank you for your feedback. We would like to clarify the novelty of our paper: (1) We are the first to investigate LLMs like ChatGPT on re-ranking tasks. A novel permutation generation method is proposed that has demonstrated state-of-the-art performance on IR benchmarks with GPT-4. (2) A new benchmark for evaluating LLMs on unseen knowledge is proposed. (3) Our paper also presents a study on a permutation distillation model.\n\n### **Presentation issues**\n> *There are many spelling and grammatical errors.*\n\nMany thanks for your valuable feedback regarding the typos and the suggestion to use vector diagrams. We will correct these typos and incorporate vector diagrams in the final version. \n\n\n### **Question A: Influence of window size**\n> *A. Does the window size in the permutation distillation technique have an influence on the performance?*\n\n- Thanks for your suggestion. To analyze the influence of parameters of the sliding window strategy, we adjust the window size and set the step size to half of the window size. The main motivation of this setup is to keep the expected overhead of the method (number of tokens required for computation), i.e., most tokens in this setup are used for PG only twice. The experimental results are as follows:\n| Window size | Step size | nDCG@1 | nDCG@5 | nDCG@10 | \n| --- | --- | --- | --- | --- |\n| 20 | 10 | 75.58 | 70.50 | 67.05 |\n| 40 | 20 | 78.30 | 71.32 | 65.51 |\n| 60 | 30 | 75.97 | 69.23 | 65.03 |\n| 80 | 40 | 72.09 | 70.59 | 65.57 |\n- The results show that the effect varies over a certain range of arrivals for different values of window size: window size=20 performs best in terms of nDCG@10, while window size=40  performs best in terms of nDCG@5 and nDCG@1. We speculate that a larger window size will enhance the model's ranking horizon, but will also present challenges in handling long context and large numbers of items.\n- The above results are obtained using gpt-3.5-turbo-16k API for processing long context.\n- We will include these results and detailed analysis in our final paper.\n\n### **Question B: Mixing ChatGPT data with labels**\n> *B. What would the model perform if the data generated by ChatGPT and MARCO were mixed together?*\n\n- The table below presents the results of an experiment where we combined MARCO labels with ChatGPT generated data, denoted as \"MARCO+ChatGPT\". \n| Method | TREC-DL19 | TREC-DL20 | BEIR |\n| --- | --- | --- | --- |\n| MARCO | 68.89 | 61.38 |  42.64 |\n| ChatGPT | 70.66 | 67.15 | **53.03** |\n| MARCO+ChatGPT | **71.38** | **67.30** | 50.88 |\n- In this experiment, we incorporated the MARCO label passages into the top-1 position of the ChatGPT predicted rank and subsequently trained the model. The results indicate that the combination of ChatGPT data with MARCO labels enhances the model's performance on the TREC, and reduces performance on the BEIR.\n\n### **Question C: Running time**\n> *C. What about the running time of the proposed instructional permutation generation approach?*\n\n- Our experiments with ChatGPT and GPT-4 are conducted using the OpenAI API. As such, the running time is contingent on the OpenAI service, e.g., API latency. Besides, the running time can also vary across different API versions and network environments.\n- In our specific testing conditions, the average latency for API calls for gpt-3.5-turbo and gpt-4 were around 1.1 seconds and 3.2 seconds, respectively. Our proposed sliding window-based permutation generation approach requires 10 API calls per query to re-rank 100 passages. Consequently, the average running time per query is 11 seconds for gpt-3.5-turbo and 32 seconds for gpt-4. \n- As shown in Table 11, the proposed permutation generation approach consumes more tokens than the previous query generation approach, but is faster in using the API due to 10 times fewer API calls.\n- We will incorporate these figures into our final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The topic investigated by this paper is new. Some conclusions drawn by this paper are essential, and may inspire other researchers.\n2. Experiments are rich and comprehensive.\n3. This paper is well written.", "reasons_to_reject": "1. The method proposed in this paper lacks novelty and is similar to previous works.\n2. There are many spelling and grammatical errors.", "questions_to_authors": "A. Does the window size in the permutation distillation technique have an influence on the performance?\nB. What would the model perform if the data generated by ChatGPT and MARCO were mixed together?\nC. What about the running time of the proposed instructional permutation generation approach？\n", "paper_topic_and_main_contributions": "This paper aims to explore the re-ranking capability of LLMs. It introduces a permutation generation approach to re-rank passages. Furthermore, a permutation distillation technique is proposed to distill ranking capabilities of LLMs into smaller ranking models. To evaluate the performance, this paper conducts experiments on several benchmarks and a newly constructed dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 038: application -> applications\nLine 054: of of -> of\nLine 059: an -> a\nLine 075: In additation -> In addition\nLine 200: illustrate -> illustrates\nLine 266: calculates -> calculate\nLine 309: a argument -> an argument\nLine 340: fromfrom -> from\nLine 469: inference -> influence\nLine 583: improving -> improve\n\nIt is better to use vector diagrams."}}, {"review_id": "mXkFl0Arka", "reviewer": "Reviewer_KPjQ", "summary": "", "questions": "A. I am not able to understand how LLM based distillation is better than supervised learning using ground truth annotations for MSMARCO. Is it due to just sparser signal in ground truth labels compared to generated permutations?\nB. Would supervised training with labels from a cross encoder trained on MSMARCO lead to better precision (than ChatGPT distilled model) for the specialised model? \nC. Is pseudo-labelling (or specialised distillation) done for setups where the target data is different from the training data? If we are not specialising to a particular target domain, then we can almost always find a large labelled source of data (like MSMARCO) to train the model on, without the need for pseudo labelling it right?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful comment. We would like to address your questions in turn.\n\n### **Definition of permutation generation**\n> *Permutation generation is not well defined and seems like a hand wavy heuristic.*\n\n- Thank you for your constructive comment. To clarify, this paper focuses on the re-ranking task: given $M$ passages $(p_1, ..., p_M)$ for a query $q$, the re-ranking aims to use an agent $f(\\cdot)$ to output their ranking results $\\mathbf{R} = (r_1, . . . , r_M)$, where $r_i ∈ [1, 2, ..., M]$ denotes the rank of $p_i$. This paper studies using the LLMs as $f(·)$.\n- Previous pointwise approaches, such as query generation or relevance generation, predict the relevance score of each passage only by leveraging the query and the passage: $\\mathbf{S} = (s_1, ..., s_M)$, where $s_i = f(q, p_i)$. Then, the overall ranks are calculated by $r_i = \\text{arg sort}_{i} (\\mathbf{S})$.\n- In contrast, our proposed permutation generation method directly assigns each passage $p_i$ to a unique ranking identifier $a_i$ (e.g., *[1]*, *[2]*), and places it at the beginning of  $p_i$: $p_{i}^{'} = \\text{concat}(a_i, p_i)$. Subsequently, a generative LLM is instructed with generating a permutation of these identifiers: $\\mathbf{Perm} = f(q, p_{1}^{'}, ..., p_{M}^{'})$, where the permutation $\\mathbf{Perm}$ indicates the rank of the identifiers $a_i$ (e.g., $[a_3, a_1,...]$). We then simply map the identifiers $a_i$ to the passages $p_i$ to obtain the ranking of the passages.\n- We appreciate your feedback and will include a definition of the proposed method in the final version of our paper for enhanced clarity.\n\n\n### **Question A: Why LLM-based distillation is better than supervised learning**\n> *A. I am not able to understand how LLM based distillation is better than supervised learning using ground truth annotations for MSMARCO. Is it due to just sparser signal in ground truth labels compared to generated permutations?*\n\n- The sparse labeling of MS MARCO does play an important role. MS MARCO only provides one relevant passage, but other passages, often deemed \"non-relevant\" by supervised learning methods, may also be relevant to the query. \n- Furthermore, research by Arabzadeh et al. demonstrates that the top items returned by a modern neural ranker often appear superior to MS MARCO labeled relevant items. In particular, human assessors favor the top passage from the neural ranker over the MARCO ground-truth in 58.6% of the cases when the two are in disagreement.\n- In contrast, distillation approaches tend to produce more comprehensive and high-quality results, thereby leading to superior performance. \n\n\n### **Question B: Distilling supervised re-ranker**\n> *B. Would supervised training with labels from a cross encoder trained on MSMARCO lead to better precision (than ChatGPT distilled model) for the specialised model?*\n\n- Thank you for your insightful question. To investigate this, we conducted an experiment where we distilled the predictions of monoT5-3B on 10k queries into a Deberta-large reranker using RankNet Loss. The table below presents the experimental results (nDCG@10):\n| Model | Training Method | TREC-DL19 | TREC-DL20 | BEIR (avg) |\n| --- | --- | --- | --- | --- |\n| ChatGPT | Zero-shot | 65.80 | 62.91 | 49.37 |\n| monoT5-3B | Supervised Learning | 71.83 | 68.89 | 51.36 |\n| Deberta | Distilling ChatGPT | 70.66 | 67.15 | **53.03** |\n| Deberta | Distilling monoT5-3B | **72.01** | **69.85** | 51.78 |\n- From these results, we can observe that the distillation of monoT5-3B outperforms the distillation of ChatGPT on the TREC benchmarks, but lags behind on the BEIR benchmarks. \n\n### **Question C: When to apply specialized distillation**\n> *C. Is pseudo-labelling (or specialised distillation) done for setups where the target data is different from the training data? If we are not specialising to a particular target domain, then we can almost always find a large labelled source of data (like MSMARCO) to train the model on, without the need for pseudo labelling it right?*\n\n- Firstly, even for tasks with substantial labeled data like MS MARCO, the quality of annotations remains crucial. Our experiments demonstrate that sparse and low-quality labels could negatively impact the results. Furthermore, the cost of collecting large-scale high-quality labeled datasets is often significantly higher than querying LLMs.\n- Secondly, we believe that tasks without extensive annotations are more common. In these scenarios, the query and document distribution, domain, and relevance criteria may significantly differ from existing labeled data like MS MARCO. In such cases, LLMs could potentially offer better performance due to their robust adaptability to various domains and distributions.\n\n\n### **References**\n\n- Negar Arabzadeh, Alexandra Vtyurina, Xinyi Yan, and Charles L. A. Clarke. 2021. Shallow pooling for sparse labels. Information Retrieval Journal, 25:365-385.\n- Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se679 bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Towards unsupervised dense information retrieval with contrastive learning. TMLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a comprehensive empirical analysis for performance of chatpgpt/gpt4 as re-ranking agent\n2. The paper proposed novel permutation generation for using prompting LLMs for ranking tasks. Also paper proposes permutation distillation for training specialised models. Both of these techniques lead to gain for their corresponding benchmarks. \n3. The sensitivity of final ranking precision on initial rank order (from BM25) is intuitive and interesting (along with the fact that multiple rerankings with LLMs lead to better nDCG@10 number but worse nDCG@1)\n\n", "reasons_to_reject": "1. Permutation generation is not well defined and seems like a hand wavy heuristic.", "questions_to_authors": "A. I am not able to understand how LLM based distillation is better than supervised learning using ground truth annotations for MSMARCO. Is it due to just sparser signal in ground truth labels compared to generated permutations?\nB. Would supervised training with labels from a cross encoder trained on MSMARCO lead to better precision (than ChatGPT distilled model) for the specialised model? \nC. Is pseudo-labelling (or specialised distillation) done for setups where the target data is different from the training data? If we are not specialising to a particular target domain, then we can almost always find a large labelled source of data (like MSMARCO) to train the model on, without the need for pseudo labelling it right?", "paper_topic_and_main_contributions": "The paper explores utilities of LLMs as re-ranking agents in an information retrieval pipeline. They argue that with right prompting techniques LLMs can be better than supervised baselines for ranking. Towards this goal, the authors propose a new technique for prompting LLMs for better leveraging their re-ranking capabilities. They also propose novel distillation technique for training smaller specialised re-ranking models using LLMs. \n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission studies a meaningful and interesting problem about whether LLMs like ChatGPT has the ability to perform passage ranking. The authors have put forth a two-fold proposition in their work. Firstly, they propose permutation generation as a technique to instruct LLMs to directly generate permutations of a set of passages. Secondly, they present a novel permutation distillation approach which aims to replicate the ranking capabilities of LLMs in a smaller, specialized model. This distillation technique showcases superior efficacy and efficiency in its performance. All concerns from reviewers have been resolved by the discussion during the rebuttal.
Decision: Accept-Main