Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models
Authors: Crystina Zhang, Sebastian Hofstätter, Patrick Lewis, Raphael Tang, Jimmy Lin


Reviewer paper summaries: The paper introduces a approach to construct effective listwise rerankers that are independent of GPT, using open-source LLMs. Through experiments on two TREC datasets, the proposed method demonstrates comparable performance to GPT-based models, while also highlighting the importance of high-quality listwise ranking data for training such rerankers and calling for future research to focus on developing these data resources.
Summary of strengths: 1. The authors implemented an LLM-based reranker through fine-tuning, which helps to lift the upper bound on the performance of LLM-based ranking models.
2. This paper not only conducts experiments on TREC DL 19 and TREC DL 20 datasets, but also conducts experiments on BEIR dataset to show how the generalization ability of the model is.
3. The setting of fine-tuning data is explained in detail in the paper to make the study reproducible.
Summary of weaknesses: 1. Experiments on BEIR benchmark show that the generalization ability of Rank-wo-GPT needs to be improved, which may limit the application scope of the method.
2. The ranking method used in this article is still the sliding window approach, which is the same as RankGPT and RankVicuna before. In the sliding window-based ranking strategy, the ranking in a window is dependent on the order of previous windows, and the ranking process within each window cannot be parallelized, which leads to the high time complexity.
3. Due to the randomness of the output of LLMs, the results of different ranking processes may have large differences. A sensitivity analysis of different initial orderings is therefore necessary, and I did not see this in this paper.
Comments, suggestions and typos: 1. The generalization of Rank-wo-GPT method needs to be improved.
2. Fine-tuning open source LLMs for ranking tasks may require a faster and more robust ranking strategy than sliding window.
3. If a sensitivity analysis to the initial order can be given and Rank-wo-GPT can show good robustness to different initial ordering of documents, I will improve my score.
Overall assessment scores: 3