Are Optimal Algorithms Still Optimal? Rethinking Sorting in LLM-Based Pairwise Ranking with Batching and Caching
Authors: Juan Wisznia, Cecilia Bolaños, Juan Tollo, Giovanni Franco Gabriel Marraffini, Agustín Andrés Gianolini, Noe Fabian Hsueh, Luciano Del Corro
Accepted at: acl25

Are Optimal Algorithms Still Optimal? Rethinking Sorting in LLM-Based Pairwise Ranking with Batching and Caching

Juan Wisznia1,3, Cecilia Bolaños1,2,
Juan Tollo1, Giovanni Marraffini1,3, Agustín Gianolini1,
Noe Hsueh1, Luciano Del Corro1,3

{jwisznia, cbolanos, jtollo, agianolini, nhsueh, ldelcorro}@dc.uba.ar, giovanni.marraffini@gmail.com

1Departamento de Computación, FCEyN, Universidad de Buenos Aires
2Instituto de Ciencias de la Computación, FCEyN, Universidad de Buenos Aires
3Lumina Labs*

Abstract

We introduce a novel framework for analyzing sorting algorithms in pairwise ranking prompting (PRP), re-centering the cost model around LLM inferences rather than traditional pairwise comparisons. While classical metrics based on comparison counts have traditionally been used to gauge efficiency, our analysis reveals that expensive LLM inferences overturn these predictions; accordingly, our framework encourages strategies such as batching and caching to mitigate inference costs. We show that algorithms optimal in the classical setting can lose efficiency when LLM inferences dominate the cost under certain optimizations.

1 Introduction

LLMs have ushered in a new era of language understanding (Brown et al., 2020). Alongside these developments, LLM-based reranking has emerged in the information retrieval (IR) domain (Nogueira et al., 2020; Zhuang et al., 2023; Ma et al., 2023; Sun et al., 2024). Instead of using custom fine-tuned rankers, off-the-shelf LLMs—often combined with a first-stage retriever can refine search results in a zero-shot manner. The practical significance of reranking is evident in its rapid commercial adoption, with major cloud platforms now offering it as a core functionality. LLM-based reranking enables robust ranking quality without the overhead of dataset-specific models, which is crucial, for example, for the widespread adoption of Retrieval-Augmented Generation across both cloud-based and on-prem deployments.

A notable exemplar in zero-shot LLM-based reranking is Pairwise Ranking Prompting (PRP) (Qin et al., 2024; Luo et al., 2024), which compares two candidate documents. Despite its conceptual elegance and model-agnostic nature, PRP faces significant computational challenges; in practice—each pairwise comparison requires an expensive LLM inference, making a naïve all-pairs approach prohibitively costly (Qin et al., 2024). This has prompted both researchers and practitioners to adopt classical sorting algorithms for minimizing the number of comparisons (Qin et al., 2024) as they offer theoretical guarantees.

We argue that classical analysis is not adequate for PRP as it treats each comparison as an atomic, uniform-cost operation, whereas in an LLM-based system, each comparison is an expensive inference call. This gap between classical and LLM-centric views can invert conventional wisdom under certain basic optimizations, causing algorithms that appear optimal under traditional assumptions to underperform in real-world scenarios, and vice versa.

To address these limitations, we introduce a framework that redefines how ranking algorithms are analyzed in an LLM context. Rather than merely counting comparisons, we focus on LLM inference calls as the primary cost driver. We show that basic optimizations—such as caching and batch inference—can significantly alter algorithms’ efficiency. Furthermore, we propose Quicksort as an efficient reranking algorithm, demonstrating its potential when leveraging these optimizations. To the best of our knowledge, this is the first time Quicksort has been applied in this context.

Caching and Batching have no effect on algorithm ranking performance; the exact same comparisons will be performed but much faster. While caching repeated queries and batching independent operations are seemingly trivial adaptations, they significantly affect the choice of the optimal algorithm challenging previous results (Qin et al., 2024; Zhuang et al., 2024). For instance, Heapsort is no longer the preferred choice. A mere batch size of 2 will result in Quicksort generating 44% less inference calls compared to Heapsort.

We validate our findings on standard ranking benchmarks (TREC DL 2019 and 2020 (Craswell
et al., 2020, 2021) and BEIR (Thakur et al., 2021)). By re-framing sorting theory around real-world LLM inference costs, we offer both practical guidance for zero-shot reranking and a theoretical basis for understanding algorithmic efficiency under modern IR constraints.

2 Related Work

Traditional IR systems require extensive labeled data and struggle with cross-domain generalization (Matveeva et al., 2006; Wang et al., 2011). LLMs have transformed this landscape by enabling zero-shot ranking. PRP emerged then as a particularly effective technique (Qin et al., 2024; Luo et al., 2024). PRP’s key advantage lies in its model-agnostic nature- by comparing document pairs through simple prompts, it can leverage any LLM without training or access to model internals, making it especially valuable as newer models emerge. However, PRP faces significant computational challenges as each pairwise comparison requires an expensive LLM inference, with costs scaling quadratically with document count.

To address these computational demands, recent work has incorporated sorting algorithms into the PRP framework (Qin et al., 2024; Zhuang et al., 2024). While theoretically well-grounded, these approaches adopt the cost framework of traditional sorting theory, where comparisons are treated as atomic operations with uniform costs. However, in LLM-based ranking, inferences are orders of magnitude more expensive than other operations. This mismatch between classical cost assumptions and LLM-specific characteristics suggests the need to reevaluate sorting algorithm selection and optimization for real-world performance.

3 Revisiting sorting algorithms

This section examines how small yet impactful optimizations (caching, batching, and top-k extraction) in the context of classical algorithms—Bubblesort, Quicksort, and Heapsort can significantly shift which algorithm is most efficient in LLM-based ranking. While these adaptations are not exhaustive, they demonstrate how our framework redefines efficiency based on LLM-specific costs, where reducing inference steps matters more than traditional complexity metrics. Importantly, these optimizations preserve the final ranking outcome: the same comparisons are performed but are batched or reused, leading to fewer inference calls and a much faster process. Table 1 summarizes the optimizations applicable to each algorithm.

<table>
  <tr>
    <th>Algorithm</th>
    <th>Batching</th>
    <th>Caching</th>
    <th>Top-k Efficiency</th>
  </tr>
  <tr>
    <td>Heapsort</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Bubblesort</td>
    <td>x</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>Quicksort</td>
    <td>✓</td>
    <td>x</td>
    <td>✓<sup>1</sup></td>
  </tr>
</table>

Table 1: Summary of optimization techniques under LLM-centric costs.

![Bubblesort with Caching. Solid arrows show inferences, dashed arrows cached comparisons.](page_755_682_393_180.png)

Figure 1: Bubblesort with Caching. Solid arrows show inferences, dashed arrows cached comparisons.

Heapsort has been favored in early PRP research Qin et al. (2024) for its \( O(n \log n) \) complexity and natural support for top-k extraction. However, it cannot be adapted to batching or caching due to its binary tree structure. Each comparison is inherently sequential and unique. This makes it impossible to group comparisons into a single inference step (batching) or to reuse prior results (caching) effectively.

Bubblesort has been considered expensive due to its \( O(n^2) \) complexity, but this can be adapted via caching from its repeated adjacent comparisons across passes (Figure 1). The memory overhead remains negligible, requiring only a small dictionary to store prior results. While its pairwise swap structure precludes batching (comparisons cannot be grouped into single inferences), it inherently supports top-k extraction (Qin et al., 2024), enhancing its practicality for ranking applications.

Quicksort uniquely enables batching through its partition phase, where multiple elements can be evaluated simultaneously against a pivot (Figure 2). However, it has limited potential for caching, as pivot comparisons are typically non-repeating. Despite this, the Partial Quicksort variant (Martinez, 2004) enhances its efficiency by enabling early termination for top-k extraction. To the best of our knowledge, we are the first to introduce Quicksort in PPR as prior research focused on Heapsort and Bubblesort due to their top-k properties.

1Using Partial Quicksort (Martinez, 2004).
Figure 2: a. One comparison per inference as classic analysis for these algorithms. b. Multiple comparisons per inference making gaining more information per inference.

4 Experimental Setup

Hardware: Our analysis is theoretical and agnostic to hardware. However, to validate that our cost assumptions align with practical throughput behavior, we ran two lightweight empirical checks on NVIDIA A100 (40GB), RTX 3090, and RTX 2080 Ti. These include single forward-pass latency measurements across batch sizes and GPUs, and full PRP reranking with Quicksort and Heapsort at batch sizes 2 and 128 for the A100 (see Section 5).

Metric: Instead of focusing on traditional comparison counts, we shifted to the number of LLM inference calls, which are the dominant computational cost. Each inference—regardless of token count or monetary cost—is treated as a uniform cost unit. We disregard token counts and dollar costs because these are determined by the dataset and pre-trained model. Moreover, standard pre-processing (e.g., chunking/truncation) ensures uniformity across documents. We show mean and standard deviation across datasets and LLMs. Individual results can be found in Appendix A.

LLMs: Following Qin et al. (2024); Zhuang et al. (2024) we used: Flan-T5-L (780M), Flan-T5-XL (3B), Flan-T5-XXL (11B) (Chung et al., 2022), Mistral-Instruct (7B) (Jiang et al., 2023), and Llama-3-Instruct (8B) (et al., 2024). For the latency analysis we implemented batch processing with Flan-T5-Large using the Hugging Face Transformers library (Wolf et al., 2020).

Algorithms: (1) Bubblesort, (2) Quicksort with median-of-three pivot strategy (other strategies are shown in Appendix A), and (3) Heapsort.

Datasets: TREC DL 2019 (43 queries) and 2020 (200) (Craswell et al., 2020, 2021) as well as subsets from BEIR (Thakur et al., 2021): Webis-Touche2020 (49), NFCorpus (295), Large-Scifact (300), TREC-COVID (50), FiQA (648), and DBpedia-Entity (400). Following standard practices, we re-ranked the top 100 BM25-retrieved documents per query (Robertson and Zaragoza, 2009; Qin et al., 2024; Zhuang et al., 2024; Luo et al., 2024) to identify the top-10 most relevant ones efficiently.

5 Results and Discussion

Cost model analysis: Figure 3 illustrates the number of inferences performed by Heapsort and Quicksort across different batch sizes. When the batch size is set to 1 (equivalent to counting individual comparisons), Heapsort emerges as the most efficient algorithm consistent with traditional sorting analysis and previous results (Qin et al., 2024; Zhuang et al., 2024). However, as the batch size increases, Quicksort is able to significantly outperform as multiple comparisons can be run in parallel. For instance, with a batch size of 2, the average number of inference calls is reduced already by almost 45%.

Figure 4 compares the number of inferences performed by Bubblesort with and without cache. Bubblesort benefits significantly more from caching at a minimal storage overhead. This is because Bubblesort involves repeated comparisons, many of which can be cached, reducing the total number of inferences by an average of 46%.

Importantly, despite these optimizations reducing the number of LLM inferences, they do not alter the final ranking outcome. The same comparisons are performed, but they are either batched together or retrieved from cache rather than recomputed, leading to fewer inference calls and a much faster process.

Latency Analysis: Figure 5 shows single-pass
Figure 3: Mean and SD inference count for Quicksort and Heapsort across batch sizes. Black number: Heapsort vs. Quicksort using batching gain; Green number: Quicksort batching vs. no batching gain.

Figure 4: Mean and SD inference count for Bubblesort, with and without cache. Green numbers indicate the percentage gain with cache. The dashed line represents the mean inference count for Heapsort and Quicksort.

speed-ups on A100, RTX 3090, and RTX 2080 Ti for different batch sizes. A100 achieves near-ideal scaling up to batch size 8, with throughput continuing to improve—albeit with diminishing returns—up to batch 128. On 3090 and 2080 Ti, ideal scaling occurs up to batch sizes 2 and 4, respectively, with throughput saturating between batch sizes 32 and 64. These results indicate that while theoretical efficiency peaks at larger batch sizes, practical efficiency is constrained by GPU architecture. The point at which near-ideal conditions are met before saturation sets in is GPU-dependent.

We also ran the full PRP pipeline over BEIR on the A100 using both batch size 2 and 128 for Quicksort and Heapsort. At batch 128, Quicksort is \(5.52 \times\) faster than Heapsort while achieving similar nDCG@10 (See Appendix A, Tables 2–3). Experiments show that the theoretical gains from batching and algorithmic design hold in end-to-end ranking performance.

Ranking Performance: Figure 6 shows that the ranking performance of all these algorithms across optimization settings remains relatively stable for a given dataset, allowing users to prioritize computational efficiency and hardware constraints before performance when choosing an algorithm.

Findings provide a detailed insight of sorting algorithms behavior in LLM-based pairwise ranking, highlighting their respective benefits and drawbacks, enabling users to select the most suitable algorithm based on their specific resources and requirements. More specifically:

Quicksort is ideal for latency-sensitive applications with batch sizes \( \geq 2 \), leveraging hardware parallelism to outperform alternatives.
Bubblesort achieves a substantial efficiency gain with caching. It’s remarkable performance in some datasets like scifact and touche2020 makes it a more competitive choice with the new adaptation. Bubblesort tends to be effective in the context of LLMs in which pairwise transitivity is not guaranteed. Pairwise adjacent comparisons seemed to be more stable and bring better results in the context of PRP (Luo et al., 2024).

Heapsort, once the gold standard for its theoretical logarithmic complexity, its advantage emerges only with no batching (rarely seen in LLM ranking).

![Line chart showing algorithms’ performance across datasets](page_186_573_627_246.png)

Figure 6: Algorithms’ performance across datasets.

6 Conclusion

We introduced a framework for optimizing sorting algorithms in LLM-based pairwise ranking by prioritizing inference calls over comparison counts. We found that classical efficiency assumptions break down under LLM workloads, revealing Quicksort as a natural, yet unexplored, choice of algorithm. This demonstrates that inference efficiency is a property deeply tied to algorithmic design. We hope this framework encourages further exploration of algorithms better aligned with LLM cost structures.

7 Limitations

While our work showcases the efficacy of batching and caching optimizations in mitigating the high inference costs of LLM-based pairwise ranking, certain limitations remain. First, sorting algorithms work best when transitivity in pairwise comparisons holds, but LLMs can yield inconsistent judgments for near-equivalent or context-sensitive documents. Addressing this inconsistency requires dedicated methods to detect and resolve intransitive preferences, which remains an open area of research. Future work could examine how much performance is degraded and whether ranking algorithms that do not assume transitivity can actually offer any practical advantage.

Additionally, although our experiments were limited to medium-sized LLMs for budgetary and computational reasons, larger models could further amplify the benefits observed here. Future research should explore how our framework performs with these more powerful models, potentially unlocking even greater gains in inference efficiency. Moreover, hybrid methods that unify the strengths of multiple algorithms, as well as active ranking strategies or noisy sorting algorithms (Mikhailuk et al., 2020; Bai and Coester, 2023), are fully compatible with our approach: they rely on additional computations separate from the LLM inferences themselves, thereby enabling more informed—and thus fewer—LLM queries. Ultimately, our findings underscore the need for ongoing algorithmic innovation that exploits LLM-specific cost structures, paving the way for more efficient, scalable, and broadly applicable ranking solutions.

References

Xingjian Bai and Christian Coester. 2023. Sorting with predictions.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharar Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models.

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the trec 2020 deep learning track.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the trec 2019 deep learning track.

Abhimanyu Dubey et al. 2024. The llama 3 herd of models.

C.A.R. Hoare. 1962. Quicksort. BCS, Computer Journal, 5(1):10–15.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b.

Jian Luo, Xuanang Chen, Ben He, and Le Sun. 2024. PRP-graph: Pairwise ranking prompting to LLMs with graph aggregation for effective text re-ranking. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 5766–5776, Bangkok, Thailand. Association for Computational Linguistics.

Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-shot listwise document reranking with a large language model.

Conrado Martinez. 2004. Partial quicksort. In In Proceedings of the 6th ACMSIAM Workshop on Algorithm Engineering and Experiments and 1st ACM-SIAM Workshop on Analytic Algorithmics and Combinatorics, pages 224–228.

Irina Matveeva, Chris Burges, Timo Burkard, Andy Lau- cius, and Leon Wong. 2006. High accuracy retrieval with multiple nested ranker. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’06, page 437–444, New York, NY, USA. Association for Computing Machinery.

Aliaksei Mikhailiuk, Clifford Wilmot, María Pérez-Ortiz, Dingcheng Yue, and Rafal Mantiuk. 2020. Active sampling for pairwise comparisons via approximate message passing and information gain maximization. CoRR, abs/2004.05691.

Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model.

Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. Large language models are effective text rankers with pairwise ranking prompting. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1504–1518, Mexico City, Mexico. Association for Computational Linguistics.

Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3:333–389.

Robert Sedgewick. 1975. Quicksort. Outstanding Dissertations in the Computer Sciences. Garland Publishing, New York.

Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2024. Is chatgpt good at search? investigating large language models as re-ranking agents.

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.

Lidan Wang, Jimmy Lin, and Donald Metzler. 2011. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’11, page 105–114, New York, NY, USA. Association for Computing Machinery.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing.

Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2023. Rankt5: Fine-tuning t5 for text ranking with ranking losses. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 2308–2313, New York, NY, USA. Association for Computing Machinery.

Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. 2024. A setwise approach for effective and highly efficient zero-shot ranking with large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, page 38–47. Association for Computing Machinery.
A Appendix

In this appendix, we present a comparison of different methods across the BEIR and TREC datasets. Each table follows the same structure and reports NDCG@10 (Normalized Discounted Cumulative Gain), the number of inferences, and the number of comparisons (#Inferences and #Comparisons) for each method. Also, latency (in seconds) is reported for Quicksort and Heapsort over the BEIR suite and corresponds to end-to-end PRP execution over each dataset on an A100 40GB GPU. Quicksort is analyzed using four pivot selection strategies: the original Hoare’s method, the middle-element selection, random and median-of-three strategies (Hoare, 1962; Sedgewick, 1975).

To highlight performance differences, we emphasize the best-performing algorithm for each dataset and LLM model in black, while the second-best is underlined. Additionally, the tables distinguish between two computational scenarios: (1) cached: The number of inferences made to the LLM. (2) non-cached: The number of comparisons performed using precomputed results, avoiding additional inferences. All results are presented with a batch size of 2 and 128 to show batch inference efficiency.

<table>
  <tr>
    <th rowspan="2"># Methods</th>
    <th colspan="3">dbpedia</th>
    <th colspan="3">nfcorpus</th>
    <th colspan="3">fiqa</th>
  </tr>
  <tr>
    <th>NDCG@10</th>
    <th>#comp</th>
    <th>#inf Lat.</th>
    <th>NDCG@10</th>
    <th>#comp</th>
    <th>#inf Lat.</th>
    <th>NDCG@10</th>
    <th>#comp</th>
    <th>#inf Lat.</th>
  </tr>
  <tr>
    <td>BM25</td>
    <td>0.318</td>
    <td>–</td>
    <td>–</td>
    <td>0.322</td>
    <td>–</td>
    <td>–</td>
    <td>0.240</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>heapsort #</td>
    <td>0.413</td>
    <td>225.1</td>
    <td>225.1 24.8</td>
    <td>0.335</td>
    <td>160.0</td>
    <td>160.0 18.4</td>
    <td><b>0.313</b></td>
    <td>200.3</td>
    <td>200.3 26.1</td>
  </tr>
  <tr>
    <td>quicksort (original, b=2)</td>
    <td>0.403</td>
    <td>245.3</td>
    <td>126.8 34.1</td>
    <td>0.321</td>
    <td>171.0</td>
    <td>88.9 16.8</td>
    <td>0.284</td>
    <td>253.3</td>
    <td>130.8 27.7</td>
  </tr>
  <tr>
    <td>quicksort (original, b=128)</td>
    <td>0.403</td>
    <td>245.3</td>
    <td>13.2 4.7</td>
    <td>0.321</td>
    <td>171.0</td>
    <td>11.1 3.0</td>
    <td>0.284</td>
    <td>253.3</td>
    <td>13.6 4.8</td>
  </tr>
  <tr>
    <td>quicksort (random, b=2)</td>
    <td>0.414</td>
    <td>236.9</td>
    <td>122.4 –</td>
    <td>0.322</td>
    <td>181.3</td>
    <td>94.1 –</td>
    <td>0.282</td>
    <td>246.0</td>
    <td>127.0 –</td>
  </tr>
  <tr>
    <td>quicksort (random, b=128)</td>
    <td>0.405</td>
    <td>241.5</td>
    <td>13.0 –</td>
    <td>0.322</td>
    <td>171.6</td>
    <td>10.9 –</td>
    <td>0.289</td>
    <td>246.6</td>
    <td>13.2 –</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=2)</td>
    <td>0.410</td>
    <td>231.9</td>
    <td>119.9 –</td>
    <td>0.315</td>
    <td>168.9</td>
    <td>87.8 –</td>
    <td>0.277</td>
    <td>243.2</td>
    <td>125.6 –</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=128)</td>
    <td>0.410</td>
    <td>231.9</td>
    <td>12.8 –</td>
    <td>0.315</td>
    <td>168.9</td>
    <td>10.9 –</td>
    <td>0.277</td>
    <td>243.2</td>
    <td>13.2 –</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=2)</td>
    <td>0.414</td>
    <td>255.2</td>
    <td>115.6 –</td>
    <td>0.326</td>
    <td>187.0</td>
    <td>83.7 –</td>
    <td>0.295</td>
    <td>284.9</td>
    <td>128.7 –</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=128)</td>
    <td>0.414</td>
    <td>255.2</td>
    <td>12.3 –</td>
    <td>0.326</td>
    <td>187.0</td>
    <td>10.5 –</td>
    <td>0.295</td>
    <td>284.9</td>
    <td>13.1 –</td>
  </tr>
  <tr>
    <td>bubblesort (classic)</td>
    <td>0.415</td>
    <td>777.6</td>
    <td>777.6 –</td>
    <td>0.343</td>
    <td>593.9</td>
    <td>593.9 –</td>
    <td>0.295</td>
    <td>662.1</td>
    <td>662.1 –</td>
  </tr>
  <tr>
    <td>bubblesort (cached)</td>
    <td><b>0.415</b></td>
    <td>777.6</td>
    <td>360.4 –</td>
    <td><b>0.343</b></td>
    <td>593.9</td>
    <td>242.2 –</td>
    <td>0.295</td>
    <td>662.1</td>
    <td>235.3 –</td>
  </tr>
  <tr>
    <td>heapsort</td>
    <td>0.419</td>
    <td>229.3</td>
    <td>229.3 –</td>
    <td>0.353</td>
    <td>144.9</td>
    <td>144.9 –</td>
    <td><b>0.361</b></td>
    <td>224.5</td>
    <td>224.5 –</td>
  </tr>
  <tr>
    <td>quicksort (original, b=2)</td>
    <td>0.404</td>
    <td>238.6</td>
    <td>123.3 –</td>
    <td>0.345</td>
    <td>160.6</td>
    <td>83.7 –</td>
    <td>0.338</td>
    <td>209.1</td>
    <td>108.3 –</td>
  </tr>
  <tr>
    <td>quicksort (original, b=128)</td>
    <td>0.404</td>
    <td>238.6</td>
    <td>12.7 –</td>
    <td>0.345</td>
    <td>160.6</td>
    <td>10.9 –</td>
    <td>0.338</td>
    <td>209.1</td>
    <td>11.9 –</td>
  </tr>
  <tr>
    <td>quicksort (random, b=2)</td>
    <td>0.412</td>
    <td>221.6</td>
    <td>114.7 –</td>
    <td>0.343</td>
    <td>168.7</td>
    <td>87.6 –</td>
    <td>0.345</td>
    <td>211.3</td>
    <td>109.5 –</td>
  </tr>
  <tr>
    <td>quicksort (random, b=128)</td>
    <td>0.411</td>
    <td>230.8</td>
    <td>12.4 –</td>
    <td>0.344</td>
    <td>159.0</td>
    <td>10.5 –</td>
    <td>0.338</td>
    <td>211.3</td>
    <td>12.0 –</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=2)</td>
    <td>0.410</td>
    <td>221.3</td>
    <td>114.5 –</td>
    <td>0.353</td>
    <td>157.6</td>
    <td>82.1 –</td>
    <td>0.341</td>
    <td>205.4</td>
    <td>106.5 –</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=128)</td>
    <td>0.410</td>
    <td>221.3</td>
    <td>12.2 –</td>
    <td><b>0.353</b></td>
    <td>157.6</td>
    <td>10.4 –</td>
    <td>0.341</td>
    <td>205.4</td>
    <td>11.9 –</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=2)</td>
    <td>0.413</td>
    <td>234.6</td>
    <td>106.3 –</td>
    <td>0.349</td>
    <td>184.9</td>
    <td>82.8 –</td>
    <td>0.357</td>
    <td>226.6</td>
    <td>102.7 –</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=128)</td>
    <td>0.413</td>
    <td>234.6</td>
    <td>11.6 –</td>
    <td>0.349</td>
    <td>184.9</td>
    <td>10.3 –</td>
    <td>0.357</td>
    <td>226.6</td>
    <td>11.1 –</td>
  </tr>
  <tr>
    <td>bubblesort (classic)</td>
    <td>0.420</td>
    <td>788.2</td>
    <td>788.2 –</td>
    <td>0.351</td>
    <td>443.8</td>
    <td>443.8 –</td>
    <td>0.355</td>
    <td>712.8</td>
    <td>712.8 –</td>
  </tr>
  <tr>
    <td>bubblesort (cached)</td>
    <td><b>0.420</b></td>
    <td>788.1</td>
    <td>376.0 –</td>
    <td>0.351</td>
    <td>443.8</td>
    <td>189.6 –</td>
    <td>0.355</td>
    <td>712.8</td>
    <td>332.5 –</td>
  </tr>
</table>

Table 2: Comparison of different methods across DBPedia, NFCorpus, and FiQA datasets.
<table>
  <tr>
    <th rowspan="2"># Methods</th>
    <th colspan="4">scifact</th>
    <th colspan="4">trec-covid</th>
    <th colspan="4">touche2020</th>
  </tr>
  <tr>
    <th>NDCG@10 #comp</th>
    <th>#inf Lat.</th>
    <th>NDCG@10 #comp</th>
    <th>#inf Lat.</th>
    <th>NDCG@10 #comp</th>
    <th>#inf Lat.</th>
  </tr>
  <tr>
    <td>BM25</td>
    <td>0.679</td>
    <td>–</td>
    <td>–</td>
    <td>0.595</td>
    <td>–</td>
    <td>–</td>
    <td>0.442</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>heapsort</td>
    <td>0.675</td>
    <td>222.4</td>
    <td>222.4</td>
    <td>26.9</td>
    <td>0.753</td>
    <td>241.0</td>
    <td>241.0</td>
    <td>28.0</td>
    <td>0.332</td>
    <td>221.0</td>
    <td>221.0</td>
    <td>26.0</td>
  </tr>
  <tr>
    <td>quicksort (original, b=2)</td>
    <td>0.579</td>
    <td>211.4</td>
    <td>109.5</td>
    <td>25.3</td>
    <td>0.752</td>
    <td>245.3</td>
    <td>126.9</td>
    <td>26.9</td>
    <td>0.268</td>
    <td>273.6</td>
    <td>141.0</td>
    <td>34.9</td>
  </tr>
  <tr>
    <td>quicksort (original, b=128)</td>
    <td>0.579</td>
    <td>211.4</td>
    <td>12.0</td>
    <td>4.3</td>
    <td>0.752</td>
    <td>245.3</td>
    <td>13.6</td>
    <td>5.2</td>
    <td>0.268</td>
    <td>273.6</td>
    <td>13.9</td>
    <td>5.6</td>
  </tr>
  <tr>
    <td>quicksort (random, b=2)</td>
    <td>0.596</td>
    <td>224.7</td>
    <td>116.2</td>
    <td>–</td>
    <td>0.759</td>
    <td>243.8</td>
    <td>126.0</td>
    <td>–</td>
    <td>0.270</td>
    <td>275.2</td>
    <td>142.0</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (random, b=128)</td>
    <td>0.611</td>
    <td>218.5</td>
    <td>12.3</td>
    <td>–</td>
    <td>0.755</td>
    <td>243.7</td>
    <td>13.6</td>
    <td>–</td>
    <td>0.256</td>
    <td>273.0</td>
    <td>13.4</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=2)</td>
    <td>0.597</td>
    <td>211.2</td>
    <td>109.4</td>
    <td>–</td>
    <td>0.763</td>
    <td>235.5</td>
    <td>121.8</td>
    <td>–</td>
    <td>0.269</td>
    <td>253.2</td>
    <td>130.8</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=128)</td>
    <td>0.597</td>
    <td>211.2</td>
    <td>11.8</td>
    <td>–</td>
    <td><b>0.763</b></td>
    <td>235.5</td>
    <td>13.0</td>
    <td>–</td>
    <td>0.269</td>
    <td>253.2</td>
    <td>13.4</td>
    <td>–</td>
  </tr>
  <tr>
    <td>Plan-15-l</td>
    <td>quicksort (median of three, b=2)</td>
    <td>0.637</td>
    <td>237.1</td>
    <td>107.6</td>
    <td>–</td>
    <td>0.763</td>
    <td>256.0</td>
    <td>115.1</td>
    <td>–</td>
    <td>0.274</td>
    <td>289.4</td>
    <td>131.6</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=128)</td>
    <td>0.637</td>
    <td>237.1</td>
    <td>11.4</td>
    <td>–</td>
    <td>0.763</td>
    <td>256.0</td>
    <td>12.9</td>
    <td>–</td>
    <td>0.274</td>
    <td>289.4</td>
    <td>13.0</td>
    <td>–</td>
  </tr>
  <tr>
    <td>bubblesort (classic)</td>
    <td><b>0.692</b></td>
    <td>805.7</td>
    <td>805.7</td>
    <td>–</td>
    <td>0.718</td>
    <td>890.1</td>
    <td>890.1</td>
    <td>–</td>
    <td><b>0.447</b></td>
    <td>845.4</td>
    <td>845.4</td>
    <td>–</td>
  </tr>
  <tr>
    <td>bubblesort (cached)</td>
    <td><b>0.692</b></td>
    <td>805.7</td>
    <td>284.9</td>
    <td>–</td>
    <td>0.718</td>
    <td>890.1</td>
    <td>437.9</td>
    <td>–</td>
    <td><b>0.447</b></td>
    <td>845.4</td>
    <td>332.8</td>
    <td>–</td>
  </tr>
  <tr>
    <td>Plan-15-xl</td>
    <td>heapsort</td>
    <td>0.710</td>
    <td>197.5</td>
    <td>197.5</td>
    <td>–</td>
    <td><b>0.783</b></td>
    <td>249.5</td>
    <td>249.5</td>
    <td>–</td>
    <td>0.284</td>
    <td>244.3</td>
    <td>244.3</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (original, b=2)</td>
    <td>0.634</td>
    <td>206.9</td>
    <td>107.2</td>
    <td>–</td>
    <td>0.761</td>
    <td>225.4</td>
    <td>116.6</td>
    <td>–</td>
    <td>0.261</td>
    <td>234.7</td>
    <td>121.3</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (original, b=128)</td>
    <td>0.634</td>
    <td>206.9</td>
    <td>11.6</td>
    <td>–</td>
    <td>0.761</td>
    <td>225.4</td>
    <td>12.5</td>
    <td>–</td>
    <td>0.261</td>
    <td>234.7</td>
    <td>12.6</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (random, b=2)</td>
    <td>0.646</td>
    <td>219.0</td>
    <td>113.3</td>
    <td>–</td>
    <td>0.777</td>
    <td>243.4</td>
    <td>125.6</td>
    <td>–</td>
    <td>0.285</td>
    <td>216.3</td>
    <td>112.1</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (random, b=128)</td>
    <td>0.639</td>
    <td>211.8</td>
    <td>12.0</td>
    <td>–</td>
    <td>0.772</td>
    <td>222.4</td>
    <td>12.8</td>
    <td>–</td>
    <td>0.265</td>
    <td>236.7</td>
    <td>12.9</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=2)</td>
    <td>0.642</td>
    <td>200.4</td>
    <td>103.9</td>
    <td>–</td>
    <td>0.777</td>
    <td>239.4</td>
    <td>123.6</td>
    <td>–</td>
    <td>0.277</td>
    <td>214.7</td>
    <td>111.0</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=128)</td>
    <td>0.642</td>
    <td>200.4</td>
    <td>11.5</td>
    <td>–</td>
    <td>0.777</td>
    <td>239.4</td>
    <td>12.4</td>
    <td>–</td>
    <td>0.277</td>
    <td>214.7</td>
    <td>11.9</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=2)</td>
    <td>0.663</td>
    <td>235.1</td>
    <td>106.7</td>
    <td>–</td>
    <td>0.775</td>
    <td>250.3</td>
    <td>113.3</td>
    <td>–</td>
    <td>0.283</td>
    <td>232.0</td>
    <td>105.1</td>
    <td>–</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=128)</td>
    <td>0.663</td>
    <td>235.1</td>
    <td>11.4</td>
    <td>–</td>
    <td>0.775</td>
    <td>250.3</td>
    <td>12.4</td>
    <td>–</td>
    <td>0.283</td>
    <td>232.0</td>
    <td>11.4</td>
    <td>–</td>
  </tr>
  <tr>
    <td>bubblesort (classic)</td>
    <td>0.713</td>
    <td>581.9</td>
    <td>581.9</td>
    <td>–</td>
    <td>0.748</td>
    <td>874.5</td>
    <td>874.5</td>
    <td>–</td>
    <td>0.428</td>
    <td>869.4</td>
    <td>869.4</td>
    <td>–</td>
  </tr>
  <tr>
    <td>bubblesort (cached)</td>
    <td><b>0.713</b></td>
    <td>581.9</td>
    <td>217.9</td>
    <td>–</td>
    <td>0.748</td>
    <td>874.5</td>
    <td>510.9</td>
    <td>–</td>
    <td><b>0.428</b></td>
    <td>869.4</td>
    <td>467.7</td>
    <td>–</td>
  </tr>
</table>

Table 3: Comparison of different methods across SciFact, TREC-COVID, and Touche2020 datasets.
<table>
  <tr>
    <th rowspan="2">#</th>
    <th rowspan="2">Methods</th>
    <th colspan="3">TREC DL 2019</th>
    <th colspan="3">TREC DL 2020</th>
  </tr>
  <tr>
    <th>NDCG@10</th>
    <th>#Comparisons</th>
    <th>#Inferences</th>
    <th>NDCG@10</th>
    <th>#Comparisons</th>
    <th>#Inferences</th>
  </tr>
  <tr>
    <td></td>
    <td>BM25</td>
    <td>0.510</td>
    <td>–</td>
    <td>–</td>
    <td>0.479</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td rowspan="8">Flan-4.5-large</td>
    <td>heapsort</td>
    <td>0.650</td>
    <td>230.9</td>
    <td>230.9</td>
    <td><b>0.626</b></td>
    <td>226.5</td>
    <td>226.5</td>
  </tr>
  <tr>
    <td>quicksort (original, b=2)</td>
    <td>0.637</td>
    <td>249.0</td>
    <td>128.8</td>
    <td>0.588</td>
    <td>237.1</td>
    <td>122.7</td>
  </tr>
  <tr>
    <td>quicksort (original, b=128)</td>
    <td>0.637</td>
    <td>249.0</td>
    <td>14.1</td>
    <td>0.588</td>
    <td>237.1</td>
    <td>13.5</td>
  </tr>
  <tr>
    <td>quicksort (random, b=2)</td>
    <td>0.639</td>
    <td>236.7</td>
    <td>122.3</td>
    <td>0.587</td>
    <td>236.7</td>
    <td>122.4</td>
  </tr>
  <tr>
    <td>quicksort (random, b=128)</td>
    <td>0.650</td>
    <td>260.5</td>
    <td>13.7</td>
    <td>0.580</td>
    <td>240.0</td>
    <td>12.8</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=2)</td>
    <td>0.650</td>
    <td>231.1</td>
    <td>119.6</td>
    <td>0.594</td>
    <td>235.5</td>
    <td>121.8</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=128)</td>
    <td>0.650</td>
    <td>231.1</td>
    <td>13.5</td>
    <td>0.594</td>
    <td>235.5</td>
    <td>13.0</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=2)</td>
    <td>0.650</td>
    <td>276.0</td>
    <td>124.8</td>
    <td>0.600</td>
    <td>259.5</td>
    <td>117.0</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=128)</td>
    <td><b>0.650</b></td>
    <td>276.0</td>
    <td>13.2</td>
    <td>0.600</td>
    <td>259.5</td>
    <td>12.8</td>
  </tr>
  <tr>
    <td>quicksort (classic)</td>
    <td>0.634</td>
    <td>843.7</td>
    <td>843.7</td>
    <td>0.586</td>
    <td>777.2</td>
    <td>777.2</td>
  </tr>
  <tr>
    <td>bubblesort (cached)</td>
    <td>0.634</td>
    <td>843.7</td>
    <td>388.3</td>
    <td>0.586</td>
    <td>777.2</td>
    <td>357.1</td>
  </tr>
  <tr>
    <td rowspan="8">Flan-4.5-xl</td>
    <td>heapsort</td>
    <td><b>0.706</b></td>
    <td>242.0</td>
    <td>242.0</td>
    <td><b>0.689</b></td>
    <td>244.9</td>
    <td>244.9</td>
  </tr>
  <tr>
    <td>quicksort (original, b=2)</td>
    <td>0.697</td>
    <td>266.6</td>
    <td>137.5</td>
    <td>0.672</td>
    <td>250.6</td>
    <td>129.3</td>
  </tr>
  <tr>
    <td>quicksort (original, b=128)</td>
    <td>0.697</td>
    <td>266.6</td>
    <td>13.9</td>
    <td>0.672</td>
    <td>250.6</td>
    <td>12.9</td>
  </tr>
  <tr>
    <td>quicksort (random, b=2)</td>
    <td>0.694</td>
    <td>232.3</td>
    <td>120.2</td>
    <td>0.676</td>
    <td>239.0</td>
    <td>123.5</td>
  </tr>
  <tr>
    <td>quicksort (random, b=128)</td>
    <td>0.697</td>
    <td>257.3</td>
    <td>13.2</td>
    <td>0.676</td>
    <td>237.6</td>
    <td>12.7</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=2)</td>
    <td>0.703</td>
    <td>230.5</td>
    <td>119.4</td>
    <td>0.668</td>
    <td>232.9</td>
    <td>120.5</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=128)</td>
    <td>0.703</td>
    <td>230.5</td>
    <td>13.1</td>
    <td>0.668</td>
    <td>232.9</td>
    <td>12.6</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=2)</td>
    <td>0.696</td>
    <td>243.5</td>
    <td>110.5</td>
    <td>0.682</td>
    <td>239.6</td>
    <td>108.9</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=128)</td>
    <td>0.696</td>
    <td>243.5</td>
    <td>11.9</td>
    <td>0.682</td>
    <td>239.6</td>
    <td>11.8</td>
  </tr>
  <tr>
    <td>quicksort (classic)</td>
    <td>0.684</td>
    <td>887.8</td>
    <td>887.8</td>
    <td>0.670</td>
    <td>869.5</td>
    <td>869.5</td>
  </tr>
  <tr>
    <td>bubblesort (cached)</td>
    <td>0.684</td>
    <td>887.8</td>
    <td>544.9</td>
    <td>0.670</td>
    <td>869.5</td>
    <td>542.6</td>
  </tr>
  <tr>
    <td rowspan="8">Flan-4.5-xxl</td>
    <td>heapsort</td>
    <td><b>0.702</b></td>
    <td>238.9</td>
    <td>238.9</td>
    <td>0.688</td>
    <td>239.4</td>
    <td>239.4</td>
  </tr>
  <tr>
    <td>quicksort (original, b=2)</td>
    <td>0.677</td>
    <td>265.9</td>
    <td>137.0</td>
    <td>0.680</td>
    <td>234.7</td>
    <td>121.3</td>
  </tr>
  <tr>
    <td>quicksort (original, b=128)</td>
    <td>0.677</td>
    <td>265.9</td>
    <td>13.6</td>
    <td>0.680</td>
    <td>234.7</td>
    <td>12.7</td>
  </tr>
  <tr>
    <td>quicksort (random, b=2)</td>
    <td>0.691</td>
    <td>239.4</td>
    <td>124.0</td>
    <td>0.678</td>
    <td>228.3</td>
    <td>117.9</td>
  </tr>
  <tr>
    <td>quicksort (random, b=128)</td>
    <td>0.685</td>
    <td>244.7</td>
    <td>12.8</td>
    <td>0.674</td>
    <td>227.4</td>
    <td>12.3</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=2)</td>
    <td>0.688</td>
    <td>226.3</td>
    <td>117.0</td>
    <td>0.677</td>
    <td>229.2</td>
    <td>118.5</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=128)</td>
    <td>0.688</td>
    <td>226.3</td>
    <td>12.3</td>
    <td>0.677</td>
    <td>229.2</td>
    <td>12.3</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=2)</td>
    <td>0.686</td>
    <td>254.8</td>
    <td>116.3</td>
    <td>0.688</td>
    <td>230.4</td>
    <td>104.7</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=128)</td>
    <td>0.686</td>
    <td>254.8</td>
    <td>11.9</td>
    <td><b>0.688</b></td>
    <td>230.4</td>
    <td>11.4</td>
  </tr>
  <tr>
    <td>quicksort (classic)</td>
    <td>0.679</td>
    <td>866.2</td>
    <td>866.2</td>
    <td>0.680</td>
    <td>827.1</td>
    <td>827.1</td>
  </tr>
  <tr>
    <td>bubblesort (cached)</td>
    <td>0.679</td>
    <td>866.2</td>
    <td>532.1</td>
    <td>0.680</td>
    <td>827.1</td>
    <td>465.0</td>
  </tr>
  <tr>
    <td rowspan="8">Meta-Diann-38B-Instruct</td>
    <td>heapsort</td>
    <td>0.662</td>
    <td>235.0</td>
    <td>235.0</td>
    <td><b>0.615</b></td>
    <td>231.9</td>
    <td>231.9</td>
  </tr>
  <tr>
    <td>quicksort (original, b=2)</td>
    <td>0.645</td>
    <td>266.5</td>
    <td>137.4</td>
    <td>0.576</td>
    <td>235.5</td>
    <td>121.8</td>
  </tr>
  <tr>
    <td>quicksort (original, b=128)</td>
    <td>0.645</td>
    <td>266.5</td>
    <td>13.5</td>
    <td>0.576</td>
    <td>235.5</td>
    <td>12.9</td>
  </tr>
  <tr>
    <td>quicksort (random, b=2)</td>
    <td><b>0.663</b></td>
    <td>231.3</td>
    <td>119.8</td>
    <td>0.580</td>
    <td>231.7</td>
    <td>119.8</td>
  </tr>
  <tr>
    <td>quicksort (random, b=128)</td>
    <td>0.660</td>
    <td>219.0</td>
    <td>12.8</td>
    <td>0.585</td>
    <td>232.9</td>
    <td>12.8</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=2)</td>
    <td>0.640</td>
    <td>220.9</td>
    <td>114.4</td>
    <td>0.564</td>
    <td>228.1</td>
    <td>118.0</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=128)</td>
    <td>0.640</td>
    <td>220.9</td>
    <td>12.6</td>
    <td>0.564</td>
    <td>228.1</td>
    <td>12.6</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=2)</td>
    <td>0.650</td>
    <td>236.0</td>
    <td>106.3</td>
    <td>0.594</td>
    <td>244.3</td>
    <td>110.1</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=128)</td>
    <td>0.650</td>
    <td>236.0</td>
    <td>12.0</td>
    <td>0.594</td>
    <td>244.3</td>
    <td>12.3</td>
  </tr>
  <tr>
    <td>quicksort (classic)</td>
    <td>0.641</td>
    <td>822.5</td>
    <td>822.5</td>
    <td>0.600</td>
    <td>797.6</td>
    <td>797.6</td>
  </tr>
  <tr>
    <td>bubblesort (cached)</td>
    <td>0.641</td>
    <td>822.5</td>
    <td>389.4</td>
    <td>0.600</td>
    <td>797.6</td>
    <td>365.9</td>
  </tr>
  <tr>
    <td rowspan="8">Mistral-7B-Instruct-v1</td>
    <td>heapsort</td>
    <td>0.559</td>
    <td>200.3</td>
    <td>200.3</td>
    <td>0.513</td>
    <td>190.1</td>
    <td>190.1</td>
  </tr>
  <tr>
    <td>quicksort (original, b=2)</td>
    <td>0.578</td>
    <td>278.9</td>
    <td>143.7</td>
    <td>0.529</td>
    <td>276.2</td>
    <td>142.2</td>
  </tr>
  <tr>
    <td>quicksort (original, b=128)</td>
    <td>0.578</td>
    <td>278.9</td>
    <td>14.0</td>
    <td>0.529</td>
    <td>276.2</td>
    <td>13.4</td>
  </tr>
  <tr>
    <td>quicksort (random, b=2)</td>
    <td>0.593</td>
    <td>293.8</td>
    <td>150.9</td>
    <td>0.511</td>
    <td>271.5</td>
    <td>139.8</td>
  </tr>
  <tr>
    <td>quicksort (random, b=128)</td>
    <td>0.573</td>
    <td>279.7</td>
    <td>12.9</td>
    <td>0.524</td>
    <td>263.9</td>
    <td>13.2</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=2)</td>
    <td>0.595</td>
    <td>257.3</td>
    <td>132.7</td>
    <td>0.531</td>
    <td>249.3</td>
    <td>128.6</td>
  </tr>
  <tr>
    <td>quicksort (middle, b=128)</td>
    <td>0.595</td>
    <td>257.3</td>
    <td>13.5</td>
    <td>0.531</td>
    <td>249.3</td>
    <td>12.9</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=2)</td>
    <td><b>0.612</b></td>
    <td>292.1</td>
    <td>132.5</td>
    <td>0.538</td>
    <td>292.5</td>
    <td>133.0</td>
  </tr>
  <tr>
    <td>quicksort (median of three, b=128)</td>
    <td><b>0.612</b></td>
    <td>292.1</td>
    <td>13.1</td>
    <td>0.538</td>
    <td>292.5</td>
    <td>12.9</td>
  </tr>
  <tr>
    <td>quicksort (classic)</td>
    <td>0.587</td>
    <td>631.0</td>
    <td>631.0</td>
    <td>0.539</td>
    <td>578.7</td>
    <td>578.7</td>
  </tr>
  <tr>
    <td>bubblesort (cached)</td>
    <td>0.587</td>
    <td>631.0</td>
    <td>250.5</td>
    <td><b>0.539</b></td>
    <td>578.7</td>
    <td>223.4</td>
  </tr>
</table>

Table 4: Comparison of methods for TREC DL 2019 and TREC DL 2020.

Reviewer paper summaries: This paper challenges conventional assumptions about sorting algorithm efficiency in the context of Large Language Model (LLM) based pairwise ranking. The authors argue that while classical sorting theory focuses on minimizing comparison counts, LLM-based ranking should prioritize reducing LLM inference calls, which are significantly more expensive. The paper introduces a framework that considers batching and caching as key optimizations. The authors demonstrate that these optimizations can dramatically shift which sorting algorithms perform best. They find that Quicksort with batching outperforms Heapsort (previously considered optimal) when batch size ≥ 2, and Bubblesort with caching becomes competitive despite its traditionally worse complexity.

This paper rethinks the efficiency of classical sorting algorithms in the context of LLM-based pairwise ranking. It introduces a new framework that treats LLM inference calls—not comparison counts—as the primary cost. The authors show that with optimizations like batching and caching, traditionally suboptimal algorithms (e.g., Quicksort and Bubblesort) can outperform Heapsort, which was previously preferred.
Summary of strengths: 1.The paper addresses an important practical issue in LLM-based reranking by reconsidering algorithm efficiency in terms of real-world costs (LLM inference calls).
2. The proposed framework provides clear, practical insights for implementing pairwise ranking systems with LLMs.
3. The paper's experiment design effectively demonstrates the impact of each optimization technique across different algorithms.

- this paper is very clearly written in its motivation and solution

- the viewpoint of this work is novel and interesting. Previous works usually use number of comparison to measure cost, but with nowadays vllm etc, batching and caching can make the real efficiency different.

- the comparison among the sorting algorithms are solid.
Summary of weaknesses: 1. The visualization in Figure 3 could be improved, as the first three columns (Quicksort without batching, Quicksort with batching, and Heapsort without batching) show the same values.
2. The analysis lacks consideration of the computational overhead introduced by larger batch sizes. While batching reduces the number of inference calls, it typically increases computation time per batch. A more comprehensive analysis would include actual time measurements to verify the real-world efficiency gains.

- it worth to have some real latency measurement using the implementation with batching and caching, it will makes the results and proposed method more convincing.
Comments, suggestions and typos: 1. I would recommend including timing measurements in addition to inference counts to provide a more comprehensive picture of real-world performance, especially for the batching experiments in Figure 3 and Table 2.
2. Consider redesigning the bar plot in Figure 3.

N/A
Overall assessment scores: 3

4