Title: AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking
Venue: NeurIPS 2025
Accepted: yes
Keywords: Listwise Reranking

Large Language Models

Adaptive Computation

Uncertainty Estimation

Inference Efficiency
Abstract: Listwise reranking with large language models (LLMs) enhances top-ranked results in retrieval-based applications.
Due to the limit in context size and high inference cost of long context, reranking is typically performed over a fixed size of small subsets, with the final ranking aggregated from these partial results.
This fixed computation disregards query difficulty and document distribution, leading to inefficiencies. 
We propose AcuRank, an adaptive reranking framework that dynamically adjusts both the amount and target of computation based on uncertainty estimates over document relevance. 
Using a Bayesian TrueSkill model, we iteratively refine relevance estimates until reaching sufficient confidence levels, and our explicit modeling of ranking uncertainty enables principled control over reranking behavior and avoids unnecessary updates to confident predictions.
Results on the TREC-DL and BEIR benchmarks show that our method consistently achieves a superior accuracy–efficiency trade-off and scales better with compute than fixed-computation baselines.
These results highlight the effectiveness and generalizability of our method across diverse retrieval tasks and LLM-based reranking models.

AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking

Abstract

Listwise reranking with large language models (LLMs) enhances top-ranked results in retrieval-based applications. Due to the limit in context size and high inference cost of long context, reranking is typically performed over a fixed size of small subsets, with the final ranking aggregated from these partial results. This fixed computation disregards query difficulty and document distribution, leading to inefficiencies. We propose AcuRank, an adaptive reranking framework that dynamically adjusts both the amount and target of computation based on uncertainty estimates over document relevance. Using a Bayesian TrueSkill model, we iteratively refine relevance estimates until reaching sufficient confidence levels, and our explicit modeling of ranking uncertainty enables principled control over reranking behavior and avoids unnecessary updates to confident predictions. Results on the TREC-DL and BEIR benchmarks show that our method consistently achieves a superior accuracy-efficiency trade-off and scales better with compute than fixed-computation baselines. These results highlight the effectiveness and generalizability of our method across diverse retrieval tasks and LLM-based reranking models. 3

1 Introduction

Modern information retrieval pipelines, such as web search and retrieval-augmented generation (RAG) systems, typically adopt a fast first-phase retriever, selecting a set of broadly relevant documents, like BM25 [1] or dense encoders [2] optimized for recall and speed. As they often produce noisy or suboptimal rankings, reranking is critical in applications where precision at the top is essential, including conversational agents and reasoning with large language models (LLMs) [3, 4].

Reranking methods can be broadly categorized by how they model document interactions and handle relative relevance. Pointwise methods [5] independently assign scores to each document, offering scalability but failing to consider the competitive context among candidates. Pairwise methods [6] improve upon this by comparing document pairs to capture local preferences, but often struggle to maintain coherent global rankings. Listwise methods [7, 8] instead evaluate a group of documents jointly, particularly effective for out-of-domain [9] or low-resource [10] settings. Recent work has shown that LLMs are especially effective as listwise rerankers, due to their ability to capture complex reasoning and fine-grained distinctions across documents [7, 11, 12].

(b)Select documents with high

(a)Initialize with first-stageretrieval

rankinguncertainty,updatescores

(c) Final Ranking

ProbabilityDensity

C

Top-1

TrueSkill

Top-2

Stopping

Criteria

Relevance

Relevance

Relevance

Despite their effectiveness, applying LLMs in listwise reranking is computationally expensive. As each LLM reranker call can process only a small group of candidates due to input length limits, covering the full list typically requires multiple calls. Prior work has proposed fixed-computation strategies such as sliding windows [13] and tournament-style reranking [14] to balance ranking effectiveness with computational cost. While these fixed strategies improve efficiency, they suffer from rigidity and lack of adaptivity by assigning a fixed number of reranker calls to predetermined candidate positions, regardless of the complexity or ambiguity of the query. As a result, documents ranked low in early iterations may never be reconsidered, even when initial decisions are based on limited or noisy context. Moreover, these methods do not leverage intermediate signals from previous reranking steps and cannot dynamically focus computation on uncertain cases where refinement is most needed.

To address these limitations, we propose AcuRank , a framework for listwise reranking with U ncertainty-aware A daptive C omputation (rearranged for pronunciation). Our method adaptively allocates computation based on each document's relevance uncertainty relative to the topk boundary, rather than treating all documents equally. As illustrated in Figure 1, we adopt TrueSkill [15], a Bayesian rating system, to maintain probabilistic relevance estimates (Figure 1(a)) and iteratively update them as reranking results are observed. At each step, only uncertain candidates are selected for reranking (Figure 1(b)), while confident ones are skipped. This targeted refinement continues until convergence (Figure 1(c)), improving both accuracy and computational efficiency by focusing resources on the most ambiguous cases.

We evaluate our method on the TREC Deep Learning [16] and BEIR [17] benchmarks using multiple LLM-based rerankers. Our approach consistently improves ranking quality while requiring fewer reranker calls across diverse datasets and reranker models, including both in-domain and out-of-domain settings. These results demonstrate the effectiveness of uncertainty-aware adaptive computation and the broad applicability of our method.

Our contributions are as follows:

We introduce probabilistic relevance modeling based on TrueSkill, which enables estimation of uncertainty in document rankings.

Building on this, we develop a novel listwise reranking framework that supports adaptive computation by selectively reranking uncertain candidates through iterative refinements.

Through extensive experiments, our method consistently improves the accuracy-efficiency tradeoff across benchmarks and generalizes well across datasets and reranker models.

2 Related work

2.1 Fixed-computation reranking strategies

Pointwise, pairwise, and setwise. Pointwise methods [18-21] score each document independently, treating reranking as a classification problem. They are efficient and compatible with neural encoders and LLMs but cannot model cross-document dependencies, limiting their effectiveness in complex

scenarios. Pairwise methods [22, 6, 23] compare two documents at a time to model pairwise dependency, but require n 2 number of reranker calls to rank n documents. Setwise methods [24] are typically compared against pairwise methods, which rerank small fixed-size sets to balance between pointwise simplicity and listwise interaction. While efficient, they typically view only one or two passages at a time, limiting their ability to model subtle inter-document relationships visible only in larger listwise contexts. A concurrent OpenReview submission explores a budget-constrained pairwise reranking strategy that selects informative comparisons using TrueSkill-based uncertainty sampling; unlike our approach, it operates under a fixed pairwise budget rather than allocating computation adaptively over listwise contexts [25].

Listwise - sliding windows. Sliding window methods rerank overlapping subsets of candidates iteratively, starting from the bottom window of the list and progressively moving to the next window, merging top results upward. This approach has been widely adopted in LLM-based reranking systems such as LRL [13], RankGPT [11], RankVicuna [8], and RankZephyr [12], which use zero-shot prompts to perform reranking over windowed chunks. While effective, fixed window sizes and traversal paths lead to inefficiencies, by incurring over-computation for simple queries with little ambiguity and under-computation for complex queries.

Listwise - tournament-style. Tournament-style methods rerank by partitioning candidates into small groups and performing listwise reranking in iterative rounds. TDPart [26] employs a top-down partitioning strategy to minimize redundant inference and enhance parallel processing. In each round, listwise reranking is applied to the groups, and the top-ranked documents advance. This process repeats over progressively smaller pools until the final ranking is achieved. While TDPart focuses on efficiency, ListT5 [27] and TourRank [14] leverage T5 models and LLMs for groupwise comparisons, achieving competitive ranking accuracy. However, like sliding window methods, they are limited by fixed schedules, leading to over-computation for straightforward queries and under-computation for complex, ambiguous ones.

Complementary to scheduling-based approaches, CoRanking [28] integrates rerankers of different model sizes to reduce end-to-end cost, while DemoRank [29] explores demonstration selection tailored for ranking tasks. Both are orthogonal to our uncertainty-driven adaptive computation, yet potentially compatible with it.

2.2 Uncertainty and adaptivity in retrieval

Uncertainty estimation in retrieval has emerged as a principled method for enhancing reliability in retrieval. Cohen et al. [30] introduce a Bayesian framework that interprets relevance scores in deep retrieval models as stochastic estimates rather than deterministic values. Using techniques like dropout-based variational inference, they capture predictive uncertainty, leading to improvements in robustness, calibration, and resilience to adversarial inputs. Their findings expose the risks associated with overconfident ranking models and highlight the need for uncertainty-aware mechanisms in downstream tasks, such as filtering and reweighting. While post-hoc calibration over static retrieval outputs is allowed in their problem context, our work requires online computation for the iterative reranking setting. Our distinction lies in integrating the rigor of Bayesian models, inspired by TrueSkill [15], into adaptive computation, thereby making our approach both theoretically grounded and practically efficient. Meanwhile, prior adaptive computation methods [31] in retrieval are largely confined to statistical ranking scenarios [32] that depend on separate estimators for judging relevance thresholds or query difficulty [33-36].

In contrast, our method unifies uncertainty estimation and adaptive computation in a single framework: we use uncertainty not only as a measure of model confidence, but also as a dynamic signal that guides the allocation of computation. This enables fine-grained reranking that selectively refines ambiguous documents while avoiding unnecessary inference on confident ones.

3 Method

Webegin by formalizing the reranking task (§3.1) and introducing our approach to relevance modeling and uncertainty quantification using TrueSkill (§3.2). Building on this foundation, we then describe the components of our reranking algorithm with adaptive computation, AcuRank (§3.3, Fig. 1).

While conventional reranking methods rely on a fixed number of reranker calls per query (§2.1), our method adaptively determines both which documents to rerank and how much computation to allocate, based on evolving document-level uncertainty estimates. This enables focused refinement of uncertain candidates, improving both computational efficiency and ranking effectiveness by avoiding unnecessary updates to already confidently ranked documents.

3.1 Problem formulation

Given a query q and a set of retrieved documents D = { D 1 , . . . , D n } from a first-stage retriever such as BM25, the goal is to reorder the documents into a ranked list [ D r 1 > · · · > D r n ] or to extract a topk list [ D r 1 > · · · > D r k ] , where { r 1 , . . . , r n } is a permutation of { 1 , . . . , n } .

We assume access to a listwise reranker g ( D ′ ; M ) , where D ′ is an ordered subset of D and M denotes the underlying reranking model. This is done by prompting an instruction-tuned or zero-shot LLM, and the reranker returns a new ordering over the documents in D ′ .

In practice, n is often large (e.g., 100-1000), making it infeasible to rerank all retrieved documents at once due to input sequence length constraints. Instead, g is applied to smaller subsets with |D ′ | = m ≪ n , where m is typically 20, and the final ranking is approximated by aggregating local reranking results across multiple batches. The ranked result is then evaluated using metrics such as NDCG@10, which only considers the topk documents in the ranking, so k is often set to a small number like 10. In RAG systems, k can be determined by the number of documents intended for inclusion in the generated context.

3.2 TrueSkill-based relevance modeling and ranking uncertainty quantification

At the core of our method is a probabilistic model of document relevance based on TrueSkill [15], a principled Bayesian ranking system originally developed for multiplayer games. It is known to produce well-calibrated estimates of both scores and uncertainty, and has been widely adopted in various applications where reliable uncertainty quantification is crucial [37-39]. Similar to prior works, we recast documents as players, and when one document is ranked above another, it is treated as having won that match. Specifically, we model the latent relevance of each document D i as a Gaussian variable x i ∼ N ( µ i , σ 2 i + β 2 ) , where µ i represents the estimated relevance, σ i captures epistemic uncertainty, and β is a fixed global parameter representing observation noise. Here, epistemic uncertainty reflects the model's lack of confidence in its current estimate and is expected to decrease as more reranking evidence accumulates through iterative updates. This probabilistic view aligns with the intuition of Cohen et al. [30], who emphasize the importance of modeling uncertainty in relevance estimation for more effective ranking decisions.

Each document's relevance is thus represented not as a point estimate but as a belief distribution, capturing both its expected value and uncertainty. Modeling document relevance as a distribution enables principled reasoning under uncertainty and supports more informed reranking decisions. Unlike traditional methods that rely solely on deterministic rankings, our approach maintains document-level belief distributions and quantifies their relative strength. This formulation allows the computation of informative quantities such as the probability that a document appears in the topk and the entropy of its rank distribution, which serve as signals for uncertainty-aware computation. The model can also incorporate external signals, such as retrieval confidence or prior knowledge of relevancy signals, when available.

̸

To estimate rank probabilities, we express the rank of D i as r i = 1 + ∑ j = i 1 x j >x i , i.e., the number of documents more relevant than D i under their latent scores. Since each x i is Gaussian, the exact probability that D i has rank r can be computed via:

where p i is the probability density function of x i and F j is the the cumulative distribution function of x j . The summation inside the integral can be evaluated using dynamic programming with O ( n 2 ) complexity by tracking the distribution over how many documents are ranked above D i . However, it

becomes costly and numerically unstable as n grows, due to the need to compute dense integrals and multiply small probabilities.

To improve scalability, we adopt a more efficient approximation. We define a threshold t ( r ) such that the expected number of documents whose relevance exceeds t ( r ) equals the target rank r , i.e., ∑ i P ( x i > t ( r )) = r . The value of t ( r ) depends on each document's relevance distribution ( µ i , σ i ) , and since t ( r ) is monotonic to r , we efficiently compute t ( r ) via binary search. We then approximate the cumulative rank probability as P ( r i ≤ r ) ≈ P ( x i > t ( r )) , which reflects the chance that x i exceeds the estimated topr threshold.

3.3 AcuRank framework

Algorithm 1 outlines AcuRank, a listwise reranking method that performs adaptive computation guided by probabilistic relevance modeling and uncertainty estimation. At each iteration, AcuRank identifies documents with uncertain rankings and focuses reranking efforts on them, updating their relevance estimates based on listwise reranker outputs.

Algorithm 1 AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking

Input: Query q , retrieved documents D = { D 1 , . . . , D n } , listwise reranker g , target rank cutoff k Output: Ranked list [ D r 1 > · · · > D r n ] , with topk used downstream

1: Initialize TrueSkill-based relevance scores ( µ i , σ i ) for all D i ∈ D

2: repeat

3: Select candidate documents C ⊂ D with high ranking uncertainty

4: Partition C into ordered groups {B 1 , . . . , B b }

5: Apply listwise reranker g to each B j and update TrueSkill scores accordingly

6: until |C| is small, topk converges, or computational budget is exhausted

Initialization (Line 1). Each document's relevance distribution ( µ i , σ i ) is initialized using first-stage retrieval scores, with µ i set to the raw score and σ i = µ i / 3 . This provides a more informative prior than uniform defaults (Supplementary Section B.4). The observation noise parameter β remains constant throughout. In ablation experiments, we also compare with the standard TrueSkill initialization ( µ i = 25 , σ i = 25 / 3 ).

Uncertainty-based selection (Line 3). To identify uncertain documents, we compute s i = P ( x i > t ( k )) for each document, where t ( k ) is the expectation threshold ( r = k ) defined at Section 3.2 and ∑ i s i = k by construction. All documents have s i = k/n for standard TrueSkill initialization, but when using first-stage retrieval scores, s i values may deviate from uniformity based on score distribution. As reranking progresses, these values polarize toward 0 or 1. We define the uncertain candidate set as C = { D i | ϵ < s i < 1 -ϵ } using a tolerance hyperparameter ϵ , selecting documents with intermediate s i values that are neither confidently in nor out of the topk .

Partitioning (Line 4). When the number of uncertain documents exceeds the reranker's capacity m , we partition the candidate set C into equally sized groups. We first sort C in descending order of µ i , resulting in [ D π 1 , . . . , D π |C| ] , where µ π 1 ≥ · · · ≥ µ π |C| . Then, we partition C into disjoint ordered lists B j = [ D π i | i ∈ [( j -1) · m, min( j · m, |C| ))] , where j = 1 , . . . , ⌈|C| /m ⌉ . Comparing documents with similar relevance scores yields higher information gain, as their relative order is more uncertain. We report random partitioning strategies in ablation experiments.

Score refinement (Line 5). Each group B j is passed to the reranker g , and the resulting rankings update TrueSkill parameters. Each outcome is interpreted as a multiplayer game among documents in B j , where each document is assigned a rank based on its position in the reranked list. We apply the TrueSkill update using listwise ranking as input, using a uniform monotonic ranking score (e.g., rank-based weights) to drive the update. Documents ranked higher than expected receive an increased µ i , while those ranked lower experience a decreased µ i . The magnitude reflects how surprising the outcome is relative to prior beliefs. This iterative process gradually sharpens the model's belief in document relevance.

Stopping criteria (Line 6). Reranking continues until one or more of the following conditions are met: (i) the number of uncertain documents |C| falls below the threshold τ , where τ is the threshold at which the number of |C| satisfies the stopping criterion, (ii) the topk set remains unchanged for a

fixed number of iterations, or (iii) the number of loops exceed the maximum computation budget. We use condition (i) by default and (iii) for budget-aware variants, with results on (ii) presented for ablation.

Final ranking. Documents are ranked by their µ i values to produce the output. By selectively reranking uncertain candidates and refining their relevance estimates through iterative feedback, AcuRank improves accuracy while reducing unnecessary computation. This uncertainty-aware approach allocates effort where most needed, focusing on ambiguous cases and avoiding redundant updates to confidently ranked documents. As a result, AcuRank provides both robust and scalable solutions for efficient listwise reranking under limited computation budgets.

4 Experimental setup

4.1 Evaluation protocol

Datasets. We evaluate reranking performance on two widely used retrieval benchmarks. For TREC Deep Learning (TREC-DL) [16], we use six standard tracks: DL19 (43), DL20 (54), DL21 (53), DL22 (76), DL23 (82), and DL-Hard (50) [40]. For BEIR [17], following Sun et al. [11], we select eight representative datasets: TREC-COVID (50), NFCorpus (323), Signal-1M (97), News (57), Robust04 (249), Touché (49), DBPedia (400), and SciFact (300). The numbers in parentheses indicate the number of queries in each dataset. These datasets span a wide range of domains, including web search, scientific literature, news articles, and argumentative retrieval.

Evaluation metrics. We measure ranking accuracy using Normalized Discounted Cumulative Gain at rank 10 (NDCG@10) [41], which reflects the quality of top-ranked documents and is the standard evaluation metric for both TREC-DL and BEIR. To assess efficiency, we count the number of reranker calls per query, which serves as a practical proxy for end-to-end latency in real-world deployments. While our evaluation is based on reranker call counts rather than actual latency, AcuRank supports parallel execution across calls of disjoint candidate groups, enabling faster computation compared to sequential reranking steps or improved ranking accuracy without increasing latency. 4

Retrievers and rerankers. We use BM25 with top-100 candidates as the default first-stage retriever, and evaluate three additional settings: BM25 top-1000, to assess scalability with larger candidate pools, along with SPLADE++ED [42] top-100 and Contriever [43] top-100 (using facebook/contriever-msmarco with mean pooling), to evaluate performance with different firststage signals. Since neural retrievers such as SPLADE and Contriever produce scores on a different scale from BM25, we normalize the top-100 first-stage scores to have a mean of 10 and a standard deviation of 1 per query before applying AcuRank. We perform retrieval from the pre-built Pyserini index [44]. For reranking, we use RankZephyr [12] as the default model, and additionally evaluate RankGPT ( gpt-4.1-mini ) to assess generalization across model families. Both rerankers operate over candidate lists of size m = 20 , formatted as prompts containing the query and document list.

4.2 Baselines

We compare AcuRank against the following reranking baselines with fixed computation per query. All experiments were conducted on a single NVIDIA A6000 GPU (48GB VRAM).

Sliding windows (SWx ): A standard listwise reranking method that processes the candidate list in overlapping windows of size m , with each pass starting from the bottom of the list and progressively refining the ordering toward the top. Since SWx uses a fixed number of reranker calls per query regardless of ranking difficulty, we evaluate multiple pass configurations (SW-1, SW-2, SW-3) to match the computational cost of our method. We use the implementation from the RankLLM codebase, 5 using the same prompt format, hyperparameters, and input size limits.

TourRankx : A listwise reranking method that runs x independent multi-stage tournaments over retrieved candidates [14]. 6 At each stage, documents are partitioned into groups based on initial

order, then randomly shuffled. At each stage, an LLM selects the most relevant documents from each group, with survivors receiving stage-specific scores. The process repeats for multiple stages (e.g., 100 → 50 → 20 → 10 → 5 → 2 ), and summing scores across x tournaments yields the final ranking. TourRank-1 uses a single tournament, while TourRank-5 employs five tournaments for lower variance at higher computational cost. For a controlled comparison, we preserve the original algorithm but reuse the exact prompt configuration employed for sliding windows and AcuRank.

TrueSkill-Static (TSc ): A TrueSkill-based baseline that selects candidates solely based on their estimated relevance scores µ i , without modeling uncertainty. At each stage j , the top-( m · c j ) documents are selected and reranked, where c = [ c 1 , . . . , c s ] defines the reranker call allocation per stage. This baseline allows us to isolate the effect of uncertainty modeling in our full method.

4.3 AcuRank configurations

Building on the general framework described in Section 3.3, we specify the experimental settings used as our default configuration, unless otherwise noted in the ablation studies (Table 4). The hyperparameters were selected based on empirical evaluation on a subset of TREC-DL19 and DL20, then applied consistently across all other datasets to ensure a fair comparison. We also provide an extended robustness analysis on how computation scales with the candidate-set size n and on ranking stability under document addition; see Supplementary Section F.

Initialization: We initialize TrueSkill scores based on first-stage retrieval scores. Specifically, we set the mean µ i to the raw retrieval score (e.g., BM25 or SPLADE), and the standard deviation to σ i = µ i / 3 . This allows the model to prioritize high-confidence documents from the beginning. Analysis on different variance initialization and justification of the design choice are discussed in Supplementary Section B and Table 5.

Uncertainty threshold: We select documents whose rank probability s i = P ( x i > t ( k )) falls within the range ( ϵ, 1 -ϵ ) . We use ϵ = 0 . 01 and k = 10 in our default configuration, unless noted otherwise in variant-specific settings.

Partitioning strategy: When the number of uncertain documents exceeds the reranker capacity m = 20 , we divide them into equally sized groups using sequential partitioning. Otherwise, we rerank using a single batch. For ablation, we also evaluate a random grouping variant.

Stopping criterion: We terminate reranking when the number of uncertain documents falls below τ = 10 , or the reranker call budget is exhausted. Alternative stopping criteria, such as topk stability, are only used in ablation studies.

Adaptive variants: To evaluate the effectiveness of our method under different computational scenarios, we introduce variants of AcuRank that reflect distinct trade-offs between efficiency and accuracy. AcuRankx is a strict-budget variant that terminates reranking after a predefined number of reranker calls x , enabling fair comparison with baselines under matched computational budgets. AcuRank-{H, HH} are high-precision variants designed for settings where ranking quality is prioritized over efficiency. AcuRank-H adopts a tighter tolerance parameter ϵ = 0 . 0001 , and AcuRank-HH additionally applies a stricter stopping criterion ( τ = 5 ), enabling more thorough refinement of uncertain candidates. Extended analyses on different choices of uncertainty thresholds are discussed in Supplementary Section D.3 and Table 13.

5 Results and analysis

5.1 Effectiveness and efficiency vs. fixed baselines

Table 1 reports NDCG@10 scores and reranker call counts across TREC-DL and BEIR datasets. Under comparable reranker budgets, AcuRank consistently outperforms fixed-computation baselines in terms of overall ranking accuracy. For example, AcuRank-9 matches the average call count of SW-1 (8.8) but achieves slightly higher average NDCG@10, showing that uncertainty-aware allocation can yield better performance even with limited computation.

As more computation becomes available, AcuRank continues to outperform. AcuRank uses 19.7 calls on average, between the budgets of SW-2 (17.6) and SW-3 (26.4), yet exceeds both in accuracy. While the gains of Sliding Windows diminish with additional calls, AcuRank shows a notably larger

56.0

AcuRank-HH

AcuRank-H

AcuRank

55.5

TS-25

TS-35

(%)

TS-14

TourRank-5

Avg.NDCG@10 (

55.0

AcuRank-10

TourRank-2

TS-10

AcuRank-9

SW-2

SW-3

54.5

SW-1

Methodfamily

54.0

SlidingWindows (SW)

TourRank

Trueskill-Static(TS)

53.5

TourRank-1

AcuRank

10

20

30

40

50

60

#Calls

100

80

60

40

20

0.0

Spearmanp=-0.27

p-value =9.3e-10

0.2

WIG(lower=harder)

improvement from AcuRank-9 to full AcuRank despite a similar increase in cost, highlighting the advantage of adaptively allocating computation based on uncertainty.

Beyond the number of reranker calls, we further evaluate efficiency on: (1) average window size, (2) input length, (3) end-to-end latency, and (4) floating-point operations (FLOPs), as reported in Table 2. Under comparable reranker budgets, AcuRank exhibits consistently higher efficiency across these measures, offering a more faithful assessment than call counts alone. In fact, the number of reranker calls disadvantages AcuRank over baselines. Since AcuRank adaptively partitions uncertain candidates with a maximum window size of C =20 , this leads to shorter inputs with smaller context numbers in some stages. Results in Table 2 confirm that AcuRank achieves faster runtime and lower input length despite operating under a similar number of calls, with up to 20% reduction in input length and latency on DL19/20, and 15-30% fewer FLOPs on average.

Figure 2 visualizes the accuracy-efficiency trade-off across systems. AcuRank consistently lies along the Pareto frontier, achieving stronger accuracy at a given budget or using fewer calls to reach the same target. In contrast, TourRank attains relatively strong performance only at much higher cost, resulting in a less favorable trade-off.

Finally, AcuRank supports flexible deployment under varying computational budgets. Its highcompute variants, AcuRank-H and AcuRank-HH, apply stricter uncertainty thresholds and use more reranker calls, yielding incremental but consistent gains without early saturation. This behavior suggests that AcuRank can function as an anytime prediction method, where users can select a variant according to their available test-time budget and desired accuracy-efficiency trade-off.

5.2 Robustness across retrieval scenarios

Table 3 shows that AcuRank maintains strong performance across varying retrieval settings with different first-stage retrievers and reranker models. AcuRank consistently outperforms the fixedcomputation baseline SW-1 under comparable reranker call budgets. With SPLADE++ED as the first-stage retriever, AcuRank achieves higher accuracy than SW-3 while using a similar number of calls as SW-1, effectively leveraging strong initial rankings to improve performance without incurring high cost. AcuRank also generalizes well to different reranking models. When using RankGPT ( gpt-4.1-mini ) instead of RankZephyr, it still achieves higher accuracy than SW-1, suggesting that uncertainty-guided allocation remains effective regardless of the underlying reranker.

5.3 Adaptive allocation behavior

We investigate how our uncertainty-guided method adaptively allocates computation across queries and whether this behavior contributes to performance gains. To isolate the effect of adaptive allocation, we compare AcuRank to TrueSkill-Static, a static variant that uses the same probabilistic scoring but selects candidates solely based on estimated relevance µ i , without considering uncertainty. As shown in Figure 2, TrueSkill-Static outperforms both Sliding Windows and TourRank in terms of accuracy-efficiency trade-off. Nonetheless, AcuRank achieves even higher accuracy with fewer or comparable reranker calls, demonstrating that ranking uncertainty is critical in further improving performance beyond static scheduling.

To further examine how AcuRank adjusts computation at the query level, we analyze the correlation between reranker call counts and query difficulty measured by Weighted Information Gain (WIG) [45], over 612 queries on TREC and BEIR. WIG is computed from BM25 top-100 scores using a log(1+ s ) transform with window size k = 50 , where lower values indicate more ambiguous relevance distributions. Figure 3 shows a significant negative correlation between WIG and the number of reranker calls issued by AcuRank ( ρ = -0 . 27 , p < 10 -8 ), confirming that more compute is allocated to harder queries . This adaptive behavior is further detailed in Supplementary Section E, comparing with the Sliding Window variant, where we observe that AcuRank allocates more computation to difficult queries while reducing computation on easier ones. We also report a qualitative example of Bayesian update in the Supplementary Section E.3.

5.4 Impact of design choices

Table 4 presents an ablation study evaluating how individual design choices in AcuRank affect its overall effectiveness. Initializing TrueSkill scores from first-stage retrieval improves accuracy over default initialization (55.5 vs. 54.8), though at the cost of more reranker calls. Sequential partitioning outperforms random grouping in both accuracy and efficiency, confirming that comparing documents with similar relevance estimates yields more informative updates. For the stopping criterion, uncertainty-based termination achieves similar accuracy to the topk -unchanged strategy

while reducing the number of reranker calls (19.7 vs. 22.7). Taken together, these results show that each component of our method meaningfully contributes to the method's effectiveness and that the default configuration offers a strong balance between accuracy and computational efficiency.

6 Conclusion

We introduce AcuRank, a novel listwise reranking method that performs adaptive computation guided by uncertainty-aware relevance modeling. Our approach maintains probabilistic relevance estimates using TrueSkill and selectively allocates reranking effort to documents with high ranking uncertainty. By focusing computation on ambiguous candidates, AcuRank consistently outperforms fixed-computation baselines across TREC-DL and BEIR, achieving better accuracy-efficiency tradeoffs. Beyond a single algorithm, AcuRank serves as a flexible framework for uncertainty-aware adaptive computation in listwise reranking, providing a principled foundation based on probabilistic modeling. This formulation enables fine-grained control over the reranking process and opens opportunities for future research in uncertainty-guided retrieval.

References

Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval , 3(4):333-389, 2009.

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6769-6781, 2020.

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107 , 2023.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 , 2, 2023.

Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv preprint arXiv:1901.04085 , 2019.

Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large language models are effective text rankers with pairwise ranking prompting. In Findings of the Association for Computational Linguistics: NAACL 2024 , pages 1504-1518, 2024.

Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. Rankt5: Fine-tuning t5 for text ranking with ranking losses. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 2308-2313, 2023.

Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankvicuna: Zero-shot listwise document reranking with open-source large language models. arXiv preprint arXiv:2309.15088 , 2023.

Ruicheng Xian, Honglei Zhuang, Zhen Qin, Hamed Zamani, Jing Lu, Ji Ma, Kai Hui, Han Zhao, Xuanhui Wang, and Michael Bendersky. Learning list-level domain-invariant representations for ranking, 2023. URL https://arxiv.org/abs/2212.10764 .

Mofetoluwa Adeyemi, Akintunde Oladipo, Ronak Pradeep, and Jimmy Lin. Zero-shot crosslingual reranking with large language models for low-resource languages. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 650-656, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. acl-short.59. URL https://aclanthology.org/2024.acl-short.59/ .

Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 14918-14937, 2023.

Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze! arXiv preprint arXiv:2312.02724 , 2023.

Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. Zero-shot listwise document reranking with a large language model. arXiv preprint arXiv:2305.02156 , 2023.

Yiqun Chen, Qi Liu, Yi Zhang, Weiwei Sun, Xinyu Ma, Wei Yang, Daiting Shi, Jiaxin Mao, and Dawei Yin. Tourrank: Utilizing large language models for documents ranking with a tournament-inspired strategy. arXiv preprint arXiv:2406.11678 , 2024.

Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill™: A bayesian skill rating system. Advances in Neural Information Processing Systems , 19, 2006.

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820 , 2020.

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021.

Andrew Drozdov, Honglei Zhuang, Zhuyun Dai, Zhen Qin, Razieh Rahimi, Xuanhui Wang, Dana Alon, Mohit Iyyer, Andrew McCallum, Donald Metzler, et al. Parade: Passage ranking using demonstrations with llms. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 14242-14252, 2023.

Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 2421-2425, 2024.

Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 3781-3797, 2022.

Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. Beyond yes and no: Improving zero-shot llm rankers via scoring fine-grained relevance labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) , pages 358-370, 2024.

Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models. arXiv preprint arXiv:2101.05667 , 2021.

Jian Luo, Xuanang Chen, Ben He, and Le Sun. Prp-graph: Pairwise ranking prompting to llms with graph aggregation for effective text re-ranking. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 5766-5776, 2024.

Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. A setwise approach for effective and highly efficient zero-shot ranking with large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 38-47, 2024.

Anonymous. Budget-constrained document re-ranking with bayesian LM-based pairwise comparisons. OpenReview preprint, ACL ARR 2025 February submission, 2025. URL https://openreview.net/forum?id=3m34mqglmh . Accessed: 2025-10-21.

Andrew Parry, Sean MacAvaney, and Debasis Ganguly. Top-down partitioning for efficient list-wise ranking. arXiv preprint arXiv:2405.14589 , 2024.

Soyoung Yoon, Eunbi Choi, Jiyeon Kim, Hyeongu Yun, Yireun Kim, and Seung-won Hwang. Listt5: Listwise reranking with fusion-in-decoder improves zero-shot retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2287-2308, 2024.

Wenhan Liu, Xinyu Ma, Yutao Zhu, Lixin Su, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. Coranking: Collaborative ranking with small and large ranking agents, 2025. URL https://arxiv.org/abs/2503.23427 .

Wenhan Liu, Yutao Zhu, and Zhicheng Dou. Demorank: Selecting effective demonstrations for large language models in ranking task, 2024. URL https://arxiv.org/abs/2406.16332 .

Daniel Cohen, Bhaskar Mitra, Oleg Lesota, Navid Rekabsaz, and Carsten Eickhoff. Not all relevance scores are equal: Efficient uncertainty and calibration modeling for deep retrieval models. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 654-664, 2021.

Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983 , 2016.

Andrei Z Broder, David Carmel, Michael Herscovici, Aya Soffer, and Jason Zien. Efficient query evaluation using a two-level retrieval process. In Proceedings of the twelfth international conference on Information and knowledge management , pages 426-434, 2003.

Debasis Ganguly and Emine Yilmaz. Query-specific variable depth pooling via query performance prediction. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 2303-2307, 2023.

Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, and Maarten de Rijke. Query performance prediction using relevance judgments generated by large language models. arXiv preprint arXiv:2404.01012 , 2024.

Negar Arabzadeh, Chuan Meng, Mohammad Aliannejadi, and Ebrahim Bagheri. Query performance prediction: From fundamentals to advanced techniques. In European Conference on Information Retrieval , pages 381-388. Springer, 2024.

Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, and Maarten de Rijke. Ranked list truncation for large language model-based re-ranking. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 141-151, 2024.

John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. Talebrush: Sketching stories with generative pretrained language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , CHI '22, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450391573. doi: 10.1145/3491102.3501819. URL https://doi.org/10.1145/3491102.3501819 .

Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023. URL https://arxiv.org/abs/2304.03442 .

Omar Shaikh, Valentino Chai, Michele J. Gelfand, Diyi Yang, and Michael S. Bernstein. Rehearsal: Simulating conflict to teach conflict resolution, 2024. URL https://arxiv.org/ abs/2309.12309 .

Iain Mackie, Jeffrey Dalton, and Andrew Yates. How deep is your learning: the dl-hard annotated deep learning dataset. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 2335-2341, 2021.

Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst. , 20(4):422-446, oct 2002. ISSN 1046-8188. doi: 10.1145/582415.582418. URL https://doi.org/10.1145/582415.582418 .

Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. From distillation to hard negative sampling: Making sparse neural ir models more effective, 2022. URL https://arxiv.org/abs/2205.04733 .

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2022. URL https://arxiv.org/abs/2112.09118 .

Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021) , pages 2356-2362, 2021.

Yun Zhou and W Bruce Croft. Query performance prediction in web search environments. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval , pages 543-550, 2007.

Meta AI. Llama 3.3 70B Instruct model card. https://huggingface.co/meta-llama/ Llama-3.3-70B-Instruct , 2024. Model release date: 6 Dec 2024. Accessed 23 May 2025.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 10014-10037, 2023.

SU Hongjin, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Liu Haisu, Quan Shi, Zachary S Siegel, Michael Tang, et al. Bright: A realistic and challenging benchmark for reasoning-intensive retrieval. In The Thirteenth International Conference on Learning Representations .

Chenghao Xiao, G Thomas Hudson, and Noura Al Moubayed. Rar-b: Reasoning as retrieval benchmark. arXiv preprint arXiv:2404.06347 , 2024.

Yuelyu Ji, Zhuochun Li, Rui Meng, and Daqing He. Reasoningrank: Teaching student models to rank through reasoning-based knowledge distillation. arXiv preprint arXiv:2410.05168 , 2024.

Tong Niu, Shafiq Joty, Ye Liu, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Judgerank: Leveraging large language models for reasoning-intensive reranking. arXiv preprint arXiv:2411.00142 , 2024.

Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, and Benjamin Van Durme. Rank1: Test-time compute for reranking in information retrieval. arXiv preprint arXiv:2502.18418 , 2025.

Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. Rankr1: Enhancing reasoning in llm-based document rerankers via reinforcement learning. arXiv preprint arXiv:2503.06034 , 2025.

Wenhan Liu, Xinyu Ma, Weiwei Sun, Yutao Zhu, Yuchen Li, Dawei Yin, and Zhicheng Dou. Reasonrank: Empowering passage ranking with strong reasoning ability, 2025. URL https://arxiv.org/abs/2508.07050 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract (lines 1-14) and Introduction explicitly state the two core contributions, which is (i) probabilistic relevance modeling with TrueSkill and (ii) uncertainty-aware adaptive computation, and the empirical scope matches these claims.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We added the the limitations section at the Appendix, at Section G.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We clearly state our problem formulation in Section 3.1 and the underlying theorem in Section 3.2, with algorithms at Algorithm 1.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 4 detail datasets, retrieval/reranking models, hyper-parameters, stopping criteria and ablation settings, providing sufficient information for reproduction.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Public datasets (TREC-DL, BEIR) are used, and a public code repository is also linked, with the github url of https://github.com/soyoung97/AcuRank .

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 4.3 enumerates retriever/reranker configurations, window size m , uncertainty threshold ϵ , stopping criterion τ , and compute-budget variants.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report statistical significance (spearman

ρ

with

p

-

value

for Fig. 3.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments were conducted on a single NVIDIA A6000 GPU (48GB VRAM) (Section 4.2. We also focus on reporting the number of reranker calls to show improved efficiency on our method.

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work builds on publicly available text corpora and complies with data-use licences; no personal or sensitive data are processed.

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work focuses on efficient and effective listwise reranking.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No new high-risk model or dataset is released.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite datasets and baseline code, and the license terms are explained in Supplementary Section C.2.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new dataset, or model checkpoint is introduced in the submission. Our method can be applied upon different listwise reranking models.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The study involves no human-subject or crowdsourcing component.

15. Institutional review board (IRB) approvals or equivalent

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human-subject research was conducted, so IRB approval is not applicable.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: Section 4 explain how large language models (RankZephyr, RankGPT) are used as listwise rerankers, including prompt size and grouping strategy.

Supplementary Materials for AcuRank: Uncertainty-Aware Adaptive Computation for Listwise reranking

A Overview

This supplementary material provides additional details and extended results for our AcuRank framework. Specifically:

Section B describes the mathematical foundations of our Bayesian modeling approach, including the TrueSkill update equations and our efficient approximation of cumulative rank distributions. We also report the computational overhead of TrueSkill and additional experimental justification of the variance initialization.

Section C specifies our experimental setup, including hardware specifications, software configurations, datasets, licenses, and modifications to baseline implementations.

Section D presents comprehensive experimental results, including dataset-wise performance metrics omitted from the main paper due to space constraints. This includes extended results for all baselines (Tables 1-3), full numbers for the TrueSkill-Static variants shown in Figure 2, additional experiments with RankVicuna-7B and Llama-3.3-70B-Instruct demonstrating the generalizability of our approach, and extended analysis on AcuRank's sensitivity to uncertainty thresholds.

Section E provides a deeper analysis of AcuRank's adaptive behavior, including query-level compute allocation patterns and a breakdown of performance on easy versus hard queries. Also, we provide a qualitative example of the Bayesian update of AcuRank.

Section F discusses the AcuRank's robustness among scalability and ranking stability over varying n and document addition.

Section G discusses current limitations and potential future directions.

Section H lists the exact prompt templates used with RankZephyr, RankVicuna, and Llama-3.370B-Instruct for listwise reranking.

B Supplement on Bayesian modeling and score updates in TrueSkill

B.1 TrueSkill as a Bayesian model

TrueSkill [15] models the latent relevance (originally, 'skill') of each document x i as a Gaussian random variable, x i ∼ N ( µ i , σ 2 i ) , starting with a prior and updating it iteratively via Bayesian inference. Each listwise reranker output, denoted g ( D ′ ; M ) , is treated as noisy evidence about pairwise preferences among a subset D ′ of documents, where M is the reranking model. Using belief propagation on a factor graph, TrueSkill analytically updates the posterior distribution to

We denote the current update round by t . At t = 0 , ( µ (0) i , σ (0) i ) represents the prior initialized from first-stage retrieval scores. After observing the reranking output, the posterior is updated to ( µ ( t +1) i , σ ( t +1) i ) .

B.2 Approximating the cumulative rank distribution

We approximate the probability that document D i ranks within the top r positions as

The left-hand side is the probability that at most r -1 other documents outrank D i . The right-hand side approximates this count by evaluating whether x i exceeds a threshold t ( r ) calibrated such that the expected number of documents exceeding this threshold equals r . This approximation becomes accurate under two assumptions: (1) the latent scores x i follow independent Gaussian distributions with similar variances, and (2) the number of candidates n is sufficiently large, so that

the expected number of exceedances concentrates near its mean by the law of large numbers. This provides an efficient closed-form alternative to the expensive O ( n 2 ) dynamic programming approach for computing the full ranking distribution over permutations. Instead of enumerating all possible rankings, we approximate P ( r i ≤ r ) using a single Gaussian tail probability, 1 -Φ( · ) .

B.3 TrueSkill score update equations

The reranking over each batch B is treated as a multi-player game . The observed ranking outcome is given by g ( B ; M ) . For every adjacent document pair in this order, a win-loss factor is defined to model their pairwise preference. Applying message passing on the factor graph yields the following canonical update equations for each document D i :

where c = √∑ D j ∈B ( σ 2 j + β 2 ) . The values λ and υ are closed-form functions derived from the win probability between adjacent items (see Herbrich et al. [15] for a full derivation). Intuitively, when a document ranks higher than expected, its mean µ i increases. Unexpected outcomes also reduce σ i , reflecting greater confidence in the document's estimated relevance.

Side note on implementation. We employ the Python trueskill library. 7 The relevance of each candidate passage is represented as a trueskill.Rating class. After the listwise reranker outputs an ordering, we update all ratings by calling trueskill.rate function with the argument rating_groups= list of ratings , and ranks=[0,1,...,n] , where n is the number of passages. The function returns a list of posterior ratings containing ( µ, σ ) , which serve as the priors for the next iteration. 8

B.4 Proportional uncertainty initialization based on retrieval scores

We initialize each document's relevance score as a Gaussian distribution parameterized by its firststage retrieval score. Specifically, for each document D i with retrieval score s i , we set the mean and standard deviation as µ i = s i and σ i = µ i / 3 . This initialization preserves the relative scale of the retrieval scores while explicitly modeling their uncertainty. A higher retrieval score indicates greater expected relevance ( µ i ↑ ), but it may also be more prone to overestimation. To reflect this, we assign a proportionally larger prior uncertainty ( σ i ↑ ) to such documents. The 1 / 3 ratio follows the standard µ : σ =3:1 default used in TrueSkill, and yields a conservative displayed rating R i = µ i -3 σ i = 0 . This provides a non-negative lower bound (semantically natural for relevance), while leaving substantial headroom for upward revision as reranking evidence arrives. Statistically, [ µ i -µ i , µ i + µ i ] = [0 , 2 µ i ] corresponds to [ µ i ± 3 σ i ] since 3 σ i = µ i , enclosing ≈ 99 . 7% of Gaussian mass; thus the prior is cautious yet flexible, centered on the retrieval signal but amenable to change. Using a larger multiple (e.g., σ i = µ i / 2 ) makes the prior overly diffuse and weakens the retrieval signal, whereas a smaller multiple would risk overconfidence and slower correction; the three-sigma choice strikes a practical middle ground that is widely adopted in TrueSkill deployments.

Experimental justification. To further support this design, we conduct an ablation study comparing three initialization variants: (1) a fixed uncertainty ( σ =8 ), (2) a more conservative variant ( σ i = µ i / 2 ), and (3) our default setting ( σ i = µ i / 3 ). As shown in Table 5, the default configuration performs best overall, while all variants preserve the same qualitative trends. This indicates that AcuRank is robust to the specific choice of initialization and supports the effectiveness of the three-sigma prior.

B.5 Score refinement (Algorithm 1, Line 5)

After each reranker call, we perform the following update procedure:

Convert the returned ranking over the current batch B into a set of pairwise win-loss observations.

Run one round of TrueSkill message passing to update each document's parameters ( µ i , σ i ) in B based on these observations.

Recompute the ranking-uncertainty scores s i = P ( x i > t ( k )) to guide selection in the next iteration.

This iterative posterior refinement progressively sharpens the relevance estimates. As a result, the uncertainty values σ i naturally decrease over time. Documents whose ranks stabilize early tend to exit the uncertain set, allowing us to avoid unnecessary reranker calls and reduce overall computation.

B.6 Computational overhead of TrueSkill

We additionally measured and reported the computational overhead of TrueSkill at Table 6, which was negligible compared to the cost of LLM reranker calls. On the DL19 benchmark, the total runtime of AcuRank was approximately 58.28 seconds, among which the cumulative time spent on TrueSkill updates was only 0.06 seconds (0.1%). The portion of running Trueskill versus the total process was consistent (0.1%) across subsets of data (DL19, DL20, TREC-COVID, News) and for AcuRank-9 and AcuRank. This was measured using time.time() around the call to trueskill.rate() , which can be further observed at the official codebase. The trueskill.rate() function performs lightweight updates to the mean and variance of document scores based on closed-form Gaussian updates, which are efficiently computed on CPU. This confirms that nearly all computation time in AcuRank is spent on LLM calls, with minimal overhead from the uncertainty modeling.

C Implementation Details

C.1 Hardware specifications and software configurations

Compute nodes: Experiments were conducted on either (i) an ASUS ESC8000-E11 server equipped with dual 4 th -Gen Intel Xeon processors, 64 CPU threads, 1.1 TB RAM, and eight NVIDIA A6000 GPUs (48 GB each), or (ii) a workstation with eight RTX 3090 GPUs (24 GB each).

Inference stack: All LLM inference was performed using the transformers library, without acceleration backends such as vLLM or FastChat. Greedy decoding was used throughout, with a fixed random seed to ensure reproducibility.

Reproducibility: We release a complete end-to-end reproduction script, packaged as a .zip archive, which replicates reported experiments.

Token limits: Following the RankLLM codebase 9 , all inputs were truncated to a maximum of 4,096 tokens.

Default hyperparameters: Unless otherwise specified, we adopt the default hyperparameter settings provided by the RankLLM implementation.

C.2 Datasets and licenses

Abbreviations. For brevity, we use the following shorthands throughout our tables: DL19-DL23 (TREC-DL 2019-2023), DL-H (DL-Hard), COVID (TREC-COVID), NFC (NFCorpus), Signal (Signal-1M), News (TREC-News), R04 (Robust04), Touche (Webis-Touche2020), DBP (DBPedia), and Sci (SciFact).

TREC-DL (2019-2023 tracks). The Topics and Qrels are in the public domain as U.S. Government work. 10 The underlying corpus (MS MARCO + ORCAS) is released by Microsoft for non-commercial research purposes only , with no IP rights to the documents and 'as-is' usage at the user's own risk. 11

DL-HARD [40]. We use the annotation files and BM25 retrieval result files hosted on GitHub ( https://github.com/grill-lab/DL-Hard ). The repository does not specify a formal license. Following author guidance and consistent with MS MARCO terms (see footnote 11), we use the data strictly for non-commercial academic research.

BEIR [17]. The framework code for BEIR is under the Apache 2.0 license. 12 Each constituent dataset retains the license of its original source (e.g., public-domain for TREC-COVID). For our experiments, we use pre-computed BM25 and SPLADE top-1000 retrieval runs (included in our anonymized supplementary code), which are released under the BigScience OpenRAIL-M license.

TrueSkill rating algorithm. We rely on the open-source Python package trueskill (v0.4.5) 13 , distributed under the permissive BSD 3-Clause License . The TrueSkill TM trademark and the original Bayesian rating system remain the property of Microsoft, who permits the name and algorithm to be used only in Xbox Live titles or non-commercial projects. 14 Our usage complies with these terms, as it is strictly limited to non-profit academic research.

C.3 Evaluation protocol

All reported metric values are rounded to one decimal place for consistency. For each method, we first compute the mean number of reranker calls within each dataset . Specifically, we calculate the average across all queries belonging to a given dataset, resulting in one mean value per dataset. We then compute the overall value reported in our tables by taking a simple unweighted average of these 14 per-dataset means. This ensures that each dataset contributes equally, regardless of the number of queries it contains. Note that this is a dataset-level macro average , not a micro average computed over all individual queries pooled together.

C.4 Sliding windows baseline

The sliding windows baselines rerank the top-100 candidates using fixed-size windows of size 20 and a stride of 10. This configuration results in nine overlapping windows per query. When the list length is not divisible by 20, we form a final, smaller window containing the remaining documents. This ensures full coverage of all retrieved candidates.

C.5 TourRank baseline

We adapted the public TourRank implementation 15 to handle candidate sets with fewer than 100 passages. The original code assumes |D| = 100 and executes five tournament stages. However, for some queries in certain datasets (e.g., NFCorpus), the query text is extremely short, so BM25 retrieves fewer than 100 candidate documents ( |D| < 100 ). To handle such cases, we added the simple stage-skip rule below and a one-line boundary check to the original grouping function, in order to avoid redundant inference and out-of-range errors. Our modifications are twofold:

Stage skip rule. We skip Stages 1 through 4 when |D| is less than or equal to 50, 20, 10, and 5, respectively. When |D| ≤ 2 , we return the BM25 order directly.

Boundary check (grouping). We retain the original index modulo grouping logic from the public implementation and insert a simple boundary check ( if idx < len(D) ) to prevent IndexError .

D Extended experimental results

D.1 Average number of reranker calls per dataset

We report the number of reranker calls per dataset with NDCG@10 scores, which correspond to the Tables and Figures reported in the main paper. Table 7 reports the NDCG@10 scores for each dataset,

along with the average number of reranker calls. These results complement Table 1 in the main paper by providing dataset-level breakdowns. Additionally, Table 9 reports the complete per-dataset results for TrueSkill-Static with different configurations shown in Figure 2. Table 8 summarizes dataset-wise results for the SPLADE++ED first-stage retrieval results and results with RankGPT ( gpt-4.1-mini ) as a reranker, including the average number of passes, corresponding to Table 3 in the main paper. Finally, Table 10 reports the per-dataset average number of reranker calls and the resulting NDCG@10 for each design-choice variant, corresponding with Table 4 in our main paper.

Akey observation from these extended results is that, unlike static approaches such as sliding windows or TourRank, our method exhibits varying numbers of calls across different datasets. This adaptive behavior reflects our uncertainty-aware framework's ability to allocate computational resources based on the inherent difficulty and characteristics of each dataset.

D.2 Generalization across rerankers

To evaluate the generalizability of AcuRank across different reranking models and scales, we conduct comprehensive experiments using both RankVicuna-7B [8] and Llama-3.3-70B-Instruct [46] models.

Evaluation with RankVicuna-7B. We first test AcuRank using RankVicuna-7B (huggingface identifier of castorini/rank_vicuna_7b_v1 ) on BM25 top-100 retrieval results. As shown in Table 11, while RankVicuna achieves lower absolute NDCG@10 scores than RankZephyr, AcuRank maintains its effectiveness pattern. Specifically, AcuRank-9 achieves +0.7 improvement over SW-1 that uses same number of calls (8.8). Meanwhile, AcuRank achieves +1.8 NDCG@10 improvement over SW-1 , and +1.2 improvement over SW-3 despite using fewer calls (19.5 vs 26.4), and +1.5 improvement over TourRank-2 using similar number of calls (25.5 vs 26.4). In summary, AcuRank consistently outperforms both sliding windows and TourRank baselines while requiring fewer reranker calls.

Evaluation with Llama-3.3 70B as a zero-shot reranker. To test whether AcuRank continues to yield benefits for larger, purely zero-shot rerankers, we replaced our 7B parameter RankZephyr

model with Llama-3.3 70B Instruct [46] 16 and reran the pipeline. Resource constraints limited us to three representative benchmarks: DL23, TREC-COVID, and TREC-NEWS. Results are reported in Table 12. Even for this stronger backbone AcuRank extracts an additional +1.6 average NDCG@10 over the single-pass baseline, while keeping the number of calls bounded. This confirms that our framework continues to provide a favorable accuracy-efficiency trade-off as model capacity scales.

D.3 Extended analysis on sensitivity to uncertainty thresholds

For AcuRank, the uncertainty thresholds ϵ and σ are adjustable parameters that control the tradeoff between efficiency and accuracy. A relaxed setting (i.e., smaller ϵ or σ ) generally improves accuracy at the cost of increased computation, while stricter values reduce LLM usage with some performance degradation. This flexibility allows users to adapt the method to their specific resource and performance requirements. Our configurations were chosen to reflect different trade-off points depending on user preference: AcuRank-H and AcuRank-HH favor accuracy, AcuRank-9 prioritizes efficiency, and the standard AcuRank strikes a practical balance. We report additional comprehensive ablation results at Table 13 to illustrate how performance (in terms of NDCG@10) and computation vary with these thresholds. The results show that the trade-off curve is smooth and monotonic, suggesting that AcuRank is robust across a wide range of hyperparameter choices.

E Extended analysis on adaptive computation allocation

Following Section 5.3 of the main paper ('Adaptive allocation behavior') and Figure 3, we provide a more fine-grained query-level analysis.

E.1 Correlations with WIG

This is an extended analysis of Figure 3 in the main paper. On designing the experiment, we pool all datasets containing fewer than 100 queries to obtain reliable correlations. This includes every benchmark except DBPedia-Entity, SciFact, NFCorpus, and Robust04. The resulting subset contains 612 queries drawn from 14 datasets. For each query, we compute the Weighted Information Gain (WIG) [45] over the BM25 top-100 retrieval scores. We then measure two Spearman correlations: (i) between WIG and the number of calls made by AcuRank to satisfy its stopping criterion, and (ii) between WIG and the final NDCG achieved by AcuRank. As shown in Table 14, both correlations are negative.

Calls versus WIG. Lower WIG, which reflects more ambiguous top-100 lists, leads to more reranker calls. This confirms that AcuRank allocates additional computation when the candidate set is uncertain.

Calls versus NDCG. Queries that result in lower final NDCG tend to receive more reranker calls. This indicates that AcuRank naturally focuses effort on harder queries.

E.2 Comparison with fixed sliding windows

We further compare AcuRank (with an average of 19.7 calls across 18 datasets) with a Sliding Windows baseline using a similar compute budget (SW-2, 18 calls). For three challenging datasets (Touche, TREC-COVID, and DL-Hard), we split the queries into Easy (NDCG ≥ µ ) and Hard (NDCG < µ ), where µ is the dataset-level mean NDCG produced by SW-2. Table 15 shows a consistent pattern:

Avg. # Reranker Calls: Hard > Easy. AcuRank spends more calls on the Hard bucket (e.g. +6.2 on TREC-COVID) and sometimes even saves calls on the Easy bucket (e.g. DL-HARD).

NDCG Gains: Hard > Easy. The additional computation translates into larger NDCG improvements on Hard queries (e.g. +7.0 on TOUCHE, +1.7 on TREC-COVID), while improvements on Easy queries are smaller than those for the Hard bucket queries.

These observations corroborate the claim that AcuRank adaptively directs compute to the most uncertain and therefore most difficult queries, yielding a superior accuracy and efficiency trade-off compared with fixed-schedule baselines.

E.3 Qualitative Example of the Bayesian Update process of AcuRank

Setup. Table 16 shows the trace for the first query of TREC-DL19 as AcuRank iterates five times over 50 candidate passages ( n =50 ). We highlight two contrasting passages to make the dynamics clear.

Over-estimated early leader (pid 6337909). On the first iteration, this non-relevant passage holds a slightly higher mean score than the gold passage ( µ = 13 . 77 > 13 . 44 ). After one unfavourable comparison, its mean collapses, confidence tightens, and because subsequent comparisons are unnecessary, the passage is frozen at rank 33.

Initially uncertain passage that surges (pid 4647186, gold). Starting lower than 6337909, this passage wins multiple pairwise contests. Each victory pushes its mean up and its σ down until it confidently enters the top-10 (rank 9).

The key takeaway is that uncertain winners can surge, overestimated items are pruned, and comparisons halt once confidence is high, saving compute.

This exemplifies how AcuRank's Bayesian evidence accumulation boosts performance while saving LLM calls by halting once posterior uncertainty is low.

F Extended robustness: scalability and ranking stability

This section extends our analysis of AcuRank's robustness along two axes that are important in practice. First, we study how the amount of computation evolves as the size of the candidate set n changes. Second, we assess the stability of rankings when the candidate pool is enlarged by adding new documents. Unless specified otherwise, we use the same default setup as in the main text (BM25 first-stage, RankZephyr listwise reranker, group size m =20 , k =10 , uncertainty tolerance ϵ =0 . 01 ,

stopping threshold τ =10 ), on the selected subset (DL19, DL20, TREC-COVID, TREC-News) of BEIR.

F.1 Scalability with candidate-set size

We vary n ∈ { 50 , 100 , 200 , 1000 } and report the average number of reranker calls until convergence and the average posterior mean µ computed over the final top-100 (top-50 when n =50 ). Sliding Windows with a single pass (SW-1) is included for reference. Table 17 (left) shows that the number of AcuRank calls grows sublinearly in n . For example, increasing n from 100 to 1000 (10 × ) raises the average calls from 18.7 to 68.6 ( ≈ 3.7 × ), while SW-1 scales almost linearly (9 → 99, ≈ 11 × ). Table 17 on the right shows a mild increase in the average µ among the top set as n grows, which is expected when a larger pool provides stronger candidates.

Overall, AcuRank focuses computation on the subset of uncertain candidates near the topk boundary and avoids redundant updates to confident items, which explains the sublinear growth in calls with larger n and indicates better scalability as n increases.

F.2 Ranking stability under document addition

We examine whether the relative order over an original set D = ( D 1 , D 2 , ... D n ) is preserved after reranking an extended set D + ⊇D . Concretely, we take BM25 top-50 as D and top-100 as D + for each query, and independently run AcuRank and SW-1 on both sets. For a subset S⊆D , we compute the ratio of preserved pairwise relations between the two rankings. We report two choices of S : all items in D , and only the gold-relevant items in D . Results in Table 18 show that AcuRank preserves relative orderings more robustly over the full D , and is comparable or better on the gold-only subset across three of four datasets, reflecting the algorithm's focus on accurately resolving the most relevant candidates.

In summary, compared with the sliding window baseline, AcuRank scales favorably with larger candidate pools and maintains stable ordering , particularly among truly relevant items, when additional documents are introduced.

G Limitations and Future Work

Latency optimization. Although AcuRank reduces the total number of reranker calls, further latency improvements may be achieved through parallelization. Unlike sequential sliding windows, our use of disjoint candidate groups enables batched inference across groups within the same iteration.

In addition, cross-query computation sharing could significantly reduce latency when processing multiple queries concurrently.

Hyperparameter adaptation. Our fixed global hyperparameters ( µ , ϵ , and stopping criteria) perform well overall, but may be suboptimal for certain domains. Query-aware or domain-specific tuning could improve effectiveness. Automatic adaptation methods represent a promising direction for future work.

Potential method enhancements. While AcuRank demonstrates strong performance, several promising directions remain for future research. Our current approach uses thresholded cumulative rank probabilities to guide reranking decisions. However, the full rank probability distribution offers a richer source of uncertainty information. For example, rank entropy , defined as H ( r i ) = -∑ n r =1 P ( r i = r ) log P ( r i = r ) , quantifies the dispersion in a document's predicted rank and could support more expressive uncertainty-based filtering strategies. 17 In addition, our current grouping strategy for uncertain documents is based on simple heuristics. Clustering-based or similarity-aware grouping methods may improve reranking efficiency by better capturing interdocument structure. Finally, modern LLMs may expose token-level or generation-level confidence scores. These signals could be integrated to refine document-level uncertainty estimates or modulate the strength of score updates during reranking.

Reasoning-aware retrieval. Apromising future direction is to extend the proposed uncertainty-aware adaptive reranking to reasoning-aware retrieval [47-49], where the objective is not only to identify topically relevant documents but also to select those that meaningfully contribute to multi-step reasoning or evidence aggregation. In such scenarios, reranking plays a critical role [50-54], as the quality of reasoning often depends on the interaction and organization of supporting information rather than on individual relevance scores. Modeling uncertainty over a document's contribution to reasoning could enable adaptive allocation of computation to the most influential candidates, potentially improving both reasoning accuracy and overall retrieval efficiency across diverse tasks.

H Example input/output prompt format for listwise reranking

We provide concrete prompt examples for RankZephyr [12], RankVicuna [8], and zero-shot Llama3.3-70B-Instruct reranking models [46].

H.1 RankZephyr [12] prompt example

H.2 RankVicuna [8] prompt example



 
 You are RankLLM, an intelligent assistant that can rank passages based on their relevancy to the query. 
 I will provide you with 20 passages, each indicated by a numerical identifier []. Rank the passages based on their relevance to the search query: what does prenatal care include. [1] Pregnancy and prenatal care go hand in hand. During the first trimester, prenatal care includes blood tests, a physical exam, conversations about lifestyle and more. Prenatal care is an important part of a healthy pregnancy. Whether you choose a family physician, obstetrician, midwife or group prenatal care, here's what to expect during the first few prenatal appointments. [2] Pregnancy and prenatal care go hand in hand. During the first trimester, prenatal care includes blood tests, a physical exam, conversations about lifestyle and more. Prenatal care is an important part of a healthy pregnancy. Whether you choose a family physician, obstetrician, midwife or group prenatal ... [3] Prenatal appointments typically ... [4] Routine prenatal screening ... [5] Standard tests include ... ... [19] What is Prenatal Care: Prenatal care is the health care that both the woman and the baby receive before giving birth. This is more than just a few doctor's visits and an ultrasound or two. [20] Medicaid coverage for prenatal care provides another opportunity to illustrate the four pathways' differences and how each state determines what services to include. (1) Right away, we know that the emergency pathway will not pay for prenatal care because preventive services are non-urgent by definition. Search Query: what does prenatal care include. Rank the 20 passages above based on their relevance to the search query. All the passages should be included and listed using identifiers, in descending order of relevance. The output format should be [] > [], e.g., [2] > [1]. Only respond with the ranking results, do not say any word or explain. 
 --------------------Example Output: [1] > [2] > [3] > [5] > [6] > [19] > [8] > [15] > [16] > [14] > [17] > [4] > [12] > [13]

> [9] > [10] > [18] > [7] > [11] > [20]

H.3 Llama-3.3-70B-Instruct [46] (zero-shot) prompt example

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe existing methods for listwise reranking using large language models are computationally expensive. While there have been various attempts to address these issues, they still suffer from problems of rigidity and lack of adaptability. To overcome these challenges, this paper proposes AcuRank, an uncertainty-aware reranking method, which utilizes the TrueSkill Bayesian rating system for reranking.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper is well written and easy to follow.\n- Using uncertainty as for the reranking metric is an interesting attempt.\n- Experimental results prove the efficacy of the proposed method.\n\nWeaknesses\n\n- It is not trivial to understand the motivation for setting the initialization variance to $\\sigma^2/3$. It would be helpful to have either an ablation experiment or more motivation to clarify this.\n- It would be great to have a qualitative example showing how the predictive mean and ranking of the retrieved documents change during the Bayesian update. For instance, documents that initially had high scores but were uncertain might have their scores decrease and uncertainty reduce, leading to a lower rank after the update, or vice versa.\n- It would be helpful for readers if there exist the ablation experiments on the uncertainty threshold.\n- It would be helpful to analyze how the scale of the updated mean and the number of updates required for the final update change as the number of retrieved documents, $n$, varies. This would provide a clearer understanding of how the method operates as $n$ changes.\n- Let’s consider a document set ${\\mathcal{D}_1, \\ldots, \\mathcal{D}_n}$. After reranking this set, we obtain the result ${\\hat{\\mathcal{D}}_1, \\ldots, \\hat{\\mathcal{D}}_n}$. Now, suppose a new set of documents ${\\bar{\\mathcal{D}}_1, \\ldots, \\bar{\\mathcal{D}}_m}$ is added to the collection. The task is to examine whether the reranked result of the combined set ${\\mathcal{D}_1, \\ldots, \\mathcal{D}_n, \\bar{\\mathcal{D}}_1, \\ldots, \\bar{\\mathcal{D}}_m^*}$ preserves the relative order of the original documents \\${\\mathcal{D}\\_1, \\ldots, \\mathcal{D}\\_n}\\$ as \\${\\hat{\\mathcal{D}}\\_1, \\ldots, \\hat{\\mathcal{D}}\\_n}\\$. An experiment focusing on this could help verify whether the method maintains the relative ranking robustly, regardless of the composition of the document set. This would ensure that the method is consistent in preserving the order of the original documents even when new documents are added.\n\nQUESTIONS:\nSee the Weaknesses section.\n\nLIMITATIONS:\nSee the weaknesses section.\n\nI am willing to increase the rating based on the authors response on my questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach for information retrieval via second-stage reranking using LLMs. The limited context length of LLMs necessitates breaking down large first-stage candidate sets into small batches, and the high inference cost motivates minimizing the number of LLM calls. Existing approaches like sliding-window and tournament-style methods apply a fixed computational budget. The proposed approach models uncertainty in document relevance scores using the TrueSkill model (represents relevance not as a single point value, but as a Gaussian distribution) and uses it to decide which documents need to be reranked by the LLM. This allows dynamic choice of documents for comparison and early stopping when the uncertainty for top ranked documents is small. The approach shows promise in terms of accuracy-efficiency trade-off on several retrieval benchmarks (TREC-DL and BEIR).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The proposed approach is principled and well justified.\n* Using relevance uncertainty to select documents for reranking with TrueSkill is an elegant solution.\n* The approach serves as a meta-framework using the LLM as a black-box reranker, making it a practical and broadly applicable approach.\n* Empirical evaluation shows compelling results in terms of accuracy-efficiency trade-off.\n\nWeaknesses:\n* Unaddressed computational overhead from TrueSkill (see detailed questions below).\n* Closely related paper needs to be added to related work.\n\nQUESTIONS:\n* Is t(k) needed for the algorithm? Is it computed from the \\mu_i’s? This was not clear to me.\n* Can you please comment on the computational cost of the TrueSkill overhead (outside LLM calls)?\n* Please detail how your approach relates to [Budget-Constrained Document Re-ranking with Bayesian LM-Based Pairwise Comparisons](https://openreview.net/pdf?id=3m34mqglmh)\n\nMinor / Typos\n* Line 80: missing space “TDPart[25]”\n* Line 149: “the the”\n* Line 189: was \\tau defined?\n\nLIMITATIONS:\nSee Weaknesses/Questions above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel adaptive reranking framework called AcuRank, aiming to improve the efficiency and accuracy of listwise reranking with large language models (LLMs). The core idea is to dynamically adjust computational resources (including computational effort and objectives) based on uncertainty estimates of document relevance. AcuRank employs a Bayesian TrueSkill model to iteratively refine relevance estimates until a sufficient confidence level is reached. This explicit modeling of ranking uncertainty enables principled control over the reranking behavior, avoiding unnecessary computations on already confident predictions. The authors conducted experiments on TREC-DL and BEIR benchmarks, across different retrieval tasks and LLM-based reranking models. The results demonstrate that AcuRank consistently outperforms fixed-computation baselines, exhibiting excellent accuracy-efficiency trade-offs and better scalability with computational resources.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1.\tClear Motivation Statement: The introduction clearly articulates the limitations of existing fixed-computation reranking methods and the motivation behind adopting an adaptive approach, presenting a very logical flow. \n\n2.\tNovelty and Significance: The concept of uncertainty-aware adaptive computation for LLM reranking is highly novel, directly addressing critical practical challenges such as the high inference cost and fixed context lengths of current LLM rerankers. Specifically, the use of the Bayesian TrueSkill model for explicitly modeling uncertainty offers a theoretically rigorous approach. \n\n3.\tStrong Experimental Results: The consistent improvements in accuracy-efficiency trade-off and better computational scalability demonstrated on established benchmarks (TREC-DL and BEIR), and with different LLM rerankers (RankZephyr, RankGPT), provide strong empirical support for AcuRank's effectiveness and generality.\n\nWeaknesses: \n\n1.\tInsufficient Theoretical Depth: Complexity of Bayesian TrueSkill Model Integration: While the Bayesian TrueSkill model excels in game and competitive rankings, its application to the iterative process of document reranking, particularly the mapping of its inherent mechanisms and parameters (e.g., skill values, uncertainty, definition of \"match outcomes\"), is not immediately obvious. The paper could provide a deeper explanation of this part, for instance, how to precisely define the \"match\" results of documents during each reranking iteration, and how these results update the skill (relevance scores) and uncertainty in the TrueSkill model. \n\n2.\tInsufficient Theoretical/Empirical Basis for Uncertainty Thresholds: A core aspect of AcuRank lies in deciding when to stop based on uncertainty thresholds. However, the paper lacks sufficient theoretical guidance or sensitivity analysis regarding the selection of these thresholds (e.g., ϵstop and σtarget). \n\n3.\tLack of Generalization Analysis across Different LLM Behaviors: The paper uses RankZephyr and RankGPT as LLM rerankers and shows AcuRank's effectiveness for both. However, there is a lack of additional baselines; for example, considering the impact of first-stage retrieval models like monoBERT, or comparisons with models like GPT-4/GPT-3.5. \n\n4.\tUnreasonable Evaluation of Efficiency. This paper uses the “number of reranker calls” to evaluate model efficiency. However, this is not an appropriate metric, as the time cost depends not only on the number of reranker calls but also on the context length of each call.\n\nQUESTIONS:\nBasis for ϵ stop and σ target Selection and Sensitivity Analysis: I noticed that AcuRank's performance heavily depends on the setting of uncertainty thresholds. Could the authors provide the rationale for choosing these hyperparameters and a more detailed sensitivity analysis?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AcuRank, an uncertainty-based adaptive computation framework for listwise re-ranking. The main contributions are as follows:\n\n1. Introduction of a TrueSkill-based probabilistic relevance modeling approach, which enables the estimation of uncertainty in document ranking.\n2. Development of a novel listwise re-ranking framework that supports adaptive computation by iteratively refining and selectively re-ranking uncertain candidate documents.\n3. Extensive experimental validation across various benchmarks and datasets, demonstrating AcuRank’s superior trade-off between accuracy and efficiency, as well as its generalization capability in different retrieval tasks and LLM-based re-ranking models.\n\n\nAcuRank employs a Bayesian TrueSkill model to maintain probabilistic estimates of document relevance, iteratively refining these estimates until a sufficient confidence level is reached. By explicitly modeling ranking uncertainty, this approach allows for rational control of re-ranking actions and avoids unnecessary updates for predictions that are already highly certain. On the TREC-DL and BEIR benchmarks, AcuRank consistently achieves a better balance between accuracy and efficiency compared to fixed-computation baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. AcuRank is the first to introduce uncertainty modeling into listwise re-ranking tasks, enabling fine-grained control over uncertainty through dynamic allocation of computational resources, which demonstrates high originality.\n2. Across multiple benchmarks, AcuRank achieves a better trade-off between accuracy and efficiency. For example, on the TREC-DL and BEIR datasets, AcuRank consistently improves ranking quality under various re-ranking models while requiring fewer re-ranker invocations.\n3. The paper is well-written and easy to understand.\n\nWeaknesses:\n\nThe main limitation lies in insufficient experimental evaluation. The authors mention using TourRank and TrueSkill-Static as baselines; however, Table 1 does not report results for TrueSkill-Static, and the lower part of Table 1 lacks results for TourRank. Table 2 exhibits similar issues. To robustly assess AcuRank’s performance across retrieval scenarios, comprehensive comparisons with these baseline models are necessary. Additionally, a comparison with setwise methods is also needed.\n\nQUESTIONS:\nQ1: AcuRank dynamically allocates computational resources to handle uncertain documents. However, in certain scenarios, could this dynamic adjustment lead to either over-allocation or under-allocation of resources? For example, when processing highly complex queries, is there a risk that the model may terminate computation prematurely and consequently overlook some important documents?\n\nQ2: AcuRank employs the TrueSkill model to estimate the uncertainty of document relevance. However, is this uncertainty estimation method sufficiently accurate?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}