JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking

## Abstract

Abstract Accurate document retrieval is crucial for the success of retrieval-augmented generation (RAG) applications, including open-domain question answering and code completion.
While large language models (LLMs) have been employed as dense encoders or listwise rerankers in RAG systems, they often struggle with reasoning-intensive tasks because they lack nuanced analysis when judging document relevance.
To address this limitation, we introduce JudgeRank , a novel agentic reranker that emulates human cognitive processes when assessing document relevance. Our approach consists of three key steps: (1) query analysis to identify the core problem, (2) document analysis to extract a query-aware summary, and (3) relevance judgment to provide a concise assessment of document relevance.
We evaluate JudgeRank on the reasoning-intensive BRIGHT benchmark, demonstrating substantial performance improvements over first-stage retrieval methods and outperforming other popular reranking approaches.
In addition, JudgeRank performs on par with fine-tuned state-of-the-art rerankers on the popular BEIR benchmark, validating its zero-shot generalization capability.
Through comprehensive ablation studies, we demonstrate that JudgeRank ’s performance generalizes well across LLMs of various sizes while ensembling them yields even more accurate reranking than individual models.

## 1 Introduction

Passage reranking is a critical component in modern information retrieval systems, designed to refine results obtained from efficient first-stage retrieval methods such as BM25 (Robertson et al., , ). By narrowing down the pool of candidate documents, reranking substantially improves the quality of downstream tasks, such as retrieval-augmented generation or RAG (Lewis et al., ). Two primary approaches have emerged to address the reranking task.
The first category comprises encoding-based approaches (Nogueira and Cho, ; Gao et al., ), which encode queries and documents into fixed-size embedding vectors. These methods use either cosine similarity as a score function or directly output a score from the model (Nogueira et al., ; Zhuang et al., ). While highly efficient, these approaches face several limitations. One major challenge is their inflexibility in defining relevance, making it difficult to accommodate diverse retrieval objectives (e.g., finding supporting vs. refuting evidence). Moreover, encoding-based models heavily rely on manual supervision signals due to the discrepancy between LLM pretraining and reranking objectives, limiting their ability to generalize to new domains or models (Nguyen, ; Izacard et al., ).

Most recently, utilizing Large Language Models (LLMs) for document reranking has led to promising progress in addressing some of these challenges, owing to their superior capabilities in language understanding, generation, interaction, and reasoning (Ouyang et al., ).
These approaches utilize an LLM either as a pointwise judge (Ma et al., ) or a listwise reranker (Sun et al., ; Zhuang et al., ). While these approaches allow for flexible definition of document relevance and support zero-shot operation, they still require the model to make decisions without intermediate analyses. Consequently, they fall short in scenarios requiring complex reasoning (Su et al., ), hampering both performance and interpretability. Moreover, listwise rerankers face significant computational challenges due to context length constraints, often compromising on individual document length when processing multiple documents simultaneously.

To bridge this gap, we propose JudgeRank, a novel zero-shot pointwise reranker tailored for reasoning-intensive text retrieval tasks.
Inspired by Chain-of-Thought (Wei et al., ) and LLM-as-a-Judge (Zheng et al., ) methods, JudgeRank utilizes highly generalizable prompts to guide instruction-tuned LLMs through explicit reasoning steps before arriving at a final judgment.

Figure 1 illustrates a real example of how our model works on the Biology dataset in the BRIGHT (Benchmark for Reasoning-Intensive Generative Retrieval Tasks) benchmark (Su et al., ). Specifically, our reranker first prompts the LLM to identify the core problem in the query, allowing it to focus on the central question while filtering out irrelevant context. Next, the model produces an extractive summary for each of the candidate documents and explains how it addresses the query. Finally, the model makes a relevance judgment based on the previous analyses. This process closely mimics how humans approach questions: by first skimming the document, identifying relevant parts, and then carefully reading these parts to obtain an answer (Masson, ).
This structured pipeline enables JudgeRank to transcend surface-level lexical matching, leveraging deeper semantic understanding to improve reranking accuracy.

We evaluate JudgeRank on the recently constructed BRIGHT benchmark, widely regarded as one of the most challenging retrieval evaluation datasets. Despite the poor performance of state-of-the-art text embedding models and rerankers on this benchmark, our method achieves significant improvements over all existing baselines and secures the top position on the BRIGHT benchmark leaderboard among $89$ models, surpassing the previous best model by a significant margin ($9$ points).(^1^11[https://brightbenchmark.github.io/](https://brightbenchmark.github.io/))
Additionally, we demonstrate that JudgeRank readily generalizes to other popular retrieval benchmarks such as BEIR and performs competitively with state-of-the-art rerankers.
We also analyze the complementarity of models at different scales by investigating the alignment of their ranking decisions.
Contrary to our expectations, we observe that models of different sizes demonstrate a surprisingly orthogonal behavior on their relevance judgments.
Leveraging this key finding, we propose a simple ensembling strategy that allows us to combine multiple models flexibly and demonstrate that it translates to considerable performance gains on the final ranking.

Figure: Figure 1: A step-by-step illustration of how JudgeRank arrives at the final judgment through query and document analyses. The query analysis identifies the core problem being asked, while the document analysis extracts relevant sentences from the document based on the query. This is a real example from the Biology task in the BRIGHT evaluation benchmark.
Refer to caption: x1.png

## 2 Method

Figure: Figure 2: (a) Prompt to analyze query, where {query name} (e.g., “Biology post”) and {query} are placeholders for the query type and content. (b) Prompt for analyzing a document, where {doc name} (e.g., “document”) and {doc} are placeholders for the document type and content. (c) Prompt for making the final one-word relevance judgment.
Refer to caption: x2.png

### 2.1 Agentic steps

Mimicing human cognitive process, JudgeRank consists of three main steps: Query Analysis, Document Analysis, and Relevance Judgment. The prompt templates for these steps are illustrated in Figure 2.

#### Query analysis

The query analysis prompt (Figure 2 (a).) directs the LLM to analyze the query by identifying the core problem being asked. Note that this prompt only depends on the query so that we can generate the query analyses separately and store them. Since the number of queries $n_{_}q$ is usually much smaller than that of documents $n_{_}d$ ($n_{_}q\ll n_{_}d$), we can afford to use a more expensive LLM (e.g., GPT-4) to handle this important step, and leave the other steps to relatively smaller LMs.

#### Document analysis

The document analysis prompt (Figure 2 (b).) asks the LLM to output an extractive summary of the document that helps answer the query, and assess the overall relevance of the document based on the summary.

#### Relevance judgment

The judgment prompt (Figure 2 (c).) asks the model to make a one-word judgment, either “Yes” or “No”. We isolate this step to make it easier to ensemble with different judgment prompts or models.

### 2.2 Generalizability of the prompts

All three steps consume natural language as input and generate a response, making it more flexible to transfer and stack across different LLMs. The templates also show that the prompts are highly generalizable: to adapt them to a new reranking task, one only needs to replace the query name, the document name, and the relation between them. Leveraging LLMs in a zero-shot setting allows us to flexibly define the relation between the query and the document. This flexibility is important because the user may define either “document that supports a query” or “document that refutes a query” as the relation, which are opposites of each other. Encoding-based models usually cannot achieve such behavior zero-shot because most of them use cosine similarity or metrics alike to represent “relevance”. One way encoding-based models could achieve such flexibility is through extensive fine-tuning. However, this requires additional training data and introduces new model parameters, potentially causing an unintended distribution shift. Similarly, models like RankZephyr also finetunes the LLM to output document ids, thus suffering from a distribution shift as well.

### 2.3 Methodology of scoring documents

#### Discrete version

The discrete version creates a binary partition between accepted (when the model outputs a “Yes”) and rejected (when the model outputs a “No”) documents, maintaining the first-stage retrieval ranking within each category. More specifically, let $D=\{d_{_}1,d_{_}2,\ldots,d_{_}k\}$ be an ordered list of top-$k$ documents ranked by the first-stage retrieval model. Let $D_{_}y$ and $D_{_}n$ be a partition of $D$, such that

$$ $\displaystyle D_{_}y\cup D_{_}n$ $\displaystyle=D$ (1) $\displaystyle D_{_}y\cap D_{_}n$ $\displaystyle=\emptyset$ (2) $$

, where $D_{_}y$ is the set of documents that the reranker judged as relevant and $D_{_}n$ is the set of documents that the reranker judged as non-relevant. Let $R$ be the reranking function which maps each document $d$ to its rank (lower rank means “more relevant”), then

$$ $\forall d\in D_{_}y\ \text{and}\ d^{\prime}\in D_{_}n,\ R(d)<R(d^{\prime})$ (3) $$

, and for the relative ranking within each partition,

$$ $\forall d_{_}i,d_{_}j\in D_{_}y,\ R(d_{_}i)<R(d_{_}j)\iff R_{_}0(d_{_}i )<R_{_}0(d_{_}j)$ (4) $$

, where $R(d)$ is the final rank of document $d$ and $R_{_}0(d)$ is the rank of $d$ from the first-stage retrieval. The same applies to $D_{_}n$.

While straightforward, this approach is sensitive to prompt wording and relies heavily on first-stage retrieval performance.

#### Continuous version

The continuous version addresses the limitations of the discrete version by using the probability of the “Yes” judgment $p_{_}y$ and the probability of the “No” judgment $p_{_}n$ to construct a complete ranking. This is similar to the relevance generation approach proposed in Liang et al. (). The scoring function $S$ is defined by normalizing the probabilities between $p_{_}y$ and $p_{_}n$ as follows:

$$ $S(d)=\frac{p_{_}y}{p_{_}y+p_{_}n}$ (5) $$

This normalization step is necessary because the combined probability mass allocated to $p_{_}y$ and $p_{_}n$ is not always a fixed value across different documents. Without normalization, the $p_{_}y$ values for different documents would not be directly comparable.

The final ranking $D_{_}\text{R}$ is then defined as $D_{_}\text{R}=\{d_{_}1,d_{_}2,\ldots,d_{_}k\}$ such that

$$ $\forall i,j\in\{1,2,\ldots,k\},\ i<j\iff S(d_{_}i)>S(d_{_}j)$ (6) $$

This continuous version provides a more fine-grained ranking compared to the binary partition, as it utilizes the full range of probabilities output by the LLM.

#### Hybrid version

Additionally, we explore a variant of the continuous version where the final score is computed by taking a weighted sum of the probability score $S_{_}\text{prob}$ and the BM25 score $S_{_}{\text{BM25}}$. More specifically, the final score is computed by

$$ $S=\alpha S_{_}\text{prob}+S_{_}{\text{BM25}}$ (7) $$

, where $S_{_}{\text{BM25}}$ is the score provided by BM25 in the first-stage retrieval, and $\alpha$ is the relative weight of the probability score. We set $\alpha=100$ in this work to bring $S_{_}\text{prob}$ to the same scale as $S_{_}{\text{BM25}}$. This version leverages model ensembling to consider both reasoning and surface-level matching, thus marrying the benefits of both approaches. Unless otherwise specified, we use this setting to compute the final score throughout the paper. In the ablation studies, we compare these three settings and show their relative performances.

## 3 Experimental Setup

### 3.1 Datasets

#### BRIGHT

We use the BRIGHT benchmark (Su et al., ) to assess the performance of our reranker. BRIGHT is specifically designed to evaluate text retrieval systems on complex, reasoning-intensive queries that go beyond simple keyword matching. The benchmark comprises $1\text{,}398$ real-world queries spanning diverse domains, including economics, psychology, robotics, math, and software engineering. These queries are carefully curated to represent challenging scenarios that require deep understanding and reasoning to identify relevant documents. We use this dataset to evaluate our approach because unlike traditional benchmarks that focus on simple information-seeking tasks, BRIGHT queries require complex reasoning to determine document relevance, making it an excellent tool for evaluating advanced retrieval systems in realistic scenarios. The benchmark has also been validated to be robust against potential data leakage, maintaining its effectiveness even when benchmark documents have been included in model training data.

Because of its challenging nature, state-of-the-art retrieval models have shown significantly lower performance on BRIGHT compared to other benchmarks (Su et al., ). For example, the leading model on the MTEB leaderboard (Muennighoff et al., ) achieves an nDCG@10 of only $18.0$ on BRIGHT, compared to $59.0$ on other benchmarks. The GPT-4 listwise reranker also only improves around $2$ points on nDCG@10 on top of the BM25 first-stage retrieval, while Gemini (Team et al., ) features less improvement than that. The cross-encoder reranker MiniLM (Wang et al., ) even significantly underperforms the BM25 baseline.

#### BEIR

To test the generalizability of our approach, we also evaluate on the BEIR benchmark (Thakur et al., ), a robust and heterogeneous evaluation benchmark for information retrieval. We evaluate on all tasks in this benchmark that are publicly available (Kamalloo et al., ). For all datasets we use the the test set, except for MSMARCO where we follow BEIR convention to evaluate on the dev set. For cqadupstack we follow BEIR convention and evaluate on all sub-datasets and compute their average.
Because BEIR is a large benchmark, and the largest dataset has more than $13$K queries, we only generate query analysis to evaluate on this dataset. This almost adds no overhead to the generation because the query analysis generation does not depend on the document. Since our model only generates a single token “Yes” or “No”, its latency is almost the same as encoding both the query and the document with an encoding-based retrieval model, making it a highly efficient alternative.

### 3.2 First-stage retrieval

For both benchmarks we evaluate on, we follow common settings from previous work to rerank the top-$100$ documents from the first-stage retrieval and use nDCG@10 score as the evaluation metric. This metric assesses the quality of the retrieved documents, taking into account both their relevance and ranking position.

#### BRIGHT

The original BRIGHT paper explores using LLMs to generate Chain-of-Thought (Wei et al., ) reasoning steps as queries (Su et al., ), resulting in up to $12.2$ point improvements on average. We thus build on top of this best first-retrieval model on the leaderboard,
which achieves an nDCG@10 score of $26.5$ with BM25 and reasoning chains generated by GPT-$4$-$0125$-preview.

#### BEIR

We follow the original BEIR paper (Thakur et al., ) and use ElasticSearch BM25 (Elasticsearch, ) as the first-stage retriever.

### 3.3 Base model

Our main model builds on top of Llama-3.1-70B-instruct (Dubey et al., ). We choose the Llama-3.1 model family because its RoPE scaling Liu et al. () allows longer context length up to $128$K, which is essential in handling long documents. To speed up experiments, we evaluate on a quantized version of this model, namely Llama-3.1-70B-instruct-awq-int4.(^2^22[https://huggingface.co/hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4](https://huggingface.co/hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4)) We also perform ablation studies where we evaluate on Llama-3.1-8B(^3^33[https://huggingface.co/meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B)) and Llama-3.1-405B-instruct-awq-int4.(^4^44[https://huggingface.co/hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4](https://huggingface.co/hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4)) Note that for the 8B model we do not use the quantized version because it can already fit on a single A$100$ GPU, while the other two bigger models require quantization to save computational cost. More specifically, the $70$B version requires at least $2$ x A$100$ GPUs, while the $405$B version requires $8$.

### 3.4 Baseline rerankers

We reproduce RankLlama (Ma et al., ) and RankZephyr (Pradeep et al., ), two state-of-the-art rerankers as evaluated by the BEIR benchmark. RankLlama is a pointwise reranker that directly outputs a score. This model is trained on the MS MARCO passage ranking dataset (Bajaj et al., ). RankZephyr is a listwise reranker that takes a query and a list of documents together as input and outputs a ranking. This model uses the queries sourced by Sun et al. () from the MS MARCO dataset to distill GPT-3.5 and GPT-4 in sequence. We use the rerankers library (Clavié, ), a lightweight unified API that allows users to run diverse reranking models out-of-the-box.(^5^55[https://github.com/AnswerDotAI/rerankers](https://github.com/AnswerDotAI/rerankers))

### 3.5 Efficiency and Optimization

To make the encoding and generation more efficient, we use vLLM (Kwon et al., ). This library leverages paged attention, which improves the throughput of popular LLMs by $2$ - $4$x with the same level of latency. Importantly, when designing the prompts used in our approach, we append the query and the document at the very end of each prompt to make the best use of Automatic Prefix Caching (Gim et al., ), which temporarily stores the KV cache of existing inputs so that a new input can directly reuse the KV cache if it shares the same prefix with one of the existing ones. This allows the new inputs to skip the computation of the shared part. This design greatly improves the efficiency of our experiments.

## 4 Results and Discussion

### 4.1 Main Results

#### BRIGHT

As shown in Table 1, JudgeRank achieves state-of-the-art results on the BRIGHT evaluation benchmark as measured by nDCG@10. Our best preforming model improves upon the no-rerank baseline by more than $9$ points, while RankLlama underperforms the baseline and RankZephyr stays barely above the baseline. The smaller Llama-$3.1$-$8$B-instruct also outperforms the baseline by more than $3$ points, showing the generalizability of our approach across different model sizes. Interestingly, increasing model size from $70$B to $405$B does not bring a significant gain on nDCG@10. We thus select the $70$B version as our main model to balance between efficiency and performance.
In contrast, the original BRIGHT paper shows that GPT-4 with listwise reranking improves on top of vanilla BM25 baseline by an average of $2.7$ points on nDCG@10, a much smaller improvement than our approach despite using a much stronger LLM.

**Table 1: JudgeRank nDCG@10 results on the BRIGHT evaluation benchmark. Best results on each dataset and the entire benchmark are boldfaced. “Ensemble” stands for model ensembling of JudgeRank-$8$B, $70$B, and $405$B.**
|  | BM25 | RankLlama | RankZephyr | JudgeRank |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | 8B | 70B | 405B | Ensemble |  |  |  |
| Biology | 53.63 | 11.05 | 44.37 | 54.44 | 57.59 | 60.33 | 60.70 |
| Earth Science | 53.65 | 11.83 | 35.24 | 54.62 | 58.36 | 55.11 | 58.72 |
| Economics | 24.28 | 9.56 | 24.44 | 28.04 | 33.01 | 32.15 | 35.39 |
| Psychology | 38.59 | 11.38 | 36.92 | 42.16 | 46.59 | 45.42 | 47.57 |
| Robotics | 18.77 | 8.53 | 18.80 | 24.09 | 28.30 | 27.64 | 28.16 |
| Stack Overflow | 22.74 | 11.40 | 19.62 | 27.18 | 27.47 | 28.30 | 29.74 |
| Sustainable Living | 25.90 | 11.75 | 29.38 | 30.69 | 39.55 | 38.54 | 41.88 |
| Leetcode | 19.27 | 20.32 | 24.58 | 17.49 | 20.06 | 22.47 | 20.23 |
| Pony | 17.73 | 18.88 | 48.96 | 22.85 | 30.82 | 31.54 | 32.74 |
| Aops | 3.92 | 3.55 | 6.98 | 6.15 | 8.24 | 7.74 | 8.57 |
| TheoremQA-Questions | 18.90 | 11.82 | 22.34 | 24.09 | 23.83 | 26.93 | 25.86 |
| TheoremQA-Theorems | 20.22 | 5.63 | 7.78 | 29.89 | 34.55 | 36.16 | 36.20 |
| Average | 26.47 | 11.31 | 27.25 | 30.14 | 34.03 | 34.36 | 35.48 |

#### BEIR

As shown in Table 2, our model delivers competitive results on the BEIR evaluation benchmark despite the fact that RankLlama and RankZephyr are heavily fine-tuned on in-domain data including MS MARCO, which is part of the BEIR benchmark.

**Table 2: JudgeRank nDCG@10 results on the BEIR evaluation benchmark. Best results on each dataset and the entire benchmark are boldfaced.**
|  | BM25 | JudgeRank | RankLlama | RankZephyr |
| --- | --- | --- | --- | --- |
| webis-touche2020 | 34.71 | 27.94 | 32.97 | 33.34 |
| trec-covid | 68.80 | 83.70 | 81.08 | 85.77 |
| scifact | 69.06 | 73.24 | 75.57 | 74.68 |
| nfcorpus | 34.28 | 37.73 | 35.92 | 38.79 |
| dbpedia-entity | 32.02 | 44.30 | 43.72 | 44.19 |
| fiqa | 25.36 | 40.35 | 42.70 | 40.40 |
| scidocs | 16.47 | 19.47 | 19.26 | 19.70 |
| arguana | 47.16 | 62.77 | 32.08 | 47.07 |
| nq | 32.61 | 56.87 | 59.19 | 60.10 |
| climate-fever | 18.61 | 19.00 | 17.75 | 24.26 |
| fever | 64.94 | 69.97 | 78.55 | 79.52 |
| msmarco | 22.75 | 34.01 | 41.40 | 37.89 |
| hotpotqa | 60.22 | 68.30 | 70.65 | 71.16 |
| quora | 80.77 | 84.25 | 82.73 | 79.60 |
| cqadupstack | 32.53 | 43.98 | 42.43 | 42.68 |
| Average | 42.69 | 51.06 | 50.40 | 51.94 |

### 4.2 Discussion

We pose several research questions to illustrate whether and how our approach works.

#### How complementary are LLMs of different scales?

In Table 1, we observe that JudgeRank-$70$B and JudgeRank-$405$B performs on par with each other. However, nDCG@10 alone does not reveal the whole picture. One natural question to ask is: do these two models make similar judgments or are complementary to each other? To answer this question, we obtain statistics on the percentage of both models agreeing and disagreeing each other and show them on the left of Figure 3. From the tables we can see that for all three combinations of the models, the majority case is always that both models rejects the documents. This is understandable because only a few out of the top-$100$ documents are supposed to be relevant. The interesting pattern emerges when we inspect the other three cases: each pair of the models spends more time disagreeing with each other than both outputting “Yes”. For the pairs $8$B vs $70$B and $8$B vs $405$B, there is a higher difference because the capabilities of the two models differ more. In contrast, for $70$B vs $405$B there is less disagreement. From these observations, we indeed see that each two models may be complementary to each other.

Motivated by this observation, we take model ensembling one step further. Recall that in Section 2.3, we ensemble the BM25 score with each of the scores output by the Llama models. Here we first take the average score output by all the Llama models, and then perform the weighted sum with the BM25 score. More specifically, let $S_{_}{8\text{B}}$, $S_{_}{70\text{B}}$, and $S_{_}{405\text{B}}$ be the score assigned by each model, respectively, the ensemble score of the three models is computed as $\alpha(S_{_}{8\text{B}}+S_{_}{70\text{B}}+S_{_}{405\text{B}})/3+S_{_}{
\text{BM25}}$, where again $\alpha=100$ and $S_{_}{\text{BM25}}$ is the score given by the BM25 model. The same equation generalizes analogously to two-model ensembles.

We present all ensembling results on the right of Figure 3. We can see that each ensembling performance is better than its individual model performances, with the strongest performance observed when ensembling all three models. This result shows that a salient performance boost can be achieved by ensembling two of the strongest models ($70$B + $405$B), while even the model with lower performance (i.e., $8$B) could contribute positively in model ensembling. Intuitively, such ensembling is equivalent to a verification or a majority voting step. The final score is the highest when both models say “Yes”, the score is medium when one of the two says “No”, and the lowest score is observed when both say “No”.

Figure: Figure 3: On the left: judgment alignment studies for models of three sizes: $8$B, $70$B, and $405$B. Percentages are shown for each quadrant. On the right: nDCG@10 of each individual model and model ensembling on the BRIGHT evaluation benchmark.
Refer to caption: x3.png

#### How does the choice of reranking score impact the final performance?

Recall that to compute the final score, we take a weighted sum of the BM25 score and the probability score from the judgment step. To justify this choice, we compare it with two other settings: the first is binary judgment, and the second only uses the normalized probability to rerank documents (introduced in Section 3). The left part of Figure 4 shows that binary judgment performs the worst among the three settings while using only probability achieves somewhere in between.
This is understandable because binary judgments are sensitive to wordings. Imagine that if we change the relation from “substantially helps answer” to “helps answer” or “at least partially helps answer,” the number of “Yes” that the model outputs will keep increasing, thus also increasing the number of false positives. However, the other two settings are not sensitive to such changes.

#### How useful are the query and document analysis steps?

To show the effectiveness of the analysis steps, we perform an ablation study on BRIGHT. We remove the two analyses steps and keep the judgment step untouched to compare its performance with the original approach.
The right of Figure 4 shows that judging with query and document analyses performs consistently better than the direct judgment approach.

Figure: Figure 4: Ablation studies of JudgeRank. On the left: Comparison of three scoring settings on the BRIGHT evaluation benchmark. Binary stands for binary judgment, Prob stands for probability, and Hybrid stands for a weighted sum of BM25 and probability scores. On the right: Comparison of direct judge and judge with query and document analyses on the BRIGHT evaluation benchmark.
Refer to caption: x5.png

#### Qualitative examples

Figure 5 demonstrates how JudgeRank enhances document relevance identification using real examples from the BRIGHT dataset. In the left example, we observe a document initially ranked high by the first-stage retriever due to significant word overlap with the query. However, JudgeRank’s deeper analysis correctly judges this document as irrelevant. Despite surface-level similarities, the reranker fails to extract any sentences that help answer the query, revealing that the document is merely an advertisement coincidentally sharing common terminology with the query. The right example presents a contrasting scenario, where a document initially ranked low by the first-stage retriever due to minimal word overlap with the query is accurately identified by JudgeRank as highly relevant. In this instance, the document analysis prompt enables the LLM to pinpoint key sentences that elucidate the underlying mechanism of funnel web spider venom’s lethality, precisely addressing the query’s intent. These extracted sentences further inform the LLM to make the final positive judgment, demonstrating JudgeRank’s ability to uncover deeply relevant content that might be overlooked by traditional retrieval methods.

Figure: Figure 5: Illustration of how agentic generations of JudgeRank help identifying the relevant documents. On the left, the document is ranked high by the first-stage retriever but judged as negative by the reranker. On the right, the document is ranked low by the first-stage retriever but judged as positive by the reranker because the document analysis prompt helps the LLM to locate the relevant sentences that answer the query.
Refer to caption: extracted/5968050/figure/judge-rank-extractive-summary.png

## 5 Related Work

The field of reranking models can be understood through two primary dimensions.

The first dimension distinguishes between encoding-based and LLM-based approaches. Encoding-based models (Nogueira and Cho, ; Gao et al., ), which have been the standard since the introduction of BERT, typically require extensive training to adapt to the specific objectives of reranking tasks. In contrast, LLM-based models (Sun et al., ; Tang et al., ; Qin et al., ; Zhuang et al., ), particularly those with decoder-only architectures, often demonstrate impressive zero-shot capabilities, allowing them to perform effectively without task-specific fine-tuning. However, it’s worth noting that some researchers have explored pretraining, fine-tuning and distillation techniques for LLM-based rerankers to further enhance their performance (Zhang et al., ; Ma et al., ; Yu et al., ).

The second dimension differentiates between pointwise and listwise reranking strategies. Pointwise rerankers (Ma et al., ; Guo et al., ) evaluate the relevance of individual query-document pairs in isolation, producing a score for each pair without direct comparison between documents. Listwise rerankers (Sun et al., ; Pradeep et al., ; Ma et al., ; Yoon et al., ; Liu et al., ; Tang et al., ), on the other hand, consider the entire set of documents for a given query, generating a fully ordered list as output. To address the challenges posed by input length limitations, researchers have developed innovative techniques such as sliding window approaches (Sun et al., ), which allow for the ranking of smaller subsets of documents before aggregating them into a comprehensive ranking. This framework encompasses pairwise (Pradeep et al., ; Qin et al., ) and setwise (Zhuang et al., ) rerankers as specific instances of the broader listwise category.

Our contribution is a novel LLM-based, pointwise reranker that leverages the capabilities of instruction-tuned language models and incorporates explicit reasoning steps in the relevance judgment process. This approach sets our work apart from previous efforts by enhancing the model’s ability to handle complex, reasoning-intensive reranking tasks while simultaneously improving the interpretability of its decisions.

## 6 Conclusion

In this work, we target document retrieval tasks that require intensive context-based reasoning, which even the strongest retrieval models struggle to achieve satisfactory performance. Through experiments and ablation studies, we show that our agentic reranker can effectively recover low-ranked documents and outperform previous state-of-the-art reranking models, while remaining flexible and efficient. In section 4.2, we have shown the significant benefit of model ensembling on document reranking; yet we do not have to stop there. There are at least two categories of ensembling that we envision.
First, sampling ensembling: For each generation, we sample several generations, each of which could lead to a different judgment. This kind of ensembling is similar to the self-consistency approach (Tang et al., ).
Second, prompt ensembling: we could leverage paraphrases of the same prompt to perform ensembling.
We leave the exploration as future work because such approaches could be generalized to many prompting-related tasks, and thus better be addressed in separate works.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: **Base models:** have you explored base models other than Llama3.1?

**Prompt design and variants:** How much impact do the prompts have on JudgeRank’s performance? The paper mentions potential prompt variations -- when would different prompts be applicable, and how might they influence retrieval effectiveness?

**Ensembling** Maybe I missed this, but can you share the model ensemble performance on the BEIR benchmark?

**Listwise reranking:** I understand this is somewhat orthogonal to your approach, but do you have any thoughts on how you might adapt your technique + intuition to listwise reranking (instead of pointwise).

**Typos / Grammar / Presentation**
- l138 - mimicking
- l312 - performing
- l271 - best-first retrieval? best retrieval-first model?
- Table 2 - I might suggest presenting the models in the same order as the prior table. Also, it might be helpful if you explicitly re-state (maybe in the caption?) the details of this JudgeRank variant (that it's the 70B variant w/ ensemble)
soundness: 3
strengths: The paper is well-written and very easy to follow. The technique is intuitive, taking retrieved top-k documents, and performing passage reranking through the use of instruction-tuned LLMs (LLama-3.1 model family). They do some interesting analysis, such as exploring the similarities between model predictions at different sizes (8b vs 70b vs 405b), exploring ensembling and finding this can improve accuracy, and also doing component-level analysis of the JudgeRank pipeline (comparing different scoring metrics, and also investigating the benefits of this modular reasoning approach).
summary: This paper introduces JudgeRank, a new method for LLM-based pointwise passage reranking. The method diverges from existing methods in that they apply LLMs to do passage-level relevance scoring, breaking this process into three LLM-based steps: 1) query decomposition to understand the core problems needing to be solved 2) query-focused summarization of each document and 3) combining these two together outputs together to assign a relevance score to each document. They test their method on two popular reranking benchmarks, finding their method outperforms vanilla first-stage retrieval, and also two baseline LLM methods fine-tuned for this task.
weaknesses: Overall, the novelty of this work is somewhat lacking -- the method is effectively comprised of a few intuitive prompting steps which, while effective, are fairly intuitive and don't introduce any fundamentally new way of tackling more challenging reasoning; the authors note this work was inspired by popular techniques such as Chain-of-Thought and LLM-as-a-Judge, and it seems this is more or less a straightforward application of existing ideas to a particular use case.

I think some additional steps/experiments could help to bolster the paper and add further takeaways; some concrete directions might be (1) to dig deeper on the base model selection, understanding if/how other model families (both open and propietary models, so long as they provide relevant confidence scores) compare on this task, (2) further exploring aggregation/ensembling methods with better cost:performance ratio (e.g., what happens if you ensemble a mixture of several smaller models from different families), and (3) investigating more 'adaptive' prompting configurations targeted to each use case (e.g. you note: "Leveraging LLMs in a zero-shot setting allows us to flexibly define the relation between the query and the document....this flexibility is important because the user may define either “document that supports a query” or “document that refutes a query...”).

It would also be interesting to understand what the current 'ceiling' is for this task. It seems there is a first-stage BM25 step applied, and I'm curious what impact this has on final performance -- ie, what are the implications if you vary k during the first-stage (from latency/scaling to performance); could your method potentially perform better without the first-stage retrieval step?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. How does your computational efficiency compare to traditional encoding-based frameworks? Could you provide more details such as throughput rate, inference speed, and memory usage?
2. What are your thoughts on using closed-source LLMs like OpenAI's models? There might be situations where you can't obtain "yes" and "no" logits in the top@k (k=20?) logits.
3. Could you provide insights on why the performance difference between Llama 70B and 405B models isn't significant?
4. Perhaps you could use different types of models (maybe same parameters) to create an ensemble?
soundness: 3
strengths: 1. JudgeRank performs well in BRIGHT and is compatible with other methods in BEIR.
2. JudgeRank is more suitable for (RAG) tasks that require complex reasoning.
3. The paper is well-written.
summary: JudgeRank represents an advancement in reasoning-intensive document retrieval, demonstrating that structured reasoning approaches can substantially improve reranking performance. The method's success in both specialized (BRIGHT) and general (BEIR) benchmarks validates its effectiveness and generalizability, while the proposed model ensembling techniques offer promising directions for future improvements.
weaknesses: 1. The computational cost of multiple LLM inferences may not be justified by the performance gains.
2. Sensitivity to prompt wording.
3. Maybe over-complex for simple retrieval tasks?
4. The inability to perform document analysis in advance may results in high costs for each query.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - Have you concluded anything from your experiments that differentiates between reasoning and knowledge heavy reranking tasks?
- Could you elaborate further on why do you think ensembling results are higher? Could it simply be the effect of randomness (especially given the large context window) of multiple retries and the advantage of a majority vote? Have you performed any experimentation to disprove this? majority vote measures with the same model for example?
- Have you explored COT judgement? (beside decoupled analyses)
- Have you explored ensembling with embedding based ranking methods?
soundness: 2
strengths: - What I appreciated most is the decoupling of analysis and judgement. Breaking down a complex task as so is an elevated approach, especially given the query-based document analysis.
- Inclusion of prompt templates.
- Experimentation across model sizes.
summary: The paper introduces an LLM-based document ranking pipeline given a query. The authors use 3 main stages, query analysis, document analysis and judgment. They evaluate their pipeline on 2 main benchmarks, BRIGHT and BEIR, with 3 different sized Llama models. The authors claim that their methodology performs better than existing state-of-the-art reranking models/methods.
weaknesses: - The paper could touch base further on the limitations of context length, this is a very intuitive limitation than can be mitigated with a parsing pipeline that creates overlapping shingles for example that are evaluated as separate documents…etc. Even in the presence of 128k context windows, models’ reasoning capabilities tend to decline with really large windows, making it harder to capture a nuanced understanding of the full context, potentially missing some aspects or attending to portions of the context in a manner biased to position. 
- Results could provide more example outputs.
- The pipeline might have a scaling limitation with a large number of documents. One thing that can be added to the pipeline is a global topic prompt per doc, followed by a filtering step per query for relevant documents given the latter (possibly even in batches).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - How do authors obtain the probability of the “Yes” judgment? This needs to be explained.
- The efficiency of JudgeRank is not examined in this paper. Including an analysis of its running time and a comparison of RankGPT and other LLM-based ranking methods in a fair setting would be better.
- What is “{definition of relevance}” set to in each task? How does it affect the re-ranking result?
soundness: 2
strengths: The proposed method is simple and effective, and generalizes well on a series of retrieval tasks.
summary: This paper presents JudgeRank, a LLM-based re-ranking framework for reasoning-extensive retrieval. It consists of three steps (components): query analysis, document analysis, and final judgement making. In query analysis, an LLM is prompted to generate the core problem or question of the query/question. In document analysis, an LLM is prompted to analyze every top-k retrieved document with the analysis of the query, and the definition of relevance. In final judgement making, an LLM is prompted to generate a Yes/No judgement given the query and document analysis, and the definition of relevance. Experiments show that JudgeRank works well in both reasoning-extensive and ordinary retrieval tasks.
weaknesses: - The authors argue that “In contrast, the original BRIGHT paper shows that GPT-4 with listwise reranking improves on top of vanilla BM25 baseline by an average of 2.7 points on nDCG@10, a much smaller improvement than our approach despite using a much stronger LLM”. However, the experiments from the BRIGHT paper is conducted on vanilla BM25, without the use of CoT reasoning on the query, whereas JudgeRank in this paper reranks top documents from BM25 with query-side CoT reasoning. So the comparison here is not fair - “GPT-4 with listwise reranking” should be run in the same setting.
- The effectiveness of this three-step framework is not clear. What is the result of only removing the query or document analysis step? Do both analysis steps help? And, what about putting the document analysis and judgment into a single prompt, simplifying the process into two steps? An in-depth analysis of the design of JudgeRank is needed.
- See questions.

Meta-review: This paper introduces JudgeRank, a novel LLM-based method for pointwise passage reranking. It employs a three-step process: query decomposition to identify core issues, query-focused document summarization, and combining these outputs for relevance scoring. Tested on BRIGHT and BEIR benchmarks, JudgeRank outperforms vanilla retrieval and fine-tuned LLM baselines.

Strength:
- The paper is very well-written and easy to follow.
- The authors presents an intuitive technique leveraging instruction-tuned LLMs for passage reranking of top-k retrieved documents.

Weakness: 
- The exploration of other model families and ensembling is framed as future work in the revised conclusion. While this acknowledges the limitation, the lack of immediate experimental results diminishes the effectiveness and reliability of the proposed approach.

- Reviewer 27LA raised the score from 3 to 5 but highlighted that several concerns remain unaddressed:

 1. The absence of results for removing the query or document analysis step individually.
2. A lack of evaluation for combining document analysis and judgment into a single prompt.
 3. No detailed analysis of the impact of "{definition of relevance}" on reranking performance.

Decision: Reject