Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Authors: Yilun Zhou
Austin Xu
PeiFeng Wang
Caiming Xiong
Shafiq Joty
Venue: ICML 2025 poster (2025)
Keywords: LLM-as-judge
test-time scaling
benchmark
Primary area: deep_learning->large_language_models
TL;DR: We propose a novel benchmark, JETTS, to evaluate LLM-as-judges from the perspective of their helpfulness as evaluators for generator's test-time scaling.

Abstract: Scaling test-time computation, or affording a generator large language model (LLM) extra compute during inference, typically employs the help of external non-generative evaluators (i.e., reward models). Concurrently, LLM-judges, models trained to generate evaluations and critiques (explanations) in natural language, are becoming increasingly popular in automatic evaluation. Despite judge empirical successes, their effectiveness as evaluators in test-time scaling settings is largely unknown. In this paper, we introduce the Judge Evaluation for Test-Time Scaling (JETTS) benchmark, which evaluates judge performance in three domains (math reasoning, code generation, and instruction following) under three task settings: response reranking, step-level beam search, and critique-based response refinement. We evaluate 10 different judge models (7B-70B parameters) for 8 different base generator models (6.7B-72B parameters). Our benchmark shows that while judges are competitive with outcome reward models in reranking, they are consistently worse than process reward models in beam search procedures. Furthermore, though unique to LLM-judges, their natural language critiques are currently ineffective in guiding the generator towards better responses.

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Yilun Zhou Austin Xu
Peifeng Wang'
Caiming Xiong'
Shafiq Joty
Abstract
Repporas
Stp-uue Search
Scaling test-time computation, affording gen- erator large language model (LLM) extra compute during inference, typically employs the help of cx- ternal non-generative evaluators (i.e., reward mod- els). Concurrently, LLM- judges,, models trained to generate evaluations and critiques (explana- tions) in natural language, becoming increas- ingly popular in automatic evaluation. Despite judge empirical successes, their effectiveness as evaluators in test-time scaling settings is largely unknown. In this paper, we introduce the Judge Evaluation for Tist-Timeaoo Scaling (JETTS) bench- mark, which evaluates judge performance three domains (math reasoning, code generation, and instruction following) under three task settings: response reranking, step-level beam search, and critique- based response refinement. We evaluate 10 different judge models (7B-70B parameters) for different base generator models (6.7B-72B parameters). Our benchmark shows that while judges are competitive with outcome reward mod- els reranking. they consistently worse than process reward models in beam search procedures. Furthermore, though unique to LLM-judges, their natural language critiques are currently ineffective in guiding the generator towards better responses.
fettweet
Rel geseunggrraattoo
Aoda ressimgund
Judge
Reranking
Beam Search
Refinement
Prometheus-2 7B
-0.107
-0.100
0.976
SFR-Judge 8B
0.035
0.001
0.941
Skywork-Critic 8B
0.039
0.046
OffsetBias
-0.020
0.015
Themis 8B
-0.149
-0.021
0.996
SFR-Judge 12B
0.047
0,943
Prometheus-2 8x7B
-0.079
-0.083
0,966
SFR-Jodge 70B
0.171
0.138
Skywork-Critic 70B
0.177
Self Raight Eval.
0.095
Best RM
0.113
Best PRM
0.195
Random
-0.193
-0.139
Baseline Value
Figure I: Top: a graphical description of the three test-time com- pute tasks in JETTS used evaluate LLM-judges. Bottom: the leaderboard of judges the three benchmark tasks. along with outcome/process reward models and random choice. "." means that the model cannot be used for the task. Achieving a value less than the baseline (grayed out) means that the judge is worse than some less compute- inttoutte-ittiveeeeeee approach (c.g., greedy decoding, Sec. 3.5 for details).
Introduction
of additional data and prohibitive cost of model training and serving. Recently, test-time scaling is being considered as an alternative solution, in which more compute dedicated at the test time in the hope that better model responses are generated (Snell et al., 2024; Jaech et al., 2024).
For last several years, the rapid increase various capa- bilities of large language models (LLMs) is mostly attributed scaling the corresponding increase in model and train- ing data sizes (Kaplan et al., 2020; Hoffmann al., 2022). Nonetheless, this effort has gradually saturated due to lack
A common component in test-time compute the evaluator model, which gives quality signal to various generated responses. Scalar reward models (RMs) are predominantly used to rerank either complete responses in best-of-N se- lection or partial responses in step-wise generation (e.g., beam search). By contrast, generative LLM judges are often favored in model evaluation, e.g., AlpacaEval (Dubois et al.,
Eqqual contribution 'Salesforce AI Research. Correspon- dence Yilun Zhou syilun.zhou@e swwesforce.ccoocccccooooo comz, Austin Xu <austin xu @salesforce.com..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Evaluating Judges as Evaluators: The JETTS Benchmark of Ludeas-uudge as Test-Time Scaling Evaluators
2024b) and MT-BBecch (Zheng et al., 2023), since they can adapt custom evaluation criteria and can (often) produce natural language critiques (or explanations) of the response.
0.20
(Nomma 1e Haloruinass
0.15-
Prometteur-2
JETTS performmncc
0.10
Seyaare-Crtic
0.03
Ofteeille 00
Despite LLM-judges achieving competitive performance on reward model benchmarks (e.g.. RewardBench (Lambert et al., 2024)) and PPE (Frick et al., 2024)), they are less com- monly used in test-time scaling scenarios compared to scalar RMs (Ji et al., 2025; Ke et al., 2025). This gap perhaps sur- prising given the potential benefits of LLM-judges. Namely, judge models are often trained to engage in thain-of-though reasoning before giving a judgment, which could not only benefit for paasoning-intensiin domains such as math and code, but also serve as effective critiques for the model to re- fine its generation, such as in Reflexion (Shinn et al., 2024) or SCORE (Zhang et al., 2024b). Furthermore, LLM-judges could seamlessly integrate tool use. Last, LLM-judges are inherently "instructable", capable of giving different types of judgment like binary yes/no answers, numerical scores, fine-grained feedback, etc., simply with different prompts.
0.00
Jrge 128
Promatsuss-2 5x78
Stre 728
-0.3
Skyner & kc 708
-0.0
JETTS Reraniing
Softough Eval. 708
-0.15
JETTS Beam Search
-0.00
RewardBench performance (Pairvise Accuracy)
Figure 2: Judge performance on RewardBench vs. JETTS, where marker size signifies judge model size (7B-70B), Small judges perform comparably to large judges on RewardBench, but lag large judges on JETTS, which simulates test-time scaling settings.
model benchmark. While judge performance between the two benchmarks is generally correlated, JETTS reveals a difference in the "fundamental judging ability" between judges of different scales. For example, if deciding between Meroo-Crritic-ae Critic-8B 8B and 70B, the practical choice based on RewardBench is to use the 8B model, trading off 4% in ac- curacy for an order of magnitude fewer parameters. JETTS, which simulates test-time scaling scenarios, reveals that this choice is suboptimal for reranking: The 8B judge yields substantively lower improvements over the greedy response than the 70B judge. The benchmark construction approach likely contributes to the difficulty gap: RewardBench forms pairs of responses from different generators, meaning can arrive the right outcome via stylistic factors, like for- matting (Tan et al., 2024). JETTS, however, requires judges to compare responses sampled from the same generator, limiting stylistic factors that lead to correct outcomes.
Skywork-
In this paper, we propose the first systematic benchmark of LLM-judges for model's 'ss test-time scaling. This bench- mark consists of three tasks (Fig. 1): (1) response reranking, where the judge picks the best response out of several ones, (2) step-level beam search, where the judge guides the model to generate the response seep-by-teep, and (3) critique- based refinement, where the judge offers natural language critique to the response for the model to refine. We evaluate 10 different judge models, ranging in size from 7B to 70B pa- rameters on these three test-time scaling tasks. We cover 3 different domains (math reasoning, code generation, and instruction following) and generate responses from up to 8 unique LLMs, ranging in size from 6.7B to 72B parameters.
2 Background and Related Work
The JETTS setup enables us to analyze the effects of judge- guided test-time scaling: Can a "weak" (e.g., 8B) judge actually help "strong" (e.g., 72B) generator? How benefi- cial are strong judges to weak generators? In what domains are the current crop of judge models best suited for test-time scaling? How useful are judge critiques in practice? We find that weak judges can help strong generators in casier tasks, such as instruction following, but not in reasoning- intensive tasks like coding or math. Larger judges bring the most benefit for math and instruction following tasks, but no evaluated judges are able to reliably improve genera- tor performance for coding. Lastly, while natural language critiques are touted as defining advantage of judges over RMs, we find that such critiques have significant room for improvement in terms of utility. We present these results with many additional analyses in Sec. 4.
The concept of using more computation at test time to in- crease model performance is not new. tf-thuhh (CoT) reasoning (Nye et al., 2021; Kojima et al., 2022; Wei et al., 2022) can be considered as one of the earliest such techniques, where the model is encouraged to generate a reasoning chain before producing the final response. When the query asks for "single-target" response (e.g., number, True/False, or name), self-consistency (Wang et al., 2022) samples multiple CoT chains and returns the majority vote answer. Orthogonally, wre-of-thoughhe (Yao et al., 2024) and graph-of-thoughe (Besta et al., 2024) break down the chain into individual steps to search over.
While some studies (Madaan et al., 2024; Saunders et al., 2022) find that a model could improve or correct its own response, others highlight various issues and limitations in these approaches based on the fundamental paradox that if model knows where it gets wrong, why it does not simply output the correct response in the beginning (Huang et al., 2023; Stechly et al., 2023; Valmeekam et al., 2023; Shridhar et al., 2024). Thus, a new trend in test-time compute is the
Comparison with RewardBench. In Fig. 2, we present a preview of our results by comparing performance on JETTS Best-of-N Reranking and Beam Search tasks against Re- wardBench (Lambert et al., 2024), popular RM and judge
2

---

Evaluating, Judges as Evaluators: The JETTS Benchmark of LLM--asJuugess as Test-Time Scaling Evaluators
use of an evaluator help with model response, commonly known the reward model (RM) (Liu et al., 2025; Cobbe et al., 2021; Ouyang et al., 2022). Beyond RMs that operate on complete responses, known as outcome RMs (ORMs), for reasoning tasks it is also possible to train process RMs (PRMs) to provide step-level rewards (Wang et al., 2023a; Lightman et al., 2023; Zhang et al., 2025b; Luo et al., 2024; Dai et al., 2024).
Task
Dataset
Size
Evaluation metric
GSM8K
1319
Accuracy via Math- Math-Verify
Math
MATH ()
1324
Accuracy Math Verify Math-Verify
Reasoning
CHAMP
270
Accuracy via GPT:40 grading
HumanEval+
164
Pass @
Code Generation
MBPP+
378
Pass @
BigCodcBench
1140
Pass @
AlpacaEval
805
Win rute vs. GPT-4 Turbo, with
Instruction
GPT-4 Turbo as judge
Following
IFEval
541
Prompt-leeel strict accuracy
Mostly parallel to this effort is the development of LLM- as-judges (Zheng et al., 2023), i.e., specializing LLMs into judges of model responses. While vanilla models are originally used (e.g.. GPT-4 in AlpacaEval (Dubois et al., 2024b)), finetuned models are often found to deliver supe- rior performance and be less susceptible to bias (Park et al., 2024). To further leverage the reasoning abilities of LLMs and improve explainability, recent judges are finetuned to generate reasoning or critiques for the response, often be- fore the final judgment. Since then, a long line of work has been proposed in developing specialized LLM-judge models, focusing both on data curation ( (Wang et al., 2024b; Kim et al., 2023; 2024; Wang et al., 2023b; Li et al., 2023; Vu al., 2024) and training methodologies (Wang et al., 2024a; Hu et al., 2024; Ye et al., 2024; Saad-Falcon al., 2024; Deshpande et al., 2024; Wang et al., 2024b).
Table 1: List of datasets and associated metrics in JETTS.
vestigates different types of verifiers for best-of-N reranking, considering only one evaluation protocol (binary correct- ness) for one prompted L.L.-judge baseline per domain.
3 The JETTS Benchmark
In this section, we present our proposed benchmark: Judge Evaluation for Test-Time Scaling (JETTS). As previewed in Sec. 1, JETTS consists of three tasks: response reranking, step-level beam search, and critique-based refinement.
3.1 Task and Dataset Selection
Judge evaluation has typically focused on judging instruc- fion following proxy for chat quality (Zeng et al., 2023). However, most recent efforts in test-time scaling have fo- cused on math and code (e.g., Snell et al. (2024); Brown et al. (2024)). To cover both typical judge and test-time scaling domains, we consider three types of tasks: math reasoning, code generation and instruction following; we list the datasets in Tab. In order to minimize the effect of randomness, we pre-compute the model responses and release them as part of the benchmark wherever possible.
As RMs and LLM-judges are both evaluators in nature, they can often be directly compared, most notably on the Reward- Bench leaderboard (Lambert et al., 2024), showing close performance among the best RMs and LLM-judges. Due to saturating performance on RewardBench, with top models achieving over 95% accuracy, new benchmarks have been proposed to evaluate the efficacy of evaluators (Feuer et al., 2024; Liu et al., 2024b). Notably, ProcessBench (Zheng et al., 2024) was proposed to evaluate how process reward models (PRMs) perform in identifying step-level mistakes in math reasoning. PPE (Frick et al., 2024) and RMB (Zhou et al., 2024) both evaluate RM and LLM-judge efficacy in pairwise preference and Best-of-N settings for alignment, focusing on chat quality and safety. JudgeBench (Tan et al., 2024) identifies shortcomings with RewardBench's 's reason- ing samples, and proposes a more difficult pairwise evalua- tion set for reasoning tasks. Complementing outcome- outsome-based judge benchmarks, Lan et al. (2024); Lin et al. (2024) eval- uate the quality of critiques for single-round refinement.
For math, we use GSMSk (Cobbe et al., 2021). MATH (Hendrycks et al., 2021) and CHAMP (Mao et al., 2024). For code, we use HumanEval+ (Liu et al., 2023a), MBPP+ (Liu et al., 2023a) and BigCodeBench (Zhuo et al., 2024). HumanEval+ and MBPP+ are further refined versions of HvmanEval (Chen et al., 2021) and MBPP (Austin et al., 2021), respectively. For instruction following, we use Al- pacaEval (Dubois et al., 2024b) and IFEval (Zhou et al., 2023), with the former being LLM--eallated and latter be- ing algorithmically evaluated. App. A contains additional details about evaluation protocols.
JETTS, unlike recent judge benchmarks, does not assess judges using fixed pairwise test sets. Instead, it simulates three distinct test-time scaling scenarios (reranking, beam search, and multi-round refinement) and evaluates the qual- ity of LLM judges by measuring the improvement they bring to the generator model. Snell et al. (2024) orthogonally in- vestigate the efficacy of different test-time-compute scaling approaches (Best-of-N reranking and step-by-step beam search) with a fixed RM. Similarly, Zhang et al. (2024a) in-
3.2 Task 1: Response Reranking
The most straightforward and used inference-time compute technique is response reranking, where the generator model samples multiple responses with stochastic decoding and the judge reranks them to select the best one. In our benchmark, recognizing that the greedy decoding often produces higher quality responses, for each input query, we generate the greedy response and 9 sampled responses with a temperature of 1.0 and top-p with p = 0.95, for total of 10 responses.
3

---

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
We ask LLM-judge to rerank and select the top response for evaluation. Since no current judge supports one-shot direct reranking of multiple responses, we consider two protocols: pairwise round-robin and single-instane instance rating. In the for- mer, for the N responses, we construct the N(N - 1)/2 response pairs, query the judge on each pair, assign the win- ner a score of and the loser a score of O, and select the response with the highest score (random tie-breaking if nec- essary)'. In the latter, we ask the judge to generate score (an integer between 1 and 5), and the select the response with the highest score. Since ties are much more common in the single-instance rating setting, we keep all tied responses and report min, average and max performance; e.g., if for a test sample, a judge rates 3 of 10 responses as the highest score, and those 3 responses have accuracy labels (1, 0, 1), then we use accuracy values (0, 2/3, 1) for this sample when aggregating (min, average, max) performance.
In a (N, -)beaam search with beam width of M, the gener- ator samples N first steps, which are reranked by the judge to keep the top-N/ /MM Then, for each chosen step, the gen- erator samples M next steps, for a total of N responses to rerank again. When a generation ends (due to the EOS token or length limit), its current step is not expanded, resulting in less than N responses. This process repeats until all selected steps are finished, at which time a final reranking selects the top response. Similar to before, in each sampling, one response is greedily decoded and the rest are sampled with temperature of 1.0 and top-p of 0.95. We use (10, 2) 2)-beam search in the benchmark and limit the tree depth to 10.
Notably, the beam search requires LLM-judges to reason about partial responses, use case not specifically designed for any of the current judge models. Nonetheless, since they are often trained with flexible judging criteria, we append the following to the existing ones in this task:
Since the base performance is very different across datasets and generators, we define and report the normalized help- fulness for the judge as
Important note: the model response is generated with streaming, meaning that may not be finished yet but we still want to get sense of the partial progress so far. If this is the case, you should provide judgment and feedback on the currently generated part, focusing on its correctness and promise of leading to correct final solution.
= Pjudge Pgreedy
(1)
Poracle Pgreedy
where Pgreedy is the average performance of the greedily decoded responses across the dataset, Poracle is the oracle judge performance over all response selections (e.g.. se- lecting the correct response for math problem whenever there is one), and Pjudge is the average performance of top responses reranked by the judge. A negative value indicates strict harm as h = 0 can be trivially achieved by selecting the greedy response. We also include random baseline (i.e., the expected helpfulness of a judge that randomly reranks the responses) to compare each judge against "average" reranker. For GSM8k and MATH, we also compute the performance of the majority answer for each problem².
Finally, we consider a lookahead variant of beam search similar to Snell et al. (2024), where instead of sending par- tial responses to the judge, we complete them with greedy decoding finish and send these "lookahead" versions (without adding the instruction above). To contextualize judge performances, we again compute the greedy, random and oracle performance and derive normalized helpfulness.
3.4 Task 3: Critique-Based Refinement
A unique feature of LLM-judges is their ability gener- ate natural language critiques, or explanations, for their judgments. The critique- based refinement task evaluates whether these critiques are helpful for the model to improve their response. Specifically, we define a (NV, --rremmmmmnnt as the setup where we start with N seed responses, and each response is iteratively refined M times, for total of N N(M + 1) responses, which is reranked at the end to se- lected the top response. Compared to reranking and beam search, in which the judge reweighs the output distribution, in refinement the judge directly modifies it with its critique.
3.3 Task 2: Step-Level Beam Search
Using an evaluator to control the flow of generation is in- creasingly popular. For example, when generating the re- sponse line-by-line to coding problem, an evaluator could decide that newly generated line is bad and hence asks the generator to resample new line. In this task, we generalize this procedure to beam search. Since this procedure requires the notion of steps in a response, we consider only math and code tasks, and define each step as a line of response from our observation that the generated math solutions often put each step on its own line and the fact that each line of code naturally represents programmatic operation.
Since most judges are finetuned on fixed judging templates, they may not be able to directly revise responses. We instead provide the generator model the judge rating and critique, and ask it to revise its response using the feedback with the prompt shown in Fig. 18 of App. A.3. In our benchmark, we consider (1, --rreiinemen,, which yields 10 responses in total to make it directly comparable to the reranking task.
'To account for positional bias, we employ additional consis- tency checks which may lead to ties; See App. B.I for details.
Majority-vote aggregation is not feasible for CHAMP since GPT-4o directly grades natural language responses against the ground truth without extracting the "final answer".
While reranking and beam search have oracle performance as a natural upper-bound to define normalized helpfulness.
4

---

Evaluating Judges as Evaluators: The JETTS Benchmark of LLMM-ssJuuges as Test-Time Scaling Evaluators
refinement has no such oracle. Therefore, we use improve-
Generator Model
Params
Generator Model
#Params
ment ratio over reranking and greedy responses as measure of judge efficacy, defined as
LJama-3. Innttruct
8, 70B
Qwen-2.5 2.5 2.5-Instruct
32, 72B
Deepseck Crepseek-Math--le Math-Instruct
7B
Qwen-2.. Laven-2.5-Matt M-thh
7B
Deepseck Coder-
6.7B
Qwen- 2.5-oeer
7B
(Ball) 5(RR) = P'judge/ Pjudge
(Rel) ((G) = Pjudge/ Pgready
(2)
Juige Model
Albrr
Params
Parrise
Supported Capability
Single
Catique
SRR-Judge
(Wang al., 20244)
SFR
708l
where, for (N, MM -refinmmmnn Pjudge is the average per- formance of the refined-and- -nde--ran-ereddd response across the dataset, and ( that of the reranked response among [(M+ +1) sampled ones (including one greedy) and Pgreedy is that of the greedy response. In our (1, --reinnnemnnt setup, the latter two quantities are directly taken from the results in Task of reranking on 10 responses.
Skywork Chitic
(Shiwen et al., 2024)
SC
8.70B
x
OfisecBias
(Park al., 2024)
OB
RB
Promueteau
(Kim
Prom
7,3478
Self Taught Eval.
(Wang al.. 2024b)
STE
70B
Themis
(Hu et al., 2024)
Thua
8B
Table 2: Top: the generator models to produce the model responses. The models on the first row are used in all tasks, and those on the second and third rows are used only in math and code tasks respectively. Bottom: The judge models used in our experiments.
Qwen math (Yang al., 2024) and coder (Yang al., 2024), and older models, DeepSeck math (Shao et al., 2024) and coder (Guo et al., 2024), to analyze how judge-based test time scaling improves models of different strengths.
In order for refinement to be a meaningful test-time scaling procedure, we need it to outperform both reranking and greedy. So we define the effective improvement ratio as
((ER)) min ( 8(0))
Tab. 2 (bottom) lists the LLM-judge models that we bench- mark. We additionally include several baseline models:
3.5 The JETTS Leaderboard
the vanilla Llama-3.1 3.1ma-3.11-1 3.B-BB-BBB... 1--8--BInstinn Instruct model with SFR-Judge prompts for response reranking;
Finally, we compute a leaderboard (Fig. 1) for three tasks.
three ORMs for response reranking: Llama-3- -affsa--OffsetBiass- RM-8B (OBRM). Skywork-Reward- -ewarrd-Llam-3.111 --.---.-...-B-..... and
We use normalized helpfulness h (Eq. 1) to measure judge
performance on reranking and refinement, and effective in-
(SRM); and
provement ratio (EE)) (Eq. 3) on refinement. Each judge's 's score is computed by averaging the task-specific metric over all datasets and generators. If a judge supports multiple protocols (e.g., both single rating and pairwise comparisons in reranking), we report the maximum aggregate perfor- mance among all protocols. For convenience, we include per-dataset and per-generator metrics for each of the three tasks for all evaluated judges in App. C.
PRM Qwen2.5--Mth-PPRM. (QPRM) for beam search.
App. A.2 includes more details on judges and RMs.
For all results presented below, unless otherwise noted, we perform statistical analyses of reported quantity differing from the baseline value (0 for reranking and beam-search, and for refinement) using two-sided one-sample /-test and indicate the significance as "*" for p S 0.05, "* *" for p S 0.01, "**** for S 0.001, and "n.s." or no marker for not significant (p > 0.05).
Practitioner note. Beam search evaluation requires signifi- cantly more compute than reranking evaluation. While Task 3 reveals interesting findings, e.g., judges generalize to some degree to assessing partial responses, we find reranking and beam search performance to be correlated; see Fig. 2 and analysis in Sec. 4.3, Our recommendation for practitioners is to use reranking performance as a proxy for beam search performance when evaluating future judge models, unless working specifically on process-based judging.
4.2 Response Reranking
As discussed in Sec. 3.2, since no judge supports direct reranking of N responses, we use either pairwise round- robin comparison or single-instance rating and evaluate each judge on the protocol(s) it supports. Furthermore, we experi- ment with to judging prompts for single rating: Likert scale, where judges are not given fine-graing criteria and simply asked to rate on a 1-5 scale, and additive criteria (Yuan et al., 2024), which instructs the model incrementally increase score based based on given criteria, with maximum score of 5. The full results are presented in Tab. 3-5 in App. B. Below, we highlight notable findings.
Experimental Results
4.1 Experiment Setup
As introduced in Sec. 3.1, JETTS consists of eight datasets from three task categories: math reasoning, code genera- tion and instruction following. Models listed in Tab. 2 (top) generate the responses. We use Llama 3.1 (Dubey et al., 2024) and Qwen 2.5 (Yang et al., 2024) models in all tasks. For math and code tasks, we also consider four domain- specific models and intentionally choose newer models,
Reranking helpfulness varies across protocols and datasets. Fig. 3 plots normalized helpfulness per dataset averaged across judges and generators, along with the re- spective random performance. There are two notable trends.
5

---

Evaluating Judges as Evaluators: The JETTS Benchmark of LLMMassJudges as Test-Time Scaling Evaluators
1.0
Nomalzzd hepruines
Nomaltzed Helptuiness
0.3
0.2
0.0
0.0
0.2
-0.4
Parroise
Single Likert
Single Additive
Random
-1.5
GSMBK
MATH
CHAspe
HumantEaal
MBP
BiyCadsBerch
Alpacafival
iPEval
10"
10'
10'
Judge Size /ennerator
Math coet: 0.16 (eer)
GSMAK
M6T
CHAMP BigCodeBanch
Code coef: 9.00 (n.s.)
HumanEval-
MBPP
Figure 3: Normalized helpfulness of judge reranking per dataset. Asterisks on the bars denote statistical significance described at the end of Sec. 1.
(n.s)
AvpacaEval
IFEval
Figure 5: Normalized helpfulness under the pairwise protocol for different judge-to-generaton size ratios, along with the best-fit lines for each task log scale.
Normalize nopruness
0.2
0.1
ferent judges. Fig. 4 plots the normalized helpfulness of each judge on each task category for the pairwise proto- col, aggregating over respective datasets and all generator models. With few exceptions, all judges, including the vanilla Llama-3.1-88 Instruct with judge prompt and the reward models, demonstrate highest helpfulness for instruc- tion following, less but mostly positive helpfulness for math reasoning. and mostly negative helpfulness for code genera- tion. For single-rating protocols, consistent with Fig. 3, the helpfulness is much lower, but the relative ranking still generally the same, as shown in Fig. 22 of App. B.1.
-0.1
-0.2
-0.3
Math Reasoring
Code Generation
Instruction Follwing
Prom
SFR
SC
080 88
SRR 128
From 8078-
STR 708
SC 108
STE 708
Lamm 88
Random-
Figure 4: Normalized helpfulness of the pairwise protocol across task categories of each judge, compared to Llama-3.1 8B with judge prompt, best reward model and random reranking, averaged across generator models and datasets. A violin plot depicting finer- grained distributions is in Fig. 21 of App. B.I.
The consistency of inter-task helpfulness among judges co- incides with the instruction-following training (and prompt- ing) emphasis of current LLM-judges (Park et al., 2024; Tan et al., 2024). Despite training data including both math and code samples (c.g.. Vu et al. (2024); Shiwen et al. (2024)). judge-speeific finetuming seems to primarily boost instruction-following evaluation abilities, sometimes the cost of other capabilities. This effect is shown in Fig. 4, where the vanilla Llama 3.1 8B model, prompted as judge, exhibits a baseline level of competency as judge, per- forming roughly at the greedy response level. Comparably sized 7B/8B judge models outperform it significantly on and moderately on math, but perfor- mance degrades, sometimes significantly, on coding. More- over, task-specific judge prompting does not appear to miti- gate this task gap, as we show in Fig. 23 in App. B.I.
First, the pairwise protocol typically performs better than the single-rating protocol with either Likert or additive rubric, with the only exception of MBPP+, with all protocols worse than the greedy baseline. This is consistent with past find- ings, where both humans (Shah et al., 2016) and LLM- judges (Liu et al., 2023b) are better at providing pairwise comparison responses than single-rating responses. Given this finding, for the next two benchmarking tasks, step-level beam search and critique-based refinement, we use the pair- wise protocol in reranking whenever possible. However, this comes significant inference cost, as the judge inference for pairwise reranking takes O(N2) time for N responses to rerank, compared to O(N) for single-instance rating.
Second, there are also significant variations across datasets, even within the same task category: pairwise protocols help on GSM8k and MATH but not CHAMP, and all protocols help on AlpacaFval (for both win rate and length-controlled win, as we show in App. B.1) but none on IFEval. Overall, the dataset, and fur- ther fine-grained analyses to understand the strengths and weaknesses of judgment would be valuable?.
Larger to-geerto- size ratio under the pairwise protocol increases helpfulness for math and instruction following, but not code. For each pair of judge and gen- erator, Fig. plots the ratio of the number of parameters in the judge to that in the generator (on log-scale) vs. the helpfulness achieved by the judge, color-coded by task cat- egory. A linear regression is plotted for each task on the log-scale size ratio. For math reasoning, a larger size ratio statistically significantly increases helpfulness, indicating the promise of larger judges for small generators, (which, nonetheless, may not be practical). At the lower end of
judge helpfulness highly depends
Inter-task helpfulness variation is consistent across dif-
3 "Alhhough different sets of judges are used to compute the average for pairwise vs. single-instance protocols due to different supported capabilities, Fig. 20 of App. B.I shows that these trends largely preserve within judges that support both capabilities.
6

---

Evaluating Judges as Evaluators: The JETTS Benchmark of LLdeas-Judges as Test-Time Scaling Evaluators
size ratio around 0.1 (e.g., 8B judge for 70B generator), normalized helpfulness is negative average, indicating the absence of "weak-to-strong" guidance ability for LLM- judges. For instruction following, all size ratios achieve positive helpfulness but the effect of ratio increase is less notable. For code generation, the size ratio does not have an effect on the helpfulness, which is negative at all size ratios. For single-ratings protocol with Likert and additive rubrics (Fig. 24 of App. B. the effect of the size ratio is mildly positive only on math datasets, at 0.07 and 0.03 respectively, and negative on both of the other two tasks.
Figure 7: Normalized helpfulness ranges for single-rating proto- cols, showing minimum and maximum performance achievable from tied top-rated responses, along with average performance.
mum and maximum achievable performances from the tied responses. If one were to employ random tie-breaking, per- formance can swing from extremely poor to near oracle.
No judge can reliably outperform -aoo on math datasets. The output format specification of GSM8k and MATH allows us to easily extract the final answer from
4.3 Step-Level Beam Search
the generated response, and compute the majority-vote (i.e., self- self-consistency (Wang et al., 2022)) performance. In Fig. 6, we plot the difference in the normalized helpfulness between each judge and this heuristic and mark all positive cells with purple stars. On GSM8k, only SFR 70B and SC 70B offer substantive improvement over majority- relatively weak generator models. For MATH, only SFR 70B is able to meaningfully beat majority-vote on small- to-medium generators only. Thus, in theory, an LLM that simply extracts the majority-vote answer could serve very strong reranker for these datasets.
As mentioned in Sec. 3.3, we focus on math and code tasks since their solutions can naturally be decomposed down into steps. We consider the (10, 2)-beam search, where the judge needs rerank 10 candidates at each step and select the top-5, which get expanded to 10 next-step candidates. We use the pairwise protocol for the all rerankings, except for Themis 8B, a single instance rating judge, where we use the additive prompt. The full results are presented in Tab. 9 of App. B.2, and we highlight notable findings below.
but only for
The effects of uadde--to-generato size ratio are similar to that of reranking. Fig. 8 shows the judge helpfulness the ratio of the number of parameters in the judge to that in the generator (on log-scale), along with linear re- gression for each task. The effects of judge-to-generatee are remarkably similar to that in reranking: for math rea- soning, a larger size ratio results in statistically significant helpfulness gains, with generally negative helpfulness for smaller size ratios. Like reranking, small judges are unable to provide "weak-to-strone ators, revealing the need for large judges for beam search. This requirement may be impractical due to the significant computational overhead required to judge outputs at each "step". For code, scaling judge size relative to the generator produces no statistically significant gains: for all size ratios, the helpfulness is negative.
Judges are over-lenient under the single rating proto- col, resulting in a large number of top-ranked responses. Despite more time-efficient, single-rating evaluation pro- tocol performance results in evaluation that is too lenient. Judges often rate significant fraction of the N responses top score (i.e., a 4 or 5), with the most lenient models consistently rating all 10 responses equally high (Fig. 25 of App. 1) Multiple responses tied as the "best" response leads to large ranges of possible reranking performance. Fig. 7 shows the average performance, as well as the mini-
improvements for large gener-
0.70
Llama uo
Liaa 708
0.35
COWN
Owen
728-
Owen M 7B
0.35
LLM-judges lag significantly behind the small QPRM- Fig. 9 compares the normalized helpfulness of non- lookahead vs. lookahead beam search for different judges, along with reranking, random and QPRM performance. For math reasoning, while some judges can achieve positive helpfulness, they struggle to outperform reranking even with much more judge inference. More notably, by far the best result is achieved by the 7B parameter QPRM- For code generation, only large judges achieve positive helpful- ness, with only SC 70B providing statistically significant improvement. QPRM. although finetuned on math data, still achieves decent helpfulness at much smaller scale. These results suggest that even though some judges are finetuned
Opsk-M 78
-070
0.50
Uiema
706
0.25
Owen 328
0.50
Owen 728
Dpak-M 70
0.50
SPR sp
SC se
08 ta
120
SPR you
scc yout
sre rad
Luma xd
Best eut Ghaddy
Prom
Prom
Figure 6: Normalized helpfulness difference between each judge and majority-vote for each generator on GSM8k and MATH. Posi-
tive value cells are marked with purple stars.
7

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLdeas-Judges as Test-Time Scaling Evaluators
0.4
Non Lickhhed Imaasted
3.4
Normalzzon Helphunwes
Nemm Hinprueee
Loona Num. (No Ioottta
02
0.0
0.2
0.0
-02
-0.4
-244
-04
10
10"
-2.
Retarinng North, phuiissss
-10
-0.5
0.0
9.5
-0.5
Nin www.olokodddddooooooo Nomn. Holpfliees
-0.6
-0.2
9.0
022
0.4
Adde Size Generator Size
G242t
MITH
CAVM Bglaaeeendh
0.00 (-*1)
GBMBK
CHANP BigCodeBench
Hkrrataal-
Code coet 0.01 In:5))
HumanEvaa++
NEFP+
het-lookahead
Lonbwheed
Figure 10: Relationship of judge's reranking vs. beam scarch performance, and non-lookahead vs. lookahead performance.
Figure 8: Normalized helpfulness of beam search under for differ- ent judge-to-gennne to-generatorr size ratios, along with the best-fit lines for each task in log scale.
4,4 Critique-Based Refinement
Since this setup requires the judge to generate critique on its judgment of an individual sample, we only evalu- ate judges with both single- ingtanin rating and critique- generation capabilitiies in Tab. 2: SFR-Judge, Prometheus and Themis. In addition, when reranking the final candidate responses, as the reranking performance is much better with pairwise round-robin than single-instance rating (Sec. 4.2), we use the former whenver the judge supports it (i.e., for SFR-Judge and Prometheus) and fall back to the latter oth- erwise (i.e., for Themis). The full results are presented in Tab. 10 App. B.3, with notable findings discussed below.
to work with flcxible rubrics, process judgment is likely too out-of-distribution for small judges to be effective.
The benefit of lookahead is inconclusive. the lookahead mode, rather than incomplete responses, judges evaluate full, rolled out responses, more aligned with judge training distributions. However, somewhat surprisingly, lookahead is not universally beneficial. For math reasoning, Fig. 9 shows that it benefits seven judges and QPRM. sometimes by two to three times, while slightly hurting the other three judges. For code generation, lookahead benefits five judges while hurting the other five and QPRM, with much milder effects. Overall, judges that perform well on reranking full responses tend to benefit from lookahead, whereas weaker ones tend to benefit from non-lookahead. This suggests that splitting full responses into shorter chunks makes evaluation easier for them, despite the -ittr--distribttion nature of step-level evaluation.
No judge yields beneficial performance on refinement across task category. For individual judges. Fig. 11 plots the average effective improvement ratio g(Ef)) for each judge, with no judge achieving ratio better than 1.0. In- struction following experiences the largest loss, with up to 10% performance degradation. Thus, we conclude that all are incapable at this task.
There is a high correlation among non-lookahead, looka- head and reranking performance. Fig. 10 plots the cor- relation between each judge's 's reranking performance and beam search performance (left), and the correlation between its lookahead and non-lookahead performance (right), with each scatter point representing a pair of dataset and genera- tor. We observe high correlation among the three quantities, suggesting that they are all likely to be tied under the same underlying "fundamental judging capability" of the judge.
We now focus our analysis on (1) quantifying the utility of refinement relative to greedy decoding and reranking and (2) precisely quantifying the behavior of the refine-then- rerank process. To do so, we define the following metrics, Towards the quantifying refinement utility, denote (Bee) and porae. to be the performance of a randomly selected re- sponse and the best performing response from all responses produced in the refinement process across the dataset, and (RR)
analogously Prand. and (RR)
on responses in reranking.
We define 5(RR) = (Ref) /p /RRR)) ((RR) = porac. / (RR)
((G) deeper insights into the refinement process, examine the
Prand. Soannd/Prreen /Pgreddy. g(G) Porae. /Pgreedy- To gain
(Reef)
(Ref)
-32
performance using the last refined response without rerank-
2.00
3.56
Figure 9: Normalized helpfulness of beam scarch by each judge and 2iwen225-MMtthPPP PRM 7B. The judge's reranking performance on the task category plotted for comparison, as well as the random beam search baseline.
Teasoring
Code Gemerattoo
Instruction To lookiigg
Prom 78
SFA 88
Thrn 8B
SFR 128
Prom 8x78
SFA 768
Figure 11: Effective improvement ratio (RHF) for each judge and task category.
8

---

Evaluating Judges as Evaluators: The JETTS Benchmark of LLL--ssJJudger as Test- Time Scaling Evaluators
166-
Math Rewsoning
Code Generation
Instruction Folnwing
0.50
1000
The 45
Prom 8076
S16 205
N 85
SFN 22
5PH 55
STB 578
3.0
1.5
Nambe of Salurrers
Choscs Trpposs Irdex
Figure 14; The distribution of number of refinement rounds con- ducted by the Llama 3.1 8B generator, and the index of the chosen response by the final reranking.
Figure 12: Various diagnostic metries refinement.
ing. We denote p to be the performance of the last response from the refinement process across the dataset and define (RR) B (BBB) and |/(c) = Plast
at giving actionable advice to encourage refinement behav- ior, which unfortunately does not translate to meaningful benefits in the final result (Fig. 11).
(Ref)) Pgreedy to be analogous to (RR) and g(G) defined in Eq. 2. Fig. 26 in App. B.3 illustrates these newly defined metrics.
The final selected response is often generated early on. For Llama 8B, Fig. 14 plots the distribution of number of refinements (excluding the greedy seed response) and distribution of the chosen response index (0 for greedy), Llama 8B tends to either refine very minimally or use all nine rounds. However, the response actually chosen by the final reranking tends to be generated much earlier in the process, with the seed response (index 0) still most likely to be chosen, revealing limited utility of the critiques.
The refinement process improves upon either reranking or greedy decoding, but never both. In Fig. 12 (top), we see that (RR) and g(G) are never meaningfully greater than 1.0 simultaneously on any dataset across all judges. As such, refinement is typically sub-optimal choice of test-time scaling method, with greedy decoding reranking leading to better performance.
Critiques tend to over-focus stylistic features than response quality. We conduct qualitative study on the critiques in App. B.3. We inspect over 100 refinements and find two major issues. First, the judge gives high scores to responses with actual mistakes for stylistic reasons such as it being easy to follow (Fig. 27), thus failing to lead the generator to correct the error. Second, the judge is overly critical about minor stylistic issues on fully correct responses (Fig. 28) so that the generator engages in mean- ingless refinements that may even flip correct answers.
The benefit of final reranking is small. While Fig. 12 (top) shows that ò consistently better than ō the difference is small. Given the O(N)) complexity of pairwise reranking, just using the last refinement may be practically favorable.
Generators and judges rarely engage in extensive refine- ment processes. A notable trend from Fig. 12 (bottom) is (/C)) 2 and dorac. < for all datasets, suggesting that the refined responses are very similar to the original greedy seed and fail to discover much better solutions. It turns out, in fact, that the refinement process is rarely extensive. In Fig. 13, we plot the total number of generated refinements (cxcluding the original greedy response) across the three task categories. Among the four generator models, Llama 8B revised for significantly more rounds than other gen- erators. This may suggest intrinsic pliability taking feedback, and/or result from its initial greedy response being much worse. Among judges, Prometheus 7B is much better
5 Conclusion and Future Work
In this paper, we seek to understand the feasibility of us- ing LLM-judges in place of typically used RMs in test- time compute procedures. To this end, we propose JETTS, the first such systematic benchmark with three tasks: re- sponse reranking. step-level beam search and critique-based refinement. Our benchmarking results reveal several lim- itations and corresponding future research directions for LLM-judges. First, the reranking task faces a performance- efficiency dilemma: the pairwise protocol shows promise in certain domains but runs in O(N2), while the O(N) single-instane protocol performs much worse. Improv- ing single-rating ability of judges is essential for compute- efficient test-time scaling. Further, current chain-of-tthuggee reasoning generated by the judges is insufficient, both for themselves to arrive at better judgments, especially in math and code domains, and for generators as critiques to refine responses. As such, a key next-step for judge development is to produce better and more useful reasoning.
Reasoning
Cade Geneeation
from 18
5:4
2,26
221
1.05
3.4
1.47
2.63
0.22
4.33
2.27
1.30
1.2
041
9.35
3.08
246
124
ame
the 18
5.7
632
0.04
3.53
1.74
3,03
5.55
0.02
2.56
039
0:31
032
5.72
033
3.28
8.50
1.21
198
2.03
150
0.5
SFR
from hor3
4.95
0.43 0.8 2.22
1.78
0.5 0:22 0.12
2.43
027 0.28
0.2
5.43
6,66
0.18 9,37
4.29
352
8.53
0.54
3.05
058
0.42
020
Uame 36
Lamo 708
Ooven 220
Liona 758
Gomm 738
Limno St
228
Goe 73
Com
Lamii
Figure 13: The average number of refinement rounds carried out by each (judge, generator) pair.

---

Evaluating Judges Evaluators:
The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Impact Statement
Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P. S., and Hashimoto, T. B. Al- pacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024b.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be
specifically highlighted here.
Feuer, B., Goldblum, M., Datta, T., Nambiar, S., Besaleli, R., Style out-
References
Dooley, S., Cembalest, M., and Dickerson, weighs substance: Failure modes of Ilm judges in align- ment benchmarking. arXiv preprint arXiv:2409.15268 2024.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q. al. Program synthesis with large language models. arXiv preprint arXiv:2 108.0220877772 2021.
Frick, E., Li, T., Chen, C., Chiang, W.-L., Angelopoulos, A. N., Jiao, J., Zhu, B., Gonzalez, J. E., and Stoica, I. How evaluate reward models for rlhf, arXiv preprint arXiv: 22..2440..77771 2024.
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Pod- stawski, M., Gianinazzi, L., Gajda, J.. Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial In- telligence, 2024.
Guo, D., Zhu, Q. Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseck-codes When the large language model meets programming- the rise of code intelligence. arXiv preprint wrXh:22411141166 2024.
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, arXiv preprint 2rXiv:2407.21787 2024.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv 2021.
preprint arXiv:2
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. O., Kaplan, Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained code. arXiv preprint arXiv: (00.:22077.03372 2021.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 1555552003.1566, 2022.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14166, 2021.
Hu, X., Lin, L., Gao, M., Yin, X., and Wan, X. Themis: reference-free nlg evaluation language model with flexibility and interpretability. arXiv preprint arXiv:2406 18365, 2024.
Dai, N., Wu, Z., Zheng, R., Wei, Z., Shi, W., Jin, X., Liu, G., Dun, C., Huang, L., and Yan, L. Process supervision- guided policy optimization for code generation. arXiv preprint ar vxXiv:2410.17622 2024.
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language mod- els cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01996 2023.
Deshpande, D., Ravi, S. S., CH- CanWang, S., Mielczarek, B., Kannappan, A., and Qian, R. Glider: Grading Ilm inter- actions and decisions using explainable ranking. arXiv preprint arXiv:2412.14144 141440 2024.
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai ol system card. arXiv preprint 24Xiv:2412.16722 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Ji, Y., Li, J., Ye, H., Wu, K., Xu, J., Mo, L., and Zhang, M. Test-time computing: from system-1 thinking system-2 thinking. arXiv preprint arXiv:2501.02497 2025.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475 2024a.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001 08361, 2020.
10

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLMM-ssJudgess as Test-Time Scaling Evaluators
Ke, Z., Jiao, E., Ming, Y., Nguyen, X.-P., Xu, A., Long, D. X., Li, M., Qin, C., Wang, P., Savarese, S., et al. A survey of frontiers in IIm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint arXiv:2504.09037, 2025.
of tricks for reward modeling in llms. arXiv preprint arXi::2410.18551 2024a.
Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty- seventh Conference on Neural Information Processing Systems, 2023a. URL https::///// / "oormmidd=quxx1100 pesttss:/oppnzeeee /pppo/eeeeeeeeeeeeee net
Kim, Shin, J., Cho, Y., Jang, J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thome, J., et al. Prometheus: Induc- ing fine uve-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2023.
Liu, Y., Fabbri, A. R., Chen, J., Zhao, Y., Han, S., Joty, S., Liu, P., Radev, D., Wu, C.-S., and Cohan, A. Bench- marking generation and evaluation capabilities of large language models for instruction controllable summariza- tion. arXiv preprint arXiv:231 2023b.
Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M. Prometheus 2: An open source language model special- ized in evaluating other language models. arXiv preprint arXiv:2405.01533 2024.
Liu, Y., Yao, Z., Min, R., Cao, Y., Hou, L., and Li, J. Rm- bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410.1614 2024b.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213,20 2022.
Liu, Y., Yao, Z., Min, R., Cao, Y., Hou, L., and Li, J. Pair- wise rm: Perform best-of-n sampling with knockout tour- nament. arXiv preprint wix::255111130000 2025.
Kwon, W., Li, Z.,, Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H.. Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Luo, L., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., Sun, J., et al. Improve mathemati- cal reasoning language models by automated process supervision. arXiv preprint arXiv:2406.0653 2024.
Kydlíček, H. Math Math Verification Library, 2025. URL hwwps:ttt//.tom https:///iitit /github. cwm cww/huggingfacee////o/ooooooo mm ath-verify.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prahhumoye, S., Yang, Y., al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L.., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint 2rxiv:2403.13787 2024.
Mao, Y., Kim, Y., and Zhou, Y. Champ: A competition- dataset for fine-grained analyses of Ilms' mathematical reasoning capabilities. arXiv preprint arXiv:2401.06%0 2024.
Lan, T., Zhang, W., Xu, C., Huang, H., Lin, D., Chen, K., and Mao, X.-l. Criticeval: Evaluating large language model as critic. arXiv preprint arXiv:2402. 1...2.4422...44 2024.
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint 11w.::::211100011111 2021.
Li, J., Sun, S., Yuan, W., Fan, R.-Z.,, Zhao, H., and Liu, P. Generative judge for evaluating alignment. arXiv preprint 10Xiv:2310.05470, 2023.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20050. 2023.
Ouyang, I.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2771 2022.
Lin, Z., Gou, Z., Liang, T., Luo, R., Liu, H., and Yang, Y. Criticbench: Benchmarking 11ms for critique-correct reasoning. arXiv preprint 2xXx:22402.14800 2024.
Park, J., Jwa, S., Ren, M., Kim, D., and Choi, S. Offsetbias: Leveraging debiased data for tuning evaluators. arXiv preprint arXiv:2407.0655// 2024.
Liu, C. Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Saad-Falcon, J., Vivek, R., Berrios, W., Naik, N. S., Franklin, M., Vidgen, B., Singh, A., Kiela, D., and Mehri, S. Lmunit: Fine-grained evaluation with natural language 2024.
Vu, T., Krishna, K., Alzubi, S., Tar, C., Faruqui, M., and Sung, Y.-H. Foundational autoraters: Taming large lan- guage models for better automatic evaluation. arXiv preprint wxXiv2240.1088/1 2024.
unit tests. arXiv preprint arXiv:2412.
Saunders, W. Yeh, C., Wu, Bills, S., Ouyang, I.,, Ward, J., and Leike, J. Self-critiquing models for assisting human evaluators, arXiv preprint trXiv:2206.0580) 2022.
Wang. P., Li, L.,, Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: A label-free step-by-step verifier for llms in mathematical reasoning. 2023a.
arXiv preprint arXiv:2312.0880
Shah, N. B., Balakrishnan, S., Bradley, J., Parekh, A., Ramch, K., Wainwright, M. J., et al. Estimation from pairwise comparisons: Sharp minimax bounds with topol- ogy dependence. Journal of Machine Learning Research, 17(58):1-47, 2016.
Wang, P., Xu, A., Zhou, Y., Xiong, C., and Joty, S. Di- rect judgement preference optimization. arXiv preprint arXiv:2409. 14664, 2024a.
Wang. T., Kulikov, I., Golovneva, O., Yu, P., Yuan, W., Dwivedi- Yu, J., Pang, R. Y., Fazel-Zarandi, M., Weston, J., and Li, X. Self-taught evaluators. arXiv preprint 2024b.
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03330 2024.
Wang, X., Wei, J., Schuurmans, D., Le, Q. Chi, E.,, Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprinl arXiv: //101, 2022.
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances in Newral Information Processing Systems, 36, 2024.
Wang, Y., Yu, Z., Yao, W., Zeng, Z., Yang, L.,, Wang, C., Chen, H., Jiang, C., Xie, R., Wang, J., et al. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. In The Twelfth International Confer- ence on Learning Representations, 2023b.
Shiwen, T., Liang, Z., Liu, C. Y., Zeng, L., and Liu, Y. Skywork critic model series. https:/ htsps://huggee /huggingf ace co/skywork, September 2024. URL https: / //wuggingface.....o///////////ee http://ggf......oooooooooo co/Skywork.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 01224244-48371 2022.
Shridhar, K., Sinha, K., Cohen, A., Wang. T., Yu, P., Pa- sunuru, R., Sachan, M., Weston, J., and Celikyilmaz, A. The art of Iim refinement: Ask, refine, and trust. In Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5872-5883, 2024.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F,, Wei, H., et al. Qwen2. 5 technical report. arXiv preprint 251 2024.
Snell, C., Lee, J.. Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.033: 03314, 2024.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances Neural Information Processing Systems, 36, 2024.
Ye, 7., Li, X., Li, Q.. Ai, Q., Zhou, Y., Shen, W., Yan, D., and Liu, Y. Beyond scalar reward model: Learning generative judge from preference data. arXiv preprint arXiv:2410.03742. 2024.
Stechly, K., Marquez, M., and Kambhampati, S. Gpt- 4 doesn't know it's wrong: An analysis of iterative prompting for reasoning problems. arXiv preprint arXiv:2310. 12397, 2023.
Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. E. Self-rewarding language mod- els. In Forty-first International Conference on Machine Learning, 2024.
Tan, S., Zhuang. S., Montgomery, K., Tang, W. Y., Cuadron, A., Wang, C., Popa, R. A., and Stoica, I. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12788 2024.
Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., and Chen, D. Evaluating large language models at evaluating in- struction following. arXiv preprint arX1v:2310.07641, 2023.
Valmeekam, K., Marquez, M., and Kambhampati, S. Can large language models really improve by self-critiquing their own plans? arXiv preprint arXiv:2310.08: 10.08/18, 2023.
12

---

Evaluating Judges Evaluators: The JETTS Benchmark of Lude-sJuugges as Test-Time Scaling Evaluators
Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Ku-
mar, A., and Agarwal, R. Generative verifiers: Re- ward modeling next-token prediction. arXiv preprint arXiv:2408. 15240, 2024a.
Zhang, Y., Khalifa, M., Logeswaran, L., Kim, J., Lee, M., Lee, H., and Wang, L. Small language models need strong verifiers to self-correct reasoning. arXiv preprint mrXiv:244.117140 2024b.
Zhang, Z., Zheng, C.,, Wu, Y., Zhang, B., Lin, R., Yu, B., Liu, D., Zhou, and Lin, The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07344 007301, 2025a.
Zhang, Z., Zheng, C.,, Wu, Y., Zhang. B., Lin, R., Yu, B., Liu, D., Zhou, and Lin, The lessons of developing process reward models in mathematical reasoning. arXiv preprint ar NKVv:25011003300 2025b.
Zheng, C., Zhang, Z., Zhang, B., Lin, R., Lu, K., Yu, B., Liu, D., Zhou, and Lin, Processbench: Identifying process emrors in mathematical reasoning. arXiv preprint arXiv:2412.06559, 2024.
Zheng, L., Chiang, W.-L., Sheng. Zhuang, S., Wu, Z., Zhuang. Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Zhou, E., Zheng, G., Wang, B., Xi, Z., Dou, S., Bao, Shen, W., Xiong, L.,, Fan, J., Mou, Y., al. Rmb: Com- prehensively benchmarking reward models in Ilm align- ment. arXiv preprint arXiv:2410.0988) 2024.
Zhou, Lu, T., Mishra, S., Brahma, S. Basu, S., Luan, Y., Zhou, D., and Hou, I.. Insllwntion-follooinng evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, N. B., Zhan, H., He, Paul, I., al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406. 15877, 2024.
13

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLM--sJJudgee as Test-Time Scaling Evaluators
A Additional benchmark details: Evaluation procedures, judge models, and prompts
In this section, we provide additional details about benchmark evaluation procedures and the judge models evaluated.
A.1 Benchmark evaluation procedures.
Math reasoning. For GSM8K, we prompt generator models with 0-shot version of the CoT prompt used by Llama-3.1 evaluations". To parse and grade results, we use Math-Verify (Kydliček, 2025), comparing the model responses against the list of accepted responses provided by Llama-3.1 evaluations.
For MATH, we prompt the generator with 0-shot CoT prompt used in Shepherd- Mathh (Wang et al., 2023a). To parse and grade results, again use Math- comparing model responses against the original solution in MATH.
For CHAMP, we utilize the default O-shot prompt released by CHAMP and their suggested evaluation pipeline: The generator produces a response and is then asked to summarize its response into a final answer. A frontier LLM is then prompted to evaluate the final answer and the ground-truth. We utilize GPT-4o for this evaluation. We evaluate all responses generated across all tasks, regardless of if they are chosen or not, and store the results to be re-used for different judges. This ensures that judge evaluation not influenced by potentially stochastic GPT-4o inference.
Code generation. We utilize the code generation and evaluation scripts provided by BigCodeBench (Zhuo et al., 2024) and EvalPlus (Liu et al., 2023a), with minor modifications accommodate grading arbitrary subsets of the full evaluation set. For fast evaluation, similar to CHAMP evaluation, we evaluate all responses once and store the results.
Instruction following. We directly prompt the generator model with the prompts provided by both AlpacaEvval (Liu et al., 2023a) and IFEval (Zhou et al., 2023) and utilize the evaluation scripts. We again making minimal changes to accommodate grading arbitrary subsets of the full evaluation set. For AlpacaEval, we follow the same procedure as CHAMP and evaluate all responses, storing results to be re-used for different judges. This again eliminates judge-by jydjudg stochasticity in GPT-4 evaluations.
A.2 Additional details about evaluated judge baselines.
As noted in Tab. 2, we benchmark 10 judges of varying capabilities. Here, we provide a more comprehensive overview of each judge evaluated. We run inference with vLLM (Kwon et al.. 2023), with 1 A100 40GB GPU for judges with sizes 12B and smaller, and 8xA100 100 40GB GPUs for judges with sizes 8x7B and larger.
SFR-Judge (Wang et al., 2024a): SFR-Judge is family of three judge models trained from Llama-3.. 1-8B-and 70B-instruct and Mistral-NeMo- 12B- Instructt using direct preference optimization (DPO). These judges are trained with data samples from a diverse array of source datasets to provide natural language critiques and perform three evaluation tasks: Pairwise comparisons, single-rating evaluation, and binary yes/no classification. Notably, SFR-Judge training data preserves original human evaluation instructions, meaning the models are amenable to flexible prompting.
Skywork- Critic (Shiwen et al., 2024): Skywork-Critic is a family of two judge models trained from Llama-3. 1------..18BBBa-a 70B-instruct using supervised finetuning (SFT). These judges are trained on smaller, curated preference dataset to perform pairwise comparison evaluation, without providing natural language critiques. Skywork- models are trained with a fixed prompt template.
OffsetBias (Park et al., 2024): OffsetBias is a pairwise comparison (without natural language critiques) model trained from -antma----B-Insstuu using SFT. is trained with data specifically aimed at mitigating common LLM-as-judge biases, such length bias. OffsetBias is trained with fixed prompt template.
Prometheus-v2.00 (Kim et al., 2024): Prometheus-v2.0 is a family of two judge models trained from Mistral-7B- and 7x--iisttu These judges are trained on purely synthetic data synthesized from GPT-4 to perform pairwise comparisons and single-rating evaluation, and provide natural language feedback. Notably, Prometheus specifically generates per-instance fine-grained evaluation criteria, making it amenable to flexible prompting.
'Meta releases evaluation data publicly. For example, met.a 1 ama foma/Llamme L.lama-2 3. 1-8B- instructtest
results: https: http:///g/gggggac.e./oo/ co.aaaasett.///aaasstttt.s...s
14

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLM-asJJudges as Test-Time Scaling Evaluators
Seffttaught-evaluator (Wang al., 2024b): Self-taught-evaluaoor is model trained from Llama-3. 1-10--3..-0B----- --stt-nnstttttt perform pairwise comparison evaluation and provide natural language eritiques. It is trained using iterative DPO self-teaching, where adversarially generated samples from the model are produced at each training round, classified into correct or incorrect judgments, and used to update the model with DPO. self-tauggt-evaluate is trained with fixed prompt template.
Themis (Hu al., 2024): Themis single-rating specific model (with natural language critiques) trained from Llama-3-8B-Innee Instructt using single-rating margin-based version of DPO. It trained from a large mixture of preference data from a diverse array of sources. Notably, original instructions given to human annotators is preserved in Themis training data, making it amenable to flexible prompting.
For each judge, we adhere to prompt templates and judgment parsing code provided by authors, minimally updating the evaluation criteria necessary. We provide examples of prompts in the next section. We additionally evaluate four reward model baselines: three ORMs for reranking and one PRM for beam search.
Skywork-Reward-ie (Liu et al., 2024a): A family of sequence classifier outcome reward models trained from 1-ama-3.1-8B and Gemma-2-278 using the same dataset as the Skywork-Critic series.
Offset RM (Park et al., 2024): A sequence classifier outcome reward model trained from Llama-3-88 using the same dataset as the OffsetBias generative judge.
Swee2.5-Math-PR.. (Zhang al., 2025a): A process reward model specific for math domain trained from Qwen2.5,
A.3 Prompts used in JETTS.
Judge prompts. Here, we provide an example of the pairwise (Fig. 15), Likert (Fig. 16), and additive rating (Fig. 17) prompts used for evaluation. We show the prompts used for SFR-Judge, with other prompts differing only in the template (e.g., the expected format of outputs, whether or not to elicit CoT, order of input elements, etc). In all of the prompts, as mentioned in Sec. 3.3, the message indicating a partial solution inserted in the seaxtaalrresponse e portion of the prompt, or is left empty (with surrounding newlines removed) for non-beam-search tasks.
Generator refinement prompt. We provide the prompt given to the generator for critique-based refinement in Fig. 18. As mentioned in Sec. 3.4, the generator is allowed to interpret the judge feedback and decide that no further refinement is necessary.
15

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLMeas-Judges as Test-Time Scaling Evaluators
You are a helpful assistant evaluating the quality of the responses for a given instruction. Your goal is to select the best response for the given instruction.
Select Response A Response B, that is better for the given instruction. The two responses are generated by two different Al chatbots respectively. Do NOT say both / neither are good.
Here are some rules of the evaluation:
(1) You should prioritize evaluating whether the output honeesty/preciielyddoeeyy executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc.
(2) Responses should NOT contain more/less than what the instruction asks for, such responses do NOT precisely execute the instruction.
(3) You should avoid any potential bias and your judgment should be as objective as possible. Here are some potential sources of bias:
The order which the responses were presented should NOT affect your judgment, as Response A and Response are *equally likely** be the better.
The length of the responses should NOT affect your judgement, longer response does necessarily correspond to better response. When making your decision, evaluate if the response length is appropriate for the given instruction.
[partial_reste response_note
Your reply should strictly follow this format: **Reasoning:*** ** <feedback evaluating the responses>
**Result:** <A B>
Here is the data.
Instruction:
(query_texr text]
Response A:
[response_A]
Response B:
[response_B]
The pairwise evaluation prompt given to SFR-Judge models. Blue text are changed to the corresponding actual value when rendering the template. Everything else is presented literally. The partial response note slot holds the beam search judging note of Sec. 3.3 is only used in the non-lookahead mode of beam search. The partial response note slot holds the beam search judging note of Sec. 3.3 only used the non-lookahead mode of beam search.
Figure 15:
16

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLMMas-Judges as Test-Time Scaling Evaluators
You are tasked with evaluating a response based on given instruction (which may contain an Input). Provide comprehensive feedback on the response quality based on the rules for evaluation. Follow this with a score between and 5. Avoid generating any additional opening, closing, or explanations.
Here are some rules of the evaluation:
(1) You should prioritize evaluating whether the output honessly/precieeyylooeeyy executes the instruction, then consider its helpfulness, aceuracy, level of detail, harmlessness, etc.
(2) Responses should NOT contain more/less than what the instruction asks for, such responses do NOT precisely
execute the instruction.
(3) You should avoid any potential bias and your judgment should be as objective as possible. Here is potential source of bias:
The length of the response should NOT affect your judgement, as a longer response does not necessarily correspond to
better response. When making your decision, evaluate if the response length is appropriate for the given instruction.
(partial restllseeesposee_____ ]
Your reply should strictly follow this format: **Rassoning:*** <Your feedbacks
**Result: ** <an integer between and 5>
Here is the data:
Instruction:
..
[query_text]
Response:
[response]
Figure 16: The Likert single-rating evaluation prompt given to SFR-Judge models. Blue text are changed to the corresponding actual value when rendering the template. Everything clse is presented literally.
17

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLMe-s-Judges as Test-Time Scaling Evaluators
You are tasked with evaluating a response based given instruction (which may contain Input). Provide comprehensive feedback the response quality based on the rules for evaluation. Follow this with a score between and 5. Avoid generating any additional opening, closing, or explanations.
Here are some rules of the evaluation:
(1) You should prioritize evaluating whether the response satisfies the provided rubric. The basis of your score should
depend exactly on the rubric.
(2) You should avoid any potential bias and your judgment should be objective possible. Here is potential source
of bias:
The length of the response should NOT affect your judgement, as a longer response does not necessarily correspond to
better response. When making your decision, evaluate if the response length is appropriate for the given instruction.
[partial_esponse_nntte
Your reply should strictly follow this format: **Reasoning:** 8* <Your feedback>
**Result:* <an integer between and 5>
Here is the data:
Instruction:
[query_text]
Response:
(response ]
Score rubrics:
Add one point the response is relevant and provides some information related to the user's inquiry, even if it is incomplete or contains some irrelevant content.
Add a second point if the response addresses a substantial portion of the user's question, but does not completely resolve the query or provide direct answer.
Add third point if the response answers the basic elements of the user's question in useful way, regardless of whether it seems to have been written by an AI Assistant if it has elements typically found in blogs or search results. Add fourth point if the response is clearly written from an AI Assistant's 's perspective, addressing the user's question directly and comprehensively, and is well-organized and helpful, even if there is slight room for improvement in clarity, conciseness or focus.
Add a fifth point for a response that is impeccably tailored to the user's question by an Assistant, without extraneous information, reflecting expert knowledge, and demonstrating high-quality, engaging, and insightful answer.
The additive single-rating evaluation prompt given SFR-Judge models, adapted from Yuan al. (2024). Blue text changed to corresponding actual value when rendering template. Everything else presented literally. partial response note slot holds the beam search judging note of Sec. 3.3 is only used in the non-lookahead mode of beam search.
Figure 17:
18

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLMMas-Judges as Test-Time Scaling Evaluators
You help revise a machine- -eririnned response to a user query. Below, you will be provided with the user query and the machine--genratten response. You will also be provided with the output of an evaluator model, which gives a score (max being (max_score)) }) and an explanation for the score.
You should revise and improve the current response, following the evaluator's recommendation. If the evalua- tor does not identify any area of improvement, you should output "No revision needed." Otherwise, you should output the revised response surrounded by the respoose>e and anything else.
tags. You do not need to output
<query> (query)
</query>
<original >
(response]
deiiginal_rsponee
<score>
(score) out of (max_score). </score>
<explanation>
(explanation)
<lexplanation>
Your revision (or "No revision needed.")
Figure 18: The refinement prompt given to the generator. Blue text are changed to the corresponding actual value when rendering the template. Everything clse is presented literally.
19

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLMeas-Judges as Test-Time Scaling Evaluators
You are helpful assistant in evaluating the quality of the responses for given instruction, which is a math problem. Your goal is to select the best response for the given instruction. Select Response A or Response B, that is better for the given math problem. The two responses are generated by two different Al chathots respectively.
Do NOT say both / neither are good.
Here are some rules of the evaluation:
(1) You should prioritize evaluating if the output arrives at the correct solution for the given math problem and if the logical reasoning to derive the solution is sound.
(2) If both responses arrive at the correct solution, choose the response that contains the better logical reasoning. For reasoning, prioritize correctness, then completeness and conciseness.
(3) Responses should NOT contain more/less than what the question asks for, as such responses do NOT precisely solve the problem.
(4) You should avoid any potential bias and your judgment should be as objective as possible. Here some potential sources of bias:
The order in which the responses presented should NOT affect your judgment, as Response A and Response B are
*qqually likely** to be the better.
The length of the responses should NOT affect your judgement, as longer response does not necessarily correspond to better response. When making your decision, evaluate the response length is appropriate for the given instruction.
You are a helpful assistant evaluating the quality the responses for a given instruction, which a coding problem. Your goal is select the best response for the given instruction. Select Response A or Response B, that is better for the given coding problem. The two responses generated by two differrntt AI chatbots respectively. Do NOT say both / neither are good.
Here are some rules of the evaluation:
(1) You should prioritize evaluating the output response code correctly implements the desired functionality in the instruction.
(2) If both responses correctly implement the desired functionality, choose the response that contains the better written
code. Prioritize conciseness and readibility.
(3) Responses should NOT contain more/less than what the question asks for, as such responses do NOT precisely solve the problem.
(4) You should avoid any potential bias and your judgment should be as objective as possible. Here are some potential sources of bias:
The order which the responses were presented should NOT affect your judgment, as Response and Response are
* *equally likely** be the better.
The length of the responses should NOT affect your judgement, longer response does not necessarily correspond to better response. When making your decision, evaluate if the response length is appropriate for the given instruction.
Figure 19: Domain-specific evaluation criteria given to SFR-Judge-8B for math reasoning (top) and code generation (bottom) evaluation.
20

---

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM--sJJugges as Test-Time Scaling Evaluators
B
Complete benchmark results and additional experiments
In this section, we present the raw benchmark performance as well as additional visualizations for all judges and reward model baselines across the three tasks.
B.1 Response reranking
Pairwise round-robin reranking details. Because judge models are prone to positional biases (Li et al., 2023), we employ a consistency-basee pairwise evaluation setup. We form all NNV 11/22 pairs of responses. Then, we run each pairwise comparison twice, swapping the order of responses in the second run. If the same response is chosen for both runs, then assign the selected response 1 "point" and the rejected response 0 "points". If the judge is inconsistent, i.., selects different a different response when the order changes, we consider this a tie and assign both response 0.5 "points". The round-robin tournament winner is selected based on the response that has the most points after all N(N - 12/2 comparisons are completed.
Full experimental results. We present the full raw benchmark performance for all judges, generators, and dataset combinations for the pairwise comparison protocol in Tab. 3, Likert rating protocol in Tab. 4, and additive rating protocol in Tab. 5. We also report the performance of Llama-3. 1-B.----Inmsse Instruct prompted with SFR- SFR-Judge prompts and three reward models to contextualize judge performance.
Prom 7B
SPR 8B
SC
OB 8B
SPR 12B
Prom 8x7B
SPR 70B
SC, 70B
STE: 70B
Llama 8B
Best RM
Greedy
Random
Omcle
Maj
Llama 8B
85.22
89.08
88.86
85.82
89:31
85.22
93.71
91.36
90.07
87.11
90.37
85.67
82.08
96. 44
90.83
Llama 70B
95.83
95.91
95.98
95.22
95. 68
94.69
96.29
96.59
94,92
95.53
96.51
95.53
95.11
98.48
96.82
GS
Qwen 32B
95.00
96.21
95.53
95.68
95.60
91,81
95.91
95.75
95.22
95.15
96.06
95.22
95.22
38.56
96.06
Qwen 728
95.07
95.53
95.60
95.75
y5
95.30
95. 68
96.21
95.75
95.22
96.29
95 68
95.52
97. 87
96.
Qwen-M M 7B
94.69
95.45
95:53
94,69
95 64)
94.47
95.68
95.68
95.07
94.77
95.68
95.15
91.65
97.88
95.83
Dpck-M 7B
80.44
86.73
87.49
82.41
81.05
91.74
87.11
84.15
$7.83
95.53
87.49
Llama SB
23.19
29.98
25.76
24.85
29.90 29.98
25.15
36.71
29.83
29.98
29.15
29.83
24.70
21.98
$3.47
33.08
Llama
47.21
42.95
45.77
51.81
49.55
47.81
68.35
MATH
Qwen 32B
54.76
59.06
57.93
53,55
60.05
$5.06
64.00 44.05
59.82
$9.59
58.01
59.34
$7.10
$2.24
78. 17
62.61
Qwen 72R
62.31
86
64.20
62.31
66.84
67.37
65.33
66.01
62.99
K2.75 77
69.26
Qwen-M 7B
66.77
70.09
67.75
65.48
69.71
67.07
71.68
69.79
68.96
69.71
68.58
66.33
55.28
80. 14
70.24
Dpsk-M Mak-MM77 7B
16.01
23.56
21.45
21.07
21.53
19.94
29.68
24.85
26.28
23.13
24.58
18.66
17.04
47.21
25.68
Llama 8B
30.74
30.37
30.74
31.11
34.07
31.11
34.07
32,96
33.33
30.37
31.85
29.26
27.30
60.00
CHAMP
Llama 70B
45,93
51,48
50.74
48.52
48.89
45,56
48.15
52,22
47.78
48.52
52.22
47:41
46.07
71.48
Qwen
70.00
71.11
67.41
72.96
68.89
70.37
68.89
70.37
67.78
71.30
75.19
70.67
Qwen 72B
65.93
66.30
70.00
66.67
68 62.52
68.15
67 1
70.37
67.78
68.52
68.70
71.48
67.11
85.56
Owen-M M 7B
70.37
68.15
67.04
67.78
67.41
72.59
70.37
66.67
70,00
46.26
Dpsk-M 7B
27.01
42.59
38.52
31.44
39.26
25.56
12,96
10,00
39.63
30.74
41,48
26.30
28.78
72.22 22.22
Llana 8B
$3.66
64.02
61.59
60.95
55.49
61.59
67.07
$6.70
HumimEbale
Lama 70H
67.07
71. 344
78.05
75.61
77.43
70.73
75.61
79.27
77,44
75.61
78.66
75.61
73.23
90.85
Qwen 32B
79.88
62.32
81.10
85.15
82.32
79.88
81.10
83.54
81.10
81.10
85.37
81.10
82.56
93.29
Owen 72B
82.93
84.76
85.98
85, .6
82.9 $2.93
81.71
36.59
87.80
82.93
82.32
87.80
82.33
82.13
93.90
Qwen-C 7B
75.00
62.32
81.10
89.88
82.32
78.05
$1.15
83.51
62,93
82.32
82,32
86.59
81.52
93.90
Dpsk-C 6.78
66.46
69.51
75.00
67.66 686
67.66 67.68
66.46
77.43
76.83
74.39
69:51
77.44
71.95
67.62
86, 59
Llama 8B
47.6
54.23
57.41
58.99
$2.38
49.21
56.88
62,70
62.17
56.61
60.85
$4.50
55.21
76.46
Llama
58.29
64.29
65.34
56.08
60.32
68.39
83.07
MBPP
Owen 32B
73.81
75.93
75.93
76.46
74.87
73.81
75.66
78.04
75.40
75.13
76.06
75.40
75.45
$4
Owen
76.16
16.19
75.40
76.16
78.01
75.71
81.66
Qwen-C 78
66.40
69:05
70.11
70.11
66.93
66.93
72.49
75.13
73.28
71.43
72.22
71.69
66.51
85.45
Dpså-C 6.7B
58.73
61.11
57.67
57.11
60.58
61.90
65.87
65.87
62.12
79.37
Big CodeBBenhh
Llama 8R
26.23
32.46
33.25
32.72
31.23
28.33
36.05
37.19
36.23
30.35
35.09
31.67
28.03
56.84
Llama 70B
40.44
42.19
42.89
42.25
41.67
40.00
42.63
43.51
43.60
43.33
43,90
45.44
41.92
62.63
Qwen 32B
43.51
16.23
45.26
41.91
45.79
43.86
47.72
16.84
46.67
45.18
46.23
15.53
41.66
65.18
Qwen 72B
45.79
46.84
47,46
47,89
47.63
45,70
47.54
47.228
47.63
45.79
48.60
46.67
46.30
60.18
Qwen-C 7B
36.32
10.26
39.74
41.23
39.91
38.33
43.33
13.86
42.63
10.18
42.37
12.37
37.89
61.58
Dpsk-C 6,78
32,28
36.14
38.42
37,54
37.19
33.60
40.18
40.09
39,30
36.84
38.99
34.91
32.49
$8.95
IPEval AlpaciEval
Llama SB
37.15
36.98
34,23
39.93
37.31
37.63
32.00
38.98
27.26
26.80
55.07
Llama 70B
16.02
44.88
42.88
40.99
45.92
45.20
46.87
43.90
45.69
40.17
45.08
35.45
34,27
61.90
Owen
41.3
44,59
45.97
41.38
69
Owen 72B
58.94
62.22
55.98
58.35
61.16
60.51
63.69
61.433
63.03
58.73
62.56
52.01
56.76
76.52
Lama 8R
67.28
70.24
70.06
71.90
71.90
68.95
74.68
71.34
72.09
69:87
73.01
72.83
67.79
87.80
Llama 70B
82.07
81.70
80.78
80.04
81.70
80.04
82.99
82.26
82.99
80.22
82.62
84.29
81.21
92.42
Qwen 32B
80.60
$1.70
60,41
79.48
80.04
60.22
82.62
82.62
82.26
79.85
82.38
80.22
79.44
90.02
Qwen 728
82.81
84.66
84.47
83.73
84.84
83.55
85.03
85.40
83.92
85.21
86.04
83.55
83,42
92.05
Table 3: Performance of the top response after LLM-judge pairwise comparison- based reranking, in the context of the vanilla Llama 3.1 8B model using the SFR-Judge pairwise comparison prompt, the best reward model (using single-instance score to rerank), and various baselines. "-M" and "-C" suffixes represent the math and code domain-specific models.
21

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLdeas-Judges as Test-Time Scaling Evaluators
Prom 7B
SFR 8B
Thm
STR 12B
Prom 8x7B
SER 70B
Llama
OBay SB
8B
SaM 27B
Greedy
Rundom
Orccle
Maj
Llama 8B
84.32
86.86
81.92
86.16
83,95
88.33
84,70
90.03
88.78
90.37
85.67
82.08
96.44
90.83
Liama 70B
95.80
95. 95.60
GSMSS
Qwen 32B
95.45
95.65
95.07
95. 34
95.30
95.47
95.35
96.06
95.91
95.79
95.22
95.22
98.56
96.06
Qwen 72B
95.85
Qwen-M M 7B
93.72
95.36
91.86
95.35
94.19
95.57
94.74
95.45
95.53
95.68
95.15
91.65
97.88
95.83
Dpsk-M
82.74
87.49
Llama 8B
24.33
26.69
21.30
26.35
24.65
29.42
24.34
29.25
28.44
29.83
24.70
21,98
53:47
33,08
Llama 70B
45.05
45.67
41.59
46.23
14.71
46.15
45.14
46.98
47.73
50.04
43.81
42.07
68.35
$2.49
MATH
Qwen 32B
54.79
54.53
52,11
55.84
53,97
54.99
55.05
55,97
59.34
57.78
57,10
52.24
78.17
62.61
Qwen 72B
62.73
63.61
61.60
63:16
62.06
63.78
62.84
64.19
66.01
65.94
62.99
61.37
82.78
(9 69.26
Qwen-M
55.81
67.35
56.26
67.33
61.71
67.78
64.14
67.86
67.71
68.58
66.39
55,28
80.14
70.24
Dpsk-M 78
17.22
21.76
16.33
19:83
18.11
24.87
20.30
23.79
24.58
24.43
18.66
17.04
47.21
25 68
Llama
31,57
30.74
CHAMP
Llama 70B
47.02
50.30
46.99
50.90
47.95
48.94
47.67
52.22
50.74
50.56
47.41
46.07
71.48
Qwen 32B
70.17
71.30
75.19
Qwen 72B
64.95
68.09
67.42
67.40
66.43
66.99
68.50
68.15
68.70
67.41
71.48
67.11
85.56
Qwen-M
67.26
70.00
Dpsk-M 78
26.70
31,83
29.15
33.02
30,52
39,11
31,96
39.44
41,48
34,81
26.30
28.78
72.22
Llama 8B
58.61
61.54
57.03
61.47
$7.49
64.87
57.55
60.37
64.33
67.07
63.35
56.70
79.88
Hummeval
Llama 70B
74.76
72.88
73.63
73.17
73.37
74.11
73.84
77.44
78.66
77.44
75.61
73.23
90.85
Qwen 328
82.90
81.30
82.29
82.53
82,50
84.85
81.80
84.76
81.71
85.37
81.10
82,56
933.29
Quen 72B
82.01
85.51
82.17
84.51
82.10
84,61
81.87
84.15
84.76
87,80
82.32
82,13
93.90
Qwen-C: 78
81.81
81.95
81:77
81 .28
81.91
84,59
81.80
81.71
79.27
82.32
86.39
81.52
933.00
Dpsk-C6.77
67.95
69.97
73.16
Llama 8B
56.75
$4,20
55.94
$7.17
56.42
60.72
56.85
60.32
60.85
56.88
54,50
55.21
76.46
Llama
MBnp+
Qwen 32B
74.94
75.42
75.77
75.40
75.41
75.59
75.84
75.13
76.06
75.40
75.40
75.45
84.13
Qwen 72B
76.95
76.01
75.26
76.46
76.19
Qwen-C 78
67.94
67,73
67.17
69 69.28
67.74
70.95
69.33
71,44
72.22
71.56
71
66.51
85.45
Dpak-C 6.7B
62.26
62.73
62.64
62.12
62,48
61,90
63.54
63.62
65.87
58,20
66.14
62,12
79.37
Dig DigCooelench
Llama 8B
29.69
30.65
28.95
32.00
29.85
34.20
30.43
32.63
34.96
35.09
31,67
28.03
56.84
LJama 708
42.5 42.37
42.75
42.55
42.88
42,73
43.31
42.74
43.63
43.90
43.86
4% 43.44
41,92
62.63
Quen 32B
45.01
45.61
44.70
45.44
44,93
45.80
45.65
46.05
46.07
46.23
45.53
44,66
65.18
Qwen 72B
46.76
46.93
46.33
47.27
46.37
47.45
46.63
46.89
48.60
48.33
46.67
46,30
60.18
Qwen-C
38.66 30
39.95
42,06
Dpsk-C6.75 6.7B
34.22
34,80
33.07
35.66
33.97
38.65
35,31
36.84
38.99
38.86
34,91
32.49
58.95
AlpacaEval
29.95
38.98
Llama 708
36.
36,86
35.01
36.55
35.50
35.60
33.99
39,10
42.05
45.08
35.48
34,27
61.90
Quen
37.08
37.08
37.21
36.17
36.54
35.98
45.17
Qwen 72B
54.40
54.75
53.80
54.99
$4,05
54.54
54.08
55.95
60.34
62.56
52.01
56.76
76.52
Llama 8B
67.85
70.34
69.13
70.63
69.15
71.80
70.41
72.46
71.07
73.01
72,83
66.43
87.80
IFEal
Llama 70B
82.26
82.27
82.26
82.29
81,99
82.72
81.45
81.15
82.62
$1.89
84.29
79.94
92.42
Qwen 328
80.40
80.10 19
80.30
79.71
79.89
82.64
80.95
82.38
80.41
14
80.22
77.38
90.02
Quen 72B
83.25
83.72
83.29
84.29
83.63
84.87
83,91
84.57
$6.04
83.46
83.55
81.44
92.24
Table 4: Judge reranking performance under the Likert scale single-instance rating protocol.
22

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLdeas-Judges as Test-Time Scaling Evaluators
Prom 7B
SFR 8B
Thm
STR 12B
Prom 8x7B
SER 708
Llama
OBay SB
8B
SaM 27B
Greedy
Rundom
Orccle
Maj
Llama 8B
83.28
85.45
82.10
85 32
85.69
86.51
84.99
90.03
88.78
90.37
85.67
82.08
96.44
90.83
Liama 70B
95.63
GSMSS
Qwen 32B
95.26
95.47
94,97
95. 95.32
95.39
95.27
95.49
96.06
95.91
95.79
95.22
95.22
98.56
96.06
Qwen 72B
95.57
Qwen-M M 7B
91.61
95.25
91.85
95.14
93.78
95.34
95.05
95.45
95.53
95.68
95.15
91.65
97.88
95.83
Dpsk-M
83.42
79.97
87.49
Llama 8B
22.79
25.16
21.19
25.96
23,97
26.96
24.59
29.25
28.44
29.83
24.70
21,98
53:47
33,08
Llama 70B
43.35
45.71
42.54
46.26
14.26
44.98
46.02
46.98
47.73
50.04
43.81
42.07
68.35
$2.49
MATH
Qwen 32B
54.11
54.49
51.81
55.16
55,09
53.55
54,49
55,97
59.34
57.78
57,10
52.24
78.17
62.61
Qwen 72B
61.71
62.97
61,85
62.84
63.08
62,46
62.49
64.19
66.01
65.94
62.99
61.37
82.78
(9 09.26
Qwen-M
50.09
67.07
54.68
67.42
61.01
67.41
63.83
67.86
67.71
68.58
66.39
55,28
80.14
70.24
Dpsk-M 78
18:35
20.05
16.46
18:45
18.76
22:45
18.64
23.79
24.58
24.43
18.66
17.04
47.21
25 68
29.77
27.78
30.74
CHAMP
Llama 70B
47.96
49.85
47.19
49.40
49.44
47.41
46.37
52.22
50.74
50.56
47.41
46.07
71.48
Qwen 32B
70:51
71.30
75.19
Qwen 72B
67.32
67.79
66.58
67.14
67.74
67.98
66.39
68.15
68.70
67.41
71.48
67.11
85.56
Qwen-M
61.98
70.00
Dpsk-M 78
27.55
34,02
29,36
31 844
29.61
37,14
28.84
39.44
41,48
34,81
26.30
28.78
72.22
Llama 8B
57.33
58.53
56.96
60.50
$7.78
61.74
59.59
60.37
64.33
67.07
63.35
56.70
79.88
Hummeval
Llama 70B
73.72
73.47
73.39
72.85
73.34
73.94
72.43
77.44
78.66
77.44
75.61
73.23
90.85
Qwen 328
83.05
79.01
82.53
82.58
82.60
82.51
81.00
84.76
81.71
85.37
81.10
82,56
933.29
Qwen 72B
82.29
84.02
82,16
84.59
82.07
83.60
83,58
84.15
84.76
87.80
82.32
82,13
93.90
Qwen-C 78
81. 96
82.20
81.93
81 98
82,29
82. 42.39
80.37
81.71
79.27
82.32
86.39
81.52
933.00
Dpsk-C6.79
66.34
Llama 8B
58.09
55.70
55.82
$6.76
$6.56
60.33
56.80
60.32
60.85
56.88
54,50
55.21
76.46
Llama
MBnp+
Qwen 32B
75.
75.38
75.66
74.74
75.51
75.39
75.67
75.13
76.06
75.40
75.40
75.45
84.13
Qwen 72B
76.32
76.85
75.26
76.46
76.19
Qwen-C 78
68.
67.77
66.85
70.15
67.96
71,83
66.79
71,43
72.22
71.56
71
66.51
85.45
Dpak-C 6.7B
6.7B
64.08
60.50
62.67
61.15
62.50
63.07
62.49
63.62
65.87
58,20
66.14
62,12
79.37
Dig DigCooeBench
Llama 8B
29.48
30.40
28.88
31.20
29.90
32.27
29.78
32.63
34.96
35.09
31,67
28.03
56.84
LJama 708
42.66
42.27
42.55
41 14
42.66
42.88
42,72
43.63
43.901
43.86
4% 45.44
41,92
62.63
Quen 32B
44.77
45.63
44.77
45.83
45.53
45.31
45.58
46.05
46.07
46.23
45.53
44,66
65.18
Qwen 72B
46.25
47.04
46.32
47.20
46.95
46.99
46.79
46.89
48.60
48.33
46.67
46,30
60.18
Qwen-C
37.97
42,06
Dpsk-C6.75 6.7B
33.61
33.83
32,90
35.20
34.66
37.64
33.63
36.84
38.99
38.86
34,91
32.49
58.95
IFEal AlpacaEval
29:02
30.78
38.98
Llama 708
36.05
38.19
35.64
36.60
36.48
36.17
36.32
39.10
42.05
45.08
35.48
34,27
61.90
Quen
37.06
38.06
37.62
37.77
45.17
Qwen 72B
55.17
$5.85
53.91
55.04
55.16
54,96
55.03
55.95
60.34
62.56
52.01
56.76
76.52
Llama 8B
67.99
69.37
69.11
69.49
69.75
70.48
69.32
72.46
71.07
73.01
72,83
66.43
87.80
Llama 70B
81.50
80.94
81.96
82.20
80.84
82.35
80.84
81.15
82.62
$1.89
84.29
79.94
92.42
Qwen 32H
79. 34
79.97
79.94
79:46
80.17
80.21
79.42
82.38
80.41
81.42 /4
80.22
77.38
90.02
Quen 72B
84.12
83.55
83.19
83.90
82.39
84.14
83.01
84.57
$6.04
83.46
83.55
81.44
92.24
Table 5: Judge reranking performance under the additive scale single-instance rating protocol.
23

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Additional visualizations for pairwise protocols. In Fig. 20, visualize the performance of judges capable of both single-rating evaluation and pairwise evaluation present wholistic picture of reranking performance under different protocols. For the majority of benchmark and judge combinations, pairwise reranking tends to perform the best, with additive and Likert single-rating protocols performing roughly equally.
Pairwise
Single Likert
Single Additive
Random
0,6
128 SHR
0.3
0.0
-0.3
-0.6
0.6
0.3
Normalized Helpfulness
0.0
00.
-0.6
0.6
5FR 708
0.3
0.0
-0.3
-0.6
0.6
Prom 78
0.3
0.0
-0.6
0.6
Prom 8k78
0.3
0.0
-0.3
-0,6
GSM8K
MATH
CHAMP
HumanEval+
MBPP
BlgCodeBench
AlpacaEval
IFEva
Figure 20: The reranking performance for different datasets, for each judge that supports both pairwise and singlc-rating protocols. In Fig. 21, we present an expanded version of Fig. 4 by plotting the minimum, maximum, mean, and distribution of normalized helpfulness for all pairwise judges. similar set plots shown in Fig. 22 for single-rating protocols. Judges tend to reach higher maximum performance under pairwise prompts, while the minimum possible performance is relatively stable between the protocols.
1.0
Normalized lizd Helpfulness
0.5
0.0
-1.5
Math Reasoning
Code Generation
Instruction Following
Prom 78
SFR 8B
SC
OB 88
SFR 12B
Prom 8x78
SFR 70B
SC 70708
STE STE700 70B
Liama 8B
Best RM RM
Random
Figure 21: The violin plot showing the full distributions of judges' pairwise reranking performance. An equivalent bar chart containing only the mean presented Fig.
Judges are able to improve AlpacaEval length-controlled win-rates as well. It is well established that L.L.jjudger are susceptible to length bias (Zeng et al., 2023; Park et al., 2024), with judges preferring longer responses, even if they do not follow user instructions. As a result, AlpacaEval designed a length- controlled win rate (Dubois et al., 2024a), which fits a generalized linear model in an attempt to normalize response preferences based on length of responses. As we show in Tab. pairwise reranking can lead large gains in conttl-contolleed win rate as well, highlighting that the responses judges
24

---

Evaluating Judges Evaluators: The JETTS Benchmark of LuMessJuugges as Test-Time Scaling Evaluators
Add.
Add.
LIK.
Add.
0.50
0.25
0.00
Normalized
-0.50
-0.75
-1.00
-1.25
-1.50
Math Reasaning
Code Generation
Instruction Following
0.2
Normaizedd Helpfuiness
-0.1
-0.2
Prom
SFR
Thm
8x7B
SFR
Llama
Random
Figure 22: Normalized helpfulness the single-rating rating protocol (with both the Likert and additive prompt) across task categorics each judge, compared to Llama-3.1 8B with judge prompt, best reward model and random reranking, averaged across generator models and datasets. The top panel shows the full distribution while the bottom panel only shows the mean.
select are high quality, regardless of length. In these settings, the Skywork- Critic models excel. Single-rating protocols also result performance gains over greedy responses, like the standard win rate setting, the gains are far less pronounced.
Full results for Likert single-rating are shown in Tab. 7 and additive single-rating are shown Tab. 8.
LC
Prom 7B
SFR 8B
SC HB
OB 8B
SFR 12B
Prom 8x7B
SER 70B
SC 70B
STE 70B
Llama HB
Best RM
Greedy
Random
Oracle
37.31
37.86
32.00
Llama 8B
31.65
32.83
36.33
31.81
35.94
32.94
37.20
37.09
33.23
29.53
37.09
25.33
25.05
52.59
Llama 70B
x
46.02
44.88
42.88
40.99
45.92
45.20
46.87
43.90
45.69
40.17
45.69
35.48
34.27
61.90
44.09
41.38
44.18
43.82
42.70
41.36
47.78
44.59
45.97
43,66
46.70
41.84
46.70
32.92
35,80
61.69
Qwen 32B
46.94
47.17
47.38
45.41
50.07
46.21
48.61
49.13
47.48
45:21
49.13
38.40
40.52
64.75
58.94
62.22
55.98
58.35
64.16
60.51
63.69
61.43
63.03
58.73
63.03
$2.01
56.76
76.52
Qwen
52.27
52.41
55.48
$2.35
$5.09
$1.56
56.07
57.42
54.24
52.13
57.42
47.18
48.56
70.85
Table 6: Judge reranking performance under the pairwise protocol on AlpacaEval, showing both non-length-coontrole (LC) win rate and length-controlled win rate. The non-LC values are the same those in Tab. 3.
Prom
8B
SFR
8x7B
SFR
Llama
Seu
Greedy
Random
28.42
29.95
27.80
29.50
27.64
29.43
27.36
33.62
36.05
38.98
27.26
26.80
55.07
26.60
27.79
25.84
27.29
25.95
27.57
26.25
32.87
34,78
35.58
25.33
25.05
52.59
Llama 70B
36.13
36.86
35.01
36.55
35.50
35.60
33.99
39.10
42.05
45.08
35.48
34.27
61.90
38.70
38.75
37.43
38.60
38.01
37.95
37.84
42.41
44.90
46.18
36.85
36.57
63.19
37.08
37.08
36.40
37.21
36.17
36.54
35.98
39.29
42.42
45.17
32.92
35,80
61.69
Qwen
42.05
41.57
41,29
42.00
41.11
41.40
41.21
45.14
46.36
46.37
38.40
40.52
64.75
54.75
54.05
54.54
55.95
60.34
56.76
Qwen 72B
49.26
49.40
48.57
49.80
48.90
49.54
49.50
53.49
$4.99
54.44
47.18
48.56
70.85
Table 7: Judge reranking performance under the Likert scale single-instance rating protocol on AlpacaEval, showing both non-length- controlled (I.C) win rate and length-controlled win rate. The non-I.C values are the same those in Tab. 4,
25

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLMMassJuugess as Test-Time Scaling Evaluators
Prom
8B
Thm 8B
SFR
Prom 8x7B
SFR
Llama
OBиa
Sux 8B
Sym 27B
Greedy
Random
Oracle
29.02
30.78
27.86
30,16
28.79
29.32
29.08
33.62
36.05
38.98
27.26
26.80
55.07
Llama 8B
27.15
28.54
25.94
27.97
26.58
27.13
26.84
32.87
34,78
35.58
25.33
25.05
52.59
Llama 708
x
36.05
38.19
35.64
36,60
36.48
36.17
36.32
39.10
42.05
45.08
35.48
34,27
61,90
39.88
38.32
38.15
42.41
44.90
36.57
37.06
38.06
36.34
37.62
31.77
36.84
37.08
39.29
42.42
45.17
32.92
35.80
61.69
Qwen
41.68
42.14
41,29
42.22
42.07
41.19
41.46
45.14
46.36
46.37
38.40
40.52
64.75
55.17
55.85
53.91
55.04
55.16
54,96
55.03
55,95
60.34
62.56
52.01
56.76
76.52
Qwea 728
50.37
48.72
49.74
49.76
53:49
54,99
48.56
Table 8: Judge reranking performance under the additive scale single-instance rating protocol on AlpacaEval, showing both non-length- controlled (LC) win rate and length-controlled win The non-LC values the those in Tab. 5.
Domain-specific prompts do not lead to judge improvements. Because the SFR-Judge family is trained to have flexible evaluation criteria, we experiment with prompting SFR-Judge-SB with domain-specific prompts for math reasoning and code generation benchmarks. We craft these prompts with an emphasis on correctness of output and validity of intermediate reasoning steps. The evaluation criteria for both prompts is shown in Fig. 19, with output format kept the same from the original prompt in Fig. 15. We show the impacts of this domain-specifie prompting in Fig. 23. Namely, performance drops on all benchmarks, though difference is statistically significant at 0.05. This demonstrates that the instruction following to reasoning domain gap discussed in Sec. 4.2 cannot be bridged with specialized prompting, which is more likely to be detrimental instead.
0.2
Math Domain Prompt
Math Original Prompt
Code Orig nal Promot
Code Demain-Specth Prompt
Normattee Helfoiness
6.1
0.0
-0.1
-6.3
GSN8K
MATH
CHAMP
HumanBrale
BigCoddBench
Figure 23: Normalized helpfulness of SFR-Judge-8B across all benchmarks, averaged across all applicable models. Domain-specific evaluation prompting performs worse in all cases.
Additional visualizations for single-rating protocols. The performance of judges based wudge-to-generatie ratio for single rating protocols is presented in Fig. 24. The positive effect for math is still present, but notably diminished compared to that of the pairwise protocol (Fig. 5), whereas the sizable instruction following advantage found in the pairwise case is eliminated.
Likert Rubrics
Additive Rubrics
0.5
0.5
Normalzzed hwwppunedhhlppumeee helpfumness
0.0
0.0
-0.5
-0.5
1.0
1.0
(n.s)
Codo coot. (ns)
Noth (n.s.
(*]
0.02 ins.)
-1.5
-15
10
10a
Judge Size Generater Size
Judge Size Generater Size
GSPBk
MATH
CHARI
HumonEval+
MBPP
BigCodzBench
AlpacaEval
IFEvOl
Figure 24: The effect of judge-to -oneerator size ratio on the normalized helpfulness for under single-rating reranking protocols.
Finally, we plot the distribution of the fraction of tied, top-rated responses for single-rating protocols in Fig. 25. Notably, under the additive rating protocol, the Prometheus models are consistently able reduce the number of top-rated responses compared to other models. Interestingly, this reduction in top-rated responses does not translate to improved average performance relative to other models, as shown in Fig. 7, which may suggest that the judge assessing based on surface- level factors. Other judges, for the most part, struggle even with finer-grained additive evaluation criteria, tending to assert a large fraction of responses as top-quality, regardless of domain.
26

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
0.4
Prom
Math Reasorinng
Code Generation
Instruction Following
Additive Rating
Lierrt Rating
Figure 25: The distribution fraction total responses as top-rated under single-rating reranking protocols.
B.2 Step-Level Beam Search
We present full results for our beam search experiments in Tab. As Mao et al. (2024) found out for the CHAMP dataset, models' full solution accuracy often much lower than their final answer accuracy. Furthermore, even state-of-ther LLMs such GPT-4 and Turbo struggle identify mistakes in intermediate steps. Thus, the oracle accuracy for math datasets, defined based final answer accuracy, is likely to be (severe) over-estimate of the true value. Such issues may also be present for code datasets, when solution contains bugs not identified by the test cases, though the issue may be less extensive we empirically find the trees to contain fewer leaf nodes (945 for math vs. 192 for code) and recent efforts in code evaluation have emphasized testing rigor (Liu et al., 2023a; Zhuo et al., 2024).
additionally report the performance of the Owen2.5-Math-i PRMM, a 7B PRM specialized for the math domain. Despite its specialization math, also offers competitive performance in coding domain tasks, largely outperforming comparably sized LLM-judge models in beam search.
27

---

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-asJJudges as Test-Time Scaling Evaluators
Lookahead
Prom 7B
SFR
SSC
8B
The
Prom SC7B
201
STE
Qves
Greedy
Rundom
Oracle
Llama 8D
/
x
85.67
83.62
59,86
59.84
90,90
58.70
87.19
87.34
85.05
85.75
89.31
89.61
85.44
83,70
92.12
94,09
91,96
92.12
89.25
90.85
93.78
94.47
87.11
52.34
99.47
Liana 708
x
96.47
94.60
95.07
95.91
95.15
95:60
95.45
95.91
94.16
94.47
95.45
95.91
96.51
96.51
95.15
96.89
96.97
55.38
94.25
99.52
GSMBI
Owen 32B
/
x
95.07
94.02
95.75
95.68
95.07
95.53
95.75
95.91
95007
94.24
95.22
96.21
95.07
94.54
95.07
96.36
95:40
96.21
95.38
94.84 84
97.12
96.97
95.15
95.03
99.01
Queen 72B
x
95.28
95.45
93.38
95.80
96.29
95.98
96.06
95,00
95.30
95.22
95.15
96.21
95.91
95.91
96.29
95.75
95.60
97.35
96.99
95.91
95.66
98.56
Owes-M
93.56
92.05
94.39
95.00
94.69
94.54
94.84
95.15
95.07
93.86
93.63
95.07
95.53
95.15
95:15
94.92
95.75
95.75
90.96
99.17
DpekMM 7B
X
79.45
53.55
53.85
81.25
80.29
83.62
81.27
89.16
88.02
56.13
92.34
81.65
79,0
99.00
80.67
$7.95
$7.04
84.31
80.82
87.64
83,70
91.96
90.30
56.70
92.12
lama 8R
22.05
28.32
24.77
23.19
22.96
26.51
24.55
34.14
29.38
27.79
42.60
93.28
24.09
29.91
27.34
25.76
24.70
29.46
25.68
40.45
32.63
48.56
Jana 708
X
43.05
43.06
49.40
44.18
46.75
45.09
44.34
45.69
63,66
42.52
48.26
46.37
44.11
44.41
47.95
53.10
44.34
48.19
45.77
44.86
60.50
55.36
43,20
41.75
94.71
MATI
Qwen 32B
x
$4.00
52.72
56.87
54,61
54.31
52.19
56.57
53.47
59.14
61.71
57.55
54.46
68.
66.01
57,10
53.54
93.50
Qwen 72B
61.78
62.24
65.26
64.65
65.63
61.33
62.61
65.18
65.48
64.20
62.39
66.09
68.20
65.63
66.92
56.00
70:92
72.89
62.99
61.35
93.83
Qwen M. 7B
x
63.50
65.18
67.37
68.73
68,58
68.15
66.39
65.01
67.32
65.86
68.58
70.00
66.54
66.77
74.09
64.95
68:96
70.69
67.00 60
70.69
72.28
76.21
05.79
$7.32
93.81
Dpsk M
16.34
16.54
19.41
24.62
18.56
17.45
24.70
21.30
25.76
32.93
24.62
28.55
26.06
20,92
34.29
39.55
19:03
17.08
85.20
Liana 88
x
27.78
30.37
37.41
35.93
30.37
36.30
34.81
34.07
28.15
28.52
35.93
36.67
28.89
31.11
33.86
43.33
38.39
38.52
35.19
53.70
39.63
44.81
28.52
28.97
98.52
Liana 708
46.67
43.33
$2.59
53.70
51.48
58.52
54.07
52.59
50.37
51.11
48.52
47.04
45.19
54.81
54.81
$5.93
$2.96
52.96
50 74
62.59
58.15
46.76
100.0
CHAMP
Qwen 30B
x
69.63
68.89
70.74
72.22
71.85
72.22
74.07
74.07
72.59
75:33
69.63
73.33
70.74
70.00
74.44
75.15
74.07
75.56
71:11
72.22
74.44
7W.52
72.96
72,
97.04
Qwen 72B
x
69:26
73.50
69.26
71.11
71.85
72.96
70.74
72.59
69.26
69.63
74.81
74.81
15.93
72.96
73:33
72.96
74.42
75.56
77.04
72.96
71.76
97.78
Owen-M
$6.67
66.67
64.44
64.44
63.70
64.07
62.96
73.20
67.41
69.63
70.37
75:03
64.45
96.67
Dpek M 7B
x
24.07
32.96
28.52
31.48
29:26
28.15
27.41
41.11
33.93
30.67
44.44
27.41
28.08
95.93
22.96
35.93
31.11
37.41
33.33
36.30
28.15
44.07
38.39
57.78
50.37
Llama 8B
$7.93
63.24
59.76
63.41
57.32
62,80
60.37
65.85
61.59
65.24
60.37
59.15 15
57.55
$0.00
58.54
62.20
60.37
62.20
59.15
65.85
59.15
63.85
65.85
65.24
67.00
Llama 708
X
68:29
70.12
73.78
73.17
75.00
76.83
73.17
70.12
71.95
75.17
71.34
73.78
71.95
69.51
74.39
70.73
73.78
77.44
76.83
73.17
77,64
76.83
74.39
71.87
92.68
HomonEoke
Qwen 32B
x
82.32
$2.32
54.76
81.71
83.54
84.76
85.98
81.71
79.88
85.37
87:00
85.98
84:76
85.37
81.71
83.54
62.32
$1.94
95.12
Owen 728
81.10
79.88
79.27
$1.21
50.49
$1.10
81.10
81.10
81.71
81.21
M2.32
82.93
30.49
81.10
82.32
82.32
N2.32
82:93
82.93
52.32
82.93
81.71
50.49
30.38
92.07
Qwen-C77B
x
78.05
78/66
79,88
78.66
51.71
$1.10
81.71
81.71
83.54
78.66
84.15
81.10
81.10
78.66
81.71
79.85
82:01
62:32
79.27
82.92
85.98
82.32
84.15
30.12
95.12
Dpsk
63.41
71.95
69.51
68.0
70.73
64.63
76.83
70.73
71:95
73.17
76.22
70.73
64.94
90.85
Llana 88
x
58.99
99.52
60.58
61.11
60005
5N.20
56.08
59.79
64.81
62.17
61.11
59.52
57.12
84.13
55.82
55.29
00.32
02.96
58.73
57.14
53.44
61.38
64.51
62.70
61.90
Liana 708
62,70
60.22
62.43
66
66.40
64.81
64.02
61.38
65.08
64.55
60.05
61.38
64.55
64.81
67.99
65.87
66.40
68.25
66.14
63.64
88.62
MBPP
Qwen 30B
X
75.66
74.34
77.25
77.51
74.60
74.34
76.72
75.40
76.46
75.13
75:03
75.13
76.72
74.00
78.04
77.51
77.78
76.46
76.72
76. 76.19
75.93
75.66 66
74.51
88.62
Qwen 73B
x
75,93
76.19
76.72
75.16
77.25
77.25
75.93
77.78
76.98
77.25
77.25
76.46
75.68
86.24
76.72
76.3%
76.46
76.38
76.46
77.51
75:93
76.98
75.40
Owen 7BB
70.11
74.07
73.61
74.87
73.02
74.07
64.70
90.21
68.52
71.17
71.95
68.78
60.84
69.58
69.31
73.54
74.60
71.43
70.37
Dpsk-C6.70
x
61.90
60.58
64.81
62.17
61.11
64.81
63.76
61.64
60:32
63.76
62.96
64.51
63.76
61.11
67.72
63,76
64.55
66.93
67.72
64.55
65.61
64.81
66, 40
61,33
$1.66
Ilama 8B
29.04
30.44
31.32
31.67
29.36
30.79
28.51
33.96
34.12
33.51
32.81
88
06.58
26.40
33.51
31.93
30.00
32.57
28.08
36.05
34.04
35.09
33.68
BleCoeellachh
Llama 708
/
X
39.56
40.00
41,75
40.95
41.58
44,47
42.59
42.72
43.77
45.16
41.40
42.37
41.05
30.64
42,46
42.28
44.82
43.68
43.42
42.98
42.46
42.37
43.07
41.98
67.11
Qwen 32B
x
43.33
43.68
45.09
45.61
45.79
45.35
45.18
44.82
45.51
45.70
45.61
46.45
45.79
45.18
45.44
44.47
44.26
69.47
Qwen 728
/
x
45.61
45.35
46.25
46.75
46.84
46.5%
67.28
46.40
45.14
46.11
45.70
45.88
46.32
46.32
46.54
46.75
45.79
46.32
46.14
45.96
45.79
46.03
63.86
Qwen-C77B 7B
/
x
35.96
37.11
40.44
40.18
41.84
40.70
41.14
41.14
80.35
38.07
40.79
39.30
38.16
18.07
42,46
40.79
43.07
41.91
41.58
40.44
42.57
42.02
41.14
38,29
71.09
Dpst-C.6
30.61
35.00
36.23
36.49
35.44
35.00
31.42
65.00
29.91
34.04
35.44
35.00
33.42
34.21
32.11
36.93
36.14
37.81
34.74
Table 9: Performance of the final selected node via the sudge-guided beam search procedure.
28

---

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-assJudges as Test-Time Scaling Evaluators
B.3
Critique-Based -aased Refinement
We present full results for our critique-based refinement experiments in Tab. 10. Here, we report the judge selected response after reranking all generated responses, as well as the performance after just taking the last generated response.
Pef.
Lox
Poens
Real
Osc
Posi
Last
3PR
Band
Ocac.
Pes.
Losk
Tlax 5B
Orec:
Per..
Lds.
12B SFR
Resl
Ose.
Par.
Laxt
Baai
Ona..
Perf
Lest
SFR R12
Road.
Orac.
Grondy
Ono, .R
19bb Ataaahal Notaaa 3000 taxabblk CHAMP MATi 03b1e
Liara 110
15.07
94.37
15:23
$3.00
05.45
98.40
16.33
35.m
15.37
$4.56
16.33
15.00
94:31
15.23
$3.51
16.22
$3.22
18.39
15.53
15.45
15.80 0/
08.90
15.53
93.11
18.45
228
10.07
08.35
08.01
$3.55
05.23
03:22
05.22
10.22
23.38
00:35
15.16
05.45
94.54
08.00
10.00
$3.38
98.3
$3.07
05.39
10.22
03:22
10.30
03.27
08.43
16.22
93:22
99.50
Owes
95.75
45.08
95.65
$5.85
95.68
65
45:6
95.88
45.53
96.53
95.56
65
99.53
45:33
95.63
65.25
95.08
55.68
45.18
95.08
95.25
05.3
95.68
45:45
97:88
25.90
34.63
25:12
33.60
25.33
35.40
25.06
25.05
36.56
26.99
25.85
25.93
25.44
25.41
25.54
18.76
25.45
25.56
25.34
27.42
29:3
28.07
27.74
32.18
24.20
23.66
53.17
Liara 7141
43.90
04.38
64:87
15.17
44.00
46:35
44.00
+4.41
13.98
$3.90
13.00
45.08
46.18
40.00
43.73
15:30
56.13
43.88
43.83
14.71
61.94
04.04
64.25
65:38
43.81
43.07
66.25
Qwex 2:38
57.02
57:35
57.03
58.31
$2.10
57.80
10
$7.10
$3.55
$7.20
57.36
58.53
36.90
56.5
56.29
58.76
56.97
55.72
56.99
57.76
5816
59.01
57.37
14
57.10
$2.34
78.17
12200
62:90
53.14
60:07
60.10
63.00
02:52
63.99
$0.78
Liana
20:03
30.00
3043
16.30
27.18
27.41
23.00
38,99
27.50
21.41
27.5%
31:11
21-26
25.89
25.41
15.20
26.57
25.07
27.39
12.22
15.30
35.05
12.26
41.85
25.25
27.50
6009
Liaaa 100
48.15
47.38
47.32
33,00
40.26
48.50
46.15
57.76
41.11
47.1
43.41
48.55
45.75
46.51
4633
$3.56
46.3
41.75
47.59
51.11
45.32
46.32
41.33
52.22
47.41
45.07
71.45
Q2c 128
13.33
73.33
75.35
15.19
14.40
15.19
15.19
Clara 18
59.76
55.54
04.35
0341
6423
2410
64.33
60:77
5341
63:11
53.11
0341
2615
56.34
20.99
65.24
62.9
6130
02.80
6341
5530
60.45
5441
07.98
10:35
35.70
70:56
Mara
11.34
71.34
7385
15.1
76-23
13.00
13.41
75-38
7611
12-27
25:1
35.0
1361
75A1
13.00
77.44
75.11
11.33
Qwee
62.93
42.33
82:47
$3.34
83.54
$3.54
02.99
14.36
13.71
81.31
$2.01
92.33
81.71
91.31
11.9
14.76
82.32
$2.32
$2.02
12.88
$3.56
12.13
$3.54
12.52
$2.13
90,90
11:32
51.00
62.57
16.38
17.26
19.36
1162
54:26
16.49
63:11
54.33
51.9
5538
1608
54.9
$1.50
59.31
120
76.34
24.09
76.31
15.55
70.55
73.44
15.75
75.06
75.00
T161
71.00
19.31
73.54
71k1
76.19
5434
15.81
54.55
77.25
15.00
75.35
$4.15
Oses 730
75:66
75:40
75.84
71.25
70.37
69:58
71.12
78.57
16.46
76.46
76.33
76.46
1063
20.63
72.17
15.57
75.1
75.15
75.55
76.72
7381
14007
14.45
76.04
76.19
75.71
$4.96
11wa 700
25.08
-5:18
06:11
4385
17.14
15.75
15.30
11:02
10:33
03.33
19:07
13:13
44.50
15.0
15.35
16
66:0
15.14
41.00
Qnc 128
45.09
43:18
45.27
46.16
43.09
44.01
43.06
47.63
45.55
45.53
45:55
45.33
44.21
46.21
46.07
45.05
44.55
44.30
44.53
45.29
45.70
4559
45.00
47.35
45.33
44.66
65.18
Qkes 728
46.38
45.57
46.33
41.10
44.59
44.11
45.10
47.55
46.74
46.75
46.74
46.55
46.14
4655
46.25
41.60
43.50
43.61
46.10
47.11
47:25
47.11
46.55
45.15
46.07
45.30
60L1S
50.30
27.54
36.2
78.56
58.34
18:66
19:35
15:11
19.55
10:23
25.00
38:00
17.36
75.00
Larra 110
37.00
36.35
36.21
41:49
36.19
15.23
33.85
36.32
15:04
38.15
33:30
36.54
15.01
35.25
15.75
31.6
35.43
35.46
33.02
36.77
15:37
34.79
15.52
36.55
15.45
34,27
61:M1
22B
37.14
37.00
35.33
1967
52.35
12.26
32.92
55.02
12.16
52.13
32.75
15.55
53.27
33.44
35.33
34.66
32.35
11.9
33.30
35.85
13.75
33.57
12.34
5433
32.92
33.00
61,91
Qwes
56.21
33.83
35.52
36.97
32.46
31.00
31.22
35.34
32.66
32.31
$1.35
32.39
12.57
51.53
5338
$3.14
53.19
32.30
53.36
51.65
51.75
53.48
55.01
35.76
76.32
Liana
6765
65.39
0664
31.15
74.6
11:36
71.6
76.65
11:00
75.30
72.16
16.09
75.05
73.06
10.70
3161
73.66
71.27
75.29
76.52
76.50
14.72
76.01
12.51
05.13
67.91
Liara 710
10.30
82.44
15.90
87:28
54.10
8336
81.89
17.90
H4.5
14.77
$4.39
13
84.0
16.53
15:00
53.82
$4.29
$4.13
15/03
15.06
15.21
1430
15.77
14.3
74.4
12-42
7607
29.35
7941
0260
12.36
11
15.40
80.30
1257
81.52
31331
1002
12:00
7687
3938
8138
02:44
12.2
11.32
81:1
10.22
71.38
10022
Table 10: Critique-based refinement result for the (1, 9)-refinemmet setup. Random and Oracle for each judge are computed from the greedy seed response and nine responses generated during refinement. Random-RR and Oracle-RR are computed from the greedy seed response and nine sampled ones used in the reranking experiments.
Graphical illustration. Fig. 26 graphically depicts the refinement process, compared to the reranking process, and illustrates the definition of various metrics used in Sec. 4.4.
(Re
Pgreedy
Refinement
Plast
(Ref) Pi P'udge
(RR) P) jjudge
Judge Random
(RR) P Prand.
(Ref) di Prand.
Y Oracle
(Ref) P Porac.
(RR)
/ Porac.
Pigraeyy
Reranking
Figure 26: Illustration various metrics, all aggregated over the dataset. Pereedy is calculated on the greedy response (seed in refinement (Ref) on the last refined response. The judge, a random selector and an oracle selects respective responses
and first in reranking), and Plaes from each refinement and reranking process and compute their metries.
Qualitative case study. Here, we provide examples of ineffective judge model critiques. We analyzed over 100 critique- response pairs and found two primary failure modes that point to lack of judge actionability: (1) False positives due to judges focusing on surface-level details rather than catching critical errors or (2) False negatives due to judges over scrutinizing stylistic errors over substance. Examples of false positive and false negative failure modes are presented in Fig. 27 and Fig. 28, respectively. In both cases, judges fail to assess based on validity and correctness of the response, offering feedback that does not appropriately progress or stop the refinement cycle.
29

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLMMassJudges as Test-Time Scaling Evaluators
Dataset: MATH
Generator: Qwen2.5-72B-Insstee Instruct
Judge: SFR-Judge-70B
Retinement Round:
Type of Error: False positive. Generator produces the wrong output, but the judge does not catch the mistake.
Question:
The 5 x 5 grid shown contains a collection of squares with sizes from 1 x 1 to 5 x 5. How many of these squares contain the black center square?
Generator Output:
## Step I: Identify the sizes of squares that contain the black center square The black center square is located at the center of the x 5 grid. We need to count the number of squares of different sizes that can contain this center square.
### Step 2: Count the number of squares of each size that contain the center square
**1 x squares**; Only the center square itself contains the black center square. There such square. ##2 x 2 squares**; The center square can be part of different 2 squares (one for each possible position of the top-left corner of the 2 x square that includes the center).
**3 x 3 " The center square be part of 4 different 3 squares (one for each possible position of the
top-left corner of the 3 x 3 square that includes the center).
4 x squares**; The center square can be part of 4 different 4 x squares (one for each possible position of the
top-left corner the 4 x 4 square that includes the center).
#*5 x squares**; The entire x grid is the only 5 x square, and it contains the center square. There such square.
## Step Sum the number of squares of each size
1+4+4-411=
Therefore, the final answer is: 14 I hope it is correct.
Judge Critique:
The response precisely executes the instruction by following the required format for complex problems and providing clear step-by-step solution. The explanation and calculations concise and easy to follow. The response also accurately counts the number of squares of each size that contain the center square and correctly sums them up to obtain the final answer. The conclusion is also in line with the instruction.
Ground- Truth Solution:
All of the squares of size x 4, and 3 contain the black square and there are
12 +3 =
of these. In addition, 4 the 2 x 2 squares and of the x squares contain the black square, for total of 14 4
19
Figure 27: Example of false positive judge critique. The judge focuses surface-level details, such as ease of reading, rather than verifying the logical errors in the model response. As result, the refinement process stops without correcting the mistake.
30

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Dataset: MATH
Generator: Llama-3.1 3.1-88-IInstrue
Judge: SFR-Judge-700
Refinement Round: 1-6; shown
Type Error: False negative. Generator produces the correct output, but the judge overemphasizes details irrelevant to response correctness.
Question:
In the diagram below, have sin LRPQ What is cos ZRPS? [Tikz omitted]
Generator Output:
## Step I: We are given that sin <RPQ 25
and we need to find cos LRPS.
Step 2: To approach this problem, we can use the relationship between sine and cosine functions in right-angled triangle.
25 ## Step 3: Since sin ZRPQ
7 can use the Pythagorean identity to find cos LRPQ.
## Step 4: The Pythagorean identity states that sin? + cos² = for any angle 0.
Step 5:
Therefore, we can find cos LRPQ by rearranging the Pythagorean identity: cos? LRPQ sisin ZRPQ.
## Step 6: Substituting the given value of sin LRPQ
get cos? LRPQ (25
(
49
576
625
625
## Step Taking the square root of both sides, find cos LRPQ
576
626
25
## Step 8: Now, need to find cos LRPS. Since LRPS is the supplement of ZRPQ, the relationship
cos(0 + 180%) co
## Step 9: Therefore, cos LRPS = COS LRPQ =
24
25
24
Therefore, the final answer is:
25
Judge Critique:
The response precisely executes the instruction by providing a step-by-step solution to the problem. It clearly explains each step and uses the required format. The response also accurate, as correctly applies the Pythagorean identity and the relationship between sine and cosine functions right-angled triangle. Additionally, the response harmless and does not contain any unnecessary information. However, the response contains more steps than necessary, as the problem be solved in fewer steps.
Ground- Truth Solution:
For any angle we have cos(180 - x) = -os so cos LRPS = cos(180 - -RRPQ) = - COs LRPQ.
Since sin2 LRPQ + cos2 <RPQ 1, we have cos² LRPQ
(2))
576
Since LRPQ acute, we have
625
24
cos / RPQ 2 which gives LRPS
/RPQ
25
Figure 28: Example negative judge critique. The judge focuses on surface-level details, such as response length, ignoring the fact that the outcome correct.
31

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLM-assJudges as Test-Time Scaling Evaluators
C Tables of Normalized Results
Table 11-15 presents the normalized helpfulness and refinement ratio statistics for reranking, beam search and refinement.
Prom 7B
SFR 8B
SC SB
OB 8B
SFR 12B
Prom 8x78
SPR 700
30B
STE 708
1aama 8B
Best RM
Greedy
Random
Oracle
Maj
Llana SB
-0.012
0.317
0.296
0.014
0.338
-0.012
0.747
0.528
0.409
0.134
0.136
0.000
-0.333
1,000
0.179
Llama 708
0.102
0.129
0.153
-0,105
0.051
-0.285
0.258
0.359
-0.207
0.000
0.332
0.000
-0.142
1,000
0.437
GSMRK
Qwen 32B
-0.066
0.296
0.093
0.138
0.114
-0.114
0.207
0.159
0.000
-0.021
0.251
0.000
0.000
1,000
0.251
Owen
-0.277
-0.036
0,032
-0,209
-0.173
0.000
0.241
0.032
-0.209
0.277
-0.073
1,000
Owen: M. 7B
-0.168
0.110
0.139
-0.168
0.165
-0.249
0.191
0.191
0.029
-0.139
0.191
0.000
-1.282
1,000
0.219
Dpsk-M 7B
0.396
0,448
0.005
0.740
0.422
0.218
0.471
-0.084
Llama 8B
-0.022
0.184
0.817
0.005
0.184
0.016
0.417
0.178
0. 0.184
0.155
0.178
0.000
-0.095
1,000
0.291
MATH
Llama 70R
Owen 32B
00.11
0.175
0.093
0,139
0.039
.0.68
-0.84
0.140
0,141
.0.097
0.080
0.336
0.330
0.129
0.118
0.234
0.163
0.043
0.254
0.106
0.000
-0.071
-0.231
1,000
1,000
0.354
0.262
Owen 72B
-0.34
0.145
0.061
-0,088
0,103
-0,034
0.263
0.195
0.221
0.118
0.153
0.000
-0.062
1,000
0.317
Qwen-M M 7B
0.028
0.269
0.099
-0.066
0.241
0.049
0.385
0.247
0.7
0.241
0.159
0.000
-0.808
1,000
0.280
Dpak- M 7R
00093
0.172
0.098
0.084
0.101
0:045
0.386
0.217
0 0.267
0.122
0.207
0.000
-0.057
1,000
0.346
SB
0.036
0.060
0.156
0.132
0.036
0.084
-0.064
L,000
CHAMP
Llama 70B
-0.66
0.169
0.138
0.046
0.061
-0.077
0.031
0.200
0.015
0.046
0,200
0.000
-0.056
1,000
Qwen
-0.393
-0.750
-0.215
-0.649
-0.608
-0,465
40.608
-0.465
40.715
-0.375
-0.436
1,000
Qwen 72B
-0.394
-03668
-0.105
.0,342
-0.210
-0.237
.0.289
-0,079
-0.263
-0.210
-0.197
0.000
-0.310
1,000
Owen M 7B
0.118
0.392
0.275
0.216
0.255
0.236
0,510
0.373
0.392
0.196
0.373
0.000
-0.884
1,000
Dosk-M 7B
0.016
0.355
0.266
0.177
0.282
-0.016
0.363
0.298
0.290
0.097
0.331
0.000
0.051
1,000
Llana SB
00,886
-0.291
0,041
-0.106
-0,143
-0.475
0.262
0,188
0.011
-0.106
0.225
0,000
-0,402
1,000
Hunsobvale
MJama 7OR
00.560
-0.280
0.160
0.001
0.119
-8.320
0.000
0.240
0.120
0.000
0.200
0.000
-0.156
1,0000
Owen 32B
-0.100
0.100
0.000
0.250
0.100
10,100
0.000
0,200
0.000
0.000
0,350
0.000
0.120
1,0000
Owen
0.211
0,316
0.053
-0.053
0.369
0.000
0.473
-0.016
1,000
Qwee-Cr C 7B
-1.585
-0.581
-0.751
-0.918
-0,581
-1.168
-0.334
-0.117
-0.501
-0,581
-0,584
0.000
-0.691
1,000
Dpsk-C6.77
0.208
-0.292
-0.292
-0.375
0.167
-0.167
0.375
-0.296
Llama SB
-0.113
.0.012
0.133
0.204
.0.997
-0.241
0.108
0.373
0.349
0.096
0.289
0.000
0.032
L,000
Llama 70B
-0.51
-0.353
-0,041
0,014
-0,500
-0,500
-0266
0.235
0.059
0.014
0,184
0.000
-0,078
1,000
MBPP
Qwen 32B
00182
0.061
0.061
0.121
-0.061
-0.182
0.000
0,302
0.000
-0.031
0.076
0.000
0.006
1,000
Qwen 72R
-0.88
0.063
0,032
0.000
40.125
-0.093
0.125
0.091
0.082
0.218
0.063
0.000
-0.057
1,0000
Owen
-0.192
.0.115
-0.115
.0,346
-0.346
0.058
0.250
0.116
-0.019
0.039
-0.376
L,000
Dpsk-C 6.7B
00.60
-0.60
-0,380
-0.360
-0.640
-0,660
0.420
-0,160
-0.320
-0.020
-0,020
0.000
-0.304
L,000
IPEval Alpocaboal BigCodBBoshh
Llama SB
0.031
0.063
0,042
-0.017
-0.133
0.174
0.181
-0.052
0.136
-0.145
L,000
Llama 708
00,291
-0189
-0.148
-0.184
-0.219
-0,316
-0.163
40.112
-0.107
-0.123
-0,090
0.000
-0.205
1,0000
Qwen 32R
0.036
-00014
-0002
-0.085
0.111
0.058
40.018
0.036
-0.044
1,000
Owen 72B
-0.655
0.013
0.058
0.090
0.071
-0.072
0.064
0.045
0.071
.0.065
0.143
0.000
-0.027
L,000
Owee- C 7B
-0.115
00.110
-0.137
-0,059
-0.128
-0,210
0.050
0.078
0.014
-0.114
0,000
0.000
-0.233
1,000
Dpsk-C 6.7B
-0.109
0.051
0.146
0.109
0.095
-0.054
0.219
0.215
0.183
0.080
0.170
0.000
-0.101
1,000
Llama SB
0.276
0.356
0.350
0.251
0.456
0,361
0.478
0.381
0.373
0.170
0.421
0,000
-0.017
1.000
Llama
0.356
0.431
0.386
0.178
0.363
-0.946
L,000
Qwen 32B
0.391
0.379
0,340
0.293
0,517
0.406
0.154
0.373
0.479
0,310
0.426
0.000
0.100
1,000
Qwen
0.417
0.347
0.477
0.274
0.194
Llama 88
-0.371
-0.173
-0.185
-0.062
<0,062
-0.259
0.124
-0.100
-0.049
-0. 40.198
0.012
0.000
-0.337
1,000
Llama 70B
-0.273
-0.319
-0.432
-0.523
-0.319
-0.523
40.100
-0.250
-0.100
-0.501
-0,205
0.000
-0.379
1,000
Owen 32B
0.039
0.151
0.019
-0.076
00.188
0.000
0.245
0.245
0,208
-0.038
0.220
0.000
-0.080
L.000
Qwen 72B
-0,067
0.131
0,108
0,021
0,152
0.000
0.174
0,218
0.0M
0.195
0,293
0.000
-0.015
1,000
Table 11: Normalized helpfulness of judge reranking (pairwise protocol), computed from raw data presented in Tab. 3.
32

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Prom 7B
SFR SB
Thm 8B
STR 12B
Prom 8x7B
SFR 70B
Llama 8B
OBxи SB
SEN 8D
Spm 27B
Greedy
Random
Oracle
Maj
Llama
-0.125
0.110
-0.48
0.045
0.247
-0.090
0.289
-0.333
0.479
Llama 70B
0.092
0.081
.0.051
0.024
00007
00,200
0.020
0.115
0.180
0.332
0.000
-0.142
1.000
0.437
CSMIK
Qwen 328
0.009
0.129
-0045
0.936
0.024
0,075
0.039
0.251
0.207
0.171
0.000
0.000
1.000
0.251
Qwen 728
0.077
-0,059
-0.123
-0,045
-0055
-0.027
0.009
0.068
0.277
0,205
0.000
-0.073
1.000
0.277
Qwen-M
-0.524
0.077
-1.205
0.073
-0.322
0.154
-0.150
0.110
0.139
0.194
0.000
-1.282
1.000
0.249
Dpsk-M 78
-0.005
0,122
-0.094
0,124
-0,008
0.423
0.020
0.389
0.464
0,471
0,000
-0.084
1.000
0.448
Llama 8D
.0.013
0.069
-0.118
0.057
0.002
0.164
.0.013
0.158
0.130
0.178
0.000
.0.095
1.000
0.291
Llama 70B
0.051
0,076
-0.090
0.099
0.037
0.095
0.054
0.129
0.160
0,254
0.000
-0,071
1.000
0,351
MATH
Qwen 32B
10.10
0,122
-0.237
-0.660
0.149
0.100
-0.097
-00544
0.106
0.032
0.000
-0.231
1.000
0.262
Qwen 728
-0.013
0.031
-0.070
0.009
-0.077
0.040
-0.008
0.056
0.153
0.149
0.000
-0,082
1.000
0.317
Qwen-M 78
-0.769
0.070
-0.737
0.068
-0,340
0.101
-0,164
0.107
0.096
0.159
0.000
-0,808
1.000
0.280
Dpsk-M
-0.050
0,109
-0.082
0.041
0.218
0.057
0.180
0:207
0.202
0.000
-0.057
1.000
0.246
Llama 8B
-0.069
-0.141
-0.028
0.031
0,016
0.075
0.055
0.0841
0.024
0.048
0.000
-0.064
1.000
CHAMP
Llama 70B
-0.016
0.120
-0.017
0.064
0.138
-0055
Qwen. 328
-0.844
-0.531
-0.422
-0,398
-0.472
-0:122
-0.562
-0.536
-0.626
-0,375
0.000
-0.136
1.000
Owen 72B
00464
-0.441
-0.288
-0,290
-0.319
-0.212
-0.237
-0.197
0.000
-0,310
Qwen-M 7B
0.080
0,155
0.183
0,228
0.066
0,290
0.165
0.255
0.373
0.275
0.000
-0.884
1.000
Dpsk-M
0.009
0.120
0.062
0,146
0.092
0.279
0.123
0.286
0,331
0.185
0.000
0.054
1.000
LJama 88
-0.287
-0.109
-0.382
-0,114
40,355
0,092
-0.351
-0.180
0.059
0,225
0.000
-0,402
1.0000
Hummmeeall
Llama 708
-0.056
-0.179
-0.130
-0.160
-0.147
-0.0%
-0.116
0.120
0,200
0.120
0.000
-0.156
1.000
Qwen 32B
0.148
0,016
0.098
0,117
0.115
0.308
0.057
0,300
0.050
0,350
0.000
0,120
1.000
Qwen 72B
-0.027
0.275
.0.013
0.189
0.019
0.198
-0.039
0.158
0.211
0.473
0.000
.0.016
1.000
Qwen-C
-0.654
-0.635
-0.659
-0.653
-0.400
-0274
-0.655
-0.668
-1.001
-0.584
0.000
-0.694
1.000
Dpsk 6.7B
-0.273
0.135
-0.241
-0.059
0.270
0,083
-0.156
0.042
0.125
0.375
0.000
.0,296
1.000
Llama
0.102
40.014
0.066
0.283
0.289
0.000
0,032
1.000
Llama 708
-0.009
-0.264
-0.099
.0.182
0.046
-0.004
-0.019
-0.029
0.184
40.103
0.000
.0,078
1.000
MBPP+
Qwen 328
-0.053
0.002
0.022
0.076
Qwen 72B
0.014
0.090
-0.014
0.043
-0.009
-0.021
0.058
0.063
-0.110
0.032
0.000
-0.057
1.000
Qwen-C
.0.273
0.288
-0.328
-0.175
-0.554
-0.172
-0.019
0,039
4.009
0.376
Dpsk-C6.78 6.78
-0.293
-0.258
-0,265
-0,301
-0.277
-0,320
-0,197
-0.190
-0.020
-0,600
0.000
-0,304
L.000
BigCoeeeench
Llama
.0.079
.0.04
-0. 108
0,913
0.101
-0.049
0.131
0.136
0.000
.0.145
Llama 708
-0.179
-0.156
-0. 168
-0,149
<0.158
-0.124
-0.157
-0.105
-0.090
-0,092
0.000
40,205
1.000
Qwen 32B
-0.026
0.004
.0042
.0,005
00.31
0,014
0.006
0.026
0.027
0.036
0.000
.0,044
1.000
Qwen 728
0.077
0.019
-0.025
0,044
-1,022
0.058
-D003
0.166
0.143
0,123
0.000
-0,027
1.0000
Qwen-C
-0.196
-0.126
-0,222
-0.105
-0.199
0,008
-0.150
-0.669
0,000
-0.016
0.000
-0,233
1.000
6.77B
-0.029
00005
-0.077
0,031
-0,039
0.156
0.017
0,080
0.170
0.164
0.000
-0,101
1.000
1FEval Alpucclivvll
Llama
0.012
0.097
0.019
0.081
0.014
0,078
0.004
0.229
0,316
0.421
0.000
-0.017
1.000
Llama 70B
0.025
0.052
-0.018
0.040
0.001
0.005
-0.056
0.137
0.249
0,363
0.000
-0.046
1.000
Qwen 328
0.145
0.145
0.121
0.149
0.113
0.126
0.106
0.221
0.330
0.426
0.000
0.100
1.000
Qwen 72B
0.098
0.112
0.073
0,122
0.083
0.103
0.084
0.161
0.340
0.430
0.000
0.194
1.000
Llama 88
-0.333
-0.166
-0.247
-0.147
-0.246
-0.069
-0.162
-0.025
-0.118
0.012
0.000
-0,4428
1.000
Llama 70B
-0.250
-0,218
-0.250
-0,246
-0.535
Owen 32B
0.018
-0.003
0.008
-0.052
-0.034
0,247
0.074
0.220
0.019
0.122
0.000
-0,290
L.000
Qwea 728
0.020
0.009
0.152
0.287
-0,010
0.000
-0.243
Table 12: Normalized helpfulness of judge reranking (single-rating protocol with Likert scale prompt), computed from raw data presented
Tab. 4.
33

---

Evaluating Judges Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Prom 7B
SFR SB
Thm 8B
STR 12B
Prom 8x7B
SFR 70B
Llama 8B
OBxи SB
SEN 8D
Spm 27B
Greedy
Random
Oracle
Maj
Llama
-0.222
-0.200
-0.331
-0,032
0,078
-0.063
0.289
-0.333
0.479
Llama 70B
0.034
0.085
0.000
0.061
0.119
-0.60
0.051
0.115
0.180
0.332
0.000
-0.142
1.000
0.437
CSMIK
Qwen 328
0.012
0.075
-0075
0,930
0.051
0,015
0.811
0.251
0.207
0.171
0.000
0.000
1.000
0.251
Qwen 728
-0.050
-0,059
-0.241
-0.068
40,005
-0,082
-0.250
0.068
0.277
0,205
0.000
-0.073
1.000
0.277
Qwen-M
-1.297
0.037
-1.300
-0.044
40.002
0.070
-00337
0.110
0.139
0.194
0.000
-1.282
1.000
0.249
Dpsk-M 78
-0.071
0,168
-0.069
0,016
0.099
0.299
0.026
0.389
0.464
0,471
0,000
-0.084
1.000
0.448
Llama 8D
-0.066
0.016
-0.122
0.044
0.025
0,079
.0.004
0.158
0.130
0.178
0.000
.0.095
1.000
0.291
Llama 70B
-0.019
0,077
-0.052
0,100
0.018
0.048
0.090
0.129
0.160
0,254
0.000
-0,071
1.000
0,351
MATH
Qwen 32B
40.42
0,124
-0.251
.0.992
0.095
0.118
-0.124
-0054
0.106
0.032
0.000
-0.231
1.000
0.262
Qwen 728
00065
40.001
-0.088
-0.008
0.005
00027
-0.025
0.056
0.153
0.149
0.000
-0,082
1.000
0.317
Qwen-M 78
-1.185
0.049
-0.852
0.075
10.39
0.074
-0.186
0.107
0.096
0.159
0.000
-0,808
1.000
0.280
Dpsk-M
-0.011
0.049
-0.077
-0.007
0.133
-0.001
0.180
0.207
0.202
0.000
-0.057
1.000
0.246
Llama 8B
0.017
-0.048
-0.017
0.091
0.076
0.062
0.047
0.084
0.024
0.048
0.000
-0.064
1.000
CHAMP
Llama 70B
0.101
-0.009
0,000
0.138
-0055
Qwen. 328
-0.535
-0.41
-0.419
-0.451
-0.155
-0.441
-0,370
-0.536
-0.626
-0,375
0.000
-0.136
1.000
Owen 72B
0.295
.02222
-0.348
-0,308
-0,249
-0.362
-0.237
-0.197
0.000
-0,300
Qwen-M 7B
-0.022
0.272
0.222
0,208
0.224
0.237
0.091
0.255
0.373
0.275
0.000
-0.884
1.000
Dpsk-M
0.027
0.168
0.067
0,121
0.072
0.236
0.055
0.286
0,331
0.185
0.000
0.054
1.000
LJama 88
-0.364
40,292
-0.387
-0,172
41,337
-0,097
-0.227
-0.180
0.059
0,225
0.000
-0,402
1.0000
Hlmmmeeall
Llama 708
-0.124
-0,140
-0.146
-0.181
-0.149
-0.110
-0.209
0.120
0,200
0.120
0.000
-0.156
1.000
Qwen 32B
0.160
-0.171
0.117
0,121
0.123
0.116
-0.008
0,300
0.050
0,350
0.000
0,120
1.000
Qwen 72B
.0.003
0,147
.0.044
0.196
0.022
0.111
0.109
0.158
0.211
0.473
0.000
.0.016
1.000
Qwen-C
-0.633
-0,001
-0.637
-0.631
-0,888
-0.575
-0.851
-0.668
-1.001
-0.584
0.000
-0.694
1.000
Dpsk 6.7B
.0.383
0.213
.0.273
-0.158
0.273
-0,027
-0.426
0.042
0.125
0.375
0.000
.0,296
1.000
Liama
0.163
0.055
0,103
0,265
0.289
0.000
0,032
1.000
Llama 708
-0.013
.0.366
-0.066
.0.166
0.053
-0044
-0.149
-0.029
0.184
40.103
0.000
.0,078
1.000
MBPP+
Qwen 328
-0,002
-0,076
0.076
Qwen 72B
0.032
0.015
-0.031
-0.031
0.034
0.078
0.135
0.063
-0.110
0.032
0.000
-0.057
1.000
Qwen-C
.0.203
.0.255
.0,352
-0.112
0.010
-0.019
0,039
4.009
0.376
Dpsk-C6.75 6.78
-0.156
-0.126
-0,262
-0,377
-0.275
-0.232
-0,276
-0.190
-0.020
-0,600
0.000
-0,304
L.000
BigCoeeeench
Llama
.0.887
.0.500
-0.111
.0,019
0.024
-0.075
0.131
0.136
0.000
.0.145
Llama 708
-0.162
-0.144
-0, 168
-0,204
<0162
-0.149
-0.158
-0.105
-0.090
-0,092
0.000
40,205
1.000
Qwen 32B
.0.039
0.005
.0.039
0.015
0,000
-0.11
0.026
0.027
0.036
0.000
.0,044
1.000
Qwen 728
-0081
0.027
-0.026
0,039
0.021
0,024
D.099
0.166
0.143
0,123
0.000
-0,027
1.0000
Qwen-C
-0.229
-0.124
-0.224
-0.144
-0.11
-0,068
-0.157
-0.069
0,000
-0.016
0.000
-0,233
1.000
6.77B
-0.054
-0,455
-0.084
0,012
0,010
0.114
-0.053
0,080
0.170
0.164
0.000
-0,101
1.000
IFEval Alpucclivvll
Llama
0.063
0,127
0.022
0.104
0.055
0,074
0.065
0.229
0,316
0.421
0.000
-0.017
1.000
Llama 70B
0.022
0,103
0.006
0.042
0.038
0.026
0.032
0.137
0.249
0,363
0.000
-0.046
1.000
Qwen 328
0.144
0.179
0.119
0.163
0.169
0.136
0.145
0.221
0.330
0.426
0.000
0.100
1.000
Qwen 72B
0.129
0.157
0.078
0,124
0.129
0.120
0.123
0.161
0.340
0.430
0.000
0.194
1.000
Llama 88
-0.323
-0.231
-0.248
-0,223
-0.206
-0.157
-0.234
-0.025
-0.118
0.012
0.000
-0,4428
1.000
Llama 70B
-0.112
-0.287
-0.257
-0.535
Owen 32B
-0.090
-0.026
-0.029
-0.078
-0.005
-0.001
.0.082
0.220
0.019
0.122
0.000
-0,290
1.000
Qwea 728
0.000
-0.011
-0.133
0,068
0.287
-0,010
0.000
-0.243
Table 13: Normalized helpfulness of judge reranking (single-rating protocol with additive prompt), computed from raw data presented
Tab. 5.
34

---

Evaluating, Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Lookahead
Prom 7B
8B
SCB8B
8B
Thmn
12B
Prom SC7B
SFR
708
STE
Qpes
Greedy
Rundom
Oracke
Llmma 8D
x
0.117
0.282
0.342
0.221
0.129
0.307
0.019
0,000
00.10
0,056
0.178
0.202
40.35
0.276
0.405
0.365
0.392
0.405
0.172
0.301
0.540
0.595
0.000
-0.786
1,000
Liana 708
x
-0.175
0.231
-0.079
0.114
0.135
-0,058
0.056
0.058
0,056
0.018
-0,300
0018
0.267
0.287
-0.058
0.079
0.404
11000
40.287
1,000
GSMBL
Owen 32B
/
x
-0,000
0.001
0.337
0.155
-00011
0.98
0.199
0.155
-0.211
00050
0.275
0.15
40158
40021
40021
0.315
0.275
-0:80
0.000
0.472
0.000
00031
1,000
Queen 72B
x
0.200
0.174
.0.374
-0.000
-0.117
0.143
0,000
0,026
0.087
0,057
0.230
0.287
0.115
0.000
0,143
0,000
0.000
0.117
0.543
0.257
0.000
-0.094
1,000
Owes-M 78
0.138
.02662
0.000
0.091
0,077
0.061
0.122
0.168
-0124
0.262
0,185
01185
0.132
0:138
0.306
0.000
-0.665
1.000
DpekMM 7B
X
0.126
0.009
0.126
0,011
0,078
0.113
0.022
0.431
0.365
0.257
0.613
0,000
-0.097
1.000
40,056
0.361
0.309
0.153
-0,948
0.343
0.188
0.591
0.496
0.404
0.600
lamo 8R
0.026
0.064
0013
0.010
0.013
0.038
0010
0.148
0,079
0.056
0.270
0.000
-0.035
0.003
0.087
0.050
0.012
0.081
0.26
0.239
0.126
0.065
Jana 708
X
-0.003
0.00
0.120
0.019
0.037
0.069
0,022
0.048
00,013
0.009
0.096
0.062
0023
0018
0.092
0.192
0.097
0.022
0.050
0.002
0.236
0.336
0.000
-0.028
1.000
MATI
Qwen 32B
x
40.120
.0085
.0000
0.877
-0.68
-2077
40.155
-0015
0.100
0.056
0.012
00003
0.052
0.245
0,000
-0.096
1.000
Qwen 72B
0.039
0.024
0.063
0.073
0.054
0.085
0.054
0.022
0.002
0,012
0.071
0.081
0,119
00039
0.100
0.169
0.085
0.127
0.068
0:007
0.320
0.000
-0.053
1.000
Qwen M. 7B
x
-0.067
-0.022
0.105
0.056
0.100
0.084
0.008
0.021
0.051
0,002
0.151
0.100
0.027
0,035
-0.300
0.296
0.175
0.113
0.175
0.085
0.282
0.372
01,000
-0.202
1,000
Dpsk M
-0,042
0.038
0.007
0.059
0.084
03057
-0.024
-0.001
0.0N6
0.034
0.102
0,10
0.084
0.148
0.106
0:029
0,000
0.029
Lluna 88
x
0.011
0.026
0.127
0.306
0.026
0.111
0,090
0,077
0,005
0,000
0.116
0.106
0.037
0.005
02212
0.036
0.143
0.148
0,098
0.074
0.159
0.233
110000
01006
1.000
Liana 708
-0.205
0.134
-0.008
-0.031
0.016
0.118
0.024
-0.008
0.055
40.094
16165
0.126
0,039
0.639
0,083
0.000
-0.047
0.000
0.110
11.000
.0.132
100.0
CHAMP
Qwen 30B
x
40.109
-0.188
-0.002
.0.33
-0046
-0.051
0.466
0.946
-0.015
0.015
-0.388
0.015
-0.125
40092
0.061
0216
0.108
0.046
-0:077
-0.081
0.231
0.000
-0.033
1.000
Qwen 72B
x
40,499
0.030
40.104
0.149
-0005
-0.015
0,000
0.089
400089
0.015
0.134
0.075
00075
0.120
0.015
0.000
0.000
0.105
0.164
0,000
-0.048
1.000
Owen-M
.0.341
0.023
0.035
0.000
0.000
-0.023
0.195
20011
40046
0.104
0.287
0.161
0:015
0.127
0.357
0.000
-0.47
1.000
Dpek M 7B
x
0,049
0.081
0.016
0.099
0,027
0.011
0.000
0.200
0.124
0.135
0.249
0,000
0.010
1.000
-0.055
0.124
0.044
0.146
0,086
0.130
0.011
0.245
0.168
0.151
0.335
Llama 8B
0,041
0.204
0.000
0.143
0.061
0.122
0.01
0.224
0.082
0.204
0.011
0.000
-0.054
.0.200
0.302
0.041
0.100
0.000
0.234
0.000
0.224
0.204
0.265
Llama 708
x
-02233
0.334
-0.033
0.067
0.133
0.033
00067
0.230
000657
0.133
0.033
0.167
0.133
0,267
-0200
0.000
0.033
0.167
0.067
0.133
0.167
0.135
0.000
0.138
1.000
HomonEoke
Qwen 32B
x
-0.048
0,000
0.000
0.191
-0.005
0,005
0.286
0.848
0.238
0.286
0.191
0.238
-0046
0.095
0,000
-0,030
1.000
Owen 728
0.053
0,053
0.105
0.105
0.053
0.053
0.053
0.105
0,105
0.158
0.211
0.000
0.053
0.158
0.158
0.158
0.211
0.15%
0.211
0.211
8.000
-0.009
1.000
Qwen-C77B
x
0.556
-0,500
-0.500
-0.389
-0.222
-0.278
00,222
-0.222
-0.056
0,500
-0.278
0.000
40500
0.278
-0.222
-0.389
.0.111
0.167
-0.167
00.45
-0.167
0.167
110000
-0.367
1.000
Dpsk
40.314
.0296
0.000
0.114
0,000
-0.129
0.057
.0.29
0.345
0.114
-0.143
0.172
0.000
-0.114
1.000
Llana 88
/
x
.0.02
0,150
.0.72
0.000
0.043
0.033
0.140
0,065
0,002
0,022
0.054
0.097
40.40
0.247
0.076
0.011
0.215
0.215
0.129
0.108
0.065
0.097
0.000
-0.098
1.000
Liana 708
x
00.153
0.259
-0.165
0.153
0.022
0.012
-0.0%0
0.000
-0.094
0.212
0.047
4.271
0212
-0001
0.35
0.059
0.082
-0.012
0.012
0.094
11000
-0.111
1.000
MBPP
Qwen 30B
x
0.102
0.000
0.343
0.123
-0082
-0.102
-0.020
0.0R3
-0.441
0.062
.0041
0.01
.0082
0,83
0.884
0.145
0.164
0.062
0:082
0:082
0.021
0.041
0.000
-0.089
1.000
Qwen 73B
x
0.027
0.054
0.028
0.033
0.027
-0,082
0.000
40,128
0,081
0.081
0.054
0.107
0.135
-0014
0.053
0.053
0.081
-0.10%
0.081
0.000
-0.080
1.000
Qwen 7B
.0.17
0.085
-0.088
0.057
0.15
-0.157
-0.100
00.14
4.120
0.000
0.157
-0.014
0:072
-0.071
0.000
0.377
1.000
D
x
.0,46
.0007
-0.900
0,261
00,33
00087
0.145
00022
00101
0:072
-0.03
0,000
-0.278
1.000
-0,319
-0.232
-0087
-0.145
-0.145
-0.188
-0290
-0145
0.020
-0.101
-0.887
Ilamm 8B
0.129
0.087
0.000
0.050
0.113
0.076
0.145
0.016
0.024
0.005
0.016
0.000
-0.134
.0.208
0.005
0005
-0842
40.100
.0.22
20.40
0.882
0.021
0:053
0.111
BieCoeellachh
Llama 708
/
X
0,146
0.128
00055
0.888
0.022
0.588
0.007
0.015
0.029
0.004
0.089
0.029
0.084
0.109
0.025
0.033
0.025
0.073
0.004
0.015
-0.025
0.029
0.000
-00445
1.000
Qwen 32B
x
-0.066
.0.46
.0.32
0018
0.025
0.060
0453
0.004
0,835
0.028
0.014
0.649
0.081
0.060
0.053
0.039
0,028
0,000
00.088
1.000
Qwen 728
/
x
0.024
0.010
.0024
0.024
0.088
0344
0,082
0,019
0.034
0.024
0.018
.0005
0.005
0.029
0.029
0.155
0.053
0:000
0.029
0.009
0.000
01013
1.000
Qwen-C77B 7B
/
x
-0.139
0,179
-0.033
-0024
-0012
0.024
0.000
0.000
-0.106
0.027
0.064
0.012
2.106
0.103
-0012
0.045
0.067
0.027
-0.074
0.015
0.030
0.042
11,000
-0.098
1.000
Dpst-C.6
/
-0.146
.0.10P
-0002
0.066
0.015
0.006
12000
40.053
-0023
-0.22
.009%
0.064
0.038
0.050
0.063
-0.009
0.000
-0.119
1.000
Table 14: Normalized helpfulness of judge beam search, computed from raw data presented in Tab. 9.
35

---

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
Prom 7B
SFR 8B
Thm 8B
SFR 12B
Prom8x7B
SFR 70B
8GG)
g(t25)
80000)
g(9)
((8m))
g(100)
8(G)
g(aa))
((A0))
g(()
g(2m)
8(R08)
8(C)
g(taE)
((588
Llama
0,998
0.993
0.993
0,960
0.998
0.960
1.043
1.000
1,000
0.962
1.003
0.962
0.995
0.989
0.989
0.962
1.052
0.962
GSM
Llama 70B
0.992
0.995
0,992
0.995
0.999
0.995
0,995
0.995
0.995
0.993
0.994
0.993
1,006
0.997
0.997
0.993
1.001
0.993
Qwen 32B
1.001
0.998
0.998
0,990
1.000
0.900
1.004
1.002
1.002
0.992
0,996
0.992
1.003
0.999
0.999
0.993
1.000
0.993
Qwen 72B
1.007
1.001
1.001
1.002
1.000
1.000
0.998
0.998
1.003
0.998
0.998
1.004
1.000
1.000
1.001
1.001
1.001
Llama 8B
1.091
1.044
1.024
0.838
1.017
0.838
1.244
1.067
1,067
0.857
1.040
0.857
1.012
1.030
1.012
0.813
1,208
0.813
MATH
Llama 70B
1.014
10033
1.003
0.915
1,004
0.915
1.024
0.995
0.995
0.934
1.008
0.934
0.964
1.007
0.964
0.867
1.026
0.867
Qwen 32B
1.041
0.999
0,999
0.967
1.000
0.967
1.111
1.008
1,008
0.941
0,989
0.941
1.033
0.996
0.996
0.907
1.017
0.907
Qwen 72B
1,010
0.999
0.999
0
1.004
0.961
1.021
1.002
1,002
0.962
0.993
0.962
1.010
0.999
0.999
0.928
1.005
0.928
CHAMP
Llama 8B
0.964
L013
0.964
0.915
0.949
0.915
0.953
0.936
0.936
0.859
1.000
0.859
0.857
0.911
0.857
1.065
1.241
1.065
Llama 70B
1.048
1.16
1.016
0.957
1.039
0.957
1.005
1,000
1,000
0.977
1.008
0.977
1.057
1.016
1.016
1,008
1.023
1.008
Qwen 32B
1,048
0.975
0,975
1.052
0.995
0.955
1.063
1.002
1,002
1.097
1.000
1,000
1.081
0.990
0.990
1,063
0,995
0.995
Qwen 72B
1.090
1.005
1.005
1.067
0.990
0.990
1.068
0.995
0.995
1.049
1.005
1.005
1.043
0.995
0.995
1.060
1.000
1.000
MBPP HumanEval-
Llama 8B
1.114
0.943
0.943
1.104
1.020
1.020
1.113
1.001
1.001
0.970
0,934
0.974
1.132
0.991
0.991
0.973
1.039
0.973
Llama 70B
1.064
0.944
1.000
0.944
0.992
0.992
0.961
0.984
0.961
1,069
1.000
1.000
1,000
1.000
1,000
Qwen 32B
1.015
1.000
1.000
0.985
1,000
0.985
0.973
0.990
0.973
0.963
0.977
0.963
1,000
0.985
0.985
1.000
1.000
1.000
Qwen 72B
1.000
1.007
1.000
0.986
1.015
0.986
0.995
0.993
0.993
0.985
0.993
0.985
1.007
1.000
1.000
0.958
1.007
0.958
Llama
1.078
0.942
0,942
0.854
0.850
0.850
0.979
1.002
0.979
0,903
0.903
1.102
0.995
0.995
1,000
1.044
1.000
Llama 70B
1.167
0.992
0,992
0.991
0.894
0.894
1.025
1.006
1.006
1.047
0.902
0.902
1.142
0.984
0.984
1,074
0,996
0.996
Qwen 32B
1.007
0.986
0.986
0.902
0.909
0.902
1.001
1.005
1,001
0.943
0,937
0.937
0.996
0.975
0.975
0.983
0.986
0.983
Qwen 72B
1.014
0.993
0.993
0.917
0.924
1.004
1.004
0.927
0.927
0.996
0.986
0.986
0.955
0.955
BigColBeack
Llama 8B
1,207
1.000
1.000
0.932
0.955
0.932
1.095
0.998
0.998
1.045
1.030
1.030
1,062
0.950
0.950
0.966
1.099
0.966
Llama 70B
1.117
0.994
0.994
1.021
0.948
0.948
1.075
1.007
1.007
1.034
0,948
0.948
1.094
0.963
0.963
1.058
0.992
0.992
Owen 32B
1.036
0.990
0.975
0,990
0.975
1.017
1.000
1,000
0.971
0.965
1.016
0.979
0.979
0.958
1.004
0.958
Qwen 72B
1.017
0.998
0.998
0.959
0.962
0.959
1.009
1.001
1,001
0.969
0.989
0.969
1,000
0.979
0.979
0.995
1.013
0.995
Alpaca Eval
Llama 8B
0.864
L1088
0.864
0.829
1.129
0.829
1.029
1.052
1.029
0.833
1.221
0.833
0.805
1.102
0.805
0.745
1.109
0.745
Llama 70B
0.804
L.043
0.806
1.020
0.806
0
0.988
0
1.003
0.775
0.784
0.999
0.784
0.755
0.997
0.755
Qwen 32B
0.841
1.128
0,841
0.736
0.980
0.736
0.884
0.976
0.884
0.696
1.011
0.696
0,731
0.990
0.731
0.712
0,994
0.712
Quen 72B
0.954
L081
0.954
0.843
1.009
0.843
0.973
1.009
0.973
0.819
1.011
0.819
0.878
1.022
0.878
0.832
1.018
0.832
Llama
1,005
0.929
0.929
1.066
1.028
1.028
1.041
0.988
0.988
1.036
1.023
1.023
1.024
0.970
0.970
1.025
1.051
1.025
IFEval
Ilama 70B
1.016
0.989
0.989
1.029
0.998
0.998
1.031
1.003
1,003
1.045
1.013
1.013
1.048
0.996
0.996
1.031
1.015
1.015
Qwen 32B
0.988
0.993
0.988
1.007
1.025
1.007
1.005
1 1.002
1.002
1.018
1.016
1.016
0.993
0.993
0.993
0.998
1.028
0.998
Qwen 72B
1.002
0.993
0.993
1.006
1.020
1.006
0.996
0.992
0.992
0.996
1.011
0.996
1.007
1.007
1.007
0.991
1.009
0.991
Table 15: The three improvement ratios for the (1, 9)-criiique- refinement setup, computed from raw data presented Tab. 10.
36


Reviews:
[{"review_id": "pf97Pb8cME", "reviewer": "Reviewer_gykS", "summary": "This paper proposes a benchmark called JETTS (Judge Evaluation for Test-Time Scaling) to evaluate the performance of LLM-as-judges in test-time scaling scenarios. The benchmark consists of three tasks: response reranking, step-level beam search, and critique-based refinement.\nThe main findings of the paper are:\n1. LLM judges can be helpful in certain domains, such as instruction following, but not in others, like math and code generation.\n2. Despite more time-efficient, single-rating evaluation protocol performance results in evaluation that is too lenient. Judges often rate a significant fraction of the N responses a top score.\n3. Current chain-of-thought reasoning generated by LLM judges is insufficient for self-improvement.\n\nThe main contributions by the paper are:\n1. The JETTS benchmark, which provides a systematic evaluation framework for LLM judges in test-time scaling scenarios.\n2. The comparision of pairwise and pointwise protocols, and the analysis of their trade-offs.\n3. The investigation of the effectiveness of chain-of-thought reasoning in LLM judges and its limitations.\nOverall, the paper highlights the challenges and opportunities in using LLM judges for test-time scaling and provides a foundation for future research in this area.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer gykS for their thoughtful review. In particular, we are happy that you found our metrics intuitive and our experimental setup sound. We respond point-by-point to questions and comments below.\n\n> Specifically, on Line 218, most judges are fine-tuned using a fixed prompt template, but in this paper's setup, a single prompt template is used for Critique-Based Refinement experiments. It would be beneficial to explain why this template was chosen and what effect using different templates might have.\n\nWe want to clarify our setup. Each judge model is asked to produce a critique and judgment **using its corresponding prompt template. That is, the judge prompt template is not fixed across all judges**. This means that each judge’s instructions and output format follow ones that are used to train the judge. \n\nHowever, we use a fixed prompt template to prompt the generator model, which for the critique experiments are all general-purpose instruction-tuned LLMs. This fixed prompt, shown in Figure 17, takes in the judge’s critique and score (which we parse out separately from the judge response), the previous response, and the original user query, and tasks the generator (i.e., instruct model) to refine its answer. Upon review of Section 3.4, we realize that we did not make this explicitly clear, and will update the final paper to clarify. Thanks! \n\n> Furthermore, while many numbers are included in the results due to the involvement of multiple generators and judges, it is difficult to determine whether the observed trends or patterns are statistically significant. For example, Figure 6 does not appear to show any significant differences between the various judges, making it less informative.\n\n\nFor the single-instance rating reranking protocol shown in Figure 6, we tested whether the min, average and max performances (i.e., the three ticks for each model, task and likert/additive prompt combination) are statistically significantly different from 0 (using a one-sample t-test with p-value threshold of 0.05). Not surprisingly, both the min and max are statistically significantly different from 0 in all cases. However,the average performances are significantly different from 0 for only a handful of math and code cases, as indicated by an “x” in the table below. *Quite concerningly, in all such cases, the average performance is negative, indicating that we have strong evidence that they perform worse than the simple greedy baseline, suggesting the unreliability of the single-rating method.*\n\n|               |         |         |        |         |           |         |\n|--------------:|:-------:|:-------:|:------:|:-------:|:---------:|:-------:|\n|               | Prom 7B | SFR 8B  | Thm 8B | SFR 12B | Prom 8x7B | SFR 70B |\n| Math Likert   | x       |         |        |         | x         |         |\n| Math Additive | x       |         |        |         |           |         |\n| Code Likert   | x       | x       | x      |         | x         |         |\n| Code Additive | x       | x       | x      | x       |           |         |\n| Instruction Following Likert   |         |         |        |         |     |         |\n| Instruction Following Additive |         |         |        |         |     |         |\n| \n\nFurthermore, we present statistical analysis for the linear regression in Figure 4 in our response to reviewer Mn8F, and will include these analyses for all results in the final version.\n\n> Minor Suggestion: Adding clear y-axis labels to each plot would improve the overall clarity of the figures.\n\nWe agree with the reviewer. As we cannot upload an updated paper version, we will update our figures for our final paper.\n\n> LLM-judges lag significantly behind the small QPRM in the task of step-level beam search.\n\nWe are excited to share some new results. Since submission, we obtained access to additional compute resources which were used to evaluate the **large judge models** on beam search (the “C!” entries in the Figure 1 result summary). Here, we provide a summary of our results, with our final paper to be updated with more comprehensive analysis.\n\n| Model                | Performance |\n|---------------------:|------------:|\n| Prometheus-7B        | -0.102      |\n| SFRJudge 8B          | -0.006      |\n| Skywork-Critic 8B    | 0.044       |\n| OffsetBias 8B        | 0.005       |\n| Themis 8B            | -0.026      |\n| SFRJudge 12B         | 0.040       |\n| **Prometheus 8x7B**      | -0.091      |\n| **SFRJudge 70B**         | 0.129       |\n| **Skywork-Critic 70B**   | 0.126       |\n| **Self-taught-eval-70B** | 0.074  |\n| Qwen PRM 7B              | 0.178  |\n| Random               | -0.141 |\n| |\n\nAs we can see, all large judges (bolded), except for Prometheus 8x7B, perform much better than smaller ones, with SFRJudge and Skywork-Critic 70B being the best. However, they still lag behind the much smaller 7B Qwen PRM, suggesting that finer-grained step-level judging has much room for improvement.", "claims_and_evidence": "The submission presents several claims about the performance and limitations of LLM-as-judges in test-time scaling scenarios. While the paper provides some evidence to support these claims, there are areas where the evidence is not clear or convincing due to lack of in-depth experimental results or improper exeprimental setup. \n\nSpecifically, on Line 218, most judges are fine-tuned using a fixed prompt template, but in this paper's setup, a single prompt template is used for Critique-Based Refinement experiments. It would be beneficial to explain why this template was chosen and what effect using different templates might have.\n\nFurthermore, while many numbers are included in the results due to the involvement of multiple generators and judges, it is difficult to determine whether the observed trends or patterns are statistically significant. For example, Figure 6 does not appear to show any significant differences between the various judges, making it less informative.\n\nMinor Suggestion: Adding clear y-axis labels to each plot would improve the overall clarity of the figures.", "methods_and_evaluation_criteria": "The proposed methods/metrics overall look intuitive and reasonable.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Upon review, most experimental designs and analyses appear to be sound.", "relation_to_broader_scientific_literature": "Yes. This paper points out the limitations of LLM-as-a-judge models in the test-time scaling setting. Important findings:\n1. Although unique to LLM-judges, their natural language critiques are currently ineffective in guiding the generator towards better responses.\n2. LLM-judges lag significantly behind the small QPRM in the task of step-level beam search.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "## update after rebuttal \nThe rebuttal has addressed my questions. On the other hand, I agree with some of points from reviewer Mn8F. Thus, I will maintain my score.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "98Co1ee0L7", "reviewer": "Reviewer_Bcf6", "summary": "The authors propose the JETTS Benchmark for evaluating LLM-as-Judge evaluators for test-time scaling where the judges are used to improve the final output from the generator.  The benchmark covers Best-of-N reranking, (2) step-level beam search, and (3) critique-based refinement across the math reasoning, code generation, and instruction following domains.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Bcf6 for their thoughtful review and are grateful that you found our work well-motivated. \n\n> Why were the 3 domains selected…\n\nThis is an excellent question. We will revise our paper to motivate our choice of domains more concretely: \n\n*Instruction following (IF):* Much recent work in judge benchmarking focuses on IF as a proxy for **chat quality** (e.g., [1,2). As such, we include IF as it is best aligned with what judges excel at. This fact is reflected in our results: Across the board, judges performed the best on IF (Fig. 4).\n\n*Math:* Math has exploded in popularity as a domain to measure progress in **LLM reasoning**. Many existing works focus on scaling inference-time compute for math (e.g., [3,4]), using benchmarks like MATH and GSM8K. Thus, we found it crucial to evaluate the judges for math.\n\n*Code:* We identified code as a challenging domain, with many recent methods (including Alphacode [5] and Reflexion [6]),  using inference-time scaling (e.g., [7, 8]) and trained evaluators (e.g., [9]). These initial works suggest that code is an emerging domain in need of strong test-time evaluators. Moreover, the line-by-line nature of code amenable to beam-search, code reranking has been the focus of small-scale judge experiments in prior work [10], and the coding domain provides a more formal reasoning language for LLMs.\n\n> …does it make sense to add general evaluation of the judges as evaluators to better understand their performance on the 3 tasks?...\n\nThank you for suggesting to contexualize JETTS performance with existing benchmarks. We compare normalized helpfulness on JETTS reranking (RR) and beam search (BS) against accuracy on RewardBench [2] and AutoJ’s EvalP test-set [11]. The former assesses reward modeling ability, while the latter assesses chat-specific evaluation. We will update our paper with a complete figure and present a subset of results below.\n\n\n|Model| RewardBench (accuracy)|EvalP (accuracy)|JETTS RR|JETTS BS|\n|-:|-:|-:|-:|-:|\n|Prometheus-7B |72.0|56.03|-0.098|-0.102|\n|Prometheus 8x7B |74.5|58.69|-0.077|-0.091|\n|SFRJudge 8B |88.7|60.34|0.024|-0.006|\n|Skywork-Critic 8B |89.0| 56.39|0.040|0.044|\n|SFRJudge 70B |92.7| 63.51|0.177|0.129|\n|Skywork-Critic 70B |93.3|57.26|0.172 |0.126|\n|\n\nThe judge performance across benchmarks are generally correlated. However, variation in performance in JETTS is much larger than that in RewardBench or EvalP. For example, in RewardBench, the gap between the 8B and 70B Skywork models is 4.3% accuracy (5% relative improvement from 8B to 70B). On JETTS RR, the gap is 0.132 normalized helpfulness, or a 330% relative improvement.\n\nWe believe JETTS more accurately reflects the difference in “fundamental judging ability” between small/large judges: Based on RewardBench, the practical choice is to use an 8B judge rather than a 70B judge for reranking/beam-search (4% acc. drop vs for 9x fewer params). However, JETTS, which realistically mimics inference-time scaling tasks, advises the opposite: the choice of 70B judge yields far more gains than the 8B judge. \n\nWe found this discussion to be rich, and will update our paper accordingly. \n\n> …there should be a consideration on task performance vs latency and memory as these are key considerations…\n\nWe agree that latency and memory are important metrics. Previous works quantify the test time scaling with respect to a compute budget (e.g., Figure 3 of [4]), but use scalar reward models that make the budget easy to quantify (i.e., reward score is only a function of input size). By comparison, LLM judge models can generate critiques/CoT reasoning, making it non-trivial to equalize the compute quantity. Instead, we equalize the experiment setup (e.g., number of responses to rerank or beam width) and leave “compute-optimal” judging to future work.\n\nThe reranking strategy, however, does show a performance-efficiency trade-off (Line 247 left). The O(n^2) pairwise round-robin delivers larger gains than the O(n) single-instance rating but requires more time: 23.58 vs. 5.64 seconds/sample for reranking Llama-8B’s BigCodeBench responses using GPT-4o as a judge in our new experiment for Reviewer Mn8F. This difference is more significant for beam search, where each beam search step requires a reranking step. We will update the paper to highlight this trade-off, and additionally include statistics about GPU VRAM needed to run each judge.\n\n[1] https://arxiv.org/abs/2310.07641 \n\n[2] https://arxiv.org/abs/2403.13787 \n\n[3] https://arxiv.org/abs/2408.03314 \n\n[4] https://arxiv.org/abs/2502.06703 \n\n[5] https://www.science.org/stoken/author-tokens/ST-905/full\n\n[6] https://arxiv.org/abs/2303.11366 \n\n[7] https://arxiv.org/abs/2407.21787 \n\n[8] https://arxiv.org/abs/2501.14723\n\n[9] https://arxiv.org/abs/2410.17621\n\n[10] https://arxiv.org/abs/2407.10817\n\n[11] https://arxiv.org/abs/2310.05470", "claims_and_evidence": "Yes. The paper provides citations where needed and the claims stated in the experimental results are backed by the evidence in the benchmark results.", "methods_and_evaluation_criteria": "Yes, the approaches used make sense and are clearly described.", "theoretical_claims": "Yes, although there is not much in theoretical claims as this is a benchmark paper.", "experimental_designs_or_analyses": "Yes.  The benchmark is run using 6 different generator models and 6 different judge models using 8 different datasets.  These are all split across the 3 tasks of math reasoning, code generation and instruction following.  The analysis is very detailed and clearly presented with key take-aways marked in bold.", "relation_to_broader_scientific_literature": "LLMs are being used as evaluators increasingly often in recent work and they provide key benefits in scaling and control.  This has led to them also being used to improve generated output at inference time as a form of reflective selection and refinement.  The benchmark presented in the paper can help researchers identify the strengths and weaknesses of different models for this purpose.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "The paper is well written and motivated and the contributions are clearly described.  I do feel that the 3 domains selected limit the applicability of the benchmark to more common use cases such as Q&A, chat and summarization and would like to see those added in a future form of the benchmark.  I would also like to see a general evaluation of the LLM judges as simple evaluators in order to better understand the impact of using them for the 3 tasks presented.", "other_comments_or_suggestions": "I would suggest adding a reasoning for why the 3 domains were chosen as opposed to others.  There is a section on task and dataset selection and some detail on model selection but no information on domain selection.", "questions_for_authors": "1) Why were the 3 domains selected as opposed to others?\n\n2) The focus of this benchmark is on test-time scaling, but does it make sense to add general evaluation of the judges as evaluators to better understand their performance on the 3 tasks?  For instance, a weak evaluator may also be weak on the tasks but some models that are in general weak evaluators may still be useful for the tasks.\n\n3). Similar to the previous question, there should be a consideration on task performance vs latency and memory as these are key considerations for deploying a model for the three tasks evaluated.  Memory impacts resource constraints while latency impacts the usability of even a great model, according to this benchmark, as a judge.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZwBFluWkj1", "reviewer": "Reviewer_i886", "summary": "This paper introduces a benchmark designed to assess the feasibility of using large language model (LLM) judges as evaluators in test-time scaling scenarios. The study compares LLM-judges to traditional reward models (RMs) and process reward models (PRMs) in three key tasks: Response Reranking, Step-Level Beam Search, Critique-Based Refinement.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their considerate review, and are happy that you found our paper well designed and interesting. \n\n> What are the possible reasons for the Critique-Based Refinement Task being largely ineffective, despite the success of self-reflection and similar methods…? …what potential solutions could improve its effectiveness\n\nWe suspect that the lack of success is because judge critiques **lack actionability**. Generally, this means that LLM-as-judge critiques tend to focus on surface-level details (e.g., formatting) rather than correctness. \n\nRecent critique benchmarks had similar findings: Models struggle to improve their performance using critiques generated from external critic models. In particular, two papers suggested by Reviewer Mn8F support this claim: CriticBench [1] and CriticEval [2] both highlight that critiques hold some promise, but in general, only critiques from extremely powerful models, like GPT-4, lead to performance gains. Our work further shows that this holds also for multi-round critique-based refinement, whereas previous work focused only on one round.\n\nFurthermore, even for self-reflection and self-correction without an external evaluator, the evidence of its utility has been mixed, with papers finding that LLMs can’t self-correct reasoning [3], or small LLMs need strong verifiers to do so [4]; See Sec 4.2 of [5] for a comprehensive review. Thus, we believe that future work is needed to further identify the fundamental mechanism and reason for reported success, e.g., to resolve the paradox posed by [3]: If an LLM possesses the ability to self-correct, why doesn’t it simply offer the correct answer in its initial attempt? \n\nTo remedy this, we believe that judge training should place a higher emphasis on critique quality, likely borrowing ideas from process supervision or critique-generation models, e.g., using human annotated critiques [6] or RL-training from verifiable feedback [7]. We will update the paper with these discussions.\n\n[1] ​​https://arxiv.org/pdf/2402.14809\n\n[2] https://arxiv.org/pdf/2402.13764 \n\n[3] https://arxiv.org/pdf/2310.01798\n\n[4] https://arxiv.org/pdf/2404.17140\n\n[5] https://arxiv.org/abs/2308.03188\n\n[6] https://arxiv.org/abs/2407.00215\n\n[7] https://arxiv.org/abs/2502.03492\n\n## Case study\nWe manually inspected over 100 judge critiques. Overwhelmingly, we found that judges fail in two ways: (1) Failure to catch mistakes by critiquing only stylistic elements of the response and (2) Overemphasis on stylistic details while ignoring a correct response. In both cases, **judge critiques lack actionability**: Judges focus on superfluous details. In the former, the judge fails to guide the generator to a correct response, while in the latter, critiques needlessly prolong the refinement process without substantive improvements.\n\nBelow, we present two representative examples of both scenarios. We use MATH, as each response can quickly be verified using the final answer. Due to space limitations, we omit the model’s intermediate reasoning steps and parts of the judge critique with [...]. We will dedicate an entire appendix section in our updated paper with complete examples. \n\n### Example 1\nIncorrect response, but the judge focuses on surface level details rather than catching the error. As a result, the generator does not revise further.\n \nGround-truth solution: 19\n\nGenerator: Qwen-72B\n``` \n## Step 1: Identify the sizes of squares that can contain the black center square [...]\nTherefore, the final answer is: $\\boxed{14}$. I hope it is correct.\n```\nJudge: SFRJudge-70B\n```\nThe response precisely executes the instruction by following the required format for complex problems and providing a clear step-by-step solution. The explanation and calculations are concise and easy to follow. The response also accurately counts the number of squares of each size that contain the center square and correctly sums them up to obtain the final answer. [...] Score: 5\n```\n\n### Example 2\nCorrect response, but the judge focuses on minor details. As a result, the generator continuously revises, but retains the correct answer in each revision. In this case, we verify that the model’s reasoning steps are correct.\n\nGround-truth solution: -7\n\nGenerator: Llama-8B\n```\n## Step 1:  We are given a polynomial equation $x^5 + 7x^4 - 2 = 0$ [...]\nTherefore, the final answer is: $\\boxed{-7}$.\n```\nJudge: SFRJudge-70B\n```\nThe response does not precisely execute the instruction. The instruction asks to provide a concise solution with minimal explanation for simple problems (2 steps or fewer) or use a step-by-step format for complex problems (3 steps or more). However, the given problem is complex and requires the step-by-step format, but the response does not provide brief explanations and calculations [...] Additionally, the response does not provide a clear and detailed explanation of how the expression is simplified to obtain the final value. Score: 2\n```", "claims_and_evidence": "I think most of the claims are well supported by the evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in JETTS are generally well-designed for assessing LLM-judges as test-time evaluators.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is mostly sound, particularly in its use of diverse benchmarks, structured evaluation tasks, and efficiency trade-off analyses.", "relation_to_broader_scientific_literature": "I think this paper is well designed and interesting overall. It has good contribution that ties test-time scaling settings and llm-as-judge.", "essential_references_not_discussed": "I think most of the related works are well cited.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "What are the possible reasons for the Critique-Based Refinement Task being largely ineffective, despite the success of self-reflection and similar methods in other tasks? If this is the case, what potential solutions could improve its effectiveness?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mrFeAMcNXj", "reviewer": "Reviewer_Mn8F", "summary": "This paper studies LLM judges as evaluators for test-time scaling and introduces a new benchmark JETTS (Judge Evaluation for Test-Time Scaling). The benchmark assesses different models across three tasks: 1) Response reranking: ) Response reranking, where judges select the best from multiple candidate responses; 2) Step-level beam search, where judges evaluate and rank partial responses during generation; and 3) Critique-based refinement, where judges provide feedback for response improvement. Key findings demonstrate that while existing LLM judges show promise in some test-time scaling scenarios, they have significant limitations, especially in domains requiring complex reasoning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Mn8F for the constructive review and are delighted that they found our evaluation criteria comprehensive. We respond point by point below.\n\n> The random tie-breaking method used for single-rating protocols introduces unquantified variability that affects result reliability; \n\nWe believe this is a misunderstanding: We did not employ random tie-breaking in the single-rating protocol precisely because of the prevalence of tied highest-scores. As explained in Sec. 3.2 (Line 129 right), we report the min, average, and max performances from tied responses, allowing us to account for the possible *range* of performance, as plotted in Fig. 6.\n\n> The paper overlooks prior work on critique-based refinement … CriticBench (Lin et al., 2024) and CriticEval (Lan et al., 2024) …\n\nThank you for pointing us to these works. We will include these works and a longer discussion of critique-related works. Both works use a single round of refinement, while in JETTS, the judge and the generator jointly decide on the number of rounds that the refinement is carried out (including no refinement at all). Despite the difference, all works arrive at similar conclusions: models struggle to improve the response from judge critiques. \n\nFor CriticBench [1], the first two “Correction” columns in Table 1 of Page 5 show that only GPT-4 can significantly improve model responses, and most other models generate worse responses than the original ones, as indicated by the red background colors. \n\nFor CriticEval [2], as shown in Table 5 of Page 7, the quality of refined responses (the CR metric) using judge-generated critiques is much lower than those using human-annotated feedbacks. Furthermore, as the authors did not share the original model performance (to the best of our knowledge), it is unclear whether the refined responses are actually better than the original ones. \n\n> The critique quality analysis lacks a systematic methodology…\n\nWe agree that additional qualitative analysis would be beneficial. We include a case study for **Reviewer i886**, and point the reviewer there due to space limitations.\n\n>2) While the judge-to-generator size ratio findings present coefficients … it's difficult to determine if these differences are statistically significant. \n\nFor the regression analysis in Fig. 4, we have the following p-values for the slope and intercept. \n\n| | |p-value|\n|-:|-:|-:|\n| Math|Slope|**9.3e-10**|\n| | Intercept (at size-ratio=0.1)|**1.6e-3**|\n| Code|Slope|0.93|\n| | Intercept (at size-ratio=0.1)|**0.038**|\n| Instruction Following | Slope|0.26|\n| | Intercept (at size-ratio=0.1)|0.064|\n|\n\nFor math, both the slope and the (negative) intercept are statistically significant, suggesting both that a large size ratio helps with performance, and the very small ratios hurt. For code, the slope is not significant but the (negative) intercept is, suggesting that all size ratios lead to negative helpfulness. For instruction following, while both slope and the intercept are slightly positive, neither is statistically significant. Thus, claiming a positive effect helpfulness would need more data to support. \n\nWe present a similar analysis for the results in Fig. 6 in our response to reviewer gykS, and will include such analyses for all results in the final version.\n\n> The comparison between Likert and Additive rating protocols … would be better placed in an appendix… \n\nThank you for the suggestion. We will make changes in the final version. Given new results of large judge beam search (see our response to reviewer gykS) and GPT-4o-as-judge (see below), we will also holistically assess the significance of each result and re-organize the main body and appendix as necessary.\n\n> Additionally, the benchmark's exclusive focus on open-source models represents a limitation.\n\nWhile JETTS focuses on benchmarking specialized LLM judge models, we started experiments with GPT-4o as the judge, using SFRJudge prompts (Fig. 14-15 on Page 15-16),  Llama-3.1-8B-Instruct as the generator, and report normalized helpfulness in reranking and relative improvement over greedy in refinement. We present preliminary results below and will update our paper with full results when experiments conclude. (Skywork-70B cannot generate critiques and hence cannot be used for refinement.)\n\n| Judge | Reranking: MATH | Reranking: BigCodeBench | Reranking: AlpacaEval | Refinement: MATH | Refinement: BigCodeBench | Refinement: AlpacaEval |\n|--|--|--|--|--|--|--|\n| GPT-4o |0.174|0.300|0.359|0.98|1.07|1.10 |\n| SFRJudge-70B|0.418|0.174|0.478|1.12|1.10|1.11 |\n| Skywork-70B|0.185|0.219|0.381|N/A|N/A|N/A |\n|\n\nExcept for BigCodeBench Reranking, GPT-4o consistently lags behind SFRJudge-70B and Skywork-70B. This suggests that general-purpose high-performance LLMs also struggle with such fine-grained judging tasks, making JETTS a valuable resource in assessing judging capability progress of future LLMs.\n\n[1] ​​https://arxiv.org/pdf/2402.14809 \n\n[2] https://arxiv.org/pdf/2402.13764", "claims_and_evidence": "The claims made in the paper are generally supported. However, several areas would benefit from stronger or more conclusive evidence: 1) In the critique-based refinement findings, the authors demonstrate that refinements rarely surpass both reranking and greedy baselines, but their explanation that critiques are \"not actionable enough\" lacks sufficient support. A qualitative analysis of critique content with examples would strengthen this claim by illustrating specifically why generators struggle to utilize the feedback effectively. 2) While the judge-to-generator size ratio findings present coefficients (0.19 for math, 0.06 for instruction following, 0.00 for code), it's difficult to determine if these differences are statistically significant. This is particularly important when making claims about domain-specific patterns.", "methods_and_evaluation_criteria": "This paper proposes a new benchmark so no new methods are introduced. The evaluation criteria is comprehensive covering various tasks, datasets, and metrics. But I do feel it is quite overwhelming and important findings are not highlighted. The comparison between Likert and Additive rating protocols, for example, appears to be included primarily for completeness rather than yielding substantive insights. Such peripheral comparisons would be better placed in an appendix to maintain focus on the more significant findings. Additionally, the benchmark's exclusive focus on open-source models represents a limitation.", "theoretical_claims": "The paper makes no theoretical claims.", "experimental_designs_or_analyses": "I analyzed several experimental designs in the JETTS paper. The normalized helpfulness metric and task diversity framework both appear sound. However, I identified several validity issues: 1) The random tie-breaking method used for single-rating protocols introduces unquantified variability that affects result reliability; 2) The critique quality analysis lacks a systematic methodology—the paper claims critiques aren't actionable enough but provides no content analysis to support this conclusion.", "relation_to_broader_scientific_literature": "The JETTS benchmark connects two popular research areas: LLM-as-a-judge and test-time scaling. The benchmark reveals significant limitations: LLM judges often fail to improve generator outputs in beam search and refinement tasks, particularly for code generation, and struggle when evaluating larger generator models. These findings challenge the optimistic assumptions in previous research about using LLMs as reliable judges for test-time scaling [1, 2].\n\n[1] Zheng L, Chiang W L, Sheng Y, et al. Judging llm-as-a-judge with mt-bench and chatbot arena[J]. Advances in Neural Information Processing Systems, 2023, 36: 46595-46623.\n\n[2] Snell C, Lee J, Xu K, et al. Scaling llm test-time compute optimally can be more effective than scaling model parameters[J]. arXiv preprint arXiv:2408.03314, 2024.", "essential_references_not_discussed": "The paper overlooks prior work on critique-based refinement. Most notably, it fails to cite CriticBench (Lin et al., 2024) and CriticEval (Lan et al., 2024), which directly evaluate LLMs' abilities to generate and utilize critiques.", "other_strengths_and_weaknesses": "Please see my comments in the sections above.", "other_comments_or_suggestions": "The font size on the images is too small to read clearly.", "questions_for_authors": "Please see my comments in the sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary:** The paper proposes ​​Judge Evaluation for Test-Time Scaling (JETTS) benchmark, which aims to provide an evaluation of LLM-judges for test-time scaling. The tasks evaluated are response reranking, step-level beam search and critique based response refinement tasks. They also provide insights on how LLM-judges compare with respect to reward models and process reward models and discuss performance across different domains. They also show that their explanations are ineffective for guiding generators for self-improving answers.

**Strengths:**
- LLM-judges are being commonly used and their connection with test-time scaling will be relevant to the ICML community.
- Claims are well supported by rigorous experiments. Experimental designs vetted.
- The paper is well written and organized.

**Weaknesses:**
- The evaluation domains of instruction following, math and code might be limited in scope.

**Suggestions:**
- Showcase the limitations of LLM-judges for critic based refinement through qualitative analysis. This is already done during the rebuttal and reviewers recommend including in the final manuscript.

**Recommendation:**
- Three out of four reviewers recommended acceptance of the paper with scores of 3, 4 and 4. The remaining reviewer recommended a weak rejection (with a score of 2).
- I believe that the paper is investigating an important area with clear research questions and well designed experiments to investigate those questions, and will be of interest to the ICML community. Therefore, I recommend its acceptance.
Decision: Accept (poster)