Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
Authors: Austin Xu
Xuan-Phi Nguyen
Yilun Zhou
Chien-Sheng Wu
Caiming Xiong
Shafiq Joty
Venue: ICLR 2026 Poster (2026)
Keywords: Automatic evaluation
LLM-as-judge
multi-task evaluators
step-level evaluation
verifers
Primary area: foundation or frontier models, including LLMs
TL;DR: We train two foundational automatic evaluators at large data scales, demonstrating state-of-the-art performance

Abstract: Finetuning specialized generative evaluators has emerged as a popular paradigm to meet the increasing demand for scalable evaluation during both training and test-time. However, recent work has largely focused on applying new methodology, such as reinforcement learning (RL), to training evaluators, shying away from large-scale, data-driven development. In this work, we focus on data scaling, curating a set of 2.5M samples spanning five unique evaluation tasks (pairwise, step-level, reference-free and reference-based verification, and single rating) and multiple domains focused on reasoning evaluation. With our data, we train Foundational Automatic Reasoning Evaluators (FARE), a family of 8B and 20B (with 3.6B active) parameter evaluators, with a simple iterative rejection-sampling supervised finetuning (SFT) approach. FARE-8B challenges larger specialized RL-trained evaluators and FARE-20B sets the new standard for open-source evaluators, surpassing specialized 70B+ evaluators. Beyond static benchmarks, we evaluate FARE in real-world tasks: As inference-time rerankers, FARE-20B achieves near-oracle performance on MATH. As verifiers in RL training, FARE improves the downstream RL-trained model performance by up to 14.1\% vs. string-matching verifiers. When initialized from FARE, a continually-finetuned FARE-Code outperforms gpt-oss-20B by 65% on evaluating test-case quality

Published as a conference paper at ICLR 2026
FOUNDATIONAL AUTOMATIC EVALUATORS: SCALING MULTI-TASK GENERATIVE EVALUATOR TRAINING FOR REASONING-CENTRIC DOMAINS
Austin Xu, Xuan-Phi Nguyen,
Yilun Zhou; Chien-Sheng Wu, Caiming Xiong, Shafiq Joty
Salesforce AI Research
Correspondence: s eestyesalesforre...
ABSTRACT
Finetuning specialized generative evaluators has emerged as a popular paradigm meet the increasing demand for scalable evaluation during both training and test-time. However, recent work has largely focused on applying new method- ology, such reinforcement learning (RL., to training evaluators, shying away from large-scale, data-driven development. In this work, we focus on data scal- ing, curating set of 2.5M samples spanning five unique evaluation tasks (pair- wise, step-level, reference-free and reference-based verification, and single rating) and multiple domains focused on reasoning evaluation. With our data, we train Foundational Automatic Reasoning Evaluators (FARE), family of 8B and 20B (with 3.6B active) parameter evaluators, with a simple iterative sejection-samplingg supervised finetuning (SFT) approach. FARE-8B challenges larger specialized Rl-ttrinned evaluators and FARE-20B sets the new standard for open-source eval- uators, surpassing specialized 70B+ evaluators. Beyond static benchmarks, we evaluate FARE in real-world tasks: As inference-time rerankers, FARE-20B achieves near-oracle performance MATH. As verifiers in RL training, FARE improves the downstream RL-trained model performance by up to 14.1% vs. trtinngmatchin verifiers. When initialized from FARE, continually finetuned FARB-Code outperforms gpt-oss-20B by 65% on evaluating test-case quality.
The FARE family of evaluators
resporse matching
0
muiti
annotaaeed
annotsted
Syntheticall
K
Updato model with SFT
Large-scale iterative training
Stpp lovel
Single rating
let Aused verificaioon
FARE: Foundational Automatic Reasoning Evaluators
task, multi-domain training samples
Statt benchirarks
Reword model for Infeeence scaling
Verifer during RL taliing
Irit for domain specific fineturing
Figure 1: Overview of our work. We curate 2.5M multi-task. multi-domain training samples (left) and use large-scale iterative rejection sampling SFT to train FARE. family of automatic evaluators (top right). We evaluate FARE on static benchmarks and on various real-world downstream tasks.
"Work done Salesforce AT Research

---

Published as a conference paper at ICLR 2026
INTRODUCTION
The past two years has seen the rapid adoption of large language models (LLMs) automatic eval- uators in response to demands for scalable evaluation of LLM outputs. LLM--bsed evaluators serve as judges for popular benchmarks (Dubois et al.
2024: (Zhngg et al. (023), generative reward models 2024). and verifiers/critics in inference- time scaling settings (McAleese et al. 2024 Zhou al. 2025b). The widespread integration of
for preference optimization Yuaan et al. 2024
Wu et al.
evaluators into nearly every phase of the LLM development cycle (Wul (225)) demands evaluators
that can handle multiple evaluation tasks while operating effectively across diverse domains.
Different settings require different evaluation abilities: Alignment needs evaluators capable of com- paring different responses, i.e., pairwise evaluation, whereas monitoring model outputs requires finding minute mistakes, i.e., step-level evaluation. More recently, generative evaluators are tasked with providing reward signals during reinforcement learning (RL) training (Team Kimi al. 2025] Jiang et al. 2025b), and are expected to grow in importance as RL moves towards unverifiable do- mains (Guujja et al, 2025; Jayalath et al. 2025) complex reasoning settings (Ke et al. 2025a Ferrag et al., 2025). As evaluators take central roles in training and evaluating the next generation of models, they must be flexible enough to evaluate as the setting demands.
Compounding the challenges of multi-task evaluation is the expanding number of domains requir- ing evaluation: RL-training has quickly moved from math reasoning (e.g.. Yu et al. (2025b)) to
general purpose reasoning (a al. 2025 (e.g.. history or economics). Agentic settings introduce
additional wrinkles: With autonomously acting single agents (OpenAT 2025) Nguyen et al., 2025]
Wei et al. 2025) and complex multi-agent workflows (Liang al. [025| Alzubi al. (2225) now being set free to browse the web and act on behalf of users with minimal oversight, evaluators must assess not only agent reasoning, but also proposed tool-use. These systems, sometimes built with in-
tricate model-generated (Hu et al., [2024b] Zhang et al. 2024a Ke et al. 2025b) interdependencies,
are bottlenecked, part, by subpar evaluation (Cemri et al. (025).
Unfortunately, recent work in the open-source automatic evaluation community has failed to meet these twin demands of multi-task, multi- evaluators, opting instead in training task- specialized evaluators at relatively small data scales. We break this trend by scaling up data, curat- ing 2.5M multi-task, multi-domain training samples that emphasize reasoning settings. As shown in Fig. [ our data mix covers 5 distinct tasks and various domains like math, code, tool-use evalua- tion, and natural language reasoning. With our data, we train, Foundational Automatic Reasoning Evaluators (FARE), two best-in-class evaluators. As shown in Fig. our contributions are
Multi-task, multi-domain dataset: We curate a large-scale, multi-task training set with an em- phasis on reasoning-cente settings. We supplement existing human-and model-annotated data with synthetic data created from challenging new seed datasets.
Scalable learning via iterative rejection sampling: We show that iterative rejection sampling su- pervised finetuning (RS-SFT) is stable approach for training evaluators scale. The semi-online nature of RS-SFT avoids problematic teacher model distribution shifts while bringing computa- tionally stable and efficient model updates. Through ablations,
quantify the impact of training pipeline features like quantity of direct judgment data and the use of a continuous curriculum. The FARE family of evaluators: We train FARE-SB and FARE-20B and rigorously assess them with challenging benchmarks and 3 practical downstream settings: test-time response reranking, RL-training verification, and domain-specific continual finetuning.
Our trained models are both well-rounded and nigh-performing. Out of the box, FARE improve generator performance at test-time, achieving near oracle reranking performance on MATH, and provide clear rewards during general-domain RL training, boosting downstream performance by 14.1% 5% over typical string-matching verifiers. With minimal continual training, FARE can be adapted to specific domains like code, beating gpt-oss-20B by 65% in code test-case quality evaluation.
2 BACKGROUND AND RELATED WORK
An automatic evaluator (AE) To X + maps input x = (p. q. R) X' to output y = (c,j) e y Input x consists of p, the evaluation protocol that specifies both the evaluation task (c.g.. pairwise comparison, verification) and evaluation rubric, g, the original question, and R, set of model re-
2

---

Published as a conference paper at ICLR 2026
sponses be evaluated. The output y consists of natural language critique and final judgment j- The AE may also be prompted to omit the critique c and directly output the judgment j. which we denote y j). The specific evaluation protocol p determines the elements of set R and the exact form of judgment j. For example, in pairwise evaluation, R consists of two responses r2) and the judgmeet is choice between the two ("A" "B"), whereas in single-rating, RR consists of a single response r and the judgment is an integer on a 1-5 scale. In this work, we focus on training automatic evaluators capable of the evaluation tasks shown in Fig.
Pairwise comparisons:
Given response set R (ri, r2). the AE selects the better of rı and r2-
Step-level evaluation: Given response set R = [s where "steps]] is a single model response broken down into steps, the AE identifies step-level errors.
Reference- beaeee verification: Given response set R = {reand, "roe), where Teand is the candidate and Fief the reference, the AE determines if reand is correct based on Tief- Reference-free verification: Given response set R {r]. the AE determines if r is correct.
Single rating: Given response set R {r), the AE assigns an integer score r.
Past work in generative automatic evaluators. Capable LLMs, like GPT-4, were originally prompted as scalable evaluators (Wang et al.) 2023a Liu et al. 2023b Fu et al. 2024 Chiang
& Leee 2023). Subsequent analysis revealed pitralls of prompted approaches, like biases with
spect to position (Wang et al.) 2023b Li et al. 2023), length (Zeng et al. 2023 Park et al. 2024),
or self-preference (Panickssery al. 2024). Finetuning specialized evaluators emerged as a result,
with early approaches using teacher model outputs to do supervised finetuning (SFT) (Kim et al.)
2023 2024b Li et al. [023 Park et al., 2024 [Shiwen al.) 2024) or direct preference optimization
(DPO) (Hu et al. [024c] Ye et al. (024), often focusing only on one or two evaluation tasks. More
recent methods moved to reasoning models as teachers (Khalifa al. 2025).
Vu et al. (2024); Wang et al. (2024a); Cao et al. 2024); Alexandru et al. (2025) train foundational
evaluators at larger data scales with multi-protocol capabilities via offline training methods like
SFT or DPO. Such approaches take inspiration from general-purpose, large-scale multi-task learn- ing (Sanh et al. 2021; Raffel et al,, [2020] Wei et al., [02T], which showed broad generalization capabilities emerge with the scaling of training data. Foundational evaluators likewise were empiri- cally shown to generalize to unseen evaluation tasks, prompts, and criteria while being more robust to common biases (Vu et al.
2024 Wang al., 2024a).
Recent work has focused on methodological advances, either using inference-time scaling (Liu et al,, 2025d Chan et al. 2025; Zhao et al.
2025) online training like reinforcement learning from
verifiable rewards (RLVR) (Chen et al. 22025a b; Whitehouse et al.) 2025 Xu et al.] 2025b) Xiong
(et al., 2025b) improve evaluator performance. Because RLVR computationally demanding
with relatively brittle training pipelines (Guo et al. 2025 Yang et al. 2025), recent evaluators are
typically trained on a small amount of data for a single le task. Our work bridges early work in training foundational evaluators with more recent methodological advancements, demonstrating that simple semi-online training approach enables stable multi-task training scale.
Desiderata for new generation of evaluators. Here, we outline our design philosophy for FARE. Beyond accuracy and robustness, we seek efficiency, as many evaluation settings like inference-time reranking or RL rollout verification demand low latency. In contrast to recent long chain-of-thouge (CoT) evaluators (Chen et al. 2025b Khalifa et al., 2025), we select base models with either no or very compact "thinking" CoTs We also explicitly avoid having the evaluator generate reference answers. Past work has used evaluators to generate references during evaluation (Zheng et al. [023]] Li et al. 2024) or training rollout (Chen et al. (025b). Not only does this risk severely degrading performance when the reference is wrong (Krumdick et al. (2025), it also converts evaluation into generation, turning relatively easier task into harder (Zhou al. 2025a).
FARE: DATA AND TRAINING RECIPE
3.1
DATA CURATION
We use two data approaches for curating final training mix: Using Existing high quality train- ing datasets created for evaluator and preference finetuning and generating Synthetic datasets through programmatic error injection and generate-then-geee strategy.
3

---

Published as a conference paper at ICLR 2026
Data by Task
Dalsa by Domann Tulee
Data by Approach
(257%)
Dosing
(EEAN
Figure 2: Breakdown of our curated training dataset of 2.5M samples by task (left), domain (center), and curation approach (right). Domain breakdown excludes step-level data, which entirely math.
Existing data consists of training samples from proven sources that have produced effective evaluators (Vu et al. 2024 Cao et al. 2024). These datasets consist of high quality annotations from humans and frontier LLMs, and cover evaluation tasks like step-level, single-rating. and pairwise evaluation and domains like chat quality, code, and safety. Following Wang et al. (2024a). we largely focus our data collection on modern (2024 and beyond) datasets, as these datasets contain the most up-to-date model responses and fresh annotations. Beyond evaluator-specific training, we take advantage of existing preference fine-tuning datasets used for RLHF (Ouyang et al., 2022) and DPO training (Rafailov et al. (023), converting these directly into pairwise eval evaluation samples. In domains with objectively correct answers, c.g., math, we also create verification training data with positive responses as correct/reference responses and negative responses incorrect responses.
We hand-craft evaluation rubrics for each source dataset that follow annotation instructions given to human annotators models, if existing. If such original instructions do not exist, write custom evaluation rubrics for each source dataset based on the data composition and domain. App. E.2 provides an example rubric. Existing data lays solid foundation, with 1,4M samples already dwarfing data scales found in recent work (e.g.. 22K samples in Whitehouse et al. (2025) or 64K samples in Chen al. (2025b)). However, upon inspection, we found three clear shortcomings: (1) Newly relevant tasks, like verification, were underrepresented. (2) Existing pairwise task data focused largely on chat-related topies and less-so on reasoning-relevant domains. (3) Questions and responses from newer, challenging datasets produced to meet the needs of reasoning focused RL trainning were absent. To address these limitations, we supplement with synthetic data.
Synthetic ii data is generated from a diverse set of challenging seed datasets using two approaches:
Programmatic error injection. We employ programmatic error injection when applicable, such as tool-use and function-calling data. For example, to create pairwise tool-use data, we inject errors (e.g., type error, extra argument, syntax errors) in correct function calls. This approach increases the amount of tool-use evaluation data, adding both pairwise and verification samples. Generate--hen--raade Here, we leverage mix of recent and established training datasets com- prised question and verifiable ground-truth answer a. We sample up to 20 responses per from various generator models, then grade each responses based a. After grouping responses by correctness, we create verification and pairwise. We use 12 unique generators from model families, covering reasoning and non-reasoning models. This ensures that FARE are trained on diverse array of model responses, enabling better generalization across distinct response distribu- tions. Generate-then- -rade enables us incorporate problems from recent, challenging datasets covering frontier math and reasoning tasks. This enhances the quality of our pairwise and verifi- cation data with difficult-to-evalua reasoning-foccee samples.
Our final dataset comprises 2.5M training samples; an overview shown Fig. 2 with breakdowns by task, domain, and curation approach, and exact dataset sources are described in App. B
3.2 MODEL TRAINING
General training recipe. We aim to train automatic evaluator ne. which call the policy model parametrized by 8. Our data curation process yields a training dataset of N samples D {24.77h J. where denotes the ground-truth judgment. Corresponding ground truth critiques c* are not typically available in evaluator training data. As result, past work has resorted to offline teacher-model based approaches online RL-based approaches, as discussed in
4

---

Published as a conference paper at ICLR 2026
These established approaches have their limitations: Teacher models introduce distribution shifts with respect to the policy model, a common problem with imitation learning (Ross et al. 2011), with choice of teacher model having large impact on downstream performance (Guha al. 2025
On the other hand, RI. training is compute and time-intensive, making it difficult to scale to large data quantities, with past work (Liu et al. 2025c) only exploring small (1.5B parameter) models via ad-hoc interventions like reference policy resetting.
We borrow desirable qualities from both paradigms and use semi-online iterative rejection sampling SET (RS-SFT) (Tourron et al., 2023 Dong et al. 2023), which was recently shown to be competi- tive with RLVR approaches (Xiong et al. 2025a). RFFSFT avoids sub-optimal distribution shift by finetuning on correct evaluation traces produced by the policy model while employing computa- tionally lightweight policy update step. This enables simple yet stable scaling to millions of training samples without sampling from teacher model. An overview of our approach is shown in Fig. |
Concretely, we train Th as follows. We split our N training samples into disjoint rollout batches B. (Tee of fixed size Nooloot and initialize the initial policy weo to be an existing post- trained LLM (e.g., spt-oss-20BB) Then for step | = .,..., 1, we perform the following:
Rollout from previous policy: For inputs zi, from B,, sample K responses per input from policy The denoted (0 ...... ........., Vi.e ((55)
Rejection sampling: For each of the K responses (0l. ..... Vs.c ). we determine correctness with ground-truti judgment Jī. For inputs with correct responses, randomly chosen response is kept. Any inputs without correct responses are discarded. We denote the collected set of inputs and corresponding correct responses as Dr.
(K)
Policy update: Use De to update the policy weights via SFT, initializing with 0,:
8i+1 arg max Σ
log (gyl)))
(x.y)ED.
Our approach draws inspiration from algorithms such as STaR (Zelikman et al. 2022) and RAFT (Dong et al. 2202)), with some key differences. STaR notably re-initializes training from Te, for each iteration and samples only one greedy response per input prompt, while RAFT relies on an external reward model to rank generated outputs. Because the automatic evaluation setting is inherently verifiable, i.e., the answer space of evaluators closed vocabulary and discrete, like A/B for pairwise comparisons or yes/no for verification, we omit the need for reward model to rank sampled responses.
Specific to evaluators, the Self-Taught Evaluator (STE) paradigm of Wang et al. (2024b) and follow- up EvalPlanner (Saha et al. 2025) are closely related to our approach. STE follows STaR with policy re-initialization and EvalPlanner uses multiple SFT and DPO training runs per iteration t. Fur- ther, these works use ln-the-loop synthetic data generation, sampling responses to a small number (< 25K) of seed questions from a fixed generator. Pairwise samples are then created from correc- Vincorrect responses and used to train the model. This data generation approach cannot be adapted to create data for other tasks, like step-level evaluation, fundamentally limiting the task abilities of STE. A secondary concern is lack of exposure to diverse response distributions; Evaluations in Wang al. (2024a) show that scaling training data with a simpler training recipe leads to better generalization across benchmarks compared to STE.
Batch composition. For each rollout batch B,, we select unseen training samples from our curated training dataset, ensuring the task mixture is consistent with global task composition. For example, 33% of our overall training data are pairwise tasks (Fig. 2}. so 33% of the input prompts in B, are sampled from unseen pairwise samples. We then sample K = 4 responses per sample with temperature of 0.9, and determine correctness based on final judgment.
Inclusion of direct judgment data. Past work (Wang et al. 2024a Cao et al.) 2024) has showed the importance of including direct judgment data samples to isolate judgment training signal. These are samples where the critique is omitted, and the input protocol p is modified to prompt directly for a judgment. To precisely control the fraction of direct judgment data, we convert a fixed percent of DD direct judgment data by dropping generated critiques and modifying the input prompt accordingly. In App.D] we ablate the proportion of direct judgment data and show such data enables FARE be prompted to exclude critiques for faster inference.
5

---

Published as a conference paper at ICLR 2026
Table Pairwise evaluation results, with best and second-best performance in each section marked. FARE achieve best-in-class performance, even outperforming frontier models in tool-use evaluation. indicates that benchmark uses consistent accuracy (25% random baseline).
JudgeBench
RJBI
PPE Correctness
RM-Bench
When2Call'
RISE-Judge-78
77.2
47.22
ExalPlanner 8B
30.20
52.8
68.1
73.4
RM-RI-14B
46.86
43.70
64.0
79.6
19.89
Compus/Judge-778
49.14
37.76
60.9
82.2
41.67
Alla Scleme 8B
21.14
12.41
53,3
71.9
$6.00
Compuschndeer-14a
50.29
37.69
62.0
77.7
44.56
FARE-8B
55.71
51,05
63.8
79.2
80.33
RISE- JuSge-3dge3328
46.86
42.35
63.5
82.2
46.44
CompassJudger
$1.89
RM-R1-32B
54.29
46.39
65.9
81.5
23.89
Sclf-Taught-7005
48.3
73.6
EvalPlanner-00f
56.60
70.2
82.1
J1-70B
60.00
72.8
82.7
FARE-20B
64.29
57.05
74.4
90.5
76.67
48.29
40.59
60.5
78.07
$9.67
Qwen3
gpt-oss-20B
59.43
50.51
71.7
89.9
61.33
gpl-uxs- 120B
70.00
GPT-5-namo
59.71
51.52
80.7
92.1
50.02
GPT-5
93.8
75.78
Table 2: ProcessRench results, with best and second-best performance in each section marked. FARE-20B almost matches GPT-5 with the same prompt, achieving best-in-class performance, while FARE-8B beats comparably sized generative (Gen.) evaluators.
GSMSK
OlympladBench
PRM
SlyworkPRM-I150
59.0
48.0
19.3
19.2
36.4
Shepherd-78
47.9
29.5
24.8
23.8
PRM
SkyworkPRM-7B
70.8
53.6
22.9
21.0
42.1
PRM
AcIRRM-7B
82.7
82.0
72.0
67.3
76.0
PRM
Dwen2.5-78B.o PRMHOOK
08.2
44.3
56.5
PRM
Owen2.5 M-MMMMlll----RRRE
82.4
77.6
67.5
66.3
73.5
PRM
Owen2.5 Math 72B-PRM
87.3
80.6
74.3
71.1
78.3
Qwen2.5 Mathh-
26.8
25.7
Gen.
RL Tango-BB
53.1
48.2
37.8
36.3
43.9
Gen.
StepWiser-
46.9
43.4
26.3
28.4
36.3
StepWiser-BB
$4.4
Gen.
FARE-8B
68.5
67.2
39.9
58.1
63.5
Gen.
Llama-3 3-7-0
82,9
59.4
46.7
43.0
$8.0
Dwen2.5-Cuder-322
Gen.
QwQ-12B
$8.0
78.7
$7.8
61.3
71.5
Gen.
Mreen2.-MMath-222
65.8
52.1
32.5
31.7
45.5
GPT-lo
79.2
Gen.
FARE-20B
89.8
87.8
80,0
79.9
84.4
Qwen3-8B- BB-ClSStttt
37.0
41.0
36.3
38.9
38.3
Qweni-818
63.2
56.7
Gca.
spt-oss-20B 20B
79.3
79.4
68.8
68.2
73.9
Gen.
gpt-oss- 1200 120B
89.6
87.6
80.8
76.0
83.5
GPT-5-mano
83.8
Gca.
GPT-5
91.4
89.5
80.6
76.9
84.6
Per-batch continuous curriculum learning. We additionally use a continuous curriculum in train- ing: For each (r. v) E De, we compute the pass percentage from the K = 4 rollout generations for r, then sort the dataset in descending order of pass percentage. That is, samples where all 4 sampled outputs correct are used to update the model first, and samples where only of 4 sampled outputs are correct are used to update the model last. We find this has negligible impact on pairwise domains but large impacts in step-level evaluation,
show in App.
Base models. We train two models starting from Qwen3-88 --B- -B--BB YYngg et al. 2025) and gpt-oss- 20B (Agarwal et al. 2025). denoted FARE-8B and FARE-20B. We find Qwen3 8B (post-trained) to be over-trained, and therefore cold-start Qwen3-88 -B--BBB- from SFT data from Qwen2.5-32B. Instruct, which denote Swen3-8B-ColdSann See App. B.2 additional details.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
We evaluate FARE on both core benchmarks, static benchmarks for automatic evaluators, and in downstream settings, which simulate real applications of evaluators. We provide descriptions of benchmarks and baselines in App. C Cnd additional ablations and analysis in App. D
4.1 CORE BENCHMARKS
Setup. We evaluate along five diverse aspects: (i) reasoning with JudgeBench (Tan et 2024),
deasoningJudgeBende (RJB) (Xu et al., 2025b), and PPE Correctness (Frick et al. (024)) (ii bias and robustness with (iv) step-level error identification with ProcessBench (Zheng et al. 2024). and (v) reference-baxed
(Liu et al. 2024b), (iii) tool-use with When2Call (Ross al. 2025),
verification with VerifyBench (Yan al. 2025).
For RM-Bench and PPE Correctness pairwise benchmarks, we adopt default evaluation setups, run- ning each benchmark once with a fixed random ordering of responses. For other pairwise bench- marks, we report consistent-accuracyy (Tan al. 2024). where each test sample is run twice, swap- ping the order of response A and response B. If the evaluator selects a different response between runs (i.e., is positionally biased), then the sample is marked incorrect; if the evaluator is consis- tent, the judgment is graded against the ground-truth. For ProcessBench and VerifyBench, we report Fl-score and accuracy, respectively. We compare FARE against other finetuned genera- tive and prompted evaluators. We report official numbers from past benchmarks, reporting sources App. C If necessary, we run each baseline using its own prompt template. For ProcessBench, we additionally compare against non-generative process reward models (PRMs).
D to 3 present our results on pairwise, step-level, and reference-based verification benchmarks, respectively. Our prompts are provided in App.
Results. Tables
Table [ shows that across diverse pair- wise benchmarks, FARE exhibit best-in- class performance, outperforming compa- rably sized baselines. FARE-8B is the strongest small judge, outperforming re- cently released RL-trained models like J1- 8B and R.M-R1-14B by 13.71 and 6.57 ab- solute points JudgeBench, respectively. FARE-20B challenges strong judges at 20B parameters, outperforming dense 70B-sized judge models despite having 3.5x fewer to- tal parameters and nearly 20x fewer ac- tive parameters. The strong performance of FARE across reasoning benchmarks, which span math to scientific domains to causal reasoning, show that our models ex- cel at discerning between objectively cor- rect and incorrect responses. Beyond rea- soning settings, FARE generally bust to subtle, stylistic biases (RM-Bench) while excelling in tool calling evaluation (When2Call), which is increasingly impor- tant agentic workflows grow popularity.
Table 3: Ref.-based verification, with best and
second-bes performance marked per section. FARE beat general-purposs verifiers in hard settings.
VerifyBench
VenifyBench-Hadt
Math Verify
45.90
32.50
GPT- mini
Llama-3,1-88
73.05
43,20
Qwen3-4B
92.00
72.40
Phi-4
89.35
56.60
Yi-1.5
87.70
61.40
FARE-8B
93.20
78.40
GPT-4o
93.15
72,60
Llama-4-Scout
90.01
48.50
Llama3.3 70B
83.25
$4.70
Qwen2.5-72B
92.35
62,40
Qwen3-32B
95.80
71.80
FARE-208
94.95
85.10
Qwen3 8B ColdStart
92.45
72.60
Qwen3-8B
94.00
70.90
gpt-oss-20B
91.95
83.60
gpt-oss-122 120B
95.35
88.30
GPPT nano
94.65
84.00
GPT-5
96.10
90.50
FARE also are extremely strong step-level evaluators, as shown by ProcessBench performance in Ta- 2 FARE-8B is the best small-sized generative critic model, outperforming the recently released Step Wiser-7B (Xiong et al. 2025b), RL-trained specialized step-level evaluator, by 1.6 points. FARE-20B outperforms the specialized Qwen2.5-Maahh722. MrM-atthh-7BBRM.. by 6.1 points, even matching GPT-5, Most notably, FARE excel on the two most challenging splits, OlympiadBench and Omni-
MATH, with FARE-8B beating StepWiser-7H: by 5.6 points and FARE-20B beating Qwen2.5-Maan
PracessBende defines their reported Fi-score differently from the traditional Fi-score; See this link

---

Published as a conference paper at ICLR 2026
Table Scaling inference-time compute for FARE typically brings additional gains in performance: The performance gaps between FARE-88/Deeppeek-GGG GRM and FARE-20B/J1-0B widens.
PPE
MMLU Pro
MATII
GPQA
MBPP+
[FEval
Overall
11-8B w/ SC#32
65.6 - 67.5
70.0
76.6
53.2
+5.77
53.1
54,6
54.0 - 54,9
59.2
61.9
11-70R w/ SCBR12
79.0 -3 79.9
86.0
88.1
65.9
66.5
66.0
66.5
67.3 -7..2
72.8
73.6
DeepSeek GRM w/ SCW32
64.8-1 65.5
68.8
69.4
56.0
49.9
59.8
59.8
DeepSeck-URM--22 w/ MetaRM
64.8-+ 68.1
68.8
70.0
53.6
00-++56.. 56.9
50.1
- 50.8
59.8 - 70.4
59.8
63.2
FARE-SB SC@ 32
69.3 70.8
79.7
80.4
58.4
54.9
55.9
63.8
FARE-200 w/ SC/0/2
80.3 83.6
94.6
97.3
68.5
7711
59.3
- 57.3
69.1 -+ 71.1
24.4
76.6
72B-PRM by 7.3 points on average. Overall, FARE are not only capable outcome-level evaluators, but are able to find subtle mistakes that manifest at the step-level.
FARE are also capable verifiers, shown in Table 3 with both models outperforming all reported baselines on VerifyyBncch-ard (Yan et al. 2025). In particular, FARE-20B excels beats the next best model, GPT-4o, by 12.5 absolute points. As we demonstrate in 4.2 when used verifiers during GRPO settings, FARE bring tangible benefits over typical string- maathing verifiers.
Scaling inference-time compute. Recently, sampling parallel judgments and aggregating via ma- jority vote, i.e., self-consistency (Wang et al.] (022), has been used to improve evaluator perfor- mance. In Table we use self-consistency with 32 responses (SC@32) 32) on PPE, comparing with J1 and DeepSeek- GRRM (Liu et al.] (025d).. DeepSeek- GRM also trains a MetaRM to perform judg- ment re-ranking test-time. Across most splits, using SC@33 32 improves performance, with up to a 3.3 point improvement for FARE-20B on the MMLU-Pro split. Even without SC@32, FARE-SB beats DeepSeek-GRMs GRM-27B + MetaRM, with the gap widening with extra compute. Similarly, the gap between FARE-20B and J1 J1-70B grows with extra compute. Interestingly, we see that across both our models, performance on MBPP+ slightly degrades, indicating that SC may not be the optimal way to use compute across all domains. Nonetheless, we observe gains in the aggregate.
4.2 DOWNSTREAM EVALUATION
Setup. We apply FARE on 3 downstream tasks: (i) Reward model for inference-time scaling, (ii) verifier for GRPO training, and (iii) initialization for continual finetuning for domain-specific evaluation. We provide detailed explanations of our downstream evaluation settings in App.
Reward model for inference-time scaling. We use the standardized setup in JETTS (Zhou al., 2025b), which provides a set of 10 outputs from various generators and various benchmarks with corresponding correctness labels. Here, automatic evaluators rerank the responses, and performance is measured as the final performance of the vvaluator-selected responses. We select the four most challenging benchmarks used in JETTS: MATH (Hendryeks et al.] 202T), CHAMP (Mao et al, 2024), MBPP+ (Liu et al. 2023a), and BigCodeBench (Zhuo et al. 2024). We compare against strong judge models benchmarked previously on JETTS: Sudg-Juge--S.. SB.70B (Wang et al. 2024a), Snyyork-Crrttc-888 8B,70B (Shiwen et al. 2024 Self Self-Taught Brrnight--vllator---a (Wang al. 2024b). and
MATH
CHAMP
MBPP+
BigCodeBench
Baseline
Oracle
STE.
HARE200
FAN
10 STE
20 AiNl
FARE-
STR.
OPE
ZIRE
Figure 3: Best-of-10 10 performance for Llama-3.1-8B generator across 4 challenging benchmarks with baseline (red line) and oracle (green line) performance with FARE and Sudge (SFR), Skywork Critic (SC), Compass-Judee (CJ), and TelghT Evaluator (STE) as basclines. FARE are the best small (< 14B) and large (2 20B) reranking models: FARE-20B achieves near oracle re-ranking performance MATH, while FARE-8B matches 70B judges.
8

---

Published as a conference paper at ICLR 2026
ONEP OA
Accurety
-5
Average
MMLL-Po
AIME24
Figure 4: Performance of downstream ERRPO-trained model with FARE as reference- -baeed verifiers. Moving from string based output matching or weaker verifiers to FARE brings tangible performance gains across both natural-lange natural-language based (e.g., GPQA) and math settings.
CompassJudger-7B. 7B, (Cao et al. 2024). We utilize the default pairwise reranking setup and prompt FARE to produce judgment directly without explanation, i.e., y (0,
Fig. shows best-of-10 10 performance with Llama-3.1-88 as generator with baseline greedy (red line) and oracle re-ranking performance (green line). FARE produce best-in-class reranking perfor- mance, with FARE-8B roughly matching the performance of larger (70B) judges in math settings and outperforming all similar sized judges in coding domains. FARE-20B excels in math domains, approaching oracle-level reranking performance on MATH, beating SFR-Judge-70B and Skywork- Critic-70B by 14 and 21 absolute points, respectively. Similarly, FARE-20B beats 70B+ judges by large margins in challenging coding domains. As we show in App. D.7 FARE improve the perfor- mance of other generators and FARE-20B improves significantly over gpt-oss-20B as a reranker.
Verifer for GRPO training. We train with WebInstruct-Verffer (Ma et al. 2025), multi-domain reasoning dataset, covering math, chemistry, etc. Verifier impact measured via the downstream performance of the trained policy model on a fixed evaluation suite of MMLU-Pro (Wang et al.)
2024c), GPQA-Diamond (Rein et al.) 2224), MATH-500, Minerva-Math (Lewkowycz et al. 2022). Olyh (He et al. 2024a), and AIME24 (Avg @ 32). We start from Qwen2.5-7B-BB Base (Yang al. 2024) with the default reward setup as Ma et al. (2025) (see App. C.2 and we compare against
training with the string- matching and trained verifier (General- Verifll-- from Ma et al. (202)).
Training with FARE-20B as a verifier improves downstream performance from 34.3 to 45.2, a nearly 11 point absolute gain, with improvements coming uniformly across the six benchmarks, as shown in Fig. Notably, with 77% fewer gradient updates
f our model was able to slightly beat the (Ma et al., 2025) several benchmarks: 38.9 vs. 38.8
performance of General-Reasoneer GPQA-Diamond, 38.4 vs 37.9 on OlynpiaBBench, and 13.9 vs. 13.8 on AIME24. This shows that using FARE-20B can significantly improve RL training convergence. Further, using FARE-SB and FARE-20B bring 8.8% and 14.1% relative gains over typically used string matching verifiers and 4.4% and 9.4% over General-Verifier which has been trained with in graining-distribung tion data. These gains appear for both natural language (MMLU-Prt GPQA) and math domains, showing that FARE can verify complex outputs across multiple challenging domains.
verifica-
Initialization for domain-specifie continual finetuning. We continually finetune FARE-20B for code evaluation with one round of RS-SFT to produce FARE-20B-Code
gob-sce-0B
IMRT-23B-Code
PAkS 216
goc upc-osis 1200
Coreestent Accursy
+8.42
train with only
62.4
E2-68
15K pairwise samples randomly chosen from Ace- Coder (Zeng et al. 2025). For evaluation, we use the recently released CodingJudge et 20255a) a pairwise benchmark covering code generation, code repair, and test-case quality eval- uation tasks. Fig. 5 which reports consistent ac- curacy across the three splits from gpt-oss-20IB, FARE-20B, MARE-20B-Code and gpt-oss- 120B for reference, shows that the first two splits are relatively easy, whereas test case evaluation is ex-
F73
(Tiang
34.40
30
Generation
Code Repair
Test Case
Figure 5: Continual training of FARE-20B for code evaluation with only 15K samples yields larger gains over gpt-oss-20B/120B.
For the same group size, SaneeaalRccasoner777 7B is trained for 700 steps with rollout batch size 768, whereas train for 120 update steps with rollout batch size 1024.

---

Published as a conference paper at ICLR 2026
tremely difficult: gpt-oss-1200 120B achieves only 32%. On the last task, FARE-20B improves
gpt-oss-20B by 11.67 absolute points, highlighting the benefits of large-scale evaluation training. Specialized continual training on coding tasks brings additional 10.48 absolute point improve- ment, with FARE-20B--CE outperforming even gpt-oss-1208 on average. In all, FARE can be readily adapted for specific applications with small amount of domain-specific data.
CONCLUSION
Using curated multi-task, multi-domain training mix and RS-SFT, train FARE, family of high performing and well-rounded automatic evaluators. FARE-8B challenges larger specialized evaluators and FARB-20B sets a new standard for locally hosted evaluators. Our evaluations include 7 challenging benchmarks and 3 practical downstream settings where we show that FARE are (1) effective reward models inference-time time, (2) effective verifiers for GRPO training, and (3) strong initializations for continual, domain-specits finetuning.
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai. Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et gpt-oss-120b. & gpt-oss-20b model card. arXiv preprint arXiv:2508. 10925, 2025.
Andrei Alexandru, Antonia Calvi, Henry Broomfield, Jackson Golden, Kyle Dai, Mathias Leys, Maurice Burger, Max Bartolo, Roman Engeler, Sashank Pisupati, et al. Atla selene mini: A general purpose evaluation model. arXiv preprint arXiv:2501. 17/95, 2025.
Salaheddin Alzubi, Creston Brooks, Purva Chiniya, Edoardo Contente, Chiara von Gerlach, Lucas Irwin, Yihan Jiang, Arda Kaz, Windsor Nguyen, Sewoong Oh, et al. Open deep search: Democ- ratizing search with open-source reasoming agents. arXiv preprint arXiv:2503. 20201, 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint urXiv:2204.0588 2022.
Maosong Cao, Alexander Lam, Haodong Duan, Hongwei Liu, Songyang Zhang, and Kai Chen. Compassjudger- All-in-one judge model helps model evaluation and evolution. arXiv preprint arXiv:2410.1623 16256, 2024.
Mert Cemri, Melissa Pan, Shuyi Yang. Lakshya Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. Why do multi-agent l1m systems fail? arXiv preprint 1XXv:2503.1366: 2025.
Tuhin Chakrabarty, Philippe Laban, and Chien-Sheng Wu. Can ai writing be salvaged? mitigat- ing idiosyncrasies and improving human-ai alignment in the writing process through edits. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp. 1-33, 2025.
Chi-Min Chan, Chunpu Xu, Jiaming Ji, Zhen Ye, Pengcheng Wen, Chunyang Jiang, Yaodong Yang. Wei Xue, Sirui Han, and Yike Guo. J1: Exploring simple test-time scaling for llm-as-a-judge. arXiv preprint 2505 2025.
Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang. Bryan Hooi, and Bingsheng He. Judgelrm: Large reasoning models as judge. arXiv preprint arXiv:2504.000 00050, 2025a.
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, and Heng Ji. Rm-rl: Reward modeling as reason- ing, 2025b.
/tatttss://rxivvii wat.aaiiivvv... org/ www.//r////..//eo////./.o/eeeeee 2505. 02387
Cheng-Han Han Chiang and Hung- Yi Lee. Can large language models be alternative human eval- uations? In Proceedings of the 6lst Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 15607-15631, 2023.
10

---

Published as a conference paper at ICLR 2026
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint Stti::2110.141660 2021.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:230.006767 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405. 240x:2405007863, 2024.
Keyu Duan, Zichen Liu, Xin Mao, Tianyu Pang. Changyu Chen, Qiguang Chen, Michael Qizhe Shieh, and Longxu Dou. Efficient process reward model training via active learning. arXiv preprint wrXiv22504.10550 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint MrXiv:2244.044777 2024.
Mohamed Amine Ferrag, Norbert Tihanyi, and Merouane Debbah. From 11m reasoning to au- tonomous ai agents: comprehensive review. arXiv preprint arXiv:2504.19678, 2025.
Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. How to evaluate reward models for rlhf. arXiv preprint
14872, 2024.
Jinlan Fu, See Kiong Ng. Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 6556- 6576, 2024.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint 2rXiv:2209.07858, 2022.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant, Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smymiis Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reason- ing models. arXiv preprint urXiv:2506. 0eee:2566604177 04178, 2025.
Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12941 2025.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi. Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, al. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.008:. 2022.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-leved bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024a.

---

Published as a conference paper at ICLR 2026
Jujie He, Tianwen Wei, Rui Yan, Jiacai Liu, Chaojie Wang. Yimeng Gan, Shiwen Tu, Chris Yuhao Liu, Liang Zeng, Xiaokun Wang, Boyang Wang, Yongcong Li, Fuxiang Zhang, Jiacheng Xu, Bo An, Yang Liu, and Yahui Zhou. Skywork-ol open series, November 2024b. :
//doi. /10 O. 5281/221122 /zwoddo. 6998085
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordonsi, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06: 06457, 2024.
Jian Hu, Xibin Wu, Zilin Zhu, Weixun Wang, Dehao Zhang, Yu Cao, al. Openrlhf: An easy-to- use, scalable and
rlhf framework. arXiv preprint arXiv:2405. 2024a.
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint arXiv:2408.0m 08435, 2024b.
Xinyu Hu, Li Lin, Mingqi Gao, Xunjian Yin, and Xiaojun Wan. Themis: A reference-free nlg evaluation language model with flexibility and interpretabilityy arXiv preprint arXiv:2406. 18365, 2024c.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint M.Niv:2410.212777 2024.
Dulhan Jayalath, Shashwat Goel, Thomas Foster, Parag Jain, Suchin Gururangan, Cheng Zhang, Anirudh Goyal, and Alan Schelten. Compute as teacher: Turning inference compute into reference-free supervision. arXiv preprint arXiv:2509. 14234, 2025.
Jiaming Ji, Mickel Liu, Josef Dai, Xuchai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang. and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via a human-preferenee dataset. Advances in Neural Information Processing Systems, 36:24678- 24704, 2023.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint vrXiv:2306.025611 2023.
Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, and Robby Tan. Codejudgebench: Benchmarking Ilm-as-ar ju-d- for coding tasks. arXiv preprint arXiv:2507.10553, 2025a.
Yuhua Jiang, Yuwen Xiong, Yufeng Yuan, Chao Xin, Wenyuan Xu, Yu Yue, Qianchuan Zhao and Lin Yan. Pag: Multi-turn reinforced 11m self-correction with policy as generative verifier. arXiv preprint a506v:2506. 10406, 2025b.
Ryo Kamoi, Yusen Zhang, Nan Zhang, Sarkar Snigdha Sarathi Das, and Rui Zhang. Training step- level reasoning verifiers with formal verification tools. arXiv preprint arXiv:2505.15960 2025.
Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, et al. A survey of frontiers in 1lm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint arXiv:2504.09037 2025a.
Zixuan Ke, Austin Xu, Yifei Ming, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. Mas-zero: Designing multi-agen systems with zero supervision. arXiv preprint 2rXiv:2505.14999 2025b.
Muhammad Khalifa, Rishabh Agarwal, I Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moon- tae Lee, Honglak Lee, and Lu Wang. Process reward models that think. arXiv preprint arXiv:2504.1683 2025.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin. Sungdong Kim, James Thome, et al. Prometheus: Inducing fine- finengrained evalua- tion capability in language models. In The Twelfth International Conference on Learning Repre- sentations, 2023.
12

---

Published as a conference paper at ICLR 2026
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaceun Kim, Dongkeun Yoon, Gui- jin Son, Yejin Cho, Sheikh Shafayat, Jinheon Back, et al. The biggen bench: A principled benchmark for fine-grainer evaluation of language models with language models. arXiv preprint arXiv:2406.05761 2024a.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01 0335, 2024b.
Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. No free labels: Limitations of llm-as-a-judge without human grounding. arXiv preprint arXiv:2503.0500 2025.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406. 18629, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. TŞ"" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating aligmment. arXiv preprint arXiv:2310.05470, 2023.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap. Tianhao Wu, Banghua Zhu, Joseph E Gon- zalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. / 1939, 2024.
Xinbin Liang, Jinyu Xiang, Zhaoyang Yu, Jiayi Zhang, Sirui Hong, Sheng Fan, and Xiao Tang. Openmanus: An apen-souree framework for building general ai agents, 2025. URL https:
//dii.org/10.522/2 wtw.oo///......2eeeeeeeee 2ee///////eoo.ooo www.ood...1.1..0 15186407
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20000. 2023.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410. 18451, 2024a.
Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. Skywork-reward-vv: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352, 2025a.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https: OCu7
Junteng Liu, Yuanxiang Fan, Zhuo Jiang, Han Ding, Yongyi Hu, Chi Zhang, Yiqi Shi, Shitong Weng, Aili Chen, Shiqi Chen, et al. Synlogic: Synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond. arXiv preprint ar arXiv:2505.1900
2025b.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint 2505 250.245000.4844.. 2025c.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using g with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 251 2511-2522, 2023b.
13

---

Published as a conference paper at ICLR 2026
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint FXX1:24410.1618, 2024b.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.02453 2025d.
Zuxin Liu, Thai Hoang. Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh RN, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances Neural Information Processing Systems, 37:54463-54482. 2024c.
Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. Uncertainty- aware reward model: Teaching reward models to know what unknown. arXiv preprint NrXiv:2410.00847, 2024.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprini urXiv:2406. 06592, 2024.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jef- frey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol- preview with a 1.5b model by scaling rl. /ttp::://ppetttyyrdaiooooooo
Sat.aaalleeSuusssgg--oooo Preview htt.://ww....a........o..eeaaaaa
2025. Notion Blog.
Xueguang Ma. Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. Gensral-reasoner: Advancing Ilm reasoning across all domains. arXiv preprint MXXIv:2505,14630 2025.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint 221.22400..222... 2024.
Yujun Mao, Yoon Kim, and Yilun Zhou Champ: A competition-level dataset for fine-grained analyses of Iims' mathematical reasoning capabilities. arXiv preprini arXiv:2401.06961 2024.
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Tre- bacz, and Jan Leike. Llm critics help catch 11m bugs. arXiv preprint urXiv:2407.00215, 2024.
Xuan-Phi Nguyen, Shrey Pandit, Revanth Gangi Reddy, Austin Xu, Silvio Savarese, Caiming Xiong, and Shafiq Joty. Str-deepresearch: Towards effective reinforcement learning for autonomously reasoning single agents. arXiv preprint wwXivv2509.06288 2025.
OpenAl. Introducing gpt-4.1 in the api, April 2025. URL gpt-4-11
OpenAl. Deep research system card. Technical report, OpenAl, August 2025. URL https:
//cen.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139, 2024.
Arjun Panickssery. Samuel Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37:68772-68802. 2024.
Junsoo Park, Seungyeon Jwa, Meiying Ren, Dacyoung Kim, and Sanghyuk Choi. Offsetbias: Lever- aging debiased data for tuning evaluators. arXiv preprint 2rXiv:2407.06551, 2024.
14

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2335./8829 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1-67, 2020.
Revanth Gangi Reddy, Tarun Suresh, JaeHyeok Doo, Ye Liu, Xuan Phi Nguyen, Yingbo Zhou, Semih Yavuz, Caiming Xiong, Heng Ji, and Shafiq Joty. Swerank: Software issue localization with code ranking. arXiv preprint arXiv:2505.07555 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. First Conference on Language Modeling, 2024.
Hayley Ross, Ameya Sunil Mahabaleshwarkar, and Yoshi Suhara. When2call: When (not) to call tools. arXiv preprint 0rXiv:2504.18857/ 2025.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Jon Saad-Falcon, Rajan Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri. Lmunit: Fine-grained evaluation with natural language unit tests. arXiv preprint arXiv:2412. 2024.
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking- -la---l--------deee arXiv preprint arXiv:2501.1809, 2025.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207. 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634 2017.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhí framework. arXiv preprint arXiv: 2409. 19256, 2024.
Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. Skywork critic model series. htttps:/huuggingtee wwwggingaacc.co///ooo Swwoo//kkwwkkk
ct Skywork September 2024. URL https://
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in neural information processing systems, 33:3008-3023 -0201, 2020.
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. Judgebench: A benchmark for evaluating Ilm-based judges. arXiv preprint arXiv:2410. 12784, 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv: 2503:22503.1999 2025.
Mistral Team.
Un ministral, des ministraux, 2024.
URL https://mistzal.aii/ce
ministraum raux
Mistral Team.
Mistral small 3. 2025a.
URL
mist sastral 1 smal 3
15

---

Published as a conference paper at ICLR 2026
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025b. URL
32b/
Team Kimi, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint ar arXiv:2507.20534 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint DxXiv:3307009288 2023.
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. Foundational autoraters: Taming large language models for better automatic evaluation. arXiv preprint arXiv:2407. 108/7, 2024.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt good nlg evaluator? preliminary study. In Proceedings of EMNLP Workshop, pp. 2023a.
Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafiq Joty. Direct judgement prefer- ence optimization. arXiv preprini arXiv:2409. (tfl:2409..4644,, 2024a.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint vrXiv:2305.17924 2023b.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: A label-free step-by-ster verifier for Iims in mathematical reasoning, arXiv preprint arXiv:2312.0893: 2023c.
Tianlu Wang. Ili Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang, Maryam Fazel -Zrannd, Jason Weston, and Xian Li. Self-taught eval- uators. arXiv preprint arXiv:2408. 02066. 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self- coffissnsstt improves chain of thought reasoning in language models. arXiv preprint 22Xx::2003.11777 2022.
Yubo Wang, Xueguang Ma. Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Jian,, et al. Mmlu-pro A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2024c.
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scoweroft, Neel Kant, Aidan Swope, et al. Helpsteer: Multi- 2023d.
attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.099
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. Advances in Neural Information Processing Systems, 37:1474- 1501, 2024d.
Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Daniel Egert, Ellie Evans, Hoo-Chang Shin, Fe- lipe Soares, Yi Dong, and Oleksii Kuchaiev. Helpsteer3: data to empower inference-time scaling in open-ended general-domain tasks. arXiv preprint M.Xiv:2503.04378, 2025a.
feedback and edit
Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Hoo-Chang Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Dong, and Oleksii Kuchaiev. Srppstter3-preferenee Open human-annotua preference data across diverse tasks and languages. arXiv preprint arXiv:2505.1 1475, 2025b.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint 1oXiv:2109001653 2021.
16

---

Published as a conference paper at ICLR 2026
Zhepei Wei, Wenlin Yao. Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, et al. Webagent-rl: Training web agents via end-to-end multi-turn rein- 2025.
forcement learning. arXiv preprint arXiv:2505.
Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swamnadeep Saha. J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning. arXiv preprint arXiv:2505. 103200 2025.
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-mest aan-as-a-meta-judgee arXiv preprint arXiv:2407. 19594, 2024.
Xiaobao Wu, Sailing ai by the stars: survey of learning from rewards in post-training and test-time scaling of large language models. arXiv preprint arXiv:2505.02686 2025.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504. 113433 2025a.
Wei Xiong, Wenting Zhao, Weizhe Yuan, Olga Golovneva, Tong Zhang, Jason Weston, and Sain- bayar Sukhbaatar. Stepwiser: Stepwise generative judges for wiser reasoning. arXiv preprint arXiv:2508. 19229, 2025b.
Austin Xu, Srijan Bansal, Yifei Ming, Semih Yavuz, and Shafiq Joty. Does context matter? contextualjudgebench for evaluating llm-based judges in contextual settings. arXiv preprint arXiv:2503. 15620, 2025a.
Austin Xu, Yilun Zhou, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. J4r: Learning to judge with equivalent initial state group relative policy optimization. arXiv preprint arXiv:2505. 2025b.
Yuchen Yan, Jin Jiang, Zhenbang Ren, Yijun Li, Xudong Cai, Yang Liu, Xin Xu, Mengdi Zhang, Jian Shao, Yongliang Shen, et al. Verifybench: Benchmarking reference-bar based reward systems for large language models. arXiv preprint arXiv:2505. 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 2024.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint 200.:2307.1028 2023.
Ziyi Ye, Xiangsheng Li, Qiuchi Li, Qingyao Ai, Yujia Zhou, Wei Shen, Dong Yan, and Yiqun Liu. Beyond scalar reward model: Learning generative judge from preference data. arXiv preprint arXiv:2410.03742 2024.
Jiachen Yu, Shaoning Sun, Xiaohui Hu, Jiaxu Yan, Kaidong Yu, and Xuelong Li. Improve llm-as- a-judge ability as general ability. arXiv preprint arXiv:2502. 1689, 2025a.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source 11m reinforcement learning system at scale. arXiv preprint 2rXiv:2503.144770 2025b.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. arXiv preprint arXiv:2002.044. 2020.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning 2024.
17

---

Published as a conference paper at ICLR 2026
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-1544 2022.
Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. arXiv preprint arXiv:2502.01718 2025.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023..
Kaiwen Zha, Zhengqi Gao, Maohao Shen, Zhang-Wa Wei Hong, Duane S Boning, and Dina Katabi. RI tango: Reinforcing generator and verifier together for language reasoning. arXiv preprint arXiv:2505. 15034, 2025.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow genera- tion. arXiv preprint arXiv:2410.1072 2024a.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. 1.5200, 2024b.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang. Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.073000 2025.
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, al. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint (tth:2504.0089)) 2025.
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical rea- soning. arXiv preprint arXiv:2412.06552 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folloiigg evaluation for large language models. arXiv preprint arXiv:2311.07911 2023.
Yefan Zhou, Austin Xu, Yilun Zhou, Janvijay Singh, Jiang Gui, and Shafiq Joty. Variation in verification: Understanding verification dynamics in large language models. arXiv preprint arXiv:2509.17991 2025a. /Rl.ptt.ps://arii..cor http//aas/22..... 2559.
Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, and Shafiq Joty. Evaluating judges as evaluators: The jetts benchmark of llm-as-judges as test-time scaling evaluators. arXiv preprint arXiv:2504. 15253, 2025b.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indrancil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406. 15877, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint
2019.
18

---

Published as a conference paper at ICLR 2026
APPENDIX
Use of LLMs. We minimally used LLMs during the writing process to (1) brainstorm revisions for short text snippets and (2) check for grammar, spelling, and writing mistakes.
Reproducibility Statement. We have released FARE checkpoints publicly and detailed training data sources in App.
Ethics Statement. The proliferation of LLM-based systems has raised concerns centered around model biases, faithfulness (c.g.. hallucinations), and accuracy. As a result, automatic evaluation with LLMs has emerged as a popular paradigm for scalable evaluation of LLM outputs. Our work falls under this paradigm, training foundational automatic evaluators for complex reasoning settings.
Despite empirical successes, automatic evaluators are not panacea for unreliable generators: Au- tomatic evaluators themselves may not be free from bias or inaccuracies. Towards quantifying any bias in our evaluators, evaluated FARE RM-Bench, which aims quantify how robust eval- uators are to style and subtle mistakes in responses. However, we advise extensive bias testing and corrective finetuning before deploying automatic evaluators in the wild. When feasible, advocate for having humans audit deployed evaluators for bias and systematic inaccuracies.
A EXTENDED RELATED WORK
Here, we cover recent advances automatic evaluation that are non-generative. Scalar reward models (RMs) (Cobbe et al. 2021], which output single scalar score for given question g and response r were popularized originally within the context of reinforcement learning from human feedback (RLHF) (Ouyang al. 2022, Stiennon al.) 2020 Ziegler et al. 2019). Here, pairwise human preferences were used to train a RM, which is then applied during RL-based alignment, e.g., with PPO (Schulman al. 2017) or DPO (Hosseini et al. 2024). Past work has focused on dataset curation (Jiang et al. 2023 Liu et al.
[024a]] 2025a Dong et al. 2024) and experimenting with 2024).
different loss formulations (Liu et al. 2024a Lou
The above class of reward models operate at the outcome level, with the entire response being assigned single score. Step-level reward models, known process reward models, attempt to provide denser feedback by assigning each "step" in a response a score. Training PRMs relies on step-level labels from humans (Lightman al., (203) models (Duan et al., 2025)) or using Monte Carlo simulation to estimate step-level quality (Wang et al. [023c] Luo et al. 2024 Xiong et al.,
20256 Zhang et al., (025). Both approaches have associated drawbacks: Step-lever annotation is rarely scalable, while Monte Carlo simulation requires careful filtering to ensure high quality.
While RMs and PRMs remain popular paradigms, recent approaches have found that generative evaluators can better leverage test-time compute for stronger evaluation performance (Zhang et al.) 20246. Mahan et al. 2024 Liu al. 2025d),, inspiring recent advances in generative evaluators, as discussed in
B
DATA AND TRAINING DETAILS
B. TRAINING DATA
We enumerate our training data sources in Table 5 and present breakdowns by curation approach
in Table 6 We took efforts to decontaminate our training sets with N-gram matching approaches, following Guha et al. (025). For the Synthetic data approach, we use 12 unique genera- tors, covering mixture of weak and strong models: Ministral-8B (Team) (024). Mistral-Small 24B Team 2025a). Gemma 3 12B (Team et al., 2025), Qwen2.5 7B, 32B (Yang et al. 2024). Qwen- QwQ (Tem 2025b), Qwen3-30B- 3--33-30-33- (Yang et al. 2025), Llama-3. 1-8B, -lama-3.3-700 (Dubey et al. 2024). GP (Hurst et al. 2024), and GPT4.I-nand 4.1 (OpenAl 2025) for seed datasets with verifiable answers. To increase diversity randomly select prompt template from a preset list for each question in the seed dataset and sample multiple responses at varying temperatures (c.g., 0.0, 0.3, 0.5, 0.7..... For open-ended datasets, such tool-use datasets, enumerated common
19

---

Published as a conference paper at ICLR 2026
Table 5: Data sources used create our training set.
Domain
Approach
AcIPRM
Duan et ml
2025
Step-Level
Math
Existing
Beavertails Preference
Pairwise
Safety
Existing
Code Preference Pnirs
Veroral one Preference- Pairs
Pairwise
Code
Existing
DeepScaleR
Pairwise, Verification
Math
Synthetic
Folio
Han ct
2022
Pairwise, Verification
NL Reasoning
Synthetic
FoVer
Katnoo
Step Seve-Level
Math
Existing
HelpSteer
Wany cl al
402.1
Single rating
Chat
Existing
HelpSteer2
Single rating
Existing
HelpSteer3
Wany al
217-4
Pairwise
Code, NI. Reasoning
Existing
HH-RLHP Harmless
Bal et
Lrnnguli et 2022
Pairwise
Safety
Existing
LAMP
Chakraburly cl
2025
Pairwise, Single rating
Chat
Existing
MATH
Hemtyyck
Pairwise, Verification
Math
Synthetic
McmGPT
MenGr L.MenuP
P-DDOODDaaa
Pairwise, Verification
Tool-Use
Existing
OffsetBias
Park et
2022
Pairwise
Chat
Existing
ReClor
Yu al 10,21
Pairwise, Verification
NL Reasoning
Synthetic
StepDPO
Lai ct 2174
Pairwise, Verification
Math
Existing
StralegyQA
2024
Pairwise, Verification
NL Reasoning
Synthetic
SWEGym
Finh or mh 40.4
Verification
Code
Existing
SWERank
Keday
Verification
Codc
Existing
SynLogic
Liu et a.
20256
Pairwise, Verification
NL Reasoning
Synthetic
Tulo-V3-IF DPO data
2024
Pairwise
Chat
Existing
WebDPO
WabDPO
Pairwise, Verification
NL Reasoning
Existing
When2Call Preference
et
2025
Pairwise
Use
Existing
Xlam-60K
Lib et ab.
2T242
Pairwise, Verification
Tool-Use
Synthetic
Table 6: Breakdown of training data by curation approach.
Task
Pairwise
Step-level
Verification
Rating
Synthetic
45. 35,2%
54.8%
Existing
27.2%
36.0%
19.7%
17.1%
Domain
General Reasoning
Muth
Code
Tool-use
Chat
Safety
Synthetic Existing
28.3% <1%
52.1% 50.4%
19.6%
14.7%
1%
25.7%
7.2%
errors found in tool calling outputs, such as invalid input types, missing arguments, malformed json format, etc., and then programmatically injected errors into ground-true -trthhh correct responses.
B.2 TRAINING DETAILS
We train batch size 128 and constant learning rate of le-6 and choose per-iteration rollout batch sizes of 50,000 and 250,000 for FARE-SB and FARE-20B, respectively. In the latter case, make the practical trade-off of RS-SFT iterations for training speed, reducing the number of times we need to reset the model for rollouts, etc. We use modified version of the OpenRLHF framework (Hu et al. 2024a) for training.
Qwen3 cold-start SFT. Public discussion from members of the Qwen organization indicate that post-trained versions of Qwen3 are difficult to continually finetune, and recommend starting from base mo Therefore, opt to cold-start SFT with iteration of rejection sampling data collected from Sive2.5532B-Insttnee As we show in App. D.6] while this cold-start model does match Qwen3-8B, FARE-SB outperforms the non-thinking Qwen3-8B many static evaluation benchmarks.
We hypothesize that a relatively short, general- pprepsse alignment phase prior evaluation-specific finetuning could further improve performance. While we did not attempt this, we believe this line of experimentation of interest for future work.
See, for example, this TwitterXX post base models I don't 't think so..."
Instruct models after RL. will pose difficulty for finetuning, hut
20

---

Published as a conference paper at ICLR 2026
C BENCHMARK AND BASELINE DETAILS
C.I CORE BENCHMARKS
For core benchmarks, we select set of challenging and contemporary benchmarks for evaluating automatic evaluators:
JudgeBench (Tan et al. 2024): A pairwise benchmark focused on evaluating LLM-as-jude judge models in reasoning settings, covering math, code, logical reasoning, and knowledge- reasoning. Responses are generated using GPT-4o.
ReasoningludgeeBnnch (Xu et al., 2025b): A pairwise benchmark that covers more diverse reasoning settings, such as multi-hop, causal, and domain-specifie reasoning. Responses are generated using GPT-4o.
PPE Correctness FFrick et al. 2024): A pairwise benchmark that covers reasoning and instruction following tasks with objectively correct answers, using seed datasets like MATH Hendrycks al. (2021). GPQA (Rein et al. (024), MBPP+ (Liu et al. (023a)), 2024c), and IFEval (Zhou et al. (023). Responses are generated using variety of weaker models, e.g., Gemma-2-9B.
MMLU-Pro (Wang et al.
RM-Bench (Liu al.
2024b): A pairwise benchmark that evaluates how robust evaluators are to stylistic biases by evaluating on pairs of responses with subtle yet critical differences.
When2Call (Ross et al. 2025): A pairwise benchmark that covers appropriate selection user prompt. We use the LLM-as-judge test split, which comprises 300 unique prompts. Each prompt has four candidate answers (refusal, direct response, tool call, follow-up question), of which one response is correct. We form all pairs, yielding 900 total pairwise comparisons.
of tools (or refusals) in response
ProcessBench (Zheng et al. 2024): ; A step-level benchmark that evaluates the ability to identify step-level errors mathematical reasoning across easy (GSM8K and MATH) and hard (Omni-Math and OlympiadBench) questions.
VerifyBench Yan et al. 2025): A reference-basee verification benchmark, comprised of Easy and Hard splits, that evaluates verifier ability to identify equivalent final answers.
For all core benchmarks, utilize officially reported numbers when available. Otherwise, run the corresponding baseline ourselves, using any prompt templates released with evaluators.
For pairwise benchmarks, select our baselines from (1) existing multi-task foundational evalua- tors, (2) recently released RL-trained judge models, and (3) strong grong-performing specialized judges:
RISE-Judge Yuu et al. 2025a): Pairwise judges trained with SFT then DPO to perform pairwise evaluation. Initialized from Qwen2.5 models.
Self-Taught Evaluators (Wang al., 2024b): A pairwise judge trained with iterative SFT with training data generated in the loop. Initialized from LJama-3.1-70B.
EvalPlanner (Saha et al. 2025): Pairwise judges trained with iterative SFT and DPO on a small seed dataset, with emphasis on learning how to plan for evaluation tasks. Initial- ized from Llama-3.3-70BB
RM-RI (Chen et al. 2025b): A family of pairwise judges trained with GRPO, initialized from -eepSeek-distilled Qwen models.
J1 (Whitehouse al.
2025): ; A pairwise and single-rating judge trained with GRPO. Ini-
tialized from Llama-3. 1/3.3 models.
CompassJudger (Cao et al. 2024): : A family of foundational evaluators trained with large-
scale SFT. Initialized from Qwen2.5 models.
Atla Selene (Alexandru al. 2025): ; foundational evaluator trained with large-scale
preference optimization. Initialized from Llama-3.1-8B.
We run gpt-oss variants with low reasoning, as (1) FARE-88 BB trained initialized from gpt-oss-200 20BB low, and (2) evaluation often demands low-latency, making long CoT undesirable if they can be
21

---

Published as a conference paper at ICLR 2026
avoided. For GPT-5, use the default API settings (medium reasoning).
For pairwise benchmarks, we use reported scores from Whitehouse et al. (2025) ): Xu et al. (202bb Liu et al. (202dd).
For ProcessBench,
officially reported numbers in Zheng al. (2024), which includes Sky- workPRMs (He et al,, 2024b). Math Shepherd PRM (Wang et al.
2023c). ActPRM (Duan et al, 2025), and Mwth PRMs (Zhang et al. 2025 We additionally report results from generative baselines: RL Tango (Zha et al. (2225) and StepWiser (Xiong et al. 2025b). For VerifyBench, we
use reported scores from the original paper (Yan al. 2025)
C.2 DOWNSTREAM SETTINGS
Reward model for inference-time scaling, We compare our models against the following base- lines, representing best-in-class performers as reported in JETTS. We utilize reported numbers rectly except for CodpassJudger, which was not included in the original JETTS evaluation. As such, we run CompassJudger ourselves.
SFR-Judge-8B, 70B (Wang et 2024a): A family of multi-task evaluators. Among the highest performing small and large judges on JETTS. Initialized from Llama-3. models.
Skywork-Critic-B88 70B (Shiwen et al. 2024): Two pairwise-specfic evaluators that do not output explanations. Among the highest performing small and large judges JETTS. Initialized from Llama-3.1 models.
Self- BeafTTaugt--Ealuuaae 70B (Wang et al. Initialized from Ilama-3.1 models.
2024b A strong performing large judge model.
CompassJudger-7BB 14B (Cao et al. 2024 As described above.
Verifier during GRPO training, We adopt the settings of Ma et al. (2025). which train General- Reasoner, family of reasoning LLMs varying model sizes using the WebInstruct- Sellntruct-Verified train- ing dataset. In particular, we train with standard GRPO, i.e., without dynamic sampling or clip higher modifications, initializing from Qwen2.5-7B-Bal We the same conditional reward setup as General- Reasoner:
If the solution parsing fails, then reward is set to -0.5.
If solution successfully parsed and deemed correct by the verifier, is assigned a reward of 1 plus length penalty of:
0.05 min (10, |1 len (ground truth) len (model response) }}-
The training framework is based on the ver 1 (Sheng et al. 2024). We use rollout batch size 1024, max response length of 4096, group size of 8, temperature of 1.0, KL coefficient of 0.001, and learning rate of 5c-7.
Initialization for domain-speeific finetuning. We randomly sample 15,000 pairwise samples from AceCoder (Zeng et al., 2025) and perform one round of rejection sampling. We adopt training setup of FARE-20B: direct judgment ratio of 60% and continuous curriculum. We train for epochs with batch size 256 and cosine decay learning rate peaking at le-5. We then evaluate on CodinJJudgeBench, reporting consistency accuracy.
Note that Bedin reports an unconventional pairwise metric, employing Z-score nor- malization between the two consistency runs. Their implementation not publicly available, and their paper lacks concrete implementation details. As such, we resort to consistent accuracy, which
more commonly used in pairwise benchmarks, e.g., (Tan et al. 2024 et 2025a b).
2023 Xu et al.)
D ABLATIONS, ANALYSIS, AND ADDITIONAL RESULTS
D.1 TRAINING RECIPE ABLATIONS.
In Table 7 we ablate three components of our training recipe and report the average on our five pairwise benchmarks and ProcessBench. First, we train multiple checkpoints using RS-SFT varying
22

---

Published as a conference paper at ICLR 2026
80
Consistency
Qwen3 8B
Qwen2.5-77
1.0
20
Number of Training Samples
Figure 6: Pairwise positional robustness emerges as at large data scales, shown with training FARE-8B and an earlier development run initialized with Qwen2.5-77BInntrree.
the proportion of direct judgment data from 30% to 70%. Direct judgment data affects FARE-8B and FARE-20B differently: endpoints (30% or 60-70%) show the best performance for FARE-20B, whereas performance peaks at 40% for FARE-8B. As such, we choose 40% and 60% to train FARE- 8B and FARE-20B.
We also ablate different strategies for training with direct judgment data specific to gpt-oss, which trained output intermediate CoT before responding. We can either keep this CoT with direct judgment data or remove the intermediate CoT by going directly to the assistant turn. The former more closely mimics the training distribution of gpt-oss, but undermines the intended purpose of direct judgment data of isolating outcome correctness training signal, as discussed in $ [3. The latter is cittoof-trannin--iisee distriitdditiiittiinnnnn but more effectively isolates training signal. As seen Table (bottom right), removing the intermediate CoT leads to gains in both pairwise and step-level settings.
Finally, we measure the impact of the continuous curriculum as compared a random data shuffling strategy with FARE-8B. As shown in Table 7 (bottom left), the continuous curriculum leads to minimal drops in pairwise performance but large gains in ProcessBench.
D.2 HOW DOES DIRECT JUDGMENT PROMPTING AFFECT PERFORMANCE?
Many settings demand low latency, such inference-time reranking evaluating rollouts during RL training. Here, we study how performance varies when FARE are prompted to skip the cri- tique e and directly output a judgment j. For FARE-20B, this involves additionally skipping the intermediate CoT, directly outputting from the assistant turn, making nut-of-distribution relative to gpt-oss-20B's original training setup. We see that performance improves for FARE-SB, but de- grades for FARE-20B. Such results for FARE-8B are in-line with prior work, which finds that direct judgment- like inference leads minimal drops in performance. For FARE-20B, we hypothesize that the post-training of gpt-oss-20B instills strong prior in favor of generating intermediate COT. Even training with direct judgment data, removing such CoT detrimental. Nonetheless, perfor- mance does not universally degrade, with When2Call performance increasing by nearly 13 points.
D.3 ROBUSTNESS TO PAIRWISE POSITIONAL BIAS EMERGES WITH DATA SCALE.
A known issue in pairwise evaluation is inconsistency WWag et al. 2023b), a form of positional bias where the evaluator judgment changes based on the order of responses the input prompt. During training, we observed that our judges become more consistent as a function of data scale; Fig.
Table 7: Ablation study varying proportion of direct judgment data, use of continuous curriculum (8B model), and ablating strategies for using direct judgment data for gpt-oss (20B model).
& dircct
Qwen3-SB- SB-dd-SB-ollSian Models
oss 20B Models
Judgmeet
Puivise
ProcessBench
Pairvis
ProcessBench
Average
61.14
58.03
70.82
82.50
50
61.56
56.85
59.31
71.97
83.00
77.49
60
62.31
54.73
58.52
72.5%
84.40
78.49
62.67
56.49
78.16
Curicolam
Phinwise:
ProassRench
Averige
Kogp Cal?
Painwis:
PreeasBench
Avenage
Yes
61.14
58.03
59.59
Yes
67.50
81.61
74.72
No
61.49
53.43
57.46
No
69.81
82.55
76.15
23

---

Published as a conference paper at ICLR 2026
Table Performance with and without critiques (and CoT for FARE-20B) for pairwise benchmarks (left) and ProcessBench (right). Directly prompting for a verdict universally improves performance for FARE-8B, but degrades performance for FARE-20B.
JudgeBench
RM
When2Call
Average
Ciympiad
Average
FARE-SB FARE-BB, no critique
55.71
79.2
00100
52.60
64.9
81.8
86.55
09.2
68.5
68.6
59.0
58.5
63.7
F-R
64-29
$7.05
74.4
90.5
76.67
72.6
89.8
87.8
80.0
79.9
84.4
FARE-20B,
crilique or CoT
62.00
55.23
68.9
85.5
89.11
72.1
79.8
73.2
70.0
70.3
73.3
Table 9: Ablation comparing training task-specific evaluators against training single multi-task evaluator. All models initialized from Qwen2.5-7B-Instteee
JudgeBench
ReasoninggTuggeBech
ProcessBench
Pairwise only
53.71
48.78
Step-level only
55.
Multi-task task
58.00
$1.11
55.81
shows the progression of pairwise consistency on the five pairwise benchmarks $ 4. over the course of training for FARE-SB and an earlier training run which initialized from Qwen2.5- 7B-Instruct. Both models steadily become positionally robust over the course of training, with the weaker Qwen2.5-77 Boon2.5-7B-Inaee -B-Insruutt showing substantial gains. This reveals that scaling evaluator training data can mitigate common judge biases, complementing mitigation strategies that use data augmentation (Saha al., 2025), label balancing (Cao al. 2024, Wang al.) 2024a), and RL- based reward or algorithmic methods (Whitehouse al. 2025
Xu et al. 2025b).
D.4 MULTI-TASK EVALUATOR TRAINING OUTPERFORMS SINGLE-TASK TRAINING.
Here, we compare training evaluators with multi-task data mix against training per-task evaluators. We initialize train from Qwen2.5-7B IIInttuut, and train a pairwise only evaluator, a step-level only evaluator, and one with pairwise and step-level data. We train all models for equivalent number of training input samples with RS-SFT on an earlier version our final data mixture. We report results on JudgeBench, ReasoningJJdgeeBencce and ProcessBench in Table We observe that multi-task training leads to significant gains over single-task evaluators, especially pairwise evaluators, with larger gains coming in pairwise evaluation settings that step-level evaluation. This result is intuitive: The skill of identifying errors at a granular (step) level improves pairwise evaluation by endowing the evaluator with the ability to catch more subtle mistakes in each response within the pair.
D.5 SINGLE-RATING EVALUATION.
We additionally evaluate FARE on Single Rating tasks with BiGGen- Bench (Kim et al. 2024a) and FLASK (Ye et al. 2023), two chat-centric evaluation datasets with human annotated 1-5 ratings. We measure Pearson correlation with human annotations, and report results in Table 10 Single-rating widely used evaluation task in reasoning settings, and thus constituting the smallest proportion of Nonetheless, FARE are competitive with chat-focused judge models, with FARE-SB outperforming foundational judge models like SFR-Judge-8B and 12B and FARE-20B approaching the performance of SFR-Juuge-70B: We use reported values from baseline papers, including LMUnnt (Saad-Falcon al. 2024). Atla Selene (Alexandru al. 2025), and SFR-Judge (Wang et al. 2024a).
our training data, as shown in Fig. 2
D.6 COMPARISON AGAINST GENERAL-PPRPOSS MODELS
Here, we compare FARE against general-purpose LLMs, selecting popular reasoning and non- reasoning models. Table II shows our results. Our cold-start SFT for Qwen3-8B produces the weakest Qwen3 variant, expected. However, after undergoing iterative SFT, FARE-SB surpasses Qwen3-8B on multiple benchmarks, improving from 43.56 to 51.05 on ReasoningJudgeeBet 56.7 63.5 ProcessBench. Likewise, yielding substantial improvements in reasoning, tool-calling, and step-level evaluation. The result- ing checkpoint approaches gpt-oss- 120B number of benchmarks.
and able to improve gpt-oss-20B across the board,
24

---

Published as a conference paper at ICLR 2026
Table 10: Single rating performance, with best and second-best performance in each section marked. Despite being trained with a focus on reasoning settings, FARE perform competitively in single- rating evaluation in chat settings.
FLASK
BiGGen Bench
Average
GPT-4o-mini
0.630
0.600
0.615
Glider-3.8B
0.615
0.604
0.610
FlowAl-Jugge-3888
0,400
0,460
0.430
Prumetheus-2-7B
0.470
0,500
0.485
Auto-J-13B
0,350
0.300
0.325
Themis-8B
0.540
SFR-Judge-81
0.520
0.590
0.555
SFR-Judge- 12B
0.590
0,570
0.580
Atla Selene 8B
0.613
0.584
0.599
LMUnit-8B
0.600
0.645
0.623
FARE-8B
0.611
0.616
0.591
GPT-4o
0.690
0,650
0.670
Nometheus-8x77
0.540
0.520
0.530
SFR-Judge-70B
0.660
0.650
0.655
LMUnit-70B
0.720
0.677
0.699
FARE-20B
0.649
0.616
0.633
Table Comparison of FARE against their initial models and other popular general-purpose mod- els. indicates some results reported in Whitehouse et al. (0025) or Zheng et al. (2024).
JudgeBench
Reasoningludgellesch
PPE Correctness
RM Bench
When2Call
Avg. consistency
ProcessBench
Beenn-88--ollSinn
Qwen3-82 (a-a3-8B-eee 2B-noo-thinkingg
52.27
43.56
64.8
79.9
61.78
74.01
56.7
FARE 8B
55.71
$1.05
63.8
79.2
80.33
82.28
63.5
gpt-oss-20B 20B (low)
39.43
50.51
71.7
89.9
61.33
77.83
73.9
gpt-oss- 120B (low)
70.29
58.26
77.8
92.0
70,00
$4.09
$3.4
Deepseek R1-671B"
68.90
58.53
76.5
88.6
$1,00
GPT-4.1
GPT-4o
50.29
45.25
68.9
80.
67.44
78.02
61.9
ol-mini!
61.20
71.3
80.8
$7.9
Table 12: Full results on JETTS. Numbers in bold indicate that the judge reranking was helpful, i.e., performance greater than baseline (greedy) performance.
FARE-SD
FARE-20B
gpt-ass-20B (CgstreuuProogt]
Model
Performance
Perforance
[ queerompt)
MATH
Llama-3.1 3.1---.1------- -1--setnsttett
24.70
53.47
35.73
50.83
49.85
29.83
Jlmm--3.1177 1-770-IInstrc
43.81
68.35
53.47
65,41
64.66
$1.06
Qwen2.5. Instruct
Qwen2.5 Ins-mmtee
62.99
82.78
70.17
79.98
78.70
70.32
GSMSK
Llama-3.1 3.1-BB- Instruct
85.67
96.41
92.04
94.77
94.77
93.78
Llama-3.1
Qwen2.5- 101-225-28---nmmm
95
98 98.56
96.21
96.29
96.74
95.75
Qwen2.5- -in-ttt-TB-
95.68
97.88
95.98
95.75
95.98
95.53
Llama-3.1 3-1--88-- Instract
47.41
71.48
51.85
58.52
$6.67
55.56
Owen2.5- 3IB Instruct
75.19
85.56
70.00
77.78
79.26
74.881
Qwen2.5.72
MBPP
Llama-3.1 I-n---nttttttt
$4.50
76.46
59.79
68.78
68.25
63.49
Llama-311 1.1-100000 101-708--aae Instroct
65
83.07
62.17
67.72
68.78
67.99
Qwen2.5-32B Qwen2.5 I28-Instruct
76.19
84.66
75.40
78.31
78.31
78.04
HamanBvval
Llama-3.1 1-1---B--nsstee Instruct
63.
79.88
64,02
74,39
74.39
68.29
Llama-3.1-
75.61
Owenn.5-332BB Instnnttt
81.10
93.29
83.54
91.46
90.24
87.850
Qwen2.5-77 100---78--nnaee Insttntt
82.32
93.90
86.59
90.24
91.85
86.59
Llama-3.1
Ilama-1 3.1- Insrrace
45.44
62.63
43,86
46.93
47.54
45.88
Qwen2.5-333 Instrutt
45.53
65.18
47.02
49,39
48.95
48.25
Qwen2.5-P7
25

---

Published as a conference paper at ICLR 2026
D.7 ADDITIONAL JETTS RESULTS.
In Table 12, we present results full results on JETTS. Concretely, we present (a) prompt ablation, denoted [Cri qquerooptt]rr where prompt FARE-20B and gpt-oss-20B for critique and judgment. We additionally report results for different generators than Llama-3. 1.tiaa-..--B---- 1--B---Bnee Instrucct. Note that unlike FARE-20B, gpt-oss-20B does not natively support prompting without intermediate CoT, making an even comparison with our results presented in Fig. unfeasible. Notably, FARE-20B is the only judge improve performance over greedy across all generators and benchmarks, regardless of prompt. While FARE-8B is relatively strong judge, it does not improve generator performance universally, struggling with larger generators on harder benchmarks. The trend of small evaluators struggling in helping larger generators was noted originally in JETTS. Across the board, FARE-20B improves in performance over gpt-oss-20B, sometimes by significant margins (e.g., 49.85 vs 29.93 for Llama-3.1
MATH performance).
E PROMPTS AND EXAMPLES
E.1 OUR EVALUATION PROMPTS
Below we provide our evaluation prompts for pairwise, step-level, and verification evaluation, along with our direct judgment evaluation prompt for pairwise.
Pairwise evaluation prompt for FARE
H# System Prompt
Flease act as an impartial juage and evaluate the quality of the responses provided ded by two Al assistants to the user prempt displayed answer and assistant H's
below. You will answer. Your job is to determine which assistant's answer is better. If assistant A is better, output [A]. If assistant B is better, output [B].
given assistans
Here are some rules for evaluation
(1) When evaluating the assistants' answers, identify any mistakes inaccurate information. Focus the content each response and select the response that logically sound and error free.
(2) If both responses contain inaccurate information, select the response that arrives at the correct response
(3) Avold any biases, such as order of responses, length, or stylistic elements like formatting
Before outputting your final judgment, provide an explanation of your judgment. Your explanation should discuss why your chosen response is better based on the evaluation criteria. The explanation should concretely discuss strengths and weaknesses of both answers.
After outputting your explanation, provide your final judgment. Use the following format:
Explanation: Your explanation here
Verdict: Your final verdict
### User Prompt
[User Question]
(question)
[The Start of Assistant A's Answer] (response_a)
[Tee End of Assistant A Answer] [The Start of Assistant Answer] (response_b]
26

---

Published as a conference paper at ICLR 2026
(The End of Assistant B' Answer]
Direct judgment pairwise evaluation prompt for FARE
### System Prompt
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user prompt displayed below. You will be given assistant A's answer and assistant B's answer. Your job is determine which assistant's answer is better. If assistant is better, output [A]. If assistant is better, output [B].
Here
rules for evaluation
(1) When evaluating the assistants' answers, identify any mistakes or inaccurate information. Focus on the content each response and select the response that is logically sound and error free.
(2) If both responses contain inaccurate information, select the response that arrives at the correct response
(3) Avoid any biases, such order responses, length, stylistic elements like formatting
Output your final judgment airectly. Do not output any explanation rationale for your decision. Use the following format:
Verdict: Your final judgment
### User Prompt
[User Question]
(question)
[The Start of Assistant A's Answer]
(response a}
(The End of Assistant A'ss Answer]
[The Start of Assistant B's Answer]
(response b}
[Thee knd Assistant B's Answer]
Step-level evaluation prompt for FARE
### System Prompt
impartial juage and evaluate the quality of the assistant the user prompt displayed below. You will be given the assistant's solution to a math problem, which is split into steps, starting with <step [step number]> tag, where [step number] is indexed from 0. Your job is to identify which step
Flease act
response provides by
occurs, if error is present. When evaluating the solution, consider each step separately. Evaluate the centent of each step for correctness. If you encounter a mistake at <step [step number]>, output [stop number] as your Verdict. If the full response is error free, then select step number -1. Avoid any biases, such length of step,
stylistic elements like formatting.
Here are some rules for evaluation.
(1) The assistant's answer does not need to complete or arrive at a final solution. You may receive a partially complete response. Your job is to assess the quality of each step.
27

---

Published as a conference paper at ICLR 2026
(2) When evaluating the assistant's answer, identify any mistakes or inaccurate inforaation. Focus on the content each step and determine if the step is logically valid.
(3) For each step, you should provide an explanation of your assessment. I= you find an error, describe the nature and cause of the error.
(4) Avoid any biases, such as answer length, or stylistic elements like formatting.
Before providing your final verdict, think through the judging process and output your thoughts as an explanation After providing your explanation, you must output the corresponding step number with an error. Use the following format:
Explanation: Your explanation here
Verdict: The step number with the error or -1 it no error occurs II User Prompt
[User Question] {question) |The Start of Assistant's Answer] {response}
[The End of Assistant's Answer]
Reference-based verification evaluation prompt for FARE
###
System Prompt
Please act as an impartial judge and evaluate if a response provided by an AI assistant (candidate answer) is consi consistent with a provided reference answer. Your job is to determine is the assistant's response consistent with the reference answer. If the response is consistent, output [A].
If the response is incorrect, output [B].
Here are some rules for evaluation.
(1) Refer to the given reference answer and determine if the cardidate's answer is consistent with the reference answer.
(2) The reference answer is always correct and the questien is perfectly valid. Take the reference answer as the ground truth.
(3) When determining it the candidate's answer is consistent with the reference answer, only compare the final answer. Ignore any potential errors the reasoning processes.
(4) Some answers may be expressed in different ways, such as some answers may be mathematical expression, textual description,
answers may be long the meaning expressed is the same. Before making a judgment, please understand the question and the reference anawer first, and then judge whether the candidate's answer is consistent with the reference ansver.
(5) Some answers may consist of multiple items, such as miltiple-choice questions, multiple-select questions, fill-in-the-blank questions, etc. Regardless of the question type, the final answer will be considered correct long it matches the standard answer, regardless whether is correct. For multiple-select questions and miltiple-blank fill-in-the-blank questions, all corresponding optiens or blanks must be anawered correctly and match the standard answer exactly to be deemed correct.
reasoning process
28

---

Published as a conference paper at ICLR 2026
Before outputting your final judgment, provide an explanation of your judgment. Your explanation should discuss why the response is correct, incorrest, or invalid. The explanation shculd concretely discuss reasons for your judgment. After cutputting your explanation, provide your final judgment. Use the follewing format:
Explanation: Your explanation here
Verdict: Your final judgment of [A] or [B]
### User Prompt
User Prompt|>
(question)
The Start of Assistant's Answer|>
(response)
<he End of Assistant's 8 Answer|>
</ Start of Reference Answer|> (reference)
<h End of Reference Answer|>
E.2 SAMPLE EVALUATION RUBRIC
Here, we provide a sample rubric that was hand-written for SWE-Rank (Reddy et al. 2025). SWE- Rank data consists of contrastive pairs for training retrieval models. We re-purposed this data into a binary verification task, asking the evaluator if the retrieved code snippet was relevant for editing given user request. "Positive" samples were assigned "Correct" labels, and "Negative" samples were assigned "Incorrect" labels.
Example hand-written rubric for code retrieval samples
Here some rules for evaluation
(1) Determine if the function provided by the assistant is relevant candidate for editing given the user's instruction
(2) A relevant function one means that needs order to address the issue described the user's instruction
medified
(3) Modi fying relevant function does not mean is is sufficient to resolve the user's issue. That is, it is ok modifying the function does not completely resolve the user issue, but it should make progress towards issue resclution.
29


Reviews:
[{"review_id": "b2vbXK0zKS", "reviewer": "Reviewer_TEm7", "summary": "This paper proposes a novel methodology to generate data and train an LLM-based evaluator and verifier. By combining different types of evaluation and verification tasks, they collect lots of training data featuring both synthetic and human preferences for downstream training. Authors stress that their result is one the examples of larger scale data scaling for training the evaluator/verifier models compared to more recent approaches that utilized relatively smaller sets for RLVR finetuning. \nAuthors rely on iterative SFT training and train a few models in the experiments. Their extensive results show that their model outperform known competing baselines on the benchmarks designed for evaluators. Moreover, their models show solid improvements when used for RL training with LLM as a judge/reward model, and outperform rule based rewards.", "questions": "* How was the model selection / early stopping performed? Is the same checkpoint used for all the evaluations in the paper?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Well executed recipe without any perplexed tuning or parameterization\n* If the model will be released, then it will be especially useful\n* Multi-tasking across different eval formats was shown to work", "weaknesses": "* W.r.t. the last strength point, i would be great to see the ablation of multi-tasking and how that affects the performance on specific eval tasks. From my understanding, there is no such experiment in the paper.\n* When compared with competing evaluator models from literature, its a bit unclear what were other models initialization ckpts i.e. either its llama or qwen. Such difference may introduce not very fair comparison if we care about the added value of the proposed data and multitasking. Adding some extra notes about what are other model's pretrained architectures will help to make this more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7f2Ai1B4e", "reviewer": "Reviewer_MQC2", "summary": "The paper introduces FARE (Foundational Automatic Reasoning Evaluators), a family of large-scale generative evaluators (8B & 20B parameters) designed for multi-task, multi-domain reasoning evaluation. It curates a 2.5M-sample dataset covering diverse evaluation types like pairwise, step-level, and verification tasks across reasoning-heavy domains such as math, code, and tool-use. FARE is trained using an iterative rejection sampling supervised fine-tuning (RS-SFT) method, enabling stable, scalable evaluator training without expensive RL or teacher models. Results show FARE-20B surpasses even larger RL-trained or specialized evaluators, achieving near-oracle reranking on MATH and significant gains in RL training and code evaluation. The work establishes FARE as a strong, open-source foundation for general-purpose evaluators in reasoning-centric LLM applications.", "questions": "1. \"Existing data lays a solid foundation, with 1.4M samples already dwarfing data scales found in recent work.\" Please quantify this properly. What recent work? What is the number of samples in that work?\n2. \"Generate-then-grade\" method. What model is used to grade the generated responses? What is the accuracy of the grader model? \n3. What is task-wise and domain-wise distribution of existing vs synthetic datasets?\n4. \"After grouping responses by correctness, we create verification and pairwise.\" This sentence needs to be reworded -- does not make sense to me. \n5. Line 237: \"Because the automatic evaluation setting is inherently verifiable\". You have automatic verifiers for all kinds of tasks? not all domains (chat, safety, reasoning) are objectively verifiable.\n6. benchmarks: does it make sense to evaluate using PPE Correctness where golden responses are generated\nusing a variety of weaker models, e.g., Gemma-2-9B (which is smaller than the FARE-20B model you train)? Also, how are golden responses for these benchmarks curated: RM-Bench, When2Call, ProcessBench, VerifyBench?\n7. Table 1: JudgeBench and RJB -- does not make sense to report GPT5 numbers and not GPT-4o numbers here. Aguably, GPT5 must be better than GPT4o. And since GPT4o was used to gather golden responses for these datasets, GPT4o must have 100% accuracy?\n8. How does your method compare with high-performing TTC methods that explicitly address confidence calibration and adaptive compute while performing generation and evaluation together, such as: Efficient Test-Time Scaling via Self-Calibration (Huang et al., 2025) https://arxiv.org/pdf/2503.00031 COME: Test-Time Adaptation by Conservatively Minimizing Entropy (Zhang et al., 2025) https://openreview.net/forum?id=506BjJ1ziZ", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The 2.5M-sample multi-task dataset is one of the largest curated for evaluation, spanning reasoning, code, math, and tool-use, enabling strong generalization across domains.\n2. The use of RS-SFT (rejection sampling SFT) achieves performance comparable to RL-based methods while being computationally more efficient and stable.\n3. Evaluations on 7 benchmarks and 3 real-world tasks (e.g., MATH reranking, RL training, code evaluation) show strong improvements over state-of-the-art baselines.\n4. The release of FARE models (8B, 20B) provides a valuable community resource for scalable evaluator research.", "weaknesses": "1. Novelty is rather low. The authors themselves mention that the method is simple and is a minor modification of methods like STE and RAFT. The paper emphasizes empirical scaling but provides little theoretical analysis of why RS-SFT works better for evaluators compared to RL or DPO approaches.\n2. While reasoning-centric, the dataset and evaluations largely focus on math, code, and tool-use; broader language understanding or multimodal evaluations are underexplored.\n3. Though large-scale, the paper lacks detailed quantitative analysis of noise, bias, or consistency across synthetic vs. real data sources.\n4. Comparisons are strong against RL-trained evaluators but do not include recent alternatives like bandit-based scaling or entropy-minimization approaches to evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y2ZSutRBc5", "reviewer": "Reviewer_Z7Xv", "summary": "This paper presents FARE (Foundational Automatic Reasoning Evaluators), a family of 8B and 20B parameter models trained to perform automatic evaluation across multiple tasks including pairwise comparison, step-level error detection, reference-based and reference-free verification, and single rating. The authors curate a 2.5M sample dataset spanning these evaluation tasks and multiple reasoning domains, then train their models using iterative rejection sampling supervised fine-tuning (RS-SFT). The paper evaluates FARE on seven static benchmarks and three practical downstream applications: inference-time reranking, verification during RL training, and domain-specific continual fine-tuning for code evaluation. The results demonstrate that FARE-8B challenges larger specialized evaluators, while FARE-20B establishes new state-of-the-art performance among open-source evaluators, even surpassing some 70B+ parameter models on certain tasks.", "questions": "1. Why was the cold-start initialization from Qwen2.5-32B-Instruct necessary for Qwen3-8B-Base, and have you experimented with alternative initialization strategies that might preserve more of the base model's capabilities?\n2. Can you provide more detailed error analysis showing specific failure modes of FARE models, particularly on the benchmarks where they underperform GPT-5 or larger baseline models?\n3. How did you determine the optimal mixing ratios for different data sources in your 2.5M sample training set, and did you experiment with dynamic reweighting during training?\n4/ What is the computational cost comparison between training FARE using RS-SFT versus training comparable evaluators using RL-based methods like RLVR?\n5. For the generate-then-grade approach, how did you select the 12 generator models, and did you analyze whether certain generator characteristics (model family, size, reasoning vs non-reasoning) led to more valuable training data?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper addresses a timely and important problem of building multi-task evaluators that can handle diverse evaluation scenarios, which is increasingly critical as LLMs become integrated into various applications.\n- The data curation strategy is thorough and well-designed, combining 1.4M existing samples with 1.1M synthetic samples using both programmatic error injection and generate-then-grade approaches across multiple domains.\n- The training methodology using iterative rejection sampling SFT is simple, stable, and scalable, avoiding the computational complexity and training instability of RL-based approaches while achieving competitive or superior results.\n- The evaluation is comprehensive, covering seven challenging static benchmarks (JudgeBench, ReasoningJudgeBench, PPE, RM-Bench, When2Call, ProcessBench, VerifyBench) and three practical downstream applications that demonstrate real-world utility.\n- FARE-20B achieves near-oracle reranking performance on MATH (Figure 3) and substantially improves RL training outcomes, with up to 14.1% relative gains over string matching verifiers (Figure 4), demonstrating strong practical impact.", "weaknesses": "- The cold-start initialization procedure for Qwen3-8B-Base using Qwen2.5-32B-Instruct data is not well-justified, and the authors acknowledge this produces a weaker baseline than the post-trained Qwen3-8B (Table 9), raising questions about whether better initialization could further improve results.\n- The paper lacks detailed analysis of failure modes or systematic error analysis that would help understand when and why FARE models struggle, particularly on the benchmarks where performance lags behind GPT-5 or larger models.\n- While the 2.5M sample dataset is impressively large, the paper provides limited information about potential data quality issues, redundancy across sources, or how the distribution of tasks and domains was optimized.\n- The comparison with Self-Taught Evaluators (STE) could be more direct and comprehensive, as STE represents closely related work but is only evaluated on a subset of benchmarks, making it difficult to fully assess the relative merits of the approaches.\n- Some experimental details are missing or relegated to appendices, such as the specific prompt templates used for each baseline model, hyperparameter selection procedures, and computational costs of training.\n- The paper does not discuss potential negative societal impacts or limitations of deploying automatic evaluators at scale, such as perpetuating biases present in training data or the risks of evaluator exploitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)