J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
Authors: Chenxi Whitehouse
Tianlu Wang
Ping Yu
Xian Li
Jason E Weston
Ilia Kulikov
Swarnadeep Saha
Venue: ICLR 2026 Poster (2026)
Keywords: LLM-as-a-Judge
Reasoning
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: The progress of AI is bottlenecked by the quality of evaluation, making powerful LLM-as-a-Judge models a core solution. The efficacy of these judges depends on their chain-of-thought reasoning, creating a critical need for methods that can effectively optimize this reasoning process. In this work, we introduce J1, a reinforcement learning framework for teaching LLM judges to think before making decisions. Our core contribution lies in converting all judgment tasks for nonverifiable and verifiable prompts into a unified format with verifiable rewards, enabling direct optimization of evaluation quality while mitigating positional bias. We then use RL to train thinking-judges at scales of 8B, 32B, and 70B and show that they obtain state-of-the-art performance across multiple benchmarks. In particular, J1-Qwen-32B, our multitasked pointwise and pairwise judge also outperforms o1-mini, o3, and a much larger 671B DeepSeek-R1 on some benchmarks, while only training on synthetic data. Through comprehensive ablations of pairwise, pointwise, and multitask J1 variants, we demonstrate the effectiveness of our approach across seed prompts, reward strategies, and training recipes. Qualitative analysis reveals that J1 develops systematic evaluation strategies, including dynamic criteria generation, reference answer creation, iterative self-correction of initial assessments, and feedback generation for low-quality responses.

Published as a conference paper at ICLR 2026
J1: INCENTIVIZING THINKING IN LLM-ASS A A
VIA REINFORCEMENT LEARNING
Chenxi Whitchouse Ilia Kulikov
Tianlu Wang
Ping Yu
Xian Li
Jason Weston
Swarnadeep Saha
FAIR at Meta chenxwh@meta.co a. com,
swa swarnndeepemetaccom
ABSTRACT
The progress of Al is bottlenecked by the quality of evaluation, making powerful LLM-as-a-Judge models core solution. The efficacy these judges depends their chain-of-thought reasoning, creating critical need for methods that effectively optimize this reasoning process. In this work, we introduce J1, a rein- forcement learning framework for teaching LLM judges to think before making decisions. Our core contribution lies in converting all judgment tasks for non- verifiable and verifiable prompts into unified format with verifiable rewards, enabling direct optimization of evaluation quality while mitigating positional bias. We then use RL to train thinking-judges at scales of 8B, 32B, and 70B and show that they obtain state-of-the aof-the-art performance across multiple benchmarks. In partic- ular, D1-Qwen 32B, our multitasked pointwise and pairwise judge also outperforms ol o1-mini, o3, and much larger 671B DeepSeek-RI R1 on some benchmarks, while only training on synthetic data. Through comprehensive ablations of pairwise, pointwise, and multitask J1 variants, we demonstrate the effecttivnness of our ap- proach across seed prompts, reward strategies, and training recipes. Qualitative analysis reveals that J1 develops systematic evaluation strategies, including dy- namic criteria generation, reference answer creation, iterative self-correction initial assessments, and feedback generation for low- oww-quaity responses.
INTRODUCTION
Better judgments can be made by learning how to reason, which is observed in both humans and machines. For models, the ability to judge predictions is vital process that is applied all stages of development: during training and inference to provide reward or verification signal, and during final benchmark evaluation to judge performance. Classical evaluation using reward models typically outputs a score directly (Ouyang al., 2022) without having explicit reasoning step. Using pre-trained and aligned language models to act as judges instead, i.c.. ALM-as---Judee allows the possibility to generate chain-of-thought reasoning before making decision, which was first. invoked by prompting (Zheng et al., 2023; Gu et al., 2024; Saha et al., 2024). Subsequently, iterative finetuning and direct preference optimization (DPO) methods were developed improve these reasoning steps (Mahan et al., 2025; Wang et al., 2024c; Yu et al., 2025a; Saha et al., 2025). In this work, we investigate approaches for further improvements to judgment reasoning via online Reinforeement Learning (RL).
We introduce J1 (Thinking- Brhinking-LLM-assse LLMdea---Judee via RL), framework that incentivizes LLMs to think for evaluation through three primary aspects: (i) Unified Verifiable Training: we design unified training recipe that converts all judgment tasks, from both verifiable (e.g., math problems) and typically subjective, non-verifiable prompts (e.g., user prompts from WildChat (Zhao et al., 2024)). into a format that can be optimized with RL from verifiable rewards. This allows us to train a single, generalist judge across diverse domains using only synthetic data. (ii) Teasonnng-Opttmmized Training: we use GRPO (Shao et al., 2024) to directly optimize the quality of evaluation thoughts, in analogy to the approach in DeepSeek-Ri R1 (Guo et al., 2025a). Guided by seed prompt and targeted reward schemes, J1 teaches the model to reason critically about evaluation. (iii) Multitask and Bias-Aware Judge: address positional bias through consistency-based rewards and, more importantly, develop

---

Published as a conference paper at ICLR 2026
Pairwise-J1 Thinking Trace
Pointwise-J1 Thinking Trace (Chosen Rejected)
Evaluation on Criteria will conaider the accursty the mathemstical solution, darity exploration, and achenence to the given fanctior and user's question.
"<think>
"thhink>
Comparison
Therafore, lotal iumber
gi Bt 28800.
Referance Answer
This motches assistont's reasoning caloulation.
Flrst, let's estaolish the reforence arswer by soiving the problem: Multiphying both sides by $56++ 1/5 gives $10 - x 15 Subtracting 1 from both siees yeicss Se = 95 Re-evalustion
</think>
Final Score
<score> 10 </scoress
Howeverr let's re evel ane the step where we ve So4 to ensure accuncy. Mittilynng both sciee by $56- gives $10 = 15, step seems have bwem done commally Asestant A's reparee, lesting tn Sx
"<tiink>
Referance Anower
The correct calculation should bei -.. This glves us 120 28800. Feedbock Actuslly, correct think about is that the parties are ananged 12 since they are blocks)...
Comparison
verily whi ih solution comeect.... .... Thas Assettent solution Sa.- correct
</thinks>
Final Verdict
Final Score
canswer> [A] <lanswers'
<score>
</scare>*
Figure 1: Illustration of the thinking patterns of pairwise and pointwise J1 models during RL training. J1 learns to outline evaluation criteria, generate reference answers. re-evaluate correctness, and compare between responses. Pairwise setup outputs a final verdict indicating the better response, pointwise generates real-valued score, with higher score for the better response. More examples of J1's thinking traces are shown in Figures 6, 7, and 8.
method to train inherently position-consisten pointwise judges using only pairwise supervision. We then unify these approaches into single multitask model that is capable of performing both pointwise and pairwise evaluations.
After applying our best J1 training recipe, we train judges on top of Llama-3. 1-Instrutt (Grattafiori et al., 2024) and Qwen3 (Yang et al., 2025) models at 8B, 32B, and 70B scales that achieve state-of- the-art performance across a variety of benchmarks: PPE (Frick et al., 2025), RewardBench (Lambert et al., 2025), JudgeBench (Tan et al., 2025), RM-Bench (Liu et al., 2025a), and Fellow (Saha et al., 2025). In particular, J1 significantly outperforms: (i) models trained with SFT or Self-Taught reasoning (Wang al., 2024c) and offline DPO (Saha al., 2025); (ii) scalar RMs trained with a Bradley-Terry objective such as Skywork-RM (Shiwen et al., 2024); (iii) recent state-of- the-art generative reward models such as DeepSeek-GRi (Liu al., 2025b) and Reasoning Reward Model (Guo et al., 2025b) that are trained on significantly more data; (iv) open-weight reasoning models like SeepSeck-R1-67111
and (v) closed reasoning models such as ol-mini (Jaech et al., 2024) and o3 (OpenAI, 2025). Crucially, J1 achieves these results while only training a 32B model and leveraging small amount of synthetic training data. Test-time scaling of via majority vote over multiple verdicts or averaging over multiple judgment scores leads to further improvements.
We provide detailed ablations and analysis of these results by comparing our best training strategy to other variants, that either modify the LLM-as-a-Judge setup (e.g., pairwise vs pointwise vs joint/multitasked, with vs without scores), bias mitigation strategies, reward modeling approaches, or seed thinking prompt. We show that our proposed join! model flexibly performs both pointwise and pairwise evaluations, while also being superior to both separately trained counterparts. In addition, also analyze the distribution of the generated scores, thought lengths, and reasoning patterns within the thought generations. Our qualitative analysis shows that J1 models learn to make better judgments by systematically outlining evaluation criteria, comparing responses against self- self-generated reference answers, critically re-evaluating their own initial assessments, and providing feedback (see Figures 1, 6, and 8 for examples).
2
J1: THINKING-LLMAASAAJJUU
VIA REINFORCEMENT LEARNING
Our J1 framework trains LLM-as-a-Judge that generates chain-of-though reasoning before scoring responses or making a preference judgment. Our primary setting is pairwise evaluation, where the judge takes instruction x and two responses (a. b) produce verdict indicating the preferred response. The model's 's output consists intermediate thought tokens t followed by the final verdict y. conditioned seed prompt designed to elicit reasoning (see Appendix B).
This section details the three core components of our method. First, we describe the synthetic data generation process that creates verifiable tasks for RL. training ($2.1). 1). Next, we define the reward
2

---

Published as a conference paper at ICLR 2026
Verdict Gorrectinn Streard if Hendil Auoi - cle Mendict lase c. eloon
CoT Goncration
Hler Prompe->
wudigasiiiilniii h.o
Pairwise-J
verd endror so0re3
und cxtaacing
Poetton-sgnotti
Consistecyy
Venfuble Teaks
User thompnus
Pattwiie Batthh
Faarditt()uu) Verdet m G. othendie
Rejected Resporenc
User Prorpg
Nonverifablle Tooks
LLM
UIM
Winerro
(z 2/ JudgePoottvisstss..
Pointwis Batch
Pointwise-J] J
Col Goneration and oxtrectig sco
(ooee > SSorro H ehc Fooon 4) =-Screl ote
Score based Reward
Multitaske
New Use Prompe
(Scomela)) Scoml))) diso
RL Training with GRPO
Reinforcement Learning recipes for training J1 models. We generate synthetic preference pairs for both verifiable and non-verifiable tasks to create position-agnostic training batches. Rewards based on verdict correctness, consistency, and score alignment jointly optimize evaluation thoughts and verdicts using online GRPO. Pointwise-J] is trained only via distant supervision from pairwise labels. Multi MultiTask-JJ Task-J1 combines pairwise and pointwise formulation.
Figure 2:
functions used to optimize for both judgment correctness and consistency ($2.2). Finally, we outline different judge formulations and training, including pairwise, pointwise, and multitask model ($2.3). An overview of the overall process is shown in Figure 2.
2.1 SYNTHETIC TRAINING DATA GENERATION
The goal of J1 is train a generalist judge for both verifiable and non-verifiable tasks. To achieve this, we follow the unified training dataset of synthetic preference pairs from Wang et al. (2024a), which removes the reliance on costly human annotations used prior work (Liu al., 2025b; Chen et al., 2025b). By building on the same data generation strategy from Saha et al. (2025). we can directly compare the effectiveness of our online RL approach against prior work using offline DPO on the same data distribution.
Our final 22K training data consists of 17K WildChat (Zhao et al., 2024) and 5K MATH (Hendrycks et al., 2021) prompts, along with their corresponding preference pairs. For WildChat, rejected responses are obtained by prompting an LLM to first generate a "noisy" variant of the original instruction and then produce response to this noisy instruction (Wang et al., 2024c). See Figure 14 for the prompt and Table 5 for an example of a synthetically generated training pair. For MATH rejected responses
sampled from generations by LLM that do not lead to the gold answer. Given these preference pairs, we are thus able to convert the evaluation task into a verifiable task (i.e., predicting the better response), enabling the use of RI. with verifiable rewards.
To address the known issue of position bias in pairwise judges ( (Wang et al., 2024b; Chen et al., 2024), we augment the training data by including both response orderings - (ar, a, b) and (zr, a). using the thinking seed prompt (Figure 9 in Appendix B). We construct training batches to be position-agnostic, i.e., both orderings of a given pair processed in the same batch. As detailed in the next section, this batching strategy is crucial for implementing our consistency-bard based rewards.
2.2 REWARD MODELING
We adopt a straightforward and effective rule-based reward system designed to encourage accurate and consistent judgments.
Verdict Correctness. JI's primary reward signal is binary. The model receives a reward of +1 if its final verdict correctly identifies the preferred response, and 0 otherwise.
Verdict Consistency. To explicitly mitigate position bias, we introduce a consistency reward. A reward of +1 is granted only if the model produces the correct verdict for both input orderings of a response pair, i.e., for (z, a. b) and (x, b, a). An incorrect verdict on either ordering results in a reward of 0.
We also explored adding format-based rewards to enforce the use of "<think>" tags, but found no noticeable performance benefit. The effects of different reward components ablated in $4.2.
3

---

Published as a conference paper at ICLR 2026
2.3 J1 FORMULATIONS AND TRAINING
Having described the training data and reward schemes, we now use GRPO (Shao et al., 2024) to jointly optimize thought generation and final judgments. We explore several training formulations that differ in their input/output formats and reward schemes.
Pairwise with Verdict (PaV). Our first formulation, J1pav prompt b) (t. y). receives a user question and response pair, and generates thought tokens and the preferred response (as the final verdict). Figure in Appendix shows the seed thinking prompt. Figure illustrates examples of judgment generation with this formulation.
Pairwise J1 with Scores (PaS). Instead of directly generating a verdict, our pairwise score-based variant Jlpas paompt (z, a, b) > (t, Sa; Sb), generates real-valued scores Sa. for response and b, respectively. The response that obtains the higher score is chosen as the final verdict. Figure 11 in Appendix B shows the corresponding thinking prompt. To train a model with this recipe, we replace the pordict-based reward with score- score-based reward: a binary reward is assigned depending on whether the predicted scores are consistent with the gold verdict.
Pairwise J1 with Scores& Verdict (PaVS). In another pairwise variant J1pavs promptpavs (zz, a, b) - St, 86, the model generates scores for both responses as well as the final verdict, where the generated scores are interpreted as observations of unknown latent variables help with the pairwise judgment task; therefore y is directly used as the final verdiet. Consequently, the reward is also computed using only the final verdict (and not the intermediate scores). Figure 12 in Appendix shows the corresponding thinking prompt.
Pointwise J1 (PoS). We also introduce a pointwise judge, J1pos prompt a) - (t, 6), which takes an instruction and a single response as input, and outputs score s that reflects the quality or reward of the response. Unlike pairwise judges, pointwise judges are inherently position-consistent. We train J1pos via distant supervision with the same pairwise training data as used for all our pairwise variants. Each preference pair is split into two separate pointwise samples and the model is trained with seed thinking prompt that assigns a score between and 10 to a given response (see Figure 13 in Appendix B). Both scores are evaluated jointly, and the model receives a reward of if the scores are consistent with the gold verdict. Since preference rankings are significantly easier to obtain than pointwise annotations, Pointwise-Thinking
represents one of our novel contributions.
Multitask Pairwise & Pointwise J1 (MT). Finally, we unify the Pointwise and Pairwise (PaS) paradigms into a single multitask model by jointly training on both pairwise and pointwise data. This model has the advantage of being a unified model that can be evaluated in both setups and improves upon separately trained judges. As we show later, given the overall superiority of pairwise judgments over pointwise judgments, we evaluate the multitasked model in a pairwise setup for best results.
EXPERIMENTAL SETUP
Training. We implement J1 on top of verl (Sheng et al., 2025). All variants are trained on the 22K synthetic preference pairs described in Section 2.1. Appendix C includes other details of our experimental setup. Unless otherwise stated, all main J1 experiments are conducted with our primary Pairwise J1 with Verdict (PaV) recipe and the multitasked J1 model evaluated in a pairwise setting.
Evaluation. We evaluate J1 on five pairwise judgment benchmarks, covering both verifiable and non-verifiable tasks, including multilingual instructions and responses from a wide range of LLMs.
Preference Proxy Evaluations (PPE) (Frick et al., 2025) is a large-scale benchmark that links reward models real-worid human preference performance. It consists of two subsets: (i) PPE Preference (10.2K samples), human preference pairs from Chatbot Arena featuring 20 L.MMs in 121+ languages, and (ii) PPE Correctness (12.7K samples), response pairs from four models across popular verifiable benchmarks (MMLU-Pro, MATH, GPQA, MBPP-Plus, IFEva)). The first subset evaluates subjective preferences, while the second tests alignment in Best-of-N tasks.
JungeBench (Tan et al., 2025) (350 preference pairs) contains challenging response pairs that span knowledge, reasoning, math, and coding categories. Following Tan et al. (2025), we report results on the subset where the responses generated by GPT--40.
4

---

Published as a conference paper at ICLR 2026
Table 1: Results PPE Correctness comparing J1 against other L.M-Judges and reward models.
All models are evaluated in a pairwise setup. t: Results from Liu et al. (2025b) and Frick et al. (2025). Trained solely on synthetically constructed preference pairs. Reported gains are relative the corresponding base models.
PPE Correctness
#Training Pref. Pairs Overall
MMLU-Pro MATH
GPQA
MBPP-Plus IFEval
Bose [LM-ar-a-Judge
Llama-3 1amma-3.1--BB I-8B-Instruct
54.7
56.3
62.9
51.4
50.1
52.8
1.mm--3.-770--nnsttt
65.7
72.1
73.1
61.2
59.6
62.3
Qwen3-32B (thinking)
66.5
75.6
85.2
53.6
55.9
62.0
SOTA Scalor Reword Models
Armo-8B-v0.1
1000K
61.2
66.0
71.0
57.0
54.0
58.0
Skywork-Reward-ok Gemuud--Gmma--2277B8 DeepSeek- WTRM-27B
80K
54.7
55.0
46.2
44.7
69.1
58.3
73.2
56.8
SOTA Generative Reward Models
DeepSeck-GRM-275
237K
39.8
64.8
68.8
55.6
50.1
59.8
EvalPlanee--Llam--88
22K
52.8
57.0
59.0
50.3
47.7
50.0
22K
70.2
78.4
81.7
64.4
62.2
64.3
JI Models (Ours)
D1-Llana-BE
22K
59.2
65.6
70.0 -2.1
53.2 .1.
53.1 +10
.1.0
31-Llana-708
22K
-12
86.0 -128
65.9 +4.7
66.0 r6a
67.3 +50
J1- 13-Qwen-322B
22K
74.6
82.2 .66
93.3
65.2
65.3
66.8
Momen-328-Mut-eee
22K
76.8
85.0
94.3
68.6.150
66.3
69.5
R.M-Bench (Liu et al., 2025a) (4K samples) assesses the robustness of reward models based on their sensitivity and resistance to subtle content differences and style biases.
Evll (Saha al., 2025) (205 preference pairs) tests reward models for their ability
validate multi-level constraints in LLM responses (e.g.. "Write one sentence summary (less than 15 words) for the following dialogue. The summary must contain the word 'stuff"...")
RewardBench (Lambert et al., 2025) (3K samples), similar to JudgeBench, consists of preference pairs from 4 categories of prompts: chat, chat-hard, safety, and reasoning.
Consistent with prior work, report accuracy over random ordering of paired responses for PPE, RewardBench, and RM-Bench. For JudgeBench and FollowBenchEval. we instead report aasition-consiste accuracy, where sample deemed correct only if the judge produces the correct verdict under both response orders. A more detailed analysis of position consistency is presented in Section 4.2. Model selection is based on overall accuracy on RewardBench. Inference is performed using vLLM (Kwon et al., 2023).
Baselines. We compare J1 to different categories of baselines: (i) non-thinking LLMs acting as judges in zero-shot manner (e.g., LLMs acting as judges (e.g., ----steee-----sslllleeeeee
GPT-4o (Hurst et al., 2024), (ii) thinking (Guo et al., 2025a), DeepSeek-R2 reward models (e.g., DeepSeck-BTRMr --mii- o3 (OpenAI, 2025)) (iii) state-of-the-an ahe scalar Reward-Gemma-
Qwen3-32DB (Yang al., 2025), OpenAI-ol
(Liu et al., 2025b), Armo (Wang et al., 2024a), Skywork- (Shiwen al., 2024)), (iv) state-of-the-ar generative reward models that belong to the same category as J1 (e.g.. EvalPlanner (Saha et al., 2025), DeepSeek-GRM-P5 al., 2025b)), and Reasoning Reward Model (Guo al., 2025b). Additionally, for RewardBench, we compare J1 to all highest-ranked Generative Reward Models according to the leaderboard."
(Liu
4 RESULTS
4.1 MAIN RESULTS ON BENCHMARKS WITH Pairwise AND MultiTask-J1
Results on PPE Correctness. We first evaluate J1 on PPE Correctness because directly tests RMs and LLM-Judges for their ability improve popular reasoning benchmarks. This benchmark also offers a broader potential for improvement compared to others, such as RewardBench. Table 1 shows the results. J1 Qwen3- -2B---32---0 32B-Mul tisas,, our best J1 model, obtains state-of-the-un performance with an overall accuracy of 76.8, outperforming all previous methods by significantly large margins
bench
5

---

Published as a conference paper at ICLR 2026
Table 2: Results five reward modeling benchmarks, where PPE includes both correctness and preference subsets of data. We compare J1 at different scales to EvalPlanner and general Thinking- LLMs (distilled-RI, ol-mini, o3, and RI). We report the default metric for each benchmark, where f: JudgeBench and NollowBenchEEve use positional consistent accuracy. Reported gains are relative to the corresponding base models.
Models
Overall
PPE
RewardBench
RM-Bench
JudgeBeuch
VollowBeuchEEaal
Llama-3.1 Baaee-..------tsteee
48,3
55.6
69.5
$4,0
32.3
30.2
DeepSeek Ri-iistlled Llana
RvaiPlamerr-I Jamaa-8
56.2
54.3
83.0
68.1
30.2
45.3
J1-Llama-88
-13.6
59.8 44.2
85.7 +J62
73.4.
42.0 vR.
48.3 419.1
Llama-3.3 1.mma--.--70--mmee In-ttutttttt
64.3
65.8
85.4
69.5
48.6
52.2
DeepSeck-R1 I--isstllet-Lllmm--00
67.4
68.6
86.9
80.8
46.0
$4.6
EvaiPlanner-Llamm Jama---0-
73.2
67.9
93.8
82.1
56.6
65.4
31-Llama-706
75.0
69.6 23,8
93.3
82.7 +182
60.0
69.3
Qwen3-32B
77.3
90.9
88.1
70.8
70.0
80.8
71.8
93.6
90.3
714
77.1
OpenAl-ol ol -liiiii mini OpenAl-o3
72.7
68.5
87.1
80.8
64.2
62.9
77.4
86.4
86.1
DeepSeek R1-671B
78.4
72.3
90.6
88.6
68.9
71.7
Table 3: Results PPE Correctness comparing judgment bias of pairwise and pointwise J1. Consistent Accuracy: proportion examples judged correctly in both response orders. Verdict Flip: proportion of cases where the pairwise verdict changes when order is swapped. Ties: proportion of examples where the pointwise judge assigns equal scores.
Models
Type
(a, b) Acc
(b, a) Acc
Consistent Acc
Verdict Flip/Ties
J1-LLama-770
Pairwise
72.9
72.3
61.2
21.9
J1-Llama-708
Pointwise
65.0
13.7
J1-Owen-328
Pairwise
74.6
74.2
65.2
14.5
J1-Qwen-30
Pointwise
69.3
13.0
J1-Qwent (0weww33---MM-0 Mit-Mu
Pairwise
76.8
76.2
67.0
17.0
J1 Miwee-33B--Mlttin
Pointwise
70.6
10.5
(p < 0.0001), including those trained much more data (see column 2). Compared to related competing approaches that are generative reward models, J1- Mwwen-33B-MMltte both EvalPlanner (Saha et al., 2025) by 6.8% (70.2 76.6), and DeepSeck-GRM (Liu et al., 2025b) by 17% (59.8 + 76.8). J1 also improves upon the base Qwen3-32B model, a strong thimking-LLM, by large 10.3% (66.5 76.8).
outperforms
Furthermore, all J1 models at three different scales outperform their base counterparts, highlighting the generalizability of our recipe. In particular, this validates the effectiveness of J1's training methodology and use of online RL, compared to EvalPlanner, which is trained on the same data but with iterations of DPO. Second, this also shows the effectiveness of J1's high-quality synthetic preference pairs, compared to the data used to train DeepSeck-GRM- tuned on 1270K judge data, followed by stages of Reinforcement Learninng on 237K samples and further scaling at test time with a meta reward model across 32 generations. At a smaller scale, J1-Llama 8B competitive with Armo-8B (scalar RM) and outperforms EvalPlanner-8F and larger respectively).
The latter is first fine-
by significant margins (52.8 + 59.2 and 54.7 - 59.2,
Results on RewardBench. Table 6 in Appendix D shows comparison of J1 with leading gen- erative reward models on RewardBench. J1- Qwen- 32B-MultiTask obtains an overall score 93.6, 6, outperforming all previous generative reward models. Importantly, J1 is equally performant four categorics RewardBench. This suggests that J1 is generalist judge that can be used for evaluating responses to both verifiable and non-verifiable prompt tasks, and in different stages of the LLM development process.
all
Comparison of J1 with Thinking-LLMs, Next, in Table 2, compare J1 several SOTA Thinking-LLAN LLMSs on all benchmarks at three different seales (8B/32B/70B). These include DeepSeck- R1-Distill- DeepSeek-R1, Qwen3-32B, OpenAl-ol-mini and OpenAI-o3. DeepSeek-R1-

---

Published as a conference paper at ICLR 2026
Mutttask.j.. Pointwise Eval
Nultitask ja Pairwise Eval
Multitask
Eval
6.00
2530
Dhunt
4200
15:0
2300
1000
500
10
Absokite Score
Absolute
Score Difference
Score Difference (A
Figure 3: Distribution of Absolute Scores and AScore (Chosen Rejected) on PPE Correct- ness from J1 -1-00 Qwen ------22---- 32B-Mul titask when used in swirwise and pointwise settings. Pairwise exhibits sparser score distribution and larger score differences between Chosen and Rejected (ground- truth) responses. Note that all samples with positive A in the range [0, 1) are correct predictions and count toward the '0' bar in the plot.
Consibtent Accurscy- Paintwis
Pointwise
Pesition Conobtent Acouracy Purreise
Verdic PlipTie
20
53
U.Sampa
Sa
Samoless
Figure 4: Test-time scaling of J1 on PPE Correctness. We show greedy decoding results in dotted lines for comparison. As we sample more N, (i) resition-consister accuracy increases and (ii) tie rate decreases, for both pairwise and pointwise models, both 8B and 70B scales.
Distilled-Llama is SFT-ed with 800K long CoTs from DeepSeek-RI (a much larger 671B MoE model), starting from the same base models as J1. Thus, in a head-to-head comparison where the base models are the same, J1 Llama models outperform benchmarks by large margins. Impressively, the-art thinking models, RI and o3, on three out of five benchmarks, while only training a 32B model with synthetic data. Through these comprehensive evaluations, J1 thus reinforces the utility of online RL and synthetic data for training state-of- oftte-of-thh-art Thinking MeikiinngLLMMM-ee LLM-as-a-Judhe models.
RI-isstileed--d
across all also outperforms state-of-
4.2 ABLATIONS AND ANALYSES OF J1 IN POINTWISE VS. PAIRWISE SETUPS
In this section, systematically analyze the impact of different J1 formulations introduced in $2.3, focusing on following research questions: (i) How does pointwise setup compare to pairwise in mitigating position bias? (ii) What benefit does multitasking bring compared to separate Pointwise- J1 and Pai Pairwise-J JJ training? (iii) How can we design effective rewards for reinforcement learning Thinking-LLan LL--ang-L------------ models? (iv) What is the impact of different thinking prompts on model behavior?
Position Consistency. There are two main ways of mitigating position bias in judgments either by improving pairwise judge itself (e.g., by training on both orders of data, adding consistency-bar based rewards, etc., like we do in J1) by training a Pointwise- JJ1 model that, by design. is position- pointwise judges on the
consistent. While on one hand, pairwise judges can be other hand are consistent but lack the context of reference candidates to ground their evaluations on and hence are more prone to ties.
To understand this long-standing issue of judgment bias, in Table 3, we report (i) individual accuracy for both orders of responses, (ii) position-consistent accuracy, and (iii) verdict-flips/ties. We use "verdict flip" and "tie" to refer to the same metric which is the fraction of samples where either the verdict changes based on the response order (for pairwise judges) or both responses are scored equally (for pointwise judges). consistently correct judge is thus expected to show higher position- consistent accuracy and lower ferdict-flips/ties
7

---

Published as a conference paper at ICLR 2026
Table 4: Results of Pairwise J1 models trained with different reward schemes and seed prompts.
Pairkise-J1 8B Variants
Overall
PPE
RewardBench
JudgeBBncch
RM-Bench
Follow
with Different Rewards
Positive Reward for Correct Verdict
61.8
59.8
85.7
42.0
73.4
48.3
Negative Reward for Incorrect Verdict
60.4
59.6
44,9
42.0)
Format Reward
61.0
59.3
85.6
40.3
71.8
49.3
with Different Seed Prompte
Thinking (default Figure 10)
61.8
59.8
85.7
42.0
73.4
48.3
Plan & Execution (Figure 9)
62.1
59.0
85.8
44.3
71.8
50.2
Training Rewward
Generation Length- Train Set
Validation Reward
0.3
0.7
0.5
0.5
0.5
353
1.00
Traving
Figure 5: Reward and average generation length during training for different Jl- 8B models. Pointwise-J1 J1 is trained via distant supervision derived from pairwise preference data.
We observe that when judgment quality is measured by a stricter Pointwise- outperforms Pai Jwinw-se-J1. However, if we consider a random ordering of response pairs, Pairwise-J] performs better. Moreover, the multitask formulation leverages the strengths of both pointwise and pairwise judges, outperforming separately trained judges in both random-order and position-consiitengg accuracy. In Table 8 and Appendix D, we also study the effect of training with both orders of data and verdict consistency rewards on Pai neinwise-]] J1 models.
accuracy,
Score Distribution of Pointwise vs Pairwise Evaluation. Recall that J1 -w--wweennnnnnee MttB-MultiTakk has the advantage of being evaluated in both pointwise and pairwise settings. To understand the difference in scores assigned by the model in both these judgment settings, Figure 3 shows the distribution of (i) absolute scores and (ii) score differences between the chosen and rejected gground-truthh responses. Both settings assign high scores more frequently (e.g., 8-10 are the tallest bars), but pairwise exhibits a sparser distribution and a larger gap between chosen and rejected responses than pointwise. This difference stems from their training objectives: pairwise directly contrasts both responses within the same input, enabling clearer differentiation reflected in the scores. In contrast, pointwise is trained with distant supervision one response at time, making fine-grained comparative judgments harder. Thus, combining both approaches can leverage their complementary strengths for improved quality.
Effect of Test-time Scaling, We explore test-time scaling of J1 by either conducting ceff-consis over multiple verdiets or averaging over multiple pointwise scores. We show that these can lead to further improvements (see Table 12 and 13 in Appendix D). Here, we specifically analyze whether J1 models that generate scores can achieve more accurate judgments and fewer ties by averaging scores across multiple generations at test time. In Figure we plot position-consisten accuracy and ties as a function of number of generations N. We observe improved position-consistent accuracy and reduction in ties for both Pairwise- and Pointwise-D both 8B and 70B scales. To the best of our knowledge, this is one of the first comprehensive analyses of Pointwise versus Pairwise judges, when both are trained using the same data.
Effect of Reward Schemes and Seed Prompts for Training J1. In Table 4, we first study the effect of different rewards for Pai Painw-se-]1 models. We obtain best results when only assigning positive rewards to correct verdicts adding additional format rewards or negative rewards for incorrect verdicts marginally degrades performance. Next, we also analyze the effect of two different seed prompts that are used to elicit "thinking" in J1 models. Our default J1 Thinking prompt is motivated by DeepSeek-R1. Additionally, similar to EvalPlanner (Saha et al., 2025), we experiment with prompt that instructs the model to first plan for the evaluation recipe, then execute the evaluation according to that recipe and the response(s), before generating the final verdict (Figure 10 in Appendix B). We find that J1 is robust to such choices, performing comparably with both prompts.
8

---

Published as a conference paper at ICLR 2026
In fact, with a simpler Thinking prompt, the model tends to generate richer reasoning traces, including evaluation criteria, reference answers, re-evaluations, and detailed comparisons (see Figure 1).
Reward and Thought Length Analysis of J1. Figure 5 illustrates the training and validation reward, as well as average generation length throughout different stages of J1 training. As training progresses, the thought lengths of most pairwise judges converge at around 500 tokens, while the pointwise judge tends to generate shorter outputs, typically between 300 and 400 tokens due to the absence of comparison-style tokens. Training rewards for the pairwise variants exhibit a similar steady increase before converging. In Table 7 of Appendix D, we provide a comparison of different Pairwise-J1 variants, all of which show comparable performance.
Qualitative Analysis of J1's Thinking Traces. In Figure 6, 7 and 8 we show three representative examples of JI's thinking traces. The first two are for verifiable math prompts and the third one is for a ner-verilia prompt. In Figure 6, we see that J1 identifies a calculation mistake in Assistant B's answer and provides critical feedback of how and where to improve the answer.
5 RELATED WORK
Reward Models. Reward Models have been instrumental in both training-time (Ouyang et al., 2022; Lambert et al., 2025) and test-time (Snell et al., 2025) alignment of LLMs. Traditional reward models are typically trained with the Bradley-Tent Terry objective and output a scalar score indicating the reward of the response. This design frequently results in poor calibration and generalization across different prompts and responses (Sun et al., 2025; Zhang et al., 2025). Furthermore, such discriminative models do not fully leverage the generative capabilities of L.MSs and therefore cannot be scaled up at test time, e.g., with long chain-of-thought or multiple generations (Wang et al., 2025; Shiwen et al., 2024). As a potential solution, generative reward models have emerged, which we discuss below.
LLM-as-a-Judge and Generative Reward Models. LLM-as-a-Judge and Generative Reward Models (GRMs) conceptually share similar motivation the language modeling head in LLMs is used to also output chain-of of-thought (CoT) reasoning (in the form of critiques) before generating preference judgments or rewards (Kim et al., 2024a;b; Ankner et al., 2024; Mahan et al., 2025; Ye et al., 2025; Yu et al., 2025b; Zhang et al., 2025; Saha et al., 2025). Rewards in such models could either come from training a separate reward head (typically done in GRMs) or from the LM head itself generating real-valued scores as tokens (typically done in L.MM-as-a-Judge) Prior work building LLM judges has mostly relied on either prompting (Zheng et al., 2023; Saha et al., 2024), rejection finetuning on self-generated CoTs (Wang et al., 2024c), or preference finetuning CoT pairs using DPO (Mahan et al., 2025; Trivedi et al., 2024; Saha et al., 2025; Yu et al., 2025a).
Recently, in some concurrent studies, methods like DeePSeek-GRM (Liu al., 2025b), JudgeLRM (Chen et al., 2025a), RM-RI (Chen et al., 2025b), and Reward Reasoning Model (Guo et al., 2025b) use Reinforcement Learning for building reasoning judge models. We compare J1 to these methods (in Tables and Table 4) and show that J1 achieves superior performance with significantly less data. This is achieved via J1's methodical novelty that span several axes. First, is a training recipe that is based only on synthetic data. Second, it focuses on mitigating position bias (a long-standing issue in LLM-as-a-Judge development) and multitask learning for single generalist judge, leading to the proposal of novel consistency rewards and Pointwise- J1 models trained with pairwise supervision only. Consequently, we are able comprehensively study different J1 variants that vary across sizes, modeling choices, seed prompts, and reward strategies, enabling us to draw important conclusions on building generalist thinking-judge models with -tate-of-the-art results.
Reinforcement Learning with Verifiable Rewards. J1 draws inspiration from the recent advance- ments in improving reasoning through Reinforcement Learning with verifiable rewards. Online optimization algorithms such GRPO, when combined with accurate and robust rewards, have been shown to elicit enhanced reasoning in LLMs (Guo et al., 2025a; Team et al., 2025; OpenAL, 2025). In our approach, we construct preference pairs and assign verifiable rewards based on the correctness of the model's 's judgments. By optimizing over the generated thinking traces, J1 encourages LLMs to spend more test-time compute before deriving scores and judgments.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
We proposed J1, an RL recipe for training Thinking-LLL--as--JJJ lies in converting the judgment task into a verifiable one for all kinds of task prompts, both verifiable and non-verifiable, and then optimizing the thoughts and judgments using an online RL method. We trained J1 at 8B, 32B, and 70B scales, exploring pointwise, pairwise, and multitask formulations. Our generalist J1 judge models outperformed all baselines at their respective sizes, with J1 Qwen 32B also surpassing the much larger R1 and o3 models on some benchmarks. Furthermore, we find that with only pairwise supervision, we can train pointwise judges that effectively mitigate position bias. Finally, we proposed a multitask training strategy that combines the strengths of pairwise judging (comparative context) and pointwise judging (position consistency), achieving the best overall performance and highlighting the potential Thinking-LLAN LLM-as--aJudge.
models. Our key innovation
REPRODUCIBILITY STATEMENT
All J1 models built upon open-weight Llama and Qwen models. The prompts used for data generation and training of J1 models are provided in the Appendix. Furthermore, the training code for J1 based on the open-source Verl repository, and detailed hyperparameters facilitate experiment reproducibility are listed in Appendix C.
REFERENCES
Andrei Alexandru, Antonia Calvi, Henry Broomfield, Jackson Golden, Kyle Dai, Mathias Leys, Maurice Burger, Max Bartolo, Roman Engeler, Sashank Pisupati, et al. Atla selene mini: A General Purpose Evaluation Model. arXiv preprint aXXiv22501177195 2025. URL https:
//arxiv. aaaxiiv.orggabs//5e... 17191
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan Daniel Chang, and Prithviraj Ammanabrolu. Critique-out-Loudd Reward Models. In Pluralistic Alignment Workshop at NeurIPS 2024, 2024.
URL https:tt//p/eeeeeieeiiee net wwtp:n//ooormiiiicccyyooii
Maosong Cao, Alexander Lam, Haodong Duan, Hongwei Liu, Songyang Zhang, and Kai Chen. Compassjudger-1: 1: All-In-One Judge Model Helps Model Evaluation and Evolution. arXiv preprint arXiv:2410.16256 2024. URL https: / / aaxiv...vvv...i. arg/abb/2423 16256.
Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs as the Judge? A Study on Judgement Bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8301-8327. Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 101555//11222..emlmmmaaaa org/2024. emnlp- main. 474.
URL
https://aclanthnn
Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang. Bryan Hooi, and Bingsheng He. JudgeLRM: Large Reasoning Models as a Judge. arXiv preprint arXiv:2504.0000 2025a. URL
https://arxiv.. http://aaxiii.oorr///////// 250000000040000.
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. RM-R1: Reward Modeling as Reasoning. arXiv preprint 0223:22005.02387 2025b. URL https://arxáv www.oorg///////aooooooooosssss//
abs 2w..2025533222
Evan Frick, Tianle Li. Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. How to Evaluate Reward Models for RL.H. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevieee nttt:/orrmmdd-ccttttLLoo http://///eeeeeooooo.////roooooo
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 Herd of Models. arXiv preprint 2rXiv:2407.21783, 2024. URL https://arxiv.or hargg/bb/2400 21783.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A Survey on LLLM-as-a-udgeee arXiv preprint arX1v:2411.15594, 2024. URL https::///// //tssi//////i///// wwtpr/iwwwwg../////eeeeooo /2411. 15594.
10

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948, 2025a. URL https: l/arxiv.org/abs/a//..../aaa/../t org/abs/2501. 12948.
Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Reward Reasoning Models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b. URL https:///pperevilie
net nett/forum22d--88k2222ccr
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https::/openrevieew wetttoorum1ddd77yytttta2
Aaron Hurst, Adam Lever, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o System Card. arXiv preprint 21276.
arXiv:2410.21276, 2024. URL
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Alcksander Madry, Alex Beutel, Alex Carney, et al. OpenAI ol System Card. arXiv preprint arXiv:2412.16720, 2024. URL https: : http::::////vvvv/// org. /2010./222...22 167..
Seungone Kim. Jamin Shin. Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Scongjin Shin, Sungdong Kim, James Thore, et al. Prometheus: Inducing Fine-grained Eval- uation Capability in Language Models. In The Twelfth International Conference on Learning Representations, 2024a. URL https: : /ttp:///ppeeeeeeeeeeeeeeeeeeeeee ew. net www.oooonnninnuaaaaa
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Sco. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4334 4353, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10. 188138//11222 http://..22222...2mmmm/e main. 248. URL hwwph:ttp//too htattp::/aaclanthol..g//// o ww2/ emn lp- main. 248.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with Attentittntion.. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. URL https:://l...t. acm. htt//do/aab////// /00. 0145/3660000 3613165.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Rewardbench: Evaluating Reward Models for Language Modeling. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 1755-1797, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89 17--9997-11111 1177-699777 doi: 10. 1ttp:///2222..mmmle findings-naacl.66 //aclantho Logy. org/ 2025. f findings- naacl.
naacl.96. URL https:
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://openreviiee fwt//fommmifi QEHrmQPBdd.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-Time Scaling for Generalist Reward Modeling. arXiv preprint arXiv:2504.02449 2025b. URL https://arxvv.org//ee
Dakota Mahan, Duy Van Phung, Alon Albalak, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato. Jan-Philipp Fränken, and Chelsea Finn. Generative Reward Models, 2025. URL
https://openreview net/ ttttfforummidd-M/222LL..
OpenAl. Openai o3 and 04-mini system card, 2025. URL https://cdm wotiddd.oppaii.co//pdoooo 2221c875- 02dc 4789-8000--00 7775813722111 and 04-0044-mmm -ini system- papstem-card..ee card.pdf.

---

Published as a conference paper at ICLR 2026
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022. URL. https:://proceediingg neurips cc/ppper_ weweerffflleepppeeeeeeeeee https/ppap///2///a////// 0bef6550e344799--58550077------2 Abstt00111111-Attaaaaaaaa cot.essetttttttcoeeeneeeeeeeeee html.
Conference.
Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch- Solve-Merge Improves Large Language Model Evaluation and Generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8345-8363, 2024. URL wrg/2024....2 naacl long. 462.
httts::/aalantholl..
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason E Weston, and Tianlu Wang. Learning to In Forty-second International Con-
Plan & Reason for Evaluation with Thinking- ference Machine Learning, 2025, URL https://openreviiee
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepseekMath: Pushing the Limits of Mathemat- ical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300, 2024. URL
https://arxiv. orggabs/24022 03300.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A Flexible and Efficient RLHF Framework. In Proceed- ings of the Twentieth European Conference on Computer Systems, EuroSys '25, pp. 1279-1297, New York, NY, USA, 2025, Association for Computing Machinery. ISBN 9798400711961. doi: 10.1 04.144/368003133696
URL https: //doi. ortoorgg////// 11e11111519990 3696075.
Tu Shiwen, Zhao Liang Chris Yuhao Liu, Liang Zeng, and Yang Liu. Skywork Critic Model Series. https: https://hugginng https///hhggnnggacc.oooo/ee co/Skywork.
September 2024. URL https://huggingffh
Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning. In The Thirteenth International Conference Learning Representations, 2025, URL https://openreview... forum?id=4 4onunttn=
Hao Sun, Yunyi Shen, and Jean-Francois Ton. Rethinking Reward Modeling in Preference-based Large Language Model Alignment. In The Thirteenth International Conference on Learning Representations, 2025, URL httpss// www./trrur/eeeeeee ttntt/forumiddrffto id=rfdb lEfrbb...@m..
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and lon Stoica. JudgeBench: A Benchmark for Evaluating LLM-Based Judges. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevieet net/fforum1d-GGOkkFaaa
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, al. Kimi k1.5: Scaling Reinforcement Learning with
LLMs. arXiv preprint arXiv:2501.12599, 2025. URL https: : //wwxiv.i/vv////////////aa//aiv
http://wwww..rrgbr.a/a///aaaa orgg 12599.
Prapti Trivedi, Aditya Gulati, Oliver Molenschot, Meghana Arakkal Rajeev, Rajkumar Ramamurthy, Keith Stevens, Tanveesh Singh Chaudhery. Jahnavi Jambholkar, James Zou, and Nazneen Rajani. Relf-Rationalization Improves LLM as a Fine-grained Judge. arXiv preprint urXiv:2410.054993 2024. URL https: /tttps:://aaiiiii http//aawvvv...gaaaaaa 2410. 05495.
Haoxiang Wang. Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts. In Findings of the Association for Computattional Linguistics: EMNLP 2024. pp. 10582-10592, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10. 1011155/112224 findiingsiinns---ee wwwdiigns-mmnlpppp emnlp.620. URL https::/acllathhology/ www.ollloy.oooo/////// 222/ findings emn womll.66.. 620.
PeiFeng Wang. Austin Xu, Yilun Zhou Caiming Xiong, and Shafiq Joty. Direct Judgement Preference Optimization. In Proceedings of the 2025 Conference Empirical Methods in Natural Language
12

---

Published as a conference paper at ICLR 2026
Processing, pp. 1979-2009, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8 8 999-8889166-33--6 doi: 10.18653 caamlppmmin.iie main. URL https: //aclantho logy wrg. emnlp- main. 103/.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large Language Models are not Fair Evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. Thailand, August 2024b. Association for Computational Linguisties. doi: 10. 18653//1/2024. acl-long.511. 1. URL https::/// /attps://aclus http:///aaaaattohooooooo org/ wrgg//22440000 acl- long. 511.
Bangkok,
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan. Jane Dwivedi- Yu, Jason Weston, and Xian Li. Self-T Evaluators. arXiv preprint arXiv:2408. 208:22008.02666, 2024c. URL https: /aaxxvv.avvvv///a///////aaaaaaa wwt./ogg//////ooooooo........../ /408. .2666..
Richard Yuanzhe Pang, Maryam
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. 2025. URL https://arxiv..rr wttp:/ii.orgaas/////oe
09388.
Zihuiwen Ye, Fraser David Greenlee, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallé. Improving Reward Models with Synthetic Critiques. pp. 4506-4520, April 2025. doi: 10.18653/v1/205. findings naacl.
ffn6d////02...indining..... naacl.254. URL https: htttts////alattholoo.ooo////
Jiachen Yu, Shaoning Sun, Xiaohui Hu, Jiaxu Yan, Kaidong Yu, and Xuelong Li. Improve LLM- as-a-Judge Ability As a General Ability. In Christos Christodoulopoulls, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 14099-14115, Suzhou, China, November 2025a. Association for Computational Linguisticss ISBN 979-8-89117 186--33--7777-66666 doi: 10. 18.588 1/202/ http0222222222maaii 72..
URL https:://atlathhoooo..oogg
emn main. 712/.
Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang. Yundi Qian, Xuewei Wang. Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, Critiques Boost Reward Modeling for Language Models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp- 11499-11514, 1e Albuquerque, New Mexico, April 2025b. Association for Computational Linguistics. URL
and Rui Hou. Self-Generatee
ISBN
http///callnooooggyyy.ooo org/ 2025 naacl- long 573.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative Verifiers: Reward Modeling as Next-Token Prediction. In The Thirteenth International Conference on Learning Representations, 2025. URL Cowp4tFEtE.
wwt.teettouonnee formm?dd
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. WildChat: 1M ChatGPT Interaction Logs in the Wild. In The Twelfth International Conference on Learning Representations, 2024. URL
net
wwt/forrneee forum? 1d=
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023. URL https://proceedinng neurips...r...p...... cc/paper_ fomeer_ffllllss httes//aper/222//aaste 9.178822227998777777755555222222 Abst wwwttratt-aattettt...o.o.s
Datasets and Benchmarks.. html.
13

---

Published as a conference paper at ICLR 2026
A JI's THINKING TRACES
Here Figure 6, Figure 7 and Figure 8 we show additional representative examples J1's thinking traces. The first for verifiable math prompts and the third one is for non-verifiable prompt.
Example of J1's Thinking Trace for Verifiable Math Question
<think
Okay, let's see. The user wants know which assistant's answer better. The problem about finding sum the sums two two-element subsets the set (1,2,3.4.5,6).
First, need to check if both answers 105. Let me verify that. The number of two-element subsets in 6-element set is C(6,2) = 15, which matches the problem statement. Each element appears in (6-1) 5 subsets. So each number from to 6 is included times in the total sum. The sum of the numbers 1 through 6 is 21. So total sum would be 5+21 = 105. That's the correct answer.
correct. The correct answer should be
Now, looking at Assistant A's answer. They all subsets, compute each sum correctly, add them up step by step, and get 105. The final answer boxed correctly. The steps clear and methodical.
Assistant B's answer also lists the subsets and computes each sum However, when adding them up, they make mistake. The sums they list are correct, but when they add them, they group them incorrectly.
For example, they say 3 + +
+ 8 + + 8 + + + 10 11. Let me add those numbers: 3+4=7, +5=12, +6=18, +7=25, +5=30, +6=36, +7=43, +8=51, +7=58, +8=66, +9=75, +9=84, +10=94, +11=105. Wait, that's 185. But Assistant says they simplified to That's clear error. They must have miscalculated when adding. The final answer they give 47, which is wrong.
+ 7 5
So Assistant A's answer is correct, while Assistant B's is incorrect. The user's problem requires the correct answer, so Assistant A's response is better. Additionally, Assistant B's explanation of the addition step is flawed. They mention grouping the sums into 13 10 11 + 13 47, which doesn't 't make sense. The correct total is 105.
Therefore, Assistant A's answer is accurate follows instructions properly, while Assistant B's answer contains critical error leading to incorrect result. The user's instructions emphasized providing clear, step-by-step solution, which Assistant did correctly. Assistant B's answer, despite being detailed, has mistake in the final addition step, leading to an incorrect answer.
</think
Figure 6: Thinking trace for a verifiable prompt. first self-generates reference answer to the math problem (highlighted in brown). Subsequently, it checks the correctness of Assistant A's answer and judges as correct (highlighted in green). Then it concludes that Assistant B's s answer is incorrect and also provides feedback by pointing to calculation mistake (highlighted in blue).
14

---

Published as a conference paper at ICLR 2026
Example 2 of J1's Thinking Trace for a Verifiable Math Question
<think>
Okay, let's see. The is asking about the of isosceles right triangle with hypotenuse endpoints at A(5, and B(- Both assistants provided answers, but I need to figure out which one is correct and better explained.
So..So Assistant A's answer is correct.
Now Assistant B. They also calculated the hypotenuse as sqrt Then they set the equation for the legs using Pythagoras: 2r²² = 52, leading to =2= Then they tried to compute the area as (1/2)=base+height, using hypotenuse as base and x as height. But wait, in a right triangle, the legs are the base and height. If you use the hypotenuse as the base, the height isn't t one of the legs. The correct height relative to the hypotenuse is (leg * leg))hypotenuse, which is 26/sqrt(52) sqrt(52) (Qrr))))))) that's (52 Let's compute that: 52*26 1352. (sgrt((5) (135() is sgrt(52)26) sqrt(4+13-26) 2xsqrt(338) 2+18.38 36.76. Divided gives 18.38, which is not 26. But Assistant B simplified it (1/2) (52) (1) 26, which is incorrect. They made mistake in simplifying the product of sqrt(52) and s2rt(26). The correct product is sqrt(52-26) sqrt 36.76, not 52. So their final answer of 26 is wrong.
when they multiplied
Therefore, Assistant A's answer is correct with 13, while Assistant B's is incorrect. The key error in B's solution is the incorrect simplification of the product the hypotenuse and the leg as instead the correct value. Assistant A's approach more straightforward and correct. Both solutions are step-by-step, but B's mistake in the final calculation leads to incorrect answer. So Assistant A's response is better.
</think>
Figure 7: Thinking trace J1 for a verifiable math prompt. The model analyzes Assistant B's answer in step-by-step manner, finds error in the simplification step, and provides detailed feedback (highlighted in blue).
15

---

to for Published as a conference paper at ICLR 2026
Example 3 of J1's Thinking Trace for a Non- Verifiable Writing Prompt
<think>
Okay, let's see. The user asking about the requit arc flash PPE, specifically mentioning maintenance, use, inspections, and shock protection areas like limited restricted approach boundaries. They want the only qualified persons.
Looking at Assistant A's answer: They talk about qualified persons needing to identify PPE, ensure maintenance and inspections, understand shock protection areas, and train others. That covers the points mentioned. But wait, the user's question is about the requirements for arc flash PPE, so maybe Assistant A is adding more about training others, which might be relevant but not directly the requitements for the itself.
Assistant B's parts. They mention being trained, knowing equipment, hazards, PPE, inspections, and understanding the approach boundaries. This seems more focused the qualifications and responsibilities of the person, which aligns with the user's instruction to make the answer for only qualified persons.
starts by stating only qualified persons work exposed
The user's question includes the requi for PPE and shock protection areas. Both answers mention these, but Assistant B's answer is more concise and directly addresses the requirements without adding extra info about training others. Also, Assistant starts by stating that only qualified persons directly
allowed, which the user's instruction. Assistant A's answer might be detailed but includes elements not directly asked for. So Assistant B's answer is more precise and better follows the user's specific request.
</think>
Figure 8: Thinking trace J1 for non-verifialle prompt.
16

---

Published as a conference paper at ICLR 2026
B PROMPT TEMPLATES
Figure 9 shows the seed prompt to train our primary J1 recipe: Pairwise-J1 with verdict. Figure 10 shows an alternative seed prompt for training a similar Pairwise-J1 with Verdict setup. Motivated by EvalPlanner (Saha et al., 2025), this prompt instructs the model to first plan the evaluation recipe and then execute it as part of the thinking process. Figure and Figure 12 show our prompts for "Pairwiz rwise-J1 with Scores" and "Pairwise- J1 with Scores& Verdict" variants respectively. Finally, we adapt our pairwise prompts to a pointwise prompt used train our Pointwise-Ji model that instructs the model to think and assign real-valued scores between 0 and 10, shown in Figure 13.
Thinking Seed Prompt Template for Training Pairwise J1 with Verdict
You are given user question and two responses from two AI assistants, Your task to an impartial judge and evaluate which response better follows the user's instructions and provides higher-quality answer.
First, provide your reasoning within <think> and </think> tags. This should include your evaluation criteria for a high-quality response, a detailed comparison of the two responses, and when helpful, reference answer as part of your evaluation. Be explicit in your thought process, referencing your criteria and explaining how each response aligns with or deviates from them.
Avoid any position biases and ensure that order which the responses were presented does not influence your decision. Do not allow the length of the responses influence your evaluation. Do not favor certain names of the assistants. Be objective possible.
Finally, provide your verdict within <answer> and </answer> tags, strictly
following this format:
<answer> [[A]] </answer> if Assistant is better
<answer> [[B]] </answer> if Assistant is better
Below the user's question and the responses:
[User Question] (instruction)
[The Start of Assistant A's Answer]
(response A}
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
{response B} [The End of Assistant B's Answer]
Figure 9: Thinking seed prompt template for Pai marieee-ao --Jl -J1
with Verdict.
17

---

Published as a conference paper at ICLR 2026
SyalPlanner-style Seed Prompt Template for Training Pairwise J1 with Verdict
You are given user question and two responses from AI assistants. Your task to act impartial judge and determine which response better follows the user's instructions and provides higher-quality answer.
First, build an evaluation plan that can then be executed to assess the response quality. Whenever appropriate, you can choose to also include beep-by-step reference answer as part of the evaluation plan. Enclase your evaluation plan between the tags <planz and </plans.
Next, execute the plan step-by-step to evaluate the responses. Avoid copying plan when doing the evaluation. Please also only stick the generated plan and provide an explanation of how the plan is executed responses. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. Enclose your plan execution between the tags <execution> and </execution>.
compare the
Finally, output your final verdict by strictly following this format: <answer> [[A]] </answer> Assistant is better
<answer> [[B]] </answer> if Assistant is better
Below are the user's question and the two responses:
[User Question] (instruction)
(The Start of Assistant A's Answer]
(response A} [The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
(response B} [The End of Assistant B's Answer]
Figure 10: EvalPlanner-style (plan + execution) Prompt template for Pairwise- J1 with Verdict.
18

---

Published as a conference paper at ICLR 2026
Thinking Seed Prompt Template for Training Pairwise- J1 with Scores
You are given a user question and two responses from two AI assistants. Your task is to act as an impartial judge and evaluate which response better follows the user's instructions and provides a higher-quality answer.
First, provide your reasoning within <think> and </think> tags. This should include your evaluation criteria for high-quality response, detailed comparison of the two responses, and when helpful, reference answer part of your evaluation. Be explicit in your thought process, referencing and explaining how each response aligns with or deviates from them.
criteria
Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses influence your evaluation. Do not favor certain names of the assistants. Be objective possible.
Finally, assign the assistant's response score from 0 to 10, using either integer decimal with up to 0.1 precision, with higher score indicating higher-quality response that better satisfies the criteria. Enclose the scores within the tags <score_A> </score_ and <score_ B> </score_B>.
Format your output like this:
<think> your chinkinggproceee process </think> <score your_score.3 </score <score sourescore_b </score_B B>
Below the user's question and the responses:
[User Question] (instruction)
[The Start of Assistant A's Answer]
(response A} [The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
(response B} [The End of Assistant B's Answer]
Figure 11: Thinking seed prompt template for training Pairwise- J1 with Scores.
19

---

Published as a conference paper at ICLR 2026
Thinking Seed Prompt Template for Training Pairwise with Verdict and Score
You are given user question and two responses from two AI assistants. Your task is to act as an impartial judge and evaluate which response better follows the user's instructions and provides a higher-quality answer.
First, provide your reasoning within <think> and </thinks tags. This should include your evaluation criteria for a high-quality response, a detailed comparison of the two responses, when helpful, reference answer as part your evaluation. and explaining how each response aligns with deviates from them.
explicit in your thought process, referencing your criteria
Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible.
Finally, assign the assistant's response a score from 0 to 10, using either an integer or decimal with up 0.1 precision, with a higher score indicating higher-quality response that better satisfies criteria. Enclose the scores within the tags <score_A> </score_A>, and <score_ </score_
Finally, provide your verdict within <answer> and </answer> tags, strictly following this format:
<answer> [[A]] </answer> if Assistant is better
<answer> [[B]] </answer> if Assistant is better
Below are the user's question and the two responses:
[User Question] (instruction)
[The Start of Assistant A's Answer]
{response A}
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
(response B} [The End of Assistant B's Answer]
Figure 12: Thinking seed prompt template for training Pai rwise-J1 J1 with Verdict and Score.
20

---

Published as a conference paper at ICLR 2026
Thinking Seed Prompt Template for Training Pointwise J1
You are given user question and response from AI assistant. Your task is to act user's instructions. You will be shown multiple responses to the same prompt, but only one at time. Evaluate each response independently.
impartial judge and evaluate how well the response fulfills the
Think carefully about how to assess the quality of the response, and enclose your reasoning within <think> and </think> tags. Your reasoning should include evaluation criteria, a clear understanding of what an ideal response would look like for this particular question, and a concrete example such an ideal or reference answer if possible. Then compare the assistant's response to your ideal reference answer, explaining how it aligns with deviates from your expectations. Be specific and avoid vague overly general judgments. Remain objective possible.
Finally, assign the assistant's response a score from 0 to 10, using either an integer or a decimal with up to 0.1 precision. A higher score should indicate a higher-quality response. Enclose the score within <score> and </score> tags.
Format your output like this: <think> thur_ihi process </think> <score> your-score </score>
Below are the user's question and the assistant's response:
[User Question] (instruction)
[The Start of the Assistant's Answer] {response) [The End of the Assistant's Answer]
Figure 13: Thinking seed prompt template for training Pointwise-Jl.
Prompt for Creating Preference Pairs for Training
Below conversation between an user and an Assistant. (instruction)
The start of Assistant's Answer
(baseline response)
Please first generate modified instruction that is highly relevant but not semantically identical to the instruction above from the user Then write a high-quality answer which is a good response to the modified instruction but not a good response to the original user question. IMPORTANT: Please strictly follow the following format:
User Question Modified
<provide a modified instruction here>
The start of Assistant's answer to the modified instruction
<provide a high-quality response to the modified instruction end Assistant's answer to the aodified instruction
Figure 14: Given prompt and a baseline response, the prompt asks the LLM to generate noisy instruction and then a response that noisy instruction. The response the noisy instruction subsequently is selected as the rejected response to the original instruction.
21

---

Published as a conference paper at ICLR 2026
Table 5: An example of synthetically generated training pair for J1 training. The rejected response is a good response to the noisy instruction, thus making it a bad response to the original instruction.
Original Question
Noisy Question
Chesee Response
Rejected Response
There are objects arranged in line. Somcone says "the table is to the right the bag, the ta- ble is the left of the hox and the bag is to the right of the box." Is the situation just scribed possible or ampossible? Before answering, take doop breath. Think stop by step be- fore giving a final answer. My survival depends on your swer being correct, so concen- trate bard Please analyze this new situation and determine whether possible or impos- sible.
There are objects arranged in a line. Someone says "the ta- ble is to the right of the bug. the box is the left of the ble and the hag is to the left of the box." " Please analyze this new situation and determine whether it is possible or impos- sible.
break down the state- ments: "The table is to the right of the bag." "The table left of the box." " "The to the right of the box." From statement 3, know the order is: box -> bag From statement I. know the order is: bag >> table Combining these two, gel: box is bug is table Now, let's look at statereett table is to the left of the box." This statement contradicts our previous conclusion. If the ta- is to the left of the box. then the correct order should be: table >> box > bug How this contradicts statement 1. which says the table is the right bag. Therefore. conclude that the situation de- scribed is IMPOSSIBLE, The statements mutually contra- dictory, and it's not possible for all three statements be true at the same time."
Let's break down given statements: The table to the right of the bag. 'The box is to the left of the table. The to the left of the box. From statement we know the order is: bag >> box From statement we know the order is: box table Now, let's combine these urders: bag >> box +> table (from statements and 3) State- ment says the table to the right
the is
'The
bag. which is con- sistent with this order. Conclu- sion Based on the analysis, POSSIBLE for all three state- ments to be true at the same time. The correct order is: -> box -> table.
C EXPERIMENTAL SETUP
For training, the policy actor generates 5 rollouts per prompt using ancestral sampling with tempera- ture 1.0. Training regime uses a learning rate of le - 6 (decayed to 3e - 7 in later steps for pairwise J1-Llama- -7B), and a total batch size of 512. The maximum sequence length is set to 4096 tokens for both inputs and outputs.
We experimented with different KL coefficients from {0.1, 0.001, 0} for J1- J1-Llama 8BB and selected 0.01 as the best-performing value based on development set accuracy. For J1-Llama- -0BB we set the KL coefficient to to encourage more exploration. Preliminarily experiment with entropy bonus during training showed degraded performance: the model tends to generate longer but more repetitive output. See Table 10 in Appendix D for comparison of KL penalty and entropy bonus.
We use A100 to train Jl-aaa 8B, 32xA100 GPUs for J1- til--wen-i 32B and 64 x A100 GPUs for J1- Llama 70B. Inference is done using 8x A 100 GPUs. Tensor parallelism is set to for both training and inference. During inference, we maintain a maximum generation length of 4096 tokens. For inference-time scaling we use sampling with top-p of 0.95 and temperature of for Llama, and the default 0.6 for Qwen3.
D ADDITIONAL RESULTS
Comparison of Different Pairwise-J1 Models. In Table 7, we compare the three Pairwise-J] variants, that as part of the final answer, generate either: (i) only the final verdict, (ii) only real-valued scores for both responses, or (iii) both. We observe that predicting the verdict (without the scores) performs as well as other variants. Having access to scores, however, has other advantages e.g., in quantifying the degree of preference or to rank across multiple responses. Next, in Table 8, we compare the effect of training Pai mairiss--] models with both orders of data and with verdict consistency rewards. We observe that adding consistency reward improves position-consistency accuracy of J1 models and consequently, also reduces the fraction of verdict flips.
Comparison of Different Decoding hyperparameters for J1 Models. In Table 9, compare greedy decoding to temperature sampling of J1 models. We find that our models are robust to such choices, exhibiting consistent performance with negligible variance.
22

---

Published as a conference paper at ICLR 2026
Table 6: Results RewardBench comparing our J1- 31-Qwen-322 32---M2B-MMM1 Task model with other top performing generative reward models. Results for most of these models are from the RewardBench leaderboard.
Models
Overall
Chat
Chat-Hard
Safety
Reasoning
Open and Closed LLM-as-a-huulee
Llama3.1 88--nnstrct
64.4
Hlmma3.37700 100-:.3.-7BBIntttuu
85.4
96.9
77.4
77.6
89.6
Llama3. 1-0ttc-Intttuct
84.1
97.2
74.6
77.6
87.1
Claude-3.5-Somec
84.2
96.4
74.0
81.6
84.7
GPT-4o
86.7
76.1
88.1
86.6
Gemini- 1.5-Pro-0514
88.2
92.3
80.6
87.9
92.0
OpenAl-ol-man
87.1
94.4
78.7
80.9
94.2
OpenAl-o3
92.7
80.5
79.8
92.7
DeepSeck-R1
90.6
95.3
83.6
86.4
97.4
SOTA Generative Reward Models
JudpussJudger 1--1-328
al., 2024)
facebook/Self- wow.eff-tttggttttttoro-oooooooo -aa.ealaammm-----........1.1111. 1-70B (Wang al., 2024c)
90.0
96.9
85.1
89.6
88.4
Salesfores/SFR nemo- 12B Judger (Wang et al., 2025)
90.3
97.2
82.2
86.5
95.1
3F-Foumdation//een Twwee/ea-eiviffeeeeeeeeee Ofii--fsssttii12222
91.0
91.9
86.6
92.0
93.6
Reward Reasoning Model (Guo al., 2025h)
98.3
RM-RI (Chen al., 2025h)
91.4
95.3
83.1
91.9
95.2
AtlaAl/Seleme- (Alexandru et al., 2025)
92.4
97.8
84.0
92.2
95.7
R-I-S- D-S-E/RSEE-Judge-OO SaarSS---uuee--eeeeee Qweese--eeeen2.-33 2.5-32B et 2025a)
92.7
96.6
83.3
91.9
98.8
Balesfore/SERR-LLMMa-3333 Judgee- (Wang 2025)
84.8
97.6
-.----lmmma--11000 3,1-70B (Shiwen et al., 2024)
93.3
96.6
87.9
93.1
95.5
Dooundation/exxtvvlllllaam
93.5
94.1
90.1
93.2
96.4
J1- Qwen Multiiakk (0urs)
93.6
96.4
89.5
90.5
98.1
Table 7: Results of Pairwise-J J1 models trained with different recipes. input instruction, b pair of responses, t: intermediate thought, y: verdict, Sa, 8b scores.
J1 88 Variants
Overall
PPE
RewardBench
JudgeBench
RM-Bench
FollowBenchhEvll
w/ Verdict: : (ir, 6) (f.3)
63.9
59.8
85.7
42.0
73.4
48.3
Scores: (2,0,8) + (t, 46]
60.2
85.8
41.7
72.3
46.3
w/ Scores& Verdict: (x, (t, 9)
61.7
59.5
85.1
41.4
71.5
41.0
Effect of KL Penalty and Entropy Bonus in GRPO for training J1. In Table 10, we study the effect of KL Penalty and Entropy Bonus GRPO when training a Pairwise J1 L-lam-88 model. In our experiments, we find that more exploration generally leads to some degradation in performance.
Comparison of J1 with a Scalar RM Trained on Same Data. In Table 11, we compare J1 scalar RM trained on the same base model (Llama-3.1 Instrrru using the same training data. We observe that J1 outperforms the corresponding sealar model on four out of five benchmarks with the maximum improvement coming in the hardest RM-Bench benchmark.
23

---

Published as a conference paper at ICLR 2026
Table Results on PPE Correctness and JudgeBench comparing different position-bias mitigation strategies for Pairwise-J1 Consistent Accuracy: examples are judged correctly in both response orders. Verdict Flip: cases where the pairwise verdict changes when response order swapped.
PPE Correctness
JudgeBench
Pairwise-31 8B Variants
(a, b)
(b, a)
Consistent Ace
Verdict-
(a, b)
(b,
Consistent Ace t
Verdict- Flip/Ties !
Acc
Ace
Flip 4
Acc f
Acc
Llama-3.1 1.0--3118111 ----nruttctt
54.7
54.1
30.2
44.1
67.4
42.3
32.3
37.4
Random Single-order Data
58.3
57.6
38.3
36.7
48.3
59,4
36.6
32.9
Both-order data Verdict Consistency Reward
59.2
58.4
39.1
36.8
63.1
51.4
42.0
27.7
58.4
58.2
43.9
28.7
52.3
64.6
45.4
26.0
Table Comparison of Greedy Decoding and Temperature Sampling showing the robustness of
models to different decoding temperatures.
Model
Decoding Temperature
PPE Correctness
J1-Llama- 708
Greedy (with t=0.0)
72.9
J1 21-LLamm--700
Temperature Sampling (8 seeds with t=0.6)
72.8 0.1
J1 Maetnn32B-MMMuttTT-
Greedy (with (=0.)
76.8
J1 J1-Qwen-30 Meten-32B-MuutiT--
Temperature Sampling (8 seeds with t=0.6)
77.0 1: 0.1
Table 10: Ablation studies on Pai mairsseJ11 J1 with verdict with KI penalty and entropy bonus.
Pairwise-JD J1 SB Variants
Overall
PPE
RewardBench
JudgeBench
RM-Bench
FollowBenchEval
w/ KI. Penalty
63.9
59.8
85.7
42.0
73.4
48.3
w/o KL Penalty
61.1
59.6
84.9
42.9
69.8
48.3
w/o Entropy Bonus
63.9
59.8
85.7
42.0
73.4
48.3
w/ Entropy Bonus
58.3
59.1
84.1
39.4
71.7
42.4
Table 11:
Comparison of J1 with scalar RM trained on same data.
Model
Overall
PPE
RewardBench
JudgeBench
RM-Bench
FollowBenchhEal
Bradley-Terr Terry (scalar)
66.6
57.5
82.6
74.0 73.4
51.1 58.5
67.8
J1-Llame 8B (generative)
69.2
59.8
85.7
68.8
Table 12: Test-time scaling of Pointwise-J1 models on PPE Correctness. Judgments are made based on the average scores of the responses.
Pointwise-J1
Overall
MMLUiPPo
MATH
GPQA
MBPP-Plus
IFEval
J1-LLama-70B Greedy
65.0
73.4
77.4
$8.9
60.7
54.6
J1-L.lama-70B -
64.9
74.0
79.7
58.2
55.7
59.7
J1-Llama-70B (Mean-Score@32)
74.8
81.2
87.6
67.3
81.9
70.8
Table 13: Test-time scaling of Pai seirwsse-D models on PPE Correctness. Judgments are made based on majority vote over multiple verdicts.
Pairwise-J1
Overall
MMLU-Pro
MATH
GPQA
MBPP-Plus
IFFyal
J1-LLama-708B Greedy
72.9
79.0 79,9
86.0 88.1
65.9
66.0
67.3
J1-LJama-70B (SC@33
73.7
66.5
66.5
67.2
24


Reviews:
[{"review_id": "D4qdZ72S6q", "reviewer": "Reviewer_Rk8V", "summary": "This paper proposed J1 model which combines verifiable rewards and reinforcement learning, enabling LLM judges to develop deeper reasoning, fairer evaluation, and more consistent decision-making across diverse tasks.", "questions": "see above weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a unified framework that converts both verifiable and non-verifiable evaluation tasks into verifiable formats using synthetic preference pairs; applies online RL to directly optimize the chain-of-thought reasoning in LLM judges; showing a novel consistency-based reward that enforces the same judgment regardless of response order; and develops a multitask J1 model that jointly learns from both pairwise and pointwise supervision. Although similar ideas have been employed in other papers, their application to LLM Judge is novel.", "weaknesses": "1.\tThe current setup focuses solely on pairwise and pointwise evaluation, without exploring extensions to multi-response or listwise judgment\n2.\tThe work defines both Verdict Correctness and Verdict Consistency rewards, but lacks any reward weighting or sensitivity analysis\n3.\tThe data used for training and evaluation primarily covers conversational and reasoning domains, with no experiments on diverse areas such as code or multimodal judgment\n4.\tThis paper omits any discussion of training cost or computational resources", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dARNZZTodR", "reviewer": "Reviewer_dZdp", "summary": "This work uses RL to train thinking LLM judges and show that it achieves SOTA performance on multiple benchmarks. Here are the contributions:\n1. a recipe that converts different judgement tasks into a unified format with verifiable reward to train with RL\n2. RL training on the thinking judges that shows strong performance improvement across various benchmarks\n3. Develop methods to address positional bias through consistency reward.", "questions": "1. It seems like for Qwen32B, there is a much lower improvement (e.g. +3.5 overall, +0.7 on JudgeBench) compared to the base model, whereas Llama model shows a much stronger improvement. Can the author comment on why this is the case?\n2. Can the authors elaborate on the compute cost (e.g., GPU-hours, convergence time) of J1 (GRPO) versus the DPO-based EvalPlanner baseline? This would provide a more complete picture of the trade-offs involved.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Strong Empirical Results: The model trained using RL showed strong and consistent improvement across benchmarks, and is able to match frontier model (e.g. o3-mini, Deepseek-R1) that is an order of magnitude bigger.\n2. Comprehensive Ablation & Analysis: The author provides thorough analysis on things like positional bias in the ablation study, which helps better understand the behavior of the model, and show that through the consistency reward, the \"Verdict Flip/Ties\" rate decreases.", "weaknesses": "1. Training Complexity: while J1 shows stronger performance than EvalPlanner, which uses offline DPO training, it is not a fully apple-to-apple comparisons. It is unclear how these two methods (GRPO vs DPO) differ under the same compute budget.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9carn8DvMO", "reviewer": "Reviewer_UxRN", "summary": "The paper proposes J1, a reinforcement-learning (RL) framework to train “LLM-as-a-Judge” models to think before deciding. The key idea is to convert both verifiable (e.g., math) and typically non-verifiable judging tasks into a unified, verifiable formulation; then optimize the thinking traces and final decisions using GRPO, with rewards for verdict correctness and order-consistency to mitigate position bias. J1 is instantiated on Llama-3.x and Qwen3 (8B/32B/70B) and evaluated on PPE, RewardBench, JudgeBench, RM-Bench, and FollowBenchEval, reporting SOTA or competitive results—especially for J1-Qwen-32B-Multitask—despite training on ~22k synthetic preference pairs. The paper includes ablations (pairwise vs pointwise vs multitask; reward variants; seed prompts) and qualitative analyses of learned evaluation strategies.", "questions": "- How sensitive are results to adding graded thought-quality rewards (e.g., rubric adherence, factual checks) vs the current binary signals?\n- For “non-verifiable” tasks, what fraction of pairwise labels are ambiguous/tied under careful human review?\n- How does J1 fare on unseen domains/languages and with adversarial perturbations (e.g., equal-quality or equal-error pairs) that should produce ties?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Unified, verifiable training recipe across “verifiable” and “non-verifiable” prompts enabling direct optimization via RL, not just DPO/SFT.\n- Bias mitigation at training time via batching both orderings and an order-consistency reward; clean and effective.\n- SOTA or competitive results on multiple judging benchmarks at reasonable model sizes, with notably strong PPE Correctness and RewardBench numbers; comprehensive comparison tables.\n- Thoughtful ablations (rewards, prompts; pairwise vs pointwise vs multitask) and qualitative traces illustrating learned strategies (criteria, reference answers, self-correction).", "weaknesses": "- “Non-verifiable” to verifiable conversion is underspecified. For subjective tasks, the paper leans on synthetic construction and pairwise labels; the validity of these labels as ground truth for reward is not rigorously validated with humans.\n- Limited causal analysis of bias reduction. While order-consistency improves, the paper doesn’t isolate the effect of batched dual-ordering vs consistency reward vs prompt phrasing with confidence intervals on all benchmarks.\n- A minor detail is that while prompts and some details are provided, RL stability (seed variance, KL control, learning rate sweeps, early stopping criteria) could be reported more completely to ensure replicability of gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6pTOaV8RKN", "reviewer": "Reviewer_TdB2", "summary": "This paper introduces an RL-based method for teaching an LLM-as-a-Judge to think before delivering a verdict/score. The authors investigate and compare various pairwise and pointwise strategies, and conduct meaningful analysis and ablations over these settings", "questions": "See weaknesses. Also,\n1. [2] is recent work that trains a pointwise judge with RL, but they observe that response length explodes to over 10k tokens during training (similar to DeepSeek-R1-Zero [3]), whereas you observe that it initially falls and settles around 500 tokens. Why is this? Would performance be better if the models reasoned for longer before providing a score/verdict?\n2. In L.262, the authors state the results are statistically significant \"(p<0.0001)\". How was this statistical testing/resampling done?\n\n[1] https://huggingface.co/spaces/allenai/reward-bench\n\n[2] https://arxiv.org/pdf/2504.10337\n\n[3] https://arxiv.org/pdf/2501.12948", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear and easy to follow.\n2. The evaluation is comprehensive, spanning many relevant datasets and baselines. \n3. The approach, particularly the formulation in Section 3.2, is clear and well-motivated. Many meaningful ablations over the approach are provided.", "weaknesses": "1. There is some disconnect between Sections 2.3 and Section 4.1. For example, it's not clear which formulations from Section 2.3 were used to train the models evaluated in Section 4.1. Section 3 could be used to provide more information about the trained models. \n2. The role of the consistency reward is unclear. From Figure 5, it appears to slightly underperform the other pairwise settings, but a more thorough analysis (e.g., as a setting in Table 4) is needed.\n3. In real-world settings, automatic evaluations often span multiple dimensions (e.g., helpfulness, factuality, safety). How could the J1 framework be adapted for this multi-task setting?\n4. A couple of statements could use some qualification. For example, in L.180, the authors claim that \"pointwise judges are inherently consistent\", which is true in reference to positional consistency (because pointwise judges have no notion of position), but more broadly incorrect (e.g., sampling from the same pointwise judge is not guaranteed to produce consistent results). Similarly, in L.311, it states that J1-Qwen-32B-MultiTask outperforms \"all previous methods\" on RewardBench, but this is only true against other generative approaches; a number of classified approaches outscore J1-Qwen-32B-MultiTask [1].\n\n[1] https://huggingface.co/spaces/allenai/reward-bench", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)