Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
Authors: Ran Xu
Jingjing Chen
Jiayu Ye
Yu Wu
Jun Yan
Carl Yang
Hongkun Yu
Venue: ICLR 2026 Poster (2026)
Keywords: LLM judge
tool-integrated reasoning
Agentic RL
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation. However, most LLM judges operate solely on intrinsic text-based reasoning, limiting their ability to verify complex constraints or perform accurate computation. Motivated by the success of tool-integrated reasoning (TIR) in numerous tasks, we propose TIR-Judge, an end-to-end RL framework for training LLM judges that integrates a Python executor for precise evaluation. TIR-Judge is built on three principles: (i) diverse training across verifiable and non-verifiable domains, (ii) flexible judgment formats (pointwise, pairwise, listwise), and (iii) iterative RL that enables bootstrapping directly from a base model without distillation. On six public benchmarks, TIR-Judge surpasses strong reasoning-based judges by up to 6.4% (pointwise) and 7.7% (pairwise), and achieves listwise performance comparable to Claude-Opus-4 despite having only 8B parameters. Remarkably, TIR-Judge-Zero—trained entirely without distillation—matches the performance of the distilled variants, showing that tool-augmented judges can self-improve through iterative reinforcement learning.

Published as a conference paper at ICLR 2026
INCENTIVIZING AGENTIC REASONING IN LLM JUDGES VIA TOOL-INTEGRATED REINFORCEMENT LEARNING
Ran Xu Jingjing Jiayu Ye?, Yu Wu², Jun Yan?, Carl Yang', Hongkun Yu²
Emory University, "Google, *Google Cloud Al Research
ABSTRACT
Large Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation. However, most LLM judges operate solely intrinsic text-based reasoning, limiting their ability ver- ify complex constraints or perform accurate computation. Motivated by the success of tool-integrated reasoning (TIR) in numerous tasks, we propose TIR-Judge, an end-to-end RL framework for training LLM judges that integrates a Python executor for precise evaluation. TIR-Judge is built on three principles: (i) diverse training across verifiable and non-verifian verifiiiie domains, (ii) flexible judgment formats (point- wise, pairwise. listwise), and (iii) iterative RL that enables bootstrapping directly from base model without distillation. On seven public benchmarks, TIR-Judge surpasses strong reasoning-basee baannd judges by up to 6.4% (pointwise) and 7.7% (pair- wise), and achieves listwise performance comparable to Claudc-Opus despite having only 8B parameters. Remarkably, TIR- LER-JJdgee-zen traooerrained entirely without distillation matches the performance of the distilled variants, showing that cool-augmented judges can self-improve through reinforcement learning alone.
INTRODUCTION
Large Language Model (LLM)-based judges emerging critical component in the LLM ecosystem, typically used with scoring and ranking model outputs. This evaluation capability is essential at multiple stages of LLM development: during post-training, judges provide preference signals for alignment (Chen et al., 2025a; Whitehouse et al., 2025); at inference time, judges verify and select responses through best-of-N decoding (Huang et al., 2025a); and during evaluation, judges deliver reliable assessments without manual human assessment (Li et al., 2024a). Thus, training accurate LLM-based judges is of great importance for building powerful language models.
Classical evaluation with reward models often outputs scores directly, which can- not fully harvest the inherent reasoning capability of L.LM. Recent progress in generative reward modeling (Zhang et al., 2025; Zhao al., 2025) and reinforce- ment learning equips judges with thinking before producing final predictions (Chen al., 2025b; Whitchouse et al., 2025; Guo et al., 2025b; Hong et al., 2025). While these approaches enhance judge quality by equipping LLMs with long chains of textual reasoning traces, they remain inherently limited in scenarios that require precise computation or symbolic reasoning capabilities that are much more challenging for text-only models (Mirzadeh et al., 2025).
Write worrtee abscul the besauty of eucoeyytiue trees and their many usee.
Your coo-eeegrttd LUM))adge
Frontot
ayython resgonoo e**ghere desert best. whaapeza -utfe prrntt (hen Orerponse, tp (2)
Where desert heat lapers and mountaing ieii rism tell. The oucalyptus ds. majcctic call.
reeponce conteins recuirements.
The response only conteins 321 words. which cloes not mest the recainementa ita.
Texs LUV Judg
Figure 1: An example of LLM judge augmented with code execution, enabling precise judgments.
Recent advances in LLM tool-use provide a promising avenue to overcome the limitations of text- only judges (Chen et al., 2023; Gao et al., 2023). By granting access to executable interfaces for enumeration, verification, and computation, tools enable exact validation of reasoning steps rather than relying on potentially error-prone text-based inference. For example, code execution
Work done during intersship Google. Corresponding Email: ran. xudemory edu..

---

Published as a conference paper at ICLR 2026
automatically verify outputs on certain instructions (Zhou et al., 2023) (as shown in Figure 1) or check intermediate calculations in math reasoning (Lu et al., 2025). Early attempts have also explored equipping LLM judges with tool-use abilities (Peng et al., 2025; Findeis et al., 2025; Li et al., 2024b; Agarwal et al., 2025), but these approaches reveal two major limitations. (i) Inference-time restriction: most methods integrate tool-use only at the inference stage, preventing deeper integration between reasoning processes and tool execution. (ii) Narrow task coverage: many are tailored to specific domains or specialized task types, which limits their applicability in general-poue ceneral-purpose judging scenarios. These gaps highlight the need for robust judges that tightly couple reasoning with tool execution and be optimized end-to-end.
Motivated by these challenges, our goal is to develop an LLM judge that can reliably integrate reasoning with code interpreter execution. Incorporating tool-integrated reasoning (TIR) (Feng et al., 2025; Li et al., 2025a; Lin & Xu, 2025), we propose TIR-Judge, a framework that leverages reinforcement learning (RL) to teach models to generate code, execute it with interpreters, and iteratively refine their reasoning based on the resulting outputs. By reinforcing this cycle of reasoning and tool-use, TIR- TIR-Judge equips LLM judges with the ability at the training time to deliver more accurate and verifiable evaluations across diverse tasks.
Then, to fully unleash the potential of RL for TIR-Judge, we introduce several key design choices. (i) Task diversity: To balance between different tasks, construct training prompts spanning both verifiable domains (e.g.. competitive programming, mathematical reasoning) and non-verifiable domains (e-g., dialogue, safety, general coding), allowing the model to learn when tool invocation is beneficial and when pure reasoning suffices. (ii) Judgment flexibiliyy: To accommodate to different input/output formats, we diversify the evaluation tasks to cover pointwise, pairwise, and listwise ranking, ensuring broad applicability across practical use cases. (iii) Data efficiency: unlike prior methods that rely on distillation as cold-start for RL (Chen et al., 2025b; Hong et al., 2025), we demonstrate that T TIR-Judge can bootstrap from the initial checkpoint. Specifically, TIR-Judge- Zero trains purely with iterative reinforcement learning for achieving self-improvemment, while T TRR-Judge-Distill provides an optional variant using a small amount of distillation data.
Our contribution can be summarized as follows:
We introduce TIR-Judge, tool-integrated framework for training LLM-basee judges with end-to-end multi-turn reinforcement learning. To the best of our knowledge, this is the first approach that jointly optimizes reasoning and tool-use for training LLM-based judges via RL.
We design several key strategies to fully exploit the power of reinforcement learning, including task diversification across verifiable and non- vor-verifiable domains, flexible judgment formats (pointwise, pairwise, listwise), as well as an iterative RI. scheme that enables self-improvement in tool use even without distillation.
We evaluate Judge on seven public benchmarks covering diverse tasks and input formats. TIR-Judge consistently outperforms strong reasoning- based judges, achieving gains of up to 6.4% (pointwise) and 7.7% (pairwise). Moreover, TIR- Judge shows strong parameter efficiency: With only 8B parameters, it surpasses the 32B reasoning reward models the PPE dataset, and reaches 96% of the performance of Claude-Opus-4 in the listwise setting in RewardBench 2. Interestingly, TIR-Judge -erog the judge trained without any distillation, achieves a 1.2% gain over its distilled counterpart at 4B scale, highlighting the power of RL to bootstrap reasoning and tool-use capabilities.
TIR-Judge
2 RELATED WORKS
Reasoning- Ceasoning-Enhancee Reward and Judge Models. A growing line of work strengthens reward models (RMs) and judges by explicitly training them to reason before scoring. Generative Veri- fiers (Zhang et al., 2025) treat verification as next-token prediction with chain-of-thought improving math and algorithmic tasks. Other methods enhance judgment by generating critiques (Ankner et al., 2024; Yu et al., 2025c;b; Wang et al., 2025a), using multi-round preference optimization (Wang et al., 2024b), or planning evaluations before issuing a decision (Saha et al., 2025). Liu et al. (2025d) study how to allocate compute and structure critiques for reliability. More recently. RL-based judges (Chen et al., 2025a;b; Whitehouse et al., 2025; Guo et al., 2025b; Hong et al., 2025) incentivize longer, higher-quality reasoning and reduce bias across pointwise and pairwise settings, Khalifa al. (2025);
2

---

Published as a conference paper at ICLR 2026
Zhao et al. (2025) leverage thinking traces to improve process reward models. However, these remain confined to text-only reasoning and often emphasize pairwise judgment.
Tosl-Assisted Reward and Judge Models. Another line of work augments judges with external tools. Li et al. (2024b) incorporate verifiable signals alongside preference data for judge training, though primarily within tool-use scenarios. Zhuge et al. (2025) evaluate agentic judge capabilities in agent settings, and Peng et al. (2025) integrate human preferences with correctness checks to construct more reliable rewards. Findeis et al. (2025) show that external tools (e.g.. code execution, search) can improve annotations, observing gains are task-dependent. Yet, most works rely on prompled tool use rather than training judges learn when and how call tools and incorporate their outputs.
Reinforcement Learning for Tool- nollinteerated Reasoning, RL has recently been applied to tool- integrated reasoning (TIR). Feng et al. (2025); Bai et al. (2025); Li et al. (2025a) train LLMs to interleave reasoning with code execution, improving math and programming. Jin et al. (2025); Song al. (2025) extend this to web search, while others study reward design (Dong et al., 2025; Wang et al., 2025b) provide theoretical analyses (Lin & Xu, 2025).
PRELIMINARIES
Problem Setup. We consider the task of I.-M-based judgment: given user prompt X' and mederlgennrated responses y = {3/ . }. the goal is to evaluate the quality of responses for the prompt. The judge model Jo produces an evaluation output conditioned on J)). In this work, we consider three evaluation settings: (i) Pointwise evaluation: given (Ir, y). the judge assigns a
scalar score, Jo( y) = (0 y) R; (ii) Pairwise evaluation: given (I. ya. Bb), the judge selects the preferred response, Jn Na, yb) = arg [a. 8p (1,, (R,);,, where 8a denotes a learned scoring function. This is also the most common evaluation setting; (iii) Listwise evaluation: given (x, )) with >> 2, the judge returns the index of the best response, (z, )) arg max (l......) so (z, V:). These settings unify broad range of evaluations under a common framework!.
TugleAugeeteed Judge. We extend the judge with the ability to call an external Python execution environment I. For the prompt E X. At step k, the judgment trajectory sg is represented as s% = {ri, C1, 01...., Try Ck2 ) where is a natural language reasoning step, is a generated code, and og = [(;)) the execution result of c; (Li al., 2025a). The iterative process defined as:
(rk, cx) ~ J(x ① 8k-1), Og = I(ck), 8 1 1 ① Fkk ① Cg @ Ok-
(1)
This cycle continues until the judge produces a final prediction a; ~ @(r with being the final step. Unlike traditional text-only reasoning, the trajectory now interleaves reasoning, code execution, and tool feedback, enabling the judge to ground its decision in verifiable evidence.
4 TRAINING TIR-JUDGE
We now describe the training procedure for TIR- which consists of four components: (1) data collection and filtering for RL, (2) the RL framework for training judges with integrated code execution tools, (3) reward design for RL, and (4) cold-start and iterative training strategies in RL. The overall framework of T TIR-Judge is exhibited in Figure 2.
4.1 DATA COLLECTION AND FILTERING
High-quality training data is crucial for RL with tool-augmented judges. Since judgment requires both prompts and candidate responses, we curate a collection of (prompt, responses) tuples spanning multiple tasks. Our corpus integrates both human-ane annotatde preference data and automatically generated synthetic pairs to ensure diversity and scalability.
We sample human-labeled preference pairs from a variety of domains: general helpfulness - HelpSteer 3 (Wang et al., 2025c); reasoning - Ultralnteract (Yuan et al., 2025), SI (Muennighoff al., 2025); coding CodeRM (Ma al., 2025b); instruction following
Real Preference Pairs.
Note that in our work, the reference answer is unseen during evaluation, different from the verification setting (Liu al., 2025a; Li et al., 2025h; Yan al., 2025) where the reference answer also a part of the input.
3

---

Published as a conference paper at ICLR 2026
Tool-Integrated Reasoning
Judge Evaluation
Training Strategies for RL
Poem form: Y is writien verse. hhighiigh eucelyptus Length requirement: Need python code evalcate
Pointwise
Gemin
Rejectinn Samplingg
SFTT Dasa Toog
JiR-Judge-Disttlll
Tram ng Cata
python
TiR-Judge-Zero
zesponse setteee desect
CacacendDC/
opere
(exaposan,
Patnwise
Listwise
TA
SF1 Dusa
Rejection Sempling
Drecstion outout
choees
rejected
reacted
Injectec
321
SFT
RL
Fime answer
SFT
Cutput formal
From the ress the reeponse only contains 321 words, K
Code csscurion X
Final Verdict:
does follow the user prompt completee plately.
variants. TIR-Judge natively supports tool use during judgment and is designed handle diverse input formats.
Figure 2: Overall framework of
(IF) preference pairs from Tulu (Lambert et al., 2024); safety - Safe-RLHI (Dai et al., 2024). Each prompt is paired with one preferred (chosen) response and one or more rejected responses.
Synthetic Preference Pairs. Because reasoning preference data is often limited in scale, we augment the corpus with synthetic preference pairs generated from verifiable prompts. For each prompt, we sample responses from multiple open-source models, including Qwen3-8B/14B (Team, 2025), Gemma-2-99 (Team et al., 2024), and Gemma-3- (Team al., 2025). The responses are automatically evaluated against verifiable functions (for IF tasks) or ground-truth solutions (for reasoning tasks) form preference pairs. For IF, we use verifiable prompts from Tulu-3 (Lambert al., 2024), where correctness can be programmatically verified using lexical or structural constraints. For reasoning, we employ MATH (Hendrycks et al., 2021) and DAPO-Math (Yu et al., 2025a) for math domain and WebInstruct (Ma et al., 2025a), and Loong (Huang et al., 2025b) for general domain, both of which provide ground ground-truth solutions for exact verification.
In total, our dataset comprises approximately 26k preference pairs, including pointwise, pairwise, and listwise annotations, covering diverse domains such as helpfulness, reasoning, coding, safety, and verifiable instruction following. We apply strict 8-gram decontamination to eliminate any overlap between training prompts and evaluation benchmarks (Oren et al., 2024). This diverse mixture of data provides strong foundation for training robust tool-augmented judges.
4.2 TOOL-INTEGRATED RL WITH VERIFIABLE REWARDS
Overall Framework. We adopt DAPO (Yu et al., 2025a), an improved variant of GRPO (Shao et al., 2024), for training the LLM judge J parameterized by Given a prompt-answer pair (g, a), we first sample group of G' rollouts { from the current policy Tew. Each rollout s, is assigned scalar reward R; = R(s;, a) with access the oracle answer a. The policy mo then updated with the following clipped policy gradient objective:
lad
(0) (x ~o (:1))
min((s..(0)A.))
D
|81|
clip(rs.(0), Etow, 1+ Ar,.) (DL
s.t. 0 < [s: is is__i________ eeu.iveeeeettia s:))] < G
where (0)) = Fe so(s
Fu (*,,) is the token-level weight, A., R, Re-meanf (R.)
is the advantage at the token level, and i i s__qui va walee step filters out the prompts with accuracy equal to 1 and The hyperparameters Etow and Ehigh control the clipping range for importance weights, while regulates the KL divergence penalty to stabilize training.
std [_,,
Additional Designs. Beyond standard RL training, we implement two enhancements to stabilize tool-augmented judgment: (i) Error Message Processing. We truncate the outputs from Interpreter to only the final error line to avoid excessive context length while preserving useful feedback in sgs (ii) Sandbox Output Masking. Since execution results o I(c;) may cause the model to overfit by memorizing outputs, we mask o; during loss computation, following Li et al. (2025a); Jin et al. (2025). This prevents reliance exact strings and improves training stability.
4

---

Published as a conference paper at ICLR 2026
Reward Designs. To effectively facilitate multi-turn RL with code execution, design structured covering three aspects, described as follows:
(i) Correctness Reward Re: This component measures whether the judge's prediction aligns with the reference preference label. Let æ denote the prompt. y f Vn the candidate responses, and 7 the ground-truth preferred response. The reward is defined as:
(1s6 (2,, ypos) > so Mneg)), for pointwise evaluation, I [(Jo(x))))
Re
for pairwise or listwise evaluation, otherwise,
0.
where I((-) is the indicator function, sæ(r, y) denotes the judge's scoring function, and (x, ) is the predicted best response under the judge's policy. Intuitively, R. if the judge's 's decision matches the ground-truth preference, and R. otherwise (i.e. incorrect predictions, or having errors when parsing the generated text).
(ii) Format Reward Ry: To ensure reliability, the judge is required to strictly follow a predefined structured output format. Specifically, prediction scores must be enclosed within <score> and </score> tags, the preference label must be wrapped in <pref ference> and < iipreference. tags, and all code segments must be enclosed using "python and In addition, to accommo- date both reasoning and non-reasoning tasks and discourage unnecessary tool calls, we introduce heuristic: for safety and general helpfulness prompts, positive format reward is granted only if the model produces valid output without invoking tools. Formally, R 1 if the output satisfies all formatting constraints (and the no-tool heuristic when applicable), and R = 0 otherwise.
(iii) Tool-Specific Reward Rş: We encourage accurate and efficient tool use by penalizing errors excessive executions (Wang et al., 2025b). We set the max number of tool calls per trajectory to 3, and set R2 1 only when code blocks
error-free and within the call budget; otherwise R, 0.
The final reward defined as a combination of correctness, format, and tool-specific rewards and assigns full credit only when correctness, format, and tool-use are all satisfied:
1, if R A Ry = 1 ^ R. = 1,
R =
0.1,
if R2 but (Re = 0 v R = 0):
(3)
0,
if R. 0.
4.3 TRAINING STRATEGIES FOR RL
Directly applying RL often leads to suboptimal outcomes, as the base model lacks sufficient reasoning and tool-use capability. To address this, we design two cold-start strategies for training TIR-Judge.
Distillation from Teacher Models UTIR-Judge-Disilll)
We leverage a stronger teacher, Gemini-2.. 5-Flash with code execution (Comanici al., 2025), to generate high-quality trajec- tories via rejection sampling. For each user prompt and corresponding V we collect trajectory s and final prediction a as (x. V, s, a) J. Only trajectories that produce correct answers are retained, yielding dataset TSFT = [ar, 8, R(s, a, = 1}. Then the student judge is trained via supervised fine-tuning (SFT) with objective
b log fo(Ts TTi,))
L = - -((r.))
where = (s, is the target trajectory with reasoning and code steps. As in RL training, interpreter feedback tokens are masked to prevent learning on execution results. In total, we collect about 10k tool-integrated trajectories for SFT, which serve the initialization before reinforcement learning.
Iterative Training without Distillation (TIR-Judge-Zero). Beyond teacher distillation, we investigate whether cool-augmented judges can improve purely through weff-bootstrapping (Yuan et al., 2024; Huang et al., 2023; Zelikman et al., 2022; Xiong et al., 2025). The process alternates between RI., rejection sampling, and supervised fine-tuning.
5

---

Published as a conference paper at ICLR 2026
Starting from the initial model Tog first obtain the checkpoint тØ, via direct RL on training data as Te, +- (Ta,) (Sec. 4.2). Then, for each prompt æ, we sample multiple trajectories from Te, as - To, =) (G 4 in study), where each trajectory contains reasoning, code, and execution results: 8, {ri, (1,,01,00 01,..0,,,.. rk, Ca. 0k)- - We retain only valid trajectories that (i) produce the correct answer 1, (ii) satisfy the output format, and (iii) execute without interpreter errors as T, = [(,, s, s, R(s,a, a,() I]. To promote efficiency, for each prompt we further keep only one trajectory, preferring the shortest response or the one with the fewest tool calls². The dataset Te is then used for SFT, and the fine-tuned model initializes the next RL round. After each cycle, we select the best checkpoint based on held-out validation accuracy and repeat the RS - SFT + RL loop:
T++1 f RS,,))), T8+11 f ( T Th+41 + -RL(Te..))
This iterative process a stable refinement loop of better examples + better rollouts - even better examples. It allows TIR- PTR--Jdde--Da -Zero progressively bootstrap stronger reasoning and tool-use capabilities entirely from a base model and facilitates self-improvement without distillation. Since judgment rewards are deterministic and unambiguous (correct vs. incorrect), this iterative process converges reliably in practice.
EXPERIMENTS
5.1 EXPERIMENT SETUPS
Following prior work (Whitehouse al., 2025; Chen et al., 2025b), we focus on reasoning lasks, evaluating TIR-Judge on PPE Correctness (Frick et al., 2025). We additionally consider two more challenging datasets on judges: IFBench (Peng et al., 2025) for instruction- instruction-following and CodeJudgeBench (Jiang et al., 2025b) for code generation. All evaluations are conducted under both pointwise and pairwise settings to demonstrate the broader applicability of TIR-Judge. We also evaluate on general-domain judge benchmarks, where reasoning constitutes subset, including RewardBench (Lambert et al., 2025), R.M-Bench (Liu et al., 2025c) and JudgeBench (Tan et al., 2025) for pointwise/pairwise evaluation, and RewardBench 2 (Malik et al., 2025) for listwise evaluation.
Evaluation Datasets.
Implementation Details. We Qwen BB and Tneen
(Team, 2025)
as backbones, without enabling thinking mode, and implement training with Verl-Tool (Jiang et al., 2025a). For SFT, we train with batch size 64, learning rate 2c-6, context length 8192, for epoch. For RL, we set the micro batchsize per gpu to 4, mini batchsize to 128 and number of rollout to 8. We set Euo = 0.2, Fhigh 0.3, 8 0.01, max response length to 8192, learning rate 1e-6 and train for 2 epochs. The experiments are run with 8 NVIDIA H100 80G GPUs. For data collection in Sec. 4.1, we generate rollouts for each model with / 0.9, p = 0.95. No external feedback (e.g., GPT annotations) is used. For inference, we set / = 0 for generating responses.
Baselines. We consider the following group of baselines: (i) Off-the-shelf LLM as judges: GPT- 4o (Hurst et al., 2024), GPT-ol-mini (Jacch et al., 2024), Deepseek-RI (Guo et al., 2025a), Claude 3.5 (Anthropic, 2025), Seemini-2.5-Flase (Comanici et al., 2025), Qwen-3 (Team, 2025); (ii) Standard Reward Models: Armo-RM (Wang et al., 2024a), Seywork-Reward-Geemiie
(Liu et al., 2024),
Deepseck-BTRe -TRM (Liu et al., 2025d); (iii) Text-based Judges trained with RL: Deepseek-GRM (Liu et al., 2025d), J1 (Whitehouse al., 2025), RM-RI (Chen et al., 2025b), RRM (Guo et al., 2025b) and Think-RM (Hong et al., 2025); (iv) Tool-augmented Judges (Inference-time): Gemini- 2.5-Flash-Tooo (Comanici al., 2025), AgentRM (Peng al., 2025)², and Qwen-3 (Team, 2025) (our backbone) that use the same code execution tool as TIR- but only inject tools at the inference time.
'In practice, prioritize trajectories with the fewest tool calls, since encouraging efficient tool usage is the primary objective. If multiple trajectories tie under this criterion, we then choose the with the shortest trajectory length to further promote concise and efficient reasoning. For fairness, we Qwen-3 as the backbone for AgentRM. Note that AgentRM additionally leverages Armo-RM assist judgment.

---

Published as a conference paper at ICLR 2026
Table 1: Main results six benchmarks. indicates results reported from the original papers, and are mainly used for reference. CJBench, RWBench, and JGBench denote CodeJudgeBench, RewardBench, and JudgeBench. "Distill?" specifies whether the model relies additional judge data distilled from teacher models. Bold highlights the overall best accuracy, while blue and red mark the best results within our direct comparisons for pointwise and pairwise settings, respectively.
Baselinss
TTriid
Diail!
IFBench
ClBench
RWBanch
RMBach
JGBlach
MILLE-P
MATH
GPQA
MBRF-
IFEval
Aug.
LL.M-a5-- Judge (Pairoise Evaluation unless specifiie)
(Pontwvn))
57.4
55.9
69.3
64.9
Gemini-2.5 2.5 Flah (Puitwia))
36.5
79.5
45.4
63.0
63.0
61.9
51.5
$3.3
50.7
70.3
66.9
GPT-4o
57.6
61.3
56.7
725
56.5
GPi-ol-min!
71.3
70.1
87.1
05.
PerpSeek-R1--711
76.5
68.0
90.6
75.1
Chadle 3.5
81.0
96.0
63.0
54.0
16.0
68.4
58.3
68.3
60.
64.3
Dwen--4f--beemm (Painsie))
81.1
67.3
75.3
63.9
Queed
01.3
(Purkise)
75.0
74.8
$1.9
Scalar Noward Models (Pointuioe)
Aras RM-6B
1000L
66.0
71.0
57.0
$4.0
58.0
61.2
62.9
90.3
62.7
Skywodl-Gcamaa 2cmmma--278 2-27B?
80L
55.0
46.2
44.7
99.1
58.3
54.7
63.2
93.81
67.3
Despocek BTRM-222
257k
68.8
73.2
55.5
68.8
66.0
66.7
$1.7
bast- Reasoning Judggs (Patrwise Evaluation untess speciited)
Deaperk-GGMM--1
3571
59.8
59.8
11-88 (Phirwisc)"
7344
31-48 (Pristwiac)
22%
58.5
HMMM-7
420k
100.0
57.9
61.3
53.6
70.4
67.0
IUSS KL -Deepppo Dieall-11
73k
67.3
91.3
62.6
55.0
66.9
56.5
61.3
16.1
72.4
67.7
R33 R. Bastactt 7B
73k.
64.1
74.5
60.7
$7.3
57.8
62.9
59.0
57.5
65.2
70.2
60.3
Think RM
63.9
64.5
57.4
73.9
64.5
augieented Judges
Owen3-68-TTwh (hriewied)
64.8
R1A6
16.9
61.0
49.8
59.1
64.1
15.0
78-4
72.1
15.6
Qwen3-88- (Printiee)
34.0
27.1
67.9
53.0
TIR-2 Distil (Ponnwise)
25k
45.5
78.9
65.50
71.9
2add-22dgee-eee Zazzge--ooo 48 (Pantxissc)
28k
62.5
87.3
54.7
64.3
79.8
69.8
65.0
61.5
77.3
72.8
70.4
TER Sad-------iiii Disill KB (Poinwac))
258
70.9
68.1
$2.3
61.0
$3.0
71.0
69.4
61.9
51.0
76.7
68.2
I1R-2adge-Jas (Poistiisc)
$3.2
77.8
70.3
66.8
76.3
67.5
AscuRM 3B (Puirwise)
75.0
65.6
67.0
09.7
59.4
Qwen3-4B-Be Soun--4---Taai TBeT- (Phirwiec)
63.5
83.3
35.0
58.9
62.5
60.8
99.3
24.3
85.2
75.7
63.0
Qwen3-KB-Tiee (Peewaac)
72.0
83.3
$6.0
$4.3
60.
65.7
$3.5
$4.9
66.2
72.3
65.0
Genini 2.5 Fish Tool (Pairoise)
75.1
87.5
60.3
N5.1
$4.0
78.0
08.5
06.3
80.9
74.6
118-2adge
54.
70.5
1edgeee222 Zero (Paresse)
25k
95.3
86.5
76.5
70.3
50.8
IIB 2ad8-22dge-Din Dhaii-iiilll 4il 6B (Painiie))
25k
72.2
90.4
53.8
63.2
85.7
73.0
74.3
70.0
57.9
82.2
22.6
TER 2all--addeezz (Parw
25k
76.6
94.0
55.5
68.8
$0.8
15.2
68.9
69.3
59.1
83.7
72.0
Reference: Text fest-bosed Rensoning Judge Bonclines >-100
Purumeters (Pairwise Exalnation)
67%
72.
10.0
RRM 1OR
420k
80.5
94.1
64.4
72.3
60.
15.5
60.8
76.3
85.4
16.0
RM KL Deepseek Disiil 149
73k
78.8
94.5
63.3
70.5
65.0
74.0
58.5
65.5
59.9
$1.5
76,2
RM KL Deepseek Disall 328
73k
79.8
95.4
65.2
74.6
63.3
75.6
00.4
63.5
90.9
539
78.4
5.2 MAIN EXPERIMENT RESULTS
Experiments for Pointwise/Pirrwse Judging tasks. Table shows the main results of TIR-Judge on six judge benchmarks. The per-task accuracy on several benchmark deferred to Table 5. From the results, we have the following key observations: (i) TIR-Judge achieves strong judging accuracy compared to baselines. Notably, on the PPE benchmark, TIR- PIR-Judge outperforms baselines with similar sizes by 4.8%-9.9% for pointwise judging and 4.5%-8.88' 8.88% 8% for pairwise judging. also achieves competitive or even better performance on other benchmarks with baselines having more parameters and trained with more data. For example, TuR-Judge achieves similar accuracy on PPE and RewardBench compared to RRM-32B despite having only 1/4-1/8 of its parameters. (ii) RL is critical for boosting tool-use capability for judges: Simply augmenting Qwen-3 models with code execution yields negligible (<1%) or even negative gains. In contrast, RL produces substantial improvements, showing that base checkpoints lack robust code generation ability and that RL is essential for unlocking tool-use capability. Moreover, RL confers strong generalization: although most IF data is verifiable, Tudge also performs well on IFBench, which contains many non-verifiall constraints. (iii) Iterative RI is surprisingly effective to serve as another alternative distillation Comparing TI TTR-Judge-Zen with TTR-Judge-DDsse
we find that T TIR- IR-JJudge-Zero delivers comparable or better performance, outperforming the distilled variant on 4/6 benchmarks (pointwise) and 3/6 benchmarks (pairwise). This demonstrates that TIR-Judge-Zent Zero offers viable alternative for data-scarce regimes, achieving competitive performance to distillation- based methods, albeit with additional overheads in training time.
Experiments on Listwise Judging tasks. We further evaluate TIR-Judge on RewardBench2 (Ma- lik et al., 2025) under listwise judge setting, where the input contains one chosen and multiple rejected responses. As shown in Table 2, TIR-Judge achieves strong performance, matching 96% perfor-
7

---

Published as a conference paper at ICLR 2026
Table 2: Results on tasks in RewardBench2, sorted by average performance.
Datasets
Math
Fact
Focus
Safety
Avg.
Claude- Opuu--
41.9
74.9
82.7
86.2
89.5
76.5
1omini-2.55-lashh ftash-Preview
55.3
81.1
65.7
86.7
90.9
75.9
TIR-Judge Zero 8B
45.6
84.1
64.8
89.5
82.7
73,4
IIR-Judge-Distle 8B
58.1
72.7
63.8
81.4
82.0
71.6
GPT4411
39.7
65.2
82.9
73.4
87.3
69.7
Claude-Somet-44
35.9
70.5
76.1
76.0
89.1
69.5
TIR-Judge-Zenu 4B
47.5
86.4
59.3
85.2
62.9
68.3
TIR-Judge-Distill 4B
55.0
78.1
55.8
75.0
73.1
67.3
GPI-4.1-mini
41.2
72.1
60.8
73.5
72.6
65.7
GPT-4o
33.1
62.3
56.8
72.9
86.2
64.9
Claude-3.5-sonme
38.8
56.8
52.8
87.0
85.2
64.7
I1K-Judge-Zaro 8B
w/ only
only Coding
ooly Ressoning
only Helpfislness Sadery
Perforaanee
50
25
PPE-MBPP-PEE
PPE-MMLU.PPo
PPE-GPQA
RMBench-Safety
Figure 3: The effect of different data mixture used in RL training of TIR- TIR-Judge-Zerr.
mance of Claude-Opus-4 the current best model on the leaderboard, despite being 8B parameter only. The advantage more notable on tasks such as instruction following and mathematical reasoning, where TIR- MER--Jude'ss integration of code execution provides clear gain.
5.3 ADDITIONAL STUDIES
Diverse Data Mixture is essential for RL. We study the impact of task composition in RL in Figure 3. Training exclusively on chat or reasoning tasks leads to poor transfer across subtasks, largely because the scarcity of tool-use prompts prevents the model from fully developing tool-use capabilities. In contrast, unifying tasks both with and without tool use into single training pipeline leads to improved generalization.
Tool Use vs. Text-Only. To rigorously evaluate the impact of tool integration, we conduct a controlled study in which code execution is disabled during RI. while keeping the training data identical. As shown in Figure 4(a), tool-augmented models achieve consistently higher accuracy on reasoning and IF benchmarks, while text-only models perform slightly better text-centric tasks such as Chat and Safety RMBench. These comparisons highlight the strength of -uggmeeedd judges for reasoning, and further suggest that mixing prompts from both tool-use and non-tool-use settings maintains robust performance without sacrificing much on cases where tools are unnecessary.
Efficiency Studies. We further evaluate the efficiency of TIR-Judge against several baselines in Figure 6. While TIR-Judge achieves higher accuracy, incorporat- ing external code execution tools introduces no additional inference-time overhead. In fact, FIR-Judge is more efficient than the baselines, benefiting from our SFT data construction strategy that favors trajectories with shorter reasoning and fewer tool calls during rejection sampling.
3MMRI-distill 32B
JiR-Jodge/eio
Avg. PPE Performance
75
3RM-328
Judo
TIR-Judge Ditiil 8B
70-
Ti-- 4B
IM-RI-dissilT
65
Think-RM
RRM-7B
Qwes-1-8B
Owen
Iterative RL progressively improves TIR-Judge- Zero. We evaluate T TIR- TIR-Judge-Zero across training stages under the pairwise setting. As shown in Figure 5, we observe substantial gains after the first round of RL. These improvements arise from rejection sampling, which teaches the model to produce more format-correct and efficient tool use, thereby strengthening its reasoning capability. Additional RL iterations further boost accuracy as RL benefits from progressively higher-quality SFT data. In contrast, rejection-sampling fine-tuning yields modest gains, highlighting the necessity of online RI.
10
15
Avg. # Prompls per Second
Figure 6: Study on Inference Efficiency.
8

---

Published as a conference paper at ICLR 2026
Dixet RL ifle) WTeel
Direei RL. (fg wel Tocl
Bert N.NNNA Acerscy
(Prividd Zano Best
TiR.lale Zen
NVgie
13
30
12
Rulllond Subr
BigCod:Bench
(a) Tool Use v.s. Text-only Judges
(b) Best-of-N Inference
Figure 4: Experimental results comparing tool-augmented judges against text-only judges under the same training data and settings, as well the best-of-N inference performance.
Bose
1B--hugg-Dimill-SF
TRR--udge-Dist.llee TIR--ndge-Zzr-83
Judge
[IR-Judge Zero
IIR-Judge RL-2
IFBench
CoddiutgoRmeh
80
Performance
70-
50-
40
SB
SB
Figure 5: Accuracy of TIR-Judge across different training stages. Base denotes the backbone model without additional training. TIR- JLR-Judge-Zero-BR is variant inspired by Zelikman al. (2022) that uses rejection sampling to construct high-quality trajectories for SFT (without RL). TIR-Juddg-ZZeo-RLLU. refer the judge after 0, 1, and rounds of RL training, respectively.
5.4 BEST-OF-N EVALUATION ON POLICY MODELS
We conduct parallel test-time compute scaling experiment to study whether TIR-Judge can im- prove the downstream performance of the policy model, where we conduct a study on reward-guided best-of-N inference over datasets from multiple domains including AIME-2024, A/ME-2025, Big- CodeBench (Zhuo et al., 2025) and IFEval (Zhou et al., 2023). The detailed experimental setup is deferred to Appendix F.
Figure 4(b) presents the average accuracy of TIR-Judge compared to strong baseline, RRM, across four datasets. We find that TIR-Judge consistently surpasses both Majority Voting (Self- Consistency; Wang et al. (2023)) and RRM by clear margins, demonstrating its effectiveness.
The improvements are especially pronounced on challenging benchmarks: BigCodeBench, which involves complex code generation and diverse functions, and AIME, which consists of competition- level math problems. On these tasks, TIR-Judge achieves absolute gains of 3.9-6.7% over RRM. This justifies its ability to handle more challenging tasks in real-world applications.
5.5
CASE STUDIES
Table 3 presents an example from the IFEval subset of the PPE benchmark. TIR-Judge successfully generates correct Python functions verify two responses and produces the correct pairwise judgment. In contrast, text-only judges struggle, as counting remains challenging and often leads to incorrect and hallucinated reasoning steps, which yield incorrect predictions. This highlights how tool integration enables T TJR-Judge to overcome failure modes that remain difficult for text-only judges.
To confirm that the gains of TIR-Judge stem from improved reasoning and coding capability rather than merely "Iearning the format," we analyzed the error breakdown for the 8B models in Table 4. The results show that format errors the Qwen backbone are already negligible (< 1%). This confirms that TI TIR-Judge' 's improvement is driven better code generation (significantly lower syntax errors) and reasoning capabilities, rather than simply correcting formatting artifacts.
CONCLUSION
In this work, we introduce TIR-Judge, the first tote-integrated framework for training LLM judges with end-to-end reinforcement learning. Different from prior works text-only judges,

---

Published as a conference paper at ICLR 2026
Table 3: A case study from IFEval demonstrating the judgement generated by TIR- Jerge-Zero 8B and Qwen3-8B. Red text stands for the incorrect reasoning step while blue text justify the ability of TIR-Judge in writing correct verification codes. Part of responses are omitted due to space limits.
User Prempt
Wriee letter your Eriend who secesily moved anuy. Your respoise should be in English, and all capiral letters The lener o should appear least simes. BEEN so LONG SENCE MOVED AWAY, AND MMSS TERRIBLY ... ICN'T WAFT HEAR
Response
MY DEAREST FRIEND.I
CATCH UP WITH
Responee B
MY DEAR FR HAS BEEN SOME TIME SINCE YOU MOVED AWAY. I WISH YOU NOTHING BUT THE BEST ON YOUR NEW
ADVENTURE, FRIEND remenbeer MI UPIATE WITH ALL THE DETAILS ABOUT YOUR LIFE THEROOOEERAA IN THE PRIN TED PAGES. PLEASE
SEND Usat THOUGHTS YOUR WAY, AS KNOW YOURS AKE ALWAY'S NEAKLY ALWAYS WITH YOUR FRIEND FOR LIFE
Judge Trajectory Zid-Judgo-Zero RB
evalute the resposses, perioem following checls:
Lasure both responses
captal letter. appens soppoee
"python
Count occurrences ror fease-aensitivee count A.A reagonee A count (ro")
_ount_n__ reagonee_ 2. count (ror)
Check it zeaponzea Are in all capital Letteca
sappppa__a reasense_k pasporae_ss.uppe 15, upper.p responce resporse b. O
'0" print (eSpppon for "ouupuu:
AlT capes se.upper_b")
Zeaponze "O" count: 5d, ALl caps: Team
Besgonce 10" counti 60, All caps1
Palse
<prterroeeAApprttereeee
Qwend-88
Ansdysi Ropuma
Langaage:
English,
capical
Letier "O"
the number of leners Response (escluding punctiation and spaces
'MY DEAREST FRIEND. 1""0"
'IT HAS BEEN SD LONG SINCE YOU MOVED AWAY AND IMSSS YOU TERRIBLY." 22'
Total 0% in Requense A: 2+2+3++ 3 24 2 + Docs NOT mas the regurieeent of lose 40 '0'l latlens
Aualyso Ropuuse
LLnguage::
capical
Letter "O"
number of
Response (escluding pusctuation and spaces clarity'
"'M DEAR FRIEND" 2-2 1'OO's
"ET HAS BEEN SOME TIME SINCE YOU MOVED AWAY" -11 1""""
(ADMET IT) TERRIBLE WONDERFUL COOKING," 4+4 "I HAVE TO ADMMII, MISS OUR THOL CHIFUL CONVERSATIONS & AND EVEN (AUMEA 'O's
Todal Response Il:
4 7 30 Dacs NOT mest regrinemnnt at "O" letern
Conclusio:: Both responees
"O" companal Resporoe instruction's 's requiemenn that the lener musi uppoar at times, However, Respoase contaits more buth speeas fully complynng lentes detailed
Respurac
eloady alignual
hightr
Table 4: Breakdown of Code Execution Errors. We categorize errors into Syntax, Runtime, and Formatting failures.
Error Type
TIR-Judge-Zero
TIR-Judge
Qwen-3-Tool
Syntax
1.20 %
3.65%
4.95%
Runtime
0.00%
0.00%
0.00%
Format
0.17%
0. 0..4%
0.97%
TIR-Judge tightly couples reasoning with code execution to enable judges to perform precise verification and computation. To maximize the benefits of RI., we propose three key design choices: task diversification flexible judgement, and iterative RL training. Experiments show that (IR-Juddae outperforms strong reasoning judges by up to 6.4% (pointwise) and 7.7% (pairwise), and matches 96% of Slaude-Opus-44 listwise performance with only 8B parameters. TIR-Judge- shows that pure RL can bootstrap tool-use without supervision, offering a scalable path toward self-improving judges. In future work, aim expand the range of tools and training tasks used in RL, and explore using TIR- TIR-Judge to enhance policy model training.
seven benchmarks
ETHICS STATEMENT
Our work aims to advance the development of LLMed judges by complementing reasoning with tool use. While TIR-Judge demonstrates strong performance across multiple domains, recognize potential ethical and safety concerns. In particular, the use of external tools (e.g..
10

---

Published as a conference paper at ICLR 2026
code execution environments) raises risks if malicious unsafe code were executed, such as file manipulation, unauthorized access, or propagation of harmful outputs. To mitigate this, our experiments conducted in sandboxed environments with restricted permissions, and no real-worll files or systems are exposed.
REPRODUCIBILITY STATEMENT
We place strong emphasis on reproducibility and provide comprehensive implementation details. Hyperparameters and training configurations are reported in Section 5.1, while the construction of synthetic training data, prompt formats, and evaluation protocols are documented in Appendices D, B, and E, respectively. Each RL iteration requires approximately 8-12 hours on 8*NVIDIA H100 80GB GPUs, and the full training of TIR-Judge can be reproduced within 24 48 hours under the reported setup.
REFERENCES
Mayank Agarwal, Ibrahim Abdelaziz, Kinjal Basu, Merve Unuvar, Luis A Lastras, Yara Rizk, and Pavan Kapanipathi. Toolrm: Outcome reward models for tool-calling large language models. arXiv preprint arXiv:2509.11966 2025.
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. arXiv preprint arXiv:2408. 11791, 2024.
Anthropic. Introducing claude 3.5 sonnet. Technical report, 2025.
Fei Bai, Yingqian Min, Beichen Zhang, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang. and Ji-Rong Wen. Towards effective code-integrated reasoning. arXiv preprint arXiv:2505.24488. 2025.
Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. JudgeIrm: Large reasoning models judge. arXiv preprint arXiv:2504. 00050, 2025a.
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W. Cohen. Program thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023.
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang. Hongru Wang. Yu Zhang, Denghui Zhang. Tong Zhang, et al. Rm-rl: Reward modeling as reasoning. arXiv preprint 02.38 2025b.
Gheorghe Comanici. Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 2025.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024.
Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen. Tool-star: Empowering llm-brained multi-too reasoner via reinforcement learning. arXiv preprint arXiv:2505. 1otr225055//41 (4,00, 2025.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprini 2r0iv:2504. 1536, 2025.
Arduín Findeis, Floris Weers, Guoli Yin, Ke Ye, Ruoming Pang, and Tom Gunter. Can external validation tools improve annotation quality for ALM-as-a-judgee In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), July 2025.
11

---

Published as a conference paper at ICLR 2026
Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and lon Stoica. How to evaluate reward models for RLHF. In The Thirteenth International Conference on Learning Representations, 2025.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aides aided language models. In International Conference Machine Learning, pp. 10764-10799. PMLR, 2023.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma. Xiao Bi, et al. Deepseck-rl incentivizes reasoning in Ilms through reinforce- ment learning. Nature, 645(808) ):633)6363333 638, 2025a.
Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Reward reasoning model. arXiv preprint arXiv:2505. 2025b.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Ilgee Hong. Changlong Yu, Liang Qiu, Weixiang Yan, Zhenghao Xu, Haoming Jiang. Qingru Zhang, Qin Lu, Xin Liu, Chao Zhang, and Tuo Zhao. Think-rm: Enabling long-horizon reasoning in generative reward models. arXiv preprint ar arXiv:2505. 162.6565,,,,, 2025.
Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Akshay Krishnamurthy, and Dylan J Foster. Is best-of-n the best of them? coverage, scaling, and optimality in inference-tine alignment. In Forty-second International Conference on Machine Learning, 2025a.
Jiaxin Huang. Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 1051-1066 December 2023.
Xingyue Huang, Gregor Franke, Ziyi Yang, Jiamu Bai, Weijie Bai, Jinhe Bi, Zifeng Ding, Yiqun Duan, Chengyu Fan, Wendong Fan, et al. Loong: Synthesize long chain-of-thoughtt at scale through verifiers. arXiv preprint 2w0.h::5009030055 2025b.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21226, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameyy, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Dongfu Jiang. Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang. Alex Su, Hui Chen, Kai Zou, Chao Du, et al. Veritool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint arXiv:2509.01053 2025a.
Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, and Robby T Tan. Codejudgebench: Benchmarking llm-as-a-judge for coding tasks. arXiv preprint arXiv:2507. 10535, 2025b.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean O Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training LLMs to reason and leverage search engines with reinforcement learning. In Second Conference on Language Modeling, 2025.
Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moon- tae Lee, Honglak Lee, and Lu Wang. Process reward models that think. arXiv preprint arXiv:2504. 16828, 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin. Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint 2XXv:2411.15/22 2024.
12

---

Published as a conference paper at ICLR 2026
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick. Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. RewardBench: Evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: NAACI, 2025, pp. 1755-1797, 2025.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. In The Twelfth International Conference on Learning Representations, 2024a.
Lei Li, Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Ningyu Zhang, and Hua Wu. Tool-augmented reward modeling. In The Twelfth International Conference on Learning Representations, 2024b.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: arXiv:2503.2338) 2025a.
Scaling tool-integrated rl. arXiv preprint
Xuzhao Li, Xuchen Li, Shiyu Hu, Yongzhen Guo, and Wentao Zhang. Verifybench: A systematic benchmark for evaluating reasoning verifiers across domains. arXiv preprint arXiv:2507.08 009884, 2025b.
Heng Lin and Zhongwen Xu. arXiv:2508. 19201, 2025.
Understanding tool-integrated reasoning. arXiv preprint
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou Skywork-reward: Bag tricks for reward modeling in 1lms. arXiv preprint 24X1::2410.18451, 2024.
Shudong Liu, Hongwei Liu, Junnan Liu, Linchen Xiao, Songyang Gao, Chenggi Lyu, Yuzhe Gu, Wenwei Zhang, Derek F Wong, Songyang Zhang, et al. Compassverifier: A unified and robust verifier for ilms evaluation and outcome reward. arXiv preprint arXiv:2508.0366 2025a.
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Pairjudge rm: Perform best-of-n sampling with knockout tournament. arXiv preprint 2rX1v:2501.13007, 2025b.
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-bench: Benchmarking
reward models of language models with subtlety and style. In The Thirteenth International Conference on Learning Representations, 2025c.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.02495, 2025d.
Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathcoder2: Better math reasoning from continued pretraining on model-translated mathematical code. In The Thirteenth International Conference on Learning Representations. 2025. URL https: http:///wneeeeeee...eeee net forum?id-iiiii www.mm/dddwwwwwiirrraa..a.
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing IIm reasoning across all domains. arXiv preprint arXiv:2505. /4652, 2025a.
Zeyao Ma, Xiaokang Zhang. Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. Dynamic scaling of unit tests for code reward modeling. arXiv preprint 2501.2250..0054 2025b.
Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A Smith, Hannaneh Hajishirzi, and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation. arXiv preprint arXiv:2506.01937, 2025.
Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: http//peeeveeeeeeeeeeeeeeeeeeeee net/ forum? A
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint (rXXi:250111999) 2025.
13

---

Published as a conference paper at ICLR 2026
Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box box language models. In The Twelfth International Conference on Learning Representations, 2024.
Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou, and Juanzi Li. Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), July 2025.
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason E Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking-LLM-ann LLa--a---judee In Forty-second International Conference on Machine Learning, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint vrXiv:2402.03300, 2024.
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI-scarcher: Incentivizing the search capability in Ilms via reinforcement learning. arXiv preprint arXiv:2503. 2025.
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica. Judgebench: A benchmark for evaluating LLM- judges. In The Thirteenth International Conference on Learning Representations, 2025. URL https: http:///ppeeeeeeeeeeeeeeeeeeeeee net/ www.nneee///o/foeoe
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size, arXiv preprint vrXiv:2408.00) 18, 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, al. Gemma technical report. arXiv preprint arXiv::5033 19786, 2025.
Qwen Team. Qwen3 technical report, 2025.
Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Qiaozhi He, MuRun Yang. Bei Li, Tong Xiao, Chunliang Zhang, Tongran Liu, and JingBo Zhu. GRAM: A generative foundation reward model for reward generalization. In Forty-second International Conference on Machine Learning. 2025a.
Haoxiang Wang. Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and ofxture-of-exppets Findlngs of the Association for Computational Linguistics: EMNLP 2024, pp. 10582-10592, November 2024a.
Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Acting less reasoning more! teaching model to act efficiently. arXiv preprint arXiv:2504. 14870, 2025b.
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators. arXiv preprint wwX.:24488.02666 2024b.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Hoo- Hoo-Chang Shin, Felipe Soares, Alexander Bukharin. Ellic Evans, Yi Dong, and Oleksii Kuchaiev. Helpsteer3 crrssteer3-reeference: Open human-ame -nnotttdd preference data across diverse tasks and languages. arXiv preprint arXiv:2505.1 /475, 2025c.
Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha. J1: Incentivizing thinking in llm-as-a a-jude via reinforcement learning. arXiv preprint arXiv:2505. 10320, 2025.
14

---

Published as a conference paper at ICLR 2026
Zhaofeng Wu, Michihiro Yasunaga, Andrew Cohen, Yoon Kim, Asli Celikyilmaz, and Marjan Ghazvininejad. rewordbench: Benchmarking and improving the robustness of reward models with transformed inputs. arXiv preprint 0xxh:22503117511, 2025.
Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, Chi Jin, Tong Zhang, and Tianqi Liu. Building math agents with multi-turn iterative preference learning. In The Thirteenth International Conference on Learning Representations, 2025.
Yuchen Yan, Jin Jiang, Zhenbang Ren, Yijun Li, Xudong Cai, Yang Liu, Xin Xu, Mengdi Zhang, Jian Shao, Yongliang Shen, et al. Verifybench: Benchmarking reference-based reward systems for large language models. arXiv preprint wxxi:2255..1580 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503.
2025a.
Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, and Rui Hou. Self-generated critiques boost reward modeling for language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), April 2025b.
Zhuohao Yu, Jiali Zeng. Weizheng Gu, Yidong Wang, Jindong Wang, Fandong Meng. Jie Zhou, Yue Zhang, Shikun Zhang, and Wei Ye. Rewardanything: Generalizable principle-folowinng reward models. arXiv preprint arXiv:2506.03633 2025c.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Boji Shan, Zeyuan Liu, Jia Deng. Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees. The Thirleenth International Conference Learning Representations, 2025.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. relf-rewa language models. In Forty-first International Conference on Machine Learning, 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STar: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 2022.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal.
Generative verifiers: Reward modeling as next-token prediction. In The Thirteenth International Conference on Learning Representations, 2025.
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint arXiv:2504. 250.0009, 2025.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. instruction-follovii evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
Yilun Zhou, Austin Xu, Peifeng Wang. Caiming Xiong, and Shafiq Joty. Evaluating judges as evaluators: The jetts benchmark of Ilm-asjjudgss as test-time scaling evaluators. arXiv preprint arXiv:2504. 15253, 2025.
Mingchen Zhuge, Changsheng Zhao Dylan R Ashley, Wenyi Wang. Dmitrii Khizbullin, Yunyang Xiong. Zechun Liu, Ernie Chang. Raghuraman Krishnamoorthi, Yuandong Tian, et al. Agent- as-a-judge: Evaluate agents with agents. In Forty-second International Conference Machine Learning, 2025.
Terry Yue Zhuo Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, et al. Bigeodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference Learning Representations, 2025.
15

---

Published as a conference paper at ICLR 2026
A USAGE OF LLMS IN THE SUBMISSION
We leveraged LLMs solely as a writing aid for paper preparation. Its application was restricted editorial tasks such improving the coherence and flow of existing text, suggesting minor grammatical corrections, and enhancing overall readability. The authors are exclusively responsible for all aspects of the research design, experimental results, and the core intellectual content and conclusions presented in this work.
B PROMPT TEMPLATE
We provide the detailed prompt template used in TIR-Judge in the following.
You are an expert judge for evaluating whether model-generated response correctly answer the user's instruction. Please carefully read both the Instruction and the Response below. Your goal is to assess whether the response accurately and completely follows the instruction, focusing strictly on correctness and fidelity. Do not consider writing style, tone, or fluency unless they affect instruction compliance.
You may use Python code to assist your evaluation, by writing the code between the "'yython and tags following with "'output. Each program between "'python and tags indepen- dent program. This is especially helpful for verifying complex requirements, such as checking capitalization, counting words, letters, keywords, placeholders, checking the correctness of calculation steps for reasoning problems, as well as running test cases on the coding responses.
Instruction: (prompt)
Response: [response]
Guidelines: In the end of your response, please give in range to 10 in the format such as <score>| 1atrc==-0/////>>>> to indicate the quality of the response. A score of 10 indicates that the response fully adheres the instruction and provides correct answer. A score of I indicates that the response is irrelevant, evasive, or fails to address key aspects of the instruction.
Figure 7: Prompt for response generation with tools on pointwise evaluation.
You are an expert judge for evaluating whether model-generated response correctly answer the user's instruction. Please carefully read Instruction, Response A and Response B below. Your goal is to assess which response accurately and completely follows the instruction, focusing strictly on correctness and fidelity. Do not consider writing style, tone, or fluency unless they affect instruction compliance.
You may use Python code to assist your evaluation, by writing the code between the "'python and tags following with "'output. Each program between "'python and tags indepen- dent program. This is especially helpful for verifying complex requirements, such as checking capitalization, counting words, letters, keywords, placeholders, checking the correctness of calculation steps for reasoning problems, as well as running test cases on the coding responses.
Instruction: (prompt)
Response A: (response A}
Response B: (response B)
Guidelines: In the end of your response, please give a preference in the format such as <preffrence>-<eppeeeeeneeee indicate the better response.
Figure 8: Prompt for response generation with tools pairwise evaluation.
16

---

Published as a conference paper at ICLR 2026
You are an expert judge for evaluating whether a model-generated response correctly answer the user's instruction. Please carefully read Instruction and all responses below. Your goal is to assess which response accurately and completely follows the instruction, focusing strictly on correctness and fidelity. Do not consider writing style, tone, or fluency unless they affect instruction compliance.
You may use Python code to assist your evaluation, by writing the code between the ""yyhhnn and tags following with ""output. Each program between ""python and tags indepen- dent program. This is especially helpful for verifying complex requirements, such as checking capitalization, counting words, letters, keywords, placeholders, checking the correctness of calculation steps for reasoning problems, as well as running test cases on the coding responses.
Instruction: {prompt]
Response A: (response A}
Response B: (response B]
Response C: (response C)
Response D: [response D}
Guidelines: In the end of your response, please give preference in the format such <prefceenee><cprrereeneee indicate better response.
Figure 9: Prompt for response generation with tools on listwise evaluation.
You are an expert judge for evaluating whether a moder-generated response correctly answers the user's instruction while being harmless. Please carefully read both the Instruction and the Response below. Your goal is to assess whether the response accurately and completely follows the instruction, focusing strictly correctness and fidelity. Do not consider writing style, tone, or fluency unless they affect instruction compliance.
Instruction: (prompt)
Response: [response]
Please first generate textual reasoning step before making final predictions. In the end your response, please give a score in range to 10 in the format such as 1score>10</scor>> to indicate the quality of the response. A score of 10 indicates that the response fully adheres to the instruction and provides a correct answer. A score of indicates that the response is irrelevant, evasive, fails to address key aspects of the instruction.
Guidelines:
Figure 10: Prompt for response generation without tools on pointwise evaluation.
17

---

Published as a conference paper at ICLR 2026
You are an expert judge for evaluating whether a model-generated response correctly answers the user's instruction while being harmless. Please carefully read the instructions and all responses below. Your goal is to assess which response accurately and completely follows the instruction, focusing strictly on correctness and fidelity.
Instruction: (prompt)
Response A: (response A}
Response B: (response B}
Guidelines: Please first generate a textual reasoning step before making final predictions. In the end of your response, please give preference in the format such as <prefer- ence>A><prefereece>> indicate the better response.
Figure 11: Prompt for response generation without tools pairwise evaluation.
You are an expert judge for evaluating whether a model-generen response correctly answers the user's instruction while being harmless. Please carefully read the instructions and all responses below. Your goal is to assess which response accurately and completely follows the instruction, focusing strictly on correctness and fidelity.
Instruction: [prompt)
Response A: (response A}
Response B: (response B)
Response C: (response C)
Response D: [response D
Please first generate a textual reasoning step before making final predictions. In the end of your response, please give preference in the format such as <prefer- ence>AA</preference> to indicate the better response.
Guidelines:
Figure 12: Prompt for response generation without tools on listwise evaluation.
C FULL PERFORMANCE ON SEVERAL BENCHMARKS
Table shows the full results of TIR-Judge and key baselines on RewardBench, RMBench, and JudgeBench. Sometimes we observe that the performance of Geminn-22.5fllas declines when additional tools are introduced. This issue arises from a maximum-ture limit on tool calls: the model sometimes generates excessive tool invocations and, in certain cases, fails to terminate properly.
D DETAILS ON TRAINING DATA COMPOSITION
Our training mixture spans reasoning, code evaluation, and safety alignment tasks for reinforcement learning. Table 6 summarizes dataset statisties across three supervision formats: pointwise, pairwise, and listwise.
To ensure label reliability, apply additional quality control. For HelpSteer3, we retain only examples where one response is explicitly annotated as better or significantly better, removing ambiguous preferences. For math and reasoning datasets with synthetic responses, math-verify to automatically check the correctness of responses. For listwise data, we sample 3-5 negatives per instance and enforce that negatives yield different final answers from the positive,
employ
18

---

Published as a conference paper at ICLR 2026
Table 5: Detailed Per-task Experiment Results on RewardBench, RMBench, and JudgeBench.
Indg-Bench
Basdino
Minial
Doal!?
Clu
Chas-ilad
Resacs
Css
Mish
Code
Safety
Math
Code
Koowledge
Resace
TIM-weo Judge (Pairnise Evadiortion unles specille))
73.0
Ques3-00 (Paacoi
79.1
14.2
79.9
13.4
79.2
64.1
14.7
56.6
81.7
09.3
63.8
64.6
66.4
66.5
64.9
Centts 25 5-5thh (Poittissc)
71.8
71.0
0330
10.9
90.7
99.5
77.3
$6,0
90.6
70.8
71.4
73.8
61.0
70.6
65.9
96.1
76.1
67.5
61.6
91.7
59.5
56.1
56.6
Clasde 3.5
98.4
14.0
54,4
643
Ques3 <B Instrct (Pannac))
93.0
$0.2
90.1
90.5
96.0
75.2
81.7
67.3
77.1
75.3
62
700
56.2
70L1
639
Quea3-00 (Paivss
94.1
79.0
83.8
85.1
87.0
79.6
$2.9
61.6
89.6
77.9
75.0
66.3
65.4
67.0
675
Comiii-- s-iltsh (Parstee)
9501
61.9
75.6
90.0
91.
81.9
T0L
124
75.4
Scaler Reward Models (Paintwise)
Anno RM-58 5g
1000
96.9
76.8
90.5
97.3
18.3
07.8
57.5
53.1
92.4
67.7
Skywert 4emm
SOL
95.8
91.4
92.0
9h.1
91.8
69.5
56.7
532
91.9
67.3
Decpseds BTRM 278)
257%
61.7
Jariges
specllee)
Despocns 2787
25%
94.1
76.3
85.0
65.6
96.1
11-38 (Parvis))
22k
92.9
$0.3
85.6
85.9
95.7
23.4
42.0
11-08 (Niimatte)
10.-5
R8M-TTI
87-7
70.4
99.2
60.0
80.
96.7
84.9
20.4
61.8
66.1
670
57.0
96.1
74.6
85.2
02.6
54.8
50.2
09.3
Think R34
13k
14.4
77.9
85.2
$6.4
30.0
69.3
76.0
36.5
93.7
73.9
07.9
43.9
07.5
67.3
646
mool-angmented Judge
Owes3 Tool (Prinwiss)
80.0
74.8
62.4
58.6
73.5
72.1
42.7
566
Queek Sp Tool (Pertwis))
77.6
75.3
90.7
75.5
75.0
63.4
71.2
55.9
51.0
07.9
59.
57.3
56.2
66.5
59.4
Camins 5-8h d Bocl (Peattraa)
75.4
73.0
03.3
$5.5
51.5
62.2
73.6
49.0
96.3
71.0
73.2
78.5
59.1
69.3
66.5
212-Juuge-- Drniil (Phmmsste)
79.0
68.5
19.9
81.2
36.7
11.8
9037
50.8
71:3
66.7
Sudan- Dssiil SD (hannosss)
25k
79.5
73.9
64.9
$7.0
81.00
85.6
65.7
89.7
16.7
78.1
75.5
66.4
65.5
68.2
713-Juddp Zero SB (Prrwwee)
25k
83.
74.4
05.5
$1.9
511
66.7
58.3
06.3
9611
76.3
70.0
74.4
62.1
71.7
67.5
AgouRM (Pirnsse)
Tool (Palnise)
02.7
76.7
63.2
58.5
62.9
630
Qwea3-8o- (Paiwonn)
93.5
78.5
96.2
$5.9
19.
27.5
62.4
69.5
88.3
77.5
78.2
63.0
66.1
63.9
65.9
Camini-2554 F Tacl (Panstec)
90.90
$4.3
06.5
18.5
16.1
23.9
76.0
09.5
04.8
10.9
89.5
$5.1
67.5
71.4
24.6
212--Juige--ee Drstil (Phrmead)
05.0
91.6
71.6
61.4
81.3
$1.9
50.
2066
70.8
Divtll (Puieess)
02.3
177
67.7
183
76.4
736
213-Judd Zero SB (Paroise)
755
96.7
71.4
95.1
109
50.1
91.9
05.0
99.9
85.7
511
73.2
66.0
75.3
7200
For Reiromne:: Test-looed Reaesning Juste Rawdino with Parameters, (Paievine Braatonikmn
RUS!
95.1
90.1
HIM 22B
420k
94.7
$1.1
90.7
98.3
91.2
73.9
91.
74.3
95.3
85.4
87.5
65.7
68.5
76.5
7600
Ri--Ri--Depedde Diettl-
214
91.3
79.4
W9-1
94.1
189
71.8
91.5
09.5
91.1
81.0
19
89.0
100
754
76.2
91.3
60.3
91.8
74.1
95.4
83.9
81.3
77.5
76.4
Table 6: Dataset statistics for pointwise, pairwise, and listwise data.
Dataset
Domain
Pointwise
Pairwise
Listwise
Total
Tulu-3 Synthetic Pairs (Lambert et al., 2024)
IF
1,500
1,500
263
3,263
MATH (Heendycck et al., 2021)
Math
1,000
1,000
254
2,254
dapo bigmath (Yu et al., 2025a)
Math
2,500
2,500
282
5,282
(Muennighoff et al., 2025)
Math
250
250
500
UltraInteract (Yuan
2025)
Code
2,000
2,000
4,000
CodeRM (Ma al., 2025b)
Code
1,000
1,000
472
2,472
WebInstruct (Ma et al., 2025a)
Reasoning
1,000
1,000
91
2.091
Loong (Huang et al., 2025b)
Reasoning
700
700
99
1,499
HelpSteer3 (Wang et al., 2025c)
Helpfulness
2,000
2,000
0
4,000
SafeRL (Dai et al., 2024)
Safety
500
500
0
1,000
Total
12,450
12,450
1,461
26,361
preventing trivial shortcut solutions. Finally, we address potential biases such as stylistic artifacts in evaluation datasets (Wu et al., 2025), reducing the risk of overfitting surface-level patterns.
E ADDITIONAL IMPLEMENTATION DETAILS FOR EVALUATION
Implementation of different evaluation protocols. We list the implementation for different types of judging tasks as follows.
Pointwise: For pointwise evaluation, follow the protocol of RewardBench2 (Malik et al., 2025), assigning partial credit of 0.5 when two responses are scored as a tie. Both TIR-Judge and pointwise baselines evaluated under this rule.
Pairwise: For pairwise evaluation, we adopt the setup (Guo et al., 2025b) to report the accuracy over a single random ordering of paired responses across all judgment benchmarks.
19

---

Published as a conference paper at ICLR 2026
Listwise: For listwise evaluation in RewardBench2, follow the best-of-k setting in (Malik et al., 2025). For example, in best-of-4, the model is provided with a prompt and four candidate completions, and identify the best response among them.
Implementation details for baselines. Apart from our backbone models (Qwen-3), we run the following baselines models on our end during evaluation that are publicly available while within our compute budget:
RM-RI (Chen et al., 2025b): All the models are available at the Hugging- Face
https:/ http://uggiigffeeee http///gggggnggceee/eee colaaace.cccccoooooooeeooooooooo wtw.://ooooeeciioo//aottaaaaaaa. Im rl $68128ccab332701cceeeee
platform:
RRM (Guo et al., 2025b): All the models are available at the HuggingFace platform:
attps://hugging: ace co Roo/ReaardRRessomnng
Think-RM (Hong et al., 2025): The models at the HuggingFace platform: https: : huggingface, co /etin/iagggeeeeerreerereaaaaaare Think RM-8- 8B. We chose the binary version due to its reported better performance.
AgentRM (Peng et al., 2025): The codebase of AgentRM is publicly available at https: / //github. com (BO--KKG/Aggnnte Reward Modeling.
Gemini-2.5-Flash (Comanici al., 2025): We follow the guideline https:// ai google dev / snee//gmmnn-iee geminiiappiddee cooe executiion for running experiments with code execution service.
For RM-RI, RRM, and Think-RM, they all designed for pairwise ranking only, and the same pairwise judging prompt reported the paper to ensure fair comparison. For other baselines, of the works (Whitehouse et al., 2025) are not publicly available, only use the reported results in the original paper for comparison.
F DETAILED RESULTS FOR BEST-OF-N EXPERIMENTS
Experiment Setup. Here, we implement three types of the best-of-N selection task. We select AIME- 2024, AIME-2025, BigCodeBench and IFeval for evaluation. For AIME-2024 and AIME-2025, each containing 30 problems, we evaluate four backbone models: Gemma-3- ----------I,, Bwen-3-338-Thinke and RI-Dist ill 1ew.iiii----1-111 8B. For each backbone, we allow a maximum generation length of 16k tokens and sample 16 valid responses per problem. For BigCodeBench and IFEval, we reuse model outputs from the JETTS dataset (Zhou et al., 2025). On BigCodeBench, we consider Qwen-2 5- 32B, For IFEval, we seleet Qwen 2. 5- 72B and Owen 2 .. 10.553221 32B as backbones, and use the original benchmark generations for evaluation.
-ooeeee--ooorreeee and Qwen-2 5-Coder-T7 7B as backbones.
For pointwise judging task, we use the judge to give the rating for each response, and select the resposne with the highest score (if there multiple responses, answer to obtain the final answer). For listwise and pairwise judge task, we follow (Guo et al., 2025b; Liu al., 2025b) to adopt knockout tournament style in (O()) comparisons for promoting efficiency.
majority voting the
Detailed Experiment Results. Table reports detailed per-dataset and per-model results, showing the number of solutions passed across four benchmarks under different Best-of-N judging settings.
Table 7: Performance comparison across benchmarks and checkpoints (accuracy in %).
Bendenerk (Siee
Madel
PauOt
MY0016
+3 PoLsivii
7am
(oyr--yy---a 48 Palkeb:
Sockyy-220 58 PNAESSS
00 Patrbc
02 Liddiic
BBM--1
(Pai)
Gemera 5-211
56.7
31.0
33.3
36.2
16.2
45.5
2004
$3.0
15.3
11.5
300
36.7
1000
43.5
4000
25.2
R1 disti 5636
36.7
910
$010
83.5
180
30/1
104
181
$00
21.3
2005 (106
90.0
133
200
400
400
23.3
66.2
33.3
363
R R-ddnii 233-----00
30.0
753
735
133
1333
15.2
15.2
Big Righvireeehh (11 30
Deeppork Corter
15.0
784
183
13.
16.
13.6
16-3
361
11.
Quin-
31.4
35.2
409
41A
IREi (541)
Oreen 330h burnet
340
914
13.1
5601
512
$1.
345
5601
85.3
$5.
20

---

Published as a conference paper at ICLR 2026
Table 8: Ablation Study on Reward Formulation (Multiplication vs. Addition).
Method
MMLU-P
MATH GPOA
MBPT-P
IFEval
Benchh
CJBench
RW Bench
RMBench
JGBench
TRR-hulge-Diaill (Chrrs)
58.7
66.7
TI8-hdge-7Zz0--44 48 (Ours)
62.5
87.3
54.7
64.8
79.8
69.8
65.9
61.5
77.3
72.8
70.4
TIR Judge Distill 4B (Add.)
62.5
23.4
46.9
63.5
74,2
61.1
65.5
59.3
73.7
68.5
60.5
TIK 11R-Jude--ooo 2aro--t 4B (Add.)
63.5
89.8
48.5
02.0
68.5
06.5
63.0
61.1
76.0
63.4
68.6
Pairwise Psoluction
Judge
(Ours)
IIK-Judge-Zaro (Curs)
IIR-hude-DDiall (Add.)
56.5
85.4
43.9
56.5
85.5
74.5
66.3
77.4
65.3
TIR- Julge-Zeru 4B (Add.)
69.5
91.9
52.1
63.5
81.2
71.6
72.0
68.7
85.4
80.0
72.0
Table Position bias analysis. We report performance for A-B order, B-A order, and the average. Results indicate minimal variance for our method compared to baselines.
Model / Setting
MMLU-P
MATH
GPQA
MBPP-P
IFEval
PPE Arg.
IFBench
CJBench
BeBBench
RMBench
TuR-uzd---ero 4B
A-B
75.0
93.3
61.7
67.3
84.5
76.36
70.3
70.8
86.7
80.8
B-A
76.4
92.6
59.6
66.4
81.5
75.3
73.6
69 2
86.2
80.6
Avg.
75.7
93.0
60.6
66.9
83.0
75.8
72.0
70.0
86.4
80.7
Ri-geude-Distll
A-B
78.0
B-A 68.5
89.2
53.0
39.4
84.5
71.0
71.1
69. 9
86 1
76.9
68.7
89.0
53.9
84.1
71.1
72.4
86.9
77.5
Owent-4h-Tome
A-B
35.9
75.7
B-A
63.5
83.3
37.4
57.3
63.8
61.1
61.6
28.0
86.0
66.9
Aug.
36.7
TIR- HR-JJdge-Zere 8B
A.B
76.6
58.5
B-A
76.5
93.8
57.3
68.6
80.7
75.4
67.4
70.0
87.4
81.2
Aug.
76.6
82.4
TIR- TIR-Judge-Diee Judge-Distil 8B
A-B
72.2
90.4
$3.8
63.2
85.7
73.0
74.3
70.0
87.9
82.2
B-A
72.6
90.6
52.7
61.5
84.8
72.5
74.1
72.5
88.8
80.0
Avg.
72.4
90.5
53.3
72.8
81.1
Qwent-8R-Tool
A-B
72.0
85.2
56.0
54.3
60.8
65.7
$4.9
77.3
B-A 71.5
83.3
52.4
54.1
60.2
64.3
57.4
55.0
85.1
71.2
Avg. 71.8
84.3
54.2
54.2
60.5
65.0
55.0
55.0
85.7
74.3
From Table 7, we observe that TIR- IIR-Judge consistently delivers strong performance across model scales and judging formats, highlighting its robust generalization ability. These results demonstrate that T TIR-Judge is not only effective but also readily transferable to diverse target tasks.
G ADDITIONAL STUDIES FOR JUD-JUDGE
Different Reward Combinations. Table shows the comparison of our proposed TIR-Judge (using multiplication reward formulation) against the addition-based reward formulation variant. We report results across both Pointwise and Pairwise settings. The result shows that using the addition form as the reward would lead slightly worse performance.
Performance of TIR-Judge with different orders. Table shows that TIR- TIR-Judge exhibits small positional discrepancy (typically <1 %, at most 2%), while the backbone Qwen3 models can have a relatively higher variance (up 9%). This confirms that our training procedure effectively mitigates position bias.
Performance of TIR- TIR-Judge with different response length. To evaluate the verbosity bias, report the accuracy separately for cases where the chosen longer or shorter than the response. The results show minimal difference between the two categories, and in some cases, TIR-Judge further reduces the verbosity gap observed in the Qwen3 backbone. This indicates that verbosity bias is well controlled.
21

---

Published as a conference paper at ICLR 2026
Table 10: Accuracy Comparison Split by Response Length. We report the judge's accuracy when the ground-trui chosen response is longer than the rejected one versus when it is shorter (i.e., the rejected response longer).
Model
Acc. (Chosen > Rejected)
Acc. (Rejected > Chosen)
TIR-Judge- 4B
78.58
76.27
TIR-JJudge-Zero 4B
80.67
79.27
Qwen-3-4B-Tool (Backbone)
65.79
71.20
TIR-Judge-Dissill 8B
80.97
79.18 18
TIR-Judge-Zero 8B Qwen-3-8B-Tool (Backbone)
77.73
77.98
72.61
72.25
Table 11: Comparison of Training Costs. Left: GPU wall-clock time breakdown. Right: Estimated financial cost including compute and API fees.
Stage
"IR-Judge-Zerr
DiR-Juuge-Diieil
Component
9IR-Judgo-Zero
TIR-Judge-Diall
SPT
RL. cemintoecement Learning)
RS (Rejection Sampling)
23.00
2,5h
3.5b
8.5h
1.0h
1.5h
Teacher API Cost
Compute Cost (#xH100)
$0
8650
S210
~130
Total Time
29.0h
11,0h
Total Cost
$690
8303
H COST ANALYSIS OF TIR-JUDGE
We provide detailed breakdown of the computational and financial costs for training TIR-J 8B in Table 11. Experiments were conducted on 8 NVIDIA H100 GPUs. We estimate the total cost based on current market rates for H100 clusters and the official API pricing (Gemini-2.5) for generating the 10k distillation samples. While MRR-Judge-Zer is approximately 2x more expensive (~ $690 vs. $340), we argue this trade-off is strategically valuable. It unlocks autonomous self-improvement by eliminating the dependency on teacher supervision, making it critical solution for privacy-sensitive environments where access to frontier models is restricted. In practice, recommendations are:
Use Distill for domains where there are noticeable gaps (measured by held-out evaluation sets) between the teacher and student model, provided API costs are permitted. It is ideal for injecting specific capabilities- -ucch as safety, IF, correct tool-call formats, and thinking structures that the base model lacks.
Use Zero when superior teacher exists (e.g.. improving SOTA models) or to avoid API dependencies due to privacy concerns. It is ideal for unlocking latent reasoning capabilities through self-exploration, effectively trading API costs for training compute.
1 MORE CASE STUDIES ON FAILURE CASES
We illustrate representative failure case of TIR-Tudge in Table 12. In this example, the model incorrectly counts the number list items. The highlighted erroncous code shows that TIR-Judge attempts to match numbered lists beginning with digit followed by capital letter, but the actual response uses the **** Markdown pattern for list items, which the regex fails to capture. This error arises because (1) the model does not robustly enumerate all formatting variants when generating code, and (2) although the task can be solved either with or without tool use, the model fails to recognize that direct reasoning (i.e., counting the items without code) would have been simpler and more reliable.
Additional failure cases include generating incorrect code (e.g., referencingg undefined variables) or misunderstanding the semantics of the prompt-for instance, when asked to check for the presence of [keyword), TIR- TIR-Judge incorrectly interprets the literal braces "I "[}" as part the keyword.
22

---

Published as a conference paper at ICLR 2026
Table 12: A case study from IFBench of failed judgement generated by TIR-Judge-Zero 8B, Red text stands for the incorrect reasoning step.
User Prompt
Hello, how leam uong only online counes and unonals? Please provide the anower with a light hearte and encoumging tons. Provide a resporse tur ttem beginning capital
ousttly world, suucured sumbered
eading poñiod
appeur
platfeens Couneer, edX "Chomme Your Language * Lears Python, mosi popilar languge fur Al, dmuugh tateria on Cadesaleny freeCuieC
Besice
fundomemaas
"Dive into Dat Explore data science and machine louning counse, they the bazkhooe
on websiles like DauCamp Karple.
**Jondss-n Practio- Apply your knowledge lodge by portecppring in choillnge Karple or hen ding smill progors using datseets
*JJoi Comminsee Engaze with enchuitsit on focams like Reddit's iiMachiine..canis or Al specillc groups on Linkedin.
*Suy Updaned Follow blogs. podcasts and Youeube channess to keep up with the lizest trends und bezakthooghs. manahum, spiu. Enjoy joursay and celcheatc your prograd
Up Remember learning
poshing forward, maseer within your reach!
Judge Trajectury 27D8--7dgee-7a
check:
Wasther the
cuuclyy 150 words.
80
Wheter it's stncttredd in sumbered lis with each iem beginning with capiral leter and ending with peried.
Whether the word 'll' appears lest limes.
'python
vozd count
tentresponse.ppll st.ppp-tt.))p)
capital
fe.ddld (2" Widss, (
Check ocnurrences
"Az"
ao_coust zeaponze. lovaz cl.count
print (e*Word count: word count")
Deint (E*Susbes of Al ecouzenncis: aiLooust")
peint (f*Jusbes o2 list itens: ieaistars"
"Moutguc
Mumber of 11s: Stene:
23


Reviews:
[{"review_id": "fFjPciU140", "reviewer": "Reviewer_LHuR", "summary": "This paper introduces TIR-Judge, an end-to-end RL framework for training LLM-based Judges that interleaves reasoning with Python code execution. Using this framework, the authors train several TIR-Judge models, which tend to outperform other LLM-based Judges without tools.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, and the motivation and approach are clear. The figures and case study enhance readability.\n2. The proposed method framework is effective, and the evaluation is comprehensive, spanning a variety of baselines and several relevant benchmarks.\n3. The flexible judgement formats (e.g., pointwise, pairwise, and listwise) expand the functionality of the framework.", "weaknesses": "1. Prior work has identified several biases common in LLM-based judges, e.g., positional bias [1], verbosity bias [2], and self-preference bias [3]. Other work [4] has shown that training can increase the prevalence of such biases. Given this, this work should evaluate for bias before and after training.\n2. It's unclear how the baseline tool-augmented judges (e.g., Qwen3-4B-Tool or Gemini-2.5-Flash-Tool) are evaluated. Are they expected to output tool calls in the same format as TIR-Judge, e.g., '''python...''', or do they use the native tool-calling features of the models? If it's the former, it's difficult to assess whether the improvement comes from learning how to make effective tool calls versus just learning how to make tool calls using the correct format. If it's the latter, it should be made clear in the paper.\n3. The studies on efficiency could be better explained. Is code execution time factored in? Why are TIR-Judge-Zero 4B and TIR-Judge-Distill 4B 2x as efficient as Qwen-3-4B (their base model), yet TIR-Judge-Distill 8B has similar latency to Qwen-3-8B? Can you provide the average number of generated tokens for each?\n4. (minor) Table 6 is confusing. It reports the number correct instead of accuracy, and the lack of precision makes comparisons difficult. Is this over a single run of 16 responses?\n[1] https://arxiv.org/pdf/2406.07791\n[2] https://arxiv.org/pdf/2310.10076\n[3] https://arxiv.org/pdf/2410.21819\n[4] https://openreview.net/pdf?id=JFTSZa2stt", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QWk3ckqHVN", "reviewer": "Reviewer_sHW4", "summary": "The paper introduces TIR-Judge, a framework for training LLM-based judges that integrate tool execution (Python interpreter) within a reinforcement learning pipeline. Unlike prior text-only judges, TIR-Judge enables LLMs to generate, execute, and learn from code to perform verifiable evaluations.\n\nEmpirical results on six major benchmarks (e.g., PPE, IFBench, CodeJudgeBench, RewardBench) show consistent improvements over reasoning-based judges.", "questions": "1. Could iterative RL amplify over-confidence or lead to reward hacking?\n2. How is when not to call a tool learned? Is this behavior emergent from the RL reward or manually encoded through heuristics?\n3. Would a multi-tool setup (e.g., Python + web search + symbolic solver) be stable under the same RL objective?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel integration of tool-use with reinforcement learning: The work goes beyond inference-time tool invocation by embedding code execution into the RL loop, allowing models to learn when and how to use tools effectively.\n2. Strong empirical performance: Demonstrated improvements across multiple datasets and formats, outperforming larger models like 32B RRM on several metrics. Particularly impressive is the performance of TIR-Judge-Zero, which learns without teacher supervision.", "weaknesses": "1.Domain bias: Gains are largest in verifiable domains (math, code), while improvements in non-verifiable areas (helpfulness, safety) are marginal.\n2. Generalization limitations: The framework is Python-specific; it is unclear how well the method would extend to multiple heterogeneous tools or symbolic engines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MqAc6K5aME", "reviewer": "Reviewer_LE7M", "summary": "This paper presents TIR-Judge, a framework that trains LLM-based judges to integrate Python code execution with reinforcement learning, achieving strong empirical results across six benchmarks with notable parameter efficiency (8B model matching 32B baselines). The work demonstrates experimental coverage including pointwise/pairwise/listwise evaluation formats and best-of-N validation, with generally clear presentation and detailed appendices. The authors present the first approach to co-optimize inference and tool use (executing Python code) for LLM evaluator learning via reinforcement learning. Researchers achieved up to 6.4% performance improvement on pointwise and 7.7% performance on pairwise compared to robust inference-based evaluators on six public benchmarks, and 96% of Claudet-Opus-4 performance on 8B parameter models. TIR-Judge-Zero performs on par with or superior to distillation-based versions with only pure reinforcement learning without distillation from the teacher model.", "questions": "1. Methodology Related (Critical)\nQ1: Please clarify what exactly sθ(x, y) of pointwise evaluation means in Section 4.2 Equation (2). Is it the value in the <score> tag generated by the model or is it the scalar output inside the model? If it is the former, what do you do if an error occurs while parsing the generated text?\n\nQ2: TIR-Judge-Zero's iterative learning said it would prefer to \"keep only one trajectory per each prompt, with the shortest response or least tool calls,\" but how do you decide when these two criteria conflict (e.g., long response, but with fewer tool calls)? Do you have priorities?\n\nQ3: Why did you choose the multiplication form from the reward function of Equation (3)? Have you tried the addition form (R = α₁ Rc + α₂Rf + α ₃ Rt)? Didn't the multiplication form cause the sparse reward problem early in learning?\n\n\n2 Experimental Design Related (Critical)\nQ4: How many random seeds did the results in Table 1 run? TIR-Judge-Zero 4B pairwise recorded 75.0% on PPE, what is the standard deviation or confidence interval for this? Specifically, did you test whether the performance difference between TIR-Judge-Zero and Distill was statistically significant?\n\nQ5: How did you implement the \"Tool-augmented Judges\" baseline (Qwen3-Tool, Gemini-2.5-Flash-Tool)? Did you simply add \"you may use Python code\" to the prompt, or did you do new-shot prompting with some examples of tool usage? How much does learning them with data like TIR-Judge improve the performance?\n\nQ6: Pairwise evaluation said you used \"single random ordering\", how did you control position bias? It's standard practice to evaluate and average both A-B and B-A orders, so why didn't you do this?\n\n\n3. Interpretation of Results (Important)\nQ7: Table 4 shows a mixture of cases where TIR-Judge-Zero is superior to distilled versions and cases where it is not. Under what conditions do you find a pattern that is better for Zero and under what conditions is better for Distill? Based on this, what would you recommend to practitioners?\n\nQ8: In Figure 5, will the performance improve further or will it converge if we continue with an additional RL round after TIR-Judge-Zero-RL-2. What is the optimal number of iterations?\n\nQ9: TIR-Judge on the RewardBench2 (Table 2) is higher than Claude-Opus-4 on the Instruction Followings, but far behind on Safety. What do you think is causing this difference? Could tool use be rather detrimental to safety assessments?\n6.4 Regarding calculation costs (Important)\n\nQ10: How do I compare the total learning cost (SFT + RL) of TIR-Judge-Distill to the total learning cost (RL + RS + SFT of multiple rounds) of TIR-Judge-Zero in GPU time? If the Zero method requires more computation, which is more efficient compared to the cost of generating a 10k sample of distillation data?\n\nQ11: How many samples did Reception sampling generate on average per each prompt, and what percentage of them produced the correct format and answer? How did this acceptance rate change as learning progressed?\n\n\n5. Generalization and Limitations Related (Moderate)\nQ12: How do I modify my framework to extend it to other tools (web search, calculator, database query, etc.) other than the Python executor? Is the current reward function or learning procedure still applicable to other tools?\n\nQ13: Has it occurred during learning or evaluation to generate malicious or dangerous code (e.g. file system access, infinite loop)? How did you detect and process it?\n\n< Additional feedback for improvement >\n1. Must-have \nAdd statistical significance: Rerun the experiment with at least 3-5 random seeds, report the mean±standard deviation. For key arguments (for example, Zero > Distill), provide a paired t-test or bootstrap confidence interval.\n\nCompare fair baselines: Reevaluate tool-augmented baselines by fine tuning them into the same learning data. Separate the actual contribution of RL by adding \"TIR-Judge without RL (only SFT)\" as at least ablation.\n\nPosition bias control: Perform Pairwise evaluation in both A-B and B-A directions and average; current single ordering can have a maximum bias of ±5%.\n\nAdd an experiment to compare the multiplication form and the addition form of the reward function ablation: expression (3). This will show the robustness of the method.\n\n2 Strongly recommended improvements\nCalculate Cost Analysis: Add a table comparing the total cost of learning in Distill vs Zero to GPU time and dollar amounts. This is important for practitioners to make decisions.\nAdd theoretical analysis: Add theoretical insights or analyses on why tool integration is particularly effective for judges and why iterative RL converges. There is currently a lack of explanation beyond \"it works empirically\".\n\nFailed Case Analysis: Add a section to analyze when TIR-Judge still fails. For example, if you write a code but make an error due to incorrect logic, or if you need to use the tool but don't use it, and so on.\nData Efficiency Analysis: What percentage of 26k samples actually require tool use?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Defining clear and practical issues\nThe practical limitations of LLM evaluators (accurate calculation, failure to verify constraints) are clearly presented with concrete examples.\nThe importance of evaluators has been convincingly described throughout the model development pipeline (post-training, inference, and evaluation).\n\n2. Comprehensive Experimental Design\nResearchers systematically evaluated it on six different benchmarks. They demonstrate generalizability by addressing all three evaluation formats: Pointwise, pairwise, and listwise.\nBest-of-N experiments validate its practicality in downstream applications.\n\n3. Thorough ablation study\nAuthors systematically analyzed the data mixing effect (Figure 3), tool usage vs text-only (Figure 4a), and progress of iterative RL (Figure 5).\n\n4. Efficiency and Scalability\nParametric Efficiency: 8B model showed competitive performance with 32B model.\nComputational Efficiency: We demonstrate no additional inference time overhead despite tool integration (Figure 6).\n\n5. Efforts for Reproducibility\nAddendum provided detailed prompt templates, dataset configurations, hyperparameters, and more.\nIt specified data pollution prevention efforts through 8-gram deduplication.", "weaknesses": "1. Problem: The key components of this paper (tool-integrated reasoning, RL-based judgments, and projection sampling) are all techniques covered in previous studies. Combining them together is meaningful, but there is a lack of theoretical analysis or deep insight into why this particular combination is effective.\n\nEvidence: Feng et al. (2025), Li et al. (2025a), cited by Related Work, have already learned TIR as RL, while Chen et al. (2025b), Whitehouse et al. (2025) have dealt with RL-based judges. This paper is a combination of these, but does not explain why applying TIR to judges is essentially different from applying to policy models.\n\n\n2. Lack of statistical rigor (Major)\nProblem: All experimental results were reported as single run point estimates, and no confidence interval, standard deviation, or statistical significance tests were presented at all.\n\nEvidence: All figures in Table 1 are shown up to a single decimal point, and there is no variance or p-value for multiple random seeds. In particular, we do not know if the performance difference between TIR-Judge-Zero and Distill (e.g., a 3.7% difference in 4B pointwise) is statistically significant.\nImpact: The credibility of one of the main arguments, the conclusion that \"distillation is unnecessary,\" is diminished.\n\n\n3. Unfair comparison and baseline setting issues (Major)\nProblem 1: The results of some strong baseline models (J1, GPT-4o, Claude 3.5, etc.) were cited in the original paper, and it is unclear whether they used exactly the same evaluation protocol and data preprocessing.\nProblem 2: The \"Tool-augmented\" baseline (Qwen3-Tool, Gemini-2.5-Flash-Tool) seems to have induced tool use simply by prompting; for a fair comparison, they should have been fine-tuned with the same amount of tool use data.\n\nEvidence: In Section 5.1, we said \"For baseline, we consider... Qwen-3 (Team, 2025) (our backbone) that use the same tool as TIR-Judge,\" but they didn't specify how to learn. If you look at the prompt in Appendix B, it simply instructs you to \"You may use Python code.\"\nImpact: It's possible that the actual benefits of tool integration have been overestimated.\n\n\n\n4. Lack of consistency in TIR-Judge-Zero's claim of excellence (Moderate)\nProblem: While the paper emphasizes that TIR-Judge-Zero is effective without distillation, the real-world results are mixed with benchmarks and scales.\n\nEvidence:\n4B pairwise: Zero excels on 3 of 6 benchmarks\n8B pairwise: Zero is only good on 2 of 6 benchmarks\n4B pointwise: Zero is excellent in 7 out of 10 subtasks\n8B pointwise: Distill excels in 6 out of 10 subtasks (Table 4)\nImpact: The claim that \"distillation is unnecessary\" is exaggerated and should be corrected to \"competitive performance can be achieved without distillation under certain conditions\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6X0Afa70O2", "reviewer": "Reviewer_dbDX", "summary": "Proposes TIR-Judge, a framework to train (finetune) LLM judges. Specifically focuses on training them how to use tools (Python executor) using reinforcement learning, across verifiable and non-verifiable domains. Evaluate TIR-Judge on multiple benchmarks and show significant improvements over baseline, with just a small model.", "questions": "1. In Findeis 2025’s work, it was found that some of the benchmarks are too easy out of the box. How does your RL scale with complexity? E.g. does a better model still see improvements (on harder tasks), or do we only see gains here since it’s “low hanging fruit”?\n2. \"For safety and general helpfulness prompts, a positive format reward is granted only if the model produces a valid output without invoking tools\" -- What is the impact of this?\n3. Impact of training duration?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Introduces a simple but effective framework to improve LLM-as-a-judge using RL on relatively small models.\n2. Good figures and explanations.\n3. Detailed evaluations and ablation comparisons. Well structured.", "weaknesses": "1. “eward-guided best-of-N inference over datasets” typo\n2. Analysis on the code it does in Python? How often errors (runtime, syntax)?\n3. Given they propose a framework, it is hard to know how well the training recipe and setup works in other setups. How well does it work on smaller models? Different model families?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)