On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalization
Authors: Janvijay Singh
Austin Xu
Yilun Zhou
Yefan Zhou
Dilek Hakkani-Tür
Shafiq Joty
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
LLM-as-a-Judge
Distribution Shift
Generalization
Evaluation Robustness
Primary area: foundation or frontier models, including LLMs
TL;DR: We study the generalization and robustness of LLM-as-judge as new LLMs emerge, addressing practical questions about judge shelf life, including future proofing, backward compatibility, and question generalization.

Abstract: The LLM-as-a-judge paradigm is widely used in both evaluating free-text model responses and reward modeling for model alignment and fine-tuning. Recently, fine-tuning judges with judge-specific data has emerged as an often preferred choice over directly prompting frontier models as judges, as the former achieves better performance with smaller model sizes while being more robust to common biases. However, the standard evaluation ignores several practical concerns of fine-tuned judges regarding their real-world deployment. In this paper, we identify and formalize three aspects that affect the *shelf life* of these judges: *future-proofing* and *backward-compatibility* $-$ how well judges fine-tuned on responses by today's generator models perform on responses by future models or past models, as well as *question generalization* $-$ how well judges generalize to unseen questions at test time. We study these three aspects under a unified framework with varying train and test distributions in two reasoning datasets, three SFT- and DPO-based fine-tuning algorithms, and three different backbone models. Experiments suggest that future-proofing is challenging for most models, while backward-compatibility is relatively easy, with DPO-trained models consistently *improving* performance. We further find that continual learning provides a more balanced adaptation to shifts between older and newer response distributions than training solely on stronger or weaker responses. Moreover, all models exhibit some degree of performance degradation when moving from questions seen during training to unseen ones, showing that current judges do not fully generalize to unseen questions. These findings provide insights into practical considerations for developing and deploying judge models in the face of ever-changing generators.

Published as a conference paper at ICLR 2026
ON THE SHELF LIFE OF FINE-TUNED LLM-JUDGES: FUTURE-PROOFING, BACKVARD-COMPATTILIIYY, AND QUESTION GENERALIZATION
Janvijay Singh 1.2 Austin Xu
Yilun Zhou? Yefan Zhou 1.3+
Dilek Hakkani-Tiie Tür?
Shafiq Joty'
Salesforce Al Research
2 University of Illinois Urbana- Chaapa-Champaign
3 Dartmouth College
ABSTRACT
The L-de-as-a-judge paradigm is widely used in both evaluating free-text model responses and reward modeling for model alignment and fine-tuning. Recently, fine-tuning judges with judge-specific data has emerged as an often preferred choice over directly prompting frontier models as judges, as the former achieves better performance with smaller model sizes while being more robust common biases. However, the standard evaluation ignores several practical concerns fine-tuned judges regarding their real-world deployment. In this paper, iden- tify and formalize three aspects that affect the shelf life of these judges: future- proofing and backwand-compatibiiy by today's 's generator models perform on responses by future models or past mod- els, as well question generalization how well judges generalize unseen questions at test time. We study these three aspects under unified framework with varying train and test distributions two reasoning datasets, three SFT- and DPO- based fine-tuning algorithms, and three different backbone models. Experiments challenging for most models, while backward- compatibility is relatively easy, with DPO-trained models consistently improving performance. We further find that continual learning provides a more balanced adaptation to shifts between older and newer response distributions than training solely on stronger or weaker responses. Moreover, all models exhibit some degree of performance degradation when moving from questions seen during training unseen ones, showing that current judges do not fully generalize to unseen ques- tions. These findings provide insights into practical considerations for developing and deploying judge models in the face of ever-changing generators.
responses how well judges fine-tuned
suggest that future-proofing
Code: http://ghhhbb.oommmmmaaaaayyyyyp analysss
INTRODUCTION
Automatic evaluators have become central part of the large language model (LLM) development cycle. They serve both as reward models during training (Stiennon et al.) 2020 Ouyang et al. [022)
Yuan et al. 2024 and as verifiers in inference-time compute scaling (Zhou al. 2025 Kim al.
2025 Singhi et al. 2025). In the L.L-Mas-judge paradigm, a generative language model evaluates the outputs of other models for given input question, providing scalable approach automatic evaluation. Past work on LLM-as-judges began with zero-shot prompting of capable LLMs (Liu
al. 2023 Dubois et al., 2023 However, such judges have been shown to prone to various biases, such as stylistic bias (Zeng et al.) 2024 Raina et al. (2024), length bias (Zheng et al. 2023
Zeng et al.] 2024), and positional bias (Wang et al. [023] Pezeshkpour & Hruschka 2024 J. As
result, recent efforts have fine-tuned specialized evaluators Li et al. (2024b); Kim et al. (2024a); Vu
al. (2024), which have been shown be more robust to common forms of bias (Zhu et al.) [025]
Wang et al. 2024 Park et al. 2024a) while matching the performance of larger prompted models.
"Work done during an internship at Salesforce Al Research.
Work done Salesforce AT Research.

---

Published as a conference paper at ICLR 2026
Gemma 3
Generator Strength
Qwen2.5
Trained Judge
Future
9
Q R
Proofing
Q R
Backward Compatibility
Question Generalization
Trained Judge
Qwen2
Gemma
Generator Release Date
Figure 1: High-level overview of our setup for studying Future-Proofine Backward- ComppaabiiCmaiiibbllttty and Question Generalization through the lens of generalization and robustness to input distribution shifts. Q and R represent questions and responses, respectively, with responses generated by the shown generator models (Gemma2, Qwen2, Gemma3, Qwen2.5). Future-Proofing evaluates how well judges trained on responses from weaker, older generators (green: Gemma2, Qwen2) assess responses from stronger, newer generators (yellow: Gemma3, Qwen2.5). Backward- Compatibility examines the reverse direction. Question Generalization measures performance in-distribution questions and corresponding responses that were both not included (dashed Q and R) in the training.
Although recent advances in judge model fine-tuning have largely focused on developing training methodology [Chnn al. (2025ale). little attention has been devoted understanding how these models behave as a function of their training inputs. In this work, we investigate this gap by asking three key questions: First, can judge models trained on fixed datasets of input questions, model responses, and ground-truth verdicts accurately evaluate the responses of newer models, i.c., are judges future-prooff Second, if we train a judge on up-to-date responses from newer models, can Third,
reliably evaluate responses from older models, i.e., is the trained judge crrkward-compattinee fixing the response generating models, how reliably can judges assess questions that differ from those seen during training, i.e., do they generalize to questions? We examine these questions, as illustrated in Figure [ through the lens of generalization and robustness, aiming to understand the shelf life of trained judges.
In this work, we propose a dual-distribution formulation of automatic evaluation. Concretely, we model the judge's 's input as comprising elements drawn from two distinct distributions: the question distribution, which characterizes the input questions to be evaluated, and the response distribution, which characterizes the responses be judged. We study the performance of trained judges when responses are drawn from relatively weak and strong generators, henceforth referred to as weak responses and strong responses. We also examine how well trained judges evaluate questions that (1) seen during training but paired with new responses, and (2) completely unseen during training. By focusing on weak and strong generators and novel questions, we gain insights into the shelf life of trained judge models through four practical questions:
Future- prrooing Given a judge trained on responses from older ("weak") models, how accurately can it evaluate responses from newer (" ("strong") models? If the goal is to evaluate strong responses, how much benefit do practitioners gain by training on strong responses rather than weak ones? Backward-come compatiiomaaiiillity. Given judge trained on responses from newer ("strong") models, can it reliably assess responses from older ("weak"") models? the goal to evaluate weak responses, does training judge strong responses provide any benefit?
Continual learning. Compared to judges trained only on weak or strong responses, how well does continually trained judge adapt distribution shifts between the response distributions? Question generalization. Does judge performance depend on whether a question was seen during training? Even for questions, can judge reliably assess new responses?
Using two verifiable datasets (DeepScaleR and MMLU-Pro), we set up a suite of controlled exper- iments to analyze the shelf life of judge models, training across three backbone models of varying
2

---

Published as a conference paper at ICLR 2026
sizes and capabilities and three popular budge-training recipes. Our findings reveal that fine-tuned judges struggle to evaluate newer, stronger model responses and therefore require training on an up-to-date response distribution. Once trained newer, stronger responses, judges exhibit some degree of back waadwwadooppabbiiiii between older and newer response distributions than training solely stronger solely on weaker responses. Finally, we find that fine-tuned judges struggle to generalize to new questions. In all, our findings inform the development and deployment of future generations of fine-tuned judge models.
Continual training provides a more balanced adaptation to shifts
2
BACKGROUND AND RELATED WORK
2.1 AN OVERVIEW OF FINE-TUNED JUDGES.
LLM-based judges are automatic evaluators that evaluate LLM outputs given some evaluation cri- teria. While many judges accommodate different evaluation tasks, such as single rating ("Rate this response on scale of 1-5") (Hu et al. 2024) or classification ("Is this response appropriate?") (Vu et al. 2024), the dominant evaluation paradigm LLMM-based judges are deployed with is pairwise evaluation. Here, judge is given a question and two candidate responses, and tasked with selecting the "better" response according to some criteria. Formally, the judge performs the transformation
(Q, R1, R2)
(C, C optional,
where Q is the question, R1, R2 are the two candidate responses, C' is an optional chain-of-thought explanation, and V is the verdict of which response is better. We denote
R1, R2) ~
X' to be the judge input and y (C, V) to be the judge output. Pairwise judges are typically evaluated using accuracy or consistent accuracy, the latter accounting for response-order bias as detailed in Appendix D Due to its popularity and practicality, pairwise evaluation forms the focus of our study.
Past work in judge fine-tuning supervised fine-tuning (SFT) Li al. 2024b Kim et al. 2024b
Zhu et al.) 2025), preference optimization methods, like direct preference optimization (DPO) (Wang et al. 2024a Ye et al. 2024 -Sadd-Falee et al. (024). more recently, reinforcement learning with verifiable rewards (RLVR) (Chen et al. 2025a c Whitehouse et al. 2025 Xu et al. 2025b). Starting from dataset of (xx, V* (x)) pairs, where V denotes the ground-truth verdict/label, each approach constructs training samples differently: SFT and DPO approaches sample judge outputs from a teacher model, then use V*xx) categorize judge outputs as either correct outputs y or incorrect outputs y Then, the judge is trained on (z, y*) pairs for SFT and (z, vt. . triplets for DPO. On the other hand, RL approaches directly make use of the (x, (xx) pairs, omitting the need for teacher model explanations.
2.2 RELATED WORK
Distribution Shifts and Generalization. Distribution shift, the mismatch between training and evaluation data, is a long-standing challenge in machine learning (Hendrycks & Dietterich) 2019 Koh et al. ).2T). Early computer vision studies demonstrated significant accuracy drops under mi- perturbations (Hendrycks & Dietterich 2019). and WILDS extended this to real-world domain 2021). In LLMs, the problem is amplified as both data and model capabilities evolve over time (Shi al. 2025). Recent frameworks explore how models transfer across distri- butions. Easy-to-hurd hharrd generalization examines whether training on easier tasks transfers to harder (Sun et al.) 2024), which relates to scalable oversight where only easy tasks can be reliably supervised (Amodei et al. (0166]; task-difficulty can be estimated using either model or data-centric measures (Swayamdipta et al.) 2020). Weak-to-strong generalization investigates improving strong models using supervision derived from weaker ones (Burns et al.) 2023). Our setting complements these efforts by focusing on distribution shifts that arise from an evolving population of generators and by evaluating how judge models adapt to both weak-to- and strong-to-weal shifts.
shifts (Koh et al.]
Analyzing LLM-as-a-Judge Prior work analyzes systematic judge biases such posi- tional (Wang et al., 2023 Li et al. 2024b), length (Zeng et al.] 2024] Park et al., (024b), and self-preference (Panickssery et al. 2024 Chen al. 2025b). Prompt design, instructions, and
3

---

Published as a conference paper at ICLR 2026
scoring format strongly affect reliability (Li et al. 2024a). with pairwise judgments often reduc- ing noise and aligning better with human preferences than pointwise scores (Tripathi et al. [025] Jeong et al. 2024). Other works have emphasized the importance of carefully selecting reference an- swers Krumdick et al. 2025), linking to how generator capabilities influence the judge's inputs (Tan et al. 2025). While most studies consider static judges fixed datasets, we instead analyze judges in a dynamic setting where generators change over time, introducing response-distribus
shifts that motivate our metries for prooffn,,,,,,g backward-comee compataiiycm and question generalization.
AUTOMATIC EVALUATION AS DUAL-DISTRIBUTION PROBLEM
We propose novel formulation of the automatic evaluation problem in terms of two distributions: the question distribution and the response distribution. Concretely, let Q denote the distribution of questions Q, and let R denote the distribution of responses R. For pairwise judges, the input distribution therefore takes the form
X
R
The question distribution is defined by characteristics such as semantic content (e.g., domains like medical, legal, finance, scientific, or math) and question difficulty (e.g., difficulty can be defined by pedagogical levels, such high school vs. olympiad-leve math problems). For example, we can consider all questions in GSM8K (Cobbe et al. 2021) to come from the same question distribution, as they are all arguably of similar difficulty and semantic content. The response distribution defines the characteristics of the model responses being evaluated, such as style, capability- or model- foddel-family-speiffi quirks. We denote the training and test input distributions to be
content,
xtrann Qurein X Rtrain
R trasn and X
Qlest x R Riest
(3)
respectively. Notably, the two responses come from the same generating model, as described in the data construction details in Section 4.2 Separating the question distribution Q from the response distribution R reflects two real-world sources of shift: (1) the emergence of more capable generators (an evolving R), and (2) the introduction of new questions (an evolving Q). This decomposition allows us to isolate and quantify the impact of each factor on judge performance. In Section 5 we instantiate this framework using the weak response distribution Rweak and the strong response distribution Rytrong to simulate a model- devell-develooment timeline (older, weaker newer, stronger responses and LLMs), along with question splits Q drawn from Q that are either seen or unseen dur- ing training. Informally, weak (strong) responses drawn from LLMS with relatively low (high) accuracy on questions Q: we precisely describe generator strength in Section This instantiation enables us to investigate the four practical questions mentioned in Section | regarding the shelf life of judges. The specifics of how dual-distribution formalization supports our analysis are detailed in Section 5 with a concise connection provided in Appendix
EXPERIMENTAL SETUP
4.1 GAUGING GENERATOR STRENGTH.
We ground our study in two datasets with verifiable solutions: DeepScaleR (Luo et al. 2025) and MMLU-Pro (Wang et al., 2024c). DeepScaleR contains 40K Olympiad-style, reasoning- intensive math problems with gold answers. MMLU-Pro, by contrast, provides verifiable MCQ- style, knowledge-intensive questions spanning 14 diverse domains, including STEM, humanities, social sciences, law, business, psychology, and philosophy, enabling us to study judge shelf-life across broad range of domains. For generators. we utilize a diverse set of popular instruction- tuned models, listed in Table [ (in Appendix B). For each generator, we sample 20 responses per question and measure
strength using Pass Pass@ I captures the probability that uni- formly sampled attempt is correct and yields two clearly separated clusters, as shown in Figure 2 and Figure g (in Appendix B). where recent or larger models achieve substantially higher scores than smaller older models. Based on this distinguishable performance difference, we cluster low- and nigh-performing generators into weak (Gemma-2-99 2-99B, Qwen-2-7B, Llama-3.1-8B Ministral- 8B) and strong (Gemma-3-128 1-BB, Qwen-2.5-7B. Qwen-2.5-322 groups, respectively, and use these clusters to define our response-distribttinn shifts. Further details on generator selection and strength estimation provided in Appendix B
3.3-7... Mistral-Small-222
4

---

Published as a conference paper at ICLR 2026
Weak Generators
0.186
0.490
0.500
Strong Generators
0.419
Pass@1 Accuracy
0.4
0.259
0.231
0.243
0.2
0.173
0.1
Gevenaa 2073))
(ed (442004))
Oomec Ta Cann sooa)
(o04)
Qpnet Khop 2038
Matnal Groat baB strey
Owent ase 2004)
Genomatt Uze 2019)
Lleaaa 2-250 Der (034y)
Figure 2: Generator strength the DeepScaleR dataset, measured using pass@1 with 20 indepen- dently sampled responses. Models fall into two well-separated strength clusters: weak (0.17-0.26) and strong (0.42-0.50). No models occupy the 0.26-0.42 gap (a 0.16-wide gap), making the clus- tering robust to any threshold chosen within this interval. This clustering also aligns with model release dates, with stronger, newer models (yellow) outperforming weaker, older ones (green).
4.2 TRAINING SETUP.
Dataset Construction. To create the training and evaluation splits, we first construct pairwise input samples for the judge, following prior work (Tan et al. 2025; Wang et al.) (024b). For each question, we sample multiple responses from each generator, and each response is then labeled as "correct" or "incorrect" according to the ground-truth answer A We then form response pairs, where each pair consists of one correct response and one incorrect response, resulting in a pairwise sample with objectively correct answer. Importantly, responses in pair are drawn from single generator only. Based on the generator strengths defined above, we construct datasets of aggregated pairwise samples consisting exclusively of either weak strong responses, which we refer to as weak dataset and strong dataset, respectively.
Judge Data Distillation & Training Objectives. We train judges using three commonly adopted recipes: supervised fine-tuning (SFT) (Li et al.) 2024b Kim et al. 2024a Vu et al. (0024), direct preference optimization (DPO) (Hu et 2024;; Wang et al. 2024b), and combined SFT and DPO objective (Wang et al. 2024a Ye et al.
22044] Saad Falcon et al. 2024). As these recipes require supervision, specifically, the CoT explanation C' (Sec. 2p. we adopt the common teacher model convention (Li et al., 2024b; Wang et al. 2024a). Based on the ground-truth verdict V*, we categorize responses as correct (positive) samples y or incorrect (negative) samples y/ Positive samples are then used for SFT, whereas nositive-negative pairs are used for DPO-based recipes.
Training and Evaluation Splits. To analyze the four practical questions described in Section using the dual-distribution framework from Section 3 split the weak and strong datasets into training and test sets. For testing, we construct two distinct splits: an unseen-questions split and seen-questions split. The unseen-questions split contains questions not present during training. while the seen-questions split reuses training questions but samples new responses, with pairs constructed following the same process as described above. Unless otherwise specified, we use the unseen- questions split for evaluation. We choose three models to train: Llama-3, 1-BB, Ministral-SB, and Mistral-24B, covering range of model sizes and intrinsic strengths.
We provide more details on different aspects of the training setup in Appendix
5 EXPERIMENTAL RESULTS
In this section, we present our analysis setup and findings on future- backward- compatibility, and question generalization of judge models. Our analysis builds the dual-
5

---

Published as conference paper at ICLR 2026
10
10
Liama3. 1-8b
Ministrai-Bb
Mistral-24b
Refresh Advantage
7.0
5.9
Future Proot
4.3
2.0
1.1
-2
Ulama31-80
Ministral-ab
Mistral-24b
SFT
DPO
SFT+DPO
SFT
DPO
S
(a) FutureProoE.
(b) Refreshadvantage.
Figure 3: Future-Proofing of DeepScaler-Trained Judges. (a) Future-proofing measured by
FutureProof; negative values show degraded performance stronger responses. All models and recipes show performance degradation, indicating poor evaluation of newer, stronger responses. (b) Benefits of re-training on strong responses, measured by Re fereess SehhAvaanaage Re-training consistently improves performance, with the largest gains under DPO.
10
10
Back Compatibliity
Compatibility Shift
1.3
-2
-1.6
-2.3
-2
..7
-4.
2.3
-4
34
Ministrai-Bb
Mistral-24b
Llma3311-8
Mistral-24b
SFT
DPO
SFT+DPO
SFT
DPO
SFT+DPO
(a) BackConpatibilityy
(b) Compat toiptttttiityysitt.
Figure 4: Backward- Compatibility of DeepScaler- Trained Judges. (a) BackCompat himiiittibbiyyyy of judges trained strong responses when evaluating older responses; positive values indicate im- proved performance relative to older-judge baselines. Judges trained on newer responses show good BackCompat ibittiiiiii lity, with minimal drops performance, newer judges still face a distribution shift, reflected by with performance drops relative proofing metrics in Figure 3 backward-compatibliiy metrics are smaller, indicating that strong- response-trainee trained judges are more backward-come coopptdl than weak reak-responss-traiie trained judges are future-proof.
Backward-Compatibine
absolute gains.
(b) Despite strong absolute
ft, evaluating strong responses. (c) Compared with future-
distribution framework introduced in Section where judge inputs are factorized into a question distribution Q and a response distribution R. We instantiate the response distribution at two levels of generator strength: Ryeeak (older, less capable models) and Rstrong (newer, more capable mod- els). The question distribution Q remains fixed but varies in whether question seen unseen during training. In this way, our setup simulates model development timelines. We measure judge performance using consistent accuracy, defined in Appendix D Raw consistent accuracy scores are reported in Table Appendix D and serve as the foundation for the results below.
Notation, For clarity, we denote the consistent accuracy a judge J, trained on response distribu- tion t (J). where t {weak, strong). The subscript indicates the evaluation distribution, with e E {weak, strong). Thus, (J,, ties back to our dist--iistibution formalism: measures the accuracy of judge trained distribution / when evaluated responses from distribution

---

Published as a conference paper at ICLR 2026
5.1 How FUTURE-PROOT ARE JUDGE MODELS?
Experimental Setup. To study futuur-proofing in our simulated model development timeline, we design the following setup: weak generators serve proxies for existing LLMs, and judges trained on their responses. Strong generators represent newly released LLMs with greater capabili- ties. By autur-prooffing we refer how well weak- judges can evaluate responses from newer, stronger LLMs. Specifically, we quantify future-proodd using the following metrics:
Future?roof is defined the difference in the performance of weak-ressoose-trainee judge between the weak and strong evaluation sets:
FutureProof = AcCetrong(Jweat) - ACCweak (Aweak))(Fweak)
(4)
This measures the change in performance when the evaluation distribution shifts from R to Rabrong" i.e., weak-to-strong response distribution shift. A positive value indicates relatively better performance on strong responses, while a negative value indicates degradation; thus, higher values correspond to more future-proof judges.
RefreshAdvantage is defined as the gain from re-training judges with strong responses:
RefreshAdvantage = ACCstrong (Jsttrong(Jstrrng))) ACCstrong ( (Jweak).
(5)
This be viewed the data advantage from changing the training response distribution from Rraan Rireir to strong when evaluating on Re Higher values indicate greater henefit from re- training judges with the latest and stronger responses.
FuoureProof Findings: For all models and training recipes, we plot the FutureProof values on DeepScaleR in Figure Bal Across all settings, do observe any instance where judges gen- eralize to new or stronger responses, with all PutureProo values being negative. Interestingly, no discernible trend emerges across training recipes or model families. Generally, we find that SFT leads to greater degradations in smaller models, but smaller degradation in the large judge. In all, our results show that current judge training approaches do not produce judges capable of reliably generalizing to new, more capable model responses. Beyond lack of generalization, current judge recipes do not exhibit consistent trends across models or scales. These findings align with those on MMLU-Pro, which we discuss in more detail in Appendix F.I] In the absence of recipe-specific or model-specific trends, we recommend evaluating Proore on a model-by-man
basis.
RefreshAdvantage Findings. Our results DeepScaleR, presented in Figure 3b indicate that re-training with up-to-date responses consistently leads to performance gains. In particular, across all training recipes and backbone models, observe positive Ref reshAdva tage val- ues. Training recipes also follow a clear trend: retraining with SFT yields minimal but positive gains, whereas DPO yields the largest improvements, providing up 7.6 absolute percentage points for larger models. The SFT+DPO loss provides additional benefit over DPO alone for couple of models. We further observe that as judge model size increases. updating training data has larger impact for DPO-based approaches. For example, with DPO, Mistral-24B exhibits an absolute gain 7.6 percentage points compared to its 8B counterpart, Mistral-8B, which improves by 4.3 points. These trends are consistent with corresponding findings on MMLU-Pro, discussed further in Ap- pendix F.I Overall, we conclude that evaluating the most capable models requires training judges on their outputs; relying stale training data leaves substantial performance gains unrealized.
5.2 How NACKWARD-COMPTTIOE ARE JUDGE MODELS?
Experimental setup. Now, we extend our setup for future-proofing in Section 5.1 to study nuckwand-compattblity in simulated model development timeline. A judge trained on strong or newer generator responses represents the current judge, which is adept at evaluating new responses, while weak generators represent older LLMs with lower capabilities. By buckward-compatiblliy we refer to how well grrongrresponse trained judges can evaluate the responses of older, weaker generators. Specifically, we quantify backward backward-compant compatibiliyy using the following metrics:
backwart-compatiility
BackCompat ibility measures the performance gap when evaluating older, weaker responses with the refreshed strong-rspoonse-traineed judge instead of the reak-response-trained judge:
BackCompatibbllin ACCweak (ooiiaakKsssrae (Jstrong) Accweake Jueak).
7

---

Published as a conference paper at ICLR 2026
Llama3.1-8b
Ministral-Bb
Mistral-24b
10
10
10
FutureProof
RetreshAdvantage
8
4.3
Metric
4-
2-
-0.0
0
-2
-1.6
-2
0..8
DPO
DPO-Cont
DPO
DPO-Cont
DPO
DPO-Cont
Figure 5: Changes in future-proofing metrics when replacing weak-response- trained judge (solid) with a continually trained judge (dashed). We observe decrease in Re FrreshAdvande increase in Futuree roof, with values approaching zero for couple of models. This suggests that continual training enables judges to evaluate strong responses more effectively than judges, well strong- strong-trained judges, and adapts better the weak-to-strong response shift.
and an
Llama3.1-8b
Ministral-Bb
Mistral-24b
BackCompatiblliy
4
2.9
CompatShift
3
2
Metric
0.7
0.7
0.4
0.8
-1
-1.3
-2.0
DPO
DPO-Cont
DPO DPO Cant
DPO
pont
Figure Changes in hackward-compaiiblityy metrics when replacing a strong-repoons--rriine judge (solid) with continually-traines judge (dashed). We couple of models, suggesting that continual training can help models better evaluate weak responses than purely traiget judges. We also observe increase in Compat Shipthhhiffi ft, showing that continually trained judges adapt better to the strong- strong-to-wead te-k response shift.
increase in BackCompat for
This setting particularly important for established evaluation pipelines: if old judge replaced by new one while the task remains the same, how much does performance differ? We view this as the data disadvantage from changing training data from Rtroin Rtrain when evaluating on Rioett- positive BackCompat ibiliiy indicates that the strong-trained judge outperforms the weak-trained judge on weak responses (good backward-compatiblle while negative value reflects performance degradation (poor backward backward-compatibie
Compat iitttiiiiii AbilityShi ft quantifies the weak-to-strong distribution shift when evaluating older, weaker responses with a surrongrespooss-traanee refreshed judge. As noted in the previous section, the reverse shift sttrong-toweeld strongly affect judge performance. Here, we measure how the nut-of-distribution nature of backward-compatbblity impacts the newly trained judge:
= ACCweak (Jetrong) AcCstrong (Jstrong).
This captures the response-distribution shift opposite intureProof, i.e., from R R or strong-to weakk. It measures how far a strong-traineed judge falls below its potential under in- distribution evaluation. A positive value indicates better relative performance on weak responses, while negative value indicates degradation.
BackCompatibil iitatiibbiittty Findings. In Figure 4a
we visualize the backward compatibility of judge models trained on strong responses for the DeepScaleR dataset. When evaluating weak responses, there is little drop in absolute performance between judges trained on strong responses and those trained on in-distribution weak responses. While methods involving SFT consistently cause small performance drops, our results show that DPO training can enable newly trained judges to outper- form weak-judge models. The drop due incompatibility is smaller than the advantage gained when moving from weak to strong responses, as noted in the peefeeshAvaataac findings. This indi- cates that judges trained newer responses are indeed backward-compatbble: they closely mimic
8

---

Published as a conference paper at ICLR 2026
the performance of weak-trained judges, even in -ittr--distributiinn settings. Likewise on MMLU- Pro, strongly trained judges perform on par with or better than yeak-trained judges when evaluating weak responses. discussed in Appendix[E] F22 Thus, combined with our findings in Section 5.1 conclude that re-training with updated responses is universally beneficial: such refreshed judges are not only much better evaluating new model responses but can also serve as drop-in replacements for their older counterparts with minimal loss in performance.
Compatibi wepiliitShiff Findings. As shown above, judges trained on strong responses roughly match the performance of those trained on weak responses when evaluating older responses. Despite strong absolute performance, such newer judges are evaluating under a strong-to-Wee to-nggttoweak distribution shift; Figure plots the drop in performance due to this shift on DeepScaleR dataset. Here, we observe that across all judges and recipes, judges still experience degradation due to the out-of- distribution nature of evaluation, with the lone exception being SFI-trained Llama3. 1-8B. Surpris- ingly, here, the largest model, finetuned from Mistral-24B, experiences the largest absolute drops across all training recipes. These findings highlight that, while stronger trained judges can serve as appropriate drop-in replacements for weaker judges, distribution shift causes them to underperform relative to their potential. We see the same pattern on MMLU-Pro, where strong-trained judges also degrade under response-dee cigponse-distributiin shift, as discussed in Appendix F2 However, on DeepScaleR, compared the degradation from the weak-to-strong esspone-distributie FuturePre foof in Section 5.1). these degradations are relatively smaller. This suggests that the teak-to-stroon evaluation essponse-distriuutio shift is harder setting than strong-to-wen teak, again highlighting the importance of re-training judges on new model responses.
shift (as measured by
5.3 CAN CONTINUAL TRAINING IMPROVE PROOIIG AND DACKWARD-COMPATIBILIYY OF JUDGE MODELS?
Experimental setup. Sections 5.1 and 5.2 show that training a judge. scratch on responses from newer generators is advantageous in evaluations. An alternative is to continually update a judge originally trained on older responses by incrementally fine-tuning it on newer, stronger responses. We simulate this continual-learming paradigm by further training Jweak on responses from stronger generators, denoting the resulting model Jwenk- (details in Appendix C All experiments in this section are restricted to training judges on DeepScaler with DPO due to compute constraints.
To assess the effect of continual training, we evaluate Jweak-sstrong on both future-proofing and backward-con compaiiompiiiiilitty metrics, comparing its performance against that of the original weakly trained judge and the strongly trained judge, respectively. Specifically, we compare rutureProof and Re feefrshhäee arreshdvvantage shown in Figure 5 strong with Ju weaak-strong in Equations (7), as shown in Figure Together, these comparisons reveal how continual training helps weak judges adapt to future distribution shifts while retaining compatibility with weaker responses, relative to training from scratch.
when replacing Jweak with -strong in Equations (4)-(5, as We also compare Compat iiilitttsiiiiiiiieiiiiiiiiiitiit
and BackCompat when replacing
Changes in Future-Proofing. Figure 5 shows that continual training consistently im- proves future-proofing. FutureProof scores increase across all three models, while RefreshAdvantage decreases, approaching zero for Ministral-8B and Mistral-24B. The reduc- tion in RefreshAdvantage indicates that the benefit of retraining a strong model from scratch, relative to continual training, largely disappears when evaluating stronger responses. At the same time, the higher FutureProof scores of Jweek-strong demonstrate that continual training enables better adaptation to the weak-to-strong distribution shift than simply retaining the weak model.
Changes in Backward-Compatiilliy Figure shows mixed but informative results on backward- compatibility. BackCompati crease for Ministral-8B. Higher BackCompat iniiattiilll iiiiiiiyyyyyy indicates that a continually trained judge remains closer to the weakly trained judge when evaluating weak responses, compared to model trained solely on strong responses. We also observe a notable increase in Compat ibiiityshiiiiii ft, highlighting that continual training improves adaptation to older, weaker responses relative to purely strong- trained models. Together, these results suggest that continual training can better preserve backward wwwdd-copptiibiiity in several settings while also enhancing adaptability to distribution shifts.
scores increase for Mistral-24B and Llama-3. 1-BB but de-

---

Published as a conference paper at ICLR 2026
Llama3.1-8b
Ministral-8b
Mistral-24b
Question denerallzation
0.6
-2
-2
2.1
-2.
2.4
-6
-45
-8.9
-6
5.5
-6
3.5
-8-
-8
-10
-10 Weak
Strong
-10
-8.4
SFT
DPO DPO
SFT
DPO
SFT+DPO
SFT
DPO SFT DPO
Figure 7: Question Generalization of DeepScaler- Trained Judges. Generalization of judges trained on weak vs. strong responses to seen and unseen questions. Judges typically fail to generalize to unseen questions, showing large performance drops relative to evaluating unseen responses on seen questions.
5.4 How DO JUDGES GENERALIZE TO UNSEEN QUESTIONS AND RESPONSES?
Experimental setup. As LLMs advance, both responses and questions evolve (e.g., AIME24 vs. AIME25). We therefore examine how judges perform on previously unseen questions by sam- pling from Q in our dual-distribution framework. To quantify the benefits of question exposure during judge training, we define two evaluation splits. In the first, select subset of training questions and sample new responses for them, which we call the seen-questions, umseen-responses split. In the second, we draw questions from Qtrain that were excluded from training and pair them with new responses, defining the umseen-questions unseen-responsee split. Comparing judge per- formance across these splits reveals the performance gap due to question generalization.
QuestionGen, weak
weak = ACCweak, uneeen (Jweak) - ACCweak, (Jweak)
(8)
QuesticnGen
CCCasrong.unsees (Jstrong) ACCstrong,seen (Jatrong).
(9)
These metrics capture how well judges generalize across questions: responses are drawn from the same generator, with only the question split (seen vs. unseen during training) varied. A positive value of QuestionGen indicates better performance on unseen questions, while negative value indicates failure to generalize unseen questions.
QuestionGen Findings. As shown in Figure 7 current judge models do not generalize well to unseen questions, with nearly all judges exhibiting performance drops compared evaluating on seen questions with unseen responses. Surprisingly, we find that SFT enables the best generalization, with SFT-trained judges showing the smallest absolute drops in most cases. Mistral-24B, however, exhibits the largest drops within each training recipe, indicating poorer generalization compared to smaller models. These trends are consistent with the corresponding findings on MMLU-Pro discussed in detail in Appendix [3 Overall, experiments reveal that exposing judges to the questions they are likely to evaluate can lead to significant performance gains.
CONCLUSION
We present a dual-distribution framework for automatic evaluation and analyze four key questions surrounding finetuned LLM-as-judge models, crucial component of the LLM development cy- cle. First, we study seture-proofin and show that judges trained on older responses struggle to evaluate outputs from newer, stronger I.LMs, but re-training on newer responses yields substantial gains. Second, we examine www.dddwooppaibblii incur only minor drops, or even improvements, when evaluating older responses. Third, we demon- strate that continual learning provides more balanced adaptation to both older and newer response distributions compared to training solely on stronger or weaker responses. Finally, we investigate question generalization and find that judges experience large drops in performance questions un- seen during training. Overall, our work highlights critical challenges and actionable strategies for developing robust, future- proofe and backward-compatible judge models.
and find that judges trained on newer responses
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Francis Christiano, John Schulman, and Dan-
delion Mané. Concrete problems in ai safety. ArXiv, 0as//6066066655, 2016. URL https:
s/aplieemaatiicchoolaa... wwt. wwwp./o/ogw.....0.22.2.2.
Axolotl maintainers and contributors. Axolotl: Post-training for ai models, 2023, URL https: wwitthub.com//xo/////o
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas R. Joglckar, Jan Leike, 1lya Sutskever, Jeff Wu, and OpenAI. Weak-to-strong generalization: Eliciting strong capabilities with weak super- vision. ArXiv, abs/2312.00390, 2023. URL https://ap: eesemantiscche
org/
CorpusID:26631261
Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang. Bryan Hooi, and Bingsheng He. Judgelrm: Large reasoning models as a judge. ArXiv, abs/2504.00050, 2025a. URL https:
//api. wet.smestatiiccholae.... htt.:/ccccoaaacccooooooooaaaaaaa
Wei-Lin Chen, Zhepei Wei, Xinyu Zhu, Shi Feng, and Yu Meng. Do Ilm evaluators prefer themselves for a reason? arXiv preprint arXiv:2554403846, 2025b.
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang. Hanghang Tong, and Heng Ji. Rm-rl: Reward modeling as rea- soning. ArXiv, abs/2505.00 003387,, 2025c. URL. http:::/t///..........tt paas://app /api iemmanticseen www.aanttsccccco.....ooo.o.....e org
[oopuuSD:278327900)
Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. ArXiv, abs/2110. 14168, 2021. URL https:
//api. www.ssssaannccccollse.eo www.oo////ooooooo....oo ID:333396D
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshom, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur'elien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cris tian Cantón Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino. Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriele Synnaeve, Gabrielle Lee, Geor- gia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guanglong Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arricta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Ju-Qing Jia, Kalyan Vasuden Alwala, K. Upasani, Kate Plawiak, Keqian Li, Ken-591 neth Heafield, Kevin R. Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuen ley Chiu, Kunal Bhalla, Lauren Rantala- Kantala-Yeary, Lau- rens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat. Luke de Oliveira. Madeline Muzzi, Mahesh Pa- supuleti, Mannat Singh, Manohar Paluri, Marein Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Has- san, Naman Goyal, Narjes Torabi, Niko lay Bashlykov, Nikolay Bogoychev, Niladri S. Chatterji, Olivier Duchenne, Onur cCelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasié, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishman, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ron nie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini,

---

Published as a conference paper at ICLR 2026
Sa hana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shao- liang Nie. Sharan Narang. Sharath Chandra Raparthy, Sheng Shen, Shengye Wan, Shruti Bhos- ale. Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitmam, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Vir ginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whit ney Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yiqian Wen, Yiwen Song, Yuchen Zhang. Yue Li, Yuning Mao. Zacharie Delpierre Coudert, Zhengxu Yan, Zhengxing Chen, Zoe Papakipos. Aaditya K. Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adi Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexci Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury. Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Ben- jamin Leonhardi, Po-Yao (Bernie) Huang, Beth Loyd, Beto de Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mcjia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu. Chris Cai, Chris Tindal, Christoph Feichten- hofer, Damon Civin, Dana Beaty, Daniel Kreymer, Shang- Wen Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzm an, Frank J. Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Geor- gia Swee. Gil Halpern, Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang. Han Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Gehoski, James Kohli, Japhet Asher, Jean- Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kaixing(Kai) Wu, U KamHou, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veer- araghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang. Lailin Chen, Lakshya Garg, A Lavender, Leandro Silva, Lee Bell, Lei Zhang, Liang- peng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthias Lennic, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Mun ish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy ata Bawa, Nayan Sing- hal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Nor- man Cheng, Oleg Chemoguz. Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pe dro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollár, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Ro- driguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng, Sheng- hao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen. Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Kumar Gupta. Sung-Bae Cho, Sunny Virk, Suraj Sub- ramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xia Tang, Xiaofang Wang, Xiaojian WI, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu,
12

---

Published as a conference paper at ICLR 2026
Ye Jia, Ye Qi. Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models. ArXiv, (bs/2407.21783, 2024. URL https:/ //ppi. It ccschol ar wwtg///ooopaeeoooo TD 2:2115714:3
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang. Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. ArXiv, ubs/2305.14387, 2023. URL https:
wwwxiv.or//pa........oo... http:///a/2233...110 14387. pdf
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor-
ruptions and perturbations. In International Conference on Learning Representations, 2019. URL
https: /ettps://ppenrevee wpw.eeeeeeeeeeeeeeeeeeeeeeeeeeee net wttpr//wwwweeeeettteeemee
Xinyu Hu, Li Lin, Mingqi Gao, Xunjian Yin, and Xiaojun Wan. Themis: A reference-free nlg
evaluation language model with flexibility and interpretability. arXiv preprint arXiv:2406.18300 18365,
2024.
Hamish Ivison, Yizhong Wang. Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradcep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in changing cli- mate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:2311.10000 2023.
Hawon Jeong, chaeHun Park, Jimin Hong, and Jaegul Choo. The comparative trap: Pairwise comparisons amplifies biased preferences of I1m evaluators.
2024. URL https://api.
org. wttp::ssssssoaaaaooo/orororos.ee
Gemma Team Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Mer- hej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ram'c, e. Morgane Rivière, Louis Rouil- lard, Thomas Mesnard, Geoffrey Cideron,
Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gael Liu, Francesco Visin, Kathleen Kenealy, Lueas Beyer, Xiaohai Zhai, Anton Tsitsulin, Róbert Istvan Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Jain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Tao-sttr Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, An- dreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, Andr'as as Gyorgy, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Boxi Wu, Bobak Shahri- ari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, Cj Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Er- win Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimezak- Pluckk nska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hus- sein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, J. Michael Wicting, Jonathan Lai, Jordi Orbay, Joe Fernandez, Joshua Newlan, Junsong Ji, Jyotinder Singh. Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Wat- son, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Pi- otr Stanczyk, Pouya Deeeani Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Ardeshir Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Gir- gin, Shariq lqbal. Shashir Reddy, Shruti Sheth, Siim Pöder. Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Higer, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vladimir Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Jessica Lo. Erica Moreira, Luiz Gustavo Mar- tins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab S. Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel. Noam M. Shazeer, Oriol Vinyals, Jeffrey Dean, Demis Hassabis, Koray Kavukcuoglu, Clément Farabet, Flena Buchatskaya, Jean- Banntttt Alayrac Rohan Anil, Dmitry
13

---

Published as a conference paper at ICLR 2026
Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and L'conard Hussenot. Gemma 3 technical report. ArXiv, abs/2503.19786. 2025. URLbttps::/// /ttps://ppp.... wtt.://maaattttccccmaaeeem o.t. htt.://o......o.oo.ooora////3333
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Scongjin Shin, Sungdong Kim, James Thome, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Confer- ence on Learning Representations, 2024a. URL BeuJaTveKw
net. forum?id
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. ArXiv, fas/24055005333 2024b. URL https://app wtw.aapisssmmmanssi.h.ho.eeeeeee wtt.:/aaa..g/oooouiiiiiii 269502668
Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Kiril Gashteovski, Carolin Lawrence, J. Hockenmaier, Graham Neubig, and S. Welleck. Scaling evaluation-time compute with reasoning models as process evaluators. ArXiv, abs/2503. 2025. URL https://apl www.setttttttccooooooeoe ott.://o........ooo.oooooaaa//.. 3|t1a///w....aaaaaaaao..........
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang- Wilds: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5637-5664. PMLR, 18-24 Jul 2021. URL httpss:/pproceedingg......
html
Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. No free labels: 2025.
Limitations of llm-as-a-judge without human grounding. arXiv preprint
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yigun Liu. LIms-as-juue ju-juges:: A comprehensive survey on Ilm-based evaluation methods. ArXiv, abs/2412.05579, 2024a. URL hnttps://pp. deeaantcsccog www.:ccchhhhhoooo.ocoooeooo http://a.a.rrrrrooooeooeeeeeeee wowg/oorrrrrss 274596907
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, hai zhao, and Pengfei Liu. Generative judge for evaluating aligmment. In The Twelfth International Conference on Learning Representations, 2024b. URL httpttt::///eeeeeeeeeeeeee wttp://wwwweeeeeeourreeaaeeaeeee wttee//orrrm..dggt.eess/s
Yang Liu, Dan Iter, Yichong Xu, Shuo Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nig eval- uation using gpt-4 with better human alignment. In Conference on Empirical Methods in Natural Language Processing, 2023. htLt:: http://aaxxxx....//d....o.e
org. wttg 108.00..66660 16634.666.. pdd
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang. Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol ol-preview with 1.5b model by scaling rl, 2025. Notion Blog.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- ral Information Processing Systems, 2022. URL https:/ http///pppeeeeeeeeeeeeeee net forum?? i TG8KACxEON
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In The eight Anmual Conference on Neural Information Processing Systems, 2024. URL
eew wwt. wtt.r//eeeeeooooooooooooooo/o///
14

---

Published as a conference paper at ICLR 2026
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias:
Leveraging debiased data for tuning evaluators. ArXiv, abs/2407.06551, 2024a. URL https:
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Lever-
aging debiased data for tuning evaluators. arXiv preprint urXiv:2407.0655/ 2024b.
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of op- tions in multiple-choice questions. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2006-2017, Mex- ico City, Mexico, June 2024. Association for Computational Linguistiis. doi: 10.18653/v1/2024. indings-naach 130. URL httos: battoss//aclaactoooooyy.c///2222 onnp://rrorooe....nnnnnaaaeaaaaa 130/
Vyas Raina, Adian Liusie, and Mark Gales. Is llm-as-a-judge robust? investigating universal adversarial attacks zero-shot I1m assessment. ArXiv, abs/2402. 1401/ 2024. URL https:
//api. www.ssseaaatiicolaa..m htt.aaaccc.aaa..oooooooaaaaa1a11
Gemma Team Morgane Riviere Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, L' L'eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram'e, e, Johan Fer- ret, Peter Liu, Pouya Dehghami Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Ku- mar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stańczyk, Ser- tan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Alanna Walton, Aliaksei Severyn, Alicia Parrish. Aliya Ahmad, Allen Hutchison, Alvin Abdagic. Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, An- tonia Paterson, Ben Bastian, Bilal Piot, Boxi Wu, Brandon Royal, Charlie Chen, Chintu Ku- mar, Chris Perry, Christoper A. Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozi 'nska, D. Herbison, Elisa Bandy, Emma Wang. Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak-Pluei nska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stan- way, Jetha Chan, Jin Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost R. van Amersfoort, Josh Gordon, Josh Lipschultz, Joshua Newlan, Junsong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Ki- ranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, L.. Sifre, Lena Heuermann, Leti cia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Mar- tins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Gorner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng. Minwoo Park, Mofi Rahman, Mohit Khat- wani, Natalie Dao, Nen shad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Peng chong Jin, Petko Georgiev, Phil Culliton. Pradcep Kuppala, Ramona Comanescu, Ramona Merhej. Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samanch Saadat, Sara Mc Carthy, Sarah Perrin, Sébastien M. R. Arnold, Se bastian Krause, Shengyang Dai, Shruti Garg. Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomás Kociský, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmad- hikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani. Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeffrey Dean, Demis Has- sabis, Koray Kavukcuoglu, Clément Farabet, Flena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size. ArXiv, abs/2408.001 1010080001188 18, 2024. URL http://appp..eemnnccccccc....aa. www.ecchh.aaa..ooooooooussa.eeae http:///sss....e.......oooooooop
Jon Saad-Falcon, Rajan Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri. Lmunit: Fine-grained evaluation with natural language unit tests. arXiv preprint arXiv:2412. 2024.
Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang. Zifeng Wang, Sayna Ebrahimi, and Hao Wang. Continual leamnnng of large language models: A comprehensive
15

---

Published as a conference paper at ICLR 2026
survey, ACM Compur. Suni, May 2025. ISSN 0360-0300. doi: 10.1145/3735633. URL https: / //doi.org//ii 1ww..3135533551 Just Accepted.
Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai-Wei Wei Chang, Marcus Rohrbach, and Anna Rohrbach. When to solve, when verify: Compute-optimal problem solving and generative verification for Ilm reasoning. ArXiv, abs/2504.01005,, 2025. URL httttt/////pt ssematiccchollee www.:aar.....gooosssssus........
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feed- back. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3008-3021. Curran Associates, Inc., 2020. URL.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:// openreview.no net forum? hwwpmmmmmdqwwhhh2332
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9275-9299 Online, November 2020. Association for Computational Linguis- antha org/202..
tics. doi: 10.186633/112222emmmllp. 746. URL/https:
emm.p ma iiain.... 7
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica. Judgebench: A benchmark for evaluating LLM-based judges. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreview.nett dF-.ormmmmmocCOoooascesc
Mistral Team. Un ministral, des ministraux, a.
mini Aaristraus
Mistral Team. Mistral small 3, b. URLLEttps///isttra htt.://a....aaeee////sssttaaaaaa
Tuhina Tripathi, Manya Wadhwa, Greg Durrett, and Scott Niekum. Pairwise or pointwise? evalu- ating feedback protocols for bias in LLMe-ased evaluation. In Second Conference on Language Modeling, 2025. URL
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuar Sung- Foundational autoraters: Taming large language models for better automatic evaluation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17086 17105, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 111116635122244.mmg 949. URL hwtphtttt//omm / www.aaaahooooooooooooooooooo org 2 2024 emmlp ma i in. 949/
main.
Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafiq Joty. Direct judgement prefer- ence optimization. arXiv preprint arXiv:2409. 14664, 2024a.
Pciyi Wang. Lei Li, Liang Chen, Dawei Zhu Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. ArXiv, abs/2305. 17926, 2023. URL. https://aas /api. www.emettttio wis.monicccchoarr..... org/ Ctrg//orree Corpus 258960339
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yis Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi,, Jason Weston and Xian Li. Self-taught eval- uators. arXiv preprint arXiv:2408. 02.4000.666,,.... 2024b.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang. Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task
16

---

Published as a conference paper at ICLR 2026
language understanding benchmark. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Pa- quet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 95290. Curran Associates, Inc., 2024c. doi: 10.52202/079011-33018
URL httpps:/pproceedings. neur htt.a//cww...c........ococ....aa
Track.
Chenxi Whitehouse, Tianlu Wang. Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha. J1: Incentivizing thinking llm-as-a-judge via reinforcement learning. arXiv preprint arXiv:2505. 10320, 2025.
Austin Xu, Srijan Bansal, Yifei Ming, Semih Yavuz, and Shafiq Joty. Does context matter? contextualjudgeennch for evaluating Ilm-based judges in contextual settings. arXiv preprint arXiv:2503.15620, 2025a.
Austin Xu, Yilun Zhou, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. J4r: Learning to judge with equivalent initial state group relative policy optimization. ArXiv, abs/2505. 13346, 2025b.
URL httpt: www.nticschooiioooeoo..ec.
org/ Co:pp/sIppppp 778650
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Yang Chen, Kexin Yang, Mei Li, Min Xue, Na Ni. Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xi- aodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yunyang Wan, Yunfei Chu, Zeyu Cui, Zhenru Zhang, and Zhi-Wei Fan. Qwen2 technical report. ArXiv, abs/2407.10671, 2024a. URL https:
//api. www.ssseaaticccccoccoo wtw.scchhlaa..gooooououuoo wrg/Corpoooooooooooooooo 27:121233
Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report. ArXiv, abs/2412.15 15115, 2024b. URL https:// //ap:::///p0 semanti cscho ar oot.ao..to/oooouttt:::22 17:78559421
Ziyi Ye, Xiangsheng Li, Qiuchi Li, Qingyao Ai, Yujia Zhou, Wei Shen, Dong Yan, and Yiqun Liu. Beyond scalar reward model: Learning generative judge from preference data. arXiv preprint arXiv:2410.03742. 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason E. Weston. Self-rewarding language models. ArXiv, abs/2401.100020. 2024. URL https:// //
arxiv. wwxiv.org/pdf/2440 /24011 10020.pco
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https:://pperrvieee net htt./oorrmmmidc
tr dwwLc
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang. Joseph E. Gonzalez, and lon Stoica. Judging
with MT-bench and chatbot arena. In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: //openrevie ew. met./forume forum? /ttp://ormmmm/cccccPPGGG. ddwcccHPGDccccccccccocc
Yilun Zhou, Austin Xu, PeiFeng Wang. Caiming Xiong, and Shafiq Joty. Evaluating judges as evaluators: The jetts benchmark of llm-as lldges-judges as test-time scaling evaluators. ArXiv,
Abs/2504.112533 2025. URL https: : / /ppp.
/
semanticscholat httpaaaiiissccoaoooooo//rrpsppas wrq/Corpusdd:::
277955867
17

---

Published as a conference paper at ICLR 2026
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeLM: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations, 2025. URL http://pwwwweteeee/eeeeeeeeeeeee http://ooruuutlocsssssslaaa
18

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
LLM Usage
20
Generators and Generator Strengths
20
Training Setup Details
21
Consistent Accuracy and Judge's Performance Across Splits
23
Research Questions in the Dual-Distributoon Formulation
23
Detailed Findings from MMLU-Pro Dataset
24
How future-proof are judge models?
25
F2
How backward are judge models?
26
How do judges generalize to unseen questions and responses?
................................
26
Prompts and Sampling Hyperparameters
26
19

---

Published as a conference paper at ICLR 2026
0.7
Weak Gonerstors
Strong Generators
0.650
0.090
0.691
Pass@1 Accuracy
0.561
0.405
0.422
0.4
0.2
0.1
Mirestakele 20241
Gemmazzoo di 20244)
Oreent 7B 2024
Liama 1480 2024)
Oheng 5-70 20241
Gemmat 3-128 2025)
Hiarral samall 2448 taan 3025)
(Gee 2034)
(Dic 2024)
3-708
Figure 8: Generator strength the MMLU-Pro dataset, measured using pass with 20 inde- pendently sampled responses. The same two-tier structure emerges: weak (0.34-0.43) vs. strong (0.56-0.69), with no models in the intermediate 0.43-0.56 range (a 0.13-wide gap). This clustering aligns with the weak-strong separation observed in DeepScaleR (Figure 2 suggesting that the pat- tern reflects underlying model strength rather than threshold choice or dataset-specific artifacts.
A
LLM USAGE
Other than being used as part the experiments conducted in this work, LLMs were used solely writing assistance tool in preparing this paper submission. Their role limited to polishing language, improving clarity, and reducing redundancy. The prompt used for this purpose was similar to "Please revise the writing of this, making ideas, experimental designs, analyses, and claims presented the paper are entirely the original work of the authors. No part of the conceptual, methodological, empirical contributions relies or originates from LLM outputs.
remove any grammatical mistakes." All research
B GENERATORS AND GENERATOR STRENGTHS
Shorthand
Full Hugging Face Identifier
Llama3.3-70B
met aalaammmllmaae
3-
JJama3.1-8B
meta [amma amm.aammmm..
Qwen2-7B
Owen Owwn2-7B-Ins- 7B
Qwen2.5-7B
Owen
Qwen2.5- 14B 14B
Owen/ 2wtp://222--11---t-----
Qwen2.5-32B
Gemma2-99
google boogle/genmase 9eoogl//enaaa22bb-11
Gemma3-12B
goog e -
Ministral-8B
mi st ra 1istralai/Miissrr
Wistral-Small-224
mi mistralai/Miss 1
Sma
DeepScaleR
agent oel.:
MMLU-Pro
TIGER-Lab/M010--1G
Table 1: Mapping from shorthand model and dataset names to their corresponding Hugging Face identifiers.
To curate generator responses, we begin with a set of candidate generators and a collection of ques- tions Q, along with verifiable ground-truth answers A For each question, sample 20 responses from each generator using temperature sampling and compute a pass@ score. This score represents the probability of obtaining least correct solution when randomly selecting solution from
20

---

Published as a conference paper at ICLR 2026
Judge Backbone LLM
Weak Response Dataset
Strong Response Dataset
Ministral-8B, Mistral-Sman Smalll--lll--
Gemma2-9B Qwen2-7B, Llama3.1-8B
Qwen2.5-7B. Gemma3-12B. Llama3.3-70B
Llama3.1-80
Gemma2-99 9Bnmma2-9B, Qwen2-7B,
Qwen2.5-7B. Gemma3- --2B, Mistral-Small-24B
Ministral-8B
Table 2: Overview of training data composition on a per-backbomm LLM basis. To mitigate bias from the difficulty of evaluating self- self-generated responses, we avoid training judge models on their own responses. This produces per-judge training datasets composed of different generator responses.
the 20 samplings, where correctness is determined by matching the generator's responses against A
Concretely, we use two verifiable datasets, as shown in Table [ DeepScaleR (Luo et al. 2025 and MMLU-Pro (Wang et al. 2024c). DeepScaleR contains 40K challenging Olympiad-level math problems spanning multiple years, each paired with a ground-truth answer. In contrast, MMLU-Pro contains 12K multiple-chon choice problems, spanning 57 subjects across 14 categories and drawing from diverse sources such as MMLU, STEM websites, TheoremQA, and SciBench. We include MMLU- Pro to demonstrate the broader applicability of our results beyond mathematics. For experiments, we use popular open-source instucction-tune models, as listed in Table [ Gemma-2-9B (Riviere) cet al. 2024). Gemma-3-122 (Kamath et al., 2025), Llama-3, 3, 1-8B. Llama-3.3-70B Dubey et al. (0044) Ministral-8B (Team aa). and Mistral-Small-22 (Team, b). Qwen2-7B (Yang et al. 2024a), Qwen2.5-7B (Yang et al. 2024b). and Qwen2.5-32B (Yang et al. 20246).
Figure 2 which plots the pass@1 scores of all candidate generators, reveals two clearly separated capability bands on DeepScaleR. Weak models fall in the 0.17-0.26 range, whereas strong models fall in the 0.52-0550 range, leaving 0.16- empty gap (0.26-0.43) with model in the in- termediate region. Thus, any threshold chosen within this interval produces the same weak-strong grouping- This separation also aligns well with model release dates, as shown in Figure 2 We ob- serve the same two-tier pattern on MMLU-Pro, as shown in Figure g weak models score 0.34-0.43, while strong models score 0.56-0.69, again with no models occupying the 0.43-0.56 interval (a 1.1d-wide gap). The alignment of weak- strong groups across two very different datasets indicates that the distinction captures genuine differences in underlying model strength, rather than artifacts of particular dataset or threshold choice.
C TRAINING SETUP DETAILS
Dataset Construction. To create the training and evaluation splits, we first construct pairwise in- put samples for the judge, following prior work (Tan et al., 2025 Wang et al., 2024b). For each question, we sample multiple responses from each generator, and each response is then labeled as "correct" or "incorrect" according to the ground-truth answer A We then form response pairs, where each pair consists of one correct response and one incorrect response, resulting in a pairwise sample with an objectively correct answer. Importantly, responses in a pair are drawn from a sin- gle generator only. This choice ensures that the judge learns distinguish correctness based on reasoning quality rather than relying on stylistic differences between models, which could occur if responses from different generators were mixed in a single pair. For each generator and question, we only keep samples where there is at least one correct and one incorrect response and if this condition not met, the question is discarded for that generator. In Table we report the percentage of ques- tions retained for each generator after applying this discarding criterion. Further, in Figure 9 we show that weak models discard many hard questions because all 20 samples are incorrect, whereas strong models discard many easy questions because all 20 samples are correct. Mid-tier models re- tain the most questions because they more frequently produce mixed outcomes, resulting in a clear U-shaped trend in the rank-retention plot. Thus, the retained subset is enriched for borderline ques- tions near each model's decision boundary, naturally inducing a medium-difficulty difficulty selection bias. Following this, and based on the generator strengths defined in Appendix B, we construct aggre- gated pairwise datasets consisting exclusively of either weak or strong responses, which we refer to the weak dataset and strong dataset, respectively.
21

---

Published as a conference paper at ICLR 2026
Generator
DeepScaleR Ret Ret
MML.U-rr Ret Krank
Gemma-2-9B
36.11
63.83
Gemma-3-12B
57.2
49.236
Qwen-2-7B
52.23
79.31
Qwen-2.5-7B
63.75
58.755
Qwen-2.5-32B
62.97
43.61s
Llama-3. 1-88B
52.22
76.384
1lama-3.3-708
47.19
34.29g
Ministral-8B Mistral-Smaall-24B
62.54
62.321
64.66
50.737
Table 3: Retention percentage (Ret%) across DeepScaleR and MMLU-Pro for various generators. The subscript rank denotes each model's Pass rank; higher ranks correspond to models with perior performance, as shown in Figure and Figure [ Retention measures the fraction of questions where generator produces both correct and an incorrect sample across 20 generations.
80
DeepScaleR MMLU-Pro
Retention (%)
70
40
4
5
Pass@1 1 Rank
Figure 9: Retention vs. Pass@1 rank, derived from Table 3 Weak generators drop hard questions be- cause all sampled responses incorrect, whereas strong generators drop easy questions because all sampled responses are correct. Mid-tier generators retain the most questions by producing responses with mixed correctness. As result, the retained concentrates borderline, medium-difficulty items near each generator's decision boundary.
Judge Data Distillation & Training Objectives. We train judges using three commonly adopted recipes: supervised fine-tuning (SFT) (Li et al.) 2024b Kim et al.) 2024a] Vu et al.] (2024). direct preference optimization (DPO) (Hu et al. 2024 Wang et al. 2024b). and combined SFT and DPO objective (Wang et al., 2024a Ye et al. 2024 Saad-Falcon et al. 2024 As these recipes require supervision, specifically. the CoT explanation (Sec. (2) adopt the common teacher model convention (Li et al. 2024b Wang et al. (024a). We prompt GPT-4o with (Q, R1. R₂) inputs, sampling multiple responses (C, V) per input. Based on the ground-truth verdict V*, categorize responses as correct (positive) samples y or incorrect (negative) samples y We only keep inputs for which at least one y* and y exists. This ensures that the inputs exactly comparable for SFT and DPO, Positive samples are then used for SFT, whereas positive-negativy pairs are used for DPO-based recipes.
Train and Evaluation Splits. To analyze the four practical questions described in Section I Jusing the dual-distribution framework from Section 3 we split the weak and strong datasets into training and test splits. For testing, we construct two distinct splits: an unseen-questions split and a seen- questions split. The unseen-questions split contains questions not present during training, while the aeen-questions split reuses training questions but samples new responses, with pairs constructed following the same process deseribed above. Unless otherwise specified, questions split for evaluation. Note that the corresponding weak and strong splits use exactly the same of questions; we remove any question that appears in only split. This prevents question-
use the unseen-
22

---

Published as a conference paper at ICLR 2026
difficulty differences from confounding our findings. Overall, for DeepScaleR each training split contains 70K samples, whereas for MMLU-Pro each contains 10K samples. For both datasets, each evaluation split includes 2.5K response- caaeeee--rddee---ppppeee samples (5K after response- order flips).
Generator and Judge Backbone Details. We choose three backbone models to finetune: Llama- 3.1-8B, Ministral-&B. and Mistral-24B, covering a range of model sizes and intrinsic strengths. Prior work (Tan et al. 2025) has shown that models often struggle to judge the correctness of pairs of their own sampled responses. Another line of work (Chen et al., 2025b; Panickssery et al. (024) has shown that models can recognize their own responses and exhibit self-bias. Thus, to disentangle any effects of training a judge on self-generated responses, we exclude the backbone judge model from serving as a generator. Specifically, we create two training sets (each with weak and strong splits), ensuring that the backbone judge model not included the list of generators. We summarize these training sets and the associated backbone models in Table
Hyperparameters. All experiments with SFT, DPO, SFT+ are implemented using the Ax- OLOTL framework Axolotl maintainers and contributors (2023). For SFT, we sweep learning rates in (1 x 10 6 2.5 x 10-o, ; 5 x 10-0, x 10-b) with a cosine decay scheduler. Across all eval- uation splits, learning rate of 2.5 x 10 consistently yields the best performance. For DPO, we adopt standard hyperparameter choices from prior work Ivison et al. 2023), using learning rate of 5 x 10 and a preference strength parameter 8 0.1. For SFT+ equal weighting between the SFT and DPO objectives, using the same DPO hyperparameters (learn- ing rate 5 x 10-7, 8 0.1). DeepScaler weak and strong judges are trained for 3 epochs (2,800 gradient steps). In contrast, MMLU-Pro weak and strong judges are trained for 10 epochs (1,500 gradient steps). For continual training experiments (section (.3). we start from weak-response DPO-trained judge (trained for 3 epochs) and further train it on strong responses for I additional epoch, amounting to roughly 1,000 additional gradient steps. We sweep 3 E {0.1, 1.0} and report results in the main text using 3 = 1.0; additional results are included in Table and in Appendix
optimize a joint loss with
D CONSISTENT ACCURACY AND JUDGE'S 'S PERFORMANCE ACROSS SPLITS
Consistent Accuracy. Since judge models are prone to positional biases (Wang et al. 2023 ei al, 2024b Xu et al. 2025b) their preference shifts depending on whether Ri R2 appears first in the prompt-in it is standard practice to evaluate judges using both response orderings (Tan et al.) 2025 Xu et al. 2025a b). Concretely, for input
(Q, R1, R2), let Z denote the same sample, but with response order flipped in the input prompt. i.c., 2 = (Q, R2, R1). Then, evaluation with consistent accuracy considers the judge correct only if it correctly identifies the better response under both orderings:
Acc-
-
(10)
[P|
PEP
where [-|] is the indicator function, P is the evaluation set consisting of pairs (z,, (x)),, and the judge's verdicts (z) are compared against the ground-truth verdicts V*(x).
Judge's ' Performance. We report all consistent- -cccuracy scores of our trained judges for both Deep- Scaler and MMLU-Pro across the different evaluation splits in Table
E RESEARCH QUESTIONS IN THE DUAL-DISTRIBUTION FORMULATION
As described in Section the dual-distribution formulation separates the question distribution Q from the response distribution R. reflecting two real-world sources of shift: (1) more capable gen- erators (an evolving R) and (2) new questions (an evolving Q). This decomposition allows to isolate and quantify the impact of each factor on judge performance. Building on this, we investigate several practical questions about the shelf life of trained judges, focusing four distinct settings:
How anture-prooff are judge models? For a judge to be future-proof, it must be able to evaluate responses from newer, stronger models. To study this, we examine how a judge trained responses from the current generation of weak models performs when evaluating responses from strong mod- els. Specifically, train judge R treat and evaluate on both Rleat and Reest This setup strong
23

---

Published as a conference paper at ICLR 2026
Ministral-B8 Mistral:22 24b
Refresh Advantage
Future Proof
3.2
2
2.0
2.3
-1.0
0.7
1.11
Ministral-Bb
Mistral-24b
DPO
SFT+DPO
DPO
SFT+DPO
(a) FutureProof.
(b) RefreshAdvantage.
Future-proofing of -MMUUPPP Pro-trri-ed judges. (a) Future-proofing measured by Future?roof: negative values indicate degraded performance on stronger responses. All models and training recipes degrade, reflecting poor evaluation of newer, stronger responses. (b) Benefits of re-training on strong responses, measured by Re Refrest aeefreshAvantaan improves performance, with larger gains under DPO-based recipes. These results largely follow the trends observed on DeepScaler (Figure[IO) except with smaller absolute magnitudes, indicating that response-disrrbution shift can depend the domain.
Figure 10:
Re-training consistently
characterizes how robust judges are to distribution shift from weak to strong responses. Addition- ally, we quantify gains from retraining a judge on strong responses by replacing training data from R Rrain with responses from R frainn
strong"
How backward-compatible are judge models? Newly trained judges are fine-tuned evaluate newer, stronger geooonse-generatingg models. However, does this focus state-of-the-ar genera- tors come at the expense of performance on older, more established generators? To complement future-proofing experiments, we examine backward-compatiiilty Specifically, given judge trained on responses from Rtrain strongg we ask: how well does match a judge trained on weaker responses from Rrainn when both are evaluating Risst, responses? Beyond this comparison, eval- uating weaker responses with a judge trained on strong responses also introduces distribution shift from strong to weak responses. We quantify any performance losses that result from this shift.
Can continual learning improve future-proofing and backwarr-compptiiblity of judge models? Rather than training a new judge from scratch Ratrongs we start with judge trained on R and continually fine-tune it on Rerein to obtain a continually trained judge. In parallel to the Reest settings above, we ask whether the continually trained judge narrows the gap on Ratrong relative to trained only Rtrain and whether it retains performance Reea relative a judge trained from scratch on Ratrong: This setup tests whether continual training helps weak judge adapt the weak to strong response shift while preserving compatibility with older responses.
How do judges generalize across unseen questions? As questions introduced for evalu- ating LLMs, judge models must accurately assess responses to these questions. Here, we quantify the benefit of judge model having seen question during training. To study this form of gener- alization, we construct two evaluation splits. The first is a seen-questions, umseen-responses split, created by selecting questions that appeared in the training set and sampling new set of responses for these questions from train, The second is an unseen-questions, umseen-responses split, gen- erated by sampling questions from Qrain that were not included in the training data, along with their corresponding responses from Rtaai Comparing performance across these splits enables us assess how well judges generalize previously seen questions versus entirely new ones.
F DETAILED FINDINGS FROM MMLU-PRE DATASET
In this section, we present the future-prut proofing, backward cookwarr-ommpaaibiiityy.. and caaatiionngeeealizattoon results for the MMLU-Pro dataset and place them in context with the corresponding findings on DeepScaleR. As discussed in Section 5 the overall trends MMLUUPro closely match those ob-
24

---

Published as a conference paper at ICLR 2026
Back Compatibiltty
Compatibiltyy Shift
2-
1.3
1.2
1.4
2
-0.8
4..9
Ministral-Bb Rb
Mistral-24b
Ministral-Bb
Mistral-24b
-2.0
DPO
SFO
DPO
SFT+DPO
(a) BackConpatibility.
(b) Compat
Figure 11: Bhccward-Coopatibe of MMLU-Pro-Trainee Judges. (a) BackCompatibil measures how well judges trained on strong responses evaluate older responses; positive values indicate improvements over weak-judge baselines. gerong-traince judges show clear gains, larger than those on DeepScaler Figure 4a suggesting that strong judges are as good as or better than (b) Despite strong absolute performance, judges still face distribution shift, reflected in Compat iemeatibilin wwwlityShiiiiiiiei fi,, which captures perfor- mance drops relative evaluating strong responses. These shifts similar those observed on DeepScaler Figure 4b
weak judges when evaluating weak responses.
Ministral-8b
Mistral-24b
Question Generalization
-1.3
-2
2
7.2
-3
33
3.8
-5
Weak
Strong
-5
SFT DPO
SFT+DPO
SFT
DPO
SFT+DPO
Question Generalization of MMLU-Pro Trained Judges. Generalization of judges trained on weak and strong responses to seen and unseen questions. Judges consistently fail to generalize to unseen questions, showing large performance drops relative their performance seen questions. These trends align with our findings on DeepScaler dataset in Figure 7
Figure 12:
served DeepScaler. However, compared to DeepScaleR, which is math-oriented and reasoning- intensive. MMLU-Pro exhibits noticeably smaller degradations across all metrics. Since MMLU- Pro is more knowledge-centered, this suggests that the severity of response-disrrbaation shift is do- main dependent. These observations imply that a judge model's shelf-life metrics can vary mean- ingfully with task domain, even when the training recipe and backbone model are held constant. Below, we describe the results for each metric on MMLU-Pro.
F.1 How FUTURE- PROOF ARE JUDGE MODELS?
FutureProof Findings. Figure 10a reports the FutureProof values for all models and train- ing recipes. Consistent with DeepScaleR, we do not observe any case where judges generalize to newer stronger responses: all PutureProof values are negative. However, the magnitudes on MMLU-Pro are noticeably smaller than those on DeepScaleR (see Figure Ba). suggesting that degra- dation under response-distree distributtniii shift less severe on knowledge-oriented, non-math tasks than reasoning-intensive math-olympiad problems. This highlights that the extent of future-proofin failure can vary by domain.
RefreshAdvantagee Findings. As shown in Figure 10b re-training up-to-date responses con- sistently improves evaluation performance: all training recipes and backbone models exhibit positive RefreshAdvanta age values. Mirroring DeepScaleR, -Poed recipes yield larger gains than SFT alone, and the benefits grow with judge model size. For instance, under SFT+DPO, Mistral-24B gains 4.4 absolute points, compared to 3.2 points for its 8B counterpart, Mistral-8B. Overall, these
25

---

Published as a conference paper at ICLR 2026
results reinforce the DeepScaleR observation that reliably evaluating stronger generators requires judges trained on strong, contemporary responses. However, the gains on MMLU-Pro are slightly weaker compared to DeepScaleR, again indicating that the absolute advantage from refreshing is domain-dependent.
F.2
HOW BACKWARD-COMPATTILE ARE JUDGE MODELS?
BackCompatibility Findings. In Figure Ila visualize the buckwarr-compattbility of judge models trained on strong responses. When evaluated weak responses, these judges show little improvement over judges trained directly on in-distribution weak responses. In com- parison, DeepScaleR (see Section 5.2). observed minimal performance drop, with the backCompatibilit, metric slightly negative. Taken together, these results indicate that judges trained newer, stronger responses indeed backward-compatible: they perform par with weak-trained judges, even when evaluated out of distribution.
Findings. Our findings in BackCompat ibility show that judges trained on strong responses perform comparably to, or better than, weak trained judges when scoring older responses. However, these stronger judges evaluated under strong-to-weak distribution mismatch, and Figure 11b illustrates the resulting drop in accuracy. These drops are consistent across models and training recipes. Thus, even though stronger judges can effectively replace weaker ones, distribution shift still limits their realized performance. We observed a similar pattern on the DeepScaleR dataset, as discussed Section 5.2
CompatibilityShiie
F.3
How DO JUDGES GENERALIZE TO UNSEEN QUESTIONS AND RESPONSES?
QuestionGen Findings. From Figure 12 we observe that judges trained on MMLU-Pro do not generalize well to unseen questions, with nearly all judges showing performance drops compared to evaluating questions with unseen responses. The trends similar those for DeepScaleR in Figure 7 more performant judges using the DPO recipe and larger backbones such as Mistral-24B exhibit larger drops when evaluated in-distribution questions encountered during training.
G PROMPTS AND SAMPLING HYPERPARAMETERS
To obtain generator responses, sample multiple completions from each generator in order to better capture the diversity of its reasoning behaviors. We use five temperature-samplin config- urations, where n denotes the number of sampled completions and the sampling temperature: t=00), (n=4,t=0.41 (n=5, t=0.5), (n=5,t=0.6) and (n=5, t=0.7), with top-p fixed at 1.0. This yields 20 total responses per question for each generator model.
To reduce prompt-formal bias and further increase response diversity, we randomly select one of four generator prompt templates (Prompts [[4] for each sampled completion in the DeepScaleR dataset. For multi-domain experiments using MML.UPrr, we use the prompt in Prompt 5
For the judge models, we provide the original question along with two generator responses, each containing both the intermediate reasoning and the final numerical answer. Judge models are de- coded greedily using (===1(=0.0), as we found pass@k judge accuracies to be highly correlated with greedy decoding while being more computationally efficient.
We use the prompt Prompt for judge evaluation.
26

---

Published as a conference paper at ICLR 2026
Generator Prompt Template DeepScaleR
Instruction:
Solve the following math problem step by step. The last 1ine of your response should be:
Answer: $Answer
where $Answer is the final answer.
Problem:
[(problem)]
Output Format:
Answer: syour answer here>
Prompt 1: Generator prompt template used in DeepScaleR for structured, step-by-step solutions.
Generator Prompt Template 2 - DeepScaleR
Instruction:
Solve the following math problem efficiently and clearly.
- For simple problems (2 steps or fewer) give a concise sclution with minimal explanation.
For complex problems: use the structured step-by-step format: Step 1: [Concise description] [Explanation calculations]
Step 2: [Concise description] [Explanation calculations]
Important:
Always conclude with:
Therefore, the final answer is: Sbooxed (answer) where answer the final numeric answer.
Problem:
Problem:
((problem))
Generator prompt template used in DeepScaleR that adapts to problem complexity, producing either concise explanations or multi-step structured reasoning.
Prompt 2:
Generator Prompt Template 3 DeepScaleR
Instruction:
Read the problem, reason through it, and provide a final answer.
Problem:
((problem))
Output Requirement:
Your response must end with:
The final answer is [answer]
where [answer] is the final computed answer.
Prompt 3: Generator prompt template that prompts models to reason and explicitly report a final answer.
Generator Prompt Template 4 DeepScaleR
Problem: ((problem}}
Prompt 4: A minimal generator prompt template presenting only the raw problem.
27

---

Published as a conference paper at ICLR 2026
Generator Prompt Template - MMLU-Pro
Instruction:
You are given a multiple-choice question from the domain of {[domain)).
Each answer option corresponds to a lettered choice. Question:
([question))
Options:
[[options)}
Task:
Provide careful, step-by-step analysis of the question. Use your reasoning to evaluate all relevant information and identify the correct option. After completing your reasoning, produce your final choice in the following format:
<answer>x</answer>
where X is the letter of the correct option.
5: 5: Prompt template used for reasoning over multiple-choice questions in MMLU-Pro.
Judge Prompt Template
Task:
You are a rigorous evaluator comparing two responses to the same math question. Judge which response is better, based solely on logical soundness and correctness.
You given:
Question
Response A
Response B
Evaluation Guidelines:
Correctness is top priority. Prefer responses with correct
reasoning and correct final answers.
2. If both have reasoning flaws, choose the one that still reaches the correct final answer.
3. Ignore style, length, formatting, verbosity, fluency.
Output Format (JSON)
Your final output must be exactly one of the following:
{ ("verdict": "A")
("verdict": "B"]
Question:
[[question]] Response A:
[[response.a]]
Response B:
((response_bl) b}}
Prompt 6: Judge prompt template used compare two generator responses based on final-answer correctness and reasoning quality.
28

---

Published as a conference paper at ICLR 2026
DeepScaler
MMLU-Pro
Train
Eval
0-Shot
SFT
DPO
SFT+DPO
0-Shot
SFT
DPO SF
Llama3.1-8B
Wk. Sa
32.44
48.14
43.95
63.40 40
St.
28.41
44.66
36.62
60.48
Jwk
Wk. Un
30.79
46.06
39.41
58.47
St, Un
27.76
41.91
33.94
55.29
Wk,
32.44
45.33
42.53
61.72
St. Sn
28.41
46.41
43.74
65.15
Js:
Wk. Un
30.79
44.12
39.61
57.60
St, Un
27.76
42.21
40.91
59.27
Wk. Sn
32.44
44.69
St. Sn
28.41
41.19
St
Wk, Un
30.79
40.09
St, Un
27.76
38.41
Wk. Sn
32.44
45.43
JN Sk--St
St, Sn
28.41
39.13
Wk, Un
30.79
40.07
St. Un
27.76
37.22
Ministral-8B
Wk, Sn
33.87
48.06
61.04
61.39
26.87
34.86
47.24
47.81
St. Sn
28.72
41.94
55.55
56.41
27.14
34.04
46.38
47.08
Jwn
Wk, Un
33.81
45.93
56.41
56.72
27.05
33.72
44.54
45.03
St, Un
29.14
41.91
54.86
53.26
25.74
33.62
43.56
43.96
Wk, Sn
33.87
45.05
60.60
62.25
26.87
35.20
46.32
48.04
St, Sn
28.72
43.31
64.69
67.30
27.14
36.34
48.98
50.38
Jsi
Wk, Un St, Un
33.81
42.62
57.15
58.82
27.05
34.46
44.48
46.18
29.14
43.90
59.15
25.74
34.90
45.82
47.12
Wk, Sn
33.87
62.11
J0.1 Wk-aSt
St, Sn
28.72
60.43
Wk. St. Un
33.81
54.67
29.14
53.13
Wk. Sn
33.87
59.24
St,
28.72
58.51
J
Wk. Un St, Un
33.81
55.28
29.14
54.84
Mistral-24B
Wk. Sn
41.00
52.18
76.57
76.90
38.51
45.14
$7.14
57.53
St. Sn
37.69
45.34
72.16
71.94
37.17
44.81
55.25
56.42
Jwk
Wk,
40.75
47.49
68.56
71.41
37.64
43.81
53.58
53.92
St, Un
38.03
46.57
65.36
65.21
36.25
42.02
52.86
52.92
Wk. Sn
41.00
47.55
73.75
75.69
38.51
44.92
56.13
57.75
St, Sn
37.69
50.07
79.12
81.31
37.17
46.22
59.20
61.82
Js:
Wk, Un
40.75
45.85
66.30
68.52
37.64
43.62
54.92
55.31
St, Un
38.03
47.70
69.73
71.14
36.25
43.98
55.81
57.31
Wk, Sn
41.00
73.70
St, Sn
37.69
73.80
J
Wk. Un
40.75
64.45
St. Un
38.03
62.37
Wk,
41.00
78.22
St. Sn
37.69
75.45
Ji-
Wk. Un
40.75
66.83
St, Un
38.03
66.08
Table 4: Judge's Consistent Accuracy. Left block: DeepScaler; right block: MMLU-Pro. Train indicates whether the judge trained on Weak data (Jwx), Strong data (Jsi), or via continual weak- to-strong training (J (J Eval indicates the type of evaluation split defined by the source of responses among Weak (Wk) or Strong (St) and whether questions Seen (Sn) Unseen (Un). Within each dataset, columns correspond to the judge-training configurations: Zero-Shot, SFT, DPO, and SFT+DPO. For both datasets, backbone (zero-shot) values repeated across all blocks to facilitate direct comparison across judge-training strategies.
29


Reviews:
[{"review_id": "CzACSYblLV", "reviewer": "Reviewer_iv6U", "summary": "This paper studies the “shelf life” of finetuned LLM judges—how well they keep working as generators evolve and as questions change. The authors introduce a dual-distribution view of judge inputs that separates the question distribution Q from the response distribution R and formalize the judge input as X = Q \\times R \\times R for pairwise judgments. They then operationalize three axes of robustness: future-proofing (weak-trained judges evaluating stronger/newer responses), backward compatibility (strong-trained judges evaluating older/weaker responses), and question generalization (seen vs unseen questions). Experiments are in math, using DeepScaleR questions with verifiable answers, three base judges (Llama-3.1-8B, Ministral-8B, Mistral-24B), and three recipes (SFT, DPO, SFT+DPO), with performance measured by consistent accuracy. Main findings: future-proofing is generally poor (performance drops on newer responses), re-training on strong responses reliably helps (especially with DPO), backward compatibility is mostly fine, continual training narrows the gap between weak-only and strong-only training, and question generalization is limited (drops on unseen questions).", "questions": "1. You group generators into “weak/strong” by pass@1 bands (~0.25 vs ~0.45). Please spell out thresholds and sensitivity: do conclusions hold if the bands shift or are defined per-dataset/epoch?  \n\n2. You only pair a correct and an incorrect response from the same generator to avoid stylistic confounds. This is reasonable; please also quantify how many questions are discarded due to lack of both labels and discuss any induced selection bias toward “borderline” items.\n\n3. Standardize “future-proofing” vs “future proofing” across abstract/body.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this is a timely and practically important study for anyone deploying LLM judges. The dual-distribution framing is clean and useful, and the metric design aligns well with the questions practitioners actually face. The experimental results are clear and—importantly—actionable (e.g., prefer DPO and keep judges refreshed with up-to-date strong responses). I see this as a strong empirical contribution. With some clarifications on novelty, dataset construction choices, and statistical reporting, I would lean toward acceptance and could increase my score.", "weaknesses": "See Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "by5W9gUk8R", "reviewer": "Reviewer_dFqw", "summary": "The paper studies the “shelf life” of finetuned LLM judges under distribution shifts. Concretely, it analyzes (i) future-proofing (weak→strong generators), (ii) backward compatibility (strong→weak), and (iii) question generalization (seen→unseen) in a unified math-domain setup, using pairwise judging data with gold answers and three training recipes (SFT, DPO, SFT+DPO). Main findings: future-proofing is consistently hard; backward compatibility is relatively easy; continual training helps balance the trade-offs; and all judges degrade on unseen questions.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper clearly defines the problem and provides good metrics. This sets a strong foundation for future-proofing the system, maintaining backward compatibility, and generalizing to new types of questions.\n\n2. A key strength is the paper's choice of a verifiable math domain. This decision smartly avoids the common problem of messy or inaccurate data, and their specific evaluation method further strengthens the results by helping to reduce positional bias.\n\n3. The study's approach of separating \"weak\" and \"strong\" generators, along with the continual-training analysis, offers practical guidance on how to roll out updates to \"judge\" models in a real-world setting.", "weaknesses": "1. A significant limitation is that the training method (DPO) and the main test (pairwise accuracy) are fundamentally aligned. The model is, in effect, being tested on the same terms for which it was trained. This gives DPO a built-in advantage that doesn't necessarily prove it's a better judge in a general sense. The paper is missing other kinds of tests (like direct scoring) that would be needed to separate this effect and make a fairer comparison.\n\n2. The paper's main conclusions about \"shelf life\" are drawn entirely from the domain of mathematics. This is a problem because math, with its clear right answers, is not a good stand-in for complex, open-ended tasks like summarization or dialogue. These real-world tasks involve messy data, complex goals, and shifting user preferences, where all factors that would likely worsen shelf-life decay, not lessen it. The paper's claims are too broad for its evidence, which is limited to a single, clean domain and a simulation of time rather than real-world data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HblV7G8kYq", "reviewer": "Reviewer_YpH3", "summary": "This paper studies the robustness and “shelf life” of finetuned LLM‑as‑judge models. The core contribution is a dual‑distribution formalization. This framing lets the authors isolate and measure three practically important dimensions: future‑proofing (judges trained on older/weaker generator outputs evaluated on newer/stronger outputs), backward compatibility (the reverse), and question generalization (performance on unseen questions). They define clear metrics—FutureProof, RefreshAdvantage, BackCompatibility, and CompatibilityShift—and evaluate across three base models (Llama‑3.1‑8B, Ministral‑8B, Mistral‑24B) and three training recipes (SFT, DPO, SFT+DPO) on a math domain with verifiable labels. This paper provides more key findings and provides a unifying view and actionable guidance for deploying judge models.", "questions": "See cons", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "1. This paper investigates some key questions in the LLM-as-a-Judge domain with interesting findings & takeaways.\n2. Extensive and commonly used setups are evaluated and conducted within the math domain.", "weaknesses": "1. All experiments are conducted in the maths domain; adding at least one open-ended question would broaden the scope of this paper. This is the major concern of this paper: are we drawing conclusions from a special case?\n2. Current experiments only study one round of weak-to-strong. It would be better to scale to multiple rounds.\n3. Currently, all experiments only compare effectiveness; adding efficiency consideration would be better.\n4. The paper could benefit more from reading if it were reorganized into a more expressive form. As this paper is not a classic paper, it may not necessarily follow the intro-method-experiment setup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VvQKJP1kbC", "reviewer": "Reviewer_X2CZ", "summary": "This paper studies the durability (“shelf life”) of finetuned LLM-as-judge models under realistically inspired distribution shifts. The authors introduce a clear dual-distribution view of automatic evaluation that factorizes a judge’s inputs into a triplet of question and pairwise responses (one correct, other incorrect, but importantly both sampled from the same model). Within this frame, they formalize three practical axes: future-proofing (weak→strong generator shift), backward compatibility (strong→weak), and question generalization (seen→unseen questions). They instantiate the study in math (DeepScaleR, olympiad-style problems) and compare three judge-training recipes (SFT, DPO, SFT+DPO) across three base models (Llama-3.1-8B, Ministral-8B, Mistral-24B). The authors find that: (i) judges trained on “weak” responses do not future-proof—performance drops on “strong” responses; (ii) retraining on “strong” responses yields consistent gains (especially with DPO); (iii) strong-trained judges are fairly backward-compatible on weak responses; (iv) continual training balances both directions better than training from scratch; and (v) question generalization is weak—performance drops notably on unseen questions.", "questions": "Q1: It is a bit strange that base models were finetuned? Why not use Instruct version of the models? What is the justification for using base models?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Originality\n\nThe paper tackles an important question of *shelf-life* of judge models which I have not seen addressed before in the community. It is a question of increasingly growing importance as the field of LLM-as-a-judge matures. The paper proposes intuitive, measurable quantities—FutureProof, RefreshAdvantage, BackCompatibility, and CompatibilityShift—to quantify judge robustness under evolving generators and questions. These metrics translate deployment concerns into experimentally testable criteria.\n\n### Quality\n\n1. Systematic experimentation: The study spans three judge bases (8B and 24B), three recipes (SFT, DPO, SFT+DPO), and multiple train/test splits (seen vs. unseen questions; weak vs. strong response pools). \n2. Clear empirical takeaways: Results are easy to interpret -- future-proofing is hard (negative FutureProof across the board), retraining on strong responses pays off (positive RefreshAdvantage, largest for DPO), strong-trained judges are mostly backward-compatible, continual learning narrows gaps in both directions, and unseen-question performance drops notably.\n3. Sound dataset engineering. Responses within a pair come from the same generator to avoid style-based shortcuts; base judge is excluded as generator to reduce self-bias; and response-order flipping is used for evaluation. These are thoughtful choices.\n\n### Clarity\n\nThe paper is generally well-structured: motivation→framework→metrics→experiments→implications. Metric definitions are explicit and the content is easy to follow.\n\n### Significance\n\nThis work can have practical impact on evaluation pipelines (though currently initiated only in math). The core message—retrain judges with up-to-date strong generator outputs (DPO works best)—is actionable but not surprising. The finding that strong-trained judges can substitute older judges with minimal losses reduces migration risk.", "weaknesses": "W1: My main concern is with the fact that authors choose only math as the domain for evaluation. For math, we already have rule-based verifiers to check answers so a judge model is not needed (unless used for step-by-step level check).  Only DeepScaler dataset is used for evaluation. Given that only a single (narrow) domain is studied in the paper, the results may not transfer to more open-ended or subjective tasks (e.g., non-math domains, creative writing). I would consider increasing my score if the authors can add at least 2 other domains (eg: science domains, or mathematical proofs if sticking just to math).\n\nW2: Reliance on a more capable teacher for supervision. SFT/DPO training uses teacher-generated CoT/verdicts; details indicate prompting GPT-4o model for supervisory signals. When studying future compatibility, it is possible that old judges are being used to evaluate frontier models in which case no \"stronger\" judge exists and it might not be possible to get training data for the judge on responses from strong (newer) models. Thus, the proposed solutions of \"retraining on stronger data\" or \"continual learning\" might be infeasible [1]. Do the authors have any approaches for how to address this bottleneck?\n\nW3: The results are mostly expected apriori and not super insightful. For example, the performance drop on “Unseen questions” is not surprising at all. Given that the models were never trained for them, it is expected that the performance of the judges on those questions will be less than the seen ones as they are OOD. Similarly, judge models becoming better with continual learning is also not surprising as that version learns from both strong and weak data compared to other judges which were training only one of the two distributions. \n\nW4: No prompt templates are provided on how the judge model was presented with the responses (was it the whole response or just the final answer?)\n\nW5: Nothing is mentioned about the how the responses were sampled (top_p, temperature values).\n\n## Presentation nits: \n\n1. Some figure y-axes are labeled “Metric” without units; the sign conventions (negative=degradation) require mental mapping. Consider captions with concrete units (percentage-point deltas of consistent accuracy).\n2. Table 2 repeats “Base (zero-shot)” blocks per model; consider a single consolidated row to reduce clutter.\n3. Fig.1 is hard to follow with the arrows. \n4. It is not clarified which models are strong and which models are weak in the main paper. While this is mentioned in the Appendix, this should also be stated in the main paper. \n\nReferences:\n\n[1] Dorner, F. E., Nastl, V. Y., & Hardt, M. (2024). Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data. arXiv preprint arXiv:2410.13341.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)