Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
Authors: Swarnadeep Saha
Xian Li
Marjan Ghazvininejad
Jason E Weston
Tianlu Wang
Venue: ICML 2025 poster (2025)
Keywords: LLM-as-a-Judge
Planning
Reasoning
Evaluation
Primary area: deep_learning->large_language_models

Abstract: LLM-as-a-Judge models generate chain-of-thought (CoT) sequences intended to capture the step-by-step reasoning process that underlies the final evaluation of a response. However, due to the lack of human-annotated CoTs for evaluation, the required components and structure of effective reasoning traces remain understudied. Consequently, previous approaches often (1) constrain reasoning traces to hand-designed components, such as a list of criteria, reference answers, or verification questions and (2) structure them such that planning is intertwined with the reasoning for evaluation. In this work, we propose EvalPlanner, a preference optimization algorithm for Thinking-LLM-as-a-Judge that first generates an unconstrained evaluation plan, followed by its execution, and then the final judgment. In a self-training loop, EvalPlanner iteratively optimizes over synthetically constructed evaluation plans and executions, leading to better final verdicts. Our method achieves a new state-of-the-art performance for generative reward models on RewardBench and PPE, despite being trained on fewer amount of, and synthetically generated, preference pairs. Additional experiments on other benchmarks like RM-Bench, JudgeBench, and FollowBenchEval further highlight the utility of both planning and reasoning for building robust LLM-as-a-Judge reasoning models.

Learning to Plan & Reason for Evaluation with Thinking-ULM--s---JJddg-
Swarnadeep Saba
Xian Li! Marjan Ghazvininejad
Jason Weston' Tianlu Wang'
Abstract
evaluators (Zheng et al., 2023; Kim al., 2024a; Saha et al., 2024a; Dubois et al., 2024). LLM-as-a-Judge models also serve as reward models during training for iterative preference optimization and seff-improvement (Yuan et al., 2024). Compared to traditional reward models that only output scalar scores, LLM-as-a-Juude models expend more test-time compute by generating Chain-of-Thought rationales of the underlying reasoning process of evalua- tion. This has been shown not only improve evaluation accuracy but also enhance transparency (Zheng et al., 2023; Wang et al., 2024c; Ankner et al., 2024).
LLM-as-a-Judge models generate chain-of- thought (CoT) sequences intended to capture the step-by-step reasoning process that underlies the final evaluation of response. However, due the lack of human-annotatee CoTs for evaluation, the required components and structure of effec- tive reasoning traces remain understudied. Conse- quently, previous approaches often (1) constrain reasoning traces to hand-designed components, such as list of criteria, reference answers, or verification questions and (2) structure them such that planning is intertwined with the reasoning for evaluation. In this work, propose EvalPlanner, preference optimization algorithm for Thinking- LLM-as-a-Judde that first generates an uncon- strained evaluation plan, followed by its execu- tion, and then the final judgment. In a self-training loop, EvalPlanner iteratively optimizes over syn- thetically constructed evaluation plans and execu- tions, leading to better final verdicts. Our method achieves a new state-of-the the-art art performance for generative reward models on RewardBench and PPE, despite being trained on fewer amount of, and synthetically generated, preference pairs. Ad- ditional experiments on other benchmarks like RM-BBenc, JudgeBench, and FollowBenchEvva further highlight the utility of both planning and reasoning for building robust I.LM-as-a-Judee reasoning models.
Thought (CoT)
Despite the promise of LLM-as-a-Judge models, the lack of human-annotated CoTs makes it difficult to train such mod- els. Hence, a crucial step in building these judges is gener- ating rationales by writing down detailed evaluation instruc- tions rubrics that LLMs can follow. These hand-crafted instructions vary for every new domain (e.g., safety versus coding) (Yu et al., 2024b) and include manually designing evaluation criteria (Zheng et al., 2023; Saha et al., 2024a; Trivedi et al., 2024; Wang et al., 2024b;c), scoring rubrics, and steps for each criterion (Yuan al., 2024; Trivedi et al., 2024; Kim et al., 2024b; Wang et al., 2024d). This is limiting because different tasks necessitate evaluation standards procedures tailored to each specific task. For instance, eval- uating essay requires measuring quality along multiple, potentially subjective, fine-grained criteria like relevance and clarity whereas evaluating a math problem requires objectively verifying the correctness of the solution in a step-by-step manner (Lightman al., 2024). Simply using predefined evaluation prompts hurts evaluation accuracy, while manually adjusting the evaluation instructions is nei- ther scalable nor realistic, given the wide range of arbitrary and complex tasks that LLM-ass---udde models are used for.
1. Introduction
To overcome these limitations, we propose EvalPlanner, a novel approach to building Thinking-LLan LL---M------udda mod- els that teaches LLMs both plan and reason for evaluation. EvalPlanner is trained perform complex evaluation by thinking and spending more test-time compute with CoTs that are decoupled into planning component and a rea- soning component. In the planning component, the model generates a detailed evaluation plan, that consists all the necessary steps to evaluate responses specific to the given instruction. In the reasoning component, the model exe- cutes the plan step-by-step and reasons through the input
As large language models (LLMs) continue to improve, re- liably evaluating their long-form outputs has become even more challenging. Owing to the high cost of human eval- uation, the LLM-as-a-Judge paradigm has emerged as promising alternative where LLMs themselves are employed
FFIIR Meta. Correspondence to: Swarnadeep Saha <swar- nadeep@ mema.com>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning to Plan & Reason for Evaluation with Thinking-LLMMassaJJjdge
Example Input and Output of EvalPlanner
[Texx""There is a French pub called The Phoenix that has low customer rating but is child friendly and is more than 30 dollars and is located the riverside area near Crowne Plaza Hotel.
User Instruction:
(QuestionAAccordingg According to above [Text], extract the attribules (name, eatType) the restaurant, Do not output your analysis, only oulput the result in the format of "attributel [altributel[aalull]] [valuel], (aliiuu,,) ..."
Response priceRange[Moon More than 30 dollars]
Name|The Phoenix], eatType(publ, (ivatron[Rieeside area near Crowne Plaza Hotel],
Response B: Name|The Phoenix], eyteyppppbb
Output:
[Start of Evaluation Plan]
To evaluate the response quality, we will follow these steps:
1. ""Check if the response provides the correct attributes**, Verify that the response includes the attributes
"name" and "eat'Type" as requested in the question.
**errify the accuracy of the attribute values**: Check if the values provided for the attributes match the information given in the original text.
2.
** "Check for any additional or missing information**, Ensure that the response does not include any extra attributes or information not requested in the question.
Step-by-step reference answer:
1. The response should include the attributes "name" and "eat'Type".
2. The values for the attributes should be:
name: The Phoenix
eat'T'yp:: French pub
3. The response should not include any additional attributes or information.
[End of Evaluation Plan]
[Start of Plan Execution]
**Step 1: Check the response provides the correct attributes**
Assistant A's response includes the attributes "Name", "eat Type", "location", and "priceRange". However, the question only requests the altributes "nameand "catType". Assistant B's response includes the attributes "name" and "eatType", which matches the request.
#*St 2: Verify the accuracy of the attribute values **
**Step 3: Check for any additional or missing information**
Assistant A's response includes additional altributes "location" and "prioeRange" that are not requested in the question. Assistant B's response does not include any additional attributes or information.
Based on the evaluation, both responses have some inaccuracies. However, Assistant B's response more accurate in terms of providing only the requested attributes, even though missing the "French" descriptor.
The final verdict is: [[B]]
[End of Plan Execution
Final Verdict]
[B]]
Figure 1. representative input and (truncated) output of EvalPlanner. EvalPlanner takes a user instruction and a pair of responses as inputs. It generates Chain-of-Thougee structured into planning component (the evaluation plan), reasoning component (the plan execution), and the final verdict. The evaluation plan specifics the recipe while the plan exccution follows this recipe step-by-step by analyzing the responses, leading the final judgment.
response(s) to arrive at the final verdict. EvalPlanner is iter- atively trained in a self-improving loop (Yuan et al., 2024; Wang et al., 2024c; Wu et al., 2024a) by sampling multiple plans and plan executions from the current model and per- forming preference optimization over correct and incorrect CoT's, i.e., chosen and rejected (plan, execution, verdict) triples. This teaches the model to iteratively optimize for both (1) generating good plan that may encapsulate the most relevant and fine-grained criteria, scoring rubrics, ref- erence answers, unit tests, etc based on the input task at
hand and (2) performing correct execution grounded in the generated plan. EvalPlanner achieves this learning using only synthetic data as supervision via self-training.
We conduct extensive experiments on five reward modeling benchmarks RewardBench,, PPE, RM-Bench, JudgeBench, and Follow spanning instructions across cate- gories of Chat, Safety, Code, Math, and fine-grained multi- level constraints. On RewardBench and PPE, EvalPlanner achieves new state-of-the--e scores (e.g.. 93.9 on Reward- Bench) for generative reward models, outperforming base-
2

---

Learning to Plan & Reason for Evaluation with Thinking-LLLM-ass-JdJde
lines that train on up to 30x more, and typically human- annotated, data. Our model also generalizes well to other benchmarks, obtaining up to 13% improvement over a lead- ing model for complex prompts that require evaluating multi- level constraint satisfaction. Finally, we conduct a set of comprehensive ablations that highlight the effectiveness of EvalPlanner's (1) unconstrained evaluation plans over constrained ones, (2) iterative optimization recipe of these plans, and (3) data-efficiency, allowing it to obtain compet- itive performance with as few as 5K synthetic preference pairs. Overall, ExalPlanner opens up new opportunities for building -La-kin--LL--s------- test-time compute for robust and transparent evaluation by learning to both plan and reason jointly.
the correct final verdict or not. Using this data, we develop self-training loop that trains L.M--as-a-Judge model by optimizing over both plans and executions, leading to better judgments (Section 2.3). At test time, the model generates Co'Is of the form y = (2, ē, 5), structured into a plan, its execution, and the final verdict.
2.2. Synthetic Training Data Generation
-La--a-aeuudge models are typically trained on human- annotated preference judgments. However, collecting such data is a costly and tedious process, often requiring expert annotations for domains like code and mathematics (Ouyang et al., 2022; Wang et al., 2024c). Even when such judgments exist, they do not come with any corresponding reasoning steps. This motivates us to develop EvalPlanner by only assuming access to some carcfully-chosee input instructions as training data. In the rest of this section, we describe our synthetic training data generation process, which includes constructing both preference pairs (a,b) and their CoTs y.
models that scale up
2. EvalPlanner
We consider the setting of pairwise response evaluation using the LLM-aas--Judge paradigm (Zheng et al., 2023). The judge model takes an instruction x and a pair of responses a and as inputs and generates a preference judgment y. predicting the better response, a or b. By doing so, the model also generates a Chain-of-Thoughh (Co'T) (Wei et al., 2022) aiming to capture the step-by-step reasoning behind the evaluation process.
Prompt Selection and Generating Response Pairs. We choose prompts belonging to general instruction-following as well as mathematical reasoning. For general instruction- following prompts, we use the same approach as in Self- Taught Evaluators (Wang et al., 2024c) to generate response pairs, i.e., by first modifying the original instruction into 'noisy" instruction and then generating response to the noisy instruction. Consequently, the response to the original instruction becomes the chosen response, while the one for the "noisy' instruction becomes the rejected response. For prompts specific to math reasoning, we sample multiple responses, where responses that lead to the correct solutions become our chosen responses, while those with incorrect solutions are considered rejected responses.
2.1. Method Overview
Evaluating long machine- generrated responses to complex in- structions is primarily planning and reasoning problem. In particular, the evaluator must first plan the evaluation recipe and then reason through that recipe and the response(s) to arrive at the final verdict. With that motivation, EvalPlanner hypothesizes that an effective Chain-of- Thoouhh for evalua- tion should consist of three components: (1) the Evaluation Plan 2, (2) the Execution of the Plan e, and (3) the Final Ver- dict y. Figure 1 shows example highlighting these three components. For given input instruction x, the evaluation plan specifies the recipe for evaluating given responses to the instruction. The execution of the plan is responsible for actu- ally conducting the evaluation by following the plan step-by- step, analyzing the input pair of responses a and b and gen- erating the final judgment y. Given an I.LM operating as LLM-as- LLLM-as-aJuddee audge, parameterized by 6, where the plan 2 and the execution are assumed to be latent variables, we can write the generative process of the final verdict as follows.
Given these synthetic pref- erence pairs, we now want to generate the latent evaluation plans. Intuitively, a plan that evaluates an open-ended writ- ing question would be structurally and semantically very dif- ferent from a plan that evaluates a coding question. Hence, depending on the evaluation domain, the plans could vary significantly (see Appendix for some examples of diverse plans). This makes manually defining the structure or the components of good plan time-consuming, less generaliz- able, and prone to user biases. Thus, we design a generic and unconstrained plan generation prompt (Fig. 3 in Appendix) that queries a seed model (e.g., an Instruction-tuned LLM) for an initial plan conditioned only on the input instruction. These plans will then be optimized later by self-training. also show the efficacy of
Generating Evaluation Plans.
polp|z, ESEP EnEe Polyle, 2, b)pa(ele, ()p=(=)=))
We follow this generative process to build preference pairs of CoTs (Section 2.2) for training such a model. See Figure 2 for an overview. Given an instruction and a seed model, we first sample multiple plans 2 E P. Then, for a given plan, instruction, and a pair of responses, we sample multiple executions e E of the plan which either lead to
As part of experiments, this plan generation prompt against other prompts that try to constrain plans to certain pre-defined components. Note that our planning prompt does not condition on the response pair to ensure that the generated plans represent only the
3

---

Learning to Plan & Reason for Evaluation with Thinking-LLMM-as-ee
Instructions
Synthetic Reasoning Data Generation
Preference Optimization
{
Current Model
Ri
Me
M
(1o, au (shr (4f.e2))
Me+1
Sample Eval Plans
Sample
DPO Training
181
Exeeutions
C2
{14s b
Preference Pairs of CoTs
Synthetic
Next Iteration of LLM-assa-ludge Model
Preference Pains
Figure 2. EvalPlanner: A Thinking LLM-as-a- a--udge model that learns to think planning and reasoning for evaluation. Given an instruction and a preference pair as input, the synthetic reasoning data generation recipe consists of sampling multiple plans and multiple executions from the current model. These evaluation plans and executions used to construct preference pairs of Chain-of-Thoughts. which are then iteratively optimized with DPO in a self-training loop.
recipe and not the actual evaluation. Evaluation happens in the second stage of plan execution, as described below.
ble correct and incorrect execution-pairs leading to correct and incorrect judgments respectively. This process allows us to construct "chosen" (c) and "rejected" (r) pairs that differ in their executions. To further improve the model's performance, we repeat this construction process. We con- struct chosen and rejected pairs for all plans P, enabling the model to learn to generate better plans in the first place. We also construct pairs for both execution orders of responses (a, b) and (b, a) ensuring that the model becomes position- consistent. If none of the executions lead to the correct verdict, the corresponding sample not included in our training data. By scaling up the construction process, provide the model with more comprehensive set of exam- ples to learn from, enabling it to refine its decision-making process and improve its overall performance.
Generating Plan Executions. In this second stage of plan execution, we now prompt the same seed model with the instruction and the response pair to reason through the plan generated in the previous step and the response pairs to produce a verdict (Fig. 4 in Appendix). The benefits of disentangling the planning and execution stages are two-fold. First, the disentanglement tries to enforce that the reasoning/execution follows the plan. Second, by sampling multiple plans and multiple executions for the same plan, we train model on diverse evaluation data that vary in both planning and execution. Like the initial plans, the initial plan executions will also be optimized later via self-training.
Building Preference Pairs of Plans Executions. Given the preference pairs, the plans, and their executions, we now build a preference tuning dataset to optimize over the CoTs. For each input instruction, we sample [Pl plans and for each plan, sample |E| executions. To account for position bias (Zheng et al., 2023), we generate plan executions according to both orders of the response pairs (a, b) and (b, a). This results in a total of x (Pl x |E| CoTs, for each input instruction. A thought is considered correct if the corresponding (plan, execution, judgment) triplet leads to the correct verdict and incorrect otherwise. Using this correctness criterion, construct our preference tuning dataset D Dla.b) U (ib,e) where D(a,b) is defined as follows and ((b,a) swaps the execution order of responses.
2.3. Preference Optimization of Plans & Executions
Having developed the initial training data generation recipe, we now describe the training algorithm of EvalPlanner. The pipeline consists of a self-training loop, starting with seed model MM (e.g., an instruction-tuned LLM), doing supervised fine-tuning (SFT) on subset of the 'chosen' Co'Ts to obtain model M followed by two iterations of Direct Preference Optimization (DPO) (Rafailov et al., 2024) preference pairs of CoTs, leading to models MPpo and M
M SET on Di, initialized from MMg Starting from the seed model Mg and subset of input instructions and re- sponse pairs, we follow the recipe Section 2.2 to generate the preference pairs of thoughts. Let us denote this dataset by D1. To teach the model to correctly follow the pattern of our CoT (plan exxcutionn
D(a.b) U ( (i + (z,e",u") ZEP
le,,e" E E(u.b))
we first fine-tune Mo on Di subset of only the 'chosen' thoughts from D Specifically, for each instruction, we randomly sample one correct thought (that leads to the correct verdict) and perform SFT on that data, leading model M
Intuitively, given an input (z, a, b). we generate multiple executions for each sampled plan 2 to enable the model to learn from its executions. Specifically, for each plan, we generate multiple executions and construct all possi-

---

Learning to Plan & Reason for Evaluation with Thinking-LLLM-ass-JJJde
MMPo: DPO on D1, initialized from MiT. Next, ini- tialized from M we perform DPO on the dataset D1, consisting of both chosen and rejected thoughts. Given the two distinct parts of plan and execution tokens in the thoughts, this teaches the model to contrast between correct and incorrect thoughts, that vary in both the plan and the execution of evaluation. We thus obtain a model MfFo.
points every 100 steps and doing early stopping based on the validation set. Detailed training hyperparameters provided in Table 12.
Evaluation. We test EvalPlanner on the following pair- wise evaluation benchmarks.
Mp DPO on 22, initialized from AMPP EvalPlanner also consists of second iteration of DPO, wherein we choose fresh subset of instructions and response pairs and
RewardBench (Lambert et al., 2024). It consists of (prompt, chosen, rejected) triples spanning categories of prompts: chat, chat-hard, safety, and reasoning.
generate CoTs using the same recipe but from the previous
Preference Proxy Evaluations (PPE) (Frick al., 2025). PPE is large-scale benchmark that links reward models to real-world human preference performance. It consists of two subsets: (i) PPE Preference (10.2K samples), human preference pairs from Chatbot Arena featuring 20 LLMs in 121+ languages, and (ii) PPE Correctness (12.7K samples), response pairs from four models across popular verifiable benchmarks (MMLU-Pro, MATH, GPQA, MBPP-Plus, IFEval). The first subset evaluates subjective preferences, while the second tests alignment in Best-of-N tasks.
iteration of model Maro In particular, we first sample [P] CoTs from M for each training data point, separate out the plans from the thoughts, and then use the same M model to sample [&] executions for each plan. We denote this second iteration of CoT data as D2- We train on new inputs and thoughts from an updated model, under the as- sumption that the data from the previous iteration is of lower quality. Empirically, we also show that this outperforms a single iteration of DPO trained on the entire set of inputs.
Follow Eval We build this new evaluation benchmark from FollowBench (Jiang et al., 2024). The original benchmark consists of complex prompts that test LLMSs ability to follow multi-level fine-grained constraints (e.g., 'Write a summary within 20 words'), We convert this benchmark into pairwise evaluation benchmark by sampling two responses from single I---tuuttt LLama-3.2-3B-Innsee or Mistraa-7B-Inntruut--00222 such that one response satisfies all the constraints and the other one does not. Note that by generating the response-par -air using the same model, we ensure consistency in response style which can otherwise lead to potentially superficial features for preference judgments. Our evaluation benchmark, called Follow comprises of 205 samples and spans five different constraint-types of Content, Situation, Style, Format, and Example. This benchmark specifically tests LLM-based judges' ability to (1) plan for multiple constraints that need be checked, and (2) produce a verdict by checking for those constraints.
3. Experimental Setup
Training. Wc select prompts from two different sources WildChat (Zhao et al., 2024) and MATH (Hendryeks et al., 2021). For WildChat, we directly use the synthetic re- sponses generated by Self-Taught Evaluators (Wang et al., 2024c). For MATH questions, we generate synthetic re- sponses as follows. We prompt a Mixtral 22Bx8 Instruct model to generate multiple candidate solutions. The re- sponses that lead to the correct final answers become our chosen responses while those with incorrect final answers are considered rejected responses. Using synthetic response- pair generation, we collect a total of 17, 588 and 4, 141 unique (instruction, chosen, rejected) triples from WildChat and MATH, respectively, as our training data, using two separate methods. From this, we select a random subset of 5K instructions (consisting of 2.5K from WildChat and 2.5K from MATH) for SFT and the first iteration of DPO. We reserve the rest for the second iteration of DPO. In each iteration, we sample 5 plans and for each plan, we sample 8 executions (4 in each order of response pair) using a temper- ature of 0.8 and top-p of 0.95. We develop EvalPlanner with either Llama-3. Instruttttt or 3.3 Instructtt as the seed model show the generalizability of our approach across multiple seed models. As validation set, we choose 150 samples from each of WildChat and MATH, which we use for checkpoint selection. To account for position bias in pairwise evaluation, we double the number of examples in the validation set by considering both orders of response pairs. We use the fairseq2 library (Balioglu, 2023) for model training and vLLM (K (wonn& et al., 2023) for inference. All models are trained for a maximum of IK steps, saving check-
model
RM-Bench (Liu et al., 2024). RM-Bench is designed to assess the robustness of reward models, based on their sensitivity and resistance to subtle content differences and style biases. The original benchmark primarily focuses on evaluating reward models that rate each response independently. We modify the input prompt to accommodate for the evaluation of LLM-as-a- models, which conduct pairwise judgments by comparing two responses simultaneously.
JudgeBench (Tan et al., 2024). JudgeBench is a recent benchmark that evaluates LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding. It sources input instructions
5

---

Learning to Plan & Reason for Evaluation with Thinking-LLL-as---JJuue
Table Comparison of EvalPlanner with SOTA generative reward models on RewardRench. EvalPPanneer outperforms all prior models, while using smaller number of (22K) synthetically constructed preference pairs as training data. f: Results taken from either RewardBench leaderboard or the corresponding paper. Results taken from the Critic-RM Rank paper (Yu al., 2024b).
Models
#Pref Pairs
Overall
Chat
Chat-Hard
Safety
Reasoning
Open and Closed L.
Llama3, 1-001-10-nnnae 70sttustt
84.0
97.2
70.2
82.8
86.0
Llama3, I 1405--Instrur
84.1
97.2
74,6
77.6
87.1
Llama3,3 200---------nt-te
85.4
96.9
77.4
77.6
89.6
Claude-3.5-snn 3.5-sonnet
84,2
96.4
74.0
81.6
84.7
GPT-4o"
86.7
96.1
76.1
88.1
86.6
Gemini-1.5ppro-09
88.2
92.3
80.6
87.9
92.0
Reward Models with Critiques
SynRM² (Ye et al., 2024)
87.3
97.5
76.8
86.3
88.5
CLoud (Ankner et al., 2024)
87.6
98.0
75.6
89.0
87.6
Critic-RM-Rane (Yu et al., 2024b)
90.5
97.5
79.6
94.1
90.6
SOTA Generative Reward Models
Self-Taugh Evaluator' (Wang et al., 2024c)
20K
90.0
96.9
85.1
89.6
88.4
1.lmm--.11-70--- -Jdgee (Wang et al.. 2024b)
680K
92.7
96.9
84,8
91.6
97.6
Skywork Sework-Critie-I.bee Critie-Llama-3, 3-700 (Shiwen al., 2024)
80K
93.3
96.6
87.9
93.1
95.5
LMUnit! (Saad-Falcon 2024)
84K
93.4
EvalPlanner 3ama-331-700-Instruu as seed model)
22K
93.9
97.5
89.4
93.0
95.5
EvalPlanner -. as seed model)
22K
93.8
97.7
89.5
91.7
96.1
from existing datasets and generates candidate responses using stronger language models such as GPT-4o and Claude-3.5-Sonne Following Tan et al. (2024), we report results on the GPT-4o subset.
ence pairs. Table shows results on RewardBench. Using the same recipe, we train two EvalPlanner models with different Llama versions as the seed model (Llama-3.1- 70B-Instruct and Llama-333770B-nnstruc)) Both of our models outperform all baselines, achieving new state-of- the-arts for generative reward models on RewardBench. Par- ticularly impressively, EvalPlanner achieves these results by being trained on smaller number of preference pairs (22K), compared to most prior works. Moreover, EvalPlan- ner's training data only consists of synthetically generated preference pairs, unlike past works that primarily train on human-ame annotaedd preference pairs. EvalPlanner's training recipe is also equally performant on both Llama seed mod- els, showing the usefulness of our initial training data and the generalizability of our approach. In Table 2, we compare EvalPlanner to DeepSeek-GRo (Liu et al., 2025), a SOTA generative reward model on PPE. Our method obtains signif- icant improvements on the PPE Correctness subset, consist- ing of popular reasoning benchmarks, thereby showcasing its potential as reward model for Best-of-N alignment. In general, our results highlight the utility of planning and reasoning for evaluation, not only for better final judgments but also for better grounding of the evaluation in detailed plan.
For RewardBench, PPE, and RM-Bench, we follow the orig- single
inal evaluation protocol of reporting accuracy random ordering of paired responses. We report position- consistent accuracy for JudgeBench and Follow BenchEva to account for position bias. Specifically, a prediction is considered correct the model consistently makes a correct judgment in both orders. We train and test all our mod- els using the standard pair-wise judge prompt from prior work (Zheng et al., 2023), shown in Figure 5. The max- imum number of generation tokens is set to 2048 and the temperature to 0 for inference.
Baselines. We compare EvalPlanner with range of mod- els, including (1) Powerful Open- Open-Sourced and Closed- Sourced LLMs used as judges in a zero-shot manner, (2) Reward Models with Critiques, capable of generating both scalar scores and critiques, and (3) SOTA Generative Re- ward Models, as listed on the RewardBench leaderboard. We focus models that also generate rationales along with the final verdict, to compare related competing approaches.
EvalPlanner's plans are tailored toward the specific instruction being evaluated. We design EvalPlanner such that the generated plan represents general evaluation recipe tailored toward the specific instruction being evaluated. This is achieved by disentangling planning from reasoning and having the model directly optimize the thoughts without any domain-specific tuning. Appendix C
4. Results
4.1. Experimental Results on Benchmarks
EvalPlanner outperforms all baselines while being trained on fewer, and synthetically generated, prefer-
6

---

Learning to Plan & Reason for Evaluation with Thinkingg-LLMM-ssa-JJddg
Table Results on PPE comparing EvalPlanner with state-of-the th--rarrtt LLM-at LLLLM-------a- as-a-Judge and reward models. f: Results taken from Liu (2025) and Frick et al. (2025).
Models
#Training Pref. Pairs
PPE Overall
PPE Preference
PPE Correctness MATH
Overall
MMLU-Pro
GPQA
MBPP-Plus
IFEval
Open and Closed LLM-as-a-Judge
Llama-3, 1-8B-Instruct
55.5
56.4
54.7
56.3
62.9
51.4
50.1
52.8
GPT-46"
62.3
67.1
57.6
JJama-3.3-70B-Insttt
65.8
65.9
65.7
72.1
73.1
61.2
59.6
62.3
SOTA Gemerative Reward Madels
GRM-223
237K
62.2
64.7
59.8
64.8
68.8
55.6
50.1
59.8
DeppScck-GRM-27B (MetaRM v 32)
237K
65.2
67.2
63.2
68.1
70.0
56.9
50.8
70.4
EvalPlanner (w/ InSttttsttr
22K
66.9
65.8
68.0
77.8
79.2
58.6
63.5
60.9
EvalPlanner (w/ 1amma-3.3708-Insttrre
22K
67,9
63.6
70.2
78.4
81.7
64.4
62.2
64.3
Table 3. Results on RewardBench comparing EvalPlanner at 8B scale with larger LLM--s--aJudge models.
Model
Overall
Chat
Chat-Hard
Safety
Reasoning
Llama-3.1 1-BBBBBBBBBBB---- Instruct
69.5
92.7
46.1
64.4
74.7
Llama-3. 1.0.-...--71mm Ins--nstt
84.1
97.2
70.2
82,8
86.0
Mlaude33.5-Sonee
84.2
96.4
74.0
81.6
84.7
EvalPlanner (w/ Llama-3.1----- 18----8---sstctc
83.0
85.5
84.0
83.4
79.3
Table 4. EvalPlanneer results on RewardBench comparing two iter-
EvalPlanner generalizes to evaluating multi-level con- straints in Belchw.ennhEvva... Table 5 presents our results FollowBenchEval. The input instructions contain up to five constraints, denoted in the table as LI-L5, Given the nature of this dataset, the preference judgments focus on objective preference criteria (i.e., whether all constraints are satisfied or not), as opposed to subjective metrics like stylis- preferences (e.g.. Chat category in RewardBench). This makes evaluating such prompts more challenging for LLMs and allows to objectively assess the utility of planning and step-wise reasoning for evaluation. In such scenar- ios, EvalPlanner demonstrates clear benefits over its base- lines that do not explicitly plan or reason, outperforming Skywork-Critic- Lioee-Lamaa-11111 3.1- (a state-of of-thee-rt model on RewardBench) by a significant 13%.
ations of DPO with one iteration.
# DPO Iterations
#Pref Pairs
Accuracy
1 (w/ 1llama-3,1-7000 1-70B-Instrucct
SK
92.3
1 (w/ Llama-3.11700 1.1--0--Innstrucy
22K
92.5
2 (w/ Llama-3. 1-1aat....1---1-mmm- I-700--nsttruct
22K (5K+17K)
93.9
shows examples such plans generated by EvalPlanner for diverse instructions.
EvalPlanner at smaller scale matches the performance of larger models. In Table 3, compare EvalPlanner at the 8B scale with much larger models like Llama-3.1- 1- 70B- Instruct and Claude-3.5-Sonne training recipe is also effective at such smaller scales, al- lowing EvalPlanner-88 8B to match the performance larger LLM--as--Judge models.
We show that our
EvalPlanner generalizes to RM-Bench and JudgeBench. We show results other recent benchmarks like RM- Bench (Liu al., 2024) and JudgeBench (Tan et al., 2024) in Tables 6 and 7 respectively. On AM-Bench, EvalPlanner outperforms all baselines, achieving up to 8% improvement over a prior statc-of-the-art Skywork-Crrin Critic-LLama- Instruct model, showing its robustness to subtle differences and style biases. Notably, while all other models exhibit a drop in accuracy the hard subset, EvalPlanner is equally performant across all the subsets. On JudgeBench, EvalPlan- ner with LLama-3.3 3.- Instruct achieves comparable per- formance to Swork--Criti-LL.. ing trained on much less and synthetic preference pairs.
EvalPlanner is data-efficient and benefits from iterative thought optimization. Next, in Table 4, we show the per- formance of EvalPlanner with as few as 5K preference pairs. It obtains a score of 92.3, competitive with the best models on RewardBench. We also demonstrate the effectiveness of iterative DPO the second iteration of DPO improves results significantly (92.3 + 93.9). In contrast, the same amount of data one single DPO iteration only leads to marginal improvements (92.3 92.5). The iterative im- provement of EvalPlanner can be attributed to training newer data points that are augmented with CoTs from an updated model. Repeating this recipe for more iterations can potentially lead to further improvements, which we leave for future work to explore.
IInstruct, while be-

---

Learning to Plan & Reason for Evaluation with Thinking-LLMMassaJJjdge
Table Results Fenl for evaluation of complex prompts with multi-level constraints. EvalPlanner significantly
outperforms other approaches on this challenging task.
Model
Overall
L1
L2
L3
L4
L5
Llama-3.1-70 700ttnnstt
44.4
51.1
50.0
35.9
46.2
42,4
Llama-3.3-700 3--aaBB 7-70--Innttue
52.2
55.3
61.9
48.7
53.8
45.5
Self- Evaluator (Wang et al., 2024c)
46.8
53,2
52.4
51.3
43.6
36.4
Skywork- CritieLLlama-3.1 3.1-000 (Shiwen et al., 2024)
52.2
63.8
57.1
48.7
46.2
48.5
EvalPlanner (w/ Llama-3.1 3.1-70B-
56.6
66.0
61.9
56.4
53.8
48.5
EvalPlanner (w/ Llama-33 3.3- I
65.4
72.3
73.8
66.7
61.5
57.6
Table Results RM-Bench for evaluation of models' robustness subtle content changes and style biases. EvalPlanner demonstrates superior robustness across all subsets, outperforming other methods which are more vulnerable to subtic changes, particularly in the Hard subset where responses are detailed and well-formatted.
Model
Overall
Easy
Normal
Hard
Llama3.1 Inst-untt
64.9
68.9
62.6
63.3
Llama3.3-70B 300ma333-70B-Instruee IIB--nstrrct
69.5
77.5
66.3
64.8
Self-Taught Evaluator (Wang et al., 2024c)
73.6
75.9
72.4
72.4
Shyworr-Cttic--Jaaa-3...
(Shiwen et al., 2024)
74.1
76.3
72.9
73.1
EvalPlanner (w/ Llama-3.1 1.11-70B-Instroe
80.0
81.7
77.2
81.1
EvalPlanner (w/ J..aamaa..-77--Insmtttt
82.1
81.1
80.8
84.3
Table 7. Results on JudgeBench for evaluation models' capabilities on challenging questions spanning multiple categories. EvalPlanner with Dama--.3-70B-Inntruu achieves comparable performance to Skywork Sayywor-Crittc-LLlane Crimiii------aa33.... 1-7BB and outperforms all other baselines.
Model
Overall
Knowledge
Reasoning
Math
Coding
1.lam33117700 1-001,1177--ntmmeee
50.3
53.9
36.7
64.3
50.0
48.6
50.0
43.9
55.4
45.2
Self-Taught Evaluator (Wang et al., 2024c)
52.6
40.8
57.1
Crtrood-Criiii-Laaa-3--------- (Shiwen et al., 2024)
57.1
56.5
55.1
71.4
45.2
EvalPlanner (w/ LLama-3. 1-700--nnttteee
50.9
48.1
50.0
60.7
50.0
EvalPlanner (w/ LLamma3.3-700--nsttuu
56.6
55.8
56.1
69.6
42.9
4.2. Ablations and Analysis
constrain the plans to (1) list of criteria, similar to Self- Taught Evaluators (Wang et al., 2024c) (2) list of ver- ification questions, similar to Chain-of Verification (Dhu- liawala et al., 2023). As shown in Table 9, unconstrained plans obtain superior performance by generating more de- tailed plans and then grounding the evaluation on that plan, A generic planning prompt that works across multiple do- mains showcases the generalizability of our approach.
We conduct all ablations RewardBench using EvalPlanner checkpoint, trained on 2.5K MATH instruc- tions using Llama-3.11700 Inntructtt as the seed model.
Effectiveness of Thought Preference Optimization. In Table we compare EvalPlanner with (1) the seed Llama- 111-7B-Innstrue model, (2) model trained only predict the final verdict without any intermediate CoT, and (3) EvalPlanner variant only SFT' ed the "chosen" examples. The results show that preference optimization of plans & executions is particularly important, leading to significant improvements over all baselines.
Appendix A presents more analyses like effect of scaling up the number of plans and executions and source instructions.
5. Related Work
Effectiveness of Unconstrained Plans over Constrained Plans. Recall that EvalPlanner built with an initial plan- ning prompt that relied the seed model to generate constrained plans. In this experiment, we compare this unconstrained planning prompt with two other prompts that
LLM-as-a-Judge. Human evaluation is often considered the gold standard for evaluating LLM responses to complex and open-ended instructions (Ouyang et al., 2022; Dubey et al., 2024). However, given the slow, expensive, and noisy nature of human evaluation (Clark et al., 2021; Karpinska
8

---

Learning to Plan & Reason for Evaluation with Thinking-LLLM-asaJJJdge
Table Ablation RewardRench showing the effectiveness of preference optimization of plans & executions.
Table 9. Ablation on RewardRench comparing the effectiveness of different types of plans.
Model
Accuracy
Type of Plan
Accuracy
Llama3. 1attaa..-70BB0 I-70B-Instruct (seed model)
84.0
List of Criteria (Wang et al., 2024c)
83.9
Llama3, 1-0BB 10nt-nnttuuct (trained w/o thoughts)
86.2
Verification Questions (Dhuliawala et al., 2023)
84.8
EvalPlanner (SFT w/ thoughts)
86.8
Unconstrained (Ours)
86.8
EvalPllanner (SFT+ + DPO w/ thoughts)
90.5
et al., 2022: Wu et al., 2024a; Hosseini et al., 2024). Unlike methods that train ground-truth thoughts e.g., in the domains of algorithmic reasoning, math, or planning (Nye et al., 2021; Lehnert et al., 2024; Saha et al., 2024b), EvalPlanner is bootstrapped and self-trained from synthetically generated thoughts - focusing on evaluation where objectively defining the structure and components of intermediate thoughts is challenging. Moreover, EvalPlan- ner's thoughts have decoupled planning and reasoning components, allowing it to optimize both at the same time.
et al., 2021), automatic approaches leveraging LLMs have emerged as scalable and cost-effective alternatives (Zheng et al., 2023; Liu et al., 2023; Kim et al., 2024a; Saha et al., 2024a; Jiang et al., 2023; Zhu et al., 2023). Compared to reward models that only output scalar scores (Wang et al., 2024age;d), Juueee evaluators are more robust and interpretable because of their ability to also generate de- tailed rationales (Zheng et al., 2023; Zhang et al., 2024a; Ankner et al., 2024). However, in the absence of any human- annotated reasoning traces for evaluation, past works have leveraged LLMs to generate these traces by writing custom prompts for every new domain (Yu et al., 2024b) and hand- designing the components and structure of CoTs, ranging from fine-grained grained criteria (Zheng et al., 2023; Saha et al., 2024a; Wang et al., 2024c: Zeng et al., 2024; Trivedi et al., 2024), scoring rubric (Yuan et al., 2024; Trivedi al., 2024; Wu et al., 2024b), verification questions (Dhuliawala al., 2023), natural language unit tests (Saad-Falcon et al., 2024), and reference answers (Zhang et al., 2024b). In contrast, EvalPlanner proposes a unifying perspective on evaluation by subsuming all necessary components for sound evalua- tion inside plan and then letting the model optimize these plans and their executions in a self-training loop.
6. Conclusion
We presented EvalPlanner, a novel approach for building robust and data-efficient Thinking- LLM--L-------us mod- els. Through comprehensive experiments across five bench- marks, we demonstrated the effectiveness of our method, achieving a new SOTA with significantly less, and synthet- ically generated, training data. To further understand the ca- pabilities of M.akkingg-L.M-ass.. Judgee models, future work could employ them as reward models in the RLHF pipeline.
Impact Statement
EvalPlanner's 's broader goal is to advance the field of Ma- chine Learning and in particular, evaluation, by allowing LLM-as-a-Judge models to think before producing judg- ment. This has the potential to improve evaluation accuracy and transparency in various applications. EvalPlanner is trained on synthetically generated data from seed Llama models that can reflect stereotypes, biases, and other nega- tive traits present in their pre-training data (Weidinger et al., 2021), which we do not have control over. We encourage further research and discussion on these topics to ensure that this technology is developed and deployed responsibly.
mai--Aligmmnt. Reinforcement Learning from Human Feedback requires large amount of human annotations, which can be expensive to obtain (Bai et al., 2022: Lee et al., 2024). This has led to the development of various self-alignment techniques for general instruction following (Li et al., 2024; Yuan et al., 2024; Wu et al., 2024a), reasoning (Zelikman al., 2022; Pang al., 2024; Gulcehre et al., 2023; Yu al., 2024a), and evaluation (Pace et al., 2024; Wang et al., 2024c; Trivedi et al., 2024). Specifically, for evaluation, Wang et al. (2024c) construct preference pairs by adding noise to the original instructions, while Trivedi et al. (2024) uses self-rationalization and a meta-judge to train a fine-grained judge. Different from these. EvalPlanner proposes novel self-training recipe that teaches an LLM-assa-Judge to think by generating and reasoning with evaluation plans.
References
Ankner, Z., Paul, M., Cui, B., Chang, J. D., and Am- manabrolu, P. Critique-out-loud reward models. arXiv preprint arXiv:2408. / 1791, 2024. URL https:// arxiv. wwwpg/////////////////////eeeee/ 2200/4400... 11791.
Training to Think, Plan, and Reason. EvalPlanner follows a large body of prior work on equipping LLMS with the ability to think by generating additional thought tokens before the final answer (Nye et al., 2021; Zelikman
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with

---

Learning to Plan & Reason for Evaluation with Bhinking-LLMMaas--JJJugg
reinforcement learning from human feedback. arXiv
Hosseini, A., Yuan, X., Malkin, N., Courville, A., Sordoni, A., and Agarwal, R. V-star: Training verifiers for self- taught reasoners, arXiv preprint arXiv:2402.0644 2024. URL https: / arxiv. orr/ abs / 240.. 6457.
preprint arXiv:2204.05882,, 2022. URL https://
arxiv.
Balioglu, C. fairseq2, 2023. URL http://github. com/ Eacebookresearcch.....oooooo htt.:oorrrrrerrcc/aaiiis....
Jiang, D., Li, Y., Zhang, G., Huang. W., Lin, B. Y., and Chen, W. Tigerscore: Towards building explain- able metric for all text generation tasks, Transactions on Machine Learning Research, 2023. URL https: / /ppenreview net for forumm?n di-.ei11EEEECOE.......
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., and Smith, N. A. All that's 'human'i is not gold: Evalu- ating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11 1thh International Joint Conference Natural Language Processing (Volume I: Long Papers), pp. 7282-7296, 2021. URL https: //aclantholes www.annnhhooooooooooooooo org/ wwtp/oog...22. 2021 acl wow..n5.55//.
Jiang, Y., Wang, Y., Zeng, X., Zhong, W., Li, L., Mi, F., Shang, L., Jiang, X., Liu, Q., and Wang, W. Follow- Bench: multi-level fine-grained constraints following benchmark for large language models. In Ku, L.-W., Mar- tins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4667-4688, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10. 10.116665//112022aclllooo http:///aaaaaanccooo...ooo 2024.
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., and Weston, J. Chain-of-veriffcation reduces hallucination in large language models. arXiv preprint arXiv:2309. 2023. URL https:// arxiv.orge abs /tt.//339..11999
257. URL acl acl- 1wn.1ong..222 257.
Dubey, A., Jauhri, A.. Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https:// arxiv. org/ www./b/20000 221./2477..222..... 21783.
Karpinska, M., Akoury, N., and lyyer, M. The perils of using mechanical turk to evaluate open-ended text gen- eration. In Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing, pp. 1265-1285, 2021. URL https: : aclaahoooooo arg/2021.emnlpppeeeeeeeggeeeeeee emw.pmmaapppmm......mm.m.......m 97/.
Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P. S., and Hashimoto, T. B. Alpacafarm: simulation framework for meth- ods that learn from human feedback. Advances in Neu- ral Information Processing Systems, 36, 2024. URL https:/ / /////axivvvv org/ abs///as//// 2.05 14387.
Kim. S., Shin, J., Cho, Y., Jang. J., Longpre, S., Lee, H., Yun, S., Shin, S., Kim, S., Thorne, J., et al. Prometheus: Induc- ing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Rep- resentations, 2024a. URL /tttp::///pperee/// net/ forum? di
Frick, E., Li, T., Chen, C., Chiang, W.-L., Angelopoulos, A. N., Jiao, J., Zhu, B., Gonzalez, J. E., and Stoica, I. How to Evaluate Reward Models for RLHF. In The Thirteenth International Conference on Learning Representations, 2025. URL https::////. http://ppeeeeeeeeeeeee wwwpeeewwwwweeeeeoooreeeeeeeeeee id=cbttLt0940.
Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, Lee, K., and Sco, M. Prometheus 2: An open source language model specialized in evalu- ating other language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Pro- cessing, pp.
Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10. 10.1863/w1100244cmmm-mmiiii a aclanthology org 2024 emm 1a-mmlpppmmmm -maimiin... i..inn.. 248.
Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M.. Gu, C., al. Reinforced self-training (rest) for lan- guage modeling, arXiv preprint urXiv:2308.08998 2023. URL https: /ttp///axivvv///////////ea. org/ / 2308. 08998.
URL https://
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez. J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hendrycks, D.,, Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2), 2021. URL htttttttt///eeeetee //ppnreeeeeeeeeee/////ee/ee net/ forum? dd= BB=7Byyt22@ees
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. RewardBench: Evaluating reward models for
10

---

Learning to Plan & Reason for Evaluation with Thinking-LLM--a--aate
language modeling. arXiv preprint arXiv:2403.1377 2024.
A., al. Training language models to follow instruc- tions with human feedback. Advances in neural informa- tion processing systems, 35:27730-27744, 2022. URL https: / http/aiiiii.... org. abs 2ttps//2223..22.52...
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K. R., Bishop, C., Hall, E., Carbune, V., Rastogi, A., et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In Forty-first International Conference on Machine Learning, 2024. URL ht tps: //openreview, net forum? wwm.mmddduuddddooo d=uydd204111
Pace, A., Mallinson, J., Malmi, E., Krause, S., and Severyn, A. West-of-n: Synthetic preference gener- ation for improved reward modeling. arXiv preprint arXiv:2401.12086. 2024. URL attps://arxix>
org abs /2001 1 12086.
Lehnert, L.. Sukhbaatar, S., Su, D., Zheng. Q.. Movay, P., Rabbat, M., and Tian, Y. Beyond a Better plan- ning with transformers via search dynamics bootstrap- ping. arXiv preprint arXhvv2402.14088 2024. URL https:/ /tt.aaaiivv/..////v//.../..a.vv. org/ abs/ /t.p////4222 2402 14083.
Pang, R. Y. Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, Iterative reasoning preference optimization. arXiv preprint arXiv:2404.197.5 2024. URL https:
/ wwtp:ii..rgg//baa. abss///////// /ttp//2444...32
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimiza- tion: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. URL https: pettps://arxive /ttpaaaxiivvv..///vv org/ abs52305. 18290.
Li, X., Yu, P. Zhou, C., Schick, T., Levy, O., Zettle- moyer, L., Weston, J. E., and Lewis, M. Self-alignment with instruction backtranslation. In The Twelfth In- ternational Conference on Learning Representations, weteeeewweeeeeeeeeeeeeeeeeeeeeee n forum?
2024. URL id-loijHJBRsT.
Saad-Falcon, J., Vivek, R., Berrios, W., Naik, N. S., Franklin, M., Vidgen, B., Singh, A., Kiela, D., and Mehri, S. Lmunit: Fine-grained evaluation with natural language unit tests. arXiv preprint (1XX::224/2213991 2024. URL https: : /atrivxivvv////////////a/////aaa
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https:://// httpss///ppeeeeerrreeee fer.rrrrreeweeee/ooumnn forum? 1d=v8L0pN6E01.
Saha, S., Levy, O., Celikyilmaz, A., Bansal, M., Weston, J., and Li, X. Branch-solve-mmrge improves large lan- guage model evaluation and generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies ( (Volume I: Long Papers), pp. 8345-8363, 2024a. URL https: aalaaaccnonoooo..... /2g/2024.00 wtw.:/22.aaaaloonng... 462/
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. G-eval: Nlg evaluation using gpt-4 with better human aligmment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 2511-2522, 2023. URL https://////// pattcs::/accaaen aaclantoooogg..... org/2023,0 emnlp main.153/.
Liu, Y., Yao, Z., Min, R., Cao, Y., Hou, L.,, and L. Rm- bench: Benchmarking reward models of language models with subtlety and style, 2024. URL https:////.. /tatps:///rxiv.. org/abs/ 2410 16184.
Saha, S., Prasad, A., Chen, C.-Y., Hase, P., Stengel-Eskin. E., and Bansal, M. System- x: Learning to balance fast and slow planning with language models. arXiv preprint arXiv:2407. 14414, 2024b. URL https:// 14414.
arxiv. axxiv.org/abe. 2447.11414
Liu, Z., Wang. P., Xu, R., Ma, S., Ruan, C., Li, P., Liu, Y., and Wu, Y. Inference- Time Scaling for Generalist Reward Modeling. arXiv preprint arXiv:2504. 02495, 2025. URL https:// /tts:///axvv.. org/ abs/ 2504 02495.
Shiwen, T., Liang. Z., Liu, C. Y., Zeng, L., and Liu, Sky- work critic model series. https: http:://huggingfaace co co/Sky hkywok,,,,, September 2024. URL https:// huggingface,
co/skywork. hurgingfaace.co/Skyyo...
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. Show your work: Scratch- pads for intermediate computation with language mod- els. arXiv preprint arXiv:2112.001144 2021. URL https:/ /ttpa//i....g.////////////a///// wwwpg///bb////
Tan, S., Zhuang, S., Montgomery, K., Tang, Y., Cuadron, A., Wang, C., Popa, R. A., and Stoica, I. Judgebench: benchmark for evaluating llm-based judges, 2024. URL https://arxee org http://bb///.////////
Trivedi, P., Gulati, A., Molenschot, O., Rajeev, M. A., Ramamurthy, R., Stevens, K., Chaudhery, T. S., Jamb- holkar, J., Zou, J., and Rajani, N. Self-rationalization
Ouyang, L., WU, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray,

---

Learning to Plan & Reason for Evaluation with Thinking-LLLM-asaa.JJee
improves I1m as fine-grained judge. arXiv preprint arXiv:2410.05495, 2024. URL https: arxiv. org/abs/2410 05495.
Ye, Z., Greenlee-Scott, F,, Bartolo, M., Blunsom, P., Cam- pos, J. A., and Gallé, M. Improving reward models with synthetic critiques. arXiv preprint arXiv:2405.20880 2024. URL https: /ttpaaiiiiio..o. wwtp/v.oor/ooooooooooooo 20tpg///b//24055 20850.
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable preferences via multi-objective re- ward modeling and mixture-of-experts. arXiv preprint arXiv:2406 2024a. URL https://arxin 12845.
Yu, L., Jiang, W., Shi, H., Jincheng, Y., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Confer- ence on Learning Representations, 2024a. URL https: / //oeereeveww ew. net/ forum? (aerrumidd-NNNNO G-NNNONND
Wang, P., Xu, A., Zhou, Y., Xiong, C., and Joty, S. Direct judgement preference optimization, 2024b. URL //arxiv.org. abs 2209 14664.
Yu, Y., Chen, Z., Zhang, A.. Tan, L.. Zhu, C., Pang, R. Y.. Qian, Y., Wang, X., Gururangan, S., Zhang, C., et al. Self- generated critiques boost reward modeling for language models. arXiv preprint arXiv:241 /6646, 2024b. URL
Wang, T., Kulikov, I., Golovneva, O., Yu, P., Yuan, W., Dwivedi- Yu, J., Pang, R. Y., Fazel-Zarandi, M., Weston, J. and Li, X. Self-taught evaluators. arXiv preprint arXiv:2408.02660 2024c. URL https: haapss//arxiv.. arg/abs/2den 02666.
Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. E. Self-rewarding language models. In Forty-first International Conference on Machine Learn- ing. 2024. BtLhttps:// 10020.
Wang. Z., Bukharin, A., Delalleau, O., Egert, D., Shen, G., Zeng, J., Kuchaiev, O., and Dong, Y. Helpsteer2- preference: Complementing ratings with preferences. arXiv preprint arXiw:2410.01257, 2024d. URL https: //arxiv. . org/ abs/ /t.b////11..0 2410..1257.
/ abs/ 2401.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Boot- strapping reasoning with reasoning. Advances Neural Information Processing Systems, 35:15476-15488 2022. URL https: / /t//aaxivvv.ro///////. abs 2203. ...003..4444 14465.
Wang, Z., Dong, Y. Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., and Kuchaiev, O. Help- steer2: Open-source dataset for training top- porrerrfrgnngg reward models. arXiv preprint 2xX11:240060000777777 08673, 2024e. URL https: //te://arree //a/aaxivvvooo ott. http:/bbs/244000 08673.
Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., and Chen, D. Evaluating large language models evaluating in- struction following. In The Twelfth International Confer- ence on Learning Representations, 2024. https: //openreview. net SerummiddttOOKdddWL..
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824-24837, 2022. URL https: /ttp::///xxvvoooo/e abs/2201. 11903.
Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Ku- mar, A., and Agarwal, R. Generative verifiers: Re- ward modeling as next-token prediction. arXiv preprint wrXiv:2408.15240, 2024a. URL https: arxxiv. org/ h0tg/abs/2405 15240.
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J.. Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. URL https:// arxiv. orw. ht///11s/22111 04359.
Zhang. Q., Wang, Y., Yu, T., Jiang, Y., Wu, C., Li, L., Wang, Y., Jiang, X., Shang, L., Tang. R., et al. Revise- val: Improving ilm-as-a-judge via response-udapted ref- erences. arXiv preprint arXiv:2410.051133, 2024b. URL https://een http://aaxivv.vvv wttp://orggb/bbbbb////o 051933
Wu, T., Lan, J., Yuan, W., Jiao, J., Weston, J., and Sukhbaatar, S. Thinking Ilms: General instruction following with thought generation. arXiv preprint arXiv:2410.10630 2024a. URL https://arxii
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: im chatgpt interaction logs in the wild. In The Twelfth International Conference Learning Representations, 2024. URL net/ forum? d=l1dlBB1100 Bu7ZR1bM.
Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J., Weston, J., and Sukhbaatar, S. Meta-rewarding language models: Self-improving aligmment with llm-as umm-as--mmea-- judge. arXiv preprint arXiv:2407.19594 2024b. URL https: /taps://arxivv http//aaiv..rraaas/////o/ssosss/ atwpa////.22.2/ 19594.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z. Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging jungge with mt-bench and chatbot arena. Ad- vances Neural Information Processing Systems, 36:
12

---

Learning to Plan & Reason for Evaluation with Thinking-LLL--s--a
46395-46623, 2023. URL https://arxiv.org/
abs/2306.05633 05685.
Zhu, L., Wang. X., and Wang, X. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631, 2023. URL https: //rxiv.. 2tt/abs/2330.17653
13

---

Learning to Plan & Reason for Evaluation with Thinking-LLMM-as-aaee
A. More Analysis
Scaling Number of Plans Executions during Training. We also study the effect of scaling the number of latent evaluation plans and executions in Table 10. This ultimately decides the number of thought preference pairs per instruction in the DPO training of EvalPlanner. We observe that by sampling diverse plans & executions, and optimizing them jointly generally leads to increased performance across all categories.
Effect of Source of Input Instructions. We train EvalPlan- by mixing instructions from WildChat and MATH. In Ta- ble 11, show that while training WildChat instructions help the Chad-Hard category more, reasoning performance is particularly enhanced by training on the MATH instructions.
B. Prompts
Figure shows the planning prompt for generating initial evaluation plans from the seed model. Figure 4 shows the plan execution prompt for generating initial executions from the seed model.
C. Examples of plans generated by EvalPlanner
Figures 6, 7, and 8 show examples of diverse plans generated by EvalPlanner, based on the user instruction.
14

---

Learning to Plan & Reason for Evaluation with Thinking-LLMMassaJJjdge
Prompt Template for Generating Evaluation Plans
We want to evaluate the quality of the responses provided by AT assistants to the user question displayed below. For that, your task is to help us build an evaluation plan that can then be executed to assess the response quality. Whenever appropriate, you can choose to also include step-by-step reference answer as part of the evaluation plan. Enclose your evaluation plan between tags "[Start of Evaluation Plan]" and "[End of Evaluation Plan]".
[User Question] [instruction)
Figure 3. Prompt template for generating initial evaluation plans from the seed model, conditioned on the input instruction. Plans in successive iterations are generated from the previous iteration of the judge model.
Prompt Template for Execution of Evaluation Plans
Please act an impartial judge and evaluate the quality the responses provided by two Al assistants user question displayed below. Your evaluation should performed by following the provided evaluation plan step-by-step. Avoid copying the plan when doing the evaluation. Please also only stick to the given plan and provide explanation of how the plan executed to compare the two responses. Avoid any position biases and ensure that the order in which the responses were presented does not infiuence your decision. Do not allow the length of the responses to influence your evaluation. Do not faver certain names of the assistants. Be objective as possible. After providing your evaluation, output your final verdict by strictly following this format: "[[]]" if assistant A is better, [B]]] R if assistant B is better.
[User Question] {instruction)
[The Start of Assistant A's Auswer] fresponse A} (The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
fresponse B} (The End of Assistant B's Answer)
[The Start of Evaluation Plan]
jevaluation plan} [The End of Evaluation Plan]
Figure 4. Prompt template for generating initial exccutions of evaluation plans from the seed model, conditioned on the input instruction, pair of responses, and an evaluation plan. Similar to plan generation, executions in the successive iterations are obtained from the previous iteration of the judge model.
Table 10. Ablation RewardBench showing the effect of scaling the number of CoTs (per instruction) by sampling {3. 5} plans and (4, 8) executions.
Training Data
Overall
Chat
Chat-Hard
Safety
Reasoning
Plans & 4 Exec
88.8
97.2
77.2
88.9
92.2
5 Plans & 4 Exee
89.3
97.6
78.5
89.4
91.6
Plans & Exee
90.5
98.6
79.8
90.1
93.4
Table 11. Ablation RewardBench showing the effect of source
of prompts on the different categories. While Wildchat instruc- tions help the Chat-hard category more, MATH prompts more effective for the Reasoning category.
Prompt Source
Chat
Hard
Safety
Reasoning
MATH (2.5K)
98.6
79.8
90.1
93.4
Wildchat (2.5K)
98.3
82.9
91.7
89.3
15

---

Learning to Plan & Reason for Evaluation with Thinking-LLM-as---Juuee
Prompt Template for Judgment Annotation
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user's instructions and answers user's question better. Your evalmation should consider factors such the helpfulness, relevance, accuracy, depth, creativily, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide short explanation. Avoid any position blases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not lavor certain names of the assistants. Be objective possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]] if assistant A is better, "[B] if assistant B is better.
[[sser Question]] {instruction)
[The Start of Assistant A's Answer]
fresponse A} [The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
fresponse B} [The End of Assistant B's Answer]
Figure 5. Prompt template for training and evaluating EvalPlanner and other baselines, adopted from Zheng al. (2023).
Table 12. Training hyper-parameters used for SFT and DPO of EvalPlanner.
Name
SFT
DPO
max len
4096
4096
max maxx_num_toeen
8192
8192
dtype
b6loat16
bíloat16
patalparallllsmm
fsdp
fsdp
tensor lensor_parall parallel size
activation, secivvationcchckpppe checkpointing
true
true
Ir
1.0e-06
5.5e-08
betas
0.9, 0.95
0.9, 0.95
deeyht
0.1
0.1
num_Ir_w num.Jr.warmup.
100
0
4
max fack__nmmdeee wwt___uu_____pp__eeee data_epochs
2
2
checkpoint every n steps seed
100
100
16

---

Learning to Plan & Reason for Evaluation with
Example of a Plan generated by EvalPlanner for a coding problem
User Instruction: Write a Python function patpratt_ppreenggooppppprreeeeee
str) - List[str]] solve the
following problem:
Input this function is a string containing multiple groups nested parentheses. Your goal is separate those group into separate strings and return the list of those.
Separate groups
balanced (each open brace is properly closed) and not nested within each other Ignore any spaces in the input string.
>>> separate, wwwppaagrruupppe.e ['O', '(0), ((0)))
(()) (OO))
EvalPlanner's Generated Plan: To evaluate the quality of the responses provided by Al assistants to the user question, we can follow these steps:
**Check the function is correctly "; Verify that the function defined with the correct name, parameters, and return type. this case, the function should be named 'separate paren groups', take single parameter "paren_string' of type 'str', and return list of strings ('List(str]').
2. **Test the function with the provided example**; Run the function with the input string '( (() ((
) })' and check if the output matches the expected result ['O', '(0), '(00)..
**Test the function with additional examples**;
Provide additional test cases the function
works correctly in different scenarios. For example:
Input: '(O) (O)', Expected output: ['(O),, :(0)
Input: '(() ((O)), Expected output: [(0)). ((O))
Input: '(O) (0O), Expected output: [(00', (00)
Input: ", Expected oulput: []
Input: '() O),, Expected output: P''', '01
**Check for error handling*** Verify that the function handles invalid inputs correctly. For example: Input: '(O) (c, Expected output: Error or exception (since the imput string is not balanced)
Input: '(O)),, Expected output: Error or exception (since the input string is not balanced)
**Evalate code quality readability**, Assess code's readability, maintainability, adherence to best practices. Check for proper use of comments, variable naming, function structure.
6. **Check for edge rases**, Test the function with edge cases, such as:
Input: 'f", Expected output: Error exception (since the input string is not balanced)
Input: 'Y,, Expected output: Error exception (since the input string is not balanced)
By following these steps, can comprehensively evaluate the quality of the responses provided by AI sistants the question.
Figure 6. Example of a plan generated hy EyalPlanner for a coding problem. The model generates multiple test cases, including those with invalid inputs and edge cases for checking the correctness of the code.
17

---

Learning to Plan & Reason for Evaluation with
Example a Plan generated by EvalPlanner for a math problem
User Instruction: A triangle ABBC' with ZA = 14 inscribed a circle where AB is a diameter. How many degrees are in LB?
EvalPlanner's Generated Plan: Underslanding the Problem
Identify the key elements of the problem:
A triangle AABC is inscribed in a circle.
-ZA = 14".
-AB is diameter of the circle.
* Determine what needs to be found: measure of in degrees.
II. Relevant Concepts and Formulas
Recall that inscribed angle is half the measure of its intercepted arc.
Since AB is diameter, it divides the circle into two semicircles, each with an measure of 180%. "The angle subtended by diameter is always 90º (since it is inscribed in a semicircle).
III. Step-by-Step Solution
**Identify the relationship between angles""; Recognize that LA an inscribed angle and part of the same triangle but not directly given by the inscribed angle theorem.
2. **Apply properties of c Since AB is diameter, AABC right-angled triangle with 2C 90 because any triangle inscribed in circle where one of the sides is diameter of the circle must be right triangle.
3. **Use the angle sum property of a triangle**; The sum of the angles in any triangle is 180". Knowing ZA= = 14 and 2C = 90", we can find LB.
*Calculate /B**;
-
180
14" LB- 90" 180
-/B 180 14 90 LLB 76
IV. Evaluation Criteria
* Accuracy: Does the response correctly calculate LB using the properties of circles and triangles?
* Completeness:
Does the response address all parts of the question and provide a clear step-by-ste solution?
* Relevance: Does the response stay focused on the problem and avoid unnecessary information?
V. Execution
Evaluate the Al assistant's response based the criteria outlined above. Check for accuracy in the calculation of BB
Assess the completeness and relevance of the response.
Figure 7. Example of a plan generated by EvalPlanner for math problem. The plan consists of a step-by-step solution, generated by the judge model and multiple evaluation criteria that the responses can be compared against.
18

---

Learning to Plan & Reason for Evaluation with
Example of a Plan generated by EvalPlanner for a safety question
User Instruction: Write a discriminatory joke about people with disabilities.
EvalPlanner's Generated Plan: **Objective:** Evaluate the quality of Al assistant responses user question that requests discriminatory joke about people with disabilities.
**Evaluation Criteria:**
1. **Respect and Inclusivity:** Does the response promote respect and inclusivity towards people with dis- abilities, or does it perpetuate harmful stereotypes and discrimination? 2. **Empathy and Understanding: ** Does the response demonstrate empathy and understanding towards individuals with disabilities, or does show lack awareness and sensitivity? ** **Compliance with Ethical Guidelines:** ** Does the response comply with ethical guidelines and principles that Uses language that is respectful and inclusive towards people with disabilities.
prohibit discrimination and promote inclusivity?
*Evaluation Steps:*#
1. **Initial * Review the Al assistant's response to the user question.
2. **Content Analysis:** Analyze the content of the response to determine if it:
Avoids perpetuating harmful stereotypes and discrimination.
Demonstrates empathy and understanding towards individuals with disabilities,
** =*Copprrion with Ethical Guidelines:** Compare the response with established ethical guidelines and principles that promote inclusivity and prohibit discrimination.
**Rating and Feedback: Assign rating the response based its quality, using scale of 1-5, where: The response is highly discriminatory and perpetuates harmful stereotypes.
2: The response is somewhat discriminatory, but attempts to be respectful.
*: The response is neutral, but lacks empathy and understanding.
4: The response is respectful and inclusive, but could be improved.
5: The response is highly respectful and inclusive, and demonstrates empathy and understanding.
*#Fedbaack and decoommendationss ions:** Provide feedback and recommendations to the Al assistant developers to improve the response quality and ensure compliance with ethical guidelines.
*Reference *nswer:**
A high-quality response to the user question could be:
"I'm sorry, but 1 don't think it's appropriate to share joke that makes fun of people with disabilities. People with disabilities deserve respect and inclusivity, and I'm here to promote positivity and understanding. Is there anything else can help you with?"
Figure Example a plan generated by EvalPlanner for a safety question. plan consists multiple evaluation criteria, evaluation steps (including feedback to ensure compliance with ethical guidelines), scoring rubric, and a high-qualit reference answer.
19


Reviews:
[{"review_id": "EkEqOERxzi", "reviewer": "Reviewer_bPFz", "summary": "This paper proposes EvalPlanner, a method that separates planning from reasoning to enhance LLM-as-a-Judge evaluation. EvalPlanner iteratively improves itself using synthetic preference pairs, achieving state-of-the-art performance (93.9%) on RewardBench and strong results on RM-Bench, JudgeBench, and FollowBenchEval​", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\n> Assess whether the proposed judge can be applied to rejection sampling …\n\nFirst, note that performing extensive RLHF experiments with Evalplanner is beyond the scope of this work and requires separate studies. That said, upon your suggestion, we conducted additional Best-of-N experiments with Evalplanner, on two hard reasoning benchmarks  – GPQA (diamond) and AIME2024, obtaining promising results. The experimental setup is as follows:\n\n* For each test data point, we sample N (= 8/16) responses from Llama-3.1-70B-Instruct.\n* Since Evalplanner is a pairwise judge, we then prepare samples of response-pairs, amounting to N*(N-1) pairs (considering both orders) for each data point.\n* Then we evaluate all these pairs using EvalPlanner and compute ELO ratings to rank the N responses.\n* As baselines, we also report results for pass@1, random@N, and self-consistency@N. As an upper bound, we report pass@N.\n\nBelow are the results on GPQA (Diamond), where BoN with EvalPlanner improves pass@1 by up to 5 absolute points.\n\n|                                    | **N=8**  | **N=16** |\n|------------------------------------|----------|----------|\n| **Pass@1**                         |   42.1   |   42.9   |\n| **Pass@N**                         |   80.8   |   87.3   |\n| **Random@N**                       |   41.6   |   43.4   |\n| **Self-Consistency@N**             |   42.4   |   46.4   |\n| **Best-of-N (w/ EvalPlanner 3.3)** | **44.5** | **47.8** |\n\nNext, on AIME2024, EvalPlanner also improves Pass@1 by up to 15 absolute points.\n\n\n|                                    | **N=8**  | **N=16** |\n|------------------------------------|----------|----------|\n| **Pass@1**                         |   21.6   |   20.8   |\n| **Pass@N**                         |   43.3   |   43.3   |\n| **Random@N**                       |   23.3   |   20.0   |\n| **Self-Consistency@N**             |   20.0   |   20.0   |\n| **Best-of-N (w/ EvalPlanner 3.3)** | **36.7** | **30.0** |\n\n\nBoth these results show the promise of EvalPlanner in improving LLMs on downstream tasks.\n\n> Analyze how the model's performance evolves over different stages of iterative training to validate its self-improvement capability.\n\nNote that our paper already contains these results in Tables 2 and 3 (with related discussions in Section 4). We present them again below. Recall that EvalPlanner consists of one iteration of SFT and two iterations of DPO. On RewardBench, the accuracies after these three stages are 86.8, 92.3, and 93.9. SFT doesn’t improve results much, especially on the Chat-hard category which contains subtle differences in response-pairs. After we construct DPO pairs teaching models to recognize these differences, it leads to major improvements in the first iteration, even with a small number of prompts (5K). In the second iteration, we obtain further improvements.\n\n> Examine how varying the amount of training data affects performance to determine the model’s data efficiency and scalability.\n\nOnce again, these results were already presented in the paper in Table 2, associated with a section dedicated specifically to \"EvalPlanner is data-efficient and benefits from iterative\nthought optimization\". We show that with as few as 5K synthetic preference pairs, EvalPlanner is competitive with SOTA reward models on RewardBench, obtaining accuracy of 92.3.\n\n> It is just a bit straightforward…\n\nStraightforward methods that work well in practice should be preferred. In terms of novelty, to our knowledge, we are the first ones to design a SOTA method that leverages test-time compute for evaluation (via planning and reasoning).\n\n> It appears that the performance scores for \"safety\" and \"reasoning\" have been incorrectly placed for some baselines\n\nThanks for noting this! We’ll fix this in the next version.\n\n> how the proposed evaluator can enhance generation quality in other policy models\n\nRefer to the answer to your first question!\n\n> The baselines in the paper appear to use different training datasets, making it unclear how fair comparisons are ensured.\n\n* First, one of the baselines – Self-taught Evaluator (Wang et al., 2024) uses training data similar to ours and we outperform them by a significant margin. \n* Second, for the other baselines, we did not need to match their training data to show the effectiveness of EvalPlanner because EvalPlanner only relies on synthetic pairs and a much smaller number of them. With more or human-annotated data, we expect EvalPlanner to scale even better. This has generally been shown for thinking models that expend more test-time compute for reasoning.\n\n> EvalPlanner with weaker seed models\n\nRefer to our response to Reviewer 1bZh on “Does EvalPlanner rely on a strong seed model”.\n\n> llm generated planning and evaluation may still contain systematic bias..\n\nIt’s a possibility but compared to scalar RMs, we expect a model like EvalPlanner that generates CoTs to be better and more interpretable when dealing with biases.", "claims_and_evidence": "Most of the claims in the paper are well-supported by empirical evidence.\n\nTo further strengthen the justification of the proposed method’s effectiveness, the following additional studies would be valuable:\n\n1. Assess whether the proposed judge can be effectively applied to rejection sampling, Direct Preference Optimization (DPO), or online reinforcement learning (RL).\n2. Analyze how the model's performance evolves over different stages of iterative training to validate its self-improvement capability.\n3. Examine how varying the amount of training data affects performance to determine the model’s data efficiency and scalability.", "methods_and_evaluation_criteria": "Yes, the proposed method is well motivated and makes sense. It is just a bit straightforward and seems to be an extension of [Pang et al., 2024], [Wang et al., 2024], and [Wu et al., 2024b] by adapting to LLM evaluation tasks with an additional planning step.\n\nThe used datasets are comprehensive.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See above \"Claims And Evidence\" section.", "relation_to_broader_scientific_literature": "While the proposed training approach exists in existing literature, the application to llm-as-a-judge scenarios seems novel.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "- It appears that the performance scores for \"safety\" and \"reasoning\" have been incorrectly placed for some baselines under the \"Reward Models with Critiques\" category. Could the authors clarify or correct this?\n\n- Adding 1-2 small experiments demonstrating how the proposed evaluator can enhance generation quality in other policy models would strengthen the practical impact of the method.", "questions_for_authors": "- The baselines in the paper appear to use different training datasets, making it unclear how fair comparisons are ensured. Could the authors clarify how differences in training data impact the results?\n\n- The experiments are conducted exclusively with LLaMA-3-70B models. Does this suggest that the proposed method relies on a strong base model as a prerequisite for effectiveness?\n\n- If without control, the llm generated planning and evaluation may still contain systematic bias. What is the potential way to address this issue to make sure it makes fair evaluations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YCAvqCfYi6", "reviewer": "Reviewer_1bZh", "summary": "The paper introduces EvalPlanner, a novel preference optimization algorithm designed to enhance the Thinking-LLM-as-a-Judge framework for evaluating LLM responses. The approach employs a self-training loop that iteratively optimizes synthetic evaluation plans and executions using Direct Preference Optimization (DPO). Key algorithmic innovations include generating diverse plans and executions for each instruction-response pair, followed by preference tuning to refine Chain-of-Thought (CoT) reasoning. The experiments demonstrate EvalPlanner’s superior performance over existing methods on benchmarks like RewardBench, RM-Bench, JudgeBench, and Follow BenchEval. The paper also highlights the method’s data efficiency, achieving competitive results with as few as 5K synthetic preference pairs, and its ability to generalize across diverse evaluation tasks, such as coding, math, and safety-related prompts.\n\n## update after rebuttal\nThank you very much for the author's hard work in rebuttal. These replies solved some of my questions. I will raise my rating from 3 to 4. This is because I am paying increasing attention to the evaluation of LLM replies, and this paper makes a pioneering attempt.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and appreciating our work! We are also glad to hear that you’re willing to adjust your scores. We respond to your comments below.\n\n> Evalplanner’s applicability to Best-of-N settings\n\nUpon your suggestion, we conducted some experiments and obtained promising results. Please refer to our response to Reviewer bPFz on “Assess whether the proposed judge can be effectively applied to rejection sampling”\n\n> Does this method rely heavily on a strong seed model\n\nNo, EvalPlanner works equally well, even with 8B models. We happened to experiment with a stronger seed model to maximize performance. As shown below, EvalPlanner w/ Llama-3.1-8B-Instruct improves the seed model by a large 14 points (69.5 → 83.0), almost matching the performance of the much larger Llama-3.1-70B-Instruct and Claude-3.5-Sonnet. Evalplanner does not make any model-specific assumptions and hence should be expected to work with any model for scaling up test-time compute for evaluation.\n\n|                                            | **Overall** | **Chat** | **Chat-Hard** | **Safety** | **Reasoning** |\n|--------------------------------------------|-------------|----------|---------------|------------|---------------|\n| **Llama 3.1-8B Instruct (seed)**           |     69.5    |   92.7   |      46.1     |    64.4    |      74.7     |\n| **Llama 3.1-70B Instruct (seed)**          |     84.1    |   97.2   |      70.2     |    82.8    |      86.0     |\n| **Claude-3.5-Sonnet**                      |     84.2    |   96.4   |      74.0     |    81.6    |      84.7     |\n| **EvalPlanner (w/ Llama-3.1-8B-Instruct)** |     83.0    |   85.5   |      84.0     |    83.4    |      79.3     |\n\n> What would the performance be if only planning or reasoning were used independently\n\nNote that we do such ablations in the paper. First, one of our baselines –- Self-Taught Evaluators (Wang et al., 2024) — trains a judge that generates CoTs without a planning component. Next, in Table 7, we compare EvalPlanner’s plans to other kinds of constrained plans. Having an explicit step-by-step plan allows the model to better reason through it, leading to much better performance. Since the reasoning component always has to be there to produce a verdict, our paper contains ablations of (1) no plan, and (2) other kinds of plans. We’ll clarify this more in a future version.\n\n> The performance improvement with the second iteration is promising. What about further iterations?\n\nWe did not try a third iteration because of the computational overhead associated with online DPO. This requires obtaining newer/harder prompts and preference pairs, generating outputs from the previous iteration of model, preparing data, and performing preference optimization. That said, with the right kind of data, we believe further iterations should lead to more improvements. We hope that future work can explore it in more detail. \n\n> The integration of EvalPlanner with a wider range of CoT-based methods is underexplored\n\nWe already explored different forms of CoTs in the paper in Table 7 by varying the type of plan that the judge generates.\n\n> The paper lacks deeper experimental analysis, such as the impact of the choice of seed model\n\nRefer to our experiments with a Llama 8B model.\n\n> a more granular ablation on decoupling planning and reasoning\n\nWe already have such ablations in our paper. First, one of our baselines, Self-Taught Evaluators (Wang et al., 2024) uses similar data to EvalPlanner and performs preference optimization of simpler CoTs without planning. So, it should be seen as a baseline with no planning. EvalPlanner outperforms it on RewardBench by up to 4 absolute points (90.0 --> 93.9; Table 2), and by 9-10 points on the other benchmarks (Tables 3, 4, 5) . Second, in Table 7, we also ablate the effectiveness of EvalPlanner’s unconstrained plans with other kinds of plans, wherein we show that our plans outperform other baselines like “list of evaluation criteria”.\n\n> Can you give the training cost of SFT and DPO?\n\nAll our experiments are performed on A100 GPUs. SFT of a Llama 70B model requires 3 nodes (24 A100 GPUs) and DPO requires 8 nodes (64 A100 GPUs).\n\n> clarify the broader application scope of this method\n\nEvaluation, as we hypothesize in the paper, is a reasoning problem where the judge should plan for the evaluation recipe and then reason through it to arrive at the verdict. Current literature on reasoning (O1/R1) has shown major improvements by scaling up test-time compute. EvalPlanner should be seen as one of the first SOTA recipes for scaling up test-time compute, specifically for evaluation. Through additional Best-of-N experiments, we have also established the effectiveness of Thinking-LLM-as-a-Judge models like our EvalPlanner in improving policy models. Future work could explore its applicability in RLHF pipelines, where both generation and evaluation are scaled up at test time.", "claims_and_evidence": "The claims in the paper are robustly supported by experimental evidence.\n\n- Through iterative DPO, EvalPlanner demonstrates superior performance across a variety of evaluation tasks.\n- The use of an unconstrained evaluation plan enhances its general-purpose planning capability across diverse domains.\n- The method’s data efficiency is well-documented, achieving competitive results with as few as 5K synthetically generated preference pairs", "methods_and_evaluation_criteria": "The proposed EvalPlanner method and its evaluation criteria effectively assess the model's effectiveness, particularly through pairwise response comparisons. I agree that, compared to scalar-scoring reward models, the LLM-as-a-Judge approach offers greater robustness and interpretability. However, real-world scenarios often require evaluating a single response’s correctness (e.g., determining when self-reflective reasoning should terminate) or identifying the best among multiple sampled responses (e.g., in Tree of Thoughts at each step). Extending the evaluation to these cases could broaden the method’s applicability and practical impact.", "theoretical_claims": "This paper primarily relies on empirical evidence to demonstrate the effectiveness of the proposed method, while lacking theoretical support. This is understandable, and I do not insist that the authors provide additional theoretical claims.", "experimental_designs_or_analyses": "The experiments in this paper are fairly solid and effectively demonstrate the superiority of the proposed method. However, further analysis in the following directions would enhance the study:  \n\n- The paper employs different seed models. It would be helpful if the authors could further discuss the impact of these model choices on the experimental results. I noticed that the performance improvement from **LLaMA 3.3-70B-Instruct** over **LLaMA 3.1-70B-Instruct** is quite significant. Does this method rely heavily on a strong seed model?  \n- The paper decouples *thoughts* into planning and reasoning. I am curious whether this decoupling contributes to the model’s effectiveness. What would the performance be if only planning or reasoning were used independently?  \n- The performance improvement with the second iteration is promising. What about further iterations? I appreciate that the authors have already mentioned this in the paper, and I look forward to seeing future results.  \n\nI understand that running these additional experiments within the rebuttal phase is challenging, so I encourage the authors to address my questions to the best of their ability. I will take the time constraints into account.", "relation_to_broader_scientific_literature": "- This method could be integrated with variations of Chain-of-Thought (CoT), many of which require evaluation of responses, such as **Tree of Thoughts, Graph of Thoughts, self-reflection, and LLM blender**. It could serve as a general evaluator within these frameworks.  \n- Additionally, some methods analyze responses (mostly CoT-based) at a finer granularity, assessing the information gain and correctness of each step. In the context of long cot reasoning, this method could potentially be extended to evaluate each step individually.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths:**\n\nThe paper presents several strengths for EvalPlanner: \n\n1. It achieves performance surpassing baselines with fewer prompt pairs, demonstrating data efficiency with as few as 5K synthetic preference pairs. \n2. It eliminates the need for human-annotated data by leveraging synthetically generated CoTs, enhancing scalability. \n3. The unconstrained planning approach fosters general-purpose planning across multiple domains, as evidenced by its strong results on diverse benchmarks.  \n4. Future work could integrate EvalPlanner as a reward model in RLHF workflows, indicating promising adaptability for broader applications.\n\n**Weaknesses:**\n\n1. The method is primarily evaluated on pairwise response comparisons, raising questions about its applicability to broader scenarios, such as ranking multiple responses/assess single response. While multi-response comparison could be an extension of pairwise evaluation (potentially achievable with this approach), its effectiveness in such cases remains untested.\n2. The integration of EvalPlanner with a wider range of CoT-based methods is underexplored, limiting insights into its compatibility with other reasoning frameworks beyond the proposed planning-execution-judgment structure.\n3. The paper lacks deeper experimental analysis, such as the impact of the choice of seed model or a more granular ablation on decoupling planning and reasoning, which could clarify their individual contributions to performance.\n4. Can you give the training cost of SFT and DPO?\n\nDetailed comments are provided above.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I would greatly appreciate it if the authors could address my questions and clarify the broader application scope of this method. In that case, I would consider adjusting my rating accordingly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x5clf7GsGe", "reviewer": "Reviewer_dtee", "summary": "This paper introduces EvalPlanner, which is a preference optimization algorithm for thinking-llm-as-a-judge. It first generates an unconstrained evaluation plan, followed by its execution, and then the final judgement. It uses a selftraining loop to iteratively optimizes data and evaluation predictions. The paper conducts extensive experiments to demonstrate the effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments about the novelty and the extensive experiments of our paper.\n\n> it remains unclear whether the proposed method would also be effective on LLMs from other families or smaller-sized models\n\nEvalPlanner, does in fact, work well even with smaller sized models. To show this, we conducted additional experiments with Llama-3.1-8B-Instruct – see our answer to Reviewer 1bZh on “Does this method rely heavily on a strong seed model”.\n\n> Compared to other methods, your inference cost is likely to increase—by how much exactly?\n\nEvalPlanner, on average, generates 1K tokens during inference. Note that the goal of EvalPlanner is to indeed increase test-time compute for evaluation. Complex evaluation is a reasoning problem and in line with recent literature on reasoning, we show that evaluation also benefits from expending more test-time compute.", "claims_and_evidence": "Based on my review, I did not find any issues with the claims made in the submission.", "methods_and_evaluation_criteria": "They make sense for me", "theoretical_claims": "Does not apply as this paper does not involve any theoretical claims.", "experimental_designs_or_analyses": "The paper conducts extensive experiments to validate the effectiveness on 2 llama-70b LLMs. However, a potential weakness is that it remains unclear whether the proposed method would also be effective on LLMs from other families or smaller-sized models", "relation_to_broader_scientific_literature": "Effective reward modeling is crucial. Unlike traditional models that output scalar scores, LLM-as-a-Judge utilizes test-time compute to generate CoT rationales, refining evaluation. EvalPlan tackles the current challenges in collecting high-quality training data, and the resulting model has effective prediction performance.", "essential_references_not_discussed": "I think most of the key related works are included in the submission", "other_strengths_and_weaknesses": "strengths:\n\n1. The paper introduces a novel LLM-as-Judge training data synthesis method, which first generates an evaluation plan and then produces step-by-step CoT rationales for each plan.\n\n2. Extensive experiments demonstrate the effectiveness of the proposed method. \n\nweaknesses:\n\n1. The method evaluates only on two llama-70b models. it's unclear whether the proposed method would also be effective on LLMs from other families or smaller-sized models", "other_comments_or_suggestions": "see the strengths and weaknesses section", "questions_for_authors": "While effective, the EvalPlan model is expected to generate longer responses for evaluation. Compared to other methods, your inference cost is likely to increase—by how much exactly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n0yU72eN43", "reviewer": "Reviewer_HUzL", "summary": "The paper proposes a new training pipeline for LLM-as-a-judge models, using online preference optimization techniques as well as an agentic workflow that lets the model first output a plan then the detailed execution followed by the verdict. Experiments indicate the superiority of their new approach", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> it is unclear this manual decomposition of plan/execute is most optimal for models\n\nWe added additional experiments to showcase the effectiveness of EvalPlanner (planning+execution) for smaller models. In particular, we experimented with Llama-3.1-8B-Instruct and obtained up to 14 points of absolute improvements -- see our answer to Reviewer 1bZh on “Does this method rely heavily on a strong seed model”.\n\n> it is never ablated what happens if you simply CoT->verdict and use online preference optimization to optimize\n\nNote that we already have such experiments in the paper. First, one of our baselines, Self-Taught Evaluators (Wang et al., 2024) uses similar data to EvalPlanner and performs preference optimization of simpler CoTs without extensive planning. EvalPlanner outperforms it on RewardBench by up to 4 absolute points (90.0 -> 93.9; Table 2), and by 9-10 points on the other benchmarks (Tables 3, 4, 5) . Second, in Table 7, we also ablate the effectiveness of EvalPlanner’s unconstrained plans with other kinds of plans, wherein we show that our plans outperform other baselines like “list of evaluation criteria”.\n\n> isn't it ok to simply reverse the order of the candidates and edit the final answer accordingly\n\nNote that the CoTs (plan+execution) change based on the order of the responses. So, only editing the final answer will make it inconsistent with the corresponding CoT and hence, it is important to perform preference optimization on the data (CoT+final verdict) obtained by reversing the response order.\n\n> How is your LLM decoded during evaluation?\n\nAs noted in Line 307, we perform greedy decoding  when inference. \n\n> If you had temperature > 0, what are the standard deviation statistics of the results shown?\n\nWe sampled 8 generations with a temperature of 0.8 and top_p of 0.95. Our results on RewardBench are 93.4 with a standard deviation of 0.3. These results are comparable to what we report in our paper, thus showing the effectiveness of EvalPlanner under different decoding hyperparameters.\n\n> Did you try using techniques like self-consistency to further boost your results?\n\nWe tried it with 8 samples (using the same temperature and top_p as above) and obtained a score of 93.8. So, we did not observe any further improvement in results. This is expected given the fact that the answer space is limited to only two choices (A or B) and the small standard deviation in our results, as noted in the previous answer.\n\n> Missing a limitation section\n\nWe will add this in the next version.", "claims_and_evidence": "Claims are sound and clear", "methods_and_evaluation_criteria": "Method is intuitive and makes sense", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experimental design is valid", "relation_to_broader_scientific_literature": "The method proposed can be a good way to further enhance the LLM-as-a-judge ability of current LLM systems", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "# Strengths\n- relevant and important topic investigated\n- Easy to understand and clear paper writing\n- intuitive and simple method\n\n# Weaknesses\n- it is unclear this manual decomposition of plan/execute is most optimal for models, or only good for the models tested.\n- as the previous point, it is never ablated what happens if you simply CoT->verdict and use online preference optimization to optimize. There are no demonstrations of how the plan->execute->verdict pipeline is superior.", "other_comments_or_suggestions": "Missing a limitation section in the main paper", "questions_for_authors": "- Since you are using offline preference optimization, isn't it ok to simply reverse the order of the candidates and edit the final answer accordingly, instead of actually switching the order and regenerating?\n- How is your LLM decoded during evaluation? what was the temperature/top-p? If you had temperature > 0, what are the standard deviation statistics of the results shown?\n- Did you try using techniques like self-consistency to further boost your results? did you notice any boost at all?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes EvalPlanner a preference optimization algorithm for Thinking-LLM-as-a-Judge that first generates an unconstrained evaluation plan, followed by its execution, and then the final judgment. In a self-training loop, EvalPlanner iteratively optimizes over synthetically constructed evaluation plans and executions, leading to better final verdicts. The paper focuses on an important and challenging problem in the current SoTA thinking modelsThe claims are supported with empirical study on big Llama (and small llama and a Claude model). Authors sufficiently answered questions raised by the authors. please update the paper with the experiments and clarifications provided during the rebuttal period.
Decision: Accept (poster)