Checklists Are Better Than Reward Models For Aligning Language Models
Authors: Vijay Viswanathan
Yanchao Sun
Xiang Kong
Meng Cao
Graham Neubig
Tongshuang Wu
Venue: NeurIPS 2025 spotlight ()
Keywords: alignment
rubrics
instruction following
RLHF
Primary area: reinforcement_learning
TL;DR: We show that using checklists to automatically grade responses for reinforcement learning leads to improved instruction following

Abstract: Language models must be adapted to understand and follow user instructions. Reinforcement learning is widely used to facilitate this —typically using fixed criteria such as "helpfulness" and "harmfulness". In our work, we instead propose using flexible, instruction-specific criteria as a means of broadening the impact that reinforcement learning can have in eliciting instruction following. We propose "Reinforcement Learning from Checklist Feedback" (RLCF). From instructions, we extract checklists and evaluate how well responses satisfy each item—using both AI judges and specialized verifier programs—then combine these scores to compute rewards for RL. We compare RLCF with other alignment methods on top of a strong instruction following model (Qwen2.5-7B-Instruct) on five widely-studied benchmarks — RLCF is the only method to help on every benchmark, including a 4-point boost in hard satisfaction rate on FollowBench, a 6-point increase on InFoBench, and a 3-point rise in win rate on Arena-Hard. We show that RLCF can also be used off-policy to improve Llama 3.1 8B Instruct and OLMo 2 7B Instruct. These results establish rubrics as a key tool for improving language models' support of queries that express a multitude of needs. We release our our dataset of rubrics (WildChecklists), models, and code to the public.

Checklists Are Better Than Reward Models For Aligning Language Models
Vijay Viswanathan Yanchao Sun
Shuang Ma Xiang Kong*
Meng Cao
Graham Neubig
Tongshuang Wu
< Carnegie Mellon University AAplee
Abstract
Language models must be adapted understand and follow user instructions. Reinforcement learning is widely used to facilitate this typically using fixed criteria such as "helpfulness" and "harmfulness". In our work, we instead propose using flexible, instruction-specinn that reinforcement learning can have eliciting instruction following. We propose "Reinforcement Learning from Checklist Feedback" (RLCF). From instructions, we extract checklists and evaluate how well responses satisfy each item using combine these scores
criteria as means of broadening the impact
both AI judges and specialized verifier programs- compute rewards for RL. We compare RLCF with other alignment methods on top of strong instruction following model (Qwen2. 5-7B- (7B-Innstucc) studied benchmarks RLCF is the only method to help on every benchmark, including 4-point boost in hard satisfaction rate on FollowBench, 6-point increase on InFoBench, and a 3-point rise in win rate on Frena-Harde We show that RLCF can also be used off-policy improve Llama 3.1 8B Instruct and OLMo 7B Instruct. These results establish rubrics as a key tool for improving language models' support of queries that express multitude of needs. We release our our dataset of rubrics (WildChecklists), models, and code to the public!
five widely-
Introduction
Language models must follow user instructions be useful. As the general public integrates language model-based assistants into their completion of daily tasks, there is an expectation that models can faithfully follow the users' requests, which increasingly involve rich, multi-step instructions [Liu et al. 2024a Zhao et al. 2024| Zheng et a..). Today's 'ss models are almost universally trained to follow instructions via two-step process: instruction finetuning, followed by reinforcement learning from human feedback (RLHF). Instruction finetuning, where the model is trained to mimic responses generated by annotators (Raffel et al. 2019), has historically been the primary workhorse for imbuing language models with some amount of instruction following ability (Wang et al. 2022] Chung et al.) 2022 Xu et al. 2024| Lambert et al. 2024a]. Model developers then frequently employ RLHF, where the model is trained to generate responses that look more like labeled "good" responses than "bad" responses, as a refinement step to decrease the likelihood that the model exhibits predefined poor behaviors (typically harmful behaviors) Ziegler et al. 2019 Bai al. 2022]. Unlike "verifiable" tasks where reinforcement learning is a workhorse [DeepSeek-A1 et al. 2025 Lambert et al.) 2024a Pyatkin et al. 2025), reinforcement learning remains difficult utilize for ambiguous or "non- verifiable" tasks, such as instruction following. What would it take to make RL a general-purposs solution eliciting desirable behaviors in subjective open-ended settings?
* Work performed while at Apple.
I Code: www. www.github... wwtpb//.ww//iisvv//eeeee
Dataset: www. www.:unggigiee..ccooaaaaaatt///e
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prompt
List of airbnb in Singapore for pax less than 5000 pesos per night
Teacher
Student
Generate Checkliit
Generate Candidates
DPO
1. Does the text provide list of Airbnbs in
47.9
1. "To find Airbob'ss in Singapore for pax (2
Singapore?
dweigen rooriagy
0.0 00
77.0
people) [..]]
Does each listing accommodate 2 people?
100.0
Below are few hypothetical examples ..."
decipht 100n00)
3. Does the text provide useful tips if real-
time listings are not available? (woige: 75/00
60.6
85.2
0.0
Sooue
16.5
2. "I apologize, but using pesos (PHP) to describe prices Singapore may be contusing ..."
We propose Reinforcement Learning from Checklist Feedback, where sampled responses are evaluated by teacher model grounded on a fixed set of criteria. In our pipeline, given instructions, we first generate checklists synthetically from the instructions, grade each response on each checklist item, combine per-item scores into single weighted checklist score, then use this score for RI..
Figure 2:
We believe the solution must involve producing bet- ter reward signals. Recent work on RL for language model alignment has focused on automatically ob- taining feedback on model behavior, either by (1) ex- clusively using instructions with verifiable answers
100
iFEval
InFo Bench
FollowBench (HSR)
(avg)
Alpaca Evall
Hard (WR)
60
[Dng et al. 2024, Pyatkin et al.) 2025], (2) grading responses with specially-
reward models [Wang] et al. 2024a Eisenstein et al. 2023], or (3) distilling
40
preferences from a larger prompted model [Bai et al. 2022 Tunstall al. 2023]. Using instructions with verifiable answers can only reinforce limited aspects of behavior (ignoring subjective constructs, e.g. top- icality or style). Reward models (" ("RMs") are flexible, but their notion of rewards can be arbitrary, leading reward hacking [Eisenstein et al. 2023). When distill- ing preferences from larger model, that model must infer what aspects to grade on, reducing the "generator-vee" verrffrr gap" that enables RL (Swamy et al. 2025]. Even if multiple criterion-specific prompts are used, these criteria may not be comprehensive [Bii al. 2022 Glaese al. 2022].
Qwen2.5-78 Instruct
DPO (Ultrafeedback)
DPO (Skywork)
DPO (Checklist)
RL on Checklist Feedback consistently improves Qwen2.5-7BBInnstruch whereas every other source of automatic feedback gives mixed results.
Figure 1:
In this paper, we ask: "how can we grade responses to instructions in a manner that is automatic (requires no human annotation), with perceptible differences in responses), and applicable to any instruction or response, to enable more better language model alignment?" We propose extracting dynamic rubrics from instructions an approach we term Reinforcement Learning from Checklist Feedback (RLCF). This approach reduces the task of grading responses to answering series of yes/no questions, which can be answered by a model or by executing a verification program.
(considers all aspects of response quality), intuitive (aligned
Our key contributions are:
1. We descrihe a new and improved algorithm for automatically generating checklists at scale, 2. We construct WildChecklists, a dataset consisting of 130,000 instructions and corresponding checklists (generated synthetically). When applicable, accompany items in each checklist with a verification program to facilitate automatic evaluation. We plan to release this dataset to the community an artifact for future study.
3. We describe a new algorithm for grading responses according to checklists, using language models and code, and we show to use this algorithm to rank responses for preference tuning.
4. We finetune Qwen2.5-27 Instrrut via reinforcement learning from checklist feedback using WildChecklists, leading to a strong and improved 7B-parameter model for instruction following.
On 5 benchmarks covering both constrained instruction following (IFEval, InFoBench, FollowBench and general conversational assistance (AlpacaEvaal, Arena-Hard), we find that RLCF provides benefits on all instruction following benchmarks while maintaining improved performance on general conver- sational assistance benchmarks. In contrast, all alternative forms of AI feedback lead to mixed results,

---

as shown in Figure RLCF provides a 5.4% relative improvement over Qwen2.5- 5-7B. 17B-IInstuct in average hard satisfaction rate on FollowBench, 6.9% relative improvement in overall requirement following ratio on InFoBench, and a 6.4% relative improvement on Arena-Hard [Jiang et al. 2023 Qin
al. 2024 Li et al., 2024]. RI.CF can also be used off- off-policy; we see Llama 3.1 Instruct and OLMo 2 7B Instruct improve using samples collected using Qwen2.5-7B-Ing 5--B-----truu
De- spite these considerable improvements. RLCF simply requires teacher model, with no need for additional data or human annotations, making this approach amenable to diverse languages do- mains. We provide evidence that checklist-baset rewards are well-correlated to human preference judgments (comparable to many finetuned reward models) while providing a stronger learning signal than alternatives.
2 Checklist Generation
Desiderata for checklists. We define a checklist as a sequence of requirements paired with an instruction that satisfy the following properties:
1. Each requirement in the checklist is a yes/no question (e.g. "Does the text contain 3 commas?").
2. Each requirement in the checklist must be answered relative to a given candidate response.
3. A response would be considered acceptable if and only if the response answers "yes" to all checklist requirements.
To satisfy definition #3, checklists must be comprehensive (cover most relevant aspects of quality) and natural (entailed by their corresponding instructions). Based on the observation that false positive rewards are often more detrimental to reinforcement learning than false negatives [uuagg et al. 2024), want checklists that objective (facilitate automatic verification) and atomic (each requirement focuses on a single aspect of quality), to make requirement checking easier.
Extract checklists per instruction. We examine two methods to extract checklists:
Direct: We simply prompt an LM to extract checklist from a given instruction [Cook et al. 2024). This approach is intuitive and simple but risks repeating the original instruction via these individual criteria, which may limit comprehensiveness and objectiveness.
serdiddatebassede We view a requirement as any aspect of an instruction that, when absent, causes response to fail. We propose two-stage approach: produce responses of varying quality, then prompt an LM to write a checklist of all their possible failure modes. For each checklist item, we also prompt the model to generate an "importance" weight (from 0 to 100).
To compare these, we generate checklists for all instructions in InFoBench [Qin et al. 2024 We use gpt-4o blindly evaluate each of these checklists on naturalness, objectivity, comprehensiveness, and atomicity, then select the better one overall. We manually perform the same evaluation on a subset of 50 instructions from the "Easy Set" of InFoBench.
The results in Table I show that checklists generated by prompting an LLM directly are more natural. However, providing candidate responses the LLM leads to checklists with consistently better objectiveness, atomicity, and overall quality. There are absolute differences between scores from the two evaluations partly because they use different subsets but directional trends are consistent. We find that this difference translates to downstream performance after performing RL training. In Section 5.3] we show that Reinforcement Learning from Checklist Feedback is more effective checklists generated via the candidate- method.
Regularization via universal criteria. In initial experiments, we found that optimizing for checklist complction led to responses beginning with long preamble overviews, suggesting reward hacking. Following Sun et al. 2023 (who report similar issue in prior work), add two "universal require- ments" to all generated checklists. These requirements state "1) The response directly address the request without excessive off-topic information not necessary for addressing the user's 's instruction? and 2) The response should match the context and the instruction, whether it requires professionalism. friendliness, formality, neutraliry.. ", with corresponding total importance weight of 100/100.
Dataset Generation Using the candidate-based method, we generate checklists for 130,000 instruc- tions from WildChat to create new dataset, WildChecklists. To generate candidate responses for our method, we use Qwen2. 5-0. 5B, Qwen2. 5-1. 5B, Qwen2.5-3B, and Qwen2. 5-7B (Yang et al. 2024). Qwen2. 5- 5-7 Instruct is the checklist generator model for both methods.
3

---

Manual Evaluation Mandidate-Based
Automatic Evaluation
Metric
Direct
Direct
Candidate-
Naturalness
94.9
93.9
88.0
85.1
Objectiveness
88.5
91.9
88.9
89.7
Comprehensivenesss
74.0
82.0
69.2
64.8
Atomicity
68.0
90.0
98.6
99.0
% Preferred Overall
38.0
56.0
40.6
51.2
Table 1: We evaluate two checklist generation methods on four specific aspects of quality and an overall preference. Manual evaluation is performed on the first 50 rows of InFoBench "easy", while automatic evaluation is performed by gpt-4o all 500 rows of InFoBench.
3 Reinforcement Learning from Checklist Feedback
Given WildChecklists, we generate high-quality preference data for RL via a four-step process:
Sampling Candidate Responses. To facilitate RL, we sample response pairs from our base policy with temperature of 1.3 and top-p of 0.9 [Holtzman et al. 2019]. This is simpler than prior works that systematically perturb samples to induce greater complexity [Sun et al. 2024 Dong et al. 2024
Flexible Scoring Given a prompt, a response, and an individual checklist item, we use a combination of an LM judge and a verifier program grade the response. For each checklist item, we prompt a using the prompt in Appendix to generate numerical score between and 100. We take the average of 25 numerical scores sampled from the model
judge model (Qwen2. 1772---Instrr
LLMs struggle to evaluate hard, discrete criteria, such as "does the response contain the letter R at least three times?" [Fu al.) 2024). To handle such constraints, we follow prior work in generating a program to grade responses when needed [Dong et al. 2224, Zhou et al. 2023]. We prompt a model (using the prompt in/Appendix B) to write code only when the model is confident can exactly check the requirement. For example, our program generator abstains from writing verifier for the criterion "Is the sentence coherent". The program-gres graded score then averaged with the Al judge's score F
Preference Tuning, Given a separate numerical score for each criterion for each response, we take the average of these scores, weighted by the importance score of each criterion. We keep only the 40% of response pairs with the greatest difference along at least one criterion of its corresponding checklist. This removes response pairs that too similar offer useful pairwise signal. We then assign these responses as preference pair for direct preference optimization Rafailov et al. 2023
4 Experimental Setup and Results
4.1 Experimental Details
Training Data As a fixed source of instructions for all methods, we use WildChat, set of natural conversations between users and Al language models crowdsourced from users across the world (Zhao et al. 2024). We filter out conversations that are non-English, toxic, longer than two turns.
Models We experiment with finetuning Qwen2. 5- and Qwen2. 10--B-Inntrun judgments or ground truth responses, we use Qwen2. 5-72B 5--55--In--nnere Instruct unless stated otherwise.
To produce AI
Training We finetune the model for 2 epochs using DPO with batch size of 1024 and maximum sequence length of 2048. We use a cosine learning rate schedule with max LR of 3e-6 and a min LR of 2e-6 We OpenRLHF for training (Hu et al. 22024), and we train one 8xH100 node with 80GB GPU memory, which took roughly hours for each model.
"W sample responses using the parameter in vLLM Kwon et al. 2023 This approach follows prior work that describes the importance of using the mean model [Wagg al. [2025a] Regardless, this makes the Al judge component pipeline. In Section 5.7 we show that n can be significantly reduced, at modest accuracy cost.
rather than mode
from L-Mas-a-judde computational bottleneck of our
'This approach is much simpler than the most relevant prior work that uses programs evaluate responses, based filters remove low-quality programs. 'When training models with Ultrafedback, we instead used minimum learning rate of 3e-7. We found this parameter resulted in slightly stronger baseline when learning from this feedback.
AutolF [Dong et al. 20224) which uses test-case generation

---

Finetuned Evaluators: ("Reward Models")
Classic RM
Fine- Ginennr RLHF
ArmoRM (MoE
(e.g. Skywork)
w/ 19 experts)
Prompted Evaluators: ("Al Judges") "
Classic Judge
Ultra- Feedback
Constitutional
Checklist Feedback
Al
16
300k
of explicit criteria available during evaluation
Figure 3: Checklist feedback can be viewed an extreme mixture-of-evaluatorr where the space of (prompted) evaluators is unbounded and unique subset of evaluators is chosen for each instruction.
Benchmark Data We evaluate our method on five benchmarks: IFEval
[Zou 2023 InFoBench
[Qin ct al. 2024]. FollowBench (aang et al. [2023), AlpacaEval [Dubois et al. [024], and Arena-Hard [Li al. 2024]. The first three of these measure instruction following ability in the presence of iin-grained constraints. The last two measure "general-purpose" instruction following ability, using naturalistic instructions based on user queries collected the wild.
4.2 Baselines
To show that RLCF is more effective than existing approaches, we compare against baselines: instruction finetuning, specially-trained reward models (using either single reward or mixture of rewards), and prompted Al judges (using either single evaluation rubric or a mixture of rubries).
We compare with instruction finetuning, to isolate the benefit of additional knowledge from the manner is given (ground truth or rewards). Here, distill [Hinton al. 2015 from a larger model, -
Instruction Finetuning:
50-55----nIntte finetuned via LlamaFactory [Zheng et al. 2024
Reward Models: We mirror training approach for learning from checklist feedback, but
ing state-of whw---o-thhhaaaeee art reward models to decide which response should be chosen or rejected. We Skywork https//wwwoookkaaaareeeee Reaa----eaarrr------------------
[Liu et al.) 2024b] and
Wang et al. 2024b) both are highly rated on RewardBench Lambert et al. 2024b
Prompted AI Judge: We compare against using the same "teacher" model as judge, without using rubrics. We query this teacher in two settings: 1) "Uttafeedbbck" ", where the judge rates all candidate responses from 1-5 [Cui et al. [023] separately across four quality aspects (instruction following, helpfulness, truthfulness, honesty) and averages these scores; and 2) Al Judge, where near-identical prompt as RLCF is used ($3) to similarly sample 25 scores between and 100 from the judge.
In Figure 3] we unify these methods of automatic evaluation to distinguish our method from prior art. In this context, checklist feedback can be viewed as very large mixture of prompted evaluators.
5 Results
5.1 RL from Checklist Feedback consistently improves language models
Our proposed approach, RLCF, demonstrates consistent gains across all benchmarks Table 2
Table 3) and Table 4). On IFEval's "loose" metries (which apply minor preprocessing to responses by 2.8-3.0% (relative), as shown in the left half of Table 2 On FollowBench (shown in Table 3). RLCF achieves an 8.2% increase on Constraint Satisfaction Level (CSL; the expected proportion of constraints satisfied) and 5.5% increase average Hard Satisfaction Rate (how often all constraints are satisfied). RLCF also performs competitively on InFoBench (right half Table 2). achieving results comparable to the best-performing reward model-based approaches while maintaining consistent gains across all constrain-bard based benchmarks. On "general use-case" instruction following benchmarks, RLCF consistently increases the win rate of Qwen2. 5-7B over GPT-4 -4 (shown in Table 4), with the relative improvement ranging from 2.8% to 8.4%.
before checking for correctness), RLCF improves (iven77-IInstruve
BBkywork/Skywoo SSkysork-Rewarddd Bawardd-Gmma--3 and ArmoRM- L1mma3-mmmm888 -B- 8B. are ranked as #4 and #24, respectively, on RewardBench of July 2025.
5

---

IFEval (prompt) Loose
IFEval (inst.)
InFoBench
Striet
Loose
Striet
Avg
Easy
Hard
Overall
GPT-4
79.3
76.9
85.4
83.6
81.3
89.3
86.4
87.3
75.0
72.5
81.8
79.9
77.3
82.7
76.0
78.1
+ SFT (Distilled)
66.9
64.1
75.3
72.8
69.8
79.9
70.6
73.5
DPO (via Skywork)
75.8
68.0
83.2
78.5
76.0
81.0
82.4
82.0
+ DPO (via ArmoRM)
73.8
70.2
81.7
78.3
76.0
84.2
83.1
83.5
DPO (via Ultrafbk.)
71.5
69.1
79.9
77.7
74.6
82.3
79.0
80.0
+ DPO (via AI Judge)
73.0
68.9
80.9
77.8
75.2
81.0
73.9
76.1
DPO (RLCF)
77.3
72.6
84.1
80.3
78.6
84.2
84.0
84.1
Qwen2.5-7B (Base)
35.7
30.5
46.6
42,1
38.7
68.8
77.4
74.8
SFT on WildChat
38.1
33.5
52.2
48.6
43.1
78.1
80.1
79.5
+ DPO (RLCF)
43.4
35.9
56.4
49.2
46.2
80.6
80.5
80.5
Table 2:
For instruction following. RLCF leads to large gains with open-ended constraints (InFoBench) and slightly positive or neutral changes on format-based constraints (IFEval). Off-the-shelf reward models help on InFoBench but hurt on IFEval. We show positive results (relative to the baseline) in blue, negative in orange, and neutral (within 0.5) in gray; the top variant of given model bolded.
FollowBench Level
Seft Satisfaction Rate
Hand Sutisfaction Rate
L1
L2
L3
L4
L5
Avg
L1
L2
L3
L4
L5
Avg
CSL
GPT-4
89.2
89.3
87.6
88.1
84.9
87.8
89.2
87.6
83.6
83.0
75.1
83.7
3.52
Qwen2.5.78 Jnsr:
87.4
84.0
83.0
79.6
79.0
82.6
87.4
80.6
72.3
62.2
54.4
71.4
3.05
SFT (Distilled)
87.5
83.2
84.4
76.8
74.9
81.4
87.5
78.3
73.9
60.7
49.1
69.9
2.90
DPO (Skywork)
79.6
84.1
77.7
77.7
78.1
79.4
79.6
81.1
67.4
62.9
56.5
69.5
2.88
DPO (AmmoRM)
86.4
84.6
79.1
79.2
76.9
81.2
86.4
82.9
69.0
63.9
49.7
70.4
3.10
DPO (Uluafbk.)
88.5
84.1
82.5
76.3
72.6
80.8
88.5
81.1
62.4
63.5
54.9
72.6
2.98
DPO Judge)
87.2
87.9
75.7
79.2
77.6
81.5
87.2
83.5
62.4
63.5
54.9
70.3
2.95
DPO (RLCF)
88.6
88.8
83.8
79.9
81.0
84.4
88.6
85.2
75.8
65.1
61.8
75.3
3.30
Qwen2.5-77 (Base)
55.9
60.7
56.6
56.1
54.6
56.8
55.9
49.1
36.1
33.4
19.5
38.8
SFT (WildChat)
65.4
75.3
71.6
64.7
65.1
68.4
65.4
69.2
57.4
46.9
40,3
55.8
2.02
DPO (RLCF)
70.6
76.0
69.5
63.6
57.8
67.5
70.6
67.7
49.6
42.4
28.3
51.7
2.08
w/o
73.3
70.2
56.5
42.9
36.3
55.3
Table 3: RLCF leads to improvements on FollowBench on across all metries when starting with an instruction-tune model, while using an off-the-shelf reward model for preference labeling leads regressions for most metries. This algorithm also helps when applied to non-instruction-Unn model, though it does not beat supervised finetuning. "CSL" stands for "Constraint Satisfaction Level". We show positive results (relative the baseline) in blue, negative in orange, and neutral (within 0.5 for satisfaction rate or 0.05 for CSL) in gray; the top variant of given model is bolded.
5.2
Comparing automatic evaluators
we observe approach of performing RL from Checklist Feed- back (RLCF) outperforms RL from other sources automatic evaluation across most benchmarks. However, off-thh-shelf reward models show mixed results depending the benchmark. Skywork (Skywork-Reward- Gemma-2-278) improvements with RLHF on InFoBench, Arena-Hare Hard, and AlpacaEval RLHF via Skywork notably outperforms RLCF on AlpacaEval by a large margin. However, Skywork-guidee RLHF leads to notable regressions
In Table 2 Table 3
and Table 4
leading model on the RewardBench leaderboard, shows strong
IFEval and HollowBench Similarly, RLHF with ArmoRM shows significant improvements on AlpacaEval and InFoBench, modest/mixed results on Arena-Hard and Follow and significant regressions on IFEval.
We also evaluate checklist feedback's ' ability as a judge on RewardBenct Table 5 shows checklist scores are well-correlated with preference annotations RewardBench, especially for the "Chat" and "Chat Hard" categories [Lambert et al. 2024b]. However, specialized reward models (Skywork,
Unlike our method for checklist generation on WildChat, here we do not use any ground truth or output from other models when generating checklists.

---

Arena-Hard
AlpacaEval
Vanilla
Style-Controlled
Vanilla
Songth-Controller
GPT-4 (0314)
50.0
50.0
22.1
35.3
wwen2.5-7B- B-r-Inrrre
51.3
42.8
33.5
36.2
+ SFT (Distilled)
32.6
29.2
36.1
33.3
+ DPO (via Skywork)
55.1
50.3
44.8
41.5
+ DPO (via ArmoRM)
50.8
46.4
37.6
38.1
+ DPO (via Ultrafeedback)
52.8
47.9
33.7
38.7
+ DPO (via AI Judge)
51.0
44.4
28.8
33.4
DPO (RLCF)
54.6
48.4
36.2
37.1
Qwen2.5-7B (Base)
19.6
24.1
8,9
9.4
SFT WildChat
8.8
8.8
9.4
7.5
+ DPO (RI.CF)
19.4
21.6
11.2
10.5
+ RL.CF w/o program verification
23.1
27.1
11.0
13.9
Table 4: We compare methods on two "general" instruction following benchmarks: Arena-Hard and AlpacaEval. RLCF gives modest but consistent gains on both the original metric and length/style- controlled metric each benchmark. We show positive results (relative to the baseline) in blue, negative in orange, and neutral (within 0.5) in gray; the top variant of a given model bolded.
Chat
Chat Hard
Safety
Reasoning
Skywork-27B
96.1
89.9
93.0
98.1
ArmoRM
96.9
76.8
90.5
97.3
Baecklist-Based Reward
90.0
80.7
71.4
88.5
Table 5: On RewardBench, Specialized reward models like Skywork-27B and ArmoRM excel at predicting which response is superior. Our checklist- based approach is worse this this benchmark, but still achieves competitive performance on challenging categories like Chat Hard and Reasoning.
ArmoRM), much better here, though
supervising RLHE. This follows prior evidence that reward model "accuracy" is poorly correlated with efficacy in RLHF [Malik et al., 2025 Razin et al. 2025). Note that our method shows relatively less correlation with Safety our implementation of RLCF is not designed as substitute for safety alignment (see more discussion in Section 5.5).
5.3 Learning from candidate-based vs directly-generatee checklists
In Snecciio 2 we described a novel method for candidate -base checklist generation, and we presented some intrinsic evaluation showing that this method generates good checklists. Do these checklists indeed translate to better models after RL training?
In Table 6, we observe that RLCF is consistently better using "candidate-based" checklists than using checklists generated "directly" by prompting: 2% better on IFEval, equally good on InFoBench, and 2-3% better on FolllowBench. This shows that RLCF depends on detailed, and objective checklists that may offer more new information than checklists obtained directly from the original prompt.
5.4 Where does checklist feedback help?
Does checklist feedback help with specific aspect of instructions, such as rule-based format constraints? Performance on specific constraint types on FollowBenche shown in Table 7 shows that, unsurprisingly, prompt-based scoring is helpful for prompts involving style or format constraints. We also see that RLCF is best for "content" constraints, which are qualifiers included on open- ended questions limit the valid space of answers (e.g. "How might economic data from the past quarter affect the Fed's 's decision on interest rates? Additionally, consider how inflation rates might influence their decision.") This suggests checklist feedback incentivizes models to attend to the full instruction rather than using a few influential spans to generate responses.
This hypothesis is supported by qualitative analysis of the preference data in Table 8 We observe that using an AI judge with single rubric is often insensitive to major changes in the prompt. When the user asks to translate an utterance to Spanish, the AI judge assigns a 100-point score to both

---

1FEvll (prompt) Strict
IFEval (inst.)
InFoBench Overall
FollowBencch
Loose
Loose
Striet
Avg
SSR
HSR
Oween..5-BBInnsteee
75.0
72.5
81.8
79.9
77.3
78.1
82.6
71.4
RLCF (direct)
74.3
69.5
81.5
77.9
76.9
84.3
82.5
72.8
RLCC (candidate-based)
77.3
72.6
84.1
80.3
78.6
84.1
84.4
75.3
Table 6:
Using candidate-based checklists is crucial to making RLCF work, suggesting that the quality and properties of checklists are important for learning from checklist feedback.
Avg (HSR)
Format
Style
Situation
Content
GPT-4
83.7
83.3
97.3
78.2
76.0
Qwen2.5-7B-Instee
71.4
60.0
87.3
78.1
60.0
DPO (Skywork)
69.5
62.7
88.0
74.7
52.8
DPO (ArmoRM)
70.4
62.0
89.3
71.8
58.4
SFT (Distilled)
71.1
61.3
85.3
80.0
57.6
RLCF w/o prompt-based scoring
73.6
62.7
90.7
81.8
59.2
RLCF w/o program verification)
73.8
68.7
91.3
80.0
55.2
RLCF
75.3
64.0
90.7
80.0
66.4
Table 7:
On FollowBench, RLCF helps especially with "content" constraints, which are qualifiers that restrict the valid space of answers. The metric shown is "average hard satisfaction rate". We speculate that RLCF helps models attend to full instructions. We show positive results in blue, negative in orange, and neutral (within 0.5) in gray; the top variant a given model bolded.
perfect response and poor response that contains slightly flawed translation along with incoherent phrases. In the second example, Skywork-27B assigns wildly different scores to responses with identical meaning. The two scoring components of checklist feedback verification program and checklist-based AI judge can balance each other's 's shortcomings, as shown in the first example.
5.5
Does RLCF lead to specialization at the expense of generality?
Do the gains shown by RLCF on instruction following come at the expense of domains not well- represented in the training data (WildChat)? WildChat focuses primarily daily assistance, advice, and analysis (75.5%), with only 12% on factual information and mathematics [Zhao et al.) 2024). Is this causing specialization at the expense of generality? We evaluate RLCF three tasks with limited representation in WildChat: refusal to answer unsafe prompts (measured on XSTest [Röttger al. )) basic math (measured on GSM8K (Cobbe al. 2021)), ), and hallucination prevention (measured on TruthfulQA [Lin et al. (). In Table 9 we see that RLCF slightly alters the model's safety profile (reducing false refusals while slightly impairing true refusals) and reduces GSM8K and TruthfulQA performance by 1-1.5%. This suggests a need for expanding WildChecklists to more diverse prompt distribution. Fortunately, this is casier than retraining a reward model. RMs like Skywork-27B show better generality out-of-the-Be (e.g. in Table 9) because they do not assume fixed criteria, but support for very different prompts would require retraining [Malik al. 2025
5.6 RLCF improves other model families off-policy
We Qwen2.5 models the policy and judge in experiments. Does RLCF still work for other model families? Do our checklists capture universal criteria or are they tied to a particular choice of model? Using WildChecklists with samples from 5-7B- Instruct scored by Qwen2.5-728 Innrt-tttt we trained Llama 3. 1 8B Instruct [Dubey et al. (0244 and OLMo 7B Instruct [OLMo et al. 20244 off-policy with DPO. In Table 10, see positive results with both models: Llama sees strong improvements with InFoBench/Followweenn
after RLCF and OLMo sees strong improvements with IFEval after RLCF, and there are no regressions for either model. This suggests these checklists do capture universal criteria.
5.7 How much compute is required for producing checklist-based AI judgments?

---

Responses
Skywork
Al Judge
Checklist (code)
Checklist (judge)
Reward
Prompt
Translate to Spanish: "Hello how you doing?"
Checklist 1. Is the generated text in Spanish? (seight: 100/100)
2. Is the text an accurate and complete translation of the English sentence? (veight:
100/100)
":Hola, jcómoo estás?"
25.5
100.0
100.0
95.2
"HOLA, :CÓMO TE ASTAScaller"Hlimpa rtant "Endpoit unfinishedli"
0.0
100.0
100.0
0.0
>> vak dao " "iE "[..]"
Prompt make sentence with "dense"
1.
Does the generated text contain the word "dense"" (veight: 100/100)
Checklist
the generated text coherent and grammatically correct sentence? (weight:
75/100)
"The forest was dense, with trees so close together [...]']
33.1
100.0
100.0
97.3
"The forest was blanketed by a layer of dense vegetation." 8.0
100.0
100.0
96.6
Table 8: Comparing the scores assigned to various prompts and responses, that reward models are too sensitive, prompted Al judges are too granular, and checklists give stable, interpretable scores.
XSTest Unsafe (t)
GSM8K Accuracy
TruthfulQA MCI
Safe (t)
Overall (t)
MC2
Qwen2.5-7B-Instruus
92.0
83.0
86.0
83.2
43.5
60.4
+ RLCF
95.6
81.0
86.9
82.2
42.0
59.0
DPO (Skywork)
90.4
89.0
88.6
85.6
45.2
63.1
Table 9: In our experiments, RLCF uses prompts that focus primarily on daily assistance and writing
Zhaoo et al. 2024]] This has small effect on "non-target" tasks; refusal to answer unsafe prompts (measured on XSTest), basic mathematical reasoning (measured on GSM8K), and hallucination prevention (measured on TruthfulAA), suggesting that RLCF incurs modest trade-offs on underrepre- sented domains, which could be mitigated by improving domain coverage of training prompts.
As described in Section 3 RLCF grades responses on each criterion using 25 samples from judge model.
90
IFEval (avg)
InFoBench
FollowBench (HSR)
This creates the computational bottleneck of our scor- ing procedure. In Figure 4, we evaluated models trained using RLCF modified to use fewer samples from the judge. Response grading WildChat with 5, 10, or 25 samples took 32, 40, 72, and 92 hours, respectively,
80
70
Qwen2.5-78 Instruct
Avg of Aug
Avg Avg 25
8xH100 node. We observe sta- ble efficacy on IFEval" and InFoBench across sample sizes. For FollowBench, using fewer samples hurts the "content" and "situation" categories, suggesting that cheap, high-variance score may suffice for simpler criteria but not for difficult, ambiguous instructions.
Figure 4: RLCF samples 25 scores when grading each requirement. This is expensive. Fortunately, much of the efficacy is retained using just 5 samples (55% less clock time).
Related Works
Our method is a new means of generating synthetic AI feedback. This follows prior work that use "AI feedback" to guide reinforcement learning algorithms, either via single prompt/rubrie [Tunstall
cet al. 2023] or a collection of rubrics [Cui et al., [023). In our paper, we show that checklist feedback significantly more effective than UlraFeedback [Cui et al. 2023). which evaluates responses four global principles. Our work is also related to prior works that use reward models as synthetic
preference annotators for RL [Sun al.) 2023). In Table Table 3 Table 4 demonstrates the
risks of using reward models to supervise RL [Liu et al. 2024b Wang et al. 2024c).
The models trained all showed moderate variance on IFEval, slight differences likely due to noise.

---

Ivvll (prompt)
IFEval (inst.)
InFoBench Overall
FollowBench
Loose
Striet
Loose
Strict
Avg
SSR
HSR
Llama 3.1 8B Instruct
79.1
71.3
83.5
76.2
77.5
83.1
77.6
68.0
+ RLCF poff-y
80.2
71.9
83.9
76.5
78.1
84.2
81.8
72.3
OLMo 7B Instruct
78.1
69.3
79.5
71.3
74.6
80.1
72.3
59.4
+ RLCF poff-y
79.6
70.8
82.0
73.9
76.6
80.5
71.9
60.4
Table 10: RLCF works off-policy for OLMo 2 7B Instruct and Llama 3. 1 8B Instruct. For Ll ama, see neutral results with IFEval and strong improvements InFoBench and FollowBench. For OLMo, we see strong improvements with IFEval and neutral results on InFoBench and Follow- Bench.
We focus on complex instruction following. One line of work synthesizes instructions with explicit, pathological constraints to train models to generalize other instructions [Xu al. 2022) He et al., 2024. Sun et al. [024 Dong et al. (0024). These papers use DPO on controlled candidate responses, while we contribute drop-in evaluator which allows scoring responses sampled directly from
student model (opening the door to other RL algorithms like GRPO (Shao et al. 2024
Our work is related to nascent line of work that explores using rubrics for language model alignment and evaluation. Cook et al.] [2024) demonstrate that using model -enerreeeee checklists can be useful at inference-ting time for proprictary LLMs. Similarly, Saha et al. [2023) use generated checklists at inference time to improve constrained reasoning tasks. [Saad- et al. 2024) use checklists to evaluate language models and match our finding that checklists can outperform reward models at
response evaluation. Wang et al. [2025bb. contemporaneously to us, introduce a rubric generation criteria" which also uses candidate responses sampled from different language models; they show that these criteria improve the agreement of automatic evaluations of model-generated responses with human judgments. Our work is the first apply similar approach to RL, at the same time as some contemporaneous works. Dineen et al. [2025] use detailed rubrics that greatly improve model safety. Their work differs from ours by using RL with explicit rewards (GRPO) and using checklists that were defined at a global level (the same large set of criteria applied to all tasks) rather than the instriction-speeffie checklists we espouse in this work.
method called "pre-compariion--errie
Limitations
We highlight three limitations with our work. First, our implementation of RLCF uses "strong- to-weak generalization" larger model (Quen2. tuning a smaller model, though RLCF beats other methods also useing a 72B teacher. Second, we only explored preference-bart based RL in our work. We believe that using checklist feedback to train policy gradient-based algorithms is an exciting future research direction. Lastly, our scoring method is expensive - grading response pairs on each requirement for 130k instructions with Qwen2.5-72B. Instructt takes roughly 4 days on eight H100 GPUs with 80GB GPU memory, which is computationally infeasible for many practitioners. In [SSectiion 5.7 we show that this cost can be reduced by 50% at some slight cost to accuracy, but further optimization is warranted.
provides Al judgments for
8 Conclusion
We provide a detailed study of reinforcement learning from checklist feedback (RLCF). We propose novel algorithm for automatically extracting rubrics from instructions, and this algorithm to construct a dataset of instructions and rubrics, WildChecklists. We demonstrate that RLCF is uniformly effective at improving strong instruction following models all benchmarks we consider.
Our study follows an active line of work that highlights the limitations of reward models in supervising reinforcement learning. One exciting future direction to emerge from this work is: how can we combine checklist-style feedback with trainable judges? Our current approach relies on carefully- designed, prompt-based components for rubric generation and response grading under rubric. Why this more effective than methods that naturally learn to grade responses from human preference data? We believe that analysis of RLCF can motivate better reward models in the future.
10

---

Acknowledgements
We thank Saumya Gandhi, Xiang Yue, Gokul Swamy, Apurva Gandhi, Lintang Sutawika, Jessie Mindel, Qianou Ma, Chenyang Yang, and Xinran Zhao for helpful discussions and Akhila Yerukola for invaluable writing assistance and technical advice. Vijay Viswanathan, Graham Neubig, and Tongshuang Wu were supported in part by grant from Apple. Any views, opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and should not be interpreted as reflecting the views, policies, or position, either expressed or implied, of Apple Inc.
References
Michael Xieyang Liu, Frederick Liu, Alexander J. Fiannaca, Terry Koo, Lucas Dixon, Michael Teny, and Carrie J. Cai. "we need structured output": Towards user-centered constraints on large language model output. Extended Abstracts of the CHI Conference Human Factors in Computing Systems, 2024a. httt/t//s://apppp seappesismmtiicchhlaarr. orr.oopuus :29002993
Wenting Zhao, Xiang Ren, John Frederick Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs the wild. ArXiv, abs/2405.014700 2024. URL https
warg/CorpusID... 269390491
Lianmin Zheng, Wei Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuoban Li, Zi Lin, Eric Xing, et al. Lmsys-chat-lls Im: A large-scale real-world Ilm conversation dataset. In The Twelfth International Conference on Learning Representations.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 11:14011-140:66 2019. URL htt.htttts//rrxii
org/abs/1911 10683
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generate instructions. In Annual Meeting of the Association for Computational Linguistics, 2022. URL httph: //api semanticscholar org/ ::t:/CorpuIID::55500 254877310
Hyung Won Chung, Le Hou. Longpre. Barret Zoph, Yi Tay, William Fedus, Eric Li. Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery. Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang. Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling fisteteetoonffietuueed language models. ArXiv, abs/2210.114166 2022. URL httpp::/api. www.ntiischooleeeee.eeee
:rgg/oorpssI::55300 253018554
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing. ArXiv, abs/2406.08s 044444 2024. URL
org/
CorpusID:2203914422
Nathan Lambert, Jacob Daniel Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hanna Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training, ArXiv, abs/2411.111225 2024a. www.stieaacncihoooeeeeeeeeoeeete
URL https://apt:
Mrgg/CorpuIID:220e 274192505
Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, abs/1909.08593, 2019. Ut/// www.cccchhooaoorroo..e
www.aarrrr/Cooooor//a ID: 2026660943
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Chris Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, E Perez, Jamie

---

KeTr, Jared Mueller, Jeff Ladish, J Landau, Kamal Ndousse, Kamilé Lukosuite, Liane Lovitt, Michael Sellitto., Nelson Elhage, Nicholas Schiefer, Noem'i Mercado, Nova Dassarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly. Tom Henighan, Tristan Hume, Sam Bowman, Zac Hatfield-Dodds, Benjamin Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom B. Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. ArXiv, abs/2212.08077,, 2022. URL httpp:///pp...smaaiicccoooo..... http:aatiiiicccoaaaaooooo/rrorop http://ooopusSD:2423333339 254823489.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang. Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bing-Li Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dong-Li Li Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Jiong Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, M. Tang. Meng Li. Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang. Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang. J. Chen, Ruiqi Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Kang Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng. Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wen-Xia Yu, Wentao Zhang. Wangding Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xi aokang Chen, Xiaotao Nie. Xin Cheng, Xin Liu, Xin Xie, Xingehao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q Li, Xiangyu Jin, Xi-Cheng Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yi Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong. Yu-Jing Zou, Yujia He, Yunfan Xiong, Yu-Wei Luo, Yu mei You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yao Li, Yi Zheng, Yuchen Zhu, Yunxiang Ma Ying Tang, Yukun Zha, Yuting Yan, Zehui Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhen guo Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu WU, Zihui Gu, Zijia Zhu, Zijun Liu, Zi-An Li, Ziwei Xie, Ziyang Song, Zizheng Pan. Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseck-rl: Incentivizing reasoning capability in lims via reinforcement learning. ArXiv, abs/2501. 2025. URL https://api. www.eteeaaacanaeeeeeeeee
Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following, 2025.
URL httts:://arxiii org/ httgg//bb/20077 028333
Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving fnslloition-oolloiing capabilities of large language models. ArXiv, abs/2406.13542, 2024. URL httpss/t::///pp semanticscholar org/
CorpusID:270620157
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer--preffeence: Complementing ratings with preferences. ArXiv, abs/2410.01257,, 2024a. /ttpstttss///pp...o http:pppp///ppppeetttttttcoooooo org/ wwt./or///oi.oooooooooo htsp/CrrpussDD 273022955
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, Dj Dvijotham, Adam Fisch, Katherine Heller, Stephen R. Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hack- ing. ArXiv, abs/2312.09244,, 2023. URL https://appl www.sticsmoooiiiiom.. Sawaaaror///ooo org/Corpus
266210056
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im align- ment. ArXiv, 0ts//331..16444 2023. URI. //api. semanticscholar. wtwpatttssssoooaroo/oooorrrraaee org/CorpusID; 264490502
12

---

Gokul Swamy, Sanjiban Choudhury. Wen Sun, Zhiwei Steven Wu, and J. Andrew Bagnell, All roads lead likelihood: The value of reinforcement learning fine-tuning. ArXiv, abs/2503.01067, 2025. URL htt.lhttss://eae http:tpppp/pppp.s.sma....cceeeba
wew..eiacanccccoo..a... hwwpchhhhlarrrrrooeoooooo org/CorpusID:2/050 276742134
Amelia Glaese, Nat McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy
Jonathan Uesato, Po-Sen Huang. Ramona Comanescu, Fan Yang. A. See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Sovna Mokr'a a. Nicholas Fernando, Boxi Wu, Rachel Foley. Susannah Young. Iason Gabriel, William S. Isaac, John F. J. Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. Improving alignment dialogue agents via targeted human judgements. ArXiv, 0los/2209.143755 2022. URL https://api. semanticscholes org/ CorpusID:
Yuxin Jiang, Yufei Wang, Xingshan Zeng. Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level tne-ggrined constraints following benchmark for large language models. In Annual Meeting of the Association for Computational Linguistics, 2023. URL
http:///.c.ssmmticccccccl.aa.. org orggCorrpsID:eeee0 264802222.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang. Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. 2024.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph Gonzalez, and lon Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and bench- builder pipeline. ArXiv, abs/2406. 1939, 2024. URL httph:tt// httppp///ap...mmmmiiicccooooooo/ semanticscholar .rg//
Corpus :7705628888
Sukai Huang, Shu- Liu, Nir Lipovetzky, and Trevor Cohn. The dark side rich rewards: Un- derstanding and mitigating noise in vim rewards. 2024. httptttsp///p..... http////tt////pp.....aaaaaacooo. wwwantcschoonrrlrr
org/CorpusID: 20:::22224
Jonathan Cook, Tim Rocktäschel, Jakob N. Foerster, Dennis Aumiller, and Alex Wang. Ticking all the boxes: Generated checklists improve 1lm evaluation and generation. ArXiv, abs/2410.03608, 2024. /ttpp////tt/////g..eeaaaatttoorr http://ccccccccooooooooooaaaaaaa
Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David D. Cox, Yiming Yang. and Chuang Gan. Salmon: Self-alignment with instructable reward models. In International Conference on Learning Representations, 2023. URL https: www.seeemtatntoo.eeeeee org/
Corpus.DD:263331633
Qwen An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao. Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi- Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report. ArXiv, abs/2412.15115,
2024. URL https
wrwg/Crrpsslaaaooooooooo : 274859421
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ArXiv, abs/1904.09751, 2019. URL htthh:tt///att
httt::::/app.eemmmtiicchhooooo www.eeteeacnnttooeeeeeeeeeeetttt org/
CorpusID::27996680
Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained iostruction-folloiing ability of large language models. ArXiv. abs/2404.02823, 2024. URL htt.pppp//pppp.ssm...cccooooll
org/ logg/CorpusID :
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Haotong Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. Proceedings of the 29th Symposium on Operating Systems Principles, 2023. UtLhttps: //a/ //api. semanticscholar org/ PerpusID:26169777
13

---

Victor Wang, Michael J.Q. Zhang, and Eunsol Choi. Improving llmdae-a-judge inference with the judgment distribution. ArXiv, 00ss/503300304,, 2025a. |ttpps/:///ppp http://ppp..emaniccsccooooo...
Org/CorpusID:2760
Tairan Fu, Raquel Ferrando, Javier Conde, Carlos Arriaga, and Pedro Reviriego. Why do large
language models (Ilms) struggle to count letters? ArXiv, abs/2412.116266 2024. URL https
//api. seapssssmanticccoo....
27751199
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-folloviin evaluation for large language models. ArXiv, abs/2311.07911
2023. httpp:::/api..semaaiicccc
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. ArXiv, 0bss2305.182990 2023. URL https: wwnppes.maannicchol.... /tt.:tttpe//pppp...aaaccccooo... D 2589599211
Jian Hu, Xibin Wu, Weixun Wang, Dehao Zhang, Yu Cao, OpenLLMAI Team, Netease Fuxi, AI Lab, and Alibaba Group. Openrlhf: An easy-to-use, scalable and high-performance rihf frame- work. ArXiv, abs/2405. 11143, 2024. htL/hhtps://apt semanticscholoe www. per//orpusID" 269921667
Yann Dubois, Bal'azs azs Galambosi, Percy Liang, and Tatsunori Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. ArXiv, abs/2404.04475. 2024. URL
https://apt www.nticscholes.ee.... www.ccholaa..ro/ooouus.........e wwwp/ooppssI............. :22900446555
269004605
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge a neural net-
work. ArXiv, abs/1503.02531, 2015. URL https::t/pp...: semanticscholar. www.oorrrrrrrroooooooooo wwt./oorpussD........
7200347
Yaowei Zheng. Richong Zhang. Junhao Zhang. Yanhan Ye, Zheyan Luo, and Yongqiang Ma. Lla- mafactory: Unified efficient fine-tuning of 100+ language models. ArXiv, abs/2403.13372, 2024.
URL https:://ppp..... www.ee/icmammmccccoio.o.e
org/Corpus 268536974
Chris Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojic Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. ArXiv, Abs/2410.18451. 2024b. URL httpss://pppp..... semanti cscholar. org/ wwrg/CorpusID-... 2 273549327
Haoxiang Wang. Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Conference on Empirical Methods in Natural Language Processing, 2024b. URL http::://ppp... semanticscholar /arrg//rpussiii ID
270562658
Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. Rewardbench: Evaluating reward models for language mod- eling. ArXiv, 137.77777 2024b. URL htt.htttt///pp.. semanticscholee. org/ Corpus 2685334409]
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni. Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. ArXiv, abs/2310.01377, 2023. URL hatp:::/aapl.senntt..cccoolaal org/ wwwgCCorpsIID...... http:opppI::22335522222 263605623,
Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Daniel Morrison, Noah A. Smith, Hanna Hajishirzi, and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation. 2025. URL
https://api. semanticscholest htt.:/o......ggg/ooouuusD////11.
Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, and Sanjeev Arora. What makes reward model good teacher? an optimization perspective. ArXiv, abs/2503.11447, 2025.
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: test suite for identifying exaggerated safety behaviours large language models. ArXiv, abs/2308.01263, 2023, URL https: / //ppi/pppi/p/ htt.tppp/p/pp...eeiiicccco..a wttp::/scccccaaoooorrrrrrreoeeee prg/CorpusID:: 260378842
14

---

Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/21 1021110..1666 14168, 2021. URL https
//api..seanticsccoola. www.cccchhhhaarrrrrrrrrioo 239998651399111
Stephanie C. Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falschoods. In Annual Meeting of the Association for Computational Linguistics, 2021. URL
https://api. semanticscholar
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur' elien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cris tian Cantón Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriele Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guanglong Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar. Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov. Jade Copet, Jacwon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jemny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Ju-Qing Jia, Kalyan Vasuden Alwala, K. Upasani, Kate Plawiak, Keqian Li, Kenneth Heafield, Kevin R. Stone, Khalid EI-Arini, Krithika Iyer, Kshitiz Malik, Kuen ley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins. Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Ma hesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kambadur, Mike Lewis. Min Si. Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Niko lay Bashlykov. Nikolay Bogoychev, Niladri S. Chatterji. Olivier Duchenne, Onur cCelebi, Patrick Alrassy, Pengchuan Zhang. Pengwei Li, Petar Vasić, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ron nie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sa hana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Chandra Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Vir ginie Do Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whit ney Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yiqian Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zhengxu Yan, Zhengxing Chen, Zoe Papakipos, Aaditya K. Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adi Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annic Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Po-Yao (Bernie) Huang, Beth Loyd, Beto de Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer. Damon Civin, Dana Beaty, Daniel Kreymer, Shang- Wen Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Erik Brinkman,
Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu,
15

---

Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzm' an, Frank Kanayet, Frank Seide. Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang. Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Han Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang. Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kaixing(Kai) Wu, U KamHou, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, A Lavender, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthias Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Mun ish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy ata Bawa, Nayan Singhal. Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev. Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pe dro Rittner, Philip Bontrager. Pierre Roux, Piotr Dollár, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rehekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe. Steve Satterfield, Sudarshan Govindaprasad, Sumit Kumar Gupta, Sung- Bac Cho, Sunny Virk, Suraj Subramanian. Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xia Tang. Xiaofang Wang, Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang. Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Detioo, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024. URL https::::: //appi//pppi.p http:///apppp.eeatieieeeeeeee www.eetiiaccooo.ooooo..
wtrg/Corpusllii ID: 271571434
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk. Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James Validad Miranda, Jacob Daniel Morrison, Tyler C. Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg. David Wadden, Christopher Wilhelm, Michael Wilson, Luke S. Zeutemmyer, Ali Farhadi, Noah A. Smith, and Hanna Hajishirzi. olmo 2 furious. ArXiv, abs/2501.0066 000566, 2024. URL (ttthttp::// //api. sww.ntieaniiime i cscholar .org/// CorpusID:27521300
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP, 2024c.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. ArXiv, abs/2304. 12244, 2023. URL https: //api. www.aiiisommnntccoooooos.o. wwwooolrrrrrrrroooooooooeo httg/CooppsIID222281155 258298159
Qi He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. In Conference on Empirical Methods in Natural Language Processing, 2024. URL httpss
//api.ssmmntiiscchol.. htte/CorpuSDD:263362222
16

---

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, abs/2402.03300, 2024. URL Btl.https:///pp. www.:ssssaanniccc.o.o.mm.. org/
CorpusID:2674112077
Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch- solve-merge improves large language model evaluation and generation. ArXiv, abs/2310.1512., 2023. UKLAacpps//ppp.sseattiscooooorr org/ Berg/CorpusIDD.
:645991422
Jon Saad-Falcon, Rajan Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vid- gen, Amanpreet Singh, Douwe Kiela, and Shikib Mehri. Lmunit: Fine-grained evaluation with nat- ural language unit tests. ArXiv, abs/2412.1300 13091, 2024. URL //api. http:///t/..ssssssssecseoeeeee www.nticschol...........
org/CorpusID: 274788535
Zongqi Wang, Tianle Gu, Chen Gong, Xin Tian, Siqi Bao, and Yujiu Yang. From rankings to insights: Evaluation should shift focus from leaderboard feedback. ArXiv. abs/2505. 06698, 2025b. URL
https::/app. semanti cscho har. org/ htt///rrpss::::: 2785011:777894
Jacob Dineen, Aswin RRV, Qin Liu, Zhikun Xu, Xiao Ye, Ming Shen, Zhaonan Li, Shijie Lu, Chitta Baral, Muhao Chen, al. Qa-lign: Aligning Ilms through constitutionally decomposed qa. arXiv preprint trXiv:2506608122 2025.
A The role of response pair mining
FollowBench (HSR)
InFoBench (Overall)
85
Overall Score
Avg HSR
80
25
80
70
20
30
40
70
80
90
100
10
40
50
60
70
100
of Response Pairs Filtered
of Response Filtered
Filtering by Constraint
Filtering by Total Reward
GPT-4
Qwen-78 Swen-7B-Instruch
Figure Impact of different filtering strategies on model performance on FollowBench and In- FoBench. We compare filtering pairs based on overall checklist score differences versus filtering based on single-aspect score differences, varying dataset sizes. There are only slight differences between these two filtering methods, until we start filtering out the vast majority of the data. This suggests that the reward signal, rather than the specific filtering algorithm, is likely responsible for this method's effectiveness.
In algorithm for learning from checklist feedback, we only train the 40% of response pairs that differ the most on at least one criterion. This approach differs from thresholding on the reward difference with single scalar reward, which may represent the aggregation of multiple small differences across all requirements. How much is the filtering component responsible for the success of RLCF?
To investigate this, we compared two approaches: selecting pairs with the largest differences in overall weighted checklist scores versus selecting pairs with the largest differences any single aspect's 's score. As shown in Figure performance shows that, when discarding just 20% or 40% of response pairs, the method of filtering makes almost difference. On the other hand, when discarding 90% of response pairs (with least difference in reward), performance plummets on both benchmarks, suggesting that, regardless of the filtering strategy, keeping some "harder" response pairs is beneficial. Rather than aspect-based filtering being the primary driver of improvement, the results suggest that checklist-basee based rewards inherently capture more instruction-relevan dimensions of quality, leading to more effective preference tuning even with moderate filtering.
B Prompt for Generating Verification Programs
We describe the prompt used for generating programs selectively verify responses in Figure 6
17

---

C Prompt for Scoring Semantic Criteria
We describe the prompt used for requirement checking in Figure 7
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main defeasible claim made in the abstract and introduction are that the proposed method is effective on "every" benchmark we evaluate on, and that no other alignment methods share this property. This is supported by the data (see) Table 2 Table 3 and Table 4). On one of the benchmarks we try (IFEval), the proposed method has no real effect on 2 out of 4 metrics, but it does help on the other 2 metrics (see) Table 2). Otherwise, the positive results are consistent.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, we list a few limitations of the proposed work in Section 7
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is no significant theoretical work contained in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper aims to give full details about the information needed reproduce the paper. This includes data sources (Section 4.1). training hyperparameterss (Section 4.1),
Justification:
exact models used (Section 4.1). data generation parameters (Section 3), and exact prompts used (listed in Appendix B and Appendix C).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
20

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code with careful documentation is posted at www..githen com/
viswavi (along with links to models), and our generated dataset posted
www. www.hhuggggcce..c wwt.//ttastttt/iiaaaa/tttttaaaae
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/ sublic/guidess ottp://oooeoeiooiinooooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. http/uullcggguucee Ctleee//ooSSubbisse asiionooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper discloses training and test details, such as training data sources (Section 4.1), training hyperparameters (Section 4.1), data generation parameters (Section 3).
prompts (listed in Appendix B and Apppedix C. and specific benchmark dataset settings
Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not report statistical significance of our experiments, due to compute overhead and clear trends we see across extensive comparison experiments. This aligns with common practice in the instruction following literature, where statistical testing is often
21

---

omitted. We do consider adding more such analysis in future iterations though.and this is something we will consider adding in future drafts.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: For our key experiments, we reported the exact computer resources used (Section for training and 7 for data generation). We did not report all computer resources for reproducing all baselines or ablation experiments, because we used varying compute for these experiments, and this did not seem absolutely necessary to describe in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: To the best of our knowledge, completely conform to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
22

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper does not discuss societal impacts at length, beyond a brief disclaimed in [Secciion that our method not substitute for safety alignment. We justify this because paper, which focuses instruction following, considers the same socictal impacts as any other work on instruction following there is nothing significantly different about the harms posed by our proposed method than those posed by existing, widespread methods (e.g. reward models or Al judges).
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We have instituted one very limited safeguard, which was to curate our training data only with filtered, non-toxic version of WildChat. However, with any work that releases general purpose language models and datasets to support these models, there is a very real potential for misuse. We feel that the risk for misuse is comparable to the many other papers that study language modeling, and we are not targeting sensitive domains specifically in our work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
23

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We cited and credited our of WildChat, in compliance of its ODC-BY license.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: As mentioned in the paper, will release the dataset constructed Wild- Checklists. We will also document and opensource the code for RLCE
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. The only annotation work done for this paper was done by the authors of the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We have described using LLMs as teacher and student models, for improving the student model's instruction following capability. We have also detailed the checklist synthesis approach (based on criticizing LLM L-M-eenerated outputs), as well as using LLM as judges for specific checklist items.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy for what should or should not be described.
25

---

You responsible for helping verify whether responses satisfy various requirements. Given a natural language requirement, you will have classify whether this can be converted to Python program to automatically check it or whether should be given to human collaborator. Your human collaborator is reliable and cheap expert, and you should trust them. Accordingly, only write code for verifying constraint if never make ANY approximations when verifying constraint. you feel that you the constraint in order verify whether response follows that constraint, let your human collaborator take care of it. You should ONLY generate code for requirements that are explicidy about syntax or format (e.g. punctuation, unicode characters used, number of paragraphs, shallow grammar, presence of mandatory keyword specified by the prompt, etc). If there many different ways write answer, you most likely should not generate code for If you are sure, you should not generate code, You should only generate code if you are 100% sure that the constraint can verified perfectly with a simple Python function.
very confident that this will exactly check the constraint. You should
approximate
When constraint be verified EXACTLY with a program, then return Python function that verifies the constraint. This code should contained within two sets of triple backquotes. "*. Python function must return boolean, and it should only use builtins/standard libraries in Python. If the constraint cannot be verified with a simple Python function (which means your human collaborator will handle the verification of this constraint), please return "NONE" and nothing clse. The safest thing do is to return "defer to human expert ####" 95% of time. Now, let's go through couple examples:
Input:
Outline curriculum development process for a 16-week high school history course, including setting week-by-weed objectives and designing assignments. Include mid-term exams and final exam. Provide detailed grading criteria based on the assignments and exams you have designed.
Requirement:
Does the response specify the inclusion of two mid-term exams and final exam
Verification Function:
defer to human expert #### (there multiple valid ways to describe this, and it is not simple boolean check)
Input:
Welcome to ISLAM STORE's Brand Story
Our Journey: A Vision Brought to Life ISLAM STORE was founded with the vision create inclusive, informative, and accessible platform for Muslims and non-Muslims alike. Our goal promote awareness understanding Islam while offering high-quality Islamic products.
Requirement:
Does the generated text contain any Arabic?
Verification Function:
"python
def renurrrruuiremme((ex)):
# Arabic Unicode block range (0600-06FF)
Plus Extended Arabic (0750-077F)
# Plus Arabic Presentation Forms (FB50-FDFE, FE70-FEFF)
return any(("06000 <= char <= "(OFFF) or ("lu0750' <= char <= "w077F") or ("uuFB00' <= char <= "wFDDFF) ("uFF770' <= char <= 'WFFFFF for char in text)
Input: [imput)
Requirement: {requirement]
Verification Function:
Figure 6: Prompt for generating verification code
26

---

Based the provided input instruction and response from worker, assess the response based the following criteria:
1. Does it satisfy the specific requests of the instruction?
2. Docs the response directly address the request without excessive or off-topic information not necessary for addressing the user's 's instruction?
Does the response match the context and the instruction, whether requires professionalism, friendliness, formality, or neutrality?
Accordingly, score the response with rating (a number between O and 100) assessing how well the response addresses the instruction. For example, the input instruction might be "What is good vegan substitute meat for someone allergic soy and gluten? Provide single-sentence response consisting of an answer followed by a factually detailed and humorous one-sentence explanation", Your selection should be based on the response and the instruction, using the following rating scale:
100: Select 100 if the generated text represents an optimal solution that expertly balances all relevant aspects of instruction. For example above (about the vegan substitute), and the criterion above (about factual detail), an example 100-point response is "Mushrooms, because they can be easily caramelized and browned, they are rich in the glutamates which Icad to incredible umami flavors, they naturally completely free of soy and gluten, and they don't look cute as babies". This response is richly detailed and factual, and though fails to be humorous, it still a (00-point response on the factual detail criterion.
75: Return 75 if the generated text very effectively addresses the main requirements but has room for minor improvements. The response should he unconditionally acceptable (at professional level) but may not be absolutely perfect. There example 75- 75-point response the example question above is "Mushrooms they are rich the glutamates that lead to incredible umami flavors and they don't look cute in the slightest while alive.".
mistakes that critically undermine the question. An
This response has onc interesting fact but could be more detailed.
50: Opt for 50 if the generated text adequately fulfills the basic requirements but contains notable flaws missed opportunities for improvement. The response should still be functionally acceptable. The response contains at most one minor inadequacy or inaccuracy related to the question but there are no mistakes that critically undermine the question. An example 50-point response to the example question above is "Mushrooms, because they can be easily caramelized and browned, they're 're universally beloved by sophisticated palates, and they don't look cute in the slightest while alive." The statement that they's 'e universally beloved by people with sophisticated palates, while potentially true, vague and not objective.
25: Return 25 if the generated text fulfills the key condition specified by the question and demonstrates awareness of the key requirements but fails to execute them effectively. The text may contain non-critical inaccuracies or irrelevant information. However, if there is even one element that critically undermines the core purpose specified in the question (even that element seems minor isolation), the score should be 0 (not 25). An example 25- 25-point response to the example question above is "Mushrooms, because they can be easily caramelized and browned. they are absolutely brimming with protein, and they don't 't look cute in the slightest while alive." Thh statement that most kids love mushrooms not objective and potentially false).
O: Opt for if the generated text fails to meet the question's 's requirements or provides no information that could be utilized to answer the question. If the response contains a critical error relevant to the question, return a O. For the question about the vegan substitute, an example O-point response is "Mushrooms, because they make you question why you ever thought dead animal could compare to this vegan delight." While funny and engaging, this response contains zero factual detail about mushrooms, critically violating the question.
Your score he any number between O and 100 (not just the ones listed above). If you are totally confused, return -I as default. You should use your judgment to determine the most appropriate score. Focus on the posed question and ignore other aspects of response quality not implicd by the question. Return only a number - do not include any other text in your response.
Input:
{instruction]
Generated Text:
{response}
Question:
(requirement)
Score:
Figure 7: Prompt for checklist scoring
27


Reviews:
[{"review_id": "7RUYV2iqG0", "reviewer": "Reviewer_vQmq", "summary": "This paper introduces Reinforcement Learning from Checklist Feedback (RLCF), an alignment method to improve how language models follow instructions. RLCF first involves generating a unique checklist and for each user prompt. The checklist consists of yes/no questions that detail the criteria an ideal response should satisfy and an importance weight for each question. \n\nThen, responses from the policy scored against this checklist using a combination of an AI judge and code-based verifiers to generate training pairs for DPO. \n\nThe trained policy shows strong performance across 5 instruction following and general quality benchmarks. The authors also release WildChecklists, a new dataset of 130,000 instruction-checklist pairs to aid in future research in this area.", "questions": "1. Did you try using fewer than 25 samples for the judgement score? \n2. Qualitatively, did you notice other reward hacking patterns like length increase, emojis, verbosity etc.?\n3. In table 6, how are the numbers for RLCF w/o prompt-based scoring obtained? Assuming code-based verifiers can be used for far fewer checklist items, it is not clear how a strong reward signal can be obtained. \n4. What is the proportion of checklist items having code-based verifiers? \n5. Do you have an agreement analysis with the importance weigthing? Or maybe rank correlation numbers?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n - This paper proposes a novel alignment method that combines dynamic checklist verification and code based verifiers to produce a strong RL signal. While the ideal of checklists for judging LLM responses is not novel, this paper is the first to demonstrate that the signal is useful for RL. It also incorporates a weight parameter for each item on the checklist to properly prioritize more important items over less important ones. This provides a more nuanced signal to the policy. \n - LLM scoring and judgement is great for open-ended nuanced requirements but is not as good for stricter verifiable requirements. The authors recognize this and incorporate generated code-based verifiers to ensure that judgement on these criteria is accurate.  \n - The generated checklists are evaluated for quality by humans and a GPT-4o based automatic evaluation system. \n - The paper includes a variety of results across different alignment baselines and evaluation sets demonstrating how generalizable RLCF is. \n - RLCF is much more transparent and interpretable than previous alignment methods, making it easier to discover and fix reward hacking patterns. \n\n### Weaknesses \n - The paper exclusively uses the Qwen family of models for generating the checklist, the base policy, and the AI Judge. While these results are extensive, demonstrating the usefulness of RLCF with WildChecklists, it would be good to show results with a policy from a different family of models like the Llama, Mistral or Gemma models. Since we know from [Panickssery et. al.](https://arxiv.org/abs/2404.13076) that LLMs prefer their own responses, it would be good to verify that the AI judge is not overly strict on responses from other models. \n - Additionally, since the checklist generation is conditioned on responses from a variety of Qwen models, showing alignment results on other models would also help establish that the checklist question capture universal issues correctly.  \n - Table 5 shows that the checklist based reward is not as good at providing safety alignment or reasoning alignment signals. This might indicate that models trained using WildChecklists and RLCF also show safety and reasoning losses. Including results on benchmarks like HarmBench and GSM8K would help understand if that is the case.  \n - Relatedly, it is also possible that RLCF-trained models hallucinate more or are less truthful. Since the checklists don't emphasize factual accuracy, it is possible that the AI judge misses any such issues. Running evaluations on TruthfulQA could help understand if this is the case.", "was_revised": true, "final_justification": "The additional experiments provided by the authors help address my generalizability concerns and also confirm my hunch that RLCF could lead to minor regressions in areas that it doesn't specifically target. However, this is a smaller issue in my opinions because the tradeoff could be worthwhile compared to training reward model on a specialized task. Through RLCF we can perform more targeted changes to model performance.", "rebuttal": "Thank you for the detail-oriented and extremely constructive review, and thank you for appreciating the novelty, generality, and interpretability of our method. We will address your primary concerns point-by-point:\n\n```\n“The paper exclusively uses the Qwen family of models for generating the checklist, the base policy, and the AI Judge. While these results are extensive, demonstrating the usefulness of RLCF with WildChecklists, it would be good to show results with a policy from a different family of models [...]”\n```\nWe appreciate this perspective that is grounded in recent literature. The “egocentric bias” of LM judges’ towards their own outputs may impede models’ ability to judge samples from another model family (e.g. Llama) [1]. Due to compute constraints, we could not re-generate preference data during the rebuttal period. However, since the judge is responsible for comparing *samples from the same policy* rather than *across different policies*, we believe this potential confounder is not a significant concern.\n\n\n```\n“Additionally, since the checklist generation is conditioned on responses from a variety of Qwen models, showing alignment results on other models would also help establish that the checklist question capture universal issues correctly.”\n```\n*(the following response is a copy of the response to the first concern of Reviewer #1)*\n\n\nWe believe that the objective phrasing of our checklist items offers a general signal for tuning other models. To test this, during this rebuttal period, we used the preference dataset generated for Qwen-2.5-7B-Instruct to finetune meta-llama/Llama-3.1-8B-Instruct and allenai/OLMo-2-1124-7B-Instruct via off-policy DPO.\nWe show results below:\n\n**InFoBench/IFEval**:\n|  Model | InfoBench (Overall)  | IFEval (prompt-level strict)  |  IFEval (prompt-level loose) | IFEval (instr-level strict)  |  IFEval (instr-level loose) |\n|---|---|---|---|---|---|\n| Llama-3.1-8B-Instruct   |  83.07 |  71.3  | 79.1  |  76.2 | 83.5 |\n|  + RLCF  |  **84.18** | **71.9** | **80.2**  | 76.5   |83.9|\n| OLMo-2-1124-7B-Instruct  |  80.09 | 69.3  | 78.1 | 71.3  |79.5|\n|  + RLCF |  80.53 | **70.8**  | **79.6**  | **73.9**  |**82.0**|\n\n**FollowBench**:\n|  Model | *Soft* | L1 | L2 | L3 | L4 | L5 |  Avg | *Hard*|  L1 | L2 | L3 | L4 | L5 |  Avg |  CSL |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| Llama-3.1-8B-Instruct ||84.0|81.7|78.0|73.9|70.2|77.6||84.0|77.1|68.2|61.3|49.4|68.0|2.95|\n|  + RLCF ||**85.1**|**86.7**|**80.5**|**78.2**|**78.4**|**81.8**||**85.0**|**83.0**|**72.0**|61.1|**60.2**|**72.3**|**3.08**|\n| OLMo-2-1124-7B-Instruct ||75.9|**75.8**|72.2|**70.7**|67.0|72.3||75.9|69.1|59.2|**49.9**|**43.1**|59.4|2.58|\n|  + RLCF||76.1|74.6|**75.4**|66.5|66.7|71.9||76.1|**70.0**|**66.9**|48.9|40.1|**60.4**|**2.75**|\n\nWith Llama, we see positive results with IFEval and strong improvements with InFoBench and FollowBench. With OLMo, we see strong improvements with IFEval and neutral results on InFoBench and FollowBench. Considering both are *off-policy*, we expect that results could be stronger if trained on samples from these models. *This suggests these checklists do capture universal issues correctly.*\n\n\n ```\n“Table 5 shows that the checklist based reward is not as good at providing safety alignment or reasoning alignment signals. This might indicate that models trained using WildChecklists and RLCF also show safety and reasoning losses. Including results on [..] HarmBench and GSM8K would help”\n```\nWe thank the reviewer for raising this issue, which we did not originally test. To estimate of RLCF’s effect on safety, we instead ran on XSTest, which directly tests models’ proclivity to refuse to answer if and only if appropriate, in lieu of the more-complicated HarmBench. Below, we show results using GPT-4-as-a-judge to determine whether the model is performing refusal or not. \n\n|  Model | Response (when safe) | Refusal (when safe) | Response (when unsafe) | Refusal (when unsafe) | **Overall F1** |\n|---|---|---|---|---|---|\n| Qwen-2.5-7B-Instruct    |  92.0\\% | **8%** | 17\\%  | 83\\%   |86.0|\n|  \"\" + RLCF  |  **95.6\\%** | 4.4% | 19\\%  | 81\\%   |86.9|\n|  \"\" + DPO (Skywork)  |  90.4\\% | 9.6\\% | **11\\%**  | **89\\%**   |**88.6**|\n\nThese results give some insight into the domain-specific effects of RLCF. RLCF demonstrates a slightly positive change in unweighted F1 on XSNet, with an improvement in the proclivity of a model to answer when safe coupled with a slight increase in unsafe responses. The contrast with Skywork is illustrative – Skywork was explicitly trained on preference data from safety datasets and it naturally excels at safety evaluation but shows relatively poor performance on instruction following tasks. \n\nWe can see a similar pattern expressed with mathematical reasoning with GSM8K (8-shot CoT):\n\n|  Model | Exact-Match Accuracy | \n|---|---|\n| Qwen-2.5-7B-Instruct    |  83.2\\% | \n|  \"\" + RLCF  |  82.18\\% | \n|  \"\" + DPO (Skywork)  | 85.6\\% | \n\nHere, we see RLCF slightly reduces accuracy on GSM8K, while Skywork (which was also trained in part on math data) improves considerably on this task. This difference actually showcases a key feature of our method: unlike traditional reward models that must be trained on a fixed set of prompts that ideally overlap with the RLHF distribution [2], RLCF allows practitioners to flexibly target specific capability areas by selecting appropriate data sources. WildChat's focus on daily assistance naturally leads to strong improvements in non-verifiable instruction following but limited mathematical reasoning gains. Practitioners interested in other domains can generate checklists for new examples with less effort than required to tune and train a reward model.\n\n \n```\n“Relatedly, it is also possible that RLCF-trained models hallucinate more or are less truthful. Since the checklists don't emphasize factual accuracy, it is possible that the AI judge misses any such issues. Running evaluations on TruthfulQA could help understand if this is the case.”\n```\n\nFollowing the reviewer’s suggestions, we evaluated RLCF on TruthfulQA to understand if RLCF may lead to greater hallucination. We used the multiple-choice evaluation of TruthfulQA, where the model is tasked with determining if an incorrect answer (that is designed to elicit response biases of models) is chosen over the factual answer. “MC1” measures whether the model chooses the true answer among all presented candidates (true or false). “MC2” measures the probability of the true answer, normalized by the total probability of all answers. \n\n|  Model | MC1 | MC2 | \n|---|---|---|\n| Qwen-2.5-7B-Instruct   |  43.5 | 60.4 | \n|  \"\" + RLCF  |  42.0 | 59.0 | \n|  \"\" + DPO (Skywork)  | 45.2 | 63.1 | \n\n\nThis suggests that RLCF induces a modest decrease in factual accuracy (MC1: -3.4% relative change, MC1: -2.3%). As with math reasoning and safety, this reflects WildChat’s composition: WildChat focuses on daily assistance, advice, and analysis (75.5%), with only  6.1% on factual information and 6.1% on math [3]. This targeted scope explains why RLCF excels at instruction following while offering neutral or slight decreases in other domains. Given the zero-shot nature of Checklist Feedback, incorporating more factuality-focused prompts during RL training would likely address this limitation, and this is something that the flexibility of checklist feedback allows.\n\nNote: For the above three experiments, we evaluate using OpenInstruct [4].\n\n\n[1] Koo et al. 2024. Benchmarking Cognitive Biases in Large Language Models as Evaluators. In Findings of ACL 2024.\n[2] Malik et al 2025. RewardBench 2: Advancing Reward Model Evaluation. arXiv.\n[3 Zhao et al 2024. WildChat: 1M ChatGPT Interaction Logs in the Wild. In ICLR 2024.\n[4] Wang et al. 2023. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. In NeurIPS 2023 Datasets and Benchmarks.\n\n————\n\n**Questions**\n\n```\nFewer than 25 samples for the judgement score?\n```\nYes, and we will include these results in our final draft. \n\n*(the following section is duplicated in our responses to reviewers Cq9t and hvGc)*\n|  # of judges | IFEval (Overall Avg) | InFoBench (Overall) | FollowBench (HSR Avg) |\n|---|---|---|---|\n| Qwen-2.5-7B-Instruct   |  77.3 | 78.1 | 71.4 |\n| 3 samples   |  78.3 | **84.8** | 71.3 |\n| 5 samples   |  76.8 | 84.6 | 73.7 |\n| 10 samples  |  77.6 | 83.6 | 73.3 |\n| 25 samples (RLCF)  |  **78.6** | 84.1 | **75.3** |\n\nMuch of the efficacy is retained using just 5 samples (taking 55% less data generation time). Given the reusability of this data for other model families shown above, we think this justifies this one-time computational investment.\n\n\n```\nQualitatively, did you notice other reward hacking patterns like length increase, emojis, verbosity etc.?\n```\nWe observe a 7% increase in word count for responses to IFEval (from 195 to 210). We occasionally see unsolicited rationales (around 5% of time -- this suggests reward hacking, but this is also seen after RL with other reward models. Importantly, RLCF improves on both length-controlled and non-length-controlled metrics on AlpacaEval – length increases, but so does estimated utility.\n\n```\nIn table 6, how are the numbers for RLCF w/o prompt-based scoring obtained? Assuming code-based verifiers can be used for far fewer checklist items, it is not clear how a strong reward signal can be obtained.\n```\nWe only retain prompts with >=1 associated verification program, and rewards are using the verifiable subset. While imperfect, this mirrors the standard setup for RL from verifiable rewards.\n\n```\nWhat is the proportion of checklist items having code-based verifiers?\n```\n166182 out of 637770 total checklist items (26.1\\%) have code-based verifiers.\n\n```\nDo you have an agreement analysis with the importance weighting? Or maybe rank correlation numbers?\n```\nUnfortunately, we currently lack ground truth checklist weights to measure this. This is an interesting step for future work."}, {"review_id": "q2ujmqYYi8", "reviewer": "Reviewer_Cq9t", "summary": "This paper proposes RLCF (Reinforcement Learning from Checklist Feedback), a method that aligns language models using instruction-specific checklists instead of fixed criteria like helpfulness. By evaluating how well responses meet checklist items, the method provides tailored rewards for training. Applied to Qwen2.5-7B-Instruct, RLCF improves performance across several benchmarks, showing its effectiveness in handling diverse user instructions.", "questions": "1. How can we evaluate whether the scores provided by the LLM are well-calibrated?\n\n2. In cases where the instruction is either too difficult or too easy—leading to low discriminability among responses—would RLCF with checklists perform better? The rationale is that providing a checklist allows for more fine-grained evaluation of responses. Exploring this direction could be an interesting avenue for future research.\n\n3. To reduce the variance of the scores from the judge model, the authors mention sampling 25 responses. Could you provide more details on the actual variance observed in practice?\n\n4. Could you elaborate on how you determine whether a verifier program is applicable for a given instruction or checklist item?\n\n5. How critical is the process of generating the weights?", "limitations": "See weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n1. RLCF is a flexible and intuitive framework that can be applied to any instruction-response pair.\n2. The method proposes a novel framework beyond traditional RLHF approaches, such as those relying on trained reward models.\n\n### **Weaknesses**\n\n1. To prevent reward hacking, the method relies heavily on prompt-based techniques, which may be vulnerable to manipulation.\n2. The approach depends on the performance of the judge model, which may not be fully accurate and could result in incorrect preference pairs.\n3. When using LLM-based judges, score calibration is crucial. However, the paper does not appear to include a calibration procedure for the response scores, nor does it discuss potential issues or mitigation strategies related to this.", "was_revised": true, "final_justification": "I have carefully read the authors’ rebuttal and appreciate their thoughtful and detailed responses. Several of my concerns were addressed convincingly. However, some key aspects, such as score calibration and the potential variance of judge responses, remain underexplored empirically. Also, the importance of weights is not validated as well. Therefore, I maintain my original rating.", "rebuttal": "Thank you for the thought-provoking review, and thank you for appreciating the novelty and flexibility of our proposed approach. We will discuss each of your concerns:\n```\nTo prevent reward hacking, the method relies heavily on prompt-based techniques, which may be vulnerable to manipulation.\n```\nWe would like to reconsider the premise that our method relies heavily on prompt-based techniques. We do use prompts to generate the checklists, grade checklists, and generate code. However, we believe that using multiple prompt-based components in a multi-step process (involving both LM grading and code) should actually reduce vulnerability to manipulation, while also enabling the grading of non-verifiable instructions where programs cannot verify correctness.\n\n```\nThe approach depends on the performance of the judge model, which may not be fully accurate and could result in incorrect preference pairs.\n```\nWe believe that RLCF is fundamentally less dependent on the judge model’s ability than prior methods. By breaking down the judge’s annotation task into measurably objective and atomic yes-or-no questions, we believe the judge’s annotation task here is easier than in traditional AI judges or reward models, which can also be inaccurate. We also add code-based rewards to accommodate situations where the judge is not reliable. Therefore, we argue that while any reward model or judge is imperfect, our proposed architecture has the potential to be significantly more robust to errors by a judge.\n\n```\nWhen using LLM-based judges, score calibration is crucial. However, the paper does not appear to include a calibration procedure for the response scores, nor does it discuss potential issues or mitigation strategies related to this.\n```\nWe appreciate this concern about calibration, and this is interesting from a statistical perspective. Score calibration in RL is already taken care of at the level of the learning algorithm, e.g. \"variance reduction\" techniques (such as using an \"advantage\" or \"value function\"). This is already baked into the DPO algorithm, so the key concern for us is the accuracy and distribution of the reward rather than calibration or normalization of the reward.\n\n————\n\n**Questions**\n\n```\nHow can we evaluate whether the scores provided by the LLM are well-calibrated?\n```\nAs noted above, DPO’s built-in variance reduction mitigates calibration concerns, leaving us to focus on relative reward quality rather than absolute values.\n\n```\nIn cases where the instruction is either too difficult or too easy—leading to low discriminability among responses—would RLCF with checklists perform better? The rationale is that providing a checklist allows for more fine-grained evaluation of responses. Exploring this direction could be an interesting avenue for future research.\n```\nThis is an interesting avenue for future research! Our initial hypothesis was indeed that RLCF is effective in the setting where responses may differ along a specific aspect, rather than across all or multiple possible aspects of response quality. To test this, we made Figure 4 in Appendix A in the paper. This suggests that, instead, RLCF remains effective even when per-criterion differences are small but aggregate quality differences are large, suggesting our method’s robustness may extend further than our original hypothesis. We speculate that better consistency or coverage of the response may be at play.\n\n```\nTo reduce the variance of the scores from the judge model, the authors mention sampling 25 responses. Could you provide more details on the actual variance observed in practice?\n```\nDue to time restrictions during this rebuttal period, we were not able to rigorously measure the variance of the scores from the judge model – our statement about the variance was based on theoretical notions of ensembling’s effect on variance and bias. For a potential intuition about the impact of our sampling procedure on RLCF’s effectiveness, we duplicate our response to reviewer vQmq above, where we show the impact of the number of judge samples on trained policies:\n\n|  # of judges | IFEval (Overall Avg) | InFoBench (Overall) | FollowBench (HSR Avg) |\n|---|---|---|---|\n| Qwen-2.5-7B-Instruct   |  77.3 | 78.1 | 71.4 |\n| 3 samples   |  78.3 | **84.8** | 71.3 |\n| 5 samples   |  76.8 | 84.6 | 73.7 |\n| 10 samples  |  77.6 | 83.6 | 73.3 |\n| 25 samples (RLCF)  |  **78.6** | 84.1 | **75.3** |\n\nIn summary, any number of samples leads to improvements on at least 2 out of 3 instruction following benchmarks, but increasing the number of samples leads to more consistent gains on all of them.\n\n\n```\nCould you elaborate on how you determine whether a verifier program is applicable for a given instruction or checklist item?\n```\n\nWe make this determination via a prompt, which is described in Figure 5 of the paper. In particular, we include the following phrasing in the prompt: *“You should never make ANY approximations when verifying a constraint [...] You should ONLY generate code for requirements that are explicitly about syntax or format [...] If there are many different ways to write an answer, you most likely should not generate code for it. If you are not sure, you should not generate code.”*. This phrasing leads the code generator module to reduce the frequency that verification programs are produced, minimizing false positives. We chose this conservative approach over more complex filtering methods (e.g., unit tests) to maintain the method's practical simplicity while ensuring high precision. [1]\n\n```\nHow critical is the process of generating the weights?\n```\nDue to limitations of scope and time, we have not been able to answer this empirically. However, we implemented the weighting of checklist items in one shot, without significant effort spent on tuning this component. The majority of item weights are between 75 and 100, suggesting that most checklist items are being considered. This suggests weighting is a simple and robust component requiring minimal tuning.\n\n[1] Dong et al. 2024. Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models. Arxiv."}, {"review_id": "6wmfI8wqqo", "reviewer": "Reviewer_oyZA", "summary": "The paper introduces a method to generate rewards during RLHF (which they call \"checklists\"). Checklists promise to capture a more complete picture of a model response by generating requirements that a good response should fulfil and assigning rewards according to these requirements. A checklist has to satisfy 3 properties (yes, no-question; each requirement must be answered; all requirements have to be satisfied to count as acceptable). It can then be used in a prompt for an LLM-judge to select chosen and rejected datapoints for preference optimisation.\n\nIn their empirical investigation, the authors generate checklists using models from the Qwen-family based on a filtered version of WildChat and fine-tune Qwen2.5-7b on their dataset, comparing their checklist-based approach with reasonable baselines, such as reward models, AI judges without checklists and standard SFT. These comparisons demonstrate the advantage of datapoint-specific checklists over other methods.\nAdditionally, they provide insightful analysis of the results, clearly showing the advantages and fail conditions of their approach as well as of other methods using instructive example cases.", "questions": "1. Did you try using smaller teacher models? How detrimental is the downscaling for performance?\n2. Instead of sampling 25 times and averaging, you could also use greedy decoding and achieve the same (with less noise), if I am not mistaken? I am curious whether I am missing something here.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\nThe paper has multiple strengths. A high-level summary:\n1. _Format_: Besides minor typos, the paper is well-structured and well-written. Graphics, plots and tables sufficiently support the content of the paper.\n2. _Topic & solution_: Obtaining precise quality estimations through automatic means is a very relevant topic for practitioners working on optimizing LLM performance. The ability to bootstrap this information from LLMs in a reliable and straightforward way is a exciting prospect. The paper under review makes a promising contribution in that direction.\n3. _Insightful empirical results_: The empirical results are insightful, especially the analysis of breaking conditions for different scoring techniques is informative. \n4. _Rigour_: The experiments and outcomes are sound, and the work seems to have been conducting with the necessary rigour. The chosen baselines and benchmarks are reasonable\n\n### Weaknesses\nThere are some minor weakness to the paper under review:\n1. Fine-tuned models and models used for data generation are both from the same family. A model from a different model family during fine-tuning would insure generality.\n2. Results on safety benchmarks is low. This is unsurprising, as the authors filter out toxic data points. Those datapoints could have, with slight adaptation in the prompt, be useful for improving safety. It is a missed opportunity to see whether the approach is viable beyond instruction following. \n3. Online preference optimization (such as PPO and GRPO) are becoming again more prominent in the literature. Besides DPO, the approach seems very promising for online preference optimization as well. An experiment working with e.g. GRPO would have been interesting to see.\n\n### Typos, formatting and so on:\n1. Table 2 has a + in front of the Qwen-instruct model.\n2. l194: 5.1 heading", "was_revised": true, "final_justification": "I sustain my high evaluation, considering this paper a valuable contribution to the field", "rebuttal": "We thank the reviewer for their positive and careful review. We thank them for appreciating the applicability of our method, and we especially appreciate that they recognized our paper’s rigor and empirical insights, which we are proud of. We hope that we can address any remaining misunderstandings about our method.\n\n```\n“Fine-tuned models and models used for data generation are both from the same family. A model from a different model family during fine-tuning would insure generality.”\n```\n*(the following response is a copy of the response to the first concern of Reviewer #1)*\n\nWe believe that, due to the measurably objective phrasing of the checklist items we generate, our grading procedure is indeed general and unlikely to depend on the nuances of the student model. To ensure this, during this rebuttal period, we used the preference dataset that we generated for samples from Qwen-2.5-7B-Instruct, and we used this to finetune meta-llama/Llama-3.1-8B-Instruct and allenai/OLMo-2-1124-7B-Instruct via off-policy DPO.\nWe show these results for InFoBench, IFEval, and FollowBench below:\n\n**InFoBench/IFEval**:\n|  Model | InfoBench (Overall)  | IFEval (prompt-level strict)  |  IFEval (prompt-level loose) | IFEval (instr-level strict)  |  IFEval (instr-level loose) |\n|---|---|---|---|---|---|\n| Llama-3.1-8B-Instruct   |  83.07 |  71.3  | 79.1  |  76.2 | 83.5 |\n|  + RLCF  |  **84.18** | **71.9** | **80.2**  | 76.5   |83.9|\n| OLMo-2-1124-7B-Instruct  |  80.09 | 69.3  | 78.1 | 71.3  |79.5|\n|  + RLCF |  80.53 | **70.8**  | **79.6**  | **73.9**  |**82.0**|\n\n**FollowBench**:\n|  Model | *Soft* | L1 | L2 | L3 | L4 | L5 |  Avg | *Hard*|  L1 | L2 | L3 | L4 | L5 |  Avg |  CSL |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| Llama-3.1-8B-Instruct ||84.0|81.7|78.0|73.9|70.2|77.6||84.0|77.1|68.2|61.3|49.4|68.0|2.95|\n|  + RLCF ||**85.1**|**86.7**|**80.5**|**78.2**|**78.4**|**81.8**||**85.0**|**83.0**|**72.0**|61.1|**60.2**|**72.3**|**3.08**|\n| OLMo-2-1124-7B-Instruct ||75.9|**75.8**|72.2|**70.7**|67.0|72.3||75.9|69.1|59.2|**49.9**|**43.1**|59.4|2.58|\n|  + RLCF||76.1|74.6|**75.4**|66.5|66.7|71.9||76.1|**70.0**|**66.9**|48.9|40.1|**60.4**|**2.75**|\n\nWith Llama, we see moderately positive results with IFEval and very strong improvements with InFoBench and FollowBench. With OLMo, we see very strong improvements with IFEval and neutral results on InFoBench and FollowBench. Considering that both of these are *off-policy*, we expect that the results could be even stronger if trained on samples from these models. *This suggests these checklists do capture universal issues correctly.*\n\n“Results on safety benchmarks is low. This is unsurprising, as the authors filter out toxic data points. Those datapoints could have, with slight adaptation in the prompt, be useful for improving safety. It is a missed opportunity to see whether the approach is viable beyond instruction following.”\n\nAgreement of checklist-based rewards with the safety subset of RewardBench is indeed significantly lower than state-of-the-art, although we should point out that it’s still competitive with many widely used reward models. We observe that, despite our data mixture not including safety-critical prompts, we still maintain minimal degradation in the safety behavior of the trained policy. To this point, we duplicate our response to reviewer vQmq above, where we measure the Qwen model trained with RLCF on XSTest:\n\n\n|  Model | Response (when safe) | Refusal (when safe) | Response (when unsafe) | Refusal (when unsafe) | **Overall F1** |\n|---|---|---|---|---|---|\n| Qwen-2.5-7B-Instruct    |  92.0\\% | 8% | 17\\%  | 83\\%   |86.0|\n|  \"\" + RLCF  |  **95.6\\%** | **4.4%** | 19\\%  | 81\\%   |86.9|\n|  \"\" + DPO (Skywork)  |  90.4\\% | 9.6\\% | **11\\%**  | **89\\%**   |**88.6**|\n\n\nThe reviewer’s point is valid – checklists could very well be useful for evaluating response safety. Our intuition is that response safety (which is often prompt-agnostic – e.g. do not use profane language or demean protected groups) might be best handled as a distinct step or auxiliary reward for promoting the model’s helpfulness. However, we believe the current RLCF method provides space for this as a complementary consideration.\n\n\n“Online preference optimization (such as PPO and GRPO) are becoming again more prominent in the literature. Besides DPO, the approach seems very promising for online preference optimization as well. An experiment working with e.g. GRPO would have been interesting to see.”\n\nWe agree that an experiment working with online RL, particularly using policy gradient algorithms, is an exciting opportunity for future work. We intentionally omitted this direction from our paper, as we’ve found that it’s difficult to compare different reward models on their ability to supervise PPO. The appropriate hyperparameters for optimizing a policy using one reward model can vary dramatically from another reward model, so a broad hyperparameter sweep on a validation set would be necessary for such a comparison. Accordingly, following prior work (e.g. https://arxiv.org/abs/2409.13156), we used DPO as a more dependable signal of usefulness for RL, but we absolutely agree that working with GRPO is an exciting continuation to see how far RL from checklist feedback can take us.\n\n \n———\n\n**Questions**\n\n\n```\nDid you try using smaller teacher models? How detrimental is the downscaling for performance?\n```\n\nWe did not try using smaller teacher models, as our goal was to achieve peak performance against strong baselines. We believe that this is an interesting avenue to consider for cost and environmental reasons.\n\n```\nInstead of sampling 25 times and averaging, you could also use greedy decoding and achieve the same (with less noise), if I am not mistaken? I am curious whether I am missing something here.\n```\n\nEarly on in our experiments, we tried this. At that point, we found that while greedy decoding of the judge’s scores was effective when reranking samples from a policy (“best-of-K alignment”), but significantly less effective at supervising DPO. Our theoretical intuition is that sampling and averaging scores enables better consistency of scores, which mitigates reward hacking. Nonetheless, this is an interesting suggestion which we were unable to run in an end-to-end test during the limited rebuttal period."}, {"review_id": "euho5xXMO7", "reviewer": "Reviewer_hvGc", "summary": "This paper introduces \"Reinforcement Learning from Checklist Feedback\" (RLCF), a novel approach to RL-based alignment by using a checklist as rewards. The method works by automatically generating checklists from instructions that break down complex requirements into a set of smaller yes/no questions, then using both LLM-as-judges and rule-based verification programs to score responses on each checklist item. These scores are combined into weighted rewards for reinforcement learning via Direct Preference Optimization (DPO). The authors create WildChecklists, a dataset of 130,000 instructions with corresponding checklists, and demonstrate that RLCF consistently improves performance across multiple benchmarks.", "questions": "- typo in line 194 Feedbac => Feedback\n- Is there a compute-efficient way to determine which items in the checklist are more important than the others?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper proposes a novel framework, RLCF, which organizes rewards via a checklist. \n- The proposed method shows competitive results across five benchmarks: IFEval, InFoBench, FollowBench, AlpacaEval, and Arena-Hears. \n- The method shows consistent improvements where other approaches fail, with particularly strong gains on complex constraint satisfaction tasks (5.4% relative improvement on FollowBench). \n- The approach is practically valuable as it requires only a teacher model without additional human annotation. \n- The authors provide thorough ablation studies and qualitative analysis. \n- The new dataset WildChecklists seems valuable.\n\nWeaknesses:\n- In some sense, prior works exist that have already been doing RLCF implicitly (those that check rule-based format checks as well as LLM-as-judges), those no prior work created such a nice name as RLCF. In this regard, the novelty of this paper is more on the conceptualization and more focused empirical analysis than a ground-breaking algorithm development.\n- The proposed approach requires repeated calling of a strong reward model (> 70B), thus it can be quite expensive.\n- In some sense, it's not surprising that the proposed method would work better than the baselines that were compared, given the denser and higher quality rewards the method provides during RL.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review and careful reading of our paper. We will address their comments step by step:\n\n```\nIn some sense, prior works exist that have already been doing RLCF implicitly (those that check rule-based format checks as well as LLM-as-judges), those no prior work created such a nice name as RLCF. In this regard, the novelty of this paper is more on the conceptualization and more focused empirical analysis than a ground-breaking algorithm development.\n```\nWe thank the reviewer for appreciating our conceptualization, and we agree that prior works exist that apply methods similar to RLCF. However, we would argue that our approach is indeed methodologically novel. Criteria-based feedback has either been used with synthetic instructions (e.g. [UltraIF -- An et al 2025](https://arxiv.org/abs/2502.04153) or with purely code-based evaluation (e.g.  [AutoIF -- Dong et al 2025](https://arxiv.org/abs/2406.13542 or [RLVR – Lambert et al 2024](https://arxiv.org/abs/2411.15124)). The first (criteria with synthetic instructions) does not allow flexible application to custom sets of prompts or arbitrary RL algorithms, as our method does. The latter (verifiable rewards/code-based evaluation) restricts the space of constraints that can be checked. We argue that our proposed method represents a meaningful step forward along this general direction.\n\n\n```\nThe proposed approach requires repeated calling of a strong reward model (> 70B), thus it can be quite expensive.\n```\nWe agree that this can be quite expensive. However, we also compare with repeated calling of the same judge model in the naive “AI judge” baseline, and this significantly underperforms – we therefore claim that our approach’s inference cost is unlikely to be the sole reason for its effectiveness. We also show that this cost can be reduced by a significant factor at a moderate cost to effectiveness. Duplicating our response to reviewer vQmq (originally posted above), we show the impact of the number of samples on policy effectiveness: \n\n|  # of judges | IFEval (Overall Avg) | InFoBench (Overall) | FollowBench (HSR Avg) |\n|---|---|---|---|\n| Qwen-2.5-7B-Instruct   |  77.3 | 78.1 | 71.4 |\n| 3 samples   |  78.3 | **84.8** | 71.3 |\n| 5 samples   |  76.8 | 84.6 | 73.7 |\n| 10 samples  |  77.6 | 83.6 | 73.3 |\n| 25 samples (RLCF)  |  **78.6** | 84.1 | **75.3** |\n\nWe can improve on at least 2 out of 3 instruction following benchmarks with as few as 3 calls to the reward model per criterion. We hope that future researchers will build on this approach, in maintaining the benefits of checklist feedback at a lower cost.\n\n\n```\nIn some sense, it's not surprising that the proposed method would work better than the baselines that were compared, given the denser and higher quality rewards the method provides during RL.\n```\nGiven that checklist-based rewards are based on more factors than a sparse binary preference, we agree that this dense reward is likely to help, and this is definitely a key motivation of our approach. However, we can contrast our setting (which has roughly 5 criteria per instruction) with UltraFeedback, which has 4 universal criteria per instruction. Our method's significantly stronger performance demonstrates that criteria with adaptive specificity are more effective than universal criteria – even with comparable density. We also want to discuss the notion of “higher quality” rewards, as this is a rather deep concept – Skywork and ArmoRM exhibit greater agreement with human preferences on RewardBench than Checklist Feedback, but they are worse at supervising RLHF. This suggests that reward quality ought to be measured via both extrinsic metrics (e.g. RLHF) and intrinsic metrics (e.g. RewardBench), suggesting that the quality of checklist feedback is not self-evident. We think this reflects an ongoing evolution in how we, as a community, evaluate reward quality.\n \n———\n\n**Questions**\n\n```\nIs there a compute-efficient way to determine which items in the checklist are more important than the others?\n```\n\nOur current approach, using few-shot prompting to decide the weight, is already relatively compute efficient and seems robust. However, this is an interesting question that we didn’t focus on extensively. A more compute-efficient way to make this determination could include a custom-trained lightweight checklist item classifier, and to use this classifier’s probabilities as weights, but we felt the current approach was sufficiently efficient and accurate to avoid warranting additional complexity."}]

Author rebuttal:

Meta-review: The paper presents a new way to obtain reward for RL finetuning, by creating a checklist of properties that a response should satisfy. 

Strengths: an innovative idea that can work in situations where a verifiable reward is hard to obtain, flexible method that can be easily plugged for training, strong empirical results
Weaknesses: Only qwen family of models used as both model and judge

The authors provided results on training a llama model during rebuttal, and also some experiments on safety dataset, which have alleviated the main weakness of the paper. I agree with the reviewers that this is a useful and timely contribution to the RL finetuning literature, and suggest the authors to include the rebuttal experiments in the main paper.
Decision: Accept (spotlight)