Title: Checklists Are Better Than Reward Models For Aligning Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: alignment

rubrics

instruction following

RLHF
Abstract: Language models must be adapted to understand and follow user instructions. Reinforcement learning is widely used to facilitate this —typically using fixed criteria such as "helpfulness" and "harmfulness". In our work, we instead propose using flexible, instruction-specific criteria as a means of broadening the impact that reinforcement learning can have in eliciting instruction following. We propose "Reinforcement Learning from Checklist Feedback" (RLCF). From instructions, we extract checklists and evaluate how well responses satisfy each item—using both AI judges and specialized verifier programs—then combine these scores to compute rewards for RL. We compare RLCF with other alignment methods on top of a strong instruction following model (Qwen2.5-7B-Instruct) on five widely-studied benchmarks — RLCF is the only method to help on every benchmark, including a 4-point boost in hard satisfaction rate on FollowBench, a 6-point increase on InFoBench, and a 3-point rise in win rate on Arena-Hard. We show that RLCF can also be used off-policy to improve Llama 3.1 8B Instruct and OLMo 2 7B Instruct. These results establish rubrics as a key tool for improving language models' support of queries that express a multitude of needs. We release our our dataset of rubrics (WildChecklists), models, and code to the public.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces \"Reinforcement Learning from Checklist Feedback\" (RLCF), a novel approach to RL-based alignment by using a checklist as rewards. The method works by automatically generating checklists from instructions that break down complex requirements into a set of smaller yes/no questions, then using both LLM-as-judges and rule-based verification programs to score responses on each checklist item. These scores are combined into weighted rewards for reinforcement learning via Direct Preference Optimization (DPO). The authors create WildChecklists, a dataset of 130,000 instructions with corresponding checklists, and demonstrate that RLCF consistently improves performance across multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper proposes a novel framework, RLCF, which organizes rewards via a checklist. \n- The proposed method shows competitive results across five benchmarks: IFEval, InFoBench, FollowBench, AlpacaEval, and Arena-Hears. \n- The method shows consistent improvements where other approaches fail, with particularly strong gains on complex constraint satisfaction tasks (5.4% relative improvement on FollowBench). \n- The approach is practically valuable as it requires only a teacher model without additional human annotation. \n- The authors provide thorough ablation studies and qualitative analysis. \n- The new dataset WildChecklists seems valuable.\n\nWeaknesses:\n- In some sense, prior works exist that have already been doing RLCF implicitly (those that check rule-based format checks as well as LLM-as-judges), those no prior work created such a nice name as RLCF. In this regard, the novelty of this paper is more on the conceptualization and more focused empirical analysis than a ground-breaking algorithm development.\n- The proposed approach requires repeated calling of a strong reward model (> 70B), thus it can be quite expensive.\n- In some sense, it's not surprising that the proposed method would work better than the baselines that were compared, given the denser and higher quality rewards the method provides during RL.\n\nQUESTIONS:\n- typo in line 194 Feedbac => Feedback\n- Is there a compute-efficient way to determine which items in the checklist are more important than the others?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method to generate rewards during RLHF (which they call \"checklists\"). Checklists promise to capture a more complete picture of a model response by generating requirements that a good response should fulfil and assigning rewards according to these requirements. A checklist has to satisfy 3 properties (yes, no-question; each requirement must be answered; all requirements have to be satisfied to count as acceptable). It can then be used in a prompt for an LLM-judge to select chosen and rejected datapoints for preference optimisation.\n\nIn their empirical investigation, the authors generate checklists using models from the Qwen-family based on a filtered version of WildChat and fine-tune Qwen2.5-7b on their dataset, comparing their checklist-based approach with reasonable baselines, such as reward models, AI judges without checklists and standard SFT. These comparisons demonstrate the advantage of datapoint-specific checklists over other methods.\nAdditionally, they provide insightful analysis of the results, clearly showing the advantages and fail conditions of their approach as well as of other methods using instructive example cases.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nThe paper has multiple strengths. A high-level summary:\n1. _Format_: Besides minor typos, the paper is well-structured and well-written. Graphics, plots and tables sufficiently support the content of the paper.\n2. _Topic & solution_: Obtaining precise quality estimations through automatic means is a very relevant topic for practitioners working on optimizing LLM performance. The ability to bootstrap this information from LLMs in a reliable and straightforward way is a exciting prospect. The paper under review makes a promising contribution in that direction.\n3. _Insightful empirical results_: The empirical results are insightful, especially the analysis of breaking conditions for different scoring techniques is informative. \n4. _Rigour_: The experiments and outcomes are sound, and the work seems to have been conducting with the necessary rigour. The chosen baselines and benchmarks are reasonable\n\n### Weaknesses\nThere are some minor weakness to the paper under review:\n1. Fine-tuned models and models used for data generation are both from the same family. A model from a different model family during fine-tuning would insure generality.\n2. Results on safety benchmarks is low. This is unsurprising, as the authors filter out toxic data points. Those datapoints could have, with slight adaptation in the prompt, be useful for improving safety. It is a missed opportunity to see whether the approach is viable beyond instruction following. \n3. Online preference optimization (such as PPO and GRPO) are becoming again more prominent in the literature. Besides DPO, the approach seems very promising for online preference optimization as well. An experiment working with e.g. GRPO would have been interesting to see.\n\n### Typos, formatting and so on:\n1. Table 2 has a + in front of the Qwen-instruct model.\n2. l194: 5.1 heading\n\nQUESTIONS:\n1. Did you try using smaller teacher models? How detrimental is the downscaling for performance?\n2. Instead of sampling 25 times and averaging, you could also use greedy decoding and achieve the same (with less noise), if I am not mistaken? I am curious whether I am missing something here.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RLCF (Reinforcement Learning from Checklist Feedback), a method that aligns language models using instruction-specific checklists instead of fixed criteria like helpfulness. By evaluating how well responses meet checklist items, the method provides tailored rewards for training. Applied to Qwen2.5-7B-Instruct, RLCF improves performance across several benchmarks, showing its effectiveness in handling diverse user instructions.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n1. RLCF is a flexible and intuitive framework that can be applied to any instruction-response pair.\n2. The method proposes a novel framework beyond traditional RLHF approaches, such as those relying on trained reward models.\n\n### **Weaknesses**\n\n1. To prevent reward hacking, the method relies heavily on prompt-based techniques, which may be vulnerable to manipulation.\n2. The approach depends on the performance of the judge model, which may not be fully accurate and could result in incorrect preference pairs.\n3. When using LLM-based judges, score calibration is crucial. However, the paper does not appear to include a calibration procedure for the response scores, nor does it discuss potential issues or mitigation strategies related to this.\n\nQUESTIONS:\n1. How can we evaluate whether the scores provided by the LLM are well-calibrated?\n\n2. In cases where the instruction is either too difficult or too easy—leading to low discriminability among responses—would RLCF with checklists perform better? The rationale is that providing a checklist allows for more fine-grained evaluation of responses. Exploring this direction could be an interesting avenue for future research.\n\n3. To reduce the variance of the scores from the judge model, the authors mention sampling 25 responses. Could you provide more details on the actual variance observed in practice?\n\n4. Could you elaborate on how you determine whether a verifier program is applicable for a given instruction or checklist item?\n\n5. How critical is the process of generating the weights?\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Reinforcement Learning from Checklist Feedback (RLCF), an alignment method to improve how language models follow instructions. RLCF first involves generating a unique checklist and for each user prompt. The checklist consists of yes/no questions that detail the criteria an ideal response should satisfy and an importance weight for each question. \n\nThen, responses from the policy scored against this checklist using a combination of an AI judge and code-based verifiers to generate training pairs for DPO. \n\nThe trained policy shows strong performance across 5 instruction following and general quality benchmarks. The authors also release WildChecklists, a new dataset of 130,000 instruction-checklist pairs to aid in future research in this area.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n - This paper proposes a novel alignment method that combines dynamic checklist verification and code based verifiers to produce a strong RL signal. While the ideal of checklists for judging LLM responses is not novel, this paper is the first to demonstrate that the signal is useful for RL. It also incorporates a weight parameter for each item on the checklist to properly prioritize more important items over less important ones. This provides a more nuanced signal to the policy. \n - LLM scoring and judgement is great for open-ended nuanced requirements but is not as good for stricter verifiable requirements. The authors recognize this and incorporate generated code-based verifiers to ensure that judgement on these criteria is accurate.  \n - The generated checklists are evaluated for quality by humans and a GPT-4o based automatic evaluation system. \n - The paper includes a variety of results across different alignment baselines and evaluation sets demonstrating how generalizable RLCF is. \n - RLCF is much more transparent and interpretable than previous alignment methods, making it easier to discover and fix reward hacking patterns. \n\n### Weaknesses \n - The paper exclusively uses the Qwen family of models for generating the checklist, the base policy, and the AI Judge. While these results are extensive, demonstrating the usefulness of RLCF with WildChecklists, it would be good to show results with a policy from a different family of models like the Llama, Mistral or Gemma models. Since we know from [Panickssery et. al.](https://arxiv.org/abs/2404.13076) that LLMs prefer their own responses, it would be good to verify that the AI judge is not overly strict on responses from other models. \n - Additionally, since the checklist generation is conditioned on responses from a variety of Qwen models, showing alignment results on other models would also help establish that the checklist question capture universal issues correctly.  \n - Table 5 shows that the checklist based reward is not as good at providing safety alignment or reasoning alignment signals. This might indicate that models trained using WildChecklists and RLCF also show safety and reasoning losses. Including results on benchmarks like HarmBench and GSM8K would help understand if that is the case.  \n - Relatedly, it is also possible that RLCF-trained models hallucinate more or are less truthful. Since the checklists don't emphasize factual accuracy, it is possible that the AI judge misses any such issues. Running evaluations on TruthfulQA could help understand if this is the case.\n\nQUESTIONS:\n1. Did you try using fewer than 25 samples for the judgement score? \n2. Qualitatively, did you notice other reward hacking patterns like length increase, emojis, verbosity etc.?\n3. In table 6, how are the numbers for RLCF w/o prompt-based scoring obtained? Assuming code-based verifiers can be used for far fewer checklist items, it is not clear how a strong reward signal can be obtained. \n4. What is the proportion of checklist items having code-based verifiers? \n5. Do you have an agreement analysis with the importance weigthing? Or maybe rank correlation numbers?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}