Title: Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: instruction following

large language model

reasoning
Abstract: Existing large language models (LLMs) face challenges of following complex instructions, especially when multiple constraints are present and organized in paralleling, chaining, and branching structures. One intuitive solution, namely chain-of-thought (CoT), is expected to universally improve capabilities of LLMs. However, we find that the vanilla CoT exerts a negative impact on performance due to its superficial reasoning pattern of simply paraphrasing the instructions. It fails to peel back the compositions of constraints for identifying their relationship across hierarchies of types and dimensions. To this end, we propose RAIF, a systematic method to boost LLMs in dealing with complex instructions via incentivizing reasoning for test-time compute scaling. First, we stem from the decomposition of complex instructions under existing taxonomies and propose a reproducible data acquisition method. Second, we exploit reinforcement learning (RL) with verifiable rule-centric reward signals to cultivate reasoning specifically for instruction following. We address the shallow, non-essential nature of reasoning under complex instructions via sample-wise contrast for superior CoT enforcement. We also exploit behavior cloning of experts to facilitate steady distribution shift from fast-thinking LLMs to skillful reasoners. Extensive evaluations on seven comprehensive benchmarks confirm the validity of the proposed method, where a 1.5B LLM achieves 11.74% gains with performance comparable to a 8B LLM. Evaluation on OOD constraints also confirms the generalizability of our RAIF.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper authors propose a method to boost LLM performance on complex instruction-following tasks by constructing a set of complex instructions that was used for training different language models with Reinforcement Learning. Data is obtained by using self-instruct technique from publicly accessible instruction tuning datasets and RL verification is done with both rules and LLM-as-a-Judge. The method shows promising results for smaller LMs (1.5B) on complex instruction-following benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Comprehensive experiments - multiple model families and model sizes (1.5B, 7B, 8B).\n- Extensive evaluations and good baselines.\n- Smaller models (1.5B parameters) with RL substantially outperform baselines.\n- Open-source models and datasets were used.\n\n### Weaknesses\n- Results are not convincing:\n\t- Table1: all larger models (7B+) show either negligible improvement over I/O or even worse performance (Llama3.1-8B) compared to baselines.\n\t- Table1: for Qwen 2.5-1.5B SFT gives already substantial boost.\n\t- Table 2: considering different compute spent, not clear if there's an advantage for ComplexBench vs baselines.\n- Comparison is better done on the aligned compute since it's not clear what training/test-time compute was spent, we can be sure that the proposed approach has any advantage.\n\nQUESTIONS:\n- Regarding Table 2:\n\t- Can authors please provide more details on oracle decomposition?\n\t- What was the compute spent for each method?\n- What LLM was used for self-evolving instruction generation?\n- Effect of Behavior Cloning. From Table 1, e.g. Qwen2.5-1.5B-Instruct (cold-start) has superior performance after RL over the baselines while DeepSeek-Qwen7B (warm-start) has a marginal performance over I/O. At the same time LLaMA3.1-8B-Instruct (cold-start) performance deteriorates drastically after RL. How this corresponds to \"Without proper guidance, cold-start LLMs can only repeat their inherent shallow thinking and consistently receive negative feedback\"?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWith a focus on improving language models' ability of following constrained instructions, this work presents a dataset construction strategy, and an appropriate task-specific reward function to be used for training models with RL (particularly GRPO). The dataset is an augmentation of existing instruction tuning datasets (e.g., WildChat) with additional constraints and associated verification from either unit tests of LLM judgments. The constraints used in the training dataset overlap with those used in the evaluation datasets, which include IFEval, CELLO, FollowBench, etc. Experiments with various <8B parameter base models show that the training with GRPO on the constructed dataset improves performance on the evaluation sets, more so than CoT prompting and SFT.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n- The proposed approach is straightforward.\n- It is nice to see that it improves performance over expected baselines with a range of base models.\n\n## Weaknesses:\n\n### Main concerns:\n-  The taxonomy of constraints used in the training dataset are the same as those in the evaluation datasets. This leaves open the question of generalizability of the proposed approach to new unseen constraints. It would be helpful to hold out some constraints for an OOD evaluation to measure generalizability.\n- The proposed idea is not new. Using RL with verifiable rewards for the problem of constrained instruction following was done for training Tulu 3 (Lambert et al., 2024; https://arxiv.org/abs/2411.15124; Section 6). Related to the point above, they show that this approach does not generalize to new constraints. It would be interesting to see if the same trend holds in the current work too. Also, it is worth discussing if and how the current work differs from that work.\n\n### Additional concerns:\nThe writing in the paper can generally be improved. Documenting specific issues here:\n- The term \"self-evolving\" occurs throughout the paper, but it is not clear what this means. Section 3.2 says this method is an adaptation of Self-Instruct (SI), but SI uses the same model to generate the instructions and the responses, but this approach augments existing instructions (so it is unclear what the \"self\" refers too), and as far as I can tell, there is no \"evolution\" of instructions in this approach. \n- In the Introduction of the paper, the problem itself can be better defined as instruction following with complex verifiable constraints, along with examples of such instructions. To someone who is not already familiar with the problem, it may not be clear what this paper is about.\n- The second paragraph says prior approaches are all task-specific and are prone to over-fitting. This claim needs a citation.\n\nQUESTIONS:\n- How well does the approach perform on unseen constraints?\n- How is the approach different from RLVR targeting IFEval in Lamber et al., 2024?\n- What do the authors mean by \"self-evolving\"?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article introduces UTU 1.0, a systematic method to enhance large language models' (LLMs') ability to follow complex instructions by incentivizing deep reasoning. Traditional chain-of-thought (CoT) approaches fail due to superficial reasoning that merely paraphrases instructions. UTU 1.0 addresses this via two key components: (1) Self-evolving instruction synthesis: Generates diverse complex instructions with verifiable constraints using atomic rule compositions and validation via code execution/LLM judges. (2) Reinforcement learning (RL) with rule-centric rewards: Uses format and accuracy rewards to enforce structured reasoning, combined with superior CoT enforcement and behavior cloning to prevent policy drift.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \nThe paper directly tackles the shallow reasoning flaw in CoT for complex instructions, a critical gap in existing LLM research.\nThe method is comprehensive, which combines self-evolving data generation with RL, providing a reproducible framework for instruction-following improvement.\nTheir results show that small LLMs can achieve large-model performance via test-time compute scaling, promoting cost-efficient LLM development.\n\nWeakness:\nThe main weakness is the applicability of the proposed method. The results show that the 1.5B model has achieved strong improvement. However, this improvement diminishes when it comes to 7B and larger models. \nIn addition, the training seems hurt the performance in other domains. For example, in Table 13 and 14, I find the math performance clearly drops. Especially with LLama3.1, its math capability after training drops to 0?\n\nQUESTIONS:\nQ: In section 3.3, experience buffer part, you mention using a replay buffer to filter data for RL training. How is this related to instruction following?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}