Title: IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Model, Instruction Following, Reinforcement Learning

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves instruction following capabilities of large language models (LLMs), but suffers from over-optimization where LLMs exploit verification shortcuts without aligning to the actual intent of user instructions. We introduce Instruction Following Decorator (IFDecorator), a framework that wraps RLVR training into a robust and sample-efficient pipeline. It consists of three components: (1) a cooperative-adversarial data flywheel that co-evolves instructions and hybrid verifications, generating progressively more challenging instruction-verification pairs; (2) IntentCheck, a bypass module enforcing intent alignment; and (3) trip wires, a diagnostic mechanism that detects reward hacking by injecting trap instructions to trigger and capture shortcut exploitation behaviors. Our Qwen2.5-32B-Instruct-IFDecorator achieves 87.43% accuracy on IFEval, outperforming larger proprietary models such as GPT-4o. Additionally, we demonstrate substantial improvements on FollowBench while preserving general capabilities. Our trip wires show significant reductions in reward hacking rates. We will release models, code, and data for future research.

TL;DR: We present IFD, a training framework that balances instruction difficulty and enforces intent alignment. It exposes shortcut behaviors and significantly improves both robustness and accuracy.