IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with
  Verifiable Rewards
Authors: name: Xu Guo; name: Tianyi Liang; name: Tong Jian; name: Xiaogui Yang; name: Ling-I Wu; name: Chenhui Li; name: Zhihui Lu; name: Qipeng Guo; name: Kai Chen
Published: 2025-08-06 17:00:54

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) improves instruction
following capabilities of large language models (LLMs), but suffers from
training inefficiency due to inadequate difficulty assessment. Moreover, RLVR
is prone to over-optimization, where LLMs exploit verification shortcuts
without aligning to the actual intent of user instructions. We introduce
Instruction Following Decorator (IFDecorator}, a framework that wraps RLVR
training into a robust and sample-efficient pipeline. It consists of three
components: (1) a cooperative-adversarial data flywheel that co-evolves
instructions and hybrid verifications, generating progressively more
challenging instruction-verification pairs; (2) IntentCheck, a bypass module
enforcing intent alignment; and (3) trip wires, a diagnostic mechanism that
detects reward hacking via trap instructions, which trigger and capture
shortcut exploitation behaviors. Our Qwen2.5-32B-Instruct-IFDecorator achieves
87.43% accuracy on IFEval, outperforming larger proprietary models such as
GPT-4o. Additionally, we demonstrate substantial improvements on FollowBench
while preserving general capabilities. Our trip wires show significant
reductions in reward hacking rates. We will release models, code, and data for
future research.