Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for
  RLVR
Authors: name: Jiakang Wang; name: Runze Liu; name: Fuzheng Zhang; name: Xiu Li; name: Guorui Zhou
Published: 2025-07-21 16:34:01

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective
post-training method for improving the reasoning abilities of Large Language
Models (LLMs), mainly by shaping higher-order behaviors such as reflection and
planning. However, previous RLVR algorithms often apply uniform training
signals to all tokens, without considering the different roles of low-entropy
knowledge-related tokens and high-entropy reasoning-related tokens. Some recent
methods try to separate these token types by gradient masking or asynchronous
updates, but these approaches may break semantic dependencies in the model
output and hinder effective learning. In this work, we propose Archer, an
entropy-aware RLVR approach with dual-token constraints and synchronous
updates. Specifically, our method applies weaker KL regularization and higher
clipping thresholds to reasoning tokens to encourage exploration, while using
stronger constraints on knowledge tokens to maintain factual knowledge.
Experimental results on several mathematical reasoning and code generation
benchmarks show that our approach significantly outperforms previous RLVR
methods, reaching or exceeding state-of-the-art performance among models of
comparable size. The code is available at
https://github.com/wizard-III/ArcherCodeR.