Thought-Augmented Policy Optimization: Bridging External Guidance and
  Internal Capabilities
Authors: name: Jinyang Wu; name: Chonghua Liao; name: Mingkuan Feng; name: Shuai Zhang; name: Zhengqi Wen; name: Pengpeng Shao; name: Huazhe Xu; name: Jianhua Tao
Published: 2025-05-21 16:06:10

Abstract: Reinforcement learning (RL) has emerged as an effective method for training
reasoning models. However, existing RL approaches typically bias the model's
output distribution toward reward-maximizing paths without introducing external
knowledge. This limits their exploration capacity and results in a narrower
reasoning capability boundary compared to base models. To address this
limitation, we propose TAPO (Thought-Augmented Policy Optimization), a novel
framework that augments RL by incorporating external high-level guidance
("thought patterns"). By adaptively integrating structured thoughts during
training, TAPO effectively balances model-internal exploration and external
guidance exploitation. Extensive experiments show that our approach
significantly outperforms GRPO by 99% on AIME, 41% on AMC, and 17% on Minerva
Math. Notably, these high-level thought patterns, abstracted from only 500
prior samples, generalize effectively across various tasks and models. This
highlights TAPO's potential for broader applications across multiple tasks and
domains. Our further analysis reveals that introducing external guidance
produces powerful reasoning models with superior explainability of inference
behavior and enhanced output readability.