ASPO: Asymmetric Importance Sampling Policy Optimization
Authors: name: Jiakang Wang; name: Runze Liu; name: Lei Lin; name: Wenping Hu; name: Xiu Li; name: Fuzheng Zhang; name: Guorui Zhou; name: Kun Gai
Published: 2025-10-07 15:54:24

Abstract: Recent Large Language Model (LLM) post-training methods rely on token-level
clipping mechanisms during Reinforcement Learning (RL). However, we identify a
fundamental flaw in this Outcome-Supervised RL (OSRL) paradigm: the Importance
Sampling (IS) ratios of positive-advantage tokens are mismatched, leading to
unbalanced token weighting for positive and negative tokens. This mismatch
suppresses the update of low-probability tokens while over-amplifying already
high-probability ones. To address this, we propose Asymmetric Importance
Sampling Policy Optimization (ASPO), which uses a simple yet effective strategy
that flips the IS ratios of positive-advantage tokens, aligning their update
direction with the learning dynamics of negative ones. AIS further incorporates
a soft dual-clipping mechanism to stabilize extreme updates while maintaining
gradient flow. Comprehensive experiments on coding and mathematical reasoning
benchmarks demonstrate that ASPO significantly mitigates premature convergence,
improves training stability, and enhances final performance over strong
GRPO-based baselines. Our analysis provides new insights into the role of
token-level weighting in OSRL and highlights the critical importance of
correcting IS in LLM RL. The code and models of ASPO are available at
https://github.com/wizard-III/Archer2.0.