Meta-Awareness Enhances Reasoning Models: Self-Alignment Reinforcement
  Learning
Authors: name: Yoonjeon Kim; name: Doohyuk Jang; name: Eunho Yang
Published: 2025-09-26 14:05:48

Abstract: Recent studies on reasoning models explore the meta-awareness of language
models, the ability to know how to think by itself. We argue that large
reasoning models lack this meta-awareness property by proving severe
misalignment between true rollouts and predicted meta information. We posit
that aligning meta-prediction with true rollouts will lead to significant
performance gains. To verify this hypothesis, we design a training pipeline
that boosts Meta-Awareness via Self-Alignment (MASA), and prove that enhanced
meta-awareness directly translates to improved accuracy. Unlike existing
meta-cognitive reasoning models, our method does not require external training
sources but leverages self-generated signals to train meta-awareness. Moreover,
our method enables efficient training by i) filtering out zero-variance prompts
that are either trivial or unsolvable and ii) cutting off lengthy rollouts when
they are unlikely to lead to correct answers. The results are inspiring: our
strategy yields significant improvements in both accuracy and training
efficiency on in-domain tasks and shows strong generalization to out-of-domain
benchmarks. More specifically, our method can speed up GRPO training by over
1.28x to reach the same performance, and achieve a 19.3% gain in accuracy on
AIME25, and a 6.2 % average gain over six mathematics benchmarks. Training with
meta-cognitive guidance enhances out-of-domain generalization, giving a 3.87 %
boost on GPQA-Diamond and a 2.08 % overall accuracy gain across 13 benchmarks
spanning logical, scientific, and coding domains.