Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm
  Enables Fine-Grained Policy Optimization
Authors: name: Yang Li, name: Zhichen Dong, name: Yuhan Sun, name: Weixun Wang, name: Shaopan Xiong, name: Yijia Luo, name: Jiashun Liu, name: Han Lu, name: Jiamang Wang, name: Wenbo Su, name: Bo Zheng, name: Junchi Yan
Organization: alibaba

The reasoning pattern of Large language models (LLMs) remains opaque, and
Reinforcement learning (RL) typically applies uniform credit across an entire
generation, blurring the distinction between pivotal and routine steps. This
work positions attention as a privileged substrate that renders the internal
logic of LLMs legible, not merely as a byproduct of computation, but as a
mechanistic blueprint of reasoning itself. We first distinguish attention heads
between locally and globally focused information processing and reveal that
locally focused heads produce a sawtooth pattern near the diagonal indicating
phrasal chunks, while globally focused heads expose tokens that exert broad
downstream influence over future tokens. We formalize these with two metrics:
1) Windowed Average Attention Distance, which measures the extent of backward
attention within a clipped window; 2) Future Attention Influence, which
quantifies a token's global importance as the average attention it receives
from subsequent tokens. Taken together, these signals reveal a recurring
preplan-and-anchor mechanism, where the model first performs a long-range
contextual reference to generate an introductory token, which is immediately
followed by or coincides with a semantic anchor token that organizes subsequent
reasoning. Leveraging these insights, we introduce three novel RL strategies
that dynamically perform targeted credit assignment to critical nodes (preplan
tokens, anchor tokens, and their temporal coupling) and show consistent
performance gains across various reasoning tasks. By aligning optimization with
the model's intrinsic reasoning rhythm, we aim to transform opaque optimization
into an actionable structure-aware process, hoping to offer a potential step
toward more transparent and effective optimization of LLM reasoning.