Revisiting the Uniform Information Density Hypothesis in LLM Reasoning
  Traces
Authors: name: Minju Gwak; name: Guijin Son; name: Jaehyung Kim
Published: 2025-10-08 12:37:04

Abstract: The Uniform Information Density (UID) hypothesis suggests that effective
communication maintains a stable flow of information. In this work, we revisit
this principle in the context of large language model (LLM) reasoning traces,
asking whether step-level uniformity reflects reasoning quality. To this end,
we propose an entropy-based stepwise information density metric and introduce
two complementary measures of uniformity, local and global uniformity scores.
Across the experiments on six different reasoning benchmarks, we find that
step-level uniformity not only provides a strong theoretical lens but also
yields practical performance benefits; for example, selecting reasoning traces
with more uniform information density at the step-level improves accuracy by
10-32\% relative gains over baselines at AIME2025. Our analysis further reveals
that correct reasoning traces tend to avoid sharp information density spikes,
while incorrect traces exhibit irregular information bursts. These results
demonstrate that UID-inspired information density measures outperform
alternative internal signals as predictors of reasoning quality. Results
highlight the uniformity of the information density as a robust diagnostic and
selection criterion for building more reliable and accurate reasoning systems.