MR-Align: Meta-Reasoning Informed Factuality Alignment for Large
  Reasoning Models
Authors: name: Xinming Wang; name: Jian Xu; name: Bin Yu; name: Sheng Lian; name: Hongzhu Yi; name: Yi Chen; name: Yingjian Zhu; name: Boran Wang; name: Hongming Yang; name: Han Hu; name: Xu-Yao Zhang; name: Cheng-Lin Liu
Published: 2025-10-27 15:00:54

Abstract: Large reasoning models (LRMs) show strong capabilities in complex reasoning,
yet their marginal gains on evidence-dependent factual questions are limited.
We find this limitation is partially attributable to a reasoning-answer hit
gap, where the model identifies the correct facts during reasoning but fails to
incorporate them into the final response, thereby reducing factual fidelity. To
address this issue, we propose MR-ALIGN, a Meta-Reasoning informed alignment
framework that enhances factuality without relying on external verifiers.
MR-ALIGN quantifies state transition probabilities along the model's thinking
process and constructs a transition-aware implicit reward that reinforces
beneficial reasoning patterns while suppressing defective ones at the atomic
thinking segments. This re-weighting reshapes token-level signals into
probability-aware segment scores, encouraging coherent reasoning trajectories
that are more conducive to factual correctness. Empirical evaluations across
four factual QA datasets and one long-form factuality benchmark show that
MR-ALIGN consistently improves accuracy and truthfulness while reducing
misleading reasoning. These results highlight that aligning the reasoning
process itself, rather than merely the outputs, is pivotal for advancing
factuality in LRMs.