Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning
Authors: name: Haolei Xu, name: Yuchen Yan, name: Yongliang Shen, name: Wenqi Zhang, name: Guiyang Hou, name: Shengpei Jiang, name: Kaitao Song, name: Weiming Lu, name: Jun Xiao, name: Yueting Zhuang

Large language models (LLMs) have achieved remarkable progress on
mathematical tasks through Chain-of-Thought (CoT) reasoning. However, existing
mathematical CoT datasets often suffer from Thought Leaps due to experts
omitting intermediate steps, which negatively impacts model learning and
generalization. We propose the CoT Thought Leap Bridge Task, which aims to
automatically detect leaps and generate missing intermediate reasoning steps to
restore the completeness and coherence of CoT. To facilitate this, we
constructed a specialized training dataset called ScaleQM+, based on the
structured ScaleQuestMath dataset, and trained CoT-Bridge to bridge thought
leaps. Through comprehensive experiments on mathematical reasoning benchmarks,
we demonstrate that models fine-tuned on bridged datasets consistently
outperform those trained on original datasets, with improvements of up to
+5.87% on NuminaMath. Our approach effectively enhances distilled data (+3.02%)
and provides better starting points for reinforcement learning (+3.1%),
functioning as a plug-and-play module compatible with existing optimization
techniques. Furthermore, CoT-Bridge demonstrate improved generalization to
out-of-domain logical reasoning tasks, confirming that enhancing reasoning
completeness yields broadly applicable benefits.