SemCoT: Accelerating Chain-of-Thought Reasoning through
  Semantically-Aligned Implicit Tokens
Authors: name: Yinhan He; name: Wendy Zheng; name: Yaochen Zhu; name: Zaiyi Zheng; name: Lin Su; name: Sriram Vasudevan; name: Qi Guo; name: Liangjie Hong; name: Jundong Li
Published: 2025-10-28 20:11:54

Abstract: The verbosity of Chain-of-Thought (CoT) reasoning hinders its mass deployment
in efficiency-critical applications. Recently, implicit CoT approaches have
emerged, which encode reasoning steps within LLM's hidden embeddings (termed
``implicit reasoning'') rather than explicit tokens. This approach accelerates
CoT by reducing the reasoning length and bypassing some LLM components.
However, existing implicit CoT methods face two significant challenges: (1)
they fail to preserve the semantic alignment between the implicit reasoning
(when transformed to natural language) and the ground-truth reasoning,
resulting in a significant CoT performance degradation, and (2) they focus on
reducing the length of the implicit reasoning; however, they neglect the
considerable time cost for an LLM to generate one individual implicit reasoning
token. To tackle these challenges, we propose a novel semantically-aligned
implicit CoT framework termed SemCoT. In particular, for the first challenge,
we design a contrastively trained sentence transformer that evaluates semantic
alignment between implicit and explicit reasoning, which is used to enforce
semantic preservation during implicit reasoning optimization. To address the
second challenge, we introduce an efficient implicit reasoning generator by
finetuning a lightweight language model using knowledge distillation. This
generator is guided by our sentence transformer to distill ground-truth
reasoning into semantically aligned implicit reasoning, while also optimizing
for accuracy. SemCoT is the first approach that enhances CoT efficiency by
jointly optimizing token-level generation speed and preserving semantic
alignment with ground-truth reasoning. Extensive experiments demonstrate the
superior performance of SemCoT compared to state-of-the-art methods in both
efficiency and effectiveness. Our code can be found at
https://github.com/YinhanHe123/SemCoT/.