Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary
  Feedback
Authors: name: Yen-Ting Lin; name: Di Jin; name: Tengyu Xu; name: Tianhao Wu; name: Sainbayar Sukhbaatar; name: Chen Zhu; name: Yun He; name: Yun-Nung Chen; name: Jason Weston; name: Yuandong Tian; name: Arash Rahnama; name: Sinong Wang; name: Hao Ma; name: Han Fang
Published: 2025-01-18 15:38:03

Abstract: Large language models (LLMs) have recently demonstrated remarkable success in
mathematical reasoning. Despite progress in methods like chain-of-thought
prompting and self-consistency sampling, these advances often focus on final
correctness without ensuring that the underlying reasoning process is coherent
and reliable. This paper introduces Step-KTO, a training framework that
combines process-level and outcome-level binary feedback to guide LLMs toward
more trustworthy reasoning trajectories. By providing binary evaluations for
both the intermediate reasoning steps and the final answer, Step-KTO encourages
the model to adhere to logical progressions rather than relying on superficial
shortcuts. Our experiments on challenging mathematical benchmarks show that
Step-KTO significantly improves both final answer accuracy and the quality of
intermediate reasoning steps. For example, on the MATH-500 dataset, Step-KTO
achieves a notable improvement in Pass@1 accuracy over strong baselines. These
results highlight the promise of integrating stepwise process feedback into LLM
training, paving the way toward more interpretable and dependable reasoning
capabilities.