CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated
  Chain-of-Thought-based Reinforced Fine-Tuning
Authors: name: Wenqiao Zhu, name: Ji Liu, name: Rongjuncheng Zhang, name: Haipang Wu, name: Yulun Zhang

Reasoning capability plays a significantly critical role in the the broad
applications of Large Language Models (LLMs). To enhance the reasoning
performance of LLMs, diverse Reinforcement Learning (RL)-based fine-tuning
approaches have been proposed to address the limited generalization capability
of LLMs trained solely via Supervised Fine-Tuning (SFT). Despite their
effectiveness, two major limitations hinder the advancement of LLMs. First,
vanilla RL-based approaches ignore annotated Chain-of-Thought (CoT) and
incorporate unstable reasoning path sampling, which typically results in model
collapse, unstable training process, and suboptimal performance. Second,
existing SFT approaches generally overemphasize the annotated CoT, potentially
leading to performance degradation due to insufficient exploitation of
potential CoT. In this paper, we propose a Contrastive learning with annotated
CoT-based Reinforced Fine-Tuning approach, i.e., , to enhance the
reasoning performance of LLMs while addressing the aforementioned limitations.
Specifically, we propose learning a representation for each CoT. Based on this
representation, we design novel contrastive signals to guide the fine-tuning
process. Our approach not only fully exploits the available annotated CoT but
also stabilizes the fine-tuning procedure by incorporating an additional
unsupervised learning signal. We conduct comprehensive experiments and in-depth
analysis with three baseline approaches, two foundation models, and two
datasets to demonstrate significant advantages of  in terms of
robustness, performance (up to 10.15\%), and efficiency (up to 30.62\%). Code
is available at https://github.com/WNQzhu/CARFT.