Unleashing the Reasoning Potential of Pre-trained LLMs by Critique
  Fine-Tuning on One Problem
Authors: name: Yubo Wang, name: Ping Nie, name: Kai Zou, name: Lijun Wu, name: Wenhu Chen

We have witnessed that strong LLMs like Qwen-Math, MiMo, and Phi-4 possess
immense reasoning potential inherited from the pre-training stage. With
reinforcement learning (RL), these models can improve dramatically on reasoning
tasks. Recent studies have shown that even RL on a single problem can unleash
these models' reasoning capabilities. However, RL is not only expensive but
also unstable. Even one-shot RL requires hundreds of GPU hours. This raises a
critical question: Is there a more efficient way to unleash the reasoning
potential of these powerful base LLMs? In this work, we demonstrate that
Critique Fine-Tuning (CFT) on only one problem can effectively unleash the
reasoning potential of LLMs. Our method constructs critique data by collecting
diverse model-generated solutions to a single problem and using teacher LLMs to
provide detailed critiques. We fine-tune Qwen and Llama family models, ranging
from 1.5B to 14B parameters, on the CFT data and observe significant
performance gains across diverse reasoning tasks. For example, with just 5 GPU
hours of training, Qwen-Math-7B-CFT show an average improvement of 15% on six
math benchmarks and 16% on three logic reasoning benchmarks. These results are
comparable to or even surpass the results from RL with 20x less compute.
Ablation studies reveal the robustness of one-shot CFT across different prompt
problems. These results highlight one-shot CFT as a simple, general, and
compute-efficient approach to unleashing the reasoning capabilities of modern
LLMs.