O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple
  Distillation, Big Progress or Bitter Lesson?
Authors: name: Zhen Huang; name: Haoyang Zou; name: Xuefeng Li; name: Yixiu Liu; name: Yuxiang Zheng; name: Ethan Chern; name: Shijie Xia; name: Yiwei Qin; name: Weizhe Yuan; name: Pengfei Liu
Published: 2024-11-25 15:31:27

Abstract: This paper presents a critical examination of current approaches to
replicating OpenAI's O1 model capabilities, with particular focus on the
widespread but often undisclosed use of knowledge distillation techniques.
While our previous work explored the fundamental technical path to O1
replication, this study reveals how simple distillation from O1's API, combined
with supervised fine-tuning, can achieve superior performance on complex
mathematical reasoning tasks. Through extensive experiments, we show that a
base model fine-tuned on simply tens of thousands of samples O1-distilled
long-thought chains outperforms O1-preview on the American Invitational
Mathematics Examination (AIME) with minimal technical complexity. Moreover, our
investigation extends beyond mathematical reasoning to explore the
generalization capabilities of O1-distilled models across diverse tasks:
hallucination, safety and open-domain QA. Notably, despite training only on
mathematical problem-solving data, our models demonstrated strong
generalization to open-ended QA tasks and became significantly less susceptible
to sycophancy after fine-tuning. We deliberately make this finding public to
promote transparency in AI research and to challenge the current trend of
obscured technical claims in the field. Our work includes: (1) A detailed
technical exposition of the distillation process and its effectiveness, (2) A
comprehensive benchmark framework for evaluating and categorizing O1
replication attempts based on their technical transparency and reproducibility,
(3) A critical discussion of the limitations and potential risks of
over-relying on distillation approaches, our analysis culminates in a crucial
bitter lesson: while the pursuit of more capable AI systems is important, the
development of researchers grounded in first-principles thinking is paramount.