o1-Coder: an o1 Replication for Coding
Authors: name: Yuxiang Zhang; name: Shangxi Wu; name: Yuqi Yang; name: Jiangming Shu; name: Jinlin Xiao; name: Chao Kong; name: Jitao Sang
Published: 2024-11-29 07:19:56

Abstract: The technical report introduces O1-CODER, an attempt to replicate OpenAI's o1
model with a focus on coding tasks. It integrates reinforcement learning (RL)
and Monte Carlo Tree Search (MCTS) to enhance the model's System-2 thinking
capabilities. The framework includes training a Test Case Generator (TCG) for
standardized code testing, using MCTS to generate code data with reasoning
processes, and iteratively fine-tuning the policy model to initially produce
pseudocode, followed by the generation of the full code. The report also
addresses the opportunities and challenges in deploying o1-like models in
real-world applications, suggesting transitioning to the System-2 paradigm and
highlighting the imperative for environment state updates. Updated model
progress and experimental results will be reported in subsequent versions. All
source code, curated datasets, as well as the derived models will be disclosed
at https://github.com/ADaM-BJTU/O1-CODER .