HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs
Authors: name: Junying Chen, name: Zhenyang Cai, name: Ke Ji, name: Xidong Wang, name: Wanlong Liu, name: Rongsheng Wang, name: Jianye Hou, name: Benyou Wang

The breakthrough of OpenAI o1 highlights the potential of enhancing reasoning
to improve LLM. Yet, most research in reasoning has focused on mathematical
tasks, leaving domains like medicine underexplored. The medical domain, though
distinct from mathematics, also demands robust reasoning to provide reliable
answers, given the high standards of healthcare. However, verifying medical
reasoning is challenging, unlike those in mathematics. To address this, we
propose verifiable medical problems with a medical verifier to check the
correctness of model outputs. This verifiable nature enables advancements in
medical reasoning through a two-stage approach: (1) using the verifier to guide
the search for a complex reasoning trajectory for fine-tuning LLMs, (2)
applying reinforcement learning (RL) with verifier-based rewards to enhance
complex reasoning further. Finally, we introduce HuatuoGPT-o1, a medical LLM
capable of complex reasoning, which outperforms general and medical-specific
baselines using only 40K verifiable problems. Experiments show complex
reasoning improves medical problem-solving and benefits more from RL. We hope
our approach inspires advancements in reasoning across medical and other
specialized domains.