Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with
  Minimalist Rule-Based RL
Authors: name: Che Liu; name: Haozhe Wang; name: Jiazhen Pan; name: Zhongwei Wan; name: Yong Dai; name: Fangzhen Lin; name: Wenjia Bai; name: Daniel Rueckert; name: Rossella Arcucci
Published: 2025-05-23 14:27:37

Abstract: Improving performance on complex tasks and enabling interpretable decision
making in large language models (LLMs), especially for clinical applications,
requires effective reasoning. Yet this remains challenging without supervised
fine-tuning (SFT) on costly chain-of-thought (CoT) data distilled from
closed-source models (e.g., GPT-4o). In this work, we present AlphaMed, the
first medical LLM to show that reasoning capability can emerge purely through
reinforcement learning (RL), using minimalist rule-based rewards on public
multiple-choice QA datasets, without relying on SFT or distilled CoT data.
AlphaMed achieves state-of-the-art results on six medical QA benchmarks,
outperforming models trained with conventional SFT+RL pipelines. On challenging
benchmarks (e.g., MedXpert), AlphaMed even surpasses larger or closed-source
models such as DeepSeek-V3-671B and Claude-3.5-Sonnet. To understand the
factors behind this success, we conduct a comprehensive data-centric analysis
guided by three questions: (i) Can minimalist rule-based RL incentivize
reasoning without distilled CoT supervision? (ii) How do dataset quantity and
diversity impact reasoning? (iii) How does question difficulty shape the
emergence and generalization of reasoning? Our findings show that dataset
informativeness is a key driver of reasoning performance, and that minimalist
RL on informative, multiple-choice QA data is effective at inducing reasoning
without CoT supervision. We also observe divergent trends across benchmarks,
underscoring limitations in current evaluation and the need for more
challenging, reasoning-oriented medical QA benchmarks.