Title: Test-Time Scaling with Reflective Generative Model
Primary area: foundation or frontier models, including LLMs
Keywords: LLM Reasoning

Abstract: We introduce a new Reflective Generative Model (RGM), which obtains OpenAI o3-mini's performance via a novel Reflective Generative Form. This form focuses on high-quality reasoning trajectory selection and contains two novelties: 1) A unified interface for policy and process reward model: we share the backbone network and use task-specific heads for reasoning trajectory predicting and scoring respectively, introducing only 50M extra parameters for trajectory scoring. 2) Eliminating the reliance on process-level annotation: we provide a self-supervised process reward model (SPRM), which can directly learn the high-quality reasoning trajectory selection from the outcome reward. Equipped with the reflective generative form, RGM is naturally suitable for test-time scaling based on the controllable thinking length. Experiments demonstrate that our RGM with 50M SPRM outperforms 72B reward models and enables QwQ-32B outperforms OpenAI o3-mini on AIME24 (84.2 vs 79.6) and HMMT25 (53.1 vs 53.0). Code will be available.