Title: Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?
Primary area: foundation or frontier models, including LLMs
Keywords: Distillation, AI-Feedback, large language model

Abstract: Distillation of large language models (LLMs) has traditionally focused on transferring teacher responses, overlooking the fact that teachers also possess strong evaluative capacity. As a result, students learn what to answer, but not which answers are preferable. This gap limits generalization, propagates teacher errors, and prevents students from improving beyond imitation. Therefore, we propose a unified distillation framework that transfers both responses and evaluation ability. Our key idea is to distill reward signals from the teacher, eliminating the need for costly human annotations. However, extracting reliable reward signals from LLMs is challenging because they are optimized for generation rather than evaluation. Therefore, we introduce an adaptive reward distillation strategy that applies majority voting for verifiable tasks and LLM-as-Judge for open-ended tasks.
This yields noisy yet effective self-supervised signals without human annotations. 
To mitigate distribution shift, we systematically collect and label both teacher- and student-generated responses, which are used to train a reward model. The student is first warmed up with supervised fine-tuning on high-quality teacher responses, then refined with reinforcement learning guided by the learned reward model. Experiments on GSM8K, GSM-Plus, MMLU-Pro, and AlpacaEval2 demonstrate consistent gains over supervised fine-tuning, with smaller students in some cases even surpassing their teachers. 
These results highlight our method as a scalable and effective paradigm for training efficient yet competitive LLMs.