Advantage-Guided Distillation for Preference Alignment in Small Language Models

Authors: Shiping Gao, Fanqi Wan, Jiajian Guo, Xiaojun Quan, Qifan Wang
Primary area: foundation or frontier models, including LLMs
Keywords: Preference Alignment; Large language model; Knowledge Distillation; Advantage Function

Abstract:
Alignment techniques enable Large Language Models (LLMs) to generate outputs that align with human preferences and play a crucial role in their effectiveness. However, their impact often diminishes when applied to Small Language Models (SLMs), likely due to the limited capacity of these models. Instead of directly applying existing alignment techniques to SLMs, we propose to utilize a well-aligned teacher LLM to guide the alignment process for these models, thereby facilitating the transfer of the teacher's knowledge of human preferences to the student model. To achieve this, we first explore a straightforward approach, Dual-Constrained Knowledge Distillation (DCKD), that employs knowledge distillation with two KL-divergence constraints from the aligned teacher to the unaligned student. To further enhance the student's ability to distinguish between preferred and dispreferred responses, we then propose Advantage-Guided Distillation for Preference Alignment (ADPA), which leverages an advantage function from the aligned teacher to deliver more nuanced, distribution-level reward signals for the student's alignment. Our experimental results show that these two approaches appreciably improve the alignment of SLMs and narrow the performance gap with larger counterparts. Among them, ADPA demonstrates superior performance and achieves even greater effectiveness when integrated with DCKD. Our code is available at https://github.com/SLIT-AI/ADPA .

Decision: accept (spotlight)
Reviewer scores: 8, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1729035316854

thread_type:
official_review

<thread_object>
creator: reviewer_3nf4

<content>
summary:
This paper explores knowledge distillation (KD) for LLMs during the preference alignment stage. It first introduces a simple baseline DCKD which applies Vanilla KLD-based KD on both the positive and negative examples. Then, the paper propose ADPA to enhance the contrastive signals of KD. Experiments show the effectiveness of ADPA and its components, suggesting the importance of leveraging the perference signals for KD.

strengths:
1. How to leverage the preference signals for KD is an important but under-explored problem.
2. The methods are concise and easy to implement.
3.  The empirical results of ADPA is strong.

weaknesses:
1. Although ADPA seems effective empirically, it is still unclear how the improvement is related to the motivation of the method. From Section 3.3 and Algorithm 1, it seems ADPA does not need the preference labels: only the prompts and the grouth truth reponses works for ADPA. How is ADPA related to preference alignment?
2. From Table 2, it seems that the reference teacher model is critical for the effectiveness of ADPA. It would be better to add more explanation on why the difference between $\pi_{dpo}$ and $\pi_{ref}$ should be considered, rather than $\pi_{dpo}$. Furthermore, what if $\pi_{dpo}$ is removed from Equation (11)? (to show the effect of $-\log \pi_{sft}$ alone)

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648416921

thread_type:
official_review

<thread_object>
creator: reviewer_ev8c

<content>
summary:
This works combines the concepts of RLHF and knoledge distillation to propose the advantage-guided distillation method for preference alignment. This realizes the impressive the performance improvement of the small models in the preference alignment stage.

strengths:
1. The idea of improving the performance of small model in the preference alignment stage is interesting.
2. Advantage-guided distillation from the preference-aligned teacher model to the student model is novel for knowledge distillation.
3. The experiments are detailed and the presentation is good.

weaknesses:
This is a good work. The proposed method is simple yet effective. I do not have additional concerns for it.

questions:
1. Why experiment on  H2O-Danube3-500M and H2O-Danube2-1.8B-Base, given the availability of more popular models today, such as LLaMA-3.2-1B, Qwen2.5-0.5B . If the experiment on more popular models can be provided, the result will be more solid.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665570441

thread_type:
official_review

<thread_object>
creator: reviewer_346s

<content>
summary:
This paper studies the alignment of small language models. It points out that directly applying alignment techniques to small language models may not work well due to the limited capacity of these models. To this end, the authors present 1) a straightforward approach Dual-Constrained KD by integrating both positive and negative signals, and 2) an enhanced approach Advantage-Guided Distillation for Preference Alignment that involves a distribution-0level reward signal given by an advantage function. Extensive experiments with three teacher-student model settings demonstrate the effectiveness of the proposed method.

strengths:
1. This paper investigates the challenge of aligning small language models, the proposed methods are well-motivated and principal.

2. The ADPA method considers the distribution-level reward signal, which is an advancement compared to the previous KD methods.

3. The KD baselines are comprehensively compared, and the studied teacher-student settings are representative.

4. This paper is well organized and the presentation is decent.

weaknesses:
1. Limited baseline of alignment methods. In Table 1, while the KD baselines are comprehensively compared, the alignment baseline only includes DPO.

2. Computation overhead of the KD methods. It is not clear whether the proposed methods (and other baseline KD methods) consume more training or inference resources compared to directly applying alignment techniques to the student models.

questions:
1. I wonder if directly applying some advanced alignment methods to the smaller language models can present a competitive performance.

2. Could you elaborate more on the computational cost of the proposed methods v.s. directly applying alignment techniques?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707243429

thread_type:
official_review

<thread_object>
creator: reviewer_pkn8

<content>
summary:
The paper explores how to enhance the effectiveness of small language models  to make their generated outputs more aligned with human preferences through preference alignment techniques. To address the issue that technologies like RLHF do not align well with human preferences on small language models, the paper proposes two methods: Dual-Constrained Knowledge Distillation (DCKD) and Advantage-Guided Distillation for Preference Alignment (ADPA). The experimental results show that both methods can significantly improve the alignment of small language models and narrow the performance gap with large models.

strengths:
- The two methods proposed in the paper combine RLHF with distillation, providing insights into the preference alignment of small language models and solving problems that previous methods had not addressed. 
- The experimental section is well-designed, with numerous comparative and ablation experiments to verify the performance improvements of the methods. 
- The paper is well-written, with detailed descriptions of the two methods, including algorithm steps and formulas, which are easy to understand.

weaknesses:
- ADPA provides more nuanced guidance signals, but the additional computations introduced by fine-grained signals may increase the computational overhead, and the paper seems to lack specific quantitative metrics on this point.
- The experimental section utilizes a rather singular evaluation dataset, without providing results on a broader range of models, domains, and data. Additionally, there is a lack of experiments assessing the impact of teachers of varying proficiency levels on the results.

questions:
- In Formula 11, $L_s$ seems to be undefined. Does it refer to $L_{SFT}$?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies alignment for small language models. The motivation is that we’d prefer to use small language models for efficiency reasons, but standard alignment approaches directly applied to these struggle. Instead, the authors propose a distillation-style approach where a (larger) aligned teacher is used to guide the small model. 

Both the motivation for the problem and the approach are nice. The results are quite strong. 

There are no strong weaknesses here; the reviewers brought up questions around the clarity of the writing, but these were addressed by the authors during rebuttal. Another potential question is whether the results hold up for other small models (or more generally, other combinations of teacher and student).

Overall, this is a solid work and worth accepting.