Simple Semi-supervised Knowledge Distillation from Vision-Language
  Models via texttt{D}ual-texttt{H}ead
  texttt{O}ptimization
Authors: name: Seongjae Kang; name: Dong Bok Lee; name: Hyungjoon Jang; name: Sung Ju Hwang
Published: 2025-05-12 15:39:51

Abstract: Vision-language models (VLMs) have achieved remarkable success across diverse
tasks by leveraging rich textual information with minimal labeled data.
However, deploying such large models remains challenging, particularly in
resource-constrained environments. Knowledge distillation (KD) offers a
well-established solution to this problem; however, recent KD approaches from
VLMs often involve multi-stage training or additional tuning, increasing
computational overhead and optimization complexity. In this paper, we propose
texttt{D}ual-texttt{H}ead
texttt{O}ptimization (texttt{DHO}) -- a simple yet
effective KD framework that transfers knowledge from VLMs to compact,
task-specific models in semi-supervised settings. Specifically, we introduce
dual prediction heads that independently learn from labeled data and teacher
predictions, and propose to linearly combine their outputs during inference. We
observe that DHO mitigates gradient conflicts between supervised and
distillation signals, enabling more effective feature learning than single-head
KD baselines. As a result, extensive experiments show that DHO
consistently outperforms baselines across multiple domains and fine-grained
datasets. Notably, on ImageNet, it achieves state-of-the-art performance,
improving accuracy by 3% and 0.1% with 1% and 10% labeled data, respectively,
while using fewer parameters.