Title: Quantifying Cross-Domain Knowledge Distillation in the Presence of Domain shift
Primary area: learning theory
Keywords: knowledge distillation;domain adaption; generalization error; random matrix

Abstract: Cross-domain knowledge distillation often suffers from domain shift. Although domain adaptation methods have shown strong empirical success in addressing this issue, their theoretical foundations remain underdeveloped. In this paper, we study knowledge distillation in a teacher–student framework for regularized linear regression and derive high-dimensional asymptotic excess risk for the student estimator, accounting for both covariate shift and model shift. This asymptotic analysis enables a precise characterization of the performance gain in cross-domain knowledge distillation. Our results demonstrate that, even under substantial shifts between the source and target domains, it remains feasible to identify an imitation parameter for which the student model outperforms the student-only baseline. Moreover, we show that the student's generalization performance exhibits the double descent phenomenon.

TL;DR: We theoretically quantify the potential performance gain from cross-domain knowledge distillation.