Title: Regressor-free Intermediate Layer Distillation via Teacher Pruning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: knowledge distillation, prunining

Abstract: In deep learning, knowledge distillation from a teacher model to a smaller student model is commonly utilized to develop compact models without significantly sacrificing performance. Among various approaches, intermediate layer distillation often encounters a dimensional discrepancy between the teacher’s layer and the corresponding student’s layer. Typically, this problem is addressed by introduc-
ing a regressor to the student’s target layer to forcibly align dimensions. However, because distillation through the regressor transfers knowledge indirectly, the teacher’s knowledge is not fully transmitted to the student. Through preliminary experiments using the probing method, we first investigate whether the regressor effectively transfers the intended knowledge and reveal inefficiency in the conventional regressor-based method. Motivated by these findings, we propose an alternative method inspired by pruning techniques that directly adjusts the teacher’s target layer dimensions to match those of the student. Extensive experiments across diverse backbone architectures and datasets demonstrate that our method consistently achieves superior accuracy compared to conventional regressor-based
approaches. In particular, for ResNet models trained on CIFAR-100 and TinyImageNet, the student trained with our method attains accuracies of 77.50% and 49.23%, respectively, even exceeding the teacher’s accuracy.