Before revision: Existing methods for distillation use the conventional training approach where all samples participate equally in the process and are thus highly inefficient in terms of datautilization. In this paper, a novel data-efficient approach to transfer the knowledge from a teacher model to a student model is presented.
After revision: Existing methods for distillation do not efficiently utilize the training data. This work presents a novel approach to a student model is presented.