SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5: Statistics of learned noise weight  $\gamma$  on MobileNetV1 student. (a) Histogram of learned  $\gamma$  during training. (b) curves of average  $\gamma$  in each epoch during training.
C.5 Effects of different feature types of distillations on object detection
We conduct experiments on COCO dataset to compare the effects of performing distillations on FPN features, classification outputs, and regression outputs. As the results shown in Tab. 12, compared with the original KD without feature denoise in DiffKD, our method obtains significant improvements on all the feature types. Besides, comparing these features, distillation on FPN feature obtains relative high performance, which demonstrates that the semantic information in intermediate features is more valuable than the responses. We also conduct experiment to combine all the feature distillations in DiffKD together, while it achieves similar performance of FPN feature, this infers that the distillation on FPN feature is sufficient to obtain a good performance.
Table 12: Comparison of original diffusion model (DM) and our efficient DM on COCO dataset.
DM	Params	GFLOPs	Training time	AP	AP	AP
UNet (DDPM)	13.62 M	650.77	1.80×8 GPU Days	39.1	58.0	41.9
Efficient DM (ours)	0.21 M	132.76	0.59×8 GPU Days	39.2	58.1	42.0
C.6 Efficiency analysis
In DiffKD, we use a light-weight diffusion model to denoise the student feature. Here we analyze the computation efficiency with comparisons to other methods. Compared to the vanilla KD, DiffKD has additional computations to denoise the intermediate feature and output logits on ImageNet. Specifically, for ResNet-18 student and ResNet-34 teacher, the additional FLOPs is 800M. However, compared to other feature-based KD methods, the cost is acceptable. For example, CRD [41] uses extra 260M FLOPs, ReviewKD has an additional computation cost of 1900M.
D.1 Limitation
In this paper, we only implement our diffusion model with simple convolutions and DDIM inference algorithm, while there exists recent advances of diffusion models that propose better transformer-based models and more efficient inference algorithm. Besides, we only use the traditional mean square error and KL divergence as our KD loss functions, while many novel losses could be leveraged
to further improve the distillation performance. The computational cost is larger than the simple logits distillation methods such as KD [14] and DIST [17], but the cost is comparable to existing feature distillation methods and does not affect the computation cost in inference.
D.2 Societal impacts
Investigating the efficacy of the proposed method would consume considerable computing resources. These efforts can contribute to increased carbon emissions, which could raise environmental concerns. However, the proposed knowledge distillation method can improve the performance of light-weight compact models, where replacing the heavy models with light models in production could save more energy consumption, and it is necessary to validate the efficacy of DiffKD adequately.
E Visualization of Predicted Classification Scores
We visualize the predicted classification scores in Fig. 6. The original predictions of student, have different sharpness compared to the teacher's, while the denoised predictions align better to the predictions of teacher.
Figure 6: Visualization of predicted scores of MobileNetV1 student on ImageNet validation set. DiffKD has an effect of matching the sharpness of the probabilistic distributions of teacher and student.
F Visualization of Features
F.1 Visualization details
We visualize the features of student and teacher models in the first output (stride 4) of FPN. The models used to extract the feature are RetinaNet-R50 (student) and RetinaNet-X101 (teacher) trained on COCO dataset. Following FGD [48], we average the feature map along channel axis and perform softmax on the spatial axis to measure the saliency of each pixel. Formally, with a given feature map  $\mathbf{X} \in \mathbb{R}^{C \times HW}$ , we first average the channels and get  $\mathbf{X}' \in \mathbb{R}^{HW}$ , where  $X_{i}' = \frac{1}{HW} \sum_{i=1}^{HW} (\mathbf{X}_{:,i})$ . Then we generate the attention map for visualization as
$$
\boldsymbol {V} = H \cdot W \cdot \operatorname {s o f t m a x} \left(\boldsymbol {X} ^ {\prime} / \tau\right), \tag {10}
$$
where  $\tau$  is the temperature factor for controlling the softness of distribution, and we set  $\tau = 0.5$ .
F.2 More visualizations
We visualize the student features, denoised student features, and teacher features in Fig. 7. First, by comparing the original student and teacher features, we can conclude that the discrepancy between student and teacher features is fairly large, and the student feature contains more noises and is not as salient as the teacher feature. While for the denoised feature generated by our DiffKD, it is very similar to the teacher feature, this infers that distillation on the denoised student feature and teacher feature can get rid of the noises that disturb the optimization.
Denoised student
Figure 7: Visualizations of student features, denoised student features and teacher features on COCO dataset.