SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Object detection performance with baseline settings on COCO val set. T: teacher. S: student. †: we replace MSE with an attention-based MSE loss.
Method	AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_S \)	\( AP_M \)	\( AP_L \)
Two-stage detectors
T: Faster RCNN-R101	39.8	60.1	43.3	22.5	43.6	52.8
S: Faster RCNN-R50	38.4	59.0	42.0	21.5	42.1	50.3
Fitnet [36]	38.9	59.5	42.4	21.9	42.2	51.6
FRS [9]	39.5	60.1	43.3	22.3	43.6	51.7
FGD [48]	40.4	-	-	22.8	44.5	53.5
DiffKD	40.6	60.9	43.9	23.0	44.5	54.0
\( DiffKD^† \)	40.7	61.0	44.3	22.6	44.6	53.7
One-stage detectors
T: RetinaNet-R101	38.9	58.0	41.5	21.0	42.8	52.4
S: RetinaNet-R50	37.4	56.7	39.6	20.0	40.7	49.7
Fitnet [36]	37.4	57.1	40.0	20.8	40.8	50.9
FRS [9]	39.3	58.8	42.0	21.5	43.3	52.6
FGD [48]	39.6	-	-	22.9	43.7	53.6
DiffKD	39.7	58.6	42.1	21.6	43.8	53.3
\( DiffKD^† \)	39.8	58.7	42.5	21.5	43.6	53.2
Anchor-free detectors
T: FCOS-R101	40.8	60.0	44.0	24.2	44.3	52.4
S: FCOS-R50	38.5	57.7	41.0	21.9	42.8	48.6
FRS [9]	40.9	60.3	43.6	25.7	45.2	51.2
FGD [48]	42.1	-	-	27.0	46.0	54.6
DiffKD	42.4	61.0	45.8	26.6	45.9	54.8
\( DiffKD^† \)	42.5	61.1	45.6	25.2	46.8	55.1
Table 5: Object detection performance with stronger teachers on COCO val set. CM RCNN: Cascade Mask RCNN. †: we replace MSE with an attention-based MSE loss.
Method	AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_S \)	\( AP_M \)	\( AP_L \)
Two-stage detectors
T: CM RCNN-X101	45.6	64.1	49.7	26.2	49.6	60.0
S: Faster RCNN-R50	38.4	59.0	42.0	21.5	42.1	50.3
COFD [13]	38.9	60.1	42.6	21.8	42.7	50.7
FKD [50]	41.5	62.2	45.1	23.5	45.0	55.3
FGD [48]	42.0	-	-	23.7	46.4	55.5
DiffKD	42.2	62.8	46.0	24.2	46.6	55.3
\( DiffKD^† \)	42.4	62.9	46.4	24.0	46.7	55.2
One-stage detectors
T: RetinaNet-X101	41.2	62.1	45.1	24.0	45.5	53.5
S: RetinaNet-R50	37.4	56.7	39.6	20.0	40.7	49.7
COFD [13]	37.8	58.3	41.1	21.6	41.2	48.3
FKD [50]	39.6	58.8	42.1	22.7	43.3	52.5
FGD [48]	40.4	-	-	23.4	44.7	54.1
DiffKD	40.7	60.0	43.2	22.2	45.0	55.2
\( DiffKD^† \)	41.4	60.7	44.0	23.0	45.4	55.8
Anchor-free detectors
T: RepPoints-X101	44.2	65.5	47.8	26.2	48.4	58.5
S: RepPoints-R50	38.6	59.6	41.6	22.5	42.2	50.4
FKD [50]	40.6	61.7	43.8	23.4	44.6	53.0
FGD [48]	41.3	-	-	24.5	45.2	54.0
DiffKD	41.7	62.6	44.9	23.6	45.4	55.9
\( DiffKD^† \)	41.9	62.8	45.0	24.4	45.7	55.3
Results for CIFAR-100 dataset are summarized in Appendix C.
4.2 Object detection
Settings. Following FGD [48], we conduct experiments on baseline settings and stronger teacher settings. On baseline settings, we adopt various network architectures, including two-stage detector Faster-RCNN [34], one-stage detector RetinaNet [23], and anchor-free detector FCOS [42], and use ResNet-50 [11] as student models, and ResNet-101 as teachers, respectively. The training strategy is the most common one in previous methods [9, 18, 48]. While for the stronger teacher settings, we train students with much stronger teacher models, including two-stage detector Cascade Mask RCNN [2], one-stage detector RetinaNet [23], and anchor-free detector RepPoints [49], and stronger backbone ResNeXt-101 (X101) [45].
We conduct feature distillation on the predicted feature maps, and train the student with our DiffKD loss  $\mathcal{L}_{\mathrm{diffkd}}$ , regression KD loss, and task loss. Note that we do not use linear autoencoder in DiffKD since the number of channels in FPN is only 256. We set  $\lambda_1 = \lambda_2 = 1$ . Besides using MSE loss in feature distillation, we also adopted a simple attention-based MSE loss (marked with  $\dagger$  in Tab. 4) inspired by FGD [48] to balance the foreground and background distillations, which is acknowledged important for detection KD [10, 18, 48]. Details can be found in Appendix B.
Results on baseline settings. Our results compared with previous methods are summarized in Table 4. Our DiffKD can significantly improve the performance of student models over their teachers on various network architectures. For instance, DiffKD improve FCOS-R50 by 4.0 AP and surpasses FGD [48] by 0.4 AP. Besides, the attention-based MSE loss affords consistent improvements on vanilla DiffKD by  $\sim$  0.1 AP.
Results on stronger teacher settings. We further investigate our efficacy on stronger teachers whose backbones are replaced by stronger ResNeXts [45]. As in Table 5, student detectors achieve
Table 6: Semantic segmentation results on Cityscapes dataset. †: trained from scratch. Other models are pretrained on ImageNet. FLOPs is measured based on an input size of  $1024 \times 2048$ .
Method	Params (M)	FLOPs (G)	mIoU (%)
Val	Test
T: DeepLabV3-R101	61.1	2371.7	78.07	77.46
S: DeepLabV3-R18	13.6	572.0	74.21	73.45
CWD [38]	13.6	572.0	75.55	74.07
CIRKD [46]	13.6	572.0	76.38	75.05
MasKD [18]	13.6	572.0	77.00	75.59
DiffKD	13.6	572.0	77.78	76.24
S: DeepLabV3-R18†	13.6	572.0	65.17	65.47
CWD [38]	13.6	572.0	67.74	67.35
CIRKD [46]	13.6	572.0	68.18	68.22
MasKD [18]	13.6	572.0	73.95	73.74
DiffKD	13.6	572.0	74.45	74.52
Method	Params (M)	FLOPs (G)	mIoU (%)
Val	Test
T: DeepLabV3-R101	61.1	2371.7	78.07	77.46
S: PSPNet-R18	12.9	507.4	72.55	72.29
CWD [38]	12.9	507.4	74.36	73.57
CIRKD [46]	12.9	507.4	74.73	74.05
MasKD [18]	12.9	507.4	75.34	74.61
DiffKD	12.9	507.4	75.83	75.61
S: DeepLabV3-MBV2	3.2	128.9	73.12	72.36
CWD [38]	3.2	128.9	74.66	73.25
CIRKD [46]	3.2	128.9	75.42	74.03
MasKD [18]	3.2	128.9	75.26	74.23
DiffKD	3.2	128.9	75.71	74.96
more enhancements with our DiffKD, especially when with a Retina-X101 teacher, DiffKD gains a substantial improvement of 4.0 AP over the Retina-R50. Additionally, our methods outperforms existing KD methods with a large margin, and significantly surpasses FGD [48] when distilling on RetinaNet and RepPoints, by 1.0 and 0.6 AP, respectively. Moreover, comparing Table 4 with 5, we can infer that when the teacher is stronger, the benefit of DiffKD is more significant, as the discrepancy between the student and a stronger teacher is larger.
4.3 Semantic segmentation
Settings. Following CIRKD [46], we use DeepLabV3 [5] framework with ResNet-101 (R101) [11] backbone as the teacher network. While for the students, we use various frameworks (DeepLabV3 and PSPNet [54]) and backbones (ResNet-18 [11] and MobileNetV2 [37]) to valid our efficacy.
We conduct feature distillation on the predicted segmentation maps, and train the student with our DiffKD loss and task loss, as formulated in Eq. (9). Note that we do not use linear autoencoder in DiffKD since the number of channels in segmentation map is only 19. Detailed training strategies are summarized in Appendix B.
Results. The experimental results are summarized in Tab. 6. Our DiffKD significantly outperforms the state-of-the-art MasKD on all settings. For example, on DeepLabV3-R18 student, DiffKD improves MasKD by  $0.78\%$  on val set and 0.65 on test set.
Effects of adopting DiffKD on different types of features. On image classification, we distill both intermediate feature (the output feature of backbone before average pooling) and output logits with DiffKD. Here we conduct experiments to compare the effects of distillations on different features of MobileNetV1 student and ResNet-50 teacher in Tab. 7. We can see that, by adopting DiffKD to align the student features, both feature-level DiffKD and logits-level DiffKD can obtain obvious improvements without changing the loss functions. Meanwhile, combining feature and logits distilla
Table 7: Comparisons of different distillation features in DiffKD.
Method	Top-1 (%)
w/o KD	70.13
MSE	72.39
KL div.	70.68
DiffKD (feature)	73.16
DiffKD (logits)	72.89
DiffKD (feature + logits)	73.62
tions together in our final method achieves the optimal  $73.62\%$  top-1 accuracy.
Effects of linear autoencoder. We compare different dimensions of linear autoencoder in Tab. 8. We can see that, the linear autoencoder can significantly reduces the FLOPs of diffusion model, but if we
Table 8: Comparisons of different dimensions of linear autoencoder in DiffKD. We report the top-1 accuracy and FLOPs of diffusion models with MobileNetV1 student and ResNet-50 teacher on ImageNet.
w/o AE	128	256	512	1024	2048
Top-1 (%)	73.47	72.84	73.22	73.53	73.62	73.58
FLOPs (G)	1.41	0.04	0.09	0.21	0.58	1.82
conduct high compression ratios like 128 and 256 dimensions, the distillation performance will be severely weakened. Interestingly, AE with 512, 1024, and 2048 dimensions can outperform the one without AE, a possible reason is that the AE encodes common and valuable information in the original feature for reconstruction and would have better feature representations than the original feature. As a result, we use AE with 1024 dimension in our experiments for better performance-efficiency trade-off.
Effects of adaptive noise matching. We propose adaptive noise matching (ANM) to match the noisy level of student feature to the correct initial level in the denoising process. Here we conduct experiments to validate its efficacy. We train MobileNetV1 student with the only removal of ANM in our final method, and the DiffKD without ANM obtains  $73.34\%$  top-1 accuracy, which has a decrease of 0.28 on our DiffKD with ANM. This indicates that ANM can improve the performance by generating better denoised student feature.
Ablation on numbers of score function evaluations (NFEs). In diffusion models, the number of score function evaluations (NFEs) is a important factor for controlling the generation quality and efficiency. The early method such as DDPM [15] requires to run a complete timesteps in the reverse denoising process as training, which leads to a heavy computational budget. Recently, some works [27, 40] have been proposed to accelerate the denoising process by sampling a small number of timesteps. In this paper, we use DDIM [40] for speedup. Here, we conduct experiments to show the influence of different NFEs. As shown in Fig. 4, compared to the  $72.39\%$  accuracy of MSE baseline without denoising on student feature, only one-step denoising also achieves a significant improvement. However, its performance is weaker than that of those larger NFEs due to the limitation of denoising
Figure 4: Results of DiffKD with different NFEs. We use MobileNetV1 student and ResNet-50 teacher on ImageNet.
quality. A NFEs of 5 would suffice in our KD setting to achieve promising performance, and we use it in all experiments for a better efficiency-accuracy trade-off.
In this paper, we investigate the discrepancy between teacher and student in knowledge distillation. To reduce the discrepancy and improve the distillation performance, we proceed from a new perspective and propose to explicitly eliminate the noises in student feature with a diffusion model. Based on this idea, we further introduce a light-weight diffusion model with a linear autoencoder to reduce the computation cost of our method, and an adaptive noise matching module to align the student feature with the correct noisy level, thus improving the denoising performance. Extensive experiments on image classification, object detection, and semantic segmentation tasks validate our efficacy and generalization.
[47] J. Yang, B. Martinez, A. Bulat, and G. Tzimiropoulos. Knowledge distillation via softmax regression representation learning. In International Conference on Learning Representations, 2020. 6
[48] Z. Yang, Z. Li, X. Jiang, Y. Gong, Z. Yuan, D. Zhao, and C. Yuan. Focal and global knowledge distillation for detectors. arXiv preprint arXiv:2111.11837, 2021. 7, 8, 13, 17
[49] Z. Yang, S. Liu, H. Hu, L. Wang, and S. Lin. Repoints: Point set representation for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9657-9666, 2019. 7
[50] L. Zhang and K. Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In International Conference on Learning Representations, 2021. 7
[51] Y. Zhang, W. Chen, Y. Lu, T. Huang, X. Sun, and J. Cao. Avatar knowledge distillation: Self-ensemble teacher paradigm with uncertainty. arXiv preprint arXiv:2305.02722, 2023. 13
[52] Y. Zhang, T. Huang, J. Liu, T. Jiang, K. Cheng, and S. Zhang. Freekd: Knowledge distillation via semantic frequency prompt, 2023. 13
[53] B. Zhao, Q. Cui, R. Song, Y. Qiu, and J. Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953-11962, 2022. 2, 6
[54] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017. 8
A.1 Representation gap in knowledge distillation
An emerging topic in knowledge distillation is the representation gap between teacher and student. Recently, the models have been designed larger and more complicated, with remarkable performance improvements compared to the traditional models. As a result, an intuitive idea to improve efficient models is to distill them from a stronger teacher model. However, recent studies [29, 39] find that KD performance with a stronger teacher is poor and even worse than the normal teacher. TAKD [29] states that the student can only learn the knowledge effectively from a teacher model up to a fixed capacity, and proposes training multiple teaching assistants that have moderate capacities compared to the narrow student and a huge teacher. The teach assistants are trained sequentially with their former larger teacher assistants, and the final smallest teach assistant is used to train the student. DAKD [39] further improves TAKD by densely connecting all models (teacher, assistant teachers, and student) together and letting the student choose the optimal teacher for each sample. SFTN [30] proposes to learn the teacher model with the supervision of student model, which obtains a student-friendly teacher with smaller representation gap to the student. However, such methods suffer from complex distillation algorithms and heavy computational cost for training the teacher model, therefore, they are not applicable in practice. More recently, DIST [17] proposes an efficient and simple approach, which aims at relaxing the exact matching in previous KL divergence loss with a correlation-based loss, which performs better when the discrepancy between teacher and student is large. However, such heuristic loss in DIST only adapts to classification outputs, and thus for dense prediction tasks such as object detection, which requires detailed semantic information in intermediate features, the efficacy is limited. Moreover, the Pearson correlation in DIST only has shift and scale invariances, whereas for the complex situation of discrepancy in features, it cannot ignore all noises for a clean distillation.
As a result, we present DiffKD, which adapts well to multiple types of features and tasks, and handles better in eliminating the discrepancy in distillation features.
A.2 KD in dense prediction tasks
Different from the image classification task that only needs to recognize the overall classification of the image, object detection and semantic segmentation are referred as dense prediction tasks that have to predict the bounding boxes and classes for all the objects inside the image, or the pixel-level segments of image. As a result, effectively distilling the knowledge of the teacher in these tasks is more challenging than classification.
Various methods [3, 10, 20, 38, 48, 51] are proposed to improve KD performance in object detection. Chen et al. [3] first proposes performing KD on the classification logits and regressions of the RoI head. Mimicking [20] states that the feature maps in detection model contains richer semantic information than the responses, and proposes to distill the FPN [22] features of teacher. However, distilling from the features suffers from severe imbalance of the foreground and background pixels in object detection. To address this issue, recent methods focus on selecting valuable features and propose various loss functions based on this [10, 18, 38, 48, 52].
In semantic segmentation, knowledge distillation techniques typically prioritize maintaining the structural semantic connections between the teacher and student models. To address the inconsistency between teacher and student features, He et al. [12] employ a pretrained autoencoder to optimize feature similarity in a transferred latent space. They also transfer non-local pairwise affinity maps to minimize feature discrepancies. SKD [25] conducts pairwise distillation among pixels to preserve pixel relationships and adversarial distillation on score maps to distill holistic knowledge. IFVD [43] transfers intra-class feature variation from teacher to student to establish more robust relations with class-wise prototypes. CWD [38] proposes channel-wise distillation to better mimic spatial scores along the channel dimension. CIRKD [46] proposes intra-image and cross-image relational distillations to learn better semantic relations from the teacher.
However, existing state-of-the-art KD approaches in dense prediction tasks are designed specifically for the tasks, which are difficult to be applied to various tasks, resulting in a large experiment cost in evaluating and adapting those methods. As a result, an interesting direction in KD is to generalize and unify KD methods in different tasks.
B.1 Diffusion model
In all experiments, we use DDIM [40] as the noise scheduler in reverse denoising process. The total range of timesteps for training is 1000, and the initial time step for denoising is 500.
B.2 Image classification
On ImageNet classification task, we simply set  $\lambda_{1} = \lambda_{2} = \lambda_{3} = 1$  in all experiments. (1) Feature distillation. For experiments that use ResNet-50 or ResNet-34 as the teacher model, we set the number of latent channels in our autoencoder to 1024; while for Swin-L teacher, its feature for distillation has 1536 channels, so we compress it to 768 with autoencoder. (2) Logits distillation. We also add a DiffKD loss on the predicted classification logits. Different from the Bottleneck block used in feature distillation, we use MLP (two linear layers associated with an activation function) as an alternate of Bottleneck since the logits have only two dimensions (no spatial dimensions). Besides, the autoencoder is not used in logits distillation. On CIFAR-100 dataset, we also implement DiffKD on the output feature before average pooling layer and classification logits, but remove the autoencoder since the computation cost on CIFAR model is small.
B.3 Object detection
For object detection task, we conduct feature distillation on the predicted feature maps, and train the student with our DiffKD loss  $\mathcal{L}_{\mathrm{diffkd}}$ , regression KD loss, and task loss. Note that we do not use linear autoencoder in DiffKD since the number of channels in FPN is only 256. We set  $\lambda_{1} = \lambda_{3} = 1$ . We adopt ImageNet pre-trained backbones during training following previous KD works [18]. All the models are trained with the official strategies (SGD, weight decay of 1e-4) of  $2\times$  schedule in MMDetection [4]. We run all the models on 8 V100 GPUs.
Loss weights: We set DiffKD loss weight to 5 and regression loss weight to 1 on Faster RCNN students. For other detection frameworks, we simply adjust the loss weight of DiffKD to keep a similar amount of loss value as Faster RCNN. Concretely, the loss weights of  $\mathcal{L}_{\mathrm{DiffKD}}$  on RetinaNet, FCOS, and RepPoints are 5, 5, and 15, respectively.
B.4 Semantic segmentation
Following CIRKD [46] and MasKD [18], we train the models with standard data augmentations including random flipping, random scaling in the range of  $[0.5, 2]$ , and a crop with size  $512 \times 1024$ . An SGD optimizer with momentum 0.9 is adopted, and the learning rate is annealed using a polynomial scheduler with an initial value of 0.02. For DiffKD, we distill the knowledge of the segmentation predictions of teacher. Since these predictions are probabilistic distributions, we use DIST [17] as the distance function  $d$  in KD loss. We do not involve autoencoder since the segmentation prediction has only 19 channels on Cityscapes dataset, and set  $\lambda_1 = \lambda_3 = 1$ .
C.1 Results on CIFAR-100 dataset
We summarize the CIFAR-100 results in Tab. 9. Our DiffKD surpasses previous methods in most cases. Moreover, comparing homogeneous architecture settings and heterogeneous architecture settings, DiffKD gains more significant improvements on heterogeneous settings compared to the standalone training results, which indicates that our method can deal better with the discrepancy between teacher and student models.
C.2 Effect of efficient diffusion model
We train RetinaNet R50 student with R101 teacher with  $1 \times$  schedule in COCO, and compare our Efficient DM with original UNet in DDPM using DiffKD. As shown in Tab. 10, with feature shape (256, 80, 124), the original UNet has much larger parameters and GFLOPs, and thus leads to  $3 \times$
Table 9: Evaluation results on CIFAR-100 dataset. The upper and lower models denote teacher and student, respectively.
Method	Homogeneous architecture style	Heterogeneous architecture style
WRN-40-2	ResNet-56	ResNet-32x4	ResNet-50	ResNet-32x4	ResNet-32x4
WRN-40-1	ResNet-20	ResNet-8x4	MobileNetV2	ShuffleNetV1	ShuffleNetV2
Teacher	75.61	72.34	79.42	79.34	79.42	79.42
Student	71.98	69.06	72.50	64.6	70.5	71.82
FitNet [36]	72.24±0.24	69.21±0.36	73.50±0.28	63.16±0.47	73.59±0.15	73.54±0.22
VID [1]	73.30±0.13	70.38±0.14	73.09±0.21	67.57±0.28	73.38±0.09	73.40±0.17
RKD [31]	72.22±0.20	69.61±0.06	71.90±0.11	64.43±0.42	72.28±0.39	73.21±0.28
PKT [32]	73.45±0.19	70.34±0.04	73.64±0.18	66.52±0.33	74.10±0.25	74.69±0.34
CRD [41]	74.14±0.22	71.16±0.17	75.51±0.18	69.11±0.28	75.11±0.32	75.65±0.10
KD [14]	73.54±0.20	70.66±0.24	73.33±0.25	67.35±0.32	74.07±0.19	74.45±0.27
DIST [17]	74.73±0.24	71.75±0.30	76.31±0.19	68.66±0.23	76.34±0.18	77.35±0.25
DiffKD	74.09±0.09	71.92±0.14	76.72±0.15	69.21±0.27	76.57±0.13	77.52±0.21
Table 10: Comparisons of different dimensions of linear autoencoder in DiffKD. We report the top-1 accuracy and FLOPs of diffusion models with MobileNetV1 student and ResNet-50 teacher on ImageNet.
w/o AE	128	256	512	1024	2048
Top-1 (%)	73.47	72.84	73.22	73.53	73.62	73.58
FLOPs (G)	1.41	0.04	0.09	0.21	0.58	1.82
training time. The original UNet only achieves similar performance as our Efficient DM, as the generation of features is easier than images, and a small model would suffice.
C.3 Effect of adaptive noise matching (ANM)
To validate the efficacy of ANM more thoroughly, we conducted further experiments in more model settings and training strategies, as summarized in Tab. 11. We can see that, when with stronger strategy and teacher, the improvement of ANM is more significant (1.2% improvement on MobileNetV2 compared to 0.3% and 0.5% improvements on MobileNetV1 and ResNet-18). One possible reason is that, when the augmentations and teacher become stronger, the noisy gaps between predicted features of teacher and student become more various, and therefore ANM is more effective in matching the noisy levels.
C.4 Statistics of the learned noise weight  $\gamma$
To analyze the effectiveness of ANM, we first show the distribution of noise weight  $\gamma$  in Fig. 5 (a) of the rebuttal PDF. Revealing that the student feature is noised with  $Z_T^{(stu)} = \gamma Z^{(stu)} + (1 - \gamma)\epsilon_T$ , the larger  $\gamma$  denotes smaller additional noise added.
We can see that a large amount of values is in the range of  $\gamma > 0.9$ , indicating that the student feature itself contains nonnegligible noises and only requires small noises to match the initial noise level, while there also exist some cleaner samples that require large noises.
We also plot the curves of average  $\gamma$  in each epoch during training. Fig. 5 (b) indicates that, at the beginning of training, the student feature contains more noises, so only small weights of noises should be added. When the model gets converged, the noise in student feature becomes smaller and  $\gamma$  goes smaller to match the noise level accordingly.
Table 11: Effects of adaptive noise matching (ANM) on various model settings.
Student	Teacher	Strategy	w/ ANM	w/o ANM
MobileNetV1	ResNet-50	B1	73.6	73.3
ResNet-18	ResNet-34	B1	72.2	71.7
MobileNetV2	ResNet-50-SB	B2	74.9	73.7