SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: (a) Negligible magnitude of inner correlation with the SACL MobileNet-V1  $(1.5\times -1\times)$  trained on ImageNet-1K. (b) Unstable training and the degraded performance caused by the  $L_{\mathrm{CD}}$  with inner correlation only. is important for contrastive learning and leads summarizes the linear evaluation accuracy of the imageNet-1K dataset. Starting from scratch with method achieves  $2\%$  higher accuracy compared In Section 4, we also compare the proposed ResNet-50 [20] without SACL. The superior optimization between cross-decorrelation  $(L_{\mathrm{SACL}})$
Combining the cross-distillation with SACL, the inner decorrelation between  $z$  and  $[\tilde{z}]$  plays the dominant role in the distillation loss  $\mathcal{L}_{\mathrm{CD}}$ , and we observe that it is also critical to the training stability and performance of SACL. As shown in Fig. 3(a), the magnitude of the inner correlation is negligible compared to the inner decorrelation, even with the asymmetric encoding of SACL. Furthermore, solely minimizing the infinitesimal inner correlation as the  $\mathcal{L}_{\mathrm{CD}}$  introduces instability and degraded performance to the training process, as shown in Fig. 3(b). As a result, we formulate the total loss as:
$$
\mathcal {L} = \alpha \mathcal {L} _ {\mathrm {S A C L}} + (1 - \alpha) \left(\lambda \sum_ {i} \sum_ {i \neq j} \left(C _ {i j} ^ {A \tilde {A}}\right) ^ {2} + \lambda \sum_ {i} \sum_ {i \neq j} \left(C _ {i j} ^ {B \tilde {B}}\right) ^ {2}\right) \tag {10}
$$
The detailed pseudocode of the proposed method is summarized in the Appendix A.1. We evaluate the impact of the weight  $\alpha$  in Section 4.4. We also investigate the conventional negative-logarithm distillation loss in Appendix A.2.
Table 2: ImageNet-1K test accuracy with linear evaluation protocol based on MobileNet-V3 [22] trained by different contrastive learning/distillation methods.
Method	Encoder	Linear Eval. (%)	Epochs	Pre-train	Teacher	Training FLOPs (e+17)
$SACL-XD (Ours)	Eff-B0 (1.5×-1×)	65.32 (+2.12)	200	X	-	24 (2.9×↓)
$SACL-XD (Ours)	Mob-V3 (1.5×-1×)	61.69 (+1.79)	200	X	-	15 (64.7×↓)
SACL-XD (Ours)	Mob-V1 (1.5×-1×)	59.34	200	X	-	19
XD only (Ours)	Mob-V3 (1×)	59.42	200	X	-	7.2
XD only (Ours)	Mob-V3 (1×)	57.16	100	X	-	3.6
XD only (Ours)	Mob-V1 (1×)	55.84	100	X	-	9.0
$SSL-Small [27]	Mob-V3 (1×)	48.70	200	2 epochs	-	19
$SSL-Small [27]	Eff-B0 (1×)	55.90	200	2 epochs	-	34
ReKD [36]	Mob-V3 (1×)	56.70	200	X	ResNet-50	67
ReKD [36]	Mob-V3 (1×)	59.60	200	X	ResNet-101	125
ReKD [36]	Eff-B0 (1×)	63.40	200	X	ResNet-50	70
OSS [10]	Eff-B0 (1×)	64.10	800+200	X	ResNet-50	67
*SEED [16]	Mob-V3 (1×)	55.20	800+200	✓	ResNet-50	512
*SEED [16]	Mob-V3 (1×)	59.90	800+200	✓	ResNet-101	971
*SEED [16]	Eff-B0 (1×)	61.30	800+200	✓	ResNet-50	516
†MoCo-V2 [8]	Mob-V3 (1×)	36.30	200	X	-	4.8
†MoCo-V2 [8]	Eff-B0 (1×)	42.20	200	X	-	8.5
*: SEED [16] uses a ResNet-50 teacher which is pre-trained by 800 epochs.
$\dagger$  : Baseline linear evaluation accuracy reported by SEED [16].
:We use ReKD [36] as the SOTA baseline of EfficientNetB0 [29] to report the accuracy improvements and computation reduction.
$\S$  :We use SEED [16] as the SOTA baseline of MobileNet-V3 [22] to report the accuracy improvements and computation reduction.
$\S$  :Weights are initialized based on SEED [16]-trained model.
In this section, we evaluate the performance of the proposed algorithm based on CNN encoders (MobileNet [23, 22], EfficientNet [29], ResNet [20]) and ViT [13] models on the ImageNet-1K and ImageNet-100 dataset. We also demonstrate the capability of the proposed method with tiny-sized ResNet on the small CIFAR dataset. We also evaluate the transferability of the lightweight model on both CIFAR classification and VOC object detection downstream tasks. We characterize the asymmetry of SACL with the style of “ $\mathrm{K} \times -1 \times$ ”, where “ $\mathrm{K}$ ” is the width of the wide host model that is employed to slice the  $1 \times$  out of it. All the models are directly trained from scratch. The detailed experiment setup and hyperparameter settings are summarized in the Appendix.
4.1 Training from Scratch with SACL+XD on Lightweight CNNs
We follow the linear evaluation protocol on ImageNet to evaluate the performance of the backbone trained by the proposed SACL and cross-distillation (XD) algorithm. We train the compact models from scratch for 100 or 200 epochs, which is the same amount of fine-tuning effort as SEED [16] and ReKD [36]. The proposed algorithm is evaluated on multiple lightweight encoders, including MobileNet-V1 [23], MobileNet-V3-Large [22], and EfficientNet-B0 [29]. Table 2 compares the top-1 linear evaluation accuracy of our work against recent SoTA works for compact model training.
With the same 200 epochs, the XD-trained MobileNet-V3  $(1\times)$  model outperforms the recent ResNet-50-aided ReKD [36] by a noticeable  $2.72\%$  accuracy improvements (59.42% vs. 56.70%) with  $9.3\times$  less training FLOPs. Furthermore, XD achieves 57.16% ImageNet accuracy with only 100 epochs training on MobileNet-V3, surpasses the ResNet-50-aided ReKD [36] and SEED [16] by 0.46% and  $1.96\%$  with  $18.6\times$  and  $142\times$  training cost reduction, respectively. Combined with the proposed asymmetrical slimmable contrastive learning (SACL), our method achieves  $2.12\%$  and  $2.09\%$  linear evaluation accuracy improvements compared to the ReKD-trained EfficientNetB0 and MobileNet-V3, respectively. Meanwhile, our method eliminates the ResNet-101 teacher from training, which leads to  $5.2\times$  and  $8.3\times$  training effort reduction compared to ReKD [36]. We are aware the wider model introduces a higher computation budget, so we choose the  $1.5\times -1\times$  asymmetrical architecture, and the linear evaluation is performed on the slimmed encoder with the same size as the vanilla  $1\times$  model.
4.2 Training from Scratch with XD on ResNet and ViT
Besides the evaluation against the lightweight models, we validate the proposed XD individually by training ResNet-50 on ImageNet-1K with 300 epochs from scratch, as reported in Table 3.
Table 3: ImageNet-1K test accuracy with linear evaluation protocol based on ResNet-50 encoder.
Method	Training Epochs	Top-1 Linear Evaluation Accuracy (%)
MoCo [19]	1000	60.6
SimCLR [7]	1000	69.3
*BYOL [18]	300	68.4
*Barlow Twins [35]	300	70.7
XD (Ours)	300	71.1
*: Reported results from [18, 35] with 300 epochs training from scratch.
Compared to the recent contrastive learning methods [35, 18], the proposed cross-distillation algorithm (XD) achieves better accuracy with  $0.4\%$  linear evaluation accuracy improvements demonstrating the generality and versatility of the proposed XD algorithm.
Besides the CNN-based encoder, the proposed cross-distillation (XD) algorithm is also capable of training the lightweight ViT encoder. Table 4 summarizes the performance of the ViT-Tiny-16-224 [13] encoder trained by XD. Compared to Barlow Twins [9] and DINO [9], our method achieves  $1.36\%$  and  $0.88\%$  accuracy improvements. The model is directly trained from scratch, and the  $\alpha$  value of the XD is set to 0.8. Specifically, we simply replace the CNN encoder by the ViT-Tiny model and slim down the embedding dimensionality, no additional architecture has been introduced to ViT training. The superior performance on ViT indicates the high versatility of the proposed method.
Table 4: ImageNet-100 test accuracy with linear evaluation protocol based on ViT-Tiny [13] encoder.
Methods	Encoder	Training Epochs	Linear Eval Acc. (%)
Barlow Twins [35]	ViT-Tiny [13]	400	62.56
*DINO [9]	(# of Param = 5.5 Million)	400	63.04
XD (Ours)	400	63.92 (+0.88)
*: Reported DINO results from [12].
4.3 SACL-XD on CIFAR datasets with Tiny-sized ResNet
Table 5: CIFAR-10 linear evaluation test accuracy based on ResNet-20 trained by SACL+XD with different asymmetrical architectures.
Method	Encoder	Linear Eval Acc. (1 × model)	Teacher	Teacher Pre-trained by	Training FLOPs (e+16)
SACL+XD (Ours)	ResNet-20 (6×-1×)	86.81 (+7.18)	-	-	8.60
SACL+XD (Ours)	ResNet-20 (4×-1×)	84.04 (+4.41)	-	-	3.90
SACL+XD (Ours)	ResNet-20 (2×-1×)	82.31 (+2.68)	-	-	0.98
*SEED [14]	ResNet-20 (1×)	82.86	ResNet-20 (6×)	MoCo [7]	180
*SEED [14]	ResNet-20 (1×)	81.36	ResNet-20 (6×)	Barlow Twins [31]	180
Barlow Twins [31]	ResNet-20 (1×)	79.63	-	-	0.25
VICReg [3]	ResNet-20 (1×)	79.13	-	-	0.25
*: Re-implementation of SEED [16] on CIFAR dataset with the official code. The ResNet-20 (6×) is pretrained by 800 epochs.
Data augmentation and hyperparameter settings are adopted from [12]. We use [35] as the baseline of ResNet-20.
We also evaluate the performance of our method on the small-sized dataset with the tiny-sized ResNet encoders (e.g., ResNet-20 with 0.27 million parameters). We follow the data augmentation setup in [12] for the CIFAR-10 dataset. For the SACL, we sweep the asymmetry from  $2 \times -1 \times$  up to  $6 \times -1 \times$ , the model is trained for 1000 epochs from scratch, and the results are summarized in Table 5.
With the “ $6 \times -1 \times$ ” asymmetry, the equivalent weight sparsity is consistently held at  $97.01\%$  throughout the entire process. The linear evaluation is performed based on the slimmed  $1 \times$  model with 0.27 million non-zero weights. Our method achieves up to  $7.18\%$  accuracy improvements compared to the Barlow Twins baseline [35]. Compared to SEED [16], the proposed method achieves  $3.95\%$  accuracy improvements. We are aware the contrastive learning-trained large-sized ResNet-50 encoder can achieve  $>96\%$  CIFAR-10 accuracy with downstream fine-tuning [18, 35], but the recent CL algorithms exhibit poor performance with tiny encoder and small-sized training samples (compared to ImageNet-1K). The superior performance of the proposed method provides valuable insights for practical self-supervised learning on limited-sized datasets with tiny-sized encoders.
Figure 4: MobileNet-V1 ImagNet-1K accuracy vs. (a) value of  $\alpha$  and (b) training epochs.
Table 6: Transfer fine-tuning of MobileNet [23, 22] pretrained by the proposed method.
Method	Encoder	CIFAR-10	CIFAR-100	Aircraft	Flowers	Food-101	Cars	Pets
Supervised (from scratch)	Mob-V3 (1×)	92.97	73.69	65.37	79.89	60.30	68.18	70.97
Supervised (fine-tune)	Mob-V3 (1×)	94.53	78.86	68.29	89.94	75.84	82.43	85.87
XD (Ours)	Mob-V3 (1×)	94.80	79.00	71.39	90.05	75.71	82.77	89.42
SACL + XD (Ours)	Mob-V1 (1.5×-1×)	94.92	79.64	72.21	90.48	76.12	83.14	90.24
SEED [16]	Eff-B0 (w. RN-50 teacher)	87.5	63.0	-	-	-	-	-
Table 7: Comparison between the proposed method and other supervised high-water marks
Model	Training Method	CIFAR-10 Acc (%)	CIFAR-100 Acc (%)	# of (remained) Param. (M)
ResNet-50	Supervised Learning	94.75	78.23	25.6
ResNet-50	Supervised + GraNet [24]	94.64	77.89	2.6 (90% sparsity)
ResNet-50	Supervised + RigL [15]	94.45	76.50	2.6 (90% sparsity)
Mob-V1	SACL+XD (Ours) + Finetune	94.92	79.64	3.2
Mob-V3	XD (Ours) + Finetune	94.80	79.00	3.0
The impact of  $\alpha$ . We introduced the weight parameter  $\alpha$  in Eq. 6 and Eq. 10 as a tunable parameter to control the importance of the inner decorrelation loss during the training process. We further evaluate the impact of the different weighting between  $\mathcal{L}_{\mathrm{SACL}}$  and  $\mathcal{L}_{\mathrm{CD}}$ . We explore the impact of  $\alpha$  on the ImageNet-1K dataset with the MobileNet-V1 ( $1\times$ ) model. The model is trained by XD only with 100 epochs from scratch. As shown in Fig. 4(a), the proposed method achieves the best performance when  $\alpha = 0.9$ . Meanwhile, the accuracy oscillation caused by  $\alpha$  is relatively stable (within  $\pm 1\%$ ).
The impact of training effort. We also evaluate the performance of the proposed algorithm with different training efforts from scratch. Fig. 4(b) demonstrate the linear evaluation accuracy of MobileNet-V1 [23] on ImageNet-1K trained by different epochs. For both SACL+XD and individual XD training, the extended training effort from 100 epochs to 300 epochs leads to evident accuracy improvements. With 300 epochs of training, the proposed XD and SACL+XD method achieves  $58.25\%$  and  $60.63\%$  Top-1 linear evaluation accuracy on ImageNet-1K.
4.5 Transfer learning to downstream tasks
We report the transfer learning performance of the MobileNet [22, 23] encoder trained by both SACL+XD and XD. For the downstream classification, we use CIFAR-10 and CIFAR-100 as our target tasks. We also validate the pre-trained lightweight encoder on the VOC2007 dataset for downstream object detection. Follow the setup in [14], we finetune the models for 10,000 steps with SGD and batch size of 64. The experimental setup is summarized in the Appendix. Table 6 summarizes the transfer learning performance of the proposed method compared to SEED [16]. For the CIFAR tasks, our method achieves  $7.42\%$  and  $16.30\%$  accuracy improvements compared to SEED [16]. Together with the  $1.83\%$  and  $5.69\%$  improvements compared to supervised learning.
4.6 Comparison to SoTA energy-efficient supervised learning
In addition to the accuracy and training cost improvement compared to distillation-based contrastive learning, the powerful lightweight backbone model trained by the proposed method reveals a new perspective of energy-efficient inference compared to conventional supervised sparse training on
ResNet [24, 15]. Table 7 summarizes the CIFAR downstream comparison between the pre-trained MobileNet [29, 23] and the ResNet-50 sparsified by the recent SoTA pruning methods [24, 15]
Despite the additional fine-tuning effort, the powerful lightweight backbone pre-trained by the proposed method achieves better accuracy-model size tradeoff compared to the conventional supervised sparse learning with high element-wise sparsity. More importantly, the powerful lightweight architecture can be accelerated and deployed to the energy-constrained hardware without the requirement of the dedicated accelerator.
In this paper, we propose a novel contrastive learning (CL) algorithm designed for lightweight encoders. We first introduce the slimmed asymmetrical contrastive learning (SACL), which treats the lightweight model CL as a slimmed sparse training task with asymmetrical encoding. On top of the SACL, we propose the cross-distillation (XD) algorithm, distilling the knowledge by minimizing the decorrelation between the embeddings encoded by SACL. Compared to previous works, the proposed algorithm achieves new SOTA accuracy without introducing the large-sized teacher or expensive pre-training. Furthermore, solely training both lightweight and large-sized ResNet with the XD can still achieve superior performance. Compared to supervised learning, the ImageNet-1K-trained lightweight encoder shows superior performance in the downstream tasks with transfer learning.
Impact and limitations. In this work, SACL+XD works well with correlation-based loss [35] along with the shared encoder. Exploring the possibility of the proposed algorithm with the momentum-based encoder updating scheme [18, 19] could be an interesting direction. With the powerful, lightweight backbone trained by the proposed algorithm, we will further investigate the post-training quantization and hardware deployment, which further unleash the superior performance in Table 6 with practical downstream applications with actual hardware. To that end, our work can further improve the versatility of contrastive learning with energy-efficient applications (e.g., AI on edge).
Acknowledgments. We thank the anonymous reviewers for their comments and suggestions. This work is supported in part by the National Science Foundation under Grant No. 2144751, 2314591, 2328803, 2342726, and CoCoSys Center in JUMP 2.0, an SRC program sponsored by DARPA.
[32] Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. Clusterfit: Improving generalization of visual representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6509-6518, 2020.
[33] Yang You, Igor Gitman, and Boris Ginsburg. Large Batch Training of Convolutional Networks. arXiv preprint arXiv:1708.03888, 2017.
[34] Sukmin Yun, Hankook Lee, Jaehyung Kim, and Jinwoo Shin. Patch-level representation learning for self-supervised vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8354-8363, 2022.
[35] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow Twins: Self-supervised Learning via Redundancy Reduction. In International Conference on Machine Learning (ICML), pages 12310-12320, 2021.
[36] Kai Zheng, Yuanjiang Wang, and Ye Yuan. Boosting contrastive learning with relation knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022.
[37] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.
Algorithm 1: PyTorch-style pseudocode for the proposed algorithm
# f: encoder model
# h: projector head
# s: slim ratio of SACL
# slicer: SACL slicer
# alpha: weight between CL loss and CD loss
# lambda: weight on the off-diagonal terms
def normalize(z):
    z_norm = (z - z.mean(dim=0)) / z.std(dim=0)
    return z_norm
for batch in loader:
    x_a, x_b = batch
    # SACL forward pass
    slicer.remove_mask()
    z1 = h(f(x_a))
    sliceractivate_mask()
    z2 = h(f(x_b))
    # reverse the order of input
    with torch.no_grad():
        slicer.remove_mask()
        z1t = h(f(x_b))
        sliceractivate_mask()
        z2t = h(f(x_a))
    # cross correlation
    cab = mm(normalize(z1).T, normalize(z2)) / N
    caat = mm(normalize(z1).T, normalize(z1t)) / N
    cbbt = mm(normalize(z2).T, normalize(z2t)) / N
# Contrastive leaning loss
cl_loss = bt_loss(cab)
# CD loss
dcorr_a = off_diagonal(caat).mul_(lambda).sum()
dcorr_b = off(diagonal(caat).mul_(lambda).sum())
cd_loss = (dcorr_a + dcorr_b) / 2
loss = cl_loss.mul(alpha) + cd_loss.mul(1-alpha)
loss.backup()
optimizer step()
A.2 Compared to the Log-based distillation loss
From the perspective of knowledge distillation, the negative logarithm-based distillation loss has been widely incorporated into the "teacher-student" learning. In Section 3.2, we proposed the cross-distillation (XD) learning scheme. The distillation objective in Eq (10) is the inner decorrelation minimization between embeddings  $z$  and  $[\tilde{z}]$ . In addition to the correlation-based distillation loss, we also investigate the negative logarithm (e.g.,  $-a\log b$ ) distillation loss that is employed in both supervised knowledge distillation [21] and contrastive learning [6].
To avoid the unbalanced loss magnitude, the distillation loss is introduced as the regularization term controlled by the penalty level  $\gamma$ :
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {S A C L}} \left(z _ {A}, z _ {B}\right) + \gamma \mathcal {L} _ {C D} \tag {11}
$$
$$
\mathcal {L} _ {C D} = \left(- \left[ \tilde {z} _ {A} \right] \log z _ {A} + - \left[ \tilde {z} _ {B} \right] \log z _ {B}\right) / 2 \tag {12}
$$
We empirically observe that the negative logarithm-based distillation loss failed to outperform the proposed cross-distillation loss  $\mathcal{L}_{CD}$  with inner-decorrelation minimization. As shown in the ImageNet-100 results below:
Method	Encoder	# of Params (M)	Linear Eval Acc. (%)
XD	MobileNet-V1 (1×)	3.2	80.30
XD (w/ negative log)	MobileNet-V1 (1×)	3.2	79.63*
Barlow Twins [35]	MobileNet-V1 (1×)	3.2	78.40
*: Best accuracy we found with  $\gamma = 1\mathrm{e - }3$
Although the negative-logarithm distillation loss is suboptimal compared to the inner decorrelation minimization, the proposed cross-distillation learning scheme is beneficial to lightweight contrastive learning, compared to the baseline [35].
A.3 Detailed Experimental Setup of Pre-training
ImageNet-1K The encoders (MobileNet, EfficientNet, ResNet-50) are trained on ImageNet-1K with 100/200/300 epochs from scratch with the proposed method. We set the batch to 256 with a learning rate  $= 0.8$ . We employ the LARS optimizer with weight decay set to 1.5e-6. We set the correlation weights  $\lambda$  to 0.005. The hidden layer dimension of the projector is 4096. The detailed data augmentation is summarized in Table 8
Table 8: Detailed image augmentation settings on ImageNet-1K.
Parameter	XA	XB
Random crop size	224 × 224	224 × 224
Horizontal flip probability	0.5	0.5
Color jitter probability	0.8	0.8
Brightness adjustment probability	0.4	0.4
Contrast adjustment probability	0.4	0.4
Saturation adjustment probability	0.2	0.2
Hue adjustment probability	0.1	0.1
Gaussian blurring probability	1.0	0.1
Solarization probability	0.0	0.2
ImageNet-100 With the proposed cross-distillation method, we train the lightweight ViT model on the ImageNet-100 dataset for 400 epochs. The batch size is set to 256 with AdamW optimizer. The learning rate and weight decay are set to 0.005 and 1e-4. The detailed data augmentation is summarized in Table 9:
Table 9: Detailed image augmentation settings on ImageNet-100.
Parameter	XA	XB
Random crop size	224 × 224	224 × 224
Horizontal flip probability	0.5	0.5
Color jitter probability	0.8	0.8
Brightness adjustment probability	0.4	0.4
Contrast adjustment probability	0.4	0.4
Saturation adjustment probability	0.0	0.2
Hue adjustment probability	0.1	0.1
Gaussian blurring probability	1.0	0.1
Solarization probability	0.0	0.2
CIFAR-10 The proposed method is trained from scratch by 1,000 epochs with LARS-SGD optimizer [33]. We use 256 batch size along with 0.3 learning rate and  $1e - 4$  weight decay. The Cosine learning rate scheduler is used with 10 epochs of warmup training. The detailed data augmentation is summarized in Table 10.
Table 10: Detailed image augmentation settings on CIFAR-10.
Parameter	XA	XB
Random crop size	32 × 32	32 × 32
Horizontal flip probability	0.5	0.5
Color jitter probability	0.8	0.8
Brightness adjustment probability	0.4	0.4
Contrast adjustment probability	0.4	0.4
Saturation adjustment probability	0.2	0.2
Hue adjustment probability	0.1	0.1
Gaussian blurring probability	0.0	0.0
Solarization probability	0.0	0.2
A.4 Detailed Experimental Setup of Downstream Fine-tuning
We evaluate the transferability of the pre-trained lightweight model on downstream tasks, including CIFAR-10, CIFAR-100, and VOC2007. Following the settings in [14], we fine-tuned the models for 10,000 steps with SGD and batch size of 64. The learning rate is set to 0.1 with no weight decay. The input samples are resized to  $224 \times 224$  to maintain the dimensionality as the pre-trained model. The checkpoint of the pre-trained lightweight model will be released soon.
A.5 Detailed Training Cost of the Proposed Method
In addition to the FLOPs comparison in Table 2, we evaluate the training cost of the proposed method based on the actual training time per epoch, together with the total activation count (feature map pixels) for each forward pass of the training process, as shown in Table 11 and Table 12, respectively.
Table 11: Training time comparison between the proposed method and the distillation-based CL
Model	Training Method	Teacher	Training time / epoch	GPU Type	Batch Size
SEED [16]	MobileNet-V3	ResNet-50	35 min 20 sec	A100 (80G)	256
SACL-XD (Ours)	MobileNet-V3 1.5×-1×	N/A	26 min 02 sec	A100 (80G)	256
XD Only (Ours)	MobileNet-V3	N/A	16 min 15 sec	A100 (80G)	256
Table 12: Activation count comparison between the proposed method and the distillation-based CL
Method	Encoder	Teacher	Act. Count (E+07)	ImageNet-1K Accuracy (%)
SACL+XD (Ours)	Eff-B0 (1.5×-1×)	N/A	1.54	65.32
XD Only (Ours)	Mob-V3 (1×)	N/A	0.90	59.34
SSL-Small [27]	Mob-V3	N/A	0.90	48.70
SSL-Small [27]	Eff-B0	N/A	0.68	55.90
ReKD [36]	Mob-V3	ResNet-101	2.03	59.60
ReKD [36]	Mob-V3	ResNet-50	1.53	56.70
ReKD [36]	Eff-B0	ResNet-50	1.75	63.40
SEED [16]	Mob-V3	ResNet-50	1.52	55.20
SEED [16]	Eff-B0	ResNet-101	2.26	61.30