Slimmed Asymmetrical Contrastive Learning and Cross Distillation for Lightweight Model Training
Authors: Jian Meng
Li Yang
Kyungmin Lee
Jinwoo Shin
Deliang Fan
Jae-sun Seo
Venue: NeurIPS 2023 poster (2023)
Keywords: Contrastive Learning
Self-supervised Learning
Energy-efficient contrastive learning
TL;DR: A novel contrastive learning algorithm designed for lightweight model training from scratch.

Abstract: Contrastive learning (CL) has been widely investigated with various learning mechanisms and achieves strong capability in learning representations of data in a self-supervised manner using unlabeled data. A common fashion of contrastive learning on this line is employing mega-sized encoders to achieve comparable performance as the supervised learning counterpart. Despite the success of the labelless training, current contrastive learning algorithms *failed* to achieve good performance with lightweight (compact) models, e.g., MobileNet, while the requirements of the heavy encoders impede the energy-efficient computation, especially for resource-constrained AI applications. Motivated by this, we propose a new self-supervised CL scheme, named SACL-XD, consisting of two technical components, **S**limmed **A**symmetrical **C**ontrastive **L**earning (SACL) and **Cross**-**D**istillation (XD), which collectively enable efficient CL with compact models. While relevant prior works employed a strong pre-trained model as the teacher of unsupervised knowledge distillation to a lightweight encoder, our proposed method trains CL models from scratch and outperforms them even without such an expensive requirement. Compared to the SoTA lightweight CL training (distillation) algorithms, SACL-XD achieves 1.79% ImageNet-1K accuracy improvement on MobileNet-V3 with 64$\times$ training FLOPs reduction.

Slimmed Asymmetrical Contrastive Learning and Cross Distillation for Lightweight Model Training
Jian Meng Li Yang/ Kyungmin Lee Jinwoo Shin Deliang Fan and Jae-sun Seq [University of North Carolina at Charlotte, USA
Cornell Tech, USA
|KAIST, South Korea John Hopkins University, USA
(kyungmnee,, ) xuf 1yang508unce eddff
dfan100jhu. eduf {jm2787, js3528) edu
Abstract
Contrastive learning (CL. has been widely investigated with various learning mech- anisms and achieves strong capability learning representations of data in a sele-suue manner using unlabeled data. A common fashion of contrastive learning on this line is employing large-sized encoders to achieve comparable performance as the supervised learning counterpart. Despite the success of the labelless training, current contrastive learning algorithms failed to achieve good performance with lightweight (compact) models, e.g., MobileNet, while the re- quirements of the heavy encoders impede the energy-efficies computation, espe- cially for resource-conen constraiied AI applications. Motivated by this, propose new self-supervised CL scheme, named SACL-XD, consisting of two technical components, Slimmed Asymmetrical Contrastive Learning (SACL) and Cross- Distillation (XD), which collectively enable efficient CL with compact models. While relevant prior works employed strong pre-trained model as the teacher of unsupervised knowledge distillation to a lightweight encoder, our proposed method trains CL models from scratch and outperforms them even without such an expensive requirement. Compared to the SoTA lightweight CL training (dis- tillation) algorithms, SACL-XD achieves 1.79% ImageNet-IK accuracy improve- ment on MobileNet- V3 with x traiming FLOPs reduction. Code is available at https:///ithur com//oongjaa00002/54400
Introduction
To overcome the labeling bottleneck for supervised training of deep neural networks (DNNs), self- supervised learning has been widely investigated to learn representations without intensive labeling. In particular, contrastive learning (CL) has demonstrated its capability of representation learning in various machine learning domains, e.g., image 7 34], video [31], language (2), speech [[], and
medical imaging (30]. The success of CL is built upon different data augmentations from the original (training) samples, and the representation is learned by maximizing the latent common knowledge between contrastive embeddings 8 7 35). which are separately encoded from the augmented images by DNN models [25). Despite the various contrastive learning techniques, learning the latent knowledge and representations requires wide and deep encoders. In particular, current SoTA CL. algorithms 3, [8] have to employ a large-sized encoder (e.g., ResNet-50) to achieve comparable performance as the supervised learning counterpart.
On the other hand, training the lightweight models [23 22 from scratch is largely under-explored in CL. Almost none of the prior works have reported the CL performance with directly-trained lightweight models. In fact, the prior success of the lightweight models in supervised learning cannot be smoothly transferred to CL. where the performance gap between the lightweight models and large-sized models is largely amplified under recent CL methods [ 19], For example, with
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

supervised learning, NebileNet-V3Largee (Mob-V33 V3) [22] can achieve 74.04% on ImageNet- 1 which is comparable to ResNet-50 (76.15%) However, the 22% accuracy difference is amplified to 530% (75.2% - 36.3%) in CI., as reported in [16] with MoCo-V2. Training MobileNet-W V3 with more recent Barlow Twins 35 improves the accuracy to 552 but this is still unsatisfactory.
To overcome the limited trainability of the lightweight DNN models in CL, con-
70
This work
SCAL+XD
ReKD (wl RN-101)
SEED (wl RN-1011
trastive lightweight model learning has
ImageNet Tap-1 Accuracy (%)
been investigated as an unsupervised knowledge distillation task. For exam-
40
SCAL+XD
RaKD
XD
SEED (WT RN-101)
XD (100 epochs)
ReKD RNN-50)
SEED RIN-50)
ple, SEED [16] divides the entire train-
ing process into a two-step process of (1)
40-
Barlow
64X Training Cost Reduction
teacher pre-training with CL. and (2) unsu- pervised knowledge distillation from the frozen teacher to the student lightweight encoder (c.g., MobileNet-V3. V3 (22]). In addi- tion to the two-step process of "pretraining- and-tuning", the authors employ different input schemes between two steps, which further elevates the complexity of the en- tire training process. On the other hand, ReKD [36] introduces the latent relation knowledge into online distillation with a
40-
Encoders (CNN backons only)
MobiloNes-VS-Lan (3.0 Million)
McCo-V22
EffictentNNe-BB 80 (4.0
30
1E18
1E19
1E20
Training (FLOPs)
Figure 1: ImageNet-IK accuracy of the proposed method (SACL+XD) vs. state-of-the-an art methods. "RN" represents the ResNet teacher of the prior works [6 36.. Following the settings in 36 [6]. the number of FLOPs is computed based on 200 epochs of training.
single-step training process. However, the large-sized encoder is still required as the online teacher.
To that end, both ReKD (36] and SEED [16] focus on unsupervised distillation with the large-sized teacher, which actually amplifies the training cost for lightweight CL, as shown in Fig. [ Meanwhile, the performance of training lightweight models with CL from scratch remains unsatisfacton (27] or requires dedicated input scaling design with low generality 23 on normal CL. This is quite meaning-
ful to explore since training a high-performance compact encoder (s 5 M parameters) [22 23 29
by necessitating large-sized ResNet (>20 M) (20] teacher largely degrades the "time-to-deplo of the model due to the magnified training cost. Given the challenges of CL and the limitations of prior works, we raise the following question as our motivation:
Is there a contrastive learning algorithm that can train the high-performance lightweight model without using large-sized teacher?
To answer this question, we propose Slimmed Asymmetrical Contrastive Learning (SACL) together with
(XD), a novel self-supervised contrastive learning algorithm designed for effi- cient and -egforerffomance CL with lightweight encoders. Specifically, SACL considers lightweight contrastive learning as asymmetrical sparse training problem based on the shared weights from dense host encoder. A lightweight encoder can be treated as a "subset" model (sub-model) sliced from a wide host model, which naturally formulates the asymmetrical encoding in CL. Different from the knowledge distillation-ban -based methods [16 36) the proposed SACL-XD algorithm completely eliminates the large-sized ResNet teacher from the lightweight contrastive learning. On top of that, we introduce cross- distiliistil (XD) to facilitate SACL. XD minimizes the decorrelation between the latent information distorted by asymmetrical CL. elevating the training stability and performance. As shown in Fig. [ the proposed method achieves 64x and 8.3x training cost reduction along with 1.79% and 2.09% ImageNet accuracy improvements, compared to SOTA [16] and (36], respectively. The major contributions of the proposed SACL-XD algorithm are:
1. Simplicity: SACL-XD does not require teacher pre-training (and input scaling) before training a lightweight student encoder.
2. Efficiency: SACL-XD does not require the large-sized teacher during training. The training cost of MobileNet-V. V3 22 model is reduced by 64x and 8.3 x compared to [36] 16).
3. Performance and generality: SACL-XD achieves up to 2.09% ImageNet- 1K accuracy improvement compared the previous SoTA method. In particular, the proposed cross- distillation (XD) algorithm solely achieves superior performance with normal contrastive learning settings on ResNet-50 without SACL.
4. Transferability: The SACI--XD-traine lightweight encoder shows high transferability to the downstream tasks. With minimum fine-tuning on top of the ImageNet- -eeee-KK-un K-trained

---

MobileNet- V3, ours achieves 94.80% accuracy on CIF outperforms the supervised learning baseline and SEED [66] with 1.83% and 14.80% improvements, respectively.
2 Related Work
Self-supervised contrastive learning. With the absence of deterministic labels, the major objective of contrastive learning is minimizing the distance between the embeddings separately encoded from the augmented input samples. Early research works [9] define "positive" and "negative" sample pairs, and the learning process maximizes the similarity between positives and repels the negative samples. The popular InfoNCE loss (26 19 8 or NT-Xentt loss 77] has been proposed the learning objective. The wide and deep DNN encoder is also the key factor of success in contrastive learning. In addition to the cross-reference between positive and negative pairs, contrastive learning is also considered a DNN-based clustering problem, where the samples and their embeddings are grouped into clusters based on similarity metrics 4 32 N. SwAV [5] introduces online cluster learning with the reduced complexity of computation. The entropy- based similarity matching and clustering CL algorithms shares the same nature, but they all require an extensive amount of negative samples to generate salient contrastiveness.
Recent methods consider contrastive learning as a knowledge distillation [2 task between the contrastive encoders with separately augmented inputs. BYOL [8] and SimSiam [9] consider the student network as the online model, while the teacher is consistently staying offline with no gradient propagation [9]. The teacher is updated as the moving averaged weights of the student encoder. The "student-teacher" relationship focuses on the distance minimization between latent information. Another perspective is exploring the content hidden inside the embeddings. Barlow Twins 335 pushes the corselcorrl between the encoded embeddings toward the identity matrix. VICReg 33 collectively optimizes the variance, covariance, and distance between and inside the embeddings achieve better training stability and even enables the CL with different encoders. Among all the previous CL. algorithms, the empirical findings show the improved training stability and performance of CL together with the relaxed requirement of batch sizes, but the large-sized encoder (e.g., ResNet-50 (1x, 2x....) (00 is a persistent and almost mandatory requirement to achieve high performance, which hinders the development of CL with high energy consumption and extensive training effort.
Contrastive learning with lightweight models. Under the context of supervised learning, various lightweight architecture designs [23 22 29] are proposed to minimize the performance difference caused by the model size gap. The intricate architecture and the compact model sizes enable energy- computer vision applications. However, the architectural efficiency of
efficient and hardware-friendee the lightweight models becomes invalid in the contrastive learning domain. Employing lightweight encoder in CL leads to poor performance due to the insufficient trainability caused by the limited model sizes. To resolve this challenge, the mainstream method is introducing a giant model as the teacher of the lightweight student encoder. SEED [6 uses pre-trained large-sized ResNet (20] as the frozen teacher of the lightweight student model. The similarity- -baed teacher distribution is generated as the target soft label for student learning. In particular, accurate distillation requires both teacher and student to encode the same input, which is inconsistent with the augmentation strategy of CL. As result, self-supervised training of the lightweight model becomes two-step process. However, the dependence on the large-sized teacher and the expensive pre-training complicates the overall learning process. ReKD (36] incorporates the online distillation during learning with the relation knowledge-bert based loss between embeddings. DisCo [17] complicates the training cost of the distillation even further. The inputs encoded separately by the teacher and two separate students. Regardless of using an online or frozen teacher, the large-sized ResNet model is always required to train lightweight encoder. The recent efforts on directly-trainet lightweight model requires fine-grained efforts on input scaling (28), but the improvements on lightweight model has low generality and has no improvements on the popular ResNet-50-basee CL.
3 Method
To resolve the training dilemma between contrastive learning and lightweight models, this section introduces 1) SACL: Slimmed Asymmetrical Contrastive Learning and 2) XD: Cross Distillation to achieve lightweight contrastive learning from scratch.
3

---

Scollti [Re]]
Expotta (all)
Fal
36
Agg
11
Pugi
Slimmed
Siimmee
Asymmetry
Asymmetry
Sharing
Shawing
Sharing
f
fo
X
X2
X
(b) Figure Overview of (a) contrastive learning with shared encoder, (b) Proposed Slimmed Contrastive Learning (SACL) algorithm and (c) Cross- Looos-Disiillltin-aadeee SACI.
(a)
(c)
method performs contrastive learning based on the dual augmentation that is transformed from the clean input image. Given a clean input batch X sampled from the dataset D, the distorted views generated from combination of data augmentations T. leading to the augmented pairs X a and XB. which are fed into the encoders for different contrastive paths. In this work, we follow the recent contrastive learning algorithms 35 and use one shared encoder fo for different contrastive branches, as shown in Fig. 2 a). The encoded outputs are fed into the subsequent projector hg. resulting in the latent embeddings 2 and øp.
Similar prior contrastive learning (CL) algorithms [35 3
3.1 Slimmed Asymmetric Contrastive Learning (SACL)
Under the standard supervised training. has been well evidenced that extracting a subset lightweight model out of wider/larger (e.g.. 2x) encoder often leads to better performance compared to directly train such a lightweight model from scratch [37]. We are inspired by the fact that a lightweight model can be considered asymmetrical relationship between the slimmed model and the original model naturally fits the independent encoding path of contrastive learning.
subset model "sliced" from the original full-sized host model. The
We propose Slimmed Asymmetric Contrastive Learning (SACL), for obtaining lightweight models under self-supervise CL. training. SACL treats the lightweight model contrastive learning sparse training optimization problem based on shared weights from a large-sized encoder. During the forward pass of each iteration, the augmented input pair (X X B) are separately encoded by the dense host model (fe) and the slimmed (fī) encoders with the disabled weight filters (i.e., output channels), shown in Fig. bb)
The host encoder model fe slimmed by removing a unified amount of weight filters (output channels) of each layer based on weight magnitude score. With the slimmed and dense model fä and fo, the relationship between 0 and is defined as:
0, C and 0, M.
(1)
Where M is the weight mask that disables the channels of fa. and the subset model f is selected when M is enabled. As shown in Fig. 2()). fo and Sī separately encode the input pair in different branches. With the shared weights, the mask M is alternatively enabled between branches to formulate the asymmetrical encoding during the forward propagation. The slimming ratio (sparsity) of M'S is defined by the Slimmed Asymmetry (SA) between fa and fä with the style of "Kx-1x" "K the width of the wide host model (fo) that is employed to slice the xx model (fz) of:
1 - 1/K card (0,) /card())
where card(-) returns the number of nonzero elements of the tensor, is the desired slimmed ratio (sparsity) controlled by K.
Starting from the initialization of the training, SACI. uniformly generates the masks to slice out the weight filters (output channels) with the least magnitude score from each layer. The masks will be

---

updated after each epoch to maintain the performance of the slimmed model based on the filter-wise L norm score. The resultant slimmed (Ix) model will be deployed as the final trained encoder. Specifically, SACI. drives contrastive learning with the following properties:
1. SACL holds consistent channel-wise architecture difference (e.g., 1.5x vs. 1x) between contrastive branches throughout the entire training process.
2. SACL removes unified amount of channels (with the lowest magnitude score) of all the layers. By doing so, the resultant slimmed model will have the exact width as the target lightweight model.
Formally, given the augmented input pair (XA, XB), the forward pass characterized as: ho(((())).
z (x(()(Xa)),
(3)
And the optimization target of SACL is:
card
min Lsaci (EA,cl(AA,,3 (zA, =n), 6,0,
such that 0. C and card(O)
(4)
Inside each mini-batch, the gradient is collectively computed all at once, and the optimizer will update the whole set parameter 0 of the dense host encoder fe- The slimmed model architecture (fj) will be updated after every epoch based on the magnitude score each filter. The model is completely online and stop gradient is excluded from learning. Minimizing the contrastive loss LSacI between 2 and z is equivalent to overcoming the distortion caused by 1) data augmentation T and 2) consistent and structural architecture difference between fg and fo. And the architecture asymmetry caused by 2) motivates us to explore the enhancement of SACL from the perspective of knowledge distillation, which is presented in the following section.
3.2 Cross Distillation (XD) on top of SACL
We propose Cross- Distillation (XD) on top of SACL, which treats the teacher-student relationship as an interconnected knowledge distillation with the correlation- basod optimization on top of the proposed SACL learning scheme. Given the asymmetrical contrastive encoders fo and fä. we first encode XA and X B based on SACL (X + fe; XB fā), leading to the embeddings 24 and 2B. Subsequently,
freeze both So and Sø while reversing the order of the inputs for encoding (XB + [So] - [ X - [5ğ] - [rA]), and characterize the resultant embeddings as [žx] and [ž where "1-1" represents the frozen encoder for the forward pass only.
As result, each forward pass will generate two pairs of latent vectors resulting from two groups of SACL forward pass. We first compute the online contrastive loss CSACI. based on Z and zn. Such online embeddings contain the distortion caused by both data augmentations and asymmetrical encoders. We empirically find out that directly optimizing such high sparssyysitt difference via single contrastive loss leads to collapsed training. Motivated by that, we compute the cross-din dost-iiitillation loss Leo as the average loss between the pair of (Z A and (2B, Zn):
C EA) + L (=n,
LeD
(5)
where LA. CB will be defined formally later (see Eq. 9 As shown in Fig. 2 each term of the cross-distillation loss is computed between the asymmetrical SACL. encoders with the same input. In other words, optimizing the cross -rossddistillation loss is equivalent to minimizing the distortion in embeddings caused by the asymmetrical sparsity only. We define the total loss of the training as the weighted sum between LSacE and Lop and weight is defined as a:
L aLsact + (1 CcD
(6)
Where n represents the index of batch, and i and represent the dimensionality indices across the latent output. Correspondingly, Ct,, represents the i, element the correlation matrix.
Regarding contrastive loss design, recent works treat contrastive learning as the optimization problem of principle correlation maximization (35) or the decorrelation minimization [35] of the encoded latent vectors. The cross-correlation between ZA and 2B is computed as:
Σn
A
Cij
Rn Rnj
v En (2A E (=
5

---

Table 1: ImageNet- 1k test accuracy with linear evaluation protocal based on MobileNet- V3 trained by different contrastive Dearningdiisiilation methods.
Method
Linear Eval. Acc. (%)
Training Epochs
Pre-training
XD. (Ours)
57.16
100
X
ReKD 36
56,70
200
X
SFRD 16
55,20
200
In this work, we first adopt the BT-loss [35] as the starting point of the online contrastive loss (EA where the principle correlation (diagonal) is maximized, and the cross-
decorrelation (off-diagonal) is minimized during training.
Laxc (=A, Σ
(1-C
CAB) + Σ Σ B)
(8)
14
Unlike the prior work that uses the SoftMax-based probability (16) unsupervised soft labels for dis- tillation, we find out the proposed cross-distillation task between z and E can still be considered a correlation-base based optimization problem, and our finding shows the proposed cross-distillation a strong facilitator for contrastive learning even without SACL, First, by duplicating LSACL for LCD. each term in Eq. becomes:
100
Disillation Lose [Lxa)
>100 X
12-4
12-6
Inner-Correlaioon
Inner-Decorelatton
(4)
C = E CA AA +A
ΣV
(9)
Training (Laae)
300
Inner Correlation
Inner Decorretation
550
200
inner correlation loes
inner-dooorelattin
By encoding the same input with the identical dense model, the inner correlation C 1, and inner de-correlation E,(1 - C:AA) -CAAA) + 0. which makes the cross-distillationn loss L and S equivalent to the internal decorrelation loss between different di- mensions inside 2A and 2B, respectively. As result, minimizing LcD avoids the aliasing feature across dif- ferent dimensions. We would like to highlight the fact that the combination Eq. TO with both cross- decorrelation (TsSact) and the inner decorrelation (Teo) is importánt for contrastive learning and leads to superior performance compared to 35]. Jummmiiismm the linear evaluation accuracy the lightweight MobileNettV3--Lnnee only 100 epochs, the proposed Cross-distillation (XD) method achieves 2% higher accuracy compared to [16), which uses ResNet-50 (1x) as the teacher. In Section 4 we also compare the proposed XD algorithm with other CL methods on VIT [3] and ResNet-50 20 without SACL. The superior performance of XD proves the importance of the co-optimization between cross- cross-eccorrelatoon (Leaci)
100
Epoch
(b)
Figure 3: (a) Negligible magnitude of in- ner correlation with the SACL MobileNet-Vi (1.5x-1x) trained on ImageNet-IK. (b) Un- stable training and the degraded performance caused by the Lep with inner correlation only.
(22 trained on the ImageNet- IK dataset. Starting from scratch with
and the inner decorrelation (Lev).
Combining the cross-distillation with SACL, the inner decorrelation between and [3] plays the dominant role in the distillation loss LCD, and we observe that it also critical to the training stability and performance of SACL. As shown in Fig. Ba), the magnitude of the inner correlation is negligible compared to the inner decorrelation, even with the asymmetric encoding of SACL. Furthermore, solely minimizing the infinitesimal inner correlation the Len introduces instability and degraded performance to the training process, as shown Fig. 3(b). As result, we formulate the total loss as:
L = aLacc + +11 (1-a)(4 ΣΣ 2 + +
(10)
The detailed pseudocode of the proposed method is summarized in the Appendix A. We evaluate the
impact of the weight a in Section 4.4 We also investigate the conventional negati Degative-lae weeiiveellppee wo thm distillation loss in Appendix A.2

---

Table 2: ImageNet- 1K test accuracy with linear evaluation protocal based on MobileNet- V3 trained by different contrastive learning/distilationn methods.
Method
Encoder
I Linear Eval. (%)
Epochs
Pre-train
Teacher
Training FLOPs (e+17)
$ACL-XD (Ours)
Elf BD (1.5x-1x)
65.32 (+2.12)
200
24 (2.9% J)
BACL-DD (Ours)
Mob (155-1)
61.69 (+1.79)
200
15 (64.7x
SACL-XD (Ours)
Mob-V1
59.34
200
19
XD only (Ours)
Mob-V3(1x)
59.42
200
7.2
XD only (Ours)
Mob-V3 (Ix)
57.16
100
XD only (Ours)
(Ix)VI
55.84
100
9.0
SL. Small
27
Mob- (Ix)
48.70
200
2 epochs
SL-SSmal
Eff-BO (1x)
55.90
200
2 epochs
34
ReKD 36
Mob V3 (1x)
56.70
200
ResNet:50
67
ReKD 36
Mob-V3 (I))
59.60
200
ResNett-10
125
ReKD 36
EfT-BO (1x)
63.40
200
ResNet-50
70
OSS [0
Eff-BO (1x)
64.10
800+200
ResNet-50
67
*SEED 16
Mob V3 (1x)
55.20
800+200
ResNet-50
512
*SEED 16
Mob-V3 (1.)
59.90
800+200
ResNet-101
971
'SEED 16
Eff-BO (1x)
61.30
800+200
ResNet-50
516
McCo-V2 g
Mob-V3 (Ix)
36.30
200
4.8
McCo-V2 g
Eff-BO (1x)
42.20
200
8.5
SEED [66
a ResNet-50 teacher which pre-trained by 800 epochs.
Baseline linear evaluation accuracy reported by SEED (16).
use ReKD Bol as SOTA baseline EfficientNe1B0 b0
to report accuracy improvements computation reduction. report the accuracy improvements and computation reduction.
We use SEED [6 as the SOTA baseline of MobileNet-V. V3
22
Weights are initialized based on SEED 116-trained model.
Experimental Results
In this section, we evaluate the performance the proposed algorithm based on CNN encoders (Mo- bileNet [23] 22]. EfficientNet (29). ResNet (20) and ViT [13] models on the ImageNet-1 and ImageNet- 100 dataset. We also demonstrate the capability of the proposed method with tiny-sized ResNet on the small CIFAR dataset. We also evaluate the transferability of the lightweight model on both CIFAR classification and VOC object detection downstream tasks. We characterize the asymmetry of SACL with the style of "Kx-1x", where "K " is the width of the wide host model that is employed to slice the 1 x out of it. All the models are directly trained from scratch. The detailed experiment setup and hyperparameter settings are summarized in the Appendix.
4.1 Training from Scratch with SACL+XD on Lightweight CNNs
We follow the linear evaluation protocol on ImageNet to evaluate the performance of the backbone trained by the proposed SACL and distillaain (XD) algorithm. We train the compact models from scratch for 100 or 200 epochs, which is the same amount of fine-tuning effort as SEED [16] and ReKD (3 The proposed algorithm is evaluated on multiple lightweight encoders, including MobileNet-V VI 23 MobileNet-V3-11
(22), and EfficientNet-Bo [29] Table 2 compares the top- linear evaluation accuracy of our work against recent SoTA works for compact model training.
With the same 200 epochs, the XD-trained MobileNet- (1 x) model outperforms the recent ResNet- 50-aided ReKD 33 by a noticeable 2.72% accuracy improvements (59.42% vs. 56.70%) with 9.3 x less training FLOPs. Furthermore, XD achieves 57.16% ImageNet accuracy with only 100 epochs training on MobileNet-Va V3, surpasses the ResNet-50-aider ReKD [36] and SEED [66] by 0.46% and 1.96% with 18.6x and 142x training cost reduction, respectively. Combined with the proposed asymmetrical slimmable contrastive learning (SACL), our method achieves 2.12% and 2.09% linear evaluation accuracy improvements compared to the ReKD-traines EfficientNetBO and MobileNet-Va. respectively. Meanwhile, our method eliminates the ResNet- 101 teacher from training, which leads to 5.2x and 8.3x training effort reduction compared to ReKD (36]. We are aware the wider model introduces higher computation budget, so we choose the 1 asymmetrical architecture, and the linear evaluation is performed on the slimmed encoder with the same size the vanilla Xx model.
7

---

4.2 Training from Scratch with XD on ResNet and VIT
Besides the evaluation against the lightweight models, we validate the proposed XD individually by training ResNet-50 on ImageNet- -K with 300 epochs from scratch, as reported in Table 3
Table 3: ImageNet- IK test accuracy with linear evaluation protocol based on ResNet-50 encoder.
Method
Training Epochs
Top-1 Linear Evaluation Accuracy (%)
MoCa
19
1000
60.6
SamCLR 7
1000
69.3
"BYOL. 18
300
68.4
Barlow
300
70.7
XD (Ours)
300
71.1
Reeported results from
B
B5] with 300 epochs training from scratch.
Compared to the recent contrastive learning methods [85 the proposed cross tross-distillation algo- rithm (XD) achieves better accuracy with 0.4% linear evaluation accuracy improvements demonstrat- ing the generality and versatility of the proposed XD algorithm.
Besides the CNN-based encoder, the proposed cross-distillation (XD) algorithm is also capable of training the lightweight VIT encoder. Table 4 summarizes the performance of the ViT-Tiny-16 224 [33 encoder trained by XD. Compared to Barlow Twins [9] and DINO [9] our method achieves 1.36% and 0.88% accuracy improvements. The model directly trained from scratch, and the a value of the XD is set to 0.8. Specifically, we simply replace the CNN encoder by the ViT-Tiny model and slim down the embedding dimensionality, no additional architecture has been introduced ViT training. The superior performance on ViT indicates the high versatility of the proposed method.
Table 4: ImageNet- 100 test accuracy with linear evaluation protocol based on ViT-Tiny
13 encoder.
Methods
Encoder
Training Epochs
Linear Eval Acc. (%)
Barlow Twins
35
400
62,56
ViT-Tiny [
"DINO
(It Param Million)
400
63.04
XD (Ours)
400
63,92 (+0.88)
Reported DINO results from
12
4.3 SACL-XD on CIFAR datasets with Tiny-sized ResNet
Table 5: CIFAR-IO linear evaluation test accuracy based on ResNet-20 trained by SACL+XD with
different asymmetrical architectures.
Method
Eucader
Lineer Eval Are. (1x model)
Towcher
Teaecbur Pre-trained
Training PL,07s (ool6))
SACL-XXD (Dur))
ResNas22 20 (B00))))
$6.81 (+7.18)
8,50
SSCT (Dur))
ReiNes (10)))))))
(e4.4)
3,90
SACE=ND (hhm))
Ro0Nes-301 H)
$2.3 (-2.00)
9196
'SEE [14]
ReeNas- 21 x]]
$2.95
Ro-Ner3 (6x))
MuCai [7]
160
[14]
20
$1,30
ResNes
Bulow Toiss [31)
Bation Twins
ResNoe-20-1
VICReg [9]
RccNas-20 x]]
70.13
Re- Re-implenenizaaud of SEED TT] oz CIEAR dutset will the official code. The RegNet 20 (6x0 pretuiied 900 specit.
Das aigrematton and peppeemeeee selinns are aiopial from [+] We and [t the bradine RooNe-21
We also evaluate the performance of our method on the small-sized dataset with the tiny-sized ResNet encoders (e.g.. ResNet-20 with 0.27 million parameters). We follow the data augmentation setup in [12] for the CIFAR-10 dataset. For the SACL,, we sweep the asymmetry from 2x-1 x up to 6x-1 x, the model is trained for 1000 epochs from scratch, and the results are summarized in Table
With the "6x-1x" asymmetry, the equivalent weight sparsity is consistently held at 97.01% through- out the entire process. The linear evaluation performed based on the slimmed 1x model with 0.27 million non-zero weights. Our method achieves up to 7.18% accuracy improvements compared to the Barlow Twins baseline (35). Compared to SEED [6]. the proposed method achieves 3.9 3.95% accuracy improvements. We are aware the contrastive carming-trainee large-sized ResNet-50 encoder can achieve >966 CIFAR-10 10 accuracy with downstream fine-tuning [88 35), but the recent CL algorithms exhibit poor performance with tiny encoder and small-sized training samples (compared to ImageNet-1K), The superior performance of the proposed method provides valuable insights for practical self-supervised learning limited-sized datasets with tiny-sized encoders.
8

---

Imagetee
Linear Evall Acc. om Imagelees
22.0
SACL+XD
23.4
Linear
0:33
100
Epoch
200
250
300
Figure 4: MobileNet-Vi ImagNet-1K IK accuracy vs. (a) value of a and (b) training epochs.
Table 6: Transfer fine-tuning of MobileNet [23 22
pretrained by the proposed method.
Method
Encoder
CIFAR-10
CIFAR-100
Aircraft
Flowers
Food-101
Cars
Pets
Supervised (from scratch)
Mob-V3
92.97
73.69
65.37
79.89
60.30
68.18
70.97
Supervised (fine-tune)
Mob-V3
94.53
78.36
68.29
89.94
75.84
82.43
85.87
(Ixx
SACL XD (Ours)
Mob-VI (1.5x-1x)
94.92
79.64
72.21
90.48
76.12
83.14
90.24
teacher)
Table 7: Comparison between the proposed method and other supervised high-wate marks
Model
Training Medhod
CIFAR-10 Ace (%)
CIFAR-100 Are (%)
of (remained) Param. (M)
ResNes-50
Supervised
78.25
RosNes-50
Supervised
24
2.6 (90% sparsity)
ResNet-30
Supervised + Rigl.
2.6 (905 sparsity)
Mob-V1
SACL-XD (Ours) Finelune
94.92
79.64
3.2
Mob-V3
XD (Ours) Finetune
94,80
79.00
3.0
4.4
Ablation Study
The impact of a. We introduced the weight parameter in Eq. 6 and Eq. I0 as a tunable parameter to control the importance of the inner decorrelation loss during the training process. We further evaluate the impact of the different weighting between LSACL and Len. We explore the impact of on ImageNet- IK dataset with the MobileNet-V V1 (1 x)) model. The model trained by XD only with 100 epochs from scratch. As shown in Fig. 4(a), the proposed method achieves the best performance when a=0.9. Meanwhile, the accuracy oscillation caused by a relatively stable (within + 1%).
The impact of training effort. We also evaluate the performance of the proposed algorithm with different training efforts from scratch. Fig. b)) demonstrate the linear evaluation accuracy of MobileNet-Vi V1 23 on ImageNet- IK trained by different epochs. For both SACL+XD and individual XD training, the extended training effort from 100 epochs to 300 epochs leads evident accuracy improvements. With 300 epochs of training, the proposed XD and SACL+XD method achieves 58.25% and 60.63% Top-1 linear evaluation accuracy on ImageNet- 1.
4.5 Transfer learning to downstream tasks
We report the transfer learning performance of the MobileNet [22] 23 encoder trained by both
SACL+XD and XD. For the downstream classification, we use CIFAR- 10 and CIFAR- 1000--0000 as our
target tasks. We also validate the pre-trained lightweight encoder on the VOC2007 dataset for downstream object detection. Follow the setup [14, we finetune the models for 10,000 steps with SGD and batch size of 64. The experimental setup is summarized in the Appendix. Table 6 summarizes the transfer learning performance the proposed method compared to SEED [16] For the CIFAR tasks, our method achieves 7.42% and 16.30% accuracy improvements compared to SEED 16. Together with the 1.83% and 5.69% improvements compared supervised learning.
4.6 Comparison to SoTA energy-efficient supervised learning
In addition to the accuracy and training cost improvement compared to distillation-based contrastive learning, the powerful lightweight backbone model trained by the proposed method reveals new perspective of energy energy-efficient inference compared to conventional supervised sparse training

---

ResNet
24
15
Table 7 summarizes the CIFAR downstream comparison between the pre-trained and the ResNet-50 sparsified by the recent SoTA pruning methods [24 [5]
MobileNet 29 23
Despite the additional fine-tuning effort, the powerful lightweight backbone pre-trained by the pro- posed method achieves better accuracy-mode size tradeoff compared to the conventional supervised sparse learning with high element-wise sparsity. More importantly, the powerful lightweight architec- ture can be accelerated and deployed the energy-constrained hardware without the requirement of the dedicated accelerator.
Conclusion
In this paper, we propose novel contrastive learning (CL) algorithm designed for lightweight encoders. We first introduce the slimmed asymmetrical contrastive learning (SACL), which treats the lightweight model CL as a slimmed sparse training task with asymmetrical encoding. On top of the SACL, we propose the cross-distillation (XD) algorithm, distilling the knowledge by minimizing the decorrelation between the embeddings encoded by SACL. Compared to previous works, the proposed algorithm achieves new SOTA accuracy without introducing the large- teacher expensive pre-training. Furthermore, solely training both lightweight and large-sized ResNet with the XD can still achieve superior performance. Compared to supervised learning, the ImageNet- 1-tranned lightweight encoder shows superior performance in the downstream tasks with transfer learning.
Impact and limitations. In this work, SACL+XD works well with correlation- based loss [35] along with the shared encoder. Exploring the possibilityy of the proposed algorithm with the momentum- based encoder updating scheme [18, [9] could be interesting direction. With the powerful, lightweight backbone trained by the proposed algorithm, we will further investigate the post-training quantization and hardware deployment, which further unleash the superior performance in Table 6 with practical downstream applications with actual hardware. To that end, our work can further improve the versatility of contrastive learning with energy-efficient applications (e.g., Al on edge).
10

---

Acknowledgments. We thank the anonymous reviewers for their comments and suggestions. This work is supported in part by the National Science Foundation under Grant No. 2144751, 2314591, 2328803, 2342726, and CoCoSys Center in JUMP 2.0, an SRC program sponsored by DARPA.
References
[1] Yuki Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clus- tering and representation learning. In International Conference Learning Representations (ICLR), 2020.
[2] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning (ICML), 2022.
[3] Adrien Bardes, Jean Ponce, and Yann Lecun. Vicregl: Self-supervised learning of local visual features. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems (NeurIPS), 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), 2021.
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations, In International Conference Machine Learning (ICML), 2020.
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved Baselines with Momentum Contrastive learning. arXiv preprint arXiv:2003.0422 040000 2020.
[9] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2021.
[10] Hee Min Choi, Hyoa Kang, and Dokwan Oh. Unsupervised representation transfer for small networks: believe can distill on-the-fly. Advances in neural information processing sys- tems (NeurIPS), 2021.
[11] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021.
[12] Victor Guilherme Turrisi Da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo- learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 2022.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 12Xi:2000.111929 2020.
[14] Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[15] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICMI.), 2020.

---

[16] Zhiyuan Fang, Jianfeng Wang, Lijuan Wang. Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. In International Conference on Learning Representations (ICI.R), 2021.
[17] Yuting Gao, Jia-Xin Xin Zhuang, Shaohui Lin, Hao Cheng, Xing Sun, Ke Li, and Chunhua Shen. Disco: Remedying self-supervised learning on lightweight models with distilled contrastive learning. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[18] Lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[19] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition (CVPR), 2020.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE conference on computer vision and pullern recognition (CVPR), 2016.
[21] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 550300253,, 2015.
[22] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chia Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision (CVPR), 2019.
[23] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 04861, 2017.
[24] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang. and Decebal Constantin Mocanu. Sparse Training via Boosting Pruning Plasticity with Neuroregeneration. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[25] Jian Meng, Li Yang, Jinwoo Shin, Deliang Fan, and Jae-sun Seo. Contrastive Dual Gating: Learning Sparse Features With Contrastive Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[26] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv preprint 10XXvv/807.0333 2018
[27] Haizhou Shi, Youcai Zhang, Siliang Tang, Wenjie Zhu, Yaqian Li, Yandong Guo, and Yueting Zhuang. On the efficacy of small self-supervised contrastive models without distillation signals. In Proceedings of the AAAI Conference on Artificial Intelligence (NeurlPS), 2022.
[28) Fuwen Tan, Fatemeh Sadat Saleh, and Brais Martinez. Effective self-supervised pre-training on low-compute networks without distillation. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[29] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning (ICML). PMLR, 2019.
[30] Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 20730-20740, 2022.
[31] Jue Wang, Gedas Bertasius, Du Tran, and Lorenzo Torresani. Long-short temporal contrastive learning of video transformers. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 14010-14020, 2022.
12

---

[32] Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. Clusterfit: Improving generalization of visual representations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6509-6518, 2020.
[33] Yang You, Igor Gitman, and Boris Ginsburg. Large Batch Training of Convolutional Networks. arXiv preprint arXiv: 038888,88, 2017.
[34] Sukmin Yun, Hankook Lee, Jaehyung Kim, and Jinwoo Shin, Patch-level representation learning for sulf-supe vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8354-8363, 2022.
[35] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow Twins: Self- supervised Learning via Redundancy Reduction. In International Conference on Machine Learning (ICML), pages 23310-12320, 2021.
[36] Kai Zheng, Yuanjiang Wang, and Ye Yuan. Boosting contrastive learning with relation knowl- edge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022.
[37] Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.
13

---

A Appendix
A.1 Algorithm
Algorithm 1: PyTorch-style pseudcoode for the proposed algorithm
f:
encoder model
# h: projector head
slim ratio of SACL
# slicer: SACL slicer
alpha: weight between loss and loss
# lambda: weight on the off-diagonal terms
def normalize(z):
z_norm = (z meamn(iimm=)) / z. (dd(m==)))
return z_norm
for batch in trainloader: x_a, x_b batch
SACL forward pass
slicer. allcee.remove....mmm rwwovee wwwooee_makk ()
z1 h(f a(_ax))) a))
activate_maske ()
z2 (x_bb)
reverse the order of input with torch. no.chd))
slicer www.veemeosse
zit = h(f((bb)
saiceraactivate mask ()
22t = h(fax))))
# cross correlation
cab mm (normalize (z1) .T (x)m
caat = mm (20r .T, normalize(zit) / N
cbbt mm (20r T, normalize (z2t)) /
# Contrastive leanring loss cl_loss = bt_loss(cab)
# CD loss
dcorr a off di__diggoaallcccc (caat) mul (lambda) sum ()
dcorr b off disgoanan (caat) .mul_ (lambda) sum ()
cd_ loas = (dcorr_a dcorr_b) / 2
loss cl_loss.mi (alpha) + cd loss. mul (1-alpha)
loss. (ackward
aptimizer.stepf
14

---

A.2 Compared to the Log-based distillation loss
From the perspective of knowledge distillation, the negative logarithm- based distillation loss has been widely incorporated into the beacher-studenn learning. In Section 3.2, we proposed the cross-distillation (XD) learning scheme. The distillation objective in Eq (10) is the inner decorrelation distillation loss, we also investigate the negative logari thm (e-g. -a log b) distillation loss that is employed both supervised knowledge distillation [2 and contrastive learning [6].
minimization between embeddings and [ž]. In addition to the correlation-base
To avoid the unbalanced loss magnitude, the distillation loss introduced as the regularization controlled by the penalty level y:
== LacCe (ZA 2B) + YCDD
(11)
LCD (-[žA]log 24 + - - log zB)/2
(12)
We empirically observe that the negative logarithm- distillation loss failed to outperform the proposed cross-distillation loss LOD with inner-decorrelation minimization. As shown in the ImageNet-100 results below:
Method
Encoder
# of Params (M)
Linear Eval Acc. (%)
XD
MobileNet-Vi (Ix)
3.2
80.30
XD (w/ negative log)
MobileNet-VI (1x)
3.2
79.63*
Barlow Twins 35
MobileNet-V1 (1x)
3.2
78.40
*: Best accuracy we found with =le-3.
Although the negative-logarite thm distillation loss is suboptimal compared the inner decor- relation minimization, the proposed distiliistil learning scheme beneficial to lightweight contrastive learning, compared to the baseline [35]
A.3 Detailed Experimental Setup of Pre-training
The encoders (MobileNet, EfficientNet, ResNet-50) are trained on ImageNet-IK with 100/200/300 epochs from scratch with the proposed method. We set the batch to 256 with learning rate = 0.8. We employ the LARS optimizer with weight decay set to 1.5e-6. We set the correlation weights À 0.005. The hidden layer dimension of the projector is 4096. The detailed data augmentation is summarized Table 8
ImageNet-IK
Table 8: Detailed image augmentation settings on ImageNet-IK.
Parameter
XA
XB
Random crop size
224 x 224
224 x 224
Horizontal flip probability
0.5
0.5
Color jitter probability
0.8
0.8
Brightness adjustment probability
0.4
0.4
Contrast adjustment probability
0.4
0.4
Saturation adjustment probability
0.2
0.2
Hue adjustment probability
0.1
0.1
Gaussian blurring probability
1.0
0.1
Solarization probability
0.0
0.2
With the proposed cross-distillation method, we train the lightweight ViT model on the ImageNet-100 100 dataset for 400 epochs. The batch size is set to 256 with Adam W optimizer. The learning rate and weight decay are set to 0.005 and le-4. The detailed data augmentation summarized in T
ImageNet-100
15

---

Table Detailed image augmentation settings on ImageNet- 100.
Parameter
X
XB
Random crop size
224
224
224 x 224
Horizontal flip probability
0.5
0.5
Color jitter probability
0.8
0.8
Brightness adjustment probability
0.4
0.4
Contrast adjustment probability
0.4
0.4
Saturation adjustment probability
0.0
0.2
Hue adjustment probability
0.1
0.1
Gaussian blurring probability
1.0
0.1
Solarization probability
0.0
0.2
CIFAR-10
The proposed method trained from scratch by 1,000 epochs with LARS-SGD opti-
We use 256 batch size along with 0.3 learning rate and le 4 weight decay. The Cosine learning rate scheduler is used with 10 epochs of warmup training. The detailed data augmentation
mizer 33
summarized in Table I0
Table 10: Detailed image augmentation settings on CIFAR-10.
Parameter
Xx
X B
Random crop size
32 x 32
32 X 32
Horizontal flip probability
0.5
0.5
Color jitter probability
0.8
0.8
Brightness adjustment probability
0.4
0.4
Contrast adjustment probability
0.4
0.4
Saturation adjustment probability
0.2
0.2
Hue adjustment probability
0.1
0.1
Gaussian blurring probability
0.0
0.0
Solarization probability
0.0
0.2
A.4 Detailed Experimental Setup of Downstream Fine-tuning
We evaluate the transferability of the pre- pre-trained lightweight model on downstream tasks, including CIFAR-10, 10, CIFAR-100 and VOC2007. Following the settings in [14, we fine-tuned the models for 10,000 steps with SGD and batch size of 64. The learning rate set to 0.1 with no weight decay. The input samples are resized to 224 224 to maintain the dimensionality as the pre-trained model. The checkpoint of the pre-trained lightweight model will be released soon.
16

---

A.5 Detailed Training Cost of the Proposed Method
In addition to the FLOPs comparison in Table we evaluate the training cost of the proposed method based on the actual training time per epoch, together with the total activation count (feature map pixels) for each forward pass of the training process, shown Table II and Table T2 respectively.
Table 11: Training time comparison between the proposed method and the distillation- based CL
Model
Training Method
Teacher
Training time / epoch
GPU Type
Batch Size
SEED [66
MobilcNet-V3
ResNet-50
35 min
A (80G)
256
SACL-XD (Ours)
MobileNet-V3 1.5x-Ix
N/A
26 min see
100 (80G)
256
XD Only (Ours)
MobileNet-V3
N/A 16 min 15 sec
1100 (80G)
256
Table 12: Activation count comparison between the proposed method and the distillation-based CL
Method
Encoder
Teacher
Act. Count (E+07)
ImageNet- 1K Accuracy (%)
SACL+XD (Ours)
Bff-BO (1)5x1x)
N/A
1.54
65.32
XD Only (Ours)
Mob-V3 (1x)
59.34
SSL-Small 27
Mob-V3
N/A
0.90
48.70
SSL-Small 27
Eff-BO
N/A
0.68
55.90
ReKD 36
Mob-V3
ResNet-101 101
2.03
59.60
ReKD
Mob-V3
ResNet-50
1.53
56.70
ReKD
Eff-BO
ResNet-50
1.75
63.40
SEED 16
Mob-V3
ResNet-50
1.52
55.20
SEED 16
Eff-BO
ResNet- 101
2.26
61.30
17


Reviews:
[{"review_id": "7xrLOBB62Y", "reviewer": "Reviewer_CDjk", "summary": "The authors propose a combination of two method: Slimmed Assymterical Contrastive Learning(SACL) , and Cross-Distilation(XD) with a correlation-maximization loss.  SACL does magnitude pruning of filters at each epoch, and the pruned model is used as a encoder of one of two views used in the contrastive loss.  They find that XD with an efficient network already provides competitive performance, and SACL brings performance to state-of-the-art. Evaluations are done with linear evaluation on ImageNet, CIFAR, and VOC2007. ", "questions": "* How does the method compared to a supervised high-water mark, beyond the experiments in Table 6? \n* How is transfer performance, beyond CIFAR and VOC?\n* How could one implement an equivalent slimming procedure for ViTs? Having a plausible path for implementation in ViT's could further strengthen the work. \n* In Table 2, row SSL-Small, why is the training flops much larger than XD-only? It is same architecture, same epoch count, no pretraining. \n* ", "limitations": "Limitations are adequately addressed? ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: Though cross-distillation and training sparse networks are both known techniques, their combination with correlation-based contrastive learning is a novel combination. \n\nQuality: The quality of the experimental design is sufficient to be convincing, and the idea makes sense, and seems easy to implement, increasing potential impact. \n\nClarity: The clarity of the writing is reasonably high, with the exception of the description of XD(see weakneses). \n\nSignificance: Self-supervised pre-training has proven to be quite useful, and efforts to remove key limitations in encoder size can achieve important practical impact. ", "weaknesses": "* Experiments: One key advanatage of SSL methods  is in transfer learning, and learning general features. However, the transfer learning study in this work is relatively small, with only CIFAR and VOC presented. In my mind, this is the biggest weakness of the work. \n*  It's not explicitly stated how XD experiments are setup. Are they cross-distillation across two instances of a network of the same architecture (but independent weights), like Figure 2c but without slimming?\n* Slimming idea is general, but implementation seems CNN-specific. \n* Line 85: Minor, but saying ResNet-50 is mega-sized is a bit of an overclaim IMO. \n* Lack of substantial discussion of different axes of efficiency (FLOPs, wall time, activation count). Each of these metrics is useful in its own way, but training FLOPS is primary focus in this work. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Rebuttal to Reviewer CDjk\n\nThank you for your thorough review on our paper and we appreciate your positive feedback. \n\n### Weaknesses\n\n**Response to Weakness 1**: Thank you for pointing this out. To address your comments, we extended the experiments in Table 6 of the original manuscript with extended datasets and results, as shown in **Table 1** of the attached PDF file.\n\nAs shown in Table 1, the lightweight model trained by the proposed method can be powerful across a wide range of %spectrums in terms of downstream tasks. Compared to the model that is pre-trained on ImageNet-1K with **supervised learning**, the proposed method consistently achieves superior performance among all tasks, which indicates the high effectiveness of the proposed method in visual representation learning. We will include this updated table in the next revised version of the paper. \n\n------\n\n**Response to Weakness 2:** Thank you for your comments. We want to clarify the settings of XD as follows: \n\n- The cross distillation (XD) and SACL-XD exhibit the same settings where the weights are **shared** between two encoders. Since the augmentation and sparsity are collectively applied to the encoder, the latent output needs to preserve the similarity by having identical non-slimmed weights.\n- As described from line 230 to line 241 of the original manuscript, minimizing the cross distillation loss $\\mathcal{L}_{\\texttt{CD}}$ avoids the aliasing features across different dimensions of the augmented input. Therefore, consistent encoding is required, which necessitates the shared weights between two encoders. \n\n------\n\n**Response to Weakness 3:** We will modify the \"mega-sized\" encoder to \"large-sized\" encoder in the next revised version of the paper. \n\n------\n\n**Response to Weakness 4:** Thank you for your feedback on this. To address your question, we extended Table 5 of the original manuscript with the results of the ViT-Tiny [12] model trained by the proposed SACL-XD method under the slimming ratio of 1.25$\\times$ - 1$\\times$. The updated table (**Table 2**) is attached in the PDF file.\n\nSpecifically, we slim down the embedding dimensionality of the ViT model, which further leads to the reduced model width in both input and output features of the fully-connected layers (and normalization layers). As shown in Table 2, the proposed SACL+XD continuously achieves improved accuracy compared to the prior baselines. \n\nWe will include the details of the transformer-based SACL-XD in the next version of the paper. \n\n------\n\n**Response to Weakness 5:** Thank you for pointing this out. To address your comment, we summarized the forward pass activation count of the training process in Table 3 of the attached document. \nSpecifically, we compute the total activation count in **both** contrastive encoders (or teacher and student) in each forward pass during the training process. For the previous distillation process, the intermediate activation resulted from both student and large-sized teacher models. As shown in **Table 3**, the proposed method trains the lightweight encoder without introducing the large ResNet teacher model, collectively achieving better accuracy and lower activation count, which will further lead to reduced memory consumption during the training process. \n\nIn **Table 4** of the attached PDF file, we also summarize the comparison of training time per epoch. These are measured by running a single training task on a single Nvidia A100 GPU. \n\n------\n\n### Questions\n\n**Q1:** How does the method compare to a supervised high-water mark, beyond the experiments in Table 6?\n\n**A1:** Thank you for your question. Regarding the general comparison with the supervised learning on the small-sized vision tasks (e.g., CIFAR-10/100), we validated the proposed algorithm from two perspectives: 1) Comparison with the large-sized dense model and 2) The sparse model pruned from the large-sized dense model. We use the widely-reported CIFAR performance as an example and summarize the comparison in **Table 5** of the attached PDF file.\n\n[Ref-1]: U.Evci, et al., \"Rigging the lottery: Making all tickets winners\", ICML, 2020\n\n[Ref-2]: S.Liu, et al., \"Sparse Training via Boosting Pruning Plasticity with Neuroregeneration\", ICLR 2021\n\nAs shown in the table, the lightweight MobileNet-V3 model trained by the proposed method achieves better performance compared to the SoTA sparse training algorithms with similar parameter sizes ($\\sim$3 million). In other words, the proposed method can successfully train a strong, lightweight vision learner with superior downstream performance **without** compression or pruning. \n\n------\n\n**Answer to Q2:** Please refer to our response to Weakness 1. \n\n------\n\n**Answer to Q3:** Please refer to our response to Weakness 4. \n\n------\n\n**Q4:** In Table 2, row SSL-Small, why is the training flops much larger than XD-only? It is same architecture, same epoch count, no pretraining.\n\n**A4**: Thank you for pointing this out. \nThe FLOPs value for SSL-Small [24] in Table 2 of the original manuscript was incorrect due to a mistake, which we apologize for.\n\nWe corrected the top part of Table 2 of the original manuscript as Table 6 in the attached PDF file. \nAs reported in Table 7 and Table 10 of [24], the model is trained by 800 epochs with contrastive learning and initialized with 2 epochs SEED pre-training.  Specifically, we adopt the Thop as the FLOPs counter. For EfficientNetB0, the total training FLOPs **per epoch** is 4.2E+15, and the total FLOPs of 800 epochs is 3.4E+18. Together with the 2 epochs pertaining with SEED [14] (5.29E+16 per epoch), the total training FLOPs of SSL-Small = 3.43E+18. Applying the same computation to MobileNet-V3 leads to the total FLOPs = 1.94E+18. \n\nDue to the larger training effort and pre-training, the training cost of [24] is still more expensive than ours.\n"}, {"review_id": "3dUvK6DfmE", "reviewer": "Reviewer_UkVS", "summary": "This paper introduces a self-supervised contrastive learning algorithm designed specifically for training lightweight models, eliminating the need for a large teacher model. The algorithm comprises two main components: slimmed asymmetrical contrastive learning (SACL) and cross-distillation (XD). The authors evaluate their approach using different lightweight models and datasets, demonstrating its superiority in terms of performance and efficiency over existing methods. ", "questions": "1, While the paper focuses on the limitations of contrastive learning (CL) for lightweight models, it is worth considering whether alternative self-supervised learning methods, such as generative pre-training or reconstruction-based pre-training, are effective for these models. Therefore, it would be valuable to include a comparison between the proposed methods and these alternative approaches to provide a comprehensive understanding of the performance of different self-supervised learning techniques on lightweight models.", "limitations": "Please refer to the weakness and questions.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1, This paper is well-motivated. This paper tackles a significant and practical challenge of training lightweight models using contrastive learning, which has received limited attention in previous research efforts.  \n2, The paper presents a novel and straightforward concept of slimming the host encoder, creating asymmetrical encoding paths for contrastive learning. This innovative approach effectively reduces training costs and enhances the performance of lightweight models.  \n3, The writing is clear and easy to follow.  \n4, The paper conducts thorough experiments on diverse models, datasets, and downstream tasks, and conducts comprehensive comparisons with state-of-the-art methods. The results strongly validate the efficacy and general applicability of the proposed method.  \n", "weaknesses": "1, To enhance the clarity and understanding of the proposed methods, it would be beneficial to provide additional intuition. Specifically, more insights can be provided on how cross-distillation aids in overcoming the distortion resulting from asymmetrical encoding. Additionally, exploring the trade-offs associated with different levels of asymmetry and sparsity would further enrich the paper.  \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer UkVS\n\n### Weaknesses\n\nThank you for your thorough review on our paper and we appreciate your positive feedback. \n\n**Weakness 1:** To enhance the clarity and understanding of the proposed methods, it would be beneficial to provide additional intuition. Specifically, more insights can be provided on how cross-distillation aids in overcoming the distortion resulting from asymmetrical encoding. Additionally, exploring the trade-offs associated with different levels of asymmetry and sparsity would further enrich the paper.\n\n**Response to Weakness 1**: Thank you for your comments. The asymmetric encoding of the proposed SACL algorithm introduces the distorted latent information in one branch of contrastive encoding. Meanwhile, the asymmetrical encoding + contrasive learning can also be treated as a distillation task. However, na\\\"ively minimizing the distance between the dense and slimmed encoders is not the perfect solution for learning, as pointed out in Section 1.2 of the supplementary material.  \n\nTherefore, the design of the contrastive loss should (1) align the features across different dimensions of the latent space and (2) avoid the feature mismatch and collapse between the slimmed and original features. As a result, the cross-distillation loss is added as a moving average term on top of the original SACL loss function. \n\nRegarding the tradeoffs with different sparsity levels, we explored the different asymmetry-accuracy tradeoffs in Table 5 of the original manuscript. To further clarify the tradeoff between performance-accuracy-training cost, we extended Table 5 of the original manuscript with training FLOPs and summarized the comparison in **Table** 7 of the attached PDF. \n\n------\n\n### Questions\n\n**Q1:** While the paper focuses on the limitations of contrastive learning (CL) for lightweight models, it is worth considering whether alternative self-supervised learning methods, such as generative pre-training or reconstruction-based pre-training, are effective for these models. Therefore, it would be valuable to include a comparison between the proposed methods and these alternative approaches to provide a comprehensive understanding of the performance of different self-supervised learning techniques on lightweight models.\n\n**A1:** Thank you for pointing this out. Currently, our paper mainly focuses on contrastive learning with lightweight models. We will further investigate the lightweight generative pre-training (GPT) in our future work. \n\n"}, {"review_id": "FDYcn5graQ", "reviewer": "Reviewer_J8Hb", "summary": "This paper proposes a self-supervised contrastive learning method to improve the light-weight network performance and reduce the training cost. It mainly consists of two components: slimmed asymmetrical contrastive learning (SACL) and cross-distillation (XD), which are able to train the efficient network from scratch without the usage of pre-trained strong teacher. Experiments the effectiveness of the proposed method in terms of accuracy and training FLOPs reduction.", "questions": "Refer to the Weaknesses.", "limitations": "Yes. The authors well address the impact and limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. SACL+XD is effective to improve the light-weight network performance with efficient training.\n2. well-written and easy to follow this paper.\n", "weaknesses": "1. The SACL+XD seems to apply the slimmable neural network[ref-1] to Disco[15]. Therefore, the novelty seems to be limited.\n\n2. [Experimental issues.] (1)This paper mainly uses several efficient backbones (e.g. MobileNets) as the base model backbone, It lacks of comparisons on the more dense backbones, such as ResNet-50. (2) SACL+XD trains the models from scratch by removing a unified amount of channels based on the lowest magnitude score. I think the initial weights should have an effect on the final performance, as the channel selection in a slimmed network f_{\\theta}^s may be different under the different weight initialization settings. However, the experiments lack such comparison about the initialization, as well as the mean accuracy with std. (3) In this paper, computation reduction is evaluated by training FLOPs. The authors better add the practical training time for a more comprehensive comparison. \n\n\n[Ref-1] Yu J, Yang L, Xu N, et al. Slimmable neural networks[J]. arXiv preprint arXiv:1812.08928, 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer J8Hb\n\nThank you for your thorough review on our paper and we appreciate your positive feedback.\n\n**Weakness 1:** The SACL+XD seems to apply the slimmable neural network[ref-1] to Disco[15]. Therefore, the novelty seems to be limited.\n\n**Response to Weakness 1:** Thank you for your feedback, but we respectfully disagree with your comments. \n\nFirstly, the slimmable network [ref-1] is designed for training **multiple** sparse subset networks with **supervised learning**. However, the proposed method introduced two novel techniques of (1) *slimmed asymmetry* and (2) *cross distillation* for **self-supervised** contrastive learning. Our paper aims to resolve the low accuracy issue of lightweight models (e.g., MobileNet) trained by contrastive learning schemes. The objective is to obtain a **single** powerful, lightweight model via contrastive learning without introducing the large-sized teacher, which can also achieve superior performance in the downstream vision tasks, as summarized in Table 2 and Table 6 of the original manuscript. \n\nSecondly, the proposed method has key essential differences  and novelties compared to Disco [15]. In particular, Disco [15] introduced an auxiliary \"mean student'' encoder in addition to the student-teacher pair. In other words, Disco [15] complicates the lightweight model contrastive learning with complex distillation design, and the large-sized ResNet teacher model **is still preserved** in the contrastive learning process. On the contrary, our proposed method **completely eliminates** the \"ResNet teacher\" distillation and teacher pre-training for lightweight contrastive learning, while achieving superior performance in both ImageNet and downstream tasks. \n\nIn conclusion, the proposed method is orthogonal to slimmable neural networks [ref-1] and Disco [15]. Furthermore, our method completely eliminates the \"student-teacher'' distillation that is heavily exploited in Disco [15]. \n\n---\n\n**Weakness 2-1:** (1) This paper mainly uses several efficient backbones (e.g. MobileNets) as the base model backbone, It lacks of comparisons on the more dense backbones, such as ResNet-50. \n\n**Response to Weakness 2-1:** While it is correct that the main objective of our paper is to train the high-performance, lightweight encoder via contrastive learning (which is a long-lasting problem in prior contrastive learning methods), our original manuscript still also validated the proposed cross-distillation (XD) method on the large-sized ResNet-50 backbone model. To that end, we respectfully disagree with your comment that it lacks comparisons on denser backbones like ResNet-50. As shown in Table 2 of the original manuscript, our method can achieve improved performance on ResNet-50 compared to the recent contrastive learning methods, which proves the versatility of our proposed method. In conclusion, the proposed method achieves good performance with **both** lightweight and large-sized encoder models.\n\n**Weakness 2-2:** (2) SACL+XD trains the models from scratch by removing a unified amount of channels based on the lowest magnitude score. I think the initial weights should have an effect on the final performance, as the channel selection in a slimmed network $f_{\\theta}^s$ may be different under the different weight initialization settings. However, the experiments lack such comparison about the initialization, as well as the mean accuracy with std.\n\n**Response to Weakness 2-2:** Thank you for your question. We want to highlight the fact the major focus of our method is providing a generic and versatile contrastive learning solution for lightweight models.\nIn particular, our motivation is directly training the lightweight encoder via contrastive learning without introducing expensive distillation [9, 14, 15, 32], additional tuning on initialization [24], or fine-grained view scaling and sampling [25]. With the simple training setup of the proposed method, the learned visual representation via our proposed method leads to superior performance on the downstream tasks compared to the lightweight model trained by supervised learning. \nUnlike previous work [24], we posit that initializing the weights with minimal distillation may have minimal impact on our asymmetrical slimming. This is because the lightweight encoder is sliced from the widened model, and this corrupts the benefits of the weight initialization.\n\n---\n\n**Weakness 2-3:** (3) In this paper, computation reduction is evaluated by training FLOPs. The authors better add the practical training time for a more comprehensive comparison.\n\nThank you for pointing this out. To address your question, we summarize the comparison of training time per epoch in **Table 4** of the attached PDF. \n\nThese are measured by running a single training task on a single Nvidia A100 GPU. We will include this table in the next version of the paper together with all the other models."}, {"review_id": "smdQuIwDZj", "reviewer": "Reviewer_8Hzi", "summary": "This paper proposed a new method to enable effective contrastive learning (CL) on the lightweight encoder without a mega-size teacher model, which can thus reduce the training cost. More specifically, it expanded the target model to a larger model, which shares its weights with the target model. Then the CL problem is formulated as slimmed training task with asymmetrical encoding. Furthermore, this work incorporated cross-distillation to further minimize the decorrelation between the embeddings of the same view but from different encoders. Experimental results show that the proposed method can outperform the existing baselines with much smaller computational costs on the lightweight models.", "questions": "1. Can you provide some explanation about why the proposed method can achieve better performance than the supervised learning in Table 6?  \n\n2. In Figure 1, there are two red spots annotated with \"SCAL + XD\". What are the differences?\n\n3. Can the authors provide an example to illustrate the meaning of the style of \"K $\\times$ -1 $\\times$ \" for the slimming ratio? For instance, if the pruning ratio of the task model is 0.5 compared with the expanded model,  what is the value of K and $s$?\n\n4. In Equation (7), what is the meaning of $n, i, j$?\n\n5. For $h_{\\phi}$ from the two different encoders, are they of the same dimension? Do $Z_{A}$ and $Z_{B}$ have the same dimension?\n \n", "limitations": "Although the author claims that their method does not need an extra mega-size teacher, their method still needs an expanded version of the task model for effective CL.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-motivated. Figure 1 clearly shows that the efficient CL of the lightweight model is challenging and what improvement the proposed method can make to this problem. \n\n2. The proposed method is reasonable and Figure 2 clearly shows the key idea.\n\n3. The proposed method can not only be effective in the CL of a lightweight model but also in a mega-size model.  \n\n4. It is good to see that the authors include the experimental results to show that the trained encoder can be well generalized to different types of downstream tasks.", "weaknesses": "1. Although the authors claim Ref [25] cannot work on the mega-size encoder, it should be compared with the proposed method on the lightweight encoder.\n\n2. It would be better if the authors could show the performance of supervised learning in the tables of the experiment section to serve as the upper bound of the proposed method.\n\n3. It would be better if the authors can add a column for the training cost in Table 3, 4 and 5.\n\n4. Some technical details are not clear, which is shown in the question part.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 8Hzi\n\nThank you for your thorough review on our paper and we appreciate your positive feedback.\n\n### Weakness\n\n**Weakness 1:** Although the authors claim Ref [25] cannot work on the mega-size encoder, it should be compared with the proposed method on the lightweight encoder.\n\n**Response to Weakness 1:** Thank you for your feedback. [25] proposed an input-scaling and cropping strategy dedicated to lightweight encoders. Noticeably, [25] introduces **four** different types of multi-view sampling parameters as the hyperparameters of the training process. Such a fine-grained scaling strategy is orthogonal to our proposed method, which focuses on a generic training solution for lightweight contrastive learning with normal initialization and commonly-used data augmentation strategy as prior contrastive learning methods. Therefore, our method has better generality compared to [25], for both lightweight models (SACL+XD) and large-sized encoders (XD only). The model-specific data augmentation and local-view scaling strategy will remain as our future work.\n\n---\n\n**Weakness 2:** It would be better if the authors could show the performance of supervised learning in the tables of the experiment section to serve as the upper bound of the proposed method.\n\n**Response to Weakness 2:** Thank you for pointing this out. Due to the page limit of rebuttal,  we will include the supervised learning results as the upper bound of the performance of all the tables in the next version of the paper. \n\n---\n\n**Weakness 3:** It would be better if the authors can add a column for the training cost in Table 3, 4 and 5.\n\n**Response to Weakness 3:** Thank you for highlighting this. We updated Table 5 with the total training FLOPs as **Table 7** of the attached PDF. Due to the page limit of rebuttal, we will include the detailed training FLOPs in Table 3 and 4 in the next version of the paper.\n\n------\n\n### Questions: \n\n**Q1:** Can you provide some explanation about why the proposed method can achieve better performance than the supervised learning in Table 6?\n\n**A1:** Thank you for your question. As presented in the prior works of contrastive learning, aligning the latent information leads to enhanced robustness in learning various visual representations. \nCompared to supervised learning with deterministic labels, the encoder (e.g., ResNet-50) trained by contrastive learning can achieve better performance in the downstream small vision tasks with minimum tuning, as shown in [16, 31]. \n\n---\n\nThe proposed cross-distillation algorithm not only maximizes the correlation between latent features but also avoids feature aliasing across different dimensionality, which further enhances the representation learning of the model.\n\n**Q2:** In Figure 1, there are two red spots annotated with \"SCAL + XD\". What are the differences?\n\n**A2:** Thank you for pointing this out and we apologize for the confusion. The large dots and small dots in Figure 1 represent the EfficientNet-B0 encoder (4.0 Million parameters) and MobileNet-V3 encoder (3.0 Million parameters), respectively, as shown in the bottom right legend of the figure. We will further clarify that in the next version of the paper. \n\n---\n\n**Q3:** Can the authors provide an example to illustrate the meaning of the style of \"K -1 \" for the slimming ratio? For instance, if the pruning ratio of the task model is 0.5 compared with the expanded model, what is the value of K and s?\n\n**A3:** Thank you for your question. By saying \"K-1\" asymmetry, it means that we slim both input and output channels by 1/K times. If $K$=2, the input/output channels are slimmed by half, which leads to $s=75\\%$ structured sparsity.\n\n---\n\n**Q4:** In Equation (7), what is the meaning of $n$, $i$, $j$?\n\n**A4:** Thank you for your question. $n$ represent the index of batch, and $i$ and $j$ represent the dimensionality indices across the latent output. Correspondingly, $C_{i,j}$ represents the ${i,j}$ element of the correlation matrix. We will further clarify the symbols in the next version of the paper. \n\n---\n\n**Q5:** For from the two different encoders, are they of the same dimension? Do $Z^A$ and $Z^B$ have the same dimension?\n\n**A5:** To make the dimensionality matched and non-empty at the output of the last layer, we skip the slimming in the output channel (input channel is still slimmed) of the final layer. Therefore, $Z^A$ and $Z^B$ will have the same dimensionality."}]

Author rebuttal:
Dear Reviewers and AC, 

We thank all the reviewers for their helpful feedback, comments, and questions. As mentioned in the individual rebuttal thread, we upload the 1-page PDF attachment that includes all the updated tables for your reference. 

Please let us know if you have any further questions or comments. 

Thank you. 

Meta-review: This paper proposes a self-supervised learning method for training lightweight models, without the help of the large teacher model. The model contains two main components: slimmed asymmetrical contrastive learning (SACL) and cross-distillation (XD). The paper demonstrates the effectiveness of the proposed algorithm on different lightweight baseline models and datasets. The paper receives four weak accepts. The rebuttal and response well resolve the reviewers' concerns. The AC recommends acceptance for this paper.
Decision: Accept (poster)