Slimmable Networks for Contrastive Self-supervised Learning
Authors: Shuai Zhao
Xiaohan Wang
Linchao Zhu
Yi Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: self-supervised learning
contrastive learning
slimmable networks

Abstract: Self-supervised learning makes great progress in large model pre-training but suffers in training small models. Previous solutions to this problem mainly rely on knowledge distillation and indeed have a two-stage learning procedure: first train a large teacher model, then distill it to improve the generalization ability of small ones. In this work, we present a new one-stage solution to obtain pre-trained small models without extra teachers: slimmable networks for contrastive self-supervised learning (SlimCLR). A slimmable network contains a full network and several weight-sharing sub-networks. We can pre-train for only one time and obtain various networks including small ones with low computation costs. However, in self-supervised cases, the interference between weight-sharing networks leads to severe performance degradation. One evidence of the interference is gradient imbalance: a small proportion of parameters produces dominant gradients during backpropagation, and the main parameters may not be fully optimized. The divergence in gradient directions of various networks may also cause interference between networks. To overcome these problems, we make the main parameters produce dominant gradients and provide consistent guidance for sub-networks via three techniques: slow start training of sub-networks, online distillation, and loss re-weighting according to model sizes. Besides, a switchable linear probe layer is applied during linear evaluation to avoid the interference of weight-sharing linear layers. We instantiate SlimCLR with typical contrastive learning frameworks and achieve better performance than previous arts with fewer parameters and FLOPs.

Under review as a conference paper at ICLR 2023
SLIMMABLE NETWORKS FOR CONTRASTIVE SELF- LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Self-supervised learning makes great progress in large model pre-training but suf- fers in training small models. Previous solutions to this problem mainly rely on knowledge distillation and indeed have a two-stage learning procedure: first train large teacher model, then distill it to improve the generalization ability of small ones. In this work, present a one-stage solution to obtain pre-trained small models without extra teachers: slimmable networks for contrastive self- supervised learning (SlimCLR). A slimmable network contains a full network and several weight-sharing sub-networks. We can pre-train for only one time and ob- tain various networks including small ones with low computation costs. However, in self-supervised cases, the interference between weight-sharing networks leads severe performance degradation. One evidence of the interference is gradient imbalance: a small proportion of parameters produces dominant gradients dur- ing backpropagation, and the main parameters may not be fully optimized. The interference between networks also result in gradient direction divergence. To overcome these problems, we make the main parameters produce dominant gradi- ents and provide consistent guidance for sub-networks via three techniques: slow start training sub-networks, online distillation, and loss re-weighting according to model sizes. Besides, a switchable linear probe layer is applied during linear evaluation to avoid the interference of weight-sharing linear layers. We instantiate SlimCLR with typical contrastive learning frameworks and achieve better perfor- mance than previous arts with fewer parameters and FL.PPs.
INTRODUCTION
In the past decade, deep learning achieves great success in different fields of artificial intelligence. A large amount of manually labeled data is the fuel behind such success. However, manually la- beled data is expensive and far less than unlabeled data in practice. To relieve the constraint of costly annotations, supe-sspe learning (Dosovitskiy et al., 2016; Wu et al., 2018; van den Oord et al., 2018; He et al., 2020; Chen et al., 2020a) aims to learn transferable representations for down- stream tasks by training networks on unlabeled data. Great progress is made in large models, i.e., models bigger than ResNet-50 (He et al., 2016) that has roughly 25M parameters. For example, ReLICv2 (Tomasev et al., 2022) achieves 77.1% accuracy on ImageNet (Russakovsky et al., 2015) under linear evaluation protocol with ResNet-50, outperforming the supervised baseline 76.5%.
In contrast the success of the large model pre-training, self-supervised learning with small mod- els lags behind. For instance, supervised ResNet- 18 with 12M parameters achieves 72.1% % accuracy on ImageNet, but its self-supervised result with MoCov2 (Chen al., 2020c) is only 52.5% (Fang et al., 2021). The gap is nearly 20%. To fulfill the large performance gap between supervised and self-supervised small models, previous methods (Fang et al., 2021; Gao et al., 2022; Xu al., 2022) mainly focus on knowledge distillation, namely, they try to transfer the knowledge of a self- supervised large model into small ones. Nevertheless, such methodology actually has a two-stage procedure: first train an additional large model, then train small model mimic the large one. Be- sides, one-time distillation only produces a single small model for a specific computation scenario.
obtain different small models through one time pre- pre-training to meet various computation scenarios without extra teachers? Inspired by the suc- cess of slimmable networks (Yu et al., 2019) in supervised learning, present novel one-stage
An interesting question naturally arises:

---

Under review as a conference paper at ICLR 2023
Top-1 Aocuracy on ImaggeNet (%5
Top Accurss sonreey or Mevooaaee 1000
72.5
11.8 Widthe of slimmable ResNet-50 (supervised, 90 epochs)
11.0 0.5)
(1.0. 0.,, 025
[10.0.75 0.5 022
(1.00
110, 0.)
11.8 0.5 022 (10,077 0.. 0.25)
Widthe of elimmable ResNet-50 (MoCov2, 200 epochs)
Figure 1: Training a slimmable ResNet-50 in supervised (left) and self-supervised (right) manners. ResNet-50 1.000.00...00.0000.. 71..0.005.0....... 05,0.500.25 means this slimmbale network can switch width [1.0, 0.75, 0.5, 0.25]. The width 0.25 represents that the number of channels is scaled by 0.25 of the full network.
method to obtain pre- pre-trained small models without adding large models: slimmable networks for contrastive self-super learning (SlimCLR). A slimmable network consists of a full network and some weight-sharing sub-networks with different widths. The width denotes the number of channels in network. Slimmable networks can execute at various widths, permitting flexible deployment on different computing devices. We can thus obtain multiple networks including small ones meeting low computing cases via one-time pre-training, Weight-sharing networks can also inherit knowledge from the large ones via the sharing parameters to achieve better generalization performance.
Weight-sharing networks in a slimmbale network cause interference to each other when training simultaneously. and the situation is worse self-supervised cases. As shown in Figure 1, with supervision, weight-sharing networks only have slight impact on each other, e.g., the full model achieves 76.6% vs. 76.0% accuracy ResNet-50 and 1,000,5555 pervision, the corresponding numbers become 67.2% vs. 64.8%. One observed phenomenon of the interference is gradient imbalance: a small proportion of parameters produces dominant gra- dients during backpropagation. The imbalance occurs because the sharing parameters receive gra- dients from multiple losses of different networks during optimization. The main parameters may not be fully optimized due to gradient imbalance. Besides, the conflicts in gradient directions of weight- networks also cause gradient direction divergence of the full network. Please refer to Appendix A.3 for detailed explanations and visualizations.
Without su-
To relieve the gradient imbalance, the main parameters should produce dominant gradients during the optimization process. To avoid conflicts in gradient directions of various networks, sub-networks should have consistent guidance. Following these principles, we introduce three simple yet effective techniques during pre-training relieve the interference of networks. 1) We adopt slow start strategy for networessr.s. The networks and pseudo supervision of contrastive learning are both unstable and fast updating at the start of training. To avoid interference making the situation worse, we only train the full model at first. After the full model becomes relatively stable, sub-networks can inherit the knowledge via sharing parameters and start with better initialization. 2) We apply online distillation to make all nub-networks consistent with the full model to eliminate divergence of networks. The predictions of the full model will serve as global guidance for all sub-networks. 3) We re-weight the losses of networks according to their widths to ensure that the full model dominates the optimization process. Besides, we adopt a switchable linear probe layer to avoid interference of weight-sharing linear layers during evaluation. A single slimmable linear layer cannot achieve several complex mappings simultancously when the data distribution is complicated.
We instantiate two algorithms for SLimCLR with typical contrastive learning frameworks, iee., Mo- Cov2 and MoCov3 (Chen et al., 2020c; 2021). Extensive experiments are done on ImageNet (Rus- sakovsky et al., 2015) dataset, and the results show that our methods achieve significant performance improvements compared to previous arts with fewer parameters and FLOPs.
2 RELATED WORKS
Self-supervised learning Self-supervised learning aims to leam transferable representations for downstream tasks from the input data itself. According to Liu et al. (2020), self-supervised methods can be summarized into three main categories according to their objectives: generative, contrastive,
2

---

Under review as a conference paper at ICLR 2023
and cinnerative-contrattie (adversarial). Methods belonging to the same categories be further classified by the difference between pretext tasks. Given input z, generative methods encode z into an explicit vector and decode to reconstruct from e.g., auto-regressive (van den Oord al., b016a;b), auto-encoding models (Ballard, 1987; Kingma & Welling, 2014; Devlin et al., 2019; He et al., 2022). Contrastive learning methods encoder input x: into explicit vector = measure similarity. The two mainstream methods below this category are context-instance contrast (info- Max Hjelm et al. (2019), CPC van den Oord al. (2018), AMDIM Bachman al. (2019)) and instance-instance contrast (DeepCluster Caron et al. (2018), MoCo He et al. (2020); Chen et al. (2021), SimCLR Chen et al. (2020a;b), SimSiam Chen & He (2021)). Generative--oomrasstvv meth- ods generate a fake sample x' from x and try to distinguish x' from real samples, e.g., DCGANs Rad- ford et al. (2016), inpainting Pathak al. (2016), and colorization Zhang et al. (2016).
Slimmable networks Slimmable networks are first proposed to achieve instant and adaptive aceuracy-effiiinncy trade-offs on different devices (Yu et al., 2019). It can execute at different widths during runtime. Following the pioneering work, universally slimmable networks (Yu & Huang, 2019b) develop systematic training approaches to allow slimmable networks to run at arbi- trary widths. AutoSlim (Yu & Huang, 2019a) further achieves one-shot architecture search for chan- nel numbers under a certain computation budget. MutualNet (Yang et al., 2020) trains slimmable networks using different input resolutions learn multi-scale representations. Dynamic slimmable networks (Li et al., 2022; 2021) change the number of channels of each layer in the fly according to the input. In contrast to weight-sharing sub-networks in slimmable networks, some methods try to train multiple sub-networks with independent parameters (Zhao et al., 2022b). A relevant concept of slimmable networks in network pruning is network slimming (Liu al., 2017; Chavan et al., 2022; Wang et al., 2021), I), which aims to achieve channel-level sparsity for computation efficiency.
METHOD
3.1 DESCRIPTION OF SLIMCLR
We develop two instantial algorithms for SlimCLR with typical contrastive learning frameworks Mo- Cov2 and MoCov3 (Chen et al., 2020c; 2021). As shown in Figure 2a (right), a slimmable network with n widths wi,...., Wn, contains multiple weight-sharing networks Je_,,.... ". which are pa- rameterized by learnable weights Ow,..... ........ Ou.,, respectively. Each network So,, in the slimmable network has its own set of weights Ow. and Ow, Ow:. A network with small width shares the weights with large ones, namely, O.,, ww, if wy < w/- Generally, we assume Wi < w, if j > i, ie., Wi..... Wr arrange in descending order, and omn represent the parameters of the full model.
view
representation
prodiction
input image
fe.,
9
maximize agrocment
T
10xx
0.5x
fim
9tv
ag
fnr
fa,e
(a) left: SlimCI right: slimmable network with two widths [1.0, 0.5).
representation
projection
prodiction
input image
fa
90.
9%
Fo.. 1
1P
maximize egreement
T
fen
4
sg
(b) SlimCLR-M -MMoo333
Figure 2: The overall framework of SlimCLR. A slimmable network produces different outputs from weight-sharing networks with various widths Wi,..., Wn. where wi is the width of the full model. the network parameters and are exponential moving average of 6. sg means stop-gree googggradeen.
3

---

Under review as a conference paper at ICLR 2023
We first illustrate the learning process of SlinCLR-MoCov2 in Figure 2a. Given set of images D, an image z sampled uniformly from D, and one distribution of image augmentation T. SlimCLR produces two data views ë t(x) and 722 (x)x) from by applying augmentations t ~ T and ~ T, respectively. For the first view, SlimCLR outputs multiple representations how,, 1- no.......ha... ho... and predictions 20 20u.. where ha.. = fe., (ă)) and 20,,, = 9a.., (how.)). 9 is a stack of slimmable linear transformation layers, i.e,, slimmable version of the MLP head in MoCov2 and SimCLR (Chen et al., 2020a). For the second view, SlimCLR only outputs a single representa- tion from the full model hew, fsw, (ãz) and prediction 2t,,, 95,,, (hgm,)) ). We minimize the InfoNCE (van den Oord et al., 2018) loss to maximize the similarity of positive pairs 20w, and Lgm,"
exp(zo.., _gm, /T1)
Lrow; =x/ - log
exp(zom.; EE,, /n) + Σ. exp(2o.; /m))
and {2- are features of negative samples. For SlimCLR-MoCowe { comes from a queue. Following
where 28..; = -o..:/ 1000 112, z = /ll=,,,, 1l.. TI is temperature
MoCov2, the queue is updated by Σen, every iteration during training. The overall objective is the sum of losses of all networks with various widths:
(20,20((=-) Σ Czem, =3)(=-)
Ç is updated by 0 every iteration: Ç < mE (1-mm),, where m E [0, 1) is momemtum coefficient.
Compared to M SlimCLR-MoCov3 has an additional projection process. It first projects the representation to another high dimensional space, then makes predictions. The projector q is stack of slimmable linear transformation layers. SlmmCL-MMoov3 also adopts the InfoNCE loss, but the negative samples come from other samples in the mini-batch.
After contrastive learning, we only keep fow, T.... Jo.. and abandon other components.
3.2 GRADIENT IMBALANCE AND SOLUTIONS
As shown in Figure 1, a vanilla implementation of the above framework leads to severe performance degradation as seight-sharing networks interfere with each other during pre-training. One evidence of such interference we observed is gradient imbalance.
Gradient imbalance refers to that small proportion of parameters produces dominant gradients during backpropagation. To quantitatively evaluate the phenomenon, we show the ratios of gradient norms main and minor parameters: ||V Cl12 and V/ Cl12 versus ||V LlI2 in Figure 3,
where the loss function. Meanwhile, the ratio of the numbers of parameters is |9,. |9.a\@o.2sl ss 10o.25l
15, where 01000.25 E 11.0 This means O1. (.0.25 contains more than 90% of the total parameters. Generally, the main parameters dominate the optimization process and produce large gradient norms, i.e., the two ratios should both be large (> 1). In Figure 3a, the two ratios are both around 3.5 when training a normal network. However, in Figure 3b and 3c, when training a slimmable network, gradient imbalance occurs because sharing parameters obtain multiple gradients from different losses. To be specific, if the widths of a slimmable network Wi,..., Wn arrange in descending order and the training loss is Lzo, Own, that only represent small part of parameters will receive gradients from n different losses and obtain a large gradient norm:
Own L20,)e,(E-) 0Lz..gi1== }
dL -ex(=-)
Σ
(3)
00 w.
00 wn
Gradient imbalance is more obvious in self-supervised cases. In the supervised case in Figure 3b, I/Vo,.o,..22 L close to e/V Lll2 at first, and the former becomes larger along with the training process. By contrast, for vanilla SlimCLR-MoCon other most time. A conjecture is that instance discrimination is harder than supervised classification. Consequently, small networks with limited capacity are hard to convergence, produce large losses, and cause more disturbances to other reight-sharing networks.
in Figure 3c, | [e, ove 25 ClI2 is smaller than the
The gradient directions of weight-sharing networks may also diverge from each other during back- propagation. This causes the gradient direction divergence of the full network during training.
4

---

Under review as a conference paper at ICLR 2023
4.0
Suprersiand, normal
1.6
Supwerssse slimmable
16
MoCCo vanilla baselin
144
14
of gr
19
1.0
4.5
3.
0.8
[P Alba Flb
[10 Clbb Fll
9.
05
1.5
20
Training step of 50
165
Training step of ResNet 50 2221
145
Troining stop of ResNet SesNett00rooo... 00
(a) 76.6%
(b) 76.0%, 72.9%, 64.4%
(c) 66.0%, 61.4%, 54.5%
SUinCLR MoCov2,
StanCLR
distil
S0mcLR MoCon2, distil reweight
Budio prodiist 231723
3.5
2.0
3.0
3.0
3.5
1.5
1.5
100
1.0
05
1.
1.5
2.3
0.5
1.3
2.5
1.0
Truning stop of Www.eeeeeN......oooo www.ee.................
167
Thiining step of Bww.eet-SS0osooneeee
162
Training step of BouNNek BaeNeet- (Q
(d) 66.3%. 62.2%. 55.1%
66.9%, 62.3%, 54.9%
(I) 67.0%, 62.4%, 10.%%,55,. 55.1%
Vera Wo Cl
Ve oro 25
The gradient norm of the network calculated by averaging the layer-wise le gradient norms. Vo, 0. C is the gradient of the final loss w.r.t parameters .1.0..25 E O1oo' 20.25, i.e., rest parameters of 91. besides 0..25
Figure 3: Ratios of gradient norms:
and
Vo Cll
Specifically, the gradient directions of the full network are unstable and change fast during training as shown in Figure in Appendix A.3.2.
To avoid gradient imbalance, one natural idea is to make the main parameters dominate the opti- mization process, i.e., the two ratios in Figure 3 should both be large. To resolve the conflicts of gradient directions, networks should have a consistent optimization goal. In order to achieve the above purposes, we develop three simple yet effective techniques during pre-training: slow start, online distillation, and loss reweighting. Besides, we further introduce switchable linear probe layer to avoid the interference of weight-sharing linear layers during linear evaluation.
slow start At the start of training, the model and pseudo supervision of contrastive learning are both fast updating. The optimization procedure is unstable. To avoid interference between weight- sharing networks making the situation harder, at the first S epochs, only update 01.0 by Vol o }. In Figure 3d, the ratios of gradient norms are large before the S-th epoch; then they dramatically become small after slow start. At the first epochs, the full model can learn certain knowledge from the data without disturbances, and sub-networks can inherit the knowledge via the sharing parameters and start with better initialization. Similar approaches are also adopted in some one-shot NAS methods (Cai al., 2020; Yu et 2020).
only train the full model, i.e.,
online distillation The full model has the greatest capacity to learn knowledge from the data. The prediction of the full model can serve as consistent guidance for all sub-networks to resolve the gradient direction conflicts of weight-sharing networks. Following Yu & Huang (2019b), minimize the Kulbback-Leibler (KL) divergence between the estimated probabilities pw: exp(Ze., -Temy /22)
exp(Fow: FEwy --- exp(3em; /r/) of sub-networks and the full model:
Cpe -Prr log Pwr where wi E {W2, ..,,..... W.
(4)
To is temperature coefficient of distillation. In Figure 3e, we observe that online distillation helps V 2..00.22 S VV.. Ll become larger than 1.0. This means that online distillation also relieves the gradient imbalance and helps the main parameters dominate the optimization process.
loss reweighting Another straightforward solution to gradient imbalance and gradient direction divergence is to assign large confidence to networks with large widths. We adopt a strategy in which
5

---

Under review as a conference paper at ICLR 2023
the strongest takes control. The weight for the loss of the network with width w, is:
X; = 1.0 + 1(2; = wi 3
(5)
where 1 equals if the inner condition true, O otherwise. In Figure 3f, both ratios become large, and [ o5o.25 Lll2/1 2o..as 2112 are larger than 1.0 by clear margin. Loss reweighting helps
the main parameters produce large gradient norms and dominate the optimization process.
The overall pre-training objective of SlimCLR is:
(Cahm =c -c + Lpmn))
Lall = Xu -2
(6)
2
switchable linear probe layer As we demonstrate theoretically in Appendix A.1, given the fea- tures extracted by a slimmable network which is pre-trained via contrastive self-supervised learning methods, single slimmable linear probe layer cannot achieve several complex mappings from dif- ferent representations to the same object classes simultaneously. The failure is because the learned representations in Figure 2 do not meet the requirement discussed in Appendix A.I. In this case, propose switchable linear probe layer mechanism. Namely, each network in the slimmable network will have its own linear probe layer for linear evaluation.
4 EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
Datatest We train SlimCLR on ImageNet (Russakovsky et al., 2015), which contains I. 1.28M train- ing and 50K validation images. During pre-training, we use training images without labels.
Learning strategies of SlimCLR-MoCor2 By default, learning rate 0.2, and weight decay 1 x 10-4, We adopt the SGD optimizer with momentum 0.9. A linear warm-up and cosine decay policy (Goyal et al., 2017; He et al., 2019) for learning rate is applied, and the warm-up epoch is 10. The temperatures are T1 = 0.2 for InfoNCE and 72 5.0 for online distillation. Without special mentions, other settings including data augmentations, queue size (65536), and feature dimension (128) are the same the counterparts of MoCov2 (Chen al., 2020c). The slow start epoch S of sub-networks set to be half of the number of total epochs.
total batch size 1024, initial
Learning strategies of SlimCLR-MoCoon 1.2, and weight decay learning rate policy with warm-up epoch 10. The temperatures start epoch is half of the total epochs. One different thing is that we increase the initial learning rate to 3.2 after S epochs. Pre-training is all done with mixed precision (Micikevicius al., 2018).
We use a total batch size 1024, an initial learning rate 10-6 We adopt the LARS (You al., 2017) optimizer and cosine
1.0 and 72 1.0. The slow
Linear evaluation Following the general linear evaluation protocol (Chen et al., 2020a; He et al., 2020), add new linear layers on the backbone and freeze the backbone during evaluation. We also apply online distillation with a temperature T2 = 1.0 when training these linear layers. For evaluation of SlomCIR-MoCov2.. rate 60, which is decayed by 10 at 60 and 80 epochs. For evaluation of total batch size 1024, epochs 90, and an initial learning rate 0.4 with cosine decay policy.
total batch size 1024, epochs 100, and initial learning we use
4.2 RESULTS OF SLIMCLR ON IMAGENET
Results of SlimCLR on ImageNet are shown in Table 1. Even though we pay huge efforts lieve the interference of weight-sharing networks as described in Section 3.2, slimmable training inevitably leads to drop in performance for the full model. When training for epochs, the degradation more obvious. However, such degradation also occurs the supervised case. Con- sidering the advantages of slimmable training will discuss below, the degradation is acceptable.
Compared to MoCov2 with individual networks, SlimCLR helps sub-networks achieve significant performance improvements. Specifically, for ResNet-500.s and ResNet-50 MimCI.R-MoCon

---

Under review as a conference paper at ICLR 2023
Method
Backbone
Teacher
Top-1
Top-5
Epochs
#Params
#FLOPs
R-50
93.2
100
G.1G
R-34
75.0
21.8
3.7
R-18
72.1
11.9
188
Supervised
R-50;.0
76.0,o
92.9
25.6'M
4.1GG
R- R-500.75
74.9
92.3
14.7 M
2.3
R R-50o.s
x
72.2
90.8
100
6.9 M
1.1 G
R-50p.23
64.4
86.0
2.0M
278 M
R-50
X
67.5
200
25.6
G.1
Baseline
R-50
67.2
87.8
25.6'M
4.1'G
(individual networks
R-50 R-500.7s
64.3
85.8
14.7 M
G.3
200
trained with MoCov2)
R-50p.5
58.9
82.2
6.9 M
1.1 G
R-50p.25
47.9
72.8
2.0M
278 M
MoCov2 (2020c, preprint)
R-50
71.1
800
MoCov3 (2021, ICCV)
R-50
72.8
300
SlimCIRR-Mooo22 SlimCLR-MoCor2
R-50,.0
x
(7.4
87.9
200
25.6
4.1
R-50
70.4:0.74)
89.6
800
SlinCLR-MoCov3
R-501.0
72.3
90.8
300
SEED (2021, ICLR)
R-34
R-50 (67.4)
58.5
82.6
200
DisCo (2022, BCCV)
R-34
R-50 (67.4)
62.5
85.4
200
BINGO (2022, ICLR)
R-34
R-50 (67.4)
63.5
85.7
200
21.8
G.7
SEED (2021, ICLR)
R-34
R-50x (77.3)
65.7
86.8
800
DisCo (2022, ECCV)
R-34
R-50x2 (77.3)
67.6
88.6
200
BINGO (2022, ICLR)
R-34
R-50x2 (77.3)
68.9
200
SlimCLR-MoCov2 MimoLL-MMooo2 SlinCLR-MoCov3
R-500.75
87.0
200
R-500.75
X
68.8
88.8
800
14.7 M
2.3
7-500.75
69.7
89.4
300
CompRess (2020, NeurIPS)
R-18
R-50 (71.1)
62.6
130
SEED (2021, ICLR)
R-18
R-50x2 (77.3)
DisCo (2022, ECCV)
R-18
R-50x2 (77.3)
86.8
200
BINGO (2022. ICLR)
R-18
R-50x2 (77.3)
65.5
87.0
200
11.9
186
SEED (,021, ICLR)
R-18
R-152 (74.1)
59.5
65.5
200
DisCo (2022, ECCV)
R-18
R-152 (74.1)
65.5
86.7
200
BINGO (2022, ICLR)
R-18
R-152 (74.1)
65.9
87,1
200
SlinCLR--MoCov
R-500.5
62.5
84.8
200
SlimCLR-MoCor2
R-500.5
X
65.6
87.2
800
6.9 M
LIG
SlimCIRRMMooo3
R-50p.5
67.6.46)
88.2
300
SlinCLR-Moooov SlinCLR-MoCov2 SlimCLR-MoCov2
R R-500.25
55.1
79.5
200
R-500.25
X
57.6
81.5
800
2.0
278 M
R-500.25
62.4
84.4
300
Linear evaluation results of SlimCLR with ResNVet-5011..00755...0000 1..0000....07...5.0000.. on ImageNet. Through only one-time pre-training, SlimCLR obtains multiple different small models without extra large teacher models. It also outperforms previous methods using ResNer as backbones. The per- formance degradation when training a slimmable network shown in cyan. The gaps between the self-supervised and supervised results are shown in orange. The smaller, the better.
Table 1:
achieves 3.5% and 6.6% improvements in performance when pre-training for 200 epochs, respec- tively. This verifies that sub-networks can inherit knowledge from the full model via sharing pa- rameters improve their generalization ability. We can also use more powerful contrastive learning framework to further boost the performance of sub-networks, i.e., SlimCLR-MoCon
Compared to previous methods that aim to distill the knowledge of large teacher models, sub- networks of ResNet-50p p...p.......5.0..233 achieve better performance with fewer parameters and FLOPs. SlimCLR also helps small models get closer performance to their supervised counterparts. Furthermore, SlimCLR does not need any additional training process of large teacher models, and all networks in SlimCIR els with various computation cost which are suitable for different devices. This demonstrates the superiority of adopting slimmable networks for contrastive learning to get pre-trained small models.
trained jointly. By only training for time, can get different mod-
4.3 DISCUSSION
In this section, will discuss the influences of different components in SlimCLR.

---

Under review as a conference paper at ICLR 2023
Model
Top-1
slimmable
Top-5
Top-1
switchable
Top-5
Model
Top-1
5=0
Top-5
Top-1
S = 100
Top-5
Model
MSE
ATKD
Top-1
DKD
KD
R-50,a o
86.8
R-50,.
66.7
87.5
R-50
66.9
66.4
66.8
R-50. p5
63.4
85.3
64.3
86.0
R-500.m
64.3
86.0
65.3
86.4
R-500 75
65.2
65.0
65. 1
65.3
R-50.s
59.6
62.9
61.3
84.1
R-50as
61.3
84.1
62.5
84,3
R-50us
62.4
62.2
62.3
62.6
25
53.0
79.1
2-50
79.1
54.9
R-500.as
55.2
54.7
(a) switchable linear probe layer
(b) slow start epoch S, 200 epochs
(c) online distillation loss choice
Model
3.0
4.0
Top-1
5.0
6.0
Model
(1)
(2)
Top-1
(3)
(4)
Model
200
300
Top- 1
100
300
R-50..
67.0
67.0
66.7
R-50Le
67.4
67.5
R-50,.s
70.2
70.0
70.1
R-50c 2:
65.2
65.3
65.3
65.2
R-50
65.5
65.7
68.9
65.8
R-500 75
68.3
68.6
68.8
68.4
R-50
62.4
62.6
62.6
62.4
R-50
62.5
62.2
62.6
62.4
R-50b
65.7
65.6
65.6
65.3
R-50
55.0
5488
55.0
R-500.8
55.1
54.4
R-50go5
57.8
57.5
57.3
(d) online distillation temperature
(c) loss reweighting
(f) slow start epoch S, 800 epochs
Table 2:
Ablation experiments with SlimCLR-MoCov2 on ImageNet. The experiment in a former
table serves as a baseline for the consequent table.
switchable linear probe layer The influence of the switchable linear probe layer is shown in Table 2a. A switchable linear probe layer brings significant improvements in accuracy compared to slimmable linear probe layer. For only one slimmable layer, the interference between weight-sharing linear layers is not unavoidable. It is also possible that the learned representations of pre-trained models do not meet the requirements as we discussed in Appendix A.1.
slow start and training time Experiments with and without slow start are shown in Table 2b. The pre-training time of SlimCLR-MoCov2 without and with slow start epoch S = 100 on 8 Tesla V100 GPUs are 45 and 33 hours, respectively. For reference, the pre-training time of MoCov2 with ResNet-50 is roughly 20 hours. Slow start largely reduces the pre-training time. It also avoids the interference between weight-sharing networks at the start stage of training and helps the system reach a steady point fast during optimization. Thus sub-networks can start with good initialization and achieve better performance. We also provide ablations of slow start epoch S when training for a longer time in Table 2f. Setting S to be half of the total epochs is a natural and appropriate choice.
online distillation Here we compare two classical distillation losses: mean-square-erro (MSE) and KL divergence (KD), and two other distillation losses from recent works: ATKD (Guo, 2022) and DKD (Zhao et al., 2022a). ATKD reduces the difference in sharpness between distributions of teacher and student to help the student better mimic the teacher. DKD decouples the classical knowledge distillation objective function (Hinton et al., 2015) into target class and non-target class knowledge distillation achieve more effective and flexible distillation. In Table 2c, we can see that these four distillation losses make trivial differences in our context.
Combining the results of distillation with pesNettp...0.5000 in Figure 3e, we find that distillation mainly improves the performance of the full model, and the improvements of sub-networks are relatively small. This violates the purpose of knowledge distillation: distill the knowledge of large models and improve the performance of small ones. This is possibly because sub-networks in a slimmable network already inherit the knowledge from the full model via the sharing weights, and feature distillation cannot help the sub-networks much in this case. The main function of online distillation in our context is to relieve the interference of sub-networks as shown in Figure 3c.
We also test the influence of different temperatures in online distillation, i.e., T2 related to Pw. in equation 4, and results are shown in Table 2d. Following classical KD (Hinton et al., 2015), we choose T2 {3.0, 4.0, 5.0, 6.03. The choices of temperatures make a trivial difference in our con- text. SEED (Fang et al., 2021) use a small temperature 0.01 for the teacher to get sharp distribution and temperature 0.2 for the student. BINGO (Xu et al., 2022) adopts single temperature 0.2. Their choices are quite different from ours, and SlimCLR is more robust to the choice of tempera- tures. We further provide an analysis of the influence of temperatures in Appendix A.2.
loss reweighting We compared four loss reweighting manners in Table 2e. They are
(1). A = 1.0 1.0+1(w; = wi1 }x x Σド Wj (3). A
(2). x 1.0 [w; = x max U2,...,,a,, wr
1.0 1-+
(4). X, Σイ [1.0+ Σi= "
8

---

Under review as a conference paper at ICLR 2023
pre-train
backbone
Ap
AP
AP
Apmk
AP
APm
R-50 o
37.4
59.6
40.5
34,9
56.5
37.3
R-500..7
36.7
58.7
39.3
34,3
55.4
36,1
supervised (Yu al., 2019)
R-500.
34.7
56.3
36.8
32,6
53.1
34,1
R-500.25
30.2
50.3
31.5
28.6
47.5
29.9
MoCo (He al., 2020)
R-50
38.5
58.9
42.0
35.1
55.9
37.7
SEED (Fang al., 2021)
R-34
38.4
57.0
41.0
33.3
53.6
35.4
BINGO (Xu al., 2022)
R-18
32.0
51.0
34.7
29.6
48.2
31.5
R-5010
38.2
59.8
41.6
35.1
56.5
37,5
R- 50..7
37.3
58.4
40.4
34,4
53.5
36,8
SlimCLR-MoCovz
R 500.5
35.0
56.0
37.6
32.6
53.0
34.7
R- 50.
30.7
50.3
32.6
28.7
47.5
30.5
Table 3: Transfer learning results on COCO va12017 set. Bounding AP (APhr) for object detection and mask AP (Apmk) for instance segmentation.
where 1(- equals to 1 if the inner condition is true, otherwise. The corresponding weights of ResNet-50 1.0.0..70.0 20..0.......00.... [1.54, 1.08, 0.77, 0.62]. It is clear that larger weight for the full model helps the system achieve better performance. This demonstrates again that it is important for the full model to lead the opti- mization direction during training. The differences of the above four loss reweighting strategies mainly reflected in the sub-networks with small sizes. To ensure the performance the smallest network, adopt the reweighting manner (I) in practice.
[2.5, 1.0, 1.0, 1.0), [1.75, 1.0, 1.0, 1.0), [1.6, 1.2, 0.8, 0.4). and
Transfer learning to object detection and instance segmentation Following previous works (He al., 2020; Fang al., 2021; Xu al., 2022), we evaluate the generalization ability of SlimCLR on object detection and instance segmentation tasks. As the supervised slimmable networks (Yu al., 2019), we use Mask R-CNN (He et al., 2017) with FPN (Lin al., 2017) for the two tasks. We fine-tune parameters of all layers including batch normalization (loffe & Szegedy, 2015) end- to-end on COCO 2017 (Lin et al., 2014) dataset. The training schedule is the default 1x in Chen et al. (2019). The pre-trained backbone we used here is ResNet-50 7.......0.5...... 0.5.0.25) pre-trained via MimCLR-MoCove (800 epochs). The transfer learning results are shown in Table 3. SlimCLR- MoCov2 achieves better transfer learning results than the supervised baseline.
Differences of training slimmable networks self-supervised and supervised cases Ap- show that optimization process of training
pendix A.3, through extensive visualizations, slimmable networks is harder in self-supervised cases than that in supervised cases. Specifically, the gradient imbalance and gradient direction divergence both more significant in sulf--uper cases as we discussed in Section 3.2 and Appendix A.3.2. supervised cases, clear global su- pervision can help slimmable networks avoid these problems to some extent during training. In self-supervised learning, we do not have such clear global supervision, and we need to pay more effort deal with the performance degradation problems.
5 CONCLUSION
In this work, we adapt slimmable networks for contrastive learning to obtain pre-trained small mod- els in self-supervised manner. By using slimmable networks, we can pre-train for one time and get several models with different sizes which are suitable for various devices with different computation resources. Besides, unlike previous distillation based methods, our methods do not require additional training process of large teacher models. However, weight-sharing sub-networks in a slimmable network cause severe interference to each other self-supervised learning. One evidence of such interference we observed is the gradient imbalance in the backpropagation process. We develop several techniques to relieve the interference of weight-sharing networks during pre-training and linear evaluation. Two specific algorithms are instantiated in this work, i.e., HimmLL-MMCool and We take extensive experiments on ImageNet and achieve better performance than previous arts with fewer network parameters and FLOPPs

---

Under review as a conference paper at ICLR 2023
REFERENCES
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information across views. In NeurIPS, 2019.
Dana H. Ballard. Modular learning in neural networks. In AAAI, 1987.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In ICLR, 2020.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018.
Arnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric P Xing. Vision transformer slimming: Multi-dimension searching in continuous optimization space. In CVPR, 2022.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 1eto.09065..5 2019.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. In NeurIPS, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momen- tum contrastive learning. CoRR, abs/2003.04297, 2020c. URL httpttt/////xii...t.// abs/ 2003. 04297.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin A. Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. TPAMI, 2016.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. SEED:
sulf-supervised distillation for visual representation. In ICLR, 2021.
Yuting Gao, Jia-Xin Zhuang, Shaohui Lin, Hao Cheng, Xing Sun, Ke Li, and Chunhua Shen. Disco: Remedy self-supervised learning lightweight models with distilled contrastive learning. In ECCV, 2022.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: //www. www.dddwepppeeeeeeeeee
org.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training ima- genet in hour. CoRR, abs/ 120500076.0,,7,, 2017.
Jia Guo. Reducing the teacher-student gap via adaptive temperatures, 2022. URL https: // openreview. net/ forum? hit
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, pp- 2980-2988, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Tong He. Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In CVPR, 2019.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1 50300533,,, 2015, URL http: /arxiv. httg//bb///er /1lb//5503 02531.
R. Devon Hjelm, Alex Fedorov, Samuel Lavove-Marccileee Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), ICML, volume 37, pp. 448-44446 2015.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Compress: Self- supervised learning by compressing representations. In NeurIPS, 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Changlin Li, Guangrun Wang. Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. CVPR, 2021.
Changlin Li, Guangrun Wang. Bing Wang, Xiaodan Liang. Zhihui Li, and Xiaojun Chang. Ds- net++: Dynamic weight slicing for efficient inference in and transformers. T-PAMI, 2022.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In NeurIPS, 2018.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zimicc.. Microsoft COCO: common objects in context. In ECCV, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature pyramid networks for object detection. In CVPR, pp.
2017.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Zhaoyu Wang, Li Mian, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. CoRR, abs/2006.08218, 2020. URL https: / //arxivv.or org/ abaaab/aaa/ 2ttpb//2000.........
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn ing efficient convolutional networks through network slimming. In ICCV, 2017.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David Gareía, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In ICLR, 2018.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Alec Radford, Luke Metz, and Soumith Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. IJCV, 2015.
11

---

Under review as a conference paper at ICLR 2023
Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? CoRR, abs/2201.05119, 2022. URL httts:////axxi
org/abs/ 2201 05119.
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In NeurIPS, 2016a.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukeuoglu. Pixel recurrent neural networks. In ICML, 2016b.
Aäron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1 807 007.03348, 2018. URL http://axxxi www. aaaaaaas/1111 03748.
Wenxiao Wang. Minghao Chen, Shuai Zhao, Long Chen, Jinming Hu, Haifeng Liu, Deng Cai, Xiaofei He, and Wei Liu. Accelerate cnns from three dimensions: comprehensive pruning framework. In ICML, 2021.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, 2018.
Haohang Xu, Jiemin Fang, Xiaopeng Zhang, Lingxi Xie, Xinggang Wang. Wenrui Dai, Hongkai Xiong, and Qi Tian. Bag of instances aggregation boosts 2022.
distillation. In ICLR,
Taojiannan Yang, Sijie Zhu, Chen Chen, Shen Yan, Mi Zhang, and Andrew R. Willis. Mutualnet: Adaptive convnet via mutual learning from network width and resolution. In ECCV, 2020.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 100.:1700.0000 708888888 2017.
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers.
arXiv preprint arXiv: /903. /128, 2019a.
Jiahui Yu and Thomas S. Huang. Universally slimmable networks and improved training techniques. In ICCV, 2019b.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas S. Huang. Slimmable neural networks. In ICLR, 2019.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In ECCV, pp. 702-717. Springer, 2020.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, 2016.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In CVPR, 2022a.
Shuai Zhao, Liguang Zhou, Wenxiao Wang. Deng Cai, Tin Lun Lam, and Yangsheng Xu. Towards better accuracy-efficiency trade-offs: Divide and co- traininnnng. 2022b.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 CONDITIONS OF INPUTS GIVEN A SLIMMABLE LINEAR LAYER
We consider the conditions of imputs when only using one slimmable linear transformation layer, i.e. consider solving multiple multi-class linear regression problems with shared weights. The parame-
01
ters of the linear layer are E BdxC C' is the number of classes, where O
01 Ráxc
021
21 E Rd d1 + d2
The first input for the full model is X E RNxx where N is the number of samples, X [X11 X12). X11 E RNxd, X12 E NNdx, The second input X E RNxd is the input feature for the sub-model parameterized by O11.
Generally, we have N 2 d d1- We assume that both X and X, have independent columns, iee, X X and x X1 are invertible. The ground truth is T NxC
The prediction of the full model is Y X0, to minimize the sum-of-least-squares loss between prediction and ground truth, get
f arg min
X0 T
0
By setting the derivative w.r.f. to O, we get
0 = (xxxx) XTT.
(8)
In the same way, get
011 = (x X²T.
(9)
For X X, we have
x X11
X12)" pn X12] =
X X11
X12 X
12
(10)
X X
Bn B12
We denote the inverse of X X is matrix, its inverse is also symmetric, B B For
B
X XB 1, as X is a symmetric
Bu B
X XB = we have
X Xu x
X12
Bu
B12
Idi Odz.
Od, da I
(11)
X X
B2
Br
Then we can get
Idi
(12)
X X = Od., da
(13)
+ +
Od., d2
(14)
x X Idz
(15)
At the same time
0 (XTX) -1 XTT = TT
BBI B12
[ X12)" T
B B
[BIXT
B
12
T.
(16)
BB XT X 12
13

---

Under review as a conference paper at ICLR 2023
and
011
(Bixiii + B12X12) T= (xf - XFT.
(17)
From equation 14, we get
B21 = - (x (12)
X X 112 B11,
(18)
B12 = - BB1 x X12 (12)
(19)
Substitute equation 18 into equation 12, we get
X -
X 12
(X 1
122 111 Ids:
(20)
B1 ( 112 (x x12
(21)
At the same time
On = (B x + B12X
= BB1 [x] - X X ( X12) -1 x72) T
(X x (x
x12X11) ) (I X X12
(x
-1 X T. (22)
Combining equation 17 and equation 22, get the condition of the input
x
X12
x 12)
x1)x) (
(XT XT 12 1 -1
X T
= (x
X (23)
To verify whether equation 23 meets in practice, we sample 2048 images from the training set of ImageNet and use a ResNet-50 0.000,55000.0007700.0..0.. pre-trained by SlimCLR-MoCov2 (800 epochs) to extract the features of these images. The features from ResNet-501.0 denote X E 20048x 1024 and features from ResNet-50p.s denote X, E 20088251, We use L to represent the left side of equation 23 and R for the right side. Then we use the extracted features to get the absolute difference between L and R. ie.. L R/. The average value entries in L - R is 1.07. This performed the validation set of ImageNet. The average value of entries in L R is 0.88. This means a total difference 903094. 19.
means total difference 1096665.50 Similar experiments
These results demonstrate that features of slimmable networks learned by contrastive self- supervised learning cannot meet the input conditions (equation 23) when using a single slimmable linear probe layer. This explains why using switchable linear probe layer achieves much better performance than a single slimmable linear probe layer in Table 2a.
A.2 INFLUENCE OF TEMPERATURES DURING DISTILLATION
In this section, we will analyze the influence of temperatures when applying distillation. One of the previous methods SEED (Fang al., 2021) uses different temperatures for the student and teacher, without loss of generalization, we will also adopt a such strategy our analysis. Specifically, we adopt 7, for teacher and т, for student.
The predicted probability for a certain category the student is 4 the output of the student model, i.e., logit of the model. The probability for a certain category of e the teacher is p: E, e where is the output of the student model. The loss is the KL.
Σj 75 where is
divergence:
C
Σ
pk log qa.
(24)
14

---

Under review as a conference paper at ICLR 2023
The gradient of 9 w.r.t. is:
/ e eti/T 15
1 e /
-
(q: - 9.))
(25)
E, 6 03//71)
Ta
ōq:
0 e / Tegei
1
-9-
(26)
0z:
[S eti /
2
T9
Similarly,
0p: 1
(pi pt)
(27)
O% Tt
Op: PiPt.
(28)
Do;
The gradient of L w.r.t z is:
Oc
ps 0q.
Pt 0qi
+ Σ
(29)
0z,
qi 0z,
qe Oz,
t/i
Tg (1 - pt)
(30)
1
6 1.
(31)
To E,
e To Σ, 6 60s1 1
Following classical KD (Hinton et al., 2015), we assume temperatures are much larger than logits and use the first-order Taylor series to approximate the exponential function:
ac Uz:
1+ C
1+ mh
~c
T
Σ
C+
(32)
Supuwrstoed, neemal
Supswrstend dimmable
SlimC! B-MoCred venille brawline
Avarage f1 worm grraiieda
76.0%
030
66.0%
2.5
Plecas
3.
Bizeas
0.0
BLovass
0.10
3.0
6.0
1.2 Troming stop of Bestict 50c. u.s.u.c.... A251
Training stop Res 50"
3x0
Tratning stop of Restiet 50c...s 223
145
(a) 76.6%
(b) 76.0%, 72.9%, 64.4%
(c) 66.0%, 61.4%, 54.5%
SImCLR MoCon2, doo start
SiImCLR MoCov2, disetl
SlmCLR MoCow2, slowe start distill
Avcrge L1 sugno of graadiett
0.30
0.30
66.9% 6yzs,
0:50
0.2
R025-55..
0.22
A0gh
6: wi2s
0.23
0.25
0.13
0.13
0.15
0.00
Treining step of ResNeet500
105
Training step of BesNeet-00g
1:5
Training step of Reslee-50g
(d) 66.3%. 62.2%. 55.1%
(e) 66.9%, 62.3%, 54.9%
(I) 67.0%. 6 55.1%
Figure 4: The average gradient norm. We calculate the & norm of gradients layer-wise and get their mean. 01.0 11.0 and 00 Oo.25 represent parameters of the full model and sub-network with width 0.25. 01.0,0.25 +1.0' )0 denote the rest parameters θ1.o besides O0.25- is normal that the gradient norm increases during training (Goodfellow et al., 2016).
15

---

Under review as a conference paper at ICLR 2023
where C' is the number of classes. Following classical KD (Hinton al., 2015), further assume Ej 2z =L, Ui = O, we can get:
ac
2j
FM
(33)
dz
CT3
TB
7t
A.3 VISUALIZATION
In this section, we provide more detailed visualization and explanations for gradient imbalance, gradient direction divergence, and optimization trajectory during training. These visualization can help readers better understand how our methods work. Plus, they are also helpful for understanding the difference of training slimmable networks in supervised and self-super learning.
A.3.1 GRADIENT IMBALANCE
Besides the ratios of gradient norms in Figure 3, we also display the absolute values of gradient norms in Figure 4 to help readers better understand the gradient imbalance phenomenon.
A.3.2 GRADIENT DIRECTION DIVERGENCE
Besides the imbalance of gradient magnitudes, the gradient directions of different weight-sharing networks also have conflicts with each other. Such conflicts result in disordered gradient directions of the full model, and we call the phenomenon gradient direction divergence in this work.
Seperteed capllar
Sirvct MaGre2 larpur sheniie valase
McCoo] the acrnd aarvet dhguir
MincLL&MoOoes dietiiation
Sinncl MeCrnd dietlittion
3.0
0.0
U Wmm.mm
0.
88
0.0
30
05
20
05.
Training zup S1aas
Transing stap Stgpaann
Thussnng stop 50gaaa EAN
(a) singular values
(b) largest singular values
(c) second largest singular values
Supervised
SümCLLB-MaCove vanlilla baseline
SimCLR MoCov2, distillatons loss reweight
Kerci Comminaton
Port d +
Te comp Pomiir
Trusiig stap
Trasseg Hay
Hap
Supervised
SlinCLB MB-Ma vanlilla
MlleCLB-MoC22 datilation
Vol..
Trating wtw.eeeeessess
(d) Principal directions of gradients (e) Principal directions of gradients (f) Principal directions of gradients
Figure 5: The gradient direction divergence. (a) (b) (c) The singular values the full gradient matrix of the last linear layer. (c) (d) (e) The principal directions of gradients of the last linear layer.
16

---

Under review as a conference paper at ICLR 2023
We show the phenomenon when training slimmable networks on ImageNet in Figure 5, supervised and sulf-supervised. In Figure Sa&5&&5c, the largest and second largest singular values of the full gradient matrix of the last linear layer in the model are displayed. In Figure 5b, the network has small eigenvalues without slimmable training, namely, no interference from other weight-sharing networks. In Figure 5b&5c, after the slow start point, we can see that training of weight-sharing net- works dramatically increase the singular values. In supervised case (Figure Sa), training of weight- sharing networks also results in large singular values.
Training of weight-sharing networks will make the directions of full gradients disordered, and such phenomenon is more serious in self-supervised learning. We visualize the gradient directions in Figure 5d&5&&ff. Specifically, we collect the gradients of weights of the last linear layer during training; after training, we perform PCA on these gradients and show their projections on the first two principal components (Li et al., 2018). In Figure 5e, we can see that the gradient directions are stable and consistent during training before slow start. However, after slow start, gradient directions become disordered due to conflicts of weight-sharing networks during training. In Figure 5d, training of weight-sharing networks also makes the gradient directions disordered. However, as the networks have the same global supervision, the gradient direction divergence is less obvious compared to Figure Se. In Figure 5f, our proposed distillation and loss reweighting techniques effectively solve the divergence of gradient directions and make the training process stable.
A.3.3 ERROR SURFACE AND OPTIMIZATION TRAJECTORY
The performance degradation caused by the interference between weight-sharing networks is more severe in self-supervised learning compared to supervised learning. This is because the gradient imbalance and gradient direction divergence are both more significant in self-supervised cases. In this subsection, we further provide the the error surface and optimization trajectory (Li et al., 2018) when training a slimmable networks to re-emphasize the point: the optimization process of a slimmable network is harder in self-supervised cases than that in supervised cases.
We train slimmable networks in both supervised and self-supervised (MoCo (He et al., 2020)) man- ners on CIFAR-I0 (Krizhevsky & Hinton, 2009). The base network is a ResNet-20x4, which has 4.3M parameters. We train the model for 100 epochs. At the end of each epoch, we save the weights of the full model and calculate the Top-1 accuracy. For supf-sseee cases, we use k-NN predic- tor (Wu et al., 2018 8) to obtain the accuracy. After training, we calculate the principle components of the differences of the saved weights at each epoch and the weights of the final model following Li et al. (2018). Then we use the first two principle components as the directions to plot the error surface and optimization trajectory in Figure 6.
The visualization shows that self-supervised learning is harder than supervised learning. In the left error surface Figure 6a and Figure 6c, we can see that the terrain around the valley is flat in supervised cases; by contrast, the terrain around the valley is more complicated in self-supervised cases. From the trajectory of ResNet-20xx4 in the left of Figure 6b and Figure 6d, the contours in supervised cases are denser, i.e., the nearby two contours supervised cases costs more time to achieve the same improvement of accuracy compared to the model in supervised cases (the gaps between two contour lines are all the same). In supervised cases, clear global guidance helps the model quickly reach the global minima. In self-supervised cases, it harder for the model to reach the global minima fast without such global guidance.
closer. Namely, the model in self-
The visualization shows that the interference of weight-sharing networks is more significant in self- supervised cases. First of all, in self-supervised cases, weight-sharing networks bring huge changes to the eтror surface in Figure 6c. In contrast, the change is not so obvious in supervised cases. Second, the interference between weight-sharing networks in self-supervised cases makes the model shift more away from the global minima (the origin in the visualization) as shown in Figure 6d. In Figure 6b, the maximal offsets from the global minima along the 2nd PCA component are 21.75 and 28.49 for ResNet-20x4 and ResNet-20 2 o.... The offset increased 31.0%. For self-supervised cases in Figure 6d, the maximal offsets from the global minima along the 2nd PCA component 13.26 and 18.75 for 2e 4 and 20xNet-20x4q..uu
The offset increased 41.4%. It is clear that the interference of weight-sharing networks is more significant in self-supervised cases compared to supervised cases.
17

---

Under review as a conference paper at ICLR 2023
80
70
10
80
40
53
40
32
40
20
20
10
-90
-10
surface (supervised). left: R-20x4, 94.07%; right: R-20x4 4-20x411.0.0 (1.00.55 93.58%, 93.03%
2nd PCA component
20-
2nd PCA component
20-
-30
-20
1st PCA component
1st PCA component
(b) trajectory (supervised), left: R-20x4, 94.07%; right: p.20x4p.0,0.6)) 93.58% 93.03%.
30
50
30
43
10
-15
(c) error surface (MoCo). left: R-20x4, 76.75%; right: 4-20x4p.0.0.3 75.74%, 74.24%.
2nd PCA component
2nd PCA component
10-
-5-
15
15
-20
-20
1st PCA component
1st PCA component
(d) trajectory (MoCo). left: R-20x4, 76.75% right: 12200x4..0.0.s) 75.74%, 74.24%.
Figure 6: Visualization of the enror surface and optimization trajectory.
18

---

Under review as a conference paper at ICLR 2023
Model
74
5.0
F2
1.0
Model
1.2
2.8
3.2
3.6
Top-1
Top-5
Top-1
Top--
Top-1
R-501.0
69.0
89.0
69.2
89.1
R-50, .0
70.7
71.9
72.3
72.1
R-50a.rs
64.0
86.0
64.5
86.3
R-50p.75
66.8
69.4
69.6
69.8
R. 500.
85.0
50
68.0
R-50g.as
57.2
80.8
57.2
80.9
R- 8-500.a3
59.1
61.8
62.1
62.4
(a) temperature 72, 100 epochs
(b) Ir for slow start, 300 epochs
Table 4: Ablation experiments with SlimCLR- on ImageNet.
A.4 MORE IMPLEMENTATION DETAILS
Slimmable networks We adopt the implementation of slimmable networks described in Yu al. (2019), which has switchable batch normalization layers. Namely, each network in the slimmable network has its own independent batch normalization process.
SlimCLR-MoCon
We train SlinCLR-MoCov2 on 8 Tesla V100 32GB GPUs without synchro- nized batch normalization across GPUs. The momentum coefficient m is 0.999 during training.
SlimCLR-MoCors We train SlimCLR-MoCov3 on Tesla V100 32GB GPUs with synchronized batch normalization across GPUs Synchronized batch normalization is important for MoCov3 to obtain a better performance in linear evaluation. The momentum coefficient m is 0.99 with cosine schedule when training for 300 epochs. The data augmentations are the same as augmentations of MoCov3 Chen al. (2021).
Ablation study with SlimCLR-MoCov3 Besides the ablation studies with SlimCLR- we also provide empirical analysis for SlimCI.R-MoCov3. Different from SlimCLR-MoCove adopts different temperatures 71 1.0 and 72 = 1.0. When applying slow start, SlimCLR-MoCov3 also increases the initial learning rate at the same time.
In Table 4a, show the influence of temperature for online distillation. Different from the better for SlimCLR-MoCov? to choose a temperature T2 for online distillation, which is close to the temperature of contrastive loss. T1 = 1.0 is the default choice of MoCov3 (Chen et al., 2021), and we do not modify it.
choice in SlimCLR (MoCov2),
Another interesting phenomenon is that when the number of training epochs of MHimCLR-MoCoL becomes larger, we need to increase the learning rate (Ir) when we start to train the sub-networks. The influence of learning rate for slimmable training in SlimCLR-MoCovs is shown in Table 4b.
Here the learning rate refers to the base learning rate, the immediate learning rate after the warm- up is calculated by this base learning rate and the current training steps: 1/r(1 + cos (TT mux mux )) Different from SlimCLR-MoCove and MOmCCLR-MoCov with fewer epochs, SimCLR-MoCovv3 will get poor performance if we do not change the learning rate when training for more epochs. We attribute the difference to the LARS (You al., 2017) optimizer we adopt for SlimCLR- MoCov.. LARS normalizes the gradient of layers in the networks to avoid the imbalance of gradient mag- nitude across layers and ensures convergence when training networks with very large batch size. LARS is sensitive to the change in learning rate and helps self-supervised models training with large batches converge fast (Chen et al., 2021: 2020a). When training for the 300 epochs, the full model reach a local minima fast in the first 150 epochs. In this case, learning rate 0.6 (half of the base learning rate) is not able to help the system walk out of the valley and reach a better local minima. Consequently, large learning rate is needed to give the system more powerful momentum. From Table 4b, we can also see that MimmCL--MoCon with LARS is sensitive to the change in learning rate, This consistent with observations of previous works (Chen al., 2021; 2020a).
19


Reviews:
[{"review_id": "t5d7dWxrim", "reviewer": "Reviewer_yYC4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a slimmable contrastive self-supervised learning framework for building small models with SimCLR. It applied the well-known slimmable network and solved the gradient imbalance problem in the training by slow start training, online distillation, and loss reweighting, etc.\n", "summary_of_the_review": "The paper studied an interesting topic and solved it by applying an existing approach with some minor improvements on the training loss. The evaluation is not sufficient to justify the motivation and contributions.\n", "strength_and_weaknesses": "1. The proposed approach achieved balanced performance between multiple architecture sizes and outperformed the other small SSL models.\n2. The paper is well written and easy to follow.\n\nWeakness\n1. The novelty is limited as this paper simply applies the slimmable architecture with some minor improvements on the training loss.\n2. Given the paper target on training a small SSL model, it would be interesting to compare the performance with the small distilled model in terms of accuracy, model size, FLOPs. The current evaluation is not sufficient to justify the motivation.\n3. The evaluation can be improved by evaluating with other downstream datasets and tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rq9Y9b-h7_", "reviewer": "Reviewer_1XPr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies slimmable networks under the contrastive self-supervised learning (SlimCLR) setting. It proposes some strategies to solve the interference between weight-sharing sub-networks during training, including slow start training of sub-networks, online distillation, and loss re-weighting according to model sizes, and a switchable linear probe layer. SlimCLR is evaluated on ImageNet based on ResNet-50 using MoCo v2 and v3 frameworks.", "summary_of_the_review": "Please refer to the above.", "strength_and_weaknesses": "### Strength\n\nThis paper is a good study of the effect of slimmable networks for contrastive self-supervised learning. The empirical analysis has some contribution to the community.\n\n### Weakness\n\n**1: The technical novelty is limited and some details are confusing.**\n\nSlimmable networks are a special case of widely studied one-shot NAS (e.g., [R2, R3, R4, R5, R6]), which only considers the width dimension (see discussion in OFA [R2]). There are many techniques to deal with interference among subnetworks. Specifically,\n* “Slow start” belongs to progressive training in one-shot NAS. For example, OFA proposes a “Progressive Shrinking” strategy, which starts with training the largest sub-network and then progressively fine-tunes the network to support smaller sub-networks by gradually adding them into the sampling space.\n\n * “Online distillation” was originally proposed in US-Nets [Yu et al., 2019b]. Apart from the inplace distillation, it also proposes the sandwich rule.\n\n* “loss reweights” aims to assign larger weights for sub-networks with large widths. However, it violates the training objective of slimmable networks. The objective is to make each supported sub-network maintain the same level of accuracy as independently training a network with the same architectural configuration, rather than only training an accurate large “supernet”. This is evidenced in Table 2 (e), where adding loss reweighting makes R-50(0.25) perform worse, so what’s the meaning there?\n\n**2:  Another concern is what are the fundamental differences between self-supervised and supervised training for slimmable networks?** This is not clear to me as all the training techniques used are common practices in supervised training.\n\n**3: What’s the relationship between unsupervised NAS (e.g., [R4, R5]), including the contrastive self-supervised one (e.g., [R6])?** \n\n4: In Page 8, the authors study 4 possible cases of loss reweighting and show the results in Table 2e. However, I find case (3) archives the best performance for most widths but the paper uses case (1) by default in Eq. (5). I disagree with the author's explanation that “To ensure the performance of the smallest network, we adopt the reweighting manner (1) in practice” as all sub-networks with different widths should be equally important. Otherwise, what’s the meaning of slimmable networks there? \n\t\t\t\t\n**5: The paper lacks mathematical modeling for the gradient divergence issue which leads to the optimization difficulty claimed by the authors.** I think there are only four possible widths and it is not difficult to analyze the gradient magnitude and directions using SGD with maths formulations. Also, some theoretical analysis on convergence is expected, even assuming a linear neural network is fine [R1].\n\n**6: The experiments are far from enough to justify the effectiveness of the proposed method.**\n\n* 6.1:  The results are merely based on the ResNet-50 backbone. However, I would like to see more ResNet backbones such as R-101 and R-152. More importantly, experiments on Vision Transformers, such as ViT-B in MoCo v3, must be included in the experiments.  \n\n* 6.2:  The paper only evaluates the representation quality using linear probing. However, it must evaluate transfer learning performance which is the standard practice in self-supervised learning (e.g., in MoCo v3). For example, experiments on dataset transfer and downstream tasks such as dense segmentation and detection on COCO and ADE20k are needed.  \n\n* 6.3: How about training the whole network (width 1.0) first then using network pruning (e.g., [R7]) to obtain small networks (width 0.25, 0.5, 0.75)? As this strategy can avoid the interference issue during training. \n\n* 6.4: It lacks comparisons with methods dealing with sub-network interference, such as switchable BN [Yu et al., 2019], sandwich rules [Yu et al., 2019b] and many others.\n\n**7:  The discussions and references in related work are far from enough.** There are few discussions with single-shot NAS and unsupervised NAS methods. In addition, as I point out in the technical novelty part, the differences and advantages with the related work must be discussed. \n\n**8:  Writing also needs to be improved.** \n\n* 8.1: What is the definition of the “main parameters” in the introduction? \n\n* 8.2: In Sec. 3.2, “..., where $L$ is the loss function”. It should be defined in Eq. (1) where it first appears.\n\n* 8.3: Many grammar issues. I only point out a few. “Slimmable neworks” in Sec. 2; “server performance degradation” in Sec. 3.2. \n\n9: In Sec. 3.2, authors argue that the two ratios in Figure 3 should be large enough. “In Figure 3f, ..., are larger than 1.0 by a clear margin”. It does provide a clear concept of how large is good enough. In my opinion, it also depends on the network architectures and self-supervised learning frameworks. So Figure 3 may not be statistically significant. \n\n\n**References:**\n\n[R1]: “On the optimization of Deep Networks: Implicit Acceleration by Overparameterization”, ICML 2018\n\n[R2]: “ONCE FOR ALL: TRAIN ONE NETWORK AND SPECIALIZE IT FOR EFFICIENT DEPLOYMENT”, ICLR 2020\n\n[R3]: “BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models”, ECCV 2020\n\n[R4]: “Are Labels Necessary for Neural Architecture Search?”, ECCV 2020\n\n[R5]: “Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?”, NeurIPS 2020\n\n[R6]: “Contrastive Self-supervised Neural Architecture Search”, Arxiv 2021\n\n[R7]: “Resrep: Lossless cnn pruning via decoupling remembering and forgetting”, CVPR 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "k4atcdIRCkI", "reviewer": "Reviewer_enLj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Motivated by the performance gap between self-supervised learning and its counter supervised model, this work provides a one-stage self-supervised small model pretraining protocol. The slimmable network idea is used to get the representations of one augmentation $x_1$. Weights are shared between the set of slimmable encoders (the small one shared with the larger one). All the outputs from slimmable encoders are used as anchors in InfoNCE. The sum of loss from different anchors is calculated eventually. However, this naive implementation will cause gradient imbalance. The work proposed updating the full encoder first, knowledge distillation from the full encoder to the sub-network, and loss reweighting to mitigate the imbalance issue.  ", "summary_of_the_review": "Overall this is an interesting work even though the idea of slammable is not new and no theory is provided. I hope the author could address my questions in the rebuttal. I’m wondering whether a similar range of performance drops will happen on R-18 like on R-50. ", "strength_and_weaknesses": "An interesting work and clear motivation. \n\nHowever, I am not convinced about the gradient imbalance in learning slimmable networks. The measurement used in the paper seems not sufficient to me. Imagine that for each iteration the network weights are updated in a cyclic way, i.e., weights at (1,2,3) are updated in the first iteration, then (4,5,6), (7,8,9), (10,11,12) .... In this way the gradient norm ratio will be low as well, but the network can still be trained reasonably. Another question is since the paper mentioned Res18 in the motivation, is there a comparison of R-18 compared with R-18 distilled by R-50 teacher? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cO0twSNKy3", "reviewer": "Reviewer_5mjb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to learn slimmable neural networks with contrastive self-supervised learning without labels. To further improve the performance of sub-networks, this paper suggests using (i) slow-start training for sub-networks, (ii) online distillation, and (iii) loss reweighting. This increases the gradient norm of the main parameters, leading to performance improvements in the main network. This paper demonstrates that the main and sub-networks outperform existing two-stage distillation approaches.\n", "summary_of_the_review": "While learning slimmable neural networks is interesting, I feel the lack of methodological novelty and the weak motivation about gradient imbalance. In addition, this paper is hard to follow with confused notations. Hence I vote for rejection.\n", "strength_and_weaknesses": "Strengths\n- I think the main strength of this paper is that the proposed method outperforms other distillation approaches.\n\nWeaknesses\n- I feel the lack of methodological novelty. The idea of training slimmable neural networks is not new, and online distillation was already used in [1]. I also think other techniques are just engineering, and they have often been utilized in other literature: e.g., loss reweighting is a common strategy for multi-task learning.\n- Although this paper obtains some gains from the proposed techniques, the gains are marginal, and most of the gains come from using a large number of training epochs or MoCo-v3, not the techniques.\n- The training cost is linearly increasing with respect to the number of sub-networks. This could be critical since self-supervised learning is often time-consuming.\n- The motivation, the gradient imbalance, is really critical? I think the motivation is somewhat weak: the gradient imbalance can occur in any neural network layer because there exist dominant and minor neurons in any layer (e.g., think about singular value decomposition of weight matrix). Training sub-networks can be considered as determining the order of neuron importance. Note that the result in Fig (3a) is due to the random order of the neurons.\n  - Could you provide the distribution of singular values of the full weight matrix instead of gradient norms? I think it would be better to understand how training slimmable networks affect the weight matrix.\n- This paper is hard to follow: there are many confusing notations and descriptions.\n  - It would be better to use colors in Figure 1. In addition, it would be better to include the results of the proposed method in Figure 1 for readers.\n  - In general, notations should be defined before using them. For example, there is no description of ξ when using the parameter.\n  - Suggest to use $\\Theta_{w_i}=\\\\{\\theta_{w_1},\\ldots,\\theta_{w_i\\}\\\\}$ instead of writing $\\Theta_{w_j}\\subset\\Theta_{w_i}$ if $w_j<w_i$.\n  - In Eq (1), why $\\xi_1$ instead of $\\xi_i$? There is no explanation for this.\n  - In the self-supervised learning literature, the first MLP is often referred to as projection and the second MLP as prediction. I recommend following the conventional terminologies. For example, SlimCLR-MoCo-v2 should use projection instead of prediction. The current usage causes some confusion.\n  - What is $\\theta_{1.0\\setminus0.25}$?\n  - What are the main and minor parameters? And why are they main and minor?\n\n[1] Yu & Huang, Universally Slimmable Networks and Improved Training Techniques, 2019\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject