DSPNet: Towards Slimmable Pretrained Networks based on Discriminative Self-supervised Learning
Authors: Shaoru Wang
Zeming Li
Jin Gao
Weiming Hu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Self-supervised Learning
Dynamic Neural Networks
Knowledge Distillation

Abstract: Self-supervised learning (SSL) has achieved promising downstream performance. However, when facing various resource budgets in real-world applications, it costs a huge computation burden to pretrain multiple networks of various sizes one by one. In this paper, we propose Discriminative-SSL-based Slimmable Pretrained Networks (DSPNet), which can be trained once and then slimmed to multiple sub-networks of various sizes, each of which faithfully learns good representation and can serve as good initialization for downstream tasks with various resource budgets. Specifically, we extend the idea of slimmable networks to a discriminative SSL paradigm, by integrating SSL and knowledge distillation gracefully. We show comparable or improved performance of DSPNet on ImageNet to the networks individually pretrained one by one under the linear evaluation and semi-supervised evaluation protocols, while reducing large training cost. The pretrained models also generalize well on downstream detection and segmentation tasks. Code will be made public.

Under review as a conference paper at ICLR 2023
TOWARDS SLIMMABLE PRETRAINED NETWORKS BASED ON DISCRIMINATIVE SELF- SUPERVISED LEARNING
DSPNET:
Anonymous authors Paper under double-bling blind review
ABSTRACT
Self-supervised learning (SSL) has achieved promising downstream performance. However, when facing various resource budgets in real-world applications. it costs huge computation burden to pretrain multiple networks of various sizes one by Slimmable Pretrained
In this paper, propose Discriminative-SS--baaee Networks (DSPNet), which can be trained once and then slimmed to multiple sub- networks of various sizes, each of which faithfully learns good representation and can serve as good initialization for downstream tasks with various resource bud- gets. Specifically, we extend the idea of slimmable networks to discriminative SSL paradigm, by integrating SSL and knowledge distillation gracefully. We show comparable improved performance of DSPNet ImageNet to the networks in- dividually pretrained one by one under the linear evaluation and semi- supersseeee evaluation protocols, while reducing large training cost. The pretrained models also generalize well on downstream detection and segmentation tasks. Code will made public.
INTRODUCTION
Recently, self- self-supervise supervied learning (SSL) draws much attention to researchers where good represen- tations are learned without dependencies on manual annotations. Such representations are consid- ered to suffer less human bias and enjoy better transferability to downstream tasks. Generally, SSI. solves a well-designed pretext task, such as image colorization (Zhang al., 2016). jigsaw puzzle solving (Noroozi & Favaro 2016). instance discrimination (Dosovitskiy et al. 2014), and masked image modeling (Bao et al. 2021 According to the different types of pretext tasks, SSL methods be categorized into generative approaches and discriminative approaches. Discriminative ap- proaches received more interest during the past few years, especially the ones following the instance discrimination pretext tasks (He et al. 2020 [Chnn et al. 2020aa Grill et al.) [2020) Chen et al. 2020bb]
Caron et al. 2020 CChnn & He 2021). These pretraining approaches have shown superiority over their Measeeet-superriiee counterpart in multiple downstream tasks. However, the pretraining is time-consuming and costs large computing resources, e.g., the pretraining of BYOL costs over 4000 TPU hours on the Cloud TPU v3 cores.
In real-world applications, the resource budgets vary in wide range for the practical deployment of deep networks. single trained network cannot achieve optimal accuracy-efffciency trade-offss across different devices. It means that we usually need multiple networks of different sizes. A naive solution apply pretraining and also meet such conditions to pretrain them one by one and then fine-tune them to accomplish specific downstream tasks. However, their pretraining cost grows approximately linearly as the number of desired networks increases. It costs so many computing resources that this naive solution is far from being practical way make use of the favorable self-supervised pretraining.
In this paper, feasible approach is proposed to address this problem developing slimmable pre- trained networks that can be trained once and then slimmed to multiple sub-networks of different sizes, each of which learn good representation and serve good initialization for downstream tasks with various resource budgets. To this end, we take inspiration from the idea of slimmable networks Yuu et al. 2018), which can be trained once and executed different scale, and permit

---

Under review as a conference paper at ICLR 2023
the network FLOPs to be dynamically configurable at runtime. Nonetheless, slimmable networks 2018) and the subsequent works (Yu & Huang 2019 Yu et al.) 2020, Cai et al. 2019 YYu et al.
Yang et al. )020) all focus specific tasks with supervision from manual annotations, while we concentrate on SSI.-based representation learning without manual annotations. To bridge this gap, build our approach upon BYOL. (Grill et al.] (020), representative discriminative SSL method. Specifically, as multiple networks of different sizes with good representations need to be pretrained one go, we thus construct them within network family that are built with the same basic building blocks but with different widths and depths. We use desired networks (DNs) to denote them. Differ- ent from the original design in BYOL that the online and target networks share the same architecture at each training iteration, our online network is specifically deployed by activating the randomly sampled sub-networks in it during pretraining, which include all the above desired networks. We also perform BYOL' similarity loss between the target branch and the sampled sub-networks in the online branch, and the target network is also updated by exponential moving averages of the online network. After pretraining, DNs can be slimmed from the online network and each of them learns good representation. In this way. DNs are pretrained together by sharing weights, which can reduce large training cost compared with pretraining them one by one. We name the proposed ap- proach as Discriminaive-SSL-baaa Slimmable Pretrained Networks (DSPNet). Our contributions summarized as follows:
We propose DSPNet, which can be pretrained once with SSL and then slimmed to multiple sub-networks of various sizes, each of which learns good representation and serves as good initialization for downstream tasks with various resource budgets.
We show that our slimmed pretrained networks achieve comparable or improved perfor- mance to individually pretrained ones under various evaluation protocols, while large train- ing cost is reduced.
With extensive experiments. show that DSPNet also performs par better than previous distillation- -ased SSL methods, which can only obtain a single network with good representation by once training.
2 RELATED WORK
Self-supervised Learning. Recent self-supervised learning (SSL) approaches have shown promi- nent results by pretraining the models on ImageNet (Deng et al., 2009) and transferring them to downstream tasks. It largely reduces the performance gap or even surpasses with respect to the su- pervised models, especially when adopting large encoders. Generally, SSL. solves a well-designed pretext task. According to the types of pretext tasks, SSL methods can be categorized into gen- crative approaches and discriminative approaches. Generative approaches focus on reconstructing original data (Pathak et al. 2016 Bao et al.) 2021 He et al. 2021 Zhang et al.) 2016). Discrimi- native approaches have drawn much attention in recent years, especially those based on the instance discrimination pretext tasks (Dosovitskiy al., 2014), which consider each image in dataset as its own class. Among them, contrastive learning (Hadsell et al., (006) methods achieve more promis- ing performance, where the representations of different views of the same image are brought closer (positive pairs) while the representations of views from different images (negative pairs) are pushed apart. YOL (Grill et al., 2020) further gets rid of negative pairs while preserving high perfor- mance, namely non-contrastive method. Most of the above methods rely on large batch size and long training schedule, which cost large computing resources.
Slimmable Networks. Slimmable networks are a class of networks executable at different scales, which permit the network FLOPs to be dynamically configurable at runtime and enable the cus- tomers to trade off between accuracy and latency while deploying deep networks. The original ver- sion YYu et al., (0188) achieves networks slimmable different widths, and US-Nets (Yu & Huang. 2019) further extends slimmable networks to be executed at arbitrary widths, and also proposes two improved training techniques, namely the sandwich rule and inplace distillation. The subsequent works go beyond only changing the network width, e.g., additionally changing the network depth (Cai et al. 2019), kernel size (Yu et al. 2020) and input resolutions (Yang et al. 2020). However, previous works all focus specific tasks with supervision from manual annotations. Our approach extends them to SSL-based representation learning, to obtain slimmable pretrained models by once training.
2

---

Under review as a conference paper at ICLR 2023
Self-supervised Learning with Distillation. Some research focuses applying the aforemen- tioned self-supervised methods to train lightweight networks, while the experimental results in Fang et al. (2020) show that the naive SSL methods do not work well for lightweight networks. To ad- dress this problem, knowledge distillation (Hinton et al. 2013) is adopted by introducing a larger self-supervised trained teacher to help the lightweight networks learn good representations. For instance. CompRess (Abbasi Koohpayegani et al., 2020) and SEED (Fang et al. (020) employ knowledge distillation to improve the self-supervised visual representation learning capability of small models, relying on MoCo (Chen et al. 2020b) framework. A more flexible framework is proposed Gao et al. (202I) with contrastive learning adopted transfer teachers' knowledge to smaller students. However, these two-stage methods all rely on the pretrained teachers and keep them frozen during distilling, which may be tedious and inefficient. Recently, some methods adopt online distillation (Bhat et al., [021] Choi et al., 2021). which recommends training the teacher and student at the same time. Our approach bears some similarity to them in that we also conduct SSL and knowledge distillation simultaneously. However, works of different sizes by training once to meet the demand of the practical deployment of deep networks under various resource budgets. Moreover, it can reduce large training cost.
method is aimed to obtain multiple net-
3 PRELIMINARIES
In this section, the representative discriminative SSL methods BYOL (Grill et al. 2020) and slimmable networks are firstly summarized as preliminaries for our method.
3.1 BYOL
BYOL Grill et al. (2020) consists of two paralleled branches, namely online and target branches. Given image from datasets, two augmented views and v' are obtained by applying different image augmentations respectively. From the first view v, the online network parameterized by first outputs a representation ye fo( and then a projection 29 - go(y). and the target network parameterized by & outputs a representation V/g & fe(w') and the projection =t & (e('') from the second view v'. A predictor 4x parametered by x is applied to the online branch, and mean squared error adopted as prediction loss:
Lo,gx = (xx &
where (zo) A 4x (x(=))/llx<<=))l and E/6 A =<//1z/[l2 are the normalized version. Usually, symmetric loss Le,s,x is also computed by separately feeding v' the online network and the target network. The parameters of the online branch and are finally updated by backpropagation from the total loss LBYOL= L T.o.,x- and the parameters of the target branch & are updated as an exponential moving average (EMA) of
3.2 SLIMMABLE NETWORKS
Slimmable networks Yu et al. 2018) are a class of networks that can be executable at different scales. As shown in Fig. [ by dropping specific blocks or inactivating parts of channels, one network can be slimmed to smaller networks.... Usually, single trained network can only be deployed at the full scale, while slimmable networks can be deployed as any one of the pre-defined sub-networks to accomplish specific tasks. To train slimmable networks, a practical and cost-effective strategy is proposed, namely sandwich rule. During training, only the smallest, largest and few randomly sampled sub-networks are used to calculate the loss in each iteration. Besides, implace distillation Yuu & Huang 2019) is also a useful training technique, in which the knowledge inside the largest network is transferred to sub-networks by simply adding distillation loss between them.
METHOD
Our goal is to obtain multiple networks of various sizes within network family by training once with SSL, all of which have good representations after optimizing. We achieve it by integrating
3

---

Under review as a conference paper at ICLR 2023
Depth
Width
Deploy at depth
Deploy at full width
Deploy at partial depth
Deploy at partial width
By dropping specific blocks (Left) and inactivating parts of channels (Right), network be slimmed to a smaller sub-network. show them separately for clarity, while they could he applied together.
Figure 1:
vo'
Target Encoder
Projector
Stop Gradient
Target Branch
x
EMA
Similarity
1
Slimmable Online Encoder
Projector
Predictor
A8
Online Branch
Discriminative-SSL--S Slimmable Pretrained Networks (DSPNet)
Figure 2: Architecture of our DSPNet. Based on BYOL (Grill et al. 2020), our online encoder is deployed by activating the randomly sampled sub- sub-networks in it at each training iteration, while the target encoder is always deployed at full depth and width. After pretraining, the optimized online encoder be slimmed to multiple networks different sizes, and of them have good representations.
BYOL (Grill et al. 2020) and slimmable networks (Yu et al. 2018). In this section, we intro- duce the architecture and training procedure of our Discriminative- baaeb Slimmable Pretrained Networks (DSPNet).
4.1 DSPNET
The overall architecture of our approach, namely DSPNet, shown in Fig. It largely follows BYOL (Grill et al. 2020), except for the encoder of the online branch. Unlike the original design BYOL, which the target encoder itself the desired network with good representations after training, our desired networks are several slices of the online encoder"] They are sub-networks by removing specific channels and blocks from the whole online network, referred to desired networks (DNs) in this paper.
To train DSPNet, we also adopt the sandwich rule Yu et al. 2018). Specifically, at each training iteration, we sample n sub-networks, involving the smallest, largest and randomly sampled (n-2) sub-networks, then calculate the loss for each them, and finally apply gradients back-propagated from the accumulated loss of them. We illustrate the construction of the online encoder and its behavior during pretraining in Fig. 3 The projector and predictor along with the loss function all follow the original design in BYOL (Grill et al., 2020). except for the first linear layer in the projector of the online branch, because should have an alterable input channel to adapt to the output representations with different dimensions from the online encoder. As for the target branch, the encoder has the same architecture as the whole online network and is always deployed at full width (i.e., all channels are activated) and depth (i.e., all blocks are maintained).
More specifically, for one sampled online network, which is defined by a set of weights 0, we denote encoder as fo F, and then the projection of the representation from the online encoder is calculated as
20 go (fo(v))
'Since the target network is obtained by exponential moving average of the online network, it should con- verge to the same status the online encoder. In our experiments, we find that the representations of the sliced networks from the target and online hranches achieve comparable results.
4

---

Under review as a conference paper at ICLR 2023
Smallest DN
share parameters
slim
Randomly
sampled DNs
Whole Network
Desired Networks (DNs) Slimmable Online Encoder
Largest DN
Figure 3: Construction of the online encoder and its behavior during pretraining in DSPNet.
for the first view where go is the projector. It is noteworthy that if the whole online encoder is defined as fd parameterized by 0*", then the parameters of any sub-network are a subset of 0' For the second view v', the projection of the representation from the encoder of the target branch is calculated as
2
ge M
(fE (t')),
where f and M are the encoder and projector of the target branch parameterized by Ç. which are deployed at full size the same as the whole online network, ie., K |. Then, our prediction loss defined as
La,gx (xx)) - 1/
(4)
where qx is the predictor parameterized by in the online branch, and (20) and &g the nor- malized version of 9x (z)) and o/$-
We compute the loss by taking unweighted sum of all training losses of sampled networks:
Leggx Σ (xx 1
(5)
where θ - 01 U02 U...UUUUUU. O,, and 0, is the parameters of the i-th sampled online network. Since the largest sub-network (the whole one) is always sampled in the online branch following the sandwich rule, we can find that è 8M, We also symmetrize the loss in Eq. (5) by interchanging and of to DSPNet
compute T.gg ,,.x following the practice in BYOL. The total loss 7 "0 _x I.gM ...x + Tigg, ..x"
0M along with X are updated based on the gradients from backpropagation, and & is updated by momentum-baat based moving average of BM with hyper-yppeeee parameter т, i.e.,
DSPNet) MM X f- optimizer(O" X. VBMx LgM
(6)
ş - T + (1 M
Since we can accumulate buck-propagated gradients of all sampled networks and do not need to always keep all intermediate results, the training procedure is memory-efficient with no more GPU memory cost than pretraining individual networks. Moreover, at each training iteration, & is shared for all sampled online encoders, which means it is also time-efficient. After pretraining, the desired networks (DNs) can be slimmed from the whole online encoder, all of which are expected to have good representations.
4.2 BEHAVIORS BEHIND THE TRAINING OF DSPNET
Overall, our online encoder varies within the predefined DNs during pretraining. When the whole network is sampled, our training behaves like BYOL (Grill et al. 2020). It is essential for our ap- proach achieve good representations, While for the other sampled pled sub-networks, the supervision is built upon the similarity between the representations from the target branch (with the same size as the whole online network) and this partially activated online network, which can be viewed as knowledge distillation so that the representation capability is transferred and compressed to the sub- networks. is noteworthy that, different from the implace distillation of slimmable networks (Yu) & Huang 2019), our teacher is the target network rather than the whole online network, which is a momentum based moving average of the whole online network and severed as a stable and re- liable teacher. In this view, training is jointly optimized towards self-supervised learning and knowledge distillation, and thus contributes to simultaneous representation learning and model com- pression.

---

Under review as a conference paper at ICLR 2023
Train Once
$2.99%
Total cost: 2.11x
00.95x
Topl Aee (96)
64
Topl (96)
58
0
01.33
Total cost: 4.42x
DSPNet
DSPNet Individually pretrained models
56
0.75%
Individually pretrained models
50
0.500
0.625
0.750
1.000
0.500
0.625
0.750
1.000
Widths
Widths
Figure Topl accuracy on ImageNet un- der linear evaluation for DNs slimmed from our DSPNet and individually pretrained models. Relative training time is also reported.
Figure 5: Semi-supervised learning on ImageNet using 10% training examples for DNs slimmed from our DSPNet and individually pretrained models. Top1 accuracy (%) is reported.
EXPERIMENTS
We train our DSPNet on ImagetNet ILSVRC-2012 (Dngg et al. 2009) without labels. Then we evaluate the representations of our obtained DNs on the validation set of ImageNet, including linear evaluation protocol and semi-supervised evaluation protocol. The transferability to the downstream tasks, such as object detection and instance segmentation on COCO (Lin et al. 2014), is also inves- tigated to study the generalization.
5.1 COMPARISONS WITH INDIVIDUALLY PRETRAINED NETWORKS
One of the advantages of our method that we can obtain multiple desired networks (DNs) by training once, which is cost-efficient compared with individually pretraining them one by one. Thus, experimental comparisons with individually pretrained networks are carried out in this section.
Implementation Details. We mainly follow the settings in BYOL (Grill al. 2020). The same set of image augmentations as in B BYO (Grill et al. 2020) is adopted. Our DNs are MobileNet v3 (Howard et 2019) with different widths, i.e., 0.5, 0.625, 0.75, 0.875, 1.0] x. The projector and predictor are the same as in BYOL (Grill et al. 2020). We also adopt momentum BN (Li et al.) 2021 for further accelerating. We use the LARS You al. 2017) optimizer with a cosine decay learning rate schedule (Loshchilov & Hutter 2016) and 10-epoch warm-up period. We set the base learning rate 0.2, scaled linearly (Goyal et al. (017)) with the batch size (i.e., 0.2 BatchSize/256). The weight decay is 1.5 10-6, The EMA parameter starts from Thuse. = 0.996 and is gradually increased to 1.0 during training. We train for 300 epochs with a mini-batch size 4096. The number of sampled sub-networks per iteration is 4. Besides, to deployed at full scale at the first warm-up epochs, for robust training. As our baselines, individually pretrain these models (MobileNet v3 at widths [0.5, 0.625, 0.75, 0.875, 1.0) x) with BYOL (Grill et al. 2020). The same training recipe as ours is adopted.
constrain the online encoder
Linear Evaluation on ImageNet. Following the ImageNet (Deng al., 2009) linear evaluation protocol (Grill et al. 2020) as detailed in Appendix A.2 we train linear classifiers top of the frozen representations with the full training set, and then report the top1 accuracy on the validation set. Note that the classifier is trained individually for each DN, As shown in Fig. 4 method achieves comparable performance to the individually pretrained models. Specifically, for smaller DNs, our method achieves minor improvement, while showing slightly inferior results only for the largest one.
Furthermore, evaluate the efficiency of our method, report the relative pretraining time w.r.t. the individually pretrained MobileNet v3 at width 1x, which is also shown in Fig. 4 is evaluated on 16 V100 32G GPUs. The pretraining of our DSPNet spends 2. 2.11. x time w.r.t. the above baseline,

---

Under review as a conference paper at ICLR 2023
37
34
30
33
35
32
10AP
34
MAP
31
-33
30
32
31
DSPNet
29
DSPNes Individually protrained models
Individually pretrsined models
0.500
0.625
0.750
0.875
1,000
0.500
0.625
0.750
0.875
1,000
Widths
Widths
Figure 6: COCO object detection (Left) and segmentation (Right) based on Mask R-CNN. mAP is reported for both DNs slimmed from our DSPNet and individually pretrained models.
68
68
Topl Aee (5)
Topl Ane (%)
62
60
60
DSP (n=3)
DSPNet (3 DNs)
68
DSPNet (n-4)
68
DSPNet (5 DNs)
DSPNet (m=5)
DSPNet DNs)
Indivicually pretesined modele
Individually prstrained models
0.750
0.750
0.875
1.000
Widths
Widths
Figure 7: Ablation study under linear evaluation. We examine the effect of the number of sampled sub- networks per iteration (Left), and the number of pre-defined DNs (Right).
while it totally costs x.42x time to pretrain all the DNs one by one. About a half of training cost can be reduced by our method.
Semi-supervised Evaluation on ImageNet. We also evaluate the representations on a classifica- tion task with small subset of ImageNet's training set available for fine-tuning them, i.e., semi- supervised evaluation, as shown Fig. 5 We fine-tune the pretrained models by only utilizing 10% of the labeled training data, including both multiple DNs slimmed from our DSPNet and individu- ally pretrained models. Similar phenomena are observed as in linear evaluation. Higher accuracy than individually pretrained models is achieved except for the largest DN. More details can be found in Appendix A.3
Transferability to Other Downstream Tasks. We use Mask R-CNN (He et al. 2017) with FPN (Lin et al. 2017) to examine whether our representations generalize well beyond classification tasks (see Appendix|AA for more details). We fine-tune on COCO train2017 set and evaluate on va/2017 set. All the model parameters are learnable. We follow the typical 2x training strategy (Wu al.) 2019). As shown in Fig. 6 our bounding box AP and mask AP both outperform the individually pretrained counterparts for most models.
5.2 ABLATION STUDY
We now explore the effects of the hyper-parament of our method. For concise ablations, all exper- iments are conducted based on MobileNet v3 (Howard et al. 2019) and the pretrained models are evaluated under the linear evaluation protocol.
Number of Sampled Sub-networks Per Iteration n. We first study the number of sampled sub- networks per training iteration. It is worth noting that larger n leads to more training time. We train three kinds of DSPNet models with n equal to 3, 4, or 5. As shown in the left of Fig. 7 the models pretrained with n 4 achieve better results than 3 under linear evaluation, but further increasing to n = does not lead to significant improvement. Thus, we adopt n = in all our experiments by default.
7

---

Under review as a conference paper at ICLR 2023
Number of DNs. We then investigate the effect of the number of DNs. The more DNs. the more pretrained models can be obtained by once training. As shown in the right of Fig. 7 we have pretrained 3-switch, 5-switch, and 1-switch DSPNet. They all show similar performance, demon- strating the scalability of our method.
5.3 APPLYING TO OTHER NETWORKS.
Our method is generally applicable to most represen- tative network architectures. We further evaluate our method on ResNet (He et al. 2016). We construct several DNs based the BasicBlock building blocks. Specifically, the channel numbers for the stages are sampled from {[64, 128, 256, 512), [80, 160, 320, 640), [96, 192, 384, 768] and the block numbers for the stages are sampled from {[2, 2,, 2], 32,3,4.3,
Topl Aor (%)
66
64
DSPNNe
Individuxlyy pretrained models
[3, 4, 6, } resulting in 9 networks as DNs. The small- est one among them is the well-known ResNet18, and the largest one is ResNet34 at width 1.5x. The topl accu- racy of linear evaluation w.r.t their parameter numbers is shown in Fig. 8 The performance of these networks individually pretrained with BYOL (Grill et al. 2020) also evaluated. The same training config- urations as those of MobileNet v3 are adopted. It can be drawn that for most smaller models, DNs slimmed from the whole network outperforms the individually pretrained ones, while falling behind for few larger ones. We attribute the conjecture that the whole online network makes a compromise between slimmablity and good representation.
10
20
30
40
50
Parumeter Number (M)
Figure 8: Applying DSPNet to ResNet.
5.4 COMPARISONS WITH SSL METHODS WITH DISTILLATION
We notice that there are some works (Fang et al., 2020 Abbasi Koohpayegami et al. 2020; Gao et al,,
2021 focusing the self-supervised training of lightweight networks with knowle loowledge distillation. From some perspectives, our method can be viewed as an online distillation method by training the teacher (the target network) and students (sub-networks of the online network) simultaneously. Thus we compare our method with them by only picking the smallest DN slimmed from our DSPNet. We directly use the models introduced in Sec. [.3 where the smallest DN is ResNet18.
First, we perform comparisons under the k-nearest--eiighbrr (kNN) classification and linear evalu- ation ImageNet (Deng et al., 2009). As shown in Tab. Tal we achieve better results than other distillation- based SSL methods (Fang et al. 2020 Abbasi Koohpayegani et al. 2020; Gao et al., 202T. It is worth noting that our method does not rely on well-pretrained teachers, and additional models with good representations can also be obtained at the same time.
Second, we evaluate the pretrained models under semi-supervised evaluation. Following previous works (Fang et al.] 2020), we train linear classifier on top of the frozen representation by only utilizing 1% and 10% of the labeled ImageNet's training data. As shown in Tab. [bb our method out- performs previous approaches on 10% semi-supervised linear evaluation by large margin. Higher accuracy is achieved we further perform fine-tuning on this semi supeviupereised classification task. Details can be found in Appendix A.3
Finally, we transfer the pretrained models to downstream tasks, i.e., object detection and segmenta- tion tasks on COCO (Lin et al. 2014). We also use Mask R-CNN (He et al. 2017) in Sec. 5. by only replacing the backbone with ResNet18. As shown in Tab. Ic our bounding box AP and mask AP both outperform the distillation-based methods. Our method also shows comparable results to the supervised baseline (Supervised-IN, entry in Tab. Ic). which relies on the labels of ImageNet dataset (Deng al. 2009).
5.5 DISCUSSION: WHY THE LARGER DNS SHOW INFERIOR RESULTS?
In Sec. 5.1 observe that the larger DNs show slightly inferior results to the individually pre- trained counterparts under linear evaluation (Fig, 4. semi-supervised evaluation (Fig. (5) and down- stream detection and segmentation tasks (Fig. Here, we provide one possible explanation for the
8

---

Under review as a conference paper at ICLR 2023
Teacher
Student (ResNer18)
Methods
Methods
Top1
kNN
Topl
Top5
Supervised
69.5
Self-supervised
McCo-v2
36.7
52.2
77.6
BYOL 15
50.2
60.8
83.6
Sell-supervised with Distillation
SEED
R50 (MoCo-v2 +200e)
67.4
43.4
57.9
82.0
R (SwAV1 +400e)
77.3
55.3
63.0
84.9
CompRess
R50 (MoCo-v2 8 (8000e)
70.8
53.5
62.6
R50 ASwAV (5 +800e)
75.6
56.0
65.6
DisCo 13
R50 (MoCo-v2) (2lo-22122000 +200ee
67.4
60.6
83.7
R (SwAVt (5++000)
77.3
65.2
86.8
Ours
expanded
56.1
65.9
86.4
(a) KNN and linear evaluation comparisons on ImageNet. R50 short ResNet50
He et al. 2020).
2016). and
R50x2 represents a R50 at width 2x. i indicates using multi-crop strategy (Caron et al.
Methods
1 %
10%
Methods
COCO Obj. Det.
COCO Inst. Segm.
(ResNet18)
(ResNet18)
AP AP
AP%
AP"M AP%
AP75
Top1
Top5
Top1
Top5
Random Init. 34.7
54.3
37.7
31.8
51.2
34.0
MoCo-v2
8
31.1
54.5
47.2
73.1
Supervised 37.1 57.3
40.0
34.0
54.3
36.4
BYOL 15
42.0
67.4
56.1
81.4
MoCo-v2 8
36.6
56.7
39.7
33.4
53.8
35.8
SEED 12
43.5
54.9
BYOL 15
36.8
57.1
39.7
33.6
54.2
35.7
DisCo
13
47.5
54.6
SEED 12
37.1
57.6
40.0
34,0
54.5
36.3
Ours Ours (fine-tuning)
47.1
71.8
59.7
82.0
CompRess
36.9
57.3
39.8
33.7
54.3
36.0
49.2
73.7
62.5
83.6
Ours
37.2 $7.8
40.4
34.1
54.9
36.3
(b) Semi-sun semi-supervised evaluation on ImageNet. Top1 and Top5 accuracy reported.
(c) COCO Lin et al., 2014) object detection and segmentation based on Mask L-CNN al. (2017).
Table 1: Comparisons with SSL methods with distillation. We compare the smallest DN slimmed from our DSPNet with the models trained by SSL methods with distillation.
above observations by investigating the working mechanism of our DSPNet. We first consider the individually pretrained baseline with MobileNet v3 at width 1.0x. Built upon that, the introducing of optimizing sampled sub-networks for the online encoder during the pretraining of our DSPNet additional regularization on the full-size online encoder. This urges the good
can be viewed representation to be compressed to the sub-networks. In other words, such training scheme makes compromise between slimmablity and good representation for the full-size online encoder. The former accounts for good imitation for the DNs to the whole one, and the latter for the qualified target network as a teacher, both of which contribute powerful DNs. Hence, smaller DNs benefit from the knowledge distillation from the teacher and achieve better performance than individually pretrained ones, while the largest one has to accommodate those DNs its sub-networks, leading to slightly inferior results in the context of missing explicit supervision in SSL. The experimental results in Fig. (Right) also support our conjecture: as the number of DNs increases, the regular- ization becomes stronger, and thus the larger DNs shows more inferior results. Nevertheless, though the only few large DNs are discarded considering their inferior performance, the others are also preferable for various downstream tasks, and our method still has advantage in the training cost. Besides, we also show that our DSPNet can sever as good initialization for dynamic neural networks in downstream tasks, e.g., Slimmable Networks eY al. 2018). in Appendix B
CONCLUSIONS
In this paper, we have proposed Discriminative-SSL-- Net), which can be trained once and then slimmed to multiple sub-networks of various sizes, and experimental results show that each of them learns good representation and can serve as good ini- tialization for downstream tasks, e.g., semi-supervised classification, detection, and segmentation tasks. Our method perfectly meets the practical use where multiple pretrained networks of different sizes demanded under various resource budgets, and large training cost be reduced.
Slimmable Pretrained Networks (DSP-

---

Under review as a conference paper at ICLR 2023
REFERENCES
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Compress: Self- supervised learning by compressing representations. Adv. Neural Inform. Process. Syst. 33: 12980-12992, 2020.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.0880 0825,4 2021.
Prashant Bhat, Elahe Arani, and Bahram Zonooz. Distill on the go: Online knowledge distillation in self-supervised learning. In IEEE Conf. Comput. Vis. Pattern Recog,, pp. 2678-2687, 2021. 2021.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In Int. Conf. Learn. Represent, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. Ady. Neural Inform. Process. Syst., 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Int. Conf. Machine Learning., pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE Conf.
Comput. Vis. Pattern Recog., pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.
Hee Min Choi, Hyoa Kang, and Dokwan Oh. Unsupervised representation transfer for small net- works: believe can distill on-the-fly. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wort- man Vaughan (eds.), Adv. Neural Inform. Process. Syst., 2021. URL.
net/Eorummiii Sor.iiidadyyyyeeeee......
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 248-255, Icce, 2009.
Alexey Dosovitskiy, Jost Tobias Springenherg. Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. Adv. Neural Inform. Process. Syst., 27:766-774, 2014.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. In Int. Conf. Learn. Represent., 2020.
Yuting Gao, Jia-Xin Zhuang, Ke Li, Hao Cheng, Xiaowei Guo, Feiyue Huang, Rongrong Ji, and Xing Sun. Disco: Remedy sule-super learning on lightweight models with distilled con- trastive learning. arXiv preprint arXiv:2104.09124 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint 1rXiv:1706.02677 2017.
Jean-Bastien Grill, Florian Strub. Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Pires, Zhaohan Guo, Mohammad Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. In Adv. Neural Inform. Process. Syst., 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In JEEE Conf. Comput. Vis. Pattern Recog., volume pp. 1735-1742, IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conf. Comput. Vis. Pattern Recog, pp. 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Int. Conf. Comput. Vis., pp. 2980-2988 IEEE, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. IEEE Conf. Comput. Vis. Pattern Recog., pp- 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:21 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint MXXiv:1533.023/11 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for mobilenetv3. In Int. Conf. Compul. Vis., October 2019.
Sergey loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Int. Conf. Machine Learning., pp. 448 456, 2015.
Zeming Li, Songtao Liu, and Jian Sun. Momentum' teacher: Momentum teacher with momentum statistics for self- supervupe learning. arXiv preprint arXiv:2101.07525, 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C LLwrence Zitnick. Microsoft coco: Common objects in context. In Eur. Conf. Comput, Vis., pp. 740-755. Springer, 2014.
Tsung-Yr Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 2117-2125, 2017.
Ily Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 100.:386,,, 2016.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Int. Conf. Machine Learning., 2010.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. Eur. Conf. Comput. Vis., pp. 69 64. Springer, 2016.
Deepak Pathak, Philipp Krahenbubl, Jeff Donahue. Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. IEEE Conf. Comput. Vis. Pattern Recog. pp. 2536- 2544, 2016.
Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: A large mini-batch object detector. In IEEE Conf. Comput. Vis. Pattern Recog,, pp- 6181-6189, 2018
Ross Wightman. Pytorch image models.
https://github.o wtw.:tthab..m//iiiii.ii. hoonn/wiigttaa// htt.g//httabbcccmm//iigtttnat/
gytorch-image-modee 2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
Taojiannan Yang, Sijie Zhu, Chen Chen, Shen Yan, Mi Zhang, and Andrew Willis. Mutualnet: Adaptive convnet via mutual learning from network width and resolution. In Eur. Conf. Comput. Vis., pp. 299-315. Springer, 2020.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arviv: 1708. 10t.::8888.88888 2017.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In Int. Conf. Comput. Vis., pp. 1803-1811, 2019.

---

Under review as a conference paper at ICLR 2023
Jiahui Yu, Linjie Yang. Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In Int. Conf. Learn. Represent., 2018.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoe Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. In Eur. Conf. Comput. Vis., pp. 702-717. Springer, 2020.
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. In Eur. Conf. Comput. Vis., pp. 649-666. Springer, 2016.
12


Reviews:
[{"review_id": "zaeLrg6cg5", "reviewer": "Reviewer_P1zK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to combine the slimmable networks with BYOL to get multiple pre-trained networks in one go during training.", "summary_of_the_review": "Due to the limited novelty and my suspicion of the practicability of the method, I vote for rejection.", "strength_and_weaknesses": "Pros:\n\nThe method is rather straightforward.\n\nCons:\n\n1. Limited novelty. This paper is just a straightforward application of slimmable networks with BYOL, namely, a combination of existing works (slimmable networks + BYOL).\n\n2. Improper conclusion. In `Sec. Introduction`, the author claims `each of which learns good representation`. However, according to the experimental results, the performance of the largest network is worse than the individually trained network. See Figure 4 and Figure 5. The phenomenon is more obvious in Figure 8, near 2% gap between the performance of DSPNet and individually trained network (ResNet-34).\n(2.1.) Besides, these individual results are provided by the authors. How about DSPNet compare with standard ResNet-50 trained individually?\n\n3. Why only show results of ResNet-18 in Sec. 5.4? ResNet-34 with DisCo is 68.1%. ResNet-34 with BINGO[1] is 69.1. For ResNet-18, BINGO[1] also get 65.9% top-1 accuracy on ImageNet. The performance of DSPNet cannot surpass previous distillation methods while DSPNet also adopts distillation during training.\n\n4. Combine point 2 and point 3, for large networks, DSPNet is worse than individually trained methods; for small networks, DSPNet is no better than previous distillation based methods. One advantage of DSPNet is that it can train for one go and get several networks. However, we have to accept its aforementioned disadvantages at the same time. Sometimes, a large drop in performance (~2% in Figure 8) is unacceptable, and it means we sacrifice a large model to obtain better small models. In this case, distillation based methods maybe more practical.\n\n5. Figure 4 vs. Figure 8, why the largest network drops more when using ResNet. Could the authors provide some explanations? In Figure 8, why ResNet-34 has nearly 50M parameters? A ResNet-50 only has 25M parameters. Please check the Figure 8.\n\n[1] BAG OF INSTANCES AGGREGATION BOOSTS SELF-SUPERVISED DISTILLATION. ICLR 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "loRfw8ZGC4", "reviewer": "Reviewer_NqU9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a self-supervised learning method to train a slimmable network. The self-supervised pre-trained model can be used to fit different computing resource for downstream tasks. The basic idea is to sample sub-networks following slimmable network and train these networks using a self-supervised learning method, e.g., BYOL. The proposed method is demonstrated to achieve comparable or better performance than individually self-supervised trained models, especially when the model is small.", "summary_of_the_review": "I have some concerns about the technical contributino of this paper, but I think showing the effectiveness of combining slimmable network with self-supervised learning can still be a good contribution. My main concern is about the experimental results as explained in the weakness part.", "strength_and_weaknesses": "Strength\n1. The idea of combining self-supervised learning method with slimmable networks is technically sound. \n2. The method achieves better performance than individually trained models.\n3. The paper is easy to follow.\n\nWeakness\n1. There may be limited technical novelty in the proposed method. It is basically training slimmable networks using a self-supervised learning (e.g., BYOL) method. But I think demonstrating its effectiveness is still good.\n2. The paper used BYOL, but the major backbone is MobileNet v3, which is not used in BYOL. I am not sure why the author didn't use the backbones used in BYOL (e.g., ResNet-50). I think a better way to demonstrate the effectiveness is following the same setting (both backbone and training hyperparameters) but just apply the slimmable training. The baseline results in the paper are much lower than those in the original BYOL, thus I am not very convinced.\n3. Fig. 1-3 show that we could sample sub-networks by width and depth, but the main results are all based on width sampling only. Why don't use both width and depth in the main results?\n4. I am not sure about the training cost shown in Fig.4. For example, for 0.5x width individually trained model, the time is 0.75x. But 0.5x width will reduce the computational cost of the model to around 0.25x.  I understand there could be some other time cost like data loading, but 0.75x still seems a lot to me for 0.5x width network.\n5. In Tab 1(a), I don't think the comparisons to SEED, CompRess and DisCo are fair because they are using different self-supervised learning method. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "d0xDTpA768", "reviewer": "Reviewer_xwkC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to train slimmable networks in the context of self-supervised learning, which can be trained once and then slimmed to multiple sub-networks of various sizes. The authors use BYOL as the baseline method and adopt sandwich rule and in-place distillation to sample & train sub-networks. Experimental results show that the performance of the largest model is lower compared to the separately trained model, and the performance of the small sub-networks can be improved. The authors also show that the pretrained models also generalize well on downstream detection and segmentation tasks.\n", "summary_of_the_review": "I agree with the motivation of this paper but my major concern is the technical novelty of this paper. I don't see any challenges when combining BYOL and slimmable networks, nor does the author seem to have made any technical tweaks and optimizations.\n\nIf the authors can address my concerns, I am willing to raise my score.\n", "strength_and_weaknesses": "Strength:\n1. The motivation of this paper is clear and the method is simple to follow.\n2. The experiments show some promising results.\n\nWeakness:\n1. I think this paper has limited technical novelty. From my perspective, this paper is just a direct combination of BYOL and Slimmable Network. On the basis of BYOL's loss, it is only necessary to introduce the sampling strategy in slimmable networks. The explanation given by the authors is that the difference is the use of a momentum teacher, but this is not the difference between self-supervised learning and supervised learning. There are also many self-supervised methods that do not require a momentum encoder. \n2. In Figure 1, the authors present sub-networks with different widths as well as different depths. However, from the implementation details in Section 5.1, it seems that only the width is changed, and the depth is not involved. Hence, I think Figure 1 is misleading.\n3. Why choose BYOL as your method, can other self-supervised methods work?\n4. Missing related work [1].\n[1] DATA: Domain-aware and task-aware self-supervised learning. CVPR2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MOj9uyC4Dm", "reviewer": "Reviewer_ssnC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes DSPNet which can be trained once and then slimmed to multiple sub-networks of various sizes (e.g., different widths and depths) suitable for various resource budgets. The experimental results show that each of these sub-networks learns relatively good representation compared to the individually pre-trained ones and can serve as initialization for downstream tasks, e.g., semi-supervised classification, detection, and segmentation tasks.", "summary_of_the_review": "The idea of using slimmable network for pre-training different networks in a single pipeline and in a self-supervised manner is novel based on my knowledge. I also would like to see the authors' answers to my concerns in the weaknesses section.", "strength_and_weaknesses": "Strengths:\n\n- In my opinion, self-supervised learning considering the resource budget and efficient network architectures is interesting and applicable in real-world scenarios.\n- The paper is well-written and easy to follow.\n- The idea of training self-supervised representation learning with slimmable network that can be trained once and then slimmed to multiple subnetworks with different resource budgets is novel.\n- The results show the effectiveness of the proposed approach in different downstream tasks.\n\nWeaknesses:\n\n1- In Table 1 (a), it would be interesting to see the results of one of the knowledge distillation-based approaches such as SEED with a teacher trained using BYOL strategy to be more comparable with the current approach.\n\n2- MobileNet_v3 is also another architecture that is hard to train individually in a self-supervised manner as mentioned in SEED and DisCo. While the authors did some efforts in reporting results on this architecture, it would be more interesting to see the results in comparison to the existing KD-based approaches such as SEED. For instance, in Fig. 4, the individual trained Mobilenet_v3 has an accuracy near 66% while in SEED or Disco it is a lot lower and when it has been trained using KD by considering ResNet50 as the teacher, it reaches ~65. I would like more clarification on the evaluations on MobileNet_v3.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject