Effective Self-supervised Pre-training on Low-compute Networks without Distillation
Authors: Fuwen Tan
Fatemeh Sadat Saleh
Brais Martinez
Venue: ICLR 2023 poster (2023)
Keywords: Self-supervised learning
Low-compute network

Abstract: Despite the impressive progress of self-supervised learning (SSL), its applicability to low-compute networks has received limited attention. Reported performance has trailed behind standard supervised pre-training by a large margin, barring self-supervised learning from making an impact on models that are deployed on device. Most prior works attribute this poor performance to the capacity bottleneck of the low-compute networks and opt to bypass the problem through the use of knowledge distillation (KD). In this work, we revisit SSL for efficient neural networks, taking a closer at what are the detrimental factors causing the practical limitations, and whether they are intrinsic to the self-supervised low-compute setting. We find that, contrary to accepted knowledge, there is no intrinsic architectural bottleneck, we diagnose that the performance bottleneck is related to the model complexity vs regularization strength trade-off. In particular, we start by empirically observing that the use of local views can have a dramatic impact on the effectiveness of the SSL methods. This hints at view sampling being one of the performance bottlenecks for SSL on low-capacity networks. We hypothesize that the view sampling strategy for large neural networks, which requires matching views in very diverse spatial scales and contexts, is too demanding for low-capacity architectures. We systematize the design of the view sampling mechanism, leading to a new training methodology that consistently improves the performance across different SSL methods (e.g. MoCo-v2, SwAV or DINO), different low-size networks (convolution-based networks, e.g. MobileNetV2, ResNet18, ResNet34 and vision transformer, e.g. ViT-Ti), and different tasks (linear probe, object detection, instance segmentation and semi-supervised learning). Our best models establish new state-of-the-art for SSL methods on low-compute networks despite not using a KD loss term. Code is publicly available at github.com/saic-fi/SSLight.

Published as a conference paper at ICLR 2023
EFFECTIVE SELF-SUPERVISED PRE-TRAINING ON Low- COMPUTE NETWORKS WITHOUT DISTILLATION
Fuwen Tan
Fatemeh Saleh
Brais Martinez Samsung Al Cambridge
Samsung AI Cambridge
Microsoft Research Cambridge
ABSTRACT
Despite the impressive progress of self-supervised learning (SSL), its applicability to low-compute networks has received limited attention. Reported performance has trailed behind standard supervised pre-training by a large margin, barring self- supervised learning from making an impact on models that are deployed on device. Most prior works attribute this poor performance to the capacity bottleneck of the low-compute networks and opt to bypass the problem through the use of knowledge distillation (KD). In this work, we revisit SSL for efficient neural networks, taking a closer look at what are the detrimental factors causing the practical limitations, and whether they are intrinsic to the self- self-super low-compute setting. We find that, contrary to accepted knowledge, there no intrinsic architectural bottleneck, we diagnose that the performance bottleneck is related to the model complexity vs reg- ularization strength trade-off. In particular, we start by empirically observing that the use of local views can have dramatic impact on the effectiveness of the SSL methods. This hints at view sampling being one of the performance bottlenecks for SSL on iow-capacity networks. We hypothesize that the view sampling strategy for large neural networks, which requires matching views in very diverse spatial scales and contexts, is too demanding for low low-capacity architectures. We systematize the design of the view sampling mechanism. leading to ology that consistently improves the performance across different SSL. methods (e.g. MoCo-v2, SWAV or DINO), different low-size networks (convolution- based networks, e.g. MobileNetV2, 1, ResNet34 and vision transformer, e.g. ViT-Ti), and different tasks (linear probe, object detection, instance segmentation and cemi-supervised learning). Our best models establish new state-of state-of-the-art for SSL methods on low-compute networks despite not using a KD loss term. Code is publicly available at github. com/ swwcs (eocc-f//sss fL/ssiig............
training method-
INTRODUCTION
In this work, we revisit self-supervised learning (SSL) for low-compute neural networks. Previous research has shown that applying SSL methods to low-compute architectures leads to comparatively poor performance (Fang al., 2021; Gao et al., 2022; Xu et al., 2022), i.e. there is a large performance gap between superveuee and self supf-vupe pre-training on low cow-compuee networks. For example, the linear probe supervised gap of MoCo-v2 (Chen al., 2020c) ImageNet1 K is 5.0% for ResNet50 (71.1% vs 76.1% 76)1%). while being 17.3% for ResNet18 (52.5% vs 69.8%) (Fang et al., 2021). More importantly, while SSL pre-training for large models often exceeds supervised pre-training on a variety of downstream tasks, that is not the case for low-compute networks. Most prior works attribute the poor performance the capacity bottleneck of the low-compute networks and resort to the use of knowledge distillation (Koohpayegani et al., 2020; Fang et al., 2021: Gao et al., 2022; Xu al., 2022; Navaneet al., 2021: Bhat al., 2021). While achieving significant gains over the stand-alone SSL models, distillation-basee approaches mask the problem rather than resolve it. The extra overhead of large teacher models also makes difficult to deploy these methods in resturce-restricted scenarios, c.g. on-device. In this work, we re-examine the performance of SSL low-compute pre-training, aiming to diagnose the potential bottleneck. We find that the performance gap could be largely filled by the training recipe introduced in the recent self-supervised works (Caron et al., 2020; 20 that leverages multiple views of the images.
Comparing multiple views of the same image is the fundamental operation in the latest self suff-supee models. For example, SimCLR (Chen et al., 2020a) learns to distinguish the positive and negative views by contrastive loss. SwAV (Caron et al., 2020) learns match the cluster assignments of

---

Published as a conference paper at ICLR 2023
the views. We revisit the process of creating and comparing the image views in prior works and observe that the configurations for high-capacity neural networks, e.g. ResNet500101/152 (He et al., 2016) and ViT (Dosovitskiy et al., 2021), are sub-optimal for low-capacity models as they typically lead to matching views in diverse spatial scales and contexts. For over-parameterizer networks, this may not be as challenging, as verified in our experiments (sec. 4.3), but could even be considered as a manner of regularization. For lightweight networks, it results in performance degradation. This reveals a potentially overlooked issue for self-supervised learning: the trade-off between the model complexity and the regularization strength. With these findings, we further perform a systematic exploration of what aspects of the view-creatin process lead to performnf self-supervised models in the context of low-compute networks. We benchmark and ablate our new training paradigm in a variety of settings with different model architectures (MobileNet-V V2 (Sandler et al., 2018), 8), ResNetl 8/34/50 (He et al., 2016), ViT-S/Ti (Dosovitskiy et 2021)). )), and different self-supervised signals (MoCo-v2 (Chen et al., 2020c), SwAV (Caron et al., 2020), DINO (Caron et al., 2021)). We report results downstream visual recognition tasks, c.g. semi-supervised visual recognition, object detection, instance segmentation. Our method outperforms the previous state-of-the-arr approaches despite not relying on knowledge distillation.
Our contributions are summarized as follows:
(1) We revisit SSL for low-compute pre-training and demonstrate that, contrary to prior belief, efficient networks can learn high quality visual representations from self-supervised signals alone, rather than relying knowledge distillation;
(2) We experimentally show that SSL low-compute pre- pre-training can benefit from a weaker self- supervised target that learns to match views in more comparable spatial scales and contexts, suggesting a potentially overlooked aspect in self-supervised learning that the pretext supervision should be adaptive to the network capacity:
(3) With a systematic exploration of the view sampling mechanism, our new training recipe consis- tently improves multiple self-supervised learning approaches (c.g. MoCo-v2, SwAV, DINO) on a wide spectrum of low-size networks, including both convolutional neural networks (e.g. MobileNetV2, ResNet18, ResNet34) and the vision transformer (e.g. ViT-Ti), even surpassing the oftte-of-theaast distillation-based approaches.
2 RELATED WORK
Self-supervised learning. The latest supervuue models typically rely on contrastive learning, consistency regularization, and masked image modeling. Contrastive approaches learn to pull together different views of the same image (positive pairs) and push apart the ones that correspond different images (negative pairs). In practice, these methods require large number of negative samples. SimCLR (Chen et al., 2020a) uses negative samples coexisting in the current batch, thus requiring large batches, and MoCo (He et al.. 2020) maintains queue of negative samples and a momentum encoder to improve the consistency of the queue. Other attempts show that visual representations can be learned without discriminating samples but instead matching different views of the same image. BYOL (Grill et al., 2020) and DINO (Caron et al., 2021) start from an augmented view of an image and train the online network (a. student) to predict the representation of another augmented view of the same image obtained from the target network (a.k.a teacher). In such approaches, the target (teacher) network is updated with slow-moving average of the online (student) network. SWAV (Caron et al., 2020) introduces an online clustering based approach that enforces the consistency between cluster assignments produced from different views of the same sample. Most recently, masked token prediction, originally developed for natural language processing, has been shown to be an effective pretext task for vision transformers. BEiT (Bao et al., 2022) adapts BERT (Devlin et al., 2019) 9) for visual recognition by predicting the visual words (Ramesh et al., 2021) of the masked patches. SimMIM (Xie et al., 2022) extends BEiT by reconstructing the masked pixels directly. MAE (He al., 2022) simplifies the pre-training pipeline by only encoding small set of visible patches.
Self-supervised learning for efficient networks. Recent works have shown that the performance of self-supervised pre-training for low-compute network architectures trails behind standard supervised pre- vre-training by large margin, barring self-supervised learning from making an impact on models that are deployed devices. One natural choice to address this problem is incorporating Knowledge
2

---

Published as a conference paper at ICLR 2023
Distillation (KD) (Hinton et al., 2015) to transfer knowledge from larger network (teacher) to the smaller architecture (student). SEED (Fang et al., 2021) and CompRess (Koohpayegani et al., 2020) transfer knowledge from the teacher in terms of similarity distributions over dynamically maintained queue. SimReg (Navaneet et al., 2021) and DisCo Gao et al. (2022) utilize extra MLP heads to transfer the knowledge from teacher model to student model by regression. BINGO Xu et al. (2022) proposes to leverage the teacher to group similar samples into "bags", which are then used to guide the optimization of the student. While these are reasonable design choices reduce the gap between supervised and self-supervised learning for low-compute architectures, the reason behind this gap still poorly understood and largely unexplored. Recently, an empirical study (Shi et al., 2022) seeks to interpret the behavior of self-supervised low-compute pre-training from the perspective of over-clustering, along with examining number of assumptions to alleviate this problem. Unlike Shi et al. (2022), we study instead from the angle of view sampling and achieve superior performance compared to the best methods with knowledge distillation.
3 REVISITING SELE-SUPERVISED LOW-COMPUTE TRAINING.
The main goal of this work is to diagnose the performance bottleneck of SSL methods when using lightweight networks, and to offer a solution to alleviate the problem.
3.1 BACKGROUND
We study representative SSL methods for the contrastive loss, MoCo-v2 (Chen et al., 2020c), clustering loss, SWAV (Caron et al., 2020) and feature matching loss, DINO (Caron et al., 2021), ),, as they all demonstrate strong performance these approaches is to match multiple augmented views of the same image, involving global views that cover broader contexts (i.e. sampled from a scale range of (0.14, 1.0)) and local views that focus on smaller regions (i.e. sampled from scale range of (0.05, 0.14)). The global and local views are typically resized different resolutions (i.e. 224 vs 96²). MoCo-v2 learns to align two global views of the same image while distinguishing views of different images from a queue. SwAV and DINO further make use of the local views by learning to match the cluster assignments (i.e. SwAV) or latent features (i.c. DINO) of the global and local views of an image. We start by re-examining the performance of these state-of state-of-the-an
large neural networks. One of the key operations within
the-art art SSL approaches in the absence of knowledge distillation loss.
3.2 PILOT EXPERIMENTS
We pre-train these methods on ImageNetIK (Russakovsky et al., 2015) using MobileNetV2 (Sandler al., 2018) as the back- bone. MoCo-v2 does not use multiple crops by default. Fol- lowing (Gansbeke et al., 2021), we re-implement a variant of MoCo-v2 with multiple crops, noted as MoCo-v2". All models are trained for 200 epochs with a batch size of 1024. In Ta- ble 1, we compare the performance of the SSL methods against the supervised pre-training, as well as the distillation-bas model SimReg (Navaneet et al., 2021), which is the current state-of-the-arr method. For the SSL models, we use the linear evaluation protocol (He et al., 2020).
Method
Top-1
Top-5
Supervised
71.9
90.3
with KD SimReg
69.1
two views MoCo-v2
53.8
77.6
multiple views
MoCo-v2*
60.6
83.3
SWAV
65.2
85.6
State-of-the-art self-supervised methods consis- tently underperform the supervised model and the distillation- based model by non-negligible margin. The performance gap between MoCo-v2 and the supervised model is the largest. This is also reported in previous literature (Fang et al., 2021; Xu et al., 2022). Incorporating multiple crops largely fills the gap, improving MoCo-v2 by 6.8% in the top-1 accuracy. While using multiple crops is reported to also boost the performance of large networks, the improvement on MobileNetV2 is more significant. SwAV and DINO further reduce the self-supervised and super- vised gap to 6.7% and 5.7% respectively. However, the distillation-bast approach achieves a top-1 accuracy of 69. 69.1%, which is 2.9% better than the best self-supervised model.
Discussion.
DINO
66.2
86.4
Table I: Performance of differ-
ent self-supervised methods on Mo- bileNetV2.
3

---

Published as a conference paper at ICLR 2023
67.0
65.0
Acpu
63.0
61.0
0.1
0.3
0.4
0.0
0.1
0.2
0.3
0.5
with set
with
Figure 1: Experiments on the scale ranges of the image crops. Left: performance of different Sy when Sj is set to 0.14, Right: performance different St when Sg is set to 0.3. The performance of the default setting is circled in Red. The best performance in each experiment is circled in Plum.
We can draw the following conclusions. For self-supervised learning low-compute networks 1) the use of multiple views has an oversized effect and 2) learning with knowledge distillation still outperforms the best SSL method by a wide margin, showing that lightweight networks can effectively learn the self-supervised knowledge given suitable supervisory signal. These two facts point to the optimization signal being the cause of the performance bottleneck.
3.3 MODEL COMPLEXITY VS REGULARIZATION STRENGTHI TRADE-OFF: FROM THE VIEW SAMPLING PERSPECTIVE
In conventional supervised learning there is a well-known direct relation between the amount of regularization required for optimal performance and the capacity of the network (Steiner et al., 2022). Small networks require less aggressive regularization and typically perform optimally under weaker data augmentation. In self-supervised learning, the regularization itself the optimization target. This poses the question of whether the underlying problem with self-supervised low-compute networks is not lack of capacity, but rather a lack of modulation of the augmentation strength. In the following we study whether this is the case, concluding in the affirmative.
We investigate the problem from the view sampling perspective as it is the fundamental operation in the latest SSL methods and due to the large performance boost seen in Sec. 1 when using local views. However, given the standard parametrization of the view generation, is difficult to disentangle the factors that make view matching challenging and, at the same time, interpretable. This is compounded by the use of local views. It is thus not clear how design experiments with controlled and progressive variation of the view-matching difficulty. To unearth the underlying factors, we observe that view matching becomes increasingly harder when (i) views sharing little support. This can be caused by (i.a) views representing different parts of the image (e.g. head vs legs of a dog) or (i.b) views having crop scale discrepancy (e.g. full dog vs head of dog); (ii) views having similar support, but different pixel scales (e.g. global and a local view of a dog at 224 x 224 and 96 x 96 pixels respectively). Note here that previous research suggests that neural networks have difficulties modeling patterns multiple pixel scales (Singh & Davis, 2018).
We design four different axes to explore the parametric space. In Sec.3.3.1, we focus on the relative crop scale explore the impact of view support. In Sec.3.3.2, we focus the relative pixel scale to explore the impact of the discrepancy in pixel-size. In Sec.3.3.3, we study the impact of the number of views, result in healthier supervisory signals. In Sec.3.3.4, we further examine ways to lower the impact of pairs without shared support through the modulation of the SSL loss function.
views mean higher likelihood of some pairs having good intersection and thus
Setting. We use DINO as it provides the best performance in Sec. 3.2. All models are pretrained on ImageNetIK with MobileNetV2 the backbone. We follow the same setting in Sec. 3.2 (see also Appendix Sec. A). For the evaluation, we reserve random subset of 50,000 images from the training set of ImageNetlK, i.e. 50 per category, and report the linear accuracy on this subset avoid over-fitting, only using the ImageNet1 validation set in the final experiments in Sec. 4.
3.3.1 RELATIVE CROP SCALE
We first examine the ranges the random scales for the global and local views. The global and local crops are created by sampling image regions from the scale ranges of (Sg. 1.0) and (0.05, S.). The
4

---

Published as a conference paper at ICLR 2023
Posel Scale
66.0
S:0.3
X
8-0.3
LC-08
2D4 224
3000 7400
0336
$4.0
8-0.3
S-0.3
LCm100
LG-128
8:0.14
LC-84
S-0.14
96.x
1801
0.34
300
S-0.14
8-0.14
LG-100
LC-128
8-0.14
LC-160
AW (PS))
(B)
Figure 2: Left: Pixel Scale, i.e., the relative area between the crop in the original image and its resized area. Right: different settings and resulting performance. It is clear from the graph that optimal performance is achieved when both local and global crops have similar PSs (mean(PS,/PS, 2 1.0).
crops then resized to 224 and 96. We hypothesize that view-matching difficulty depends an interplay between these two scales. To explore this, we examine the impact of the scale range for the global views by varying Sg while fixing S1 to 0. the default value, and then searching S with the obtained Sg. Here we decouple the search for Sg and S for simplicity.
Fig. confirms that training sensitive to the relative crop scale. Making Sg too large or small hurts performance. A larger Sg reduces the variance of the global crops. This makes the matching of global views trivial and increases the discrepancy between global and local views, making their matching challenging. On the other hand, values of Sg smaller than S may lead to the mismatch in pixel-scale between views: the two views the same scale are resized to different resolutions (see Sec. 3.3.2). The best performance is achieved by setting both Sg and S to [0.3, 0.4].
3.3.2 RELATIVE PIXEL SCALE
Even if two views have similar semantic meanings, they might not correspond to the same pixel scale. We define the pixel scale of view as PS final size
cropped size* which can be considered an indication the
"size of the pixel". For each image, the global views {g.} and local views {4, are resized to different resolutions (GC 224 and LC 96 respectively). Thus, the pixel scale of global view g (PSg) and a local view (PS,) are:
GC x GC Area(g)*
LC x LC Area(l)
PSg
PS
Fig. 2(A) provides an example.
The ratio PSg/PS /PS, indicates the discrepancy in pixel scale between the global and local views. We investigate if such discrepancy has any impact on the self-supervised training. To do this, we control PSp/PS, by varying the final resolution of the local view, LC. To avoid resizing views of the same scale to different resolutions, we keep S₂ = S2 S. We perform experiments with S equals to 0.14, which is the default value, and 0.3, the optimal value obtain in Sec.3.3.1.
Fig. 2(B) shows that matching views of different pixel scales results in inferior performance. Here, the optimal resolution for the local views (LC) is not golden value but depends on mean (PSg/PS)). which is the mean ratio of the pixel scales between the global and local views over the training set. With set either 0.14 0.3, the best accuracy is achieved when mean(PS,/PS)) 1.0, i.e. LC = 103 when S = 0.14, LC = 128 when = 0.3.
3.3.3 NUMBER OF LOCAL VIEWS
The relative crop and pixel scales play a fundamental role, yet they are randomly sampled at every iteration. Intuitively, could increase the number of views sampled to improve the likelihood of each image having at least some good pairs that would keep the supervisory signal healthy.
We perform experiments in different settings with different LC and However, shown in Fig.3 (left), increasing the number of local views leads to marginal improvements which also saturate quickly. We posit that randomly sampling local views may include redundant crops that provide no
5

---

Published as a conference paper at ICLR 2023
60.0
S-0.14, LC-66
69.0
S-0.14, LC-103
68.0
S-0.3, LC-120
68.0
ADDA
67.0
67.0
66.0
85.0
10
15
20
02
0.4
N2
Figure 3: Left: number of local views (z axis) vs performance (y axis). Performance improves for more local views but saturates quickly. Right: experiments of re-balancing Ly and Li by varying
extra knowledge. Given the extra overhead of more local views and the limited benefit, we maintain the default number of views.
3.3.4 RE-BALANCING THE LOSS
The loss function in SSL models with local views typically includes two terms: given Ng global views {g No and Nt local views {4) the loss computed as:
Cy + Pag + Pot
(2)
by = Eipg SL 9j) aggregates losses between global views (i.e. global-global pairs), Lt = Σn SL(g,, 4)) aggregates losses between global and local views (i.e. global-locas pairs), SL is the self-supervised loss, which could be contrastive loss, cluster loss, or feature matching loss, etc, Poo- = =N, (N,, - 1) the number of dlobal-global pairs, whereas Pat .2 Ni is the number of global-local pairs. As N, is usually smaller than N₁, Pog < 1. For example, with the default Pat
P. 0.167. In other words, the loss L. is dominated by L1- On the other hand, aligning global-local pairs is also more difficult than aligning global-global pairs, especially when the network capacity is low. Therefore, we re-balance the loss by re-weighting Lg and Lı:
N₂ 2, 2, N₁ 6. Pae
Cg (1-
L1 Pot
L = a. Pgo
a).
(3)
Here a is a hyper-paramett that re-weights the contributions of L, and Lt- Note that with the default numbers of views, the original formulation is equivalent setting 0.143. Fig.3 (right) depicts that re-balancing L, and L, improves the performance. The optimal a falls in the range [0.3, 0.4]. We believe using a further helps modulate the crucial trade-off between the strength of the self-supervision and the model capacity.
With the findings discussed, our best pre-trained model with Mobile achieves top-1 linear accuracy of 68.33 68.3% on the ImageNetIK K validation set, gain of 2. 1% over the DINO model, as shown in Table Note that the exploration not exhaustive as different axes may correlate. We conjecture that larger scale search in the design space could lead to further improvement.
DINO
DINO with new crop scales
DINO with new crop&pixel scales
Ours
Top-1(%) 1(%)
66.2
67.0 (+0.7)
67.9 (+1.7)
68.3 (+2.1)
Table 2: Experiments on the ImageNet1 validation set.
4 EVALUATION
We perform evaluations across three dimensions; i) different self-supervised methods; ii) different low-compute architectures; iii) different downstream tasks, and further compare our pre-trained

---

Published as a conference paper at ICLR 2023
model to the current cate-of-the-an methods. Our experiments follow the benchmarks set forth by prior works on the topic. However, we also include results for ViT-Ti and ViT-S (Dosovitskiy et al., 2021), which while they have not been covered in prior work we believe offer important data point. We re-use the best view sampling recipe from Sec. 3.3, found for DINO with MobileNetV2 through linear probing ImageNet K, and apply it to all architectures and SSL methods. We remark that the positive and consistent results show conclusively the generality of the findings. Dedicated studies for different models/tasks could potentially lead to further improvement.
4.1 IMPLEMENTATION
All models are pre-trained on ImageNetIK (Russakovsky et al., 2015) for 200 epochs. Follow- ing Caron et al. (2021), we use the LARS optimizer (You et al., 2017) for the convolution- networks, and A (Loshchilov & Hutter, 2019) for the vision transformers. We use a batch size of 1024 and linear learning rate warm-up in the first 10 epochs. After the warm-up, the learning rate decays with a cosine schedule (Loshchilov Hutter, 2017). Most of the other hyper-pan parameerrsssers are inherited from the original literature. We provide further details in Appendix Sec. A.
4.2 EXPERIMENTS ON DIFFERENT SELF-SUPERVISED METHODS.
We start by verifying if the new training paradigm could transferred to different self- supervised approaches. Here we perform experi- ments on representative methods as discussed in Sec.3.2: MoCo-v2" (Chen et al., 2020c), SWAV (Caron et al., 2020), and DINO (Caron al., 2021). The visual backhone used in these experiments is MobileNetV2. As reported in Ta- ble 3, all self-supervised methods consistently improve under the proposed training regime, con- firming the findings extend SSL approaches that rely on contrastive learning, clustering, and feature matching. The performance gap between the self-supervised and fully-supervised methods is reduced to 3.6% in the top-1 accuracy, and 2.5% in the top-5 accuracy. Note that the mod- els presented here are pre-trained for only 200 epochs with batch size of 1024, while the sfateoof-the-an self-supervised training is shown to perform best on larger settings, e.g. 800 epochs with batch size of 4096. We conjecture the self-supervised and fully-supervised gap could be further reduced by scaling up the experiments.
Linear eval. ImageNet-IK
Method
Top-1(%)
Top-5(%)
Supervised
71.9
90.3
MoCo-v2*
Baseline
60.6
83.3
Ours
61.6 (+1.0)
(4..9)
SWAV
Baseline
65.2
85.6
Ours
67.3 (+2.1)
87.2 (+1.6)
DINO
Baseline
66.2
86.4
Ours
68.3 (+2.1)
87.8 (+1.4)
Table 3: Experiments on MoCo-v2, SWAV, DINO with MobileNetV2. The green numbers depict the improvement over the baseline model.
4.3 EXPERIMENTS ON DIFFERENT LOW-COMPUTE ARCHITECTURES.
We perform experiments other lightweight architectures that were widely evaluated in prior literature (Koohpayegani et al., 2020; Fang et al., 2021; Gao et al., 2022; Xu et al., 2022). We include both the convolution-based networks, e.g. ResNet18 and ResNet34 as well as the vision transformers, e.g. ViT-Ti (Dosovitskiy et al., 2021), and use DINO as the base model. Results of medium-capacio networks, ResNet50 and ViT-S/16 (Dosovitskiy et al., 2021), are also included. For the medium-capacity networks, we use batch size of 640 instead of 1024 due to hardware limitations. Table and report the evaluations under the linear probe protocol. Our pre-trained models consistently outperform the DINO baseline on all low-compute networks: gain of 3.5%, 2.0%, 4.7%, 2.8% in top-1 for ResNet18, ResNet34, ViT-Ti/32, and ViT-T//16. shows that the low-compute networks can all benefit from the new view-sampling recipe. On the other hand, our model performs on par with DINO on ResNet50. For ViT-S/16, the improvement also drops to 0.7%. From Fig. 4, we can also observe that the smaller the network the larger the improvement. These results confirm our hypothesis that the best SSL performance achieved by modulating the regularization in accordance to the network capacity.
7

---

Published as a conference paper at ICLR 2023
Lincar eval. ImageNet-Ik
Backbone
T())
Top-5(%)
T(p-
T(p-5(%)
Top-
Top-S(56)
Tup-I(%)
MobileNerV2
ResNet18
ResNet34
ResNet50
(#Pur OFLare 0.31)
(#Par 2M, GPLOPS 1.8)
(#Par.21 oFLIFa 3.67)
(# oFLare 4.12)
Supervised
71.9
90.3
69.8
89.1
73.3
91.4
76.1
92.9
DINO
66.2
86.4
62.2
84.0
67.7
88.4
73.4
91.5
Ours
68.3 (+2.1)
87.8 (+1.4)
65.7 (+3.5)
86.6 (+2.6)
69.7 (+2.0)
89.5 (+1.))
73.4 (+0.))
91.5 (+0.0)
Table Experiments on the Convolutional Networks using DINO (Caron et al., 2021) the base model. The green numbers depict the improvement over the DINO baseline.
Top-1 Acou.
Lincar eval. ImageNet-Ik
Baseline Ours
+0.0
Backbone
(())
Top-5(%)
70
+2.0
VIT-Ti/32 (#For.5 OFLORS 0.31)
+3.5
Supervised
85
DINO
55.4
78.1
Ours
60.1 (+4.7)
81.8 (+3.7)
60
ResNet18 18
ResNet34
ResNet50
VIT-Ti/16 (#Par.5 SM: OFLOPS 1.26)
Supervised
72.2
91.1
Besaline Ours
+0.7
DINO
66.7
86.6
Ours
69.5 (+2.8)
88.5 (+1.9)
70
+2.8
Supervised
ViT-S/16 (#Par 41
79.9
95.0
60
+4.7
DINO
75.4
92.3
Ours
76.1 (+0.7)
92.7 (+0.4)
50
VIT-TV32
VIT-TV16
VIT-S/16
Table Experiments on the Vision Trans- formers. The supervised results are from DeiT (Touvron et al., 2021).
Figure 4: Improvements vs model complexity. The improvement of our model decreases when the network architecture becomes larger.
4.4 FINETUNING FOR OBJECT DETECTION/INSTANCE SEGMENTATION.
We demonstrate the transfer capacity of our pre-trained models on the COCO object detection and instance segmentation task (Lin et al., 2014). Following He et al. (2020), we train the Mask R-CNN FPN model (He al., 2017) using the pre-trained MobileNetV2, ResNet18, ResNet34 as the feature
backbones. We re-use all the hyper- parameters in He et (2020). The mod- els are finetuned for 1x schedule (i.e. 12 epochs) by SGD (Loshchilov & Hutter, 2017) 8-GPU server. We train the models on the COCO 2017 training set (117k samples) and report the mean erage precision score (mAP@100) on the COCO 2017 validation (5000 sam- ples).
Object Detection APh
Instance Segmentation Ap*k
Backbone
AP
APab
AP
AP?:
MobileNerV2 Supervised DINO Ours
33.1
52.9
35.5
29.8
49.5
31.0
30.9
51.3
32.1
28.1
47.6
29.1
32.1
52.5
33.8
29.1
48.8
30.2
(+1.2)
(+1.2)
(+1.7)
(+1.2)
(+1.1)
ResNer18
Supervised
34.5
54.7
37.4
31.6
51.6
33.6
DINO Ours
32.7
53.5
34.8
30.6
50.4
32.3
34.1
54.8
36.3
31.8
51.8
33.8
(+1.4)
(+1.3)
(+1.5)
(+1.2)
(+1.4)
(+1.5)
As shown in Table 6, our pre-trained models achieve consistent improvements over DINO on all lightweight back- bones for both the box and mask pre- dictions. Similar to the evaluation on ImageNet1K (sec. 4.3), we observe larger improvements on smaller back- bones. Compared to the fully-supervised backbone, our pretrained (esNet18// 18/3/ models perform comparably in general, even slightly better on the mask predic- tion task: +0.5% Apak for ReaNet.34.
ResNer34
Supervised
38.7
59.1
42.4
35.0
56.0
37.4
DINO Ours
37.6
58.7
40.8
34.6
55.6
36.8
38.6
59.9
(+1.0)
(+1.2)
(+1.))
(+0.9)
(+1.3)
(+1.1)
Table 6: Experiments COCO object detection//nstanee segmentation. Apbb/Appk are the mean average precisions of the bounding boxes/instance masks. Numbers in green indicate the performance gain compared to the DINO baseline.
8

---

Published as a conference paper at ICLR 2023
Teacher
Training Epoch
MobileNetV2
ResNet18
ResNet34
Method
Model
Epoch
Top-1
Top.5
Top-1
Top-5
Top-1
Top-5
Supervised
71.9
90.3
69.8
89.1
73.3
91.4
Shi al. (2022)
800
55.7
ReSSI.
58.1
CompRess
MoCoV2 R50
800
130
65.8
62.6
SimReg
MoCoV2 R50
800
130
69.1
65.1
SEED
MoCoV2 R152
400
200
59.5
83.3
62.7
85.8
DisCo
MoCoV2 R152
65.5
86.7
68.1
88.6
BINGO
MoCoV2 R152
400
200
65.9
87.1
69.1
88.9
SEED
SWAV R50x2
800
400
63.0
84,9
65.7
86.8
DisCo
SWAV R50x2
800
200
86.8
67.6
88.6
BINGO
SWAV R50x2
800
200
65.5
87.0
68.9
89.0
200
68.3 (-0.8)
87.8
65.7
(-0.2)
86.6 (-0.5)
69.7 (40.6)
89.5 (40.5)
Ours
400
68.8 (.0.3)
88.3
66.8 (+0.9)
87.3 (+0.2)
70.8 (+1.7)
90.0 (-1.0)
Table 7: Comparison to the tfte-of-the-ar methods ImageNetI under the linear evaluation protocol. The best number is in bold. The best number from the baseline models is underlined. The green/red text indicates the performance gain or gap compared to the best state-of-the-art model.
4.5 COMPARISON THE STATE-OF-THE-ART ON IMAGENETIK
In Table 7, we benchmark our pre-trained models with the of low-compute pretraining works on ImageNetIK, under the linear probe protocol. We compare to both the standalone self- supervised methods, e.g. Shi et al. (2022) and ReSSI. (Zheng et al., 2021), as well as the KD based methods, e.g. CompRess (Koohpayegani et al., 2020). SimReg (Navaneet al., 2021), SEED (Fang et al., 2021), DisCo (Gao et al., 2022), and BINGO (Xu et al., 2022).
Our models perform significantly better than the standalone self-supervised approaches, e.g. + +10.0%/+7.6% compared to Shi et al. (2022) and ReSSL on ResNet18. When pre-trained for similar numbers of epochs, our models perform on par with the best KD-based models on all the lowccoppuee backbones, c.g. -0.8% 0.//0/000//0.66666 0.66% in the top-1 accuracy for MobileNetV2/Rees//e 8/ResNet344 Note that these latest approaches all rely on knowledge distillation (KD), where the teacher models (i.c. significantly larger than the target low-compute network (i.e. MobileNetV2) and are pre-trained in significantly larger settings (batch size of 4096 for 400/800 epochs). On the other hand, our models are pre-trained from scratch in a smaller setting (batch size of 1024 for 200 epochs) without using any distillation signals. When pre-trained for 400 epochs with the same batch size (1024), the performance of our models is further improved, even surpassing the tate-of-the-aa art methods on 8 (+0.9% in Top-1) and ResNet34 (+1.7% in Top-1). l). We believe these results show the feasibility of applying self-supervvsed pretraining on low-compute networks.
ResNet50/50x2/152
On COCO, our model outperforms the best competitor when using MobileNerV2 by 1.7% in mAP for object detection, and by 1.6% for object segmentation. While Sec. 4.4 showed that the gains of the proposed method also transfer to downstream tasks, the current results show that it actually transfers better than that of competing methods. This the first time that an SSL method on low-compute networks roughly bridges the gap with supervised pre-training for downstream tasks. Complete object detection and semantic segmentation results on COCO are shown in the Appendix Sec. C.
We provide comparisons to the state-of-the the--artt methods on semi-supervised learning ImageNetIK. Our model outperforms the best existing approach with ResNet18 by 1.3% and 2.5% for 1% and 10% labeled data respectively. Again, more complete results are included in the Appendix Sec. B.
5 CONCLUSION
In this paper, we investigate the feasibility of self-supervised low-compute pre-training, seeking to diagnose the performance bottleneck reported in previous literature. We reveal the previously overlooked problem of the trade-off between the model complexity and the regularization strength for superviee learning and demonstrate the importance of the design choices for view sampling. We show that by learning with proper image views, supf-super models could achieve strong performance without knowledge distillation. We hope that these encouraging results could motivate more interesting research efficient representation learning for low-compute models/devices.

---

Published as a conference paper at ICLR 2023
REFERENCES
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael G. Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In IEEE International Conference Computer Vision, 2021.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022.
Prashant Bhat, Elahe Arani, and Bahram Zonooz. Distill on the go: Online knowledge distillation in self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR) Workshops, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances Neural Information Processing Systems, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self--upprvvsed vision transformers. In IEEE International Conference on Computer Vision, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In Advances on Neural Information Processing Systems, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint MxXiv:2003.04297 2020c.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. Conf. of the North American Chapter of the Association for Computational Linguistics, 2019.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Zhiyuan Fang, Jianfeng Wang. Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. SEED: Self-super distillation for visual representation. In International Conference on Learning Representations, 2021.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Revisiting contrastive methods for unsupervised learning of visual representations. In Advances on Neural Information Processing Systems, 2021.
Yuting Gao, Jia Zhuang, Ke Li, Hao Cheng, Xiaowei Guo, Feiyue Huang, Rongrong Ji, and Xing Sun. DisCo: Remedy self-supervised learning on lightweight models with distilled contrastive learning. European Conference on Computer Vision, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Advances on Neural Information Processing Systems, 2020.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In /EEE Conference Computer Vision and Patlern Recognition, 2016.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Geoffrey E. Hinton, Oriol Vinyals, and Jeftrey Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 1503.0253,, 2015.
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. CompRess: Self-supervised learning by compressing representations. In Advances on Neural Information Processing Systems, 2020.
Tsung-Wo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, 2014.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Reprexentations, 2019.
K Navaneet, Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. SimReg: Regression simple yet effective tool for self--upervised knowledge distillation. In British Machine Vision Conference, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal on Computer Vision, 2015.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chie Chieh Chen. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and seg- mentation. In IEEE Conference on Computer Vision and Puttern Recognition, 2018
Haizhou Shi, Youcai Zhang, Siliang Tang, Wenjie Zhu, Yaqian Li, Yandong Guo, and Yueting Zhuang. On the efficacy of small self-supervised contrastive models without distillation signals. In AAAI Conference on Artificial Intelligence, 2022.
Bharat Singh and Larry S. Davis. An analysis of scale invariance object detection SNIP. In IEEE Conference on Computer Vision and Pattern Recognition, 2018 88
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your ViT? data, augmentation, and regularization in vision transformers. Transactions Machine Learning Research (TMLR), 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers and distillation through attention. In International Conference on Machine Learning, 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Yen Lo, and Ross Girshick. Detectron2. 2019.
https: /ttp//ggtttta wtt./coccaaaaeeaae Facebookreseate....oooooc..ae... wwwpreeeaaeeeeeeeeeeeeeeeeeeeeee
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling, In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
11

---

Published as a conference paper at ICLR 2023
Haohang Xu, Jiemin Fang, Xiaopeng Zhang, Lingxi Xie, Xinggang Wang, Wenrui Dai, Hongka Xiong, and Qi Tian. Bag of instances aggregation boosts self-vuue learning. In International Conference Learning Representations, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size 32K for ImageNet training. 2017.
arXiv preprint 170xvv/708.000
Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. ReSSL: Relational self-supervised learning with weak augmentation. In Advances on Neural Information Processing Systems, 2021.
12

---

Published as a conference paper at ICLR 2023
A FURTHER DETAILS OF THE EXPERIMENTAL SETTINGS
We provide further details of the pre-training and linear evaluation experiments discussed in and 4. Table 8 presents the implementation details shared by all the experiments covering different self-supervised approaches (i.e. MoCo-v2, SwAV, DINO) and different network backbones (i.c. MobileNetV2, ResNetl 8/34/50, ViT-Ti/S). Note that the learning rates of the pretraining for different SSL approaches are different, depending the pretext tasks. Particularly, we use 0.48 for MoCo-v2 and DINO, 0.3 for SWAV. With LARS as the optimizer, the depth-wise convolution layers and bias terms are excluded from the layer-wise LR scaling. All the other settings, including the intensity- based data augmentations, are inherited from the original literature. Code for the pre-training and evaluation is publicly available at github. com/ saic- waeee-ii/SLihhte
CNNs
ViTs
Pre-training config.
MobileNerV2 ResNet18/34
ViT-Ti/32 VIT-Ti/16
ResNet50
VIT-S/16
Multiview sampling
Sg. S1 (Sec.3.3.1) GC/LC (Sec.3.3.2)
0.3
0.3
0.3
0.3
224/128
224/128
224/128
224/128
N: (Sec.3.3.3) (Sec.3.3.4)
6
6
10
10
0.4
0.4
0.4
0.4
Pre-training optimizer
LARS
LARS
Adam
Adam
optimizer momentum
0.9
0.9
81, 92=0.9, 0.999
B2-9099 0.999
training epochs
200
200
200
200
warmup epochs batch size
10
10
10
10
1024
640
1024
640
base learning rate (Ir) minimum learning rate base weight decay (wd) minimum weight decay Ir/wd schedule
1r X
batcheize
1r 256
5e-1 batchaize
5e-4
256
256 le-5
1e-3
1r x 16-3
1e-5
1e-6
1e-6
0.04
0.04
1e-6
le-6
0.4
0.4
cosine decay
cosine decay
cosine decay
cosine decay
Linear probing
optimizer
LARS
LARS
SGD
SGD
optimizer momentum
0.9
0.9
0.9
0.9
training epochs
100
100
100
100
warmup epochs batch size
4096
4096
4096
4096
base learning rate (Ir) minimum learning rate
0.05 x batchaize
0.05
batcheize
0.01 x batchaize
0.01 x batcheize
256
256
256
256
le-6
le-6
le-5
le-5
weight decay Ir schedule
cosine decay
cosine decay
cosine decay
cosine decay
Table 8: Implementation details for the pretraining and evaluation experiments discussed in 3, and 4.
B SEMI-SUPERVISEDD EVALUATION ON IMAGENETIK
Following CompRress (Koohpayeganí et al., 2020), SEED (Fang et al., 2021), DisCo (Gao et al., 2022), and
BINGO (Xu et al., 2022), we perform semi- cuni-supervised evaluations on ImageNetIK (Russakovsky et al., 2015) by finetuning the pre- trainedid models the 1% and 10% labeled data defined in SimCLR Chen et al. (2020a)
Recall that for linear prohing, the classifier is applied on top of the spatially pooled features of the convolutional networks. For the semi-supervised evaluation however, we apply the classifier on the second layer of the MI.P head (Caron et al., 2021)), similar to SimCLR (Chen et al., 2020n;b)), PAWS (Assran et al., 2021). We find this significantly improves the classification accuracy under a low-data/label regime. During training, only random cropping and horizontal flipping are used for pre-processing, the images are then resized to 224 x 224. During inference, images are first resized to have a minimum edge 256 then cropped the center 224 x 224. All models are optimized by the SGD (w. Nesterov) optimizer with a batch size of 1024 and momentum of 0.9. The initial lcarning rate is set to 0.03 x batchsize /256 and decay to 1e-6 by cosine schedule (Loshchilov & Hutter, 2017). For the experiment with 10% labeled data, we finetune for 30 epochs. For the experiment with 1% labeled data, we finetune for 60 epochs.
'htpps: http://wwwwttt.fffeeee
cotps////aaaaaaaaaaoooooooo mtts://mmaenee22111m1 hagp://aatettttaaaao//a//a/eeeaa 2_s_set
13

---

Published as a conference paper at ICLR 2023
Pretrained
1% labels
10% labels
Backbone
Epochs
Top-1(%0
Top-5(%)
Top-1
Top-5(%)
MobileNetV2
(fPar.2. 2.2M, GFLOPS 0.31)
DINO
200
47.9
75.1
61.3
84.2
Ours
200
50.6 (+2.7)
77.6 (+2.5)
63.5 (+2.2)
85.8 (+1.6)
Ours
400
52.6 (+4.7)
78.6 (+3.5)
64.0 (+2.7)
85.8 (+1.6)
ResNet18
(#Par.111.h) GFLOPS 1.8)
DINO
200
44.5
72.1
59.2
82,8
Ours
200
49.8 (45.3)
77.2 (+5.1)
63.0 (+3.8)
85.5 (+2.7)
Ours
400
51.6 (47.))
78.5 (+6.4)
63.7 (+4.5)
86.0 (+3.2)
ResNet34
(#Par.2113M. GELORS 3.67)
DINO
200
52.4
79.0
65.4
86.9
Ours
200
55.2 (+2.8)
81.5 (+2.5)
67.2 (+1.8)
88.4 (+1.5)
Ours
400
56.8 (+4.4)
82.6 (+3.6)
67.5 (+2.1)
88.5 (+1.6)
Table Semi-superviied evaluation on ImageNetIK, with MobileNetV2, ResNet18, ResNet34 as backbones. The green numbers depict the improvement the baseline model.
Teacher
Pretrained Epoch
Method
1% labels
10% labels
Model (Accu.)
Epoch
SEED Fang et al. (2021)
R152 (74.1)
400
200
44.3
54.8
DisCo Gao al. (2022)
R152 (74.1)
400
200
47.1
54.7
BINGO Xu et al. (2022)
R152 (74.1)
400
200
50.3
61.2
BINGO Xu et al. (2022)
R50x2 (77.3)
800
200
48.2
60.2
200
49.8 (-0.5)
63.0 (+1.8)
Ours
400
51.6 (+1.3)
63.7 (42.5)
Table 10: Comparison the state-of-the-art methods semi-supervised ImageNet1 K recognition, using ResNet1 as the backbone. The best number is in bold. The best number from the baseline models underlined, The green/red text indicates the performance gain gap compared the best state-of-the model.
Table 9 shows that our pre-training paradigm significantly improves the semi-supervised performance in all settings. For example, with 1% labeled data, we improve the DINO baseline (Caron et al., 2021) by 99%/5.3%/..7...
Note that the performance gain in semi-supervised learning larger than that in linear probing, demonstrating further advantages of our model under low-data/label regime.
In Table 10, we compare to the state-of-the-ar approaches on semi-supervised ImageNet1 K recognition, using ResNet18 as the backbone. Recall that the baseline models are all optimized by knowledge distillation. Our models pre-trained from scratch without using any teacher. With the (i.e. 200 epochs), our model performs par with (i.e. -0.5) the best baseline 1% labeled data, favorably (i.c. +1.8) for 10% labeled data. With more pre-trained epochs (i.c. 400 epochs), our model outperforms the best statc-of-the- thet method by +1.3//2.5 for 1%/10% labeled data. Note that, compared to the large-scale pre- pre-training of the teacher models used in the basclines, the overhead of the extra epochs for model is marginal.
number of pre-trained epochs
C EXTRA EXPERIMENTS ON COCO
We compare our model to the sate-of-the-are self-supervised low-compute methods on COCO Instance Seg- mentation(Lin et al. (2014)). While similar evaluations have been reported in previous literature (Fang et al.. 2021; Gao et al., 2022: Xu et al., 2022), the detail configuration is not made public. For fair comparison, the pre-trained weights released the baseline approaches (Navaneet et al., 2021; Fang et al., 2021; Gao
14

---

Published as a conference paper at ICLR 2023
Mask R-CNN FPN MobileNet-V2
Method
Aphi
AP
AP
Apm
APm
A APm
SimReg
30.7
49.4
32.7
27.7
46.3
28.7
Ours
32.4 (+1.7)
52.4 (+3.0)
34.4 (+1.7)
29.3 (+1.6)
49.1 (+2.8)
30.3 (+1.6)
2x
SimReg
33.7
52.8
36.1
30.0
49.3
31.3
Ours
35.1 (+1.4)
55.4 (+2.6)
37.8 (+1.7)
31.7 (+1.7)
52.1 (+2.8)
33.3 (+2.0)
Table 11: Comparison to the state-of-the-art method, SimReg (Navaneet et al., 2021), on COCO Instance Segmentation, using MobileNet-V V2 as the backbone. The best number is in bold. The green numbers depict the improvement over the SimReg model.
Mask R-CNN FPN on ResNet18
Method
Teacher
A
APM
AP
A
AP
A AP
1x
SimReg SEED
BYOL R50
33.1
52.1
36.1
30.2
49.2
32.1
SWAV R50x2
34.7
54.8
37.5
31.9
51.8
34.1
DisCo
SWAV R50x2
34,6
55.1
37.1
32.0
52.0
34.0
Ours
34.7 (+0.0)
55.6 (+0.5)
37.5(
32.3 (+0.3)
52.8 (+0.8)
34.3 (+0.2)
2x
SimReg SEED
BYOL R50
35.7
55.6
38.9
32.6
52.5
35.1
SWAV R50x2
36,9
57.2
40.3
33.6
54.3
35.8
DisCo
SWAV R50x2
36.6
56.9
39.7
33.6
54.0
35.9
Ours
37.3 (+0.4)
58.2 (+1.0)
40.7
(+0.4)
34.3 (+0.7)
55.1 (+0.8)
36.8 (+0.9)
Table 12: Comparison to the state-of-the-at methods COCO Instance Segmentation, using ResNet18 as the backbone. The best number is in bold. The best number from the baseline models is underlined. The green text indicates the performance gain compared to the best state-of-the oftth-of-tee-rt model.
al., 2022) and finetune all the models the Mask R-CNN FPN framework(He et (2017)), using the default contiguration defined in Detectron2 (Wu et al., 2019). Our models used in this experiment are pre-trained for 400 epochs. All models are evaluated under both the 1x and 2x schedules. For the MobileNetV2 backbone that not supported in Detectron2(Wuu al. (2019)), re-implement the Torchvision Instance Segmentation framework in Detectron2(Wu al. (2019)).
Table 11 shows the comparison the SimReg model (Navancet et al., 2021) on COCO Instance Segmentation, using MobileNetV2 as the backbone. We use the best model released by the SimReg authors, which is trained with MoCo-v2 R50 (Chen et al.. 2020c) as the teacher and achieves top-1 accuracy of 69.1 under linear evaluation (see Sec.4.5 in the main paper). Our models consistently outperform SimReg in all settings.
In Table 12, we further compare model the state-of-the-art approaches using ResNet18 the backhone. experiment is pre-trained with BYO R50 (Grill et al., 2020) as
Note that SimReg baseline used teacher: It is shown to be the best model for ResNet18. Our model demonstrates better performance over the best state-of-the-an method in all settings, especially for the mask prediction task. However, the improvements smaller (i.e. 0.0% 1.0%) than the linear/semi-supervisedd tasks in general. We conjecture that, with more in-domain data/labels COCO, the advantage of good pre-trained models becomes marginal.
hntppa httpp://///tttthtooeoooo org/vi e /modell dobjjct-t- on- www.nnnnttennteeoooom sogmentatetttttooooo lon-and-pe pertd-peroo----- keypol no det act
15


Reviews:
[{"review_id": "eY0608j6Efr", "reviewer": "Reviewer_rtAP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors investigate the problem of self-supervised pre-training on compact networks. They study the impact of view sampling strategy and design a strategy which can boost the SSL pretraining performance without knowledge distillation. The developed method consistently shows improvement on different compact models with different SSL frameworks.", "summary_of_the_review": "In conclusion, this work improves the self-supervised pretraining on compact models by tuning several hyperparameters in the view sampling strategy, which is simple but shows decent performance boost. From my perspective, I would like to see more analyzes on how these changes influence the training, instead of just showing the empirical results.", "strength_and_weaknesses": "Strength:\n- This paper is well-motivated. The proposed method provides a good alternative to the current distillation based methods for ssl pretraining on low-compute networks.\n- The article clearly shows the thought process and is easy to follow.\n- The proposed method shows decent improvement over baseline results.\n\nWeakness:\n- I'm not convinced by how S_l and S_g are produced. 0.14 and 0.4 for S_l are quite different and I'm wondering whether it will influence the conclusion.\n- Lack of analyzes. Most parts of the method are only driven by empirical results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cTPKdGIQXpk", "reviewer": "Reviewer_aQkw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of improving self-supervised learning (SSL) methods on compact neural networks. The authors investigate the relations between data augmentation strength and sizes of neural networks in SSL, and propose a new objective and data augmentation scheme to improve the performance.", "summary_of_the_review": "This paper studies an important but less-explored aspect of self-supervised pretraining. They investigate the problem with interesting experiments to support intuitions and provide motivations, and their final results justify their contributions.", "strength_and_weaknesses": "Strengths:\n1. The research goal is well-motivated. The wider performance gap between supervised and self-supervised methods on smaller models is long observed and dubbed the benefits of larger model sizes. However, as many applications of deep learning are still confined by memory and compute limits, it is important to improve these methods in smaller models. \n2. The proposed method in the paper is motivated due to their careful experiments and observations. The authors conduct controlled experiments supporting their intuitions about the correlations of model sizes and data augmentation strength in SSL. They define a rather novel measure pixel scale to examine the effects of different view construction on the performances. And based on these results, they conjectured the smaller performance gap between SSL and supervised for larger models is due to that the smaller models lack the ability to learn on such adversarial augmented data. \n3.  The comparison between different methods on different downstream tasks are solid and convincing. The author compared their improved SSL method with others on classification, object detection, and segmentation, and over multiple architectures including resnet, mobilenetV2 and small transformers. The performance looks good and surpasses distillation-based methods in many cases, which is clearly an important contribution.\n\nWeaknesses:\n1. There are some clarity issues due to arrangement of contents, which are mentioned below.\n2. The authors can dig into the local vs global augmentations with more principled metrics. The $\\mathrm{mean}(PS_g/PS_{\\ell})$ seems rather arbitrary and less intuitive.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PJVri-A8w-O", "reviewer": "Reviewer_h6ge", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents  a training recipe for helping smaller computer vision models achieved improved performance from Self-Supervised Learning. The paper presents the success of said recipe as a evidence that it is not the capacity of the smaller models that limits their performance but rather the extreme regularization effects of some of the data augmentation procedures used in classical SSL training.", "summary_of_the_review": "The authors do a good job of presenting a training recipe that improves performance of SSL methods on smaller models.\nHowever, they do not establish the key claim they make in the paper which is that \n> We find that, contrary to accepted knowledge, there is no intrinsic architectural bottleneck, we diagnose that the performance bottleneck is related to the model complexity vs regularization strength trade-off\n\nFor my score to increase, I would have to be convinced that the positioning of the paper is correct. \n\n[Edit - ] The authors rightly pointed out that Table 5 is an unfair comparison to their method. They directed me to table 3, which I agree is the better table to compare to.\nI have updated my score based on the rebuttal", "strength_and_weaknesses": "**Strengths**\n1. The paper has extensive ablations that validate the impact of various modifications to the classical training procedure for vision-based SSL\n2. We see non-trivial improvements of the method across multiple benchmarks and on top of multiple SSL methods.\n\n**Weaknesses**\n\nMy main issue with the paper is that it is not properly positioned in that I do not believe it has provided adequate evidence for its major claim.  The paper claims to show that excessive regularization from the augmentation strategy is the reason why smaller models under-perform KD  with larger models. \nWhat the authors present however, is a set of  very useful **training recipes** for getting improved performance on smaller models. The existence and efficacy of this recipe however does not support their claim that capacity is not a bottleneck for smaller models. Specifically, to fully show this they would have to :\n1. Demonstrate that the present training recipe **does not improve results for larger models**. If it does lead to improved results for the larger model, then the authors have only presented a better training procedure but not an explanation for the performance gap. Improved results on a larger model roughly implies improved KD performance which would mean that the purported gap isn't actually closed but the baselines are just shifted up.\n\nTable 5 buttresses my suspicion that the procedure presented will benefit models at larger scales, given that the delta improvement actually **increases** from Resnet18 to Resnet34 \n\n[Edit - ] The authors rightly pointed out that Table 5 is an unfair comparison to their method. They directed me to table 3, which I agree is the better table to compare to.\nI have updated my score based on the rebuttal", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FhKLpxTGKWh", "reviewer": "Reviewer_MNWJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self-supervised learning approach in the context of low-compute deep-learning models (e.g., DeiT-Tiny). The authors observe that a weaker self-supervised target is beneficial for small networks, and propose to match multiple views in more comparable spatial scales and contexts. Experimental results are provided on ImageNet and COCO.", "summary_of_the_review": "The experimental analysis is extensive. However, I have some concerns on the novelty and importance of this paper. See the weaknesses above. ", "strength_and_weaknesses": "Strength:\n\n- The experimental analysis in Section 3 is extensive.\n- The improvements in Table 3 are impressive.\n\nWeaknesses:\n\n- The authors find that a weaker training target is beneficial for the self-supervised learning of low-compute networks, and demonstrate that this idea can be implemented by adjusting the relative scale of different views. Personally, I do not think this is a significant scientific contribution. The conclusion seems straightforward, while the authors conduct a series of experiments, yielding an empirical principle for hyper-parameter searching. In general, I think that, currently, this paper may be too engineering-oriented to be published on ICLR.\n- Is the proposed method compatible with knowledge distillation? For low-compute networks, the most significant concern may be how to improve the accuracy without introducing additional computational costs during inference. I think that \"without distillation\" is not an important advantage.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster