Moving Beyond Handcrafted Architectures in Self-Supervised Learning
Authors: Sharath Girish
Debadeepta Dey
Neel Joshi
Vibhav Vineet
Shital Shah
Caio Cesar Teodoro Mendes
Abhinav Shrivastava
Yale Song
Venue: Submitted to ICLR 2023 (2023)
Keywords: NAS
Self-supervised learning

Abstract: The current literature on self-supervised learning (SSL) focuses on developing learning objectives to train neural networks more effectively on unlabeled data. The typical development process involves taking well-established architectures, e.g., ResNet demonstrated on ImageNet, and using them to evaluate newly developed objectives on downstream scenarios. While convenient, this does not take into account the role of architectures which has been shown to be crucial in the supervised learning literature. In this work, we establish extensive evidence showing that architecture plays a significant role in SSL. We conduct a large-scale study with over 100 variants of ResNet and MobileNet architectures and evaluate them across 11 downstream scenarios in the SSL setting. We show that there is no one network that performs consistently well across the scenarios. Based on this, we propose to learn not only network weights but also architecture topologies in the SSL regime. We show that ``self-supervised architectures'' significantly outperform popular handcrafted architectures (ResNet-50 and MobileNetV2) on major image classification benchmarks (ImageNet-1K, iNat2021, and more). Our results suggest that it is time to consider moving beyond handcrafted architectures in SSL and start thinking about incorporating architecture search into self-supervised learning objectives.

Under review as a conference paper at ICLR 2023
MOVING BEYOND HANDCRAFTED ARCHITECTURES IN SELF-SUPERVISED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The current literature on self-supervised learning (SSL) focuses on developing learning objectives to train neural networks more effectively on unlabeled data. The typical development process involves taking well-established architectures, e.g., ResNet or ViT demonstrated on ImageNet, and using them to evaluate newly developed objectives on downstream scenarios. While convenient, this neglects the role of architectures which has been shown to crucial in the supervised learning literature. In this work, we establish extensive empirical evidence show- ing that a network architecture plays a significant role in contrastive SSL. We conduct large-scale study with over 100 variants of ResNet and MobileNet ar- chitectures and evaluate them across 11 downstream scenarios in the contrastive SSL setting. We show that there is one network that performs consistently well across the scenarios. Based on this, we propose to learn not only network weights but also architecture topologies in the SSL regime. We show that "self-supervised architectures" outperform popular handcrafted architectures (ResNet18 8 and Mo- bileNetV2) while performing competitively with the larger and computation- ally heavy ResNet50 on major image classification benchmarks (ImageNet-IK, IK, iNat2021, and more). Our results suggest that is time to consider moving be- yond handcrafted architectures in contrastive SSL and start thinking about incor- porating architecture search into supervide learning objectives.
INTRODUCTION
Self-supervised learning (SSL.) achieves impressive results on challenging tasks involving image, video, audio, and text. Models pretrained on large unlabeled data perform nearly as good and sometimes even better than their supervised counterparts (Caron et al., 2020; Chen & He, 2021). So far, the focus has been designing effective learning objectives e.g., pretext tasks (Gi- daris et al., 2018; Caron et al., 2018), contrastive (Oord et al., 2018; Chen et al., 2020a) and non- contrastive (Grill et al., 2020) tasks together with empirical (Cole et al., 2021; Feichtenhofer al., 2021) and theoretical (Arora et al., 2019; Poole et al., 2019) studies providing key insights and un- derpinnings. Recent works propose new objectives with different class of network architectures such as vision transformers (ViT) (Bao et al., 2021) and masked autoencoders (He et al., 2022).
However, there has been little focus on the role of architectures in SSL. Currently, the de facto pro- tocol in SSL is to take architectures that perform well on established benchmarks in the supervised setting and adapt them to the self-supervised setting by plugging in different learning objectives. For example, several existing work on contrastive learning use ResNet (He et al., 2016) as the back- bone (Chen et al., 2020a; He al., 2020). This partly for convenience. Evaluating different architectures in SSL is computationally expensive; selecting architecture in advance and fixing throughout makes easy to evaluate different learning objectives. This also stems from strong empirical success of those architectures in transfer learning, c.g., CNNs trained on large labeled data provide "unreasonable effectiveness" (Sun et al., 2017; Zhang et al., 2018; Sejnowski, 2020) in variety of downstream cases. Nonetheless, one implicit assumption is that an architecture that works well in the supervised learning scenario will continue to be effective in the SSL regime.
We argue that this assumption is incorrect and dangerous. It is valid only to limited extent and the performance starts deteriorating significantly when SSL is conducted on data whose distribution deviates much from the original distribution the architecture trained This is counter to

---

Under review as a conference paper at ICLR 2023
the promise of SSL, where one can learn optimal representation for wide range of tasks, One main reason for performance degradation is that different data distributions benefit from different inductive biases: An architecture with specific layer types and the wiring between them naturally encodes inductive biases, which may be optimal only for a certain data distribution (e.g., object- centric imagery such ImageNet) and not for others (e.g., medical and satellite imagery). In fact, numerous studies have shown that standard "recipes" for architecture design do not translate well across different data distributions (Tuggener et al., 2021; Dey et al., 2021; Kolesnikov et al., 2019).
The main objective of this work is to show that the choice of network architecture crucially matters in SSL, and that it is not easy to handcraft architectures that are effective across different SSL sce- narios. To see this, recall that the goal of SSL is to learn data representations capturing important features and attributes that generalize well across various downstream tasks. There has been exten- sive literature on the expressivity of neural networks (Raghu et al. (2017); Zhang et al. (2021a) and references therein); one of the important conclusions is that the network topology plays signifi- cant role in determining the expressivity, i.e., the kinds of functions network can approximate is bounded by the network capacity and available sample size in the finite sample regime. This im- plies that, in practice, SSL with a fixed architecture learns representations only within the scope of function space induced by the pre-selected architecture topology, and therefore, the ultimate success of SSL can be achieved when it finds optimal architecture from certain search space in conjunction with its weights for specific data distributions.
In this paper, we establish extensive empirical evidence showing that architecture matters in self- supervised learming. We do this in two sets of large-scale studies. First, we sample 116 variants of ResNet (He al., 2016) and MobileNet (Sandler et al., 2018) architectures with different topologies and evaluate them on 11 downstream tasks in the SSL setting. We pretrain all models under the same setting, optimizing the SimCLR objective (Chen et al., 2020a) on ImageNet (Deng et al., 2009), and investigate if there exist any correlation between these models in downstream performance on different datasets. We observe no strong correlation, except for tasks highly similar to ImageNet. We further show that ImageNet downstream performance, the gold standard benchmark in the SSL. literature, is not indicative of performance on other downstream tasks. This implies that we need to be careful in choosing an architecture for evaluating any newly developed SSL objectives, as one might get different conclusions based on different network architectures.
This subsequently raises the question: Can we improve SSL by learning not only network weights but also architectures directly optimized for the given dataset? It removes the burden of manually searching for effective architectures in SSL., and if succeed, it can substantially improve perfor- mance of SSL. To test this hypothesis, as the second set of our study, we apply a well-established NAS algorithm (Cai et al., 2018) to the SSL setting. Unlike the typical NAS setting that optimize on a labeled target dataset, we search for optimal architectures directly on an unlabeled pretraining dataset via contrastive learning (Chen al., 2020a). We evaluate our "NAS + SSL" framework on datasets with different distributions, ImageNet-IKG 1K and iNat 2021 (Van Homn et al., 2021), and show that supervueee architectures consistently outperform handcrafted ones in the same parameter range NMt2 and ResNet18) across 11 downstream tasks. This provides strong evidence suggesting the importance of learning architecture topologies in addition to their weights in SSL.
Our work focuses on studying the role of architectures in contrastive SSL. with the SimCLR frame- architectures such as ResNets and MobileNets. As a first step in this direction,
work for we provide an in-depth analysis through large-scale experiments in this specific (yet limited) setting. Extending our study to different SSL approaches (He et al., 2020; Grill et al., 2020; He et al., 2022) and architectures such as ViTs would be interesting direction but beyond the scope of this paper.
In summary, our main contributions are: 1) We establish extensive evidence showing that there isn't 't one network architecture performing consistently well across different downstream scenarios in the SimCLR setting. We show this using 16 variants of ResNet and MobileNet architectures pretrained on ImageNet and evaluated 11 downstream datasets. 2) We show that ImageNet performance (the gold standard in SSL benchmark) is not always indicative of downstream performance. This means that findings about SSL objectives shown only on ImageNet do not generalize across other data distributions. 3) We propose to self- self-supervise CNN architecture topology and its network weights on unlabeled data. We show that self-supervised architectures outperform handcrafted ones a similar parameter range for the SimCLR setting across different downstream datasets.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Role of architectures in SSL. There has been significant progress in learning representations via SSL (Gidaris et al., 2018; Chen et al., 2020a;c; Caron et al., 2020). Ericsson et al. (2021) provide an overview of different SSL setups and their performance on downstream tasks. Most works focus on improving self-supervised objectives to develop better representations while keeping architectures fixed. Our focus is orthogonal to this line of work. We study the role of architectures in SSL. and investigate the benefits of seff-supervising architecture topologies along with network weights.
Similar to ours, Kornblith et al. (2019) analyze transfer performance of different architectures pre- trained on ImageNet across several downstream datasets. However, they focused on supervised learning, which need not translate to self-supervised setups (Kolesnikov et al., 2019). Caron et al. (2021) propose self-distillation based SSL objective and compare the performance of ResNets with ViTs. In contrast, we focuse on contrastive SSL and the importance of architecture topologies for the class of CNNs. Kolesnikov et al. (2019) is the most related to ours but are limited to pretext-task based SSL. and show results only for a few variants of VGG and ResNet. In comparison, we con- duct study on much larger scale in contrastive learning setup. Also, we provide insights into generalization performance of the networks across different datasets. Crucially, unlike all previous work in SSL, we propose to learn both architecture topologies and their network weights.
NAS and SSL. The literature on NAS has rapidly progressed in the past years; we refer the reader to the NAS survey by Elsken al. (2019). Here focus on the most directly relevant work to SSL. Mellor et al. (2021) search for networks without any training and verify their effectiveness of supervised benchmarks. Liu et al. (2020) show that highly performant architectures can be found without using any supervised labels during the search itself, and propose using DARTS (Liu et al., 2019) with sulf-suuer proxy tasks to search for architectures which will perform well on super- vised datasets. In similar vein, Zhang et al. (2021b) use random labels during the search phase of NAS, and Yan et al. (2020) learn representations of architectures via SSL and use them to improve NAS. Li et al. (2021) introduce new self- supervisdd training scheme to search for architectures in a new hybrid search space. One common theme in this line of work is that they aim to harness SSL in aid of NAS. In contrast, we aim to utilize NAS to hunt for architectures which will perform well for self-supervised learning, thereby harnessing NAS in aid of SSL..
3 DOES ONE NETWORK RULE THEM ALL IN SSL?
We conduct a large scale study to investigate whether a particular architecture topology can be consistently effective across a wide range of SSL scenarios. To this end, we sample 1 116 models with different architecture topologies and analyze their performance 11 downstream tasks.
To maximize generality while taming complexity of our study, we choose two most representative CNN architectures: ResNets and MobileNets. The former is the de facto backbone for numerous modern visual models (Kirillov et al., 2019; Wu et al., 2019) and SSL approaches (Caron et al., 2020; Chen & He, 2021), while the latter is used in low-resource setups (Cheng al., 2017). We create 69 ResNet-like and 47 MobileNet-Hike architectures for our evaluation, varying the number of blocks at each of the 4 stages of ResNet, the block structure (BasicBlock and Bottleneck), width, and number of groups For MobileNet, we vary the width multiplier parameter and the number of blocks in each of the 6 stages. We choose the SimCLR objective (Chen et al., 2020a) for our experiments and pretrain each of the 116 architectures on ImageNet- IK. Owing to the large scale nature of the experiments and computational constraints, we limit the pretraining to 100 epochs and a batch size of 512. Each of these jobs takes roughly 1 day to finish on system with 8x V100 (32GB) GPUs We report linear evaluation results in all cases.
ImageNet performance is not indicative of downstream performance for SSL. To examine the correlation between ImageNet vs. downstream performance, we compute the Spearman's 's rank correlation coefficient p on top-1 validation accuracy between every dataset pair, shown in Fig. 1. We also show scatter plots in Fig. revealing the relationship between ImageNet vs. downstream performance on the most representative cases; the complete set of scatter plots are in the appendix.
For RisNet-like architectures (Fig. 2 top row), we see strong correlation between ImageNet and CIFAR- 100 (p .8) as these datasets contain similar categories; about 90 classes in CIFAR-10N
3

---

Under review as a conference paper at ICLR 2023
Pro ST167 (pe 50D
Frohed Feahee-Sports (
Rested Staddrr Ca's 3em 220-70
705
57
44.0
703
91.3
555
35.0
425
02.0
52
345
47-
ImagiRet Acoussy
magetecccuuaa Serlete (ara
Medii tabc///a0 (ym 23)
Spheo tpm-
Mestvlatrove Reast (1m21)
2>
140
15.
23
620
155
Wrageler AUNorr
Trugrton Acondcy
Playe aetrrrry
Irogeter MOOEeyy
Figure 2: ImageNet performance is not indicative of downstream performance in SSL. We show linear evaluation top-1 accuracy of ImageNet (x-axis) vs. downstream datasets (y-axis) obtained from variations of ResNet (top) and MobileNet (bottom); all models are pretrained on ImageNct- KK using SimCLR under the same protocol. The solid lines and shaded areas indicate fitted regression models and their confidence intervals.
the same class or superclass in ImageNet. A similar observation is made (in Fig. 1) for Stanford Dogs (p- = .77) due to the 120 dog categories in ImageNet. However, we observe high variance in transfer performance for out-of-domain datasets, e.g., Flowers (p .35), represented by only two ImageNet categories (daisy and yellow lady slipper). Correlation becomes negative on Stanford Cars (p -29) and FGVC Aircraft (p 22).. likely because ImageNet contains only few categories of cars (10 classes) and aircraft (4 classes). Low correlation means network ranks are inconsistent and models performing well ImageNet do not keep their precedence in other tasks.
For MobileNet-like architectures (Fig. 2 bottom row), we overall much lower correlation with Im- ageNet performance; the ResNet space showed correlation at least for in-distribution datasets. For e.g., we see correlation between ImageNet and CIFAR-100 drops from P = .8 (ResNet) to For other datasets like MIT67, correlation is higher (p .52) but still less meaningful due to high variability in performance (notice the cluster around 42% accuracy). These results indicate that MobileNets even less tuned towards ImageNet than ResNets and any handcrafted architectures in this space is likely to be suboptimal on ImageNet and other downstream datasets.
Spearman rank correlation coufficient
ImspeNet
1
0.8
0.77
0.77
0.5L
0.45
0:43
0:35
0.08
0.24
-0.29
GRAR120
6.8
1
0.86
0.85
0.56
0.6
0.5
0:35
0:15
60338
-0199
ORAR30
0.77
0.86
0.01
0:59
0.54
0.5
0:27
0.64
6.337
-11
0.6
Dogs
0.77
0.85
0:38
0.2
00.05
0.38
-3.15
.23 (MobileNet).
MIT37
0.51
0.01
0.05
-0.4
CUB
0.43
0.5
0.24
0.66
0.35
0..05
-0.2
0.35
0.35
0.27
6.01
0.24
0.25
-0.22
-2.16
0.0
0.08
0.15
0.04
6.05
-0.91
0.06
0.26
0.1
-0.12
Arcewtt
0.24
0.38
0.37
0:38
0.34
0.15
6:15
0.22
0.1
6.12
Cars
-0.29
-0.39
-0.1
-0.15
-0.09
-0.17
-0.05
-0.16
-0.12
6.12
Imaghet
ORR160
Doga
HTT27
SAYN
Syeraa
Aroalt
Our results highlight that the same architecture recipes in terms of Ima- geNet accuracy, which is frequently used as predictor for various self- supervised tasks, do not work well for different datasets in the SimCLR setting.
ImageNet performance isn't indicative of down- stream performance in SSL. We show rank correlation between each pair of top-1 accuracy on 11 downstream tasks obtained from ImageNet- pretrained pretenne ResNets. We see no strong correlation except for ones similar to ImageNet, e.g., CIFAR- and Dogs
Figure 1:
Larger networks do not always perform better in contrastive SSL. In general, darge-parameter models yield better performance ImageNet, both in supervised (Komblith et al., 2019) and SSL setups (Chen et al., 2020a). We examine whether this trend holds for downstream datasets some of which are widely different from ImageNet. We follow the same setup above and compute the
4

---

Under review as a conference paper at ICLR 2023
Pec maggelee (gn A05
Reahs Sperts Spw E41
Rolled 50 Cpa
Redhe www. Aac (p-.3))
5
953
406
M45
Parsint MD
Rrans Jo
Puraric M]
Paisin MI
(pm-de)
06.5
275
95.6
266
20
25.6
400
155
Figure 3: Larger models do not always perform better SSL. We show the model size in terms of parameter counts (x-axis) vs. top-1 accuracy on different datasets obtained from variations of ResNet (top) and MobileNet (bottom) architectures; all models are pretrained on ImageNet- IK using SimCLR under the same protocol.
correlation between number of model parameters and top-1 validation accuracy on different datasets. Fig. 3 shows scatter plots of the most representative results; full results are in the appendix.
For ResNet-like architectures (Fig. 3 top row), we do not see strong trends indicating larger models always perform better. In fact, some downstream tasks we negative correlation (Aircraft, p -337) with lighter networks being favored for better performance, or even non non-linear relationship, e.g., notice the slight "U" pattern on Stanford Cars (p- = 14),, indicating the behavior of ResNets on these datasets are wildly unexpected. Unsurprisingly, there is strong correlation with ImageNet (p=88h) .85) and CIFAR- 100 (p- = .87). likely because ResNets are heavily hand-tuned on the kinds of images observed in these datasets. These results clearly suggest that the same architecture recipes which work well for ImageNet (increasing parameters through depth and width) do not hold for other downstream scenarios.
For MobileNet-like architectures (Fig. bottom row), we see that there exists almost no interpretable trend. The fitted regression models (solid lines) and their confidence intervals (shaded area) show that the relationships are highly non-linear and non-monotonic; we shouldn't 't read too much into the correlation coefficients (reported for completenes)). These results indicate that MobileNets do not favor any one architecture and it is heavily reliant on the dataset it is trained on.
Our results suggest that larger models not always better in SSL; lighter models can outperform heavier models on tasks like FGVC Aircraft. Previous work (Chen et al., 2020b) showed that in- creasing network depth/width improves downstream performance. However, they vary depth at coarser level with 50/100/150 layers and width with 1x/2x, leading to a large swing in network pa- rameters (24-795M); here we show the same is not true at finer level. Our results provide evidence that ResNet architectures are tuned to scale well on ImageNet but not the others; the trend is even weaker for MobileNet-like architectures, suggesting they are optimized for computational efficiency and not for achieving high accuracy on any particular dataset.
There is no winner in the battle of top vs. bottom heavy networks in SSL. Raghu et al. (2017) showed that CNNs are more sensitive to lower (initial) layer weights, suggesting that "not all weights are created equal" across layers. This raises the question: If CNNs are more sensitive to lower layers, will increasing the parameter count for lower layers yield better performance in SSL? To get insights into this, we split a network into two halves: "top" (layers closer to output) and "bottom" (closer to input). We use the ratio of top to bottom parameters as a measure of networks being "top-heavy" (high ratio) or "bottom- "bottom-hcavy" (low ratio).
Fig. 4 shows the downstream accuracy for ImageNet, Stanford Cars and Sports against this ra- tio. From the top-left subplot, we see that ResNet--ImageNet accuracy tend to increase with high top:bottom ratio, showing hea-y networks generally perform better than the bottom-heavy coun- terparts. However, we no longer observe such trend in other datasets, and with MobileNets (Fig. 4 bottom row) we do not see such trend even for ImageNet.
An important point to realize: It is necessary to allocate the right portion of parameters to dif- ferent layers of given network topology, instead of allocating parameters in the top-heavy or
5

---

Under review as a conference paper at ICLR 2023
bottom-heay fashion assuming one will generally lead to better performance. ResNets and many other handerafted CNNs (Simonyan & Zisserman, 2014; Szegedy et al., 2016; Huang et al., 2017) usually top-heavy because of GPU memory limits; bottom-heavy networks occupy more mem- ory in terms of activation maps. MobileNets alleviate this to some extent with lighter convolutions, allowing it to have more parameters in early layers. In object detection, Liang et al. (2019) also show that allocation of computational resources in the backbone is important for improved perfor- mance. While this architectural difference provides explanations about the wildly different trends we observe above, the key message here is that one recipe (top vs. bottom heavy) does not apply equally to different architectures, bolstering our claim that one network doesn't rule them all in SSL.
Key takeaway: We need to move beyond hand- crafted architectures in SSL. The three main ob- servations above imply that finding an optimal archi- tecture could be impor- tant missing piece for self- supervised learning. Our results show that the cur- rent practice in designing SSL objectives i.c., op- timizing for ImageNet per- formance based on ResNet backbones could lead to misleading conclusions which do not generalize to other downstream scenar- ios. Also, the general belief that "the larger the better" in model size do not really hold in SSL, e.g., smaller ResNets can outperform larger ones even if they are pretrained following the same protocol. Let's say, based on these observations, one is compelled to hunt for a new architecture geared specifically towards SSL. Our top vs. bottom analysis suggests that it can be extremely tricky to find the right architecture topology with optimal parameter allocation across different layers. All this suggests that it is time to consider moving beyond handcrafted architectures in SSL. and start thinking about searching for optimal architectures as part of self-supe learning objectives.
Reshottimageeke
Resttet-Stnforr Cars
Spp
55
36.0
96.5
Battom para amm
Actumey
35.5
95.0
15.0
95.5
54
345
340
Top/B ottom Param Radio
Top/Bottom Faram Rabio
Tog/Bottom Faram Rati
Mobiistuttimageett
MobiuNat Stanford Cars
MobreNiitSports
0.40
Accureyy
28
Bistom para
26
52
0.20
15
TopeBottom Paramr Ratio
TogeBeettom Param Ratio
Top.Rootto Param Ratio
Figure 4: There is no winner in the top bottom battle in SSL. Except for ResNet-ImageNet (top-left), we see no strong trend that suggests either top-heavy or bottom-heavy networks perform better.
4 NEURAL ARCHITECTURE SEARCH FOR SSL
We turn to the idea of learning both the architecture topology and its network weights in an SSL framework, using NAS to improve SSL (rather than using SSL to improve NAS). It is important to draw clear distinction between our idea and prior work that used SSL improve NAS (Li al., 2021; Liu et al., 2020; 2019; Yan et al., 2020; Zhang et al., 2021b) as well as work that used NAS to improve supervised learning (Elsken al., 2019); our goal here is to show the benefit of hamnsssing NAS in aid of SSL and not for comparison with more recent SOTA NAS approaches.
While examining this idea appears to be straightforward, it requires careful design of experiments. The biggest hurdle is that both NAS and SSL require heavy compute resources; the former needs a search space large enough to cover a comprehensive range architecture topologies, while the latter requires large datasets and batch sizes to be effective. This calls for efficient framework to conduct our study. Furthermore, we need datasets large enough to pretrain the models on, and different enough to investigate the importance of detendeennd architectures in SSL. To meet desiderata, we choose ProxylessNAS (Cai et al., 2018) as our NAS algorithm. MobileNet as our search space, and ImageNet-11 1 1K and iNat2021 (Van Hom et al., 2021) as our pretraining datasets.

---

Under review as a conference paper at ICLR 2023
ImageNel images, 1000 classes)
INaturalist 2021 images, 10000
Small COntoo aatarett Lage sierra titirriingg d5000e
CIFAR (10/100 chAmAee
Oclord Flowers (500 ciessee)
FSVC
Stanford Cara. (185 cleopee))
Figure 5: Samples from datasets used in our study. We choose these datasets hecause of the apparent domain shift across them. ImageNet contains general yet coarsely categorized images compared to iNar2021, which contains an order of magnitude higher number of fine-grained categories; although some images look similar to each other, every image shown belongs to a different category highlighting the fine-grained nature of this dataset. The downstream datasets, except for CIFAR, are similarly fine-grained but on different domains.
4.1 EXPERIMENTAL SETUP
SSL objective. We use SimCLR (Chen et al., 2020a), one of the most well-established contrastive SSL framework. We follow the same augmentation methods and hyperparameter settings as in Chen et al. (2020a). While more recent SSL. works exist (Caron et al., 2020; Chen He, 2021), they are similar to SimCLR by utilizing contrastive learning based objective. We adopt SimCLR for its simplicity and leave analysis of other SSL approaches for future work.
NAS algorithm. We choose ProxylessNAS for two reasons: efficiency and flexibility. One-shot NAS algorithms produce an architecture topology in a two-step process. They first find an optimal cell structure by solving proxy task over small dataset (c.g., CIFAR-10) and smaller archi- tecture (e.g., 8 cells), and then stack/repeat the best found cell topology for the target task (e.g., ImageNet with 20 cells). This reduces complexity at the cost of flexibility and introduces an opti- mization gap (Chen et al., 2021), requiring strong correlation between proxy and the actual target datasets. In contrast, ProxylesNNAS produces an architecture by directly optimizing on a target task, as it can significantly ameliorate memory requirements of one-shot NAS methods. To ensure flexi- bility, it uses "supernet" with a broad range of candidate operations orchestrating depth (via zero operations), width (via wider convolutions), and block structure (by allowing for operations to differ by level). At each training step, it optimizes one "subnet" on the target task as a surrogate, which greatly reduces compute and memory requirements.
Datasets. We use ImageNet-1K and iNat2021 to pretrain our models and evaluate them on their validation sets as well as on 10 downstream datasets used in Section 3. We deliberately choose the two pretraining datasets as they exhibit widely different characteristics, i.e., ImageNet-IK K contains variety of objects and scenes, while iNat2021 contains fine- grained species covering the tree of life. The former contains many inorganic object categories not present in the latter. This creates a domain gap, which allows to investigate the importance of data data-dependent architectures in SSL.
iNat2021 contains 2.7 million images (twice the size of ImageNet) representing 10K species (ten times more than ImageNet). To investigate the effect of dataset size during architecture search and pretraining, we use both the full and the mini versions of iNat2021 - the latter contains 500K images representing the same 10K classes. This gives us pretraining datasets at three different scales: 500K (iNat202l-mini), 1. 1.2M (ImageNet), 2.7M (iNat2021).
Implementation details. For ProxylessNAS, we replace the original supervised classification loss with the contrastive loss of SimCLR and remove the latency loss as we currently do not consider hardware-constrraned scenarios. We follow the original training schedule, i.c., warmup phase for 40 epochs. which optimizes only the network weights and not the NAS parameters, followed by a search phase for 120 epochs. We use the SGD optimizer for network weights and the ADAM optimizer for NAS parameters, using initial learning rates of 0.25 and 0.1, respectively, and use the cosine decay schedule for both.
7

---

Under review as a conference paper at ICLR 2023
To make our experiments tractable, we use the MobileNetV2 search space which typically yields 3 to 18 million parameters; the ResNet search space is larger, yielding 20 to 60 million parameters. Working with smaller models means we use large batch sizes, which is important for contrastive learning to work effectively; we use the hatch size 640 given our computational budget. The can- didate set of NAS operations consists of mobile inverted bottleneck convolution (MBConv) with kernel sizes f3, 5, 7), expansion ratios {3, 6] and zero operations. A higher expansion ratio enables a wider network with more channels for convolutions, while zero operations allow for choosing to remove operations, thereby learning the optimal depth. Once the search is done, we take the architecture and discard the learned weights; we train it again from scratch using the SimCLR ob- jective on different datasets. This allows us to compare different architectures on fair ground. After pretraining, we conduct linear evaluation on all downstream datasets.
4.2 RESULTS AND DISCUSSION
Self-supervised architectures out- perform handcrafted architectures in SSL. Table compares our self-supervised architectures to Mo- bileNetV2 and ResNet18/50, by searching, pretraining, and evaluat- ing on ImageNet-IK,, iNat2021 and iNat2021-mint We report linear evaluation results on validation splits. The results show that our self- supervised architectures outperform MobileNetV2 by large margin, even with similar parameters (about 3M). The self supervieee architectures also beat ResNet18 and ResNet50, with even smaller model size than ResNet50. The superior downstream performance of our approach should not be attributed solely to NAS, the architecture search was performed without ever solving the downstream tasks. It is rather the incorporation of NAS into SSL that improved the quality of representations, leading to downstream performance boost. This shows the effectiveness of learning both the architecture topology and its weights in SSL.
Table 1: Searched architectures vs. handcrafted architecture
results. We search, pretrain, and evaluate ours on each of the three datasets in the last three columns. SSTA results (in gray) from Chen al. (2020a) for ImageNet and Cole ct al. (2021) for iNat21 require larger batch sizes and longer training.
Model
Params
Batch
Epochs
ImageNet
iNat21
iNar21
MobileV2
3.5M
640
100
41.9
30.2
13.6
Ours
3.3M
640
100
55.3
40.3
14.7
ResNer18 ResNet50
11M
640
100
49.8
30.3
20.1
23.5M
640
100
58.9
41.3
23.4
12-18M
640
100
39.1
25.1
ResNer50
23.5M
4096
1000
69.3t
50.61
Do self-supervised architectures generalize well to different data distributions? We take the three architectures searched on each dataset, discard their learned weights, and pretrain them on each dataset, yielding 9 pretrained models. We then evaluate the performance directly on validation splits of the respective datasets. Ta- ble 2 shows that transferring an architecture from iNat2021 to ImageNet leads to a marginal
Table 2: Self-supervised architecture transfer sults. We evaluate architectures in the cross-dataset setting, pretraining and evaluating the searched archi- tectures across three datasets (last three columns).
Searched on Params
ImageNet
iNar21
iNar21 mini
ImageNet iNai21 iNar21-min -ini
12-18M
59.1
21.5
23.9
12-18M
58.3
43.8
27.9
12-18M
58.0
22.4
25.1
performance drop compared an architecture optimized directly on ImageNet (59.1% to 58.3%);
both these architectures still outperform handcrafted ResNer18 (49.8%) with a comparable model size. However, transferring an architecture from ImageNet iNat2021 deteriorates performance significantly (43.8% to 21.5%). This implies an interesting finding, i.e., INat21-searde architec- tures seem be more resilient to domain shift than ImageNet-searched architectures. This could be due to the difference in dataset size (iNat21 has twice as many images as ImageNet), or due to the fine-gran ane-grained nature of iNat21 resulting in an overall more difficult instance discrimination task (Chen et al., 2020a) that leads to more discriminative representations. The effect of dataset size on architecture search also shown on iNat2 INat21-mini results. While transferring an architecture from ImageNet to iNat21-mii shows expected drop in accuracy (25.1% to 23.9%), transferring from the larger 1 improves performance (25. 1% to 27.9%). As both datasets are in the same domain and only differ in number of samples per class, higher search dataset size is the driving factor behind the gains in accuracy while pretraining on a smaller version of the dataset.
Downstream transfer experiments. The results above show that self-supervised architectures superior to handcrafted architectures when tested in -n-distibutiinn settings, which might reflect a practical use case of self- self-supervised pretraining in the real-world setting (e.g., one has access to only
8

---

Under review as a conference paper at ICLR 2023
Table 3: Downstream transfer results. We categorize downstream datasets in in-distribution (green) and out- of of-distribution (red) relative to the pretraining dataset based on class overlap: best viewed in color. We see that supe--uueeeieed architectures generally perform better on nn-distibuttin downstream scenarios.
Pretrain
Params
Pretrain Set
CUB
NABirds
CIFARIO
Oxford Flowers
Stanford Dogs
Food101
Sport
Stanford
Cars
MIT67
RGVC
MuhikeV2
3.5M
41.9
20.1
14.2
75.8
85.4
35.7
51.6
94.01
26,4
57.5
35.8
Ours
3.3M
55.3
31.8
24.4
78.8
91.7
49.2
61.7
94.3
30.4
62.5
38.1
ImNet
ResNet18
11M
49.8
27.0
19.0
79.9
89.9
44.1
55.8
94.6
27.8
62.1
36.7
ResNet50
23.5M
85.9
52.3
94.3
35.1
Ours
3-18M
59.1
34.3
26.1
81.8
92.2
51.0
64.7
94.7
33.2
66.1
39.4
ResNet18
11M
30.3
26.1
19.0
73.2
92.8
31.3
55.3
92.1
18.9
49.9
32.7
iNar21
RexNet50
23.5M
41.3
31.2
23.2
75.1
95.1
39.7
65.1
94.3
22.3
55.0
37.6
Ours
3-1MM
20.1
small labeled but large unlabeled data from the same distribution). We now evaluate our approach on downstream transfer scenario with possible domain shift and with much smaller datasets. To this end, we again use the 10 downstream tasks used in Section 3, which contain datasets coming from both in-distributions and out-of-distributions relative to the pretraining datasets.
Table summarizes the results (we color code -fuuttof-distribuution datasets based on our crude categorization; see appendix for our justification). We first compare self-supervised architec- tures to MobileNetV2 in the same parameter range (3.5M vs 3.3M; top two rows). We notice that self-supervised architectures significantly outperform MobileNerV2 in all datasets regardless of distributional shift. This is encouraging (i.e., self-supervised architectures can learn generaliz- able representations) but at the same time not totally surprising (i.e., MobileNet optimized for efficiency and not for accuracy). Next, we compare ours to ResNet18 that has a similar parameter range although belonging to a class of architectures much different from our search space. Ours out- performs R 8 on all pretraining and evaluation datasets by considerable margin. This shows that our approach is generalizable and can outperform architectures in the ResNet18 search space even though they are generally more computationally expensive than the MobileNet search space.
Finally, we compare ours to ResNet50 which is computationally heavier compared to ResNet18. We preface our analysis with caveat that supervuee architectures are almost half the capacity of ResNet50, limiting their representational power. Keeping this in mind, we see that our approach starts fail in some of the in-distribution and all of the diitii-iistiiuttion scenarios (red shaded cells). This is somewhat disappointing but perhaps expected: self-supervised architectures naturally encode inductive biases specific to the dataset they were optimized on. When a distributional shift happens, their performance can start deteriorating because out-of-domain data might require a dif- ferent set of inductive biases. The strong performance by ResNet50 imply that the model might be striking the right balance across those datasets in terms of inductive biases, but our results in Table and show that ResNet50 can be less effective on newly developed datasets such as iNat2021.
5 CONCLUSION
This work lays the ground for moving beyond handcrafted architectures in SSL. By conducting large-scale experiments with 116 architectures and 11 downstream tasks, we established extensive empirical evidence showing that there isn't one architecture that performs consistently well across different downstream scenarios in SSL. Motivated by this, we proposed to move beyond hand- crafted architectures and learn both an architecture topology and its network weights in SSL. We provided convincing results demonstrating that the self-supervised architectures significantly out- perform handcrafted MobileNetV2 and ResNet18 architectures on 11 downstream tasks, and com- petitively with ResNet50 even with almost half the model size. We re-emphasize that improvements are not solely due to NAS, as the architecture search was performed by solving SSL and not by optimizing directly downstream tasks as in the typical NAS setting.
Our work barely scratches the surface and opens up many doors for future directions. Will our findings hold for different architectures such as Transformers and different modalities such as video and text? How can we make architecture search more effective for SSL? Can ideas from domain generalization improve the transferability of self-supervised architectures in the distrri-uttiitooonn setting? Or is it even the right idea to expect learned architectures to generalize to widely different domains? We hope the readers are excited us investigate these challenging questions.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019. 1
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint
200.::2006..8254, 2021." 1
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:/812.0033: 00332, 2018. 2, 6, 15
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision, pp. 133-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020. 3,
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9660-9660, 2021. 3
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020a. 1, 2, 4, 8, 15, 17,
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information pro- cessing systems, 33:22243-22255, 2020b. 5
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive darts: Bridging the optimization gap for nas in the wild. International Journal of Computer Vision, (29(3):38-6555 2021. 7
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021. 1,3.7
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXhv:2003.04297 2020c.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv: 1710.0922, 2017. 3
Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongic. When does contrastive visual representation learning work? arXiv preprint arXiv.2105.05837, 2021. 1,
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-2482222 148-255.. Icee, 2009. 2, 16, 17, 18, 20
Debadeepta Dey, Shital Shah, and Sebastien Bubeck. Fear: A simple lightweight method to rank architectures. arXiv preprint arXiv:2106.04011 2021.
Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al. Neural architecture search: survey. J. Mach. Learn. Res., 20(55):1-21 2019. 3,
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self- self-supervised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 5414-5423, 2021. 3
Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 000.3299-3300 2021.
10

---

Under review as a conference paper at ICLR 2023
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728, 2018. 1,
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint 2011:20006.07333, 2020. 1,2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016. 1,2
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729 9738, 2020. 1,2
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li. Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16000 2022. 1,2
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017. 6
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- grained image categorization: Stanford dogs. In Proc. CVPR Workshop Fine-Grained Visual Categorization (FGVC), volume 2. Citeseer, 2011. 16, 20
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9404 0411,9413,211 2019. 3
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting sule-supervised visual repre- sentation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp.
2019. 2,
Simon Komblith, Jonathon Shlens. and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2661-2671, 2019. 3.4
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference computer vision work- shops, pp. 554-561, 2013. 16, 20
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 16, 20
Changlin Li, Tao Tang, Guangrun Wang, Jiefeng Peng, Bing Wang, Xiaodan Liang, and Xiaojun Chang. Bossnas: Exploring hybrid cnn-transformers with block- self-supervised neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12281-12291 2021. 3,
Li-Jia Li and Li Fei-Fei. What, where and who? classifying events by scene and object recognition. In 2007 IEEE 1 11th international conference on computer vision, pp. 1-8. IEEE, 2007. 16, 20
Feng Liang, Chen Lin, Ronghao Guo, Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang. Compu- tation reallocation for object detection. arXiv preprint arXiv: 1234, 2019.
Chenxi Liu, Piotr Dollár, Kaiming He, Ross Girshick, Alan Yuille, and Saining Xie. Are labels necessary for neural architecture search? In European Conference on Computer Vision, pp. 798- 813. Springer, 2020. 3, 6
11

---

Under review as a conference paper at ICLR 2023
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. URL BRLntttss///ppeeeeeiies net/Eorumm2 (tw.orummnndsleeeooossee
6,6
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 1306.5/5,, 2013. 16
Joseph Mellor, Jack Tumer, Amos Storkey, and Elliot J. Crowley. Neural architecture search without training, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 201 16, 20
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pp. 722-729. IEEE, 2008. 16, 20
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 1a::::00077370000 2018.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171- 5180. PMLR, 2019.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. 413-420. IEEE, 2009. 16. 20
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the pressive power deep neural networks. In international conference on machine learning, pp- 2847-2854. PMLR, 2017. 5,5
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chin Ching Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520, 2018.
Terrence Sejnowski. The unreasonable effectiveness of deep learning in artificial intelligence.
Proceedings of the National Academy of Sciences, 1 117448)33033330033 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 140Kv::140.15566 2014. 6
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843 852, 2017.
Christian Szegedy. Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016. 6
Lukas Tuggener, Jürgen Schmidhuber, and Thilo Stadelmann. Is enough to optimize enn architec- tures on imagenet? arXiv preprini 100.09990 2021. 2
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mae Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12884-128933 2021. 2,
Welinder, Branson, Mita, C. Wah, F. Schroff, Belongie, and P. Perona. Caltech-UCSS Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010. 16, 20
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. 2019. URL https://github. com/facchonkreserreddeeeett 2(3), 2019.
12

---

Under review as a conference paper at ICLR 2023
Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng. and Mi Zhang. Does unsupervised architecture repre- sentation learning help neural architecture search? Advances in Neural Information Processing Systems, 33, 2020. 6,6
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021a.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Xuanyang Zhang, Pengfei Hou, Xiangyu Zhang, and Jian Sun. Neural architecture search with random labels. arXiv preprint 11X1v:2101.11834, 2021b. 3, 6
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
Fig. 6 provides an overview of our work. We show that no single handcrafted architecture performs consistently well across different tasks. is therefore imperative to optimize for architecture topolo- gies along with network weights for specific task. Through extensive empirical results we show that such self-super architectures outperform their handcrafted counterparts in the same search space on the respective tasks.
Handcrafted Architecture
Architecture Search Space
SSL + Unlabeled dataset [A or orr C]
Self-supervised Architectures
A/B/C
Accuracy
Handerafted
Self-supervsed
Task
Task B
Task
Figure 6: Conventional SSL frameworks leam network weights for a fixed handcrafted architecture (left), We show that learning architecture topologies along with their weights can improve performance in SSL (right).
A SUPERVISED TRAINING PERFORMANCE OF SELF-SUPERVISED ARCHITECTURES
In addition to evaluating the performance the searched architectures for SSL, we analyze their supervised training performance. We use the searched architectures and directly train them, from scratch without any pretraining, on downstream datasets using the supervised labels. Results are summarized in Table 4. We include architectures searched on ImageNet and iNat21, and Mo- bileNetV2 for reference. It shows the searched architectures perform well even in the supervised setting, outperforming the handcrafted MobileNetV2 most of the downstream datasets. How- ever, a performance degradation observed out-o--distributioon datasets like Stanford Cars and FGVC Aircraft. This is in line with the discussion in Section 4.2 of the main paper where the searched architecture performances deteriorate with distributional shift. Nevertheless, for the more in-distribution datasets, obtain higher accuracies showing that the searched architectures suitable for supervised training as well.
Table 4: Supervised performance of searched architectures.
CUB
CIFAR10
CIFAR100
Food
Flowers
Sport
Cars
Aireraft
MobileNetV2
58.2
93.7
71.9
80.8
90.0
94.2
88.6
81.4
Ours (ImNet)
57.9
93.1
74.7
78.1
96.7
95.0
71.0
75.5
Ours (iNat21)
64.5
93.9
75.8
81.6
98.1
96.3
72.3
72.9
14

---

Under review as a conference paper at ICLR 2023
B CLASS MAPPING FROM IMAGENET TO DOWNSTREAM DATASETS
We provide a justification for characterizing downstream datasets as in-distributiionoutt--- distribution with respect to ImageNet shown in Table of the main paper. We provide rough class mapping between ImageNet and the 10 downstream datasets. Note that obtaining an exact class mapping is difficult due to only an approximate mapping existing between any 2 datasets. In addition, there can be classes which contribute to improved features for another class while still be- ing semantically different. For example, zebra (n02391044 (049) contribute improved features for horses (sorrel-n02388)) classes/superclassss. While some superclasses erarchy, we restrict to only those classes in the ImageNet-Ik dataset. Numbers in bracket denote the total number of classes roughly overlapping.
due to similar shapes. We now list datasets with corresponding ImageNet
contain additional subelasses the WordNet hi-
CIFAR-IO (270): vehicle (n4524313), bird (n1503061), feline (n2120997), frog (n1639765), dog (n2084071), sorrel (n2389026)
Stanford Dogs (120): dog (n2084071)
CUB (60): bird (n1503061)
NABirds (60): bird (n1503061)
Food101 (20): nutriment (n7570720), beverage (n7881800), foodstuff (n7566340), sand- wich (n7695965), bagel (n7693725). guacamole (n7583066), chocolate sauce (n7836838),
carbonara (n7831 146), french loaf (n7684084), pretzel (n7695742)
Stanford Cars (10): car (n2958343)
FGVC Aireraft (3): airliner (n2690373), warplane (n4552348), airship (n2692877)
Oxford Flowers (2): yellow lady's 's slipper (n12057211), daisy (n1 1939491)
MIT67 (0):
Sports(O):
Due to the inductive biases encoded during the search process specific to the dataset it is searched on, the self-supervised architecture performs well on more in-distribution datasets like CUB or NABirds. However, we see that for datasets like Stanford Cars and subsequent ones, there is lit- tle direct class overlap with ImageNet classes. This leads to lesser images being available for self-supervised pretraining which are in-distribution for these datasets. Consequently, due to the relatively -utto--disttiuution nature of these datasets we see in Table 3 of the main paper, our self- supervised architectures outperformed by the ResNet-50 baseline.
C ADDITIONAL IMPLEMENTATION DETAILS
We sample ResNet architectures by varying the number of blocks at each of the stages choosing from the set of 2, 3, blocks and choose the ones in the parameter range shown in Fig. 3 while also fitting in GPU memory. For MobileNets, we have 7 sequences (stages) and a higher vari- ation of the number of blocks from [2-6] while also choosing the width parameter from the set 1.0, 1.2, 1.4, 1.6, 1.8, 2.0 and choose the ones in the 2M-7M parameter range and fitting in GPU memory. Note that high number of blocks in the earlier stages take significantly more GPU mem- ory due to larger feature map sizes.
For the architecture search phase, of the main paper. We use a weight decay of 4e-5 for the weight parameters excluding batch normalization parameters. The initial convolution is a 3x3 convolution with stride 2. The network consists of stages with 4 cells in the first 5 stages and I cell in the last stage. By default, the number of channels each stage 24, 40, 80, 96, 192, 320, which is multiplied by a constant width multiplier. We downsample it by factor of 2 at the beginning of the first, second, third and fifth stage. Other architecture details are the default ones used in Cai et al. (2018). We use the same projection head as used normally for SimCLR Chen et al. (2020a) on top of the backbone network, which a 2048 dimensional hidden layer and 128 dimensional output layer. For evaluation, we remove the projection head and use the output of the network backbone the feature extractor. Augmentations are the same as in SimCLR with random resize scaling and cropping. flipping and color jitter. A temperature value of 0.1 is set for the contrastive loss.
the optimizer hyperparameters as explained in Sec. 4.1
15

---

Under review as a conference paper at ICLR 2023
Gonv3x3
MB6 7x7
MB6 3x3
MB6 7x7
MB6 7x7
MB6 5xd
MB6 Tx7
MB6 3x3
MB6 7xx
MB6 7x7
MB6 7x7
MB6 7
MB6 5x5
MB6 5x5
MB6 Exs
MB6 Tx7
MB6 7xx
MiB6
MB6
MB66
Searched architecture for ImageNet
Convax3
MB6 7x7
MB: 3x3
MB: 3x3
MBS 5x3
MBS 7x7
MB6
MB6 Sx5
MB6 7x7
MBG 7x7
MBS 7x7
MBS 7x7
MB6 5x55
MB6
MB6 7x7
7x7
MBB
MB6
MB6
Searched architecture for iNat21
Convax3
MB3 3x3
MB3 3x3
MB3 7x]
MB3 1xl
MBB 7x7
3x3
MB6 3x3
MB6 Sx5
5MS
MB6 7x7
MB6 9x5
MB3 5x5
MB6 3x3
MB6 7x7
MB6 7xx
MB6 7x7
MB3 7x7
MBG 5x3
MBS
Searched architecture for iNat21Mim
Figure 7: Self-supervised architectures for different pretraining datasets. MB3 MB6 are the mobile inverted convolutions with expansion ratio and respectively. see that majority of the preferred convolutions is MB6 7 x 7 suggesting that the network prefers convolutions with more parameters for the self-supervised regime due to lots of data. For smaller datasets like iNat2 MB6 convolutions are not as strongly preferred.
D VISUALIZING SELF-SUPERVISED ARCHITECTURES FOR DIFFERENT PRETRAINING DATASETS
We visualize the types of convolutions searched at 1.75x width multiplier for the different pre- training datasets: ImageNet, iNat21 and Mini. Results are shown in Fig. 7. MB3 and MB6 are the mobile inverted convolutions with expansion ratio of and 6 respectively. The grey lines denote the downsampling of image due to strided convolutions. All architectures are followed by pooling layer to reduce the image size to I x 1. In contrast to standard handcrafted architec- tures, larger 7 convolutions are preferred even in the later stages of the network. We also see that the majority of the preferred convolutions is MB6 7 x 7 suggesting that the network prefers convolutions with more parameters for the self-supervised regime due to lots of data. This less preferred in smaller datasets like iNar21 where MB3 convolutions earlier stages of the network. is difficult to draw conclusions on the type of network preferred between ImageNet and iNat21 showing that imperative search for optimal architecture rather than handcraft them.
common especially in
E DOWNSTREAM DATASET PERFORMANCE CORRELATION WITH IMAGENET
We show the downstream dataset correlation for all 10 downstream datasets in addition to ImageNet- 1K Deng al. (2009): CIFAR 00/100 Krizhevsky et al. (2009), Stanford Cars Krause al. (2013) and Dogs Khosla et al. (2011), CUB-200 Welinder et al. (2010), MIT-67 Quattoni & Torralba (2009), SVHN Netzer et al. (2011), Flowers- 102 Nilsback Zisserman (2008), AirC-Air Maji et al. (2013), Sports8 Li & Fei-Fei (2007). These are shown for both ResNets (Fig. 8) and MobileNets (Fig. 9). We see similar results for the 5 datasets in addition to those shown Fig. of main paper. High correlation exists for datasets which visually similar to ImageNet while is less correlated for datasets which are out of domain. For MobileNets this correlation is even less pronounced with high variance in performance higher ImageNet accuracies.
16

---

Under review as a conference paper at ICLR 2023
005
05.
140
10.5
Irapilit.ccomm
Iragatec
Iragelie Accurs)
Iageeear Acceacy
Figure 8: ImageNet performance correlation with different downstream datasets for various ResNets. show linear evaluation top-1 accuracy of ImageNet (x-axis) vs. different downstream datasets (y-axis) obtained from variations of ResNet; all models are pretrained on ImageNet-IK Deng et al. (2009) using Sim- CLR Chen et al. (2020a) under the same protocol. The solid lines and shaded areas indicate fitted regression models and their confidence intervals.
OutiihieDDetrrdd (en77]
Meb kafullldd- CF
Mabi
21.
714
Mociider
111
153
343
204
1ragetet ACCLECO
Figure 9: ImageNet performance correlation with 10 different downstream datasets for various Mo- bileNets. We show linear evaluation top-1 accuracy of ImageNet (x-axis) vs. different downstream datasets (y-axis) obtained from variations of MobileNet; all models pretrained on ImageNet-IR IK Deng al. (2009) using SimCLR Chen et al. (2020a) under the same protocol. The solid lines and shaded indicate fitted regression models and their confidence intervals.
17

---

Under review as a conference paper at ICLR 2023
meteeeee (am
MITST (pe55)
06.5
315
66.5
$1.5
51.
55..0
563
131
20.5
51.0
283
Restetcollii
74.0
73.0
12.5
42.5
111
0.4
345
12.
21.
Tarata (nP))
Tarom In
Figure 10: Dataset performance correlation with 10 different datasets for various ResNets. show the model size in terms of parameter counts (X-axis) vs. top-1 accuracy on different datasets obtained from variations of ResNet-like architectures; all models are pretrained on ImageNet- K Deng et al. (2009) using SimCLR Chen et al. (2020a) under the same protocol. The solid lines and shaded areas indicate fitted regression models and their confidence intervals.
Midiidee
226
42.0
23
35
235
Parane
Ryrnn
Planchh
264
62
475
28
365
1
15
925
24.
925
Parara d1 MT
Naruru MD
TaraTo Io
Paramu 01
Tarmr In
Figure 11: Dataset performance correlation with 10 different datasets for various MobileNets. We show the model size in terms of parameter counts (x-axis) vs. top-1 accuracy on different datasets obtained from variations of MobileNet-like architectures; all models are pretrained ImageNet- 1K Deng al. (2009) using SimCLR Chen et al. (2020a) under the same protocol. The solid lines and shaded areas indicate fitted regression models and their confidence intervals.
18

---

Under review as a conference paper at ICLR 2023
Pearsen trear comdars conffic
comdlatto coutddaam
00
CMM1000
Oega
0oga
MB767
3/2
5:02
3050
4.00
4.35
0..05
M0142
0.53
0.50
0:55
227
0.02
0.34
-04
0.0
3.37
3:25
2:27
0.24
-22
6.19
Cu3
5.45
06
5:51
5.3
6.01
1.115
Rawess
2.33
5.43
2:57
21
0.22
00:25
0.14
62
SWM
5.42
05
25
0:20
5.27
2.4L
244
0.05
0:
0.10
9P-M
2.22
5.42
2/33
3:34
222
6.47
0.34
0.27
Phwers
5.22
6.35
2.77
02
2.24
0:22
0.22
0.35
-08
Sants
311
0.07
3M
0.12
4.00
0.67
0:17
46.13
Speri
6.68
6:5
E.11
3.34
5.0
6.11
Arooh
7.15
0.4
3.2
0.23
1034
6:27
Aiosi
4.24
6:38
4.37
9:50
2.15
922
6.2
6.17
4.08
4.16
0.77
4.74
9:17
-.00
GUM
mopil
Figure 12: Linear and rank correlation of ResNets between different pairs of 11 datasets We show correla- tion between every pair of top-1 accuracy on 11 downstream tasks obtained from ImageNet-pretraing ResNets.
We sec strong correlation except for ones highly similar to the data the models were originally pretrained on, e.g., CIFAR-IO/ 10010- and Dogs
Irageter
0.52
5:72
0.71
0.81
2.38
Inagelee
0.27
0.53
0.32
2:27
Daga
0.9
088
6.8
0:73
0.52
05
3.51
0:44
0.4
-30
Dogs
0.22
2:45
0:46
0:37
0:54
3.34
5.09
0.37
0.16
2.
Fawar
0.02
0.5
5:73
0.73
0:42
0.60
3/4
5:42
0.32
0.53
Flwardd
0:71
55
0.52
0.25
0.52
3.33
5:15
0:17
0.57
0.01
0.93
3.25
0.56
0.82
0.
3.55
5.82
0.45
0.51
CEPRID
0:51
46
5.52
0.33
0.39
3.41
5:35
0.47
0:33
0.12
0:
3.73
5:60
0:40
0.66
3.63
0:55
0:52
5.76
0:23
3:01
0:20
0.13
0RA1200
SWNV
CMr
0.53
0:52
3.43
5.43
0.02
0.52
0.00
4.22
Cer
3.39
0.33
0.22
3.55
0.22
0.
20
0.51
0.44
3.4L
2.42
0.32
0.21
0.35
237
0:42
0.20
GP43300
0.22
0.37
2:47
0.47
9.32
3.37
0:25
0.0
0.00
Aacio
0.4
0.4
3.32
2.42
0.32
0.31
0.24
3.03
C.22
0.04
41007
0.22
0.1
5.67
0.20
0.17
0.24
3.25
0:54
6.28
0.00
-00
0.02
0.01
2,01
.0.2
0..2
10.00
2.11
0:24
0.11
.0.2
-2.03
10.12
-.33
0.23
C.R
Aroest
Dege
09000
MIST
Figure 13: Linear and rank correlation of MobileNets between different pairs of 11 datasets We show correlation between every pair of top-1 accuracy 11 downstream tasks obtained from ImageNet--retrained MobileNets. The correlation
F DATASET PERFORMANCE AS A FUNCTION OF NUMBER OF PARAMETERS
We show the dataset correlation with respect to number of parameters for 5 more datasets in addition to that shown in Fig. 4 of main paper. Fig. 10 summarizes the results for ResNets while Fig. 11 shows results for MobileNets. We see that similar results hold for the additional datasets where more parameters, and consequently larger networks, does not always lead to better downstream performance.
G LINEAR AND RANK CORRELATION FOR RESNETS/MOBILENETS
We show the summary of the correlation across different datasets for both ResNets (Fig. 12) and MobileNets (Fig. 13). In addition to Spearman's 's rank correlation coefficient, also show Pearson's linear correlation coefficient. While Pearson's 's linear coefficient is higher in the case of MobileNets, the linear fit still exhibits high variance for higher ImageNet accuracies, as seen in Fig. 9.
19

---

Under review as a conference paper at ICLR 2023
Octgve repere
Delaed clar000
Droodd cegs322
Octare rid
Decost + cu0200
Modd
Figure 14: ImageNet performance correlation with 10 different downstream datasets for various ResNets, NobileNets and searched architectures. The searched architectures outperform MobileNets while being comparable with ResNets at fewer parameters.
H
DATASET LICENSES
Table 5 lists some datasets we used and their licenses.
Table 5: Licenses datasets.
Dataset
License
CIFAR-10 Krizhevsky al. (2009)
MIT
CIFAR-100 Krizhevsky et al. (2009)
MIT BSD 3-Clause
ImageNet Deng et al. (2009)
Sport8 Li & Fei-Fei (2007)
CCO: Public Domain
Stanford Dogs Khosla et al. (2011)
BSD 3-Clause
Stanford Cars Krause et al. (2013)
BSD 3-Clause
CUB-200 Welinder et al. (2010)
Data files @ Original Authors MIT
MIT-67 Quattoni & Torralba (2009)
SVHN Netzer al. (2011)
CCO: Public Domain
Flowers- 102 Nilsback & Zisserman (2008)
GNU General Public License, version 2
20


Reviews:
[{"review_id": "ze960HmjZFC", "reviewer": "Reviewer_JRjb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attempts to combine the neural architecture search (NAS) process with self-supervised learning (SSL) so as to learn structures suitable for pre-trained datasets (e.g. ImageNet-1K, iNat-2021) in a self-supervised task like SimCLR. Extensive analysis and experiments are included to demonstrate its arguments.", "summary_of_the_review": "This paper wants to show that for SSL, the architecture obtained by NAS will have a significant advantage over the handcrafted architecture. However, neither the theoretical analysis nor the experimental part can exclude the benefits from the NAS itself, and no further experiments have been conducted for other self-supervised learning methods.", "strength_and_weaknesses": "Strength\n- Extensive experiments and analysis.\n- Interesting perspective and attempt to try NAS in self-supervised learning.\n\nWeaknesses\n- In short, I think this paper does not decouple the advantages of NAS itself from SSL, while the architecture obtained from the search may have significant advantages in terms of MAdds.\n\n-----\n\n- **About the statements in Section 3** I agree with this statement \"ImageNet performance is not indicative of downstream performance for SSL\" alone, as evidenced by many previous attempts in pixel-level contrastive learning like DenseCL and PixPro. However, the statement in here does not support the main point of this paper. \n\t- First of all, the paper does not give a specific setting for the 116 sampled models, which means that many of them may be in a situation where they perform poorly and are unlikely to be adopted. \n\t- In addition, the sampling of the MobileNet structure is too concentrated in terms of the size and performance of the sampled models, as illustrated by the distribution of ImageNet results for MobileNet in Figure.2, the model size for MobileNet in Figure.3, and the top/bottom param ratio in Figure.4. \n\t- And this means that the conclusion obtained using spearman rank correlation on such a small space has a high probability of being problematic. We can also see from Figure.13 in the Appendix that MobileNet presents better correlations than ResNet on almost most of the datasets when Pearson coefficients are used. For the few remaining datasets, the lower overlap (e.g. Aircraft) or higher performance (e.g. Sports 8, also too small with only about 1.5K images in total) do not make it universally available.\n\t- At the same time, if the analysis is performed only at the architectural level, the influence of SSL on it should be stripped, because this part of SSL are pre-trained on ImageNet-1K while using linear evaluation as a metric, which will inevitably be helpful for ImageNet-1K related tasks (e.g. CIFAR). If measured using supervised training, I think the performance on ImageNet would be at least highly correlated with the performance of other **classification tasks** (Especially considering the higher correlation already seen in terms of Pearson coefficients so far).\n\n- The experimental section is missing an important baseline, i.e., the performance of the model obtained by using supervised training instead of SSL for search.\n- For MobileNet, MAdds is a very important metric. The model given in the paper seems to have an overall stride of only 16 (inferred from Appendix D), and the latency loss is removed in the search process, which may result in a model with significantly larger MAdds than MobileNetV2.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "zO6rRX7nMqT", "reviewer": "Reviewer_ZDfX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducts an empirical investigation on the role of architectures in self-supervised learning. Based on the empirical study, the authors apply the neural architecture search in the current SSL framework and demonstrate improved performance in several downstream tasks.", "summary_of_the_review": "I recognize the value of empirical study and believe the architecture choice is crucial for semi-supervised learning.\nHowever, the novelty of the paper is limited and the message of the study is vague, unsurprising, and not rigorous. The paper will be more suitable for venues like workshops or benchmark tracks.", "strength_and_weaknesses": "**Strength**: the structure of the paper is clear and the paper is well-written. The experiments are sufficient and well-designed.\n\n\n**Weaknesses**:\nThe novelty of the paper is very limited. There are several messages from section 3:\n\n    1. ImageNet performance is not indicative of downstream performance for SSL.\n    2. Larger networks do not always perform better in contrastive SSL.\n    3. There is no winner in the battle of top vs. bottom heavy networks in SSL.\n    4. It is necessary to allocate the right portion of parameters to different layers of a given network topology.\n\nAll the messages are trivial,  not mathematically rigorous, and are generally observed in different machine-learning tasks.    Although I do agree that the empirical study is sufficient  as a motivation and can support the  general key takeaway: ``we need\nto move beyond handcrafted architectures in SSL'', but section 3 doesn't convey any new insights about SSL. \n\nThe second part of the paper (section 4) applies NAS in SSL, this combination is also a trivial extension and the resulting improvement is not surprising. \n\nI suggest the author treat the empirical study as the first step towards designing a new NAS algorithm to improve the SSL results or provide a more theoretical understanding of the described phenomenon.\n\n\n**Others**:\n1. The title of table 1 is misleading,  ``NAS vs handcraft architecture '' will be more clear.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "pne0s6lk0lh", "reviewer": "Reviewer_6dkS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The study provides evidence that a network architecture plays a significant role in contrastive SSL, by utilizing 116 variants of ResNet and MobileNet architecture, which were evaluated across 11 downstream tasks in the contrastive SSL setting.\nIt showed that no one architecture demonstrated a consistently good result, thus suggesting future researchers focus on learning architecture as well as the weights of the network in the SSL setting. They conducted two experiments: 1) network variation experiment on downstream tasks with observation of correlation between the models in downstream performance, and 2) applying NAS algorithm to the SSL setting to search for the optimal architectures on unlabeled pretraining dataset via contrastive learning.\n", "summary_of_the_review": "From the extensive study, readers can agree to the statement that there is no one architecture that performs well across different downstream tasks in SSL, and that NAS+SSL can be a solution for mitigating the problem, where researchers have to handcraft the network architecture. However, as mentioned in the section above, the novelty of this idea is limited in that the combination of NAS and SSL has been experimented widely in the literature.\nHowever, this study opens up many possible research questions regarding the effectiveness on architecture search for SSL, thus can be act as a ground research of future studies.", "strength_and_weaknesses": "[Strengths]\n\nS1: The study found out the implicit underlying assumption of the current literature and pointed out that it can be incorrect.\n\nS2: The results have shown that the SSL architecture outperforms the handcrafted architectures, and have included concrete experiments regarding the distributional shift.\n\n[Weaknesses]\n\nW1: Only one optimization objective (SimCLR) has been used. They could have checked if the proposing method works across many different learning objectives.\n\nW2: Idea of combing NAS and SSL is not novel.\n\nW3: In the downstream transfer experiment, the searched architecture did not show promising performance on out-of-distribution dataset, except for the comparison of MobileNetV2, and this results are not surprising.\n\nW4: Although 116 variations of network were experimented, there are just two backbone network architectures.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hW0hPExN-M", "reviewer": "Reviewer_5FVM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the role of architectures in self-supervised learning. The authors conduct exhaustive experiments with different architectures and downstream datasets under the setting of SSL. They show that there is no one network that performs consistently well across the scenarios and hence they propose to learn the network architectures in SSL. Experimental results show that self-supervised searched architectures outperform handcrafted ones across different downstream datasets.", "summary_of_the_review": "I am willing to raise my score if the authors can address my concerns.", "strength_and_weaknesses": "Pros:\n1. The motivation is clear. I think it is important to rethink the design of architectures in the context of self-supervised learning.\n2. This paper provides insight for future research. For instance, Fig. 3 shows that larger models do not always perform better in SSL, which is counterintuitive but interesting.\n\nCons:\n1. Can the problems of Figures 1 to 3 be solved if the author's proposed method (NAS+SSL) is used? Is it true, for example, that the larger the size of the model discovered by NAS, the better the performance? Are the problems with the hand-designed models in Figure 2 still present? I have the same question for Figures 1&3 and I hope the authors can draw Figures 1 to 3 with the NAS searched model for comparison. \n\n2. What can we conclude from the results of the NAS to design a model for SSL? Currently, SSL gradually surpasses supervised learning on various downstream tasks by pretraining weights. So I am curious if NAS under SSL can also outperform NAS under supervised learning, by pretraining the model structure.\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject