Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper conducts an empirical investigation on the role of architectures in self-supervised learning. Based on the empirical study, the authors apply the neural architecture search in the current SSL framework and demonstrate improved performance in several downstream tasks.
strength_and_weaknesses: **Strength**: the structure of the paper is clear and the paper is well-written. The experiments are sufficient and well-designed.


**Weaknesses**:
The novelty of the paper is very limited. There are several messages from section 3:

    1. ImageNet performance is not indicative of downstream performance for SSL.
    2. Larger networks do not always perform better in contrastive SSL.
    3. There is no winner in the battle of top vs. bottom heavy networks in SSL.
    4. It is necessary to allocate the right portion of parameters to different layers of a given network topology.

All the messages are trivial,  not mathematically rigorous, and are generally observed in different machine-learning tasks.    Although I do agree that the empirical study is sufficient  as a motivation and can support the  general key takeaway: ``we need
to move beyond handcrafted architectures in SSL'', but section 3 doesn't convey any new insights about SSL. 

The second part of the paper (section 4) applies NAS in SSL, this combination is also a trivial extension and the resulting improvement is not surprising. 

I suggest the author treat the empirical study as the first step towards designing a new NAS algorithm to improve the SSL results or provide a more theoretical understanding of the described phenomenon.


**Others**:
1. The title of table 1 is misleading,  ``NAS vs handcraft architecture '' will be more clear.



clarity,_quality,_novelty_and_reproducibility: **Clarity**:
The discussions of the experiment setting and results are sufficient. However, an introduction to the SSL and NAS is required to make the paper self-contained.

**Quality**:
The experiment section is qualified but 


**Novelty**:
There is no new model or algorithm provided in the paper.  The heuristic conclusion from the empirical study is not surprising and lacks of scientific rigors. The novelty of applying NAS in SSL is limited.

**Reproducibility**:
No code is provided for this empirical study, so the difficulty of reproducing the results is high.
summary_of_the_review: I recognize the value of empirical study and believe the architecture choice is crucial for semi-supervised learning.
However, the novelty of the paper is limited and the message of the study is vague, unsurprising, and not rigorous. The paper will be more suitable for venues like workshops or benchmark tracks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The study provides evidence that a network architecture plays a significant role in contrastive SSL, by utilizing 116 variants of ResNet and MobileNet architecture, which were evaluated across 11 downstream tasks in the contrastive SSL setting.
It showed that no one architecture demonstrated a consistently good result, thus suggesting future researchers focus on learning architecture as well as the weights of the network in the SSL setting. They conducted two experiments: 1) network variation experiment on downstream tasks with observation of correlation between the models in downstream performance, and 2) applying NAS algorithm to the SSL setting to search for the optimal architectures on unlabeled pretraining dataset via contrastive learning.

strength_and_weaknesses: [Strengths]

S1: The study found out the implicit underlying assumption of the current literature and pointed out that it can be incorrect.

S2: The results have shown that the SSL architecture outperforms the handcrafted architectures, and have included concrete experiments regarding the distributional shift.

[Weaknesses]

W1: Only one optimization objective (SimCLR) has been used. They could have checked if the proposing method works across many different learning objectives.

W2: Idea of combing NAS and SSL is not novel.

W3: In the downstream transfer experiment, the searched architecture did not show promising performance on out-of-distribution dataset, except for the comparison of MobileNetV2, and this results are not surprising.

W4: Although 116 variations of network were experimented, there are just two backbone network architectures.
clarity,_quality,_novelty_and_reproducibility: [Clarity]
1. Clarified the assumptions of the current research.
2. Clarified the main objective of the study is to show that the choice of network is highly impactful in SSL and handcrafting the architecture is very hard.
3. Authors raised research questions in a clear context.
4. Clearly states the three main contributions
5. Clearly explains the reason behind the choice of the NAS algorithm and dataset


[Quality]
1. Appropriate references were made.
2. Tables, figures, and appendix are effectively supporting the arguments made.


[Novelty]
This work possesses novelty to a certain degree in that the author(s) tried to use NAS in aid of SSL, and conducted a large scale experiments on the variant of networks. However, the novelty is not substantial (limited) as there are a number of work that have similar approach in combining NAS and SSL. A preprint titled ‘CSNAS: Contrastive Self-supervised Learning Neural Architecture Search via Sequential Model-Based Optimization’ (https://arxiv.org/abs/2102.10557) is one of the examples. Despite the fact, as author(s) mentioned, in the beginning of Section 4, there is a distinction between their idea and prior work.


[Reproducibility]
1. Provided the details of 1) how they varied the architecture of each network they used , 2) the number of epochs and batch size when pretraining with the machine they used for training.
2. Provided the metric used (Spearman’s rank correlation coefficient) when evaluating the correlation between ImageNet and downstream performance.
summary_of_the_review: From the extensive study, readers can agree to the statement that there is no one architecture that performs well across different downstream tasks in SSL, and that NAS+SSL can be a solution for mitigating the problem, where researchers have to handcraft the network architecture. However, as mentioned in the section above, the novelty of this idea is limited in that the combination of NAS and SSL has been experimented widely in the literature.
However, this study opens up many possible research questions regarding the effectiveness on architecture search for SSL, thus can be act as a ground research of future studies.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper attempts to combine the neural architecture search (NAS) process with self-supervised learning (SSL) so as to learn structures suitable for pre-trained datasets (e.g. ImageNet-1K, iNat-2021) in a self-supervised task like SimCLR. Extensive analysis and experiments are included to demonstrate its arguments.
strength_and_weaknesses: Strength
- Extensive experiments and analysis.
- Interesting perspective and attempt to try NAS in self-supervised learning.

Weaknesses
- In short, I think this paper does not decouple the advantages of NAS itself from SSL, while the architecture obtained from the search may have significant advantages in terms of MAdds.

-----

- **About the statements in Section 3** I agree with this statement "ImageNet performance is not indicative of downstream performance for SSL" alone, as evidenced by many previous attempts in pixel-level contrastive learning like DenseCL and PixPro. However, the statement in here does not support the main point of this paper. 
	- First of all, the paper does not give a specific setting for the 116 sampled models, which means that many of them may be in a situation where they perform poorly and are unlikely to be adopted. 
	- In addition, the sampling of the MobileNet structure is too concentrated in terms of the size and performance of the sampled models, as illustrated by the distribution of ImageNet results for MobileNet in Figure.2, the model size for MobileNet in Figure.3, and the top/bottom param ratio in Figure.4. 
	- And this means that the conclusion obtained using spearman rank correlation on such a small space has a high probability of being problematic. We can also see from Figure.13 in the Appendix that MobileNet presents better correlations than ResNet on almost most of the datasets when Pearson coefficients are used. For the few remaining datasets, the lower overlap (e.g. Aircraft) or higher performance (e.g. Sports 8, also too small with only about 1.5K images in total) do not make it universally available.
	- At the same time, if the analysis is performed only at the architectural level, the influence of SSL on it should be stripped, because this part of SSL are pre-trained on ImageNet-1K while using linear evaluation as a metric, which will inevitably be helpful for ImageNet-1K related tasks (e.g. CIFAR). If measured using supervised training, I think the performance on ImageNet would be at least highly correlated with the performance of other **classification tasks** (Especially considering the higher correlation already seen in terms of Pearson coefficients so far).

- The experimental section is missing an important baseline, i.e., the performance of the model obtained by using supervised training instead of SSL for search.
- For MobileNet, MAdds is a very important metric. The model given in the paper seems to have an overall stride of only 16 (inferred from Appendix D), and the latency loss is removed in the search process, which may result in a model with significantly larger MAdds than MobileNetV2.

clarity,_quality,_novelty_and_reproducibility: - Good clarity and reproducibility.
- Please refer to the previous section for possible problems in quality and novelty.
summary_of_the_review: This paper wants to show that for SSL, the architecture obtained by NAS will have a significant advantage over the handcrafted architecture. However, neither the theoretical analysis nor the experimental part can exclude the benefits from the NAS itself, and no further experiments have been conducted for other self-supervised learning methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the role of architectures in self-supervised learning. The authors conduct exhaustive experiments with different architectures and downstream datasets under the setting of SSL. They show that there is no one network that performs consistently well across the scenarios and hence they propose to learn the network architectures in SSL. Experimental results show that self-supervised searched architectures outperform handcrafted ones across different downstream datasets.
strength_and_weaknesses: Pros:
1. The motivation is clear. I think it is important to rethink the design of architectures in the context of self-supervised learning.
2. This paper provides insight for future research. For instance, Fig. 3 shows that larger models do not always perform better in SSL, which is counterintuitive but interesting.

Cons:
1. Can the problems of Figures 1 to 3 be solved if the author's proposed method (NAS+SSL) is used? Is it true, for example, that the larger the size of the model discovered by NAS, the better the performance? Are the problems with the hand-designed models in Figure 2 still present? I have the same question for Figures 1&3 and I hope the authors can draw Figures 1 to 3 with the NAS searched model for comparison. 

2. What can we conclude from the results of the NAS to design a model for SSL? Currently, SSL gradually surpasses supervised learning on various downstream tasks by pretraining weights. So I am curious if NAS under SSL can also outperform NAS under supervised learning, by pretraining the model structure.






clarity,_quality,_novelty_and_reproducibility: Clarity: Good

Quality: Good

Novelty: Fair

Reproducibility: Fair
summary_of_the_review: I am willing to raise my score if the authors can address my concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper discusses the role of architecture in contrastive SSL, finding that there is no clear recommended architecture that provides good downstream task performance. The authors therefore argue to search for an appropriate architecture. Whilst the authors back us this statement with experimental evidence, the main issue the reviewers had was how significant this observation is. There is no one willing to champion this paper and I think the consensus is that whilst this is an interesting piece of work, it doesn't rank highly enough for ICLR.
justification_for_why_not_higher_score: There isn't enough enthusiasm amongst the reviewers (nor myself) to support this paper to be accepted in its current form. It's essentially an interesting observation (which also isn't particularly surprising).
Decision: Reject