Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper provides a comprehensive study and a set of improvements for the self-supervised method MoCo. First, the authors challenge the common assumptions that the training dataset has to be object-centric, not long-tailed and the crops overlapping, which provides very useful insights. Then, the authors propose a bag of tricks to improve the performance of MoCo in practice.
main_review: ### Good:

- Very interesting and practically useful analysis of MoCo in Section 3. I would love to see a more extensive study on this matter in the paper.
- Extremely useful for practitioners bag of tricks presented in Section 4. I particularly appreciate k-NN positives presented in Section 4.3.

### Bad:

- Submission to the wrong venue. While the paper contains very useful content, there is no technical novelty at all, rather a set of experiments applying one model to different datasets and providing a bag of tricks for better performance. This would be a perfect fit for representation learning workshop, or perhaps a purely vision conference.
- The paper is lacking a holistic message. While Section 3 studies the inner working of MoCo and how it performs in different settings, Section 4 proposes technical improvements that have little to do with discoveries in Section 3. The paper has a feel of "we've made these experiments and wrote about all of them in this paper".
- It is not clear whether the improvements will translate to other self-supervised models rather than MoCo and whether it will translate to longer training schedules on full-size training datasets, i.e. the mode where SoTA self-supervised results are reported.

### Questions:

- It is surprising for me that having littler overlap between crops in scene-centric training datasets does not affect the performance on the target dataset (see Figure 2-3). While I can understand this effect to some extent when testing on a scene-centric dataset, what would happen if the test dataset is object-centric?
limitations_and_societal_impact: -
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents rigorous analyses of self-supervised learning from different aspects. Here are their findings:

1. The pretraining dataset should match the target dataset in terms of being object- or scene-centric.
1. MoCo is robust to changes in the class distribution of the dataset (pre-trained on uniform ImageNet vs long-tailed ImageNet)
1. The model trained on data with discriminative features (ImageNet) performs equally well with the model trained on data without (BDD100K) in the fine-tuning setting for semantic segmentation tasks.

The authors also propose three strategies to improve MoCo:
1. Multi-scale constrained cropping data augmentation
1. Mixture of standard augmentation and AutoAugment
1. Including nearest neighbors as positive pairs
main_review: ### Strength
- This paper provides comprehensive and rigorous studies for different aspects of the pre-training data being used for SSL.
- Results in Tab. 2 challenge the observation in [38, 39]. The authors identify that the non-overlapping between views is in fact not the major issue. What really matters is to match the pre-training and the target dataset in terms of being object- or scene-centric. Analyses in Fig. 1 and Fig. 2 further support their claims.
- The proposed augmentation and training strategies in Sec. 4 are effective for improving MoCo's performance.

### Weakness

- The connection between the first part of the paper (Sec. 3) and the second part of the paper (Sec. 4) is weak. It looks like a concatenation of two papers. The reviewer expects to see how the analyses from Sec. 3 motivate the strategies proposed in Sec. 4.

- Is the proposed strategies in Sec. 4 generalizable to other SSL methods? For example, a clustering-based method like SwAV.

- For fine-tuning results, the reviewer suggests including baseline methods of random init for reference.

- In Sec. 3.3, the experiment of BDD on linear evaluation is missing. Is it possible that training on BDD has a negative effect in the linear evaluation setting?

- In Sec 4.2, the authors only explain why using AutoAugment leads to worse results. The explanations/analyses of why standard + auto-augment leads to better performance are not provided.
limitations_and_societal_impact: - (copy and paste from above) Is the proposed strategies in Sec. 4 generalizable to other SSL methods? For example, a clustering-based method like SwAV.

- It is still unclear what kind of feature, bias, invariance, etc, the model captures when pre-trained on different datasets given only the performance in the linear evaluation setting and the fine-tuning setting presented in Sec. 3. For example, does the model learn a more robust representation? Is the model biased to shape or to texture?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies the current stream of instance discrimination-based self-supervised learning algorithm (MoCo) beyond the typical object-centric dataset, i.e., ImageNet. Experiments are conducted on scene-centric datasets, long-tail datasets, and general-domain datasets. Results show that MoCo works well on these datasets. Furthermore, the authors studied multi-crop and stronger data augmentation for MoCo and showed that these enhanced augmentation techniques improved performance.
main_review: Strengths:
-	The paper is clearly rewritten and very easy to follow.
-	The paper studies some of the essential problems in SSL. SSL models are trained mostly on ImageNet dataset, which by itself has inductive bias from data collecting procedure. Studying on other types of datasets, i.e., scene-centric and general-domain datasets, is an important steppingstone to better scale up the training of SSL models.
-	Experiments are thorough and convincing. Benchmarks are properly set up and follow the common practice.

Weaknesses:
-	Missing supervised baselines. Since most experiments are done on datasets of scale ~100k images, it is reasonable to assume that full annotation is available for a dataset at this scale in practice. Even if it isn’t, it’s an informative baseline to show where these self-supervised methods are at comparing to a fully supervised pre-trained network.
-	The discussion in section 3 is interesting and insightful. The authors compared training datasets such as object-centric versus scene-centric ones, and observed different properties that the model exhibited. One natural question is then what would happen if a model is trained on \emph{combined} datasets. Can the SSL model make use of different kinds of data?
-	The authors compared two-crop and multi-crop augmentation in section 4, and observed that multi-crop augmentation yielded better performance. One important missing factor is the (possible) computation overhead of multi-crop strategies. My estimation is that it would increase the computation complexity (i.e., slowing the speed) of training. Therefore, one could argue that if we could train the two-crop baseline for a longer period of time it would yield better performance as well. To make the comparison fair, the computation overhead must be discussed.
It can also be seen from Figure 7, for the KNN-MoCo, that the extra positive samples are fed into the network \emph{that takes the back-propagated gradients}. It will drastically increase training complexity as the network not only performs forward passing, but also the backward passing as well.
-	Section 4.2 experiments with AutoAugment as a stronger augmentation strategy. One possible trap is that AutoAugment’s policy is obtained by supervise training on ImageNet. Information leaking is likely.

Questions
-	In L114 the authors concluded that for linear classification the pretraining dataset should match the target dataset in terms of being object or-scene centric. If this is true, is it a setback for SSL algorithms that strive to learn more generic representations? Then it goes back again to whether by combining two datasets SSL model can learn better representations.
-	In L157 the authors discussed that for transfer learning potentially only low- and mid-level visual features are useful. My intuition is that low- and mid-level features are rather easy to learn. Then how does it explain the model’s transferability increasing when we scale up pre-training datasets? Or the recent success of CLIPs? Is it possible that \emph{only} MoCo learns low- and mid-level features?

Minor things that don’t play any role in my ratings.
-	“i.e.” -> “i.e.,”, “e.g.” -> “e.g.,”
-	In Eq.1, it’s better to write L_{contrastive}(x) = instead of L_{contrastive}. Also, should the equation be normalized by the number of positives? 
-	L241 setup paragraph is overly complicated for an easy-to-explain procedure. L245/246, the use of x+ and x is very confusing.
-	It’s better to explain that “nearest neighbor mining” in the intro is to mine nearest neighbor in a moving embedding space in the same dataset.

Overall, I like the objective of the paper a lot and I think the paper is trying to answer some important questions in SSL. But I have some reservation to confidently recommend acceptance due to the concerns as written in the “weakness” section, because this is an analysis paper and analysis needs to be rigorous. I’ll be more than happy to increase the score if those concerns are properly addressed in the feedback.

limitations_and_societal_impact: The authors didn't discuss the limitations of the study. I find no potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: - Current contrastive methods mainly focus on curated iconic datasets like ImageNet.
- The paper studies the effect of dataset biases (like centered object, uniform data distribution) on performance of existing self-supervised learning methods.
- The paper shows that MoCo works well across a wide range of datasets. The paper analyses these datasets by separating them into different categories.
- In addition to the analysis, the paper proposes improvement by modifications to multi-scale cropping, stronger augmentations and a NN based methods. In particular, the authors observe that random cropping for instance discrimination tasks works well across scenarios. 
- The paper shows that multi-crop transform leads to significant gains and gives rise to spatially structured representations which are useful for tasks like semantic segment retrieval and video instance segmentation.

main_review: Strengths:
- The paper investigates the effect of dataset biases on self-supervised learning approaches. Further, the focus on uncurated datasets is an important, and use of these datasets for self-supervised learning has only started recently.
- The authors quantitatively verified that small IoU in scene centric dataset rarely occurs. As pointed out by the authors, this would effectively ensure that for decently sized objects, the two crops would have a significant overlap between the positives.
- The observation that transfer performance doesn't change much wrt changing Max IoU is also quite interesting.
- MC and CC forces the network to focus less on the discriminative parts but rather the whole object. This is also qualitatively shown using gradcam masks. 
- The paper proposes a simple neat trick to mine for more examples by finding the nearest neighbors for anchors using features before the projection layer.
- The paper seems good improvements using a combination of the three proposed modifications to MoCo. 
- The paper is an empirical one that shows some interesting analysis before delving into some easy fixes to improve performance.

Concerns:
- Would you have a significant memory penalty since you are storing pre projection features for the entire queue and these are typically of a larger dimension (2048D?) than the features after the final projection layer (typically 128D)
- The novelty of the proposed approaches is a bit lacking since it builds on top of methods that have been proposed previously (Multi-crop, Autoaugment). But, the combination of these approaches and the simple fixes to improve performance will be beneficial to the community. Further, the analysis on dataset biases and self-supervised learning will be beneficial to the community. 
- The paper should mention the motivation behind choosing a 118K sized subset for ImageNet and OpenImages dataset. It looks like this is motivated from the size of the COCO dataset but it should be clarified.
- The authors note that the improvements start to reduce when the size of the dataset increases. It will be beneficial to show a $\Delta$ from a baseline trained with the same amount of data.
- L136 : I am curious if you have some experiments to show that this is indeed what's happening. While I understand that camera parameters would remain the same, wouldn't that be an incorrect signal for the self-supervised learning task, ie a shortcut that we don't want the network to rely on ? Further, for color/shape, it makes sense if the two crops are a part of the same object but that cannot be ensured for datasets like OpenImages.
- L145 : How do you explain the better performance of the long tailed model in Tables 2/3 ?
- L246 : should be $g(x^+)$ ? Also, the equation is a bit confusing. It might help to have a call to the projection layer explicitly here.

limitations_and_societal_impact: - The authors have adequately addressed the potential negative societal impact of their work.
- The supplementary material includes a discussion on the drawbacks of the proposed approach.
review_title: null

Meta-review: This paper had thorough reviews, rebuttals, and discussion. The reviewers all agreed that the work presents a very interesting analysis and set of conclusions, along with some simple methods to improve results of MoCo-learned representations. While individual contributions are simple and not necessarily novel from a technical perspective, as agreed by the reviewers the set of narratives, experiments, interesting findings, and methods globally provide an interesting novel perspective to the community. Analysis of invariances and especially dataset types (e.g. object/scene-specific) is indeed interesting and one that may exist as common knowledge but has not been thoroughly and rigorously analyzed. In other words, while the paper is largely an empirical investigation it is well-executed in making claims/hypothesis and resulting analysis. 

One of the main concerns expressed by multiple reviewers is the lack of a strong connection and holistic perspective offered across the two sections of the paper. In the rebuttal, the authors provided several arguments and expanded this narrative and connections, which the reviewers were satisfied with. The authors should comprehensively incorporate this (and other great suggestions made by the reviewers) into the final version. While not mandatory, one of the remaining weaknesses that have not been addressed is the applicability of the findings to other self-supervised learning methods. We encourage the authors to add this if at all possible, as it would significantly increase the impact of the paper. 
Decision: Accept (Poster)