Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an object-aware contrastive learning framework which can localize the objects in a self-supervised manner (ContraCAM) and then uses these localized objects to learn de-biased representations. The authors motivate the proposed approach through spurious scene correlations between different objects or between objects and the background. The proposed ContraCAM approach is an intuitive extension of CAM to contrastively trained models. ContraCAM can detect multiple discriminative objects in an image using an iterative method. 
main_review: The paper proposes an interesting and novel approach which extends Class Activation Maps to the contrastive learning framework. The proposed approach, called ContraCAM, can be used to obtain activation maps for discriminative objects in an image. The authors also propose an iterative extension of ContraCAM which enables localization of multiple objects. The paper further proposes some object-aware augmentations for reducing the contextual biases in contrastive learning methods. The authors show that the proposed approach achieves good performance improvements on an existing contrastive learning method. I believe that the novelty of the proposed approach and the strong results and detailed ablation studies presented by the authors make this a strong paper. 

Edit: After reading the other reviews and the authors responses, I will maintain my rating and recommend this paper for acceptance. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new learning scheme for representation learning. Using class activation maps, augmentations are obtained for contrastive learning that are aware of the objects in the image. Results include results on multiple classification, segmentation, and detection datasets. 

Note on 10 sept 2021: rebuttal has been read and discussed with colleague reviewers and AC
main_review: ## Strengths:

* Uses object-aware signals for representation learning.

* Figure 3 clearly explains the different steps in the generation of training data. 

* Table 1, 2, and 3 show results on multiple datasets.

* Table 4 studies a case where crops obtained from the proposed method might even work better than crops based on the ground truth boxes. This conclusion could have a strong impact on future object-aware contrastive learning. 

## Weaknesses:


* How much does the proposed method rely on a good initialized model? The class activation maps in equation (2) will only point to an object of interest whenever the model achieves “good enough” discriminability. 
In this light, line 181 confuses me. Do I understand correctly a model is fully trained (on a labelled dataset), then provides CAM masks from which a from-scratch model is learned? If so, the results should be compared to a supervised model that uses the same labelled data.

* Table 1 & table 3: given that this method requires labels (line 179-180), it would be fair to compare with a supervised model.

* Table 4: Comparison with a supervised model would be useful in this table. If I understand correctly from line 178, the ContraCAM maps were obtained from a model that trained 800 epochs on the COCO dataset.

## Comments:

* Figure 1B: the message of this diagram is unclear. The blue line might as well be overfitting and possible to mitigate via regularisation. Is the suggestion that the proposed method adds additional learning signals and so is less amenable to overfitting?


* Line 51: Object net [3] also evaluates this kind of co-occurrences. 

* Line 74: effect of image background was also studied in [4].

* LIne 100: how does the iterative procedure influence results? How good are results 
obtained with just one iteration?

* Line 274: Object-aware self supervised learning was also studied in [1] and [2]




[1] Pirk et al. "Online object representations with contrastive learning." arXiv preprint arXiv:1906.04312 (2019).

[2] Romijnders et al. "Representation learning from videos in-the-wild: An object-centric approach." WACV 2021.

[3] Barbu et al. "Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models." (2019).

[4] Yung et al. "SI-Score: An image dataset for fine-grained analysis of robustness to object location, rotation and size." arXiv (2021)


Edits after rebuttal:
Authors clarified that models are pre-trained with self-supervised learning. (I erroneously interpreted lines 176-181 as supervised pre-training.) This clarification puts the results in a better light and I will increase review score. I *strongly* encourage authors to clarify this paragraph in a subsequent version of the paper.
limitations_and_societal_impact: Section 5 discusses limitations and concerns of the proposed method.

Methodologically, a major limitation is the reliance on labels on the training set. Lines 179-180 outline a protocol that trains 800 epochs on COCO dataset before obtaining the CAM masks. Therefore, the method is not entirely “self-supervised”.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes object-aware contrastive learning to learn the visual representation from unlabeled images by debiasing the spurious scene correlations. The authors claim there are two types of spurious scene correlations, contextual bias and background bias, which harm the generalization of the model. A novel self-supervised object localization method ContraCAM is proposed to find the unbiased pairs of positives and negatives. Lots of experiments are well done and show the advantages of the proposed ContraCAM.
main_review: The main contribution of this paper is the ContraCAM, which is developed upon the CAM algorithm and to find the most discriminative regions. As the name of ContraCAM, the authors introduce contrastive scores into the CAM algorithm to find the salient regions or objects iteratively. The proposed solution is novel and effective. Combining the unsupervised object localization and unsupervised representation learning with the self-supervised manner may be effective in scene representation.
limitations_and_societal_impact: - the authors emphasize the generalization. But the experiments about the validation of the model's generalization are limited. I suggest the authors provide detailed experiments and analysis to demonstrate the advantages of the proposed method on generalization. For example, trained on a dataset while testing on another unseen dataset.
- the authors claim there are two types of scene bias, contextual bias and background bias. but how do those biases affect scene representation? what is the co-occurrence in scene representation?
-if there are multiple types of objects in the same image, how to guarantee the generalization of scene representation?
review_title: null

Meta-review: All reviewers are positive about this paper. The idea of using CAM to localize the most salient object in an image and use that to reduce background and contextual bias in scene-centric (as opposed to object-centered) datasets for selfsupervised learning is interesting and seems to indeed alleviate the issues. I recommend to accept the paper as a poster.
Decision: Accept (Poster)