Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to learn representative features from multi-view images of objects without requiring category labels. The authors propose a Deep Object Part Encodings (DOPE) framework that leverages sparse depths, foreground masks and camera poses to obtain pixel-level corresponds between different views and performs self-supervised learning from these views. DOPE works well on low-shot classification of novel categories.
strengths_and_weaknesses: Strengths:
1) The overall idea is interesting and novel. Compared to previous work [1], this paper explicitly leverages the information of 3D geometry in multi-view images for self-supervised learning.
2) The performance is good. DOPE outperforms other counterparts and performs competitively with the supervised methods.
3) The paper is well written.

Weaknesses:

The key weakness is that DOPE relies on ground-truth foreground masks, which will limit its application. It is necessary to explore using unsupervised foreground masking methods (e.g., [2]) to extract masks and show if DOPE can still work well.



[1] Exploit clues from views: Self-supervised and regularized learning for multiview object recognition. 

[2] Unsupervised Foreground Extraction via Deep Region Competition

questions: Please refer to weakness. My major concern is the foreground mask is not always available. It is necessary to show DOPE can work well with unsupervised foreground masking methods.

Update:
A minor update of review comments:
I think it is fine that the author can also leverage class-agnostic segmentation models or segmentation models pre-trained on other classes instead of unsupervised methods. In other words, any segmentation models that are NOT trained on the target categories will be fine for me. Please feel free to reply this question.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a self-supervised learning task to perform similarity metric learning across multiple view for the local representation of an object with the assistance of sparse depths, foreground masks, and known camera parameters (from COLMAP.) It achieves superior performance even better than some supervised baseline methods for low-shot classification of novel categories.
strengths_and_weaknesses: Strength:

1. The work shows that by performing metric learning for the local representation of an object across multiple view could significantly improve the low-shot classification of novel categories.
2. They conduct the experiments on both synthetic and real datasets and achieve consistent improvement for different scenarios.

Weakness:

1. The evaluation part is insufficient. The compared methods are all before 2021, and there is no 2022 method for comparison, and only two self-supervised baseline method for comparison, VISPE and VISPE++, which are proposed in 2020.
2. The training heavily relies on the accuracy of the camera parameter estimator, COLMAP and foreground segmentation method, PointRend. There is not much discussion when the preprocessing results are noisy.
questions: It is great that the authors show that to enforcing the self-supervised loss for the local invariant representation of an object across different view can benefit low-shot classification. However, I have few questions:

1. How are the views sampled for the training? If the object views are too extreme, the estimation of COLMAP may fail.
2. How do the preprocessing errors from the camera parameters, sparse depth, and segmentation map affect the performance? Although w in appendix, the authors do provide the ablation studies of w/o random background remove and w/o mask prediction.
3. Is the 3D-based self-supervision necessary since the correspondence may have errors? Will other some other 2D-based self-supervision-based method for correspondence learning for an image with different transformations could also be used to assist the proposed method? Can the author have a short discussions about them with the proposed method. For some examples, there are 2D-based self-supervised learning work for co-part segmentation and part correspondence.

Co-part segmentation:
1. Hung, Wei-Chih, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuan Yang, and Jan Kautz. "Scops: Self-supervised co-part segmentation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 869-878. 2019.
2. Ziegler, Adrian, and Yuki M. Asano. "Self-Supervised Learning of Object Parts for Semantic Segmentation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14502-14511. 2022.

Semantic correspondence:
1. Novotny, David, Samuel Albanie, Diane Larlus, and Andrea Vedaldi. "Self-supervised learning of geometrically stable features through probabilistic introspection." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3637-3645. 2018.
limitations: Yes, the authors do address the limitations of the proposed method and potential negative societal impact of their work at the end of the main paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel method for low-shot category generalization via the object part correspondence learning. The object part correspondence is learnt by contrastive learning on multiple views of object instances. The anchors are constructed by positive features at pixel locations of the same 3D point on the object surface and negatives features from different points on the object surface. To obtain the dense object feature, the autors propose a Deep Object Part Encoding (DOPE), which is a simple framework with the Panoptic FPN. During the inference, several keypoints of the shot image are first sampled and extract the pixel-level feature by DOPE. Then, the keypoint features are compared with all pixel feature in gallery image to obtain the similarity score. Finally, the nearest neighbor method is used to determine the category.
strengths_and_weaknesses: Strengths:

The proposed method is well-defined and esay to understand and it outperforms the state-of-the-arts approachs on few shot classification.

The authors conduct comprehensive experiments on several benchmarks including both synthetic data and real-world data. And the consistent improvement can be observed on different few shot learning setting.

Weakness:

In Figure4, the k pixels on the object surface are randomly sampled from each shot image and compare them with all pixel-wise feature of query image, however, from Line #192 to Line #200, the k pixels are sampled from the query image while compare them with all pixel-wise feature of  the shot image. These two description are conflict, and whick one is correct?

Although this paper claims it doesn't require any category-level annotations, but it still need lots of additional supervision signal, i.e., foreground mask, depth map and camera pose during training. I think it is unfair to compare the proposed method with the existing work. Especially, the depth map actually involves lots of geometry information than the pure RGB image.






questions: As mentioned before, the proposed method requires more information, i.e., depth and camera pose. Although these information are free for synthetic data, it still not very easy to capture for real images. I wonder if you have tried to transfer the model trained on synthetic data to the real images directly or with some tricks. I think these results could be interesting and if the generalization ability is not so bad, the contribution of this work would be improved.
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: -- Post rebuttal update --
The authors have addressed all the questions and weaknesses in my original review using new experiments. In particular the main thesis of the paper has become clear to me now, that it is a comparison between geometric and semantic supervision for low shot learning. The authors have agreed to change their title and description to reflect that the method is patch level and not part level. New experiments show that the methods is much better than Vader and the message remains the same even after supervised baselines are given the same data augmentation. Multi object experiments show that the method is not severely limited to single object datasets as I had originally anticipated. I have updated my rating to 7.
-- End of post rebuttal update ---

This paper presents a method for learning grid structured embeddings from images, called DOPE.

DOPE leverages pairs of images which are two views of a single object with known foreground segmentation, depth maps, camera parameters and camera poses. This information is used to derive pixel level correspondence between the two images which create a positive pair for contrastive learning. Other pairs are treated as negatives. Contrastive learning is used to learn the required embeddings which exhibit the following properties.

1. Embeddings can be used to match across instances of the same category.
2. Such properties extend to novel classes that were not used in training.

These properties combined are used to do nearest neighbor classification to demonstrate SoTA low-shot classification.
strengths_and_weaknesses: ### Strengths

- The main thesis of this paper is that local features matching provides better downstream generalization for low-shot recognition. This has been supported by comparing against baselines.

- Results include both real and synthetic data and compare against both supervised and self-supervised prior work.

- Qualitative visualizations show that the embedding is indeed local. For instance the front of the motorcycle in figure 5 has a different embedding than the middle and back of the motorcycle. Such embeddings might be useful for more fine-grained tasks in future work.

- The tables show that DOPE performs well, beating even supervised baselines on the ModelNet40-LS dataset.

### Weaknesses and questions.
- While the method does not use category labels for self-supervised pre-training, it requires foreground segmentation, depth maps, camera parameters and camera pose. This is very rich information which they derive using COLMAP for the CO3D-LS dataset, but that might not be work well for in-the-wild large scale video corpora. This limitation has been discussed in section 7.

- The paper refers to embeddings as capturing object parts. However, nothing in the formulation is part specific, rather it goes down to the pixel or patch level. The visualization show a continuous variation from one point on the object to another as opposed to a part level segmentation emerging from the algorithm. The presentation would be much clearer if the paper referred to these as patch embeddings from the beginning. Both the paper title and the method name convey that it is "part" based which is confusing.

- All experiments and the formulation assume that there is only one object in the scene. This wasn't discussed explicitly even in section 7. Specifically, what can we expect from DOPE when trained in cluttered natural environments?

- Missing comparison against VADeR (SimCLR style contrastive learning but at the pixel level using two views generated by data augmentation) [1], FLow-E (Similar to VADeR but using BYOL style loss instead of SimCLR) [2], DenseCL [3] or PixPro [4], some recent local embedding learning methods. In fact the architecture borrows from VADeR. FLow-E code is not available. But PixPro has public code [here](https://github.com/zdaxie/PixPro), DenseCL [here](https://github.com/WXinlong/DenseCL). Basically what does a SoTA self-supervised local-embedding learning method do on these datasets? On a related note, how does DOPE perform when the two views come from data augmentation?

[1] Pedro O O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsupervised learning of dense visual representations. NeurIPS 2020.

[2] Xiong, Yuwen, et al. "Self-supervised representation learning from flow equivariance." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[3] Wang, Xinlong, et al. "Dense contrastive learning for self-supervised visual pre-training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[4] Xie, Zhenda, et al. "Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
questions: ### More questions
- Why use the predicted foreground mask, \hat{M}, instead of the ground truth mask, M, in line 182. Same in line 192?

- Is PointRend in line 236 unsupervised / self-supervised?

### Minor comments
- Please discuss the observation that local embeddings work better than global ones again in the experiments. Which two rows should the reader compare to know the performance gap? Is this an apples-to-apples comparison?

- In section 2.1, please state which bucket DOPE falls into. "Metric-learning".

- Do supervised baselines also get the random background masking data augmentation?

### Typos
- Line 2: datasets for training feature representations
- Line 151: For illustration 'p'lease refer to Figure 2.
- Line 245: trained on the support data
limitations: Limitations around data simplicity need to be discussed in section 7. Can DOPE be pre-trained using say the epic kitchens dataset?

Societal impact has been adequately discussed focusing on the environmental effects of training these models.
review_title: null

Meta-review: metareview: **Summary**: This paper aims to learn representative features of objects (without category labels) from multi-view images. A Deep Object Part Encodings (DOPE) framework is proposed, which leverages sparse depths, foreground masks, and camera poses (from COLMAP) to obtain correspondence across different views and performs self-supervised learning from these views. DOPE works well on the low-shot classification of novel categories.

**Strengths**: The overall idea is interesting, novel, and effective. It explicitly leverages 3D geometry in multi-view images for self-supervised learning. The performance gain is notable. The experimental design is solid. The paper is generally well written.

**Weaknesses**: DOPE relies on ground-truth foreground masks and the estimated camera poses by COLMAP. The compared baselines are not sufficient (mostly before 2021). Some technical parts and claims/definitions are not clear.

**Recommendation**: The paper receives positive ratings and reviews in general. After rebuttal, most of the reviewers’ concerns are addressed (e.g., additional experiments) and the paper clearly has strengths. The AC thus suggests acceptance. The AC strongly suggests that the authors incorporate their rebuttal (e.g., responses to Reviewer 3MZa; change the title) into their camera-ready version.
confidence: Certain
Decision: Accept