Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a self-supervised method for Category-level 6D Object Pose Estimation. Specifically, the paper proposes an architecture and a number of geometry-based consistency losses allowing simultaneous estimation of the shape of specific instance and its pose. The experimental sections, demonstrates improvements over SOTA.
strength_and_weaknesses: The self-supervised nature of the method is attractive and makes it more applicable due to lower costs of dataset acquisition. The results improve the SOTA. Interestingly, the paper achieves better results than some supervised methods. Ablation study validates the architecture and loss choices. The methods is validated on two domains (images and videos) and two tasks (6d pose estimation itself and key points transfer)
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, probably reproducible, and to my knowledge is novel.
summary_of_the_review: I give positive rating because in my opinion the paper is well written and novel, and will be interesting to the community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper describes to learn a model for category-level object pose estimation. The proposed approach relies on a dataset of RGB-D images of a certain category of objects, e.g. mugs or bottles, with a template mesh of an object belonging to the category. The method does not require any pose annotation to train, however it leverages recent DINO features during training. The model can still be called self-supervised because DINO learning is self-supervised itself. The approach is evaluated on the Wild6D dataset and is shown to be on par with the current state-of-the-art.
strength_and_weaknesses: I see at least three strong points in the submitted work. First of all, the idea to formulate 2D-3D pose regression through cross-attention between the image pixel features and the mesh vertex vertices seems new, original and interesting. It is one step forward compared to the direct regression of 3D coordinates from 2D images, which became a traditional tool in dense correspondence and pose estimation models wince the Vitruvian Manifold work of 2012. As the ablation study shows, without this element, the model starts to perform significantly worse.

The second strong point is that the evaluation is done on a big real dataset of objects (1722 videos of 5 object categories). The baselines are chosen as rather recent methods, and the metrics are standard and widely used.

The third strong point is the evaluation on a keypoint correspondence task, which shows the power of the learned model beyond the primary task.

However, while the current dataset is diverse in terms of objects, it is very constrained in terms of object categories (only 5 of them, and all are rather simple). It is unclear whether a method that performs well on these categories will generalize to arbitrarily complex categories, e.g. a vehicle, etc., and there is a very popular and recent dataset (CO3D) that can be used as well.

Next, it is not well described in the paper, in my opinion, that the method needs depth data to train. It would be very interesting to see what metrics can we have if we don't use depth for training. It is often the case that the dataset of RGB images is much easier to collect. As an intermediate baseline, maybe one may use some self-supervised depth estimation models here as well. I see no problem in relying on some machine learning models for annotation if they are self-supervised as well. 

While I see that the reconstruction loss is important, probably I'd suggest explaining clearly why is it needed if we have the instance consistency loss. The problem is, the reconstruction loss enforces the colour consistency constraint between the projected vertices and the pixels. The instance consistency loss basically enforces consistency between the projected vertices and the pixels as well. It is stronger, because colours may be the same or similar, but coordinates may be very different. It is also not very clear why removing the depth loss makes things worse, because in my opinion (correct me if I'm wrong) but the depth supervision is already inside the instance consistency loss.

The experiment with keypoint transfer is interesting. But I would like to see the result of keypoint transfer by DINO features as well, to be able to compare and understand if there are any gains compared to this model.

Moreover, there is one question. Why are tthe DINO features used in the cycle consistency, but not the proposed 2D features?

In the Fig. 2, the cycle losses should be shown as well.




clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. There are some questions regarding the terminology used. In particular, (Umeyama 1991) algorithm is for the 3D-3D alignment problem, and it is incorrect to say that it is applied for the 2D-3D alignment problem.

Also, the authors call the competitive papers 'supervised or semi-supervised', while the authors of those papers call their approaches 'self-supervised'. It relates, for example, to the work (He et al., 2022), (Peng et al., 2022), RePoNet. Would like to understand clearly why does this happen. In my opinion, those works also fall into 'self-supervised' category.



summary_of_the_review: Summarising, the paper contains good and valuable ideas. The evaluation shows, that the method is the best on Wild6D in terms of IoU .25 metric as well as pose metrics. However, the situation is mixed on the REAL275 dataset, and here there are question regarding the adequate baselines, because the paper refuses to accept that the competitive methods also fall into the 'self-supervised' category, while the competitive methods are also called 'self-supervised'. This is a difficult situation. Also, the paper is evaluated on just one in-the-wild dataset with 5 simple object categories. I would suggest evaluation on a ore diverse set of object categories.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn category-level 6D object pose in an unsupervised manner. The key contribution lies in a framework to learn 2D-3D correspondences between image and a category-level canonical shape as a mesh. The correspondences are learned by regressing surface embedding on both image pixels and mesh vertices, which are then matched to obtain correspondences. The unsupervised losses mainly consist of differentiable rendering loss and various cycle-consistency losses.
strength_and_weaknesses: Strength
+ Overall the paper is clearly written and easy to follow.
+ The problem of 6D object pose estimation in the wild is of importance in practice, such as in robotic applications. 
+ The major strength over prior arts lies in its unsupervised nature, which does not require any human annotation of object pose. 
+ The experiments demonstrate state-of-the-art performance.

Weakness:
- Some of the technical deteils are not clear.  For example, it is not clear whether the mean shape is optimized or fixed - Sec 3.1 states that the mean shape is learnable but according to Sec. A.2 the mean shape is selected from synthetic data in ShapeNet.  
- If the method learns both the category-level mean shape and per-instance shape deformation, the decompotion of mean shape and deformation is inherently ambiguious. Any change in the mean shape can be compansated by its inverse change in the deformation. It seems there is not any regularization to resolve such ambiguity. 
- The paper has been vague about whether the depth is required in the input. My understanding is yes, because the method estimates pose by Umeyama algorithm. It is then a limitation if RGBD is required as input rather than RGB image only. The paper also acknowledges that removing depth loss leads to a drop in performance. Without depth data, how is the absolute scale of translation determined, given the ambiguity between objece size and distance caused by perspective projection?  
- There is no ablation study on the impact of deformation compared to just using a canonical shape.
-  Many of the objects shown in the paper are symetric, hence it is again ambiguious to define the pose. Does this cause instabability in training and how is it handled?  
 - The backprojection from 2D to 3D in the right equation of Eq.(4) needs depth as well. It is better to point out this for clarity.  

clarity,_quality,_novelty_and_reproducibility: The paper is overall clearly written but some important details need to be more crisp, such as requirement of depth data in input and if the mean shape is learned from scratch or taken from synthetic data.

Overall the paper has some degree of the novelty due to its unsupervised learning for 6D object pose. But majority of the component such as differentiable rendering, cycle-consistency loss are not new. 
summary_of_the_review: The paper is solving a practically important problem, i.e. unsupervised learning of 6D object pose. The overall framework makes senses and works reasoablely in the experiments. That said, some important details needs further clarification as discussed above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a self-supervised framework for category-level 3D object pose estimation. A novel category surface embedding module is proposed which can help establishing dense correspondences within a single instance, between two different instances of the same category, and instances in different time frames. Experiment shows that the proposed algorithm achieves impressive performance on 3D pose estimation and keypoint transfer.
strength_and_weaknesses: Strength:
1. The proposed Categorical Surface Embedding module and the corresponding cycle-consistency loss function is interesting and novel. 
2. The experiment result on pose estimation in the wild and dense key-point transfer on CUB dataset is impressive and the ablation study clearly shows the effectiveness of suface embedding and cycle loss.

Weakness:
1. There are some hyperparameters which weight different items in the overall loss function, such as \beta_texture, \beta_mask, \beta_2D-3D, etc. From Table 7 in Sec. A.3, it seems that such hyperparameters varies across Wild6D and CUB in training, does the algorithm's performance depends significantly on these hyper-parameters?
2. In Table 2, the proposed algorithm get good performance on the 'Self-supervised' setting, but significantly lower performance than algorithms in 'Synth supervised' setting, since the labeled synthetic images can be obtained with limited effort, what is the benefit of self-supervised training on Wild6D without pose label? and is it possible to test the setting of W*+R* data settting and compare to the method of UDA-COPE (Lee et al., 2022) with C+R* data setting?
3. In Figure 2, it is not clear whether and how the deformation \delta_V is regularized, there should be some sentences in sec.3.1 referring to Appendix A.2 and the regularization loss should be reflected in Figure 2.
clarity,_quality,_novelty_and_reproducibility: 1. In general, the paper is well written and easy to follow.
2. The formulation of the algorithm is complete and the experiment is solid and convincing
3. The idea of per-vertex per pixel category surface embedding is interesting
4. The reproducibility would be good since in section 5 the author committed to open their source code once accepted 
summary_of_the_review: In summary, I recommend that the paper could be accepted if all the questions in the weakness part are addressed
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Current 6dof mesh pose predictors are either trained with supervised learning or 3d simulators. There is little work in self supervised learning in this particular domain. The authors present an architecture to take RGB-D images and a template mesh of the considered object categories, which is then used to learn a surface embeddings to map the 3D surface to corresponding pixel values. A geometric cycle consistency loss maps the 2D to 3D. The novelty is that this approach does not require human labeling and seems to be the first result on a large enough 6dof dataset (Wild6D). 

This paper is tackling an open research problem and presents a sound solution. The writing is reasonable and clear enough. The biggest weakness is that it dosen't yet compare to all the datasets, especially things like CO3D or objectron. However, it is not fair to judge the paper because those datasets do not provide RGBD information which is needed for this approach. I do not think RGBD is a big limitation and it is interesting and realistic enough setting to consider and work on. 

I also think that most of the issues brought up during the discussion phase were resolved by the authors. 
justification_for_why_not_higher_score: The paper would be stronger if it had relaxes the RGBD assumption so that it can be compared to other datasets like objectron. This is not a strong criticism for rejecting the paper but relaxing this assumption would make it applicable to a lot more broader audience. 
justification_for_why_not_lower_score: Self supervised 6dof prediction is an important open problem in computer vision and robotics. This is the first paper to demonstrate good results on a real world 6dof dataset, with a reasonable baselines and formulation.
Decision: Accept: poster