Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes a neural mesh renderer that operates on a feature level for 3D object pose estimation. The optimization surface of the loss between the render (for a set of pose parameters) with the actual object image is improved using comparison in the feature space compared to RGB pixel space. The 3D CAD model of the object is converted to a mesh, which is converted to feature space - one feature per 3D mesh vertex. Contrastive learning of foreground vs background mesh features further improves the optimization landscape, even in the presence of occlusion. Experiments performed on the occluded Pascal3D+ dataset and the ObjectNet3D datasets illustrate the utility of conversion to feature space, and outperform some baselines. The paper is reasonably well written, but could do with a spell-check. Contrastive loss and learning in feature space is not novel, but their application to a mesh model for render and compare-type object pose estimation seems to be new. It could do with comparisons with some other state of the art render and compare baselines like [Chen et. al. 20] Category Level Object Pose Estimation via Neural Analysis-by-Synthesis.
review_title: Reasonable paper: Marginally above acceptance threshold.

final_score: rating: 7: Good paper, accept
review_content: This paper tackles the task of pose prediction and takes a render-and-compare approach. However, instead of rendering pixel colors, the key insight is to render features -- each mesh vertex is associated with 3D (learned) features which are encouraged to match computed 2D image features. This 'neural mesh' representation allows pose inference via SGD as one can optimize for pose s.t. the rendered features best match the image features, and is also robust to foreground occlusion. The paper demonstrates results on ObjectNet3d and PASCAL3D+ where the proposed approach is shown to be more robust to occlusion and also better at precise pose estimation.



**Strengths**
I feel the overall approach proposed here is novel, interesting and elegant. While prior render-and-compare based pose estimation approaches have been proposed, they typically rely on semantic keypoints. This work essentially 'densifies' the notion of keypoints to any mesh vertex and instead of a one-hot identity vector, associates a semantic feature with it. The visualizations and the error landscapes shown do convince the reader that this allows accurate inference and gives robustness to occlusion for free.

The empirical results and ablations (assuming concerns below are addressed) evaluate the method across two datasets and occlusion settings, and show clear improvements over existing SOTA. The approach is also well ablated e.g. the necessity of the contrastive term is shown, and the benefits of handling outliers as means for robustness to occlusion are also analyzed.

It is also encouraging to see that the method works even when using a coarse geometry e.g. a cuboid as the reference mesh and this in particular shows the benefits of having learned features.


**Concerns**
I have several concerns/questions regarding the empirical setup and results, and feel that these have not been detailed sufficiently in the text. On this aspect, I feel the paper lacks clarity and would hope to see these answered before recommending acceptance.

- Each category in PASCAL3D (or Objectnet3D) has different 'subtypes'. The text in 'training setup' states 'we .. learn a NMM for each subtype separately'. This raises several questions:
a) Are separate Neural Mesh Models for subtypes learned even in the 'single-cuboid' case i.e. is only the geometry the same and features learned separately for each subtype?
b) More importantly, how is the subtype known at inference? The optimization objective in Eqn 12 assumes a known neural mesh model, but if there are many possible subtypes, it is not explained how this is inferred given a new image.

- Related to the above point, the optimization in Eq 12 also assumes a known 'FB/BG' mask. It is also not described how this is known for a new image. Is an additional predictor used?

- The text does not describe how the optimization for inference is performed? Does simple gradient descent with a fixed starting point work? Or are many different/random starting points used? Also, how efficient is inference?

- The failure cases are not analyzed/highlighted. For example, all the qualitative examples are shown for cases where the template matches the object rather well. What happens if this is not true e.g. for chairs?

- One result I am very puzzled by is the 'NeMo w/o contrastive' case. In this case, why don't all the learned features  (both 2D and 3D) tend to a constant? While this is shown to be worse than the full method, I am surprised this works at all! For example, Eqn 12 would be optimal if all f_i and N_y are zeros and all poses would be equally optimal! I would be curious on why this does not happen - is there some other objective term / inductive bias that prevents this.

- As a relatively minor point, while the empirical results show gain over 'StarMap' in cases with severe occlusion, this method is similar in case of the entire dataset (L0) without additional cropped occluders. If the information available to all methods at inference is the same, then this is not a major concern.

---------

Overall, while I feel the paper presents an interesting idea, there are just too many unknowns w.r.t. the inference setup (and also some surprising results) for me to recommend acceptance. If these can be clarified in the response and the comparisons are indeed fair to the baselines, I would be happy to update my rating.

--------------------------------------
Updates after author response:
---------------------------------------
I think the revisions and the responses did address all the concerns I had, in particular towards assuring that the method and baselines leverage the same information for inference. Additionally, the experiments where one can 'search' for the optimal subtype instead of assuming known subtype at inference also showed encouraging results.

Overall, I think the paper writing and presentation is much improved, and I would argue for acceptance as the paper presents a simple and intuitive idea which is shown to work (rather surprisingly!) well.
review_title: Interesting direction but concerns regarding empirical setup

final_score: rating: 7: Good paper, accept
review_content: The authors propose a novel 3D neural mesh model of objects that is generative. They demonstrate that standard deep learning approaches to 3D pose estimation are highly sensitive to partial occlusion. Since their method works in a render and compare manner, it enables the method to be more robust to artifacts in general and partial occlusion in particular. They also achieve a highly competitive 3D pose estimation performance on popular dataset. They go on to show that even very crude prototypical approximation of the object geometry using a cuboid. 

The neural mesh model representation is novel and clearly leads to superior robustness. Also the contrastive loss is clever and effective. Finally even when the 3D models are approximated by just cuboids, the results are very competitive. Their experiments are exhaustive and convincing. I would vote for acceptance of the paper.

However, the writing has a lot of scope for improved. E.g. it is difficult to understand how exactly the 3D mesh is converted to its neural mesh representation, etc. For understanding how much is a 3D model approximated when using its corresponding low resolution cubiod, a figure which shows a few classes, their 3D models and also their low resolution decimated 3D model will be helpful. 
review_title: Novel method to estimate object pose which works well also with occlusion

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper presents a novel approach for 3d pose estimation by combining render-and-compare (analysis-by-synthesis) and contrastive feature learning. The key idea is to render and compare learned latent features instead of synthesized RGB colors to optimize 6D pose parameters. The proposed method learns latent feature vectors on a template mesh as well as target images via backbone neural networks such that matched regions have similar features while latent features are as distinctive as possible. The paper evaluates the novel formulation on  PASCAL3D+, the occluded PASCAL3D+, and ObjectNet3D dataset, demonstrating the render-and compare optimization with the proposed approach is more robust to appearance change and partial occlusions.

Although the paper presents very interesting idea, the paper leaves several critical aspects unclear (please see the comments below). Thus, I believe the paper is not ready for publication.

Pros:

\- The paper addresses the sensitivity of render-and-compare approaches by leveraging the learned latent features that are agnostic to appearance change.

\- Partial occlusion is handled by jointly predicted binary mask, allowing the render-and-compare optimization focuses on visible regions.

\- The evaluation shows that contrastive feature learning for distinctiveness plays a critical role to robust optimization. 


Cons:

\- As the authors mentioned in the related work, the traditional R&C approaches are sensitive to initialization. However, this paper misses any form of ablation study on the sensitivity on initialization, which is critical for practical use. 

\- One major limitation of the proposed method is to rely on a fixed template model. As the size and shape of objects in the same category often vary, it is not clear why not to incorporate simple shape parametrization in the template model (via PCA or axis aligned scale factors) as in traditional R&C methods.

\- The experiments in Sec. 4.2. do not fully support the advantage of NeMo over the baselines especially in case of partial occlusions. What if we provide explicit segmentation mask for each baseline as in NeMo? If the performance becomes comparable, to make deep networks more robust, we would need only explicit segmentation, not necessarily render-and-Compare optimization with contrastive feature learning. 


Questions/Remarks:

\- How sensitive is the proposed approach to the choice of differentiable renderer?

\- As the entire pipeline can be trained end-to-end, the proposed method could incorporate pose alignment process as part of objective functions during feature learning process. Why is such an objective function not incorporated? Also I’m wondering if such an optimization-driven feature learning would eliminate the need of contrastive feature learning.

\- The paper explains how to compute z_i only in the caption of Fig. 2. I would recommend explaining it in the main text.

\- As the latent features on the mesh is fixed throughout the optimization once it’s trained, calling it a generative model would not be appropriate.

\- In Sec. 3.2., it’s not clear how binary mixture coefficients \alpha_k is used.

================================================================
Post Rebuttal:

Thanks for the revision and detailed rebuttal. I think the clarity is largely improved now. As the paper has sufficient technical novelty with fairly clear description, I'm slightly inclined towards acceptance. However, I would recommend further clarifying the following point.

>We want to point out that NeMo has exactly the same amount of supervision as all other methods
- The fact that NeMo jointly reason occlusions in an unsupervised manner was not clearly explained. As the authors described, this property makes difference in terms of robustness to partial occlusions. Clarification on this in the text is very helpful. 
review_title: Neat combination of differentiable rendering and contrastive feature learning, but experiments can be improved.

Meta-review: This paper received 4 reviews with mixed initial ratings: 4,5,6,7. The main concerns of R2 and R4, who gave unfavorable scores, included lack of clarity around design choices and inconclusiveness of some of the experiments. In response to that, the authors submitted a new revision with a summary of changes and provided detailed responses to each of the reviews, which seemed to have addressed these concerns. R2 and R4 upgraded their scores and recommended acceptance.
As a result, the final recommendation is to accept for presentation at ICLR as a poster.
Decision: Accept (Poster)