Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a probabilistic generative method leveraging learned 2d-to-3d mappings from SurfEMB and depth (3DP3) that generates and optimizes (MCMC) 6d pose hypotheses based on RGB-D images. The method is purely trained on synthetic images and evaluated on real data showing significant improvements to the baseline method by evaluating multiple hypothesis and refining them using MCMC.
strength_and_weaknesses: Strength:

The probabilistic formulation is simple and seems robust.
Proposal generation method based on spherical voting from depth and correspondences is independent of 2d detections priors.
The authors show the usefulness of the probabilistic model to generate possible poses under occlusion.

Weaknesses:

A remaining question is which hypothesis are actually sampled in the MCMC optimization. An ablation for the impact of the different methods Voting, ICP and Random Walk would help to evaluate their impact on the final result.

It seems that at the moment the proposed method leverages the knowledge of how many objects are present in the scene to create the initial proposals, which is a hugh advantage over other methods that do not have this assumption. While it seems like the assumption could be relaxed, it is unclear if the method remains stable in this case.

In 3.3 the authors write "ICP proposals iterate over all objects and use ICP to align each object’s 3D model to the observed 3D point cloud with the point cloud from all the other objects masked out,", however it is unclear how they can mask out all other objects if they only have semantic masks and not instance masks. While using synthetic data learning instance masks would probably not pose a larger problem than predicting SufEmbeddings, it is unclear how these masks are generated at the moment.


Corrections:
In 3.1 the dimensions of Q should probably be HxWxE instead of HxWx3
clarity,_quality,_novelty_and_reproducibility: The joined probabilistic formulation for RGB and depth is novel and interesting.
While the basis is a mix of 3dp3 and SurfEMB, bringing them together is a novel idea and the proposal generation seems also different to existing ideas.
summary_of_the_review: After the rebuttal of the authors the explanation of the method is very clear and the probabilistic formulation shows advantages over existing methods especially also for estimating the uncertainties. There remain a few open points that would be interesting to follow up on, but the proposed idea and results are already showing the benefits of the proposed approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The presented paper aims to introduce a method for the robust sim-to-real transfer to address the domain gap between synthetically generated data and real data. The goal of the method is to support training a model with synthetic data so as to generalize to real data. The proposed method -- called 3DNEL -- uses neural embeddings to predict 2D-3D correspondences from RGB and depth images. 3DNEL is used as part of a multi-stage inverse graphics pipeline that is tested on a 6D pose prediction task.  

strength_and_weaknesses: Strengths: 

- The topic of using synthetic data for effectively training neural networks is interesting and important. 
- The proposed approach of combining RGB and depth images by employing neural embeddings seems novel although somewhat marginal compared to SurfEMB.  
- The presented experiments are mostly sounds and the results seem promising. 

Weaknesses: 

- The technical section of the paper is quite convoluted and difficult to follow. 
- Contrary to the author's claims it does not seem as if the method is easy to apply to other tasks and training setups. 
- The method is only shown to work on pose estimation. 
clarity,_quality,_novelty_and_reproducibility: While the paper is mostly easy to follow, the technical description of the method appears quite convoluted and difficult to understand. Several aspects of the method need to be described more clearly. To provide a few examples: 

- What is meant by latent 3D scene? 
- Section 3.2 refers to embeddings Q^t, while ' the key model an input 3D model coordinate into an embedding'. What is the difference between these embeddings? As is this is not clear. 
- In Section 3.3. what is meant by 'the learned neural embedding share used to define a likelihood that computes the probability of an RGB-D image given a hypothesized latent 3D scene'? Why hypothesized latent 3D scene?
- The motivation Section 3.3 is difficult to follow and should be described more clearly. Why are the semantic segmentation maps observed? How exactly is the object coordinate array defined?
- How exactly is the 'latent point cloud generated'? Why is it latent?
- What is meant by 'additionally have a uniform distribution on the bounded region as an outlier component'? What is meant by 'outlier component'?
- Why is the 'normalization constant' well defined? Why is there a finite number of RGB images? 
- Table 1: Why are the results only reported for 5 runs? Wouldn't it make sense to validate on a larger number?
- Figure 4: What is the arrow on the x-axis supposed to indicate? 
- I would suggest to provide runtime/inference runtime performance measures. This would especially be interesting for the MCMC + ICP experiment discussed in Section 5.3. 

As is, not enough information is provided to allow for reproducing the proposed method. 
summary_of_the_review: Overall, I have the impression that the topic this paper address is important and interesting to the ICLR community. However, it its current form I cannot recommend accepting this work. This is due to the following reasons: 

- The technical description of the method appears quite convoluted and is difficult to follow. The method section needs to be rewritten carefully and more details need to be provided. 
- While the method seems to be sound for the setup defined in the paper, it is not clear how it could be used for other tasks or setups. 
- Compared to SurfEMB the technical contribution appears somewhat marginal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The author proposed an inverse-graphics approach for robust object pose estimation: the model first evaluates pose performance on discretized pose hypothesis with a spherical voting, then finetunes the pose with MCMC. The performance on YCB-dataset is strong and achieves the state-of-the-art.
strength_and_weaknesses: + A 2d/3d correspondence approach to handle sim-to-real pose estimation.
+ the probabilistic formulation in Eqn (2) and (3) makes sense.
+ Spherical voting idea is interesting, related in spirit to hough voting. 
+ Experimentally, the model achieves strong performance compared to SOTA Ransac-based-proposal approaches.
clarity,_quality,_novelty_and_reproducibility: The idea is clear and easy to follow.
summary_of_the_review: In summary, the author proposed an interesting idea of 2D/3D-correspondence-based  approach to solve the object pose estimation problem. The idea is clearly presented and easy to follow. And the experimental results on YCB is strong.

I have a few questions:
* Any insight why the proposed approach can be better than RANSAC+SurfEMB? the evidence model proposed in Eqn (1) could be covered by RANSAC if a correct correspondence is proposed to generate a hypothesis.
* Robustness: if a RANSAC-based algorithm tries enough hypothesis, will it be more robust as well?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method for inference of 3D object pose and locations in RGB-D images and videos. It follows an analysis-by-synthesis paradigm where object pose hypotheses are continuously predicted and evaluated in an MCMC framework. The hypothesis prediction and the evaluation leverages learned 2d-to-3d feature mappings from previous work (SurfEMB) and depth matching from (3DP3). The method is purely trained on synthetic images and evaluated on real data showing some improvements to baselines. During rebuttal time, the authors showed results where the uncertainty provided by their formulation is beneficial for tracking and object 3d pose inference. 

There is a long line of work that does 2D to 3D inference or 2.5D to 3D inference similar to this paper:
https://arxiv.org/pdf/2205.15768.pdf and many papers before use what they call a camera multiplex representation that scores multiple camera hypotheses jointly:
 Cosypose  https://github.com/ylabbe/cosypose searches over 3D poses of multiple objects .
 MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare seem to have very similar objectives as the present work, 
Most recent work: Learning to Imitate Object Interactions from Internet Videos that again infers 3D object and hand poses via iterative refinement. 
First, the paper has not put enough effort to compare against those methods, e.g. CozyPose or MegaPose in an apples to apples manner.  For example, for CosyPose the code is available and the model could be retrained using only simulated data to have an apples to apples comparison with the proposed method in Table 1. 
Another important baseline missing whose code is publicly available and shows SOTA performance across multiple benchmarks is: Coupled Iterative Refinement for 6D Multi-Object Pose Estimation. 
 Second, the contribution of the work appears minor, in the sense that the matching function is taken from  previous works. Third, the improved performance of the present method costs 6X the inference time due to gradient-free optimization. In fact, the right comparison plot should also have the inference time on one axis, and methods should be compared both in terms of accuracy but also in terms of inference time efficiency. Forth, given the slow inference of the proposed method, it is unclear why it is disadvantageous to use real world annotations, given that building simulators is also very time consuming, similar  to annotating real world images. The authors are encouraged to use a SOTA previous work, e.g. CosyPose or the most recent MegaPose, and train it in an apples to apples method, as well as show comparisons and contributions in apples-to-apples inference time. Many of previous works would alos benefit from additional inference time by checking more hypotheses, similar to the presented method.   The work is in a great direction, perception as analysis by synthesis, and we encourage the authors to add the suggested comparisons and submit to a future venue. 
justification_for_why_not_higher_score: 
First, the paper has not put enough effort to compare against those methods, e.g. CozyPose or MegaPose in an apples to apples manner. For example, for CosyPose the code is available and the model could be retrained using only simulated data to have an apples to apples comparison with the proposed method in Table 1. Second, the contribution of the work appears minor, in the sense that the matching function is taken from  previous works. Third, the improved performance of the present method costs 6X the inference time due to gradient-free optimization. In fact, the right comparison plot should also have the inference time on one axis, and methods should be compared both in terms of accuracy but also in terms of inference time efficiency. Forth, given the slow inference of the proposed method, it is unclear why it is disadvantageous to use real world annotations, given that building simulators is also very time consuming, similar  to annotating real world images. The authors are encouraged to use a SOTA previous work, e.g. CosyPose or the most recent MegaPose, and train it in an apples to apples method, as well as show comparisons and contributions in apples-to-apples inference time. Many of previous works would alos benefit from additional inference time by checking more hypotheses, similar to the presented method.   The work is in a great direction, perception as analysis by synthesis, and we encourage the authors to add the suggested comparisons and submit to a future venue.
Decision: Reject