Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method for multi-object 6DoF pose estimation with a "probabilistic programming" model. Before inference, the model learns priors on 3D occupancy/shape for objects. Then, given the number of objects in the scene and the classes for objects in a test scene (observed in a depth map), inference proceeds by MCMC sampling scene graphs and poses and occupancies, to minimize a depth reconstruction error. The results look good on YCB and on a new synthetic dataset based on YCB objects, against two recent 6DoF pose estimators, showing good results. 
main_review: This paper was very hard for me to understand, and there still may be parts I have misunderstood. I will set up my review as a list of confusion points, which might all be easily answered by the authors.

- What is the purpose of assigning the names top/bottom/left/right/front/back to the planes of the box? It doesn't seem like this information is ever used. 

- How exactly do you model contact relationships, and learn priors about them? I found many variables that relate to contacts in the "Scene graphs" paragraph (89-109) but they never come up again, either in the generative model or the inference algorithm. In general there is a lot of notation for the reader to keep track of, and it seemed like much of it was declared and then forgotten, so perhaps it can just be deleted. (But I do very much want to learn about those contact relationships.) 
 
- The voxel grids have "fixed dimensions h, w, l ∈ N", but they also have a "grid resolution s ∈ R". What is the difference between a dimension and a resolution?

- The text says that the scene graph prior (equation 1) "favors scenes that can be parsimoniously described via either fewer objects, or fewer objects with independently-sampled 6DoF poses". The idea of "fewer objects" occurs in both sides of the "either", so it seems that part should always be true, but actually the number of objects is held constant by an assumption stated earlier: "an a-priori known number of objects N0 (used in the experiments in Section 5)". So the parsimonious/fewer part seems inactive here, and we are just left with "independently-sampled 6DoF poses". It seems like little-to-no information or inductive bias has actually been introduced here. Am I reading that right?

- Is the pre-trained object occupancy prior conditioned on class? 

- The paper says the model performs well "despite being trained on only a few synthetic depth images". How many is a few? (Is it the "5 depth images of each object type" mentioned later? Why does the abstract say "5 or fewer"?) Also, why was it trained on only a few? Since they are synthetic, I expect them to be plentiful. 

- Do the baselines also assume known class and number of objects? They seem to involve detection and segmentation as well as pose estimation, so maybe they are a harder problem, and it's not an apples-to-apples comparison. 

- I do not quite follow Table 1, and how to compare it with prior work. The Robust6D and DenseFusion papers report ADD-S accuracy across far more classes (~21 instead of 5), their values are in the range 90-100, and they also report the mean across all classes. Table 1 reports "ADD-S error" at three quartiles, and the values are under 2, and it looks like the mean across objects is never calculated. It would be good to do the same kind of evaluation as the previous work. 
limitations_and_societal_impact: No concerns.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents an approach for inferring the 3D pose and shape of multi-object tabletop scenes from a set of multi-view depth frames as input.  The approach is based on a generative model incorporating 3D scene graph generation, as well as per-object category voxelized shape priors.  At test time, the inference algorithm parses the input to a 3D scene graph using MCMC-based updates to the scene graph structure, constrained by physical contact relations between the objects and the object shape priors.

The approach is evaluated on the 6DoF 3D pose estimation task using two datasets: a set of real scenes from the YCB-video dataset, and a set of synthetically generated scenes using reconstructed YCB objects (YCB-Challenging).  The experiments compare the proposed approach against baselines from prior work (specifically, DenseFusion and Robust6D -- both trained in a supervised fashion on larger RGB-D training datasets).  An ablation of the approach without scene graph structure inference is also used as a comparison point.  The experiments show that the approach mostly outperforms the baselines and ablation in terms of accuracy of the predicted 6DoF pose for the objects in the scene.  An additional analysis incorporates one of the baselines (DenseFusion) to initialize the estimated pose used by the proposed method. Improvements to the estimated pose are observed in some challenging scenarios from the YCB-Video test scenes.
main_review: Originality:
On the plus side, the proposed approach is quite comprehensive and formalized in a way that admits an impressive level of generality.  As such, I can see the approach being applicable to domains and datasets beyond what has been presented in the paper.  I believe that the community would benefit from more work in this vein, leveraging probabilistic programming and generative models for 3D scene understanding.  On the other hand, there is a highly relevant line of prior work that is not cited or discussed: "Attend, Infer, Repeat:
Fast Scene Understanding with Generative Models" by Eslami et al. NIPS 2016 have proposed a similar probabilistic inference framework and demonstrated that it can be used to parse 3D scenes (i.e. infer the object that are present, and their 6DoF poses).  A detailed discussion of how the proposed approach relates to this prior work, and potentially comparison against it in the evaluation is in my opinion necessary.

Quality:
The paper is technically sound, as far as I can tell and the claims are mostly supported by the empirical evidence.  That being said, there is no evaluation of the object shape priors either independently, or as part of the proposed approach.  In other words, the impact of Section 4 on the results is not clear in the current paper.

Clarity:
Overall, the paper is very well written and organized coherently.  There were a few minor typos etc. that could be resolved in a proofreading pass.  A couple of examples that I detected: L98 "corresponds to rigid object with" should be "corresponds to a rigid object with"; and L201 "a uninformed" should be "an uninformed".

Significance:
The paper has an impressive scope in terms of the presented formalism and its generality.  However, the empirical results only partially back up the complexity of the proposed approach, and some discussion of relation to prior work is missing (see above points).  Therefore, I am not as positive as I could be with respect to acceptance.
limitations_and_societal_impact: Some limitations are discussed at a high level in the last section.  The speed of inference, and integrating object prior learning with scene graph parsing were identified as avenues for future work.  Potential negative societal impacts were not discussed (the checklist notes that the authors do not believe there are any at this stage of the work).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a probabilistic model for reconstructing a scene of multiple 3D objects. The key advantage of the approach is that it is able to reason about the shapes of objects, the relations between them (via the scene graph) and their contacts. This ability to use physical constraints for scene parsing is a major limitation of deep learning systems. The results show that the proposed method is significantly more accurate than two recent approaches that are based on deep models (i.e. DenseFusion and Robust6D).  The main limitation of the model seems to be the inference speed, which is currently not realtime.

main_review: **TL;DR:** An intriguing, principled, probabilistic approach for object shape and pose estimation that is able to utilize object relations and physical constraints. In the main review below I have labelled each point as positive (+), negative (-) or mixed (+/-).
 
**Originality**: 

(+) As far as I can tell the method is new in the sense that this is the first probabilistic, generative model that takes into account both object shapes, physical constraints such as their spatial relations and contact between individual objects. This is interesting as physical reasoning can help to overcome the limitations of traditional object reconstruction methods that only take into account multi-view geometry and simple shape priors. The results presented in the paper confirm this as the proposed system is significantly more accurate than the two approaches against which it is compared. 

(+) Although this type of analysis-by-synthesis approach based on MCMC has been very popular and well explored in computer vision in the past, the probabilistic model presented in the paper is novel. Furthermore, the meticulously crafted MCMC kernels used for inference are also novel and seem to be effective.  

(+/-) In terms of related work, the authors do a good job at describing which contributions are novel. I think the related work Section 2 on “Probabilistic programs for vision” is particularly clear and gives a concise description of what is novel compared to general probabilistic programming works that have addressed computer vision problems. However, I would probably like to have seen a discussion about more closely related work such as [A,B] that are probabilistic approaches for addressing the same problem described in this paper (i.e. joint scene parsing and object reconstruction). At the moment the related work doesn’t cover such approaches.

[A] Du, Yilun, et al. "Learning to Exploit Stability for 3D Scene Parsing." NeurIPS. 2018.

[B] Bao, Sid Yingze, and Silvio Savarese. "Semantic structure from motion: A novel framework for joint object recognition and 3D reconstruction." Outdoor and Large-Scale Real-World Scene Analysis. Springer, Berlin, Heidelberg, 2012. 376-397.

**Quality**: 

(+) Although I have not checked every detail of the proofs in the submission (particularly the ones in the supplementary), the general approach appears to be technically sound as far as I can tell.  

(+) The claims in the paper are well supported. Particularly the results show that the proposed method is able to learn new objects from a very small number of views i.e. 5 and using this model is able to obtain state of the art pose estimation results on the YCB dataset. 

(-) There is a caveat with these results though, in that the method appears to be initialized with a state-of-the-art deep learning model, which makes the results slightly less impressive.  However this still shows that the proposed approach is able to refine the poses to a level beyond that which can be obtained with current methods. I would suggest that the authors make it clear in the paper where the method has been initialized using an existing approach and where it hasn’t because this isn’t very clear at the moment. Also, what is the performance of the system when no initialization is used?

(-) Overall I would say that the authors could do a better job of describing the weaknesses of the approach. The most obvious weakness that comes to mind is computational efficiency. There should probably be some form of inference time comparison in the paper to show how the proposed approach compares to the existing methods (especially deep learning based ones) which are known to be very fast. Another weakness is the complexity of the approach - is it not possible to simplify the components such as the transition kernels or the scene graph structure and obtain similar results?
    
**Clarity**: 

(-) The paper is very dense and not always that easy to follow.  This is partly due to the nature of the approach - the model contains many quantities and distributions that need to be defined, but it might be useful for the reader if there was some summary or overview that describes all the components of the method in one place.

(+/-) Overall, the paper is well organized and the individual sections are quite verbose and not that easy to follow. I don’t have a great suggestion to fix this, but maybe separating the definitions from the actual description might help. 

(+) As far as I can tell, the paper provides enough information for an expert to reimplement the method (with the help of a probabilistic programming language, like Gen) and reproduce results. The authors have also supplied the source code which helps with reproducibility.

**Significance**: 

(+) In my view, the results presented in the paper are significant. This is one of the best examples of how probabilistic programming can be applied to solve challenging vision tasks.  It is quite surprising how data-efficient and accurate the approach is compared to deep learning systems that need many training samples to achieve similar accuracy for object pose estimation. 
I think the great results obtained in the paper could help to encourage more computer vision researchers to explore probabilistic programming approaches. There are also quite a few areas in which the proposed method could be improved (such as inference time etc.) which might inspire other researchers to try to improve upon the method.

(+) In summary, the method addresses a difficult task of jointly estimating object shape and pose, and also  advances the state of the art in terms of accuracy.

# Post author response and discussion
After considering the other reviews, and the authors' responses, the main concerns raised in my review have been addressed. Specifically, the issues related to how the initialization is done and a more thorough discussion about the weaknesses of the approach have been addressed. In terms of accessibility / reproducibility the authors have committed to open-sourcing their approach. Overall, I think this is a good paper which opens up lots of possibilities for future research at the intersection of probabilistic modelling and computer vision. Therefore, my final rating is "7: Good paper, accept".

limitations_and_societal_impact: The paper could do with a better description of the limitations of the approach (as mentioned above), and there is no mention of the potential negative societal impacts.
review_title: null

Meta-review: This paper presents a method for multi-object 6DoF pose estimation with a "probabilistic programming" model. Before inference, the model learns priors on 3D occupancy/shape for objects. Then, given the number of objects in the scene and the classes for objects in a test scene (observed in a depth map), inference proceeds by MCMC sampling scene graphs and poses and occupancies, to minimize a depth reconstruction error. 
The reviewers raised concerns regarding fairness of comparisons to baselines, known versus unknown number of objects at inference time, violation of priors, (lack of) comparisons to other object-centric generative models. The rebuttal of the authors addressed those concerns and generated extensive discussion. The paper is suggested for publication. 

Decision: Accept (Poster)