Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Overview:
This paper is certainly very interesting, unlike other papers and makes some very solid contributions. The qualitative results are very impressive. Unfortunately, the paper is very poorly written. If the authors can address the issues below and improve the quality of the writing, I would recommend that this paper be accepted. However, in its current state, I recommend that this paper be rejected.

Major:
The claim that “This is the first unsupervised model that can identify objects in a 3D scene” is not true. There is MONet and Iodine that can identify objects without supervision in 2D projections of 3D scenes. This claim should be revised.

What is r_C in p(z|c, r_C)?

The authors are using non-standard GQN notation and their notion is not consistent. The authors should make their notation consistent or use the standard GQN notation. Section two would not make sense to people that are not already familiar with GQN.

The scene volume map is interesting, the inductive bias preventing two objects being present in the same location is interesting, however one could imagine a case where uncertainty in the model could lead to two objects being represented in the same cell.

The terms in Equation 3 should be explained more explicitly. The way I understand it p(s_n|z, x) is the view-point dependent representations of the objects and this is why you condition on z, x. Placing p(s_n|z, x) in the text where you talk about s_n being view dependant with some explanation would help.

Why do you use a Gaussian distribution for the position? Would it not make more sense to use a uniform distribution? Could this be related to bias in your data? I.e. more objects in the centre of the scene?

What happens if object n is not present in the context image, y_c? In this case what is s_{c,n}^pos. Also, this notation: r_n = f({y_c,n}c)  is a little ambiguous. I assume it means that you are applying the function f to the set of all patches in y_c? Also, what is the object invariant object encoder? A reference to details in the appendix or a footnote would suffice.

It’s great that you are able to exploit the coordinate transform and use it for rendering.

The results are very impressive: being able to swap in and out objects in a scene, showing the 2D renderings of single objects from different view-points and the scene decompositions and predicting where the “missing” object is (Figure 6).

Minor:
The introduction could be strengthened with additional references. There are claims that object-wise factorisation will help with transfer, it would be good to have references to other work that supports this view. Also the claim that humans have 3D representations for objects requires a reference.

Typos (there are too many to list here, these are just a few):
* Abstract: and and rendering
* “and”s should be replaced with commas in the second line of the intro.
* Generally the paper is not written well.
* The GQN, as a conditional → is a conditional
* Target observations (in section 2) does not need a capital.
* This sentence does not make sense: “instead of encoding compressing the whole scene”
* Because of intractable posterior

There are many additional grammatical errors. 


-----------
Edit: Following changes made to the paper, I am now more satisfied. The writing should still be improved further and suggest that the authors fully revise the paper before the camera ready version, if the paper is accepted. I have increased my score to 6.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper presents a framework for 3D representation learning from images of 2D scenes. The proposed architecture, which the authors call ROOTS (Representation of Object-Oriented Three-dimension Scenes), is based on the CGQN (Consistent Generative Query Networks) network. The paper provides 2 modifications. The representation is 1. factorized to differentiate objects and background and 2. hierarchical to first have a view point invariant 3D representation and then a view-point dependent 2D representation. Qualitative and qualitative experiments are performed using the MuJoCo physics simulator [1] (please add citation in the paper).  

[1]Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based
control. In ICIRS, 2012.

+Learning 3D representations from 2D images is an important problem. 
+The proposed methodology learns representations that are more interpretable, with higher compositionally. 

While the paper takes a step towards a potentially impactful work, I cannot recommend it for publication in its current form. 

1. There are claims in the paper that are not supported by the experiments. For example, “As seen in Figure 2, ROOTS has clearer generations than GQN. ” However, Figure 2 does not show this at all. It shows no difference between ROOTS and GQN. 

2. The paper can benefit from further clarity throughout—in general it seems a bit rushed. For example, the caption on Figure 3 reads “For example, GoodNet segments a scene into foreground and background first and decompose foreground into each individual object further…” The text has not discussed what GoodNet is. Its also unclear what is depicted in each of the columns in Figure 3. This should be clearly explained. 

3. I suggest clarifying Figure 1 further and referring to it in section 3. Its currently not referred in the text although it is an overview of the proposed architecture. 


Other comments
-Table 2: Why not have a precision-recall curve (as is standard) and report average precision numbers?
-Table 2: Why not compare to CGQN?

Minor
-There are typos throughout the text. 
review_title: Official Blind Review #2400

final_score: rating: 3: Weak Reject
review_content: TLDR: Interesting idea that seems promising, but lacks the maturity required to pass the ICLR bar: Lacks proper citations, comparisons with the latest works, no ablation study of their contributions.

The paper presents an extension of the Generative Query Network to incorporates 1) 3D grid for the state representations 2) hierarchical representation and 3) unsupervised model for explicit object representation (which is tied to 1.

The unsupervised representation is interesting, but this is a minor contribution on top of GCN and 3D representations have been widely studied in the vision community.
 
Also, except for the 3D representation, I am not sure how much the hierarchical representation helps. This leads to the question of why the authors did not perform ablation studies on each component.

Finally, it seems that the authors did not add proper citations. First, the 3D representation has been studied widely in the vision community and 3D-R2N2, ECCV'16 proposed using an RNN to encode a series of 2D images to learn 3D grid representation which seems quite similar to what the authors are proposing as an encoder and representation. Secondly, there are numerous methods on 3D neural rendering such as DeepVoxels, CVPR19 and all of the baselines in their experiments. The paper seems to completely ignore the works in this field.


Questions:

Please point out the equation number of implementation of Eq.2 in the appendix.

Is the y in Eq.3 the same y defined in the preliminary? Also, consider defining y again. There are almost two pages between the definition and Eq.3.


Minor:
Why learn the camera projection? $f_{3D\rightarrow 2D}$? Isn't this deterministic using a camera matrix?

Is the lighting fixed throughout the training and testing?

Sec.2 first sentence: an query --> a query
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes a model building off of the generative query network model that takes in as input multiple images, builds a model of the 3D scene, and renders it. This can be trained end to end. The insight of the method is that one can factor the underlying representation into different objects. The system is trained on scenes rendered in mujoco.

Summary of positives: 
+The factoring makes sense, and the use of voxels + the physical property to enforce that two objects can't be superimposed in z_pres is a good strategy.
+There are a number of good qualitative results for understanding the learned object-oriented representation.
+The approach of learning 3D representations by comparing projections to observations is a good direction.

Summary of negatives:
- The method is quite complex and explained, in my view poorly (although I'm open to the other reviewers' opinion on the matter).
- The experiments are weak
- The manuscript makes fairly broad claims that aren't substantiated and ignores a great deal of work in the vision community on this topic. 

Given the three largely orthogonal and fairly strong negatives, I lean heavily towards rejecting this paper. Independently each of these is an issue that would be push me to at least lean towards rejection. However, I would encourage a revision and resubmission with improved method explanation, stronger experiments, and a clearer picture with respect to existing work.

In more detail:

Method: 
-I found the method section quite difficult to read, in part because the method is quite complex. This isn't intrinsically a bad thing, but complex methods with lots of steps should come with few surprises and descriptions that make the method accessible. In particular, the method section would benefit from a stronger figure that in part introduces the notation, as well as a little more thought in terms of the introduction of the method. A few instances: 
1) "This is done by an order-invariant object encoder r_n = f_{order-invar-obj}(...)". One turns to the appendix, and tries to find this function. It's not explicitly there -- instead you need to match r_{n,C} = \sum_{i} .... , then look up above at the note that "ObjectRepNet is the module we use for object level order invariant encoder", then remember that sum is order invariant. 
2) I searched throughout the paper and couldn't find precisely what model f3d->2d was. The figure suggests a projective camera and the text says "For more details on the coordinate projection, refer to Appendix.", but there's none in the appendix as far as I can see. 
3) STN-INV is nowhere defined -- inverse spatial transformer? 
4) s^{what} doesn't appear to be anywhere in the appendix -- is s^{what} factored into y^att and alpha^{att}? By matching the RHS, this seems to be the only possibility, but in the main body it's called ConvDRAW aka the GQN decoder, but in the appendix it's called Renderer.   
5) There are lots of other little things -- e.g., a figure that refers to a variable that doesn't exist (see small details section )

I don't see why a paper this complex can't be accepted at ICLR, but I think at a minimum, the appendix should be more complete so that things are well-defined. I'm open to the possibility that I may just be slow so long as the other reviewers think the paper is crystal clear down to the details. However, I think even if I'm just the slow one, the authors should think about writing this more clearly and using consistent notation and function names. 

Experiments:
-As far as I can see, the difference between ROOTS and GQN is that GQN is a little more blurry in its output (Figure 2) and ROOTS has a slightly better MSE for lots of objects (Table 1) but produces NLLs similar to GQN. There are a few issues with this:
(a) It's surprising that the correlation between larger numbers of objects and better MSE isn't really investigated -- why not show that GQN breaks at some point? The differences right now are fairly small, and I think the paper ought to delve into details to demonstrate that the differences are important.
(b) There are so many changes between ROOTS and GQN that I don't know if this has to do with the object-bits of it, or something else. This is part of a larger problem where there are no ablations. A large complex system is trained, and lots of changes are made to GQN. But when there are no ablations, it's unclear what parts of the changes are important and which parts aren't.
(c) It's not clear whether the GQN and ROOTS are being trained fairly -- do they have the same capacity? Why are they both trained for the same number of epochs? It seems entirely likely that ROOTS may train faster than GQN (or the reverse!). If there's only one experiment like this, why not train for a long enough time to ensure convergence and then take the checkpoint with best validation performance? 
-The NLL results are very weak and probably not worth putting in, at least without some sort of explanation for why this gap is significant.
-The object detection quality experiment is incomplete -- I just do not know how to parse the numbers that are presented without some sort of simple baseline in order to make sense of things. Why not also try something like this on GQN? 
-The qualitative experiments are nice but would be substantially improved by showing that:(a) that GQN doesn't do any of these (b) that ROOTs can train on 2-3 objects and test on 3-5 objects by simply changing the prior on K -- this is one of the primary advantages of object-centric representations of scenes (the ability to handle arbitrary numbers of objects). 


Related work:

The paper really needs to make its claims more specific and position itself better with respect to related work. 

Two gratuitous examples: 
1) The title is "Object-oriented representation of 3D scenes", which covers decades of work in robotics and vision. This title should be changed. 

2) "First unsupervised model that can identify objects in a 3D scene" is exceptionally broad: voxel segmentation is already a standard feature in point cloud libraries (e.g., "Voxel Cloud Connectivity Segmentation - Supervoxels for Point Clouds" Papon et al CVPR 2013). Is the manuscript and the Papon paper the same at all? No. But are they both unsupervised models that can identify objects in scenes. I'm not demanding that the authors write out a claim of novelty that's like a patent, but claiming "first unsupervised model that can identify objects in a 3D scene" is, in my opinion, clearly incorrect and needs to be qualified.


The paper should also better position itself compared to the wide variety of work that's been done on unsupervised 3D shape estimation/feature learning using reprojection. For instance (among many): 
(1) Geometry-Aware Recurrent Neural Networks for Active Visual Recognition. Cheng et al. NeurIPS 2018
(2) Learning a multi-view stereo machine. Kar et al. NeurIPS 2017.
(3) Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency. Tulsiani et al. 2017.(4) Unsupervised Learning of Depth and Ego-Motion from Video. Zhou et al. CVPR 2017 (not voxels, but 2.5D or a form of 3D)

as well as the vast array of work on 3D reconstruction, including work that is object-oriented
(1) Learning to Exploit Stability for 3D Scene Parsing. Du et al. NeurIPS 2018.
(2) Cooperative Holistic Scene Understanding: Unifying3D Object, Layout, and Camera Pose Estimation. Huang et al. NeurIPS 2018
(3) Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene, Tulsiani et al. CVPR 2018
(4) Potentially not out at ICLR submission deadline, but 3D Scene Reconstruction with Multi-layer Depth and Epipolar Transformers. Shin et al. ICCV 2019.

I agree that there are differences between these works and the manuscript, but it's really peculiar to work on inferring a 3D volume of scenes from a 2D image or set of images, and only cite YOLO, faster RCNN, and FCNs from the world of CVPR/ICCV/ECCV etc where this work is done very frequently. These works do indeed sometimes rely on a bit more supervision (but not always). But they're tested on data that's far more complex than a set of spheres and cubes.



Small issues that do not affect my score.
- The claim that the method is unsupervised when it has access to precise camera poses seems a bit like a stretch to me. It's common enough that I've given up quibbling about it. Peoples' sense of distance is not exact. This deserves some further thought.
-The authors should go through and double check their use of GQN and CGQN -- it's said at the beginning that GQN just means CGQN, but then it's occasionally dropped (e.g., right before Table 1)
- Fig 1 shows z^{where}, which I guess got renamed?
- "The Scene Representation Network is modified bases on Representation Network in GQN." -> this sentence is presumably a typo/cut off halfway through.
- " This helps remove the sequential object processing as dealing with an object does not need to consider other objects if the features are already from spatially distant areas." -> this is unclear
- Eqn 11 appendix "sacle" -> "scale"
- "Object Representation Network" in A.2 "objcet" -> "object"
-Equation 15 -- the parentheses put i \in \mathcal{I}(D) inside the Renderer, which is presumably not true.
-Table 1 -- table captions go on top








----------------------------------------------------
Post-rebuttal Update: 

AC: I would give a rating of 5 if the full revision is considered acceptable (since the paper is more clear), and increase to 4 if it is not (since there are some more experiments, although I think they are still quite weak). 

I'm still inclined to reject the paper on the grounds of experimental comparisons and the open question of whether  but recognize that my concerns are ones which may not be shared by all communities (and that this is not my community).
1) Ablations/Comparisons to GQN: This may just be a cultural thing, but I'm puzzled by the claims that certain things (direct comparisons to GQN on feature representation, and ablations) don't need to be empirically done. 

In my view, results really need to be empirically demonstrated. Simply stating that ROOTS has the capacity to decompose things into 3D and GQN doesn't have this built-in isn't enough. GQN has a feature vector, and it would not be surprising if it implicitly already did some of this. There have been far too many recent results in ML where a complex method is presented and it is asserted this complex method is necessary, followed by work that shows that  simple method does as well, typically due to issues in the dataset. I don't think expecting linear readouts of systems is too much of a burden to ask, if only to put the proposed work in context. 

Indeed: the experiments in the revision show that ROOTS often does *worse* in generalization performance to previously unseen objects (improving in only 6/9). This is surprising -- if GQN is supposed to break, why doesn't it break here? I appreciate the author's response that there's a latent variable of # of objects that needs to be adjusted in the case of ROOTs, but this should be investigated.

The same thing goes with the claim that an ablation study is only necessary for improving results. This is just baffling -- is it possible that only certain parts of the method are necessary? Surely this is a problem that is worth studying. What if it's just that some aspect of the system has higher capacity than the equivalent in GQN and just works better?

AC: I realize that I'm just the cranky computer vision person shouting about numbers and I may be out of my element here, so take this as you want. But in my view, things really need to be evaluated (since vision struggled for many years because people showed a few nice qualitative results and didn't put their ideas to the test).

2) Clarity: I completely disagree with the authors that clarity issues were minor -- they really weren't and all reviewers agreed on this. Typos are thing lke tihs that are easy to read through without thinking. These were things that required thinking, flipping back-and-forth-etc. 

The revision appears to be close to a full-rewrite, to the point where the diff system is useless -- it's all red/green. It appears to be clearer, but I haven't checked thoroughly. The ICLR 2020 guide is unclear how you should treat this (the AC guide says "you can ignore this revision if it is substantially different from the original version."). Personally, I don't think it's fair to authors who spent their time making their paper clear in the first place rather than on new results.



Smaller stuff:

-The authors have misunderstood my statement on paper complexity (although I now realize the comment has been edited)-- my point is that people who present a complex system have a strong obligation to present a clear explanation (since there's little opportunity for redundancy in explanation unlike a simple approach). 

-"AP interpretable metric without a relative comparison": this is just not true although openreview is probably not the place to litigate this and I recognize that this is my outlook. Accuracy is highly interpretable: 90% top-1 accuracy on mnist would have been boring in 2005, and 90% top-1 accuracy on imagenet would be very exciting today. 

-f_{3D-2D} There are multiple camera models. Skimming the revision suggests it's perspective projection, but the authors should realize that there are others (orthographic, weak perspective, etc) and they're often used because they're easier to learn with.
review_title: Official Blind Review #1

Meta-review: The author proposes a object-oriented probabilistic generative model of 3D scenes.  The model is based on the GQN with the key innovation being that there is a separate 3D representation per object (vs a single one for the entire scene).  A scene-volume map is used to prevent two objects from occupying the same space. The authors show that using this model, it's possible to learn the scene representation in an unsupervised manner (without the 3D ground truth). 

The submission has received relatively low scores with one weak accept and 3 weak rejects.  All reviewers found the initial submission to be unclear and poorly written (with 1 reject and 3 weak rejects initially).  The initial submission also failed to acknowledge prior work on object based representations in the 3D vision community.  Based on the reviewer feedback, the authors greatly improved the paper by reworking the notation and the description of the model, and included a discussion of related work from 3D vision.  Overall, the exposition of the paper was substantially improved.  Some of the reviewers recognize the improvement, and lifted their scores.  

However, the work still have some issues:
1. The experimental section is still weak
The reviewers (especially those from an computer vision background) questioned the lack of baseline comparisons and ablation studies, which the authors (in their rebuttal) felt to be unnecessary. It is this AC's opinion that comparisons against alternatives and ablations is critical for scientific rigor, and high quality work aims not to just propose new models, but also to demonstrate via experimental analysis how the model compares to previous models, and what parts of the model is necessary, coming up with new metrics, baselines, and evaluation when needed.

It is the AC's opinion that the authors should attempt to compare against other methods/baselines when appropriate.  For instance, perhaps it would make sense to compare the proposed model against IODINE and MONet.  Upon closer examination of the experimental results, the AC also finds that the description of the object detection quality to be not very precise.  Is the evaluation in 2D or 3D?  The filtering of predictions that are too far away from any ground truth also seems unscientific.  

2. The objects and arrangements considered in this paper is very simplistic.  

3. The writing is still poor and need improvement.
The paper needs an editing pass as the paper was substantially rewritten.  There are still grammar/typos, and unresolved references to Table ?? (page 8,9).


After considering the author responses and the reviewer feedback, the AC believe this work shows great promise but still need improvement.  The authors have tackled a challenging and exciting problem, and have provided a very interesting model.  The work can be strengthened by improving the experiments, analysis, and the writing.  The AC recommend the authors further iterate on the paper and resubmit.  As the revised paper was significantly different from the initial submission, an additional review cycle will also help ensure that the revised paper is properly fully evaluated.  The current reviewers are to be commended for taking the time and effort to look over the revision. 
Decision: Reject