Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: -- Update --
Thank you for the detailed response and updates. The quality has improved but as mentioned in the original review, it would be good to see results on a more significant non-synthetic task before I would argue for acceptance.

------------------
The authors propose a generative model with a hierarchy of latent variables corresponding to a scene, objects, and object parts. The method is evaluated on a synthetic dataset with manual inspection, MSE, and object counting performance. The method can form bounding boxes around objects and parts in the examples that are shown, and the MSE is similar to that of a VAE.

The idea of decomposing a probabilistic model hierarchically is potentially interesting, but this paper has drawbacks in terms of experimental quality, significance, and presentation.

-- Experiments -- 
The experiments are done on a manually constructed synthetic dataset, so it is not clear whether the proposed method would work in more realistic or more challenging settings. For instance, the dataset was constructed using a recursive process which probably does not resemble a realistic distribution of objects and their parts, and the parts are well-separated within the object. 

The experiments on the synthetic dataset are not thorough, and could be more interesting with ablations, evaluating performance as the dataset parameters vary (e.g. number of objects, amount of occlusion), better baselines, and an evaluation metric rather than manual inspection.

-- Significance --
With respect to significance, the formulation is fairly straightforward and builds off of previous techniques introduced in e.g. SPAIR. The lack of demonstrating usefulness on a downstream application limits this paper's significance; for instance the SPAIR paper used the generative model as a front-end for a card game and Atari game. The paper should more clearly demonstrate the benefits/tradeoffs of the hierarchical aspect.

-- Presentation --
The paper leaves many terms undefined so the paper is not self contained, e.g. the 'problem of routing', 'memory buffer', 'memory template', 'variational memory addressing', 'spatial transformer' (missing a reference here). I suspect the related work is missing references, e.g. "Unsupervised Discovery of Parts, Structure, and Dynamics" Xu et al ICLR 2019. Finally, the paper should be proof-read for grammatical errors.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: Update: I thank the auhors for the detailed response. The updated paper does look more convincing, but my main concern remains - all considered datasets and tasks are synthetic and specifically made for the method. I think experiments on more real data would be crucial to show the potential of the method. Some examples I can imagine would be part segmentation in images or 3D models, or robotic control.

---

The paper proposes an unsupervised approach to learning objects and their parts from images. The method is based on the "Attend, Infer, Repeat" (AIR) line of work and adds a new hierarchy level to the approach, corresponding to object parts. The method is evaluated on two custom synthetic dataset: one composed of simple 2D geometric shapes and another one with 3D geometric shapes. On these datasets the method successfully infers the object-part structure and can parse and reconstruct provided images.

While the general topic of the paper is interesting, I do not think it is fit for publication. First and foremost, the experiments are very incomplete: the method is only evaluated on two custom synthetic datasets and not compared against any baselines or ablated versions of the method. Moreover, the proposed approach seems like a relatively minor modification of SPAIR (Crawford & Pineau).

Pros:
1) The topic is interesting and the method generally makes sense.
2) The method works on the tasks studied in the paper, including relatively challenging scenarios with 3D occlusions or ambiguity in assigning parts to objects.
3) The paper shows generalization to a number of objects different from that seen during training.

Cons:
1) Experiments are limited:
1a) The method is evaluated on custom and relatively toy datasets. It is unclear if it would apply to more practical situations. While I agree that at the high level the question of decomposition of object into parts is interesting, it is still important to connect research in this direction to potential practical applications. Where could the proposed method be used? Perhaps in some control tasks, such as robotics? Could it be applied to more realistic data, such as for instance ShapeNet objects?
1b) There are no comparisons to baselines. One might argue that baselines do not exist since there are no methods addressing the same task. Generally, it is the job of the authors to come up with relevant baselines to show that the proposed model actually improves upon some simpler methods. One variant of obtaining baselines is via ablating the proposed model. Another is by taking prior approaches, for instance object-based ones without the part decomposition, and showing that the part decomposition allows to improve upon those in some sense.

2) The novelty is somewhat limited: the method seems like a relatively straightforward extension of SPAIR by adding another hierarchy layer. It might be sufficient if the experimental results would be strong, but given that they are not, becomes somewhat concerning. If the method does indeed include significant technical innovation, it might be helpful to better highlight it.

3) The related work overview is extremly limited. It is authors' job to provide a comprehensive ovreview of prior literature and I cannot do it for them here, but below are a few papers that come to mind. This is by no means a complete list.

[1] Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu. Unsupervised Discovery of Parts, Structure, and Dynamics. ICLR 2019
[2] Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, Jitendra Malik. Learning Shape Abstractions by Assembling Volumetric Primitives. CVPR 2017
[3] Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, Leonidas Guibas. GRASS: Generative Recursive Autoencoders for Shape Structures. SIGGRAPH 2017.
[4] Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, Subhransu Maji. CSGNet: Neural Shape Parser for Constructive Solid Geometry. CVPR 2018.
[5] Adam R. Kosiorek, Hyunjik Kim, Ingmar Posner, Yee Whye Teh. Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects. NeurIPS 2018.

4) Presentation is at times suboptimal:
4a) It would be very helpful to have more visuals and intuitions about the functioning of the method, as opposed to equations. Equations are definitely good to have, but they are not the easiest to parse, especially by those not intimately familiar with this specific line of work. 
4b) It is quite unclear to me how and why is the memory used in the model. If this is described in another paper, it would be useful to point there, but still briefly summarize in this paper to make it self-contained.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Contributions: this submission proposes a generative framework which employs a hierarchical decomposition of scene, objects and parts. It modifies the SPAIR framework (Crawford & Pineau) and replaces the recurrent generation with parallel generation, by assuming the number of objects is known and fixed. Results are presented on two synthetic datasets with 2D or 3D shapes.

Assessment:
- The proposed model and learning framework are closely related to previous work (AIR, SPAIR). The authors replaced the "sequential processing ... with single forward pass", which requires the number of objects to be fixed.
- In the proposed model, there is no constraint on the part-object hierarchy, and it is unclear whether it can learn to discover such hierarchies directly by reconstructing single static images. It is also not evaluated in the submission.
- There are no comparisons with AIR or SPAIR on the two datasets the authors created.
- The authors are recommended to compare with previous work on hierarchical generative modelings with objects and parts, such as (Xu et al, ICLR 2019), and other related work, such as SPIRAL (Ganin et al.)

Due to the limited contribution, lack of comparison with related work and limited empirical evaluation, I recommend rejection of the submission.

[1] Xu el al, Unsupervised Discovery of Parts, Structure, and Dynamics, ICLR 2019.
[2] Ganin et al., Synthesizing Programs for Images using Reinforced Adversarial Learning.


--------------------------------
Post-rebuttal:
Thank you for your detailed answers to my questions and updated manuscript. The writing has indeed significantly improved and some questions (e.g. varying number of objects) have been addressed. After reading the rebuttal, my concerns remain that: (1) lack of empirical evaluation on more complex real / synthetic datasets; (2) it is still unclear to me how such hierarchy is inferred from single images. 
review_title: Official Blind Review #3

Meta-review: The authors proposes a generative model with a hierarchy of latent variables corresponding to a scene, objects, and object parts. 

The submission initially received low scores with 2 rejects and 1 weak reject.  After the rebuttal, the paper was revised and improved, with significant portions of the paper completely rewritten (the description of the model was rewritten and a new experiment comparing the proposed model to SPAIR was added).  While the reviewers acknowledged the improvement in the paper and accordingly adjusted their score upward, the paper is still not sufficiently strong enough to be accepted (it currently has 3 weak rejects). 

The reviewer expressed the following concerns:
1. The experiments uses only a toy dataset that does not convincingly demonstrate the generalizability of the method to more realistic/varied scenarios. In particular, the reviewers voiced concern that the dataset is tailored to the proposed method

2. Lack of comparisons with baseline methods such as AIR/SPAIR and other work on hierarchical generative models such as SPIRAL.
In the revision, the author added an experiment comparing to SPAIR, so this is partially addressed.  As a whole, the paper is still weak in experimental rigor.  The authors argue that as their main contribution is the design and successful learning of a probabilistic scene graph representation, there is no need for ablation studies or to compare against baselines because their method "can bring better compositionality, interpretability, transferability, and generalization".  This argument is unconvincing as in a scientific endeavor, the validity of such claims needs to be shown via empirical comparisons with prior work and ablation studies.

3. Limited novelty
The method is a fairly straightforward extension of SPAIR with another hierarchy layer. This would not be a concern if the experimental aspects of the work was stronger. 

The AC agrees with the issues pointed by the reviewers.  In addition, the initial presentation of the paper was very poor.  While the paper has been improved, the changes are substantial (with the description of the method and intro almost entirely rewritten). Regardless, despite the improvements in writing, the paper is still not strong enough to be accepted.  I would recommend the authors improve the evaluation and resubmit. 
Decision: Reject