Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: == Update ==

Thank you for your response and clarifications. I have left my score as is.

== Original Review ==

The paper presents an unsupervised method for inferring scene graphs from images. Building upon
scene-attention methods such as AIR and SPACE, it hierarchically decomposes a scene into objects and
those objects into parts, giving rise to a tree structure. It is shown that this model successfully
recovers the hierarchies underlying the data on two newly proposed hierarchical variants of the
Sprites and CLEVR datasets.

Strengths:
 1. The paper is well written, and despite the considerable complexity of the method, its
    presentation is relatively easy to follow.
 2. The task of interest is well-defined, and has clearly been effectivly solved on the datasets
    considered. Both quantitative and qualitative evaluations make it very clear that the model has
    learned to infer the correct scene graphs as desired. Its ability to infer the appearance of
    occluded parts is especially impressive.
 3. While there are no direct competitors on this newly defined task, the paper does a decent job of
    comparing to the closest available baseline, showing how the additional structure can be
    beneficial.

Weaknesses:
 1. One may argue that the datasets have been deliberately constructed to showcase the model. While
    that is probably true, I think this is a valid approach given the novel nature of the task and
    the lack of supervision. Despite the clearly helpful structure (limited number of objects and
    parts), the datasets still appear sufficiently challenging.
 2. In the experiments, scene graphs are limited to trees of height 2 and degree 4. This is a
    significant constraint, however, each additional level of hierarchy introduces ambiguities and
    makes it harder to learn the graph in an unsupervised manner. More complicated structures would
    likely require supervision.
 3. As far as I can tell, the object types were chosen once to generate the datasets, and then kept
    fixed across the experiments. Reporting results for multiple different datasets with randomly
    chosen object types would be somewhat more convincing.
 4. As is common for unsupervised scene models, the proposed method likely only works on synthetic
    images in its current state. However, due to the additional structural assumptions on the data,
    it seems especially challenging to find suitable real-world use-cases.

Overall, the paper presents an effective new method for the task it sets out to solve. While it is
questionable how it would work on real-world data, I believe the paper is of sufficient interest as
a proof of concept, and am therefore leaning towards acceptance.

Questions:
 1. It is stated that auxiliary KL terms are added, with the sparseness constraint on $z^{pres}$
    being one of them. But is not clear if there are others. This would be important to know in
    order to evaluate how strong the model's inductive biases are.
 2. The downstream task used for Fig. 5 is not clear to me. If the number of parts is computed for
    each object, and these numbers are then summed, isn't the result equal to the total number of
    parts in the scene, which SPACE-P can also infer? If only distinct parts are counted,
    how is equality of parts defined on the dataset?
review_title: A convincing proof-of-concept solution for a difficult new task

final_score: rating: 6: Marginally above acceptance threshold
review_content: Generative Scene Graph Networks (GSGN) is a variational auto-encoder with the intermediate representation being tree-like scene graphs. The leaf nodes stand for primitive parts and edges stand for poses to compose parts into objects recursively. The experiments are done in two image datasets of single color, simple shape 2D/3D objects: Multi-dSprites and CLEVR, and the model is able to discover objects without supervision.

**Strength**: I find the direction important, and the method well established in a variational inference framework with graphics-inspired designs. Experiment numbers look good in general.

**Weakness**: Perhaps my biggest concern is that the current datasets are a bit weak. First, objects are too simple (single color, simple shape), so we are not sure if GSGN can work with more realistic visual domains where objects have more complex 3D structure or texture. Second, the object/primitive decomposition is slightly weird to me. I would expect hierarchal structure like an object being human body, and parts being legs, arms, head and so on. But in this work a "part" is a single-color object, and an “object” is a bunch of single-color adjacent objects. Based on these two points, I believe the paper will be much stronger with experiments on more complex objects like humans or tables or 

Another concern is the application of learned scene graph. The paper only shows unconditional sampling, but not really how to use the learned scene graph. For example, I'd expect scene graphs to be used for image manipulation, as one can change part of the object (shape, color, pose) without changing the rest. Showing the learned scene graph is useful for any downstream tasks can be a great plus for the current work.

Finally, I wonder how variational the learned scene graphs can be, as the objects in the datasets are fairly simple and the learning might be easy. I'd be happy to see some analysis but this is not my main concern.

---------

After rebuttal: I'm glad they added some experiments and analysis I wanted to see, so I raise the score to 6. As the authors said, the paper is a proof-of-concept of unsupervised hierarchal scene graph learning, and the rebuttal to some degree reassured me. For example, modeling a cube on top of another top as two parts of an object (which was weird to me: why not each cube as an object?) helps edit tasks where the top cube is enlarged but the "on top" relation is maintained. The downstream representation transfer also makes sense. Of course experiments are still toy from computer vision perspective, but I'm now okay with acceptance.

review_title: auto-encoder for object-part scene graphs in simple environments

final_score: rating: 4: Ok but not good enough - rejection
review_content: =====Post-Rebuttal Comment=====

I thank the authors for the detailed response and the updated results. While my overall opinion of the work is slightly more positive post-rebuttal, I still maintain that this is a clear reject, primarily for the following reason:

- The technical contribution (adding a hierarchical layer to SPAIR and demonstrate the hierarchy can also be learned without supervision) is not significant enough to accept purely based on a "proof-of-concept" of a "new" direction.
- For incremental contributions, I expect the experimental results to be more convincing to be acceptable, a few points that I still really expect to see:
    - comparisons on harder datasets when one doesn't have to go into a specific metric to show an edge over a prior art that is targeting a different application
    - results on decomposition with more significant overlaps (especially in 2D) and on objects where part boundaries are harder to infer (actual 3D objects is still preferable)
    - object-level manipulation (row 3&4 in fig 4 did not match description) and latent space interpolation

I would also strongly encourage the authors to highlight the similarity and different between SPAIR and SPACE when introducing the latent code formulation.

=====Summary=====

This work proposes a method that can learn a three-level (scene, object, part) hierarchical representation of scenes in an unsupervised manner. A variational autoencoder is used here, where the encoder recursively infers the shape and pose of individual objects and parts, and the decoder recomposites the inferred parts are then recomposited to the inferred poses. The proposed method is evaluated on two datasets created for this paper. The results suggest that the proposed method can reliably breaks down scenes into meaningful objects and parts, and performs slightly better than another method designed for a slightly different task in terms of reconstruction quality, learned representation, and data efficiency for downstream tasks.

=====Strengths=====

- The motivation of the need for hierarchy is solid, and the solution proposed seems to me to be a reasonable way to impose some sort of hierarchy.
- The model seems to be well-tuned, utilizing appropriate training tricks and architectures.
- Good performance for the evaluations chosen in the paper.

=====Weaknesses=====

- *Very* inadequate attribution of ideas. I am not too familiar with the AIR line of work, but I think quite a few ideas can be traced back to prior works. It would be much better if the authors can, in addition to a brief one sentence mention in related works, add clear discussions for the inspirations of the main design choices.
- Missing discussions of relevant works that do: 1. unsupervised part decomposition e.g. “UCSG-NET - Unsupervised Discovering of Constructive Solid Geometry Tree”, "Bae-net: Branched autoencoder for shape co-segmentation", “CvxNet: Learnable Convex Decomposition”; 2. Learning hierarchical representations e.g. “StructureNet: Hierarchical Graph Networks for 3D Shape Generation”.
- I think the comparisons in this work are neither adequate nor fair. I am not convinced that the two toy dataset used here can prove the superiority of the method. The authors claim that “other works can’t work on our dataset”, but I think the burden of the proof is on the authors to show that their method is superior, even under a more specific setting. In other words, if the method is indeed “general” and can learn good decompositions, then I would expect it to perform better even under an slightly unfair setting i.e. comparing against metrics/datasets adopted in other works. Furthermore, the datasets used in this paper appears to be way too simple as compared to real world data. The authors argue that dataset with a single shape is easier, but I disagree: datasets like partnet contains much more complicated part structures, as well as joints between parts, than what is used here, even with only a single shape. (And it is pretty evident from the qualitative examples that the challenging part is decomposing objects into parts, not decomposing scenes into objects). Last but not least, I want to see more evidence that the proposed method is actually useful in real applications.
- The learned representation does not seem to be of very good quality, as seen in Figure 4.
- A lot of overclaims, to name a few: 1. “GSGN is a general framework for representing and inferring scene graphs of arbitrary depth”: I don’t think a model being able to work a toy setting with three levels will mean that the same framework can be used for more complex settings of arbitrary depth (as an analogy: MLP works for MNIST but not on ImageNet). If the framework can handle more general cases, then show it. 2. “Closely follow the rendering process in graphics engines”: I don’t think applying affine transformations and compositing alone is enough to warrant this claim, it is pretty clear that the learned representation lacks a good sense of “objectness”, as textures, lighting and etc. are all entangled together (evident in Fig 3). There does not seem to be a straightforward way to extend the method to truly parallel the 3D rendering process, neither. 3. “First deep generative model for unsupervised scene-graph discovery”: there are a lot of works that infer structures in an unsupervised way, I don’t think it’s fair to give a very narrow definition of “scene graph” and claim “the first”. 4. “GSGN has capture many predefined object types in the dataset”: I don’t think one can make this claim when there are only three primitives and ten types of objects…

=====Reasons for Score=====

Overall, I have the impression that this work is cherry picking a very specific setting where the proposed architecture works reasonably well. For a work making a quite big claim of being “the first deep generative model that learns …”, I would expect much more comprehensive evaluations than what is currently shown here. Furthermore, many ideas in this work are not attributed properly, making the novelty of the method quite unclear. From my limited knowledge of the direct predecessors of this work, I don’t think there is too much novelty in this paper. I would be more than willing to change my score if the authors can 1. Provide more comprehensive evaluations 2. State the novelty / discuss prior works more clearly. But for now, I tend to give a pretty clear reject.

=====Additional Comments & Questions=====

- I am not sure if translation & rotation alone is a good way to handle 2D renders of 3D objects, since any translation & rotation will result in change of perspectives and illumination e.g. rotating the bronze sphere in Figure 3, row 1 will make the specular highlight inaccurate and translating the blue cube in row 3 will make the top surface less visible. Could the authors justify why predicting translation/rotation make sense, when the perspective/illumination of the object already provides a really strong cue?
- Following previous point: would like to see examples of the same learned object being used in multiple scenes.
- The quality of the learned primitives, as seen in Figure 4, seems to be pretty underwhelming. If the aim of the work is discovering those primitives, would it make more sense to impose a stronger prior on the properties of the primitives?
- Table 1 & 2: why are all the ELBO terms the same? I would imagine them to be different, especially for SPACE-O/P, which, if I understand correctly, is a *completely different* model with different architecture and loss formulation? 
- Table 2 & 3: why are the metrics so close between SPACE-P & GSCN in table 2 but so different in Table 3? Does that suggest unbalanced dataset?
- Still Table 2 & 3: why no comparison between SPACE-O & GSGN for object level occlusion?
- The paper claims that being able to handle background is a unique advantage as compared to other works, but the background used in the toy dataset is quite simple. Would like to see more complex examples.

review_title: Not Sure If the Method is Significant Enough

final_score: rating: 6: Marginally above acceptance threshold
review_content: **PAPER SUMMARY**

The paper presents a generative model for scenes that uses tree-structured latent variables to recursively decompose images into objects and parts, without any object or part supervision during training. The model is trained using variational inference. Experiments are performed on two new datasets (2D Shapes and Compositional CLEVR), demonstrating that the model is able to successfully uncover recursive scene/object/part decompositions in an unsupervised setting. The model is compared against prior work (SPACE) that performs non-hierarchical scene modeling.

**STRENGTHS**

- The paper presents a novel generative model that can infer tree-structured latent variables
- The method is technically impressive, and a clear improvement over the non-hierarchical modeling used in prior work
- The paper presents two new datasets (2D Shapes and Compositional CLEVR) for studying hierarchical scene decomposition. I hope these can be publicly released!

**WEAKNESSES**
- The method is quite complex, and though it is technically impressive I wish that it had been compared against very simple baselines
- No experiments on real-world data
- Unclear how the model will scale to wider and deeper trees
- Many implementation details are unclear

**SCALABILITY**

A big selling point of the proposed model is that it can model the hierarchy of scenes into objects and parts, and the tree-based formulation of the latent space used to achieve this is technically impressive. However, how well would the method scale to larger scenes? All experiments used a relatively small three-level hierarchy where all nodes have a fixed out-degree of 4; thus in all experiments the full tree has just 21 nodes total. How well can the method scale to much larger trees, with deeper hierarchies or wider out-degrees?

**PRIOR KNOWLEDGE**

The method requires choosing a tree depth and node out-degree as hyperparameters. To some extent, these hyperparameters encode very strong prior knowledge about the data being modeled (levels of hierarchy, and number of subparts within each part). In the experiments, these hyperparameters are perfectly matched to the synthetic datasets: you use a two-level tree with out-degree of four, and each image has between 1 and 4 objects, each of which is composed of between 1 and 3 parts. However in more complex real-world scenarios, you may not have such detailed knowledge of the world’s compositional structure. For this reason, I’m curious as to how the method would behave when the structural hyperparameters are mismatched to the underlying statistics of the dataset.

**SIMPLE BASELINES**

The model is evaluated on two synthetic datasets -- 2D Shapes, and Compositional CLEVR. How difficult is the scene graph inference problem on these datasets? I wish that the authors had compared to very simple baselines in order to give a sense for how difficult the problem really is.

For example, I suspect that a simple “handcrafted” baseline that oversegmented the image into superpixels, then recursively merged superpixels based on proximity and simple appearance features (e.g. color histograms) could produce very plausible scene graphs for these two datasets.

At a high level, the promise of an end-to-end learning-based approach compared to a “handcrafted” approach like the above is that a learning-based approach should require less tuning, should be more adaptable to new datasets and tasks, and should scale better to complex real-world datasets where the assumptions of the model don’t perfectly match the statistics of messy data from the real world. However in this case, I’m not convinced that this complex variational method would be any simpler to implement or tune, or even give better results than, a very simple “handcrafted” baseline like the above on these synthetic datasets. There are also no experiments to demonstrate its scalability to real-world datasets.

This leads to a pointed question: If someone wanted to infer scene graphs from images, why should they prefer your approach over a very simple “handcrafted” approach?

**MANY IMPLEMENTATION DETAILS UNCLEAR**

There are many implementation details that are unclear from the paper and supplementary material, without which reproducing the results are practically impossible. Even if some of these details do make sense as part of the main text, they should be specified more explicitly in the supplementary material. For example:

What is the generative process for the presence variables $z_v^{pres}$? This is not clear from Equations 3 or 4, nor the surrounding discussion. From Equation 9, the encoder predicts presence variables conditioned on image patches; but how can you predict the presence variables when sampling an image from scratch?

What is the dimension of the latent variables $z^{appr}_r?$

You say that the pose variables are Gaussian, but how is the pose parameterized in terms of relative location and scale?

From the discussion after Equation 6, the pose variables also include a depth map -- how are these depth maps represented and parameterized? Are they per-pixel Gaussians like the other pose variables?

How exactly are the transparency maps $\alpha_v$ computed from the depth maps? The text states that they are computed “by the softmax over negative depth values”, but this is hard to understand -- what are the sets of values being fed to softmax? If the depth map contains a per-pixel depth then a softmax over space wouldn’t make sense, since this would cause the transparency of an object to depend on its spatial size (since it would have more pixels competing in the softmax).

In Equation 9, the distributions of the latents $z_v^{pres}$, $z_v^{pose}$ are conditioned only on data from the parent node $z_{pa(v)}^{appr}$ and $z_{pa(v)}$. Thus all children of a node will have the same distribution for their pose and appearance. Is this correct? If so, how do you encourage the child nodes to cover all parts of the parent, and not collapse to a single part of the parent?

There are no details about any of the neural network architectures used to implement the model, nor any details about any training hyperparameters (e.g learning rates, training schedule, regularization strengths etc).

**SUMMARY**

The model is technically impressive, and a clear improvement over prior work. However on the whole I’m not sure whether the complexity of the method is actually necessary to solve the problem at hand; I wish that the authors had done a better job demonstrating the benefits of the proposed method over very simple baselines. There are also many implementation details that are very unclear, for which reason I fear that the paper as written is utterly unreproducible.

On the whole I lean slightly toward acceptance, but I hope the authors can address my concerns in their rebuttal.


**AFTER REBUTTAL**

The rebuttal largely addresses my concerns about implementation details.

I am pleased to see the additional experimental results provided by the authors; I think that these do improve the paper. I still feel that some well-tuned handcrafted approach could likely perform on-par with the results of the proposed method, but the comparison with [Wei et al] show that achieving such results is at least not trivial, which does help to better ground the complexity of the task. The additional experiments with a three-level hierarchy show a bit more evidence for scalability than provided in the original paper. While these extra experiments do strengthen the paper, I feel that they don't really address the core issue with the paper, which is whether there is any hope for the proposed method to scale to more complex and realistic datasets.

Overall I think that this is a reasonable paper and I still lean slightly toward acceptance, so I maintain my original rating of 6.
review_title: Technically impressive, but is the complexity necessary?

Meta-review: This submission generated significant discussion between the reviewers; three of them ended up on the "accept" side, but one remained firmly in the "reject" camp.

The main strength of the paper is that it tackles a very hard problem: learning an unsupervised generative model (and accompanying inference model) of scene graph structures given only image data. As one reviewer mentioned, it is remarkable that the authors were able to get their system to work at all, given the seeming intractability of this problem. The work builds upon a clear line of prior work in this area, and the type of data on which it is evaluated ("toy" synthetic datasets a la CLEVR) is consistent with prior art.

Multiple reviewers brought up the "toy" nature of the dataset as a drawback to the paper, but most agreed that this is not reason to reject the paper. Rather, the paper demonstrates a convincing proof of concept that this kind of model can be built, and improvements in the elements out of which the model is composed (generative and inference networks) should improve its applicability to real-world data.

Another question mark raised by multiple reviewers: could a simpler, handcrafted inference procedure work just as well or better? The authors included a new experiment against a hand-coded heuristic in their rebuttal, and their method outperforms it. One reviewer noted that more careful tuning might make a heuristic perform as well as the proposed method, but it is still clear that it is not trivial to get a hand-coded solution to perform well (even for this "toy" data). Another reviewer pointed out that this is one of the main attractions of variational inference methods: the ability to specific knowledge as simple generative priors rather than complex bottom-up inference procedures.

One reviewer, R1, remains negative about the paper. His (it is a he; I know this reviewer) main concern is that the scene graphs used are shallow and have a simple structure, and thus (a) it's not clear what value they add, (b) a simple postprocess could reconstruct them, assuming the individual object parts could be detected, and (c) it's not clear whether the method would generalize to deeper/more complex hierarchies. He believes this calls into question the validity of the entire method.

I am sympathetic to this argument, but I think setting the bar this high may prevent progress in this field. For point (a), the authors included an image-manipulation application in their rebuttal--again, a proof of concept, not a directly useful tool. For point (b), the authors did compare against a hand-coded inference baseline and achieved better results, so while this may be possible, it is probably not as easy as the reviewer suggests. (c) remains an open question, to me. But even if this method as presented cannot generalize to more complex scene graphs, it likely paves the way for future work that can.
Decision: Accept (Poster)