Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The current method proposes a scene generation technique that models explicit knowledge by imposing semantic rules on the objects in the scene. It generates meaningful scene layouts and shown to be perform better than some simple baselines. 
main_review: 
## Strengths 
* Training a model to generate scene layouts while enforcing explicit constraints during training is novel. However, training a neural network while enforcing explicit constraints on the output has been studied before [Ref 1]. 

## Weaknesses
* "GVAE leverages knowledge by integrating rules during the training stage; however, it cannot explicitly integrate semantic knowledge during the generation." -- Disagree, rules are explicit knowledge about the scenes and can certainly be incorporated in the similar fashion as  [Ref 2] as long as those are simple co-occurrence based. However, the advantages of the current model over the sg-vae seem to be the followings: (1) the current method could model any constraints of the objects (possible higher order constraints)  whereas sg-vae could model only co-occurrences of different objects, and (2) the optimization-based formulation of the current method enforces a soft-constraints where as the grammar-based structure of sg-vae enforces hard-constraints on the object co-occurrences. 

* "When combining the three knowledge, even from a random initialization, our T-VAE can finally find the target scene, leading to a small error in Table. 1." --- The external knowledge is adding explicit constraints in the scene generation. Those constraints reduce the search space and that leads to a better minimum? That being said, the main point here to discuss is how one could incorporate the external knowledge for the current scene generation task. This work does so by fine-tuning a t-vae while enforcing the constraints on the scene layout. An interesting evaluation however would be the comparison w.r.t. The following baselines (1) T-vae + imposing the constraint during test time, (2) current model (trained with explicit constraints, ie external knowledge) without explicit constraints during testing time. Since the constraints were imposed during the training time of the current method, the network might already know how to impose the constraints.  Also, the external knowledge constraints were not employed in the baselines Direct Search (DS) and VAE. 

* The scene attack in the current method is not convincing. To me the adversarial attack is something where we modify the image / scene by a small amount (invisible to the naked eye) to deceive an existing trained model. Now, if a trained model performs poorly on a synthesized scene then the scene could be bad. It would be more convincing if the attack changes only a tiny amount of the 3d point cloud and still could  fool a trained model. 

* The synthetic scene layouts utilized in this work are rather simple. An experiment on the suncg datasets comparing against a stronger baseline [Ref. 3] would be good.  


## References 
* [Ref. 1] Gould, Stephen, Richard Hartley, and Dylan Campbell. "Deep declarative networks: A new hope." arXiv preprint arXiv:1909.04866 (2019). 
* [Ref. 2] Purkait, Pulak, Christopher Zach, and Ian Reid. "SG-VAE: Scene Grammar Variational Autoencoder to generate new indoor scenes." European Conference on Computer Vision. Springer, Cham, 2020.  
* [Ref 3] Li, Manyi, et al. "Grains: Generative recursive autoencoders for indoor scenes." ACM Transactions on Graphics (TOG) 38.2 (2019): 1-16. 
summary_of_the_review: ### Summary
* A novel formulation and a sound approach to generate novel scenes 
* Weak evaluation -- very simple synthetic dataset, unconvincing attack 
* Weak baselines -- no comparison against s.o.t.a scene layout generation methods
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a way to incorporate explicitly defined (e.g. rule based) knowledge into generative models for scene generation. Through the addition of explicit knowledge the approach can ensure that generated scenes follow specific requirements, e.g. physical rules. The approach is evaluated on a synthetic dataset and as a way to generate adversarial examples for scene based segmentation models.
main_review: ---
Disclaimer: I am unfamiliar with this area and can only give a very high-level review.
---

The ability to inject explicit knowledge into generative models would be very helpful for many tasks. Many generative models struggle and ignore even simple physical rules and a way to explicitly incorporate them would likely lead to better results and more data-efficient models. The authors propse to represent scenes in a tree-based manner where nodes correspond to objects and edge to relationships between objects. A VAE model is then trained to learn a structured representation of the data. After the VAE is trained the tree's characeristics can be adapted with explicit knowledge to ensure that generated scenes follow these rules.

My first question in this regard is why you focus on tree-based instead of graph-based representations? It feels to me that graph-based representations would offer more flexibility. Additionally, there is a large amount of work on graph neural networks that you could build upon. Additionally, there is a large amount of literature for graph-to-image which you could compare against.

I also don't follow the motivation for evaluating your approach on generating adversarial traffic scenes. I feel there are many other applications to evaluate the general generation capabilities. Generating adversarial attacks may not be the first thing that comes to mind for incorporating knowledge into generative models. Besides just modeling "general" scenes this could be applied to biological or chemical settigns were we have clear pre-defined rules.

As a follow-up ont this: Where do you get the knowledge from? Does the knowledge need to be hand-designed? Can the system easily scale to large rule-bases? Or is it possible to learn the rules directly from the data such that they only need to be "verified" or adjusted afterwards?

Finally, could this approach be used for editing, e.g. relationships between objects? Given that we have the tree representation, could we obtain the tree representation of a given scene, edit it, and then render the new, edited scene?
summary_of_the_review: I am not experienced in this domain but it feels to me that this should be more closely investigated from the view point of graph-based neural networks and graph-based image representations and rendering. The tree-based representation feels very restrictive to me. Also, I'm not sure if generating adversarial traffic scenes is the best way to evaluate this approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to incorporate domain knowledge into the physical scene generation process. They extend the synthetic example to realistic environments. Apart from this, they also propose the semantic point could attack against stoa segmentation methods.

main_review: 
Strength:
- The tree-structured generative model looks interesting, and they propose a two-stage training method to learn such a model.
- The comparison with baseline methods looks good, and the authors also show some interesting visualization results.

Weakness:
- I found this paper is a little bit hard to follow; they start from VAE and introduce the T-VAE, which is reasonable. However, aligning those proposed new functions to the basic VAE equation is not easy. I think the authors should briefly introduce the previous approaches instead of simply citing them, e.g., [49] and [55]. 
- The contribution of this paper is the proposed T-VAE and knowledge-guided generation. The knowledge part adopts some explicit knowledge rules. How to get those rules is not clear? Why only three rules? Also, it seems explicit rules are dependent on human experts. Can we introduce some implicit rules?
- The baseline methods are not state-of-the-art methods, and they should consider the more recent works.
summary_of_the_review: The T-VAE and knowledge-guided generation proposed in this paper looks new, and the attack looks interesting. However, the current version is a little bit hard to follow, and some contributions seem incremental, for example, the proposed T-VAE compared with VAE. I can reconsider my rating if the authors point anything I was missing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the use of tree-structured VAE as a mechanism for
encoding several forms of constraint knowledge. The VAEs create samples
of scenes that aim to conform to the constraints. A synthetic scene context
was used to demonstrate and explore the approach. A more realistic LIDAR
segmentation scenario was also explored, where the goal was to generate
realistic but adversarial scenes.

main_review: This paper is quite far from my expertise so it is hard to evaluate the
originality. I have not seen this approach previously, and the literature
review supports the claimed originality.

The proposed approach does generate examples of the target scene from
random initializations, unlike most of the compared approaches that did
not have the constraints embedded. With the LIDAR scenes, the proposed approach
successfully generated more adversarial examples which also conformed
better to road traffic constraints. The adversarial examples generated
wrt one system were generally also usable as adversaries for another
system (why? what made these so effective?).

There are 2 weaknesses:
1) The paper is quite hard to follow. There is a lot of precise notational
description which could use some 'intuitive' explanation to help readers
make sense of the theory. In the case of the experiments, there is a lot of
precise detail that is missing that leads one to wonder if the claims are
a bit exaggerated. There is more detail in the appendices, and it would
help the paper to at least reference the relevant sections in the appendices.
2) There are a lot of worries about the experiments: a) why does the synthetic
target configuration have the given precise box and plate configuration,
since these are not in the constraint specification? b) The LIDAR pose attack
'cars' are at what looks like random orientations. Since the generator is
created by the authors, it is unclear why the cars could not be aligned with
the lines. c) It looks like a lot of hand-crafting is needed for constructing
the knowledge / constraint tree, and it looks like one would need a new
tree for each application. d) In the 8 cases of the LIDAR pose and scene
adversary generation, the pose approach was better in 5 of the cases, and
in the 3 cases where the proposed scene approach was better, the adversaries
were not that successful (assuming that IoU was a good measure of
adversariality).

There are a number of confusing details that could be clarified:
1) How is 'adversarial' defined in the case of the LIDAR example?
2) It is unclear what transferabilty between the Point and Scene
attack scenarios means.
3) In the case of at least 2 of the LIDAR systems, the proposed approach is
not very effective at generating adversaries. Why?
4) The paper did not clearly show how the constraints were implemented based
on the theory given in the paper, although the diagrams in the
appendices helped.
5) It was never clear what constitutes an adversary in terms of the
compared algorithms, nor why the generated LIDAR structures should be
considered an adversary.
6) Why is this? Does it not bias the experiment? "We also inject the target sce\
ne 10 times into the dataset to make
sure it accessible to all models."
7) What is the blue curve in Fig 4b?

summary_of_the_review: The approach seems reasonably novel, and the paper's writing could be improved. The experiments and applications are not very exciting.
review_title: null

Meta-review: This paper presents a semantically controllable generative framework by integrating explicit knowledge. In particular, a tree-structured generative model is proposed based on knowledge categorization. Reviewers raised concerns about technical details, experiments, and missing references. In the revised paper, the authors provided more justifications and clarifications, such as the definition of adversarial attack. During the discussion, reviewers agreed that the previous concerns have been partially addressed, but there are still concerns on experiments, e.g., more recent work should be considered as baselines.

Overall, I recommend to reject this paper. I encourage the authors to take the review feedback into account and submit a future version to another venue.
Decision: Reject