Semantically Controllable Generation of Physical Scenes with Explicit Knowledge
Authors: Wenhao Ding
Bo Li
Ji Eun Kim
Ding Zhao
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Generative Models
Knowledge-intergrated Neural Networks
Physical Scene Generation

Abstract: Deep Generative Models (DGMs) are known for their superior capability in generating realistic data. Extending purely data-driven approaches, recent specialized DGMs satisfy additional controllable requirements such as embedding a traffic sign in a driving scene by manipulating patterns implicitly in the neuron or feature level. In this paper, we introduce a novel method to incorporate domain knowledge explicitly in the generation process to achieve the semantically controllable generation of physical scenes. We first categorize our knowledge into two types, the property of objects and the relationship among objects, to be consistent with the composition of natural scenes. We then propose a tree-structured generative model to learn hierarchical scene representation, whose nodes and edges naturally corresponded to the two types of knowledge, respectively. Consequently, explicit knowledge integration enables semantically controllable generation by imposing semantic rules on the properties of nodes and edges in the tree structure. We construct a synthetic example to illustrate the controllability and explainability of our method in a succinct setting. We further extend the synthetic example to realistic environments for autonomous vehicles and conduct extensive experiments: our method efficiently identifies adversarial physical scenes against different state-of-the-art 3D point cloud segmentation models and satisfies the traffic rules specified as the explicit knowledge.


Under review as a conference paper at ICLR 2022
SEMANTICALLY CONTROLLABLE GENERATION OF PHYSICAL SCENES WITH EXPLICIT KNOWLEDGE
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Generative Models (DGMs) are known for their superior capability in gener- ating realistic data. Extending purely data-driven approaches, recent specialized DGMs satisfy additional controllable requirements such embedding a traffic sign in a driving scene by manipulating patterns implicitly in the neuron or feature level. In this paper, we introduce novel method incorporate domain knowledge explic- itly in the generation process to achieve the semantically controllable generation physical scenes. We first categorize our knowledge into two types, the property of objects and the relationship among objects, to be consistent with the composition of natural scenes. We then propose a tree-structured generative model to learn hierarchical scene representation, whose nodes and edges naturally corresponded to the two types of knowledge, respectively. Consequently, explicit knowledge integration enables semantically controllable generation by imposing semantic rules on the properties of nodes and edges in the tree structure. We construct a synthetic example to illustrate the controllability and explainability of our method in a succinct setting. We further extend the synthetic example to realistic environ- ments for autonomous vehicles and conduct extensive experiments: our method efficiently identifies adversarial physical scenes against different state-of-the-e 3D point cloud segmentation models, and satisfies the traffic rules specified the explicit knowledge.
INTRODUCTION
The recent breakthrough in machine learning enables us to learn complex distributions behind data with sophisticated models. These models help us understand the data generation process so to
realize controllable data generation [165]] [6]. Deep Generative Models (DGMss) [23 34 17), which approximate the data distribution with neural networks (NN), are representative methods to generate data targeting specific style, category, or attribute. However, existing controllable generative models focus on manipulating implicit patterns in the neuron or feature level. For instance, [7] dissects DGMs to build the relationship between neurons and generated data, while (55) interpolates in the latent space to obtain vectors that control the poses of objects. One main limitation of these existing models is that they cannot explicitly incorporate unseen semantic rules, which may lead to meaningless generated data that violates common sense. For example, to build diverse physical scenes for evaluating autonomous vehicles, the generated cars should follow semantic traffic rules and physical laws, which cannot be enforced by directly manipulating neurons. In light of the limitations of previous work, we aim to develop a structured generative framework integrate explicit knowledge [15) during the generation process and thus control the generated scene to be compliant with semantic rules.
Natural scenes can be described with objects and their various relationships [5. Thus, in this paper, we categorize the semantic knowledge that describes scenes into two types, where the first type denoted as node-level knowledge represents the properties of single objects and the second type denoted as edge-level knowledge represents the relationship among objects. We also observe that tree structure is highly consistent with this categorization for constructing scenes, where nodes of the tree represent objects and edges the relationship. By automatically controlling the tree structure during the generation, explicitly integrate the node-level and edge-level knowledge.

---

Under review as a conference paper at ICLR 2022
(a)
(b)
Latent Code
Object
Decoder
Local Information
Encoder
Emply
Latent Code
Information
Decoder
Node-loval Knowledge
Edge-level Knowledge
Scane Renderer
Global
Task Feedback
Information
Forward Generation
Knowledge Guidence
Downstream Task
Figure I: Diagram of proposed SCG. (a) Stage one. Train T-VAE model to learn the representation of structured data. (b) Stage two. Integrate node-level and edge-level knowledge during the generation, and generate controllable samples for the downstream task.
In detail, propose general framework, Semantically Controllable Generation (SCG), which consists of two stages as shown in Figure In stage one, we train a tree-structured generative model that parameterizes nodes and edges of trees with NN learn the representation of structured data. In stage two, explicit knowledge is applied to different levels of the tree to achieve semantically controllable generation for downstream tasks such satisfying certain conditions or reducing the performance of recognition algorithms.
To verify the proposed SCG, we first construct synthetic scene reconstruction example to illustrate the advantages of SCG and provide analysis on its controllability and explainability. With SCG, is possible to generate natural scenes that follow semantic rules, e.g., boxes with the same color should be positioned close to each other. To demonstrate the practicality of SCG, we conduct extensive experiments on adversarial LiDAR scene generation against state-of-the-ar 3D segmentation models. We show that our generated safety-critical physical scenes successfully attack victim models and meanwhile follow the specified traffic rules. addition, compared with traditional attack methods, scenes generated by method achieve stronger adversarial transferability across different victim models. Our technical contributions are summarized below:
We propose semantically controllable generative framework (SCG) via integrating explicit knowledge and categorize the knowledge into two types according to the composition of scenes.
We propose a tree-structured generative model based on our knowledge categorization and construct synthetic example to demonstrate the effectiveness of our knowledge integration.
We propose the first semantic adversarial point cloud attack based on SCG, named Scene Attack, against state-of-the-art segmentation algorithms, demonstrating several essential properties.
SEMANTICALLY CONTROLLABLE GENERATION
We define the scene x E X' in the data space and the latent code Z in the latent space. This paper aims to generate scene x that satisfies certain semantic rules IK, which are related to the downstream task i T. The scene x: will be used to solve the downstream task by minimizing the corresponding objective function (z).. In this section, we first describe the tree-based generative model for learning the hierarchical representations of x, which is important and necessary for applying knowledge to achieve semantic controllability. Then we explain the two types of knowledge to be integrated into the generative model together with the generation algorithm that uses explicit knowledge K, as guidance for downstream tasks.
2.1 FREE-STRUCTURED VARIATIONAL AUTO-ENCORE (T- (A-VAE)
a powerful DGM that combines auto-encoder and variational inference ( estimates a mapping between data and latent code to find the low-dimensional manifold of the data space. The objective function of training VAE to maximize lower bound of the log-like likelihood training data, which is so-called Evidence Lower Bound (ELBO)
VAE 34
ELBO Eg())20)) [log (I) 0)] (gT (1((((()))) ()){)()))
2

---

Under review as a conference paper at ICLR 2022
(a)
(b)
(c)
(n, Node Index
Blue
11. 1)
Ki
12. 13
X
12 3)
12 4)
(3.
(3. 2)
(3.
Object Node
Root Node
Empty Node
Object
Red color
Figure 2: (a) Example of tree of a scene with four types of nodes, where the index (n, i) is explained in (2). The purple and green nodes are internal nodes with child nodes. (b) The breaking process of a 2D space. The color is corresponding to the node type in (a). (c) The knowledge integration example described Section 2.2 the child nodes of the blue color node should have red color.
where KLL is Kullback-Leibie (KL) divergence. {<]| Φ) the encoder with parameters Ø, and p[z|2;; 0) is the decoder with parameters 0. The prior distribution of the latent code p(z) is usually a Gaussian distribution for simplification of KI. divergence calculation.
One typical characteristic of natural scenes is the variable data dimension caused by the variable number of objects, which very challenging represent with a fixed number of parameters in traditional models [44] Besides, many existing structured generative models (64 40 do not consider hierarchy natural scenes nor have the capability to incorporate explicit knowledge.
In general, graph is commonly used to represent structured data [11, but sometimes is too complicated to describe the hierarchy and inefficient to generate. As a special case of graph, tree naturally embed
hierarchical information via recursive generation with depth- sicctr traversal [29] 48). This
hierarchy is not only highly consistent with natural physical scenes, but also makes easier to apply explicit knowledge, supported by previous works in cognition literature [46].
In this work, we propose a novel tree-structured generative model, which is inspired by the stick- breaking approach [58]: Assume we have stick with length W and we iteratively break it into segments ((.1) with:
K
W (1,1) (2.11) + (2.2)
(a.i)
where (n, :) means the i-th segment of the n2-hh level. K,, is the total number of segments in the n-th level. The index starts from and entire length has index (1, 1). The recursive function of breaking the stick follows
w (n+1J)) wil (m.i) ((+1++11) (n.))) (1 - (1.8))
where (m.i) E [0, 1] is the splitting ratio for (m,)) and j is the index the n + 1-th level. Intuitively, this breaking process creates tree structure where the i-th level is corresponding of the tree and segments are nodes in the tree. We extent the above division to 2D space with two parameters a and B. A 2D plane example is illustrated in Figure where each node has 4 child nodes in the next layer.
the i-th layer
Assume there are M kinds of nodes in the scene, we define a batch of encoders Em and decoders Dm for all m E M and n E {I, "1,...,,N' - 1}:
f(m.)) = Em Em((f(nt1,) f(a......,,.... f(at1,ithm) , (:ntt1.)):: dm) [f(++1]]... f f(xtl,4lm) g(++1d)] Dm mm Dm.(((n(nn (f(n.); 0m)
where f(n.i) is named as the feature vector that passes the messages through the tree structure. (m.i) named the property vector of node (n, that stories specific properties such color of the object generated by node (n, i). g(n+lo) is the predicted property vector and I,, is the number of
3

---

Under review as a conference paper at ICLR 2022
children of node m. Besides the encoders and decoders, we also define Classifier to determine the child node type and a Sampler to infer the posterior distribution the latent code,
(l Classifier( (( Oe): [žps ]م] = Sampler( ( (ba)
(5)
where alm,i) is the predicted node type and [Z 2a] used to calculate the latent code z with the reparameterization trick [9]. 6g and 0, are models parameters for the Classifier and Sampler.
Parameters for the encoders q=]|>; Φ)) and decoders p(z|2; 0) are denoted respectively Φ {Ø1. ... &m, &s and 0 = {01.... 8m. Be)- The final structures of the encoder and decoder depend on the tree structure of the data point and vary the dataset. We follow Recursive Neural Networks (RvNN) [6]] to build the tree structure recursively. Finally, the tree is summarized as
σ = {c.g}= = (c
(N,,
(1.1) g (N,
(6)
where c represents the node type. Following T the ELBO of T-VAE to be maximized is
ELBO = By p(c);(c)2;;)
Eng [logr(gl;;0)
KL. (N(ep, =n)1 ())
(c(E,))
Ln(3.9)
where the equality holds because e and g are conditionally independent given The Lc term represents the cross-entropy loss (CE) of all nodes in p(a: O),
N K
ΣΣ Co(d,c) = Σ K m=li=
1
CE(Cin.)), cln.0) (
(8)
where the prior distribution of node type p(c) is calculated from the training dataset and serves as weights. To make the reconstructed tree have same structure as the original one, we use Teacher Foreing [67] during the training stage. However, in the generation stage, take the node with maximum probability as the child to expand the tree. The Ln term uses mean square error (MSE) to approximate the log-likel of node property,
M
K. ΣΣΣ R= t=1
N
CR(0.9)
Σ m=
(mm))
m [lg(m))) 11
where Nm is the times that node m appears in the tree and 1 [-] is the indicator function. In ( we normalize the MSE with Nm instcad of > Kn, to avoid the influence caused by imbalanced node type in the tree. Please refer to Appendix B for detailed model definition and generative process.
The advantage of this hierarchical structure is that we only need to store the global information the root node and use local information in other nodes, making the model easier to capture the feature from different scales in the scene. Moreover, this tree structure makes it possible to explicitly apply semantic knowledge in Stage 2, which will be explained in Section 2.2
2.2 KNOWLEDGE-GUIDEE GENERATION
Suppose there is function set F, where the function ((A) E F returns true or false a given input node A of tree v. Then, we define the two types of propositional knowledge K,, for particular downstream task (we omit below for simplification) using the first-order logic [60] as following.
Definition (Knowledge Set) The node-level knowledge kn is denoted as for a function f F. where A is single node. The edge-level knowledge ke is denoted as fi(A) + vi f2(Bi) for two functions f1, E F. where we apply knowledge fz to all A's child nodes B. Then, The knowledge (1)
set is constructed as KK = -1: (1)
In the tree context, hin describes the properties of a single node, and hic describes the relationship between the parent node and its children. Thus, the implementation of knowledge is based on the recursive traversal of the tree. We summarized this process in Algorithm [ For the specific knowledge used in our experiment, please refer to Appendix C
4

---

Under review as a conference paper at ICLR 2022
Algorithm 2: SCG Framework
Input: Dataset Task loss (x),, Searching budget B, Knowledge set K
Stage 2: Generation
Initialize latent code 2 ~ (0,
Output: Generated scene T,
10
while B is not used up do
Stage 1: Train T-VAE Initialize model parameters (0.
if Le(x) is difserentiable then
12
2
v/ (V
for in D do
else
4
Encode 2 + q(2
14
2 - Black-box optimization
Decode è 2;
Update parameters {0. Φ) by
15
x' < Appl yk (IK, x:)
6
2 < prox (2) with 10
maximizing ELBO 7
Decode the scene 2, p;
Store the learned decoder p;z|2;;)
Then, we apply explicit knowledge to the decoder of T-VAE using the proximal gradient algorithm. The original task objective (xr is augmented to La Lx(x) + (x(x, x²),,), where the second term measures the distance between the original tree and modified tree x'. Specifically, k, changes the property vector of node A in x to g to satisfy f(A), and li traverses the tree x find node A that satisfy fi in ke then the property vector of the children of A to and g respectively. The reference vector c' and g' are related to the downstream task and pre-defined in the function f. Then Lg consists two parts, MSE is used between the property vector g and g', and CE is used between the type vector e and c'.
Algorithm 1: Apply Knowledge
Input: Knowledge IK, Tree (root) Output: Modified Tree x'
Function ApplyK (K, x)
for each knowledge k(m) do
L x' + modify x according k(m)
if has child nodes then for all child nodes x of a do
change the type vector
6
x¹, < Applyk (K, 2;)
Add node n as a child to xx'
return x
For instance, the explicit knowledge described "if one box is blue, its child nodes should be red" will implemented by the following operations. Starting from the root, we find all box nodes whose colors are blue and collect the property vectors g its child nodes; then change g to g', which means the color red. Next, Lx is obtained by calculating MSE between all g' and the original This process is illustrated in Figure
In most applications, Cx(x) requires many resources to compute, while Cg is efficient to evaluate since it only involves the inference of p(r) 0). Therefore, resort to Proximal algorithms (52), type of optimization method, to restrict the searching step to finite region. In our setting, the explicit knowledge can be treated as the trusted region to limit and guide the optimization of the downstream task. The knowledge loss and task objective are alternatively optimized under the proximal optimization framework. We define the proximal operator as
prox (2) = arg min (x<(p<<)>'; x')
=1113
(10)
which projects the candidate to z' to minimizes the knowledge inconsistency caused by Lg. The second term in 10 is regularize to make the projected point close to the original point.
This problem can be solved by gradient descent since the decoder p(z 0) of T-VVE is differentiable. After the projection, we need to solve the original optimization problem (z))). We use gradient descent for differentiable (x) change to black-box optimization methods [6] when (x) is non-differentiable. The complete algorithm is summarized in Algorithm
3
EXPERIMENTS
First, we design synthetic scene to illustrate the controllability and explainability of the proposed framework. The synthetic physical scene provides a simplified setting to unveil the essence of the knowledge-guidee generation. After that, evaluate the performance of SCG realistic traffic
5

---

Under review as a conference paper at ICLR 2022
Direct Search
WAEl
GVAE
T-VAE who SCG
T-AE wi SCG (Random miblization)
Initial Final
Initial
Final
Initial
Final
Initial
Final
Inttal
Final
Good intialization
Random inttaliizaion
Figure 3: Results of synthetic scene reconstruction experiment from 5 methods with random and good initialization. I shows the results of T-VAE using SCG. With the combination of knowledge @ewe can almost reach the optimal solution even from a random initialization, while baseline methods can realize the target only when start from the good initialization.
scenes represented by point clouds. Based on SCG, we propose a new adversarial attack method, Scene Attack, against multiple state-of-the-art segmentation methods.
3.1 SYNTHETIC SCENE RECONSTRUCTION
Task Description. We aim to reconstruct a scene to match a given image as shown in Figure 4 The task objective a reconstruction error Le(x)) = |S - R(x)]]2,, where R a differentiable image renderer (33] and S is the image of target scene. Under this succinct setting, it is possible to analyze and compare the contribution of explicit knowledge integration, since we can access the optimal solution. According to the understanding of the target scene, we define three knowledge rules: ! The scene has at most two plates; ② The boxes that belong to the same plate should have the same color; 3 The boxes belong to the same plate should have distance smaller than a threshold y. We compare our method with three baselines: Direct Search (DS), VAE, and Grammar- VAE (GVAE) (37. DS directly optimizes the positions and colors of boxes and plates in the data space. In contrast, VAE and GVAE search in the latent space and generate the scene from their decoders.
Experiment Settings. We synthesize the dataset by randomly generating 10,000 samples with varying number of boxes and plates. We also inject the target scene 10 times into the dataset to make sure it accessible to all models. Unlike GVAE and T-VAE, DS and VAE need to access the number of boxes and plates in the target im- age (e.g.. two plates and eight boxes) to fix the dimension of the input feature. To get the good initial points for DS, we add small perturbation to the positions and colors of all objects in the target scene. Similarly, for other methods, we add the perturbation the optimal latent code, which is obtained by passing the target scene to the encoder to get good initialization.
Table 1: Reconstruction Error
Initialization
Method
Random
Good
Direct Search
86.0+-9.4
1.9
VAE [34
110.4
13.416.1
GVAE (37
123.7=9.5
19.7#10.2
T-VAE
135.1
14.1#2.5
T-VAE w/ SCG
14.5++1.3
11.8#2.1
Evaluation Results. The generated samples from five models are displayed in Figure and the final errors are summarized in Table. [ With good initialization, all models find a similar scene to the target one, while with the random initialization, all models are trapped in local minimums. However, obtaining good initialization is not practical in most real-world applications, indicating that this task non-trivial and all models without knowledge cannot solve After integrating the knowledge into the T-VAE model, we obtain I of Figure 3 We can see that all three knowledge have positive guidance for the optimization, e.g., the boxes concentrate on the centers of plates with knowledge 2 When combining the three knowledge, even from a random initialization, our T-VAE can finally find the target scene, leading to a small error in Table. We also want to mention that it is also possible to apply simple knowledge to GVAE during the generation. However, the advantages of our method are that (1) we can integrate any constraints as long as they can be represented by whereas GVAE can only apply hard constraints to objects with co-occurrences.

---

Under review as a conference paper at ICLR 2022
(a)
(B)
Krowledge Loss-00
(c)
Stan
(d)
Krowledoe Less-g 9
plate
plate
Kriwledige Loee 3
Random Stort
Totel Krandedge loes
Random in
Krunbsder Loke
Lows
Tosk Loss
Random in AA Stor
Random in KC Ond
Teration
Figure 4: (a) Target scene. (b) Knowledge losses of integrating semantic rules @@separately. (c) The influence of knowledge to the trajectories with same initialization. (d) After applying explicit knowledge, optimization trajectories are diverse when start from different initialization.
A PointNetee
PolerSeg
D Cylndern
Higheray Stame Alucc
Opeeactto-Paaet afters
Intessction lon Pocc Attcck
Indersacionn tary Anackk
0.30
0.73 20
20
40
00
0C
20
-0
60
60
30
20
30
Itorations
onS
Nerations
Itcrabons
Figure 5: The IoU values during the attack process of four vietim models two backgrounds. Both Pose Attack and Scene Attack successfully attack all victims, and in some cases, Pose Attack outperforms Scene Attack in terms of the IoU value. However, scenes generated by Pose Allack not realistic and violate basic physical rules (shown in Figure
Analysis of Knowledge and Controllability. To amalyze the contribution of each knowledge, plot the knowledge losses of Diin Figure(4(b) together with the task loss. All knowledge losses decrease quickly at the beginning and guide the searching in the downstream task. Next, made ablation studies to explore the reason why knowledge helps the generation. In Figure bb) we compare the optimization trajectories of TAV (red (red-xyellow) and T-VAE w/ SCG (blue ggeenn) with the same initialization. For T-VAE, the generated samples are diverse but totally different from the target scene, while for T-VAE w/ SCG, the trajectories go to another direction and the generated samples are good. However, the interesting thing is that although the knowledge help us find good scenes, it does not reach the same point in the latent space with the trajectories from good initializaaton (cyan->purplec This can be explained by the entanglement of the latent space (43, which makes multiple variables control the same property. To further study this point, we plot Figure where we use 3 different initialization for T-VAE w/ SCG. The result shows that all three cases find the target scene but with totally different trajectories, which provides evidence to our conjecture. In summary, we believe the contribution of knowledge can be attribute to the entanglement of the latent space, which makes the searching easily escape the local minimum and find the nearest optimal points.
3.2 ADVERSARIAL TRAFFIC SCENES GENERATION
Task Description. We aim to generate realistic traffic scenes against segmentation algorithms as well as satisfying certain semantic knowledge rules. The adversarial scenes are defined as scenes that reduces the performance of downstream victim systems. To generated adversarial LiDAR scenes containing various fore-/background rather than the point cloud of a single 3D object as existing studies [38]] 63 a couple of challenges should be considered: First, LiDAR scenes with millions of points are hard to be directly operated; Second, generated scenes need to be realistic and follow traffic rules. Since there are no existing methods to compare with directly, we compare three methods: (1) Point Attack: point-wiss wise attack baseline [69] that adds small disturbance to points; (2) Pose Attack: a scene generation method developed by us that searches pose of vehicles in the scene; (3) Scene Attack: semantically controllable generative method based on our T-VAE and SCG. We explore
7

---

Under review as a conference paper at ICLR 2022
Table 2: Transferability of Adversarial Scenes (Point Attack IoU / Scene Attack loU). Scene Attack has lower IoU for all evaluation pairs, which demonstrates its better adversarial transferability.
Source Target
PointNet++
SqueezeSegV3
PolarSeg
Cylinder3D
PointNet++*
-/-
0.916 0.768
0.936 / 0.854
0.955 0.918
SqueezeSegV
0,954 0.606
-/-
0.932 0.855
0,956 0.892
PolarSeg
0,952 0.528
0.904 / 0.753
-
0.953 0.908
Cylinder3D
0,951 0.507
0.903 0.688
0.934 0.877
-/-
For fair comparison, we run 20,000 iterations to get the IoU for Point Attack.
the attack effectiveness against different models of these methods, well their transferability. For Pose Attack and Scene Attack, we implement an efficient LiDAR model R(x, B) 44 (refer to Appendix (rrendiiAAAorrr details) to convert the generated scene x a point cloud scene with an background B. The task objective min (z(z)= max (R((, B)) is defined by maximizing the loss function LP of segmentation algorithms P. We design three explicit knowledge rules: ! roads follow given layout (location, width, and length); 2 vehicles on the lane follow the direction of the lane; 3 vehicles should gather logether but keep a certain distance. I ensure generated vehicles follow the layout of the background B and mmakes the scene contain more vehicles.
PolarSeg (74]. Experiment Settings. We select segmentation algorithms (PointNet++ 57
SqueezeSegV V3 (7)). Cylinder3D (75) as victim models, all of which are pre-trained on Semantic Kitti dataset [ We collect two backgrounds B (Highway and Intersection) in Carla simulator [18. Since it is usually unable access the parameters of segmentation algorithms,, we focus the black- box attack in this task. Point Attack optimizes Cx(x) with SimBA (25], while Pose Attack and Scene Attack optimizes (x(x) with Bayesian Optimization (BO) (54). For the training of T-VAE,
build a dataset by extracting the pose information of vehicles together with road and lane information from Argoverse dataset (11..
Evaluation Results. We show the Intersection over Union (IoU) metric for the vehicle during the attack in Figure Gen- erally, it is harder to find adversarial scenes in the highway background than in the intersection background since the latter has much more vehicles. Within 100 iterations, Point Attack method nearly has no influence to the performance since operates in very high dimensions. In contrast, Pose Attack and Scene Attack efficiently reduce the IoU value. Although Pose Attack achieves comparable results to our method, scenes generated by it (shown in Figure 6 are unrealistic due to the overlaps between vehicles. In contrast, scenes generated by method only modify the vehicles within the traffic constraints. In Table [ explore the transferability of Point Altack and Scene Attack. Transferability, which means using generated samples from the Source model attack other Target models, is crucial for evaluating adversarial attack algorithms. Better transferability indicates that the samples carry important pat- terns that are ignored in most victim systems. Although Point Attack method dramatically reduces the performance of all four victims, the generated scenes have weak transferability and cannot decrease the performance of other victim models. However, scenes generated by Scene Attack successfully attack all models, even those not targeted during the training, which demonstrates strong adversarial transferability. More generated scenes can be found in Appendix E
Pose Attack
Scene Attack
Figure 6: Scene generated by Scene Attack follows basic traffic rules, while scene generated by Pose At- tack violates basic physical laws.
RELATED WORK
Incorporating Knowledge in Neural Networks. Integrating knowledge to data-driven models has been explored in various forms from training methods, meta-modeling, embedding to rules used for reasoning. (27] distills logical rules with a teacher-studens framework under Posterior Regularization 22 Another way of knowledge distillation is encoding knowledge into vectors then refining the
8

---

Under review as a conference paper at ICLR 2022
These methods need to access Knowledge Graphs [19] during the training, which heavily depends on human experts. Meta-modeling of complex fluid is integrated into the NN to improve the performance of purely data-driven networks in [45]. In addition, [72) restricted generative models' output to satisfy physical laws expressed by partial differential equations. In the reinforcement learning area, reward shaping [50] is recognized as one technique to incorporate heuristic knowledge to guide the training.
features from the model that in line with the encoded knowledge 124]
Structured Deep Generative Models. The original DGMs, such as GAN (33 and VAE [34 are mostly used for unstructured data. They leverage the powerful feature extraction of neural networks to achieve impressive results [3] [0]. However, the physical world is complex since objects have diverse and structured relationships. Domain specific structured generative models are developed
via tree structure (RvNN- VAF (40) or graph structure (Graph-VAr VAF (59 ) Rule-based generative
models are also explored by sampling from pre-defined rules (37
31 14 Typical applications of
this kind of model are molecular structure generation [26] 29 30, natural scene generation [3]
and automatic program generation [12]. Unlike these existing methods, our approach explicitly integrates knowledge during the generation process, One valuable application of DGMs is generating samples that meet the requirements of downstream tasks [20 65
searches in the latent space of StyleGAN 32] to obtain images that are similar to a given image. For structured data, such a searching framework transforms discrete space optimization continuous space optimization, which was shown to be more efficient [4]. However, it may not guarantee the rationality of generated structured data due to the loss of interpretability and constraints in the latent space [12]
Physical Scene Generation. Traditional ways of scene generation focus on sampling from pre- defined rules and grammars, such as probabilistic scene graphs used in (56) and heuristic rules applied in [18). These methods rely on domain expertise and cannot be easily extensible to large- scale scenes. Recently, data-driven generative models [14 64 [5] 40 36 are proposed to learn the distribution of objects and decouple the generation of scene into sequence [64] and graph [51, 40 of objects. Although they reduce the gap between simulation and reality, generated scenes cannot satisfy specific constraints. Another substantial body of literature [2 35 24 explored learning scene graphs from images or gencrating scene images directly via an end-to-end framework. Their generalization to high-ddmenniooal data is very challenging, making them less effective than modularized methods proposed by 336 68 14).
Semantically Adversarial Attacks. Early adversarial attack methods focused on the pixel-wise attack in the image field, where Lp-norm is used to constrain the adversarial perturbation. For the sake of the interpretability of the adversarial samples, recent studies begin to consider semantic attacks. They attack the rendering process of images by modifying the light condition
42
73 or
manipulating the position and shape of objects [4] 770 28 This paper explores the generation of
adversarial point cloud scenes, which already has similar prior works [66] 3 62). [66] 3 modify the
environment by adding objects on the top of existing vehicles make them disappear. [+ create a
ghost vehicle by adding an ignorable number of points. However, they modify single object without considering the structural relationship of the whole scene.
CONCLUSION
In this paper, we explore semantically controllable generation tasks with explicit knowledge integra- tion. Inspired by the categorization of knowledge for the scene description, we design tree-structured generative model to represent structured data. We show that the two types of knowledge can be explicitly injected into the tree structure to guide and restrict the generation process efficiently and effectively. After considering explicit semantic knowledge, we verify that the generated data contain dramatically fewer semantic constraint violations. Mcanwhile, the generated data still maintain the diversity property and follow its original underlying distribution. Although we focus on the scene generation application, the SCG framework can be extended to other structured data generation tasks, such as chemical molecules and programming languages, showing the hierarchical properties. One assumption of this work is that the knowledge is helpful or least harmless they are summarized and provided by domain experts, which needs examination in the future.
9

---

Under review as a conference paper at ICLR 2022
Ethics Statement
Our proposed method aims to enhance existing data-driven generative models with explicit knowledge summarized by human experts. In this paper, focus on the application of physical scene generation. Considering the broad usage of this method in human society, we identify three potentially harmful insights:
1) The explicit knowledge we integrate is summarized by human experts, therefore, could be biased. With such kind of knowledge, the generated data could also biased and even results in discrimination.
2) The knowledge we use might potentially contain private information. After integrating it into the generative models, there a risk of user information disclosure.
3) The knowledge-enhancee generation model is powerful to generate realistic adversarial amples. These examples might be used to attack autonomous systems in the real world, e.g. self-driving algorithms, and cause damages.
To mitigate the potential negative ethics impacts mentioned above, we encourage research to follow these instructions:
Construct a strict mechanism of knowledge screening and monitor the generated samples against bias and discrimination.
2) We can also propose a knowledge screening mechanism or an intelligent discriminator to reduce the private information content.
3) Since our generated scenes realistic and follow traffic rules, it is reasonable to make the defense models drive safely in our generated scenes. However, we should restrict the usage of generated scenes only in evaluation and testing stages before deployment help self-driving algorithms overcome their shortcomings.
Reproducibility Statement
All of our experiments are conducted on a desktop with Intel 19-9900K CPU @ 3.60GHz, NVIDIA GeForce GTX 1080Ti, and 64GB memory. The settings the experiment environment are described in the code folder in the supplementary. To make the experiment results reproductable, we provide all pre-trained models (please follow the instruction in code/README, md to download). The parameters of the two experiments are shown in Table 3 and Table 4 in the Appendix.
The dataset we used in Synthetic Scene Reconstruction Experiment synthesized by ourselves, which can be downloaded from the link provided in the README.md file. This dataset is published under MIT License. The dataset we used LiDAR Scene Generation Experiment is based on the Argoverse dataset, which is published by Argo Al with MIT License. We the entire tracking part of the dataset for training and the processed dataset can be downloaded from the link provided in the README file.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432 4441, 2019.
[2] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan+4: How edit the embedded images? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8296-8305, 2020.
[3] Mazen Abdelfattah, Kaiwen Yuan, Z. Jane Wang, and Rabab Ward. Towards universal physical attacks cascaded camera-lidar 3d object detection models. arXiv preprint (XXi::2011110477 2021.
[4] Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4845 4854, 2019.
[5] Saeed Amizadeh, Hamid Palangi, Alex Polozov, Yichen Huang, and Kazuhito Koishida. Neuro- symbolic visual reasoning: Disentangling" visual" from" reasoning". In ICML, pp. 279-290, 2020.
[6] Charles Audet and Warren Hare. Derivative-free and blackbox optimization, 2017.
[7] David Bau, Jun- Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National 2020.
Academy of Sciences, 17(48):30071-3007
[8] Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall. Se- manticKIlTI: Dataset for Semantic Scene Understanding of LiDAR Sequences. In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV), 2019.
[9] David M Blci, Alp Kucukelbir, and Jon D McAuliffe, Variational inference: A review for 2017.
statisticians. Journal the American statistical Association,
[10] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: /809. /096, 2018
[11] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagject Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1148--777, 2019.
[12] Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Le Song. Syntax-directed variational autoencoder for structured data. arXiv preprint arXiv:1802.088: 2018.
[13] Fei Deng, Zhuo Zhui, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In International Conference on Learning Representations, 2021.
[14] Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. Meta-sim2: Unsupervised learning of scene structure for synthetic data generation. In European Conference on Computer Vision, pp. 715-733. Springer, 2020.
[15] Zoltan Dienes and Josef Perner. A theory of implicit and explicit knowledge. Behavioral and brain sciences, 22(5):735-808 1999.
[16] Wenhao Ding, Baiming Chen, Bo Li, Kim Ji Eun, and Ding Zhao. Multimodal safety-critical scenarios generation for decision socision-making algorithms evaluation. IEEE Robotics and Automation Letters, 1551:155 2021.
[17] Laurent Dinh, Jascha Sohl-Dickster Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 16Xiv:1655.00 08803, 2016.
11

---

Under review as a conference paper at ICLR 2022
[18] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun, Carla: An open urban driving simulator. Conference on robot learning, pp. 1-16. PMLR, 2017.
[19] Lisa Ehrlinger and Wolfram WöB. Towards definition of knowledge graphs. SEMANTiCS (Posters, Demos, SuCCESS), 48:1-4, 2016.
[20] Jesse Engel, Matthew Hoffman, and Adam Roberts. Latent constraints: Learning to generate conditionally from unconditional generative models. arXiv preprint arXiv:1711.05772, 2017.
[21] SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. arXiv preprint arXiv: 16X.v::/63.08880 2016.
[22] Kuzman Ganchev, Joao Graça, Jennifer Gillenwater, and Ben Taskar. Posterior regularization for structured latent variable models. The Journal of Machine Learning Research, 11 11:2001-2049, 2010.
[23] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 2014.
[24] Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. Scene graph generation with external knowledge and image reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1969-1978, 2019.
[25] Chuan Guo, Jacob Gardner, Yurong You. Andrew Gordon Wilson, and Kilian Weinberger. Simple black-box adversarial attacks. In International Conference on Machine Learning, pp. 2484-2493. PMLR, 2019.
[26] Xiaojic Guo and Liang Zhao. A systematic survey on deep generative models for graph generation. arXiv preprint arXiv:2007 06686, 2020.
[27] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logic rules. arXiv preprint arXiv:/603. 1603.06318, 2016.
[28] Lakshya Jain, Varun Chandrasekaran, Uyeong Jang, Wilson Wu, Andrew Lee, Andy Yan, Steven Chen, Somesh Jha, and Sanjit A Seshia. Analyzing and improving neural networks by generating semantic counterexamples through differentiable rendering. arXiv preprint arXiv: 1910.00727, 2019.
[29] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. International Conference on Machine Learning, pp. 2323-2332. PMLR, 2018
[30] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In International Conference Machine Learning, pp. 4839- 4848. PMLR, 2020.
[31] Amlan Kar, Aayush Prakash, Ming- Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-sim: Learning to generate synthetic datasets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4551-4566 2019.
[32] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410 2019.
[33] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEFE conference computer vision and pattern recognition, pp. 3907-3916, 2018.
[34] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 131266 14, 2013.
12

---

Under review as a conference paper at ICLR 2022
[35] Adam R Kosiorek, Hyunjik Kim, Ingmar Posner, and Yee Whye Teh. Sequential attend, infer, repeat: Generative modelling of moving objects. arXiv preprint 1eX:v::806... 01794, 2018.
[36] Abhijit Kundu, Yin Li, and James M Rehg. 3d-renn: Instance-level 3d object reconstruction via render-and-compare. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3559-3568, 2018.
[37] Matt Kusner, Brooks Paige, and José Miguel Hernandez-Lobato. Grammar variational autoencoder. In International Conference on Machine Learning, pp. 1945 1954. PMLR, 2017.
[38] Itai Lang, Uriel Kotlicki, and Shai Avidan. Geometric adversarial attacks and defenses on 3d point clouds. arXiv preprint arXiv:2012.05653) 2020.
[39] Der- Lee and Bruce Schachter. Two algorithms for constructing a delaunay triangulation. International Journal of Computer & Information Sciences, 9(3):219-42,, 1980.
[40] Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao Zhang. Grains: Generative recursive autoencoders for indoor scenes. ACM Transactions on Graphics (TOG), 38(2):1-16, 2019.
[41] Renjie Liao, Yujia Li, Yang Song, Shenlong Wang. Charlie Nash, William L. Hamilton, David Duvenaud, Raquel Urtasun, and Richard S Zemel. Efficient graph generation with graph recurrent attention networks. arXiv preprini arXiv: 1XX0:009100006// 2019.
[42] Hsueh-Ti Derek Liu, Michael Tao, Chun-Liang Li, Derek Nowrouzezahrai, and Alec Jacobson. Beyond pixel norm-balls: Parametric adversaries using an analytically differentiable renderer. arXiv preprint arXiv: 10XX:v1808.000 080..0258, 2018.
[43] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114- 4124. PMLR, 2019.
[44] Rengian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimiza- 2018.
tion. arXiv preprint
[45] Mohammadamin Nahmoudaaaudbozchelou,, Marco Caggioni, Setarch Shahsavari, William H Hartt, Em Karniadakis, Safa Jamali, et al. Data-driven physics-informed constitutive metamod- eling of complex fluids: A multifidelity neural network (mfnn) framework. Journal of Rheology, 65(2), 2021.
[46] George L Malcolm. Iris IA Groen, and Chris Baker. Making sense of real-world scenes. Trends in Cognitive Sciences, 20(1 1):843)8586 2016.
[47] Sivabalan Manivasagam, Shenlong Wang. Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang, Wei-Chiu Ma, and Raquel Urtasun. Lidarsim: Realistic lidar simulation by leveraging the real world. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 11 117-1111 2020.
[48] Kaichun Mo. He Wang, Xinchen Yan, and Leonidas Guibas. P(2pe: Learning to generate 3d point cloud shapes from part tree conditions. In European Conference Computer Vision, pp. 683-701. Springer, 2020.
[49] Tomas Möller and Ben Trumbore. Fast, minimum storage ray-triangle intersection. Journal of graphics tools, 2(1):21-28, 1997.
[50] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. In Icml, volume 99, pp. 278 787,, 1999.
[51] Wamiq Para, Paul Guerrero, Tom Kelly, Leonidas Guibas, and Peter Wonka. Generative layout modeling using constraint graphs. arXiv preprint 1rX1v 2020.
[52] Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in optimization, 1(3):127-239, 2014.
13

---

Under review as a conference paper at ICLR 2022
[53] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, rere-performacce deep learning library. arXiv preprint arXiv:1912.01703. 2019.
[54] Martin Pelikan, David E Goldberg, Erick Cantú-Paz, et al. Boa: The bayesian optimization algorithm. In Proceedings of the genetic and evolutionary computation conference GECCO-99, volume 1, pp.
Citeseer, 1999.
[55]
Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors of variations. arXiv preprint arXiv:2001.10233 2020.
[56] Aayush Prakash, Shaad Boochoon, Mark Brophy, David Acuna, Eric Cameracci, Gavriel State, Omer Shapira, and Stan Birchfield. Structured domain randomization: Bridging the reality gap by context-aware synthetic data. 2019 International Conference on Robotics and Automation (ICRA), pp. 7249-7255. IEEE, 2019.
[57] Charles R Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv: 1eXxi::170.... 777700000233 2017.
[58] Jayaram Sethuraman. constructive definition of dirichlet priors. Statistica sinica, pp. 639-650, 1994.
[59] Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference on Artificial Neural Networks, pp. 412-422. Springer, 2018.
[60] Raymond M Smullyan. First-order logic. Courier Corporation, 1995.
[61] Richard Socher, Cliff Chiung- Yu Lin, Andrew Y Ng, and Christopher D Manning. Parsing natural scenes and natural language with recursive neural networks. In ICML, 2011.
[62] Jiachen Sun, Yulong Cao, Qi Alfred Chen, and Z Morley Mao. Towards robust lidar-based perception in autonomous driving: General black-box adversarial sensor attack and countermea- sures. In 29th USENIX Security Symposium (USENIX Security 20), pp. 877-894, 2020.
[63] Jiachen Sun, Karl Koenig, Yulong Cao, Qi Alfred Chen, and Z Morley Mao. On the adversarial robustness of 3d point cloud classification. arXiv preprint arXiv:201 2020.
[64] Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. arXiv preprint arXiv:2101.00 065511, 2021.
[65] Austin Tripp, Erik Daxberger, and José Miguel Hernández-Lobaad Sample-efficient optimiza- tion in the latent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems, 33, 2020.
[66] James Tu, Mengye Ren, Sivabalan Manivasagam, Ming Liang, Bin Yang, Richard Du Frank Cheng, and Raquel Urtasun. Physically realizable adversarial examples for lidar object detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13716 11716-13725, 2020.
[67] Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280) 1989.
[68] Jajjun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 699-707, 2017.
[69] Chong Xiang, Charles R Qi, and Bo Li. Generating 3d adversarial point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9136-9144, 2019.
[70] Chaowei Xiao, Dawei Yang, Li, Jia Deng, and Mingyan Liu. Meshadv: Adversarial meshes for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6898 6907, 2019.
14

---

Under review as a conference paper at ICLR 2022
[71] Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In European Conference Computer Vision. pp. 1-19. Springer, 2020.
[72] Yibo Yang and Paris Perdikaris. Physics-informed deep generative models. arXiv preprint
11Xiv:1812.035111 2018.
[73] Xiaohui Zeng, Chenxi Liu, Yu-Siang Wang, Weichao Qiu, Lingxi Xie, Yu-Wing Tai, Chi-Keung Tang, and Alan L Yuille. Adversarial attacks beyond the image space. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4302-4311, 2019.
[74] Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Hassan Foroosh. Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9601-9610, 2020.
[75] Hui Zhou, Xinge Zhu, Xiao Song, Yuexin Ma, Zhe Wang, Hongsheng Li, and Dahua Lin. Cylinder3d: An effective 3d framework for driving- scene lidar semantic segmentation. arXiv preprint arXiv: 01550, 2020.
15

---

Under review as a conference paper at ICLR 2022
The appendices organized as follows:
In Appendix A, we provide the details of our LiDAR model implementation used in the traffic scene generation experiment.
In Appendix B, provide the details of the structure of our proposed T-VAE model, including definitions of encoder-decoder pairs and generation example.
In Appendix C, we show the details of the definition of knowledge in both experiments.
In Appendix D, we describe the three baselines used in the Synthetic Scene Reconstruction experiment.
In Appendix we provide more experiment results.
In Appendix F, we describe the four point cloud segmentation victim models used in the LiDAR
Scene Generation experiment.
A LIDAR MODEL IMPLEMENTATION
The LiDAR model implemented by Moller-Trumbord algorithm 49 with the PyTorch [3] package for high-efficiency computation. We assume there is plane A V. V2 in the 3D space constructed by points Vo. Vi. V2. A ray R(I) with origin and normalized direction D (we make || = for simplification) is represented point D the ray. If D the intersection between the ray and the plane AV V2, we can represent the ray with barycenteric coordinate:
R(t) O tD, where is the distance between O and the end
T(u, v) (1 - u - Vo + uVi + 0V22 = O tD = R(t)
(11)
where u and v weights. To simplify the equations, define three new notations:
Ej = V/1 Vu E2 = 1/2 Vu
(12)
T =0 Vi
Then, we can solve the distance in [11 by:
t
T, E1, Eg
(T B1) E2 (D En (T E1)
u v
D,T, Ez
(13)
D, Ez
D, E,ET
(D x E2) Ei
Since we know the ray direction D, we can get the 3D coordinate the intersection with this distance During the implementation, we reuse D Eg and T x E₁ speed up the computation. To make sure the intersecting point T'u, v) is inside the triangle A V/ V2, we need to have:
u, U. (1
v) E [o,
(14)
these three conditions are not fulfilled, the intersection point will be removed.
To calculate the point cloud generated by LiDAR, first convert vehicle mesh models triangles FF with Delaunay triangulation [39]. Then, we create the array of LiDAR rays with width W and height H for 360 view. Finally, F and LiDAR rays Ra, in parallel to get the final range map HW The background pointcloud
(13) to calculate the intersection point between all triangles
will also be converted to range map BH XW by:
0 arctan
x y
T
(15)
Φ = arctan
22
where (I,, y. ) the coordinate one point in pointcloud, is used to calculate the index of row, and Φ is used calculate the index of column. Then mix xx and BHx by taking the minimal value for each element: Ma min Bigh Vi W, jEH
(16)
16

---

Under review as a conference paper at ICLR 2022
0 123"
Take the minimal value
120 240"
Vehicle Mesh
LIDAR Raycast
Foreground Range Map
Mixed Range Map (with Label)
Project
Mixed
240"
Project
390"
Background Range Map
Background Pointcloud
Pointcloud (with Label)
Figure 7: The pipeline of LiDAR scene generation with our developed model.
where Mr, represent the (entry of the mixed range map scene M. Then, we convert the range map to the final output pointcloud scene S with:
t X sin 0
cos cos
(17)
t x cos 0 sin Ø
The entire pipeline of the above process is summarized in Figure 7 The parameters we used for LiDAR follow the configuration of the Semantic Kitti dataset, where the channel H 64, the horizontal resolution W 2048, the upper angle is 2 and the lower angle is -25 The gap between reality and simulation can reduced by realistic simulation and sensor models 47 but this will not be explored in this paper.
B DETAILED T- VAE MODEL STRUCTURE
Our T-VAE model consists of several encoders and decoders that related to the definition of the scene. In this paper, we explored two experiments with two scenes: a synthetic box placement image scene and traffic point cloud scene. In Figure we show the details of the modules for two scenes. We also use an synthetic scene as an example to show the encoding and decoding processes Figure 9 where original data point stored in tree structure and the generated data is also a tree structure. The encoding process converts the tree into a stack and then encodes the information using the node type defined in the tree. The decoding process expands the tree with the predict node type from the Classifier.
For the synthetic box placement image scene, there are 5 Encoder-Decodee pairs, Sampler, and Classifier. W node determines the global information such as location, and orientation of the entire scene. P node spawns plate object the scene with positions and colors determined by the property vector. Both Q and B nodes spawn a box object in the scene with positions and colors determined by the property vector. E node serves as a stop signal to end the expansion of a branch, therefore, will not spawn anything in the scene and does not have model parameters.
The traffic point cloud scene has similar definitions. node contains the information of the road and only has two children. L node determines the lane information such as the width and direction. Both Q and V nodes spawn a vehicle in the scene with positions and orientations determined by the property vector.
17

---

Under review as a conference paper at ICLR 2022
(Lazal) flxx
Box Scene Example
Child
clibd
Chaid Vestor
Crac Vedlor
Pagtnt
Vecte
veder
Proveety
Property
veder
Q Enocde
Node Type
Parent
Paweed Vector
Parent Viddar
Poreed Mocia
Panant Vodla
Parend Vocto
Deander
Deccoder
Semshe
Child Vector
Child
Child
Chic
(Locall
Vecter
Proserty
(Local) Property
Child
Chisd
Chiid
Picperty
Vector
Vestor
Parcnt Vocter
Parent Viddor
Plutec
Box
Quad
Empty
World
Sampler
Cáche Property
Vestor
Vector Vestor
(lace)
Piula
I'opertt
(Luee)
Traffic
Child Vector
Child wederr
Vehride Property
Vestor
vector
Enocda
V. Enender
Eincoder
Closster
Paaeet/vedcr Vedder
Parent Vector
Parent Vector
Parent
Porant Victor
Porent Viedor
Forent Vedic
Teecoter
Docoder
Chil
Child
(uccal: Priserty
(Looa) Proserty
Zero Vesiaa
(Looal Lane Property
Chic Vegtor
Chisd Vedior 2
Globol Property
Chisd Vedker
GNidd Veeter
Child Vestor 3 Vector
CNi
R Encoder
Panert Viedor
Porert Vietor
Plewrt Vecter
It Decoder
Decoder
Quad
Emply
Road
Sampler
Uhii
Crad
Chiil
Chile
Chid
(Loce)
Vidie
Vodier
Projerty
Vedor
Vector 2 Vector 3 Vectord
Figure 8: The definition of each module in our proposed T-VAE. A: There are 5 kinds of nodes in Synthetic Scene Reconstruction experiment. Therefore, we have encoders and decoders in total, plus Classifier and Sampler. B: There are 5 kinds of nodes in the LiDAR scene experiment. Therefore, we have encoders and 5 decoders in total, plus a Classifier and a Sampler.
18

---

Under review as a conference paper at ICLR 2022
R
Encoder
Decoderr
Ganenad mag
Figure 9: An example of the encoding and decoding processes our T-VAE model. The circles have the same meaning in Figure g
Table 3: papee--arramttrs of the Synthetic Scene Reconstruction Experiment
Parameters
Value
Description
Ir
0.001
Learning rate T-VAE training
E
1000
Maximum training epoch
B
128
Batch size during training
0.1
Learning rate in stage 2.
T
100
Maximum searching iteration
d=
64
Dimension of latent code
dj
128
Dimension of feature vector f
dg
Dimension of property vector g The threshold used in knowledge Normalization factor of location
Ni
Table 4: Hyper-parameters of the LiDAR Scene Generation Experiment
Parameters
Value
Description
Ir 0.001
Learning rate of T-VAE training
e 0.01
Max value for point-wise disturbance
E
1000
Maximum training epoch
B
128
Batch size during training
T
100
Maximum searching iteration
d2
32
dimension of latent code
d
64
dimension of feature vector
d
dimension of property vector g Normalization factor of location
Ni
40
(w, h)
(1.5, 3)
The thresholds used in knowledge ?
19

---

Under review as a conference paper at ICLR 2022
C KNOWLEDGE DEFINITION
For each experiment in this paper, we design three knowledge rules. We explain the details of the implementation of these rules.
In the Synthetic Scene Reconstruction Experiment, we calculate Ly (a, (x()) with the following implementations:
The scene has at most two plates, which can be implemented by W node has at most two P children nodes. We traverse the entire generated tree æ to find W node, then we collect the children nodes of W and count the number of P node. If the number is larger than 2, we calculate the cross-entropy loss between the node type and E node label.
@ The colors of the boxes that belong to the same plate should be the same, which can be implemented by The color of P node's 's children nodes should be the same. We traverse the entire generated tree find all P nodes, then we collect the colors of the children of P. 'he average color č is calculated for each P node and č is used as label to calculate the MSE for all children nodes of the corresponding P' node.
@ The distance between the boxes that belong the same plate should be smaller than a threshold y which can be implemented by The distance between P node's children nodes should be smaller than threshold y. We traverse the entire generated tree find P node, then we collect the absolute position of all its children nodes and calculate the MSE between this position and the position of P.
In the LiDAR Scene Generation Experiment, we calculate Cy Y((x)) with the following imple- mentations:
@ Roads follow given layout (location, width, and length), which can implemented by R node follows a given layout (location, width, and length). We traverse the entire generated tree to find R node, then calculate the MSE between the property vector of R and the given layout.
Vehicles on the lane follow the direction of the lane, which can implemented by L node follows pre-defined directions. We traverse the entire generated tree find L node, then calculate the MSE between the property vector of It and the given layout.
Vehicles should gather logether but keep a certain distance, which can implemented by Q node has at least two Q nodes its children until the absolute width and height of current block is smaller than thresholds w and h. We traverse the entire generated tree x to find Q node, then collect the type of its children nodes. When the collected Q node type is less than 2, calculate the cross-entropy loss between two collected node type and Q node label. When the width and height of current block is smaller than and h, we stop applying this rule.
After calculating all errors in Ly (xx, (z)), we can directly use back-prorogation to calculate the gradient of latent code = and update with gradient descent method.
D BASELINES IN SYNTHETIC SCENE RECONSTRUCTION
D.1 DIRECT SEARCH
The dimension of the physical space is x (2 12+8) = 60, where 6 is the property dimension including position, orientation, and colors, 2 is the number of plates, and 8 is the number of boxes. We use gradient descent with the learning rate to directly search in the physical space. Since there is no constraint to avoid overlaps between boxes, the generated scenes could be unrealistic.
D.2 VARIATIONAL AUTO-ENCODER (VAE)
The input dimension of the encoder is the same the searching space of Direct Search. The VAE model has a encoder and a decoder, both of which has a fixed number model parameters. There 4 hidden layers in the encoder and each layer has 128 neurons. The decoder also has 4 hidden layers with 128 neurons. For the output color, we add a Sigmoid(() function to normalize it to the range [0, 1]. The location is normalized by N, before it is taken into the encoder.
20

---

Under review as a conference paper at ICLR 2022
sample
Sample 2
Sample
Sample 4
Sample
Sample
Sample /
Sample 8
Sample
Sample 10
VAE
GVAE
HT-VAE
Figure 10: We randomly generate 10 samples by sampling in the latent space of VAE, GVAE, and T-VAE.
D.3 GRAMMAR VAE (GVAE)
GVAE (37) requires the input data be described with set of pre-defined grammars, According to the task of synthetic scene reconstruction experiment, we design 9 rules,
W + P, W B, P PIE, P PIB, P B, P - F, - B, B B-BBAR E, (18) and they represented in one-hot vector in the dataset. The original GVAE designed only for rule-based discrete data generation (e.g. molecules), thus we modify the structure to add continuous attribute representation. The encoder consists of 3 I-dimensional Convolution layers with kernel size 3 x 3. The numbers of channels for the Convolution layers are [32, 64, 128]. The decoder is an LSTM model with 128 neurons, therefore, the decoding process sequential. During the training stage, the maximal length of rules is fixed to 20 with E - E as padding, and the decoder will output 20 rules. The cross-entropy loss is used between the input rules and decoded rules. During the generation stage, the rules generated from the decoder will be firstly stored in stack and convert to the tree with the first-in-last-out (FILO) principle.
E ADDITIONAL QUALITATIVE RESULTS
In Figure 10 show samples randomly generated from VAE, GVAE, and T-VAE. The results show that all three models are able to generate diverse samples. Specifically, samples generated by VAE always have 2 plates and boxes due the fixed input data dimension. In contrast, samples from GVAE and T- T-VAE have variable numbers of plates and boxes.
show 4 more generated scenarios from the Point Attack methods with prediction results from 4 segmentation models. For better visualization, we also show three detailed figures. In Figure 12 and Figure [3] we show more results from Pose Atlack and Scene Attack methods. Scenes generated by Scene Atlack follow basic traffic rules.
In Figure 11
F SEGMENTATION MODELS IN LIDAR SCENE GENERATION
F.1 POINTNET++
This model [57 directly uses point-wise features in the 3D space as the backbone to deal with the segmentation problem. Although this model does not have impressive results on the Semantic KITTI dataset, select it because it influences a lot of existing point cloud processing models. We use the code from this repository and train the model on Semantic KITTI dataset by ourselves following the original training and testing split setting.
F.2 POLARSEG
This model 77] converts the data representation from 3D Cartesian coordinate to Polar coordinate and extracts features with 2D convolution layers. We the code from this repository and the
21

---

Under review as a conference paper at ICLR 2022
Groundrruth
Pointee2
Prediction
Grounctruth
Polarseg
Prediction
Groundiruth
SqueezaSegVa
ctinn
Groundtruth
Cyllnder3D
Prediction
Figure 11: More results for Point Attack method with the Intersection background.
22

---

Under review as a conference paper at ICLR 2022
Grounctruth
PointNel2
Prediction
Groundiruth
Polarseg
Prediction
Groundrruth
SqueezeSegV3
Prediciton
Groundruuhh
Cylinder3D
Prediction
Figure 12: More results for Pose Attack method with the Intersection background.
23

---

Under review as a conference paper at ICLR 2022
PointNot2
Prediction
Groundrruth
Polarsog
Prediction
Groundtruth
SqueczoSegVa
Prediction
Groundtruth
CylinderDD
Prediction
Figure 13: More results for Scene Attack method with the Intersection background.
24

---

Under review as a conference paper at ICLR 2022
pre-trained model provided by the authors. Since only consider the vehicle class, change all other labels to non-vehicle class.
F.3 SQUEEZESEGV3
This model 71] projects the 3D point cloud 2D range maps and extract features with 2D convolution layers from the range maps. We use the code from this repository and use the pre-trained model provided by the authors. Since we only consider the vehicle class, we change all other labels to non-vehicle class.
F.4 CYLINDER3D
This model 75] converts the data representation from 3D Cartesian coordinate to the Polar coordinate and divides the space into blocks with cylinder representation. We use the code from this repository and use the pre-trained model provided by the authors. Since we only consider the vehicle class, we change all other labels non-vehicle class.
25


Reviews:
[{"review_id": "_c4sa-pW0O0", "reviewer": "Reviewer_EqAi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Strengths \n* Training a model to generate scene layouts while enforcing explicit constraints during training is novel. However, training a neural network while enforcing explicit constraints on the output has been studied before [Ref 1]. \n\n## Weaknesses\n* \"GVAE leverages knowledge by integrating rules during the training stage; however, it cannot explicitly integrate semantic knowledge during the generation.\" -- Disagree, rules are explicit knowledge about the scenes and can certainly be incorporated in the similar fashion as  [Ref 2] as long as those are simple co-occurrence based. However, the advantages of the current model over the sg-vae seem to be the followings: (1) the current method could model any constraints of the objects (possible higher order constraints)  whereas sg-vae could model only co-occurrences of different objects, and (2) the optimization-based formulation of the current method enforces a soft-constraints where as the grammar-based structure of sg-vae enforces hard-constraints on the object co-occurrences. \n\n* \"When combining the three knowledge, even from a random initialization, our T-VAE can finally find the target scene, leading to a small error in Table. 1.\" --- The external knowledge is adding explicit constraints in the scene generation. Those constraints reduce the search space and that leads to a better minimum? That being said, the main point here to discuss is how one could incorporate the external knowledge for the current scene generation task. This work does so by fine-tuning a t-vae while enforcing the constraints on the scene layout. An interesting evaluation however would be the comparison w.r.t. The following baselines (1) T-vae + imposing the constraint during test time, (2) current model (trained with explicit constraints, ie external knowledge) without explicit constraints during testing time. Since the constraints were imposed during the training time of the current method, the network might already know how to impose the constraints.  Also, the external knowledge constraints were not employed in the baselines Direct Search (DS) and VAE. \n\n* The scene attack in the current method is not convincing. To me the adversarial attack is something where we modify the image / scene by a small amount (invisible to the naked eye) to deceive an existing trained model. Now, if a trained model performs poorly on a synthesized scene then the scene could be bad. It would be more convincing if the attack changes only a tiny amount of the 3d point cloud and still could  fool a trained model. \n\n* The synthetic scene layouts utilized in this work are rather simple. An experiment on the suncg datasets comparing against a stronger baseline [Ref. 3] would be good.  \n\n\n## References \n* [Ref. 1] Gould, Stephen, Richard Hartley, and Dylan Campbell. \"Deep declarative networks: A new hope.\" arXiv preprint arXiv:1909.04866 (2019). \n* [Ref. 2] Purkait, Pulak, Christopher Zach, and Ian Reid. \"SG-VAE: Scene Grammar Variational Autoencoder to generate new indoor scenes.\" European Conference on Computer Vision. Springer, Cham, 2020.  \n* [Ref 3] Li, Manyi, et al. \"Grains: Generative recursive autoencoders for indoor scenes.\" ACM Transactions on Graphics (TOG) 38.2 (2019): 1-16. ", "summary_of_the_paper": "The current method proposes a scene generation technique that models explicit knowledge by imposing semantic rules on the objects in the scene. It generates meaningful scene layouts and shown to be perform better than some simple baselines. ", "summary_of_the_review": "### Summary\n* A novel formulation and a sound approach to generate novel scenes \n* Weak evaluation -- very simple synthetic dataset, unconvincing attack \n* Weak baselines -- no comparison against s.o.t.a scene layout generation methods ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Qcpwph6Ucpm", "reviewer": "Reviewer_gJAm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "---\nDisclaimer: I am unfamiliar with this area and can only give a very high-level review.\n---\n\nThe ability to inject explicit knowledge into generative models would be very helpful for many tasks. Many generative models struggle and ignore even simple physical rules and a way to explicitly incorporate them would likely lead to better results and more data-efficient models. The authors propse to represent scenes in a tree-based manner where nodes correspond to objects and edge to relationships between objects. A VAE model is then trained to learn a structured representation of the data. After the VAE is trained the tree's characeristics can be adapted with explicit knowledge to ensure that generated scenes follow these rules.\n\nMy first question in this regard is why you focus on tree-based instead of graph-based representations? It feels to me that graph-based representations would offer more flexibility. Additionally, there is a large amount of work on graph neural networks that you could build upon. Additionally, there is a large amount of literature for graph-to-image which you could compare against.\n\nI also don't follow the motivation for evaluating your approach on generating adversarial traffic scenes. I feel there are many other applications to evaluate the general generation capabilities. Generating adversarial attacks may not be the first thing that comes to mind for incorporating knowledge into generative models. Besides just modeling \"general\" scenes this could be applied to biological or chemical settigns were we have clear pre-defined rules.\n\nAs a follow-up ont this: Where do you get the knowledge from? Does the knowledge need to be hand-designed? Can the system easily scale to large rule-bases? Or is it possible to learn the rules directly from the data such that they only need to be \"verified\" or adjusted afterwards?\n\nFinally, could this approach be used for editing, e.g. relationships between objects? Given that we have the tree representation, could we obtain the tree representation of a given scene, edit it, and then render the new, edited scene?", "summary_of_the_paper": "The paper proposes a way to incorporate explicitly defined (e.g. rule based) knowledge into generative models for scene generation. Through the addition of explicit knowledge the approach can ensure that generated scenes follow specific requirements, e.g. physical rules. The approach is evaluated on a synthetic dataset and as a way to generate adversarial examples for scene based segmentation models.", "summary_of_the_review": "I am not experienced in this domain but it feels to me that this should be more closely investigated from the view point of graph-based neural networks and graph-based image representations and rendering. The tree-based representation feels very restrictive to me. Also, I'm not sure if generating adversarial traffic scenes is the best way to evaluate this approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PNPMaier24p", "reviewer": "Reviewer_GPnn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is quite far from my expertise so it is hard to evaluate the\noriginality. I have not seen this approach previously, and the literature\nreview supports the claimed originality.\n\nThe proposed approach does generate examples of the target scene from\nrandom initializations, unlike most of the compared approaches that did\nnot have the constraints embedded. With the LIDAR scenes, the proposed approach\nsuccessfully generated more adversarial examples which also conformed\nbetter to road traffic constraints. The adversarial examples generated\nwrt one system were generally also usable as adversaries for another\nsystem (why? what made these so effective?).\n\nThere are 2 weaknesses:\n1) The paper is quite hard to follow. There is a lot of precise notational\ndescription which could use some 'intuitive' explanation to help readers\nmake sense of the theory. In the case of the experiments, there is a lot of\nprecise detail that is missing that leads one to wonder if the claims are\na bit exaggerated. There is more detail in the appendices, and it would\nhelp the paper to at least reference the relevant sections in the appendices.\n2) There are a lot of worries about the experiments: a) why does the synthetic\ntarget configuration have the given precise box and plate configuration,\nsince these are not in the constraint specification? b) The LIDAR pose attack\n'cars' are at what looks like random orientations. Since the generator is\ncreated by the authors, it is unclear why the cars could not be aligned with\nthe lines. c) It looks like a lot of hand-crafting is needed for constructing\nthe knowledge / constraint tree, and it looks like one would need a new\ntree for each application. d) In the 8 cases of the LIDAR pose and scene\nadversary generation, the pose approach was better in 5 of the cases, and\nin the 3 cases where the proposed scene approach was better, the adversaries\nwere not that successful (assuming that IoU was a good measure of\nadversariality).\n\nThere are a number of confusing details that could be clarified:\n1) How is 'adversarial' defined in the case of the LIDAR example?\n2) It is unclear what transferabilty between the Point and Scene\nattack scenarios means.\n3) In the case of at least 2 of the LIDAR systems, the proposed approach is\nnot very effective at generating adversaries. Why?\n4) The paper did not clearly show how the constraints were implemented based\non the theory given in the paper, although the diagrams in the\nappendices helped.\n5) It was never clear what constitutes an adversary in terms of the\ncompared algorithms, nor why the generated LIDAR structures should be\nconsidered an adversary.\n6) Why is this? Does it not bias the experiment? \"We also inject the target sce\\\nne 10 times into the dataset to make\nsure it accessible to all models.\"\n7) What is the blue curve in Fig 4b?\n", "summary_of_the_paper": "This paper proposes the use of tree-structured VAE as a mechanism for\nencoding several forms of constraint knowledge. The VAEs create samples\nof scenes that aim to conform to the constraints. A synthetic scene context\nwas used to demonstrate and explore the approach. A more realistic LIDAR\nsegmentation scenario was also explored, where the goal was to generate\nrealistic but adversarial scenes.\n", "summary_of_the_review": "The approach seems reasonably novel, and the paper's writing could be improved. The experiments and applications are not very exciting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5vvRx33PSRu", "reviewer": "Reviewer_sFaB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrength:\n- The tree-structured generative model looks interesting, and they propose a two-stage training method to learn such a model.\n- The comparison with baseline methods looks good, and the authors also show some interesting visualization results.\n\nWeakness:\n- I found this paper is a little bit hard to follow; they start from VAE and introduce the T-VAE, which is reasonable. However, aligning those proposed new functions to the basic VAE equation is not easy. I think the authors should briefly introduce the previous approaches instead of simply citing them, e.g., [49] and [55]. \n- The contribution of this paper is the proposed T-VAE and knowledge-guided generation. The knowledge part adopts some explicit knowledge rules. How to get those rules is not clear? Why only three rules? Also, it seems explicit rules are dependent on human experts. Can we introduce some implicit rules?\n- The baseline methods are not state-of-the-art methods, and they should consider the more recent works.", "summary_of_the_paper": "This paper proposes a method to incorporate domain knowledge into the physical scene generation process. They extend the synthetic example to realistic environments. Apart from this, they also propose the semantic point could attack against stoa segmentation methods.\n", "summary_of_the_review": "The T-VAE and knowledge-guided generation proposed in this paper looks new, and the attack looks interesting. However, the current version is a little bit hard to follow, and some contributions seem incremental, for example, the proposed T-VAE compared with VAE. I can reconsider my rating if the authors point anything I was missing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a semantically controllable generative framework by integrating explicit knowledge. In particular, a tree-structured generative model is proposed based on knowledge categorization. Reviewers raised concerns about technical details, experiments, and missing references. In the revised paper, the authors provided more justifications and clarifications, such as the definition of adversarial attack. During the discussion, reviewers agreed that the previous concerns have been partially addressed, but there are still concerns on experiments, e.g., more recent work should be considered as baselines.

Overall, I recommend to reject this paper. I encourage the authors to take the review feedback into account and submit a future version to another venue.
Decision: Reject