Year: 2021

Abstract:
Deep Generative Models (DGMs) are known for their superior capability in generating realistic data. Extending purely data-driven approaches, recent specialized DGMs satisfy additional controllable requirements such as embedding a traffic sign in a driving scene by manipulating patterns implicitly in the neuron or feature level. In this paper, we introduce a novel method to incorporate domain knowledge explicitly in the generation process to achieve the semantically controllable generation of physical scenes. We first categorize our knowledge into two types, the property of objects and the relationship among objects, to be consistent with the composition of natural scenes. We then propose a tree-structured generative model to learn hierarchical scene representation, whose nodes and edges naturally corresponded to the two types of knowledge, respectively. Consequently, explicit knowledge integration enables semantically controllable generation by imposing semantic rules on the properties of nodes and edges in the tree structure. We construct a synthetic example to illustrate the controllability and explainability of our method in a succinct setting. We further extend the synthetic example to realistic environments for autonomous vehicles and conduct extensive experiments: our method efficiently identifies adversarial physical scenes against different state-of-the-art 3D point cloud segmentation models, and satisfies the traffic rules specified as the explicit knowledge.

text: Deep Generative Models (DGMs) are known for their superior capability in generating realistic data. Extending purely data-driven approaches, recent specialized DGMs satisfy additional controllable requirements such as embedding a traffic sign in a driving scene by manipulating patterns implicitly in the neuron or feature level. In this paper, we introduce a novel method to incorporate domain knowledge explicitly in the generation process to achieve the semantically controllable generation of physical scenes. We first categorize our knowledge into two types, the property of objects and the relationship among objects, to be consistent with the composition of natural scenes. We then propose a tree-structured generative model to learn hierarchical scene representation, whose nodes and edges naturally corresponded to the two types of knowledge, respectively. Consequently, explicit knowledge integration enables semantically controllable generation by imposing semantic rules on the properties of nodes and edges in the tree structure. We construct a synthetic example to illustrate the controllability and explainability of our method in a succinct setting. We further extend the synthetic example to realistic environments for autonomous vehicles and conduct extensive experiments: our method efficiently identifies adversarial physical scenes against different state-of-the-art 3D point cloud segmentation models, and satisfies the traffic rules specified as the explicit knowledge.

heading: 1 INTRODUCTION
text: The recent breakthrough in machine learning enables us to learn complex distributions behind data with sophisticated models. These models help us understand the data generation process so as to realize controllable data generation [1, 65, 16]. Deep Generative Models (DGMs) [23, 34, 17], which approximate the data distribution with neural networks (NN), are representative methods to generate data targeting a specific style, category, or attribute. However, existing controllable generative models focus on manipulating implicit patterns in the neuron or feature level. For instance, [7] dissects DGMs to build the relationship between neurons and generated data, while [55] interpolates in the latent space to obtain vectors that control the poses of objects. One main limitation of these existing models is that they cannot explicitly incorporate unseen semantic rules, which may lead to meaningless generated data that violates common sense. For example, to build diverse physical scenes for evaluating autonomous vehicles, the generated cars should follow semantic traffic rules and physical laws, which cannot be enforced by directly manipulating neurons. In light of the limitations of previous work, we aim to develop a structured generative framework to integrate explicit knowledge [15] during the generation process and thus control the generated scene to be compliant with semantic rules.
Natural scenes can be described with objects and their various relationships [5]. Thus, in this paper, we categorize the semantic knowledge that describes scenes into two types, where the first type denoted as node-level knowledge represents the properties of single objects and the second type denoted as edge-level knowledge represents the relationship among objects. We also observe that tree structure is highly consistent with this categorization for constructing scenes, where nodes of the tree represent objects and edges the relationship. By automatically controlling the tree structure during the generation, we explicitly integrate the node-level and edge-level knowledge.
In detail, we propose a general framework, Semantically Controllable Generation (SCG), which consists of two stages as shown in Figure 1. In stage one, we train a tree-structured generative model that parameterizes nodes and edges of trees with NN to learn the representation of structured data. In stage two, explicit knowledge is applied to different levels of the tree to achieve semantically controllable generation for downstream tasks such as satisfying certain conditions or reducing the performance of recognition algorithms.
To verify the proposed SCG, we first construct a synthetic scene reconstruction example to illustrate the advantages of SCG and provide analysis on its controllability and explainability. With SCG, it is possible to generate natural scenes that follow semantic rules, e.g., boxes with the same color should be positioned close to each other. To demonstrate the practicality of SCG, we conduct extensive experiments on adversarial LiDAR scene generation against state-of-the-art 3D segmentation models. We show that our generated safety-critical physical scenes successfully attack victim models and meanwhile follow the specified traffic rules. In addition, compared with traditional attack methods, scenes generated by our method achieve stronger adversarial transferability across different victim models. Our technical contributions are summarized below:
• We propose a semantically controllable generative framework (SCG) via integrating explicit knowledge and categorize the knowledge into two types according to the composition of scenes.
• We propose a tree-structured generative model based on our knowledge categorization and construct a synthetic example to demonstrate the effectiveness of our knowledge integration.
• We propose the first semantic adversarial point cloud attack based on SCG, named Scene Attack, against state-of-the-art segmentation algorithms, demonstrating several essential properties.

heading: 2 SEMANTICALLY CONTROLLABLE GENERATION
text: We define the scene x ∈ X in the data space and the latent code z ∈ Z in the latent space. This paper aims to generate scene x that satisfies certain semantic rules Kt, which are related to the downstream task t ∈ T . The scene x will be used to solve the downstream task t by minimizing the corresponding objective function Lt(x). In this section, we first describe the tree-based generative model for learning the hierarchical representations of x, which is important and necessary for applying knowledge to achieve semantic controllability. Then we explain the two types of knowledge to be integrated into the generative model together with the generation algorithm that uses explicit knowledge Kt as guidance for downstream tasks.

heading: 2.1 TREE-STRUCTURED VARIATIONAL AUTO-ENCODER (T-VAE)
text: VAE [34] is a powerful DGM that combines auto-encoder and variational inference [9]. It estimates a mapping between data x and latent code z to find the low-dimensional manifold of the data space. The objective function of training VAE is to maximize a lower bound of the log-likelihood of training data, which is so-called Evidence Lower Bound (ELBO)
ELBO = Eq(z|x;φ) [log p(x|z; θ)]−KL(q(z|x;φ)||p(z)) (1)
where KL is Kullback–Leibler (KL) divergence. q(z|x;φ) is the encoder with parameters φ, and p(x|z; θ) is the decoder with parameters θ. The prior distribution of the latent code p(z) is usually a Gaussian distribution for simplification of KL divergence calculation.
One typical characteristic of natural scenes is the variable data dimension caused by the variable number of objects, which is very challenging to represent with a fixed number of parameters as in traditional models [34]. Besides, many existing structured generative models [64, 40] do not consider the hierarchy of natural scenes nor have the capability to incorporate explicit knowledge.
In general, graph is commonly used to represent structured data [41], but sometimes is too complicated to describe the hierarchy and inefficient to generate. As a special case of graph, tree naturally embed hierarchical information via recursive generation with depth-first-search traversal [29, 48]. This hierarchy is not only highly consistent with natural physical scenes, but also makes easier to apply explicit knowledge, supported by previous works in cognition literature [46].
In this work, we propose a novel tree-structured generative model, which is inspired by the stickbreaking approach [58]: Assume we have a stick with length W and we iteratively break it into segments w(n,i) with:
W = w(1,1) = w(2,1) + w(2,2) = · · · = Kn∑ i=1 w(n,i) (2)
where (n, i) means the i-th segment of the n-th level. Kn is the total number of segments in the n-th level. The index starts from 1 and entire length has index (1, 1). The recursive function of breaking the stick follows
w(n+1,j) = w(n,i)α(n,i), w(n+1,j+1) = w(n,i)(1− α(n,i)) (3)
where α(n,i) ∈ [0, 1] is the splitting ratio for w(n,i) and j is the index in the n+ 1-th level. Intuitively, this breaking process creates a tree structure where the i-th level is corresponding to the i-th layer of the tree and segments are nodes in the tree. We extent the above division to 2D space with two parameters α and β. A 2D plane example is illustrated in Figure 2, where each node has 4 child nodes in the next layer.
Assume there are M kinds of nodes in the scene, we define a batch of encoders Em and decoders Dm for all m ∈M and n ∈ {1, · · · , N − 1}:
f (n,j) = Em([f (n+1,i), · · · , f (n+1,i+lm), g(n+1,i)];φm) [f (n+1,j), · · · , f (n+1,j+lm), ĝ(n+1,j)] = Dm(f (n,i); θm) (4)
where f (n,i) is named as the feature vector that passes the messages through the tree structure. g(n,i) is named as the property vector of node (n, i) that stories specific properties such as color of the object generated by node (n, i). ĝ(n+1,j) is the predicted property vector and lm is the number of
children of node m. Besides the encoders and decoders, we also define a Classifier to determine the child node type and a Sampler to infer the posterior distribution of the latent code,
ĉ(n,i) = Classifier(f (n,i); θc), [zµ, zσ] = Sampler(f (1,1);φs) (5)
where ĉ(n,i) is the predicted node type and [zµ, zσ] is used to calculate the latent code z with the reparameterization trick [9]. θc and θs are models parameters for the Classifier and Sampler.
Parameters for the encoders q(z|x;φ) and decoders p(x|z;θ) are denoted respectively as φ = {φ1, · · · , φm, φs} and θ = {θ1, · · · , θm, θc}. The final structures of the encoder and decoder depend on the tree structure of the data point and vary in the dataset. We follow Recursive Neural Networks (RvNN) [61] to build the tree structure recursively. Finally, the tree x is summarized as
x = {c, g} = {c(1,1), · · · , c(N,KN ), · · · , g(1,1), · · · , g(N,KN )} (6)
where c represents the node type. Following (1), the ELBO of T-VAE to be maximized is
ELBO = Eq [log p(c|z; θ)]︸ ︷︷ ︸ −LC(ĉ,c) +Eq [log p(g|z; θ)]︸ ︷︷ ︸ −LR(ĝ,g) −KL (N (zµ, zσ)‖ N (0, I)) (7)
where the equality holds because c and g are conditionally independent given z. The LC term represents the cross-entropy loss (CE) of all nodes in p(x|z;θ),
LC(ĉ, c) = 1∑N n Kn N∑ n=1 Kn∑ i=1 CE(ĉ(n,i), c(n,i); p(c)) (8)
where the prior distribution of node type p(c) is calculated from the training dataset and serves as weights. To make the reconstructed tree have same structure as the original one, we use Teacher Forcing [67] during the training stage. However, in the generation stage, we take the node with maximum probability as the child to expand the tree. The LR term uses mean square error (MSE) to approximate the log-likelihood of node property,
LR(ĝ, g) = M∑ m=1 1 Nm N∑ n=1 Kn∑ i=1 1 [ c(n,i) = m ] ‖ĝ(n,i) − g(n,i)‖22 (9)
where Nm is the times that node m appears in the tree and 1 [·] is the indicator function. In (9), we normalize the MSE with Nm instead of ∑N n Kn to avoid the influence caused by imbalanced node type in the tree. Please refer to Appendix B for detailed model definition and generative process.
The advantage of this hierarchical structure is that we only need to store the global information in the root node and use local information in other nodes, making the model easier to capture the feature from different scales in the scene. Moreover, this tree structure makes it possible to explicitly apply semantic knowledge in Stage 2, which will be explained in Section 2.2.

heading: 2.2 KNOWLEDGE-GUIDED GENERATION
text: Suppose there is a function set F , where the function f(A) ∈ F returns true or false for a given input node A of a tree x. Then, we define the two types of propositional knowledge Kt for a particular downstream task t (we omit t below for simplification) using the first-order logic [60] as following.
Definition 1 (Knowledge Set) The node-level knowledge kn is denoted as f(A) for a function f ∈ F , where A is a single node. The edge-level knowledge ke is denoted as f1(A) → ∀i f2(Bi) for two functions f1, f2 ∈ F , where we apply knowledge f2 to all A’s child nodes Bi. Then, The knowledge set is constructed as K = {k(1)n , · · · , k(1)e , · · · }.
In the tree context, kn describes the properties of a single node, and ke describes the relationship between the parent node and its children. Thus, the implementation of knowledge is based on the recursive traversal of the tree. We summarized this process in Algorithm 1. For the specific knowledge used in our experiment, please refer to Appendix C.
Algorithm 2: SCG Framework
Input: Dataset D, Task loss Lt(x), Searching budget B, Knowledge set K
Output: Generated scene xs 1 Stage 1: Train T-VAE 2 Initialize model parameters {θ,φ} 3 for x in D do 4 Encode z ← q(z|x;φ) 5 Decode x̂← p(x|z;θ) 6 Update parameters {θ,φ} by
maximizing ELBO (7)
7 Store the learned decoder p(x|z;θ)
8 Stage 2: Generation 9 Initialize latent code z ∼ N (0, I)
10 while B is not used up do 11 if Lt(x) is differentiable then 12 z ← z − η∇Lt(p(x|z;θ)) 13 else 14 z ← Black-box optimization 15 x′ ← ApplyK(K, x) 16 z ← proxLK(z) with (10)
17 Decode the scene xs = p(x|z;θ)
Algorithm 1: Apply Knowledge Input: Knowledge K, Tree (root) x Output: Modified Tree x′
1 Function ApplyK(K, x) 2 for each knowledge k(n) ∈ K do 3 x′ ← modify x according to k(n)
4 if x has child nodes then 5 for all child nodes xi of x do 6 x′i ← ApplyK(K, xi) 7 Add node x′i as a child to x ′
8 return x′
Then, we apply explicit knowledge to the decoder of T-VAE using the proximal gradient algorithm. The original task objective Lt(x) is augmented to La(x) = Lt(x) + LK(x, x′), where the second term measures the distance between the original tree x and modified tree x′. Specifically, kn changes the property vector g of node A in x to g′ to satisfy f(A), and ke traverses the tree x to find node A that satisfy f1 in ke then change the type vector c or the property vector g of the children of A to c′ and g′ respectively. The reference vector c′ and g′ are related to the downstream task and pre-defined in the function f . Then LK consists of two parts, MSE is used between the property vector g and g′, and CE is used between the type vector c and c′.
For instance, the explicit knowledge described as “if one box is blue, its child nodes should be red” will be implemented by the following operations. Starting from the root, we find all box nodes whose colors are blue and collect the property vectors g of its child nodes; then we change g to g′, which means the color red. Next, LK is obtained by calculating MSE between all g′ and the original g. This process is illustrated in Figure 2(c).
In most applications, Lt(x) requires many resources to compute, while LK is efficient to evaluate since it only involves the inference of p(x|z;θ). Therefore, we resort to Proximal algorithms [52], a type of optimization method, to restrict the searching step to a finite region. In our setting, the explicit knowledge can be treated as the trusted region to limit and guide the optimization of the downstream task. The knowledge loss and task objective are alternatively optimized under the proximal optimization framework. We define the proximal operator as
proxLK(z) = arg minz′
( LK(p(x|z′;φ), x′) + 1
2 ‖z − z′‖22
) (10)
which projects the candidate z to z′ to minimizes the knowledge inconsistency caused by LK. The second term in (10) is a regularize to make the projected point close to the original point.
This problem can be solved by gradient descent since the decoder p(x|z;θ) of T-VAE is differentiable. After the projection, we need to solve the original optimization problem Lt(x). We use gradient descent for a differentiable Lt(x) or change to black-box optimization methods [6] when Lt(x) is non-differentiable. The complete algorithm is summarized in Algorithm 2.

heading: 3 EXPERIMENTS
text: First, we design a synthetic scene to illustrate the controllability and explainability of the proposed framework. The synthetic physical scene provides a simplified setting to unveil the essence of the knowledge-guided generation. After that, we evaluate the performance of SCG on realistic traffic
scenes represented by point clouds. Based on SCG, we propose a new adversarial attack method, Scene Attack, against multiple state-of-the-art segmentation methods.

heading: 3.1 SYNTHETIC SCENE RECONSTRUCTION
text: Task Description. We aim to reconstruct a scene to match a given image as shown in Figure 4. The task objective is a reconstruction error Lt(x) = ‖S − R(x)‖2, where R is a differentiable image renderer [33] and S is the image of target scene. Under this succinct setting, it is possible to analyze and compare the contribution of explicit knowledge integration, since we can access the optimal solution. According to the understanding of the target scene, we define three knowledge rules: 1© The scene has at most two plates; 2© The boxes that belong to the same plate should have the same color; 3© The boxes belong to the same plate should have distance smaller than a threshold γ. We compare our method with three baselines: Direct Search (DS), VAE, and Grammar-VAE (GVAE) [37]. DS directly optimizes the positions and colors of boxes and plates in the data space. In contrast, VAE and GVAE search in the latent space and generate the scene from their decoders.
Table 1: Reconstruction Error
Initialization
Method Random Good
Direct Search 86.0±9.4 7.9±1.2 VAE [34] 110.4±10.6 13.4±6.1 GVAE [37] 123.7±9.5 19.7±10.2 T-VAE 135.1±16.9 14.1±2.5 T-VAE w/ SCG 14.5±1.3 11.8±2.1
Experiment Settings. We synthesize the dataset by randomly generating 10,000 samples with a varying number of boxes and plates. We also inject the target scene 10 times into the dataset to make sure it accessible to all models. Unlike GVAE and T-VAE, DS and VAE need to access the number of boxes and plates in the target image (e.g., two plates and eight boxes) to fix the dimension of the input feature. To get the good initial points for DS, we add a small perturbation to the positions and colors of all objects in the
target scene. Similarly, for other methods, we add the perturbation to the optimal latent code, which is obtained by passing the target scene to the encoder to get good initialization.
Evaluation Results. The generated samples from five models are displayed in Figure 3, and the final errors are summarized in Table. 1. With good initialization, all models find a similar scene to the target one, while with the random initialization, all models are trapped in local minimums. However, obtaining good initialization is not practical in most real-world applications, indicating that this task is non-trivial and all models without knowledge cannot solve it. After integrating the knowledge into the T-VAE model, we obtain I of Figure 3. We can see that all three knowledge have positive guidance for the optimization, e.g., the boxes concentrate on the centers of plates with knowledge 3© When combining the three knowledge, even from a random initialization, our T-VAE can finally find the target scene, leading to a small error in Table. 1. We also want to mention that it is also possible to apply simple knowledge to GVAE during the generation. However, the advantages of our method are that (1) we can integrate any constraints as long as they can be represented by 1, whereas GVAE can only apply hard constraints to objects with co-occurrences.
Analysis of Knowledge and Controllability. To analyze the contribution of each knowledge, we plot the knowledge losses of 1© 3©in Figure 4(b) together with the task loss. All knowledge losses decrease quickly at the beginning and guide the searching in the downstream task. Next, we made ablation studies to explore the reason why knowledge helps the generation. In Figure 4(b), we compare the optimization trajectories of T-VAE (red→yellow) and T-VAE w/ SCG (blue→green) with the same initialization. For T-VAE, the generated samples are diverse but totally different from the target scene, while for T-VAE w/ SCG, the trajectories go to another direction and the generated samples are good. However, the interesting thing is that although the knowledge help us find good scenes, it does not reach the same point in the latent space with the trajectories from good initialization (cyan→purple). This can be explained by the entanglement of the latent space [43], which makes multiple variables control the same property. To further study this point, we plot Figure 4(c), where we use 3 different initialization for T-VAE w/ SCG. The result shows that all three cases find the target scene but with totally different trajectories, which provides evidence to our conjecture. In summary, we believe the contribution of knowledge can be attribute to the entanglement of the latent space, which makes the searching easily escape the local minimum and find the nearest optimal points.

heading: 3.2 ADVERSARIAL TRAFFIC SCENES GENERATION
text: Task Description. We aim to generate realistic traffic scenes against segmentation algorithms as well as satisfying certain semantic knowledge rules. The adversarial scenes are defined as scenes that reduces the performance of downstream victim systems. To generated adversarial LiDAR scenes containing various fore-/background rather than the point cloud of a single 3D object as existing studies [38, 63], a couple of challenges should be considered: First, LiDAR scenes with millions of points are hard to be directly operated; Second, generated scenes need to be realistic and follow traffic rules. Since there are no existing methods to compare with directly, we compare three methods: (1) Point Attack: a point-wise attack baseline [69] that adds small disturbance to points; (2) Pose Attack: a scene generation method developed by us that searches pose of vehicles in the scene; (3) Scene Attack: a semantically controllable generative method based on our T-VAE and SCG. We explore
the attack effectiveness against different models of these methods, as well as their transferability. For Pose Attack and Scene Attack, we implement an efficient LiDAR modelR(x,B) [49] (refer to Appendix A for details) to convert the generated scene x to a point cloud scene with an background B. The task objective minLt(x) = maxLP (R(x,B)) is defined by maximizing the loss function LP of segmentation algorithms P . We design three explicit knowledge rules: 1© roads follow a given layout (location, width, and length); 2© vehicles on the lane follow the direction of the lane; 3© vehicles should gather together but keep a certain distance. 1© ensure generated vehicles follow the layout of the background B and 3©makes the scene contain more vehicles. Experiment Settings. We select 4 segmentation algorithms (PointNet++ [57], PolarSeg [74], SqueezeSegV3 [71], Cylinder3D [75]) as our victim models, all of which are pre-trained on Semantic Kitti dataset [8]. We collect two backgrounds B (Highway and Intersection) in Carla simulator [18]. Since it is usually unable to access the parameters of segmentation algorithms, we focus on the black-box attack in this task. Point Attack optimizes Lt(x) with SimBA [25], while Pose Attack and Scene Attack optimizes Lt(x) with Bayesian Optimization (BO) [54]. For the training of T-VAE, we build a dataset by extracting the pose information of vehicles together with road and lane information from Argoverse dataset [11].
victims, the generated scenes have weak transferability and cannot decrease the performance of other victim models. However, scenes generated by Scene Attack successfully attack all models, even those not targeted during the training, which demonstrates strong adversarial transferability. More generated scenes can be found in Appendix E

heading: 4 RELATED WORK
text: Incorporating Knowledge in Neural Networks. Integrating knowledge to data-driven models has been explored in various forms from training methods, meta-modeling, embedding to rules used for reasoning. [27] distills logical rules with a teacher-student framework under Posterior Regularization [22]. Another way of knowledge distillation is encoding knowledge into vectors then refining the
features from the model that are in line with the encoded knowledge [24]. These methods need to access Knowledge Graphs [19] during the training, which heavily depends on human experts. Meta-modeling of complex fluid is integrated into the NN to improve the performance of purely data-driven networks in [45]. In addition, [72] restricted generative models’ output to satisfy physical laws expressed by partial differential equations. In the reinforcement learning area, reward shaping [50] is recognized as one technique to incorporate heuristic knowledge to guide the training.
Structured Deep Generative Models. The original DGMs, such as GAN [23] and VAE [34], are mostly used for unstructured data. They leverage the powerful feature extraction of neural networks to achieve impressive results [32, 10]. However, the physical world is complex since objects have diverse and structured relationships. Domain specific structured generative models are developed via tree structure (RvNN-VAE [40]) or graph structure (Graph-VAE [59]). Rule-based generative models are also explored by sampling from pre-defined rules [37, 31, 14]. Typical applications of this kind of model are molecular structure generation [26, 29, 30], natural scene generation [51, 13], and automatic program generation [12]. Unlike these existing methods, our approach explicitly integrates knowledge during the generation process. One valuable application of DGMs is generating samples that meet the requirements of downstream tasks [20, 65]. [1, 2] searches in the latent space of StyleGAN [32] to obtain images that are similar to a given image. For structured data, such a searching framework transforms discrete space optimization to continuous space optimization, which was shown to be more efficient [44]. However, it may not guarantee the rationality of generated structured data due to the loss of interpretability and constraints in the latent space [12].
Physical Scene Generation. Traditional ways of scene generation focus on sampling from predefined rules and grammars, such as probabilistic scene graphs used in [56] and heuristic rules applied in [18]. These methods rely on domain expertise and cannot be easily extensible to largescale scenes. Recently, data-driven generative models [14, 64, 51, 40, 36] are proposed to learn the distribution of objects and decouple the generation of scene into sequence [64] and graph [51, 40] of objects. Although they reduce the gap between simulation and reality, generated scenes cannot satisfy specific constraints. Another substantial body of literature [21, 35, 24] explored learning scene graphs from images or generating scene images directly via an end-to-end framework. Their generalization to high-dimensional data is very challenging, making them less effective than modularized methods proposed by [36, 68, 14].
Semantically Adversarial Attacks. Early adversarial attack methods focused on the pixel-wise attack in the image field, where Lp-norm is used to constrain the adversarial perturbation. For the sake of the interpretability of the adversarial samples, recent studies begin to consider semantic attacks. They attack the rendering process of images by modifying the light condition [42, 73] or manipulating the position and shape of objects [4, 70, 28]. This paper explores the generation of adversarial point cloud scenes, which already has similar prior works [66, 3, 62]. [66, 3] modify the environment by adding objects on the top of existing vehicles to make them disappear. [62] create a ghost vehicle by adding an ignorable number of points. However, they modify a single object without considering the structural relationship of the whole scene.

heading: 5 CONCLUSION
text: In this paper, we explore semantically controllable generation tasks with explicit knowledge integration. Inspired by the categorization of knowledge for the scene description, we design a tree-structured generative model to represent structured data. We show that the two types of knowledge can be explicitly injected into the tree structure to guide and restrict the generation process efficiently and effectively. After considering explicit semantic knowledge, we verify that the generated data contain dramatically fewer semantic constraint violations. Meanwhile, the generated data still maintain the diversity property and follow its original underlying distribution. Although we focus on the scene generation application, the SCG framework can be extended to other structured data generation tasks, such as chemical molecules and programming languages, showing the hierarchical properties. One assumption of this work is that the knowledge is helpful or at least harmless as they are summarized and provided by domain experts, which needs examination in the future.

heading: Ethics Statement
text: Our proposed method aims to enhance existing data-driven generative models with explicit knowledge summarized by human experts. In this paper, we focus on the application of physical scene generation. Considering the broad usage of this method in human society, we identify three potentially harmful insights:
1) The explicit knowledge we integrate is summarized by human experts, therefore, could be biased. With such kind of knowledge, the generated data could also be biased and even results in discrimination.
2) The knowledge we use might potentially contain private information. After integrating it into the generative models, there is a risk of user information disclosure.
3) The knowledge-enhanced generation model is powerful to generate realistic adversarial examples. These examples might be used to attack autonomous systems in the real world, e.g. self-driving algorithms, and cause damages.
To mitigate the potential negative ethics impacts mentioned above, we encourage research to follow these instructions:
1) Construct a strict mechanism of knowledge screening and monitor the generated samples against bias and discrimination.
2) We can also propose a knowledge screening mechanism or an intelligent discriminator to reduce the private information content.
3) Since our generated scenes are realistic and follow traffic rules, it is reasonable to make the defense models drive safely in our generated scenes. However, we should restrict the usage of our generated scenes only in evaluation and testing stages before deployment to help self-driving algorithms overcome their shortcomings.

heading: Reproducibility Statement
text: All of our experiments are conducted on a desktop with Intel i9-9900K CPU @ 3.60GHz, NVIDIA GeForce GTX 1080Ti, and 64GB memory. The settings of the experiment environment are described in the code folder in the supplementary. To make the experiment results reproductable, we provide all pre-trained models (please follow the instruction in code/README.md to download). The hyper-parameters of the two experiments are shown in Table 3 and Table 4 in the Appendix.
The dataset we used in Synthetic Scene Reconstruction Experiment is synthesized by ourselves, which can be downloaded from the link provided in the README.md file. This dataset is published under MIT License. The dataset we used in LiDAR Scene Generation Experiment is based on the Argoverse dataset, which is published by Argo AI with MIT License. We use the entire tracking part of the dataset for training and the processed dataset can be downloaded from the link provided in the README.md file.

heading: A LIDAR MODEL IMPLEMENTATION
text: The LiDAR model is implemented by Moller-Trumbore algorithm [49] with the PyTorch [53] package for high-efficiency computation. We assume there is a plane4V0V1V2 in the 3D space constructed by points V0, V1, V2. A ray R(t) with origin O and normalized direction D (we make ‖D‖2 = 1 for simplification) is represented as R(t) = O + tD, where t is the distance between O and the end point D of the ray. If D is the intersection between the ray and the plane4V0V1V2, we can represent the ray with barycenteric coordinate:
T (u, v) = (1− u− v)V0 + uV1 + vV2 = O + tD = R(t) (11) where u and v are weights. To simplify the equations, we define three new notations:
E1 =V1 − V0 E2 =V2 − V0 T =O − V0
(12)
Then, we can solve the distance t in (11) by:[ t u v ] = 1 | −D,E1, E2| [ | − T,E1, E2| | −D,T,E2| | −D,E1, T | ] =
1
(D × E2) · E1 [ (T × E1) · E2 (D × E2) · T (T × E1) ·D ] (13)
Since we know the ray direction D, we can get the 3D coordinate of the intersection with this distance t. During the implementation, we reuse D × E2 and T × E1 to speed up the computation. To make sure the intersecting point T (u, v) is inside the triangle4V0V1V2, we need to have:
u, v, (1− u− v) ∈ [0, 1] (14) If these three conditions are not fulfilled, the intersection point will be removed.
To calculate the point cloud generated by a LiDAR, we first convert vehicle mesh models to triangles F with Delaunay triangulation [39]. Then, we create the array of LiDAR rays with width W and height H for 360◦ view. Finally, we use (13) to calculate the intersection point between all triangles F and LiDAR rays Ri,j(t) in parallel to get the final range map FH×W . The background pointcloud will also be converted to range map BH×W by: θ = arctan z√ x2 + y2 φ = arctan x y
t = √ x2 + y2 + z2
(15)
where (x, y, z) is the coordinate of one point in pointcloud, θ is used to calculate the index of row, and φ is used to calculate the index of column. Then we mix FH×W and BH×W by taking the minimal value for each element:
Mi,j = min{Fi,j , Bi,j}, ∀i ∈W, j ∈ H (16)
where Mi,j represent the (i, j)-entry of the mixed range map scene M . Then, we convert the range map to the final output pointcloud scene S with:
z = t× sin θ x = t× cos θ cosφ y = t× cos θ sinφ
(17)
The entire pipeline of the above process is summarized in Figure 7. The parameters we used for LiDAR follow the configuration of the Semantic Kitti dataset, where the channel H = 64, the horizontal resolution W = 2048, the upper angle is 2◦, and the lower angle is −25◦. The gap between reality and simulation can be reduced by realistic simulation and sensor models [47], but this will not be explored in this paper.

heading: B DETAILED T-VAE MODEL STRUCTURE
text: Our T-VAE model consists of several encoders and decoders that are related to the definition of the scene. In this paper, we explored two experiments with two scenes: a synthetic box placement image scene and a traffic point cloud scene. In Figure 8, we show the details of the modules for two scenes. We also use an synthetic scene as an example to show the encoding and decoding processes in Figure 9, where original data point is stored in tree structure and the generated data is also a tree structure. The encoding process converts the tree into a stack and then encodes the information using the node type defined in the tree. The decoding process expands the tree with the predict node type from the Classifier.
For the synthetic box placement image scene, there are 5 Encoder-Decoder pairs, a Sampler, and a Classifier. W node determines the global information such as location, and orientation of the entire scene. P node spawns a plate object in the scene with positions and colors determined by the property vector. Both Q and B nodes spawn a box object in the scene with positions and colors determined by the property vector. E node serves as a stop signal to end the expansion of a branch, therefore, will not spawn anything in the scene and does not have model parameters.
The traffic point cloud scene has similar definitions. R node contains the information of the road and only has two children. L node determines the lane information such as the width and direction. Both Q and V nodes spawn a vehicle in the scene with positions and orientations determined by the property vector.

heading: C KNOWLEDGE DEFINITION
text: For each experiment in this paper, we design three knowledge rules. We explain the details of the implementation of these rules.
In the Synthetic Scene Reconstruction Experiment, we calculate LY (x, Yt(x)) with the following implementations:
¬ The scene has at most two plates, which can be implemented by W node has at most two P children nodes. We traverse the entire generated tree x to find W node, then we collect the children nodes of W and count the number of P node. If the number is larger than 2, we calculate the cross-entropy loss between the node type and E node label.  The colors of the boxes that belong to the same plate should be the same, which can be implemented by The color of P node’s children nodes should be the same. We traverse the entire generated tree x to find all P nodes, then we collect the colors of the children of P . The average color c̄ is calculated for each P node and c̄ is used as label to calculate the MSE for all children nodes of the corresponding P node. ® The distance between the boxes that belong to the same plate should be smaller than a threshold γ which can be implemented by The distance between P node’s children nodes should be smaller than a threshold γ. We traverse the entire generated tree x to find P node, then we collect the absolute position of all its children nodes and calculate the MSE between this position and the position of P .
In the LiDAR Scene Generation Experiment, we calculate LY (x, Yt(x)) with the following implementations:
¬ Roads follow a given layout (location, width, and length), which can implemented by R node follows a given layout (location, width, and length). We traverse the entire generated tree x to find R node, then calculate the MSE between the property vector of R and the given layout.  Vehicles on the lane follow the direction of the lane, which can implemented by L node follows pre-defined directions. We traverse the entire generated tree x to find L node, then calculate the MSE between the property vector of R and the given layout. ® Vehicles should gather together but keep a certain distance, which can implemented by Q node has at least two Q nodes as its children until the absolute width and height of current block is smaller than thresholds w and h. We traverse the entire generated tree x to find Q node, then collect the type of its children nodes. When the collected Q node type is less than 2, we calculate the cross-entropy loss between two collected node type and Q node label. When the width and height of current block is smaller than w and h, we stop applying this rule.
After calculating all errors in LY (x, Yt(x)), we can directly use back-prorogation to calculate the gradient of latent code z and update it with gradient descent method.

heading: D BASELINES IN SYNTHETIC SCENE RECONSTRUCTION

heading: D.1 DIRECT SEARCH
text: The dimension of the physical space is 6× (2 + 8) = 60, where 6 is the property dimension including position, orientation, and colors, 2 is the number of plates, and 8 is the number of boxes. We use gradient descent with the learning rate η to directly search in the physical space. Since there is no constraint to avoid overlaps between boxes, the generated scenes could be unrealistic.
D.2 VARIATIONAL AUTO-ENCODER (VAE)
The input dimension of the encoder is the same as the searching space of Direct Search. The VAE model has a encoder and a decoder, both of which has a fixed number of model parameters. There are 4 hidden layers in the encoder and each layer has 128 neurons. The decoder also has 4 hidden layers with 128 neurons. For the output of color, we add a Sigmoid(·) function to normalize it to the range [0, 1]. The location is normalized by Nl before it is taken into the encoder.

heading: D.3 GRAMMAR VAE (GVAE)
text: GVAE [37] requires the input data to be described with a set of pre-defined grammars. According to the task of synthetic scene reconstruction experiment, we design 9 rules, W → P, W → B, P → P |E, P → P |B, P → B, P → E, B → B, B → E, E → E (18)
and they are represented in one-hot vector in the dataset. The original GVAE is designed only for rule-based discrete data generation (e.g. molecules), thus we modify the structure to add continuous attribute representation. The encoder consists of 3 1-dimensional Convolution layers with kernel size 3×3. The numbers of channels for the Convolution layers are [32, 64, 128]. The decoder is an LSTM model with 128 neurons, therefore, the decoding process is sequential. During the training stage, the maximal length of rules is fixed to 20 with E → E as padding, and the decoder will output 20 rules. The cross-entropy loss is used between the input rules and decoded rules. During the generation stage, the rules generated from the decoder will be firstly stored in a stack and convert to the tree with the first-in-last-out (FILO) principle.

heading: E ADDITIONAL QUALITATIVE RESULTS
text: In Figure 10, we show samples randomly generated from VAE, GVAE, and T-VAE. The results show that all three models are able to generate diverse samples. Specifically, samples generated by VAE always have 2 plates and 8 boxes due to the fixed input data dimension. In contrast, samples from GVAE and T-VAE have variable numbers of plates and boxes.
In Figure 11, we show 4 more generated scenarios from the Point Attack methods with prediction results from 4 segmentation models. For better visualization, we also show three detailed figures. In Figure 12 and Figure 13, we show more results from Pose Attack and Scene Attack methods. Scenes generated by Scene Attack follow basic traffic rules.

heading: F SEGMENTATION MODELS IN LIDAR SCENE GENERATION

heading: F.1 POINTNET++
text: This model [57] directly uses point-wise features in the 3D space as the backbone to deal with the segmentation problem. Although this model does not have impressive results on the Semantic KITTI dataset, we select it because it influences a lot of existing point cloud processing models. We use the code from this repository and train the model on Semantic KITTI dataset by ourselves following the original training and testing split setting.

heading: F.2 POLARSEG
text: This model [74] converts the data representation from 3D Cartesian coordinate to Polar coordinate and extracts features with 2D convolution layers. We use the code from this repository and use the
pre-trained model provided by the authors. Since we only consider the vehicle class, we change all other labels to non-vehicle class.

heading: F.3 SQUEEZESEGV3
text: This model [71] projects the 3D point cloud to 2D range maps and extract features with 2D convolution layers from the range maps. We use the code from this repository and use the pre-trained model provided by the authors. Since we only consider the vehicle class, we change all other labels to non-vehicle class.

heading: F.4 CYLINDER3D
text: This model [75] converts the data representation from 3D Cartesian coordinate to the Polar coordinate and divides the space into blocks with a cylinder representation. We use the code from this repository and use the pre-trained model provided by the authors. Since we only consider the vehicle class, we change all other labels to non-vehicle class.

Review summaries:
This paper proposes a method to incorporate domain knowledge into the physical scene generation process. They extend the synthetic example to realistic environments. Apart from this, they also propose the semantic point could attack against stoa segmentation methods. 

The paper proposes a way to incorporate explicitly defined (e.g. rule based) knowledge into generative models for scene generation. Through the addition of explicit knowledge the approach can ensure that generated scenes follow specific requirements, e.g. physical rules. The approach is evaluated on a synthetic dataset and as a way to generate adversarial examples for scene based segmentation models.

This paper proposes the use of tree-structured VAE as a mechanism for encoding several forms of constraint knowledge. The VAEs create samples of scenes that aim to conform to the constraints. A synthetic scene context was used to demonstrate and explore the approach. A more realistic LIDAR segmentation scenario was also explored, where the goal was to generate realistic but adversarial scenes.