Semantically Controllable Generation of Physical Scenes with Explicit Knowledge

Deep Generative Models ( DGMs ) are known for their superior capability in generating realistic data . Extending purely data-driven approaches , recent specialized DGMs satisfy additional controllable requirements such as embedding a traffic sign in a driving scene by manipulating patterns implicitly in the neuron or feature level . In this paper , we introduce a novel method to incorporate domain knowledge explicitly in the generation process to achieve the semantically controllable generation of physical scenes . We first categorize our knowledge into two types , the property of objects and the relationship among objects , to be consistent with the composition of natural scenes . We then propose a tree-structured generative model to learn hierarchical scene representation , whose nodes and edges naturally corresponded to the two types of knowledge , respectively . Consequently , explicit knowledge integration enables semantically controllable generation by imposing semantic rules on the properties of nodes and edges in the tree structure . We construct a synthetic example to illustrate the controllability and explainability of our method in a succinct setting . We further extend the synthetic example to realistic environments for autonomous vehicles and conduct extensive experiments : our method efficiently identifies adversarial physical scenes against different state-of-the-art 3D point cloud segmentation models , and satisfies the traffic rules specified as the explicit knowledge . 1 INTRODUCTION . The recent breakthrough in machine learning enables us to learn complex distributions behind data with sophisticated models . These models help us understand the data generation process so as to realize controllable data generation [ 1 , 65 , 16 ] . Deep Generative Models ( DGMs ) [ 23 , 34 , 17 ] , which approximate the data distribution with neural networks ( NN ) , are representative methods to generate data targeting a specific style , category , or attribute . However , existing controllable generative models focus on manipulating implicit patterns in the neuron or feature level . For instance , [ 7 ] dissects DGMs to build the relationship between neurons and generated data , while [ 55 ] interpolates in the latent space to obtain vectors that control the poses of objects . One main limitation of these existing models is that they can not explicitly incorporate unseen semantic rules , which may lead to meaningless generated data that violates common sense . For example , to build diverse physical scenes for evaluating autonomous vehicles , the generated cars should follow semantic traffic rules and physical laws , which can not be enforced by directly manipulating neurons . In light of the limitations of previous work , we aim to develop a structured generative framework to integrate explicit knowledge [ 15 ] during the generation process and thus control the generated scene to be compliant with semantic rules . Natural scenes can be described with objects and their various relationships [ 5 ] . Thus , in this paper , we categorize the semantic knowledge that describes scenes into two types , where the first type denoted as node-level knowledge represents the properties of single objects and the second type denoted as edge-level knowledge represents the relationship among objects . We also observe that tree structure is highly consistent with this categorization for constructing scenes , where nodes of the tree represent objects and edges the relationship . By automatically controlling the tree structure during the generation , we explicitly integrate the node-level and edge-level knowledge . In detail , we propose a general framework , Semantically Controllable Generation ( SCG ) , which consists of two stages as shown in Figure 1 . In stage one , we train a tree-structured generative model that parameterizes nodes and edges of trees with NN to learn the representation of structured data . In stage two , explicit knowledge is applied to different levels of the tree to achieve semantically controllable generation for downstream tasks such as satisfying certain conditions or reducing the performance of recognition algorithms . To verify the proposed SCG , we first construct a synthetic scene reconstruction example to illustrate the advantages of SCG and provide analysis on its controllability and explainability . With SCG , it is possible to generate natural scenes that follow semantic rules , e.g. , boxes with the same color should be positioned close to each other . To demonstrate the practicality of SCG , we conduct extensive experiments on adversarial LiDAR scene generation against state-of-the-art 3D segmentation models . We show that our generated safety-critical physical scenes successfully attack victim models and meanwhile follow the specified traffic rules . In addition , compared with traditional attack methods , scenes generated by our method achieve stronger adversarial transferability across different victim models . Our technical contributions are summarized below : • We propose a semantically controllable generative framework ( SCG ) via integrating explicit knowledge and categorize the knowledge into two types according to the composition of scenes . • We propose a tree-structured generative model based on our knowledge categorization and construct a synthetic example to demonstrate the effectiveness of our knowledge integration . • We propose the first semantic adversarial point cloud attack based on SCG , named Scene Attack , against state-of-the-art segmentation algorithms , demonstrating several essential properties . 2 SEMANTICALLY CONTROLLABLE GENERATION . We define the scene x ∈ X in the data space and the latent code z ∈ Z in the latent space . This paper aims to generate scene x that satisfies certain semantic rules Kt , which are related to the downstream task t ∈ T . The scene x will be used to solve the downstream task t by minimizing the corresponding objective function Lt ( x ) . In this section , we first describe the tree-based generative model for learning the hierarchical representations of x , which is important and necessary for applying knowledge to achieve semantic controllability . Then we explain the two types of knowledge to be integrated into the generative model together with the generation algorithm that uses explicit knowledge Kt as guidance for downstream tasks . 2.1 TREE-STRUCTURED VARIATIONAL AUTO-ENCODER ( T-VAE ) . VAE [ 34 ] is a powerful DGM that combines auto-encoder and variational inference [ 9 ] . It estimates a mapping between data x and latent code z to find the low-dimensional manifold of the data space . The objective function of training VAE is to maximize a lower bound of the log-likelihood of training data , which is so-called Evidence Lower Bound ( ELBO ) ELBO = Eq ( z|x ; φ ) [ log p ( x|z ; θ ) ] −KL ( q ( z|x ; φ ) ||p ( z ) ) ( 1 ) where KL is Kullback–Leibler ( KL ) divergence . q ( z|x ; φ ) is the encoder with parameters φ , and p ( x|z ; θ ) is the decoder with parameters θ . The prior distribution of the latent code p ( z ) is usually a Gaussian distribution for simplification of KL divergence calculation . One typical characteristic of natural scenes is the variable data dimension caused by the variable number of objects , which is very challenging to represent with a fixed number of parameters as in traditional models [ 34 ] . Besides , many existing structured generative models [ 64 , 40 ] do not consider the hierarchy of natural scenes nor have the capability to incorporate explicit knowledge . In general , graph is commonly used to represent structured data [ 41 ] , but sometimes is too complicated to describe the hierarchy and inefficient to generate . As a special case of graph , tree naturally embed hierarchical information via recursive generation with depth-first-search traversal [ 29 , 48 ] . This hierarchy is not only highly consistent with natural physical scenes , but also makes easier to apply explicit knowledge , supported by previous works in cognition literature [ 46 ] . In this work , we propose a novel tree-structured generative model , which is inspired by the stickbreaking approach [ 58 ] : Assume we have a stick with length W and we iteratively break it into segments w ( n , i ) with : W = w ( 1,1 ) = w ( 2,1 ) + w ( 2,2 ) = · · · = Kn∑ i=1 w ( n , i ) ( 2 ) where ( n , i ) means the i-th segment of the n-th level . Kn is the total number of segments in the n-th level . The index starts from 1 and entire length has index ( 1 , 1 ) . The recursive function of breaking the stick follows w ( n+1 , j ) = w ( n , i ) α ( n , i ) , w ( n+1 , j+1 ) = w ( n , i ) ( 1− α ( n , i ) ) ( 3 ) where α ( n , i ) ∈ [ 0 , 1 ] is the splitting ratio for w ( n , i ) and j is the index in the n+ 1-th level . Intuitively , this breaking process creates a tree structure where the i-th level is corresponding to the i-th layer of the tree and segments are nodes in the tree . We extent the above division to 2D space with two parameters α and β . A 2D plane example is illustrated in Figure 2 , where each node has 4 child nodes in the next layer . Assume there are M kinds of nodes in the scene , we define a batch of encoders Em and decoders Dm for all m ∈M and n ∈ { 1 , · · · , N − 1 } : f ( n , j ) = Em ( [ f ( n+1 , i ) , · · · , f ( n+1 , i+lm ) , g ( n+1 , i ) ] ; φm ) [ f ( n+1 , j ) , · · · , f ( n+1 , j+lm ) , ĝ ( n+1 , j ) ] = Dm ( f ( n , i ) ; θm ) ( 4 ) where f ( n , i ) is named as the feature vector that passes the messages through the tree structure . g ( n , i ) is named as the property vector of node ( n , i ) that stories specific properties such as color of the object generated by node ( n , i ) . ĝ ( n+1 , j ) is the predicted property vector and lm is the number of children of node m. Besides the encoders and decoders , we also define a Classifier to determine the child node type and a Sampler to infer the posterior distribution of the latent code , ĉ ( n , i ) = Classifier ( f ( n , i ) ; θc ) , [ zµ , zσ ] = Sampler ( f ( 1,1 ) ; φs ) ( 5 ) where ĉ ( n , i ) is the predicted node type and [ zµ , zσ ] is used to calculate the latent code z with the reparameterization trick [ 9 ] . θc and θs are models parameters for the Classifier and Sampler . Parameters for the encoders q ( z|x ; φ ) and decoders p ( x|z ; θ ) are denoted respectively as φ = { φ1 , · · · , φm , φs } and θ = { θ1 , · · · , θm , θc } . The final structures of the encoder and decoder depend on the tree structure of the data point and vary in the dataset . We follow Recursive Neural Networks ( RvNN ) [ 61 ] to build the tree structure recursively . Finally , the tree x is summarized as x = { c , g } = { c ( 1,1 ) , · · · , c ( N , KN ) , · · · , g ( 1,1 ) , · · · , g ( N , KN ) } ( 6 ) where c represents the node type . Following ( 1 ) , the ELBO of T-VAE to be maximized is ELBO = Eq [ log p ( c|z ; θ ) ] ︸ ︷︷ ︸ −LC ( ĉ , c ) +Eq [ log p ( g|z ; θ ) ] ︸ ︷︷ ︸ −LR ( ĝ , g ) −KL ( N ( zµ , zσ ) ‖ N ( 0 , I ) ) ( 7 ) where the equality holds because c and g are conditionally independent given z . The LC term represents the cross-entropy loss ( CE ) of all nodes in p ( x|z ; θ ) , LC ( ĉ , c ) = 1∑N n Kn N∑ n=1 Kn∑ i=1 CE ( ĉ ( n , i ) , c ( n , i ) ; p ( c ) ) ( 8 ) where the prior distribution of node type p ( c ) is calculated from the training dataset and serves as weights . To make the reconstructed tree have same structure as the original one , we use Teacher Forcing [ 67 ] during the training stage . However , in the generation stage , we take the node with maximum probability as the child to expand the tree . The LR term uses mean square error ( MSE ) to approximate the log-likelihood of node property , LR ( ĝ , g ) = M∑ m=1 1 Nm N∑ n=1 Kn∑ i=1 1 [ c ( n , i ) = m ] ‖ĝ ( n , i ) − g ( n , i ) ‖22 ( 9 ) where Nm is the times that node m appears in the tree and 1 [ · ] is the indicator function . In ( 9 ) , we normalize the MSE with Nm instead of ∑N n Kn to avoid the influence caused by imbalanced node type in the tree . Please refer to Appendix B for detailed model definition and generative process . The advantage of this hierarchical structure is that we only need to store the global information in the root node and use local information in other nodes , making the model easier to capture the feature from different scales in the scene . Moreover , this tree structure makes it possible to explicitly apply semantic knowledge in Stage 2 , which will be explained in Section 2.2 .

Summary: The paper proposes a way to incorporate explicitly defined (e.g. rule based) knowledge into generative models for scene generation. Through the addition of explicit knowledge the approach can ensure that generated scenes follow specific requirements, e.g. physical rules. The approach is evaluated on a synthetic dataset and as a way to generate adversarial examples for scene based segmentation models.