Semantically Controllable Generation of Physical Scenes with Explicit Knowledge

Deep Generative Models ( DGMs ) are known for their superior capability in generating realistic data . Extending purely data-driven approaches , recent specialized DGMs satisfy additional controllable requirements such as embedding a traffic sign in a driving scene by manipulating patterns implicitly in the neuron or feature level . In this paper , we introduce a novel method to incorporate domain knowledge explicitly in the generation process to achieve the semantically controllable generation of physical scenes . We first categorize our knowledge into two types , the property of objects and the relationship among objects , to be consistent with the composition of natural scenes . We then propose a tree-structured generative model to learn hierarchical scene representation , whose nodes and edges naturally corresponded to the two types of knowledge , respectively . Consequently , explicit knowledge integration enables semantically controllable generation by imposing semantic rules on the properties of nodes and edges in the tree structure . We construct a synthetic example to illustrate the controllability and explainability of our method in a succinct setting . We further extend the synthetic example to realistic environments for autonomous vehicles and conduct extensive experiments : our method efficiently identifies adversarial physical scenes against different state-of-the-art 3D point cloud segmentation models , and satisfies the traffic rules specified as the explicit knowledge . 1 INTRODUCTION . The recent breakthrough in machine learning enables us to learn complex distributions behind data with sophisticated models . These models help us understand the data generation process so as to realize controllable data generation [ 1 , 65 , 16 ] . Deep Generative Models ( DGMs ) [ 23 , 34 , 17 ] , which approximate the data distribution with neural networks ( NN ) , are representative methods to generate data targeting a specific style , category , or attribute . However , existing controllable generative models focus on manipulating implicit patterns in the neuron or feature level . For instance , [ 7 ] dissects DGMs to build the relationship between neurons and generated data , while [ 55 ] interpolates in the latent space to obtain vectors that control the poses of objects . One main limitation of these existing models is that they can not explicitly incorporate unseen semantic rules , which may lead to meaningless generated data that violates common sense . For example , to build diverse physical scenes for evaluating autonomous vehicles , the generated cars should follow semantic traffic rules and physical laws , which can not be enforced by directly manipulating neurons . In light of the limitations of previous work , we aim to develop a structured generative framework to integrate explicit knowledge [ 15 ] during the generation process and thus control the generated scene to be compliant with semantic rules . Natural scenes can be described with objects and their various relationships [ 5 ] . Thus , in this paper , we categorize the semantic knowledge that describes scenes into two types , where the first type denoted as node-level knowledge represents the properties of single objects and the second type denoted as edge-level knowledge represents the relationship among objects . We also observe that tree structure is highly consistent with this categorization for constructing scenes , where nodes of the tree represent objects and edges the relationship . By automatically controlling the tree structure during the generation , we explicitly integrate the node-level and edge-level knowledge . In detail , we propose a general framework , Semantically Controllable Generation ( SCG ) , which consists of two stages as shown in Figure 1 . In stage one , we train a tree-structured generative model that parameterizes nodes and edges of trees with NN to learn the representation of structured data . In stage two , explicit knowledge is applied to different levels of the tree to achieve semantically controllable generation for downstream tasks such as satisfying certain conditions or reducing the performance of recognition algorithms . To verify the proposed SCG , we first construct a synthetic scene reconstruction example to illustrate the advantages of SCG and provide analysis on its controllability and explainability . With SCG , it is possible to generate natural scenes that follow semantic rules , e.g. , boxes with the same color should be positioned close to each other . To demonstrate the practicality of SCG , we conduct extensive experiments on adversarial LiDAR scene generation against state-of-the-art 3D segmentation models . We show that our generated safety-critical physical scenes successfully attack victim models and meanwhile follow the specified traffic rules . In addition , compared with traditional attack methods , scenes generated by our method achieve stronger adversarial transferability across different victim models . Our technical contributions are summarized below : • We propose a semantically controllable generative framework ( SCG ) via integrating explicit knowledge and categorize the knowledge into two types according to the composition of scenes . • We propose a tree-structured generative model based on our knowledge categorization and construct a synthetic example to demonstrate the effectiveness of our knowledge integration . • We propose the first semantic adversarial point cloud attack based on SCG , named Scene Attack , against state-of-the-art segmentation algorithms , demonstrating several essential properties . 2 SEMANTICALLY CONTROLLABLE GENERATION . We define the scene x ∈ X in the data space and the latent code z ∈ Z in the latent space . This paper aims to generate scene x that satisfies certain semantic rules Kt , which are related to the downstream task t ∈ T . The scene x will be used to solve the downstream task t by minimizing the corresponding objective function Lt ( x ) . In this section , we first describe the tree-based generative model for learning the hierarchical representations of x , which is important and necessary for applying knowledge to achieve semantic controllability . Then we explain the two types of knowledge to be integrated into the generative model together with the generation algorithm that uses explicit knowledge Kt as guidance for downstream tasks . 2.1 TREE-STRUCTURED VARIATIONAL AUTO-ENCODER ( T-VAE ) . VAE [ 34 ] is a powerful DGM that combines auto-encoder and variational inference [ 9 ] . It estimates a mapping between data x and latent code z to find the low-dimensional manifold of the data space . The objective function of training VAE is to maximize a lower bound of the log-likelihood of training data , which is so-called Evidence Lower Bound ( ELBO ) ELBO = Eq ( z|x ; φ ) [ log p ( x|z ; θ ) ] −KL ( q ( z|x ; φ ) ||p ( z ) ) ( 1 ) where KL is Kullback–Leibler ( KL ) divergence . q ( z|x ; φ ) is the encoder with parameters φ , and p ( x|z ; θ ) is the decoder with parameters θ . The prior distribution of the latent code p ( z ) is usually a Gaussian distribution for simplification of KL divergence calculation . One typical characteristic of natural scenes is the variable data dimension caused by the variable number of objects , which is very challenging to represent with a fixed number of parameters as in traditional models [ 34 ] . Besides , many existing structured generative models [ 64 , 40 ] do not consider the hierarchy of natural scenes nor have the capability to incorporate explicit knowledge . In general , graph is commonly used to represent structured data [ 41 ] , but sometimes is too complicated to describe the hierarchy and inefficient to generate . As a special case of graph , tree naturally embed hierarchical information via recursive generation with depth-first-search traversal [ 29 , 48 ] . This hierarchy is not only highly consistent with natural physical scenes , but also makes easier to apply explicit knowledge , supported by previous works in cognition literature [ 46 ] . In this work , we propose a novel tree-structured generative model , which is inspired by the stickbreaking approach [ 58 ] : Assume we have a stick with length W and we iteratively break it into segments w ( n , i ) with : W = w ( 1,1 ) = w ( 2,1 ) + w ( 2,2 ) = · · · = Kn∑ i=1 w ( n , i ) ( 2 ) where ( n , i ) means the i-th segment of the n-th level . Kn is the total number of segments in the n-th level . The index starts from 1 and entire length has index ( 1 , 1 ) . The recursive function of breaking the stick follows w ( n+1 , j ) = w ( n , i ) α ( n , i ) , w ( n+1 , j+1 ) = w ( n , i ) ( 1− α ( n , i ) ) ( 3 ) where α ( n , i ) ∈ [ 0 , 1 ] is the splitting ratio for w ( n , i ) and j is the index in the n+ 1-th level . Intuitively , this breaking process creates a tree structure where the i-th level is corresponding to the i-th layer of the tree and segments are nodes in the tree . We extent the above division to 2D space with two parameters α and β . A 2D plane example is illustrated in Figure 2 , where each node has 4 child nodes in the next layer . Assume there are M kinds of nodes in the scene , we define a batch of encoders Em and decoders Dm for all m ∈M and n ∈ { 1 , · · · , N − 1 } : f ( n , j ) = Em ( [ f ( n+1 , i ) , · · · , f ( n+1 , i+lm ) , g ( n+1 , i ) ] ; φm ) [ f ( n+1 , j ) , · · · , f ( n+1 , j+lm ) , ĝ ( n+1 , j ) ] = Dm ( f ( n , i ) ; θm ) ( 4 ) where f ( n , i ) is named as the feature vector that passes the messages through the tree structure . g ( n , i ) is named as the property vector of node ( n , i ) that stories specific properties such as color of the object generated by node ( n , i ) . ĝ ( n+1 , j ) is the predicted property vector and lm is the number of children of node m. Besides the encoders and decoders , we also define a Classifier to determine the child node type and a Sampler to infer the posterior distribution of the latent code , ĉ ( n , i ) = Classifier ( f ( n , i ) ; θc ) , [ zµ , zσ ] = Sampler ( f ( 1,1 ) ; φs ) ( 5 ) where ĉ ( n , i ) is the predicted node type and [ zµ , zσ ] is used to calculate the latent code z with the reparameterization trick [ 9 ] . θc and θs are models parameters for the Classifier and Sampler . Parameters for the encoders q ( z|x ; φ ) and decoders p ( x|z ; θ ) are denoted respectively as φ = { φ1 , · · · , φm , φs } and θ = { θ1 , · · · , θm , θc } . The final structures of the encoder and decoder depend on the tree structure of the data point and vary in the dataset . We follow Recursive Neural Networks ( RvNN ) [ 61 ] to build the tree structure recursively . Finally , the tree x is summarized as x = { c , g } = { c ( 1,1 ) , · · · , c ( N , KN ) , · · · , g ( 1,1 ) , · · · , g ( N , KN ) } ( 6 ) where c represents the node type . Following ( 1 ) , the ELBO of T-VAE to be maximized is ELBO = Eq [ log p ( c|z ; θ ) ] ︸ ︷︷ ︸ −LC ( ĉ , c ) +Eq [ log p ( g|z ; θ ) ] ︸ ︷︷ ︸ −LR ( ĝ , g ) −KL ( N ( zµ , zσ ) ‖ N ( 0 , I ) ) ( 7 ) where the equality holds because c and g are conditionally independent given z . The LC term represents the cross-entropy loss ( CE ) of all nodes in p ( x|z ; θ ) , LC ( ĉ , c ) = 1∑N n Kn N∑ n=1 Kn∑ i=1 CE ( ĉ ( n , i ) , c ( n , i ) ; p ( c ) ) ( 8 ) where the prior distribution of node type p ( c ) is calculated from the training dataset and serves as weights . To make the reconstructed tree have same structure as the original one , we use Teacher Forcing [ 67 ] during the training stage . However , in the generation stage , we take the node with maximum probability as the child to expand the tree . The LR term uses mean square error ( MSE ) to approximate the log-likelihood of node property , LR ( ĝ , g ) = M∑ m=1 1 Nm N∑ n=1 Kn∑ i=1 1 [ c ( n , i ) = m ] ‖ĝ ( n , i ) − g ( n , i ) ‖22 ( 9 ) where Nm is the times that node m appears in the tree and 1 [ · ] is the indicator function . In ( 9 ) , we normalize the MSE with Nm instead of ∑N n Kn to avoid the influence caused by imbalanced node type in the tree . Please refer to Appendix B for detailed model definition and generative process . The advantage of this hierarchical structure is that we only need to store the global information in the root node and use local information in other nodes , making the model easier to capture the feature from different scales in the scene . Moreover , this tree structure makes it possible to explicitly apply semantic knowledge in Stage 2 , which will be explained in Section 2.2 .

Summary: This paper proposes the use of tree-structured VAE as a mechanism for encoding several forms of constraint knowledge. The VAEs create samples of scenes that aim to conform to the constraints. A synthetic scene context was used to demonstrate and explore the approach. A more realistic LIDAR segmentation scenario was also explored, where the goal was to generate realistic but adversarial scenes.