Semantic Hierarchy Emerges in the Deep Generative Representations for Scene Synthesis

1 INTRODUCTION . Success of deep neural networks stems from the representation learning , which identifies the explanatory factors underlying the high-dimensional observed data ( Bengio et al . ( 2013 ) ) . Prior work has shown that many concept detectors spontaneously emerge inside the deep representations trained for classification task . For example , Gonzalez-Garcia et al . ( 2018 ) shows that networks for object recognition are able to detect semantic object parts , and Bau et al . ( 2017 ) confirms that deep representations from classifying images learn to detect different categorical concepts at different layers . Analyzing the deep representations and their emergent structures gives insight into the generalization ability of deep features ( Morcos et al . ( 2018 ) ) as well as the feature transferability across different tasks ( Yosinski et al . ( 2014 ) ) . But current efforts on interpreting deep representations mainly focus on discriminative models ( Zhou et al . ( 2015 ) ; Gonzalez-Garcia et al . ( 2018 ) ; Zeiler and Fergus ( 2014 ) ; Agrawal et al . ( 2014 ) ; Bau et al . ( 2017 ) ) . Recent advance of Generative Adversarial Networks ( GANs ) ( Goodfellow et al . ( 2014 ) ; Karras et al . ( 2018a ; b ) ; Brock et al . ( 2019 ) ) is capable of transforming random noises into high-quality images , however , the nature of the learned generative representations and how a photo-realistic image is being composed over different layers of the generator in GAN remain much less explored . It is known that the internal units of Convolutional Neural Networks ( CNNs ) emerge as object detectors when trained to categorize scenes ( Zhou et al . ( 2015 ) ) . Representing and detecting informative categorical objects provides an ideal solution for classifying scenes , such as sofa and TV are representative of living room while bed and lamp are of bedroom . However , synthesizing a scene demands far more knowledge for the generative models to learn . Specifically , in order to produce highly-diverse scene images , the deep representations might be required to not only generate every individual object relevant to a specific scene category , but also decide the underlying room layout as well as render various scene attributes , e.g. , the lighting condition and color scheme . Very recent work 1Source code will be made available . Please see the demo video at this link . on interpreting GANs Bau et al . ( 2019 ) visualized that the internal filters at intermediate layers are specialized for generating some certain objects , but studying scene synthesis from object aspect only is far from fully understanding how GAN is able to compose a photo-realistic image , which contains multiple variation factors from layout level , category level , to attribute level . The original StyleGAN work ( Karras et al . ( 2018b ) ) pointed out that the layer-wise latent codes actually control the synthesis from coarse to fine , but how these variation factors are composed together and how to quantify such semantic information are still uncertain . Differently , this work gives a much deeper interpretation on the hierarchical generative representations in the sense that we match these layer-wise variation factors with human-understandable scene variations at multiple abstraction levels , including layout , category ( object ) , attribute , and color scheme . Starting with the state-of-the-art StyleGAN models ( Karras et al . ( 2018b ) ) as the example , we reveal that highly-structured semantic hierarchy emerges from the deep generative representations with layer-wise stochasticity trained for synthesizing scenes , even without any external supervision . Layerwise representations are first probed with a broad set of visual concepts at different abstraction levels . By quantifying the causality between the layer-wise activations and the semantics occurring in the output image , we are able to identify the most relevant variation factors across different layers of a GAN model with layer-wise latent codes : the early layers specify the spatial layout , the middle layers compose the category-guided objects , and the later layers render the attributes and color scheme of the entire scene . We further show that identifying such a set of manipulatable latent variation factors from layouts , objects , to scene attributes and color schemes facilitates the semantic image manipulation with large diversity . The proposed manipulation technique is further generalized to other GANs such as BigGAN ( Brock et al . ( 2019 ) ) and ProgressiveGAN ( Karras et al . ( 2018a ) ) . 1.1 RELATED WORK . Deep representations from classifying images . Many attempts have been made to study the internal representations of CNNs trained for classification tasks . Zhou et al . ( 2015 ) analyzed hidden units by simplifying the input image to see which context region gives the highest response , Simonyan et al . ( 2014 ) applied back-propagation technique to compute the image-specific class saliency map , Bau et al . ( 2017 ) interpreted the hidden representations via the aid of segmentation mask , Alain and Bengio ( 2016 ) trained independent linear probes to analyze the information separability among different layers . There are also some studies transferring the features of CNNs to verify how learned representations fit with different datasets or tasks ( Yosinski et al . ( 2014 ) ; Agrawal et al . ( 2014 ) ) . In addition , reversing the feature extraction process by mapping a given representation back to image space ( Zeiler and Fergus ( 2014 ) ; Nguyen et al . ( 2016 ) ; Mahendran and Vedaldi ( 2015 ) ) also gives insight into what CNNs actually learn to distinguish different categories . However , these interpretation techniques developed for classification networks can not be directly applied for generative models . Deep representations from synthesizing images . Generative Adversarial Networks ( GANs ) ( Goodfellow et al . ( 2014 ) ) advance the image synthesis significantly . Some recent models ( Karras et al . ( 2018a ) ; Brock et al . ( 2019 ) ; Karras et al . ( 2018b ) ) are able to generate photo-realistic faces , objects , and scenes , making GANs applicable to real-world image editing tasks , such as image manipulation ( Shen et al . ( 2018 ) ; Xiao et al . ( 2018a ) ; Wang et al . ( 2018 ) ; Yao et al . ( 2018 ) ) , image painting ( Bau et al . ( 2019 ) ; Park et al . ( 2019 ) ) , and image style transfer ( Zhu et al . ( 2017 ) ; Choi et al . ( 2018 ) ) . Despite such a great success , it remains uncertain what GANs have actually learned to produce such diverse and realistic images . Radford et al . ( 2016 ) pointed out the vector arithmetic phenomenon in the underlying latent space of GAN , however , discovering what kinds of semantics exist inside a well-trained model and how these semantics are structured to compose high-quality images are still unsolved . A very recent work ( Bau et al . ( 2019 ) ) analyzed the individual units of the generator in GAN and found that they learn to synthesize informative visual contents such as objects and textures spontaneously . Unlike Bau et al . ( 2019 ) which focuses on the intermediate filters , our work quantitatively explores the emergence of multi-level semantics inside the very early latent space . 2 VARIATION FACTORS FOR SCENE SYNTHESIS . 2.1 MULTI-LEVEL SCENE SEMANTICS . Imagine an artist drawing a picture of living room . The very first step , before drawing every single object , is to choose a perspective and set up the layout of the room . After the spatial structure is decided , the next step is to add objects that typically occur in a living room , such as sofa and TV . Finally , the artist will refine the details of the picture with specified decoration styles , e.g. , warm or cold , natural lighting or indoor lighting . The above process reflects how a human interprets a scene to draw it . As a comparison , generative models such as GANs follow a completely end-to-end training for synthesizing scenes , without any prior knowledge about the drawing techniques and relevant concepts . Even so , the trained GANs are able to produce photo-realistic scenes , which makes us wonder if the GANs have mastered any human-understandable drawing knowledge as well as the variation factors of scenes spontaneously . Therefore , in this work we aim at interpreting how GANs learn to synthesize a photo-realistic scene image from scratch . To align the synthesized scenes with human perception , we use off-the-shelf classifiers to extract semantics from the output image . As shown in Fig.1 , given a scene image , semantics at multiple abstraction levels are extracted , including layout , object ( category ) , and attribute . These concepts are treated as candidates and we propose a quantification technique in Sec.2.2 to identify which variation factor has been encoded into the well-learned generative representation . We surprisingly find that GAN synthesizes a scene in a manner highly consistent with human . Over the convolutional layers , GAN manages to compose these multi-level abstractions hierarchically . In particular , GAN constructs the spatial layout at early stage , synthesizes category-specified objects at middle stage , and renders the scene attribute ( e.g. , color scheme ) at later stage . We will describe the method we use to quantify the emergent variation factors as follows . 2.2 IDENTIFYING THE EMERGENT VARIATION FACTORS . Among the multi-level candidate concepts described in Sec.2.1 , not all of them are meaningful to a particular scene synthesis model . For instance , “ indoor lighting ” will never happen in outdoor scenes such as bridge and tower . Accordingly , we come up with a method to quantitatively identify the variation factors that emerge inside the learned generative representation . Fig.2 illustrates the identification process which consists of two steps , i.e. , probing and verification . Probing latent space . The generator of GAN , G ( · ) , typically learns the mapping from latent space Z to image space X . Latent vectors z ∈ Z can be considered as the generative representation learned by GAN . To study the emergence of variation factors inside Z , we need to first extract semantic information from z , which is not trivial . To solve this problem , we employ synthesized image , x = G ( z ) , as an intermediate step and use a broad set of off-the-shelf image classifiers to help assign semantic scores for each sampled latent code z . Taking “ indoor lighting ” as an example , the scene attribute classifier is able to output the probability on how an input image looks like having indoor lighting , which we use as semantic score . Recall that we divide scene representation into layout , category , and attribute levels , we introduce layout estimator , scene category recognizer , and attribute classifier to predict semantic scores from these abstraction levels respectively , forming a hierarchical semantic space S. After establishing the one-on-one mapping from latent space Z to sematic space S , we search the decision boundary for each concept by treating it as a bi-classification problem , as shown in Fig.2 . Here , taking “ indoor lighting ” as an instance , the boundary separates the latent space Z to two sets , i.e. , present or absent of indoor lighting . Verifying manipulatable variation factors . After probing the latent space with a broad set of candidate concepts , we still need to figure out which ones are most relevant to the generative model acting as the variation factors . The key issue is how to define “ relevance ” , or say , how to verify whether the learned representation has already encoded a particular variation factor . We argue that if the target concept is manipulatable from latent space perspective ( e.g. , change the indoor lighting status of the synthesized image via simply varying the latent code ) , the GAN model is able to capture such variation factors during the training process . As mentioned above , we have already got separation boundaries for each candidate . Let { ni } Ci=1 denote the normal vectors of these boundaries , where C is the total number of candidates . For a certain boundary , if we move a latent code z along its normal direction ( positive ) , the semantic score should also increase correspondingly . Therefore , we propose to re-score the varied latent code to quantify how a variation factor is relevant to the target model for analysis . As shown in Fig.2 , this process can be formulated as ∆si = 1 K K∑ k=1 max ( Fi ( G ( zk + λni ) ) − Fi ( G ( zk ) ) , 0 ) , ( 1 ) where 1K ∑K k=1 stands for the average of K samples to make the metric more accurate . λ is a fixed moving step . To make this metric comparable among all candidates , all normal vectors { ni } Ci=1 are normalized to fixed norm 1 and λ is set as 2 . With this re-scoring technique , we can easily rank the score ∆si among all C concepts to retrieve the most relevant latent variation factors .

Summary: The paper proposes an approach to analyze the latent space learned by recent GAN approaches into semantically meaningful directions of variation, thus allowing for interpretable manipulation of latent space vectors and subsequent generated images.  The approach is based on using pre-trained classifiers for semantic attributes of the images at a variety of levels, including indoor room layout, objects present, illumination (indoor lightining, outdoor lighting), etc. By forming a decision boundary in the latent space for each of these classifiers, the latent code is then manipulated along the boundary normal direction, and re-scored by the classifiers to determine the extent to which the boundary is coupled to the semantic attribute.