Semantic Hierarchy Emerges in the Deep Generative Representations for Scene Synthesis
Authors: Ceyuan Yang
Yujun Shen
Bolei Zhou
Venue:  (2020)
Keywords: Feature visualization
feature interpretation
generative models

Abstract: Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what networks have learned inside the deep generative representations and how photo-realistic images are able to be composed from random noises. In this work, we show that highly-structured semantic hierarchy emerges from the generative representations as the variation factors for synthesizing scenes. By probing the layer-wise representations with a broad set of visual concepts at different abstraction levels, we are able to quantify the causality between the activations and the semantics occurring in the output image. Such a quantification identifies the human-understandable variation factors learned by GANs to compose scenes. The qualitative and quantitative results suggest that the generative representations learned by GAN are specialized to synthesize different hierarchical semantics: the early layers tend to determine the spatial layout and configuration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme. Identifying such a set of manipulatable latent semantics facilitates semantic scene manipulation.

Under review as a conference paper at ICLR 2020
SEMANTIC HIERARCHY EMERGES IN DEEP GENERA- TIVE REPRESENTATIONS FOR SCENE S SYNTHESIS
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the success of Generative Adversarial Networks (GANs) in image syn- thesis, there lacks enough understanding on what networks have learned inside the deep generative representations and how photo-realistie images able be composed from random noises. In this work, we show that highly-structured semantic hierarchy emerges variation factors for synthesizing scenes from the generative representations in state-of-the-ar GAN models, like StyleGAN and BigGAN. By probing the layer-wise representations with broad set of semantics at different abstraction levels, we are able to quantify the causality between the activations and semantics occurring in the output image. Such a quantification identifies the human understandable variation factors learned by GANs to com- pose scenes. The qualitative and quantitative results suggest that the generative representations learned by the GANs with layer-wise latent codes are specialized to synthesize different hierarchical semanties: the early layers tend to determine the spatial layout and contiguration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme. Identifying such set of manipulatable latent variation factors facilitates semantic scene manipulation
INTRODUCTION
Success of deep neural networks stems from the representation learning. which identifies the explana- tory factors underlying the high-dimensional observed data (Bengio al. (2013). Prior work has shown that many concept detectors spontaneously emerge inside the deep representations trained for classification task. For example, Gonzalez- Garciaa et al. (2018) shows that networks for object recog- nition are able to detect semantic object parts, Bau et al. (2017) confirms that deep representations from classifying images learn to detect different categorical concepts at different layers.
Analyzing the deep representations and their emergent structures gives insight into the generalization ability of deep features (Morcos et al. (2018) as well as the feature transferability across different tasks (Yosinski et al. (2014)). But current efforts interpreting deep representations mainly focus on discriminative models (Zhou et al. 2013);;
Manzzllz--Gacica et al. (018)); Zeiler and Fergus 2014);
Agrawal et al. 2014); Bau al. (2017)) Recent advance of Generative Adversarial Networks (GANs) [Goodfellow et al. 2014); Karras et al. (2018a b); Brock et al. (2019) is capable of transforming random noises into high-quality images, however, the nature of the learned generative representations
and how a photo-realistic image is being composed over different layers of the generator in GAN remain much less explored.
It is known that the internal units of Convolutional Neural Networks (CNNs) emerge as object detectors when trained to categorize scenes (Zhou al. (011)) ). Representing and detecting infor- mative categorical objects provides an ideal solution for classifying scenes, such as sofa and TV are representative of living room while bed and lamp are bedroom. However, synthesizing scene demands far more knowledge for the generative models to learn. Specifically, in order to produce highly-diverse scene images, the deep representations might be required to not only generate every individual object relevant to specific scene category, but also decide the underlying room layout as well as render various scene attributes, e.g., the lighting condition and color scheme. Very recent work
Source code will be made available. Please the demo video this link

---

Under review as a conference paper at ICLR 2020
on interpreting GANs Bau et al. (2019) visualized that the internal filters at intermediate layers specialized for generating some certain objects, but studying scene synthesis from object aspect only is far from fully understanding how GAN is able to compose photo-realistic image, which contains multiple variation factors from layout level, category level, to attribute level. The original StyleGAN work (Karras et al. (2018b) pointed out that the layer-wise latent codes actually control the synthesis from coarse to fine, but how these variation factors are composed together and how to quantify such semantic information are still uncertain. Differently, this work gives much deeper interpretation on the hierarchical generative representations in the sense that we match these layer-wise variation factors with human- underssaaadat scene variations at multiple abstraction levels, including layout, category (object), altribule, and color scheme.
Starting with the state-of-the- StyleGAN models (Karras et al. (2018b)) as the example, we reveal that highly-strun structstt semantic hierarchy emerges from the deep generative representations with layer-wise stochasticity trained for synthesizing scenes, even without any external supervision. Layer- wise representations are first probed with a broad set of visual concepts at different abstraction levels. By quantifying the causality between the layer-wise activations and the semanties occurring in the output image, we are able to identify the most relevant variation factors across different layers of GAN model with layer-wise latent codes: the early layers specify the spatial layout, the middle layers compose the category-guided objects, and the later layers render the attributes and color scheme of the entire scene. We further show that identifying such a set of manipulatable latent variation factors from layouts, objects, to scene attributes and color schemes facilitates the semantic image manipulation with large diversity. The proposed manipulation technique is further generalized to other GANs such as BigGAN (Brock al. (2019) and ProgressiveGAN (Karras al. 20188a
1.1 RELATED WORK
Deep representations from classifying images. Many attempts have been made to study the internal representations of CNNs trained for classification tasks. [Zhou et al. (2015) analyzed hidden units by simplifying the input image to see which context region gives the highest response, Simonyaa iet al. (2014) applied back-propagation technique to compute the image-specific class saliency map. Bau et al. (2017) interpreted the hidden representations via the aid of segmentation mask. AAlai and Bengio (016)) trained independent linear probes to analyze the information separability among different layers. There are also some studies transferring the features of CNNs to verify how learned representations fit with different datasets or tasks (Yosinski et al. (2014); Agrawal et al. (0144). In addition, reversing the feature extraction process by mapping given representation back to image space (Zeiler and Fergus (2014); Nguyen et al. (2016); Mahendran and Vedaldi (2055) also gives insight into what CNNs actually learn to distinguish different categories. However, these interpretation techniques developed for classification networks cannot be directly applied for generative models.
Deep representations from synthesizing images. Generative Adversarial Networks (GANs) (Good. fellow et al.] (2014) advance the image synthesis significantly. Some recent models (Karras et al. (2018a): Brock et al.) (2019); Karras et al. (2018b)) are able to generate photo-realistic faces, objects, and scenes, making GANs applicable to real-world image editing tasks, such as image manipulation (Shen et al. (2018 8); Xiao et al. (2018a); Wang et al. (2018); Yao et al. (2118). image painting (Bau ctt al. (2019 9 ; Park et al. (2019)) and image style transfer (Zhu et al. (2017); Choi et al. (2018)). Despite such great success, it remains uncertain what GANs have actually learned to produce such diverse and realistic images. Radford et al. (2016) pointed out the vector arithmetic phenomenon in the underlying latent space of GAN, however, discovering what kinds of semanties exist inside a well-trained model and how these semantics are structured to compose high-quality images are still unsolved. A very recent work (Bau al. (2019)) analyzed the individual units of the generator in GAN and found that they learn to synthesize informative visual contents such as objects and textures spontaneously. Unlike Bau et al. (2019) which focuses on the intermediate filters, our work quantitatively explores the emergence of multi-level semantics inside the very early latent space.
2
VARIATION FACTORS FOR SCENE SYNTHESIS
2.1 MULTI-LEVEL SCENE SEMANTICS
Imagine an artist drawing a picture of living room. The very first step, before drawing every single object, is to choose perspective and set up the layout of the room. After the spatial structure is
2

---

Under review as a conference paper at ICLR 2020
Synthesis
Layout
Objects
Synthesis
Layout
Objects
uttributes: nature lighting
Scene attributes: enclosed area eluttered space socializzin
wood
folinge
soothing
no barizen
Scene category: bedroom
Scene cutegory: living
Figure 1: Multiple levels of semantics extracted from two synthesized scenes.
Indoor lighting
Probing
Verification
FO
2.74
0.32
Sp
Sereetic Spoces
Latrrt Spoce
F(G + 2n))
F(G&)
Figure 2: Method for identifying the emergent variation factors in generative representation. By deploying broad set of off-the-shelf image classifiers as scoring functions, F(.), we are able to assign a synthesized image with semantic scores corresponding to cach candidate variation factor. For particular concept, we learn a decision boundary in the latent space considering as binary classification task. Then move the sampled latent code towards the boundary quantitatively verify the emergence of the target concept.
how the semantic varies in the synthesis, and re-scoring technique
decided, the next step is to add objects that typically occur in a living room, such as sofa and TV. Finally, the artist will refine the details of the picture with specified decoration styles, e.g., warm or cold, natural lighting or indoor lighting. The above process reflects how human interprets a scene to draw it. As comparison, generative models such GANs follow completely end-to-end training for synthesizing scenes, without any prior knowledge about the drawing techniques and relevant concepts. Even so, the trained GANs are able produce photo-realistic scenes, which makes us wonder if the GANs have mastered any human-understandable drawing knowledge as well as the variation factors of scenes spontaneously.
Therefore, in this work we aim at interpreting how GANs learn to synthesize photo-realistic scene image from scratch. To align the synthesized scenes with human perception, we use off-the-shelf classifiers to extract semantics from the output image. As shown in Fig[]] given a scene image, semanties at multiple abstraction levels are extracted, including layout, object (category). and attribute. These concepts are treated as candidates and we propose quantification technique in Sec 2ec identify which variation factor has been encoded into the well-learned generative representation. We surprisingly find that GAN synthesizes scene in a manner highly consistent with human. Over the convolutional layers, GAN manages to compose these multi-level abstractions hicrarchically. In particular, GAN constructs the spatial layout early stage, synthesizes category-speciffe objects at middle stage, and renders the scene attribute (e.g., color scheme) at later stage. We will describe the method to quantify the emergent variation factors as follows.
2.2 IDENTIFYING THE EMERGENT VARIATION FACTORS
Among the multi-level candidate concepts described in Sec 2.1 not all of them are meaningful to particular scene synthesis model. For instance, "indoor lighting" will never happen in outdoor scenes such as bridge and tower. Accordingly, we come up with method to quantitatively identify the variation factors that emerge inside the learned generative representation. Fig 2 illustrates the identification process which consists of two steps, i.e., probing and verification.
Probing latent space. The generator GAN, G(-), typically learns the mapping from latent space Z to image space X. Latent vectors z E Z can be considered as the generative representation learned by GAN. To study the emergence of variation factors inside 2, we need first extract semantic information from z, which is not trivial. To solve this problem, we employ synthesized image, x =Gzz,, as an intermediate step and use broad set of off-the-shelf image classifiers to help assign semantic scores for each sampled latent code z. Taking "indoor lighting" attribute classifier able to output the probability on how an input image looks like having indoor lighting, which
example, the scene
semantic score. Recall that we divide category, and attribute levels, we introduce layout estimator, scene category recognizer, and attribute classifier to predict semantic scores from these abstraction levels respectively, forming hierarchical
representation into layout,
3

---

Under review as a conference paper at ICLR 2020
semantic space S. After establishing the one-on-one mapping from latent space 2 sematic space S. we search the decision boundary for each concept by treating it as a bi-classification problem, as shown in Fig[2 Here, taking "indoor lighting" an instance, the boundary separates the latent space 2 to two sets, Le., present or absent of indoor lighting.
Verifying manipulatable variation factors. After probing the latent space with broad set of candidate concepts, we still need to figure out which ones are most relevant to the generative model acting as the variation factors. The key issue how define "relevance", or say, how verify whether the learned representation has already encoded particular variation factor. We argue that the target concept is manipulatable from latent space perspective (e.g.. change the indoor lighting status of the synthesized image via simply varying the latent code), the GAN model is able to capture such variation factors during the training process.
As mentioned above, we have already got separation boundaries for each candidate. Let (n; ) denote the normal vectors of these boundaries, where C' is the total number of candidates. For a certain boundary, if we move a latent code z along its normal direction (positive), the semantic score should also increase correspondingly. Therefore, we propose to re-score the varied latent code to quantify how a variation factor is relevant to the target model for analysis. As shown in Fig 2 this process can be formulated as
K Σ max (F((G(z* + + F(((()))))))
Ast
(1)
K
where i Σx=1 stands for the average of K samples to make the metric more accurate. À is fixed moving step. To make this metric comparable among all candidates, all normal vectors [n:) are normalized to fixed norm 1 and À is set as 2. With this re-scoring technique, easily rank the score As; among all C' concepts to retrieve the most relevant latent variation factors.
EXPERIMENTAL RESULTS
In the generation process, the deep representation each layer (especially for StyleGAN and BigGAN) is actually directly derived from the projected latent code. Therefore, we consider the latent code as the "generative representation", which may be slightly different from the conventional definition in the classification networks. We conduct a detailed empirical analysis on the variation factors identified across the layers of the generators in GANs. We show that the hierarchy of variation factors emerges in the deep generative representations as result of synthesizing scenes, Sec 3. contains the layer-wise analysis on the statc-of-the-a art StyleGAN model (Karras et al. (2018b)), quantitatively and qualitatively verifying that the multi-level variation factors are encoded in the latent space. In Sec we explore the question on how GANs represent categorical information such as bedroom v.s. living room. We reveal that GAN synthesizes the shared objects some intermediate layers. By controlling their activations only, we can easily overwrite the category of the output image, e.g. turning bedroom into living room, while preserve its original layout and high-level attributes such as indoor lighting. 3.3 further shows that our approach can faithfully identify the most relevant attributes associated with particular scene, facilitating semantic scene manipulation.
Experimental Setting. The main experiment conducted on StyleGAN (Karras et al. (2018b)) but also extend our analysis PGGAN (Karras et al. (2018a) and BigGAN (Brock et al. (2019)) ) Most models are trained to synthesize scene images within a particular scene category, but we also train mixed StyleGAN model on a collection of images including bedroom, living room, and dining room to better understand how GAN encodes the categorical information and their associated objects. We off-the-shelf image classifiers to assign synthesized scenes with semantic scores, including a layout estimator (Zhang et al. (2019)) a scene category recognizer (Zhou et al. (2017)), and an attribute classifier (Zhou al. (2017)). We further extract color scheme of scene image through its hue histogram in HSV space. More details of the GAN models, the image classifiers, and the semantic boundary search process can be found in Appendix.
3.1 EMERGING SEMANTIC HIERARCHY
Humans typically interpret a scene in a hierarchy of semantics, from its layout, underlying objects, to the detailed attributes and the color scheme. This section will show that GAN composes scene over
4

---

Under review as a conference paper at ICLR 2020
Layout
Category
Attribute
Color Scheme
Bottom
Lower
Upper
Top
OE
(4)
Layer Index
13
Aftribate Indeor
Bottom
Upper
Top
Layout
95%
5%
0%
0%
Category
10%
90%
0%
0%
Schemne
Aftribute
0%
85%
5%
User
Figure 3: (a) Four levels of visual abstractions emerge at different layers of StyleGAN, Vertical axis shows the normalized perturbation score As; (b) User study on different layers correspond variation factors from different abstraction levels. (c) Layer-win wise manipulation result. The first column is the original synthesized images, and the other columns are the manipulated images at layers from four different stages respectively. Blue boxes highlight the results from varying the latent code at the most proper layers for the target concept.
Glossy
Wood
Cluttered space
Layout
Category
Indoor lighting
Joint
RR
B
Figure 4: (a) Independent attribute manipulation results high layers. The middle row the source images. We able both decrease (top row) and increase (bottom row) the variation factors the images. (b) Joint manipulation results, where the layour is manipulated at carly layers, the categorical objects are manipulated middle layers, while indoor lighting attribute is manipulated at later layers. The first column indicates the source images, the middle three columns are the independently manipulated images.
the layers in a similar way with human perception. To enable analysis on layout and category, we take the mixed StyleGAN model trained indoor scenes the target model. StyleGAN (Karras) ct al. (2018b)) learns a more disentangled latent space W on top of the conventional latent space Z. Besides, StyleGAN feeds the latent code transformations instead of only feeding it to the first layer. Specifically, for é-th layer, w is linearly transformed to layer-wise transformed latent code y(i) with y(!) A (0 w + b(0), where A (4) (()) the weight and bias for style transformation respectively. We thus perform layer-wise analysis by studying (f) instead of in Eq. ().
W each convolutional layer with different
To quantify the importance each layer with respect to each variation factor, we use the re-scoring technique identify the causality between the layer-wise generative representation y()) and the semantic emergence. The normalized score in Fig(Baa) shows that the layers of the generator in GAN are specialized to compose semantics in a hicrarchical manner: the bottom layers determine the layout, the lower layers and upper layers control category- level and attribute-level variations respectively, while color scheme is mostly rendered at the top. This is consistent with human perception.
To visually inspect the identified variation factors, move latent vector along the boundaries at different layers to show how the synthesis varies correspondingly. For example, given boundary in regard room layout, vary the latent code towards the normal direction bottom, lower, upper,

---

Under review as a conference paper at ICLR 2020
Living Room
Bedroom
Dining Room
wall
curtain
wall
wall
windowpane
painting
windowpane
windowpane =
pillow
painting
table
sofa
chair
cushion
cushion
table
ceiling
ceiling
floor
ceiling
floor
floor
Figure 5: Objects are transformed by GAN to represent different scene categories. On the top shows that the object segmentation mask varies when manipulating living to bedroom, and further dining the bottom visualizes object mapping that appears during category transition, where pixels are counted only from object level instead of instance level. GAN is able learn shared objects as well as the transformation of objects with similar appearance when trained to synthesize scene images from more than one category.
On
and top layers respectively. Fig F shows the qualitative results for several concepts. We see that the emerged variation factors follow highly-structured semantic hierarchy, e.g., layout can be best controlled at early stage while color scheme can only be changed at final stage. Besides, varying latent code at the inappropriate layers may also change the image content, but the changing might be inconsistent with the desired output. For example, the second row, modulating the code bottom layers for category only leads random change in the scene viewpoint.
To better evaluate the manipulahility across layers, we conduct a user study. We first generate 500 samples and manipulate them with respect to several concepts different layers. For each concept, 20 users are asked to choose the most appropriate layers for manipulation. Fig shows the user study results, where most people think bottom layers best align with layout, lower layers control scene category, etc. This is consistent with our observations in Fig () and (c). suggests that hierarchical variation factors emerge inside the generative representation for synthesizing scenes. and that our re-scoring method indeed helps identify the variation factors from broad set of semantics.
Identifying the semantic hierarchy and the variation factors across layers facilitates semantic scene manipulation. We can simply push the latent code toward the boundary of the desired attribute at the appropriate layer. Fig (a) shows that rial of furniture (cloth to wood), or even the cleanliness (tidy to cluttered) respectively. Furthermore, we can jointly manipulate the hierarchical variation factors. In Fig we simultaneously change the room layout (rotating viewpoint) at early layers, scene category (converting bedroom to living room) at middle layers, and scene attribute (increasing indoor lighting) later layers.
change the decoration style (crude glossy), the mate-
3.2 WHAT MAKES A SCENE?
As mentioned above, GAN models for synthesizing scenes are capable of encoding hierarchical semanties inside the generative representation, i.e., from layout, category, to scene attribute and color scheme. One of the most noticeable properties is that the middle layers of GAN actually synthesize different objects for different scene categories. raises the question on what makes scene as living room rather than bedroom. Thus further dive into the encoding of categorical information in GANs, to quantify how GAN interprets a scene category as well as how the scene category is transformed from object perspective.
We employ the StyleGAN model trained on the mixture of bedroom, living room, and dining room, and then search the semantic boundary between each two categories. To extract the objects from

---

Under review as a conference paper at ICLR 2020
Bridge
10
Church
Tower
busking
dlou
lores
liligge
meta
Ariage
Uree
we eepeeiionn
Peeping
cloud
vaary
vemioa
repitation
Senae Bonno
hiid
17
Living
Kitchen
Restaurant
Weating
wradd
Indeoy lightig
gless
scoting
lightnn
Blaay
elattoee spore
caktig
remasi lighing
indoor Highted
duterad spnce
indove lightng
Wend
waggrigating
natiral ligning
Ferarion
marting
notuoal
Figure 6: Comparison of the top scene attributes identified in the generative representations learned by StyleGAN
models for synthesizing different scenes. Vertical axis shows the perturbation score Ast.
Sunny
Boating
I Far-away horizon
Bridge
Vigetation
Cloud
Sunny
Eating
Kitchen
Figure 7: Manipulation results on StyleGAN models trained for synthesizing different scenes. For each triple, top shows the target attribute, the first image is the source image, the other two images generated by increasing the manipulation magnitude. Please the demo video for continuous manipulation via this link
the synthesized images, we apply semantic segmentation model (Xiao et al. (018b)). which can segment 150 objects (tv, sofa, etc) and stuff (ceiling. floor, etc). Specifically, we first randomly synthesize 500 living room images, and then vary the corresponding latent codes towards the "living room-bedroom" boundary and "bedroom-din dining room" boundary in turn. We segment the images before and after manipulation to get the segmentation masks, shown in Fig After tracking label mapping for each pixel via the image coordinate during the manipulation process, we are able to compute the statistics on how objects are transformed along with category changing.
Fig 5 shows the objects mapping in the category transformation process. We can see that (I) When image is manipulated among different categories, most of the stuff classes (e.g., ceiling and floor) remain the same, but some objects are mapped into other classes. For example, sofa living room is mapped to pillow and bed in bedroom, and bed in bedroom is further mapped to table and chair in dining room. This phenomenon happens because sofa, bed, dining table and chair are distinguishable objects for living room, bedroom, and dining room respectively. (2) Some objects between different scene categories, and the GAN model is able to spot such property and learn to generate these shared objects across different classes. For example, the lamp in living room (on the left boundary of the image) still remains after the image is converted to bedroom. (3) With the ability to learn object mapping as well as share objects across different classes, we are able to turn an unconditional GAN into GAN that can control category. Typically, to make GAN produce images from different categories, class labels have be fed into the generator to learn a categorical embedding, like BigGAN (Brock et al. (2019)). Our result suggests alternative approach.
sharable
3.3 DIVERSE ATTRIBUTE MANIPULATION
The emergence of variation factors for scene synthesis depends the training data. Here apply our method to a collection of StyleGAN models, to capture a wide range manipulatable attributes out of the 102 scene attributes use. Each styleGAN in the collection is trained to synthesize scene images from certain category, including both outdoor (bridge, church, tower) and indoor scenes (living room, kitchen).
7

---

Under review as a conference paper at ICLR 2020
Bedroom
Bridge
Living room
Layoul
natural lightip
symmettrial
boick
no hurizon
martle
Staney vegitasioo
nstural lightng warm
woathing oe toanting
Indaer lightii nstural lightig
weed Elway
sociataing Saainng
Siming
connecting tin hoino
Werking
Figure 8: Effects scene attributes (already sorted) when varying particular variation factor (in red color). Vertical axis shows the perturbation score As, in log scale.
Lower
Upper
All
Attribute
Wood
VegetatSinn
(1)
(b)
Indoor lighting
Cloud
Layer Index
Figure 9: (a) Some variation factors identified from PGGAN (bedroom). (b) Layer-wise analysis on BigGAN
from attribute level.
Fig 6 shows the top-10 relevant semantics to each model. We can that "sunny" has high scores on all outdoor categories, while "lighting" has high scores on all indoor categories. Furthermore, "boating" is identified for bridge model, "touring" for church and tower, "reading" for living room, "eating" for kitchen, and "socializing" for restaurant. These results are highly consistent with human perception, suggesting the effectiveness of the proposed quantification method. Fig 7 7 further shows manipulation results with respect to the scene attributes identified by our approach. We realistically manipulate the synthesized image with desired semantics. More results can be found in Appendix.
DISCUSSION AND CONCLUSION
Disentanglement of Semantics. Some variation factors we detect in the generative representation are more disentangled with each other than other semantics. Compared to the perceptual path length and linear separability described in Karras et al. (2018b) and the cosine similarity proposed in Shen et al. (2019), our work offers a new metric for disentanglement analysis. In particular, we move the latent code along one semantic direction and then check how the semantic scores of other factors change accordingly. As shown in Fig a]a, when we modify the spatial layout, all scene attributes are barely affected, suggesting that GAN learns to disentangle layout-level semantic from attribute-level. However, there are also some scene attributes (from same abstraction level) entangling with each other. Taking Fig 8 c) as an example, when modulating "indoor lighting", "natural lighting" also varies. This is also aligned with human perception, further demonstrating the effectiveness of our proposed quantification metric.
Application to Other GANs. We further apply our method for two other GAN structures, i.e., PGGAN (Karras et al.] (2018a) and BigGAN (Brock et al. (2019). These two models are trained on L.UN dataset Yuu al. (2015)) and Places dataset (Zhou et al. (2017) respectively. Compared to StyleGAN, PGGAN feeds the latent vector only to the very first convolutional layer and hence does not support layer-wise analysis. But the proposed re-scoring method can still be applied to help identify manipulatable semantics, as shown in Fig aa). BigGAN is the state-of-the-a conditional GAN model that concatenates the latent vector with class-guided embedding code before feeding to the generator, and it also allows layer-wise analysis like StyleGAN. Fig (b) gives analysis results on BigGAN from attribute level, where we can tell that scene attribute can be best modified at upper layers compared to lower layers or all layers. Meanwhile, the quantitative curve shows consistent result with the discovery on StyleGAN as F).. These results demonstrate the generalization ability of our approach as well as the emergence of manipulatable factors in other GANs.
In this paper, we show the emergence of highly-structured variation factors inside the deep generative representations learned by GANs with layer-wise stochasticity. In particular, the GAN model spontaneously learns to set up layout at early layers, generate categorical objects at middle layers, and render scene attribute and color scheme at later layers when trained to synthesize scenes. A re- scoring method is proposed to quantitatively identify the manipulatable semantic concepts within well-trained model, enabling photo-realistic scene manipulation.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
P. Agrawal, R. Girshick, and J. Malik. Analyzing the performance of multilayer neural networks for object recognition. In Proc. ECCV, 2014.
G. Alain and Y. Bengio. arXiv: 1610.01644, 2016.
Understanding intermediate layers using linear classifier probes.
D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. Proc. CVPR, 2017.
D. Bau, J.-Y. Zhu, H. Strobelt, B. Zhou, J. B. Tenenbaum, W. T. Freeman, and A. Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. In International Conference on Learning Representations, 2019.
Y. Bengio, A. Courville, and P. Vincent. Representation learning; A review and new perspectives. In IEEE Trans. on Pattern Analysis and Machine Intelligence, 2013.
A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo. Stargan: Unified generative adversarial networks for multi-domas image-to- tonggge translation. In Proc. CVPR, 2018.
A. Bonzalez-Garchhn D. Modolo, and V. Ferrari. Do semantic parts emerge convolutional neural networks? In International Journal of Computer Vision, 2018.
I. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
Karras, T. Aila, S. Laine, and Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference Learning Representations, 2018a.
T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv: 18/2.04924 2018b.
A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In Proc. CVPR, 2015.
A. S. Morcos, D. G. Barrett, N. C. Rabinowitz, and M. Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations, 2018.
A. Nguyen, A. Dosovitskiy, J. Yosinski. T. Brox, and J. Clune. Synthesizing the preferred inputs for neurons neural networks via deep generator networks. In Advances in Neural Information Processing Systems, 2016.
T. Peuyen--Puuc,, C. Li, L. Theis, C. Richardt, and Y.-L. Yang. Hologan: Unsupervised learning of 3d representations from natural images. In Proc. ICCV, 2019.
T. Park, M.-Y. Liu, T.C. Wang, and J.-Y. Zhu. Semantic image synthesis with spatially-adattive normalization. arXiv preprint arXiv:1903.00 07.991 2019.
A. Radford, Metz, and Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference Learning Representations, 2016.
T. R. Shaham, T. Dekel, and T. Michaeli. Singan: Learning generative model from single natural image. In Proc. ICCV, 2019.
Y. Shen, P. Luo, J. Yan, X. Wang, and X. Tang. Faceid-gan, Learning a symmetry three-player gan for
face synthesis. In Proc. CVPR, 2018.

---

Under review as a conference paper at ICLR 2020
Y. Shen, J. Gu, X. Tang, and B. Zhou. Interpreting the latent space of gans for semantic face editing. arXiv preprint arXiv 10786, 2019.
K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks; Visualising image classification models and saliency maps. In International Conference on Learning Representations Workshop, 2014.
T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proc. CVPR, 2018.
Xiao, J. Hong, and J. Ma. Elegant: Exchanging latent encodings with gan for transferring multiple face attributes. In Proc. ECCV, 2018a.
T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun. Unified perceptual parsing for scene understanding. In Proc. ECCV, 2018b.
S. Yao, T. M. Hsu, J.-Y. Zhu, J. Wu, A. Torralba, B. Freeman, and J. Tenenbaum. 3d-aware scene manipulation via inverse graphics. Advances in Neural Information Processing Systems, 2018.
Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, 2014.
F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1XXiv:506.033600 2015.
M. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Proc. ECCV, 2014.
W. Zhang, W. Zhang, and J. Gu. Edge-semantic learning strategy for layout estimation in indoor environment. In IEEE Transactions on Cybernetics, 2019.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene cnns. In International Conference on Learning Representations, 2015.
B. Zhou, A. Lapedriza, A. Khosla, Oliva, and Torralba. Places: 10 million image database for scene recognition. In IEEE Trans. on Pattern Analysis and Machine Intelligence, 2017.
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In Proc. ICCV, 2017.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
Sec A introduces the implementation details, including the GAN models used in this work, the off-the-shelf classifiers used for semantic score prediction, and the process of semantic identification. Sec [ contains the ablation study for showing why the proposed re-scoring technique is essential for identifying variation factors GAN. Sec[Cdiscusses the limitation of our method as well as some future directions. Dec|Dcontain more semantic scene manipulation results for wide range of scene categories and concepts. See ReeEEsh shows the details model structures of StyleGAN and BigGAN, both of which employ layer-wiss latent codes. provides the ablation study on layer-witd wise manipulation from different abstraction levels.
A IMPLEMENTATION DETAILS
A.1 GAN MODELS
We conduct experiments on three state-of-the-art generative models, including PGGAN Karras et al. (2018a), StyleGAN (Karras et al. (2018b)), and BigGAN (Brock et al. (2199). ). Among them, PGGAN and StyleGAN are trained on LSUN dataset (Yu et al. (2015)) while BigGAN is trained on Places dataset (Zhou al. (017)). LSUN dataset consists of indoor scene categories and 3 outdoor scene categories, and Places dataset contains 10 million images across 434 categories.
For PGGAN model, we use the officially released model each of which trained synthesize scene within a particular category of LSUN dataset. For StyleGAN, only one model related to scene synthesis (i.e., bedroom) is released For a more thorough analysis, we use the official implementation to train some additional models on other scene categories, including both indoor scenes (living room, kitchen, restaurant) and out door scenes (bridge, church, tower). We also train a mixed model the combination of images from bedroom, living room, and dining room with same implementation. This model specifically used for categorical analysis. For each StyleGAN model, Tab[[ shows the category, the number of training samples, well as the corresponding Fréchet inception distances (FID) (Heusel et al.) (2017)) which can reflect the synthesis quality to some extent. For BigGAN, a conditional generative model by taking category label as constraint. The resolution of the scene images synthesized by all of the above models is 256 256.
the author's officially unofficial PyTorch BigGAN implementation to train
Table I: Description of the StyleGAN models trained on different categories.
Scene Category
Indoor Outdoor Training Samples FID (lower is better)
bedroom (official)
Indoor
3M
2.65
living room
Indoor
1.3M
5.16
kitchen restaurant
Indoor
1M
5.06
Indoor
626K
4.03
bridge
Outdoor
819K
6.42
church
Outdoor
126K
4.82
tower
Outdoor
708K
5.99
Mixed
Indoor
500K each
3.74
A.2 SEMANTIC CLASSIFIERS
To extract semantic from synthesized images, we employ some off-the-shelf image classifiers to assign these images with semantic scores from multiple abstraction levels, including layout. category, scene altribute, and color scheme. Specifically, we use (1) a layout estimator (Zhang et al. (2019)). which is able to predict the spatial structure of a indoor place, (2) scene category classifier (Zhou
ct al. (2017), which is able to classify scene image to 365 categories, and (3) an attribute predictor
"These PGGAN models can be found at htt.ss//arrve...ggg..c..///////
"The StyleGAN model can he found at
*The implementation of StyleGAN can be found at http:p///rriieegggggooo////////a
IMA3GyN5m0voPcxx 9korsgobhwvPups7
'The implementation of BigGAN can he found at
htcps://gtthibbcom/ajj.oo.
Bi Pynnoch
11

---

Under review as a conference paper at ICLR 2020
Nugudive
Poslive
Nogatie
Pasitive
Figure 10: The definition of layout for indoor scenes. Green lines represent for the outline prediction from the layout estimator. The dashed line indicates the horizontal center, and the red point the center point of intersection line of two walls. The relative position between the vertical line and the center point is used to split the dataset. For example, image the left is treated as positive sample, while the one on the right is treated as negative sample.
Layout
Calegory
Chuttered space
Cloud
Swnny
Foliage
Color Scheme
Positive Samples
Negative
Samples
Figure 11: Samples for training decision boundary with respect layout, scene category, and various scene attributes.
(Zhou et al. (217)), which is capable of predicting 102 pre-defined scene attributes (e.g., sunny and dirty). We also extract color scheme of a scene image through its hue histogram in HSV space. Among them, the category classifier and attribute predictor can directly output the probability of how likely an image belongs certain category how likely an image has particular attribute. As for the layout estimator, it only detects the outline structure of indoor place, shown as the green line in To
A.3 SEMANTIC PROBING AND VERIFICATION
Given a well-trained GAN model for analysis, we first generate a collection of synthesized scene images by randomly sampling latent codes. To ensure capturing all the potential variation factors, set N = 500, 000. We then the aforementioned image classifiers to assign semantic scores for each visual concept. It is worth noting that we use the relative position between image horizontal center and the intersection line of walls to quantify layout, as shown in Fig TO After that, for each candidate, we select 2, 000 images with the highest response as positive samples, and another 0000 with the lowest response as negative ones. Fig 17 shows some examples, where living room and bedroom are treated as positive and negative for scene category respectively. We then train linear SVM by treating label is binary indicating whether the target semantic appears in the corresponding synthesis or not) to get a linear decision boundary. Finally, we re-generate K 1, 000 samples for semantic verification as described in Sec 2.2
bi-classification problem (i.e., data the sampled latent code while
B ABLATION STUDY ON RE-SCORING TECHNIQUE
Before performing the proposed re-scoring technique, we have two more steps, which are (1) assigning semantic scores for synthesized samples, and (2) training SVM classifiers to search semantic boundary. We would like to verify the essentiality of the re-scoring technique in identifying manipulatable semantics. We conduct ablation study on the StyleGAN model trained for synthesizing bedrooms. As shown in T2 the left figure sorts the scene attributes by how many samples labeled positive
12

---

Under review as a conference paper at ICLR 2020
Highest Perecstage
Accurncy SVM
Most Manipulotable
100%
Score
nstial Spliing
woou
en Joruing
distonal sper
Indorr
Figure 12: Ablation study the proposed re-scoring technique with StyleGAN model for bedroom synthesis.
ones, the middle figure sorts by the accuracy of the trained SVM classifiers, while the right figure sorts by our proposed quantification metric.
In left figure, "no horizon", "man-made", and "enclosed area" are attributes with highest percentage. However, all these three attributes are default properties of bedroom and thus not manipulatable. On the contrary, with the re-scoring technique for verification, our method successfully filters out these invariable candidates and reveals more meaningful semantics, like "wood" and "indoor lighting". In addition, our method also manages to identify some less frequent but actually manipulatable scene attributes, such as "cluttered space".
In middle figure, almost all attributes get similar scores, making them indistinguishable. Actually, even the worst SVM classifier (i.e., "railroad") achieves 72.3% accuracy. That is because even some variation factors are not encoded in the latent representation (or say, not manipulatable), the corresponding attribute classifier still assign synthesized images with different scores. Training SVM on these inaccurate data can also result in a separation boundary, even is not expected as the target concept. Therefore, only relying on the SVM classifier is not enough to detect relevant variation factors. By contrast, our method pays more attention to the score modulation after varying the latent code, which is not biased by the initial response of attribute classifier or the performance of SVM. As a result, we are able to thoroughly yet precisely detect the variation factors the latent space from a broad candidate set.
C LIMITATION AND FUTURE WORK
Despite the success of our proposed re-scoring technique in quantitatively identifying the hierarchi- cal manipulatable latent variation factors the deep generative representations, there are several limitations for future improvement.
First, the layout classifier can only detect the layout structure of indoor scenes. But for more general analysis on both indoor and outdoor scene categories, there lacks of unified definition of the spatial layout. For example, our framework cannot change the layout of outdoor church images. In future work, we will leverage the computational photography tools that recover the 3D camera pose of the image, thus we can extract more universal viewpoint representation for the synthesized images. Second, our proposed re-scoring technique relies on the performances of the off-the-shelf classifiers. For some of the attributes, the classifiers are not so accurate, which leads to poor manipulation boundary. This problem could be addressed with more powerful discriminative models. Third, for simplicity we only use the linear SVM for semantic boundary search. This limits our framework from interpreting the latent semantic subspace with more complex and nonlinear structure.
D MANIPULATION OF S SYNTHESIZED SCENES
Our proposed method can not only identify hierarchical variation factors from learned generative representation, but futher facilitate semantic scene manipulation. Fig 13 shows the manipulation results from layout level and category level. Fig and Fig 15 show the manipulation results from attribute level indoor scenes and outdoor scenes respectively. 1o the joint manipulation by modulating the latent code along the direction of desired semantics at the most appropriate layer. All experiments conducted on StyleGAN model.
13

---

Under review as a conference paper at ICLR 2020
Layout
Category
2
Fat
Tm
Figure 13: Layout and category manipulation results.
14

---

Under review as a conference paper at ICLR 2020
Kitchen
Glossy
Glass
Metal
Cluttered space
Living room
Glossy
Cluttered space
Wood
Indoor lighting
Restaurant
Wood
Indoor lighting
Eating
Glass
Figure 14: Manipulating the attributes of indoor scenes at different scores (low to high).
15

---

Under review as a conference paper at ICLR 2020
Bridge
Sunny
Boating
Vegetation
Cloud
EPF
Church
Sunny
Cloud
Vegetation
Vertical Part
Tower
Vegetation
Brick
Cloud
Sunny
Figure 15: Manipulating the attributes of outdoor seenes at different scores (low to high).
16

---

Under review as a conference paper at ICLR 2020
Layout
t
B
Lighting
t
=
M
Joint
Layout
Lighting
15
Joint
Category
25.
Lighting
Joint
Category
Lighting
Joint
Figure 16: Independent and joint manipulation results.
17

---

Under review as a conference paper at ICLR 2020
Latent 2 E
Noise
Typical archifocture
ResBlock in BigGAN
Syuthesis network
Nermaitee
4x4e5112
Mapping network
Eppit
BaidhNuum
Lanoa
Linear
+
LI
PC
ResBllok
Concall
Upaemple
Uppeepii
Condel
FC
Upareele
ResBlock
Concal
Ixt Conv
3id Corve
Cany Aus
Batchheem
Linoul
Non local
A
MAN
Cone Aes
Concal
A
MalN
8x3
Image
StyleGAN
BigGAN
Figure 17: Generator architectures of StyleGAN (Karras et al. 2018b)) and BigGAN (Brock et al. (2019) ). both of which introduces luser-wise stochasticity (i.e.. latent codes are fed into all convolutional layers instead of only the first layer). Note that the diagrams are horrowed from the original papers.
E GENERATOR STRUCTURES
This work conducts experiments on state-of of- the deep generative models for high-resolution scene synthesis, including StyleGAN (Karras et al. (2018b)), BigGAN (Brock et al. (2019)) and PGGAN (Karras et al. (2) Among them, PGGAN employs the conventional generator structure where the latent code is only fed into the very first layer. Differently, StyleGAN and BigGAN introduces layer-wise stochasticity by feeding latent codes to all convolutional layers as shown in 17 is worth mentioning that more and more latest GAN models inherit the design of using layer-wise latent codes to achieve better generation quality, such as SinGAN (Shaham et al. (2019) ) and HoloGAN ( Phuoc et al. (2019). ). And our layer-wisg wise analysis sheds light on why
effective.
In StyleGAN model that is trained to produce 256 256 scene images, there are totally 14 con- volutional layers. According to our experimental results, layout, category, attribute, color scheme correspond to bottom, lower, upper, and top layers respectively, which are actually [0, 2), [2, 6). [6, 12) and [12, 14) layers. As for BigGAN model with 256 256 resolution, there are total 12 convolutional layers. As the category information is already encoded in the "class" code as shown in Fig 17 we only separate the layers two groups, which are lower (bottom layers) and upper (top 6 layers). Since our layout model can only be applied to indoor scenes yet the BigGAN model trained on Places dataset which contains both indoor and outdoor categories, we only analyze BigGAN from attribute level shown in Fig b). Both visualization results and quantitative curve suggest that attribute-level semantics are better controlled by upper layers of BigGAN. For example, when manipulating "vegetation" attribute lower layers all layers, the spatial information varies unexpectedly, while manipulating at upper layers gives desired output.
18

---

Under review as a conference paper at ICLR 2020
Upper
Upper
All
Figure 18: Comparison results between manipulating latent codes at only upper cattribute-relevant layers and manipulating latent codes at all layers with respect to indoor lighting StyleGAN.
Layout
Category
Indoor lighting
Color Scheme
Figure 19: Manipulation at the bottom layers in 4 different directions, including layout, category, indoor lighting. and color scheme StyleGAN.
ABLATION STUDY ON LAYER-WISE MANIPULATION
To further validate the emergence of semantic hierarchy, we make ablation study on layer-wise manipulation with StyleGAN model.
First, we select "indoor lighting" as the target semantic, and vary the latent code only on upper (atribbute-reeevan) layers v.s. on all layers. We can easily tell from Fig that when manipulation "indoor lighting" all layers, the objects inside the room are also changed. By contrast, manipulating latent codes only at attribute-relevant layers can satisfyingly increase the indoor lighting without affecting other factors.
Second, we select bottom layers as the target layers, and select boundaries from all four abstraction levels for manipulation. As shown in Fig 19 no matter what level of semantics we choose, as long the latent code is modified at bottom (layout-relevant) layers, only layout instead of all other semantics varies.
These two experiments further verify our discovery about the emergence of semantic hierarchy.
19


Reviews:
[{"review_id": "rJegBk1SqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a visually-guided interpretation of activations of the convolution layers in the generator of StyleGAN on four semantic abstractions (Layout, Scene Category, Scene Attributes and Color), which are referred to as the \"Variation Factors\" and validates/corroborates these interpretations quantitatively using a re-scoring function. The claim of the paper is that there is a hierarchical encoding in the layers of the StyleGAN generator with respect to the aforementioned \"Variation Factors\".  Figure 3(a) illustrates how these \"Variation Factors\" emerge in the layers of the StyleGAN generator.\n\nThe basic GAN architecture used in this work is that of StyleGAN. However, details on the architecture of this particular GAN are missing, including in the Appendix. How many Convolution layers are present in its generator? Not everyone is aware of StyleGAN architecture -- A better illustration of their architecture in the main paper and its correspondence with the layer levels (bottom, lower, middle, top) is desired, mainly because the paper is built upon this. The dataset used to train the StyleGAN model is not clear either. In Appendix, Table 1 tabulates the training details, but nowhere is it clearly mentioned if the N=500,000 latent codes are sampled from a GAN model that was trained on a mixture of datasets (i.e., bedroom, living rooms, kitchen etc.) or individual datasets. As well, the unit of training time in Table 1 of the Appendix seems to be M; is it Million or Minutes? Both of them seem unrealistic units for training a GAN.\n\nSince the training dataset is not clear, my understanding of the method is that a range of datasets are used to produce the results, especially for the effect where the transition of Semantic Category results is studied. As a first step, StyleGAN model trained on \"bedroom\" scenes from LSUN dataset is used to randomly sample codes from the learned distribution, which are further passed through the generator to obtain the respective image mappings. Off-the-shelf image classifiers are employed on each of the images to classify them to one of the four \"Visual concepts\", which is nothing but the aforementioned four semantic abstractions. Here, I would like to encourage the authors to use a consistent terminology -- The four semantic abstractions have been referred to as \"Variation Factors\" (page 3), \"Candidate concepts\" (page 6), \"Visual concepts\"(page 12), \"Semantics\" (page 8) interchangeably throughout the literature, which is confusing.  Then, 2000 top positive examples and 2000 top negative examples identified by the image classifiers are used to train a linear SVM, i.e., a binary-SVM all the four scene abstractions (\"Varying Factors\"), and the separation boundary is obtained. I assume the separation boundary is only obtained once, and not after every layer of the generator. Otherwise, it would not make much sense. \nWith the separation boundary (in the form of a normal vector) known for each of the four scene semantics, different feature activations are obtained by moving the latent code towards/away from the separation boundary. A scoring function is obtained to quantify (Equation 1) how the corresponding images vary in a particular semantic aspect when the latent code is moved from the separation boundary.  As per the last line of Paragraph 2 on page 4, a ranking of such scores using this function is used to understand the most relevant latent semantics. Does this mean that initially, a large set of semantics is used to observe whether the output of GAN is manipulated by probing each of them? Or are only four scene semantics chosen to begin with? And what happens when there is a tie? And, what value of K makes this metric more accurate? Any lower bound? Please explain. More questions on the effect of lamda later below.\n\nIn the next step, the authors sample a latent code from the learned distribution and pass it through\nevery layer of the GAN generator. The output code y is varied along the boundary of the SVM classifier. This is repeated at every layer of the GAN generator and the same lamda is used to perturb the resulting output code from the separation boundary. The results are visualized in Fig 3(c). The claim here is that with the same perturbation of the resulting codes (lambda=2) at the output of different GAN layers, the change in the visualized output demonstrates what kind of, if any, semantic is being captured by different layers of GAN. This is also claimed to have been validated through the \"re-scoring\" function. I am not very clear on this. \nI request the following experiment:\n1) Within just a single layer (be it bottom, lower, middle or top), how does the output change when the output code of that layer is perturbed in all directions? This is to see the effect (by visualizing) of the range of lamda values on the output at all the layers. Do you discover any changes weakening your claim?\n2) I would like to see the visualizations of the latent codes at the separation boundaries, just to see how well the binary-SVM performs and whether or not, non-binary information is lost/unaccounted for.\n\nOn Page-5, see the fourth line from the bottom (going up): how do we know the desired output apriori? Are the four semantic abstractions decided based on the desired output? This takes us back to a question I asked earlier.\n\nMoreover, Layout variation is just view-point variation. So I think it will be appropriate to call it \"view-Point Variation\" rather than \"Layout Variation\". This is because Layout is associated with spatial arrangement of objects in a scene, with functionality goals. \nOne last question I have is: What is so special about StyleGAN that it was used as the guiding architecture in this work? How generalizable is this approach to other kinds of GANs other than PGGAN and BigGAN (or rather, why is this approach relatable to StyleGAN, PGGAN and BigGAN alone)? \n\nThe paper has grammatical errors  (sentences are not well written), typos (ex; \"manipulabe\" on page-8 which should be \"manipulatable\") and is not polished. I also suggest the authors to change the title of the paper, which right now, is a bit odd; if you decide to keep it, there should not be a \"the\" before \"Deep\" in the title.\n\nAll in all, the paper is interesting but lacks persuasiveness. \nI may jump my score if the authors address all the aforementioned questions and concerns convincingly, and work on the presentation.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1gTx_WOcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to analyze the latent space learned by recent GAN approaches into semantically meaningful directions of variation, thus allowing for interpretable manipulation of latent space vectors and subsequent generated images.  The approach is based on using pre-trained classifiers for semantic attributes of the images at a variety of levels, including indoor room layout, objects present, illumination (indoor lightining, outdoor lighting), etc. By forming a decision boundary in the latent space for each of these classifiers, the latent code is then manipulated along the boundary normal direction, and re-scored by the classifiers to determine the extent to which the boundary is coupled to the semantic attribute.\n\nBy taking advantage of the structured composition of the latent space into per-layer contributions in the StyleGAN approach, experiments are performed to show that different levels of semantics are captured at different layers: layout being localized in lower layers, object categories in middle layers, followed by other scene attribute, and lastly the color scheme of the image in the highest layers.  A user study shows that human judgments of the coupling between layers and semantic attribute being manipulated are consistent with this observation.  A set of qualitative experiments demonstrate manipulation along several axes.  Another set of experiments demonstrate that the importance of different semantic attribute dimensions for different scene categories varies in an interpretable way, and also that certain attribute dimensions influence each other strongly (e.g. \"indoor lighting\" and \"natural lighting\"), whereas other ones are decoupled (e.g. \"layout\" and other dimensions).\n\nI am somewhat positive with respect to acceptance of the paper. On the one hand, the key idea is simple, and has been demonstrated compellingly with a broad set of experiments.  On the other hand, the insight gained is fairly superficial, boiling down to the statement that the learned latent code has structure that corresponds to semantically meaningful axes of variation, and that such structure is localized to particular levels of the layer hierarchy for particular semantic axes.\n\nThere are a few small issues with the clarity of the paper that would be good to fix:\n- Fig 3a: the interpretation of the vertical axis here was not clearly described in the caption or the main text\n- Fig 4 caption: typo \"while lindoor\" -> \"while indoor\"\n- Fig 5: the construction of the pixel area flow visualization is not explained in the caption, and needs a bit more clarity in the main text (e.g., how are multiple instances of the same class handled?)\n- Fig 6: the caption could use a bit more explanation for making these plots interpretable: e.g. say what value the vertical axis is reporting\n- Fig 8: same issue as above\n- p8 typo: \"that contacts the latent vector\" -> \"that concatenates the latent vector\"\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BylIovgxcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Updates after author response:\nI'd like to thank the authors for their detailed responses. Some of my primary concerns were regarding the presentation, and I feel they have been mostly addressed with the changes to the introduction and abstract (I'd still recommend using 'layerwise latent code' instead of 'layerwise representation' everywhere in the text). The additional qualitative results showing the benefits of manipulating 'z' vs y_l were also helpful. Finally, I agree that given the popularity of StyleGAN like models, the investigation methodology proposed, and the insights presented might be useful to a broad audience. Overall, I am inclined to update my rating to lean towards acceptance.\n\n---------------------------\nThis paper investigates the aspects encoded by the latent variables input to different layers in StyleGAN (Karras et. al.), and demonstrates that these correspond to encoding different aspects of the scene across layers e.g. initial ones correspond to layout, final ones to lighting.\n\nThe ’StyleGAN’ work first-generates a per-layer latent code y_l (from a global latent variable w), and uses these in a generative model. This paper investigates which layer’s latent codes best explain certain variations in scenes. To formalize the notion of how a latent vector is causally related to a scene property, the approach here is to use an off-the-shelf classifier for the property, and a) find a linear decision boundary in the latent space, and b) quantifying whether changing the latent code indeed affects the predicted score.\n\nPositives:\n1. The analysis presented in the work is thorough and results interesting. The paper analyzes the relation of various scene properties w.r.t the latent variables across layers, and does convincingly show that aspects like layout, category, attribute etc, are related to different layers.\n\n2. The visual results depicting manipulation of specific properties of scenes by changing specific variables in the latent space, and the ones in Sec 3.2 studying transitions across scene types, are also impressive and interesting.\n\n3. The proposed way of measuring the ‘manipulability’ of an aspect of a scene w.r.t a latent variable is simple and elegant, thought I have some concerns regarding its general applicability (see below).\n\nDespite these positives, I am not sure about accepting the paper because I feel the investigation methods and the results are both very specific to a particular sort of GAN, and the writing (introduction, abstract, related work etc.) pitch the paper as being more general than it is, and claim the insights to be more applicable. More specifically:\n\n1) The text claims the approach ‘probes the layer-wise representations’. However, what is actually investigated is the layer-wise latent code (NOT ‘representation’ which is typically defined to mean the responses of filters/outputs of each layer). In fact, I do not think this work is directly applicable to probing ‘representations’ as the term is normally used because it may be too high-dimensional to infer meaningful linear decision boundaries, or directly manipulate it.\n \n2) All the initial text in the paper’s abstract, introduction etc. leads the reader to believe that the findings here are generally applicable e.g. the sentence “the generative representations learned by GAN are specialized to synthesize different hierarchical semantics” should actually be something like “the per-layer latent variables for StyleGAN affect different levels of scene semantics“. Independent of any other concerns, I would be hesitant to accept the paper with the current writing given the very general nature of assertions made despite experiments in far more specific settings.\n\n3) In Sec 4, this paper only shows some sample results other models e.g. BIGGAN, but no ’semantic hierarchy in deep generative representation’ is shown (not surprising given only a global latent code).  As the discussion also alludes to, I do not think this approach would yield any insights if a GAN does not have a multi-layered latent code.\n\n4) Finally, while the results obtained for StyleGAN do convincingly show the causal relations claimed, these results are essentially backing up the insights that led to the design of StyleGAN i.e. having a single-level latent variable capture all source of variation is sub-optimal.\n\n5) This is not a really weakness, but perhaps an ablation that may help. The results showing scene property manipulation e.g. in Fig 4 are obtained by varying a certain y_l, and it’d help to also show the results if the initial latent code w was modified directly (therefore affecting all layers!). It would be interesting to know if this adversely affects constancy of some aspects e.g. maybe objects also change in addition to layout.\n\nOverall, while the results are interesting, they are only in context of a specific GAN, and using an approach that is applicable to generative models having a multi-layer code. I feel the paper should also be written better to be more precise regarding the claims. While the rating here only allows me to give a ‘3’ as a weak reject, I am perhaps a bit more towards borderline (though leaning towards reject) than that indicates.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to study what information is encoded in different layers of StyleGAN.  The authors do so by training classifiers for different layers of latent codes and investigating whether changing the latent code changes the generated output in the expected fashion.

The paper received borderline reviews with two weak accepts and one weak reject.  Initially, the reviewers were more negative (with one reject, one weak reject, and one weak accept).  After the rebuttal, the authors addressed most of the reviewer questions/concerns.  

Overall, the reviewers thought the results were interesting and appreciated the care the authors took in their investigations.  The main concern of the reviewers is that the analysis is limited to only StyleGAN.  It would be more interesting and informative if the authors applied their methodology to different GANs.  Then they can analyze whether the methodology and findings holds for other types of GANs as well. R1 notes that given the wide interest in StyleGAN-like models, the work maybe of interest to the community despite the limited investigation.  The reviewers also point out the writing can be improved to be more precise.

The AC agrees that the paper is mostly well written and well presented.  However, there are limitations in what is achieved in the paper and it would be of limited interest to the community.  The AC recommends that the authors consider improving their work, potentially broadening their investigation to other GAN architectures, and resubmit to an appropriate venue.
Decision: Reject