RPGAN: random paths as a latent space for GAN interpretability
Authors: Andrey Voynov
Artem Babenko
Venue:  (2020)
Keywords: generative models
GAN
interpretability

Abstract: In this paper, we introduce Random Path Generative Adversarial Network (RPGAN) --- an alternative scheme of GANs that can serve as a tool for generative model analysis. While the latent space of a typical GAN consists of input vectors, randomly sampled from the standard Gaussian distribution, the latent space of RPGAN consists of random paths in a generator network. As we show, this design allows to associate different layers of the generator with different regions of the latent space, providing their natural interpretability. With experiments on standard benchmarks, we demonstrate that RPGAN reveals several interesting insights about roles that different layers play in the image generation process. Aside from interpretability, the RPGAN model also provides competitive generation quality and allows efficient incremental learning on new data.

Under review as a conference paper at ICLR 2020
RPGAN: RANDOM PATHS AS A LATENT SPACE FOR GAN INTERPRETABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we introduce Random Path Generative Adversarial Network (RP- GAN) an alternative design of GANs that can serve a tool for generative model analysis. While the latent space of a typical GAN consists of input vec- tors, randomly sampled from the standard Gaussian distribution, the latent space of RPGAN consists of random paths in generator network. As we show, this de- sign allows to understand factors of variation, captured by different generator lay- ers, providing their natural interpretability. With experiments standard bench- marks, we demonstrate that RPGAN reveals several interesting insights about the roles that different layers play in the image generation process. Aside from inter- pretability, the RPGAN model also provides competitive generation quality and data.
allows efficient incremental learning
INTRODUCTION
Nowadays, deep generative models are an active research direction in the machine learning com- munity. The dominant methods for generative modeling, such as Generative Adversarial Networks (GANs), are currently able to produce diverse photorealistic images (Brock et al.) 2019 Karras øt al. 2019). These methods are not only popular among academicians, but are also crucial com- ponent in a wide range of applications, including image editing (Isola et al., 2017 Zhu et al. 2017). super-resolution (Ledig et al. 2017). video generation (Wang et al.] 2018) and many others.
Along with practical importance, key benefit of accurate generative models is a more complete understanding of the internal structure of the data. Insights about the data generation process can result both in the development of new machine learning techniques well as advances in industrial applications. However, most srate-of-the-aee generative models employ deep multi-laya architec- tures, which difficult to interpret explain. While many works investigate interpretability of discriminative models (Zeiler & Fergus Simonyan et al. 2013 Mahendran & Vedaldi 2015).
only few (Chen al. 2016 Bau al. 2019 address the understanding of generative ones.
In this work, we propose the Random Path GAN (RPGAN) - an alternative design of GANs that allows natural interpretability of the generator network. In traditional GAN generators, the stochastic component that influences individual samples is noisy input vector, typically sampled from the standard Gaussian distribution. In contrast, RPGAN generators instead use stochastic routing during the forward pass as their source of stochasticity. In nutshell, the RPGAN generator contains several instances of the corresponding layer. For each sample, only one random instance of each layer is activated during generation. The training of the RPGAN can then be performed in the same adversarial manner as in traditional GANs. the sections below, we show how RPGAN allows to understand the factors of variation captured by the particular layer and reveals several interesting findings about the image generation process, e.g. that different layers are "responsible for" coloring objection location. As practical advantage, RPGANs can be efficiently updated to new data via the simple addition of new instances to the bucket, avoiding re- re-training the full model from scratch. Finally, observe that RPGANs allow the construction of generative models without nonlinearities, which can significantly speed up the generation process for fully-connected layers.
In summary, the main contributions of our paper the following:

---

Under review as a conference paper at ICLR 2020
We introduce RPGAN - GAN with alternative source of stochasticity, based random routing. While being close to traditional GANs in terms of generation quality, RPGAN allows natural interpretability and efficient model updates with new data.
With extensive experiments on standard benchmarks we reveal several insights about the image generation process. Many of our insights confirm and extend recent findings from Bau et al. (2019). Note, that our scheme is more general compared to the technique from RPGAN does not require labeled datasets or pretrained segmentation
Bau et al. 2019 models.
We open-source the PyTorch implementation of RPGAN with common generator architec- ture
The rest of this paper is organized as follows. In Section we review relevant ideas from prior art. The proposed Random Path GAN design is described in Section 3 and experimentally evaluated in Section Section 5 concludes the paper and discusses possible directions for future work.
RELATED WORK
In this section we briefly describe connections of RPGAN to existing ideas from prior works
Generative adversarial networks. GANs are currently one of the main paradigms in generative modelling. Since the seminal paper GANs by Goodfellow al. (2014), plethora of alternative loss functions, architectures, normalizations, and regularization techniques were developed (Kurach et al. 2019 Today, oftte-of-thh-art GANs able to produce high-fidelity images, often indis- tinguishable from real ones (Brock et al. 2019 Karras et al.] 2019). In essence, GANs consist of two networks a generator and discriminator, which are trained jointly in adversarial manner. In standard GANS, the generation stochasticity is provided by the input noise vector. In RPGANs, we propose an alternative source of stochasticity by using fixed input but random routes during forward pass in the generator.
Specific GAN architectures. Many prior works investigated different design choices for GANs, but to the best of our knowledge, none of them explicitly aimed to propose an interpretable GAN model. Hoang et al. (2018) proposed the use of several independent generators to address the mode collapse problem. Chavdarova & Fleuret (2018) employ several auxiliary local generators and dis- criminators to improve mode coverage as well. |Huang et al. (2017) use layer-wise generators and discriminators to enforce hidden representations produced by layers of the generator to be similar to the corresponding representations produced by a reversed classification network. Important differ- ences of RPGAN compared to the works described above is that it uses random routes as its latent space and does not enforce to mimic the latent representations of pretrained classifiers.
Interprefability. While interpretability of models based on deep neural networks is important research direction, most existing work addresses the interpretability of discriminative models. These works typically aim understand the internal representations of networks (Zeiler & Fergus 2014
Simonyan et al. 2013 Mahendran & Vedaldi 2015) Dosovitskiy & Brox 2016) or explain deci-
sions produced by the network for particular samples (Sundararajan al. [017]] Bach et al.) 2015
Simonyan et al. 2013).. However, only a few works address interpretability of generative models. A related work by [Bu et al. (2019) develops technique that allows to identify which parts of the generator are responsible for the generation of different objects. In contrast, we propose GANs with alternative source of stochasticity that allows natural interpretation by design. Some of our findings confirm the results from Bau et al. (2019). which provides stronger evidence about the responsibili- ties of different layers in the generation process. Note, that the technique (Bau et al., 2019) requires a pretrained segmentation network and cannot be directly applied to several benchmarks, e.g. CIFAR- 10 or MNIST. In contrast, RPGAN does not require any auxiliary models or supervision and can be applied to any data.
rwgan- ICI (CI.R220 RPGAN
2

---

Under review as a conference paper at ICLR 2020
B
22
Bu
B
B2
Ba
H21
Ral
By
BAF
Z
By2
B2
Bxt
D
Z
B22
B,a
...
...
...
...
B pm
Bam-
jaw
R/ec
Generator
Generator
(a) The generator consists buckets, each containing several blocks.
(b) Forward pass.
Figure 1: The RPGAN generator. During forward pass, only one block in each bucket activated.
RANDOM PATH GAN
3.1 MOTIVATION
Before the formal description, we provide an intuition behind the RPGAN model. Several prior works have demonstrated that in discriminative convolutional neural networks different layers "responsible" for different levels of abstraction (Zeiler & Fergus 2014 Babenko et al.] 2014). For instance, earlier layers aim detect small texture patterns, while activations in deeper layers typi- cally correspond to semantically meaningful concepts. Similarly, in our paper we aim to understand the roles that different GAN layers play image generation. Thus, we propose an architecture that provides a direct way to interpret the impact of individual layers. For given generator architecture, we construct several copies of each layer in its architecture. During the forward pass we randomly choose layer instance that will be used when generating particular image. Therefore, we can analyze the role of each RPGAN layer by visualizing how different instances that layer affect the generated image.
3.2 MODEL
Here we formally describe the structure of the RPGAN model. The model highly flexible to the choice of generator and discriminator architectures as well as to the loss function and learning strategy. Similarly to the standard GAN architectures, our model consists two networks - generator and discriminator. The RPGAN discriminator operates exactly like discriminators in common GANs, hence below we focus on the generator description.
Unlike existing GANs, the RPGAN generator always receives fixed input vector Z during for- ward pass and aims to produce an image from the real image distribution. The generator con- sists of several consequent buckets B,...., Bn, Each bucket is union of independent blocks: (B= Bim,). ). where each block is an arbitrary computational unit and m,= Bz]. A typical example of block is ResNet block (He al. (0055). convolutional layer with nonlinearity or any other (see Figure Tab layer type. In our experiments, all the units from the same bucket have the same architecture.
a block from the bucket B₁ produces an intermediate output tensor that passed block from the next bucket B++1- Typically we associate each bucket with a layer (or several layers) in the generator architecture, which we aim to interpret or analyze. A block from the first bucket B, always receives fixed input vector Z, which the same for different forward passes. The stochasticity of the generator arises from a random path that goes from Z to an output image, using only a single block from each bucket. Formally, during each forward pass, we randomly choose indices 81, 8n with < 8; < m.j The generator output is then computed as Bus., .... B2s2 o B1e (Z) (see Figure lb). Thus, the generator defines a map from the Cartesian product (m1) X (mz)
For each i=1,
take an arbitrary existing GAN model, group its generator layers into buckets and replicate them into multiple blocks. In these
(m,) to the image space. Note that
terms, the original model can be treated as the RPGAN model with single block in each bucket and
3

---

Under review as a conference paper at ICLR 2020
random imput noise. Note that during image generation perform the same number of operations as in the standard GAN generator.
By its design, RPGAN with buckets BI..... B,, and constant input Z is able to generate at most [B||x... x...
...... x B, different samples were [Bx] is the number of blocks in the bucket Bg. Nevertheless, this number typically much larger compared to the training set size. We argue that the probability space of random paths can serve as a latent space to generate high-quality images, as confirmed by the experiments below.
To guarantee that blocks in a particular bucket are different, we also add specific diversity term in the generator loss function. The motivation for this ferm is prevent blocks BRs, Bay from learning the same weights. Let W be the set of all parameters of the generator. For each parameter E W there is set of its instances {w(1), ,... w(mh)} } in the RPGAN model. Then we enforce the instances be different by the loss term
Block diversity loss.
w()
MSE
8w
Swr
WE W.
Here we also normalize by the standard deviation Sw of all parameters from different blocks that correspond to the same layer. This normalization effectively guarantees that all buckets contribute to the diversity term.
4 EXPERIMENTS
Architecture. In all the experiments in this section, normalization and the hinge loss (SN-ResNet) described in Miyato et al. (2018). The blocks in the first bucket are fully-connected layers, the blocks in the last bucket are convolutional layers and blocks in all other buckets are residual blocks with two convolutions and a skip connection. If not stated otherwise, all the buckets have the same number of blocks. Additional experiments with other architectures are provided in Appendix.
ResNet-like generators with spectral
Datasets. We performed experiments on CIFAR-10 (Krizhessky et 2009), LSUN-bedroodd Yu et al., 2015) and Anime Faces (Jin et al. 2017) datasets. For different datasets we use different numbers of discriminator steps per one generator step dsteps and different numbers of blocks in bucket Toockk We summarize the main parameters used for three datasets in Table [ In the last column we also report Coverage, which the ratio the latent space cardinality (which equals the number of buckets to the power Nblocks) to the dataset size. Intuitively, large coverage guarantees that RPGAN has a sufficiently rich latent space of generator routes to capture the reference dataset. In the experiments below, we demonstrate that even moderate coverage is sufficient to generate high-fidelity images (see the LSUN-bedroom dataset with coverage Rs 3.3).
Dataset CIFAR-10
Image size
Number of buckets
Tiblokks
dsteps
Batch size
Coverage
32 x 32
5
40
5
64
2048
Anime Faces
64 x 64
6
20
1
32
2970
LSUN-bedroom
128 x 128
7
10
1
16
~ 3.3
Table 1: The details of architectures and training protocols used for different datasets.
Training details. We use the Adam optimizer with learning rate equal to 0.25 x 10-3, 81 3, equal 0.5, 0.999 and train the model for 45 10* generator steps for CIFAR-10 and 25 10a generator steps for Anime Faces and LSUN-bedroom datasets. During training we also learn the unique input vector Z. We observed that a learnable Z slightly improves the final generation quality and stabilizes the learning process. During the training step, we pass Z through N independent random paths. Formally, let {xI,..., ZN be a batch of samples received from bucket Bg. pass this batch through the bucket B8+1 take random blocks B Bai and form new batch {Bku (æ1)...., Baiv (IN)). } In all the experiments, we use the same training protocols for both the RPGAN and the standard GAN of the generator architecture. Note, that despite larger number of learnable parameters, RPGAN does not require more data or training time to achieve the same quality, compared standard GANs.
4

---

Under review as a conference paper at ICLR 2020
2
B3
B4
Bs
B31
B2
Z
B:
R33
jou
Ban
"frozen" blacks
"Frozee" blocks
Figure 2: The five-bucket RPGAN model with "frozen" blocks in buckets 1, 2, 4 and 5. In this case the stochasticity comes only from the third bucket. Analyzing the generated images allows to identify factors of variation, influenced by the third bucket.
unfrozen
original
bucket 1
bucket 2
bucket 3
bucket 4
bucket 5
Figure 3: Top image: sample generated with fixed sequence blocks. Horizontal lines: samples generated by the same sequence of blocks in all buckets and one unfrozen bucket. In the selected bucket we choose ten arbitrary blocks to avoid excessively large figures. The samples produced in this way allow to interpret factors of variation, captured by different buckets.
4.1 LAYERS INTERPRETATION
In the first series of experiments we investigate the responsibility areas" of different generator lay- This can be performed with technique schematically presented Figure 2 The goal is to in- terpret the role of the third bucket B3 in a five-bucket generator. For all other buckets Bi B2, Ba, Bs fix arbitrary blocks, shown in blue Figure Then generate images corresponding routes that contain all the fixed blocks, with the stochasticity coming only from varying blocks inside the target bucket B3. By inspecting the distribution of the obtained images, we can understand what factors of variation are influenced by B3.
On Figure 3 plot the samples generated with only one "unfrozen" bucket for image from the CIFAR- 10 dataset. Thus, each row shows how the original generated image could change if differ- blocks from the corresponding bucket are used. Several observations from Figure 3 Figure g Figure and and Figure 15 are listed below. The first bucket typically does not influence coloring and mostly affects small objects' deformations. The intermediate buckets have the largest influence on semantics. The last two buckets mostly responsible for coloring and do not influence the content shape. In particular, on Figure the fourth layer widely varies color, while the fifth acts as general tone corrector. Note that these findings consistent with the insights revealed by Bau et al. 2019)
To confirm the observations quantitatively, we perform the following experiment. We define a metric dang that evaluates the similarity between two generated images. Different metrics are able to capture different variations (e.g. in terms of semantic, color histogram, etc.) and we describe two particular choices of ding below. Then we choose random route in the RPGAN generator and for each bucket

---

Under review as a conference paper at ICLR 2020
25
4.5
Red channe
Dreen
semantic diversity
color diversity
3.5
2.5
varying bucket
varying bucket
(a) Averaged relative inception impact
Averaged relative coloring impact
Figure 4: Layer diversity compared to the first layer on CIFAR-10.
Bi generate four images possible pairs, we take four random samples from each line of the table the Figure Then measure diversity w.r.t. dang captured by bucket B₁ as a ratio
$4 varying blocks in Bt. In other words, rather then taking all
Σ dime (8) 588
Di- I, ctingg
> ding
(1))
Intuitively, we compute the relative diversity with respect the first layer, which typically captures the smallest amount of variations in our experiments. We then average these ratios over 100 inde- pendent evaluations. High values of averaged ratio Di-,1, imply higher diversity of the bucket B, compared to the first bucket in terms of the metric dmmg- For dung we experimented with the following two metries, capturing semantics and color differences correspondingly.
Inspired by the well-known Fréchet Inception Score concept (Heusel et al. 2017). we consider the Euclidean distance between the outputs of the last layer of the pretrained Inception network Szegedy et al. 2016) for semantic distance evaluation. Namely, we define dsemantic (img1, imgz) as Iv3 immg)) |2 were Ivmg))g) is the Inception V3 model activations for image img.
Iv3 (imgi)
To measure differences in color, we take the Hellinger distance between the color histograms of gen- erated samples. Namely, for each color channel, we split the range [O,...: 255] into equal seg- ments and evaluate the discrete distribution defined by the frequencies the sample's 's pixel intensities
appear in a given segment. Then the Hellinger distance between two quantified color distributions
defined as
Σ
(VP.
We compute this metric for each color channel independently.
The average values of Di with the standard deviations are shown on Figure 4 It demonstrates that the semantic diversity the largest for the intermediate layers. On the contrary, the last buckets, which are closer to the output, do not influence semantics but have higher impact in terms of color. Note that the first layer always shows the smallest variability in terms of both semantics and colors. The last bucket seems to be responsible for color correction and color inversion and has lower pallet variability impact. Note, that the plots from Figure reaffirm the findings coming from Figure 3 Note that the similar empirical results also hold for other datasets (see figures appendix).
16 in
Overall, summarize the main findings common for CIFAR- LSUN and Anime Faces datasets as:
The earlier layers have smaller variability and seem to be responsible for the viewpoint and the position of the object on the image.

---

Under review as a conference paper at ICLR 2020
The semantic details of the image content mostly determined by the intermediate layers.
The last layers typically affect only coloring scheme and do not affect content semantics or image geometry.
Note, that these conclusions can differ for other datasets or other generator architectures. For in- stance, for the four-bucket generator and MNIST (Figure 6 left) or randomly colored MNIST ( (Fig- ure g left) the semantics are mostly determined by the first two buckets.
4.2 RPGAN STANDARD GAN
In this subsection, we argue that the interpretations of different layers, obtained with RPGAN, are also valid for the standard GAN generator of the same architecture. First, we demonstrate that both standard GAN and RPGAN trained under the same training protocol provide almost the same generation quality. As a standard evaluation measure, we use the Fréchet Inception Distance (FID) introduced in Heusel et al. (2017). We also compute precision-recall curve defined in (Sajaadi al. (2018). For evaluation on CIFAR-10 10, we use 50000 generated samples and the whole train dataset. We also take ten independently trained generators and report minimal and average FID values. See far FID comparison and Figure 11 in appendix for precision-recall, RPGAN and SN-ResNet perform with the same quality both in terms of FID and precision-recall curves.
model Five-bucket RPGAN 16.9
min FID
average FID
20.8
SN-ResNet
16.75
18.7
Table 2: FID values for CIFAR-10.
To confirm that the layers of the standard GAN generator can be interpreted in the same way as the corresponding layers of its RPGAN counterpart, perform the following experiment. We take standard SN-ResNet GAN, consisting of five layers associated with the correspondent buckets in RPGAN, and train it on CIFAR-10. Then for each layer, we add normal noise to its weights. In- tuitively, we expect that the noise injection in the particular layer would change generated samples in terms of characteristics, influenced by this layer. For instance, noise in the last two layers is ex- pected to harm coloring scheme, while noise in the intermediate layers is expected to bring maximal semantic damage. Several samples, produced by perturbed layers, are presented on Figure [5] The images support the intuition described above and confirm that RPGAN may serve an analysis tool for the corresponding generator model. Note, however, that injecting noise per se not sufficient for interpretability. The perturbed generators produce poor images, which are difficult analyze. Meanwhile, RPGAN always generates good-looking images, which allows to identify the factors of variation, corresponding to the particular layer. For instance, see Figure g for the colored MNIST dataset. Figure & (left) shows plausible images, generated by varying RPGAN blocks. In contrast, Figure 8 (right) demonstrates images from generators perturbed with small and large noise. For both noise magnitudes, these images are difficult to interpret. Of course, given the interpretations obtained via RPGAN, one can perceive similar patterns in the noisy generations, but noise injection alone is not sufficient for interpretability.
4.3 INCREMENTAL LEARNING WITH RPGAN
In the next experiment, we demonstrate that the RPGAN model is also a natural fit for the generative incremental learning task (see e.g., Wu et al. 2018)). Let us assume that the whole train dataset D is split into two disjoint subsets D = Dj U Dz. Suppose that initially we have no samples from D2 and train generative model to approximate a distribution defined by the subset Dj. Then, given additional samples from D2, we aim to solve incremental learning task to update the model with new data without re-training it from scratch. The RPGAN model allows solving this task naturally. First we train generator with buckets Bi...., B, to reproduce the subset D,- Once we want to extend the generator with samples from D2, we add several new blocks to the buckets that are responsible for the features that capture the difference between D2 and Dz. Then optimize the generator to reproduce both D, and D2 by training only the new blocks. Thus, instead of training a new generator from scratch, we exploit the pretrained blocks that are responsible for features, which

---

Under review as a conference paper at ICLR 2020
Noised layer
Noised layer
2
5
5
Figure 5: Images, produced by the standard SN-ResNet GAN with noise injection the parameters of different generator layers. First column: original images, produced without generator perturba- tion. Other columns: images produced by generators perturbed different layers.
bucket
O
4
2
5 /
2 0
0
5
4
9
bucket 2
S
5
S 0
5 4
6
3
3
9
2
bucket
5
5 5 5
2
7 7135
2 26/4 4
5945
bucket
5
5
5
5
Figure 6: Incremental learning with RPGAN. Left: variations captured by different buckets with generator trained on MNISTo-6. Center: samples picked from a generator trained on MNISTo-6: Right: samples picked from a generator tuned on MNIST with only ten new blocks training (see details in the main text).
are common for D1 and D2. To illustrate this scenario. we take partition of the MNIST hand- written digits dataset (LeCun) 1989) into two subsets MNIST and MNIST;-9 of digits from O and from to correspondingly, As for generator for MNIST we take a 4-bucket RPGAN model with number of blocks equal to 20, 20, 20, 8. Note that the last bucket is much thinner than the others, as it turns out to responsible for variations writing style, which does not change much across the dataset. Then train the generator the subset MNIST of first 7 digits (see Figure 6 left and center). After that we add five additional blocks to each of the first two layers, obtaining generator with number of blocks equal to 25, 25, 20, 8 and pretrained weights in all blocks but five in the first and in the second buckets. Then we train the extended model to fit the whole MNIST by optimizing only the ten new blocks (see Figure 6 right).
4.4 PURELY LINEAR GENERATOR
As surprising side effect of our model, discovered that decent generation quality be achieved by the RPGAN generator with no nonlinearities, i.e., one can train the RPGAN gen- erator with all blocks consisting of linear transformations only. To demonstrate that, we take an RPGAN with the same ResNet-like generator architecture as in the experiments above. Then we replace all nonlinearities in the generator model by identity operations and train on the CIFAR-10 dataset. The model demonstrates FID equal to 22.79 that is competitive to the state-of-the-art gen- erative models of comparable sizes (see Figure 23 for generated images examples). Note that this approach fails for a standard GAN generator that maps Gaussian distribution to an images distri- bution. Indeed, that generator would be linear operator from latent space to the images space with Gaussian distribution in the images domain.
This purely linear generator architecture allows us to significantly speed up the image generation process for fully-connected layers. We group consequent buckets of fully-connected layers to form a new bucket. Blocks in the new bucket are linear transformations that are products of the blocks from the original buckets. To demonstrate this, train fully-connected generator network on the MNIST dataset (see Table in appendix, left column). Then we join the last three buckets into a single one. We form bucket by blocks defined the linear operators Bsg Baj B3, where
8

---

Under review as a conference paper at ICLR 2020
2.j, k are random indices of blocks from the buckets B3. B1. B, of the original generator. Thus instead of performing three multiplications features vector from the second layer by matrices of the shapes 256 512, 512 x 1024, 1024 x 784 perform single multiplication by a 256 784 matrix. In our experiments, we achieved x speed up. Note, however, that after the compression, the latent space cardinality
k) is used to populate the new bucket. Nevertheless, as random products of joining buckets are used, we expect that the gener- ated images would be uniformly distributed in the space of images, produced by the uncompressed generator (see Figure 24 for samples comparison).
decrease if small subset of tuples (i
5 CONCLUSION
In this paper, we address the interpretability of generative models. In particular, we have introduced RPGAN, alternative design of generative adversarial networks, which allows natural interpreta- tion of different generator layers via using random routing as source of stochasticity. With experi- ments on several datasets, we provide evidence that different layers are responsible for the different factors of variation in generated images, which is consistent with findings from previous work. As possible direction of future research, one can use the RPGAN analysis construct efficient models, e.g., via identification of redundant parts of the generator for pruning or inference speedup.
REFERENCES
Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. Neural codes for image retrieval. European conference on computer vision, pp. 584-599, Springer, 2014.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wist wise relevance propagation. PloS one, 10(7):e0130140, 2015.
David Bau, Jun- Zhu, Hendrik Strobelt, Zhou Bolei, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial net- works. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
https://cpenreet
net/ forum?id=Bi111 x
Tatjana Chavdarova and François Fleuret. Sgan: An alternative training of generative adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp- 2172-2180 2016.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in neural information processing systems, pp. 658 558-666, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, pp. 3769-5779, USA, 2017. Curran Asso- ciates Inc. ISBN 978-1-5108-6096-1 URL http: al. acm. org/ citation cfmiid
3395222 3295327
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770- 778, 2015.

---

Under review as a conference paper at ICLR 2020
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. MGAN: Training generative adversarial nets with multiple generators. In International Conference on Leurning Representations, 2018.
URL
net/
Xun Huang, Yixuan Li, Omid Poursaeed, John Hopcroft, and Serge Belongie. Stacked generative adversarial networks. In The IEEE Conference on Computer Vision and Patlern Recognition (CVPR), July 2017.
Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 1125-1132 2017.
Jin, J Zhang, M Li, Tian, and H Zhu. Towards the high-quality anime characters generation with generative adversarial networks. In Proceedings of the Machine Learning for Creativity and Design Workshop at NIPS, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Karol Kurach, Mario Lučić, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly, A large-scale study on regularization and normalization in gans. In International Conference on Machine Learning, 2019.
pp.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmist// 1989.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-4690, 2017.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188-5196, 2015.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018.
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lučié, Olivier Bousquet, and Sylvain Gelly. Assessing Generative Models via Precision and Recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 2013.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning- KcarningVVolume 70, pp. 3319- 3328. JMLR. org, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Ting- Chun-Chu Wang, Ming- Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao. Jan Kautz, and Bryan Catan- zaro. Video-to-video synthesis. arXiv preprint arXiv: 1808.06601, 2018.
10

---

Under review as a conference paper at ICLR 2020
50
FID
40
30
30
50
blocks in bucket
Figure 7: The FID values for RPGAN with different number of blocks.
Chenshen Wu, Luis Herranz, Xialei Liu, Joost van de Weijer, Bogdan Raducanu, et al. Memory replay gans: Learning to generate new categories without forgetting. In Advances In Neural Information Processing Systems, pp. 5962-5972, 2018.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506.03003333 2015.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 318-833, Springer, 2014.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pp. 2223-2223332 2017.
A ADDITIONAL FIGURES AND EXPERIMENTS
A.1 ABLATION ON NUMBER OF BLOCKS
Here investigate the impact of the number of blocks in each RPGAN bucket the generation quality. We train RPGAN with the SN-ResNet generator on CIFAR-10 10 with a different numbers of blocks in each bucket. For simplicity, all buckets have the same number of blocks. The resulting FID values are presented on Figure
If the number of blocks is too low, the resulting latent space appears have insufficient cardinality to cover the dataset. On the other hand, too high number of blocks results a difficult training procedure and also fails.
A.2 LSUN BEDROOM DATASET
To generate CUUNN-hdrroo-llice images, we use the 7-bucket RPGAN with ResNet-like generator with five residual blocks, the first fully-connected layer, and the last convolutional layer. Similarly to CIFAR-I0 experiments, during the generation, we freeze random path and vary blocks single bucket to investigate its responsibility. See Figure 12 for blocks variations and Figure 13 for buckets responsibility analysis. Note that similarly to CIFAR-10 10, the central buckets have a maximal semantic impact. Last two buckets mostly responsible for coloring. The first two buckets responsible for local geometrical features. Note that here we face mode collapse for the third bucket: mainly, it affects only tiny local features. See Figure for samples generated by the model.

---

Under review as a conference paper at ICLR 2020
8
8
bucket
3
2
6
8
4
3
8
6
2
8
3
3
bucket 2
9
4
2
4
O
O
8
X
8
8
Z
bucket 3
8
8
8
8
8
8
8
8
bucket
8
8
8
8
8
8
8
8
low variance
large noise variance
Figure 8: Left: images produced by varying blocks in particular bucket of RPGAN, Right: images produced by the standard GAN after parameters perturbation in a particular generator layer, with low and high normal noise variance.
A.3 ANIME FACES DATASET
Though this dataset is not standard for GANs, it in the experiments it nicely reveals the RPGAN analysis tools. Here we use the 6-bucket RPGAN with ResNet-like generator with four residual blocks, the first fully-connected layer, and the last convolutional layer. See Figure [3 for block variations and Figure 16 for bucket responsibility analysis. Again, the content semantics mostly defined by the intermediate buckets. The last two buckets are mostly responsible for coloring: the fifth bucket has the maximal impact coloring, and the last bucket varies tones. The first buckets are responsible for small details (one can note the hair on the character's forehead). See Figure 17for for samples generated by the model.
A.4 WASSERSTEIN GAN
Here we show that the concept of RPGAN works well with different generator architectures and learning strategies. Here present plots for DCGAN-like generators consisted of consequent convolutional layers without skip connections, All the models were trained with the same parameters described in Section Despite of Spectral Normalization, we train these models WGANs with weight penalty (Gulrajani et al. 2017). On the Figure 19 we show plots for a trained on CIFAR-IO. We also train bocr-bucket generator on colored MNIST, see Figure left.
generator
Finaly, we show plots for the five-bucket generator and CelebA-64ke dataset on Figure 21 See
Figure Figure 21
Figure 22 for buckets analysis.
B RPGAN INTERPRETABILITY VS WEIGHTS NOISING
In this section we show that injecting noise the generator weights cannot be used as a stand-alone interpretability method. Namely, we compare images produces by RPGAN and noise injection for models trained on randomly colored MNIST samples. We train both RPGAN and standard generators as a Wasserstein GAN with weights penalty.
12

---

Under review as a conference paper at ICLR 2020
bucket
bucket
bucket 3
bucket
bucket 5
Figure 9: Frozen paths individual blocks variation in 5-bucket RPGAN.
Figure 10: 32 x 32 samples generated by 5-bucket RPGAN.
13

---

Under review as a conference paper at ICLR 2020
1.0
0.8
Precision
0.6
0.4
0.2
RPGAN SN-ResNet
0.0
0.00
0.25
0.50
0.75
1.00
Recall
Figure 11: Comparison of precision-recall of RPGAN and its backbone SN- trained on CI- FAR10.
bucket
bucket 2
bucket 3
bucket 4
bucket
bucket
bucket 7
Figure 12: Frozen paths individual blocks variation in 7-bucket RPGAN.
14

---

Under review as a conference paper at ICLR 2020
Rid cherre
Bluc
semantic diversity
color diversity
1.5
0.5
varying bucket
varying bucket
(a) Averaged relative inception impact
(b) Averaged relative coloring impact
Figure 13: LSUN bedroom buckets specification. See Section 4.1 for details.
Figure 14: 128 128 samples generated by 7-bucket RPGAN
15

---

Under review as a conference paper at ICLR 2020
bucket 1
bucket 2
bucket
bucket 4
bucket 5
bucket
Figure 15: Frozen paths individual blocks variation 6-bucket RPGAN.
Green
20
semantic diversity
color diversity
15
10
2
4
2
4
varying bucket
varying bucket
(a) Averaged relative inception impact
(b) Averaged relative coloring impact
Figure 16: Anime Faces buckets specification. See Section 4.1 for details.
16

---

Under review as a conference paper at ICLR 2020
Figure 17: 64 x 64 samples generated by 6-bucket RPGAN.
13
cherne
2.5
Green
semantic divensity
1.1
color diversity
0.9
1.5
0.7
0.6
varying bucket
varying bucket
(a) Averaged relative inception impact
Averaged relative coloring impact
Figure 18: Buckets specification of RPGAN with DCGAN backbone trained on colored MNIST.
See Section 4.1 for details.
bucket
bucket 2
bucket 3
bucket 4
Figure 19: Frozen paths individual blocks variation in 4-bucket RPGAN with DCGAN backbone. Generated images size is 32 x 32.
17

---

Under review as a conference paper at ICLR 2020
1.2
Red channe
1.15
Blue
Semantic diversity
1.0
1.4
color diversity
1.6
1.05
1.4
1.2
0.96
0.9
0.8
0.85
0.8
varying bucket
varying bucket
(a) Averaged relative inception impact
(b) Averaged relative coloring impact
Figure 20: Buckets specification of RPGAN with DCGAN backbone trained CIFARIO dataset.
See Section for details.
bucket
bucket
bucket
bucket 4
bucket
Figure 21: Frozen paths individual blocks variation in 5-bucket RPGAN with DCGAN backbone. Generated images size is 64 x 64.
18

---

Under review as a conference paper at ICLR 2020
Green
semantic diversity
color diversity
2.5
15
varying bucket
varying bucket
(a) Averaged relative inception impact
(b) Averaged relative coloring impact
Figure 22:
Buckets specification of RPGAN with DCGAN backbone trained on CelebA dataset. See
Section 4.1 for details.
Figure 23: Samples from 5-bucket ResNet-like generator without nonlinearities.
G
5
7 8 5
3 6
9
0
6
0
3
3
6
7
3
f
3
4 7 4
6
3 2
7
6
04559
8
7
2
7
7
6 7 9 8
$ 4 0 0
0
Figure 24: Digits generated by RPGAN without nonlinearities (left) and by its x2.2 faster compres- sion (right).
19

---

Under review as a conference paper at ICLR 2020
Figure 25: SN-ResNet generator with different layers noising. Left: low normal noise variance.
Right: high normal noise variance
Original
Compressed
2 128
fc, 32 blocks, 128
fc, 32 blocks, 256
fc, 32 blocks, 512
fc, 16 blocks, 1024
fc, 128 blocks, 784
fc, 16 blocks, 784
Tanh, reshape to 28 x
Table 3: Fully connected RPGAN without nonlinearities and its compressed modification.
20


Reviews:
[{"review_id": "SyeceWbLcr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a variation on the generator of GANs. The authors modify the generator by adding a concept of \"blocks\" which are randomly activated based on part of the random input vector. It is similar to adding random dropout in the generator, except that the dropout would apply to larger sets of activations instead of single component.\n\nThe authors also add a diversity term to force the different blocks to have different blocks. This is a term based on the L2 distance between the weights of different blocks. Although this term is ad-hoc and could probably be refined into something more grounded in theory, it should indeed provide some diversity.\n\nThis block structure allows for more understanding of what each layer of the generator does, since it is easy to change the discrete variables that switch blocks. The paper presents an empirical evaluation of what each switch does, and show that concepts are well disentangled between layers (for instance, one layer changes the background whereas another one changes the color of the foreground).\n\nThey also show that blocks can be added after training is done which is a nice property for incremental training.\n\nThey also show that this framework can train a generator without non-linearities (except for the block switching), which could potentially simplify the analysis of such networks.\n\nGenerated samples are presented up to 128x128 pixels which, although far from state of the art, proves that the concept works.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklH2Ycr9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the issue of interpretability of GAN generation through an alternative approach to the introduction of variability. To seed the generation, instead of providing a random input vector (typically sampled from a standard Gaussian distribution), the authors instead modify the generator architecture so as to allow for randomization in the routing: each layer is replaced by a bucket consisting of several blocks, and in forward propagation only through randomly chosen blocks. In this case, the input vector is chosen to be a constant - the only source of randomization\nprovided to the generator is in the choice of blocks through which to propagate. The explanability derives from the tendency of blocks to associate with a common interpretation after training. Their use of blocks necessitates the introduction of a block diversity loss, to discourage mode collapse. The scheme is referred to as RPGAN, for \"Random Path GAN\".\n\nThe main strengths of the paper:\n\n(1) Their proposed approach is highly flexible. In principle, any underlying GAN architecture can be adapted by assigning each layer to a distinct bucket, and then replicating the layer across the blocks. \n\n(2) Experimental results do show that different block sequences are associated with common image characteristics after training, especially for the initial and final layers.\n\n(3) The use of non-standard ways of introducing stochasticity to GAN generation is an interesting idea in itself.\n\nThe main weaknesses:\n\n(1) Although the authors provide experimental examples showing images associated with various paths through the architecture, is not clear how interpretations can be associated with these paths. In the examples presented, there seems to be a tendency for greater interpretability at the initial and final layers, with the explanation given for the intermediate layers being less convincing.\n\n(2) The number of experimental examples is low, yet the authors draw rather firm conclusions (end of Section 4.1) regarding interpretability across layers. I am not sure that their conclusions adequately capture what is going on here, nor am I convinced that they generalize to other situations.\n\n(3) The number of buckets limits the numbers of explanations. Essentially, the method has the same difficulties as in clustering, where specifying too many or too few groups can profoundly influence the nature and quality of the result. Although the authors do discuss an approach by which the number of buckets can be incrementally increased (thereby allowing for variation in the number of explanations generated), the experimental evidence is insufficient.\n\n(4) Presumably, the replication of a layer across the blocks assigned to its bucket would require more training data and/or greater training times. What is the relationship in both time and quality between the original GAN network and its RPGAN versions?\n\n(5) There are many presentational problems with this paper, in grammar, vocabulary and terminology, sentence structure, etc.\n\nOverall, in its current state (not least due to presentational issues) the paper appears to be below the acceptance threshold.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lOQlo7cB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the Random Path Generative Adversarial Network (RP-GAN) to serve as a tool for generative model analysis.  The main idea is to have several different buckets in each block of the generator and then train the generator with random paths. To interpret the features captured by each block, the authors unfreeze one block and show the variance of the generated images via different buckets.\n\nThe contribution of the paper is limited. Most of the observations proposed in this paper are just to confirm the findings in the recent paper Bau et al (2019). And the authors did not clarify why their methods are better than the previous work. In addition, this work changes some standard ways of generating images. For example,  they use a fixed input vector Z rather than a random vector Z following Gaussian distribution. Then when they claim that their findings are also valid to standard GAN generators, e.g., SN-ResNet, they add the noise to the weights in SN-ResNet and claim that we can conclude similar findings as RP-GAN. Therefore, my question is if adding noise to the weights to a generator is sufficient for the interpretation of the generator, why do we still need this work for further interpretation?\n\nMinor:\nIn Figure 3, there are 10 images in each line but the number of buckets in each block is 40 for CIFAR10, as you claimed in Section 4. It should be clarified how you get those 10 images from 40 blocks in the unfreezed bucket.\n\nIn conclusion, I vote for a weak reject for this work. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received mixed scores: Weak Reject (R1 and R2) and Accept (R3). AC has closely read the reviews/comments/rebuttal and examined the paper. After the rebuttal, R2's concerns still remain. AC sides with R2 and feels that the generated interpretations are not convincing, and that the conclusions drawn are not fully supported. Thus the paper just falls below the acceptance threshold, unfortunately. The work has merits however and the authors should revise their paper to incorporate the constructive feedback.
Decision: Reject