Stochastic Conditional Generative Networks with Basis Decomposition
Authors: Ze Wang
Xiuyuan Cheng
Guillermo Sapiro
Qiang Qiu
Venue:  (2020)

Abstract: While generative adversarial networks (GANs) have revolutionized machine learning, a number of open questions remain to fully understand them and exploit their power. One of these questions is how to efficiently achieve proper diversity and sampling of the multi-mode data space. To address this, we introduce BasisGAN, a stochastic conditional multi-mode image generator. By exploiting the observation that a convolutional filter can be well approximated as a linear combination of a small set of basis elements, we learn a plug-and-played basis generator to stochastically generate basis elements, with just a few hundred of parameters, to fully embed stochasticity into convolutional filters. By sampling basis elements instead of filters, we dramatically reduce the cost of modeling the parameter space with no sacrifice on either image diversity or fidelity. To illustrate this proposed plug-and-play framework, we construct variants of BasisGAN based on state-of-the-art conditional image generation networks, and train the networks by simply plugging in a basis generator, without additional auxiliary components, hyperparameters, or training objectives. The experimental success is complemented with theoretical results indicating how the perturbations introduced by the proposed sampling of basis elements can propagate to the appearance of generated images.

Published as a conference paper at ICLR 2020
STOCHASTIC CONDITIONAL GENERATIVE NETWORKS WITH BASIS DECOMPOSITION
Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, Qiang Qiu
Duke University
{ze.w. W, xiuyuan. cheng, gwillermo.sapeeeeeeeeeeeeeoo wiwp:iuuuu/uueeccommm edu
ABSTRACT
While generative adversarial networks (GANs) have revolutionized machine learning, a number of open questions remain fully understand them and exploit their power. One of these questions is how to efficiently achieve proper diversity and sampling of the multi-mode data space. To address this, we introduce Ba- sisGAN, a stochastic conditional multi-mode image generator. By exploiting the observation that a convolutional filter can be well approximated as a linear combi- nation of a small set of basis elements, we learn a plug and-ed basis generator to stochastically generate basis elements, with just few hundred of parameters, fully embed stochasticity into convolutional filters. By sampling basis elements instead of filters, we dramatically reduce the cost of modeling the parameter space with sacrifice either image diversity fidelity. To illustrate this proposed plug-and sang--nd-layy framework, we construct variants of BasisGAN based on state-of- the-art conditional image generation networks, and train the networks by simply plugging basis generator, without additional auxiliary components, hyperpa- rameters, or training objectives. The experimental success is complemented with theoretical results indicating how the perturbations introduced by the proposed sampling of basis elements can propagate to the appearance of generated images.
INTRODUCTION
Conditional image generation networks learn mappings from the condition domain to the image do- main by training on massive samples from both domains. The mapping from condition, e.g., map, an image, e.g., satellite image, essentially one-to-many as illustrated in Figure [ In other words, there exists many plausible output images that satisfy a given input condition, which motivates us explore multi- conditional image generation that produces diverse images ditioned on one single input condition.
One technique to improve image generation diversity is to feed the image generator with an addi- tional latent code in the hope that such code can carry information that not covered by the input condition, so that diverse output images are achieved by decoding the missing information conveyed through different latent codes. However, as illustrated in the seminal work Isola et al. (2077), encod- ing the diversity with an input latent code can lead to unsatisfactory performance for the following reasons. While training using objectives like GAN loss Goodfellow al. (2014), regularizations like LI loss Isola et al. (2017) and perceptual loss Wang et al. (2018) are imposed to improve both visual fidelity and correspondence to the input condition. However, no similar regularization is im- posed to enforce the correspondence between outputs and latent codes, so that the network is prone to ignore input latent codes in training, and produce identical images from an input condition even with different latent codes. Several methods are proposed explicitly encourage the network to take into account input latent codes to encode diversity. For example, Mao et al. (2019) explicitly max- imizes the ratio of the distance between generated images with respect to the corresponding latent codes; while Zhu et al. (201bb) applies an auxiliary network for decoding the latent codes from the generative images. Although the diversity of the generative images is significantly improved, these methods experience drawbacks. In Mao et al. (2019), at least two samples generated from the same condition are needed for calculating the regularization term, which multiplies the memory footprint while training each mini-batch. Auxiliary network structures and training objectives in Zhu et al. (20176) unavoidably increase training difficulty and memory footprint. These previously proposed methods usually require considerable modifications to the underlying framework.

---

Published as a conference paper at ICLR 2020
Convolve
Convolve
Deterministi
Sab Sab-modee
Stochastie
Sub cub-model
Canditiar
Generated Images with Different Filters
Hasis Generator
Basis Generator
Filter Rec onstonstucttonn
Generator
Reconstruction
Filter
Ground Truth
T 8
Sampled
Figure 1: Illustration of the proposed BasisGAN. The diversity generated images are achieved by the parameter generation in the stochastic sub-model, where basis generators take samples from a prior distribution and generate low dimensional basis elements from the learned spaces. The sampled basis elements linearly combined using the deterministic bases coefficients and used to reconstruct the convolutional filters. Filters in each stochastic layer are modeled with a separate basis generator. By convolving the same feature from the deterministic sub-model using different convolutional filters, images with diverse appearances are generated.
In this paper, we propose stochastic model, BasisGAN, that directly maps an input condition to diverse output images, aiming building networks that model the multi-mode intrinsically. The proposed method exploits a known observation that a well-trained deep network can converge to significantly different sets of parameters across multiple trainings, due to factors such as different parameter initializations and different choices mini-batches. Therefore, instead of treating a con- ditional image generation network as deterministic function with fixed parameters, modeling the filter each convolutional layer as a sample from filter space, and learning the corre- sponding filter space using a tiny network for efficient and diverse filter sampling. In Ghosh et al. 2018), parameter non uon-uniqueeess used for multi-mode image generation by training several gen- erators with different parameters simultancously as a multi-agent solution. However, the maximum modes of Ghosh et (2018) are restricted by the number of agents, and the replication increases memory as well as computational cost. Based on the above parameters con-uniqueness property, introduce into deep network stochastic convolutional layers, where filters sampled from learned filter spaces. Specifically, we learn the mapping from simple prior to filter space us- ing neural networks, here referred to as filter generators. To empower a deterministic network with multi-mode image generation, we divide the network into a deterministic sub-model and stochastic sub-mode as shown in Figure [] where standard convolutional layers and stochastic convolutional deployed, respectively. By optimizing adversarial loss, filter gen-
propose
layers with filter generators erators can be jointly trained with a conditional image generation network. In each forward pass, filters at stochastic layers sampled by filter generators. Highly diverse images conditioned on the same input are achieved by jointly sampling of filters in multiple stochastic convolutional layers.
However, filters of a convolutional layer are usually high-dimensional while being together written as one vector, which makes the modeling and sampling of filter space highly costly practice in terms of training time, sampling time, and filter generator memory footprint. Based on the low-rank property observed from sampled filters, we decompose each filter linear combination of a small set basis elements Qiu et al. (2018). and propose to only sample www-dimensional spatial basis elements instead of filters. By replacing filter generators with basis generators, the proposed method becomes highly efficient and practical. Theoretical arguments are provided on how perturbations introduced by sampling basis elements can propagate to the appearance of generated images.
The proposed BasisGAN introduces a generalizable concept to promote diverse modes the con- ditional image generation. As basis generators act as plug-and-play modules, variants of BasisGAN be easily constructed by replacing in various state-of-the-art conditional image generation net-
2

---

image gen- Published as a conference paper at ICLR 2020
works the standard convolutional layers by stochastic layers with basis generators. Then, we directly train them without additional auxiliary components, hyperparameters, or training objectives on top of the underlying models. Experimental results consistently show that the proposed BasisGAN is simple yet effective solution multi-mode conditional image generation. We further empirically show that the inherent stochasticity introduced by our method allows training without paired sam- ples, and the one-to-many image-to-image translation is achieved using stochastic auto-encoder where stochasticity prevents the network from learning a trivial identity mapping.
Our contributions are summarized follows:
We propose blug-and-played basis generator stochastically generate basis elements, with just a few hundred parameters, to fully embed stochasticity into network filters. Theoretic arguments are provided to support the simplification of replacing stochastic filter generation with basis generation.
Both the generation fidelity and diversity of the proposed BasisGAN with basis generators are validated extensively, and state-of-the-art performances are consistently observed.
2 RELATED WORK
Conditional image generation. Parametric modeling of the natural image distribution has been studied for years, from restricted Boltzmann machines (Sooeesky) (1986) to variational autoen-
coders Kingma & Welling (2013); in particular variants with conditions Oord et al. (2016); Sohn
al. (2015) R Van den Oord et al. (2016) show promising results. With the great power of GANs
Goodfellow et al. (2014), conditional generative adversarial networks (cGANs) Isola et al. (2017)
Pathak et al. (201)): Sangkloy et al. (2177); Wang et al. 2); Xian et al. 2018 Zhu et al. (2017a)
achieve great progress on visually appealing images given conditions. However, the quality of im- ages and the loyalty to input conditions come with sacrifice on image diversity discussed in Zhu
al. (2017b), which is addressed by the proposed BasisGAN.
Multi-mode conditional image generation. To enable the cGANs with
eration, pioneer works like infoGAN Chen et al. (2016) and pix2pix [sola et al. (2017) propose to encode the diversity in an input latent code. enforce the networks take into account input latent codes, /Zhu et al.) 2017b) deploys auxiliary networks and training objectives to impose the recovery of the input latent code from the generated images. MSGAN Mao al. 2019) and DSGAN Yang iet al.] (2019) propose regularization terms for diversity that entorces a larger distance between gen- erated images with respect to different input latent codes given input condition. These methods require considerable modifications to the underlying original framework.
Neural network parameters generating and uncertainty. Extensive studies have been con- ducted for generating network parameters using another network since Hypernetworks Ha et al. (2016). As a seminal work on network parameter modeling, Hypernetworks successfully reduce learnable parameters by relaxing weight-sharing across layers. Followup works like Bayesian Hy- pernetworks Krueger et al. (017)) further introduce uncertainty to the generated parameters. Varia- tional inference based methods like Bayes by Backprop Blundell et al. (2015) solve the intractable posterior distribution of parameters by assuming prior (usually Gaussian). However, the assumed prior unavoidably degrades the expressiveness of the learned distribution. The parameter predic- tion of neural network is intensively studied under the context of few shot learning (Betinetto al. (2016): Qiao et al. (2018); Wang et al.] (2019). which aims to customize network to a new task adaptively and efficiently in data-driven way. Apart from few shot learning, Denil al. (2013) suggests parameter prediction as a way to study the redundancy in neural networks. While study- ing the representation power of random weights. [Saxe et al. (20TT) also suggests the uncertainty and non-uniqueness of network parameters. Another family network with uncertainty is based on variational inference Blundell et al. (205)). where an assumption of the distribution on network weights imposed for a tractable learning on the distribution of weights. Works on studying the relationship between local and global minima of deep networks Haeffele & Vidal (2015): Vidal et al. (017) also suggest the unnuuniueessss of optimal parameters of deep network.
3

---

Published as a conference paper at ICLR 2020
3 STOCHASTIC FILTER GENERATION
conditional generative network (cGAN) Mirza & Osindero (014) learns the mapping from input condition domain A to output image domain & using deep neural network. The conditional im- age generation is essentially one-to-many mapping as there could be multiple plausible instances B E B that map to condition A E A et al. (2017b), corresponding to a distribution p p(B However, the naive mapping of the generator formulated by neural network G A + B is de- terministic, and is incapable of covering the distribution p(B A). We exploit the non-uniqueness of network parameters discussed above, and introduce stochasticity into convolutional filters through plug-ande and-aaddpaaa play filter generators. To achieve this, we divide a network into two sub-models:
A deterministic sub-model with convolutional filters @ that remain fixed after training;
A stochastic sub-model whose convolutional filters w are sampled from parameter spaces modeled by neural networks T, referred to as filter generators, parametrized by with inputs from a prior distribution, e.g., e.g., (0,, I) for all experiments in this paper.
Note that filters in each stochastic layer are modeled with a separate neural network, which is not explicitly shown in the formulation for notation brevity. With this formulation, the conditional image generation becomes Go.0 + B, with stochasticity achieved by sampling filters Tg for the stochastic sub-model in each forward pass. The conditional GAN loss Goodfellow et al. (2014) Mirza & Osindero (2014) then becomes
min max V(D,G) G D
log ([A,
(1)
(A-p(A))s=ptss)log - D(A, (o,
where D denotes a standard discriminator. Note that we represent the generator here as (o. to emphasize that the generator uses stochastic filters
Tz(2).
Given stochastic generative network parametrized by & and f, and input condition A, the generated images form a conditional probability 9o.0 (B)o(B|A A), so that ( can be simplified
min max V(D, =EA B~p(B) log D(A, B)+ G
D
(2)
BA-)(A).B-q..1l A) log[1 D(A,
When the optimal discriminator is achieved, ② can be reformulated as
C(G) max VVD,G) EA~M(A)] log(4) - JSD()( go.0 (B
D
where JSD the lensen-Shannon divergence (the proof is provided the supplementary mate- rial). The global minimum of (3 is achieved when given every sampled condition A, the generator perfectly replicates the true distribution p(B|A), which indicates that by directly optimizing the loss in (]. conditional image generation with diversity is achieved with the proposed stochasticity in the convolutional filters.
To optimize TT. we train D as in Goodfellow et al. (2014) to maximize the probability of assigning the correct label to both training examples and samples from Gø Simultaneously, we train Go to minimize the following loss, where filter generators To are jointly optimized to bring stochasticity:
L EA- [log(1 D(A, Go (A; To()))))
(4)
We describe in detail the optimization of the generator parameters (á. 0) in supplementary material Algorithm
Discussions on diversity modeling in eGANs. The goal of cGAN is to model the conditional
probability AB. Previous cGAN models Mao et al. (2019; Mirza & Osindero (2014); Zhu
et al. (017b) typically incorporate randomness in the generator by setting == G 2),2 ~ p(ı),
where G is a deep network with deterministic parametrization and the randomness is introduced via z, e.g., latent code, as an extra inpul. This formulation implicitly makes the following two assumptions: (A1) The randomness of the generator is independent from that of p(A); (A2) Each realization (ww) conditional on can be modeled by CNN, i... B (A). where G is draw from an ensemble of CNNs, w being the random event. (A1) is reasonable as long as the source of variation to be modeled by cGAN is independent from that contained in A, and the rational of
4

---

Published as a conference paper at ICLR 2020
(A2) lies in the expressive power of CNNs for image to image translation. The previous model adopts a specific form of (A(A) via feeding random input =ww) to G, yet one may observe that the most general formulation under (A1), (A2) would be to sample the generator itself from certain distribution p(G), which is independent from p(A). Since generative CNNs are parametrized by convolutional filters, this would be equivalent to set B = ((A,w), w ~ (W), where we use "." in the parentheses to emphasize that what after is parametrization of the generator. The proposed cGAN model in the current paper indeed takes such an approach, where we model p( (w)) by separate filter generator network.
4 STOCHASTIC BASIS GENERATION
Using the method above, filters of each stochastic layer w are generated in the form of high- dimensional vector of size L x L C' C, where C", and C' correspond to the kernel size, num- bers of input and output channels, respectively. Although directly generating such high-dimensional vectors is feasible, it can be highly costly in terms of training time, sampling time, and memory footprint when the network scale grows. We present a throughout comparison in terms of generated quality and sample filter size in supplementary material Figure A.I] where it is clearly shown that filter generation is too costly to afford. In this section, we propose to replace filter generation with basis generation to achieve a quality/cost effect shown by the red dot in Figure I Details on the memory. parameter number, and computational cost are also provided at the end of the supplemen-
tary material, Section G
For convolutional filters, the weights w is a 3-way tensor involving spatial index and two channel indices for imput and output channel respectively. Tensor low-rank decomposition cannot be defined in unique way. For convolutional filters, natural solution then is separate out the spatial index, which leads to depth- rank factorization of convolutional layers. Qiu et al. (2018) proposes to approximate convolutional filter using a set of prefixed basis element linearly combined by learned reconstruction coefficients.
network architectures Chollet (2017). Among other studies of low-
Given that the weights in convolutional layers may have a low-rank structure, we collect a large amount of generated filters and reshape the stack of N sampled filters to a 2-dimensional matrix F with size of x J', where J = N x L x L and J' C" x C. We consistently observe that F is always of low effective rank, regardless the network scales we use to estimate the filter distribution. If we assume that a collection of generated filters observe such a low-rank structure, the following theorem proves that it suffices to generate bases in order to generate the desired distribution of filters.
Theorem 1. Let F), be probability space and F : Ω + R 1O'xxx a 3-way random tensor, where F maps each event w to F", X, A)), W [L] x [4]. N' [C', XE X For each fixed and u, F (u) = (F" (u, X, A)bxAA E C(TRO,, R If there exists set of deterministic linear transforms age, k 1,... K in L(RC' RC) FW (u) E paantax)k=1Bp any w and u, then there exists K random vectors by Ω + RT?, k 1, K. s.t. F(u, = Ek bx(u)ax(X, A) in distribution. F has probability density, then so do 3==1
The proof of the theorem is provided in the supplementary material.
We simplify the expensive filter generation problem by decomposing each filter as a linear combi- nation of small set of basis elements, and then sampling basis elements instead of filters directly. In our method, we assume that the diverse modes of conditional image generations are essentially caused by the spatial perturbations, thus we propose to introduce stochasticity to the spatial basis elements. Specifically, we apply convolutional filer decomposition as in Qiu al. (2018) to write w= $da v E RL.XXX where v are basis elements, a are decomposition coefficients, and K is pre-defined small value, e.g., K- 7. We keep the decomposition coefficients a deterministic and learned directly from training samples. Instead of using predefined basis elements as in Qiu et al. 2018). adopt basis generator network T(0, 2) parametrized by 6. that learns the mapping from random latent vectors & to basis elements & with stochasticity. The basis generator networks are jointly trained with the main conditional image generation network in an end-to-end manner. Note that we inherit the term 'basis' from DCFNet Qiu et al. (2018) for the intuition behind the proposed framework, and we do not impose additional constrains such as orthogonality or linear indepen- dence to the generated elements. Sampling the basis elements sa using basis generators dramatically reduces the difficulty on modeling the corresponding probability distribution. The costly filter gener- ators in Section 3 is now replaced by much more efficient basis generators, and stochastic filters
5

---

Published as a conference paper at ICLR 2020
then constructed by linearly combining sampled basis elements with the deterministic coefficients, w = th = T(0, a)a. The illustration on the convolution filter reconstruction is shown as a part of Figure [ As illustrated in this figure, BasisGAN is constructed by replacing convolutional layers with the proposed stochastic convolutional layers with basis generators, and the network parameters can be learned without additional auxiliary training objective or regularization.
5 EXPERIMENTS
In this section, we conduct experiments on multiple conditional generation task. Our preliminary objective is to show that thanks to the inherent stochasticity of the proposed BasisGAN, multi-mode conditional image generation can be learned without any additional regularizations that explicitly promote diversity. The effectiveness of the proposed BasisGAN is demonstrated by quantitative and qualitative results multiple tasks and underlying models. We start with stochastic auto-encoder example to demonstrate the inherent stochasticity brought by basis generator. Then we proceed to image to image translation tasks, and compare the proposed method with: regularization based methods DSGAN Yang et al. (2019) and MSGGA[Mao et al. (2019) that adopt explicit regularization terms that encourages higher distance between output images with different latent code; the model based method MUNIT Huang et al. (2018) that explicitly decouples appearance with content and achieves diverse image generation by manipulating appearance code; and BicycleGAN Zhu et al. (2017b) that uses auxiliary networks to encourage the diversity of the generated images with respect to the input latent code. We further demonstrate that as an essential way to inject randomness to conditional image generation, which can be adopted together with our proposed method for further performance improvements. Finally, ablation studies on the size of basis generators and the effect of K are provided in the supplementary material, Section
method is compatible with existing regularization based methods,
5.1 STOCHASTIC AUTO-ENCODER
The inherent stochasticity of the proposed BasisGAN allows learning conditional one-to- to-yymyyyy map- ping even without paired samples for training. We validate this by a variant of BasisGAN referred as stochastic auto-encoder, which is trained to do simple seff-reconstructionn with real-world images as inputs. Only L1 loss and GAN loss are imposed to promote fidelity and correspondence. However, thanks to the inherent stochasticity of BasisGAN, we observe that the network does not collapse to trivial identity mapping, and diverse outputs with strong correspondence to the input images are generated with appealing fidelity. Some illustrative results are shown in Figure
Input
Generated diverse samples
Input
Generated diverse samples
Figure 2: Stochastic auto-encoder: one-to- many conditional image generation without paired sam-
ple. The network is trained directly to reconstruct the input real-world images, and the inherent stochasticity of the proposed method successfully promotes diverse output appearances with strong fidelity and correspondence to the inputs.
5.2 IMAGE TO IMAGE TRANSLATION
To faithfully validate the fidelity and diversity of generated images, we follow Mao et al. 2019 to evaluate the performance quantitatively using the following metrics:
LPIPS. The diversity of generated images are measured using LPIPS Mao et al. 2011 LPIPS computes the distance of images in the feature space. Generated images with higher diversity give higher LPIPS scores, which are more favourable in conditional image generation.
FID. FID Heusel et al. (201)) is used measure the fidelity of the generated images. It computes

---

Published as a conference paper at ICLR 2020
Input
Ground truth
Generated diverse samples
Figure 3: BasisGAN adapted from Pix2Pix. The network is trained without any auxiliary loss functions regularizations. From top bottom, the image image translation tasks are: edges + handbags, edges + shoes, maps + satellite, nights + days, facades + buildings. Additional examples are provided in the supplementary material, Figure A.2
the distance between the distribution of the generated images and the true images. Since the entire GAN family is to faithfully model true data distribution parametrically, lower FID is favourable in our case since it reflects closer fit the desired distribution.
Pix2Pix - BasisGAN. As one of the most prevalent conditional image generation network, Pix2Pix Isola al. 2017) serves solid baseline for many multi-mode conditional image genera- tion methods. achieves conditional image generation by feeding the generator a conditional image, and training the generator to synthesize image with both GAN loss and L1 loss to the ground truth image. Typical applications for Pix2Pix include edge and so on. We adopt the ResNet based Pix2Pix model, and impose the proposed stochasticity in the successive residual blocks, where regular convolutional layers and convolutional layers with basis generators convolve alternatively with the feature maps. The network is re-trained from scratch di- rectly without any extra loss functions or regularizations. Some samples
or handbags, maps maps->satellites,
visualized in Figure For a fair comparison with previous works Isola et al.) (2017); Mao et al. (201);; Zhu et al. (201 7b); Yang et al. (2019); Huang et al. (2018). perform the quantitative evaluations on image image translation tasks and the results are presented Table [ Qualitative comparisons are presented in Figure A.3 As discussed, all the state-of-the-art methods require considerable modifications to the underlying framework. By simply using the proposed stochastic basis generators as plug-and-play modules to the Pix2Pix model, the BasisGAN generates significantly more diverse images but still comparable quality with other state-of-the-art methods. Moreover, shown in Table A.3] Ba- sisGAN reduces the number of trainable parameters comparing to the underlying methods thanks to the small number of basis elements and the tiny basis generator structures. While regularization based methods like Mao et al. (2019); Yang et al. (2019) maintain the parameter numbers of the underlying network models.
Pix2PixHD BasisGAN. In this experiment, we report results on high-resolution scenarios, which particularly demand efficiency and have not been previously studied by other conditional image generation methods.
We conduct high resolution image synthesis on Pix2PixHD Wang et al. (2018), which is proposed to conditionally generate images with resolution up to 2048 1024. The importance of this experiment arises from the fact that existing methods Mao et al. 2019 Zhu al. (2017b) require considerable modifications to the underlying networks, which in this case, are difficult to be scaled to very high resolution image synthesis due the memory limit of modern hardware. Our method requires no auxiliary networks structures or special batch formulation, thus is easy to be scaled to large scale scenarios. Some generated samples visualized in Figure 4 Quantitative results and comparisons
7

---

Published as a conference paper at ICLR 2020
Table Quantitative results on image image translation. Diversity and fidelity are measured using LPIPS and FID, respectively. Pix2Pix Isola et al. (2017), BicycleGAN /huu et al. (2017b), MSGAN Mao al. 2 and DSGAN Yang al. (219)) included in the comparisons. DSGAN adopts a different setting (denoted as 20s in the table) by generating 20 samples per imput for computing the scores, We report results under both settings,
Dataset
Labels + Facade
Methods
Pix2Pix
BicycleGAN
MSGAN
BasisGAN
DSGAN (20s)
BasisGAN (20s)
Diversity Fidelity
0.0003 0,0000 139.19 =2 .4
0.1413 0.0005 98.85 + 1.21
0.1894 0.0011 92.84 1,00
0.2648 0.004 88.7 + 1.28
0.18
0.2594 0.004
57.20
24.14 + 0.76
Darasets
Map - Satellite
Methods
Pix2Pix
BicycleGAN
MSGAN
BasisGAN
DSGAN (20s)
BasisGAN (20s)
Diversity Fidelity
0,0016 + 0.0003 168.99 = 2.58
0.1150 0.0007 145.78 + 3,90
0,2189 + 0.0004
0.2417 0.005 35,54 2.19
0.13
0.2398 0.005
152.43 + 2.52
49.92
28.92 + 1.88
Dataset
Edge - Handhag
Edge MUNIT
Shoe
Methods
MUNIT
BasisGAN
BasisGAN
Diversity
0.32 +0.624
0.35 +0.810
0.217 0.512 62.57 0.917
0.242 +00.443 64.17 + 1.14
Fidelity
92.84 = 0.121
88.76 10.513
Input condition
Generated diverse samples
Figure 4: High resolution conditional image generation. Additional examples provided in the
supplementary material, Figure A.4
against DSGAN Yang et al. 2019) reported in Table 2 BasisGAN significantly improves both diversity and fidelity with little overheads in terms training time, testing time, and memory.
Image inpainting. We conduct one-to-many image inpainting experiments face images. Fol- lowing Yang et al. (2019), centered face images in the celebA dataset are adopted and parts of faces are discarded by removing the center pixels. We adopt the exact same network used in Yang al. 2019) and replace the convolutional layers by layers with basis generators. To show the plug- and-plat play compatibility of the proposed BasisGAN, we conduct experiments by both training Basis- GAN alone and combining BasisGAN with regularization based methods DSGAN (BasisGAN DSGAN). When combining BasisGAN with DSGAN, we feed all the basis generator in BasisGAN with the same latent code and use the distance between the latent codes and the distance between generated samples to compute the regularization term proposed in Yang et al. (2019). Quantitative results and qualitative results are in Table and Figure [] respectively. BasisGAN delivers good bal- ance between diversity and fidelity, while combining BasisGAN with regularization based DSGAN further improves the performance.
Table 2: Quantitative results on high resolution image Table Quantitative results on face in- to image translation. Diversity and fidelity are mea- painting. Diversity and fidelity are mea- sured using LPIPS and FID, respectively.
sured using LPIPS and FID, respectively.
Methods
Pix2PsdHD
DSGAN
BasiGANN
Methods
DSGAN
BasiGGAN
BasisGAN DSGAN
Diversity 1
0.0
0.12
0.168
Diversity Fidelity
0.05
0.062
0.073
Fidelity
48.85
28.8
25.12
13.94
12.88
12.82
8

---

Published as a conference paper at ICLR 2020
DVD
Input condition
BasisCAN
BasisGAN DSGAN
Figure 5: Face inpainting examples.
CONCLUSION
In this paper, proposed BasisGAN model the multi-mode for conditional image generation in an intrinsic way. We formulated BasisGAN as a stochastic model to allow convolutional filters to be sampled from a filter space learned by neural network instead of being deterministic. To sig- nificantly reduce the cost of sampling high-dimensional filters, we adopt parameter reduction using filter decomposition, and sample low-dimensional basis elements, supported by the theoretical re- sults here presented. Stochasticity introduced by replacing deterministic convolution layers with stochastic layers with basis generators. BasisGAN with basis generators achieves high-fidelity and high-diversity, state-of-the-arr conditional image generation, without any auxiliary training objec- tives or regularizations. Extensive experiments with multiple underlying models demonstrate the effectiveness and extensibility of the proposed method.
7
ACKNOWLEDGMENTS
Work partially supported by ONR, ARO, NGA, NSF, and gifts from Google, Microsoft, and Ama- zon.
REFERENCES
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed- forward one-shot learners. In Advances in Neural Information Processing Systems, pp. 523-531, 2016.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613-1622, 2015.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2172-2180, 2016.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In JEEE Con- ference on Computer Vision and Pattern Recognition, pp. 1251-1258, 2017.
Misha Denil, Babak Shakibi, Laurent Dinh, Nando De Freitas, et al. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, pp.
2013.
Arnab Ghosh, Viveka Kulharia, Vinay P Namboodiri, Philip HS Torr, and Puneet K Dokania. Multi- agent diverse generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 8513-8521, 2018.
Ian Goodfellow, Jean Pouget-Abadee Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
David Ha, Andrew Dai, and Quoe V Le. Hypernetwrkss. arXiv preprint 1rXiv:1609.09106, 2016.
Benjamin D Haeffele and René Vidal. Global optimality in tensor factorization, deep learning, and beyond. arXiv preprint 156v:1506607544 2015.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.

---

arXiv preprint Published as a conference paper at ICLR 2020
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. Proceedings of the European Conference on Computer Vision (ECCV), pp- 172-189, 2018.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In JEEE Conference on Computer Vision and Pattern Recognition, 2017.
pp.
Diederik Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint arXiv: 170.00770, 2017.
Qi Mao, Ying Lee, Hung-Ya Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking gener- ative adversarial networks for diverse image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv:1411.178% 2014.
arXiv preprint
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv: (6X1.066000675) 2016.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2544, 2016.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan Yuille. Few-shot image recognition by predicting parameters from activations. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 7229-7238, 2018.
Qiang Qiu, Xiuyuan Cheng, Robert Calderbank, and Guillermo Sapiro. DCFNet: Deep neural network with decomposed convolutional filters. International Conference on Machine Learning, 2018.
Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Scribbler: Controlling deep image synthesis with sketch and color. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400-5409, 2017.
Andrew M Saxe, Pang Wei Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, and Andrew Y Ng. On random weights and unsupervised feature learning. In International Conference on Machine Learning, volume 2, pp. 6, 2011.
Paul Smolensky. Information processing in dynamical systems: Foundations of harmony theory. Technical report, Colorado Univ Boulder Dept of Computer Science, 1986.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems, pp- 3483-3491, 2015.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Condi- tional image generation with pixelenn decoders. In Advances in Neural Information Processing Systems, pp. 4798, 2016.
Rene Vidal, Joan Bruna, Raja Giryes, and Stefano Soatto. Mathematics of deep learning. arXiv preprint 1712 2017.
Ting Chuu Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional GANs. In IEEE Confer- ence on Computer Vision and Pattern Recognition, 2018.
Xin Wang, Fisher Yu, Ruth Wang, Trevor Darrell, and Joseph E Gonzalez. Tafe-net: Task-aware feature embeddings for low shot learning. arXiv preprint arXiv: /204.00900005 05967, 2019.
10

---

Published as a conference paper at ICLR 2020
Wenqi Xian, Patsorn Sangkloy, Varun Agrawal, Amit Raj, Jingwan Lu, Chen Fang. Fisher Yu, and James Hays. Texturegan: Controlling deep image synthesis with texture patches. In IEEE Conference on Computer Vision and Puttern Recognition, pp. 8456-8465, 2018.
Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee. Diversity- sensitive conditional generative adversarial networks. arXiv preprint arXiv:/901.09024. 2019.
Jun-Yan Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Eiros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2223-2232, 2017a.
Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Bli Shechtman. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems, pp. 465 465-476, 2017b.

---

Published as a conference paper at ICLR 2020
A PROOF OF EQUATION (3)
Proof. Given (2) in Section 3, the minimax game of adversarial training is expressed as:
min max (DD -AA log ( B)+
Ba (BIA) log(1 - D(A,B)) ( A) log ( B) B (B)A) log(1 D(A, B)]}
(A.1)
By fixing A and only consider:
V' BB log D(A, B) + B (B/A) log - D(A.B)
(A.2)
p (B log ( B) + 4s.0 (B. log ((A, dB
The optimal discriminator D in is achieved when
p (B|A)
D' (A, A[B + 9o.0 (B(A)
(A.3)
Given the optimal discriminator D", A.2 is expressed as:
V' = (BAA) log (*,A, B) + EB Bo.6 (B|A) log/1 (A,
(B
9 B
B
BB llog (B + 1s.0 (B A EB ol BB llog pl
(BIA)
9o.0 B A)
P AB ( KL A)
AB
+ Jo.8 BIA)
log((4) + K ( A)
log((4) + (SS((((B) A (
(A.4)
where KL is the Dallback-Leibler divergence. The minimum of V is achieved iff the Jensen- Shannon divergence 0 and p(B A) (B)A.. And the global minimum of (A.l) 1 is achieved when given every sampled A, the generator perfectly replicate the conditional distribution p(B
B PROOF OF THEOREM 4.1
Proof. We first consider the case when {a*]k= is a linearly independent set in the space of C(RC,, RC), which is finite dimensional (the space of C" C"-by-C matrices). Then (*(u) is in the span of {a for any w, u means that there are unique coefficients b(k; w, u) s.t.
K
(m( = Σ b(k; w, u)ak.
and the vector {b(k; w, u)}x E R can determined from ("(u) by (deterministic) linear trans- form. Since each entry F(u, X', A) is random variable, i.c. measurable function on (St, IP). then so b(k; viewed mapping from 52 to IR, for each i and u, due to that linear transform between finite dimensional spaces preserves measurability. For same reason, if F(u, X' X) has probability density, then so does each b(k;;, u). Letting {b(k;, u) [E] be the random vectors bş proves the statement.
linearly dependent, the dimensionality of the subspace where F²(u) lie in is K < K. Suppose {āx K is linearly independent set which spans the subspace, and T R K is the linear transform to map to ag 's from às Using the argument above, there exist random vectors by F Σk byāg, and using the pseudo-inverse of T' to construct random vectors KK we have that = Ex byag- This proves the existence of the K random vectors b4- D
When {a
s.
C PARAMETER OPTIMIZATION IN FILTER GENERATION
The optimization of the parameters {d) 0) in filter generation is presented in Algorithm
12

---

Published as a conference paper at ICLR 2020
Algorithm Optimization of the generator parameters {Ø, )
for number iterations
Sample minibatch of pairs of samples (A, B A.
Sample 2 No I).
Calculate the gradient w.r.t. the convolutional filters e and w as in the standard setting
UC
UC
Aw
OΦ
8w
where Ei D(A, Go. (A; (=)))))
Calculate the gradient w.r.t. 0 in the filter generator A9 = Aw de
Sw
Update the parameters d: @ + & - adsi
aao. where a the learning rate.
end for
6007
Quaa ity & Cost Ploting
Ganzwted with bass groentooss
Candition
Grunnd trull
Chourntad wals filer geneentoos
0007
600
ConyTias Merray)
(a) Quality/cost comparison.
(b) Generated images.
Figure A.I: (a) shows the comparison between basis generation and filter generation in terms of quality and cost. In (b), top row shows images generated with basis generators (the red dot in (a)), bottom row shows images generated with filter generators at the highest cost (highest in (a)). Basis generation achieves better performance with significantly less cost comparing filter generation. The quality metrics are introduced in Section
D COMPUTATION COMPARISON
We present a throughout comparison terms of generated quality and sample filter size Fig- ure A.l where is clearly shown that filter generation is too costly to afford, and basis generation achieves a significantly better quality/cost effect shown by the red dot in Figure A.I
E ABLATION STUDIES
In this section, perform ablation studies the proposed BasisGAN, and evaluate multiple factors that can affect generation results. We perform ablation studies on BasisGAN adapted from the Pix2Pix model with the maps satellite dataset.
Size of basis generators. We model a basis generator using a small neural network, which consists of several hidden layers and inputs latent code sampled from prior distribution. We consistently observe that basis generator with single hidden layer achieves the best performance while main- tains fast basis generation speed. Here we perform further experiments on the size of intermediate layers and input latent code size, and the results are presented in Table A.I It is observed that the size of basis generator does not significantly effect the final performance, and we use the 64 + 64 setting in all the experiments for good balance between performances and costs.
Number basis elements K. By empirically observing the low rank generated filters, we use K = 7 in all the aforementioned experiments. We conduct further experiments to show the perfor- mances with larger and show the results Table A.2 clearly shown that by increasing K, the quality of the generated images do not increase. And when K gets larger, e.g, 128, even significantly degrades the diversity of the generated images.
13

---

Published as a conference paper at ICLR 2020
Table A.l: Quantitative results with different sizes of input latent code and intermediate layer. m denotes the size of latent code and intermediate layer.
Dimensions 16 + 16
32 32
64
128
128
256 +56
512 + 512
Diversity 0.2242
0.2388
0.2417
0.2448 34.36
0.2452 33.70
0.2433 32.31
Fidelity 40.16
37.41
35.54
Table A.2: Quantitative results with different sizes of input latent code and intermediate layer. m++ denotes the size of latent code and intermediate layer.
K
7
16
32
64
128
Diversity Fidelity
0.2417
0.2409
0.2382
0.2288
0.2006
35.54
36.08
35.17
34.97
36.49
F QUALITATIVE RESULTS
F.I PIX2PIX -BASISGAN
Additional qualitative results for Pix2Pix BasisGAN are presented in Figure A.2 Qualitative
comparisons against MSGAN Mao et al. (2199) and DSGAN Yang al. (2019) presented in Figure A.3 We directly use the official implementation and the pretrained models provided by the authors. For each example, the first 5 generated samples are presented without any selection. For the satellite - map comparison, we often observe missing correspondence in the samples generated by DSGAN. BasisGAN consistently provides samples with diverse details and strong correspondence the input conditions.
F.2 PIX2PIxHD -BSSIGAN
Additional qualitative results for Pix2PixHD -BasisGAN are presented in Figure A.4
G SPEED, PARAMETER, AND MEMORY
We use Py PoTch for the implementation of the experiments. The training and testing are per- formed on single NVIDIA 1080Ti graphic card with 11GB memory. The comparisons on testing speed, parameter number, and training memory are presented in Table A.3] The training memory measured under standard setting with resolution of 256 x 256 for Pix2Pix. and 1024 x 512 for Pix2PixHD, Since we are using small number of basis elements (typically 7), and tiny basis gen- erators, the overall trainable parameter number of the networks are reduced. Note that we only compute the parameter number of the generator networks since do adopt any change the discriminators.
Table A.3: Speed in testing, memory usage in training, and overall trainable parameter numbers.
Methods
Testing speed (s)
Training memory (MB)
Parameter number
Pix2Pix
0.01017
1,465
11,330,243
Pix2Pix BasisGAN
0.01025
1,439
10,261,763
Pix2PixHD
0.0299
8,145
182,546,755
Pix2PixHD - BasisGAN
0.0324
8,137
154,378,051
14

---

Published as a conference paper at ICLR 2020
Input
Ground truth
Generated diverse samples
Figure A.2: Pix2Pix BasisGAN.
15

---

Published as a conference paper at ICLR 2020
MSGAN
DSGAN
BasisGAN
MSGAN
DSGAN
BasisGAN
MSGAN
DSGAN
BasisGAN
MSGAN
DSGAN
BasisGAN
Input and GT
Generated diverse samples
Figure A.3: Qualitative comparisons with MSGAN and DSGAN. Please zoom in for details.
16

---

Published as a conference paper at ICLR 2020
Input condition
Generated diverse samples
Figure A.4: Pix2PixHD + BasisGAN
17


Reviews:
[{"review_id": "rJed5l9tqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new conditional GAN architecture. In particular, in order to allow for further diversity in conditional signal generation, the BasiGAN proposes to model the convolutional layers as a combination of basis which is stochastically sampled. The idea of the paper is interesting and some interesting experiments are presented. Nevertheless, I do not quite get why a set of predefined random basis would enforce more variability than the non-parametric way of training which is currently applied for conditional-GANs. If I get a convincing answer from the authors, I would definitely accept the paper (which otherwise is well-written and quite interesting to read). ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SkxALfA6YS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors introduce BasisGAN, a novel method for introducing stochasticity in conditional GANs, i.e., a way of conducting one-to-many mappings. This is a good addition in the literature as: (a) most of the widely-used conditional GANs such as pix2pix (Isola et al., 2016) or pix2pixHD (Wang et al., 2018) are deterministic (i.e., for a specific input a single output is always generated), (b) it improves upon the current SOTA in one-to-many mappings, (c) it is very useful application-wise. As also stated in the paper, there is a number of applications where this method is handy (e.g., converting a sketch to images varying in colors, etc.).\n\nI am leaning towards accepting this paper as this work is well-motivated and found the idea of using the basis generator to learn the bases for the generation of the parameters quite interesting. This is the main contribution and difference of this paper in comparison to DCFNet (Qiu et al., ICML 2018), where the bases are not learned. \n\nNevertheless, I have the following questions/requests:\n\n- How can we tell that the generated bases are indeed bases (e.g., are they orthogonal?)\n- Please report the number of parameters used in your implementation in comparison to the rest of the methods. \n- Please provide qualitative results against the compared methods and especially against DSGAN (Qin et al., 2018).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgrY8ih9S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a model for stochasticity for conditional image generation, building upon the previously available (DCFNet) results on composition of  convolutional filters out of the elements of the filter basis. \n\nThe idea of introducing stochasticity by convolutional filters into the conditional generative models seems to be novel and the reviewer thinks it could be of interest for the community.\n\nThe following remarks could be given to improve the presentation:\n1) Theorem 1 is an existence theorem, so it does not give the procedure for construction of the basis. Does the construction procedure for the basis, described under the theorem formulation, meet the conditions of Theorem 1? \n2) The Theorem 1 formulation states that “ If there exists a set of deterministic linear transforms”. Should the linear independence be stated as well as one of the theorem conditions ( so that the space dimensionality would indeed be K)? \n3) The reviewer finds the structure of Section 4 confusing: it starts from the problem statement (first paragraph 'Using the method above, filters of each stochastic layer…’), then provides the description of the approach and only then outlines Theorem 1. It might be that stating Theorem 1 and then defining the method for generation of the basis (how exactly could we get to the basis? ) could improve readability of the paper. Essentially, the question is: is there any way to emphasise the procedure for filter generation and inform the reader in which circumstances these filters would be the basis (e.g. why it wouldn't be prone to the analogue of mode collapse when the filters do not effectively have enough diversity for linear independence)? \n***\nIn addition to this list, it might be useful to provide some evidence on whether there is any inherent mechanism to regulate the diversity of filters and therefore of samples (so that to change the variability of the conditional samples from the model with the impact analogous to the one of temperature in Glow (Kingma et al, 2018)). If there is one, further experimental evidence, which shows the impact on diversity of filters, would contribute to improvement of the paper.  \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content: BasiGAN, a novel method for  introducing stochasticity in conditional GANs
Summary of discussion:
reviewer1: interesting work and results on GANs. Reviewer had a question on pre-defned basis but i think it was answered by the authors. 
reviewer3: interesting and novel work on GANS, wel-written paper and improves on SOTA. The main uestion is around bases again like reviewer 1, but it seems the authors have addressed this.
reviewer4: Novel interesting work. Main comments are around making Theorem 1 more theoretically correct, which it sounds like the authors addressed.
Recommendation: Poster. Well written and novel paper and authors addressed a lot of concerns. 
Decision: Accept (Poster)