MVP: Multivariate polynomials for conditional generation
Authors: Grigorios Chrysos
Yannis Panagakis
Venue:  (2021)
Keywords: conditional image generation
generative models
polynomial neural networks

Abstract: Conditional Generative Adversarial Nets (cGANs) have been widely adopted for image generation. cGANs take  i) a noise vector and ii) a conditional variable as input. The conditional variable can be discrete (e.g., a class label) or continuous (e.g., an input image) resulting into class-conditional (image) generation and image-to-image translation models, respectively. However, depending on whether the conditional variable is discrete or continuous, various cGANs employ substantially different deep architectures and loss functions for their training. In this paper, we propose a novel framework, called MVP, for conditional data generation. MVP resorts to multivariate polynomials of higher-order and treats in a unified way both discrete and continuous conditional variables. MVP is highly expressive, capturing higher-order auto- and cross-correlations of input variables (noise vector and conditional variable). Tailored sharing schemes are designed between the polynomial’s parameter tensors, which result in simple recursive formulas. MVP can synthesize realistic images in both class-conditional and image-to-image translation tasks even in the absence of activation functions between the layers. 

Under review as a conference paper at ICLR 2021
MVP: MULTIVARIATE POLYNOMIALS FOR CONDITIONAL DATA GENERATION
Anonymous authors Paper under double-bling review
ABSTRACT
Conditional Generative Adversarial Nets (cGANs) have been widely adopted for image generation. cGANs take i) a noise vector and ii) a conditional variable as input. The conditional variable can be discrete (e.g., class label) or continuous (e.g., an input image) resulting into class-conditional (image) generation and image- to-image translation models, respectively. However, depending conditional variable is discrete or continuous, various cGANS employ substantially different deep architectures and loss functions for their training. In this paper, we propose a novel framework, called MVP. for conditional data generation. MVP resorts multivariate polynomials of higher-order and treats in a unified way both discrete and continuous conditional variables. MVP is highly expressive, capturing higher-order auto- and cross-correlations of input variables (noise vector and condi- tional variable). Tailored sharing schemes are designed between the polynomial 's parameter tensors, which result in simple recursive formulas. MVP can synthesize realistic images in both class-conditions and image-to tongetto-mmage translation tasks even in the absence activation functions between the layers.
whether the
INTRODUCTION
Modelling high-dimensional distributions and generating samples from complex distributions are fundamental tasks in machine learning. Generative adversarial networks (GANs) (Goodfellow et al. 2014) have demonstrated spectacular results in the two tasks using both unsupervised (Miyato et al. 2018) and supervised (Brock et al. 2019) learning. In the unsupervised setting, (the generator of) a GAN accepts input noise vector 2, and maps the noise vector to high-dimensional output. The supervised models, called conditional Generative Adversarial Nets (cGANs) (Mirza) & Osindero 2014), accept both noise vector 2 and an additional conditional variable 2n that facilitates the generation. The conditional variable can be discrete (e.g., a class or an attribute label) or continuous (c.g.. a low-resolution image). The impressive results obtained with both discrete conditional input (Brock et al. 2019) and continuous conditional input (Park et al.] 2019 et al. 2017) have led to plethora of applications that range from text-to-image synthesis (Qiao et al. 2019)
to deblurring Yan & Wang 2017) and medical analysis You et al.) 2019).
Despite the similarity in the formulation for discrete and continuous conditional input (i.e., learning the function G(2,, 21, 2n)). the literature has focused on substantially different architectures and losses. Frequently, techniques are simultaneously developed, e.g., the self-attention in the class-conditional Self-Attention GAN (Zhang et al. 2019) and in the Attention-GAN (Chen et al., (2018) with continu- ous conditional input. This delays the progress since practitioners develop twice as many architectures and losses for every case. A couple of straightforward ideas can be employed to unify the behavior of the two conditional variable types. One idea to use an encoder network to obtain representations that are independent of the conditional variable. This has two drawbacks: i) the network ignores the noise and a deterministic one-variable mapping is learned (Isola et al. 2017). ii) such encoder has not been successful far for discrete conditional input. An alternative idea is to directly concatenate the labels in the latent space instead of finding an embedding. In AC-GAN (Odena et al. (017)) the class labels are concatenated with the noise; however, the model does not scale well beyond 10 classes. We argue that concatenation of the input is only capturing additive correlation and not higher-orde order interactions between the inputs. A detailed discussion is conducted on sec. D (in the Appendix).

---

Under review as a conference paper at ICLR 2021
A polynomial expansion with respect to the input variables capture such higher-order correlations. II-Net (Chrysos et al. 2020) casts the function approximation into a polynomial expansion of single input variable. By concatenating the input variables, polynomial of the fused variable. However, the concatenation reduces the flexibility of the model significantly, e.g.. enforces the same order of expansion with respect to the different variables and it only allows the same parameter sharing scheme to all variables.
express the function approximation
We introduce multivariate framework, called MVP, for conditional data generation. MVP resorts to multivariate polynomials with two input variables, i.e., ž for the noise vector and ž for the conditional variable. MVP captures higher-order auto- and corss-corrssations between the variables. By imposing a tailored structure in the higher-order interactions, we obtain an intuitive, recursive formulation for MVP. The formulation is flexible and enables different constraints to be applied to each variable and its associated parameters. The formulation can be trivially extended to M input variables. In summary, our contributions are the following:
We introduce a framework, called MVP, that expresses a high-order, multivariate polynomial for conditional data generation. Importantly, MVP treats both discrete and continuous conditional variables in unified way.
We offer an in in-depth relationship with state-of sitte-of-the-aae the-art works, such as SPADE (Park et al. 2019), that can interpreted as polynomial expansions. We believe this perspective better explains the success of such architectures and offers new direction for their extension.
MVP is trained on eight different datasets for both class-conditional generation and image- to-image translation tasks. The trained models rely on both input variables, i.c., they do not ignore the noise vector.
To illustrate the expressivity of the model, also experiment with generators that do not use activation functions between the layers. We verify that MVP can synthesize realistic images even in the absence of activation functions between the layers.
The source code of MVP will be published upon the acceptance of the paper.
2 RELATED WORK
The literature on conditional data generation is vast; dedicated surveys per task (Agnese et al. 2019 Wu et al. 2017b) can be found for the interested reader. Below, we review representative works in conditional generation and then we summarize the recent progress in multiplicative interactions.
2.1 CONDITIONAL GENERATIVE MODELS
The challenging nature of image/video generation has led to a proliferation of conditional models. Although cGAN (Mirza & Osindero| 2014) is general framework, since then the methods developed for conditional generation differ substantially depending on the type of conditional data. We present below representative works of the two categories, i.c., discrete and continuous conditional data, and their combination.
Discrete conditional variable: This is most frequently used for class-conditional generation (Miyato al. 2018; Brock et al. 2019; Kaneko al. 2019). Conditional normalization (Dumoulin et al. 2017 De Vries et al.) 2017) techniques have been popular in the case of discrete conditional input, e.g., in generation of natural scenes images (Miyato et al.) 2018 Brock al., 2019). Conditional normalization cannot trivially generalize to a continuous conditional variable. In AC-GAN (Odena al. 2017), they concatenate the class labels with the noise; however, their model does not scale well (i.c.. they train model per 10 classes). The aforementioned methods cannot be trivially used or modified for continuous conditional input. Text-to-image generation models (Qiao et al. 2019: Li specialized branch to embed the text labels.
et al. 2019 Zhang et al. 2018 Xu al. 2018)
Continuous conditional variable: The influential work of pix2pix (Isola et al. 2017 has become the reference point for continuous conditional input. The conditional input is embedded in low- dimensional space (with an encoder), and then mapped to high-dimensional output (through a decoder). The framework has been widely used for inverse tasks (Ledig et al. 2017 Pathak et al.
2

---

Under review as a conference paper at ICLR 2021
2016
Wu et al.
201 7a; lizuka et al., 2017 Huang al. 2017 Yu al. 2018a Grm et al.
2019
Xie
et al.
2018 Yan & Wang 2017). conditional pose generation (Ma et al. [017 Siarohin et al.
2018
Liang et al. et al. 2018a), generation from semantic labels (Wang et al. 2018h), image blending (Wu et al. 2019
2019), representation learning (Tran et al.] (017). conditional video generation WWng
Zhan et al. 2019). We recognize
major drawhacks in the aforementioned methods: a) they cannot be easily adapted for discrete conditional input, b) they learn deterministic mapping, i.e. the noise typically ignored. However, in many real applications, such inverse tasks, the mapping is not one-to-one; there are multiple plausible outputs for every conditional input. The auxiliary losses used in such works, e.g., l loss (Isola et al., 2017), perceptual loss (Ledig et al.) 2017), are an additional drawback. Those losses both add hyper- paremeerrsssrrss that require tuning and are domain-specific, thus is challenging to transfer them to different domains or even different datasets. On the contrary, in our experiments, we do not use any additional loss.
Discrete and continuous conditional variables: Few works combine both discrete and continuous conditional inputs (Yu et al. 2018b 8b Xu et al. 2017 Lu et al. 2018). However, these methods include significant engineering (e.g., multiple discriminators (Xu et al. 20177, auxiliary losses), while often the generator learns to ignore the noise (similarly to the continuous conditional input). Antipov et al. (2017) design generator for face aging. The generator combines continuous with discrete variables (age classes), however there is no Gaussian noise utilized, i.e., a deterministic transformation is learned for each input face. InfoGAN (Chen et al.] (016)) includes both discrete and continuous conditional variables. However, the authors explicitly mention that additional losses are required, otherwise the generator is 'free to ignore' the additional variables.
The idea of al. (2020) most closely related our work. They introduce a unifying framework for paired (Tsola et al.] (017) and unpaired (Zhu et al. (2017a) learning. However, their framework assumes a continuous conditional input, while ours can handle discrete conditional input (e.g., class labels). In addition, their method requires pre-trained teacher generator, while ours consists of a single generator trained end-to-end.
Diverse data generation: Conditional image generation often suffers from deterministic mappings, i.e., the noise variable has often negligible or negative impact in the generator (Zhu al., [017b] Isola et al. 2017). This has been tackled in the literature with additional loss terms and/or auxiliary network modules. A discussion of representative methods that tackle diverse generation is deferred to sec.[Inn the Appendix. In Table I the differences of the core techniques summarized. Even though diverse generation is a significant task, we advocate that learning a generator does not ignore the input variables can be achieved without such additional loss terms. We highlight that diverse generation a byproduct of MVP and not our main goal. Particularly, we believe that diverse images can be synthesized because the higher-order correlations of the input variables captured effectively the proposed method.
Table 1: Comparison of techniques used for diverse, conditional generation. The majority of the methods insert additional loss terms, while some of them even require additional networks to be trained to achieve diverse generation results. MVP learns non-deterministic mapping without additional networks or loss terms, thus simplifying the training. Nevertheless, as we empirically exhibit sec. H..7 dedicated works that tackle diverse generation can be used in conjunction with the proposed MVP to further boost the diversity of the synthesized images.
Methods for diverse generation.
additional loss terms
auxiliary networks
Model
BicycleGAN (Zhu et al. 2017b)
Yang el al. (2019); Lee el al. (2019)
V
X
Huang et (2018) ; Lee et (202))
V
MVP (ours)
X
2.2 MULTIPLICATIVE INTERACTIONS
Multiplicative connections have long been adopted in computer vision and machine learning (Shin & Ghosh. 1991 Hochreiter Schmidhuber 1997 Bahdanau et al., 2013). The idea is to combine the inputs through elementwise products other diagonal forms. Even though multiplicative connections
3

---

Under review as a conference paper at ICLR 2021
have successfully been applied to different tasks, until recently there comprehensive study of their expressivity versus the standard feedforward networks. Jayakumar et al. (2020) include the proof that second order multiplicative operators represent greater class of functions than classic feed-forward networks. Even though we capitalize on the theoretical argument, our framework can express any higher-order interactions while the framework of oflaaykummar al. (2020) is limited to second order interactions.
Table 2: Comparison of attributes of polynomial-like neural networks. Even though the architectures of Karras et al. (2019); Chen et (209)); Park et. (2019) were not posed as polynomial expansions, believe that their success can be (partly) attributed the polynomial expansion (please check sec. For further information). 1I-Net and StyleGAN are not designed for conditional data generation. In practice, learning complex distributions requires high-order polynomial expansions; this can be effectively achieved with products of polynomials as detailed in sec. 3.2 Only II-Net and MVP include such formulation. Additionally, the only work that enables multiple conditional variables (and includes experiments with both continuous and discrete conditional variables) is the proposed MVP.
Attributes of polynomial-like networks.
product of polynomials
discrete variable
continuous cond. variable
multiple cond. variables
Model
IT-Net (Chrysos et al., 2020
StyleGAN (Karras al. 20199)
x
X
x
sBN (Chen et al.)
2019)
x
X
X
SPADE (Park et 2019)
X
MVP (ours)
V
Higher-order interactions have been studied in the tensor-related literature (Kolda & Bader 2009
Debals & De Lathauwer 2017]. However, their adaptation modern deep architectures has been slower. Chrysos et al. (2020) propose high-order polynomial for mapping the input z to the output =Gz). II-Netfocuses on single input variable and cannot handle the multivariate cases that are the focus of this work. Three additional works that can be thought of as polynomial expansions are Karras et al. (2019); Park et al. (2019); Chen et al. (2019). The three works were originally introduced as (conditional) normalization variants, but we attribute their improvements in the expressiveness of their polynomial expansions. Under the polynomial expansion perspective, they can be expressed as special cases of the proposed MVP. A detailed discussion is conducted in sec. the Appendix. We believe that the proposed framework offers direction to further extend the results of such works, e.g., by allowing more than conditional variables.
METHOD
The framework for a multivariate polynomial with two-variable input is introduced (sec. 3.1 The
derivation, further intuition and additional models are deferred the Appendix (sec. B The crucial
technical details, including the stability of the polynomial, are developed in sec. sec. 3.2 We emphasize
that a multivariate polynomial can approximate any function (Stone 1948 Nikol skil multivariate polynomial is universal approximator.
2013 i.e.,
Table 3: Symbols
TeaaaronnnTessoossmmarrrreseesee calligraphicuppprrass/lowweee
are symbolized by boldface letters e.g..
Symbol N
Role
W.W ,w. order M) with vector u E R is W X m and results in a tensor of order M - 1. We assume that symbols are sum-
www. The mode-m vector product of W (of
Expansion order the polynomial
k
Rank of decompositions Inputs the polynomial
#s, Fi n. Wyln.ol
Auxiliary variables
b. The IT 2; when while a detailed tensor notation is marized in Table B,
Purameter tensor of the polynomial
U Upq
C
Learnalee parameters
deferred to the Appendix (sec. B.I)
Hadamard product
4

---

Under review as a conference paper at ICLR 2021
3.1 TWo INPUT VARIABLES
Given two input variables 21, Zn E K where K S R or K s N, the goal is to learn a function G: Kaxd R that captures the higher-degree interactions between the elements of the two inputs. We can learn such higher-degree interactions as polynomials of two input variables. polynomial of expansion order N with output æ Ro has the form:
a = G(
Σ
Wlme
11 x 1 --pp
II
X + B
(1)
where B E Ro and Wl E - Xnd for n [1, E [1,n + are the learnable
parameters. The expansion depends on two (independent) variables, hence we the and p as auxiliary variables. The two products of [ do not overlap, i.e., the first multiplies the modes [2, p] (of wlleel) with 2 and the other multiplies the modes [p 1, with zn.
Z
Upl
UP2
U
[eNW....
U (ar
U12.11
Z1I
So G ( 2111
Figure 1: Abstract schematic for /ith order approximation of = G(2r, 211). The inputs 21, #17 are symmetric in our formulation. We denote with æ sample from prior distribution (e.g., Gaussian), while zι symbolizes a sample from a conditional input (e.g., class label or low-resolution image).
Recursive relationship: The aforementioned derivation can be generalized to arbitrary expansion order. The recursive formula for an arbitrary order N the following:
F, In-1 (U + U 11) #n-1
(2)
for ....... with 21 UT.n2 + U [1,11 =u and æ CaN + B. The parameters C Book UU|n,l E Baak for ,,
N and [1, are learnable.
The intuition behind this model the following: An embedding initially found for each of the two input variables, then the embeddings added together and they multiplied elementwise with the previous approximation. The different embeddings for each of the input variables allows us implement URR,J] and U Un.Ir] with different constraints, e.g., [|,,1] to be dense layer and U to be convolution.
3.2 MODEL EXTENSIONS AND TECHNICAL DETAILS
There are three limitations in 2 Those are the following: a) (2) describes polynomial expansion of two-variable input, b) each expansion order requires additional layers, c) high-order polynomials might suffer from unbounded values. Those limitations are addressed below.
Our model can be readily extended beyond two-variable input; an extension with three-variabl input developed in sec. C The pattern (for each order) is similar to the two-vare two-variable input: a) a different embedding is found for each input variable, b) the embeddings are added together, c) the result is multiplied elementwise with the representation of the previous order.
The polynomial expansion of (2 requires O(N) layers for an Nith order expansion. That is, each new order n of expansion requires new parameters Un,,2] and U UJ], However, the order of expansion
"To avoid cluttering the notation we same dimensionality for the two inputs. However, the derivations apply for different dimensionalities, only the dimensionality of the tensors change slightly.
5

---

Under review as a conference paper at ICLR 2021
can be increased without increasing the parameters substantially. To that end, we can capitalize on the product of polynomials. Specifically, let N1 be the order of expansion of the first polynomial. The output of the first polynomial is fed into second polynomial, which has expansion order of N2- Then, the output of the second polynomial will have an expansion order of N1 N2. The product of polynomials can he used with arbitrary number of polynomials; it suffices the output of the 7th polynomial to be the input to the (T+ 1)th polynomial. For instance, if we assume a product of o E N polynomials, where each polynomial has an expansion order of two, then the polynomial expansion is of 2 order. In other words, we need O(log2(N)) layers to achieve an N order expansion.
In algebra, higher-order polynomials are unbounded and can thus suffer from instability for large values. To avoid such instability, we take the following three steps: a) MVP samples the noise vector from the uniform distribution, i.e., from the bounded interval of [-1, b) hyperbolic tangent is used in the output of the generator the bounded interval of [-1, 1]. c) batch normalization (loffe Szegedy 2015) used to convert the representations to zero-mean. We emphasize that in GANs the hyperbolic tangent is the default activation function in the output of the generator, hence it is not an additional requirement of our method. Additionally, in our preliminary experiments, the uniform distribution can be changed for a Gaussian distribution without any instability. A theoretical analysis on the bounds of such multivariate polynomials would be interesting subject for future work.
normalization, i.e., it constrains the outputs in
EXPERIMENTS
The proposed MVP is empirically evaluated in three settings: a) a class-conditional generation, i.c., with discrete conditional input, b) an image-to-image translation, i.e., with continuous conditional input, c) a mixed conditional setting with two conditional variables. The goal is to showcase how MVP can be used with both discrete and continuous conditional inputs. Even though architectures specialized for a single task (e.g., Ledig et al. (0017) perform well in that task, their well-selected inductive biases (e.g., perceptual or Ł loss) do not generalize well in other domains or different conditional inputs. Hence, our goal is not to demonstrate state-of-the-ee arr results in specific tasks, but rather to propose one generic formulation. Further experiments (e-g., with SVHN or MNIST to SVHN translation; sec[H], the details on the datasets and the evaluation metries (sec. G are deferred to the Appendix. Throughout the experimental section, we reserve the symbol 2n for the conditional input (e.g., class label).
generation
Our framework, e.g., 2 does not include any activation functions. To verify the expressivity of our framework, we maintain the same setting for the majority of the experiments below. Particularly, the generator does not have activation functions between the layers; there is only a hyperbolic tangent in the output space for normalization. Training generator without activation functions between the layers also emerged in II-Net (Chrysos et al.) (2020), where the authors demonstrate the challenges in such framework. However, conduct one experiment using strong baseline with activation functions. That is, a comparison with SNGAN (Miyato & Koyama 2018) in class-conditional generation is performed (sec. 4.1).
Baselines: 'IINeetSICCONC' implements a polynomial expansion of a single variable, i.c., by concatenating all the input variables. 'SPADE' implements polynomial expansion with respect to the conditional variable. Also, 'GAN-CONC' and 'GAN-ADD' are added as baselines, where we replace the Hadamard products with concatenation and addition respectively. An abstract schematic of the differences between the compared polynomial methods is depicted in Fig. 6 while detailed description of all methods is deferred to sec. G Each experiment conducted five times and the mean and the standard deviation are reported.
4.1 CLASS-CONDITIONAL GENERATION
The first experiment is class-conditional generation, where the conditional input is class label in the form of one-hos hot vector. Two types of networks are utilized: a) a resnet-based generator (SNGAN), b) polynomial generator (II-Net) based on Chrysos al. (2020). The former network has exhibited strong performance the last few years, while the latter bears resemblance to the formulation we propose in this work.

---

Under review as a conference paper at ICLR 2021
Table 4: Quantitative evaluation on class-conditional generation with resnet-based generator (i.e.. SNGAN). Higher Inception Score (IS) (Salimans et al. (016)) (lower Frechet Inception Distance (FID) (Heusel al,, a (017)) indicates better performance. The baselines improve the IS of SNGAN, however they cannot improve the FID. Nevertheless, SNGAN-MVP improves upon all the baselines both the and the FID.
class-conditional generation on CIFARIO
Model
IS (T)
FID (J)
SNGAN
8.30 t 0.11
14.70 + 0.97
SNGAN-CONC
8.50 + 0.49
30.65 + 3.55
SNGAN-ADI
8.65 0.11
15.47 0.74
SNGAN-SPADE
8.69 + 0.19
21.74 + 0.73
SNGAN-MVP
8.77 + 0.12
14.22 + 0.66
Resnet-based generator: The experiment is conducted by augmenting the resnet-based generator of SNGAN. The quantitative results are in Table synthesized samples are illustrated in Fig. 2 a). SNGAN-MVE improves upon all the baselines both the Inception score (IS) (Salimans et al. 2016
and the FID (Heusel al. (017). The proposed formulation enables inter-class interpolations, That is, the noise 2 is fixed, while the class zn is interpolated. In Fig. bbb) and Fig. (2c), intra-class and
inter-class linear interpolations are illustrated respectively. Both the quantitative and the qualitative results exhibit the effectiveness of our framework.
Random samples per class
(b) Intra-class interpolation
(c) Inter-class interpolation
Figure 2: Synthesized images by MVP in the class-conditional CIFAR10 (with resnet-based genera- tor): (a) Random samples where each row depicts the same class, (b) Intra-class linear interpolation from source the target, (c) inter-class linear interpolation. In inter-class interpolation, the class labels of the leftmost and rightmost images are one-hot vectors, while the rest are interpolated in-between; the resulting images are visualized. In all three cases, MVP synthesizes realistic images.
Table Quantitative evaluation on class-conditional generation with II-Net-based generator. CI- FAR10, 10, there is considerable improvement the IS, while in Cars FID drops dramatically with MVP. We hypothesize that the dramatic improvement in Cars arises because of the correlations of the classes. For instance, the SUV cars (of different carmakers) share several patterns, which are captured by our high-order interactions, while they might be missed when learning different normalization statistics per class.
class-conditional generation on CIFARIO I0
class-conditional generation on Cars 196
Model
IS (T)
FID ({)
Model
FID ({)
GAN-CONC
3.73 + 0.32
294.33 + 8.16
GAN-CONC
240.45 + 16.79
GAN-ADD
3.74 + 0.60
298.53 + 16.54
GAN-ADD
208.72 + 12.65
SPADE
4.00 + 0.53
294.21
16.33
SPADE
168.19 39.71
TI-Ne--ICONC
6.65 + 0.60
71.81 + 33.00
II-Ne-SICONC
153.39 + 27.93
II-Net
7.54 + 0.16
37.26 + 1.86
II-Net
120.40 + 28.65
MVP
7.87 + 0.21
34.35
2.68
MVP
55.48 3.16

---

Under review as a conference paper at ICLR 2021
Il-Net-based generator: A product of polynomials, based on II-Net, is selected the baseline architecture for the generator. II-Net has conditional batch normalization (CBN) in the generator, while in the rest compared methods CBN is replaced by batch normalization. The results in CIFARIO are summarized in Table 5 (left), where MVP outperforms all the baselines by a large margin. An additional experiment is performed in Cars196 that has 196 classes. The results in Table 5 (right) depict substantial improvement over the all the baselines (53.9% reduction over the best-performing baseline). We should note that the baseline was not built for conditional generation, however we We hypothesize that the
have done our best effort to optimize the respective improvement arises because of the correlations of the classes. That is, the 196 classes might be correlated (e.g., the SUV cars of different carmakers share several patterns). Such correlations are captured by our framework, while they might be missed when learning different normalization statistics per class. Overall, MVP synthesizes plausible images (Fig. 11) even in the absence of activation functions.
4.2 CONTINUOUS CONDITIONAL INPUT
The performance of MVP is scrutinized in tasks with continuous conditional input, e.g., super- resolution. The conditional input žu is an input image, e-g., a low-resolution sample or corrupted sample. Even though the core architecture remains the same, a single change is made in the structure of the discriminator: Motivated by (Miyato & Koyama 2018), we include an elementwise product of Zu with the real/fake image in the discriminator. This stabilizes the training and improves the results. wealth of literature is available on such continuous conditional inputs (sec. 2.T), however we select the challenging setting of using generator without activation functions between the layers.
Table 6: Quantitative evaluation on super-resolution with II-Net-based generator on Cars196. The task the left is super-resolution 16x, while on the right the task is super-resolution 8x. Our variant of SPADE, i.e., SPADE-MVP (details in sec. C). vastly improves the original SPADE. The full two-variable model, i.e., MVP, outperforms the compared methods.
Super-resolution 16x Cars196
Super-resolution 8x Cars196
Model
FID(L)
Model
FID ())
SPADE
111.75 + 13.41
SPADE
119.18 + 14.82
LINNet-SICONG
80.16 : 12.42
SICONC
186.42 I 40.84
SPADE-MVE
72.63 + 3.18
MVV
64.76 Σ 8.26
8.26
MVP 60.42 1 6.19
MVP
62.76 4.37
The experiments are performed in (a) super-resolution, (b) block-inpainting Super-resolution assumes a low-resolution image is available, while in block inpainting, (rectangular) part of the image is missing. The two tasks belong in the broader category of 'inverse tasks', and they are significant both for academic reasons but also for commercial reasons (Sood et al., 2018: You et al., 2019). Such inverse tasks underdetermined; each input image corresponds to several plausible output images.
The FID scores in Cars for the task of suser-renolution are reported in Table In super-resolution 16x, Σn has 48 dimensions, while in super-resolution 8x, zu has 192 dimensions. Notice that the performance of TINNe-SICONN deteriorates substantially when the dimensionality of the conditional variable increases. That validates our intuition about the concatenation in the input of the generator (sec. D.. We also report the SPADE-MVP, which captures higher-order correlations with respect to the first variable as well (further details in sec. G). The proposed SPADE-MVP outperforms the original SPADE, however it cannot outperform the full two-variable model, i.e., MVP. MVP maintains outperforms all baselines by a large margin.
The qualitative results on (a) super-resolution 8x on CelebA, (b) super-resolution 8x on Cars 196, (c) super-resolution 16x on Cars Cars196 196 are illustrated in Fig. Similarly the qualitative results on block-inpainting are visualized in Fig. 11 For each conditional image, different noise vectors are sampled. Notice that the corresponding synthesized images differ in the fine-details. For instance, changes in the mouth region, the car type position and even background changes are observed. Thus, MVP results in high-resolution images that i) correspond to the conditional input, ii) vary in fine-details. Similar variation has emerged even when the source and the target domains differ substantially, e.g., in the translation of MNIST digits to SVHN digits (sec. H.3). We should mention that regularization techniques have been proposed specifically for image-to-image translation, e.g.,
8

---

Under review as a conference paper at ICLR 2021
(a) Super-resolution 8x
(b) Super-resolution 8x
(c) Super-resolution 16x
Figure 3: Synthesized images for super-resolution by (a), (b) 8x, (c) 16x. The first row depicts the conditional imput (i.e., low-resolution image). The rows 2-6 depict outputs of the MVP when noise vector is sampled per row. Notice how the noise changes (a) the smile or the pose of the head, (b) the color, car type or even the background, (c) the position of the car.
Yang et al. (2019) ; Lee et al. (2019). However, such works utilize additional losses and even require additional networks for training, which makes the training more computationally heavy and more sensitive to design choices.
5 CONCLUSION
The topic of conditional data generation is the focus of this work. A multivariate polynomial model, called MVP, is introduced. MVP approximates function G(2,, zn) with inputs 2 (e.g.. sample from Gaussian distribution) and žn (e.g., class low-resolution image). MVP resorts multivariate polynomials with arbitrary conditional inputs, which capture high-order correlations of the inputs. The empirical evaluation confirms that our framework can synthesize realistic images in both dlass-condition generation (trained on CIFARIO. Cars196 and SVHN), attribute- attribute-guided guided generation and image-to-imes to-image translation (i.e., super-resolution. block-inpainting, edges-to-shees bilges-to-handbarn MNIST-to-SVHNN We also showcase that it can be extended to three-vare variable input with class-conditional super-resolution. In addition to conditional data generation, the proposed framework can be used in tasks requiring fusion of different types of variables.
REFERENCES
Jorge Agnese, Jonathan Herrera, Haicheng Tao, and Xingquan Zhu. A survey and taxonomy of adversarial neural networks for text-to-image synthesis. arXiv preprint 1910.0990009999 2019.
Amjad Almahairi, Sai Rajeswar, Alessandro Sordoni, Philip Bachman, and Aaron Courville. Aug- mented cyclegan: Learning many-to-man mappings from unpaired data. In International Confer- ence on Machine Learning (ICML), 2018.
Grigory Antipov, Moez. Baccouche, and Jean-Luc Dugelay. Face aging with conditional generative adversarial networks. In International Conference on Image Processing (ICIP), pp. 2089-2093,
2017.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In International Conference on Learning Representations (ICLR), 2015.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference Learning Representations (ICLR), 2019.

---

Under review as a conference paper at ICLR 2021
Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems (NeurlPS), pp. 2172-2180, 2016.
Xinyuan Chen, Chang Xu, Xiaokang Yang, and Dacheng Tao, Attention gan for object transfiguration in wild images. In European Conference Computer Vision (ECCV), pp. 164-180, 2018.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Conference on Computer Vision and Pattern Recognition (CVPR), pp.
$188-8197, 2020.
Grigorios Chrysos, Stylianos Moschoglou, Yannis Panagakis, and Stefanos Zafeiriou. Polygan: High-order polynomial generators. arXiv preprint arXiv: 2019.
Grigorios
Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis, Jiankang Deng. and Stefanos Zafeiriou. x-nets: Deep polynomial neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1): 53-65, 2018.
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron Courville. Modulating early visual processing by language. In Advances in neural information processing systems (NeurIPS), pp. 6594-6604, 2017.
Debals and L De Lathauwer. The concept of tensorization. Technical report, Technical Report 17-99, ESAT-STADIUS, KU Leuven, Belgium, 2017.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems (NeurIPS), 2014.
Klemen Grm, Walter J Scheirer, and Vitomir Štrue. Face hallucination using cascaded super-resolution and identity priors. IEEE Transactions in Image Processing (TIP), 29()):150--2165, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems (NeurIPS), pp. 6626-6637, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Rui Huang, Shu Zhang, Tianyu Li, Ran He, et al. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. In International Conference on Computer Vision (ICCV), 2017.
Xun Huang, Ming-Yo Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In European Conference on Computer Vision (ECCV), pp. 172-189, 2018.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (TOG), 36(4):107,
Sergey Toffe and Christian Szegedy. Batch normalization: reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015.
Accelerating deep network training by
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to- to--tnag-egeee translation with conditional adversarial networks. Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
10

---

Under review as a conference paper at ICLR 2021
Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Tch, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In International Conference Learning Representations (ICLR), 2020.
Y Jin, J Zhang, M Li, Y Tian, and H Zhu. Towards the high-quality anime characters generation with generative adversarial networks. In Proceedings of the Machine Learning for Creativity and Design Workshop at NeurIPS, 2017.
Takuhiro Kaneko, Yoshitaka Ushiku, and Tatsuya Harada. Label-noise robust generative adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2467-2476, 2019.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51()): 455-500 2009.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Conference on Computer Vision and Pattern Recognition Workshops (CVPR')), pp. 554-561, 2013.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton, The cifar-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar. hmml, 55, 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-basee learning applied to document recognition. Proceedings of the IEEE, 86(1 96(11))2278-2324 1998.
Christian Ledig. Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Hsin- Ying
Hsin-Ying Lee, Hung- Yu Tseng, Qi Mao, Jia-Bin Huang. Yu- Yu-Ding Lu, Mancesh Singh, and Ming- Hsuan Yang. Drit++: Diverse image-to-ime to-ag translation via disentangled representations. Inter- national Journal of Computer Vision (IJCV), pp. 1-16, 2020.
Soochan Lee, Junsoo Ha, and Gunhee Kim. Harmonizing maximum likelihood with gans for multimodal conditional generation. In International Conference on Learning Representations (ICLR), 2019.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. In Advances neural information processing systems (NeurIPS), pp. 2063-2073, 2019.
Muyang Li, Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, and Song Han. Gan compression: Efficient architectures for interactive conditional gans. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Dong Liang, Rui Wang, Xiaowei Tian, and Cong Zou. Pegan: Partttion-cottrllll human image generation. In AAAI Conference Artificial Intelligence, volume 33, pp. 8698-8705, 2019.
Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, and Antonio Torralba. Diverse image generation via celficonditioned gans. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14286-14295, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision (ICCV), pp. 3730-3738, 2015.
Yongyi Lu, Yu-Wing Tai, and Chi-Keung Tang. Attribute-guided guided face generation using conditional cyclegan. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 282-297, 2018.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In Advances in neural information processing systems (NeurIPS), pp. 700-709, 2018.
11

---

Under review as a conference paper at ICLR 2021
Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. Pose guided person image generation. In Advances in neural information processing systems (NeurIPS), pP- 406-416, 2017.
Qi Mao, Hsin Hsin-Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming- Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 1429-1437, 2019.
Maxim Maximov, Ismail Elezi, and Laura Leal-Taixé. Ciagan: Conditional identity anonymization generative adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 14844141, 2014.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In International Conference Learning Representations (ICLR), 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
S.M. Nikol'skii. Analysis III: Spaces of Differentiable Functions. Encyclopaedia of Mathematical Sciences. Springer Berlin Heidelberg, 2013. ISBN 9783662099612.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International Conference on Machine Learning (ICML), pp. 2642-2651. JMLR.
org. 2017.
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2337-2346, 2019.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context en- coders: Feature learning by inpainting. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2536 2544, 2016.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1505-1514, 2019.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems (NeurIPS), pp. 2234-2242, 2016.
Yoan Shin and Joydeep Ghosh. The pi-sigma network: An efficient higher-order neural network for pattern classification and function approximation. International Joint Conference on Neural Networks, volume 1, pp. 13-18, 1991.
Aliaksandr Siarohin, Enver Sangineto, Stephane Lathuiliere, and Nicu Sebe. Deformable gans for based human image generation. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 2408-3416, 2018
Rewa Sood, Binit Topiwala, Karthik Choutagunta, Rohit Sood, and Mirabela Rusu. An application of generative adversarial networks for super resolution medical imaging. In International Conference Machine Learning and Applications (ICMLA), pp. 326-331, 2018
Marshall H Stone. The generalized weierstrass approximation theorem. Mathematics Magazine, 21 (5):237-254, 1948.
12

---

Under review as a conference paper at ICLR 2021
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 2015.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. In International Conference on Learning Representations (ICLR), 2016.
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose-invariant face recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), volume 3, pp. 7, 2017.
Ting- Ting-Chun Wang. Ming-Yu Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In Advances in neural information processing systems (NeurIPS), 2018a.
Ting-Chun Chgu Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans, In Conference on Computer Vision and Puttern Recognition (CVPR), pp. 8798-8807, 2018b.
Bingzhe Wu, Haodong Duan, Zhichao Liu, and Guangyu Sun. Srpgan: Perceptual generative adversarial network for single image super resolution. arXiv preprint arXiv: 1712.0557 2017a.
Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. Gp-gan: Towards realistic high-resolution image blending. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 2487-2495, 2019.
Xian Wu, Kun Xu, and Peter Hall. A survey of image synthesis and editing with generative adversarial networks. Tsinghua Science and Technology. 22(6):660-6744 2017b.
Saining Xie and Zhuowen Tu. Holistically-nester edge detection. In International Conference Computer Vision (ICCV), 2015.
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempogan: temporally coherent, volumetric gan for super-resolution fluid flow. ACM Transactions on Graphics (TOG), 37(4): 2018.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.
Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1316-1324, 2018.
Xiangyu Xu, Deqing Sun, Jinshan Pan, Yujin Zhang, Hanspeter Pfister, and Ming-Hsuan Yang. Learning to super-resolve blurry face and text images. In International Conference on Computer Vision (ICCV), pp. 251-260, 2017.
Qiaojing Yan and Wei Wang. Degans for image super-resolution, denoising and debluring. Advances in neural information processing systems (NeurIPS), pp. 487-495, 2017
Dingdong Yang. Seunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee. Diversity- sensitive conditional generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
Chenyu You, Guang Li, Yi Zhang, Xiaoliu Zhang, Hongming Shan, Mengzhou Li, Shenghong Ju, Zhen Zhao, Zhuiyang Zhang, Wenxiang Cong, et al. Ct super-resolution gan constrained by the identical, residual, and eycle learning ensemble (gan-circle). IEEE Transactions on Medical Imaging, 39(1): (18)))882200)) 200,203,222 2019.
A. Yu and K. Grauman. Fine-Grained Visual Comparisons with Local Learning. In Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5505-5514, 2018a.
Xin Yu, Basura Fernando, Richard Hartley and Fatih Porikli. Super-resolving very low-resolution face images with supplementary attributes. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 908-917, 2018b.
13

---

Under review as a conference paper at ICLR 2021
Fangneng Zhan, Hongyuan Zhu, and Shijian Lu. Spatial fusion gan for image synthesis. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6653-3662, 2019.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 4188):1947-196,, 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Machine Learning (ICML), 2019.
Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon.
Bias and generalization in deep generative models: An empirical study. In Advances in neural information processing systems (NeurIPS), pp. 10792-10801, 2018.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. Generative visual manip- ulation the natural image manifold. In European Conference Computer Vision (ECCV), 2016.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-imane translation using cycle-consistent adversarial networks. In International Conference on Computer Vision (ICCV), pp. 2223-2232, 2017a.
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems (NeurIPS), pp. 465-476, 2017b.
A SUMMARY OF SECTIONS IN THE APPENDIX
In the following sections, further details and derivations are provided to elaborate the details of the MVP. Specifically, in sec. B the decomposition and related details on the method are developed. The extension of our method beyond two-input variables is studied in sec. A method frequently used in the literature for fusing information concatenation; we analyze how concatenation captures only additive and not more complex correlations (e.g.. multiplicative) in sec. D The differences from IT-Net (Chrysos et al., 2020) is explored in sec. F In sec. F some recent (conditional) data generation methods are cast the polynomial neural network framework and their differences from the proposed framework are analyzed. The experimental details including the evaluation metrics and details on the baselines are developed in sec. G In sec. H. additional experimental results are included. Lastly, the differences from works that perform diverse generation are explored in sec.
B METHOD DERIVATIONS
In this section, we expand on the method details, including the scalar output case or the notation. Specifically, a more detailed notation is determined in sec. [..T] the scalar output case is analyzed in sec. B.2 In sec. B.3 second order expansion is assumed to illustrate the connection between the polynomial expansion and the recursive formula. Sequentially, we derive an alternative model with different factor sharing. This model, called Nested-MVP, has nested factor sharing format (sec. B.4
B.1 NOTATION
Our derivations rely tensors (i.e., multidimensional equivalent of matrices) and (tensor) products. We relay below the core notation used in our work, the interested reader can find further information
in the tensor-related literature (Kolda & Bader 2009 Debals & De Lathauwer 2017).
Symbols of variables: Tensors/maarices//eetoos are symbolized by calligzphicupprrase/loweeesssss boldface letters e.g., W.W.w. ,w.
Matrix products: The Hadamard product of A, B E | defined as and is equal to ((3))(1))) for the (i, j) element. The Khatri-Rao product of matrices A R and B IR is
14

---

Under review as a conference paper at ICLR 2021
denoted by A O B and yields matrix of dimensions (IJ) x N. The Khatri-Rat product for set of matrices { [m] [Rm mm=1 is abbreviated by A(i + A ② ..... A = O [m)
Tensors: Each element of an M order tensor W is addressed by M indices, i.e., (W)s W Am An M tensor W defined over the tensor space R/xxxxxxxx00 -xla where Im Z for 1,2,..., M. The mode-m unfolding
tensor W R x xxxx maps W matrix A-1 1s such that the tensor element Wi d2uuddr is mapped to the kym
W ((m) E IR xFm with In = III
M
k-1 In- The mode-m vector product of W with a vector u E R Fm, denoted by W u IR x Ta xexex Im-1 Im+1 x.xx Tur results in tensor of order M 1:
matrix element Winj where j
IM hml (ix 1)Jx with JR -11 kym
(W m fm......mmm...
Way ba....e Vim
(3)
We denote W X1 (1) u X3 u (M)
W WIT=>
u (m)
The CP decomposition (Kolda & Bader 2009) factorizes tensor into a sum of component rank-one tensors. The rank-R CP decomposition of an -odder tensor W written as:
R
W
[U]]] Up (UA........mm) UiMll -
Σ
u
u (2)
.... u (M)
(4)
where is the vector outer product. The factor matrices (U(m) [u] (m) R'.. x 12 M
u2
ft
collect the vectors from the rank-one components. By considering the mode-1 unfolding of W, the CP decomposition can written in matrix form as:
T
W( Up O
[[m]
(5)
M
The following lemma is useful in our method:
Lemma 1. For set of N matrices [A RRI.K Jv-1 and (B E RIXLIN equality holds:
J==1 the following
O A TT (0 B = (A Bpj) ............... ( B(N)
(6)
An indicative proof can be found in the Appendix of Chrysos et al. (2019)
B.2 SCALAR OUTPUT
The proposed formulation expresses higher-order interactions of the input variables. elaborate that, we develop the single output case below. That is, we focus on an element of the output vector, e.g.. a single pixel. the next few paragraphs, we consider the case of a scalar output 2y, with 7 E [1, o] when the input variables 2,, &n E IKd, To avoid cluttering the notation we only refer to the scalar output with x, the next few paragraphs.
As reminder, the polynomial of expansion order N N with output e R has the form:
n+1
x G(z,, kn)
Wlmme
x 221
2n
+ B
J-2
r-p+1
We assume second order expansion (N 2) and let denote an arbitrary scalar output of æ. The [1.2]
[1,1]] first order correlations can be expressed through the sums Σd= W Zn,A and Σx= w 21, The second order correlations include both auto- and cross-correlations. The tensors [y(2.1] and ()(33 capture the auto-correlations, while the tensor (1(222 captures the cross-correlations.
15

---

Collecting all the terms in an Under review as a conference paper at ICLR 2021
=11.1
=1.1
P.ll
+
11.
+
FILA
12.1] A,y
Fil
21A
[2.3] Y.Auu
RI
+ ...
Z1,
12.2 Pi
211,
Schematic for second order expansion with scalar output 2, R. The abbreviations 21,A, 2 are elements of 2; with A, d.. Similarly, 211,,, 21 are elements of 211. The first two terms (on the right side of the equation) the first-order correlations; the next two terms the second order auto-correlations. The last term expresses the second order cross-correlations.
Figure 4:
A pictorial representation of the correlations are captured in Fig. equation, each output is expressed as:
Ir = Br
+
Σ 1=1
[1.21 2n +
w 4A
+
Σ
w
|2.1| 7.
2n,
+
Σ
T.
12.3| w
LAA
Σ
w T.A. 2
[2,2]
pmm
(8)
where By R. Notice that all the correlations of up second order are captured in equation
B.3 SECOND ORDER DERIVATION FOR TWO-VARIABLE INPUT
In all our derivations, the variables associated with the first input have an 7 notation, e.g., Un. Respectively for the second input Zn. the notation T/ used.
Even though equation 7 enables any order of expansion, the learnable parameters increase exponen- tially, therefore we can use coupled factorization reduce the parameters. Next, we derive the factorization for a second order expansion (i.e., N 2) and then provide the recursive relationship that generalizes for an arbitrary order.
Second order derivation: For a second order expansion (i.e., N 2 in equation D we factorize each parameter tensor Vy[n.e), We assume coupled CP decomposition for each parameter as follows:
Let w CU
(1) [1. and wil.21 CUTT,
(1) [1,1] the parameters for n
Let w C(URm O "n and w/2.31 C(Up2 O Up.n)* capture the
(1)
(1)
second order correlations of single variable (2g and z, respectively).
The cross-terms are expressed 12 X2 z X3 Zn- The output of the elemen is 2 x3 also results in the same elementwise
- w [2.2] T,A., &u,, The product v [2.2]
(1) expression. Hence, to allow for symmetric expression, we factorize the term W= as the sum of the terms C C Un.r))" and C(U c Up.rn)". For each of the two terms, we assume that the vector- valued inputs are accordingly multiplied.
The parameters C' E Book U [m..). Rdxk (m 1,2 and @ = (T, IT) are learnable. The aforementioned factorization results in the following equation:
CU + CUT " c
T (=00)++ (1Nnn Unm)) ( c( o ((PpnoUp,i) (= + CUaun (zn +3
CU
An elementwise analysis (with scalar output) provided the Appendix (sec. B.2
16

---

Under review as a conference paper at ICLR 2021
This expansion captures the correlations (up to second order) of the two imput variables 21, Au- To make the proof more complete, we remind the reader that the recursive relationship (i.c., 2 in the main paper) is:
In = In-1 + (T..11 + U (m.11 * 2n-1
(10)
for N with = =UiL,121 U 11.11 and x CaN + B.
Claim 1. The equation 2 a special formal of polynomial that is visualized as Fig. of the
main paper. Equivalently, prove that ( follows the recursive relationship of T70).
Proof. We observe that the first two terms of equation Pare equal to Ca (from equation 10 By
9
applying Lemma iin the terms that have Khatri-Rao product, obtain:
3 Cz of ( (e (URun)2)) + (U
(U
U22n)2) (UR.mRa)) + (VErrzz))) (V%..nz))
(11)
Ca c(((TE,n))) (UE)
*01
Ca2 + B
The last equation is precisely the one that arises from the recursive relationship from equation 10
To prove the recursive formula for the 4 order expansion, similar pattern in sec.C of Poly- GAN (Chrysos et al. 2019) can be followed. Specifically, the difference here is that because of the two input variables, the auto- and cross-correlation variables should be included. Other than that, the same factor sharing is followed.
B.4 NESTED-MV VP MODEL FOR TWO- VARIABLE INPUT
The model proposed above (i.e.. equation 10), relies on a single coupled CP decomposition, however more flexible model can factorize each level with CP decomposition. To effectively do that, utilize learnable prper-parameters byn e R for n E [1, .N. which act as scaling factors for each parameter tensor. Then, polynomial of expansion order N e N with output R has the form:
N
n+2
22
x
G Z1; 2nd)
Σ p-2
Winni-11 x2 b2
I1 X1²ı
II r-p+1
X+Z1
+ +.3
(12)
1=3
To demonstrate the factorization without cluttering the notation, we assume a second order expansion
in equation T2
Second order derivation:
The second order expansion, i.e., N = 2, is derived below. We jointy factorize all parameters of equation 12 with a nested decomposition follows:
w C(Ap.m B and wil.21 C(A 2 BB)) (1)
First order parameters
(1)
Let 112.11 (1)
cf
A @
{ O Buj Ves and w
cf Ann
(An.n O Bui) Vaa
capture the second order correlations of a single variable (zu
and 2, respectively).
The cross-terms are included 1112.21 X2 lij X3 2 X Zn- The output of the element is expressed D. Ed
Ex w [2,2]
"eeeh, ((j w Similarly, the product v [2.2]
(2(2.2]
x2 b X3
17

---

Under review as a conference paper at ICLR 2021
Zn X4 2 has output Σ Σ -A T.K.p.. only change in the expressions is the permutation of the third and forth modes of the tensor; the rest of the expression remains the same. Therefore, to account for this symmetry we factorize the term wy(2.2] as the sum of two terms and assume that each term is multiplied
" R1 for the element, Notice that the
[2:21
by the respective terms. Let w C A o (1)
( (An.1n 3 Bpn (22) + Ap.m O
T
(Anin B111) 122
The parameters C Boxk (n.d) E R [n R [r] R for 1,2 and [T, IT are learnable. Collecting all the terms above and extracting C as common factor (we ommit C' below to avoid cluttering the notation):
(Ap,m 2 B(z)) (zu (p2) + (Ap..1 C
A O
(An.m1 6 Bm) 122 ( 2n 0 bb1)+
(An.11 3 B1n) M21
A O
(a (x0))bb)++
T
(An.11 B11 V/2)
12,
(z ④ an 0 bp1++
(13)
A 12. O
AAn 3 B11) 12
(zu 1 Ob11)
A T 12,11 Ale.n
b1a))
(u + AT 21)
VT Vial
(AT
1 AT.1)a)
B (BL())))(B((((b
The last equation is precisely recursive equation that can be expressed with the Fig. Jor equivalently the generalized recursive relationship below.
Z
A
A/211
A
[RMW.... or
Ap
Aw
Z11
Vp21
ViN
C S. B
G 211
1
B
Figure 5: Abstract schematic for 7 order approximation of x = G( z11) with Nested-MVP model. The inputs 21, 211 are symmetric in our formulation. We denote with sample from the noise distribution (e.g.. Gaussian), while 2ı symbolizes sample from a conditional input (e.g., class label or low-resolution image).
Recursive relationship: The recursive formula for the Nested-MVP model with arbitrary expansion order N N is the following:
Cr (Afr Alr (f..1122))
" (Vinsen- + bpm)
(14)
18

---

Under review as a conference paper at ICLR 2021
where n E [ and æ1 = (Af (A + Ah
BT (m)) The parameters C [1
Boxk A [m.d] E Raxk Vin] E Baxk B|n] E Ruuk for Φ = {1, T are learnable. Then, the output CaN + B.
The Nested-MVP model manifests an alternative network that relies on slightly modified assumptions on the decomposition. Thus, changing the underlying assumptions of the decomposition can modify the resulting network. This can be an important tool for domain-specific applications, e.g., when the comain-knowledee should be inserted in the last layers.
C BEYOND TWO VARIABLES
Frequently, more than one conditional inputs are required Yu et al. 2018b Xu et al. 2017 Maximov al. 2020). In such tasks, the aforementioned framework can be generalized more than two ipput variables. We demonstrate how this is possible with three variables; then it can trivially extended to an arbitrary number of input variables.
Let 2, Zn, Zm E K6 denote the three input variables. We aim to learn function that captures the higher-order interactions the input variables. The polynomial of expansion order N E with output Io has the form:
N
n+1 n
8
n+1
x = G(1,; 2n, 2mn)
Σ 8=p
x X
II T-p+1
x uu
xx
+ B
(15)
(=0+1
where 3 E Ro and Vylm.p.66 E Ro Xnd (for n [J and p, J [1, n + 1J) are the learnable parameters. As in the two woovvariable imput, the unknown parameters increase exponentially. To that end, we utilize joint factorization with factor sharing. The recursive relationship of such factorization is:
Iy = In-1 (UT 21 UT 2u + U (1 In-1
(16)
for n 2,..., with x = - U = + UT f and CæN B.
Notice that the pattern (for each order) is similar the two-variable imput: a) a different embedding is found for each input variable, b) the embeddings are added together, c) the result is multiplied elementwise with the representation of the previous order.
D CONCATENATION OF INPUTS
A popular method used for conditional generation is to concatenate the conditional input with the noise labels. However, as we showcase below, concatenation has two significant drawbacks when compared to framework. To explain those, will define concatenation model.
Let K K where K1, IK2 can be subset of real or natural numbers. The output T
of a concatenation layer x pT 2 where the symbol ' denotes the concatenation
and P E xx is affine transformation on the concatenated vector. The jth output is I Edh Pr Edz Pr,
Therefore, the two differences from the concatenation case are:
If the input variables are concatenated together we obtain an additive format, not multi- plicative that can capture cross-term correlations. That is, the multiplicative format does allow achieving higher-order auto- and cross- term correlations.
The concatenation changes the dimensionality the embedding space. Specifically, the input space has dimensionality d1 dz. That has significant toll on the size of the filters (i.e., it increases the learnable parameters), while still having an additive impact. On the contrary, framework does not change the dimensionality of the embedding spaces.
19

---

Under review as a conference paper at ICLR 2021
E IN-DEPTH DIFFERENCES FROM II-NET
In the next few paragraphs, we conduct an in-depth analysis of the differences between NT-Net and MVP. The analysis assumes knowledge of the proposed model, i.e., (2)
Chrysos et al. (2220) introduce II-Net as a polynomial expansion of a single input variable. Their goal is model functions towards using a single-input variable z, which can be noise in case of image generation or an image in discriminative experiments. The authors express the StyleGAN architecture (Karras al. 2019) as polynomial expansion, while they advocate that the impressive results can be attributed to the polynomial expansion.
G(z) as high-order polynomial expansions of z. Their focus is
To facilitate the in-depth analysis, the recursive relationship that corresponds to An N order expansion in II-Net is expressed as:
is provided below.
A (m)=) In-1 1 + In-1
(17)
for nt = .,..., N with 2 Pijz and x TEN + B. The parameters A, I are learnable.
In this work, we focus on conditional data generation, i.e., there are multiple input variables available as auxiliary information. The trivial application of II-Net would be to concatenate all the M input variables 21, 2m; ..,.... The input variable z becomes
Z žuii Zm: ... where the symbol denotes the concatenation. Then, the polynomial expansion of IT-Net can be learned on the concatenated z. However, there are four significant reasons that we believe that this is not as flexible as the proposed MVP.
When we refer to II-Net below, we refer to the model with concatenated input. In addition, let 2, E Ky 2u E K. denote the input variables where K1, 12 can be subset of real or natural numbers.
Parameter sharing: MVP allows additional flexibility in the structure of the architecture, since MVP utilizes different projection layer for each input variable. We utilize this flexibility to share the parameters of the conditional input variable; as we detail in (19), we set U|N.11] Un.1 on (2. If we want perform similar sharing in IT-Net, the formulation equivalent 17 would be (Aia1); = (Ami) for = di,...,4 dj + dz. However, sharing only part of the matrix might be challenging. Additionally, when A a convolution, the sharing pattern not straightforward be computed. Therefore, MVP enables additional flexibility to the model, which is hard to be included in II-Net.
Inductive bias: The inductive bias is crucial in machine learning (Zhao et al. 2018 however concatenating the variables restricts the flexibility of the model (i.e. II-Net). To illustrate that, let us use the super-resolution experiments an example. The input variable is the noise vector and z the (vectorized) low-resolution image. If we concatenate the two variables, then we should use a fully-comnected (dense) layer, which does not model well the spatial correlations. Instead, with MVP, we use a connecco layer for the noise vector and convolution for žn (low-resolution image). The convolution reduces the number of parameters and captures the spatial correlations in the image. Thus, by concatenating the variables, we reduce the flexibility of the model.
Dimensionality of the inputs: The dimensionality of the inputs might vary orders of magnitude, which might create an imbalance during learning. For instance, in dass-conditionaa generation con- catenating the one-hot labels in the input does not scale well when there are hundreds of classes (Odena et al. 2017). We observe a similar phenomenon in cass-conditional generation: in Cars (with 196 classes) the performance of TT-Net deteriorates considerably when compared to its (relative) performance in CIFAR10 (with 10 classes). On the contrary, MVP does not fuse the elements of the input variables directly, but it projects them into a subspace appropriate for adding them.
Order of expansion with respect to each variable: Frequently, the two inputs do not require the same order of expansion. Without loss of generality, assume that we need correlations up to N, and NI order (with N, Nrr) from z, and žn respectively. MVP includes different transformation
20

---

Under review as a conference paper at ICLR 2021
for each variable, i.e., 1]..1] [] for and (n,, for zu. Then, we set [n, 0 for Ny. On the contrary, the concatenation of inputs (in II-Net) constrains the expansion to have the same order with respect to each variable.
All in all, we can use concatenation to fuse variables and use I1-Net, however an inherently multivari- ate model is more flexible and can better encode the types of inductive bias required for conditional data generation.
F DIFFERENCES FROM OTHER NETWORKS CAST AS POLYNOMIAL NEURAL NETWORKS
number of networks with impressive results have emerged in (conditional) data generation the last few years. Three such networks that are particularly interesting in our context are Karras et al. (2019); Park al. (2019): Chen et al. (2019). We analyze below each method and how it relates to polynomial expansions:
Karras et al. 2019) propose an Adaptive instance normalization (AdaIN) method for unsupervised image generation. An AdaIN layer expresses a second-order interaction h = (A w) ())(hmm)), where is normalization, the convolution operator and w the transformed noise w = M (z)) (mapping network). The parameters are learnable, while ha. is the input to the AdaIN. Stacking AdaIN layers results in polynomial expansion with a single variable.
Chen et al. (219)) propose a normalization method, called sBN, to stabilize the GAN training. The method performs a 'self-modulation" with respect to the noise variable and optionally the conditional variable in the condiiinnai work. sBN injects the network layers with a multiplicative interaction of the input variables. Specifically, sBN projects the conditional variable into the space of the variable z, through embedding function. Then, the interaction of the two vector-like variables is passed through a fully-connected layer (and a ReLU activation function); the result injected into the network through the batch normalization parameters. If cast as a polynomial expansion, a network with sBN layers expresses a single polynomial expansior
we focus on the candsccondiiinnll setting that is closer generation setting. Henceforth,
Park et al. (2019) introduce spatially normalization, i.e., SPADE, to improve semantic image synthesis. Their model, referred to as SPADE in the remainder of this work, assumes semantic layout conditional input that facilitates the image generation. We analyze in sec. how to obtain the formulation of their spatially-adaptii normalization. If cast as a polynomial expansion, SPADE expresses a polynomial expansion with respect to the conditional variable.
The aforementioned works propose or modify the batch normalization layer to improve the per- formance or stabilize the training. while in our work we propose the multivariate polynomial as a general function approximation technique for conditional data generation. Nevertheless, given the interpretation of the previous works in the perspective of polynomials, we still can express them as special cases of MVP. Methodologically, there are two significant limitations that none of the aforementioned works tackle:
The aforementioned architectures focus on no or one conditional variable. Extending the frameworks to multiple conditional variables might not be trivial, while MVP naturally extends to arbitrarily many conditional variables.
Even though the aforementioned three architectures use (implicitly) a polynomial expansion, a significant factor is the order of the expansion. In our work, the product of polynomi- als enables capturing higher-order correlations without increasing the amount of layers substantially (sec. 3.2)
The formulation is derived from the public implementation of the authors.
'In MVP, we do not learn a single embedding function for the conditional variable. In addition, we do not project the (transformed) conditional variable to the space of the noise-variable. Both of these can he achieved by making simplifying assumptions on the factor matrices of MVP.
21

---

Under review as a conference paper at ICLR 2021
In addition the aforementioned methodological differences, work is the only polynomial expansion that conducts experiments on a variety of conditional data generation tasks. Thus, we both demonstrate methodologically and verify experimentally that MVP can be used for a wide range of conditional data generation tasks.
F.1 IN-DEPTH DIFFERENCES FROM SPADE
In the next few paragraphs, we conduct an in-depth analysis of the differences between SPADE and MVP.
Park al. (019) introduce spatially-adaptive normalization, i.c. SPADE, to improve semantic image synthesis. Their model, referred to as SPADE in the remainder of this work, assumes semantic layout as conditional input that facilitates the image generation.
The n model block applies normalization on the representation En-1 of the previous layer and then performs an elementwise multiplication with a transformed semantic layout. The transformed semantic layout can be denoted as A (1. 2n where denotes the conditional input to the generator. The output of this elementwise multiplication is then propagated to the next model block that performs the same operations. Stacking N such blocks results in an N order polynomial expansion which is expressed as:
En (APm.11)2n) * [n]
(V Zn + Blm(D(m))
(18)
where n E N] and æ1 A [1,1) The parameters C' E Book A [n,0] Rddk Vin
Rkck B
[n] Bu for o [I, are learnable. Then, the output CxN
SPADE as expressed in (18) resembles one of the proposed models of MVP (specifically (4)). In
particular, it expresses polynomial with respect to the conditional variable. The parameters [p. are set as zero, which means that there are no higher-order correlations with respect to the input variable 2, Therefore, our work bears the following differences from Park et al. (0119)
SPADE proposes normalization scheme that is only applied to semantic image generation. On the contrary. our proposed MVP can be applied to any conditional data generation task, e.g., class-conditional generation image-to -o-ratt--mage translation.
SPADE is special case of MVP. In particular, by setting i) A equal to zero, ii) A [p,7] in (14) equal zero, we obtain SPADE. In addition, MVP allows different assumptions on the decompositions which lead to alternative structure, such as (
SPADE proposes a polynomial expansion with respect to a single variable. On the other hand, our model extend to arbitrary number of input variables account for auxiliary labels, e.g.,
16
Even though SPADE models higher-order correlations of the conditional variable, still does not leverage the higher-order correlations of the representations (e.g., in the product polynomials) and hence without activation functions it might not work as well as the two-variable expansion.
Park et al. results using NVIDIA DGX with 8 V100 GPUs). Our goal is not compete in computationally heavy, large-scale experiments, but rather to illustrate the benefits of the generic formulation of MVP.
2019) exhibit impressive generation results with large-scale computing (i.e., they report
SPADE is an important baseline for work. In particular, we augment SPADE in wo ways: a) by extending it to accept both continuous and discrete variables in žn and b) by adding polynomial terms with respect to the input variable z, The latter model is referred as SPADE-MVE (details on the next section).
G EXPERIMENTAL DETAILS
Metrics: The two most popular metrics (Lucic et al. 2018 Creswell et al. 2018 for evaluation of the synthesized images the Inception Score (IS) (Salimans et al. 2016) and the Frechet Inception
22

---

Under review as a conference paper at ICLR 2021
Distance (FID) (Heusel et al. 2017). The metrics utilize the pretrained Inception network (Szegedy et 2015) 2015)) to extract representations of the synthesized images. FID assumes that the representations extracted follow Gaussian distribution and matches the statistics (i.e., mean and variance) of the representations between real and synthesized samples. Alternative evaluation metrics have been reported as inaccurate, e.g., in Theis al. (20T6), thus use the IS and FID. Following the standard practice of the literature, the IS is computed by synthesizing 5, 000 samples, while the FID computed using 10, 000 samples.
The IS is used exclusively for images of natural scenes as a metric. The reasoning behind that is that the Inception network has been trained on images of natural scenes. On the contrary, the FID metric relies on the first and second-order moments of the representations, which are considered more robust to different types of images. Hence, we only report IS for the CIFAR1O related experiments, while for the rest the FID is reported.
Dataset details:
There are five main datasets used in this work:
Large-scale CelebFaces Attributes (or CelebA for short) (Liu al., 2013) is Large-scale face attributes dataset with 202, 000 celebrity images. We use 160, 000 images for training our method.
Cars196 (Krause et al., 2013) is a dataset that includes different models of cars in different positions and backgrounds. Cars196 has 16, 000 images, while the images have substantially more variation than CelebA faces.
CIFARIO (Krizhevsky et al. 2014) contains 60, 000 images of natural scenes. Each image is of resolution 32 x 32 x 3 and is classified in one of the 10 classes. CIFARIO is frequently used as a benchmark for image generation.
The Street View House Numbers dataset (or SVHN for short) (Netzer et al., 201 has 100, 000 images of digits (73, 257 of which for training). SVHN includes color house- number images which are classified in 10 classes; each class corresponds SVHN images are diverse (e.g., with respect to background, scale).
digit
MNIST (LeCun et al.) 1998) consists of images with handwritten digits. Each images depicts single digit (annotated from 0 to 9) in 28 x 28 resolution. The dataset includes 60, 000 images for training.
Shoes Yu & Grauman 2014 Xie & Tu 2015) consists of 50, 000 images of shoes, where the edges of each shoe extracted (Isola al. 2017).
Handbags (Zhu et al. 2016 Xie & Tu 2015) consists of more than 130, 000 images of handbag items. The edges have been computed for each image and used as conditional input to the generator (Isola et al. 2017).
Anime characters dataset (Jin et al.) 2017) consists of anime characters that are generated based on specific attributes, e.g., hair color. The public version u contains annotations on the hair color and the eye color. We consider 7 classes on the hair color and 6 classes on the eye color, with a total of 14, 000 training images.
All the images of CelebA, Cars 196, Shoes and Handbags are resized to 64 x 64 resolution.
The discriminator structure is left the same for each experiment, we focus only on the generator architecture. All the architectures are based on two different generator schemes, i.e., the SNGAN (Miyato & Koyama 2018) and the polynomial expansion of Chrysos et al. (2020) that does not include activation functions in the generator.
Architectures:
The variants of the generator of SNGAN are described below:
SNGAN (Miyato & Koyama 2018): The generator consists of a convolution, followed by three residual blocks. The discriminator also based on successive residual blocks. The public implementation of SNGAN with conditional batch normalization (CBN) is used as the baseline.
'The version is downloaded following the instructions of htcpp.//githubb.com//ooee
An Go
23

---

Under review as a conference paper at ICLR 2021
SNGAN-MVP [proposed]: We convert the resnet-based generator of SNGAN an MVP model. To obtain MVP, the SNGAN is modified in two ways: a) the Conditional Batch Normalization (CBN) is converted into batch normalization (loffe & Szegedy 2013). b) the injections of the two embeddings (from the inputs) are added after each residual block, i.e. the formula of (2). In other words, the generator is converted product of two-variable polynomials.
Based on SNGAN-MVP, we replace each Hadamard product with concatenation. This implements the variant mentioned in sec. D
SNGAN-CONC:
SNGAN-SPADE (Park et al. 2019): : As described in sec. F.I SPADE is polynomial
with respect the conditional variable zn- The generator of SNGAN-MVP modified to perform the Hadamard product with respect to the conditional variable every time.
The variants of the generator of II-Net are described below:
TT-Net (Chrysos al. 2020): The generator is based on product of polynomials. The first polynomials use fully-connected connections, while the next few polynomials use cross-correlations,, The discriminator is based on the residual blocks of SNGAN. We stress out that the generator does not include any activation functions apart from hyperbolic tangent in the output space for normalization. The authors advocate that this exhibits the expressivity of the designed model.
The generator structure is based on IT-Net with two modifications: a) the Conditional Batch Normalization is converted into batch normalization (loffe & Szegedy 2015), b) the second-input is concatenated with the first (i.e., the noise) in the input of the generator. Thus, this is single variable polynomial, i.e., a II-Net, where the second-input vectorized and concatenated with the first. This baseline implements the II-Net described in sec.
TT-Net-SCOONC:
The generator of 11-Net is converted to MVP model with two modifi- cations: a) the Conditional Batch Normalization is converted into batch normalization (loffe) & Szegedy 2015), b) instead of having Hadamard product with single variable in 11-Net, the formula with the two-variable input (e.g., ( is followed.
MVP [proposed]:
GAN-CONC: Based on MVP, each Hadamard product is replaced by a concatenation. This implements the variant mentioned in sec.
GAN-ADD: Based on MVP, each Hadamard product is replaced by an addition. This modifies (4) to 2 (A A Ab,,112)) + (Vinjan- + b11))1)))
SPADE (Park al.) 2019): As described in sec. F.I SPADE defines a polynomial with respect to the conditional variable zn. The generator of II-Net is modified to perform the Hadamard product with respect the conditional variable every time.
SPADE-MVP [proposed]: This is variant develop bridge the gap between SPADE and the proposed MVP. Specifically, we augment the aforementioned SPADE twofold: a) the dense layers in the input space are converted into polynomial with respect to the variable 2, and b) we also convert the polynomial in the output (i.e., the rightmost polynomial in the Fig. 6 schematics) higher-order correlations of the variable z; that SPADE did not not originally include. This model still includes single variable polynomials, however the input in each polynomial varies and is not only the conditional variable.
polynomial with respect to the variable 2, This model captures
The two baselines GAN-CONC and GAN-ADD capture only additive correlations, hence they cannot effectively model complex distributions without activation functions. Nevertheless, they are added as reference point to emphasize the benefits of higher-order polynomial expansions.
An abstract schematic the generators that are in the form of products of polynomials depicted in Fig. 66 Notice that the compared methods from the literature use polynomials of single variable, while we propose polynomial with an arbitrary number of inputs (c.g., two-input shown in the schematic).
24

---

Under review as a conference paper at ICLR 2021
1
Polyd
Polyc-in
Polyu
Polyc
x
1-var
1-var
1-var
1-var
II-Net
Chrysos al.
2020
(oMB:
Z11
Polyo
Polyc-in
Polyu
Polyc-out
21
1-var
1-var
1-var
1-var
(b) II-Net-SICONC (baseline)
JIW....
Z11
Z1
Dense
Polyc-in
Polyu
Polyc
1-var
1-var
1-var
(c) SPADE
Park et al. 2019
Z11
Z
Polyd
Poly
Polyu
Poly
2-var
2-var
2-var
2-var
(d) MVP (proposed)
Figure 6: Abstract schematic of the different compared generators. All the generators are products of polynomials. Each colored box represents a different type of polynomial, i.c., the green box symbolizes polynomial(s) with dense layers, the blue box denotes convolutional or cross-correlation layers. The red box includes the up-sampling layers.
(a) II-Net implements a variae-variable G(z). II-Net enables clas-conniiional generation by using
polynomial for modeling functions conditional batch normalization (CBN). (b) An alternative to CBN is to concatenate the conditional variable in the input, in II-NetSICONN. This also enables the non-discrete conditional variables (e.g., fow-resolution images) to be concatenated. (c) SPADE implements a single-variabe polynomial for conditional image generation. The polynomial built with respect the conditional variable Z11- This is substantially different from the polynomial with multiple-input variables, i.c., MVP. Two additional differences are that (i) SPADE is motivated as a continuous conditional variables), while MVP can be used both for discrete and for continuous type variables, (ii) there no polynomial the dense layers in the SPADE. However, as illustrated in II-Net converting the dense layers into higher-order polynomial can further boost the performance. (d) The proposed generator structure.
method (i.e., for
Implementation details MVP: Throughout this work, we reserve the symbol æn for the condi- tional input (c.g.. class label). In each polynomial, reduce further the parameters by using the same embedding for the conditional variables. That is expressed as:
U Um. Up.
(19)
for n = 2, ......N N. Equivalently, that would be A = A in (14) Additionally, Nested-MVP performed better in our preliminary experiments, thus (14) design each polynomial. Given
25

---

Under review as a conference paper at ICLR 2021
the aforementioned sharing, the N order expansion is described by:
Zn A [m. + A fL.1122) (V (n + Bl( (m)
(20)
for n = 2....,N: N. Lastly, the factor An, is a convolutional layer in the case continuous conditional input, while it is conneced layer in the case of discrete conditional input.
H ADDITIONAL EXPERIMENTS
Additional experiments and visualizations are provided in this section. Additional visualizations for class-conditional generation provided in sec. H.. An additional experiment with class-conditional
generation with SVHN digits is performed in sec. H.2 An experiment that learns the translation of MNIST SVHN digits is conducted in sec. H.3
explore further the image-to-image translation, An attribute autribute-guided guided generation is performed in sec. H.5 to illustrate the benefit of our framework with respect to multiple, discrete conditional inputs. This is further extended in sec. (ec. where an experiment with mixed conditional input is conducted. Finally, an additional diversity diversity-inducing inducing regularization term is used to assess whether can further boost the diversity the synthesized images in sec. H.7
two additional experiments are conducted in sec.
H.1 ADDITIONAL VISUALIZATIONS IN CLASS-CONDITIONAL GENERATION
In Fig. the qualitative results of the compared methods in class-conditional generation on CIFARIO are shared. Both the generator of SNGAN and ours have activation functions in this experiment.
(c) MVP Figure 7: Qualitative results on CIFARIO. Each row depicts random samples from single class.
(a) Ground-trutt samples
(b) SNGAN Miyato al. 2018
In Fig. 8 samples from the baseline 11-Net (Chrysos et al. 2020 and our method are depicted for the class-conditional generation on CIFARIO 10. The images have a substantial difference. Similarly, in Fig. 9 visual comparison between II-Net and MVP is exhibited in Cars196 dataset. To
knowledge, no framework in the past has demonstrated such expressivity; MVP synthesizes images that approximate the quality of synthesized images from networks with activation functions.
In Fig. 10 an inter-class interpolation various compared methods in CIFAR10 are visualized. The illustrations of the intermediate images in SNGAN-CONC and SNGAN-ADD either blurry not realistic. On the contrary, in SPADE and MVP the higher-order polynomial expansion results in more realistic intermediate images. Nevertheless, MVP results in sharper shapes and images even in the intermediate results when compared to SPADE.
H.2 CLASS-CONDITIONAL GENERATION ON HOUSE DIGITS
An experiment cans-conniiional generation with SVHN is conducted below. SVHN images include (substantial) blur or other distortions, which insert noise in the distribution to be learned. In addition, some images contain contain central digit (i.e., based which the class is assigned), and
26

---

Under review as a conference paper at ICLR 2021
(a) I[-Net Chrysos et al. 2020
(b) MVP
Figure 8: Qualitative results CIFARIO when the generator does not include activation functions between the layers. Each row depicts random samples from a single class; the same class is depicted in each pair of images. For instance, the last corresponds to boats.
II-Net
(b) MVP
Figure 9: Qualitative results Cars196 when the generator does not include activation functions between the layers. Each row depicts random samples from a single class; the same class is depicted in each pair of images. The differences between the synthesized images dramatic.
partial visibility of other digits. Therefore, the generation of digits of SVHN is challenging for generator without activation functions between the layers.
Our framework, e.g., equation [4 does not include any activation functions. To verify the expressivity of our framework, we maintain the same setting for this experiment. Particularly, the generator does not have activation functions between the layers; there is only hyperbolic tangent in the output space for normalization. The generator receives a noise sample and a class as input, i.c. it is class-conditional polynomial generator.
27

---

Under review as a conference paper at ICLR 2021
(a) SNGAN-CONC
(b) SNGAN-ADD
(c) SNGAN-SPADE
(d) SNGAN-MVP
Figure 10: Inter-class linear interpolations across different methods. In inter-class interpolation, the class labels of the leftmost and rightmost images are one-hot vectors, while the rest are interpolated tn-between; the resulting images are visualized. Many of the intermediate images SNGAN-CONC and SNGAN-ADD are either blurry not realistic. On the contrary, in SPADE and MVP the higher-order polynomial expansion results in more realistic intermediate images. Nevertheless, MVP results in sharper shapes and images even in the intermediate results when compared to SPADE.
The results in Fig. 12
b) illustrate that despite the noise, MVP learns the distribution. As mentioned the main paper, our formulation enables both inter-class and intra-class interpolations naturally. In the inter-class interpolation the noise 2; is fixed, while the class 2n is interpolated. In Fig, d) several inter-class interpolations are visualized. The visualization exhibits that our framework is able synthesize realistic images even with inter-class interpolations.
H.3 TRANSLATION OF MNIST DIGITS TO SVHN DIGITS
An experiment on image translation from the domain of binary digits to house numbers is conducted below. The images of MNIST used as the source domain (i.e., the conditional variable 2n). while
28

---

Under review as a conference paper at ICLR 2021
(a) Class-conditional generation (b) Class-conditional generation
(c) Block-inpainting
Figure 11: Synthesized images by MVP in the (a), (b) class-conditional generation (sec. 4.1) and (b) block-inpainting (sec. 4.2) The networks do not include activation functions between the layers. In class- classccondiiional generation, each row depicts a single class. Notice how the MVP syntesiiees diverse images even the absence of activation functions.
9 9
2222
01011010000001001110000000000000
0
0
8
8
8
8858
8
3
8
8
1818188818
9
91
9
91
81
8
81
INO
10
50
10
20
5
5
5
0
2
22222
22
20
6
6
06
4
14
14
18
10
10
10
6:6
60 681
1672
16:5
46
9:
9
19999999
3
31
31
16
10
10 0
3
225
65
010
10
10
20
20
20
5
16
16
16
18
18
18
13
(a) Ground-trun samples
(b) MVP
(c) Intra-class interpolation(d) Inter-class interpolation
Figure 12: Synthesized images by MVP in the class-conditional SVHN: (a) Ground-truth samples, class, (c) Intra-class linear interpolation from
(b) Random samples where each depicts the a source (leftmost image) to the target (rightmost image), (d) inter-class linear interpolation. In inter-class interpolation, the class labels of the lefumost and rightmost images are one-hot vectors. while the rest are interpolated in in-between; the resulting images are visualized. In all three cases ((b)-dd), MVP synthesizes realistic images.
the images of SVHN used the target domain. The correspondence of the source the target domain is assumed to be many-to-many, i.e., each MNIST digit can synthesize multiple SVHN images. No additional loss used, the setting of continuous conditional input from sec. 4.2 is used.
The images in Fig. 13 illustrate that MVP can translate MNIST digits into SVHN digits. Additionally, for each source digit, there is significant variation in the synthesized images.
0
6
2
8
2
O
3
0
36
32
58
23
0
3 3
10
51
2
18
2
0
0
16
2
8
207
10
15
6F
2
8
12
10
3
58
2 8
0333
Figure 13: Qualitative results MNIST-to-SVHI translation. The first row depicts the conditional
input (i.c., a MNIST digit). The rows 2-6 depict outputs of the MVP when a noise vector is sampled per row. Notice that for each source digit, there is significant variation in the synthesized images.
29

---

Under review as a conference paper at ICLR 2021
H.4 TRANSLATION OF EDGES TO IMAGES
An additional experiment on translation is conducted, where the source domain depicts edges and the target domain the output image. Specifically, the tasks of edges-to-handbags (on Handbags dataset) and edges-to-shoes (on Shoes dataset) have been selected Isola et al. (017).
In this experiment, the MVP model of sec. 4.2 is utilized, i.e., a generator without activation functions between the layers. The training is conducted using only the adversarial loss. Visual results for both the case of edges-to-handhage and edges-to-shoes are depicted in Fig. 14 The first row depicts the conditional input En. i.e., an edge, while the rows 2-6 depict the synthesized images. Note that in both the case of handbags and shoes there is significant variation in the synthesized images, while they follow the edges provided as input.
(a) Figure 14: Qualitative results on edges-to-image translation. The first row depicts the conditional input (i.e., the edges). The rows 2-6 depict outputs of the MVP when we vary æ1. Notice that for each edge, there is significant variation in the synthesized images.
to- bigges-to-handole
(b) edges-to-shoes
H.S MULTIPLE, DISCRETE CONDITIONAL INPUTS
Frequently, than type of input conditional inputs available. Our formulation can be extended beyond two input variables (sec. []; we experimentally verify this case. The task selected is attribute-giided generation trained images of Anime characters. Each image is annotated with respect to the color of the eyes (6 combinations) and the color of the hair (7 combinations).
Since SPADE only accepts single conditional variable, should concatenate the two attributes in a single variable. We tried simply concatenating the attributes directly, but this did not work well. Instead, we can use the total number of combinations, which is the product of the individual attribute combinations, i.c., in our case the total number of combinations is 42, Obviously, this causes 'few' images to belong in each unique combination, i.e., there are 340 images on average that belong to each combination. On the contrary, there are 2380 images average for each eye color.
SPADE and II-Net are trained by using the two attributes in a single combination, while in our case, we consider the multiple conditional variable setting. In each case, only the generator differs depending on the compared method. In Fig. [5] few indicative images are visualized for each method; each row depicts single combination of attributes, i.c., hair and eye color. Notice that SPADE results in a single image per combination, while in SINCNNC there is considerable repetition each case. The single image in SPADE can be explained by the lack of higher-order correlations with respect to the noise variable 2,.
In addition to the diversity of the images per combination, image from every combination is visualized in Fig. 16, MVP synthesizes more realistic images than the compared methods of II-Net- SINCONC and SPADE.
30

---

Under review as a conference paper at ICLR 2021
(a) SINNO--INCCNC
(b) SPADE
(c) MVP
Figure 15: Each row depicts single combination of attributes, i.e., hair and eye color. Please zoom-in to check the finer details. The method of SPADE synthesizes a single image per combination. II-Net- SINCONC synthesizes few images, but not has many repeated elements, while some combinations result unrealistic faces, e.g., the 5th or the 7th row. On the contrary, MVP synthesizes much more diverse images for every combination.
(a) Ne-NCON
SPADE
(c) MVP
Figure 16: Each row depicts single chair color, while each column depicts single eye color. SPADE results in some combinations that do not follow the structure of the face, e.g., 3rd column in the last row. Similarly, II-Ne-SINCCON some of the synthesized images are not completely realistic, egg. penultimate row. MVP synthesizes images that resemble faces for every combination.
31

---

Under review as a conference paper at ICLR 2021
H.6 MULTIPLE CONDITIONAL INPUTS WITH MIXED CONDITIONAL VARIABLES
We extend the previous experiment with multiple conditional variables to the case of mixed conditional variables, i.e., there is one discrete and one continuous conditional variable. The discrete conditional variable captures the class label, while the continuous conditional variable captures the low-resolution image. Thus, the task is class-conditional super-resolution
We use the experimental details of sec. 4.2 in super-resolution 8x. In Fig. 17 we visualize how for each low-resolution image the results differ depending on the randomly sampled class label. The FID in this case is 53. 53.63, which is similar to the previous two cases. Class- conditonndil super-resolution (or similar tasks with multiple conditional inputs) can be of interest the community and MVP results in high-dimensional images with large variance.
H.7 IMPROVE DIVERSITY WITH REGULARIZATION
As emphasized in sec. various methods have been utilized for synthesizing more diverse images conditional image generation tasks. A reasonable question is whether our method can be used in conjunction with such methods, since it already synthesizes diverse results. Our hypothesis is that when MVP is used in conjunction with any diversity-inductei improve the diversity of the synthesized images. To assess the hypothesis, we conduct an experiment on edges to images that is a popular benchmark in such diverse generation tasks (Zhu et al., 2017b) Yang al. 2019).
technique, it will further
The plug-n-play regularization term of Yang et al. (2019) is selected and added to the GAN loss during the training. The objective of the regularization term Freg is to maximize the following term:
Figure 17: Three- variable input gener- ative model.
Greg min ||G(zL,, kn) - G( zn) ,
(21)
|Z
21 1
where r is predefined constant, 1; 21,2 are different noise samples. The motivation behind this term lies in encouraging the generator to produce outputs that differ when the input noise samples differ. our experiments, we follow the implementation of the original paper with T = 10.
The regularization loss of equation 21 is added to the GAN loss; the architecture of the generator remains similar to sec. H.4 The translation task is edges-to-handbags (on Handbags dataset) and edges-to-shoes (on Shoes dataset). In Fig. T8 the synthesized images are depicted. The regularization loss causes more diverse images to be synthesized (i.c., when compared to the visualization Fig. [4] that was trained using only the adversarial loss). For instance, in both the shoes and the handbags, new shades of blue are now synthesized, while yellow handbags can now be synthesized.
The empirical results validate the hypothesis that model can be used in conjunction with diversity regularization losses in order to improve the results. Nevertheless, the experiment in sec. HH.4| indicates that a regularization term necessary synthesize images that do not ignore the noise as feed-forward generators had previously.
I DIFFERENCE OF MVP FROM OTHER DIVERSE GENERATION TECHNIQUES
One challenge that often arises in conditional data generation is that one of the variables gets ignored by the generator (Isola al. 2017). This has been widely acknowledged in the literature, e.g., Zhu et al. (2017b) advocates that it is hard to utilize a simple architecture, like Isola et al. (2017). with noise. similar conclusion is drawn in InfoGAN (Chen al. (016)) where the authors explicitly mention that additional losses are required, otherwise the generator is 'free to ignore' the additional variables. To mitigate this, a variety of methods have been developed. We summarize the most prominent methods from the literature, starting from image-to-ing to-iaggg translation methods:
BicycleGAN (Zhu et al. (0176) proposes a framework that can synthesize diverse images in image-to to--tt----nm to-image translation. The framework contains 2 encoders, 1 decoder and 2 discriminators. This results in multiple loss terms (e.g., eq.9 of the paper). Interestingly, the authors utilize a separate training scheme for the encoder-decoder and the second encoder
32

---

Under review as a conference paper at ICLR 2021
(a) edges-to-handhags
(h) edges-to-shoes
Figure 18: Qualitative results edges-to-image translation with regularization loss for diverse generation (sec. H.7). The first row depicts the conditional input (i.e.. the edges). The rows 2-6 depict outputs of the MVP when vary 21. Diverse images synthesized for each edge. The regularization loss results in 'new' shades blue to emerge in the synthesized images both the shoes and the handbags cases.
as training together' 'hides the information of the latent code without learning meaningful modes'.
Almahaini et al. 2018) augment the deterministic mapping of CycleGAN (Zhu et al. 2017a) with a marginal matching loss. The framework learns diverse mappings utilizing the additional encoders. The framework includes 4 encoders, 2 decoders and discriminators.
MUNIT (Huang et al. 2018) focuses on diverse generation in unpaired saage-to-image translation. MUNIT demonstrates impressive translation results, while the inverse translation is also learnt simultancously. That is, in case of edges-to-shoce the translation shoes-to- edges is also learnt during the training. The mapping learnt comes at the cost of multiple network modules. Particularly, MUNIT includes 2 encoders, decoders, 2 discriminators for learning. This also results in multiple loss terms (e.g., eq.5 of the paper) along with additional hyper-parameters and network parameters.
Drit++ (Lee et al. 2020) extends unpaired image-to-image translation with disentangled representation learning, while they allow multi-domain image-to-image translations. Drit++ uses 4 encoders, 2 decoders, 2 discriminators for learning. Similarly to the previous methods, this results in multiple loss terms (e.g., eq.6-7 of the paper) and additional typer-parameters.
Choi et al. (020) introduce a method that supports multiple target domains. The method includes four modules: generator, mapping network, style encoder and discriminator. All modules (apart from the generator) include domain-specific sub-networks in case of mul- tiple target domains. To ensure diverse generation, Choi et al. (2020) utilize regularization loss (i.c., eq. 3 of the paper), while their final objective consists of multiple loss terms.
The aforementioned frameworks contain additional network modules for training, which also results additional syper-parameters the loss-function and the network architecture. Furthermore, the frameworks focus exclusively image-to-image translation and not all conditional generation cases, e.g., they do not tackle class-conditional or attribute-base generation.
An interesting technique for diverse, caas-conniiional generation is the self-conditional GAN of Liu cct al. (2020). The method conditions the generator with pscudo-labels that are automatically derived from clustering on the feature space of the discriminator. This enables the generator synthesize more diverse samples. This method is orthogonal to our, i.e., the generator of Liu et al.) (2000) can be replaced with MVP.
Using regularization terms in the loss function has been an alternative way to achieve diverse generation. Mao al. (2019) Yang et al. (2019) propose simple regularization terms that can be
33

---

Under review as a conference paper at ICLR 2021
plugged into any architecture encourage diverse generation. Lee et al. (2019) propose two variants of regularization term, with the 'more stable variant' requiring additional network modules.
We emphasize that method can be used in conjunction with many of the aforementioned techniques to obtain more diverse examples. We demonstrate that this possible in an experiment sec. H.7
34


Reviews:
[{"review_id": "yoXjDxxZUSv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Post rebuttal (round #3)**\n\nThanks to the authors' effort on the rebuttal. Despite the extensive efforts, I feel the review/rebuttal iteration is not satisfactory, possibly due to some miscommunication.\n\nTo be clear, I want to re-emphasize that significant parts of my concerns were about **misleading claims** on prior work, and comparison to them was the next step.\n- For example, I just wanted to clarify that the claim \"type A ignores the noise and cannot learn the stochastic mapping\" is wrong. The paper could simply fix the claim instead of including a massive related work section. Maybe my review also has some responsibility: I could simply say **fix** the wrong claims, instead of indirectly delivering by pointing them out.\n\nAlso, as I explicitly mentioned the concerns A,C,D,F, the rebuttal could address them point-by-point.\n- In particular, I'm not convinced that cBN/sBN is **not applicable** to continuous conditions, as sBN predicts the BN parameters from the continuous latent variable.\n\nDespite the remaining concerns, I raised the score (from 5) to 6 as the architectures with multiplicative interactions are an important and timely topic. However, I think the paper is on the borderline, and the rebuttal and revised paper could be much stronger.\n\n------\n\n**Summary**\n\nThis paper extends $\\Pi$-Net, deep polynomial neural networks, to a multivariate setting. The proposed network, MVP, is applied to various conditional GANs, including discrete, continuous, and mixed condition scenarios.\n\n\n**Pros**\n\n- Extending $\\Pi$-Net to multivariate setting is a natural and necessary research direction.\n- Good experimental results on class-conditional, image-conditional, and mixed (class + image)-conditional scenarios.\n\n\n**Concerns/questions**\n\nI. Novelty over $\\Pi$-Net is not significant\n\n- $\\Pi$-Net has shown that deep polynomials can be useful for an unconditional generation. Extending it to conditional generation (using two variables) is quite straightforward. While the paper compares with other design choices (e.g., SICONC and SPADE), it is natural that MVP performs the best since it has the strongest expressive power.\n- While the paper claims that \"unifying discrete and continuous conditions\" is a key property of MVP, standard conditional GANs can also handle those cases and are already discussed in the literature. For example, [1] considers image (face) + class (quantized age) conditions.\n- While the paper claims that \"network without activation function\" is one of the main contributions, it was originally investigated and heavily discussed in $\\Pi$-Net.\n\n[1] Antipov et al. Face Aging With Conditional Generative Adversarial Networks. ICIP 2017.\n\n\nII. Wrong claims on the (drawbacks of) prior work\n\nThe paper claims that prior conditional GANs: (Type A) \"encoder network to obtain representations that are independent of the conditional variable\" and (Type B) \"directly concatenate the labels in the latent space\" have several drawbacks. However, the claims are incorrect, as stated below:\n- Type A does not ignore the noise and can learn the stochastic mapping (with proper training) [2,3]\n- Type A can be successful for discrete conditions [1]\n- Type B can scale beyond 10 class, especially with nonlinear mapping such as conditional BN [4,5]\n\nThe paper also claims that \"inter-class interpolations in CIFAR10 have not emerged\", but there is no backup for the claim. To verify this, the paper should compare the generated samples of standard GANs and MVP in Figure 2.\n\n[2] Zhu et al. Toward Multimodal Image-to-Image Translation. NeurIPS 2017.\\\n[3] Huang et al. Multimodal Unsupervised Image-to-Image Translation. ECCV 2018.\\\n[4] Miyato & Koyama. cGANs with Projection Discriminator. ICLR 2018.\\\n[5] Brock et al. Large Scale GAN Training for High Fidelity Natural Image Synthesis. ICLR 2019.\n\nIII. Why polynomial should work better than standard GANs?\n\nThe paper claims that MVP performs better than standard GANs in various conditional generation setups.\n- Is there any intuition that MVP should work better than standard GANs?\n- How about the number of parameters or sampling speed? Is the comparison (in Table 2-4) fair in terms of complexity?\n\n**Rating**\n\nDue to the concerns above, I currently recommend a rating of 5.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IBAufCv8RWB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a conditional generation framework (cGAN) that bridges the gap between discrete and continuous variable used in the generation. They do so by proposing a new network architecture that implements higher order multi variate polynomials (MVP). They show that MVP generalizes well to different types of conditional variables and has good expressivity even in the absence of activation functions.\t\n\nPros:\nThe figures are succinct and informative, giving a clear picture of the points the authors want to illustrate. I appreciate the multiple evaluations with mean and standard deviation reported. \n\nThe methods section is well-written and the intuition provided to readers who are not familiar with the concept of polynomial networks is very useful.\n\nCons/Comments:\nSection 2.1 Discrete conditional variable: the authors claim that conditional normalization might be an obstacle towards generalizing to unseen classes. They do not show their method is able to generalize. \n\nSection 2.1 Continuous conditional variable: The authors do not cite or mentioned papers that do multi modal image generation [1,2,3] that display diversity and do not require pixel-wise loss such as l1 or perceptual. They single out one-to-one translation models that are not designed for such tasks. \n\nTable 2: Why is there a huge disparity between IS and FID scores of SNGAN-CONC compared to SNGAN? The IS is slightly better but the FID is significantly worse. Furthermore, the FID of the original SNGAN paper [4] is 21.7 and the FID of BigGAN [5] is 14.73. What causes the disparity the results between the papers? In my experience, BigGAN performs much better than SNGAN yet the FID of SNGAN quoted in this paper is 14.7, virtually the same as BigGAN.\n\nSection 4.1 Resnet-based generator: The authors claim that inter-class interpolations have been done for other datasets but not CIFAR. Is BigGAN capable of doing inter-class interpolations for CIFAR since it is able to do for ImageNet?\n\nThe experiments include conditional generation based on discrete variables and continuous variables. One of the main comparisons is with SPADE. However, SPADE was designed for image generation from semantic maps, a task not tested in this paper. Super resolution was chosen as the one of the two tasks for continuous variable. This is a task not traditionally done with multi-modal output in mind. In my opinion, this is not the most useful task to base the experiments on. A more interesting task could be image-to-image translation or semantic image synthesis.\n\nThe experiments are all done at 64x64 resolution and the datasets are relatively simple compared to those used in SOTA models (512x512 ImageNet on BigGAN and 1024x1024 FFHQ on StyleGAN [6]). The quantitative improvements on the most commonly used ResNet architecture is minimal. While MVP remains expressive without activations and improves significantly using a Π net architecture, those to my knowledge are not settings widely used now in literature.  \n\n\nIn general, this paper provides a clear unified framework for conditional image generation. The method is well explained and illustrated. However, their improvements on a commonly used architecture (ResNet) is minimal. It is my opinion that more justification and evidence is needed on why MVP is needed under common settings.\n\n[1] Choi, Yunjey, et al. \"Stargan v2: Diverse image synthesis for multiple domains.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[2] Huang, Xun, et al. \"Multimodal unsupervised image-to-image translation.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n\n[3] Lee, Hsin-Ying, et al. \"Drit++: Diverse image-to-image translation via disentangled representations.\" International Journal of Computer Vision (2020): 1-16.\n\n[4] Miyato, Takeru, et al. \"Spectral normalization for generative adversarial networks.\" arXiv preprint arXiv:1802.05957 (2018).\n\n[5] Brock, Andrew, Jeff Donahue, and Karen Simonyan. \"Large scale gan training for high fidelity natural image synthesis.\" arXiv preprint arXiv:1809.11096 (2018).\n\n[6] Karras, Tero, Samuli Laine, and Timo Aila. \"A style-based generator architecture for generative adversarial networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5cieSyV6La", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a framework for generating conditional data using multivariate polynomials, which treats both discrete and continuous conditional variables in a unified way. From my perspective,  neither the effect nor the actual use is very clear.\n\nBelow are my major concerns:\n\n1. The author compares the proposed method with multiple methods on the class-conditional generation and image2image translation tasks, but none of the comparison methods are state-of-the-art. The choice of dataset and task is not very appropriate. \nWhy compare with SPADE on the class conditional generation and super-resolution tasks? SPADE is the SOTA of  **semantic image synthesis** in 2019. It is unfair to compare specific methods for inappropriate tasks.\nIn addition, the selected dataset is also weird. Why not use a standard super-resolution dataset or use downsampled FFHQ and CelebAHQ images for face super-resolution tasks. \n2. The author mentioned that the disadvantage of the traditional encoder/decoder architecture (pix2pix Isola et al., 2017) is that it ignores noise,  but in fact, this problem was solved to a certain extent as early as 2017 e.g. (BicycleGAN zhu al., 2017). In fact, in Figure 4, I did not see enough of the variability caused by the noise claimed in this paper, even though the task is so simple.\n3. Although the method proposed in this paper is novel, I am not sure it really makes sense in actual complex input scenarios. Besides, the results mentioned in this paper do not show any superiority over traditional architectural design.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new network architecture that implements higher order multivariate polynomials (MVP). They show that MVP generalizes well to different types of conditional variables, and can be applied to a broad range of tasks.   However, unifying discrete and continuous conditions and network without activation function are both well studied in literatures. The inappropriate discussions on the prior works and the advantages of the proposed method over prior approaches are not clearly justified.  Although outperforming SOTA is not necessary, the compared methods need to be well chosen which can provide convincing evidence on why MVP is needed under common settings.
Decision: Reject