Conditional Generation Using Polynomial Expansions
Authors: Grigorios Chrysos
Markos Georgopoulos
Yannis Panagakis
Venue: NeurIPS 2021 Poster (2021)
Keywords: polynomial expansions
polynomial neural networks
GAN
VAE
unseen combinations
conditional data generation

Abstract: Generative modeling has evolved to a notable field of machine learning. Deep polynomial neural networks (PNNs) have demonstrated impressive results in unsupervised image generation, where the task is to map an input vector (i.e., noise) to a synthesized image. However, the success of PNNs has not been replicated in conditional generation tasks, such as super-resolution. Existing PNNs focus on single-variable polynomial expansions which do not fare well to two-variable inputs, i.e., the noise variable and the conditional variable. In this work, we introduce a general framework, called CoPE, that enables a polynomial expansion of two input variables and captures their auto- and cross-correlations. We exhibit how CoPE can be trivially augmented to accept an arbitrary number of input variables. CoPE is evaluated in five tasks (class-conditional generation, inverse problems, edges-to-image translation, image-to-image translation, attribute-guided generation) involving eight datasets. The thorough evaluation suggests that CoPE can be useful for tackling diverse conditional generation tasks. The source code of CoPE is available at https://github.com/grigorisg9gr/polynomial_nets_for_conditional_generation.

Conditional Generation Using Polynomial Expansions
Grigorios G Chrysos
Markos Georgopoulos
EPFL, Switzerland
Imperial College London, UK
grigorios. ctt.::ssss.crsssspp......oo.
wr.gerggoooollsst ieme@rmpppriee www.oosssmmmemaaaa....mmaa ac..
Yannis Panagakis
University of Athens, GR yannispēdi, uua.
Abstract
Generative modeling has evolved to notable field of machine learning. Deep polynomial neural networks (PNNs) have demonstrated impressive results in unsu- pervised image generation, where the task is to map an input vector (i.e., noise) synthesized image. However, the success of PNNs has not been replicated in conditional generation tasks, such as super-resolution. Existing PNNs focus on single-variable polynomial expansions which do not fare well to two-variable inputs, i.e., the noise variable and the conditional variable. In this work, we introduce a general framework, called CoPE, that enables a polynomial expan- sion of two input variables and captures their auto- and cross-correlations, We exhibit how CoPE can trivially augmented to accept an arbitrary number input variables. CoPE is evaluated in five tasks conds-conniiionall generation, inverse problems, edges-to-imane translation, image-to-image translation, attribute- guided generation) involving eight datasets. The thorough evaluation suggests that CoPE can be useful for tackling diverse conditional generation tasks. The source code of CoPE is available polynomia sylyoommallettteeeee nets _frrr__o______ theeessor_oooeettteeeeeeeeeeeeee coor_condiiionneeoeeoeeoeeeeeoee cot.aiiiiollleeeeeetttoono. generation.
https://git
com/ siggorisgggs
Introduction
Modelling nigh-dimensional distributions and generating samples from complex distributions are fundamental tasks in machine learning. Among prominent generative models, StyleGAN [Karras et al., 2019] has demonstrated unparalleled performance in unsupervised image generation. Its success can be attributed to the higher-order correlations of the input vector captured by the generator. As Chrysos et al. [2019] argue, StyleGAN' is best explained as a deep polynomial neural network (PNN). PNNs have demonstrated impressive generation results in faces, animals, cars [Karras al., 2020b], paintings, medical images [Karras et al., 2020a]. Nevertheless, PNNs have to demonstrate similar performance conditional generation tasks, such as super-resolution or image-to-imen
translation.
In contrast to unsupervised generators that require a single-variable input 2, in conditional generation (at least) two inputs required: i) (or more) conditional variables c, e.g., low-resolution image, and ii) a noise sample 2. A trivial extension of PNNs for conditional generation would be to concatenate all the input variables into fused variable. The fused variable is then the input the single-variable polynomial expansion of PNNs. However, the concatenation reduces the flexibility of the model significantly. For instance, concatenating noise vector and vectorized low-resolution image results in sub-optin cub-optimal super-resolution, since the spatial correlations of the input image are lost in the vectorization. Additionally, the concatenation of the vectorized conditional variable and
This work focuses on the generator network; any reference StyleGAN refers to its generator.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

leads to huge number of parameters when we use fully-connected layer as typically done in the input of StyleGAN, especially when e depicts an image.
In this work, we introduce a framework, called CoPE, for conditional data generation. CoPE resorts to multivariate polynomials that capture the higher-order auto- and cross-correlations between the two input variables. By imposing tailored structure in the higher-order correlations, we obtain an intuitive, recursive formulation for CoPE. The formulation enables different constraints to be applied to each variable and its associated parameters. In CoPE, different architectures can be defined simply by changing the recursive formulation. Our contributions can be summarized as follows:
We introduce a framework, called CoPE, that expresses high-order, multivariate polynomial for conditional data generation. We exhibit how CoPE can be applied on diverse conditional generation tasks.
We derive two extensions to the core two-variable model: a) we augment the formulation to enable an arbitrary number of conditional input variables, b) we design different architectures that arise by changing the recursive formulation.
CoPE is evaluated on five different lasks (class-conditional generation, inverse problems, edges-to-image translation, amage-to-image translation, attribute-guided generation); overall eight datasets are used for the thorough evaluation.
The diverse experiments suggest that CoPE can be useful for a variety of conditional generation tasks, e.g., by defining tpsk-specific recursive formulations. To facilitate the reproducibility, the source code is available at https: / //gi higtthu....m com/ httpl/cccooggggrroeeeeeeee httpe/gggrooooooooooeooo a nets_
2 Related work
Below, we review representative works in conditional generation and then we summarize the recent progress in multiplicative interactions (as low-order polynomial expansion).
2.1 Conditional generative models
The literature on conditional generation is vast. The majority of the references below focus on Generative Adversarial Networks (GANs) [Goodfellow et al., 2014] since GANs have demonstrated the most impressive results to date, however similar methods can be developed for other generative models, such as Variational Auto-encoders (VAEs) [Kingma and Welling, 2014]. Four groups of conditional generative models are identified below based on the type of conditional information.
The first group is the class-conditionul generation [Miyato et al., 2018, Brock et al., 2019, Kaneko et al., 2019], where the data are divided into discrete categories, e.g., a cat or a ship. During training class label is provided and the generator should synthesize sample from that category. One popular method of including class-conditional information is through conditional normalization techniques [Dumoulin et al., 2017, De Vries al., 2017]. An alternative way is to directly concatenate the class labels with the input noise; however, as Odena et al. [2017] observe, this model does not scale well to a hundred thousand classes.
The second group is inverse problems that have immense interest for both academic and commercial reasons [Sood et al., 2018, You et al., 2019]. The idea is to reconstruct latent signal, when corrupted measurements are provided [Ongie et al., 2020]. Well-known inverse problems in imaging include super-resolution, deblurring, inpainting. Before the resurgence of deep neural networks, the problems were tackled with optimization based techniques [Chan and Chen, 2006, Levin et al., 2009]. The recent progress in conditional generation has fostered the interest in inverse problems [Ledig et al., 2017, Pathak et al., 2016, Huang et al., 2017]. A significant challenge that is often overlooked the literature [Ledig al., 2017, Yu et al., 2018a] is that there are many possible latent signals for each measurement. For instance, given a now-resolution image, we can think of several high-resolution images that when samplsa provide the same low-resolution image.
Another group is touge- translation. The idea is to map an image from one domain to an image to another domain, e.g., day-time image to a night-time image. The influential work of Isola et al. [2017] has become the reference point for mage-to-imaae translation. Applications condi- tional pose generation [Ma et al., 2017, Siarohin al., 2018], conditional video generation [Wang

---

Table 1: Comparison of polynomial neural networks (PNNSs) Even though the architectures' of Karras et al. [2019], Chen et al. [2019], Park et al. [2019] were not posed as polynomial expansions, we believe that their success can be (partly) attributed the polynomial expansion (please check sec. F for further information). II-Net and StyleGAN are not designed for conditional data generation. In practice, learning complex distributions requires high-order polynomial expansions; this can be effectively achieved with products of polynomials as detailed in sec. 3.2. Only IT-Net and CoPE include such a formulation. The columns on discrete and continuous variable refer to the type of conditional variable the method was originally proposed c.g., sBN was only tried on class- conditional generation. Additionally, the only work that enables multiple conditional variables (and includes related experiments) is the proposed CoPE.
Attributes of polynomial like networks.
products of polynomials
discrete
continuous cond. variable
multiple cond. variables
Model
II-Net [Chrysos ct al., 2020]
X
X
StyleGAN [Karras al., 2019]
X
x
X
sBN [Chen et al., 2019]
x
x
X
SPADE [Park et al., 2019]
X
CoPE (ours)
et al., 201 8a] or generation from semantic labels [Wang et al., 2018b] have appeared. Despite the success, converting the mapping from one-to-one to one-to-many, i.e., having multiple plausible outputs for a single input image, has required some work [Zhu et al., 2017b, Huang et al., 2018]. A more dedicated discussion on diverse generation is deferred to sec. I.
The fourth group uses multiple conditional variables for generation, e.g., sttributeggiiee gener- ation [Choi et al., 2018]. These methods include significant engineering (c.g.. multiple discrimi- nators [Xu et al., 2017]. auxiliary losses). The influential InfoGAN [Chen et al., 2016] explicitly mentions that without additional losses the generator is 'free to ignore' the additional variables.
Each technique above is typically applied to single group of conditional generation tasks, while our goal is to demonstrate that CoPE can be applied to different tasks from these groups.
2.2 Multiplicative interactions
Multiplicative connections have long been adopted in machine learning (Shin and Ghosh, 1991, Hochreiter and Schmidhuber, 1997, Bahdanau et al., 2015, Rendle, 2010]. The idea is to combine the inputs through elementwise products or other diagonal forms. Jayakumar et al. [2020] prove that second order multiplicative operators can represent greater class of functions than classic feed-forward networks. Even though we capitalize on the theoretical argument, our framework can express any higher-order correlations while the framework of Jayakumar al. [2020] is limited to second order interactions.
Higher-order correlations have been studied in the tensor-related literature [Kolda and Bader, 2009, Debals and De Lathauwer, 2017]. However, their adaptation in modern deep architectures has been slower. IT-Net [Chrysos et al., 2020] resorts to a high-order polynomial expansion for mapping the input z to the output æ = G(z). I--eet focuses on a single-variable polynomial expansion; an in-depth difference of our work with I1-Net can be found in sec. E in the supplementary. Two efforts on conditional generation which can be cast as polynomial expansions are SPADE [Park et al., 2019] and sBN [Chen al., 2019]. SPADE can be interpreted as a single-variable polynomial expansion with respect to the conditional variable c. SPADE does not capture the higher- cross-core coss-correlations between the input variables. Similarly, sBN can be interpreted as a polynomial expansion of the two variables for class-con generation. However, SPADE and sBN do not use the product of polynomial formulation, which enables high-order expansions without increasing the number of layers [Chrysos et al., 2020]. Importantly, SPADE and sBN are constructed for specific applications (i.e., semantic image generation and mnsupervieeddclas-coniiii remains unclear whether PNN can effectively tackle purppoe conditional generation task.
generation respectively) and
3

---

3 Method
In the following paragraphs we introduce the two-variable polynomial expansion (sec. 3.1), while the detailed derivation, along with additional models deferred the supplementary (sec. B). The crucial technical details, including the stability of the polynomial, are developed in sec. 3.2. We emphasize that multivariate polynomial can approximate any function [Stone, 1948, Nikol'skii, 2013), i.e., a multivariate polynomial is a universal approximator.
hattiion.eennnos/mattrees///eeee by calli-
Table 2: Symbols
are graphiclluperrasee/oooerasss
symbolized
boldface letters e.g.. W. W.w. The mode-m vector product of W (of order M) with vector u R/m is W X u and results in a tensor of order M 1. We assume that Ii T4 1 when >> b. The core symbols are summarized in Table while detailed tensor notation is deferred to the supplementary (sec. B.1).
Symbol
Role
N
Expansion order of the polynomial
k
Rank of the decompositions
21, 21
Inputs to the polynomial
n.
Auxiliary variables
Winel
Parameter tensor of the polynomial
U [n]] C
Learnable parameters
Hadamard product
3.1 Two input variables
Given two input variables 21, 2n E IKd where S R N, the goal is to learn function G Kaxd - I that captures the higher-order correlations between the elements of the two inputs. can learn such higher-order correlations as polynomials of two input variables. polynomial expansion of order N E N with output E I (such that æ G(2,, 2nr) has the form:
1++
Wlmel
π j-2
x
II r-p+1
x 3
where B E R' and Wlnmel E Rox11=-- Xnd for E [1,N], [1, + 1] are the learnable (independent) variables, hence the and p as
parameters. The expansion depends auxiliary variables. The two products of (1) do not overlap, i.e., the first multiplies the modes [2, (of wllmell with and the other multiplies the modes Le + 1, + with zn.
Z
Upi
(2
U
[MMW....
U
U2n11
U
or
Z11
S.. B
G G 211)
Figure 1: Abstract schematic for Th order approximation of x G(z,, z1r). The inputs 21, Z1r are symmetric in our formulation. We denote with æ a noise vector, e.g., a samples from Gaussian distribution, while æ symbolizes sample from conditional input (e.g., class label low- resolution image).
Recursive relationship: The aforementioned derivation can be generalized to an arbitrary expansion order. The recursive formula for arbitrary order N N is the following:
x In-1 (Uir.112 + U U7..11721 #n-1
"To avoid cluttering the notation we use same dimensionality for the two inputs. However, the derivations apply for different dimensionalities, only the dimensionality of the tensors change slightly.

---

for n = =2... 2...... N with œ1 = UR. f..na + U (1.1 2n and x = CaN + B. The parameters C' Roxk; U((n,o] E Rddk for n I,....N N and Φ = {I, are learnable.
The intuition behind this model is the following: An embedding is initially found for each of the two input variables, then the two embeddings are added together and they are multiplied elementwise with the previous approximation. The different embeddings for each of the input variables allows us implement Un.J] and Un.IrJ with different constraints, e.g., U U[n,1] to be dense layer and (1, to be convolution.
(a) Random samples per class
(b) Intra-class interpolation
(c) Inter-class interpolation
Figure 2: Synthesized images by CoPE in the class consitonndi CIFARIO (with resnet-based genera- tor): (a) Random samples where each row depicts the same class, (b) Intra-class linear interpolation from source the target, (c) inter-class linear interpolation. In inter-class interpolation, the class labels of the leftmost and rightmost images are one-hot vectors, while the rest are interpolated in-between; the resulting images visualized. In all three cases, CoPE synthesizes realistic images.
3.2 Model extensions and technical details
There are three limitations (2). Those are the following: a) (2) describes a polynomial expansion of two-variable input, b) each expansion order requires additional layers, c) high-order polynomials might suffer from unbounded values. Those limitations are addressed below.
Our model can be readily extended beyond two varviariable input; an extension with varialla input developed sec. C. The pattern (for each order) is similar to the two-variable input: a) a different embedding is found for each input variable, b) the embeddings are added together, c) the result is multiplied elementwise with the representation of the previous order.
The polynomial expansion of (2) requires O(N) layers for an 1ith order expansion. That is, each new order n of expansion requires new parameters Ui,.1] and UI], However, the order of expansion can be increased without increasing the parameters substantially. To that end, we can capitalize on the product of polynomials. Specifically, let N₂ be the order of expansion of the first polynomial. The output of the first polynomial is fed into a second polynomial, which has expansion order of N2. Then, the output of the second polynomial will have an expansion order of Nı N2. The second polynomial of degree N2 can either be polynomial of one variable (i.e., the output of the previous polynomial) or more variables (i.e., the output the previous polynomial and one or more of the inputs). In both cases the total degree of the output of the second polynomial will be N₁ N₂. The choice of the type of polynomial in each case is a design option. The product of polynomials can be used with arbitrary number of polynomials; it suffices the output of the 7th polynomial to be the input to the (r + th polynomial. For instance, we assume a product of 2 e N polynomials, where each polynomial has an expansion order of two, then the polynomial expansion is of 2 order. In other words, we need ((log,(N)) layers to achieve an Mh order expansion.
In algebra. higher-order polynomials are unbounded and can thus suffer from instability for large values. To avoid such instability, we take the following three steps: a) CoPE samples the noise vector from the uniform distribution, i.e., from the bounded interval of [-1, b) hyperbolic tangent is used the output of the generator as normalization, i.e., it constrains the outputs in the bounded interval of [-1, 1]. c) batch normalization [Ioffe and Szegedy, 2015] is used to convert
5

---

(a) Class-conditional generation
(b) Class-conditional generation
(c) Block-inpainting
Figure 3: Synthesized images by CoPE in the (a), (b) class-conditional generation (sec. 4.1) and (c) block-inpainting (sec. 4.3). In class-conditiona generation, each row depicts single class.
the representations zero-mean. We emphasize that in GANs the hyperbolic tangent is the default activation function in the output of the generator, hence it is not an additional requirement of our method. Additionally, our preliminary experiments, the uniform distribution can be changed for a Gaussian distribution without any instability. A theoretical analysis on the bounds of such multivariate polynomials would be interesting subject for future work.
Lastly, we highlight the flexibility of the proposed CoPE, sBN and SPADE can be considered as special cases of the wwo-variable polynomial expansion. In particular, exhibit in sec. F.1 how SPADE can be extended into a general-purposs two-variable polynomial expansion. In addition, the products of polynomials would enable both sBN and SPADE to perform higher-order expansions without increasing the number of layers.
Experiments
To validate the proposed formulation, the following diverse conditional generation tasks are consid- ered:
class-conditional generation trained CIFARIO, Cars 196 and SVHN in sec. 4.1 and sec. H.2.
generation of unseen attribute combinations in sec. 4.2.
attribute-guided generation in sec. H.5.
inverse problems imaging, e.g., super-resolution and block-inpainting, trained Cars and CelebA in sec. 4.3.
edges-to-image translation trained handbags and shoes in sec. H.4.
image-to-image translation in sec. H.3.
The details on the datasets and the evaluation metrics are deferred to the supplementary (sec. G) along with additional visualizations and experiments. Additionally, the source code of CoPE is available at https: /ttp//gittttt com/ httpg/riiiigigrr... http://grr/ooommmmlalamm.m Lettomiaalleaaa___eeee _etli_nts_______________ for condit ona _eeneration.
Our framework, e.g., (2), does not include any activation functions. To verify the expressivity of our framework, maintain the same setting for the majority of the experiments below. Particularly, the generator does not have activation functions between the layers; there is only hyperbolic tangent in the output space for normalization as typically done in GAN generators. However, we conduct one experiment using a strong baseline with activation functions. That is, comparison with SNGAN [Miyato and Koyama, 2018] in class-conditional generation is performed (sec. 4.1).
Baselines: TT-NNe-SICCOE implements a polynomial expansion of a single variable by concate- nating all the input variables. 'SPADE' implements polynomial expansion with respect to the conditional variable. Also, 'GAN-CONC' and 'GAN-ADD' are added as baselines, where we replace the Hadamard products with concatenation and addition respectively. A schematic of the differences between the compared polynomial methods depicted in Fig. 7, while a detailed description of all methods is deferred to sec. G. Each experiment is conducted five times and the mean and the

---

Table 3: Quantitative evaluation on class-con generation with resnet-based generator (i.e., SNGAN). Higher Inception Score (IS) [Salimans et al., 2016] (lower Frechet Inception Distance (FID) [Heusel al., 2017)) indicates better performance. The baselines improve the IS of SNGAN, however they cannot improve the FID. Nevertheless, SNGAN-CoPE: improves upon all the baselines in both the and the FID.
class-conditional generation on CIFARTO
Model
(1)
FID (J)
SNGAN
8.30 + 0.11
14.70 + 0.97
SNGAN-CONC
8.50 0.49
30.65 3.55
SNGAN-ADD
8.65 0.11
15.47 0.74
SNGAN-SPADE
8.69 + 0.19
21.74 + 0.73
SNGAN-CoPE
8.77 + 0.12
14.22 + 0.66
Table 4: Quantitative evaluation on class conss-oondiional generation with II-Net-based generator. CI- FARIO, there is considerable improvement on the IS, while in 19rs196 FID drops dramatically with CoPE. We hypothesize that the dramatic improvement in Cars 196 arises because of the correlations of the classes. For instance, the SUV cars (of different carmakers) share several patterns, which are captured by our high-order interactions, while they might be missed when learning different normalization statistics per class. The generator does not have activation functions between the layers. so the deteriorated performance of GAN-CONC and GAN-ADD is reasonable.
class-conditional generation on CIFARTO
class-conditional generation Cars
Model
IS
FII ())
Model
FID (()
GAN-CONC
3.73 + 0.32
294.33 1 8.16
GAN-CONC
240.45 t 16.79
GAN-ADD
3.74 0.60
298.53 + 16.54
GAN-ADD
208.72 + 12.65
SPADE
4.00 + 0.53
294.21 + 16.33
SPADE
168.19 1 39.71
II-Net-SICONC
6.65 + 0.60
71.81 + 33.00
II-Net-SCCONC
153.39 + 27.93
IT-Net
7.54 + 0.16
37.26 + 1.86
IT-Net
120.40 + 28.65
CoPE
7.87 + 0.21
34.35 + 2.68
CoPE
55.48 + 3.16
standard deviation are reported. Throughout the experimental section, we reserve the symbol zn for the conditional input (e.g., a class label).
4.1 Class-conditional generation
In class-conditional generation the conditional input is a class label in the form of one-hot vector. The experiments we conduct below modify only the generator, while in all cases we assume there is the same discriminator. In particular, two types of generators (SNGAN), b) a polynomial generator (II-Net). The former network has exhibited strong performance the last few years, while the latter is recently proposed PNN.
used: resnet-basee based generator
Resnet-based generator: The experiment is conducted by augmenting the resnet-based generator of SNGAN Each compared method will be named according to the modification on the generator of SNGAN, e.g., SNGAN-CoPE utilizes a resnet-based generator following the proposed framework of (14). All methods are trained using CIFAR10 images; CIFARIO is a popular benchmark in class-conditional generation. The quantitative results are in Table 3 and synthesized samples are illustrated in Fig. 2(a). ANGAN-CoPE improves upon all the baselines in both the Inception score (IS) [Salimans et al., 2016] 6] and the FID [Heusel et al., 2017]. The proposed formulation enables inter-class interpolations. That is, the noise 2, fixed, while the class žn is interpolated. In Fig. 2(b) and Fig. 2(c), intra-class and inter-class linear interpolations are illustrated respectively. Both the quantitative and the qualitative results exhibit the effectiveness of our framework.
Il-Net-based generator: A polynomial expansion is selected as the baseline architecture for the generator In the original II-Net conditional batch normalization (CBN) was used in the generator; this is replaced by batch normalization in the rest of the compared methods. The quantitative results in CIFARIO are summarized in Table 4 (left). SPADE does not utilize the products of polynomials
Further implementation details offered G.
7

---

FN
FS
MN
MS
(a)
(b)
Figure 4: Synthesized images with CoPE-VAE In (a), all combinations are illustrated (the red the combination missing during training, i.e. Female+Smile), while in (b), only images from the missing combination are visualized.
formulation, which explains its poor performance. Additionally, even though II-Net-SINCONC and I1-Net both express a single-variable polynomial expansion, the inductive bias inserted into the network has substantial effect in the final performance. Notice that CoPE outperforms all the baselines by large margin.
We also evaluate flass-conditional generation in Cars196 that has 196 classes. Cars is selected as reasonably larger dataset than CIFAR 10; it contains 196 classes and yet it can be trained on single GPU. The compared methods and the training details remain the same. The results in Table 4 (right) demonstrate substantial difference between CoPE and the compared methods. Namely, the proposed method achieves a 53.9% reduction of the FID over the best- best-performing baseline. We emphasize that both SPADE and I1-Net were not originally built for class-conditional generation, however we have tried to optimize the respective pyper-parameters to optimize their performance. The performance gap between SPADE and the rest PNNs can be explained by the lack of products of polynomials. We also verify the experimental results on CIFAR10 that demonstrate how II-Net improves upon II-Net- SINCONC. However, even II-Net obtains substantially higher FID than CoPE; we hypothesize that the improvement arises because the correlations between the classes. For instance, the SUV cars of different carmakers share several patterns. Such correlations captured by our framework, while they might be missed when learning different normalization statistics per class. Overall, CoPE synthesizes plausible images (Fig. even in the absence of activation functionsn.
4.2 Polynomial conditioning for generating unseen attribute combinations
The proposed CoPE is general framework for conditional generation, and we have already demon- strated how it can be used for conditional generation in GANs. In this section, we extend our method to the Variational Autoencoder (VAE) [Kingma and Welling, 2014] to showcase a useful byproduct of our formulation, namely the ability to generate unseen label combinations. In particular, intro- duced in [Georgopoulos et al., 2020], this is a multi-label setting where one (or more) combinations not seen in the training set. The method is then evaluated based on its ability to generate the unseen attribute combinations. To this end, we implement CoPE- VAE, a variation of the conditional VAE [Kingma and Welling, 2014] where both the encoder and decoder are polynomials. We perform experiments on CelebA using the annotated attributes of smile and gender. Similar to |Georgopoulos et al., 2020] remove the combination (Smiling, Female) from the training set. The results in Figure highlight the efficacy of the proposed conditioning method in disentangling the two labels and leveraging the multiplicative interactions synthesize the missing combination.
4.3 Inverse problems imaging
In the following paragraphs we evaluate the performance of CoPE inverse problems. We select super-resolution and block-inpaintin as two popular tasks.
8

---

Table 5: Quantitative evaluation super-resolution with II-Net-based generator Cars196. The
task on the left is super-resolution 16x, while on the right the task is super-resolution 8x. Our
variant of SPADE, i.e., SPADE (details in sec. G), vastly improves the original SPADE. The full two-variable model, i.e., CoPE, outperforms the compared methods.
Super-resolution 16x Cars196
Super-resolution 8x
Cars196
Model
FID (.)
Model
FID (J)
SSIM (1)
LPIPS (.)
SPADE
111.75 + 13.41
SPADE
119.18 + 14.82
0.32
0.178
II-Ne--SCCONC
80.16 + 12.42
SINNNC
186.42 + 40.84
0.31
0.200
SPADE-CoPE
72.63 - 3.18
SPADE-COR CoPE
64.76 I 8.26
0.49
0.135
CoPE
60.42 + 6.19
CoPE
62.76 + 4.37
0.53
0.127
The core architectures remain as in the experiment above, i... CoPE and II-Net-IICOC implement products of polynomials. A single change is made in the structure of the discriminator: Motivated by [Miyato and Koyama, 2018]. we include an elementwise product of Σn with the real/fake image in the discriminator. This stabilizes the training and improves the results. Even though architectures specialized for a single task (e.g., Ledig et al. [2017]) perform well in that task, their well-selected inductive biases (e.g., perceptual or & loss) do not generalize well in other domains or different conditional inputs. Our goal is not to demonstrate state-of-the-ae results, but rather to scrutinize the effectiveness of the proposed formulation in different conditional generation tasks. To that end, we consider II-Net-SICONC, SPADE and SPADE-COPE as the baselines.
We experiment with two settings in super-resolution: one that the input image is down-sampled 8x and one that it is down- wwwn-sampled 16x The two settings enable us to test the granularity of CoPE at differentt scales. In super-resolution 16x, Eu (i.e., the low-resolution input) has 48 dimensions, while in super-resolution 8x, Zu has 192 dimensions. The FID scores Cars196 for the task of super-resolution are reported in Table 5. In addition, for the experiment super-resolution 8x, the SSIM [Wang et al., 2004] and LPIPS [Zhang et al., 2018] are reported as widely-used metrics in inverse imaging tasks. Notice that the performance of II-Net-SICONC deteriorates substantially when the dimensionality of the conditional variable increases. That validates our intuition about the concatenation in the input of the generator (sec. E), i.e., that the inductive bias of single-variable PNNs might not fare well in conditional generation tasks. We also report the SPADE-CoPE: which captures higher-order correlations with respect the first variable as well (further details in sec. G). The proposed SPADE-CoPE outperforms the original SPADE, however it cannot outperform the full two-variable model, i.e., CoPE. The results indicate that CoPE performs well even when the conditional input is an image.
Beyond the quantitative results, qualitative results provide different perspective on what the mappings learn. Qualitative results on the super-resolution experiments on Cars 196 are provided in Fig. 8. We also provide synthesized results on both super-resolution 8x and block-inpainting on CelebA in Fig. 8 and Fig. 3 respectively. For each conditional image, different noise vectors z1 are sampled. Notice that the corresponding synthesized images differ in the fine details. For instance, changes in the mouth region, the car type or position and even background changes are observed. Thus, CoPE synthesizes realistic images that i) correspond the conditional input, ii) vary in the fine details. Similar variation has emerged even when the source and the target domains differ substantially, e.g., the translation of MNIST digits to SVHN digits (sec. H.3). We should mention that the aforementioned experiments were conducted only using the adversarial learning loss. In the literature, regularization techniques have been proposed specifically for mmaee-to-maage translation, e.g., Yang et al. [2019]. Lee et al. [2019]. However, such works utilize additional losses and even require additional networks for training, which makes the training computationally demanding and more sensitive to design choices.
Discussion
CoPE can be used for various conditional generation tasks as the experimental evaluation in both sec. 4 and the supplementary illustrate. Namely, CoPE can synthesize diverse content, which is typically tackled using auxiliary losses or networks in conditional GANs. We expect this attribute of diverse generation to be useful in inverse tasks, where multiple latent sharp images can correspond to single corrupted image. The diverse generation can be attributed to the higher-order correlations between

---

the noise and the conditional variable. Such higher-order correlations also enable synthesizing images with unseen attribute combinations (sec. 4.2).
One limitation of our work is that our method has not been tried in large-scale synthesis, e.g., like Brock al. [2019]. However, only very limited number of labs/institutions have access to such resources. In addition, our single-GPU training does have a reduced energy footprint when compared the multi-GPU setups of large scale GANs. We believe that the proposed method has merit despite the single-GPU training. Indeed, we demonstrate that CoPE can perform well in different tasks, which could help reduce the search for independent methods for every single task.
An interesting future step would be to evaluate the performance of the proposed CoPE in training with limited data, e.g., using techniques similar to Karras et al. [2020a]. This task can also encourage architecture discovery, similar to the various architectures devised in Chrysos et al. [2020]. In this work, we have demonstrated two such architectures with recursive formulations in (2) and (14), however additional architectures can be designed by changing the tensor decomposition.
The equations of (2) and (14) express polynomial expansions of arbitrary order, which can approxi- mate the target function without using activation functions between the recursive terms. However, in sec. 4.1 we have also experimented with generators that include activation functions between recursive terms (i.e., the experiment with SNGAN variants). In the future, we intend to study how to include the activation functions in the formulation and study the properties of such piecewise polynomial expansions. Lastly, demystifying the relationship between the order of the polynomial expansion and the expressivity (e.g., the implementation details for the order in sec. G) is a promising direction.
Societal impact of image generation: Manipulation of images is made possible through algorithms and architectures like ours with well-studied potential negative applications. The rapid progress in GAN-based image synthesis has made the discussion imperative. We encourage further work to be conducted on understanding how to detect synthesized images, e.g., in the conditional generation setting. For instance, our method can be used for training powerful classifiers that detect synthesized images.
Conclusion
We have introduced CoPE for conditional data generation. CoPE expresses polynomial expansion of two input variables, i.c., a noise vector and a conditional variable. We exhibit how previously published methods, such as SPADE and sBN, can be considered as special forms of this two-variable polynomial expansion. Notably, CoPE can be augmented to accept an arbitrary number of conditional variables inputs. The empirical evaluation confirms that our framework images in five diverse tasks, including inverse problems and class- conditionl generation. Inverse problems, such as super-resolution, sampling different noise vectors results in plausible differences in the synthesized image. We derive two recursive formulations. i.e., (2) and (14), but new task-specific formulation can be easily defined. We expect this to be useful in learning from different modalities, such as visual question answering (VQA) text-to-speech synthesis, since CoPE capture high-order auto- and cross-correlations among the input variables.
synthesize realistic
benefit from the proposed framework; we showcase that
Acknowledgements
This project was sponsored by the Department of the Navy, Office of Naval Research(ONR)) under a grant number N62909-17-1-2111, We are thankful to the reviewers for their constructive feedback and recommendations on how to improve our manuscript.
References
A. Almahairi, Rajeswar, A. Sordoni, P. Bachman, and A. Courville. Augmented cyclegan: Learning many-to-many mappings from unpaired data. In International Conference on Machine Learning (ICML), 2018.
D. Bahdanau, K. Cho, and Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015.
10

---

A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
T. F. Chan and K. Chen. An optimization-based multilevel algorithm for total variation image denoising. Multiscale Modeling & Simulation, 5(2):615-645 2006.
Chen, M. Lucic, N. Houlsby, and S. Gelly. On self modulation for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
X. Chen, Y. Duan, R. Houthooft, Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems (NeurIPS), pages 2172-2180, 2016.
Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image toniaagg translation. Conference on Computer Vision and Pultern Recognition (CVPR), pages 8789-8797, 2018.
Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. Stargan v2: Diverse image synthesis for multiple domains. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 8188-818 2020.
G. Chrysos, S. Moschoglou, Y. Panagakis, and S. Zafeiriou. Polygan: High-order polynomial generators. arXiv preprint arXiv: 06571, 2019.
G. Chrysos, S. Moschoglou, G. Bouritsas, Y. Panagakis, J. Deng, and S. Zafeiriou. xenets: Deep polynomial neural networks. Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1):53-65, 2018.
H. De Vries, F. Strub, J. Mary, H. Larochelle, O. Pietquin, and A. C. Courville. Modulating early visual processing by language. In Advances in neural information processing systems (NeurIPS), pages 6594-6604, 2017.
O. Debals and L. De Lathauwer. The concept of tensorization. Technical report, Technical Report 17-99, STADIS, KU Leuven, Belgium, 2017.
V. Dumoulin, J. Shlens, and M. Kudlur. learned representation for artistic style. 2017.
M. Georgopoulos, G. Chrysos, M. Pantic, and Y. Panagakis. Multilinear latent conditioning for gencrating unseen attribute combinations. International Conference on Machine Learning (ICML), pages 1442-3451. PMLR, 2020.
Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems (NeurIPS), 2014.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances neural information processing systems (NeurIPS), pages 6626-6637 2017.
Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 98))1735-1178) 1997.
R. Huang, S. Zhang, T. Li, R. He, al. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. In International Conference on Computer Vision (ICCV), 2017.
X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz. Multimodal unsupervised image- toageett-iie to-image translation. In European Conference Computer Vision (ECCV), pages 172-189, 2018.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference Machine Learning (ICML), 2015.

---

P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. mmage-to-image translation with conditional adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR). 2017.
M. Jayakumar, W. M. Czarnekki, Menick, Schwarz, J. Rae, S. Osindero, W. Teh, Harley, and R. Pascanu. Multiplicative interactions and where to find them. In International Conference on Learning Representations (ICLR), 2020.
Y. Jin, Zhang, M. Li, Tian, and H. Zhu. Towards the high-quality anime characters generation with generative adversarial networks. In Proceedings of the Machine Learning for Creativity and Design Workshop at NeurIPS, 2017.
Kaneko, Y. Ushiku, and T. Harada. Label-noise robust generative adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2467-2476, 2019.
T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Karras, M. Aittala, Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. In Advances in neural information processing systems (NeurIPS), 2020a.
Karras, Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 81 110-8119, 2020b.
D. P. Kingma and M. Welling. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR), 2014.
T. G. Kolda and B. W. Bader. Tensor decompositions and applications. SIAM review, 51(3):455-500, 2009.
Krause, Stark, Deng, and L. Fei-Fei. 3d object representations for fine- fine-grained categorization. In Conference on Computer Vision and Pattern Recognition Workshops (CVPR"W) pages 554 561, 2013.
A. Krizhevsky, V. Nair, and G. Hinton. The cifar-10 dataset. online: http://www. cs. toronto. edu/kriz/cffar html, 55, 2014.
Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2322 1998.
C. Ledig, Theis, Huszár, J. Caballero, A. Cunningham, A. Acosta, Aitken, Tejani, Totz, Z. Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
H.-Y. Lee, H.-Y. Tseng, Q. Mao, J.-B. Huang, Y.-D. Lu, M. Singh, and M.-H. Yang. Drit++: Diverse image-to-image translation via disentangled representations. International Journal of Computer Vision (ICC), pages 1-16, 2020.
Lee, J. Ha, and G. Kim. Harmonizing maximum likelihood with gans for multimodal conditional generation. In International Conference on Learning Representations (ICLR), 2019.
A. Levin, Y. Weiss, F. Durand, and W. T. Freeman. Understanding and evaluating blind deconvolution algorithms. In Conference on Computer Vision and Patern Recognition (CVPR), pages 1964-1971, 2009.
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision (ICCV), pages 3730-3738, 2015.
M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet. Are gans created equal? large-scale study. In Advances in neural information processing systems (NeurIPS), pages 700-709, 2018.
L. Ma, X. Jia, Q. Sun, B. Schiele, T. Tuytelaars, and L. Van Gool. Pose guided person image generation. In Advances neural information processing systems (NeurIPS), pages 406-416, 2017.
12

---

Q. Mao, H.-Y. Lee, H.-Y. Tseng, S. Ma, and M.-H. Yang. Mode seeking generative adversarial networks for diverse image synthesis. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1429-1437, 2019.
M. Maximov, I. Elezi, and L. Laix- Ciagan: Conditional identity anonymization generative adversarial networks. Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Miyato and M. Koyama. egans with projection discriminator. International Conference on Learning Representations (ICLR), 2018.
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICI.R, 2018.
Y. Netzer, T. Wang, A. Coates, Bissacco, B. Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
S. Nikol'skii. Analysis III: Spaces of Differentiable Functions. Encyclopaedia of Mathematical Sciences. Springer Berlin Heidelberg. 2013. ISBN 9783662099612.
A. Odena, Olah, and Shlens. Conditional image synthesis with auxiliary classifier gans. In International Conference Machine Learning (ICMI.,, pages 2642-2651, JMLR. org, 2017.
G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39-500 2020.
T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu. Semantic image synthesis with spatially normalization. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2337- 2346, 2019.
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2016.
S. Rendle. Factorization machines. In 2010 IEEE International conference on data mining, pages 995-1000. IEEE, 2010.
Salimans, I. Goodfellow, W. Zaremba, V. Cheung. Radford, and X. Chen. Improved techniques for training gans. In Advances in neural information processing systems (NeurIPS), pages 2234- 2242, 2016.
Y. Shin and Ghosh. The pi-sigma network: An efficient higher-order neural network for pattern classification and function approximation. In International Joint Conference on Neural Networks, volume 1, pages 13-18, 1991.
A. Siarohin, E. Sangineto, S. Lathuiliere, and N. Sebe. Deformable gans for pose-based human image generation. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 3408-3416, 2018
K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, pages 3483-3491, 2015.
R. Sood, B. Topiwala, K. Choutagunta, R. Sood, and M. Rusu. An application of generative adversarial networks for super resolution medical imaging. In International Conference on Muchine Learning and Applications (ICMLA), pages 326-331, 2018.
M. H. Stone. The generalized weierstrass approximation theorem. Mathematics Magazine, 21(5): 237-254, 1948.
C. Szegedy, W. Liu, Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Conference Computer Vision and Pattern Recognition (CVPR), pages 1-9, 2015.
I. Theis, A. v. d. Oord, and M. Bethge. A note on the evaluation of generative models. In International Conference Learning Representations (ICLR), 2016.
13

---

J. M. Tomczak and M. Welling. Vae with a vampprior. arXiv preprint arXiv: arXiv: 2017.
T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, G. Liu, A. Tao, J. Kautz, and B. Catanzaro. Video-to-video synthesis. In Advances in neural information processing systems (NeurIPS), 2018a.
T.-.. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Conference Computer Vision and Pattern Recognition (CVPR), pages 8798-8807, 2018b.
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions in Image Processing (TIP), 13(4):600-612, 2004.
S. Xie and Z. Tu. Holistically-neste edge detection. In International Conference on Computer Vision (ICCV), 2015.
X. Xu, Sun, Pan, Y. Zhang, H. Pfister, and M.-H. Yang. Learning to super-resolve blurry face and text images. In International Conference Computer Vision (ICCV), pages 251-260, 2017.
D. Yang. S. Hong, Y. Jang, T. Zhao, and H. Lee. Diversity sensitive conditional generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
C. You, G. Li, Y. Zhang, X. Zhang, H. Shan, M. Li, S. Ju, Z. Zhao, Z. Zhang, W. Cong, et al. Ct super-resolution gan constrained by the identical, residual, and cycle learning ensemble (gan-circle). IEEE Transactions on Medical Imaging, 99(1):1882000 2019.
A. Yu and K. Grauman. Fine-Grained Visual Comparisons with Local Learning. In Conference on Computer Vision and Puttern Recognition (CVPR), June 2014.
Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. Huang. Generative image inpainting with contextual attention. In Conference Computer Vision and Pattern Recognition (CVPR), pages 5505-5514 2018a.
X. Yu, B. Fernando, R. Hartley, and F. Porikli. Super-resolving very low-resolution face images with supplementary attributes. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 908-917, 2018b.
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 586-595, 2018.
S. Zhao, H. Ren, A. Yuan, J. Song, N. Goodman, and S. Ermon. Bias and generalization in deep generative models: An empirical study. In Advances neural information processing systems (NeurIPS), pages 10792-10801,200
J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros. Generative visual manipulation the natural image manifold. In European Conference on Computer Vision (ECCV), 2016.
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In International Conference on Computer Vision (ICCV), pages 2223-2232, 2017a.
J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman. Toward multimodal image-to-image translation. Advances neural information processing systems (NeurIPS), pages 465-476 2017b.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] The datasets we use in this work are all publicly available, our setup can be reproduced by interested practitioners. The code is open sourced.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We use standard data splits from well-established datasets (i.e. using the wrappers of Pytorch). The source code is publicly available.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] A single GPU was used in each experiment. The GPU is part of our lab cluster.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] The databases used in our experimental validation have been properly cited. In addition, the code is written by the authors and the models trained by the authors.
(b) Did you mention the license of the assets? [Yes] All the datasets used in this work are publicly available datasets; in addition, they quite popular benchmarks for diverse tasks. All the datasets enable their use for research purposes.
(c) Did you include any new assets either in the supplemental material or as URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] The datasets we utilize are publicly available and contain tens or hudrends of thousands of images. We refer to the authors original papers describing the datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] All the datasets do not have offensive content or personally identifiable information. Celeb-A contains images of celebrities and the images do not contain any offensive content.
(except for Celeb-A)
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "V1_B56uvNKv", "reviewer": "Reviewer_Rugm", "summary": "This paper proposes a model for conditional generation of images, which relies on the creation of polynomials of the inputs. The paper evaluates this generation process in a multitude of tasks, showing improvement over similar techniques.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The generation method (CoPE) proposed in this paper is closely related to $\\Pi$-net, directly designed for conditional generation (accepting two inputs at its base form, one latent and one conditional input). The authors point out several aspects which differentiate their technique from $\\Pi$-net, and I think there is significant novelty. I would suggest furthering this discussion in the main paper, since the two techniques are quite similar.\n\nThe experimental section (which is the main focus of the paper) considers a multitude of problems, such as class conditional generation, inverse problems, and generation of unseen attribute combinations. This demonstrates the suitability of the proposed method in various useful tasks. Comparisons are made with several polynomial-like architectures, but I think it would be interesting to make comparisons with other architectures as well. For example, in the conditional CIFAR-10 experiment, Karras et al. (2020) report some architectures which appear to have better IS/FID scores than the proposed CoPE technique (Figure 11b in that paper). Of course, the actual scores received also depend on the size of the architecture used in each method, but I believe they should be mentioned at least. I also think that the baselines considered should be the same in Tables 4 and 5 (Sections 4.1 and 4.3 respectively), to enable a better comparison of the proposed method with previous work.\n\nThe paper is clear in general and easy to follow. I have the following comments to make which can improve the clarity of the paper.\n- Tables should be in the beginning or the end of the page. Right now there are a few lines of text between tables which are easy to miss.\n- I believe it would be useful to include a better explanation of the baselines used (mainly GAN-CONC and GAN-ADD), as well as the architecture of the discriminator and the order of CoPE used in the experiments (at least in the supplementary).\n- In Figure 4, I think it would be useful to include whether the images presented are randomly picked or curated.\n\nThere are also a couple of very minor errors, namely Tables 1 and 4 extending beyond the limits of the page slightly, as well as a typo in line 78.\n\nOverall, this is an interesting work in an equally interesting field, given the possibility of providing improvements for both image generation and inverse problems. I lean towards accepting this work, especially if the issues regarding the experiments I addressed above are resolved during the rebuttal process.\n\nReferences:\n\nKarras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020). Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06676.\n\n**Update after rebuttal:**\n\nThe reviewers have clarified the points I noted about the experiments, and have fully addressed my concerns. I believe the added material from the feedback and all of the reviewers' responses should be included in the final version of the paper.\n\nAs such, I am raising my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately described the limitations of their work in a dedicated section of the paper (Section 5). \n\nIn the checklist they also state that they only present methods for a task which exists in the literature, so no discussion on negative societal impact is needed. I agree with them, but I also believe that a short statement regarding the societal impact of the task itself (e.g. manipulation of images) could be included for completeness.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "PfMokfTnAen", "reviewer": "Reviewer_7vAs", "summary": "This paper presents a method of conditional image generation where the conditional variables can be discrete or continuous (including other images) or a combination of both. The main idea is based on recently proposed polynomial neural networks which were unconditional and based on continuous latent noise variables as input. This paper extends it to include multiple inputs aiming to model correlations between them. Experiments conducted on 5 different tasks show improvements over considered baselines. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea is simple and intuitive with sound motivation. And the experiments are very thorough and show clear improvements over baselines.\n\nOther comments: \n\nI am not an expert in this field but the writing is mostly clear but is hard to follow at the beginning of the experiment section. Section 3 largely focuses on simple PNNs and their generalization to conditional settings. However, right at the beginning of section 4, the authors mention SNGAN-CoPE and SPACE-CoPE without much explanation of how they are created. \n\nAdditionally, in section 3.2 about the logarithmic number of layers required, it was not clear how the output of the first polynomial is fed to the other. The initial input has two latent variables z_I and z_II which will result in a single output which is a polynomial of degree N_1. How exactly is this input to obtain a polynomial of degree N1.N2? \n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "OeGHmBsPLiZ", "reviewer": "Reviewer_kHDT", "summary": "Deep polynomial neural networks (PNNs) have demonstrated great results on unsupervised image generation, but has not been applied to conditional generation tasks. This paper introduces a framework which allows PNNs for conditional data generation.\nThe paper proposes two extensions to the core model: a) a formulation that enables an arbitrary number of conditional input variables, b) a new architecture. Experiments show that the proposed conditional model can be applied to five different tasks (class-conditional generation, inverse problems, edges-to-image translation, image-to-image translation, attribute-guided generation).\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. This paper introduces a framework that extends deep polynomial neural network to conditional data generation, which does not seem to be extensively studied before. However, extending an unconditional model to a conditional model seems to be a straightforward follow-up.\n2. The paper is well written.\n3. The proposed method can be applied to different conditional generation tasks. However, it is unclear how competitive the results are compared to state-of-the-art models for each application and other state-of-the-art conditional generative models that do not use polynomial neural network architectures. It seems that no such comparison is performed. In order to show the usefulness of the method, such comparisons are needed.\n\n**Detailed comments**:\n1. Class-conditional generation: As mentioned at line 234-236 \"both SPADE and $\\Pi$-Net were not originally built for class-conditional generation\", comparing with SPADE or $\\Pi$-Net might not successfully show the strength of the proposed method. The FID scores for GAN-CONC, GAN-ADD, SPADE in Table 4 are very bad (e.g., 290+ for CIFAR-10), which raises questions on whether the baselines are tuned and optimized properly. It would be more convincing to compare with state-of-the-art class-conditional generation methods.\n2. Section 4.2: It would be useful to mention that comparison with other models are in the appendix. Is there any reason why VAE is used while the other sections are discussing GANs? If the goal is to show that the proposed method extends beyond GANs, it would be nice to also include the VAE results in the other sections, and discuss the related background and details (instead of directly presenting the VAE results in only one section). \n3. Super-resolution: SPADE is not originally designed for super-resolution, it might be important to compare with other stronger super-resolution baselines in order to show the strength of the proposed method. It is unclear if FID score alone is able to capture how good the super-resolution results are. For instance, the generated high resolution images could completely ignore the low resolution images but still achieve a good FID score (e.g., unconditional generation). It would be more convincing to also report other commonly used metrics (e.g., PSNR, SSIM, etc) for super-resolution.\n4. minor: It would be good to label which methods are the proposed methods in the table.\n5. Will the proposed method work for denoising?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impacts are discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "1J_Kx-XBNQ8", "reviewer": "Reviewer_tbWn", "summary": "The paper introduces conditional polynomial expansion method (CoPE) that expands on the polynomial network framework and adopts it for conditional generation with GANs. CoPE can be seen as a generalization of existing work that allows better use of high-order correlations between input variables and polynomial multiplication for parameter efficiency. Conditional generation experiments suggest that CoPE is indeed a useful mechanism for mixing information from different sources. Revealing more details on the technical implementation would make even stronger submission.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty:\n\nCoPE is a novel construction to my knowledge which can be seen as a natural continuation of the existing work such as Pi-Net or SPADE. A potentially useful reference for a similar (albeit arguably simpler) technique for incorporating interactions between multiple input variables can be found in [1].\n\nClarity:\n\nThe paper is generally well-written and adequately commincates the non-trivial construction of CoPE. \n\nI would still appreciate if authors could provide more details on technical implementation on some of the mentioned ideas.\n\n1) As authors mention, if one of the inputs is an image a convolution can be applied for embedding it. For example, is it a problem that the dimensionality of the image should remain constant (and low) or this can be circumvented? \n2) Would it be anyhow beneficial to include non-linear components in computing the multiplicative update in eq. 2? As I understand, it would break the connection to the general polynomial expansion method but also make the architecture closer to StyleGAN which could (along with the standard StyleGAN) serve as another baseline.\n3) Implementation of  SNGAN-CoPE.\n\nAnother question is why in Appendix D matrix P has dimensionality d1 * d2 x o and not (d1 + d2) x o? I'm puzzled here.\n\nA more general question is why authors are specifically interested in the GAN setting and not, for example, in VAEs where the problem of ignoring conditional information is also present. The benefit of considering VAEs would allow to compare models directly in likelihoods and alleviate the possibility of hacking the imperfect image-quality metrics.\n\nSignificance:\n\nCoPE seems to be a useful mechanism for conditioning in polynomial networks and will likely be re-used in further work.\n\nQuality:\n\nOverall I don't have many concerns about the method of the evaluation. I would still very much be interested in applying CoPE in a controllable, toy setting where the underlying data distribution is known to verify it indeed recovers the right interaction between variables because mere relying on metrics such as inception score is somewhat futile (this just my subjective opinion).\n\nReferences:\n\n[1] Rendle, S. (2010, December). Factorization machines. In 2010 IEEE International conference on data mining (pp. 995-1000). IEEE.\n\n**Updated after authors feedback**\n\nAuthors addressed my questions and concerns reasonably and I'm therefore raising my score. I encourage authors to incorporate the received feedback (and their replies) into the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No concerns.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed an extension of polynomial neural networks (PNNs) for application to conditional generation tasks. Experiments show that the proposed conditional model can be applied to five different tasks (class-conditional generation, inverse problems, edges-to-image translation, image-to-image translation, attribute-guided generation) with relevance to the NeurIPS conference.

The reviewers all agree that the contribution is useful and novel. All reviewers were satisfied with the author response to the minor concerns that were raised in the first round of reviewing, with one reviewer raising their score. The reviewers engaged in committee discussion and reached a consensus recommendation to accept the paper. 
Decision: Accept (Poster)