Conditional Generation Using Polynomial Expansions
Authors: Grigorios Chrysos
Markos Georgopoulos
Yannis Panagakis
Venue: NeurIPS 2021 Poster (2021)
Keywords: polynomial expansions
polynomial neural networks
GAN
VAE
unseen combinations
conditional data generation

Abstract: Generative modeling has evolved to a notable field of machine learning. Deep polynomial neural networks (PNNs) have demonstrated impressive results in unsupervised image generation, where the task is to map an input vector (i.e., noise) to a synthesized image. However, the success of PNNs has not been replicated in conditional generation tasks, such as super-resolution. Existing PNNs focus on single-variable polynomial expansions which do not fare well to two-variable inputs, i.e., the noise variable and the conditional variable. In this work, we introduce a general framework, called CoPE, that enables a polynomial expansion of two input variables and captures their auto- and cross-correlations. We exhibit how CoPE can be trivially augmented to accept an arbitrary number of input variables. CoPE is evaluated in five tasks (class-conditional generation, inverse problems, edges-to-image translation, image-to-image translation, attribute-guided generation) involving eight datasets. The thorough evaluation suggests that CoPE can be useful for tackling diverse conditional generation tasks. The source code of CoPE is available at https://github.com/grigorisg9gr/polynomial_nets_for_conditional_generation.

Conditional Generation Using Polynomial Expansions
Grigorios G Chrysos
Markos Georgopoulos Imperial College London, UK
EPFL, Switzerland
grigorios. ctt.::ssss.crsssspp......oo.
wr.gerggoooollsst ieme@rmpppriee wwp.rial.ac.c...m..........
Yannis Panagakis
University of Athens, GR yannispēdi, uua.
Abstract
Generative modeling has evolved to notable field of machine learning. Deep polynomial neural networks (PNNs) have demonstrated impressive results in unsu- pervised image generation, where the task is to map an input vector (i.e., noise) synthesized image. However, the success of PNNs has not been replicated in conditional generation tasks, such as super-resolution. Existing PNNs focus on single-variable polynomial expansions which do not fare well to two-variable inputs, i.e., the noise variable and the conditional variable. In this work, we introduce a general framework, called CoPE, that enables a polynomial expan- sion of two input variables and captures their auto- and cross-correlations, We exhibit how CoPE can trivially augmented to accept an arbitrary number input variables. CoPE is evaluated in five tasks conds-conniiionall generation, inverse problems, edges-to-imane translation, image-to-image translation, attribute- guided generation) involving eight datasets. The thorough evaluation suggests that CoPE can be useful for tackling diverse conditional generation tasks. The source code of CoPE is available polynomia sylyoommallettteeeel nets _frrr__o______ thteessor_oooeeteteeeeeeeeeeeeee coor_condiiionneeoeeoeeoeeeeeoee cor.iiiiiollleeeeettttonn. generation.
https://git
com/ siggorisgggs
Introduction
Modelling nigh-dimensional distributions and generating samples from complex distributions are fundamental tasks in machine learning. Among prominent generative models, StyleGAN [Karras et al., 2019] has demonstrated unparalleled performance in unsupervised image generation. Its success can be attributed to the higher-order correlations of the input vector captured by the generator. As Chrysos et al. [2019] argue, StyleGAN' is best explained as a deep polynomial neural network (PNN). PNNs have demonstrated impressive generation results in faces, animals, cars [Karras al., 2020b], paintings, medical images [Karras et al., 2020a]. Nevertheless, PNNs have to demonstrate similar performance conditional generation tasks, such as super-resolution or image-to-imen
translation.
In contrast to unsupervised generators that require a single-variable input 2, in conditional generation (at least) two inputs required: i) (or more) conditional variables c, e.g., low-resolution image, and ii) a noise sample 2. A trivial extension of PNNs for conditional generation would be to concatenate all the input variables into fused variable. The fused variable is then the input the single-variable polynomial expansion of PNNs. However, the concatenation reduces the flexibility of the model significantly. For instance, concatenating noise vector and vectorized low-resolution image results in sub-optin cub-optimal super-resolution, since the spatial correlations of the input image are lost in the vectorization. Additionally, the concatenation of the vectorized conditional variable and
This work focuses on the generator network; any reference StyleGAN refers to its generator.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

leads to huge number of parameters when we use fully-connected layer as typically done in the input of StyleGAN, especially when e depicts an image.
In this work, we introduce a framework, called CoPE, for conditional data generation. CoPE resorts to multivariate polynomials that capture the higher-order auto- and cross-correlations between the two input variables. By imposing tailored structure in the higher-order correlations, we obtain an intuitive, recursive formulation for CoPE. The formulation enables different constraints to be applied to each variable and its associated parameters. In CoPE, different architectures can be defined simply by changing the recursive formulation. Our contributions can be summarized as follows:
We introduce a framework, called CoPE, that expresses high-order, multivariate polynomial for conditional data generation. We exhibit how CoPE can be applied on diverse conditional generation tasks.
We derive two extensions to the core two-variable model: a) we augment the formulation to enable an arbitrary number of conditional input variables, b) we design different architectures that arise by changing the recursive formulation.
CoPE is evaluated on five different lasks (class-conditional generation, inverse problems, edges-to-image translation, amage-to-image translation, attribute-guided generation); overall eight datasets are used for the thorough evaluation.
The diverse experiments suggest that CoPE can be useful for a variety of conditional generation tasks, e.g., by defining tpsk-specific recursive formulations. To facilitate the reproducibility, the source code is available at https: / //gi higtthu....m com/ httpl/cccoo/gggrroeeeeeeee http/rgiiigiggggeeeeeeeg https/gggroooooooooo a nets_
2 Related work
Below, we review representative works in conditional generation and then we summarize the recent progress in multiplicative interactions (as low-order polynomial expansion).
2.1 Conditional generative models
The literature on conditional generation is vast. The majority of the references below focus on Generative Adversarial Networks (GANs) [Goodfellow et al., 2014] since GANs have demonstrated the most impressive results to date, however similar methods can be developed for other generative models, such as Variational Auto-encoders (VAEs) [Kingma and Welling, 2014]. Four groups of conditional generative models are identified below based on the type of conditional information.
The first group is the class-conditionul generation [Miyato et al., 2018, Brock et al., 2019, Kaneko et al., 2019], where the data are divided into discrete categories, e.g., a cat or a ship. During training class label is provided and the generator should synthesize sample from that category. One popular method of including class-conditional information is through conditional normalization techniques [Dumoulin et al., 2017, De Vries al., 2017]. An alternative way is to directly concatenate the class labels with the input noise; however, as Odena et al. [2017] observe, this model does not scale well to a hundred thousand classes.
The second group is inverse problems that have immense interest for both academic and commercial reasons [Sood et al., 2018, You et al., 2019]. The idea is to reconstruct latent signal, when corrupted measurements are provided [Ongie et al., 2020]. Well-known inverse problems in imaging include super-resolution, deblurring, inpainting. Before the resurgence of deep neural networks, the problems were tackled with optimization based techniques [Chan and Chen, 2006, Levin et al., 2009]. The recent progress in conditional generation has fostered the interest in inverse problems [Ledig et al., 2017, Pathak et al., 2016, Huang et al., 2017]. A significant challenge that is often overlooked the literature [Ledig al., 2017, Yu et al., 2018a] is that there are many possible latent signals for each measurement. For instance, given a now-resolution image, we can think of several high-resolution images that when samplsa provide the same low-resolution image.
Another group is touge- translation. The idea is to map an image from one domain to an image to another domain, e.g., day-time image to a night-time image. The influential work of Isola et al. [2017] has become the reference point for mage-to-imaae translation. Applications condi- tional pose generation [Ma et al., 2017, Siarohin al., 2018], conditional video generation [Wang

---

Table 1: Comparison of polynomial neural networks (PNNSs) Even though the architectures' of Karras et al. [2019], Chen et al. [2019], Park et al. [2019] were not posed as polynomial expansions, we believe that their success can be (partly) attributed the polynomial expansion (please check sec. F for further information). II-Net and StyleGAN are not designed for conditional data generation. In practice, learning complex distributions requires high-order polynomial expansions; this can be effectively achieved with products of polynomials as detailed in sec. 3.2. Only IT-Net and CoPE include such a formulation. The columns on discrete and continuous variable refer to the type of conditional variable the method was originally proposed c.g., sBN was only tried on class- conditional generation. Additionally, the only work that enables multiple conditional variables (and includes related experiments) is the proposed CoPE.
Attributes of polynomial like networks.
products of polynomials
discrete
continuous cond. variable
multiple cond. variables
Model
II-Net [Chrysos ct al., 2020]
X
X
StyleGAN [Karras al., 2019]
X
x
X
sBN [Chen et al., 2019]
x
x
X
SPADE [Park et al., 2019]
X
CoPE (ours)
et al., 201 8a] or generation from semantic labels [Wang et al., 2018b] have appeared. Despite the success, converting the mapping from one-to-one to one-to-many, i.e., having multiple plausible outputs for a single input image, has required some work [Zhu et al., 2017b, Huang et al., 2018]. A more dedicated discussion on diverse generation is deferred to sec. I.
The fourth group uses multiple conditional variables for generation, e.g., sttributeggiiee gener- ation [Choi et al., 2018]. These methods include significant engineering (c.g.. multiple discrimi- nators [Xu et al., 2017]. auxiliary losses). The influential InfoGAN [Chen et al., 2016] explicitly mentions that without additional losses the generator is 'free to ignore' the additional variables.
Each technique above is typically applied to single group of conditional generation tasks, while our goal is to demonstrate that CoPE can be applied to different tasks from these groups.
2.2 Multiplicative interactions
Multiplicative connections have long been adopted in machine learning (Shin and Ghosh, 1991, Hochreiter and Schmidhuber, 1997, Bahdanau et al., 2015, Rendle, 2010]. The idea is to combine the inputs through elementwise products or other diagonal forms. Jayakumar et al. [2020] prove that second order multiplicative operators can represent greater class of functions than classic feed-forward networks. Even though we capitalize on the theoretical argument, our framework can express any higher-order correlations while the framework of Jayakumar al. [2020] is limited to second order interactions.
Higher-order correlations have been studied in the tensor-related literature [Kolda and Bader, 2009, Debals and De Lathauwer, 2017]. However, their adaptation in modern deep architectures has been slower. IT-Net [Chrysos et al., 2020] resorts to a high-order polynomial expansion for mapping the input z to the output æ = G(z). I--eet focuses on a single-variable polynomial expansion; an in-depth difference of our work with I1-Net can be found in sec. E in the supplementary. Two efforts on conditional generation which can be cast as polynomial expansions are SPADE [Park et al., 2019] and sBN [Chen al., 2019]. SPADE can be interpreted as a single-variable polynomial expansion with respect to the conditional variable c. SPADE does not capture the higher- cross-core coss--orrelations between the input variables. Similarly, sBN can be interpreted as a polynomial expansion of the two variables for class-con generation. However, SPADE and sBN do not use the product of polynomial formulation, which enables high-order expansions without increasing the number of layers [Chrysos et al., 2020]. Importantly, SPADE and sBN are constructed for specific applications (i.e., semantic image generation and mnsupervieeddclas-coniiii remains unclear whether PNN can effectively tackle purppoe conditional generation task.
generation respectively) and
3

---

3 Method
In the following paragraphs we introduce the two-variable polynomial expansion (sec. 3.1), while the detailed derivation, along with additional models deferred the supplementary (sec. B). The crucial technical details, including the stability of the polynomial, are developed in sec. 3.2. We emphasize that multivariate polynomial can approximate any function [Stone, 1948, Nikol'skii, 2013), i.e., a multivariate polynomial is a universal approximator.
hattiion.eennnos/mattrees//eeeee by calli-
Table 2: Symbols
are graphiclluperrasee/oooerasss
symbolized
boldface letters e.g.. W. W.w. The mode-m vector product of W (of order M) with vector u R/m is W X u and results in a tensor of order M 1. We assume that Ii T4 1 when >> b. The core symbols are summarized in Table while detailed tensor notation is deferred to the supplementary (sec. B.1).
Symbol
Role
N
Expansion order of the polynomial
k
Rank of the decompositions
21, 21
Inputs to the polynomial
n.P
Auxiliary variables
Winel
Parameter tensor of the polynomial
U [n]] C
Learnable parameters
Hadamard product
3.1 Two input variables
Given two input variables 21, 2n E IKd where S R N, the goal is to learn function G Kaxd - I that captures the higher-order correlations between the elements of the two inputs. can learn such higher-order correlations as polynomials of two input variables. polynomial expansion of order N E N with output E I (such that æ G(2,, 2nr) has the form:
1++
Wlmel
π j-2
x
II r-p+1
x 3
where B E R' and Wlnmel E Rox11=-- Xnd for E [1,N], [1, + 1] are the learnable (independent) variables, hence the and p as
parameters. The expansion depends auxiliary variables. The two products of (1) do not overlap, i.e., the first multiplies the modes [2, (of wllmell with and the other multiplies the modes Le + 1, + with zn.
Z
Upi
(2
U
[MMW....
U
U2n11
U
or
Z11
S.. B
G G 211)
Figure 1: Abstract schematic for Th order approximation of x G(z,, Z1r). The inputs 21, Z1r are symmetric in our formulation. We denote with æ a noise vector, e.g., a samples from Gaussian distribution, while æ symbolizes sample from conditional input (e.g., class label low- resolution image).
Recursive relationship: The aforementioned derivation can be generalized to an arbitrary expansion order. The recursive formula for arbitrary order N N is the following:
x In-1 + (Uir.112 + U U7..11721 * #n-1
"To avoid cluttering the notation we use same dimensionality for the two inputs. However, the derivations apply for different dimensionalities, only the dimensionality of the tensors change slightly.

---

for n = =2... 2...... N with œ1 = UR. f..na + U (1.1 2n and x = CaN + B. The parameters C' Roxk; U((n,o] E Rddk for n I,....N N and Φ = {I, are learnable.
The intuition behind this model is the following: An embedding is initially found for each of the two input variables, then the two embeddings are added together and they are multiplied elementwise with the previous approximation. The different embeddings for each of the input variables allows us implement Un.J] and Un.IrJ with different constraints, e.g., U U[n,1] to be dense layer and (1, to be convolution.
(a) Random samples per class
(b) Intra-class interpolation
(c) Inter-class interpolation
Figure 2: Synthesized images by CoPE in the class consitonndi CIFARIO (with resnet-based genera- tor): (a) Random samples where each row depicts the same class, (b) Intra-class linear interpolation from source the target, (c) inter-class linear interpolation. In inter-class interpolation, the class labels of the leftmost and rightmost images are one-hot vectors, while the rest are interpolated in-between; the resulting images visualized. In all three cases, CoPE synthesizes realistic images.
3.2 Model extensions and technical details
There are three limitations (2). Those are the following: a) (2) describes a polynomial expansion of two-variable input, b) each expansion order requires additional layers, c) high-order polynomials might suffer from unbounded values. Those limitations are addressed below.
Our model can be readily extended beyond two varviariable input; an extension with varialla input developed sec. C. The pattern (for each order) is similar to the two-variable input: a) a different embedding is found for each input variable, b) the embeddings are added together, c) the result is multiplied elementwise with the representation of the previous order.
The polynomial expansion of (2) requires O(N) layers for an 1ith order expansion. That is, each new order n of expansion requires new parameters Ui,.1] and UI], However, the order of expansion can be increased without increasing the parameters substantially. To that end, we can capitalize on the product of polynomials. Specifically, let N₂ be the order of expansion of the first polynomial. The output of the first polynomial is fed into a second polynomial, which has expansion order of N2. Then, the output of the second polynomial will have an expansion order of Nı N2. The second polynomial of degree N2 can either be polynomial of one variable (i.e., the output of the previous polynomial) or more variables (i.e., the output the previous polynomial and one or more of the inputs). In both cases the total degree of the output of the second polynomial will be N₁ N₂. The choice of the type of polynomial in each case is a design option. The product of polynomials can be used with arbitrary number of polynomials; it suffices the output of the 7th polynomial to be the input to the (r + th polynomial. For instance, we assume a product of 2 e N polynomials, where each polynomial has an expansion order of two, then the polynomial expansion is of 2 order. In other words, we need ((log,(N)) layers to achieve an Mh order expansion.
In algebra. higher-order polynomials are unbounded and can thus suffer from instability for large values. To avoid such instability, we take the following three steps: a) CoPE samples the noise vector from the uniform distribution, i.e., from the bounded interval of [-1, b) hyperbolic tangent is used the output of the generator as normalization, i.e., it constrains the outputs in the bounded interval of [-1, 1]. <) batch normalization [Ioffe and Szegedy, 2015] is used to convert
5

---

(a) Class-conditional generation
(b) Class-conditional generation
(c) Block-inpainting
Figure 3: Synthesized images by CoPE in the (a), (b) class-conditional generation (sec. 4.1) and (c) block-inpainting (sec. 4.3). In class-conditiona generation, each row depicts single class.
the representations zero-mean. We emphasize that in GANs the hyperbolic tangent is the default activation function in the output of the generator, hence it is not an additional requirement of our method. Additionally, our preliminary experiments, the uniform distribution can be changed for a Gaussian distribution without any instability. A theoretical analysis on the bounds of such multivariate polynomials would be interesting subject for future work.
Lastly, we highlight the flexibility of the proposed CoPE, sBN and SPADE can be considered as special cases of the wwo-variable polynomial expansion. In particular, exhibit in sec. F.1 how SPADE can be extended into a general-purposs two-variable polynomial expansion. In addition, the products of polynomials would enable both sBN and SPADE to perform higher-order expansions without increasing the number of layers.
Experiments
To validate the proposed formulation, the following diverse conditional generation tasks are consid- ered:
class-conditional generation trained CIFARIO, Cars 196 and SVHN in sec. 4.1 and sec. H.2.
generation of unseen attribute combinations in sec. 4.2.
attribute-guided generation in sec. H.5.
inverse problems imaging, e.g., super-resolution and block-inpainting, trained Cars and CelebA in sec. 4.3.
edges-to-image translation trained handbags and shoes in sec. H.4.
image-to-image translation in sec. H.3.
The details on the datasets and the evaluation metrics are deferred to the supplementary (sec. G) along with additional visualizations and experiments. Additionally, the source code of CoPE is available at https: /ttp//gittttt com/ httpg/riiiigigrr... http://grr/ooommmmlalamm.m Lettomiaalleaaa___eeee _eili_nts___________ for condit ona _eeneration.
Our framework, e.g., (2), does not include any activation functions. To verify the expressivity of our framework, maintain the same setting for the majority of the experiments below. Particularly, the generator does not have activation functions between the layers; there is only hyperbolic tangent in the output space for normalization as typically done in GAN generators. However, we conduct one experiment using a strong baseline with activation functions. That is, comparison with SNGAN [Miyato and Koyama, 2018] in class-conditional generation is performed (sec. 4.1).
Baselines: TT-NNe-SICCOE implements a polynomial expansion of a single variable by concate- nating all the input variables. 'SPADE' implements polynomial expansion with respect to the conditional variable. Also, 'GAN-CONC' and 'GAN-ADD' are added as baselines, where we replace the Hadamard products with concatenation and addition respectively. A schematic of the differences between the compared polynomial methods depicted in Fig. 7, while a detailed description of all methods is deferred to sec. G. Each experiment is conducted five times and the mean and the

---

Table 3: Quantitative evaluation on class-con generation with resnet-based generator (i.e., SNGAN). Higher Inception Score (IS) [Salimans et al., 2016] (lower Frechet Inception Distance (FID) [Heusel al., 2017)) indicates better performance. The baselines improve the IS of SNGAN, however they cannot improve the FID. Nevertheless, SNGAN-CoPE: improves upon all the baselines in both the and the FID.
class-conditional generation on CIFARTO
Model
(1)
FID (J)
SNGAN
8.30 + 0.11
14.70 + 0.97
SNGAN-CONC
8.50 0.49
30.65 3.55
SNGAN-ADD
8.65 0.11
15.47 0.74
SNGAN-SPADE
8.69 + 0.19
21.74 + 0.73
SNGAN-CoPE
8.77 + 0.12
14.22 + 0.66
Table 4: Quantitative evaluation on class conss-oondiional generation with II-Net-based generator. CI- FARIO, there is considerable improvement on the IS, while in 19rs196 FID drops dramatically with CoPE. We hypothesize that the dramatic improvement in Cars 196 arises because of the correlations of the classes. For instance, the SUV cars (of different carmakers) share several patterns, which are captured by our high-order interactions, while they might be missed when learning different normalization statistics per class. The generator does not have activation functions between the layers. so the deteriorated performance of GAN-CONC and GAN-ADD is reasonable.
class-conditional generation on CIFARTO
class-conditional generation Cars
Model
IS
FII ())
Model
FID (()
GAN-CONC
3.73 + 0.32
294.33 1 8.16
GAN-CONC
240.45 t 16.79
GAN-ADD
3.74 0.60
298.53 + 16.54
GAN-ADD
208.72 + 12.65
SPADE
4.00 + 0.53
294.21 + 16.33
SPADE
168.19 1 39.71
II-Net-SICONC
6.65 + 0.60
71.81 + 33.00
II-Net-SCCONC
153.39 + 27.93
IT-Net
7.54 + 0.16
37.26 + 1.86
IT-Net
120.40 + 28.65
CoPE
7.87 + 0.21
34.35 + 2.68
CoPE
55.48 + 3.16
standard deviation are reported. Throughout the experimental section, we reserve the symbol zn for the conditional input (e.g., a class label).
4.1 Class-conditional generation
In class-conditional generation the conditional input is a class label in the form of one-hot vector. The experiments we conduct below modify only the generator, while in all cases we assume there is the same discriminator. In particular, two types of generators (SNGAN), b) a polynomial generator (II-Net). The former network has exhibited strong performance the last few years, while the latter recently proposed PNN.
used: resnet-basee based generator
Resnet-based generator: The experiment is conducted by augmenting the resnet-based generator of SNGAN Each compared method will be named according to the modification on the generator of SNGAN, e.g., SNGAN-CoPE utilizes a resnet-based generator following the proposed framework of (14). All methods are trained using CIFAR10 images; CIFARIO is a popular benchmark in class-conditional generation. The quantitative results are in Table 3 and synthesized samples are illustrated in Fig. 2(a). ANGAN-CoPE improves upon all the baselines in both the Inception score (IS) [Salimans et al., 2016] 6] and the FID [Heusel et al., 2017]. The proposed formulation enables inter-class interpolations. That is, the noise 2, fixed, while the class žn is interpolated. In Fig. 2(b) and Fig. 2(c), intra-class and inter-class linear interpolations are illustrated respectively. Both the quantitative and the qualitative results exhibit the effectiveness of our framework.
Il-Net-based generator: A polynomial expansion is selected as the baseline architecture for the generator In the original II-Net conditional batch normalization (CBN) was used in the generator; this is replaced by batch normalization in the rest of the compared methods. The quantitative results in CIFARIO are summarized in Table 4 (left). SPADE does not utilize the products of polynomials
Further implementation details offered G.
7

---

FN
FS
MN
MS
(a)
(b)
Figure 4: Synthesized images with CoPE-VAE In (a), all combinations are illustrated (the red the combination missing during training, i.e. Female+Smile), while in (b), only images from the missing combination are visualized.
formulation, which explains its poor performance. Additionally, even though II-Net-SINCONC and I1-Net both express a single-variable polynomial expansion, the inductive bias inserted into the network has substantial effect in the final performance. Notice that CoPE outperforms all the baselines by large margin.
We also evaluate flass-conditiona generation in Cars196 that has 196 classes. Cars is selected as reasonably larger dataset than CIFAR 10; it contains 196 classes and yet it can be trained on single GPU. The compared methods and the training details remain the same. The results in Table 4 (right) demonstrate substantial difference between CoPE and the compared methods. Namely, the proposed method achieves a 53.9% reduction of the FID over the best- best-performing baseline. We emphasize that both SPADE and I1-Net were not originally built for class-conditional generation, however we have tried to optimize the respective pyper-parameters to optimize their performance. The performance gap between SPADE and the rest PNNs can be explained by the lack of products of polynomials. We also verify the experimental results on CIFAR10 that demonstrate how II-Net improves upon II-Net- SINCONC. However, even II-Net obtains substantially higher FID than CoPE; we hypothesize that the improvement arises because the correlations between the classes. For instance, the SUV cars of different carmakers share several patterns. Such correlations captured by our framework, while they might be missed when learning different normalization statistics per class. Overall, CoPE synthesizes plausible images (Fig. even in the absence of activation functionsn.
4.2 Polynomial conditioning for generating unseen attribute combinations
The proposed CoPE is general framework for conditional generation, and we have already demon- strated how it can be used for conditional generation in GANs. In this section, we extend our method to the Variational Autoencoder (VAE) [Kingma and Welling, 2014] to showcase a useful byproduct of our formulation, namely the ability to generate unseen label combinations. In particular, intro- duced in [Georgopoulos et al., 2020], this is a multi-label setting where one (or more) combinations not seen in the training set. The method is then evaluated based on its ability to generate the unseen attribute combinations. To this end, we implement CoPE- VAE, a variation of the conditional VAE [Kingma and Welling, 2014] where both the encoder and decoder are polynomials. We perform experiments on CelebA using the annotated attributes of smile and gender. Similar to |Georgopoulos et al., 2020] remove the combination (Smiling, Female) from the training set. The results in Figure highlight the efficacy of the proposed conditioning method in disentangling the two labels and leveraging the multiplicative interactions synthesize the missing combination.
4.3 Inverse problems imaging
In the following paragraphs we evaluate the performance of CoPE inverse problems. We select super-resolution and block-inpaintin as two popular tasks.
8

---

Table 5: Quantitative evaluation super-resolution with II-Net-based generator Cars196. The
task on the left is super-resolution 16x, while on the right the task is super-resolution 8x. Our
variant of SPADE, i.e., SPADE (details in sec. G), vastly improves the original SPADE. The full two-variable model, i.e., CoPE, outperforms the compared methods.
Super-resolution 16x Cars196
Super-resolution 8x
Cars196
Model
FID (.)
Model
FID (J)
SSIM (1)
LPIPS (.)
SPADE
111.75 + 13.41
SPADE
119.18 + 14.82
0.32
0.178
II-Ne--SCCONC
80.16 + 12.42
SINNNC
186.42 + 40.84
0.31
0.200
SPADE-CoPE
72.63 - 3.18
SPADE-COR CoPE
64.76 I 8.26
0.49
0.135
CoPE
60.42 + 6.19
CoPE
62.76 + 4.37
0.53
0.127
The core architectures remain as in the experiment above, i... CoPE and II-Net-IICOC implement products of polynomials. A single change is made in the structure of the discriminator: Motivated by [Miyato and Koyama, 2018]. we include an elementwise product of Σn with the real/fake image in the discriminator. This stabilizes the training and improves the results. Even though architectures specialized for a single task (e.g., Ledig et al. [2017]) perform well in that task, their well-selected inductive biases (e.g., perceptual or & loss) do not generalize well in other domains or different conditional inputs. Our goal is not to demonstrate state-of-the-ae results, but rather to scrutinize the effectiveness of the proposed formulation in different conditional generation tasks. To that end, we consider II-Net-SICONC, SPADE and SPADE-COPE as the baselines.
We experiment with two settings in super-resolution: one that the input image is down-sampled 8x and one that it is down- wwwn-sampled 16x The two settings enable us to test the granularity of CoPE at differentt scales. In super-resolution 16x, Eu (i.e., the low-resolution input) has 48 dimensions, while in super-resolution 8x, Zu has 192 dimensions. The FID scores Cars196 for the task of super-resolution are reported in Table 5. In addition, for the experiment super-resolution 8x, the SSIM [Wang et al., 2004] and LPIPS [Zhang et al., 2018] are reported as widely-used metrics in inverse imaging tasks. Notice that the performance of II-Net-SICONC deteriorates substantially when the dimensionality of the conditional variable increases. That validates our intuition about the concatenation in the input of the generator (sec. E), i.e., that the inductive bias of single-variable PNNs might not fare well in conditional generation tasks. We also report the SPADE-CoPE: which captures higher-order correlations with respect the first variable as well (further details in sec. G). The proposed SPADE-CoPE outperforms the original SPADE, however it cannot outperform the full two-variable model, i.e., CoPE. The results indicate that CoPE performs well even when the conditional input is an image.
Beyond the quantitative results, qualitative results provide different perspective on what the mappings learn. Qualitative results on the super-resolution experiments on Cars 196 are provided in Fig. 8. We also provide synthesized results on both super-resolution 8x and block-inpainting on CelebA in Fig. 8 and Fig. 3 respectively. For each conditional image, different noise vectors z1 are sampled. Notice that the corresponding synthesized images differ in the fine details. For instance, changes in the mouth region, the car type or position and even background changes are observed. Thus, CoPE synthesizes realistic images that i) correspond the conditional input, ii) vary in the fine details. Similar variation has emerged even when the source and the target domains differ substantially, e.g., the translation of MNIST digits to SVHN digits (sec. H.3). We should mention that the aforementioned experiments were conducted only using the adversarial learning loss. In the literature, regularization techniques have been proposed specifically for mmaee-to-maage translation, e.g., Yang et al. [2019]. Lee et al. [2019]. However, such works utilize additional losses and even require additional networks for training, which makes the training computationally demanding and more sensitive to design choices.
Discussion
CoPE can be used for various conditional generation tasks as the experimental evaluation in both sec. 4 and the supplementary illustrate. Namely, CoPE can synthesize diverse content, which is typically tackled using auxiliary losses or networks in conditional GANs. We expect this attribute of diverse generation to be useful in inverse tasks, where multiple latent sharp images can correspond to single corrupted image. The diverse generation can be attributed to the higher-order correlations between

---

the noise and the conditional variable. Such higher-order correlations also enable synthesizing images with unseen attribute combinations (sec. 4.2).
One limitation of our work is that our method has not been tried in large-scale synthesis, e.g., like Brock al. [2019]. However, only very limited number of labs/institutions have access to such resources. In addition, our single-GPU training does have a reduced energy footprint when compared the multi-GPU setups of large scale GANs. We believe that the proposed method has merit despite the single-GPU training. Indeed, we demonstrate that CoPE can perform well in different tasks, which could help reduce the search for independent methods for every single task.
An interesting future step would be to evaluate the performance of the proposed CoPE in training with limited data, e.g., using techniques similar to Karras et al. [2020a]. This task can also encourage architecture discovery, similar to the various architectures devised in Chrysos et al. [2020]. In this work, we have demonstrated two such architectures with recursive formulations in (2) and (14), however additional architectures can be designed by changing the tensor decomposition.
The equations of (2) and (14) express polynomial expansions of arbitrary order, which can approxi- mate the target function without using activation functions between the recursive terms. However, in sec. 4.1 we have also experimented with generators that include activation functions between recursive terms (i.e., the experiment with SNGAN variants). In the future, we intend to study how to include the activation functions in the formulation and study the properties of such piecewise polynomial expansions. Lastly, demystifying the relationship between the order of the polynomial expansion and the expressivity (e.g., the implementation details for the order in sec. G) is a promising direction.
Societal impact of image generation: Manipulation of images is made possible through algorithms and architectures like ours with well-studied potential negative applications. The rapid progress in GAN-based image synthesis has made the discussion imperative. We encourage further work to be conducted on understanding how to detect synthesized images, e.g., in the conditional generation setting. For instance, our method can be used for training powerful classifiers that detect synthesized images.
Conclusion
We have introduced CoPE for conditional data generation. CoPE expresses polynomial expansion of two input variables, i.c., a noise vector and a conditional variable. We exhibit how previously published methods, such as SPADE and sBN, can be considered as special forms of this two-variable polynomial expansion. Notably, CoPE can be augmented to accept an arbitrary number of conditional variables inputs. The empirical evaluation confirms that our framework images in five diverse tasks, including inverse problems and class- conditionl generation. Inverse problems, such as super-resolution, sampling different noise vectors results in plausible differences in the synthesized image. We derive two recursive formulations. i.e., (2) and (14), but new task-specific formulation can be easily defined. We expect this to be useful in learning from different modalities, such as visual question answering (VQA) text-to-speech synthesis, since CoPE capture high-order auto- and cross-correlations among the input variables.
synthesize realistic
benefit from the proposed framework; we showcase that
Acknowledgements
This project was sponsored by the Department of the Navy, Office of Naval Research(ONR)) under a grant number N62909-17-1-2111, We are thankful to the reviewers for their constructive feedback and recommendations on how to improve our manuscript.
References
A. Almahairi, Rajeswar, A. Sordoni, P. Bachman, and A. Courville. Augmented cyclegan: Learning many-to-many mappings from unpaired data. In International Conference on Machine Learning (ICML), 2018.
D. Bahdanau, K. Cho, and Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015.
10

---

A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
T. F. Chan and K. Chen. An optimization-based multilevel algorithm for total variation image denoising. Multiscale Modeling & Simulation, 5(2):615-645 2006.
Chen, M. Lucic, N. Houlsby, and S. Gelly. On self modulation for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
X. Chen, Y. Duan, R. Houthooft, Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems (NeurIPS), pages 2172-2180, 2016.
Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo. Stargan: Unified generative adversarial networks for multi-domain taage-t--image translation. Conference on Computer Vision and Pultern Recognition (CVPR), pages 8789-8797, 2018.
Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha. Stargan v2: Diverse image synthesis for multiple domains. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 8188-818 2020.
G. Chrysos, S. Moschoglou, Y. Panagakis, and S. Zafeiriou. Polygan: High-order polynomial generators. arXiv preprint arXiv: 06571, 2019.
G. Chrysos, S. Moschoglou, G. Bouritsas, Y. Panagakis, J. Deng, and S. Zafeiriou. xenets: Deep polynomial neural networks. Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1):53-65, 2018.
H. De Vries, F. Strub, J. Mary, H. Larochelle, O. Pietquin, and A. C. Courville. Modulating early visual processing by language. In Advances in neural information processing systems (NeurIPS), pages 6594-6604, 2017.
O. Debals and L. De Lathauwer. The concept of tensorization. Technical report, Technical Report 17-99, ESAT- STADIS, KU Leuven, Belgium, 2017.
V. Dumoulin, J. Shlens, and M. Kudlur. learned representation for artistic style. 2017.
M. Georgopoulos, G. Chrysos, M. Pantic, and Y. Panagakis. Multilinear latent conditioning for gencrating unseen attribute combinations. International Conference on Machine Learning (ICML), pages 1442-3451. PMLR, 2020.
Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems (NeurIPS), 2014.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances neural information processing systems (NeurIPS), pages 6626-6637 2017.
Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 98))1735-1178) 1997.
R. Huang, S. Zhang, T. Li, R. He, al. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. In International Conference on Computer Vision (ICCV), 2017.
X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz. Multimodal unsupervised image- toagge-t-iiee to-image translation. In European Conference Computer Vision (ECCV), pages 172-189, 2018.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference Machine Learning (ICML), 2015.

---

P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. mmage-to-image translation with conditional adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR). 2017.
M. Jayakumar, W. M. Czarnekki, Menick, Schwarz, J. Rae, S. Osindero, W. Teh, Harley, and R. Pascanu. Multiplicative interactions and where to find them. In International Conference on Learning Representations (ICLR), 2020.
Y. Jin, Zhang, M. Li, Tian, and H. Zhu. Towards the high-quality anime characters generation with generative adversarial networks. In Proceedings of the Machine Learning for Creativity and Design Workshop at NeurIPS, 2017.
Kaneko, Y. Ushiku, and T. Harada. Label-noise robust generative adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2467-2476, 2019.
T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Karras, M. Aittala, Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. In Advances in neural information processing systems (NeurIPS), 2020a.
Karras, Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 81 110-8119, 2020b.
D. P. Kingma and M. Welling. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR), 2014.
T. G. Kolda and B. W. Bader. Tensor decompositions and applications. SIAM review, 51(3):455-500, 2009.
Krause, Stark, Deng, and L. Fei-Fei. 3d object representations for fine- fine-grained categorization. In Conference on Computer Vision and Pattern Recognition Workshops (CVPR"W) pages 554 561, 2013.
A. Krizhevsky, V. Nair, and G. Hinton. The cifar-10 dataset. online: http://www. cs. toronto. edu/kriz/cffar html, 55, 2014.
Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2322 1998.
C. Ledig, Theis, Huszár, J. Caballero, A. Cunningham, A. Acosta, Aitken, Tejani, Totz, Z. Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
H.-Y. Lee, H.-Y. Tseng, Q. Mao, J.-B. Huang, Y.-D. Lu, M. Singh, and M.-H. Yang. Drit++: Diverse image-to-image translation via disentangled representations. International Journal of Computer Vision (ICC), pages 1-16, 2020.
Lee, J. Ha, and G. Kim. Harmonizing maximum likelihood with gans for multimodal conditional generation. In International Conference on Learning Representations (ICLR), 2019.
A. Levin, Y. Weiss, F. Durand, and W. T. Freeman. Understanding and evaluating blind deconvolution algorithms. In Conference on Computer Vision and Patern Recognition (CVPR), pages 1964-1971, 2009.
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision (ICCV), pages 3730-3738, 2015.
M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet. Are gans created equal? large-scale study. In Advances in neural information processing systems (NeurIPS), pages 700-709, 2018.
L. Ma, X. Jia, Q. Sun, B. Schiele, T. Tuytelaars, and L. Van Gool. Pose guided person image generation. In Advances neural information processing systems (NeurIPS), pages 406-416, 2017.
12

---

Q. Mao, H.-Y. Lee, H.-Y. Tseng, S. Ma, and M.-H. Yang. Mode seeking generative adversarial networks for diverse image synthesis. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1429-1437, 2019.
M. Maximov, I. Elezi, and L. Laix- Ciagan: Conditional identity anonymization generative adversarial networks. Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Miyato and M. Koyama. egans with projection discriminator. International Conference on Learning Representations (ICLR), 2018.
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICI.R, 2018.
Y. Netzer, T. Wang, A. Coates, Bissacco, B. Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
S. Nikol'skii. Analysis III: Spaces of Differentiable Functions. Encyclopaedia of Mathematical Sciences. Springer Berlin Heidelberg. 2013. ISBN 9783662099612.
A. Odena, Olah, and Shlens. Conditional image synthesis with auxiliary classifier gans. In International Conference Machine Learning (ICMI.,, pages 2642-2651, JMLR. org, 2017.
G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39-500 2020.
T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu. Semantic image synthesis with spatially normalization. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2337- 2346, 2019.
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2016.
S. Rendle. Factorization machines. In 2010 IEEE International conference on data mining, pages 995-1000. IEEE, 2010.
Salimans, I. Goodfellow, W. Zaremba, V. Cheung. Radford, and X. Chen. Improved techniques for training gans. In Advances in neural information processing systems (NeurIPS), pages 2234- 2242, 2016.
Y. Shin and Ghosh. The pi-sigma network: An efficient higher-order neural network for pattern classification and function approximation. In International Joint Conference on Neural Networks, volume 1, pages 13-18, 1991.
A. Siarohin, E. Sangineto, S. Lathuiliere, and N. Sebe. Deformable gans for pose-based human image generation. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 3408-3416, 2018
K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, pages 3483-3491, 2015.
R. Sood, B. Topiwala, K. Choutagunta, R. Sood, and M. Rusu. An application of generative adversarial networks for super resolution medical imaging. In International Conference on Muchine Learning and Applications (ICMLA), pages 326-331, 2018.
M. H. Stone. The generalized weierstrass approximation theorem. Mathematics Magazine, 21(5): 237-254, 1948.
C. Szegedy, W. Liu, Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Conference Computer Vision and Pattern Recognition (CVPR), pages 1-9, 2015.
I. Theis, A. v. d. Oord, and M. Bethge. A note on the evaluation of generative models. In International Conference Learning Representations (ICLR), 2016.
13

---

J. M. Tomczak and M. Welling. Vae with a vampprior. arXiv preprint arXiv: arXiv: 2017.
T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, G. Liu, A. Tao, J. Kautz, and B. Catanzaro. Video-to-video synthesis. In Advances in neural information processing systems (NeurIPS), 2018a.
T.-.. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Conference Computer Vision and Pattern Recognition (CVPR), pages 8798-8807, 2018b.
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions in Image Processing (TIP), 13(4):600-612, 2004.
S. Xie and Z. Tu. Holistically-neste edge detection. In International Conference on Computer Vision (ICCV), 2015.
X. Xu, Sun, Pan, Y. Zhang, H. Pfister, and M.-H. Yang. Learning to super-resolve blurry face and text images. In International Conference Computer Vision (ICCV), pages 251-260, 2017.
D. Yang. S. Hong, Y. Jang, T. Zhao, and H. Lee. Diversity sensitive conditional generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
C. You, G. Li, Y. Zhang, X. Zhang, H. Shan, M. Li, S. Ju, Z. Zhao, Z. Zhang, W. Cong, et al. Ct super-resolution gan constrained by the identical, residual, and cycle learning ensemble (gan-circle). IEEE Transactions on Medical Imaging, 99(1):1882000 2019.
A. Yu and K. Grauman. Fine-Grained Visual Comparisons with Local Learning. In Conference on Computer Vision and Puttern Recognition (CVPR), June 2014.
Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. Huang. Generative image inpainting with contextual attention. In Conference Computer Vision and Pattern Recognition (CVPR), pages 5505-5514 2018a.
X. Yu, B. Fernando, R. Hartley, and F. Porikli. Super-resolving very low-resolution face images with supplementary attributes. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 908-917, 2018b.
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 586-595, 2018.
S. Zhao, H. Ren, A. Yuan, J. Song, N. Goodman, and S. Ermon. Bias and generalization in deep generative models: An empirical study. In Advances neural information processing systems (NeurIPS), pages 10792-10801,200
J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros. Generative visual manipulation the natural image manifold. In European Conference on Computer Vision (ECCV), 2016.
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In International Conference on Computer Vision (ICCV), pages 2223-2232, 2017a.
J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman. Toward multimodal image-to-image translation. Advances neural information processing systems (NeurIPS), pages 465-476 2017b.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] The datasets we use in this work are all publicly available, our setup can be reproduced by interested practitioners. The code is open sourced.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We use standard data splits from well-established datasets (i.e. using the wrappers of Pytorch). The source code is publicly available.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] A single GPU was used in each experiment. The GPU is part of our lab cluster.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] The databases used in our experimental validation have been properly cited. In addition, the code is written by the authors and the models trained by the authors.
(b) Did you mention the license of the assets? [Yes] All the datasets used in this work are publicly available datasets; in addition, they quite popular benchmarks for diverse tasks. All the datasets enable their use for research purposes.
(c) Did you include any new assets either in the supplemental material or as URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] The datasets we utilize are publicly available and contain tens or hudrends of thousands of images. We refer to the authors original papers describing the datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] All the datasets do not have offensive content or personally identifiable information. Celeb-A contains images of celebrities and the images do not contain any offensive content.
(except for Celeb-A)
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "clbbPEtCZAZ", "reviewer": "Reviewer_41qE", "summary": "This paper proposes CoPE, a new neural network architecture for conditional generative modeling. The architecture is derived from a coupled CP decomposition of the parameter tensors of multivariate polynomials, which gives rise to a recursive formula that can be interpreted as a neural network. The method is evaluated on a variety of conditional image generation tasks. \n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe presentation is relatively clear (the supplementary material is especially so), and the method is evaluated on a broad range of tasks. \nThe proposed architecture is in the relatively under-explored area of polynomial neural networks, as opposed to an over-explored area.\n\nWeaknesses:\n\nThe proposed architecture, CoPE, is a special case of \\Pi-net, where the input variable is a concatenation of the conditioning input and the latent noise. This calls into question the novelty of the CoPE compared to \\Pi-net. The main claim of novelty made in the paper (i.e. in Table 1 of the main paper) is that \\Pi-net considers a single input variable only, whereas CoPE considers multiple input variables. But the two are equivalent once the multiple input variables are concatenated together. This is acknowledged on L555 in the supplementary material, and the claim is revised to \\Pi-net being \"not as flexible as the proposed CoPE\" on L547 in the supplementary material. By flexibility, the authors refer to the fact that different structures can be imposed on different submatrices of the embedding matrices, \\Lambda_[i]. I believe all the four differences between CoPE and \\Pi-Net pointed out in Sect. E of the supplementary material boil down to this, and therefore, this forms the core methodological contribution made by this paper. \n\nBased on a close reading of the supplementary material (in particular Sect. E), I understand the paper claims the following concrete contributions:\n1) The precise way in which the embedding matrices are partitioned into submatrices (namely along the boundaries of the conditioning input and the latent noise)\n2) The idea of sharing the columns of embedding matrices corresponding to the conditioning input across different orders (eqn. 19 of the supplementary material)\n3) The idea of imposing convolution structure on the columns of embedding matrices corresponding to the conditioning input, but *not* on the columns corresponding to the latent noise\n4) The concept of setting the columns of embedding matrices corresponding to the latent noise to zero for all orders above a threshold (though it's unclear if this concept was actually used)\n\nI think there are a number of issues:\n\n1) In terms of presentation, I would have preferred for these to be stated upfront in the main paper, rather than having them in a middle section of the supplementary and claiming the single variable input vs. multiple variable input distinction as the novel contribution in the main paper. The latter distinction comes down to the way the recursive formulas (Eqs. (2) and (17)) are written - the original novelty claim is essentially about the writing the formulas in terms of submatrices, rather than the full matrix. A new way of writing a formula is not a valid claim of novelty. \n2) These four concrete contributions are more like design choices than methodological contributions. These normally don't constitute substantial contributions; however, because the paper is on a relatively under-explored area, it could be that these are actually non-trivial design choices and are important in obtaining good empirical results in practice. However, I don't see evidence of this: for example, in Figs. 10 and 17, there isn't a big qualitative difference between \\Pi-Net and CoPE results. Furthermore, for several tasks (e.g.: super-resolution, edge-to-image), no qualitative comparison between \\Pi-Net and CoPE results is shown. Moreover, no ablation study is provided for the different changes that are made relative to \\Pi-Net. Given the limited methodological novelty, substantial improvements should be demonstrated over \\Pi-Net to claim the non-trivialness of these design choices. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While limitations are discussed in Sect. 3.2, the proposed solutions to address the limitations have fundamental limitations of their own. I outline two below:\n\n1) The proposed method for increasing the order of the polynomial in a parameter-efficient way (described on L156-165) assumes a specific factorization of the high-order polynomial. Whereas the original formulation can model any polynomial whose parameters are sufficiently low-rank tensors, the proposed method cannot. \n2) The proposed method for improving stability of high-order polynomials (described on L166-175) only makes the outputs bounded, but does not make the gradients bounded, which means that a tiny change in the input can cause a huge change in the output. So I don't think this technique alone will overcome stability issues. More discussion should be included in the paper on how to get around this issue in practice. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12 hours"}, "extra_text": {}}, {"review_id": "TrntlwMD6Bt", "reviewer": "Reviewer_oDAB", "summary": "The paper has shown an improvisation on the Pi-Net, deep polynomial neural networks, on a conditional multiplicative setting that is shown to work on different computer vision tasks such as conditional image generation, inverse tasks like translation, inpainting, etc. The paper has shown experimental results improvement on class-conditional, image-conditional, and mixed conditional scenarios.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n----------------------\n\n1. The paper has shown an organic way to condition the Pi-Net. The methodology is shown to perform well in various alternate computer vision tasks.\n2. The methods section is well-written and the intuition provided to readers who are not familiar with the concept of polynomial networks is very useful. I really liked the detailed discussion in Sup. Mat. \n3. The figures are succinct and informative, giving a clear picture of the points the authors want to illustrate. I appreciate the multiple evaluations with mean and standard deviation reported.\n\n\nWeakness\n----------------------\n\n+++++++++++++++++++++\n\n1. Significance of the Study: \n\n1.1. The paper has shown a comparison with SPADE, StyleGAN, sBN methods. But, none of them is a state-of-the-art method (if I follow the \"Does it advance the state of the art in a demonstrable way?\" of the reviewer guideline). The choice of SPADE for the inverse problem and conditional generation is not very appropriate when there are more recent conditional generations and inverse problem-related wors such as [1], [2], etc.\n\n1.2. The reviewer is not convinced that cBN/sBN is not applicable to continuous conditions, as sBN predicts the BN parameters from the continuous latent variable. Clarification on why the paper has given such information is much needed. \n\n++++++++++++++++++++\n\n2. Experiments:\n\n2.1.  Discussion on Quantitative Result is missing: The experiments are all done at 64x64 compared to other SOTAs, like BigGAN 512 X 512 on ImageNet. Any justification why this is so?\n\n2.2.   Moreover, the improvement on a commonly used architecture (ResNet) is minimal. It will be nice if the paper provides theoretical justification why the method does not perform well under common settings. (if I follow the \"Does it advance the state of the art in a demonstrable way?\" of the reviewer guideline)\n\n2.3.  Score Disparity: I am a bit confused on why the paper has shown the FID of SNGAN 21.7? Any reason for that? I tried the original code but failed to reproduce the FID of the SNGAN [3] shown in the paper. Which source code the paper has tried? What is the seed value, what are the learning rate, batch size, and other hyperparameter values?\n\n2.4. Time complexity: How about the number of parameters or sampling speed? Is the comparison in the qualitative results table fair in terms of complexity?\n\n2.5. What is the seed value we have taken for this experiment? Are those results are shown for one seed or an interval? If, interval what was the distribution that we sampled seeds?\n\n2.6. [Minor] The paper can move the MNIST-SVHN, Edge-Image translation results to the main draft.\n\n3. Novelty: Using the conditional on an unconditional Pi-Net seems not a good novelty. What architectural changes happen here instead of adding a new path for the conditional? Why the setup works well than a GAN? Is there any theoretical evidence in the paper? Am I missing something?\n\n\nRef\n-------------------\n[1]  Image2StyleGAN++: How to Edit the Embedded Images?\n\n[2]  GAN Compression: Efficient Architectures for Interactive Conditional GANs\n\n[3] Spectral normalization for generative adversarial networks", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " The authors have adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "10hrs"}, "extra_text": {}}, {"review_id": "AdoYwGog3lO", "reviewer": "Reviewer_Mncu", "summary": "## Paper Summary:\n\nDeep Polynomial Neural Networks [Chrysos et al 2020] introduce layers that model higher-order polynomial interactions of a single input. This paper extends PNN’s to the setting where atleast two inputs are provided. (For eg, in conditional image generation, the inputs are the noise variable (z) and class variable ©).  In particular, they incorporate an additional learnable dense projection to incorporate this into the initial PNN formulation (Eq 2)\n\nThe authors provide experimental results on a number of benchmarks to showcase the applicability of their technique.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n\n* Generative modeling to incorporate multiple auxiliary variables is an interesting direction to pursue. \n* I appreciate the authors efforts to showcase the applicability of their method to a number of benchmarks.\n\n## Cons:\n\nThe paper is not well structured (for eg, self-contained with all the background material) and the experiments are weak. Even if the proposed architectural change is incremental (wrt PNN's), consistent and strong improvements over multiple benchmarks with solid experiments would attain a more favourable rating,\nHere are my suggestions for potential improvements.\n\n### Structure:\n* The paper highlights in the introduction that a trivial modification would be to concatenate the two inputs. However, the modification proposed amounts to tiling the noise input across the image and then applying a dense projection which is arguably almost as trivial.\n* A majority of the methods section is the contribution of [Chrysos et al 2020]. For eg, the recursive formulation, product of polynomials and the normalization stability tricks. I suggest that the methods section should solely focus on how [Chrysos et al 2020] is modified to incorporate multiple variables. The prerequisites have to be either moved to a background section or the methods section should be rewritten to make this distinction.\n\n### Experiments:\n* What are N (order) and K (projection dim) used in the experiments? Are they fixed across all tasks and all baselines?\n* [Karras et al 2020] discuss the 3 variants used in their paper, i.e CCP, NCP and NCP-skip. Why was only the CPP-variant used?\n* How was SPADE modified to incorporate conditioning?\n* SPADE is the only baseline, while `GAN-CONC, GAN-ADD and maybe Π-Net-SICONC` are ablations of the current technique. The tables could be changed to reflect this.\n* Some bolded results may not be significant.\n  * Table 3: On IS, SNGAN-CoPE and SNGAN-Spade are within standard deviation\n  * Table 3: On FID, SNGAN-CoPe and SNGAN are comparable.\n  * Table 4: On CIFAR 10, CoPe is comparable to pi-net.\n  *  Table 5: On CARS196, CoPE is comparable to SPADE-CoPE.\n* Section 4.2 is qualitative and hard to judge because of no baselines. I suggest to move it to the appendix.\n* In Table 5, Baseline SNGAN is missing. Why is SPADE-CoPE introduced here but not in the other tables? Why are there two modifications of SPADE? The “CONC” and “ADD” variants seem to be missing as well.\n* The paper mentions StyleGAN in the introduction, but there are no StyleGAN variants (for eg, style + label) in the experiments.\n\nMinor Comments:\n* Line 20: PNNs have demonstrated impressive generation results in faces, animals, cars [Karras et al., 2020b], 22 paintings, medical images [Karras et al., 2020a]:\nC: I suggest the citation be changed to Chrysos et al. [2019]. StyleGAN is a specific type of PNN + other innovations. StyleGan itself has been applied to image-to-image in StarGanv2 (that the authors cite)\n* Line 22: paintings, images -> paintings and images.\n* Line 34 - for conditional data generation -> “for conditional data generation that employs or uses PNNs”\n* Line 39 and Line 44: In CoPE, different architectures can be defined simply by changing the recursive formulation.\n\tIt is not clear what “different architectures” mean. I do not believe the “formulation” is really changed since Eq 2 is applied everywhere \n* Line 152 - Line 155: Is this used in any experiments? In any case, this can be easily incorporated in equation 2.\n* Line 156 - Line 165. The description of the product of polynomials method is confusing. I found the description in Section 3.2 and Figure 5 of [Karras et al 2020] much easier and intuitive to follow.\n* Line 170 -> What does “representations” mean here? Is batch-normalization applied at the output of Eq 4?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 - 7 hours"}, "extra_text": {}}, {"review_id": "4lFUJ1qLvNG", "reviewer": "Reviewer_La4a", "summary": "In this paper, the author proposes a new framework to extend deep polynomial neural networks (PNNs) to multi-variable polynomial neural networks. The proposed PNN framework accepts multiple variables, thus enabling conditional generation where at least two variables, noise and conditioned, are needed. Extensive experiments are conducted to demonstrate the feasibility of conditional PNN under the framework.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies extending deep polynomial neural networks (PNNs) to multi-variable polynomial neural networks. The author argues that the although PNN or PNN-like net can (at least partially) explain the success of current state-of-the-art GANs, current framework either only accepts single variable and thus are only enough for unconditional generation where only noise (i.e. latent vector), or needs special treatment of the conditions that are tailored to a particular tasks.\n\nThus the author proposed to extend PNNs, such that PNNs in a new framework are framed as polynomials of multiple variables (Sec 3.1). Details are taken care of in Sec 3.2 to for empericaly efficiency and stability. The proposed extension looks new in the context of PNN and sets itself from other PNN works in (1) introducing extra variables and deriving formulas  connecting multiple variables in polynomial networks. and (2) providing a universal way to represent conditions that could be either continuous or discrete.\n\nConducted experiments look pretty solid and comprehensive. The authors cover diverse conditional generations tasks where consciousness can be either delicate classes/attributes or continuous images. As this work is to present a new framework that works for a wide range of tasks, it's ok that the qualitative results are not beating the vis-a-vis SOTA model in particular tasks.\n\nAlso this paper is mostly well written and easy to follow. details are well covered in the appendix. Overall, I think this solid work deserves a clear acceptance.\n\nNitpicking: Typo on  L78 \"Alnother\"\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After discussion, there's general consensus among reviewers to reject the work, and with no one strongly in favor of acceptance. The paper's contributions could be better highlighted against related work such as Pi-Nets, and experimental Results are not thorough (both criticisms shared across reviewers). I highly recommend the authors use the reviewer feedback to improve their work in any future conference submission.
Decision: Accept (Poster)