Conditional Generation Using Polynomial Expansions

Abstract: Generative modeling has evolved to a notable field of machine learning. Deep polynomial neural networks (PNNs) have demonstrated impressive results in unsupervised image generation, where the task is to map an input vector (i.e., noise) to a synthesized image. However, the success of PNNs has not been replicated in conditional generation tasks, such as super-resolution. Existing PNNs focus on single-variable polynomial expansions which do not fare well to two-variable inputs, i.e., the noise variable and the conditional variable. In this work, we introduce a general framework, called CoPE, that enables a polynomial expansion of two input variables and captures their auto- and cross-correlations. We exhibit how CoPE can be trivially augmented to accept an arbitrary number of input variables. CoPE is evaluated in five tasks (class-conditional generation, inverse problems, edges-to-image translation, image-to-image translation, attribute-guided generation) involving eight datasets. The thorough evaluation suggests that CoPE can be useful for tackling diverse conditional generation tasks. The source code of CoPE is available at https://github.com/grigorisg9gr/polynomialnetsforconditionalgeneration.


Reviews and author responses:

 Thank you for the response. The response, especially the tables, has addressed most of my concerns, and I decide to increase my score.

Deep polynomial neural networks (PNNs) have demonstrated great results on unsupervised image generation, but has not been applied to conditional generation tasks. This paper introduces a framework which allows PNNs for conditional data generation.
The paper proposes two extensions to the core model: a) a formulation that enables an arbitrary number of conditional input variables, b) a new architecture. Experiments show that the proposed conditional model can be applied to five different tasks (class-conditional generation, inverse problems, edges-to-image translation, image-to-image translation, attribute-guided generation).
  1. This paper introduces a framework that extends deep polynomial neural network to conditional data generation, which does not seem to be extensively studied before. However, extending an unconditional model to a conditional model seems to be a straightforward follow-up.
2. The paper is well written.
3. The proposed method can be applied to different conditional generation tasks. However, it is unclear how competitive the results are compared to state-of-the-art models for each application and other state-of-the-art conditional generative models that do not use polynomial neural network architectures. It seems that no such comparison is performed. In order to show the usefulness of the method, such comparisons are needed.

**Detailed comments**:
1. Class-conditional generation: As mentioned at line 234-236 "both SPADE and $\Pi$-Net were not originally built for class-conditional generation", comparing with SPADE or $\Pi$-Net might not successfully show the strength of the proposed method. The FID scores for GAN-CONC, GAN-ADD, SPADE in Table 4 are very bad (e.g., 290+ for CIFAR-10), which raises questions on whether the baselines are tuned and optimized properly. It would be more convincing to compare with state-of-the-art class-conditional generation methods.
2. Section 4.2: It would be useful to mention that comparison with other models are in the appendix. Is there any reason why VAE is used while the other sections are discussing GANs? If the goal is to show that the proposed method extends beyond GANs, it would be nice to also include the VAE results in the other sections, and discuss the related background and details (instead of directly presenting the VAE results in only one section). 
3. Super-resolution: SPADE is not originally designed for super-resolution, it might be important to compare with other stronger super-resolution baselines in order to show the strength of the proposed method. It is unclear if FID score alone is able to capture how good the super-resolution results are. For instance, the generated high resolution images could completely ignore the low resolution images but still achieve a good FID score (e.g., unconditional generation). It would be more convincing to also report other commonly used metrics (e.g., PSNR, SSIM, etc) for super-resolution.
4. minor: It would be good to label which methods are the proposed methods in the table.
5. Will the proposed method work for denoising?
 No potential negative societal impacts are discussed.

This paper proposes a model for conditional generation of images, which relies on the creation of polynomials of the inputs. The paper evaluates this generation process in a multitude of tasks, showing improvement over similar techniques.  The generation method (CoPE) proposed in this paper is closely related to $\Pi$-net, directly designed for conditional generation (accepting two inputs at its base form, one latent and one conditional input). The authors point out several aspects which differentiate their technique from $\Pi$-net, and I think there is significant novelty. I would suggest furthering this discussion in the main paper, since the two techniques are quite similar.

The experimental section (which is the main focus of the paper) considers a multitude of problems, such as class conditional generation, inverse problems, and generation of unseen attribute combinations. This demonstrates the suitability of the proposed method in various useful tasks. Comparisons are made with several polynomial-like architectures, but I think it would be interesting to make comparisons with other architectures as well. For example, in the conditional CIFAR-10 experiment, Karras et al. (2020) report some architectures which appear to have better IS/FID scores than the proposed CoPE technique (Figure 11b in that paper). Of course, the actual scores received also depend on the size of the architecture used in each method, but I believe they should be mentioned at least. I also think that the baselines considered should be the same in Tables 4 and 5 (Sections 4.1 and 4.3 respectively), to enable a better comparison of the proposed method with previous work.

The paper is clear in general and easy to follow. I have the following comments to make which can improve the clarity of the paper.
- Tables should be in the beginning or the end of the page. Right now there are a few lines of text between tables which are easy to miss.
- I believe it would be useful to include a better explanation of the baselines used (mainly GAN-CONC and GAN-ADD), as well as the architecture of the discriminator and the order of CoPE used in the experiments (at least in the supplementary).
- In Figure 4, I think it would be useful to include whether the images presented are randomly picked or curated.

There are also a couple of very minor errors, namely Tables 1 and 4 extending beyond the limits of the page slightly, as well as a typo in line 78.

Overall, this is an interesting work in an equally interesting field, given the possibility of providing improvements for both image generation and inverse problems. I lean towards accepting this work, especially if the issues regarding the experiments I addressed above are resolved during the rebuttal process.

References:

Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020). Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06676.

**Update after rebuttal:**

The reviewers have clarified the points I noted about the experiments, and have fully addressed my concerns. I believe the added material from the feedback and all of the reviewers' responses should be included in the final version of the paper.

As such, I am raising my score. The authors have adequately described the limitations of their work in a dedicated section of the paper (Section 5). 

In the checklist they also state that they only present methods for a task which exists in the literature, so no discussion on negative societal impact is needed. I agree with them, but I also believe that a short statement regarding the societal impact of the task itself (e.g. manipulation of images) could be included for completeness.

The paper introduces conditional polynomial expansion method (CoPE) that expands on the polynomial network framework and adopts it for conditional generation with GANs. CoPE can be seen as a generalization of existing work that allows better use of high-order correlations between input variables and polynomial multiplication for parameter efficiency. Conditional generation experiments suggest that CoPE is indeed a useful mechanism for mixing information from different sources. Revealing more details on the technical implementation would make even stronger submission.  Novelty:

CoPE is a novel construction to my knowledge which can be seen as a natural continuation of the existing work such as Pi-Net or SPADE. A potentially useful reference for a similar (albeit arguably simpler) technique for incorporating interactions between multiple input variables can be found in [1].

Clarity:

The paper is generally well-written and adequately commincates the non-trivial construction of CoPE. 

I would still appreciate if authors could provide more details on technical implementation on some of the mentioned ideas.

1) As authors mention, if one of the inputs is an image a convolution can be applied for embedding it. For example, is it a problem that the dimensionality of the image should remain constant (and low) or this can be circumvented? 
2) Would it be anyhow beneficial to include non-linear components in computing the multiplicative update in eq. 2? As I understand, it would break the connection to the general polynomial expansion method but also make the architecture closer to StyleGAN which could (along with the standard StyleGAN) serve as another baseline.
3) Implementation of  SNGAN-CoPE.

Another question is why in Appendix D matrix P has dimensionality d1 * d2 x o and not (d1 + d2) x o? I'm puzzled here.

A more general question is why authors are specifically interested in the GAN setting and not, for example, in VAEs where the problem of ignoring conditional information is also present. The benefit of considering VAEs would allow to compare models directly in likelihoods and alleviate the possibility of hacking the imperfect image-quality metrics.

Significance:

CoPE seems to be a useful mechanism for conditioning in polynomial networks and will likely be re-used in further work.

Quality:

Overall I don't have many concerns about the method of the evaluation. I would still very much be interested in applying CoPE in a controllable, toy setting where the underlying data distribution is known to verify it indeed recovers the right interaction between variables because mere relying on metrics such as inception score is somewhat futile (this just my subjective opinion).

References:

[1] Rendle, S. (2010, December). Factorization machines. In 2010 IEEE International conference on data mining (pp. 995-1000). IEEE.

**Updated after authors feedback**

Authors addressed my questions and concerns reasonably and I'm therefore raising my score. I encourage authors to incorporate the received feedback (and their replies) into the paper. No concerns.

 We are thankful to the reviewer for the appreciation of our work. 
> I think it would be interesting to make comparisons with other architectures as well. For example, in the conditional CIFAR-10 experiment, Karras et al. (2020) report some architectures which appear to have better IS/FID scores than the proposed CoPE technique (Figure 11b in that paper). Of course, the actual scores received also depend on the size of the architecture used in each method, but I believe they should be mentioned at least. I also think that the baselines considered should be the same in Tables 4 and 5 (Sections 4.1 and 4.3 respectively), to enable a better comparison of the proposed method with previous work.
* We appreciate the reference from the reviewer; the discussion with Karras et al. 2020 will be included in the camera-ready version of the paper. One difference in the reported FID scores is computed using 10k images (following the convention of SNGAN and Π-nets), while Karras et al. 2020 use 50k images. 
* We can indeed include the additional baselines in table 5, however those baselines are not optimized for image-to-image translation tasks, i.e., dense regression. For instance, GAN-CONC concatenates the two inputs, hence when the one input is an image with large dimensionality, it will result in a large number of additional parameters. Also, the conditional batch normalization of Π-nets works only with discrete distributions; discretizing the images (e.g., low-resolution input image) is not trivial to do; we would need to replace the batch normalization with an alternative scheme. 
________
> Tables should be in the beginning or the end of the page. Right now there are a few lines of text between tables which are easy to miss.

We thank the reviewer for pointing this out and will modify the camera-ready version of the paper accordingly. 
________
> I believe it would be useful to include a better explanation of the baselines used (mainly GAN-CONC and GAN-ADD), as well as the architecture of the discriminator and the order of CoPE used in the experiments (at least in the supplementary).


The *discriminator* that we have used in all cases is the one from SNGAN. That discriminator includes 4 ResNet blocks and includes spectral normalization in its layers. In the experiments with inverse problems and image-to-image translation, we have included a projection loss, following [Miyato and Koyama, 2018] (mentioned in lines 258-260 in sec. 4.3).

As for the baselines: There is some explanation on lines 727-759 in the supplementary (sec. G); we offer a succinct description below: 
* GAN-CONC: This is a GAN-variant that concatenates the two input variables. Each block of GAN-CONC can be expressed as: $x_{n} = (A_{[n, I]}^T s_{I} +  A_{[n, II]}^T s_{II}) \parallel (V_{[n]}^T x_{n-1} + B_{[n]}^TB_{[n]})$, where $s_{I}$ and $s_{II}$ are the inputs for the $n^{th}$ recursive term and $\parallel$ is the concatenation operation.
* GAN-ADD: This is a GAN-variant that adds the embeddings of the two input variables. Each block of GAN-ADD can be expressed as: $x_{n} = (A_{[n, I]}^T s_{I} +  A_{[n, II]}^T s_{II}) + (V_{[n]}^T x_{n-1} + B_{[n]}^TB_{[n]})$, where $s_{I}$ and $s_{II}$ are the inputs for the $n^{th}$ recursive term.


Order of CoPE: We have used a different order for the different polynomials in CoPE, as we demonstrate in Fig. 7(d) (supplementary) there are a number of polynomials used in CoPE. The first set of polynomials with fully-connected layers typically include low-degree polynomials (i.e., 1-3 degree), while the main polynomial includes 6-9 degree polynomial, and the output polynomial is also 1-3 degree polynomial. We believe that our open source code will allow the interested practitioner to understand all the details. 

The aforementioned information will be included in the camera-ready version of the paper/supplementary.
________
> In Figure 4, I think it would be useful to include whether the images presented are randomly picked or curated.
The images used in Figure 4 were randomly picked. We have randomly sampled 100 more images (25 for each combination) that can be found in the following [link](https://anonymous.4open.science/r/images_neurips_rebuttal_21-C60D/readme.txt). 

________
> There are also a couple of very minor errors, namely Tables 1 and 4 extending beyond the limits of the page slightly, as well as a typo in line 78.
We appreciate the thorough reading of the reviewer; the corrections will be amended in the camera-ready paper. 
________
> I also believe that a short statement regarding the societal impact of the task itself (e.g. manipulation of images) could be included for completeness.

We appreciate the direct feedback, the following short paragraph will be included: 

Manipulation of images is made possible through algorithms and architectures like ours with well-studied potential negative applications , however the rapid progress in GAN-based image synthesis has made the discussion imperative. We encourage further work to understand how to detect synthesized images, e.g., in the conditional generation setting. For instance, our method can be used for training powerful classifiers that detect synthesized images.

References:

Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020). Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06676.


 We appreciate the effort of the reviewer and their positive comments on our work.  We address below their concerns:

> A useful reference for a similar (albeit arguably simpler) technique for incorporating interactions between multiple input variables can be found in [1].

We thank the reviewer for highlighting this work ; it will be included in the related work section of the final manuscript. 

________

> As authors mention, if one of the inputs is an image a convolution can be applied for embedding it. For example, is it a problem that the dimensionality of the image should remain constant (and low) or can this be circumvented? 

* The dimensionality of the image can vary (e.g., to enable multi-resolution analysis) and since the convolutions per order of the polynomial can be independent, it can be seamlessly handled. 
* Images of higher-dimensionality can also be handled. Higher-dimensionality images contain more details, so naturally a larger-dimensionality of embedding is required to encode those details. Indeed if the parameter $k$ (projection dim) can be appropriately scaled in this case. 

________

> Would it be anyhow beneficial to include non-linear components in computing the multiplicative update in eq. 2? As I understand, it would break the connection to the general polynomial expansion method but also make the architecture closer to StyleGAN which could (along with the standard StyleGAN) serve as another baseline.

This experiment is indeed valuable and this is the reason why we also conduct an experiment with non-linear activation functions. In Table 3 we have used SNGAN which includes non-linearities and we convert it into a polynomial (see also next answer). The results highlight the superior learning capacity of the proposed framework compared to the baselines. Furthermore, table 4 and table 5 include networks without activation functions between the layers. We will clarify the difference between the two settings in the final version of the paper. 
________

> Implementation of SNGAN-CoPE.

We convert the resnet-based generator of SNGAN to a polynomial expansion by making the following modification: After each residual block, we project the embeddings of the two-input variables and do an elementwise multiplication with the output of the residual block. That is, we use the equation 14 of the paper for this purpose. In addition, to provide a fair comparison we do not use the Conditional Batch Normalization (CBN) used in SNGAN (it provides different statistics per class), but convert it to batch normalization.

___________

> Another question is why in Appendix D matrix P has dimensionality d1 * d2 x o and not (d1 + d2) x o? I'm puzzled here.

We appreciate the attentive reviewer for the  remark; the typo has been corrected on our version already. It will be reflected in the camera-ready version. 
________

> A more general question is why authors are specifically interested in the GAN setting and not, for example, in VAEs where the problem of ignoring conditional information is also present. 

The main contribution of our paper is a general framework for conditional generation, which can be easily extended beyond the GAN setting. We focus on the GAN setting due to its status as the state-of-the-art approach for conditional image synthesis. However, we note that we have already performed experiments with VAE in Sec. 4.2, where we study the generation of unseen attribute combinations. We will update the final version of the manuscript to highlight the VAE in this section.

________

> I would still very much be interested in applying CoPE in a controllable, toy setting where the underlying data distribution is known to verify it indeed recovers the right interaction between variables because mere relying on metrics such as inception score is somewhat futile (this just my subjective opinion).

We are thankful to the reviewer for the suggestion. We have implemented a toy setting using a mixture of Gaussians as the data distribution. We use a mixture of three Gaussians and indeed CoPE can recover the data distribution. The $\ell_1$ distance of the recovered means from the ground-truth is 0.12 (the original means are (0, 0), (0, 0.7) and (0.7, 0.25)). 


[1] Rendle, S. (2010, December). Factorization machines. In 2010 IEEE International conference on data mining (pp. 995-1000). IEEE.


 We are thankful to the reviewer for the appreciation of our work. 

>  The writing is mostly clear but is hard to follow at the beginning of the experiment section. Section 3 largely focuses on simple PNNs and their generalization to conditional settings. However, right at the beginning of section 4, the authors mention SNGAN-CoPE and SPADE-CoPE without much explanation of how they are created. 

We thank the reviewer for the suggestion. We will revise the beginning of the experimental section in the camera ready version of the paper (there is an additional content page). 

More concretely: 

* SNGAN-CoPE: We convert the resnet-based generator of SNGAN to a polynomial expansion by making the following modification: After each residual block, we project the embeddings of the two-input variables and do an elementwise multiplication with the output of the residual block. That is, we use the equation 14 of the paper for this purpose. In addition, to provide a fair comparison we do not use the Conditional Batch Normalization (CBN) used in SNGAN (it provides different statistics per class), but convert it to batch normalization.

* SPADE-CoPE: This is a variant we develop to bridge the gap between SPADE and the proposed CoPE. Specifically, we augment the aforementioned SPADE twofold: a) the dense layers in the input space are converted into a polynomial with respect to the variable $z_{I}$ and b) we also convert the polynomial in the output (i.e., the rightmost polynomial in the Figure 7 schematics) to a polynomial with respect to the variable $z_{I}$. This model captures higher-order correlations of the variable  $z_{I}$ that SPADE did not not originally include. This model still includes single variable polynomials, however the input in each polynomial varies and is not only the conditional variable.


________

> It was not clear how the output of the first polynomial is fed to the other. The initial input has two latent variables $z_I$ and $z_{II}$ which will result in a single output which is a polynomial of degree $N_1$. How exactly is this input to obtain a polynomial of degree $N_1 \cdot N_2$? 

The second polynomial of degree $N_2$ can either be a polynomial of one variable (i.e., the output of the previous polynomial) or more variables (i.e., the output of the previous polynomial and one or more of the inputs). In both cases the total degree of the output of the second polynomial will be $N_1 \cdot N_2$. The choice of the type of polynomial in each case is a design option. For instance, in the class-conditional setting we find that the case with two-variable polynomials (i.e., the output of the previous polynomial along with the conditional variable) works better. We will clarify this in the final version of the paper.



 We appreciate the detailed comments of the reviewer and their elaborate review. We address their comments below:

> Class-conditional generation: As mentioned at line 234-236 "both SPADE and Π-Net were not originally built for class-conditional generation", comparing with SPADE or Π-Net might not successfully show the strength of the proposed method. The FID scores for GAN-CONC, GAN-ADD, SPADE in Table 4 are very bad (e.g., 290+ for CIFAR-10), which raises questions on whether the baselines are tuned and optimized properly. It would be more convincing to compare with state-of-the-art class-conditional generation methods.

Please allow us to clarify: 
**Results on Table 4**: Please allow us to clarify: the results in Table 4 (e.g., 290+ for CIFAR-10) are not for the baseline GAN-CONC, GAN-ADD and SPADE models. The results are for a version of the models that does NOT use non-linearities (Π-Net-based). Therefore, the low score is expected, as GAN-CONC and GAN-ADD can express only first-degree polynomials. Similarly, SPADE only expresses a polynomial of the conditional variable and not the input noise, which explains its performance. This comparison is done to stress the expressivity of polynomial functions of different degrees.  For results of the baselines WITH non-linearities, we point the reviewer to Table 3. We will highlight this in the caption of the corresponding tables (e.g. table 5).

**On the choice of baselines**: The baselines of GAN-CONC and GAN-ADD are alternative options to the CoPE model. Those are used in the literature to tackle conditional generation. Those models share similar architecture with the CoPE model for a fair comparison; the difference is that the Hadamard product is converted into a concatenation and an addition respectively. The baselines are compared against the proposed model both in the settings WITH and WITHOUT activation functions in tables 3 and 4 respectively.

Furthermore, as per the request of the reviewer, we compare our model with other class-conditional generative models. sBN and BigGAN are highly cited as the state-of-the-art models in the literature and result in the following table assuming the SNGAN-based setting that is close to ours:


|    Method            |Inception Score |FID                         |
|---------------------|-----------------|-----------------------|
|sBN        |8.34|18.40|
|BIGGAN        |-|14.70|
|CoPE        |**8.77**|**14.22**|

The results demonstrate how **CoPE outperforms both methods**.

________

> Section 4.2: It would be useful to mention that comparisons with other models are in the appendix. Is there any reason why VAE is used while the other sections are discussing GANs? If the goal is to show that the proposed method extends beyond GANs, it would be nice to also include the VAE results in the other sections, and discuss the related background and details (instead of directly presenting the VAE results in only one section). 

The main contribution of our paper is a general framework for conditional generation, which can be easily extended beyond the GAN setting. However, we focus on the GAN setting due to its status as the state-of-the-art approach for conditional image synthesis. The reason we include the study in Sec. 4.2 is not just to showcase how our method extends to the VAE setting, but most importantly, to showcase a useful byproduct of our formulation, namely the ability to generate unseen label combinations. This setting was first studied in [Georgopoulos et al.] and hence, we use their setting (VAE) for fair comparison in their proposed benchmark. We will explain this choice in the main text of the final version of the manuscript.

________

> Super-resolution: SPADE is not originally designed for super-resolution, it might be important to compare with other stronger super-resolution baselines. It would be more convincing to also report other commonly used metrics (e.g., PSNR, SSIM, etc) for super-resolution.

* Stronger baselines: We agree with the reviewer that very strong methods would outperform the proposed work as they are specifically engineered for the task of super-resolution.  However, the goal of this work is not to demonstrate state-of-the-art performance in a particular task, but rather to introduce a way to construct conditional models that perform well in a variety of tasks. Indicatively, we highlight the superiority of the proposed model in conditional generation in GAN and VAE, super-resolution and in-painting. 

* Commonly used metrics: We are thankful to the reviewer for the remark. We have indeed opted to report the SSIM as a standard image-processing metric along with LPIPS. LPIPS [1] is a perceptual metric that has gained popularity in the image processing tasks due to its correlation with human vision. A higher SSIM (or lower LPIPS) illustrates better performance. The following table reports the results on super-resolution $8\times$:

|    Method            |SSIM |LPIPS                         |FID                         |
|---------------------|-----------------|-----------------------|-----------------------|
|SPADE        |0.32|0.178|119.18|
|Π-net        |0.31|0.200|186.42|
|CoPE        |**0.53**|**0.127**|**62.76**|

________

> Will the proposed method work for denoising?

The setting has similarities with super-resolution and block-inpainting, therefore we assume a similar architecture would work. We intend to extend the model for denoising as part of our future work.

[1] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 586-595).


This paper presents a method of conditional image generation where the conditional variables can be discrete or continuous (including other images) or a combination of both. The main idea is based on recently proposed polynomial neural networks which were unconditional and based on continuous latent noise variables as input. This paper extends it to include multiple inputs aiming to model correlations between them. Experiments conducted on 5 different tasks show improvements over considered baselines.   The main idea is simple and intuitive with sound motivation. And the experiments are very thorough and show clear improvements over baselines.

Other comments: 

I am not an expert in this field but the writing is mostly clear but is hard to follow at the beginning of the experiment section. Section 3 largely focuses on simple PNNs and their generalization to conditional settings. However, right at the beginning of section 4, the authors mention SNGAN-CoPE and SPACE-CoPE without much explanation of how they are created. 

Additionally, in section 3.2 about the logarithmic number of layers required, it was not clear how the output of the first polynomial is fed to the other. The initial input has two latent variables z_I and z_II which will result in a single output which is a polynomial of degree N_1. How exactly is this input to obtain a polynomial of degree N1.N2? 



 Yes

Meta-review: This paper proposed an extension of polynomial neural networks (PNNs) for application to conditional generation tasks. Experiments show that the proposed conditional model can be applied to five different tasks (class-conditional generation, inverse problems, edges-to-image translation, image-to-image translation, attribute-guided generation) with relevance to the NeurIPS conference.

The reviewers all agree that the contribution is useful and novel. All reviewers were satisfied with the author response to the minor concerns that were raised in the first round of reviewing, with one reviewer raising their score. The reviewers engaged in committee discussion and reached a consensus recommendation to accept the paper. 

Decision: accept