Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes CoPE, a new neural network architecture for conditional generative modeling. The architecture is derived from a coupled CP decomposition of the parameter tensors of multivariate polynomials, which gives rise to a recursive formula that can be interpreted as a neural network. The method is evaluated on a variety of conditional image generation tasks. 

main_review: Strengths:

The presentation is relatively clear (the supplementary material is especially so), and the method is evaluated on a broad range of tasks. 
The proposed architecture is in the relatively under-explored area of polynomial neural networks, as opposed to an over-explored area.

Weaknesses:

The proposed architecture, CoPE, is a special case of \Pi-net, where the input variable is a concatenation of the conditioning input and the latent noise. This calls into question the novelty of the CoPE compared to \Pi-net. The main claim of novelty made in the paper (i.e. in Table 1 of the main paper) is that \Pi-net considers a single input variable only, whereas CoPE considers multiple input variables. But the two are equivalent once the multiple input variables are concatenated together. This is acknowledged on L555 in the supplementary material, and the claim is revised to \Pi-net being "not as flexible as the proposed CoPE" on L547 in the supplementary material. By flexibility, the authors refer to the fact that different structures can be imposed on different submatrices of the embedding matrices, \Lambda_[i]. I believe all the four differences between CoPE and \Pi-Net pointed out in Sect. E of the supplementary material boil down to this, and therefore, this forms the core methodological contribution made by this paper. 

Based on a close reading of the supplementary material (in particular Sect. E), I understand the paper claims the following concrete contributions:
1) The precise way in which the embedding matrices are partitioned into submatrices (namely along the boundaries of the conditioning input and the latent noise)
2) The idea of sharing the columns of embedding matrices corresponding to the conditioning input across different orders (eqn. 19 of the supplementary material)
3) The idea of imposing convolution structure on the columns of embedding matrices corresponding to the conditioning input, but *not* on the columns corresponding to the latent noise
4) The concept of setting the columns of embedding matrices corresponding to the latent noise to zero for all orders above a threshold (though it's unclear if this concept was actually used)

I think there are a number of issues:

1) In terms of presentation, I would have preferred for these to be stated upfront in the main paper, rather than having them in a middle section of the supplementary and claiming the single variable input vs. multiple variable input distinction as the novel contribution in the main paper. The latter distinction comes down to the way the recursive formulas (Eqs. (2) and (17)) are written - the original novelty claim is essentially about the writing the formulas in terms of submatrices, rather than the full matrix. A new way of writing a formula is not a valid claim of novelty. 
2) These four concrete contributions are more like design choices than methodological contributions. These normally don't constitute substantial contributions; however, because the paper is on a relatively under-explored area, it could be that these are actually non-trivial design choices and are important in obtaining good empirical results in practice. However, I don't see evidence of this: for example, in Figs. 10 and 17, there isn't a big qualitative difference between \Pi-Net and CoPE results. Furthermore, for several tasks (e.g.: super-resolution, edge-to-image), no qualitative comparison between \Pi-Net and CoPE results is shown. Moreover, no ablation study is provided for the different changes that are made relative to \Pi-Net. Given the limited methodological novelty, substantial improvements should be demonstrated over \Pi-Net to claim the non-trivialness of these design choices. 
limitations_and_societal_impact: While limitations are discussed in Sect. 3.2, the proposed solutions to address the limitations have fundamental limitations of their own. I outline two below:

1) The proposed method for increasing the order of the polynomial in a parameter-efficient way (described on L156-165) assumes a specific factorization of the high-order polynomial. Whereas the original formulation can model any polynomial whose parameters are sufficiently low-rank tensors, the proposed method cannot. 
2) The proposed method for improving stability of high-order polynomials (described on L166-175) only makes the outputs bounded, but does not make the gradients bounded, which means that a tiny change in the input can cause a huge change in the output. So I don't think this technique alone will overcome stability issues. More discussion should be included in the paper on how to get around this issue in practice.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper has shown an improvisation on the Pi-Net, deep polynomial neural networks, on a conditional multiplicative setting that is shown to work on different computer vision tasks such as conditional image generation, inverse tasks like translation, inpainting, etc. The paper has shown experimental results improvement on class-conditional, image-conditional, and mixed conditional scenarios.
main_review: Strengths
----------------------

1. The paper has shown an organic way to condition the Pi-Net. The methodology is shown to perform well in various alternate computer vision tasks.
2. The methods section is well-written and the intuition provided to readers who are not familiar with the concept of polynomial networks is very useful. I really liked the detailed discussion in Sup. Mat. 
3. The figures are succinct and informative, giving a clear picture of the points the authors want to illustrate. I appreciate the multiple evaluations with mean and standard deviation reported.


Weakness
----------------------

+++++++++++++++++++++

1. Significance of the Study: 

1.1. The paper has shown a comparison with SPADE, StyleGAN, sBN methods. But, none of them is a state-of-the-art method (if I follow the "Does it advance the state of the art in a demonstrable way?" of the reviewer guideline). The choice of SPADE for the inverse problem and conditional generation is not very appropriate when there are more recent conditional generations and inverse problem-related wors such as [1], [2], etc.

1.2. The reviewer is not convinced that cBN/sBN is not applicable to continuous conditions, as sBN predicts the BN parameters from the continuous latent variable. Clarification on why the paper has given such information is much needed. 

++++++++++++++++++++

2. Experiments:

2.1.  Discussion on Quantitative Result is missing: The experiments are all done at 64x64 compared to other SOTAs, like BigGAN 512 X 512 on ImageNet. Any justification why this is so?

2.2.   Moreover, the improvement on a commonly used architecture (ResNet) is minimal. It will be nice if the paper provides theoretical justification why the method does not perform well under common settings. (if I follow the "Does it advance the state of the art in a demonstrable way?" of the reviewer guideline)

2.3.  Score Disparity: I am a bit confused on why the paper has shown the FID of SNGAN 21.7? Any reason for that? I tried the original code but failed to reproduce the FID of the SNGAN [3] shown in the paper. Which source code the paper has tried? What is the seed value, what are the learning rate, batch size, and other hyperparameter values?

2.4. Time complexity: How about the number of parameters or sampling speed? Is the comparison in the qualitative results table fair in terms of complexity?

2.5. What is the seed value we have taken for this experiment? Are those results are shown for one seed or an interval? If, interval what was the distribution that we sampled seeds?

2.6. [Minor] The paper can move the MNIST-SVHN, Edge-Image translation results to the main draft.

3. Novelty: Using the conditional on an unconditional Pi-Net seems not a good novelty. What architectural changes happen here instead of adding a new path for the conditional? Why the setup works well than a GAN? Is there any theoretical evidence in the paper? Am I missing something?


Ref
-------------------
[1]  Image2StyleGAN++: How to Edit the Embedded Images?

[2]  GAN Compression: Efficient Architectures for Interactive Conditional GANs

[3] Spectral normalization for generative adversarial networks
limitations_and_societal_impact:  The authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the author proposes a new framework to extend deep polynomial neural networks (PNNs) to multi-variable polynomial neural networks. The proposed PNN framework accepts multiple variables, thus enabling conditional generation where at least two variables, noise and conditioned, are needed. Extensive experiments are conducted to demonstrate the feasibility of conditional PNN under the framework.
main_review: This paper studies extending deep polynomial neural networks (PNNs) to multi-variable polynomial neural networks. The author argues that the although PNN or PNN-like net can (at least partially) explain the success of current state-of-the-art GANs, current framework either only accepts single variable and thus are only enough for unconditional generation where only noise (i.e. latent vector), or needs special treatment of the conditions that are tailored to a particular tasks.

Thus the author proposed to extend PNNs, such that PNNs in a new framework are framed as polynomials of multiple variables (Sec 3.1). Details are taken care of in Sec 3.2 to for empericaly efficiency and stability. The proposed extension looks new in the context of PNN and sets itself from other PNN works in (1) introducing extra variables and deriving formulas  connecting multiple variables in polynomial networks. and (2) providing a universal way to represent conditions that could be either continuous or discrete.

Conducted experiments look pretty solid and comprehensive. The authors cover diverse conditional generations tasks where consciousness can be either delicate classes/attributes or continuous images. As this work is to present a new framework that works for a wide range of tasks, it's ok that the qualitative results are not beating the vis-a-vis SOTA model in particular tasks.

Also this paper is mostly well written and easy to follow. details are well covered in the appendix. Overall, I think this solid work deserves a clear acceptance.

Nitpicking: Typo on  L78 "Alnother"

limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: ## Paper Summary:

Deep Polynomial Neural Networks [Chrysos et al 2020] introduce layers that model higher-order polynomial interactions of a single input. This paper extends PNN’s to the setting where atleast two inputs are provided. (For eg, in conditional image generation, the inputs are the noise variable (z) and class variable ©).  In particular, they incorporate an additional learnable dense projection to incorporate this into the initial PNN formulation (Eq 2)

The authors provide experimental results on a number of benchmarks to showcase the applicability of their technique.

main_review: ## Pros

* Generative modeling to incorporate multiple auxiliary variables is an interesting direction to pursue. 
* I appreciate the authors efforts to showcase the applicability of their method to a number of benchmarks.

## Cons:

The paper is not well structured (for eg, self-contained with all the background material) and the experiments are weak. Even if the proposed architectural change is incremental (wrt PNN's), consistent and strong improvements over multiple benchmarks with solid experiments would attain a more favourable rating,
Here are my suggestions for potential improvements.

### Structure:
* The paper highlights in the introduction that a trivial modification would be to concatenate the two inputs. However, the modification proposed amounts to tiling the noise input across the image and then applying a dense projection which is arguably almost as trivial.
* A majority of the methods section is the contribution of [Chrysos et al 2020]. For eg, the recursive formulation, product of polynomials and the normalization stability tricks. I suggest that the methods section should solely focus on how [Chrysos et al 2020] is modified to incorporate multiple variables. The prerequisites have to be either moved to a background section or the methods section should be rewritten to make this distinction.

### Experiments:
* What are N (order) and K (projection dim) used in the experiments? Are they fixed across all tasks and all baselines?
* [Karras et al 2020] discuss the 3 variants used in their paper, i.e CCP, NCP and NCP-skip. Why was only the CPP-variant used?
* How was SPADE modified to incorporate conditioning?
* SPADE is the only baseline, while `GAN-CONC, GAN-ADD and maybe Π-Net-SICONC` are ablations of the current technique. The tables could be changed to reflect this.
* Some bolded results may not be significant.
  * Table 3: On IS, SNGAN-CoPE and SNGAN-Spade are within standard deviation
  * Table 3: On FID, SNGAN-CoPe and SNGAN are comparable.
  * Table 4: On CIFAR 10, CoPe is comparable to pi-net.
  *  Table 5: On CARS196, CoPE is comparable to SPADE-CoPE.
* Section 4.2 is qualitative and hard to judge because of no baselines. I suggest to move it to the appendix.
* In Table 5, Baseline SNGAN is missing. Why is SPADE-CoPE introduced here but not in the other tables? Why are there two modifications of SPADE? The “CONC” and “ADD” variants seem to be missing as well.
* The paper mentions StyleGAN in the introduction, but there are no StyleGAN variants (for eg, style + label) in the experiments.

Minor Comments:
* Line 20: PNNs have demonstrated impressive generation results in faces, animals, cars [Karras et al., 2020b], 22 paintings, medical images [Karras et al., 2020a]:
C: I suggest the citation be changed to Chrysos et al. [2019]. StyleGAN is a specific type of PNN + other innovations. StyleGan itself has been applied to image-to-image in StarGanv2 (that the authors cite)
* Line 22: paintings, images -> paintings and images.
* Line 34 - for conditional data generation -> “for conditional data generation that employs or uses PNNs”
* Line 39 and Line 44: In CoPE, different architectures can be defined simply by changing the recursive formulation.
	It is not clear what “different architectures” mean. I do not believe the “formulation” is really changed since Eq 2 is applied everywhere 
* Line 152 - Line 155: Is this used in any experiments? In any case, this can be easily incorporated in equation 2.
* Line 156 - Line 165. The description of the product of polynomials method is confusing. I found the description in Section 3.2 and Figure 5 of [Karras et al 2020] much easier and intuitive to follow.
* Line 170 -> What does “representations” mean here? Is batch-normalization applied at the output of Eq 4?

limitations_and_societal_impact: Yes
review_title: null

Meta-review: After discussion, there's general consensus among reviewers to reject the work, and with no one strongly in favor of acceptance. The paper's contributions could be better highlighted against related work such as Pi-Nets, and experimental Results are not thorough (both criticisms shared across reviewers). I highly recommend the authors use the reviewer feedback to improve their work in any future conference submission.
Decision: Accept (Poster)