MVP: Multivariate polynomials for conditional generation

Conditional Generative Adversarial Nets ( cGANs ) have been widely adopted for image generation . cGANs take i ) a noise vector and ii ) a conditional variable as input . The conditional variable can be discrete ( e.g. , a class label ) or continuous ( e.g. , an input image ) resulting into class-conditional ( image ) generation and imageto-image translation models , respectively . However , depending on whether the conditional variable is discrete or continuous , various cGANs employ substantially different deep architectures and loss functions for their training . In this paper , we propose a novel framework , called MVP , for conditional data generation . MVP resorts to multivariate polynomials of higher-order and treats in a unified way both discrete and continuous conditional variables . MVP is highly expressive , capturing higher-order auto- and cross-correlations of input variables ( noise vector and conditional variable ) . Tailored sharing schemes are designed between the polynomial ’ s parameter tensors , which result in simple recursive formulas . MVP can synthesize realistic images in both class-conditional and image-to-image translation tasks even in the absence of activation functions between the layers . 1 INTRODUCTION . Modelling high-dimensional distributions and generating samples from complex distributions are fundamental tasks in machine learning . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have demonstrated spectacular results in the two tasks using both unsupervised ( Miyato et al. , 2018 ) and supervised ( Brock et al. , 2019 ) learning . In the unsupervised setting , ( the generator of ) a GAN accepts as input a noise vector zI and maps the noise vector to a high-dimensional output . The supervised models , called conditional Generative Adversarial Nets ( cGANs ) ( Mirza & Osindero , 2014 ) , accept both a noise vector zI and an additional conditional variable zII that facilitates the generation . The conditional variable can be discrete ( e.g. , a class or an attribute label ) or continuous ( e.g. , a low-resolution image ) . The impressive results obtained with both discrete conditional input ( Brock et al. , 2019 ) and continuous conditional input ( Park et al. , 2019 ; Ledig et al. , 2017 ) have led to a plethora of applications that range from text-to-image synthesis ( Qiao et al. , 2019 ) to deblurring ( Yan & Wang , 2017 ) and medical analysis ( You et al. , 2019 ) . Despite the similarity in the formulation for discrete and continuous conditional input ( i.e. , learning the function GpzI , zIIq ) , the literature has focused on substantially different architectures and losses . Frequently , techniques are simultaneously developed , e.g. , the self-attention in the class-conditional Self-Attention GAN ( Zhang et al. , 2019 ) and in the Attention-GAN ( Chen et al. , 2018 ) with continuous conditional input . This delays the progress since practitioners develop twice as many architectures and losses for every case . A couple of straightforward ideas can be employed to unify the behavior of the two conditional variable types . One idea is to use an encoder network to obtain representations that are independent of the conditional variable . This has two drawbacks : i ) the network ignores the noise and a deterministic one-variable mapping is learned ( Isola et al. , 2017 ) , ii ) such encoder has not been successful so far for discrete conditional input . An alternative idea is to directly concatenate the labels in the latent space instead of finding an embedding . In AC-GAN ( Odena et al. , 2017 ) the class labels are concatenated with the noise ; however , the model does not scale well beyond 10 classes . We argue that concatenation of the input is only capturing additive correlation and not higher-order interactions between the inputs . A detailed discussion is conducted on sec . D ( in the Appendix ) . A polynomial expansion with respect to the input variables can capture such higher-order correlations . Π-Net ( Chrysos et al. , 2020 ) casts the function approximation into a polynomial expansion of a single input variable . By concatenating the input variables , we can express the function approximation as a polynomial of the fused variable . However , the concatenation reduces the flexibility of the model significantly , e.g. , it enforces the same order of expansion with respect to the different variables and it only allows the same parameter sharing scheme to all variables . We introduce a multivariate framework , called MVP , for conditional data generation . MVP resorts to multivariate polynomials with two input variables , i.e. , zI for the noise vector and zII for the conditional variable . MVP captures higher-order auto- and cross-correlations between the variables . By imposing a tailored structure in the higher-order interactions , we obtain an intuitive , recursive formulation for MVP . The formulation is flexible and enables different constraints to be applied to each variable and its associated parameters . The formulation can be trivially extended to M input variables . In summary , our contributions are the following : • We introduce a framework , called MVP , that expresses a high-order , multivariate polynomial for conditional data generation . Importantly , MVP treats both discrete and continuous conditional variables in a unified way . • We offer an in-depth relationship with state-of-the-art works , such as SPADE ( Park et al. , 2019 ) , that can be interpreted as polynomial expansions . We believe this perspective better explains the success of such architectures and offers a new direction for their extension . • MVP is trained on eight different datasets for both class-conditional generation and imageto-image translation tasks . The trained models rely on both input variables , i.e. , they do not ignore the noise vector . • To illustrate the expressivity of the model , we also experiment with generators that do not use activation functions between the layers . We verify that MVP can synthesize realistic images even in the absence of activation functions between the layers . The source code of MVP will be published upon the acceptance of the paper . 2 RELATED WORK . The literature on conditional data generation is vast ; dedicated surveys per task ( Agnese et al. , 2019 ; Wu et al. , 2017b ) can be found for the interested reader . Below , we review representative works in conditional generation and then we summarize the recent progress in multiplicative interactions . 2.1 CONDITIONAL GENERATIVE MODELS . The challenging nature of image/video generation has led to a proliferation of conditional models . Although cGAN ( Mirza & Osindero , 2014 ) is a general framework , since then the methods developed for conditional generation differ substantially depending on the type of conditional data . We present below representative works of the two categories , i.e. , discrete and continuous conditional data , and their combination . Discrete conditional variable : This is most frequently used for class-conditional generation ( Miyato et al. , 2018 ; Brock et al. , 2019 ; Kaneko et al. , 2019 ) . Conditional normalization ( Dumoulin et al. , 2017 ; De Vries et al. , 2017 ) techniques have been popular in the case of discrete conditional input , e.g. , in generation of natural scenes images ( Miyato et al. , 2018 ; Brock et al. , 2019 ) . Conditional normalization can not trivially generalize to a continuous conditional variable . In AC-GAN ( Odena et al. , 2017 ) , they concatenate the class labels with the noise ; however , their model does not scale well ( i.e. , they train one model per 10 classes ) . The aforementioned methods can not be trivially used or modified for continuous conditional input . Text-to-image generation models ( Qiao et al. , 2019 ; Li et al. , 2019 ; Zhang et al. , 2018 ; Xu et al. , 2018 ) use a specialized branch to embed the text labels . Continuous conditional variable : The influential work of pix2pix ( Isola et al. , 2017 ) has become the reference point for continuous conditional input . The conditional input is embedded in a lowdimensional space ( with an encoder ) , and then mapped to a high-dimensional output ( through a decoder ) . The framework has been widely used for inverse tasks ( Ledig et al. , 2017 ; Pathak et al. , 2016 ; Wu et al. , 2017a ; Iizuka et al. , 2017 ; Huang et al. , 2017 ; Yu et al. , 2018a ; Grm et al. , 2019 ; Xie et al. , 2018 ; Yan & Wang , 2017 ) , conditional pose generation ( Ma et al. , 2017 ; Siarohin et al. , 2018 ; Liang et al. , 2019 ) , representation learning ( Tran et al. , 2017 ) , conditional video generation ( Wang et al. , 2018a ) , generation from semantic labels ( Wang et al. , 2018b ) , image blending ( Wu et al. , 2019 ; Zhan et al. , 2019 ) . We recognize two major drawbacks in the aforementioned methods : a ) they can not be easily adapted for discrete conditional input , b ) they learn a deterministic mapping , i.e. , the noise is typically ignored . However , in many real applications , such as inverse tasks , the mapping is not one-to-one ; there are multiple plausible outputs for every conditional input . The auxiliary losses used in such works , e.g. , ` 1 loss ( Isola et al. , 2017 ) , perceptual loss ( Ledig et al. , 2017 ) , are an additional drawback . Those losses both add hyper-parameters that require tuning and are domain-specific , thus it is challenging to transfer them to different domains or even different datasets . On the contrary , in our experiments , we do not use any additional loss . Discrete and continuous conditional variables : Few works combine both discrete and continuous conditional inputs ( Yu et al. , 2018b ; Xu et al. , 2017 ; Lu et al. , 2018 ) . However , these methods include significant engineering ( e.g. , multiple discriminators ( Xu et al. , 2017 ) , auxiliary losses ) , while often the generator learns to ignore the noise ( similarly to the continuous conditional input ) . Antipov et al . ( 2017 ) design a generator for face aging . The generator combines continuous with discrete variables ( age classes ) , however there is no Gaussian noise utilized , i.e. , a deterministic transformation is learned for each input face . InfoGAN ( Chen et al. , 2016 ) includes both discrete and continuous conditional variables . However , the authors explicitly mention that additional losses are required , otherwise the generator is ‘ free to ignore ’ the additional variables . The idea of Li et al . ( 2020 ) is most closely related to our work . They introduce a unifying framework for paired ( Isola et al. , 2017 ) and unpaired ( Zhu et al. , 2017a ) learning . However , their framework assumes a continuous conditional input , while ours can handle discrete conditional input ( e.g. , class labels ) . In addition , their method requires a pre-trained teacher generator , while ours consists of a single generator trained end-to-end . Diverse data generation : Conditional image generation often suffers from deterministic mappings , i.e. , the noise variable has often negligible or negative impact in the generator ( Zhu et al. , 2017b ; Isola et al. , 2017 ) . This has been tackled in the literature with additional loss terms and/or auxiliary network modules . A discussion of representative methods that tackle diverse generation is deferred to sec . I in the Appendix . In Table 1 the differences of the core techniques are summarized . Even though diverse generation is a significant task , we advocate that learning a generator does not ignore the input variables can be achieved without such additional loss terms . We highlight that diverse generation is a byproduct of MVP and not our main goal . Particularly , we believe that diverse images can be synthesized because the higher-order correlations of the input variables are captured effectively the proposed method .

Summary: This paper proposes a conditional generation framework (cGAN) that bridges the gap between discrete and continuous variable used in the generation. They do so by proposing a new network architecture that implements higher order multi variate polynomials (MVP). They show that MVP generalizes well to different types of conditional variables and has good expressivity even in the absence of activation functions.