Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) have delivered tremendous success in learning to generate samples from high-dimensional distributions ( Goodfellow et al. , 2014 ; Cao et al. , 2018 ; Jabbar et al. , 2021 ) . In the GAN framework , two models are trained simultaneously : a generator G which attempts to generate data from the desired distribution , and a discriminator D which learns to distinguish between real data samples and the fake samples generated by generator . The GAN training problem is typically posed as a zero-sum game for which the generator and discriminator compete to optimize certain objective f p∗ = min G max D f ( G , D ) . The ultimate goal of the GAN training problem is thus to find a saddle point ( also called a Nash equilibrium ) of the above optimization problem over various classes of ( G , D ) . By allowing the generator and discriminator to be represented by neural networks , great advances have been made in generative modeling and signal/image reconstruction ( Isola et al. , 2017 ; Karras et al. , 2019 ; Radford et al. , 2015 ; Wang et al. , 2018 ; Yang et al. , 2017 ) . However , GANs are notoriously difficult to train , for which a variety of solutions have been proposed ; see e.g. , ( Nowozin et al. , 2016 ; Mescheder et al. , 2018 ; Metz et al. , 2016 ; Gulrajani et al. , 2017 ) . One such approach pertains to leveraging Wasserstein GANs ( WGANs ) ( Arjovsky et al. , 2017 ) , which utilize the Wasserstein distance with the ` 1 metric to motivate a particular objective f . In particular , assuming that true data is drawn from distribution px , and the input to the generator is drawn from distribution pz , we represent the generator and discriminator with parameters θg and θd respectively , to obtain the WGAN objective p∗ = min θg max θd Ex∼px [ Dθd ( x ) ] − Ez∼pz [ Dθd ( Gθg ( z ) ) ] . ( 1 ) When G and D are neural networks , neither the inner max , nor , the outer min problems are convex , which implies that min and max are not necessarily interchangeable . As a result , first , there is no guarantees if the saddle points exists . Second , it is unclear to what extent heuristic methods such as Gradient Descent-Ascent ( GDA ) for solving WGANs can approach saddle points . This lack of transparency about the loss landscape of WGANs and their convergence is of paramount importance for their utility in sensitive domains such as medical imaging . For instance , WGANs are commonly used for magnetic resonance image ( MRI ) reconstruction ( Mardani et al. , 2018 ; Han et al. , 2018 ) , where they can potentially hallucinate pixels and alter diagnostic decisions . Despite their prevalent utilization , GANs are not well understood . To shed light on explaining WGANs , in this work , we analyze WGANs with two-layer neural network discriminators through the lens of convex duality and affirm that many such WGANs provably have optimal solutions which can be found with convex optimization , or can be equivalently expressed as convex-concave games , which are well studied in the literature ( Žaković & Rustem , 2003 ; Žaković et al. , 2000 ; Tsoukalas et al. , 2009 ; Tsaknakis et al. , 2021 ) . We further provide interpretation into the effect of various activation functions of the discriminator on the conditions imposed on generated data , and provide convex formulations for a variety of generator-discriminator combinations ( see Table 1 ) . We further note that such shallow neural network architectures can be trained in a greedy fashion to build deeper GANs which achieve state-of-the art for image generation tasks ( Karras et al. , 2017 ) . Thus , our analysis can be extended deep GANs as they are used in practice , and motivates further work into new convex optimization-based algorithms for more stable training . Contributions . All in all , the main contributions of this paper are summarized as follows : • For the first time , we show that WGAN can provably be expressed as a convex problem ( or a convex-concave game ) with polynomial-time complexity for two-layer discriminators and two-layer generators under various activation functions ( see Table 1 ) . • We uncover the effects of discriminator activation on data generation through moment matching , where quadratic activation matches the covariance , while ReLU activation amounts to piecewise mean matching . • For linear generators and quadratic discriminators , we find closed-form solutions for WGAN training as singular value thresholding , which provides interpretability . • Our experiments demonstrate the interpretability and effectiveness of progressive convex GAN training for generation of CelebA faces . 1.1 RELATED WORK . The last few years have witnessed ample research in GAN optimization . While several divergence measures ( Nowozin et al. , 2016 ; Mao et al. , 2017 ) and optimization algorithms ( Miyato et al. , 2018 ; Gulrajani et al. , 2017 ) have been devised , GANs have not been well interpreted and the existence of saddle points is still under question . In one of the early attempts to interpret GANs , ( Feizi et al. , 2020 ) shows that for linear generators with Gaussian latent code and the 2nd order Wasserstein distance objective , GANs coincide with PCA . Others have modified the GAN objective to more implicitly enforce matching infinite-order of moments of the ground truth distribution ( Li et al. , 2017 ; Genevay et al. , 2018 ) . Further explorations have yielded specialized generators with layer-wise subspaces , which automatically discover latent “ eigen-dimensions '' of the data ( He et al. , 2021 ) . Others have proposed explicit mean and covariance matching GAN objectives which provide stable training ( Mroueh et al. , 2017 ) . Regarding convergence of Wasserstein GANs , under the fairly simplistic scenario of linear discriminator and a two-layer ReLU-activation generator with sufficiently large width , saddle points exist and are achieved by GDA ( Balaji et al. , 2021 ) . Indeed , linear discriminators are not realistic as then simply match the mean of distributions . Moreover , the over-parameterization is of high-order polynomial compared with the ambient dimension . For more realistic discriminators , ( Farnia & Ozdaglar , 2020 ) identifies that GANs may not converge to saddle points , and for linear generators with Gaussian latent code , and continuous discriminators , certain GANs provably lack saddle points ( e.g. , WGANs with scalar data and Lipschitz discriminators ) . The findings of ( Farnia & Ozdaglar , 2020 ) raises serious doubt about the existence of optimal solutions for GANs , though finite parameter discriminators as of neural networks are not directly addressed . Convexity has been seldomly exploited for GANs aside from ( Farnia & Tse , 2018 ) , which studies convex duality of divergence measures , where the insights motivate regularizing the discriminator ’ s Lipschitz constant for improved GAN performance . For supervised two-layer networks , a recent of line of work has established zero-duality gap and thus equivalent convex networks with ReLU activation that can be solved in polynomial time for global optimality ( Pilanci & Ergen , 2020 ; Sahiner et al. , 2020a ; Ergen & Pilanci , 2021a ; Sahiner et al. , 2020b ; Bartan & Pilanci , 2021 ; Ergen et al. , 2021 ) . These works focus on single-player networks for supervised learning . However , extending those works to the two-player GAN scenario for unsupervised learning is a significantly harder problem , and demands a unique treatment , which is the subject of this paper . 1.2 NOTATION . Throughout the paper , we denote matrices and vectors as uppercase and lowercase bold letters , respectively . We use 0 ( or 1 ) to denote a vector or matrix of zeros ( or ones ) , where the sizes are appropriately chosen depending on the context . We also use In to denote the identity matrix of size n. For matrices , we represent the spectral , Frobenius , and nuclear norms as ‖ · ‖2 , ‖ · ‖F , and ‖ · ‖∗ , respectively . Lastly , we denote the element-wise 0-1 valued indicator function and ReLU activation as 1 [ x ≥ 0 ] and ( x ) + = max { x , 0 } , respectively . 1.3 PRELIMINARIES . In this paper , we consider the WGAN training problem as expressed in equation 1 . We consider the case of a finite real training dataset X ∈ Rnr×dr which represents the ground truth data from the distribution we would like to generate data . We also consider using finite noise Z ∈ Rnf×df as the input to the generator as fake training inputs . The generator is given as some function Gθg : Rdf → Rdr which maps noise from the latent space to attempt to generate realistic samples using parameters θg , while the discriminator is given by Dθd : Rdr → R which assigns values depending on how realistically a particular input models the desired distribution , using parameters θd . Then , the primary objective of the WGAN training procedure is given as p∗ = min θg max θd 1 > Dθd ( X ) − 1 > Dθd ( Gθg ( Z ) ) +Rg ( θg ) −Rd ( θd ) , ( 2 ) whereRg andRd are convex regularizers on the generator and discriminator parameters , respectively . We will subsequently analyze various realizations of discriminators and generators , and analyze the saddle point problem via convex duality . In particular , one such architecture is that of the two-layer neural network with md neurons and activation function σ , given by Dθd ( X ) = md∑ j=1 σ ( Xuj ) vj 1 . Two activation functions that we will analyze in this work include polynomial activation σ ( t ) = at2 + bt+ c ( of which quadratic and linear activations are special cases where ( a , b , c ) = ( 1 , 0 , 0 ) and ( a , b , c ) = ( 0 , 1 , 0 ) respectively ) , and ReLU activation σ ( t ) = ( t ) + . As a crucial part of our convex analysis , we first need to obtain a convex representation for the ReLU activation . Therefore , we introduce the notion of hyperplane arrangements similiar to ( Pilanci & Ergen , 2020 ) . 1In the case of networks with bias , one can write Dθd ( X ) = ∑m j=1 σ ( Xuj + 1bj ) vj . Hyperplane arrangements . We define the set of hyperplane arrangements asHx : = { diag ( 1 [ Xu ≥ 0 ] ) : u ∈ Rdr } , where each diagonal matrix Hx ∈ Hx encodes whether the ReLU activation is active for each data point for a particular hidden layer weight u . Therefore , for a neuron u , the output of the ReLU activation can be expressed as ( Xu ) + = HxXu , with the additional constraint that ( 2Hx − Inr ) Xu ≥ 0 . Further , the set of hyperplane arrangements is finite , i.e . |Hx| ≤ O ( r ( nr/r ) r ) , where r : = rank ( X ) ≤ min ( nr , dr ) ( Stanley et al. , 2004 ; Ojha , 2000 ) . Thus , we can enumerate all possible hyperplane arrangements and denote them asHx = { H ( i ) x } |Hx|i=1 . Similarly , one can consider the set of hyperplane arrangements from the generated data as { H ( i ) g } |Hg|i=1 , or of the noise inputs to the generator : { H ( i ) z } |Hz|i=1 . With these notions established , we now present the main results2 .

Summary: In this paper, novel theoretical results that express two-layer WGAN training as convex-concave games are established. Ample theoretic theorems and propositions, as well as insightful interpretations, are given under various activation function settings, including linear, quadratic, and ReLU. Finally, proof-of-concept experiments are conducted to verify the theoretic results, where the proposed method that utilizes the explicit convex-concave game solutions are shown to generate better visual quality images than traditional non-convex trained results.   The main novelty of the paper comes from the theoretic results of the convex interpretation of the non-trivial WGAN training problem. Novel closed forms are given for certain training settings. The proposed ProCoGAN model based on the theoretic results is also novel.