Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) have delivered tremendous success in learning to generate samples from high-dimensional distributions ( Goodfellow et al. , 2014 ; Cao et al. , 2018 ; Jabbar et al. , 2021 ) . In the GAN framework , two models are trained simultaneously : a generator G which attempts to generate data from the desired distribution , and a discriminator D which learns to distinguish between real data samples and the fake samples generated by generator . The GAN training problem is typically posed as a zero-sum game for which the generator and discriminator compete to optimize certain objective f p∗ = min G max D f ( G , D ) . The ultimate goal of the GAN training problem is thus to find a saddle point ( also called a Nash equilibrium ) of the above optimization problem over various classes of ( G , D ) . By allowing the generator and discriminator to be represented by neural networks , great advances have been made in generative modeling and signal/image reconstruction ( Isola et al. , 2017 ; Karras et al. , 2019 ; Radford et al. , 2015 ; Wang et al. , 2018 ; Yang et al. , 2017 ) . However , GANs are notoriously difficult to train , for which a variety of solutions have been proposed ; see e.g. , ( Nowozin et al. , 2016 ; Mescheder et al. , 2018 ; Metz et al. , 2016 ; Gulrajani et al. , 2017 ) . One such approach pertains to leveraging Wasserstein GANs ( WGANs ) ( Arjovsky et al. , 2017 ) , which utilize the Wasserstein distance with the ` 1 metric to motivate a particular objective f . In particular , assuming that true data is drawn from distribution px , and the input to the generator is drawn from distribution pz , we represent the generator and discriminator with parameters θg and θd respectively , to obtain the WGAN objective p∗ = min θg max θd Ex∼px [ Dθd ( x ) ] − Ez∼pz [ Dθd ( Gθg ( z ) ) ] . ( 1 ) When G and D are neural networks , neither the inner max , nor , the outer min problems are convex , which implies that min and max are not necessarily interchangeable . As a result , first , there is no guarantees if the saddle points exists . Second , it is unclear to what extent heuristic methods such as Gradient Descent-Ascent ( GDA ) for solving WGANs can approach saddle points . This lack of transparency about the loss landscape of WGANs and their convergence is of paramount importance for their utility in sensitive domains such as medical imaging . For instance , WGANs are commonly used for magnetic resonance image ( MRI ) reconstruction ( Mardani et al. , 2018 ; Han et al. , 2018 ) , where they can potentially hallucinate pixels and alter diagnostic decisions . Despite their prevalent utilization , GANs are not well understood . To shed light on explaining WGANs , in this work , we analyze WGANs with two-layer neural network discriminators through the lens of convex duality and affirm that many such WGANs provably have optimal solutions which can be found with convex optimization , or can be equivalently expressed as convex-concave games , which are well studied in the literature ( Žaković & Rustem , 2003 ; Žaković et al. , 2000 ; Tsoukalas et al. , 2009 ; Tsaknakis et al. , 2021 ) . We further provide interpretation into the effect of various activation functions of the discriminator on the conditions imposed on generated data , and provide convex formulations for a variety of generator-discriminator combinations ( see Table 1 ) . We further note that such shallow neural network architectures can be trained in a greedy fashion to build deeper GANs which achieve state-of-the art for image generation tasks ( Karras et al. , 2017 ) . Thus , our analysis can be extended deep GANs as they are used in practice , and motivates further work into new convex optimization-based algorithms for more stable training . Contributions . All in all , the main contributions of this paper are summarized as follows : • For the first time , we show that WGAN can provably be expressed as a convex problem ( or a convex-concave game ) with polynomial-time complexity for two-layer discriminators and two-layer generators under various activation functions ( see Table 1 ) . • We uncover the effects of discriminator activation on data generation through moment matching , where quadratic activation matches the covariance , while ReLU activation amounts to piecewise mean matching . • For linear generators and quadratic discriminators , we find closed-form solutions for WGAN training as singular value thresholding , which provides interpretability . • Our experiments demonstrate the interpretability and effectiveness of progressive convex GAN training for generation of CelebA faces . 1.1 RELATED WORK . The last few years have witnessed ample research in GAN optimization . While several divergence measures ( Nowozin et al. , 2016 ; Mao et al. , 2017 ) and optimization algorithms ( Miyato et al. , 2018 ; Gulrajani et al. , 2017 ) have been devised , GANs have not been well interpreted and the existence of saddle points is still under question . In one of the early attempts to interpret GANs , ( Feizi et al. , 2020 ) shows that for linear generators with Gaussian latent code and the 2nd order Wasserstein distance objective , GANs coincide with PCA . Others have modified the GAN objective to more implicitly enforce matching infinite-order of moments of the ground truth distribution ( Li et al. , 2017 ; Genevay et al. , 2018 ) . Further explorations have yielded specialized generators with layer-wise subspaces , which automatically discover latent “ eigen-dimensions '' of the data ( He et al. , 2021 ) . Others have proposed explicit mean and covariance matching GAN objectives which provide stable training ( Mroueh et al. , 2017 ) . Regarding convergence of Wasserstein GANs , under the fairly simplistic scenario of linear discriminator and a two-layer ReLU-activation generator with sufficiently large width , saddle points exist and are achieved by GDA ( Balaji et al. , 2021 ) . Indeed , linear discriminators are not realistic as then simply match the mean of distributions . Moreover , the over-parameterization is of high-order polynomial compared with the ambient dimension . For more realistic discriminators , ( Farnia & Ozdaglar , 2020 ) identifies that GANs may not converge to saddle points , and for linear generators with Gaussian latent code , and continuous discriminators , certain GANs provably lack saddle points ( e.g. , WGANs with scalar data and Lipschitz discriminators ) . The findings of ( Farnia & Ozdaglar , 2020 ) raises serious doubt about the existence of optimal solutions for GANs , though finite parameter discriminators as of neural networks are not directly addressed . Convexity has been seldomly exploited for GANs aside from ( Farnia & Tse , 2018 ) , which studies convex duality of divergence measures , where the insights motivate regularizing the discriminator ’ s Lipschitz constant for improved GAN performance . For supervised two-layer networks , a recent of line of work has established zero-duality gap and thus equivalent convex networks with ReLU activation that can be solved in polynomial time for global optimality ( Pilanci & Ergen , 2020 ; Sahiner et al. , 2020a ; Ergen & Pilanci , 2021a ; Sahiner et al. , 2020b ; Bartan & Pilanci , 2021 ; Ergen et al. , 2021 ) . These works focus on single-player networks for supervised learning . However , extending those works to the two-player GAN scenario for unsupervised learning is a significantly harder problem , and demands a unique treatment , which is the subject of this paper . 1.2 NOTATION . Throughout the paper , we denote matrices and vectors as uppercase and lowercase bold letters , respectively . We use 0 ( or 1 ) to denote a vector or matrix of zeros ( or ones ) , where the sizes are appropriately chosen depending on the context . We also use In to denote the identity matrix of size n. For matrices , we represent the spectral , Frobenius , and nuclear norms as ‖ · ‖2 , ‖ · ‖F , and ‖ · ‖∗ , respectively . Lastly , we denote the element-wise 0-1 valued indicator function and ReLU activation as 1 [ x ≥ 0 ] and ( x ) + = max { x , 0 } , respectively . 1.3 PRELIMINARIES . In this paper , we consider the WGAN training problem as expressed in equation 1 . We consider the case of a finite real training dataset X ∈ Rnr×dr which represents the ground truth data from the distribution we would like to generate data . We also consider using finite noise Z ∈ Rnf×df as the input to the generator as fake training inputs . The generator is given as some function Gθg : Rdf → Rdr which maps noise from the latent space to attempt to generate realistic samples using parameters θg , while the discriminator is given by Dθd : Rdr → R which assigns values depending on how realistically a particular input models the desired distribution , using parameters θd . Then , the primary objective of the WGAN training procedure is given as p∗ = min θg max θd 1 > Dθd ( X ) − 1 > Dθd ( Gθg ( Z ) ) +Rg ( θg ) −Rd ( θd ) , ( 2 ) whereRg andRd are convex regularizers on the generator and discriminator parameters , respectively . We will subsequently analyze various realizations of discriminators and generators , and analyze the saddle point problem via convex duality . In particular , one such architecture is that of the two-layer neural network with md neurons and activation function σ , given by Dθd ( X ) = md∑ j=1 σ ( Xuj ) vj 1 . Two activation functions that we will analyze in this work include polynomial activation σ ( t ) = at2 + bt+ c ( of which quadratic and linear activations are special cases where ( a , b , c ) = ( 1 , 0 , 0 ) and ( a , b , c ) = ( 0 , 1 , 0 ) respectively ) , and ReLU activation σ ( t ) = ( t ) + . As a crucial part of our convex analysis , we first need to obtain a convex representation for the ReLU activation . Therefore , we introduce the notion of hyperplane arrangements similiar to ( Pilanci & Ergen , 2020 ) . 1In the case of networks with bias , one can write Dθd ( X ) = ∑m j=1 σ ( Xuj + 1bj ) vj . Hyperplane arrangements . We define the set of hyperplane arrangements asHx : = { diag ( 1 [ Xu ≥ 0 ] ) : u ∈ Rdr } , where each diagonal matrix Hx ∈ Hx encodes whether the ReLU activation is active for each data point for a particular hidden layer weight u . Therefore , for a neuron u , the output of the ReLU activation can be expressed as ( Xu ) + = HxXu , with the additional constraint that ( 2Hx − Inr ) Xu ≥ 0 . Further , the set of hyperplane arrangements is finite , i.e . |Hx| ≤ O ( r ( nr/r ) r ) , where r : = rank ( X ) ≤ min ( nr , dr ) ( Stanley et al. , 2004 ; Ojha , 2000 ) . Thus , we can enumerate all possible hyperplane arrangements and denote them asHx = { H ( i ) x } |Hx|i=1 . Similarly , one can consider the set of hyperplane arrangements from the generated data as { H ( i ) g } |Hg|i=1 , or of the noise inputs to the generator : { H ( i ) z } |Hz|i=1 . With these notions established , we now present the main results2 .

Summary: In this paper, the authors analyzed the training of Wasserstein GANs with two-layer neural network discriminators.  They showed that WGAN can provably be expressed as a convex problem (or a convex-concave game) with polynomial-time complexity for two-layer discriminators and two-layer generators under various activation functions.   They uncovered the effects of discriminator activation on data generation through moment matching, where quadratic activation matches the covariance, while ReLU activation amounts to piecewise mean matching. They found closed-form solutions for WGAN training as singular value thresholding, which provides interpretability. Their experiments demonstrate the interpretability and effectiveness of progressive convex GAN training for generation of CelebA faces.