Minimax Optimality (Probably) Doesn't Imply Distribution Learning for GANs

1 INTRODUCTION . When will a generative adversarial network ( GAN ) trained with samples from a distribution D actually output samples from a distribution that is close to D ? This question is one of the most foundational questions in GAN theory—indeed , it was raised since the original paper introducing GANs . However , despite significant interest , this question still remains to be fully understood for general classes of generators and discriminators . A significant literature has developed discussing the role of the training dynamics ( Liu et al. , 2017 ; Li et al. , 2018 ; Arora et al. , 2018 ; Berard et al. , 2019 ; Wiatrak et al. , 2019 ; Thanh-Tung & Tran , 2020 ; Allen-Zhu & Li , 2021 ) , as well as the generalization error of the GAN objective ( Zhang et al. , 2017 ; Arora et al. , 2017 ; Thanh-Tung et al. , 2019 ) . In most cases , researchers have demonstrated that given sufficient training data , GANs are able to learn some specific form of distributions after successful training . Underlying these works appears to be a tacit belief that if we are able to achieve the minimax optimal solution to the population-level GAN objective , then the GAN should be able to learn the target distribution . In this work , we take a closer look at this assumption . What does it mean to learn the target distribution ? As a starting point , we must first formally define what we mean by learning a distribution ; more concretely , what do we mean when we say that two distributions are close ? The original paper of ( Goodfellow et al. , 2020 ) proposed to measure closeness with KL divergence . However , learning the target distribution in KL divergence is quite unlikely to be satisfied for real-world distributions . This is because learning distributions in KL divergence also requires us to exactly recover the support of the target distribution , which we can not really hope to do if the distribution lies in an unknown ( complicated ) low-dimensional manifold . To rectify this , one may instead consider learning in Wasserstein distance , as introduced in the context of GANs by ( Arjovsky et al. , 2017 ) , which has no such “ trivial ” barriers . Recall that the Wasserstein distance between two distributions D1 , D2 over Rd is given by W1 ( D1 , D2 ) = sup Lip ( f ) 1 ED1 [ f ] ED2 [ f ] , where for any f : Rd ! R , we let Lip ( f ) denote the Lipschitz constant of f . That is , two densities are close in Wasserstein distance if no Lipschitz function can distinguish between them . In this work we will focus on Wasserstein distance as it is the most standard notion of distance between probability distributions considered in the context of GANs . Note that if the class of discriminators contains sufficiently large neural networks , then minimax optimality of the GAN objective does imply learning in Wasserstein distance . This is because we can approximate any Lipschitz function arbitrarily well , with an exponentially large network with one hidden layer ( see e.g . ( Poggio et al. , 2017 ) ) . Thus , in this case , minimizing the population GAN objective is actually equivalent to learning in Wasserstein distance . Of course in practice , however , we are limited to polynomially large networks for both the generator and the discriminator . This raises the natural question : Does achieving small error against all poly-size neural network discriminators imply that the poly-size generator has learned the distribution in Wasserstein distance ? One might conjecture that this claim is true , since the generator is only of poly-size . Thus , using a ( larger ) poly-size discriminator ( as opposite to the class of all 1-Lipschitz functions ) might still be sufficient to minimize the actual Wasserstein distance . In this paper , however , we provide strong evidence to the contrary . We demonstrate that widely accepted cryptographic assumptions imply that this is false , even if the generator is of constant depth : Theorem 1.1 ( Informal , see Theorem 3.1 ) . For any n 2 N , let n be the standard Gaussian measure over Rn . Assuming local pseudorandom generators exist , the following holds for any sufficiently large m 2 Z , d , r poly ( m ) , and any diverse1 target distribution D⇤ over [ 0 , 1 ] d given by the pushforward of the uniform distribution Ir on [ 0 , 1 ] r by a constant depth ReLU network of polynomial size/Lipschitzness : 2 There exist generators G : Rm ! Rd computed by ( deterministic ) ReLU networks of constant depth and polynomial size for which no ReLU network of polynomial depth/size/Lipschitzness can tell apart the distributions G ( m ) and D⇤ , yet G ( m ) and D⇤ are ⌦ ( 1 ) -far in Wasserstein distance . While Theorem 1.1 pertains to the practically relevant setting of continuous seed and output distributions , we also give guarantees for the discrete setting . In fact , if we replace D⇤ and m by the uniform distributions over { ±1 } d and { ±1 } m , we show this holds for generators whose output coordinates are given by constant-size networks ( see Theorem 3.2 ) . We defer the formal definition of local pseudorandom generators ( PRGs ) to Section 2.2 . We pause to make a number of remarks about this theorem . First , our theorem talks about the population loss of the GAN objective ; namely , it says that the true population GAN objective is small for this generator G , meaning that for every ReLU network discriminator f of polynomial depth/size/Lipschitzness , we have that |E [ f ( D⇤ ) ] E [ f ( G ( m ) ) ] | 1 d ! ( 1 ) . In other words , our theorem states that even optimizing the true population minimax objective is insufficient for distribution learning . In fact , we show this even when the target distribution can be represented perfectly by some other generative model . Second , notice that our generator is extremely simple : notably , it is only constant depth . On the other hand , the discriminator is allowed to be much more complex , namely any ReLU network of polynomial complexity . This discriminator class thus constitutes the most powerful family of functions we could hope to use in practice . Despite this , we show that the discriminators are still not powerful enough to distinguish the output of the ( much simpler ) generator from the target distribution . Third , our conclusions hold both for d m and d m , so long as the input and output dimensions are related by polynomial factors . 1See Definition 6 . In the discussion proceeding this definition , we give a number of examples making clear that this is a mild and practically relevant assumption to make . 2When we say “ polynomial , ” we are implicitly referring to the dependence on the parameter m , though because d , r are bounded by poly ( m ) , “ polynomial ” could equivalently refer to the dependence on those parameters if they exceeded m. Finally , we formally define the class of “ diverse ” target distributions for which our conclusions hold in Section 2.3 . We note that this class is quite general : for instance , it includes pushforwards of the uniform distribution under random leaky ReLU networks ( see Lemma 2.3 ) . Empirical results . To complement these theoretical results , we also perform some empirical validations of our findings ( see Section 4 ) . Our theorem is constructive ; that is , given a local PRG , we give an explicit generator which satisfies the theorem . We instantiate this construction with Goldreich ’ s PRG with the “ Tri-Sum-And ” ( TSA ) predicate ( Goldreich , 2011 ) , which is an explicit function which is believed to satisfy the local PRG property . We then demonstrate that a neural network discriminator trained via standard methods empirically can not distinguish between the output of this generator and the uniform distribution . While of course we can not guarantee that we achieve the truly optimal discriminator using these methods , this still demonstrates that our construction leads to a function which does appear to be hard to distinguish in practice . GANs , PRGs , and circuit lower bounds . At a high level , our results and techniques demonstrate surprising and deep connections between GANs and more “ classical ” problems in cryptography and complexity theory . Theorem 1.1 already shows that cryptographic assumptions may pose a fundamental barrier to the most basic question in GAN theory . In addition to this , we also show a connection between this question and circuit lower bounds . In the supplementary material , we show that if we are able to unconditionally exhibit generators which can fool polynomially large ReLU network discriminators , then we would obtain breakthrough circuit lower bounds against TC0 ( see Theorem C.2 and Remark C.3 ) . This complements Theorem 1.1 , as it says that if we can unconditionally construct generators which fool realistic discriminators , then we make progress on long-standing questions in circuit complexity . We believe that exploring these connections may be crucial to achieving a deeper understanding of what GANs can and can not learn . 1.1 RELATED WORK . GANs and Distribution Learning The literature on GAN theory is vast and we can not hope to do it full justice here . For a more extensive review , see e.g . ( Gui et al. , 2020 ) . Besides the previously mentioned work on understanding GAN dynamics and generalization , we only mention the most relevant papers here . One closely related line of work derives concrete bounds on when minimax optimality of the GAN objective implies distribution learning ( Bai et al. , 2018 ; Liang , 2018 ; Singh et al. , 2018 ; Uppal et al. , 2019 ; Chen et al. , 2020a ; Schreuder et al. , 2021 ) . However , the rates they achieve scale poorly with the dimensionality of the data , and/or require strong assumptions on the class of generators and discriminators , such as invertability . Another line of work has demonstrated that first order methods can learn very simple GAN architectures in polynomial time ( Feizi et al. , 2017 ; Daskalakis et al. , 2017 ; Gidel et al. , 2019 ; Lei et al. , 2020 ) . However , these results do not cover many of the generators used in practice , such as ReLU networks with > 1 hidden layers . Local PRGs and Learning PRGs have had a rich history of study in cryptography and complexity theory ( see e.g . ( Vadhan , 2012 ) ) . From this literature , the object most relevant to the present work is the notion of a local PRG . These are part of a broader research program of building constant parallel-time cryptography ( Applebaum et al. , 2006 ) . One popular local PRG candidate was suggested in ( Goldreich , 2011 ) . By now there is compelling evidence that this candidate is a valid PRG , as a rich family of algorithms including sum-of-squares ( ODonnell & Witmer , 2014 ) and statistical query algorithms ( Feldman et al. , 2018 ) provably can not break it . Finally , we remark that Goldreich ’ s PRG and , more generally , hardness of refuting random CSPs have been used in a number of works showing hardness for various supervised learning problems ( Daniely & Vardi , 2021 ; Daniely et al. , 2014 ; Daniely , 2016 ; Daniely & Shalev-Shwartz , 2016 ; Applebaum et al. , 2006 ; Applebaum & Raykov , 2016 ) . We consider a very different setting , and our techniques are very different from the aforementioned papers .

Summary: This paper studies the problem of learning generative adversarial networks using a ploy-size ReLU generator and discriminator under the standard Wasserstein-1 metric. The main result is that there exists a "bad" generator that can cheat all discriminators under the estimation of the Wasserstein-1 metric while being far from the data distribution under the true Wasserstein-1 metric. The proof relies on two assumptions. The first one is on the diversity of the target data distribution. The second one is a standard assumption in cryptography as claimed by this paper while I'm not familiar with cryptography. The results explicitly consider the computation complexity of the model and may show the learnability of the GAN model in some sense.