Approximability of Discriminators Implies Diversity in GANs

1 INTRODUCTION . In the past few years , we have witnessed great empirical success of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) in generating high-quality samples in many domains . Various ideas have been proposed to further improve the quality of the learned distributions and the stability of the training . ( See e.g. , ( Arjovsky et al. , 2017 ; Odena et al. , 2016 ; Huang et al. , 2017 ; Radford et al. , 2016 ; Tolstikhin et al. , 2017 ; Salimans et al. , 2016 ; Jiwoong Im et al. , 2016 ; Durugkar et al. , 2016 ; Xu et al. , 2017 ) and the reference therein . ) However , understanding of GANs is still in its infancy . Do GANs actually learn the target distribution ? Recent work ( Arora et al. , 2017a ; b ; Dumoulin et al. , 2016 ) has both theoretically and empirically brought the concern to light that distributions learned by GANs suffer from mode collapse or lack of diversity — the learned distribution tends to miss a significant amount of modes of the target distribution ( elaborated in Section 1.1 ) . The main message of this paper is that the mode collapse can be in principle alleviated by designing proper discriminators with strong distinguishing power against specific families of generators such as special subclasses of neural network generators ( see Section 1.2 and 1.3 for a detailed introduction . ) 1.1 BACKGROUND ON MODE COLLAPSE IN GANS . We mostly focus on the Wasserstein GAN ( WGAN ) formulation ( Arjovsky et al. , 2017 ) in this paper . Define the F-Integral Probability Metric ( F-IPM ) ( Müller , 1997 ) between distributions p , q as WF ( p , q ) : = sup f∈F ∣∣EX∼p [ f ( X ) ] − EX∼q [ f ( X ) ] ∣∣ . ( 1 ) Given samples from distribution p , WGAN sets up a family of generators G , a family of discriminators F , and aims to learn the data distribution p by solving min q∈G WF ( p̂ n , q̂m ) ( 2 ) where p̂n denotes “ the empirical version of the distribution p ” , meaning the uniform distribution over a set of n i.i.d samples from p ( and similarly q̂m . ) When F = { all 1-Lipschitz functions } , IPM reduces to the Wasserstein-1 distance W1 . In practice , parametric families of functions F such as multi-layer neural networks are used for approximating Lipschitz functions , so that we can empirically optimize this objective eq . ( 2 ) via gradient-based algorithms as long as distributions in the family G have parameterized samplers . ( See Section 2 for more details . ) One of the main theoretical and empirical concerns with GANs is the issue of “ modecollapse ” ( Arora et al. , 2017a ; Salimans et al. , 2016 ) — the learned distribution q tends to generate high-quality but low-diversity examples . Mathematically , the problem apparently arises from the fact that IPM is weaker than W1 , and the mode-dropped distribution can fool the former ( Arora et al. , 2017a ) : for a typical distribution p , there exists a distribution q such that simultaneously the followings happen : WF ( p , q ) . ε and W1 ( p , q ) & 1 . ( 3 ) where . , & hide constant factors . In fact , setting q = p̂N with N = R ( F ) /ε2 , where R ( F ) is a complexity measure of F ( such as Rademacher complexity ) , q satisfies eq . ( 3 ) but is clearly a mode-dropped version of p when p has an exponential number of modes . Reasoning that the problem is with the strength of the discriminator , a natural solution is to increase it to larger families such as all 1-Lipschitz functions . However , Arora et al . ( Arora et al. , 2017a ) points out that Wasserstein-1 distance doesn ’ t have good generalization properties : the empirical Wasserstein distance used in the optimization is very far from the population distance . Even for a spherical Gaussian distribution p = N ( 0 , 1dId×d ) ( or many other typical distributions ) , when the distribution q is exactly equal to p , letting q̂m and p̂n be two empirical versions of q and p with m , n = poly ( d ) , we have with high probability , W1 ( p̂ n , q̂m ) & 1 even though W1 ( p , q ) = 0 . ( 4 ) Therefore even when learning succeeds ( p = q ) , it can not be gleaned from the empirical version of W1 . The observations above pose a dilemma in establishing the theories of GANs : powerful discriminators cause overfitting , whereas weak discriminators result in diversity issues because IPM doesn ’ t approximate the Wasserstein distance The lack of diversity has also been observed empirically by ( Srivastava et al. , 2017 ; Di & Yu , 2017 ; Borji , 2018 ; Arora et al. , 2017b ) . 1.2 AN APPROACH TO DIVERSITY : DISCRIMINATOR FAMILIES WITH RESTRICTED APPROXIMABILITY . This paper proposes a resolution to the conundrum by designing a discriminator class F that is particularly strong against a specific generator class G. We say that a discriminator class F ( and its IPM WF ) has restricted approximability w.r.t . a generator class G and the data distribution p , if F can distinguish p and any q ∈ G approximately as well as all 1-Lipschitz functions can do : WF has restricted approximability w.r.t . G and p , ∀q ∈ G , γL ( W1 ( p , q ) ) .WF ( p , q ) . γU ( W1 ( p , q ) ) , ( 5 ) where γL ( · ) and γU ( · ) are two monotone nonnegative functions with γL ( 0 ) = γU ( 0 ) = 0 . The paper mostly focuses on γL ( t ) = tα with 1 ≤ α ≤ 2 and γU ( t ) = t , although we use the term “ restricted approximability ” more generally for this type of result ( without tying it to a concrete definition of γ ) . In other words , we are looking for discriminatorsF so thatF-IPM can approximate the Wasserstein distance W1 for the data distribution p and any q ∈ G. Throughout the rest of this paper , we will focus on the realizable case , that is , we assume p ∈ G , in which case we say WF has restricted approximability with respect to G if eq . ( 5 ) holds for all p , q ∈ G. We note , however , that such a framework allows the non-realizible case p /∈ G in full generality ( for example , results can be established through designingF that satisfies the requirement in Lemma 4.3 ) . A discriminator class F with restricted approximability resolves the dilemma in the following way . First , F avoids mode collapse – if the IPM between p and q is small , then by the left hand side of eq . ( 5 ) , p and q are also close in Wasserstein distance and therefore significant mode-dropping can not happen . 1 Second , we can pass from population-level guarantees to empirical-level guarantees – as shown in Arora et al . ( 2017a ) , classical capacity bounds such as the Rademacher complexity of F relate WF ( p , q ) to WF ( p̂n , q̂m ) . Therefore , as long as the capacity is bounded , we can expand on eq . ( 5 ) to get a full picture of the statistical properties of Wasserstein GANs : ∀q ∈ G , γL ( W1 ( p , q ) ) .WF ( p , q ) ≈WF ( p̂n , q̂m ) . γU ( W1 ( p , q ) ) . Here the first inequality addresses the diversity property of the distance WF , and the second approximation addresses the generalization of the distance , and the third inequality provides the reverse guarantee that if the training fails to find a solution with small IPM , then indeed p and q are far away in Wasserstein distance.2 To the best of our knowledge , this is the first theoretical framework that tackles the statistical theory of GANs with polynomial samples . The main body of the paper will develop techniques for designing discriminator class F with restricted approximability for several examples of generator classes including simple classes like mixtures of Gaussians , exponential families , and more complicated classes like distributions generated by invertible neural networks . In the next subsection , we will show that properly chosen F provides diversity guarantees such as inequalities eq . ( 5 ) . 1.3 DESIGN OF DISCRIMINATORS WITH RESTRICTED APPROXIMABILITY . We start with relatively simple families of distributions G such as Gaussian distributions and exponential families , where we can directly design F to distinguish pairs of distribution in G. As we show in Section 3 , for Gaussians it suffices to use one-layer neural networks with ReLU activations as discriminators , and for exponential families to use linear combinations of the sufficient statistics . In Section 4 , we study the family of distributions generated by invertible neural networks . We show that a special type of neural network discriminators with one additional layer than the generator has restricted approximability3 . We show this discriminator class guarantees that W1 ( p , q ) 2 . WF ( p , q ) . W1 ( p , q ) where here we hide polynomial dependencies on relevant parameters ( Theorem 4.2 ) . We remark that such networks can also produce an exponentially large number of modes due to the non-linearities , and our results imply that if WF ( p , q ) is small , then most of these exponential modes will show up in the learned distribution q . One limitation of the invertibility assumption is that it only produces distributions supported on the entire space . The distribution of natural images is often believed to reside approximately on a low-dimensional manifold . When the distribution p have a Lebesgue measure-zero support , the KLdivergence ( or the reverse KL-divergence ) is infinity unless the support of the estimated distribution coincides with the support of p.4 Therefore , while our proof makes crucial use of the KL-divergence in the invertible case , the KL-divergence is fundamentally not the proper measurement of the statistical distance for the cases where both p and q have low-dimensional supports . The crux of the technical part of the paper is to establish the approximation of Waserstein distance by IPMs for generators with low-dimensional supports . We will show that a variant of an IPM can still be sandwiched by Wasserstein distance as in form of eq . ( 5 ) without relating to KL-divergence ( Theorem 4.5 ) . This demonstrates the advantage of GANs over MLE approach on learning distributions 1Informally , if most of the modes of p are ε-far away from each other , then as long as W1 ( p , q ) ε , q has to contain most of the modes of p. 2We also note that the third inequality can hold for all p , q as long as F is a subset of Lipschitz functions . 3This is consistent with the empirical finding that generators and discriminators with similar depths are often near-optimal choices of architectures . 4The formal mathematical statement is that Dkl ( p‖q ) is infinity unless p is absolutely continuous with respect to q. with low-dimensional supports . As the main proof technique , we develop tools for approximating the log-density of a smoothed neural network generator . We demonstrate in synthetic and controlled experiments that the IPM correlates with the Wasserstein distance for low-dimensional distributions with measure-zero support and correlates with KLdivergence for the invertible generator family ( where computation of KL is feasible ) ( Section 5 and Appendix G. ) The theory suggests the possibility that when the KL-divergence or Wasserstein distance is not measurable in more complicated settings , the test IPM could serve as a candidate alternative for measuring the diversity and quality of the learned distribution . We also remark that on real datasets , often the optimizer is tuned to carefully balance the learning of generators and discriminators , and therefore the reported training loss is often not the test IPM ( which requires optimizing the discriminator until optimality . ) Anecdotally , the distributions learned by GANs can often be distinguished by a well-trained discriminator from the data distribution , which suggests that the IPM is not well-optimized ( See Lopez-Paz & Oquab ( 2016 ) for analysis of for the original GANs formulation . ) We conjecture that the lack of diversity in real experiments may be caused by sub-optimality of the optimization , rather than statistical inefficiency .

Summary: This paper analyzes that the Integral Probability Metric (IPM) can be a good approximation of Wasserstein distance under some mild assumptions. They first showed two theorems based on simple cases (Gaussian Distribution and Exponential Families). Then, they proved that, for an invertible generator, a special designed neural network can approximate Wasserstein distance with IPM. The main contribution is that, for a stable generator (i.e., invertible generator), a discriminator can reversely “re-visit” inner status of the generator, then use this information to make a decision.