Approximability of Discriminators Implies Diversity in GANs
While Generative Adversarial Networks have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse.The theoretical work of Arora et al. suggests a dilemma about GANs’ statistical properties: powerful discriminators cause overfitting, whereas weak discriminators cannot detect mode collapse.By contrast, we show in this paper that GANs can in principle learn distributions in Wasserstein distance with polynomial sample complexity, if the discriminator class has strong distinguishing power against the particular generator class.For various generator classes such as mixture of Gaussians, exponential families, and invertible and injective neural networks generators, we design corresponding discriminators such that the Integral Probability Metric induced by the discriminators can provably approximate the Wasserstein distance and/or KL-divergence.This implies that if the training is successful, then the learned distribution is close to the true distribution in Wasserstein distance or KL divergence, and thus cannot drop modes.Our preliminary experiments show that on synthetic datasets the test IPM is well correlated with KL divergence or the Wasserstein distance, indicating that the lack of diversity in GANs may be caused by the sub-optimality in optimization instead of statistical inefficiency.
GANs can in principle learn distributions sample-efficiently, if the discriminator class is compact and has strong distinguishing power against the particular generator class.Proposes the notion of restricted approximability, and provides a sample complexity bound, polynomial in the dimension, which is useful in investigating lack of diversity in GANs.Analyzes that the Integral Probability Metric can be a good approximation of Wasserstein distance under some mild assumptions.