Learning Disconnected Manifolds: Avoiding The No Gan's Land by Latent Rejection

1 INTRODUCTION . GANs ( Goodfellow et al. , 2014 ) are an effective way to learn complex and high-dimensional distributions , leading to state-of-the-art models for image synthesis in both unconditional ( Karras et al. , 2019 ) and conditional settings ( Brock et al. , 2019 ) . However , it is well-known that a single generator with a unimodal latent variable can not recover a distribution composed of disconnected sub-manifolds ( Khayatkhoei et al. , 2018 ) . This leads to a common problem for practitioners : the necessary existence of very-low quality samples when covering different modes . This is formalized by Tanielian et al . ( 2020 ) which refers to this area as the no GAN ’ s land and provides impossibility theorems on the learning of disconnected manifolds with standard formulations of GANs . Fitting a disconnected target distribution requires an additional mechanism inserting disconnectedness in the modeled distribution . A first solution is to add some expressivity to the model : Khayatkhoei et al . ( 2018 ) propose to train a mixture of generators while Gurumurthy et al . ( 2017 ) make use of a multi-modal latent distribution . A second solution is to improve the quality of a trained generative model by avoiding its poorest samples ( Tao et al. , 2018 ; Azadi et al. , 2019 ; Turner et al. , 2019 ; Grover et al. , 2019 ; Tanaka , 2019 ) . This second line of research relies heavily on a variety of Monte-Carlo algorithms , such as Rejection Sampling or the Metropolis-Hastings . These methods aim at sampling from a target distribution , while having only access to samples generated from a proposal distribution . This idea was successfully applied to GANs , using the previously learned generative distribution as a proposal distribution . However , one of the main drawback is that Monte-Carlo algorithms only guarantee to sample from the target distribution under strong assumptions . First , we need access to the density ratios between the proposal and target distributions or equivalently to a perfect discriminator ( Azadi et al. , 2019 ) . Second , the support of the proposal distribution must fully cover the one of the target distribution , which means no mode collapse . This is known to be very demanding in high dimension since the intersection of supports between the proposal and target distribution is likely to be negligible ( Arjovsky and Bottou , 2017 , Lemma 3 ) . In this setting , an optimal discriminator would give null acceptance probabilities for almost any generated points , leading to a lower performance . To tackle the aforementioned issue , we propose a novel method aiming at reducing the Wasserstein distance between the previously trained generative model and the target distribution . This is done via the adversarial training of a third network that learns importance weights in the latent space . The goal is to learn the redistribution of mass of the modeled distribution that best fits the target distribution . To better understand our approach , we first consider a simple 2D motivational example where the real data lies on four disconnected manifolds . To approximate this , the generator splits the latent space into four distinct areas and maps data points located in the frontiers , areas in orange in Figure 1b , out of the true manifold ( see Figure 1a ) . Our method consequently aims at learning latent importance weights that can identify these frontiers and simply avoid them . This is highlighted in Figure 1d where the importance weighter has identified these four frontiers . When sampling from the new latent distribution , we can now perfectly fit the mixture of four gaussians ( see Figure 1c ) . Our contributions are the following : • We discuss works improving the sampling quality of GANs and identify their limitations . • We propose a novel approach that directly modifies the latent space distribution . It provides a principled way to reduce the Wasserstein distance to the target distribution . • We thorougly compare our method with a large set of previous approaches on a variety of datasets and distributions . We empirically show that our solution significantly reduces the computational cost of inference while demonstrating an equal efficiency . Notation . Before moving to the related work section , we shortly present notation needed in the paper . The goal of the generator is to generate data points that are “ similar ” to samples collected from some target probability measure µ ? . The measure µ ? is defined on a potentially high dimensional spaceRD , equipped with the euclidean norm ‖ · ‖ . To approach µ ? , we use a parametric family of generative distribution where each distribution is the push-forward measure of a latent distribution Z and a continuous function modeled by a neural network . In most of all practical applications , the random variable Z defined on a low dimensional space Rd is either a multivariate Gaussian distribution or uniform distribution . The generator is a parameterized class of functions from Rd to RD , say G = { Gθ : θ ∈Θ } , where Θ ⊆ Rp is the set of parameters describing the model . Each function Gθ takes input from Z and outputs “ fake ” observations with distribution µθ = Gθ ] Z . On the other hand , the discriminator is described by a family of functions from RD to R , say D = { Dα : α ∈Λ } , Λ ⊆RQ , where each Dα . Finally , for any given distribution µ , we note Sµ its support . 2 RELATED WORK . 2.1 DISCONNECTED MANIFOLD LEARNING : HOW TO TRAIN AND EVALUATE GANS . Goodfellow et al . ( 2014 ) already stated that when training vanilla GANs , the generator could ignore modes of the target distribution : this is the mode collapse . A significant step towards understanding this phenomenon was made by Arjovsky and Bottou ( 2017 ) who explained that the standard formulation of GANs leads to vanishing or unstable gradients . The authors proposed the Wasserstein GANs ( WGANs ) architecture ( Arjovsky et al. , 2017 ) where , in particular , discriminative functions are restricted to the class of 1-Lipschitz functions . WGANs aim at solving : sup α∈A inf θ∈Θ Ex∼µ ? Dα ( x ) −Ez∼Z Dα ( Gθ ( z ) ) ) ( 1 ) The broader drawback of standard GANs is that , since any modeled distribution is the push-forward of a unimodal distribution by a continuous transformation , it consequently has a connected support . This means that when the generator covers multiple disconnected modes of the target distribution , it necessarily generates samples out of the real data manifold ( Khayatkhoei et al. , 2018 ) . Consequently , any thorough evaluation of GANs should assess simultaneously both the quality and the variety of the generated samples . Sajjadi et al . ( 2018 ) argue that a single-digit metric such as the Inception Score ( Salimans et al. , 2016 ) or the Frechet Inception distance ( Heusel et al. , 2017 ) is thus not adequate to compare generative models . To solve this issue , the authors propose a Precision/Recall metric that aims at measuring both the mode dropping and the mode inventing . In the Improved Precision/Recall ( Kynkäänniemi et al. , 2019 ) , the precision refers to the portion of generated points that belongs to the target manifold , while the recall measures how much of the target distribution can be re-constructed by the model distribution . Building on this metric , Tanielian et al . ( 2020 ) highlighted the trade-off property of GANs deriving upper-bounds on the precision of standard GANs . To solve this problem , a common direction of research consists in over-parameterizing the generative model . Khayatkhoei et al . ( 2018 ) enforce diversity by using a mixture of generators while Gurumurthy et al . ( 2017 ) suggest that a mixture of Gaussians in the latent space is efficient to learn diverse and limited data . 2.2 IMPROVING THE QUALITY OF TRAINED GENERATORS . To better fit disconnected manifolds with standard GANs architectures , another line of research consists in inserting disconnectedness into a previously learned generative distribution µθ . Tanielian et al . ( 2020 ) proposed an heuristic to remove the no GAN ’ s land ( i.e . samples mapped out of the true manifold ) : rejecting data points with a high Jacobian Frobenius norm . Another possibility would be to use one of the different Monte-Carlo methods ( Robert and Casella , 2013 ) and apply it to GANs . Building up on the well-known inference theory , Azadi et al . ( 2019 ) suggests the use of rejection sampling to improve the quality of the proposal distribution µθ . One can compute density ratios using either a classifier trained from scratch or the discriminator obtained at the end of the training . Consequently , in this Discriminator Rejection Sampling ( DRS ) , any generated data point x∼ µθ is accepted with the following acceptance probability Pa : Pa ( x ) = µ ? ( x ) Mµθ ( x ) where M = max x∈Sµθ µ ? ( x ) µθ ( x ) , ( 2 ) where µ ? and µθ here refers to the density functions . Similarly , Turner et al . ( 2019 ) use the same density ratios and derive MH-GAN , an adaptation of the Metropolis-Hasting algorithm ( Hastings , 1970 ) , that improves the sampling from µθ . Finally , Grover et al . ( 2019 ) use these density ratios r as importance weights and define an importance resampled generative model whose density is now defined by µ̂θ ∝ µθ × r ( x ) . In order to perform discrete sampling from µ̂θ , authors rely on the Sampling-Importance-Resampling ( SIR ) algorithm ( Rubin , 1988 ; Liu and Chen , 1998 ) . This defines a new distribution µ̂θ SIR : µ̂SIRθ ( xi ) = r ( xi ) n ∑ j=1 r ( x j ) where x1 , . . . , xn ∼ µnθ . Note that these algorithms rely on the same density ratios and an acceptance-rejection scheme . In Rejection Sampling , the acceptance rate is uncontrollable but sampling from µ ? is assured . With SIR and MH , the acceptance rate is controllable but sampling from µ ? is no longer guaranteed . 3 ADVERSARIAL LEARNING OF LATENT IMPORTANCE WEIGHTS . 3.1 OUR APPROACH . Similar to previous works , our method consists in improving the performance of a given generative model , post-training . Given a trained WGANs ( Gθ , Dα ) , we now propose to learn importance weights in the latent space . To do so , we use a feed-forward neural network from Rd to R+ , say Ω = { wϕ : ϕ ∈ Φ } . The neural network wϕ is trained using an adversarial process with the discriminator Dα , whilst keeping the weights of Gθ frozen . We now want to solve the following : sup α∈A inf ϕ∈Φ Ex∼µ ? Dα ( x ) −Ez∼Z ( wϕ ( z ) ×Dα ( Gθ ( z ) ) ) ) ( 3 ) Note that our formulation can also be plugged on top of many different objective functions . Interestingly , the use of the predictor wϕ defines a new latent space distribution whose density γ̂ is defined by γ̂ ( z ) ∝ wϕ ( z ) × γ ( z ) . Consequently , the newly defined modeled distribution µ̂θ is defined as the pushforward µ̂θ = Gθ ] γ̂ . The proposed method can be seen as minimizing the Wasserstein distance to the target distribution , over an increased class of generative distributions . The network wϕ thus learns how to redistribute the mass of µθ such that µ̂θ is closer to µ ? in terms Wasserstein distance . However , as in the field of counterfactual estimation , a naive optimization of importance weights by gradient descent can lead to trivial solutions . First , if for example , the Wasserstein critic Dα outputs negative values for any generated samples , the network wϕ could simply learn to avoid the dataset and output 0 everywhere . To avoid this issue , we follow Swaminathan and Joachims ( 2015c ) and scale the output of the discriminator such that the reward is always positive . A second problem comes from the fact that equation 3 can now be minimized not only by putting large importance weights wϕ ( z ) on the examples with high likelihoods Dα ( G ( z ) ) , but also by maximizing the sum of the weights : this is the propensity overfitting ( Swaminathan and Joachims , 2015a ) . To stabilize the optimisation process , we consequently introduce two important regularization techniques : Self-normalization . Similarly to Swaminathan and Joachims ( 2015a ) , we advocate the use of a normalization of the importance weights . To be more precise , we enforce the expectation of the importance weights to be close 1 by adding a penalty term . By doing so , we prohibit the propensity overfitting since the sum of the importance weights in the batch is bounded . Soft-Clipping To avoid cases where small areas of z have really high wϕ ( z ) values , which would lead to mode collapse , we enforce a soft-clipping on the weights ( Bottou et al. , 2013 ; Grover et al. , 2019 ) . Note that this constraint on wϕ ( z ) could also be implemented with a bounded activation function on the final layer , such as a re-scaled sigmoid or tanh activation . Finally , we thus get the following objective function : sup ϕ∈Φ Ez∼Z wϕ ( z ) ( Dα ( Gθ ( z ) ) ) −∇ ) ︸ ︷︷ ︸ discriminator reward −λ1 ( Ez∼Zwϕ ( z ) −1 ) 2︸ ︷︷ ︸ self-normalization −λ2Ez∼Z max ( 0 , ( wϕ ( z ) −m ) ) 2︸ ︷︷ ︸ soft-clipping , ( 4 ) where ∇ = minz∼Z Dα ( G ( z ) ) . λ1 , λ2 , and m are hyper-parameters ( values displayed in Appendix ) .

Summary: The paper proposes a method for an improvement of generative adversarial models via post-processing its latent variable distribution. To be more precise, the method proposes to train an additional neural network that outputs an important weight for each point of the latent space, thus reweighting the final distribution in the space of images. For the optimization of this network, the authors use the dual form of the Wasserstein distance, where they multiply the initial latent density by the output of the network. To fix the ill-behaved objective, the authors add two regularization terms to it. The proposed objective is then validated on 3 MNIST-like datasets quantitatively and on CelebA qualitatively.