LEARNING DISCONNECTED MANIFOLDS: AVOIDING THE NO GAN’S LAND BY LATENT REJECTION
Year: 2020

Abstract:
Standard formulations of GANs, where a continuous function deforms a connected latent space, have been shown to be misspecified when fitting disconnected manifolds. In particular, when covering different classes of images, the generator will necessarily sample some low quality images in between the modes. Rather than modify the learning procedure, a line of works aims at improving the sampling quality from trained generators. Thus, it is now common to introduce a rejection step within the generation procedure. Building on this, we propose to train an additional network and transform the latent space via an adversarial learning of importance weights. This idea has several advantages: 1) it provides a way to inject disconnectedness on any GAN architecture, 2) since the rejection happens in the latent space, it avoids going through both the generator and the discriminator saving computation time, 3) this importance weights formulation provides a principled way to reduce the Wasserstein’s distance to the target distribution. We demonstrate the effectiveness of our method on different datasets, both synthetic and high dimensional.

heading: 1 INTRODUCTION
text: GANs (Goodfellow et al., 2014) are an effective way to learn complex and high-dimensional distributions, leading to state-of-the-art models for image synthesis in both unconditional (Karras et al., 2019) and conditional settings (Brock et al., 2019). However, it is well-known that a single generator with a unimodal latent variable cannot recover a distribution composed of disconnected sub-manifolds (Khayatkhoei et al., 2018). This leads to a common problem for practitioners: the necessary existence of very-low quality samples when covering different modes. This is formalized by Tanielian et al. (2020) which refers to this area as the no GAN’s land and provides impossibility theorems on the learning of disconnected manifolds with standard formulations of GANs. Fitting a disconnected target distribution requires an additional mechanism inserting disconnectedness in the modeled distribution. A first solution is to add some expressivity to the model: Khayatkhoei et al. (2018) propose to train a mixture of generators while Gurumurthy et al. (2017) make use of a multi-modal latent distribution. A second solution is to improve the quality of a trained generative model by avoiding its poorest samples (Tao et al., 2018; Azadi et al., 2019; Turner et al., 2019; Grover et al., 2019; Tanaka, 2019).
This second line of research relies heavily on a variety of Monte-Carlo algorithms, such as Rejection Sampling or the Metropolis-Hastings. These methods aim at sampling from a target distribution, while having only access to samples generated from a proposal distribution. This idea was successfully applied to GANs, using the previously learned generative distribution as a proposal distribution. However, one of the main drawback is that Monte-Carlo algorithms only guarantee to sample from the target distribution under strong assumptions. First, we need access to the density ratios between the proposal and target distributions or equivalently to a perfect discriminator (Azadi et al., 2019). Second, the support of the proposal distribution must fully cover the one of the target distribution, which means no mode collapse. This is known to be very demanding in high dimension since the intersection of supports between the proposal and target distribution is likely to be negligible (Arjovsky and Bottou, 2017, Lemma 3). In this setting, an optimal discriminator would give null acceptance probabilities for almost any generated points, leading to a lower performance.
To tackle the aforementioned issue, we propose a novel method aiming at reducing the Wasserstein distance between the previously trained generative model and the target distribution. This is done via
the adversarial training of a third network that learns importance weights in the latent space. The goal is to learn the redistribution of mass of the modeled distribution that best fits the target distribution. To better understand our approach, we first consider a simple 2D motivational example where the real data lies on four disconnected manifolds. To approximate this, the generator splits the latent space into four distinct areas and maps data points located in the frontiers, areas in orange in Figure 1b, out of the true manifold (see Figure 1a). Our method consequently aims at learning latent importance weights that can identify these frontiers and simply avoid them. This is highlighted in Figure 1d where the importance weighter has identified these four frontiers. When sampling from the new latent distribution, we can now perfectly fit the mixture of four gaussians (see Figure 1c).
Our contributions are the following:
• We discuss works improving the sampling quality of GANs and identify their limitations. • We propose a novel approach that directly modifies the latent space distribution. It provides
a principled way to reduce the Wasserstein distance to the target distribution. • We thorougly compare our method with a large set of previous approaches on a variety of
datasets and distributions. We empirically show that our solution significantly reduces the computational cost of inference while demonstrating an equal efficiency.
Notation. Before moving to the related work section, we shortly present notation needed in the paper. The goal of the generator is to generate data points that are “similar” to samples collected from some target probability measure µ?. The measure µ? is defined on a potentially high dimensional spaceRD, equipped with the euclidean norm ‖ · ‖. To approach µ?, we use a parametric family of generative distribution where each distribution is the push-forward measure of a latent distribution Z and a continuous function modeled by a neural network. In most of all practical applications, the random variable Z defined on a low dimensional space Rd is either a multivariate Gaussian distribution or uniform distribution. The generator is a parameterized class of functions from Rd to RD, say G = {Gθ : θ ∈Θ}, where Θ ⊆ Rp is the set of parameters describing the model. Each function Gθ takes input from Z and outputs “fake” observations with distribution µθ = Gθ ]Z. On the other hand, the discriminator is described by a family of functions from RD to R, say D = {Dα : α ∈Λ}, Λ ⊆RQ, where each Dα . Finally, for any given distribution µ , we note Sµ its support.

heading: 2 RELATED WORK

heading: 2.1 DISCONNECTED MANIFOLD LEARNING: HOW TO TRAIN AND EVALUATE GANS
text: Goodfellow et al. (2014) already stated that when training vanilla GANs, the generator could ignore modes of the target distribution: this is the mode collapse. A significant step towards understanding this phenomenon was made by Arjovsky and Bottou (2017) who explained that the standard formulation of GANs leads to vanishing or unstable gradients. The authors proposed the Wasserstein GANs (WGANs) architecture (Arjovsky et al., 2017) where, in particular, discriminative functions are restricted to the class of 1-Lipschitz functions. WGANs aim at solving:
sup α∈A inf θ∈Θ Ex∼µ? Dα(x)−Ez∼Z Dα(Gθ (z))) (1)
The broader drawback of standard GANs is that, since any modeled distribution is the push-forward of a unimodal distribution by a continuous transformation, it consequently has a connected support. This means that when the generator covers multiple disconnected modes of the target distribution, it necessarily generates samples out of the real data manifold (Khayatkhoei et al., 2018). Consequently, any thorough evaluation of GANs should assess simultaneously both the quality and the variety of the generated samples. Sajjadi et al. (2018) argue that a single-digit metric such as the Inception Score (Salimans et al., 2016) or the Frechet Inception distance (Heusel et al., 2017) is thus not adequate to compare generative models. To solve this issue, the authors propose a Precision/Recall metric that aims at measuring both the mode dropping and the mode inventing.
In the Improved Precision/Recall (Kynkäänniemi et al., 2019), the precision refers to the portion of generated points that belongs to the target manifold, while the recall measures how much of the target distribution can be re-constructed by the model distribution. Building on this metric, Tanielian et al. (2020) highlighted the trade-off property of GANs deriving upper-bounds on the precision of standard GANs. To solve this problem, a common direction of research consists in over-parameterizing the generative model. Khayatkhoei et al. (2018) enforce diversity by using a mixture of generators while Gurumurthy et al. (2017) suggest that a mixture of Gaussians in the latent space is efficient to learn diverse and limited data.

heading: 2.2 IMPROVING THE QUALITY OF TRAINED GENERATORS
text: To better fit disconnected manifolds with standard GANs architectures, another line of research consists in inserting disconnectedness into a previously learned generative distribution µθ . Tanielian et al. (2020) proposed an heuristic to remove the no GAN’s land (i.e. samples mapped out of the true manifold): rejecting data points with a high Jacobian Frobenius norm. Another possibility would be to use one of the different Monte-Carlo methods (Robert and Casella, 2013) and apply it to GANs. Building up on the well-known inference theory, Azadi et al. (2019) suggests the use of rejection sampling to improve the quality of the proposal distribution µθ . One can compute density ratios using either a classifier trained from scratch or the discriminator obtained at the end of the training. Consequently, in this Discriminator Rejection Sampling (DRS), any generated data point x∼ µθ is accepted with the following acceptance probability Pa:
Pa(x) = µ?(x)
Mµθ (x) where M = max x∈Sµθ µ?(x) µθ (x) , (2)
where µ? and µθ here refers to the density functions. Similarly, Turner et al. (2019) use the same density ratios and derive MH-GAN, an adaptation of the Metropolis-Hasting algorithm (Hastings, 1970), that improves the sampling from µθ . Finally, Grover et al. (2019) use these density ratios r as importance weights and define an importance resampled generative model whose density is now defined by µ̂θ ∝ µθ × r(x). In order to perform discrete sampling from µ̂θ , authors rely on the Sampling-Importance-Resampling (SIR) algorithm (Rubin, 1988; Liu and Chen, 1998). This defines a new distribution µ̂θ SIR:
µ̂SIRθ (xi) = r(xi)
n ∑ j=1
r(x j) where x1, . . . ,xn ∼ µnθ .
Note that these algorithms rely on the same density ratios and an acceptance-rejection scheme. In Rejection Sampling, the acceptance rate is uncontrollable but sampling from µ? is assured. With SIR and MH, the acceptance rate is controllable but sampling from µ? is no longer guaranteed.

heading: 3 ADVERSARIAL LEARNING OF LATENT IMPORTANCE WEIGHTS

heading: 3.1 OUR APPROACH
text: Similar to previous works, our method consists in improving the performance of a given generative model, post-training. Given a trained WGANs (Gθ ,Dα), we now propose to learn importance weights in the latent space. To do so, we use a feed-forward neural network from Rd to R+, say Ω = {wϕ : ϕ ∈ Φ}. The neural network wϕ is trained using an adversarial process with the discriminator Dα , whilst keeping the weights of Gθ frozen. We now want to solve the following:
sup α∈A inf ϕ∈Φ
Ex∼µ?Dα(x)−Ez∼Z ( wϕ(z)×Dα(Gθ (z))) ) (3)
Note that our formulation can also be plugged on top of many different objective functions. Interestingly, the use of the predictor wϕ defines a new latent space distribution whose density γ̂ is defined by γ̂(z) ∝ wϕ(z)× γ(z). Consequently, the newly defined modeled distribution µ̂θ is defined as the pushforward µ̂θ = Gθ ]γ̂ . The proposed method can be seen as minimizing the Wasserstein distance to the target distribution, over an increased class of generative distributions. The network wϕ thus learns how to redistribute the mass of µθ such that µ̂θ is closer to µ? in terms Wasserstein distance.
However, as in the field of counterfactual estimation, a naive optimization of importance weights by gradient descent can lead to trivial solutions. First, if for example, the Wasserstein critic Dα outputs negative values for any generated samples, the network wϕ could simply learn to avoid the dataset and output 0 everywhere. To avoid this issue, we follow Swaminathan and Joachims (2015c) and scale the output of the discriminator such that the reward is always positive. A second problem comes from the fact that equation 3 can now be minimized not only by putting large importance weights wϕ(z) on the examples with high likelihoods Dα(G(z)), but also by maximizing the sum of the weights: this is the propensity overfitting (Swaminathan and Joachims, 2015a). To stabilize the optimisation process, we consequently introduce two important regularization techniques:
Self-normalization. Similarly to Swaminathan and Joachims (2015a), we advocate the use of a normalization of the importance weights. To be more precise, we enforce the expectation of the importance weights to be close 1 by adding a penalty term. By doing so, we prohibit the propensity overfitting since the sum of the importance weights in the batch is bounded.
Soft-Clipping To avoid cases where small areas of z have really high wϕ(z) values, which would lead to mode collapse, we enforce a soft-clipping on the weights (Bottou et al., 2013; Grover et al., 2019). Note that this constraint on wϕ(z) could also be implemented with a bounded activation function on the final layer, such as a re-scaled sigmoid or tanh activation.
Finally, we thus get the following objective function:
sup ϕ∈Φ
Ez∼Z wϕ(z) ( Dα(Gθ (z)))−∇ )︸ ︷︷ ︸ discriminator reward −λ1 ( Ez∼Zwϕ(z)−1 )2︸ ︷︷ ︸ self-normalization −λ2Ez∼Z max ( 0,(wϕ(z)−m) )2︸ ︷︷ ︸ soft-clipping , (4)
where ∇ = minz∼Z Dα(G(z)). λ1, λ2, and m are hyper-parameters (values displayed in Appendix).

heading: 3.2 SAMPLING FROM THE NEW DISTRIBUTION
text: As mentionned above, the scale and variance of the learned importance weights are actively controlled, as it is done in counterfactual estimation (Bottou et al., 2013; Swaminathan and Joachims, 2015b; Faury et al., 2020). Doing so, we explicitly control the acceptance rates Pa(z) of the rejection sampling algorithm performed on γ̂ , since for any given z∼ Z, we have:
Pa(z) = γ̂(z)
mγ(z) and EZ Pa(z) = ∫ Rd γ̂(z) mγ(z) γ(z)dz = ∫ Rd γ̂(z) m dz = 1 m ,
where m is the maximum output of the importance weighter as defined in equation 4. We define as Latent Rejection Sampling (latentRS), the method that performs the Rejection Sampling algorithm on top of the learned importance weights. Since the exact sampling of the distribution γ̂ is now tractable with a rejection sampling algorithm, we do not need to implement neither the Metropolis-Hasting nor the Sampling Importance Resampling algorithm.
Inspired from the literature on latent space optimal transport (Salimans et al., 2018; Agustsson et al., 2019; Tanaka, 2019), we also propose a second method where we perform gradient ascent in the latent space. To be more precise, for any given sample in the latent space, we follow the path maximizing the learned importance weights. This method is denoted latent Gradient Ascent (latentGA). In high-dimension, similarly to Tanaka (2019, Algorithm 2), gradients are projected to restrict z on the training support. Note that the learning rate and the number of updates used for this method are hyper-parameter that need to be tuned.

heading: 3.3 ADVANTAGES OF THE PROPOSED APPROACH
text: We now discuss, in detail, the flaws of previous Monte-Carlo based approaches:
1) Computational cost. By using sampling algorithms in the latent space, we avoid going through both the generator and the discriminator, leading to a significant computational speed-up. This is of particular interest when dealing with high-dimensional spaces since we do not need to pass through deep CNNs generator and discriminator (Brock et al., 2019).
2) Existence of density functions. Every Monte-Carlo based methods assume that both µ? and µθ are probability distributions with associated density functions. However, in high dimension, the hypothesis that data tend to lie near a low dimensional manifold (Fefferman et al., 2016) is now commonly accepted. Besides, it is often the case that GANs be defined as the push-forward from much lower dimensional space, that is d << D. In that case, neither µ? nor µθ have density function in RD. Note that our method based on Wasserstein distance does not require this assumption.
3) Covering of the support Sµ? . First, Monte-Carlo methods are well-known to suffer from the curse of dimensionality (Mengersen et al., 1996; Robert and Casella, 2013). Besides, in the context of GANs, Arjovsky and Bottou (2017, Theorem 2.2) have shown that the intersection Sµ? ⋂ Sµθ is likely to be a negligible set under µθ . In this specific case, the density ratios would evaluate close to 0 almost everywhere on Sµθ , increasing the time complexity. More generally, Monte-Carlo based methods tend to avoid any area within Sµθ \Sµ? which could lead to a deteriorated sampling quality. To better illustrate this phenomenon, we represent in Figure 2a a synthetic experiment, where Sµθ does not recover Sµ? (by slightly shifting the mean of two modes after training the WGAN). In this setting, we clearly see in Figure 2b that Monte-carlo based methods worsen the WGAN. when Sµ? 6⊂ Sµθ , density ratios focus on local information and lead to non-optimal solutions. On the opposite, our method suggests to learn the optimal re-weighting of mass within the support Sµθ . Interestingly, on this synthetic dataset, it significantly reduces the Wasserstein distance to µ?, see Figure 2c.
4) Non-optimal discriminators. Knowing that optimal discriminators would lead to non-optimal objectives (very low acceptance probabilities), previous approaches made sure that their obtained classifier is sufficiently far from the optimal classifier (Section 3.2 in (Azadi et al., 2019)). Authors have thus come up with heuristics to approximate density ratios: for example Azadi et al. (2019) fine tune a regularized discriminator, Grover et al. (2019) use a pre-trained neural network on ImageNet classification and only fine-tune the final layers for the binary classification task. In our method, on the contrary, we are still looking for the discriminator maximizing the Integral Probability Metric (Müller, 1997) in equation 3, linked to optimal transport.

heading: 4 EXPERIMENTS
text: In the following section, we illustrate the efficiency of the proposed methods, latentRS and latentGA on synthetic datasets. Then, we compare their performances with previous works on image datasets. On this image generation tasks, we empirically stress that both latentRS or latentGA methods slightly surpass density ratios based methods while significantly reducing the time complexity.

heading: 4.1 EVALUATION METRICS
text: To measure performances of GANs when dealing with low dimensional applications - as with synthetic datasets - we equip our space with the standard Euclidean distance. However, for high dimensional applications such as image generation, Brock et al. (2019); Kynkäänniemi et al. (2019) have shown that embedding images into a feature space with a pre-trained convolutional classifier provides more semantic information. In this setting, we consequently use the euclidean distance between the images’ embeddings from a classifier. For a pair of images (a,b), we define the distance d(a,b) as d(a,b) = ‖φ(a)−φ(b)‖2 where φ is a pre-softmax layer of a supervised classifier, trained specifically on each dataset. Doing so, they will more easily separate images sampled from the target distribution µ? from the ones sampled by the distribution µθ .
We compare the performance of the different methods with a panel of evaluation metrics. To begin with, we use the Improved Precision/Recall (Improved PR) metric (Kynkäänniemi et al., 2019), a more robust version of the Precision/Recall metric which was first applied to the context of GANs by Sajjadi et al. (2018). The Improved PR metric is based on a non-parametric estimation of the support of both generated and real distributions using k-Nearest Neighbors. Besides, we also report two well-known metrics: the Earth Mover’s Distance (EMD), the discrete version of the Wasserstein distance, and the average Hausdhorff distance (Hausd). EMD is a distance between probability distributions while Hausd. focuses on support estimations. These two measures are particularly interesting in GANs since one can compute them with collections of discrete points. Let (x1, . . . ,xn) and (y1, . . . ,yn) be two collections of n data points and S be the set of permutations of [1,n], then:
EMD(X ,Y ) = min σ∈S
n
∑ i=1 ‖xi,yσi‖ and average Hausd(X ,Y ) = 1 n ∑xi∈X min y j∈Y ‖xi,y j‖
Besides, we argue that the Wasserstein distance should be a metric of reference when evaluating WGANs since it is directly linked to their objective function. Finally, for completeness, we report FID Heusel et al. (2017).

heading: 4.2 MIXTURE OF GAUSSIANS
text: Further experiments were ran on synthetic datasets with mixtures of 2D Gaussians, with either 4, 9, 16 or 25 components. When dealing with 2D mixtures of Gaussians, we used MLP with 4 hidden layers of 30 nodes for the generator, the discriminator and the importance weighter. As expected in this setting, a standard WGAN-GP combined with a connected latent space (i.e. multivariate normal or uniform), necessarily generates samples in-between two modes. Both Figure 1 and Figure 2a have stressed how the importance weighter can truncate latent space areas that are mapped outside the real data manifold and improve the EMD metric. More figures and details of the different evaluation metrics are given in Appendix.

heading: 4.3 IMAGE DATASETS
text: MNIST, F-MNIST and Stacked MNIST. We further study the efficiency of the proposed methods on three image datasets: MNIST (LeCun et al., 1998), FashionMNIST (F-MNIST) (Xiao et al., 2017), and Stacked MNIST (Metz et al., 2016) a highly disconnected datasets with 1,000 classes. For MNIST, F-MNIST and Stacked MNIST, we follow Khayatkhoei et al. (2018) and use a standard CNN architecture composed of a sequence of 3x3 convolution layer, relu activation with nearest neighbor upsampling. To exhibit the efficiency of the proposed methods in different settings, we use hinge loss with gradient penalty (Hinge-GP) (Miyato et al., 2018) on MNIST and F-MNIST, and a Wasserstein loss with gradient penalty (Gulrajani et al., 2017) on Stacked Mnist. For the importance weighter wϕ , we use an MLP architecture with fully-connected layers and relu activation. wϕ has 4 hidden layers, each having its width four times larger than the dimension of the latent space.
For exhaustivity, we compare latentRS and latentGA with previous works leveraging density ratios. In particular, we implemented a wide set of post-processing methods for GANs: DRS (Azadi et al., 2019), MH-GAN (Turner et al., 2019), SIR-GAN (Grover et al., 2019) and DOT (Tanaka, 2019). Similarly to Azadi et al. (2019), we take the discriminator at the end of the adversarial training, fine-tune it with the binary cross-entropy loss and select the best model in terms of EMD. During fine-tuning, we keep the gradient penalty or spectral normalization, otherwise the classifier easily separates real from generated data, which leads to a degraded performance, as shown in Figure 2a. Following Azadi et al. (2019); Grover et al. (2019), we do not include explicit mechanism to calibrate the classifier. To the extent of our knowledge, we are the first to empirically compare such a wide variety of Monte-Carlo methods on different datasets and metrics.
The main results of this comparison are shown in Table 1 (see Appendix for more details). We see that, except for Stacked MNIST, both of our methods outperform every other method on precision, av. Hausdhorff and the EMD metric. Interestingly, latentGA seems to be the strongest one. In Figure 3, we show how samples evolve when performing latent gradient ascent on the importance weights. As expected, as importance weights are increased, the quality of the generated images significantly improves. Besides, a strong contribution of the paper also resides in notably speeding-up the inference procedure. As shown in Table 1, the inference time of a given data point is 25 times faster with latentRS than with SIR-GAN.
CelebA is a large-scale dataset of faces covering a variety of poses. We train the models at 64x64 resolution. Following recent studies (Brock et al., 2019), the discriminator is trained with the hinge loss and spectral normalization (Miyato et al., 2018). For the generator network, residual connections (He et al., 2016) are used alongside self-modulation layers (Chen et al., 2019). The importance weighter is a simple 4 hidden-layer MLP with a width 10 times larger than the latent space dimension.
In this one-class high-dimensional dataset, the importance weighter still managed to learn some meaningful features. First, Figure 3 higlights a subtle improvement of the generated images when performing latentGA. Second, when ranking generated images with the importance weighter and comparing the top-5% vs worst-5% in Figure 4, we observe some differences in quality. However, on a broader scale, the importance weighter does not bring a clear improvement on neither the EMD nor the Hausdhorff metric. Interestingly, this is also the case for any of the different rejection methods (see Appendix for details).
We argue that in this one-class generation task, post-processing the generated samples is not as efficient as in a multi-modal setting (e.g. MNIST, FMNIST, Stacked MNIST). Intuitively, it is a much easier task to remove generated samples that are out of the target manifold than to discriminate between samples that already share similarities with training samples. It further stresses that this family of methods is useful if one needs to insert disconectedness in the modeled distribution. However, when the target distribution is a single-class distribution with a connected support, their efficiency decrease. To illustrate this, we added in Appendix a figure highlighting samples generated by a trained WGAN on Celeba 64x64, ranked by the discriminator. We observe that on these images, the discriminator does not correlate well with human judgement prohibiting the importance weighter to learn a meaningful signal.

heading: 5 CONCLUSION
text: In this paper, we provide insights on improving the learning of disconnected manifolds with GANs. Given the existence of the no GAN’s land, latent space areas mapping outside the target manifold, we provide two methods to truncate them. Contrary to previous works focusing on learning density ratios in the output space, both of our methods are based on training adversarially a neural network learning importance weights in the latent space. On the task of image generation, both of the proposed methods were shown to be empirically efficient while significantly reducing the inference time (latentRS by an order of 20), when compared to density ratio based methods. This paper has specifically stressed the efficiency of post-training methods when dealing with highly disconnected target distributions.
However, when dealing with single-class connected distributions or class-conditioning generative models, the efficiency of such methods is not clear. We argue that one of the reason is that, once the generator maps correctly inside the target manifold, it is a much harder task to discriminate between realistic and fake samples. A potential future work would therefore be to investigate how can we help the discriminator better classify among the set of generated images.

heading: A EVALUATION DETAILS
text: Precision recall metric. For the precision-recall metric, we use the algorithm from Khayatkhoei et al. (2018). Namely, when comparing the set of real data points (x1, ...,xn) with the set of fake data points (y1, ...,yn):
A point xi has a recall r(xi) = 1 if there exists y j, such that ‖xi− y j‖ ≤ ‖y j− y j(k)‖, where y j(k) is the k-nearest neighbor of n. Finally, the recall is the average of individual recall: 1n ∑i r(xi).
A point yi has a precision p(yi) = 1 if there exists x j, such that ‖yi− x j‖ ≤ ‖x j− x j(k)‖, where x j(k) is the k-nearest neighbor of n. Finally, the precision is the average of individual precision: 1n ∑i p(xi).
Images’ embeddings. As mentioned earlier, for images we use the distance between embeddings of images in a neural network trained specifically for classification on this dataset. For Stacked Mnist, we use a MNIST classifier on each output channel and simply stack the three embedding vectors.
Parameters. For all datasets, we use k = 3 (3rd nearest neighbor). For MNIST, F-MNIST and Stacked MNIST, we use a set of n = 2048 points. For CelebA, we use a set of n = 1024 points. This is also valid for the other metrics used: EMD, Av. Hausd. For FID on CelebA, we use the standard protocol evaluation with Inception Net and 50k data points.

heading: B HYPER-PARAMETERS.
text: SIR: Model selection: we fine-tune with a binary cross-entropy loss the discriminator from the end of the adversarial training and select the best model in terms of EMD.
We use then use Sampling-Importance-Resampling algorithm, with sets of n=40 points.
DRS: Model selection: we fine-tune with a binary cross-entropy loss the discriminator from the end of the adversarial training and select the best model in terms of EMD.
We use the standard Rejection Sampling algorithm, without artificially increasing the acceptance rate such as Azadi et al. (2019). We use regularized discriminator (with gradient penalty or spectral normalization), which avoids the acceptance rate falling to almost zero.
MH-GAN: Model selection: we fine-tune with a binary cross-entropy loss the discriminator from the end of the adversarial training and select the best model in terms of EMD. We use the independance Metropolis-Hastings algorithm with Markov Chains of 40 points, and select the last point.
DOT: Model selection: we fine-tune with the dual wasserstein loss the discriminator from the end of the adversarial training and select the best model in terms of EMD. We then perform a projected gradient descent as described in Tanaka (2019) with SGD, with Nsteps = 10 and ε = 0.01.
LRS: For MNIST, F-MNIST and Stacked MNIST, we use the same hyper-parameters: λ1 = 10, λ2 = 2 and m = 3. wϕ is a standard MLP with 4 hidden layers, each having 600 nodes (6x dimension of latent space), and relu activation. The output layer is 1-dimensional and with a relu activation.
For CelebA, we use: λ1 = 50, λ2 = 2 and m = 3. wϕ is a standard MLP with 4 hidden layers, each having 1280 nodes (10x dimension of latent space), and relu activation. The output layer is 1-dimensional and with a relu activation.
For the adversarial training of importance weights, we use the discriminator from the end of the standard adversarial training (generator vs discriminator). We then alternate between 1 step of wϕ and 1 update of Dα .
LGA: We use the same neural network than in LRS. The hyper-parameters for this method are similar to DOT: number of steps of gradient ascent Nsteps and learning rate ε . We choose Nsteps = 10 and ε = 0.1.
C VISUALIZATION AND RESULTS FOR SYNTHETIC DATASETS
(a) WGAN for the mixture of 9 Gaussians.
(b) WGAN for the mixture of 16 Gaussians. (c) WGAN for the mixture of 25 Gaussians.
(d) Heatmap in the latent space of the distance between a generated sample and its nearest neighbor for the mixture of 9 Gaussians. (e) Heatmap in the latent space of the distance between a generated sample and its nearest neighbor for the mixture of 16 Gaussians. (f) Heatmap in the latent space of the distance between a generated sample and its nearest neighbor for the mixture of 25 Gaussians.

heading: D COMPARISONS WITH CONCURRENT METHODS ON SYNTHETIC AND REAL-WORLD DATASETS

Review summaries:
The paper proposes a method for an improvement of generative adversarial models via post-processing its latent variable distribution. To be more precise, the method proposes to train an additional neural network that outputs an important weight for each point of the latent space, thus reweighting the final distribution in the space of images. For the optimization of this network, the authors use the dual form of the Wasserstein distance, where they multiply the initial latent density by the output of the network. To fix the ill-behaved objective, the authors add two regularization terms to it. The proposed objective is then validated on 3 MNIST-like datasets quantitatively and on CelebA qualitatively.

This work aims at improving the sample quality of generative models through better sampling, which is a relevant problem and has brought about a line of work [1,2,3,4,5], to name a few. By leveraging the idea of importance sampling, the authors train an additional network. The latter uses the information contained in the learned discriminator to assign importance weights to the latent points, thus defining a new distribution in the latent space. Subsequently, rejection sampling on the newly defined latent distribution is applied to obtain inputs for a generator network. By treating the problem in the latent space, the paper introduces latentRS method that compares favourably to several existing methods in terms of computational complexity for generating a sample. The authors propose one more method, latentGA, following the path in the latent space that maximizes the learned importance weights. The paper also discusses the limitations of the previously proposed methods and presents their empirical comparison on several datasets and metrics.

The paper proposes a new algorithm for improved sampling of GANs. Since GANs are continuous functions that act on a connected latent space, they will have trouble learning distributions whose support is disconnected (for e.g., clustered data). The proposed method tries to fix this issue and is motivated by rejected sampling. However, instead of using density based algorithms for rejecting samples, the authors take a fixed pre-trained generative model and train a neural network that learns to reject samples from the latent space.