Learning Disconnected Manifolds: Avoiding The No Gan's Land by Latent Rejection
Authors: Thibaut Issenhuth
Ugo Tanielian
David Picard
Jeremie Mary
Venue:  (2021)

Abstract: Standard formulations of GANs, where a continuous function deforms a connected latent space, have been shown to be misspecified when fitting disconnected manifolds. In particular, when covering different classes of images, the generator will necessarily sample some low quality images in between the modes. Rather than modify the learning procedure, a line of works aims at improving the sampling quality from trained generators. Thus, it is now common to introduce a rejection step within the generation procedure.
Building on this, we propose to train an additional network and transform the latent space via an adversarial learning of importance weights. This idea has several advantages: 1) it provides a way to inject disconnectedness on any GAN architecture, 2) the rejection avoids going through both the generator and the discriminator saving computation time, 3) this importance weights formulation provides a principled way to estimate the Wasserstein's distance to the true distribution, enabling its minimization. We demonstrate the effectiveness of our method on different datasets, both synthetic and high dimensional, and stress its superiority on highly disconnected data.

Under review as a conference paper at ICLR 2021
LEARNING DISCONNECTED MANIFOLDS: AVOIDING THE No GAN'S LAND BY LATENT REJECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Standard formulations of GANs, where continuous function deforms connected latent space, have been shown to misspecified when fitting disconnected mani- folds. In particular, when covering different classes of images, the generator will necessarily sample some low quality images between the modes. Rather than modify the learning procedure, a line of works aims at improving the sampling quality from trained generators. Thus, it is now common to introduce a rejection step within the generation procedure. Building additional network and transform the latent space via an adversarial learning of importance weights. This idea has several advantages: 1) it provides a way inject disconnectedness on any GAN architecture, 2) since the rejection happens in the latent space, it avoids going through both the generator and the discrimina- tor saving computation time, 3) this importance weights formulation provides a principled way reduce the Wasserstein's distance to the target distribution. We demonstrate the effectiveness of our method on different datasets, both synthetic and high dimensional.
this, we propose to train an
INTRODUCTION
GANs (Goodfellow al. 2014 are an effective way learn complex and high-dimensional distribu- tions, leading to state-of- the-trtert art models for image synthesis in both unconditional (Karras et al. (019)) and conditional settings (Brock et al., 2019). However, it well-known that a single generator with unimodal latent variable cannot recover distribution composed of disconnected sub-manifolds (Khayatkhoei al. 2018). This leads to common problem for practitioners: the necessary existence of very-low quality samples when covering different modes. This is formalized by Tanielian et al. (2020) which refers to this area as the no GAN's land and provides impossibility theorems on the learning of disconnected manifolds with standard formulations of GANs. Fitting disconnected target distribution requires an additional mechanism inserting disconnectedness in the modeled distribution. A first solution is to add some expressivity to the model: Khayatkhoei et al. (2018) propose train mixture of generators while Gurumurthy et al. (2017) make use of a multi-modal latent distribution. A second solution is improve the quality of trained generative model by avoiding its poorest
samples (Ta et al. 2018 Azadi et al. 2019 Turner et al. 2019 Grover et al. 2019 Tanaka 2019).
This second line of research relies heavily on variety of Monte-Carlo algorithms, such as Rejection These methods aim at sampling from a target distribution, while
Sampling or the Metropolis-Hasiine having only access samples generated from proposal distribution. This idea was successfully applied to GANs, using the previously learned generative distribution as a proposal distribution. However. one of the main drawback is that Monte-Carle algorithms only guarantee to sample from the target distribution under strong assumptions. First, we need access to the density ratios between the proposal and target distributions or equivalently to a perfect discriminator (Azadi al. 2019). Second, the support of the proposal distribution must fully cover the one of the target distribution. which means no mode collapse. This is known to be very demanding in high dimension since the intersection of supports between the proposal and target distribution is likely to be negligible (Arjovsky and Bottou| [0177] Lemma 3). In this setting, an optimal discriminator would give null acceptance probabilities for almost any generated points, leading to lower performance.
To tackle the aforementioned issue, we propose a novel method aiming at reducing the Wasserstein distance between the previously trained generative model and the target distribution. This is done via

---

Under review as a conference paper at ICLR 2021
the adversarial training of third network that learns importance weights in the latent space. The goal to learn the redistribution mass of the modeled distribution that best fits the target distribution. To better understand our approach, first consider simple 2D motivational example where the real data lies on four disconnected manifolds. To approximate this, the generator splits the latent space into four distinet areas and maps data points located in the frontiers, areas in orange in Figure Tbj of the true manifold (see Figure Tap. Our method consequently aims at learning latent importance weights that can identify these frontiers and simply avoid them. This is highlighted in Figure [d] where the importance weighter has identified these four frontiers. When sampling from the new latent distribution, we can now perfectly the mixture of four gaussians (see Figure Ic
(a) WGAN: real samples in green and fake ones in bluc.
(b) Latent space: heatmap of the distance between a generated sample and its nearest real sample.
(c) WGAN with latent rejection sampling: real samples in green and fake ones in blue.
(d) Latent space: heatmap of the learned importance weights. The blue frontiers have weights.
Learning disconnected manifolds leads to the apparition of an area the latent space generating points outside the target manifold. With the of the importance weighter, one can avoid this specific area and better fit the target distribution.
Figure 1:
Our contributions are the following:
We discuss works improving the sampling quality of GANs and identify their limitations.
We propose novel approach that directly modifies the latent space distribution. It provides a principled way to reduce the Wasserstein distance to the target distribution.
We thorougly compare our method with large set of previous approaches on variety of datasets and distributions. We empirically show that our solution significantly reduces the computational cost of inference while demonstrating an equal efficiency.
Notation. Before moving to the related work section, we shortly present notation needed in the paper. The goal of the generator is generate data points that are "similar" samples collected from some target probability measure 44, The measure is defined on potentially high dimensional space IR,, equipped with the euclidean norm ||- I|. To approach μl., we use parametric family of generative distribution where each distribution is the push-forward measure of a latent distribution Z and continuous function modeled by a neural network. In most all practical applications, the random variable Z defined or uniform distribution. The generator is parameterized class functions from Rd to RD, say )). where O C RR is the set of parameters describing the model. Each function
low dimensional space Rd is either multivariate Gaussian distribution
9 = (G Ge takes input from and outputs "fake" observations with distribution μ4 Ge On the other hand, the discriminator is described by family of functions from RD to R, say 9 = (Da E A),
C R2, where each Da. Finally, for any given distribution μ. note Sy its support.
2

---

Under review as a conference paper at ICLR 2021
2
RELATED WORK
2.1 DISCONNECTED MANIFOLD LEARNING: HOW TO TRAIN AND EVALUATE GANS
Goodfellow al. 2014) already stated that when training vanilla GANs, the generator could ignore modes of the target distribution: this is the mode collapse. A significant step towards understanding this phenomenon was made by Arjovskk and Bottou (017)) who explained that the standard formulation of GANs leads to vanishing or unstable gradients. The authors proposed the Wasserstein GANs (WGANs) architecture (Arjovsky al., 2017) where, in particular, discriminative functions are restricted to the class of I-Lipschitz functions. WGANS aim at solving:
sup inf Ex-u, (x - Ez-z (G(G)()))
acA 0ee
The broader drawback of standard GANs is that, since any modeled distribution the push-forward of unimodal distribution by continuous transformation, it consequently has a connected support. This means that when the generator covers multiple disconnected modes of the target distribution, necessarily generates samples out of the real data manifold (Khayatkhoei et al., 2018). Consequently, any thorough evaluation of GANs should assess simultaneously both the quality and the variety of the generated samples. (Sajaadi et al. (2088) argue that a single-digit metric such as the Inception Score (Salimans et al. (2016) or the Frechet Inception distance (Heusel et al. (017) is thus not adequate to compare generative models. To solve this issue, the authors propose Precision/Recall metric that aims at measuring both the mode dropping and the mode inventing.
In the Improved Precision/Recall KYnkääänieemi et al. 2019). the precision refers to the portion of generated points that belongs to the target manifold, while the recall measures how much of the target distribution can be re-constructe by the model distribution. Building on this metric, Taaielian et al. (2020) highlighted the trade-off property of GANs deriving upper- bounds on the precision of standard GANs. To solve this problem, a common direction of research consists in wer-ppraameterizing the generative model. Khayatkhoei et al. (2018) enforce diversity by using a mixture of generators while Gurumurthy et al. (2017) suggest that a mixture of Gaussians in the latent space is efficient to learn diverse and limited data.
2.2 IMPROVING THE QUALITY OF TRAINED GENERATORS
To better fit disconnected manifolds with standard GANs architectures, another line of research consists in inserting disconnectedness into a previously learned generative distribution μlg Taniclian ctt al. (020) proposed an hcuristic to remove the no GAN's land (i.e. samples mapped out of the true manifoldy: rejecting data points with a high Jacobian Frobenius norm. Another possibility would be to use one of the different Monte-Carlo methods (Robert and Casella (013) and apply it to GANs. Building up the well-known inference theory, Azadi et al. (2019) suggests the use of rejection sampling to improve the quality of the proposal distribution (4g. One can compute density ratios using either a classifier trained from scratch or the discriminator obtained the end of the training. Consequently, in this Discriminator Rejection Sampling (DRS), any generated data point x ~ μl is accepted with the following acceptance probability Pa:
Pa(x) (x)
Mxa(x) where M- max (x)
(2)
XES Hg (x)
where μl, and μl here refers to the density functions. Similarly, Turner et al. (2019) use the same density ratios and derive MH-GAN, an adaptation of the Metropolis-Hastinn algorithm (Hastings) 1970). that improves the sampling from Hlg. Finally, Grover et al. (2019) use these density ratios as importance weights and define importance resampled generative model whose density is now defined by fie x μą x r(xx. In order to perform discrete sampling from μlg authors rely on the Sampling-Importance. Resamplin (SIR) algorithm (Rubin) 1988 Liu and Chen. 1998). This defines new distribution (.y SIR.
r(x;) "
A (xx)
where XI, ~ ub-
Σ r(x))
Note that these algorithms rely the same density ratios and acceptance-rejection scheme. In Rejection Sampling, the acceptance rate is uncontrollable but sampling from μl, is assured. With SIR and MH, the acceptance rate is controllable but sampling from μ4 is no longer guaranteed.
3

---

Under review as a conference paper at ICLR 2021
3 ADVERSARIAL LEARNING OF LATENT IMPORTANCE WEIGHTS
3.1 OUR APPROACH
Similar to previous works, our method consists in improving the performance of given generative model, post-training. Given a trained WGANs (Ga. Du), we now propose to learn importance weights in the latent space. To do so, we use feed-forward neural network from R to IR', say . = {wo Q E Φ). The neural network We is trained using an adversarial process with the discriminator Du, whilst keeping the weights of Gg frozen. We now want to solve the following:
sup inf Ex--p, Pa (x)xx= Fegzz (Wo(zz) Da(GG(z))))
acA PE
Note that our formulation can also be plugged on top of many different objective functions. Interest- ingly, the of the predictor We defines new latent space distribution whose density ÿ defined by (z)) x w (z) x 7(z). Consequently, the newly defined modeled distribution fig is defined as the pushforward Pgg = Gott. The proposed method can be seen as minimizing the Wasserstein distance to the target distribution, over an increased class of generative distributions. The network w' thus learns how redistribute the mass of μl such that Ag closer to J., terms Wasserstein distance.
However, as in the field of counterfactual estimation, a naive optimization of importance weights by gradient descent can lead to trivial solutions. First, if for example, the Wasserstein critic Da outputs negative values for any generated samples, the network we could simply learn to avoid the dataset and output 0 everywhere. To avoid this issue, we follow [Swaminathan and Joachims (2015c) and scale the output of the discriminator such that the reward is always positive. A second problem comes from the fact that equation 3 can now be minimized not only by putting large importance weights (z) on the examples with high likelihoods Da (G(z)) but also by maximizing the sum of
the weights: this is the propensity overfitting (Swaminathan and Joachims) (015a)). To stabilize the optimisation process, we consequently introduce two important regularization techniques:
nollinormaliaaionn Similarly to Swaminathan and Joachims (2015a). we advocate the use of normalization of the importance weights. To be more precise, we enforce the expectation of the importance weights to be close 1 by adding a penalty term. By doing so, we prohibit the propensity overfitting since the sum of the importance weights in the batch is bounded.
To avoid cases where small areas of have really high wz(z) values, which would lead to mode collapse, we enforce a soft-clipping on the weights (Bottou et al. (0T3)/rove et al.. 2019). Note that this constraint function on the final layer, such as a re-scaled sigmoid or tanh activation.
Soft-Clipping
(z) could also be implemented with bounded activation
Finally, we thus get the following objective function:
sup Erz (P( v) -1 (E.-zw (z) 1)² -22 Eg-Z max pe
max (0, (wp(z) (wp(t))m)) (4)
discriminator reward
self normalizaaion
soft-clipping
where V ming-z DG(G(z)), 2, 22 and are hyper-parameter (values displayed in Appendix).
3.2 SAMPLING FROM THE NEW DISTRIBUTION
As mentionned above, the scale and variance of the learned importance weights are actively controlled, as it is done in counterfactual estimation (Bottou al. 2013 Swaminathan and Joachims 2015b Faury et al. 2020). Doing so, we explicitly control the acceptance rates Pz(z) of the rejection sampling algorithm performed on Q. since for any given z ~ have:
7(2) and F2 Pa(t)= PL(()) = my(z)
72) my(z)
122 m
((=)d== Jed
dz
m
where is the maximum output of the importance weighter defined in equation Latent Rejection Sampling (latentRS), the method that performs the Rejection Sampling algorithm on top of the learned importance weights. Since the exact sampling of the distribution y is now tractable with rejection sampling algorithm, we do not need to implement neither the Metropolis-Hasting nor the Sampling Importance Resampling algorithm.
4 We define as
4

---

Under review as a conference paper at ICLR 2021
(a) WGAN-GP: EMD= 0.12
(b) WGAN-GP with DRS: EMD 0.36
(c) WGAN-GI GP with LRS: EMD 0.03 0.01.
Figure 2: In this synthetic experiment, the target distribution has two modes slightly shifted from the generated distribution. When using an optimal discriminator, DRS only selects the intersection of the two supports, thus removing important data. Our method achieves a much better fit and does not suppress any mode. The EMD metric confirms the interest of the method in this specific case.
Inspired from the literature latent space optimal transport (Salimans et al. 2018 Agustsson et al. 2019 Tanaka 2019), we also propose a second method where we perform gradient ascent in the latent space. To be more precise, for any given sample in the latent space, we follow the path maximizing the learned importance weights. This method is denoted latent Gradient Ascent (latentGA). In high-dimension, similarly to Tanaka (2019) Algorithm 2), gradients are projected to restrict 2 on the training support. Note that the learning rate and the number of updates used for this method are hyper- parameter that need be tuned.
3.3 ADVANTAGES OF THE PROPOSED APPROACH
We now discuss, in detail, the flaws of previous Monte-Carlo based approaches:
1) Computational cost. By using sampling algorithms in the latent space, we avoid going through both the generator and the discriminator, leading to a significant computational speed-up. This is of particular interest when dealing with high-dimensional spaces since we do not need to pass through deep CNNs generator and discriminator (Brock al. 2019).
2) Existence of density functions. Every Monte-Care based methods assume that both μ4, and μlg are probability distributions with associated density functions. However, in high dimension, the hypothesis that data tend to lie near low dimensional manifold (Fefferman et al., (016) is now commonly accepted. Besides, it often the case that GANs be defined as the push-forward from much lower dimensional space, that is d << D. In that case, neither μ. nor Hg have density function in RD, Note that our method based on Wasserstein distance does not require this assumption.
3) Covering of the support Sy.. First, Monte-Carlo methods are well-known to suffer from the curse of dimensionality (Mengersen et al. 1996 Robert and Casella 2013). Besides, in the context of GANs, Arjovsky and Bottou (2017) Theorem 2.2) have shown that the intersection Sp, Umμo is likely to be a negligbble set under [g.. In this specific case, the density ratios would evaluate close to almost everywhere on Spg increasing the time complexity. More generally, Monte- Caree based methods tend to avoid any area within Spb \Sp,. which could lead to a deteriorated sampling quality. To better illustrate this phenomenon, we represent in Figure 2a synthetic experiment, where Sue does not recover Sp. (by slightly shifting the mean of two modes after training the WGAN), In this setting, we clearly see in Figure 2b that Monte-carlo based methods worsen the WGAN. when Sp. & Spo density ratios focus on local information and lead to non-optimal solutions. On the opposite, our method suggests to learn the optimal re-weighting of mass within the support Sup. Interestingly, on this synthetic dataset, it significantly reduces the Wasserstein distance to μ4, see Figure 2c
4) Non-optimal discriminators. Knowing that optimal discriminators would lead to non-optimal objectives (very low acceptance probabilities), previous approaches made sure that their obtained classifier is sufficiently far from the optimal classifier (Section 3.2 in (Azadi et al. 2019)). Authors have thus come up with heuristics approximate density ratios: for example Azadi al. (2019) fine tune regularized discriminator, Grover et al. (2019) use a pre-trained neural network on ImageNet classification and only fine-tune the final layers for the binary classification task. In our method, on the contrary, we are still looking for the discriminator maximizing the Integral Probability Metric (Müller 1997) in equation 3 linked optimal transport.
5

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
In the following section, we illustrate the efficiency of the proposed methods, latentRS and latentGA on synthetic datasets. Then, we compare their performances with previous works on image datasets. On this image generation tasks, we empirically stress that both latentRS or latentGA methods slightly surpass density ratios based methods while significantly reducing the time complexity.
4.1 EVALUATION METRICS
To measure performances of GANs when dealing with low dimensional applications as with synthetic datasets we equip our space with the standard Euclidean distance. However, for high dimensional applications such as image generation, Brock et al. (019); Kynkäänniemi et al. (2019) have shown that embedding images into feature space with pre-trained convolutional classifier provides more semantic information. In this setting, we consequently use the euclidean distance between the images' embeddings from a classifier. For pair of images (a, we define the distance d(a,b) as d(a, b) (a)(a) & (b)ll2 where & is pre-softmax layer of a supervised classifier, trained specifically on each dataset. Doing so, they will more easily separate images sampled from the target distribution μd, from the ones sampled by the distribution μlg.
We compare the performance of the different methods with a panel evaluation metrics. To begin with, we use the Improved Precision/Recall (Improved PR) metric (Kynkäänniemi et al. 2019), more robust version of the Precision/Recall metric which was first applied to the context of GANs by (Sajjadi et al. (2018). The Improved PR metric is based on non-parametric estimation of the support of both generated and real distributions using k-Nearest Neighbors. Besides, we also report two well-known metrics: the Earth Mover's Distance (EMD), the discrete version of the Wasserstein distance, and the average Hausdhorff distance (Hausd). EMD is a distance between probability distributions while Hausd. focuses on support estimations. These two measures are particularly interesting in GANs since one can compute them with collections of discrete points. Let (x)....,kn) and (yi..... "i..... Yn) be two collections of data points and y be the set of permutations of [1,n], then:
1 Σ min |x,,y|| n XEX MeEY
EMD(X, min GE
Σ ||,l and average Hausd
Besides, we argue that the Wasserstein distance should be a metric of reference when evaluating WGANs since it is directly linked to their objective function. Finally, for completeness, we report FID Heusel et al. (0177).
4.2 MIXTURE OF GAUSSIANS
Further experiments were on synthetic datasets with mixtures of 2D Gaussians, with either 4, 9, 16 or 25 components. When dealing with 2D mixtures of Gaussians, we used MLP with 4 hidden layers of 30 nodes for the generator, the discriminator and the importance weighter. As expected in this setting, a standard WGAN-GP combined with a connected latent space (i.e. multivariate normal or uniform), necessarily generates samples in-between stressed how the importance weighter can truncate latent space areas that are mapped outside the real data manifold and improve the EMD metric. More figures and details of the different evaluation metrics are given in Appendix.
modes. Both Figure [ and Figure 2a have
4.3 IMAGE DATASETS
MNIST, F-MNIST and Stacked MNIST. We further study the efficiency of the proposed methods on three image datasets: MNIST (LeCun et al. 1998), FashionMNIST (F-MNIST) (Xiao et al. 2017), and Stacked MNIST (Metz et al. 2016) a highly disconnected datasets with 1,000 classes. For MNIST, F-MNIST and Stacked MNIST, we follow Khayatkhoei et al. (2018) and use a standard CNN architecture composed of sequence of 3x3 convolution layer, relu activation with nearest neighbor upsampling. To exhibit the efficiency of the proposed methods in different settings, use hinge loss with gradient penalty (Hinge-GP) (Miyato et al. 2018) on MNIST and F-MNIST, and a Stacked Mnist. For the importance
Wasserstein loss with gradient penalty (Gulrajani et al.) 2017) weighter We. we use an MLP architecture with ally-connected layers and relu activation. Wo has hidden layers, each having its width four times larger than the dimension of the latent space.

---

Under review as a conference paper at ICLR 2021
MNIST
Prec. (f)
Rec. (f)
EMD (C)
Hausd (1)
FT)
Inference
Hinge-GP
87 47.4
94.6=#04
24.9+0.3
21.9=02
53.6#72
0.9
HGP: SIR-GAN
89.4
94.3+00.5
24.1+0.1
21.2=02
35.923.1
52.0
HGP: DOT
89.5
94.011.3
24.810.2
21.7=0.2
43.3+3.4
43
HGP: latentRS (+)
88.940.4
94.7.10.7
24.2:10.3
21.4=002
37.3
2.0
HGP: latentGA (x)
91.4+1.0
92.9+0.4
23.5+0.1
19.8
38.2+3.8
18
F-MNIST
Hinge-GP
86.420.6
86.8+0.6
68.640.4
58,4
598.94551
0.9
HGP: SIR-GAN
86.8=0.4
87 5..5
68, 28.9
57.4=0.3
558. 9
52.0
HGP: DOT
88.2+0.6
85 5.0.5
70.4
57.0=02
797.5=88.9
43
HGP: latentRS (+)
86.8
37.5:109
67.6110.6
58.6=0.5
A38.315002
2.1
HGP: latentGA (+)
88.4
86.8.1.
67. 0.1.99
56.4=0.2
475.5
Stacked MNIST
WGAN-GP
88.7+0.8
87.3+0.5
81.1+0.2
75.7-0.2
594.2
0..9
WGP: SIR-GAN
89.9
88.4
80.3
15.1=0.1
528. 113.44
52.0
WGP: DOT
90.8=0.5
90.2.10.7
79.4::01
74.1=022
43
WGP: latentRS (+)
90.4=1.4
86.6.0.5
80.3.10.1
75.
568.5
1.9
WGP: latentGA (+)
90.3:107
87 87.2.0.5
80.1
74.9=0.2
530.1
18
Table 1: On the image generation task, latentGA seems the best performer and latentRS matches
sato with a significantly reduced inference cost (by order of 20). + is 97% confidence interval. Inference refers to the time needed to compute one image on a NVIDIA V100 GPU.
3
Figure 3: Gradient ascent on latent importance weights (latentGA): the quality is gradually improved as we move to higher importance weights. Each image is generated only for the purpose visualiza- tion, and one can run this gradient ascent directly in the latent space with the importance weighter.
For exhaustivity, compare latentRS and latentGA with previous works leveraging density ratios. In particular, we implemented a wide set of post-processing methods for GANs: DRS (Azadi et al. 2019), MH-GAN (Turner al. 2019), SIR-GAN (Grover al. 2019) and DOT (Tanaka (019)) Similarly to Azadi et al. (2019). we take the discriminator at the end of the adversarial training, fine-tune it with the binary cross-entropy loss and select the best model in terms of EMD. During fine-tuning, we keep the gradient penalty or spectral normalization, otherwise the classifier easily separates real from generated data, which leads a degraded performance, as shown Figure 2a Following Azadi et al. 2019); Grover et al. (2019), we do not include explicit mechanism to calibrate the classifier the extent of our knowledge, we are the first to empirically compare such wide variety of Monte-Carlo methods on different datasets and metrics.
The main results of this comparison are shown in Table I (see Appendix for more details). We see that, except for Stacked MNIST, both of our methods outperform every other method on precision, av. Hausdhorff and the EMD metric. Interestingly, latentGA seems to be the strongest one. In Figure we show how samples evolve when performing latent gradient ascent the importance weights. As expected, as importance weights are increased, the quality of the generated images significantly improves. Besides, strong contribution of the paper also resides in notably speeding-uv up the inference procedure. As shown Table the inference time of a given data point is 25 times faster with latentRS than with SIR-GAN.
7

---

Under review as a conference paper at ICLR 2021
Ranking CelebA faces with the trained importance weighter. For each line, we compare the worst-5% vs best-5% of the generated samples. We a significant difference in terms of quality.
Figure 4:
CelebA is large-scale dataset of faces covering variety of poses. We train the models at 64x64 resolution. Following recent studies (Brock al. (209)) the discriminator trained with the hinge loss and spectral normalization (Miyato al. (0188) 8j. For the generator network, residual connections (He et al., 2016) are used alongside self- ceff-modulation layers (Chen et al. 2019). The importance weighter is simple 4 hidden-layer MLP with width 10 times larger than the latent space dimension.
In this one-class high-dimensional dataset, the importance weighter still managed to learn some meaningful features. First, Figure 3 higlights subtle improvement of the generated images when performing latentGA. Second, when ranking generated images with the importance weighter and comparing the top-5% vs worst-5% in Figure 4 observe some differences in quality. However, a broader scale, the importance weighter does not bring a clear improvement on neither the EMD nor the Hausdhorff metrie. Interestingly, this also the case for any of the different rejection methods (see Appendix for details).
We argue that in this one-class generation task, post-processing the generated samples is as efficient as in a multi-modaa setting (e.g. MNIST, FMNIST, Stacked MNIST). Intuitively, it is much easier task remove generated samples that are out of the target manifold than discriminate between samples that already share similarities with training samples. It further stresses that this family of methods useful if one needs to insert disconectedness in the modeled distribution. However, when the target distribution is single-class distribution with connected support, their efficiency decrease. To illustrate this, we added in Appendix a figure highlighting samples generated by trained WGAN Celeba 64x64, ranked by the discriminator. We observe that these images, the discriminator does not correlate well with human judgement prohibiting the importance weighter to learn meaningful signal.
CONCLUSION
In this paper, we provide insights improving the learning of disconnected manifolds with GANs. Given the existence of the no GAN's land, latent space areas mapping outside the target manifold, we provide two methods to truncate them. Contrary to previous works focusing learning density ratios in the output space, both our methods are based on training adversarially a neural network learning importance weights in the latent space. On the task of image generation, both of the proposed methods were shown to be empirically efficient while significantly reducing the inference time (latentRS by an order of 20), when compared density ratio based methods. This paper has specifically stressed the efficiency of post-traiil post-training methods when dealing with highly disconnected target distributions.
However, when dealing with single-class connected distributions or conditionii generative models, the efficiency of such methods is not clear. We argue that one of the reason is that, once the generator maps correctly inside the target manifold, it is much harder task to discriminate between realistic and fake samples. A potential future work would therefore be to investigate how can help the discriminator better classify among the set of generated images.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Eirikur Agustsson, Alexander Sage, Radu Timofte, and Luc Van Gool. Optimal transport maps for distribution preserving operations on latent spaces of generative models. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
M. Arjovsky and L.. Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations, 2017.
M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. D. Precup and Y.W. Tch, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 214-223. PMLR, 2017.
Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discrimi- nator rejection sampling. In International Conference on Learning Representations, 2019.
Léon Bottou, Jonas Peters, Joaquin Quinonero Candela, Denis Xavier Charles, Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Y Simard, and Ed Snelson. Counterfactual Reasoning and Learning Systems: the example of Computational Advertising. Journal of Machine Learning Research, 14(1):300733260, 2013.
A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019.
Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. In International Conference on Learning Representations, 2019. URL
https:://ppeneviiee --
L. Faury, U. Tanielian, E. Dohmatob, E. Smirnova, and F. Vasile. Distributionally robust counterfactual risk minimization. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 3850- 3857. AAAI Press, 2020.
C. Fefferman, S. Mitter, and H. Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29:983-1049, 2016.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio. Generative adversarial nets. Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 1672-2680. Curran Associates, Inc., 2014.
Aditya Grover, Jiaming Song, Ashish Kapoor, Kenneth Tran, Alekh Agarwal, Eric Horvitz, and Stefano Ermon. Bias correction of learned generative models using likelihood importance weighting. In Advances in Neural Information Processing Systems, pages I 1056-11068. 2019.
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A.C. Courville. Improved training of Wasser- stein GANs. I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5767-5777. Curran Associates, Inc., 2017.
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 166-174, 2017.
W Keith Hastings. Monte carlo sampling methods using markov chains and their applications. 1970.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.

---

Under review as a conference paper at ICLR 2021
T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
Mahyar Khayatkhoei, Maneesh K Singh, and Ahmed Elgammal. Disconnected manifold learning for generative adversarial networks. In Advances in Neural Information Processing Systems, pages 7343-7353, 2018.
Kynkäänniemi, T. Karras, Laine, Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Alc Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems 32, pages 3927-3936. Curran Associates, Inc., 2019.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278-2324, 1998.
Jun S Liu and Rong Chen. Sequential monte carlo methods for dynamic systems. Journal of the American statistical association, 93(443):1032-1044, 1998.
Kerrie Mengersen, Richard L Tweedie, al. Rates of convergence of the hastings and metropolis algorithms. The annals of Statistics, 24(1):101-121, 1996.
L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein. Unrolled generative adversarial networks. arXiv. 1,100226,,, 2016.
T. Miyato, Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
A. Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29:429-443, 1997.
Christian Robert and George Casella. Monte Carlo statistical methods. Springer Science & Business Media, 2013.
Donald B Rubin. Using the sir algorithm to simulate posterior distributions. Bayesian statistics, 395-402, 1988.
M.S.M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall, In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- BiannBinnchi, and R. Garnett, editors, Advances Neural Information P Processing Systems 31, pages 5228-5237. Curran Associates, Inc., 2018.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training GANs. In D.D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 2234-2242 Curran Associates, Inc., 2016.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport. In International Conference Learning Representations, 2018.
A. Swaminathan and T. Joachims. The seff-normalized estimator for counterfactual learning. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances Neural Information Processing Systems 28, pages 3231-3239. Curran Associates, Inc., 2015a.
Adith Swaminathan and Thorsten Joachims. Batch Learning from Logged Bandit Feedback through Counterfactual Risk Minimization. Journal of Machine Learning Research, 16(1):13111175. 2015b.
Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. volume 37 of Proceedings of Machine Learning Research, pages 814-823, Lille, France, 07-09 Jul 2015c. PMLR.
Akinori Tanaka. Discriminator optimal transport. In Advances in Neural Information Processing Systems, pages 2019..
2019.
10

---

Under review as a conference paper at ICLR 2021
U. Tanielian, T. Issenhuth, E. Dohmatob, and J. Mary. Learning disconnected manifolds: a gan's land. International Conference on Machine Learning, 2020.
Chenyang Tao, Liqun Chen, Ricardo Henao, Jianfeng Feng, and Lawrence Carin Duke. Chi-square
generative adversarial network. In International conference on machine learning, pages 4887-4896, 2018.
Ryan Turner, Jane Hung, Eric Frank, Yunus Saatchi, and Jason Yosinski. Metropolis-hasting generative adversarial networks. International Conference on Machine Learning, pages 6345- 6353, 2019.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms.

---

Under review as a conference paper at ICLR 2021
A EVALUATION DETAILS
Precision recall metric. For the precision-recall metric, we use the algorithm from Khayatkhoei al. 2018). Namely, when comparing the set of real data points (x1,)))))) with the set of fake data points (a):
point x; has recall r(x;) if there exists yy, such that ||x; -yyl |1y; 155-yp yy(k)],, where yy(k) is the k-nearest neighbor n. Finally, the recall is the average of individual recall: 1 à [.)(x)))
point y; has precision p(x) if there exists xj, such that |yy; -xyl xyll S xx))]] where xy(k)
the k-nearest neighbor of n. Finally, the precision is the average of individual precision: p(x;).
Images' embeddings. As mentioned earlier, for images we use the distance between embeddings of images in neural network trained specifically for classification on this dataset. For Stacked Mnist, we use MNIST classifier on each output channel and simply stack the three embedding vectors.
Parameters. For all datasets, we use k = (3rd nearest neighbor). For MNIST, F-MNIST and Stacked MNIST, we use a set of n 2048 points. For CelebA, we use a set of 1024 points. This also valid for the other metrics used: EMD, Av. Hausd. For FID on CelebA, we use the standard protocol evaluation with Inception Net and 50k data points.
B HYPER-PARAMETERS.
SIR: Model selection: we fine-tune with binary cross-entropy loss the discriminator from the end of the adversarial training and select the best model in terms of EMD.
We use then use Saaptgg-ooooootteeeeeeeeee
algorithm, with sets of n=40 points.
DRS: Model selection: we fine-tune with a binary cross-entropy loss the discriminator from the end of the adversarial training and select the best model in terms of EMD.
We use the standard Rejection Sampling algorithm, without artificially increasing the acceptance rate such as Azadi et al. (2019). We use regularized discriminator (with gradient penalty or spectral normalization), which avoids the acceptance rate falling to almost zero.
MH-GAN: Model selection: we fine-tune with binary cross-entropy loss the discriminator from the end of the adversarial training and select the best model in terms of EMD. We use the independance Metropolis-Hasting algorithm with Markov Chains of 40 points, and select the last point.
we fine-tune with the dual wasserstein loss the discriminator from the end of the adversarial training and select the best model in terms of EMD. We then perform projected gradient descent as described in Tanaka 2019) with SGD, with Nstepe = 10 and E = 0.01.
DOT: Model selection:
LRS: For MNIST, F-MNIST and Stacked MNIST,
the same hyper-parameters: 2 10, 22 2 and m = 3. We a standard MI.P with hidden layers, each having 600 nodes (6x dimension of latent space), and relu activation. The output layer I-dimensional and with relu activation.
For CelebA, we use: A = 50, 2 = 2 and m = 3. We is standard MLP with 4 hidden layers, each having 1280 nodes (10x dimension of latent space), and relu activation. The output layer is 1 I-dimensional and with a relu activation.
For the adversarial training of importance weights, we use the discriminator from the end of the standard adversarial training (generator vs discriminator). We then alternate between I step of We and update of Da-
LGA: We use the same neural network than in LRS. The hyper-pann parameters for this method are similar DOT: number of steps of gradient ascent Naeps and learning rate e. We choose Nateps 10 and g = 0.1.
12

---

Under review as a conference paper at ICLR 2021
C
VISUALIZATION AND RESULTS FOR SYNTHETIC DATASETS
a WGAN for the mixture of 9
(b) WGAN for the mixture of 16 Gaussians.
(c) WGAN for the mixture of 25 Gaussians.
Gaussians.
(d) Heatmap in the latent space of the distance between generated sample and its nearest neighbor for the mixture 9 Gaussians.
(e) Heatmap in the latent space of the distance between generated sample and its nearest neighbor for mixture of Gaussians.
(i) Heatmap in the latent space of the distance between generated sample and its nearest neighbor for the mixture 25 Gaussians.
(g) Heatmap of the importance weights in the latent space for the mixture of Gaussians.
(h) Heatmap of the importance weights in the latent space for the mixture of Gaussians.
(i) Heatmap of the importance weights in the latent space for the mixture 25 Gaussians.
4.
(i) WGAN with latentRS for the mixture of Gaussians.
(k) WGAN with latentRS for the mixture of 16 Gaussians.
(l) WGAN with latentRS for the mixture of 25 Gaussians.
Figure 5: Synthetic examples on mixtures of Gaussians with respectively (per column) and 25 components. Real samples in green and generated points in blue.
13

---

Under review as a conference paper at ICLR 2021
Mixture of Gaussians
Prec.
Rec.
EMD
Hausd.
WGAN-GP
95.0+0.5
78.5+1.
0.50+0.05
0.85+0.05
WGAN-GP with latentRS (*)
99.2+0.1
70.8+2.2
0.37=0.07
0. 0.102..02
Mixture of Gaussians
Hinge-GP
86.3
76,2+0.8
0.45
0.57
HGP: latentRS (+)
91.0+0.3
73.0+14
0.40+0.06
0.23 0.04
Mixture of 16 Gaussians
Hinge-GP
70.6+0.3
68.2+0.8
1.1020.07
0.85
HGP: latentRS (+)
74.0+0.3
69.5+1.
1.00+0.04
0.54+0.04
Mixture of 25 Gaussians
Hinge-GP
63.410.3
64.61008
1.35.0.08
1. 1020.07
HGP: latentRS (+)
70.6+0.3
67.0+14
1.25+0005
0.8+0.04
Table 2: Performance of the latentRS method on the estimation of 2D mixtures of Gaussians with
respectively 4, 9, 16 or components. Each mode variance σ² 0.025 separated by
distance = 5. As expected the latentRS method removes data points that are off the target manifold and therefore significantly improves the Hausdhorff metric.
D COMPARISONS WITH CONCURRENT METHODS ON SYNTHETIC AND REAL-WORLD DATASETS
.
(a) WGAN-GP.
(b) WGAN-GP with SIR.
(c) WGAN-GP with DOT.
(d) WGAN-GP with LRS.
(e) WGAN-GP.
(I) WGAN-GP with SIR.
(g) WGAN-GP with DOT.
(h) WGAN-GP with
LRS.
Figure 6: Comparisons of concurrent methods on synthetic datasets: mixtures of 25 Gaussians and on 2D Swiss Roll. Experimental setting from Tanaka (2019). Real samples in green and generated points in blue.
14

---

Under review as a conference paper at ICLR 2021
Mixture of 25 Gaussians
Prec.
Rec.
EMD
WGAN-GP
24.7+40.5
90.4+1.0
0.004
WGAN-GP with DRS
86.9
84.0+0.5
0.038+0.002
WGAN-GP with SIR
84.3=0.8
90.0=0.7
0.041
WGAN-GP with DOT
72.7=10
76.320.8
0.035=#0002
WGAN-GH with latentRS (+)
36.2.10.8
88.5.0.9
0.036:10001
Swiss Roll
WGAN-GP
77.9+0.5
90.3+0.7
0.030++0002
WGAN-GP: DRS
97.7+402
94.4+0..5
0.036+0.004
WGAN-GP: SIR
97.3+0.6
93.9=0.7
0.037:0.003
WGAN-GP: DOT
92.9=0.6
77.7=0.5
0.029=0.008
WGAN-GP: latentRS (+)
84.3:108
90.1
0.025 :00.002
Table 3: Comparison of latentRS with concurrent methods on two synthetic datasets in the same
setting as [Tankaa 2019). Our method enables consistent gain EMD, surpassing other methods on Swiss Roll and slightly behind DOT on Mixture of 25 Gaussians.
MNIST
Prec. (t)
Rec. (t)
EMD (L)
av. Hausd (L)
FID (L)
Hinge-Gr GPgge-GP...
87.4=0.9
94.6204
24.940.3
21.9=0.2
53.617.2
HGP: DRS
89.0-0.5
94.4+005
24.3+03
21.2-n2
41.3+8.2
HGP: SIR-GAN
89.4-0.5
94.3+0.5
24.140.1
21.2-0.2
35.9+3.1
HGP: MH-GAN
89.4=0.6
94.520.3
24.420.3
21.1=0.1
43.2+5.7
HGP: DOT
89.5=006
94.0=0.3
24.8+0.2
21.7-02
43.3+3.4
HGP: latentRS (+)
88.9=044
94.7=107
24.2+0.3
21.4-02
37.3#3.2
HGP: latentGA (+)
91.4
92.9#0.4
23.5
19.8
45.2+3.8
F-MNIST
Hinge-GP
86.4
86.8+0.6
68.6404
58.4-n.3
598.9
HGP: DRS
86.7+0.7
87.4
68.7
57.7-0.2
546.8 +37.3
HGP: SIR-GAN
86.8=0.4
87,420.5
68 68.9408
57.4=033
558.9458.7
HGP: MH-GAN
87.8=0.5
87 $7.0=0.4
68.8+0.6
57.6-02
555.3
HGP: DOT
88.2=0.6
85.8
70.4
57.0=022
797.5
HGP: latentRS (+)
86.8
87.5
67. 67.6400.
58.6
438.3
HGP: latentGA (+)
88.4
86.8+0.7
67.040.9
56.4-0.2
475.5
Stacked MNIST
WGAN-GP
88.7=0.8
87.3
81.140.2
75.7=02
594.2=8.2
WGP DRS
89.9
88.540.6
80.2+0.1
75.1=0.1
527 22..2=9.
WGP: SIR-GAN
89.9
88.4
80.3
75.1-0.1
528. +:3.4
WGP: MH-GAN
90.3=006
88.940.5
80.210.1
75.1=0.1
527.2
WGP: DOT
90.8=005
90.21007
79.410.
74.1-02
484.0=85
WGP: latentRS (+)
90.4=0.4
86.6+0.5
80.3+0.1
75.3=0.1
568.5
WGP: latentGA (*)
90.3
87.2+0.5
80.1
74.9-02
530.1
CelebA 64x64
Hinge-SN
88.2-066
66.7408
37.7402
34.5-02
17.820.1
HSN: DRS
88.1
66.5+1.
38.3+07
34.5-02
19. 10.22
HSN: SIR
87.9=0.6
66.941.3
38. 1802
34,4-022
19.240.2
HSN: MH
88.1=0.8
67.3
38.140.3
34.4=0.3
19.2+0.2
HSN: DOT
88.0+0.8
67.4+1..
37.4+003
34.3-n.3
18.0+n.1
HSN: latentRS (x)
88.0-06
68.1
40.3
34.1-0.3
18.0+0.1
HSN: latentGA (+)
88. 10..6
66.8+111
37.440.2
34.3-02
18.12002
Table Comparing latentRS and latentGA with previous works (DRS, MH-GAN, SIR-GAN, DOT) on the image generation task. Interestingly, latentGA seems the best performer and latentRS matches sato with a significantly reduced inference cost (by an order 20). + 97% confidence interval. Inference refers to the time needed to compute one image on NVIDIA V100 GPU. + is 97% confidence interval.
15

---

Under review as a conference paper at ICLR 2021
(a) Generated samples the Celeba dataset 64x64 ordered by the discriminator (left right, top bottom),
(b) Generated samples the Celeba dataset 64x64 ordered by the importance weighter (left to right, top to bottom).
(c) Generated samples on the Celeba dataset 64x64 ordered by the discriminator (left to right, top
(d) Generated samples the Celeba dataset 64x64 ordered by the importance weighter (left to right, top bottom).
bottom).
Figure 7: We observe that the ranking with the discriminator does not perfectly correlate with human judgment, prohibiting the importance weighter learn meaningful features. Consequently, the target distribution is not better approximated.
16


Reviews:
[{"review_id": "jafny_CBEOw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for an improvement of generative adversarial models via post-processing its latent variable distribution. To be more precise, the method proposes to train an additional neural network that outputs an important weight for each point of the latent space, thus reweighting the final distribution in the space of images. For the optimization of this network, the authors use the dual form of the Wasserstein distance, where they multiply the initial latent density by the output of the network. To fix the ill-behaved objective, the authors add two regularization terms to it. The proposed objective is then validated on 3 MNIST-like datasets quantitatively and on CelebA qualitatively.\n\nReview:\nMy major concern is the limited theoretical novelty together with modest empirical study. Let me clarify. I think the idea to put the filtering stage into the latent space is indeed worthy. However, the straightforward amortization of the discriminator network via a fully connected network is challenging due to the described computational problems and usually high dimensionality of the latent space. Furthermore, the verification of the method on MNIST-like data does not seem convincing, especially when the relevant works provide a comparison on ImageNet (Azadi 2018, Neklyudov 2019).\n\nAdditional comments:\n1. perhaps, I'm missing something, but for me, it is not clear why the objective in equation (3) corresponds to the optimization of Wasserstein distance in the space of images w.r.t. the parameters alpha and phi. I mean that there are even no guarantees that \\widehat{\\gamma} is a distribution.\n2. \"since the rejection sampling scheme is now tractable, we do not need to implement the MH algorithm or the importance sampling\". Firstly, I do not understand why the rejection sampling is tractable. The regularization term does not provide any guarantees for the maximum value of the density ratio. Secondly, even if the rejection sampling is tractable, I still find the MH algorithm more efficient: it does not require the evaluation of the constant; given the same proposal, MH's acceptance rate is greater or equal to the acceptance rate of the rejection sampling.\n3. the authors claim that reweighting in the latent space allows for better support coverage than the methods operating on the image space. Although I believe that such an effect occurs, I wouldn't expect the quality of images to be high. Indeed, this additional coverage could be produced by sampling from the low-density regions of the latent distribution. It is clear that such regions are underrepresented during the training. Moreover, there is empirical evidence of the deteriorating quality of images for latent distributions with higher variance (see Brock 2018).\n4. the bottom of page 3. DRS does not assure sampling from the target distribution since it adjusts the constant and uses an approximation of density ratio. In contrast, the MH algorithm provides some guarantees by upper bounding the total variation distance between the stationary distribution and the target (see Neklyudov 2019).\n\nminor comments:\n1. abstract. I would suggest finding a better analog for the phrase \"inject disconnectedness\". It does not sound like a desirable feature of your model when we speak about GANs, especially at the beginning of the paper, where few context is given. I would propose something like \"postselection\" or \"filtering\".\n2. eq. 4, the signs of regularization terms are incorrect\n3. typo on page 5, item 2). every methods -> every method\n\nReferences:\n1. (Azadi 2018) Azadi, Samaneh, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. \"Discriminator rejection sampling.\" arXiv preprint arXiv:1810.06758 (2018).\n2. (Brock 2018) Brock, Andrew, Jeff Donahue, and Karen Simonyan. \"Large scale gan training for high fidelity natural image synthesis.\" arXiv preprint arXiv:1809.11096 (2018).\n3. (Neklyudov 2019) Neklyudov, Kirill, Evgenii Egorov, and Dmitry P. Vetrov. \"The Implicit Metropolis-Hastings Algorithm.\" In Advances in Neural Information Processing Systems, pp. 13954-13964. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "axrkiy837Zz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work aims at improving the sample quality of generative models through better sampling, which is a relevant problem and has brought about a line of work [1,2,3,4,5], to name a few. By leveraging the idea of importance sampling, the authors train an additional network. The latter uses the information contained in the learned discriminator to assign importance weights to the latent points, thus defining a new distribution in the latent space. Subsequently, rejection sampling on the newly defined latent distribution is applied to obtain inputs for a generator network. By treating the problem in the latent space, the paper introduces latentRS method that compares favourably to several existing methods in terms of computational complexity for generating a sample. The authors propose one more method, latentGA, following the path in the latent space that maximizes the learned importance weights. The paper also discusses the limitations of the previously proposed methods and presents their empirical comparison on several datasets and metrics.\n\nThe method is concise and straightforward to be applicable by a broad community of ML practitioners. One of the proposed methods, latentRS, offers a significant speedup at the inference stage compared to analyzed methods while being similar in performance metrics. The paper also raises an interesting question of whether the existing enhanced sampling methods help when the target distribution is not sufficiently disconnected.\n\nHowever, there are several weaknesses in the experiments which lead to questioning the claims. While the paper's claim of careful comparison with the existing methods and the discussion of existing methods' limitations is indeed well-presented, the paper neglects the already recurring standard experiments for such methods or brushes them under the Appendix section. The generated samples from the Appendix figure for the mixture of Gaussians with n>9 show that the results are not as promising as the same experiments in the literature (the 'fake' clouds are not as nicely located on top of the true ones). n=25 is a recurring setting and seems to be a standard check for algorithms that refine GAN's sampling (e.g. DRS, DOT, DDLS). Table 2 in the Appendix lacks computer metrics for existing methods since it would demonstrate the tangible interpretable difference in this setting between comparable methods.\n\nThe paper misses some essential experiments to be faithfully compared with existing methods. It would be helpful to see the Swiss Roll experiment and the statistics on recovered modes and quality on 25 Gaussians for all the considered methods. As for more realistic image spaces, the CIFAR10 is a dataset that represents an undoubtfully disconnected manifold, and it has more potential to show the advantage of the proposed methods.\n\nReturning to the presented empirical study, these too raise a number of questions. The IPR results in Table 1 (and Table 3 in the Appendix) do not show consistent advantages for the proposed methods over the existing ones. It either favours latentRS or latentGA in terms of precision or recall alone, not both at the same time. It's understandable that when maximizing importance weights with latentGA we get higher precision; we force the generated samples to stay within true points at the cost of their diversity (which can also be seen in the synthetic experiment with Gaussians), so I guess the method is highly reliant on the hyperparameter m, which controls the 'conservativeness' of the trained importance weights network. It would be helpful to see an ablation study for the hyperparameters.\n\nGiven all the above, I am leaning towards a reject and my main concerns are as follows. The experiment with 25 Gaussians doesn't show as much improvement in sampling as existing methods implying there might be little effect in real-world datasets. I believe that the proposed methods have not been faithfully compared to the existing methods. There is no ablation study on the hyperparameters of the proposed methods.\n\nWhile the argument about one class CelebA has grounds, the DRS technique shows that it improves face generation by producing less warped nightmare-like faces. Thus, better GAN sampling techniques should ideally not only help avoid empty regions in the latent space between the nodes (inject disconnectedness) but also grasp the shape of those modes. In this regard, using an energy-based model for the latent variable might be an apt direction [5].\n\nThe authors state that they use image embeddings from corresponding classification networks for each of their datasets, but how do they obtain image embeddings for CelebA?\n\nSome minor points — the notation for the proposal and true distribution we want to sample from in section 3 is a bit confusing as the hat is usually used to denote an approximation. Also, the submission has quite a few typos — it needs proofreading.\n\nReferences:\n[1] Azadi, Samaneh, et al. \"Discriminator rejection sampling.\" arXiv preprint arXiv:1810.06758 (2018).\n\n[2] Turner, Ryan, et al. \"Metropolis-hastings generative adversarial networks.\" International Conference on Machine Learning. 2019.\n\n[3] Neklyudov, Kirill, Evgenii Egorov, and Dmitry P. Vetrov. \"The Implicit Metropolis-Hastings Algorithm.\" Advances in Neural Information Processing Systems. 2019.\n\n[4] Tanaka, Akinori. \"Discriminator optimal transport.\" Advances in Neural Information Processing Systems. 2019.\n\n[5] Che, Tong, et al. \"Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling.\" arXiv preprint arXiv:2003.06060 (2020).", "extra_scores": {}, "extra_text": {}}, {"review_id": "TopfdzCOt8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary:\nThe paper proposes a new algorithm for improved sampling of GANs. Since GANs are continuous functions that act on a connected latent space, they will have trouble learning distributions whose support is disconnected (for e.g., clustered data). The proposed method tries to fix this issue and is motivated by rejected sampling. However, instead of using density based algorithms for rejecting samples, the authors take a fixed pre-trained generative model and train a neural network that learns to reject samples from the latent space.\n\n## Significance:\nThe problem is well motivated and seems significant. Learning distributions with disjoint support can be difficult using the traditional GAN training, and this paper addresses this problem by learning which areas in the latent space must be avoided. \n\n## Quality:\nWhile the problem is significant, I find that the proposed method has some weaknesses in its formulation and empirical evaluation. Please see the section \"Cons\" below.\n\n## Originality:\nThe proposed method seems sufficiently novel, but I am not familiar enough with this area to know if something closely related has been done before.\n\n## Pros:\n1. On synthetic data, the proposed method can capture modes better than the considered baselines.\n1. The proposed technique produces better quality samples on GANs trained on CelebA.\n\n## Cons:\n1. While some of the experiments are convincing, I do not buy some of the arguments made in the paper. Specifically, under eqn (3), it is argued that if the GAN $G$ is kept fixed and the following adversarial training is performed for a classifier $w_\\phi$ and discriminator $D_\\alpha$, then the following procedure:\n$$ \\sup_{\\alpha \\in A} \\inf_{\\phi \\in \\Phi} E_{x \\sim \\mu} D_\\alpha(x) - E_{z \\sim Z} [ w_\\phi(z) \\cdot D_\\alpha( G(z) )]$$\nwill produce a new distribution on $z$ which is $\\widehat{\\gamma}(z) \\propto \\gamma(z) w_\\phi(z)$.\nThere is no proof of this claim, and I further suspect that this claim in not correct.\n\n1. There exist several baseline methods that consider the problem of mode collapse. Examples like PAC-GAN [Lin et al 2017] have shown to be effective, and are also provably good. Other examples include those considered in table 5 of https://arxiv.org/pdf/2010.00654v1.pdf\nFor the Stacked-MNIST dataset, the algorithms listed in table 5 seem to have much better mode coverage than the algorithms in Table 1 of this submission.  While the paper I have linked is very recent, the baseline algorithms considered in the paper are published works from 2017 onwards.\nComparing to these algorithms would make this paper much stronger.\n\n1. There are a lot of unsubstantiated claims. Some include:\n    1. Modifying the loss in equation 3 gives a new distribution $\\widehat{\\gamma}$ defined underneath equation 3.\n    1. At the bottom of page 5, the authors remark \"In our method, on the contrary, we are still looking for the discriminator maximizing the Integral Probability Metric (Müller, 1997) in equation 3, linked to optimal transport.\" .\nIn equation 3, the authors take the dual optimization problem for estimating the Wasserstein distance, and modify it. With this modification, optimizing equation 3 will not give the Wasserstein distance any more.\n\n## Minor:\n1. Should there be a quanitifer in eqn (3) restricting $D_\\alpha$ to $1-$Lipschitz models?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to train a rejection sampler in the latent space of a GAN to learn disconnected data manifolds. Reviewers raised concerns about some theoretical aspects of the method as well as about the lack of larger scale datasets (ImageNet) in the experiments. Authors responded to these concerns but some of them still remain (including $\hat{\gamma}(z)$ not guaranteed to be a probability distribution and lack of more convincing experiments). I still think the work is promising, and encourage the authors to revise and resubmit the paper addressing these points highlighted by the reviewers. 
Decision: Reject