MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining
Authors: Ahmed Imtiaz Humayun
Randall Balestriero
Richard Baraniuk
Venue: ICLR 2022 Poster (2022)
Keywords: Deep Generative Networks
Uniform Sampling
Fairness
Data Augmentation

Abstract: Deep Generative Networks (DGNs) are extensively employed in Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and their variants to approximate the data manifold, and data distribution on that manifold. However, training samples are often obtained based on preferences, costs, or convenience producing artifacts in the empirical data distribution e.g. the large fraction of smiling faces in the CelebA dataset or the large fraction of dark-haired individuals in FFHQ). {\em These inconsistencies will be reproduced when sampling from the trained DGN, which has far-reaching potential implications for fairness, data augmentation, anomaly detection, domain adaptation, and beyond.} In response, we develop a differential geometry based sampler -coined MaGNET- that, given any trained DGN, produces samples that are uniformly distributed on the learned manifold. We prove theoretically and empirically that our technique produces a uniform distribution on the manifold regardless of the training set distribution. We perform a range of experiments on various datasets and DGNs. One of them considers the state-of-the-art StyleGAN2 trained on FFHQ dataset, where uniform sampling via MaGNET increases distribution precision \& recall by 4.12\% \& 3.01\% and decreases gender bias by 41.2\%, without requiring labels or retraining.

Published as a conference paper at ICLR 2022
MaGNET: UNIFORM SAMPLING FROM DEEP GENERA- TIVE NETWORK MANIFOLDS WITHOUT RETRAINING
Ahmed Imtiaz Humayun Rice University
Randall Balestriero
Richard Baraniuk Rice University
Rice University
mt a edu
randal 1ba rstrrrrrggm com
i chb@r ce.edu
ABSTRACT
Deep Generative Networks (DGNs) are extensively employed in Generative Ad- versarial Networks (GANs). Variational Autoencoders (VAEs), and their variants approximate the data manifold and distribution. However, training samples often distributed non-uniformly on the manifold, due the cost convenience collection. For example, the CelebA dataset contains large fraction of smiling faces. These inconsistencies will be reproduced when sampling from the trained DGN, which is not always preferred, e.g., for fairness or data augmentation. In response, we develop MaGNET, novel and theoretically motivated latent space sampler for any pre-trained DGN that produces samples uniformly distributed on the learned manifold. We perform a range of experiments on several datasets and DGNs, e.g.. for the state-of-the-ard StyleGAN2 trained on the FFHQ dataset, uni- form sampling via MaGNET increases distribution precision by 4.1% 1% and recall by 3.0% and decreases gender bias by 41.2%, without requiring labels or retrain- ing. Since uniform sample distribution does not imply uniform semantic distri- bution, we also explore how semantic attributes of generated samples vary under MaGNET sampling. Colab and codes bit. 1y/magnet -ampeettssaannnnn
Figure 1: Random batches of StyleGAN2 (1) 0.5) samples with 1024 x 1024 resolution, generated using standard sampling (left), uniform sampling via MaGNET the learned pixel-space manifold (middle), and uniform sampling on style-space manifold (right) of the same model. MaGNET sampling yields higher number of young faces, better gender balance, and greater background/accesson variation, without the need for labels retraining. Images are sorted by gender-age and color coded red-green (female-male) according Microsoft Cognitive API predictions. Larger batches of images and attribute distributions furnished in Appendix E.

---

Published as a conference paper at ICLR 2022
INTRODUCTION
Deep Generative Networks (DGNs) are Deep Networks (DNs) trained to learn latent representations of datasets; such frameworks include Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), Variational Autoencoders (VAEs) (Kingma & Welling, 2013), flow-based models such as NICE (Dinh et al., 2014), and their variants (Dziugaite et al., 2015; Zhao et al., 2016; Durugkar et al., 2017: Arjovsky et al., 2017; Mao et al., 2017; Yang al., 2019; Fabius & van Amersfoort, 2014; van den Oord et al., 2017; Higgins et al.. 2017; Tomczak & Welling, 2017: Davidson et al., 2018; Dinh et al., 2017; Grathwohl et al., 2018; 80 Kingma & Dhariwal, 2018). A common assumption that we will carry through our study is that the datasets of interest are not uniformly distributed in their ambient space, but rather are concentrated on, or around, manifolds of lower intrinsic dimension, e.g., the manifold of natural images (Peyré, 2009). Different DGN training methods have been developed and refined to obtain models that approximate as closely as possible the training set distribution. This becomes an Achilles heel when the training set, regardless of its size, is not representative of the true data distribution, i.e., when the training samples have been curated based on cost or availability that result in implicit/explici biases. In such scenarios, while the training samples will lic on the true data manifold, the density distribution of the training set will be different from the natural distribution of the data.
Deploying DGN trained with biased data distribution can be catastrophic, in particular, when employed for tasks such as data augmentation (Sandfort al., 2019), controlled data generation for exploration/interrretaiionn (Thirumuruganathan et al., 2020), or estimation of statistical quantities of the data geometry, such as the Lipschitz constant of the data manifold (Gulrajani et al., 2017: Scaman & Virmaux, 2018). Biased data generation from DGNs due to skewed training distributions also raises serious concerns in terms of fair machine learning (Hwang et al., 2020; Tan et al., 2020).
While ensuring semantic uniformity in samples is an extremely challenging task, we take one step in the more reachable goal of controlling the DGN sampling distribution to be uniform in terms of the sample distribution on the data munifold. To that end, we propose MaGNET (for Maximum entropy Generative NETwork, a simple and efficient modification to any DGN that adapts its latent space distribution to provably produce samples uniformly distributed on the learned DGN manifold. Importantly, MaGNET can be employed on any pre pre-trained and differentiable DGN regardless of its training setting, reducing the requirement of fine-tuning or retraining of the DGN. This is crucial as many models, such as BigGAN (Brock et al., 2019) and StyleGAN (Karras et al., 2020), have significant computational and energy requirements for training. A plug-and-play method is thus greatly preferred to ease deployment in any already built/trained deep learning pipeline.
Previously, there has been rigorous work on DGNs aimed at improving the training stability of mod- els, deriving thcoretical approximation results, understanding the role of the DGN architectures, and numerical approximations to speed-up training and deployment of trained models (Mao al., 2017: Chen et al., 2018; Arjovsky & Bottou; Miyato et al., 2018; Xu & Durrett, 2018; Liu et al., 2017; Zhang et al., 2017: Biau et al., 2018: Li et al., 2017: Kodali al., 2017: Roy et al., 2018; Andrés-Tees Terés-TTrré&& & Lió, 2019; Chen et al., 2018; Balestriero et al., 2020; Tomczak & Welling, 2016; Berg et al., 2018). Existing methods (Metz al., 2016; Tanaka, 2019; Che et al., 2020) also try to tackle mode dropping by improving approximation of the data distribution, but this can potentially increase the bias learned implicitly by the DGN. We are the first to consider the task of providing uniform sampling on the DGN underlying manifold, which has far-reaching consequences, rang- ing from producing DGNs stable to data curation and capable of handling inconsistencies such as repeated samples in the training set. We provide first-of-its-kind provable uniform sampling on the data manifold that can be used to speed up estimation of various geometric quantities, such as estimation of the Lipschitz constant.
MaGNET applies to any (pretrained) DGN architecture (GAN, VAE, NF, etc.) using continuous piecewise affine (CPA) nonlinearities, such as the (leaky) ReLU; smooth nonlinearities can be dealt with via first-order Taylor approximation argument. Our main contributions are as follows: [C1] We characterize the transformation incurred by a density distribution when composed with a CPA mapping (Sec. 3.1) and derive the analytical sampling strategy that enables one to obtain uniform distribution on a manifold that is continuous and piecewise affine (Sec 3.2).
[C2] We observe that current DGNs produce CPA manifolds, and demonstrate how to leverage
[CI] to produce uniform sampling on the manifold of any DGN (Sec. 3.2).
[C3] We conduct several carefully controlled experiments that validate the importance of uniform
2

---

Published as a conference paper at ICLR 2022
sampling and showcase the performance of MaGNET pretrained modelss such as BigGAN (Brock et al., 2019), StyleGAN2 (Karras et al., 2020), progGAN (Karras et al., 2017), and NVAE (Vahdat & Kautz, 2020), e.g., we show that MaGNET can be used to increase distribution precision by 4% and recall by 3% for StyleGAN2 and decrease gender bias by 41 %, without requiring labels or retraining (Sec. 4.2 and Sec. 4.3).
Plug and play codes for various models are made available at our Git repository. Computation and software details are provided Appendix H. with the proofs of our results in Appendix Discussion of the settings in which MaGNET is desirable and possible limitations is provided in Sec. 5.
2 BACKGROUND
Continuous Piecewise Affine (CPA) Mappings. A rich class of functions emerges from piecewise polynomials: spline operators. In short, given a partition Ω of a domain R$, a spline of order l is a mapping defined by a polynomial of order l on each region E Ω with continuity constraints on the entire domain for the derivatives of order Ok... we only define this case for concreteness. An affine spline produces its output via
k- - 1. As we will focus affine splines (k 1),
S(z)
Σ (A = + bw) ( WEN
with input and Aw. b.. the per-region slope and offset parameters respectively, with the key constraint that the entire mapping is continuous over the domain S e (R Spline operators and especially affine spline operators have been extensively used in function approximation theory (Cheney & Light, 2009), optimal control (Egerstedt & Martin, 2009), statistics (Fantuzzi et al., 2002), and related fields.
Deep Generative Networks. A deep generative network (DGN) is a (nonlinear) operator Ge with parameters θ mapping latent input R$ to observation intermediate layer mappings. The only assumption we require for our study is that the nonlinearities present in the DGN are CPA, as is the with (leaky-)ReLU, absolute value, max-pooling. For smooth nonlinearities, our results hold from a first-order Taylor approximation argument. Precise definitions of DGN operators can be found in Goodfellow et al. (2016). We will omit θ from the Ge operator for conciseness unless needed. It is also common to refer to as the latent representation, and as the generated/observe data, e.g., a time-series or image. One property of DGNs that employ nonlinearities such as (leaky-)ReLU, max max-pooling, and the likes, is that the entire input-output mapping becomes a CPA spline.
RD by composing L
CONTINUOUS PIECEWISE AFFINE MAPPING OF A PROBABILITY DENSITY
In this section, we study the properties of a probability density that is transformed by a CPA map- ping. Our goal is to derive the produced density and characterize its properties, such how the per-region affine mappings in Eq. impact the density concentration. We present some key results that serve as the backbone of our core result in the next section: how sample uniformly from the manifold generated by DGNs.
3.1 DENSITY ON THE GENERATED MANIFOLD
Consider an affine spline operator S (Eq. 1) going from space of dimension S to space of dimension D with D > S. The image of this mapping CPA manifold of dimension at most S, the exact dimension is determined by the rank of the per-region slope matrices. Formally, the span, or the image, of S is given by
Im(S) 2 (S(=) ZER)
U Aff(w; An, bw.)
with Aff(w: Aw, = (A,z+b, : E w the affine transformation of region w by the per-region parameters Aw, bw.
From Eq. 2 .e observe that the generated manifold surface is made of regions that are the affine transformations of the latent space partition regions w E Q based on the coordinate change induced by A., and the shift induced by b We visualize this in Fig. 2 for toy spline operator with
3

---

Published as a conference paper at ICLR 2022
Figure 2: Visual depiction Eq. with a toy affine spline mapping S It² + R. Left: latent space parti- tion f made of different regions shown with different colors and with bound- aries shown in black. Right: affine spline image Im((S) which is a contin- uous piecewise affine surface composed of the latent space regions affinely trans- formed by the per-region affine mappings (Eq. 1). The per-region colors maintain correspondence from the left to the right.
2- 2-dimensional latent space and 3-dimensional ambient/output space. In the remainder of our study we will denote for conciseness (ww) A Aff(w; Aw. bw).
When the imput space is equipped with density distribution, then this density is transformed by the mapping S and "lives" on the surface of the CPA manifold generated by S. Given a distribution pN over the latent space, we can explicitly compute the output distribution after the application of S, which leads to an intuitive result exploiting the CPA property of the generator. For this result, we require that the operator S be bijective between its domain and range. That is, each slope matrix A Yu E Ω should be full rank, and there should not be any folding of the generated CPA surface w' We now derive the key result of this section that characterizes the density distribution on the manifold.
that intersects with itself, i.e., S(w) ((w') o 0 >
Lemma 1. The volume of a region w E Ω denoted by p(w) is related to the volume of the affinely transformed region S(w) by
((S(w))
(3)
m(w)
where ((S(w)) is the measure on the S-dimensional affine subspace spanned by the CPA mapping. (Proof in Appendix I.I.)
Theorem 1. The probability density ps(z) generated by S for latent space distribution pa is given by,
P= (ATA.)) A (x -b,)
ps(
(+ES(w))-
Vdet(ATA.)
WER
(Proof in Appendix 1.2.)
In words. the distribution obtained in the output space naturally corresponds to a piecewise affine transformation of the original latent space distribution, weighted by the change in volume of the per-region mappings from Eq. 3. For Gaussian and Uniform distributed pz, we use the above results to obtain the analytical form of the density covering the output manifold, we have provided proof and differential entropy derivations Appendix B.
3.2 MAKING THE DENSITY ON THE MANIFOLD UNIFORM
The goal of this section is build on Thm. provide novel latent space distribution such that the density distribution lying on the generated manifold is uniform.
One important point that we highlight is that having a Uniform density distribution in the latent space of the affine spline not sufficient have a uniform density lying on the manifold; would be if (ot(ATAA mapping equal for all w). This is evident from Appendix (Eq. 8). Therefore we propose here a novel latent space sampler with the purpose that once it is transformed by the affine spline (i.e.. the DGN) a distribution becomes uniform on the DGN manifold. We focus here on the technical aspect and defer precise motivations behind such construction to the next section that deals with practical applications. To obtain K samples uniformly distributed on the output manifold of S using the proposed MaGNET procedure:
det(A m/ Awr) Ai # w' (in words, the change in volume of the per region
4

---

Published as a conference paper at ICLR 2022
1. For K MaGNET samples, sample N > K (as large as possible) iid latent vectors with U being the latent space domain of S (z1,..... kN), with 2, ~ U(U).
2. Compute the per-region slope matrices A, (s( (Eq. 1), and the change of volume V det A where A; = A (zlew)
scalar (oI..... NN)
(ddt(AA
3. Sample (with replacement) K latent vectors (Z1,.... zx) with probability x (a1...., oN)
We discuss possible choices of N and K in Appendix where we observe that even for state-of- tre-art models like StyleGAN2, N =250,000 is sufficient to provide a stable approximation of the true latent space target distribution. In practice, A, is simply obtained through backpropagation, since it is the Jacobian matrix of the DGN at 2,, as in A; = (z:)
The above Monte-Carlo approximation does not require knowledge of the DGN spline partition S nor the per-region slope matrices (Eq. 1). Those are computed on-demand as z, are sampled. The above procedure produces uniform samples on the manifold learned by a DGN regardless of how has been trained.
MAGNET: MAXIMUM ENTROPY GENERATIVE NETWORK SAMPLING
The goal of this section is to first bridge current DGNs with affine splines & leverage Thm. 1 and Sec. 3.2 to effectively produce uniform samples on the manifold of DGNs such as BigGAN, Style- GAN. We build this affine spline DGN bridge and motivate for uniform sampling in Sec. 4.1 and present various experiments across architectures in Sec. 4.2, 4.3, and 4.4.
4.1
UNIFORM SAMPLING ON THE DEEP GENERATIVE NETWORK MANIFOLD
We provided in Sec. 3.2 thorough study of affine splines and how those mappings transform a given input distribution. This now takes high relevance as per the following remark.
Remark 1. Any DGN (or part of it) that employs CPA nonlinearities (as in Sec. 2) is itself a CPA; that is, the input-output mapping can be expressed as in (Eq. 1).
This observation in the context of classifier DNs goes back to Montufar et al. (2014) and has been further studied in Unser (2018); Balestriero & Baraniuk (2018). We also shall emphasize that op- erators such Batch-Normalization (loffe & Szegedy, 2015) are not continuous piecewise affine during training but become affine operators during evaluation time. For completeness, we also pro- vide that analytical form of the per-region affine mappings Aw. bw of Eq. for the DGNs featured Appendix C. The key for our method is thus to combine the above with the results from Sec. 3.2 to obtain the following statement.
Theorem 2. Consider a training set sampled from a manifold M and a (trained) CPA DGN S. As long as M C Im))), sampling from S as per Sec. 3.2 produces uniform samples on M, regardless of the training set sampling. (Proof in 1.4.)
This result follows by leveraging the analytical DGN distribution from Thm. and by replacing Ps with the proposed one, leading to (T xx Ewga 11 (wES()) which is uniform on the DGN manifold. By using the above one can take any (trained) DGN and produce uniform samples on the learned underlying manifold. Hence, our solution produces a generative process that becomes invariant to the training set distribution. While this provides a theoretical guarantee for uniform sampling, it also highlights the main limitation of MaGNET: the uniform samples will lie on CPA manifold. That is, unless the true manifold M is also continuous, MaGNET will occasionally introduce abnormal samples that correspond to sampling from the regions of discontinuity of M. We will see in the following sections how even on high-quality image datasets, MaGNET produces very few abnormal samples, reason being that for complicated data manifolds. state-of- th--the-rt art DGNs are often built with (class) conditioning. In such cases, the above continuity assumption on M. lessens only to within-class continuity assumption which is much more realistic. Sampling uniformly on the DGN manifold has many important applications that are deferred to the following sections.
4.2 QUANTITATIVE VALIDATION: -BALI. CONCENTRATION, GMM LIKELIHOOD AND FRÉCHET INCEPTION DISTANCE
We now report three controlled experiments to validate the applicability of the theoretical results from Sec. 3.2 for the MaGNET sampling procedure.
5

---

Published as a conference paper at ICLR 2022
standard sampling
MaGNET 2 sampling
samples with 1) generated neighbors
104
==100
10a
1000
2000
2000
5000
5500
10a
10a
Number (log) of training
N= 10000
N 10000
20²
10a
10
10a
100
100
150
200
250
300
350
400
12
number of samples within e-ball radius,
Figure 3: Distribution of the number of MNIST training samples with neighbors generated within an e-ball radius. N samples are generated using standard sampling and MaGNET sampling using NVAE model. Here taken to be the average nearest neighbor distance for the training samples. For Vanilla NVAE, heavier tails indicative of larger density variations on the manifold as N is increased, whereas for MaGNET the shorter tails are indicative of fewer variations neighborhood density, i.c., uniform generation on the data manifold. 10,000 MNIST samples are used for comparison, for additional see Fig. 13 and Fig. 14 in the Appendix.
First, we consider MNIST and assume that the entire data manifold is approximately covered by the training samples. Regardless of the training data distribution the manifold (uniform not), we can pick a datum at random, count how many generated samples (17) are within this datum c- ball neighborhood and repeat this process for 10,000 training samples. If does not vary between training datum, then it strongly indicates that the generated samples are uniformly distributed on the manifold covered by the training data. We perform this experiment using a pretrained state-of- the-art variational autoencoder NVAE (Vahdat & Kautz, 2020) to compare between standard and MaGNET sampling with the number of generated samples N ranging from 1,000 10,000. We report the distribution of in Fig. 3. Again, uniform sampling is equivalent to having the same v for all training samples, i.c., a Dirac distribution in the reported histograms. We can see that MaGNET sampling approaches that distribution while standard sampling has a heavy-tail r distribution, i.e., the generated digits have different concentrations at different parts of the data manifold. Another quantitative measure consists of fitting Gaussian Mixture Model (GMM) with varying number of clusters, on the generated data, and comparing the likelihood obtained for standard and MaGNET sampling. As we know that in both cases the samples lie the same manifold and domain, the sam- pling with lower likelihood will correspond to the one for which samples are spread more uniformly on the manifold. We report this in Fig. 4, further confirming the ability of MaGNET uniformly spread samples. We report the generated samples in Appendix E. Lastly, we compare the Fréchet Inception Distance (FID) (Heusel et al., 2017) between 50,000 generated samples and 70,000 training samples for StyleGAN2 (config-f) trained on FFHQ Since uniform sampling via MaGNET increases the diversity of generated samples, we see that MaGNET sampling improves the FID for truncation (Karras et al., 2019), v {.4, 55, 66, .7) by 2.76 points on average (see Ap- pendix F). While for the aforementioned v MaGNET samples alone provide an improved FID, for introduce increasing amount of MaGNET samples for FID caleulation. We observe in Fig. 4 that by progressively increasing the percentage of MaGNET samples, we are able exceed the state-of-the--rt FID of 2.74 for StyleGAN2 (17 1), reaching FID of 2.66 with ~ 4% of MaGNET samples.
produce
higher v values,
4.3
QUALITATIVE VALIDATION: HIGH-DIMENSIONAL STATE-OF-THE-ARE IMAGE GENERATION
We now turn into the qualitative evaluation of MaGNET sampling, to do so we propose extensive experiments on various of-te-of-the-art image DGNs. We also remind the reader that in all cases, standard and MaGNET sampling are performed on the same DGN (same weights) as discussed in Sec. 3.2.
2-Dimensional Dataset and CoLOred-MNIST. The first set of controlled experiments is designed such that the training set contains inconsistencies while it is known that the original distribution is uniform on the data manifold. Such inconsistencies can occur in real datasets due to challenges related dataset compilation. We provide illustrative examples in Fig. 5, where demonstrate

---

Published as a conference paper at ICLR 2022
2.85
4000
Max
Mean
Min
log-likelihood
2,80
3000
2.75
2.70
1000
StyleGANZ FFFHQ
100 200 300 400
0%
2% 4% 6%
8%
10%
percentage of MaGNET samples
of clusters
Figure 4: (Left) Average log- log-likelihood and baoddahdwi for runs of GMM trained on 10,000 samples using standard sampling (blue) and MaGNET sampling (black) for an NVAE trained on MNIST. The higher log-likelihood (given the same number of clusters) in the standard sampling case demonstrates an increased concentration around few modes, as opposed MaGNET. (Right) FID (L) of StyleGAN2 (config-f) trained on FFHQ for 50,000 generated samples and runs. With an incrcasing percentage of uniformly generated samples increase diversity, MaGNET reaches state-of-the-art FID of 2.66 achieved at 4.1% mixture.
Samples from original distribution
Artificially bissed
Standard DGN Sampling
Mat:NET sampling
08
0.0
0.6
0.4
02
0.2
0.0
0.0
0.0
0.2
0.8
0.2
0.5
0.6
0.8
0.0
0.2
0.1
0.0
0.0
1.0
0.2
04
Figure 5: From left to right, samples from a toy 2D distribution with triangular support, biased samples ob- tained for training GAN, standard sampling showing biased distribution learned by the GAN, and MaGNET sampling recovering uniformly distributed samples on the support of the true distribution. Note that the same number of samples are obtained for both standard and MaGNET sampling.
0.007
Hue (color) distribution of samples ob- tained from standard and MaGNET sampling from trained BVAE model colored 8 digits from MNIST. The original datasct purposefully favored cyan color- ing to represent training set inconsistency. MaGNET BVAE can approximately generate uniformly colored MNIST samples. The density drop around red repre- sent regions where training density was low, therefore the DGN manifold approximation was incorrect and uniform sampling can not recover those samples.
Figure 6:
Vanilla BVAE MaGNET BVAE
0.006
0.005
0.004
0.003
0.002
0.001
0.000
0.0m
D.5mT
1.0m 1.5m
2.
that unless uniform sampling is employed, the trained DGN reproduces the inconsistencies present in the training set, expected. This toy dataset visualization validates our method from Sec. 3.2. Going further, we take the MNIST dataset (in this case, only digit 8 samples) and apply imbalanced coloring based on the hue distribution provided in Appendix Fig. 12, which favors cyan color. We train a VAE DGN VAE) on that cyan-inclined dataset, and present in Fig. 6 the hue distributions for samples obtained via standard sampling and MaGNET sampling. We observe that MaGNET cor- rects the hue distribution back to uniformity.
Uniform Face Generation: CelebA-HQ and Flickr-Faces-HH with progGAN and Style- GAN2. Our first experiment concerns sampling from the StyleGAN2 (Karras et al., 2020) model pretrained on the Flickr-Faces-Hit (FFHQ) dataset. StyleGAN2 has two DGNs, one that maps to an intermediate latent space, termed style-space and another DGN that maps style-space vectors to the pixels-space (output of StyleGAN2). Implementation details are contained in Appendix H. We focus here on applying MaGNET onto the entire StyleGAN2 model (the composition of both DGNs), in Sec. 4.4 we discuss applying MaGNET to the style-space DGN. Fig. 1 we provide random samples from the same StyleGAN2 model obtained via standard and MaGNET sampling. Upon qualitative evaluation, it can be seen that the samples obtained via MaGNET (MaGNET Style- GAN2) have significantly larger variety of age distribution, background variations and wearable
7

---

Published as a conference paper at ICLR 2022
Figure 7: Random hatch of samples generated from BigGAN (left) and MaGNET BigGAN (right), condi- tioned on the Samoyed class of ImageNet. While BigGAN samples contain homogeneous postures. MaGNET samples represent the true span the data manifold learned by BigGAN.
Lioschitz constant estimation
Estimator Std.
4.75
4.50
Standard
4.25
MaGNET
4.00
0.04
Samoles tlog)
Number of Samples
Figure 8: Lipschitz constant estimation using standard DGN sampling and MaGNET sampling (left). Each line represents the mean over 200 Monte-Carlo runs. As expected, estimation of such statistics from samples converges faster when employed uniformly distributed samples. The standard deviation of the Monte-Carlo estimations are also provided (right), where it is clear that the uniform sampling via MaGNET reaches smaller standard deviation earlier steps i.e., is faster to converge. This is key application MaGNET: speeding- up convergence of statistical estimation of quantities, such as the Lipschitz constant in this case. The x-axis represents the number of samples used for estimation in log-scale.
accessories compared to standard sampling.
For experiments with the CelebA-HQ dataset, adopt the Progressively Growing GAN (prog- GAN) (Karras et al., 2017), trained on 1024 x 1024 resolution images. In Fig. 9 we provide random samples from standard and MaGNET sampling, the latter portraying more qualitative diversity. We see that uniform manifold sampling via MaGNET recovers samples containing a number of at- tributes that are generally underrepresented in the samples generated by vanilla progGAN. (See Ap- pendix E for larger batches and attribute distributions.) Note that uniform sampling not only recovers under-represented groups e.g., age < 30, head-wear, and bald hair, also increases the presence of neutral emotion and black hair. One interesting observation is that MaGNET also increases the number samples off the true data manifold (images that are not celebrity faces), exposing regions where the manifold is not well approximated by progGAN.
Conditionally Uniform Generation: ImageNet with BigGAN. We present experiments on the state-of-the the- conditional generative model BigGAN (Brock et al., 2019) using MaGNET sam- pling. In Fig. 7 we provide random samples from standard and MaGNET sampling. More ex- periments different classes presented in Appendix E. We see that uniform sampling the learned data manifold yields a large span of backgrounds and textures, including humans, while standard sampling produces examples closer the modes of the training dataset. This is quite un- derstandable considering that ImageNet was curated using a large number of images scraped from the internet. MaGNET therefore could be used for data exploration/mode interpretation and also as a diagnostic tool to assess the quality of the learned manifold posteriori of training.
4.4 APPLICATION: MONTE- MONTE-CARLO ESTIMATION AND ATTRIBUTE REBALANCING
We conclude this section with two more practical aspects of MaGNET.
Monte-Carlo Estimator. The first to speed-up (in terms of number of re- quired samples) basic Monte-Carlo estimation of arbitrary topological quantities of the generated manifold. Suppose that one's goal is to estimate the Lipschitz constant of a DGN. A direct estima- tion method would use the known bound given by max, [Js(z)]| F (Wood & Zhang, 1996). This estimation can be done by repeatedly sampling latent vectors z from the same distribution that one used for training DGN. However, this implies that the produced samples will not be uniformly dis- tributed on the manifold in turn leading to slower convergence of the estimator. Instead, we propose to use MaGNET, and report our findings in Fig. 8. More domains of application, where MaGNET
8

---

Published as a conference paper at ICLR 2022
Figure 9: Random hatch of samples generated from vanilla progGAN (left) and MaGNET ProgGAN (right). Some samples contain fusion of attributes which are not frequently represented in the data distribution, e.g. female w/ facial hair, male w/ makeup, etc. MaGNET also generates more samples from regions of the manifold less approximated by the DGN. Larger batches in Appendix E.
can be used for estimator variance reduction, can be found in Baggenstoss (2017).
Style-space MaGNET sampling rebalances attributes. When thinking of uniform sampling manifold, it might seem natural to expect fairness i.e., fair representation of different attributes such as equal representation of gender, ethnicity, hair color, etc. However, this is not necessarily true in all cases. In fact, it is trivial to show that each attribute category will be equally represented iff their support on the true data manifold is of equal volume (integrated with respect the data man- ifold). Fortunately, as we mentioned in Sec. 4.3, architectures such as StyleGAN2 have explicitly built a style-space, which a latent space in which attributes are organized along affine subspaces occupying similar volumes (Karras et al., 2019) i.e., MaGNET applied on the style-space DGN should improve fairness. By applying MaGNET sampling on the stylc-space, we are able to reduce gender bias from 67-33% (female-male) in standard StyleGAN2 to 60-40%. This simple result demonstrates the importance of our proposed sampling and how it can be used to increase fairness for DGNs trained on biased training sets. MaGNET in the style-space also yields improvements in terms of recall and precision (Sajjadi et al., 2018). Given a reference distribution (e.g.. FFHQ dataset) and learned distribution, precision measures the fidelity of generated samples while recall measures diversity. We compare the metrics for face images generated via ~ ((0, where 1.5, 1.5, 2. 2 ~ 2, 2). and MaGNET sampling on style-space. For 70k samples generated for each case, MaGNET sampling obtains a recall and precision of (0.822, 0.92) with a 4. 12% rela- tive increase in recall and 3.01% % relative increase in precision compared to the other latent sampling methods (metrics were averaged for 10 seeds).
a E 0.5,
CONCLUSIONS, LIMITATIONS AND FUTURE WORK
We have demonstrated how the affine spline formulation of DGN provides new theoretical results to provably provide uniform sampling on the manifold learned by a DGN. This allows becoming robust to possibly incorrect training set distributions that any DGN would learn to replicate after its train- ing. We have reported on several experiments using pretrained state-of-the-an generative models and demonstrated that uniform sampling on the manifold offers many benefits from data exploration to statistical estimation. Beyond the sole goal of uniform sampling on manifold, MaGNET opens many avenues, yet MaGNET is not a "one size fits all" solution.
When not to sample uniformly. We can identify the general cases in which one should not em- ploy uniform sampling of the DGN manifold. The first case occurs whenever the true manifold is known to be discontinuous and one needs to avoid sampling in those regions of discontinuities. In fact, in the discontinuous case, DGN training will adapt to put zero (or near zero) density in those discontinuous regions preventing standard sampling to reach those regions (Balestriero et al., 2020). However, MaGNET will reverse this process and introduce samples back in those regions. The second case occurs if one aims to produce samples from the same distribution as the training set distribution (assuming training of the DGN was successful). In this scenario, one should use the same latent distribution evaluation time as the one used during training.
Future work. Currently, there are two main limitations of our MaGNET sampling strategy. The first one lies in the assumption that the trained DGN able to learn a good enough approximation of the true underlying data manifold. In future work, we plan to explore how MaGNET can be used to test such an assumption. One potential direction is as follows; train a DGN using several sub-sampled datasets (similar to bootstrap methods) and then study if MaGNET samples populate manifolds that all coincide between the different DGNs. If training is successful, then those sam- pled manifolds should coincide. Another direction could be understanding the relationship between uniform sampling and uniform attribute representation. We demonstrated how uniform sampling in the style-space of StyleGAN2 ensures that relationship by construction.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Reproducible data and
code for various experiments is made available at bit sit.ly/ magnet- sampling Computation and software details provided in Appendix H, with the proofs of our results in Appendix I.
ACKNOWLEDGEMENT
This work supported by NSF grants CCF-1911094, IIS- and IIS- ONR grants N00014- 18- 1---188--1111 12571, 10001--20-125344 and MURI N00014-20-1-27877 AFOSR grant FA9550-22- 1-0060; and Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2044,
REFERENCES
Helena Andrés-Ters and Pietro Lió. Perturbation theory approach to study the latent space degen- eracy of variational autoencoders. arXiv preprint 190.v0/097.002600 2019.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint 100.::170..46
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein
gan. arXiv preprint
arXiv:1701.07875. 2017.
Paul M Baggenstoss..
Uniform manifold sampling (ums): Sampling the maximum entropy pdf. IEEE Trans. Signal Processing, 65(9):2455-244 2017.
Randall Balestriero and Richard Baraniuk.
A spline theory of deep networks. In Proc. ICML,
volume 80, pp. 374-383, Jul. 2018
Randall Balestriero and Richard Baraniuk. Mad max: Affine spline insights into deep learning. Proceedings of the IEEE, 2020.
Randall Balestriero, Romain Cosentino, Behnaam Aazhang, and Richard Baraniuk. The geometry of deep networks: Power diagram subdivision. In Proc. NeurIPS, pp. 15806-15815. 2019.
Randall Balestriero, Sébastien Paris, and Richard Baraniuk. Analytical probability distributions and exact expeccation-maxiiizatiio for deep generative networks. Proc. NeurlPS, volume 33, 2020.
Matan Ben-Yosef and Daphna Weinshall. Gaussian mixture generative adversarial networks for diverse datasets, and the unsupervised clustering of images. arXiv preprint arXiv: 10356, 2018.
Rianne van den Berg, Leonard Hasenclever, Jakub M Tomezak, and Max Welling. Sylvester nor- malizing flows for variational inference. arXiv preprint arXiv:1803.0550 05540, 2018.
Gérard Biau, Benoît Cadre, Maxime Sangnier, and Ugo Tanielian. Some theoretical properties of gans. arXiv preprint arXiv: (003:080007//11 2018.
Leo Breiman, William Meisel, and Edward Purcell. Variable kernel estimates of multivariate densi- ties. Technometrics, 19(2):135-144, 1977.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. 2019.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly energy-based model and you should use discriminator driven latent sampling. arXiv preprint wxXiv:2003.06060 2020.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. volume 31, pp. 2610-2620, 2018.
Elliott Ward Cheney and William Allan Light. A course in approximation theory, volume 101.
American Mathematical Soc., 2009.
10

---

Published as a conference paper at ICLR 2022
Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyperspheri- cal variational auto-encoders. arXiv preprint 1rXi::8804000891,, 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nyp. In Proc.
ICLR, 2017.
Ishan Durugkar, lan Gemp, and Sridhar Mahadevan. Generative multi-adversarial networks. In Proc. ICLR, 2017.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint 1xXiv:5505.0333 2015.
Magnus Egerstedt and Clyde Martin. Control theoretic splinee: optimal control, statistics, and path planning. Princeton University Press, 2009.
Otto Fabius and Joost van Amersfoort. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.658// 2014.
Cesare Fantuzzi, Silvio Simani, Sergio Beghelli, and Riccardo Rovatti. Identification piecewise affine models in noisy environment. International Journal of Control, 75(18))1472-1485 2002.
I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning, volume MIT Press, 2016.
J Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Proc. NIPS, pp. 2672-2680, 2014.
Will Grathwohl, Ricky TQ Chen, Jesse Betterncourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.011367 2018.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. arXiv preprint arXiv: 000000000 2017.
L. A. Hannah and D. B. Dunson. Multivariate convex regression with adaptive partitioning. Mach. Learn. Res., ((1):3261-329)) 2013.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proc. NeurlPS, volume 30, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Veta-vae: Learning basic visual concepts with constrained variational framework. In Proc. ICLR, 2017.
Sunhee Hwang, Sungho Park, Dohyung Kim, Mirae Do, and Hyeran Byun. Fairfacegan: Fairness- aware facial image-to-inenn to-iage translation. In Proc. BMVC, pp. 1309-1320, 2020.
Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167. 2015.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint 170.:11710.10196 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. CVPR, pp. 4401-4410 2019.
Tero Karras, Samuli Laine, Miia Aittala, Janne Hellsten. Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proc. CVPR, pp. 8110-8119, 2020.
Diederík Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Proc. NewrIPS, pp. 10215-10224, 2018.

---

Published as a conference paper at ICLR 2022
Diederik P Kingma and Max Welling. 1rXiv:13/2.614, 2013.
Auto-encoding variational
bayes.
arXiv preprint
Naveen Kodali, Jacob Abemnetyy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv: 07211..... 2017.
Jerry Li, Aleksander Madry, John Peebles, and Ludwig Schmidt. Towards understanding the dy-
namics of generative adversarial networks, arXiv preprint arXiv:1706. 09884, 2017.
Shuang Liu, Olivier Bousquet, and Kamalika Chaudhuri. Approximation and convergence proper-
ties of generative adversarial learning. In Proc. NeurIPS, pp. 5545-5553, 2017.
Alessandro Magnani and Stephen P Boyd. Convex piecewise-linear fitting. Optim. Eng., 10(1): 2009.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proc. ICCV, pp. 2794-2802, 2017.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. arXiv preprint arXiv: 16/1.02163 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXhv:1802.05957. 2018.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Proc. NeurlPS, pp. 2924-2932, 2014.
Gabriel Peyré. Manifold models for signals and images. Computer vision and image understanding, 113(2):249-660, 2009.
Murray Rosenblatt. Remarks on some nonparametric estimates of density function. The Annals of Mathematical Statistics, pp. 832-837, 1956.
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments vector quantized autoencoders. arXiv preprint wrxv::805.111063 2018.
Walter Rudin. Real and Complex Analysis. Tata McGraw-hill education, 2006.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucie, Olivier Bousquet, and Sylvain Gelly. Assessing 2018.
generative models via precision and recall. arXiv preprint 1806.
Veit Sandfort, Ke Yan, Perry Pickhardt, and Ronald M Summers. Data augmentation using gener- ative adversarial networks (cyclegan) improve generalizability in et segmentation tasks. Scien- rific reports, 9(1):1-9, 2019.
Kevin Scaman and Aladin Virmaux. Lipschitz regularity of deep neural networks: analysis and efficient estimation. arXiv preprint 1095. 2018.
Michael Spivak. Calculus manifolds: modern approach to classical theorems of advanced calculus. CRC press, 2018
Shuhan Tan, Yujun Shen, and Bolei Zhou. Improving the faimess of deep generative models without retraining, arXiv preprint arXiv:2012.04842. 2020.
Akinori Tanaka. Discriminator optimal transport. arXiv preprint arXiv: 2019.
Saravanan Thirumuruganathan. Shohedul Hasan, Nick Koudas, and Gautam Das. Approximate query processing for data exploration using deep generative models. In Proc. ICDE, pp. 1309- 1320, 2020.
Jakub M Tomezak and Max Welling. Improving variational auto-encoders using householder flow. arXiv preprint arXiv: 1.06630 2016.
Jakub M Tomezak and Max Welling. Vae with vampprior. arXiv preprint arXiv: 1705.07 2017.
12

---

Published as a conference paper at ICLR 2022
Michael Unser. A representer theorem for deep neural networks. arXiv preprint arXiv: 180.90180,.09210, 2018.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. In Proc. NeurlPS, 2020.
Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Proc. NeurlPS, pp. 6306-6315, 2017.
Shuning Wang and Xusheng Sun. Generalization of hinging hyperplanes. IEEE Trans. Information Theory, (4111):4425-444 4431, 2005.
GR Wood and BP Zhang. Estimation of the lipschitz constant of a function. J. Global Optim., 8(1): 91-103, 1996.
Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. arXiv preprint arXiv:1808.100
2018.
Dingdong Yang. Seunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee. Diversity- sensitive conditional generative adversarial networks. arXiv preprint arXiv: Seee11991..99444 2019.
Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu, and Xiaodong He. On the discrimination- generalization tradeoff in gans. arXiv preprint arXiv: 2017.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network.
arXiv preprint wfXvv1669.033266 2016.
13

---

Published as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
The following appendices support the main paper and are organized follows: Appendix A ad- dresses some details on the background of continuous piecewise affine deep networks, that were omitted in the main paper. Appendices and H provide additional figures and training details for all the experiments that were studied in the main text, and Appendix provides the proofs of all the theoretical results. Due filesize constraints, the high-quality batch of samples are provided in the supplementary files.
A BACKGROUND ON CONTINUOUS PIECEWISE AFFINE DEEP NETWORKS
A max-affine spline operator (MASO) concatenates independent max-affine spline (MAS) functions, with each MAS formed from the point-wise maximum It affine mappings (Magnani & Boyd, 2009; Hannah & Dunson, 2013). For our purpose each MASO will express DN layer and is thus an operator producing a D dimensional vector from a D dimensional vector and is formally given by
MASO( (A,, b.) r=l...,
A + by,
(5)
where A, E D' are the slopes and b, E D are the offset/bias parameters and the max- imum is taken coordinate-wis wise. For example, layer comprising fully connected operator with weights W' and biases De followed by ReLU activation operator corresponds to a (single) MASO with R = 2, A W A2 0, bi bc, b2 O. Note that MASO continuous piecewise- affine (CPA) operator (Wang & Sun, 2005).
The key background result for this paper that the layers of DNs constructed from piecewise affine operators (e.g., convolution, ReLU, and max-pooling) are MASOs (Balestriero & Baraniuk, 2018);
3R No; ,,,,N,,,, ",A, 6, s.t. MASO(u; (A,, b, b (')) E D
(6)
making the entire DGN composition of MASOs.
The CPA spline interpretation enabled from a MASO formulation DGNs provides a powerful global geometric interpretation of the network mapping based on partition of its input space RS into polyhedral regions and a per-region affine transformation producing the network output. The partition regions are built up the layers via subdivision process and closely related to Voronoi and power diagrams (Balestriero et al., 2019). We now propose to greatly extend such insights carefully characterize and understand DGNs well provide theoretical justifications to various observed behaviors e.g. mode collapse.
B UNIFORM AND GAUSSIAN MANIFOLD DISTRIBUTIONS
We now demonstrate the use of the above result by considering practical examples for which we able to gain insights into the DGN data modeling and generation. We consider the two most common cases: (i) the latent distribution is set as ~ N(0, 1) and (ii) the latent distribution is set as U(0, 1) (on the hypercube of dimension S). We obtain the following result by direct application of Thm. 1.
Corollary 1. The generated density distribution ps of the Gaussian and uniform densities are given by
H(0- )w) (A?)" (AL) TT (w--bw
P2(
(mEE((w))
(Gaussian)
((T) det det(AAAA.)
WED
Vol(U)-1
pS(x)
(A (xES(w))
(Uniform)
(8)
WER
The two above formulae provide precise description of the produced density given that the latent space density is Gaussian Uniform. In the Gaussian case, the per-region slope matrices act upon
14

---

Published as a conference paper at ICLR 2022
the Eg distance by rescaling it from the coordinates of A,, and the per-region offset parameters b are the mean against which the input a compared against. the Uniform case, the change of volume (recall Eq. 3) is the only quantity that impacts the produced density. We will heavily rely this observation for the next section where we study how to produce a uniform sampling onto the CPA manifold of affine spline.
We derive the analytical form for the case of Gaussian and Uniform latent distribution in Ap- pendix L.3, From the analytical derivation of the generator density distribution, we obtain its dif- ferential entropy.
Corollary 2. The difserential Shannon entropy of the output distributtio pc of the DGN is given by E(pe) = [(px)) Ewes P(z E w) log( log(ydett det (A A.
As a result, the differential entropy of the output distribution PG corresponds to the differential entropy of the latent distribution Ps plus convex combination of the per-region volume changes. It is thus possible to optimize the latent distribution ps to better fit the target distribution entropy as in Ben- Yosef Ben-Yosef & Weinshall (2018) and whenever the prior distribution fixed, any gap between the latent and output distribution entropy imply the need for high change in volumes between and G(w)).
C PER-REGION AFFINE MAPPINGS
For completeness we also provide that analytical form of the per-region affine mappings
A = II
diag
(ō (w)) w
(9)
1=0
T-1
T-8-1
bw =b
Σ
II 1=0
diag (ol-((w) wh-t diag (ō' 0e
(10)
where (2) the pointwise derivative of the activation function of layer based on its input w + bc which note function of directly. For precise definitions of those oper- ators see Balestriero & Baraniuk (2020). The diag operator simply puts the given vector into a diagonal square matrix. For convolutional layers (or else) can simply replace the corresponding W' with the correct slope matrix parametrization as discussed in Sec. Notice that since the em- ployed activation functions of, Vl E (1,.... L. L} are piecewise affine, their derivative is piecewise constant, in particular with values [d](=)]& E {a,1) 1} with a 0 for ReLU, a -1 for absolute value, and in general with for Leaky-ReLU for l E D').
D NUMBER OF SAMPLES AND UNIFORMITY
Exact uniformity reached when the Monte Carlo samples have covered each region of the DGN partition boundary. For large state-of-the-an models this condition requires sampling on the order of millions. However, we conducted an experiment to see how the number of samples really impacted the uniformity of the generated manifold as follows. We compute precision and recall metries [4] for StyleGAN2 with K generated samples obtained from N Monte Carlo samples based on our sampling strategy by varying N. We use K 5000 and N ranging from 10,000 to 500,000. Based on the metrics, we identify that increasing beyond = 250, 000 no longer impacts the metrics, showing that this number of monte carlo samples is enough converge (approximately) to the uniform sampling in that case; see Fig. 10.
We report here the Jacobian computation times for Tensorflow 2.5 with CUDA 11 and Cudnn 8 on an NVIDIA Titan RTX GPU, For StyleGAN2 pixel space, 5.03s/it; StyleGAN2 style-space, 1.12s/it; BigGAN 5.95s/it; ProgGAN 3.02s/it. For NVAE on Torch 1.6 it takes 20.3s/it. Singular value calculation for StyleGAN2 pixel space takes 0.005s/it, StyleGAN2 style space 0.008s/it, BigGAN 0.001 ProgGAN 0.004s/it and NVAE 0.02sfit on NumPy.
E ADDITIONAL FIGURES
This section contains samples from our proposed methods, more samples along with attribute data and pretrained weights available at our project link.
15

---

Published as a conference paper at ICLR 2022
1.0
0.8
Precision
0.6
Figure 10:
Evolution of the precision/recall curves for varying number of samples N form the monte-carlo sampling against the number of samples K 5k for StyleGAN2.
0.4
10000
100000
0.2
250000
500000
0.0
0.00
0.25
0.75
1.00
Recall
1.0
0.8
Precision
0.6
Figure 11: Precision- curves for K 70k: samples from Vanilla StyleGAN2 and MaGNET StyleGAN2
0.4
0.2
Vanilla MaGNET
0.0
0.00
0.25
0.50
0.75
1.00
Recall
0.007
Original
0.006
Figure 12: Depiction of the imbalance hue distri- bution applied to color the MNIST digits.
0.001
0.000
0.0m
0.5m
1.0m
1.5m
2.0n
Number of training samples having
(X-axis) samples at most e aaway
101
N 1000
104
N 1000
2000
2000
5000
N 5000
10²
10a
- 10000
-- 10000
10a
10a
10a
10a
10
12
10
12
number of samples within 0.5c-ball radius
Figure 13: Reprise of Fig. 3. Vanilla NVAE Left, MaGNET NVAE Right
16

---

Published as a conference paper at ICLR 2022
Number of training samples having
(X-axis) samples at most away
1000
10a
1000
2000
2000
N 5000
N 5000
107
10a
N 10000
N 10000
10*
10a
109
200
400
600
800
1000
1200
1400
1600
100
200
300
400
500
number of samples within 1.5e-hall radius
Figure 14: Reprise of Fig. 3. Vanilla NVAE Left, MaGNET NVAE Right
17

---

Published as a conference paper at ICLR 2022
Figure 15: Random batches of 245 samples from a StyleGAN2 trained on FFHQ, generated via standard sampling (left), MaGNET sampling in the pixel-space (middle) and MaGNET sampling in the style-space.
18

---

Published as a conference paper at ICLR 2022
Figure 16: Random Samples from vanilla progGAN (left) and MaGNET ProgGAN (right) trained on the Ce dataset. Samples are sorted by gender & age and color coded by gender as visually predicted by the Microsoft Cognitive API. Samples not recognized by the API are color coded as white the bottom.
19

---

Published as a conference paper at ICLR 2022
Figure 17: Random Samples from vanilla BigGAN (left) and MaGNET BigGAN (right) from the Collie class of Imagenet.
20

---

Published as a conference paper at ICLR 2022
Figure 18: Random Samples from vanilla BigGAN (left) and MaGNET BigGAN (right) from the Siamese class of Imagenet.
21

---

Published as a conference paper at ICLR 2022
Figure 19: Random Samples from vanilla BigGAN (left) and MaGNET BigGAN (right) from the Tabby class of Imagenet.
22

---

Published as a conference paper at ICLR 2022
Gender
Gasses
Virig MeCxET-sty
vond
20
MaCNET
Vegnen Style
MOONET plee
NOONE iMee!
Magaer Waet
desay (PSt
1211
1a
lemakd
made
Ansetton
Age
Emotion
Accessories
Nalve
Ton
Wanla
Nagrat-Sryl
(bg)
22
MacNET
Vagnet-dog Style
Mayrett
0.20
0.04
0.25
Irjer
Figure 20: Facial Attributes of 5000 StyleGAN2 samples using vanilla sampling, MaGNET style- space sampling and MaGNET pixel-sper space sampling. We that MaGNET style-space increases uniformity in gender and age distributions whereas MaGNET pixel-space yields more variations in physical attributes and accessories.
Gonder
Hair
Glasses
Veniee
Nenila
107
0.2
Dendly Ixgi
22
0.3
12
ReotingDlwees
Age
Emotion
Accessoris
0.00
Mike Magret
Dng cottereee Zegh
von f1
107
vall
0.04
0.89
contenge
orgport
Iepponena
andreg
0.00
10
20
30
30
60
75
mask
Figure 21: Facial Attributes of 5000 ProgGAN samples using standard sampling and MaGNET sampling.
23

---

Published as a conference paper at ICLR 2022
3
3
Z
3
9
Z
4
5
6
5
G
S
3
3
O
R
5
L
E
9
S
C
3
H
5
9
S
&
G
5
5
O
&
T
6
P
2
R
E
6
2
J
R
&
5
O
3
C
O 2
2
A
7 7 6
D
&
9
6
3
D E 5 D
P
5 2
3
4
S
3
2
5
F 4
T
R
9
5
6
O
3
9
7
7
4
&
5
7
&
S
&
7
3
Z
6 2
O
T 4
6
6
6
2
O
D
D
6
R
6
4
Y G Q
S
O
O
6
7
8
6
&
S
O
O
3
Y
2
G
9
7
2
O
7
7
2
8
9
6
6
O
O
9
9
S
3
O
2
O
3
6
3
2
9
6
4
2
O
O
3
3
2
Z
6
2
7
4
7
?
Figure 22: Random Samples from vanilla NVAE (left) and MaGNET NVAE (right) trained on the MNIST dataset.
Figure 23: Random Samples from vanilla NVAE (left) and MaGNET NVAE (right) trained on the
CIFAR dataset.
24

---

Published as a conference paper at ICLR 2022
F ADDITIONAL TABLES
Table 1: FID obtained by StyleGAN2 (config-f) trained FFHQ using standard and MaGNET sampling (pixel-space) for varying degrees of truncation (t)) (Left) FID score obtained for 50,000 samples generated by mixture of MaGNET and standard sampling. MaGNET samples can be used to increase diversity of the model, resulting in better FID than current state-of-the-art. (Right) FID)
score for varying truncation without mixing.
Truncation
Sampling
Percent Mixture
Truncation
Sampling Method
FID (L)
FID (L)
Method
v
Standard
2.74
Standard
58.33
.5
MaGNET
4.1%
2.66
MaGNET
54.47
Standard
5.05
Standard
83.84
9
.4
MaGNET
20%
4.29
MaGNET
82.41
Standard
10.94
Standard
112.08
.8
.3
MaGNET 33%
8.57
MaGNET
112.89
Standard
21.34
Standard
142.27
.7
.2
MaGNET 100%
19,41
MaGNET
144.93
Standard
36.98
Standard
176.20
.6
.1
MaGNET
100%
33.19
MaGNET
178.75
G ALGORITTS
Algorithm 1: MaGNET Sampling as described Sec. 3.2
Input:
Latent space domain, U; Generator G; Number of regions to sample N; Number of samples K;
Output: MaGNET Samples, {x;)
Initialize, 2, S + 0
= 1 do 2 ~~ -(U::
1,...,
Get Slope Matrix, A Ja(z);
Get volume scalar at z, a. / (AT))) Z.append(2): S.append
end
for n 1,... K do
Categorical(prob softmax(8)).
< 2[1]
end
Algorithm 2: Online Rejection Sampling algorithm for MaGNET
Input: Latent space domain, U; Generator G; N change of volume scalars {01, o2...., NN):
Output: MaGNET Sample,
while True do
Sample U(T);
Sample a ~ U[0, 1):
Get Slope Matrix, A = J=(z);
Get volume scalar at 2, a= / det((AA)) then
if
x G(z);
break;
end end
25

---

Published as a conference paper at ICLR 2022
H ARCHITECTURE, HARDWARE AND IMPLEMENTATION DETAILS
All the experiments were run on a Quadro RTX 8000 GPU, which has 48 GB of high-speed GDDR6 memory and 576 Tensor cores. For the software details we refer the reader to the provided codebase. In short, we employed TF2 (2.4 the time of writing), all the usual Python scientific libraries such as NumPy and Py Porch.. We employed the official repositories of the various models we employed with official pre-trained weights. As a note, most of the architectures can not run on GPUs with less equal to 12 GB of memory.
For StyleGAN2, use the official config-e provided in the GitHub StyleGAN2 repo', unless spec- ified. We use the recommended default of t/ = 0.5 as the interpolating stylespace truncation, to ensure generation quality of faces for the qualitative experiments. For BigGAN we the BigGAN- deep architecture with no truncation, available on TFHub". We also use the NVAE and ProgGAN4 models and weights from their respective official implementations. For the Jacobian determinant calculation of images w.r.t latents, we first use random orthogonal matrix to project generated im- ages into a lower dimensional space, calculate the Jacobian of the projection w.r.t the latents and calculate the singular values of the jacobian to estimate the volume scalar. We use a projection of 256 dimensions for StyleGAN2-pixel. ProgGAN and BigGAN, and 128 dimensions for NVAE. To estimate the volume scalar we use the top 30, 20, 15 singular values for StyleGAN2 MaGNET pixel, ProgGAN and BigGAN; 40 for StyleGAN2 MaGNET style, and 30 for NVAE.
I
PROOFS
1.1 PROOF OF LEMMA
Proof. In the special case of affine transform of the coordinate given by the matrix A E R DxD the well known result from demonstrates that the change of volume is given by det( (see Theorem 7.26 in Rudin (2006)). However in our case the mapping is a rectangular matrix span an affine subspace in the ambient space " making det( not defined.
First, we shall note that in the case of a Riemannian manifold (as is the produced surface from the per-region affine mapping) the volume form used in the usual change of variable formula can be defined via the square root of the determinant of the metric tensor. Now, for surface of intrin- sic dimension n embedded in Euclidean space of dimension m (in our case, the per-region affine mapping produces an affine subspace) parametrized by the mapping M RR + Rm (in our case this mapping is simply the affine mapping (z) = zu22 + b. for each region) the metric tensor is given by g DM DM with D) the jacobian/diiferenniaa operator (in our case g AT A.,, for each region). This result is also known as Sard's theorem (Spivak, 2018). We thus obtain that the change of volume from the region w to the affine subspace G(w) is given by /det(AATA) A) which can also be written as follows with USV the syd-decomposition of the matrix A:
V det(AA (T A) ( (de(((VSUUT))UVV
(det(VSSSVTT)
del(
II (A
is0,, 200
leading to
J JAMCLAAD dx ddet(A A A)
/ dz
http:://htbb.eeeeeeeemmmmmaaaaad "htpps//gituu..oo//Waaaaaa growing of gans
26

---

Published as a conference paper at ICLR 2022
1.2 PROOF OF THEOREM 1
Proof. We will be doing the change of variables = (A b.) A ( bwb, also notice that (a) A+. First, we know that PG(=)( E w) (= G (w)) (a-1(w) Pz (a(z))d which well defined based on our full rank assumptions. We then proceed by
Pa(x w) = -
/
Pz Pz (z=) det((J (x)) dx)
wel who
Σ /
Pz Pz (et(( d da
wEl wii
= Σ
(x))
II oi
wEl whi
eo;; (At)>0
Σ/
(x)
II (A,())) dx Etape
WEN who
(A.))=)
Σl WE
(x)))
dx
/del A Aw)
Let now prove the Etape step by proving that (,(At (o,(A)) - where we lighten notations as A. and USVT is the svd-decoonositionn of A:
A (A" AT
=SV"""""VV" (USVT)
-V"""s"""'v'""
-V(S"S)-'S") UT
oi(At)= (o:(A))-
with the above it is direct to see that
V det((At) A
follows
Vdet(A A1
V dde((AA))
II a( II (A,)
i:
-1
II (:(Au)) 1:00
1
/del A.)
which gives the desired result.
I.3 PROOF OF COROLLARY
Proof. We now demonstrate the use of Thm. where we consider that the latent distribution is set as x ~ N(0, 1). We obtain that
Pc E w) = Σ
det (et do
WEn whw
Σ
LnEG(W) (21) (2mt (ee ||l 2
WE whw
Σ
ba)
bw) ((A) (x dx
1REG(W) (22) AT
WEn whw
= Σ
leEG(w) (22) deee(etTAAA
dx
WEQ wNw
27

---

Published as a conference paper at ICLR 2022
giving the desired result that is reminiscent of Kernel Density Estimation (KDE) (Rosenblatt, 1956) and in particular adaptive KDE (Breiman et al., 1977), where partitioning the data manifold is performed each cell (w in our case) different kernel parameters are used.
Proof. We now turn into the uniform latent distribution case on a bounded domain in the DGN input space. By employing again Thm. 1, the given formula can directly obtain that the output density given by
Ewer Leew det(A Vol(U)
PG(x)
(11)
1.4 PROOF OF THM.
successful training, then regardless of the actual distribution pa, the DGN will learn the correct underlying manifold, and learn the best approximation to as possible onto this manifold. Now, applying MaGNET sampling i.c. Sec. 3.2 equivalent to sampling from a distri- bution Pms such that after DGN mapping, that distribution is uniform on the learned manifold (see Thm. 1). As we assumed that regardless of Pr the DGN approximates correctly the true manifold, and as we then adapt the sampling distribution pm= to always obtain uniform sampling on that man- ifold, we see that this final sampling becomes invariant upon the data distribution (on the manifold) leading to the desired result.
Proof. As
28


Reviews:
[{"review_id": "NQaIozH2RXc", "reviewer": "Reviewer_5nSv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a sampling method that aims at providing uniform samples from the latent space for deep generative models. Providing uniform samples from the latent space is very important, but the manuscript requires more quantitative results to support their claims of uniform samples:\n\nMajor concerns: \n1. How does the MaGNET sampling affect the quality of the generated images? Common GANs literature provides some quantitative evaluation metrics (inception score, FID score, KID score) as a justification for their proposed methods, but this submission does not provide any justification using the prevalent evaluation metrics. The visual quality of some generated images using the MaGNET method is poorer than using the original GAN alone (Figure 7 &9) In addition, in addition, the provided samples of the generated images (Figure 15 - 20) are not clear enough to provide a good justification for their visual quality.\n\n2. It is hard to tell whether the proposed method truly improves the uniformity of the sampled data (in Figure 22):\n (1) (Gender) it seems that the MaGNET-style reduces gender bias but the MaGNET-pixel increases gender bias\n (2) (Hair) due to the low clarity of this subfigure, it is hard to draw any conclusion\n (3) (Glasses) it seems that the MaGNET-pixel improves the occurrence of sunglasses and the improvement of MaGNET-style is limited\n (4) (Age) it seems that both methods have some improvement in age\n (5) (Emotion) the improvement of this subfigure is hard to tell (both improves in fear, but decrease in disgust)\n (6) (Accessaries) it seems that both methods (MaGNET-pixel and MaGNET-style) can improve the occurrence of headwear\nBut all the results are qualitative, the reviewer would like to see some quantitative results, such as how close the new distribution sampled using the MaGNET methods is to the uniform distribution, under the same categories (Gender, Hair, Glasses, Age, Emotion, Accessaries) provided by the authors comparing to the previous methods.\n\nMinor comments:\n1. What is the per-region slope matrices $\\mathbf{A}_i = \\mathbf{J_S}(\\mathbf{z}_i)$ in the sampling procedure of the MaGNET?\n2. The resolution of some figures (Figure 5, 8, 15 - 20) are too low, especially in Figure 15 - 20 where the quality of generated samples is hard to verify.\n\n", "summary_of_the_paper": "This paper proposes a sampling method called MaGNET for generative models which aims at sampling data from the latent distribution of the images uniformly.", "summary_of_the_review": "This paper does not provide enough experimental results (both qualitatively and quantitively) to support their claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BvIpRXcoe6", "reviewer": "Reviewer_orST", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(a) Given any trained DGN, the paper gives a novel theoretical method to produce samples that are uniformly distributed on the learned manifold, regardless of the training set distribution. The approach is novel and solves the problem elegantly. \n(b) It proves the proposed method for a mild assmption that DGN only comprises continuous piecewise affine (CPA) non-linearities, such as ReLu, absolute value, max-pooling. \n(c) It gives convincing experiments on synthetic dataset showing that regardless of the training set distribution their MaGNET approach produces samples that are uniformly distributed.\n\nWeakness: The paper needs improvement in writing. \n(a) In section 3.2, the notation J_s(z_i) is used without explaining it. \"compute the per-region slope matrices A_i = J_s(z_i)\". Please define the notation and explain how to compute the slope matrices. \n(b) A high level proof of the main theorem 2, in the main paper will help the reader understand the theorem better. \n(c) the x-axis values of the two plots in Figure 3 are different by order of 100, it does not seem correct. ", "summary_of_the_paper": "This paper concerns with uniform sampling from deep generative networks such as GANs and VAEs. The training samples of DGNs are often biased as they are obatined based on preferences, costs, or convenience that leads to DGNs producing biased examples. This paper gives a gemoetry based sampler MaGNET, that given any trained DGN, produces samples that are uniformly distributed on the learned manifold. It theoretically proves, and empirically shows that the MaGNET produces a uniform distrbution on the manifold regardless of the training set distribution. The theoretical proofs require that the DGNs only comprise continuous piecewise affine (CPA) non-linearities, such as ReLu, absolute value, max-pooling. The three main contributions of the paper are as following:\n(a) It characterizes the transformation incurred by a density distribution when composed with a CPA mapping.\n(b) It derives an analytical sampling strategy that allows to obtain a uniform distribution on a manifold that is continuous and piecewise affine.\n(c) It provides multiple numerical experiments validating the gains of their proposed method MaGNET.\n", "summary_of_the_review": "The paper provides a novel provable method to produce samples that are uniformly distributed on the learned manifold, regardless of the training set distribution. The method is also well proven empirically through numerous experiments. The proposed method help address the problem fairness in samples produced from DGNs trained on not-so-well-representative training datasets.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "321izR4qZhm", "reviewer": "Reviewer_TJkz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Applying the change of variables formula to augment sampling from DGNs is a novel idea. Moreover, the method is interesting and theoretically well-motivated. Finally, the sampling algorithm itself is very straightforward.\n\nHowever, I take issue with the framing of the algorithm in the abstract and introduction. Namely, the authors use the colloquial understanding of “uniform” side-by-side with the differential geometric / measure theoretic understanding of “uniform”. For example, in the abstract, the authors state that 1) many generative models today are trained on non-uniform data, which has “potential implications for fairness, data augmentation, anomaly detection, domain adaptation, and beyond,” and 2) their algorithm “produces a uniform distribution on the manifold regardless of the training set distribution”. This creates the false impression that the present technique is capable of neutralizing the negative \"implications\" on fairness, data augmentation, etc etc. This juxtaposition may imply parity between the two definitions of \"uniform\" to the inattentive reader. While the authors do emphasize the difference between the term in these two contexts much later in the paper, I feel that it is not appropriate for the abstract to mislead in this manner. Again, I only have this issue with the framing of the technique, not the technique itself.\n\nWIth regard to the uniform sampling property of MaGNET, I have two concerns about the practicality of the method. \n1. The authors have touched upon this, but uniform sampling from the data manifold does not imply uniform sampling of attributes. This is exacerbated when the model has not fully learned the manifold. Therefore MaGNET’s sampling is only as “uniform” as the DGN and the data manifold itself is.\n2. Since the DGN can only be trained on the training data distribution, sample quality will vary across the true data manifold. Namely, sample quality will likely correlate with density w.r.t. the training data distribution. Therefore, I imagine that sampling uniformly will reduce sample quality overall. This seems to be corroborated by qualitative comparison of original v. MaGNET samples in the paper figures.\n\nComputationally, the authors demonstrate in Appendix D that sampling with $N$ past 250k does not affect the Precision-Recall metric, but I could not find what $N$ is in the experiments shown. And since each image sample requires computing the Jacobian of the DGN w.r.t. its input, I wonder what is the approximate computation time needed to sample $N=250,000$ times for each of the models.\n", "summary_of_the_paper": "The authors propose a uniform sampling technique for deep generative networks (DGNs) inspired by the probabilistic change of variables formula. The technique works with any already trained DGN and does not involve any further training. (Though it does require back propagation w.r.t. the input $x$.) In essence, the algorithm works by drawing many samples N >> K from the DGN, then sampling from these $N$ samples with probability inversely related to their pushforward density (as computed by the change-of-variables formula).\n", "summary_of_the_review": "This method itself is novel and interesting, and warrants acceptance into the conference. However, the current wording of the title and abstract can be misleading, and should be edited to remove confusion.\n\nPros:\n+ Theoretically motivated\n+ Algorithmically simple\n\nCons:\n- Seems computationally expensive\n- Oversells the capabilities of the technique in the abstract, namely: mathematically uniform is implied to mean semantically uniform\n- Needs proofreading", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a simple method for uniform sampling from generative manifold using change of variables formula. The method works by first sampling a much larger number of samples (N) from uniform distribution in the latent space and then does sampling by replacement (using probability proportional to change in volume) to generate a smaller number of final samples (k << N) that are seen as approximately sampled from a uniform distribution from the generative manifold. 

Reviewers had some questions/concerns about the confusing language in the abstract and introduction around the use of the term "uniform" which the authors have addressed satisfactorily. Authors have also provided results on quality (FID metric) of the generated samples as asked by the reviewers. 

While the proposed method is rather simple, has high computational cost, and novelty is marginal (as noted by two of the reviewers), reviewers agree it is above the acceptance bar.
Decision: Accept (Poster)