On the optimal precision of GANs
Authors: Thibaut Issenhuth
Ugo Tanielian
Jeremie Mary
David Picard
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep learning theory
GANs
Deep generative modelling

Abstract: Generative adversarial networks (GANs) are known to face model misspecification when learning disconnected distributions. Indeed, continuous mapping from a unimodal latent distribution to a disconnected one is impossible, so GANs necessarily generate samples outside of the support of the target distribution. In this paper, we make the connection between the performance of GANs and their latent space configuration. In particular, we raise the following question: what is the latent space partition that minimizes the measure of out-of-manifold samples? Building on a recent result of geometric measure theory, we prove there exist optimal GANs when the dimension of the latent space is larger than the number of modes. In particular, we show that these generators structure their latent space as a `simplicial cluster' - a Voronoi partition where centers are equally distant.  We derive both an upper and a lower bound on the optimal precision of GANs learning disconnected manifolds. Interestingly, these two bounds have the same order of decrease: $\sqrt{\log m}$, $m$ being the number of modes. Finally, we perform several experiments to exhibit the geometry of the latent space and experimentally show that GANs have a geometry with similar properties to the theoretical one.

Under review as conference paper at ICLR 2023
ON THE OPTIMAL PRECISION OF GANS
A Anonymous authors
Paper under double-blind review
ABSTRACT
Generative adversarial networks (GANs) known face model misspecification when learning disconnected distributions. Indeed, continuous mapping from a uni- modal latent distribution to a disconnected one is impossible, so GANs necessarily generate samples outside of the support of the target distribution. In this paper, we make the connection between the performance of GANs and their latent space configuration. In particular, we raise the following question: what is the latent space partition that minimizes the measure of out-of-manifold samples? Building on a recent result of geometric measure theory, we prove sufficient condition for GANs to be optimal when the dimension of the latent space is larger than the number of modes. In particular, their latent space as a 'simplicial cluster' a Voronoi partition where centers are equally distant. We derive both an upper and lower bound on the optimal preci- sion of GANs learning disconnected manifolds. Interestingly, these two bounds have the same order of decrease: vlogm, m being the number of modes. Finally, we perform several experiments to exhibit the geometry of the latent space and experimentally show that GANs have geometry with similar properties the theoretical one.
show the optimality of generators that structure
INTRODUCTION
GANs (Goodfellow et al., 2014), a family of deep generative models, have shown great capacities to generate photorealistic images (Karras et al., 2019). State-of-the-ar models, like StyleGAN (Karras et al., 2019) or TransforneGGAN (Jiang al., 2021), show empirical benefits from relying on overparametrized networks with high-dimensional latent spaces. Besides, manipulating the latent representation of GAN also helpful for diverse tasks such as image editing (Shen al., 2020; Wu et al., 2021) or unsupervised learning of image segmentation (Abdal et al., 2021). However, there is still a poor theoretical understanding of how GANs organize their latent space. We argue that this is a crucial step in better apprehending the behavior of GANs.
To better understand GANs, the setting of disconnected distributions learning is enlightening. Ex- perimental and theoretical works (Khayatkhoei et al., 2018; Tanielian et al., 2020) have shown fundamental limitation of GANs when dealing with such distributions. Since the distribution modeled by GANs is connected, some areas of GANs' support are necessarily mapped outside the true data distribution. When covering modes of a disconnected distribution, GANs try to minimize the measure of the generated distribution lying outside the true modes (e,g. the purple area on the right of Figure 1). In other words, GANs need to minimize the measure of the borders between the modes in the latent space. Considering a Gaussian latent space, minimizing this measure is closely linked the field of Gaussian isoperimetric inequalities (Ledoux, 1996). This field aims at deriving the partitions that decompose Gaussian space with a minimal Gaussian Gaussian-weighted perimeter. We argue that the optimal partitions derived in Gaussian isoperimetric inequalities cast light on the structure of the latent space of GANs. Most notably, recent result (Milman and Neeman, 2022) shows that, as long as the number of components space are such that m < d + 1, the optimal partition a 'simplicial cluster': a Voronoi diagram obtained from the cells of equidistant points (see left of Figure for = 3 and d 3).
in the partition and the number of dimensions d of the Gaussian
In this paper, we apply this result to the field of GANs and show, both experimentally and theoretically, generated samples. We
that GANs with 'simplicial cluster' latent space minimize draw the connection between GANs and Gaussian isoperimetric inequalities by using the precision metric (Sajjadi et al., 2018: Kynkäänniemi al., 2019), which quantifies the portion of generated

---

Under review as conference paper at ICLR 2023
2
4
Figure I: Illustration of the ability of GANs find optimal configuration in the latent space. On the left, the propeller shape is partition of 3D Gaussian space with the smallest Gaussian Gaussian-weighted perimeter (Figure from Heilman et (2013)). On the right, we show the 3D Gaussian latent space of GAN trained on three classes of MNIST. Each area colored in blue, green, or red maps samples in one of the three classes. In purple, we observe the samples that are classified with low confidence. We see that the partition reached by the GAN (right) is close to optimality (left), since the latent space partition is similar to the intersection of the propeller on a sphere.
points that support the target distribution. We show that GANs with latent space organized as simplicial cluster reach optimal precision levels and derive both an upper and a lower bound on the precision of such GANs. Experimentally, we show that the GANs with higher performances tend to organize their latent space as simplicial clusters. To summarize, our contributions are the following:
We are the first to import the latest results from Gaussian isoperimetric inequalities by (Milman and Neeman, 2022) to the study and understanding of GANs. We use it show that the latent space structure has major implications on the precision of GANs.
We derive a new theoretical analysis, stating both an upper bound and a lower bound on the precision of GANs. We show that GANs with latent space organized as simplicial cluster have an optimal precision whose lower bound decrease in the same order as the upper bound: Vlogm, where m is the number of modes.
Experimentally, show that GANs tend to structure their latent space 'simplicial clusters' on image datasets. First, we explore two properties of the latent space: linear separability and convexity of classes. Then, we play with the latent space dimension and highlight how it impacts the performance of GANs. Finally, we show that overparametrization helps approaching the optimal structure and improving GANs performance.
RELATED WORK
2.1 NOTATION
Data. We assume that the target distribution 14, is defined on Euclidean space Rº (potentially high-dimensionnl space), equipped with the Huclidean norm I|- We denote Sy- the support of this unknown distribution μu^ In practice, however, we only have access finite collection of i..dd observations XI,..., distributed according to ur. Thus, for the remainder of the article, we let pm be the empirical measure based on XI,...,Xm.
Generative model. We consider 9 the set of L-Lipschitz continuous functions from the latent space Rd to the high-dimensional space RD_ Each generator aims at producing realistic samples, The latent space distribution defined Rd is supposed to be Gaussian and is noted y. Thus, each candidate distribution is the push forward between y and a generator and is noted as Gty.
This Lipschitzness assumption on 91 is reasonable since Virmaux and Scaman (2018) has shown and presented an algorithm that upper-bounde the Lipschitz constant of any deep neural network. In practice, enforce the Lipschitzness of generator functions by clipping the neural networks'
2

---

Under review as conference paper at ICLR 2023
parameters (Arjovsky et al., 2017). penalizing the discriminative functions' gradient (Gulrajani et al., 2017; Kodali et al., 2017: Wei al., 2018; Zhou al., 2019), or penalizing the spectral norms (Miyato et al., 2018). Note that some large-scale generators such as SAGAN (Zhang et al., 2019) and BigGAN (Brock et al., 2019) also make use of spectral normalization for the generator.
2.2 GANS AND DISCONNECTED DISTRIBUTIONS
A significant flaw of GANs (Goodfellow al., 2014) is their difficulty in learning multi-modal distributions. This phenomenon has been analyzed by Khayatkhoei et al. (2018) and Tanielian et al. (2020). The problem comes from this fundamental trade-off: GANs can either cover all modes and generate out-of-manifee samples or generate only good quality samples and neglect some modes (mode collapse). Some methods have proposed ways train disconnected generators (Khayatkhoei et al., 2018: Gurumurthy et al., 2017), but with little benefits compared to single overparametrized generators with rejection mechanisms (Taniclian et al., 2020; Azadi al., 2018).
Empirically, different works give intuition on the latent space structure of GANs. Karras et al. (2019) show that binary attributes are linearly separable in the Gaussian latent space and even better separated in an intermediate latent space. Shen et al. (2020) stress that face attributes are separated by hyperplanes, and edit images only by moving in the latent space orthogonally to these hyperplanes. Arvanitidis et al. (2018) 8) and Chen et al. (2018g 8a) view the latent space of generative models with Riemannian perspective and define metric tensor using the generator's 's Jacobian to find the shortest paths on the data manifold.
However, these findings might not be sufficient for a clear understanding of the required geometry of the latent space. For instance, Karras al. (2019) while Sauer et al. (2022) argue that the optimal latent space dimension is close to the intrinsic dimension of images (RR44 for ImtgeNet). Tanielian et al. (2020) stress the relevance of this problem by showing that the precision of GANs can converge to O when the number of modes or the distance between them increases. In this paper, make step towards better understanding of the behavior of GANs and expose an optimal latent space configuration when the number of modes m and the dimension of the latent space d are such that < d I.
very large latent space dimension (R-112,
2.3 EVALUATING GANS
When learning disconnected manifolds, Sajjadi al. (2018) illustrated the need for new measures that simultaneously evaluate the quality (Precision), and the diversity (Recall) of the generated samples. Kynkäänniemi et al. (2019) highlighted an important drawhack of this PR metrie: it cannot correctly interpret situations when large numbers of samples are packed together. They propose an Improved PR metric based on the non-parametric estimation of manifolds to correct this.
Improved PR metric: Informally, for generator G, precision (ag) quantifies the proportion of generated samples that can be approximated with true samples, while recall (Bc) measures the proportion of true samples that can be approximated with generated ones. Applying this to GANs, using the target distribution μ and modeled distribution Gły, the Improved PR metric was shown, by Tanielian et al. (2020, Theorem 1), to be asymptotically equivalent to:
of - 0G G)r(S and BC
+ Bo μ (Saty).
where Sy- denotes the support of μr" More recently, Naeem et al. (2020) have shown that the Improved PR metric (Kynkäänniemi et al., 2019) is not robust to outlier samples of both the target and the generated distribution. To correct this and fix the overestimation of the manifold around real outliers, Naeem et al. (2020) propose the Density/Coverage metric.
Density/Covvrage:
Instead of counting how many fake samples belong to real sample neighbor- hood, density counts how many real sample neighborhoods contain generated sample. On the other hand, coverage counts the number of real sample neighborhoods that contain at least one fake sample.
In the next section, will use the notion of precision and recall defined in (1). Using this definition allows us to circumvent the non- parameeeiii estimators involved in the existing metrics (Kynkääänniemi et al., 2019; Naeem al., 2020).
3

---

Under review as conference paper at ICLR 2023
DETERMINING OPTIMAL PRECISION IN GANS
We want to better understand the latent space of GANs and stress which GANs have the highest precision under specific constraints. GANs are push-forward distributions of unimodal (connected) Gaussian distribution y and continuous function parameterized by G. Consequently, the modeled generative distribution G#y will have connected support.
When learning a target distribution μ** with disconnected manifolds, GANs necessarily map fake data points out of the true manifold. This leads us to the following question: given that generator samples data points in each of the distinet modes, what can be its maximum precision? To begin with, let's ss' assume target distribution μ composed of m disconnected modes.
Assumption (Disconnected manifolds). The target distribution μ² lays equally measured spheres M,, i E [1.m] of radius r, each located at equal distance D (with D/2>> r).
The use of A Assumption is reasonable. First, on many real-world datasets, data is correctly balanced in between the different modes. The equal distance assumption can be justified from the concentration of distances in high-dimensional spaces: centers of modes will be approximately at equal distance (Beyer et al., 1999; Aggarwal et al., 2001). It has also been shown that embeddings of deep neural networks trained for classification tend to collapse around means that are equidistant to one another (Kothapalli et al., 2022). This could thus pave the way for new analysis where the chosen distance is no longer the euclidean distance in R" but a distance in the feature space of the generator or any pre-trained classifier. We also further discuss how this assumption could be relaxed. For the rest of the paper, let us define the set of well-balanced generators that maps equally data points to the different modes of the data distribution:
Definition 1. A generator G is well-balanced, for all i E [1,m), GBy(M)) =.... Gty(MM).
Considering well-balanced generators is also fair since many empirical improvements such as WGAN-GP (Gulrajani et al., 2017) or BigBiGAN (Donahue and Simonyan, 2019) have significantly decreased the mode collapse. GANs generate diverse output distribution on datasets such as CIFAR 10, CIFAR 100, and ImageNet. To validate the use of well- balanced generators, we ran a small experiment and evaluate the proportion of each class generated by GANs on MNIST and CIFARIO. On MNIST, the minimal proportion of a class is 9.2 and the maximal 10.9, while it is respectively 8.3 and 11.9 on CIFARIO (in %). The variance/mean ratio is equal 0.03 for MNIST and 0.22 for CIFARIO.
3.1 PRECISION AND THE ASSOCIATED PARTITION
Now that the prerequisites for both the data and the model have been given, we propose to define our approach. We create the connection between the set of generators from Rd to RD and the set of partitions in the latent space. In particular, for each given partition in Rd there exists set of associated generators defined as follows:
Definition 2. For a given partition of = (Ai...., Am) on Rd, we say that G is associated to sd if: for all i E m1, for all 2 A;, i= ==arg min d M)), where d(X,M,)) = min || - yll- yEM,
je[1.m]
It is clear that each given generator associated with a unique partition in the latent space. Moreover, the geometry of the partition partly explains its behavior and performance. We are interested in maximizing the precision of generative models. Any point in the intersection of two cells A nA ji E m is at equal distance of M, and M, and thus does not belong to any of these modes (since D/2 >>>>> r). Besides, due to the generator's 's Lipschitzness, there is a small neighborhood of the boundary such that any points in this neighborhood will be mapped out of the target manifold. This region in the latent space thus reduces the precision. For a given E >> O, we now define the epsilon-boundary of the partition a as follows:
Definition 3. For a given partition add =(A And of Rd and given E Rt, denote de of the epsilon- bounddrybburryyy of ot, defined as follows:
m
2 U (U
1=1
4

---

Under review as conference paper at ICLR 2023
where A corresponds to the E-extension of set A. To better understand the link between the precision of generator dg and its associated partition s, we state the following lemma:
Lemma 1. Assume that Assumption I is satisfied and of be a partition in R Then, any generator G E $1, associated with of verifies:
0G < 1 1-1(( 7(3mmm of), where Emin DDLL.
(2)
Interestingly, this result holds independently of the partition sd This result highlights that the geometry of the partition gives an upper- upper-bound on the precision of the generator. Consequently, to properly determine this bound on the precision levels of generative models, one might be interested in d o Furthermore, to exhibit generative models
determining the measure of this epsilon-boundee with optimal precision levels, one must look partitions with the smallest epsilon-boundary measures y(d of This is tightly connected to the theoretical field of Gaussian isoperimetric inequalities.
3.2 OPTIMALITY IN GANS
Isoperimetric inequalities link the measure of sets with their perimeters. More specifically, isoperi- metric incqualities highlight minimizers of the perimeter for fixed measure, e.g. the sphere in an euclidean space with a given Lebesgue measure. Gaussian isoperimetric inequalities study a similar problem in Gaussian space. Borell (1975) and Sudakov and Tsirel' son (1978) show that in finieedimeeniionll Gaussian space, among all sets of a given measure, half-spaces have a minimum Gaussian perimeter. More formally, for any Borel set A in Rd and half-space H, if we have ((A 2 ((1),, then y(A*) > y(11F) for any E > O, where A denotes the e-extension of the set A.
The Gaussian multi-bubble conjecture was formulated when looking for a way partition the Gaussian space in m parts and with the least-weighted boundary. It was recently proved by Milman and Neeman (2022) who showed that the best way to split a Gaussian space Rd in m clusters of equal measure, with 2 & m & d + 1, is by using 'simplicial clusters' obtained as the Voronoi cells of m equidistant points in Rd Convex geometry theory tells us that each cell convex cone, whose borders are hyperplanes going through the origin of Rd. We note of any partition corresponding to this optimal configuration, see Figure 1.
The aim of the present paper is to leverage this result to better understand the behaviour of GANs. We argue that in the case where m & d 1, optimal models in levels of precision are closely linked to the optimal partitions e8 derived in the Gaussian Multi-Bubble conjecture (Milman and Neeman, 2022). Besides, using results on the Gaussian boundary measure of those sets (Schechtman, 2012), we can also derive upper-bound Theorem (Upper-bounding the precision). Assume that Assumption I satisfied and m & d+1. For any 8 >>000 O, if I. is large enough, then for any well-balanced generator G E 91, we have:
the maximal precision of generative models, as follows:
ag E 1 (() of") +
In particular, there exists L with L3 D /logmm)) such that for any well-balanced generator G E gy:
0g & 1 Emin v log m e 3/22 where Emin = D/L.
Theorem 1 links the precision of well-balanced generators with the optimal partition from Milman and Neeman (2022). In particular, result in (3) gives an interesting insight when training GANs on a finite number of modes. Tanielian et al. (2020, Theorem 3) showed similar result but for the asymptotic case when the number of modes increases:
Ag < e Siiin Emin Ing[m)/2
(4)
For Emin / log(m) o1) both (3) and (4) have the same behaviour w.r.t. to the number of modes. Now, to further show the usefulness of st we prove the following theorem:
Theorem (Lower-bounding the precision). Assume that Assumption I is satisfied and & dd+ + and a well-balanced generator
For any 8 > O, there exists C large enough and I, > Dm/mm x (olo((m) G e 31 associated with of such that for any other well-halanced generator G E 91, we have:
ag > dG-
(5)
Moreover. if S d:
DVm L
OGG 2 1 1 Emax V It (log(Cm) Cm) where Emax
(6)
5

---

Under review as conference paper at ICLR 2023
M1
A1
d A3
22
A3
M2
M3
Latent space
Output space
Figure 2: An optimal generator maps 2D latent space to 2D output space with three modes (M,,M2,M)) The latent space has an optimal 'simplicial cluster' geometry. In the latent space, all the E e-boundaries intersect each other in the gray circle, which is mapped in the output space in the convex hull of the three modes.
This theorem, which proof is delayed to Appendix, shows that the set of generators associated with d contains optimal generators w.r.f. precision. More importantly, shows that when L is large enough, the bound in (3) may be tight, as it is almost reached by optimal generators defined in Theorem 2. An example of such optimal generators for the 2D case is given in Figure 2. This specific generator memorizes the dataset, since all samples
mapped to of the center of the modes M [1, m], except for those E-boundaries. It is not clear however, whether those are the only generators with optimal precision. We see that when the number of modes is less than the number of dimensions in the latent space, the only factor that impacts the precision is the number of modes.
What if modes are not equally distant? This assumption is needed for the definition of a well- balanced generator as the proposed in Figure 2. In R²² for example, if there is no assumption all on the location the modes, there might not be any well-balanced generators associated with the optimal partition at* As shown Figure 3, the latent space configuration obtained by the GANs on 3 aligned data points (right) is made of two parallel hyperplanes, much different from et defined by Milman and Neeman (2022) (left).
Figure 3: GANs training with 3 equidistant modes and 3 (almost) aligned modes. The first and third figure show the data points in the output space. The second and fourth stress the boundaries in the latent space using heatmaps of the norm of the gradient of the generator.
What if the dimension d+11? 1? This setting is much more complex. First, Assumption is no longer valid, since one cannot find m points equally distant from one another. Second, the result from Milman and Neeman (2022) is longer valid and the optimal partition of the Gaussian space in m equal cells is unknown. In this generalized context, GANs could hint at the optimal partition geometry. We show in Figure examples when training GAN from I² to R" with m equidistant target points in R This could give some insights on how to divide the Gaussian space into m equitable areas with least Gaussian -eeihhte perimeter.
EXPERIMENTS: UNDERSTANDING THE LATENT SPACE OF GANS
In the following experiments, we validate our theoretical analysis and derive insights for GANs trained toy and image datasets. We verify if the latent space geometry of GANs has similar

---

Under review as conference paper at ICLR 2023
Figure 4: Extending the multi-bubble conjecture when m > d++ 1. We plot the partition of the I² latent space of GAN that maps m equidistant points in RR" from = (left) 12 (right). Each colored cell maps to a distinct data point in the output space.
properties than the minimizers of the Gaussian isoperimetric inequality (Milman and Neeman, 2022), and compute series experiments on the latent space of GANs better understand its properties.
In all the following experiments, we train WGANs with gradient penalty (Arjovsky et al., 2017; Gulrajani et al., 7). For mixture of Gaussians, generator and discriminator are MI.P networks. For MNIST (LeCun et al., 1998), both the generator and discriminator are standard convolutional architectures. On CIFAR- 10 and CTFAR- datasets (Krizhevsky et al., 2009), we use Resnet- based (He et al., 2016) convolutional architecture with self-modulation in the generator (Chen et al., 2018b), and the transformer- based architecture from Jiang et al. (2021). To evaluate the performance of GANs, the density/coverage (Naeem et al., 2020). As recommended by recent works (Naeem et al., 2020; Kynkäänniemi et al., 2022), use dataset-specific classifier extract image features instead of an ImageNet pre-trained classifier, and thus refer to the FID as FD for Fréchet Distance. Implementation details are given in Appendix and code provided in Supplementary Material.
both the precision (Kynkäänniemí al., 2019), the FID (Heusel et al., 2017), and
4.1 LINEAR SEPARABILITY AND CONVEXITY
Milman and Neeman (2022) show that the optimal configuration in the latent space obtained as the Voronoi cells of equidistant points in R if m S d+ 1. This means that if GANs reach this optimal configuration, each of the cells must be convex polytopes and thus verify the following properties 1) each cell has 'flat' sides, and bounded exclusively by faces; 2) each cell is convex. In the following experiments, we study whether GANs' latent spaces feature these two properties.
Are classes linearly separable in the latent space of GANs? To verify this, we leverage a labeled dataset and investigate if simple linear model (e.g., multinomial logistic regression) can map from latent space to label space. If cells the latent space are bounded by hyperplanes, the linear model expected to be good predictor of a generated sample's label.
We consider labeled dataset of samples with fixed number of classes. Gg is pre-trained generator, and Cp pre-trained classifier considered of latent vectors E Rd and their associated labels Co (Z) On CIFAR-10/100 similarly to Razavi et al. (2019), only data points with above a confidence threshold are accepted. This dataset is later split into 100k training points and 10k test points. The mapping from latent vectors z to their labels y is learned by multinomial logistic regression. We report the test-set results in Table I under the column LogReg Accuracy. This accuracy reaches 90% on MNIST and 70% on CIFAR- 10. Interestingly, there is also a correlation between the linear separability of the latent space and the precision metric, which validates the optimality of the simplicial cluster partition.
oracle. Using Go and Co. we construct dataset
Are classes convex in the latent space of GANs? In this experiment, we draw two random latent vectors zo and zi that belong to the same class. Then, we generate linear interpolations E + (1 -1 E)zı and verify if these new samples belong to the same class zo and 2ı, i.e. whether Co (Z)) equals to Co (Z) We report the mean accuracy of this experiment in Table 1 under the column Conver Accuracy. Again, the higher the precision, the 'more convex' each cell in the latent space seems to be. For a qualitative evaluation, we show this phenomenon in Figure 5 and stress that linear interpolations conserve the image class.

---

Under review as conference paper at ICLR 2023
Dataset
Architecture
Latent dim
Precision
LogReg Acc.
Convex Acc.
100 Gauss.
MLP
100
75.5
78.5
87.2
MNIST
CNN
64
93.2
90.4
98.7
CIFAR-10
ResNet
64
75.6
65.3
75.2
CIFAR-10
Transformer
256
79.5
70.7
84.3
CIFAR-100 ResNet
64
65.9
34.4
47.1
CIFAR-100
Transformer
64
70.1
26.5
42.3
Table In this experiment, we verify 1) the latent spaces of GANs are linearly separable (LogReg Accuracy); 2) if each cell of the latent space is convex. (Convex Accuracy). In par with Theorem 1, the higher the precision is, the more each cell in the latent space is linearly separable and convex. The architecture Transformer refers to the TransGAN model from Jiang et al. (2021).
Illustration of convexity of classes in the latent space of GANs trained CIFAR-10 -10. Visual inspection confirms that latent linear interpolations between two samples of the same class most often conserve the class.
Figure 5:
Impact of the latent space dimension. Now, we evaluate the impact of the latent space dimension on MNIST, CIFAR10 and CIFARI00 datasets. To do so, we vary the number of latent dimensions d from 2 to 128 for each given dataset. In Figure we exhibit two phases the performance of GANs when changing the number of latent dimensions: first, when d is below dataset-specific threshold, the precision the model falls when reducing the number of latent dimensions. When d is above this threshold, the precision becomes constant, and increasing the number of latent dimensions does not bring any apparent improvement. This threshold is a function of the complexity of the dataset and its number of modes. We observe from Figure 6, that the more complex the dataset, the more it requires large latent space for high precision levels. This is coherent with the theoretical results, where the precision decreases w.r.t. to the number of modes in / /log(m) when m & d+1.
Disentangling the manifold dimension. As discussed in Roth et al. (2017), two different problems can arise when training GANs: i) dimensional misspecification where the true and modeled distri- butions do not have density functions w.r.t. the same base measure, and ii) density misspecification. where GANs try to fit a disconnected manifold with a unimodal disitribution. To isolate the density misspecification studied in the current paper, we train conditional GAN with a low-dimensional latent space Rd (e.g. R in our setting), so that the dimension of the generated manifold is at most 5. We later collect dataset of synthetic generated samples Synthetic CIFAR-IO. and train unconditional GANs by varying the dimension of the latent space. Figure 6 shows both the Synthetic CIFAR- 10 and the standard CIFAR-10 converge to the same limits for FD, Precision and Density, showing that with large latent space dimensions, the density misspecification seems to be the main issue to cope with. A synthetic experiment showing the importance of density misspecification over dimensional misspecification is available in appendix.
Impact of overparametrization. Balaji et al. (2020) already showed the importance of GANs' overparametrizzatoon in both their convergence and performance. Knowing that, we study whether overparametriizaion can help GANs obtain the optimal geometry of latent space. In Table 2, we vary the width of ResNet generators, and highlight that overparametrized GANs better fit the target distribution. More importantly, we observe that overparamettization helps achieving better linear separability of their latent space, shown by LogReg Accuracy.
8

---

Under review as conference paper at ICLR 2023
MNES Syrthelic CHAAN1D
Precision
Density
Latent dimension
Latent dimension
Latent dimension
Figure 6: Performance of GANs w.r.t. the number of modes and latent space dimensions on image datasets. Left, the FD gets lower with increased latent dimension. Center and right, precision and density improve when the latent dimension increases and saturates from threshold.
Dataset
Width
LogReg Acc.
Convex Acc.
Prec.
Dens.
Cov.
CIFAR-
32
53.4 + 0.5
61.1 + 0.3
447.9 15.6
63.8 + 0.7
52.9 + 1.3
73.0 = 1.0
10
64
60.7 + 0.5
72.1 + 0.6
154.3 4: 10.8
72.1 + 0.4
69.4 4 1.5
82.9 0.7
63.4 +
4.6
74.9
74.2
85.7 =
256
65.0 + 0.4
7.4
7.68
75.6
7
87.5 = 0.8
512
65.3 0.6
0.7
10.8
10.::00
77.5
87.0 = 0.6
CIFAR-
32
20.3
2.5
4
4.6
37.9
69.5 1.6
100
0.9
3.5
286.4 10.4
0.7
1.4
78.7 =
128
0.3
3.7
139.1
6.76
6.33
85.9 = 0.7
256
0.5
0.6
1.7.4
66.0
1.8
86.4 = 0.5
512
+
0.5
115.3
65.9 0.5
64.3
85.9 = 0.5
Table 2: Overparameerizztioon study: for a latent dimension equal to 64, we vary the width of the generator (confidence intervals computed 10 checkpoints). Increasing the capacity of GANs tend to structure their latent space in simplicial clusters (better LogReg accuracy) and improve their performance on precision, density and coverage.
CONCLUSION
This paper aims make a step toward better understanding of GANs learning disconnected distributions. When the latent space dimension is large enough, we present an optimal latent space geometry of GANs: 'simplicial clusters', Voronoi partition where each cell is convex cone. We further show experimentally that GANs with sufficient latent capacity tend to respect this optimal geometry. We believe that our analysis can foster exciting research GANs, with both theoretical and practical impacts. For example, understanding the optimal latent space's geometry could help design semi-supervised or transfer algorithms from GANs. Also, could inspire new neural architectures with bias for this 'simplicial cluster' partitioning of the latent space. Finally, let note that our results could potentially be extended to other types of generative models with Gaussian latent space and, thus, would allow better understanding of their properties. To adapt analysis for variational auto-encoders or diffusion models, one would need to adapt our results to a stochastic generator. This could be exciting follow-up of our work.
Limitations. We showed the existence of optimal generators and have shown experimentally that overparametrization plays a key role. However, limitation of our work is that we could not prove their uniqueness. This is linked to partitions with the lowest è e-boundaries measures in the Gaussian space, which a complex, unknown result. A second limitation is that the derived optimal generators are not valid in the case > d 1, because the minimizers of Gaussian isoperimetric inequalities are known in this case.
Potential negative societal impacts. This work is mainly about understanding the behavior of deep generative models. Thus, it may lead to practical improvements in this technology and increase its potential negative impacts, such deepfakes (Fallis, 2020).

---

Under review as conference paper at ICLR 2023
REFERENCES
Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka. Labels4free: Unsupervised segmentation using stylegan. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19970-13999 2021. (Cited on
Charu C Aggarwwa, Alexander Hinneburg, and Daniel A Keim, On the surprising behavior of distance metries high dimensional space. In International conference on database theory, pages 420 434. Springer, 2001. (Cited on 4)
M. Arjovsky, Chintala, and L. Bottou. Wasserstein generative adversarial networks. In D. Precup and Y.W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 214-223. PMLR, 2017. (Cited on 3,
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. International Conference on Learming Representations, 2018. (Cited on 3)
Samanch Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discrimi- nator rejection sampling. In International Conference on Learning Representations, 2018. (Cited
on 3)
Yogesh Balaji. Mohammadmahdi Sajedi, Neha Mukund Kalibhat, Mucong Ding. Dominik Stöger, Mahdi Soltanolkotabi, and Soheil Feizi. Understanding over-paramettriization in generative adversarial networks. In International Conference on Learning Representations, 2020. (Cited on 8)
Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. When is "nearest neighbor" meaningful? In International conference on database theory, pages 217-235. Springer, 1999. (Cited 4)
Christer Borell. The brunn-minkowski inequality in gauss space. Inventiones mathematicae, 30(2): 207-216, 1975. (Cited on 5)
A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. (Cited 3)
Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick Smagt. Metrics for deep generative models. International Conference on Artificial Intelligence and Statistics, pages 1540-1550. PMLR, 2018st 8a. (Cited on 3)
Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. In International Conference on Learning Representations, 201 8b. (Cited on 7, 14)
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in neural information processing systems, 32, 2019. (Cited on 4)
Don Fallis. The epistemic threat of deepfakes. Philosophy & Technology, pages 1-21, 2020. (Cited on 9)
Herbert Federer. Geometric measure theory. Springer, 1969. (Cited on 16)
I. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and J. Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2672 2680. Curran Associates, Inc., 2014. (Cited on 3)
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A.C. Courville. Improved training of Wasser- stein GANs. In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan. and R. Gameet, editors, Advances in Neural Information Processing Systems 30, pages 5767-5777. Curran Associates, Inc., 2017. (Cited on 3, 7)
10

---

Under review as conference paper at ICLR 2023
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 166-174, 2017. (Cited on 3)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference computer vision and pattern recognition, pages 770-778, 2016. (Cited on 7)
Steven Heilman, Aukosh Jagannath, and Assaf Naor. Solution of the propeller conjecture in R. Discrete & Computational Geometry, 50(2))263-335 2013. (Cited on 2)
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637 2017. (Cited on 7, 13)
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances Neural Information Processing Systems, 34, 2021. (Cited on 1, 7, 8)
T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4401-4410, 2019. (Cited on 1,
Mahyar Khayatkhoei, Maneesh K Singh, and Ahmed Elgammal. Disconnected manifold learning for generative adversarial networks. In Advances Neural Information Processing Systems, pages 7343-7353, 2018. (Cited on 1)
N. Kodali, J. Abernethy, J. Hays, and Z. Kira. On convergence and stability of GANs. arXiv. 17Xi:1705..0215. 2017. (Cited on 3)
Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: review on modelling principles and generalization. arXiv preprint arXiv:2206.0400 2022. (Cited on 4)
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. (Cited on 7)
T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. In H. Wallach, H. Larochelle, A. Beygelzimer, A'cché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 3927-3936. Curran Associates, Inc., 2019. (Cited on 1, 3, 7, 13)
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. arXiv preprint Stt::2203306020 2022. (Cited on 7)
Y. LeCun, L. Bottou, Bengio, and Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278-2324, 1998. (Cited on 7)
Michel Ledoux. Isoperimetry and gaussian analysis. In Lectures on probubility theory and statistics, pages 165-294. Springer, 1996. (Cited on
Emanuel Milman and Joe Neeman. The gaussian double-bubble and multi-bubble conjectures. Annals of Mathematics, 195(1):89-006, 2022. (Cited on 1, 2, 6, 7, 16)
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. (Cited 3)
Muhammad Ferjad Nacem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reli- able fidelity and diversity metrics for generative models. International Conference on Machine Learning, pages 71 7176-7185, 2020. (Cited on 3, 7, 13)

---

Under review as conference paper at ICLR 2023
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019. (Cited on
K. Roth, A. Lucchi, S. Nowozin, and T. Hofmann. Stabilizing training of generative adversarial networks through regularization. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, Curran Associates, Inc., 2017. (Cited on 8)
pages 2018-2028.
M.S.M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and Gelly. Assessing generative models via precision and recall. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neurul Information Processing Systems 31, pages 5228-5237. Curran Associates, Inc., 2018. (Cited on 3)
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. arXiv preprint arXiv:220..0027: 2022. (Cited on 3)
Gideon Schechtman. Approximate gaussian isoperimetry for k sets. In Geometric Aspects of Functional Analysis, pages 373-379. Springer, 2012. (Cited on 5, 19)
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9243-9252, 2020. (Cited on 1,
Vladimir Sudakov and Boris S Tsirel' son. Extremal properties of half-spaces for spherically invariant measures. Journal of Soviet Mathematics, 9(1):9-18, 1978. (Cited on 5)
Ugo Tanielian, Thibaut Issenhuth, Elvis Dohmatob, and Jeremie Mary. Learning disconnected manifolds: gan's 's land. In International Conference on Machine Learning, pages PMLR, 2020. (Cited on 3)
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. In Advances in Neural Information Processing Systems, pages 3835-3844, 2018. (Cited on 2)
X. Wei, B. Gong, Z. Liu, W. Lu, and L. Wang. Improving the improved training of Wasserstein GANs: A consistency term and its dual effect. In International Conference on Learning Representations, 2018. (Cited on 3)
Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12863- 12872, 2021. (Cited on
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7354-7363, 2019. (Cited on 3)
Z. Zhou, Liang, Y. Song, L. Yu, H. Wang, W. Zhang, Y. Yu, and Z. Zhang. Lipschitz generative adversarial nets. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th Interna- tional Conference on Machine Learning, volume 97, pages 7593-7593. PMLR, 2019. (Cited on 3)
12

---

Under review as conference paper at ICLR 2023
A
IMPLEMENTATION DETAILS
Table 3: GANs training details on MNIST
Operation
Kernel
Strides
Feature Maps
Activation
Generator G(z)
2 ~ ((O.f) Fully Connected Convolution
dim(z)
7 7 X 128
x 3
1x
7 64
LReLU LReLU
Convolution
1x1
7 7 64
Nearest Up Sample Convolution
14 14 x 64
3x3
1x1
14 14 x 32
LReLU LReLU
Convolution
3
1x1
14 14 x 32
Nearest Up Sample
14 14
Convolution
3
1x1
28 16
LReLU
Convolution
x
1x1
x8
Tanh
D(x)
28 28 1
Convolution
4x4
2x2
14 14 512
LReLU
Convolution
3x3
1x1 1
14 14 512
LReLU
Convolution
4x4
2x2
7x7x
LReLU
Convolution
3
1x1
7
LReLU
Fully Connected
Batch size
256
Leaky ReLU slope
0.2
Gradient Penalty weight
10
Learning Rate Discriminator
1 10
Learning Rate Generator
5 x 10
Disciminator steps Optimizer
2
Adam
B1 0.5 B2:
Training. We use the Wasserstein loss with gradient-penatyy on interpolations of fake and real data. At each iteration, the discriminator is trained 2 steps and the generator I step with Adam optimizer. The batch size is 256. The learning rate of the discriminator is two times larger (Heusel et al., 2017), i.e. 5% x 10 for the generator and 1 10 10-4 for the discriminator. GANs trained for 80k steps on MNIST and for 100k steps on CIFAR datasets. Architectures of generator and discriminator are described in Table and Table
Evaluation. Features extracted with classifier with simple architecture (convolutions, relu activation, no batch normalization). The classifier trained on each dataset with cross-entropy loss. Weights of the classifiers given in the code. For evaluation metrics, by the authors. For FID (Heusel et al., 2017), we use 50k real images and 50k fake images. For precision, recall, density and coverage (Kynkäänniemi et al., 2019; Nacem et al., 2020), we use 10k real images and 10k fake images with nearest-k=$ 5. Full results the study on latent space dimension presented in Table 5.
follow the setting proposed
We also share the code for a better reproducibility.
GPUs. For all datasets, the training of GANs was run on NVIDIA TESLA V100 GPUs (16 GB). The training of a GAN for 100k steps on CIFAR takes around 30 hours.
13

---

Under review as conference paper at ICLR 2023
Table 4: GANs training details CIFAR datasets. BN stands for batch aattchnnomalliztion.
Conditional
Operation
Kernel
Strides
Feature Maps
BN (Chen et al., 2018b)
Activation
Generator G(z)
~~ Id)
128
Fully Connected
x 4 x 128
ResBlock
[3x3] x2 1x1
4 4 x 128
Y
ReLU
Nearest Up Sample
8 8 128
ResBlock
[3x3] x2
1x1
8 128
Y
ReI.U
Nearest Up Sample
16 16 x 128
ResBlock
[3x3] x2
1x1
16 16 x 128
Y
ReLU
Nearest Up Sample
32 128
Convolution
3x3
1x1
x2 x
Tanh
Discriminator D(x)
32 x
ResBlock
33 x
1x
32 32 x 256
ReLU
AvgPool
2x2
16 256
ResBlock
33 x2
1x1
16 256
ReLU
AvgPool
2x2
1x1
x
ResBlock
3
x2
1x1
x
ReLU
ResBlock
3x3
x2
1x1
x x 256
ReLU
Mean spatial pooling
256
Fully Connected
Batch size
256
Gradient Penalty weight
10
Learning Rate Discriminator
1 10
Learning Rate Generator
5 x 10 -1
Discriminator steps
2
Optimizer
Adam
B1
B 0.999
14

---

Under review as conference paper at ICLR 2023
Dataset MNIST
L. Dim.
Acc.
FD 112.3 22.6
Prec.
Rec.
Dens.
Cov.
44.3 + 2.5
2.9 1.0
8.0
0.9 + 0.3
0.5
59,4 + 0.7
36.4
75.7 9..0
47.9 9.5
6..0
83.9 0.4
3.0
9..4
83. 7.0
8.44
1..2
91.1
9..
8..
87.1
10
86.7
10.7
93.
89.0 9..5
89.8 0.7
12
87.4
1.6
9..6+
0.5
9.
91.8
16
86.9 0.4
1.0
03.
9.9 0.6
90.7+
32
9.4
1.6
0.3
0.7
91.8 1.7
90.3
64
9.34
2.6
93.2
9.5
9.01
90.2
128
87.0
2.3
9.3
91.8
90.4 9.5
CIFAR-10
1.7
4 25.6
5.9
8.99
0.2
4.1
0.5
183.1
10.7
6.60
0.6
51.8
0.7
10
5..
1.2
7.22
0.5
65.6
8.7
30
6.5
+.7
8.84
7.80
85.9+02.
128
6.5
$1.3710.7
7.5
0.5
1.6
86.7
Synthetic CIFAR-10
2.00
175.9
7.9
7.2
0.3
5.1
3.5
183.1=101
6.60
0.6
1.0
74.8
10
131.7 28.5 0.6
58.8
1.0
6.2
7.76
30
57.3 0.9
$7.7
7.
76.7
4.4
90.4 + 1.3
128
55.7 +
79.0 12.4
7.
7.6
7.1
84.2 + 6.3
CIFAR-100
12.1 + 1.2
9 125.8
1.7
86.2
0.9
0.5
0.9
11.10..3 0.3
244.9 = 8.7
4..
0.3
0.9
64.4 1.2
10
0.3
220.2 2 10.2
5.7
81.6
1.3
76.9 + 1.0
30
27.3 1 0.4
198.7 + 14.8
0.6
1.46 0.6
81.7 09
28.4
+ 0.3
139.1 11.8
6.76
7.6
2.3
85.9 + 0.7
128
26.7
+ 0.5
191.9 + 10.7
0.87
80.2 2 0.8
5..4
82.5 + 1.0
Table 5: Performance of GANs when varying latent space dimension. Confidence intervals are computed 10 checkpoints of the same training. See main paper for curves of precision and FID with regard to the latent space dimension.
15

---

Under review as conference paper at ICLR 2023
B TECHNICAL RESULTS
B. PROOF OF LEMMA
We want to show that generator G E 9 is such that dg < 1 - - of where
m
Fenin of = U
(U)HAA)) Smin (U))iAA)
il
Proof by contradiction.
Assume a generator G such that there exists E Fmin of and E [1,m] such that G(z) E M, Since G associated with s we have using Definition 2, that there exists z' and j [I.m].] * such that || - 2'|l
< Emin/ 12 and j = arg min [[G(=') - M Thus, we have:
ke [E
[[G(z) - G(2')l Mi)(
d M))
D miin/2 /
[((() G(Z))l
And,
Dmin/ Emin-
2 2
L.
This contradicts G being in 9T.
B.2 PROOF THEOREM 1
Proving that: for & d for any 8 >> 0, if I. is large enough, then, for any well-balanced G we have ag & 1- of ) Let be well-balanced generator and of the partition associated with Let us first define the gaussian boundary measure Py of a partition a of Rd For partitions with smooth boundaries, it coincides with the (d 1)-dimensional gaussian measure of the boundary, defined as follows:
Py = lim r(ac ol) -
22/
Moreover, for sets with smooth boundaries, we have from Federer (1969, Theorem 3.2.29):
liminfe-0 7( ( -(d)
r(de of) ((a))
lim
v 2/TTE
Let us denote _²",, the optimal partition defined in Milman and Neeman (2022). based simplicial clusters. A is a standard partition where y(A7) =... y(A", for all and Σ, ((A,) = 1. By the multi-bubble theorem (Milman and Neeman, 2022), simplicial clusters (such ef*) are the unique minimizers of the gaussian isoperimetric problem, thus:
lime +0 & lime-ve of
of)
L & Log.
where Los = -imm pf and Last lime-10 Moder
Then, for any O, there exists E' O such that, for any E << <',,
r(d of Logal < s y(de
Loa < S and Lyg & Lod
Thus, for any 8 > 0, there exists g > O such that, for any 8 < E',
r(a5 al²) & r(05 (dd 28g
16

---

Under review as conference paper at ICLR 2023
Besides, we know that Consequently, we have that:
dg < 1 y(d Fmin of
dg & 1 y() Fmin of)
< 1
odd a Smin
20E using (7).
We conclude by choosing L big enough such that Emin strictly smaller than 8/2"
Second part of Theorem 1. Let L, be such that L DVlog(m). Let's prove that for any well- balanced generator G L., have:
ag & 1 -1--Em Emin v llgm 3/2,
Using the method from Schechtman (2012), we have the measure of the border of cell i:
(
e2d,, where is such that 1
I. 212ds 1/m,
2n
V2n
8
(142)?/2
2n
8 /logm
e1 -
(using logm S 22logm,
m
8 Vlgm
evllom 122
m
Thus:
((d (00mmoo))= of =
Σ y
12
> Emin v llogm e Emin Togm 1ogmm-cmmm122 /2
Thus, we have
04; < Rmd Emin of
Emin logm Gmin Vlogm e
Moreover, using Emin L and L > Dogom, so we get Emin v llgm & 1:
ag < 11 Emin logm 3/2
B.3 PROOF OF THEOREM 2
For a given partition ad, and target distribution μ with m disconnected components M,, E [1,m]. defined X,, [1,m] m] set of sampled data points such that for all E [1,m], have X; M,. Now, we define G with > O, a generative model such that:
(A9)())
G)(() = Σ wi(z) X; with wx(2) ΣjEs, d(z, (A))
(8)
iES.
where d(z,A)= = minged ||z-a||, and S. {j E [1,x] such that 2 E A4) denotes the set of cell- extensions the point belongs to. We can see that C memorizes the dataset since every z close to the center of the cell A, such that [S.| = verifies G²(z) X,. An illustration is given in Figure 2.
To be more precise, all samples are mapped to one of the center of the modes X;, E [1.m], except for those c-boundaries. When belongs to the intersection of two e-boundaries, Ge (z) is a simple linear combination of points. It is only when [S_| > 3 that more complex samples are generated. A simple illustration of G for = 2 and m = 3 is given in Figure Interestingly, one can also show that the image of G is equal to the convex hull of the diracs X,, [1,m]. In particular, there exists particularly interesting neighborhood v where G)(v) is equal to the whole convex hull of the points X;, m1.
17

---

Under review as conference paper at ICLR 2023
Proof that G2 is well-balanced.
We recall that a generator is well-balanced if we have G1y(Mi) =
(F)(Mm). By construction
(8), we have that for any E [1.m]
(
X:Il =
Σ kp
wx (Xx xi)
x (1 Wi).
So, for any z Aj have that
1= arg min arg min je[[mn] wy Jm (Gyl)
Thus Gg is associated with the optimal partition ot*..
Besides, for given radius r of the different modes, since everything is symmetrical, we have that f({z E Rd ([G(() - -illl < r) =..- 7(12 E R [[G(2)) - Xmll & rj. Thus, the generator is well-balanced.
Proof that G Emax
is Jz.
To begin with, have Rd = =F of (R" (e.
Ra add of UAA((U)AA))) nAi):
i=1
and, we know that for each E [1.m]. (z(z) constant and thus L-Liscchitz
Now,
ae of = U
NA.
Se (e. (m)) ieS SS
Now, let S E ([1, with (S) = 2 2. Let 2, E S². (BI...., Bm) be two vectors, both in RR", such that for i [1,m]:
Let a = (at.... and B
d(z, d(z,(A))) (A
d(z (AP)C)
and B.
Ejemt ((z,(A5)) (A5)C)
Σ Ejedh ((21(A9))
We have that
(((z))G((2)) |1
Σ x (1 Σ
+ Σ 0;
Σ BxIl 111
1
F1
111
|| Σ
(0: Bx) - -x:)lI
1=
max
| -xxxll
X,ll || - -B1:
max
|X Xyll (14(2) - h(2))l
where h is the function from Rd + RM defined as:
h(z) = d(z, (A))
d(z, (A
Lea d(z,, (A5)
Lie ((z,(A))) (A
We can write h = fog with / the function defined from Rd R" by
f(z)
d(e, (AE))))
and the function defined RM [0) by
g(2) ||2||
18

---

Under review as conference paper at ICLR 2023
We have that f is a /m--Lipshhiz functions (given that こ d(z, (Am))) is 1-Lipschitz). Besides, we know that outside the ball (0), the function is (L/s)-Lipschitz.
Using the convexity of function z Ljeat d(z, (A))) (as sum of convex functions), we can show that for all z E oe we have that f(z) & (m E)e and f(z) is not B₂ Finally, the function is w/t E -ipschittz Thus, we have that:
DVm 8
[G G((II &
||2 1|
with D = X; (1,)
B.mmiii j. Consequently, by noting Emax
Dym have
| Emax (z) - Emux
(E')l < 2 - 2/11-
We conclude on the Lipschitzness of G on Rd_
Proving that: for < d + for any 8 > O, if L is large enough, then, for any well-balanced E Ii, we have dci 2 ag 8. From the proof Theorem I, we have that for any ő >> 0, there
exists Emin such that:
0G E (fa ol)
((iin of 28Emin
using (7).
Now, by construction of G mmxx have that
2 1 - 11> Pmx
mma of*)
Consequently,
0g < 1 111-y(22 y(dfmi ad*) 2.emmaxx + (() Fias odd - fmamro ef(²)
$ ac; + Emax + (E Emax sf") 7(0 Smin sfr)
< ac; + 2mmxx E(EEm Emax of") 2L Emax (1) of ) + + + 22 Eminn)
< 0c; + 6mmmmax 2L Emux.
< (o; + Emax (48 + 21.
We conclude by choosing I. big enough such that Emax is strictly smaller than 4842L
Proving the second part of Theorem 2. The precision of G is thus such that:
11 y(d of)
1- -
However, since d el C A Af, we have that for any E
r10 of
(A
Using results from Schechtman (2012, Proposition 1), when d, there exists C large enough, such that
7(A Pmm
Emax
π llog(CC))
m
Thus, we have
Emax
π Ioog(mm),
To have CrGr
O, we must have Emax
< 1/ log(Cm)). This is the case since we have
Emax Dvm/I. and
I D /m htmmmlllga llog(Cm).
19

---

Under review as conference paper at ICLR 2023
C ADDITIONAL RESULT
Impact of the number modes. To illustrate the results from Theorem and Theorem 2, propose to vary the number of modes of the data distribution. On real-world data, the number of modes set but usually unknown, and removing/adding classes as proxy for modes usually does not give insightful results since some classes can be much more complex than others. We thus use a synthetic setting, where we can easily control both the number of modes and their complexity. Figure stresses that as the number of modes increase, the precision decrease. Interestingly, using large latent space dimension can relieve the problem, even if the latent space dimension clearly below that of the target. Recall the two problems that arise when training GANs: i) dimensional misspecification where the true and modeled distributions do not have density functions w.r.t. the same base measure. and density misspecification, where GANs try fit disconnected manifold with a unimodal disitribution. From the results we conclude that:
With very low latent space dimensions, both problems i) and ii) have to addressed and this leads to poor precision the number of modes increases.
With larger latent space dimensions, the problem i) is less of a burden even when there is clear dimensional misspecification and thus the GANs' performance is more tied to problem ii).
903
Precision
Coverage
22.5
013
dimizi-100
dimiz)=100
dim(Z)=20
dim/2)-20
442
diml2)=5
932
diniZi-5
100
20
100
Number of modes
Number of modes
Figure 7: Training on a mixture of Gausians in IR 100 with varying number of modes and varying latent space dimension. The bigger the number of modes, the lower the precision. Increasing the latent space dimension helps up to limit depending on the number of modes.
20


Reviews:
[{"review_id": "xn1RGkKgSQ", "reviewer": "Reviewer_xjYQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the optimal precision of GANs trained on disconnected distributions under some simplifying assumptions about the distributions. The paper derives lower and upper bounds for the precision of GANs under these assumptions, and also studies what partitioning of the latent space that can give rise to the optimal precision in GANs. The paper also provides some smaller scale experiments on image datasets that show GANs tend to construct the predicted Voronoi partitioning to a certain extent, and also behave consistently with the predicted effects of the dimensionality of the latent space on precision.\n", "summary_of_the_review": "Overall, I like the paper and think the findings about what kind of latent space partitioning can be optimal for GANs with respect to precision are valuable. The provided bounds are also insightful for designing new GANs. My main concern is that the paper does not restrict its claims as well as I’d consider appropriate.\n\n### Update post-rebuttal\nI thank the authors for their response and answering my concerns. I think the paper merits acceptance upon making its assumptions and restrictions more clear, but I agree with the other reviewers that the assumptions are somewhat strong, as such keeping my score at 6.", "strength_and_weaknesses": "**Strengths**\n\nThe paper is well-written and easy to follow. The posed question is also very interesting to me, understanding how partitioning of the latent space affects the precision of GANs, and vice versa, finds many potential applications for improving GAN precision and searching/editing in their latent space. The Theorems and claims are mostly correct, and the provided bounds are non-trivial and interesting.\n\n\n**Weaknesses**\n\nIn general I like the paper, but I have some questions and minor concerns that I’ll go over below:\n\n1) The paper neither shows Voronoi partitioning to be a necessary nor a sufficient condition for achieving optimal precision. While the authors do touch on this point in several parts of the paper (including right after Theorem 2 and then again at limitations), I still think the language is vague and must be made very explicit to avoid misleading a not so very careful reader. In particular, since the paper starts by raising the following question in its abstract: “what is the latent space partition that minimizes the measure of out-of-manifold samples”, a reader is likely to think that the answer is a Voronoi partition, yet this is not true! The question remains open in that no necessary condition on the latent space is discovered. Please make this explicit in your abstract – mentioning the question is great for motivation, but it needs to follow the explicit mention that you could not answer this question, rather could find one particular generator with Voronoi partitioning that minimizes the measure of out-of-manifold samples, which is still plenty valuable as a starting point in my opinion.\n\n2) In Figure 1’s caption, can you be more specific about in what sense left is close to right?\n\n3) The equal distance assumption is not well justified, either cite or elaborate more specifically.\n\n4) Can you be more specific on the statement “If L is large enough”; and elaborate in what way small L breaks the theory, and discuss any connections to practical settings where L cannot be very large without causing instabilities.\n\n5) The claim in Table 1 that “the higher the precision is, the more each cell in the latent space is linearly separable” is not strictly true when comparing Precision and LogReg columns, same goes for the Convex column. Moreover, the table lacks error bars, so significance of any trend is questionable. Including a proper correlation test can help clarify the extent of your claim.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "luzZom1jCW", "reviewer": "Reviewer_ApLT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses a recent mathematical result -- about the form of Gaussian-isoperimetric partitionings of Euclidean space -- to conjecture that successfully learned GANs learn such \"optimal\" partitionings of the space. This is used to try to quantify the precision of such learned GANs in terms of other parameters of the problem. The paper contains experiments on GAN latent spaces, on which the data are Gaussian by construction, and which can link learned GAN latent spaces to theory.", "summary_of_the_review": "The overall idea, of using deep results in Gaussian isoperimetry to explore learned GAN representations, is great and could be a very useful avenue of study in several ways. The theoretical narrative is very incomplete, and though the experiments are promising, the theory is not sufficiently supported or explored by the experiments.", "strength_and_weaknesses": "Strengths:\n\nThe paper adopts a novel perspective on the latent space of GANs, by using results in Gaussian isoperimetry (GI) to conjecture about the form of the learned space of optimally-learned GANs. Using this perspective and powerful tools from GI, the authors try to relate precision to the structure of the latent space learned by a GAN. This is a really nice idea. The proofs are fairly clearly and straightforwardly done, with an exception (see below).\n\n\nWeaknesses:\n\nThe overall structure of the theory is not supported by the practical considerations. \n\nThe experiments don't really support the theoretical results as much as they could. The theory consists of an upper bound on precision which holds for any model, and a lower bound which holds for an \"optimal\" model. The lower bound is inherently more interesting in this situation, but its statement is weaker. A natural next question is: do the backprop-learned GANs actually follow this lower bound? \n\nThe relevant quantities are L, D, and m, which can all be measured to empirically investigate this question. The theorems state a number of tradeoffs betweeen these quantities, which could be empirically investigated without much effort, but are not. The lower/upper bounds don't actually match, because the quantification over the partitions is different; and I'd therefore like to see empirical verification of the \\sqrt{log m} rate. This is easy to test not only for MNIST, but more multiclass problems like ImageNet-1000; you can subsample classes and get precision numbers. \n\nThe issue of precision itself is a little muddled in this paper. It's not clearly defined initially. Then it's assumed that r << D, i.e. effectively that the data are on a set of m point masses. This ignores the data-dependent effects of distance with space, and leads to the fundamental D/L scaling of the epsilons in the theorems. How to interpret the theorems on real data is therefore somewhat unclear. \n\nThe Lipschitz constant L is a parametrization the authors choose, which ends up making the proofs track with proofs of the analogous GI results. Like other alternative parametrizations, it is something that could be reasonably estimated in practice - what is it for the networks used in the experiments? Is L large enough in practice to support the desired bounds? From the cited Virmaux+Scaman reference, L is ~150 for an MNIST CNN of depth 5; it would be good to include the results of such an evaluation, to complement these experiments. Meanwhile, D can also be estimated for the dataset, and m is known. So the bounds on precision and recall can actually be numerically computed and compared. This would provide some notion of how good the parametrization is, and how much more theoretical understanding is needed.\n\nIt would also be good to see the \"convex accuracy\" reported for all the experiments - I find that and the logistic regression to be very flexible and useful aids to understanding the relationship between simplicial clusters and the empirical learned space. Experimenting on more networks, grid-searching across approximate {L, D, m} configurations, is needed.\n\nRelaxing the GI assumption to encompass data-dependent Gaussian concentration of measure (without isoperimetric symmetries) would be a very interesting avenue of future work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "je1T47-b4E3", "reviewer": "Reviewer_jLeB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses existing theoretical results on the boundary measure of dimensional Voronoi cells of m<d+1 equal distance modes to show a bound of the precision of a m-mode GAN. Some empirical study is carried out to study the geometry of the latent space. However, the empirical result is not particularly well-connected with the theoretical bound. ", "summary_of_the_review": "Interesting usage of an existing theory for GAN precision bounds. But the theoretical results is not well connected with practical usage of GANs.", "strength_and_weaknesses": "The usage of the theoretical result seems valid. However, it still seems that the assumption (Assumption 1) is a bit too strong. In practice, the prior on different modes can be unbalanced. Some discussion is provided but no solution is proposed. Also the distribution can be a mixture model instead of the case that each distribution is strictly restricted within a (d+1)-dimensional Voronoi cell. \n\nThe implication of the bound is unclear. No practical insights are revealed as a consequence of the bound. The experiments explore the geometry like linearly separability and convexity. But these results are still quite far from the bound. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D3OjUpLRPX", "reviewer": "Reviewer_q2EK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A Lipschitz-continuous generator of GAN cannot map a unimodal distribution to disconnected distributions. Therefore, in order to transform a unimodal distribution, e.g., Gaussian, of the latent variable to cover all modes of the disconnected target distribution, the generator must map a subset of the latent space outside the support of the target distribution. \n\nIn order to minimize the probability of the generator outputting an out-of-support sample, the paper studies some sufficient condition of the shape of the latent space partitions via Isoperimetric inequality. By invoking results from (Milman and Neeman, 2022), the author indicates that the latent space forms a  Voronoi partition when $ m < d+1$ ($m$ is the number of modes, $d$ is the latent space dimension). Next, the authors perform several experiments on structure of the latent space of GANs, e.g., the linear separability of the latent space partition, the convexity fo the latent space partition, etc.", "summary_of_the_review": "\nThe paper bridges the configuration of the latent space of a generator and the precision through Isoperimetric inequality. The idea is interesting but there are several weaknesses in both the theory and the experiments.", "strength_and_weaknesses": "Strengths:\n\nThe paper poses an interesting angel towards understanding generative models (not restricted to GANs). A Lipschitz generator of cannot map a unimodal distribution to disconnected distributions. Therefore, there is a tradeoff between mode-collapse and precision of the generator. Through Gaussian Isoperimetric inequality, the simplicity and regularity of the latent space (linearity & convexity) can be related to the quality of the generation. \n\n\nWeaknesses:\n\nTheory:\n\n1. The assumptions are too strong and fail to capture the essence of the real generative models. \n\n(1) The theory depends on the $L_2$ Lipschitz continuity $L$ of the generator network. For deep models $L$ is often a very large constant. Just think about the existence of adversarial examples. \n\n(2) The abstraction of the target distribution is artificial. The paper assumes the target distribution is a mixture of $m$ balls of equal distance. This assumption can be readily relaxed without modifying the main theorems, e.g., just defind $D$ as the minimum distance between those modes. \n\n2. The technical contributions of the theorems are weak. \n\n(1) The only interesting theoretical result is Lemma 1, which bridges the surface area of the latent space partition and the precision of the generator. The result is weak since empirically $D/L$ is very small and hence the upper bound is loose. Then the behavior of the generator is influenced by other factors.   \n\n(2) Theorem 1 & 2 are straight applications of the Gaussian Isoperimetric inequality.\n\nExperiments: \n\n1. The experiments are based on an implicit assumption that different classes correspond to different disconnected \"modes\". Besides, the empirical results (LogReg Acc & Convex Acc) are weak, suggesting latent space partition according to different classes is not predicted by the theory. \n\n2. Several experiments are disconnected from the theory. E.g., latent space dimension, disentangling the manifold dimension, and the Impact of overparametrization. Those experiments are far-fetched from the theory part.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4GoNniomps", "reviewer": "Reviewer_GvAm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper adapts recent results on the Gaussian isoperimetric inequality to study the latent space organization of GANs with disconnected modes. This proves bounds on the precision of GANs in certain circumstances, and implies various properties about how these latent spaces \"should\" be organized.", "summary_of_the_review": "An interesting paper bringing recent mathematical results to a relevant community who probably wouldn't otherwise hear about them, while also exploring GAN-specific properties in an interesting way. Although the paper has its limitations, I think it's a nice step in the understanding of latent-space generative models.", "strength_and_weaknesses": "Strengths:\n- The paper gives a clear exposition of a fairly thorough examination of the results in question.\n- It increases our understanding of \"optimal\" latent space organization for Gaussian latents.\n\nWeaknesses:\n- The scope of the results is somewhat limited.\n- A few interesting questions are raised without being fully explored.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject