Title: UNIVERSALITY THEOREMS FOR GENERATIVE MOD-ELS

Abstract: Despite the fact that generative models are extremely successful in practice, the theory underlying this phenomenon is only starting to catch up with practice. In this work we address the question of the universality of generative models: is it true that neural networks can approximate any data manifold arbitrarily well? We provide a positive answer to this question and show that under mild assumptions on the activation function one can always find a feedforward neural network that maps the latent space onto a set located within the specified Hausdorff distance from the desired data manifold. We also prove similar theorems for the case of multiclass generative models and cycle generative models, trained to map samples from one manifold to another and vice versa.

1. INTRODUCTION:

Generative models such as Generative Adversarial Networks (GANs) are widely used for tasks such as image synthesis, semi-supervised learning, and domain adaptation (Brock et al., 2018; Radford et al., 2015; Zhang et al., 2017; Isola et al., 2017) . Such generative models are trained to perform a mapping from a latent space of a small dimension to some specified data manifold, typically represented by a dataset of natural images. Despite their success and excellent performance, the theory behind such models is not yet well understood. A recent survey of open questions about generative models (Odena, 2019) among others presents the following question: what sorts of distributions can GANs model? In particular, what does it even mean for a GAN to model a distribution?

To answer these questions we adopt the following geometric approach, very amenable to precise mathematical analysis. Under the assumption of the Manifold Hypothesis (Goodfellow et al., 2016) , data comes from a certain data manifold. Then the goal of a generator network is to reproduce this data manifold as closely as possible by mapping the latent space into the ambient space of the data manifold. This intuitive understanding can be written more concretely as follows. Suppose that we are given the latent space M z , feedforward neural network f θ as a generator, and some target data manifold M. In order for the manifold M to be generated by f θ we require that the image of M z under f θ is sufficiently close to M, more specifically that the Hausdorff distance between f θ (M z ) and M is less than the given parameter ε. Hausdorff distance is a well-defined metric on the space of all compact subsets of Euclidean space and hence is equal to zero if and only if f θ (M z ) = Mthe case of precise replication of the data manifold. Thus, the question at hand can be formulated as follows: is it possible to approximate in the sense of the Hausdorff distance an arbitrary compact (connected) manifold using standard feedforward neural networks? By combining techniques from Riemannian geometry with well-known properties of neural networks we provide a positive answer to this question. We also show that the condition of being smooth is not necessary and the results are also valid for just topological manifolds.

We further extend the discussed geometric approach for the theoretical analysis of many practical situations, for instance, to the case of data manifolds, which consist of multiple disjoint manifolds and correspond to multiclass datasets, and cycle generative models (Zhu et al., 2017; Isola et al., 2017) , which for two manifolds learn an approximately invertible mapping from one manifold to another. For the latter case we prove a somewhat surprising result that for any given pair of data manifolds of the same dimension, one can always train a pair of neural networks which are approximately inverses of one another, and map the first manifold almost onto the second one, and vice versa. In this work, we ignore specifics of the training algorithm (for instance, what loss function is used) and merely focus on understanding the generative capabilities of neural networks.

2. RELATED WORK:

A large body of papers is devoted to analyzing the universality of neural networks. Classical works on universality (Cybenko, 1989; Hornik, 1991; Haykin, 1994; Hassoun et al., 1995) prove that neural networks with one hidden layer are universal approximators and can approximate arbitrary continuous functions on compact sets. Similar results also stand for deep wide networks with ReLU nonlinearities (Lu et al., 2017) , convolutional neural networks (Cohen & Shashua, 2016) and recurrent neural networks (Khrulkov et al., 2019) .

GANs were mostly studied from point of view of convergence properties (Feizi et al., 2017; Balduzzi et al., 2018; Lucic et al., 2018) . Several works focus on the relationship between geometric properties of datasets and the behavior of GANs. To analyze what characteristics of datasets lead to better convergence, synthetic datasets were studied in (Lucic et al., 2018) . A case of disconnected data manifold (similar in spirit to our analysis in Section 5) was analyzed in (Khayatkhoei et al., 2018) . A metric for analyzing the quality of GANs based on comparing geometric properties of the original and generated datasets was proposed in (Khrulkov & Oseledets, 2018) .

3. NOTATION AND ASSUMPTIONS:

We will denote the d-cube [-1, 1] d by I d . We will often use an approximation of a continuous function by a neural network, in that case, the "network version" of the function will be indicated by a subscript θ or φ indicating a collection of trainable parameters, e.g., f θ or g φ .

In this work, we deal with data manifolds. We assume that all these manifolds are smooth, orientable, compact and connected unless stated explicitly. We also assume that all the manifolds are embedded into a Euclidean space R n , and inherit the Riemannian metric tensor g. By smooth we will mean infinitely differentiable manifolds (functions), i.e, of class C ∞ ; all the results, however, will stay true if we consider class C r for some finite r. As a norm of a function f defined on some compact set D we will use the C-norm: f D = max x∈D |f (x)|, and for vectors we use the 2-norm.

We will often make use of a natural geometric measure µ on a manifold, which can be constructed by integrating the volume form associated with the Riemannian metric tensor over the corresponding set.

4. BACKGROUND:

Let us first present some background material necessary for understanding the proofs. We will freely use the term manifold in the precise mathematical sense. Due to limited space, we do not provide the definition and refer the reader to thorough introductions such as (Lee, 2013; Sakai, 1996) .

First important construction in the proof is the exponential map.

4.1. EXPONENTIAL MAP:

Let M be a Riemannian manifold endowed with a metric tensor g. Recall that geodesics are locally length minimizing curves, defined as a solution of a certain second-order differential equation. An important property of geodesics is that the length of the velocity vector is preserved along the curve, i.e., for a geodesic γ(t) we have

EQUATION

.

The exponential map is defined in the following manner. Let q ∈ M and v ∈ T p M, and suppose that there exists a geodesic γ : [0, 1] → M with

γ(0) = q, γ(0) = v.

Then the point γ(1) ∈ M is denoted by exp q (v) and called the exponential of the tangent vector v. The geodesic γ can then be written as γ(t) = exp q (vt). While apriori the exponential map is defined only if v is small enough, for certain class of manifolds it is globally defined. Namely, if a manifold is geodesically complete, then exp q (v) is defined for all q and v ∈ T q M. Our proof is based on the following classical result. Theorem 4.1 (Hopf-Rinow). Let (M, g) be a connected Riemannian manifold. Then the following statements are equivalent.

• The closed and bounded subsets of M are compact;

• M is a complete metric space;

• M is geodesically complete.

Furthermore, any of the above implies that any points p and q in M can be connected by a minimal (length-minimizing) geodesic.

In particular, this implies that any compact connected manifold M is geodesically complete.

4.2. HAUSDORFF DISTANCE:

The Hausdorff distance between two sets X, Y ⊂ R n is defined as follows.

EQUATION

where

EQUATION

It is well-known that the set of all compact subsets of R n endowed with the Hausdorff distance becomes a complete metric space (Henrikson, 1999) .

4.3. UNIVERSAL APPROXIMATION PROPERTY OF NEURAL NETWORKS:

In this paper we heavily rely on the ability of neural networks to approximate functions. In particular, we use the following classical results on neural networks (Cybenko, 1989; Hornik, 1991) . Theorem 4.2 (Universal Approximation Theorem). Let φ : R → R be a nonconstant, bounded and continuous function. Then for any continuous function f : I n → R and ε > 0 there exists a fully connected neural network f θ with the activation function φ and one hidden layer, such that

max x∈In |f (x) -f θ (x)| < ε.

Similarly, our analysis is also valid for deep networks with Rectified Linear Unit (ReLU) nonlinearities by means of the following result (Arora et al., 2018, Theorem 2.2 ). Theorem 4.3 (Arora et al. (2018) ). Every piecewise linear function R n → R can be represented by a ReLU DNN with at most log 2 (n + 1) + 1 depth.

Since piecewise linear functions are dense in the space of continuous function, we obtain a simple corollary.

Corollary 4.1. For any continuous function f : I n → R and ε > 0 there exists a fully connected neural network f θ with ReLU nonlinearity, such that

max x∈In |f (x) -f θ (x)| < ε.

For conciseness, we will call nonlinearities satisfying Theorem 4.2 or Theorem 4.3 simply universal nonlinearities, explicitly specifying other required properties when necessary. Similarly, our results can also be extended to the case of non-compact latent space R d . In Appendix B we provide relevant theorems and show how results of next sections generalize.

5. GEOMETRIC UNIVERSALITY THEOREM:

In this section we prove that for an arbitrary manifold it is possible to construct a neural network, mapping the cube I d approximately onto this manifold. Our analysis is based on the following lemma.

In fact, this is a particular case of a much stronger theorem valid even for topological manifolds (without smooth structure), for which we provide a discussion and reference further in the text. We, however, believe that this particular case is instructive and provides an intuition on how the generative mappings may look like. Lemma 5.1. Let M ⊂ R n be a compact connected d-dimensional manifold. Then there exists a smooth map f :

I d → R n , such that f (I d ) = M.

Proof. We will construct this map explicitly. Choose an arbitrary point q ∈ M, and consider

exp q : T q M → M.

Since M is compact and connected, it is geodesically complete and the Hopf-Rinow theorem applies. Thus, this map is defined on T q M ∼ = R d and surjective.

We now need to show that we can choose a compact subset of T q M such that the restriction of exp q to this subset is also surjective. To do this observe that since M is compact it has finite diameter, namely ∀p, q : d(p, q) ≤ R 0 for some finite constant R 0 . Here d is the Riemannian distance, defined as the arc length of a minimizing geodesic. From Eq. ( 1) it instantly follows that for the (Euclidean) ball

B R0 = {v ∈ T q M : v ≤ R 0 } we have exp q (B R0 ) = M.

Indeed, since any point on M is within distance R 0 from q, there exists a minimal geodesic connecting these points with length bounded by R 0 . But for any vector v ∈ T q M from Eq. ( 1) we obtain that the length of the corresponding geodesic connecting q and exp q (v) is exactly v , which proves the claim. Statement of the lemma then follows after selecting an arbitrary cube containing B R0 and appropriate rescaling. Previously we have noted that our Lemma 5.1 is a particular case of a much stronger result (Brown, 1962) . Namely, it can be stated as follows.

z M M ✓ f (z) f ✓ (z) kf f✓kI 2 < " I 2

Lemma 5.2 (Brown's mapping theorem). Let M be a compact connected d-dimensional topological manifold (with or without boundary). Then there exists a continuous map

f : I d → R n , such that f (I d ) = M.

Based on this lemma Theorem 5.1 can be generalized to include the more general case of topological data manifolds (as well as of manifolds with boundary). Corollary 5.1 (Strong Geometric Universality). Theorem 5.1 holds true for M being an arbitrary compact connected topological manifold with or without boundary.

This class of manifolds is extremely general, and it seems plausible that manifolds of natural images satisfy these conditions, which may partially explain the success of generative models. Indeed, spaces of natural images of shape H × W × C are closed subsets of I HW C , and thus are compact.

We hypothesize that the property of being connected holds for manifolds representing single-class datasets. We will now address the case of multiclass manifolds.

Multiclass case:

The previous theorem considers only the case of a single data manifold. However, commonly in practice, single datasets contain samples from multiple data manifolds (e.g, MNIST digits, ImageNet classes). Since we can assume that these manifolds do not intersect, it is impossible to map a connected latent space surjectively onto this disconnected joint data manifold. To counteract this effect we can allow small pieces of latent space to map into thin "tunnels" connecting those manifolds. This can be made precise by the following statement. Theorem 5.2 (Geometric Universality for Multiclass Manifolds). Let M = c i=1 M i be a "multiclass" data manifold, with each M i being a compact connected d-dimensional topological manifold (with or withour boundary). Then for every ε > 0 and δ > 0 and every universal nonlinearity σ there exists a fully connected neural network f θ (z) : I d → R n with the activation function σ such that the following properties hold.

• There exists a collection {D i } c i=1 of disjoint compact subsets of

I d such that ∀i d H (f θ (D i ), M i ) < ε. (4) • µ(I d \ c i=1 D i ) ≤ δ.

We refer the reader to Appendix A for the proof.

6. INVARIANCE PROPERTY OF DEEP EXPANDING NETWORKS:

Our previous results state that it is possible to approximate any given manifold M up to some accuracy. However, neural networks used in the proof are shallow (they have one hidden layer) and are not practical. In this section, we study how the set M θ looks like for more practical networks consisting of a series of fully connected and convolutional layers. We will show a somewhat surprising result that under certain mild conditions such networks cannot significantly transform the latent space, more precisely the generated set M θ will be diffeomorphic to the open unit cube (-1, 1) d . In fact, our results will be more general and will demonstrate that this property holds for arbitrary latent spaces, that is if z is sampled from some manifold M z , then M θ will be diffeomorphic to M z .

6.1. REMINDER ON EMBEDDINGS:

Recall the following definition. Definition 6.1 (Smooth embedding). Let M and N be smooth manifolds and f : M → N be a smooth map. Then f is called an embedding is the following conditions hold.

• Derivative of f is everywhere injective;

• f is an injective, continuous and open map (i.e, maps opens sets to open sets).

The main property of a smooth embedding is the following (Lee, 2013). Proposition 6.1. The domain of an embedding is diffeomorphic to its image.

We will show that certain neural networks commonly used for generative models are in fact smooth embeddings, and thus their image is diffeomorphic to the domain (latent space). We analyze the two most commonly used layers in such models: fully connected and convolutional layers (both standard and transposed). For the sake of simplicity we assume that convolutions are circularly padded, i.e., the input presents a two-dimensional torus; in this case, when the offset calls for a pixel that is off the left end of the image, the layer "wraps around" to take it from the opposite end. We consider arbitrary stride, in order to allow for a layer to increase the spatial size of a feature tensor, as commonly done.

Let us fix the nonlinearity σ(z) to be an arbitrary smooth monotonous function without saddle points (σ (z) = 0). Then the following two lemmas hold. Let us first assume that the latent space is the Euclidean space R d (or equivalently, an open unit cube (-1, 1) d ). Lemma 6.1. Let f (z) = σ(Az + b) with A ∈ R n×m be a fully connected layer. If n ≥ m then f (z) is a smooth embedding for all A except for a set of measure zero. We will call such a layer an expanding fully connected layer.

Proof. Indeed, such a map is injective. It is open as a composition of a linear map (which is trivially open), and of σ(z) which is open since it is a continuous monotonous function. Then for all matrices A of full rank (which form a set of full measure in the space of matrices of size n × m) the derivative is injective by a simple application of the chain rule and the fact that σ (z) = 0.

Let us now deal with the convolutional layers. Lemma 6.2. Let z be a 3rd-order tensor tensor representing a feature tensor of size m × m with k channels. Suppose that f (z) = σ(Conv(z) + b) is a standard convolutional or transposed convolutional layer with an arbitrary stride. Suppose that Conv is parameterized via a kernel parameter C ∈ R l×k×s×s , such that f (z) is a feature tensor of size n × n with l channels. If n 2 l ≥ m 2 k then f (z) is a smooth embedding for all C except for a set of measure zero. We will call such a layer an expanding convolutional layer.

Proof. The only non-trivial part of the proof is showing injectivity of this layer for all C but measure zero. Note that if n 2 l ≥ m 2 k then the matrix representing the linear map performing the Conv operation is vertical, hence it is sufficient to show that generically it is of full rank. In the case of the transposed convolution, we can transpose this matrix and analyze the corresponding convolutional layer.

Stride one Let us start with the most important case of stride being one, in which case m = n. Denote the matrix of the linear map underlying Conv by C ∈ R n 2 l×n 2 k , that is vec(Conv(x)) = Cvec(x), where vec denotes the vectorization operator. We need to show that for all C but measure zero this matrix is of full rank.

To prove the lemma we use the following simple argument coming from algebraic geometry. The condition of matrix C not being a full rank is algebraic (i.e., is given by polynomial equations) in the space of parameters C. Indeed, the operation of constructing C based on C is linear with respect to C, and the condition of not being a full rank in the space of all matrices is specified by a set of polynomial equations (namely, determinants of all maximal square submatrices should be zero). Thus, we have shown that set C singular = {C ∈ R l×k×s×s | C is not of full rank} is algebraic; and by the well-known property of algebraic sets there are two options: either µ(C singular ) = 0 or C singular = R l×k×s×s (with µ being the standard Lebesgue measure). To show that the latter does not hold, we provide a concrete example of a weight C not in C singular . Namely, consider the following C.

EQUATION

Here δ ij denotes the Kronecker delta symbol:

δ ij = 1, i = j, 0, i = j.

We observe that the corresponding matrix C is of particularly simple structure:

C[i, j] = δ ij ,

which trivially is of full rank.

Arbitrary stride The same argument as before applies. Notice that selection of a bigger stride corresponds to selecting specific rows from the matrix C obtained for stride one. By using the same weight tensor C as in the case of stride one, we find that the obtained matrix C contains min(m 2 k, n 2 l) distinct rows of the identity matrix, followed by possible zero rows and thus also has full rank.

After these preliminary results, we are ready to extend them to the case of arbitrary latent space. Namely, suppose that z is sampled from an arbitrary manifold M z ⊂ R d . We use the following simple lemma and refer the reader to Appendix A for the proof. Lemma 6.3. Let f : M → N be an arbitrary smooth embedding. Let S ⊂ M be a smooth embedded submanifold. Then f | S is also a smooth embedding.

By combining Lemmas 6.1 to 6.3 and Proposition 6.1 we obtain the following result. Theorem 6.1. Let f θ (z) be an arbitrary neural network consisting of expanding fully connected layers and expanding convolutions, and

z ∈ M z ⊂ R d . Denote M θ = f θ (M z ).

Then for all parameters θ but measure zero the following properties hold:

• M θ is a smooth embedded manifold;

• M θ M z .

Proof. Theorem follows from Lemmas 6.1 to 6.3 and Proposition 6.1 and the fact that a composition of embeddings is also an embedding.

For many datasets used in practice, it seems very unlikely that the data comes from manifolds with very simple topological properties, as even basic visual patterns may possess quite non-trivial topological structure (Ghrist, 2008) . Thus on the first sight, it seems that Theorem 6.1 suggests that using only expanding architectures, it is impossible to approximate an arbitrary data manifold with latent space being R d (or an open unit cube). Such models are, however, extremely successful in practice. While we do not provide a precise theorem for this case, based on the discussion in Section 7, we hypothesize that it may possible to approximate an arbitrary compact data manifold using expanding networks up to a subset of arbitrary small measure, and thus limitations imposed by Theorem 6.1 are negligible in practice.

7. CYCLE GENERATIVE MODELS:

Another popular class of models used for instance for the unsupervised image to image translation (Zhu et al., 2017; Isola et al., 2017) learn a mapping along with its inverse from one data manifold to another. We specify this task as follows. Given two data manifolds M and N of the same dimension, the goal is two train two neural networks f θ (x) and g φ (y) such that f θ (x) is a diffeomorphism of M and N with g being inverse of f .

First of all, let us notice that we do not expect for such f and g to exist for two general manifolds since two manifolds of different topological properties cannot be diffeomorphic. However, based on Theorem 6.1 we expect that the desired properties may hold approximately. Let us start with lemmas ensuring existence of functions f and g which map M approximately to N and N approximately to M correspondingly. In this section, we again consider only the case of smooth data manifolds.

First of all, we recall the following result (Sakai, 1996) , proved in a very similar manner to Lemma 5.1. We use this result to obtain the following lemma (see Appendix A for the proof). Lemma 7.2. Let M and N be two manifolds of the same dimension. For every δ > 0 there exist compact subsets M δ ⊂ M and N δ ⊂ N such that µ(M \ M δ ) < δ and µ(N \ N δ ) < δ and M δ is diffeomorphic to N δ .

We are now ready to provide our main result on cycle generative models. As before, recall from Section 4.3 that universal nonlinearities include nonconstant bounded continuous functions as well as ReLU.

Theorem 7.1 (Geometric Universality for Cycle Models). Fix any two compact connected manifolds M and N of the same dimension and a universal nonlinearity σ. Then for every δ > 0 and ε > 0 there exist compact subsets M δ ⊂ M and N δ ⊂ N and a pair of feedforward neural networks f θ (x), g φ (y) with the activation function σ(x) satisfying the following conditions:

• µ(M \ M δ ) < δ and µ(N \ N δ ) < δ; Proof. Let us start by selecting subsets M δ and N δ and a diffeomorphism f : M δ → N δ along with its inverse g as specified by Lemma 7.2. For simplicity let us also assume that M ⊂ I n and N ⊂ I n . By means of the Whitney extension theorem (Whitney, 1934) we can smoothly extend f and g to the entire cube I n , and by universality of σ we construct two feedforward neural networks f θ (x) and g φ (y) such that f θf In < ε, (6) and g φg In < ε, (7) with all the functions defined on the unit cube I n . This proves first two points in the theorem. To show the last property we find that ∀x ∈ M δ the following estimate holds.

• d H (f θ (M δ ), N δ ) < ε and d H (g φ (N δ ), M δ ) < ε; • g φ • f θ -id M δ < Cε and f θ • g φ -id N δ <

EQUATION

where he have used the fact that g • f (x) = x for x ∈ M δ and properties ( 6) and ( 7). The second part of the claim is proved similarly.

Neural networks f θ and g φ constructed in the proof perform translation from data sampled from M δ to data coming from approximately N δ , and existence of such networks for arbitrary manifolds may partially explain huge empirical success of cyclic models. Even though the theorem is valid for an arbitrary pair of manifolds, we hypothesize that for datasets containing visually similar images such a map may be much easier to model, than for two arbitrary manifolds without such a connection.

Latent space R d Results we provided in previous sections are valid for the compact latent space I d . However, we can generalize them to include the other popular case of latent space being the entire space R d (even though for a smaller set of activation functions). See Appendix B for the discussion of this case.

8. CONCLUSION AND FUTURE WORK:

In this work we have attempted to partially explain huge empirical success of generative models.

Our results show only existence of neural networks approximating arbitrary manifolds, and do not specify how one can estimate the size of a network required for any given manifold. We hypothesize, however, that there might exist a connection between certain geometrical properties of a manifold (curvature, various topological properties), and the width/depth of a neural network required. One interesting direction of research left for a future work is analyzing this relation for datasets popular in computer vision, such as MNIST or CelebA, or toy datasets sampled from simple small dimensional manifolds (tori, circles), where one can easily vary the topological properties.

:



Figures and Tables:

Figure 1: Visualization of the construction in the proof of Theorem 5.1. The latent space I 2 is mapped onto the manifold M via the function f . This mapping is then approximated via neural network f θ , which in turn maps I 2 onto the compact set M θ . If f θ is sufficiently close to f then so are M and M θ .Recall from Section 4.3 that universal nonlinearities include ReLU and nonconstant bounded continuos functions. Theorem 5.1 (Geometric Universality of Generative Models). Let M be a compact connected d-dimensional manifold. For every universal nonlinearity σ there exists a fully connected neural network f θ (z) :I d → R n with the activation function σ such that d H (M, M θ ) < ε. Here M θ = f θ (I d ).Proof. Choose an arbitrary f as in Lemma 5.1. By universality of σ we can find such a neural network that ff θ I d < ε. Statement of the theorem then follows from the definition of the Hausdorff distance. Indeed, by surjectivity of f we find that every point x 0 = f (z 0 ) ∈ M is within distance ε from the point f θ (z 0 ) ∈ M θ , and thus M ⊂ [M θ ] ε as in Eq. (3), and conversely M θ ⊂ [M] ε . See Fig. 1 for illustration of the proof.

Lemma 7.1. Every compact connected d-dimensional manifold M contains an open dense set diffeomorphic to R d . Moreover, complement of this set has measure zero in M.

Cε with constant C depending only on manifolds M and N .

Peer review (iclr2020):
This paper theoretically shows that data manifolds can be approximated by neural-network-based generative models including GANs. In particular, they prove that under mild assumptions neural networks can map a given latent space onto a set which is close to the given data manifold within a small Hausdorff distance. There results hold for the case of both single data manifold (i.e. data generated from a single class) and multiple data manifolds (i.e. data generated from multiple classes). In the latter, they need to allow insignificant parts of the latent space to be mapped onto thin “tunnels” connecting the manifolds. The construction of the mapping from the latent space onto the data manifold provided in their proofs is only for shallow networks. Section 6 of the paper - Invariance Property of Deep Expanding Networks - characterizes the property of the manifold computed by deep neural networks that expanding the dimensionality of the input. In particular, they show that is a smooth embedded manifold and it is diffeomorphic to the latent manifold , which shows that neural networks cannot significantly transform the latent space. The paper also extends the results for the geometric universality of generative models to cycle generative models including the CycleGANs or Pix2Pix architectures. In particular, they show that under mild conditions, there exists a pair of neural networks that can approximately map back and forth between two given data manifolds. Potentially, this paper could be an interesting contribution. However, some points in the theory need to be verified, and at least toy experiments should be provided to validate their theoretical claims. Furthermore, Section 6 - Invariance Property of Deep Expanding Networks - does not seem related to the main story of the paper and should be a separate paper itself. I weakly reject this paper, but given these clarifications in an author response, I would be willing to increase the score. Main arguments and questions: 1. Theorem 5.1 and 5.2 - Geometric Universality of Generative Models for signal class and multiclass case - seem to be straight-forward applications of Lemma 5.1 and Theorem 4.2. Lemma 5.1 is a special case of the Brown’s mapping theorem and Theorem 4.2 is the well-known theorem on the universal approximation power of neural networks. I, therefore, question the significance of Theorem 5.1 and 5.2. 2. Since this is clearly a theory paper, lack of experiments can be tolerated. However, the paper does not provide even a single toy simulation result to at least shed light on their theoretical claims. For example, the results from Theorem 6.1 which show that neural networks cannot significantly transform the latent space can be empirically justified. 3. Will the results in Section 5 and 7 hold for Convolutional Neural Networks, or only for Fully Connected Networks? 4. Theorem 6.1 shows that the manifold computed by a neural network is diffeomorphic to the latent manifold . However, the mappings by deep generative models like GANs, in most cases, are not invertible. It would be great if the authors can elaborate this point more. Otherwise, the results in Theorem 6.1 are questionable. 5. The paper admits that according to theorem 6.1, “it is impossible to approximate an arbitrary data manifold with latent space being R^{d}”, but in practice, deep generative models like GANs can generate good-looking images. As mentioned in the paper, a precise theorem for this case is not provided. The authors, however, say they “hypothesize that it may be possible to approximate an arbitrary compact data manifold using expanding networks up to a subset of arbitrary small measure, and thus limitations imposed by Theorem 6.1 are negligible in practice.”. However, I didn’t see how this hypothesis is related to the results and discussion in Section 7. It would be great if the authors can provide more explanation for this hypothesis and its connection to the results in Section 7. 6. Geometric universality does not guarantee that a good approximation to data manifolds can be found after training the generative model. The optimization, especially the objective losses used to train the model, plays an important role in understanding the manifold represented by the trained model. As a result, I am not quite sure about the significance of this paper and would appreciate if the authors can discuss the relevance of their results. Things to improve the paper that did not impact the score: 1. The paper should define what fully connected neural networks are. 2. Some notations in the paper need to be defined such as the ? in Theorem 6.1. 3. More discussion on exponential map and the intuitions behind it should be provided. Minor comments: There are some typos in the paper such as “continuos” in page 4, “an embedding is the following” in Definition 6.1, “may possible” in page 7, and “is two train two neural networks” in page 7.