user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Universality Theorems for Generative Models

Paper content:
Despite the fact that generative models are extremely successful in practice, the theory underlying this phenomenon is only starting to catch up with practice.
In this work we address the question of the universality of generative models: is it true that neural networks can approximate any data manifold arbitrarily well?
We provide a positive answer to this question and show that under mild assumptions on the activation function one can always find a feedforward neural network that maps the latent space onto a set located within the specified Hausdorff distance from the desired data manifold.
We also prove similar theorems for the case of multiclass generative models and cycle generative models, trained to map samples from one manifold to another and vice versa.
Generative models such as Generative Adversarial Networks (GANs) are widely used for tasks such as image synthesis, semi-supervised learning, and domain adaptation (Brock et al., 2018; Radford et al., 2015; Zhang et al., 2017; .
Such generative models are trained to perform a mapping from a latent space of a small dimension to some specified data manifold, typically represented by a dataset of natural images.
Despite their success and excellent performance, the theory behind such models is not yet well understood.
A recent survey of open questions about generative models (Odena, 2019) To answer these questions we adopt the following geometric approach, very amenable to precise mathematical analysis.
Under the assumption of the Manifold Hypothesis (Goodfellow et al., 2016) , data comes from a certain data manifold.
Then the goal of a generator network is to reproduce this data manifold as closely as possible by mapping the latent space into the ambient space of the data manifold.
This intuitive understanding can be written more concretely as follows.
Suppose that we are given the latent space M z , feedforward neural network f θ as a generator, and some target data manifold M. In order for the manifold M to be generated by f θ we require that the image of M z under f θ is sufficiently close to M, more specifically that the Hausdorff distance between f θ (M z ) and M is less than the given parameter ε.
Hausdorff distance is a well-defined metric on the space of all compact subsets of Euclidean space and hence is equal to zero if and only if f θ (M z ) = Mthe case of precise replication of the data manifold.
Thus, the question at hand can be formulated as follows: is it possible to approximate in the sense of the Hausdorff distance an arbitrary compact (connected) manifold using standard feedforward neural networks?
By combining techniques from Riemannian geometry with well-known properties of neural networks we provide a positive answer to this question.
We also show that the condition of being smooth is not necessary and the results are also valid for just topological manifolds.
We further extend the discussed geometric approach for the theoretical analysis of many practical situations, for instance, to the case of data manifolds, which consist of multiple disjoint manifolds and correspond to multiclass datasets, and cycle generative models , which for two manifolds learn an approximately invertible mapping from one manifold to another.
For the latter case we prove a somewhat surprising result that for any given pair of data manifolds of the same dimension, one can always train a pair of neural networks which are approximately inverses of one another, and map the first manifold almost onto the second one, and vice versa.
In this work, we ignore specifics of the training algorithm (for instance, what loss function is used) and merely focus on understanding the generative capabilities of neural networks.
A large body of papers is devoted to analyzing the universality of neural networks.
Classical works on universality (Cybenko, 1989; Hornik, 1991; Haykin, 1994; Hassoun et al., 1995) prove that neural networks with one hidden layer are universal approximators and can approximate arbitrary continuous functions on compact sets.
Similar results also stand for deep wide networks with ReLU nonlinearities (Lu et al., 2017) , convolutional neural networks (Cohen & Shashua, 2016) and recurrent neural networks (Khrulkov et al., 2019) .
GANs were mostly studied from point of view of convergence properties (Feizi et al., 2017; Balduzzi et al., 2018; Lucic et al., 2018) .
Several works focus on the relationship between geometric properties of datasets and the behavior of GANs.
To analyze what characteristics of datasets lead to better convergence, synthetic datasets were studied in (Lucic et al., 2018) .
A case of disconnected data manifold (similar in spirit to our analysis in Section 5) was analyzed in (Khayatkhoei et al., 2018) .
A metric for analyzing the quality of GANs based on comparing geometric properties of the original and generated datasets was proposed in (Khrulkov & Oseledets, 2018) .
We will denote the d-cube [−1, 1] d by I d .
We will often use an approximation of a continuous function by a neural network, in that case, the "network version" of the function will be indicated by a subscript θ or φ indicating a collection of trainable parameters, e.g., f θ or g φ .
In this work, we deal with data manifolds.
We assume that all these manifolds are smooth, orientable, compact and connected unless stated explicitly.
We also assume that all the manifolds are embedded into a Euclidean space R n , and inherit the Riemannian metric tensor g. By smooth we will mean infinitely differentiable manifolds (functions), i.e, of class C ∞ ; all the results, however, will stay true if we consider class C r for some finite r. As a norm of a function f defined on some compact set D we will use the C-norm: f D = max x∈D |f (x)|, and for vectors we use the 2-norm.
We will often make use of a natural geometric measure µ on a manifold, which can be constructed by integrating the volume form associated with the Riemannian metric tensor over the corresponding set.
Let us first present some background material necessary for understanding the proofs.
We will freely use the term manifold in the precise mathematical sense.
Due to limited space, we do not provide the definition and refer the reader to thorough introductions such as (Lee, 2013; Sakai, 1996) .
First important construction in the proof is the exponential map.
Let M be a Riemannian manifold endowed with a metric tensor g. Recall that geodesics are locally length minimizing curves, defined as a solution of a certain second-order differential equation.
An important property of geodesics is that the length of the velocity vector is preserved along the curve, i.e., for a geodesic γ(t) we have
.
The exponential map is defined in the following manner.
Let q ∈ M and v ∈ T p M, and suppose that there exists a geodesic γ : [0, 1] → M with
Then the point γ(1) ∈ M is denoted by exp q (v) and called the exponential of the tangent vector v. The geodesic γ can then be written as γ(t) = exp q (vt).
While apriori the exponential map is defined only if v is small enough, for certain class of manifolds it is globally defined.
Namely, if a manifold is geodesically complete, then exp q (v) is defined for all q and v ∈ T q M. Our proof is based on the following classical result.
Theorem 4.1 (Hopf-Rinow).
Let (M, g) be a connected Riemannian manifold.
Then the following statements are equivalent.
•
The closed and bounded subsets of M are compact;
• M is a complete metric space;
• M is geodesically complete.
Furthermore, any of the above implies that any points p and q in M can be connected by a minimal (length-minimizing) geodesic.
In particular, this implies that any compact connected manifold M is geodesically complete.
The Hausdorff distance between two sets X, Y ⊂ R n is defined as follows.
where
It is well-known that the set of all compact subsets of R n endowed with the Hausdorff distance becomes a complete metric space (Henrikson, 1999) .
In this paper we heavily rely on the ability of neural networks to approximate functions.
In particular, we use the following classical results on neural networks (Cybenko, 1989; Hornik, 1991) .
Theorem 4.2 (Universal Approximation Theorem).
Let φ : R → R be a nonconstant, bounded and continuous function.
Then for any continuous function f : I n → R and ε > 0 there exists a fully connected neural network f θ with the activation function φ and one hidden layer, such that
Similarly, our analysis is also valid for deep networks with Rectified Linear Unit (ReLU) nonlinearities by means of the following result (Arora et al., 2018 , Theorem 2.2).
Theorem 4.3 (Arora et al. (2018) ).
Every piecewise linear function R n → R can be represented by a ReLU DNN with at most log 2 (n + 1) + 1 depth.
Since piecewise linear functions are dense in the space of continuous function, we obtain a simple corollary.
Corollary 4.1.
For any continuous function f : I n → R and ε > 0 there exists a fully connected neural network f θ with ReLU nonlinearity, such that
For conciseness, we will call nonlinearities satisfying Theorem 4.2 or Theorem 4.3 simply universal nonlinearities, explicitly specifying other required properties when necessary.
Similarly, our results can also be extended to the case of non-compact latent space R d .
In Appendix B we provide relevant theorems and show how results of next sections generalize.
In this section we prove that for an arbitrary manifold it is possible to construct a neural network, mapping the cube I d approximately onto this manifold.
Our analysis is based on the following lemma.
In fact, this is a particular case of a much stronger theorem valid even for topological manifolds (without smooth structure), for which we provide a discussion and reference further in the text.
We, however, believe that this particular case is instructive and provides an intuition on how the generative mappings may look like.
Lemma 5.1.
Let M ⊂ R n be a compact connected d-dimensional manifold.
Then there exists a smooth map
Proof.
We will construct this map explicitly.
Choose an arbitrary point q ∈ M, and consider
Since M is compact and connected, it is geodesically complete and the Hopf-Rinow theorem applies.
Thus, this map is defined on T q M ∼ = R d and surjective.
We now need to show that we can choose a compact subset of T q M such that the restriction of exp q to this subset is also surjective.
To do this observe that since M is compact it has finite diameter, namely ∀p, q : d(p, q) ≤ R 0 for some finite constant R 0 .
Here d is the Riemannian distance, defined as the arc length of a minimizing geodesic.
From Eq. (1) it instantly follows that for the (Euclidean) ball
Indeed, since any point on M is within distance R 0 from q, there exists a minimal geodesic connecting these points with length bounded by R 0 .
But for any vector v ∈ T q M from Eq. (1) we obtain that the length of the corresponding geodesic connecting q and exp q (v) is exactly v , which proves the claim.
Statement of the lemma then follows after selecting an arbitrary cube containing B R0 and appropriate rescaling.
Figure 1: Visualization of the construction in the proof of Theorem 5.1.
The latent space I 2 is mapped onto the manifold M via the function f .
This mapping is then approximated via neural network f θ , which in turn maps I 2 onto the compact set M θ .
If f θ is sufficiently close to f then so are M and M θ .
Recall from Section 4.3 that universal nonlinearities include ReLU and nonconstant bounded continuos functions.
Theorem 5.1 (Geometric Universality of Generative Models).
Let M be a compact connected d-dimensional manifold.
For every universal nonlinearity σ there exists a fully connected neural network f θ (z) :
Proof.
Choose an arbitrary f as in Lemma 5.1.
By universality of σ we can find such a neural network that f − f θ I d < ε.
Statement of the theorem then follows from the definition of the Hausdorff distance.
Indeed, by surjectivity of f we find that every point Previously we have noted that our Lemma 5.1 is a particular case of a much stronger result (Brown, 1962) .
Namely, it can be stated as follows.
Lemma 5.2 (Brown's mapping theorem).
Let M be a compact connected d-dimensional topological manifold (with or without boundary).
Then there exists a continuous map
Based on this lemma Theorem 5.1 can be generalized to include the more general case of topological data manifolds (as well as of manifolds with boundary).
Corollary 5.1 (Strong Geometric Universality).
Theorem 5.1 holds true for M being an arbitrary compact connected topological manifold with or without boundary.
This class of manifolds is extremely general, and it seems plausible that manifolds of natural images satisfy these conditions, which may partially explain the success of generative models.
Indeed, spaces of natural images of shape H × W × C are closed subsets of I HW C , and thus are compact.
We hypothesize that the property of being connected holds for manifolds representing single-class datasets.
We will now address the case of multiclass manifolds.
Multiclass case The previous theorem considers only the case of a single data manifold.
However, commonly in practice, single datasets contain samples from multiple data manifolds (e.g, MNIST digits, ImageNet classes).
Since we can assume that these manifolds do not intersect, it is impossible to map a connected latent space surjectively onto this disconnected joint data manifold.
To counteract this effect we can allow small pieces of latent space to map into thin "tunnels" connecting those manifolds.
This can be made precise by the following statement.
Theorem 5.2 (Geometric Universality for Multiclass Manifolds).
Let M = c i=1 M i be a "multiclass" data manifold, with each M i being a compact connected d-dimensional topological manifold (with or withour boundary).
Then for every ε > 0 and δ > 0 and every universal nonlinearity σ there exists a fully connected neural network f θ (z) :
n with the activation function σ such that the following properties hold.
• There exists a collection
We refer the reader to Appendix A for the proof.
Our previous results state that it is possible to approximate any given manifold M up to some accuracy.
However, neural networks used in the proof are shallow (they have one hidden layer) and are not practical.
In this section, we study how the set M θ looks like for more practical networks consisting of a series of fully connected and convolutional layers.
We will show a somewhat surprising result that under certain mild conditions such networks cannot significantly transform the latent space, more precisely the generated set M θ will be diffeomorphic to the open unit cube (−1, 1) d .
In fact, our results will be more general and will demonstrate that this property holds for arbitrary latent spaces, that is if z is sampled from some manifold M z , then M θ will be diffeomorphic to M z .
Recall the following definition.
Definition 6.1 (Smooth embedding).
Let M and N be smooth manifolds and f : M → N be a smooth map.
Then f is called an embedding is the following conditions hold.
• Derivative of f is everywhere injective;
• f is an injective, continuous and open map (i.e, maps opens sets to open sets).
The main property of a smooth embedding is the following (Lee, 2013).
Proposition 6.1.
The domain of an embedding is diffeomorphic to its image.
We will show that certain neural networks commonly used for generative models are in fact smooth embeddings, and thus their image is diffeomorphic to the domain (latent space).
We analyze the two most commonly used layers in such models: fully connected and convolutional layers (both standard and transposed).
For the sake of simplicity we assume that convolutions are circularly padded, i.e., the input presents a two-dimensional torus; in this case, when the offset calls for a pixel that is off the left end of the image, the layer "wraps around" to take it from the opposite end.
We consider arbitrary stride, in order to allow for a layer to increase the spatial size of a feature tensor, as commonly done.
Let us fix the nonlinearity σ(z) to be an arbitrary smooth monotonous function without saddle points (σ (z) = 0).
Then the following two lemmas hold.
Let us first assume that the latent space is the Euclidean space R d (or equivalently, an open unit cube (−1, 1) d ).
Lemma 6.1.
Let f (z) = σ(Az + b) with A ∈ R n×m be a fully connected layer.
If n ≥ m then f (z) is a smooth embedding for all A except for a set of measure zero.
We will call such a layer an expanding fully connected layer.
A of full rank (which form a set of full measure in the space of matrices of size n × m) the derivative is injective by a simple application of the chain rule and the fact that σ (z) = 0.
Let us now deal with the convolutional layers.
Lemma 6.2.
Let z be a 3rd-order tensor tensor representing a feature tensor of size m × m with k channels.
Suppose that f (z) = σ(Conv(z) + b) is a standard convolutional or transposed convolutional layer with an arbitrary stride.
Suppose that Conv is parameterized via a kernel parameter C ∈ R l×k×s×s , such that f (z) is a feature tensor of size n × n with l channels.
If n 2 l ≥ m 2 k then f (z) is a smooth embedding for all C except for a set of measure zero.
We will call such a layer an expanding convolutional layer.
Proof.
The only non-trivial part of the proof is showing injectivity of this layer for all C but measure zero.
Note that if n 2 l ≥ m 2 k then the matrix representing the linear map performing the Conv operation is vertical, hence it is sufficient to show that generically it is of full rank.
In the case of the transposed convolution, we can transpose this matrix and analyze the corresponding convolutional layer.
Stride one Let us start with the most important case of stride being one, in which case m = n. Denote the matrix of the linear map underlying Conv by C ∈ R n 2 l×n 2 k , that is vec(Conv(x)) = Cvec(x), where vec denotes the vectorization operator.
We need to show that for all C but measure zero this matrix is of full rank.
To prove the lemma we use the following simple argument coming from algebraic geometry.
The condition of matrix C not being a full rank is algebraic (i.e., is given by polynomial equations) in the space of parameters C. Indeed, the operation of constructing C based on C is linear with respect to C, and the condition of not being a full rank in the space of all matrices is specified by a set of polynomial equations (namely, determinants of all maximal square submatrices should be zero).
Thus, we have shown that set C singular = {C ∈ R l×k×s×s | C is not of full rank} is algebraic; and by the well-known property of algebraic sets there are two options: either µ(C singular ) = 0 or C singular = R l×k×s×s (with µ being the standard Lebesgue measure).
To show that the latter does not hold, we provide a concrete example of a weight C not in C singular .
Namely, consider the following C.
Here δ ij denotes the Kronecker delta symbol:
We observe that the corresponding matrix C is of particularly simple structure:
which trivially is of full rank.
Arbitrary stride The same argument as before applies.
Notice that selection of a bigger stride corresponds to selecting specific rows from the matrix C obtained for stride one.
By using the same weight tensor C as in the case of stride one, we find that the obtained matrix C contains min(m 2 k, n 2 l) distinct rows of the identity matrix, followed by possible zero rows and thus also has full rank.
After these preliminary results, we are ready to extend them to the case of arbitrary latent space.
Namely, suppose that z is sampled from an arbitrary manifold M z ⊂ R d .
We use the following simple lemma and refer the reader to Appendix A for the proof.
Lemma 6.3.
Let f : M → N be an arbitrary smooth embedding.
Let S ⊂ M be a smooth embedded submanifold.
Then f | S is also a smooth embedding.
By combining Lemmas 6.1 to 6.3 and Proposition 6.1 we obtain the following result.
Theorem 6.1.
Let f θ (z) be an arbitrary neural network consisting of expanding fully connected layers and expanding convolutions, and
.
Then for all parameters θ but measure zero the following properties hold:
• M θ is a smooth embedded manifold;
Proof.
Theorem follows from Lemmas 6.1 to 6.3 and Proposition 6.1 and the fact that a composition of embeddings is also an embedding.
For many datasets used in practice, it seems very unlikely that the data comes from manifolds with very simple topological properties, as even basic visual patterns may possess quite non-trivial topological structure (Ghrist, 2008) .
Thus on the first sight, it seems that Theorem 6.1 suggests that using only expanding architectures, it is impossible to approximate an arbitrary data manifold with latent space being R d (or an open unit cube).
Such models are, however, extremely successful in practice.
While we do not provide a precise theorem for this case, based on the discussion in Section 7, we hypothesize that it may possible to approximate an arbitrary compact data manifold using expanding networks up to a subset of arbitrary small measure, and thus limitations imposed by Theorem 6.1 are negligible in practice.
Another popular class of models used for instance for the unsupervised image to image translation ) learn a mapping along with its inverse from one data manifold to another.
We specify this task as follows.
Given two data manifolds M and N of the same dimension, the goal is two train two neural networks f θ (x) and g φ (y) such that f θ (x) is a diffeomorphism of M and N with g being inverse of f .
First of all, let us notice that we do not expect for such f and g to exist for two general manifolds since two manifolds of different topological properties cannot be diffeomorphic.
However, based on Theorem 6.1 we expect that the desired properties may hold approximately.
Let us start with lemmas ensuring existence of functions f and g which map M approximately to N and N approximately to M correspondingly.
In this section, we again consider only the case of smooth data manifolds.
First of all, we recall the following result (Sakai, 1996) , proved in a very similar manner to Lemma 5.1.
Lemma 7.1.
Every compact connected d-dimensional manifold M contains an open dense set diffeomorphic to R d .
Moreover, complement of this set has measure zero in M.
We use this result to obtain the following lemma (see Appendix A for the proof).
Lemma 7.2.
Let M and N be two manifolds of the same dimension.
For every δ > 0 there exist compact subsets M δ ⊂ M and N δ ⊂ N such that µ(M \ M δ ) < δ and µ(N \ N δ ) < δ and M δ is diffeomorphic to N δ .
We are now ready to provide our main result on cycle generative models.
As before, recall from Section 4.3 that universal nonlinearities include nonconstant bounded continuous functions as well as ReLU.
Theorem 7.1 (Geometric Universality for Cycle Models).
Fix any two compact connected manifolds M and N of the same dimension and a universal nonlinearity σ.
Then for every δ > 0 and ε > 0 there exist compact subsets M δ ⊂ M and N δ ⊂ N and a pair of feedforward neural networks f θ (x), g φ (y) with the activation function σ(x) satisfying the following conditions:
• µ(M \ M δ ) < δ and µ(N \ N δ ) < δ;
Cε with constant C depending only on manifolds M and N .
Proof.
Let us start by selecting subsets M δ and N δ and a diffeomorphism f : M δ →
N δ along with its inverse g as specified by Lemma 7.2.
For simplicity let us also assume that M ⊂ I n and N ⊂ I n .
By means of the Whitney extension theorem (Whitney, 1934) we can smoothly extend f and g to the entire cube I n , and by universality of σ we construct two feedforward neural networks f θ (x) and g φ (y) such that
and
with all the functions defined on the unit cube I n .
This proves first two points in the theorem.
To show the last property we find that ∀x ∈ M δ the following estimate holds.
where he have used the fact that g • f (x) = x for x ∈ M δ and properties (6) and (7).
The second part of the claim is proved similarly.
Neural networks f θ and g φ constructed in the proof perform translation from data sampled from M δ to data coming from approximately N δ , and existence of such networks for arbitrary manifolds may partially explain huge empirical success of cyclic models.
Even though the theorem is valid for an arbitrary pair of manifolds, we hypothesize that for datasets containing visually similar images such a map may be much easier to model, than for two arbitrary manifolds without such a connection.
Latent space R d Results we provided in previous sections are valid for the compact latent space I d .
However, we can generalize them to include the other popular case of latent space being the entire space R d (even though for a smaller set of activation functions).
See Appendix B for the discussion of this case.
In this work we have attempted to partially explain huge empirical success of generative models.
Our results show only existence of neural networks approximating arbitrary manifolds, and do not specify how one can estimate the size of a network required for any given manifold.
We hypothesize, however, that there might exist a connection between certain geometrical properties of a manifold (curvature, various topological properties), and the width/depth of a neural network required.
One interesting direction of research left for a future work is analyzing this relation for datasets popular in computer vision, such as MNIST or CelebA, or toy datasets sampled from simple small dimensional manifolds (tori, circles), where one can easily vary the topological properties.
A PROOFS Theorem 5.2 (Geometric Universality for Multiclass Manifolds).
Let M = c i=1 M i be a "multiclass" data manifold, with each M i being a compact connected d-dimensional topological manifold (with or withour boundary).
Then for every ε > 0 and δ > 0 and every universal nonlinearity σ there exists a fully connected neural network f θ (z) : I d → R n with the activation function σ such that the following properties hold.
• There exists a collection
Proof.
Similar to the proof of Theorem 5.1 we will apply the universal approximation theorem to a certain function constructed with the help of Lemma 5.2.
To construct such function let us select sets D i in the following way.
We divide the interval [−1, 1] uniformly into c intervals, namely
Intuition is very simple: we chop down the cube D on the first axis into smaller boxes, and remove some space between them.
On each of the chunks D i we can now apply Lemma 5.2 for the corresponding manifold M i , obtaining a collection of maps
.
To construct a global continuous map f we can now simply linearly interpolate each of the maps f i from the right boundary
of the neighboring one.
By applying the universal approximation theorem to this function f , we finalize the proof.
Lemma 6.3.
Let f : M → N be an arbitrary smooth embedding.
Let S ⊂ M be a smooth embedded submanifold.
Then f | S is also a smooth embedding.
Proof.
The proof follows from the definition.
Indeed, for every point x ∈ S ⊂ M we have T x S ⊂ T x M and restriction of the derivative of f onto this subspace is also injective.
Note that f | S is also injective and open map.
In this section we discuss how the results in the main text can be generatlized to the case of the latent variable z sampled from R d rather than I d .
The only principal difference is that we now have to deal with approximating functions defined on the noncompact space, which is less trivial.
We make use of the following result (Ito (1992) ), where we for simplicity provide concrete formulas for the activation functions for which the results hold.
Theorem B.1.
Let σ belong to one of the three families:
• Gaussian distribution family: σ(t) = (2π)

assistant: text: We shot that a wide class of manifolds can be generated by ReLU and sigmoid networks with arbitrary precision.