Statistical Regeneration Guarantees of the Wasserstein Autoencoder with Latent Space Consistency
Authors: Anish Chakrabarty
Swagatam Das
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Autoencoders
Optimal Transport
VC dimension
Deep Generative Models

Abstract: The introduction of Variational Autoencoders (VAE) has been marked as a breakthrough in the history of representation learning models. Besides having several accolades of its own, VAE has successfully flagged off a series of inventions in the form of its immediate successors. Wasserstein Autoencoder (WAE), being an heir to that realm carries with it all of the goodness and heightened generative promises, matching even the generative adversarial networks (GANs). Needless to say, recent years have witnessed a remarkable resurgence in statistical analyses of the GANs. Similar examinations for Autoencoders however, despite their diverse applicability and notable empirical performance, remain largely absent. To close this gap, in this paper, we investigate the statistical properties of WAE. Firstly, we provide statistical guarantees that WAE achieves the target distribution in the latent space, utilizing the Vapnik–Chervonenkis (VC) theory. The main result, consequently ensures the regeneration of the input distribution, harnessing the potential offered by Optimal Transport of measures under the Wasserstein metric. This study, in turn, hints at the class of distributions WAE can reconstruct after suffering a compression in the form of a latent law. 

Statistical Regeneration Guarantees of the Wasserstein Autoencoder with Latent Space Consistency
Anish Chakrabarty Statistics and Mathematics Unit Indian Statistical Institute, Kolkata chakrabarty, anish@gmail. com
Swagatam Das Electronics and Communication Sciences Unit Indian Statistical Institute, Kolkata swagatam wwagatam.das@isicaal. ac. in
Abstract
The introduction of Variational Autoencoders (VAE) has been marked as a break- through in the history of representation learning models. Besides having several accolades its own, VAE has successfully flagged off a series of inventions in the form of its immediate successors. Wasserstein Autoencoder (WAE), being heir to that realm carries with it all of the goodness and heightened generative promises, matching even the generative adversarial networks (GANs). Needless say, recent years have witnessed a remarkable resurgence in statistical analyses the GANs. Similar examinations for Autoencoders, however, despite their diverse applicability and notable empirical performance, remain largely absent. To close this gap, in this paper, we investigate the statistical properties of WAE. Firstly, we provide statistical guarantees that WAE achieves the target distribution in the latent space, utilizing the Vapnik- Chervonenkis (VC) theory. The main result, consequently ensures the regeneration of the input distribution, harnessing the potential offered by Optimal Transport of measures under the Wasserstein metric. This study, in turn, hints at the class of distributions WAE can reconstruct after suffering compression in the form latent law.
Introduction
Unsupervised data generation has been a hime-hallowed problem in machine learning theory. Though its strongest suit, Variational Autoencoders (VAE) []] have been implemented for generative purposes to a great extent. Once emerged as a crucial unsupervised method for representation learning, its poor performances in this particular area somewhat depreciate its appeal. On the contrary, generative adversarial networks (GANs) ( have proven to be superior when it comes to the quality of the generated image samples. The theoretical elegance of the adversarial training method can be duly credited for this achievement. In an attempt to adopt this merit, such 'adversarial loss' functions were incorporated into VAE objectives 3 4. The Wasserstein metric, also being used in GANs (5) previously, was an inevitable choice, since it brings along the beckoning of Transportation Theory. Using the same to measure the reconstruction loss in a VAE. Wasserstein Autoencoder (WAE) (6) was conceived. By approaching generative modeling from Optimal Transport (OT) point of view, it not only achieved remarkable reconstruction standards but also strengthened the theoretical backbone.
An appropriate WAE framework consists of two components, namely an encoder (E) and decoder (D), which represented by neural networks in practice. Samples from input distribution (Px) are fed into the encoder, which aims at resulting in target law (Pz) in the latent space (Z). The decoder deals with the reconstruction of the original distribution from this 'encoded' signal. Mathematically, the objective of WAEs is to minimize L = R(D, E) + ?(E) over the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

set of probabilistic encoders E. Here, R() ensures the reconstruction of samples put in whereas, ?() ensures the attainment of the latent distribution. Needless to say, R() is represented by the Wasserstein discrepancy.
As the name suggests, our primary interest lies in investigating whether suitably characterized WAE can learn a distribution, which represents its latent behaviour, without losing the memory of the initial signal it was fed. When it comes to the question of memorizing the input distribution in the form of reconstruction, theoretical guarantees in the context of GANs, have been provided rigorously. In the absence of flexible discriminator, WAEs need more care and closer observation. Moreover, the concurrent task of sculpting desired latent distribution adds nuances to its mechanism. Most of the existing literature on deep generative models, under adversarial loss, misses out on this very point. Against such a backdrop, this first of its kind study substantiates the simultaneous performances of WAE, from theoretical viewpoint.
Our analysis starts with a special family of WAEs, namely f- Wasserstein Autoencoders (f-WA)) (7]. which arises due to the specific choice of (2() as f-divergences (I). We follow non-parametric approach in defining both the networks involved and the distributions under consideration. The transformations induced by the encoder and the decoder are conceived as measurable functions between spaces. The nature of an 'ideal' encoder in our language, is a network that preserves information. Whereas, the decoder is characterized by a map that propagates the information retained, precisely. Our statistical assumptions on the same reflect these very ideas. As a result, the two concurrent objectives of f-WE boil down to non-mmraitrric estimation of laws, under vastly different circumstances.
Our major findings can be summarized in the following way,
Given non-negative margin of f f-WAEs can learn latent space distributions with a convergence rate of Onn -2), under fairly gentle assumptions on the input and target law (T. The proof involves realistic characterization of the transformation, induced by the encoder network.
Under geometry preserving representation of the decoder map, Hölder densities can be successfully reconstructed by f-WAEs up to a constant margin (2). Our work also produces deterministic concentration bounds on the regeneration error and hence, the rate of convergence of the empirical laws.
A particular instance of our analysis goes on to show that the input distribution converges strongly to its latent counterpart under encoding, in the form of empirical measures. Also, f- T-WAs can exactly reproduce samples from smooth densities in the sense of weak conver- gence (4).
Inferences, identical to that case f-WEE, can be drawn for the original WAE 6 under additional assumptions on the decoder 2
2 Related Work
In recent times, scrutiny of the empirical achievements of deep generative models from theoretical point of view has gained great momentum. GANs (2], being at the helm, have received most of the attention 8 19. Possibly, the cornerstone of success for GAN lies in adversarial training and the accompanying loss functions. A plethora of GAN variants have been devised by incorporating differ- ent losses, such as Wasserstein distance (WD) (5) -divvereence [10, Maximum Mean Discrepancy [I] and Sobolev distances 112, to name a few. The generative aspect, in particular, has attracted lot of research in the form of distributional estimation studies [13 [4]. Additionally, [15] has proved weak convergence of distributions under several of the aforementioned adversarial losses. In a recent study, 16] has also discussed the convergence of the empirical measures in the case of WGAN, under parametrized network assumptions. VAEs, despite being used vastly generative tools, have not experienced such a level of independent theoretical investigation, which comprehends its latent space mechanism.
In a novel approach to unify the deep generative techniques, Adversarial AEs 3 were introduced. Subsequent work to establish resemblance 17 4. has only strengthened that foundation. Eventually, the formulation of WAE [6] bridged the gap significantly by matching regeneration standards. The equivalence in performance is direct reflection of the similarity of its objective to that of GANs' [7].

---

This "primal-dual relationship' prepares basis to explore the generative facets of WAE, in particular, now with the collective experience provided by GANs.
Typically, WD serves as the measure of discrepancy used to represent the reconstruction loss in WAFs. Non-parametrie density estimation under this metric has the establishment of minimax bounds based on covering numbers, under fairly general assumptions [18 [9 The same has also been achieved by [20], for unbounded metric spaces, assuming that some moments are finite. Singh et al. [2] established minimax convergence rates in the same context, under smoothness assumptions on the networks employed in the generation. Despite being influential in their own ways. none the above aim at discussing the coinciding objectives of WAEs. In a recent empirical study, Diffusion VAE [22] was introduced, which attempts capturing arbitrary latent manifolds using kernels of Brownian motion. However, without precise conceptual insight. As such, the theoretical questions regarding the parallel machinery of WAEs remain largely unanswered. Our analysis is a humble attempt at filling this void of statistical understanding.
3 Preliminaries
3.1 Notations
We start with denoting the input space by X, which can be assumed to be Polish (i.e., separable and completely metrizable). We will use for the latent space. Both may be thought of as Euclidean spaces, since it is typical characterization. The space of probability measures (distributions) defined on X' is denoted by (X). The set of all measurable functions from X' to is refered to as (X, 2). For example, observe that probabilistic encoder E belongs to I (X', (2)). For any function
+ Z, we denote the pushforward measure μ as T# E (Z). i.c. for any measurable set w Z we have Tœp(w) p(ix E X T(x) E 2}) p(T (w)). To denote
absolute continuity of measure j with respect to X, we use the notation < X. The Support of p is denoted by Supp(p). To indicate the Closure of X' of its Interior Points. Given S C X, the S-distance between pair of functions f and is given by, 1 -glls sup
X and [nt(X)) denotes the set
g(x)]. When the sup taken over the whole space, we call the uniform norm. The total variation distance between probability measures μ and y E (X) is denoted by ||I - 7/xrv. Now we introduce some useful concepts that are made use of later.
Definition (f-Divergence). Let P. E (X) such that P < Q. For convex function f with f(1) = 0, the f-dieergence of P from Q is defined
{
(dP
dP
D2(P.Q)=
(dQ)
dQ.
Definition 2 (Yatracos Family (23)). For a class of real-valued functions F. defined on Z, the Yatracos family associated to F is collection of subsets of Z defined as
((F) fa E 2: f(x) 2 g(r); f.g E F).
Definition (Empirical Yatracos Minimizer
24 The Empirical Yatracos Minimizer a function
(2) U t (Z) defind a=1
( argmin ||g -ulllree
(E)(
f where pn is an empirical distribution based on a sequence of observations from Z.
Definition 4 (c-Closeness). A pair of distributions f.g (X)) are said to be c-close to each other with respect to the metric i cf g) S c.
Definition 5 (Quasi-Isometrie Embedding). A map (Z,di) (X, called a quasi-
isometric embedding if there exists a constant A 2 such that,
A d1(2. S (zlf S Ad1(z.)) Vz,yEZ.
Definition 6 (Covering Number 18] Let S X. Then, the minimum integer m such that there
exists m closed balls { {B,. of diameter each e 5 U1sism B, is called the e-covering number of S. We denote it by (()S).

---

3.2 Wasserstein Duality: A Brief Exposition
The equivalence relation between Wasserstein Distance and Integral Probability Metries (IPMs) 25 is a fairly well-studied topic in optimal transport theory. We exploit this fact to serve our purpose. For detailed elucidation on this subject, one may take refuge [26] 27
Let P,Q E (X) and e: X' x X + Rzo forms a metric. Then,
Q) = inf (xrr (x,,))d((r,)) (1)
(ET(P()
{ l ((x))P()) - / (1))d((z) IPM (P.Q). 1E
= sup
where ( P(X x X): fx (x,
fx, y)dy P, fx 7x2, y)dx - Q} denotes the coupling between P and Q. The class of I-Lipschitz functions with respect the metric represented by L.
x
Later in the study however, we will use the notation dæ for Wasserstein metric, for clarity.
3.3 Problem Setup
typical WAE architecture intends to solve the minimization problem.
U E.-me)lda,, D(e))ddxx PAC(E (I) WAE D) (2)
inf
where c:: X' X R>0 and A 0 serves hyperparameter. Arbitrary divergence metrics between measures, belonging to (Z), are deployed the form of Q. In this paper however, we modify this formulation to facilitate theoretical analyses. The reshaped objective that our study is based on is given by,
inf f- f-WAE, (Px, (2)
(2)
This category of AEs are called f- /-assersteiin Autoencoders 77. The key difference between this family and the ordinary WAE arises due the choice of the reconstruction loss. W. is the manifestation of the expected cost of transportation, over all couplings the pair of distributions under consideration. This alteration, however, does not impact the characterization significantly. Equivalence between these two systems can be attained by imposing mild restrictions on the decoder transformation, which we elaborate foundation to work on.
later stage. As such, this can be marked valid
Observe that, the regularizer metric used in 2 to measure the discrepancy between the target latent distribution and the encoded signal is taken to be Dy. In this analysis, our specific choice of f f-Divergence is Total Variation Distance (TV). Its rich theoretical appeal motivates
do so. TV as a measure deviation is symmetric with respect to the distributions under consideration. Moreover, it can pose both a -ff-iiergeence and an IPM (28 The original WAE, in of its characterizations, uses Jensen-Shannon divergence (JS) to penalize the loss instead (6. TV, on the other hand, acts as an upper bound JS 229] which also justifies our choice.
Remark. WAEs as L3 in ( establishes an equivalence between the two [6]. Keeping that mind, our analysis does not restrict the choice of and hence can also cater to the theory of AAEs.
generalization AAEs [3. We stress fact that, specific choice of
4 Theoretical Analysis
Before moving forward, let us rewrite the minimization problem ( in the so-called constrained form
inf (Wx(Px, (Do F)+ Px)} subject to (y Px, Pz) <
(2)
for t 0. By Lagrangian duality 30), for every value of t where the constraint Dy Px) < t holds, there exists a value of À that produces the same solution from (2) Con- versely, the solution to the problem 2). say É also solves the bound version with t (E Px. Pz).

---

As such, there is an one-to-one correspondence between the constrained problem and the La- grangian one 2 This formulation gives us a direction move ahead to establish regeneration guarantees and consistency of measures in the latent space.
Our analysis consists of two stages, going forward. Firstly show that probabilistic encoders can successfully transform input densities into realistically characterized latent space laws, under TV distance, up to a constant margin of error. The mild assumptions imposed in the process are fairly practical. Once the constraint of the minimization problem is met in the form of latent space consistency, we provide deterministic upper bounds to the empirical reconstruction loss.
Throughout our discussion, we denote with μ E (Xt) the input data distribution. The target distribution which characterizes the latent space is denoted by p. For simplicity, assumee that both and p happen to have corresponding densities, namely Py and pp with respect to the Lebesgue measure X, in their respective spaces. Both densities need explicit characterization, which we state in the form of certain assumptions.
Assumption 1. P (i) based on a compact and convex support is (ii) bounded away from 0 by a 2
constant factor > 0, i.e.
Assumption 2. Pp- (i) also defined on compact support, is easy-to-sample-from and (ii) infinitely
smooth, ie. Pp (2).
The input density being far from nullity, passes on the same nature the reconstructed law, even under near-ideal situations. This poses as a measure to prevent the regenerated law from being a degenerated one, zero. We point out that, in the non-parametric approach of this work, we often use empirical distributions corresponding to p. which are based on observations from the same. feasible to
In high-dimensional statisties, there are distributions from which it is practically draw samples. For example, the drawing of uniform samples from high-dimensional unit sphere. Although in our analysis, the need for such samples is strictly theoretical and the practical feasibility is somewhat immaterial, we describe p this way to avoid complication. Given these assumptions, suppose the 'encoded' pushforward measure En μ E (Z) has density P Then, every such P is a potential candidate to represent Pp. However, our analysis does not assume the existence of a density. We denote Supp(pp) by C, such that Assumption 3. VC-dim (FP(() (C)) is finite.
This assumption too is very much plausible in its own right. For example, let Ga and Aa denote the class of d-dimensional Gaussian distributions and the class of d-dimensional axis-aligned Gaussian distributions respectively. It can be shown that, VC-dim ([)) (c)) (31. The introduction of Yatracos class of distributions, in turn, facilitates the search for the minimum distance estimate of a measure. For detailed exposition on the same, one may turn to
(dd) and VC-dim (CCdiim)"(AA)) (AAa)
Perhaps the most challenging of tasks in studying the theoretical aspects of deep neural networks lies in their basic characterization. Several depictions of networks as collection of functions, indexed by parametric classes have produced notable results in the past 99 13). This paper rather explores the mathematical properties network has embody to function in WAE system. The following representation of an encoder map reflects our motivation. We consider an E E F (X, (2)) such that,
Assumption 4. (i) Egg f. is supported on C and is also easy to draw samples from.
(ii) Given any arbitrary empirical measure Paa (based on n samples) corresponding to p. there exists an empirical counterpart (E4H),, of Eggu which is c-close to inu under E, with probability at least
1 kexp(=n'e") where 2 1 and k 2 0. In other words, IP ((EEp (E44), se)
1 =e'pl_n"e)).
Part (ii) of Assumption 4 is provided only to give coherence to the definition of E. It suggests that the probability of 'the discrepancy between the two quantities involved being arbitrarily small increases exponentially with the r-th power of n. As such, if one has a superlinear rate of increment in information in the form of samples, the portion lost in transition decays sharply. This embodies our idea of an encoder, that preserves information. It is often seen in practice that AFs, with ReLU encoders, can reconstruct images accurately, despite having the encoded observations to be almost degenerate. In such case, the two measures under consideration can always be said to be arbitrarily
5

---

close. Part (ii) is gentler approximation of the same idea. We now present the main result that enables one prove consistency estimating the latent space distribution.
Theorem 1. Let, (E+ stands for empirical measure based 2 i.i.d. samples from E μs. Then, under assumptions ii. 2 i), 41).. for positive constants 01. C22 C3 and E (0, 1) there exists x 2 0 such that,
|| - (++1))) Ilv - - 1 Val a in (
holds with probability 2 1 - where VC-dim)D(
To prove this theorem, we introduce some crucial results from the estimation theory of measures. Proofs of all the lemmas stated henceforth can be found in the supplement.
Lemma 1, For J.9 E (C)) øllrv = |f 9 ylly)
Lemma 2. Let (Z). Also, let In denote the empirical distribution based 2 0 i.d. samples drawn from ", Then there exist positive constants k1, k2 such that
P P(I - Pnlloo S ki / +1) 2 exp (-kgnt)
n
where v stands for VC-dim
Proof of Theorem 7 Given observations from p. let in denote the corresponding empirical measure. Now, using the triangle inequality we have,
| (F+1)
|| S -11p F
FF (F#1))))
S 11p Pullzv + F#g (E*1)) bllmy lia ExullTT
(4)
Define,
2 argmin 11q Pall)(on))) (C))(C)
Observe that,
|:n E S ll GillT + 1| Exullrv
1l Caby(erc + 1| E*ullev
(5)
$ el - -0-alll(a(e)) + || E
(6)
Lemma 1 ensures the equality in 5 6 due to the definition of L.p Now,
inf (ar, /
|
E
v)
(7)
inf TET TET' (C) / dxx, E(y)ddr
where T and T are couplings of measures. The metric c: 2 x 2 - R20 in 7 is the trivial metric c(x, y) 1 [26]. This implies, transformation induced by the encoder should be such that, it minimizes the adversarial loss 88. In case e is lower semi-continuous, there exists T E T which minimizes (8), and hence an E For example, c(æ,, y) (o y), where (c(x, y) the indicator on an open set C. In our situation however, we can always choose a sub-optimal E such that (8) < X*, for a nen-negative A Villani et al. [26] describes various ways to get hold of such a
Going back to 4
(E+H),
X* < 21|p Pull
Ex (E#1), (
Thus. 99) along with lemma
proves the theorem.

---

It is worth mentioning that in reality however, we only have sample of size n E N+ from the imput distribution, say [X,] Based on the same, an empirical distribution (ú, =1 2i=1 dx, is what we have at hand.
Corollary 1. Under the additional assumptions (J}
| plITV
Op(n
Remark. Despite the fact that the optimal value of (8) cannot be made arbitrarily small directly, in case we obtain Am. n E N. such that, lim limsupp--zee An X"; the same conclusion can be drawn, We call Corollary 7 "consistency up X* in this light. Note that, for a given margin of error X in the latent space, E needs to be chosen such that, X I X. On the other way round, starting with an unspecified X, one can learn about the feasible margin of ervor by observing X"
Theorem ① shows that the discrepancy between the 'encoded' and the 'target' law can be made arbitrarily small around a constant with high probability. Corollary ( consequently establishes our idea of consistency in latent space. Proof for the same can be found in the supplement. As such, for any suitably chosen (*, the constraint in problem 3 satisfied. We now investigate whether the decoder network is capable of regenerating a distribution 'close" to μ. from E ps. A probabilistic decoder, in this context, is formally defined as function D Z X such that D#p E (X).
The residual task at hand can be formally written as the minimization of
d2: (eD 11,
sup (e [{r) 1E.2
Eazn [w)]]
over E E FX,, (2)) which follows assumption Which, however under an observed empirical measure ir boils down to its sample counterpart, given by
d (DD +) p))
sup ie
(DoR()))) [(x)) Ezep [(x))]
(10)
We remark that the role of the class of functions Lt while minimizing (10) is to behave as 'critics'. GANs, during their training, gradually learn to a better judge by casting this functional class through a discriminator [5). This freedom of choice in turn gives one more flexibility to interpret its regenerative capabilities [3 14] In the case of AEs however, the choice depends solely on the preassigned discrepancy metric. Hence, the theoretical justification of reconstruction for AEs needs additional maneuvering. Our first step in doing so an assumption on the input distribution.
Assumption 5. (i) Pμ E (X)) with O, where H (X) denotes the Hölder class based X;
such that, the 8th order derivative Pp
(i) exists (i S [ax]) and is bounded on &. Also,
p (((=)))))) p()-)))))
[a]] PM
sup2.y
< cc, for x # Int(A)).
I ln-vlla"
Remark. Assumption enables one to assign the Hölder norm, defined as
|| H = max
(
+ DDaa
Moreover. |pullzge turns out to bounded.
Smoothness assumptions on input data distribution, in the form of Hölder or Sobolev densities are
quite familiar in the theory of density estimation for generative models [13] 14 21.. Observe that, earlier results still hold under this additional assumption. The smoothness of both the input and latent densities ii))) together ensures the existence of certain regenerative maps, which are crucial the forthcoming discussion. Lemma (3) formalizes the same.
Lemma
33 Under assumptions
5 there exists a map T E H from Z to X such that,
Typ= = μ.
In our discussion, similar the encoder transformation, we do not impose specific structural restrictions on the decoder network. However, the mathematical assumptions that are made, are inspired and supported by real architectures.
Assumption 6. The quasi-isometric embedding D (Z, llrv) - network e e-close to T in uniform norm, where is as defined in lemma 3 Also, (D F) y shares the same support with p.
(X, c) induced by the decoder
7

---

Classes of functions, learned by generator networks in GANs, are often taken to be smooth [3 21 achieve minimax convergence rate. Our assumption of quasi-isometric embedding is gentler in principle. We mention that an alternative weaker assumption of Lipschitz continuity would have sufficed in our analysis. Assumption (6) on the other hand, takes one step forward to comment on the broader geometry of the two spaces under consideration. Moreover, precise specification of parameters of a ReLU network can always result a map, which is capable of approximating the optimal Monge solution (34 [35] which in our case turns out to be smooth. Note that, the metric on 2 can be equivalently taken as l
The following lemma provides an oracle inequality by fragmenting the empirical loss as in 10p into disjoint parts.
Lemma 4. Given an encoder transformation E",
yy: ((D E" #bn, E E2 28
where & = ddg F*) DHp) is the 'Encoder Approximation Error'; E = b==dyg(D#p. PD)) stands for the 'Decoder Approximation Error' and Eg - = (Pn, H). the 'Statistical Estimation Error'.
We remark that successful regeneration by the f f-WAE can be ensured by showing: these individual errors are arbitrarily small or they approach zero in any probabilistic sense. To demonstrate the same in the case of E3, we introduce some concepts from [36 [8].
Definition (I-Upper Wasserstein Dimension). For (X), the (e, r)-ceerring number is given by
Nd(r,T) inf (Mf((N((S) : ((S) 2 T. SCX
Consequently, we define the (c, -imeensio as 8c(7,T) = .Na(tt) -log(e)*
Under this setup, the Upper
Wasserstein Dimension for p 1 is
(1) = inf E (2) lim (7, Ss.
c
Lemma 5 (18 Let (Sapp(() = B. Then, for n 2 0 and s > (1))
2nt
P (d:2 (fm, S On +1) 2 exp
B2
Now, we propose the main theorem which provides a guarantee regarding regeneration.
Theorem 2. Given margin of error X* in latent space, let fa, denote empirical imput measure, consistent up to X", under encoder map E". Also, let (Samp(Su = Then, there exists a constant 5* and E (0, such that
dæg ((D F² (") (1) O(= ((
with probability at least 1 6, where: s > (1).
Proof of Theorem Observe that, the error committed by the encoder
I VET(EARM.O)
E dg) E² E²) D+P)
inf
(11)
$ inf vET (ET / ||x vllrv du(x, v)
AB + (n +),
(12)
where 12) holds with probability 2 1 - (using Theorem T in 11 denotes the associated coupling and A is the constant related to the embedding D. Denote, Similarly,
ABB
E dey TDp) = sup { 1E
- Expp [Emxp((((T))]
< Exnp
[I - -Tllool <(' D-
(13)
8

---

which is a consequence of the fact that, bhat.|D-- - S (Assumption 6
Here, e is an arbitrarily
small quantity corresponding to e.
Also, lemma (5) suggests that for 8 > 81
Es = d2p (Pn. - ((+
B vn
1
In
(14)
with probability at least 1-6.
As such, [12
13 and
14) together prove the theorem.
Remark. Let us weigh up on a special case, which carries notable significance. Suppose, one obtains
Xn upper bound to (8. corresponding Cu; Nt, such that Xn = o(1). We do not prove the existence of such sequence, which might be taken up as a future work. However, under the assumption of existence, corollary
7 implies that,
|E - p(A)]| 0, AEF
sup
where F is the Borel o-algebra on Z. As such, Exir converges strongly to This theoretically ensures the achievement of target latent law.
Similarly, such an { will have a remarkable effect on the regeneration process. In that case, Theorem (2) will imply,
dy E² EE" P p)) O,
i.e. (Do 4 converges weakly in (X) μ 266) guarantee.
This result serves as an exact reconstruction
Now, let us look back at the original WAE objective in T Denote by
(Px (E, = E
the reconstruction loss. In general, this acts as an upper bound to the M- counterpart (7]. However, under additional restrictions on D, they become equivalent (7]. In such case, if 22 ① is taken to be D,, the two objectives in their entirety also follow the same trait. In our study, the measure of discrepancy between distributions in latent space was throughout taken be TV. JS divergence, being upper bounded by TV, is obliged to follow the results established exactly. As such, the only task remaining the transferral of regeneration guarantees to WAEs.
Corollary 2.
There exists a specific margin of error in the latent space such that, Theorem holds exactly for dt (pro (F*, D)), where D is also invertible.
Proof of Corollary (2). Husain et al. 7 had suggested the existence of 0, such that for all margin or error À K. (T) and (2) are exactly equal. The proof however, demands the decoder transform to be invertible. In such case, for encoder E* and Ω Dy imply that
(p,n (E" (E*,D)) D)) / (
S dog((D ((D =")+0n 11) /1) + does (Pr. H.).
Hence, using lemma 2 and Theorem we say that
do (E', D)) 5* on O(n {(m
holds with probability at least 6, for > (p).
emphasize the fact that the argument of weak convergence, however, as given (4), does not apply directly to this case. The Wasserstein distance metrizes ( i.e. weak convergence holds if and only if d - O. On the other hand, d does not satisfy the axioms a distance. As such, WD produces stronger inferences, even though under carefully taken assumptions they tend to be equal.

---

Further Theoretical Implications
An important aspect of the VAE's operations lies in its involvement in "disentanglement" While the concept lacks robust definition, it can arguably be described as the process of achieving factorial representation; one with "expressive' and perhaps independent coordinates, which characterize the features. A somewhat gentle depiction of the same might be a distribution with a diagonal dispersion matrix. The non-parametric portrayal of latent distributions in this analysis does not comment on the covariance structure of the underlying random variables. However, this seemingly unrestrained characterization in turn may allow for such specific distributions also, as far as they follow the assumptions of our framework. For example, we mention that if PC)) is taken to be the class of finie-dimennional axis-aligned Gaussians, our results hold true.
A practitioner might also feel divided between the two architectures that appear in [6] Although we focus mainly on the one deploying IS divergence, there is no evidence theoretical superiority that MMD enjoys, to the best of our knowledge. Uniform convergence bounds on empirical MMD between distributions, as established by (7) in a different context, however, also produce convergence rate of On ). As such, this section can be regarded as note of encouragement for closer observations on these very topics.
6 Conclusion
Our analysis, besides serving as the first of its kind theoretical promises regarding the simultaneous tasks WAE has to perform, also points toward untraveled avenues. An immediate extension might look into broader classes of discrepancy measures in latent space. The latent space and hence the distribution that characterizes it might also be in focus. An even elaborate characterization can deliver it further justice. Our study includes a commentary on the geometry preserving nature of deep generative networks from fairly new perspective. Being a novel representation, it creates new opportunities for inventive scrutiny. We also restrict our analysis to non parametric regime, which may encourage similar endeavours under a parametrized setup. Such an approach could be the key to revealing the effect of network parameters such as depth. width, activation functions etc. on regeneration and consistency in the latent space. This, in turn, may improve the theoretical results by allowing training plans on board as well.
References
[1] Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference Learning Representations, ICLR 2014, 2014.
[2] Ian J. Goodfellow, Jean Bouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS' 14, page 2680, 2014.
[3] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adver- sarial autoencoders, 2016, 151.:64414.
[4] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks, 2018, arXiv:1701.0mm 1701.0422..
[5] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214-223. PMLR, 2017.
[6] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. In International Conference on Learning Representations, 2018.
[7] Hisham Husain, Richard Nock, and Robert C Williamson. A primal-dual link between gans and autoencoders. In Advances in Neural Information Processing Systems, volume 32, 2019.
[8] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 224-232. PMLR, 06-11 Aug 2017.
10

---

[9] Gérard Biau, Benoît Cadre, Maxime Sangnier, and Ugo Tanielian. Some theoretical properties of GANS, The Annals of Statistics, 48:1539 1566. 2020.
[10] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural sam- plers using variational divergence minimization. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[11] Chun-Liang Li, Wei-Cheng Chang. Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network, 2017, arXiv: 1.div:1705.008 705.00884.
[12] Youssef Mroueh, Chun-Liang Li, Tom Sercu, Anant Raj, and Yu Cheng. Sobolev gan, 2017,
arXiv:1711 1tt::1711.0094. 1.04944.
[13] Tengyuan Liang. How well generative adversarial networks learn distributions, 2020, arXiv:1811.03179.
[14] Minshuo Chen, Wenjing Liao, Hongyuan Zha, and Tuo Zhao. Statistical guarantees of generative adversarial networks for distribution estimation, 2020, arXiv:2002.03938
[15] Shuang Liu, Olivier Bousquet, and Kamalika Chaudhuri. Approximation and convergence properties of generative adversarial learning. 2017, arXiv: trxXiv:1755.055 700.08991.
[16] Gérard Biau, Maxime Sangnier, and Ugo Tanielian. Some theoretical insights into wasserstein gans, 2020, arXiv:2006. 202.22...0....22..
[17] Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric P. Xing. On unifying deep generative models, 2018, arXiv: 10ff::170000055000
[18] Jonathan Weed and Francis Bach. Sharp asymptotic and saipee rates of convergence of empirical measures in wasserstein distance, 2017, arXiv: (tt::177.000 707.00087.
[19] Jing Lei. Convergence and concentration of empirical measures under wasserstein distance in unbounded functional spaces. Bernoulli, 26(1), Feb 2020.
[20] Shashank Singh and Barnahás P6ezos. Minimax distribution estimation in wasserstein distance,
2019, arXiv:1802.08855.
[21] Shashank Singh, Ananya Uppal, Boyue Li, Chun-Liang Li, Manzil Zaheer, and Barnabás Póczos. Nonparametrie density estimation with adversarial losses. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 10246-10257, 2018
[22] Luis A. Perez Rey, Vlado Menkovski, and Jim Portegies. Diffusion variational autoencoders. In Proceedings of the Twenty-inth International Joint Conference on Artificial Intelligence, IJCAI- 20, pages 2704-2710. International Joint Conferences on Artificial Intelligence Organization, 2020.
[23] Yannis G. Yatracos. Rates convergence of minimum distance estimators and kolmogorov's entropy. The Annals of Statistics, 13(2):768-774, 1985.
[24] L. Devroye and G. Lugosi. Combinatorial methods in density estimation. In Springer series in statistics, 2001.
[25] Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):499443,, 1997.
[26] Villani Cédric. Optimal transport : old and new. Grundlehren der mathematischen Wis- senschaften. Springer, 2009.
[27] Gabriel Peyré and Marco Cuturi. Computational optimal transport, 2020, wwXiv:1803000566
(28)
Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Berhard Schölkopf, and Gert R. G. Lanckriet. On integral probability metrics, g-divergences and binary classification, 2009, arXiv:0901.2698.
[29] Igal Sason and Sergio Verdu. f -divergence inequalities. IEEE Transactions on Information Theory, 62(11)::973-600 Nov 2016.
[30] Stephen Boyd and Lieven Vandenherghe. Convex Optimization. Cambridge University Press,
USA, 2004.
[31] Hassan Ashtiani and Abbas Mehrabian. Some techniques density estimation, 2018, arXiv:1801.04000

---

[32]
Villani Cédric. Topics in optimal transportation. Graduate studies in mathematics. American mathematical society, C 2003.
[33] L. Caffarelli. The regularity of mappings with convex potential. Journal of the American Mathematical Society, 5:99-104, 1992.
[34] Minshuo Chen, Haoming Jiang, Wenjing Liao, and Tuo Zhao. Efficient approximation of deep relu networks for functions low dimensional manifolds. In Advances in Neural Information Processing Systems, volume 32, 2019.
[35] Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103-114, 2017.
[36] R. M. Dudley, The Speed of Mean Glivenko-Cantelli Convergence. The Annals of Mathematical Statistics, ):(1)):00 50, 1969.
[37] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723-773, 2012.
Funding Transparency Statement
The authors did receive any third-party funding third-party support during the last 36 months prior to this submission toward pursuing this work. The authors have no financial relationships with entities that could potentially be perceived to influence the submitted work.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? YYes] We avoid any form of exaggeration
in our claims throughout the discussion. The precisely stated results give clear impression of the achievements and the limitations of our work.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes] All assumptions included in the analyses are orderly placed throughout the article.
(b) Did you include complete proofs of all theoretical results? [Yes] Rigorous proofs of our main results only have been provided the paper. The same for all other claims are added to the supplement.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [N/A]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [N/A]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "mavUg27XYn", "reviewer": "Reviewer_wdFS", "summary": "This paper analyzes the statistical properties of WAE, providing theoretical guarantees for both sample reconstruction and latent distribution tasks. To do so, the authors use the family of so-called f-Wasserstein autoencoders due to the specific choice of discrepancy selected for measuring the target and latent distribution. Here total variation distance is chosen. After setting up necessary preliminaries, and assumptions the authors prove the first of two main theorems, consistency in estimating the latent space distribution. Under suitable conditions, the second theorem provides a guarantee regarding the regeneration. Since they analysis was conducted using TV, which upper bounds the Jenson-Shannon divergence, the final corollary transfers the regeneration guarantees to WAEs. Fairly gentle assumptions and interesting theoretical results are discussed in detail. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThis paper offers novel analysis of the simultaneous tasks of WAE, and reveled new statistical guarantees for latent space consistency and regeneration quality. To my knowledge such an analysis has not been done. The related work seems appropriately cited. \n\nQuality: \nThe analysis presented here doesn’t appear to have technical issues. The work is self-contained, and the authors appear to fairly describe where any limitations may lie. \n \nClarity: \nThe submission is clearly written and well organized though some polishing could be done. Line 82-84 reads awkwardly, and I’m not sure it’s fair to say VAEs haven’t had much theoretical work done. Nevertheless, the logic from lemmas to theorems is nicely built up, with discussions where needed. I enjoyed reading it.\n\nSignificance:\nGiven the pace at which the AI community advances, theoretical work is often playing catch up. Results like the ones presented here are a delight, and I’d imagine others may find them interesting as well.  While it’s difficult to determine whether it advances state-of-the-art (no empirical results), it principally offers statistical guarantees of WAE using VC theory. I believe the approach is of significance and that the NeurIPS community would benefit from the contributions presented in this paper. \n\nTypo: \nEquation 10, the last parenthesis under the square root should be on the outside.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "fBaG-jBRQkI", "reviewer": "Reviewer_TEbm", "summary": "This paper provides a statistical guarantee of f-WAEs on learning the latent distribution by VC theory.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written with accurate and proper definition on related notations and probability measures which is appreciated. \n\nThe author studied statistical properties of f-WAE on learning latent space and reconstruction. This is the first kind of such result from my knowledge. \n\nMy main concern is about the implication of those theoretical results.\n\n--- The current theoretical result is a bit rough and hard for the practitioner to gain some insights why WAE is better than vanilla VAE and AEs. \n\n--- The main theoretical results in Theorems 1 and 2 are independent of training and specific model architecture. For example, whether the VC-dimension will be related to the depth and width of the networks? In Theorem 1, how large lambda^* it is? Is that an absolute constant or can be arbitrarily large? Do you have any model assumption on the networks? For me, assumption 4 is a bit strong.  I suggest the author could discuss under what cases it can be satisfied.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It's also better to discuss if there is any technical contribution. For the proof of Theorem 1, it looks naive and just uses the basic concentration. In assumption 4, \"$E_{\\#}\\mu$ is easy to draw sample from\". This is not a rigorous statement.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "_0GP2YPyav", "reviewer": "Reviewer_sBV7", "summary": "The submission aims to provide rigorous theoretical results for the Wasserstein Autoencoder (WAE) - A model that has achieved great empirical success yet remains severely understudied. In particular, their main contribution in showing learning theoretic bounds such as complexity and convergence guarantees. As a feature of the Autoencoder set-up, they introduce the notion of statistical regeneration and go out to prove this statement for their learning setting.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of this paper is the problem they tackle which is an extremely topical issue with little theoretical studies, especially for the autoencoder settings. The contribution of the paper is original and makes important and clear references to related work when appropriate. The significance of the paper is also clear by virtue of the importance of the problem.\n\nIn particular, the authors establish a connection to the VC-dimension of the encoder distribution classes which is an interesting manifestion of complexity in the autoencoder setting. This opens up fruitful future work in the direction of studying autoencoders and generative models are large from a learning theoretic perspective. The work also establishes the notion of statistical regeneration, which although somewhat not interpretable, can be an important step for understanding the generalization performance of autoencoder models.\n\nThe clarity of the paper can definitely be improved as I found it quite challenging to follow notation and had to refer to external references to better understand some definitions. For example, in Assumption 3, I could not properly understand what is this space \\mathcal{Y}( \\mathscr{P}(\\mathcal{C})) formally? I understand it refers to the choice of models however I would like to better understand what this function class is precisely. However, I understand that the area of generative modelling and autoencoders are quite notation dense, given the differing ways they are defined so this is not so much an issue for a paper with such results.\n\nSome questions I had for the authors:\n\n(1) One of the main interesting questions about WAE is how should one pick the divergence between distributions in the latent space. This paper comments on this however I was wondering if there is anything extremely actionable that you can comment? I am aware that the most common choices are MMD due to computational convenience and perhaps this has some stronger effect on the convergence?\n\n(2) There is a common discussion in the autoencoder literature between the choice of lambda (the regularization weighting) and 'disentanglement'. While disentanglement is not properly defined, I wonder if the results of your work can establish this formally from the perspective of statistical regeneration since it seems to be intimately linked with the latent space. Do you have any comments on this?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes they have.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "13"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers liked the paper and support acceptance. So it is a clear accept.

The reviewers provided good feedback on shortcomings in the presentation. The authors are strongly encouraged to take these comments into account for the final version of the paper.
Decision: Accept (Spotlight)