Non-asymptotic Error Bounds for Bidirectional GANs
Authors: Shiao Liu
Yunfei Yang
Jian Huang
Yuling Jiao
Yang Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Convergence analysis
Dudley distance
empirical process
function approximation
neural networks

Abstract: We derive nearly sharp bounds for the bidirectional GAN (BiGAN) estimation error under the Dudley distance between the latent joint distribution and the data joint distribution with appropriately specified  architecture of the neural networks used in the model. To the best of our knowledge, this is the first theoretical guarantee for the bidirectional GAN learning approach. An appealing feature of our results is that they do not assume the reference and the data distributions to have the same dimensions or these distributions to have bounded support. These assumptions are commonly assumed in the existing convergence analysis of the unidirectional GANs but may not be satisfied in practice. Our results are also applicable to the Wasserstein bidirectional GAN if the target distribution is assumed to have a bounded support. To prove these results, we construct neural network functions that push forward an empirical distribution to another arbitrary empirical distribution on a possibly different-dimensional space. We also develop a novel decomposition of the integral probability metric for the error analysis of bidirectional GANs. These basic theoretical results are of independent interest and can be applied to other related learning problems.

Non-Asymptotic Error Bounds for Bidirectional GANs
Shiao Liu
Department of Statistics and Actuarial Science, University of lowa lowa City, IA 52242, USA
shiao-1iu@uiowa.ed
Yunfei Yang
Department Mathematics, The Hong Kong University of Science and Technology Clear Water Bay, Hong Kong, China
yyangdc@connect. ust.
Jian Huang
Department of Statistics and Actuarial Science, University of lowa Iowa City, IA 52242, USA
jian-huanguuioxa.ee edu
Yuling Jiao
School of Mathematics and Statistics, Wuhan University Wuhan, Hubei, China 430072
eew.ch@wuuuueu.... cn
Yang Wang
Department Mathematics, The Hong Kong University of Science and Technology Clear Water Bay, Hong Kong, China yangwang@ust.cr hk
Abstract
We derive nearly sharp bounds for the bidirectional GAN (BiGAN) estimation етrоr under the Dudley distance between the latent joint distribution and the data joint distribution with appropriately specified architecture of the neural networks used in the model. To the best of our knowledge, this is the first theoretical guar- antee for the bidirectional GAN learning approach. results is that they do not assume the reference and the data distributions to have the same dimensions or these distributions to have bounded support. These as- sumptions are commonly assumed in the existing convergence analysis of the unidirectional GANs but may not be satisfied in practice. Our results are also applicable to the Wasserstein bidirectional GAN if the target distribution is sumed to have a bounded support. To prove these results, we construct neural network functions that push forward empirical distribution to another arbitrary empirical distribution on a possibly dirreeentddimessinnnll space. We also develop novel decomposition of the integral probability metric for the error analysis of bidirectional GANs. These hasic theoretical results are of independent interest and can be applied to other related learning problems.
appealing feature of our
"Corresponding authors
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Introduction
Generative adversarial networks (GAN) (Goodfellow et al. 2014) is an important approach to im- plicitly learning and sampling from high-dimensional complex distributions. GANs have been shown to achieve impressive performance in many machine learning tasks (Radford et al. 2016 Reed et al., 2016; Zhu et al. 2017: Karras et al., 2018 2019 Brock et al., 2019). Several recent studies have generalized GANs to bidirectional generative learning, which learns an encoder map- ping the data distribution to the reference distribution simultaneously together with the generator doing reversely. These studies include the adversarial autoencoder (AAE) (Makhzani et al. 2015). bidirectional GAN (BiGAN) (Donahue et al. 2016), adversarially learned inference (ALI) (u- moulin et al., 2016). and bidirectional generative modeling using adversarial gradient estimation (AGES) (Shen al. 2020). A common feature of these methods is that they generalize the basic adversarial training framework of the original GAN from unidirectional to bidirectional. Dumoulin al. (2016) showed that BiGANs make use of the joint distribution of data and latent representa- tions, which can better capture the information of data than the vanilla GANs. Comparing with the unidirectional GANs, the joint distribution matching the training of bidirectional GANs alleviates mode dropping and encourages cycle consistency (Shen et al. 2020).
Several elegant and stimulating papers have analyzed the theoretical properties of unidirectional GANs. Arooa et al. (2017) considered the generalization error of GANs under the neural net distance. Zhang et al. (2018) improved the generalization error bound in Arora et al. (2017). [Ling]] (2020) studied the minimax optimal rates for learning distributions with empirical samples under Sobolev evaluation class and density class. The minimax rate is O(n -1/2 V n -2a+ where a and are the regularity parameters for Sobolev density and evaluation class, respectively. Bai et al. (2019) analyzed the estimation error of GANs under the Wasserstein distance for a special class of distributions implemented by generator. while the discriminator is designed to guarantee zero bias. Chen et al.] (2020) studies the convergence properties of GANs when both the evaluation class and the target density class are Hölder classes and derived O(n O(n-8/(2++)) (/8++4)))))))) log" n) bound, where is the dimension of the data distribution and and 3 are the regularity parameters for Hölder density and evaluation class, respectively. While impressive progresses have been made on the theoretical understanding of GANs, there are still some drawbacks in the existing results. For example,
(a) The reference distribution and the target data distribution are assumed to have the same dimen- sion, which is not the actual setting for GAN training.
(b) The reference and the target data distributions are assumed be supported on bounded sets.
(c) The prefactors in the convergence rates may depend on the dimension of the data distribution exponentially.
In practice, GANs are usually trained using reference distributions with lower dimension than that of the target data distribution. Indeed, an important strength of GANs is that they can model low-dimensional latent structures via using a low-dimensional reference distribution. The bounded support assumption excludes some commonly used Gaussian distributions as the reference. There- fore, strietly speaking. the existing convergence analysis results do not apply to what have been done in practice. In addition, there has been theoretical analysis of bidirectional GANs in the literature.
1.1 Contributions
We derive nearly sharp non-asy non-asymptotic bounds for the GAN estimation error under the Dudley dis- tance between the reference joint distribution and the data joint distribution. To the best of our knowledge, this is the first result providing theoretical guarantees for bidirectional GAN estimation error rate. We do not assume that the reference and the target data distributions have the same dimen- sion or these distributions have bounded support. Also, our results are applicable to the Wasserstein distance if the target data distribution is assumed to have a bounded support.
The main novel aspects of our work are as follows.
(1) We allow the dimension of the reference distribution be different from the dimension of the target distribution, in particular, it can be much lower than that of the target distribution.
We allow unbounded support for the reference distribution and the target distribution under mild conditions on the tail probabilities of the target distribution.

---

(3) We explicitly establish that the prefactors in the enror bounds depend on the square root of the dimension of the target distribution. This significant improvement over the exponential dependence on d in the existing works.
Moreover, we develop novel decomposition of integral probability metric for the error analysis of bidirectional GANs. We also show that the pushforward distribution of an empirical distribution based on neural networks can perfectly approximate another arbitrary empirical distribution long as the number of discrete points are the same.
Notation We max(2: 0), x E R. We use 7 to denote the identity map. Without further indication, ||. represents the L2 norm. For any function let ||g]|>> = supg [[](()]]-- We use notation O(-) and (-) to express the order of function slightly differently, where O-)) omits the universal constant independent of while O-) omits the constant depending on d. We use (a to denote the Lg ball in Rd with center at O and radius a. Let I#V be the pushforward distribution of by function g in the sense that (*v(A) (A)) for any measurable A. We use E to denote taking expectation with respect to the empirical distribution.
denote the ReLU activation function in neural networks, which is (
Bidirectional generative learning
We describe the setup of the bidirectional GAN estimation problem and present the assumptions need our analysis.
2.1 Bidirectional GAN estimators
Let be the target data distribution supported on Rd for d 2 1. Let v be a reference distribution which is easy to sample from. We first consider the case when v is supported on R and then extend it to R&* where k 2 1 can be different from d. Usually, k < d in practical machine learning tasks such as image generation. The goal to learn functions g
- Rdd and e IRd - such that (g, f) and == e), J4 is the pushforward distribution of v under
J4" éw where 9 and ē is the pushforward distribution of μ under é. We call J'v/ the joint latent distribution or joint reference distribution and é the joint data distribution joint target distribution. At the population level, the bidirectional GAN solves the minimax problem:
",e,f"" (g e" f*) E arg min max Ez Ef Z)]
gEg.eEE fEJ
where F.EFF are referred to as the generator class, the encoder class, and the discriminator class, respectively. Suppose we have two independent random samples Zi,..., Zn i and
XI....,X, Lid. μ. At the sample level, the bidirectional GAN solves the empirical version of the above minimax problem:
(go, Cip,epp fw) = arg
min EEN ENE
max
fa (Z:), 2x)
Σ ex),
(2.1)
where NNN and n are two classes of neural networks approximating the generator class and the encoder class E respectively, and FNN is class of neural networks approximating the discriminator class F.
2.2 Assumptions
We assume the target and the reference v satisfy the following assumptions. Assumption (Subexponential tail). For large n, the target distribution ls Rdd and the reference distribtuion v on R satisfies the first moment tail condition for some > 0,
fong m)
max E, / (12))
1 (IX])>) = O(n
Assumption (Absolute continuity). Both the target distribution μ on Ted and the reference distri- bution I. are absolutely continuous with respect to the Lebesgue measure X.
3

---

Assumption is a technical condition for dealing with the case when and v are supported on Fd and I instead of compact subsets. For distributions with bounded supports, this assumption automatically satisfied. Here the factor (log n) ensures that the tails of f and are sub-exponential, and it can be easily satisfied if the distributions are sabl-gassiin. For the reference distribution, Assumption [ and 2and2can can be easily satisfied by specifying as some common distribution with easy- to-sample density such as Gaussian or uniform, which is usually done in the applications of GANs. For the target distribution, Assumption bidirectional GAN with our theoretical guarantees. Note that Assumption is also necessary in our proof for bounding the generator and encoder approximation error in the sense that the results will not hold if we replace (logn) with 1. Assumption is also necessary for Theorem 4.3 in mapping between empirical samples, which is essential in bounding generator and encoder approximation emoo.
specifies the type of distributions that are learnable by
2.3 Generator, encoder and discriminator classes
Let FNN WW, LI) be the discriminator class consisting of the feedforward ReLU neural networks fw R2+1 + E with width W1 and depth L1. Similarly, let GNN == (W,, L2) be the generator class consisting of the feedforward ReL.U neural networks go -> Ted with width W2 and depth I2g and ENN- == NWWWW,, L2) the encoder class consisting of the feedforward ReLU neural networks Cp Rd + R. with width W3 and depth Lg-
The functions fw FNN have the following form:
fu(x) AL ((LL-1 ... a (a + b1)... ...................
b11-1) 1......bbb1
P7g
where A, are the weight matrices with number of rows and columns no larger than the width W1, b; the bias vectors with compatible dimensions, and a is the ReLU activation function o (xx) z'V.. Similarly, functions ge E INN and e E ENN have the following form:
ge(x) AL, oA - + b1) (x) AZ, -.. b()
b/2-1) 6/22
b%,,-) + b%,
where A and Aパ" are the weight matrices with number of rows and columns larger than W22 and W3, respectively, and b4 and b"" are the bias vectors with compatible dimensions.
We impose the following conditions GNN, ENN, and FNN-
Condition 1. For any ge E GNN and eip E ExN, we have max max{|lga ll 1le S log n.
Condition onn NNN can be easily satisfied by adding an additional clipping layer l after the original output layer, with Gn (log n /d.
{(a) A Cr.d V (-Cr,d) = a(a + Cn.d) a(a Cn.a) - Cn.d-
(2.2)
We truncate the output of ||go| to increasing interval [- log n, log support for the evaluation function class. Condition on εNN can be satisfied in the same manner. This condition is technically necessary in our proof (see appendix).
n] include the whole Rd
3 Non-asymptotic error bounds
We characterize the bidirectional GAN solutions based on minimizing the integral probability metric (IPM, Müller (197)) between two distributions μ and v with respect symmetric evaluation function class F. defined by
dr(μ, v) sup E.,. F.
(3.1)
fef
By specifying the evaluation function class F. differently, we can obtain many commonly -osm met- rics Liu al. 2017). Here focus the following two
F = bounded Lipschitz function class: dyr = dBL. (bounded Lipschitz (or Dudley) metric:
metrizing weak convergence, Dudley (2018)),
JF = Lipschitz function class dz = W, (Wasserstein GAN, Arjovsky et al. (2017))

---

We consider the estimation enror under the Dudley metric døL. Note that in the case when μ and have bounded supports, the Dudley metric dBr. is equivalent to the I-Wasserstein metric W Therefore, under the bounded support condition for and all our convergence results also hold under the Wasserstein distance Wi. Even if the support of f and are unbounded, we can still apply the result of Lu and Lu (2020) to avoid empirical process theory and obtain an stochastic error bound under the Wasserstein distance Wi. However, the result Lu and Lu (2020) requires sub-gaussianity to obtaín the vd prefactor. In order to make it more general, we use the empirical processes theory to get the explicit prefactor. Also, the discriminator approximation error will be unbounded if we consider the Wasserstein distance W1. Hence, we can only consider dBL for the unbounded support case.
The bidirectional GAN solution (ğo. (p) in 2.1 also minimizes the distance between (go 2 and # An under drww
min drwN ((āo ( &nn ( 80 GExw
However, even two distributions are close with respect to drwNs there is no automatic guarantee that they will still be close under other metrics, for example, the Dudley or the Wasserstein distance (Arora et al. 2017). Therefore, it is natural to ask the question:
How close are the two bidirectional GAN estimators -= = (ğo. n) and ju == (1
under some other stronger metrics?
We consider the IPM with the uniformly bounded I-Lipschitz function class on Rd+1 the evalu- ation class, which is defined as, for some finite B > 0,
RR+1 + R If(=) f(v)
= | ull x, E IR+11 and II
S B)
(3.2)
In Theorem 3.1 consider the bounded support where dy Wi: In Theorem 3.2) extend the result to the unbounded support case; In Theorem 3.3 we extend the result to the case where the dimension of the reference distribution is arbitrary.
We first present a result when j is supported on a compact subset [-M, C. IRd and v is sup- ported on -M, M]] C IR for finite M > O.
Theorem 3.1. Suppose that the target is supported on [-M, 1-M,MY M C Rd and the reference v is supported on M, .M] C. IR for a finite M > and Assumption 2 holds. Let the outputs of go and G, be within [-M, M1 and [-M, M] for go GNNN and ep ENN, respectively. By specifying the three network structures as Wi Li 2 [VT], W3 Cidn, and W La = C2n for some constants 12 < C1, C2 < 384 and properly choosing parameters, we have Edy- (i, j S Covdn a (log n) 4t1, ais,, where Co > 0 is a constant independent of d and n.
The prefactor Covd in the error bound depends on d//2 linearly. This is different from the existing works where the dependence of the prefactor on d is either not clearly described or is exponential. In high-dimensional settings with large d. this makes a substantial difference the quality of the error bounds. These remarks apply to all the results stated below.
The next theorem deals with the case of unbounded support.
Theorem 3.2. Suppose Assumption D and 2 hold, and Condition D is satisfied. By specifying the structures of the three network classes W,I, [vn], W Cidn, and W La C2n for some constants 12 < C1, C22 S 384 and properly choosing parameters, we have
Edy: (i) min
पोх (logn)*+at यtं, Can योュ logn),
where Co is constant independent of d and n. but C. depends on d.
Note that two methods used in bounding stochastic errors (see appendix), which leads two different bounds: one with explicit Vd prefactor with the cost that have additional log factor. Another one with an implicit prefactor but with a better log n factor. Hence, it is a tradeoff between the explicitness of prefactor and the order of log n.
Our next result generalizes the results to the case when the reference distribution supported on k for k N
5

---

Assumption 3. The target distribution on R is absolutely continuous with respect to the Lebesgue measure on I and the reference distribution v on IK is absolutely continuous with respect to the Lebesgue measure on TR and k < d.
With the above assumption, we have the following theorem providing theoretical guarantees for validity of any dimensional reference
Theorem 3.3. Suppose Assumption 1 fand3holl and Condition is satisfied. By specifying gener- ator and discriminator class structure as W,L, 2 [vn]. W L2 C,dn, and W Ls C2kn for some constants 12 S C1. C2 S 384 and properly choosing parameters, we have
Bd (i: pA) S min {Co ddn बेत (log t Can येн log n
where Co constant independent and n, but Ca depends d.
Note that the errors bounds established in Theorems 3.1 3.3 1.30g are tight up to a logarithmic factor, since the minimax rate measured in Wasserstein distance for learning distributions when the Lipschitz evaluation class defined on IRd ((-+)) +) (Liang 2020
Approximation and stochastic errors
In this section present novel inequality for decomposing the total error into approximation and stochastic errors and establish bounds on these errors.
4.1 Decomposition of the estimation error
Define the approximation enror of function class F to another function class H by
EHH,F) == sup inf h NEMIEF
flloo-
We decompose the Dudley distance d (i, (s) between the latent joint distribution and the data joint distribution into four different error terms,
the approximation error of the discriminator class FNN to F!:
Σ1 = E
the approximation error of the generator and encoder classes:
Eg
inf ge C ENNNNN n.
sup
Σ (foloo()), fa(())
the stochastic error for the latent joint distribution é:
Eg sup Eld -B)(((()),)) fWet
the stochastic eиror for the latent joint distribution je:
= sup (L ES,( FEEFI
Lemma 4.1. Let (go, E.p) be the bidirectional GAN solution in 22. and F be the uniformly bounded 1-Lipschitz function class defined in (3.2). Then the Dudley distance between the latent joint distribution ò (ğo. I) 4v and the data joint distribution = (I, &, 4 be decomposed as follows
functiion
d.)(1,)) S 28 + Eg Es Ea.
(4.1)
The novel decomposition 4.1 is fundamental to our error analysis. Based on (.1 we bound
each error term on the right side (4.1) and balance the bounds to obtain an overall bound for the bidirectional GAN estimation.

---

For proving Lemma 4.1 we introduce the following useful inequality, which states that for any two probability distributions, the difference in IPMs with distinct evaluation classes will not exceed 2 times the approximation error between the two evaluation classes, that is, for any probability distributions y and and symmetric function classes and H.
du(p,v) dy (1, - (H..
(4.2)
is easy to check that if replace (z(u, v) by d=((,))= sup [Bnh Eyd,, 44.2) still holds. hEH
Proof of Lemma 4.1
We have
dr (D, (1) = sup (f=(O(=), -(L()d)))) L
I sup (f= 2) (f-(((=), 2) + sup (f)()()), (f,((, wuem
WEE
+ sup (x, ((x)) (x,
f_EFi
=8g Est + sup (f(=) (x, f
Denote A = E (O(), (z, By 4.2) and the optimality of the bidirec- tional GAN solutions, A satisfies
A = sup 1 fWEF
Σ5
- -ulo,
Σ (I)(((Z)).2) - -fa (xx, + +EE(F) FFNN)
$ sup f.EFNN n
inf guEDNN EENN ENNNN
sup
1
Σ
(Fo()o(z))) fuloi, + 23,
1 i=1
2E, Ez.
Note that we cannot directly apply the symmetrization technic (see appendix) to Es and & since e* and g" are correlated with r; and 2,. However, this problem can be solved by replacing the samples (I:, 2:) in the empirical terms in Eg and & with ghost samples (a/, 2() independent of (æ;, 24) and replacing 9 and ex with g²* and e** which are obtained from the ghost samples, respectively. That is, we replace f. (x), 2) and (x, (x)) with L (g
(g""(2"), (z"), and (x', , (x")) in &a and Es, respectively. Then can proceed with the same proof of Lemma 4.1 and apply the sym- metrization technic to Eg and E,, since (g"(2) (2+), 2)) and (g" (2()( 2f) have the same distribution. To simplify the notation, we will just use (g (z), 2) and f (x, e (x)) to denote (g (z'),2') and ( e (x')) here, respectively.
4.2
Approximation errors
We now discuss the errors due to the discriminator approximation and the generator and encoder approximation.
4.2.1 The discriminator approximation error E₁
The discriminator approximation error E1 describes how well the discriminator neural network class approximates functions from the Lipschitz class Fi. Lemma 4.2 below can applied to obtain the neural network approximation enror for Lipschitz functions. It leads to quantitative and non- asymptotic approximation rate in terms of the width and depth of the neural networks when bounding E,.
7

---

Lemma 4.2 (Shen et al. (2021)). Let f be a Lipschitz continuous function defined on [-R, Rj For arbitrary W, N.. there exists a function v implemented by a ReLU feedforward neural network with width W and depth I, such that
|f - = =
By Lemma 4.2 and our choice of the architecture of discriminator class FNN in the theorems, have & ((v((W,L)) - *ें logn). Theorem 4.2 also informs about how choose the architecture of the discriminator networks based on how small we want the approximation error & be. By setting (1) 2 n, & dominated by the stochastic terms &g and E
4.2.2 The generator and encoder approximation error &2
The generator and encoder approximation error &g describes how powerful the generator and en- coder classes are in pushing the empirical distributions f., and i, each other. A natural question
Can we find some generator and encoder neural network functions such that Eg = 0?
Most of the current literature concerning the error analysis of GANs applied the optimal transport theory [Villani 2008) to minimize an eтror term similar to Eg, see, for example, Chen et al. (020))
However, the existence of the optimal transport function from R F not guaranteed. Therefore, the existing analysis of GANs can only deal with the scenario when the reference and the target data distribution are assumed to have the same dimension. This equal dimensionality assumption not satisfied in the actual training of GANs or bidirectional GANs in many applications. Here, instead of using the optimal transport theory, we establish the following approximation results Theorem 4.3 which enables to forgo the equal dimensionalityy assumption.
Theorem 4.3. Suppose that supported on R and p. supported on I& are both absolutely continuous w.E. the Lebesgue measures, and z/s and x/s are i.i.d. samples from v and μ.. respectively for Rd and
S i < n. Then there exist generator and encoder neural network functions g R + TR. such that g and e are inverse bijections of each other between {z, 1 s < n} and {z; -1si iis S. n} up to a permutation. Moreover, such neural network functions and be obtained by properly specifying W c2dn and W L3 =3n for some constant 12 < 02 03 S 384.
Proof. By the absolute continuity of u and μr, all the z/8 and w/s are distinct a.s.. We can reorder as from the smallest to the largest, so 21 22 <... . 2n. Let 2141/2 /2 be any point between 2; and Z441 for E (1, ...... 1}. We define the continuous piece-wise linear function g R + Rd by
F1
2 S 21,
3-3+1//2 +
8-2- Ti41
2 = (Zi 141122) for = 1,..., n 1,
g(2) =
2-4
D4+11
2 E [a+4] 1/2 2141), for = 1,..., 22
2 2 2n 1+1/2-
By Yang al. 22021
Lemma 3.1) g E (,,(W,,,2) n (W2 d 1) W2- W2-d-1 422 Taking 6d
n (W₂-
La 2 J. a simple calculation shows W L.2 edn for some constant 12 < c < 384. The existence of neural net function e can be constructed in the same way due to the
1)
d--d-1 6d
fact that the first coordinate of zía are distinct almost surely.
When the number of point masses of the empirical distributions are relatively moderate compared with the structure of the neural nets, we can approximate empirical distributions arbitrarily well with any empirical distribution with the same number of point masses pushforwarded by the neural nets.
Theorem 4.3 provides an effective way to specify the architecture of generator and encoder classes.
According to this lemma, we can take n 2
Wa-d
La
+ 2 W,-1 Wa-1 A + 2.
6d
which gives rise to W L2/d x L3 x More importantly, Theorem 4.3 can be applied to
8

---

bound & as follows.
"
& =
inf B EENN fWEJNN 1=1
sup
Σ (o(((())) - (an ex))))
Σ (o((()z)) - fa(a))2))
inf sup goEUNN ENEFNN "
sup EEENNN JWEFNN
inf
Σ (1(1)))))) (x(,,,)
=0.
We simply reordered z/s and æ/s in the proof. Therefore, this error term can be perfectly elimi- nated.
4.3 Stochastic errors
The stochastic error Eg (E4) quantifies how close the empirical distribution and the true latent joint distribution (data joint distribution) are with the Lipschitz class F as the evaluation class under
IPM. We apply the results in the refined Dudley inequality (Schreuder 2020 in Lemma to bound Eg and ε..
Lemma 4.4 (Refined Dudley Inequality). For symmetric function class F with sup ll S M, we have
Eidy(Pnn [J] S
inf 0<B<M
46 +
11o0
Vn
The original Dudley inequality (Dudley 1967: Van der Vaart and Wellner 1996) suffers from the problem that if the covering number (Cc,FFF (oo) increases too fast goes to 0, then the upper bound will be infimity, which is totally meaningless. The improved Dudley inequality circumvents such a problem by only allowing to integrate from 6 0 as is shown in Lemma 4.4 which also indicates that [EE3 scales with the covering number [e, 1lso).
By calculating the covering number of Fl and utilizing the refined Dudley inequality, we can obtain the upper bound
max KE1} = =0(can- - क logn A Vdn शे (log))++at
(4.3)
Related work
Recently, several impressive works have studied the challenging problem the convergence prop- erties of unidirectional GANs. AAooa al. (2017) noted that training of GANs may not have good generalization properties in the sense that even if training may appear successful but the trained dis- tribution may be far from target distribution in standard metrics. On the other hand, Bai et al. (2019) showed that GANs can learn distributions in Wasserstein distance with polynomial sample com- plexity. [Liang] (2020) studied the rates of convergence of class of GANs, including Wasserstein, Sobolev and MMD GANs. This work also established the nonparametrie minimax optimal rate un- der the Sobolev IPM. The results of Bai al. (2019) and Liang] (2020) require invertible generator networks, meaning all the weight matrices need to be full-rank, and the activation function needs be the invertible leaky ReLU activation. Chen et al. (2020) established an upper bound for the estimation error rate under Hölder evaluation and target density classes, where 7 is Hölder class with regularity 3 and the density of the target j assumed to belong H!" They assumed that the reference distribution has the same dimension as the target distribution and applied the optimal transport theory to control the generator approximation error. However, how prefactor depends in the error bounds on the dimension d in the existing results (Liang 2020 Chen et al. (020) is either not clearly described or is exponential. In high-dimensional settings with large this makes substantial difference in the quality of the error bounds.

---

Singh et al. (2019) studied minimax convergence rates of nonparametric density estimation under class of adversarial losses and investigated how the choice of loss and the assumed smoothness of the underlying density together determine the minimax rate; they also discussed connections to learning generative models
minimax statistical sense. [Uppal et al. (2019) generates the idea of Sobolev IPM to Besov IPM, where both target density and the evaluation classes are Besov classes. They also showed how their results imply bounds on the statistical error of a GAN.
These results provide important insights in the understanding of GANs. However, as we mentioned earlier, some of the assumptions made in these results, including equal dimension between the ref- erence and target distributions and bounded support of the distributions, are not satisfied in the training of GANs in practice. Our results avoid these assumptions. Moreover, the prefactors in our error bounds are clearly described as being dependent on the square root of the dimension d. Finally, the aforementioned results only dealt with unidirectional GANs. Our work the first to address the convergence properties of bidirectional GANs.
6
Conclusion
This paper derives the error bounds for the bidirectional GANs under the Dudley distance between the latent joint distribution and the data joint distribution. The results are established without the two crucial conditions that are commonly assumed in the existing literature: equal dimensionality between the reference and the target distributions and bounded support for these distributions. Ad- ditionally, this work contributes to the neural network approximation theory by constructing neural network functions such that the pushforward distribution of an empirical distribution can perfectly approximate another arbitrary empirical distribution with a different dimension as long as their num- ber of point masses equal. A novel decomposition of integral probability metric is also developed for eтror analysis of bidirectional GANs, which can be useful in other generative learning problems.
A limitation of our results, well as all the existing results the convergence properties of GANs, is that they suffer from the curse of dimensionality, which cannot be circumvented by assuming sufficient smoothness assumptions. In many applications, high-dimensional complex data such as images, texts and natural languages, tend to be supported on approximate lower-dimensional man- ifolds. It is desirable take into such structure in the theoretical analysis. An important extension of the present results is to show that bidirectional GANs can circumvent the curse of dimensionality if the target distribution assumed be supported on an approximate lower-dimensional manifold. This appears to be technically challenging problem and will be pursued in our future work.
Acknowledgements
The authors wish thank the three anonymous reviewers for their insightful comments and con- structive suggestions that helped improve the paper significantly.
The work of J. Huang is partially supported by the U.S. NSF grant 1MS-1916199. The work of Y. Jiao is supported in part by the National Science Foundation of China under Grant 1 1871474 and by the research fund of KLATASDSMOE. The work of Y. Wang is supported in part by the Hong Kong Research Grant Council grants 1630851 8 and 16317416 and HK Innovation Technology Fund ITS/044/18XX, as well Guangdong- Kong-Macao Joint Laboratory for Data-Driven Fluid Mechanics and Engineering Applications.
10

---

References
Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein generative adversarial networks. In ICML.
Arora, S., Ge, R., Liang, Y. Ma, T., and Zhang, Y. (2017). Generalization and equilibrium in generative adversarial nets (gans). International Conference on Machine Learning, pages 224- 232. PMLR.
Bai, Y. Ma, T., and Risteski, (2019). Approximability of discriminators implies diversity in GANs. In International Conference on Learning Representations.
Brock, A., Donahue, J., and Simonyan, (2019). Large scale gan training for high fidelity natural image synthesis.
Chen, M., Liao, W. Zha, H., and Zhao, T. (2020). Statistical guarantees of generative adversarial networks for distribution estimation. arXiv preprint arXiv:2002.0395:
Donahue, J. Krähenbühl, and Darrell, T. (2016). Adversarial feature learning. arXiv preprint arXiv:1605.09782.
Dudley, R. (1967). The sizes of compact subsets of hilbert space and continuity of gaussian pro- cesses, Journal of Functional Analysis, 1(3):290-330.
Dudley, M. (2018). Real Analysis and Probability. CRC Press.
Dumoulin, V., Belghazi, I., Poole, B., Mastropictro, O., Lamb, A., Arjovsky, M., and Courville, A. (2016). Adversarially learned inference. arXiv preprint urXiv:1606.0074 00704.
Goodfellow, I. J., Pouget-Abadic, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, (2014). Generative adversarial networks. arXiv preprint 1406.2240.2660
Karras, T., Aila, T., Laine, S., and Lehtinen, J. (2018). Progressive growing of gans for improved quality, stability, and variation.
Karras, T., Laine, S., and Aila, T. (2019). A style-based generator architecture for generative adver- sarial networks.
Liang, T. (2020). How well generative adversarial networks learn distributions.
Liu, S., Bousquet, O., and Chaudhuri, K. (2017). Approximation and convergence properties of generative adversarial learning. arXiv preprint arXiv: 1705.0899.
Lu, Y. and Lu, (2020). universal approximation theorem of deep neural networks for expressing distributions. arXiv preprint 0788
Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., and Frey, B. (2015). Adversarial autoencoders. arXiv preprint /5
Müller, (1997). Integral probability metries and their generating classes of functions. Advances in Applied Probability, pages 429 443.
Radford, A., Metz, I.,, and Chintala, (2016). Unsupervised representation learning with deep convolutional generative adversarial networks.
Reed, S., Akata, 7.,, Yan, X., Logeswaran, I.., Schiele, B., and Lee, H. (2016). Generative adversarial text image synthesis. In ICML.
Schreuder, (2020). Bounding the expectation of the supremum of empirical processes indexed by hölder classes.
Shen, X., Zhang, T., and Chen, (2020). Bidirectional generative modeling using adversarial gradient estimation. arXiv preprint 2rX1v.200..09161.

---

Singh, S., Uppal. A., Li, B., Li, C.-L., Zaheer, M., and Póczos, B. (2019). Nonparametric density estimation with adversarial losses. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 10246-10257.
Uppal, A., Singh, S., and Póczos. B. (2019). Nonparametric density estimation & convergence rates for gans under besov ipm losses. arXiv preprint arXiv:1902.03511.
Van der Vaart, A. W. and Wellner, A. (1996). Weak convergence. In Weak convergence and empirical processes. Springer.
Villani, C. (2008). Optimal Transport: Old and New, volume 338, Springer Science & Business Media.
Yang, Y., Li, Z., and Wang, Y. (2021). On the capacity of deep generative networks for approximat- ing distributions. arXiv preprint arXiv:2101. 12353..
Zhang, P., Liu, Q.. Zhou, D., Xu, T., and He, X. (2018). On the discrimination-genraalaaiiio tradeoff in GANs. In International Conference Learning Representations.
Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV.
12


Reviews:
[{"review_id": "wpZ_ZuY5FjR", "reviewer": "Reviewer_eow5", "summary": "The paper presents non-asymptotic error analysis for bidirectional GANs. The distinctive feature of the error analysis is relaxing the assumption that the data and latent variable should have the same dimension.  The constants in the error bound is also explicit in dimension.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and discusses the related literature well. I could not go through the details of error analysis, but the statement of the results is rigorous. I have some question/comments about the assumptions and results.  \n\n\n1- What is the motivation to study bi directional GAN as opposed to GAN? Can these results be extended to GAN, or is there a special property of GAN that allows relaxing the assumptions?\n\n2- What are the limitations of Assumption 2? Does it exclude data distributions that lie in low-dimensional manifolds? How necessary is this assumption? \n\n3- For Theorem 3.2., the constant for the unbounded case does not seem to be explicit as the paper suggests. It seems to better distinguish this in the introduction\n\n4- It is surprising that the dimension k does not appear in result in Theorem 3.3. Is there any explanations for this?  \n\n5- I did not understand how the results in Theorem 4.3 is extended to multi-dimensional case for $\\nu$.  \n\nOverall, the paper is well written and comparison is given with the related literature. I only suggest including an outline to navigate the theoretical results ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "pt6qEHuXxO-", "reviewer": "Reviewer_jcaY", "summary": "This paper studies the statistical properties of bidirectional GANs. It studies the sample complexity of the bidirectional GAN estimate under the Dudley metric. They relax a couple of assumptions usually made in GAN theory literature; they do not assume that the dimension of the latent distribution and the data distribution are the same or that the true data distribution has bounded support. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting paper that provides useful insight for understanding bidirectional GANs. It is clearly written and well organized. This work shows to theoretically handle the bidirectionality in the particular GAN framework and unboundedness of the support of the data. They also show via a decomposition of the generalization error how the different dimensions of the latent and the data distribution affect this error. Moreover, they provide explicit prefactors for the upper bound of the generalization error. \n\nIt is important to note that explicit prefactors are only provided in the case of bounded support of the data. In this case the Dudley metric and the Wasserstein distance is the same. Since the paper by Lu and Lu (2020) cited in the related work study the Wasserstein distance, does their proof imply the same dependence of the pre factor on the dimension? Moreover, given that the generalization bound has exponential dependence on the dimension when is a better estimate of the prefactor important?\n\nMy other concerns/suggestions with this work are as follows: \n- It would be nice to have lower bounds for statistical complexity under the setting considered in this work to better understand the results. \n- Often the number of samples from the latent distribution used to train GANs far exceeds that of the true data. It would provide useful perspective to incorporate this into the results. \n- Some discussion on the computational complexity of computing the GAN estimator proposed would be very beneficial even if only under the setting where the discriminator is replaced by an oracle that provides the loss. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed some limitations of this work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "_wVtLMxbjmH", "reviewer": "Reviewer_jF5w", "summary": "This paper presents non-asymptotic error bounds for bidirectional GAN estimators. Their primary contribution is a dependence on the square root of dimension under certain assumptions on the data distribution and the latent distribution.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality\n\nThe task is not new, but the results are new. One minor comment for highlighting differences with previous contributions is to write out previously known results in terms of equations, rather than only in plain words as is the case of the current version. Related work is adequately cited.\n\n2. Quality\n\nThe submission seems technically sound. It might be a good idea to add a bit more discussion into the dependence on n and C_d as well---the current paper focuses on the square root dependence on d, but the main theorems (3.1., 3.2., 3.3.) all depend on specifying generator and discriminator class structure in terms of n, d., and C_d \"depends on d\". I think it would be helpful to have some more discussion in terms of these dependencies. Following the above point, it would be nice to address whether general results would also hold without a specification of the class structures; otherwise, the claims seem to be fairly specific and only holds for these particular specifications.  \n\n3. Clarity\n\nThe paper is fairly well-written and organized. Some comments on typos are addressed in Section 5. of the main review. Some comments regarding clarity:\n\na. The notation $\\hat{\\mathbb{E}}$ is not clearly defined in Line 188-189.\n\nb. There is no notion of specifying network structures in the abstract, and it leaves an impression that the results are general and hold true for a wide range of bidrectional GANs. This may need to be addressed in further revisions of this paper.\n\nc. Following the above point, it leads to the question of whether the assumptions and conditions are suitable for this particular specification, and what would happen under looser assumptions: even in Section 2.2 the authors pointed out that \"Assumptions 1 and 2 can be easily satisfied by specifying $\\nu$ as some common distribution with easy-to-sample density such as Gaussian or uniform...\". It would be interesting if the necessity of these assumptions are discussed in a clearer sense.\n\nd. The use of the $\\implies$ signs in Lines 137-139 are slightly ambiguous.\n\ne. Perhaps a minor point, but in the abstract and Section 1 the term \"target distribution\" is used but is not clearly defined. It may be helpful to clarify in further revisions the difference/similarities of the \"target distribution\" compared with the \"data distribution\" and \"latent distribution\".\n\n\n4. Significance\n\nThe results are interesting and bring insight into understanding bidirectional GANs. From this perspective, the results are important. However, I think the significance would be clearer if some of the above points are addressed.\n\n5. Typos\n\nSome typos:\n\na. In Theorems 3.2. and 3.3., \"Conditions 1\" may be \"Condition 1\".\n\nb. In Line 145: \"defined\" may be \"define\".\n\nUpdate after reviewing comments: Increased score to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss some societal impact, but there are no societal issues due to the theoretical nature of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents non-asymptotic error analysis for the bidirectional GANs.
In the analysis the authors have succeeded in relaxing those assumptions usually adopted in the GAN literature, such that the latent distribution and the data distribution has the same dimension and that the true data distribution has a bounded support. They have also succeeded in making some of the coefficients in the error bounds explicit. These constitute the main contributions of this paper.

The reviewers raised several questions, and the authors have addressed them adequately, as evidenced by two of the reviewers having raised their scores from below the threshold to above. Now all the review scores are above the threshold, I am happy to recommend acceptance of this paper for presentation at the NeurIPS conference.

Minor points:
- A learnability issue: Lemma 4.2 is based on existence of a neural network $\\psi$, and Theorem 4.3 is based on existence of neural networks $g$ and $e$, with desired properties. These lemma and theorem do not care about the learnability of these neural networks. It would therefore be nicer if some empirical results be presented to at least suggest that the learnability issue would be minor in practice, so that the stated error bounds are indeed relevant.
- Line 34: join(t)
- Line 59: the data joint distribution.( )To the best
- Lines 93, 94: $(\sim)\nu$, $(\sim)\mu$
- Lines 121, 124: vector(s)
- Line 138: metri(ci)zing
- Equation between lines 186 and 187: $h\in(\mathcal{F}\to\mathcal{H})$
- Definitions of $\mathcal{E}_3$ and $\mathcal{E}_4$: In the proof of Lemma 4.1, it seems that the authors used slightly different definitions of these quantities, where $g^*$ and $e^*$ are replaced with $\hat{g}$ and $\hat{e}$, respectively.
- Line 267: cover(ing) numbers
- SM, line 26: $|z_0|\le\log n$ $\to$ $\\\|z_0\\\|\le\log n$
Decision: Accept (Poster)