Non-asymptotic Error Bounds for Bidirectional GANs

∗Corresponding authors 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Introduction . Generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) is an important approach to implicitly learning and sampling from high-dimensional complex distributions . GANs have been shown to achieve impressive performance in many machine learning tasks ( Radford et al. , 2016 ; Reed et al. , 2016 ; Zhu et al. , 2017 ; Karras et al. , 2018 , 2019 ; Brock et al. , 2019 ) . Several recent studies have generalized GANs to bidirectional generative learning , which learns an encoder mapping the data distribution to the reference distribution simultaneously together with the generator doing reversely . These studies include the adversarial autoencoder ( AAE ) ( Makhzani et al. , 2015 ) , bidirectional GAN ( BiGAN ) ( Donahue et al. , 2016 ) , adversarially learned inference ( ALI ) ( Dumoulin et al. , 2016 ) , and bidirectional generative modeling using adversarial gradient estimation ( AGES ) ( Shen et al. , 2020 ) . A common feature of these methods is that they generalize the basic adversarial training framework of the original GAN from unidirectional to bidirectional . Dumoulin et al . ( 2016 ) showed that BiGANs make use of the joint distribution of data and latent representations , which can better capture the information of data than the vanilla GANs . Comparing with the unidirectional GANs , the joint distribution matching in the training of bidirectional GANs alleviates mode dropping and encourages cycle consistency ( Shen et al. , 2020 ) . Several elegant and stimulating papers have analyzed the theoretical properties of unidirectional GANs . Arora et al . ( 2017 ) considered the generalization error of GANs under the neural net distance . Zhang et al . ( 2018 ) improved the generalization error bound in Arora et al . ( 2017 ) . Liang ( 2020 ) studied the minimax optimal rates for learning distributions with empirical samples under Sobolev evaluation class and density class . The minimax rate is O ( n−1/2 ∨ n−α+β/ ( 2α+β ) ) , where α and β are the regularity parameters for Sobolev density and evaluation class , respectively . Bai et al . ( 2019 ) analyzed the estimation error of GANs under the Wasserstein distance for a special class of distributions implemented by a generator , while the discriminator is designed to guarantee zero bias . Chen et al . ( 2020 ) studies the convergence properties of GANs when both the evaluation class and the target density class are Hölder classes and derived O ( n−β/ ( 2β+d ) log2 n ) bound , where d is the dimension of the data distribution and α and β are the regularity parameters for Hölder density and evaluation class , respectively . While impressive progresses have been made on the theoretical understanding of GANs , there are still some drawbacks in the existing results . For example , ( a ) The reference distribution and the target data distribution are assumed to have the same dimension , which is not the actual setting for GAN training . ( b ) The reference and the target data distributions are assumed to be supported on bounded sets . ( c ) The prefactors in the convergence rates may depend on the dimension d of the data distribution exponentially . In practice , GANs are usually trained using a reference distributions with a lower dimension than that of the target data distribution . Indeed , an important strength of GANs is that they can model low-dimensional latent structures via using a low-dimensional reference distribution . The bounded support assumption excludes some commonly used Gaussian distributions as the reference . Therefore , strictly speaking , the existing convergence analysis results do not apply to what have been done in practice . In addition , there has been no theoretical analysis of bidirectional GANs in the literature . 1.1 Contributions . We derive nearly sharp non-asymptotic bounds for the GAN estimation error under the Dudley distance between the reference joint distribution and the data joint distribution . To the best of our knowledge , this is the first result providing theoretical guarantees for bidirectional GAN estimation error rate . We do not assume that the reference and the target data distributions have the same dimension or these distributions have bounded support . Also , our results are applicable to the Wasserstein distance if the target data distribution is assumed to have a bounded support . The main novel aspects of our work are as follows . ( 1 ) We allow the dimension of the reference distribution to be different from the dimension of the target distribution , in particular , it can be much lower than that of the target distribution . ( 2 ) We allow unbounded support for the reference distribution and the target distribution under mild conditions on the tail probabilities of the target distribution . ( 3 ) We explicitly establish that the prefactors in the error bounds depend on the square root of the dimension of the target distribution . This is a significant improvement over the exponential dependence on d in the existing works . Moreover , we develop a novel decomposition of integral probability metric for the error analysis of bidirectional GANs . We also show that the pushforward distribution of an empirical distribution based on neural networks can perfectly approximate another arbitrary empirical distribution as long as the number of discrete points are the same . Notation We use σ to denote the ReLU activation function in neural networks , which is σ ( x ) = max { x , 0 } , x ∈ R. We use I to denote the identity map . Without further indication , ‖ · ‖ represents theL2 norm . For any function g , let ‖g‖∞ = supx ‖g ( x ) ‖ . We use notationO ( · ) and Õ ( · ) to express the order of function slightly differently , where O ( · ) omits the universal constant independent of d while Õ ( · ) omits the constant depending on d. We use Bd2 ( a ) to denote the L2 ball in Rd with center at 0 and radius a . Let g # ν be the pushforward distribution of ν by function g in the sense that g # ν ( A ) = ν ( g−1 ( A ) ) for any measurable set A . We use Ê to denote taking expectation with respect to the empirical distribution . 2 Bidirectional generative learning . We describe the setup of the bidirectional GAN estimation problem and present the assumptions we need in our analysis . 2.1 Bidirectional GAN estimators . Let µ be the target data distribution supported on Rd for d ≥ 1 . Let ν be a reference distribution which is easy to sample from . We first consider the case when ν is supported on R , and then extend it to Rk , where k ≥ 1 can be different from d. Usually , k d in practical machine learning tasks such as image generation . The goal is to learn functions g : R → Rd and e : Rd → R such that g̃ # ν = ẽ # µ , where g̃ : = ( g , I ) and ẽ : = ( I , e ) , g̃ # ν is the pushforward distribution of ν under g̃ and ẽ # µ is the pushforward distribution of µ under ẽ . We call g̃ # ν the joint latent distribution or joint reference distribution and ẽ # µ the joint data distribution or joint target distribution . At the population level , the bidirectional GAN solves the minimax problem : ( g∗ , e∗ , f∗ ) ∈ arg min g∈G , e∈E max f∈F EZ∼ν [ f ( g ( Z ) , Z ) ] − Ex∼µ [ f ( X , e ( X ) ) ] , where G , E , F are referred to as the generator class , the encoder class , and the discriminator class , respectively . Suppose we have two independent random samples Z1 , . . . , Zn i.i.d.∼ ν and X1 , . . . , Xn i.i.d.∼ µ . At the sample level , the bidirectional GAN solves the empirical version of the above minimax problem : ( ĝθ , êϕ , f̂ω ) = arg min gθ∈GNN , eϕ∈ENN max fω∈FNN 1 n n∑ i=1 fω ( gθ ( Zi ) , Zi ) − 1 n n∑ j=1 fω ( Xj , eϕ ( Xj ) ) , ( 2.1 ) where GNN and ENN are two classes of neural networks approximating the generator class G and the encoder class E respectively , andFNN is a class of neural networks approximating the discriminator class F . 2.2 Assumptions . We assume the target µ and the reference ν satisfy the following assumptions . Assumption 1 ( Subexponential tail ) . For a large n , the target distribution µ on Rd and the reference distribtuion ν on R satisfies the first moment tail condition for some δ > 0 , max { Eν‖Z‖1 { ‖Z‖ > logn } , Eµ‖X‖1 { ‖X‖ > logn } } = O ( n− ( logn ) δ d ) . Assumption 2 ( Absolute continuity ) . Both the target distribution µ on Rd and the reference distribution ν on R are absolutely continuous with respect to the Lebesgue measure λ . Assumption 1 is a technical condition for dealing with the case when µ and ν are supported on Rd and R instead of compact subsets . For distributions with bounded supports , this assumption is automatically satisfied . Here the factor ( log n ) δ ensures that the tails of µ and ν are sub-exponential , and it can be easily satisfied if the distributions are sub-gaussian . For the reference distribution , Assumption 1 and 2 can be easily satisfied by specifying ν as some common distribution with easyto-sample density such as Gaussian or uniform , which is usually done in the applications of GANs . For the target distribution , Assumption 1 and 2 specifies the type of distributions that are learnable by bidirectional GAN with our theoretical guarantees . Note that Assumption 1 is also necessary in our proof for bounding the generator and encoder approximation error in the sense that the results will not hold if we replace ( log n ) δ with 1 . Assumption 2 is also necessary for Theorem 4.3 in mapping between empirical samples , which is essential in bounding generator and encoder approximation error . 2.3 Generator , encoder and discriminator classes . Let FNN : = NN ( W1 , L1 ) be the discriminator class consisting of the feedforward ReLU neural networks fω : Rd+1 7→ R with width W1 and depth L1 . Similarly , let GNN : = NN ( W2 , L2 ) be the generator class consisting of the feedforward ReLU neural networks gθ : R 7→ Rd with width W2 and depth L2 , and ENN : = NN ( W3 , L3 ) the encoder class consisting of the feedforward ReLU neural networks eϕ : Rd 7→ R with width W3 and depth L3 . The functions fω ∈ FNN have the following form : fω ( x ) = AL1 · σ ( AL1−1 · · ·σ ( A1x+ b1 ) · · ·+ bL1−1 ) + bL1 where Ai are the weight matrices with number of rows and columns no larger than the width W1 , bi are the bias vectors with compatible dimensions , and σ is the ReLU activation function σ ( x ) = x∨0 . Similarly , functions gθ ∈ GNN and eϕ ∈ ENN have the following form : gθ ( x ) = A ′ L2 · σ ( A ′ L2−1 · · ·σ ( A ′ 1x+ b ′ 1 ) · · ·+ b′L2−1 ) + b ′ L2 eϕ ( x ) = A ′′ L3 · σ ( A ′′ L3−1 · · ·σ ( A ′′ 1x+ b ′′ 1 ) · · ·+ b′′L3−1 ) + b ′′ L3 where A′i and A ′′ i are the weight matrices with number of rows and columns no larger than W2 and W3 , respectively , and b′i and b ′′ i are the bias vectors with compatible dimensions . We impose the following conditions on GNN , ENN , and FNN . Condition 1 . For any gθ ∈ GNN and eϕ ∈ ENN , we have max { ‖gθ‖∞ , ‖eϕ‖∞ } ≤ log n. Condition 1 on GNN can be easily satisfied by adding an additional clipping layer ` after the original output layer , with cn , d ≡ ( log n ) / √ d , ` ( a ) = a ∧ cn , d ∨ ( −cn , d ) = σ ( a+ cn , d ) − σ ( a− cn , d ) − cn , d . ( 2.2 ) We truncate the output of ‖gθ‖ to an increasing interval [ − log n , log n ] to include the whole Rd support for the evaluation function class . Condition 1 on ENN can be satisfied in the same manner . This condition is technically necessary in our proof ( see appendix ) .

Summary: This paper studies the statistical properties of bidirectional GANs. It studies the sample complexity of the bidirectional GAN estimate under the Dudley metric. They relax a couple of assumptions usually made in GAN theory literature; they do not assume that the dimension of the latent distribution and the data distribution are the same or that the true data distribution has bounded support.