A Uniform Generalization Error Bound for Generative Adversarial Networks
Authors: Hao Chen
Zhanfeng Mo
Qingyi Gao
Zhouwang Yang
Xiao Wang
Venue:  (2020)
Keywords: GANs
Uniform Generalization Bound
Deep Learning
Weight normalization

Abstract: 	This paper focuses on the theoretical investigation of unsupervised generalization theory of generative adversarial networks (GANs). We first formulate a more reasonable definition of general error and generalization bounds for GANs. On top of that, we establish a  bound for generalization error with a fixed generator in a general weight normalization context. Then, we obtain a width-independent bound by applying $\ell_{p,q}$ and spectral norm weight normalization. To better understand the unsupervised model, GANs,  we establish the generalization bound, which uniformly holds with respect to the choice of generators. Hence, we can explain how the complexity of discriminators and generators contribute to generalization error. For $\ell_{p,q}$ and spectral weight normalization, we provide explicit guidance on how to design parameters to train robust generators. Our numerical simulations also verify that our generalization bound is reasonable.

Under review as a conference paper at ICLR 2020
A UNIFORM GENERALIZATION BOUND FOR GENERA- TIVE ADVERSARIAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
This paper focuses the theoretical investigation of unsupervised generalization theory of generative adversarial networks (GANs). We first formulate a more rea- sonable definition of general error and generalization bounds for GANs. On top of that, we establish a bound for generalization error with a fixed generator in gen- eral weight normalization context. Then, we obtain a bidth-independen bound by applying lp.g and spectral norm weight normalization. To better understand the unsupervised model, GANs, we establish the generalization bound, which uni- formly holds with respect to the choice of generators. Hence, we can explain how the complexity of discriminators and generators contribute to generalization error. For f. and spectral weight normalization, to design parameters to train robust generators. Our numerical simulations also verify that our generalization bound is reasonable.
provide explicit guidance how
INTRODUCTION
The generative adversarial network (GAN) (Goodfellow al., 2014) is one of most powerful gen- erative models for modeling complex high-dimensional tasks, such as image generation, dialogue generation, and image impainting. Many variants of GANs (Ho & Ermon, 2016; Abadi & Andersen, 2016; Goodfellow et al., 2014: Li et al., 2017: Yu et al., 2018) have also been introduced to reinforce the stability of training processes to obtain more realistic models.
A GAN consists of two neural networks: discriminator and generator. Literally, the generator generates simulated data, while the discriminator tries to discriminate between simulated data and real data. The training process of GANs is tantamount to two-player game between generator and a discriminator. The main goal is to obtain a good generator, which is able to successfully approximate the distribution of real data. We denote the distribution of real data and the generator- induced distribution by Dreat and D respectively. Our goal is to find a generator such that D Dreat = Dg. We revise the goal d(Dreat: Dgy) O, with distribution distance d(., -). The Lensen-Shanme (JS) divergence is implicitly used in Vanillar GANs (Goodfellow et al., 2014), and the 1-Wasserstein distance is employed in WGANs (Arjovsky et al., 2017). Empirical experiments suggest that the Wasserstein distance is a more sensible measure for differentiating probability measures supported in low-dimensional manifolds.
The generalization properties of GANs are less explored in the literature, and some exceptions are these works(Jiang al., 2019; Arora et al., 2017: Bartlett et al., 2017: Zhang al., 2017). Motivated by the supervised learning context, where we say training to be generalized if the gap between the training loss and the test loss is small, we can define the generalization for GANs in a similar way. Concretely, generalization for GANs means that, the population distance between Dreat and D2 is closed to the empirical distance between the empirical distributions of Dreal and Dz. Hence, we define the gap between the population and empirical distance the generalization error. Though our ultimate goal is to minimize the former distance, the latter one what we minimize in practice. Given that our training process provides small distance between empirical distributions, small generalization error indicates that the population distance is also small. In other words, small generalization error guarantees that the generated distribution is close to the real data distribution.
In fact, the training process of GANs is sample-dependent. In other words, the generator depends on the training data sets, which are random samples from Dreat- The training process minimizes

---

Under review as a conference paper at ICLR 2020
d(D d(Dreat, Dg), where Dreat denotes the empirical distribution over samples. The deviation be- tween Peal and D Dreat leads the generalization error, i.c., the gap between d(Dreat. P₂) and d(Dreat: Dg). This motivates us to establish a bound for generalization error, that is, the general- ization bound. A tight generalization bound guarantees that the generalization emror is small. The highlights and main contributions of this article are summarized as follows:
We formulate new definitions for both generalization error and generalization bound, which are more reasonable than the definitions in previous work (Arora et al., 2017; Jiang et al., 2019).
We establish the generalization enror bound in general version, with a fixed generator. By applying bp.9 weight normalization, we obtain a tighter bound.
We establish the generalization error bound, which uniformly holds over any choice of gen- erator. Hence, we can explain how the complexity of the generator class and discriminator class contribute to the generalization error.
Numerical experiments Gaussian Mixture models verify that the theory of generalization error bound is consistent with the numerical performance.
1.1
RELATED WORK
Some previous works provide theoretical investigations of the generalization of GANs. Arora al. (2017) introduces new metric for distributions called F-distance, and defines the generalization for GANs based on this distance. On top of that, the paper shows that generalization does happen with a moderate number of training examples (i.e., when the generator wins, the two distributions must be close in F-distance). However, they analyze the generalization with a fixed generator. Hence, the result is not guaranteed to hold uniformly across all generators. In this paper, we establish generalization error bounds for both scenarios. When a generator is fixed, we provide a tighter bound for generalization error than that in Arora et al. (2017).
Jiang et al. (2019) also established a bound for generalization error using spectral normalization of GANs with a fixed generator. They show the advantages of spectrum control for generalization by constraining discriminator class. By adopting the Rademacher Complexity, Bartlett et al. (2017) yielded bound of order O( D(vdkk/m) where d, k m stand for the largest discriminator width, the discriminator depth, and the training data size, respectively. Jiang et al. (2019) derived bound
In our work, we establish a bound of order Odf /k/m) with 1/p" < 1, d1 & d, which is tighter than those from previous works. Moreover, we provide a more general version of the bound for generalization error with a fixed generator, and the result in Jiang et al. (2019) is special case under the spectral weight normalization. To the best of our knowledge, we are the first to establish a generalization bound for GANs that holds uniformly across all generators.
of order ((Vdkk/m)
PRELIMINARIES
We first introduce the formulation of the generalization bound for GANs. We use Dreat for the real data distribution over dR, and (0, for d-dimensional standard Gaussian distribution. We define the sample set as S 2 {x where {x, are i.i.d samples from Dreat and de- note z (0.1) as Gaussian noise. We denote the -nom of a matrix A as || &
and the spectral norm as ||A||2 We denote the conjugate of p as pe, with 1/2" + 1/p- = 1. For an arbitrary distribution μt, fa denotes the empirical distribution over a random sample of size m from nt.
We let F = (flf: R - 1], denote the function class of discriminators, and g {g g Rlo Rd denote the function class of generators. Every generator g induces a distribution
Dy: by applying to a random sample z ~ Ni, (0, I), then we generate a sample g(z) from D,,, In the context of GANs, both F and & are neural network function classes. Specifically, V ʃ J,
dd f(x) Ty oooTT,
ooro 1 ox, where Y 1 S i < ki +1,TT, & w J J.uu+byi aassa--llla p-Lipschitz active function. Note that, by.; E Rd W,,:
and o-)) is a
f.z E Rd 1x4, where d, is the width of the ith layer of f, and hn is the depth of f. For convenience, introduce
2

---

Under review as a conference paper at ICLR 2020
M - (by,s. W f.a". Similarly, we define the generator class g _ {g g g2 Tg. o ..... o g. c o, 2 E R4). Here, Tg.:(u) - WT 9.1 bg.s I S h2 + 1. Note that, bo. E 14x1, Wa,t 9. R where 4 is the width of the ith layer of f. and l 1 is the depth of g. For neural network functions f F and E G. we parameterize them as 9w. respectively, where
are the weight parameters. We denote W and V as the parameter space of F,g, respectively. We denote the Lipshchitz constant (with respect to the input x E R24 of f as L.
Weight normalization. Weight normalization is efficient regularization method for training robust models. We introduce the Pp. and spectral weight normalizations, and establish the general- ization theory for such weight normalized neural networks.
Assume that, F is a neural network function class, which is parameterized as F {Sw w (M M M,)., E W). We define FF with fp.g weight normalization |M f,,llpp. & CF.r 1, ,,... h 1. In this context, we define the parameter norm as ||w w Plp. & Iktt [M - pall
For spectral weight normalization, we repeat the definition Jiang et al. (2019)'s work. As- sume that, F is a neural network function class without bias terms. It is parameterized as F == [fw|w w = (W, fik+tl..... 5,,), w E W). We define F with spectral weight normaliza- tion as (WWalll & B I,..., k1 + 1. In this context, define the parameter norm as || - wwwl212 E1S [ W', BF,
For F and G. the parameter norm || || induces the metric parameter space (W, || II) and (V. II). respectively. Hence, we define the Lipschitz constants of F and g (with respect || as Ly and Lg. respectively.
GANs. According the training process of GANs, formulate the objective functions
min max E gEō JEF x--Drenf [x([(x)] x~D, [d([())))
where (b.): : 11-1, R is monotone Læ-Lipschitz continuous function. The objective function shows that, the discriminator f should give high values to Dreat and low values x Dg- When Dreat, Dg are the same, f is expected to output O.
The training process for GAN is tantamount to minimizing a specific distance, between Dg and Dreat- To measure the distance between distributions, we consider a general distance. Let μr, be two distributions supported on R:d:;
d=(u, v) sup E [x([(x))] - E [o([(x))]
JEFX-M
The objective function is equivalent to mingeg dy (Dreat, Dg). Without a loss of generality, we take dbxx) A in our work. Therefore, our distribution distance can be revised as d v) sup [x)] - x~- [xx().. Though the distribution distance is similar to Wasserstein Distance fEFx~N
(Arjovsky et al., 2017), our discriminator function class F is not forced to be an 1-Lipschitz function class. According to Arora et al. (2017)'s work, if we constrain the range of f to [0, and utilize the F-distance (Arora et al., 2017), the representation can be reduced to the original GAN and WGAN by specific o(-). Thus, it can unify the IS divergence and the Wasserstein distance. Since our generalization theory also holds in these cases, we omit repetitive discussions of F-distance.
Rademacher Complexity. The complexity capacity of network function class has direct effect on the generalization properties of a network. Since GAN model consists of two network structures, the complexities of J and are the keys further investigation of the generalization properties of GANs. The definition of the Rademacher complexity is given as follows. If we as- sume that FF is a class of real value functions, and e; is the Rademacher Random Variable, then the empirical and expected Rademacher complexities are defined accordingly.
(s) A E,
sup JEV
Σ
exf(z))
RM, A ESSDI (pts(F))
where EI....,6, 6, are independent Rademacher random variables, i.c., Plee = 1) = P(G = -1) = 1/2.
3

---

Under review as a conference paper at ICLR 2020
3 GENERALIZATION ERROR BOUND WITH A FIXED 9
We first introduce the definition of generalization error for GANs. In supervised learning, general- ization error refers to the gap between the training error and the test error. However, in the context of GANs, neither the training error nor the test error is well defined. This is because the discrimi- nator f. which is the counterpart of the loss function, varies throughout the training process. In the following, we provide a reasonable measure for the counterparts of the training eтror and the test error for GANs using the distribution distance d. For GAN model with the discriminator f E and the generator g E G, we define the training error as dz D₂) and the real error as dD Dg), where Dreat is the empirical distribution over the sample set S. The generalization error bound for GANs is defined as the difference between these two errors.
We compare our definition of the generalization error with that in Arora et al. (2017); Jiang et al. (2019); Zhang et al. (2017). Arora et al. (2017) defined the training error as dy (Dreat, Dg), which is related to the empirical distribution of Dg- In other words, noise samples are regarded as a training set for GANs, while D3 is regarded as an unknown distribution. However, such a consideration is not commensurate with the experiment process. Instead of being reinput in every itcration like the training set S, new noise sample set of size is generated in every epoch. Thus, noise sample sets are not equivalent to the training set S, since every noise sample sets is utilized only once.
The definition of the generalization bound in Jiang et al. (2019); Zhang et al. (2017) is slightly different from our intuitionn In these works, 9 is defined as the generator, which is obtained by the training process, and g" is defined the optimal generator for geg (D(Dreat: D,). Then, they define the generalization bound as dDrDreaat: D₂) d (Drrttt Dg-)) However, in practise, exact values for g" and Dreal are not accessible. Thus, the first term fails to represent the training error, and the second term fails to represent the testing error.
It is more reasonable assume that D3 is a known distribution for a fixed g. The process of generating new noise samples in every epoch, is an empirical approximation of Da, rather than a simple "training noise" data collection process. From this perspective, our definition provides a more reasonable theoretical measure of the generalization error for GANs.
Definition 3.1. For a GAN model with the discriminator f E F and the generator g E 9, the Dz)l- We say is a generalization
generalization error is defined as (Dy(Dreatt Dg) - error bound if the following holds:
|d reats Dg) - dy reals D,yl c,, gEG
sup
where only relies on the parameter settings of F and g.
Intuitively, a low generalization error bound guarantees that the generalization error is low. Hence, the discriminator successfully discriminates between real data and unseen data. In these cases, the generator generates distribution close to Dreat- An explicit generalization error bound provides us with guidance to design a GAN for designing a GAN to adequately fit real data. The following theorem provides an upper bound for the generalization error, with fixed g.
Theorem 3.2. For any fixed g E G. with a probability of at least ô over the choice of samples S:
[og(1188
dD D,) - do (Dreat, DØ) < 29m.. (F) +
(1)
m
Though the result of Theorem 3.2 seems similar to the Theorem 3.1 in Zhang et al. (2017), our def- inition of generalization eπтоr is different. Theorem 3.2 shows that, for a fixed E G, the bound for the generalization error mainly depends on the complexity of F. Although the discriminator class should be complex enough to discriminate between Dreat and DDy- grossly complicated discrimi- nator class generates extra generalization errors. Theorem 3.2 also shows the relationship between Rm, Dreat (F) and the generalization error bound. For some specific neural network function classes, such as the class of Ep.g weight normalized neural networks, we can compute the Rademacher com- plexity to obtain explicit upper bound for the generalization error. There some previous works of Rademacher Complexity bounds, including Golowich et al. 22018); Neyshabur et al. (2015); Chen et al. (2019). We adopt Chen al. (2019) to obtain tighter and more general bound.
4

---

Under review as a conference paper at ICLR 2020
Corollary 3.3. Assume |x||g* < 1, S. For any fixed G, with probability of at least 6 over the choice of samples S, with bn weight normalization:
dz D2) - d= dD D2)
(2h1 + log
ki+1 crapa
bl+
C(p)
log(1/6)
22
8ki
c
d
V m
+
m
m
where
Sk41
h1 ki+1
cp + -11
do
$ +1
cF pd.
+
i11
and
1=1
1og()2))
P (1) p E (1.2).
C(p)
min(p" - 1, log
By utilizing the big O notation, Theorem 3.2 provides a bound of order O(d5 v //kim/) where 1/p" & usually holds. Notice that, some previous works (Bartlett et al., 2017; Jiang et al., 2019) provide bounds of order O(ydkk//m and Old k2 /m), where d ) Hence, bound is tighter than previous works.
Remark 3.4. A fair comparison between our bound and the bound in Arora et al. (2017) shows our bound is tighter. We denote Py as the number of parameters of f E F, and Ly as the Lipschitz constant with respect to the parameters of f E F. We assess the two results un- der our definition of generalization bound in Section
According to Arora et al. (2017). if m > 3P, log( I.r P we have a probability of at least 1 exp(-P7) (P) over the choice of S. [dy(Dreat: D.) (D))-dy(D,rrr) (DrraDr Dy)1 & c/2. This holds because the formula (6) in the Theorem B.2 of Arora al. (2017) no longer contributes the generalization bound. We convert our result into a similar fashion by utilizing 1/m & e (3P log LyPr/c) and = exp(- - P7). We obtain
|d> (Dreat, D,) df (Dreat. Do)l
11+1
II cr. F- + + VCl)) + (
2( (2hg + log +
/3Pr log(L>
Since 3h, (e can be constrained to small values by applying weight normalization, we can force the following holds:
2(5k) +1 v (2k) + 1) log 2
b +1
CJ
spd
+ dø vC(p))
1=1
EVPF (g)
21 3og
t1+1
p
d
:1+ cF (s/1)
log(LyPs/e) so Log(PFvmi) and PF are large numbers, which only depend on the structure of ki F. Hence, there exits {C that is small enough to satisfy the inequality above. In these cases, our bound is tighter than Arora et al. (2017)'s bound. In fact, the weight normalization contracts the range of network parameters, so the complexity of F is reduced, thereby leading to a tighter generalization error bound.
Inspired by the probabilistic inequality in corollary 3.3, we formulate a hypothesis testing process to judge whether a generator produces data with the same distribution that of the real data. The appendix contains the theory and experiments related to this novel hypothesis test on a toy dataset.
If we adopt ReLU homogeneous function) ization, the bound for the generalization error can be further reduced to become width-independdnt.
active function and apply the lp.g weight normal-
Corollary 3.5. Under the same settings as Corollary 3.3, 1/p 1/9 q > 1, we adopt the ReLU function as active function. Then, for any fixed g E G. with a probability of at least 8 over the choice of samples S:
ds (Dreat, Sicatt,Dy, Dg) - dr (Dreat Da)
k1+1
(2k) + 4)
k1+1
/C(p)
log(1/8) m
(1
CF
dg
m
E=1
5

---

Under review as a conference paper at ICLR 2020
We can easily extend results to cases with spectral weight normalization (Jiang et al., 2019).
Corollary 3.6. Let d max 9 E g. with probability of at least - 6 over the choice of samples S with spectral weight normalization:
fi=i m
1 Under the same settings as Corollary 3.3, for any fixed
d (Dreals Do)) dx dx((Dral D
24( Br (mg(2 I +1 Br
8 mm
log(1/0)
V
m
Corollary 3.6 shows the advantage of applying spectral weight normalization. We constrain I B such that is small enough to force the first term to be small. we set I BF, 1, the conclusion reduced to Olv O(vdky//m)
Experiments in Miyato et al. (2018) also show that, spectral weight normalization render the discriminator generated data and real data. Hence, we can suffer less from model collapse. Since d depends on the largest width of the network, the bound in spectral normalization not actually width-independent. For this reason, we prefer to utilize the lp.9 weight normalization to obtain a width-independent bound.
powerful for distinguishing between
UNIFORM GENERALIZATION ERROR BOUND
We have established the generalization error upper bound with fixed generator g. that is, g is independent of the training process and the choice of sample set S. However, such a bound is not uniform bound for A g G. For any fixed g. let S(g) denote the set of samples where the
inequality (1) in Theorem 3.2 holds, i.e., S(g) A {8 S i.i.d Dmm Dreatt bound (1) holds with S). We emphasize the dependence of S(g) on because this set varies as g changes. In other words, different S(g) values lead different probability levels, at which the bound holds. Hence, the probability that (1) holds with S E (g) is not guaranteed to be greater than 1 5. An upper bound for the generalization error with a fixed g is tantamount to the generalization error bound for neural networks in supervised learning. To further understand GANs, it is necessary for to establish a uniform bound for generalization error with varying g. The following theorem provides generalization bound for GANs.
Theorem 4.1. With probability of at least 1 - 2/ -xpp(-ee /4) over the choice of samples S:
sup [dy D,)- dy(Dreat, D,)1 < 2 (F) E,
gEG
where X is a 2LL6 net V. and V is the parameter space of G.
Theorem 4.1 depicts the error that is contributed by G. Note that, if G is complicated, then | can be large number. In other words, if we reduce the complexity of G by applying weight normalization, the generalization error bound will consequentially decrease. In fact, weight normalization is an approach for reducing the complexity of function class and provides generalization error bound control. For the fp.9 weight normalization, the next theorem provides an explicit expression of the generalization error bound.
Corollary 4.2. With probability of at least - 8 over the choice of samples S:
sup (d=(Dreat D,) - dz (D (reeat,D))) Dz)l DeG
(2kg + logo Ski
k
1o 11
C(p) m
2P
<2
crapd
d5"
+ cs V
log(bkz m
m
where es satisfies
2P log LLg),
log(1/6) <
11 log(6k, + Pg log(cs
2
m
8ky+1 and C(p) are given in theorem 3.3, and Pg denotes the number of parameters in g E G.

---

Under review as a conference paper at ICLR 2020
Notice that with the fp.s weight normalization, obtain upper bound for L, Lg, in l..g norm to
- cg.-- Hence, for an arbi- version: L II cF +1+ .Lg < Etetl p -1 (,7 1,7
1++1
trary weight normalized GAN, we calculate the generalization bound with Pe. CF and cg.- For every 6, we calculate the right hand side of the constraint and pick a feasible and small 6g- Ac- cording to theorem 4.2, the choice of g contributes to the second term. Since the bounds { for weight normalization can be artificially constrained to a small range, the first two terms can be constrained to a small value. Since Pg is determined by the network structure of g E G and is usually much larger than 22g the third term is dominate. Similarly, we obtain width wudth-inddeendent generalization bound by adopting ReLU functions and applying specific lp. weight normalization, with 1/p 1/q 1
Our result can be extended to cases with spectral weight normalization.
Corollary 4.3, Assume ||x||2 & for E S. With probability of at least 1 - over the choice of
samples S with spectral weight normalization:
k1 log(2 dmmks
sup (Dr D₂) dx (Drraat Dz)l <24 sEG
By ad
II Br.)
m
k1
log +1
cs
Pg log
6ikz
B &
II
Bas).
m
m
j=
where es satisfies
h1+1
k2+1
log(1/6) <(
Pg (og( i=1
II
By,
II Bo.j)
j=
11+1
2Ps
+ Pg log(cs
log (6k2
By II j=l
Bo.))).
m
NUMERICAL EXPERIMENTS.
In this section, we illustrate some numerical experiments verify that our generalization error bound is consistent with numerical studies. We train Wasserstein generative adversarial networks (WGANs) to learn a three-Gaussian Mixture distribution. The structure of the discriminator is three-layer (2 50 FC)-ReLU--EE x 50 FC)- x 50 FeL-ReL x 1 FC) network, where FC denotes a fully connected layer. The generator is three-layer (2 x FC)-ReLU-dd x 50 FC)- ReLU-(50 x 50 FC)-ReLU-(50 x 2 FC) network, where d takes value in (50, 70, 90, 110, 130). After the training process, we calculate the generalization error and generalization error bound. The details of the experimental settings are included in the appendix.
In order to compute the generalization error with an empirical approach, we generate two data sets, Strain and Stest, while the sample size of Stest is much larger than that of Strain We regard the empirical distribution over Stest as an approximation of Dreat- The noise is generated from (0, I), in the training process.
We compare the generalization error and generalization bound of the WGAN model. We adopt an empirical approach to calculate d== D2)) and d=<(Dreat D2)) Since theorem 4.2 provides a computable generalization bound with f2.2 weight normalization, for trained WGAN, we can cal- culate the generalization error and generalization bound. For each d E {50, 70, 90, 110, 130). we repeat the following process 50 times to compute the generalization error and the average gener- alization error. At the same time, we calculate the generalization error bound for each d and the average value. Figure displays two visualizations of the simulated results. The left panel shows a visibly good generator and the right panel shows visibly bad generator.
According to the previous discussions, generalization error defined as (Dreat Dz)
dy (D, Dy)l. To compute dy D,), dy (Dreat, Dg), we approximate the distribution dis- tance (D Dg): dr(Dreat D,) by substituting D Dreat, Dreat with the uniform distribution
7

---

1b Under review as a conference paper at ICLR 2020
02
22
1.5
-10 -4
20
15
do
(a) A visibly good generator.
(b) A visibly bad generator.
Figure 1: The blue point cloud represents Strain, sample from a Gaussian Mixture distribution. The red points are (g(2,) Marrnn while (z, Ti=1
morraan 1,ddd (2.
Average generalization Bound
2.6
130
2.2
32.0
70
0.141
0.144
0.147
0.150
Average generalization Error
Figure 2: Each dot on graph represents average generalization error and the generalization bound of WGAN with d E (50, 70, 90, 110, 1303.
Stest; Strain (denoted Drest; Dirain), respectively. The detailed process is included in the ap- pendix.
According figure 2, there is positive correlation between the generalization error and general- ization bound. As the number of parameters in the generator increases, the generalization bound increases. In other words, the experiment verifies that, our bound provides generalization error con- trol. A low generalization bound guarantees that the generalization error is low, whereas generator with large number of parameters introduces more error.
In fact, by applying fp weight normalization to f.g with small cg.j, we control the general- ization bound so that it remains small value. The number of parameters the generator should not be extremely large. So far, the experiment and our theory have explained how fp.q weight nor- malization and the parameter settings of f, g affect the generalization of GANs. Our generalization bound provides explicit guidance parameters designing to train GANs with small generalization error. Thus, we can obtain robust generators.
CONCLUSION
In this paper, we establish the generalization theory for GANs and provide more reasonable def- inition of generalization eиror. We first establish general bound for generalization error, with fixed generator. To further understand GANs, we establish the generalization error bound, which uniformly holds models verify that, our theory is consistent with the numerical studies. the Appendix, we also formulate novel hypothesis testing procedure to judge whether the generated distribution equals the distrbution of observed data. Notice that, in these high dimension cases, the ordinary statistical approaches do work well. Our hypothesis test is capable of discriminating between good and bad generators. One interesting future research topic is to develop generalization error bounds for autoencoder GANs with an additional encoder network.
any choice of generators. Our numerical experiments Gaussian Mixture
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi and David G. Andersen. Learning to protect communications with adversarial neu- ral cryptography. CoRR, abs/1610.06918, 2016. URL http://arxiv. arggabbs/61CC: 06918.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv: 2017.
Wasserstein gan.
arXiv preprint
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang, Generalization and equilibrium in generative adversarial nets (gans). In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 224-232. JMLR. org. 2017.
Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normaiizee margin bounds for neural networks. CoRR, (bs/17066,8498, 2017. URL. 08498.
Quentin Berthet. Topics statistical theory.
Hao Chen, Zhanfeng Mo, Zhouwang Yang, and Xiao Wang. Theoretical investigation of gen- eralization bound for residual networks. In Proceedings the Twenty-Eighih International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 2081-2087. International Joint Con- ferences on Artificial Intelligence Organization. 7 2019. doi: 10.24963/jica.21199255 URL
https: doi. wtt.:o/..//..........
24963/ hwtp//ijaa..../ ttt.:aaii............1 2019/288.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size- lize-independent sample complexity of neu- ral networks. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet (eds.), Proceedings of the s1 Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Re- search, pp- 297-299. PMLR, 06-09 Jul 2018. URL http: / http:://rroceeeeee ngs mlr. .press// v75//olcwichh hww.lwiichhla.. html..
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS' 14, pp. 2672-2680, Cambridge, MA, USA, 2014. MIT Press. URL http:/ http:/dl... ht/./llaa acm org/ citation cfmmid==2999000 2969125.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. CoRR, abs/1606.03476, 2016. URL http: /ttp:://///vvvvvee wrgg 60660666 63
Haoming Jiang, Zhehui Chen, Minshuo Chen, Feng Liu, Dingding Wang. and Tuo Zhao On compu- tation and generalization of generative adversarial networks under spectrum control. In Interna- tional Conference on Learning Representations, 2019. URL httptst:///peereeiieee net./
forum?i (erummiddrJNH6sAA 6sAqY7
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint arXiv:/412.6900 1412.6980 2014.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2157-2169, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. (0.1865//1/DD1--1330 URL https: //www/ /wwwwww.. ac web. org/
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. CoRR, abs/1802.05957,, 2018. URL UtLphttp://axxii abs/1802. 05957.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401, 2015.

---

Under review as a conference paper at ICLR 2020
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. Generative image inpainting with contextual attention. CoRR, abs/1801.07892, 2018. URL http:://axxxi atg/abs/1800 07892.
Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu, and Xiaodong He. On the discrimination- generalization tradeoff in gans. CoRR, abs/1711..0271/ 2017. URL htttp://axxiv.oo
abs/1711.02771.
A
PROOF DETAILS FOR MAIN RESULTS
PROOF OF THEOREM 3.2
Proof.
sup SEF
E f(x) - E fog(t)]
-sup
E f(x) E f [()]]
2-N
fEF
znD
2~N
sup
[(x)
E J [og[z)]
[[ IND
E
f(x) I [o
fET
avD
2~N
2
sup
E r>D
f(x)
E
f(x)
feF
RND
Using the well-known estimation method for the generalization bound in supervised learning Mohri al. (2018), with a probability of at least 5 over the choice of samples, we have
log(1/o)
sup fEF
f(x) - E f(x) RND
< 29 Druat (F) +
aND
m
PROOF OF COROLLARY 3.3
We adopt Chen al. (2019) obtain tighter and more general bound of the Rademacher Com- plexity:
Rm (F)
k +1
(2h1 + 4) log
1
#1+ dp
C(p)
Esk
cp
Where
k +1 kn
by
cripd -
+ll -
is Hh cr.pp,
8k +1
[=1
2 2log (2dg)
p= 1
C(p)
mmnnp" 1, log (2do))
PROOF OF THEOREM 4.1
Lemma A.1. V|[X||p- & 1,f F, we have f(x) & CF D
d3
CF.
Where p is the Lipsehitz constant of the active function.
Proof. Define series of variables (Zo, Zı, ..., Zzn }
Zo ||x||/-
and
Z41 (1f41(()))xl) = 0. 1....,k.,
10

---

Under review as a conference paper at ICLR 2020
where fi is the combination of the first layers. Then, prove by induction that for / 1,..., k1,
+ cF.
Zi x
1=1
When 00, 0, we have Zo = ||x|1p < 1. For 1 = 1,2, k1
Z, f 10-
(W f,((x) + by.))lp-
< pd + +++ (I [W + lIbrallo-)
- + (cr.b(-1+c5))
<pd,
+ シ cF.j
j=1
Finally, we have
(x)
W Fiyt Say (x)llr-
f.bytille Ilp Ilfx, (x)lp=
h1
+
CCF
CF.
j=1
Lemma A.2. For generator class G. which is parameterized as the previous discussions, there is a L/Lg-net X for parameter space V, with respect to the p. parameter norm, such that
log |X| < Pg log 1og
+
where Lg = cg Ete p tl
cg.s.
Proof. For two generators Iv Iv' E g, (M' k.ttts...... M
(M. M,ghH.,..,,,M..... Mg1), v
M' 9.1)
V. we have
1 (x)(x) gv (x)l
U/M,,oatin M..11 - 9IM's 9. 9. 1. M.. +
19(M''..2 I MY I...,
2,.1) - 9(M M.M.,ng. 1 B.....,,,,.... 2.1
(AMg 1pp.
+ Lg ( g. l..
LG
CG.1
kx+1
AWT g. T1pp
Lg
cg.i
Notice that, we derive the second inequality by applying lemma A.1 on g.
Then, we focus on the c of set (x E RR4 Ip. & (g.)). According to lemma 1.3 of K2Log
cp. -net X; that satisfyies log |X| < (1-((++1) log(3k2 Lg/e. We construct Berthet, there exists a TzLLo
negeet X' of V, by taking X (v = (M, Mg.k. kv +1, Hence, we have log | < Pg Lgg
M2,)) E V Mg E X, = 1,... k2 1}.
Lemma A.3. For any generator g E g
[d Dy) E.s |dy (D Dz)l & 29t Drem

---

Under review as a conference paper at ICLR 2020
Proof.
(Dreal Dg) Es|dy (Dr p₂)l
sup
E xr-Dreal
[(x) I g(2)] - E. sup E
[(x) E I g(z)]
fEF
2
fer x--Drenf
<Bs
sup
E X X--Dree!
f(x)
f [(()]]
sup
E f(x)
fo oz(z))
SEF
x-N
SEF XmeDraa
am
< Es sup
E XmDrrel
f(x)
E
ʃ g(2)] - E
f(x)
E I [ and
FEF
z-N
Xr-Dreal
22 (F),
where we denote the noise distribution Ni. (0,1) I) as N for short.
Lemma A... With F,9 fixed, the following holds with a probability of at least 28 over the choice of samples S:
sup gE
sup JET xrDrraa
[(x)
mix
So
sup feJ
[(x)
So
x--Dreal
Cá / 2P's log(3kz Vmn
where sutisfies
log (1g(1/6) <(
1) log(3k2 Pg log(es/ log (3gg33
Proof. Let Si and Si be two sample sets from real data, with #S = #S = m. They differ by exactly one element, which is denoted as E Si and x St, respectively. For fixed g, we have
sup fEF
f(x)
E 2~N
fo [og(z)]
sup feF xnDi
f(x) E f 9())]
x
2~N
< sup /EF
E f(x)
E
fo o [g(2)]
-
E
f(x) E fo g(z)]
xvD
zmN
x~Di
zn
sup
f(x)
[(x)]
/EF
x
x-Di
1 (f(x.)) f(x))
2
sup
JEF
where D, D' stands for the uniform distribution over S₁, St. According to McDiarmid's inequality, it holds that:
Ps
Es sup fer xnD
f(x)
E. 2"N"
So g(())]
ssup E.
f(x)
I > 2
[og[z)]
feF
xnD
z-N
2//2
<2exp(- 2/m
where D stands for the uniform distribution over S. By following Lemma A.2, we let X be zíz. net of V that satisfies log | & Pg Log//k)LLg/ee). Then, by union bound over all g, whose parameter belongs to X. we have:
Ps
sup xE
Es sup JEF
E f(x)
E fo - suu E f(x) - E f (..oo(z))]
E >
xnD
JEF xnD
2
2/2
2 eeeppPp (Pg log (kggLg/)) exp(-
2/m
22/2PG (kg Vm
Take
then the proof is finished.
12

---

Under review as a conference paper at ICLR 2020
Proof of theorem 4.1.
sup
(Dral) Dg) dy pg) 9Eg
dy Dg) Esldy (Dreat Pg)l| | d (Dreat, D,)l [dx Dz)
We complete our ultimate proof by combining lemma A.3 and A.4.
B DETAILED EXPERIMENTAL SETTINGS
B.I DATA SET.
The training data set Stroin is point cloud in IR? We generate data sample S accordingly:
1. Set the size of three Gaussian mixtures as {m, | (m1, m2, ma) ~ PN 3: 1).
2. Generate three Gaussian mixtures, respectively. S Si U U S3, where Vi, S; denotes
{{x 1-=1//// { 1 fidd N₂ (p, E,), Vý, | 10}. We constrain the training data with
vj. |X, 12 10 to simulate the assumption of S without rejecting much generated data.
According to the previous discussions, the generalization error [d=(Dreat: D₂) dr rDeat D₂)l depends on Dreat, which is an unknown distribution. In order to compute the generalization error by an empirical approach, we generate two data sets: Strain Stest, with #S < Stest In other words, regard the uniform distribution over Stest concretely, we generate Strain with Mergin
approximation of Dreat- More 2 x 10 and Stest with Mestt 2 x 10a. We set f (-0.5,-1), 14g (0.2, 0), 14g (-1, 1), A E; 0.05 I2. We generate noise as z ~ (0, I) the training process.
B.2 NETWORK STRUCTURE AND PARAMETERS SETTING.
The structure of the discriminator is three-layer(2 50 FC)-ReLUU-(00 50 FC)-ReLLU-(500 50 FC)-ReLU-(50 x 1 FC) network, where FC denotes fully connected layer. The generator is threc-layer(2 x d FC)-ReLU-(d 50 FC)-ReLU-(50 50 FC)-ReLU-(50 2 FC) network, where d takes value in (50, 70, 90, 110, 130). For the generator, apply the Xavier initialization with gain 1.3 to the weights, and we set the initial bias 0.1. For the discriminator, we fill the weights with random numbers from N(0, 0.02²) and bias with 0.
We use the optimizer of Adam Algorithm Kingma & (2014) with 5 (0, 0.9) for both the discriminator and generator. The leaning rate of both the discriminator and generator are set to 8e-4. To apply 2,2 weight normalization to the discriminator, 1)a, to the loss function of WGANs. We train 150 epochs with batch size of 1000 to train generator. Since there is no sufficient stopping criterion we stop the training when the generated point cloud visually resembles Strain without further variance.
add 622 penalty, (II: ||,
05
-35
11
-.5
-33
23
(a) A visually good generator.
(b) visually bad generator.
Figure 3: The blue points cloud represents gtrain sample from Gaussian Mixture distribution. The red points are {g(zz)))"murnn Marain while { mirrain i.5d (2,
The process for computing dy Dg), (D real, D,) is executed as follows:
13

---

Under review as a conference paper at ICLR 2020
1. Train WGAN over Strain and save the parameters of the optimal g.
2. Train discriminator f to maximize Exwiirai [f(x) - Ex- [f]]] to obtain ftrain.
3. Train discriminator f to maximize Ex [x()] 1x [[(x) to obtain ftast-
4. We calculate dy d= P.) by following section 3.1.1. We obtain ds Dg) by substituting 'train' with 'test".
C A NOVEL HYPOTHESIS TEST FOR Ho Dreal = DDg
For given generator g E G. we are insterested in whether this generator generates real data. To answer this question, we formulate a hypothesis test.
Since our goal is to provide bypothesis-testing process to figure out whether D, = Dreat holds, confidential level 1 a, state the null hypothesis Ho Dg Dreat and the alternative hypothesis as H1 Dg + Dreat- With the distribution distance, we revise the null hypothesis as: Hn dD real, Dg) O. Intuitively, in the context of image-generating GANs, visually bad generator should give evidence to reject the null hypothesis and embrace the alternative hypothesis.
Assume that Ho holds, we have dy (Dreatt D₂) & Id=(D Dg) - dd Dg)l. According to corollary 3.5, for given g G, with probability least 1 a over the choice of S, it holds that: d, (D real, D₂)= & T, where denotes the right hand side of corollary 3.5. Hence, d)=(Dreal: Pg) > I' provides significant evidence to reject Ho at - a confidential level. In other words, for a given E G, if d (Dreat. Dg) > T holds. This means that the artificial data is discriminable and g fails to adequately the real data. Hence, to make a hypothesis testing conclusion, we compare the values of d (D D2) and T.
C.0.1 HYPOTHESIS TESTING PROCESS.
For a given y E G, we train a discriminator / to maximize E. xx [[()) Ex [x()] Notice that, training such a discriminator is equivalent training a WGAN with g fixed. Hence, we obtain ftrain.
2. For i (1,...,55) 50), have (DrDratt: Da); BxwDreal [ Ex (Jtrain (x))- In each iterations, we generate an m-size noise sample set from Ni, (0, I), to feed the
second term. Then, let df Dg) = Σ. d= dD((Dratt D2),/50.
3. We set C.. [[ foreen..illls. For given confidential level 1 a, by plugging ( into the right hand side of corollary 3.5, we obtain 7.
4. If dy (Dreat, Dg) > T, we reject H, at 1 a level.
The hypothesis testing for H, an application of corollary 3.5. Instead providing a statistic with an explicit distribution, our hypothesis testing is based probabilistic bound for generalization error with a fixed generator.
Notice that, there tistical way, such as the Kolmogorov-Smirae test, Shapiro-Win Wilk test, Mann- WhinnWy U Test and Bootstrap Methods. However, most of these methods are heavily based on some strong assump- tions (i.e., normality, the conditions for central limit theorem and low dimension). Hence, these methods do not work with some unusual and high dimensional distributions. cases, corollary 3.5 guarantees that, our novel hypothesis testing has the capacity to discriminate be- tween high dimensional distributions. Our numerical experiments show that, this hypothesis testing is commensurate with our observations.
distribution-free methods compare distributions in purely sta-
high dimensional
D MORE EXPERIMENTAL RESULTS FOR OUR HYPOTHESIS TEST
The following experiment illustrates the hypothesis testing process in section 3.1.1. For a specific generator, convey the hypothesis process on 100 sample sets 1100 where VS is derived from
14

---

Under review as a conference paper at ICLR 2020
the same distribution of Strain If generator is visually bad, the hypothesis process is expected to reject Hy almost 100 times, at a 0.95 confidential level. In these cases, it is highly probable that the generated distribution is not close to the real distribution. Notice that, even if the hypothesis process embraces Ho. the generator is not guaranteed to fit the real distribution well. Instead, we should say, the sample data is not rich enough provide significant evidence reject Ho high confidential level. In fact, for a visually good generator, the hypothesis process fails to reject Ho. even with rich sample data
D.0.1 HYPOTHESIS TESTING FOR Ho Dreat Dg-
L.875-
230)
Ioo
20200
0 100
0.2925
130
25
80
75
100
25
103
Experment
Expertment
(a) For the previous visually good generator.
(b) For the previous visually bad generator.
Figure 4: A representative result of the hypothesis test with the generators in figure In every experiment,
reject Ho if the training loss is larger than
We show a representative result in figure 4. Recall that, the hypothesis testing process reject Ho if dy (Dreat Dg) T, where refer to dy (D Dg) as the training loss and T as the test statistic for Ho- Figure 4 suggests that, the hypothesis testing process embraces Ho with a visually good generator, while reject Ho with visually bad one, 0.95 confidential level. In fact, our experiments show that the conclusions from the hypothesis testing are commensurate with our observation. In our toy model, Ho is rejected when the generated data forms a single clique (what we call Model Collapse) or it obviously disjoints with the real data. Hence, our hypothesis test is capable of discriminating between good and bad generators, with sufficient sample data.
In our experiments, an extremely small and negative value of gap dD Dg) - î implics that the generator fits the real data very well and vice versa. For the previous bad generator in figure 3, the gap is positive but small. This implies that, though the generator fails to fit the real data, it does not violate the real data grossly.
We display some representative results for hypothesis test in section 4. The experimental settings the same in previous discussions.
D.1 DISCRIMINATION
To verify that our hypothesis test shows discrimination in choice of good generators, we conduct hypothesis test on 79 generators with the same training set Strain and d = 50.
In figure 6 and 7, juxtapose the hypothesis test conclusion and the corresponding generator.
Recall that, for a given g. we reject Ho Dreat D, (D D,) > Í. Numerical experiments show that, our hypothesis is capable to detect the visually bad generators, at a 0.95 confidential level.
Our hypothesis test is based on a probabilistic inequality, which depends on the artificially designed sample set Strain and confidential level 1 Thus, for visually bad generators, there may be some rare acceptances of Hp. these cases, instead of stating 'the hypothesis test provided an incorrect conclusion'. we should say 'the sample set Strain is not rich enough provide evidence to reject Ho. at a 1 a confidential level'. Such an example is included in 7.
D.2 DISTRIBUTION DEVIATION SENSITIVITY
The numerical experiments also imply that, the hypothesis test can detect how D, violates Dreat- We provide example in figure 5.
15

---

Under review as a conference paper at ICLR 2020
According to figure 5, the hypothesis test provides stable rejection of Hy with S i i 1, ..... 100. Moreover, there is an evident gap between the training loss and T. Recall that the gap between the training loss and T of the bad generator in figure 3 is small and unstable. Obviously, the generator in figure 5 is much worse than the previous ones. The result suggests that, our hypothesis test is sensitive the deviation between Dg: Dreat-
Concretely, for generators that fit the data very well (or very badly), the hypothesis test provides a training loss which is much larger (or lower) than T; for generators that perform modestly, the hypothesis test provides training loss that is close to T to show its hesitation. In latter cases, to obtain an unambiguous conclusion, we can choose richer sample set or set a larger confidential level.
Train
0.0
-0.5
0.50
-1.0
-15
-1.0
-0.5
0.0
1.5
(a) An extremely bad generator, which is even (b) We convey the hypothesis test with S., i worse than figure 3. Model collapse also occurs. I.... 100. All the experiments reject Ho at a 0.95 con- fidential level.
Figure 5: In such extreme case, the values of training loss and T are stable.
16

---

Under review as a conference paper at ICLR 2020
1.75
1.50
1.25
Train loss
1.00
0.75
0.50
0.25
0.00
12
38
Generator order
(a) The hypothesis test result. For fixed 9, reject Ho if the training loss is larger than T. According the figure, reject Ho with the 1, 10, 12, 19, 25, 39 generators.
5m
7t
58
18 1n
12
55h
(b) generator examples in order. Notice that generators 1, 12, 19, 25 and 39 visually bad (or occur model collapse), which is commensurate with our hypothesis test conclusion.
Figure 6: every figure, blue points cloud represents Strain sample Gaussian Mixture distribu- tion. The red points are (g(z,) M while (z, ) 1.3. No I).
17

---

Under review as a conference paper at ICLR 2020
Train loss
Value
0.25
10
32
Generator order
(a) The hypothesis test result. For fixed 9, reject Ho if the training loss is larger than T. According to the figure, reject Ho with the 26 th generators.
10th
17
18
98
20
21
28
28
28
28
(b) 40 generator examples in order. Notice that generator 9, 24 and 26 are visually bad (or even occur model collapse).
Figure 7: Though our hypothesis test fails to detect the 24 th generator, shows that train loss is extremely closed to for the 24 th generator. This unusual acceptance is caused by randomness. In this case, the sample set Strain is not rich enough reject Ho, at 0.95 confidential level.
18


Reviews:
[{"review_id": "ryle5_OpKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors study the generalization bound for GANs based on a new definition of generalization error where the distribution corresponding to the generator is assumed to be known for each generator (i.e., there is no empirical distribution for generators). For this generalization error, the authors give both bounds for a fixed generator and a uniform bound for a class of generators.\n\nIn my opinion, most of the theoretical results seem follow directly from standard tools in statistical learning theory and existing results on capacity bounds of neural networks. It seems that the authors do not introduce new ideas or techniques in the analysis.\n\nThe authors made comparisons with the related results in Arora et al (2017). However, since the generalization bound in Arora et al (2017) is based on a different definition of generalization error where empirical distributions are considered for both discriminators and generators, the comparison seems not fair.\n\n----------------------\nAfter rebuttal:\n\nI have read the authors' response. The contribution seems not novel and enough. I would like to keep my original score.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ryevDNCidS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe authors give new generalization bounds for GANs.  The argue for a new definition of generalization for GANs, which isolates the effect of sampling error arising from sampling from\nthe real distribution.  (Their argument, which I find convincing, may be paraphrased as saying that sampling from the generator should be viewed more as a computational cost than a data-gathering cost, since a procedure may sample from a generator as many times as it wants.)  They give a bound that is uniform both over discriminators and generators.\n\nIn my opinion, the mathematical writing is not up to the standard for publication in ICLR.  There are many cases where the paper is unclear, and, in many other cases, I have to guess what they mean, and I have limited confidence in my guess.  Since the theorems of the paper are quite technical, it is very difficult to be confident what their exact statements are.  One example is where they write \n\"We define F with weight normalization as ...\".  The fact that the c in this definition has a subscript of f led me to think that the bound can depend on f, but this does not make sense.  I assume that the bound is independent of f.   Later, when they write \"we define the Lipschitz constant of f\", since they write that this Lipschitz constant is with respect to a norm on parameterizations, I take it that they are defining the Lipschitz constant of a mapping from parameters to functions (and not a Lipschitz constant of f).  But they don't indicate what metric is used to define the distance between functions.  \n\nWhen the compare their bound with previous work, they treat quantities as constants which can be large.  For example, they treat the product of the operator norms of the layers as a constant.  It also is not clear how they get the bound that they attribute to Bartlett, et al from the bound in that paper.  \n\nMost of the technical heavy lifting appears to have been borrowed from the Chen, et al paper.\n\nA more detailed account of how they get their covering bound for their (p,q) norm from the\nDumer paper is needed.  \n\nSince the authors assume that phi is the identity, it seems unnecessary to keep subscript d with it.  On the other hand, subscripting d with calF would make the theorems easier to interpret at a glance.\n\n(Edit on 11/14/19: I have read the response and checked the revision.  Some of the above criticisms have been addressed by the revision.  I have increased my rating.)", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lWfzcP9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proves generalization bounds for GANs. I think the paper can be improved significantly in several ways:\n\n1- Writing: The first two sections are relatively well-written. The problem starts at section 3 and continues after that. Some of the things that can be improved:\n\na) The discussion on the different definitions of generalizations is not really helpful in the current format. You might want to explain how these different definitions relate to each other. For example, if generalization in one of them implies generalization in the other one, etc.\n\nb)Theorem 2.3 is a general statement but it is followed by Corollary 3.3 which is a very specific generalization bound. There is no explanation how one can show the corollary.  Even worse is mixing these two in the proof of the theorem in the appendix.   Please consider improving the use of Theorems, Lemmas and Corollaries.\n\nc) Section 3 and 4 have bunch of theorems and collieries without much explanation. It is not clear that all of these are actually helpful for the main purpose of the paper.\n\nd) I don't completely understand the notation in Corollary 3.3. Eg. what is d_{f,\\ell}?\n\n2) Related Work: I think authors need to do a more comprehensive literature review on generalization bounds. Since the generalization bounds presented here are built on the supervised learning bounds, authors discuss the generalization bounds in supervised learning. For example, authors heavily rely on Chen et al. (2019) for their generalization bounds while very similar results where shown before by [1] and [2].\n[1] Neyshabur, Behnam, Ryota Tomioka, and Nathan Srebro. \"Norm-based capacity control in neural networks.\" Conference on Learning Theory. 2015.\n[2] Golowich, Noah, Alexander Rakhlin, and Ohad Shamir. \"Size-independent sample complexity of neural networks.\" Conference on Learning Theory. 2018.\n\n3) Definition of generalization: I don't think the definition of generalization suggested in this work is much different than Arora et. al. since f really doesn't depend on samples from D_g and hence the empirical and true distributions are not very different. In fact, I think the definition provided by Arora et. al. 2017 is preferred because at the end of the day, we have to estimate the distribution D_g by generating some samples.\n\n4) Generalization bound for fixed g: Unfortunately, the novelty of these generalization bounds are very limited as they are a direct application of known generalization bounds in the supervised settings. Therefore, the authors contributions are very limited here.\n\n5) Generalization bounds for all generators: Again, here the novelty and final result is very limited since the bounds achieved by a union bound arguments and does not really go beyond that.\n\n6) Experiments: Experiments can also be improved significantly. Currently, the correlation is reported for 5 trained networks and it is not clear to me that this result is statistically significant. Moreover, only one hyper-parameter is changed in the experiments which could be problematic. I suggest authors to change multiple hyper-parameters and train more networks to improve the evaluation.\n\n\n******************************\n\nAfter author rebuttals:\n\nI have read the authors response and looked at the revision. Unfortunately, many of my concerns are not addressed adequately so my score remains the same.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors received reviews from true experts and these experts felt the paper was not up to the standards of ICLR. 

Reviewer 3 and Reviewer 1 disagree as to whether the new notion of generalization error is appropriate. I think both cases can be defended. I think the authors should aim to sharpen their argument in this regard.Several reviewers at one point remark that the results follow from standard techniques: shouldn't this be the case? I believe the actual criticism being made is that the value of these new results do not go above and beyond existing ones. There is also the matter of what value should be attributed to technical developments on their own. On this matter, the reviewers seem to agree that the derivations lean heavily on prior work. 
Decision: Reject