Statistical Error Bounds for GANs with Nonlinear Objective Functionals
Authors: Jeremiah Birrell
Venue: Accepted by TMLR ()

Abstract: Generative adversarial networks (GANs) are unsupervised learning methods for training a generator distribution to produce samples that approximate those drawn from a target distribution. Many such methods can be formulated as minimization of a metric or divergence between probability distributions. Recent works have derived statistical error bounds for GANs that are based on integral probability metrics (IPMs), e.g., WGAN which is based on the 1-Wasserstein metric. In general, IPMs are defined by optimizing a linear functional (difference of expectations) over a space of discriminators.  A much larger class of GANs, which we here call $(f,\Gamma)$-GANs, can be constructed using $f$-divergences (e.g., Jensen-Shannon, KL, or $\alpha$-divergences) together with a regularizing discriminator space $\Gamma$ (e.g., $1$-Lipschitz functions). These GANs have nonlinear objective functions, depending on the choice of $f$, and have  been shown to exhibit improved performance in a number of applications. In this work we derive statistical error bounds for $(f,\Gamma)$-GANs for general classes of $f$ and $\Gamma$ in the form of finite-sample concentration inequalities.  These results prove the   statistical consistency of $(f,\Gamma)$-GANs and reduce to the known results for IPM-GANs in the appropriate limit.  Our results use novel Rademacher complexity bounds which  provide new insight into the performance of IPM-GANs for distributions with unbounded support and have application to  statistical learning tasks beyond GANs.

Published in Transactions on Machine Learning Research (05/2025)
Statistical Error Bounds for GANs with Nonlinear Objective Functionals
Jeremiah Birrell
jbireel@@tastate.edl
Deportment of Mathematics Teaas State University San Marcos, TX 78666, USA
Reviewed on OpenReview: https: openreview. net/ ftrmmm id= 2g.hykPSaU
Abstract
Generative adversarial networks (GANs) are unsupervised learning methods for training a generator distribution to produce samples that approximate those drawn from a target distribution. Many such methods can be formulated as minimization of a motric or divergence between probability distributions. Recent works have derived statistical error bounds for GANs that are based integral probability metrics (IPMs), e.g., WGAN which is based on the 1-Wasserstein metric. In general, IPMs are defined by optimizing linear functional (difference of expectations) over a space of discriminators. A much larger class of GANs, which we here call T)NNN,,, can be constructed using f-divergences (e.g., Jensen-Shannon, KL, or a-divergences) together with regularizing discriminator space I' (e.g., 1-Lipschitz functions). These GANs have nonlinear objective functions, depending on the choice of f, and have been shown to exhibit improved performance in number of applications. In this work derive statistical error bounds for T)-ANs for general classes of and T in the form of finite-sample concentration inequalities. These results prove the statistical consistency of -f.T)- and reduce to the known results for IPM-GANs in the appropriate limit. Our results use novel Rademacher complexity bounds which provide new insight into the performance of IPM-GANs for distributions with unbounded support and have application lo statistical learning tasks beyond GANs.
1 Introduction
Generative adversarial networks (GANs) are unsupervised learning methods for training a generator distri- bution to approximate a target distribution by using samples from the target in a minmax game between
generator and a discriminator network (Goodfellow al. 2014 Dziugaite et al. 2015 Li al.) 2015
Arjovsky et al. 2017p. Mathematically, many such methods can be formulated in terms of minimizing integral probability metric (IPM):
inf dr (Q, IE0
(1)
dr(Q, = sup Eq - Ep, h]).
(2)
hel
Here dr is the IPM with test function space I' (i..,, discriminators), Q is the distribution to be learned (i.e., the distribution of the data) and Po is the generator distribution, depending on parameters A @. More specifically, Po = (1e) + Pz is the pushforward of a noise source Pz under a generator network 4g. For instance, Wasserstein GAN [Arjovsky et al. 2017) corresponds to P being the set of 1-Lipschitz functions. A key distinguishing feature the IPM-GANs the linearity in of the objective functional in (2).
A larger class of GAN methods can be formulated by generalizing the minmax game to use a nonlinear objective functional; such methods generally be viewed minimizing divergence (a generalized notion

---

Published in Transactions on Machine Learning Research (05/2025)
of "distance" discrepancy) between Q and Pg. Such methods include the original GAN (Goodfellow 2014) which was based on the Jensen-Shannon (JS) divergence, the more general d divergeenes defined in Arora et al. (2017), which use nonlinear objective functionals that generalize the original JS- GAN objective by replacing the log with another concave function Φ, the nonlinear objectives f(x: y) introduced in Liu et al. (2017), the convex duality framework of Farnia & The (2018), and the restricted f-divergences of Liu & Chaudhuri (2018). In this paper we will focus on GANs expressed in terms of (f. diivergggnss these divergences generalize and interpolate between IPMs and f-divergences (e.g., Kl-divergence) and the corresponding GANs have nonlinear objectives; see [irrell et al. (2022b a) for the general theory of these objects which, in contrast to number of other approaches, applies to probability distributions having unbounded support. The corresponding (f, --ANNs encompass and generalize large number of successful methods in the literature (Goodfellow et al.) 2014 Nowozin el al. 2016; Belghazi el al.
et al.
2018 Miyalo el. al.,
2018: Arjovsky et al.) 2017 Guirajani et al. 2017 Song & Ermon 2020a Nguyen et al.
2010 Gretton (202bb).
ct al. 2012: Glasor et al., 2021 Dupuis, Paul & Mao, Yixiang 2022) see Table in Birrell et al.
Moreover, we further motivated to study the statistical properties of (J,T)S due to the observation that the nonlinearity of the objective functional confers several advantages over Wasserstein GAN (WGAN), widely-used IPM GAN. Specifically, in Birrell et al. (2022b), (f.1')-AAs were shown to be effective on heavy-tailed data for which WGAN fails. (f.N)-GANs were also shown in experiments to be significantly less sensitive to hyperparameter tuning, with (ʃ, T)-GAN training being successful under hyperparameter perturbations which caused WGAN training to fail. Those experiments focused on f's coming from the lamily a-divergence; see Table below.
The (J,T)- are constructed from the ((vT)-divereence, which are defined follows. Given convex function f with f(1) and test function space r, the (f, ddiveeeieeee between probability distributions Q and P is defined by
(F(()PP) sup [Fo Af[M] ker
(3)
where, denoting the Legendre transform of f by f", we define
Aflh] == inf (v + Eplf" v)]} VER
(4)
We refer to Af as the generalized cumulant generating function because in the KL-divergence case (xK =
= =log(=)) it is straightforward to show Fxx [b] = log Ep(e9], which is the classical cumulant generating
function. The relation of AP to f-divergences was proviously studied in Broniatowski & Keziou (2006);
Ben-Tal & Teboulle [2007]: Nguyen et al. (2010); Ruderman et al. (012)). Under appropriate assumptions on f and r, D) P) provides meaningful notion of discrepancy between Q and P due to it satisfying
the divergence property, i.e,, (QII)) 2 with equality if and only if Q = P; see Theorem 2.8 in Birrell
let al. (2022b). This property, along with the variational characterization (3) motivates the definition of
(f, U)ANNs:
inf (F())))) 8EB
= inf sup { [Eol - Af1h)
(5)
aet ker
We note that 3 can alternatively be written as
D)(Q1IP)= = sup heT
(h))
(6)
with the shifts I absorbed into the definition of the discriminator space i= (h (h-v: Eeve R} (i.e. v is the bias parameter of the final layer). The form ② is preferable for implementation purposes, however for the purposes of our analysis we will focus on the form (3) in order to emphasize the special role that the parameter v plays. More specifically, our methods for proving statistical consistency will rely on the assumption that I' consists of uniformly bounded functions, while the priori unbounded shift parameter v will require special attention. Such considerations are not relevant for IPM GANs since bias parameter in the final layer will exactly cancel due to the linearity of the objective functions, and hence can be neglected.
2

---

Published in Transactions on Machine Learning Research (05/2025)
The statistical consistency theory of IPM-GANs has recently been studied in number of works (Liang 2021
Biau et al. 2021: Huang et al. 2022 Chen et al. 2023b Chakraborty & Bartlett 2024). These derivations take advantage of the special structure of IPMs, namely the linearity of the objective functional. As previously noted, a number of works have introduced GANs with nonlinear objective, however the statistical consistency theory for such GANs largely lacking. A notable exception is Arora et al. (2017) which studied the original JS-GAN and their generalization to dr..-diiergences: we will provide a detailed comparison of this earlier work with ours in Section 1.2 below. In this paper we develop theory that allows to prove statistical consistency of the (f.T)-GANs, in the form of finite-sample concentration inequalities. The key technical hurdle is the nonlinearity of the objective functional due to the presence of the generalized cumulant. generating function E Section 2 is dedicated to properties of and bounds on Aÿ which will be needed in Section 3 to derive concentration inqqualities for ()DNs. We note that our theory does not make any compactness assumptions on the support of the data distribution or generator noise source; our method for handling distributions with unbounded support in Section 3.3 provides new insight even in the IPM-GAN case.
1.1 Summary of Results
Key to our results decomposition of the (f.T)-GAN error into optimization error, approximation error, and various statistical errors. Given an approximate discriminator space I' C I' (e.g., a neural network with spectral normalization as an approximation to the space of 1-Lipschitz functions), and empirical measures Qn and Pa.m = (#o) constructed using n ii.d. samples from the data distribution, Q, and i.id. samples from the generator noise source, Pz, respectively, let lx mm denote solution (with optimization nnm
error tolerance copt 2 0) to the empirical (f,N)-GAN problem:
DT,QQulIPo m)) S inf DF (Po...) + Eopt
n.m
(7)
dee
Given this, we derive the following (f.T)-AAN error bound in Lemma 3.4
PF(Q)Pp inf (Q11)))
(8)
AEG
{APam
Pa. [ho Tb) - -A (f(hhoo)))
{AP2 [h Tb) - A.7am pho h bol
< sup keT,
sup
her
statistical error from sampling Pz
+ sup [Bolh] Eoa [M]] + sup (Eq. [h] Eggh]} ke
ke
statistical error from sampling Q
opt
+ (f*) + 8 - a)) sup inf |h Rill
m.m
heTher her
optimisation crror
discriminator approximation error
The distribution Po: (Do )z Pz is the ()-N)GAN estimator, i.c., the approximation to Q obtained by solving the empirical GAN problem 77. The primary difference between the decomposition {8) and the corresponding result for IPMs Lemma 9 of Huang et al. (022) the presence of the generalized cumulant generating function Af= in the statistical error from sampling Pz. The functional A Pz is nonlinear in the discriminator and so treating those terms requires new techniques which we develop in Section 2 Using these new results, in the case of discriminator space I such that sup,,a [h(x;) - -h(7)] < 1 for all h E r, we derive concentration inequalities of the following form; see Theorem 3.7 for details:
PPC ( - 1-innn inf ()(())P) 2 c Copprox + BEB
T.F
n.m Cope
+ 4R ARn + 4K, jKF.foP....
S exp
2 7
m
7.
Here ..F
approo is the discriminator approximation error, R T.Q. denotes the Rademacher complexity of (see
A Appendix C K0 y. Pz.m Pz.m depends on RTo..Po P and A f.m can be bounded uniformly in m. We nole that
3

---

Published in Transactions on Machine Learning Research (05/2025)
e.r. one can always choose r= r, thereby making the discriminator approximation error zero; in general, 'approw simply quantifies the tradeoff between using a weaker discriminator space I' in the Rademacher complexity terms, compared to the stronger space I' used in the -fiveeeiieeeeence The approximation error has alroady been studied for IPM GANs and, apart from the f dependence of the prefactor, there is no difference here. Hence we refer the reader to Section 2.2.2 in Huang et al. (2022) for further discussion of this term.
Assuming the discriminator and optimization errois are zero and assuming that one chooses discriminator and generator classes whose Rademacher complexities decay as n, m - 00, our result (9) shows that the (f. (f.N)-GAN estimator Po:... approximately solves the exact (f.T)-GAN problem (5) with high probability. If the generator class is sufficiently rich to ensure inface Pb) = 0 then (9 implies the (f.T)-GAN
estimator is close to the distribution of the data source, Q. with high probability, where closeness is measured by the divereeeee: in this latter scenario we in fact obtain a tighter result than (9), see (35). The exponentially docaying nature of the bounds (9) mcans that one can use the Borel-Cantelli lemma
also conclude almost-sure convergence of ( n.m to inface (F) provided that the errors and + cc. As part of these derivations prove bounds the mean of the error; see Remark 3.10 Our techniques can also be used to derive concentration inequalities for the (f. divvereeeee estimation problem, which is of independent interest for, e.g., mutual information estimation; ion; see Theorem F.1 Due to the asymmetry of PPI in Q and P, we also provide a concentration inequality where the positions of Q and Pg:,. are reversed; see Theorem F.4 Finally, we note that the and A s.m approach their IPM counterparts in the limit where f* bocomes lincar; see
Rademacher complexities approach zero when m
quantitics K Too.Pz.m Pz.m
Remarks 2.8 and 2.11
The concentration incquality 9 and other related results presented below reduce the problem of statistical performance guarantees for (J.N)S to the problem of bounding the Rademacher complexities of the discriminator and the composition of discriminator and generator function classes. This is a well-studied problem which also ariscs the case of IPM-GANs and the same techniques and results from that sctting can be applied here. The discussion in Section 3.1 of Huang el al. (0222) and also Chakraborty & Bartlett (2024) shows how covering number bounds imply that, under appropriate assumptions, the IPM-GAN performance depends on the intrinsic dimension, d", of the support of Q and not the (possibly much larger) ambient dimension, d. As our new bounds depend on the Rademacher complexity in the same qualitative manner as the TPM results, the aforementioned insights regarding dimension dependence also immediately apply to (f. ()NNNs Furthermore, in Scction 3.3 we discuss Radenacher complexity bounds for distributions with unbounded support. While several approaches to this problem can be found in the IPM-GAN literature, see Biau et all (2021) and |Huang et al. (2022). our result in Theorem 3.13 requires significantly less restrictive assumptions and so provides new insight even in the IPM-GAN case. Morcover, Theorem 3.13 has many
applications beyond the study of GANs, such as for neural estimation of mutual information Bel BBelghee Melggazi el al.
2018
Song & Ermon 2020b
Birrell et al. 2022c
1.2
Comparison with Generalization Bounds for the JS-GAN
The closest existing work on GANs with nonlinear objectives can be found in Arora et al. (2017), which dcrived statistical guarantecs for nonlinear GANs with JS-style nonlincar objectives. The present work is distinguished in the following ways.
1. The nonlinearity in the (f.N)-GAN objective is of a different character than what was considered in Arora et al. (017). Specifically, in addition to composing the discriminators with the nonlinear function f", the minimization over the (unbounded) shift parameter in (4) introduces several complications, as one can no longer a priori assume that f is bounded or uniformly Lipschitz, as was assumed for é in Arora et al. (2017) (see their Theorem 3.1); the necessity of including shifts was discussed in Birrell et al. (022b) and also in Theorem 2 of Farnia & Tse (2018). ; and hence is inherent complication of rigorously studying GANs that are based on infimal convolutions, such (J,N)-GANs.
2. The statistical consistency results proven here in Theorem 3.7 and in Theorem 3.1 of Arora et al. (2077) are of different natures. The latter derives bound the difference between empirical and
4

---

The fundamental difference between the more commonly studied IPM-GANs (1) and the (f.T)-GANs (5) is the noulinearity of the objective functional (3, coming from the generalized cumulant generating function E detailed study of the properties of the generalized cumulant generating function, Af. is therefore Published in Transactions on Machine Learning Research (05/2025)
exact errors (as measured by the chosen divergence) while our result (9) implies that the solution to the empirical GAN problem provides near-optimal solution to the exact GAN problem. While both results are meaningful, they have different interpretations and their derivations require different techniques.
3. Finally, note that the generalization bound in Theorem 3.1 Arora et al. 2017 assumes the space of discriminators is Lipschitz in its parameters. Generally speaking, for discriminator architectures used in practice this property holds only when the data and noise distributions have compact support. In contrast, our novel Rademaccer complexity approach in Section 3.3 allows for discriminators that satisfy a more general local Lipechitz property in their parameters. This allows our framework to be be applied distributions with unbounded support, including a range of heavy-tailed distributions.
2 Properties of Af
required in order to extend the techniques used for IPM-GANs to obtain statistical guarantees for (f, -)-GAN.
We undertake that study in this section. Detailed proofs can be found in Appendix D
Going forward, we let X' be a measurable space and Ms(X)) be the space of bounded measurable real-valued functions on X. We let f: (a, + R, 0 S < 1<bs 0o be convex function that satisfies f(1) = O; we denote the set of such functions by Fi b). We will make regular use of various properties of such convex functions and their Legendre transforms that are collected in Appendix A In particular, we note that a 2 0 implies f* non-decreasing. The value of the right derivative of at will play a key role many of the proofs and so make the following definition:
20 f1)..
(10)
By assumption, 1 is in the interior of the set where the convex function f is finile and so this right derivative is guaranteed to exist and be finite.
The first key property is that bounds on a function h translate to bounds on Af and also allow the minimization over v (4) to be restricted to correspouding compact interval.
Lemma 2.1. Let f E b)) with 0, P be probubility measure on X, and h E Mb(x) with a S B. Then:
a E A 3
2. If f is also strictly conver on neightorhood of 1 and 20 E {f" < oo then
Af(M
inf v 8-20
(u+ Fpps v)]}
(11)
Remark 2.2. We use A to denole the interior of a set A.
Next we give a Lipschitz bound for AP.
Lemma 2.3. Let P a probability measure on X, and h, h E My(X)) with a S h < 3. Let f (i,
with and assume f is strictly conver in neighborhood of 1 and 20 + 3 - a E {f < xo.. Then
-97h-A910 S(ST)(EE + 3 - a)]l h||
(12)
We now aim to obtain a tight bound on the difference between A A [h] and A [h] when P. and P,, are empirical measures that differ by only a single point. This will be key for an effective use of McDiarmid's inequality, thus allowing us derive statistical guarantees for (f.N)- in Section For this purpose it will be convenient to make the following definition.
5

---

Published in Transactions on Machine Learning Research (05/2025)
Definition 2.4. Let f E b) with 0. Define A, :R R by
(iaa-vi) Σ f (at (x==-11) -m
Aイ(x) = inf
V +
(13)
PER
The connection to A is given by the following lemma, which a simple consequence the respective definitions.
Lemma 2.5. Let ʃ E FF, with a 2 O. Let Mo(x) and P. = = - oz be an empirical mensure on X. Then
Af t = o
(14)
where hn (x))==== (h((s))..,, h(xn))
The Lipschitz bound in Lemma 2.3 translates into the following Lipschitz bound for Ay-
Corollary 2.6. Let f b)) with a 2 0. Let a S 3 and assume f is strictly conver in a neighborhood of and z0 B a 15" < xx). For x, E [a, 31 we have
(y - ,y(=)1 _ (F) + B ②3)
Σ 1=1
5. - -xsl-
(15)
In particular, A is continuous on [a, 3
The following lemma gives tight perturbation bounds on A, when the inputs differ in only a single component, which in turn gives the desired tight bound on Pm [h] P [h] when Pn and Pn are empirical measures that differ by only single point via Lemma 2.5
Lemma 2.7. Let f (a, b) with a 2 O. Let a S 3 and assume f is strictly convex in a neighborkood of and za + 3 a E ( < o Given j E n), let E ((x, [a, 3)m (a, 3)th æi 5z for + j}. Then
-2 n
f a x2)
sup - (H,ā)EE
inf 2E|3o- (B80)-(B-)),0)
(2)
(16)
and we have the simpler loose bounds
8 a
- - a+2) -2+ n (z) + (x)))
inf
(17)
n
=E( (3 (B-ax).o))
n
n
1 (f a 20) (3 20) 3-a
n
Remark 2.8. Note that 17p implies that the bound 16 continuously appronches the result in the linear case
(i.e., where Af = Ep, which corresponds to (z) = z on sufficiently large interval) as (f*) a 20) approaches 1.
Finally, we present a pair of uniform law large numbers (ULLN) results that bound the maximum difference between A P. f [h] and Af[h] over a set of test function h E T in terms of the Rademacher complexity of T, where P. is the empirical measure for i.i.d. samples from P. These results should be compared with the ULLN for means which provides uniform bounds on the difference between the empirical mean and expectation; Appendix C for the relevant background. First obtain simpler result that uses [I] along with a Lipschitz bound to reduce the problem to the ULLN for means; this can be thought of as a substantially more general version of the estimate in the proof of Lemin 2 in Belghazi et al. (2018), which studied a KL-divergence based mutual- informatioooo method. However, as we will show, the resulting estimate is overly pessimistic it does not reduce the IPM case in the appropriate limit. Following this simpler lemma we will derive tighter bound that does possess the desired limiting property, though requires slightly more restrictive assumptions on

---

Published in Transactions on Machine Learning Research (05/2025)
Lemma 2.9. Let V be nonempty countable collection of measurable functions y and suppose we have 8 E IR such that I dh S 3 for all va E V. Let E (i, with 2 0 and assume is strietly conver in neighborhood of 1 and zo + 3 E {f < co.o
Let P a probability measure on y and Ye, i = 1,.., n independent samples from P with Pn the corresponding empiricnl measure. Then
E sup NEW
+
(Aflm)
8-a (0 (Ro.pn + 2n
(18)
Lemma 2.10. Let a counlable collection of measurable functions on y that contains al least one constant function and suppose we have a, 3 E R. such that S th 3 for all vb E v. Let f E (i with and assume f is strictly conver in neighborhood of 1 and za a E (s" < <oo) Also assume that (f*)4 is Lips-Lipschttz on [20- (3 -a), 20
at.
Let P be probability measure on y and Y,, i 1, be independent samples from P with P. the corresponding empirical measure. Then
{ -f
sup NEW
(19)
52min { [1+26 La 2) Run (B (8-a) (B") a 20) Rv Rw. B B-a
2/
2n
Remark 2.11. The result in Lemma 2.9 does not reduce to the ULLN for means in the case where (2) (on a sufficiently large interval), i.e., when A, [t] = Ep hh]; specifically the bound (18) differs from the ULLN for means, as recalled in (64) of Appendix C by the term 8-a in that case. However, 19 does reduce to the ULLN for means when (z) z. Moreover, the bound
approuches the bound in (64) as the Lipschitz constant for (f*Y, on [zu- 3aa-67 (B-a)) a), 3 aa] approaches zero. We emphasize that, when comparing our resulls with the IPM case, the sub-oplimal second term in (2.9), which comes from the parameler v, cannol be absorbed into the Rademacher complezity as the bias of the final layer. This is because the bios of the final layer always cancels in an IPM and hence does not contribute to the Rademacher complexity terms for IPM-GAN, Thus the improved Lemma 2.10 is necessary in order to derive general result that appropriately reduces to the IPM case.
3
Error Bounds for (f.T)GAAS
In this section we use the properties of A from Section to derive concentration inequalities for (-DANS.NS.
3.1 (f.T)-GAN Error Decomposition
We start by deriving a decomposition of the (f, error into statistical, approximation, and optimization error terms. First we consider the error that comes from approximating the idealized discriminator space r by smaller space (c.g., a neural network) r, step that is generally required when implementing GAN method:
Lemma 3.1. Let F c I' C Mo(X)) be nonempty and suppose have a,8 R. such that S h S B for all h E T. Let f E Fi with a 2 0 and assume is strictly conver in neighborhood of and + 8 a a {f" < cofo Then for any probability measures Q. on X' we have
< DF((1IP) - < (1++)))))) + +3 -a) sup inf ||h Talloo
(20)
hel ker
7

---

Published in Transactions on Machine Learning Research (05/2025)
Proof. For any h r, h I use the definition and Lemma 2.3 to compute
- D((P)))) <BQM Eq [1] + Af(h 1
(21)
s - hl]m.(Q) 51h-bllm>(0)) + (f)) + a)ll h(lls(p)
<1+ + - -a) 111 11-
Minimizing over gives
Eq - (F a)) inf |h her
(22)
for all h E T. Maximizing over h then completes the proof.
Next we outline the assumptions we make regarding the discriminator, generator, and the empirical GAN optimization.
Assumption 3.2 ((f,))-GAN Assumptions). Let f E Fi(a,)) with a 2 and a, IR, a < B that satisfy the following:
f is strictly conver in a nbhd of 1.
20 + a E (f" oo)o, where 2a was defined in
10
Lel (2,Bx) be lopological space with the Borel sigma algebra. Suppose T C Ca(X) (the space of bounded continuous functions) are nonempty (the discriminator spaces) and satisfy the following:
S for all h e.
2. There erists countable To c I' such that for all h E F there exists sequence hy E To such that
h + pointwise.
There erists a countable To c f such that for all h e f there erists a sequence h, E fo such that
hg + pointwise.
Let Z be another measurable space, Pz probabilily mensure 2, and to 2 - X be measurable for 0 E O where θ is separable metric space. Suppose 0 + 40(2) is continuous for all E Z. Define Pa = (4e) Pz, which is a probability measure on X.
Let Q be probability mensure on and i= n, 7,, 1- 1. m be independent and distributed as Q, Pz respectively. Let Qn, Pz.m- and Pe.m be the empirical measures corresponding X;, Zi, and Dg Zi respectively. Finally, suppose that for each m, have error tolerance copt and O-valued random variables 87_m that approrimate optimizers to the empirical GAN problem, i.e., that satisfy
DJ (Q Po: m) < inf (Q.l + BEe
opt n,m
P-a8.
(23)
Remark 3.3. The assumplions regurding To, To. separability of O and continuily of D9 and h E allow to address the issue of measurability of the various suprema that arise in the derivations below by enabling one to restrict them to countable subsets. These assumptions hold in most cases of interest and can also be replaced by any alternatives that serve the same purpose.
Several important examples I's that satisfy the required assumptions can be found in Table 1 Specifically, these examples all satisfy 2a E (f* < )0 and hence 20 + 3 E (f* < oo for appropriate choices of a, 3.
The concentration inequalities that we derive below rely on the following decompositions of the (f.A) error. These should be compared with the result for IPM-GANs, Lemma. 9 of Huang et al. 2022
8

---

Table 1: Convex functions, f, that satisfy the assumptions required by the theorems below, along with their convex conjugates, /", and the value of zo, as defined in (10). Note that the representation of the Jensen-Shaan Shannon divergence obtained by combining the first row of the table with (6) is different from the one Published in Transactions on Machine Learning Research (05/2025)
Divergence
f(t)
[*)
20
JS-divergence
(log(t) - (t + log (+2)
- log(22 - E²)
0
KL-divergence
+log(t)
E=-1
1
a-divergence (a 1)
1
aa 1)a/(a-1) max 2, 0)a/(a-1)) a(a-1)
used in the original GAN (Goodfellow al. 2014) or in the analysis of JS-style GANs in Arora al. (2017)
Lemma 3.4 ((f.))-GAN Error Decomposition). Under the Assumption 3.2 the (f.T)-GAN error can be decomposed P-a.8. as follows:
inf (Q))PP) 050
(24)
< sup
(A
Ph.m
[h Pbl - A [ho bol)
sup
( [h Dbl -
-A, Pa.m [h Pel
heT ,E0
NEñ
+ sup {Eqlh] Eq. E + Ret
sup {Eq, [h] -Eqlh] her
+ (1+ (f*)4 (zo + aa) sup inf ||h - 011000 + opt keT hel
Proof. Using
23
we can compute the P-a.s. bound
P/ ... - inf (P)P)
(25)
= =DJ inf (F Pa.m) + inf
( inf (Q1(PP)
OEB
dE0
0EO
SDJ(Q) (Q (o;.m)))
opt f.m
inf (Qal
inf Dj (Q).
050
Using the fact that I C r implies Dj < DJ. along with Lemma 3.1 we then find
DI inf (Q())P)) 8E0
(26)
<DF(Q - Df (Qnll Po m + inf DJ (Q.)) inf (Q11)))
0E0
dee
+ +1+ (1+) 3 - a)) sup inf ||/- hilloo + opt
hell heT
Next make use of the simple bound
+ (supd, supc)) S (d -4)
(27)
ie7
ief
iET
9

---

Published in Transactions on Machine Learning Research (05/2025)
whenever c₁ d, E R for all E I and sup, Ci sup, d2 are finite. Using the definition (3 along with we can compute
78
and
27
Ay Po [h]] Egg, [bb] -A, Pon.m m [h]]
DF(Q)PPo..) m. ) inf P)())) S sup 0EB
Eo[h]
(28)
k
[r), - -A, -A,00m()) - (Fo(i)) - A, ()))
sup sup 0E NET
+ (1 + (S*) + - sup inf || Tollo + 2
NET MET
sup (Fi [h] -Fq. [i]]] + sup AET,
{ { (APom [M] -Afom1)
ker
+ supf Eq, [ht] - Eq[hh} | + sup
[APo[A] - , fmim [h]
her
hef
(1 + (S ( (20 + 3 a)) sup inf ||| T + 6x heT heT
We have AP0M) = A [h bb) and Afam ( [b] A 2 [h dD) and so this completes the proof.
We note that the setting of Lemmas 3.4 differs in few ways from that of Huang et al. (0022). Namely we assume I and we do not assume that inføce (Q11 = O. Under appropriate assumptions, the latter can be proven to hold for a sufficiently rich class of generators. More specifically, the analogue of the approach in Huang et al. (2022) would be to work under the assumption that inface (M, (Pm (P) = 0 for all
empirical distributions Mu. As DJ= S dr (see (60) in Appendix B), this zero generator approximation error property holds for the (f. divveeeieee whenever it holds for the the corresponding IPM; see Yang al. (2022) and the discussion in Section 2.2.1 of Huang et al. (2022) for sufficient conditions. Below we give an error decompositions that is adapted to the zenoapproxination-ervo the proof, which is very similar to that of Lemma 3.4 can be found in Appendix D
Lemma 3.5 (U,T)-GAN Error Decomposition 2). Under Assumplion 3.2. and supposing that infsee DJ (u (P) = 0 for all empirical distribulions pa, the (JTT)GAN error can be decomposed P-a.s. as follows:
PF i
{A Pz. [h 2o] - -A [h- Dol] + sup NEÑ,EE
{A/=th [h Yol - -A,- -f [h [ho Dol} (29)
sup heT ,E0
+ sup her
(Fo [h] Fq., [h]] + (1 + (f4)4(20 + - ()))
sup inf || Rull + copt het her
Note that the terms in the bounds (2) and c.
29 reduce to the terms in IPM case, Lemma 9 in Huang et al. (0222). if
3.2 Concentration Inequalities for (AAT)
We are now ready to derive concentration incqualities for (f.N), we present two variants, depending on whether one assumes the zero generator approximation error property for empirical measures. The key ingredients are the ULLN for the generalized cumulant generating function Lemmas 2.9 and all0gllog with the perturbation bound from Lemma 2.7] the latter being nceded in order to apply McDiarmid's incquality to the lerms involving the generalized cumulant generating function. The following definition lists several quantities that will appear in the concentration inequalitics below. We note that these quantities approach their counterparts in the linear (i..,, IPM) case in the appropriate limit; see the discussion in Remarks 2.8
and 2.11
Definition 3.6. With notation as in either Assumption 9.9
10

---

Published in Transactions on Machine Learning Research (05/2025)
1. Define the discriminator-space approrimation ervor
r.r Caprroz
= (f* + + - a)) sup inf || - boloo.
(30)
hel kel
Note that this vanishes when I =T r.
2. For n E z define
[*(=) (-2+ n- fnn =E(20 181200(8-aj) =8=
inf
++())+ + n
(31)
n
and note that fnn
fm 68 a, (3 + 20) -20l.
Let Z+, V be nonemply family of measurable functions X, and P a probability measure on X. If ( is La,s- Lipschitz |20 (B a), 20 3 a] and v contains constant function then define
K.J.P.P..
(32)
:mmin { { (L+ (3)3-a)La,n))ee (3 - (=)))²
B - a
(f a 20) RR...n +
2n
2n2/2
and otherwise define
B 3-0 a =1 a Ruv.mn +
(33)
2n
With these definitions, present the following concentration inequalities.
Theorem 3.7 ((f.))-GAN Concentration Inequalities). Under Assumption 3.2 and in particular with o the approzimate solution to the empirical (f,1")GAAN problem 233 for E we have
P (F DEG
inf (T(Q))P)) 2
r Capprox + Copt 4R Q. 4k f.To0,P2.
(34)
2
Sexp
2
a) +
where we refer to the quantities in Definition 3.6
If. in addition, infece (Hm))PP) for all possible empórical distributions pn then we obtain the tighter bound
P (DI(QIPPz.... ........) )
r.r approz
opt
2R
T.q.
4K. (.
S exp
(-I
a)
62
2 f.m
(35)
2m
Remark 3.8. See Table 7 for several important eramples of nonlinear f to which this theorem applies.
together with the dominated convergence theorem imply + f/2 hh Øo] is are continuous and also that h + pointwise implies A [h, o 2b] + A o &a), Eqlhyl Eq and similarly for the empirical variants. Therefore, letting O, denote a countable dense subset of θ, the suprema in Lemma 3.4 can be restricted to the countable subsets Io and Oo, giving the P-a.s. bound
Proof. First. note that Lemma 2.3 and Corollary 2.6
lEe DF(()1Po:..) inf (1((PP)))
(36)
{ (Arxm
Pa. [h Tol - -, Ap21M [h fol)
{ Φol -A,- p/ [ Tol
sup heTo.0E@p
sup heTo,
+ sup [Ellh] - Eq, [1]} + sup {EQ.!M [b] EEl11}
keño
keto
+ approz + opt
r.F
1,m
11

---

Published in Transactions on Machine Learning Research (05/2025)
Now we will apply McDiarmid's inequality to the right-hand side. The map H : x x Tm +R defined by
(x, = sup DEG,
{ {A Pkmm [ [ Øo] - A 2 [/ bol)} +
sup NETo,
{ Qol - A Pz.m [I [
(37)
+ sup {Eq [h] Eq, [h]} sup AETo
(Eq, | [[[h))
heto
is measurable and if a, E X" differ only in the j'th component then
(ar) 1 Σ M(:s)
Hx 2) ((, =)1 2 sup
(38)
heTo
12
2 sup {1((2)) ((a0)))>h(a(3333 < 2 (3 a)
n NETo
while if 2, E z differ only in the j'th component then Lemma 2.5 and the perturbation bound from
Lemma 2.7 imply
2 {la (h (2) Ay (h (2) = A
(H(x,)) HI,, 2)
2
sup
p.m
(39)
AcTo.EEoo
m
Therefore we apply McDiarmid's inequality to H, see, e.g,, Theorem D.8 in
Mohri et al.
2018 to obtain
2 2 2 a)²
P
Z)) S exp
(40)
2
m 1.
In terms of H, the error bound
36
becomes the a.s. bound
inf (Q11P))) < H) + #iee
Caprrox +
r.r
opt c,
(41)
Using the ULLN for means, see Theorem generating function Lemmas 2.9 and 2.10 we obtain
along with the new ULIN for the generalized cumulant
E[ Z)] < 4RT T. 4K J,To0,P..
(42)
Combining
40
41 and
42
we arrive at the claimed result
(34)
If we also assume ine DJ (Mm Pa) = 0 for all possible empirical distributions pn then we can write the error decomposition 3.5 as
[] SH(X, + opproo opt
rF
(43)
P-a.s., where now define
H(x,))= = sup
{AP Ps.m m [h Dol - A f + sup
{ Do) A, p P
(44)
sup { {Eqlh) [h] - [hl}
heTo
Similar to the above, when and & differ in only single component we can bound [x 2) H(, z)1 - - (3 aa)
(45)
and when z and ă differ in only a single component we can bound
H -) (xx, 2) < A.m
(46)
The mean of H(X, Z) can be bounded via Thcorem C.4 and Lemmas 2.9 and 2.10
E(H((, < 2R T.Q.a 4IC
(47)
P.Fo
Combining 43 and
45
47
with McDiarmid's inequality completes the proof of
35
12

---

Published in Transactions on Machine Learning Research (05/2025)
We also note that the methods employed above can also be used to obtain concentration incqualities for the estimation of (Q])P) from samples; see Appendix F. for details. Due to the asymmetry of DJ in its arguments, it also desirable to obtain concentration inequalities for the reverse (f,N') ie., with the role of the data source and generator reversod. The analysis in this case is very similar and the corresponding results can be found in Appendix F
By combining Remarks 2.8 and 2.11 one finds that the result of Theorem 3.7 reduces the IPM-GAN case when f is linear (on sufficiently large interval). Morcover, the terms appearing in the bound contimuously approach their IPM counterparts (2) approaches 2 in the appropriate sense, outlined in the aforementioned remarks. Morcover, in practice, while n is often inherently constrained by the availability of data, there is no inherent constraint on m, as Pz is specifically chosen so that it is easy to sample from. Therefore one can choose to work with m >> n, in which case the terms involving 1 A J.m on the hight-hand
sides of (34) and (35) are negligible. Similarly, under appropriate assumptions, /TTo .p... will be negligible Therefore, apart from the approximation and optimization errors, the error terms in both (34) and 35 will be nearly the same as those in the IPM case in the regime m > n. The same is not true if one reverses the order of Q and Pe in the arguments of the divergence, as in Theorem F.4
compared to R see Theorem 3.13
Remark 3.9. As stated, the above results assume a space of discriminalors, T, that salisfies a uniform bound of the form a S h S 3 for all h r. However, we note that the divverenee objective functional in (3 is invariant under constant shifts, due to the identity Af +c= [h] + for all R. Therefore, if we have discriminators ye such that the ranges of vb v have uniformly bounded diameter, i.e,, there erists B such that supa,a (x(z)) - xb(3)) S B for all vp E V, then we can write
(48)
where := (1b inf tb 1p E V) satisfies 5 h S for all h ET Thus our theory can be applicd to DJ and hence also to Dÿ via 448 In this way, our theorems can be applied to -.ANN with discriminators whose ranges have uniformly bounded diameter, e.g., 1-Lipschitz functions on a compact domain.
Remark 3.10 (L*-Bounds on the (f,r)) error). Bounds on the mean of the (f,I)-GAN error are implicil in the above derivations. They are oblained by combising the error decompositions in Section 3.1 with the ULLN results in Lemma 2.9, Lemma 22.0, and Theorem C.A. In addition, standard technique can be used to turn the concentration inequalities into L error bounds for any q > 1 follows: Let Y be non-negative random variable, a E [0, xi), and K (c) + (0, be measurable such that
P(Y 2 2 + a
S K(t)
(49)
for all E > O. Then for q we have
E|y S a" +9 / (a K(c)dc.
(50)
This follows from rewriting
E 9 = //00 10 P(Y 2 r)dr.
(51)
breaking the domain of integration into [0, a and [ag, oo), then changing variables in the second term and using the bound 449
3.3 Rademacher Complexity Bounds for Distributions with Unbounded Support
The (J,N) concentration inequalities derived above do not explicitly make any assumptions regarding the distributions Q or Pz. However, for the bounds to be meaningful one requires additional assumptions to ensure the Rademacher complexities are finite and approach zero as the number of samples increases to infinity. If the distributions have compact support, the decay of the Rademacher complexity for typical discriminator and generator classes (e.g., neural networks) follows from standard covering number arguments
13

---

Published in Transactions on Machine Learning Research (05/2025)
without any further assumptions the distributions. However, the case of unbounded support is more subtle. As the the noise source Pz is often chosen to be Gaussian in practice, the ability to handle distributions with unbounded support is of interest even when the data distribution naturally has compact support. In this section we provide an approach to this problem that only assumes the distributions have finite second moments, as opposed to the much stronger assumptions made in previous approaches, e.g., the aub-Gaussian and sub-oxponential assumptions required in Proposition 20 of Biau al. 2021 and Theorem of Huang et al. )022)) respectively.
Assumption 3.11. Suppose (O,de) is metric space and we have collection of measurable real-valued functions on V, V = {ih : A E } that satisfy the following properties.
For all y E y there erists L(y) such that 0 tho(y) is Lip)-Iipshhitz.
P probability mensure on y and L E (P)
Remark 3.12. In this section are thinking of the discriminator being parameterized family of functions (c.g., NNj and the A E θ represents either the discriminator or the composition of discriminator and generator, depending on which Rademacher complexity term in (34) is being bounded.
Theorem 3.13. Under Assumplion 3.11 for all n E Et we have the Rademacher complexity bound
[o De
R.... S 12n 1/2p/11
Vlog N(e, O de)de.
(52)
where De == supe, #.ce de (0,,,,0, is the diameter of @ and N(c, O, de) denotes the covering number of e by e-balls in the metric de.
In particular, 0 is the unit ball in R under some norm then the entropy integral is finite and we have
Re.
(53)
Proof. Using Dudley's entropy integral, soc, c.g., Corollary 5.25 in van Handel (0166) or Thcorem 5.22 Wainwright 2019 we can bound the empirical Rademacher complexity at sample y y by
Dn(s)
Re, (v) <12n 112n-1/2 1/2 L
log ((. (v), (lza(m))de
(54)
where (e, Vn (v), || ((())) is the e-covering number of (n) == (((()), v(un)) vb E V/ under the norm (M)2)())) v Σm= 12 and (y) the diameter of V. under this norm.
The map θ + Vn, (2). 0 + (v,00, (0, yn)) is onto and is Ln) = EE L(wi))) 1/2 Lipschitz with respect to (de. [l))(m)). ) as demonstrated by the calculation
vi.,.., ((((01,)1),... (01, - (sb(0,, v1), ... (02,, Un)))))))))))))))
(vb(01, vs) - vb)(2,))))
(55)
L 2 (0,,01,02)
These properties imply the following relation between covering numbers, N(e, (,))), |l (()(()) S (v)) O, as well as the diamcter bounds (v) < In De) Combining these pieces and changing variables in the integral we arrive at
(56)
De, where De == supo, o.ee (e 02) is the diameter of e.
Ry.n (v) 112n 11/2
((/LL//...)) 0, de
(57)
( ((, ((e,
De
-12n 1// Ln( 1//)1//))))))
14

---

Published in Transactions on Machine Learning Research (05/2025)
Taking the expectation of both sides, we obtain at the Rademacher complexity bound
De
Rv.ra 12n 1/2
1/2 F/pppm pmmm1 [Im]]
log (,,)dd,
where
n
I(vi)² 1)
1/2
Ep [mn]] -
Ep Σ
= =F
(58)
If θ is the unit ball in R with respect to the norm |lele then De S 2 and we have the covering number bound O,e, lle) s (1 + 2/e)*, see, e.g., Example 5.8 in Wainwright 2019 Therefore
De Vlog N(e, θ le)dde S k2/ /////
1-" Vlog(1+ 2/e)de S v2k { E - de 1/2
(59)
Thus we arrive at
53
The Lipschitz property from Assumption 3.11 with L(y) a + blllll, E IRd holds for many neural network architectures, (0, y). parameterized by E C RR In such cases, Theorem 3.13 implics Om-11/3) (resp. 0(m-1/2) Rademacher complexity bounds for GANs whenever the discriminator (resp. and generator) spaces are appropriate neural networks, assuming that Q and Pz have finite second moments respectively. When combined with Theorem 3.7 this implies statistical consistency of the corresponding (J,T)-GANs. We emphasize that existence of the second moment is much weaker assumption than the sub-exponential or sub-Gaussian requirements of previous approaches, and thus our Theorem 3.13 improves on the state of the art bounds even in the IPM-GAN case, i.e., Theorem 3.7 with [=) = 2. We also note that in cases where P Y + Z where Y has compact support and Z is (possibly unbounded) perturbation with mean zero with (6)) variance then Theorem 3.13 yields a Rademacher complexity bound that differs from the cetprbbutioo-indeendeeen bound in the Z=0 case by a ((8k/nn)1/2) term; thus, perturbing the data with a unbounded noise that has small variance results in a negligible difference in the statistical guarantees for all n.
4 Conclusion
We have derived statistical error bounds for (f.N),GANs, a large class of GANs with nonlinear objective functionals. These GANs are based on the ddiveedivesssscss, which generalize and interpolate between integral probability metrics (IPMs, e.g., 1- 1-Wasserstein) and f-divergences (e.g., JS, KL, a-divergences) and have been show to outperform both IPM and f-divergence-based based methods in a number of applications. This paper extends earlier techniques for proving consistency of GANs with linear objective functionals (IPM-GANs) to the nonlinear objective setting. The key technical results are the tight perturbation bound in Lemma 2.7 uniform law of large numbers bounds for a class of nonlinear functionals in Lemmas 2.9 and 2.10 and the (f.T)-GAN error decompositions in Section 3.1 These results allow for the derivation of finito-sample concentration incqualities for (f, ()-NNs in Theorem 3.7 We also presented a new Rademacher complexity bound in Section 3.3 that implies the statistical consistency of (f.T)-GANs for distributions with unbounded support that have finite second moment. In particular, our results can be applied to certain heavy-tailed distributions, where the MGF does not exist, thus providing new insight even in the previously studied IPM-GAN case. Moreover, these novel Rademacher complexity bounds are of independent interest for analyzing her statistical learning tasks.
References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Doina Precup and Yee Whye Tch (ods.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 214-2112222 214-223, International Convention Centre, Sydney, Australia, 06 11 Aug 2017. PMTRR
15

---

Published in Transactions on Machine Learning Research (05/2025)
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 224-232. PMLR, 06-11 Aug 2017. URL https::/prooeeedingg mml. http:///ww......er.eeeaee.eeeeee pptps//////ooorraa.......... hat.//aaraaaaaaall...aa
Mohamed Ishmacl Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 531 540, Stockholmsmässan, Stockholm Sweden, 10 15 Jul 2018. PMLR. URL http://proceediing. .mmpeiiinnnmmm..l....mm.mmrm mlr. wwwps/ http:////w/baaaaa..aaaa....
Aharon Ben-Tal and Mare Teboulle. An old-now concopt of convex risk measures: The optimized cortainty
equivalent. Mathematical Finance, 17(3):449 476, 2007. doi: 10.1111/j.1467999666 000.996..00...000 2007.00311.x URL 991.--11911111119--99955
1467-99665... 2007 00311.x
Gérard Biau, Maxime Sangnier, and Ugo Tanielian. Some theoretical insights into Wasserstein GANs. Journal of Machine Learning Research, 22(119):1-44 45, 2021.
Birrell, M Katsoulakis, L Rey-Bellet, and W Zhu. Struclure-preserving GANs. Proceedings of Machine Learning Research, 162, 2022a.
Jeremiah Birrell, Paul Dupuis, Markos A Katsoulakis, Yannis Pantazis, and Luc Rey-Bellet. (S, D)vvevvrrreences:: Interpolating between f-divergences and integral probability metrics. Journal of machine learning research, 23(39):1-70, 2022b.
Jeremiah Birrell, Markos A. Katsoulakis, and Yannis Pantazis. Optimizing variational representations of divergences and accelerating their statistical estimation. IEEE Trunsactions on Information Theory, 68(7): 1553-45722 2022c. doi: 10.1109/TIT 00.1109/TIT.2022.316000 2022..166665.
Michel Broniatowski and Amor Keziou. Minimization of divergences on sets of signed measures. Studia Scientiorum Mathematicarum Hungarica, 43(4)::03-444) 2006.
Saptarshi Chakraborty and Peter L Bartlett. On the statistical properties of generative adversarial models for low intrinsic data dimension. arXiv preprint NNN::2441.15001 2024.
Ziyu Chen, Markos Katsoulakis, Lue Rey-Bellet, and Wei Zhu. Sample complexity of probability divergences under group symmetry. In International Conference Machine Learning, pp.
PMLR, 2023a.
Ziyu Chen, Markos A Katsoulakis, Lue Bellet,,,, and Wei Zhu. Statistical guarantees of group-invariant GANs. arXiv preprint arXin:2305.190 13517, 2023b.
Ziyu Chen, Hyemin Gu, Markos Katsoulakis, Luc Rey-Bellet, and Wei Zhu. Learning heavy-tailed distributions with Wasserstein- propeeateieeroiiinimaeeeeiieeee a-divergences. arXiv preprint 2XXv:2405.13962 2024.
Dupuis, Paul and Mao, Yixiang. Formulation and properties of a divergence used to compare probability measures without absolute continuity. ESAIM: COCV, 28:10, 2022. doi: 10. 10.1051 11101/cocx/2022000 URL
https:///do.orrg/11 htt.g://Aoo.og///00000////ooo222 101/////000///0
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. Proceedings of the Thirby First Conference on Uncerlainty in Artificial Intelligence, 2015.
Farzan Farnia and David Tse. A convex duality framework for GANs. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL http:::/prooeeeiigg..no http://eeeeeeeeeeeeereeeeeeeeeee http://ppprffflllepppppr.pr o00p:/1144Blaaal0000000444000000
G.B. Folland. Real Analysis: Modern Techniques and Their Applications. Pure and Applied Mathematics:
A Wiley Series of Texts, Monographs and Tracts. Wiley, 2013. ISBN 9781118626399, URL https
//bbokk.googge.ccoo http://cccc/ooooo///eeee//AAAAAa
16

---

Published in Transactions on Machine Learning Research (05/2025)
Pierre Glaser, Michael Arbel, and Arthur Gretton. KALE flow: A relaxed KL gradient flow for probabilities with disjoint support. Advances in Neural Information Processing Systems, 34:8018-800 2021.
Tan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS'14, pp. 2672-2680, Cambridge, MA, USA, 2014. MIT Press.
Arthur Gretion, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. J. Mach. Learn. Res., 13(1):723 773, 2012. ISSN 1532-4435. URL http://dl. htt.m/.l.cmmm..mmmmmmmmmm acm org/
citation. cfm 2508308 2188410
Ishaan Gulrajani, Faruk Ahmod, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved Training of Wasserstein GANs. In Proceedingg of the 31st International Conference on Newral Information Processing Systems, NTPS'17, pp. 5769-5779, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Jian Huang, Yuling Jiao, Zhen Li, Shiao Liu, Yang Wang, and Yunfei Yang. An error analysis of generative adversarial networks for learning distributions. Journal of machine learning research, 23(116):1144) 11:1, 2022.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International conference machine learning, pp. 1718-1722 PMLR, 2015.
Tengyuan Liang. How well generative adversarial networks learn distributions. Journal of Machine Learning Research, 22(228): 41, 2021.
Shuang Liu and Kamalika Chaudhuri. The inductive bias of restricted f-GANs. arXie preprint Sfff:18099045.2 2018.
Shuang Liu, Olivier Bousquet, and Kamalika Chaudhuri. Approximation and convergence properties of generative adversarial learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL https::/pproceeddngg
(49177bbaaa85332cee66ddeeee -appe-aaperrr..-..-- .df
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. URL
M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning, second edition. Adaptive Computation and Machine Learning series. MIT Press, 2018. ISBN 9780262039406. URL httpps//books
X. Nguyen, M. J. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk mininization. IEEE Transaclions on Information Theory, 56(11):5847 5861, 2010.
Sebastian Nowozin, Botond Cseke, and Ryola Tomioka. F-GAN:
Training Generative Neural Samplers Using Variational Divergence Minimization, In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pp. 271-279, Red Hook, NY, USA, 2016. Curran Associates Inc ISBN 9781510838819.
R.T. Rockafellar. Conver Analysis. Princeton Landmarks in Mathematics and Physics. Princeton University Press, 1970. ISBN 9780691015866.
Avraham Ruderman, Mark D. Reid, Darío García- Garcia, and James Petterson. Tighter variational represen- tations of f-divergences via restriction probability measures. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, pp. 1155-1162, Madison, WI, USA, 2012. Onnipress. ISBN 9781450312851.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Jiaming Song and Stefano Ermon. Bridging the gap between f-GANs and Wasserstein GANs. In International Conference on Machine Learnöng, pp. 9078 90888-907770000 PMLR, 2020a.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information cstimators. In International Conference on Learning Represenlations, 2020b.
R. van Handel. Probobility in High Dimension. APC 550 Lecture Notes. Princeton University, 2016.
M.J. Wainwright. High--Dimensional Statisties: A Non-Asympiotic Viewpoënt. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. ISBN 9781108498029, URL https
7/books. gowgle.
Yunfci Yang, Zhen Li, and Yang Wang. On the capacity of deep generative networks for approximating distributions. Neural nelworks, 145:144 154, 2022.
A Properties of f and fr
In this appendix collect number of important properties of the convex function f and its Legendre transform f that are needed in the study of D)-NNs.. For b satisfying -oo S < < b S co we define (a to be the set of convex f (a, b) - R with f(1) = 0. For f E Fi (a, b), standard convex functions theory, see, e.g., Rockafellar (1770) and Appendix A in Birrell et al. (202bb), implies that / and its Legendre transform f : + (-00, 00), (z) = suPte(a,b) {zt - f(t)}. have the following properties:
(2) 2 z for a E R (this uses f(1) 0).
2. a then f is non-decreasing.
3. (m")=
f" is convex and LSC.
f is continuous on {f+ < oc), where A denotes the closure of a set A.
6. The right derivative (f*), exists and is finite on {f oo); where A denotes the interior of set A. Similarly, f exists and is finite on (a, b).
(f")4, is non-decrcasing and absolutely contimuous on compact intervals; the latter implics that f satisfies the fundamental theorem of calculus, see, e.g., Theorem 3.35 and exercise 42 in Folland (2013).
Of particular relevance will be the value 2n == = f1), which exists and finite due to the assumption that 1 ba, The importance of zo the -diveevives was observed in Birrell et al. (2022b), where the following properties were proven (see Lemma A.9):
Lemma A.1. Define 20 == (1).
(20) = 20
I is strictly conver on a neighborhood of and za (f* oo)" then (f*)
These properties will be key our analysis and so we will generally assume that f is strictly convex on a neighborhood of and zn {s" < oc).
18

---

Published in Transactions on Machine Learning Research (05/2025)
B Properties of the Divereeiees
In this appendix collect several important properties of the (d.V"-divergences. For X' a measurable space we let MM(22) denote the space of bounded measurable real-valued functions on X and P(X) be the set of probability measures on X. The following results are taken from Theorem 2.8 of Birrell et al. (2022b)
Theorem B.1. Let f (i,ab)) C (cMM(X) be nonemply, and Q, P. E P(X).
()1P)))) & inf + nEP(X)
(60)
In particular, (Q1))) min drQ P)).
2. The map (Q,P) P(S) P(S) - (F is convex.
3. If there crists cu I MIRR then (Q11)) 2 0.
4. Suppose f and P satisfy the following:
(a) There exist a nonemply set & CT with the following properties:
i. V is P(X) dedeerminn, i.e., for all Q, P(X), EQli] Ep(p) for all E v implies Q P.
H.. For all uh there erists co R, en > 0 such that co esp E r for all c0.
(b) f is strictly conver on neighborhood of 1.
(c) f* is finite and C on a neighborhood of f4(...
Then D has the divergence properly, i.e., PPI) with equality and only if P.
Remark B.2. Under stronger assumptions one can show that 60 is in fact an equality; see Theorem 2.15 in Birrell et al. (0226)).
Remark B.3. Assumptions 4(b) and 4(c) hold, for instance, if is strictly convex on (a, b) and f1(1) e
(5" < )0; see Theorem 26.3 in Rockafellar (977).
C Rademacher Complexity and Uniform Law of Large Numbers
In this appendix we recall the definition of Rademacher complexity and its use in proving uniform law of large numbers (ULLN) results. There are two definitions of Rademacher complexity in common use, those being with and without absolute value. In this work we use the version without absolute value as defined below.
Definition C.1. Let v be collection of functions on y and n E zt. The empirical Rademacher complerity of at E yn is defined by
R = Es sup Loa NeW n
en (en 1
(61)
where o,, i= 1,.,,, are independent uniform random variables taking values in [-,,1), i.e., Rademacher random variables, and v. = (b(1)),.., ((y))). Given probability distribution P on V, the Rodemacher complezity of relative is defined by
Rv,p, = Epn [
(62)
where Pd is the n-fold product of P, i.e., the y: become i.i.d. samples from P.
Remark C.2. Note that in 62 we are implicitly assuming that the empirical Rademacher complerity is mensurable. When use 62 in the main text, this will be guaranteed by other assumptions.
19

---

Published in Transactions on Machine Learning Research (05/2025)
The definition 61 is particularly convenient for purposes due to the following version of Talagrand's lemma; see Lemma 5.7 in Mohri et al. (2018)
Lemma C.3. Let a, B E R and vp be collection of real-valued functions on y such that + S B for all & E V. If o: [a, $ + R is L-Lipschitz then
Roo nn S L
(63)
for all n E _t, y E y", where o 6 V = J
Bounds on the Rademacher complexity can be used to prove ULLN bounds; see, c.g., Theorem 3.3, Eq. (3.8) (3.13) in Mohri et al. (2018) (this reference assumes [0, IJ-valued functions but the result. can be freely shifted and scaled to apply to a set uniformly bounded functions):
Theorem C.4 (ULLN). Let P be a probability measure on y and U countable family of real-valued measurable functions on y that are uniformly bounded. If Y,, i i= 1, n are i.i.d. V-valued random variables that are distributed as P then for n Zt we have
Σ (Y,) - Ep Epp()) )H 2R..P 12
)
E
sup DEV
(64)
Remark C.5. We have slated the ULLN in terms of a countable collection of functions avoid measurabilllyy issues, but under appropriate assumptions one can apply to an uncountable V, e.g., if there exists countable Ma V such that for every tb V there erists sequence thy E Vo with thy - 1/ pointwise.
D Proofs of Important Lemmas
In this appendix we present the proofs of several important lemunas from the main text. We begin by deriving the properties of A that were stated in Section
Proof of Lemma 2.1
1. f is non-decreasing, therefore
inf (v + fa v1) s inf fu HER
inf + fB -)} VER
(65)
PER
for all v. For any R we have
inf
(v + 1e (c - inffc - (c-
(66)
PER
sup(2 f²)) c (1) c -(-))=
EER
Applying this to c= a and c = 3 we see that a S <APTh] & 3 as claimed.
For the following it will be useful recall that f*(20) = 2o and (f')4 1; see Lemma A.1 Suppose 3 2g and first consider the case where (hh sc. We have h-v < h h- zu) Zu: therefore h + 1/n, h- (3 =o) E (ʃ" < co)o for sulficiently large. S is absolutely continuous on compact intervals, therefore the fundamental theorem of calculus holds, see, e.g., Theorem 3.35 and exercise 42 in Folland (2013), and hence we have
/ h (3--2))
(- (8- = (1h
jn-ut1/a (2)).
(67)
For 2 < h (8 20) we have S 20, hence (z) (ft)4(2)) 1. Therefore
(h (3 =0)) SJ (h- 1/n) h (3 20) - (h 1/n)
(68)
=h 1/n) - (B 20) + 1/n.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Using the continuity of f on {f" < oo) we can take - to get
μ =+ (h-
f (3-20))
(69)
This was proven under the assumption that (* 15-v) v) < oc, but it also trivially holds when this is infinite. Taking the expectation of both sides we therefore find
4 Eplf (h- v)] 2 (3 20) + Eplf (h (B 20)) 2 inf
+ Eplf - (70)
vEla 80.8-01
for all V > 8 - 20.
Now suppose a 20 and first consider the case where (' D) < xc We have h - h (a- 20) 2 20. Therefore h 1/n, h (a 20) E (f < co)o o for sufficiently large and
- ((/n)- = f (h- - (a - 20)) + h-v-1/n
ha (2)dz.
(71)
For 2 2 ht (a - =0) we have 2 20 and so (#) 2 Hence
- 1/n) 2f (h - (a 20)) + h y - 1/n (h (a z0))
(72)
= f (h - (a - 20)) V 1/n + (a - -20)
Taking n - 00 gives
μ+ 11 z0) + f (a-=)))
(73)
This was proven under the assumption that (* - u) < oo, but also trivially holds when this is infinite. Taking the expectation of both sides we therefore find
v - Eplf v)] 2(a - z0) + Eplf (h- (a - 20))] 2 inf
{v + Eplf w)}}
(74)
vela-2o. 3=z0l
for all v < a - 2g- Therefore we conclude that
Af[h] inf 20(0-20,88-8)
{v+ + (hlf*(h-vi)
(75)
as claimed.
Proof of Lemma 2.3 f is non-decreasing, hence 2o - (5 < oo implies 2q E (5" < oo.% Therefore
we can use Lemma to write
A AMM
inf wEla- 10.a-20.8-0 80.8-10)
(u+ Frlf"(h (hv)))
(76)
The fact that f* is non-decreasing also implics that h - 20 S f v) S f a + 20) < oo for all
V E [a 20. 3 - zo] and hence Eplf* v)] is finite. The same holds for h, therefore
<
sup vElo 8clo-20.8-30 2-.8-20
Epllf( v) - f w)[l
(77)
Here we made use of the simple bound
#(inf d; inf ci) 5 sup - c:)
(78)
ie!
whenever ca de R for all i E 7 and inf, G, inf, d, are finite.
21

---

Published in Transactions on Machine Learning Research (05/2025)
f is absolutely continuous on the interval between h v and h v, therefore can use the fundamental theorem of calculus to write
f. nn (f))())d2
h-v
f(h (6-v) v) - (h - v)
(79)
and so
-) v)1 s h h
sup
(f)+
(80)
The fact that f is non-dccreasing implics (f*) 2 0 on {s < oo.o Combined with the fact that (f*) is non-decreasing, we can therefore conclude
(5 -) v) - (" w s Ih
33
(81)
Taking the expectation of both sides and combining the result with 77 we obtain the claimed Lipschitz bound
| - -- (<0 + 3 a)llh h())
(82)
Proof of Lemma
2.7
First let z, = a for all i, E; = a for i i j and ăة = B. We have (x, 2) E E and therefore
(i) - (x)) (a,5)EE
sup
(83)
1 n
r - v) 11
inf ME 00-20.0-20 20.
V+= Σ
(ig 1)
inf
Σ V
PER
n
f=1
i=1
1
f' - -v) v)+ + pr (1)
1
inf we 1e-000000000
a +
+supfa - -- (a v/}
n
veR
inf =<[<0- 10:0-((8 o0)
-2 +
1
(2) + _ a 2)
((')((1)
n
inf
-2 n
(z) + f -a+z
sE(so-(8 (00a-(B-o)) or)
To prove the reverse inequality, let (z, 2) E E be arbitrary and compute
(Z) - (x)x)
(84)
inf 1+ f (3: - -)
inf
V+ Σ
((x--1)) vi}
20,9 =zol
vEla-zo.1 8-20
n i=1
-+ 1 n
n
sup
inf
Σ
f (r: - -0) - n
-
Σ
(" - v)
- De(a- 20(0-20.8810
n
(=1,4
(=1,f
1 - D) (x, ]
1
n1
sup
inf
1 v+
1
Σ
8' (1:--0) - Σ (" v)
= 20) pE(a- 20.8-20
n
1=1,4
(=1,1
2) n
where in the last line we used the fact that f is non-decreasing. We have 5, D. - V E (5" < oo)o and for each i # j the terms involving I, are absolutely continuous on compact intervals with right derivative
d (_ ;) 1 f ( (e) w) n
= ((F) (z. - D) (x, -
(85)
dx. m
22

---

Published in Transactions on Machine Learning Research (05/2025)
which is non-positive when & 2 and non-negative when D S (due to (f*) being non-decreasing). The fundamenial theorem of calculus then implies that 1 [ B)
R [xx - v) is non-increasing in 2; when
V 224 V and non-decreasing in 24 when v v. Therefore
-
(86)
- u u ---
i } D) 1/22/ ʃ
< sup
inf ve(n 20.n- - 20| vEla-20.55
1 - "
a- - V --1
1/<)< if a < 2
= sup
inf a)
vEla-20.8 - 20
2E 20
a- V ((a- aa+
if 1/ 2
where we changed variables to = = a- - ú in the last line. For = E (a - u, z0), the right derivative of the objective with respect to 2 is given by
d dzt (a-2-u+
1 1 f = -1 (1) --
((8-v)
(87)
n
n
(f*)², is non-decreasing and 3 a + 20. therefore -1+ implies that the objective is non-decreasing in 2 [a (0--2) V, 20 (the extension to the endpoint follows from continuity of the objective). Therefore
= =0.
0. This
(z) (z)
(88)
a 2 + n (²) s' - v) + 1 r
((B-aaz)) a a++) 2)}
sup
inf
wEla- 20] =E(30- (8 (B-a);a-0
n
n
{
{ - fa -v)+ (1))
1 fr 13- a f3 5 at))) a- n - 1 (*) +
sup vEla-so.0-20
+
inf
xE| (8 a)
Let z. be a minimizer of
inf =E|=o- (8-0)-(--)).o)) 1--2 = n 1
+xx)++ n
1 (10)++)(0)++)+++++
(89)
which exists due to compactness of the domain and continuity of the objective. The objective is convex in 2, therefore it is non- non-increasing on (-00, 2.] and so
A²()) - (x)
(90)
{
- - (aa-v) a- 1e-2. a- (())
if - 1 2 2, if < 2,
}
< sup "
(U+ - - +/
w w (w) +,, +
n (1))((z)) + n f - if if 2,
sup =E[=0- (5 - a):0
1 - a + w) (w))
if w <
where we changed variables to w = a - v. We have supweR (w (w)) = (1*) = and w + f²3 a + +w) - (ww is non-decreasing on w S z0 (this follows from (f*) + w) (f+)4 2 0), therefore
Ay( - (5(x) & max { -2, + n
a (* + 1 r a + 2.), - (1(8 -+) ((x))) n
(91)
n
Using the bound f*(2) 2 2 we can compute
-2+t n - 1
1 n
1 - -z))
f (2x) +
=- + (z)) > 0.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Hence
Ay(8) - Ay(x) S- 2x n-1
1 f (3 a + 2x)
(z,)+
(92)
n
n-1 (x))
1
inf =E| a))
-
n
1
This holds for all (xr 3) E E and so when combined with
83
we obtain the claimed equality.
To derive the looser bounds, first use (z) 2 = to compute the lower bound
+ n (z) +
1 fr a a2))
2 8-a
inf xE(so- -0(=)(0--)),2)
(93)
For the upper bound we compute
- n- (2) ( a ((88-++))) +2)
inf
(94)
=E(20 (8 a0)
S 20 + 1-
f*(2)) + f a+ 2)) = - (s a 20) - -z0)
3 a <(M)) - + zn)
where to obtain the last line used the fundamental theorem of calculus together with the facts that (zo) = 20 and (f*)4 + is non-decreasing.
Proof of Lemma 2.9
The result is trivial if = B so suppose < B. Note that f" is non-decreasing, and if < oc} implies E {f < oo)" Using Lemma 2.1 along with 78 we obtain
hence 20 + B
sup - A VEla vEla-20.3
(+ v)] (r.10"
(95)
Note that 20 (B - a) S (3 v) S lr 20) < 0o and the expectations finite. Also, the supremum over be restricted to rational due to continuity. Maximizing over i and taking expectations gives
E sup NEW
+ Af
S
sup 2 8- 20
[+EEl[ -1]] 15 m1]33
(96)
The ULLN (see Theorem C.4
implies
E
NE NEW, 30.00/0000000 {+(Ep]f v)] Ep, (f v))1}
sup
(97)
- Σ 1(Yi)) }
sup WEH
Ep[h]
2M.
11
where H = (bb v) v E V,N E 1a1- 20,3 2o] Q). /" and (s+Y are decreasing, therefore I is
(B + z0) - Lipschitz on (-o0, - a + 20] and so Talagrand's lemma (see Lemma C.3 implies
RH.P, <( a 20) -0-0
(98)
We can compute
Ry- Ry-laato0--000..11 Rv. + 8- P,
(99)
24

---

Published in Transactions on Machine Learning Research (05/2025)
and
20 80.00
= Es
sup
v
Eg
sup
(2 (8 + +a/2 20)
01
n
vEla- So 8-20l
n1
=E|- (1-0))))/22 (2,00/22)))))
im1
(100)
27
1/2
B 0z
8
Eg
sup
2
0i
Eg
03
S
Eg
Σ
0g
11
=El- (8 (B)
2n
2n
1=1
i=1
8
2n
Combining these completes the proof.
Proof of Lemma 2.10 As in the proof of Lemma 2.9 the result is trivial when a = B. For a < B we have
E
sup NEV
+
-A
[Afle]
A
s E
sup
[# - w) - (V (101)
NEW. - 20.8- mQ
Now use the fundamental theorem of calculus to compute
(* - v) = (b - (3 - z0)) (3 20 v) S( ( (1- t) za) - -tv)d -tv)dt
(102)
for 3 E V E la 20, B - z0l Qn. Therefore
sup NEV.NEla-20 nn
(+ (v: -)] Frr, [f" (ib w]}}}
(103)
(Epls (1b (B -20))] - Ep, (xp (B -0))
sup
vee,vE|a-20.0 so
+ (8 -33-2 20 uv S Er(([*)) (1 (1- 1) 20) 1v)] Fr, 1(1' Y4 (1b (1 1)(B =n -(w]jet
S suup ( (Frr(s (v) (B - 20))] -Er, Fr (1b (B 20})]}
New
vEla 8- { (8- (8-20-13 v) /''' (EPl(T) (1 (1-t)) () - 20) - -t/]]
+
sup
- -Ep,(00))( - -11 1)(3 - - -[w)] di
<
sup
{1E)plh) - Ep. [pp.[h]}
(104)
(V
(B- 20 -v) J.' (EPILPX (EL- -00 - -tv)]
+
sup
(1-+)(3-
Pev vEV,ve(n-20.8 20 n2
Ep., (1- -1)((B - =0 - [wwi]] dl
The ULLN (Theorem C.4 implies
sup
(#(Ep[h] - Er, 1h)3
(105)
hEf
(3-20))
We have @ - (3 2p) < 20 and f is (2") on (-00, 20] where (5") (20) = 1, hence Talagrand's 's
lemma (Lemma C.3 implies
( RRv-(8-20)PPP Rg, Pm
(106)
25

---

Published in Transactions on Machine Learning Research (05/2025)
We note that the shift invariance of the Rademacher complexity follows from the definition, recalled in
Eq. (61 of Appendix C
As for the second term in 104,, we have
(3 (3-22) 20 -0) -v) ( (1- 1)(3 20) - -tv)]
sup VEV.VE(a-so 80. z0
(107)
(E, (1- =o [w)] dt [tt
-til)Jat)
5(B - a) [-
{ |EP1(0)) (1 -1)(1 - =o) - [v] wef 2 8-.8-z0) zonee
sup
EEr, - (1 (13 20 w1ll } at.
Using the fact that v contains a constant (which implies that the terms in the following bound are non-negative) and then employing the ULLN (Theorem C.4 we have
E
sup NEW wE (a- 8-
( (3 20) - -tz]] -x,.l(f)) - t) - z0) - tt)l|
(108)
<
sup
-1-100 20) - 1)(3 =u) - -w
BEV.VE la 2 8- =0
+ E
sup
(-(Epl(f") (1 =0) - tw) - Ep, (P. -1- - 20) -m)))}
MEV VEla 88.88 2
< 4RH.
where H4 == ((f*)( -11 20) - : E V 3- 20] nQ). Using the Lipschitz assumption
(f*) logether with Talagrand's lemma (Lemma C.3 and the result of the calculation 100
obtain
Ra
(109)
S La. Ru. + + +..+,18-00 B-a
-Lv, +
2n
Putting these bounds logether we find
E sup { + (Aflu) A3th [w))]
S2R + (B aa) //' 4La.9 RR. + 3-0 a
dt
(110)
2n
NEW
-2(1 + 2 a) Ru + (B L))
n/2
Combining this with the bound from Lemma 2.9 gives the claimed result..
Finally, we derive the alternative error decomposition from Lemma 3.5
Proof of Lemma 3.5 Using the assumption
23 along with Lemma 3.1 we can compute the P-a.s. bound
D)(Q1PPo SDF 3
+nl aE0
inf DF (f. - (Q Pos Po
(111)
+ +(1+ (f*) + 3 - a)) sup inf ||A Tulloo +
opt
vm
hel heT
26

---

Published in Transactions on Machine Learning Research (05/2025)
As simple consequence of the definition 3). for all 0 E θ we have
(Lll S DJ (Q.)(
sup
[A][
[4] A Po.m [h]]
(112)
heñ
Minimizing over 0 E θ and using the rero-approxiimation-erroo property we obtain
inf BEB
(2 Po.m) s inf (T(())))))+
sup
{A [h] - ,, Pm.m [h]
(113)
DEe
heñ
{A/ - A Po.m [1]]
= sup
beñ
Therefore
1) Po 1
sup
f - , Po. [A] + (1+ + aa) sup inf /|/ - holoc + copt hel her
2,m
(114)
NEt
D.. (Q] 6.m (2 mm)
{A, Pe [k]
A Pa.m [T]] (1 (J*)4 + 3- (-)) sup inf || 11l1xx + fm het AeT
sup
Pob.
[h] - Aj m [h]
+ sup Eq [b] - Eq, [h] + sup het
A
ket
Bounding the last term by maximizing over O and then using A [h] A Øol and A Po.m [h]
A P..m [ bb) completes the proof.
E
Concentration Inequalities for the Estimation of (f, DDvverreesss
The tools developed Section can also be used to address the simpler problem of concentration incqualities for the estimation of (Q) P) using samples from Q and P. Special cases of this problem considered in Chen et al. (2023a 2024). Both considered the case where corresponded to the family of a-divergences and I was the set of L-Lipschitz functions; the former focused on the consequences of group symmetry while the latter focused on heavy-tailed distributions. The techniques developed in this work allow us to derive concentration inequalities for a much wider class of f's and N's, though we do not consider the consequences of group symmetry this work.
previously
Thcorem E.1. Let I be a nonempty set of measurable functions on X' and suppose we have a, B E such that a < 3 and a S h S B for all T. Assume there erists a countable To C such that for all T there exists sequence h; E To such that hy + h pointwise. Let f E Fi( with a 0 and assume is strictly conver in neighborhood of 1 and 2g
a (f" co))
Let Q. P.P. be probabilily measures on X and X,, i= 1, n, X -- 1,..,, be jointly independent samples from Q and P respectively with Qu Pm the corresponding empirical measures. Then for e > 0 we have
22
P PDD[(Q)P) - (F(Qu. -) 2 Sexp
(115)
1 a) + 1
22
(DD DF(Q1P) 2 c 2Rr. 2K.. Sexp
(116)
1 a) +
AA
where amm and P. were defined in Definition 3.6
Proof. Similar to the proof of Theorem 3.7 if we define
H = (IQ)
(117)
27

---

Published in Transactions on Machine Learning Research (05/2025)
then when and I differ by single component we have and when x' and &' differ by a single component, Lemma 2.7 implies
- -((,,)))H((,1)))) - (3 a)
(118)
H (xx, a') _ f.
(119)
m
Therefore we can use McDiarmid's incquality to conclude that
262
EIDT( 2 S exp
(120)
[
a)2
1 A
1.m
Combining 120 (lower sign) with the result of the calculation
E [f =E
sup
EEq, [h - 1/ Ep If - ri}}
(121)
ve 1 3
sup
B [EEq, 1h vl - -Ep, fd (h D\]l
he 10 20.8
{Eohh v] - Eplf -)] = Df(1))
sup hET 20
we obtain 1155
Combining
120
(upper sign) with the result the calculation
E - -D] <E
sup { [Eq, [h] Eq [eeT
+ E
sup
{APIM] - Af [b]]
(122)
wer
and then using Theorem C.4 and Lemmas
2.9 and
2.10
obtain
116
F Concentration Inequalities for Reverse (f, D)-GAN
The asymmetry of the (f, -ivveeeeeeess requires us to separately treat the cases where the generator the first argument and where it the second; we emphasize that this choice can make a significant difference in practice shown in the examples in Birrell et al. (2022b). With this in mind give second version of the (J,N)-GAN assumptions. The proofs for this case are very similar and so omit them.
Assumption F.1 (Reverse (,,N)-AAN Assumptions). Assume that Assumplion 3.2 holds, with the sole exception being that 0% satisfies
(P (1))) S inf (Po.....))))) + C,m P-a.s.
(123)
lEe
as opposed to 23
Lemma F.2 (Reverse (f.N)-GAN Error Decomposition). Under Assumption F.1 the (f,T)-AA error can be decomposed P-a.S. as follows:
DJ 11)) - inf (Po BE0
(124)
< sup {Epz [h Φe] AEñ
EEP. [/ 2ol}
sup NET 8ce
{Epz. ( Lo] Epz 1h o Φe]}
[1 2m [h] - -191h]] + sup heT
[991)] - - [b]
+ sup
heT
+ (1 + (S*)4 (zo 3 a)) sup inf ||| Jellos w.m her ket
28

---

Published in Transactions on Machine Learning Research (05/2025)
Lemma F.3 (Reverse (f,I)-GAN Error Decomposition 2). Under Assumption F.1 and supposing that infeee DJ (Pollm))) 0 for all empirical distributions pa, the (f.I)-GAN error can be decomposed P-a.s. follows:
(Po: [1) -
sup Nef
EEpz ( D8] - -EP,,m [h [b + sup hET,0EG
{Epz..) 1 hol Dol - - 1h del} (125)
- 17 (1+0)))) oi) sup inf ||/ illoo En.m her het
+ sup keT
Theorem F.4 (Reverse (S.T)-GAN Concentration Inequalities). Under Assumplion with 0% mm the approrimate solution to the empirical (J,T)-GAN problem
and in parlicular have
123 for
P
PP 1Q1) - inf (PolI)) BEG
2ct c
aaproox +
r.F.
n.m Copt
+ 4R Fox T Pam + 4K f.
(126)
22
exp [B a)2 + A
fm
where we refer the quantities Definition 3.6
If, in addition, infece D) 0 for all possible empirical distributions Mn then we obtain the tighter bound
2
TP (D) n.m 1Q) 2 + opppoo +
r.r
n.m Cpt + 4R P2.m Pm.m + 2K, jK....))))))
eex
2 m
2
(3
a
(127)
2n
j.n
29


Reviews:
[{"review_id": "JmV8x4R7WP", "reviewer": "Reviewer_xG3i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The nonlinear dependence on the test function introduces various technical hurdles which make it more challenging to establish properties that would otherwise be straightforward in the IPM setting, e.g. the sensitivity bound in Lemma 2.7 where one shows that the generalized cgf for an empirical distribution does not change too much when one data point is changed.\n- The main conceptual contribution is that this consistency result allows one to interpolate between existing consistency results for GANs based on IPMs and GANs based on $f$-divergences.\n- They establish finite-sample concentration bounds (the aforementioned Lemma 2.7 is crucial here)\n\nWeaknesses:\n- As with any paper proving generalization bounds, there is the drawback that it does not touch upon optimization aspects\n- My main reservation about this work is the motivation, see Requested Changes below", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers a class of GAN-style objectives that interpolates between $f$-divergences and integral probability metrics, previously introduced in work of Birrell et al. '22. They key difference between this and typical IPM-style GANs like Wasserstein GAN is that the objective depends in a nonlinear fashion on the discriminator. The main finding of this work is a generalization bound: if one tries to minimize this measure of discrepancy between {empirical distribution over i.i.d. samples from the target distribution} and any {generator that pushes forward an empirical distribution over i.i.d. samples from the seed distribution}, the discrepancy between the target distribution and the pushforward of the seed distribution under the resulting generator can be controlled by the usual three types of terms: finite-sample error, optimization error, and approximation error term. When the space of discriminators has bounded range, this can be further strengthened to a high-probability bound.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "While the paper is technically sound, I would have liked further discussion on the motivation for this paper and for $(f,\\Gamma)$-divergences. While these are a generalization of existing GAN approaches, what is the advantage of this formulation over previous ones, e.g. with regards to training stability and mode collapse? I would have imagined that the more complicated objective makes these even more finicky to train. Given the decline in popularity of this method for generative modeling, it is especially important that papers in this space try to offer counterpoints that make a case for this approach or variants thereof.", "broader_impact_concerns": "N/A: while the paper is on generative modeling, it is a theory paper proving statistical error bounds and there are no relevant ethical implications", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E5A3EO7Da0", "reviewer": "Reviewer_GdXz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n - The paper is structured logically, with clear definitions, theorems, and remarks, helping understand of the complex theoretical arguments. The manuscript is enjoyable and instructive to read.\n- Rigorous and comprehensive theoretical analysis to prove the statistical consistency for the general class of GANs. Compared to the exiting literature that focuses on IPM-GAN with linear objectives, this paper extends further towards more general (f,\\Gamma) GANs with notably weaker assumptions. \n\n**Weaknesses**\n\n- While theoretically strong, the paper currently offers limited direct practical guidance for designing and training GANs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The primary contribution is the derivation of finite-sample concentration inequalities for GANs, establishing theoretical guarantees on the statistical consistency of $(f,\\Gamma)$ GANs.\n- This paper uses new techniques to handle the nonlinearity of the objective function introduced by the generalized cumulant generating function. These methods hold potential applicability beyond GANs.\n- The authors derive the improved Rademacher complexity bounds under less restrictive distributional assumptions, making the analysis more closely aligned with practical scenarios.\n- The derived bounds generalize previously known results for IPM-GANs, recovering them explicitly as a special case when $f^*=z$.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Minors:\n    - Currently, citations appear as inline text, which can disrupt readability. Using parenthetical citations when it is not part of the sentence would be better.\n    - Define $m,n$ when they are first introduced in Section 1.1. Their definitions weren’t clear to me until I saw Lemma 2.5.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7W7iNDhT7u", "reviewer": "Reviewer_NE8H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper studies the theory of GANs, which is a highly interesting subject.\n2. The paper presents an error bound for $(f,\\Gamma)$-GANs with a non-linear function $f$, an interesting result that applies to several successful GAN formulations.\n\n**Weaknesses**\n\n1. Although I find the paper's main result in Theorem 3.7 interesting, I am unsure whether this result is sufficient for an independent publication at the level of TMLR. While it is nice to have a separate bound for the authors' defined $(f,\\Gamma)$-divergence, the result appears to only address the case of a non-linear $f$. Furthermore, the paper lacks bounds or novel discussion about the main error components that remain in the bound: 1) the approximation error term, 2) the optimization error term, and 3) the Rademacher complexity. These generally defined error terms are left unanalyzed, and in my opinion, Theorem 3.7 alone is not sufficient for a top-tier paper on the theory of GANs.\n2. To elaborate on my previous comment regarding the paper's result, let's discuss the error terms in Theorem 3.7. The approximation error in equation (30) includes the term $\\sup_{h\\in\\Gamma} \\inf_{\\tilde{h}\\in\\tilde{\\Gamma}} || h - \\tilde{h} ||_{\\infty}$. I am concerned that this term may not be small for standard choices where $\\Gamma$ is the set of 1-Lipschitz functions and $\\tilde{\\Gamma}$ is a set of neural networks. For this error term to be bounded by $\\epsilon$, we would require an everywhere $\\epsilon$-accurate approximation of every 1-Lipschitz function, which seems highly unlikely in moderately high dimensions. As the authors do not analyze this error term, it is unclear if such a bound is useful for the discussed case where $\\Gamma$ is the set of 1-Lipschitz functions.\n3. Similar to the approximation error term, there is no discussion on the optimization error term, which is indeed a critical challenge in training GANs. Additionally, the paper uses a standard uniform convergence analysis for bounding the estimation error, based on a general Rademacher complexity term, which would likely lead to a vacuous bound for a standard deep neural network discriminator. Including some discussion on the applicability of the result to standard GANs would address this concern.\n4. Based on the above, the paper's contribution appears limited to deriving the additional constants in the bound for a non-linear $f$-divergence function $f$, assuming the scalar input to $f$ varies within a limited range $[\\alpha,\\beta]$. To demonstrate the utility of this result, it would be beneficial to discuss its implications for standard $f$-divergence cases such as JS or KL divergences. Currently, the paper presents the result for a general $f$, leaving the potential value of the result in determining which function $f$ to use unclear.\n5. The paper lacks numerical results. While running image-scale GAN experiments may be challenging, synthetic experiments with uniform or Gaussian data distributions would be valuable.\n6. The paper's literature review misses several related works on the theoretical analysis of $f$-GAN and IPM-GANs. The following papers also theoretically study $f$-GANs and IPM-GANs with restricted discriminator sets:\n    * Arora et al, \"Generalization and Equilibrium in Generative Adversarial Nets\", ICML 2017\n    * Liu et al, \"Approximation and Convergence Properties of Generative Adversarial Learning\", NeurIPS 2017\n    * Farnia et al, \"A Convex Duality Framework for GANs\", NeurIPS 2018\n    * Liu et al, \"The Inductive Bias of Restricted f-GANs\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies generative adversarial network (GAN) formulations formulated using a constrained discriminator set with an $f$-divergence loss function. This family of GANs, which the authors define as $(f,\\Gamma)$-GANs, has been often used in the literature. The paper shows a statistical consistency result aimed at $(f,\\Gamma)$-GANs. Theorem 3.7 bounds the gap between the $(f,\\Gamma)$-divergence of the optimized generator $\\theta^*_{n,m}$ and the infimum achievable divergence by assuming bounded discriminator approximation and optimization error, and the Rademacher complexity of the discriminator set.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following would make the paper stronger:\n\n1. Discussing the application of the result to standard choices of $f$-divergence measures, such as JS and KL divergence. Especially, it would help to show a particular choice of $f$ might be preferred based on the paper's theorems.\n\n2. Adding some numerical results on synthetic and perhaps image data that examine or support the main theorem.\n\n3. Including more discussion on the utility of the theoretical results and their implication on the design and performance of GANs in practice.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work provided a theoretical analysis on the statistical error bound of a family of generative adversarial networks based on the so-called $(f, \Gamma)$-divergence (an interpolation between the $f$-divergence and the integral probability metric, IPM). The main technical challenge is to deal with the nonlinear objective (as opposed to the linear one in IPMs) due to explicitly accounting for translation invariance. The paper is expertly written and the results seem to strictly extend existing analysis. 

One one hand, all reviewers recognized the technical contributions, including the presentation and soundness. On the other hand, the reviewers voiced concerns on the applicability and relevance of the theoretical analysis to existing practices. In the response, the author further explained how the current results extend previous ones and pointed out the value of this work: more on justifying some of the existing practices than motivating new designs, which I tend to agree. 

In the end, I believe the contributions of this work outweigh the (practical) concerns: while it may not (directly) inform practitioners how to train GANs better, it is nevertheless assuring to know when some of the GAN variants are statistically sound (not to mention that some of the results may be of independent interest).
Decision: Accept (poster)