Bridging the Gap Between f-divergences and Bayes Hilbert Spaces
Authors: Linus Lach
Alexander Willi Fottner
Yarema Okhrin
Venue: ICLR 2025 Poster (2025)
Keywords: f-divergences
Bayes Hilbert spaces
fenchel conjugates
centered-log-ratio
posterior approximation
Primary area: learning theory
TL;DR: We generalize f-divergences to bridge the gap to Bayes Hilbert spaces by introducing a novel pseudo divergence framework that allows locally non-convex divergence generating functions.

Abstract: We introduce a novel framework that generalizes $f$-divergences by incorporating locally non-convex divergence-generating functions.
Using this extension, we define a new class of pseudo $f$-divergences, encompassing a wider range of distances between distributions that traditional $f$-divergences cannot capture.
Among these, we focus on a particular pseudo divergence obtained by considering the induced metric of Bayes Hilbert spaces.
Bayes Hilbert spaces are frequently used due to their inherent connection to Bayes's theorem. They allow sampling from potentially intractable posterior densities, which has remained challenging until now.
In the more general context, we prove that pseudo $f$-divergences are well-defined and introduce a variational estimation framework that can be used in a statistical learning context.
By applying this variational estimation framework to $f$-GANs, we achieve improved FID scores over existing $f$-GAN architectures and competitive results with the Wasserstein GAN, highlighting its potential for both theoretical research and practical applications in learning theory.

Published as a conference paper at ICLR 2025
BRIDGING THE GAP BETWEEN f-DIVERGENCES AND BAYES HILBERT SPACES
Linus Lach", Alexander Fottner, Yarema Okhrin Department of Statistics and Data Science
University of Augsburg {linus. lach, alexander. fot fottnrr,,.. www.aaaa....innn.........
ABSTRACT
We introduce a novel framework that generalizes f-divergences by incorporating divergence-gennrraingg functions that exhibit non-convex behavior in neighbor- hood of the origin. Using this extension, we define new class of pseudo f- divergences, encompassing a wider range of distances between distributions that traditional f-divergences cannot capture. Among these, we focus particular pseudo divergence, obtained by considering the induced metric Bayes Hilbert spaces. Bayes Hilbert spaces are frequently used due their inherent connection to Bayes's s theorem as they allow sampling from potentially intractable posterior densities, challenging task until now. In the more general context, prove that pseudo f-divergences are well-defined and introduce a variational estimation framework that can be used in a statistical learning context. By applying this vari- ational estimation framework to f-GANs, we achieve improved FID scores over existing f-GAN architectures and competitive results with the Wasserstein GAN, highlighting its potential for both theoretical research and practical applications in learning theory.
INTRODUCTION
central challenge in learning theory estimating statistical divergences between distributions from finite samples to quantify their dissimilarity. ffl::..-vvverr f-divergences are a popular class of diver- gences (Csiszár et al. 2004]] Liese & Vajda, (006) including the Kullback-Leibler (KL) and Pear- divergence. Their general form is given by Dy(p, v)
[(dg))d where f is convex lower-semicontinuuus function satisfying f(1 = 0. f-divergences and the subsequent estima- tion have been extensively studied in Nguyen al. (2010); Rubenstein al. (2019); Nguyen et al. 2009); Hero 2014); Keziou (2003). However, research on generalizing these divergences 2022i et al., 2024), particularly by relaxing the convexity
Gimenez & Zou 2022 Birrell et al.
constraint, still needs to be explored.
We introduce novel pseudo f-divergence framework that relaxes the convexity constraint divergence-generaen
functions, offering greater flexibility and applicability various machine- learning tasks. This flexibility is crucial tasks like generative image modeling (Nowozin al., 2016), or topic modeling on large text data sets (Hoffman et al.) 2013). where the underlying data diverse. Previously, the convexity requirement hindered connections between f-divergences and other dissimilarity measures like the Bayes Hilbert space metric (Van den Boogaart et al.) 2014). fre- quently used Bayesian statistics. In Bayesian statistics, inference typically requires sampling from intractable densities (Kroese et al.) 2013). Sampling from intractable densities using divergences in a bayesian setting has been studied Blei et al. (2017); Camphell Broderick (2018); Campbell & Beronov 2019). However, these methods have only been applied using the KL-divergence (Blei ct al.) 2017 Campbell Beronov 2019) or lower dimensional synthetic data sets (Campbell & Broderick 2018).
To highlight the application of general f-divergences Bayesian statistics context, we therefore explore connection between f-divergences and Bayes Hilbert spaces, which provide an alterna- tive structure for measuring distances between distributions. Bayes Hilbert spaces are distribution spaces with vector structure, intrinsically connected to Bayes's theorem, where addition aligns with

---

Published as a conference paper at ICLR 2025
Bayesian updating (Van den Boogaart al. 2014). Due to their Hilbert space structure, they equipped with an induced metric, providing natural notion of distance between measures that can be incorporated into the proposed pseudo f-divergence framework. Another advantage of estimating the Bayes Hilbert space metric over traditional divergences arises from the Hilbert space structure. The Hilbert space structure implies that every Cauchy sequence converges within the space, which can be exploited in an optimization setting. By modeling the loss function as a sequence in a Bayes Hilbert space and using gradient-based approaches. convergence ensured, preventing instability near the optimum. However, defining a Hilbert space for probability measures not straightforward (Van den Boogaart et al., 2010). First, probability measures must be identified as vectors before imposing a linear structure on them obtain linear space. This linear space can then be endowed with an inner product to define Hilbert space. The linear space structure allows probability measures and their associated densities to be embedded in a subspace of L2, cnabling a clear understanding of the distances between distributions.
solution near the optimum is
The connection between f-divergences and Bayes Hilbert spaces can be established using the cir transformation for distributions, which is defined as clr(u) = log( # log( du )dA, where du is ax
аx the Sadon-Nikodym derivative of μ with respect to some continuous base measure X. The induced metric of Bayes Hilbert space, which involves the squared clr transformation, closely relates to a pseudo f-divergence with the preuudo-divergene-geeeee the interval 00, cxp(-1)], we apply newly de-
function I + x ).. Since this
function encompasses local non-convexity veloped framework, which allows us to estimate the so-called BHS divergence and also exploit the characteristics of Bayes Hilbert spaces obtain lower bound for the pseudo divergence that is attained.
1.1 CONTRIBUTIONS
Our main contributions can be summarized as follows:
We develop novel theoretical framework bridging the gap between f-divergences and Bayes Hilbert spaces.
We generalize f-divergences by allowing for locally non-convex divergence-genenee functions.
We present an efficient sampling algorithm for high dimensional distributions.
We apply our extended pseudo-divergence estimation framework to f-GANs, outperform- ing traditional f-GANs achieving lower FID scores.
1.2 RELATED WORK
Our work extends the existing body of research on f-divergences and their ap- plication in learning theory, particularly in the context of sampling from unknown distributions. A foundational approach to estimating /-divergences was developed by Nowozin et al. (2016) Nguyen al. (2010), where the authors developed and utilized variational estimation framework esti- mate f f-divergences with an application to GANs. Their framework allows for flexible divergence estimation but remains constrained by the convexity of livergence-generating functions. Our ap- proach addresses this limitation by relaxing the convexity constraint, allowing for a broader range of pseudo f f-divergences that can be tailored specific tasks.
f-divergences:
Besides f f-divergences, Rényi divergences, which generalize the KL divergence by introducing a pa- rameter a > 0, a * that corresponds to weight controlling the sensitivity of divergence regarding areas where the distributions are similar or dissimilar have been developed to provide greater flexi- bility (Van Erven & Harremos 2014). While Rényi divergences provide greater flexibility compared to KL, they are still limited by the convexity constraint inherent in f-divergences.
different approach to measure the dissimilarity between probability measures is offered by inte- gral probability metric (IPM) such as the Wasserstein distance. The Wasserstein distance has gained popularity in generative modeling due to its robustness in comparing distributions with different supports (Sriperumbudur et al.) 2012). However, as Yatracos (2022) has noted, the Wasserstein dis- tance is unsuitable for estimating distributions with heavy tails, as it struggles to handle distributions whose tails are not exponentially bounded. Recent work by Chen et al. (2024) addresses this issue by
2

---

Published as a conference paper at ICLR 2025
proposing a penalized version of the Wasserstein distance, which combines it with a-divergences to estimate heavy-tailed distributions better. While promising, their method requires careful tuning of the penalty terms and relies on the Wasserstein metric's s limitations, particularly in nigh-dimensional
spaces.
In contrast, Birrell et al. (002) combines IPMs and f-divergences introducing (f, diveerggess that inherit the ability to compare distributions that are not absolutely continuous from IPMs, while also retaining the property of f-divergences control heavy-tailed distributions. Although this hybrid approach broadens the applicability of divergence estimation, it still imposes the convexity function.
constraint on the
Bayes Hilbert Spaces: Bayes spaces, and their extension to Bayes Hilbert spaces (Van den Boogaart al. 2010; 2014. Egozcue et al.) 2006) while still in their infancy, were introduced generalization of the theory of compositional data, originally popularized by Aitchison in (Aitchison 1986); Van den Boogaart & Tolosana- Delgado (2013). These spaces address the challenge of defining meaningful distances between probability measures for specific data analysis tasks. The core idea is to introduce vector space structure for measures, where the group operations align with Bayes updating (Van den Boogaart et al. 2014), thus creating a framework in which Bayesian reasoning can be incorporated directly into the space measures.
natural
[1982]
Bayes Hilbert spaces frequently used in functional regression, including Maier al. (2021) Arata (2017); Machalova et al. (2016); Machalová et al. (2021); Talská et al. (2018), where Radon- Nikodym derivatives are modeled using spline functions within the Bayes Hilbert space.
However, one significant challenge with these approaches is their reliance on functional regression, where the targets or features are typically density functions. Since these densities not directly observed in practice, the models must infer them from finite samples generated by the underlying distributions. This creates gap between the theoretical framework of Bayes Hilbert spaces and practical learning scenarios.
Our work addresses this gap by incorporating generative adversarial network (GAN) to approxi- mate the underlying probability distribution while simultaneously minimizing the distance between the learned and true density in the Bayes Hilbert space. This integration allows us to bridge the functional models of Bayes spaces with finite-sampl learning, offering more practical and scal- able solution to regression tasks involving density functions.
In parallel, Wynne (Wynne 2023) explored the use of Bayes Hilbert spaces for Bayesian posterior approximation, concluding that these spaces provide a natural and appropriate framework for ap- proximating posterior distributions. Our approach builds on this insight by extending its application to finite sample learning and divergence minimization in the Bayes Hilbert space.
1.3 OUTLINE
We begin by reviewing key properties of f-divergences and establish theoretical assumptions on the underlying spaces. Next, we address challenges in quantifying dissimilarities hetween measures using f-divergences. Since estimating f-divergences relies on variational framework involving Fenchel conjugates, we discuss how these conjugates are used in dual representation of convex divergence--eneraaing functions to derive an estimable lower bound. We then outline the conditions for realizing this bound and present its analytical form, which remains valid under milder assump- tions.
In Section 3 review Bayes Hilbert spaces as framework for measuring distances between prob- ability measures by embedding them into function spaces with a separable Hilbert space structure defined through the clr. The induced metrie corresponds to the log-variance, which connects to f-divergences.
Finally, we apply this generalized divergence estimation framework f-GANs, demonstrating that the more flexible pseudo-divergence approach outperforms traditional f-divergences.
Proofs of our theoretical results are provided in the Appendix.
3

---

Published as a conference paper at ICLR 2025
2 T-DIVERGENCES
Statistical divergences measure the dissimilarity between probability distributions and are often used estimate parameters statistical models, such generative algorithms designed to sample from unknown distributions. f-divergences are widely used class of these measures defined as:
Dy(,, = ( Pu(2)) (a)
X(dz).
where μ and y/ are assumed to be two probability distributions on the same measure space (S2, B, A) (B denoting the Borel a-field over !2) that are both absolutely continuous with respect to the a-finite base measure X. Py and Pe denote the Radon-Nikodym densities of μ. and v with respect to X. Common choices for À are the Lebesgue measure or counting measure. f is an arbitrary convex, lower-semicontinouus function f 1 IR, satisfying f(1) 0 called divergence generating function. By requiring that f(1) O. it is ensured that = V a.s., Dy(,,v) = and the lower-semiee sower-semicontinuity of ensures that lim f(t) f(0). Note, that f-divergences generally do not induce metrics as they are not symmetric in most cases, i.c., (y, & (v, ()). However, pseudo distances between measures.
they can still effectively be used for measuring dissimilarity Since the function f must be convex, which can be restrictive when defining new divergences, potentially limits the range of distributions generative model can learn effectively. A challenge with f f-divergences is their difficulty in estimation from finite samples of μ and v. To address this, particularly estimating likelihood ratios for multivariate distributions, convex optimization tech- niques are employed (Nguyen et al., 2010). The optimization uses variational estimation involving the Fenchel conjugate of a continuous convex function defined as:
f (v) = sup
{xTy y f(z)}, E dom()")
rEdom(()
This implies the dual representation of f, known the Fenchel convex duality:
Theorem 2.1 (Rockafellar) 1970). Let f R" + R be convex function. If f is lower semi- continuous, then the duality (' = f(z) for all x E IR" holds.
Leveraging the dual representation f(x) = f (x) sup se (x"y-f((y) in the sense of Theorem 2.1 lower bound for the divergence of f1 and can be derived (Nguyen al. 2010):
ET) Ef( o 3m)
Dy(μ,V) sup TET
Here, T denotes an arbitrary class of Borel measurable functions T + dom(f*) By adding the additional assumption that f is continuously differentiable on its entire domain, the bound in equation] is is tight and the supremum can be represented analytically as
P
T(x) = f (x)
By adapting the definition of convex conjugates to allow concave functions instead of convex func- tions, Theorem 2. can be adjusted to hold for upper-semicontinuus,, concave functions as well. The concave conjugate of a function f R. R is defined
fv)
inf {x* y- -=)f()) y dfm(f=). =Edom(s)
Fenchels duality the concave case then states f..(x) = f(x) for x E R² (Rockafellar 1970).
3 BAYES HILBERT SPACES
Another idea for quantifying distances between probability measures is the embedding into suitable spaces of functions such as Hilbert spaces. The induced metric of these spaces then be used to measure the distance between those probability measures. This basic idea will be formulated mathematically in the following paragraph which is based on Van den Boogaart et al. 2014 2010)
4

---

Published as a conference paper at ICLR 2025
Denote by M the set of o-finite measures some Borel measurable space (Ω, B) that equiv- alent to a a finite base measure À on B. Then, I and y are called B e-quivalent, denoted by =(() v, if there exists constant > 0 such that p(A)
cи(A) for all A B with the convention (+o)) +oo. Define (X) as the set of = B equivvllll classes, i.c., the quotient space / =- The relation = an equivalence relation on M(A). Since the considered mea- sures are equivalent each other, the Radon aadon-Nikodym theorem ensures that the densities du/dv exist for every μ1, V E (M)).
Given the equivalence class of = equivaleen measures, addition and multiplication can be defined: For R E B define the perturbation of by u, the powering of y for a scalar a E IR, and the negative perturbation in BA))
du dd. dv In Bayes Hilbert space literature it is common to identify measures with their corresponding Radon- de for every μ E B(A) it Nikodym densities. This is coherent in the sense that by setting μ dλ holds that μ =B ax ax
(A e (R) Jx du dv
d' da ax, (a = (ax) dx, (mOv) di
du
(B(A), θ O) is a real linear space called Bayes linear space with base measure A. The additive neutral element is given by Og == and the additive inverse element Oμ == du
dA To derive a Hilbert
space based on this vector space, we now assume that the base measure À is a probability measure. The space (A),, p 2 of À --euuivalent measures defined
/
|LoE Jog (xx)
l" as < +00
B²()) (
vector subspace of B(A). For details and more basic properties, see Van den Boogaart et al. 2010).
Let a E (A) with corresponding density ax Then, define the centered log-ratio transformation clr of
dp ax
dj ax
clr(()) = log
log
dx.
The cir transformation has the desirable property of maintaining the relation of o with multiplica- tion and with exponentiation respectively which is enforced by the properties of the logarithm. Furthermore, the map clr (A) - (A) is linear.
Recall, that (A) = (g E (A) : gdd = 0}. Given the map clr, an inner product on B² can be defined which in turn induces a metric.
Let therefore Pp and Pe be two Radon-Nikodym densities in B² (A), i.e., for p.v E B² define (r((x) = ax (ar) and px(x)) = (xx) for all x E RR, Then, the inner product (r)) B²()) defined
L (Iog(en(x))) - -
(log(en(w) X(d)) (bostou(e))) - log(p(())) M(dy) A(dr).
(A) is separable Hilbert space and the map clr B²(A) I.((A) is isometry of Hilbert spaces.
The map d'a(a)) (2))) (2(A) + (0, o0), (P)())(P))))))) Px) = ((p, Pv. e Bp)))) an induced metric and the pair d is a complete metric space.
If want to construct f-divergence that is closely related to the aforementioned metrie, its explicit form is major interest. We can derive such an analytical expression of the Bayes Hilbert space metric using the following Lemma.
Lemma 3.1. Let Pn and pv be two Radon-Nikodie densities in B2). Then,
()B())(Pa p2) Varx(log(H e v)).
Therefore the Bayes Hilbert space metric directly corresponds the log-variance loss studied in e.g. Richter et al.) (2020); Nüsken & Richter (202T); Richter & Berner (2024), embedding those works in the area of Bayes Hilbert space learning.
5

---

Published as a conference paper at ICLR 2025
4 BRIDGING THE GAP BETWEEN BAYES HILBERT SPACES AND J-DIVERGENCES
In this section, show how pseudo f-divergence can be formulated to directly relate to the Bayes Hilbert space metric dga, enabling sampling from a probability distribution within a well-defined Bayes Hilhert space. This approach addresses key challenge in Bayes Hilbert space learning, as highlighted in Maier et al. (202T): estimating densities through direct sampling from the learned distribution.
4.1 MIXED CONJUGATES
A key challenge is that f-divergences are only well-defined when f is lower semi semi-continuous and convex. Thus establish the desired connection, we must first extend the definition of fddivergenees to handle local subsets near the origin where is not convex. Consider continuous function f: 00, x)) - R. which is concave on [0, a) for some a and convex on [a, oo). To account for this local non-convexity, we define the mixed conjugate f of f as
13(1== = sup f1x - [(x) ((EM) + inf (lx - [ zE(0.a)
xE(a,x)
where
M {t dom(f?): argmax (tr - f(x)) oo)] =Edom(f)
N = {t dom(f.) argmin (tx f(x)) [0.a)).
#Edom(f)
The set M (N) ensures that consider the convex (concave) conjugate the domain of S where is convex (concave). Under mild conditions, this definition allows a disjoint characterization the subdomains M and N. The motivation behind this mixed conjugate is straightforward: it enables well-defined dual representation for locally non-convex functions.
Lemma 4.1. For any continuous function f: [0, oc) TR. that conver [a, 00), 0, concave on [0, a), and satisfies limz-400 f(x) = +o0, the mixed conjugate satisfies:
1. M N 0 and M U U dfm(ff)
2. f=: = f for all E dom(f)) where ft: denotes the biconjugate of f. i.e. ftr (ft):.
The unboundedness assumption on f is not only theoretical necessity in the proof of the lemma above but also carries significance in the context of divergence -ennereingggggggggggg functions. By assuming that f is unbounded, we permit the dissimilarity between probability measures to grow arbitrarily large.
At first glance, it is not clear why such pseudo-divergences are well-defined, however, the following Lemma proves their coherence.
Lemma 4.2. Let f [0, 20) oo) be concave interval (0, O and convex 0.). For any probability measures μl, E (²),), Dy(1,v) v) is well-defined in the sense that μ
Dj(1, v) and Dy(u, =
To address the challenge of estimating likelihood ratios from finite samples, we build on the ap- proach in Nguyen al. (2010) by developing generalized variational framework applicable to pseudo f-divergences.
Theorem 4.3. Let [0, I. convex on Ja, oo) with 0 < a oc, concave on [0, a), assume that j is twice continuously differentiable, and that ((ft))-1 is X-as. well-defined. Furthermore, let u.v E (2) with de (x) = Pp(x), du (x) == (x)), x E R. Then, T(x) == (Pn(x) is an
optimizer for
sup { (T) - (f: o r))} TEC(M*)
inf TEC(N*)
(T) - Ex o T))}
(3)
Here, C(M*) (C(N*) denotes the set of continuous functions T Ω dom(f") such that f: T convex (concave).

---

Published as a conference paper at ICLR 2025
Table
Table of different pseudo-divergence-
generating functions and their respective pseudo divergence
powas dun grice geneuding fatcteno
f(u)
Dj(μ,
u (l))
log
P((x)) P((
u(u - 1)²
Bp
Pa(ar)
2
(u(
u/u 11
Pr(a))
03
(2)
12
15
23
Vūlu
Ex
V (ar)
v (x)
(P)(a)
(x)
Figure Plot of different pseudo-divergence-
V (t)
generating functions f
Vuluu 1
Ep
(2)(
Theorem 4.3 can be applied to the extended class of divergence evvergenc-geeneraiin functions. A brief selec- tion of such wewrruddddiiieeenneggeeeeeeeeeee functions and their respective pseudo-divergenn be in Figure 1 and [
The optimization problem in equation 3 can be transformed to reduce the two objectives into single equivalent objective with the same optimum. This transformation leverages the property i-f(-) sup(-) and substituting T with T ([ ((rec(-)))) For details,
Proof of Corollary
Corollary 4.4. For any function f satisfying the conditions Theorem 4.3
sup Dy( TEC(M-JUC(("))
(T) - ( (7
(4)
Therefore, by applying the generalizing mixed conjugate the variational estimation framework of Nguyen et al. (2010), we can still derive lower bound for the pseudo didieergence.
4.2 BAYES HILBERT SPACE DIVERGENCE
We can now utilize this newly established mixed conjugate framework to bridge connection be- tween the induced pseudo f diverggessss and Bayes Hilbert spaces. One locally non-convex func- tion we already considered in Table is closely related the centered log- lag-ratio transformation and of special interest for the remainder of the paper. It is defined as
fans: [0. oo) R. + lz)
(5)
is easy to verify that faHS is concave on the interval [0, e0 as well as fBHS(1) = 0. We can therefore use faHs to define the pseudo f-divergence Dinns-
Let (A), then we can derive a connection between this newly constructed divergence and the Bayes Hilbert space metrie:
Lemma 4.5. For JaHS as defined in equation 5 it holds that
D Diaus (14, v) = d Po)2 + Ey (log(p e v))²
Minimizing Dians also minimizes ( (Pp. (ipppPP)) Pv), ensuring that two probability measures close with respect to Daas similarly close in the corresponding Bayes Hilbert space. This aligns in- tuitively with statistical learning, where we assume yu as the true underlying measure and aim to learn measure that closely approximates it. Consequently, i generates Bayes Hilbert space (2()) which the approximations must reside. Since densities relative to μ are rarely of primary interest, convert them more familiar densities, such as those with respect Lebesgue measures.
The mixed conjugate of saHs is given by
(2(-1 V1+z) exp exp v1+z)
xE(-1,0)
Sr (x) ( 2xpp
220

---

Published as a conference paper at ICLR 2025
Here, M [0, 00) and N 1), Applying Lemma 4.1 implies that f... fBHS and
Theorem 4.3 implies that T(x) JBHS
is and optimizer for equation Since BHs Po(z))
Pa(#)
T(x) = 2 log it follows that
(T) - Ex(f. (TEM + inf ((B,(T) Ex T)) TEC(N-)
sup TEC(M-)
= (T) -B T) juus (),
Thus, the bound in Corollary 4.4 is tight for fBHS. In summary, we have demonstrated that despite the local non-convexity of Jauss. defines a pseudo f-divergence closely related to the Bayes Hilbert space metric. A slight adaptation of the results from Nowozin et al. (2016); Nguyen et al. (210)) enables the minimization of a lower bound for D Daans with this bound being attained by the optimal estimate of T.
In the next section, we will apply these results to generative modeling, introducing a new pseudo f-GAN (Bayes Hilbert space GAN) that enables sampling from learned Bayes Hilbert space dis- tribution.
COMPUTATIONAL RESULTS
5.1
SETUP
Using the PyTorch framework (Paszke et al. 2019), we implemented the Bayes Hilbert space GAN (BHSGAN) and compared its results to traditional f-GANs, shown in Table 2 as well as Wasserstein-GAN (Arjovsky al. 2017) the MNIST (Deng) 2012) (Appendix A.2) and CI- FARI0 (Krizhevsky et al.) 2009) data sets. Furthermore. we trained the benchmarking GAN ar- chitectures approximate heavy-tailed distribution, specifically the lognormal distribution with parameters (loc = 0, shape = Ocaee = 1).
Table 2: Specification of f-GAN activation functions and Fenchel conjugates similar to Nowozin al. 2016).
Name
91
dom(ff.)
(Mixed) Conjugate ((,
Kullback-Leibbe (KL)
exp(t 1)
Reverse KL.
exp(- -v)
log( ()
Pearson
R
1 +
GAN
log(1 exp(-1))
R_
log( exp(t))
BHS
-1 + exp(v)
[-1,
2 (-11
/ + exp (-1 VT+1
In terms of implementation, our approach closely resembles Nowozin al. (2016) where the au- thors estimate generative models using variational divergence minimization (Nguyen et al.) (2110).
According to Corollary 4.4 we need to optimize equation with respect to T. which is equivalent
to optimizing
min max F(0,w) == min max (T,) Evg (S* o Tw))
(6)
Here, Pa represents the generative model taking random vector as an input and returning a sample be evaluated by the discriminatory model. The diseriminatory model Tu, returns some real-valued constant we call score. A high value for the score indicates that the discriminatory model is confident of the sample being generated by the generative model. In contrast, a low score indicates that the discriminatory model evaluated the underlying sample stem from the training data. & and w denote the parametrizations of the respective models. We optimize equation 6 by sampling from a finite training data set to estimate the underlying distribution. As underlying models for the approximation on the MNIST and synthetic heavy tailed data set, we chose basic -erwwrd neural networks identical the ones used in Nowozin et al. (0166).
8

---

Published as a conference paper at ICLR 2025
For the CIFAR-I0 dataset, we employ the DCGAN architecture introduced by Radford et al. We also utilize the same representation of the variational functions as in Nowozin et al. 2016), summarized follows: To ensure that T. respects the domain of f. represent the variational function as Tw by (x) = gj o Vi (a). Here, V. denotes the discriminatory model with no restric- tions the output, meaning Im (V,) C R while 9, is final output activation function tailored to the domain of f (see Table 2). This approach accommodates various f-divergences, particularly our newly introduced approach that allows for locally non-convex divergence--eneeraing function. In our specific case, where f() x log(x)?, we choose gf(z) -1 + exp(e), to ensure the the output lies in |-1,00 00). To train the generative adversarial network, we slightly adjust the approach in Radford et al. (2015). The specific adjustments, configurations, and algorithm can be found in the appendix.
2015
5.2 DISCUSSION OF RESULTS
To evaluate the performance of each GAN architecture in terms of estimating a heavy- tailed distribution, we estimate the shape and scale parameters of the generated distri- compared to the
Table 3: Comparison of GAN architectures with re- spect to shape and scale parameters.
bution. These estimates true parameters to assess each model's 's abil- ity to capture the distribution's 's heavy-tailed nature.
Name
shape = 0.5
scale =
BHSGAN
0.5049 + 0.0101
1.0937 + 0.0227
Wasserstein
0.4220 0.0071
1.6670 0.0127
GAN
0.1952 0.0028
1.3422 = 0.0130
Pearson
0.6181 + 0.0240
0.6863 + 0.0183
Table 3 indicates that BHSGAN approxi- mates the true shape parameter of the log-
KL
0.9951 + 0.0167
1.0126 + 0.0207
Reverse KL.
0.8949 + 0.0178
0.7376 + 0.0120
normal distribution more accurately than every other architecture. While the KI.
GAN slightly outperforms the BHS GAN in scale estimation, KI. GAN consistently overestimates the shape parameter, with an average value nearly twice the true parameter. Thus, BHSGAN emerges as a robust choice for generating heavy-tailed distributions like the lognormal distribution.
In addition to the descriptive analysis, we provide and discuss histogram and KDE estimate for each GAN architecture in Appendix A.2 which allows for a visual comparison of the different generated distributions.
However, the performance of the BHSGAN on the CI- FARI0 dataset can be assessed by comparing the Fréchet Inception Distance (FID) (Heusel et al., 2017) of gener- ated samples to different f-GAN and Wasserstein GAN architectures. Tammariss FID scores of the tested architectures.
Table 4: Mean FID Scores and two times the standard deviation for differ- ent architectures on the CIFAR10 10 data set over 100 sceds. Lower FID scores indicate better model performance.
The BHSGAN outperforms all f-GAN variants, proving its effectiveness in generating higher-quality and more di- verse images. While the WGAN achieves a marginally better FID score, the difference is negligible and unlikely to be conclusive. Moreover, our approach guarantees that the approximated and true distributions remain closely aligned in the Bayes Hilbert space.
MODEL
FID
BHSGAN
31.26 0.08
KL GAN
37.50 0.13
REVERSE KL GAN
85.27 = 0.19
PEARSON GAN
33.60 0.38
GAN
33,60 0.10
Upon inspecting Figure 2 the BHSGAN architecture stands out among the other f-GAN architectures by hav- ing the most refined picture structures. The KL, Reverse KL, and Pearson GAN produce less recognizable shapes using the same training procedure as the others.
WGAN
30.81 0.12
As shown in Table 4. the WGAN and BHSGAN architectures produce the most recognizable sam- plcs. We can therefore conclude, that the BHSGAN outperforms traditional f-GANs and is on par with the Wasserstein architecture.

---

Published as a conference paper at ICLR 2025
Figure 2: Experimental results for the CIFARIO data set: Top row, from left to right: BHSGAN, KLGAN, and Reverse KLGAN. Bottom row, from left to right: Pearson GAN, original GAN, and Wasserstein GAN. Samples were drawn from the generators trained on the CIFARIO dataset.
CONCLUSION
In this paper, presented novel framework for generalizing f-divergences, wherein proved that the iivergence-generatinn function of an f-divergence does not need to be convex on the whole functions. We then showed, that
domain, allowing for even larger class of weteerennce-geeeratie there is a direct connection between the metric induced by the Bayes Hilbert space inner product and pseudo f ddiiergeence with ascudo-divergeed triiergence-generaing generating function fnus.
This connection f-divergences and Bayes Hilbert spaces poses an interesting future direction as it circumvents one of the drawbacks of Bayes Hilbert space learning: learning from samples from distributions. Additionally, our framework enables sampling from estimated Bayes Hilbert space distributions in high dimensional settings, a feature that functional methods of estimation lack by design due to the curse of dimensionality. The framework therefore poses an interesting approach for sampling from posterior distributions Bayesian statistics, which has be addressed in detail in future research.
Since the primary goal of this publication is extend the theoretical framework of f f-divergences and bridge the gap to Bayes Hilbert spaces, we want to improve GAN performance in future work. Performance in terms of FID--cores could be improved by considering a more advanced training procedure like progressively growing GAN (Karras et al.] (201)) state-of-the-a architectures like StyleGAN (Karras et al.) (219)) and VQGAN (Esser et al. 2021). The proposed framework can also easily be implemented
addition to existing J-GAN implementations as it only requires defining the new pseudo-diie pseuuo-divergence generating function faHS and adjusting the output activation function g specific to the domain of the generator.
REPRODUCIBILITY STATEMENT
We have taken several steps ensure the reproducibility of our work. In Section we provide de- tailed explanations of the framework and model architectures, with pseudocode for the key algorithm included in A.3 Proofs of technical results can be found in A.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
J. Aitchison. The Statistical Analysis of Compositional Data. Monographs on Statistics and Applied Probability. Springer Netherlands, 1986. ISBN 9780412280603.
John Aitchison. The statistical analysis of compositional data. Journal of the Royal Statistical Society: Series B (Methodological), 44(2):139-16 1982.
Yoshiyuki Arata. A functional linear regression model in the space of probability density functions. Technical report, Research Institute of Economy, Trade and Industry (RIETI), 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp.
PMLR, 2017.
Jeremiah Birrell, Paul Dupuis, Markos A Katsoulakis, Yannis Pantazis, and Luc Rey-Bellet. (f, diverrenees: Interpolating between f-divergences and integral probability metrics. Jour- nal of machine learning research, 23(39):1-70, 2022.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisti- cians. Journal of the American statistical Association, 11((518):859-88 877, 2017.
Trevor Campbell and Boyan Beronov. Sparse variational inference: Bayesian coresets from seratch. Advances in Neural Information Processing Systems, 32, 2019.
Trevor Campbell and Tamara Broderick. Bayesian coreset construction via greedy iterative geodesic ascent. In International Conference on Machine Learning, pp. 698-706. PMLR, 2018.
Ziyu Chen, Hyemin Gu, Markos A Katsoulakis, Luc Rey-Bellet, and Wei Zhu. Learning heavy-tailed distributions with wasserstein- proximal-reggularized a-divergences. arXiv preprint arXiv:2405.1396 2024.
Imre Csiszár, Paul C Shields, et al. Information theory and statistics: A tutorial. Foundations and Trends@ in Communications and Information Theory, 144):117552, 2004.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 2966):141-142, 2012.
J. J. Egozcue, J. L. Díaz-Barrero and V. Pawlowsky-Glahn Hilbert space of probability density functions based on aitchison geometry. Acta Mathematica Sinica, 22(4): Jul 2006. ISSN 7617. doi: 10.1007/s101 14-005-0678-2.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and pattern recogni- fion, pp. 12873-12883, 2021.
Izrail Moiseevitch Gelfand, Richard A Silverman, al. Calculus of variations. Courier Corporation, 2000.
Jaime Roquero Gimenez and James Zou. A unified f-divergence framework generalizing vae and gan. arXiv preprint arXiv:2205.05211 052144, 2022.
Jean-Pierre Gossez, EJ Lami Dozo, Jean Mawhin, and Lucien Waelbroeck. Nonlinear Operators and the Calculus of Variations: Summer School Held in Bruxelles, 8-9 September 1975, volume 543. Springer, 2006.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational infer- ence. Journal of Machine Learning Research, 2013.

---

Published as a conference paper at ICLR 2025
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint 0rXi::1710.11196,2011
Tero Karras, Samuli Laine, and Timo Aila. A ayylebased generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Amor Keziou. Dual representation of --divergences and applications. mathématique, 336 336(10):577862, 2003.
Comptes rendus
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
URL
http::/www... atoronto.ca/
learning features
featturee-eeuuuesseerr centures-2009-- 2009- TR. pdE
Dirk Kroese, Thomas Taimre, and Zdravko I Botev. Handbook of monte carlo methods. John
Wiley & Sons, 2013.
Friedrich Liese and Igor Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10))4394 4412, 2006.
Jitka Machalova, Karel Hron, and Gianna Serafina Monti. Preprocessing of centred logratio trans- formed density functions using smoothing splines. Journal of Applied Statistics, 4418):1419- 1435, 2016.
Jitka Machalová, Renáta Talská, Karel Hron, and Aleš Gába. Compositional splines for representa- tion of density functions. Computational Statistics, 36:1031 1034-1044, 2021.
Eva-Maria Maier, Almond Stocker, Bernd Fitzenberger, and Sonja Greven. Additive density-on-
scalar regression in bayes hilbert spaces with an application gender economies. 2021.
Kevin R Moon and Alfred O Hero. Ensemble estimation of multivariate f-divergence. In 2014 IEEE International Symposium on Information Theory, pp. 356-360. IEEE, 2014.
XuanLong Nguyen, Martin Wainwright, and Michael I Jordan. On surrogate loss functions and f-divergences. 2009.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11))5847-553 5861, 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
Nikolas Nüsken and Lorenz Richter. Solving high-dimensional hamillon-jacobi--elllnn pdes using neural networks: perspectives from the theory controlled diffusions and measures on path space. Partial Differential Equations and Applications, 2(4):48, Jun 2021. ISSN 2662-2971. doi: (0077//42985-021-000000 s42985- 02005--221-0000-200
URL /tctps://dc..ogg// 1007/
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06433 2015.
Lorenz Richter and Julius Berner. Improved sampling via learned diffusions. In The Twelfth In- ternational Conference on Learning Representations, 2024. URL http::::///// //tp:///////ee/ieee/ net het:/forum2ioonh4MMooos
12

---

Published as a conference paper at ICLR 2025
Lorenz Richter, Ayman Boustati, Nikolas Nüsken, Francisco J. R. Ruiz, and Ömer Deniz Akyildiz. Vargrad: low-variance gradient estimator for variational inference. In Proceedings of the 34th International Conference Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Ralph Tyrell Rockafellar. Convex Analysis. Princeton University Press, 1970.
Paul Rubenstein, Olivier Bousquet, Josip Djolonga, Carlos Riquelme, and Ilya O Tolstikhin. Prac- tical and consistent estimation of f-divergences. Advances in Neural Information Processing Systems, 32, 2019.
Bharath K. Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert R. G. Lanckriet. On the empirical estimation of integral probability metrics. Electronic Journal of Statistics, 6:1550 1599, 2012. doi:
R. Talská, A. Menafoglio, J. Machalová, K. Hron, and E. Fišerová. Compositional regression with functional response. Computational Statistics & Data Analysis, 123:66-85, 2018. ISSN 0167- 9473. doi: https://doi.org/10 10..o/ooorr/01.01/......... 1016/j.csda.201866 2018.0...............
Van den Boogaart, Juan José Egozeue, and Vera Pawlowsky-Glahn Bayes linear spaces. SORT: statistics and operations research transactions, 2010, vol. 34, núm. 4, p. 201-222, 2010.
K Gerald Van den Boogaart and Raimon Tolosana-eelgado Analyzing compositional data with R, volume 122. Springer, 2013.
Karl Gerald Van den Boogaart, Juan José Egozcue, and Vera Pawlowsky-Glahn. Bayes hilbert spaces. Australian New Zealand Journal of Statistics, 16(2):171-194, 2014.
Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions Information Theory, 6007):3777-3800, 2014.
George Wynne. Bayes hilbert spaces for posterior approximation. arXiv preprint 20000:04093 2023.
Yannis G Yatracos. Limitations of the wasserstein mde for univariate data. Statistics and Computing, 32(6):95, 2022.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX / SUPPLEMENTAL MATERIAL
A. PROOFS OF THEORETICAL RESULTS
In this section, we provide proofs of several results we introduced throughout the publication.
While the following Lemma does not concern BHSGANs directly, we noticed that this connection has not yet been drawn in previous publications.
Proof of Lemma 3.1
(Pn. = (pa θ Pv. O po)
(clr(pn @ pu), clr(p @
(og(pa(>) e (pt)) ( (ogginll))@ppee)eCM
Jn
Varx(1og(14 v))
The proof of Lemma 4.1 is straightforward but shows nevertheless the importance the restriction to functions that are unbounded the convex part of the domain.
Proof of Lemma 4.1
1. (a) Lett" M. Then & = a(t*) argmax (xt* - f(x)) [o). Since & E |a, oo), =Edom(f)
the function x + xt" - f(x) is concave in à and monotonically decreasing for any f. Thus, lim zt f(x) -o0, i.e., argmin zt f(x) [a, 0o) implying
rEdom(f)
that & N. Analogously, 1, E N = 1, M.
(b) In order to show that M U N = dom(f:), let E dom(f"). Then, there exists a Zi such that (1) xıी - (*z).. Since x; E domy, either zy (0,a) or (a, oo). As was chosen arbitrarily, M U N dom(S.) holds.
2. Since M and N are disjoint,
write
f = sup (tr (z)) + inf {tx - 1(())) (1EN-) sEN
#eM
sup (tx f"(x)) + inf (tx - ((ENN)) sEN
3EM
[LEM + (IEN)
f.
In Lemma 4.2 we restrict ourselves to the class of functions that have nonnegative domain and range. The reason for this restriction is simply ensure that the introduced measure of dissimilarity respects homogeneity.
Proof of Lemma 4.2 Let U, E (2).). Then,
Pn(x))
Dj(m,u)=
(c()
A(dx)
(c)
P=(x))
(a(x))
A(dx)
Pr (x)
20
20
14

---

Published as a conference paper at ICLR 2025
and
Dy(mu)= 1 ( (Pu(T) (p(x))
v(dx)) - ==v A-as.
we have to resort to a more technical argument involving the fun- damental lemma of the calculus of variations ensure that we can indeed derive an analytical representation for the optimizer of equation 3
In order to prove Theorem 4.3
Proof of Theorem 4.3 order optimize equation we have maximize
(E, - E5 T)) ((rec(M))
and minimize
(E) (ft T)) (TEC(N)))
with respect T E (()om( Consider the term
(T) - -f T))
can be rewritten as
T(r)u(de) - 1o 00 ( (T))v(d))))
(T) (x o T)
T(x)pu(2) (ft (=)
=E(TT)
Then, for an optimizer of equation
E(T + +U) 1
OE ==
with U Cg is necessary condition for an extremum (Gelfand al. 2000
Theorem 4.1).
Now,
E(T +=U)= f/- [//- [(T+))()))))))) ft (T (U) ())
and
de ( 000
E(T + =U))
((dx)
0=
=
- UUb-4e---CCCYT))+-UUU))CChchhhc-
I -
Applying the fundamental lemma of the calculus of variations Gelfand al. 2000 Lemma 1.1) to
yields equation 7
(ttpaacc)-C(YY((()))))ccccc.)la =0 = pm(x) - (f:) = a.e..
Thus,
T(x) = (J:')= a
Pe Pu
(x)
15

---

Published as a conference paper at ICLR 2025
candidate for optimizer. Note, that if (ff")=1 is well-defined, then (ft")-((x) f((x) (ef. Rockafellar (1970)). For the second derivative, we have
8²
E(T + +U)==
-(TY(( (ddr).
2=2
==
20
Depending on the properties of f the candidate
T(x) = (F:')
(x)
Pu
thus maximizer or minimizer. The restriction in the maximization part of equation enforcess that the optimizer 7 is convex, implying that
(J:Y'
(5:')~ -1 Pe
(x)
Pv
and therefore
8²
E(T + =U) <
0=2
==0
Conversely, to optimize the minimization part of equation cave part of fr, implying that
consider the restriction to the
(1:Y
(1+T) -1 Pμ
(x) so
Pv
and therefore
8²
E(T + =U) 0.
0=²
==0
plays an important role when considering the computational perspective of equation Since we wanted to avoid optimizing a min max and min min game simultancously, we derived lower bound for the pseudo divergence that can be solved as simple min max game instead. The core idea is to rewrite the infimum as supremum and combine both suprema to obtain a min max game framework.
Corollary 4.4
Proof of Corollary
4.4
Jn ff( Pulx)
Dy(1v))
M(dz)
(x(
sup
.Pu(x)
1:(0) ((dx)
I inf {
(pPult)
fi(()) X(dx)
(8)
(o(x)
EEN Po(x)
EE((foo7] 77)
( m0 A(dx)
2 sup TEC(M*)
[T)( - -Eo( o
+ inf TEC(N))
(9)
Po(x)
sup TEC(M))
(E,((T Ex(F. o TT)
sup TEC(N*)
(E -E o off TT)
(10)
Eu((TL(reecuun [[TEC(N=))))-
sup TEC(dom(ft))
Ex o oT -
ET( - Ex T)) }
sup TEC(M))
(11)
where C(M') (C(N)) denotes the set of continuous functions such that ft oT' convex (concave). The second equality, equation g can be derived by applying the definition of the mixed conjugate.
16

---

Published as a conference paper at ICLR 2025
equation 9 derived in the same way as equation 1 and equation 10 application of Gossez et al. (2006) where we used that - inf = sup. In equation 11 we used the fact, that C(MP))C(N')) Defming T T( (TeC(M))) L ((TEC(N))) yields the desired result.
Proof of Lemma 4.5
/ (v(x) Pu(a)
(a(tt)
Dians (1, v)
log
A(dx)
12
Do
Pa(ar)
/ Pa(x)) log
Pulx))
X(dz)
px(2)
E (log(u 2 )))
dBB(w) (pp: po) + E, (log(u v)))
A.2 ADDITIONAL EXPERIMENTS
In this section we present generated MNIST digits as well as histograms and KDEs for an estimated lognormal distribution for each benchmarking GAN architecture.
The findings of Table 3 can be validated further by considering Figure 3 BHS GAN approximates
the true density well regarding mode and tail behavior. In comparison, the Wasserstein architecture fails to capture the left tail behavior completely, and the right tail shows some discrepancies in terms of monotonicity. While the KL GAN captures the true distributions' scale relatively well, the right tail seems to be much heavier than the tail of the true underlying distribution. Pearson GAN and Reverse KL GAN struggle with oscillations and deviations from the true density, especially in the tails. The basic GAN performs poorly in terms of alignment, with significant underestimation near the mode and noisy approximation in the tails.
The generated MNIST samples in Figure 4 visibly depict digits between O and 9, indicating that all GAN architectures learned the underlying distribution of the images well. Therefore we can conclude that the BHSGAN is par with the other GAN architecture for the MNIST dataset. We limit ourselves to this high-level evaluation as a quantitative comparison for generative models on this kind of data is not meaningful (Nowozin et al. 2016).
A.3 PRACTICAL CONSIDERATION AND ALGORIHTM
During the experiments we found that training (pseudo) f-GANs is highly unstable, suffering from exploding and diminishing losses. To stabilize the training process we implemented two methods. Firstly, we introduced gradient penalty identical to the one used for WGANs (Gulrajani et al.) 2017) for all used GANs except the vanilla and reverse KL-GAN. Additionally, contrary to the popular approach of training the discriminatory model first, we decided to start by training the gen- erative model first. We interpret this finding as (prior) practical drawback of f-GANs, in the way that in theory, the supremum has to be found over all continuous functions, but in practice, this class of functions is too general to be learned during training. This means that the discriminator is allowed to be too flexible it starts "remembering" real images before the generator good fake ones, resulting in exploding losses for the discriminator and vanishing ones for the gen- erator. Accordingly, forcing the discriminator to Lipschizz-contiuouus constrains the supremum and infimum in 3 to be found over all Lipschitz-continuuu functions, which stabilizes the training process by hindering the discriminator from remembering real images too fast.
generate
All experiments were conducted on an AORUS RIX 4090 eGPU 24GB.
17

---

Published as a conference paper at ICLR 2025
listogaae KD sogramaa Dondyy with shapord. and scalent Appensimated using BHIGAM
1:30
02
<0
0.00
Heroorran Lagnoeral Donatyy thopemt
Halogra Logramaal
00
15
000
60
.etogom SOE of Logiormnl Coretry with stapers and acalemt Agaratrmmt using GMW
Hittgpoo and Ltgogoomss Caruty with stopert Land scalons Approal ssing Revars XL
60
Figure 3: Histograms and KDE estimates for different GAN architectures
18

---

Published as a conference paper at ICLR 2025
8
7
5
3
8
3
4
3
9
4
2
0
2
7
2
8
4
7
1
1
4
2
2
3
3
5
5
8
3
+
6
5
9
2
O
9
1
1
7
8
3
9
5
6
/
6
/
4 2
K
&
8
9
4
/
q
8
4
3
2
4
/
y
9 A
/
/
5
/
4
3
9
8
I
7
0
2
3
S
3
O
4
9
7
7
5
3
3
7
7
9
9
/
0
6
4
1
4
5
9
8
8
/
4
4
5
/
7
5
6
1
2
1
3
0
8
7
/
q
0
8
0
0
0 9 3
7
2
6
7
8
Figure 4: Experimental results for MNIST datasets: Left column, from top to bottom: BHSGAN, KLGAN, and Reverse KLGAN. Right column, from top bottom: Pearson GAN, original GAN, and Wasserstein GAN. Samples were drawn from the generators trained on the MNIST dataset.
Algorithm Default values for the training procedure in this paper were og at 0.0002 for all GANs except BHSGAN and REVKLGAN, where = Oid 0.00005, nı 2, = 2. m 128 for MNIST and m 64 for CIFAR, Nepochs 50, d2 100 and À 10 except for PEARSONGAN, where À = 20
Input: Og. learning rate of generative model. ag, learning rate of discriminatory model. nd. number of discriminator updates. Ng. number of generator updates. m, batch size. Nepochs> number of epochs. d,, dimension of noise, X, gradient penalty coefficient, gp. gradient penalty, wo, initial parameters of discriminatory model dg, initial parameters of generative model.
for 1,..., N Nepochs do
for each batch = (x(1), .... x (m)) in training set do
noise = (2( ..., (m)) vid. No0,, Id,xd,)
real sample = (1) x (m))
for 1,... ng do
Lv, m Σn iim ft ooggon Vu vo((0)
i = 9 Adam ag. 1. Lrn)
end for
for 1 ..... Mi do
Lv m Em
(s: og Vo ug(2()))- gr Vu((i))))) + +h. gp
W = w + Ad Lv,.
end for end for
end for
19


Reviews:
[{"review_id": "xMBOBy19Dr", "reviewer": "Reviewer_ScwD", "summary": "The paper studies a more general class of f-divergences that does not need to be convex on the whole domain and can be concave on the interval of $[0, a)$ and convex on $[a, \\infty)$, called pseudo-divergence-generating function. The authors also link the pseudo f-divergence with the metric induced by the Bayes Hilbert space. The paper uses the proposed framework to extend f-GANs to include their locally non-convex divergence function, showing comparable results with older methods.", "questions": "The authors say that their framework helps with the high-dimensional sampling for Bayes Hilbert space distributions. I am not sure if I could follow how the proposed framework solves this problem. I would appreciate if the authors could elaborate on that.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The idea of generalizing the convex divergence functions to locally non-convex is interesting and is a step towards a more general family of divergences. The paper is well-motivated and the connection to Bayes Hilbert space (BHS) is neat. I enjoyed this connection as it helps with the challenge of estimating the distributions using some samples (something that BHS is not good at while the f-divergences use optimization tools and are more flexible).", "weaknesses": "While the idea is interesting, the experimental setup does not show a big advancement in having locally concave functions, and one can see that WGAN is still outperforming the proposed method. Did the authors try to have some synthetic example that clearly shows the benefit of the pseudo f-divergence family?\n\nOn another note, I think the paper's writing could improve, particularly some details could help the reader. For example:\n- Line 201, Equation 1 refers to a wrong equation, I believe the authors wanted to point to the equation line 196.\n- Line 221, is the constant c, dependent on A? If yes, maybe the authors could specify that by writing $c_A$.\n- Jumping from Eq. 2 to Corollary 4.4's result could seem surprising as it seems to be a jump from sup and inf to only sup. Checking the appendix, I believe explaining the relation between $T$ and $\\tilde{T}$ would clarify this.\n- Double Equation in line 431\n- The images of Fig 2(a) are barely distinguishable. It would help if the authors could change the size of the image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qhD8pEvp7A", "reviewer": "Reviewer_zX4e", "summary": "The paper proposes a new way to generalize $f$-divergence where the divergence-generating function may not be convex on the whole domain, which they coined as pseudo $f$-divergence. Furthermore, the authors provide a nice connection between the introduced pseudo $f$-divergence and the metric induced by the Bayes Hilbert Space. An illustration of using pseudo $f$-divergence in Bayes Hilbert space GAN demonstrates the potential usefulness of such a generalization.", "questions": "Thanks for the interesting work. Two questions I have are\n\n1) I wonder if the authors could clarify how the pseudo-divergence could yield a new sampling approach at a high level. I understand that these could serve as an alternative approach to measuring the difference between distributions, but it's not clear to me if any sampling procedures can be based on them. Perhaps the more natural way to utilize this result would be variational inference, but I am not sure about the sampling strategies, so any clarification would help.\n\n2) Could authors perhaps include any concrete example where the generalization of $f$-divergence significantly improved or robustified the performance of learning? Or at least, an extensive numerical comparison with the existing divergences to further gain insights (even for simple tasks).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The presentation of the submitted manuscript is very clear and its goals are well articulated.\n\n2) Generalization of $f$-divergence and its close connection to Bayes Hilbert Space provides a nice insight as well as practically useful theory to extend existing Bayesian inferences, or probabilistic machine learning tasks.\n\n3) Illustration of introduced methods in GAN further provides soundfulness of the established results.", "weaknesses": "No major weakness. Minor weakness below:\n\n1) For the sake of improving the presentation, I would suggest reducing Section 1 and include more details about Bayes Hilbert Space in Section 3—for instance, the paragraph before 1.1 contributions (lines 64-72), the Second paragraph in Bayes Hilbert Spaces part in Section 1(lines 124-131), the second paragraph of section 1.3 (lines 156-160) all substantially overlap. One could summarize three paragraphs into one and just give a high-level depiction in the introduction and provide more details in Section 3 on Bayes Hilbert Space.\n\n2) Many prepositions, or phrases missing in front of references. For instance, in lines 128-129, it is better to write \"including\" before you list the references. Or perhaps include references inside the brackets [ ].\n\n3) In the Fenchel conjugate definition, I believe it should be x^Ty, not just xy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9OuJTeUYEM", "reviewer": "Reviewer_Tns5", "summary": "This work proposes novel connections between Bayes Hilbert spaces and a modified version of $f$-divergences (namely pseudo $f$-divergences) requiring $f$ to be concave on $[0,a)$ and convex on $[a,+\\infty), a>0$ instead of convexity on all $[0,+\\infty)$. More precisely. \n\n- Basic results and definition for both Bayes Hilbert spaces and $f$-divergences are recalled in Sections 2 and 3.\n- Extensions of classical results of $f$-divergences for pseudo $f$-divergences are proposed in Section 4.1 \n- Authors unveil a connection between a specific pseudo $f$-divergence and a Bayes Hilbert Space\n- Experiments on GANs are proposed in Section 5, showing the pseudo $f$-divergence approach on a certain learning problem outperforms the classical $f$-GAN approach and matches the performance of Wasserstein GANs", "questions": "- Can you precise why the addition, multiplication and substraction in the Bayes Hilbert is well-defined only if we consider $B(\\lambda)$ and not $\\mathcal{M}(\\lambda)$ ?\n- You said in the abstract that your framework 'generalizes f -divergences by incorporating locally non-convex divergence-generating functions'. This should be clarified as, to my understanding of Section 4, you only consider a certain type of local non-convexity, being functions concave on $[0,a)$ and convex on $[a,+\\infty)$? Do you think it is possible to consider functions $f$ with several areas of non-convexity, eg  a function convex on $[0,a], [b,c], [d,+\\infty)$ and concave on $(a,b), (c,d)$?\n- Experiments: can you please detail the nature of your experiment? It is not clear to understand what you are doing and why you are doing it?\n- You mentioned in your intro (l.104) that in case of heavy-tail distribution, Wasserstein distance is unsuitable to modeling distributions with heavy-tails. Given that your experiments shows similar performances with WGAN, it seems that you do not consider heavy-tailed distributions, could you think of an experiment (even a toy one) with such a heavy-tailed measure where the WGAN would fail and your approach succeed?\n- l 431 typo for 'equation'\n- I find the nature of your theoretical contribution unclear: you claim that you are bridging the gap between Bayes Hilbert space and $f$-divergences. However, you are not creating such a link as you focus only on pseudo divergences, with $f$ concave on $[0,a], a>0$ and convex on $[a,+\\infty]$ with $a>0$. As this does not encompass the case $a=0$ (ie the case of a true $f$-divergence, if we add a lower semicontinuous assumption), I don't know whether:\n\t1. the case $a=0$ has already been dealt with in the literature, then it would be good to precise such links more clearly and that your contribution is an extension of those links to the case of pseudo-divergences\n\t2. The case $a=0$ has not been previously considered, but you were only able to design theoretical links for pseudo $f$-divergences. In this case you are not truly reaching $f$-divergences\n\tIn both cases, claiming that you are bridging the gap between Bayes Hilbert space and $f$-divergences seems inaccurate.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The three first sections are well-written and pedagogical.\n- The theoretical connection of Section 4.2 looks strong and promising.", "weaknesses": "- The precise nature of the contribution would gain to be clearer\n- The experimental section is poorly written and does not exploit the flexibility provided by the proposed theoretical framework.\n\nPlease see the Questions for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xXbAl8YWHj", "reviewer": "Reviewer_Y69u", "summary": "This paper introduces a new loss for GANs, which builds on a connection between so-called Bayes Hilbert spaces and pseudo f divergences. First, Bayes Hilbert space are introduced, which is a linear space with additive and multiplicative structure coherent with Bayes theorem. Then, pseudo f divergences, which do not require f to be convex everywhere are introduced. The main part is that for a certain choice of f. the pseudo f divergence can be written as a sum of the BHS norm and some other 'entropy' term. Then, it is verified numerically, how well this certain choice of GAN performs in image generation.", "questions": "1) Can the connection between Bayes Hilbert spaces and f divergences be leveraged for other generative models? (GANs are not in fashion anymore, due to well-known stability, mode collapse and interpretability reasons). Many generative models have a loss that can be bounded via the KL. I guess this goes in the direction of Richter et al.?\n\n2) Bayes Hilbert spaces are introduced as they adhere to the structure of Bayes theorem, i.e., the likelihood is the radon nikodym derivative of posterior with prior. Can you somehow use these ideas to reformulate the distance between posterior and generation measure in terms of BHS losses?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper deals with a niche topic, but does it so very nicely. The definitions and theorems are perfectly clear, although I did not check the proofs. The connection between pseudo f divergences and Bayes Hilbert spaces is very interesting. The paper comes with code and seems reproducible. It is also nice that the paper rediscovers the log variance loss used in Richter et al and motivates it more theoretically.", "weaknesses": "1) My biggest problem apart from some nitpicks is that I do not really see the point of all this formalism. It is nice that there is an intrinsic connection between pseudo f divergences and Bayes Hilbert spaces, but there are no additional insights based on that. One could just introduce pseudo f divergences and use the specific f. One thing that seems a bit odd and might yield additional insights is that in Lemma 4.5, there is an additional entropy like term. Does this give any other interpretation?\n\n2) Numerically, the results are okayish, mostly due to the use of the GAN algorithm. Unfortunately GANs suffer from many problems: unstable training, mode collapse,... Did you find if your loss alleviates some of these problems? \n\n3) The interpretation of the numerical results is also a bit bold: The difference between WGAN and BHS-GAN \"is on a par\", while the BHSGAN \"outperforms traditional f-GANs\". Looking at the table, f-GANs (Pearson) are 2 FID away from BHS-GAN, so I would say there is not really anything conclusive about outperforming. In particular, the table states that the results are \"over 100 seeds\", which i interpret as you draw with one trained model a lot of images and calculate the FID and average this 100 times? Then, to say anything conclusive, one should try to get different training runs. However, I see this paper more of a theory paper and therefore I would suggest to tone the claims a bit down.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)