Bridging the Gap Between f-divergences and Bayes Hilbert Spaces

Authors: Linus Lach, Alexander Willi Fottner, Yarema Okhrin
Primary area: learning theory
Keywords: f-divergences, Bayes Hilbert spaces, fenchel conjugates, centered-log-ratio, posterior approximation

Abstract:
We introduce a novel framework that generalizes $f$-divergences by incorporating locally non-convex divergence-generating functions.
Using this extension, we define a new class of pseudo $f$-divergences, encompassing a wider range of distances between distributions that traditional $f$-divergences cannot capture.
Among these, we focus on a particular pseudo divergence obtained by considering the induced metric of Bayes Hilbert spaces.
Bayes Hilbert spaces are frequently used due to their inherent connection to Bayes's theorem. They allow sampling from potentially intractable posterior densities, which has remained challenging until now.
In the more general context, we prove that pseudo $f$-divergences are well-defined and introduce a variational estimation framework that can be used in a statistical learning context.
By applying this variational estimation framework to $f$-GANs, we achieve improved FID scores over existing $f$-GAN architectures and competitive results with the Wasserstein GAN, highlighting its potential for both theoretical research and practical applications in learning theory.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1729446618606

thread_type:
official_review

<thread_object>
creator: reviewer_y69u

<content>
summary:
This paper introduces a new loss for GANs, which builds on a connection between so-called Bayes Hilbert spaces and pseudo f divergences. First, Bayes Hilbert space are introduced, which is a linear space with additive and multiplicative structure coherent with Bayes theorem. Then, pseudo f divergences, which do not require f to be convex everywhere are introduced. The main part is that for a certain choice of f. the pseudo f divergence can be written as a sum of the BHS norm and some other 'entropy' term. Then, it is verified numerically, how well this certain choice of GAN performs in image generation.

strengths:
The paper deals with a niche topic, but does it so very nicely. The definitions and theorems are perfectly clear, although I did not check the proofs. The connection between pseudo f divergences and Bayes Hilbert spaces is very interesting. The paper comes with code and seems reproducible. It is also nice that the paper rediscovers the log variance loss used in Richter et al and motivates it more theoretically.

weaknesses:
1) My biggest problem apart from some nitpicks is that I do not really see the point of all this formalism. It is nice that there is an intrinsic connection between pseudo f divergences and Bayes Hilbert spaces, but there are no additional insights based on that. One could just introduce pseudo f divergences and use the specific f. One thing that seems a bit odd and might yield additional insights is that in Lemma 4.5, there is an additional entropy like term. Does this give any other interpretation?

2) Numerically, the results are okayish, mostly due to the use of the GAN algorithm. Unfortunately GANs suffer from many problems: unstable training, mode collapse,... Did you find if your loss alleviates some of these problems? 

3) The interpretation of the numerical results is also a bit bold: The difference between WGAN and BHS-GAN "is on a par", while the BHSGAN "outperforms traditional f-GANs". Looking at the table, f-GANs (Pearson) are 2 FID away from BHS-GAN, so I would say there is not really anything conclusive about outperforming. In particular, the table states that the results are "over 100 seeds", which i interpret as you draw with one trained model a lot of images and calculate the FID and average this 100 times? Then, to say anything conclusive, one should try to get different training runs. However, I see this paper more of a theory paper and therefore I would suggest to tone the claims a bit down.

questions:
1) Can the connection between Bayes Hilbert spaces and f divergences be leveraged for other generative models? (GANs are not in fashion anymore, due to well-known stability, mode collapse and interpretability reasons). Many generative models have a loss that can be bounded via the KL. I guess this goes in the direction of Richter et al.?

2) Bayes Hilbert spaces are introduced as they adhere to the structure of Bayes theorem, i.e., the likelihood is the radon nikodym derivative of posterior with prior. Can you somehow use these ideas to reformulate the distance between posterior and generation measure in terms of BHS losses?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730115009258

thread_type:
official_review

<thread_object>
creator: reviewer_tns5

<content>
summary:
This work proposes novel connections between Bayes Hilbert spaces and a modified version of $f$-divergences (namely pseudo $f$-divergences) requiring $f$ to be concave on $[0,a)$ and convex on $[a,+\infty), a>0$ instead of convexity on all $[0,+\infty)$. More precisely. 

- Basic results and definition for both Bayes Hilbert spaces and $f$-divergences are recalled in Sections 2 and 3.
- Extensions of classical results of $f$-divergences for pseudo $f$-divergences are proposed in Section 4.1 
- Authors unveil a connection between a specific pseudo $f$-divergence and a Bayes Hilbert Space
- Experiments on GANs are proposed in Section 5, showing the pseudo $f$-divergence approach on a certain learning problem outperforms the classical $f$-GAN approach and matches the performance of Wasserstein GANs

strengths:
- The three first sections are well-written and pedagogical.
- The theoretical connection of Section 4.2 looks strong and promising.

weaknesses:
- The precise nature of the contribution would gain to be clearer
- The experimental section is poorly written and does not exploit the flexibility provided by the proposed theoretical framework.

Please see the Questions for more details.

questions:
- Can you precise why the addition, multiplication and substraction in the Bayes Hilbert is well-defined only if we consider $B(\lambda)$ and not $\mathcal{M}(\lambda)$ ?
- You said in the abstract that your framework 'generalizes f -divergences by incorporating locally non-convex divergence-generating functions'. This should be clarified as, to my understanding of Section 4, you only consider a certain type of local non-convexity, being functions concave on $[0,a)$ and convex on $[a,+\infty)$? Do you think it is possible to consider functions $f$ with several areas of non-convexity, eg  a function convex on $[0,a], [b,c], [d,+\infty)$ and concave on $(a,b), (c,d)$?
- Experiments: can you please detail the nature of your experiment? It is not clear to understand what you are doing and why you are doing it?
- You mentioned in your intro (l.104) that in case of heavy-tail distribution, Wasserstein distance is unsuitable to modeling distributions with heavy-tails. Given that your experiments shows similar performances with WGAN, it seems that you do not consider heavy-tailed distributions, could you think of an experiment (even a toy one) with such a heavy-tailed measure where the WGAN would fail and your approach succeed?
- l 431 typo for 'equation'
- I find the nature of your theoretical contribution unclear: you claim that you are bridging the gap between Bayes Hilbert space and $f$-divergences. However, you are not creating such a link as you focus only on pseudo divergences, with $f$ concave on $[0,a], a>0$ and convex on $[a,+\infty]$ with $a>0$. As this does not encompass the case $a=0$ (ie the case of a true $f$-divergence, if we add a lower semicontinuous assumption), I don't know whether:
	1. the case $a=0$ has already been dealt with in the literature, then it would be good to precise such links more clearly and that your contribution is an extension of those links to the case of pseudo-divergences
	2. The case $a=0$ has not been previously considered, but you were only able to design theoretical links for pseudo $f$-divergences. In this case you are not truly reaching $f$-divergences
	In both cases, claiming that you are bridging the gap between Bayes Hilbert space and $f$-divergences seems inaccurate.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730151581033

thread_type:
official_review

<thread_object>
creator: reviewer_zx4e

<content>
summary:
The paper proposes a new way to generalize $f$-divergence where the divergence-generating function may not be convex on the whole domain, which they coined as pseudo $f$-divergence. Furthermore, the authors provide a nice connection between the introduced pseudo $f$-divergence and the metric induced by the Bayes Hilbert Space. An illustration of using pseudo $f$-divergence in Bayes Hilbert space GAN demonstrates the potential usefulness of such a generalization.

strengths:
1) The presentation of the submitted manuscript is very clear and its goals are well articulated.

2) Generalization of $f$-divergence and its close connection to Bayes Hilbert Space provides a nice insight as well as practically useful theory to extend existing Bayesian inferences, or probabilistic machine learning tasks.

3) Illustration of introduced methods in GAN further provides soundfulness of the established results.

weaknesses:
No major weakness. Minor weakness below:

1) For the sake of improving the presentation, I would suggest reducing Section 1 and include more details about Bayes Hilbert Space in Section 3—for instance, the paragraph before 1.1 contributions (lines 64-72), the Second paragraph in Bayes Hilbert Spaces part in Section 1(lines 124-131), the second paragraph of section 1.3 (lines 156-160) all substantially overlap. One could summarize three paragraphs into one and just give a high-level depiction in the introduction and provide more details in Section 3 on Bayes Hilbert Space.

2) Many prepositions, or phrases missing in front of references. For instance, in lines 128-129, it is better to write "including" before you list the references. Or perhaps include references inside the brackets [ ].

3) In the Fenchel conjugate definition, I believe it should be x^Ty, not just xy.

questions:
Thanks for the interesting work. Two questions I have are

1) I wonder if the authors could clarify how the pseudo-divergence could yield a new sampling approach at a high level. I understand that these could serve as an alternative approach to measuring the difference between distributions, but it's not clear to me if any sampling procedures can be based on them. Perhaps the more natural way to utilize this result would be variational inference, but I am not sure about the sampling strategies, so any clarification would help.

2) Could authors perhaps include any concrete example where the generalization of $f$-divergence significantly improved or robustified the performance of learning? Or at least, an extensive numerical comparison with the existing divergences to further gain insights (even for simple tasks).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730422735783

thread_type:
official_review

<thread_object>
creator: reviewer_scwd

<content>
summary:
The paper studies a more general class of f-divergences that does not need to be convex on the whole domain and can be concave on the interval of $[0, a)$ and convex on $[a, \infty)$, called pseudo-divergence-generating function. The authors also link the pseudo f-divergence with the metric induced by the Bayes Hilbert space. The paper uses the proposed framework to extend f-GANs to include their locally non-convex divergence function, showing comparable results with older methods.

strengths:
The idea of generalizing the convex divergence functions to locally non-convex is interesting and is a step towards a more general family of divergences. The paper is well-motivated and the connection to Bayes Hilbert space (BHS) is neat. I enjoyed this connection as it helps with the challenge of estimating the distributions using some samples (something that BHS is not good at while the f-divergences use optimization tools and are more flexible).

weaknesses:
While the idea is interesting, the experimental setup does not show a big advancement in having locally concave functions, and one can see that WGAN is still outperforming the proposed method. Did the authors try to have some synthetic example that clearly shows the benefit of the pseudo f-divergence family?

On another note, I think the paper's writing could improve, particularly some details could help the reader. For example:
- Line 201, Equation 1 refers to a wrong equation, I believe the authors wanted to point to the equation line 196.
- Line 221, is the constant c, dependent on A? If yes, maybe the authors could specify that by writing $c_A$.
- Jumping from Eq. 2 to Corollary 4.4's result could seem surprising as it seems to be a jump from sup and inf to only sup. Checking the appendix, I believe explaining the relation between $T$ and $\tilde{T}$ would clarify this.
- Double Equation in line 431
- The images of Fig 2(a) are barely distinguishable. It would help if the authors could change the size of the image.

questions:
The authors say that their framework helps with the high-dimensional sampling for Bayes Hilbert space distributions. I am not sure if I could follow how the proposed framework solves this problem. I would appreciate if the authors could elaborate on that.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733220606744

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response by Authors

comment:
We thank the reviewers for carefully reading our paper, providing positive feedback, and suggesting minor revisions that improved our paper's presentation and soundness. 
In the following paragraphs, we summarize revisions of the individual reviewers and the changes we subsequently incorporated into our manuscript.

## Experimental Section

Reviewers *ScwD, zX4e, Tns5,* and *Y69u* all pointed out that the experiments showcasing applications of the newly proposed $BHS$-divergence should be improved by providing additional experiments.
(Pseudo) $f$-divergences in a GAN context outperform Wasserstein GANs in approximating heavy-tailed distributions. 
We, therefore, enriched our experimental Section by comparing different GAN architectures in terms of their ability to generate a lognormal distribution with parameters $\mathrm{shape}=0.5$ and $\mathrm{scale}=1.0$.
To evaluate the performance, we estimated the shape and scale parameters of the generated distributions and compared them to the true parameter pair.
BHSGAN provides the best estimate for the shape parameter and a close-to-optimal estimate for the scale parameter.

Reviewer *ScwD* subsequently asked us to test multiple parameter combinations to rule out a potential selection bias.
Therefore, we trained each architecture for nine different pairs of shape and scale parameters and compared the estimated parameters of the generated distributions to the true shape and scale parameters.
In this setting, BHSGAN performed best in most experiments by consistently providing estimates close to the true parameter values.

## Presentation

Reviewer *ScwD* pointed out that the reformulation of the optimization problem in Corollary 4.4. is surprising and requires checking the Appendix to explain the underlying transformation.
We, therefore, added some context in a paragraph preceding the Corollary to motivate this reformulation.
Reviewer *ScwD* also suggested increasing the size of Figure 2(a), which we incorporated by moving the generated MNIST digits (Figure 2(b)) to the Appendix and increasing the size of the generated CIFAR10 images (Figure 2(a)). 

Reviewer *zX4e* suggested improving the presentation by reducing Section 1, which contained overlapping information regarding Bayes Hilbert spaces.
We revised our manuscript by removing redundant information and restructuring Section 1 so that Bayes Hilbert spaces are introduced more thoroughly in Section 3.

As requested by Reviewer *Y69u*, we adjusted the wording regarding the experimental result to a more nuanced tone.

## Theoretical Results

Reviewer *Tns5* pointed out that our proposed theoretical divergence framework does not include traditional $f$-divergences since we initially only considered divergence-generating functions $f$ that are convex on $[a,\infty)$ with $a>0$.
We adapted the definition of pseudo divergences to include the case $a=0$, which did not lead to any changes in our main contribution.

Reviewer *zX4e* pointed out that our definition of the convex and concave conjugate should be adjusted by replacing $xy$ with $x^Ty$ to adhere to standard literature notation. 
We adapted the definitions to adhere to these conventions.

## Closing Remarks

We sincerely thank the reviewers for their insightful feedback, which has significantly strengthened our manuscript.
We hope the revisions and discussions have addressed their questions and resolved any potential concerns about our results.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes pseudo f-divergence, an extension of the traditional concept of f-divergence, and introduces a novel estimation algorithm based on it. Specifically, the study extends the conventional f-divergence, which relied solely on convex functions, to pseudo f-divergence, which incorporates combinations of convex and concave functions. Furthermore, the paper reveals a fundamental connection between such divergences and the Bayes Hilbert space for certain choices of f.　The estimation method is based on GANs, and the proposed approach demonstrates performance comparable to Wasserstein GANs on several tasks. While many reviewers acknowledged the importance of the paper’s mathematical contributions, they also pointed out insufficient discussion regarding the practical utility of the method and the implications of its relationship with the Bayes Hilbert space.　Despite these shortcomings, the paper introduces a significant new direction for f-divergence in the context of machine learning, making it a valuable contribution to the community. Therefore, I recommend acceptance.