Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows
Authors: Alexandre Verine
benjamin negrevergne
Muni Sreenivas Pydi
Yann Chevaleyre
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative Models
Precision
Recall
Optimization
f-Divergeces
TL;DR: Training Generative models on a principled trade-off between precision and recall.

Abstract: Achieving a balance between image quality (precision) and diversity (recall) is a significant challenge in the domain of generative models. Current state-of-the-art models primarily rely on optimizing heuristics, such as the Fr\'echet Inception Distance. While recent developments have introduced principled methods for evaluating precision and recall, they have yet to be successfully integrated into the training of generative models. Our main contribution is a novel training method for generative models, such as Generative Adversarial Networks and Normalizing Flows, which explicitly optimizes a user-defined trade-off between precision and recall.  More precisely, we show that achieving a specified precision-recall trade-off corresponds to minimizing a unique $f$-divergence from a family we call the \mbox{\em PR-divergences}. Conversely, any $f$-divergence can be written as a linear combination of PR-divergences and  corresponds to a weighted precision-recall trade-off. Through comprehensive evaluations, we show that our approach improves the performance of existing state-of-the-art models like BigGAN in terms of either precision or recall when tested on datasets such as ImageNet.

Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows
Alexandre Verine LAMSADE, CNRS,
Benjamin Negrevergne
LAMSADE, CNRS, Université Paris- Paris-Dauphine-PSL.
Université Paris-Dauphine-PSA.
Paris, France al exeenrdeee.. wwwineddauphine psl.
Paris, France
benjamin negroovergneddaupiine psl.
Muni Sreenivas Pydi
Yann Chevaleyre
LAMSADE, CNRS,
LAMSADE, CNRS,
Université Paris-Dauphine-PSI Paris, France
Université Paris-Dapphine-PLL.
Paris, France
muni pydi @yyddddeeuiineeeeemmamee pst.pp.l.....
yann. yamn.chovaleerreddauuhi... psl.
Abstract
Achieving balance between image quality (precision) and diversity (recall) a significant challenge in the domain of generative models. Current state-of-the- art models primarily rely on optimizing heuristics, such as the Fréchet Inception Distance. While recent developments have introduced principled methods for evaluating precision and recall, they have yet to be successfully integrated into the training of generative models. Our main contribution is a novel training method for generative models, such as Generative Adversarial Networks and Normalizing Flows, which explicitly optimizes user-defined trade-off between precision and recall. More precisely, we show that achieving a specified precision- recall trade-off corresponds to minimizing unique f-divergence from family we call the PR-divergences. Conversely, any f-divergence can be written as a linear combination of PR-divergences and corresponds to weighted precision- recall trade-off. Through comprehensive evaluations, we show that our approach improves the performance of existing state-of-ther of-the-art models like BigGAN in terms either precision or recall when tested on datasets such as ImageNet.
Introduction
Evaluation of generative models has always been a challenging task. The metric used must reflect both the quality of the sample generated (precision) and how much the sample covers the targeted probability distribution (recall). Inception Score (IS) or Fréchet Inception Distance (FID) have been introduced and are now widely used by the community these metrics are favored because they "correlate well with the visual fidelity of the samples" and are "sensitive to both the addition of spurious modes as well mode dropping" [44]. However as pointed by Kynkäääniiemi et al. [28), FID and IS group these two aspects into a single value without clear trade-off. Depending on the use-case, generative models might require a good precision (high-resolution image and video generation, artistic synthesis, 3D model design) (data augmentation, drug discovery, anomaly detection). For that reason, a number more principled methods [9, 15, 28, 44, 47] have emerged assess precision and recall (hereafter abbreviated by
select the best models. Typically,
good recall
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

P&R) independently, however these methods cannot be optimized during training, because they are not differentiable or because they are too computationally demanding.
To enhance either the precision or the recall of particular model, an array of strategies and techniques have been employed. Usually these techniques involve altering the latent distribution posteriori (e.g. truncation and temperature post-training). Other methods exists such as through rejection sampling, boosting, or instance selection [3, 12. 13, 19, 23, 33, 49]. Although these approaches may utilize proxies of P&R, they are not theoretically grounded in the principled methods of evaluating these metries, leaving their alignment with the foundational concepts of P&R unverified. This leads the following question.
Question 1: Is possible to train generative model that achieves a specified trade-off between precision and recall?
In addition to this question, another of our goals is to understand existing generative models in terms of P&R. Modern generative models, such as Generative Adversarial Networks (GANs) or Normalizing Flows (NFs), are typically designed to minimize specific divergence measures. Given target distribution P and a set of parameterized distributions { {Poe E }), a generative model aims to find the best fit P Po. that minimizes a divergence between P and P. These divergences induce different behaviors at convergence. For instance, optimizing the Kullback-Leible (KL) divergence tends to favor mass-covering models [35], contrasting with the mode- mode-seeking behavior observed with other generative models, leading to the infamous problem of mode collapse. Transitioning from one divergence to another does indeed alter the model's ' behavior in terms of precision and recall, however, the implicit trade-offs made during the optimization of a general divergence remain somewhat ambiguous. This motivates the following question.
Question 2: What precision-red recall trade-off does an arbitrary f-divergence minimize?
In this paper, we bridge the gap between principled methods of P&R evaluation and controlling their trade-off. In doing so, address Questions I and 2 by making the following contributions:
We show that achieving a specified precision-recall trade-off corresponds to minimizing a particular f-divergence between P and P. Specifically, in Theorem 4.3 we give a family of J-divergences (denoted by Dx.rms À E col) that are associated with various points along the precision-recall curve of the generative model.
We show that any arbitrary f-divergence can be written a linear combination of f- divergences from the DA.r. family. This result makes explicit, the implicit precision-recall trade-offs made by generative models that minimize an arbitrary f-divergence.
We propose a novel approach to train or fine-tune a generative model on notoriously hard to train f-divergences, with guarantees on divergences defined by the lipschitz constant of f.
We use this approach to train models on a user specified trade-off between P&R by minimizing Dx-mr for any given A. We specifically focus on GANs and NFs. For instance, Figure I shows how our model performs under various settings of X. With a high X, we can train the model to favor precision over recall and vice-versa.
Through extensive experiments, we show that our approach enables effective model training to minimize the RR-DDiergence, particularly for fine-tuning pre-trained models, with notable impact from the choice of trade-off parameters X, while also demonstrating scalability with larger dimensions and datasets.
Related works
Generative model evaluation metrics: IS or FID are widely adopted due to their ability to assess visual fidelity and sensitivity mode variations. However, these metrics fall short in providing a trade-off between P&R. Principled approaches were introduced by Djolonga et al. [15] and Sajjadi et al. [44], and later extended by Simon al. [47], providing a definition P&R independently using PR- PR-Cure detailed in Section 3.3. We adopt this extended approach in our work. In image generative modeling, the current consensus for P&R evaluation are two methods. First, the method presented by Kynkäänniemi et al. [28], which provides simpler evaluation of P&R based on the estimation

---

(a) X 0.1
(b) A 1
(c) 10
(d) DEL
(e) DAKL
Figure 1: NFs- RealNVP [14] trained on 2D Gaussians. Fig. 1(a) to Fig. 1()): models trained to minimize D Fig. (d) and Fig. I(e): models trained to minimize DKL and DiKL respectively. Samples drawn from the true distribution /' are shown in black, samples drawn from the estimated distribution P are shown in green and the log-likelihhod of P is shown in blue (darker means higher density). (a) À = 0.1, favors recall over precision. (b) À = 1, balanced precision vs. recall trade-off. (c) = 10, which favors precision over recall. As observed in [35], and demonstrated in Theorem 4.5, the DKL. (I(d) is mass covering and the DiKL (1()) in mode-seeking. The corresponding PR-Curve are presented in Figure 2(c).
of the support of the distribution using a k-NN algorithm, akin to the most recent work by Cheema and Urner [9]. Second, a method introduced by [36], computing the Density (D) and the Coverage (C) which account for local density and are robust to outliers in contrast to [28]. In natural language processing, one popular method, MAUVE [41], consists of the area under PR-Curves defined by [15]. However, because of their mathematical foundation, these methods remain unsuitable for training due to con-differentiabiliyy and high computational requirements.
Trade-offs and techniques in generative model: The challenge of balancing P&R in generative
model training has led to a variety of techniques. Methods for enhancing precision often involve
manipulating the latent distribution post-training, such hard truncation [7, 24, 25, 46] standard deviation adjustment, also called soft truncation [27]. One can also use rejection sampling in the image space [4, 50], in the latent space [23, 49], or in the dataset [13]. There are also numerous methods [12, 16, 19] and mixture based latent models [6, 40]. However, should be noted that these approaches lack a foundation in the principled methods of P&R evaluation. This underscores the need for a theoretically grounded approach to balance these crucial aspects in generative models, a gap that our current work aims to address. Note also that all these methods can be applied to any model, and in particular to the one trained using our proposed method.
strategies to prevent mode collapse and boost recall [8, 32], gradient-boostee
Divergence training in generative models: Modern generative models, such as GANs and NFs, often employ specific divergence measures for model optimization. With works such as Grover et al. [20], Nowozin et al. [39], models can be trained with a variety of divergences such as f-divergence, thus observing different results of P&R. Another notable work by Midgley et al. [34] proposed training with a-Divergence, under the assumption of access to data density. The work of [29], closely aligned with our approach, defines different orders of mode seeking and evaluates the corresponding f-divergences at these levels. Although its methodology employs the training of f-GAN models on simple datasets to illustrate the mode-seeking property, it does not offer the flexibility to establish user-defined trade-off. The implicit trade-offs made by these divergence measures is still a challenge: the recent work of [48] shows the links between P&R and any DeGroot's 's divergences.
3 Background
Throughout the paper, we use X' C Rd to refer to the input space and 2 C Rm to the latent space of the model we consider. We also denote P(X') and P(Z) the set of all probability measures over measurable subsets of X' and respectively. P and P are consistently used to denote the target and estimated distributions (both members of P(x'). We also assume that P and P share the same support in X, and that they admit densities denoted by the corresponding lower case letters p and p. respectively. Finally, for any function f E IR - TR, we define the convex conjugate (or Fenchel transform) of f given by (t) suP.eR {tu f())
Notation:
3

---

3.1 f-divergences
Given a convex lower semi-continuous (l.s.c) function f R - R satisfying f(1) 0, the f-divergence between two probability distributions and P is defined as follows.
[(x) p(x))
D)(P1P)= -AP1)))-{A000 Mx)s
da.
(I)
Dj is invariant to an affine transformation in f i.c., Dj(PIPP = (P((P))) for fi(u) = f(u) + c(u - 1) for any constant R. Many well-known statistical divergences such as the KL divergence (DKL), the reverse KL (DeKL) or the Total Variation (Drv) are f-divergences (see Table 1). Importantly, any Dj admits dual variational form [37]. with T denoting the set of all measurable functions X' + R:
(P(P)) = sup Df (PIP), where TET
Di (PII)) = Ex- [T())] - E. E P
We use Topt E T to denote the function that achieves the supremum in (15).
Table I: List of common f-divergences. The generator f is given with its Fenchel conjugate f" The optimal discriminator 7'ppt is given compute the likelihood ratio p(æ)/f(a) Vf* (T (x)). Then (1(AA)//A* is given to compute the D as combination of Dx.en using Theorem 4.4.
DIVERGENCE
NOTATION
f(u)
(t)
(x)
("(1/2//X
KL
DRL.
log
exp(t-1) 1)
1 + logr(z)/f(ar)
1/X²
REVERSE KL
DALL
log a
log
-(xx)((>)
1/A
-PERSON
Dx²
(u
1)²
t//++
(p(x))(aa) -
2/2²
3.2 Generative models
Generative Adversarial Networks (GANs): A GAN consists of two functions, generator Z + X, discriminator 7' X IR, as well prior distribution Q E P(Z) which is usually the standard normal (0, I. The estimated data distribution Pc the push-forward distribution of Q by G. In the original work of Goodfellow et al. [17], specific Dj is used to optimize the divergence between P and Pg. In this paper, we consider the more general framework of Nowozin et al. [39] that can be used to train a GAN with any DDy by solving the following min-max optimization problem:
min max Ddual [P] G
min max Re- (T(x)) E Ea-Pa G
Pro [T))))
Normalizing Flows (NFs): An NF consists an invertible function G - and a prior distribution Q E P()). As for GANs, the estimated distribution Po is the push-forward of Q by G. However, because G is invertible, it is possible to compute the density p(-) using simple change of variable formula, p(x) = q(G ( (x)) det Jaco- (æ)], where det Jacg- (x) is the determinant of the Jacobian matrix of G-1 æ, and train the generator using the DKL between P and Pp, or cquivalently, by maximizing the log likelihood:
min DKL H(P) max ExoP llog (x)]
(4)
where (P) is the continuous entropy of P. In practice, the generator function G is typically represented by neural networks such GLOW [27], RealNVP [42], or ResFlow [5, 10] for which det Jacq-= (z) is easy to compute. Grover et al. [20] showed that is possible to train an NF using most f-divergences. In practice, the lig-liieldhod is added to the min-max objective stabilize learning. This gives us the following optimization problem:
min max Ddual (Pl) ?PEE Dog (x))),
(5)

---

Target
Model P
1.0
Model P.
Model
Model P
0.8 4
9.5
A=01 0.1
045
10
0.4
02-
00 064
0.6
0.8
3
04
01.5
1
(a) Distributions
(h) (PRD(P, P) curves
(c) @PRD(P, P) for Fig.
Figure 2: PR curves for two models Pi and P3 of P. Figure 2(a) shows P1, P3 and P. Figure 2(b) shows PR curves for Pı, P2 against P. Pj has good recall since it covers both modes of P, but low precision since it generates points between the modes. P₂ has good precision since it does generate samples outside of P but low recall since it can generate samples from only one mode. Figure 2(c) are the PR curves corresponding Figure and are detailed Section 6.
3.3 Precision-Recal curve for generative models
Generative models are usually evaluated using a single criterion such as the IS [45] or the FID [21].
However, these criteria unable to distinguish between the distinet failure modes of low precision (i.e., failure to produce quality samples) and low recall (i.e., failure to cover all modes of P). The following definition introduced by Sajjadi et al. [44] and later extended by Simon et al. [47].
Definition 3.1 (PRD set, adapted from Simon et al. [47]). For P. P E P(X), the P&R set PRD(P, P) defined as the set of pairs of precision a and recall in IR+ R+ such that there exist Ja E ((X) for which 2 Bu and P 2 ouu. The precision-recall curve (or PR curve) is defined as (P, = [(a,3) E PRD(P, B(a', with a' 2 a and 3 2 3}.
An equivalent definition of PRD(P, P) is found in Sajjadi et al. [44], where (a,, B) E PRD(P, P) if P and can be decomposed as in (6) for some common component E P(X) and complementary components vp, Mp P(X).
P ap (1 a)vp and P 3p -1-1111 B)up-
(6)
Simon al. [47] show that the PR curve is parameterized by X E [0, +oo] @PRD P) ( J [0, + with (x(PI)P) =
Sx. min (Ap(m), ((x) da and (P P) = (Pll P)/ Here, À is called the trade-off parameter and can be used to adjust the sensitivity to precision recall.
illustration of the PR curve is given in Figure 2 for a target distribution P that is a mixture of two Gaussians and two candidate models P1 and Pz. We can see on Figure that Pt offers better results for large values of À (with high sensitivity to precision) whereas P2 offers better results for low values of (with high sensitivity recall).
Precision and Recall trade-off as an f-divergence
In this section, we formalize the link between P&R trade-off and f-divergences, and address Ques- tion 2. We will exploit this link in Section to train models that optimize particular P&R trade-off.
4.1 Precision-Reeall as an fdiiverge
We start by introducing the PR DR-eivergence as follows.
5

---

Definition 4.1 (PR -iveeveege,,,,) Given a trade-off parameter À E [0. too), the PR-divergence (denoted by D defined as 1,, for fx- R + R given by fx(u) = max(Xu, 1) max (A, 1) for X E Ikt and fa(u) = for X +X0
Note that fx is continuous, convex, and satisfies f1(1) 0 for all X. A graphical representation of fA can be found in Appendix B.2. The following proposition gives some properties of D D.. Proposition 4.2 (Properties of the PR-Divergence).
The Fenchel conjugate fr of ʃx is defined on dom (fx) = A| and given by, SX (1) = 1/2
for X < and J (1) 1/\ X otherwise.
The optimal discriminator for the dual form is ToP( = Asign (p(x)/p(a)) - 1}.
-Dxma *Dxr((P1PP) XD xDarra
Observe that Dxven (PIIP) + Dx-ea (PIPP in general, but for X = 1, De-en(PllP)))) = D 1- (PP()P Drv (ml(Pl)))) P)/2. Having defined the PR-divergence, we can now show that P&R w.I.t can be expressed as a function of the divergence between P and P.
Theorem 4.3 (P&R as a function of D..m).. Given P. P P(X)) and X E [0, +00), the PR curve @PR P) is related to the PR-divergence Dx.re (xma(Pa (PI[P) as follows.
(PI P) min(11A)- DDxxx(PPP))
(7)
Conversely, suppose that there exists a strictly decreasing linear function [0,11] + R+ and an f-divergence Dy such that h[ax(P] P)) = Dy(PIIP) for all P.P E P(X), then f(u) = c1 fx(ux + (1)
direct consequence of Theorem 4.3 is that minimizing Dxee is equivalent to maximizing ax, In a more explicit way. Theorem 4.3 suggests that Dx-ea is the only f-divergence (up to an affine transformation) for which this property holds. This makes D XASR a uniquely suitable candidate for training generative model with a specific P&R trade-off. The proof of Theorem 4.3 is in Appendix B.1
4.2 Relation between PR-divergences and other f-divergences
In the previous subsection, we showed that for each trade-off parameter X, there exists a Dx.en that corresponds to optimizing for it. This raises the converse question of what trade-off is achieved by optimizing for an arbitrary f-divergence. We answer this by showing in the following theorem that any f-divergence can be expressed weighted sum of PR- -R-vivergesces.
Theorem 4.4 (d-divergence weighted sums of PR- PR-divergences). For any P, P(X) supported on all of and any X IR,, with m minx (f(x)) /(x))) and M = maxx (f(x)/((x))
()(P)|P)= = J.m .M 1
(1) DA (PII
(8)
X3
As sanity check, observe that the weights (1(1/A)/X² remain invariant under an affine transforma- tion in ff much like Dy.
Corollary 4.5 (DKL and DIKL as an average of DX-r). The DEL Divergence and the DeKL can be written as a weighted average of PR-Divergence D
(PI(P) .M 1 - _ D
M and (iKL(PIIP) = [ *
(9)
As we can see in this Corollary, both DKL and D,KL can be decomposed into a sum of PR-divergences terms DA-Fa. each weighted with 1/X² and 1/A respectively. Note that if A < 1 then 1/X > 1/X², and conversely, if X > DKL is thus associating more weights on recall than D,KL. This explains the mass covering behavior observed in NFs trained with DKL. Comparatively, the D,KL assigns more weight to terms with a large lambda, leading to the mode seeking behavior empirically observed with flows trained with the 1, [33]. However, as it will be observed in the Section 6, D is still favoring low values of Other weights for other /ddivergences are Appendix A.
An equivalent result can be found in Corollary 19 of [48]

---

Minimizing the Precision-Recal divergence
In this section, address Question 1 by introducing new method that can be used optimize model with a specific precision-recall trade-off A. A first naive strategy to achieve this is to use the f-GAN framework introduced by Nowozin et al. [39], and minimize the dual variational form of Divnn presented in Theorem 4.3. Together, this results in solving the min-max problem:
min max D D (PIPP) min max E [T()) - 2 (f)()
(10)
G
where G and T are both represented using neural networks. In practice, this strategy would fail because training a neural network with loss functions such as fiv (or in this case fx) is notoriously difficult due to vanishing gradients that lead to poor training performance [20, 39]. Instead, training neural networks on functions f" such fkI. fo+. results in much better empirical performance [29, 39, 51].
To avoid these issues, show how train the discriminator using auxiliary divergence (based on a function g # f) to better estimate the target f-divergence. The main idea is to choose an auxiliary g g-divergence that is adequate for training (e.g. DKL or Dx²) and use compute the likelihood ratio p(z)/p(z)
Because, optimality, have Vg" (T)) (o(x://(())) as follows:
then compute the f-divergence
D)(P)|P)= D
p(x) p(x))
dæ ((V (ToP)())))dl
(II)
In practice, we do not have access to Topt Instead, we have a discriminator 7 trained to maximize Deual. At any time during training, we can estimate the f-divergence based I using the primal T.T estimate, which we define as follows.
Definition 5.1 (Primal estimate Dprimal). Let P.P P(X). For any function T X R. f: + R, and g R+ + R, we define the primal estimate Dyrimal as follows.
Dprimal (PIIP) p(æ)f (r(z))
(12)
where r X + IR is given by, r(æ)= = Vg* (T())).
The success of this approach depends on how well r approximates p(z)/f(z), which depends on T. We first show that the approximation error of r measured in terms of the Bregman divergence [2] (also defined in Appendix A.2). It corresponds exactly the approximation error of Ddual so minimizing the latter will also minimize the former.
g.T
Theorem 5.2 (Error of the estimation of an f-divergence under the dual form.). For any discriminator
T: X + R and (x) (T())),
p(x) p(x)
(PIPP) Diiaal (PI[P) -Ea Breg, r(())
(13)
On the basis of this result, the quality of the approximation will crucially depend on Vg. We can show that if the auxiliary g is strongly convex, the error in the estimation of D is bounded:
Theorem 5.3 (Bound on the estimation of an f-divergence using an auxiliary g-divergence). Let f.g : + + R be such that j-stroogly convex and f a-Lipschitz. For the discriminator T X. + R, let = Vg* (Tx)) Then
2e
D, (2(P) - <c - (D((P))) - D
- (PI <
'The explanation for this vanishing gradient phenomenon primarily relies on the fact that the optimal discriminator for these functions is "0tt(a = Sign(((a) <)/p(=)) - 1). To approximate it, [39] recommends the of tonh activation function that known to induce vanishing gradients.

---

I successfully maximizes D then the primal estimation converges to D,. To implement this approach, we propose the following simplified version of the algorithm: Repeat until convergence these 3 steps:
1. Let x1
real NN
P and sfttee
fake ... NN
Pa.
2. Update the parameters of T by ascending the gradient
vLr
1 V
2i=1
(C Enso" (T(afake))
N
3. Update the parameters of G by descending the gradient
VC N V N
( If (Vg" (T(efak)))
This method closely parallels the GAN training procedure, with the key distinction that T and G optimize objectives based on different f. practice, our implementation uses a stochastic gradient descent, fully detailed in Algorithm 1 in Appendix C.
Experiments
In this section, we employ the auxiliary loss approach outlined in Section 5 to train various models.
Specifically, we train NFs on 2D synthetic data, MNIST and FashionMNIST,, while we train BigGAN on CIFAR-10, CelebA64, ImageNet128 128, and FFHQ256. All models are tested in terms of IS and FID with 50k samples, and on P&R with 10k samples using the method of Kynkäääniemi et al. [28] with k = 3 for MNIST and FNSS and with & = 5 for CelebA64, ImageNet128 and FFHQ256. Also, we test every model in terms of Density and Coverage [36] on 10k samples with k 5. In this paper, we present a selection of experimental results. For a comprehensive set of results, including model parameters, optimizers, learning rates, and samples, please refer to Appendix D. We also included in this Appendix set of experiments run using the naive approach based on Equation 10, thus showing that the discriminator fails to train as explained in Section 5. To ensure reproducibility, our models and code are available the GitHub repository of the project².
We show that: 1) the auxiliary loss approach effectively enables the training of model to minimize the PR- PR-Divergence, 2) this method is suitable for fine-tuning pre-trained models, 3) the choice of trade-off parameter significantly influences the results on P&R, and finally, 4) our method scales well with larger dimensions and datasets.
Normalizing Flows synthetic data: NFs are typically trained to minimize DKL in addition to their structural limitation [11, 52], resulting in good recall but poor precision. Prior work has employed various techniques [27, 49] to improve model precision post-training, we our method to directly train the model on given trade-off. We demonstrate our approach by training RealNVP models 2D synthetic dataset using D Daven with various À values and using DKL and DIKL for baseline comparison. As we can see in Figure 1, increasing À leads to an increase in precision in the resulting models. Using DA.ra estimation, we compute the corresponding PR curves (Figure 2(c)). The À = 0.1 model, while best at 0p.1, performs poorly at 01 and 0'00 clearly demonstrating the impact of maximizing axx- This pattern across models validates the efficacy of our method in minimizing the desired trade-off.
Using DKL. vs Dz= on MNIST and FashionMNIST:: We now demonstrate that we can improve the precision by directly minimizing the DA.ra with the correct À using pre-trained GLOW models [27] on both MNIST [54] and MNST [53]. Figure 3 and Figure 4 present the samples obtained. First, we observe that while À increases, the visual quality improves, but the models focus on a few modes only (0, 1. 7, 6, 9 for MNIST and "Trouser" for FashionMNIST). Then training with both DKL and D divergences aligns with our expectations: DKL training leads to high recall, while DiKL. to higher precision. However, according Corollary 4.5, DiKL still favors low À values; consequently, our models trained with A= > 0.1 demonstrate better precision than standard flow-GAN models. Furthermore, we find that both auxiliary g functions (fKL and fx²) used to train the discriminator T perform well. In practice, training with fy² proves to be more stable, particularly for FashionMNIST, with results reported in Appendix D.3. For larger models, we use exclusively g = =xx..

---

Q
S
2
Y
G
R
7
D
9
0.00-
(a) À = 0.1
(b)
(c)
À = 10
0.79
0.12
PREKL
MLE
0.70
KLL
0.22
0.68
0.00
10'
12"4
30'
10
(d)
10
Figure 4: Precision and Recall as func- tion on À for GLOW trained on MNIST with T trained with fKI. or fx2, and for models trained DEL. DeKL and MLE.
Figure 3: Samples from NFs GLOW trained on MNIST
(3(a) to 3(c)) and FashionMNIST (3(d) to 3()). Recall decreases as precision increases for between 0.1 and 10.
Training BigGAN on CIFAR-10 and CelebA64 Now we demonstrate that our method can also be used to train large generative models. Our choice to adopt the BigGAN architecture [7] was informed by several factors: competitive performance close to state-of-the-as models; its versatility, permitting diverse experimental explorations: and the fact that it is publicly accessible, ensuring experiment reproducibility. We train Big- GAN using both the baseline method (i.e., hinge loss) and our proposed method on CIFAR-10 [1] and CelebA64 [30]. A notable observation when training with different precision recall trade-off's the early elim- ination of modes from the target distribution at higher values of X. As illustrated in Figure models with low values maximum recall, while those with À > rapidly saturate to lower recall value. similar behavior can be observed for models trained on CelebA, as shown in Figure D.18. In Table 2, we present the quanti- tative metrics (Precision, Recall, and FID) for the baseline BigGAN, the BigGAN models trained with varying trade-offs and the current state-of-the-an models: EDM-G++ [26] for CIFAR-10 and ADM-IP [38] for CelebA64. Employing our proposed method enables us to ad- just the trade-off, allowing us to train models that closely approach the state-of-the-an recall and, for high X, even outperform statc-offthe-aar models in terms of precision.
Recall
1,50
5.00
2,00
10.00
3100
20.00
converge achieve
232
BiG
lreeationns
Figure 5: Recall during training of BigGAN op- timizing different trade- off on CIFAR-10. Mod- els for high saturates in an early stage at decreas- ing values of recall.
Table 2: BigGAN trained with the vanilla approach [7] and with a variety using our approach on CIFAR- 10 and CelebA64. We compare approach with hard truncation the baseline model. FID (L), Precision (t), Recall (t), Density (f) and Coverage (f) are reported. In bold, our best model highlighted and the state-of-the-ard FID is marked with exponent "
MODEL
CIFAR
10
32
CELEBA
64
FID
P
FID
BASELINE BIGGAN
13.37
86.51
65.66
0.76
0.81
9.16
78.41
51.43
0.89
0.48
HARD V 2.0
13.95
86.82
63.58
0.77
0.79
10.60
80.81
48.21
0.96
0.50
HARD 3 1.0
17.23
88.03
53.63
0.83
0.75
17.97
84.30
37.46
1.11
0.49
HARD & = 0.5
20.11
87.87
44.98
0.83
0.70
25.70
83.70
28.81
1.08
0.42
À 0.05
13.29
81.10
70.63
0.61
0.80
À 0.1
11.62
81.78
74.58
0.66
0.83
13.36
84.85
65.13
0.74
0.82
8.79
83.37
44.07
1.09
0.54
A = 0.5
14.50
83.27
68.23
0.70
0.81
6.03
77.60
55.98
0.88
0.50
A 1.0
14.03
83.04
69.35
0.68
0.79
13.07
81.70
36.85
1.00
0.47
A 2.0
16.94
84.93
59.79
0.75
0.78
14.23
82.98
32.87
1.16
0.49
A 5.0
32.54
83.39
56.94
0.68
0.73
22.45
83.96
25.81
1.21
0.43
10.0
39.69
84.11
39.29
0.75
0.67
X = 20.0
67.03
90.03
21.81
0.98
0.56
DENSEFLOW [18]
88.90
60.81
0.86
0.71
85.83
38.22
1.17
0.82
ADM-IP [38]
3.25
80.67
83.65
0.65
0.87
1.53*
23.42
64.48
0.09
0.24
EDM G++ [26]
1.77*
78.48
85.83
0.60
0.87
STYLEGAN XL [46]
1.85
85.11
70.04
0.75
0.85

---

In every experiment, we compare our approach with traditional post-traimmn techniques. In Table 2. we give the results for the hard truncation (also called the truncation trick [7]) of the latent distribution Q for different tb. We observe that this method enables to improve solety the precision by trading off the recall; however, note that the truncation can be use in addition our approach.
Fine-tuning BigGAN on Imagenet128 and FFHQ Finally, we apply our method to pre- pre-trained BigGAN models. To accomplish this, we implement a straightforward technique: initially, we train the discriminator for brief period, allowing the model to transition from the vanilla training objective to the Ddual, This approach enables us to train BigGAN on large datasets such as ImageNet122 [43] and datasets with high dimensions such as FFHQ256 [24]. We compare method with both hard truncation and soft truncation (also denoted temperature in [27]). Both methods can be used in addition to our approach. The metrics presented in Table 3 demonstrate that (1) we enhance a given model's precision (by +2.83% or recall (by +1.17%) on ImageNet, thereby achieving ffate-of-the-art art precision, and (2) method compromises less on the trade-off than truncation. For instance, in FFHQ, for a similar precision improvement (== +15.5%), recall is decreased by more than 5% for truncation methods and only by 1.63% with our approach.
Table BigGAN fine-tune with the vanilla approach [7] and with a variety of À using our approach on IntgeNet128 and FFHQ256. We compare our approach with hard truncation on the baseline model. FID (L), Precision (f), Recall (f), Density (f) and Coverage (1) are reported. In bold, our best model is highlighted and the state-of-the-ane FID is marked with an exponent".
MODEL
IMAGENET 128 X 128
FFHQ 256
256
FID
R
D
FID
P
R
D
C
BASELINE BIGGAN
9.83
28.04
41.21
0.14
0.17
41.41
65.57
10.17
0.52
0.47
SOFT 3 0.7
11.39
23.04
31.13
0.11
0.15
56.43
76.59
4.87
0.70
0.41
SOFT 0.5
15.49
20.20
19.83
0.10
0.14
82.05
84.48
1.58
0.89
0.32
HARD 1 2.0
9.69
25.83
39.89
0.13
0.18
43.32
68.84
8.66
0.58
0.47
HARD D 1.0
12.12
21.86
35.42
0.11
0.15
56.19
76.44
4.76
0.75
0.44
HARD & 0.5
15.21
21.13
29.55
0.10
0.13
71.32
80.99
4.84
0.84
0.36
À 0.2
9.92
26.69
42.04
0.13
0.17
35.66
78.70
9.45
0.88
0.60
= 0.5
10.82
26.83
42.38
0.13
0.16
35.24
78.41
9.66
0.89
0.60
À = 1.0
20.42
29.72
28.21
0.15
0.15
35.91
78.95
8.32
0.90
0.57
A 2.0
20.21
30.27
30.49
0.14
0.14
36.33
81.10
8.69
1.05
0.64
A 5.0
20.76
30.87
28.38
0.15
0.15
38.16
84.31
8.52
1.15
0.63
ADM [22]
2.97
26.63
68.54
0.14
0.16
STYLEGAN-XI [46]
1.81
11.35
68.04
0.04
0.09
2.19"
79.91
38.79
0.86
0.73
7
Conclusion
In this paper, we present novel method for training generative models using new PR-divergence, DA.mn Our approach offers unique advantage over existing methods as it allows for explicit control of the precision-recall trade-off in generative models. By varying the trade-off parameter X, one can train variety of models ranging from mode seeking (high precision) to mode covering (high recall), as well as more balanced models that may be more suitable for various applications. Through extensive experiments, we demonstrate the validity of our method and show that it scales well with larger dimensions and datasets. Our approach also provides insights into the implicit P&R trade-offs made by models trained with other /-divergences. By introducing the Dx.mm divergence and providing systematic approach for training generative models based on user-specified trade-offs, we contribute to the development of more customizable generative models. Our method currently applies to GANs and NFs only, it is still unclear if similar approach can be applied to trained diffusion models: promising work [26] uses a discriminator to refine diffusion models, and could be used to estimate the D X-mn-
Acknowledgment
We are grateful for the grant of access to computing resources at the IDRIS Jean Zay cluster under allocations No. ADO1 101 1296 and No. AD011014053 made by GENCI.
10

---

References
[1] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[2] Shun-ichi Amari and Hiroshi Nagaoka. Methods of Information Geometry. volume 191 of Translations of Mathematical Monographs, Providence, Rhode Island, April 2007. American doi: 10. 00.1090/mono/191.
Mathematical Society. ISBN 978-0-8218-4302- 978- 978-1-4704-4605-5
URL http::/www ams org/mmono/199.
[3] Lynton Ardizzone, Radek Mackowiak, Carsten Rother, and Ullrich Köthe. Training Nor- malizing Flows with the Information Bottleneck for Competitive Generative Classification.
arXiv:200J. 064448 [cs, stat], January 2021. URL http://arxx wwwpsiiv.orggaae org/abs/ 2201..20...00 20044004110....... arXiv: 2001.06448.
[4] Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discriminator Rejection Sampling, February 2019. URL http://arxiv http//aaavv.oo.gbr..raa/ wwtp:/o////////////
06758. arXiv:1810.066 [cs, stat].
[5] Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible Residual Networks. Proceedings of the 36 th International Conference on Machine Learning, Long Beach, California, PMLR 97, 2019, May 2019. arXiv: 1811.00995.
[6] Matan Ben- Yosef and Daphna Weinshall. Gaussian Mixture Generative Adversarial Networks for Diverse Datasets, and the Unsupervised Clustering of Images. arXiv:/808 10356 [cs, stat], August 2018. URL http:http///eeeet //axxiv
htgg/abs/18000 10356. arXiv: 1808. 10356.
[7] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fi- delity Natural Image Synthesis, February 2019. URL httt:://ttpiiii arXiv:1809. 1096 [cs, stat].
abtpa//./...////////// 11096.
[8] Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode Regularized Generative Adversarial Networks, March 2017. htt.tt/////axiv org/aat//a att11/2b/661...... /1////11121.......11
arXiv:1612.021366 [cs].
[9] Fasil Cheema and Ruth Urner. Precision Recall Cover: A Method For Assessing Generative Models. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, pages 6571-6594 1-6594. PMLR, April 2023. URL https: //troc/eeoocc....ee//oe///c mlr, .urrssss//// v206/ chr6//cheema2a.... html. ISSN: 2640-3498.
[10) Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, and Jörn-Herrik Jacobsen. Residual Flows for Invertible Generative Modeling. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada., July 2020. arXiv: 1906.02735.
[11] Rob Cornish, Anthony L. Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows. In Proceedings of the 37th International Conference on Machine Learning, April 2021. URL http: / attaaaxiiviooo/ org/ abs/1909. 13833. arXiv: 1909.13833.
[12] Zac Cranko and Richard Nock. Boosted Density Estimation Remastered. In Proceedings of the 36th International Conference on Machine Learning, pages 1416-1425, PMLR, May 2019. URL htttps://ppettee http:///////rreeenng......rrrrr .ww............. www.pppeessvvgge wtwpall.ppeeeee///eeeeeeeee http://///rao11..b/tt.. html. ISSN: 2640-3498.
[13] Terrance Vries, Michal Drozdzal, and Graham W. Taylor Instance Selection for GANs, 15255. M.Xiv:2007.1525: [cs, stat].
October 2020. URL http: http:tppp//aaii.ooo/aaaaaae
[14] Laurent Dinh, Jascha Sohl-Dickstein,, and Samy Bengio. Density estimation using Real NVP, February 2017. URL
a105 16050660.... 08803. 1exiv:1605. .88003 [cs, stat].
[15] Josip Djolonga, Mario Lucic, Marco Cuturi, Olivier Bachem, Olivier Bousquet, and Sylvain Gelly. Precision-Recall Curves Using Information Divergence Frontiers, June 2020. URL http:///arxi..og/bbbs/////s
10768. 190. [cs, stat].
[16] Robert Giaquinto and Arindam Banerjee. Gradient Boosted Normalizing Flows. arXiv:2002. 11896 [cs, stat), October 2020. URL http://arxiv. org/abs/ 2ttp:/202...111 arXiv: 2002.11896.
[17] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. In 27th Conference on Neural Information Processing Systems (NeurIPS 2014), June 2014. URL http://arxivv org htgg/abs/1406 2661. arXiv: 1406.2661.
11

---

[18] Matej Greić, Ivan Grubišić, and Siniša Segvić. Densely connected normalizing flows, November
2021. URL http:///aax...rrr//bb///1...... 04627. arXiv:2106.0462% [cs].
[19] Aditya Grover and Stefano Ermon. Boosted Generative Models, December 2017. URL
http:///aaxii...gabbbb../eso /tt.//////............ 1rXiv:1702.08484 [es, stat].
[20] Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models, January 2018. URL http::/arre org/ abs/ 08868.
arXiv:
[cs, stat].
[21] Martin Heusel, Hubert Ramsauer. Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. GANs Trained by Two Time-Scale Update Rule Converge a Local Nash Equi- librium. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URI. https: httpt:p::/ppooceeediin ittpeeedddinnn.nnniiisise. www.nnuurp.... cwtprrppp..pppppp/1111111 8a1d694707eb0fe e658 139556871669749992668 www.raat.ttttateeeee html.
[22] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising Diffusion Probabilistic Models, December 2020. URL http: / /ttp//aaxvvv.//////i///////aaav/ org/ htrg/abb/20005 11239. arXiv:2006. 1239 [cs, stat].
[23] Thibaut Issenhuth, Ugo Tanielian, David Picard, and Jeremie Mary. Latent reweighting, an almost free improvement for GANs. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1574-35833 Waikoloa, HI, USA, January 2022. IEEE. ISBN 978-1-66540-915-33 doi: 10.1109///CV55548.2222222 ieee. org/ /wtp///t/m//////////////// /tccummnt//7709934////
URL
[24] Tero Karras, Samuli Laine, and Timo Aila. Style-Based Generator Architecture for Gen- erative Adversarial Networks, March 2019. URL http:://rxtis http://aawxvv..o/aa/////s/a /aabs/1812..m 04948. arXiv:1812.04944 [cs, stat].
[25] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and Improving the Image Quality of StyleGAN, March 2020. URL http://arxiv. org/abs/
/1taaa////.../// 1912. 04958. arXiv: 1912. 1et.1.112.999.. [cs, eess, stat].
[26] Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and I1-Chul Moon. Refining Gen- erative Process with Discriminator Guidance in Score- based Diffusion Models. In Proceedings of the 40 th International Conference Machine Learning.. volume 202, Honolulu, Hawaii, USA, April 2023. JMLR. URL http: /arxivv. org/ abs/ 2211. 17091. arXiv:2211.17091 [cs] version: 3.
[27] Durk P. Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolu- tions. In 32nd Conference Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada., volume 31, 2018
[28] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved Precision and Recall Metric for Assessing Generative Models. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada., October 2019. arXiv: 1904.06991.
[29] Cheuk Ting Li and Farzan Farnia. Mode-Seeking Divergences: Theory and Applications to GANs. In Proceedings of The 26th International Conference Artificial Intelligence and Statistics, pages 8321-8350. PMLR, April 2023. URL https: httt::::/prrceedinng mlr. .tressss/////
~206/ting-1it 1i20a html. ISSN: 2640-3498.
[30] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes the www.rriiv.rggaaa///ee .7666. 1.Xiv:1411.7741 [es].
Wild, September 2015. URL.
[31] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks, June 2018 URL http://xxii..rrraas/////// 04722.
[es].
[32] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled Generative Adversarial Networks, May 2017. URL http: http::///axxiv.i http://axxvvv../raa// aa1// 02163. arXiv:1611.02163 [cs, stat].
[33] Laurence IIling Midgley, Vincent Stimper, Gregor N. C. Simm, and José Miguel Hernández- Lobato. Bootstrap Your Flow, March 2022. URL http://t////a http:::/arriv.orr/aaa///11111 11510. 111 [cs, stat].
12

---

[34]
Laurence Illing Midgley, Vincent Stimper, Gregor N. C. Simm, Bernhard Schölkopf, and José Miguel Hemández-Lobato Flow Annealed Importance Sampling Bootstrap. November
2022. URL http:://axxxii orggabs//200 01893. arXiv:2208.00 01 893 [cs, q-bio, stat].
[35]
Thomas Minka. Divergence measures and message passing. page 17, 2005.
[36] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh. Yunjey Choi, and Jaejun Yoo.
Reliable Fidelity and Diversity Metrics for Generative Models, June 2020. URL http:// /// 09797 SEi::2002.09797 [es, stat].
[37] XuanLong Nguyen. Martin J. Wainwright, and Michael I. Jordan. On surrogate loss functions and $ff-ddierrences. The Annaly of Statistics, 37(2), April 2009. ISSN 0090-5364. doi:
10.1214/08-A0S595, URL http::tttt///iiiae www.ooog/aba..
arXiv:math/051052..
[38] Mang Ning. Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input Perturbation Reduces Exposure Bias in Diffusion Models, February 2023. URL http://
arxiv. arxiv.org/abs/2300
11706. arXiv:2301. 1706 [cs].
[39] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization, June 2016. URL http:t///////ii.oo
abs/1606 0tw//66660079..... 007090 1.XXv:1606.000090 [cs, stat].
[40] Teodora Pandeva and Matthias Schubert. MMGAN: Generative Adversarial Networks for Multi-Modal Distributions. arXiv: 1etp:9911.06633 [cs, stat], November 2019. URL http://
arxiv arxiv.org/abs.aeooooooooo... 06663 arXiv: 1911.06663.
[41] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Mcasuring the Gap Between Neural Text and Human Text using Divergence Frontiers, November 2021. URL http:/tttaaaat //apiv.////ivv/// wrgg/ag///////ee//e//ooo 01454. arXiv:2102.01454 [cs].
[42] Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows.
In arXiv:/505.0577 05570 [cs, stat), June 2016.
[43] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge, January 2015. URL http: //axxv. org/abs/ http//149..5500 0575. 4rXiv.1409.0575 [cs].
[44] Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing Generative Models via Precision and Recall. In 32nd Conference on Neural In- formation Processing Systems (NeurIPS 2018), Montréal, Canada, October 2018. URL /ttp://arxiv.ooo wwwpsiii..gg/b////e/e 1atp///10...ooo 00035. arXiv: 1806.00035.
[45] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs, June 2016. URL http: httppp///axxvv.eeeeeeeeeeeeeeava wwwxiiv.or wrtp.bs/////// 03498. trXiv:16066 03498 [cs].
[46] Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings, pages 1-10, Vancouver BC Canada, August 2022. ACM. ISBN URL https: //dl. acm. org/ doi/10.
doi: 111114555222233353000 1145/3528233. 3530738.
[47] Loie Simon, Ryan Webster, and Julien Rabin. Revisiting precision recall definition for generative modeling. In Proceedings of the 36th International Conference on Machine Learning, pages
5799-5808 PMLR. May 2019. URL https: stttps://prceeedinng mlr, wrw.s/rssvvvveee http:///////mmmm/////// simon19a. html. ISSN: 2640-3498.
[48] Rodrigue Siry. Ryan Webster, Loic Simon, and Julien Rabin. On the Theoretical Equivalence of Several Trade-Off Curves Assessing Statistical Proximity. Journal of Machine Learning Research, 24, 2023.
[49] Vincent Stimper, Bernhard Schölkopf, and José Miguel Hemández--Lobao. Resampling Base Distributions of Normalizing Flows. arXiv:21 10. 15828 [cs, stat], February 2022. URL http://arrivv.or/bbb.s.s/
15828. arXiv: 21 10.118888.
[50] Ugo Tanielian, Thibaut Issenhuth, Elvis Dohmatob, and Jeremie Mary. Learning disconnected manifolds: a no GANs land. In Proceedings of the 37 th International Conference on Machine Learning, Vienna, Austria, PMLR 119, 2020, December 2020. arXiv: 2006.04596.
13

---

[51] Chenyang Tao, Liqun Chen, Ricardo Henao, Jianfeng Feng, and Lawrence Carin Duke. Chi- square Generative Adversarial Network. In Proceedings of the 35th International Conference on Machine Learning, pages 4887-4896. PMLR, July 2018. URL. www.thttt///otoeeeeee
mlr. wttppp//////totee//// html. ISSN: 2640-3498.
[52] Alexandre Verine, Benjamin Negrevergne, Yann Chevaleyre, and Fabrice Rossi. On the expressivity of bi-Lipschitz normalizing flows. In Proceedings of The 14th Asian Conference
on Machine Learning, pages 1054-1069. PMLR, April 2023. URL https: h/tos::::///rocedingss...
mlr. #TTes//1199/eerine233 html. ISSN: 2640-3498.
[53] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms, September 2017. URL atttp://arxia org/ abs/1708.071 07747. arXiv: 708.07747 [cs, stat].
[54] Yann LeCun, Corinna Cortes, and CJ Burges, MNIST handwritten digit database. ATT Labs, 2010. URL http: /ttt:://aan..eee lecun, homm/xxbbmniittt
14

---

A Supplementary background
A.1 f-divergences
f-divergence between two probability distributions P and Q over a common support X' is defined as:
D)(P)1Q)=
- a(x)))
p(x)) g(x)
da,
(14)
where p(æ) and ((x)) denote the densities of and Q, respectively, and f is convex function such that f1) = 0. Any D, admits dual variational form [37], with T denoting the set of all measurable functions R:
DP(P)))
sup TET
-Ep
(15)
The properties of f-divergence are as follows:
Non-Negativity: For any two probability distributions P and P on X. we have Dy(PI >
0. The equality holds if and only if P P.
Convexity f: f(u)) is convex function, then (P(P)) is convex in the input distributions P and P.
Non-Symmetry: Generally, f f-divergence is not symmetric, i.e., Dy + (PIP))
Linearity: f-divergence is linear in f, i.e., for any two convex functions fu( and fz(u). and for any two real numbers a and the divergence P = aD,, (PIIP) 6D,, (PIIP)
Specific choices of the function in the f-divergence definition can yield various well-known divergence measures. We will review some them of the principal properties. For every divergence, report here the generator function its convex conjugate f*, the domain of the convex conjugate dfm(f"). the typical activation function used to ensure that (()) e dom(f*) and finally the optimal discriminator 1"*
Kullback Leibler:
Notation: DKL.
Generator Function: f(u) ulooguu..
Convex Conjugate domain: dom((") = TR
Convex Conjugate: (t) exp(t 1).
Activation: v + 0,
Optimal discriminator: (x)) = 1 log
p(a) p(x)
Reverse Kullback Leibler:
Notation: DIKL.
Generator Function: (u) = - -log(u),
Convex Conjugate domain: dom(f" (f*)
Convex Conjugate: (1) = -1 log 1,
Activation: 4 expet
Optimal discriminator: T*xx) = f(m)
15

---

Pearson:
Notation: xx2,
Generator Function: (u) = (u - 1)?
Convex Conjugate domain: dom f = R,
Convex Conjugate: (t) = f² / /4+1
Activation: v v,
Optimal discriminator: (x)x) =
(p(m) pla)
1).
GAN:
Notation: DAAN.
Generator Function: f(u) uloo(u)) (u log(u (u + 1),
Convex Conjugate domain: dom( R
Convex Conjugate: (1) = log (1 -xp(()),
Activation: V + - exp log (1 - exp(v)),
Optimal discriminator: (x)x log p(n)
p(a)
Total Variation:
Notation: Drv.
Generator Function: f(u) 2 1/.
Convex Conjugate domain: dom(f" (f*) [- 21
Convex Conjugate: (1) 1,
Activation: tanh(v),
Optimal discriminator: T (xx sign p(w)
1). p(
PR- PR-Divergence:
Notation: DA.mm for À 0,
Generator Function: fu) max(Au, 1) max(A, (A, 1),
Convex Conjugate domain: dom(f*) [0, A].
Convex Conjugate: fx (t) t/X for À S and fx (t) t/A + A - -1 otherwise,
Activation: v + (v)), where a is the sigmoid function,
Optimal discriminator: T (x) = Asign
p(w) p(z)
A.2 Bregman Divernce
The Bregman Divergence under strictly convex function f R R with continuously differentiable interior, between two points x and y in the interior of the domain of f. is denoted by Breg, and is defined as:
Breg, (x. 3) f(x) f(y)
(Vf(w), 8)
where Vf(y) the gradient of at y. and (., .) the inner product in R". follows some properties
as distance metrics:
Non-Negativity: For any æ, y the interior of the domain of f, we have Breg (2,, y) 2
The equality holds if and only if x y.
16

---

Convexity: The Bregman Divergence Bregy(æ, y) is convex in its first argument. That is,
for any #2, y in the interior of the domain of f and any E [0, 1], we have:
Breg (tx1 (1 2) y) S / y) + (1 B) ( y) (22.
Non-Symmetry: Unlike some other distances or divergences, the Bregman Divergence is not symmetric, meaning that in general. Bregy (z. y) & Breg, (v.
Additivity: If f(x) = fz(1)) +... + fa(aa) where each f. is convex function, then the Bregman Divergence decomposes into of Bregman Divergences, i.e., f y) Breg,, (21, M1) +..... Bregy, (Ia; ya).
Taylor Approximation: Bregman Divergence essentially the error of the first-order Taylor approximation of f around the point y at the point æ.
Connection with Dual Functions: If f convex, then it has convex conjugate (or dual function) f The Bregman Divergence Breg (a:, y) is related the Bregman Divergence Br (w), fx))) between the gradients of f and y, where the gradient map f serves Legendre transformation between the primal and dual spaces.
B Proof and supplementary for Section 4
B.1 Proof for Theorem 4.3
We have to prove that ((A) can be written as function of an f-divergence for any À E R+ First we can develop the expression of (A):
a(A) = / min (Ap(x), dæ
(16)
{ p(xz) dx p(x)
((x) min
(17)
For this integral to be considered as an f-divergence, we need f to be first convex lower semi- continuous and then satisfy f(1) 0. However, for every a, E R, the min satisfies min(a, + b max(a, b). Therefore,
a(A) / ((x)
X
p(xx ((x)
p(z) p(x)
+ max
dæ
(18)
/ max
X p(xx)
p(x)d +1
p(x)) dx
(19)
1 / ((x)) max > p(xx)
dx
(20)
f(x))
Thus, we can take f(u) = max(Au, 1) max(A, 1) such that f(1) = O. The precision becomes:
a(A) x 1 / p(x)f((
p(x)) max(A, 1) / f(æ)
(21)
5(x))
mmnnn,, 1) / f(x)
p(xx) dæ min(A, D P).
(22)
((x))
Consequently, (A)) can be written as function of an --divergence Dxven with f(u) = max (Au, 1) max (A, 1).
Now we prove the converse. Suppose there exists a strictly decreasing linear* function h: [0, 1]
R and an f-divergence D, such that h (PI) P)) (P(P))) for all P, P e P(X).
For P P. we get from the definition of as that a (P min(A, 1). Hence,
0 (P h(ax(P))))) h(mmn 1))-
*We omitted the critical constraint of h being linear in the original statement of the theorem in our submission by mistake. We apologize for this oversight. We stress again that the first part of the Theorem, which is the most important part, remains completely unaffected.
17

---

Combining the above with the fact that is strictly decreasing linear function, we see that for any fixed X, h must be of the form, h(u) = cx (min(A, 1) u). where cx > 0 constant. Now,
Dy(PIPP) h(ax(P] P)) - (min(AA, ax P)) Cx CxDx)enPP1P))
where the last equality follows from the first part of the theorem, which shows that ax P) min(A, 1) xa (P|P) P). Rewriting the above inequality, we get the following.
1 - CA = -
X (PIP)
By the uniqueness theorem of f-divergence f(u) = S2(u)) + +22(u 1) for some constants
Ciy O2 E R.
B.2 Proof of Proposition 4.2
the generator function f of the Precision-Recal Divergence is f(u) = max(Au, 1) max(A, 1)
then its Fenchel conjugate function is:
(*(t)
sup {tu f(u)} max(A, 1) + sup {tu max (Au, 1)} wfdom(f)
(23)
NER+
If or < O, then the SuPuER+ (lu - max (Au, 1)) = xe for respectively u - xo and + -00. The domain of f is thus restricted to [0, A). Thus for 0 S f. S X, the supremum is obtained for a = 1/A since is in the sub-differential of the function 1/A as Figure 6(b).
6M-1
69-1
R60
AA10
3/2
9A/10
Function f(u) for different values of A.
(b) Function u max(Au, 1) for values of between and
Figure B.6: Graphical illustration of fx and f*- Both piece-wise linear
Consequently the Fenchel conjugate is:
[t/A (t/A
i
E [0, (*(t== max
(24)
1
otherwise.
Finally, the optimal discriminator uut by taking the derivative of f in p(x) p(a)
we get:
p(x)
A if p(x)
1/A, p(m)
(opt(a)) v.
(25)
((x))
otherwise.
18

---

Then we can compute the compute the reverse Dxrs:
[Mar)
Da ([(PP)
p(a:).x
dx
(26)
(((x)
x
max(Xp(()) (2), p(z)) p(z) max(A, 1) da
(27)
A (/x max p(.))p(())))))))())))) p(a)//)da max 1/2))
(28)
X / ((x) max(1,
P(æ)
()) - (ax) max(1, 1/2) dx
(29)
p(a))
p(æ)
À X
f1/A
dæ
(30)
((ar)
XD AD P (PIIP)
(31)
With this results, we can show that
/ |m(x)) - f(xx)) dr
Drv(P||P)
(32)
= f max(p(x)) p(xx), 0) + max(f(x)) 0)dx
(33)
Then since D1-re(P112 Sxx (ax(((()),paa)) p(z)dx Sxx max(f(x)) p(x),.)ds and
Diea (P] = Dr.a (P)PP) we have:
Drv(P][P) = D (PIIP) + (P
(34)
= 2D P).
(35)
B.3 Proof of Theorem 5.2
From now on, assume the support of P and P coincide. For any T X' + R,
Di (PI P) = ExLP p [d (x)] - E.~ 1f (d()))
p(z)) d (x) f (d(x)) p(x)
E
Let Topt sug f.r (P (P) For any T X IR.
(P(P)PP) - f.T
P] = Diual
(PI - Dduall P)
p(z)
(TmPt T(x)) f (I)) + fr (T(z))
Ep
p(x))
It is known that for all x X' we have fz (Tont (x))
p(0)
p(x)
-
Ep (T))t()) (ToM - T(x)) fr (x)) + f" (T()))
Recall that for any continuously differentiable strictly convex functions, the Bregman divergence of I is Breg, (a, = [(a) f(b) - (V/(b), a b). So we have
Dy(P1PP) - P) Rs [Bregy- (T(x), (x)]
Let us now use the following property: Breg, (a,b) Breg,- (a*, b²) where a" Vs(a) and 6" v(b)..
19

---

D)(PI|P) (PIP) =Ep Breg, (Vf(((())) (f
= EPp Bregg
(T)
p(z)
p(x)
Let us define (æ) (a) as estimator of p(a)//((a). So finally, we have
(P (PII)) Ep Breg, rr(x), p(x:) p(x))
B.4 Proof of Theorem 5.3
Now assume that f is j--ttronlly convex, then Bregy (a,b)
NV
||a b1l
Ep Bregg (r(æz) p(m) ] < and if f is p-strongly convex, then p(x)
Ep
r(zz)
p(x)
2e I
(36)
f(z))
p
Consider an arbitrary f-divergence Dy(PIIP) = و (g(d0) dP g (r(x)) dP. Then,
dP.
Define De
(PII
dF
p(x)
- g.T
(PIIP)
Ep
g (r(z))
f(x))
p(x)
g (r(z))
p(x)
(a)
S Ep 10 |e(x)]
o EE [|{(x)]]
(
0/ Ep [e(x)]]
(9) 2c 0 p
where (a) follows from the o-Lipschitz assumption on g, (b) follows from Jensen's inequality and
finally, (c) follows from equation (36).
B.5
Proof of Theorem 8
Let C R + be C² function and take Umin and Umax- The goal is to express f(u) for all [Uminy [maxx] weighted average of fPR.
1/Mentn
Vu CR (((A)fRR(u)d))
("()) [max((u, max (A, 1)) dd
(37)
11/emax
1/umax
20

---

First, let us assume that Umin I 1 and Umax 2 1, then the terms can be decomposed and the integral split to evaluate the max:
[ 1/maaax
//mmin
1/
1/Main
(A) max(Xu, 1)dd
c'(2) max 12,1)
(38)
///Waax
1/a
f d"()) 1jdA
//Maiin
e"()) max(Xu, 1)dA +
1/
// ("() (A, 1)
max (2,
1/ //maxxx
(39)
Sa e'(jjx+ + fym d(A))aud -
1/u
S J1/Memax
1/ 1/uminn
('(A)d
d'()Adx.
(40)
By integrating by parts, we have: /i/umin e'(2)dd> = (c(N)))))
// d' c(2)d) so
11/uma
satisfies:
1f /'tmii ((A)SRRRu)d
V/u
//Manin
f
((A))x+uld(() /
Mumin
c((A)d
(41)
1 Muuaa
1/a
1/Muiin
d'())d- - (c(A)))) Mniin
c(())d
1/
[d' (1) 1/umax
1/u
+ +uld(()))l/M
//mm Mmmin u [o(A)]]
//
(42)
1/u
[c' (A)li/
[Cc(A)))) 1/mmia + / Mhmin
1
1
1 c(())
= =d'
c((0) + uc'
ucc
WC
+ uc
Umin
Ummin
1
)
+ +c((0) - c'
+ tc((1) x x
c(1)
Umin
Vamin
Umin
(43)
d
(u - 1) + uc ( c(1).
c
(44)
Munin
Umnin
Umin
We would like fi/siass f-divergences generated by and g are equals if there is a E R such that f(u) g(u) c(u
/2/Mmin fA (T)))F to be equal to f between Umin and Umax- But since two
1), the Divergence generated by /1//min c((A) (u)dA is equal to the divergence generated by (u)dd
uc (1) c(1). Therefore, we require e to satisfy:
Yu [umin: Umax], f(u)= = uc ( c(1).
By differentiating with respect to u, we have:
1
('(x) = lim (c' c(A)] +
c'
(45)
X0000
And finally:
1
1
{"(w)= u
+
c'
+
(46)
u
3
1
(47)
Consequently, with À = 1/u, we have that:
(1)
d'(2)
J"
(48)
23
21

---

With such a results, with m = minx
[( p(z))
and M max
p
we can write any f-divergence
p
as:
Dy(PIIP) = //x f(a)))
p(x)) ((xr)
da
M
((x) p(x))
((x)
x 8"
fx
dAdæ
x
(((x) p(x))
PPR
dAdæ
X3
M
(((z)
f(a=)
dæ dd
X
p(x))
M
X f"
(
Dx
B.6 Proof of Corollary 4.5
In particular for the DKL f(u) = u log u, therefore f"(u) = 1/u which gives:
(KL(PIIP)= S. M
1 x
(49)
And for the DeKL. we can either use Equation 48 with f(u)
logg or use the fact that
D P) (el/)ra(Pll)))..
DikL(PIIP) = f.n M Dxea
(50)
C Minimizing the Precision-Recall divergence
In Section 5, we outline novel strategy to train models for challenging f-divergences. Rather than directly employing the f-GAN framework and minimizing the dual variational form of Px.PRs introduce an auxiliary function, denoted by which is easier to train. Practical choices for could include functions such fKL or fx2, which have been shown be easily trainable.
Using this approach, discriminator T is trained with the objective of maximizing Ddusl This is achieved by drawing samples from both P and and computing the estimate in the dual form. Under optimal conditions, we find Vg (Tops (a>) = pfx))/((x). enabling us to estimate the f-divergence based on I' using what term the primal estimate.
The primal estimate is defined as Derimal (PII = fxx P())] (r(æ)) da,, where r(w) = Vg*(T(x)) This strategy and its corresponding training procedure are visually represented in Figure C.7.
M(a)
tple)
(PI()) 98
T
Rx)
p ④ P Φ
G
primal
(PI
Used to train Used to compute
Figure C.7: Training procedure: the discriminator T is trained based Ddual and G is trained on
Dprimal f.I. f.T.
22

---

We have shown that the choice of g affects the quality of the estimation of Dy. We have shown that to have guarantees on the quality of the estimation, 9 must be strictly convex. We show empirically that g fx² leads to better approximation than g == SKI,- We train 200 discriminators on 20 Gaussian random two-dimensional mixtures to maximize D with g fKL. and 9 fx2. In Figure C.8, we report the results. On the z-axis, we report the training loss quality PPIP) - Dirt' (PI
and on the y-axis we report the estimation quality: (P Dj, D (PIIP)). The dotted lines correspond to the Mahalanobis distance of each set of experiments. In this experiment, we use a linear layers 2-1002--12-2555 neural network with LeakyRelu activation between layers. The last activation is set to match dom()") (see Section A.I). We use learning rate of 2.10 with Adam optimizer.
D (P(P)
Dprimal f/T
(|| P)
fx² fKL
3.5
3.0
2.5
2.0
1.5
1.0
0.5
D P P
D Ddual (PI) P
g
0.00
0.05
0.10
0.15
0.20
0.25
Figure C.8: We compare the quality of the primal estimation using two discriminators, one trained using the DkL, and 2x= over 200 experiments. The lower the point is on the y-axis, the better the estimate. As we can see, even distant dual estimates of the fxa (points on the right side of the chart) tend to provide better estimates of the primal.
We observe in this experiment that 1) the difference between DKL and its dual is usually lower (the blue circle is lower on ær-xii) and yet 2) the estimation of DD is usually better with g = fx² (the red circle is lower on y-axis). These observations corroborate our claim that employing fx² the auxil- iary divergence a more advantageous choice compared to fKL- Although both divergence measures exhibit satisfactory performance in the context of the MNIST and FashionMNIST experiments, find fx² to demonstrate superior stability, reinforcing its suitability in this training framework.
Complete version of the algorithm In Section we introduce condensed version of our training algorithm. For practical purposes, however, we implement a more refined variant of this algorithm using stochastic gradient descent, detailed in Algorithm This approach involves segmenting the dataset into batches of size N and computing the distinct losses for each individual batch. Our algorithm is closely related the GAN training procedure, operating on an iterative principle of alternating updates to the discriminator T and the generator G until convergence state is reached. In particular, our method differs from the standard GAN training protocol in two key aspects. First, the generator G in our case is trained based on an estimation of distinct f-divergence. Second, this specific /ddivergence calculated in terms of its primal form, rather than dual form, a marked departure from the established GAN training practice.
23

---

Algorithm 1 Stochastic Gradient Descent for the two-step approach for minimizing Da.m. For each batch B composing the dataset D, the discriminator T is trained on Lr. the empirical estimation of D P) and the generator trained on Lgs the empirical estimation of D (P)) P).
Input: Generator G, Discriminator Dataset D
{ fa
for epoch e e= I.......... do
for B E do
CT, LG + 0,0
for xeeal ..... N E B do
for i = to N do
Generate 2;; fake = G(z) with (0,, Id)
LT s LT + T l (a.ake)
end for
Update parameters of T by ascending the gradient V CL. i N do La La 9 (VI (T(:akk))
end for Update parameters of G by descending the gradient V VL.. end for end for end for
D Experiments
D.1 Naive Approach
In this section, we juxtapose the straightforward method with our technique delineated in Section 5. We achieve this by evaluating the optimization process of two identical G models: one is trained using the conventional approach and the other using our methodology. In reality, there exist two significant variations between these two training procedures: training losses and the final layer of the discriminator T. In both procedures, we use (T(((=)), implying that the co-domain must be encapsulated within dom(f*). More information on activation can be found the Appendix A.I. Aside from these disparities, section to train the models on CIFAR-10. 10. The training procedures are compared in Figure D.9. To evaluate training procedures, in the naive approach or the primal estimation of Dxvmu in our method. To track the evolution of the discriminator, we plot discriminating predictions Precisely, we compute the accuracy on batch of real or fake images, how many the discriminator achieves identify such. Finally, we plot the Precision and Recall computed at each epoch.
utilize the identical training process outlined in the previous
plot the loss Lg. used to train G, either the dual estimation of D
the input conditioned the class (real fake).
We observe that for varying values X, the discriminator undergoes the training process using the naive approach, leading to unsatisfactory estimation of Ddual

---

Loss on G: Cr:
Accuracy of (in 80)
Precision
Recall
000
11.00
602
2008
40.30
1000
20000
(200))
20000
40.03
3800
0340
on G: Co
Accuracy of 56)
I Precision
Recall
3.300
012
60
001
001
1200
00400
(0000
2600
99003
23000
1200
Less on G: La
Accunay (in %3)
Precision
Recall
010
--000
62-
-15
2000
6086
91331
1000
2400
100
1340
$6000
2000
81.00
9X00
000
28.00
1380
1006
Enenaiione
Iteratioon
Boratinn
Itaaatione
Figure D.9: Results for training BigGAN models CIFAR-10 10 using the naive approach (in blue) and our method (in red). For À = 0.2, À = and 5, the loss represented by in learned. We obverse that for the naive approach, this loss is constant. Then, plot the accuracy of the discriminator. We plot the accuracy conditioned on the class: in the dotted line the accuracy on the images of the dataset real and in the solid line the accuracy the generated images x fake Then plot the Precision and Recall during training.
D.2 Synthetic data: 8 Gaussians
For the synthetic data, we use RealNVP [14] for the generator G. We use an 8-coupling step composed of each of 2 linear layers 2-256-2 with LeakyRelu activation in between. For the discriminator, we used a 4 linear layers 2-1004-512-22 network with LeakyRelu activation between layers. For both, we use Adam optimizer with learning rate of 2.10-5 for G and 1.10-1 for T. G has 540k parameters and 660k for T.
A-
0.5
neural
0.1
10
DxL
Dan.
Then, to estimate the PR curves, both methods from Sajjadi et [44] and Simon et al. [47] are developed for image dataset. To estimate these curves, we use our own estimation of Dx-eR- We computed Dx-Fn. for À tan(f) with 0 between 0 and 2/2. The observations from the generated curves underscore a compelling finding: Fach model has been effectively optimized for unique, specific X, resulting in its superior performance on the correspond- ing D Dx.R- This indicates clear correspondence between the intended optimization and the achieved performance, suggest- ing the efficacy of training algorithm in optimizing models for specific performance metrics. Such a successful matching of model optimization underscores the feasibility of tailoring model training according to specific precision-recal trade-off objective.
0.0
Figure D.10: PR Curves of the models in Figure 1. shades of blue, the curves correspond to the model trained to min- imize D Dx-PR- with dark blue for À 0.1, medium blue for X 1, and light blue for X 1. The green curve cor- responds to DKL and the red curve to DIKL-
D.3 MNIST and FaShionMNIST
The training procedure for MNIST and FashionMNIST is strictly the same. For both we use multiscale glow [27]. The model has three levels processing images size 4 x x 16, x 64 8 8. Each level has 16 blocks of affine coupling with 3 layers of 512 channels of convolutional
25

---

operations, leading to total of 85.2M parameters, For the discriminator, we use a linear layers 1024- 1024-512-256-1 neural network with LeakyRelu activation between layers. with 1.7M parameters. Both are trained with Adam using learning rate of 1.10 for 7 and 1.10-6 for G. For both dataset, we train model for 250 epochs using maximum likelihood estimation (MLE) in GPUs V100 (~> 200 hours). The models are then fine-tuned with their different losses on 12 V100 GPUs for 30 epochs (~> hours). For two epochs we train the discriminator only and then both. For MNIST, we report results graphs in Figure D.11, the quantitative results in Table 4 and samples in Figure D.3. For FashionMNIST, we report the results as graph in Figure D.13, the quantitative results in Table and samples in Figure D.3.
Table 4: Quantitative evaluation of various Glow [27] models on the MNIST and FashionMNIST datasets. Models differ by their choice of y, ,, pg. and The performance metrics include the FID (L), P (L) and R (t). The best model is highlighted in bold. FID is calculated for 50k samples.P and are culculated for 10k samples and k 3.
Model
MNIST
FashionMNIST
D,
Dj
FID
P
R
FID
P
R
MLE
7.68
70.03
76.52
66.64
58.19
47.35
DKL.
DKL.
7.76
70.68
77.19
71.72
51.44
45.61
DILL
DILL
10.50
74.64
67.21
51.65
67.68
40.76
Dx²
DA-eR
0.10
6.84
78.24
72.11
48.11
66.57
45.80
D2²
Daver
6.48
78.84
71.02
41.97
46.62
D2²
DA-eR
0.50
6.57
79.09
70.26
46.04
71.20
46.37
Dx²
DAvrn
1.00
7.95
79.55
69.08
52.88
73.32
36.44
D/²
Dxvra
2.00
10.01
80.37
67.84
101.33
76.47
39.40
D₃a
Daira
5.00
15.31
81.34
67.00
118.54
73.44
35.14
D22
DAvRE
10.00
20.88
81.71
66.62
91.20
72.14
19.48
DKL DAvea
0.10
4.66
76.06
73.88
42.85
66.48
49.73
DKL DA-ER.
0.20
4.45
76.92
71.92
48.25
72.87
49.52
DKL Da.en
0.50
5.94
77.98
71.42
54.12
71.28
41.70
DKL DAvrE
1.00
8.21
79.92
70.02
62.33
62.58
12.89
DKL DA-iE:
2.00
9.40
79.89
67.15
64.74
72.78
41.41
DKL DAvrE
5.00
14.01
82.08
66.13
83.25
73.12
33.80
DKL
DA-ER
10.00
27.61
83.20
65.09
79.37
70.74
27.05
2477
25
PR-KL KL
2.30
E375
6.72
MLE
KL.
3.00
TKL
5370
18
1071
117
Figure D.I1: MNIST: Glow models [27] are trained for different X. From left to right, we plot IS (t), FID (L), P (t) and R (t). IS and FID are calculated using 50k samples, and P and R calculated using 5k samples with & = 3 using Ky Kynkäänniemi et al. [28]'s method. For comparison, we also report models trained with MLE (in black), for DKL (in blue) and for D.KL red.
26

---

6
2
2
4
/
7
4
6
1
4
8
9
6
6
3
5
2
8
7
3
6
9
2
6
7
8
6
6
9
8
2
2
9
&
6
7
7
9
3
7
7
3
8
9
3
8
4
5
2
O
7
Σ
7
7
7
5
1
O
5
9
8
7
3
9
7
8
6
(
7
7
6
7
4
9
4
5
&
8
3
9
5
4
2
/
7
2
1
4
7
7
8
3
C
5
O
/
5
7
0
6
3
3
5
{
7
3
6
/
O
7
8
6
MLE
(b)
À
0.1
9
7
8
6
8
3
6
\
4
9
9
6
2
7
7
4
1
7
4
/
3
7
5
7
6
/
9
\
3
7
7
0
/
7
7
9
1
/
7
9
7
6
3
7
8
θ
6
/
9
6
9
/
/
6
8
4
8
7
8
9
1
7
/
9
/
9
7
7
6
3
3
8
O
6
O
9
9
9
1
9
6
7
5
7
/
4
/
7
7
9
7
9
1
0
4
8
9
/
9
7
7
7
/
(c)
=
(d)
X
10
Samples are drawn from different models (only models trained with = From 12(a) to 12(d), we observe that visual quality improves while diversity decreases. Models geared towards high recall, specifically MI.E and those with = 0.1, are found to generate wide variety of samples. However, these are characterized by lower precision, and approximately 20% of the generated samples are incoherent. On the contrary, the model trained with A = 10 appears to focus smaller subset of modes, demonstrating higher precision but limited diversity. Specifically, this model primarily generates classes 1, 6, 7, and 9.
Figure D.12: MNIST:
f2²).
FID
P
155
1050
12
PR C5a2
105
PR-KL
1000
MLE
1000
KL
+KI.
03
307
11
Figure D.13: FashionMNIST: Glow models [27] are trained for different X. From left to right, we plot IS (t), FID (1), P (t) and (t). IS and FID calculated using 50k samples, and P and calculated using 10k samples with k = 3 using Kynkäänniemi et al. [28]'s method. For comparison, we also report models trained with MLE (in black), for DKL (in blue) and for DAKL in red.
27

---

MLB
(b) 0.1
M
IT
N
7
(c) A =1
(d) A 10
Figure D.14: FashionMNISTT: Samples are drawn from different models (only models trained with 9 fy²). From 14(a) to 14(d). we observe that visual quality improves while diversity decreases. Models geared towards high recall, specifically MLE and those with 0.1, are found to generate a wide variety of samples. However, these characterized by lower precision, and approximately 15% of the generated samples are incoherent. On the contrary. the model trained with À = 10 appears to focus smaller subset of modes, demonstrating higher precision but limited diversity. Specifically, this model primarily generates the class "trouser".
D.4 Training BigGAN CIEAR-10 and CelebA64
In this section give the details of the experiments training BigGAN [7] models. To do this, modify the official implementation count 4.3M and 4.2M parameters for CIFAR-10 and 32.0M and 19.5M for CelebA64. CIFAR-10's models are trained on 2 A 80GB GPUs with a batch size 128 for approximately 100k iterations (~ 7 hours), while CelebA64's models have been trained on A A100 32GB GPUs with batch size of 128 for 95k iteration (~~ 20 hours). Quantitative results are reported in Table 2. The graphic representation of the results is in Figures 1.16 and D.20. The training curves representative of FID, IS, P and R during training are reported in Figures D.17 and D.18.
PyTorch² of BigGAN by Brock et al. [7]. G and respectively
28

---

(a) BigGAN Bascline
(b) A = 0.10
(c) ==1
(d) À = 10
Figure 1.15: CIFAR-10: Samples are drawn from different BigGAN trained on the PR-Divergence.
We observe that when À increases, the recall decreases going to various range of colored images mostly brown images.
P
102
130
025
002
R
Appoed
Figure D.16: CIFAR-10 10: BigGAN models [7] are trained for different A. From left to right, we plot IS (1). P (1), (1), FID (4), (f) and C (1). IS and FID are calculated using 50k samples. and P and R are calculated using 10k samples with A = 5 using Kynkäänniemi et al. [28]'s method. For comparison, we also report model trained hinge loss (in black), following the vanilla training procedure.
29

---

Ieception
14-
10-
02-
0.05
200
0.20
0.50
1.00
20300
1008
10.0
1006
3000
1368
(0000)
530)
3.00
4000) Brorations
11:00
003E
3000
.000 Ieeracions
(000)
10000
Teerations
Tterations
Inception Store
FID
Precision
Recall
2.-
1.00
2.20
2-
5:00
10.00
2000
10910
COXE
1986
2000
1340
53003
0.
2000
400.0
(200
20000
J000
Tteratios
Boralima
Itera innn
Figure D.17: CIFAR-10:
Evolution of the IS, FID, P and R during training. We observe that the precision quickly achieves its maximal value and saturates. We can also observe that the model is evicting modes of the target distribution early in the training, R is quickly saturates to different levels depending on X. However, for low values of X, the recall does not increase when À decreases, indicating that the model capacity limits the maximum recall.
Iax eaception
FID
Precision
Recall
38-
0.-
135
01.0
48-
64-
0.-
18
(1500
0.15
0.67
33
07
1.00
30
1800
(869)
00
Exeadions
Torrat/one
Trerations
Itarations
Incepttion Store
FID
Procision
Roolll
1.3
1.50
0.00
48
13
0.2
5,00
39
Excations
Banaloon
Issations
Figure D.18: CelebA Evolution of the IS, FID, and during training. We can observe that precision quickly achieves its maximal value and saturates. We can also observe that the model is evicting modes of the target distribution early the training, quickly saturates to different level depending on X.
30

---

(a) BigGAN Bascline
(b) À = 0.5
(c) 1
(d) X == 10
Figure 1.19: CelebA64: Samples are drawn from different BigGANs with our apprach. For À = 0.5, the model generated faces with various background, while the model generate only grey, black and brown backgrounds. However there are less artifacts on the latter model.
272
031
0:33
0.75
FID
6.5
Dr Apoii
0.2
0.0
Figure D.20: CelabA64: BigGAN models [7] are trained for different X. From left to right, plot IS (t), P (t), D (f), FID (L), R (*) and C (1). IS and FID are computed using 50k samples and P and R are computed using 10k samples with & using Kynkäänniemi et al. [28]'s method. For comparison, we also report model trained hinge loss (in black), following the vanilla training procedure.
31

---

D.5 Fine-tuning BigGAN on ImageNet128 and FFHQ256.
In this section, fine-tune the pre-trained BigGAN models. For ImageNet128, of the model trained by Brock et al. [7], while for FFHQ256 we use a model pre-trained by our self, explaining why the FFHQ models is under performing. We train models V100 GPUs with batch size of 128 for 10k iterations each. The graphic representation of the results is reported in Figures D.21 and D.22 and samples shown in Figures D.5 and D.5.
the weights
033
110
610
0113
172
0.70
1.10
Appodh
0.30
110
0.05
1.00
Figure D.21: ImageNet1 28: BigGAN models [7] are trained for different A. From left to right, we plot IS (t), P (?), D (f), FID (4), R (f) and C (t). IS and FID are calculated using 50k samples, and P and R are calculated using 10k samples with l = 5 using Kynkäänniemi et al. [28]'s 's method. For comparison, also report model trained hinge loss (in black), following the vanilla training procedure.
18
P
D
087
32
000
65-
0.5
1.00
Apped
0.00
Figure D.22: FFHQ: BigGAN models [7] are trained for different X. From left to right, we plot IS (t), P (t). (t), FID (L), (f) and (f). IS and FID are computed using 50k samples and and computed using 10k samples with & = 5 using Kynkäänniemi et al. [28]'s method. For comparison, we also report a model trained hinge loss (in black), following the vanilla training procedure.
32

---

(a) BigGAN Bascline
(b) A= 0.2
(c) x=1
(d) 5= 5
Figure D.23: ImageNet1 28: Samples are drawn from different BigGANs trained on the PR-
Divergence.
33

---

(a) BigGAN Bascline
(b) A 0.5
(c) x=1
(d) À = 2
Figure D.24: FFHQ256: Samples are drawn from different BigGAN trained on the PR-Divergence.
E
Training the AUC
In this paper, we propose method to optimize any trade-off between precision and recall. In practice, we optimize model maximize any (P(l))) We could also optimize for the area under the PR-Curves PRD(P, (),P, P). In this section, will show how compute the AUC in terms of x (P) and train a model on a small dimension dataset and compare the results with the model trained on several X. First, we must compute the AUC:
Proposition E.1 (AUC under the @PRD(P, PJ).
The area under the curve is:
+00 a
())
AUC
Yp
(51)
X2
Proof. The AUC can be computed by integrating with respect to an angle 0 in the first quadrant:
1//2
(0)
1/2 (2)
AUC
udud0
d0.
(52)
1=0
Juzo
2
34

---

Therefore, with A = tan we have r(0) = Otan( (PIIPP/ sin & (see Figure E.25). Thus:
x/2 tan( (0) (PI) P)2
AUC
d0
(53)
sin 2 0
T/2
(tan (PIP)) 11 tan 0
(54)
to0
a)(P[| (1 2)
1+ dd with dd (1 tan (an2
(55)
(PI
As dA.
(56)
Using such expression, we can estimate the AUC and train model to directly optimize the AUC. It loses the focus on a specific trade-off between precision and recall, but instead optimizes overall performance. We trained a RealNVP model on 2D 8 Gaussians example, similar to Section 6. We observe Figure 27(d) that while the model trained to minimize the AUC performs poorly in terms of X = 0.1 and À 1. However, we can clearly see that the AUC is greater and the model performs better in than the model trained for X = itself. We can also observe in Figure 27(d), the resulting model.
A=1
1.0
0.8
A
A 10
ax
AUC
(10)
(P P))
B
A 0.1
8
Illustration of the change of variable compute the AUC. Instead of parametrising the frontier @PRD(P, P) with
Figure E.25:
Figure E.26: PR-Curves corresponding the models represented in Figure E.
E E U [0, oo]. we take 0 [0, ] with tan 0.
(a) = 0.1
(b) A
(0) A= 10
(d) AUC
35


Reviews:
[{"review_id": "Of64nlh4in", "reviewer": "Reviewer_jwf3", "summary": "This paper proposes a training method for generative models (normalizing flows and GANs), which can control the precision-recall trade-off of the generative models. The method is to design a new divergence named precision-recall divergence to bridge the precision-recall curve and the f-divergence. Then, the generative models optimized using the precision-recall divergence can control the precision-recall tradeoff by adjusting the hyperparameter lambda. The experiments show that the proposed method can control the precision-recall tradeoff and improve the performance of the baseline model (i.e., BigGAN).", "questions": "In addition to the theoretical one, what’s the practical advantages of the proposed method comparing to the existing methods such as truncation, rejection sampling, and combination of different f-divergences (e.g., KL + reverse KL)?", "limitations": "The authors have addressed the limitation in the paper.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper proposes a divergence that is directly related to the precision-recall curve, and the proposed method can control the precision-recall tradeoff during training. \n2. The results show that the proposed method can control the precision-recall tradeoff.\n3. The results show that the proposed method can improve the performance of BigGAN.", "weaknesses": "1. I think some combination of different kinds of f-divergence can also control the precision-recall tradeoff during the training. For example, KL-divergence + lambda * reverse KL-divergence. The paper does not compare with this kind of method, and does not explain the advantages of the proposed method in practice.\n2. Although the existing methods including truncation and rejection sampling can not control the precision-recall during training, they can control the tradeoff through post-processing. The authors do not compare with these methods. What are the practical advantages of the proposed method compared to these methods?\n3. The proposed method is not evaluated on larger resolutions (e.g., 512*512) and larger datasets. The ImageNet dataset is just used in the fine-tuning setting. \n4. The proposed method is only evaluated on the BigGAN backbone. I am more interested in the use of StyleGAN, since StyleGAN is more widely used. \n5. In Table 2, StyleGAN-XL performs much better than the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to express our gratitude for the time and effort you dedicated to reviewing our paper. We appreciate the constructive comments. We would like to address the concerns you raised:\n\n**1. Combination of Different Kinds of $f$-divergence:**\nYou rightly pointed out that a combination of different f-divergences, such as KL-divergence and reverse KL-divergence, might also control the precision-recall tradeoff. However, a combination of KL and reverse-KL, while being a suitable f-divergence, does not have a easy close form convex conjugate function $f^*$. The conjugate function depends on the Lambert function. So, to train a model to minimize a combination of KL and reverse KL, it would either require  two discriminators (one per divergence) or using a more complex version of our algorithm. While being a nice intuitive idea, combining different divergences is more complex than it appears to be. Secondly, even if we were to train for a combination of KL and reverse KL using multiple discriminators or another method, it is not a priori clear what PR trade-off is achieved by a combination, whereas the PR-divergence makes this trade-off very explicit. We discuss this in lines 193-199, but we will add a paragraph explaining this more clearly in the paper. \n\n**2. Comparison with Pre or Post processing Methods:**\nWe recognize the importance of comparing our method with post-processing techniques like truncation and rejection sampling. Our method's primary advantage is the ability to control the precision-recall tradeoff during training. The other methods for sampling [1] or instance selection [2], can only shift the trade-off to improve the Precision while degrading the Recall. We have added in the general rebuttal some results for truncated latent distribution on the baseline model (BigGAN) and show that the method marginally improves the precision by limiting the recall.    \n\n**3. Evaluation on Larger Resolutions, larger Datasets and more popular models:**\nWe acknowledge the limitation regarding the evaluation of larger resolutions and datasets. Our primary focus is to demonstrate the efficacy and versatility of our method, and we chose the settings that best facilitate this.  Our choice of BigGAN was based on its compatibility with our method and the easier implementation/training procedure of BigGAN in practice. As a matter of fact, BigGAN is often used as a base to test new techniques and approaches [2, 3, 4]. We strongly believe that our experimental set up shows the efficiency of the method for a variety of models and datasets. In particular, a larger dataset, a higher dimension or larger model (like StyleganXL) would drastically increase the computational resources and environmental impact.\n\n\n**4. Practical Advantages Over Existing Methods:**\nBeyond the theoretical advantages, our method offers several practical benefits, the most important being:\n\n**Explicit Trade-off:** The precision-recall tradeoff is made explicit in our approach, allowing for more predictable and controlled outcomes during training. We directly optimize well-established precision and recall measures, which others do not.\n\nAlso, our method provides:\n\n**Minimal Cost:** One of the standout features of our method is that it introduces minimal computational overhead. The complexity of training remains unchanged, ensuring that the benefits of our approach come at no additional cost.\n\n**Improved Recall:** Our method has the potential to enhance recall, a feat that other techniques like rejection sampling, importance sampling, and truncation cannot achieve.\n\n\n[1] Humayun, A. I.; Balestriero, R.; and Baraniuk, R. 2022. Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Values. ArXiv:2203.01993 [cs].\n\n[2] Terrance DeVries, Michal Drozdzal, and Graham W. Taylor. Instance Selection for GANs,\nOctober 2020. arXiv:2007.15255 \n\n[3] Hanxiao Liu, Andrew Brock, Karen Simonyan, and Quoc V Le. Evolving normalizationactivation layers. In NeurIPS, 2020.\n\n[4] Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. arXiv:1903.02271, 2020\n\n"}, {"review_id": "r5qCq7SJX3", "reviewer": "Reviewer_oBuL", "summary": "This paper focuses on the fine-grind optimization regarding the generation precision and recall. More specifically, considering the ambiguity of FID, the authors propose to directly optimize the precision and recall of generated images, by developing a certain class of f-divergence, that is, the PR-divergence. This paper also shows that a linear combination of the proposed PR-divergence can represent arbitrary existing f-divergence. This paper also proposes a practical primal-dual estimation on the PR-divergence, for relieving the gradient vanishing issues during training. The experimental results verify the feasibility of adjusting lambda to adjust the generation precision and recall.", "questions": "1. The notation \\leq or \\geq when comparing two distributions should also be explained. For example, the notation \\geq appears in  P\\geq\\beta\\mu in Definition 3.\n\n2. It seems Theorem 4.4 is not straightforward, where the proof or brief explanation is necessary. \n\n3. The authors propose the g-divergence for the primal estimation. How the g-divergence is chosen in practice?", "limitations": "Please see my weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a unique class of f-divergence named PR-divergence (up to an affine transform) that can explicitly optimize the precision and recall of GANs. The established theory of the PR-divergence is well aligned with the precision and recall of GANs, namely, the PR-curve. The author prove its connection to the vanilla f-divergence. The primal estimation of the divergence has also been provided for the ease of optimization, which is proved to be equivalent to minimizing the dual object in terms of the Bregman divergence. The presentation of this paper is clear for me. Experimental results have verified the effectiveness of optimizing different lambda-valued PR-divergence can alter the precision and recall.", "weaknesses": "One of my main concerns is regarding whether the PR-divergence can impede the generation quality, although it is proved to be able to alter the precision and recall during training. I was wondering, whether optimizing the PR-divergence for some lambda could at least retain the same best FIDs with other existing methods, under the same architecture and batch sizes. For example, as I read from the details in the supplementary material, the authors employed larger batch size (equal to 128) than the default setting (equal to 64) when training CIFAR-10 under the BigGAN architecture. However, the default setting of 64bs already achieved FIDs lower than 10, whereby the results are reported by https://github.com/POSTECH-CVLab/PyTorch-StudioGAN. All the FIDs for CIFAR-10 reported in this paper are larger than 10, even trained by 128bs. Why would this happen? Will the PR-divergence deteriorate the overall generation quality?\n\nIf I understood correctly, the proposed PR-divergence can be well applied to divergence-based or likelihood-based generative models. So I was wondering whether this method can be applied to diffusion models? Also for the normalization flows, did the authors compare the NFs on real-world datasets, other than on the synthetic datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you dedicated to reviewing our paper, and we would like to address the concerns you raised:\n\n**1. Training Settings and FID Scores:**\nWe acknowledge the discrepancies in the batch sizes used for training. Due to the unavailability of pretrained weights for BigGAN on CIFAR-10 (in PyTorch), we had to retrain the model ourselves. The batch size was chosen based on the computational resources available to us. Our primary goal is to demonstrate the tunability of models using our method, and we believe that a minor difference in FID scores (13.37 (ours) vs. 14.73 ([1])) does not significantly impact our main findings. It's worth noting that, as per your reference, most of the FID scores for CIFAR-10 are indeed higher than 10.\n\n**2. Applicability to Diffusion Models:**\nOur method can be extended to diffusion models. While the KL divergence in diffusion models is computed using a closed form between two Gaussians, our PR-Divergence doesn't have a closed form. However, we can approximate it using our technique. To achieve this, the discriminator would need to be $t$-dependent, as seen in some papers [2]. We are confident that this adaptation is feasible, but this would be an entirely different paper.\n\n**3. Training Normalizing Flows:**\nIn our experiments (Section 6), we trained GLOW, a type of Normalizing Flow, on MNIST and Fashion MNIST using our method. Given the large model size of Normalizing Flows, we deemed it unnecessary to train on higher dimensions for the scope of this paper.\n\n**4. Choice of g-divergence and Theorem 4.4:**\n In practice, we utilize the $\\chi^2$ divergence. Theorem 4.4 essentially states that the auxiliary function $g$ should exhibit strong convexity. In Appendix C, we demonstrate that using an auxiliary $\\chi^2$ divergence yields better results compared to KL. We will certainly provide a clearer explanation for the notation $P\\geq \\mu$ and offer more insights into Theorem 4.4 in our revised manuscript.\n\n\n\n\n[1]: Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis, February 2019.\narXiv:1809.11096 [cs, stat].\n\n[2]: Dongjun Kim, Yeongmin Kim, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models. ArXiv, abs/2211.17091, 2022.\n"}, {"review_id": "ZBjQ7Sssap", "reviewer": "Reviewer_hgHq", "summary": "The paper aims to balance the precision and recall of generative models including GANs and normalizing flows. Theoretically, the paper shows that the f-divergences to minimize can be reformulated as sums of PR-divergences, which provides a theoretical explanation between the optimization objective and the precision-recall trade-off. Empirically, the paper proposes a series of techniques to optimize the precision-recall divergence.", "questions": "Please refer to the weaknesses.", "limitations": "Limitations are addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is qualified to be accepted to NeurIPS in the following aspects:\n\n1.\tNovelty: Unlike previous heuristic works on the precision-recall trade-off, this paper established a fundamental theoretical understanding of the PR trade-off. To the best of our knowledge, the induced method is the first work to achieve a good PR trade-off from the perspective of divergence training.\n\n2.\tSignificance: The proposed theoretical results can not only serve as a guide in training generative models but also as an extension of studies on the traditional PR trade-off, thus might be applicable to other tasks like retrieval and recommendation systems.\n\n3.\tClarity: The paper is overall clear and well-written even with the heavy mathematics.\n\n4.\tSoundness: No obvious errors were found in the proofs. The theories are consistent with the empirical results (Fig. 4). \n", "weaknesses": "The main concern is the proposed optimization algorithm involves an additional hyperparameter $\\lambda$, which needs to be tuned for a new dataset. Therefore, it might be an alternative to optimize the Area Under the PR Curve (AUPRC) instead of a single point in the PR curve, which has been studied in ranking problems [1,2,3,4]. The paper could be more instructive if the theoretical results can inspire the extension along this path.\n\nRef:\n\n[1] Wang et. al. Momentum accelerates the convergence of stochastic auprc maximization. ICML, 2022.\n\n[2] Wen et. al. Exploring the algorithm-dependent generalization of auprc optimization with list stability. NeurIPS, 2022.\n\n[3] Cakir et. al. Deep metric learning to rank. CVPR, 2019.\n\n[4] Chen et. al. Ap-loss for accurate one-stage object detection. T-PAMI, 2020.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply appreciative of your thorough review and the positive feedback on our paper. Your insights are invaluable, and we would like to address the concern you raised:\n\n**Optimizing for AUC:**\nYour suggestion to optimize for the Area Under the Curve (AUC) is indeed insightful. In fact, we have previously explored this avenue for lower-dimensional datasets such as 2D, MNIST, and CIFAR-10. Our approach was formulated as maximizing\n\n$$\n\\mathrm{AUC} = \\int_{0}^{+\\infty}\\alpha_\\lambda(P\\Vert \\widehat P)^2d\\lambda.\n$$\n\nBy parameterizing with  $\\lambda=\\tan(\\theta)$ where $\\theta\\in[0, 1]$, we trained models to optimize the AUC. However, our observations indicated that the model behavior was closely aligned with $\\lambda=1$. Given that this did not truly lead to  a trade-off between precision and recall, we chose not to include this study in the main paper. However, considering your interest, we are more than willing to add a dedicated section in the Appendix detailing our experiments with AUC optimization.\n"}, {"review_id": "VOWWwwcWmF", "reviewer": "Reviewer_Pocn", "summary": "The paper introduces a technique for training generative models using an objective which approximates so called PR-divergence.  By adjusting a parameter, the paper claims that it is possible to train an array of models, from those that prioritize  high precision to those that prioritize high recall (mode seeking vs mode covering), as well as more balanced models. The supporting experiments involving BigGAN training on CIFAR10, CelebA64 and finetuning on ImageNET128 and FFHQ256 are described. ", "questions": "1. What is the complexity of the training procedure?\n\n2. In several places throughout the paper the likelihood ratio $p(x)/\\hat{p}(x)$ is used. How to deal with regions with $\\hat{p}(x)=0$  in each of the  encounters of this ratio ?\n\n3. In the experiments reported in Table 2, was only a single initialization used in each case to test the proposed method? ", "limitations": "The limitations related with the complexity of the training procedure are not provided.  Also the limitations concerning the convergence of the proposed minimax procedure based on two different f are not discussed. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Authors consider the family of f-divergences and show that , given a trade-off parameter $\\lambda$, for  a particular element \n${\\cal D}_{\\lambda -PR}$  of this family, the minimization of  this element is equivalent to the maximization \nof the value at $\\lambda$ of the first component of so called PR curve from Saijidi et al [40]. \n\n2. As the minimization of ${\\cal{D}}_{\\lambda -PR}$  via f-GAN approach would fail, the authors describe a way to minimize a certain approximation to this objective and give an estimate for the error of this approximation. \n\n3. The provided experiments show that  for NFs-GLOW on 2D synthetic dataset, MNIST and FMNIST, and for BigGAN on CIFAR-10, CelebA64,  the training with the approximation  objective and small $\\lambda$ leads indeed to models with better mode covering, and, to a much lesser extent for BigGAN, the bigger tradeoff parameter $\\lambda$ leads to better quality of generated samples. ", "weaknesses": "Although the approach looks interesting, the paper raises several red flags that prevent me from endorsing it. \n\n1. The experimental validation is limited, especially the gain in precision, i.e. the sample quality, for high tradeoff is insubstantial, compared with other methods. One example of this is the quality of samples on Figure D.11.d : all 100 supposedly high quality samples, at the highest tradeoff $\\lambda=20$, are clearly of bad quality. \n\n2. Similarly, the NFs results on Fig 1 in the simple 2D synthetic setup of eight gaussians do not really demonstrate excellent precision despite choosing the parameter lambda supposedly largely favoring the quality of generated samples.  Compare this with e.g. Wasserstein-GP GAN from 2017 paper (https://github.com/caogang/wgan-gp).\n\n3. The code for the reproducibility check is not provided, despite the mentioning of an anonymized repository on page 8, the link is absent. This is especially disappointing as some evaluation numbers looks somewhat strange, like the P-value at $\\lambda=20$ in Table 2 compared with FID, but it is not possible to try to reproduce this number or to see the details of its calculation. \n\n4. Despite the fact that there are numerous variants in the literature of precision-recall scores, only one variant, based on k=3-NN only, from [27] is provided for evaluation of the models, which is known to not always work properly especially in regions of low density. \n\n5. The related works description has some lacunas like \"Reliable Fidelity and Diversity Metrics for Generative Models\" (ICML'2020) from which another variant of PR scores can be used for evaluation, or \"A Domain Agnostic Measure for Monitoring and\nEvaluating GANs\" (NeurIPS'2019). \n\n6. Out of big diversity of GAN models, only the training of BigGan is tested with the proposed approximation objective. \n\n7. In several places throughout the paper, the likelihood ratio $p(x)/\\hat{p}(x)$ is used, e.g. in Theorem 5.2. However it is known that,  especially in high-dimensional spaces, the support of distributions is rather small and there can be large regions with  $\\hat{p}(x)=0$ on which the methods involving the ratios as above give infinite or not well-defined answers. How to deal with this in each of the  encounters of this quantity ?\n\n\n8. The clarity of the presentation can be improved, in particular the relation with other, more intuitive definitions of Precision and Recall scores from e.g. [27] and the \"Reliable Fidelity and Diversity Metrics for Generative Models\" (ICML'2020) paper,  is not explained. \n\n9. The complexity of the training is not described.  \n\n10. The error bars are absent in the main experiment reported in Table 2.\n\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and feedback on our paper. We appreciate the time and effort you've dedicated to understanding our work. We would like to address the concerns you raised:\n\n**1. Improving Recall:**\nWhile methods like rejection sampling, instance selection, or truncation primarily focus on enhancing precision, often at the expense of recall, our method aims to improve recall without significantly deteriorating precision. For the sake of completeness, we have added results for BigGAN using truncation for comparison to further illustrate this point.\n\n**2. Experimental Setup and Model Choice:**\nThe primary objective of our experimental setup is to demonstrate the efficacy of our method in tuning any given model. Our theoretical framework establishes that our approach can be applied universally across GAN or NF architectures to balance precision and recall. We believe it's unnecessary to test our method on every existing architecture, especially considering the environmental impact of training large models like StyleGanXL. Moreover, in our 2D experiments, we intentionally used a model with limited expressivity to clearly illustrate how our method operates. Using a more complex model would have perfectly matched the distribution, distinguishing from real-world dataset and thus, making our method redundant.\n\n**3. Code Availability and Additional Results:**\nWe apologize for the oversight regarding the code. We have provided the code to the area chair. Additionally, we've included extra results in the general rebuttal, showcasing the Precision Recall (Kynkäänniemi et al.) for k=5 and the Density Coverage (Naeem et al.) for k=5. These new metrics further validate our findings.\n\nAddressing the Questions:\n\n**Training Complexity:** As discussed in our paper, our training algorithm closely mirrors the original GAN training procedure. Consequently, both share similar algorithmic complexities, contingent on the neural network architecture used. We will make this clear in the paper, and add precise training times of our architectures.\n\n**Estimating PR Divergence:** We estimate the PR divergence using the primal form of the f-divergence. This primal form is estimated by sampling generated data points from $\\widehat{ P}$:\n\n$$\n\\mathcal{D}_{\\lambda}(P \\Vert\\widehat{P}) = {\\mathrm{E}} \\_{\\widehat{P}}\n\\left[f \\_{\\lambda}\\left(\\frac{p(x)}{\\widehat{p}(x)}\\right)\\right].\n$$\n\nPoints where $ \\widehat p(x) = 0$ will *not* be sampled for estimating the primal form. Thus, the value of the density ratio estimator on those points has no effect on the estimation of the PR divergence. We will make this clearer in the paper.\n\n**Metrics Evaluation:** Due to computational considerations, the metrics in Table 2 are evaluated on only one instance of the model. We observed similar numbers on other runs as well, and are open to adding error bars for the revised manuscript. \n\n**Related Works:**  The two papers you mention do make a good addition to the related works section - we thank you for pointing them out. However, we emphacize that our training method is focused on improving the precision and recall as defined in the works of [40, 43], and thereafter used in numerous follow-up works.\n\n**Complexity of training:** The complexity of our training method (both computational and memory) is comparable or no greater than that of existing approaches like f-GAN. We will make this clear in the paper. \n\nWe hope that our clarifications address your concerns, and we are committed to refining our manuscript based on your valuable feedback.\n"}, {"review_id": "CcLYk7XOaA", "reviewer": "Reviewer_VDUJ", "summary": "The paper proposes a way to train generative models such that they obtain a user defined tradeoff between sample fidelity and variety. Main findings are that the definition of precision and recall from previous works (by Simon et al.) can be written as a $f$-divergence (named PR-divergence) and that other $f$-divergences, such as KL or reverse-KL, can be formulated in terms of PR-divergence.", "questions": "1. To further demonstrate the usefulness of explicitly controlling PR-tradeoff, I would like to see FID and PR curves from a following comparison: truncating BigGAN vs. applying classifier-free guidance to ADM models vs. training a separate model using different $\\lambda$ to control the PR-tradeoff (on CIFAR-10, ImageNet 128x128 datasets).\n\n2. Why do the FID results using BigGAN differ from the official results, reported by Brock et al. [1], on CIFAR-10 (13.37 vs. 14.73) and ImageNet 128x128 (9.83 vs. 8.7)? Two things that come to mind could be that you either use a validation set to compute FID or a different instance of Inception-V3 network (see App. A from [2] for other possible explanations). Also a minor note, on ImageNet 128x128 the baseline actually has the best FID instead of the model with $\\lambda=0.2$.\n\nMinor notes from checking the proofs:\n\n3. There are different versions of defining $\\lambda \\in [0, \\inf]$ in Definition 4.1 and Theorem 4.3. What is the difference?\n4. Proof of Theorem in App. Eq. (6): Is the last term missing $\\hat{p}(x)$, but it is correctly there in Eq. (7)?\n5. App. Eq. (8): the first integral term has extra $($ and is missing $\\textrm{d}\\boldsymbol{x}$?\n6. App. B.5: Title should be Proof of Theorem 4.4.\n7. App. Eq. (30): Third term is missing $u_{\\textrm{min}}$ from the denominator but it reappears correctly in Eq.(31)? Why does $c(1/u_{\\textrm{max}})$ change suddenly to $c(0)$ (it cancels out but I’m still curious)?\n8. Why in App. Eq (32) $\\textrm{lim}$-term appears when differentiating w.r.t $u$?\n9. Is there a typo in Theorem 5.2? Should it be $r(x) = \\nabla g^*(T(x))$ instead of $f^*$?\n\n[1]: Brock et al., Large Scale GAN Training for High Fidelity Natural Image Synthesis\n\n[2]: Kynkäänniemi et al., The Role of ImageNet Classes in Fréchet Inception Distance\n", "limitations": "The authors adequately addressed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strengths of the paper are thorough analysis of the $f$-divergences and solid theoretic foundation of PR-divergence. The paper provides interesting insights into what PR-tradeoff often used divergences, such as the KL, optimize for. Additionally, it proposes an algorithm to train generative models using an auxiliary divergence, since training models with $f$-GAN is difficult in practice.", "weaknesses": "The main weakness of the paper are empirical results. Although the method works as expected from the theory and it can be used to explicitly control the tradeoff between precision and recall it is far behind the current state-of-the-art. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your comprehensive review and insightful comments are greatly valued, and we thank you for them. We would like to address the concerns and questions you raised:\n\n**1. Empirical Results and PR-tradeoff:**  \nWe acknowledge the importance of empirical results. We've added results for truncation on the Baseline BigGAN in the General Rebuttal. As observed, truncation primarily enhances the baseline precision, but it does not significantly improve recall.\n\n**2. FID Differences:**  \nThe discrepancy in FID for CIFAR-10 arises from our use of PyTorch, for which pretrained weights for BigGAN are unavailable. Consequently, we utilized a version of BigGAN that we trained ourselves. Additionally, as you rightly pointed out, we employed the PyTorch version of the Inception model, leading to FID differences from the official TensorFlow metrics. To address this, we recalculated FIDs for every different model using a consistent method.\n\n**3.  Addressing the Main Weakness:**  \nThe primary objective of our paper is to introduce a loss function that can effectively trade off between precision and recall. This loss can be applied to a wide range of generative models, including StyleGANXL and diffusion models. We firmly believe that our experimental setup validates the efficacy of our method. It's essential to note that our focus isn't on achieving only state-of-the-art results. Instead, we aim to showcase the versatility of our loss function. The notion of \"state-of-the-art\" is contingent on the specific model used and the user's preference for either recall or precision.\n\n**Typos and Clarifications:**\n\n- **Lambda Domain:** The domain of $ \\lambda $ only differs in notation. In both cases, it encompasses all positive values, including 0 and $ +\\infty $. We will ensure consistent notation in the final manuscript.\n\n- **Equation Typos:** We appreciate your keen observation of the typos in Eq (6), Eq (8), and the title for App B.5. These will be corrected.\n\n- **Bounds on Lambda:** We transitioned from 0 and $+\\infty $ bounds on $ \\lambda $ to $ 1/u_{max} $ and $ 1/u_{min} $ in theorem 4.4. The discrepancies in the proofs that you highlighted will be addressed. Thank you for pointing them out.\n\n- **Theorem 5.2 Notation:** The notation in theorem 5.2 is accurate. However, we acknowledge the reversed $f $ and $ g $ notations in the appendix. This will be rectified in the final manuscript.\n"}]

Author rebuttal:
Dear Reviewers, 

In response to the feedback from the reviewers, we have added some additional results for a more comprehensive comparison. Two reviewers inquired about a comparison with other methods such as truncation, and one recommended adding Density and Coverage metrics from Naeem et al. Moreover, Precision and Recall are computed for $k=5$. Below are the results for four different datasets:



### CIFAR-10

| Model | FID | Precision | Recall | Density | Coverage |
|---|---|---|---|---|---|
| Baseline BigGAN $\psi=1.0$ | 13.38 | 86.54 | 65.63 | 0.76 | 0.81 |
| Baseline BigGAN $\psi=0.7$ | 22.25 | 90.81 | 48.01 | 0.90 | 0.67 |
| Baseline BigGAN $\psi=0.5$ | 36.10 | 92.34 | 22.11 | 1.00 | 0.48 |
|---|---|---|---|---|---|
| $\lambda=0.05$ | 13.88 | 85.29 | 68.40 | 0.72 | 0.83 |
| $\lambda=0.10$ | 11.62 | 81.78 | 74.58 | 0.66 | 0.83 |
| $\lambda=0.20$ | 13.36 | 84.85 | 65.13 | 0.74 | 0.82 |
| $\lambda=0.30$ | 14.41 | 84.24 | 69.42 | 0.71 | 0.82 |
| $\lambda=0.50$ | 14.50 | 83.27 | 68.23 | 0.70 | 0.81 |
| $\lambda=0.67$ | 15.15 | 82.57 | 68.34 | 0.69 | 0.81 |
| $\lambda=1.00$ | 15.26 | 81.96 | 72.51 | 0.65 | 0.80 |
| $\lambda=1.50$ | 16.68 | 84.64 | 63.92 | 0.73 | 0.79 |
| $\lambda=2.00$ | 18.25 | 79.53 | 72.90 | 0.59 | 0.78 |
| $\lambda=3.00$ | 26.66 | 85.28 | 55.49 | 0.76 | 0.74 |
| $\lambda=5.00$ | 32.54 | 83.39 | 56.94 | 0.68 | 0.73 |
| $\lambda=10.00$ | 39.69 | 84.11 | 39.29 | 0.75 | 0.67 |
| $\lambda=20.00$ | 67.03 | 89.64 | 20.52 | 0.97 | 0.56 |

 
### CelebA 64

| Model | FID | Precision | Recall | Density | Coverage |
|---|---|---|---|---|---|
| Baseline BigGAN $\psi=1.0$ | 9.17 | 78.48 | 51.36 | 0.89 | 0.49 |
| Baseline BigGAN $\psi=0.7$ | 23.72 | 87.82 | 31.11 | 1.29 | 0.49 |
| Baseline BigGAN $\psi=0.5$ | 43.64 | 91.01 | 11.54 | 1.53 | 0.39 |
|---|---|---|---|---|---|
| $\lambda=0.2$ | 8.79 | 83.37 | 44.07 | 1.09 | 0.54 |
| $\lambda=0.5$ | 6.03 | 77.60 | 55.98 | 0.88 | 0.50 |
| $\lambda=0.7$ | 9.24 | 81.08 | 46.71 | 1.03 | 0.51 |
| $\lambda=1.0$ | 13.07 | 81.70 | 36.85 | 1.00 | 0.47 |
| $\lambda=1.5$ | 13.21 | 83.56 | 38.89 | 1.09 | 0.51 |
| $\lambda=2.0$ | 14.23 | 82.98 | 32.87 | 1.16 | 0.49 |
| $\lambda=5.0$ | 22.44 | 84.04 | 25.67 | 1.21 | 0.43 |

### ImageNet 128


| Model | FID | Precision | Recall | Density | Coverage |
|---|---|---|---|---|---|
| Baseline BigGAN $\psi=1.0$ | 9.84 | 27.97 | 40.92 | 0.14 | 0.17 |
| Baseline BigGAN $\psi=0.7$ | 11.39 | 23.12 | 31.77 | 0.11 | 0.15 |
| Baseline BigGAN $\psi=0.5$ | 15.49 | 20.25 | 20.08 | 0.10 | 0.14 |
|---|---|---|---|---|---|
| $\lambda=0.2$ | 9.92 | 26.69 | 42.04 | 0.13 | 0.17 |
| $\lambda=0.5$ | 10.82 | 26.83 | 42.38 | 0.13 | 0.16 |
| $\lambda=1.0$ | 20.42 | 29.72 | 28.21 | 0.15 | 0.15 |
| $\lambda=2.0$ | 20.21 | 30.27 | 30.49 | 0.14 | 0.14 |
| $\lambda=5.0$ | 20.76 | 30.87 | 28.38 | 0.15 | 0.15 |

### FFHQ 256

| Model | FID | Precision | Recall | Density | Coverage |
|---|---|---|---|---|---|
| Baseline BigGAN $\psi=1.0$ | 41.42 | 65.54 | 10.02 | 0.52 | 0.47 |
| Baseline BigGAN $\psi=0.7$ | 56.44 | 76.60 | 4.83 | 0.70 | 0.41 |
| Baseline BigGAN $\psi=0.5$ | 82.04 | 84.51 | 1.50 | 0.89 | 0.32 |
|---|---|---|---|---|---|
| $\lambda=0.2$ | 35.66 | 78.70 | 9.45 | 0.88 | 0.60 |
| $\lambda=0.5$ | 35.24 | 78.41 | 9.66 | 0.89 | 0.60 |
| $\lambda=1.0$ | 35.91 | 78.95 | 8.32 | 0.90 | 0.57 |
| $\lambda=2.0$ | 36.33 | 81.10 | 8.69 | 1.05 | 0.64 |
| $\lambda=5.0$ | 38.16 | 84.31 | 8.52 | 1.15 | 0.63 |

**Observations on the Tables:**

1. **Cifar and CelebA:** 
   - Truncation primarily enhances Precision at the expense of Recall, yielding results comparable to our method. 
   - However, when it comes to improving Recall, our method stands out. By training models with $\lambda < 1$, we've been able to boost the baseline Recall, a feat truncation fails to achieve.

2. **Imagenet:** 
   - Truncation appears to be counterproductive, diminishing both Precision and Recall. This suggests that the mean of the Gaussian might be mapped outside the support of the target distribution.

3. **FFHQ:** 
   - Our method demonstrates a superior trade-off compared to truncation. For instance, at a Precision of 84%, our method achieves a Recall of 8.52% versus truncation's 1.5%.

---


Thank you for the constructive feedback. We've made efforts to address the concerns raised and hope that the additional results and explanations provided here shed more light on our approach. 


Meta-review: This paper proposes a family of divergences for training generative models (e.g., GANs) over a user-defined trade-off between precision and recall. They is some good theoretical analysis, e.g., w.r.t. f-divergences and empirical results support the claim that this approach can be used to improve precision / recall. The work is interesting, novel, and important / well motivated and justified, it appears that the presentation and while there were concerns about soundness of some of the theory, sufficient clarifications were made. It would be nice to see this method applied to more generative models (e.g, styleGAN, etc), but I believe BigGAN was sufficient for demonstrating the core ideas work. I therefore recommend this paper to be accepted as a poster.
Decision: Accept (poster)