Title: Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generative Models

Precision

Recall

Optimization

f-Divergeces
Abstract: Achieving a balance between image quality (precision) and diversity (recall) is a significant challenge in the domain of generative models. Current state-of-the-art models primarily rely on optimizing heuristics, such as the Fr\'echet Inception Distance. While recent developments have introduced principled methods for evaluating precision and recall, they have yet to be successfully integrated into the training of generative models. Our main contribution is a novel training method for generative models, such as Generative Adversarial Networks and Normalizing Flows, which explicitly optimizes a user-defined trade-off between precision and recall.  More precisely, we show that achieving a specified precision-recall trade-off corresponds to minimizing a unique $f$-divergence from a family we call the \mbox{\em PR-divergences}. Conversely, any $f$-divergence can be written as a linear combination of PR-divergences and  corresponds to a weighted precision-recall trade-off. Through comprehensive evaluations, we show that our approach improves the performance of existing state-of-the-art models like BigGAN in terms of either precision or recall when tested on datasets such as ImageNet.

Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows

Abstract

Achieving a balance between image quality (precision) and diversity (recall) is a significant challenge in the domain of generative models. Current state-of-theart models primarily rely on optimizing heuristics, such as the Fréchet Inception Distance. While recent developments have introduced principled methods for evaluating precision and recall, they have yet to be successfully integrated into the training of generative models. Our main contribution is a novel training method for generative models, such as Generative Adversarial Networks and Normalizing Flows, which explicitly optimizes a user-defined trade-off between precision and recall. More precisely, we show that achieving a specified precisionrecall trade-off corresponds to minimizing a unique f -divergence from a family we call the PR-divergences . Conversely, any f -divergence can be written as a linear combination of PR-divergences and corresponds to a weighted precisionrecall trade-off. Through comprehensive evaluations, we show that our approach improves the performance of existing state-of-the-art models like BigGAN in terms of either precision or recall when tested on datasets such as ImageNet.

1 Introduction

Evaluation of generative models has always been a challenging task. The metric used must reflect both the quality of the sample generated (precision) and how much the sample covers the targeted probability distribution (recall). Inception Score (IS) or Fréchet Inception Distance (FID) have been introduced and are now widely used by the community to select the best models. Typically, these metrics are favored because they "correlate well with the visual fidelity of the samples" and are "sensitive to both the addition of spurious modes as well as mode dropping" [44]. However as pointed by Kynkäänniemi et al. [28], FID and IS group these two aspects into a single value without a clear trade-off. Depending on the use-case, generative models might require a good precision (high-resolution image and video generation, artistic synthesis, 3D model design) or a good recall (data augmentation, drug discovery, anomaly detection). For that reason, a number of more principled methods [9, 15, 28, 44, 47] have emerged to assess precision and recall (hereafter abbreviated by

P&R) independently, however these methods cannot be optimized during training, because they are not differentiable or because they are too computationally demanding.

To enhance either the precision or the recall of a particular model, an array of strategies and techniques have been employed. Usually these techniques involve altering the latent distribution a posteriori (e.g. truncation and temperature post-training). Other methods exists such as through rejection sampling, boosting, or instance selection [3, 12, 13, 19, 23, 33, 49]. Although these approaches may utilize proxies of P&R, they are not theoretically grounded in the principled methods of evaluating these metrics, leaving their alignment with the foundational concepts of P&R unverified. This leads to the following question.

Question 1: Is it possible to train a generative model that achieves a specified trade-off between precision and recall?

In addition to this question, another of our goals is to understand existing generative models in terms of P&R. Modern generative models, such as Generative Adversarial Networks (GANs) or Normalizing Flows (NFs), are typically designed to minimize specific divergence measures. Given a target distribution P and a set of parameterized distributions { P θ | θ ∈ Θ } , a generative model aims to find the best fit ̂ P = P θ ∗ that minimizes a divergence between P and ̂ P . These divergences induce different behaviors at convergence. For instance, optimizing the Kullback-Leibler (KL) divergence tends to favor mass-covering models [35], contrasting with the mode-seeking behavior observed with other generative models, leading to the infamous problem of mode collapse. Transitioning from one divergence to another does indeed alter the model's behavior in terms of precision and recall, however, the implicit trade-offs made during the optimization of a general divergence remain somewhat ambiguous. This motivates the following question.

Question 2: What precision-recall trade-off does an arbitrary f -divergence minimize?

In this paper, we bridge the gap between principled methods of P&R evaluation and controlling their trade-off. In doing so, we address Questions 1 and 2 by making the following contributions:

We show that achieving a specified precision-recall trade-off corresponds to minimizing a particular f -divergence between P and ̂ P . Specifically, in Theorem 4.3 we give a family of f -divergences (denoted by D λ -PR , λ ∈ [0 , ∞ ] ) that are associated with various points along the precision-recall curve of the generative model.

We show that any arbitrary f -divergence can be written as a linear combination of f -divergences from the D λ -PR family. This result makes explicit, the implicit precision-recall trade-offs made by generative models that minimize an arbitrary f -divergence.

We propose a novel approach to train or fine-tune a generative model on notoriously hard to train f -divergences, with guarantees on divergences defined by the lipschitz constant of f .

We use this approach to train models on a user specified trade-off between P&R by minimizing D λ -PR for any given λ . We specifically focus on GANs and NFs. For instance, Figure 1 shows how our model performs under various settings of λ . With a high λ , we can train the model to favor precision over recall and vice-versa.

Through extensive experiments, we show that our approach enables effective model training to minimize the PR-Divergence, particularly for fine-tuning pre-trained models, with a notable impact from the choice of trade-off parameters λ , while also demonstrating scalability with larger dimensions and datasets.

2 Related works

Generative model evaluation metrics: IS or FID are widely adopted due to their ability to assess visual fidelity and sensitivity to mode variations. However, these metrics fall short in providing a trade-off between P&R. Principled approaches were introduced by Djolonga et al. [15] and Sajjadi et al. [44], and later extended by Simon et al. [47], providing a definition P&R independently using PR-Curves detailed in Section 3.3. We adopt this extended approach in our work. In image generative modeling, the current consensus for P&R evaluation are two methods. First, the method presented by Kynkäänniemi et al. [28], which provides a simpler evaluation of P&R based on the estimation

(a)

λ

= 0

.

1

(b)

λ

= 1

(c)

λ

= 10

(d)

D

KL

(e)

D

rKL

of the support of the distribution using a k-NN algorithm, akin to the most recent work by Cheema and Urner [9]. Second, a method introduced by [36], computing the Density (D) and the Coverage (C) which account for local density and are robust to outliers in contrast to [28]. In natural language processing, one popular method, MAUVE [41], consists of the area under PR-Curves defined by [15]. However, because of their mathematical foundation, these methods remain unsuitable for training due to non-differentiability and high computational requirements.

Trade-offs and techniques in generative model: The challenge of balancing P&R in generative model training has led to a variety of techniques. Methods for enhancing precision often involve manipulating the latent distribution post-training, such as hard truncation [7, 24, 25, 46] or standard deviation adjustment, also called soft truncation [27]. One can also use rejection sampling in the image space [4, 50], in the latent space [23, 49], or in the dataset [13]. There are also numerous strategies to prevent mode collapse and boost recall [8, 31, 32], gradient-boosting methods [12, 16, 19] and mixture based latent models [6, 40]. However, it should be noted that these approaches lack a foundation in the principled methods of P&R evaluation. This underscores the need for a theoretically grounded approach to balance these crucial aspects in generative models, a gap that our current work aims to address. Note also that all these methods can be applied to any model, and in particular to the one trained using our proposed method.

Divergence training in generative models: Modern generative models, such as GANs and NFs, often employ specific divergence measures for model optimization. With works such as Grover et al. [20], Nowozin et al. [39], models can be trained with a variety of divergences such as f -divergence, thus observing different results of P&R. Another notable work by Midgley et al. [34] proposed training with α -Divergence, under the assumption of access to data density. The work of [29], closely aligned with our approach, defines different orders of mode seeking and evaluates the corresponding f -divergences at these levels. Although its methodology employs the training of f -GAN models on simple datasets to illustrate the mode-seeking property, it does not offer the flexibility to establish a user-defined trade-off. The implicit trade-offs made by these divergence measures is still a challenge: the recent work of [48] shows the links between P&R and any DeGroot's divergences.

3 Background

Notation: Throughout the paper, we use X ⊂ R d to refer to the input space and Z ⊂ R m to the latent space of the model we consider. We also denote P ( X ) and P ( Z ) the set of all probability measures over measurable subsets of X and Z respectively. P and ̂ P are consistently used to denote the target and estimated distributions (both members of P ( X ) ). We also assume that P and ̂ P share the same support in X , and that they admit densities denoted by the corresponding lower case letters p and ̂ p , respectively. Finally, for any function f ∈ R → R , we define the convex conjugate (or Fenchel transform) of f given by f ∗ ( t ) = sup u ∈ R { tu -f ( u ) } .

3.1 f -divergences

Given a convex lower semi-continuous (l.s.c) function f : R + → R satisfying f (1) = 0 , the f -divergence between two probability distributions P and ̂ P is defined as follows.

D f is invariant to an affine transformation in f i.e., D f ( P ∥ ̂ P ) = D f † ( P ∥ ̂ P ) for f † ( u ) = f ( u ) + c ( u -1) for any constant c ∈ R . Many well-known statistical divergences such as the KL divergence ( D KL ), the reverse KL ( D rKL ) or the Total Variation ( D TV ) are f -divergences (see Table 1). Importantly, any D f admits a dual variational form [37], with T denoting the set of all measurable functions X → R :

We use T opt ∈ T to denote the function that achieves the supremum in (15).

3.2 Generative models

Generative Adversarial Networks (GANs): A GAN consists of two functions, generator G : Z → X , discriminator T : X → R , as well as a prior distribution Q ∈ P ( Z ) which is usually the standard normal N (0 , I m ) . The estimated data distribution ̂ P G is the push-forward distribution of Q by G . In the original work of Goodfellow et al. [17], a specific D f is used to optimize the divergence between P and ̂ P G . In this paper, we consider the more general framework of Nowozin et al. [39] that can be used to train a GAN with any D f by solving the following min-max optimization problem:

Normalizing Flows (NFs): An NF consists of an invertible function G : Z → X and a prior distribution Q ∈ P ( Z ) . As for GANs, the estimated distribution ̂ P G is the push-forward of Q by G . However, because G is invertible, it is possible to compute the density ˆ p ( · ) using a simple change of variable formula, ˆ p ( x ) = q ( G -1 ( x )) | det Jac G -1 ( x ) | , where det Jac G -1 ( x ) is the determinant of the Jacobian matrix of G -1 at x , and train the generator using the D KL between P and ̂ P F , or equivalently, by maximizing the log-likelihood:

where H ( P ) is the continuous entropy of P . In practice, the generator function G is typically represented by neural networks such as GLOW [27], RealNVP [42], or ResFlow [5, 10] for which det Jac G -1 ( x ) is easy to compute. Grover et al. [20] showed that it is possible to train an NF using most f -divergences. In practice, the log-likelihood is added to the min-max objective to stabilize learning. This gives us the following optimization problem:

α

p

(

x

)

x

Target

P

Model

̂

P

1

Model

̂

P

2

(a) Distributions

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

β

α

Model

̂

P

1

Model

̂

P

2

(b)

∂

PRD(

P,

̂

P

)

curves

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

β

λ

= 0

.

1

λ

= 1

λ

= 10

λ

= 0

.

1

λ

= 1

λ

= 10

D

KL

D

rKL

(c)

∂

PRD(

P,

̂

P

)

for Fig. 1

3.3 Precision-Recall curve for generative models

Generative models are usually evaluated using a single criterion such as the IS [45] or the FID [21]. However, these criteria are unable to distinguish between the distinct failure modes of low precision (i.e., failure to produce quality samples) and low recall (i.e., failure to cover all modes of P ). The following definition was introduced by Sajjadi et al. [44] and later extended by Simon et al. [47].

Definition 3.1 (PRD set, adapted from Simon et al. [47]) . For P, ̂ P ∈ P ( X ) , the P&R set PRD( P, ̂ P ) is defined as the set of pairs of precision α and recall β in R + × R + such that there exist µ ∈ P ( X ) for which P ≥ βµ and ̂ P ≥ αµ . The precision-recall curve (or PR curve) is defined as ∂ PRD( P, ̂ P ) = { ( α, β ) ∈ PRD( P, ̂ P ) | ∄ ( α ′ , β ′ ) with α ′ ≥ α and β ′ ≥ β } .

An equivalent definition of PRD( P, ̂ P ) is found in Sajjadi et al. [44], where ( α, β ) ∈ PRD( P, ̂ P ) if P and ̂ P can be decomposed as in (6) for some common component µ ∈ P ( X ) and complementary components ν P , ν ̂ P ∈ P ( X ) .

Simon et al. [47] show that the PR curve is parameterized by λ ∈ [0 , + ∞ ] as ∂ PRD( P, ̂ P ) = { α λ ( P ∥ ̂ P ) , β λ ( P ∥ ̂ P ) | λ ∈ [0 , + ∞ ] } , with α λ ( P ∥ ̂ P ) = ∫ X min( λp ( x ) , ̂ p ( x )) d x and β λ ( P ∥ ̂ P ) = α λ ( P ∥ ̂ P ) /λ . Here, λ is called the trade-off parameter and can be used to adjust the sensitivity to precision or recall.

An illustration of the PR curve is given in Figure 2 for a target distribution P that is a mixture of two Gaussians and two candidate models ̂ P 1 and ̂ P 2 . We can see on Figure 2 that ̂ P 1 offers better results for large values of λ (with high sensitivity to precision) whereas ̂ P 2 offers better results for low values of λ (with high sensitivity to recall).

4 Precision and Recall trade-off as an f -divergence

In this section, we formalize the link between P&R trade-off and f -divergences, and address Question 2. We will exploit this link in Section 5 to train models that optimize a particular P&R trade-off.

4.1 Precision-Recall as an f -divergence

We start by introducing the PR-Divergence as follows.

Definition 4.1 (PR-divergence) . Given a trade-off parameter λ ∈ [0 , + ∞ ] , the PR-divergence (denoted by D λ -PR ) is defined as D f λ for f λ : R + → R given by f λ ( u ) = max( λu, 1) -max( λ, 1) for λ ∈ R + and f λ ( u ) = 0 for λ = + ∞ .

Note that f λ is continuous, convex, and satisfies f λ (1) = 0 for all λ . A graphical representation of f λ can be found in Appendix B.2. The following proposition gives some properties of D λ -PR . Proposition 4.2 (Properties of the PR-Divergence) .

The Fenchel conjugate f ∗ λ of f λ is defined on dom( f ∗ λ ) = [0 , λ ] and given by, f ∗ λ ( t ) = t/λ for λ ≤ 1 and f ∗ λ ( t ) = t/λ + λ -1 otherwise.

The optimal discriminator for the dual form is T opt ( x ) = λ sign { p ( x ) / ̂ p ( x ) -1 } .

D λ -PR ( ̂ P ∥ P ) = λ D 1 λ -PR ( P ∥ ̂ P ) .

̸

Observe that D λ -PR ( ̂ P ∥ P ) = D λ -PR ( P ∥ ̂ P ) in general, but for λ = 1 , D 1 -PR ( P ∥ ̂ P ) = D 1 -PR ( ̂ P ∥ P ) = D TV ( P ∥ ̂ P ) / 2 . Having defined the PR-divergence, we can now show that P&R w.r.t λ can be expressed as a function of the divergence between P and ̂ P .

Theorem 4.3 (P&R as a function of D λ -PR ) . Given P, ̂ P ∈ P ( X ) and λ ∈ [0 , + ∞ ] , the PR curve ∂ PRD( P, ̂ P ) is related to the PR-divergence D λ -PR ( P ∥ ̂ P ) as follows.

Conversely, suppose that there exists a strictly decreasing linear function h : [0 , 1] → R + and an f -divergence D f such that h ( α λ ( P ∥ ̂ P )) = D f ( P ∥ ̂ P ) for all P, ̂ P ∈ P ( X ) , then f ( u ) = c 1 f λ ( u ) + c 2 ( u -1) .

A direct consequence of Theorem 4.3 is that minimizing D λ -PR is equivalent to maximizing α λ . In a more explicit way, Theorem 4.3 suggests that D λ -PR is the only f -divergence (up to an affine transformation) for which this property holds. This makes D λ -PR a uniquely suitable candidate for training a generative model with a specific P&R trade-off. The proof of Theorem 4.3 is in Appendix B.1

4.2 Relation between PR-divergences and other f -divergences

In the previous subsection, we showed that for each trade-off parameter λ , there exists a D λ -PR that corresponds to optimizing for it. This raises the converse question of what trade-off is achieved by optimizing for an arbitrary f -divergence. We answer this by showing in the following theorem that any f -divergence can be expressed as a weighted sum of PR-divergences.

Theorem 4.4 ( f -divergence as weighted sums of PR-divergences) . 1 For any P, ̂ P ∈ P ( X ) supported on all of X and any λ ∈ R + , with m = min X ( ̂ p ( x ) /p ( x )) and M = max X ( ̂ p ( x ) /p ( x )) :

As a sanity check, observe that the weights f ′′ (1 /λ ) /λ 3 remain invariant under an affine transformation in f much like D f .

Corollary 4.5 ( D KL and D rKL as an average of D λ -PR ) . The D KL Divergence and the D rKL can be written as a weighted average of PR-Divergence D λ -PR :

As we can see in this Corollary, both D KL and D rKL can be decomposed into a sum of PR-divergences terms D λ -PR , each weighted with 1 /λ 2 and 1 /λ respectively. Note that if λ < 1 then 1 /λ > 1 /λ 2 , and conversely, if λ > 1 . D KL is thus associating more weights on recall than D rKL . This explains the mass covering behavior observed in NFs trained with D KL . Comparatively, the D rKL assigns more weight to terms with a large lambda, leading to the mode seeking behavior empirically observed with flows trained with the D rKL [33]. However, as it will be observed in the Section 6, D rKL is still favoring low values of λ . Other weights for other f -divergences are in Appendix A.1.

5 Minimizing the Precision-Recall divergence

In this section, we address Question 1 by introducing a new method that can be used to optimize a model with a specific precision-recall trade-off λ . A first naive strategy to achieve this is to use the f -GAN framework introduced by Nowozin et al. [39], and minimize the dual variational form of D λ -PR presented in Theorem 4.3. Together, this results in solving the min-max problem:

where G and T are both represented using neural networks. In practice, this strategy would fail because training a neural network with loss functions such as f ∗ TV (or in this case f ∗ λ ) is notoriously difficult due to vanishing gradients 2 that lead to poor training performance [20, 39]. Instead, training neural networks on functions f ∗ such as f ∗ KL or f ∗ χ 2 , results in much better empirical performance [29, 39, 51].

̸

To avoid these issues, we show how to train the discriminator using an auxiliary divergence (based on a function g = f ) to better estimate the target f -divergence. The main idea is to choose an auxiliary g -divergence that is adequate for training T (e.g. D KL or D χ 2 ) and use it to compute the likelihood ratio p ( x ) / ̂ p ( x ) .

Because, at optimality, we have ∇ g ∗ ( T opt ( x )) = p ( x ) / ̂ p ( x ) , we can then compute the f -divergence as follows:

In practice, we do not have access to T opt . Instead, we have a discriminator T trained to maximize D dual g,T . At any time during training, we can estimate the f -divergence based on T using the primal estimate , which we define as follows.

Definition 5.1 (Primal estimate D primal f,T ) . Let P, ̂ P ∈ P ( X ) . For any function T : X → R , f : R + → R , and g : R + → R , we define the primal estimate D primal f,T as follows.

where r : X → R + is given by, r ( x ) = ∇ g ∗ ( T ( x )) .

The success of this approach depends on how well r approximates p ( x ) / ̂ p ( x ) , which depends on T . We first show that the approximation error of r measured in terms of the Bregman divergence [2] (also defined in Appendix A.2). It corresponds exactly to the approximation error of D dual g,T , so minimizing the latter will also minimize the former.

Theorem 5.2 (Error of the estimation of an f -divergence under the dual form.) . For any discriminator T : X → R and r ( x ) = ∇ f ∗ ( T ( x )) ,

On the basis of this result, the quality of the approximation will crucially depend on ∇ g . We can show that if the auxiliary g is strongly convex, the error in the estimation of D primal f,T is bounded:

Theorem 5.3 (Bound on the estimation of an f -divergence using an auxiliary g -divergence) . Let f, g : R + → R be such that g is µ -strongly convex and f is σ -Lipschitz. For the discriminator T : X → R , let r ( x ) = ∇ g ∗ ( T ( x )) . Then

If T successfully maximizes D dual g,T , then the primal estimation converges to D f . To implement this approach, we propose the following simplified version of the algorithm: Repeat until convergence these 3 steps:

Let x real 1 . . . x real N ∼ P and x fake 1 . . . x fake N ∼ ̂ P G .

Update the parameters of T by ascending the gradient

Update the parameters of G by descending the gradient

This method closely parallels the GAN training procedure, with the key distinction that T and G optimize objectives based on different f . In practice, our implementation uses a stochastic gradient descent, fully detailed in Algorithm 1 in Appendix C.

6 Experiments

In this section, we employ the auxiliary loss approach outlined in Section 5 to train various models.

Specifically, we train NFs on 2D synthetic data, MNIST and FashionMNIST, while we train BigGAN on CIFAR-10, CelebA64, ImageNet128, and FFHQ256. All models are tested in terms of IS and FID with 50k samples, and on P&R with 10k samples using the method of Kynkäänniemi et al. [28] with k = 3 for MNIST and FashionMNIST and with k = 5 for CelebA64, ImageNet128 and FFHQ256. Also, we test every model in terms of Density and Coverage [36] on 10k samples with k = 5 . In this paper, we present a selection of experimental results. For a comprehensive set of results, including model parameters, optimizers, learning rates, and samples, please refer to Appendix D. We also included in this Appendix a set of experiments run using the naive approach based on Equation 10, thus showing that the discriminator fails to train as explained in Section 5. To ensure reproducibility, our models and code are available on the GitHub repository of the project 3 .

We show that: 1) the auxiliary loss approach effectively enables the training of a model to minimize the PR-Divergence, 2) this method is suitable for fine-tuning pre-trained models, 3) the choice of trade-off parameter λ significantly influences the results on P&R, and finally, 4) our method scales well with larger dimensions and datasets.

Normalizing Flows on synthetic data: NFs are typically trained to minimize D KL , in addition to their structural limitation [11, 52], resulting in good recall but poor precision. Prior work has employed various techniques [27, 49] to improve model precision post-training, we use our method to directly train the model on a given trade-off. We demonstrate our approach by training RealNVP models on a 2D synthetic dataset using D λ -PR with various λ values and using D KL and D rKL for baseline comparison. As we can see in Figure 1, increasing λ leads to an increase in precision in the resulting models. Using our D λ -PR estimation, we compute the corresponding PR curves (Figure 2(c)). The λ = 0 . 1 model, while best at α 0 . 1 , performs poorly at α 1 and α 10 , clearly demonstrating the impact of maximizing α λ .This pattern across models validates the efficacy of our method in minimizing the desired trade-off.

Using D KL vs D χ 2 on MNIST and FashionMNIST: We now demonstrate that we can improve the precision by directly minimizing the D λ -PR with the correct λ using pre-trained GLOW models [27] on both MNIST [54] and FashionMNIST [53]. Figure 3 and Figure 4 present the samples obtained. First, we observe that while λ increases, the visual quality improves, but the models focus on a few modes only ( 0 , 1 , 7 , 6 , 9 for MNIST and "Trouser" for FashionMNIST). Then training with both D KL and D rKL divergences aligns with our expectations: D KL training leads to high recall, while D rKL , to higher precision. However, according to Corollary 4.5, D rKL still favors low λ values; consequently, our models trained with λ > 0 . 1 demonstrate better precision than standard flow-GAN models. Furthermore, we find that both auxiliary g functions ( f KL and f χ 2 ) used to train the discriminator T perform well. In practice, training with f χ 2 proves to be more stable, particularly for FashionMNIST, with results reported in Appendix D.3. For larger models, we use exclusively g = f χ 2 .

(a)

λ

= 0

.

1

(b)

λ

= 1

(c)

λ

= 10

(d)

λ

= 0

.

1

(e)

λ

= 1

(f)

λ

= 10

10

-

1

10

0

10

1

0

.

700

0

.

725

0

.

750

0

.

775

0

.

800

0

.

825

P

10

-

1

10

0

10

1

0

.

66

0

.

68

0

.

70

0

.

72

0

.

74

0

.

76

R

PR-Chi2

PR-KL

MLE

KL

rKL

λ

λ

Training BigGAN on CIFAR-10 and CelebA64 Now we demonstrate that our method can also be used to train large generative models. Our choice to adopt the BigGAN architecture [7] was informed by several factors: its competitive performance close to state-of-the-art models; its versatility, permitting diverse experimental explorations; and the fact that it is publicly accessible, ensuring experiment reproducibility. We train BigGAN using both the baseline method (i.e., hinge loss) and our proposed method on CIFAR-10 [1] and CelebA64 [30]. A notable observation when training with different precision-recall trade-offs is the early elimination of modes from the target distribution at higher values of λ . As illustrated in Figure 5, models with low values of λ converge to achieve maximum recall, while those with λ > 1 rapidly saturate to a lower recall value. A similar behavior can be observed for models trained on CelebA, as shown in Figure D.18. In Table 2, we present the quantitative metrics (Precision, Recall, and FID) for the baseline BigGAN, the BigGAN models trained with varying trade-offs and the current state-of-the-art models: EDM-G++ [26] for CIFAR-10 and ADM-IP [38] for CelebA64. Employing our proposed method enables us to adjust the trade-off, allowing us to train models that closely approach the state-of-the-art recall and, for high λ , even outperform state-of-the-art models in terms of precision.

0

20k

40k

60k

80k

Iterations

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Recall

1.50

2.00

3.00

5.00

10.00

20.00

In every experiment, we compare our approach with traditional post-training techniques. In Table 2, we give the results for the hard truncation (also called the truncation trick in [7]) of the latent distribution Q for different ψ . We observe that this method enables to improve solely the precision by trading off the recall; however, note that the truncation can be use in addition to our approach.

Fine-tuning BigGAN on Imagenet128 and FFHQ Finally, we apply our method to pre-trained BigGAN models. To accomplish this, we implement a straightforward technique: initially, we train the discriminator for a brief period, allowing the model to transition from the vanilla training objective to the D dual g . This approach enables us to train BigGAN on large datasets such as ImageNet128 [43] and datasets with high dimensions such as FFHQ256 [24]. We compare our method with both hard truncation and soft truncation (also denoted temperature in [27]). Both methods can be used in addition to our approach. The metrics presented in Table 3 demonstrate that (1) we enhance a given model's precision (by +2 . 83% ) or recall (by +1 . 17% ) on ImageNet, thereby achieving state-of-the-art precision, and (2) our method compromises less on the trade-off than truncation. For instance, in FFHQ, for a similar precision improvement ( ≈ +15 . 5% ), recall is decreased by more than 5% for truncation methods and only by 1 . 65% with our approach.

7 Conclusion

In this paper, we present a novel method for training generative models using a new PR-divergence, D λ -PR . Our approach offers a unique advantage over existing methods as it allows for explicit control of the precision-recall trade-off in generative models. By varying the trade-off parameter λ , one can train a variety of models ranging from mode seeking (high precision) to mode covering (high recall), as well as more balanced models that may be more suitable for various applications. Through extensive experiments, we demonstrate the validity of our method and show that it scales well with larger dimensions and datasets. Our approach also provides insights into the implicit P&R trade-offs made by models trained with other f -divergences. By introducing the D λ -PR divergence and providing a systematic approach for training generative models based on user-specified trade-offs, we contribute to the development of more customizable generative models. Our method currently applies to GANs and NFs only, it is still unclear if a similar approach can be applied to trained diffusion models: a promising work [26] uses a discriminator to refine diffusion models, and could be used to estimate the D λ -PR .

Acknowledgment

We are grateful for the grant of access to computing resources at the IDRIS Jean Zay cluster under allocations No. AD011011296 and No. AD011014053 made by GENCI.

References

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Shun-ichi Amari and Hiroshi Nagaoka. Methods of Information Geometry. volume 191 of Translations of Mathematical Monographs , Providence, Rhode Island, April 2007. American Mathematical Society. ISBN 978-0-8218-4302-4 978-1-4704-4605-5. doi: 10.1090/mmono/191. URL http://www.ams.org/mmono/191 .

Lynton Ardizzone, Radek Mackowiak, Carsten Rother, and Ullrich Köthe. Training Normalizing Flows with the Information Bottleneck for Competitive Generative Classification. arXiv:2001.06448 [cs, stat] , January 2021. URL http://arxiv.org/abs/2001.06448 . arXiv: 2001.06448.

Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discriminator Rejection Sampling, February 2019. URL http://arxiv.org/abs/1810. 06758 . arXiv:1810.06758 [cs, stat].

Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible Residual Networks. In Proceedings of the 36 th International Conference on Machine Learning, Long Beach, California, PMLR 97, 2019 , May 2019. arXiv: 1811.00995.

Matan Ben-Yosef and Daphna Weinshall. Gaussian Mixture Generative Adversarial Networks for Diverse Datasets, and the Unsupervised Clustering of Images. arXiv:1808.10356 [cs, stat] , August 2018. URL http://arxiv.org/abs/1808.10356 . arXiv: 1808.10356.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis, February 2019. URL http://arxiv.org/abs/1809.11096 . arXiv:1809.11096 [cs, stat].

Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode Regularized Generative Adversarial Networks, March 2017. URL http://arxiv.org/abs/1612.02136 . arXiv:1612.02136 [cs].

Fasil Cheema and Ruth Urner. Precision Recall Cover: A Method For Assessing Generative Models. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , pages 6571-6594. PMLR, April 2023. URL https://proceedings.mlr.press/ v206/cheema23a.html . ISSN: 2640-3498.

Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual Flows for Invertible Generative Modeling. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. , July 2020. arXiv: 1906.02735.

Rob Cornish, Anthony L. Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows. In Proceedings of the 37th International Conference on Machine Learning , April 2021. URL http://arxiv.org/ abs/1909.13833 . arXiv: 1909.13833.

Zac Cranko and Richard Nock. Boosted Density Estimation Remastered. In Proceedings of the 36th International Conference on Machine Learning , pages 1416-1425. PMLR, May 2019. URL https://proceedings.mlr.press/v97/cranko19b.html . ISSN: 2640-3498.

Terrance DeVries, Michal Drozdzal, and Graham W. Taylor. Instance Selection for GANs, October 2020. URL http://arxiv.org/abs/2007.15255 . arXiv:2007.15255 [cs, stat].

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP, February 2017. URL http://arxiv.org/abs/1605.08803 . arXiv:1605.08803 [cs, stat].

Josip Djolonga, Mario Lucic, Marco Cuturi, Olivier Bachem, Olivier Bousquet, and Sylvain Gelly. Precision-Recall Curves Using Information Divergence Frontiers, June 2020. URL http://arxiv.org/abs/1905.10768 . arXiv:1905.10768 [cs, stat].

Robert Giaquinto and Arindam Banerjee. Gradient Boosted Normalizing Flows. arXiv:2002.11896 [cs, stat] , October 2020. URL http://arxiv.org/abs/2002.11896 . arXiv: 2002.11896.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. In 27th Conference on Neural Information Processing Systems (NeurIPS 2014) , June 2014. URL http://arxiv.org/abs/1406.2661 . arXiv: 1406.2661.

Matej Grci´ c, Ivan Grubiši´ c, and Siniša Šegvi´ c. Densely connected normalizing flows, November 2021. URL http://arxiv.org/abs/2106.04627 . arXiv:2106.04627 [cs].

Aditya Grover and Stefano Ermon. Boosted Generative Models, December 2017. URL http://arxiv.org/abs/1702.08484 . arXiv:1702.08484 [cs, stat].

Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models, January 2018. URL http://arxiv.org/ abs/1705.08868 . arXiv:1705.08868 [cs, stat].

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 8a1d694707eb0fefe65871369074926d-Abstract.html .

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, December 2020. URL http://arxiv.org/abs/2006.11239 . arXiv:2006.11239 [cs, stat].

Thibaut Issenhuth, Ugo Tanielian, David Picard, and Jeremie Mary. Latent reweighting, an almost free improvement for GANs. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 3574-3583, Waikoloa, HI, USA, January 2022. IEEE. ISBN 978-1-66540-915-5. doi: 10.1109/WACV51458.2022.00363. URL https://ieeexplore. ieee.org/document/9706934/ .

Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks, March 2019. URL http://arxiv.org/abs/1812.04948 . arXiv:1812.04948 [cs, stat].

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and Improving the Image Quality of StyleGAN, March 2020. URL http://arxiv. org/abs/1912.04958 . arXiv:1912.04958 [cs, eess, stat].

Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and Il-Chul Moon. Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models. In Proceedings of the 40 th International Conference on Machine Learning. , volume 202, Honolulu, Hawaii, USA, April 2023. JMLR. URL http://arxiv.org/abs/2211.17091 . arXiv:2211.17091 [cs] version: 3.

Durk P. Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada. , volume 31, 2018.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved Precision and Recall Metric for Assessing Generative Models. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. , October 2019. arXiv: 1904.06991.

Cheuk Ting Li and Farzan Farnia. Mode-Seeking Divergences: Theory and Applications to GANs. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , pages 8321-8350. PMLR, April 2023. URL https://proceedings.mlr.press/ v206/ting-li23a.html . ISSN: 2640-3498.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild, September 2015. URL http://arxiv.org/abs/1411.7766 . arXiv:1411.7766 [cs].

Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks, June 2018. URL http://arxiv.org/abs/1701.04722 . arXiv:1701.04722 [cs].

Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled Generative Adversarial Networks, May 2017. URL http://arxiv.org/abs/1611.02163 . arXiv:1611.02163 [cs, stat].

Laurence Illing Midgley, Vincent Stimper, Gregor N. C. Simm, and José Miguel HernándezLobato. Bootstrap Your Flow, March 2022. URL http://arxiv.org/abs/2111.11510 . arXiv:2111.11510 [cs, stat].

Laurence Illing Midgley, Vincent Stimper, Gregor N. C. Simm, Bernhard Schölkopf, and José Miguel Hernández-Lobato. Flow Annealed Importance Sampling Bootstrap, November 2022. URL http://arxiv.org/abs/2208.01893 . arXiv:2208.01893 [cs, q-bio, stat].

Thomas Minka. Divergence measures and message passing. page 17, 2005.

Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable Fidelity and Diversity Metrics for Generative Models, June 2020. URL http:// arxiv.org/abs/2002.09797 . arXiv:2002.09797 [cs, stat].

XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. On surrogate loss functions and $f$-divergences. The Annals of Statistics , 37(2), April 2009. ISSN 0090-5364. doi: 10.1214/08-AOS595. URL http://arxiv.org/abs/math/0510521 . arXiv:math/0510521.

Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input Perturbation Reduces Exposure Bias in Diffusion Models, February 2023. URL http:// arxiv.org/abs/2301.11706 . arXiv:2301.11706 [cs].

Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization, June 2016. URL http://arxiv.org/ abs/1606.00709 . arXiv:1606.00709 [cs, stat].

Teodora Pandeva and Matthias Schubert. MMGAN: Generative Adversarial Networks for Multi-Modal Distributions. arXiv:1911.06663 [cs, stat] , November 2019. URL http:// arxiv.org/abs/1911.06663 . arXiv: 1911.06663.

Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers, November 2021. URL http://arxiv.org/abs/2102.01454 . arXiv:2102.01454 [cs].

Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In arXiv:1505.05770 [cs, stat] , June 2016.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge, January 2015. URL http://arxiv. org/abs/1409.0575 . arXiv:1409.0575 [cs].

Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing Generative Models via Precision and Recall. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada , October 2018. URL http://arxiv.org/abs/1806.00035 . arXiv: 1806.00035.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs, June 2016. URL http://arxiv.org/abs/1606. 03498 . arXiv:1606.03498 [cs].

Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings , pages 1-10, Vancouver BC Canada, August 2022. ACM. ISBN 978-1-4503-9337-9. doi: 10.1145/3528233.3530738. URL https://dl.acm.org/doi/10. 1145/3528233.3530738 .

Loic Simon, Ryan Webster, and Julien Rabin. Revisiting precision recall definition for generative modeling. In Proceedings of the 36th International Conference on Machine Learning , pages 5799-5808. PMLR, May 2019. URL https://proceedings.mlr.press/v97/simon19a. html . ISSN: 2640-3498.

Rodrigue Siry, Ryan Webster, Loic Simon, and Julien Rabin. On the Theoretical Equivalence of Several Trade-Off Curves Assessing Statistical Proximity. Journal of Machine Learning Research , 24, 2023.

Vincent Stimper, Bernhard Schölkopf, and José Miguel Hernández-Lobato. Resampling Base Distributions of Normalizing Flows. arXiv:2110.15828 [cs, stat] , February 2022. URL http://arxiv.org/abs/2110.15828 . arXiv: 2110.15828.

Ugo Tanielian, Thibaut Issenhuth, Elvis Dohmatob, and Jeremie Mary. Learning disconnected manifolds: a no GANs land. In Proceedings of the 37 th International Conference on Machine Learning, Vienna, Austria, PMLR 119, 2020 , December 2020. arXiv: 2006.04596.

Chenyang Tao, Liqun Chen, Ricardo Henao, Jianfeng Feng, and Lawrence Carin Duke. Chisquare Generative Adversarial Network. In Proceedings of the 35th International Conference on Machine Learning , pages 4887-4896. PMLR, July 2018. URL https://proceedings. mlr.press/v80/tao18b.html . ISSN: 2640-3498.

Alexandre Verine, Benjamin Negrevergne, Yann Chevaleyre, and Fabrice Rossi. On the expressivity of bi-Lipschitz normalizing flows. In Proceedings of The 14th Asian Conference on Machine Learning , pages 1054-1069. PMLR, April 2023. URL https://proceedings. mlr.press/v189/verine23a.html . ISSN: 2640-3498.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms, September 2017. URL http://arxiv.org/ abs/1708.07747 . arXiv:1708.07747 [cs, stat].

Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs , 2, 2010. URL http://yann.lecun.com/exdb/mnist .

A Supplementary background

A.1 f -divergences

f -divergence between two probability distributions P and Q over a common support X is defined as:

where p ( x ) and q ( x ) denote the densities of P and Q , respectively, and f is a convex function such that f (1) = 0 . Any D f admits a dual variational form [37], with T denoting the set of all measurable functions X → R :

The properties of f -divergence are as follows:

Non-Negativity: For any two probability distributions P and ̂ P on X , we have D f ( P ∥ ̂ P ) ≥ 0 . The equality holds if and only if P = ̂ P .

Convexity in f : If f ( u ) is a convex function, then D f ( P ∥ ̂ P ) is convex in the input distributions P and ̂ P .

̸

Non-Symmetry: Generally, f -divergence is not symmetric, i.e., D f ( P ∥ ̂ P ) = D f ( ̂ P ∥ P ) .

Linearity: f -divergence is linear in f , i.e., for any two convex functions f 1 ( u ) and f 2 ( u ) , and for any two real numbers a and b , the divergence D af 1 + bf 2 ( P ∥ ̂ P ) = a D f 1 ( P ∥ ̂ P ) + b D f 2 ( P ∥ ̂ P ) .

Specific choices of the function f in the f -divergence definition can yield various well-known divergence measures. We will review some of them of the principal properties. For every divergence, we report here the generator function f , its convex conjugate f ∗ , the domain of the convex conjugate dom( f ∗ ) , the typical activation function used to ensure that T ( x ) ∈ dom( f ∗ ) and finally the optimal discriminator T ∗ .

Kullback Leibler:

Notation: D KL ,

Generator Function: f ( u ) = u log( u ) ,

Convex Conjugate domain: dom( f ∗ ) = R ,

Convex Conjugate: f ∗ ( t ) = exp( t - 1) ,

Activation: v ↦→ v ,

Optimal discriminator: T ∗ ( x ) = 1 + log ( p ( x ) ̂ p ( x ) ) .

Reverse Kullback Leibler:

Notation: D rKL ,

Generator Function: f ( u ) = - log( u ) ,

Convex Conjugate domain: dom( f ∗ ) = R - ,

Convex Conjugate: f ∗ ( t ) = - 1 - log - t ,

Activation: v ↦→- exp v ,

Optimal discriminator: T ∗ ( x ) = - ̂ p ( x ) p ( x )

χ 2 Pearson:

Notation: D χ 2 ,

Generator Function: f ( u ) = ( u - 1) 2 ,

Convex Conjugate domain: dom( f ∗ ) = R ,

Convex Conjugate: f ∗ ( t ) = t 2 / 4 + t ,

Activation: v ↦→ v ,

Optimal discriminator: T ∗ ( x ) = 2 ( p ( x ) p ( x ) - 1 )

GAN:

Notation: D GAN ,

Generator Function: f ( u ) = u log( u ) - ( u +1)log( u +1) ,

Convex Conjugate domain: dom( f ∗ ) = R - ,

Convex Conjugate: f ∗ ( t ) = - log (1 - exp( t )) ,

Activation: v ↦→- exp - log (1 - exp( v )) ,

Optimal discriminator: T ∗ ( x ) = log p ( x ) p ( x )+ ̂ p ( x ) .

Total Variation:

Notation: D TV ,

Generator Function: f ( u ) = | u -1 | ,

Convex Conjugate domain: dom( f ∗ ) = [ -1 2 , 1 2 ] ,

Convex Conjugate: f ∗ ( t ) = t ,

Activation: v ↦→ tanh( v ) ,

Optimal discriminator: T ∗ ( x ) = 1 2 sign ( p ( x ) p ( x ) -1 )

PR-Divergence:

Notation: D λ - PR for λ > 0 ,

Generator Function: f ( u ) = max( λu, 1) - max( λ, 1) ,

Convex Conjugate domain: dom( f ∗ ) = [0 , λ ] ,

Convex Conjugate: f ∗ λ ( t ) = t/λ for λ ≤ 1 and f ∗ λ ( t ) = t/λ + λ - 1 otherwise,

Activation: v ↦→ λσ ( v ) , where σ is the sigmoid function,

Optimal discriminator: T ∗ ( x ) = λ sign ( p ( x ) ̂ p ( x ) - 1 ) .

A.2 Bregman Divernce

The Bregman Divergence under a strictly convex function f : R n → R with a continuously differentiable interior, between two points x and y in the interior of the domain of f , is denoted by Breg f and is defined as:

where ∇ f ( y ) is the gradient of f at y , and ⟨ ., . ⟩ is the inner product in R n . It follows some properties as a distance metrics:

Non-Negativity: For any x , y in the interior of the domain of f , we have Breg f ( x , y ) ≥ 0 . The equality holds if and only if x = y .

Convexity: The Bregman Divergence Breg f ( x , y ) is convex in its first argument. That is, for any x 1 , x 2 , y in the interior of the domain of f and any t ∈ [0 , 1] , we have:

̸

Non-Symmetry: Unlike some other distances or divergences, the Bregman Divergence is not symmetric, meaning that in general, Breg f ( x , y ) = Breg f ( y , x ) .

Additivity: If f ( x ) = f 1 ( x 1 ) + · · · + f d ( x d ) where each f i is a convex function, then the Bregman Divergence decomposes into a sum of Bregman Divergences, i.e., Breg f ( x , y ) = Breg f 1 ( x 1 , y 1 ) + · · · +Breg f d ( x d , y d ) .

Taylor Approximation: Bregman Divergence is essentially the error of the first-order Taylor approximation of f around the point y at the point x .

Connection with Dual Functions: If f is convex, then it has a convex conjugate (or dual function) f ∗ . The Bregman Divergence Breg f ( x , y ) is related to the Bregman Divergence Breg f ∗ ( ∇ f ( y ) , ∇ f ( x )) between the gradients of f at x and y , where the gradient map ∇ f serves as a Legendre transformation between the primal and dual spaces.

B Proof and supplementary for Section 4

B.1 Proof for Theorem 4.3

We have to prove that α ( λ ) can be written as a function of an f -divergence for any λ ∈ R + . First we can develop the expression of α ( λ ) :

For this integral to be considered as an f -divergence, we need f to be first convex lower semicontinuous and then to satisfy f (1) = 0 . However, for every a, b ∈ R , the min satisfies min( a, b ) = a + b -max( a, b ) . Therefore,

Thus, we can take f ( u ) = max( λu, 1) -max( λ, 1) such that f (1) = 0 . The precision becomes:

Consequently, α ( λ ) can be written as a function of an f -divergence D λ -PR with f ( u ) = max( λu, 1) -max( λ, 1) .

Now we prove the converse. Suppose there exists a strictly decreasing linear 4 function h : [0 , 1] → R + and an f -divergence D f such that h ( α λ ( P ∥ ̂ P )) = D f ( P ∥ ̂ P ) for all P, ̂ P ∈ P ( X ) .

For P = ̂ P , we get from the definition of α λ that α λ ( P ∥ P ) = min( λ, 1) . Hence,

Combining the above with the fact that h is a strictly decreasing linear function, we see that for any fixed λ , h must be of the form, h ( u ) = c λ (min( λ, 1) -u ) , where c λ > 0 is a constant. Now,

where the last equality follows from the first part of the theorem, which shows that α λ ( P ∥ ̂ P ) = min( λ, 1) -D λ -PR ( P ∥ ̂ P ) . Rewriting the above inequality, we get the following.

By the uniqueness theorem of f -divergence f ( u ) = c 1 c λ f λ ( u ) + c 2 ( u -1) for some constants c 1 , c 2 ∈ R .

B.2 Proof of Proposition 4.2

If the generator function f of the Precision-Recall Divergence is f ( u ) = max( λu, 1) -max( λ, 1) then its Fenchel conjugate function is:

If t > λ or λ < 0 , then the sup u ∈ R + { tu -max( λu, 1) } = ∞ for respectively u → ∞ and u → -∞ . The domain of f ∗ is thus restricted to [0 , λ ] . Thus for 0 ≤ t ≤ λ , the supremum is obtained for u = 1 /λ since 0 is in the sub-differential of the function in 1 /λ as Figure 6(b).

1/

2

1

1/

3

u

0

f

(

u

)

1<1

2=1

3>1

(a) Function

f

(

u

)

for different values of

λ

.

0

1/

u

t

1/

1

t

2/

1

t

3/

1

ut

u

, 1)

t

1 = /10

t

1 = /2

t

1 =9 /10

(b) Function

u

ut

-

λu,

1)

for values of

t

between

0

and

λ

.

Consequently the Fenchel conjugate of f is:

Finally, the optimal discriminator T opt by taking the derivative of f in p ( x ) ̂ p ( x ) , we get:

Then we can compute the compute the reverse D λ -PR :

With this results, we can show that :

Then since D 1 -PR ( P ∥ ̂ P ) = ∫ X max( ̂ p ( x ) , p ( x )) -p ( x )d x = ∫ X max( ̂ p ( x ) -p ( x ) , 0)d x and D 1 -PR ( P ∥ ̂ P ) = D 1 -PR ( ̂ P ∥ P ) , we have:

B.3 Proof of Theorem 5.2

From now on, assume the support of P and ̂ P coincide. For any T : X → R ,

Let T opt ∈ arg sup D dual f,T ( P ∥ ̂ P ) . For any T : X → R

It is known that for all x ∈ X we have ∇ f ∗ ( T opt ( x )) = p ( x ) ̂ p ( x ) :

Recall that for any continuously differentiable strictly convex function f , the Bregman divergence of f is Breg f ( a, b ) = f ( a ) -f ( b ) -⟨∇ f ( b ) , a -b ⟩ . So we have

Let us now use the following property: Breg f ( a, b ) = Breg f ∗ ( a ∗ , b ∗ ) where a ∗ = ∇ f ( a ) and b ∗ = ∇ f ( b ) .

Let us define r ( x ) = ∇ f ∗ T ( x ) as our estimator of p ( x ) / ̂ p ( x ) . So finally, we have

B.4 Proof of Theorem 5.3

Now assume that f is µ -strongly convex, then Breg f ( a, b ) ≥ µ 2 ∥ a -b ∥ 2 If E ̂ P [ Breg f ( r ( x ) , p ( x ) ̂ p ( x ) )] ≤ ϵ and if f is µ -strongly convex, then

Consider an arbitrary f-divergence D g ( P ∥ ̂ P ) = ∫ g ( dP d ̂ P ) d ̂ P . Define D primal g,T ( P ∥ ̂ P ) = ∫ g ( r ( x )) d ̂ P . Then,

where ( a ) follows from the σ -Lipschitz assumption on g , ( b ) follows from Jensen's inequality and finally, ( c ) follows from equation (36). s

B.5 Proof of Theorem 8

Let c : R + ↦→ R be a C 2 function and take u min and u max . The goal is to express f ( u ) for all u ∈ [ u min , u max ] as a weighted average of f PR λ :

First, let us assume that u min ≤ 1 and u max ≥ 1 , then the terms can be decomposed and the integral split to evaluate the max :

1 /u 1 /u 1 /u

We would like ∫ 1 /u min 1 /u max c ′′ ( λ ) f PR λ ( u )d λ to be equal to f between u min and u max . But since two f -divergences generated by f and g are equals if there is a c ∈ R such that f ( u ) = g ( u ) + c ( u -1) , the Divergence generated by ∫ 1 /u min 1 /u max c ′′ ( λ ) f PR λ ( u )d λ is equal to the divergence generated by uc ( 1 u ) -c (1) . Therefore, we require c to satisfy:

By differentiating with respect to u , we have:

And finally:

Consequently, with λ = 1 /u , we have that:

With such a results, with m = min X ( ̂ p ( x ) p ( x ) ) and M = max X ( ̂ p ( x ) p ( x ) ) , we can write any f -divergence as:

B.6 Proof of Corollary 4.5

In particular for the D KL , f ( u ) = u log u , therefore f ′′ ( u ) = 1 /u which gives:

And for the D rKL we can either use Equation 48 with f ( u ) = -log u or use the fact that D λ -PR ( P ∥ ̂ P ) = λ D 1 /λ -PR ( ̂ P ∥ P ) :

C Minimizing the Precision-Recall divergence

In Section 5, we outline a novel strategy to train models for challenging f -divergences. Rather than directly employing the f -GAN framework and minimizing the dual variational form of D λ -PR , we introduce an auxiliary function, denoted by g , which is easier to train. Practical choices for g could include functions such as f KL or f χ 2 , which have been shown to be easily trainable.

Using this approach, a discriminator T is trained with the objective of maximizing D dual g,T . This is achieved by drawing samples from both P and ̂ P and computing the estimate in the dual form. Under optimal conditions, we find ∇ g ∗ ( T opt ( x )) = p ( x ) / ̂ p ( x ) , enabling us to estimate the f -divergence based on T using what we term the primal estimate.

The primal estimate is defined as D primal f,T ( P ∥ ̂ P ) = ∫ X ̂ p ( x ) f ( r ( x )) d x , where r ( x ) = ∇ g ∗ ( T ( x )) . This strategy and its corresponding training procedure are visually represented in Figure C.7.

We have shown that the choice of g affects the quality of the estimation of D f . We have shown that to have guarantees on the quality of the estimation, g must be strictly convex. We show empirically that g = f χ 2 leads to a better approximation than g = f KL . We train 200 discriminators on 20 Gaussian random two-dimensional mixtures to maximize D dual g,T with g = f KL and g = f χ 2 . In Figure C.8, we report the results. On the x -axis, we report the training loss quality D g ( P ∥ ̂ P ) -D dual g,T ( P ∥ ̂ P ) and on the y -axis we report the estimation quality : D f ( P ∥ ̂ P ) -D primal f,T ( P ∥ ̂ P ) | . The dotted lines correspond to the Mahalanobis distance of each set of experiments. In this experiment, we use a 4 linear layers 2-1024-512-256-1 neural network with LeakyRelu activation between layers. The last activation is set to match dom( f ∗ ) (see Section A.1). We use a learning rate of 2 . 10 -5 with Adam optimizer.

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

4

.

0

|D

f

(

P

‖

̂

P

)

-D

primal

f,T

(

P

‖

̂

P

)

|

D

g

(

P

‖

P

)

-D

dual

g,T

(

P

‖

P

)

fχ

2

f

KL

̂

̂

We observe in this experiment that 1) the difference between D KL and its dual is usually lower (the blue circle is lower on x -axis) and yet 2) the estimation of D f is usually better with g = f χ 2 (the red circle is lower on y -axis). These observations corroborate our claim that employing f χ 2 as the auxiliary divergence is a more advantageous choice compared to f KL . Although both divergence measures exhibit satisfactory performance in the context of the MNIST and FashionMNIST experiments, we find f χ 2 to demonstrate superior stability, reinforcing its suitability in this training framework.

Complete version of the algorithm In Section 5, we introduce a condensed version of our training algorithm. For practical purposes, however, we implement a more refined variant of this algorithm using stochastic gradient descent, detailed in Algorithm 1. This approach involves segmenting the dataset into batches of size N and computing the distinct losses for each individual batch. Our algorithm is closely related to the GAN training procedure, operating on an iterative principle of alternating updates to the discriminator T and the generator G until a convergence state is reached. In particular, our method differs from the standard GAN training protocol in two key aspects. First, the generator G in our case is trained based on an estimation of a distinct f -divergence. Second, this specific f -divergence is calculated in terms of its primal form, rather than its dual form, a marked departure from the established GAN training practice.

Algorithm 1 Stochastic Gradient Descent for the two-step approach for minimizing D λ -PR . For each batch B composing the dataset D , the discriminator T is trained on L T , the empirical estimation of D dual f,T ( P ∥ ̂ P ) and the generator G is trained on L G , the empirical estimation of D primal λ -PR ( P ∥ ̂ P ) .

D Experiments

D.1 Naive Approach

In this section, we juxtapose the straightforward method with our technique delineated in Section 5. We achieve this by evaluating the optimization process of two identical G models: one is trained using the conventional approach and the other using our methodology. In reality, there exist two significant variations between these two training procedures: training losses and the final layer of the discriminator T . In both procedures, we use f ∗ ( T ( x )) , implying that the co-domain must be encapsulated within dom( f ∗ ) . More information on activation can be found in the Appendix A.1. Aside from these disparities, we utilize the identical training process as outlined in the previous section to train the models on CIFAR-10. The training procedures are compared in Figure D.9. To evaluate training procedures, we plot the loss L G , used to train G , either the dual estimation of D λ -PR in the naive approach or the primal estimation of D λ -PR in our method. To track the evolution of the discriminator, we plot discriminating predictions on the input conditioned on the class (real or fake). Precisely, we compute the accuracy on a batch of real or fake images, how many the discriminator achieves to identify as such. Finally, we plot the Precision and Recall computed at each epoch.

We observe that for varying values of λ , the discriminator T undergoes the training process using the naive approach, leading to an unsatisfactory estimation of D dual f,T .

0

20000

40000

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Precision

0

20000

40000

0

.

0

0

.

2

0

.

4

0

.

6

Recall

0

20000

40000

60000

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

λ

= 0

.

20

Loss on

G

:

L

G

0

20000

40000

60000

0

25

50

75

100

Accuracy of

T

(in %)

0

20000

40000

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Precision

0

20000

40000

0

.

0

0

.

2

0

.

4

Recall

0

20000

40000

60000

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

λ

= 1

.

00

Loss on

G

:

L

G

0

20000

40000

60000

0

25

50

75

100

Accuracy of

T

(in %)

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Precision

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

2

0

.

4

Recall

0

20000

40000

60000

80000

Iterations

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

λ

= 5

.

00

Loss on

G

:

L

G

0

20000

40000

60000

80000

Iterations

0

25

50

75

100

Accuracy of

T

(in %)

D.2 Synthetic data: 8 Gaussians

For the synthetic data, we use a RealNVP [14] for the generator G . We use an 8-coupling step composed of each of 2 linear layers 2-256-2 with LeakyRelu activation in between. For the discriminator, we used a 4 linear layers 2-1024-512-256-1 neural network with LeakyRelu activation between layers. For both, we use Adam optimizer with a learning rate of 2 . 10 -5 for G and 1 . 10 -4 for T . G has 540 k parameters and 660 k for T .

Then, to estimate the PR curves, both methods from Sajjadi et al. [44] and Simon et al. [47] are developed for image dataset. To estimate these curves, we use our own estimation of D λ -PR . We computed D λ -PR for λ = tan( θ ) with θ between 0 and π/ 2 . The observations from the generated curves underscore a compelling finding: Each model has been effectively optimized for a unique, specific λ , resulting in its superior performance on the corresponding D λ -PR . This indicates a clear correspondence between the intended optimization and the achieved performance, suggesting the efficacy of our training algorithm in optimizing models for specific performance metrics. Such a successful matching of model optimization to λ underscores the feasibility of tailoring model training according to a specific precision-recall trade-off objective.

D.3 MNIST and FashionMNIST

The training procedure for MNIST and FashionMNIST is strictly the same. For both we use a multiscale glow [27]. The model has three levels processing images of size 4 × 16 × 16 , 16 × 8 × 8 , 64 × 8 × 8 . Each level has 16 blocks of affine coupling with 3 layers of 512 channels of convolutional

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

β

α

λ

= 0

.

1

λ

= 1

λ

= 10

λ

= 0

.

1

λ

= 1

λ

= 10

D

KL

D

rKL

operations, leading to a total of 85 . 2 Mparameters. For the discriminator, we use a 4 linear layers 10241024-512-256-1 neural network with LeakyRelu activation between layers, with 1 . 7 Mparameters. Both are trained with Adam using a learning rate of 1 . 10 -5 for T and 1 . 10 -6 for G . For both dataset, we train a model for 250 epochs using maximum likelihood estimation (MLE) in 4 GPUs V100 ( ∼ 200 hours). The models are then fine-tuned with their different losses on 12 V100 GPUs for 30 epochs ( ∼ 2 hours). For two epochs we train the discriminator only and then both. For MNIST, we report results as graphs in Figure D.11, the quantitative results in Table 4 and samples in Figure D.3. For FashionMNIST, we report the results as graph in Figure D.13, the quantitative results in Table 4 and samples in Figure D.3.

10

-

1

10

0

10

1

λ

2

.

375

2

.

400

2

.

425

2

.

450

2

.

475

2

.

500

IS

10

-

1

10

0

10

1

λ

5

10

15

20

25

FID

10

-

1

10

0

10

1

λ

0

.

700

0

.

725

0

.

750

0

.

775

0

.

800

0

.

825

P

10

-

1

10

0

10

1

λ

0

.

66

0

.

68

0

.

70

0

.

72

0

.

74

0

.

76

R

PR-Chi2

PR-KL

MLE

KL

rKL

(a) MLE

(b)

λ

= 0

.

1

(c)

λ

= 1

(d)

λ

= 10

9

2

h

0

4

8

8

6

3

S

2

R

7

3

9

2

7

6

9

8

2

2

9

6

7

3

7

>

9

9

3

8

4

6

2

7

1

7

1

6

1

9

1

8

F

S

9

7

6

1

7

9

7

3

9

6

7

4

9

4

3

8

3

4

2

/

2

2

1

4

7

9

8

3

B

/

I

3

3

7

3

I

。

9

8

6

b

7

9

8

G

8

2

1

4

9

6

2

7

h

7

4

3

7

7

/

9

3

1

7

4

7

7

1

9

/

/

9

7

6

3

7

S

θ

6

/

9l

6

9

/

1

6

4

7

6

G

9

9

1

?

7

/

9

1

9

7

7

3

3

8

b

/

0

9

9

6

9

2

7

9

9

4

8

a

9

7

7

1

2

1

10

-

1

10

0

10

1

λ

2

.

2

2

.

4

2

.

6

2

.

8

3

.

0

3

.

2

IS

10

-

1

10

0

10

1

λ

40

60

80

100

120

FID

10

-

1

10

0

10

1

λ

0

.

55

0

.

60

0

.

65

0

.

70

0

.

75

P

10

-

1

10

0

10

1

λ

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

0

.

45

0

.

50

R

PR-Chi2

PR-KL

MLE

KL

rKL

(a) MLE

(b)

λ

= 0

.

1

(c)

λ

= 1

(d)

λ

= 10

D.4 Training BigGAN on CIFAR-10 and CelebA64

In this section we give the details of the experiments on training BigGAN [7] models. To do this, we modify the official implementation of PyTorch 5 of BigGAN by Brock et al. [7]. G and T respectively count 4 . 3 Mand 4 . 2 Mparameters for CIFAR-10 and 32 . 0 Mand 19 . 5 Mfor CelebA64. CIFAR-10's models are trained on 2 A100 80GB GPUs with a batch size of 128 for approximately 100k iterations ( ∼ 7 hours), while CelebA64's models have been trained on 4 A1200 32GB GPUs with a batch size of 128 for 95k iteration ( ∼ 20 hours). Quantitative results are reported in Table 2. The graphic representation of the results is in Figures D.16 and D.20. The training curves representative of FID, IS, P and R during training are reported in Figures D.17 and D.18.

(a) BigGAN Baseline

(b)

λ

= 0

.

10

(c)

λ

= 1

(d)

λ

= 10

10

-

1

10

0

10

1

λ

4

5

6

7

IS

10

-

1

10

0

10

1

λ

0

.

82

0

.

84

0

.

86

0

.

88

0

.

90

P

10

-

1

10

0

10

1

λ

0

.

7

0

.

8

0

.

9

D

10

-

1

10

0

10

1

λ

10

20

30

40

50

60

FID

10

-

1

10

0

10

1

λ

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

R

10

-

1

10

0

10

1

λ

0

.

55

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

C

Our Appoach

Baseline

0

20000

40000

60000

80000

Iterations

0

1

2

3

4

5

6

7

Inception Score

0.05

0.10

0.20

0.30

0.50

0.67

1.00

20000

40000

60000

80000

Iterations

0

50

100

150

200

250

FID

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Precision

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Recall

0

20000

40000

60000

80000

Iterations

0

1

2

3

4

5

6

7

Inception Score

1.00

1.50

2.00

3.00

5.00

10.00

20.00

20000

40000

60000

80000

Iterations

10

20

30

40

50

60

70

80

FID

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Precision

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Recall

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Inception Score

0.20

0.50

0.67

1.00

0

20000

40000

60000

80000

Iterations

10

20

30

40

50

60

FID

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Precision

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Recall

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

Inception Score

1.00

1.50

2.00

5.00

0

20000

40000

60000

80000

Iterations

20

40

60

80

100

120

FID

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Precision

0

20000

40000

60000

80000

Iterations

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Recall

(a) BigGAN Baseline

(b)

λ

= 0

.

5

(c)

λ

= 1

(d)

λ

= 10

10

0

λ

2

.

50

2

.

55

2

.

60

2

.

65

2

.

70

2

.

75

IS

10

0

λ

0

.

78

0

.

79

0

.

80

0

.

81

0

.

82

0

.

83

0

.

84

P

10

0

λ

0

.

9

1

.

0

1

.

1

1

.

2

D

10

0

λ

10

15

20

FID

10

0

λ

0

.

25

0

.

30

0

.

35

0

.

40

0

.

45

0

.

50

0

.

55

R

10

0

λ

0

.

42

0

.

44

0

.

46

0

.

48

0

.

50

0

.

52

0

.

54

C

Our Appoach

Baseline

D.5 Fine-tuning BigGAN on ImageNet128 and FFHQ256.

In this section, we fine-tune the pre-trained BigGAN models. For ImageNet128, we use the weights of the model trained by Brock et al. [7], while for FFHQ256 we use a model pre-trained by our self, explaining why the FFHQ models is under performing. We train models on 4 V100 GPUs with a batch size of 128 for 10k iterations each. The graphic representation of the results is reported in Figures D.21 and D.22 and samples are shown in Figures D.5 and D.5.

10

0

λ

50

60

70

80

90

IS

10

0

λ

0

.

27

0

.

28

0

.

29

0

.

30

0

.

31

P

10

0

λ

0

.

135

0

.

140

0

.

145

0

.

150

0

.

155

D

10

0

λ

10

12

14

16

18

20

FID

10

0

λ

0

.

300

0

.

325

0

.

350

0

.

375

0

.

400

0

.

425

R

10

0

λ

0

.

145

0

.

150

0

.

155

0

.

160

0

.

165

0

.

170

C

Our Appoach

Baseline

10

0

λ

2

.

8

2

.

9

3

.

0

3

.

1

3

.

2

3

.

3

IS

10

0

λ

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

P

10

0

λ

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

.

0

1

.

1

D

10

0

λ

35

36

37

38

39

40

41

FID

10

0

λ

0

.

085

0

.

090

0

.

095

0

.

100

R

10

0

λ

0

.

50

0

.

55

0

.

60

C

Our Appoach

Baseline

(a) BigGAN Baseline

(b)

λ

= 0

.

2

(c)

λ

= 1

(d)

λ

= 5

(a) BigGAN Baseline

(b)

λ

= 0

.

5

(c)

λ

= 1

(d)

λ

= 2

E Training the AUC

In this paper, we propose a method to optimize any trade-off between precision and recall. In practice, we optimize a model to maximize any α λ ( P ∥ ̂ P ) . We could also optimize for the area under the PR-Curves PRD( P, ̂ P )( P, ̂ P ) . In this section, we will show how we can compute the AUC in terms of α λ ( P ∥ ̂ P ) and train a model on a small dimension dataset and compare the results with the model trained on several λ . First, we must compute the AUC:

Proposition E.1 (AUC under the ∂ PRD( P, ̂ P ) ) . The area under the curve is:

Proof. The AUC can be computed by integrating with respect to an angle θ in the first quadrant:

Therefore, with λ = tan θ , we have r ( θ ) = α tan( θ ) ( P ∥ ̂ P ) / sin θ (see Figure E.25). Thus:

Using such expression, we can estimate the AUC and train model to directly optimize the AUC. It loses the focus on a specific trade-off between precision and recall, but instead optimizes overall performance. We trained a RealNVP model on a 2D 8 Gaussians example, similar to Section 6. We observe in Figure 27(d) that while the model trained to minimize the AUC performs poorly in terms of λ = 0 . 1 and λ = 1 . However, we can clearly see that the AUC is greater and the model performs better in λ = 1 than the model trained for λ = 1 itself. We can also observe in Figure 27(d), the resulting model.

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

β

λ

= 0

.

1

λ

= 1

λ

= 10

λ

= 0

.

1

λ

= 1

λ

= 10

AUC

α

(a)

λ

= 0

.

1

(b)

λ

= 1

(c)

λ

= 10

(d) AUC

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a way to train generative models such that they obtain a user defined tradeoff between sample fidelity and variety. Main findings are that the definition of precision and recall from previous works (by Simon et al.) can be written as a $f$-divergence (named PR-divergence) and that other $f$-divergences, such as KL or reverse-KL, can be formulated in terms of PR-divergence.\n\nSTRENGTHS:\nThe strengths of the paper are thorough analysis of the $f$-divergences and solid theoretic foundation of PR-divergence. The paper provides interesting insights into what PR-tradeoff often used divergences, such as the KL, optimize for. Additionally, it proposes an algorithm to train generative models using an auxiliary divergence, since training models with $f$-GAN is difficult in practice.\n\nWEAKNESSES:\nThe main weakness of the paper are empirical results. Although the method works as expected from the theory and it can be used to explicitly control the tradeoff between precision and recall it is far behind the current state-of-the-art.\n\nQUESTIONS:\n1. To further demonstrate the usefulness of explicitly controlling PR-tradeoff, I would like to see FID and PR curves from a following comparison: truncating BigGAN vs. applying classifier-free guidance to ADM models vs. training a separate model using different $\\lambda$ to control the PR-tradeoff (on CIFAR-10, ImageNet 128x128 datasets).\n\n2. Why do the FID results using BigGAN differ from the official results, reported by Brock et al. [1], on CIFAR-10 (13.37 vs. 14.73) and ImageNet 128x128 (9.83 vs. 8.7)? Two things that come to mind could be that you either use a validation set to compute FID or a different instance of Inception-V3 network (see App. A from [2] for other possible explanations). Also a minor note, on ImageNet 128x128 the baseline actually has the best FID instead of the model with $\\lambda=0.2$.\n\nMinor notes from checking the proofs:\n\n3. There are different versions of defining $\\lambda \\in [0, \\inf]$ in Definition 4.1 and Theorem 4.3. What is the difference?\n4. Proof of Theorem in App. Eq. (6): Is the last term missing $\\hat{p}(x)$, but it is correctly there in Eq. (7)?\n5. App. Eq. (8): the first integral term has extra $($ and is missing $\\textrm{d}\\boldsymbol{x}$?\n6. App. B.5: Title should be Proof of Theorem 4.4.\n7. App. Eq. (30): Third term is missing $u_{\\textrm{min}}$ from the denominator but it reappears correctly in Eq.(31)? Why does $c(1/u_{\\textrm{max}})$ change suddenly to $c(0)$ (it cancels out but I’m still curious)?\n8. Why in App. Eq (32) $\\textrm{lim}$-term appears when differentiating w.r.t $u$?\n9. Is there a typo in Theorem 5.2? Should it be $r(x) = \\nabla g^*(T(x))$ instead of $f^*$?\n\n[1]: Brock et al., Large Scale GAN Training for High Fidelity Natural Image Synthesis\n\n[2]: Kynkäänniemi et al., The Role of ImageNet Classes in Fréchet Inception Distance\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a technique for training generative models using an objective which approximates so called PR-divergence.  By adjusting a parameter, the paper claims that it is possible to train an array of models, from those that prioritize  high precision to those that prioritize high recall (mode seeking vs mode covering), as well as more balanced models. The supporting experiments involving BigGAN training on CIFAR10, CelebA64 and finetuning on ImageNET128 and FFHQ256 are described.\n\nSTRENGTHS:\n1. Authors consider the family of f-divergences and show that , given a trade-off parameter $\\lambda$, for  a particular element \n${\\cal D}_{\\lambda -PR}$  of this family, the minimization of  this element is equivalent to the maximization \nof the value at $\\lambda$ of the first component of so called PR curve from Saijidi et al [40]. \n\n2. As the minimization of ${\\cal{D}}_{\\lambda -PR}$  via f-GAN approach would fail, the authors describe a way to minimize a certain approximation to this objective and give an estimate for the error of this approximation. \n\n3. The provided experiments show that  for NFs-GLOW on 2D synthetic dataset, MNIST and FMNIST, and for BigGAN on CIFAR-10, CelebA64,  the training with the approximation  objective and small $\\lambda$ leads indeed to models with better mode covering, and, to a much lesser extent for BigGAN, the bigger tradeoff parameter $\\lambda$ leads to better quality of generated samples.\n\nWEAKNESSES:\nAlthough the approach looks interesting, the paper raises several red flags that prevent me from endorsing it. \n\n1. The experimental validation is limited, especially the gain in precision, i.e. the sample quality, for high tradeoff is insubstantial, compared with other methods. One example of this is the quality of samples on Figure D.11.d : all 100 supposedly high quality samples, at the highest tradeoff $\\lambda=20$, are clearly of bad quality. \n\n2. Similarly, the NFs results on Fig 1 in the simple 2D synthetic setup of eight gaussians do not really demonstrate excellent precision despite choosing the parameter lambda supposedly largely favoring the quality of generated samples.  Compare this with e.g. Wasserstein-GP GAN from 2017 paper (https://github.com/caogang/wgan-gp).\n\n3. The code for the reproducibility check is not provided, despite the mentioning of an anonymized repository on page 8, the link is absent. This is especially disappointing as some evaluation numbers looks somewhat strange, like the P-value at $\\lambda=20$ in Table 2 compared with FID, but it is not possible to try to reproduce this number or to see the details of its calculation. \n\n4. Despite the fact that there are numerous variants in the literature of precision-recall scores, only one variant, based on k=3-NN only, from [27] is provided for evaluation of the models, which is known to not always work properly especially in regions of low density. \n\n5. The related works description has some lacunas like \"Reliable Fidelity and Diversity Metrics for Generative Models\" (ICML'2020) from which another variant of PR scores can be used for evaluation, or \"A Domain Agnostic Measure for Monitoring and\nEvaluating GANs\" (NeurIPS'2019). \n\n6. Out of big diversity of GAN models, only the training of BigGan is tested with the proposed approximation objective. \n\n7. In several places throughout the paper, the likelihood ratio $p(x)/\\hat{p}(x)$ is used, e.g. in Theorem 5.2. However it is known that,  especially in high-dimensional spaces, the support of distributions is rather small and there can be large regions with  $\\hat{p}(x)=0$ on which the methods involving the ratios as above give infinite or not well-defined answers. How to deal with this in each of the  encounters of this quantity ?\n\n\n8. The clarity of the presentation can be improved, in particular the relation with other, more intuitive definitions of Precision and Recall scores from e.g. [27] and the \"Reliable Fidelity and Diversity Metrics for Generative Models\" (ICML'2020) paper,  is not explained. \n\n9. The complexity of the training is not described.  \n\n10. The error bars are absent in the main experiment reported in Table 2.\n\nQUESTIONS:\n1. What is the complexity of the training procedure?\n\n2. In several places throughout the paper the likelihood ratio $p(x)/\\hat{p}(x)$ is used. How to deal with regions with $\\hat{p}(x)=0$  in each of the  encounters of this ratio ?\n\n3. In the experiments reported in Table 2, was only a single initialization used in each case to test the proposed method?\n\nLIMITATIONS:\nThe limitations related with the complexity of the training procedure are not provided.  Also the limitations concerning the convergence of the proposed minimax procedure based on two different f are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to balance the precision and recall of generative models including GANs and normalizing flows. Theoretically, the paper shows that the f-divergences to minimize can be reformulated as sums of PR-divergences, which provides a theoretical explanation between the optimization objective and the precision-recall trade-off. Empirically, the paper proposes a series of techniques to optimize the precision-recall divergence.\n\nSTRENGTHS:\nThe paper is qualified to be accepted to NeurIPS in the following aspects:\n\n1.\tNovelty: Unlike previous heuristic works on the precision-recall trade-off, this paper established a fundamental theoretical understanding of the PR trade-off. To the best of our knowledge, the induced method is the first work to achieve a good PR trade-off from the perspective of divergence training.\n\n2.\tSignificance: The proposed theoretical results can not only serve as a guide in training generative models but also as an extension of studies on the traditional PR trade-off, thus might be applicable to other tasks like retrieval and recommendation systems.\n\n3.\tClarity: The paper is overall clear and well-written even with the heavy mathematics.\n\n4.\tSoundness: No obvious errors were found in the proofs. The theories are consistent with the empirical results (Fig. 4).\n\nWEAKNESSES:\nThe main concern is the proposed optimization algorithm involves an additional hyperparameter $\\lambda$, which needs to be tuned for a new dataset. Therefore, it might be an alternative to optimize the Area Under the PR Curve (AUPRC) instead of a single point in the PR curve, which has been studied in ranking problems [1,2,3,4]. The paper could be more instructive if the theoretical results can inspire the extension along this path.\n\nRef:\n\n[1] Wang et. al. Momentum accelerates the convergence of stochastic auprc maximization. ICML, 2022.\n\n[2] Wen et. al. Exploring the algorithm-dependent generalization of auprc optimization with list stability. NeurIPS, 2022.\n\n[3] Cakir et. al. Deep metric learning to rank. CVPR, 2019.\n\n[4] Chen et. al. Ap-loss for accurate one-stage object detection. T-PAMI, 2020.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nLimitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the fine-grind optimization regarding the generation precision and recall. More specifically, considering the ambiguity of FID, the authors propose to directly optimize the precision and recall of generated images, by developing a certain class of f-divergence, that is, the PR-divergence. This paper also shows that a linear combination of the proposed PR-divergence can represent arbitrary existing f-divergence. This paper also proposes a practical primal-dual estimation on the PR-divergence, for relieving the gradient vanishing issues during training. The experimental results verify the feasibility of adjusting lambda to adjust the generation precision and recall.\n\nSTRENGTHS:\nThis paper proposes a unique class of f-divergence named PR-divergence (up to an affine transform) that can explicitly optimize the precision and recall of GANs. The established theory of the PR-divergence is well aligned with the precision and recall of GANs, namely, the PR-curve. The author prove its connection to the vanilla f-divergence. The primal estimation of the divergence has also been provided for the ease of optimization, which is proved to be equivalent to minimizing the dual object in terms of the Bregman divergence. The presentation of this paper is clear for me. Experimental results have verified the effectiveness of optimizing different lambda-valued PR-divergence can alter the precision and recall.\n\nWEAKNESSES:\nOne of my main concerns is regarding whether the PR-divergence can impede the generation quality, although it is proved to be able to alter the precision and recall during training. I was wondering, whether optimizing the PR-divergence for some lambda could at least retain the same best FIDs with other existing methods, under the same architecture and batch sizes. For example, as I read from the details in the supplementary material, the authors employed larger batch size (equal to 128) than the default setting (equal to 64) when training CIFAR-10 under the BigGAN architecture. However, the default setting of 64bs already achieved FIDs lower than 10, whereby the results are reported by https://github.com/POSTECH-CVLab/PyTorch-StudioGAN. All the FIDs for CIFAR-10 reported in this paper are larger than 10, even trained by 128bs. Why would this happen? Will the PR-divergence deteriorate the overall generation quality?\n\nIf I understood correctly, the proposed PR-divergence can be well applied to divergence-based or likelihood-based generative models. So I was wondering whether this method can be applied to diffusion models? Also for the normalization flows, did the authors compare the NFs on real-world datasets, other than on the synthetic datasets?\n\nQUESTIONS:\n1. The notation \\leq or \\geq when comparing two distributions should also be explained. For example, the notation \\geq appears in  P\\geq\\beta\\mu in Definition 3.\n\n2. It seems Theorem 4.4 is not straightforward, where the proof or brief explanation is necessary. \n\n3. The authors propose the g-divergence for the primal estimation. How the g-divergence is chosen in practice?\n\nLIMITATIONS:\nPlease see my weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training method for generative models (normalizing flows and GANs), which can control the precision-recall trade-off of the generative models. The method is to design a new divergence named precision-recall divergence to bridge the precision-recall curve and the f-divergence. Then, the generative models optimized using the precision-recall divergence can control the precision-recall tradeoff by adjusting the hyperparameter lambda. The experiments show that the proposed method can control the precision-recall tradeoff and improve the performance of the baseline model (i.e., BigGAN).\n\nSTRENGTHS:\n1. This paper proposes a divergence that is directly related to the precision-recall curve, and the proposed method can control the precision-recall tradeoff during training. \n2. The results show that the proposed method can control the precision-recall tradeoff.\n3. The results show that the proposed method can improve the performance of BigGAN.\n\nWEAKNESSES:\n1. I think some combination of different kinds of f-divergence can also control the precision-recall tradeoff during the training. For example, KL-divergence + lambda * reverse KL-divergence. The paper does not compare with this kind of method, and does not explain the advantages of the proposed method in practice.\n2. Although the existing methods including truncation and rejection sampling can not control the precision-recall during training, they can control the tradeoff through post-processing. The authors do not compare with these methods. What are the practical advantages of the proposed method compared to these methods?\n3. The proposed method is not evaluated on larger resolutions (e.g., 512*512) and larger datasets. The ImageNet dataset is just used in the fine-tuning setting. \n4. The proposed method is only evaluated on the BigGAN backbone. I am more interested in the use of StyleGAN, since StyleGAN is more widely used. \n5. In Table 2, StyleGAN-XL performs much better than the proposed method.\n\nQUESTIONS:\nIn addition to the theoretical one, what’s the practical advantages of the proposed method comparing to the existing methods such as truncation, rejection sampling, and combination of different f-divergences (e.g., KL + reverse KL)?\n\nLIMITATIONS:\nThe authors have addressed the limitation in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}