SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alexandre Verine
LAMSADE, CNRS,
Université Paris-Dauphine-PSL,
alexandre. verine@dauphine.psl.eu
Benjamin Negrevergne
benjamin.negrevergne@dauphine.psl.eu
Muni Sreenivas Pydi
muni.pydi@dauphine.psl.eu
Yann Chevaleyre
yann.chevaleyre@dauphine.psl.eu
Achieving a balance between image quality (precision) and diversity (recall) is a significant challenge in the domain of generative models. Current state-of-the-art models primarily rely on optimizing heuristics, such as the Fréchet Inception Distance. While recent developments have introduced principled methods for evaluating precision and recall, they have yet to be successfully integrated into the training of generative models. Our main contribution is a novel training method for generative models, such as Generative Adversarial Networks and Normalizing Flows, which explicitly optimizes a user-defined trade-off between precision and recall. More precisely, we show that achieving a specified precision-recall trade-off corresponds to minimizing a unique  $f$ -divergence from a family we call the PR-divergences. Conversely, any  $f$ -divergence can be written as a linear combination of PR-divergences and corresponds to a weighted precision-recall trade-off. Through comprehensive evaluations, we show that our approach improves the performance of existing state-of-the-art models like BigGAN in terms of either precision or recall when tested on datasets such as ImageNet.
Evaluation of generative models has always been a challenging task. The metric used must reflect both the quality of the sample generated (precision) and how much the sample covers the targeted probability distribution (recall). Inception Score (IS) or Fréchet Inception Distance (FID) have been introduced and are now widely used by the community to select the best models. Typically, these metrics are favored because they "correlate well with the visual fidelity of the samples" and are "sensitive to both the addition of spurious modes as well as mode dropping" [44]. However as pointed by Kynkänniemi et al. [28], FID and IS group these two aspects into a single value without a clear trade-off. Depending on the use-case, generative models might require a good precision (high-resolution image and video generation, artistic synthesis, 3D model design) or a good recall (data augmentation, drug discovery, anomaly detection). For that reason, a number of more principled methods [9, 15, 28, 44, 47] have emerged to assess precision and recall (hereafter abbreviated by
P&R) independently, however these methods cannot be optimized during training, because they are not differentiable or because they are too computationally demanding.
To enhance either the precision or the recall of a particular model, an array of strategies and techniques have been employed. Usually these techniques involve altering the latent distribution  $a$  posteriori (e.g. truncation and temperature post-training). Other methods exist such as through rejection sampling, boosting, or instance selection [3, 12, 13, 19, 23, 33, 49]. Although these approaches may utilize proxies of P&R, they are not theoretically grounded in the principled methods of evaluating these metrics, leaving their alignment with the foundational concepts of P&R unverified. This leads to the following question.
Question 1: Is it possible to train a generative model that achieves a specified trade-off between precision and recall?
In addition to this question, another of our goals is to understand existing generative models in terms of P&R. Modern generative models, such as Generative Adversarial Networks (GANs) or Normalizing Flows (NFs), are typically designed to minimize specific divergence measures. Given a target distribution  $P$  and a set of parameterized distributions  $\{P_{\theta}|\theta \in \Theta \}$ , a generative model aims to find the best fit  $\hat{P} = P_{\theta^*}$  that minimizes a divergence between  $P$  and  $\hat{P}$ . These divergences induce different behaviors at convergence. For instance, optimizing the Kullback-Leibler (KL) divergence tends to favor mass-covering models [35], contrasting with the mode-seeking behavior observed with other generative models, leading to the infamous problem of mode collapse. Transitioning from one divergence to another does indeed alter the model's behavior in terms of precision and recall, however, the implicit trade-offs made during the optimization of a general divergence remain somewhat ambiguous. This motivates the following question.
Question 2: What precision-recall trade-off does an arbitrary  $f$ -divergence minimize?
In this paper, we bridge the gap between principled methods of P&R evaluation and controlling their trade-off. In doing so, we address Questions 1 and 2 by making the following contributions:
- We show that achieving a specified precision-recall trade-off corresponds to minimizing a particular  $f$ -divergence between  $P$  and  $\widehat{P}$ . Specifically, in Theorem 4.3 we give a family of  $f$ -divergences (denoted by  $\mathcal{D}_{\lambda_{\mathrm{PR}}}$ ,  $\lambda \in [0,\infty]$ ) that are associated with various points along the precision-recall curve of the generative model.
- We show that any arbitrary  $f$ -divergence can be written as a linear combination of  $f$ -divergences from the  $\mathcal{D}_{\lambda_{\mathrm{PR}}}$  family. This result makes explicit, the implicit precision-recall trade-offs made by generative models that minimize an arbitrary  $f$ -divergence.
- We propose a novel approach to train or fine-tune a generative model on notoriously hard to train  $f$ -divergences, with guarantees on divergences defined by the lipschitz constant of  $f$ .
- We use this approach to train models on a user specified trade-off between P&R by minimizing  $\mathcal{D}_{\lambda - \mathrm{PR}}$  for any given  $\lambda$ . We specifically focus on GANs and NFs. For instance, Figure 1 shows how our model performs under various settings of  $\lambda$ . With a high  $\lambda$ , we can train the model to favor precision over recall and vice-versa.
- Through extensive experiments, we show that our approach enables effective model training to minimize the PR-Divergence, particularly for fine-tuning pre-trained models, with a notable impact from the choice of trade-off parameters  $\lambda$ , while also demonstrating scalability with larger dimensions and datasets.
Generative model evaluation metrics: IS or FID are widely adopted due to their ability to assess visual fidelity and sensitivity to mode variations. However, these metrics fall short in providing a trade-off between P&R. Principled approaches were introduced by Djolonga et al. [15] and Sajjadi et al. [44], and later extended by Simon et al. [47], providing a definition P&R independently using PR-Curves detailed in Section 3.3. We adopt this extended approach in our work. In image generative modeling, the current consensus for P&R evaluation are two methods. First, the method presented by Kynkäänniemi et al. [28], which provides a simpler evaluation of P&R based on the estimation
(a)  $\lambda = 0.1$
(b)  $\lambda = 1$
(c)  $\lambda = 10$
Figure 1: NFs - RealNVP [14] trained on 2D Gaussians. Fig. 1(a) to Fig. 1(c): models trained to minimize  $\mathcal{D}_{\lambda_{\mathrm{PR}}}$ . Fig. 1(d) and Fig. 1(e): models trained to minimize  $\mathcal{D}_{\mathrm{KL}}$  and  $\mathcal{D}_{\mathrm{rKL}}$  respectively. Samples drawn from the true distribution  $P$  are shown in black, samples drawn from the estimated distribution  $\widehat{P}$  are shown in green and the log-likelihood of  $\widehat{P}$  is shown in blue (darker means higher density). (a)  $\lambda = 0.1$ , favors recall over precision. (b)  $\lambda = 1$ , balanced precision vs. recall trade-off. (c)  $\lambda = 10$ , which favors precision over recall. As observed in [35], and demonstrated in Theorem 4.5, the  $\mathcal{D}_{\mathrm{KL}}$  (1(d)) is mass covering and the  $\mathcal{D}_{\mathrm{rKL}}$  (1(e)) in mode-seeking. The corresponding PR-Curve are presented in Figure 2(c).
(d)  $\mathcal{D}_{\mathrm{KL}}$
(e)  $\mathcal{D}_{\mathrm{rKL}}$
of the support of the distribution using a k-NN algorithm, akin to the most recent work by Cheema and Urner [9]. Second, a method introduced by [36], computing the Density (D) and the Coverage (C) which account for local density and are robust to outliers in contrast to [28]. In natural language processing, one popular method, MAUVE [41], consists of the area under PR-Curves defined by [15]. However, because of their mathematical foundation, these methods remain unsuitable for training due to non-differentiability and high computational requirements.
Trade-offs and techniques in generative model: The challenge of balancing P&R in generative model training has led to a variety of techniques. Methods for enhancing precision often involve manipulating the latent distribution post-training, such as hard truncation [7, 24, 25, 46] or standard deviation adjustment, also called soft truncation [27]. One can also use rejection sampling in the image space [4, 50], in the latent space [23, 49], or in the dataset [13]. There are also numerous strategies to prevent mode collapse and boost recall [8, 31, 32], gradient-boosting methods [12, 16, 19] and mixture based latent models [6, 40]. However, it should be noted that these approaches lack a foundation in the principled methods of P&R evaluation. This underscores the need for a theoretically grounded approach to balance these crucial aspects in generative models, a gap that our current work aims to address. Note also that all these methods can be applied to any model, and in particular to the one trained using our proposed method.
Divergence training in generative models: Modern generative models, such as GANs and NFs, often employ specific divergence measures for model optimization. With works such as Grover et al. [20], Nowozin et al. [39], models can be trained with a variety of divergences such as  $f$ -divergence, thus observing different results of P&R. Another notable work by Midgley et al. [34] proposed training with  $\alpha$ -Divergence, under the assumption of access to data density. The work of [29], closely aligned with our approach, defines different orders of mode seeking and evaluates the corresponding  $f$ -divergences at these levels. Although its methodology employs the training of  $f$ -GAN models on simple datasets to illustrate the mode-seeking property, it does not offer the flexibility to establish a user-defined trade-off. The implicit trade-offs made by these divergence measures is still a challenge: the recent work of [48] shows the links between P&R and any DeGroot's divergences.
Notation: Throughout the paper, we use  $\mathcal{X} \subset \mathbb{R}^d$  to refer to the input space and  $\mathcal{Z} \subset \mathbb{R}^m$  to the latent space of the model we consider. We also denote  $\mathcal{P}(\mathcal{X})$  and  $\mathcal{P}(\mathcal{Z})$  the set of all probability measures over measurable subsets of  $\mathcal{X}$  and  $\mathcal{Z}$  respectively.  $P$  and  $\widehat{P}$  are consistently used to denote the target and estimated distributions (both members of  $\mathcal{P}(\mathcal{X})$ ). We also assume that  $P$  and  $\widehat{P}$  share the same support in  $\mathcal{X}$ , and that they admit densities denoted by the corresponding lower case letters  $p$  and  $\widehat{p}$ , respectively. Finally, for any function  $f \in \mathbb{R} \to \mathbb{R}$ , we define the convex conjugate (or Fenchel transform) of  $f$  given by  $f^*(t) = \sup_{u \in \mathbb{R}} \{tu - f(u)\}$ .
3.1  $f$ -divergences
Given a convex lower semi-continuous (l.s.c) function  $f: \mathbb{R}^+ \to \mathbb{R}$  satisfying  $f(1) = 0$ , the  $f$ -divergence between two probability distributions  $P$  and  $\hat{P}$  is defined as follows.
$$
\mathcal {D} _ {f} (P \| \widehat {P}) = \int_ {\chi} \widehat {p} (\boldsymbol {x}) f \left(\frac {p (\boldsymbol {x})}{\widehat {p} (\boldsymbol {x})}\right) d \boldsymbol {x}. \tag {1}
$$
$\mathcal{D}_f$  is invariant to an affine transformation in  $f$  i.e.,  $\mathcal{D}_f(P\| \widehat{P}) = D_{f^\dagger}(P\| \widehat{P})$  for  $f^{\dagger}(u) = f(u) + c(u - 1)$  for any constant  $c\in \mathbb{R}$ . Many well-known statistical divergences such as the KL divergence  $(\mathcal{D}_{\mathrm{KL}})$ , the reverse KL  $(\mathcal{D}_{\mathrm{rKL}})$  or the Total Variation  $(\mathcal{D}_{\mathrm{TV}})$  are  $f$ -divergences (see Table 1). Importantly, any  $\mathcal{D}_f$  admits a dual variational form [37], with  $\mathcal{T}$  denoting the set of all measurable functions  $\mathcal{X}\to \mathbb{R}$ :
$$
\mathcal {D} _ {f} (P \| \widehat {P}) = \sup  _ {T \in \mathcal {T}} \mathcal {D} _ {f, T} ^ {\text {d u a l}} (P \| \widehat {P}), \quad \text {w h e r e} \quad \mathcal {D} _ {f, T} ^ {\text {d u a l}} (P \| \widehat {P}) = \mathbb {E} _ {\boldsymbol {x} \sim P} [ T (\boldsymbol {x}) ] - \mathbb {E} _ {\boldsymbol {x} \sim \widehat {P}} [ f ^ {*} (T (\boldsymbol {x})) ] \tag {2}
$$
We use  $T^{\mathrm{opt}} \in \mathcal{T}$  to denote the function that achieves the supremum in (15).
Table 1: List of common  $f$ -divergences. The generator  $f$  is given with its Fenchel conjugate  $f^*$ . The optimal discriminator  $T^{\mathrm{opt}}$  is given to compute the likelihood ratio  $p(\boldsymbol{x}) / \widehat{p}(\boldsymbol{x}) = \nabla f^*(T^{\mathrm{opt}}(\boldsymbol{x}))$ . Then  $f''(1/\lambda)/\lambda^3$  is given to compute the  $\mathcal{D}_f$  as a combination of  $\mathcal{D}_{\lambda_{\mathrm{PR}}}$  using Theorem 4.4.
DIVERGENCE	NOTATION	f(u)	f*(t)	T^opt(x)	f''(1/λ)/λ^3
KL	D_KL	u log u	exp(t-1)	1 + log p(x)/p(x)	1/λ^2
REVERSE KL	D_rKL	- log u	-1 - log -t	- p(x)p(x)	1/λ
χ^2-PEARSON	Dχ^2	(u-1)^2	t^2/4 + t	2 (p(x)p(x) - 1)	2/λ^3
3.2 Generative models
Generative Adversarial Networks (GANs): A GAN consists of two functions, generator  $G: \mathcal{Z} \to \mathcal{X}$ , discriminator  $T: \mathcal{X} \to \mathbb{R}$ , as well as a prior distribution  $Q \in \mathcal{P}(\mathcal{Z})$  which is usually the standard normal  $\mathcal{N}(0, \mathcal{I}_m)$ . The estimated data distribution  $\widehat{P}_G$  is the push-forward distribution of  $Q$  by  $G$ . In the original work of Goodfellow et al. [17], a specific  $\mathcal{D}_f$  is used to optimize the divergence between  $P$  and  $\widehat{P}_G$ . In this paper, we consider the more general framework of Nowozin et al. [39] that can be used to train a GAN with any  $\mathcal{D}_f$  by solving the following min-max optimization problem:
$$
\min  _ {G} \max  _ {T} \mathcal {D} _ {f, T} ^ {\text {d u a l}} (P \| \widehat {P} _ {G}) = \min  _ {G} \max  _ {T} \mathbb {E} _ {\boldsymbol {x} \sim P} [ T (\boldsymbol {x}) ] - \mathbb {E} _ {\boldsymbol {x} \sim \widehat {P} _ {G}} [ f ^ {*} (T (\boldsymbol {x})) ], \tag {3}
$$
Normalizing Flows (NFs): An NF consists of an invertible function  $G: \mathcal{Z} \to \mathcal{X}$  and a prior distribution  $Q \in \mathcal{P}(\mathcal{Z})$ . As for GANs, the estimated distribution  $\widehat{P}_G$  is the push-forward of  $Q$  by  $G$ . However, because  $G$  is invertible, it is possible to compute the density  $\hat{p}(\cdot)$  using a simple change of variable formula,  $\hat{p}(\boldsymbol{x}) = q(G^{-1}(\boldsymbol{x})) |\det \operatorname{Jac}_{G^{-1}}(\boldsymbol{x})|$ , where  $\det \operatorname{Jac}_{G^{-1}}(\boldsymbol{x})$  is the determinant of the Jacobian matrix of  $G^{-1}$  at  $\boldsymbol{x}$ , and train the generator using the  $\mathcal{D}_{\mathrm{KL}}$  between  $P$  and  $\widehat{P}_F$ , or equivalently, by maximizing the log-likelihood:
$$
\min  _ {G} \mathcal {D} _ {\mathrm {K L}} (P \| \widehat {P} _ {G}) = H (P) - \max  _ {G} \mathbb {E} _ {\boldsymbol {x} \sim P} [ \log \hat {p} _ {G} (\boldsymbol {x}) ], \tag {4}
$$
where  $H(P)$  is the continuous entropy of  $P$ . In practice, the generator function  $G$  is typically represented by neural networks such as GLOW [27], RealNVP [42], or ResFlow [5, 10] for which  $\det \operatorname{Jac}_{G^{-1}}(\pmb{x})$  is easy to compute. Grover et al. [20] showed that it is possible to train an NF using most  $f$ -divergences. In practice, the log-likelihood is added to the min-max objective to stabilize learning. This gives us the following optimization problem:
$$
\min  _ {G} \max  _ {T} \mathcal {D} _ {f, T} ^ {\text {d u a l}} (P \| \widehat {P} _ {G}) - \gamma \mathbb {E} _ {\boldsymbol {x} \sim P} [ \log \hat {p} _ {G} (\boldsymbol {x}) ], \tag {5}
$$
(a) Distributions
(b)  $\partial \mathrm{PRD}(P,\widehat{P})$  curves
(c)  $\partial \mathrm{PRD}(P,\widehat{P})$  for Fig. 1
Figure 2: PR curves for two models  $\widehat{P}_1$  and  $\widehat{P}_2$  of  $P$ . Figure 2(a) shows  $\widehat{P}_1$ ,  $\widehat{P}_2$  and  $P$ . Figure 2(b) shows PR curves for  $\widehat{P}_1$ ,  $\widehat{P}_2$  against  $P$ .  $\widehat{P}_1$  has good recall since it covers both modes of  $P$ , but low precision since it generates points between the modes.  $\widehat{P}_2$  has good precision since it does not generate samples outside of  $P$  but low recall since it can generate samples from only one mode. Figure 2(c) are the PR curves corresponding to Figure 1 and are detailed in Section 6.
3.3 Precision-Recall curve for generative models
Generative models are usually evaluated using a single criterion such as the IS [45] or the FID [21]. However, these criteria are unable to distinguish between the distinct failure modes of low precision (i.e., failure to produce quality samples) and low recall (i.e., failure to cover all modes of  $P$ ). The following definition was introduced by Sajjadi et al. [44] and later extended by Simon et al. [47].
Definition 3.1 (PRD set, adapted from Simon et al. [47]). For  $P, \widehat{P} \in \mathcal{P}(\mathcal{X})$ , the P&R set PRD(P,  $\widehat{P}$ ) is defined as the set of pairs of precision  $\alpha$  and recall  $\beta$  in  $\mathbb{R}^{+} \times \mathbb{R}^{+}$  such that there exist  $\mu \in \mathcal{P}(\mathcal{X})$  for which  $P \geq \beta \mu$  and  $\widehat{P} \geq \alpha \mu$ . The precision-recall curve (or PR curve) is defined as  $\partial \mathrm{PRD}(P, \widehat{P}) = \{ (\alpha, \beta) \in \mathrm{PRD}(P, \widehat{P}) \mid \not\exists (\alpha', \beta') \text{ with } \alpha' \geq \alpha \text{ and } \beta' \geq \beta \}$ .
An equivalent definition of  $\mathrm{PRD}(P,\widehat{P})$  is found in Sajjadi et al. [44], where  $(\alpha ,\beta)\in \mathrm{PRD}(P,\widehat{P})$  if  $P$  and  $\widehat{P}$  can be decomposed as in (6) for some common component  $\mu \in \mathcal{P}(\mathcal{X})$  and complementary components  $\nu_{P},\nu_{\widehat{P}}\in \mathcal{P}(\mathcal{X})$
$$
P = \alpha \mu + (1 - \alpha) \nu_ {P} \quad \text {a n d} \quad \widehat {P} = \beta \mu + (1 - \beta) \nu_ {\widehat {P}}. \tag {6}
$$
Simon et al. [47] show that the PR curve is parameterized by  $\lambda \in [0, +\infty]$  as  $\partial \mathrm{PRD}(P, \widehat{P}) = \left\{\alpha_{\lambda}(P \| \widehat{P}), \beta_{\lambda}(P \| \widehat{P}) \mid \lambda \in [0, +\infty]\right\}$ , with  $\alpha_{\lambda}(P \| \widehat{P}) = \int_{\mathcal{X}} \min(\lambda p(\boldsymbol{x}), \widehat{p}(\boldsymbol{x})) \, \mathrm{d}\boldsymbol{x}$  and  $\beta_{\lambda}(P \| \widehat{P}) = \alpha_{\lambda}(P \| \widehat{P}) / \lambda$ . Here,  $\lambda$  is called the trade-off parameter and can be used to adjust the sensitivity to precision or recall.
An illustration of the PR curve is given in Figure 2 for a target distribution  $P$  that is a mixture of two Gaussians and two candidate models  $\widehat{P}_1$  and  $\widehat{P}_2$ . We can see on Figure 2 that  $\widehat{P}_1$  offers better results for large values of  $\lambda$  (with high sensitivity to precision) whereas  $\widehat{P}_2$  offers better results for low values of  $\lambda$  (with high sensitivity to recall).
4 Precision and Recall trade-off as an  $f$ -divergence
In this section, we formalize the link between P&R trade-off and  $f$ -divergences, and address Question 2. We will exploit this link in Section 5 to train models that optimize a particular P&R trade-off.
4.1 Precision-Recall as an  $f$ -divergence
We start by introducing the PR-Divergence as follows.
Definition 4.1 (PR-divergence). Given a trade-off parameter  $\lambda \in [0, +\infty]$ , the PR-divergence (denoted by  $\mathcal{D}_{\lambda - \mathrm{PR}}$ ) is defined as  $D_{f_{\lambda}}$  for  $f_{\lambda} : \mathbb{R}^{+} \to \mathbb{R}$  given by  $f_{\lambda}(u) = \max(\lambda u, 1) - \max(\lambda, 1)$  for  $\lambda \in \mathbb{R}^{+}$  and  $f_{\lambda}(u) = 0$  for  $\lambda = +\infty$ .
Note that  $f_{\lambda}$  is continuous, convex, and satisfies  $f_{\lambda}(1) = 0$  for all  $\lambda$ . A graphical representation of  $f_{\lambda}$  can be found in Appendix B.2. The following proposition gives some properties of  $\mathcal{D}_{\lambda -\mathrm{PR}}$ .
Proposition 4.2 (Properties of the PR-Divergence).
- The Fenchel conjugate  $f_{\lambda}^{*}$  of  $f_{\lambda}$  is defined on  $\operatorname{dom}(f_{\lambda}^{*}) = [0, \lambda]$  and given by,  $f_{\lambda}^{*}(t) = t / \lambda$  for  $\lambda \leq 1$  and  $f_{\lambda}^{*}(t) = t / \lambda + \lambda - 1$  otherwise.
- The optimal discriminator for the dual form is  $T^{\mathrm{opt}}(\pmb{x}) = \lambda \mathrm{sign}\{p(\pmb{x}) / \widehat{p}(\pmb{x}) - 1\}$ .
$\mathcal{D}_{\lambda \text{-PR}}(\widehat{P}\| P) = \lambda \mathcal{D}_{\frac{1}{\lambda}\text{-PR}}(P\| \widehat{P}).$
Observe that  $\mathcal{D}_{\lambda -\mathrm{PR}}(\widehat{P}\| P)\neq \mathcal{D}_{\lambda -\mathrm{PR}}(P\| \widehat{P})$  in general, but for  $\lambda = 1$ ,  $\mathcal{D}_{1 - \mathrm{PR}}(P\| \widehat{P}) = \mathcal{D}_{1 - \mathrm{PR}}(\widehat{P}\| P) = \mathcal{D}_{\mathrm{TV}}(P\| \widehat{P}) / 2$ . Having defined the PR-divergence, we can now show that P&R w.r.t  $\lambda$  can be expressed as a function of the divergence between  $P$  and  $\widehat{P}$ .
Theorem 4.3 (P&R as a function of  $\mathcal{D}_{\lambda_{-}\mathrm{PR}}$ ). Given  $P, \widehat{P} \in \mathcal{P}(\mathcal{X})$  and  $\lambda \in [0, +\infty]$ , the PR curve  $\partial \mathrm{PRD}(P, \widehat{P})$  is related to the PR-divergence  $\mathcal{D}_{\lambda_{-}\mathrm{PR}}(P \| \widehat{P})$  as follows.
$$
\alpha_ {\lambda} (P \| \widehat {P}) = \min  (1, \lambda) - \mathcal {D} _ {\lambda - \mathrm {P R}} (P \| \widehat {P}). \tag {7}
$$
Conversely, suppose that there exists a strictly decreasing linear function  $h:[0,1]\to \mathbb{R}^+$  and an  $f$ -divergence  $\mathcal{D}_f$  such that  $h(\alpha_{\lambda}(P\| \widehat{P})) = \mathcal{D}_f(P\| \widehat{P})$  for all  $P,\widehat{P}\in \mathcal{P}(\mathcal{X})$ , then  $f(u) = c_1f_\lambda (u) + c_2(u - 1)$ .
A direct consequence of Theorem 4.3 is that minimizing  $\mathcal{D}_{\lambda -\mathrm{PR}}$  is equivalent to maximizing  $\alpha_{\lambda}$ . In a more explicit way, Theorem 4.3 suggests that  $\mathcal{D}_{\lambda -\mathrm{PR}}$  is the only  $f$ -divergence (up to an affine transformation) for which this property holds. This makes  $\mathcal{D}_{\lambda -\mathrm{PR}}$  a uniquely suitable candidate for training a generative model with a specific P&R trade-off. The proof of Theorem 4.3 is in Appendix B.1
4.2 Relation between PR-divergences and other  $f$ -divergences
In the previous subsection, we showed that for each trade-off parameter  $\lambda$ , there exists a  $\mathcal{D}_{\lambda - \mathrm{PR}}$  that corresponds to optimizing for it. This raises the converse question of what trade-off is achieved by optimizing for an arbitrary  $f$ -divergence. We answer this by showing in the following theorem that any  $f$ -divergence can be expressed as a weighted sum of PR-divergences.
Theorem 4.4 (f-divergence as weighted sums of PR-divergences). For any  $P$ ,  $\widehat{P} \in \mathcal{P}(\mathcal{X})$  supported on all of  $\mathcal{X}$  and any  $\lambda \in \mathbb{R}^+$ , with  $m = \min_{\mathcal{X}} (\widehat{p}(\mathbf{x}) / p(\mathbf{x}))$  and  $M = \max_{\mathcal{X}} (\widehat{p}(\mathbf{x}) / p(\mathbf{x}))$ :
$$
\mathcal {D} _ {f} (P \| \widehat {P}) = \int_ {m} ^ {M} \frac {1}{\lambda^ {3}} f ^ {\prime \prime} \left(\frac {1}{\lambda}\right) \mathcal {D} _ {\lambda - \mathrm {P R}} (P \| \widehat {P}) \mathrm {d} \lambda , \tag {8}
$$