SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As a sanity check, observe that the weights  $f''(1 / \lambda) / \lambda^3$  remain invariant under an affine transformation in  $f$  much like  $D_f$ .
Corollary 4.5 ( $\mathcal{D}_{\mathrm{KL}}$  and  $\mathcal{D}_{\mathrm{rKL}}$  as an average of  $\mathcal{D}_{\lambda - \mathrm{PR}}$ ). The  $\mathcal{D}_{\mathrm{KL}}$  Divergence and the  $\mathcal{D}_{\mathrm{rKL}}$  can be written as a weighted average of PR-Divergence  $\mathcal{D}_{\lambda - \mathrm{PR}}$ :
$$
\mathcal {D} _ {\mathrm {K L}} (P \| \widehat {P}) = \int_ {m} ^ {M} \frac {1}{\lambda^ {2}} \mathcal {D} _ {\lambda - \mathrm {P R}} (P \| \widehat {P}) \mathrm {d} \lambda , \quad a n d \quad \mathcal {D} _ {\mathrm {r K L}} (P \| \widehat {P}) = \int_ {m} ^ {M} \frac {1}{\lambda} \mathcal {D} _ {\lambda - \mathrm {P R}} (P \| \widehat {P}) \mathrm {d} \lambda . \tag {9}
$$
As we can see in this Corollary, both  $\mathcal{D}_{\mathrm{KL}}$  and  $\mathcal{D}_{\mathrm{rKL}}$  can be decomposed into a sum of PR-divergences terms  $\mathcal{D}_{\lambda_{-\mathrm{PR}}}$ , each weighted with  $1 / \lambda^2$  and  $1 / \lambda$  respectively. Note that if  $\lambda < 1$  then  $1 / \lambda > 1 / \lambda^2$ , and conversely, if  $\lambda > 1$ .  $\mathcal{D}_{\mathrm{KL}}$  is thus associating more weights on recall than  $\mathcal{D}_{\mathrm{rKL}}$ . This explains the mass covering behavior observed in NFs trained with  $\mathcal{D}_{\mathrm{KL}}$ . Comparatively, the  $\mathcal{D}_{\mathrm{rKL}}$  assigns more weight to terms with a large lambda, leading to the mode seeking behavior empirically observed with flows trained with the  $\mathcal{D}_{\mathrm{rKL}}$  [33]. However, as it will be observed in the Section 6,  $\mathcal{D}_{\mathrm{rKL}}$  is still favoring low values of  $\lambda$ . Other weights for other  $f$ -divergences are in Appendix A.1.
An equivalent result can be found in Corollary 19 of [48]
5 Minimizing the Precision-Recall divergence
In this section, we address Question 1 by introducing a new method that can be used to optimize a model with a specific precision-recall trade-off  $\lambda$ . A first naive strategy to achieve this is to use the  $f$ -GAN framework introduced by Nowozin et al. [39], and minimize the dual variational form of  $\mathcal{D}_{\lambda_{-}\mathrm{PR}}$  presented in Theorem 4.3. Together, this results in solving the min-max problem:
$$
\min  _ {G} \max  _ {T} \mathcal {D} _ {f _ {\lambda}, T} ^ {\text {d u a l}} (P \| \widehat {P}) = \min  _ {G} \max  _ {T} \mathbb {E} _ {\boldsymbol {x} \sim P} [ T (\boldsymbol {x}) ] - \mathbb {E} _ {\boldsymbol {x} \sim \widehat {P}} [ f _ {\lambda} ^ {*} (T (\boldsymbol {x})) ], \tag {10}
$$
where  $G$  and  $T$  are both represented using neural networks. In practice, this strategy would fail because training a neural network with loss functions such as  $f_{\mathrm{TV}}^{*}$  (or in this case  $f_{\lambda}^{*}$ ) is notoriously difficult due to vanishing gradients² that lead to poor training performance [20, 39]. Instead, training neural networks on functions  $f^{*}$  such as  $f_{\mathrm{KL}}^{*}$  or  $f_{\chi^2}^{*}$ , results in much better empirical performance [29, 39, 51].
To avoid these issues, we show how to train the discriminator using an auxiliary divergence (based on a function  $g \neq f$ ) to better estimate the target  $f$ -divergence. The main idea is to choose an auxiliary  $g$ -divergence that is adequate for training  $T$  (e.g.  $\mathcal{D}_{\mathrm{KL}}$  or  $\mathcal{D}_{\chi^2}$ ) and use it to compute the likelihood ratio  $p(\pmb{x}) / \widehat{p}(\pmb{x})$ .
Because, at optimality, we have  $\nabla g^{*}(T^{\mathrm{opt}}(\pmb {x})) = p(\pmb {x}) / \widehat{p} (\pmb {x})$ , we can then compute the  $f$ -divergence as follows:
$$
\mathcal {D} _ {f} (P \| \widehat {P}) = \int \widehat {p} (\boldsymbol {x}) f \left(\frac {p (\boldsymbol {x})}{\widehat {p} (\boldsymbol {x})}\right) d \boldsymbol {x} = \int \widehat {p} (\boldsymbol {x}) f \left(\nabla g ^ {*} \left(T ^ {\text {o p t}} (\boldsymbol {x})\right)\right) d \boldsymbol {x}. \tag {11}
$$
In practice, we do not have access to  $T^{\mathrm{opt}}$ . Instead, we have a discriminator  $T$  trained to maximize  $\mathcal{D}_{g_iT}^{\mathrm{dual}}$ . At any time during training, we can estimate the  $f$ -divergence based on  $T$  using the primal estimate, which we define as follows.
Definition 5.1 (Primal estimate  $\mathcal{D}_{f,T}^{\mathrm{primal}}$ ). Let  $P, \widehat{P} \in \mathcal{P}(\mathcal{X})$ . For any function  $T: \mathcal{X} \to \mathbb{R}$ ,  $f: \mathbb{R}^{+} \to \mathbb{R}$ , and  $g: \mathbb{R}^{+} \to \mathbb{R}$ , we define the primal estimate  $\mathcal{D}_{f,T}^{\mathrm{primal}}$  as follows.
$$
\mathcal {D} _ {f, T} ^ {\text {p r i m a l}} (P \| \widehat {P}) = \int_ {\mathcal {X}} \widehat {p} (\boldsymbol {x}) f (r (\boldsymbol {x})) \mathrm {d} \boldsymbol {x}, \tag {12}
$$
where  $r:\mathcal{X}\to \mathbb{R}^+$  is given by,  $r(\pmb {x}) = \nabla g^{*}(T(\pmb {x}))$
The success of this approach depends on how well  $r$  approximates  $p(x) / \widehat{p}(x)$ , which depends on  $T$ . We first show that the approximation error of  $r$  measured in terms of the Bregman divergence [2] (also defined in Appendix A.2). It corresponds exactly to the approximation error of  $\mathcal{D}_{g,T}^{\mathrm{dual}}$ , so minimizing the latter will also minimize the former.
Theorem 5.2 (Error of the estimation of an  $f$ -divergence under the dual form.). For any discriminator  $T:\mathcal{X}\to \mathbb{R}$  and  $r(\pmb {x}) = \nabla f^{*}(T(\pmb {x}))$
$$
\mathcal {D} _ {g} (P \| \widehat {P}) - \mathcal {D} _ {g, T} ^ {\text {d u a l}} (P \| \widehat {P}) = \mathbb {E} _ {\widehat {P}} \left[ \operatorname {B r e g} _ {g} \left(r (\boldsymbol {x}), \frac {p (\boldsymbol {x})}{\widehat {p} (\boldsymbol {x})}\right) \right]. \tag {13}
$$
On the basis of this result, the quality of the approximation will crucially depend on  $\nabla g$ . We can show that if the auxiliary  $g$  is strongly convex, the error in the estimation of  $\mathcal{D}_{f,T}^{\mathrm{primal}}$  is bounded:
Theorem 5.3 (Bound on the estimation of an  $f$ -divergence using an auxiliary  $g$ -divergence). Let  $f, g: \mathbb{R}^{+} \to \mathbb{R}$  be such that  $g$  is  $\mu$ -strongly convex and  $f$  is  $\sigma$ -Lipschitz. For the discriminator  $T: \mathcal{X} \to \mathbb{R}$ , let  $r(\boldsymbol{x}) = \nabla g^{*}(T(\boldsymbol{x}))$ . Then
$$
\underline {{\mathcal {D} _ {g} (P \| \widehat {P}) - \mathcal {D} _ {g , T} ^ {\mathrm {d u a l}} \leq \epsilon}} \Longrightarrow \left| \mathcal {D} _ {f} (P \| \widehat {P}) - \mathcal {D} _ {f, T} ^ {\mathrm {p r i m a l}} (P \| \widehat {P}) \right| \leq \sigma \sqrt {\frac {2 \epsilon}{\mu}}.
$$
2The explanation for this vanishing gradient phenomenon primarily relies on the fact that the optimal discriminator for these functions is  $T^{\mathrm{opt}}(\pmb {x}) = \mathrm{sign}(\widehat{p} (\pmb {x}) / p(\pmb {x}) - 1)$ . To approximate it, [39] recommends the use of a tanh activation function that is known to induce vanishing gradients.
If  $T$  successfully maximizes  $\mathcal{D}_{g,T}^{\mathrm{dual}}$ , then the primal estimation converges to  $\mathcal{D}_f$ . To implement this approach, we propose the following simplified version of the algorithm: Repeat until convergence these 3 steps:
1. Let  $x_1^{\mathrm{real}} \ldots x_N^{\mathrm{real}} \sim P$  and  $x_1^{\mathrm{fake}} \ldots x_N^{\mathrm{fake}} \sim \widehat{P}_G$ .
2. Update the parameters of  $T$  by ascending the gradient
$$
\nabla \mathcal {L} _ {T} = \frac {1}{N} \nabla \left\{\sum_ {i = 1} ^ {N} T \left(x _ {i} ^ {\mathrm {r e a l}}\right) - \sum_ {i = 1} ^ {N} g ^ {*} \left(T (x _ {i} ^ {\mathrm {f a k e}})\right) \right\}.
$$
3. Update the parameters of  $G$  by descending the gradient
$$
\nabla \mathcal {L} _ {G} = \frac {1}{N} \nabla \left\{\sum_ {i = 1} ^ {N} f \left(\nabla g ^ {*} \left(T (x _ {i} ^ {\mathrm {f a k e}})\right)\right) \right\}.
$$
This method closely parallels the GAN training procedure, with the key distinction that T and G optimize objectives based on different  $f$ . In practice, our implementation uses a stochastic gradient descent, fully detailed in Algorithm 1 in Appendix C.
In this section, we employ the auxiliary loss approach outlined in Section 5 to train various models.
Specifically, we train NFs on 2D synthetic data, MNIST and FashionMNIST, while we train BigGAN on CIFAR-10, CelebA64, ImageNet128, and FFHQ256. All models are tested in terms of IS and FID with 50k samples, and on P&R with 10k samples using the method of Kynkänniemi et al. [28] with  $k = 3$  for MNIST and FashionMNIST and with  $k = 5$  for CelebA64, ImageNet128 and FFHQ256. Also, we test every model in terms of Density and Coverage [36] on 10k samples with  $k = 5$ . In this paper, we present a selection of experimental results. For a comprehensive set of results, including model parameters, optimizers, learning rates, and samples, please refer to Appendix D. We also included in this Appendix a set of experiments run using the naive approach based on Equation 10, thus showing that the discriminator fails to train as explained in Section 5. To ensure reproducibility, our models and code are available on the GitHub repository of the project<sup>3</sup>.
We show that: 1) the auxiliary loss approach effectively enables the training of a model to minimize the PR-Divergence, 2) this method is suitable for fine-tuning pre-trained models, 3) the choice of trade-off parameter  $\lambda$  significantly influences the results on P&R, and finally, 4) our method scales well with larger dimensions and datasets.
Normalizing Flows on synthetic data: NFs are typically trained to minimize  $\mathcal{D}_{\mathrm{KL}}$ , in addition to their structural limitation [11, 52], resulting in good recall but poor precision. Prior work has employed various techniques [27, 49] to improve model precision post-training, we use our method to directly train the model on a given trade-off. We demonstrate our approach by training RealNVP models on a 2D synthetic dataset using  $\mathcal{D}_{\lambda -\mathrm{PR}}$  with various  $\lambda$  values and using  $\mathcal{D}_{\mathrm{KL}}$  and  $\mathcal{D}_{\mathrm{rKL}}$  for baseline comparison. As we can see in Figure 1, increasing  $\lambda$  leads to an increase in precision in the resulting models. Using our  $\mathcal{D}_{\lambda -\mathrm{PR}}$  estimation, we compute the corresponding PR curves (Figure 2(c)). The  $\lambda = 0.1$  model, while best at  $\alpha_{0.1}$ , performs poorly at  $\alpha_{1}$  and  $\alpha_{10}$ , clearly demonstrating the impact of maximizing  $\alpha_{\lambda}$ . This pattern across models validates the efficacy of our method in minimizing the desired trade-off.
Using  $\mathcal{D}_{\mathrm{KL}}$  vs  $\mathcal{D}_{\chi^2}$  on MNIST and FashionMNIST: We now demonstrate that we can improve the precision by directly minimizing the  $\mathcal{D}_{\lambda -\mathrm{PR}}$  with the correct  $\lambda$  using pre-trained GLOW models [27] on both MNIST [54] and FashionMNIST [53]. Figure 3 and Figure 4 present the samples obtained. First, we observe that while  $\lambda$  increases, the visual quality improves, but the models focus on a few modes only (0, 1, 7, 6, 9 for MNIST and "Trouser" for FashionMNIST). Then training with both  $\mathcal{D}_{\mathrm{KL}}$  and  $\mathcal{D}_{\mathrm{rKL}}$  divergences aligns with our expectations:  $\mathcal{D}_{\mathrm{KL}}$  training leads to high recall, while  $\mathcal{D}_{\mathrm{rKL}}$ , to higher precision. However, according to Corollary 4.5,  $\mathcal{D}_{\mathrm{rKL}}$  still favors low  $\lambda$  values; consequently, our models trained with  $\lambda >0.1$  demonstrate better precision than standard flow-GAN models. Furthermore, we find that both auxiliary  $g$  functions ( $f_{\mathrm{KL}}$  and  $f_{\chi^2}$ ) used to train the discriminator  $T$  perform well. In practice, training with  $f_{\chi^2}$  proves to be more stable, particularly for FashionMNIST, with results reported in Appendix D.3. For larger models, we use exclusively  $g = f_{\chi^2}$ .
<sup>3</sup>https://github.com/AlexVerine/PrecisionRecallBigGan
(c)  $\lambda = 10$
(d)  $\lambda = 0.1$
(e)  $\lambda = 1$
(f)  $\lambda = 10$
Figure 3: Samples from NFs - GLOW trained on MNIST (3(a) to 3(c)) and FashionMNIST (3(d) to 3(f)). Recall decreases as precision increases for  $\lambda$  between 0.1 and 10.
Figure 4: Precision and Recall as a function on  $\lambda$  for GLOW trained on MNIST with  $T$  trained with  $f_{\mathrm{KL}}$  or  $f_{\chi^2}$ , and for models trained  $\mathcal{D}_{\mathrm{KL}}$ ,  $\mathcal{D}_{\mathrm{rKL}}$  and MLE.
Training BigGAN on CIFAR-10 and CelebA64 Now we demonstrate that our method can also be used to train large generative models. Our choice to adopt the BigGAN architecture [7] was informed by several factors: its competitive performance close to state-of-the-art models; its versatility,
permitting diverse experimental explorations; and the fact that it is publicly accessible, ensuring experiment reproducibility. We train BigGAN using both the baseline method (i.e., hinge loss) and our proposed method on CIFAR-10 [1] and CelebA64 [30]. A notable observation when training with different precision-recall trade-offs is the early elimination of modes from the target distribution at higher values of  $\lambda$ . As illustrated in Figure 5, models with low values of  $\lambda$  converge to achieve maximum recall, while those with  $\lambda > 1$  rapidly saturate to a lower recall value. A similar behavior can be observed for models trained on CelebA, as shown in Figure D.18. In Table 2, we present the quantitative metrics (Precision, Recall, and FID) for the baseline BigGAN, the BigGAN models trained with varying trade-offs and the current state-of-the-art models: EDM-G++ [26] for CIFAR-10 and ADM-IP [38] for CelebA64. Employing our proposed method enables us to adjust the trade-off, allowing us to train models that closely approach the state-of-the-art recall and, for high  $\lambda$ , even outperform state-of-the-art models in terms of precision.
Figure 5: Recall during training of BigGAN optimizing different tradeoff on CIFAR-10. Models for high  $\lambda$  saturates in an early stage at decreasing values of recall.
Table 2: BigGAN trained with the vanilla approach [7] and with a variety of  $\lambda$  using our approach on CIFAR-10 and CelebA64. We compare our approach with hard truncation on the baseline model. FID (↓), Precision (↑), Recall (↑), Density (↑) and Coverage (↑) are reported. In bold, our best model is highlighted and the state-of-the-art FID is marked with an exponent * .
MODEL	CIFAR-10 32 × 32	CELEBA 64 × 64
FID	P	R	D	C	FID	P	R	D	C
BASELINE BIGGAN	13.37	86.51	65.66	0.76	0.81	9.16	78.41	51.42	0.89	0.48
HARD ψ = 2.0	13.95	86.82	63.58	0.77	0.79	10.60	80.81	48.21	0.96	0.50
HARD ψ = 1.0	17.23	88.03	53.63	0.83	0.75	17.97	84.30	37.46	1.11	0.49
HARD ψ = 0.5	20.11	87.87	44.98	0.83	0.70	25.70	83.70	28.81	1.08	0.42
λ = 0.05	13.29	81.10	70.63	0.61	0.80	-	-	-	-	-
λ = 0.1	11.62	81.78	74.58	0.66	0.83	-	-	-	-	-
λ = 0.2	13.36	84.85	65.13	0.74	0.82	8.79	83.37	44.07	1.09	0.54
λ = 0.5	14.50	83.27	68.23	0.70	0.81	6.03	77.60	55.98	0.88	0.50
λ = 1.0	14.03	83.04	69.35	0.68	0.79	13.07	81.70	36.85	1.00	0.47
λ = 2.0	16.94	84.93	59.79	0.75	0.78	14.23	82.98	32.87	1.16	0.49
λ = 5.0	32.54	83.39	56.94	0.68	0.73	22.45	83.96	25.81	1.21	0.43
λ = 10.0	39.69	84.11	39.29	0.75	0.67	-	-	-	-	-
λ = 20.0	67.03	90.03	21.81	0.98	0.56	-	-	-	-	-
DENSEFLOW [18]	-	88.90	60.81	0.86	0.71	-	85.83	38.22	1.17	0.82
ADM-IP [38]	3.25	80.67	83.65	0.65	0.87	1.53*	23.42	64.48	0.09	0.24
EDM G++ [26]	1.77*	78.48	85.83	0.60	0.87	-	-	-	-	-
STYLEGAN-XL [46]	1.85	85.11	70.04	0.75	0.85	-	-	-	-	-
In every experiment, we compare our approach with traditional post-training techniques. In Table 2, we give the results for the hard truncation (also called the truncation trick in [7]) of the latent distribution  $Q$  for different  $\psi$ . We observe that this method enables to improve solely the precision by trading off the recall; however, note that the truncation can be used in addition to our approach.
Fine-tuning BigGAN onImagenet128 and FFFQ Finally, we apply our method to pre-trained BigGAN models. To accomplish this, we implement a straightforward technique: initially, we train the discriminator for a brief period, allowing the model to transition from the vanilla training objective to the  $\mathcal{D}_g^{\mathrm{dual}}$ . This approach enables us to train BigGAN on large datasets such as ImageNet128 [43] and datasets with high dimensions such as FFHQ256 [24]. We compare our method with both hard truncation and soft truncation (also denoted temperature in [27]). Both methods can be used in addition to our approach. The metrics presented in Table 3 demonstrate that (1) we enhance a given model's precision (by +2.83%) or recall (by +1.17%) on ImageNet, thereby achieving state-of-the-art precision, and (2) our method compromises less on the trade-off than truncation. For instance, in FFHQ, for a similar precision improvement ( $\approx +15.5\%$ ), recall is decreased by more than 5% for truncation methods and only by 1.65% with our approach.
Table 3: BigGAN fine-tune with the vanilla approach [7] and with a variety of  $\lambda$  using our approach on ImageNet128 and FFHQ256. We compare our approach with hard truncation on the baseline model. FID (↓), Precision (↑), Recall (↑), Density (↑) and Coverage (↑) are reported. In bold, our best model is highlighted and the state-of-the-art FID is marked with an exponent *
MODEL	FID	IMAGENET 128 × 128	FFHQ 256 × 256
P	R	D	C	FID	P	R	D	C
BASELINE BIGGAN	9.83	28.04	41.21	0.14	0.17	41.41	65.57	10.17	0.52	0.47
SOFT ψ = 0.7	11.39	23.04	31.13	0.11	0.15	56.43	76.59	4.87	0.70	0.41
SOFT ψ = 0.5	15.49	20.20	19.83	0.10	0.14	82.05	84.48	1.58	0.89	0.32
HARD ψ = 2.0	9.69	25.83	39.89	0.13	0.18	43.32	68.84	8.66	0.58	0.47
HARD ψ = 1.0	12.12	21.86	35.42	0.11	0.15	56.19	76.44	4.76	0.75	0.44
HARD ψ = 0.5	15.21	21.13	29.55	0.10	0.13	71.32	80.99	4.84	0.84	0.36
λ = 0.2	9.92	26.69	42.04	0.13	0.17	35.66	78.70	9.45	0.88	0.60
λ = 0.5	10.82	26.83	42.38	0.13	0.16	35.24	78.41	9.66	0.89	0.60
λ = 1.0	20.42	29.72	28.21	0.15	0.15	35.91	78.95	8.32	0.90	0.57
λ = 2.0	20.21	30.27	30.49	0.14	0.14	36.33	81.10	8.69	1.05	0.64
λ = 5.0	20.76	30.87	28.38	0.15	0.15	38.16	84.31	8.52	1.15	0.63
ADM [22]	2.97	26.63	68.54	0.14	0.16	-	-	-	-	-
STYLEGAN-xL [46]	1.81*	11.35	68.04	0.04	0.09	2.19*	79.91	38.79	0.86	0.73
In this paper, we present a novel method for training generative models using a new PR-divergence,  $\mathcal{D}_{\lambda -\mathrm{PR}}$ . Our approach offers a unique advantage over existing methods as it allows for explicit control of the precision-recall trade-off in generative models. By varying the trade-off parameter  $\lambda$ , one can train a variety of models ranging from mode seeking (high precision) to mode covering (high recall), as well as more balanced models that may be more suitable for various applications. Through extensive experiments, we demonstrate the validity of our method and show that it scales well with larger dimensions and datasets. Our approach also provides insights into the implicit P&R trade-offs made by models trained with other  $f$ -divergences. By introducing the  $\mathcal{D}_{\lambda -\mathrm{PR}}$  divergence and providing a systematic approach for training generative models based on user-specified trade-offs, we contribute to the development of more customizable generative models. Our method currently applies to GANs and NFs only, it is still unclear if a similar approach can be applied to trained diffusion models: a promising work [26] uses a discriminator to refine diffusion models, and could be used to estimate the  $\mathcal{D}_{\lambda -\mathrm{PR}}$ .
We are grateful for the grant of access to computing resources at the IDRIS Jean Zay cluster under allocations No. AD011011296 and No. AD011014053 made by GENCI.
[51] Chenyang Tao, Liquan Chen, Ricardo Henao, Jianfeng Feng, and Lawrence Carin Duke. Chisquare Generative Adversarial Network. In Proceedings of the 35th International Conference on Machine Learning, pages 4887-4896. PMLR, July 2018. URL https://proceedings.mlr.press/v80/tao18b.html. ISSN: 2640-3498.
[52] Alexandre Verine, Benjamin Negrevergne, Yann Chevaleyre, and Fabrice Rossi. On the expressivity of bi-Lipschitz normalizing flows. In Proceedings of The 14th Asian Conference on Machine Learning, pages 1054-1069. PMLR, April 2023. URL https://proceedings.mlr.press/v189/verine23a.html. ISSN: 2640-3498.
[53] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms, September 2017. URL http://arxiv.org/abs/1708.07747. arXiv:1708.07747 [cs, stat].
[54] Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs, 2, 2010. URL http://yann.learcun.com/exdb/mnist.
A.1  $f$ -divergences
$f$ -divergence between two probability distributions  $P$  and  $Q$  over a common support  $\mathcal{X}$  is defined as:
$$
\mathcal {D} _ {f} (P \| Q) = \int_ {\chi} q (\boldsymbol {x}) f \left(\frac {p (\boldsymbol {x})}{q (\boldsymbol {x})}\right) \mathrm {d} \boldsymbol {x}, \tag {14}
$$
where  $p(\boldsymbol{x})$  and  $q(\boldsymbol{x})$  denote the densities of  $P$  and  $Q$ , respectively, and  $f$  is a convex function such that  $f(1) = 0$ . Any  $\mathcal{D}_f$  admits a dual variational form [37], with  $\mathcal{T}$  denoting the set of all measurable functions  $\mathcal{X} \to \mathbb{R}$ :
$$
\mathcal {D} _ {f} (P \| \hat {P}) = \sup  _ {T \in \mathcal {T}} \left(\mathbb {E} _ {P} [ T (\boldsymbol {x}) ] - \mathbb {E} _ {\hat {P}} [ f ^ {*} (T (\boldsymbol {x})) ]\right), \tag {15}
$$
The properties of  $f$ -divergence are as follows:
- Non-Negativity: For any two probability distributions  $P$  and  $\widehat{P}$  on  $\mathcal{X}$ , we have  $\mathcal{D}_f(P\| \widehat{P}) \geq 0$ . The equality holds if and only if  $P = \widehat{P}$ .
- Convexity in  $f$ : If  $f(u)$  is a convex function, then  $\mathcal{D}_f(P\| \widehat{P})$  is convex in the input distributions  $P$  and  $\widehat{P}$ .
- Non-Symmetry: Generally,  $f$ -divergence is not symmetric, i.e.,  $\mathcal{D}_f(P\| \widehat{P}) \neq \mathcal{D}_f(\widehat{P}\| P)$ .
- Linearity:  $f$ -divergence is linear in  $f$ , i.e., for any two convex functions  $f_{1}(u)$  and  $f_{2}(u)$  and for any two real numbers  $a$  and  $b$ , the divergence  $\mathcal{D}_{af_1 + bf_2}(P\| \widehat{P}) = a\mathcal{D}_{f_1}(P\| \widehat{P}) + b\mathcal{D}_{f_2}(P\| \widehat{P})$ .
Specific choices of the function  $f$  in the  $f$ -divergence definition can yield various well-known divergence measures. We will review some of them of the principal properties. For every divergence, we report here the generator function  $f$ , its convex conjugate  $f^*$ , the domain of the convex conjugate  $\mathrm{dom}(f^*)$ , the typical activation function used to ensure that  $T(\boldsymbol{x}) \in \mathrm{dom}(f^*)$  and finally the optimal discriminator  $T^*$ .
Kullback Leibler:
Notation:  $\mathcal{D}_{\mathrm{KL}}$
- Generator Function:  $f(u) = u\log (u)$ ,
- Convex Conjugate domain:  $\mathrm{dom}(f^{*}) = \mathbb{R}$
- Convex Conjugate:  $f^{*}(t) = \exp (t - 1)$ ,
- Activation:  $v \mapsto v$ ,
- Optimal discriminator:  $T^{*}(\pmb{x}) = 1 + \log \left( \frac{p(\pmb{x})}{\widehat{p}(\pmb{x})} \right)$ .
Reverse Kullback Leibler:
Notation:  $\mathcal{D}_{\mathrm{rKL}}$
- Generator Function:  $f(u) = -\log (u)$ ,
- Convex Conjugate domain:  $\mathrm{dom}(f^{*}) = \mathbb{R}^{-}$
- Convex Conjugate:  $f^{*}(t) = -1 - \log - t$ ,
Activation:  $v\mapsto -\exp v$
- Optimal discriminator:  $T^{*}(\pmb{x}) = -\frac{\widehat{p}(\pmb{x})}{p(\pmb{x})}$ .
$\chi^2$  Pearson:
Notation:  $\mathcal{D}_{\chi^2}$
- Generator Function:  $f(u) = (u - 1)^2$ ,
- Convex Conjugate domain:  $\mathrm{dom}(f^{*}) = \mathbb{R}$
- Convex Conjugate:  $f^{*}(t) = t^{2} / 4 + t$ ,
Activation:  $v\mapsto v$
- Optimal discriminator:  $T^{*}(\boldsymbol{x}) = 2\left(\frac{p(\boldsymbol{x})}{\widehat{p}(\boldsymbol{x})} - 1\right)$ .
GAN: