SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {D} _ {f} (P \| \widehat {P}) = \int_ {\mathcal {X}} \widehat {p} (\boldsymbol {x}) f \left(\frac {p (\boldsymbol {x})}{\widehat {p} (\boldsymbol {x})}\right) \mathrm {d} \boldsymbol {x} \\ = \int_ {\mathcal {X}} \widehat {p} (\boldsymbol {x}) \int_ {m} ^ {M} \frac {1}{\lambda^ {3}} f ^ {\prime \prime} \left(\frac {1}{\lambda}\right) f _ {\lambda} ^ {\mathrm {P R}} \left(\frac {p (\boldsymbol {x})}{\widehat {p} (\boldsymbol {x})}\right) \mathrm {d} \lambda \mathrm {d} \boldsymbol {x} \\ = \int_ {m} ^ {M} \int_ {\mathcal {X}} \frac {1}{\lambda^ {3}} f ^ {\prime \prime} \left(\frac {1}{\lambda}\right) \widehat {p} (\boldsymbol {x}) f _ {\lambda} ^ {\mathrm {P R}} \left(\frac {p (\boldsymbol {x})}{\widehat {p} (\boldsymbol {x})}\right) \mathrm {d} \lambda \mathrm {d} \boldsymbol {x} \\ = \int_ {m} ^ {M} \frac {1}{\lambda^ {3}} f ^ {\prime \prime} \left(\frac {1}{\lambda}\right) \left[ \int_ {\mathcal {X}} \widehat {p} (\boldsymbol {x}) f _ {\lambda} ^ {\mathrm {P R}} \left(\frac {p (\boldsymbol {x})}{\widehat {p} (\boldsymbol {x})}\right) \mathrm {d} \boldsymbol {x} \right] \mathrm {d} \lambda \\ = \int_ {m} ^ {M} \frac {1}{\lambda^ {3}} f ^ {\prime \prime} \left(\frac {1}{\lambda}\right) \mathcal {D} _ {\lambda - \mathrm {P R}} (P \| \widehat {P}) \mathrm {d} \lambda \\ \end{array}
$$
B.6 Proof of Corollary 4.5
In particular for the  $\mathcal{D}_{\mathrm{KL}}$ ,  $f(u) = u\log u$ , therefore  $f^{\prime \prime}(u) = 1 / u$  which gives:
$$
\mathcal {D} _ {\mathrm {K L}} (P \| \widehat {P}) = \int_ {m} ^ {M} \frac {1}{\lambda^ {2}} \mathcal {D} _ {\lambda - \mathrm {P R}} (P \| \widehat {P}) \mathrm {d} \lambda . \tag {49}
$$
And for the  $\mathcal{D}_{\mathrm{rKL}}$  we can either use Equation 48 with  $f(u) = -\log u$  or use the fact that  $\mathcal{D}_{\lambda_{-}\mathrm{PR}}(P\| \widehat{P}) = \lambda \mathcal{D}_{1 / \lambda_{-}\mathrm{PR}}(\widehat{P}\| P)$ :
$$
\mathcal {D} _ {\mathrm {r K L}} (P \| \widehat {P}) = \int_ {m} ^ {M} \frac {1}{\lambda} \mathcal {D} _ {\lambda - \mathrm {P R}} (P \| \widehat {P}) \mathrm {d} \lambda . \tag {50}
$$
C Minimizing the Precision-Recall divergence
In Section 5, we outline a novel strategy to train models for challenging  $f$ -divergences. Rather than directly employing the  $f$ -GAN framework and minimizing the dual variational form of  $\mathcal{D}_{\lambda_{\mathrm{PR}}}$ , we introduce an auxiliary function, denoted by  $g$ , which is easier to train. Practical choices for  $g$  could include functions such as  $f_{\mathrm{KL}}$  or  $f_{\chi^2}$ , which have been shown to be easily trainable.
Using this approach, a discriminator  $T$  is trained with the objective of maximizing  $\mathcal{D}_{g,T}^{\mathrm{dual}}$ . This is achieved by drawing samples from both  $P$  and  $\widehat{P}$  and computing the estimate in the dual form. Under optimal conditions, we find  $\nabla g^{*}(T^{\mathrm{opt}}(\pmb{x})) = p(\pmb{x}) / \widehat{p}(\pmb{x})$ , enabling us to estimate the  $f$ -divergence based on  $T$  using what we term the primal estimate.
The primal estimate is defined as  $\mathcal{D}_{f,T}^{\mathrm{primal}}(P\| \widehat{P}) = \int_{\mathcal{X}}\widehat{p} (\boldsymbol {x})f(r(\boldsymbol {x}))\mathrm{d}\boldsymbol{x}$ , where  $r(\boldsymbol {x}) = \nabla g^{*}(T(\boldsymbol {x}))$ . This strategy and its corresponding training procedure are visually represented in Figure C.7.
Figure C.7: Training procedure: the discriminator  $T$  is trained based on  $\mathcal{D}_{g,T}^{\mathrm{dual}}$  and  $G$  is trained on  $\mathcal{D}^{\mathrm{primal}}f,T$ .
We have shown that the choice of  $g$  affects the quality of the estimation of  $\mathcal{D}_f$ . We have shown that to have guarantees on the quality of the estimation,  $g$  must be strictly convex. We show empirically that  $g = f_{\chi^2}$  leads to a better approximation than  $g = f_{\mathrm{KL}}$ . We train 200 discriminators on 20 Gaussian random two-dimensional mixtures to maximize  $\mathcal{D}_{g,T}^{\mathrm{dual}}$  with  $g = f_{\mathrm{KL}}$  and  $g = f_{\chi^2}$ . In Figure C.8, we report the results. On the  $x$ -axis, we report the training loss quality  $\mathcal{D}_g(P\| \widehat{P}) - \mathcal{D}_{g,T}^{\mathrm{dual}}(P\| \widehat{P})$  and on the  $y$ -axis we report the estimation quality:  $\mathcal{D}_f(P\| \widehat{P}) - \mathcal{D}_{f,T}^{\mathrm{primal}}(P\| \widehat{P})$ . The dotted lines correspond to the Mahalanobis distance of each set of experiments. In this experiment, we use a 4 linear layers 2-1024-512-256-1 neural network with LeakyRelu activation between layers. The last activation is set to match  $\operatorname{dom}(f^{*})$  (see Section A.1). We use a learning rate of  $2.10^{-5}$  with Adam optimizer.
Figure C.8: We compare the quality of the primal estimation using two discriminators, one trained using the  $\mathcal{D}_{\mathrm{KL}}$  and  $\mathcal{D}_{\chi^2}$  over 200 experiments. The lower the point is on the  $y$ -axis, the better the estimate. As we can see, even distant dual estimates of the  $f_{\chi^2}$  (points on the right side of the chart) tend to provide better estimates of the primal.
We observe in this experiment that 1) the difference between  $\mathcal{D}_{\mathrm{KL}}$  and its dual is usually lower (the blue circle is lower on  $x$ -axis) and yet 2) the estimation of  $\mathcal{D}_f$  is usually better with  $g = f_{\chi^2}$  (the red circle is lower on  $y$ -axis). These observations corroborate our claim that employing  $f_{\chi^2}$  as the auxiliary divergence is a more advantageous choice compared to  $f_{\mathrm{KL}}$ . Although both divergence measures exhibit satisfactory performance in the context of the MNIST and FashionMNIST experiments, we find  $f_{\chi^2}$  to demonstrate superior stability, reinforcing its suitability in this training framework.
Complete version of the algorithm In Section 5, we introduce a condensed version of our training algorithm. For practical purposes, however, we implement a more refined variant of this algorithm using stochastic gradient descent, detailed in Algorithm 1. This approach involves segmenting the dataset into batches of size  $N$  and computing the distinct losses for each individual batch. Our algorithm is closely related to the GAN training procedure, operating on an iterative principle of alternating updates to the discriminator  $T$  and the generator  $G$  until a convergence state is reached. In particular, our method differs from the standard GAN training protocol in two key aspects. First, the generator  $G$  in our case is trained based on an estimation of a distinct  $f$ -divergence. Second, this specific  $f$ -divergence is calculated in terms of its primal form, rather than its dual form, a marked departure from the established GAN training practice.
Algorithm 1 Stochastic Gradient Descent for the two-step approach for minimizing  $\mathcal{D}_{\lambda -\mathrm{PR}}$ . For each batch  $B$  composing the dataset  $D$ , the discriminator  $T$  is trained on  $\mathcal{L}_T$ , the empirical estimation of  $\mathcal{D}_{f,T}^{\mathrm{dual}}(P\| \widehat{P})$  and the generator  $G$  is trained on  $\mathcal{L}_G$ , the empirical estimation of  $\mathcal{D}_{\lambda -\mathrm{PR}}^{\mathrm{primal}}(P\| \widehat{P})$ .
Input: Generator  $G$  , Discriminator  $T$  ,Dataset  $D$ $g^{*}\gets f_{\chi^{2}}^{*}\mathrm{or}f_{\mathrm{KL}}^{*}$ $f\gets f_{\lambda}$    
for epoch  $e = 1,\dots ,E$  do   
for  $B\in D$  do   
 $\mathcal{L}_T,\mathcal{L}_G\gets 0,0$    
for  $\pmb{x}_{1}^{\mathrm{real}},\ldots ,\pmb{x}_{N}^{\mathrm{real}}\in B$  do   
for  $i = 1$  to  $N$  do Generate  $\pmb{x}_i^{\mathrm{fake}} = G(\pmb {z})$  with  $\pmb {z}\sim \mathcal{N}(\pmb {0}_d,\pmb {I}_d)$ $\mathcal{L}_T\gets \mathcal{L}_T + T(\pmb{x}_i^{\mathrm{real}}) - f^* (T(\pmb{x}_i^{\mathrm{fake}}))$    
end for   
Update parameters of  $T$  by ascending the gradient  $\nabla \mathcal{L}_T$    
for  $i = 1$  to  $N$  do   
 $\mathcal{L}_G\gets \mathcal{L}_G + g\left(\nabla f^* (T(\pmb{x}_i^{\mathrm{fake}})\right)$    
end for   
Update parameters of  $G$  by descending the gradient  $\nabla \mathcal{L}_G$    
end for   
end for
D.1 Naive Approach
In this section, we juxtapose the straightforward method with our technique delineated in Section 5. We achieve this by evaluating the optimization process of two identical  $G$  models: one is trained using the conventional approach and the other using our methodology. In reality, there exist two significant variations between these two training procedures: training losses and the final layer of the discriminator  $T$ . In both procedures, we use  $f^{*}(T(\pmb{x}))$ , implying that the co-domain must be encapsulated within  $\mathrm{dom}(f^{*})$ . More information on activation can be found in the Appendix A.1. Aside from these disparities, we utilize the identical training process as outlined in the previous section to train the models on CIFAR-10. The training procedures are compared in Figure D.9. To evaluate training procedures, we plot the loss  $\mathcal{L}_G$ , used to train  $G$ , either the dual estimation of  $\mathcal{D}_{\lambda_{\mathrm{PR}}}$  in the naive approach or the primal estimation of  $\mathcal{D}_{\lambda_{\mathrm{PR}}}$  in our method. To track the evolution of the discriminator, we plot discriminating predictions on the input conditioned on the class (real or fake). Precisely, we compute the accuracy on a batch of real or fake images, how many the discriminator achieves to identify as such. Finally, we plot the Precision and Recall computed at each epoch.
We observe that for varying values of  $\lambda$ , the discriminator  $T$  undergoes the training process using the naive approach, leading to an unsatisfactory estimation of  $\mathcal{D}_{f,T}^{\mathrm{dual}}$ .
Figure D.9: Results for training BigGAN models on CIFAR-10 using the naive approach (in blue) and our method (in red). For  $\lambda = 0.2$ ,  $\lambda = 1$  and  $\lambda = 5$ , the loss represented by  $G$  in learned. We obverse that for the naive approach, this loss is constant. Then, we plot the accuracy of the discriminator. We plot the accuracy conditioned on the class: in the dotted line the accuracy on the images of the dataset  $\pmb{x}^{\mathrm{real}}$  and in the solid line the accuracy on the generated images  $\pmb{x}^{\mathrm{fake}}$ . Then we plot the Precision and Recall during training.
D.2 Synthetic data: 8 Gaussians
For the synthetic data, we use a RealNVP [14] for the generator  $G$ . We use an 8-coupling step composed of each of 2 linear layers 2-256-2 with LeakyRelu activation in between. For the discriminator, we used a 4 linear layers 2-1024-512-256-1 neural network with LeakyRelu activation between layers. For both, we use Adam optimizer with a learning rate of  $2.10^{-5}$  for  $G$  and  $1.10^{-4}$  for  $T$ .  $G$  has 540k parameters and 660k for  $T$ .
Then, to estimate the PR curves, both methods from Sajjadi et al. [44] and Simon et al. [47] are developed for image dataset. To estimate these curves, we use our own estimation of  $\mathcal{D}_{\lambda_{\mathrm{-PR}}}$ . We computed  $\mathcal{D}_{\lambda_{\mathrm{-PR}}}$  for  $\lambda = \tan (\theta)$  with  $\theta$  between 0 and  $\pi /2$ . The observations from the generated curves underscore a compelling finding: Each model has been effectively optimized for a unique, specific  $\lambda$ , resulting in its superior performance on the corresponding  $\mathcal{D}_{\lambda_{\mathrm{-PR}}}$ . This indicates a clear correspondence between the intended optimization and the achieved performance, suggesting the efficacy of our training algorithm in optimizing models for specific performance metrics. Such a successful matching of model optimization to  $\lambda$  underscores the feasibility of tailoring model training according to a specific precision-recall trade-off objective.
Figure D.10: PR Curves of the models in Figure 1. In shades of blue, the curves correspond to the model trained to minimize  $\mathcal{D}_{\lambda_{\mathrm{PR}}}$ , with dark blue for  $\lambda = 0.1$ , medium blue for  $\lambda = 1$ , and light blue for  $\lambda = 1$ . The green curve corresponds to  $\mathcal{D}_{\mathrm{KL}}$  and the red curve to  $\mathcal{D}_{\mathrm{rKL}}$ .
D.3 MNIST and FashionMNIST
The training procedure for MNIST and FashionMNIST is strictly the same. For both we use a multiscale glow [27]. The model has three levels processing images of size  $4 \times 16 \times 16$ ,  $16 \times 8 \times 8$ ,  $64 \times 8 \times 8$ . Each level has 16 blocks of affine coupling with 3 layers of 512 channels of convolutional
operations, leading to a total of 85.2M parameters. For the discriminator, we use a 4 linear layers 1024-1024-512-256-1 neural network with LeakyRelu activation between layers, with 1.7M parameters. Both are trained with Adam using a learning rate of  $1.10^{-5}$  for  $T$  and  $1.10^{-6}$  for  $G$ . For both dataset, we train a model for 250 epochs using maximum likelihood estimation (MLE) in 4 GPUs V100 ( $\sim$  200 hours). The models are then fine-tuned with their different losses on 12 V100 GPUs for 30 epochs ( $\sim$  2 hours). For two epochs we train the discriminator only and then both. For MNIST, we report results as graphs in Figure D.11, the quantitative results in Table 4 and samples in Figure D.3. For FashionMNIST, we report the results as graph in Figure D.13, the quantitative results in Table 4 and samples in Figure D.3.
Table 4: Quantitative evaluation of various Glow [27] models on the MNIST and FashionMNIST datasets. Models differ by their choice of  $\mathcal{D}_f$ ,  $\mathcal{D}_g$ , and  $\lambda$ . The performance metrics include the FID (↓), P (↓) and R (↑). The best model is highlighted in bold. FID is calculated for 50k samples.P and R are calculated for 10k samples and  $k = 3$ .
Model	MNIST	FashionMNIST
Dg	Df	λ	FID	P	R	FID	P	R
MLE	7.68	70.03	76.52	66.64	58.19	47.35
DKL	DKL	-	7.76	70.68	77.19	71.72	51.44	45.61
DrKL	DrKL	-	10.50	74.64	67.21	51.65	67.68	40.76
Dx2	Dλ-PR	0.10	6.84	78.24	72.11	48.11	66.57	45.80
Dx2	Dλ-PR	0.20	6.48	78.84	71.02	41.97	66.41	46.62
Dx2	Dλ-PR	0.50	6.57	79.09	70.26	46.04	71.20	46.37
Dx2	Dλ-PR	1.00	7.95	79.55	69.08	52.88	73.32	36.44
Dx2	Dλ-PR	2.00	10.01	80.37	67.84	101.33	76.47	39.40
Dx2	Dλ-PR	5.00	15.31	81.34	67.00	118.54	73.44	35.14
Dx2	Dλ-PR	10.00	20.88	81.71	66.62	91.20	72.14	19.48
DKL	Dλ-PR	0.10	4.66	76.06	73.88	42.85	66.48	49.73
DKL	Dλ-PR	0.20	4.45	76.92	71.92	48.25	72.87	49.52
DKL	Dλ-PR	0.50	5.94	77.98	71.42	54.12	71.28	41.70
DKL	Dλ-PR	1.00	8.21	79.92	70.02	62.33	62.58	42.89
DKL	Dλ-PR	2.00	9.40	79.89	67.15	64.74	72.78	41.41
DKL	Dλ-PR	5.00	14.01	82.08	66.13	83.25	73.12	33.80
DKL	Dλ-PR	10.00	27.61	83.20	65.09	79.37	70.74	27.05
Figure D.11: MNIST: Glow models [27] are trained for different  $\lambda$ . From left to right, we plot IS ( $\uparrow$ ), FID ( $\downarrow$ ), P ( $\uparrow$ ) and R ( $\uparrow$ ). IS and FID are calculated using 50k samples, and P and R are calculated using 5k samples with  $k = 3$  using Kynkänniemi et al. [28]'s method. For comparison, we also report models trained with MLE (in black), for  $\mathcal{D}_{\mathrm{KL}}$  (in blue) and for  $\mathcal{D}_{\mathrm{rKL}}$  in red.
(a) MLE
(b)  $\lambda = 0.1$
(c)  $\lambda = 1$
(d)  $\lambda = 10$
Figure D.12: MNIST: Samples are drawn from different models (only models trained with  $g = f_{\chi^2}$ ). From 12(a) to 12(d), we observe that visual quality improves while diversity decreases. Models geared towards high recall, specifically MLE and those with  $\lambda = 0.1$ , are found to generate a wide variety of samples. However, these are characterized by lower precision, and approximately  $20\%$  of the generated samples are incoherent. On the contrary, the model trained with  $\lambda = 10$  appears to focus on a smaller subset of modes, demonstrating higher precision but limited diversity. Specifically, this model primarily generates classes 1, 6, 7, and 9.
Figure D.13: FashionMNIST: Glow models [27] are trained for different  $\lambda$ . From left to right, we plot IS ( $\uparrow$ ), FID ( $\downarrow$ ), P ( $\uparrow$ ) and R ( $\uparrow$ ). IS and FID are calculated using 50k samples, and P and R are calculated using 10k samples with  $k = 3$  using Kynkänniemi et al. [28]'s method. For comparison, we also report models trained with MLE (in black), for  $\mathcal{D}_{\mathrm{KL}}$  (in blue) and for  $\mathcal{D}_{\mathrm{rKL}}$  in red.
(d)  $\lambda = 10$
Figure D.14: FashionMNIST: Samples are drawn from different models (only models trained with  $g = f_{\chi^2}$ ). From 14(a) to 14(d), we observe that visual quality improves while diversity decreases. Models geared towards high recall, specifically MLE and those with  $\lambda = 0.1$ , are found to generate a wide variety of samples. However, these are characterized by lower precision, and approximately  $15\%$  of the generated samples are incoherent. On the contrary, the model trained with  $\lambda = 10$  appears to focus on a smaller subset of modes, demonstrating higher precision but limited diversity. Specifically, this model primarily generates the class "trouser".
D.4 Training BigGAN on CIFAR-10 and CelebA64
In this section we give the details of the experiments on training BigGAN [7] models. To do this, we modify the official implementation of PyTorch<sup>5</sup> of BigGAN by Brock et al. [7].  $G$  and  $T$  respectively count 4.3M and 4.2M parameters for CIFAR-10 and 32.0M and 19.5M for CelebA64. CIFAR-10's models are trained on 2 A100 80GB GPUs with a batch size of 128 for approximately 100k iterations ( $\sim$  7 hours), while CelebA64's models have been trained on 4 A1200 32GB GPUs with a batch size of 128 for 95k iteration ( $\sim$  20 hours). Quantitative results are reported in Table 2. The graphic representation of the results is in Figures D.16 and D.20. The training curves representative of FID, IS, P and R during training are reported in Figures D.17 and D.18.
<sup>5</sup>https://github.com/ajbrock/BigGAN-PyTorch
(a) BigGAN Baseline
(b)  $\lambda = 0.10$
(d)  $\lambda = 10$
Figure D.15: CIFAR-10: Samples are drawn from different BigGAN trained on the PR-Divergence. We observe that when  $\lambda$  increases, the recall decreases going to a various range of colored images to mostly brown images.
Figure D.16: CIFAR-10: BigGAN models [7] are trained for different  $\lambda$ . From left to right, we plot IS ( $\uparrow$ ), P ( $\uparrow$ ), D ( $\uparrow$ ), FID ( $\downarrow$ ), R ( $\uparrow$ ) and C ( $\uparrow$ ). IS and FID are calculated using 50k samples, and P and R are calculated using 10k samples with  $k = 5$  using Kynkänniemi et al. [28]'s method. For comparison, we also report a model trained hinge loss (in black), following the vanilla training procedure.
Figure D.17: CIFAR-10: Evolution of the IS, FID, P and R during training. We can observe that the precision quickly achieves its maximal value and saturates. We can also observe that the model is evicting modes of the target distribution early in the training, R is quickly saturates to different levels depending on  $\lambda$ . However, for low values of  $\lambda$ , the recall does not increase when  $\lambda$  decreases, indicating that the model capacity limits the maximum recall.
Figure D.18: CelebA Evolution of the IS, FID, P and R during training. We can observe that precision quickly achieves its maximal value and saturates. We can also observe that the model is evicting modes of the target distribution early in the training, R is quickly saturates to different level depending on  $\lambda$ .
(b)  $\lambda = 0.5$
(d)  $\lambda = 10$
Figure D.19: CelebA64: Samples are drawn from different BigGANs with our approach. For  $\lambda = 0.5$ , the model generated faces with various background, while the model generate only grey, black and brown backgrounds. However there are less artifacts on the latter model.
Figure D.20: CelabA64: BigGAN models [7] are trained for different  $\lambda$ . From left to right, we plot IS ( $\uparrow$ ), P ( $\uparrow$ ), D ( $\uparrow$ ), FID ( $\downarrow$ ), R ( $\uparrow$ ) and C ( $\uparrow$ ). IS and FID are computed using 50k samples and P and R are computed using 10k samples with  $k = 5$  using Kynkänniemi et al. [28]'s method. For comparison, we also report a model trained hinge loss (in black), following the vanilla training procedure.
In this section, we fine-tune the pre-trained BigGAN models. For ImageNet128, we use the weights of the model trained by Brock et al. [7], while for FFHQ256 we use a model pre-trained by our self, explaining why the FFHQ models is under performing. We train models on 4 V100 GPUs with a batch size of 128 for 10k iterations each. The graphic representation of the results is reported in Figures D.21 and D.22 and samples are shown in Figures D.5 and D.5.
Figure D.21: ImageNet128: BigGAN models [7] are trained for different  $\lambda$ . From left to right, we plot IS ( $\uparrow$ ), P ( $\uparrow$ ), D ( $\uparrow$ ), FID ( $\downarrow$ ), R ( $\uparrow$ ) and C ( $\uparrow$ ). IS and FID are calculated using 50k samples, and P and R are calculated using 10k samples with  $k = 5$  using Kynkänniemi et al. [28]'s method. For comparison, we also report a model trained hinge loss (in black), following the vanilla training procedure.
Figure D.22: FFHQ: BigGAN models [7] are trained for different  $\lambda$ . From left to right, we plot IS ( $\uparrow$ ), P ( $\uparrow$ ), D ( $\uparrow$ ), FID ( $\downarrow$ ), R ( $\uparrow$ ) and C ( $\uparrow$ ). IS and FID are computed using 50k samples and P and R are computed using 10k samples with  $k = 5$  using Kynkänniemi et al. [28]'s method. For comparison, we also report a model trained hinge loss (in black), following the vanilla training procedure.
D.5 Fine-tuning BigGAN on ImageNet128 and FFHQ256.
(b)  $\lambda = 0.2$
(d)  $\lambda = 5$
Figure D.23: ImageNet128: Samples are drawn from different BigGANs trained on the PR-Divergence.
(d)  $\lambda = 2$
Figure D.24: FFHQ256: Samples are drawn from different BigGAN trained on the PR-Divergence.
E Training the AUC
In this paper, we propose a method to optimize any trade-off between precision and recall. In practice, we optimize a model to maximize any  $\alpha_{\lambda}(P\|\widehat{P})$ . We could also optimize for the area under the PR-Curves  $\mathrm{PRD}(P,\widehat{P})(P,\widehat{P})$ . In this section, we will show how we can compute the AUC in terms of  $\alpha_{\lambda}(P\|\widehat{P})$  and train a model on a small dimension dataset and compare the results with the model trained on several  $\lambda$ . First, we must compute the AUC:
Proposition E.1 (AUC under the  $\partial \mathrm{PRD}(P,\widehat{P})$ ). The area under the curve is:
$$
\mathrm {A U C} = \int_ {0} ^ {+ \infty} \frac {\alpha_ {\lambda} (P \| \widehat {P}) ^ {2}}{\lambda^ {2}} \mathrm {d} \lambda \tag {51}
$$
Proof. The AUC can be computed by integrating with respect to an angle  $\theta$  in the first quadrant:
$$
\mathrm {A U C} = \int_ {\theta = 0} ^ {\pi / 2} \int_ {u = 0} ^ {r (\theta)} u \mathrm {d} u \mathrm {d} \theta = \int_ {0} ^ {\pi / 2} \frac {r ^ {2} (\theta)}{2} \mathrm {d} \theta . \tag {52}
$$
Therefore, with  $\lambda = \tan \theta$ , we have  $r(\theta) = \alpha_{\tan (\theta)}(P\| \widehat{P}) / \sin \theta$  (see Figure E.25). Thus:
$$
\begin{array}{l} \mathrm {A U C} = \int_ {0} ^ {\pi / 2} \frac {\alpha_ {\tan (\theta)} (P \| \widehat {P}) ^ {2}}{\sin^ {2} \theta} \mathrm {d} \theta (53) \\ = \int_ {0} ^ {\pi / 2} \alpha_ {\tan (\theta)} (P \| \widehat {P}) ^ {2} \left(1 + \frac {1}{\tan^ {2} \theta}\right) \mathrm {d} \theta (54) \\ = \int_ {0} ^ {+ \infty} \alpha_ {\lambda} (P \| \widehat {P}) ^ {2} \frac {\left(1 + \frac {1}{\lambda^ {2}}\right)}{1 + \lambda^ {2}} \mathrm {d} \lambda \quad \text {w i t h} \quad \mathrm {d} \lambda = (1 + \tan^ {2} \theta) \mathrm {d} \theta , (55) \\ = \int_ {0} ^ {+ \infty} \frac {\alpha_ {\lambda} (P \| \widehat {P}) ^ {2}}{\lambda^ {2}} \mathrm {d} \lambda . (56) \\ \end{array}
$$