SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 4 For any three  $n$ -dimensional Gaussians  $\mathcal{N}(\pmb{\mu}_i, \pmb{\Sigma}_i) (i \in \{1, 2, 3\})$  such that  $KL(\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1) || \mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)) \leq \varepsilon_1$  and  $KL(\mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2) || \mathcal{N}(\pmb{\mu}_3, \pmb{\Sigma}_3)) \leq \varepsilon_2$  for  $\varepsilon_1, \varepsilon_2 \geq 0$ ,
$$
\begin{array}{l} K L \left(\mathcal {N} \left(\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1}\right) | | \boldsymbol {\Sigma} \left(\boldsymbol {\mu} _ {3}, \boldsymbol {\Sigma} _ {3}\right)\right) <   \varepsilon_ {1} + \varepsilon_ {2} + \frac {1}{2} \left(W _ {- 1} \left(- e ^ {- (1 + 2 \varepsilon_ {1})}\right) W _ {- 1} \left(- e ^ {- (1 + 2 \varepsilon_ {2})}\right) + W _ {- 1} \left(- e ^ {- (1 + 2 \varepsilon_ {1})}\right) \right. \\ \left. + W _ {- 1} \left(- e ^ {- (1 + 2 \varepsilon_ {2})}\right) + 1 - W _ {- 1} \left(- e ^ {- (1 + 2 \varepsilon_ {2})}\right) \left(\sqrt {2 \varepsilon_ {1}} + \sqrt {\frac {2 \varepsilon_ {2}}{- W _ {0} \left(- e ^ {- (1 + 2 \varepsilon_ {2})}\right)}}\right) ^ {2}\right) \\ \end{array}
$$
Overview of Proof of Theorem 4
The proof of Theorem 4 is the most technical part of this paper. Please see Appendix I for details. Here we give the overview of the proof.
We want to solve the following optimization problem  $P_{3}$  analytically.
$$
\begin{array}{l} P _ {3}: \text {m a x i m i z e} K L (\mathcal {N} (\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1}) | | \mathcal {N} (\boldsymbol {\Sigma} (\boldsymbol {\mu} _ {3}, \boldsymbol {\Sigma} _ {3})) \\ s. t. K L (\mathcal {N} (\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1}) | | \mathcal {N} (\boldsymbol {\mu} _ {2}, \boldsymbol {\Sigma} _ {2})) \leq \varepsilon_ {1} \\ K L (\mathcal {N} (\boldsymbol {\mu} _ {2}, \boldsymbol {\Sigma} _ {2}) | | \mathcal {N} (\boldsymbol {\mu} _ {3}, \boldsymbol {\Sigma} _ {3})) \leq \varepsilon_ {2} \\ \end{array}
$$
Unfortunately, it is hard to find the supremum due to the complexity caused by Lambert  $W$  function. So we relax the constraints to simplify the problem. The proof mainly consists of the following four steps.
1. Invertible linear transformation. The first step is similar to that of Theorem 1. We apply a linear transformation on  $\mathcal{N}_1$ ,  $\mathcal{N}_2$ , and  $\mathcal{N}_3$  simultaneously such that  $\mathcal{N}_2$  is converted to standard Gaussian. The transformed problem will be proved in Lemma 2 (shown below). In the end, these three Gaussians are transformed back to general case.
2. Relaxing constraints. In the proof of Lemma 2, we relax the constraints and finally reduce to the following core problem  $P_4$ .
$$
\boldsymbol {P} _ {4}: \text {m a x i m i z e} \sum_ {i = 1} ^ {n} \lambda_ {1, [ i ]} \lambda^ {\prime} _ {2, [ i ]} - \log \lambda_ {1, [ i ]} \lambda^ {\prime} _ {2, [ i ]} \tag {6}
$$
$$
\begin{array}{l} s. t. \lambda_ {1, [ i ]} - \log \lambda_ {1, [ i ]} = 1 + \varepsilon_ {1, [ i ]} (1 \leq i \leq n) \\ \bigwedge_ {i = 1} ^ {n} \varepsilon_ {1, [ i ]} \geq 0 \wedge \sum_ {i = 1} ^ {n} \varepsilon_ {1, [ i ]} = 2 \varepsilon_ {1} \\ \lambda^ {\prime} _ {2, [ i ]} - \log \lambda^ {\prime} _ {2, [ i ]} = 1 + \varepsilon_ {2, [ i ]} (1 \leq i \leq n) \\ \bigwedge_ {i = 1} ^ {n} \varepsilon_ {2, [ i ]} \geq 0 \wedge \sum_ {i = 1} ^ {n} \varepsilon_ {2, [ i ]} = 2 \varepsilon_ {2} \\ \end{array}
$$
where  $\lambda_{1,[i]}$  and  $\lambda_{2,[i]}^{\prime}$  are the eigenvalues of  $\pmb{\Sigma}_{1}$  and  $\pmb{\Sigma}_{2}^{-1}$  arranged in decreasing order, respectively.  $\varepsilon_{1,[i]}$  and  $\varepsilon_{2,[i]}$  are arranged in decreasing order too.
3. Allocating  $2\varepsilon_{1}$  and  $2\varepsilon_{2}$ . The value of objective function in  $\pmb{P_4}$  is determined by  $\lambda_{1,[i]}$  and  $\lambda_{2,[i]}^{\prime}$ , which are in turn determined by how  $2\varepsilon_{1}$  and  $2\varepsilon_{2}$  are allocated to  $(\varepsilon_{1,[1]},\dots ,\varepsilon_{1,[n]})$  and  $(\varepsilon_{2,[1]},\dots ,\varepsilon_{2,[n]})$ , respectively. We prove that an "extreme allocation" can maximize the objective function. In other words, the objective function in Equation (6) takes its maximum when  $\varepsilon_{1,[1]} = 2\varepsilon_{1},\varepsilon_{2,[1]} = 2\varepsilon_{2}$ , and  $\varepsilon_{1,[i]} = \varepsilon_{2,[i]} = 0$  for  $1 < i\leq n$ . In the proof, we use a key Lemma G.5 to deal with the 2-dimensional case ( $n = 2$ ). Finally, we extend to arbitrary dimensional cases.
4. Dealing with 2-dimensional case. The proof of Lemma G.5 is the most technical part in this work. In the proof, concentrating  $\varepsilon_{1}$  and  $\varepsilon_{2}$  is much harder than that in the last section for Theorem 1.  $f(x) = x - \log x$  is a transcendental function whose inverse function is expressed by Lambert  $W$  function. This makes even a 2-dimensional case of problem  $P_{4}$  hard to solve. Our proof of Lemma G.5 is like coordinate descent but much difficult. We show that, for any fixed "non-extreme allocation" of  $2\varepsilon_{1}$  (i.e.,  $(\varepsilon_{1,[1]},\varepsilon_{1,[2]})$  where  $\varepsilon_{1,[2]} > 0$ ), there exists
a "more extreme" allocation of  $2\varepsilon_{2}$  (i.e.,  $(\varepsilon_{2,[1]},\varepsilon_{2,[2]})$  where  $\varepsilon_{2,[2]} > 0$ ) that maximizes the objective function. Note that it is hard to state that  $(\varepsilon_{2,[1]},\varepsilon_{2,[2]})$  is "more extreme" than  $(\varepsilon_{1,[1]},\varepsilon_{1,[2]})$  because  $\varepsilon_{1} \neq \varepsilon_{2}$ . Then we fix  $(\varepsilon_{2,[1]},\varepsilon_{2,[2]})$  and find a "more extreme" allocation of  $2\varepsilon_{1}(\varepsilon_{1,[1]}^{\prime},\varepsilon_{1,[2]}^{\prime})$  to lift the objective function further. Using these iterations, we can construct an infinite sequence of allocations whose limitation is an "extreme" one (i.e.,  $\varepsilon_{1,[1]} = 2\varepsilon_{1},\varepsilon_{2,[1]} = 2\varepsilon_{2}$ ). Then we prove the extreme allocation can make the objective function reach its supremum. In Appendix G we present the key Lemma G.5 and its proof. We also give its proof sketch before the long proof. Essentially, Lemma G.5 plays the most vital role in eliminating the dimension  $n$  in case of  $n = 2$  from the bound in 2-dimensional case. Finally, we will make the bound in high-dimensional problem dimension-free as well.
In summary, we apply a linear transformation to simplify the problem. Then we solve the simplified problem in the following Lemma 2, accomplishing the above steps  $2 \sim 4$ . Theorem 4 can be seen as the generalization of Lemma 2.
Lemma 2 For any two  $n$ -dimensional Gaussian distributions  $\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1)$  and  $\mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)$  such that  $KL(\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1) || \mathcal{N}(0, I)) \leq \varepsilon_1$ ,  $KL(\mathcal{N}(0, I) || \mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)) \leq \varepsilon_2$  ( $\varepsilon_1, \varepsilon_2 \geq 0$ ),
$$
\begin{array}{l} K L \left(\mathcal {N} \left(\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1}\right) \| \mathcal {N} \left(\boldsymbol {\mu} _ {2}, \boldsymbol {\Sigma} _ {2}\right)\right) <   \varepsilon_ {1} + \varepsilon_ {2} + \frac {1}{2} \left(W _ {- 1} \left(- e ^ {- (1 + 2 \varepsilon_ {1})}\right) W _ {- 1} \left(- e ^ {- (1 + 2 \varepsilon_ {2})}\right) + W _ {- 1} \left(- e ^ {- (1 + 2 \varepsilon_ {1})}\right) \right. \\ \left. + W _ {- 1} \left(- e ^ {- (1 + 2 \varepsilon_ {2})}\right) + 1 - W _ {- 1} \left(- e ^ {- (1 + 2 \varepsilon_ {2})}\right) \left(\sqrt {2 \varepsilon_ {1}} + \sqrt {\frac {2 \varepsilon_ {2}}{- W _ {0} \left(- e ^ {- (1 + 2 \varepsilon_ {2})}\right)}}\right) ^ {2}\right) \\ \end{array}
$$
Proof 4 Please see Appendix H for details.
Remark 3 The bound in Lemma 2 is dimension-free and becomes  $O$  when  $\varepsilon_1 = \varepsilon_2 = 0$ .
Similarly, we can expand Lambert  $W$  function by series [19, 14] and simplify the bound in Theorem 4 as follows [36].
Theorem 5 For any three  $n$ -dimensional Gaussian distributions  $\mathcal{N}(\pmb{\mu}_i, \pmb{\Sigma}_i) (i \in \{1, 2, 3\})$  such that  $KL(\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1) || \mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)) \leq \varepsilon_1$  and  $KL(\mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2) || \mathcal{N}(\pmb{\mu}_3, \pmb{\Sigma}_3)) \leq \varepsilon_2$  for small  $\varepsilon_1, \varepsilon_2 \geq 0$ ,
$$
K L \left(\mathcal {N} \left(\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1}\right) | | \mathcal {N} \left(\boldsymbol {\mu} _ {3}, \boldsymbol {\Sigma} _ {3}\right)\right) <   3 \varepsilon_ {1} + 3 \varepsilon_ {2} + 2 \sqrt {\varepsilon_ {1} \varepsilon_ {2}} + o \left(\varepsilon_ {1}\right) + o \left(\varepsilon_ {2}\right)
$$
Proof 5 Please see Appendix J for details.
Finally, in the proof of Theorem 4, we use invertible linear transformation to convert  $\mathcal{N}_2$  to standard Gaussian while preserving KL divergence. This proof also applies to the case when  $\mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)$  is fixed. Therefore, we obtain the following corollary.
Corollary 2 Theorem 4 and 5 hold when  $\mathcal{N}(\mu_2,\Sigma_2)$  is fixed.
Remark 4 Our theorem is different from existing generalized Pythagoras inequalities satisfied by KL divergence. Please see Appendix K.1 for more discussion.
6 Discussion and Applications
In our theorems, we allow all parameters are unknown or one Gaussian is fixed. Therefore, our theorems are suitable for problems where the parameters can vary. This is common in deep learning where the parameters are learned from data. In this section, we discuss motivation application and other applications ranging from anomaly detection to reinforcement learning to sample complexity research.
6.1 Motivation Application: Anomaly Detection with Flow-based Model
The research problems in this paper are motivated by our research on deep anomaly detection using flow-based model [57]  $^{3}$ . Flow-based model assigns higher likelihoods to Out-of-Distribution (OOD) data than In-Distribution (ID) data (i.e., training data) [39, 51, 13, 52, 40, 30]. For example, Glow [29] assigns higher likelihoods for SVHN when trained on CIFAR-10. Furthermore, we cannot sample OOD data from the model. Existing explanation is based on the discrepancy of typical set and high probability density regions of model distribution [40]. Such typicality-based explanation and OOD detection method fail when OOD data has coinciding likelihoods with ID data [40].
We focus on two research problems in this context. (1) why we cannot sample OOD data from flow-based model with prior regardless of when OOD data have higher, lower, or coinciding likelihoods? (2) How to detect OOD data using flow-based model? We investigate these problems from a statistical divergence perspective. Let  $z = f(x)$  be the flow-based model mapping data  $x$  in data space to representation  $z$  in latent space. Suppose the prior distribution  $p_{Z}^{r}$  is the most commonly used Gaussian distribution. Let  $X_{1} \sim p_{X}(x)$ ,  $X_{2} \sim q_{X}(x)$  be the distributions of ID and OOD datasets, respectively.  $Z_{1} = f(X_{1}) \sim p_{Z}(z)$ ,  $Z_{2} = f(X_{2}) \sim q_{Z}(z)$  be the distributions of representations of ID and OOD datasets, respectively. Let  $p_{X}^{r}$  be the model induced distribution such that  $Z_{r} \sim p_{Z}^{r}$  and  $X_{r} = f^{-1}(Z_{r}) \sim p_{X}^{r}$ . Flow-based model is usually trained by maximum likelihood estimation, which is equal to minimizing forward KL divergence  $KL(p_{X}||p_{X}^{r})$  [44, 23]. We conduct generalized Shapiro-Wilk test for multivariate normality on representations. As shown in the original Table C.3 in [57] (also in supplementary material),  $p_{Z}$  is Gaussian-like for all datasets.  $q_{Z}$  is also Gaussian-like for OOD datasets with higher or coinciding likelihoods except for just one case. These results allow us to approximate  $p_{Z}$  and  $q_{Z}$  with Gaussians when possible.
The theorems proved in this paper provide solid theoretical guarantee for our analysis and algorithm. On one hand, flow-based model preserves KL divergence (see Proposition D.1 in Appendix D), so  $KL(p_Z||p_Z^r)$  which equals  $KL(p_X||p_X^r)$  is minimized. According to the approximate symmetry of small KL divergence (Theorem 1), we can know  $KL(p_Z^r ||p_Z)$  is small too. So we can assume  $p_Z^r\approx p_Z$  when  $KL(p_{Z}||p_{Z}^{r})$  is sufficiently small. On the other hand, we can also assume that the distributions of ID and OOD data are far from each other. This implies that  $KL(p_{X}||q_{X})$  equaling  $KL(p_{Z}||q_{Z})$  can be any large. This implies  $KL(p_Z^r ||q_Z)\approx KL(p_Z||q_Z)$  is large too. Specially, when  $q_{Z}$  is Gaussian-like, we can apply the relaxed triangle inequality (Theorem 4) and infer that  $KL(p_{Z}^{r}||q_{Z})$  must be large. Note that, when  $q_{Z}$  is not Gaussian-like, we can still apply the theorems presented in this paper to perform analysis on the lower bound of KL divergence (see original Theorem 5 in [57]). Overall, the large KL divergence between  $p_Z^r$  and  $q_{Z}$  reveals why we cannot sample OOD data from flow-based model with prior. It is also notable that flow-based model constructs diffeomorphism between data and latent space with thousands of dimensions. Thus, it is critical that the bounds found in this paper are dimension-free. Furthermore, we decompose the KL divergence further into group-wise KL divergence and mutual information. Based on these analysis, we propose an unified OOD detection algorithm KLODS both for group (GAD) and point-wise (PAD) anomaly detection. We conduct extensive experiments to compare our method with 13 baseline methods including  $t$  -test, KS-test, MMD [24], KSD [35], Annulus Method [13], typicality test [40], the state-of-the-art (SOTA) GAD method GOD2KS [28], input complexity compensated likelihood [50], last-scale likelihood [49], ODIN [34], Joint confidence loss [33], and DoSE [37]. Experimental results demonstrate the superiority of our method. For example, as shown in Table 2, our method outperforms the SOTA group-wise anomaly detection method GOD2KS on flow-based model by  $9.1\%$  AUROC. Our method also outperforms the SOTA point-wise anomaly detection method DoSE with Glow by  $5.2\%$  AUROC. More details of the algorithm and experimental results on both group and point-wise anomaly detection can be referred to [57].
6.2 Applications of Approximate Symmetry of Small KL divergence
Theorem 1 can be also applied widely in deep reinforcement learning and sample complexity research. In many contexts, researchers are hindered by the asymmetry of KL divergence. Theorem 1 on the approximate symmetry of KL divergence between Gaussians brings the following convenience to us.
1. Minimizing one of forward and reverse KL divergences also bounds another.
2. We can exchange forward and reverse KL divergences for small  $\varepsilon$ .
<sup>3</sup>We append an anonymous version of our work [57] in the supplementary material for convenience.
Table 2: Group-wise anomaly detection Results (AUROC and AUPR in percentage) of our method KLODS and the SOTA method GOD2KS on Glow with batch sizes 5 and 10. We run our method for 5 times. Results of GOD2KS are referred from [28]. The higher the better.
ID	OOD	batch size=5	batch size=10
KLODS	GOD2KS	KLODS	GOD2KS
AUROC	AUPR	AUROC	AUPR	AUROC	AUPR	AUROC	AUPR
FashionMNIST	MNIST	99.8±0.0	99.8±0.0	98	98	100.0±0.0	100.0±0.0	100	100
KMNIST	99.9±0.0	99.9±0.0	97	96	100.0±0.0	100.0±0.0	100	100
Omniglot	100.0±0.0	100.0±0.0	100	100	100.0±0.0	100.0±0.0	100	100
SVHN	CelebA	100.0±0.0	100.0±0.0	100	99	100.0±0.0	100.0±0.0	100	100
CIFAR-10	100.0±0.0	100.0±0.0	92	84	100.0±0.0	100.0±0.0	99	98
CIFAR-100	100.0±0.0	100.0±0.0	93	86	100.0±0.0	100.0±0.0	99	98
LSUN	100.0±0.0	100.0±0.0	99	98	100.0±0.0	100.0±0.0	100.0	100.0
CIFAR-10	CelebA	99.2±0.1	99.4±0.1	86	92	100.0±0.0	100.0±0.0	96	98
SVHN	97.6±0.2	97.8±0.2	96	98	99.8±0.0	99.8±0.0	100	100
LSUN	100.0±0.0	100.0±0.0	60	58	100.0±0.0	100.0±0.0	58	56
CelebA	CIFAR-10	99.6±0.0	99.6±0.0	84	73	100.0±0.0	100.0±0.0	94	91
CIFAR-100	99.8±0.0	99.8±0.0	82	71	100.0±0.0	100.0±0.0	94	90
SVHN	100.0±0.0	100.0±0.0	97	98	100.0±0.0	100.0±0.0	100	100
LSUN	100.0±0.0	100.0±0.0	85	75	100.0±0.0	100.0±0.0	96	92
average	99.7	99.7	90.6	87.6	100.0	100.0	95.4	94.5
We summarize the applications of Theorem 1 briefly in the following. The details of these applications are discussed in Appendix L.1.
Providing Theoretical Guarantee for Continuous Gaussian Policy in Reinforcement Learning. In [38], Nair et al. propose AWAC method to accelerate online reinforcement learning with offline datasets. They obtain theoretical guarantee in offline reinforcement learning for discrete policies. Theorem 1 can extend their guarantee to continuous Gaussian policy.
Bringing New Insights to Existing Reinforcement Learning Algorithm. In [1], Abdelmaleki et al. propose the MPO algorithm for reinforcement learning. They use Expectation-Maximization (EM) to solve control problems and use constraints on KL terms in both E and M-steps. Theorem 1 can eliminate such a difference for continuous Gaussian policies.
Bridging Research on Sample Complexity of Learning Gaussian Distribution. Theorem 1 can bridge existing research on sample complexity of Gaussian distribution. Researchers have proposed algorithms for learning a multivariate Gaussian distribution with error bounds in forward and reverse KL divergence separately so far [6, 8]. Theorem 1 can eliminate the difference between forward and reverse KL divergence in this scenario.
6.3 Application of Relaxed Triangle Inequality
Extending One-step Safety Guarantee to Multiple Steps in Reinforcement Learning. The relaxed triangle inequality (Theorem 5) has been applied in safe reinforcement learning. Liu et al. propose an Expectation-Maximization style approach for learning safe policy in reinforcement learning [36]. They utilize our relaxed triangle inequality to extend one-step robustness guarantee to multiple steps. In the original Proposition 4 in [36], they simplify the bound in Theorem 4 in case  $\varepsilon_1 = \varepsilon_2$ . Please see Appendix L.2 for details.
In this paper, we research the properties of KL divergence between Gaussians. First, we find the supremum of reverse KL divergence  $KL(\mathcal{N}_2||\mathcal{N}_1)$  if the forward KL divergence  $KL(\mathcal{N}_1||\mathcal{N}_2)\leq \varepsilon$ $(\varepsilon >0)$ . This conclusion quantifies the approximate symmetry of small KL divergence between Gaussians. We also find the infimum of  $KL(\mathcal{N}_2||\mathcal{N}_1)$  if  $KL(\mathcal{N}_1||\mathcal{N}_2)\geq M(M > 0)$ . We give the conditions when the supremum and infimum can be attained. Second, we find a bound for  $KL(\mathcal{N}_1||\mathcal{N}_3)$  when  $KL(\mathcal{N}_1||\mathcal{N}_2)$  and  $KL(\mathcal{N}_2||\mathcal{N}_3)$  are bounded. This indicates that KL divergence between Gaussians follows a relaxed triangle inequality. All the bounds in this paper are independent of the dimension of distributions. Finally, we discuss the applications of our theorems in deep anomaly detection, reinforcement learning, and sample complexity research.
This work is supported by the National Natural Science Foundation of China (Grant No. 62002107, 62372162, 62032024, 62172429, 61872371, 62202150, 62102442) and National Key Research and Development Program of China (Grant No. 2021ZD40300).
[1] Abbas Abdelmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Martin A. Riedmiller. Maximum a posteriori policy optimisation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[2] Karim T Abou-Moustafa and Frank P Ferrie. A note on metric properties for some divergence measures: The gaussian case. In Asian Conference on Machine Learning, pages 1-15. PMLR, 2012.
[3] Rohit Agrawal, Yi-Hsiu Chen, Thibaut Horel, and Salil Vadhan. Unifying Computational Entropies via Kullback-Leibler Divergence. In Alexandra Boldyreva and Daniele Micciancio, editors, Advances in Cryptology - CRYPTO 2019, pages 831-858, Cham, 2019. Springer International Publishing.
[4] Rohit Agrawal and Thibaut Horel. Optimal bounds between f-divergences and integral probability metrics. Journal of Machine Learning Research, 22(128):1-59, 2021.
[5] Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological), 28(1):131-142, 1966.
[6] Hassan Ashtiani, Shai Ben-David, Nicholas J. A. Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Near-optimal sample complexity bounds for robust learning of gaussian mixtures via compression schemes. J. ACM, 67(6), oct 2020.
[7] Arindam Banerjee, Srujana Merugu, Inderjit S. Dhillon, and Joydeep Ghosh. Clustering with Bregman divergences. Journal of Machine Learning Research, 6(58):1705-1749, 2005.
[8] Arnab Bhattacharyya, Davin Choo, Rishikesh Gajjala, Sutanu Gayen, and Yuhao Wang. Learning sparse fixed-structure gaussian bayesian networks. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 9400–9429. PMLR, 28–30 Mar 2022.
[9] Jeremiah Birrell, Paul Dupuis, Markos A. Katsoulakis, Yannis Pantazis, and Luc Rey-Bellet.  $(f,\Gamma)$ -divergences: Interpolating between  $f$ -divergences and integral probability metrics. Journal of Machine Learning Research, 23(39):1-70, 2022.
[10] Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg, 2006.
[11] N. Bouhlel and A. Dziri. Kullback-Leibler divergence between multivariate generalized gaussian distributions. IEEE Signal Processing Letters, 26(7):1021-1025, 2019.
[12] Lev M Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200-217, 1967.
[13] Hyunsun Choi and Eric Jang. WAIC, but why?: Generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392, 2018.
[14] Robert M Corless, Gaston H Gonnet, David EG Hare, David J Jeffrey, and Donald E Knuth. On the Lambert W function. Advances in Computational mathematics, 5(1):329-359, 1996.
[51] Alireza Shafaei, Mark Schmidt, and James J Little. Does your model know the digit 6 is not a cat? a less biased evaluation of "outlier" detectors. arXiv preprint arXiv:1809.04729, 2018.
[52] Vít Škářa, Tomáš Pevný, and Václav Šmídl. Are generative deep models for novelty detection truly better? KDD Workshop on Outlier Detection De-Constructed (ODD v5.0), 2018.
[53] W. Stummer and I. Vajda. On Bregman distances and divergences of probability measures. IEEE Transactions on Information Theory, 58(3):1277-1288, mar 2012.
[54] Tim van Erven and Peter Harremos. Rényi divergence and Kullback-Leibler divergence. IEEE Transactions on Information Theory, 60(7):3797-3820, 2014.
[55] Rui F. Vigelis, Luiza H. F. De Andrade, and Charles C. Cavalcante. Properties of a generalized divergence related to Tsallis generalized divergence. IEEE Transactions on Information Theory, 66(5):2891-2897, 2020.
[56] Pengfei Yang and Biao Chen. Robust Kullback-Leibler divergence and universal hypothesis testing for continuous distributions. IEEE Transactions on Information Theory, 65(4):2360-2373, 2019.
[57] Yufeng Zhang, Jialu Pan, Wanwei Liu, Zhenbang Chen, Kenli Li, Ji Wang, Zhiming Liu, and Hongmei Wei. Kullback-leibler divergence-based out-of-distribution detection with flow-based generative models. IEEE Transactions on Knowledge and Data Engineering, pages 1-14, 2023.