SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yufeng Zhang, Jialu Pan * Kenli Li
College of Computer Science and Electronic Engineering
Hunan University, Changsha, China
{yufengzhang, jialupan, lkl}@hnu.edu.cn
Wanwei Liu, Zhenbang Chen, Xinwang Liu
College of Computer
Key Laboratory of Software Engineering for Complex Systems
National University of Defense Technology, Changsha, China
{wwliu，zbchen，xinwangliu，wj}@nudt.edu.cn
Ji Wang
State Key Laboratory for High Performance Computing
Key laboratory of Software Engineering for Complex Systems
wj@nudt.edu.cn
Kullback-Leibler (KL) divergence is one of the most important measures to calculate the difference between probability distributions. In this paper, we theoretically study several properties of KL divergence between multivariate Gaussian distributions. Firstly, for any two  $n$ -dimensional Gaussian distributions  $\mathcal{N}_1$  and  $\mathcal{N}_2$ , we prove that when  $KL(\mathcal{N}_2||\mathcal{N}_1)\leq \varepsilon$  ( $\varepsilon >0$ ) the supremum of  $KL(\mathcal{N}_1||\mathcal{N}_2)$  is  $(1 / 2)\left((-W_0(-e^{-(1 + 2\varepsilon)}))^{-1} + \log (-W_0(-e^{-(1 + 2\varepsilon)})) - 1\right)$ , where  $W_{0}$  is the principal branch of Lambert  $W$  function. For small  $\varepsilon$ , the supremum is  $\varepsilon +2\varepsilon^{1.5} + O(\varepsilon^2)$ . This quantifies the approximate symmetry of small KL divergence between Gaussian distributions. We further derive the infimum of  $KL(\mathcal{N}_1||\mathcal{N}_2)$  when  $KL(\mathcal{N}_2||\mathcal{N}_1)\geq M$  ( $M > 0$ ). We give the conditions when the supremum and infimum can be attained. Secondly, for any three  $n$ -dimensional Gaussian distributions  $\mathcal{N}_1,\mathcal{N}_2$ , and  $\mathcal{N}_3$ , we theoretically show that an upper bound of  $KL(\mathcal{N}_1||\mathcal{N}_3)$  is  $3\varepsilon_1 + 3\varepsilon_2 + 2\sqrt{\varepsilon_1\varepsilon_2} +o(\varepsilon_1) + o(\varepsilon_2)$  when  $KL(\mathcal{N}_1||\mathcal{N}_2)\leq \varepsilon_1$  and  $KL(\mathcal{N}_2||\mathcal{N}_3)\leq \varepsilon_2$  ( $\varepsilon_1,\varepsilon_2\geq 0$ ). This reveals that KL divergence between Gaussian distributions follows a relaxed triangle inequality. Note that, all these bounds in the theorems presented in this work are independent of the dimension  $n$ . Finally, we discuss several applications of our theories in deep learning, reinforcement learning, and sample complexity research.
A statistical divergence measures the "distance" between probability distributions. Let  $X$  be a space of probability distributions with the same support. A statistical divergence  $D: X \times X \to \mathbb{R}^{+}$  ( $\mathbb{R}^{+}$ )
*Jialu Pan is the corresponding author.
is the set of non-negative real numbers) should satisfy (a) non-negativity:  $D(p,q) \geq 0$  and (b) identity of indiscernibles:  $D(p,p) = 0$ , where  $p, q$  are probability densities. Another stricter concept, statistical distance, also measures the distance between probability distributions. A statistical distance should satisfy two additional properties, including (c) symmetry:  $D(p,q) = D(q,p)$  and (d) triangle inequality:  $D(p,q) \leq D(p,g) + D(g,q)$ , where  $p, q$  and  $g$  are probability densities.
Kullback-Leibler (KL) divergence, also referred to as relative entropy [31], is applied broadly in many fields, such as machine learning [10, 23], information theory [15], and statistics [45]. The KL divergence between two continuous probability densities  $p(x)$  and  $q(x)$  is defined as  $KL(p(x)||q(x)) = \int p(x)\log \frac{p(x)}{q(x)}\mathrm{d}x$ . KL divergence is not a proper distance [31]. First, KL divergence is not symmetric. It might happen that the forward KL divergence $^2$ $KL(p||q)$  is very small but the reverse KL divergence  $KL^{*}(p||q) = KL(q||p)$  is very large. Second, KL divergence does not satisfy the triangle inequality. This hinders the application of KL divergence in many contexts.
KL divergence also has connections with other information measures. For example, by taking the second-order Taylor expansion, KL divergence can be approximated with fisher information matrix [31]. Therefore, KL divergence is locally approximately symmetric when two distributions are close to each other.
Meanwhile, Gaussian distribution is one of the most important distributions and is central to statistics. It is also pervasive in many fields. The probability density function of an  $n$ -dimensional Gaussian distribution is  $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma}) = ((2\pi)^{n/2} |\boldsymbol{\Sigma}|^{1/2})^{-1} \exp \left(- (1/2) (\boldsymbol{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\boldsymbol{x} - \boldsymbol{\mu})\right)$ . Here  $\boldsymbol{\mu} \in \mathbb{R}^n$  is the mean and  $\boldsymbol{\Sigma} \in S_{++}^n$  is the covariance matrix, where  $S_{++}^n$  is the space of symmetric positive definite  $n \times n$  matrices. Gaussian distribution is the basis for more complicated distributions such as Gaussian Mixture Model [10]. In this paper, we refer to Gaussian distribution as Gaussian for brevity.
The KL divergence between two  $n$ -dimensional Gaussians  $\mathcal{N}_1, \mathcal{N}_2$  has the following closed form [45]
$$
K L \left(\mathcal {N} _ {1} \left(\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1}\right) \mid \mid \mathcal {N} _ {2} \left(\boldsymbol {\mu} _ {2}, \boldsymbol {\Sigma} _ {2}\right)\right) = \frac {1}{2} \left(\log \frac {\left| \boldsymbol {\Sigma} _ {2} \right|}{\left| \boldsymbol {\Sigma} _ {1} \right|} + \operatorname {T r} \left(\boldsymbol {\Sigma} _ {2} ^ {- 1} \boldsymbol {\Sigma} _ {1}\right) + \left(\boldsymbol {\mu} _ {2} - \boldsymbol {\mu} _ {1}\right) ^ {\top} \boldsymbol {\Sigma} _ {2} ^ {- 1} \left(\boldsymbol {\mu} _ {2} - \boldsymbol {\mu} _ {1}\right) - n\right) \tag {1}
$$
where the logarithm is taken to base  $e$  and  $\mathrm{Tr}$  is the trace of matrix. Like many other distributions, KL divergence between Gaussians is neither symmetric nor satisfies the triangle inequality.
In this work, we investigate the following two research problems, which are motivated by our research on out-of-distribution detection with flow-based model [57].
1. How to quantify the relation between forward and reverse KL divergences between Gaussian distributions?
2. Does the KL divergence between Gaussian distributions satisfy some property similar to triangle inequality?
Contributions. The contributions of this work are as follows. Let  $\mathcal{N}_i(\boldsymbol{\mu}_i, \boldsymbol{\Sigma}_i) (i \in \{1, 2, 3\})$  be any three  $n$ -dimensional Gaussians.
1. We prove that when  $KL(\mathcal{N}_1||\mathcal{N}_2)\leq \varepsilon$  ( $\varepsilon \geq 0$ ) the supremum of  $KL(\mathcal{N}_2||\mathcal{N}_1)$  is  $(1/2)\left((-W_0(-e^{-(1 + 2\varepsilon)}))^{-1} + \log (-W_0(-e^{-(1 + 2\varepsilon)})) - 1\right)$ , where  $W_0$  is the principal branch of Lambert  $W$  function. We give the conditions when the supremum can be attained. For small  $\varepsilon$ , the supremum is  $\varepsilon + 2\varepsilon^{1.5} + O(\varepsilon^2)$ . This quantifies the approximate symmetry of small KL divergence between Gaussians.
2. We find the infimum of  $KL(\mathcal{N}_1||\mathcal{N}_2)$  if  $KL(\mathcal{N}_2||\mathcal{N}_1)\geq M$ $(M > 0)$ . We also give the conditions when the infimum can be attained.
3. We find an upper bound of  $KL(\mathcal{N}_1||\mathcal{N}_3)$  if  $KL(\mathcal{N}_1||\mathcal{N}_2)\leq \varepsilon_1$  and  $KL(\mathcal{N}_2||\mathcal{N}_3)\leq \varepsilon_2$  for  $\varepsilon_{1},\varepsilon_{2}\geq 0$ . For small  $\varepsilon_{1}$  and  $\varepsilon_{2}$ , the upper bound is  $3\varepsilon_{1} + 3\varepsilon_{2} + 2\sqrt{\varepsilon_{1}\varepsilon_{2}} +o(\varepsilon_{1}) + o(\varepsilon_{2})$ . This indicates that KL divergence between Gaussians follows a relaxed triangle inequality.
4. All the bounds in our theorems are independent of the dimension  $n$ . This is a critical property in contexts where dimensionality has a fundamental impact.
5. The theorems proved in this paper can extend the applications of KL divergence in many contexts. We discuss the motivation application in out-of-distribution detection with flow-based model and multiple applications in reinforcement learning and sample complexity research.
2We can choose to call  $KL(p||q)$  or  $KL(q||p)$  as forward KL divergence.
The rest part of this paper is organized as follows. In Section 2 we discuss related work. In Section 3 we prepare lemmas and notations. In Section 4 we investigate the supremum (infimum) of reverse KL divergence between Gaussians when forward KL divergence is bounded. In Section 5 we investigate the relaxed triangle inequality of KL divergence between Gaussians. In Section 6, we discuss applications. Finally, we conclude in Section 7. We put long proofs in Appendix.
KL divergence has a wide range of applications [15, 10, 23, 45, 20, 25, 3]. Researchers have investigated KL divergence between many distributions, including Markov sources [47], GMM models [18, 27], multivariate generalized Gaussians [11], discrete normal distributions [42], etc. In [45], a bound of KL divergence between Gaussians is given. As far as we know, no related work focuses on the similar properties of KL divergence between Gaussians as in this paper.
KL divergence is one member of more general divergences such as Bregman divergence [12, 7, 21, 53],  $f$ -divergence [5, 45, 4], Rényi divergence [48, 45, 54], and  $(f,\Gamma)$ -divergence [9]. Bregman divergence defines a class of divergences [7] in vector space. KL divergence between multinomial distributions is a special form of Bregman divergence when the convex function for Bregman divergence is chosen as  $\sum_{i=1}^{n} p_i \log p_i$ , where  $p_i \geq 0$  define a multinomial distribution. Frigyik et. al. [21] extends vector Bregman divergence to functional Bregman divergence in  $L^p$ . Similarly, KL divergence is a special form of functional Bregman divergene. (Functional) Bregman divergence also satisfies generalized Pythagoras theorem [7, 21].
The asymmetry of KL divergence has restricted the application of KL divergence in practical applications. Many other divergences have been investigated [46, 16, 2, 43, 26, 17, 7, 22, 56, 55]. Pardo gives a comprehensive survey on a wide range of statistical divergences in his book [45].
3 Lemmas and Notations
Table 1: Notations.
f(x)	x - log x (x ∈ R++)
W(x)	the Lambert W function
W0(x)	the principal branch (branch 0) of W(x)
W-1(x)	the branch -1 of W(x)
w1(t)	the smaller solution of f(x) = 1 + t (t ≥ 0)
w2(t)	the larger solution of f(x) = 1 + t (t ≥ 0)
f(x1, ...,xn)	Σni=1n f(xi)
λ	the eigenvalue of matrix
N(0,I)	standard Gaussian distribution, dimension n is eliminated for brevity
We introduce the following Lambert  $W$  function before our theoretical results.
Definition 1 Lambert W Function[32, 14]. The inverse function of function  $y = xe^{x}$  is called Lambert W function  $y = W(x)$ .
When  $x \in \mathbb{R}$ ,  $W$  is a multivalued function with two branches  $W_0, W_{-1}$ , where  $W_0$  is the principal branch (also called branch 0) and  $W_{-1}$  is the branch -1. Figure A.1 in Appendix A shows the graph of  $W_0$  and  $W_{-1}$ .
**Lemmas.** Function  $f(x) = x - \log x$  ( $x \in \mathbb{R}^{++}$ ) ( $\mathbb{R}^{++}$  is the set of positive real numbers) lies in the core of our problems. In Lemma B.1 in Appendix B, we prove several properties of  $f(x)$ . We show that the inverse function of  $f$  is  $f^{-1}(x) = -W(-e^{-x})$  ( $x \geq 1$ ). Equation  $f(x) = x - \log x = 1 + t$  ( $t \geq 0$ ) has two solutions  $w_{1}(t) = -W_{0}(-e^{-(1 + t)}) \in (0,1]$  and  $w_{2}(t) = -W_{-1}(-e^{-(1 + t)}) \in [1, +\infty)$ . We treat  $w_{1}(t), w_{2}(t)$  as functions of  $t$ .
Table 1 summarizes some notations used in this paper. Please see Table A.1 in Appendix A for a full list of notations.
4 Bounds of Forward and Reverse KL Divergence Between Gaussians
In this section, we investigate the relation between forward and reverse KL divergences between Gaussians. These conclusions quantify the approximate symmetry of small KL divergence between Gaussians.
4.1 Supremum of Reverse KL Divergence Between Gaussians
The following Theorem 1 gives the supremum of reverse KL divergence when forward KL divergence is bounded by a number  $\varepsilon$ .
Theorem 1 For any two  $n$ -dimensional Gaussian distributions  $\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1)$  and  $\mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)$ , if  $KL(\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1) || \mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)) \leq \varepsilon$  ( $\varepsilon \geq 0$ ), then
$$
K L (\mathcal {N} (\boldsymbol {\mu} _ {2}, \boldsymbol {\Sigma} _ {2}) | | \mathcal {N} (\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1})) \leq \frac {1}{2} \left(\frac {1}{- W _ {0} (- e ^ {- (1 + 2 \varepsilon)})} - \log \frac {1}{- W _ {0} (- e ^ {- (1 + 2 \varepsilon)})} - 1\right)
$$
The supremum is attained when the following two conditions hold.
(1) There exists only one eigenvalue  $\lambda_{j}$  of  $B_{2}^{-1}\pmb{\Sigma}_{1}(B_{2}^{-1})^{\top}$  or  $B_{1}^{-1}\pmb{\Sigma}_{2}(B_{1}^{-1})^{\top}$  equal to  $-W_{0}(-e^{-(1 + 2\varepsilon)})$  and all other eigenvalues  $\lambda_{i}$  ( $i\neq j$ ) are equal to 1, where  $B_{1} = P_{1}D_{1}^{1 / 2}$ ,  $P_{1}$  is an orthogonal matrix whose columns are the eigenvectors of  $\pmb{\Sigma}_{1}$ ,  $D_{1} = \text{diag}(\lambda_{1},\dots,\lambda_{n})$  whose diagonal elements are the corresponding eigenvalues,  $B_{2}$  is defined in the similar way as  $B_{1}$  except on  $\pmb{\Sigma}_{2}$ .
(2)  $\pmb{\mu}_1 = \pmb{\mu}_2$
Overview of Proof of Theorem 1. Please see Appendix D for the full proof. Here we give the overview of the proof. Theorem 1 can be proved by solving the following optimization problem  $P_{1}$  analytically.
$$
\boldsymbol {P} _ {1}: \text {m a x i m i z e} K L (\mathcal {N} \left(\boldsymbol {\mu} _ {2}, \boldsymbol {\Sigma} _ {2}\right) | | \mathcal {N} \left(\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1}\right)) \tag {2}
$$
$$
s. t. K L \left(\mathcal {N} \left(\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1}\right) \mid \mid \mathcal {N} \left(\boldsymbol {\mu} _ {2}, \boldsymbol {\Sigma} _ {2}\right)\right) \leq \varepsilon \tag {3}
$$
The proof consists of the following several steps.
1. Invertible linear transformation. We apply a linear transformation to  $\mathcal{N}(\mu_1, \Sigma_1)$  and  $\mathcal{N}(\mu_2, \Sigma_2)$  such that one of them is transformed to a standard Gaussian. In this way, we can simplify the objective function and the constraint in  $P_1$ . Note that diffeomorphism preserves KL divergence [41] (see Proposition D.1 in Appendix D). In the end, we use the inverse linear transformation to transform  $\mathcal{N}(\mu_1, \Sigma_1)$  and  $\mathcal{N}(\mu_2, \Sigma_2)$  back.
2. Reducing to new optimization problem. We reduce  $P_{1}$  to the following core optimization problem  $P_{2}$ .
$$
P _ {2}: \text {m a x i m i z e} \bar {f} \left(\frac {1}{x _ {1}}, \dots , \frac {1}{x _ {n}}\right) \tag {4}
$$
$$
s. t. \bar {f} (x _ {1}, \dots , x _ {n}) \leq n + \varepsilon^ {\prime} \tag {5}
$$
where  $\bar{f} (x_{1},\ldots ,x_{n}) = \sum_{i = 1}^{n}f(x_{i}) = \sum_{i = 1}^{n}x_{i} - \log x_{i}$ $(x_{i}\in (0,\infty))$
3. Investigating  $f(x)$ .  $f(x)$  lies in the core of the problem. We have proven several properties of  $f(x)$  in Section 3. A fundamental property of  $f(x)$  (stated in Lemma B.1a) is that  $f(x)$  is strictly convex and takes its minimum value 1 at  $x = 1$ . These lemmas allow us to conduct further analysis in other parts of this paper.
4. Allocating  $\varepsilon'$ . In problem  $P_{2}$ , the supremum of  $\bar{f}(\frac{1}{x_1}, \ldots, \frac{1}{x_n})$  is affected by the domain of each dimension, which is in turn determined by how  $\varepsilon'$  is allocated to these dimensions. We call  $(\varepsilon_1, \dots, \varepsilon_n)$  where  $\sum_{i=1}^{n} \varepsilon_i = \varepsilon'$  as an allocation. We prove that  $\bar{f}(\frac{1}{x_1}, \ldots, \frac{1}{x_n})$  takes its maximum when  $\varepsilon'$  is allocated to only one dimension (i.e., an "extreme" allocation). In other words, there exists one  $\varepsilon_j = \varepsilon'$  and  $\varepsilon_i = 0$  for all  $i \neq j$ . The key is to prove the convexity of function  $\Delta(\varepsilon) = f\left(\frac{1}{w_1(\varepsilon)}\right) - f(w_1(\varepsilon))$ , where  $w_1(\varepsilon) = -W_0(-e^{-(1 + \varepsilon)})$ .
In summary, we use a linear transformation to simplify problem  $P_{1}$  into standard Gaussian case. Then we deal with the simplified problem in the following Lemma 1, which accomplishes the above steps  $2 \sim 4$ . Theorem 1 can be seen as the generalization of Lemma 1.
Lemma 1 Let  $\mathcal{N}(0, I)$  be  $n$ -dimensional standard Gaussian distribution,  $\varepsilon$  be a positive number. For any  $n$ -dimensional Gaussian distribution  $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ ,
(a) If  $KL(\mathcal{N}(\pmb{\mu}, \pmb{\Sigma})||\mathcal{N}(0, I)) \leq \varepsilon$ , then
$$
K L (\mathcal {N} (0, I) | | \mathcal {N} (\boldsymbol {\mu}, \boldsymbol {\Sigma})) \leq \frac {1}{2} \left(\frac {1}{- W _ {0} (- e ^ {- (1 + 2 \varepsilon)})} - \log \frac {1}{- W _ {0} (- e ^ {- (1 + 2 \varepsilon)})} - 1\right)
$$
(b) If  $KL(\mathcal{N}(0,I)||\mathcal{N}(\pmb {\mu},\pmb {\Sigma}))\leq \varepsilon$  , then
$$
K L (\mathcal {N} (\boldsymbol {\mu}, \boldsymbol {\Sigma}) | | \mathcal {N} (0, I)) \leq \frac {1}{2} \left(\frac {1}{- W _ {0} (- e ^ {- (1 + 2 \varepsilon)})} - \log \frac {1}{- W _ {0} (- e ^ {- (1 + 2 \varepsilon)})} - 1\right)
$$
Proof 1 Please see Section C in the Appendix for details.
To further investigate the bound in Theorem 1, we expand the Lambert  $W$  function using the series presented in [19, 14] for small  $\varepsilon$ . This result is stated in the following Theorem 2, which can help users to apply our theorem conveniently.
Theorem 2 For any two  $n$ -dimensional Gaussian distributions  $\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1)$ ,  $\mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)$ , and a small positive number  $\varepsilon$ , if  $KL(\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1) || \mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)) \leq \varepsilon$ , then
$$
K L (\mathcal {N} (\boldsymbol {\mu} _ {2}, \boldsymbol {\Sigma} _ {2}) | | \mathcal {N} (\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1})) \leq \varepsilon + 2 \varepsilon^ {1. 5} + O (\varepsilon^ {2})
$$
Proof 2 Please see Appendix E for details.
Theorem 1 holds for any two Gaussians  $\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1)$  and  $\mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)$ . According to the proof of Theorem 1 (Lemma 1), one of  $\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1)$  and  $\mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)$  can be fixed. Thus, it is not hard to extend Lemma 1 to the case where the fixed Gaussian is not standard. We can apply linear transformation on the fixed Gaussian (see Equation (D.46)) as what we have done in the main proof of Theorem 1 (see Appendix D). Other parts of the proof are the same. Therefore, we obtain the following corollary.
Corollary 1 Theorem 1 and Theorem 2 hold when one of  $\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1)$  and  $\mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)$  is fixed.
Remark 1 The supremum in Theorem 1 is small (zero) when  $\varepsilon$  is small (zero). Figure A.2 in Appendix A shows the graph of the supremum of KL divergence. Due to the strict conditions, it is hard to reach the supremum in typical applications (e.g. in machine learning). Notably, the bound is independent of the dimension  $n$ . This is critical in high-dimensional problems (see motivation application in Section 6). We also note that the condition  $KL(\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1) || \mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)) \leq \varepsilon$  in Theorem 1 is strict in high-dimensional problems.
Remark 2 Theorem 1 gives the strict conditions when the supremum can be attained. We can benefit from these strict conditions in applications. When the forward KL divergence is small, we want a guarantee that the reverse KL divergence is also small such that bounding forward KL divergence also bounds the reverse KL divergence. Theorem 1 has the following two meanings.
1. The supremum of reverse KL divergence  $\varepsilon + 2\varepsilon^{1.5} + O(\varepsilon^2)$  is small, implying that the worst case is acceptable.
2. The strict conditions for reaching the supremum imply that the worst case barely happens in practice. When these strict conditions do not hold, the reverse KL divergence is smaller than the supremum, which is what we want in practice.
Toy Examples.
Figure 1 shows some toy examples in one dimensional case. The black line represents standard Gaussian distribution  $\mathcal{N}_0(0,1)$ . All other four Gaussian distributions  $\mathcal{N}_i$  ( $1 \leq i \leq 4$ , in colored lines) have the same forward KL divergence  $KL(\mathcal{N}_i||\mathcal{N}_0) = 0.01$ . The second distribution has the maximized reverse KL divergence  $KL(\mathcal{N}_0||\mathcal{N}_1(0,0.90173^2)) \approx 0.01148$ , which is equal to the supremum  $\frac{1}{2}\left(\frac{1}{-W_0(-e^{-(1 + 2\times 0.01)})} - \log \frac{1}{-W_0(-e^{-(1 + 2\times 0.01)})} - 1\right)$ . Other reverse KL divergences are  $KL(\mathcal{N}_0||\mathcal{N}_2(0,1.10161^2)) \approx 0.00879$ ,  $KL(\mathcal{N}_0||\mathcal{N}_3(0.14143,1)) \approx 0.01$ ,  $KL(\mathcal{N}_0||\mathcal{N}_4(0.1,1.07153^2)) \approx 0.00892$ .
Figure 1: One dimensional toy examples.
4.2 Infimum of Reverse KL Divergence Between Gaussians
In this subsection, we give the infimum of  $KL(\mathcal{N}_2||\mathcal{N}_1)$  when  $KL(\mathcal{N}_1||\mathcal{N}_2)\geq M$ $(M > 0)$ . The result is presented in Theorem 3.
Theorem 3 For any two  $n$ -dimensional Gaussian distributions  $\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1)$  and  $\mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)$ , if  $KL(\mathcal{N}(\pmb{\mu}_1, \pmb{\Sigma}_1) || \mathcal{N}(\pmb{\mu}_2, \pmb{\Sigma}_2)) \geq M (M > 0)$ , then
$$
K L \left(\mathcal {N} \left(\boldsymbol {\mu} _ {2}, \boldsymbol {\Sigma} _ {2}\right) | | \mathcal {N} \left(\boldsymbol {\mu} _ {1}, \boldsymbol {\Sigma} _ {1}\right)\right) \geq \frac {1}{2} \left(\frac {1}{- W _ {- 1} (- e ^ {- (1 + 2 M)})} - \log \frac {1}{- W _ {- 1} (- e ^ {- (1 + 2 M)})} - 1\right)
$$
The infimum is attained when the following two conditions hold.
(1) There exists only one eigenvalue  $\lambda_{j}$  of  $B_{2}^{-1}\pmb{\Sigma}_{1}(B_{2}^{-1})^{\top}$  or  $B_{1}^{-1}\pmb{\Sigma}_{2}(B_{1}^{-1})^{\top}$  equal to  $-W_{-1}(-e^{-(1 + 2M)})$  and all other eigenvalues  $\lambda_{i}$  ( $i\neq j$ ) are equal to 1, where  $B_{1} = P_{1}D_{1}^{1 / 2}$ ,  $P_{1}$  is an orthogonal matrix whose columns are the eigenvectors of  $\pmb{\Sigma}_{1}$ ,  $D_{1} = \text{diag}(\lambda_{1},\dots,\lambda_{n})$  whose diagonal elements are the corresponding eigenvalues,  $B_{2}$  is defined in the similar way as  $B_{1}$  except on  $\pmb{\Sigma}_{2}$ .
(2)  $\mu_1 = \mu_2$
Proof 3 Theorem 1 and Theorem 3 form a duality. These two theorems can be proved independently in a similar way and also be derived from each other. We give two proofs of Theorem 3 in Appendix F. The first proof presented in Appendix F.1 has the similar structure as that of Theorem 1, except that Theorem 3 uses  $W_{-1}$ . The second proof shown in Appendix F.2 derives Theorem 3 from Theorem 1. These two proofs can verify each other.
5 Relaxed Triangle Inequality
In this section, we give a dimension-free bound of  $KL(\mathcal{N}_1||\mathcal{N}_3)$  when  $KL(\mathcal{N}_1||\mathcal{N}_2)$  and  $KL(\mathcal{N}_2||\mathcal{N}_3)$  are bounded for any three Gaussians  $\mathcal{N}_1$ ,  $\mathcal{N}_2$ , and  $\mathcal{N}_3$ . Proving the relaxed triangle inequality is much more difficult. The main result is presented in Theorem 4 and 5. We use two key Lemmas G.5 and 2 to accomplish the key steps of the proof.