which is exactly the claimed bound (121). This bound gives in particular that

$$ \xi' \mapsto \int_{\Upsilon \times \mathbb{Z}} \ell^T \circ S(y) k^{(0)}(\ell(\xi'), y) \tilde{\varphi}(\ell) \nu(dy) \tau_0(d\ell) $$

is well-defined in $F$ for $\xi' \in D_x(\xi, \epsilon_0)$, and continuous at $\xi$ in direction $x$. Moreover, the Fubini theorem applies so that (i) integrating first with respect to $y$ and (ii) using the expression of $\nabla A$ given in (32), we obtain that this mapping is in fact $\xi' \mapsto w_{\tau_0}^{\nabla}(\xi')$. This being true for all $\xi \in \Upsilon(\tilde{T})$ and $x \in F$ such that $\|x\|_F = 1$, we get the last claim by observing that, by (57), the expectation in (122) is another way to express the same integral in the case where $\xi = \Upsilon(\vartheta)$. ■

We can now prove Theorem 5.

**Proof of Theorem 5** We use Proposition 7 with $\varphi(\ell, y)$ given by (52). We first prove that Conditions (106) and (107) hold in this setting, under the assumptions of Theorem 5.

We use again that the ratio in (52) is the conditional density of $L$ applied to $\ell$ given $Y = y$ under $\mathbb{P}_{\vartheta_0, \tau_0}$, and that the marginal $\mathbb{P}_{\vartheta_0, \tau_0}$ on $Y$ is equivalent to the dominating measure $\nu$. Hence we can write the integral in (106) as

$$ \int_{\Upsilon} \tilde{\varphi}(y) \|S(y)\|_E g(y) \nu(dy) \quad \text{with} \quad g(Y) = \mathbb{E}_{\vartheta_0, \tau_0} \left[ \|L\|_{\text{op}} \mid Y \right]. $$

By (C2) and (58), we obtain Condition (106). Similarly the integral in (107) reads, for all $\xi \in \Upsilon(\tilde{T})$,

$$ \int_{\Upsilon} \tilde{\varphi}(y) g(y) \nu(dy) \quad \text{with} \quad g(Y) = \mathbb{E}_{\vartheta_0, \tau_0} \left[ \|L^{\nabla}(\xi)\|_F \mid Y \right]. $$

Clearly, for this $g$, (C3) gives that, for some $\epsilon, C > 0$ $g(y) \le C\epsilon^{-1} e^{\tilde{s}_{\vartheta_0, \tau_0}(y)}$ for $\nu$-a.e. $y$, and (58) implies (107).

We can thus apply Proposition 7 and to conclude, we only need to check that the conditions of Assertions (i) and (ii) of Proposition 7 also applies, that is, we need to show that $w_{\tau_0}^{\nabla}$ is continuous on $\Upsilon(\tilde{T})$ in every direction and that (109) holds for all $\xi \ne \xi' \in \Upsilon(\tilde{T})$. The continuity of $w_{\tau_0}^{\nabla}$ follows from Lemma 12 since (120) holds by (58) (and we obtain in passing (61), which is exactly (122)). Finally, Condition (109) follows from (C4), since in the setting of Theorem 5, $\tilde{\varphi}(\ell) > 0$ for all $\ell \in \mathbb{Z}$. ■

### C.4.3 PROOF OF COROLLARY 10

The proof of Corollary 10 relies on some preliminary results. We first show in Proposition 8 below that Theorem 5 applies to a general mixture of Gaussian distributions (which notably includes the Student's $t$ distribution family considered in Example 5).

**Proposition 8** Let $\tilde{M}$ be a class of probability distributions on $(0, \infty)$ and suppose that for all $\tilde{\tau}_0 \in \tilde{M}$, there exists $\epsilon_0 > 0$ such that

$$ \int_0^\infty e^{\epsilon_0 z} \tilde{\tau}_0(dz) < \infty. \qquad (123) $$
Let $d \ge 1$ and define for any $\vartheta = (m, \Sigma) \in \tilde{T} = \mathbb{R}^d \times \mathcal{M}_{>0}(d)$

$$ \check{k}(\vartheta, y) = \int_{0}^{\infty} \mathcal{N}(y; m, z^{-1}\Sigma) \check{\tau}_{0}(dz), \qquad (124) $$

seen as a kernel density with respect to $\nu$ being the Lebesgue measure on $\mathbb{R}^d$. Then the following assertions hold.

(i) Let $k^{(1)}$ be ELM family given by Definition 2 with $F = E = \mathbb{R}^d \times \mathbb{R}^{d \times d}$, $E_0 = \tilde{T} = \mathbb{R}^d \times \mathcal{M}_{>0}(d)$, $\Upsilon(m, \Sigma) = (\Sigma^{-1}m, \Sigma^{-1})$,

$$ k^{(0)}(\xi, y) = \mathcal{N}(y; m, \Sigma) \quad \text{for all } \xi = \Upsilon(\vartheta) \text{ with } \vartheta = (m, \Sigma), \qquad (125) $$

and $M$ the class of all push-forward probabilities $\tau_0$ of $\check{\tau}_0 \in \check{M}$ through the mapping $z \mapsto z \text{Id}$ defined from $(0, \infty)$ to $\mathcal{L}(E)$, where $\text{Id}$ is the identity operator on $E$. Then, $M$ is a class of distributions on positive scalar operators on $E$ and, for all $\vartheta = (m, \Sigma) \in \tilde{T}$, we have $\check{k}(\vartheta, y) = k^{(1)}((\vartheta, \tau_0), y)$.

(ii) Setting, for all $y \in \mathbb{R}^d$ and $\vartheta_0 = (m_0, \Sigma_0) \in \tilde{T}$,

$$ q_{\vartheta_0}(y) = \frac{1}{2}(y - m_0)^T \Sigma_0^{-1} (y - m_0), \qquad (126) $$

$$ g_{\check{\tau}_0}(u, v) = \int_{0}^{\infty} z^u e^{-v z} \check{\tau}_0(dz) \quad \text{for all } u > -1 \text{ and } v > -\epsilon_0, \qquad (127) $$

this ELM family satisfies (C1)-(C4) with

$$ \tilde{m}_{\vartheta_0, \tau_0}(y) = \frac{g_{\check{\tau}_0}(d/2 + 1, q_{\vartheta_0}(y))}{g_{\check{\tau}_0}(d/2, q_{\vartheta_0}(y))}, \qquad (128) $$

$$ \check{s}_{\vartheta_0, \tau_0}(y) = \log \left( \frac{g_{\check{\tau}_0}(d/2, q_{\vartheta_0}(y) - \epsilon_0)}{g_{\check{\tau}_0}(d/2, q_{\vartheta_0}(y))} \right). \qquad (129) $$

(iii) Theorem 5 applies for any $\theta_0 = (\vartheta_0, \tau_0) \in \tilde{T} \times M$ and any probability density function $\check{\varphi}: \mathbb{R}^d \to \mathbb{R}_+$ satisfying

$$ \int_{\mathbb{Y}} \check{\varphi}(y) \left( (\|y\| + \|y\|^2) \tilde{m}_{\vartheta_0, \tau_0}(y) + e^{\check{s}_{\vartheta_0, \tau_0}(y)} \right) \nu(dy) < \infty, \qquad (130) $$

in which case the argmax problem (62) has a unique solution $\xi^* = \Upsilon(\vartheta^*)$ with $\vartheta^* = (m^*, \Sigma^*)$ given by

$$ m^* = \frac{\mathbb{E}_{\vartheta_0, \check{\tau}_0}[Z \check{Y}]}{\mathbb{E}_{\vartheta_0, \check{\tau}_0}[Z]}, \qquad (131) $$

$$ \Sigma^* = \mathbb{E}_{\vartheta_0, \check{\tau}_0} [Z \check{Y} \check{Y}^T] - \frac{\mathbb{E}_{\vartheta_0, \check{\tau}_0} [Z \check{Y}] \mathbb{E}_{\vartheta_0, \check{\tau}_0} [Z \check{Y}]^T}{\mathbb{E}_{\vartheta_0, \check{\tau}_0} [Z]}, \qquad (132) $$

where, under $\mathbb{P}_{\vartheta_0, \check{\tau}_0}$, $Z$ has density with respect to $\check{\tau}_0$ given by

$$ \check{\varphi}(z) = \int_{\mathbb{Y}} \check{\varphi}(y) \frac{z^{d/2} e^{-z q_{\vartheta_0}(y)}}{g_{\check{\tau}_0}(d/2, q_{\vartheta_0}(y))} dy \qquad (133) $$
and the conditional distribution of $\tilde{Y}$ given $Z = z$ has density

$$ \psi[y|z] = \frac{1}{1+b} \frac{z^{d/2} e^{-z q_{\vartheta_0}(y)} \tilde{\varphi}(y)}{g_{\tilde{\tau}_0}(d/2, q_{\vartheta_0}(y)) \tilde{\varphi}(z)} + \frac{b}{1+b} \mathcal{N}(y; m_0, z^{-1}\Sigma_0). \quad (134) $$

**Proof** We prove the claimed assertions successively.

**Proof of Assertion (i)** The spaces $F = E = \mathbb{R}^d \times \mathbb{R}^{d \times d}$ are endowed with the usual inner product

$$ \langle (x, M), (x', M') \rangle_E = x^T x' + \mathrm{Tr}(M^T M'). $$

We have $\mathrm{Int}E_0 = E_0 = \tilde{T} = \mathbb{R}^d \times \mathcal{M}_{>0}(d)$, and set, for all $y \in \mathbb{R}^d$, $S(y) = (y, -yy^T/2)$ and, for all $(x, M) \in \mathrm{Int}E_0$, $A(x, M) = \frac{1}{2}(x^T M^{-1}x - \log|M|)$. In this setting, $\ell^T = \ell$, $\ell = \|\ell\|_{\mathrm{op}}$ Id for $\tau_0$-almost all $\ell \in \mathbb{Z}$, and $\|L\|_{\mathrm{op}}$ has distribution $\check{\tau}_0$ for $L \sim \tau_0$. Then, for all $\vartheta = (m, \Sigma) \in \tilde{T}$ and $\tau_0$-almost all $\ell \in \mathbb{Z}$, we have $\ell \circ \Upsilon(m, \Sigma) = \Upsilon(m, \Sigma/\|\ell\|_{\mathrm{op}})$. Consequently, for all $\vartheta \in \tilde{T}$, $y \in \mathbb{R}^d$ and $\tau_0$-almost all $\ell \in \mathbb{Z}$, we have

$$ \begin{aligned} k^{(o)}(\ell \circ \Upsilon(\vartheta), y) &= \left( \frac{1}{2\pi} \|\ell\|_{\mathrm{op}}^{-1} \Sigma \right)^{1/2} e^{-(y-m)^T (\|\ell\|_{\mathrm{op}}^{-1} \Sigma)^{-1} (y-m)/2} \\ &= \mathcal{N} \left( y; m, \|\ell\|_{\mathrm{op}}^{-1} \Sigma \right). \end{aligned} \quad (135) $$

Thus the obtained ELM family $k^{(1)}$ of Definition 2 satisfies $\check{k}(\vartheta, y) = k^{(1)}((\vartheta, \tau_0), y)$ for all $\vartheta = (m, \Sigma) \in \tilde{T}$.

**Proof of Assertion (ii)** Assumption (C1) is obvious in the above setting. We then check (C4), (C2) and (C3), successively. For all $\xi \neq \xi' \in \Upsilon(\tilde{T})$ and $\tau_0 \in \mathbb{M}$, we easily see that

$$ \tau_0(\{\ell \in \mathbb{Z} : \ell(\xi - \xi') = 0\}) = \check{\tau}_0(\{0\}) = 0, $$

and (C4) follows. To check (C2), we compute, for all $\vartheta_0 = (m_0, \Sigma_0)$ and $\tau_0 \in \mathbb{M}$,

$$ \begin{aligned} \mathbb{E}_{\vartheta_0, \tau_0} [\|L\|_{\mathrm{op}} | Y] &= \int_0^\infty z \frac{\mathcal{N}(Y; m_0, z^{-1}\Sigma_0)}{\check{k}(\vartheta_0, Y)} \check{\tau}_0(dz) \\ &= \frac{\int z^{d/2+1} e^{-z q_{\vartheta_0}(Y)} \check{\tau}_0(dz)}{\int z^{d/2} e^{-z q_{\vartheta_0}(Y)} \check{\tau}_0(dz)} \end{aligned} $$

where we successively used (57) with (135) and (126). Hence (C2) follows from (127) and (128). Let us now check (C3). Straightforward computations yield, for all $\xi = (x, M) \in \Upsilon(\tilde{T})$ and $\tau_0$-almost all $\ell \in \mathbb{Z}$,

$$ \begin{aligned} \nabla A \circ \ell(\xi) &= (M^{-1}x, -(M^{-1}xx^T M^{-1} + \|\ell\|_{\mathrm{op}}^{-1} M^{-1})/2) \\ \ell^\nabla(\xi) &= (\|\ell\|_{\mathrm{op}} M^{-1}x, -(\|\ell\|_{\mathrm{op}} M^{-1}xx^T M^{-1} + M^{-1})/2) \\ \|\ell^\nabla(\xi)\|_F &\le (1 + \|\ell\|_{\mathrm{op}}) \|(M^{-1}x, M^{-1}xx^T M^{-1} + M^{-1})\|_F. \end{aligned} $$
It follows, that, for all $\vartheta_0 = (m_0, \Sigma_0) \in \tilde{T}$, $\tau_0 \in M$, $\xi = (x, M) \in \Upsilon(\tilde{T})$ and $\epsilon > 0$,

$$ 
\begin{aligned}
\mathbb{E}_{\vartheta_0, \tau_0} \left[ e^{\epsilon \|L^{\nabla}(\xi)\|_F} \middle| Y \right] &\le C_1 \mathbb{E}_{\vartheta_0, \tau_0} \left[ e^{C_2 \epsilon \|L\|_{op}} \middle| Y \right] \mathbb{P}_{\vartheta_0, \tau_0} - \text{a.s.} \\
&= C_1 \log \left( \frac{g_{\tilde{\tau}_0}(d/2, q_{\vartheta_0}(y) - C_2\epsilon)}{g_{\tilde{\tau}_0}(d/2, q_{\vartheta_0}(y))} \right) \mathbb{P}_{\vartheta_0, \tau_0} - \text{a.s.},
\end{aligned}
$$

where $C_1, C_2 > 0$ only depend on $\xi$. Hence, since we assumed (123), the definition of $g_{\tilde{\tau}_0}$ in (127) holds for all $v > -\epsilon_0$, and (C3) holds with $\tilde{s}_{\vartheta_0, \tau_0}$ as in (129) by taking $\epsilon < \epsilon_0/C_2$.

**Proof of Assertion (iii)** From the previous assertion, we can apply Theorem 5 and use (66) to solve the argmax problem (62) for a density $\tilde{\varphi}$ with respect to the Lebesgue measure on $\mathbb{R}^d$ satisfying (58), which, since $S(y) = (y, -yy^T/2)$ and

$$ \|S(y)\|_E = \left( \|y\|^2 + \frac{1}{4} \mathrm{Tr}(yy^Tyy^T) \right)^{1/2} = O \left( \|y\| + \|y\|^2 \right), $$

is equivalent to (130). Let us compute both sides of the equation (66) which characterizes the solution $\xi^* = \Upsilon(\vartheta^*)$. The left-hand side of (66) reads

$$ \mathbb{E}_{\vartheta^*, \tilde{\tau}_0} [L^T \circ S(Y)] = \mathbb{E}_{\vartheta^*, \tilde{\tau}_0} [Z S(Y)], $$

where $Z = \|L\|_{op}$ has density with respect to $\tilde{\tau}_0$ given by (133). The right-hand side of (66) reads

$$ \mathbb{E}_{\vartheta_0, \tilde{\tau}_0} [L^T \circ S(\tilde{Y})] = \mathbb{E}_{\vartheta_0, \tilde{\tau}_0} [Z S(\tilde{Y})], $$

where $Z$ is the same as above and the conditional distribution of $\tilde{Y}$ given $Z = z$ has density given by (134). Finally, the solution $\xi^* = \Upsilon(\vartheta^*)$ is given by the equation

$$ \mathbb{E}_{\vartheta^*, \tilde{\tau}_0} [Z S(Y)] = \mathbb{E}_{\vartheta_0, \tilde{\tau}_0} [Z S(\tilde{Y})] \quad (136) $$

and this equation leads to the solutions (131) and (132). Indeed, note that

$$ \mathbb{E}_{\vartheta^*, \tilde{\tau}_0} [Y|Z] = m^* \quad \text{and} \quad \mathbb{E}_{\vartheta^*, \tilde{\tau}_0} [YY^T|Z] - m^*(m^*)^T = \Sigma^*/Z \quad (137) $$

Applying the first identity in (137), then the tower property and the identity (136) applied to the first component of $S(Y) = (Y, -YY^T/2)$,

$$ m^* = \frac{\mathbb{E}_{\vartheta^*, \tilde{\tau}_0}[Zm^*]}{\mathbb{E}_{\vartheta^*, \tilde{\tau}_0}[Z]} = \frac{\mathbb{E}_{\vartheta^*, \tilde{\tau}_0}[ZY]}{\mathbb{E}_{\vartheta^*, \tilde{\tau}_0}[Z]} = \frac{\mathbb{E}_{\vartheta_0, \tilde{\tau}_0}[Z\tilde{Y}]}{\mathbb{E}_{\vartheta^*, \tilde{\tau}_0}[Z]} = \frac{\mathbb{E}_{\vartheta_0, \tilde{\tau}_0}[Z\tilde{Y}]}{\mathbb{E}_{\vartheta_0, \tilde{\tau}_0}[Z]} $$

where the last equality follows from the fact that under $\mathbb{P}_{\vartheta^*, \tilde{\tau}_0}$ or $\mathbb{P}_{\vartheta_0, \tilde{\tau}_0}$, $Z$ has the density $\tilde{\varphi}$ with respect to $\tilde{\tau}_0$. Similarly, using the second equality in (137), then the tower property and the identity (136) applied to the second component of $S(Y) = (Y, -YY^T/2)$

$$ 
\begin{aligned}
\Sigma^* &= \mathbb{E}_{\vartheta^*, \tilde{\tau}_0}[Z(\Sigma^*/Z)] = \mathbb{E}_{\vartheta^*, \tilde{\tau}_0}[ZYY^T] - \mathbb{E}_{\vartheta^*, \tilde{\tau}_0}[Z]m^*(m^*)^T \\
&= \mathbb{E}_{\vartheta_0, \tilde{\tau}_0}[Z\tilde{Y}\tilde{Y}^T] - \mathbb{E}_{\vartheta^*, \tilde{\tau}_0}[Z]m^*(m^*)^T
\end{aligned}
$$

This proves (131) and (132).
**Remark 5** The probability density function (124) is called an elliptical distribution. In particular if $\tilde{\tau}_0$ is the $\chi^2$ distribution with $a_0$ degrees of freedom, then it is the density of the Student's t distribution with parameter $(m, \Sigma, a_0)$. In this case we have

$$g_{\tilde{\tau}_0}(u, v) = \frac{(a_0/2)^{a_0/2} \Gamma(a_0/2 + u)}{(a_0/2 + v)^{a_0/2+u} \Gamma(a_0/2)},$$

and $\tilde{m}_{\vartheta_0, \tau_0}$ and $\tilde{s}_{\vartheta_0, \tau_0}$ in (128) and (129) read, for any $\epsilon_0 \in (0, a_0/2]$,

$$\tilde{m}_{\vartheta_0, \tau_0}(y) = \frac{a_0/2 + d/2}{a_0/2 + q_{\vartheta_0}(y)} = O(1),$$

$$\tilde{s}_{\vartheta_0, \tau_0}(y) = (a_0/2 + d/2) \log \left( \frac{a_0/2 + q_{\vartheta_0}(y)}{a_0/2 - \epsilon_0 + q_{\vartheta_0}(y)} \right) = O(1),$$

so that Condition (130) boils down to $\int_Y \check{\varphi}(y) \|y\|^2 \nu(dy) < \infty$.

The following lemma is used in Example 5.

**Lemma 13** Define, for all $x \in (0, \infty)$,

$$\kappa(x) = \log(x) + \Gamma'(x)/\Gamma(x).$$

Then $\kappa$ is increasing and bijective from $(0, \infty)$ to $\mathbb{R}$.

**Proof** This result follows from the fact that the digamma function $x \mapsto \Gamma'(x)/\Gamma(x)$ is increasing from $(0, \infty)$ to $\mathbb{R}$.

**Proof of Corollary 10** We apply Theorem 2 in the setting of Example 5. Condition (18) holds by Theorem 3 and the update (i) in Example 5.

The remainder of the proof consists in proving Condition (19) for given $n \ge 1$ and $j = 1, \dots, J$. We apply Proposition 8, whose Assertion (i) with Remark 5 provides an ELM family $k^{(1)}$ such that, for any $\vartheta = (m, \Sigma) \in \mathbb{R}^d \times \mathcal{M}_{>0}(d)$ and $a > 0$, $\check{k}(\vartheta, y)$ in (124) with $\tilde{\tau}_0 = \tilde{\tau}_a$ given by (73) is $k^{(1)}((\vartheta, \tau_0), y)$ but also $k(\theta, y)$ in Example 5 for $\theta = (m, \Sigma, a)$. We will use Proposition 4 to this ELM family with $\check{\varphi} = \check{\varphi}_{j,n}^{(\alpha)}$, $\theta = (\vartheta, \tau)$ defined with $\vartheta = (m_{j,n+1}, \Sigma_{j,n+1})$ and $\tau$ the measure on obtained by pushing forward $\tilde{\tau}_{a_{j,n+1}}$ through the mapping $z \mapsto z\mathrm{Id}$ and $\theta_0 = (\vartheta_0, \tau_0)$ defined similarly but using the parameters $(m_{j,n}, \Sigma_{j,n}, a_{j,n})$. Then conclusion (49) of Proposition 4 is exactly Condition (19) with $k$ as in Example 5. Hence, to conclude the proof we only need to show that Proposition 4 applies in this setting. Namely, we need to prove (50) and (51), which respectively read

$$\int_Y \left( \int_0^\infty \varphi_{j,n}(z, y) \log \left( \frac{d\tilde{\tau}_{a_{j,n+1}}(z)}{d\tilde{\tau}_{a_{j,n}}(z)} \right) \tilde{\tau}_{a_{j,n}}(dz) \right) dy \ge 0, \quad (138)$$

$$\int_Y \left( \int_Z \varphi_{j,n}(\|\ell\|_{\mathrm{op}}, y) \log \left( \frac{k^{(o)}(\ell \circ \Upsilon(\vartheta), y)}{k^{(o)}(\ell \circ \Upsilon(\vartheta_0), y)} \right) \tau_{a_{j,n}}(d\ell) \right) dy \ge 0, \quad (139)$$
where $\varphi_{j,n}$ is defined by (71) and $k^{(o)}$ is the Gaussian canonical kernel defined by (125). We thus conclude with the proof of these two conditions in the opposite order.

**Proof of (139) Assertion (ii) of Proposition 8 with Remark 5 gives that the ELM family $k^{(1)}$ satisfies (C1)-(C4) with $\tilde{\vartheta}_{0,\tau_0}$ and $\tilde{\vartheta}_{0,\tau_0}$ such that Condition (130) boils down to $\int \tilde{\varphi}(y) \|y\|^2 d\nu < \infty$. This latter condition is satisfied with $\tilde{\varphi} = \tilde{\varphi}_{j,n}^{(\alpha)}$ as a consequence of (67) by using Lemma 8. Therefore Condition (130) holds and Assertion (iii) of Proposition 8 provides the solution of the argmax problem (62). A careful comparison of (131) and (132) with (68) and (69) tells us that $\vartheta = (m_{j,n+1}, \Sigma_{j,n+1})$ is the unique argmax of (62). Applying Corollary 9 in this setting finally gives us (139).**

**Proof of (138) Since the left-hand side of (138) is zero for $a = a_{j,n}$, it suffices to show that $a_{j,n+1}$ in (70) satisfies**

$$a_{j,n+1} = \underset{a \in (0, \infty)}{\operatorname{argmax}} \int_{\mathbb{Y}} \left( \int_{0}^{\infty} \varphi_{j,n}(z, y) \log \left( \frac{d\check{\tau}_a}{d\check{\tau}_{a,j,n}}(z) \right) \check{\tau}_{a,j,n}(dz) \right) dy. \quad (140)$$

By (73), we have, for all $z > 0$,

$$\log \left( \frac{d\check{\tau}_a}{d\check{\tau}_{a,j,n}}(z) \right) = \frac{a}{2} \log \left( \frac{a}{2} \right) + \log \Gamma \left( \frac{a}{2} \right) + \left( \frac{a}{2} - \frac{a_{j,n}}{2} \right) \left( \log z - z \right) + C_{j,n},$$

where $C_{j,n}$ is a positive constant not depending on $a$. The derivative of $x \mapsto x \log(x) + \log \Gamma(x)$ is $\kappa$ as defined in Step (ii) of Example 5 and in Lemma 13. Is it easy to check that $\int \varphi_{j,n}(z, y) \check{\tau}_{a,j,n}(dz) dy = \int \tilde{\varphi}(y) dy = 1$. Thus (140) follows from (70) with Lemma 13 if we can show that

$$\int_{\mathbb{Y}} \left( \int_{0}^{\infty} \varphi_{j,n}(z, y) (|\log z| + z) \check{\tau}_{a,j,n}(dz) \right) dy < \infty.$$

In fact, we will show that there exists $\epsilon > 0$ such that for all $t \in (-\epsilon, 1]$,

$$\int_{\mathbb{Y}} \left( \int_{0}^{\infty} \varphi_{j,n}(z, y) z^t \check{\tau}_{a,j,n}(dz) \right) dy < \infty, \quad (141)$$

which indeed implies the previous display. By definition of $\varphi_{j,n}$ in (71), we find that (141) is implied by

$$I(t) := \int_{\mathbb{Y}} \left( \int_{0}^{\infty} \check{\varphi}_{j,n}^{(\alpha)}(y) z^{t+d/2} e^{-zq_{j,n}(y)} \left( 1 + \frac{2}{a_{j,n}} q_{j,n}(y) \right)^{(a_{j,n}+d)/2} \check{\tau}_{a,j,n}(dz) \right) dy < \infty,$$

where we set $q_{j,n}(y) = \frac{1}{2}(y - m_{j,n})^T \Sigma_{j,n} (y - m_{j,n})$. Since, as in Remark 5, for all $t > -d/2$,

$$\int_{0}^{\infty} z^{t+d/2} e^{-zq_{j,n}(y)} \check{\tau}_{a,j,n}(dz) = \frac{(a_{j,n}/2)^{a_{j,n}/2} \Gamma(a_{j,n}/2 + t + d/2)}{(a_{j,n}/2 + q_{j,n}(y))^{a_{j,n}/2 + t + d/2} \Gamma(a_{j,n}/2)},$$

we get that for all $t > -d/2$, there exists $C'_{j,n}(t) > 0$ such that

$$I(t) = C'_{j,n}(t) \int_{\mathbb{Y}} \check{\varphi}_{j,n}^{(\alpha)}(y) \left( 1 + \frac{2}{a_{j,n}} q_{j,n}(y) \right)^{-t} dy.$$
Since $q_{j,n}(y)$ is a quadratic form and we already checked that $\int \check{\varphi}_{j,n}^{(\alpha)}(y) (1 + \|y\|^2) \, d\nu < \infty$ in the proof of (139), we finally find that $I(t) < \infty$ if $t > -d/2$ and $t \ge -2$, which concludes the proof of (141). ■

D. Additional Numerical Experiments

In this section we provide further plots based on the numerical experiments from Section 8.

<figure><img src="image_11.png" /><figcaption>Figure 5: Error bounds for the Monte Carlo estimate of the VR Bound in the MG-IS-n approach (fixed mixture weights) when considering each of the target distributions (i), (ii) and (iii).</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>Figure 6: Error bounds for the Monte Carlo estimate of the VR Bound in the RGD-IS-n approach (fixed mixture weights) when considering each of the target distributions (i), (ii) and (iii).</figcaption></figure>

References

Pierre Alquier and James Ridgway. Concentration of tempered posteriors and of their variational approximations. *The Annals of Statistics*, 48(3):1475–1497, 2020. doi: 10.1214/19-AOS1855. URL https://doi.org/10.1214/19-AOS1855.

David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. *Journal of the American Statistical Association*, 112(518):859–877, 2017.

Alexander Buchholz, Florian Wenzel, and Stephan Mandt. Quasi-Monte Carlo variational inference. In Jennifer Dy and Andreas Krause, editors, *Proceedings of the 35th International Conference on Machine Learning*, volume 80 of *Proceedings of Machine Learning Research*,
<table><caption>Table 3: LogMSE for the RGD and the MG approaches ($\eta = 0.1$) when considering each of the target distributions (i), (ii) and (iii).</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">J = 10</th><th colspan="3">J = 50</th></tr><tr><th>γ = 0.1</th><th>γ = 0.5</th><th>γ = 1.0</th><th>γ = 0.1</th><th>γ = 0.5</th><th>γ = 1.0</th></tr></thead><tbody><tr><td rowspan="4">(i)</td><td>RGD-IS-n(γ)</td><td>0.372</td><td>0.510</td><td>0.384</td><td>-0.616</td><td>-0.713</td><td>-0.778</td></tr><tr><td>MG-IS-n(γ)</td><td>1.104</td><td>1.074</td><td>0.387</td><td>1.135</td><td>-0.077</td><td>-0.060</td></tr><tr><td>RGD-IS-unif(γ)</td><td>0.359</td><td>0.469</td><td>0.458</td><td>-0.688</td><td>-0.670</td><td>-0.583</td></tr><tr><td>MG-IS-unif(γ)</td><td><b>-0.200</b></td><td><b>-0.229</b></td><td><b>-0.515</b></td><td><b>-1.500</b></td><td><b>-1.462</b></td><td><b>-1.246</b></td></tr><tr><td rowspan="4">(ii)</td><td>RGD-IS-n(γ)</td><td>-0.025</td><td>-0.056</td><td>-0.087</td><td>-1.027</td><td>-0.997</td><td>-0.969</td></tr><tr><td>MG-IS-n(γ)</td><td>-0.270</td><td>-0.126</td><td>0.235</td><td>-0.269</td><td>-0.417</td><td>-0.487</td></tr><tr><td>RGD-IS-unif(γ)</td><td>-0.121</td><td>-0.111</td><td>0.052</td><td>-1.097</td><td>-0.966</td><td>-0.883</td></tr><tr><td>MG-IS-unif(γ)</td><td><b>-1.120</b></td><td><b>-0.938</b></td><td><b>-0.957</b></td><td><b>-1.764</b></td><td><b>-1.889</b></td><td><b>-1.192</b></td></tr><tr><td rowspan="4">(iii)</td><td>RGD-IS-n(γ)</td><td>-0.329</td><td>-0.197</td><td>-0.238</td><td>-1.691</td><td>-1.612</td><td>-1.637</td></tr><tr><td>MG-IS-n(γ)</td><td>1.101</td><td>0.758</td><td>0.524</td><td>0.181</td><td>-0.181</td><td>0.893</td></tr><tr><td>RGD-IS-unif(γ)</td><td>-0.370</td><td>-0.224</td><td>-0.212</td><td>-1.708</td><td>-1.627</td><td><b>-1.649</b></td></tr><tr><td>MG-IS-unif(γ)</td><td><b>-1.211</b></td><td><b>-1.313</b></td><td><b>-1.083</b></td><td><b>-2.013</b></td><td><b>-1.882</b></td><td>-0.491</td></tr></tbody></table>

<table><caption>Table 4: LogMSE for the RGD and the MG approaches ($\gamma = 0.5$) when considering each of the target distributions (i), (ii) and (iii).</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="3">J = 10</th><th colspan="3">J = 50</th></tr><tr><th>η = 0.05</th><th>η = 0.1</th><th>η = 0.5</th><th>η = 0.05</th><th>η = 0.1</th><th>η = 0.5</th></tr></thead><tbody><tr><td rowspan="4">(i)</td><td>RGD-IS-n(γ)</td><td>0.045</td><td>0.510</td><td>1.299</td><td>-1.355</td><td>-0.713</td><td>0.924</td></tr><tr><td>MG-IS-n(γ)</td><td>0.087</td><td>1.074</td><td>1.343</td><td>-1.205</td><td>-0.077</td><td>1.329</td></tr><tr><td>RGD-IS-unif(γ)</td><td>-0.018</td><td>0.469</td><td>1.328</td><td>-1.385</td><td>-0.670</td><td>0.928</td></tr><tr><td>MG-IS-unif(γ)</td><td><b>-1.244</b></td><td><b>-0.229</b></td><td><b>1.100</b></td><td><b>-2.524</b></td><td><b>-1.462</b></td><td><b>0.309</b></td></tr><tr><td rowspan="4">(ii)</td><td>RGD-IS-n(γ)</td><td>-0.096</td><td>-0.056</td><td>0.522</td><td>-1.509</td><td>-0.997</td><td>0.542</td></tr><tr><td>MG-IS-n(γ)</td><td>-0.629</td><td>-0.126</td><td>0.100</td><td>-1.430</td><td>-0.417</td><td>0.348</td></tr><tr><td>RGD-IS-unif(γ)</td><td>-0.195</td><td>-0.111</td><td>0.489</td><td>-1.542</td><td>-0.966</td><td>0.529</td></tr><tr><td>MG-IS-unif(γ)</td><td><b>-1.814</b></td><td><b>-0.938</b></td><td><b>-0.149</b></td><td><b>-1.711</b></td><td><b>-1.889</b></td><td><b>-0.282</b></td></tr><tr><td rowspan="4">(iii)</td><td>RGD-IS-n(γ)</td><td>-0.091</td><td>-0.197</td><td>0.339</td><td>-1.596</td><td>-1.612</td><td>-0.282</td></tr><tr><td>MG-IS-n(γ)</td><td>-0.772</td><td>0.758</td><td>1.358</td><td>-0.878</td><td>-0.181</td><td>0.927</td></tr><tr><td>RGD-IS-unif(γ)</td><td>-0.113</td><td>-0.224</td><td>0.322</td><td>-1.611</td><td>-1.627</td><td>-0.300</td></tr><tr><td>MG-IS-unif(γ)</td><td><b>-1.608</b></td><td><b>-1.313</b></td><td><b>-0.253</b></td><td><b>-1.879</b></td><td><b>-1.882</b></td><td><b>-0.716</b></td></tr></tbody></table>
<figure><img src="image_13.png" /><figcaption>Figure 7: Monte Carlo estimate of the VR Bound for the RGD and the MG approaches ($\eta = 0.05$) when considering each of the target distributions (i), (ii) and (iii).</figcaption></figure>

pages 668–677. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/buchholz18a.html.

Trevor Campbell and Xinglong Li. Universal boosting variational inference. In *Advances in Neural Information Processing Systems*, volume 32, 2019.

Olivier Cappé, Randal Douc, Arnaud Guillin, Jean-Michel Marin, and Christian P Robert. Adaptive importance sampling in general mixture classes. *Statistics and Computing*, 18 (4):447–459, 2008.

Andrzej Cichocki and Shun-ichi Amari. Families of alpha-beta- and gamma-divergences: Flexible and robust measures of similarities. *Entropy*, 12(6):1532–1568, 2010.

Kamélia Daudel and Randal Douc. Mixture weights optimisation for alpha-divergence variational inference. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and
<figure><img src="image_14.png" /><figcaption>Figure 8: Monte Carlo estimate of the VR Bound for the RGD and the MG approaches ($\eta = 0.5$) when considering each of the target distributions (i), (ii) and (iii).</figcaption></figure>

J. Wortman Vaughan, editors, *Advances in Neural Information Processing Systems*, volume 34, pages 4397–4408. Curran Associates, Inc., 2021.

Kamélia Daudel, Randal Douc, and François Portier. Infinite-dimensional gradient-based descent for alpha-divergence minimisation. *The Annals of Statistics*, 49(4):2250–2270, 2021.

Adji Bousso Dieng, Dustin Tran, Rajesh Ranganath, John Paisley, and David Blei. Variational inference via $\chi$ upper bound minimization. In *Advances in Neural Information Processing Systems 30*, pages 2732–2741, 2017.

Justin Domke. Provable smoothness guarantees for black-box variational inference. In *International Conference on Machine Learning*, pages 2587–2596. PMLR, 2020.
<figure><img src="image_15.png" /><figcaption>Figure 9: Monte Carlo estimate of the VR Bound for the RGD and the MG approaches when considering the Bayesian Logistic Regression on the Covertype data set.</figcaption></figure>

Samuel Gershman, Matt Hoffman, and David Blei. Nonparametric variational inference. In *Proceedings of the 29 th International Conference on Machine Learning*, Edinburgh, Scotland, UK, 2012.

Jose Hernandez-Lobato, Yingzhen Li, Mark Rowland, Thang Bui, Daniel Hernandez-Lobato, and Richard Turner. Black-box alpha divergence minimization. In *Proceedings of The 33rd International Conference on Machine Learning*, volume 48 of *Proceedings of Machine Learning Research*, pages 1511–1520, New York, USA, 2016.

Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic variational inference. *Journal of Machine Learning Research*, 14(4):1303–1347, 2013.

Matthew James Beal. *Variational algorithms for approximate Bayesian inference*, PhD thesis, The Gatsby Computational Neuroscience Unit, University of London, UK, 2003.

Ghassen Jerfel, Serena Wang, Clara Fannjiang, Katherine A. Heller, Yian Ma, and Michael I. Jordan. Variational refinement for importance sampling using the forward kullback-leibler divergence. In *37th Conference on Uncertainty in Artificial Intelligence (UAI 2021)*, forthcoming, 2021.

Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. *Machine Learning*, 37(2): 183–233, 1999.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In *Proceedings of the 2nd International Conference on Learning Representations*, 2014.

Yingzhen Li and Richard E Turner. Rényi divergence variational inference. In *Advances in Neural Information Processing Systems 29*, pages 1073–1081, 2016.

Tom Minka. Power ep. Technical Report MSR-TR-2004-149, Microsoft Research Ltd, Cambridge, UK, 2004.

Tom Minka. Divergence measures and message passing. Technical Report MSR-TR-2005-173, Microsoft Research Ltd., Cambridge, UK, 2005.

Dennis Prangle. Distilling importance sampling. *Available on arXiv:1910.03632v3*, last accessed on 10 Sep 2021, 2021.

Rajesh Ranganath, Sean Gerrish, and David Blei. Black Box Variational Inference. In *Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics*, volume 33 of *Proceedings of Machine Learning Research*, pages 814–822, Reykjavik, Iceland, 2014. PMLR.

Alfréd Rényi. On measures of entropy and information. In *Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics*, pages 547–561, Berkeley, Calif., 1961. University of California Press.

Tim van Erven and Peter Harremoes. Rényi divergence and kullback-leibler divergence. *IEEE Transactions on Information Theory*, 60(7):3797–3820, 2014.

Dilin Wang, Hao Liu, and Qiang Liu. Variational inference with tail-adaptive f-divergence. In *Advances in Neural Information Processing Systems 31*, pages 5737–5747, 2018.

Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman. Yes, but did it work?: Evaluating variational inference. In *Proceedings of the 35th International Conference on Machine Learning*, volume 80 of *Proceedings of Machine Learning Research*, pages 5581–5590, Stockholmsmässan, Stockholm Sweden, 2018. PMLR.

Cheng Zhang, Judith Butepage, Hedvig Kjellstrom, and Stephan Mandt. Advances in variational inference. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 41(8):2008–2026, 2019.

Huaiyu Zhu and Richard Rohwer. Information geometric measurements of generalisation. Technical Report NCRG/4350, Neural Computing Research Group Dept. of Computer Science and Applied Mathematics Aston University, Aston Triangle, UK, 1995a.

Huaiyu Zhu and Richard Rohwer. Bayesian invariant measurements of generalization. *Neural Processing Letters*, 2:28–31, 1995b.