SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[50] Yang, Duraiswami, Gumerov, and Davis. Improved fast Gauss transform and efficient kernel density estimation. In Proceedings Ninth IEEE International Conference on Computer Vision, pages 664-671 vol.1, 2003.
[51] Jiyan Yang, Vikas Sindhwani, Quanfu Fan, Haim Avron, and Michael W. Mahoney. Random Laplace feature maps for semigroup kernels on histograms. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pages 971-978. IEEE Computer Society, 2014.
[52] Tianbao Yang, Yu-Feng Li, Mehrdad Mahdavi, Rong Jin, and Zhi-Hua Zhou. Nyström method vs random Fourier features: A theoretical and empirical comparison. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 485-493, 2012.
[53] Manzil Zaheer, Guru Prashanth Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Minh Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Mahmoud El Houssieny Ahmed. Big bird: Transformers for longer sequences. 2020.
[54] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In IEEE international conference on computer vision, pages 19-27, 2015.
Proof. By the definition of  $\langle \nu_{\mathrm{DE}}, f_{\mathrm{DE}}^{(1)}, f_{\mathrm{DE}}^{(2)} \rangle$ , we have:
$$
\begin{array}{l} \mathbb {E} _ {\nu_ {\mathrm {D E}}} f _ {\mathrm {D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) f _ {\mathrm {D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y}) = (2 \pi) ^ {- \frac {d}{2}} D ^ {2} \int_ {\mathbb {R} ^ {d}} \exp \left(- \frac {1}{2} \| \boldsymbol {\omega} \| ^ {2} + 2 \boldsymbol {\omega} ^ {\top} \mathbf {A} \boldsymbol {\omega} \right. \\ \left. + \omega^ {\top} \left(\mathbf {B} ^ {(1)} \mathbf {x} + \mathbf {B} ^ {(2)} \mathbf {y}\right) + \mathbf {x} ^ {\top} \mathbf {C} ^ {(1)} \mathbf {x} + \mathbf {y} ^ {\top} \mathbf {C} ^ {(2)} \mathbf {y}\right) d \boldsymbol {\omega} \\ = (2 \pi) ^ {- \frac {d}{2}} D ^ {2} \exp \left(\mathbf {x} ^ {\top} \mathbf {C} ^ {(1)} \mathbf {x} + \mathbf {y} ^ {\top} \mathbf {C} ^ {(2)} \mathbf {y}\right) \\ \times \int_ {\mathbb {R} ^ {d}} \exp \left(- \frac {1}{2} \boldsymbol {\omega} ^ {\top} (\mathbf {I} _ {d} - 4 \mathbf {A}) \boldsymbol {\omega} + \boldsymbol {\omega} ^ {\top} (\mathbf {B} ^ {(1)} \mathbf {x} + \mathbf {B} ^ {(2)} \mathbf {y})\right) d \boldsymbol {\omega}. \\ \end{array}
$$
Since  $8\mathbf{A} \prec \mathbf{I}_d$ , we have  $4\mathbf{A} \prec 0.5\mathbf{I}_d \prec \mathbf{I}_d$ , meaning that  $\mathbf{I}_d - 4\mathbf{A}$  is positive definite and invertible. The following identity is straightforward to check:
$$
\begin{array}{l} - \frac {1}{2} \boldsymbol {\omega} ^ {\top} (\mathbf {I} _ {d} - 4 \mathbf {A}) \boldsymbol {\omega} + \boldsymbol {\omega} ^ {\top} (\mathbf {B} ^ {(1)} \mathbf {x} + \mathbf {B} ^ {(2)} \mathbf {y}) = - \frac {1}{2} \left(\boldsymbol {\omega} - \boldsymbol {\mu}\right) ^ {\top} \boldsymbol {\Sigma} ^ {- 1} (\boldsymbol {\omega} - \boldsymbol {\mu}) + \frac {1}{2} \boldsymbol {\mu} ^ {\top} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {\mu}, \\ \boldsymbol {\Sigma} = (\mathbf {I} _ {d} - 4 \mathbf {A}) ^ {- 1}, \quad \boldsymbol {\mu} = \boldsymbol {\Sigma} (\mathbf {B} ^ {(1)} \mathbf {x} + \mathbf {B} ^ {(2)} \mathbf {y}). \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {\nu_ {\mathrm {D E}}} f _ {\mathrm {D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) f _ {\mathrm {D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y}) = (2 \pi) ^ {- d / 2} D ^ {2} \exp \left(\mathbf {x} ^ {\top} \mathbf {C} ^ {(1)} \mathbf {x} + \mathbf {y} ^ {\top} \mathbf {C} ^ {(2)} \mathbf {y} + \frac {1}{2} \boldsymbol {\mu} ^ {\top} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {\mu}\right) \\ \times \int_ {\mathbb {R} ^ {d}} \exp \left(- \frac {1}{2} (\boldsymbol {\omega} - \boldsymbol {\mu}) ^ {\top} \boldsymbol {\Sigma} ^ {- 1} (\boldsymbol {\omega} - \boldsymbol {\mu})\right) d \boldsymbol {\omega}. \\ \end{array}
$$
Next, we use the fact that the integral of the multivariate Gaussian distribution with mean  $\mu$  and variance  $\Sigma$  is 1:
$$
(2 \pi) ^ {- d / 2} \det (\boldsymbol {\Sigma}) ^ {- 1 / 2} \int_ {\mathbb {R} ^ {d}} \exp \left(- \frac {1}{2} (\boldsymbol {\omega} - \boldsymbol {\mu}) ^ {\top} \boldsymbol {\Sigma} ^ {- 1} (\boldsymbol {\omega} - \boldsymbol {\mu})\right) d \boldsymbol {\omega} = 1.
$$
From that we conclude:
$$
\begin{array}{l} \mathbb {E} _ {\nu_ {\mathrm {D E}}} f _ {\mathrm {D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) f _ {\mathrm {D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y}) = D ^ {2} \det  (\boldsymbol {\Sigma}) ^ {1 / 2} \exp \left(\mathbf {x} ^ {\top} \mathbf {C} ^ {(1)} \mathbf {x} + \mathbf {y} ^ {\top} \mathbf {C} ^ {(2)} \mathbf {y} + \frac {1}{2} \boldsymbol {\mu} ^ {\top} \boldsymbol {\Sigma} ^ {- 1} \boldsymbol {\mu}\right) \\ = D ^ {2} \det  (\mathbf {I} _ {d} - 4 \mathbf {A}) ^ {- 1 / 2} \exp \left(\mathbf {x} ^ {\top} \mathbf {C} ^ {(1)} \mathbf {x} + \mathbf {y} ^ {\top} \mathbf {C} ^ {(2)} \mathbf {y} \right. \\ \left. + \frac {1}{2} \left(\mathbf {B} ^ {(1)} \mathbf {x} + \mathbf {B} ^ {(2)} \mathbf {y}\right) ^ {\top} \left(\mathbf {I} _ {d} - 4 \mathbf {A}\right) ^ {- 1} \left(\mathbf {B} ^ {(1)} \mathbf {x} + \mathbf {B} ^ {(2)} \mathbf {y}\right)\right) \\ = D ^ {2} \det  (\mathbf {I} _ {d} - 4 \mathbf {A}) ^ {- 1 / 2} \exp \left(\mathbf {x} ^ {\top} \left(\mathbf {C} ^ {(1)} + \frac {1}{2} (\mathbf {B} ^ {(1)}) ^ {\top} (\mathbf {I} _ {d} - 4 \mathbf {A}) ^ {- 1} \mathbf {B} ^ {(1)}\right) \mathbf {x} \right. \\ + \mathbf {y} ^ {\top} \left(\mathbf {C} ^ {(2)} + \frac {1}{2} (\mathbf {B} ^ {(2)}) ^ {\top} (\mathbf {I} _ {d} - 4 \mathbf {A}) ^ {- 1} \mathbf {B} ^ {(2)}\right) \mathbf {y} + \mathbf {x} ^ {\top} (\mathbf {B} ^ {(1)}) ^ {\top} (\mathbf {I} _ {d} - 4 \mathbf {A}) ^ {- 1} \mathbf {B} ^ {(2)} \mathbf {y}\left. \right). \\ \end{array}
$$
Based on this expression, we conclude that, indeed,  $\mathbb{E}_{\nu_{\mathrm{DE}}}f_{\mathrm{DE}}^{(1)}(\pmb {\omega},\mathbf{x})f_{\mathrm{DE}}^{(2)}(\pmb {\omega},\mathbf{y}) = K^{(0)}(\mathbf{x},\mathbf{y})$  for all  $\mathbf{x},\mathbf{y}\in \mathbb{R}^d$  if the conditions from theorem's statement are satisfied.
Next, we calculate expression for the variance. For any random variable  $Z$ ,  $\operatorname{Var} Z = \mathbb{E}Z^2 - (\mathbb{E}Z)^2$ . In particular, if  $Z = f_{\mathrm{DE}}^{(1)}(\omega, \mathbf{x})f_{\mathrm{DE}}^{(2)}(\omega, \mathbf{y})$ ,  $\omega \sim \nu_{\mathrm{DE}}$ , we get:
$$
\begin{array}{l} \operatorname {V a r} _ {\nu_ {\mathrm {D E}}} f _ {\mathrm {D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) f _ {\mathrm {D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y}) = \mathbb {E} _ {\nu_ {\mathrm {D E}}} f _ {\mathrm {D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) ^ {2} f _ {\mathrm {D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y}) ^ {2} - \left(\mathbb {E} _ {\nu_ {\mathrm {D E}}} f _ {\mathrm {D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) f _ {\mathrm {D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y})\right) ^ {2} \\ = \mathbb {E} _ {\nu_ {\mathrm {D E}}} f _ {\mathrm {D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) ^ {2} f _ {\mathrm {D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y}) ^ {2} - K ^ {(0)} (\mathbf {x}, \mathbf {y}) ^ {2}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {\nu_ {\mathrm {D E}}} f _ {\mathrm {D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) ^ {2} f _ {\mathrm {D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y}) ^ {2} = (2 \pi) ^ {\frac {d}{2}} D ^ {4} \int_ {\mathbb {R} ^ {d}} \exp \left(- \frac {1}{2} \| \boldsymbol {\omega} \| ^ {2} + 4 \boldsymbol {\omega} ^ {\top} \mathbf {A} \boldsymbol {\omega} + 2 \boldsymbol {\omega} ^ {\top} (\mathbf {B} ^ {(1)} \mathbf {x} + \mathbf {B} ^ {(2)} \mathbf {y}) \right. \\ \left. + 2 \mathbf {x} ^ {\top} \mathbf {C} ^ {(1)} \mathbf {x} + 2 \mathbf {y} ^ {\top} \mathbf {C} ^ {(2)} \mathbf {y}\right) d \boldsymbol {\omega} = (2 \pi) ^ {\frac {d}{2}} D ^ {4} \exp \left(2 \mathbf {x} ^ {\top} \mathbf {C} ^ {(1)} \mathbf {x} + 2 \mathbf {y} ^ {\top} \mathbf {C} ^ {(2)} \mathbf {y}\right) \\ \times \int_ {\mathbb {R} ^ {d}} \exp \left(- \frac {1}{2} \boldsymbol {\omega} ^ {\top} (\mathbf {I} _ {d} - 8 \mathbf {A}) \boldsymbol {\omega} + 2 \boldsymbol {\omega} ^ {\top} (\mathbf {B} ^ {(1)} \mathbf {x} + \mathbf {B} ^ {(2)} \mathbf {y})\right) d \boldsymbol {\omega}. \\ \end{array}
$$
Evaluation of the integral above can be done in the same way as calculation of  $\mathbb{E}\nu_{\mathrm{DE}}f_{\mathrm{DE}}^{(1)}(\boldsymbol {\omega},\mathbf{x})f_{\mathrm{DE}}^{(2)}(\boldsymbol {\omega},\mathbf{y})$  , noticing that  $\mathbf{I}_d - 8\mathbf{A}$  is positive definite and invertible. The result is as follows:
$$
\begin{array}{l} \mathbb {E} _ {\nu_ {\mathrm {D E}}} f _ {\mathrm {D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) ^ {2} f _ {\mathrm {D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y}) ^ {2} = D ^ {4} \det  (\mathbf {I} _ {d} - 8 \mathbf {A}) ^ {- 1 / 2} \\ \times \exp \left(2 \mathbf {x} ^ {\top} \left(\mathbf {C} ^ {(1)} + (\mathbf {B} ^ {(1)}) ^ {\top} (\mathbf {I} _ {d} - 8 \mathbf {A}) ^ {- 1} \mathbf {B} ^ {(1)}\right) \mathbf {x} \right. \\ + 2 \mathbf {y} ^ {\top} \left(\mathbf {C} ^ {(2)} + (\mathbf {B} ^ {(2)}) ^ {\top} (\mathbf {I} _ {d} - 8 \mathbf {A}) ^ {- 1} \mathbf {B} ^ {(2)}\right) \mathbf {y} + 4 \mathbf {x} ^ {\top} (\mathbf {B} ^ {(1)}) ^ {\top} (\mathbf {I} _ {d} - 8 \mathbf {A}) ^ {- 1} \mathbf {B} ^ {(2)} \mathbf {y}\Big). \\ \end{array}
$$
We conclude that the variance expression given in the theorem's statement is correct.
A.2 Important lemma
Below, we prove an important lemma which is used in the subsequent proofs:
Lemma A.1. Consider a function  $f:(-\infty ,\frac{1}{8})$  defined as
$$
f (A) = \log (1 - 4 A) - \frac {1}{2} \log (1 - 8 A) + \frac {\phi}{1 - 8 A} \tag {14}
$$
where  $\phi \geq 0$ . Then, the minimum of  $f$  on  $(- \infty, \frac{1}{8})$  is achieved at
$$
A ^ {*} = \frac {1}{1 6} \left(1 - 2 \phi - \sqrt {\left(2 \phi + 1\right) ^ {2} + 8 \phi}\right). \tag {15}
$$
Proof. Set  $\gamma = (1 - 8A)^{-1} \in (0, +\infty)$ . Note that there is a one-to-one correspondence between  $\gamma \in (0, +\infty)$  and  $A \in (-\infty, \frac{1}{8})$ . Hence, we can substitute  $\gamma^{-1} = 1 - 8A$  and  $1 - 4A = ((1 - 8A) + 1)/2 = (\gamma^{-1} + 1)/2 = \frac{1 + \gamma}{2\gamma}$  in (14) and equivalently perform minimization with respect to  $\gamma$ :
$$
\min  _ {\gamma \in (0, + \infty)} h (\gamma) = \log \left(\frac {\gamma + 1}{2 \gamma}\right) + \frac {1}{2} \log \gamma + \phi \gamma = \log (\gamma + 1) - \frac {1}{2} \log \gamma - \log 2 + \phi \gamma .
$$
For  $h(\cdot)$ 's derivative, we have:
$$
\begin{array}{l} h ^ {\prime} (\gamma) = \phi + \frac {1}{\gamma + 1} - \frac {1}{2 \gamma} = \phi + \frac {\gamma - 1}{2 \gamma (\gamma + 1)} (16) \\ = \frac {2 \phi \gamma (\gamma + 1) + \gamma - 1}{2 \gamma (\gamma + 1)} = \frac {2 \phi \gamma^ {2} + (2 \phi + 1) \gamma - 1}{2 \gamma (\gamma + 1)}. (17) \\ \end{array}
$$
Based on (16), we see that  $h'(\gamma) \to -\infty$  as  $\gamma \to 0$  and  $h'(\gamma) > \phi \geq 0$  for all  $\gamma > 1$ . Hence, we conclude that  $h(\cdot)$  is bounded from below on  $(0, +\infty)$  and the global minimum  $\gamma^*$  on  $(0, +\infty)$  exists and it is one of the points satisfying  $h'(\gamma^*) = 0$ . Hence, it's one of the positive roots of the polynomial in numerator of (17).
If  $\phi = 0$ , there is a single root  $\gamma^{*} = 1$  of the polynomial in the numerator of (17), hence it is a global minimum of  $h(\cdot)$ . If  $\phi > 0$ , then there are two roots of the polynomial in the numerator of (17):
$$
\gamma_ {-} ^ {*} = \frac {- (2 \phi + 1) - \sqrt {(2 \phi + 1) ^ {2} + 8 \phi}}{4 \phi},
$$
$$
\gamma_ {+} ^ {*} = \frac {- (2 \phi + 1) + \sqrt {(2 \phi + 1) ^ {2} + 8 \phi}}{4 \phi}. \tag {18}
$$
Note that, if  $\phi > 0$ , then  $2\phi + 1 > 0$  and  $(2\phi + 1)^{2} + 8\phi \geq (2\phi + 1)^{2}$ . Hence,  $\gamma_{-}^{*} < 0$  and  $\gamma_{+}^{*} > 0$ . We conclude that  $\gamma^{*} = \gamma_{+}^{*}$  is the minimum of  $h(\cdot)$  on  $(0, +\infty)$ . We multiply numerator and denominator of (18)'s right hand side by  $(2\phi + 1) + \sqrt{(2\phi + 1)^{2} + 8\phi} > 0$ :
$$
\gamma^ {*} = \gamma_ {+} ^ {*} = \frac {\left((2 \phi + 1) ^ {2} + 8 \phi\right) - (2 \phi + 1) ^ {2}}{4 \phi \left((2 \phi + 1) + \sqrt {(2 \phi + 1) ^ {2} + 8 \phi}\right)} = \frac {2}{2 \phi + 1 + \sqrt {(2 \phi + 1) ^ {2} + 8 \phi}}. \tag {19}
$$
Note that the right hand side of (19) is equivalent to (18) when  $\phi > 0$  but also holds for the case when  $\phi = 0$  (i.e. when  $\gamma^{*} = 1$ ). We conclude that  $f(\cdot)$  is minimized at  $\mathbf{A}^{*} = \frac{1}{8}(1 - (\gamma^{*})^{-1})$  since  $\gamma^{*} = (1 - 8A^{*})^{-1}$ . It's easy to see that (15) follows from (19) directly.
A.3 Proof of Theorem 4.2
Proof. With  $\mathbf{A} = A\mathbf{I}_d$ , the conditions from Theorem 4.1 read as
$$
8 A <   1, \quad \frac {1}{1 - 4 A} \left(\mathbf {B} ^ {(1)}\right) ^ {\top} \mathbf {B} ^ {(2)} = \mathbf {I} _ {d}, \quad \mathbf {C} ^ {(k)} = - \frac {1}{2 (1 - 4 A)} \left(\mathbf {B} ^ {(k)}\right) ^ {\top} \mathbf {B} ^ {(k)}, D = (1 - 4 A) ^ {d / 4} \tag {20}
$$
for  $k\in \{1,2\}$ . And the variance expression (9) for all  $\mathbf{x},\mathbf{y}\in \mathbb{R}^d$  transforms into
$$
\begin{array}{l} \operatorname {V a r} _ {\nu_ {\mathrm {A D E}}} f _ {\mathrm {A D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) f _ {\mathrm {A D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y}) = D ^ {4} (1 - 8 A) ^ {- d / 2} \exp \left(2 \mathbf {x} ^ {\top} \left(\mathbf {C} ^ {(1)} + \frac {1}{1 - 8 A} (\mathbf {B} ^ {(1)}) ^ {\top} \mathbf {B} ^ {(1)}\right) \mathbf {x} \right. \\ + 2 \mathbf {y} ^ {\top} \left(\mathbf {C} ^ {(2)} + \frac {1}{1 - 8 A} (\mathbf {B} ^ {(2)}) ^ {\top} \mathbf {B} ^ {(2)}\right) \mathbf {y} + \frac {4}{1 - 8 A} \mathbf {x} ^ {\top} (\mathbf {B} ^ {(1)}) ^ {\top} \mathbf {B} ^ {(2)} \mathbf {y}\left. \right) - K ^ {(0)} (\mathbf {x}, \mathbf {y}) ^ {2}. \\ \end{array}
$$
We express  $\mathbf{C}^{(k)}$  through  $A$ ,  $\mathbf{B}^{(k)}$  and  $D$  through  $A$  using (20) in the equation above:
$$
\begin{array}{l} \operatorname {V a r} _ {\nu_ {\mathrm {A D E}}} f _ {\mathrm {A D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) f _ {\mathrm {A D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y}) = \left(\frac {1 - 4 A}{\sqrt {1 - 8 A}}\right) ^ {d} \exp \left(\left(\frac {2}{1 - 8 A} - \frac {1}{1 - 4 A}\right) \mathbf {x} ^ {\top} (\mathbf {B} ^ {(1)}) ^ {\top} \mathbf {B} ^ {(1)} \mathbf {x} \right. \\ + \left(\frac {2}{1 - 8 A} - \frac {1}{1 - 4 A}\right) \mathbf {y} ^ {\top} (\mathbf {B} ^ {(2)}) ^ {\top} \mathbf {B} ^ {(2)} \mathbf {y} + \frac {4}{1 - 8 A} \mathbf {x} ^ {\top} (\mathbf {B} ^ {(1)}) ^ {\top} \mathbf {B} ^ {(2)} \mathbf {y}\left. \right) - K ^ {(0)} (\mathbf {x}, \mathbf {y}) ^ {2}. \\ \end{array}
$$
Since  $\frac{1}{1 - 4A} (\mathbf{B}^{(1)})^\top \mathbf{B}^{(2)}$  is a full-rank matrix  $\mathbf{I}_d$  (20), both  $\mathbf{B}^{(1)}$  and  $\mathbf{B}^{(2)}$  are full-rank. Hence, we can express  $\mathbf{B}^{(2)} = (1 - 4A)(\mathbf{B}^{(1)})^{-\top}$ . Also, note that
$$
\frac {2}{1 - 8 A} - \frac {1}{1 - 4 A} = \frac {2 - 8 A - 1 + 8 A}{(1 - 8 A) (1 - 4 A)} = (1 - 8 A) ^ {- 1} (1 - 4 A) ^ {- 1}.
$$
We rewrite the expression for the variance using the identity above and the formula for  $\mathbf{B}^{(2)}$ :
$$
\begin{array}{l} \mathrm {V a r} _ {\nu_ {\mathrm {A D E}}} f _ {\mathrm {A D E}} ^ {(1)} (\pmb {\omega}, \mathbf {x}) f _ {\mathrm {A D E}} ^ {(2)} (\pmb {\omega}, \mathbf {y}) = \left(\frac {1 - 4 A}{\sqrt {1 - 8 A}}\right) ^ {d} \exp \bigg ((1 - 8 A) ^ {- 1} (1 - 4 A) ^ {- 1} \mathbf {x} ^ {\top} (\mathbf {B} ^ {(1)}) ^ {\top} \mathbf {B} ^ {(1)} \mathbf {x} \\ \left. + (1 - 8 A) ^ {- 1} (1 - 4 A) \mathbf {y} ^ {\top} ((\mathbf {B} ^ {(1)}) ^ {\top} \mathbf {B} ^ {(1)}) ^ {- 1} \mathbf {y} + 4 (1 - 8 A) ^ {- 1} (1 - 4 A) \mathbf {x} ^ {\top} \mathbf {y}\right) - K ^ {(0)} (\mathbf {x}, \mathbf {y}) ^ {2}. \\ \end{array}
$$
We use the expression above to rewrite (8) for  $\langle \nu, f^{(1)}, f^{(2)} \rangle = \langle \nu_{\mathrm{ADE}}, f_{\mathrm{ADE}}^{(1)}, f_{\mathrm{ADE}}^{(2)} \rangle$  as follows:
$$
\begin{array}{l} \overline {{\mathcal {L}}} (\boldsymbol {\theta} _ {\mathrm {A D E}}; \mathcal {X}, \mathcal {Y}, \mathcal {T} _ {\mathrm {A D E}}) = L ^ {- 2} \sum_ {1 \leq i, j \leq L} \log (\operatorname {V a r} _ {\nu_ {\mathrm {A D E}}} f _ {\mathrm {A D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x} ^ {(i)}) f _ {\mathrm {A D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y} ^ {(j)}) + K ^ {(0)} (\mathbf {x} ^ {(i)}, \mathbf {y} ^ {(j)})) \\ = d \log (1 - 4 A) - \frac {d}{2} \log (1 - 8 A) + (1 - 8 A) ^ {- 1} (1 - 4 A) ^ {- 1} L ^ {- 1} \sum_ {i = 1} ^ {L} (\mathbf {x} ^ {(i)}) ^ {\top} (\mathbf {B} ^ {(1)}) ^ {\top} \mathbf {B} ^ {(1)} \mathbf {x} ^ {(i)} \\ + (1 - 8 A) ^ {- 1} (1 - 4 A) L ^ {- 1} \sum_ {j = 1} ^ {L} (\mathbf {y} ^ {(j)}) ^ {\top} (\mathbf {B} ^ {(1)}) ^ {- 1} (\mathbf {B} ^ {(1)}) ^ {- \top} \mathbf {y} ^ {(j)} \\ \end{array}
$$
$$
+ 4 (1 - 8 A) ^ {- 1} (1 - 4 A) L ^ {- 2} \sum_ {1 \leq i, j \leq L} \left(\mathbf {x} ^ {(i)}\right) ^ {\top} \mathbf {y} ^ {(j)}. \tag {21}
$$
Denote  $\mathbf{E} = (\mathbf{B}^{(1)})^{\top}\mathbf{B}^{(1)}$ . Then (21) becomes:
$$
\begin{array}{l} \overline {{\mathcal {L}}} (\boldsymbol {\theta} _ {\mathrm {A D E}}; \mathcal {X}, \mathcal {Y}, \mathcal {T} _ {\mathrm {A D E}}) = d \log (1 - 4 A) - \frac {d}{2} \log (1 - 8 A) \\ + (1 - 8 A) ^ {- 1} (1 - 4 A) ^ {- 1} L ^ {- 1} \sum_ {i = 1} ^ {L} (\mathbf {x} ^ {(i)}) ^ {\top} \mathbf {E} \mathbf {x} ^ {(i)} \\ + (1 - 8 A) ^ {- 1} (1 - 4 A) L ^ {- 1} \sum_ {j = 1} ^ {L} \left(\mathbf {y} ^ {(j)}\right) ^ {\top} \mathbf {E} ^ {- 1} \mathbf {y} ^ {(j)} + 4 (1 - 8 A) ^ {- 1} (1 - 4 A) L ^ {- 2} \sum_ {1 \leq i, j \leq L} \left(\mathbf {x} ^ {(i)}\right) ^ {\top} \mathbf {y} ^ {(j)}. \tag {22} \\ \end{array}
$$
We next prove the following lemma:
Lemma A.2. Let  $\mathbf{B}^{(1)*} = \sqrt{1 - 4A}\pmb{\Sigma}^{1/2}\mathbf{U}^{\top}(\pmb{\Lambda}^{(1)})^{-1/2}(\mathbf{Q}^{(1)})^{\top}$ . When  $A$  ( $8A < 1$ ) is fixed,  $\mathbf{E} = \mathbf{E}^{*} = (\mathbf{B}^{(1)*})^{\top}\mathbf{B}^{(1)*}$  minimizes the right hand side of (22) with respect to  $\mathbf{E}$ .
$$
\begin{array}{l} L ^ {- 1} \sum_ {i = 1} ^ {L} (\mathbf {x} ^ {(i)}) ^ {\top} \mathbf {E} \mathbf {x} ^ {(i)} = L ^ {- 1} \sum_ {i = 1} ^ {L} \mathrm {T r a c e} ((\mathbf {x} ^ {(i)}) ^ {\top} \mathbf {E} \mathbf {x} ^ {(i)}) = L ^ {- 1} \sum_ {i = 1} ^ {L} \mathrm {T r a c e} (\mathbf {E} \mathbf {x} ^ {(i)} (\mathbf {x} ^ {(i)}) ^ {\top}) \\ = \operatorname {T r a c e} \left(\mathbf {E} \left(L ^ {- 1} \sum_ {i = 1} ^ {L} \mathbf {x} ^ {(i)} (\mathbf {x} ^ {(i)}) ^ {\top}\right)\right) = \operatorname {T r a c e} \left(\mathbf {E M} ^ {(1)}\right) \\ \end{array}
$$
where we use the cyclic property of trace  $\mathrm{Trace}(\cdot)$  and linearity of trace. Analogously, we obtain  $L^{-1}\sum_{j = 1}^{L}(\mathbf{y}^{(j)})^{\top}\mathbf{E}^{-1}\mathbf{y}^{(j)} = \mathrm{Trace}(\mathbf{E}^{-1}\mathbf{M}^{(2)})$ . Assuming that  $A$  is fixed, optimization of (22) with respect to  $\mathbf{E}$  reduces to the following minimization problem:
$$
\min  _ {\mathbf {E} \in \mathbb {S} _ {d}, \mathbf {E} \succ 0} \quad \mathcal {F} (\mathbf {E}) = \beta_ {1} \operatorname {T r a c e} \left(\mathbf {E} \mathbf {M} ^ {(1)}\right) + \beta_ {2} \operatorname {T r a c e} \left(\mathbf {E} ^ {- 1} \mathbf {M} ^ {(2)}\right) \tag {23}
$$
where  $\beta_{1} = (1 - 8A)^{-1}(1 - 4A)^{-1}$ ,  $\beta_{2} = (1 - 8A)^{-1}(1 - 4A)$  and the constraint  $\mathbf{E} \in \mathbb{S}_d$ ,  $\mathbf{E} \succ 0$  follows from the fact that  $\mathbf{E} = (\mathbf{B}^{(1)})^\top \mathbf{B}^{(1)}$  and  $\mathbf{E}$  is invertible. We have  $1 - 8A > 0$  and  $1 - 4A = (1 - 8A)/2 + 1/2 > 0$ . Hence,  $\beta_{1}, \beta_{2} > 0$ . For any  $\mathbf{E} \succ 0$  and any  $\Delta \in \mathbb{S}_d$  there is  $t \in \mathbb{R}$  small enough such that  $\mathbf{E} + t\mathbf{B}$  is invertible and the following Neumann series is convergent:
$$
\left(\mathbf {E} + t \boldsymbol {\Delta}\right) ^ {- 1} = \mathbf {E} ^ {- 1} \left(\mathbf {I} _ {d} + t \boldsymbol {\Delta} \mathbf {E} ^ {- 1}\right) ^ {- 1} = \sum_ {l = 0} ^ {\infty} (- t) ^ {l} \mathbf {E} ^ {- 1} \left(\boldsymbol {\Delta} \mathbf {E} ^ {- 1}\right) ^ {l}
$$
We further deduce:
$$
\begin{array}{l} \operatorname {T r a c e} ((\mathbf {E} + t \boldsymbol {\Delta}) ^ {- 1} \mathbf {M} ^ {(2)}) = \operatorname {T r a c e} \left(\left(\sum_ {l = 0} ^ {\infty} (- t) ^ {l} \mathbf {E} ^ {- 1} (\boldsymbol {\Delta} \mathbf {E} ^ {- 1}) ^ {l}\right) \mathbf {M} ^ {(2)}\right) = \\ = \sum_ {l = 0} ^ {\infty} (- t) ^ {l} \operatorname {T r a c e} \left(\mathbf {E} ^ {- 1} \left(\boldsymbol {\Delta} \mathbf {E} ^ {- 1}\right) ^ {l} \mathbf {M} ^ {(2)}\right) \\ \end{array}
$$
and, therefore,
$$
\begin{array}{l} \mathcal {F} (\mathbf {E} + t \pmb {\Delta}) = \beta_ {1} \mathrm {T r a c e} ((\mathbf {E} + t \pmb {\Delta}) \mathbf {M} ^ {(1)}) + \beta_ {2} \sum_ {l = 0} ^ {\infty} (- t) ^ {l} \mathrm {T r a c e} \left(\mathbf {E} ^ {- 1} (\pmb {\Delta} \mathbf {E} ^ {- 1}) ^ {l} \mathbf {M} ^ {(2)}\right) \\ = \beta_ {1} \operatorname {T r a c e} \left(\mathbf {E M} ^ {(1)}\right) + t \beta_ {1} \operatorname {T r a c e} \left(\boldsymbol {\Delta} \mathbf {M} ^ {(1)}\right) + \beta_ {2} \sum_ {l = 0} ^ {\infty} (- t) ^ {l} \operatorname {T r a c e} \left(\mathbf {E} ^ {- 1} \left(\boldsymbol {\Delta} \mathbf {E} ^ {- 1}\right) ^ {l} \mathbf {M} ^ {(2)}\right). \tag {24} \\ \end{array}
$$
Further, we have:
$$
\begin{array}{l} \frac {\partial}{\partial t} \mathcal {F} (\mathbf {E} + t \boldsymbol {\Delta}) = \beta_ {1} \operatorname {T r a c e} \left(\boldsymbol {\Delta} \mathbf {M} ^ {(1)}\right) + \beta_ {2} \sum_ {l = 1} ^ {\infty} (- 1) ^ {l} l t ^ {l - 1} \operatorname {T r a c e} \left(\mathbf {E} ^ {- 1} \left(\boldsymbol {\Delta} \mathbf {E} ^ {- 1}\right) ^ {l} \mathbf {M} ^ {(2)}\right), \\ \frac {\partial^ {2}}{(\partial t) ^ {2}} \mathcal {F} (\mathbf {E} + t \boldsymbol {\Delta}) = \beta_ {2} \sum_ {l = 2} ^ {\infty} (- 1) ^ {l} l (l - 1) t ^ {l - 2} \mathrm {T r a c e} \left(\mathbf {E} ^ {- 1} (\boldsymbol {\Delta} \mathbf {E} ^ {- 1}) ^ {l} \mathbf {M} ^ {(2)}\right), \\ \left. \frac {\partial^ {2}}{(\partial t) ^ {2}} \mathcal {F} (\mathbf {E} + t \boldsymbol {\Delta}) \right| _ {t = 0} = 2 \beta_ {2} \operatorname {T r a c e} \left(\mathbf {E} ^ {- 1} \boldsymbol {\Delta} \mathbf {E} ^ {- 1} \boldsymbol {\Delta} \mathbf {E} ^ {- 1} \mathbf {M} ^ {(2)}\right). \tag {25} \\ \end{array}
$$
We replace  $\mathbf{M}^{(2)} = \mathbf{Q}^{(2)}(\mathbf{\Lambda}^{(2)})^{1 / 2}(\mathbf{\Lambda}^{(2)})^{1 / 2}(\mathbf{Q}^{(2)})^{\top}$  and apply the cyclic property of trace in (25):
$$
\begin{array}{l} \frac {\partial^ {2}}{(\partial t) ^ {2}} \mathcal {F} (\mathbf {E} + t \boldsymbol {\Delta}) \bigg | _ {t = 0} = 2 \beta_ {2} \operatorname {T r a c e} \left((\boldsymbol {\Lambda} ^ {(2)}) ^ {1 / 2} (\mathbf {Q} ^ {(2)}) ^ {\top} \mathbf {E} ^ {- 1} \boldsymbol {\Delta} \mathbf {E} ^ {- 1} \boldsymbol {\Delta} \mathbf {E} ^ {- 1} \mathbf {Q} ^ {(2)} (\boldsymbol {\Lambda} ^ {(2)}) ^ {1 / 2}\right) \\ = 2 \beta_ {2} \operatorname {T r a c e} \left(\mathbf {T E} ^ {- 1} \mathbf {T} ^ {\top}\right) \\ \end{array}
$$