SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathbf{T} = (\boldsymbol{\Lambda}^{(2)})^{1/2}(\mathbf{Q}^{(2)})^\top \mathbf{E}^{-1}\boldsymbol{\Delta}$ . Since  $\mathbf{E}$  is positive definite,  $\mathbf{E}^{-1}$  is also positive definite and  $\mathbf{TE}^{-1}\mathbf{T}^\top$  is at least positive semidefinite. Hence,  $\operatorname{Trace}\left(\mathbf{TE}^{-1}\mathbf{T}^\top\right) \geq 0$  and also  $\frac{\partial^2}{(\partial t)^2}\mathcal{F}(\mathbf{E} + t\boldsymbol{\Delta})|_{t=0} \geq 0$ . We conclude that  $\mathcal{F}(\mathbf{E})$  is a convex function on  $\{\mathbf{E} \in \mathbb{S}_d | \mathbf{E} \succ 0\}$ . Since  $\{\mathbf{E} \in \mathbb{S}_d | \mathbf{E} \succ 0\}$  is an open set, (every) global minimum  $\mathbf{E}$  of (23) satisfies two conditions
$$
1) \mathbf {E} \succ 0, \quad \text {a n d} \quad 2) \nabla \mathcal {F} (\mathbf {E}) = \mathbf {0} _ {d \times d} \tag {26}
$$
Set  $t = 1$  and assume that  $\Delta \in \mathbb{S}_d$  is small enough by norm so that  $\mathbf{E} + \Delta$  is invertible and the Neumann series for  $(\mathbf{I}_d + \Delta \mathbf{E}^{-1})^{-1}$  is convergent. Then, (24) holds for  $t = 1$ :
$$
\begin{array}{l} \mathcal {F} (\mathbf {E} + \boldsymbol {\Delta}) = \beta_ {1} \operatorname {T r a c e} \left(\mathbf {E M} ^ {(1)}\right) + \beta_ {1} \operatorname {T r a c e} \left(\boldsymbol {\Delta} \mathbf {M} ^ {(1)}\right) + \beta_ {2} \sum_ {l = 0} ^ {\infty} (- 1) ^ {l} \operatorname {T r a c e} \left(\mathbf {E} ^ {- 1} \left(\boldsymbol {\Delta} \mathbf {E} ^ {- 1}\right) ^ {l} \mathbf {M} ^ {(2)}\right) \\ = \mathcal {F} (\mathbf {E}) + \beta_ {1} \operatorname {T r a c e} \left(\boldsymbol {\Delta} \mathbf {M} ^ {(1)}\right) + \beta_ {2} \sum_ {l = 1} ^ {\infty} (- 1) ^ {l} \operatorname {T r a c e} \left(\mathbf {E} ^ {- 1} \left(\boldsymbol {\Delta} \mathbf {E} ^ {- 1}\right) ^ {l} \mathbf {M} ^ {(2)}\right) \\ = \mathcal {F} (\mathbf {E}) + \beta_ {1} \operatorname {T r a c e} \left(\boldsymbol {\Delta} \mathbf {M} ^ {(1)}\right) - \beta_ {2} \operatorname {T r a c e} \left(\mathbf {E} ^ {- 1} \boldsymbol {\Delta} \mathbf {E} ^ {- 1} \mathbf {M} ^ {(2)}\right) \\ + \beta_ {2} \sum_ {l = 2} ^ {\infty} (- 1) ^ {l} \mathrm {T r a c e} \left(\mathbf {E} ^ {- 1} (\boldsymbol {\Delta} \mathbf {E} ^ {- 1}) ^ {l} \mathbf {M} ^ {(2)}\right). \\ \end{array}
$$
Clearly,  $\beta_{2}\sum_{l = 2}^{\infty}(-1)^{l}\mathrm{Trace}\left(\mathbf{E}^{-1}(\pmb {\Delta}\mathbf{E}^{-1})^{l}\mathbf{M}^{(2)}\right) = o(\| \pmb {\Delta}\|)$  where  $\| \cdot \|$  is an  $L_{2}$ -norm. Also, using the cyclic property of trace, we get:
$$
\operatorname {T r a c e} \left(\mathbf {E} ^ {- 1} \boldsymbol {\Delta} \mathbf {E} ^ {- 1} \mathbf {M} ^ {(2)}\right) = \operatorname {T r a c e} \left(\boldsymbol {\Delta} \mathbf {E} ^ {- 1} \mathbf {M} ^ {(2)} \mathbf {E} ^ {- 1}\right).
$$
$$
\mathcal {F} (\mathbf {E} + \boldsymbol {\Delta}) = \operatorname {T r a c e} \left(\boldsymbol {\Delta} \left(\beta_ {1} \mathbf {M} ^ {(1)} - \beta_ {2} \mathbf {E} ^ {- 1} \mathbf {M} ^ {(2)} \mathbf {E} ^ {- 1}\right)\right) + o (\| \boldsymbol {\Delta} \|). \tag {27}
$$
Since  $\pmb{\Delta},\mathbf{E}^{-1},\mathbf{M}^{(1)},\mathbf{M}^{(2)}\in \mathbb{S}_d$  , from (27) it follows that
$$
\nabla \mathcal {F} (\mathbf {E}) = \beta_ {1} \mathbf {M} ^ {(1)} - \beta_ {2} \mathbf {E} ^ {- 1} \mathbf {M} ^ {(2)} \mathbf {E} ^ {- 1}. \tag {28}
$$
Let  $\mathbf{E}^{*} = (\mathbf{B}^{(1)*})^{\top}\mathbf{B}^{(1)*}\succeq 0$ . Note that
$$
\sqrt [ 4 ]{\frac {\beta_ {2}}{\beta_ {1}}} = \sqrt [ 4 ]{\frac {(1 - 8 A) ^ {- 1} (1 - 4 A)}{(1 - 8 A) ^ {- 1} (1 - 4 A) ^ {- 1}}} = \sqrt {1 - 4 A}.
$$
Since  $\sqrt{\beta_2 / \beta_1} \neq 0$ ,  $\pmb{\Sigma}$ ,  $\mathbf{U}$ ,  $\Lambda^{-1/2}$ ,  $\mathbf{Q}^{(1)}$  are full-rank,  $\mathbf{E}^*$  is also full-rank, therefore  $\mathbf{E}^* \succ 0$  and it satisfies condition 1 from (26). Observe that
$$
\mathbf {E} ^ {*} \mathbf {Q} ^ {(1)} (\boldsymbol {\Lambda} ^ {(1)}) ^ {1 / 2} = \sqrt {\beta_ {2} / \beta_ {1}} \mathbf {Q} ^ {(1)} (\boldsymbol {\Lambda} ^ {(1)}) ^ {- 1 / 2} \mathbf {U} \boldsymbol {\Sigma} \mathbf {U} ^ {\top} (\boldsymbol {\Lambda} ^ {(1)}) ^ {- 1 / 2} (\mathbf {Q} ^ {(1)}) ^ {\top} \mathbf {Q} ^ {(1)} (\boldsymbol {\Lambda} ^ {(1)}) ^ {1 / 2}
$$
$$
\begin{array}{l} = \sqrt {\beta_ {2} / \beta_ {1}} \mathbf {Q} ^ {(1)} (\mathbf {A} ^ {(1)}) ^ {- 1 / 2} \mathbf {U} \boldsymbol {\Sigma} \mathbf {U} ^ {\top} \\ = \sqrt {\beta_ {2} / \beta_ {1}} \mathbf {Q} ^ {(1)} (\mathbf {\Lambda} ^ {(1)}) ^ {- 1 / 2} (\mathbf {U} \boldsymbol {\Sigma} \mathbf {V} ^ {\top}) \mathbf {V} \mathbf {U} ^ {\top} \\ = \sqrt {\beta_ {2} / \beta_ {1}} \mathbf {Q} ^ {(1)} (\boldsymbol {\Lambda} ^ {(1)}) ^ {- 1 / 2} \left(\left(\boldsymbol {\Lambda} ^ {(1)}\right) ^ {\frac {1}{2}} \left(\mathbf {Q} ^ {(1)}\right) ^ {\top} \mathbf {Q} ^ {(2)} \left(\boldsymbol {\Lambda} ^ {(2)}\right) ^ {\frac {1}{2}}\right) \mathbf {V} \mathbf {U} ^ {\top} \\ = \sqrt {\beta_ {2} / \beta_ {1}} \mathbf {Q} ^ {(2)} (\mathbf {A} ^ {(2)}) ^ {1 / 2} \mathbf {V U} ^ {\top} \\ \end{array}
$$
where we use definitions of  $\mathbf{E}^*$ ,  $\mathbf{U}$ ,  $\boldsymbol{\Sigma}$ ,  $\mathbf{V}$  and orthogonality of  $\mathbf{Q}^{(1)},\mathbf{Q}^{(2)}$ ,  $\mathbf{U},\mathbf{V}$ . Hence, we deduce that
$$
\begin{array}{l} \beta_ {1} \mathbf {E} ^ {*} \mathbf {M} ^ {(1)} \mathbf {E} ^ {*} = \beta_ {1} \mathbf {E} ^ {*} \mathbf {Q} ^ {(1)} \left(\boldsymbol {\Lambda} ^ {(1)}\right) ^ {1 / 2} \left(\left(\boldsymbol {\Lambda} ^ {(1)}\right) ^ {1 / 2} \left(\mathbf {Q} ^ {(1)}\right) ^ {\top} \mathbf {E} ^ {*}\right) (29) \\ = \beta_ {1} \frac {\beta_ {2}}{\beta_ {1}} \mathbf {Q} ^ {(2)} \left(\boldsymbol {\Lambda} ^ {(2)}\right) ^ {1 / 2} \left(\left(\boldsymbol {\Lambda} ^ {(2)}\right) ^ {1 / 2} \left(\mathbf {Q} ^ {(2)}\right) ^ {\top}\right) \\ = \beta_ {2} \mathbf {M} ^ {(2)} (30) \\ \end{array}
$$
by the definition of  $\mathbf{Q}^{(1)},\mathbf{\Lambda}^{(1)},\mathbf{Q}^{(2)},\mathbf{\Lambda}^{(2)}$  and due to orthogonality of  $\mathbf{V}$ ,  $\mathbf{U}$ . By left- and right-multiplication of (30) by  $(\mathbf{E}^{*})^{-1}$  we deduce that
$$
\beta_ {1} \mathbf {M} ^ {(1)} = \beta_ {2} (\mathbf {E} ^ {*}) ^ {- 1} \mathbf {M} ^ {(2)} (\mathbf {E} ^ {*}) ^ {- 1}
$$
or, in other words,  $\nabla \mathcal{F}(\mathbf{E}^{*}) = \mathbf{0}_{d\times d}$  and the condition 2 from (26) is also satisfied. We conclude that the global minimum of (23) is achieved at  $\mathbf{E}^*$ .
According to Lemma A.2,  $\mathbf{B}^{(1)} = \mathbf{B}^{(1)*}$  is a global minimum of (21)'s right hand side when  $A$  is fixed. Indeed, if there is  $\mathbf{B}^{(1)}$  which leads to a smaller value of (21),  $\mathbf{E} = (\mathbf{B}^{(1)})^\top \mathbf{B}^{(1)}$  would lead to a smaller value of (22)'s right hand side. Also, this  $\mathbf{E}$  is positive definite by definition (note that  $\mathbf{B}^{(1)}$  is nonsingular), leading to contradiction with Lemma A.2.
Substituting  $\mathbf{E}^*$  instead of  $\mathbf{E}$  in (22) corresponds to the minimum value of  $\overline{\mathcal{L}} (\theta_{\mathrm{AGE}};\alpha ,\mathcal{X},\mathcal{Y},\mathcal{T}_{\mathrm{AGE}})$  for a fixed  $A$ . Our next step is to minimize this expression with respect to  $A$ . Denote  $\mathbf{F} = \mathbf{Q}^{(1)}(\boldsymbol{\Lambda}^{(1)})^{-1 / 2}\mathbf{U}\boldsymbol{\Sigma}\mathbf{U}^{\top}(\boldsymbol{\Lambda}^{(1)})^{-1 / 2}(\mathbf{Q}^{(1)})^{\top}$ . Then  $\mathbf{E}^* = (1 - 4A)\mathbf{F}$  where  $\mathbf{F}$  doesn't depend on  $A$ . We substitute  $\mathbf{E}^*$  into (22) and get:
$$
\begin{array}{l} d \log (1 - 4 A) - \frac {d}{2} \log (1 - 8 A) + (1 - 8 A) ^ {- 1} (1 - 4 A) ^ {- 1} \operatorname {T r a c e} ((1 - 4 A) \mathbf {F M} ^ {(1)}) \\ + (1 - 8 A) ^ {- 1} (1 - 4 A) \operatorname {T r a c e} ((1 - 4 A) ^ {- 1} \mathbf {F} ^ {- 1} \mathbf {M} ^ {(2)}) \\ + 4 (1 - 8 A) ^ {- 1} (1 - 4 A) L ^ {- 2} \sum_ {1 \leq i, j \leq L} (\mathbf {x} ^ {(i)}) ^ {\top} \mathbf {y} ^ {(j)} \\ = d \log (1 - 4 A) - \frac {d}{2} \log (1 - 8 A) + (1 - 8 A) ^ {- 1} \operatorname {T r a c e} (\mathbf {F M} ^ {(1)}) + (1 - 8 A) ^ {- 1} \operatorname {T r a c e} (\mathbf {F} ^ {- 1} \mathbf {M} ^ {(2)}) \\ + 2 \left(1 + (1 - 8 A) ^ {- 1}\right) d \mu^ {(3)} \tag {31} \\ \end{array}
$$
where we also replace
$$
L ^ {- 2} \sum_ {1 \leq i, j \leq L} \left(\mathbf {x} ^ {(i)}\right) ^ {\top} \mathbf {y} ^ {(j)} = L ^ {- 2} \left(\sum_ {i = 1} ^ {L} \mathbf {x} ^ {(i)}\right) ^ {\top} \left(\sum_ {j = 1} ^ {L} \mathbf {y} ^ {(j)}\right) = d \mu^ {(3)}
$$
$$
(1 - 8 A) ^ {- 1} (1 - 4 A) = \frac {(1 - 8 A) + 1}{2 (1 - 8 A)} = \frac {1}{2} (1 + (1 - 8 A) ^ {- 1})
$$
Based on (30) and since  $\mathbf{F} = \sqrt{\beta_1 / \beta_2}\mathbf{E}^*$ , we conclude that  $\mathbf{FM}^{(1)}\mathbf{F} = \mathbf{M}^{(2)}$ , or  $\mathbf{M}^{(1)}\mathbf{F} = \mathbf{F}^{-1}\mathbf{M}^{(2)}$ . Using the cyclic property of trace, we get:
$$
\operatorname {T r a c e} \left(\mathbf {F M} ^ {(1)}\right) = \operatorname {T r a c e} \left(\mathbf {M} ^ {(1)} \mathbf {F}\right) = \operatorname {T r a c e} \left(\mathbf {F} ^ {- 1} \mathbf {M} ^ {(2)}\right).
$$
By the definition of  $\mathbf{F}$ ,  $\boldsymbol{\Lambda}^{(1)}, \mathbf{Q}^{(1)}$  and using the cyclic property and orthogonality of  $\mathbf{Q}^{(1)}, \mathbf{U}$ , we have:
$$
\operatorname {T r a c e} \left(\mathbf {F M} ^ {(1)}\right) = \operatorname {T r a c e} \left(\mathbf {Q} ^ {(1)} \left(\boldsymbol {\Lambda} ^ {(1)}\right) ^ {- 1 / 2} \mathbf {U} \boldsymbol {\Sigma} \mathbf {U} ^ {\top} \left(\boldsymbol {\Lambda} ^ {(1)}\right) ^ {- 1 / 2} \mathbf {Q} ^ {(1)}\right) ^ {\top} \left(\mathbf {Q} ^ {(1)} \boldsymbol {\Lambda} ^ {(1)} \left(\mathbf {Q} ^ {(1)}\right) ^ {\top}\right)\left. \right)
$$
$$
\begin{array}{l} = \operatorname {T r a c e} \left(\mathbf {Q} ^ {(1)} \left(\boldsymbol {\Lambda} ^ {(1)}\right) ^ {- 1 / 2} \mathbf {U} \boldsymbol {\Sigma} \mathbf {U} ^ {\top} \left(\boldsymbol {\Lambda} ^ {(1)}\right) ^ {1 / 2} \left(\mathbf {Q} ^ {(1)}\right) ^ {\top}\right) \\ = \operatorname {T r a c e} \left(\boldsymbol {\Sigma} \mathbf {U} ^ {\top} \left(\boldsymbol {\Lambda} ^ {(1)}\right) ^ {1 / 2} \left(\mathbf {Q} ^ {(1)}\right) ^ {\top} \mathbf {Q} ^ {(1)} \left(\boldsymbol {\Lambda} ^ {(1)}\right) ^ {- 1 / 2} \mathbf {U}\right) \\ = \operatorname {T r a c e} (\boldsymbol {\Sigma}) = \sum_ {l = 1} ^ {d} \boldsymbol {\Sigma} _ {l, l}. \\ \end{array}
$$
Hence, (31) finally becomes:
$$
\begin{array}{l} d \log (1 - 4 A) - \frac {d}{2} \log (1 - 8 A) + 2 (1 - 8 A) ^ {- 1} \sum_ {l = 1} ^ {d} \boldsymbol {\Sigma} _ {l, l} + 2 \left(1 + (1 - 8 A) ^ {- 1}\right) d \mu^ {(3)} \\ = d \left(\log (1 - 4 A) - \frac {1}{2} \log (1 - 8 A) + 2 (1 - 8 A) ^ {- 1} \left(d ^ {- 1} \sum_ {l = 1} ^ {d} \boldsymbol {\Sigma} _ {l, l} + \mu^ {(3)}\right) + 2 \mu^ {(3)}\right). \tag {32} \\ \end{array}
$$
Next, we use Lemma A.1 ( $\phi = d^{-1}\sum_{l=1}^{d}\pmb{\Sigma}_{l,l} + \mu^{(3)} \geq 0$ ) for deriving expression for  $A$  which minimizes (32). This expression coincides with the one in Theorem's statement. The expressions for  $\mathbf{B}^{(2)}, \mathbf{C}^{(1)}, \mathbf{C}^{(2)}$  follow directly from (20), optimal  $\mathbf{B}^{(1)} = \mathbf{B}^{(1)*}$  and  $A$ . (10) follows from (32). The proof is concluded.
Proof. With  $\mathbf{B}^{(1)} = \mathbf{B}^{(2)} = \mathbf{B}$  and  $\mathbf{C}^{(1)} = \mathbf{C}^{(2)} = \mathbf{C}$ , the conditions from Theorem 4.1 read as
$$
8 \mathbf {A} \prec \mathbf {I} _ {d}, \quad \mathbf {B} ^ {\top} \left(\mathbf {I} _ {d} - 4 \mathbf {A}\right) ^ {- 1} \mathbf {B} = \mathbf {I} _ {d}, \quad \mathbf {C} = - \frac {1}{2} \mathbf {B} ^ {\top} \left(\mathbf {I} _ {d} - 4 \mathbf {A}\right) ^ {- 1} \mathbf {B} = - \frac {1}{2} \mathbf {I} _ {d}, \quad D = \det  (\mathbf {I} _ {d} - 4 \mathbf {A}) ^ {1 / 4}. \tag {33}
$$
Denote  $\mathbf{Q} = (\mathbf{I}_d - 4\mathbf{A})^{-1/2}\mathbf{B} \in \mathbb{R}^{d \times d}$ . Then, according to (33),  $\mathbf{Q}^\top \mathbf{Q} = \mathbf{I}_d$ , that is  $\mathbf{Q} \in \mathbb{O}_d$ . We rewrite (9) using (33) and then substitute  $\mathbf{B} = (\mathbf{I}_d - 4\mathbf{A})^{1/2}\mathbf{Q}$ :
$$
\begin{array}{l} \operatorname {V a r} _ {\nu_ {\mathrm {S D E}}} f _ {\mathrm {S D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x}) f _ {\mathrm {S D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y}) = \det  (\mathbf {I} _ {d} - 4 \mathbf {A}) \det  (\mathbf {I} _ {d} - 8 \mathbf {A}) ^ {- 1 / 2} \exp \left(- \| \mathbf {x} \| ^ {2} \right. \\ \left. + 2 \mathbf {x} ^ {\top} \mathbf {B} ^ {\top} \left(\mathbf {I} _ {d} - 8 \mathbf {A}\right) ^ {- 1} \mathbf {B} \mathbf {x} - \| \mathbf {y} \| ^ {2} + 2 \mathbf {y} ^ {\top} \mathbf {B} ^ {\top} \left(\mathbf {I} _ {d} - 8 \mathbf {A}\right) ^ {- 1} \mathbf {B} \mathbf {y} + 4 \mathbf {x} ^ {\top} \mathbf {B} ^ {\top} \left(\mathbf {I} _ {d} - 8 \mathbf {A}\right) ^ {- 1} \mathbf {B} \mathbf {y}\right) \\ - K ^ {(0)} (\mathbf {x}, \mathbf {y}) ^ {2} = \det (\mathbf {I} _ {d} - 4 \mathbf {A}) ^ {1 / 4} \det (\mathbf {I} _ {d} - 8 \mathbf {A}) ^ {- 1 / 2} \exp \left(- \| \mathbf {x} \| ^ {2} - 2 \mathbf {x} ^ {\top} \mathbf {Q} ^ {\top} \mathbf {E} \mathbf {Q} \mathbf {x} - \| \mathbf {y} \| ^ {2} \right. \\ \left. - 2 \mathbf {y} ^ {\top} \mathbf {Q} ^ {\top} \mathbf {E Q y} - 4 \mathbf {x} ^ {\top} \mathbf {Q} ^ {\top} \mathbf {E Q y}\right) - K ^ {(0)} (\mathbf {x}, \mathbf {y}) ^ {2} \tag {34} \\ \end{array}
$$
where we denote:
$$
\begin{array}{l} \mathbf {E} = - (\mathbf {I} _ {d} - 4 \mathbf {A}) ^ {1 / 2} (\mathbf {I} _ {d} - 8 \mathbf {A}) ^ {- 1} (\mathbf {I} _ {d} - 4 \mathbf {A}) ^ {1 / 2} = - (\mathbf {I} _ {d} - 4 \mathbf {A}) (\mathbf {I} _ {d} - 8 \mathbf {A}) ^ {- 1} \\ = - \frac {1}{2} \left(\left(\mathbf {I} _ {d} - 8 \mathbf {A}\right) + \mathbf {I} _ {d}\right) \left(\mathbf {I} _ {d} - 8 \mathbf {A}\right) ^ {- 1} = - \frac {1}{2} \mathbf {I} _ {d} - \frac {1}{2} \left(\mathbf {I} _ {d} - 8 \mathbf {A}\right) ^ {- 1} \tag {35} \\ \end{array}
$$
which is in  $\mathbb{D}_d$  since  $\mathbf{A}\in \mathbb{D}_d$ . Next, we observe:
$$
2 \mathbf {x} ^ {\top} \mathbf {Q} ^ {\top} \mathbf {E} \mathbf {Q} \mathbf {x} + 2 \mathbf {y} ^ {\top} \mathbf {Q} ^ {\top} \mathbf {E} \mathbf {Q} \mathbf {y} + 4 \mathbf {x} ^ {\top} \mathbf {Q} ^ {\top} \mathbf {E} \mathbf {Q} \mathbf {y} = 2 (\mathbf {x} + \mathbf {y}) ^ {\top} \mathbf {Q} ^ {\top} \mathbf {E} \mathbf {Q} (\mathbf {x} + \mathbf {y})
$$
We plug this into (34) and use the resulting expression to rewrite (8) for  $\langle \nu, f^{(1)}, f^{(2)} \rangle = \langle \nu_{\mathrm{SDE}}, f_{\mathrm{SDE}}^{(1)}, f_{\mathrm{SDE}}^{(2)} \rangle$  as follows:
$$
\begin{array}{l} \overline {{\mathcal {L}}} (\boldsymbol {\theta} _ {\mathrm {S D E}}; \mathcal {X}, \mathcal {Y}, \mathcal {T} _ {\mathrm {S D E}}) = L ^ {- 2} \sum_ {1 \leq i, j \leq L} \log (\operatorname {V a r} _ {\nu_ {\mathrm {S D E}}} f _ {\mathrm {S D E}} ^ {(1)} (\boldsymbol {\omega}, \mathbf {x} ^ {(i)}) f _ {\mathrm {S D E}} ^ {(2)} (\boldsymbol {\omega}, \mathbf {y} ^ {(j)}) + K ^ {(0)} (\mathbf {x} ^ {(i)}, \mathbf {y} ^ {(j)})) \\ = \log \det (\mathbf {I} _ {d} - 4 \mathbf {A}) - \frac {1}{2} \log \det (\mathbf {I} _ {d} - 8 \mathbf {A}) - L ^ {- 1} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} - L ^ {- 1} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} \\ - 2 L ^ {- 2} \sum_ {1 \leq i, j \leq L} \left(\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}\right) ^ {\top} \mathbf {Q} ^ {\top} \mathbf {E} \mathbf {Q} \left(\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}\right). \tag {36} \\ \end{array}
$$
Using linearity and cyclic property of trace, we deduce that
$$
\begin{array}{l} L ^ {- 2} \sum_ {1 \leq i, j \leq L} (\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}) ^ {\top} \mathbf {Q} ^ {\top} \mathbf {E} \mathbf {Q} (\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}) = \\ = L ^ {- 2} \sum_ {1 \leq i, j \leq L} \operatorname {T r a c e} \left(\left(\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}\right) ^ {\top} \mathbf {Q} ^ {\top} \mathbf {E} \mathbf {Q} \left(\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}\right)\right) \\ = L ^ {- 2} \sum_ {1 \leq i, j \leq L} \operatorname {T r a c e} \left(\mathbf {Q} ^ {\top} \mathbf {E} \mathbf {Q} (\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}) (\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}) ^ {\top}\right) \\ = \operatorname {T r a c e} \left(\mathbf {Q} ^ {\top} \mathbf {E} \mathbf {Q} \left(L ^ {- 2} \sum_ {1 \leq i, j \leq L} \left(\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}\right) \left(\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}\right) ^ {\top}\right)\right) \\ \end{array}
$$
$$
\begin{array}{l} L ^ {- 2} \sum_ {1 \leq i, j \leq L} (\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}) (\mathbf {x} ^ {(i)} + \mathbf {y} ^ {(j)}) ^ {\top} = \\ = L ^ {- 2} \sum_ {1 \leq i, j \leq L} \left(\mathbf {x} ^ {(i)} (\mathbf {x} ^ {(i)}) ^ {\top} + \mathbf {x} ^ {(i)} (\mathbf {y} ^ {(j)}) ^ {\top} + \mathbf {y} ^ {(j)} (\mathbf {x} ^ {(i)}) ^ {\top} + \mathbf {y} ^ {(j)} (\mathbf {x} ^ {(j)}) ^ {\top}\right) \\ = L ^ {- 1} \sum_ {i = 1} ^ {L} \mathbf {x} ^ {(i)} \left(\mathbf {x} ^ {(i)}\right) ^ {\top} + \left(L ^ {- 1} \sum_ {i = 1} ^ {L} \mathbf {x} ^ {(i)}\right) \left(L ^ {- 1} \sum_ {j = 1} ^ {L} \mathbf {y} ^ {(j)}\right) ^ {\top} + \left(L ^ {- 1} \sum_ {j = 1} ^ {L} \mathbf {y} ^ {(j)}\right) \left(L ^ {- 1} \sum_ {i = 1} ^ {L} \mathbf {x} ^ {(i)}\right) ^ {\top} \\ + L ^ {- 1} \sum_ {j = 1} ^ {L} \mathbf {y} ^ {(j)} (\mathbf {y} ^ {(j)}) ^ {\top} = \mathbf {M} ^ {(1)} + \boldsymbol {\mu} ^ {(4)} (\boldsymbol {\mu} ^ {(5)}) ^ {\top} + \boldsymbol {\mu} ^ {(5)} (\boldsymbol {\mu} ^ {(4)}) ^ {\top} + \mathbf {M} ^ {(2)}. \\ \end{array}
$$
Denote  $\mathbf{N} = \mathbf{M}^{(1)} + \pmb{\mu}^{(4)}(\pmb{\mu}^{(5)})^{\top} + \pmb{\mu}^{(5)}(\pmb{\mu}^{(4)})^{\top} + \mathbf{M}^{(2)}$ . We conclude that
$$
\begin{array}{l} \overline {{\mathcal {L}}} (\pmb {\theta} _ {\mathrm {S D E}}; \mathcal {X}, \mathcal {Y}, \mathcal {T} _ {\mathrm {S D E}}) = \log \det (\mathbf {I} _ {d} - 4 \mathbf {A}) - \frac {1}{2} \log \det (\mathbf {I} _ {d} - 8 \mathbf {A}) - L ^ {- 1} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} \\ - L ^ {- 1} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} - 2 \operatorname {T r a c e} \left(\mathbf {Q} ^ {\top} \mathbf {E Q N}\right). \tag {37} \\ \end{array}
$$
With  $\mathbf{A}$  fixed, we minimize the right hand side of (37) with respect to  $\mathbf{Q}$  which is equivalent to minimizing  $\overline{\mathcal{L}} (\theta_{\mathrm{SDE}};\mathcal{X},\mathcal{Y},\mathcal{T}_{\mathrm{SDE}})$  with respect to  $\mathbf{B}$  with fixed  $\mathbf{A}$ , since there is a one-to-one correspondence between  $\mathbf{B}$  and  $\mathbf{Q}$ . This is equivalent to maximizing, again using the cyclic property of trace,
$$
\operatorname {T r a c e} \left(\mathbf {Q} ^ {\top} \mathbf {E Q N}\right) = \operatorname {T r a c e} \left(\mathbf {E Q N Q} ^ {\top}\right) \tag {38}
$$
with respect to  $\mathbf{Q}$ . We prove the following lemma first:
Lemma A.3. Suppose that diagonal entries of  $\mathbf{E}$  are all distinct, and the same holds for  $\boldsymbol{\Lambda}^{(3)}$ . Let  $\Pi \in \{0,1\}^{d\times d}$  be a permutation matrix sorting diagonal entries of  $\mathbf{E}$  (i.e. by applying  $\Pi \mathbf{E} \Pi^{\top}$ ) in a descending order corresponding to a permutation  $\pi \in \mathbb{N}^d$ . Set  $\mathbf{Q}^{*} = \Pi^{\top}(\mathbf{Q}^{(3)})^{\top} \in \mathbb{O}_{d}$ . Then we have:
$$
\begin{array}{l} \operatorname {T r a c e} \left(\mathbf {E Q} ^ {*} \mathbf {N} \left(\mathbf {Q} ^ {*}\right) ^ {\top}\right) = \sum_ {l = 1} ^ {d} \mathbf {E} _ {\pi_ {l}, \pi_ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} (39) \\ = \sup  _ {\mathbf {Q} \in \mathbb {O} _ {d}} \operatorname {T r a c e} \left(\mathbf {E Q N Q} ^ {\top}\right) (40) \\ \end{array}
$$
Proof. First of all, we have:
$$
\operatorname {T r a c e} \left(\mathbf {E Q} ^ {*} \mathbf {N} \left(\mathbf {Q} ^ {*}\right) ^ {\top}\right) = \operatorname {T r a c e} \left(\mathbf {E I I} ^ {\top} \left(\mathbf {Q} ^ {(3)}\right) ^ {\top} \mathbf {N Q} ^ {(3)} \boldsymbol {\Pi}\right) = \operatorname {T r a c e} \left(\mathbf {E I I} ^ {\top} \boldsymbol {\Lambda} ^ {(3)} \boldsymbol {\Pi}\right) \tag {41}
$$
$$
= \operatorname {T r a c e} \left(\boldsymbol {\Pi} \boldsymbol {\mathbf {E}} \boldsymbol {\Pi} ^ {\top} \boldsymbol {\Lambda} ^ {(3)}\right) = \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)}, \tag {42}
$$
i.e. (39) is satisfied.
Optimization for finding  $\sup_{\mathbf{Q} \in \mathbb{O}_d} \operatorname{Trace}\left(\mathbf{EQNQ}^\top\right)$  is a well-studied problem [9]. By the definition,  $\pmb{\Lambda}^{(3)}$  has eigenvalues of  $\mathbf{N}$  on the main diagonal and  $\mathbf{E} \in \mathbb{D}_d$  hence it contains its eigenvalues on its main diagonal. Then, as proven in [9],  $\mathbf{Q}^*$  is indeed a global maximum of this problem in the case of distinct eigenvalues for  $\mathbf{E}$  and  $\mathbf{N}$ . That is, (40) is proven.
Next, we prove a generalization of Lemma A.3 when diagonal entries of  $\mathbf{E}$  and  $\pmb{\Lambda}^{(3)}$  are not necessarily distinct:
Lemma A.4. Let  $\Pi \in \{0,1\}^{d\times d}$  be a permutation matrix sorting diagonal entries of  $\mathbf{E}$  (i.e. by applying  $\Pi \mathbf{E} \Pi^{\top}$ ) in any non-ascending order corresponding to a permutation  $\pi \in \mathbb{N}^d$ . Set  $\mathbf{Q}^{*} = \Pi^{\top}(\mathbf{Q}^{(3)})^{\top} \in \mathbb{O}_{d}$ . Then we have:
$$
\begin{array}{l} \operatorname {T r a c e} \left(\mathbf {E Q} ^ {*} \mathbf {N} \left(\mathbf {Q} ^ {*}\right) ^ {\top}\right) = \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} (43) \\ = \sup  _ {\mathbf {Q} \in \mathbb {O} _ {d}} \operatorname {T r a c e} \left(\mathbf {E Q N Q} ^ {\top}\right) (44) \\ \end{array}
$$
Proof. In the same way as (41-42), we show that Trace  $\left(\mathbf{E}\mathbf{Q}^{*}\mathbf{N}(\mathbf{Q}^{*})^{\top}\right) = \sum_{l=1}^{d} \mathbf{E}_{\pi_{l},\pi_{l}} \boldsymbol{\Lambda}_{l,l}^{(3)}$ , i.e. (43) is satisfied. Next we prove that for any  $\mathbf{Q} \in \mathbb{O}_{d}$ ,
$$
\operatorname {T r a c e} \left(\mathbf {E Q N Q} ^ {\top}\right) \leq \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)}. \tag {45}
$$
which would imply (44).
Our proof is by contradiction. First of all, we can assume that  $\mathbf{E},\boldsymbol{\Lambda}^{(3)}$  are nonzero matrices since otherwise we have (44) trivially. Since Trace  $\left(\mathbf{EQNQ}^{\top}\right)$  is a continuous function of  $\mathbf{Q}$  and  $\mathbb{O}_d$  is compact,  $\sup_{\mathbf{Q}\in \mathbb{O}_d}\operatorname {Trace}\left(\mathbf{EQNQ}^\top\right)$  is finite. Suppose that there is  $\delta >0$  such that
$$
\delta = \sup  _ {\mathbf {Q} \in \mathbb {O} _ {d}} \operatorname {T r a c e} \left(\mathbf {E Q N Q} ^ {\top}\right) - \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \Lambda_ {l, l} ^ {(3)}. \tag {46}
$$
Let  $\widetilde{\mathbf{E}},\widetilde{\boldsymbol{\Lambda}}^{(3)}\in \mathbb{D}_d$  be matrices with all distinct values on the diagonal such that
$$
\left\| \widetilde {\mathbf {E}} - \mathbf {E} \right\| _ {\mathrm {F}} \leq \min  \left(\left\| \mathbf {E} \right\| _ {\mathrm {F}}, \frac {\delta}{1 2 \left\| \boldsymbol {\Lambda} ^ {(3)} \right\| _ {\mathrm {F}}}\right), \quad \left\| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} - \boldsymbol {\Lambda} ^ {(3)} \right\| _ {\mathrm {F}} \leq \frac {\delta}{1 2 \left\| \mathbf {E} \right\| _ {\mathrm {F}}} \tag {47}
$$