SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\| \cdot \|_{\mathrm{F}}$  denotes Frobenius norm and  $\| \mathbf{E}\|_{\mathrm{F}},\| \boldsymbol{\Lambda}^{(3)}\|_{\mathrm{F}}\neq 0$  since these are nonzero matrices. Further, we assume that diagonal entries of  $\widetilde{\boldsymbol{\Lambda}}^{(3)}$  are sorted in a descending order and, in addition to  $\boldsymbol{\Lambda}^{(3)},\pi$  also sorts entries of  $\widetilde{\mathbf{E}}$  in a non-ascending (descending) order. Clearly, such  $\widetilde{\mathbf{E}},\widetilde{\boldsymbol{\Lambda}}^{(3)}$  can be obtained by small perturbations of  $\mathbf{E},\boldsymbol{\Lambda}^{(3)}$ . Also, denote  $\widetilde{\mathbf{N}} = \mathbf{Q}^{(3)}\widetilde{\boldsymbol{\Lambda}}^{(3)}(\mathbf{Q}^{(3)})^{\top}$ . Since  $\mathbb{O}_d$  is a compact closed set and Trace  $(\mathbf{EQNQ}^{\top})$  is a continuous function of  $\mathbf{Q}$ , there exists  $\mathbf{Q}^{**}\in \mathbb{O}_d$  such that
$$
\operatorname {T r a c e} \left(\mathbf {E Q} ^ {* *} \mathbf {N} \left(\mathbf {Q} ^ {* *}\right) ^ {\top}\right) = \sup  _ {\mathbf {Q} \in \mathbb {O} _ {d}} \operatorname {T r a c e} \left(\mathbf {E Q N Q} ^ {\top}\right). \tag {48}
$$
By the definition of  $\widetilde{\mathbf{E}},\widetilde{\boldsymbol{\Lambda}}^{(3)},\widetilde{\mathbf{N}}$  , we have:
$$
\begin{array}{l} \operatorname {T r a c e} \left(\mathbf {E Q} ^ {* *} \mathbf {N} (\mathbf {Q} ^ {* *}) ^ {\top}\right) - \operatorname {T r a c e} \left(\widetilde {\mathbf {E}} \mathbf {Q} ^ {* *} \widetilde {\mathbf {N}} (\mathbf {Q} ^ {* *}) ^ {\top}\right) \\ = \left(\mathrm {T r a c e} \left(\mathbf {E Q} ^ {* *} \mathbf {N} (\mathbf {Q} ^ {* *}) ^ {\top}\right) - \mathrm {T r a c e} \left(\mathbf {E Q} ^ {* *} \widetilde {\mathbf {N}} (\mathbf {Q} ^ {* *}) ^ {\top}\right)\right) + \left(\mathrm {T r a c e} \left(\mathbf {E Q} ^ {* *} \widetilde {\mathbf {N}} (\mathbf {Q} ^ {* *}) ^ {\top}\right) \right. \\ \end{array}
$$
$$
\begin{array}{l} - \operatorname {T r a c e} \left(\widetilde {\mathbf {E}} \mathbf {Q} ^ {* *} \widetilde {\mathbf {N}} (\mathbf {Q} ^ {* *}) ^ {\top}\right) \\ = \operatorname {T r a c e} \left(\mathbf {E Q} ^ {* *} \left(\mathbf {N} - \widetilde {\mathbf {N}}\right) \left(\mathbf {Q} ^ {* *}\right) ^ {\top}\right) + \operatorname {T r a c e} \left(\left(\mathbf {E} - \widetilde {\mathbf {E}}\right) \mathbf {Q} ^ {* *} \widetilde {\mathbf {N}} \left(\mathbf {Q} ^ {* *}\right) ^ {\top}\right). \\ \end{array}
$$
Next, we apply Cauchy-Schwarz inequality to both terms:
$$
\operatorname {T r a c e} \left(\mathbf {E Q} ^ {* *} \left(\mathbf {N} - \widetilde {\mathbf {N}}\right) \left(\mathbf {Q} ^ {* *}\right) ^ {\top}\right) \leq \| (\mathbf {Q} ^ {* *}) ^ {\top} \mathbf {E} \| _ {\mathrm {F}} \| (\mathbf {N} - \widetilde {\mathbf {N}}) (\mathbf {Q} ^ {* *}) ^ {\top} \| _ {\mathrm {F}} = \| \mathbf {E} \| _ {\mathrm {F}} \| \mathbf {N} - \widetilde {\mathbf {N}} \| _ {\mathrm {F}},
$$
$$
\operatorname {T r a c e} \left(\left(\mathbf {E} - \widetilde {\mathbf {E}}\right) \mathbf {Q} ^ {* *} \widetilde {\mathbf {N}} (\mathbf {Q} ^ {* *}) ^ {\top}\right) \leq \| (\mathbf {Q} ^ {* *}) ^ {\top} (\mathbf {E} - \widetilde {\mathbf {E}}) \| _ {\mathrm {F}} \| \widetilde {\mathbf {N}} (\mathbf {Q} ^ {* *}) ^ {\top} \| _ {\mathrm {F}} = \| \mathbf {E} - \widetilde {\mathbf {E}} \| _ {\mathrm {F}} \| \widetilde {\mathbf {N}} \| _ {\mathrm {F}}
$$
where we use invariance of the Frobenius norm under multiplications by orthogonal matrices. Using this invariance again, we deduce that
$$
\begin{array}{l} \| \mathbf {N} - \widetilde {\mathbf {N}} \| _ {\mathrm {F}} = \| \mathbf {Q} ^ {(3)} (\boldsymbol {\Lambda} ^ {(3)} - \widetilde {\boldsymbol {\Lambda}} ^ {(3)}) (\mathbf {Q} ^ {(3)}) ^ {\top} \| _ {\mathrm {F}} = \| \boldsymbol {\Lambda} ^ {(3)} - \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {\mathrm {F}}, \\ \| \widetilde {\mathbf {N}} \| _ {\mathrm {F}} = \| \mathbf {Q} ^ {(3)} \widetilde {\boldsymbol {\Lambda}} ^ {(3)} (\mathbf {Q} ^ {(3)}) ^ {\top} \| _ {\mathrm {F}} = \| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {\mathrm {F}}. \\ \end{array}
$$
$$
\operatorname {T r a c e} \left(\mathbf {E Q} ^ {* *} \mathbf {N} \left(\mathbf {Q} ^ {* *}\right) ^ {\top}\right) \leq \operatorname {T r a c e} \left(\widetilde {\mathbf {E}} \mathbf {Q} ^ {* *} \widetilde {\mathbf {N}} \left(\mathbf {Q} ^ {* *}\right) ^ {\top}\right) + \| \mathbf {E} \| _ {\mathrm {F}} \| \boldsymbol {\Lambda} ^ {(3)} - \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {\mathrm {F}} + \| \mathbf {E} - \widetilde {\mathbf {E}} \| _ {\mathrm {F}} \| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {\mathrm {F}}. \tag {49}
$$
Next, we apply Lemma A.3 to  $\mathbf{E} = \widehat{\mathbf{E}}$ ,  $\boldsymbol{\Lambda}^{(3)} = \widehat{\boldsymbol{\Lambda}}^{(3)}$  and deduce that
$$
\begin{array}{l} \operatorname {T r a c e} \left(\widetilde {\mathbf {E}} \mathbf {Q} ^ {* *} \widetilde {\mathbf {N}} (\mathbf {Q} ^ {* *}) ^ {\top}\right) \leq \sum_ {l = 1} ^ {d} \widetilde {\mathbf {E}} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \widetilde {\boldsymbol {\Lambda}} _ {l, l} ^ {(3)} = \sum_ {l = 1} ^ {d} \left(\mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + \left(\widetilde {\mathbf {E}} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} - \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}}\right) \boldsymbol {\Lambda} _ {l, l} ^ {(3)} \right. \\ \left. + \widetilde {\mathbf {E}} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \left(\widetilde {\boldsymbol {\Lambda}} _ {l, l} ^ {(3)} - \boldsymbol {\Lambda} _ {l, l} ^ {(3)}\right)\right) \\ = \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + \sum_ {l = 1} ^ {d} \left(\widetilde {\mathbf {E}} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} - \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}}\right) \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + \sum_ {l = 1} ^ {d} \widetilde {\mathbf {E}} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \left(\widetilde {\boldsymbol {\Lambda}} _ {l, l} ^ {(3)} - \boldsymbol {\Lambda} _ {l, l} ^ {(3)}\right) \\ = \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + \mathrm {T r a c e} \left(\boldsymbol {\Pi} (\widetilde {\mathbf {E}} - \mathbf {E}) \boldsymbol {\Pi} ^ {\top} \widetilde {\boldsymbol {\Lambda}} ^ {(3)}\right) + \mathrm {T r a c e} \left(\boldsymbol {\Pi} \widetilde {\mathbf {E}} \boldsymbol {\Pi} ^ {\top} (\widetilde {\boldsymbol {\Lambda}} ^ {(3)} - \boldsymbol {\Lambda} ^ {(3)})\right). \\ \end{array}
$$
We apply Cauchy-Schwarz inequality again to the second and the third term:
$$
\operatorname {T r a c e} \left(\boldsymbol {\Pi} (\widetilde {\mathbf {E}} - \mathbf {E}) \boldsymbol {\Pi} ^ {\top} \widetilde {\boldsymbol {\Lambda}} ^ {(3)}\right) \leq \| (\widetilde {\mathbf {E}} - \mathbf {E}) \boldsymbol {\Pi} ^ {\top} \| _ {\mathrm {F}} \| \boldsymbol {\Pi} ^ {\top} \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {\mathrm {F}} = \| \widetilde {\mathbf {E}} - \mathbf {E} \| _ {\mathrm {F}} \| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {\mathrm {F}},
$$
$$
\operatorname {T r a c e} \left(\boldsymbol {\Pi} \widetilde {\boldsymbol {\mathbf {E}}} \boldsymbol {\Pi} ^ {\top} \left(\widetilde {\boldsymbol {\Lambda}} ^ {(3)} - \boldsymbol {\Lambda} ^ {(3)}\right)\right) \leq \| \widetilde {\boldsymbol {\mathbf {E}}} \boldsymbol {\Pi} ^ {\top} \| _ {\mathrm {F}} \| \boldsymbol {\Pi} ^ {\top} \left(\widetilde {\boldsymbol {\Lambda}} ^ {(3)} - \boldsymbol {\Lambda} ^ {(3)}\right) \| _ {\mathrm {F}} = \| \widetilde {\boldsymbol {\mathbf {E}}} \| _ {\mathrm {F}} \| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} - \boldsymbol {\Lambda} ^ {(3)} \| _ {\mathrm {F}}
$$
where we use invariance of the Frobenius norm under column and row permutations. We conclude that
$$
\mathrm {T r a c e} \left(\widetilde {\mathbf {E}} \mathbf {Q} ^ {* *} \widetilde {\mathbf {N}} (\mathbf {Q} ^ {* *}) ^ {\top}\right) \leq \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + \| \widetilde {\mathbf {E}} - \mathbf {E} \| _ {\mathrm {F}} \| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {\mathrm {F}} + \| \widetilde {\mathbf {E}} \| _ {\mathrm {F}} \| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} - \boldsymbol {\Lambda} ^ {(3)} \| _ {\mathrm {F}}.
$$
We combine this inequality with (49) and obtain:
$$
\operatorname {T r a c e} \left(\mathbf {E Q} ^ {* *} \mathbf {N} \left(\mathbf {Q} ^ {* *}\right) ^ {\top}\right) \leq \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + 2 \| \mathbf {E} \| _ {\mathrm {F}} \| \boldsymbol {\Lambda} ^ {(3)} - \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {\mathrm {F}} + 2 \| \mathbf {E} - \widetilde {\mathbf {E}} \| _ {\mathrm {F}} \| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {\mathrm {F}}. \tag {50}
$$
Next, we use triangle inequality and deduce that
$$
\| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {F} \leq \| \boldsymbol {\Lambda} ^ {(3)} \| _ {F} + \| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} - \boldsymbol {\Lambda} ^ {(3)} \| _ {F}.
$$
Hence, we continue (50):
$$
\mathrm {T r a c e} \left(\mathbf {E Q} ^ {* *} \mathbf {N} (\mathbf {Q} ^ {* *}) ^ {\top}\right) \leq \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + 2 \| \mathbf {E} \| _ {\mathrm {F}} \| \boldsymbol {\Lambda} ^ {(3)}
$$
$$
\begin{array}{l} - \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {F} + 2 \| \mathbf {E} - \widetilde {\mathbf {E}} \| _ {F} \left(\| \boldsymbol {\Lambda} ^ {(3)} \| _ {F} + \| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} - \boldsymbol {\Lambda} ^ {(3)} \| _ {F}\right) \\ = \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + 2 \| \mathbf {E} \| _ {\mathrm {F}} \| \boldsymbol {\Lambda} ^ {(3)} - \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {\mathrm {F}} + 2 \| \boldsymbol {\Lambda} ^ {(3)} \| _ {\mathrm {F}} \| \mathbf {E} - \widetilde {\mathbf {E}} \| _ {\mathrm {F}} + 2 \| \mathbf {E} - \widetilde {\mathbf {E}} \| _ {\mathrm {F}} \| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} - \boldsymbol {\Lambda} ^ {(3)} \| _ {\mathrm {F}} \\ \leq \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + 2 \| \mathbf {E} \| _ {\mathrm {F}} \| \boldsymbol {\Lambda} ^ {(3)} - \widetilde {\boldsymbol {\Lambda}} ^ {(3)} \| _ {\mathrm {F}} + 2 \| \boldsymbol {\Lambda} ^ {(3)} \| _ {\mathrm {F}} \| \mathbf {E} - \widetilde {\mathbf {E}} \| _ {\mathrm {F}} + 2 \| \mathbf {E} \| _ {\mathrm {F}} \| \widetilde {\boldsymbol {\Lambda}} ^ {(3)} - \boldsymbol {\Lambda} ^ {(3)} \| _ {\mathrm {F}} \\ \end{array}
$$
where in the last transition we use  $\| \mathbf{E} - \widetilde{\mathbf{E}}\|_{\mathrm{F}}\leq \| \mathbf{E}\|_{\mathrm{F}}$  which is according to (47). We continue this chain of inequalities using (47) again:
$$
\begin{array}{l} \operatorname {T r a c e} \left(\mathbf {E Q} ^ {* *} \mathbf {N} (\mathbf {Q} ^ {* *}) ^ {\top}\right) \leq \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + \frac {2}{1 2} \delta + \frac {2}{1 2} \delta + \frac {2}{1 2} \delta = \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + \frac {\delta}{2} \\ <   \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} _ {l}, \boldsymbol {\pi} _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} + \delta . \\ \end{array}
$$
This is a contradiction with (46) taking into account  $\mathbf{Q}^{**}$ 's definition (48). Hence, (44) is proven.
Let  $\mathbf{Q}^*$  be defined as in Lemma A.4's statement. Further, we denote  $\pi(\mathbf{A}) = \pi$ ,  $\Pi(\mathbf{A}) = \Pi$  where  $\pi, \Pi$  are defined as in Lemma A.4's statement. That is,  $\pi(\mathbf{A})$  denotes some permutation which sorts diagonal entries of  $\mathbf{E}$  in a non-ascending order. It's a function of  $\mathbf{A}$  since  $\mathbf{E}$  is a function of  $\mathbf{A}$  defined in (35). In fact, based on (35), we see that  $\pi(\mathbf{A})$  is some permutation which sorts diagonal entries of  $\mathbf{A}$  in a non-descending order.  $\Pi(\mathbf{A})$  denotes a permutation matrix corresponding to  $\pi(\mathbf{A})$ . That is, diagonal entries of  $\Pi(\mathbf{A})\mathbf{A}\Pi(\mathbf{A})^\top$  are sorted in a non-descending order.
Let  $\mathcal{G}(\mathbf{A})$  denote the right hand side of (37) where we substitute  $\mathbf{Q} = \mathbf{Q}^*$ . That is,  $\mathcal{G}(\mathbf{A})$  is an optimal value of  $\overline{\mathcal{L}}(\theta_{\mathrm{SDE}}; \mathcal{X}, \mathcal{Y}, \mathcal{T}_{\mathrm{SDE}})$  with  $\mathbf{A}$  fixed:
$$
\begin{array}{l} \mathcal {G} (\mathbf {A}) = \log \det  (\mathbf {I} _ {d} - 4 \mathbf {A}) - \frac {1}{2} \log \det  (\mathbf {I} _ {d} - 8 \mathbf {A}) - L ^ {- 1} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} - L ^ {- 1} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} \\ - 2 \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} (\mathbf {A}) _ {l}, \boldsymbol {\pi} (\mathbf {A}) _ {l}} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} \\ = \log \det  (\mathbf {I} _ {d} - 4 \mathbf {A}) - \frac {1}{2} \log \det  (\mathbf {I} _ {d} - 8 \mathbf {A}) - L ^ {- 1} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} - L ^ {- 1} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} \\ + \sum_ {l = 1} ^ {d} \left(1 + \left(1 - 8 \mathbf {A} _ {\boldsymbol {\pi} (\mathbf {A}) _ {l}, \boldsymbol {\pi} (\mathbf {A}) _ {l}}\right) ^ {- 1}\right) \boldsymbol {\Lambda} _ {l, l} ^ {(3)} \tag {51} \\ \end{array}
$$
where we use  $\mathbf{E}$ 's definition (35). Let  $\pi^{-1}(\mathbf{A}) \in \mathbb{N}^d$  denote a permutation inverse to  $\pi(\mathbf{A})$ . By rearranging terms in the sum, we have:
$$
\sum_ {l = 1} ^ {d} \left(1 + \left(1 - 8 \mathbf {A} _ {\boldsymbol {\pi} (\mathbf {A}) _ {l}, \boldsymbol {\pi} (\mathbf {A}) _ {l}}\right) ^ {- 1}\right) \boldsymbol {\Lambda} _ {l, l} ^ {(3)} = \sum_ {l = 1} ^ {d} \left(1 + \left(1 - 8 \mathbf {A} _ {l, l}\right) ^ {- 1}\right) \boldsymbol {\Lambda} _ {\boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}, \boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)}.
$$
$$
\begin{array}{l} \mathcal {G} (\mathbf {A}) = \log \det  (\mathbf {I} _ {d} - 4 \mathbf {A}) - \frac {1}{2} \log \det  (\mathbf {I} _ {d} - 8 \mathbf {A}) - L ^ {- 1} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} - L ^ {- 1} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} \\ + \sum_ {l = 1} ^ {d} \left(1 + (1 - 8 \mathbf {A} _ {l, l}) ^ {- 1}\right) \Lambda_ {\boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}, \boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)}. \tag {52} \\ \end{array}
$$
Define a new function  $\mathcal{G}(\widehat{\mathbf{A}},\mathbf{A})$  , where  $\widehat{\mathbf{A}}\in \mathbb{D}_d$  satisfies  $8\widehat{\mathbf{A}}\prec \mathbf{I}_d$  , as follows:
$$
\mathcal {G} (\widehat {\mathbf {A}}, \mathbf {A}) = \log \det (\mathbf {I} _ {d} - 4 \widehat {\mathbf {A}}) - \frac {1}{2} \log \det (\mathbf {I} _ {d} - 8 \widehat {\mathbf {A}}) + \sum_ {l = 1} ^ {d} (1 - 8 \widehat {\mathbf {A}} _ {l, l}) ^ {- 1} \boldsymbol {\Lambda} _ {\boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}, \boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)}.
$$
By the definition of  $\mathcal{G}(\widehat{\mathbf{A}},\mathbf{A})$  , we have:
$$
\mathcal {G} (\mathbf {A}) = \mathcal {G} (\mathbf {A}, \mathbf {A}) - L ^ {- 1} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} - L ^ {- 1} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} + \sum_ {l = 1} ^ {d} \boldsymbol {\Lambda} _ {\boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}, \boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)}.
$$
Hence, it holds:
$$
\mathcal {G} (\mathbf {A}) \geq \inf  _ {\widehat {\mathbf {A}} \in \mathbb {D} _ {d}, 8 \widehat {\mathbf {A}} \prec \mathbf {I} _ {d}} \mathcal {G} (\widehat {\mathbf {A}}, \mathbf {A}) - L ^ {- 1} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} - L ^ {- 1} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} + \sum_ {l = 1} ^ {d} \boldsymbol {\Lambda} _ {\pi^ {- 1} (\mathbf {A}) _ {l}, \boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)}. \tag {53}
$$
Next, we show that there is a closed-form expression for the solution of  $\inf_{\widehat{\mathbf{A}}\in \mathbb{D}_d,8\widehat{\mathbf{A}}\prec \mathbf{I}_d}\mathcal{G}(\widehat{\mathbf{A}},\mathbf{A})$  Since  $\widehat{\mathbf{A}}\in \mathbb{D}_d$ , we have:  $\log \det (\mathbf{I}_d - 4\widehat{\mathbf{A}}) = \sum_{l = 1}^{d}\log (1 - 4\widehat{\mathbf{A}}_{l,l}),\log \det (\mathbf{I}_d - 8\widehat{\mathbf{A}}) = \sum_{l = 1}^{d}\log (1 - 8\widehat{\mathbf{A}}_{l,l})$ . We further have:
$$
\mathcal {G} (\widehat {\mathbf {A}}, \mathbf {A}) = \sum_ {l = 1} ^ {d} \left(\log (1 - 4 \widehat {\mathbf {A}} _ {l, l}) - \frac {1}{2} \log (1 - 8 \widehat {\mathbf {A}} _ {l, l}) + (1 - 8 \widehat {\mathbf {A}} _ {l, l}) ^ {- 1} \boldsymbol {\Lambda} _ {\boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}, \boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)}\right). \tag {54}
$$
From (54), we see that minimization  $\inf_{\hat{\mathbf{A}}\in \mathbb{D}_d,8\hat{\mathbf{A}}\prec \mathbf{I}_d}\mathcal{G}(\widehat{\mathbf{A}},\mathbf{A})$  reduces to  $d$  independent minimization problems with respect to  $\widehat{\mathbf{A}}_{l,l}$  such that  $8\widehat{\mathbf{A}}_{l,l} < 1$ .  $l$ th problem,  $1\leq l\leq d$ , is solved using Lemma A.1 where we set  $\phi = \Lambda_{\pi^{-1}(\mathbf{A})_l,\pi^{-1}(\mathbf{A})_l}^{(3)}$ . Let  $\mathbf{A}^{**}\in \mathbb{D}_d$  denote the corresponding solution. Then, for all  $1\leq l\leq d$ , we have:
$$
\mathbf {A} _ {l, l} ^ {* *} = \frac {1}{1 6} \left(1 - 2 \Lambda_ {\pi^ {- 1} (\mathbf {A}) _ {l}, \pi^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)} - \sqrt {\left(2 \Lambda_ {\pi^ {- 1} (\mathbf {A}) _ {l} , \pi^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)} + 1\right) ^ {2} + 8 \Lambda_ {\pi^ {- 1} (\mathbf {A}) _ {l} , \pi^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)}}\right). \tag {55}
$$
From (53) it follows that
$$
\begin{array}{l} \mathcal {G} (\mathbf {A}) \geq \mathcal {G} (\mathbf {A} ^ {* *}, \mathbf {A}) - L ^ {- 1} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} - L ^ {- 1} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} + \sum_ {l = 1} ^ {d} \boldsymbol {\Lambda} _ {\boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}, \boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)} \\ = \log \det (\mathbf {I} _ {d} - 4 \mathbf {A} ^ {* *}) - \frac {1}{2} \log \det (\mathbf {I} _ {d} - 8 \mathbf {A} ^ {* *}) - L ^ {- 1} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} - L ^ {- 1} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} \\ + \sum_ {l = 1} ^ {d} \left(1 + \left(1 - 8 \mathbf {A} _ {l, l} ^ {* *}\right) ^ {- 1}\right) \boldsymbol {\Lambda} _ {\boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}, \boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)}. \tag {56} \\ \end{array}
$$
Denote  $\mathbf{E}^{**} = -\frac{1}{2}\mathbf{I}_d - \frac{1}{2} (\mathbf{I}_d - 8\mathbf{A}^{**})^{-1}$ . Then we have:
$$
\begin{array}{l} \sum_ {l = 1} ^ {d} \left(1 + \left(1 - 8 \mathbf {A} _ {l, l} ^ {* *}\right) ^ {- 1}\right) \boldsymbol {\Lambda} _ {\boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}, \boldsymbol {\pi} ^ {- 1} (\mathbf {A}) _ {l}} ^ {(3)} = - 2 \operatorname {T r a c e} \left(\mathbf {E} ^ {* *} \boldsymbol {\Pi} (\mathbf {A}) ^ {- 1} \boldsymbol {\Lambda} ^ {(3)} \left(\boldsymbol {\Pi} (\mathbf {A}) ^ {- 1}\right) ^ {\top}\right) \\ \leq - 2 \sum_ {l = 1} ^ {d} \mathbf {E} _ {\boldsymbol {\pi} (\mathbf {A} ^ {* *}) _ {l}, \boldsymbol {\pi} (\mathbf {A} ^ {* *}) _ {l}} ^ {* *} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} \tag {57} \\ \end{array}
$$
where the second transition follows from Lemma A.4 and the fact that  $\pi (\mathbf{A}^{**})$  sorts diagonal entries of  $\mathbf{A}^{**}$  in a non-descending order, hence its sorts diagonal entries of  $\mathbf{E}^{**}$  in a non-ascending order (recall the definition of  $\pi (\mathbf{A}^{**})$  and  $\mathbf{E}^{**}$ ). Denote  $\mathbf{E}^* = \Pi (\mathbf{A}^{**})\mathbf{E}^{**}\Pi (\mathbf{A}^{**})^\top$ . Then  $\mathbf{E}_{\pi (\mathbf{A}^{**})_l,\pi (\mathbf{A}^{**})_l}^{**} = \mathbf{E}_{l,l}^*$  for all  $1\leq l\leq d$  and
$$
\sum_ {l = 1} ^ {d} \mathbf {E} _ {\pi (\mathbf {A} ^ {* *}) _ {l}, \pi (\mathbf {A} ^ {* *}) _ {l}} ^ {* *} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} = \sum_ {l = 1} ^ {d} \mathbf {E} _ {l, l} ^ {*} \boldsymbol {\Lambda} _ {l, l} ^ {(3)}. \tag {58}
$$
$$
\begin{array}{l} \mathbf {E} ^ {*} = \boldsymbol {\Pi} (\mathbf {A} ^ {* *}) \left(- \frac {1}{2} \mathbf {I} _ {d} - \frac {1}{2} (\mathbf {I} _ {d} - 8 \mathbf {A} ^ {* *}) ^ {- 1}\right) \boldsymbol {\Pi} (\mathbf {A} ^ {* *}) ^ {\top} \\ = - \frac {1}{2} \mathbf {I} _ {d} - \frac {1}{2} \left(\mathbf {I} _ {d} - 8 \boldsymbol {\Pi} \left(\mathbf {A} ^ {* *}\right) \mathbf {A} ^ {* *} \boldsymbol {\Pi} \left(\mathbf {A} ^ {* *}\right) ^ {\top}\right) ^ {- 1} \\ = - \frac {1}{2} \mathbf {I} _ {d} - \frac {1}{2} \left(\mathbf {I} _ {d} - 8 \mathbf {A} ^ {*}\right) ^ {- 1} \\ \end{array}
$$
where we denote  $\mathbf{A}^{*} = \boldsymbol{\Pi}(\mathbf{A}^{**})\mathbf{A}^{**}\boldsymbol{\Pi}(\mathbf{A}^{**})^{\top}$ , i.e.  $\mathbf{A}_{l,l}^{*} = \mathbf{A}_{\pi (\mathbf{A}^{**})_{l},\pi (\mathbf{A}^{**})_{l}}^{**}$  for all  $1\leq l\leq d$ . Given the definition of  $\mathbf{A}^{**}$  (55), for all  $1\leq l\leq d$  we have:
$$
\mathbf {A} _ {l, l} ^ {*} = \frac {1}{1 6} \left(1 - 2 \boldsymbol {\Lambda} _ {l, l} ^ {(3)} - \sqrt {\left(2 \boldsymbol {\Lambda} _ {l , l} ^ {(3)} + 1\right) ^ {2} + 8 \boldsymbol {\Lambda} _ {l , l} ^ {(3)}}\right). \tag {59}
$$
That is,  $\mathbf{A}^*$  is independent of  $\mathbf{A}$ . Based on (59), we see that smaller values of  $\boldsymbol{\Lambda}_{l,l}^{(3)}$  result in bigger values of  $\mathbf{A}_{l,l}^*$ . Since  $\boldsymbol{\Lambda}_{1,1}^{(3)},\ldots,\boldsymbol{\Lambda}_{d,d}^{(3)}$  are ordered in a non-ascending order, we deduce that  $\mathbf{A}_{1,1}^*,\ldots,\mathbf{A}_{d,d}^*$  are ordered in a non-descending order. By the definition of  $\pi(\mathbf{A}^*)$ , we then have  $\mathbf{A}_{l,l}^* = \mathbf{A}_{\pi(\mathbf{A}^*)_l,\pi(\mathbf{A}^*)_l}^*$  for all  $1 \leq l \leq d$ . Therefore,
$$
\sum_ {l = 1} ^ {d} \mathbf {E} _ {l, l} ^ {*} \boldsymbol {\Lambda} _ {l, l} ^ {(3)} = - \frac {1}{2} \sum_ {l = 1} ^ {d} (1 + (1 - 8 \mathbf {A} _ {l, l} ^ {*}) ^ {- 1}) \boldsymbol {\Lambda} _ {l, l} ^ {(3)} = - \frac {1}{2} \sum_ {l = 1} ^ {d} (1 + (1 - 8 \mathbf {A} _ {\boldsymbol {\pi} (\mathbf {A} ^ {*}) _ {l}, \boldsymbol {\pi} (\mathbf {A} ^ {*}) _ {l}} ^ {*}) ^ {- 1}) \boldsymbol {\Lambda} _ {l, l} ^ {(3)}.
$$
Combining this with (58), (57), we can continue the chain of inequalities (56):
$$
\begin{array}{l} \mathcal {G} (\mathbf {A}) \geq \log \det (\mathbf {I} _ {d} - 4 \mathbf {A} ^ {* *}) - \frac {1}{2} \log \det (\mathbf {I} _ {d} - 8 \mathbf {A} ^ {* *}) - L ^ {- 1} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} - L ^ {- 1} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} \\ + \sum_ {l = 1} ^ {d} \left(1 + \left(1 - 8 \mathbf {A} _ {\boldsymbol {\pi} (\mathbf {A} ^ {*}) _ {l}, \boldsymbol {\pi} (\mathbf {A} ^ {*}) _ {l}} ^ {*} \right) ^ {- 1}\right) \boldsymbol {\Lambda} _ {l, l} ^ {(3)} \\ = \log \det (\mathbf {I} _ {d} - 4 \mathbf {A} ^ {*}) - \frac {1}{2} \log \det (\mathbf {I} _ {d} - 8 \mathbf {A} ^ {*}) - L ^ {- 1} \sum_ {i = 1} ^ {L} \| \mathbf {x} ^ {(i)} \| ^ {2} - L ^ {- 1} \sum_ {j = 1} ^ {L} \| \mathbf {y} ^ {(j)} \| ^ {2} \\ + \sum_ {l = 1} ^ {d} \left(1 + \left(1 - 8 \mathbf {A} _ {\boldsymbol {\pi} (\mathbf {A} ^ {*}) _ {l}, \boldsymbol {\pi} (\mathbf {A} ^ {*}) _ {l}} ^ {*}\right) ^ {- 1}\right) \boldsymbol {\Lambda} _ {l, l} ^ {(3)} = \mathcal {G} (\mathbf {A} ^ {*}) \tag {60} \\ \end{array}
$$
where in the second transition we use the fact that