SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now, considering  $\mathbf{J} = 2n\left(\widetilde{\boldsymbol{\Sigma}}_B - \frac{4pq}{(p + q)^2}\boldsymbol {\Sigma}_B\right)$ , we formulate  $\partial_{kjl}\widehat{\mathcal{R}}^{\mathcal{F}'}(\mathbf{H})$  as:
$$
\partial_ {k j l} \widehat {\mathcal {R}} ^ {\mathcal {F} ^ {\prime}} (\mathbf {H}) = \frac {- n}{2 N} \mathrm {T r} \left\{\partial_ {k j l} \left(\left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] [ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} ] ^ {- 1}\right) \right\} + \frac {N \lambda_ {H}}{4 n} \left(2 \mathbf {e} _ {l} ^ {\top} \mathbf {h} _ {k, j}\right)
$$
Since  $C = 2$  in our analysis, the derivative expands into:
$$
\begin{array}{l} \partial_ {k j l} \widehat {\mathcal {R}} ^ {\mathcal {F} ^ {\prime}} (\mathbf {H}) = - \frac {1}{4} \operatorname {T r} \left\{\partial_ {k j l} \left(\mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n}\right) \left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \right\} \\ - \frac {1}{4} \operatorname {T r} \left\{\left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] \partial_ {k j l} \left(\left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1}\right) \right\} + \lambda_ {H} \mathbf {e} _ {l} ^ {\top} \mathbf {h} _ {k, j} \\ \end{array}
$$
Where the second term can be expanded as:
$$
\begin{array}{l} \frac {1}{4} \mathrm {T r} \left\{\left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] \partial_ {k j l} \left(\left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1}\right) \right\} \\ = - \frac {1}{4} \operatorname {T r} \left\{\left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] \left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \partial_ {k j l} \left(\mathbf {J} + \Delta_ {2}\right) \left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \right\} \\ = - \frac {1}{4} \operatorname {T r} \left\{\left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] \left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \partial_ {k j l} \left(\mathbf {J} + \Delta_ {2}\right) \right\} \\ \end{array}
$$
Now, by expanding  $\partial_{kjl}(\mathbf{J})$  in terms of covariance matrix derivatives, we get:
$$
\begin{array}{l} \partial_ {k j l} \left(\mathbf {J}\right) = \partial_ {k j l} 2 n \left(\widetilde {\boldsymbol {\Sigma}} _ {B} - \frac {4 p q}{(p + q) ^ {2}} \boldsymbol {\Sigma} _ {B}\right) \\ = \left(\mathbf {e} _ {l} \overline {{\mathbf {h}}} _ {k} ^ {\top} + \overline {{\mathbf {h}}} _ {k} \mathbf {e} _ {l} ^ {\top}\right) - \frac {4 p q}{(p + q) ^ {2}} \left(\mathbf {e} _ {l} \left(\overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G}\right) ^ {\top} + \left(\overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G}\right) \mathbf {e} _ {l} ^ {\top}\right) \\ = \mathbf {e} _ {l} \left(\left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G}\right) ^ {\top} + \left(\left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G}\right) \mathbf {e} _ {l} ^ {\top} \\ \end{array}
$$
This leads to the following formulation for  $\partial_{kjl}\widehat{\mathcal{R}}^{\mathcal{F}'}(\mathbf{H})$ :
$$
\begin{array}{l} \partial_ {k j l} \widehat {\mathcal {R}} ^ {\mathcal {F} ^ {\prime}} (\mathbf {H}) = - \frac {1}{4} \operatorname {T r} \left\{\left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \partial_ {k j l} (\mathbf {J}) \right\} \\ + \frac {1}{4} \operatorname {T r} \left\{\left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] \left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \partial_ {k j l} (\mathbf {J}) \right\} \\ + \lambda_ {H} \mathbf {e} _ {l} ^ {\top} \mathbf {h} _ {k, j} + \mathcal {P} _ {k j l} \\ = - \frac {1}{2} \operatorname {T r} \left\{\left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \left[ \mathbf {e} _ {l} \left(\left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G}\right) ^ {\top} \right] \right\} \\ + \frac {1}{2} \operatorname {T r} \left\{\left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] \left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \right. \\ \left. \cdot \left[ \mathbf {e} _ {l} \left(\left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G}\right) ^ {\top} \right] \right\} + \lambda_ {H} \mathbf {e} _ {l} ^ {\top} \mathbf {h} _ {k, j} + \mathcal {P} _ {k j l} \\ \end{array}
$$
Where  $\mathcal{P}_{kjl}$  represents the remaining trace terms pertaining to the partial derivatives of  $\Delta_2$ ,  $\tilde{\Delta}_3$ :
$$
\begin{array}{l} \mathcal {P} _ {k j l} = - \frac {1}{4} \operatorname {T r} \left\{\left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \partial_ {k, j, l} \left(\frac {\widetilde {\Delta} _ {3}}{n}\right) \right\} \\ + \frac {1}{4} \operatorname {T r} \left\{\left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] \left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1} \partial_ {k j l} \left(\Delta_ {2}\right) \right\} \\ \end{array}
$$
We now denote  $\mathbf{M} \coloneqq [\mathbf{J} + \Delta_2 + \lambda_{W_2}N\mathbf{I}]^{-1}$  to obtain:
$$
\begin{array}{l} \partial_ {k j l} \widehat {\mathcal {R}} ^ {\mathcal {F} ^ {\prime}} (\mathbf {H}) = - \frac {1}{2} \left(\left[ \mathbf {M} - \mathbf {M} \left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] \mathbf {M} \right] \left[ \left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G} \right] - 2 \lambda_ {H} \mathbf {h} _ {k, j}\right) ^ {\top} \mathbf {e} _ {l} \\ + \mathcal {P} _ {k j l} \tag {102} \\ \end{array}
$$
Without loss of generality, since  $\mathcal{P}_{kjl} \in \mathbb{R}$ , we consider  $\mathcal{P}_{kjl} = \mathbf{p}_{k,j}^{\top} \mathbf{e}_l$  where  $\mathbf{p}_{k,j} \in \mathbb{R}^{d_{L-1}}$  is a random vector which represents the overall perturbation effect of  $\mathcal{P}_{kjl}$ . Note that the randomness is associated with the  $\mathbf{E}$  matrix in  $\Delta_2, \widetilde{\Delta}_3$ . We can now represent  $\partial_{kjl} \widehat{\mathcal{R}}^{\mathcal{F}'}(\mathbf{H}) = \langle \mathbf{R}_{k,j}, \mathbf{e}_l \rangle$ , where:
$$
\begin{array}{l} \mathbf {R} _ {k, j} = - \frac {1}{2} \left(\widetilde {\mathbf {M}} \left[ \left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G} \right] - 2 \lambda_ {H} \mathbf {h} _ {k, j} - 2 \mathbf {p} _ {k, j}\right) \tag {103} \\ \widetilde {\mathbf {M}} = \left[ \mathbf {M} - \mathbf {M} \left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] \mathbf {M} \right] \\ \end{array}
$$
- Derivative of  $\Sigma_B$ : By denoting  $\Sigma_B(a,b) = \mathbf{e}_a^\top \Sigma_B\mathbf{e}_b$  as the  $(a,b)$ -element of  $\Sigma_B$ , we use the above result for  $\partial_{kjl}\widehat{\mathcal{R}}^{\mathcal{F}'}(\mathbf{H})$  and the chain rule to compute  $\frac{d\Sigma_B}{dt}$  along the flow (14), as follows:
$$
\begin{array}{l} \frac {d \boldsymbol {\Sigma} _ {B} (a , b)}{d t} = \sum_ {k, j, l} \partial_ {k, j, l} \boldsymbol {\Sigma} _ {B} (a, b) \frac {d \mathbf {h} _ {k , j} [ l ]}{d t} = \sum_ {k, j, l} \partial_ {k, j, l} \boldsymbol {\Sigma} _ {B} (a, b) \left(- \partial_ {k, j, l} \widehat {\mathcal {R}} ^ {\mathcal {F} ^ {\prime}} (\mathbf {H})\right) \\ = \sum_ {k, j} \sum_ {l} - \frac {1}{2 n} \left(\langle \mathbf {e} _ {a}, \mathbf {e} _ {l} \rangle \langle \mathbf {e} _ {b}, \overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G} \rangle + \langle \mathbf {e} _ {a}, \overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G} \rangle \langle \mathbf {e} _ {l}, \mathbf {e} _ {b} \rangle\right) \langle \mathbf {R} _ {k, j}, \mathbf {e} _ {l} \rangle \\ = \sum_ {k, j} - \frac {1}{2 n} \left(\langle \mathbf {e} _ {a}, \mathbf {R} _ {k, j} \rangle \langle \mathbf {e} _ {b}, \overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G} \rangle + \langle \mathbf {e} _ {a}, \overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G} \rangle \langle \mathbf {R} _ {k, j}, \mathbf {e} _ {b} \rangle\right) \\ = \frac {1}{2 n} \mathbf {e} _ {a} ^ {\top} \left(\sum_ {k, j} - \mathbf {R} _ {k, j} \left(\overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G}\right) ^ {\top} - \left(\overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G}\right) \mathbf {R} _ {k, j} ^ {\top}\right) \mathbf {e} _ {b} \\ = \frac {1}{4 n} \mathbf {e} _ {a} ^ {\top} \left(\sum_ {k, j} \left(\widetilde {\mathbf {M}} \left[ \left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G} \right] - 2 \lambda_ {H} \mathbf {h} _ {k, j} - 2 \mathbf {p} _ {k, j}\right) \left(\overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G}\right) ^ {\top}\right) \mathbf {e} _ {b} \\ + \frac {1}{4 n} \mathbf {e} _ {a} ^ {\top} \left(\sum_ {k, j} \left(\overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G}\right) \left(\widetilde {\mathbf {M}} \left[ \left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G} \right] - 2 \lambda_ {H} \mathbf {h} _ {k, j} - 2 \mathbf {p} _ {k, j}\right) ^ {\top}\right) \mathbf {e} _ {b} \\ \end{array}
$$
For further simplification, let's consider the following term:
$$
\begin{array}{l} \sum_ {k, j} \left[ \left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G} \right] \left[ \overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G} \right] ^ {\top} \\ = \frac {1}{(p + q) ^ {2}} \sum_ {k, j} \left[ (p - q) ^ {2} \overline {{\mathbf {h}}} _ {k} + 4 p q \overline {{\mathbf {h}}} _ {G} \right] \left[ \overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G} \right] ^ {\top} \\ = \frac {1}{(p + q) ^ {2}} \sum_ {k, j} \left[ (p - q) ^ {2} \overline {{\mathbf {h}}} _ {k} \overline {{\mathbf {h}}} _ {k} ^ {\top} - (p - q) ^ {2} \overline {{\mathbf {h}}} _ {k} \overline {{\mathbf {h}}} _ {G} ^ {\top} + 4 p q \overline {{\mathbf {h}}} _ {G} \overline {{\mathbf {h}}} _ {k} ^ {\top} - 4 p q \overline {{\mathbf {h}}} _ {G} \overline {{\mathbf {h}}} _ {G} ^ {\top} \right] \\ = \frac {1}{(p + q) ^ {2}} \left[ (p - q) ^ {2} 2 n \widetilde {\boldsymbol {\Sigma}} _ {B} - 8 n p q \boldsymbol {\Sigma} _ {G} - 2 n \boldsymbol {\Sigma} _ {G} \left((p - q) ^ {2} - 4 p q\right) \right] \\ = \frac {1}{(p + q) ^ {2}} \left[ 2 n (p - q) ^ {2} \boldsymbol {\Sigma} _ {B} + 2 n (p - q) ^ {2} \boldsymbol {\Sigma} _ {G} - 8 n p q \boldsymbol {\Sigma} _ {G} - 2 n \boldsymbol {\Sigma} _ {G} ((p - q) ^ {2} - 4 p q) \right] \\ = 2 n \left(\frac {p - q}{p + q}\right) ^ {2} \boldsymbol {\Sigma} _ {B} \\ \end{array}
$$
Next, we proceed with the simplification of  $\sum_{k,j} - \lambda_H\mathbf{h}_{k,j}(\overline{\mathbf{h}}_k - \overline{\mathbf{h}}_G)^\top -\lambda_H(\overline{\mathbf{h}}_k - \overline{\mathbf{h}}_G)\mathbf{h}_{k,j}^\top$
$$
\begin{array}{l} - \lambda_ {H} \left(\sum_ {k, j} \mathbf {h} _ {k, j} \left(\overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G}\right) ^ {\top} + \left(\overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G}\right) \mathbf {h} _ {k, j} ^ {\top}\right) \\ = - \lambda_ {H} \left(n \bar {\mathbf {h}} _ {1} \left(\bar {\mathbf {h}} _ {1} - \bar {\mathbf {h}} _ {G}\right) ^ {\top} + n \bar {\mathbf {h}} _ {2} \left(\bar {\mathbf {h}} _ {2} - \bar {\mathbf {h}} _ {G}\right) ^ {\top} + n \left(\bar {\mathbf {h}} _ {1} - \bar {\mathbf {h}} _ {G}\right) \bar {\mathbf {h}} _ {1} ^ {\top} + n \left(\bar {\mathbf {h}} _ {2} - \bar {\mathbf {h}} _ {G}\right) \bar {\mathbf {h}} _ {2} ^ {\top}\right) \\ = - \lambda_ {H} \left(2 n \bar {\mathbf {h}} _ {1} \bar {\mathbf {h}} _ {1} ^ {\top} + 2 n \bar {\mathbf {h}} _ {2} \bar {\mathbf {h}} _ {2} ^ {\top} - n \bar {\mathbf {h}} _ {1} \bar {\mathbf {h}} _ {G} ^ {\top} - n \bar {\mathbf {h}} _ {2} \bar {\mathbf {h}} _ {G} ^ {\top} - n \bar {\mathbf {h}} _ {G} \bar {\mathbf {h}} _ {1} ^ {\top} - n \bar {\mathbf {h}} _ {G} \bar {\mathbf {h}} _ {2} ^ {\top}\right) \\ = - \lambda_ {H} \left(2 n \bar {\mathbf {h}} _ {1} \bar {\mathbf {h}} _ {1} ^ {\top} + 2 n \bar {\mathbf {h}} _ {2} \bar {\mathbf {h}} _ {2} ^ {\top} - 4 n \bar {\mathbf {h}} _ {G} \bar {\mathbf {h}} _ {G} ^ {\top}\right) = - \lambda_ {H} \left(4 n \tilde {\Sigma} _ {B} - 4 n \boldsymbol {\Sigma} _ {G}\right) = - 4 n \lambda_ {H} \boldsymbol {\Sigma} _ {B} \\ \end{array}
$$
These results now simplify  $\frac{d\pmb{\Sigma}_B}{dt}$  as:
$$
\frac {d \boldsymbol {\Sigma} _ {B}}{d t} = \frac {1}{4 n} \mathbf {e} _ {a} ^ {\top} \left(2 n \left(\frac {p - q}{p + q}\right) ^ {2} \widetilde {\mathbf {M}} \boldsymbol {\Sigma} _ {B} + 2 n \left(\frac {p - q}{p + q}\right) ^ {2} \boldsymbol {\Sigma} _ {B} \widetilde {\mathbf {M}} - 8 n \lambda_ {H} \boldsymbol {\Sigma} _ {B} - \widetilde {\mathbf {P}} _ {B}\right) \mathbf {e} _ {b} \tag {104}
$$
Where  $\widetilde{\mathbf{P}}_B = 2\sum_{k,j}\mathbf{p}_{k,j}(\overline{\mathbf{h}}_k - \overline{\mathbf{h}}_G)^\top + (\overline{\mathbf{h}}_k - \overline{\mathbf{h}}_G)\mathbf{p}_{k,j}^\top.$
- Derivative of  $\Sigma_W$ : By denoting  $\Sigma_W(a,b) = \mathbf{e}_a^\top \Sigma_W\mathbf{e}_b$  as the  $(a,b)$ -element of  $\Sigma_W$ , we use a similar line of analysis as above to compute  $\frac{d\Sigma_W}{dt}$  as follows:
$$
\begin{array}{l} \frac {d \boldsymbol {\Sigma} _ {W} (a , b)}{d t} = \sum_ {k, j, l} \partial_ {k, j, l} \boldsymbol {\Sigma} _ {W} (a, b) \frac {d \mathbf {h} _ {k , j} [ l ]}{d t} = \sum_ {k, j, l} \partial_ {k, j, l} \boldsymbol {\Sigma} _ {W} (a, b) \left(- \partial_ {k, j, l} \widehat {\mathcal {R}} ^ {\mathcal {F} ^ {\prime}} (\mathbf {H})\right) \\ = \sum_ {k, j} \sum_ {l} - \frac {1}{2 n} \left(\langle \mathbf {e} _ {a}, \mathbf {e} _ {l} \rangle \langle \mathbf {e} _ {b}, \mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k} \rangle + \langle \mathbf {e} _ {a}, \mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k} \rangle \langle \mathbf {e} _ {l}, \mathbf {e} _ {b} \rangle\right) \langle \mathbf {R} _ {k, j}, \mathbf {e} _ {l} \rangle \\ = \sum_ {k, j} - \frac {1}{2 n} \left(\langle \mathbf {e} _ {a}, \mathbf {R} _ {k, j} \rangle \langle \mathbf {e} _ {b}, \mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k} \rangle + \langle \mathbf {e} _ {a}, \mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k} \rangle \langle \mathbf {R} _ {k, j}, \mathbf {e} _ {b} \rangle\right) \\ = \frac {1}{2 n} \mathbf {e} _ {a} ^ {\top} \left(\sum_ {k, j} - \mathbf {R} _ {k, j} \left(\mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k}\right) ^ {\top} - \left(\mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k}\right) \mathbf {R} _ {k, j} ^ {\top}\right) \mathbf {e} _ {b} \\ = \frac {1}{4 n} \mathbf {e} _ {a} ^ {\top} \left(\sum_ {k, j} \left(\widetilde {\mathbf {M}} \left[ \left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G} \right] - 2 \lambda_ {H} \mathbf {h} _ {k, j} - 2 \mathbf {p} _ {k, j}\right) \left(\mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k}\right) ^ {\top}\right) \mathbf {e} _ {b} \\ + \frac {1}{4 n} \mathbf {e} _ {a} ^ {\top} \left(\sum_ {k, j} \left(\mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k}\right) \left(\widetilde {\mathbf {M}} \left[ \left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G} \right] - 2 \lambda_ {H} \mathbf {h} _ {k, j} - 2 \mathbf {p} _ {k, j}\right) ^ {\top}\right) \mathbf {e} _ {b} \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {k, j} \left[ \left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G} \right] \left[ \mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k} \right] ^ {\top} \\ = \sum_ {k} \left[ \left(\frac {p - q}{p + q}\right) ^ {2} \overline {{\mathbf {h}}} _ {k} + \frac {4 p q}{(p + q) ^ {2}} \overline {{\mathbf {h}}} _ {G} \right] \left[ \sum_ {j} \left(\mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k}\right) \right] ^ {\top} = \mathbf {0} \\ \end{array}
$$
Next, we simplify  $\sum_{k,j} - \lambda_H\mathbf{h}_{k,j}(\mathbf{h}_{k,j} - \overline{\mathbf{h}}_k)^\top - \lambda_H(\mathbf{h}_{k,j} - \overline{\mathbf{h}}_k)\mathbf{h}_{k,j}^\top$  as follows:
$$
\begin{array}{l} - \lambda_ {H} \sum_ {k, j} \left(\mathbf {h} _ {k, j} \left(\mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k}\right) ^ {\top} + \left(\mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k}\right) \mathbf {h} _ {k, j} ^ {\top}\right) \\ = - \lambda_ {H} \sum_ {k, j} \left(\mathbf {h} _ {k, j} \mathbf {h} _ {k, j} ^ {\top} - \mathbf {h} _ {k, j} \overline {{\mathbf {h}}} _ {k} ^ {\top} + \mathbf {h} _ {k, j} \mathbf {h} _ {k, j} ^ {\top} - \overline {{\mathbf {h}}} _ {k} \mathbf {h} _ {k, j} ^ {\top}\right) \\ = - \lambda_ {H} \sum_ {k, j} \left(\mathbf {h} _ {k, j} \mathbf {h} _ {k, j} ^ {\top} - \mathbf {h} _ {k, j} \overline {{\mathbf {h}}} _ {k} ^ {\top} + \mathbf {h} _ {k, j} \mathbf {h} _ {k, j} ^ {\top} - \overline {{\mathbf {h}}} _ {k} \mathbf {h} _ {k, j} ^ {\top} + \overline {{\mathbf {h}}} _ {k} \overline {{\mathbf {h}}} _ {k} ^ {\top} - \overline {{\mathbf {h}}} _ {k} \overline {{\mathbf {h}}} _ {k} ^ {\top}\right) \\ = - \lambda_ {H} \left(2 n \boldsymbol {\Sigma} _ {W} + 2 n \widetilde {\boldsymbol {\Sigma}} _ {T} - 2 n \widetilde {\boldsymbol {\Sigma}} _ {B}\right) = - 4 n \lambda_ {H} \boldsymbol {\Sigma} _ {W} \\ \end{array}
$$
Where the last inequality is based on the fact that  $\widetilde{\Sigma}_T = \Sigma_W + \widetilde{\Sigma}_B$ . These results simplify  $\frac{d\Sigma_W}{dt}$  as:
$$
\frac {d \boldsymbol {\Sigma} _ {W}}{d t} = \frac {1}{4 n} \mathbf {e} _ {a} ^ {\top} \left(- 8 n \lambda_ {H} \boldsymbol {\Sigma} _ {W} - \widetilde {\mathbf {P}} _ {W}\right) \mathbf {e} _ {b} \tag {105}
$$
Where  $\widetilde{\mathbf{P}}_W = 2\sum_{k,j}\mathbf{p}_{k,j}(\mathbf{h}_{k,j} - \overline{\mathbf{h}}_k)^\top +(\mathbf{h}_{k,j} - \overline{\mathbf{h}}_k)\mathbf{p}_{k,j}^\top$
- Trace of covariance matrices along the flow: Taking the derivative of  $\operatorname{Tr}(\Sigma_W)$  gives us:
$$
\begin{array}{l} \frac {d \operatorname {T r} (\boldsymbol {\Sigma} _ {W})}{d t} = - 2 \lambda_ {H} \operatorname {T r} (\boldsymbol {\Sigma} _ {W}) - \frac {1}{4 n} \operatorname {T r} \left(\widetilde {\mathbf {P}} _ {W}\right) \\ = - 2 \lambda_ {H} \operatorname {T r} \left(\boldsymbol {\Sigma} _ {W}\right) - \frac {1}{n} \operatorname {T r} \left(\sum_ {k, j} \mathbf {p} _ {k, j} \left(\mathbf {h} _ {k, j} - \overline {{\mathbf {h}}} _ {k}\right) ^ {\top}\right) \tag {106} \\ \end{array}
$$
Similarly, the derivative of  $\operatorname{Tr}(\Sigma_B)$  gives us:
$$
\begin{array}{l} \frac {d \operatorname {T r} (\boldsymbol {\Sigma} _ {B})}{d t} = - 2 \lambda_ {H} \operatorname {T r} (\boldsymbol {\Sigma} _ {B}) + \frac {1}{4 n} \operatorname {T r} \left(2 n \left(\frac {p - q}{p + q}\right) ^ {2} \widetilde {\mathbf {M}} \boldsymbol {\Sigma} _ {B} + 2 n \left(\frac {p - q}{p + q}\right) ^ {2} \boldsymbol {\Sigma} _ {B} \widetilde {\mathbf {M}} - \widetilde {\mathbf {P}} _ {B}\right) \\ = - 2 \lambda_ {H} \operatorname {T r} \left(\boldsymbol {\Sigma} _ {B}\right) + \frac {1}{2 n} \operatorname {T r} \left(2 n \left(\frac {p - q}{p + q}\right) ^ {2} \widetilde {\mathbf {M}} \boldsymbol {\Sigma} _ {B} - 2 \sum_ {k, j} \mathbf {p} _ {k, j} \left(\overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G}\right) ^ {\top}\right) \\ = - 2 \lambda_ {H} \operatorname {T r} \left(\boldsymbol {\Sigma} _ {B}\right) + \operatorname {T r} \left(\left(\frac {p - q}{p + q}\right) ^ {2} \widetilde {\mathbf {M}} \boldsymbol {\Sigma} _ {B}\right) - \frac {1}{n} \operatorname {T r} \left(\sum_ {k, j} \mathbf {p} _ {k, j} \left(\overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G}\right) ^ {\top}\right) \\ = \operatorname {T r} \left(\left[ \left(\frac {p - q}{p + q}\right) ^ {2} \widetilde {\mathbf {M}} - 2 \lambda_ {H} \mathbf {I} \right] \boldsymbol {\Sigma} _ {B}\right) - \frac {1}{n} \operatorname {T r} \left(\sum_ {k, j} \mathbf {p} _ {k, j} \left(\overline {{\mathbf {h}}} _ {k} - \overline {{\mathbf {h}}} _ {G}\right) ^ {\top}\right) \tag {107} \\ \end{array}
$$
Observe that for small enough perturbation matrix  $\mathbf{E}$ , formally for  $\| \mathbf{E} \| < E$  for sufficiently small  $E$ , the sign of  $\frac{d \operatorname{Tr}(\boldsymbol{\Sigma}_W)}{dt}$  and  $\frac{d \operatorname{Tr}(\boldsymbol{\Sigma}_B)}{dt}$  depends on the first term in each of them, and not on the second term that depends on  $\mathbf{E}$ . Therefore, since  $\lambda_H > 0$ , we have that  $\operatorname{Tr}(\boldsymbol{\Sigma}_W)$  decreases. It is left to show that  $\operatorname{Tr}(\boldsymbol{\Sigma}_B)$  increases in this regime.
Since the trace of the product of a positive definite matrix and a non-zero positive semidefinite matrix is positive by Von-Neumann trace inequality, we aim for conditions that allow  $\left[\left(\frac{p - q}{p + q}\right)^2\widetilde{\mathbf{M}} - 2\lambda_H\mathbf{I}\right]$  to be positive definite. First, observe that  $\widetilde{\mathbf{M}}$  is symmetric:
$$
\widetilde {\mathbf {M}} = \left[ \mathbf {M} - \mathbf {M} \left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] \mathbf {M} \right]
$$
$$
\mathbf {M} = \left[ \mathbf {J} + \Delta_ {2} + \lambda_ {W _ {2}} N \mathbf {I} \right] ^ {- 1}
$$
$$
\mathbf {J} = 2 n \left(\widetilde {\boldsymbol {\Sigma}} _ {B} - \frac {4 p q}{(p + q) ^ {2}} \boldsymbol {\Sigma} _ {B}\right)
$$
Since  $\widetilde{\Sigma}_B, \Sigma_B, \mathbf{I}, \Delta_2$  and  $\widetilde{\Delta}_3$  are symmetric.
Observe that  $\frac{4pq}{(p + q)^2} \leq 1$ , because
$$
0 \leq (p - q) ^ {2} = p ^ {2} + 2 p q + q ^ {2} - 4 p q = (p + q) ^ {2} - 4 p q.
$$
$$
\mathbf {J} = 2 n \left(\widetilde {\boldsymbol {\Sigma}} _ {B} - \frac {4 p q}{(p + q) ^ {2}} \boldsymbol {\Sigma} _ {B}\right) \geq 2 n \left(\widetilde {\boldsymbol {\Sigma}} _ {B} - \boldsymbol {\Sigma} _ {B}\right) = 2 n \boldsymbol {\Sigma} _ {G} \geq 0
$$
Thus also  $\mathbf{M} > 0$  for small  $\Delta_2$ . Note also that  $[\mathbf{J} + \lambda_{W_2}N\mathbf{I}]^{-1}\mathbf{J} < \mathbf{I}$ . Therefore, for small enough  $\mathbf{E}$  (and thus small  $\Delta_2,\widetilde{\Delta}_3$ ), we have that
$$
\widetilde {\mathbf {M}} = \mathbf {M} - \mathbf {M} \left[ \mathbf {J} + \frac {\widetilde {\Delta} _ {3}}{n} \right] \mathbf {M} > 0.
$$
Thus,  $\left[\left(\frac{p - q}{p + q}\right)^2\widetilde{\mathbf{M}} - 2\lambda_H\mathbf{I}\right]$  is positive definite when:
$$
2 \lambda_ {H} <   \left(\frac {p - q}{p + q}\right) ^ {2} \lambda_ {\min } (\widetilde {\mathbf {M}}) \tag {108}
$$
Here  $\lambda_{min}\left(\widetilde{\mathbf{M}}\right)$  represents the smallest eigenvalue of  $\widetilde{\mathbf{M}}$
G Proof of Theorem 4.1
Let's begin by calculating the expected value and covariance of features  $\mathbf{X}^{(l)}$ , which are obtained after the graph convolution operation based on equation 3.
- Case  $c = 1$ :
We begin by considering the features of a node belonging to class  $c = 1$  as follows:
$$
\mathbf {x} _ {1, i} ^ {(l)} = \mathbf {W} _ {1} ^ {* (l)} \mathbf {h} _ {1, i} ^ {(l - 1)} + \mathbf {W} _ {2} ^ {* (l)} \mathbf {H} ^ {(l - 1)} \widehat {\mathbf {A}} _ {t} \mathbf {e} _ {1, i} \tag {109}
$$