SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} X _ {i + 1} (U _ {\Sigma} X _ {0}, B _ {j}) \\ = X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) + B _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) M X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) ^ {T} A _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) \\ = U _ {\Sigma} X _ {i} \left(X _ {0}, B _ {j}\right) + U _ {\Sigma} B _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) M X _ {i} \left(X _ {0}, B _ {j}\right) ^ {T} A _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) \\ = U _ {\Sigma} X _ {i + 1} \left(X _ {0}, B _ {j}\right) \\ \end{array}
$$
where the second equality uses the inductive hypothesis, and the fact that  $A_{i} = a_{i}\Sigma^{-1}$ , so that  $U_{\Sigma}^{T}A_{i}U_{\Sigma} = A_{i}$ , and the fact that  $B_{i} = b_{i}I$ , from the definition of  $S$  and our assumption that  $(A,B)\in S$ . This concludes the proof of (42).
We now present the second main result of this step. Let  $U_{\Sigma}^{-1} \coloneqq \Sigma^{1/2}U^T\Sigma^{-1/2}$ , so that it satisfies  $U_{\Sigma}U_{\Sigma}^{-1} = U_{\Sigma}^{-1}U_{\Sigma} = I$ . For all  $i$ ,
$$
\left. U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right) \right| _ {t = 0} = \left. \frac {d}{d t} X _ {i} \left(X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}\right) \right| _ {t = 0}. \tag {43}
$$
To reduce notation, we will not write  $\cdot|_{t=0}$  explicitly in the subsequent proof. We first write down the dynamics for the right-hand-side term of (43): From (34), for any  $\ell \leq j$ , and for any  $i \geq j + 1$ , and
for any  $C\in \mathbb{R}^{d\times d}$
$$
\begin{array}{l} \frac {d}{d t} X _ {\ell} (X _ {0}, B _ {j} + t C) = 0 \\ \frac {d}{d t} X _ {j + 1} \left(X _ {0}, B _ {j} + t C\right) = C X _ {j} \left(X _ {0}, B _ {j}\right) M X _ {j} \left(X _ {0}, B _ {j}\right) ^ {\top} A _ {j} X _ {j} \left(X _ {0}, B _ {j}\right) \\ \frac {d}{d t} X _ {i + 1} \left(X _ {0}, B _ {j} + t C\right) = \frac {d}{d t} X _ {i} \left(X _ {0}, B _ {j} + t C\right) \\ + B _ {i} \left(\frac {d}{d t} X _ {i} \left(X _ {0}, B _ {j} + t C\right)\right) M X _ {i} \left(X _ {0}, B _ {j}\right) ^ {\top} A _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) \\ + B _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) M \left(\frac {d}{d t} X _ {i} \left(X _ {0}, B _ {j} + t C\right)\right) ^ {\top} A _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) \\ + B _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) M X _ {i} \left(X _ {0}, B _ {j}\right) ^ {\top} A _ {i} \left(\frac {d}{d t} X _ {i} \left(X _ {0}, B _ {j} + t C\right)\right) \tag {44} \\ \end{array}
$$
We are now ready to prove (43) using induction. For the base case, we verify that for  $\ell \leq j$ ,  $U_{\Sigma}^{-1}\frac{d}{dt} X_{\ell}\left(U_{\Sigma}X_{0},B_{k} + tS_{j}\right) = 0 = \frac{d}{dt} X_{\ell}\left(X_{0},B_{j} + tU_{\Sigma}^{-1}S_{j}U_{\Sigma}\right)$  (see first equation in (44)). For index  $j + 1$ , we verify that
$$
\begin{array}{l} U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {j + 1} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right) \\ = U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma} X _ {j} (X _ {0}, B _ {j}) M X _ {j} \big (U _ {\Sigma} X _ {0}, B _ {j} \big) ^ {\top} A _ {j} X _ {j} (U _ {\Sigma} X _ {0}, B _ {j}) \\ = \frac {d}{d t} X _ {j + 1} \left(X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}\right) \tag {45} \\ \end{array}
$$
where we use two facts: 1.  $X_{i}(U_{\Sigma}X_{0},B_{j}) = U_{\Sigma}X_{i}(X_{0},B_{j})$  from (42), 2.  $A_{i} = a_{i}\Sigma^{-1}$  so that  $U_{\Sigma}^{\top}A_{i}U_{\Sigma} = A_{i}$ . We verify by comparison to the second equation in (44) that  $U_{\Sigma}^{-1}\frac{d}{dt} X_{j}\left(U_{\Sigma}X_{0},B_{j} + tS_{j}\right) = 0 = \frac{d}{dt} X_{j}\left(X_{0},B_{j} + tU_{\Sigma}^{-1}S_{j}U_{\Sigma}\right)$ . These conclude the proof of the base case.
Now suppose that (43) holds for some  $i$ . We will now prove (43) holds for  $i + 1$ . From (34),
$$
\begin{array}{l} U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i + 1} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right) \\ = U _ {\Sigma} ^ {- 1} \frac {d}{d t} \left(X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right) \\ + U _ {\Sigma} ^ {- 1} \frac {d}{d t} \left(B _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right) M X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right) ^ {\top} A _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right) \\ = U _ {\Sigma} ^ {- 1} \frac {d}{d t} \left(X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right) \\ + U _ {\Sigma} ^ {- 1} B _ {i} \left(\frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right) M X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) ^ {\top} A _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) \\ + U _ {\Sigma} ^ {- 1} B _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) M \left(\frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right) ^ {\top} A _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) \\ + U _ {\Sigma} ^ {- 1} B _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) M X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) ^ {\top} A _ {i} \left(\frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i)} {=} U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right) \\ + B _ {i} \left(U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right) M X _ {i} \left(X _ {0}, B _ {j}\right) ^ {\top} A _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) \\ + B _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) M \left(U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right) ^ {\top} A _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) \\ \left. - B _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) M X _ {i} \left(X _ {0}, B _ {j}\right) ^ {\top} A _ {i} \left(U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right) \right. \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i i)} {=} \frac {d}{d t} X _ {i} \left(X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}\right) \\ + B _ {i} \left(\frac {d}{d t} X _ {i} \left(X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}\right)\right) M X _ {i} \left(X _ {0}, B _ {j}\right) ^ {\top} A _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) \\ + B _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) M \left(\frac {d}{d t} X _ {i} \left(X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}\right)\right) ^ {\top} A _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) \\ + B _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) M X _ {i} \left(X _ {0}, B _ {j}\right) ^ {\top} A _ {i} \left(\frac {d}{d t} X _ {i} \left(X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}\right)\right) \tag {46} \\ \end{array}
$$
In (i) above, we crucially use the following facts: 1.  $B_i = b_i I$  so that  $U_{\Sigma}^{-1} B_i = B_i U_{\Sigma}^{-1}$ , 2.  $X_i(U_{\Sigma}X_0, B_j) = U_{\Sigma}X_i(X_0, B_j)$  from (42), 3.  $A_i = a_i \Sigma^{-1}$ , so that  $U_{\Sigma}^{\top} A_i U_{\Sigma} = A_i$ , 4.  $U_{\Sigma} U_{\Sigma}^{-1} = U_{\Sigma}^{-1} U_{\Sigma} = I$ . (ii) follows from our inductive hypothesis. The inductive proof is complete by verifying that (46) exactly matches the third equation of (44) when  $C = U_{\Sigma}^{-1} SU_{\Sigma}$ .
3.  $G$  and  $\frac{d}{dt} G$  under random transformation of  $X_0$ . We now verify that  $G(U_{\Sigma}X_0, B_j) = U_{\Sigma}G(X_0, B_j)$ . This is a straightforward consequence of (42) as
$$
\begin{array}{l} G (U _ {\Sigma} X _ {0}, B _ {j}) \\ = U _ {\Sigma} X _ {0} \prod_ {i = 0} ^ {k} \left(I + M X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) ^ {T} A _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right)\right) \\ = U _ {\Sigma} X _ {0} \prod_ {i = 0} ^ {k} \left(I + M X _ {i} \left(X _ {0}, B _ {j}\right) ^ {T} A _ {i} X _ {i} \left(X _ {0}, B _ {j}\right)\right) \\ = U _ {\Sigma} G \left(X _ {0}, B _ {j}\right), \tag {47} \\ \end{array}
$$
where the second equality uses (42), as well as the fact that  $U_{\Sigma}^{\top}A_{i}U_{\Sigma} = A_{i}$ . Next, we will show that
$$
\left. U _ {\Sigma} ^ {- 1} \frac {d}{d t} G \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right) \right| _ {t = 0} = \left. \frac {d}{d t} G \left(X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}\right) \right| _ {t = 0}. \tag {48}
$$
To see this, we can expand
$$
\begin{array}{l} U _ {\Sigma} ^ {- 1} \frac {d}{d t} G \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right) \\ = U _ {\Sigma} ^ {- 1} \frac {d}{d t} \left(U _ {\Sigma} X _ {0} \prod_ {i = 0} ^ {k} \left(I + M X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right) ^ {T} A _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right)\right) \\ = X _ {0} \sum_ {i = 0} ^ {k} \left(\prod_ {\ell = 0} ^ {i - 1} \left(I + M X _ {\ell} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) ^ {T} A _ {\ell} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {\ell}\right)\right)\right) \\ \cdot M \frac {d}{d t} \left(X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right) ^ {T} A _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j}\right)\right) \\ \cdot \left(\prod_ {\ell = i + 1} ^ {k} \left(I + M X _ {\ell} \left(U _ {\Sigma} X _ {0}, B _ {j}\right) ^ {T} A _ {\ell} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {\ell}\right)\right)\right) \\ \stackrel {(i)} {=} X _ {0} \sum_ {i = 0} ^ {k} \left(\prod_ {\ell = 0} ^ {i - 1} \left(I + M X _ {\ell} \left(X _ {0}, B _ {j}\right) ^ {T} A _ {\ell} X _ {\ell} \left(X _ {0}, B _ {\ell}\right)\right)\right) \\ \cdot M \left(\left(U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right) ^ {T} A _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) + M X _ {i} \left(X _ {0}, B _ {j}\right) ^ {T} A _ {i} \left(U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}\right)\right)\right) \\ \cdot \left(\prod_ {\ell = i + 1} ^ {k} \left(I + M X _ {\ell} \left(X _ {0}, B _ {j}\right) ^ {T} A _ {\ell} X _ {\ell} \left(X _ {0}, B _ {\ell}\right)\right)\right) \\ \stackrel {(i i)} {=} X _ {0} \sum_ {i = 0} ^ {k} \left(\prod_ {\ell = 0} ^ {i - 1} \left(I + M X _ {\ell} (X _ {0}, B _ {j}) ^ {T} A _ {\ell} X _ {\ell} (X _ {0}, B _ {\ell})\right)\right) \\ \end{array}
$$
$$
\begin{array}{l} \cdot M \left(\left(\frac {d}{d t} X _ {i} \left(X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}\right)\right) ^ {T} A _ {i} X _ {i} \left(X _ {0}, B _ {j}\right) + M X _ {i} \left(X _ {0}, B _ {j}\right) ^ {T} A _ {i} \left(\frac {d}{d t} X _ {i} \left(X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}\right)\right)\right) \\ \cdot \left(\prod_ {\ell = i + 1} ^ {k} \left(I + M X _ {\ell} \left(X _ {0}, B _ {j}\right) ^ {T} A _ {\ell} X _ {\ell} \left(X _ {0}, B _ {\ell}\right)\right)\right) \\ \stackrel {(i i i)} {=} \frac {d}{d t} G (X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}) \\ \end{array}
$$
In (i) above, we the following facts: 1.  $X_{i}(U_{\Sigma}X_{0},B_{j}) = U_{\Sigma}X_{i}(X_{0},B_{j})$  from (42), 2.  $A_{i} = a_{i}\Sigma^{-1}$  so that  $U_{\Sigma}^{\top}A_{i}U_{\Sigma} = A_{i}$ , 3.  $U_{\Sigma}U_{\Sigma}^{-1} = U_{\Sigma}^{-1}U_{\Sigma} = I$ . (ii) follows from (43). (iii) is by definition of  $G$ .
4. Putting everything together. Let us now continue from (41). We can now plug (47) and (48) into (41):
$$
\begin{array}{l} \left. \frac {d}{d t} f (A, B (t S _ {j}, j)) \right| _ {t = 0} \\ = 2 \mathbb {E} _ {X _ {0}, U} \left[ \operatorname {T r} \left((I - M) G (U _ {\Sigma} X _ {0}, B _ {j}) ^ {\top} \Sigma^ {- 1} \frac {d}{d t} G (U _ {\Sigma} X _ {0}, B _ {j} + t S _ {j}) \Bigg | _ {t = 0} (I - M)\right) \right] \\ \stackrel {(i)} {=} 2 \mathbb {E} _ {X _ {0}, U} \left[ \operatorname {T r} \left((I - M) G (X _ {0}, B _ {j}) ^ {\top} \Sigma^ {- 1} \frac {d}{d t} G (X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}) \Big | _ {t = 0} (I - M)\right) \right] \\ = 2 \mathbb {E} _ {X _ {0}} \left[ \operatorname {T r} \left(\left(I - M\right) G \left(X _ {0}, B _ {j}\right) ^ {\top} \Sigma^ {- 1} \mathbb {E} _ {U} \left[ \frac {d}{d t} G \left(X _ {0}, B _ {j} + t U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma}\right) \Bigg | _ {t = 0} \right] (I - M)\right) \right] \\ \stackrel {(i i)} {=} 2 \mathbb {E} _ {X _ {0}} \left[ \operatorname {T r} \left((I - M) G (X _ {0}, B _ {j}) ^ {\top} \Sigma^ {- 1} \frac {d}{d t} G (X _ {0}, B _ {j} + t \mathbb {E} _ {U} [ U _ {\Sigma} ^ {- 1} S _ {j} U _ {\Sigma} ]) \Big | _ {t = 0} (I - M)\right) \right] \\ = 2 \mathbb {E} _ {X _ {0}} \left[ \operatorname {T r} \left((I - M) G (X _ {0}, B _ {j}) ^ {\top} \Sigma^ {- 1} \frac {d}{d t} G (X _ {0}, B _ {j} + t s _ {j} I) \Big | _ {t = 0} (I - M)\right) \right] \\ = \frac {d}{d t} f (A, B (t s _ {j} I, j)) \Bigg | _ {t = 0} \\ \end{array}
$$
where  $s_j \coloneqq \frac{1}{d} \operatorname{Tr}\left(\Sigma^{-1/2} S_j \Sigma^{1/2}\right)$ . In the above, (i) uses 1. (47) and (48), as well as the fact that  $U_{\Sigma}^{\top} \Sigma^{-1} U_{\Sigma} = \Sigma^{-1}$ . (ii) uses the fact that  $\left.\frac{d}{dt} G(X_0, B_j + tC)\right|_{t=0}$  is affine in  $C$ . To see this, one can verify from (44), using a simple induction argument, that  $\left.\frac{d}{dt} X_i(X_0, B_j + tC)\right|_{t=0}$  is affine in  $C$  for all  $i$ . We can then verify from the definition of  $G$ , e.g. using similar algebra as the proof of (48), that  $\frac{d}{dt} G(X_0, B_j + C)$  is affine in  $\frac{d}{dt} X_i(X_0, B_j + tC)$ . Thus  $\mathbb{E}_U\left[G(X_0, B_j + tU_{\Sigma}^{-1} S_j U_{\Sigma})\right] = G(X_0, B_j + t \mathbb{E}_U\left[U_{\Sigma}^{-1} S_j U_{\Sigma}\right])$ .
With this, we conclude our proof of (40).
5. Proof of (39). We will now prove (39) for fixed but arbitrary  $j$ , i.e. there is some  $r_j$  such that
$$
\frac {d}{d t} f (A (t \cdot r _ {j} \Sigma^ {- 1}, j), B) \leq \frac {d}{d t} f (A (t R _ {j}, j), B).
$$
The proof is very similar to the proof of (40) that we just saw, and we will essentially repeat the same steps from Step 2-4 above.
Since we now consider perturbations to  $A$  instead of to  $B$ , we will need to redefine some notation: let  $X_{i}(X,C)$  (resp  $Y_{i}(X,C)$ ) to denote the value of  $X_{i}$  (resp  $Y_{i}$ ) from (34), with  $X_0 = X$ , and  $A_{j} = C$  (previously it was with  $B_{j} = C$ ). Let  $G(X,A_{j} + C) := X\prod_{i = 0}^{i}\left(I + M\left(X_{i}(X,A_{j} + C)^{T}A(C,j)\right)_{i}X_{i}(X,A_{j} + C)\right)$ ), where recall that  $A(C,j) := A_j + C$ , and  $A(C,j)_{\ell} := A_{\ell}$  for all  $\ell \in \{0\dots k\} \setminus \{j\}$ .
We first verify that
$$
\begin{array}{l} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j}\right) = U _ {\Sigma} X _ {i} \left(X _ {0}, A _ {j}\right) \\ G \left(U _ {\Sigma} X _ {0}, A _ {j}\right) = U _ {\Sigma} G \left(X _ {0}, A _ {j}\right). \tag {49} \\ \end{array}
$$
The proofs are identical to the proofs of (42) and (47) so we omit them. Next, we show that for all  $i$ ,
$$
\left. U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right) \right| _ {t = 0} = \left. \frac {d}{d t} X _ {i} \left(X _ {0}, A _ {j} + t U _ {\Sigma} ^ {\top} R _ {j} U _ {\Sigma}\right) \right| _ {t = 0}. \tag {50}
$$
We establish the dynamics for the right-hand-side of (50):
$$
\begin{array}{l} \frac {d}{d t} X _ {\ell} \left(X _ {0}, A _ {j} + t C\right) = 0 \\ \frac {d}{d t} X _ {j + 1} \left(X _ {0}, A _ {j} + t C\right) = B _ {j} X _ {j} \left(X _ {0}, A _ {j}\right) M X _ {j} \left(X _ {0}, A _ {j}\right) ^ {\top} C X _ {j} \left(X _ {0}, A _ {j}\right) \\ \frac {d}{d t} X _ {i + 1} \left(X _ {0}, A _ {j} + t C\right) = \frac {d}{d t} X _ {i} \left(X _ {0}, A _ {j} + t C\right) \\ + B _ {i} \left(\frac {d}{d t} X _ {i} \left(X _ {0}, A _ {j} + t C\right)\right) M X _ {i} \left(X _ {0}, A _ {j}\right) ^ {\top} A _ {i} X _ {i} \left(X _ {0}, A _ {j}\right) \\ + B _ {i} X _ {i} \left(X _ {0}, A _ {j}\right) M \left(\frac {d}{d t} X _ {i} \left(X _ {0}, A _ {j} + t C\right)\right) ^ {\top} A _ {i} X _ {i} \left(X _ {0}, A _ {j}\right) \\ + B _ {i} X _ {i} \left(X _ {0}, A _ {j}\right) M X _ {i} \left(X _ {0}, A _ {j}\right) ^ {\top} A _ {i} \left(\frac {d}{d t} X _ {i} \left(X _ {0}, A _ {j} + t C\right)\right) \tag {51} \\ \end{array}
$$
Similar to (45), we show that for  $i \leq j$ ,
$$
U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right) = 0 = U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j} + t U _ {\Sigma} R _ {j} U _ {\Sigma}\right)
$$
$$
\begin{array}{l} U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {j + 1} \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right) \\ = U _ {\Sigma} ^ {- 1} B _ {j} U _ {\Sigma} X _ {j} \left(X _ {0}, A _ {j}\right) M X _ {j} \left(U _ {\Sigma} X _ {0}, A _ {j}\right) ^ {\top} A _ {j} X _ {j} \left(U _ {\Sigma} X _ {0}, A _ {j}\right) \\ = \frac {d}{d t} X _ {j + 1} \left(X _ {0}, A _ {j} + t U _ {\Sigma} ^ {\top} R _ {j} U _ {\Sigma}\right). \\ \end{array}
$$
Finally, for the inductive step, we follow identical steps leading up to (46) to show that
$$
\begin{array}{l} U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i + 1} (U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}) \\ = \frac {d}{d t} X _ {i} \left(X _ {0}, A _ {j} + t U _ {\Sigma} ^ {\top} R _ {j} U _ {\Sigma}\right) \\ + B _ {i} \left(\frac {d}{d t} X _ {i} \left(X _ {0}, A _ {j} + t U _ {\Sigma} ^ {\top} R _ {j} U _ {\Sigma}\right)\right) M X _ {i} \left(X _ {0}, A _ {j}\right) ^ {\top} A _ {i} X _ {i} \left(X _ {0}, A _ {j}\right) \\ + B _ {i} X _ {i} \left(X _ {0}, A _ {j}\right) M \left(\frac {d}{d t} X _ {i} \left(X _ {0}, A _ {j} + t U _ {\Sigma} ^ {\top} R _ {j} U _ {\Sigma}\right)\right) ^ {\top} A _ {i} X _ {i} \left(X _ {0}, A _ {j}\right) \\ + B _ {i} X _ {i} \left(X _ {0}, A _ {j}\right) M X _ {i} \left(X _ {0}, A _ {j}\right) ^ {\top} A _ {i} \left(\frac {d}{d t} X _ {i} \left(X _ {0}, A _ {j} + t U _ {\Sigma} ^ {\top} R _ {j} U _ {\Sigma}\right)\right) \tag {52} \\ \end{array}
$$
The inductive proof is complete by verifying that (52) exactly matches the third equation of (51) when  $C = U_{\Sigma}^{-1}SU_{\Sigma}$ . This concludes the proof of (50).
Next, we study the time derivative of  $G(U_{\Sigma}X_0, A_j + tR_j)$  and show that
$$
U _ {\Sigma} ^ {- 1} \frac {d}{d t} G \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right) = \frac {d}{d t} G \left(X _ {0}, A _ {j} + t U _ {\Sigma} ^ {\top} R _ {j} U _ {\Sigma}\right). \tag {53}
$$
This proof differs significantly from that of (48) in a few places, so we provide the whole derivation below. By chain-rule, we can write
$$
U _ {\Sigma} ^ {- 1} \frac {d}{d t} G \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right) = \spadesuit + \odot
$$
$$
\spadesuit := U _ {\Sigma} ^ {- 1} \frac {d}{d t} \left(U _ {\Sigma} X _ {0} \prod_ {i = 0} ^ {k} \left(I + M X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right) ^ {T} A _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right)\right)\right)
$$
$$
\begin{array}{l} \left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left.\right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left. \text {令} X _ {0} = U _ {\Sigma} X _ {0}, A _ {j}\right) ^ {T} A _ {i} X _ {i} (U _ {\Sigma} X _ {0}, A _ {j})\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right)\right) \\ \cdot M X _ {j} \left(U _ {\Sigma} X _ {0}, A _ {j}\right) ^ {T} R _ {j} X _ {j} \left(U _ {\Sigma} X _ {0}, A _ {j}\right) \\ \cdot \left(\prod_ {i = j + 1} ^ {k} \left(I + M X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j}\right) ^ {T} A _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j}\right)\right)\right). \\ \end{array}
$$
We will separately simplify  $\spadesuit$  and  $\heartsuit$ , and verify at the end that summing them recovers the right-hand-side of (53). We begin with  $\spadesuit$ , and the steps are almost identical to the proof of (48).
$$
\begin{array}{l} = U _ {\Sigma} ^ {- 1} \frac {d}{d t} \left(U _ {\Sigma} X _ {0} \prod_ {i = 0} ^ {k} \left(I + M X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right) ^ {T} A _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right)\right)\right) \\ = X _ {0} \sum_ {i = 0} ^ {k} \left(\prod_ {\ell = 0} ^ {i - 1} \left(I + M X _ {\ell} \left(U _ {\Sigma} X _ {0}, A _ {j}\right) ^ {T} A _ {\ell} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {\ell}\right)\right)\right) \\ \cdot M \frac {d}{d t} \left(X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right) ^ {T} A _ {i} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right)\right) \\ \cdot \left(\prod_ {\ell = i + 1} ^ {k} \left(I + M X _ {\ell} \left(U _ {\Sigma} X _ {0}, A _ {j}\right) ^ {T} A _ {\ell} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {\ell}\right)\right)\right) \\ \stackrel {(i)} {=} X _ {0} \sum_ {i = 0} ^ {k} \left(\prod_ {\ell = 0} ^ {i - 1} \left(I + M X _ {\ell} (X _ {0}, A _ {j}) ^ {T} A _ {\ell} X _ {\ell} (X _ {0}, A _ {\ell})\right)\right) \\ \cdot M \left(\left(U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right)\right) ^ {T} A _ {i} X _ {i} \left(X _ {0}, A _ {j}\right) + M X _ {i} \left(X _ {0}, A _ {j}\right) ^ {T} A _ {i} \left(U _ {\Sigma} ^ {- 1} \frac {d}{d t} X _ {i} \left(U _ {\Sigma} X _ {0}, A _ {j} + t R _ {j}\right)\right)\right) \\ \cdot \left(\prod_ {\ell = i + 1} ^ {k} \left(I + M X _ {\ell} \left(X _ {0}, A _ {j}\right) ^ {T} A _ {\ell} X _ {\ell} \left(X _ {0}, A _ {\ell}\right)\right)\right) \\ \stackrel {(i i)} {=} X _ {0} \sum_ {i = 0} ^ {k} \left(\prod_ {\ell = 0} ^ {i - 1} \left(I + M X _ {\ell} (X _ {0}, A _ {j}) ^ {T} A _ {\ell} X _ {\ell} (X _ {0}, A _ {\ell})\right)\right) \\ \cdot M \left(\left(\frac {d}{d t} X _ {i} \left(X _ {0}, A _ {j} + t U _ {\Sigma} ^ {\top} R _ {j} U _ {\Sigma}\right)\right) ^ {T} A _ {i} X _ {i} \left(X _ {0}, A _ {j}\right) + M X _ {i} \left(X _ {0}, A _ {j}\right) ^ {T} A _ {i} \left(\frac {d}{d t} X _ {i} \left(X _ {0}, A _ {j} + t U _ {\Sigma} ^ {\top} R _ {j} U _ {\Sigma}\right)\right)\right) \\ \cdot \left(\prod_ {\ell = i + 1} ^ {k} \left(I + M X _ {\ell} \left(X _ {0}, A _ {j}\right) ^ {T} A _ {\ell} X _ {\ell} \left(X _ {0}, A _ {\ell}\right)\right)\right) \\ = X _ {0} \sum_ {i = 0} ^ {k} \left(\prod_ {\ell = 0} ^ {i - 1} \left(I + M X _ {\ell} \left(X _ {0}, A _ {j}\right) ^ {T} A _ {\ell} X _ {\ell} \left(X _ {0}, A _ {\ell}\right)\right)\right) \\ \cdot M \frac {d}{d t} \left(X _ {i} \left(X _ {0}, A _ {j} + t U _ {\Sigma} ^ {\top} R _ {j} U _ {\Sigma}\right) ^ {T} A _ {i} X _ {i} \left(X _ {0}, A _ {j} + t U _ {\Sigma} ^ {\top} R _ {j} U _ {\Sigma}\right)\right) \\ \cdot \left(\prod_ {\ell = i + 1} ^ {k} \left(I + M X _ {\ell} \left(X _ {0}, A _ {j}\right) ^ {T} A _ {\ell} X _ {\ell} \left(X _ {0}, A _ {\ell}\right)\right)\right) \tag {54} \\ \end{array}
$$