SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {t r} \left(B ^ {T} P B\right) = \operatorname {t r} \left(X ^ {T} X\right) - (8 L ^ {2} - 1 0 L + 4) \frac {\theta^ {T}}{\| \theta \| ^ {2}} X ^ {T} X \theta .
$$
Now removing the  $\mu$  subscript of the derived  $\sum_{\rho, \sigma, i} H_{\sigma, \mu}^{\rho} B_{\sigma i} B_{i\rho}$  and recovering the matrix notation, we have the final form of  $\spadesuit$  by combining it with the result of  $\operatorname{tr}(B^T P B)$
$$
\begin{array}{l} \spadesuit = - \frac {k ^ {2} \lambda_ {L}}{2 \| \theta \| ^ {\lambda_ {L} + 2}} \left[ 2 (2 L - 1) \left(I + 2 (L - 1) \frac {\theta \theta^ {T}}{\| \theta \| ^ {2}}\right) X ^ {T} X \theta \right] \\ - \frac {k ^ {2} \lambda_ {L}}{2 \| \theta \| ^ {\lambda_ {L} + 2}} \left[ \theta \operatorname {t r} \left(X ^ {T} X\right) - (8 L ^ {2} - 1 0 L + 4) \frac {\theta \theta^ {T}}{\| \theta \| ^ {2}} X ^ {T} X \theta \right] \\ = \frac {k ^ {2}}{2} \left[ - \frac {\lambda_ {L}}{\| \theta \| ^ {\lambda_ {L} + 2}} \operatorname {t r} \left(X ^ {T} X\right) \theta - \frac {4 (L - 1)}{\| \theta \| ^ {\lambda_ {L} + 2}} \left(I - \frac {L}{2 L - 1} \frac {\theta \theta^ {T}}{\| \theta^ {2} \|}\right) X ^ {T} X \theta \right] d t. \tag {109} \\ \end{array}
$$
Thus the SDE of  $D$  now becomes
$$
\begin{array}{l} d D = \partial_ {\theta} D d \theta \\ + \left[ \partial_ {t} D - \frac {k ^ {2} \lambda_ {L}}{2 \| \theta \| ^ {\lambda_ {L} + 2}} \operatorname {t r} \left(X ^ {T} X\right) \theta - \frac {2 k ^ {2} (L - 1)}{\| \theta \| ^ {\lambda_ {L} + 2}} \left(I - \frac {L}{2 L - 1} \frac {\theta \theta^ {T}}{\| \theta^ {2} \|}\right) X ^ {T} X \theta \right] d t \\ \end{array}
$$
To find the particular  $g(t)$  such that Eq. (98) is satisfied, we need to require that
$$
\begin{array}{l} \partial_ {t} D - \frac {k ^ {2} \lambda_ {L}}{2 \| \theta \| ^ {\lambda_ {L} + 2}} \operatorname {t r} \left(X ^ {T} X\right) \theta - \frac {2 k ^ {2} (L - 1)}{\| \theta \| ^ {\lambda_ {L} + 2}} \left(I - \frac {L}{2 L - 1} \frac {\theta \theta^ {T}}{\| \theta^ {2} \|}\right) X ^ {T} X \theta \\ = - \frac {8 \eta \mathcal {L}}{n} \frac {(L - 1) \xi^ {2}}{\| \theta \| ^ {2}} \left(I - \frac {1}{2} \frac {\theta \theta^ {T}}{\| \theta \| ^ {2}}\right) X ^ {T} X \theta . \\ \end{array}
$$
Recall that  $D = \| \theta \|^{-\lambda_L}\theta +\gamma +g(t)$  , we have
$$
\partial_ {t} D = g ^ {\prime} (t),
$$
which, noting that  $k^2 = 4\xi^4\eta \mathcal{L} / n$ , further gives us that the following relation should be satisfied
$$
\begin{array}{l} g ^ {\prime} (t) - \frac {2 \xi^ {4} \eta \mathcal {L}}{n \| \theta \| ^ {\lambda_ {L} + 2}} \left[ \lambda_ {L} \operatorname {t r} \left(X ^ {T} X\right) \theta + 4 (L - 1) \left(I - \frac {L}{2 L - 1} \frac {\theta \theta^ {T}}{\| \theta^ {2} \|}\right) X ^ {T} X \theta \right] \\ = - \frac {8 \eta \mathcal {L}}{n} \frac {(L - 1) \xi^ {2}}{\| \theta \| ^ {2}} \left(I - \frac {1}{2} \frac {\theta \theta^ {T}}{\| \theta \| ^ {2}}\right) X ^ {T} X \theta . \\ \end{array}
$$
As a result of this, we can, noting that  $\xi^2 = \| \theta \|^{2\lambda_L}$  from  $\| \theta \| ^2 = \xi^2\| v_1\| ^2$  , give the required  $g(t)$  that makes Eq. (98) satisfied by solving the following equation:
$$
\begin{array}{l} g ^ {\prime} (t) = \frac {2 \lambda_ {L} \eta \operatorname {t r} \left(X ^ {T} X\right) \mathcal {L} \| \theta \| ^ {\lambda_ {L} - 2}}{n} \theta \\ + \frac {8 \eta (L - 1) \mathcal {L} \| \theta \| ^ {\lambda_ {L} - 2}}{n} \left[ I - I - \left(\frac {L}{2 L - 1} - \frac {1}{2}\right) \right] X ^ {T} X \theta \\ = \frac {2 \lambda_ {L} \eta \mathcal {L} \| \theta \| ^ {\lambda_ {L} - 2}}{n} \operatorname {t r} \left(\left(I - \frac {\theta \theta^ {T}}{\| \theta \| ^ {2}}\right) X ^ {T} X\right) \theta . \tag {110} \\ \end{array}
$$
Now let the orthogonal projection operator of  $\theta$  be  $P_{\perp}(\theta) = I - \frac{\theta\theta^T}{\|\theta\|^2}$ , then we can solve  $g(t)$  as
$$
g (t) = \frac {2 \lambda_ {L} \eta}{n} \int_ {0} ^ {t} \mathcal {L} (\theta) \| \theta \| ^ {\lambda_ {L} - 2} \theta \operatorname {t r} \left(P _ {\perp} (\theta) X ^ {T} X\right) d s. \tag {111}
$$
With this particular  $g(t)$ , we can then give  $V^{S}(\theta ,t)$
$$
V ^ {\mathcal {S}} (\theta , t) = \frac {1}{\Omega_ {L}} \| \theta \| ^ {\Omega_ {L}} + \theta^ {T} \left[ \frac {2 \lambda_ {L} \eta}{n} \int_ {0} ^ {t} \mathcal {L} (\theta) \| \theta \| ^ {\lambda_ {L} - 2} \theta \operatorname {t r} \left(P _ {\perp} (\theta) X ^ {T} X\right) d s - \frac {\theta (0)}{\| \theta (0) \| ^ {\lambda_ {L}}} \right] \tag {112}
$$
that satisfies the relation
$$
d \partial_ {\theta} V ^ {\mathcal {S}} (\theta) = - \nabla_ {\theta} \mathcal {L} d t + 2 \sqrt {\frac {\eta \mathcal {L} \xi}{n}} X ^ {T} d \mathcal {W} _ {t}.
$$
Moreover, a particular interesting case is that, as  $L\to \infty$
$$
\lim  _ {L \rightarrow \infty} V (\theta , t) = \| \theta \| + \theta^ {T} \left[ \frac {2 \eta}{n} \int_ {0} ^ {T} \mathcal {L} (\theta) \left\| P _ {\perp} (\theta) X ^ {T} \right\| _ {F} ^ {2} \frac {\theta}{\| \theta \|} d s - \frac {\theta (0)}{\| \theta (0) \|} \right]. \tag {113}
$$