SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Take a sequence  $\{t_k\}_{k\in \mathbb{N}}$  such that  $t_k > 0$  and  $\lim_{k\to \infty}t_k = 0$ . We wish to show  $\lim_{k\to \infty}V(t_k) = \lim_{n\to \infty}V_{\lambda_n}(0)$ . Note the arguments in Appendix E.3.2 are valid when  $\| \tilde{\mathbf{A}} (X(t_k))\|$  and  $\frac{\|X(t_k) - X_0\|}{t_k}$  are bounded for all  $k\in \mathbb{N}$ . The boundedness of  $\| \tilde{\mathbf{A}} (X(t_k))\|$  comes from Corollary E.2. And from Lemma B.12 we have
$$
\frac {\| X (t _ {k}) - X _ {0} \|}{t _ {k}} = \lim  _ {\lambda \to 0 +} \frac {\| X _ {\lambda} (t _ {k}) - X _ {0} \|}{t _ {k}} \leq \lim  _ {\lambda \to 0 +} \frac {\int_ {0} ^ {t _ {k}} \left\| \dot {X} _ {\lambda} (s) \right\| d s}{t _ {k}} \leq M _ {d o t} (t _ {k}) \leq \sup  _ {k \in \mathbb {N}} M _ {d o t} (t _ {k}).
$$
Therefore applying the arguments in Appendix E.3.2 we have
$$
\lim  _ {k \rightarrow \infty} V ^ {0} (t _ {k}) = \lim  _ {k \rightarrow \infty} \frac {C (t _ {k}) ^ {2}}{2} \left\| \tilde {\mathbf {A}} (X (t _ {k})) \right\| ^ {2}.
$$
Thus it remains to show the limit on the right hand side exists and is equal to  $\lim_{n\to \infty}V_{\lambda_n}^0 (0)$ .
For  $p \geq 1$ , we know  $\lim_{k \to \infty} C(t_k)^2 = 0$ , thus  $\lim_{k \to \infty} \frac{C(t_k)^2}{2} \left\| \tilde{\mathbf{A}}(X(t_k)) \right\|^2 = 0$  since  $\left\| \tilde{\mathbf{A}}(X(t_k)) \right\|$  is bounded. As  $\lim_{n \to \infty} V_{\lambda_n}^0(0) = 0$  from (32), we're done.
Now consider the case  $0 < p < 1$ . Suppose  $\tilde{\mathbf{A}}(X(t_{k_l}))$  is a convergent subsequence of  $\tilde{\mathbf{A}}(X(t_k))$ . First observe from (i) we have
$$
V ^ {0} \left(t _ {k _ {l}}\right) \leq \lim  _ {n \rightarrow \infty} V _ {\lambda_ {n}} ^ {0} (0) = \frac {\left\| m \left(\mathbb {A} \left(X _ {0}\right)\right)\right\| ^ {2}}{2}.
$$
From above inequality, recalling  $\lim_{l\to \infty}C(t_{k_l})^2 = 1$  we have
$$
\frac {\left\| \lim  _ {l \rightarrow \infty} \tilde {\mathbb {A}} (X (t _ {k _ {l}})) \right\| ^ {2}}{2} = \lim  _ {l \rightarrow \infty} V ^ {0} (t _ {k _ {l}}) \leq \frac {\| m (\mathbb {A} (X _ {0})) \| ^ {2}}{2}.
$$
By the way as  $\lim_{l\to \infty}X(t_{k_l}) = X_0$  and  $\tilde{\mathbb{A}} (X(t_{k_l}))\in \mathbb{A}(X(t_{k_l}))$ , by closed graph theorem we have  $\lim_{l\to \infty}\tilde{\mathbb{A}} (X(t_{k_l}))\in \mathbb{A}(X_0)$ . As of  $m(\mathbb{A}(X_0))$  is the element in  $\mathbb{A}(X_0)$  with smallest norm, we have  $\lim_{l\to \infty}\tilde{\mathbb{A}} (X(t_{k_l})) = m(\mathbb{A}(X_0))$ . As all convergent subsequence converges to the same limit  $m(\mathbb{A}(X_0))$ , we conclude  $\lim_{k\to \infty}\tilde{\mathbb{A}} (X(t_k)) = m(\mathbb{A}(X_0))$ .
$$
\lim _ {k \to \infty} V ^ {0} (t _ {k}) = \lim _ {k \to \infty} \frac {C (t _ {k}) ^ {2}}{2} \left\| \tilde {\mathbf {A}} (X (t _ {k})) \right\| ^ {2} = \frac {\| m (\mathbf {A} (X _ {0})) \| ^ {2}}{2} = \lim _ {n \to \infty} V _ {\lambda_ {n}} ^ {0} (0).
$$
As  $t_k$  was arbitrary positive sequence converges to 0, we conclude  $\lim_{n\to \infty}V_{\lambda_n}^0 (0) = \lim_{t\to 0 + }V^0 (t) = V^0 (0)$ . Therefore
$$
\lim  _ {n \rightarrow \infty} V _ {\lambda_ {n}} (0) = V ^ {0} (0) - \int_ {0} ^ {t _ {0}} \frac {d}{d s} \left(\frac {C (s) ^ {2} \dot {\beta} (s)}{2}\right) \| X (s) - X _ {0} \| ^ {2} d s = \lim  _ {t \rightarrow 0 +} V (t).
$$
E.4 Proof of Lemma 3.4
Most of the proof is done in the main text. Recall  $\Phi(t)$  is defined as  $\Phi(t) = \left\| \tilde{\mathbf{A}}(X(t)) \right\|^2 + 2\beta(t) \left\langle \tilde{\mathbf{A}}(X(t)), X(t) - X_0 \right\rangle$ , and from (8) we have
$$
\Phi (t) \geq \frac {1}{2} \left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} - 2 \beta (t) ^ {2} \| X _ {0} - X _ {\star} \| ^ {2}.
$$
$$
\frac {2 V (t)}{C (t) ^ {2}} - \Phi (t) = (\beta (t) ^ {2} + \dot {\beta} (t)) \| X (t) - X _ {0} \| ^ {2} - \frac {2}{C (t) ^ {2}} \int_ {t _ {0}} ^ {t} \frac {d}{d s} \left(\frac {C (s) ^ {2} \dot {\beta} (s)}{2}\right) \| X (s) - X _ {0} \| ^ {2} d s.
$$
Note  $C(t) \neq 0$  if  $t > 0$ , we can divide with  $C(t)^2$ . Now from Corollary 3.3 we have we have  $V(t) \geq V(0)$  for  $t > 0$ , and so  $\frac{V(t)}{C(t)^2} - \frac{V(0)}{C(t)^2} \geq 0$ . Thus for  $t > \epsilon$
$$
\begin{array}{l} \frac {2 V (0)}{C (t) ^ {2}} - \left(\beta (t) ^ {2} + \dot {\beta} (t)\right) \| X (t) - X _ {0} \| ^ {2} + \frac {2}{C (t) ^ {2}} \int_ {t _ {0}} ^ {t} \frac {d}{d s} \left(\frac {C (s) ^ {2} \dot {\beta} (s)}{2}\right) \| X (s) - X _ {0} \| ^ {2} d s \\ = \frac {2 V (0)}{C (t) ^ {2}} - \left(\frac {2 V (t)}{C (t) ^ {2}} - \Phi (t)\right) \\ = \left(\frac {2 V (0)}{C (t) ^ {2}} - \frac {2 V (t)}{C (t) ^ {2}}\right) + \Phi (t) \geq \Phi (t) \geq \frac {1}{2} \left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} - 2 \beta (t) ^ {2} \| X _ {0} - X _ {\star} \| ^ {2}. \\ \end{array}
$$
Moving  $2\beta (t)^{2}\| X_{0} - X_{\star}\|^{2}$  to the left hand side and multiplying with 2 we get the desired result
$$
\begin{array}{l} \left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} \leq 4 \beta (t) ^ {2} \| X _ {0} - X _ {\star} \| ^ {2} + \frac {4 V (0)}{C (t) ^ {2}} - 2 (\beta (t) ^ {2} + \dot {\beta} (t)) \| X (t) - X _ {0} \| ^ {2} \\ + \frac {2}{C (t) ^ {2}} \int_ {t _ {0}} ^ {t} \frac {d}{d s} \left(C (s) ^ {2} \dot {\beta} (s)\right) \| X (s) - X _ {0} \| ^ {2} d s. \\ \end{array}
$$
E.5 Proof of Theorem 3.1
Restating Lemma 3.4, we know for  $t > 0$
$$
\begin{array}{l} \left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} \leq 4 \beta (t) ^ {2} \| X _ {0} - X _ {\star} \| ^ {2} + \frac {2 V (0)}{C (t) ^ {2}} - 2 (\beta (t) ^ {2} + \dot {\beta} (t)) \| X (t) - X _ {0} \| ^ {2} \tag {7} \\ + \frac {2}{C (t) ^ {2}} \int_ {t _ {0}} ^ {t} \frac {d}{d s} \left(C (s) ^ {2} \dot {\beta} (s)\right) \| X (s) - X _ {0} \| ^ {2} d s. \\ \end{array}
$$
As we know  $\| X(t) - X_0 \| \leq 2 \| X_0 - X_\star \|$  from Lemma D.1, it is clear that
$$
4 \beta (t) ^ {2} \| X _ {0} - X _ {\star} \| ^ {2} = \mathcal {O} (\beta (t) ^ {2})
$$
$$
\frac {2 V (0)}{C (t) ^ {2}} = \mathcal {O} \left(\frac {1}{C (t) ^ {2}}\right) \tag {38}
$$
$$
2 \left(\beta (t) ^ {2} + \dot {\beta} (t)\right) \| X (t) - X _ {0} \| ^ {2} = \mathcal {O} \left(\beta (t) ^ {2}\right) + \mathcal {O} \left(\dot {\beta} (t)\right).
$$
Therefore it remains to show
$$
\frac {2}{C (t) ^ {2}} \int_ {t _ {0}} ^ {t} \frac {d}{d s} \left(C (s) ^ {2} \dot {\beta} (s)\right) \| X (s) - X _ {0} \| ^ {2} d s = \mathcal {O} (\beta (t) ^ {2}) + \mathcal {O} \left(\frac {1}{C (t) ^ {2}}\right) + \mathcal {O} (\dot {\beta} (t)).
$$
We can check there is  $T > 0$  such that for  $s > T$  the sign of  $\frac{d}{ds}\left(C(s)^2\dot{\beta}(s)\right)$  does not change, for  $\beta(t) = \frac{\gamma}{t^p}$  with  $\gamma > 0$ ,  $p > 0$ . We first proceed our proof with assuming this condition. The point for this condition is that following equality holds for  $t > T$ .
$$
\left| \int_ {T} ^ {t} \frac {d}{d s} (C (s) ^ {2} \dot {\beta} (s)) d s \right| = \int_ {T} ^ {t} \left| \frac {d}{d s} (C (s) ^ {2} \dot {\beta} (s)) \right| d s.
$$
Applying above equality and using  $\| X(t) - X_0\| \leq 2\| X_0 - X_\star \|$ , we see
$$
\begin{array}{l} \left| \frac {2}{C (t) ^ {2}} \int_ {t _ {0}} ^ {t} \frac {d}{d s} \left(C (s) ^ {2} \dot {\beta} (s)\right) \| X (s) - X _ {0} \| ^ {2} d s \right| \\ \leq \frac{2}{C(t)^{2}}\underbrace{\left|\int_{t_{0}}^{T}\frac{d}{ds}\left(C(s)^{2}\dot{\beta} (s)\right)\|X(s) - X_{0}\|^{2}ds\right|}_{= M} + \frac{2}{C(t)^{2}}\int_{T}^{t}\left|\frac{d}{ds}\left(C(s)^{2}\dot{\beta} (s)\right)\| X(s) - X_{0}\|^{2}\right|  ds \\ \leq \frac {2 M}{C (t) ^ {2}} + \frac {2}{C (t) ^ {2}} \int_ {T} ^ {t} \left| \frac {d}{d s} \left(C (s) ^ {2} \dot {\beta} (s)\right) 4 \| X _ {0} - X _ {\star} \| ^ {2} \right| d s \\ = \frac {2 M}{C (t) ^ {2}} + \frac {8 \| X _ {0} - X _ {\star} \| ^ {2}}{C (t) ^ {2}} \left| C (t) ^ {2} \dot {\beta} (t) - C (T) ^ {2} \dot {\beta} (T) \right| \\ \leq \frac {2}{C (t) ^ {2}} \left(M + 4 \| X _ {0} - X _ {\star} \| ^ {2} C (T) ^ {2} \left| \dot {\beta} (T) \right|\right) + 8 \| X _ {0} - X _ {\star} \| ^ {2} \left| \dot {\beta} (t) \right| = \mathcal {O} \left(\frac {1}{C (t) ^ {2}}\right) + \mathcal {O} \left(\dot {\beta} (t)\right). \\ \end{array}
$$
Therefore from (7) and (38), we conclude
$$
\left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} = \mathcal {O} (\beta (t) ^ {2}) + \mathcal {O} \left(\frac {1}{C (t) ^ {2}}\right) + \mathcal {O} (\dot {\beta} (t)).
$$
It remains to show there is  $T > 0$  such that for  $s > T$  the sign of  $\frac{d}{ds}\left(C(s)^2\dot{\beta}(s)\right)$  does not change. It can be shown by easy, but a little complicated calculations. Since  $\dot{C}(t) = C(t)\beta(t)$ , we see
$$
\frac {d}{d s} \left(C (s) ^ {2} \dot {\beta} (s)\right) = 2 C (s) ^ {2} \beta (s) \dot {\beta} (s) + C (s) ^ {2} \ddot {\beta} (s) = C (s) ^ {2} \left(2 \beta (s) \dot {\beta} (s) + \ddot {\beta} (s)\right).
$$
Therefore it is enough to check the sign of  $2\beta(s)\dot{\beta}(s) + \ddot{\beta}(s)$ . Observe
$$
2 \beta (s) \dot {\beta} (s) + \ddot {\beta} (s) = - \frac {2 p \gamma^ {2}}{s ^ {2 p + 1}} + \frac {p (1 + p) \gamma}{s ^ {p + 2}} = \left\{ \begin{array}{l l} \frac {2}{s ^ {3}} \gamma (1 - \gamma) & \text {i f} p = 1 \\ \frac {p (p + 1) \gamma}{s ^ {2 p + 1}} \left(s ^ {p - 1} - \frac {2 \gamma}{p + 1}\right) & \text {i f} p \neq 1. \end{array} \right.
$$
Thus when  $p = 1$ , for all  $s > 0$  it is nonpositive if  $\gamma \geq 1$  and nonnegative if  $0 < \gamma < 1$ .
When  $p \neq 1$ , we see
$$
\lim  _ {s \to 0 +} s ^ {p - 1} = \left\{ \begin{array}{l l} \infty & \text {i f} 0 <   p <   1 \\ 0 & \text {i f} p > 1 \end{array} \right., \qquad \qquad \qquad \lim  _ {s \to \infty} s ^ {p - 1} = \left\{ \begin{array}{l l} 0 & \text {i f} 0 <   p <   1 \\ \infty & \text {i f} p > 1 \end{array} \right.
$$
Therefore by intermediate value theorem there is  $T > 0$  such that  $T^{p - 1} - \frac{2\gamma}{p + 1} = 0$ , and for that  $T$  we have  $s^{p - 1} - \frac{2\gamma}{p + 1}$  is nonpositive for  $s > T$  if  $0 < p < 1$  and nonnegative for  $s > T$  if  $p > 1$ . This concludes the proof.
E.5.1 Proof for the convergence rate in Table 1
Recall, from (31) we have
$$
C (t) = \left\{ \begin{array}{l l} t ^ {\gamma} & p = 1 \\ e ^ {\frac {\gamma}{1 - p}} t ^ {1 - p} & p > 0,   p \neq 1. \end{array} \right.
$$
We now observe  $\dot{\beta}(t)$  does not effect to convergence rate. In other words, we show  $\dot{\beta}(t)$  is not the slowest one that goes to zero compared to  $\beta(t)^2$  and  $\frac{1}{C(t)^2}$  for every case.
Comparing  $\dot{\beta}(t) = -\frac{p\gamma}{t^{p+1}}$  with  $\beta(t)^2 = \frac{\gamma^2}{t^{2p}}$ , we see  $\dot{\beta}(t) = \mathcal{O}(\beta(t)^2)$  when  $0 < p \leq 1$ .
For  $p > 1$ , we see  $\lim_{t\to \infty}\frac{1}{C(t)^2} = 1\neq 0$ . As  $\lim_{t\to \infty}\dot{\beta} (t) = -\lim_{t\to \infty}\frac{p\gamma}{tp + 1} = 0$ , we have  $\dot{\beta} (t) = \mathcal{O}\left(\frac{1}{C(t)^2}\right)$ .
From (i) and (ii), we conclude
$$
\left\| \tilde {\mathbf {A}} (X (t)) \right\| ^ {2} = \mathcal {O} (\beta (t) ^ {2}) + \mathcal {O} \left(\frac {1}{C (t) ^ {2}}\right).
$$
Now the results in Table 1 is straightforward.
-  $p = 1$
When  $p = 1$ , we have  $C(t) = t^{\gamma}$ . Comparing  $\frac{1}{C(t)^2} = \frac{1}{t^{2\gamma}}$  and  $\beta (t)^2 = \frac{\gamma^2}{t^2}$ ,
(1)  $\left\| \tilde{\mathbf{A}} (X(t))\right\| ^2 = \mathcal{O}\left(\beta (t)^2\right) = \mathcal{O}\left(\frac{1}{t^2}\right)$  if  $\gamma \geq 1$
(2)  $\left\| \tilde{\mathbf{A}} (X(t))\right\| ^2 = \mathcal{O}\left(\frac{1}{C(t)^2}\right) = \mathcal{O}\left(\frac{1}{t^{2\gamma}}\right)$  if  $0 <   \gamma <  1$
-  $0 < p < 1$
When  $0 < p < 1$ , we have
$$
\lim  _ {t \rightarrow \infty} \frac {1}{\beta (t) ^ {2}} \cdot \frac {1}{C (t) ^ {2}} = \frac {1}{\gamma^ {2}} \lim  _ {t \rightarrow \infty} \frac {t ^ {2}}{e ^ {\frac {2 \gamma}{1 - p}} t ^ {1 - p}} = 0.
$$
Therefore,  $\left\| \tilde{\mathbf{A}} (X(t))\right\| ^2 = \mathcal{O}\left(\beta (t)^2\right) = \mathcal{O}\left(\frac{1}{t^{2p}}\right)$ .
$p > 1$
We observed previously that  $\lim_{t\to \infty}\frac{1}{C(t)^2} = 1\neq 0$  when  $p > 1$ . Therefore  $\left\| \tilde{\mathbf{A}} (X(t))\right\| ^2 = \mathcal{O}\left(\frac{1}{C(t)^2}\right) = \mathcal{O}\left(1\right)$ .
E.6 Proof of Theorem 3.5
$$
\bar{X}_{\star} = \Pi_{\text{Zer}\mathbf{A}}(X_{0}) = \operatorname *{argmin}_{z\in \text{Zer}\mathbf{A}}\left\| z - X_{0}\right\| .
$$
We first show
$$
\limsup_{t\to \infty}\big\langle X(t) - \bar{X}_{\star},X_{0} - \bar{X}_{\star}\big\rangle \leq 0.
$$
Proof by contradiction. Suppose not.
Then there is  $\epsilon > 0$  such that for every  $k \in \mathbb{N}$ , there is  $n_k \in [0, \infty)$  such that
$$
\left\langle X \left(n _ {k}\right) - \bar {X} _ {\star}, X _ {0} - \bar {X} _ {\star} \right\rangle > \epsilon
$$
By the way, from Lemma D.1 we know  $X(n_{k})\in \bar{B}_{\| X_{0} - \bar{X}_{\star}\|}(\bar{X}_{\star})$ . Thus by Bolzano-Weierstrass theorem, there is a converging subsequence  $\{X\left(n_{k(l)}\right)\}_{l\in (N)}$ . Name the limit as  $X_{\infty}$ . Since  $\lim_{t\to \infty}\|\tilde{\mathbf{A}}(X(t))\| = 0$ ,  $\{\tilde{\mathbf{A}}(X(n_{k(l)})\}_{l\in (N)}$  converges to 0. Then from [15, Proposition 20.38],  $(X_{\infty},0)\in \operatorname{Gra}\mathbb{A}$ , i.e.  $X_{\infty}\in \operatorname{Zer}\mathbb{A}$ .
On the other hand, as  $\mathbb{A}$  is maximal monotone, by [15, Proposition 23.39] Zer  $\mathbb{A}$  is closed and convex. Since  $\bar{X}_{\star} = \Pi_{\mathrm{Zer}\mathbb{A}}(X_0)$ , by [15, Theorem 3.16] we have  $\langle X_{\infty} - \bar{X}_{\star}, X_0 - \bar{X}_{\star} \rangle \leq 0$ . Thus
$$
0 <   \epsilon \leq \lim _ {l \rightarrow \infty} \left\langle X \left(n _ {k (l)}\right) - \bar {X} _ {\star}, X _ {0} - \bar {X} _ {\star} \right\rangle = \left\langle X _ {\infty} - \bar {X} _ {\star}, X _ {0} - \bar {X} _ {\star} \right\rangle \leq 0,
$$
this is a contradiction, therefore we get the desired result.
Now we show  $\lim_{t\to \infty}\left\| X(t) - \bar{X}_{\star}\right\| = 0$ . Recall, for  $t > 0$
$$
\dot {X} \in - \mathbb {A} (X (t)) - \beta (t) (X - X _ {0}).
$$
From  $\dot{C}(t) = C(t)\beta(t)$ , and monotonicity of  $\mathbb{A}$ , we observe
$$
\begin{array}{l} \frac {d}{d t} \left(C (t) ^ {2} \| X (t) - \bar {X} _ {\star} \| ^ {2}\right) = 2 C (t) ^ {2} \left\langle \dot {X} (t), X (t) - \bar {X} _ {\star} \right\rangle + 2 C (t) ^ {2} \beta (t) \| X (t) - \bar {X} _ {\star} \| ^ {2} \\ = 2 C (t) ^ {2} \left\langle - \tilde {\mathbf {A}} (X (t)) - \beta (t) (X (t) - X _ {0}), X (t) - \bar {X} _ {\star} \right\rangle + 2 C (t) ^ {2} \beta (t) \| X (t) - \bar {X} _ {\star} \| ^ {2} \\ \leq - 2 C (t) ^ {2} \beta (t) \left\langle X (t) - X _ {0}, X (t) - \bar {X} _ {\star} \right\rangle + 2 C (t) ^ {2} \beta (t) \left\| X (t) - \bar {X} _ {\star} \right\| ^ {2} \\ = 2 C (t) ^ {2} \beta (t) \left\langle X _ {0} - \bar {X} _ {\star}, X (t) - \bar {X} _ {\star} \right\rangle . \\ \end{array}
$$
Now take  $\epsilon$ . From  $\lim \sup_{t \to \infty} \left\langle X(t) - \bar{X}_{\star}, X_0 - \bar{X}_{\star} \right\rangle \leq 0$ , there is  $M > 0$  such that
$$
t > M \quad \Longrightarrow \quad \left\langle X _ {0} - \bar {X} _ {\star}, X (t) - \bar {X} _ {\star} \right\rangle <   \epsilon .
$$
For that  $M$  and  $t > M$ , integrating from  $M$  to  $t$  we get
$$
\begin{array}{l} \left[ C (s) ^ {2} \left\| X (s) - \bar {X} _ {\star} \right\| ^ {2} \right] _ {M} ^ {t} = C (t) ^ {2} \left\| X (t) - \bar {X} _ {\star} \right\| ^ {2} - C (M) ^ {2} \left\| X (M) - \bar {X} _ {\star} \right\| ^ {2} \\ \leq \int_ {M} ^ {t} 2 C (s) ^ {2} \beta (s) \left\langle X _ {0} - \bar {X} _ {\star}, X (s) - \bar {X} _ {\star} \right\rangle d s \\ \leq \int_ {M} ^ {t} \left| 2 C (s) ^ {2} \beta (s) \left\langle X _ {0} - \bar {X} _ {\star}, X (s) - \bar {X} _ {\star} \right\rangle \right| d s \\ \leq \int_ {M} ^ {t} 2 C (s) ^ {2} \beta (s) \epsilon d s = \epsilon \left[ C (s) ^ {2} \right] _ {M} ^ {t} = \epsilon \left(C (t) ^ {2} - C (M) ^ {2}\right). \\ \end{array}
$$
By dividing  $C(t)^2$  and organizing,
$$
\left\| X (t) - \bar {X} _ {\star} \right\| ^ {2} \leq \epsilon \left(1 - \left(\frac {C (M)}{C (t)}\right) ^ {2}\right) + \left(\frac {C (M)}{C (t)}\right) ^ {2} \left\| X (M) - \bar {X} _ {\star} \right\| ^ {2}.
$$
By the way from assumption, we have  $\lim_{t\to \infty}\frac{1}{C(t)} = 0$ . Therefore we conclude
$$
\lim  _ {t \rightarrow \infty} \left|\left| X (t) - \bar {X} _ {\star} \right|\right| ^ {2} \leq \epsilon .
$$
Since  $\epsilon > 0$  was arbitrary, we get the desired result.
F Proof for worst case examples
The explicit solution for linear  $A$  is crucially used in worst case examples. Therefore, we first provide proof for it.
F.1 Proof for explicit solution for linear  $A$
F.1.1 Proof of Lemma 4.1
$$
X (t) = \sum_ {n = 0} ^ {\infty} \frac {(- t A) ^ {n}}{\Gamma (n + \gamma + 1)} \Gamma (\gamma + 1) X _ {0} = X _ {0} + \sum_ {n = 1} ^ {\infty} \frac {(- t A) ^ {n}}{\Gamma (n + \gamma + 1)} \Gamma (\gamma + 1) X _ {0},
$$
we can check  $X(0) = X_0$ . Now by using the property of Gamma function  $\Gamma(x + 1) = x\Gamma(x)$ , and paying attention to the lower bound of the summation index we have
$$
\begin{array}{l} \dot {X} (t) = \sum_ {n = 1} ^ {\infty} \frac {(- n A) (- t A) ^ {n - 1}}{\Gamma (n + \gamma + 1)} \Gamma (\gamma + 1) X _ {0} = A \sum_ {n = 1} ^ {\infty} \frac {(- n - \gamma + \gamma) (- t A) ^ {n - 1}}{\Gamma (n + \gamma + 1)} \Gamma (\gamma + 1) X _ {0} \\ = - A \sum_ {n = 1} ^ {\infty} \frac {(- t A) ^ {n - 1}}{\Gamma (n + \gamma)} \Gamma (\gamma + 1) X _ {0} + \gamma \sum_ {n = 1} ^ {\infty} \frac {(- t) ^ {n - 1} A ^ {n}}{\Gamma (n + \gamma + 1)} \Gamma (\gamma + 1) X _ {0} \\ = - A \sum_ {n = 0} ^ {\infty} \frac {(- t A) ^ {n}}{\Gamma (n + \gamma + 1)} \Gamma (\gamma + 1) X _ {0} + \frac {\gamma}{(- t)} \sum_ {n = 1} ^ {\infty} \frac {(- t) ^ {n} A ^ {n}}{\Gamma (n + \gamma + 1)} \Gamma (\gamma + 1) X _ {0} = - A X (t) - \frac {\gamma}{t} (X (t) - X _ {0}). \\ \end{array}
$$
The solution can be written in another form
$$
X (t) = \gamma e ^ {- t A} t ^ {- \gamma} \int_ {0} ^ {t} u ^ {\gamma - 1} e ^ {u A} d u X _ {0}. \tag {39}
$$
As this is a special case of (9), here we just briefly check this satisfies the ODE and check other details in the proof of Lemma 4.2. By product rule of differentiation,
$$
\begin{array}{l} \dot {X} = \gamma \left(\frac {d}{d t} e ^ {- t A}\right) t ^ {- \gamma} \int_ {0} ^ {t} u ^ {\gamma - 1} e ^ {u A} d u X _ {0} + \gamma e ^ {- t A} \left(\frac {d}{d t} t ^ {- \gamma}\right) \int_ {0} ^ {t} u ^ {\gamma - 1} e ^ {u A} d u X _ {0} + \gamma e ^ {- t A} t ^ {- \gamma} \frac {d}{d t} \left(\int_ {0} ^ {t} u ^ {\gamma - 1} e ^ {u A} d u X _ {0}\right) \\ = - A (X (t)) - \frac {\gamma}{t} X (t) + \gamma e ^ {- t A} t ^ {- \gamma} \left(t ^ {\gamma - 1} e ^ {t A}\right) X _ {0} \\ = - A (X (t)) - \frac {\gamma}{t} (X (t) - X _ {0}). \\ \end{array}
$$
F.1.2 Proof of Lemma 4.2
$$
X _ {*} (t) = \left(I - \frac {A e ^ {- t A}}{C (t)} \left(\int_ {0} ^ {t} e ^ {s A} C (s) d s\right)\right) X _ {0}.
$$
We show  $X_{*}$  is a well-defined solution for (3) with linear  $\mathbb{A}$ , then show  $X_{*}$  is equal to  $X(t)$  defined in (9).
We first check well-definedness. By definition,  $C(t) = e^{\int_{t_0}^t \beta(s) ds}$  is nondecreasing. Also,  $\| e^{tA} \|$  is also nondecreasing since from monotonicity of  $A$  we have for all  $x \in \mathbb{R}^n$
$$
\frac {d}{d t} \left\| e ^ {t A} x \right\| ^ {2} = 2 \left\langle A (e ^ {t A} x), e ^ {t A} x \right\rangle \geq 0.
$$
Now we see  $X(t)$  is well-defined since
$$
\begin{array}{l} \left\| e ^ {- t A} \frac {1}{C (t)} \left(\int_ {0} ^ {t} e ^ {s A} C (s) d s\right) \right\| = \left\| \int_ {0} ^ {t} e ^ {(s - t) A} \frac {C (s)}{C (t)} d s \right\| \\ \leq \int_ {0} ^ {t} \left\| e ^ {(s - t) A} \right\| \left| \frac {C (s)}{C (t)} \right| d s \leq \int_ {0} ^ {t} (1 \cdot 1) d s = t <   \infty . \\ \end{array}
$$
Also above inequality implies second term reaches to zero as  $t \to 0+$ , we have  $\lim_{t \to 0+} X_*(t) = X_0$ . Thus defining  $X_*(0) = X_0$  if necessary, we see  $X_*$  satisfies the initial condition with  $X_* \in \mathcal{C}([0, \infty), \mathbb{R}^n)$ .
We then check  $X_{*}(t)$  becomes the solution. This is immediate from product rule for differentiation. Since  $\frac{d}{dt}\left(\frac{1}{C(t)}\right) = -\frac{1}{C(t)^2} C(t)\beta(t) = -\frac{1}{C(t)}\beta(t)$ , we have
$$
\begin{array}{l} \dot {X} _ {*} (t) = - \left(\frac {d}{d t} e ^ {- t A}\right) \frac {1}{C (t)} \left(\int_ {0} ^ {t} e ^ {s A} C (s) d s\right) X _ {0} \\ - A e ^ {- t A} \left(\frac {d}{d t} \frac {1}{C (t)}\right) \left(\int_ {0} ^ {t} e ^ {s A} C (s) d s\right) X _ {0} \\ - A e ^ {- t A} \frac {1}{C (t)} \frac {d}{d t} \left(\int_ {0} ^ {t} e ^ {s A} C (s) d s\right) X _ {0} \\ = - A \left(X _ {*} (t) - X _ {0}\right) - \beta (t) \left(X _ {*} (t) - X _ {0}\right) - A (X _ {0}) = - A (X (t)) - \beta (t) \left(X _ {*} (t) - X _ {0}\right). \\ \end{array}
$$
Finally we now show  $X(t) = X_{*}(t)$ , where  $X(t)$  is defined as (9). From integral by parts we have
$$
\begin{array}{l} X (t) = e ^ {- t A} \frac {1}{C (t)} \left(\int_ {0} ^ {t} e ^ {s A} C (s) \beta (s) d s + C (0) I\right) X _ {0} \\ = e ^ {- t A} \frac {1}{C (t)} \left(\left[ e ^ {s A} C (s) \right] _ {0} ^ {t} - \int_ {0} ^ {t} A e ^ {s A} C (s) d s + C (0) I\right) X _ {0} \\ = \left(I - A e ^ {- t A} \frac {1}{C (t)} \left(\int_ {0} ^ {t} e ^ {s A} C (s) d s\right)\right) X _ {0} = X _ {*} (t). \\ \end{array}
$$
F.2 Proof of Theorem 4.3
If  $\beta (t)\equiv 0$  , for  $A\coloneqq \left( \begin{array}{ll}0 & 1\\ -1 & 0 \end{array} \right)$  , we have  $X(t) = e^{-tA}$  , so
$$
\lim  _ {t \to \infty} \| A (X (t)) \| = 1 \neq 0.
$$
So let's consider the case  $\beta$  is not  $\beta(t) \equiv 0$  and  $\beta(t) \geq 0$ . For
$$
A _ {\xi} = 2 \pi \xi \left( \begin{array}{c c} 0 & 1 \\ - 1 & 0 \end{array} \right)
$$
name the solution for ODE  $\dot{X}_{\xi} = -A_{\xi}(X_{\xi}) - \beta (t)(X_{\xi} - X_0)$  as  $X_{\xi}$ . By Lemma 4.2 we have
$$
X _ {\xi} (t) = \frac {e ^ {- t A _ {\xi}}}{C (t)} \left(\int_ {0} ^ {t} e ^ {s A _ {\xi}} C (s) \beta (s) d s + C (0) I\right) X _ {0}.
$$
We want to show,
$$
\exists \xi \in \mathbb {R}, \quad \lim  _ {t \rightarrow \infty} \| A _ {\xi} (X _ {\xi} (t)) \| \neq 0
$$
From  $\lim_{t\to \infty}\frac{1}{C(t)}\neq 0$  we see  $\lim_{t\to \infty}\left\| \frac{e^{-tA\xi}}{C(t)}\right\| = 1\cdot \frac{1}{C(t)}\neq 0.$  And since  $A_{\xi}$  is invertible except the case  $\xi = 0$  , we have
$$
\begin{array}{l} \lim  _ {t \rightarrow \infty} \| A _ {\xi} (X _ {\xi} (t)) \| = 0 \Longleftrightarrow \lim  _ {t \rightarrow \infty} \| X _ {\xi} (t) \| = 0 \\ \Longleftrightarrow \left\| \lim  _ {t \rightarrow \infty} \int_ {0} ^ {t} e ^ {s A _ {\xi}} C (s) \beta (s) d s + C (0) I \right\| = 0. \\ \end{array}
$$
Now let assume  $\forall \xi \in \mathbb{R}$ ,  $\lim_{t\to \infty}\| A_{\xi}(X_{\xi}(t))\| = 0$  and lead to contradiction. Define  $f:\mathbb{R}\rightarrow \mathbb{R}$  as
$$
f (s) = \left\{ \begin{array}{l l} C (s) \beta (s) & s > 0 \\ 0 & s \leq 0. \end{array} \right.
$$
Then  $\int_{-\infty}^{\infty}f(s)ds = [C(s)]_0^\infty = \lim_{t\to \infty}C(t) - C(0) < \infty$  we have  $f\in L^{1}(\mathbb{R},\mathbb{R})$ . Note  $\lim_{t\to \infty}C(t)$  exists, since  $C$  is nondecreasing and by the assumption is bounded.
Now setting  $X_0 = (1,0)^T$  and from  $e^{sA_{\xi}} = \begin{pmatrix} \cos 2\pi \xi s & \sin 2\pi \xi s \\ -\sin 2\pi \xi s & \cos 2\pi \xi s \end{pmatrix}$ , we see
$$
\begin{array}{l} \int_ {0} ^ {\infty} e ^ {s A _ {\xi}} C (s) \beta (s) d s X _ {0} = \left(\int_ {0} ^ {\infty} \cos (2 \pi s \xi) f (s) d s, \int_ {0} ^ {\infty} \sin (- 2 \pi s \xi) f (s) d s\right) ^ {T} \\ = \left(R e (\hat {f} (\xi)), I m (\hat {f} (\xi))\right) ^ {T} \\ \end{array}
$$
where  $\hat{f} (\xi)$  is Fourier transform of  $f$
From assumption we have  $\forall \xi \in \mathbb{R},\left\| \int_0^\infty e^{sA_\xi}C(s)\beta (s)dsX_0 + C(0)X_0\right\| = 0$  , we have
$$
\left(R e (\hat {f} (\xi)), I m (\hat {f} (\xi))\right) ^ {T} \equiv - C (0) X _ {0}
$$
By the way, from Fourier theory we know  $\hat{f}$  vanishes at infinity, thus
$$
\| C (0) X _ {0} \| = \lim  _ {\xi \rightarrow \infty} \left| \hat {f} (\xi) \right| = 0.
$$