SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Therefore,  $w_{t,i}^{2}$  increases at an exponential rate while  $v_{t,i}^{2}$  decreases at an exponential rate. Stage One ends when  $\beta_{t,i}$  exceeds  $\beta_{i}^{*} / 2$ , and our goal is to estimate  $t_{i,0}$  that satisfies
$$
\beta_ {t, i} \geq \left(1 + \frac {\eta \beta_ {i} ^ {*}}{\gamma n}\right) ^ {t _ {i, 0}} \alpha^ {2} - \left(1 - \frac {\eta \beta_ {i} ^ {*}}{\gamma n}\right) ^ {t _ {i, 0}} \alpha^ {2} \geq \beta_ {i} ^ {*} / 2.
$$
Note that  $\{v_{t,i}^2\}_{t\geq 0}$  forms a decreasing sequence and thus is bounded by  $\alpha^2$ . Hence, it suffices to solve the following inequality for  $t_{i,0}$ ,
$$
\left(1 + \frac {\eta \beta_ {i} ^ {*}}{\gamma n}\right) ^ {t _ {i, 0}} \alpha^ {2} \geq \beta_ {i} ^ {*} / 2 + \alpha^ {2},
$$
which is equivalent to obtain  $t_{i,0}$  satisfying
$$
t _ {i, 0} \geq T _ {i, 0} = \log \left(\frac {\beta^ {*}}{2 \alpha^ {2}} + 1\right) / \log \left(1 + \frac {\eta \beta_ {i} ^ {*}}{\gamma n}\right).
$$
For  $T_{i,0}$ , by direct calculation, we have
$$
\begin{array}{l} T _ {i, 0} \stackrel {(i)} {\leq} \log \left(\frac {\beta^ {*}}{2 \alpha^ {2}} + 1\right) \left(1 + \frac {\eta \beta_ {i} ^ {*}}{\gamma n}\right) / \left(\frac {\eta \beta_ {i} ^ {*}}{\gamma n}\right) \\ \stackrel {(i i)} {\leq} 2 \log \left(\frac {\beta_ {i} ^ {*}}{\alpha^ {2}}\right) / \eta \beta_ {i} ^ {*}, \\ \end{array}
$$
where  $(i)$  follows from  $x\log x - x + 1\geq 0$  when  $x\geq 0$  and  $(ii)$  holds due to  $\log (x / 2 + 1)\leq \log x$  when  $x\geq 2$  as well as the assumption on  $\gamma$  and  $\eta \beta_i^*\leq 1$ . Thus, we set  $t_{i,0} = 2\log (\beta_i^* /\alpha^2) / (\eta \beta_i^*)$  such that for all  $t\geq t_{i,0}$ ,  $\beta_{t,i}\geq \beta_i^* /2$  for all  $i\in S_1$ .
Stage Two. Define  $\epsilon = C_{\epsilon}\sqrt{\log p / n}$  and  $a_{i,1} = \lceil \log_2(\beta_i^* /\epsilon)\rceil$  . Next, we refine the element-wise bound  $\xi_{t,i}$  according to (13). For any fixed  $i\in S_1$  , if there exists some  $a$  such that  $1\leq a\leq a_{i,1}$  and  $(1 - 1 / 2^{a})\beta_{i}^{*}\leq \beta t,i\leq (1 - 1 / 2^{a + 1})\beta_{i}^{*}$  , then, based on the analytical thinking in Stage One, we can easily deduce that
$$
\left| \xi_ {t, i} \right| \lesssim \frac {\eta m}{2 ^ {a} \gamma n \log p}. \tag {20}
$$
Using this element-wise bound (20), we get the increment of  $w_{t+1,i}^2$  and decrement of  $v_{t+1,i}^2$ ,
$$
w _ {t + 1, i} ^ {2} \geq w _ {t, i} ^ {2} \left(1 + \frac {\eta \beta_ {i} ^ {*}}{2 ^ {a} \gamma n}\right) a n d v _ {t + 1, i} ^ {2} \leq v _ {t, i} ^ {2} \left(1 - \frac {\eta \beta_ {i} ^ {*}}{2 ^ {a} \gamma n}\right).
$$
We define  $t_{i,a}$  as the smallest  $t$  such that  $\beta_{t + t_{i,a},i} \geq (1 - 1/2^{a+1})\beta_{i}^{*}$ . Intuitively, suppose that the current estimate  $\beta_{t,i}$  is between  $(1 - 1/2^{a})\beta_{i}^{*}$  and  $(1 - 1/2^{a+1})\beta_{i}^{*}$ , we aim to find the number of iterations required for the sequence  $\{\beta_{t,i}\}_{t \geq 0}$  to exceed  $(1 - 1/2^{a+1})\beta_{i}^{*}$ . To obtain a sufficient condition for  $t_{i,a}$ , we construct the following inequality,
$$
\beta_ {t + t _ {i, a}, i} \geq w _ {t, i} ^ {2} \left(1 + \frac {\eta \beta_ {i} ^ {*}}{2 ^ {a} \gamma n}\right) ^ {t _ {i, a}} - v _ {t, i} ^ {2} \left(1 - \frac {\eta \beta_ {i} ^ {*}}{2 ^ {a} \gamma n}\right) ^ {t _ {i, a}} \geq (1 - 1 / 2 ^ {a + 1}) \beta_ {i} ^ {*}.
$$
Similar with Stage One, the sequence  $\{v_{t,i}^2\}_{t\geq 0}$  is bounded by  $\alpha^2$ . Then, it is sufficient to solve the following inequality for  $t_{i,a}$ ,
$$
t _ {i, a} := T _ {i, a} = \log \left(\frac {(1 - 1 / 2 ^ {a + 1}) \beta_ {i} ^ {*} + \alpha^ {2}}{w _ {t , i} ^ {2}}\right) \Bigg / \log \left(1 + \frac {\eta \beta_ {i} ^ {*}}{2 ^ {a} \gamma n}\right).
$$
To obtain a more precise upper bound of  $T_{i,a}$ , we have
$$
\begin{array}{l} T _ {i, a} = \log \left(\frac {(1 - 1 / 2 ^ {a + 1}) \beta_ {i} ^ {*} + \alpha^ {2}}{w _ {t , i} ^ {2}}\right) \bigg / \log \left(1 + \frac {\eta \beta_ {i} ^ {*}}{2 ^ {a} \gamma n}\right) \\ \stackrel {(i)} {\leq} \log \left(\frac {(1 - 1 / 2 ^ {a + 1}) \beta_ {i} ^ {*} + \alpha^ {2}}{(1 - 1 / 2 ^ {a}) \beta_ {i} ^ {*}}\right) \Bigg / \log \left(1 + \frac {\eta \beta_ {i} ^ {*}}{2 ^ {a} \gamma n}\right) \\ \stackrel {(i i)} {\leq} \log \left(1 + \frac {1 / 2 ^ {a + 1}}{1 - 1 / 2 ^ {a}} + \frac {\alpha^ {2}}{(1 - 1 / 2 ^ {a}) \beta_ {i} ^ {*}}\right) \left(1 + \frac {\eta \beta_ {i} ^ {*}}{2 ^ {a} \gamma n}\right) \Bigg / \left(\frac {\eta \beta_ {i} ^ {*}}{2 ^ {a} \gamma n}\right), \\ \end{array}
$$
where  $(i)$  is based on the condition  $w_{t,i}^{2}\geq (1 - 1 / 2^{a})\beta_{i}^{*}$ ,  $(ii)$  stands since  $x\log x - x + 1\geq 0$  when  $x\geq 0$ . By direct calculation, we obtain
$$
\begin{array}{l} T _ {i, a} \stackrel {(i i i)} {\leq} \left(\frac {1 / 2 ^ {a + 1}}{1 - 1 / 2 ^ {a}} + \frac {\alpha^ {2}}{(1 - 1 / 2 ^ {a}) \beta_ {i} ^ {*}}\right) / \left(\frac {\eta \beta_ {i} ^ {*}}{2 ^ {a + 1} \gamma n}\right) \tag {21} \\ \stackrel {(i v)} {\leq} \frac {2}{\eta \beta_ {i} ^ {*}} + \frac {2 ^ {a + 2} \alpha^ {2}}{\eta \beta_ {i} ^ {* 2}}, \\ \end{array}
$$
where we use  $\log (x + 1)\leq x$  when  $x\geq 0$  in step  $(iii)$  and  $1 - 1 / 2^{a}\geq 1 / 2$  in step  $(iv)$ , respectively.
Recall the assumption  $a \leq a_{i,1} = \lceil \log_2(\beta_i^* /\epsilon) \rceil$  with  $\epsilon = C_{\epsilon}\sqrt{\log p / n}$ , then we get  $2^{a + 2} \leq$
$4\beta_{i}^{*} / \epsilon \leq 4\sqrt{n / \log p}\beta_{i}^{*} / C_{\epsilon}$ . Moreover, by the assumption on the initialization size  $\alpha$ , we have  $\alpha^2 \leq C_\alpha / p^2$ . Thus, we can bound  $2^{a + 2}\alpha^2 / \eta \beta_i^{*2}$  in (21) as
$$
\frac {2 ^ {a + 2} \alpha^ {2}}{\eta \beta_ {i} ^ {* 2}} \leq \sqrt {\frac {n}{\log p}} \frac {4 C _ {\alpha}}{C _ {\epsilon} p ^ {2} \eta \beta_ {i} ^ {*}} \stackrel {(i)} {\leq} \frac {1}{\eta \beta_ {i} ^ {*}}, \tag {22}
$$
where  $(i)$  holds when  $p^2\log p\geq 4C_\alpha \sqrt{n} /C_\epsilon$ . Combined (21) and (22), we calculate the final bound of  $T_{i,a}$  as
$$
T _ {i, a} \leq \frac {3}{\eta \beta_ {i} ^ {*}},
$$
for any  $a \leq a_{i,1}$ . If there exists an  $1 \leq a \leq a_{i,1}$  such that  $\beta_{t,i} \in [(1 - 1/2^a)\beta_i^*, (1 - 1/2^{a+1})\beta_i^*]$ , after  $t_{i,a} \geq 3/\eta\beta_i^*$  iterations, we can guarantee that  $\beta_{t,i} \geq (1 - 1/2^{a+1})\beta_i^*$ . Now recalling the definition of  $a_{i,1}$ , we have  $\beta_i^* / 2^{a_{i,1}} \leq \epsilon = C_\epsilon \sqrt{\log p/n}$ . Therefore, with at most  $\sum_{a=0}^{a_{i,1}} T_{i,a}$  iterations, we have  $\beta_{t,i} \geq \beta_i^* - \epsilon$ . By the assumption of  $\alpha$ , we obtain
$$
\begin{array}{l} \sum_ {a = 0} ^ {a _ {i, 1}} T _ {i, a} \leq 2 \log \left(\beta_ {i} ^ {*} / \alpha^ {2}\right) / \left(\eta \beta_ {i} ^ {*}\right) + 3 \left[ \log_ {2} \left(\frac {\beta_ {i} ^ {*}}{\epsilon}\right) \right] / \left(\eta \beta_ {i} ^ {*}\right) \\ \leq 5 \log \left(\frac {\beta_ {i} ^ {*}}{\alpha^ {2}}\right) / (\eta \beta_ {i} ^ {*}). \\ \end{array}
$$
Since the function  $\log (x / \alpha^2) / (\eta x)$  is decreasing with respect to  $x$ , we have after  $5\log (m / \alpha^{2}) / (\eta m)$  iterations that  $|\beta_{t,i} - \beta_i^* |\lesssim \sqrt{\log p / n}$  for any fixed  $i\in S_1$ . Thus, we complete the proof of Proposition 2.