SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Model 1:  $\mathbf{x} \sim MN(\mathbf{0}_p, \mathbf{\Sigma})$ ,  $\boldsymbol{\Sigma} = (\sigma_{ij})$  with nonzero elements  $\sigma_{ij} = 0.4^{|i - j|}$  for  $1 \leq i, j \leq p$ ,  $P(y = 1|\mathbf{x}) = \Phi(\mathbf{x}^T\boldsymbol{\beta}^*)$ , where  $\Phi(\cdot)$  is the cumulative density function of the standard normal distribution,  $\boldsymbol{\beta}^* = (1.1, 1.1, 1.1, 1.1, 1.1, 0, \dots, 0)^T$  and  $s = 4$ .
- Model 2:  $P(y = 1) = P(y = -1) = 0.5$ ,  $\mathbf{x} \mid (y = 1) \sim MN(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ ,  $\mathbf{x} \mid (y = -1) \sim MN(-\boldsymbol{\mu}, \boldsymbol{\Sigma})$ ,  $s = 5$ ,  $\boldsymbol{\mu} = (0.1, 0.2, 0.3, 0.4, 0.5, 0, \dots, 0)^T$ ,  $\boldsymbol{\Sigma} = (\sigma_{ij})$  with diagonal entries equal to 1, nonzero entries  $\sigma_{ij} = -0.2$  for  $1 \leq i \neq j \leq s$  and other entries equal to 0. The bayes rule is  $\mathrm{sign}(1.39\mathbf{x}_1 + 1.47\mathbf{x}_2 + 1.56\mathbf{x}_3 + 1.65\mathbf{x}_4 + 1.74\mathbf{x}_5)$  with bayes error  $6.3\%$ .
We follow the default parameter setup from Section 4, and the experiments are repeated 30 times. The averaged estimation and prediction results are presented in Figure 11. From Figure 11, it's evident that the GD estimator closely approximates the oracle estimator in terms of estimation error and prediction accuracy.
C Gradient Descent Dynamics
First, let's recall the gradient descent updates. We over-parameterize  $\beta$  by writing it as  $\mathbf{w} \odot \mathbf{w} - \mathbf{v} \odot \mathbf{v}$ , where  $\mathbf{w}$  and  $\mathbf{v}$  are  $p \times 1$  vectors. We then apply gradient descent to the following optimization problem:
$$
\min  _ {\mathbf {w}, \mathbf {v}} \max  _ {\boldsymbol {\mu} \in \mathcal {P} _ {1}} \left\{\frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - y _ {i} \mathbf {x} _ {i} ^ {T} (\mathbf {w} \odot \mathbf {w} - \mathbf {v} \odot \mathbf {v})\right) \mu_ {i} - d _ {\gamma} (\boldsymbol {\mu}) \right\},
$$
(a) Results in Figure 2
(b) Results in Figure 3
(c) Results in Figure 4
(d) Results in Figure 5
Figure 8: The True Negative results correspond to the settings illustrated in Figures 2, 3, 4 and 5, respectively.
where  $d_{\gamma}(\pmb{\mu}) = \frac{\gamma}{2}\|\pmb{\mu}\|^2$ . The gradient descent updates with respect to  $\mathbf{w}$  and  $\mathbf{v}$  are given by
$$
\mathbf {w} _ {t + 1} = \mathbf {w} _ {t} + 2 \eta \frac {1}{n} \sum_ {i = 1} ^ {n} y _ {i} \mu_ {t, i} \mathbf {x} _ {i} \odot \mathbf {w} _ {t} \quad a n d \quad \mathbf {v} _ {t + 1} = \mathbf {v} _ {t} - 2 \eta \frac {1}{n} \sum_ {i = 1} ^ {n} y _ {i} \mu_ {t, i} \mathbf {x} _ {i} \odot \mathbf {v} _ {t}.
$$
For the sake of convenience, let  $\mathcal{G}_t\in \mathbb{R}^p$  represent the gradients in the form of  $\mathcal{G}_t = n^{-1}\mathbf{X}^T\mathbf{Y}\boldsymbol{\mu}_t$  where  $\mathbf{Y}$  is a diagonal matrix composed of the elements of  $y$ . Consequently, the  $i$ -th element of  $\mathcal{G}_t$  can be expressed as  $G_{t,i} = n^{-1}\sum_{k = 1}^{n}\mu_{t,k}y_kx_{ki}$ , where  $\mu_{t,k} = \mathrm{median}(0,(1 - y_k\mathbf{x}_k^T\boldsymbol {\beta}_t) / \gamma n,1)$ . Subsequently, the updating rule can be rephrased as follows:
$$
\mathbf {w} _ {t + 1} = \mathbf {w} _ {t} + 2 \eta \mathbf {w} _ {t} \odot \mathcal {G} _ {t} \quad a n d \quad \mathbf {v} _ {t + 1} = \mathbf {v} _ {t} - 2 \eta \mathbf {v} _ {t} \odot \mathcal {G} _ {t}. \tag {7}
$$
Furthermore, the error parts of  $\mathbf{w}_t$  and  $\mathbf{v}_t$  are denoted by  $\mathbf{w}_t\odot \mathbf{1}_{S_1^c}$  and  $\mathbf{v}_t\odot \mathbf{1}_{S_1^c}$ , which include both weak signal parts and pure error parts. In addition, strong signal parts of  $\mathbf{w}_t$  and  $\mathbf{v}_t$  are denoted by  $\mathbf{w}_t\odot \mathbf{1}_{S_1}$  and  $\mathbf{v}_t\odot \mathbf{1}_{S_1}$ , respectively.
We examine the dynamic changes of error components and strong signal components in two stages. Without loss of generality, we focus on analyzing entries  $i \in S_1$  where  $\beta_i^* > 0$ . The analysis for the case where  $\beta_i^* < 0$  and  $i \in S_1$  is similar and therefore not presented here. Specifically, within **Stage One**, we can ensure that with a high probability, for  $t \geq T_0 = 2\log(m / \alpha^2) / (\eta m)$ , the following
(c) False Positive Result
(d) True Negative Result
Figure 9: Performance on complex signal structure under  $[-1, 1]$  uniform distribution. The boxplots are depicted based on 30 runs.
results hold under Assumptions 1 and 2:
$$
\text {s t r o n g s i g n a l d y n a m i c s :} \quad \beta_ {t, i} \geq \min  \left\{\frac {\beta_ {i} ^ {*}}{2}, \left(1 + \frac {\eta \beta_ {i} ^ {*}}{\gamma n}\right) ^ {t} \alpha^ {2} - \alpha^ {2} \right\} f o r i \in S _ {1}, \beta_ {i} ^ {*} > 0,
$$
$$
\max  \left\{\| \mathbf {w} _ {t} \odot \mathbf {1} _ {S _ {1}} \| _ {\infty}, \| \mathbf {v} _ {t} \odot \mathbf {1} _ {S _ {1}} \| _ {\infty} \right\} \leq \alpha^ {2} f o r i \in S _ {1},
$$
$$
\text {e r r o r c o m p o n e n t d y n a m i c s :} \quad \max  \left\{\| \mathbf {w} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} \| _ {\infty}, \| \mathbf {v} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} \| _ {\infty} \right\} \leq \sqrt {\alpha} f o r i \in S _ {1} ^ {c}. \tag {8}
$$
From (8), we can observe that for  $t \geq T_0$ , the iterate  $(\mathbf{w}_t, \mathbf{v}_t)$  reaches the desired performance level. Specifically, each component  $\beta_{t,i}$  of the positive strong signal part  $\beta_t \odot \mathbf{1}_{S_1}$  increases exponentially in  $t$  until it reaches  $\beta_i^* / 2$ . Meanwhile, the weak signal and error part  $\beta_t \odot \mathbf{1}_{S_1^c}$  remains bounded by  $\mathcal{O}(\alpha)$ . This observation highlights the significance of small initialization size  $\alpha$  for the gradient descent algorithm, as it restricts the error term. A smaller initialization leads to better estimation but with the trade-off of a slower convergence rate.
After each component  $\beta_{t,i}$  of the strong signal reaches  $\beta_i^* /2$ , Stage Two starts. In this stage,  $\beta_{t,i}$  continues to grow at a slower rate and converges to the true parameter  $\beta_i^*$ . After this time, after  $3\log (m / \alpha^{2}) / (\eta m)$  iterations,  $\beta_{t,i}$  of the strong signal enters a desired interval, which is within a distance of  $C_{\epsilon}\sqrt{\log p / n}$  from the true parameter  $\beta_i^*$ . Simultaneously, the error term  $\beta_t\odot 1_{S_i^c}$  remains bounded by  $\mathcal{O}(\alpha)$  until  $\mathcal{O}(\log (1 / \alpha) / (\eta \log n))$  iterations.
We summarize the dynamic analysis results described above in Proposition 1 and Proposition 2. By combining the results from these two propositions, we can obtain the proof of Theorem 2 in the subsequent subsection.
(a) Signal 3
(b) Signal 4
Figure 10: Sensitivity analysis of smoothness parameter  $\gamma$ . The boxplots are depicted based on 30 runs.
Proof. We first utilize Proposition 1 to upper bound the error components  $\beta_{t} \odot \mathbf{1}_{S_{\mathrm{i}}^{\mathrm{c}}}$ . By Proposition 1, we are able to control the error parts of the same order as the initialization size  $\alpha$  within the time interval  $0 \leq t \leq T^{*} = \log(1 / \alpha) / (4\sigma\eta\log n)$ . Thus, by direct computation, we have
$$
\begin{array}{l} \left\| \boldsymbol {\beta} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} - \boldsymbol {\beta} ^ {*} \odot \mathbf {1} _ {S _ {1} ^ {c}} \right\| ^ {2} = \left\| \boldsymbol {\beta} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} + \left(\boldsymbol {\beta} _ {t} \odot \mathbf {1} _ {S _ {2}} - \boldsymbol {\beta} ^ {*} \odot \mathbf {1} _ {S _ {2}}\right) \right\| ^ {2} \\ \stackrel {(i)} {\leq} \left\| \boldsymbol {\beta} ^ {*} \odot \mathbf {1} _ {S _ {2}} \right\| ^ {2} + p \| \mathbf {w} _ {t} \odot \mathbf {w} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} - \mathbf {v} _ {t} \odot \mathbf {v} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} \| _ {\infty} ^ {2} \tag {9} \\ \stackrel {(i i)} {\leq} C _ {w} ^ {2} \cdot \frac {s _ {2} \log p}{n} + 2 C _ {\alpha} ^ {4} \cdot \frac {1}{p}, \\ \end{array}
$$
where  $(i)$  is based on the relationship between  $\ell_2$  norm and  $\ell_{\infty}$  norm and  $(ii)$  follows from the results in Proposition 1. As for the strong signal parts, by Proposition 2, that with probability at least  $1 - c_{1}n^{-1} - c_{2}p^{-1}$ , we obtain
$$
\left\| \boldsymbol {\beta} _ {t} \odot \mathbf {1} _ {S _ {1}} - \boldsymbol {\beta} ^ {*} \odot \mathbf {1} _ {S _ {1}} \right\| ^ {2} \leq s _ {1} \left\| \boldsymbol {\beta} _ {t} \odot \mathbf {1} _ {S _ {1}} - \boldsymbol {\beta} ^ {*} \odot \mathbf {1} _ {S _ {1}} \right\| _ {\infty} ^ {2} \leq C _ {\epsilon} ^ {2} \cdot \frac {s _ {1} \log p}{n}. \tag {10}
$$
Finally, combining (9) and (10), with probability at least  $1 - c_{1}n^{-1} - c_{2}p^{-1}$ , for any  $t$  that belongs to the interval
$$
[ 5 \log (m / \alpha^ {2}) / (\eta m), \log (1 / \alpha) / (4 \sigma \eta \log n) ],
$$
$$
\begin{array}{l} \| \pmb {\beta} _ {T _ {1}} - \pmb {\beta} ^ {*} \| ^ {2} = \| \pmb {\beta} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} - \pmb {\beta} ^ {*} \odot \mathbf {1} _ {S _ {1} ^ {c}} \| ^ {2} + \| \pmb {\beta} _ {t} \odot \mathbf {1} _ {S _ {1}} - \pmb {\beta} ^ {*} \odot \mathbf {1} _ {S _ {1}} \| ^ {2} \\ \leq C _ {\epsilon} ^ {2} \cdot \frac {s _ {1} \log p}{n} + C _ {w} ^ {2} \cdot \frac {s _ {2} \log p}{n} + 2 C _ {\alpha} ^ {4} \cdot \frac {1}{p}. \\ \end{array}
$$
Since  $p$  is much larger than  $n$ , the last term,  $2C_{\alpha}^{4} / p$ , is negligible. Considering the constants  $C_w$ ,  $C_{\alpha}$  and  $C_{\epsilon}$ , we finally obtain the error bound of gradient descent estimator in terms of  $\ell_2$  norm. Therefore, we conclude the proof of Theorem 2.
E Proof of Propositions 1 and 2
In this section, we provide the proof for the two propositions mentioned in Section 3. First, we introduce some useful lemmas, which are about the coherence of the design matrix  $\mathbf{X}$  and the upper bound of sub-Gaussian random variables.
(a) Model 1: Estimation Error
(b) Model 1: Non-normalized Error
(c) Model 1: Prediction Result
(d) Model 2: Estimation Error
(e) Model 2: Non-normalized Error
(f) Model 2: Prediction Result
Figure 11: Estimation error and prediction performance on Model 1 and Model 2. The boxplots are depicted based on 30 runs. The non-normalized error is calculated by  $\| \beta_t - \beta^*\| /\| \beta^*\|$
E.1 Technical Lemmas
Lemma 1. (General Hoeffding's inequality) Suppose that  $\theta_1, \dots, \theta_m$  are independent mean-zero sub-Gaussian random variables and  $\pmb{\mu} = (\mu_1, \dots, \mu_m) \in \mathbb{R}^m$ . Then, for every  $t > 0$ , we have
$$
\mathbb {P} \left[ \left| \sum_ {i = 1} ^ {m} \mu_ {i} \theta_ {i} \right| \geq t \right] \leq 2 \exp \left(- \frac {c t ^ {2}}{d ^ {2} \| \boldsymbol {\mu} \| ^ {2}}\right),
$$
where  $d = \max_{i}\| \theta_{i}\|_{\psi_{2}}$  and  $c$  is an absolute constant.
Proof. General Hoeffding's inequality can be proofed directly by the independence of  $\theta_{1},\dots ,\theta_{m}$  and the sub-Gaussian property. The detailed proof is omitted here.
Lemma 2. Suppose that  $\mathbf{X} / \sqrt{n}$  is a  $n\times p$ $\ell_2$ -normalized matrix satisfying  $\delta$ -incoherence; that is  $1 / n|\langle \mathbf{x}_i\odot \mathbf{1}_K,\mathbf{x}_i\odot \mathbf{1}_K\rangle |\leq \delta ,i\neq j$  for any  $K\subseteq [n]$ . Then, for s-sparse vector  $\mathbf{z}\in \mathbb{R}^p$ , we have
$$
\left\| \left(\frac {1}{n} \mathbf {X} _ {K} ^ {T} \mathbf {X} _ {K} - \mathbf {I}\right) \mathbf {z} \right\| _ {\infty} \leq \delta s \| \mathbf {z} \| _ {\infty},
$$
where  $\mathbf{X}_K$  denotes the  $n\times p$  matrix whose  $i$ -th column is  $\mathbf{x}_i$  if  $i\in K$  and  $\mathbf{0}_{p\times 1}$  otherwise.
Proof. The proof of Lemma 2 is similar to Lemma 2 in [23]. According to the condition
$$
\frac {1}{n} | \langle \mathbf {x} _ {i} \odot \mathbf {1} _ {K}, \mathbf {x} _ {i} \odot \mathbf {1} _ {K} \rangle | \leq \delta , i \neq j, K \subseteq [ n ],
$$
we can verify that for any  $i\in [p]$
$$
\left| \left(\frac {1}{n} \mathbf {X} _ {K} ^ {T} \mathbf {X} _ {K} \mathbf {z}\right) _ {i} - z _ {i} \right| \leq \delta s \| \mathbf {z} \| _ {\infty}.
$$
$$
\left\| \left(\frac {1}{n} \mathbf {X} _ {K} ^ {T} \mathbf {X} _ {K} - \mathbf {I}\right) \mathbf {z} \right\| _ {\infty} \leq \delta s \| \mathbf {z} \| _ {\infty}.
$$
E.2 Proof of Proposition 1
Proof. Here we prove Proposition 1 by induction hypothesis. It holds that our initializations  $\mathbf{w}_0 = \alpha \mathbf{1}_{p\times 1}$  and  $\mathbf{v}_0 = \alpha \mathbf{1}_{p\times 1}$  satisfy our conclusion given in Proposition 1. As we initialize  $w_{0,i}$  and  $v_{0,i}$  for any fixed  $i\in S_1^c$  with
$$
\left| w _ {0, i} \right| = \alpha \leq \sqrt {\alpha} \quad a n d \quad \left| v _ {0, i} \right| = \alpha \leq \sqrt {\alpha},
$$
Proposition 1 holds when  $t = 0$ . In the following, we show that for any  $t^*$  with  $0 \leq t^* \leq T^* = \log(1 / \alpha) / (4\sigma \eta \log n)$ , if the conclusion of Proposition 1 stands for all  $t$  with  $0 \leq t < t^*$ , then it also stands at the  $(t^* + 1)$ -th step. From the gradient descent updating rule given in (7), the updates of the weak signals and errors  $w_{t,i}, v_{t,i}$  for any fixed  $i \in S_1^c$  are obtained as follows,
$$
w _ {t + 1, i} = w _ {t, i} (1 + 2 \eta G _ {t, i}) \quad a n d \quad v _ {t + 1, i} = v _ {t, i} (1 - 2 \eta G _ {t, i}).
$$
Recall that  $|\mu_{t,k}|\leq 1$  for  $k = 1,\ldots ,n$ , for any fixed  $i\in S_1^c$ , then we have
$$
\left| G _ {t, i} \right| = \left| \frac {1}{n} \sum_ {k = 1} ^ {n} \mu_ {t, k} y _ {k} x _ {k i} \right| \leq \frac {1}{n} \sum_ {k = 1} ^ {n} \left| x _ {k i} \right|.
$$
For ease of notation, we denote  $M_{i} = n^{-1}\sum_{k = 1}^{n}|x_{ki}|$  and  $M = \max_{i\in S_1^c}M_i$ . Then we can easily bound the weak signals and errors as follows,
$$
\left\| \mathbf {w} _ {t + 1} \odot \mathbf {1} _ {S _ {1} ^ {c}} \right\| _ {\infty} \leq (1 + 2 \eta M) \left\| \mathbf {w} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} \right\| _ {\infty} \quad a n d \quad \left\| \mathbf {v} _ {t + 1} \odot \mathbf {1} _ {S _ {1} ^ {c}} \right\| _ {\infty} \leq (1 + 2 \eta M) \left\| \mathbf {v} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} \right\| _ {\infty}. \tag {11}
$$
By General Hoeffding's inequality of sub-Gaussian random variables, with probability at least  $1 - cn^{-n}$ , we obtain  $M_{i} \leq \sigma \sqrt{\log n}, i \in S_{1}^{c}$ , where  $c$  is a constant. Since  $p$  is much larger than  $n$ , the inequality  $(1 - cn^{-n})^p > 1 - cn^{-1}$  holds. Then, with probability at least  $1 - cn^{-1}$ , where  $c$  is a constant, we have  $M \leq \sigma \sqrt{\log n}$ .
Combined (11) and the bound of  $M$ , we then have
$$
\begin{array}{l} \left\| \mathbf {w} _ {t ^ {*} + 1} \odot \mathbf {1} _ {S _ {1} ^ {c}} \right\| _ {\infty} \leq \left(1 + 2 \eta M\right) ^ {t ^ {*} + 1} \left\| \mathbf {w} _ {0} \odot \mathbf {1} _ {S _ {1} ^ {c}} \right\| _ {\infty} \\ = \exp ((t ^ {*} + 1) \log (1 + 2 \eta M)) \alpha \\ \stackrel {(i)} {\leq} \exp (T ^ {*} \log (1 + 2 \eta M)) \alpha \\ \stackrel {(i i)} {\leq} \exp (T ^ {*} \cdot 2 \eta M) \alpha \\ \begin{array}{l} \stackrel {(i i i)} {\leq} \exp ((1 / 2) \log (1 / \alpha)) \alpha = \sqrt {\alpha}, \end{array} \\ \end{array}
$$
where  $(i)$  and  $(ii)$  follow from  $t^* + 1 \leq T^*$  and  $\log(1 + x) < x$  when  $x > 0$ , respectively. Moreover,  $(iii)$  is based on the definition of  $T^*$ . Similarly, we can prove that  $\|v_{t^*} \odot \mathbf{1}_{S_i^\alpha}\|_\infty \leq \sqrt{\alpha}$ . Thus, the induction hypothesis also holds for  $t^* + 1$ . Since  $t^* < T^*$  is arbitrarily chosen, we complete the proof of Proposition 1.
E.3 Proof of Proposition 2
Proof. In order to analyze strong signals:  $\beta_{t,i} = w_{t,i}^2 - v_{t,i}^2$  for any fixed  $i \in S_1$ , we focus on the dynamics  $w_{t,i}^2$  and  $v_{t,i}^2$ . Following the updating rule (7), we have
$$
\begin{array}{l} w _ {t + 1, i} ^ {2} = w _ {t, i} ^ {2} + 4 \eta w _ {t, i} ^ {2} G _ {t, i} + 4 \eta^ {2} w _ {t, i} ^ {2} G _ {t, i} ^ {2}, \tag {12} \\ v _ {t + 1, i} ^ {2} = v _ {t, i} ^ {2} - 4 \eta v _ {t, i} ^ {2} G _ {t, i} + 4 \eta^ {2} v _ {t, i} ^ {2} G _ {t, i} ^ {2}. \\ \end{array}
$$
Without loss of generality, here we just analyze entries  $i \in S_1$  with  $\beta_i^* > 0$ . Analysis for the negative case with  $\beta_i^* < 0$ ,  $i \in S_1$  is almost the same and is thus omitted. We divide our analysis into two stages, as discussed in Appendix C.
Specifically, in **Stage One**, since we choose the initialization as  $\mathbf{w}_0 = \mathbf{v}_0 = \alpha \mathbf{1}_{p\times 1}$ , we obtain  $\beta_{0,i} = 0$  for any fixed  $i\in S_1$ , we show after  $2\log (\beta_i^* /\alpha^2) / \eta \beta_i^*$  iterations, the gradient descent coordinate  $\beta_{t,i}$  will exceed  $\beta_i^* /2$ . Therefore, all components of strong signal part will exceed half of the true parameters after  $2\log (m / \alpha^{2}) / (\eta m)$  iterations. Furthermore, we calculate the number of iterations required to achieve  $\beta_{t,i}\geq \beta_i^* -\epsilon$ , where  $\epsilon = C_{\epsilon}\sqrt{\log p / n}$ , in **Stage Two**. Thus, we conclude the proof of Proposition 2.
Stage One. First, we introduce some new notations. In  $t$ -th iteration, according to the values of  $\mu_{t,i}$ , we divide the set  $[n]$  into three parts,  $K_{t,1} = \{i \in [n] : \mu_{t,i} = 1\}$ ,  $K_{t,2} = \{i \in [n] : 0 < \mu_{t,i} < 1\}$  and  $K_{t,3} = \{i \in [n] : \mu_{t,i} = 0\}$ , with cardinalities of  $k_{t,1}$ ,  $k_{t,2}$  and  $k_{t,3}$ , respectively. From [28, 45], most elements of  $\mu_t$  are 0 or 1 and the proportion of these elements will rapidly increase with the decreasing of  $\gamma$ , which means that  $k_{t,2} \ll (k_{t,1} + k_{t,3})$  controlling  $\gamma$  in a small level.
According to the updating formula of  $w_{t+1,i}^2$  in (12), we define the following form of element-wise bound  $\xi_{t,i}$  for any fixed  $i \in S_1$  as
$$
\begin{array}{l} \xi_ {t, i} := 1 - \frac {w _ {t , i} ^ {2}}{w _ {t + 1 , i} ^ {2}} \left(1 - \frac {4 \eta}{\gamma n} (\beta_ {t, i} - \beta_ {i} ^ {*})\right) \\ = \frac {1}{w _ {t + 1 , i} ^ {2}} \left(w _ {t + 1, i} ^ {2} - w _ {t, i} ^ {2} + \frac {4 \eta}{\gamma n} w _ {t, i} ^ {2} \left(\beta_ {t, i} - \beta_ {i} ^ {*}\right)\right) \tag {13} \\ = \frac {w _ {t , i} ^ {2}}{w _ {t + 1 , i} ^ {2}} \left(4 \eta \left(G _ {t, i} + \frac {1}{\gamma n} (\beta_ {t, i} - \beta_ {i} ^ {*})\right) + 4 \eta^ {2} G _ {t, i} ^ {2}\right). \\ \end{array}
$$
Rewriting (13), we can easily get that
$$
w _ {t + 1, i} ^ {2} = w _ {t, i} ^ {2} \left(1 - \frac {4 \eta}{\gamma n} \left(\beta_ {t, i} - \beta_ {i} ^ {*}\right)\right) + \xi_ {t, i} w _ {t + 1, i} ^ {2}. \tag {14}
$$
From (14), it is obvious that if the magnitude of  $\xi_{t,i}$  is sufficiently small, we can easily conclude that  $w_{t + 1,i}^2\geq w_{t,i}^2$ . Therefore, our goal is to evaluate the magnitude of  $\xi_{t,i}$  in (13) for any fixed  $i\in S_1$ . First, we focus on the term  $G_{t,i} + (\beta_{t,i} - \beta_i^*) / (\gamma n)$  in (13). In particular, recall the definition of  $G_{t,i}$  and we expand  $G_{t,i} + (\beta_{t,i} - \beta_i^*) / (\gamma n)$  in the following form
$$
\begin{array}{l} G _ {t, i} + \frac {1}{\gamma n} \left(\beta_ {t, i} - \beta_ {i} ^ {*}\right) = \frac {1}{n} \left(\sum_ {k \in K _ {t, 1}} y _ {k} x _ {k i} + \sum_ {k \in K _ {t, 2}} y _ {k} x _ {k i} \frac {1 - y _ {k} \mathbf {x} _ {k} ^ {T} \boldsymbol {\beta} _ {t}}{n \gamma}\right) + \frac {1}{\gamma n} \left(\beta_ {t, i} - \beta_ {i} ^ {*}\right) \\ = \frac {1}{n} \left(\sum_ {k \in K _ {t, 1}} y _ {k} x _ {k i} + \sum_ {k \in K _ {t, 2}} \frac {y _ {k} x _ {k i}}{n \gamma}\right) - \frac {1}{\gamma n ^ {2}} \left(\sum_ {k \in K _ {t, 2}} x _ {k, i} \mathbf {x} _ {k} ^ {T} \boldsymbol {\beta} _ {t} - n \left(\beta_ {t, i} - \beta_ {i} ^ {*}\right)\right) \\ = \frac {1}{n} \left(\sum_ {k \in K _ {t, 1}} y _ {k} x _ {k i} + \sum_ {k \in K _ {t, 2}} \frac {\left(y _ {k} - \mathbf {x} _ {k} ^ {T} \boldsymbol {\beta} ^ {*}\right) x _ {k i}}{n \gamma}\right) \\ - \frac {1}{\gamma n} \left(\frac {1}{n} \sum_ {k \in K _ {t, 2}} x _ {k, i} \mathbf {x} _ {k} ^ {T} \left(\boldsymbol {\beta} _ {t} - \boldsymbol {\beta} ^ {*}\right) - \left(\beta_ {t, i} - \beta_ {i} ^ {*}\right)\right) \\ \hat {=} (I _ {1}) + (I _ {2}). \\ \end{array}
$$
For the term  $(I_1)$ , by the condition  $k_{t,2} \ll k_{t,1}$  and General Hoeffding's inequality, we have
$$
\frac {1}{n} \left| \sum_ {k \in K _ {t, 1}} y _ {k} x _ {k i} + \sum_ {k \in K _ {t, 2}} \frac {\left(y _ {k} - \mathbf {x} _ {k} ^ {T} \boldsymbol {\beta} ^ {*}\right) x _ {k i}}{n \gamma} \right| \leq \frac {1}{n} \sigma \sqrt {n \log p} = \sigma \sqrt {\frac {\log p}{n}}, \tag {15}
$$
holds with probability at least  $1 - cp^{-1}$ , where  $c$  is a constant.
For the term  $(I_2)$ , let  $\mathbf{X}_{t,2} \in \mathbb{R}^{n \times p}$  denote the matrix whose  $k$ -th column is  $\mathbf{x}_k$  if  $k \in K_{t,2}$ . Then, we approximate  $(I_2)$  based on the assumption on the design matrix  $\mathbf{X}$  via Lemma 2,
$$
\left\| \frac {1}{n} \mathbf {X} _ {t, 2} ^ {T} \mathbf {X} _ {t, 2} \left(\boldsymbol {\beta} _ {t} - \boldsymbol {\beta} ^ {*}\right) - \left(\boldsymbol {\beta} _ {t} \odot \mathbf {1} _ {S _ {1}} - \boldsymbol {\beta} ^ {*} \odot \mathbf {1} _ {S _ {1}}\right) \right\| _ {\infty} \lesssim \delta s \kappa m. \tag {16}
$$
By (15), (16), condition of the minimal strength  $m \geq \sigma \log p \sqrt{\log p / n}$  and condition  $\delta \lesssim 1 / (\kappa s \log p)$ , we have
$$
\left| G _ {t, i} + \frac {1}{\gamma n} \left(\beta_ {t, i} - \beta_ {i} ^ {*}\right) \right| \lesssim \frac {m}{\gamma n \log p}. \tag {17}
$$
Then, we can bound  $G_{t,i}$  through
$$
\begin{array}{l} \left| G _ {t, i} \right| \leq \left| G _ {t, i} + 1 / (\gamma n) \left(\beta_ {t, i} - \beta_ {i} ^ {*}\right) \right| + \left| 1 / (\gamma n) \left(\beta_ {t, i} - \beta_ {i} ^ {*}\right) \right| \\ \lesssim \frac {1}{\gamma n} \left(\frac {m}{\log p} + \kappa m\right). \tag {18} \\ \end{array}
$$
Note that for any fixed  $i \in S_1$ ,  $G_{t,i} \leq n^{-1} \sum_{k=1}^{n} |x_{ki}|$ . By General Hoeffding's inequality, we have with probability at least  $1 - c_1 n^{-1}$ ,  $|G_{t,i}| \lesssim \log n$  and then  $|\eta G_{t,i}| < 1$  based on the assumption of  $\eta$ . Therefore,  $w_{t,i}^2 / w_{t+1,i}^2$  is always bounded by some constant greater than 1.
Recalling the definition of element-wise bound  $\xi_{t,i}$ , and combining (17) and (18), we have that
$$
\left| \xi_ {t, i} \right| \stackrel {(i)} {\lesssim} \frac {1}{\gamma n} \left(\frac {\eta m}{\log p} + \eta^ {2} \kappa^ {2} m ^ {2}\right) \stackrel {(i i)} {\lesssim} \frac {\eta m}{\gamma n \log p}, \tag {19}
$$
where we use the bound of  $w_{t,i}^2 / w_{t+1,i}^2$  for any fixed  $i \in S_1$  in step(i) and step(ii) is based on  $\eta \kappa m \leq m / \log p$  and  $\kappa m \lesssim 1$ . Thus, we conclude that  $\xi_{t,i}$  is sufficiently small for any fixed  $i \in S_1$ .
Now, for any fixed  $i \in S_1$ , when  $0 \leq \beta_{t,i} \leq \beta_i^* / 2$ , we can get the increment of  $w_{t,i}^2$  for any fixed  $i \in S_1$  according to (19),
$$
\begin{array}{l} w _ {t + 1, i} ^ {2} = w _ {t, i} ^ {2} \left(1 - \frac {4 \eta}{\gamma n} \left(\beta_ {t, i} - \beta_ {i} ^ {*}\right)\right) / \left(1 + c \frac {\eta m}{\gamma n \log p}\right) \\ \stackrel {(i)} {\geq} w _ {t, i} ^ {2} \left(1 + \frac {2 \eta \beta_ {i} ^ {*}}{\gamma n}\right) / \left(1 + c \frac {\eta m}{\gamma n \log p}\right) \\ \stackrel {(i i)} {\geq} w _ {t, i} ^ {2} \left(1 + \frac {\eta \beta_ {i} ^ {*}}{\gamma n}\right), \\ \end{array}
$$
where  $(i)$  follows from  $0\leq \beta_{t,i}\leq \beta_i^* /2$  and  $(ii)$  holds since  $m / \log p\lesssim \beta_i^*$ . Similarly, we can analyze  $v_{t,i}^{2}$  to get that when  $0\leq \beta_{t,i}\leq \beta_i^* /2$
$$
v _ {t + 1, i} ^ {2} \leq v _ {t, i} ^ {2} \left(1 - \eta \beta_ {i} ^ {*} / \gamma n\right).
$$