SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.4. Let us define a  $k$ -sparse vector pair domain to be  $\Omega_k \coloneqq \{(x, y) \in \mathbb{R}^p \times \mathbb{R}^p : \| x \|_0 \leq k, \| y \|_0 \leq k, \| x - y \|_0 \leq k\}$ . Any  $M_1$  satisfying  $f(y) \leq f(x) + \nabla f(x)^T (y - x) + \frac{M_1}{2} \| y - x \|_2^2$  for all  $(x, y) \in \Omega_1$  is called a restricted smooth parameter with support size 1, and any  $m_{2k}$  satisfying
$f(\pmb{y}) \geq f(\pmb{x}) + \nabla f(\pmb{x})^T (\pmb{y} - \pmb{x}) + \frac{m_{2k}}{2} \| \pmb{y} - \pmb{x} \|_2^2$  for all  $(\pmb{x}, \pmb{y}) \in \Omega_{2k}$  is called a restricted strongly convex parameter with support size  $2k$ . If  $\hat{\beta}$  is our heuristic solution by the beam-search method, and  $\beta^*$  is the optimal solution, then:
$$
\mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {*}\right) \leq \mathcal {L} _ {\text {r i d g e}} (\hat {\boldsymbol {\beta}}) \leq \left(1 - e ^ {- m _ {2 k} / M _ {1}}\right) \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {*}\right). \tag {51}
$$
Proof. The left-hand-side inequality  $\mathcal{L}_{\mathrm{ridge}}(\beta^{*}) \leq \mathcal{L}_{\mathrm{ridge}}(\hat{\beta})$  is obvious because the optimal loss should always be less than or equal to the loss found by our heuristic method. Therefore, we focus on the right-hand-side inequality  $\mathcal{L}_{\mathrm{ridge}}(\hat{\beta}) \leq (1 - e^{-m_{2k} / M_1})\mathcal{L}_{\mathrm{ridge}}(\beta^{*})$ .
We first give an overview of our proof strategy, which is similar to the proof in [29], but our decay factor  $m_{2k} / M_1$  in the exponent is much larger than theirs, which is  $m_{2k} / M_{2k}$  (we can obtain the definition of  $M_{2k}$  by extending the definition of  $M_1$  to the support size  $2k$ ). Let  $\mathcal{L}_{\mathrm{ridge}}(\beta^t)$  and  $\mathcal{L}_{\mathrm{ridge}}(\beta^{t + 1})$  be the losses at the  $t$ -th and  $(t + 1)$ -th iterations of our beam search algorithm. We first derive a lower bound formula on the decrease of the loss function  $\mathcal{L}_{\mathrm{ridge}}(\cdot)$  between two successive iterations, i.e.,  $\mathcal{L}_{\mathrm{ridge}}(\beta^t) - \mathcal{L}_{\mathrm{ridge}}(\beta^{t + 1})$ . We will then show that this quantity of decrease can be expressed in terms of the parameters  $M_1$  and  $m_{2k}$  defined in the statement of the theorem. Finally, we use the relationship between the decrease of the loss and  $M_1$  and  $m_{2k}$  to derive the right-hand inequality in Equation (51).
Step 1: Decrease of Loss between Two Successive Iterations:
For an arbitrary coordinate  $j$  that is not part of the support of  $\beta^t$ , or more explicitly  $\beta_j^t = 0$ , if we change the coefficient on the  $j$ -th coordinate  $\beta^t$  from 0 to  $\alpha$ , the loss becomes:
$$
= \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {\beta} ^ {t}) - \frac {1}{4} \frac {\left(\nabla_ {j} \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {\beta} ^ {t})\right) ^ {2}}{\| \boldsymbol {X} _ {: j} \| _ {2} ^ {2} + \lambda_ {2}} + \left(\| \boldsymbol {X} _ {: j} \| _ {2} ^ {2} + \lambda_ {2}\right) \left(\alpha + \frac {1}{2} \frac {\nabla_ {j} \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {\beta} ^ {t})}{\| \boldsymbol {X} _ {: j} \| _ {2} ^ {2} + \lambda_ {2}}\right) ^ {2}
$$
The first two terms are constant, and the third term is a quadratic function of  $\alpha$ .
The minimum loss we can achieve for  $\mathcal{L}_{\mathrm{ridge}}(\beta^t + \alpha e_j)$  by optimizing on the  $j$ -th coefficient is:
$$
\min _ {\boldsymbol {\alpha}} \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {t} + \alpha \boldsymbol {e} _ {j}) = \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {t}) - \frac {1}{4} \frac {(\nabla_ {j} \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {t})) ^ {2}}{\| \boldsymbol {X} _ {: j} \| _ {2} ^ {2} + \lambda_ {2}},
$$
and the maximum decrease of our ridge regression loss by optimizing a single coefficient is then:
$$
\max _ {\alpha} \left(\mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) - \mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {t} + \alpha \boldsymbol {e} _ {j}\right)\right) = \mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) - \min _ {\alpha} \mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {t} + \alpha \boldsymbol {e} _ {j}\right) = \frac {1}{4} \frac {\left(\nabla_ {j} \mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right)\right) ^ {2}}{\left\| \boldsymbol {X} _ {: j} \right\| _ {2} ^ {2} + \lambda_ {2}}, \tag {52}
$$
where  $X_{:j}$  is the  $j$ -th column of the design matrix  $X$ .
Now, for our beam-search method, the decrease of loss between two successive iterations will be greater than the decrease of loss by optimizing on a single coordinate (because we are fine-tuning the coefficients on the newly expanded support and picking the best expanded support among all support candidates in the beam search), we have
$$
\mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) - \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t + 1}\right) \geq \max  _ {j} \max  _ {\alpha} \left(\mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) - \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t} + \alpha \boldsymbol {e} _ {j}\right)\right) = \max  _ {j} \frac {1}{4} \frac {\left(\nabla_ {j} \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right)\right) ^ {2}}{\left\| \boldsymbol {X} _ {: j} \right\| _ {2} ^ {2} + \lambda_ {2}}. \tag {53}
$$
For an arbitrary set of coordinates  $\mathcal{J}'$ , we have the following inequality:
$$
\max  _ {j} \frac {1}{4} \frac {\left(\nabla_ {j} \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right)\right) ^ {2}}{\| \boldsymbol {X} _ {: j} \| _ {2} ^ {2} + \lambda_ {2}} \geq \frac {1}{| \mathcal {I} ^ {\prime} |} \sum_ {j ^ {\prime} \in \mathcal {I} ^ {\prime}} \frac {1}{4} \frac {\left(\nabla_ {j ^ {\prime}} \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right)\right) ^ {2}}{\| \boldsymbol {X} _ {: j ^ {\prime}} \| _ {2} ^ {2} + \lambda_ {2}}. \tag {54}
$$
Inequality 54 holds because the largest decrease of loss among all possible coordinates is greater than or equal to the average decrease of loss among any arbitrary set of coordinates  $\mathcal{J}'$ .
If we choose  $\mathcal{J}' = \mathcal{S}^R \coloneqq \mathcal{S}^* \setminus \text{supp}(\beta^t)$ , where  $\mathcal{S}^*$  is the set of coordinates for the optimal solution  $\beta^*$  and  $\text{supp}(\cdot)$  denotes the support of the solution, we have
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) - \mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {t + 1}\right) \geq \max  _ {j} \frac {1}{4} \frac {\left(\nabla_ {j} \mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right)\right) ^ {2}}{\| \boldsymbol {X} _ {: j} \| _ {2} ^ {2} + \lambda_ {2}} \\ \geq \frac {1}{| \mathcal {S} ^ {R} |} \sum_ {j ^ {\prime} \in \mathcal {S} ^ {R}} \frac {1}{4} \frac {\left(\nabla_ {j ^ {\prime}} \mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right)\right) ^ {2}}{\left\| \mathbf {X} _ {: j ^ {\prime}} \right\| _ {2} ^ {2} + \lambda_ {2}} \\ = \frac {1}{4 | \mathcal {S} ^ {R} |} \sum_ {j ^ {\prime} \in \mathcal {S} ^ {R}} \frac {(\nabla_ {j ^ {\prime}} \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {\beta} ^ {t})) ^ {2}}{\| \boldsymbol {X} _ {: j ^ {\prime}} \| _ {2} ^ {2} + \lambda_ {2}} \quad \# p u l l \frac {1}{4} t o t h e f r o n t \\ \geq \frac {1}{4 k} \sum_ {j ^ {\prime} \in \mathcal {S} ^ {R}} \frac {\left(\nabla_ {j ^ {\prime}} \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right)\right) ^ {2}}{\left\| \boldsymbol {X} _ {: j ^ {\prime}} \right\| _ {2} ^ {2} + \lambda_ {2}} \# b e c a u s e \mathcal {S} ^ {R} \subset \mathcal {S} ^ {*}, | \mathcal {S} ^ {R} | \leq | \mathcal {S} ^ {*} | = k \\ \geq \frac {1}{4 k} \frac {1}{\max _ {j ^ {\prime} \in \mathcal {S} ^ {R}} (\| \boldsymbol {X} _ {: j ^ {\prime}} \| _ {2} ^ {2} + \lambda_ {2})} \sum_ {j ^ {\prime} \in \mathcal {S} ^ {R}} (\nabla_ {j ^ {\prime}} \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {t})) ^ {2} \\ \end{array}
$$
take the maximum of the denominator and pull it to the front
$$
\Rightarrow \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) - \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t + 1}\right) \geq \frac {1}{4 k} \frac {1}{\max  _ {j ^ {\prime} \in \mathcal {S} ^ {R}} \left(\left\| \boldsymbol {X} _ {: j ^ {\prime}} \right\| _ {2} ^ {2} + \lambda_ {2}\right)} \sum_ {j ^ {\prime} \in \mathcal {S} ^ {R}} \left(\nabla_ {j ^ {\prime}} \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right)\right) ^ {2}. \tag {55}
$$
Inequality (55) is the lower bound of decrease of loss between two successive iterations of our beam-search algorithm.
In Step 2 and Step 3 below, we will provide bounds on  $\max_{j' \in S^R} (\|X_{:j'}\|_2^2 + \lambda_2)$  and  $\sum_{j' \in S^R} (\nabla_{j'} \mathcal{L}_{\mathrm{ridge}}(\beta^t))^2$  in terms of  $M_1$  and  $m_{2k}$ , respectively.
Step 2: Upper bound on  $\max_{j' \in S^R} (\| \mathbf{X}_{:j'} \|_2^2 + \lambda_2)$
We will show below that  $\frac{M_1}{2} \geq \max_{j' \in S^R} (\|X_{:j'}\|_2^2 + \lambda_2)$ .
According to the definition of the restricted smoothness of  $M_{1}$  (See Background Concepts in Appendix B), we have that
$$
\frac {M _ {1}}{2} \left\| \boldsymbol {w} _ {2} - \boldsymbol {w} _ {1} \right\| _ {2} ^ {2} \geq \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {w} _ {2}) - \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {w} _ {1}) - \nabla \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {w} _ {1}) ^ {T} (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) \tag {56}
$$
for  $\forall \pmb{w}_1, \pmb{w}_2 \in \mathbb{R}^p$  with  $\| \pmb{w}_1 \|_0 \leq 1$ ,  $\| \pmb{w}_2 \|_0 \leq 1$ , and  $\| \pmb{w}_2 - \pmb{w}_1 \|_0 \leq 1$ .
The right-hand side can be rewritten as
$$
\mathcal {L} _ {\text {r i d g e}} (\boldsymbol {w} _ {2}) - \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {w} _ {1}) - \nabla \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {w} _ {1}) ^ {T} (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1})
$$
$$
\begin{array}{l} = (\| \boldsymbol {X} \boldsymbol {w} _ {2} - \boldsymbol {y} \| _ {2} ^ {2} + \lambda_ {2} \| \boldsymbol {w} _ {2} \| _ {2} ^ {2}) - (\| \boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y} \| _ {2} ^ {2} + \lambda_ {2} \| \boldsymbol {w} _ {1} \| _ {2} ^ {2}) - 2 (\boldsymbol {X} ^ {T} (\boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y}) + \lambda_ {2} \boldsymbol {w} _ {1}) ^ {T} (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) \\ \quad \# w r i t e o u t \mathcal {L} _ {\text {r i d g e}} (\cdot) e x p l i c i t l y a n d s u b s t i t u t e \nabla \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {w} _ {1}) = \\ \quad 2 (\boldsymbol {X} ^ {T} (\boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y}) + \boldsymbol {w} _ {1}) a c c o r d i n g t o L e m m a B. 7 \end{array}
$$
$$
= \left(\left\| (\boldsymbol {X} \boldsymbol {w} _ {2} - \boldsymbol {X} \boldsymbol {w} _ {1}) + (\boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y}) \right\| _ {2} ^ {2} + \lambda_ {2} \left\| (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) + \boldsymbol {w} _ {1} \right\| _ {2} ^ {2}\right)
$$
$$
- \left(\| \boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y} \| _ {2} ^ {2} + \lambda_ {2} \| \boldsymbol {w} _ {1} \| _ {2} ^ {2}\right) - 2 \left(\boldsymbol {X} ^ {T} (\boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y}) + \lambda_ {2} \boldsymbol {w} _ {1}\right) ^ {T} (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1})
$$
We subtract and add identical terms inside the first parentheses
$$
\begin{array}{l} = \left(\| X w _ {2} - X w _ {1} \| _ {2} ^ {2} + \| X w _ {1} - y \| _ {2} ^ {2} + 2 \left(X w _ {2} - X w _ {1}\right) ^ {T} \left(X w _ {1} - y\right)\right) \\ + \lambda_ {2} \| \boldsymbol {w} _ {2} - \boldsymbol {w} _ {1} \| _ {2} ^ {2} + \lambda_ {2} \| \boldsymbol {w} _ {1} \| _ {2} ^ {2} + 2 \lambda_ {2} (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) ^ {T} \boldsymbol {w} _ {1} \\ - \left(\| \boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y} \| _ {2} ^ {2} + \lambda_ {2} \| \boldsymbol {w} _ {1} \| _ {2} ^ {2}\right) - 2 \left(\boldsymbol {X} ^ {T} (\boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y}) + \lambda_ {2} \boldsymbol {w} _ {1}\right) ^ {T} (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) \\ \end{array}
$$
expand the terms inside  $\| \cdot \|$  for the first line into two lines
$$
\begin{array}{l} = \left(\left\| \boldsymbol {X} \boldsymbol {w} _ {2} - \boldsymbol {X} \boldsymbol {w} _ {1} \right\| _ {2} ^ {2} + \left\| \boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y} \right\| _ {2} ^ {2} + 2 \left(\boldsymbol {X} ^ {T} (\boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y})\right) ^ {T} (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1})) \right. \\ + \lambda_ {2} \| \boldsymbol {w} _ {2} - \boldsymbol {w} _ {1} \| _ {2} ^ {2} + \lambda_ {2} \| \boldsymbol {w} _ {1} \| _ {2} ^ {2} + 2 (\lambda_ {2} \boldsymbol {w} _ {1}) ^ {T} (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) \\ \begin{array}{c} - \left(\| \boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y} \| _ {2} ^ {2} + \lambda_ {2} \| \boldsymbol {w} _ {1} \| _ {2} ^ {2}\right) - 2 \left(\boldsymbol {X} ^ {T} (\boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y}) + \lambda_ {2} \boldsymbol {w} _ {1}\right) ^ {T} (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) \\ \# r e w r i t e   \left(\boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {X} \boldsymbol {w} _ {2}\right) ^ {T} (\boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y}) = \left(\boldsymbol {X} ^ {T} (\boldsymbol {X} \boldsymbol {w} _ {1} - \boldsymbol {y})\right) ^ {T} (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) \end{array} \\ \end{array}
$$
$$
\begin{array}{l} = \| \boldsymbol {X} \boldsymbol {w} _ {2} - \boldsymbol {X} \boldsymbol {w} _ {1} \| _ {2} ^ {2} + \lambda_ {2} \| \boldsymbol {w} _ {2} - \boldsymbol {w} _ {1} \| _ {2} ^ {2} \\ = (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) ^ {T} (\lambda_ {2} \mathbf {I} + \boldsymbol {X} ^ {T} \boldsymbol {X}) (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) \\ \Rightarrow \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {w} _ {2}) - \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {w} _ {1}) - \nabla \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {w} _ {1}) ^ {T} (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) = (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}) ^ {T} (\lambda_ {2} \mathbf {I} + \boldsymbol {X} ^ {T} \boldsymbol {X}) (\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}). \\ \end{array}
$$
Therefore, together with Inequality (56), we have that
$$
\frac {M _ {1}}{2} \left\| \boldsymbol {w} _ {2} - \boldsymbol {w} _ {1} \right\| _ {2} ^ {2} \geq \left(\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}\right) ^ {T} \left(\lambda_ {2} \mathbf {I} + \boldsymbol {X} ^ {T} \boldsymbol {X}\right) \left(\boldsymbol {w} _ {2} - \boldsymbol {w} _ {1}\right) \tag {57}
$$
for any  $\pmb{w}_2 - \pmb{w}_1 \in \mathbb{R}^p$  and  $\| \pmb{w}_2 - \pmb{w}_1 \|_0 \leq 1$ . If we let  $\pmb{w}_2 - \pmb{w}_1 = \pmb{e}_j$  where  $\pmb{e}_j$  is a vector with 1 on the  $j$ -th entry and 0 otherwise, we have
$$
\frac {M _ {1}}{2} \geq \lambda_ {2} + \| \pmb {X} _ {: j} \| _ {2} ^ {2}
$$
for  $\forall j\in [1,\dots,p]$ . Thus, we can derive our upper bound at the beginning of Step 2:
$$
\frac {M _ {1}}{2} \geq \max  _ {j ^ {\prime} \in S ^ {R}} \left(\left\| \boldsymbol {X} _ {: j ^ {\prime}} \right\| _ {2} ^ {2} + \lambda_ {2}\right) \tag {58}
$$
Step 3: Lower bound on  $\sum_{j' \in S^R} (\nabla_{j'} \mathcal{L}_{\mathrm{ridge}}(\beta^t))^2$
We will show that  $\sum_{j' \in \mathcal{J}'} (\nabla_{j'} \mathcal{L}_{\mathrm{ridge}}(\beta^t))^2 \geq 2m_{2k} (\mathcal{L}_{\mathrm{ridge}}(\beta^t) - \mathcal{L}_{\mathrm{ridge}}(\beta^*))$ .
According to the definition of the restricted strong convexity parameter  $m_{2k}$ , we have:
$$
\nabla \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {t}) ^ {T} (\boldsymbol {\beta} ^ {*} - \boldsymbol {\beta} ^ {t}) + \frac {m _ {2 k}}{2} \| \boldsymbol {\beta} ^ {*} - \boldsymbol {\beta} ^ {t} \| _ {2} ^ {2} \leq \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {*}) - \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {t}),
$$
where  $\beta^t\in \mathbb{R}^p$  is our heuristic solution at the  $t$ -th iteration,  $\beta^{*}$  is the optimal  $k$ -sparse ridge regression solution, and  $\| \beta^t\| _0 = t\leq k < 2k$ ,  $\| \beta^{*}\|_{0} = k < 2k$ , and  $\| \beta^{*} - \beta^{t}\|_{0}\leq 2k$
By rearranging some terms, we have:
$$
\begin{array}{l} \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) - \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {*}\right) \\ \leq - \nabla \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) ^ {T} \left(\boldsymbol {\beta} ^ {*} - \boldsymbol {\beta} ^ {t}\right) - \frac {m _ {2 k}}{2} \| \boldsymbol {\beta} ^ {*} - \boldsymbol {\beta} ^ {t} \| _ {2} ^ {2} \\ \end{array}
$$
always holds because of restricted strong convexity
$$
= - \nabla \mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) ^ {T} \left(\boldsymbol {\beta} ^ {t} + \sum_ {j = 1} ^ {p} \alpha_ {j} \boldsymbol {e} _ {j} - \boldsymbol {\beta} ^ {t}\right) - \frac {m _ {2 k}}{2} \| \boldsymbol {\beta} ^ {t} + \sum_ {j = 1} ^ {p} \alpha_ {j} \boldsymbol {e} _ {j} - \boldsymbol {\beta} ^ {t} \| _ {2} ^ {2}
$$
$e_j$  is a vector with 1 on the  $j$ -th entry and 0 otherwise;  $\alpha_j$  is a fixed number with  $\alpha_j = \beta_j^* - \beta_j^t$
$$
\begin{array}{l} = - \nabla \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) ^ {T} \left(\sum_ {j = 1} ^ {p} \alpha_ {j} \boldsymbol {e} _ {j}\right) - \frac {m _ {2 k}}{2} \| \sum_ {j = 1} ^ {p} \alpha_ {j} \boldsymbol {e} _ {j} \| _ {2} ^ {2} \quad \# \alpha_ {j} \text {i s a f i x e d n u m b e r w i t h} \alpha_ {j} = \beta_ {j} ^ {*} - \beta_ {j} ^ {t} \\ = - \sum_ {j = 1} ^ {p} \left(\nabla_ {j} \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) \alpha_ {j} - \frac {m _ {2 k}}{2} \alpha_ {j} ^ {2}\right) \quad \# \text {C o m b i n e i n t o o n e s u m m a t i o n} \\ = - \sum_ {j \in S ^ {R}} \left(\nabla_ {j} \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {t}) \alpha_ {j} - \frac {m _ {2 k}}{2} \alpha_ {j} ^ {2}\right) - \sum_ {j \in s u p p (\boldsymbol {\beta} ^ {t})} \left(\nabla_ {j} \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {t}) \alpha_ {j} - \frac {m _ {2 k}}{2} \alpha_ {j} ^ {2}\right) \\ - \sum_ {j \notin \mathcal {S} ^ {R} \cup s u p p (\boldsymbol {\beta} ^ {t})} \left(\nabla_ {j} \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {t}) \alpha_ {j} - \frac {m _ {2 k}}{2} \alpha_ {j} ^ {2}\right) \\ \end{array}
$$
Recall  $S^R = \mathcal{S}^*\setminus supp(\beta^t)$ . We divide the indices  $\{1,2,\dots,p\}$  into three groups:  $\mathcal{S}^R$ ,  $supp(\beta^t)$ , and the rest.
$$
= - \sum_ {j \in \mathcal {S} ^ {R}} \left(\nabla_ {j} \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {\beta} ^ {t}) \alpha_ {j} - \frac {m _ {2 k}}{2} \alpha_ {j} ^ {2}\right)
$$
$$
\# \nabla_ {j} \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {\beta} ^ {t}) = 0 \text {f o r} j \in \operatorname {s u p p} (\boldsymbol {\beta} ^ {t}), \text {a n d} \alpha_ {j} = 0 \text {f o r} j \notin \mathcal {S} ^ {R} \cup \operatorname {s u p p} (\boldsymbol {\beta} ^ {t})
$$
$$
= - \sum_ {j ^ {\prime} \in \mathcal {J} ^ {\prime}} \left(\nabla_ {j ^ {\prime}} \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {\beta} ^ {t}) \alpha_ {j ^ {\prime}} - \frac {m _ {2 k}}{2} \alpha_ {j ^ {\prime}} ^ {2}\right) \quad \# C h a n g e t h e i n d e x n o t a t i o n f r o m j t o j ^ {\prime}
$$
$$
\leq \max  _ {\boldsymbol {\nu}} - \sum_ {j \in \mathcal {J} ^ {\prime}} \left(\nabla_ {j} \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {\beta} ^ {t}) \nu_ {j} - \frac {m _ {2 k}}{2} \nu_ {j} ^ {2}\right)
$$
Allow  $\alpha$  to take any values with a new variable  $\nu$ . The inequality holds if we take the maximum with respect to  $\nu$
$$
= \frac {1}{2 m _ {2 k}} \sum_ {j ^ {\prime} \in \mathcal {J} ^ {\prime}} \left(\nabla_ {j ^ {\prime}} \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right)\right) ^ {2} \# \text {e a c h t e r m c a n b e m a x i m i z e d b e c a u s e i t i s a q u a d r a t i c f u n c t i o n}
$$
This gives us the lower bound:
$$
\sum_ {j ^ {\prime} \in \mathcal {J} ^ {\prime}} \left(\nabla_ {j ^ {\prime}} \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right)\right) ^ {2} \geq 2 m _ {2 k} \left(\mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) - \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {*}\right)\right) \tag {59}
$$
Step 4: Final Bound for the Loss of Our Heuristic Solution
If we plug Inequalities (58) and (59) into the Inequality (55), we have:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {r i d g e}} (\pmb {\beta} ^ {t}) - \mathcal {L} _ {\mathrm {r i d g e}} (\pmb {\beta} ^ {t + 1}) \geq \frac {1}{4 k} \frac {1}{\max _ {j ^ {\prime} \in \mathcal {S} ^ {R}} (\| \pmb {X} _ {: j ^ {\prime}} \| _ {2} ^ {2} + \lambda_ {2})} \sum_ {j ^ {\prime} \in \mathcal {S} ^ {R}} (\nabla_ {j ^ {\prime}} \mathcal {L} _ {\mathrm {r i d g e}} (\pmb {\beta} ^ {t})) ^ {2} \\ \Rightarrow \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) - \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t + 1}\right) \geq \frac {m _ {2 k}}{k M _ {1}} \left(\mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {t}\right) - \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {*}\right)\right) \tag {60} \\ \end{array}
$$
Lastly, let's use Inequality (60) to derive the right-hand-side of Inequality (51) stated in the Theorem. Inequality (60) can be rewritten as:
$$
(\mathcal {L} _ {\mathrm {r i d g e}} (\pmb {\beta} ^ {t}) - \mathcal {L} _ {\mathrm {r i d g e}} (\pmb {\beta} ^ {*})) - (\mathcal {L} _ {\mathrm {r i d g e}} (\pmb {\beta} ^ {t + 1}) - \mathcal {L} _ {\mathrm {r i d g e}} (\pmb {\beta} ^ {*})) \geq \frac {m _ {2 k}}{k M _ {1}} (\mathcal {L} _ {\mathrm {r i d g e}} (\pmb {\beta} ^ {t}) - \mathcal {L} _ {\mathrm {r i d g e}} (\pmb {\beta} ^ {*})),
$$
which can be rearranged to the following expression:
$$
(1 - \frac {m _ {2 k}}{k M _ {1}}) (\mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {t}) - \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {*})) \geq (\mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {t + 1}) - \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {*}))
$$
Therefore, by applying the above inequality  $k$  times, we have:
$$
(1 - \frac {m _ {2 k}}{k M _ {1}}) ^ {k} (\mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {0}) - \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {*})) \geq (\mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {k}) - \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {*}))
$$
However,  $\mathcal{L}_{\mathrm{ridge}}(\mathbf{0}) = 0$ . This gives us
$$
\mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {k}\right) \leq \left(1 - \left(1 - \frac {m _ {2 k}}{k M _ {1}}\right) ^ {k}\right) \mathcal {L} _ {\mathrm {r i d g e}} \left(\boldsymbol {\beta} ^ {*}\right)
$$
To get the final inequality, we start from a well-known inequality result  $1 + x \leq e^{x}$  for any  $x \in \mathbb{R}$ .
Let  $x = -\frac{a}{k} > -1$ , then we have  $1 - \frac{a}{k} \leq e^{-a / k}$ . If we take both sides to the power of  $k$ , we have  $(1 - \frac{a}{k})^k \leq e^{-a}$ .
If we let  $a = \frac{m_2k}{M_1}$  (because  $\frac{m_2k}{M_1} < 1$  [29] we have  $x = -\frac{a}{k} = -\frac{m_{2k}}{M_1k} > -1$  for any  $k \geq 1$ ), then we have
$$
\left(1 - \frac {m _ {2 k}}{k M _ {1}}\right) ^ {k} \leq e ^ {- \frac {m _ {2 k}}{M _ {1}}} \tag {61}
$$
Because  $\mathcal{L}_{\mathrm{ridge}}(\beta^{*})$  is a negative number, we finally have