SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {k}\right) \leq \left(1 - \left(1 - \frac {m _ {2 k}}{k M _ {1}}\right) ^ {k}\right) \mathcal {L} _ {\text {r i d g e}} \left(\boldsymbol {\beta} ^ {*}\right) \\ \leq (1 - e ^ {- \frac {m _ {2 k}}{M _ {1}}}) \mathcal {L} _ {\mathrm {r i d g e}} (\boldsymbol {\beta} ^ {*}). \\ \end{array}
$$
C.5 Derivation of  $g(a, b) = \max_{c} ac - \frac{c^2}{4} b$  via Fenchel Conjugate
Recall our function  $g:\mathbb{R}\times \mathbb{R}\to \mathbb{R}\cup \{+\infty \}$  is defined as:
$$
g (a, b) = \left\{ \begin{array}{l l} \frac {a ^ {2}}{b} & \text {i f} b > 0 \\ 0 & \text {i f} a = b = 0 \\ \infty & \text {o t h e r w i s e} \end{array} \right. \tag {62}
$$
The convex conjugate of  $g(\cdot ,\cdot)$  , denoted as  $g^{*}(\cdot ,\cdot)$  , is defined as [8]:
$$
g ^ {*} (c, d) = \sup  _ {a, b} (a c + b d - g (a, b)) \tag {63}
$$
Since function  $g(a, b)$  is convex and lower semi-continuous, the biconjugate of  $g(a, b)$  is equal to the original function, i.e.,  $(g^*)^* = g$  [8]. Writing this explicitly, we have:
$$
g (a, b) = \sup  _ {c, d} \left(a c + b d - g ^ {*} (c, d)\right) \tag {64}
$$
For now, let us first derive an analytic formula for the convex conjugate function  $g^{*}(\cdot ,\cdot)$
According to the definition of  $g(a,b)$  in Equation (62), we have
$$
h (a, b, c, d) = a c + b d - g (a, b) = \left\{ \begin{array}{l l} a c + b d - \frac {a ^ {2}}{b} & \text {i f} b > 0 \\ 0 & \text {i f} a = b = 0 \\ - \infty & \text {o t h e r w i s e} \end{array} \right..
$$
For any fixed  $c$  and  $d$ , the optimality of the first case can be achieved if the first derivatives of  $ac + bd - \frac{a^2}{b}$  with respect to  $a$  and  $b$  are both equal to 0. This means that,
$$
c - 2 \frac {a}{b} = 0 \quad \text {a n d} \quad d + \frac {a ^ {2}}{b ^ {2}} = 0.
$$
If  $d = -\frac{c^2}{4}$ , the optimality condition is achieved if we let  $a = \frac{bc}{2}$ , and then  $ac + bd - \frac{a^2}{b} = \frac{bc}{2} c + b\left(-\frac{c^2}{4}\right) - \frac{c^2}{4} b = 0$ .
If  $d \neq -\frac{c^2}{4}$ , the optimality condition can not be achieved. However, if we still let  $a = \frac{bc}{2}$ , we have  $ac + bd - \frac{a^2}{b} = \frac{bc}{2} c + b(d) - \frac{c^2}{4} b = (\frac{c^2}{4} + d)b$ . Then, we have two specific cases to discuss: 1) if  $\frac{c^2}{4} + d > 0$ , the supremum is  $\infty$  if we let  $b \to \infty$ ; 2) if  $\frac{c^2}{4} + d < 0$ , the supremum is  $0$  if we let  $b \to 0$ .
With all these cases discussed, we have that
$$
\begin{array}{l} g ^ {*} (c, d) = \sup  _ {a, b} h (a, b, c, d) \\ = \left\{ \begin{array}{l l} 0 & \text {i f} \frac {c ^ {2}}{4} + d \leq 0 \\ \infty & \text {o t h e r w i s e} \end{array} \right. \\ \end{array}
$$
Now let us revisit Equation (64), where  $g(a,b) = \sup_{c,d} ac + bd - g^{*}(c,d)$ . Below, we discuss three cases and derive the alternative formulas for  $g(a,b)$  under different scenarios: (1)  $\frac{c^2}{4} + d = 0$ , (2)  $\frac{c^2}{4} + d \leq 0$ , and (3)  $\frac{c^2}{4} + d > 0$ .
1.  $\frac{c^2}{4} + d = 0$ :
$$
g _ {1} (a, b) = \sup  _ {c, d, \frac {c ^ {2}}{4} + d = 0} a c + b d - g ^ {*} (c, d) = \sup  _ {c} a c + b (- \frac {c ^ {2}}{4})
$$
2.  $\frac{c^2}{4} + d < 0$ :
$$
g_{2}(a,b) = \sup_{c,d,\frac{c^{2}}{4} +d <   0}ac + bd - g^{*}(c,d) = \sup_{c,d}ac + bd
$$
3.  $\frac{c^2}{4} + d > 0$ :
$$
g_{3}(a,b) = \sup_{c,d,\frac{c^{2}}{4} +d > 0}ac + bd - g^{*}(c,d) = -\infty
$$
After discussing these three cases, we can obtain the final results by taking the maximum of the three optimal values, i.e.,  $g(a,b) = \max(g_1(a,b), g_2(a,b), g_3(a,b))$ . The third case  $\frac{c^2}{4} + d > 0$  is not interesting and can be easily eliminated. The second case  $\frac{c^2}{4} + d < 0$  is also not interesting because  $ac + bd = ac + b\left(-\frac{c^2}{4}\right) + b\left(\frac{c^2}{4} + d\right) \leq ac + b\left(-\frac{c^2}{4}\right)$  for any  $b \geq 0$ , which is the domain we are interested in. This gives us  $g_2(a,b) \leq g_1(a,b)$ , and the equality is only achieved when  $b = 0$ . Thus,
$$
\begin{array}{l} g (a, b) = \max  \left(g _ {1} (a, b), g _ {2} (a, b), g _ {3} (a, b)\right) \\ = g _ {1} (a, b) (65) \\ = \sup  _ {c} a c + b \left(- \frac {c ^ {2}}{4}\right) \\ = \max  _ {c} a c - b \frac {c ^ {2}}{4}, (66) \\ \end{array}
$$
where in the last step we convert  $\sup (\cdot)$  to  $\max (\cdot)$  because the function is quadratic and the optimum can be achieved.
D Existing MIP Formulations for  $k$ -sparse Ridge Regression
We want to solve the following  $k$ -sparse ridge regression problem:
$$
\min  _ {\boldsymbol {\beta}, \boldsymbol {z}} \quad \boldsymbol {\beta} ^ {T} \boldsymbol {X} ^ {T} \boldsymbol {X} \boldsymbol {\beta} - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {\beta} + \lambda_ {2} \sum_ {j = 1} ^ {p} \beta_ {j} ^ {2} \tag {67}
$$
subject to  $(1 - z_{j})\beta_{j} = 0,\sum_{j = 1}^{p}z_{j}\leq k,z_{j}\in \{0,1\}$
However, we can not handle the constraint  $(1 - z_{j})\beta_{j} = 0$  directly in a mixed-integer programming (MIP) solver. Currently in the research community, there are four formulations we can write before sending the problem to a MIP solver. They are (1) SOS1, (2) big-M, (3) perspective, and (4) optimal perspective. We write each formulation explicitly below:
1. SOS1 formulation:
$$
\min  _ {\boldsymbol {\beta}, \boldsymbol {z}} \quad \boldsymbol {\beta} ^ {T} \boldsymbol {X} ^ {T} \boldsymbol {X} \boldsymbol {\beta} - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {\beta} + \lambda_ {2} \sum_ {j = 1} ^ {p} \beta_ {j} ^ {2} \tag {68}
$$
subject to  $(1 - z_{j},\beta_{j})\in \mathrm{SOS1}$ $\sum_{j = 1}^{p}z_{j}\leq k,\quad z_{j}\in \{0,1\} ,$
where  $(a,b) \in \mathrm{SOS1}$  means that at most one variable (either  $a$  or  $b$  but not both) can be nonzero.
2. big-M formulation:
$$
\min  _ {\boldsymbol {\beta}, \boldsymbol {z}} \quad \boldsymbol {\beta} ^ {T} \boldsymbol {X} ^ {T} \boldsymbol {X} \boldsymbol {\beta} - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {\beta} + \lambda_ {2} \sum_ {j = 1} ^ {p} \beta_ {j} ^ {2} \tag {69}
$$
where  $M > 0$ . For the big-M formulation, as we mentioned in the main paper, it is very challenging to choose the right value for  $M$ . If  $M$  is too big, the lower bound given by the relaxed problem is too loose; if  $M$  is too small, we risk cutting off the optimal solution.
3. perspective formulation:
$$
\min  _ {\boldsymbol {\beta}, \boldsymbol {z}, \boldsymbol {s}} \quad \boldsymbol {\beta} ^ {T} \boldsymbol {X} ^ {T} \boldsymbol {X} \boldsymbol {\beta} - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {\beta} + \lambda_ {2} \sum_ {j = 1} ^ {p} s _ {j} \tag {70}
$$
subject to  $\beta_j^2\leq s_jz_j,\sum_{j = 1}^{p}z_j\leq k,z_j\in \{0,1\}$
where  $s_j$ 's are new variables which ensure that when  $z_j = 0$ , the corresponding coefficient  $\beta_j = 0$ . The problem is a mixed-integer quadratically-constrained quadratic problem (QCQP). The problem can be solved by a commercial solver which accepts QCQP formulations.
4. eigen-perspective formulation:
$$
\min  _ {\boldsymbol {\beta}, \boldsymbol {z}, \boldsymbol {s}} \quad \boldsymbol {\beta} ^ {T} \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda_ {\min } \left(\boldsymbol {X} ^ {T} \boldsymbol {X}\right) \boldsymbol {I}\right) \boldsymbol {\beta} - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {\beta} + \left(\lambda_ {2} + \lambda_ {\min } \left(\boldsymbol {X} ^ {T} \boldsymbol {X}\right)\right) \sum_ {j = 1} ^ {p} s _ {j} \tag {71}
$$
subject to  $\beta_j^2\leq s_jz_j,\sum_{j = 1}^{p}z_j\leq k,z_j\in \{0,1\} ,$
where  $\lambda_{\mathrm{min}}(\cdot)$  denotes the smallest eigenvalue of a matrix. This formulation is similar to the QCQP formulation above, but we increase the  $\ell_2$  coefficient by subtracting  $\lambda_{\mathrm{min}}(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{I}$  from the diagonal of  $\boldsymbol{X}^T\boldsymbol{X}$ .
5. optimal perspective formulation:
$$
\min  _ {\boldsymbol {B}, \beta , \boldsymbol {z}} \left\langle \boldsymbol {X} ^ {T} \boldsymbol {X}, \boldsymbol {B} \right\rangle - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {\beta} \tag {72}
$$
$$
\text {s u b j e c t} \quad \boldsymbol {B} - \boldsymbol {\beta} \boldsymbol {\beta} ^ {T} \succeq 0, \quad \beta_ {j} ^ {2} \leq B _ {j j} z _ {j}, \quad \sum_ {j = 1} ^ {p} z _ {j} \leq k, \quad z _ {j} \in \{0, 1 \},
$$
where  $A \succeq 0$  means that  $A$  is positive semidefinite. Such formulation is called a mixed-integer semidefinite programming (MI-SDP) problem and is proposed in [62, 28, 37]. Currently, there is no commercial solver that can solve MI-SDP problems: Gurobi does not support SDP; MOSEK can solve SDP but does not support MI-SDP. As shown by [28], SDP is not scalable to high dimensions. We will confirm this observation in our experiments by solving the relaxed convex SDP problem (relaxing  $\{0,1\}$  to  $[0,1]$ ) via MOSEK, the state-of-the-art conic solver. After we get the solution from the relaxed problem, we find the indices corresponding to the top  $k$  largest  $z_{j}$ 's and retrieve a feasible solution on these features.
E Algorithmic Charts
E.1 Overall Branch-and-bound Framework
We first provide the overall branch-and-bound framework. Then we go into details on how to calculate the lower bound, how to obtain a near-optimal solution and upper bound (via beam search), and how to do branching. A visualization of the branch-and-bound tree can be found in Figure 5.
Figure 5: Branch-and-bound diagram.
Algorithm 1 OKRidge  $(\mathcal{D},k,\lambda_2,\epsilon ,T_{\mathrm{iter}},T_{\mathrm{max}})\to \beta$
Input: dataset  $\mathcal{D}$ , sparsity constraint  $k$ ,  $\ell_2$  coefficient  $\lambda_{2}$ , optimality gap  $\epsilon$ , iteration limit  $T_{\mathrm{iter}}$  for subgradient ascent, and time limit  $T_{\mathrm{max}}$ .  
Output: a sparse continuous coefficient vector  $\beta \in \mathbb{R}^p$  with  $\| \beta \| _0\leq k$
E.2 Fast Lower Bound Calculation
Algorithm 2 Node.lowerSolveFast()  $\rightarrow$ $\mathcal{L}_{\mathrm{lower}}$
Attributes in the Node class: dataset  $\mathcal{D} = \{(x_i,y_i)\}_{i = 1}^n$ , sparsity constraint  $k$ ,  $\ell_2$  coefficient  $\lambda_{2}$ , set of features that must be selected  $\mathcal{N}_{\mathrm{select}}$ , set of features that must be avoided  $\mathcal{N}_{\mathrm{avoid}}$ .
Output: a lower bound for the node  $\mathcal{L}_{\mathrm{lower}}$
1:  $\pmb{\gamma} = \operatorname{argmin}_{\pmb{\beta}, \pmb{\beta}_j = 0}$  if  $j \in \mathcal{N}_{\text{avoid}}$ $\pmb{\beta}^T \pmb{X}^T \pmb{X} \pmb{\beta} - 2 \pmb{y}^T \pmb{X} \pmb{\beta} + \lambda_2 \| \pmb{\beta} \|_2^2$  Solve the ridge regression problem. We avoid using coordinates belonging to  $\mathcal{N}_{\text{avoid}}$  by fixing the coefficients on those coordinates to be 0.
2:  $\mathcal{L}_{\mathrm{ridge}} = \gamma^T\pmb {X}^T\pmb {X}\pmb {\gamma} - 2\pmb {y}^T\pmb {X}\pmb {\gamma} + \lambda_2\| \pmb {\gamma}\| _2^2$  Loss for the ridge regression.
3:  $\mathcal{L}_{\mathrm{extra}} = (\lambda_2 + \lambda_{\min}(\pmb{X}^T\pmb {X}))\cdot \mathrm{SumBottom}_{p - |\mathcal{N}_{\mathrm{avoid}}| - k}(\{\beta_j^2,\mathrm{for}\forall j\in [1,\dots ,p]\setminus (\mathcal{N}_{\mathrm{avoid}}\cup \mathcal{N}_{\mathrm{select}})\})$  ▷First of all, we are restricted to the coordinates not belonging to  $\mathcal{N}_{\mathrm{avoid}}$ . Second, coordinates in the set  $\mathcal{N}_{\mathrm{select}}$  must be selected. If we work through the strong convexity argument in Section 3.1 in the main paper, we would get this formula.
4:  $\mathcal{L}_{\mathrm{lower}} = \mathcal{L}_{\mathrm{ridge}} + \mathcal{L}_{\mathrm{extra}}$
5: return  $\mathcal{L}_{\mathrm{lower}}$
E.3 Refine the Lower Bound through the ADMM method
Next, we show how to get a tighter lower bound through the ADMM method.
Algorithm 3 Node.lowersolveTightADMM()  $\rightarrow$ $\mathcal{L}_{\mathrm{lower}}$
Attributes in the Node class: dataset  $\mathcal{D} = \{(x_i,y_i)\}_{i = 1}^n$ , sparsity constraint  $k$ ,  $\ell_2$  coefficient  $\lambda_{2}$ , set of features that must be selected  $\mathcal{N}_{\mathrm{select}}$ , and iteration limit  $T_{\mathrm{iter}}$  for ADMM (For simplicity of presentation, we assume  $\mathcal{N}_{\mathrm{avoid}}$  is empty. If  $\mathcal{N}_{\mathrm{avoid}}\neq \emptyset$ , we can do a selection and relabeling step on  $X,Y,Q_{\lambda},\mathcal{N}_{\mathrm{select}}$  to focus solely on coordinates not belonging to  $\mathcal{N}_{\mathrm{avoid}}$ ).
1:  $\pmb{\gamma}^{1} = \mathrm{argmin}_{\pmb{\beta}}\pmb{\beta}^{T}\pmb{X}^{T}\pmb{X}\pmb{\beta} - 2\pmb{y}^{T}\pmb{X}\pmb{\beta} + \lambda_{2}\|\pmb{\beta}\|_{2}^{2}\triangleright$  Initialize  $\gamma$  by solving the ridge regression problem. This will be used as a warm start for the ADMM method below. To avoid redundant computation, we can use the value for  $\gamma$  from Node.lowerSolveFast().
2:  $\pmb{p}^{1} = \pmb{X}^{T}\pmb{y} - \pmb{Q}_{\lambda}\pmb{\gamma}$  ▷Initialize  $\pmb{p}$  as stated in Problem 18.  
3:  $\pmb{q}^{1} = \pmb{0}$  ▷Initialize the scaled dual variable  $\pmb{q}$  in the ADMM.
4:  $\rho = 2 / \sqrt{\lambda_{\max}(\mathbf{Q}_{\lambda})\lambda_{\min >0}} (\mathbf{Q}_{\lambda})$ $\triangleright$  Calculate the step size used in the ADMM
5: for  $t = 2, 3, \ldots T_{\mathrm{iter}}$ . do
6:  $\pmb{\gamma}^t = \left(\frac{2}{\rho}\pmb {I} + \pmb {Q}_\lambda\right)^{-1}(\pmb {X}^T\pmb {y} - \pmb{p}^{t - 1} - \pmb{q}^{t - 1})$  ▷Update  $\gamma$  in the ADMM.
7:  $\pmb{\theta}^{t} = 2\pmb{Q}_{\lambda}\pmb{\gamma}^{t} + \pmb{p}^{t - 1} - \pmb{X}^{T}\pmb{y}$  ▷Update  $\pmb{\theta}$  in the ADMM.
8:  $\pmb{a} = \pmb{X}^T\pmb{y} - \pmb{\theta}^t - \pmb{q}^{t-1} \triangleright$ This line and the next 3 lines are used to calculate variables used for setting up the isotonic regression problem.
9: Let  $\mathcal{J}$  be the indices of the top  $k - |\mathcal{N}_{\mathrm{select}}|$  terms of  $|a_{j}|$  with the constraint that any element in  $\mathcal{J}$  does not belong to  $\mathcal{N}_{\mathrm{select}}$ .
10: Let  $w_{j} = 1$  if  $j \notin \mathcal{J}$  and  $j \notin \mathcal{N}_{\mathrm{select}}$  and let  $w_{j} = \frac{2}{\rho(\lambda_{2} + \lambda) + 1}$  otherwise.
11: Let  $b_{j} = \frac{|a_{j}|}{w_{j}}$ .
12: Solve the problem  $\hat{\pmb{v}} = \mathrm{argmin}_{\pmb{v}}\sum_{j}w_{j}(v_{j} - b_{j})^{2}$  such that for  $\forall i,l\notin \mathcal{N}_{\mathrm{select}},v_i\geq v_l$  if  $|a_i|\geq |a_l|$ . We can decompose this optimization problem into two independent problems with coordinates belonging to  $\mathcal{N}_{\mathrm{select}}$  (can be solved individually) and coordinates not belonging to  $\mathcal{N}_{\mathrm{select}}$  (can be solved via standard isotonic regression).
13:  $\pmb{p}^{t} = \text{sign}(\pmb{a}) \odot \hat{\pmb{v}}$  Update  $\pmb{p}$  in the ADMM. The symbol  $\odot$  denotes the element-wise (Hadamard) product.
14:  $\pmb{q}^{t} = \pmb{q}^{t - 1} + \pmb{\theta}^{t} + \pmb{p}^{t} - \pmb{X}^{T}\pmb{y}$  ▷Update q in the ADMM.
15: end for
16:  $\mathcal{L}_{\mathrm{lower}} = h(\pmb{\gamma}^{T_{\mathrm{iter}}})$ $\triangleright$  Calculate the lower bound.
17: return  $\mathcal{L}_{\mathrm{lower}}$
E.4 (Optional) An alternative to the Lower Bound Calculation via Subgradient Ascent and the CMF method
In addition to the ADMM method, we also provide an alternative algorithm showing how to get a tighter lower bound through the subgradient ascent algorithm using the CMF method. Note that this subgradient method has been superseded by the ADMM method, which converges faster. We provide this method in the Appendix for the sake of completeness and for future readers to consult.
Algorithm 4 Node.lowerSolveTightSubgradient()  $\rightarrow$ $\mathcal{L}_{\mathrm{lower}}$
Attributes in the Node class: dataset  $\mathcal{D} = \{(x_i,y_i)\}_{i = 1}^n$ , sparsity constraint  $k$ ,  $\ell_2$  coefficient  $\lambda_{2}$ , set of features that must be selected  $\mathcal{N}_{\mathrm{select}}$ , set of features that must be avoided  $\mathcal{N}_{\mathrm{avoid}}$ , and iteration limit  $T_{\mathrm{iter}}$  for subgradient ascent
1:  $\pmb{\gamma} = \operatorname{argmin}_{\pmb{\beta}, \beta_j = 0}$  if  $j \in \mathcal{N}_{\text{avoid}}$ $\pmb{\beta}^T \pmb{X}^T \pmb{X} \pmb{\beta} - 2 \pmb{y}^T \pmb{X} \pmb{\beta} + \lambda_2 \| \pmb{\beta} \|_2^2$  Solve the ridge regression problem and avoid using coordinates in  $\mathcal{N}_{\text{avoid}}$ . This will be used as a warm start for the subgradient ascent procedure below. To avoid redundant computation, we can use the value for  $\gamma$  from Node.lowerSolveFast()
2:  $\pmb{v} = \pmb{\gamma}$  ▷Initialization for  $v^t$  at iteration step  $t = 1$
3:  $h^* = \mathcal{L}_{\mathrm{ridge}}(\beta_{upper}^{root})$  get an estimate of  $max_{\gamma}h(\gamma)$ ; if  $\beta_{upper}^{root}$  has not been solved at the root node, solve  $\beta_{upper}^{root}$  by calling the upperSolve() function at the root node.
4:  $\mathcal{L}_{\mathrm{lower}} = h(\pmb{\gamma})$
5: for  $t = 2,3,\ldots T_{\mathrm{iter}}$  do
6:  $\gamma, \pmb{v} = \text{Node.subgradAscentCMF}(\pmb{\gamma}, \pmb{v})$  ▷Perform one step of subgradient ascent using the CMF method
7:  $\mathcal{L}_{\mathrm{lower}} = \max (\mathcal{L}_{\mathrm{lower}},h(\boldsymbol {\gamma}))$  Loss decreases stochastically during the subgradient ascent procedure. We keep the largest value of  $h(\boldsymbol {\gamma})$
8: end for
9: return  $\mathcal{L}_{\mathrm{lower}}$
In the above algorithm, we call a new sub algorithm called subGradAscentCMF. We provide the details on how to do this below. We try to maximize  $h(\gamma)$  through iterative subgradient ascent. A more sophisticated algorithm (CMF method [23]) can be applied to adaptively choose the step size, which has been shown to achieve faster convergence in practice. The computational procedure for the subgradient ascent algorithm using the CMF method [18, 23] is as follows. Given  $\gamma^{t-1}$  at iteration  $t - 1$ ,  $\gamma^t$  can be computed as:
$$
\boldsymbol {w} ^ {t} = \left. \frac {\partial \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {s a d d l e}} (\gamma , Z (\gamma^ {t - 1}))}{\partial \gamma} \right| _ {\gamma = \gamma^ {t - 1}} \quad \# \text {g e t t h e s u b g r a d i e n t}
$$
$$
s _ {t} = \max  (0, - d _ {t} (\boldsymbol {v} ^ {t - 1}) ^ {T} \boldsymbol {w} ^ {t} / \| \boldsymbol {v} ^ {t - 1} \| _ {2} ^ {2})   \# s _ {t} w i l l b e u s e d a s a s m o o t h i n g f a c t o r b e t w e e n \boldsymbol {v} ^ {t - 1} a n d \boldsymbol {w} ^ {t}
$$
$$
\boldsymbol {v} ^ {t} = \boldsymbol {w} ^ {t} + s _ {t} \boldsymbol {v} ^ {t - 1} \quad \# c o m b i n e \boldsymbol {v} ^ {t - 1} a n d \boldsymbol {w} ^ {t} u s i n g t h e s m o o t h i n g f a c t o r
$$
$$
\alpha_ {t} = \left(h ^ {*} - h \left(\boldsymbol {\gamma} ^ {t - 1}\right)\right) / \left\| \boldsymbol {v} ^ {t} \right\| _ {2} ^ {2}
$$
calculate the step size;  $h^*$  is an estimate of  $\max_{\gamma} h(\gamma)$ ; here we let  $h^* = \mathcal{L}_{\mathrm{ridge}}(\beta_{\mathrm{upper}}^{root})$ .
$$
\boldsymbol {\gamma} ^ {t} = \boldsymbol {\gamma} ^ {t - 1} + \alpha_ {t} \boldsymbol {v} ^ {t} \quad \# p e r f o r m s u b g r a d i e n t a s c e n t
$$
where at the beginning of the iteration  $(t = 1)$ ,  $\pmb{v}^{1} = \mathbf{0}$  and we choose  $d_{t} = 1.5$  as a constant for all iterations as recommended by the original paper. Below, we provide the detailed algorithm to perform subgradient ascent using the CMF method.
Attributes in the Node class: dataset  $\mathcal{D} = \{(x_i,y_i)\}_{i = 1}^n$ , sparsity constraint  $k$ ,  $\ell_2$  coefficient  $\lambda_{2}$ , set of features that must be selected  $\mathcal{N}_{\mathrm{select}}$ , set of features that must be avoided  $\mathcal{N}_{\mathrm{avoid}}$
Input: Current variable  $\gamma$  for the lower bound function  $h(\cdot)$  for Equation 14, current ascent vector  $\mathbf{v}$ , and  $h^*$  is an estimate of  $\max_{\gamma} h(\gamma)$ .
Output: updated variable  $\gamma$  and updated ascent vector  $\pmb{v}$
Algorithm 5 Node.subgradAscentCMF  $(\gamma ,v,h^{*})\to \gamma ,v$
1:  $\hat{z} = \operatorname{argmin}_{\pmb{z}} \mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{saddle}}(\gamma, z)$  ▷get the minimizer for  $z$  at the current value for  $\gamma$ ; optimization is performed under the constraint  $\sum_{j=1}^{p} z_{j} \leq k, z_{j} \in [0,1], z_{j} = 1$  for  $j \in \mathcal{N}_{\text{select}}$  and  $z_{j} = 0$  for  $j \in \mathcal{N}_{\text{select}}$
2:  $\pmb{w} = \left(\partial \mathcal{L}_{\mathrm{ridge - }\lambda}^{\mathrm{saddle}}(\pmb {\gamma},\hat{\pmb{z}})\right) / (\partial \pmb {\gamma})$
3:  $\pmb{w}_j = 0$  for  $j \in \mathcal{N}_{\text{avoid}}$ . We don't perform any ascent steps on the coordinates belong to  $\mathcal{N}_{\text{avoid}}$ .
4:  $s = \max (0, - 1.5\pmb{v}^T\pmb {w} / \| \pmb {v}\| _2^2)$  >Calculate the smoothing factor
5:  $\pmb{v} = \pmb{w} + s\pmb{v}$  ▷update the ascent vector by combining the current subgradient and previous ascent vector
6:  $\alpha = (h^{*} - h(\pmb {\gamma})) / \| \pmb {v}\|_{2}^{2}$
7:  $\pmb {\gamma} = \pmb {\gamma} + \alpha \pmb{v}$  Perform one step of ascent along the direction of the ascent vector
8: return  $\gamma, v$
E.5 Beam Search Algorithm
We provide a visualization of the beam search algorithm in Figure 6
Figure 6: We add into our support one feature at a time by picking the feature which would result in the largest decrease in the objective function. After that, we finetune all coefficients on the support. We keep the top  $B$  solutions during each stage of support expansion.
Algorithm 6 Node.upperSolve  $(B = 50)\rightarrow \beta_{\mathrm{upper}},\mathcal{L}_{\mathrm{upper}}$
Attributes in the Node class: dataset  $\mathcal{D} = \{(x_i,y_i)\}_{i = 1}^n$ , sparsity constraint  $k$ ,  $\ell_2$  coefficient  $\lambda_{2}$ , a set of feature coordinates that must be selected  $\mathcal{N}_{\mathrm{select}}$ , and a set of feature coordinates that must be avoided  $\mathcal{N}_{\mathrm{avoid}}$ .
Input: beam search size  $B$ , which is set to be 50 as the default value.
Output: a near-optimal solution  $\beta_{\mathrm{upper}}$  with  $\| \beta_{\mathrm{upper}}\| _0 = k$  and the corresponding loss  $\mathcal{L}_{\mathrm{upper}}$ .
1:  $\pmb{\beta}^{\prime} = \mathrm{argmin}_{\pmb{\beta},\beta_{j} = 0\text{for} j\notin \mathcal{N}_{\mathrm{select}}}\pmb{\beta}^{T}\pmb{X}^{T}\pmb{X}\pmb{\beta} - 2\pmb{y}^{T}\pmb{X}\pmb{\beta} + \lambda_{2}\|\pmb{\beta}\|_{2}^{2}$  ▷ Initialize the coefficients by fitting on the must-be-selected coordinates.
2:  $\mathcal{W} = \{\beta^{\prime}\}$  ▷collection of solutions at each iteration (initially we only have one solution)
3:  $\mathcal{F} = \emptyset$  Initialize the collection of found supports as an empty set
4: for  $t = |\mathcal{N}_{\mathrm{select}}| + 1, \dots, k$  do
5:  $\mathcal{W}_{\mathrm{tmp}}\gets \emptyset$
6: for  $\beta^{\prime}\in \mathcal{W}$  do
7:  $(\mathcal{W}',\mathcal{F})\gets \text{ExpandSuppBy1}(\mathcal{D},\beta',\mathcal{F},B,\mathcal{N}_{\text{avoid}})$ .  $\triangleright$  Returns  $\leq B$  models with supp.  $t$
8:  $\mathcal{W}_{\mathrm{tmp}} \gets \mathcal{W}_{\mathrm{tmp}} \cup \mathcal{W}'$
9: end for
10: Reset  $\mathcal{W}$  to be the  $B$  solutions in  $\mathcal{W}_{\mathrm{tmp}}$  with the smallest ridge regression loss values.
1: end for
12: Pick  $\beta_{\mathrm{upper}}$  from  $\mathcal{W}$  with the smallest ridge regression loss.
13:  $\mathcal{L}_{\mathrm{upper}} = \beta_{\mathrm{upper}}^{T}\pmb{X}^{T}\pmb{X}\beta_{\mathrm{upper}} - 2\pmb{y}^{T}\pmb{X}\beta_{\mathrm{upper}} + \lambda_{2}\| \beta_{\mathrm{upper}}\|_{2}^{2}$
14: return  $\beta_{\mathrm{upper}}$ ,  $\mathcal{L}_{\mathrm{upper}}$ .
Algorithm 7 ExpandSuppBy1  $(\mathcal{D},\beta^{\prime},\mathcal{F},B,\mathcal{N}_{\mathrm{avoid}})\to \mathcal{W},\mathcal{F}$
Input: dataset  $\mathcal{D} = \{(x_i,y_i)\}_{i = 1}^n$ , current solution  $\beta^\prime$ , set of found supports  $\mathcal{F}$ , beam search size  $B$ , and a set of feature coordinates that must be avoided  $\mathcal{N}_{\mathrm{avoid}}$ .