SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f (\boldsymbol {y}) \geq f (\boldsymbol {x}) + \boldsymbol {g} ^ {T} (\boldsymbol {y} - \boldsymbol {x}) + \frac {\alpha}{2} \| \boldsymbol {y} - \boldsymbol {x} \| _ {2} ^ {2} \tag {31}
$$
for  $\pmb{x}, \pmb{y} \in \mathbb{R}^p$ , where  $\pmb{g} \in \partial f$  and  $\partial f$  is the subdifferential of  $f$ .
Definition B.2 (Smoothness [20]). A continuously differentiable function  $f: \mathbb{R}^p \to \mathbb{R}$  is smooth with parameter  $\beta > 0$  if the gradeint  $\nabla f$  is  $\beta$ -Lipschitz. Mathematically, this means that:
$$
\| \nabla f (\boldsymbol {x}) - \nabla f (\boldsymbol {y}) \| _ {2} \leq \beta \| \boldsymbol {x} - \boldsymbol {y} \| _ {2} \tag {32}
$$
for  $x,y\in \mathbb{R}^p$
Lemma B.3 (Smoothness Property [20]). Let  $f: \mathbb{R}^p \to \mathbb{R}$  be a  $\beta$ -smooth function. Then, we have the following sandwich property ( $f(\boldsymbol{y})$  is "sandwiched" between two quadratic functions):
$$
f (\boldsymbol {x}) + \nabla f (\boldsymbol {x}) ^ {T} (\boldsymbol {y} - \boldsymbol {x}) + \frac {\beta}{2} \| \boldsymbol {y} - \boldsymbol {x} \| _ {2} ^ {2} \geq f (\boldsymbol {y}) \geq f (\boldsymbol {x}) + \nabla f (\boldsymbol {x}) ^ {T} (\boldsymbol {y} - \boldsymbol {x}) - \frac {\beta}{2} \| \boldsymbol {y} - \boldsymbol {x} \| _ {2} ^ {2} \tag {33}
$$
for  $\pmb{x}, \pmb{y} \in \mathbb{R}^p$ .
Definition B.4 ( $k$ -sparse Pair Domain [29]). A domain  $\Omega_k \subset \mathbb{R}^p \times \mathbb{R}^p$  is a  $k$ -sparse vector pair domain if it contains all pairs of  $k$ -sparse vectors that differ in at most  $k$  entries, i.e,
$$
\Omega_ {k} := \left\{\left(\boldsymbol {x}, \boldsymbol {y}\right) \in \mathbb {R} ^ {p} \times \mathbb {R} ^ {p}: \| \boldsymbol {x} \| _ {0} \leq k, \| y \| _ {0} \leq k, \| \boldsymbol {x} - \boldsymbol {y} \| \leq k \right\}. \tag {34}
$$
Definition B.5 (Restricted Strong Convexity [29]). A function  $f: \mathbb{R}^p \to \mathbb{R}$  is restricted strongly convex with parameter  $\alpha > 0$  on the  $k$ -sparse vector pair domain  $\Omega_k$  if it satisfies the following inequality:
$$
f (\boldsymbol {y}) \geq f (\boldsymbol {x}) + \nabla f (\boldsymbol {x}) ^ {T} (\boldsymbol {y} - \boldsymbol {x}) + \frac {\alpha}{2} \| \boldsymbol {y} - \boldsymbol {x} \| _ {2} ^ {2} \tag {35}
$$
for  $(\pmb {x},\pmb {y})\in \Omega_{k}$
Definition B.6 (Restricted Smoothness [29]). A function  $f: \mathbb{R}^p \to \mathbb{R}$  is restrictedly smooth with parameter  $\beta > 0$  on the  $k$ -sparse vector pair domain  $\Omega_k$  if it satisfies the following inequality:
$$
f (\boldsymbol {x}) + \nabla f (\boldsymbol {x}) ^ {T} (\boldsymbol {y} - \boldsymbol {x}) + \frac {\beta}{2} \| \boldsymbol {y} - \boldsymbol {x} \| _ {2} ^ {2} \geq f (\boldsymbol {y}) \tag {36}
$$
Lemma B.7 (Gradient of Ridge Regression). At  $\beta \in \mathbb{R}^p$ , the gradient of our ridge regression function is given by:
$$
\nabla \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {\beta}) = 2 \boldsymbol {X} ^ {T} (\boldsymbol {X} \boldsymbol {\beta} - \boldsymbol {y}) + 2 \lambda_ {2} \boldsymbol {\beta} \tag {37}
$$
for  $\beta \in \mathbb{R}^p$
$$
\begin{array}{l} \nabla \mathcal {L} _ {\text {r i d g e}} (\boldsymbol {\beta}) = \nabla \left(\boldsymbol {\beta} ^ {T} \boldsymbol {X} ^ {T} \boldsymbol {X} \boldsymbol {\beta} - 2 \boldsymbol {y} ^ {X} \boldsymbol {X} \boldsymbol {\beta} + \lambda_ {2} \boldsymbol {\beta} ^ {T} \boldsymbol {\beta}\right) \\ = \nabla (\boldsymbol {\beta} ^ {T} \boldsymbol {X} ^ {T} \boldsymbol {X} \boldsymbol {\beta}) - \nabla (2 \boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {\beta}) + \lambda_ {2} \nabla (\boldsymbol {\beta} ^ {T} \boldsymbol {\beta}) \\ = 2 \boldsymbol {X} ^ {T} \boldsymbol {X} \boldsymbol {\beta} - 2 \boldsymbol {X} ^ {T} \boldsymbol {y} + 2 \lambda_ {2} \boldsymbol {\beta} \\ = 2 \boldsymbol {X} ^ {T} (\boldsymbol {X} \boldsymbol {\beta} - \boldsymbol {y}) + 2 \lambda_ {2} \boldsymbol {\beta} \\ \end{array}
$$
C Derivations and Proofs
$$
\max  _ {\boldsymbol {\gamma}} \min  _ {\boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {s a d d l e}} (\boldsymbol {\gamma}, \boldsymbol {z}) \quad s. t. \quad \sum_ {j = 1} ^ {p} z _ {j} \leq k, \quad z _ {j} \in [ 0, 1 ], \tag {38}
$$
where  $\mathcal{L}_{ridge - \lambda}^{saddle}(\pmb {\gamma},\pmb {z}):= -\pmb{\gamma}^T\pmb {Q}_\lambda \pmb {\gamma} - \frac{1}{\lambda_2 + \lambda} (\pmb {X}^T\pmb {y} - \pmb {Q}_\lambda \pmb {\gamma})^T diag(\pmb {z})(\pmb {X}^T\pmb {y} - \pmb {Q}_\lambda \pmb {\gamma}),$  (39)
and  $\text{diag}(z)$  is a diagonal matrix with  $z$  on the diagonal.
Before we give the proof for Theorem 3.1, we first state one useful lemma below:
Lemma C.1. For an optimal solution  $\pmb{c}^*$  to Problem (40), there exists some  $\beta^*$  such that  $\pmb{c}^* = \frac{2}{\lambda_2 + \lambda} (\pmb{X}^T \pmb{y} - \pmb{Q}_\lambda \pmb{\beta}^*)$ .
Without using this lemma, there is some concern with reparametrizing  $c$  with a new parameter  $\gamma$  when  $\lambda = \lambda_{\min}(X^T X)$ . In this case,  $X^T X - \lambda I$  becomes singular and our reparametrization trick could miss some subspace. This issue of having  $X^T X - \lambda I$  being singular would lead to the concern that Problem (39) is a relaxation of Problem (40), not equivalent. Fortunately, due to this lemma, the subspace which we miss via the reparametrization trick will not prevent Problem (39) from achieving the same optimal value as Problem (40).
We now prove this lemma by the method of contradiction.
Proof. Suppose the optimal solution  $\pmb{c}$  to Problem (40) cannot be written by Equation (41). Through the rank-nullity theorem in linear algebra, we can find  $\pmb{d}_1 \in \operatorname{col}(\pmb{Q}_{\lambda})$  and  $d_2 \in \ker(\pmb{Q}_{\lambda}), d_2 \neq \mathbf{0}$  so that  $\pmb{c} = \frac{2}{\lambda_2 + \lambda} (\pmb{X}^T \pmb{y} - \pmb{d}_1 + \pmb{d}_2)$ , where  $\operatorname{col}(\cdot)$  denotes the column space of a matrix and  $\ker(\cdot)$  denotes the kernel of a matrix. If we let  $\beta = \alpha d_2$  for some real value  $\alpha$ , then the inner optimization of Problem (40) (ignore the last term involving  $z_j$ 's for now since  $\beta$  and  $z$  are separable in the objective function) becomes
$$
\begin{array}{l} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {F e n c h e l}} (\boldsymbol {\beta}, \boldsymbol {z}, \boldsymbol {c}) = \boldsymbol {\beta} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\beta} - \left(\left(\lambda_ {2} + \lambda\right) \boldsymbol {c} - 2 \boldsymbol {X} ^ {T} \boldsymbol {y}\right) ^ {T} \boldsymbol {\beta} \\ = \boldsymbol {\beta} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\beta} - \left(\left(\lambda_ {2} + \lambda\right) \frac {2}{\lambda_ {2} + \lambda} \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {d} _ {1} + \boldsymbol {d} _ {2}\right) - 2 \boldsymbol {X} ^ {T} \boldsymbol {y}\right) ^ {T} \boldsymbol {\beta} \\ = \beta^ {T} Q _ {\lambda} \boldsymbol {\beta} - (2 (\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {d} _ {1} + \boldsymbol {d} _ {2}) - 2 \boldsymbol {X} ^ {T} \boldsymbol {y}) ^ {T} \boldsymbol {\beta} \\ = \boldsymbol {\beta} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\beta} + 2 \left(\boldsymbol {d} _ {1} - \boldsymbol {d} _ {2}\right) ^ {T} \boldsymbol {\beta} \\ = \boldsymbol {\beta} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\beta} + 2 \boldsymbol {d} _ {1} ^ {T} \boldsymbol {\beta} - 2 \boldsymbol {d} _ {2} ^ {T} \boldsymbol {\beta}. \\ \end{array}
$$
Because  $\beta = \alpha d_{2}$ , we have  $\beta \in \ker(Q_{\lambda})$  and  $\beta \perp d_{1}$ . Thus, the first two terms become  $\mathbf{0}$ , and we are left with  $\mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{Fenchel}} = -2d_{2}^{T}\beta = -2\alpha \| d_{2} \|_{2}^{2}$ . For the inner optimization of Problem (40), because we want to minimize over  $\beta$ , we can let  $\alpha \to \infty$ , and we will get  $\mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{Fenchel}} \to -\infty$ . This is obviously not the optimal value for Problem (40) (a simple counter-example is to let  $c = \frac{2}{\lambda_2 + \lambda} X^T y$ , and we can easily show that  $\min_{\beta, z} \mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{Fenchel}}(\beta, z, c)$  is finite). Therefore,  $d_{2}$  must be  $\mathbf{0}$ , and the optimal solution  $c$  to Problem (40) must be represented by some  $\beta$  through the equation  $c = \frac{2}{\lambda_2 + \lambda} (X^T y - Q_{\lambda} \beta)$ .
Now that we have this lemma, we can prove Theorem 3.1 rigorously. We provide two alternative proofs so that readers can understand the logic from different angles. The first proof is based on using the lemma directly; the second proof is based on the sandwich argument.
Proof. For notational convenience, let us use  $\mathcal{D}_z\coloneqq \{\pmb {z}\mid \sum_{j = 1}^p z_j\leq k,z_j\in [0,1]\}$  as a shorthand for the domain of  $\pmb{z}$ .
Recall that Problem (10) is the following optimization problem:
$$
\max  _ {\boldsymbol {c}} \min  _ {\boldsymbol {\beta}, \boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {F e n c h e l}} (\boldsymbol {\beta}, \boldsymbol {z}, \boldsymbol {c}) \tag {40}
$$
$$
\text {s u b j e c t} \quad \sum_ {j = 1} ^ {p} z _ {j} \leq k, \quad z _ {j} \in [ 0, 1 ].
$$
where  $\mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{Fenchel}}(\pmb {\beta},\pmb {z},\pmb {c}) = \pmb{\beta}^{T}\pmb{Q}_{\lambda}\pmb {\beta} - 2\pmb{y}^{T}\pmb {X}\pmb {\beta} + (\lambda_{2} + \lambda)\sum_{j = 1}^{p}(\beta_{j}c_{j} - \frac{c_{j}^{2}}{4})z_{j}$
For any fixed  $c$ , if we take the derivative of  $\mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{Fenchel}}(\beta, z, c)$  with respect to  $\beta$  and set the derivative to  $\mathbf{0}$ , we obtain the optimality condition for  $\beta$ :
$$
\boldsymbol {c} = \frac {2}{\lambda_ {2} + \lambda} \left[ \boldsymbol {X} ^ {T} \boldsymbol {y} - \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \boldsymbol {\beta} \right] \tag {41}
$$
If we reparametrize  $\pmb{c}$  with a new parameter  $\gamma$  and let  $\pmb{c} = \frac{2}{\lambda_2 + \lambda} [\pmb{X}^T\pmb{y} - (\pmb{X}^T\pmb{X} - \lambda\pmb{I})\pmb{\gamma}]$ , then  $\beta = \gamma$  satisfies the optimality condition for  $\beta$  in Equation (41). Therefore, whenever we reparametrize  $\pmb{c}$  with  $\pmb{c} = \frac{2}{\lambda_2 + \lambda} [\pmb{X}^T\pmb{y} - (\pmb{X}^T\pmb{X} - \lambda\pmb{I})\pmb{\gamma}]$ , we can just let  $\beta = \gamma$ , and this ensures us that we can achieve the inner minimum of Equation (40) with respect to  $\beta$ .
Then, we can reduce Problem (40) to Problem (38) in the following way:
$$
\max  _ {\boldsymbol {c}} \min  _ {\boldsymbol {\beta}, \boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {F e n c h e l}} (\boldsymbol {\beta}, \boldsymbol {z}, \boldsymbol {c})
$$
$$
\# \text {s u b j e c t} \sum_ {j = 1} ^ {p} z _ {j} \leq k, z _ {j} \in [ 0, 1 ]
$$
$$
= \max  _ {\boldsymbol {c}} \min  _ {\boldsymbol {z} \in \mathcal {D} _ {z}, \boldsymbol {\beta}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {F e n c h e l}} (\boldsymbol {\beta}, \boldsymbol {z}, \boldsymbol {c})
$$
use  $\mathcal{D}_z$  as the shorhand for the domain of  $z$
$$
= \max  _ {\boldsymbol {c}} \min  _ {\boldsymbol {z} \in \mathcal {D} _ {z}, \boldsymbol {\beta}} \boldsymbol {\beta} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\beta} - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {\beta} + (\lambda_ {2} + \lambda) \sum_ {j = 1} ^ {p} \left(\beta_ {j} c _ {j} - \frac {c _ {j} ^ {2}}{4} z _ {j}\right)
$$
plug in the details for  $\mathcal{L}_{\text{ridge} - \lambda}^{\text{Fenchel}}(\beta, z, c)$
$$
= \max  _ {\boldsymbol {c}} \min  _ {\boldsymbol {z} \in \mathcal {D} _ {z}} \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {\gamma} + (\lambda_ {2} + \lambda) \sum_ {j = 1} ^ {p} \left(\gamma_ {j} c _ {j} - \frac {c _ {j} ^ {2}}{4} z _ {j}\right)
$$
letting  $\beta = \gamma$  achieves the inner minimum for  $\beta$
$$
= \max  _ {\boldsymbol {c}} \min  _ {\boldsymbol {z} \in \mathcal {D} _ {z}} \gamma^ {T} \boldsymbol {Q} _ {\lambda} \gamma - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \gamma + (\lambda_ {2} + \lambda) (\gamma^ {T} \boldsymbol {c} - \frac {1}{4} \boldsymbol {c} ^ {T} d i a g (\boldsymbol {z}) \boldsymbol {c})
$$
#write the summation in vector forms;  $\operatorname {diag}(z)$  denotes a diagonal matrix with  $z$  on the diagonal
$$
= \max  _ {\boldsymbol {\gamma}} \min  _ {\boldsymbol {z} \in \mathcal {D} _ {z}} \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {\gamma} + (\lambda_ {2} + \lambda) \left(\boldsymbol {\gamma} ^ {T} \frac {2}{\lambda_ {2} + \lambda} \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma}\right) \right.
$$
$$
\left. - \frac {1}{4} \frac {2}{\lambda_ {2} + \lambda} \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma}\right) ^ {T} \operatorname {d i a g} (\boldsymbol {z}) \frac {2}{\lambda_ {2} + \lambda} \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma}\right)\right)
$$
apply the reparametrization  $\pmb{c} = \frac{2}{\lambda_2 + \lambda} (\pmb{X}^T\pmb{y} - \pmb{Q}_\lambda \pmb{\gamma})$ .
remember that due to Lemma C.1, we will not miss the optimal solution  $c^*$
$$
= \max  _ {\gamma} \min  _ {z \in \mathcal {D} _ {z}} \gamma^ {T} Q _ {\lambda} \gamma - 2 y ^ {T} X \gamma + 2 \gamma^ {T} (X ^ {T} y - Q _ {\lambda} \gamma) - \frac {1}{\lambda_ {2} + \lambda} (X ^ {T} y - Q _ {\lambda} \gamma) ^ {T} d i a g (z) (X ^ {T} y - Q _ {\lambda} \gamma)
$$
distribute  $\lambda_{2} + \lambda$  inside the parentheses
$$
= \max  _ {\gamma} \min  _ {z \in \mathcal {D} _ {z}} \gamma^ {T} Q _ {\lambda} \gamma - 2 y ^ {T} X \gamma + 2 \gamma^ {T} X ^ {T} y - 2 \gamma^ {T} Q _ {\lambda} \gamma - \frac {1}{\lambda_ {2} + \lambda} (X ^ {T} y - Q _ {\lambda} \gamma) ^ {T} d i a g (z) (X ^ {T} y - Q _ {\lambda} \gamma)
$$
distribute  $2\gamma$  inside the parentheses
$$
= \max  _ {\gamma} \min  _ {z \in \mathcal {D} _ {z}} - \gamma^ {T} Q _ {\lambda} \gamma - \frac {1}{\lambda_ {2} + \lambda} \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - Q _ {\lambda} \gamma\right) ^ {T} d i a g (\boldsymbol {z}) \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - Q _ {\lambda} \gamma\right)
$$
# simplify the linear algebra
$$
= \max  _ {\boldsymbol {\gamma}} \min  _ {\boldsymbol {z}} - \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} - \frac {1}{\lambda_ {2} + \lambda} (\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma}) ^ {T} d i a g (\boldsymbol {z}) (\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma})
$$
subject to  $\sum_{j = 1}^{p}z_{j}\leq k,z_{j}\in [0,1]$
$$
= \max  _ {\gamma} \min  _ {z} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {s a d d l e}} (\gamma , z)
$$
subject to  $\sum_{j=1}^{p} z_{j} \leq k, z_{j} \in [0,1]$ ; apply the definition of  $\mathcal{L}_{ridge-\lambda}^{saddle}(\gamma, z)$
This completes the derivation for Theorem 3.1.
Proof. We will show  $\max_c\min_{\beta ,z}\mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{Fenchel}}(\beta ,z,c)\leq \max_\gamma \min_z\mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{saddle}}(\gamma ,z)$  and  $\max_c\min_{\beta ,z}\mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{Fenchel}}(\beta ,z,c)\geq \max_\gamma \min_z\mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{saddle}}(\gamma ,z)$  (both under the constrains  $\sum_{j = 1}^{p}z_{j}\leq k,z_{j}\in [0,1]$ ) and therefore establish the equivalence of optimal values.
For the first inequality, let  $\beta^{*}, c^{*}$  be the optimal solution to Problem (40) with the relation  $c^{*} = \frac{2}{\lambda_{2} + \lambda} (\pmb{X}^{T}\pmb{y} - \pmb{Q}_{\lambda}\pmb{\beta}^{*})$  (this is always true because of Lemma C.1). Furthermore, let  $\gamma^{*} = \beta^{*}$ . Then we have
$$
\max  _ {\boldsymbol {c}} \min  _ {\boldsymbol {\beta}, \boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {F e n c h e l}} (\boldsymbol {\beta}, \boldsymbol {z}, \boldsymbol {c}) = \min  _ {\boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {F e n c h e l}} (\boldsymbol {\beta} ^ {*}, \boldsymbol {z}, \boldsymbol {c} ^ {*}) = \min  _ {\boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {s a d d l e}} (\boldsymbol {\gamma} ^ {*}, \boldsymbol {z}) \leq \max  _ {\gamma} \min  _ {\boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {s a d d l e}} (\boldsymbol {\gamma}, \boldsymbol {z})
$$
The derivation for the second equality roughly follows the derivation in the first proof we gave.
For the second inequality, let  $\hat{\gamma}$  be the optimal solution to Problem (38). Let  $\hat{\pmb{c}} = \frac{2}{\lambda_2 + \lambda} (\pmb{X}^T\pmb{y} - \pmb{Q}_\lambda \hat{\gamma})$
Furthermore, let  $\hat{\beta} = \hat{\gamma}$  (notice that this  $\hat{\beta}$  satisfies the optimality condition of Problem (40)). Then we have
$$
\begin{array}{l} \max  _ {\boldsymbol {\gamma}} \min  _ {\boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {s a d d l e}} (\boldsymbol {\gamma}, \boldsymbol {z}) = \min  _ {\boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {s a d d l e}} (\hat {\boldsymbol {\gamma}}, \boldsymbol {z}) \\ = \min  _ {\boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {F e n c h e l}} (\hat {\beta}, \boldsymbol {z}, \hat {\boldsymbol {c}}) \\ = \min  _ {\boldsymbol {\beta}, \boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {F e n c h e l}} (\boldsymbol {\beta}, \boldsymbol {z}, \hat {\boldsymbol {c}}) \\ \leq \max  _ {\boldsymbol {c}} \min  _ {\boldsymbol {\beta}, \boldsymbol {z}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {F e n c h e l}} (\boldsymbol {\beta}, \boldsymbol {z}, \boldsymbol {c}). \\ \end{array}
$$
Similarly, the derivation for the second equality roughly follows the derivation (but this time in reverse order) in our first proof.
Using these two inequalities, we can establish the equivalence of optimal values between Problem (40) and Problem (38), i.e.,  $\max_c\min_{\beta ,z}\mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{Fenchel}}(\beta ,z,c) = \max_\gamma \min_z\mathcal{L}_{\mathrm{ridge} - \lambda}^{\mathrm{saddle}}(\gamma ,z)$ .
$$
h (\boldsymbol {\gamma}) \geq - \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} - \frac {1}{\lambda_ {2} + \lambda} \| \boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} \| _ {2} ^ {2}. \tag {42}
$$
Furthermore, the right-hand size of Equation (42) is maximized if  $\gamma = \hat{\gamma} = \operatorname{argmin}_{\alpha} \mathcal{L}_{ridge}(\alpha)$ , where in this case,  $h(\gamma)$  on the left-hand side of Equation (42) becomes
$$
h (\hat {\boldsymbol {\gamma}}) = \mathcal {L} _ {\mathrm {r i d g e}} (\hat {\boldsymbol {\gamma}}) + (\lambda_ {2} + \lambda) S u m B o t t o m _ {p - k} (\{\hat {\gamma} _ {j} ^ {2})),
$$
Proof. We first derive Inequality (42).
$$
h (\boldsymbol {\gamma}) = \min  _ {\boldsymbol {z} \in \mathcal {D} _ {\boldsymbol {z}}} \mathcal {L} _ {\text {r i d g e} - \lambda} ^ {\text {s a d d l e}} (\boldsymbol {\gamma}, \boldsymbol {z}) \quad \# \mathcal {D} _ {\boldsymbol {z}} := \left\{\boldsymbol {z} \mid \sum_ {j = 1} ^ {p} z _ {j} \leq k, z _ {j} \in [ 0, 1 ] \right\}
$$
$$
= \min  _ {\boldsymbol {z} \in \mathcal {D} _ {\boldsymbol {z}}} - \gamma^ {T} \boldsymbol {Q} _ {\lambda} \gamma - \frac {1}{\lambda_ {2} + \lambda} (\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \gamma) ^ {T} d i a g (\boldsymbol {z}) (\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \gamma)
$$
apply the definition of  $\mathcal{L}_{ridge - \lambda}^{saddle}(\gamma ,z)$
$$
= \min  _ {\boldsymbol {z} \in \mathcal {D} _ {\boldsymbol {z}}} - \gamma^ {T} \boldsymbol {Q} _ {\lambda} \gamma - \frac {1}{\lambda_ {2} + \lambda} \boldsymbol {d} ^ {T} \operatorname {d i a g} (\boldsymbol {z}) \boldsymbol {d} \quad \# \operatorname {l e t} \boldsymbol {d} := \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \gamma\right)
$$
notice that  $z_{j}\in [0,1]$  ; letting  $z_{j} = 1$  for each  $j$  gives us a lower bound
$$
\begin{array}{l} = \min  _ {\boldsymbol {z} \in \mathcal {D} _ {\boldsymbol {z}}} - \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} - \frac {1}{\lambda_ {2} + \lambda} \sum_ {j = 1} ^ {p} d _ {j} ^ {2} z _ {j} \# u s e s u m m a t i o n n o t a t i o n i n s t e a d o f m a t r i x n o t a t i o n \\ \geq - \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} - \frac {1}{\lambda_ {2} + \lambda} \sum_ {j = 1} ^ {p} d _ {j} ^ {2} \\ = - \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} - \frac {1}{\lambda_ {2} + \lambda} \boldsymbol {d} ^ {T} \boldsymbol {d} \quad \# \text {u s e m a t r i x n o t a t i o n i n s t e a d o f s u m m a t i o n n o t a t i o n} \\ = - \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} - \frac {1}{\lambda_ {2} + \lambda} \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma}\right) ^ {T} \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma}\right). \quad \# a p p l y d := \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma}\right) \\ \end{array}
$$
This completes the derivation for Inequality (42).
The right-hand side of Inequality (42) is maximized if the gradient with respect to  $\gamma$  is 0.
The gradient can be derived and simplified as:
$$
\frac {\partial}{\partial \boldsymbol {\gamma}} \left(- \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} - \frac {1}{\lambda_ {2} + \lambda} (\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma}) ^ {T} (\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma})\right)
$$
gradient of the right-hand side of Inequality (42)
$$
= \frac {\partial}{\partial \boldsymbol {\gamma}} \left(- \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} - \frac {1}{\lambda_ {2} + \lambda} (\boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {X} ^ {T} \boldsymbol {y} - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} + \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma})\right)
$$
expand the quadratic term
$$
= \left(- 2 \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} - \frac {1}{\lambda_ {2} + \lambda} \left(- 2 \boldsymbol {Q} _ {\lambda} ^ {T} \boldsymbol {X} ^ {T} \boldsymbol {y} + 2 \boldsymbol {Q} _ {\lambda} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma}\right)\right)
$$
apply the differential operator to each term
$$
= \left(- 2 \boldsymbol {Q} _ {\lambda} ^ {T} \boldsymbol {\gamma} - \frac {1}{\lambda_ {2} + \lambda} \left(- 2 \boldsymbol {Q} _ {\lambda} ^ {T} \boldsymbol {X} ^ {T} \boldsymbol {y} + 2 \boldsymbol {Q} _ {\lambda} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma}\right)\right)
$$
change the first term from  $Q_{\lambda}\gamma$  to  $Q_{\lambda}^{T}\gamma$  because  $Q_{\lambda} = X^T X - \lambda I$  is symmetric
$$
= - \frac {2}{\lambda_ {2} + \lambda} \boldsymbol {Q} _ {\lambda} ^ {T} \left(\left(\lambda_ {2} + \lambda\right) \boldsymbol {\gamma} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma}\right)
$$
pull the common factor  $-\frac{2}{\lambda_2 + \lambda} Q_\lambda^T$  out of the parentheses
$$
= - \frac {2}{\lambda_ {2} + \lambda} \boldsymbol {Q} _ {\lambda} ^ {T} \left(\left(\lambda_ {2} + \lambda\right) \boldsymbol {\gamma} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \boldsymbol {\gamma}\right)
$$
substitute  $Q_{\lambda}^{T} = \boldsymbol{X}^{T}\boldsymbol{X} - \lambda \boldsymbol{I}$  for the last term