SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
= - \frac {2}{\lambda_ {2} + \lambda} \boldsymbol {Q} _ {\lambda} ^ {T} \left(\left(\lambda_ {2} + \lambda\right) \boldsymbol {I} \gamma - \boldsymbol {X} ^ {T} \boldsymbol {y} + \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \gamma\right))
$$
add an identity matrix in the first term
$$
= - \frac {2}{\lambda_ {2} + \lambda} \boldsymbol {Q} _ {\lambda} ^ {T} \left(\left(- \boldsymbol {X} ^ {T} \boldsymbol {y} + \left(\boldsymbol {X} ^ {T} \boldsymbol {X} + \lambda_ {2} \boldsymbol {I}\right) \boldsymbol {\gamma}\right)\right). \quad \# a d d a n i d e n t i t y m a t r i x i n t h e f i r s t t e r m
$$
Therefore, if we set  $(\pmb{X}^T\pmb{X} + \lambda_2\pmb{I})\pmb{\gamma} = \pmb{X}^T\pmb{y}$ , then the gradient of the right-hand side of Inequality (42) will be  $\mathbf{0}$ , and the optimality condition is achieved. However, this means that  $\pmb{\gamma} = (\pmb{X}^T\pmb{X} + \lambda_2\pmb{I})^{-1}\pmb{X}^T\pmb{y}$ , which gives us the minimizer of the ridge regression loss  $\mathcal{L}_{\mathrm{ridge}}(\cdot)$ .
Lastly, we show that when  $\hat{\gamma} = \operatorname{argmin}_{\pmb{\gamma}} \mathcal{L}_{\mathrm{ridge}}(\pmb{\gamma})$ ,  $h(\hat{\gamma}) = \mathcal{L}_{\mathrm{ridge}}(\hat{\gamma}) + (\lambda_2 + \lambda) \text{SumBottom}_{p-k}(\{\hat{\gamma}_j^2\})$ .
First, let us derive a useful property. We claim that when  $\hat{\gamma} = \operatorname{argmin}_{\gamma} \mathcal{L}_{\mathrm{ridge}}(\gamma), \boldsymbol{X}^T \boldsymbol{y} - \boldsymbol{Q}_\lambda \hat{\gamma} = (\lambda_2 + \lambda) \hat{\gamma}$ . Below, we prove this by showing that when we subtract the two terms, we get  $\mathbf{0}$ .
$$
\begin{array}{l} \boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \hat {\gamma} - (\lambda_ {2} + \lambda) \hat {\gamma} \\ = \boldsymbol {X} ^ {T} \boldsymbol {y} - \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \hat {\gamma} - (\lambda_ {2} + \lambda) \hat {\gamma} \quad \# \text {s u b s t i t u t e} Q _ {\lambda} = \boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I} \\ = \boldsymbol {X} ^ {T} \boldsymbol {y} - \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \hat {\gamma} - (\lambda_ {2} + \lambda) \boldsymbol {I} \hat {\gamma} \quad \# a d d t h e i d e n t i t y m a t r i x \boldsymbol {I} \\ = \boldsymbol {X} ^ {T} \boldsymbol {y} - \left(\boldsymbol {X} ^ {T} \boldsymbol {X} + \lambda_ {2} \boldsymbol {I}\right) \hat {\gamma} \quad \# a g g r a g e t h e r e l e v a n t t e r m s i n f r o n t o f \hat {\gamma} \\ \end{array}
$$
$$
\begin{array}{l} = \boldsymbol {X} ^ {T} \boldsymbol {y} - \left(\boldsymbol {X} ^ {T} \boldsymbol {X} + \lambda_ {2} \boldsymbol {I}\right) \left(\boldsymbol {X} ^ {T} \boldsymbol {X} + \lambda_ {2} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {X} ^ {T} \boldsymbol {y} \quad \# \hat {\gamma} = \operatorname {a r g m i n} _ {\gamma} \mathcal {L} _ {\text {r i d g e}} (\gamma) = \boldsymbol {X} ^ {T} \boldsymbol {X} + \lambda_ {2} \boldsymbol {I} \\ = \boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {X} ^ {T} \boldsymbol {y} \\ = 0. \\ \end{array}
$$
Therefore, we have  $\pmb{X}^T\pmb{y} - \pmb{Q}_{\lambda}\hat{\gamma} = (\lambda_2 + \lambda)\hat{\gamma}$  when  $\hat{\gamma} = \mathrm{argmin}_{\pmb{\gamma}}\mathcal{L}_{\mathrm{ridge}}(\pmb{\gamma})$ . With this property, we can derive the final simplified formula for  $h(\hat{\gamma})$ :
$$
h(\hat{\boldsymbol{\gamma}}) = \min_{\boldsymbol {z}\in \mathcal{D}_{\boldsymbol{z}}}\mathcal{L}_{\text{ridge} -\lambda}^{\text{saddle}}(\hat{\boldsymbol{\gamma}},\boldsymbol {z})
$$
$$
\# \mathcal {D} _ {z} := \left\{z \mid \sum_ {j = 1} ^ {p} z _ {j} \leq k, z _ {j} \in [ 0, 1 ] \right\}
$$
$$
= \min  _ {\boldsymbol {z} \in \mathcal {D} _ {\boldsymbol {z}}} - \hat {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \hat {\gamma} - \frac {1}{\lambda_ {2} + \lambda} \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \hat {\gamma}\right) ^ {T} d i a g (\boldsymbol {z}) \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {Q} _ {\lambda} \hat {\gamma}\right)
$$
apply the definition of  $\mathcal{L}_{\text{ridge} - \lambda}^{\text{saddle}}(\hat{\gamma}, z)$
$$
= \min  _ {\boldsymbol {z} \in \mathcal {D} _ {\boldsymbol {z}}} - \hat {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \hat {\gamma} - \frac {1}{\lambda_ {2} + \lambda} \left(\lambda_ {2} + \lambda\right) \hat {\gamma} ^ {T} d i a g (\boldsymbol {z}) \left(\lambda_ {2} + \lambda\right) \hat {\gamma}
$$
apply the property  $X^T\mathbf{y} - Q_\lambda \hat{\gamma} = (\lambda_2 + \lambda)\hat{\gamma}$
$$
= \min  _ {\boldsymbol {z} \in \mathcal {D} _ {\boldsymbol {z}}} - \hat {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \hat {\gamma} - (\lambda_ {2} + \lambda) \hat {\gamma} ^ {T} d i a g (\boldsymbol {z}) \hat {\gamma}
$$
cancel out  $\lambda_{2} + \lambda$
$$
= \min  _ {\boldsymbol {z} \in \mathcal {D} _ {\boldsymbol {z}}} - \hat {\gamma} ^ {T} \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \hat {\gamma} - (\lambda_ {2} + \lambda) \hat {\gamma} ^ {T} d i a g (\boldsymbol {z}) \hat {\gamma}
$$
apply  $Q_{\lambda} = X^T X - \lambda I$
$$
= \min  _ {\boldsymbol {z} \in \mathcal {D} _ {\boldsymbol {z}}} - \hat {\gamma} ^ {T} (\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}) \hat {\gamma} - (\lambda_ {2} + \lambda) \sum_ {j = 1} ^ {p} \hat {\gamma} _ {j} ^ {2} z _ {j}
$$
use summation instead of matrix notation
$$
= - \hat {\gamma} ^ {T} \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \hat {\gamma} - (\lambda_ {2} + \lambda) \max  _ {\boldsymbol {z} \in \mathcal {D} _ {\boldsymbol {z}}} \sum_ {j = 1} ^ {p} \hat {\gamma} _ {j} ^ {2} z _ {j}
$$
move  $\min (\cdot)$  inside the negative sign and convert it to  $\max (\cdot)$
$$
= - \hat {\gamma} ^ {T} \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \hat {\gamma} - (\lambda_ {2} + \lambda) \operatorname {T o p S u m} _ {k} \left\{\hat {\gamma} _ {j} ^ {2} \right\}
$$
TopSum $_k(\cdot)$  denotes summation over the top  $k$  terms
$$
= - \hat {\gamma} ^ {T} \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \hat {\gamma} - (\lambda_ {2} + \lambda) \left(\sum_ {j = 1} ^ {p} \hat {\gamma} _ {j} ^ {2} - B o t t o m S u m _ {p - k} \left\{\hat {\gamma} _ {j} ^ {2} \right\}\right)
$$
BottomSum $_{p-k}(\cdot)$  denotes summation over the bottom  $p-k$  terms
$$
= - \hat {\gamma} ^ {T} (\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}) \hat {\gamma} + (\lambda_ {2} + \lambda) B o t t o m S u m _ {p - k} \left\{\hat {\gamma} _ {j} ^ {2} \right\} - (\lambda_ {2} + \lambda) \sum_ {j = 1} ^ {p} \hat {\gamma} _ {j} ^ {2}
$$
distribute into the parentheses
$$
= - \hat {\gamma} ^ {T} \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \hat {\gamma} + (\lambda_ {2} + \lambda) B o t t o m S u m _ {p - k} \left\{\hat {\gamma} _ {j} ^ {2} \right\} - (\lambda_ {2} + \lambda) \hat {\gamma} ^ {T} \hat {\gamma}.
$$
use matrix notation instead of summation notation
We note that  $-\hat{\gamma}^T(\boldsymbol{X}^T\boldsymbol{X} - \lambda \boldsymbol{I})\hat{\gamma} - (\lambda_2 + \lambda)\hat{\gamma}^T\hat{\gamma} = \mathcal{L}_{\mathrm{ridge}}(\hat{\gamma})$ . To show this, we subtract the two terms and prove that the subtraction is equal to 0 below:
$$
- \hat {\gamma} ^ {T} \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \hat {\gamma} - (\lambda_ {2} + \lambda) \hat {\gamma} ^ {T} \hat {\gamma} - \mathcal {L} _ {\text {r i d g e}} (\hat {\gamma})
$$
$$
= - \hat {\gamma} ^ {T} \left(\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}\right) \hat {\gamma} - (\lambda_ {2} + \lambda) \hat {\gamma} ^ {T} \hat {\gamma} - \left(\hat {\gamma} ^ {T} \boldsymbol {X} ^ {T} \boldsymbol {X} \hat {\gamma} - 2 \boldsymbol {y} ^ {T} \boldsymbol {X} \hat {\gamma} + \lambda_ {2} \hat {\gamma} ^ {T} \hat {\gamma}\right)
$$
plug in the formula for  $\mathcal{L}_{\mathrm{ridge}}(\hat{\gamma})$
$$
= - 2 \left(\hat {\gamma} ^ {T} \left(\boldsymbol {X} ^ {T} \boldsymbol {X} + \lambda_ {2} \boldsymbol {I}\right) \hat {\gamma} - \boldsymbol {y} ^ {T} \boldsymbol {X} \hat {\gamma}\right)
$$
$$
= - 2 \left(\hat {\gamma} ^ {T} \left(\boldsymbol {X} ^ {T} \boldsymbol {X} + \lambda_ {2} \boldsymbol {I}\right) \left(\boldsymbol {X} ^ {T} \boldsymbol {X} + \lambda_ {2} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {y} ^ {T} \boldsymbol {X} \hat {\gamma}\right)
$$
plug in the equation  $\hat{\gamma} = (X^T X + \lambda_2 I)^{-1} X^T y$  because  $\hat{\gamma}$  is the minimizer of  $\mathcal{L}_{\mathrm{ridge}}(\cdot)$ .
$$
= - 2 \left(\hat {\gamma} ^ {T} \boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {y} ^ {T} \boldsymbol {X} \hat {\gamma}\right)
$$
cancel out  $X^T X + \lambda_2 I$
$$
= 0.
$$
Using the property that  $-\hat{\gamma}^T(\boldsymbol{X}^T\boldsymbol{X} - \lambda \boldsymbol{I})\hat{\gamma} - (\lambda_2 + \lambda)\hat{\gamma}^T\hat{\gamma} = \mathcal{L}_{\mathrm{ridge}}(\hat{\gamma})$ , we have:
$$
h (\hat {\boldsymbol {\gamma}}) = - \hat {\boldsymbol {\gamma}} ^ {T} (\boldsymbol {X} ^ {T} \boldsymbol {X} - \lambda \boldsymbol {I}) \hat {\boldsymbol {\gamma}} - (\lambda_ {2} + \lambda) \hat {\boldsymbol {\gamma}} ^ {T} \hat {\boldsymbol {\gamma}} + (\lambda_ {2} + \lambda) B o t t o m S u m _ {p - k} \{\hat {\gamma} _ {j} ^ {2} \}
$$
$$
= \mathcal {L} _ {\text {r i d g e}} (\hat {\boldsymbol {\gamma}}) + (\lambda_ {2} + \lambda) \text {B o t t o m S u m} _ {p - k} \{\hat {\gamma} _ {j} ^ {2} \}.
$$
This completes the proof for Theorem 3.2.
C.3 Proof of Theorem 3.3
Theorem 3.3. Let  $F(\pmb{\gamma}) = \pmb{\gamma}^T\pmb{Q}_{\lambda}\pmb{\gamma}$  and  $G(\pmb{p}) = \frac{1}{\lambda_2 + \lambda} SumTop_k(\{p_j^2\})$ . Then the solution for the problem  $\pmb{\gamma}^{t+1} = \operatorname*{argmin}_{\pmb{\gamma}} F(\pmb{\gamma}) + \frac{\rho}{2} \| \pmb{Q}_{\lambda}\pmb{\gamma} + \pmb{p}^t - \pmb{X}^T\pmb{y} + \pmb{q}^t \|_2^2$  is
$$
\boldsymbol {\gamma} ^ {t + 1} = \left(\frac {2}{\rho} \boldsymbol {I} + \boldsymbol {Q} _ {\lambda}\right) ^ {- 1} \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {p} ^ {t} - \boldsymbol {q} ^ {t}\right). \tag {43}
$$
Furthermore, let  $\mathbf{a} = \mathbf{X}^T\mathbf{y} - \pmb{\theta}^{t + 1} - \pmb{q}^t$  and  $\mathcal{I}$  be the indices of the top  $k$  terms of  $\{|a_j|\}$ . The solution for the problem  $\pmb{p}^{t + 1} = \mathrm{argmin}_{\pmb{p}}G(\pmb {p}) + \frac{\rho}{2}\| \pmb{\theta}^{t + 1} + \pmb {p} - \pmb{X}^T\pmb {y} + \pmb{q}^t\| _2^2$  is  $p_j^{t + 1} = sign(a_j)\cdot \hat{v}_j$
where  $\hat{\pmb{v}} = \underset {\pmb{v}}{\mathrm{argmin}}\sum_{j = 1}^{p}w_{j}(v_{j} - b_{j})^{2}$  s.t.  $v_{i}\geq v_{l}$  if  $|a_{i}|\geq |a_{l}|$  (44)
Problem (44) is an isotonic regression problem and can be efficiently solved in linear time [12, 21].
Proof. Part I. We first show why Equation (43) is the analytic solution to the first optimization problem  $\gamma^{t + 1} = \mathrm{argmin}_{\gamma}F(\gamma) + \frac{\rho}{2}\| Q_{\lambda}\gamma +\pmb{p}^t -\pmb{X}^T\pmb {y} + \pmb{q}^t\| _2^2$
The loss function we want to minimize is (writing out  $F(\pmb {\gamma}) = \pmb{\gamma}^{T}\pmb{Q}_{\lambda}\pmb{\gamma}$  explicitly)
$$
\begin{array}{l} \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} + \frac {\rho}{2} \| \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} + \boldsymbol {p} ^ {t} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {q} ^ {t} \| _ {2} ^ {2} \\ = \boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} + \frac {\rho}{2} \left(\boldsymbol {\gamma} ^ {T} \boldsymbol {Q} _ {\lambda} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} + 2 \left(\boldsymbol {p} ^ {t} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {q} ^ {t}\right) ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} + \| \boldsymbol {p} ^ {t} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {q} ^ {t} \| _ {2} ^ {2}\right). \\ \end{array}
$$
write out the square term as a sum of three terms
This loss function is minimized if we take the gradient with respect to  $\gamma$  and set it to 0:
$$
\mathbf {0} = 2 \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} + \frac {\rho}{2} \left(2 \boldsymbol {Q} _ {\lambda} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} + 2 \boldsymbol {Q} _ {\lambda} ^ {T} \left(\boldsymbol {p} ^ {t} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {q} ^ {t}\right)\right)
$$
take the gradient w.r.t.  $\gamma$  and set it to 0; ignore the last term because it is a constant
$$
\begin{array}{l} = 2 Q _ {\lambda} \boldsymbol {\gamma} + \rho \left(Q _ {\lambda} ^ {T} Q _ {\lambda} \boldsymbol {\gamma} + Q _ {\lambda} ^ {T} \left(\boldsymbol {p} ^ {t} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {q} ^ {t}\right)\right) \quad \# d i s t r i b u t e 2 i n s i d e p a r e n t h e s e s \\ = 2 \boldsymbol {Q} _ {\lambda} ^ {T} \boldsymbol {\gamma} + \rho (\boldsymbol {Q} _ {\lambda} ^ {T} \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} + \boldsymbol {Q} _ {\lambda} ^ {T} (\boldsymbol {p} ^ {t} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {q} ^ {t})) \quad \# \boldsymbol {Q} _ {\lambda} i s s y m m e t r i c \\ \end{array}
$$
$$
\begin{array}{l} = \boldsymbol {Q} _ {\lambda} ^ {T} (2 \boldsymbol {\gamma} + \rho (\boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} + (\boldsymbol {p} ^ {t} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {q} ^ {t}))) \quad \# t a k e o u t t h e c o m m o n f a c t o r \boldsymbol {Q} _ {\lambda} ^ {T} \\ = \boldsymbol {Q} _ {\lambda} ^ {T} (2 \boldsymbol {I} \boldsymbol {\gamma} + \rho \boldsymbol {Q} _ {\lambda} \boldsymbol {\gamma} + \rho (\boldsymbol {p} ^ {t} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {q} ^ {t})) \quad \# m u l t i p l y b y i d e n t i t y m a t r i x \boldsymbol {I} \\ = \boldsymbol {Q} _ {\lambda} ^ {T} \left(\left(2 \boldsymbol {I} + \rho \boldsymbol {Q} _ {\lambda}\right) \boldsymbol {\gamma} + \rho \left(\boldsymbol {p} ^ {t} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {q} ^ {t}\right)\right) \quad \# a g g r e g a t e (\text {t h e c o e f f i c i e n t f o r} \gamma \\ = \rho Q _ {\lambda} ^ {T} \left(\left(\frac {2}{\rho} \boldsymbol {I} + \boldsymbol {Q} _ {\lambda}\right) \gamma + \left(\boldsymbol {p} ^ {t} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {q} ^ {t}\right)\right). \quad \# t a k e o u t t h e c o m m o n f a c t o r \rho \\ \end{array}
$$
A sufficient condition for the gradient to be 0 is to have
$$
\left(\frac {2}{\rho} \boldsymbol {I} + \boldsymbol {Q} _ {\lambda}\right) \boldsymbol {\gamma} + \left(\boldsymbol {p} ^ {t} - \boldsymbol {X} ^ {T} \boldsymbol {y} + \boldsymbol {q} ^ {t}\right) = \mathbf {0}.
$$
If we solve the above equation, we obtain Equation (43):
$$
\boldsymbol {\gamma} = \left(\frac {2}{\rho} \boldsymbol {I} + \boldsymbol {Q} _ {\lambda}\right) ^ {- 1} \left(\boldsymbol {X} ^ {T} \boldsymbol {y} - \boldsymbol {p} ^ {t} - \boldsymbol {q} ^ {t}\right).
$$
**Part II.** Next, we show why  $p_j^{t+1} = \text{sign}(a_j) \cdot \hat{v}_j$  together with Equation (44) gives us the solution to the second optimization problem  $\pmb{p}^{t+1} = \operatorname{argmin}_{\pmb{p}} G(\pmb{p}) + \frac{\rho}{2} \| \pmb{\theta}^{t+1} + \pmb{p} - \pmb{X}^T \pmb{y} + \pmb{q}^t \|_2^2$ .
If we write out  $G(\pmb{p}) = \frac{1}{\lambda_2 + \lambda} \mathrm{SumTop}_k(\{p_j^2\})$  explicitly and use  $\pmb{a} = \pmb{X}^T \pmb{y} - \pmb{\theta}^{t+1} - \pmb{q}^t$  as a shorthand to represent the constant, the objective function can be rewritten as:
$$
\begin{array}{l} \boldsymbol {p} ^ {t + 1} = \operatorname * {a r g m i n} _ {\boldsymbol {p}} \frac {1}{\lambda_ {2} + \lambda} \operatorname {S u m T o p} _ {k} \left(\left\{p _ {j} ^ {2} \right\}\right) + \frac {\rho}{2} \| \boldsymbol {p} - \boldsymbol {a} \| _ {2} ^ {2} \\ = \operatorname * {a r g m i n} _ {\boldsymbol {p}} \frac {2}{\rho (\lambda_ {2} + \lambda)} \operatorname {S u m T o p} _ {k} \left(\left\{p _ {j} ^ {2} \right\}\right) + \| \boldsymbol {p} - \boldsymbol {a} \| _ {2} ^ {2}. \\ \end{array}
$$
# multiplying by  $\frac{2}{\rho}$  doesn't change the optimal solution
According to Proposition 3.1 (b) in [30], the optimal solution  $\pmb{p}^{t + 1}$  has the property that  $\mathrm{sign}(p_j^{t + 1}) = \mathrm{sign}(a_j)$ .
Inspired by this sign-preserving property, let us make a change of variable by writing  $p_j = \mathrm{sign}(a_j) \cdot v_j$ . Due to this change of variable, our optimal solution  $\pmb{p}^{t + 1}$  can be obtained by solving the following optimization problem:
$$
\begin{array}{l} p _ {j} ^ {t + 1} = \operatorname {s i g n} (a _ {j}) \cdot \hat {v _ {j}} \\ \text {w h e r e} \hat {\boldsymbol {v}} = \underset {\boldsymbol {v}} {\operatorname {a r g m i n}} \frac {2}{\rho (\lambda_ {2} + \lambda)} \operatorname {S u m T o p} _ {k} \left(\left\{v _ {j} ^ {2} \right\}\right) + \sum_ {j = 1} ^ {p} \left(v _ {j} - \left| a _ {j} \right|\right) ^ {2}. \tag {45} \\ \end{array}
$$
Note that in Problem (45), if we add the extra constraints  $v_{j} \geq 0$ , this will not change the optimal solution. We can show this by the argument of contradiction. Suppose there exists an optimal solution  $\hat{v}$  and an index  $j$  with  $\hat{v}_{j} < 0$ . Then, if we flip the sign of  $\hat{v}_{j}$ , then the objective loss in Problem (45) is decreased. This contradicts with the assumption that  $\hat{v}$  is the optimal solution. Therefore, we are safe to add the extra constraints  $v_{j} \geq 0$  into Problem (45) without cutting off the optimal solution.
Thus, our optimal solution  $\pmb{p}^{t + 1}$  becomes:
$$
p _ {j} ^ {t + 1} = \operatorname {s i g n} \left(a _ {j}\right) \cdot \hat {v} _ {j}
$$
$$
\text {w h e r e} \hat {\boldsymbol {v}} = \underset {\boldsymbol {v}} {\operatorname {a r g m i n}} \frac {2}{\rho (\lambda_ {2} + \lambda)} \operatorname {S u m T o p} _ {k} \left(\left\{v _ {j} ^ {2} \right\}\right) + \sum_ {j = 1} ^ {p} \left(v _ {j} - \left| a _ {j} \right|\right) ^ {2} \text {s . t .} v _ {j} \geq 0 \text {f o r} \forall j. \tag {46}
$$
Next, according to Proposition 3.1 (c) in [30], we have  $\hat{v}_i \geq \hat{v}_l$  if  $|a_i| \geq |a_l|$  in Problem (46). Thus, we add these extra constraints into Problem (46) without changing the optimal solution and  $\pmb{p}^{t+1}$  becomes:
$$
\text {w h e r e} \hat {\boldsymbol {v}} = \underset {\boldsymbol {v}} {\operatorname {a r g m i n}} \frac {2}{\rho (\lambda_ {2} + \lambda)} \operatorname {S u m T o p} _ {k} \left(\left\{v _ {j} ^ {2} \right\}\right) + \sum_ {j = 1} ^ {p} \left(v _ {j} - \left| a _ {j} \right|\right) ^ {2} \tag {47}
$$
s.t.  $v_{j} \geq 0$  for  $\forall j$ , and  $v_{i} \geq v_{j}$  if  $|a_{i}| \geq |a_{j}|$ . For  $\forall i, j$ .
Note that in Problem (47), because of the extra constraints  $|a_i| \geq |a_j|$  for  $\forall i, j$ , we can rewrite  $\mathrm{SumTop}_k(\{v_j^2\}) = \sum_{j \in \mathcal{J}} v_j^2$  where  $\mathcal{J}$  is the set of indices of the top  $k$  terms of  $\{|a_j|\}$ 's. Then the optimal solution  $p^{t+1}$  becomes:
$$
p _ {j} ^ {t + 1} = \operatorname {s i g n} (a _ {j}) \cdot \hat {v _ {j}}
$$
$$
\text {w h e r e} \hat {\boldsymbol {v}} = \underset {\boldsymbol {v}} {\operatorname {a r g m i n}} \frac {2}{\rho (\lambda_ {2} + \lambda)} \sum_ {j \in \mathcal {J}} v _ {j} ^ {2} + \sum_ {j = 1} ^ {p} \left(v _ {j} - \left| a _ {j} \right|\right) ^ {2} \tag {48}
$$
s.t.  $v_{j} \geq 0$  for  $\forall j$ , and  $v_{i} \geq v_{j}$  if  $|a_{i}| \geq |a_{j}|$ . for  $\forall i, j$ .
Next, we drop the constraints  $v_{j} \geq 0$  for  $\forall j$  as this does not change the optimal solution, using the same argument of contradiction we have shown before. Therefore, our optimal solution  $\pmb{p}^{t + 1}$  can be obtained in the following way:
$$
p _ {j} ^ {t + 1} = \operatorname {s i g n} (a _ {j}) \cdot \hat {v} _ {j}
$$
$$
\text {w h e r e} \hat {\boldsymbol {v}} = \underset {\boldsymbol {v}} {\operatorname {a r g m i n}} \frac {2}{\rho \left(\lambda_ {2} + \lambda\right)} \sum_ {j \in \mathcal {J}} v _ {j} ^ {2} + \sum_ {j = 1} ^ {p} \left(v _ {j} - \left| a _ {j} \right|\right) ^ {2} \tag {49}
$$
s.t.  $v_{i} \geq v_{j}$  if  $|a_{i}| \geq |a_{j}|$ . for  $\forall i, j$ .
We can obtain the formulation in Problem (44) by performing some linear algebra manipulations as follows:
$$
\underset {\boldsymbol {v}} {\operatorname {a r g m i n}} \frac {2}{\rho (\lambda_ {2} + \lambda)} \sum_ {j \in \mathcal {J}} v _ {j} ^ {2} + \sum_ {j = 1} ^ {p} (v _ {j} - | a _ {j} |) ^ {2}
$$
$$
= \operatorname * {a r g m i n} _ {\boldsymbol {v}} \sum_ {j \in \mathcal {J}} \left(\frac {2}{\rho (\lambda_ {2} + \lambda)} v _ {j} ^ {2} + (v _ {j} - | a _ {j} |) ^ {2}\right) + \sum_ {j \notin \mathcal {J}} (v _ {j} - | a _ {j} |) ^ {2}
$$
divide the summation into two groups based on the set  $\mathcal{I}$
$$
= \operatorname {a r g m i n} _ {\boldsymbol {v}} \sum_ {j \in \mathcal {J}} \left(\left(\frac {2}{\rho \left(\lambda_ {2} + \lambda\right)} + 1\right) v _ {j} ^ {2} - 2 \left| a _ {j} \right| v _ {j} + \left| a _ {j} \right| ^ {2}\right) + \sum_ {j \notin \mathcal {J}} \left(v _ {j} - \left| a _ {j} \right|\right) ^ {2}
$$
combine the coefficients for  $v_{j}^{2}$
$$
= \operatorname {a r g m i n} _ {\boldsymbol {v}} \sum_ {j \in \mathcal {J}} \left(\left(\frac {2}{\rho (\lambda_ {2} + \lambda)} + 1\right) \left(v _ {j} - \frac {| a _ {j} |}{\frac {2}{\rho (\lambda_ {2} + \lambda)} + 1}\right) ^ {2} - \frac {| a _ {j} | ^ {2}}{\frac {2}{\rho (\lambda_ {2} + \lambda)} + 1} + | a _ {j} | ^ {2}\right) + \sum_ {j \notin \mathcal {J}} (v _ {j} - | a _ {j} |) ^ {2}
$$
complete the square
$$
= \operatorname * {a r g m i n} _ {\boldsymbol {v}} \sum_ {j \in \mathcal {J}} \left(\left(\frac {2}{\rho (\lambda_ {2} + \lambda)} + 1\right) \left(v _ {j} - \frac {| a _ {j} |}{\frac {2}{\rho (\lambda_ {2} + \lambda)} + 1}\right) ^ {2}\right) + \sum_ {j \notin \mathcal {J}} (v _ {j} - | a _ {j} |) ^ {2}
$$
# get rid of the constant terms which do not affect the optimal solution
$$
= \operatorname * {a r g m i n} _ {\boldsymbol {v}} \sum_ {j \in \mathcal {J}} \left(w _ {j} \left(v _ {j} - \frac {| a _ {j} |}{w _ {j}}\right) ^ {2}\right) + \sum_ {j \notin \mathcal {J}} w _ {j} \left(v _ {j} - \frac {| a _ {j} |}{w _ {j}}\right) ^ {2}
$$
$$
\# w _ {j} = \frac {2}{\rho (\lambda_ {2} + \lambda)} + 1 i f j \in \mathcal {J} a n d w _ {j} = 1 i f j \notin \mathcal {J}
$$
$$
= \operatorname * {a r g m i n} _ {\boldsymbol {v}} \sum_ {j = 1} ^ {p} w _ {j} (v _ {j} - b _ {j}) ^ {2}.
$$
let  $b_{j} = \frac{|a_{j}|}{w_{j}}$  and combine the two summations into a single summation
Using this new property, we can obtain the final result stated in the theorem for our optimal solution  $\pmb{p}^{t + 1}$  to the second optimization problem:
$$
\text {w h e r e} \hat {\boldsymbol {v}} = \underset {\boldsymbol {v}} {\operatorname {a r g m i n}} \sum_ {j = 1} ^ {p} w _ {j} \left(v _ {j} - b _ {j}\right) ^ {2} \text {s . t .} v _ {i} \geq v _ {j} \text {i f} \left| a _ {i} \right| \geq \left| a _ {j} \right| \text {f o r} \forall i, j \tag {50}
$$
$$
w _ {j} = \left\{ \begin{array}{l l} 1 & \text {i f} j \notin \mathcal {J} \\ \frac {2}{\rho (\lambda_ {2} + \lambda)} + 1 & \text {o t h e r w i s e} \end{array} , \right. \quad b _ {j} = \frac {| a _ {j} |}{w _ {j}}.
$$
Therefore, the task of evaluating the proximal operator is essentially reduced to solving an isotonic regression problem. We would like to acknowledge that the connection between  $SumTop_{k}(\cdot)$  and isotonic regression has also been recently discovered by [55]. However, the context and applications are totally different. While [55] discovers this relationship in the context of training neural networks, we find this connection in the context of ADMM and perspective relaxation of k-sparse ridge regression. Mathematically, our objective function is also different as our objective function contains a linear term of  $p$  in the proximal operator evaluation.
C.4 Proof of Theorem 3.4