SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \langle \nabla M _ {g, \gamma} (\boldsymbol {x}) - \nabla M _ {g, \gamma} (\boldsymbol {y}), \boldsymbol {x} - \boldsymbol {y} \rangle \\ = \gamma^ {- 1} \left\langle \boldsymbol {x} - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x}) - \boldsymbol {y} + \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y}), \boldsymbol {x} - \boldsymbol {y} \right\rangle \\ = \gamma^ {- 1} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2} + \gamma^ {- 1} \left\langle \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y}) - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x}), \boldsymbol {x} - \boldsymbol {y} \right\rangle \\ \geq \gamma^ {- 1} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2} - \frac {1}{2 \gamma} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2} - \frac {1}{2 \gamma} \| \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x}) - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y}) \| ^ {2} \\ \geq \frac {1}{2 \gamma} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2} - \frac {3}{2 \gamma} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2} \\ = - \frac {1}{\gamma} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2}, \\ \end{array}
$$
where the first equality follows from (8) and the last inequality follows from Lemma 4. Thus  $g$  is  $\gamma^{-1}$ -weakly convex by Lemma 3.
A.2 Proofs for ADMM in Section 3
A.2.1 Proof of Proposition 1
Proof: First we define  $h_{[a,b]}(z) = \sum_{i=a}^{b} \theta_i(z)$ . Due to convexity, we have
$$
0 \in \partial h _ {[ m, n ]} (v _ {[ m, n ]}) \quad \text {a n d} \quad 0 \in \partial h _ {[ n + 1, p ]} (v _ {[ n + 1, p ]}).
$$
Noting that  $v_{[m,n]} > v_{[n+1,p]}$ , due to the strong convexity of  $h$ , we further have
$$
\partial h _ {[ m, n ]} (v _ {[ n + 1, p ]}) <   0, \quad \text {a n d} \quad \partial h _ {[ n + 1, p ]} (v _ {[ m, n ]}) > 0,
$$
where we use the convention that a set  $A > 0$  (or  $< 0$ ) denotes that all elements  $g \in A$  satisfies  $g > 0$  (or  $< 0$ ). Therefore we obtain that there exist  $s_{[m,n]}$ ,  $s_{[n+1,p]} \in \mathbb{R}$  such that
$$
\partial h _ {[ m, p ]} (v _ {[ m, n ]}) = \partial h _ {[ m, n ]} (v _ {[ m, n ]}) + \partial h _ {[ n + 1, p ]} (v _ {[ m, n ]}) \ni s _ {[ m, n ]} > 0,
$$
$$
\partial h _ {[ m, p ]} (v _ {[ n + 1, p ]}) = \partial h _ {[ m, n ]} (v _ {[ n + 1, p ]}) + \partial h _ {[ n + 1, p ]} (v _ {[ n + 1, p ]}) \ni s _ {[ n + 1, p ]} <   0.
$$
As  $h_{[m,p]}$  is strongly convex in  $[v_{[n+1,p]}, v_{[m,n]}]$ , the subgradient  $\partial h_{[m,p]}$  is strictly increasing in  $[v_{[n+1,p]}, v_{[m,n]}]$ . That is,  $\forall x, y \in [v_{[n+1,p]}, v_{[m,n]}], \forall s_x \in \partial h_{[m,p]}(x), s_y \in \partial h_{[m,p]}(y), (s_x - s_y)(x - y) \geq 0$ . The above facts, together with the strong convexity of  $h_{[m,p]}$ , the unique minimizer  $v_{[m,p]}$  of  $h_{[m,p]}$  must lie in the interval  $[v_{[n+1,p]}, v_{[m,n]}]$ .
A.2.2 Proof of Theorem 1
Proof: We use the extended formula for Clark generalized gradient of a sum of two functions in our proofs:  $\partial (f_1 + f_2)(\pmb{x}) \subset \partial f_1(\pmb{x}) + \partial f_2(\pmb{x})$  if  $f_1$  and  $f_2$  are finite at  $\pmb{x}$  and  $f_2$  is differentiable at  $\pmb{x}$ . The equality holds if  $f_1$  is regular at  $\pmb{x}$ . [10, Theorem 2.9.8].
By Assumption 3, we have
$$
\begin{array}{l} \mathbf {0} \in \partial_ {\mathbf {z}} \left(\Omega \left(\mathbf {z} ^ {k + 1}\right) + \frac {\rho}{2} | | \mathbf {z} ^ {k + 1} - D \mathbf {w} ^ {k} + \frac {\boldsymbol {\lambda} ^ {k}}{\rho} | | ^ {2}\right) \\ \subset \partial \Omega \left(\boldsymbol {z} ^ {k + 1}\right) + \boldsymbol {\lambda} ^ {k} + \rho \left(\boldsymbol {z} ^ {k + 1} - D \boldsymbol {w} ^ {k}\right) \tag {15} \\ = \partial \Omega (\boldsymbol {z} ^ {k + 1}) + \boldsymbol {\lambda} ^ {k} + \rho (\boldsymbol {z} ^ {k + 1} - D \boldsymbol {w} ^ {k + 1}) + \rho D (\boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k}) \\ = \partial \Omega (\boldsymbol {z} ^ {k + 1}) + \boldsymbol {\lambda} ^ {k + 1} + \rho D (\boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k}), \\ \end{array}
$$
$$
L _ {\rho} \left(\boldsymbol {z} ^ {k}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}\right) - L _ {\rho} \left(\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}\right) \geq 0. \tag {16}
$$
Here the last equality of (15) follows from  $\pmb{\lambda}^{k + 1} = \pmb{\lambda}^k +\rho (\pmb {z}^{k + 1} - D\pmb{w}^{k + 1})$
By Assumption 3 and  $(r - c)$ -strong convexity of the  $w$ -subproblem, we have
$$
\begin{array}{l} \mathbf {0} \in \partial_ {\boldsymbol {w}} \left(\frac {\rho}{2} \| \boldsymbol {z} ^ {k + 1} - D \boldsymbol {w} ^ {k + 1} + \frac {\boldsymbol {\lambda} ^ {k}}{\rho} \| ^ {2} + g (\boldsymbol {w} ^ {k + 1}) + \frac {r}{2} \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2}\right) \tag {17} \\ = \partial g (\boldsymbol {w} ^ {k + 1}) - D ^ {\top} \boldsymbol {\lambda} ^ {k} - \rho D ^ {\top} \left(\boldsymbol {z} ^ {k + 1} - D \boldsymbol {w} ^ {k + 1}\right) + r \left(\boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k}\right) \\ = \partial g (\boldsymbol {w} ^ {k + 1}) - D ^ {\top} \boldsymbol {\lambda} ^ {k + 1} + r (\boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k}), \\ \end{array}
$$
$$
L _ {\rho} \left(\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}\right) - L _ {\rho} \left(\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k + 1}, \boldsymbol {\lambda} ^ {k}\right) \geq \frac {2 r - c}{2} \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2}. \tag {18}
$$
The second equality of (17) is due to the fact that weakly convex functions are regular [47, Proposition 4.5]. By the dual update we have
$$
\begin{array}{l} L _ {\rho} (\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k + 1}, \boldsymbol {\lambda} ^ {k}) - L _ {\rho} (\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k + 1}, \boldsymbol {\lambda} ^ {k + 1}) \\ = \left(\boldsymbol {\lambda} ^ {k} - \boldsymbol {\lambda} ^ {k + 1}\right) ^ {\top} \left(\boldsymbol {z} ^ {k + 1} - D \boldsymbol {w} ^ {k + 1}\right) \tag {19} \\ = - \frac {1}{\rho} \| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \| ^ {2}. \\ \end{array}
$$
Summing (16), (18) and (19) we obtain that
$$
L _ {\rho} \left(\boldsymbol {z} ^ {k}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}\right) - L _ {\rho} \left(\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k + 1}, \boldsymbol {\lambda} ^ {k + 1}\right) \geq \frac {2 r - c}{2} \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2} - \frac {1}{\rho} \| \boldsymbol {\lambda} ^ {k} - \boldsymbol {\lambda} ^ {k + 1} \| ^ {2}. \tag {20}
$$
By Assumptions 1 and 2 we have that
$$
\begin{array}{l} L _ {\rho} \left(\boldsymbol {z} ^ {k}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}\right) = \Omega \left(\boldsymbol {z} ^ {k}\right) + g \left(\boldsymbol {w} ^ {k}\right) + \left(\boldsymbol {\lambda} ^ {k}\right) ^ {\top} \left(\boldsymbol {z} ^ {k} - D \boldsymbol {w} ^ {k}\right) + \frac {\rho}{2} \left\| \boldsymbol {z} ^ {k} - D \boldsymbol {w} ^ {k} \right\| ^ {2} \\ = \Omega (\boldsymbol {z} ^ {k}) + g (\boldsymbol {w} ^ {k}) + \frac {\rho}{2} \| \boldsymbol {z} ^ {k} - D \boldsymbol {w} ^ {k} + \frac {\boldsymbol {\lambda} ^ {k}}{\rho} \| ^ {2} - \frac {\| \boldsymbol {\lambda} ^ {k} \| ^ {2}}{2 \rho} \tag {21} \\ \geq - \frac {\| \boldsymbol {\lambda} ^ {k} \| ^ {2}}{2 \rho} > - \infty . \\ \end{array}
$$
So  $L_{\rho}(\pmb{z}^k, \pmb{w}^k, \pmb{\lambda}^k)$  is bounded below by some  $L^{\star}$ . Moreover,  $\sum_{k=1}^{K} \|\pmb{\lambda}^{k+1} - \pmb{\lambda}^k\|^2 \leq \sum_{k=1}^{\infty} \|\pmb{\lambda}^{k+1} - \pmb{\lambda}^k\|^2 < \infty$  for  $\forall K \geq 1$ .
Let  $\hat{L} := L_{\rho}(\pmb{z}^1, \pmb{w}^1, \pmb{\lambda}^1) - L^{\star} + \sum_{k=1}^{\infty} \frac{2}{\rho} \|\pmb{\lambda}^{k+1} - \pmb{\lambda}^k\|^2 < \infty$ . Then telescoping (20) from  $k = 1$  to  $K$ , we obtain that
$$
\begin{array}{l} \hat {L} \geq L _ {\rho} \left(\boldsymbol {z} ^ {1}, \boldsymbol {w} ^ {1}, \boldsymbol {\lambda} ^ {1}\right) - L _ {\rho} \left(\boldsymbol {z} ^ {K}, \boldsymbol {w} ^ {K}, \boldsymbol {\lambda} ^ {K}\right) + \sum_ {k = 1} ^ {K} \frac {2}{\rho} \left\| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \right\| ^ {2} \tag {22} \\ \geq \sum_ {k = 1} ^ {K} \frac {2 r - c}{2} \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2} + \sum_ {k = 1} ^ {K} \frac {1}{\rho} \| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \| ^ {2}. \\ \end{array}
$$
$$
\min _ {k \leq K} \frac {2 r - c}{2} \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2} + \frac {1}{\rho} \| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \| ^ {2} \leq \frac {\hat {L}}{K}.
$$
Letting  $k = \arg \min_{i\leq K}\frac{2r - c}{2}\| \pmb{w}^{i + 1} - \pmb{w}^i\| ^2 +\frac{1}{\rho}\| \pmb{\lambda}^{i + 1} - \pmb{\lambda}^i\| ^2$ , we have
$$
\left\| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \right\| \leq \sqrt {\frac {2 \hat {L}}{K (2 r - c)}},
$$
$$
\left\| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \right\| \leq \sqrt {\frac {\rho \hat {L}}{K}}.
$$
Letting  $K = \frac{1}{\epsilon^2}$ , by (15) and (17) we further have
$$
\operatorname {d i s t} \left(- \boldsymbol {\lambda} ^ {k + 1}, \partial \Omega \left(\boldsymbol {z} ^ {k + 1}\right)\right) \leq \rho \| D \| \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| \leq \rho \| D \| \sqrt {\frac {2 \hat {L}}{K (2 r - c)}} = O (\epsilon),
$$
$$
\operatorname {d i s t} \left(D ^ {T} \boldsymbol {\lambda} ^ {k + 1}, \partial g \left(\boldsymbol {w} ^ {k + 1}\right)\right) \leq r \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| \leq r \sqrt {\frac {2 \hat {L}}{K (2 r - c)}} = O (\epsilon),
$$
$$
\left\| \boldsymbol {z} ^ {k + 1} - D \boldsymbol {w} ^ {k + 1} \right\| = \frac {1}{\rho} \left\| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \right\| \leq \sqrt {\frac {\hat {L}}{\rho K}} = O (\epsilon).
$$
The proof of Theorem 1 is adapted from [33, Theorem 4.1].
A.3 Proofs for smoothed ADMM in Section 4
A.3.1 Proof of Proposition 2
Proof: By Assumption 3, similar to (15), we have
$$
\begin{array}{l} \mathbf {0} \in \partial \Omega \left(\boldsymbol {z} ^ {k + 1}\right) + \boldsymbol {\lambda} ^ {k} + \rho \left(\boldsymbol {z} ^ {k + 1} - D \boldsymbol {w} ^ {k}\right) \\ = \partial \Omega (\boldsymbol {z} ^ {k + 1}) + \boldsymbol {\lambda} ^ {k} + \rho (\boldsymbol {z} ^ {k + 1} - D \boldsymbol {w} ^ {k + 1}) + \rho D (\boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k}) \\ = \partial \Omega (\boldsymbol {z} ^ {k + 1}) + \boldsymbol {\lambda} ^ {k + 1} + \rho D (\boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k}), \\ \end{array}
$$
where the last equality follows from  $\pmb{\lambda}^{k + 1} = \pmb{\lambda}^k +\rho (\pmb{z}^{k + 1} - D\pmb{w}^{k + 1})$
By the first order condition of the  $w$ -subproblem and Assumption 3, we have
$$
\begin{array}{l} \mathbf {0} = \nabla M _ {g, \gamma} (\boldsymbol {w} ^ {k + 1}) - D ^ {\top} \boldsymbol {\lambda} ^ {k} - \rho D ^ {\top} \left(\boldsymbol {z} ^ {k + 1} - D \boldsymbol {w} ^ {k + 1}\right) + r \left(\boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k}\right) \tag {23} \\ = \nabla M _ {g, \gamma} (\boldsymbol {w} ^ {k + 1}) - D ^ {\top} \boldsymbol {\lambda} ^ {k + 1} + r (\boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k}). \\ \end{array}
$$
Note that  $\nabla M_{g,\gamma}(\boldsymbol{w}^{k+1}) \in \partial g\left(\operatorname{prox}_{g,\gamma}\left(\boldsymbol{w}^{k+1}\right)\right) = \partial g(\tilde{\boldsymbol{w}}^{k+1})$ . This completes the proof.
A.3.2 Proof of Lemma 1
Lemma 5 Under Assumption 4, if  $0 < c\gamma \leq \frac{1}{2}$ , then for  $\forall k = 0,1,\ldots$ , we have
$$
\left\| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \right\| ^ {2} \leq \sigma^ {- 1} \left(\gamma^ {- 2} + r ^ {2}\right) \left\| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \right\| ^ {2} + \sigma^ {- 1} r ^ {2} \left\| \boldsymbol {w} ^ {k - 1} - \boldsymbol {w} ^ {k} \right\| ^ {2},
$$
where  $\sigma$  is the smallest positive eigenvalue of  $DD^{\top}$ .
Proof: By Assumption 4 we have  $\pmb{\lambda}^{k + 1} - \pmb{\lambda}^k = \rho (\pmb{z}^{k + 1} - D\pmb{w}^{k + 1})\in Im(D)$ . Then we obtain that
$$
\begin{array}{l} \left\| \boldsymbol {\lambda} ^ {k} - \boldsymbol {\lambda} ^ {k + 1} \right\| ^ {2} \leq \sigma^ {- 1} \left\| D ^ {\top} \left(\boldsymbol {\lambda} ^ {k} - \boldsymbol {\lambda} ^ {k + 1}\right) \right\| ^ {2} \\ \leq \sigma^ {- 1} \| \nabla M _ {g, \gamma} (\boldsymbol {w} ^ {k + 1}) - \nabla M _ {g, \gamma} (\boldsymbol {w} ^ {k}) \| ^ {2} + \sigma^ {- 1} r ^ {2} \| \boldsymbol {w} ^ {k} - \boldsymbol {w} ^ {k + 1} \| ^ {2} + \sigma^ {- 1} r ^ {2} \| \boldsymbol {w} ^ {k - 1} - \boldsymbol {w} ^ {k} \| ^ {2} \\ \leq \sigma^ {- 1} (\gamma^ {- 2} + r ^ {2}) \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2} + \sigma^ {- 1} r ^ {2} \| \boldsymbol {w} ^ {k - 1} - \boldsymbol {w} ^ {k} \| ^ {2}, \\ \end{array}
$$
where the second inequality follows from (23) and the last inequality follows from the fact that  $M_{g,\gamma}$  has  $\gamma^{-1}$  Lipschitz continuous gradient when  $0 < c\gamma \leq \frac{1}{2}$  [7].
Proof: By Assumption 3 we have
$$
L _ {\rho} \left(\boldsymbol {z} ^ {k}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}\right) - L _ {\rho} \left(\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}\right) \geq 0. \tag {24}
$$
Since  $0 < c\gamma \leq \frac{1}{3}$ ,  $M_{g,\gamma}(\pmb{w})$  is  $\gamma^{-1}$ -weakly convex by Lemma 2. By the optimality of  $\pmb{w}^{k+1}$  and  $(r - \gamma^{-1})$ -strong convexity of the  $\pmb{w}$ -subproblem we have
$$
L _ {\rho} \left(\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}\right) - L _ {\rho} \left(\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k + 1}, \boldsymbol {\lambda} ^ {k}\right) \geq \frac {2 r - \gamma^ {- 1}}{2} \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2}. \tag {25}
$$
By the update of dual variable
$$
L _ {\rho} \left(\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k + 1}, \boldsymbol {\lambda} ^ {k}\right) - L _ {\rho} \left(\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k + 1}, \boldsymbol {\lambda} ^ {k + 1}\right) = - \frac {1}{\rho} \| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \| ^ {2}. \tag {26}
$$
Summing (24), (25) and (26) we have
$$
L _ {\rho} \left(\boldsymbol {z} ^ {k}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}\right) - L _ {\rho} \left(\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k + 1}, \boldsymbol {\lambda} ^ {k + 1}\right) \geq \frac {2 r - \gamma^ {- 1}}{2} \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2} - \frac {1}{\rho} \| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \| ^ {2}. \tag {27}
$$
Finally, we obtain that
$$
\begin{array}{l} \Phi^ {k} - \Phi^ {k + 1} \\ = L _ {\rho} (\boldsymbol {z} ^ {k}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}) - L _ {\rho} (\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k + 1}, \boldsymbol {\lambda} ^ {k + 1}) + \frac {2 r ^ {2}}{\rho \sigma} \| \boldsymbol {w} ^ {k - 1} - \boldsymbol {w} ^ {k} \| ^ {2} - \frac {2 r ^ {2}}{\rho \sigma} \| \boldsymbol {w} ^ {k} - \boldsymbol {w} ^ {k + 1} \| ^ {2} \\ \geq \left(\frac {2 r - \gamma^ {- 1}}{2} - \frac {2 r ^ {2}}{\rho \sigma}\right) \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2} + \frac {2 r ^ {2}}{\rho \sigma} \| \boldsymbol {w} ^ {k} - \boldsymbol {w} ^ {k - 1} \| ^ {2} + \left(- \frac {2}{\rho} + \frac {1}{\rho}\right) \| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \| ^ {2} \\ \geq \left(\frac {2 r - \gamma^ {- 1}}{2} - \frac {4 r ^ {2}}{\sigma \rho} - \frac {2}{\sigma \rho \gamma^ {2}}\right) \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2} + \frac {1}{\rho} \| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \| ^ {2}, \\ \end{array}
$$
where the first inequality follows from (27) and the second inequality follows from Lemma 5.
A.3.3 Proof of Theorem 2
Proof: First note that  $0 < c\gamma \leq \frac{1}{3}$ , so Lemma 1 holds. Let  $\mathcal{E}_0 := \frac{1}{3c}$  and  $\mathcal{E}_1 := \frac{r - \gamma^{-1}}{2} - \frac{4r^2}{\sigma\rho} - \frac{2}{\sigma\rho\gamma^2}$ . Using the definitions of  $\gamma, \rho$  and  $r$  we have
$$
\begin{array}{l} \mathcal {E} _ {1} = \frac {2 r - \gamma^ {- 1}}{2} - \frac {4 r ^ {2}}{\sigma \rho} - \frac {2}{\sigma \rho \gamma^ {2}} \\ = \frac {1}{2 \sigma \rho \gamma^ {2}} \left(2 \sigma \rho r \gamma^ {2} - \sigma \rho \gamma - 8 r ^ {2} \gamma^ {2} - 4\right) \\ = \frac {2 \sigma C _ {1} C _ {2} - \sigma C _ {1} - 8 C _ {2} ^ {2} - 4}{2 \sigma C _ {1} \epsilon} \\ > \frac {\mathcal {E} _ {0}}{2 \sigma C _ {1} \epsilon} > \frac {1}{2 \sigma C _ {1}} > 0. \\ \end{array}
$$
Since  $\{\lambda^k\}$  is bounded, we obtain that  $L_{\rho}(z^{k},\pmb{w}^{k},\pmb{\lambda}^{k})$  is bounded below by a similar manner of (21) and Assumption 1. Thus  $\Phi^k$  is lower bounded by some  $\Phi^{\star}$  as well, i.e.,  $\Phi^k\geq \Phi^\star$ , for  $\forall k\geq 1$ . Telescoping (14) from  $k = 1$  to  $K$  we have:
$$
\Phi^ {1} - \Phi^ {\star} \geq \sum_ {k = 1} ^ {K} \mathcal {E} _ {1} \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2} + \sum_ {k = 1} ^ {K} \frac {1}{\rho} \| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \| ^ {2}.
$$
Thus we have:
$$
\min  _ {k \leq K} \mathcal {E} _ {1} \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2} + \frac {1}{\rho} \| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \| ^ {2} \leq \frac {\Phi^ {1} - \Phi^ {\star}}{K}.
$$
Let  $k = \arg \min_{i\leq K}\mathcal{E}_1\| \pmb{w}^{i + 1} - \pmb{w}^i\| ^2 +\frac{1}{\rho}\| \pmb{\lambda}^{i + 1} - \pmb{\lambda}^i\| ^2$ , we have
$$
\begin{array}{l} \left\| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \right\| \leq \sqrt {\frac {\Phi^ {1} - \Phi^ {\star}}{K \mathcal {E} _ {1}}}, \\ \left\| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \right\| \leq \sqrt {\frac {\rho \left(\Phi^ {1} - \Phi^ {\star}\right)}{K}}. \\ \end{array}
$$
Recall  $\tilde{\boldsymbol{w}}^k = \mathrm{prox}_{g,\gamma}(\boldsymbol{w}^k)$ . Note that by (28),  $\mathcal{E}_1^{-1}$  is upper bounded by the constant  $2\sigma C_1$ . Letting  $K = \frac{1}{\epsilon^4}$ , by Proposition 2, the definitions of  $r$  and  $\rho$  and the above two equations, we have
$$
\begin{array}{l} \operatorname {d i s t} \left(- \boldsymbol {\lambda} ^ {k + 1}, \partial \Omega \left(\boldsymbol {z} ^ {k + 1}\right)\right) \leq \rho \| D \| \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| = O (\epsilon), \\ \left. \operatorname {d i s t} \left(D ^ {T} \boldsymbol {\lambda} ^ {k + 1}, \partial g \left(\tilde {\boldsymbol {w}} ^ {k + 1}\right)\right) \leq r \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| = O (\epsilon), \right. \\ \left\| \boldsymbol {z} ^ {k + 1} - D \tilde {\boldsymbol {w}} ^ {k + 1} \right\| \leq \left\| \boldsymbol {z} ^ {k + 1} - D \boldsymbol {w} ^ {k + 1} \right\| + \left\| D \left(\tilde {\boldsymbol {w}} ^ {k + 1} - \boldsymbol {w} ^ {k + 1}\right) \right\| \\ \leq \frac {1}{\rho} \| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \| + \gamma \| D \| M = O (\epsilon), \\ \end{array}
$$
where the second inequality follows from (8) and Assumption 5. This completes the proof.
Our proof of Theorem 2 draws inspiration from [33, Theorem 4.1] and [32]. In our method, we employ the Moreau envelope  $M_{g,\gamma}$  instead of  $g(w)$  to obtain a smoothed version of Problem (1). However, the solution obtained through Algorithm 1 may not satisfy the  $\epsilon$ -KKT conditions of the original problem. Nonetheless, our analysis demonstrates that by setting  $\tilde{\boldsymbol{w}}^k = \mathrm{prox}_{g,\gamma}(\boldsymbol{w}^k)$ , our algorithm guarantees an  $\epsilon$ -KKT point of the original problem in at most  $O(1 / \epsilon^4)$  iterations.
B Additional Experimental Details
B.1 Source Code
The source code is available in the https://github.com/RufengXiao/ADMM-for-rank-based-loss.
B.2 Experimental Environment
All algorithms are implemented in Python 3.8 and all the experiments are conducted on a Linux server with 256GB RAM and 96-core AMD EPYC 7402 2.8GHz CPU.
B.3 Details for the PAVA
Proposition 3 Given two consecutive blocks  $[m, n]$  and  $[n + 1, p]$ , if they are out-of-order, then we have
$$
\frac {\sum_ {i = m} ^ {n} \sigma_ {i}}{n - m + 1} <   \frac {\sum_ {i = n + 1} ^ {p} \sigma_ {i}}{p - n}.
$$
Proof: Suppose on the contrary that  $v_{[m,n]} > v_{[n+1,p]}$  but
$$
\frac {\sum_ {i = m} ^ {n} \sigma_ {i}}{n - m + 1} \geq \frac {\sum_ {i = n + 1} ^ {p} \sigma_ {i}}{p - n}.
$$
We use the conventions that  $h_{[a,b]}(z) = \sum_{i=a}^{b} \theta_i(z)$ . For any  $g \in \partial l(v_{[m,n]})$ , we must have  $g \geq 0$  because  $l$  is monotonically increasing (see the beginning of Section 3.1). Therefore we have
$$
\begin{array}{l} \frac {\sum_ {i = m} ^ {n} \sigma_ {i}}{n - m + 1} g + \rho \left(v _ {[ m, n ]} - \frac {\sum_ {i = m} ^ {n} m _ {i}}{n - m + 1}\right) \\ \geq \frac {\sum_ {i = n + 1} ^ {p} \sigma_ {i}}{p - n} g + \rho \left(v _ {[ m, n ]} - \frac {\sum_ {i = m} ^ {n} m _ {i}}{n - m + 1}\right) \\ \geq \frac {\sum_ {i = n + 1} ^ {p} \sigma_ {i}}{p - n} g + \rho \left(v _ {[ m, n ]} - \frac {\sum_ {i = n + 1} ^ {p} m _ {i}}{p - n}\right), \\ \end{array}
$$
where the first inequality is due to the non-negativity of  $g$ , and the second inequality is due to the fact that  $m_i \leq m_{i+1}$  for  $i = m, m+1, \ldots, p$ . Note also that
$$
\frac {\sum_ {i = m} ^ {n} \sigma_ {i}}{n - m + 1} g + \rho \left(v _ {[ m, n ]} - \frac {\sum_ {i = m} ^ {n} m _ {i}}{n - m + 1}\right) \in \frac {1}{n - m + 1} \partial h _ {[ m, n ]} (v _ {[ m, n ]})
$$
$$
\frac {\sum_ {i = n + 1} ^ {p} \sigma_ {i}}{p - n} g + \rho \left(v _ {[ m, n ]} - \frac {\sum_ {i = n + 1} ^ {p} m _ {i}}{p - n}\right) \in \frac {1}{p - n} \partial h _ {[ n + 1, p ]} (v _ {[ m, n ]})
$$
We also have  $0 \in \partial h_{[m,n]}(v_{[m,n]})$  from the optimality condition. This implies that there exists some  $s \in \partial h_{[n+1,p]}(v_{[m,n]})$  such that  $s \leq 0$ .
Now if there exists another  $s' \in \partial h_{[n+1,p]}(v_{[m,n]})$  such that  $s' > 0$ , then we have  $0 \in \partial h_{[n+1,p]}(v_{[m,n]}')$ , which contradicts with the uniqueness of the optimal point  $v_{[n+1,p]}$  to  $h_{[n+1,p]}$  (due to its strongly convexity). So we must have  $s \leq 0$  for all  $s \in \partial h_{[n+1,p]}(v_{[m,n]})$ . Then we must have  $v_{[m,n]} \leq v_{[n+1,p]}$  from the optimality of  $v_{[n+1,p]}$  to  $h_{[n+1,p]}$  and convexity of  $h_{[n+1,p]}$ . However, this contradicts the fact that  $[m,n]$  and  $[n+1,p]$  are out-of-order.
Proposition 3 presents an additional opportunity to accelerate the PAVA, particularly for the top- $k$  loss and AoRR framework. Consider the case of top- $k$  loss, where  $\sigma_{i} = 1$  if  $i = k$  and  $\sigma_{i} = 0$  otherwise. In this scenario, the condition for merging blocks is satisfied only when  $i = k$ . Therefore, we only need to examine the block containing  $k$ , its preceding block, and its subsequent block. Consequently, the time complexity of searching for out-of-order blocks in line 5 of Algorithm 2 is reduced to  $O(1)$ .
By leveraging this insight, we enhance the efficiency of our algorithm, specifically for top- $k$  loss. An analogous acceleration can also be applied to the AoRR loss. This optimization dramatically reduces computational complexity, thereby facilitating faster execution of the PAVA for both top- $k$  loss and AoRR loss.
In our implementation of the PAVA, we use either the bisection method or Newton's method to find a minimizer of the convex function  $h_{[m,n]}$ . Assuming the maximum number of iterations for these methods as  $T$ , the computation of the minimizer  $v_{[m,n]}$  exhibits a time complexity of  $O(T)$ . In the PAVA, as we merge out-of-order blocks, the size of the index set  $J$  decreases by at least one.