SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 1 provides a crucial insight: when we encounter consecutive out-of-order blocks with a length greater than 2, we can merge them simultaneously, rather than performing individual block merges. This approach significantly improves the efficiency of the algorithm. To illustrate this concept, consider a scenario where we have a sequence of values arranged as  $v_{[1,2]} > v_{[3,4]} > v_{[5,6]}$ . Initially, we merge the blocks [1,2] and [3,4], resulting in  $v_{[1,4]} \geq v_{[3,4]} > v_{[5,6]}$ . However, since  $v_{[1,4]}$  is still greater than  $v_{[5,6]}$ , we need to merge the blocks [1,4] and [5,6] as well. Rather than calculating  $v_{[1,4]}$  separately, we can streamline the process by directly merging the entire sequence into a single block, namely [1,6], in a single operation. By leveraging this approach, we eliminate the need for intermediate calculations and reduce the computational burden associated with merging individual blocks. This results in a more efficient version of the PAVA. Further details can be found in Appendix B.3.
It is worth noting that when  $\theta_{i}(z_{i})$  is convex with respect to  $z_{i}$ , the PAVA guarantees to find a global minimizer [6]. However, when considering the empirical human risk minimization with CPT-weight in (4), the function  $\theta_{i}(z_{i})$  is nonconvex, which is because  $\sigma_{i}$  is a piecewise function with two different values. In such cases, we can still find a point that satisfies the first-order condition [12, Theorem 3]. In summary, we always have
$$
\mathbf {0} \in \partial_ {\boldsymbol {z}} L _ {\rho} \left(\boldsymbol {w} ^ {k + 1}, \boldsymbol {z}; \boldsymbol {\lambda} ^ {k}\right). \tag {11}
$$
3.3 Convergence Analysis of Algorithm 1
We now demonstrate that Algorithm 1 is guaranteed to converge to an  $\epsilon$ -KKT point of problem (9). To this end, we shall make the following assumptions.
Assumption 2 The sequence  $\{\lambda^k\}$  is bounded and satisfies  $\sum_{k=1}^{\infty} \| \lambda^k - \lambda^{k+1} \|^2 < \infty$ .
It is worth noting that Assumption 2 is commonly employed in ADMM approaches [51, 44, 3].
Next, we present our convergence result based on the aforementioned assumptions. As mentioned in Section 3.2, the PAVA can always find a solution for the  $z$ -subproblem that satisfies the first-order condition (11). When  $\sigma_{i}$  is constant, which is the case of the spectral risk and the AoRR risk, the subproblems of PAVA are strongly convex, and we can observe the descent property of the  $z$ -subproblem:
$$
L _ {\rho} \left(\boldsymbol {z} ^ {k}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}\right) - L _ {\rho} \left(\boldsymbol {z} ^ {k + 1}, \boldsymbol {w} ^ {k}, \boldsymbol {\lambda} ^ {k}\right) \geq 0. \tag {12}
$$
However, if  $\sigma_{i}$  has different values for  $z$  less than and larger than the reference point  $B$  as in (4), the subproblems of PAVA may lose convexity, and the descent property may not hold. Thus, we assume that (12) holds for simplicity. By noting that the  $w$ -subproblem can be solved exactly as it is strongly convex, we make the following assumption.
Assumption 3 The  $\mathbf{w}$ -subproblem is solved exactly, and the  $\mathbf{z}$ -subproblem is solved such that (11) and (12) hold.
Theorem 1 Assume that Assumptions 1, 2 and 3 hold. Then Algorithm 1 can find an  $\epsilon$ -KKT point  $(\pmb{z}^{k+1}, \pmb{w}^{k+1}, \pmb{\lambda}^{k+1})$  of problem (9) within  $O(1/\epsilon^2)$  iterations, i.e.,
$$
\operatorname {d i s t} \left(- \boldsymbol {\lambda} ^ {k + 1}, \partial \Omega \left(\boldsymbol {z} ^ {k + 1}\right)\right) \leq \epsilon , \quad \operatorname {d i s t} \left(D ^ {\top} \boldsymbol {\lambda} ^ {k + 1}, \partial g \left(\boldsymbol {w} ^ {k + 1}\right)\right) \leq \epsilon , \quad \| \boldsymbol {z} ^ {k + 1} - D \boldsymbol {w} ^ {k + 1} \| \leq \epsilon ,
$$
where  $\mathrm{dist}(\pmb {a},A) = \min \{\| \pmb {a} - \pmb {x}\| :\pmb {x}\in A\}$  defines the distance of a point  $\pmb{a}$  to a set  $A$
In the absence of Assumption 2, a common assumption for nonconvex ADMM is that  $g(\boldsymbol{w})$  possesses a Lipschitz continuous gradient [48, 32], e.g.,  $g(\boldsymbol{w}) = \frac{\mu}{2} \| \boldsymbol{w} \|^2$ . Under this assumption, Algorithm 1 guarantees to find an  $\epsilon$ -KKT point within  $O(1 / \epsilon^2)$  iterations [48, 32].
4 ADMM for a Smoothed Version of Problem (1)
In Section 3.3, our convergence result is established under Assumption 2 when a nonsmooth regularizer  $g(\boldsymbol{w})$  is present. In this section, to remove this assumption, we design a variant of the proximal ADMM by smoothing the regularizer  $g(\boldsymbol{w})$ . We employ the Moreau envelope to smooth  $g(\boldsymbol{w})$  and replace  $g(\boldsymbol{w})$  in Algorithm 1 with  $M_{g,\gamma}(\boldsymbol{w})$ . We point out that  $M_{g,\gamma}(\boldsymbol{w})$  is a  $\gamma^{-1}$  weakly convex function if  $0 < c\gamma \leq \frac{1}{3}$ . Thus the  $w$ -subproblem is still strongly convex and can be solved exactly. See Appendix A.1 for proof. We assume that the sequence  $\{\lambda^k\}$  is bounded, which is much weaker than Assumption 2. We will show that within  $O(1/\epsilon^4)$  iterations,  $(z^k, \tilde{w}^k, \lambda^k)$  is an  $\epsilon$ -KKT point, where  $\tilde{w}^k = \mathrm{prox}_{g,\gamma}(\boldsymbol{w}^k)$ .
To compensate for the absence of Assumption 2, we introduce the following more practical assumptions.
Assumption 4 We assume  $z^k \in \operatorname{Im}(D) \forall k$ , where  $\operatorname{Im}(D) = \{D\mathbf{x} : \mathbf{x} \in \mathbb{R}^d\}$ .
A stronger version of Assumption 4 is that  $DD^{\top} = (\mathrm{diag}(\pmb{y})X)(\mathrm{diag}(\pmb{y})X)^{\top} = \mathrm{diag}(\pmb{y})XX^{\top}\mathrm{diag}(\pmb{y}) \succ 0$  [32]. It is worth noting that the full row rank property of the data matrix is often assumed in the high dimensional setting classification ( $m < d$  and each entry of  $\pmb{y}$  belongs to  $\{-1,1\}$ ). Here we do not impose any assumptions on the rank of data matrix  $X$ .
Assumption 5  $\nabla M_{g,\gamma}(\pmb{w})$  is bounded by  $M > 0$ , i.e.,  $\| \nabla M_{g,\gamma}(\pmb{w})\| \leq M$ ,  $\forall \pmb{w} \in \mathbb{R}^n$ .
Regarding nonsmooth regularizers, Assumption 5 is satisfied by weakly convex functions that are Lipschitz continuous [7], due to the fact that Lipschitz continuity implies bounded subgradients and (8). Common regularizers such as  $l_{1}$ -norm, MCP and SCAD are all Lipschitz continuous.
The following proposition establishes the relationship between  $\{(z^k, \boldsymbol{w}^k, \boldsymbol{\lambda}^k)\}$  and  $\{(z^k, \tilde{\boldsymbol{w}}^k, \boldsymbol{\lambda}^k)\}$ .
Proposition 2 Suppose  $g$  is  $c$ -weakly convex and Assumption 3 holds. Suppose the sequence  $\{(z^k, \boldsymbol{w}^k, \boldsymbol{\lambda}^k)\}$  is produced during the iterations of Algorithm 1 with replacing  $g(\boldsymbol{w})$  by  $M_{g,\gamma}(\boldsymbol{w})$  and  $\tilde{\boldsymbol{w}}^k = \mathrm{prox}_{g,\gamma}(\boldsymbol{w}^k)$ . Then we have
$$
\mathbf {0} \in \partial \Omega \left(\boldsymbol {z} ^ {k + 1}\right) + \boldsymbol {\lambda} ^ {k + 1} + \rho D \left(\boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k}\right), \quad D ^ {T} \boldsymbol {\lambda} ^ {k + 1} + r \left(\boldsymbol {w} ^ {k} - \boldsymbol {w} ^ {k + 1}\right) \in \partial g (\tilde {\boldsymbol {w}} ^ {k + 1}). \tag {13}
$$
Before presenting our main results, we introduce a Lyapunov function that plays a significant role in our analysis:
$$
\Phi^ {k} = L _ {\rho} (\boldsymbol {w} ^ {k}, \boldsymbol {z} ^ {k}, \boldsymbol {\lambda} ^ {k}) + \frac {2 r ^ {2}}{\sigma \rho} \| \boldsymbol {w} ^ {k} - \boldsymbol {w} ^ {k - 1} \| ^ {2}.
$$
The following lemma demonstrates the sufficient decrease property for  $\Phi^k$ .
Lemma 1 Let  $r > \gamma^{-1}$ . Under Assumptions 1, 3, 4 and 5, if  $0 < c\gamma \leq \frac{1}{3}$ , then we have
$$
\Phi^ {k} - \Phi^ {k + 1} \geq \left(\frac {2 r - \gamma^ {- 1}}{2} - \frac {4 r ^ {2}}{\sigma \rho} - \frac {2}{\sigma \rho \gamma^ {2}}\right) \| \boldsymbol {w} ^ {k + 1} - \boldsymbol {w} ^ {k} \| ^ {2} + \frac {1}{\rho} \| \boldsymbol {\lambda} ^ {k + 1} - \boldsymbol {\lambda} ^ {k} \| ^ {2}. \tag {14}
$$
The sufficient decrease property for the Lyapunov function is crucial in the proof of nonconvex ADMM. Using Lemma 1, we can control  $\| \pmb{w}^{k + 1} - \pmb{w}^k\|$  and  $\| \lambda^{k + 1} - \lambda^k\|$  through the decrease of the Lyapunov function. We are now ready to present the main result of this section.
Theorem 2 Suppose  $\{\pmb{\lambda}^k\}$  is bounded. Set  $\gamma = \epsilon \leq \frac{1}{3c}$ ,  $\rho = \frac{C_1}{\epsilon}$ ,  $r = \frac{C_2}{\epsilon}$ , where  $C_1$  and  $C_2$  are constants such that  $C_2 > 1$  and  $C_1 > \frac{8C_2^2 + \frac{1}{3c} + 4}{\sigma(2C_2 - 1)}$ . Under the same assumptions in Lemma 1, Algorithm 1 with replacing  $g(\pmb{w})$  by  $M_{g,\gamma}(\pmb{w})$  finds an  $\epsilon$ -KKT point  $(\pmb{z}^{k+1}, \tilde{\pmb{w}}^{k+1}, \pmb{\lambda}^{k+1})$  within  $O(1/\epsilon^4)$  iterations, i.e.,
$$
\operatorname {d i s t} \left(- \boldsymbol {\lambda} ^ {k + 1}, \partial \Omega \left(\boldsymbol {z} ^ {k + 1}\right)\right) \leq \epsilon , \quad \operatorname {d i s t} \left(D ^ {T} \boldsymbol {\lambda} ^ {k + 1}, \partial g \left(\tilde {\boldsymbol {w}} ^ {k + 1}\right)\right) \leq \epsilon , \quad \| \boldsymbol {z} ^ {k + 1} - D \tilde {\boldsymbol {w}} ^ {k + 1} \| \leq \epsilon .
$$
5 Numerical Experiment
In this section, we perform binary classification experiments to illustrate both the robustness and the extensive applicability of our proposed algorithm.
When using the logistic loss or hinge loss as individual loss, the objective function of problem (1) can be rewritten as  $\sum_{i=1}^{n} \sigma_i \log \left(1 + \exp \left(z_{[i]}\right)\right) + g(\boldsymbol{w})$  or  $\sum_{i=1}^{n} \sigma_i \left[1 + z_{[i]}\right]_+$  +  $g(\boldsymbol{w})$ , where  $\boldsymbol{z} = -\boldsymbol{y} \odot (X\boldsymbol{w})$ , and  $g(\boldsymbol{w}) = \frac{\mu}{2} \| \boldsymbol{w} \|_2^2$  or  $g(\boldsymbol{w}) = \frac{\mu}{2} \| \boldsymbol{w} \|_1$ .
We point out some key settings:
1) For the spectral risk measures, we use the logistic loss and hinge loss as individual loss, and use  $\ell_1$  and  $\ell_2$  regularization with  $\mu$  taken as  $10^{-2}$ .
2) For the average of ranked range aggregate loss, we use the logistic loss and hinge loss as individual loss and use  $\ell_2$  regularization with  $\mu = 10^{-4}$ .
3) For the empirical human risk minimization, we use the logistic loss as individual loss, and use  $\ell_2$  regularization with  $\mu = 10^{-2}$ .
As shown in Section 2.1, we can apply our proposed algorithm to a variety of frameworks such as the spectral risk measure (SRM), the average of ranked-range (AoRR) aggregate loss, and the empirical human risk minimization (EHRM). We compare our algorithm with LSVRG, SGD, and DCA: LSVRG(NU) denotes LSVRG without uniformity, and LSVRG(U) denotes LSVRG with uniformity, as detailed in [36]; SGD denotes the stochastic subgradient method in [36]; DCA denotes the difference-of-convex algorithm in [40]; sADMM refers to the smoothed version of ADMM.
The details of our algorithm setting, more experiment settings and detailed information for each dataset are provided in Appendix B. Additional experiments with synthetic datasets are presented in Appendix C.
5.1 Spectral Risk Measures
In this section, we select  $\sigma(t) \equiv 1$  in (3) to generate the empirical risk minimization (ERM) and  $\sigma(t) = 1_{[q,1]}(t) / (1 - q)$  to generate superquantile, the latter having garnered significant attention in the domains of finance quantification and machine learning [30].
Setting. For the learning rates of SGD and LSVRG, we use grid search in the range  $\{10^{-2}, 10^{-3}, 10^{-4}, 1 / N_{\mathrm{samples}}, 1 / (N_{\mathrm{samples}} \times N_{\mathrm{features}})\}$ , which is standard for stochastic algorithms, and then we repeat the experiments five times with different random seeds.
Results. Tables 1 and 2 present the mean and standard deviation of the results in different regularizations. The first row for each dataset represents the objective value of problem (1) and the second row shows test accuracy. Bold numbers indicate significantly better results. Tables 1 and 2 show that our ADMM algorithm outperforms existing methods in terms of objective values for most instances and has comparable test accuracy. In addition, we highlight the necessity of reselecting the learning rate for SGD and LSVRG algorithms, as the individual loss and dataset undergo modifications—a highly intricate procedure. Conversely, our proposed algorithm circumvents the need for any adjustments in this regard.
Table 1: Results in real data with SRM superquantile framework and  ${\ell }_{2}$  regularization.
Datasets	Logistic Loss	Hinge Loss
SVMguide	ADMM	LSVRG(NU)	LSVRG(U)	SGD	ADMM	LSVRG(NU)	LSVRG(U)	SGD
0.6522 (0.0058)	0.6522 (0.0058)	0.6522 (0.0058)	0.6523 (0.0058)	0.7724 (0.0214)	0.7724 (0.0214)	0.7724 (0.0214)	0.7738 (0.0219)
0.9566 (0.0035)	0.9563 (0.0038)	0.9561(0.0036)	0.9566 (0.0032)	0.9568 (0.0033)	0.9568 (0.0033)	0.9568 (0.0033)	0.9574 (0.0035)
AD	0.1539 (0.0092)	0.1538 (0.0092)	0.1538 (0.0091)	0.1560 (0.0114)	0.0787 (0.0132)	0.0852 (0.0129)	0.0833 (0.0134)	0.0815 (0.0137)
0.9629 (0.0071)	0.9633 (0.0070)	0.9633 (0.0070)	0.9631 (0.0067)	0.9521 (0.0112)	0.9559 (0.0099)	0.9578 (0.0098)	0.9576 (0.0097)
Table 2: Results in real data with SRM superquantile framework and  ${\ell }_{1}$  regularization.
Datasets	Logistic Loss	Hinge Loss
SVMguide	ADMM	sADMM	LSVRG(NU)	LSVRG(U)	SGD	ADMM	sADMM	LSVRG(NU)	LSVRG(U)	SGD
0.6280 (0.0129)	0.6280 (0.0129)	0.6280 (0.0129)	0.6280 (0.0129)	0.6282 (0.0128)	0.6789 (0.0273)	0.6789 (0.0273)	0.9121 (0.0133)	0.6789 (0.0273)	0.8569 (0.0210)
0.9561 (0.0037)	0.9561 (0.0037)	0.9563 (0.0039)	0.9561 (0.0037)	0.9561 (0.0038)	0.9566 (0.0036)	0.9566 (0.0036)	0.9558 (0.0039)	0.9563 (0.0035)	0.9555 (0.0040)
AD	0.2601 (0.0079)	0.2602 (0.0079)	0.2614 (0.0080)	0.2614 (0.0080)	0.2641 (0.0105)	0.1352 (0.0094)	0.1370 (0.0096)	0.1457 (0.0110)	0.1449 (0.0100)	0.1402 (0.0101)
0.9673 (0.0054)	0.9673 (0.0054)	0.9680 (0.0054)	0.9680 (0.0054)	0.9673 (0.0039)	0.9589 (0.0060)	0.9589 (0.0060)	0.9644 (0.0045)	0.9639 (0.0039)	0.9635 (0.0023)
Figure 1 illustrates the relationship between time and sub-optimality of each algorithm in the ERM framework with logistic loss, which is a convex problem whose global optimum can be achieved. The learning rate of the other algorithms is the best one selected by the same method. The Figure 1 depicts that ADMM exhibits significantly faster convergence towards the minimum in comparison to the other algorithms. Conversely, although sADMM fails to achieve the accuracy level of ADMM, it nevertheless demonstrates accelerated convergence when compared to other algorithms.
5.2 Empirical Human Risk Minimization
Since [31] utilizes gradient-based algorithms akin to those in [36], we compare our algorithm with the one referenced in Section 5.1.
Results. The mean and standard deviation of the results are tabulated in Table 3. SPD, DI, EOD, AOD, TI, and FNRD are fairness metrics detailed in Appendix B.6. The bold numbers highlight the superior result. It is evident that our proposed algorithm outperforms the other algorithms in terms of objective values and test accuracy, and also shows enhanced performance across almost all fairness metrics.
5.3 Average of Ranked Range Aggregate Loss
As mentioned in Section 2.1, DCA can be used to solve for AoRR aggregate loss minimization [27]. Therefore, we compare our algorithm with the DCA algorithm in this section.
Results. Table 4 displays the mean and standard deviation of both the objective value and time results. Given that this is a nonconvex problem, the objective value may not be the optimal value.
(g)
Figure 1: Time vs. Sub-optimality gap in synthetic datasets with ERM framework. (a-d) for  $\ell_2$  regularization, and (e-f) for  $\ell_1$  regularization. Sub-optimality is defined as  $F^k - F^*$ , where  $F^k$  represents the objective function value at the  $k$ -th iteration or epoch and  $F^*$  denotes the minimum value obtained by all algorithms. Plots are truncated when  $F^k - F^* < 10^{-8}$ .
Table 3: Comparison with EHRM framework. 'objval' denotes the objective value of problem (1).
ADMM	LSVRG(NU)	LSVRG(U)	SGD	ADMM	LSVRG(NU)	LSVRG(U)	SGD
objval	0.4955 (0.0045)	0.5383 (0.0022)	0.5384 (0.0023)	0.6093 (0.0013)	EOD	-0.0521 (0.0160)	-0.0238 (0.0114)	-0.0231 (0.0110)	-0.0160(0.0103)
Accuracy	0.7759 (0.0076)	0.7445 (0.0050)	0.7448 (0.0054)	0.6334 (0.0205)	AOD	0.0130 (0.0090)	0.0449 (0.0199)	0.0450 (0.0196)	0.0426 (0.0078)
SPD	0.0300 (0.0071)	0.0585 (0.0161)	0.0587 (0.0158)	0.0459 (0.0050)	TI	0.0838 (0.0018)	0.0933 (0.0018)	0.0934 (0.0018)	0.1062 (0.0052)
DI	1.0533 (0.0126)	1.1034 (0.0300)	1.104 (0.0297)	1.0586 (0.0065)	FNRD	0.0521 (0.0160)	0.0238 (0.0114)	0.0231 (0.0110)	0.0160 (0.0103)
Nevertheless, as evidenced in Table 4, our proposed algorithm achieves better objective value in a shorter time for all the instances.
Table 4: Comparison with AoRR framework. 'objval' denotes the objective value of problem (1).
Datasets	Monks	Australian	Phoneme	Titanic	Splice
ADMM	DCA	ADMM	DCA	ADMM	DCA	ADMM	DCA	ADMM	DCA
Logistic	objval	0.0025 (0.0001)	0.5857 (0.1258)	0.0011 (0.0001)	0.6919 (0.0003)	0.0031 (0.0001)	0.685 (0.0090)	0.0027 (0.0001)	0.6912 (0.0003)	0.0018 (0.0001)	0.6914 (0.0003)
time (s)	3.15 (0.11)	46.73 (0.29)	3.54 (0.34)	53.85 (0.98)	55.36 (5.89)	116.24 (27.16)	40.98 (9.36)	51.8 (0.21)	17.58 (0.37)	63.91 (5.41)
Hinge	objval	0.0093 (0.0017)	0.8556 (0.1701)	0.0017 (0.0010)	0.9949 (0.0012)	0.0060 (0.0005)	0.8842 (0.1657)	0.0127 (0.0021)	0.9923 (0.0013)	0.0038 (0.0005)	0.9931 (0.0013)
time (s)	1.05 (0.02)	23.80 (0.13)	1.62 (0.05)	24.15 (0.07)	20.28 (3.31)	39.09 (1.60)	7.22 (1.26)	14.53 (1.53)	9.12 (1.42)	114.52 (26.94)
This paper considers rank-based loss optimization with monotonically increasing loss functions and weakly convex regularizers. We propose a unified ADMM framework for rank-based loss minimization. Notably, one subproblem of the ADMM is solved efficiently by the PAVA. Numerical experiments illustrate the outperformance of our proposed algorithm, with all three practical frameworks delivering satisfactory results. We also point out some limitations of our algorithm. To effectively utilize our algorithm, individual losses must exhibit monotonicity, as this allows us to use the PAVA to solve subproblems. If the sample size is large, the PAVA's computational efficiency may be hindered, potentially limiting its overall effectiveness. Future work may explore a variant using mini-batch samples, potentially improving early-stage optimization performance and overall computational efficiency.
Acknowledgement Rujun Jiang is partly supported by NSFC 12171100 and Natural Science Foundation of Shanghai 22ZR1405100.
[40] Phan, D. N. (2016). DCA based algorithms for learning with sparsity in high dimensional setting and stochastic learning. PhD thesis, Université de Lorraine.
[41] Prashanth, L., Jie, C., Fu, M., Marcus, S., and Szepesvári, C. (2016). Cumulative prospect theory meets reinforcement learning: Prediction and control. In International Conference on Machine Learning, pages 1406-1415. PMLR.
[42] Schmidt, M., Fung, G., and Rosales, R. (2007). Fast optimization methods for 11 regularization: A comparative study and two new approaches. In Machine Learning: ECML 2007: 18th European Conference on Machine Learning, Warsaw, Poland, September 17-21, 2007. Proceedings 18, pages 286-297. Springer.
[43] Shalev-Shwartz, S. and Wexler, Y. (2016). Minimizing the maximal loss: How and why. In International Conference on Machine Learning, pages 793-801. PMLR.
[44] Shen, Y., Wen, Z., and Zhang, Y. (2014). Augmented lagrangian alternating direction method for matrix separation based on low-rank factorization. Optimization Methods and Software, 29(2):239-263.
[45] Tversky, A. and Kahneman, D. (1992). Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5:297-323.
[46] Vapnik, V. (2013). The Nature of Statistical Learning Theory. Springer Science & Business Media.
[47] Vial, J.-P. (1983). Strong and weak convexity of sets and functions. Mathematics of Operations Research, 8(2):231-259.
[48] Wang, Y., Yin, W., and Zeng, J. (2019). Global convergence of admm in nonconvex nonsmooth optimization. Journal of Scientific Computing, 78:29-63.
[49] Williamson, R. and Menon, A. (2019). Fairness risk measures. In International Conference on Machine Learning, pages 6786-6797. PMLR.
[50] Wnek, J. (1992). MONK's Problems. UCI Machine Learning Repository. DOI: 10.24432/C5R30R.
[51] Xu, Y., Yin, W., Wen, Z., and Zhang, Y. (2012). An alternating direction algorithm for matrix completion with nonnegative factors. Frontiers of Mathematics in China, 7:365-384.
[52] Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics, pages 894-942.
[53] Zhang, Z., Song, Y., and Qi, H. (2017). Age progression/regression by conditional adversarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5810-5818.
A.1 Proofs for properties of Moreau envelope
Lemma 2 Let  $g: \mathbb{R}^n \to \mathbb{R} \cup \{\infty\}$  be a c-weakly convex function. For  $0 < c\gamma \leq \frac{1}{3}$ , the Moreau envelope  $M_{g,\gamma}$  of  $g$  is  $\gamma^{-1}$ -weakly convex.
A.1.1 Proof of Lemma 2
Lemma 3 Let  $g: \mathbb{R}^n \to \mathbb{R} \cup \{\infty\}$ . Then  $g$  is c-weakly convex if and only if the following holds
$$
\left\langle \boldsymbol {v} - \boldsymbol {w}, \boldsymbol {x} - \boldsymbol {y} \right\rangle \geq - c \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2},
$$
for  $\pmb{x}, \pmb{y} \in \mathbb{R}^n$ ,  $\pmb{v} \in \partial g(\pmb{x})$ ,  $\pmb{w} \in \partial g(\pmb{y})$ .
Proof: See [16, Lemma 2.1].
Lemma 4 Suppose  $g(\boldsymbol{w})$  is  $c$ -weakly convex. If  $0 < c\gamma \leq \frac{1}{3}$ , then we have
$$
\| \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x}) - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y}) \| ^ {2} \leq 3 \| \boldsymbol {y} - \boldsymbol {x} \| ^ {2}.
$$
Proof: Suppose  $\pmb{x}, \pmb{y} \in \mathbb{R}^n$ . By (8) and Lemma 3 we have
$$
\begin{array}{l} - c \| \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x}) - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y}) \| ^ {2} \\ \leq \gamma^ {- 1} \left\langle \left(\boldsymbol {y} - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y})\right) - \left(\boldsymbol {x} - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x})\right), \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y}) - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x}) \right\rangle \\ = \frac {1}{2 \gamma} \left[ - \| \boldsymbol {y} - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y}) - \boldsymbol {x} + \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x}) \| ^ {2} - \| \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y}) - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x}) \| ^ {2} \right. \\ + \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2} ]. \\ \end{array}
$$
$$
\begin{array}{l} (1 - 2 c \gamma) \| \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y}) - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x}) \| ^ {2} \leq - \| \boldsymbol {y} - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y}) - \boldsymbol {x} + \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x}) \| ^ {2} + \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2} \\ \leq \left\| \boldsymbol {x} - \boldsymbol {y} \right\| ^ {2}. \\ \end{array}
$$
Note that  $(1 - 2c\gamma)^{-1} \leq 3$  when  $0 < c\gamma \leq \frac{1}{3}$ . Thus we have
$$
\left\| \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {y}) - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {x}) \right\| ^ {2} \leq (1 - 2 c \gamma) ^ {- 1} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2} \leq 3 \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2}.
$$
Proof: For any  $\pmb{x}, \pmb{y} \in \mathbb{R}^n$ , we have