SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rufeng Xiao* Yuze Ge* Rujun Jiang† Yifan Yan
School of Data Science, Fudan University
{rfxiao21,yzge23}@m.fudan.edu.cn
rjjiang@fudan.edu.cn
yanyf21@m.fudan.edu.cn
The empirical loss, commonly referred to as the average loss, is extensively utilized for training machine learning models. However, in order to address the diverse performance requirements of machine learning models, the use of the rank-based loss is prevalent, replacing the empirical loss in many cases. The rank-based loss comprises a weighted sum of sorted individual losses, encompassing both convex losses like the spectral risk, which includes the empirical risk and conditional value-at-risk, and nonconvex losses such as the human-aligned risk and the sum of the ranked range loss. In this paper, we introduce a unified framework for the optimization of the rank-based loss through the utilization of a proximal alternating direction method of multipliers. We demonstrate the convergence and convergence rate of the proposed algorithm under mild conditions. Experiments conducted on synthetic and real datasets illustrate the effectiveness and efficiency of the proposed algorithm.
The empirical risk function is the cornerstone of machine learning. By minimizing this function, machine learning models typically achieve commendable average performance. However, as these models find applications across a diverse spectrum of real-world scenarios, the evaluation standards for machine learning performance have evolved to include factors such as risk aversion and fairness [49, 9]. This has led to the empirical loss function often being supplanted by other loss functions, many of which fall under the category of the rank-based loss, which consists of a weighted sum of sorted individual losses. We consider the following optimization problem in this paper, which minimizes the rank-based loss plus a regularizer:
$$
\min  _ {\boldsymbol {w} \in \mathbb {R} ^ {d}} \sum_ {i = 1} ^ {n} \sigma_ {i} \boldsymbol {l} _ {[ i ]} (- \boldsymbol {y} \odot (X \boldsymbol {w})) + g (\boldsymbol {w}). \tag {1}
$$
Here,  $l(\pmb{u}) \triangleq [l(u_1), \dots, l(u_n)]^\top : \mathbb{R}^n \to \mathbb{R}^n$  represents a vector-valued mapping where  $l : \mathbb{R} \to \mathbb{R}$  denotes an individual loss function and  $u_i$  is the  $i$ -th element of  $\pmb{u}$ ,  $l_{[1]}(\cdot) \leqslant \dots \leqslant l_{[n]}(\cdot)$  denotes the order statistics of the empirical loss distribution,  $g : \mathbb{R}^d \to \mathbb{R}$  is a regularizer that induces desired structures,  $\pmb{w} \in \mathbb{R}^d$  represents the parameters of the linear model,  $X \in \mathbb{R}^{n \times d}$  is the data matrix, and  $\pmb{y} \in \{\pm 1\}^n$  is the label vector. The subscript  $[i]$  denotes the  $i$ -th smallest element,  $\sigma_i \in [0,1]$  is the weight corresponding to the  $i$ -th smallest element, and “ $\odot$ ” represents the Hadamard product. Depending on the values of  $\sigma_i$ , the rank-based loss can encompass a significant range of losses often
used in machine learning and other fields: the L-risk or the spectral risk [1, 2, 35], the trimmed risk or the sum of the ranked-range loss [27], the human-aligned risk based on the cumulative prospect theory [31].
Related work. When the values of  $\sigma_{i}$  are constant and monotonically increase, i.e.,  $\sigma_{1} \leq \dots \leq \sigma_{n}$ , the rank-based loss is referred to as the spectral risk [1]. Several methods have been proposed in the literature, including: a derivative-free method proposed by [25] that utilizes a stochastic smoothing technique; an adaptive sampling algorithm proposed by [14] for the minimization of conditional value at risk (CVaR), which is a special case of the spectral risk; and a stochastic algorithm developed by [36] for minimizing the general spectral risk by characterizing the subdifferential of the rank-based loss. In the case of nonconvex rank-based losses, [27] utilizes the difference of convex algorithm for minimizing the average of ranked-range (AoRR) loss. The nonconvex human-aligned risk was minimized by [31] by directly computing the gradient of the function, even though the rank-based loss is nondifferentiable, without any convergence guarantee. Despite the increasing applications of the rank-based loss, a unified framework for addressing the rank-based loss minimization remains elusive. Moreover, stochastic methods may encounter bias issues [36], existing methods for the spectral risk cannot deal with nonconvex rank-based losses, and existing methods for human-aligned risk lack convergence guarantee [31].
Our Contributions. We present a unified framework for the minimization of the rank-based loss to overcome the challenges of existing methods. Specifically, we focus on monotone increasing loss functions, such as the hinge loss and the logistic loss, and use weakly convex regularizers. We leverage the alternating direction multiplier method (ADMM), which is widely used in solving nonsmooth nonconvex composite problems [8, 32, 48]. To utilize the ADMM framework, we introduce an auxiliary variable to represent the product of the data matrix and parameter vector. The two subproblems are either strongly convex (with a proximal term), which can be solved by various existing methods, or can be efficiently solved by the pool adjacent violators algorithm (PAVA) [6]. We demonstrate that our algorithm can find an  $\epsilon$ -KKT point in at most  $O(1/\epsilon^2)$  iterations under mild conditions. To relax the assumptions further, we propose a variant of the ADMM with a smoothing technique when the regularizer is nonsmooth. Notably, when the Moreau envelope is applied to smooth the nonsmooth regularizer, we show that our algorithm can find an  $\epsilon$ -KKT point within  $O(1/\epsilon^4)$  iterations.
1) We propose a unified framework for the rank-based loss minimization for monotonically increasing loss functions. This approach is versatile, effectively dealing with both convex and nonconvex loss functions by allowing different settings of weights  $\sigma_{i}$ .
2) Furthermore, the regularizer in our problem can be weakly convex functions, extending existing works that only consider convex or smooth regularizers, such as the  $l_{1}$  and  $l_{2}$  penalty.
3) We theoretically demonstrate that our ADMM algorithm converges to an  $\epsilon$ -approximate KKT point under different assumptions.
4) The experiments in three different aggregate loss function frameworks demonstrate the advantages of the proposed algorithm.
In this section, we explore the wide variety of rank-based losses and introduce the Moreau envelope as a smooth approximation of weakly convex functions.
2.1 Rank-Based Loss
Let  $\{(X_1, y_1), \ldots, (X_n, y_n)\}$  be an i.i.d. sample set from a distribution  $\mathbb{P}$  over a sample space  $(\mathcal{X}, \mathcal{Y})$ , and  $\mathcal{L}$  be the individual loss function. Then  $Y = \mathcal{L}(\boldsymbol{w}, X, y)$  is also a random variable, and  $Y_i = \mathcal{L}(\boldsymbol{w}, X_i, y_i)$  represents the  $i$ -th training loss with respect to the sample  $(X_i, y_i)$ ,  $i \in \{1, \ldots, n\}$ . Let  $Y_{[1]} \leq \ldots \leq Y_{[n]}$  be the order statistics. The rank-based loss function in problem (1) equals
$$
\sum_ {i = 1} ^ {n} \sigma_ {i} Y _ {[ i ]}. \tag {2}
$$
Spectral Risk The spectral risk, as defined in [1], is given by:
$$
\mathrm {R} _ {\sigma} (\boldsymbol {w}) = \int_ {0} ^ {1} \operatorname {V a R} _ {t} (\boldsymbol {w}) \sigma (t) d t, \tag {3}
$$
where  $\mathrm{VaR}_t(\pmb{w}) = \inf \left\{u:\mathrm{F}_{\pmb{w}}(u)\geq t\right\}$  denotes the quantile function, and  $F_{\mathbf{w}}$  represents the cumulative distribution function of  $Y$ . The function  $\sigma :[0,1]\to \mathbb{R}_+$  is a nonnegative, nondecreasing function that integrates to 1, known as the spectrum of the rank-based loss. The discrete form of (3) is consistent with (2), with  $\sigma_{i} = \int_{(i - 1) / n}^{i / n}\sigma (t)dt$  [36]. The spectral risk builds upon previous aggregate losses that have been widely used to formulate learning objectives, such as the average risk [46], the maximum risk [43], the average top-  $k$  risk [21] and conditional value-at-risk [39]. By choosing a different spectral  $\sigma (t)$ , such as the superquantile [30], the extremile [15], and the exponential spectral risk measure [11], can be constructed for various spectral risks.
Human-Aligned Risk With the growing societal deployment of machine learning models for aiding human decision-making, these models must possess qualities such as fairness, in addition to strong average performance. Given the significance of these additional requirements, amendments to risk measures and extra constraints have been introduced [28, 23, 19]. Inspired by the cumulative prospect theory (CPT) [45], which boasts substantial evidence for its proficiency in modeling human decisions, investigations have been carried out in bandit [24] and reinforcement learning [41]. Recently, [31] also utilized the concept of CPT to present a novel notion of the empirical human risk minimization (EHRM) in supervised learning. The weight assigned in (3) is defined as  $\sigma_{i} = w_{a,b}\left(\frac{i}{n}\right)$ , where
$$
w _ {a, b} (t) = \frac {3 - 3 b}{a ^ {2} - a + 1} \left(3 t ^ {2} - 2 (a + 1) t + a\right) + 1.
$$
These weights assign greater significance to extreme individual losses, yielding an S-shaped CPT-weighted cumulative distribution. Moreover, we consider another CPT-weight commonly employed in decision making [45, 41]. Let  $\mathbf{z} = -\mathbf{y} \odot (X\mathbf{w})$  and  $B$  be a reference point. Unlike previous weight settings,  $\sigma_{i}$  is related to the value of  $z_{[i]}$  and is defined as
$$
\sigma_ {i} \left(z _ {[ i ]}\right) = \left\{ \begin{array}{l l} \omega_ {-} \left(\frac {i}{n}\right) - \omega_ {-} \left(\frac {i - 1}{n}\right), & z _ {[ i ]} \leq B, \\ \omega_ {+} \left(\frac {n - i + 1}{n}\right) - \omega_ {+} \left(\frac {n - i}{n}\right), & z _ {[ i ]} > B, \end{array} \right. \tag {4}
$$
$$
\omega_ {+} (p) = \frac {p ^ {\gamma}}{\left(p ^ {\gamma} + (1 - p) ^ {\gamma}\right) ^ {1 / \gamma}}, \quad \omega_ {-} (p) = \frac {p ^ {\delta}}{\left(p ^ {\delta} + (1 - p) ^ {\delta}\right) ^ {1 / \delta}}, \tag {5}
$$
with  $\gamma$  and  $\delta$  as hyperparameters.
Ranked-Range Loss The average of ranked-range aggregate (AoRR) loss follows the same structure as (2), where
$$
\sigma_ {i} = \left\{ \begin{array}{l l} \frac {1}{k - m}, & i \in \{m + 1, \dots , k \}, \\ 0, & \text {o t h e r w i s e}, \end{array} \right. \tag {6}
$$
with  $1 \leq m < k \leq n$ . The ranked-range loss effectively handles outliers, ensuring the robustness of the model against anomalous observations in the dataset [27]. It is clear that AoRR includes the average loss, the maximum loss, the average top- $k$  loss, value-at-risk [39] and the median loss [34]. [27] utilized the difference-of-convex algorithm (DCA) [40] to solve the AoRR aggregate loss minimization problem, which can be expressed as the difference of two convex problems.
2.2 Weakly Convex Function and Moreau Envelope
A function  $g: \mathbb{R}^d \to \mathbb{R}$  is said to be  $c$ -weakly convex for some  $c > 0$  if the function  $g + \frac{c}{2} \| \cdot \|^{2}$  is convex. The class of weakly convex functions is extensive, encompassing all convex functions as well as smooth functions with Lipschitz-continuous gradients [37]. In our framework, we consider weakly convex functions as regularizers. It is worth noting that weakly convex functions constitute a rich class of regularizers. Convex  $L^{p}$  norms with  $p \geq 1$  and nonconvex penalties such as the Minimax Concave Penalty (MCP) [52] and the Smoothly Clipped Absolute Deviation (SCAD) [20] are examples of weakly convex functions [7].
Next, we define the Moreau envelope of  $c$ -weakly convex function  $g(\boldsymbol{w})$ , with proximal parameter  $0 < \gamma < \frac{1}{c}$ :
$$
M _ {g, \gamma} (\boldsymbol {w}) = \min  _ {\boldsymbol {x}} \left\{g (\boldsymbol {x}) + \frac {1}{2 \gamma} \| \boldsymbol {x} - \boldsymbol {w} \| ^ {2} \right\}. \tag {7}
$$
The proximal operator of  $g$  with parameter  $\gamma$  is given by
$$
\operatorname {p r o x} _ {g, \gamma} (\boldsymbol {w}) = \arg \min _ {\boldsymbol {x}} \left\{g (\boldsymbol {x}) + \frac {1}{2 \gamma} \| \boldsymbol {x} - \boldsymbol {w} \| ^ {2} \right\}.
$$
We emphasize that  $\mathrm{prox}_{g,\gamma}(\cdot)$  is a single-valued mapping, and  $M_{g,\gamma}(\boldsymbol{w})$  is well-defined since the objective function in (7) is strongly convex for  $\gamma \in (0,c^{-1})$  [7].
The Moreau envelope is commonly employed to smooth weakly convex functions. From [7, Lemma 3.1-3.2] we have
$$
\nabla M _ {g, \gamma} (\boldsymbol {w}) = \gamma^ {- 1} \left(\boldsymbol {w} - \operatorname {p r o x} _ {g, \gamma} (\boldsymbol {w})\right) \in \partial g \left(\operatorname {p r o x} _ {g, \gamma} (\boldsymbol {w})\right). \tag {8}
$$
Here  $\partial$  represents Clarke generalized gradient [10]. For locally Lipschitz continuous function  $f: \mathbb{R}^d \to \mathbb{R}$ , its Clarke generalized gradient is denoted by  $\partial f(\pmb{x})$ . We say  $\pmb{x}$  is stationary for  $f$  if  $\pmb{0} \in \partial f(\pmb{x})$ . For convex functions, Clarke generalized gradient coincides with subgradient in the sense of convex analysis. Under assumptions in Section 3.3, both  $\Omega(z) \coloneqq \sum_{i=1}^{n} \sigma_i l_{[i]}(z)$  and  $g(\pmb{w})$  are locally Lipschitz continuous [12, 47]. Thus, the Clarke generalized gradients of  $\Omega(z)$  and  $g(\pmb{w})$  are well defined, and so is that of  $L_\rho(\pmb{w}, z; \lambda)$  defined in Section 3.1.
3 Optimization Algorithm
Firstly, we make the basic assumption about the loss function  $l$  and the regularizer  $g$ :
Assumption 1 The individual loss function  $l(\cdot) : \mathcal{D} \to \mathbb{R}$  is convex and monotonically increasing, where  $\mathcal{D} \subset \mathbb{R}$  is the domain of  $l$ . The regularizer  $g(\cdot) : \mathbb{R}^d \to \mathbb{R} \cup \{\infty\}$  is proper, lower semicontinuous, and c-weakly convex. Furthermore,  $l(\cdot)$  and  $g(\cdot)$  are lower bounded by 0.
Many interesting and widely used individual loss functions satisfy this assumption, such as logistic loss, hinge loss, and exponential loss. Our algorithm described below can also handle monotonically decreasing loss functions, by simply replacing  $\sigma_{i}$  with  $\sigma_{n - i + 1}$  in (1). We assume  $l$  and  $g$  are lower bounded, and take the infimum to be 0 for convenience. The lower bounded property of  $l$  implies that the  $\Omega$  is also lower bounded by 0 since  $\sigma_{i} \geq 0$  for all  $i$ .
3.1 The ADMM Framework
By Assumption 1, the original problem is equivalent to
$$
\min  _ {\boldsymbol {w}} \sum_ {i = 1} ^ {n} \sigma_ {i} l \left(\left(D \boldsymbol {w}\right) _ {[ i ]}\right) + g (\boldsymbol {w}),
$$
where  $l$  is the individual loss function,  $D = -\operatorname{diag}(\boldsymbol{y})X$  and  $\operatorname{diag}(\boldsymbol{y})$  denotes the diagonal matrix whose diagonal entries are the vector  $\boldsymbol{y}$ .
To make use of the philosophy of ADMM, by introducing an auxiliary variable  $z = Dw$ , we reformulate the problem as
$$
\min  _ {\boldsymbol {w}, \boldsymbol {z}} \quad \Omega (\boldsymbol {z}) + g (\boldsymbol {w}) \quad \text {s . t .} \quad \boldsymbol {z} = D \boldsymbol {w}, \tag {9}
$$
where  $\Omega (\pmb {z}) = \sum_{i = 1}^{n}\sigma_{i}l\left(z_{[i]}\right)$
Note that for human-aligned risk,  $\sigma_{i}$  should be written as  $\sigma_{i}(z_{[i]})$  since it is a piecewise function with two different values in (4), but for simplicity, we still write it as  $\sigma_{i}$ .
The augmented Lagrangian function can then be expressed as
$$
L _ {\rho} (\boldsymbol {w}, \boldsymbol {z}; \boldsymbol {\lambda}) = \sum_ {i = 1} ^ {n} \Omega (\boldsymbol {z}) + \frac {\rho}{2} | | \boldsymbol {z} - D \boldsymbol {w} + \frac {\boldsymbol {\lambda}}{\rho} | | ^ {2} + g (\boldsymbol {w}) - \frac {| | \boldsymbol {\lambda} | | ^ {2}}{2 \rho}.
$$
The ADMM, summarised in Algorithm 1, cyclically updates  $\boldsymbol{w}$ ,  $\boldsymbol{z}$  and  $\lambda$ , by solving the  $\boldsymbol{w}$ - and  $\boldsymbol{z}$ -subproblems and adopting a dual ascent step for  $\lambda$ . In Algorithm 1, we assume the  $\boldsymbol{w}$ -subproblem can be solved exactly (for simplicity). When  $g(\boldsymbol{w})$  is a smooth function, the  $\boldsymbol{w}$ -subproblem is a smooth problem and can be solved by various gradient-based algorithms. Particularly, the  $\boldsymbol{w}$ -subproblem is the least squares and admits a closed-form solution if  $g(\boldsymbol{w}) \equiv 0$  or  $g(\boldsymbol{w}) = \frac{\mu}{2} ||\boldsymbol{w}||_2^2$ , where  $\mu > 0$  is a regularization parameter. When  $g(\boldsymbol{w})$  is nonsmooth, if  $\mathrm{prox}_{g,\gamma}(\boldsymbol{w})$  is easy to compute, we can adopt the proximal gradient method or its accelerated version [4]. Particularly, if  $g(\boldsymbol{w}) = \frac{\mu}{2} ||\boldsymbol{w}||_1$ , then the subproblem of  $\boldsymbol{w}$  becomes a LASSO problem, solvable by numerous effective methods such as the Coordinate Gradient Descent Algorithm (CGDA) [22], the Smooth L1 Algorithm (SLA) [42], and the Fast Iterative Shrinkage-Thresholding Algorithms (FISTA) [4].
Algorithm 1 ADMM framework
Input:  $X, y, w^0, z^0, \lambda^0, \rho, r,$  and  $\sigma_i, i = 1, \dots, n$ .
1: for all  $k = 0,1,\ldots$  do
2:  $z^{k + 1} = \arg \min_{\boldsymbol{z}}\Omega (\boldsymbol {z}) + \frac{\rho}{2} ||\boldsymbol {z} - D\boldsymbol{w}^k +\frac{\boldsymbol{\lambda}^k}{\rho} ||^2.$
3:  $\pmb{w}^{k + 1} = \arg \min_{\pmb{w}}\frac{\rho}{2} ||\pmb{z}^{k + 1} - D\pmb{w} + \frac{\pmb{\lambda}^k}{\rho} ||^2 +g(\pmb {w}) + \frac{r}{2}\| \pmb {w} - \pmb{w}^k\| ^2.$
4:  $\pmb{\lambda}^{k + 1} = \pmb{\lambda}^k +\rho (\pmb{z}^{k + 1} - D\pmb{w}^{k + 1}).$
5: end for
At first glance, the  $z$ -subproblem is nonconvex and difficult to solve. However, we can solve an equivalent convex chain-constrained program that relies on sorting and utilize the pool adjacent violators algorithm (PAVA). More specifically, we follow the approach presented in [13, Lemma 3] and introduce the auxiliary variable  $\boldsymbol{m} = D\boldsymbol{w}^k - \frac{\boldsymbol{\lambda}^k}{\rho}$  (we remove the superscript for  $\boldsymbol{m}$  for simplicity). This enables us to express the  $z$ -subproblem in the equivalent form below
$$
\min  _ {z} \quad \sum_ {i = 1} ^ {n} \sigma_ {i} l \left(z _ {p _ {i}}\right) + \frac {\rho}{2} \left(z _ {p _ {i}} - m _ {p _ {i}}\right) ^ {2} \quad \text {s . t .} z _ {p _ {1}} \leq z _ {p _ {2}} \leq \dots \leq z _ {p _ {n}},
$$
where  $\{p_1,p_2,\ldots ,p_n\}$  is a permutation of  $\{1,\dots ,n\}$  such that  $m_{p_1}\leqslant m_{p_2}\leqslant \dots \leqslant m_{p_n}$
3.2 The Pool Adjacent Violators Algorithm (PAVA)
To introduce our PAVA, for ease of notation and without loss of generality, we consider  $m_{1} \leqslant m_{2} \leqslant \dots \leqslant m_{n}$ , i.e., the following problem
$$
\min  _ {z} \quad \sum_ {i = 1} ^ {n} \sigma_ {i} l \left(z _ {i}\right) + \frac {\rho}{2} \left(z _ {i} - m _ {i}\right) ^ {2} \quad \text {s . t .} z _ {1} \leq z _ {2} \leq \dots \leq z _ {n}. \tag {10}
$$
The above problem constitutes a convex chain-constrained program. Although it can be solved by existing convex solvers, the PAVA [6] is often more efficient.
The PAVA is designed to solve the following problem
$$
\min  _ {z} \quad \sum_ {i = 1} ^ {n} \theta_ {i} (z _ {i}) \qquad \text {s . t .} z _ {1} \leq z _ {2} \leq \dots \leq z _ {n},
$$
where each  $\theta_{i}$  represents a univariate convex function. In our problem,  $\theta_{i}(z_{i}) = \sigma_{i}l(z_{i}) + \frac{\rho}{2} (z_{i} - m_{i})^{2}$ . The PAVA maintains a set  $J$  that partitions the indices  $\{1,2,\ldots ,n\}$  into consecutive blocks  $\{[s_1 + 1,s_2],[s_2 + 1,s_3],\dots ,[s_k + 1,s_{k + 1}]\}$  with  $s_1 = 0$  and  $s_{k + 1} = N$ . Here,  $[a,b]$  represents the index set  $\{a,a + 1,\dots ,b\}$  for positive integers  $a < b$ , and by convention, we define  $[a,a] = a$ . A block  $[p,q]$  is termed a single-valued block if every  $z_{i}$  in the optimal solution of the following problem has the same value,
$$
\min  _ {z _ {p}, \dots , z _ {q}} \quad \sum_ {i = p} ^ {q} \theta_ {i} (z _ {i}) \qquad \text {s . t .}   z _ {p} \leq z _ {p + 1} \leq \dots \leq z _ {q},
$$
i.e.,  $z_{p}^{*} = z_{p + 1}^{*} = \dots = z_{q}^{*}$ . In line with existing literature, we use  $v_{[p,q]}$  to denote this value. For two consecutive blocks  $[p,q]$  and  $[q + 1,r]$ , if  $v_{[p,q]} \leq v_{[q + 1,r]}$ , the
two blocks are in-order; otherwise, they are out-of-order. Similarly, the consecutive blocks  $\{[s_k,s_{k + 1}],[s_{k + 1} + 1,s_{k + 2}],\dots ,[s_{k + t} + 1,s_{k + t + 1}]\}$  are said to be in-order if  $v_{[s_k s_{k + 1}]}\leq v_{[s_{k + 1} + 1,s_{k + 2}]}\leq \dots \leq v_{[s_{k + t} + 1,s_{k + t + 1}]}$ ; otherwise they are out-of-order. Particularly, if  $v_{[s_k s_{k + 1}]}>$ $v_{[s_{k + 1} + 1,s_{k + 2}]} > \dots >v_{[s_{k + t} + 1,s_{k + t + 1}]}$ , the consecutive blocks are said to be consecutive out-of-order. The PAVA initially partitions each integer from 1 to  $n$  into single-valued blocks  $[i,i]$  for  $i = 1,\ldots ,n$ . When there are consecutive out-of-order single-valued blocks  $[p,q]$  and  $[q + 1,r]$ , the PAVA merges these two blocks by replacing them with the larger block  $[p,r]$ . The PAVA terminates once all the single-valued blocks are in-order.
Algorithm 2 A refined pool-adjacent-violators algorithm for solving problem (10)
Input:  $J = \{[1,1],[2,2],\ldots ,[n,n]\} ,\theta_{i},i = 1,2,\ldots ,n.$    
Output:  $\{y_1^*,y_2^*,\dots ,y_N^*\}$    
1: for each  $[i,i]\in J$  do   
2: Compute the minimizer  $v_{[i,i]}$  of  $\theta_{i}(\pmb {y})$    
3: end for   
4: while exists out-of-order blocks in  $J$  do   
5: Find consecutive out-of-order blocks  $C = \{[s_k,s_{k + 1}],[s_{k + 1} + 1,s_{k + 2}],\dots ,[s_{k + t}+$ $1,s_{k + t + 1}]\}$    
6:  $J\gets J\backslash C\cup \{[s_k,s_{k + t + 1}]\}$  , compute the minimizer  $v_{[s_k,s_{k + t + 1}]}$  of  $\sum_{i = s_k}^{s_k + t + 1}\theta_i(z)$    
7: end while   
8: for each  $[m,n]\in J$  do   
9:  $y_{i}^{*} = v_{[m,n]}$ $\forall i = m,m + 1,\dots ,n$    
10: end for
Traditional PAVA processes the merging of out-of-order blocks one by one, identifying two consecutive out-of-order blocks and then solving a single unconstrained convex minimization problem to merge them. For constant  $\sigma_{i}$ , our proposed Algorithm 2, however, leverages the unique structure of our problem to improve the PAVA's efficiency. Specifically, we can identify and merge multiple consecutive out-of-order blocks, thereby accelerating computation. This acceleration is facilitated by the inherent properties of our problem. Notably, in case the spectral risk and the rank-ranged loss, the function  $\theta_{i}(z_{i})$  demonstrates strong convexity due to the presence of the quadratic term  $\frac{\rho}{2} (z_i - m_i)^2$  and the convex nature of  $l(z_{i})$ . This key observation leads us to the following proposition.
Proposition 1 For constant  $\sigma_{i}$ , suppose  $v_{[m,n]} > v_{[n+1,p]}$ , the blocks  $[m,n]$  and  $[n+1,p]$  are consecutive out-of-order blocks. We merge these two blocks into  $[m,p]$ . Then the block optimal value, denoted by  $v_{[m,p]}$ , satisfies  $v_{[n+1,p]} \leq v_{[m,p]} \leq v_{[m,n]}$ .