Title: A Unified Framework for Rank-based Loss Minimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: rank-based loss

ADMM

nonconvex nonsmooth optimization

conditional Value-at-Risk

human-aligned risk

ranked range loss
Abstract: The empirical loss, commonly referred to as the average loss, is extensively utilized for training machine learning models. However, in order to address the diverse performance requirements of machine learning models, the use of the rank-based loss is prevalent, replacing the empirical loss in many cases. The rank-based loss comprises a weighted sum of sorted individual losses, encompassing both convex losses like the spectral risk, which includes the empirical risk and conditional value-at-risk, and nonconvex losses such as the human-aligned risk and the sum of the ranked range loss. In this paper, we introduce a unified framework for the optimization of the rank-based loss through the utilization of a proximal alternating direction method of multipliers. We demonstrate the convergence and convergence rate of the proposed algorithm under mild conditions. Experiments conducted on synthetic and real datasets illustrate the effectiveness and efficiency of the proposed algorithm.

A Unified Framework for Rank-based Loss Minimization

Abstract

The empirical loss, commonly referred to as the average loss, is extensively utilized for training machine learning models. However, in order to address the diverse performance requirements of machine learning models, the use of the rank-based loss is prevalent, replacing the empirical loss in many cases. The rank-based loss comprises a weighted sum of sorted individual losses, encompassing both convex losses like the spectral risk, which includes the empirical risk and conditional value-at-risk, and nonconvex losses such as the human-aligned risk and the sum of the ranked range loss. In this paper, we introduce a unified framework for the optimization of the rank-based loss through the utilization of a proximal alternating direction method of multipliers. We demonstrate the convergence and convergence rate of the proposed algorithm under mild conditions. Experiments conducted on synthetic and real datasets illustrate the effectiveness and efficiency of the proposed algorithm.

1 Introduction

The empirical risk function is the cornerstone of machine learning. By minimizing this function, machine learning models typically achieve commendable average performance. However, as these models find applications across a diverse spectrum of real-world scenarios, the evaluation standards for machine learning performance have evolved to include factors such as risk aversion and fairness [49, 9]. This has led to the empirical loss function often being supplanted by other loss functions, many of which fall under the category of the rank-based loss, which consists of a weighted sum of sorted individual losses. We consider the following optimization problem in this paper, which minimizes the rank-based loss plus a regularizer:

Here, l ( u ) ≜ [ l ( u 1 ) , . . . , l ( u n )] ⊤ : R n → R n represents a vector-valued mapping where l : R → R denotes an individual loss function and u i is the i -th element of u , l [1] ( · ) ⩽ · · · ⩽ l [ n ] ( · ) denotes the order statistics of the empirical loss distribution, g : R d → R is a regularizer that induces desired structures, w ∈ R d represents the parameters of the linear model, X ∈ R n × d is the data matrix, and y ∈ {± 1 } n is the label vector. The subscript [ i ] denotes the i -th smallest element, σ i ∈ [0 , 1] is the weight corresponding to the i -th smallest element, and ' ⊙ " represents the Hadamard product. Depending on the values of σ i , the rank-based loss can encompass a significant range of losses often

used in machine learning and other fields: the L-risk or the spectral risk [1, 2, 35], the trimmed risk or the sum of the ranked-range loss [27], the human-aligned risk based on the cumulative prospect theory [31].

Related work. When the values of σ i are constant and monotonically increase, i.e., σ 1 ≤ · · · ≤ σ n , the rank-based loss is referred to as the spectral risk [1]. Several methods have been proposed in the literature, including: a derivative-free method proposed by [25] that utilizes a stochastic smoothing technique; an adaptive sampling algorithm proposed by [14] for the minimization of conditional value at risk (CVaR), which is a special case of the spectral risk; and a stochastic algorithm developed by [36] for minimizing the general spectral risk by characterizing the subdifferential of the rank-based loss. In the case of nonconvex rank-based losses, [27] utilizes the difference of convex algorithm for minimizing the average of ranked-range (AoRR) loss. The nonconvex human-aligned risk was minimized by [31] by directly computing the gradient of the function, even though the rank-based loss is nondifferentiable, without any convergence guarantee. Despite the increasing applications of the rank-based loss, a unified framework for addressing the rank-based loss minimization remains elusive. Moreover, stochastic methods may encounter bias issues [36], existing methods for the spectral risk cannot deal with nonconvex rank-based losses, and existing methods for human-aligned risk lack convergence guarantee [31].

Our Contributions. We present a unified framework for the minimization of the rank-based loss to overcome the challenges of existing methods. Specifically, we focus on monotone increasing loss functions, such as the hinge loss and the logistic loss, and use weakly convex regularizers. We leverage the alternating direction multiplier method (ADMM), which is widely used in solving nonsmooth nonconvex composite problems [8, 32, 48]. To utilize the ADMM framework, we introduce an auxiliary variable to represent the product of the data matrix and parameter vector. The two subproblems are either strongly convex (with a proximal term), which can be solved by various existing methods, or can be efficiently solved by the pool adjacent violators algorithm (PA V A) [6]. We demonstrate that our algorithm can find an ϵ -KKT point in at most O (1 /ϵ 2 ) iterations under mild conditions. To relax the assumptions further, we propose a variant of the ADMM with a smoothing technique when the regularizer is nonsmooth. Notably, when the Moreau envelope is applied to smooth the nonsmooth regularizer, we show that our algorithm can find an ϵ -KKT point within O (1 /ϵ 4 ) iterations.

Our contributions can be summarized as follows:

We propose a unified framework for the rank-based loss minimization for monotonically increasing loss functions. This approach is versatile, effectively dealing with both convex and nonconvex loss functions by allowing different settings of weights σ i .

Furthermore, the regularizer in our problem can be weakly convex functions, extending existing works that only consider convex or smooth regularizers, such as the l 1 and l 2 penalty.

We theoretically demonstrate that our ADMM algorithm converges to an ϵ -approximate KKT point under different assumptions.

The experiments in three different aggregate loss function frameworks demonstrate the advantages of the proposed algorithm.

2 Preliminaries

In this section, we explore the wide variety of rank-based losses and introduce the Moreau envelope as a smooth approximation of weakly convex functions.

2.1 Rank-Based Loss

Let { ( X 1 , y 1 ) , . . . , ( X n , y n ) } be an i.i.d. sample set from a distribution P over a sample space ( X , Y ) , and L be the individual loss function. Then Y = L ( w , X, y ) is also a random variable, and Y i = L ( w , X i , y i ) represents the i -th training loss with respect to the sample ( X i , y i ) , i ∈ { 1 , . . . , n } . Let Y [1] ≤ . . . ≤ Y [ n ] be the order statistics. The rank-based loss function in problem (1) equals

Spectral Risk The spectral risk, as defined in [1], is given by:

where VaR t ( w ) = inf { u : F w ( u ) ≥ t } denotes the quantile function, and F w represents the cumulative distribution function of Y . The function σ : [0 , 1] → R + is a nonnegative, nondecreasing function that integrates to 1, known as the spectrum of the rank-based loss. The discrete form of (3) is consistent with (2), with σ i = ∫ i/n ( i -1) /n σ ( t ) dt [36]. The spectral risk builds upon previous aggregate losses that have been widely used to formulate learning objectives, such as the average risk [46], the maximum risk [43], the average topk risk [21] and conditional value-at-risk [39]. By choosing a different spectral σ ( t ) , such as the superquantile [30], the extremile [15], and the exponential spectral risk measure [11], can be constructed for various spectral risks.

Human-Aligned Risk With the growing societal deployment of machine learning models for aiding human decision-making, these models must possess qualities such as fairness, in addition to strong average performance. Given the significance of these additional requirements, amendments to risk measures and extra constraints have been introduced [28, 23, 19]. Inspired by the cumulative prospect theory (CPT) [45], which boasts substantial evidence for its proficiency in modeling human decisions, investigations have been carried out in bandit [24] and reinforcement learning [41]. Recently, [31] also utilized the concept of CPT to present a novel notion of the empirical human risk minimization (EHRM) in supervised learning. The weight assigned in (3) is defined as σ i = w a,b ( i n ) , where

These weights assign greater significance to extreme individual losses, yielding an S-shaped CPTweighted cumulative distribution. Moreover, we consider another CPT-weight commonly employed in decision making [45, 41]. Let z = -y ⊙ ( X w ) and B be a reference point. Unlike previous weight settings, σ i is related to the value of z [ i ] and is defined as

where

with γ and δ as hyperparameters.

Ranked-Range Loss The average of ranked-range aggregate (AoRR) loss follows the same structure as (2), where

with 1 ≤ m < k ≤ n . The ranked-range loss effectively handles outliers, ensuring the robustness of the model against anomalous observations in the dataset [27]. It is clear that AoRR includes the average loss, the maximum loss, the average topk loss, value-at-risk [39] and the median loss [34]. [27] utilized the difference-of-convex algorithm (DCA) [40] to solve the AoRR aggregate loss minimization problem, which can be expressed as the difference of two convex problems.

2.2 Weakly Convex Function and Moreau Envelope

A function g : R d → R is said to be c -weakly convex for some c > 0 if the function g + c 2 ∥ · ∥ 2 is convex. The class of weakly convex functions is extensive, encompassing all convex functions as well as smooth functions with Lipschitz-continuous gradients [37]. In our framework, we consider weakly convex functions as regularizers. It is worth noting that weakly convex functions constitute a rich class of regularizers. Convex L p norms with p ≥ 1 and nonconvex penalties such as the Minimax Concave Penalty (MCP) [52] and the Smoothly Clipped Absolute Deviation (SCAD) [20] are examples of weakly convex functions [7].

Next, we define the Moreau envelope of c -weakly convex function g ( w ) , with proximal parameter 0 < γ < 1 c :

The proximal operator of g with parameter γ is given by

We emphasize that prox g,γ ( · ) is a single-valued mapping, and M g,γ ( w ) is well-defined since the objective function in (7) is strongly convex for γ ∈ ( 0 , c -1 ) [7].

The Moreau envelope is commonly employed to smooth weakly convex functions. From [7, Lemma 3.1-3.2] we have

Here ∂ represents Clarke generalized gradient [10]. For locally Lipschitz continuous function f : R d → R , its Clarke generalized gradient is denoted by ∂f ( x ) . We say x is stationary for f if 0 ∈ ∂f ( x ) . For convex functions, Clarke generalized gradient coincides with subgradient in the sense of convex analysis. Under assumptions in Section 3.3, both Ω( z ) := ∑ n i =1 σ i l [ i ] ( z ) and g ( w ) are locally Lipschitz continuous [12, 47]. Thus, the Clarke generalized gradients of Ω( z ) and g ( w ) are well defined, and so is that of L ρ ( w , z ; λ ) defined in Section 3.1.

3 Optimization Algorithm

Firstly, we make the basic assumption about the loss function l and the regularizer g :

Assumption 1 The individual loss function l ( · ) : D → R is convex and monotonically increasing, where D ⊂ R is the domain of l . The regularizer g ( · ) : R d → R ∪ {∞} is proper, lower semicontinuous, and c -weakly convex. Furthermore, l ( · ) and g ( · ) are lower bounded by 0.

Many interesting and widely used individual loss functions satisfy this assumption, such as logistic loss, hinge loss, and exponential loss. Our algorithm described below can also handle monotonically decreasing loss functions, by simply replacing σ i with σ n -i +1 in (1). We assume l and g are lower bounded, and take the infimum to be 0 for convenience. The lower bounded property of l implies that the Ω is also lower bounded by 0 since σ i ≥ 0 for all i .

3.1 The ADMM Framework

By Assumption 1, the original problem is equivalent to

where l is the individual loss function, D = -diag( y ) X and diag( y ) denotes the diagonal matrix whose diagonal entries are the vector y .

To make use of the philosophy of ADMM, by introducing an auxiliary variable z = D w , we reformulate the problem as

Note that for human-aligned risk, σ i should be written as σ i ( z [ i ] ) since it is a piecewise function with two different values in (4), but for simplicity, we still write it as σ i .

The augmented Lagrangian function can then be expressed as

The ADMM, summarised in Algorithm 1, cyclically updates w , z and λ , by solving the w - and z -subproblems and adopting a dual ascent step for λ . In Algorithm 1, we assume the w - subproblem can be solved exactly (for simplicity). When g ( w ) is a smooth function, the w -subproblem is a smooth problem and can be solved by various gradient-based algorithms. Particularly, the w -subproblem is the least squares and admits a closed-form solution if g ( w ) ≡ 0 or g ( w ) = µ 2 || w || 2 2 , where µ > 0 is a regularization parameter. When g ( w ) is nonsmooth, if prox g,γ ( w ) is easy to compute, we can adopt the proximal gradient method or its accelerated version [4]. Particularly, if g ( w ) = µ 2 || w || 1 , then the subproblem of w becomes a LASSO problem, solvable by numerous effective methods such as the Coordinate Gradient Descent Algorithm (CGDA) [22], the Smooth L1 Algorithm (SLA) [42], and the Fast Iterative Shrinkage-Thresholding Algorithms (FISTA) [4].

Algorithm 1 ADMMframework

Input: X , y , w 0 , z 0 , λ 0 , ρ , r , and σ i , i = 1 , ..., n .

1: for all k = 0 , 1 , ... do

2: z k +1 = arg min z Ω( z ) + ρ 2 || z - D w k + λ k ρ || 2 .

5: end for

At first glance, the z -subproblem is nonconvex and difficult to solve. However, we can solve an equivalent convex chain-constrained program that relies on sorting and utilize the pool adjacent violators algorithm (PAVA). More specifically, we follow the approach presented in [13, Lemma 3] and introduce the auxiliary variable m = D w k -λ k ρ (we remove the superscript for m for simplicity). This enables us to express the z -subproblem in the equivalent form below

where { p 1 , p 2 , . . . , p n } is a permutation of { 1 , . . . , n } such that m p 1 ⩽ m p 2 ⩽ · · · ⩽ m p n .

3.2 The Pool Adjacent Violators Algorithm (PAVA)

To introduce our PAVA, for ease of notation and without loss of generality, we consider m 1 ⩽ m 2 ⩽ · · · ⩽ m n , i.e., the following problem

The above problem constitutes a convex chain-constrained program. Although it can be solved by existing convex solvers, the PA V A [6] is often more efficient.

The PAVA is designed to solve the following problem

n

∑

i

z

=1

where each θ i represents a univariate convex function. In our problem, θ i ( z i ) = σ i l ( z i ) + ρ 2 ( z i -m i ) 2 . The PAVA maintains a set J that partitions the indices { 1 , 2 , . . . , n } into consecutive blocks { [ s 1 +1 , s 2 ] , [ s 2 +1 , s 3 ] , · · · , [ s k +1 , s k +1 ] } with s 1 = 0 and s k +1 = N . Here, [ a, b ] represents the index set { a, a +1 , . . . , b } for positive integers a < b , and by convention, we define [ a, a ] = a . A block [ p, q ] is termed a single-valued block if every z i in the optimal solution of the following problem has the same value,

i.e., z ∗ p = z ∗ p +1 = · · · = z ∗ q . In line with existing literature, we use v [ p,q ] to denote this value. For two consecutive blocks [ p, q ] and [ q + 1 , r ] , if v [ p,q ] ≤ v [ q +1 ,r ] , the min

θ

i

(

z

i

)

z

1

z

2

≤ · · · ≤

z

n

,

two blocks are in-order ; otherwise, they are out-of-order . Similarly, the consecutive blocks { [ s k , s k +1 ] , [ s k +1 +1 , s k +2 ] , · · · , [ s k + t +1 , s k + t +1 ] } are said to be in-order if v [ s k s k +1 ] ≤ v [ s k +1 +1 ,s k +2 ] ≤ · · · ≤ v [ s k + t +1 ,s k + t +1 ] ; otherwise they are out-of-order . Particularly, if v [ s k s k +1 ] > v [ s k +1 +1 ,s k +2 ] > · · · > v [ s k + t +1 ,s k + t +1 ] , the consecutive blocks are said to be consecutive out-oforder . The PAVA initially partitions each integer from 1 to n into single-valued blocks [ i, i ] for i = 1 , . . . , n . When there are consecutive out-of-order single-valued blocks [ p, q ] and [ q +1 , r ] , the PAVA merges these two blocks by replacing them with the larger block [ p, r ] . The PAVA terminates once all the single-valued blocks are in-order.

Traditional PAVA processes the merging of out-of-order blocks one by one, identifying two consecutive out-of-order blocks and then solving a single unconstrained convex minimization problem to merge them. For constant σ i , our proposed Algorithm 2, however, leverages the unique structure of our problem to improve the PAVA's efficiency. Specifically, we can identify and merge multiple consecutive out-of-order blocks, thereby accelerating computation. This acceleration is facilitated by the inherent properties of our problem. Notably, in case the spectral risk and the rank-ranged loss, the function θ i ( z i ) demonstrates strong convexity due to the presence of the quadratic term ρ 2 ( z i -m i ) 2 and the convex nature of l ( z i ) . This key observation leads us to the following proposition.

Proposition 1 For constant σ i , suppose v [ m,n ] > v [ n +1 ,p ] , the blocks [ m,n ] and [ n + 1 , p ] are consecutive out-of-order blocks. We merge these two blocks into [ m,p ] . Then the block optimal value, denoted by v [ m,p ] , satisfies v [ n +1 ,p ] ≤ v [ m,p ] ≤ v [ m,n ] .

Proposition 1 provides a crucial insight: when we encounter consecutive out-of-order blocks with a length greater than 2, we can merge them simultaneously, rather than performing individual block merges. This approach significantly improves the efficiency of the algorithm. To illustrate this concept, consider a scenario where we have a sequence of values arranged as v [1 , 2] > v [3 , 4] > v [5 , 6] . Initially, we merge the blocks [1 , 2] and [3 , 4] , resulting in v [1 , 4] ≥ v [3 , 4] > v [5 , 6] . However, since v [1 , 4] is still greater than v [5 , 6] , we need to merge the blocks [1 , 4] and [5 , 6] as well. Rather than calculating v [1 , 4] separately, we can streamline the process by directly merging the entire sequence into a single block, namely [1 , 6] , in a single operation. By leveraging this approach, we eliminate the need for intermediate calculations and reduce the computational burden associated with merging individual blocks. This results in a more efficient version of the PA V A. Further details can be found in Appendix B.3.

It is worth noting that when θ i ( z i ) is convex with respect to z i , the PAVA guarantees to find a global minimizer [6]. However, when considering the empirical human risk minimization with CPT-weight in (4), the function θ i ( z i ) is nonconvex, which is because σ i is a piecewise function with two different values. In such cases, we can still find a point that satisfies the first-order condition [12, Theorem 3]. In summary, we always have

3.3 Convergence Analysis of Algorithm 1

We now demonstrate that Algorithm 1 is guaranteed to converge to an ϵ -KKT point of problem (9). To this end, we shall make the following assumptions.

It is worth noting that Assumption 2 is commonly employed in ADMM approaches [51, 44, 3].

Next, we present our convergence result based on the aforementioned assumptions. As mentioned in Section 3.2, the PA V A can always find a solution for the z -subproblem that satisfies the first-order condition (11). When σ i is constant, which is the case of the spectral risk and the AoRR risk, the subproblems of PAVA are strongly convex, and we can observe the descent property of the z -subproblem:

However, if σ i has different values for z less than and larger than the reference point B as in (4), the subproblems of PAVA may lose convexity, and the descent property may not hold. Thus, we assume that (12) holds for simplicity. By noting that the w -subproblem can be solved exactly as it is strongly convex, we make the following assumption.

Assumption 3 The w -subproblem is solved exactly, and the z -subproblem is solved such that (11) and (12) hold.

Theorem 1 Assume that Assumptions 1, 2 and 3 hold. Then Algorithm 1 can find an ϵ -KKT point ( z k +1 , w k +1 , λ k +1 ) of problem (9) within O (1 /ϵ 2 ) iterations, i.e.,

where dist( a , A ) = min {∥ a -x ∥ : x ∈ A } defines the distance of a point a to a set A .

In the absence of Assumption 2, a common assumption for nonconvex ADMM is that g ( w ) possesses a Lipschitz continuous gradient [48, 32], e.g., g ( w ) = µ 2 ∥ w ∥ 2 . Under this assumption, Algorithm 1 guarantees to find an ϵ -KKT point within O (1 /ϵ 2 ) iterations [48, 32].

4 ADMMfor a Smoothed Version of Problem (1)

In Section 3.3, our convergence result is established under Assumption 2 when a nonsmooth regularizer g ( w ) is present. In this section, to remove this assumption, we design a variant of the proximal ADMMby smoothing the regularizer g ( w ) . We employ the Moreau envelope to smooth g ( w ) and replace g ( w ) in Algorithm 1 with M g,γ ( w ) . We point out that M g,γ ( w ) is a γ -1 weakly convex function if 0 < cγ ≤ 1 3 . Thus the w -subproblem is still strongly convex and can be solved exactly. See Appendix A.1 for proof. We assume that the sequence { λ k } is bounded, which is much weaker than Assumption 2. We will show that within O (1 /ϵ 4 ) iterations, ( z k , ˜ w k , λ k ) is an ϵ -KKT point, where ˜ w k = prox g,γ ( w k ) .

To compensate for the absence of Assumption 2, we introduce the following more practical assumptions.

A stronger version of Assumption 4 is that DD ⊤ = ( diag ( y ) X )( diag ( y ) X ) ⊤ = diag ( y ) XX ⊤ diag ( y ) ≻ 0 [32]. It is worth noting that the full row rank property of the data matrix is often assumed in the high dimensional setting classification ( m < d and each entry of y belongs to {-1 , 1 } ). Here we do not impose any assumptions on the rank of data matrix X .

Regarding nonsmooth regularizers, Assumption 5 is satisfied by weakly convex functions that are Lipschitz continuous [7], due to the fact that Lipschitz continuity implies bounded subgradients and (8). Common regularizers such as l 1 -norm, MCP and SCAD are all Lipschitz continuous.

The following proposition establishes the relationship between { ( z k , w k , λ k ) } and { ( z k , ˜ w k , λ k ) } .

Proposition 2 Suppose g is c -weakly convex and Assumption 3 holds. Suppose the sequence { ( z k , w k , λ k ) } is produced during the iterations of Algorithm 1 with replacing g ( w ) by M g,γ ( w ) and ˜ w k = prox g,γ ( w k ) . Then we have

Before presenting our main results, we introduce a Lyapunov function that plays a significant role in our analysis:

The following lemma demonstrates the sufficient decrease property for Φ k .

Lemma 1 Let r > γ -1 . Under Assumptions 1, 3, 4 and 5, if 0 < cγ ≤ 1 3 , then we have

The sufficient decrease property for the Lyapunov function is crucial in the proof of nonconvex ADMM. Using Lemma 1, we can control ∥ w k +1 -w k ∥ and ∥ λ k +1 -λ k ∥ through the decrease of the Lyapunov function. We are now ready to present the main result of this section.

Theorem 2 Suppose { λ k } is bounded. Set γ = ϵ ≤ 1 3 c , ρ = C 1 ϵ , r = C 2 ϵ , where C 1 and C 2 are constants such that C 2 > 1 and C 1 > 8 C 2 2 + 1 3 c +4 σ (2 C 2 -1) . Under the same assumptions in Lemma 1, Algorithm 1 with replacing g ( w ) by M g,γ ( w ) finds an ϵ -KKT point ( z k +1 , ˜ w k +1 , λ k +1 ) within O (1 /ϵ 4 ) iterations, i.e.,

5 Numerical Experiment

In this section, we perform binary classification experiments to illustrate both the robustness and the extensive applicability of our proposed algorithm.

When using the logistic loss or hinge loss as individual loss, the objective function of problem (1) can be rewritten as ∑ n i =1 σ i log ( 1 + exp ( z [ i ] )) + g ( w ) or ∑ n i =1 σ i [ 1 + z [ i ] ] + + g ( w ) , where z = -y ⊙ ( X w ) , and g ( w ) = µ 2 ∥ w ∥ 2 2 or g ( w ) = µ 2 ∥ w ∥ 1 .

We point out some key settings:

For the spectral risk measures, we use the logistic loss and hinge loss as individual loss, and use ℓ 1 and ℓ 2 regularization with µ taken as 10 -2 .

For the average of ranked range aggregate loss, we use the logistic loss and hinge loss as individual loss and use ℓ 2 regularization with µ = 10 -4 .

For the empirical human risk minimization, we use the logistic loss as individual loss, and use ℓ 2 regularization with µ = 10 -2 .

As shown in Section 2.1, we can apply our proposed algorithm to a variety of frameworks such as the spectral risk measure (SRM), the average of ranked-range (AoRR) aggregate loss, and the empirical human risk minimization (EHRM). We compare our algorithm with LSVRG, SGD, and DCA: LSVRG(NU) denotes LSVRG without uniformity, and LSVRG(U) denotes LSVRG with uniformity, as detailed in [36]; SGD denotes the stochastic subgradient method in [36]; DCA denotes the difference-of-convex algorithm in [40]; sADMM refers to the smoothed version of ADMM.

The details of our algorithm setting, more experiment settings and detailed information for each dataset are provided in Appendix B. Additional experiments with synthetic datasets are presented in Appendix C.

5.1 Spectral Risk Measures

In this section, we select σ ( t ) ≡ 1 in (3) to generate the empirical risk minimization (ERM) and σ ( t ) = 1 [ q, 1] ( t ) / (1 -q ) to generate superquantile, the latter having garnered significant attention in the domains of finance quantification and machine learning [30].

Setting. For the learning rates of SGD and LSVRG, we use grid search in the range { 10 -2 , 10 -3 , 10 -4 , 1 /N samples , 1 / ( N samples × N features ) } , which is standard for stochastic algorithms, and then we repeat the experiments five times with different random seeds.

Results. Tables 1 and 2 present the mean and standard deviation of the results in different regularizations. The first row for each dataset represents the objective value of problem (1) and the second row shows test accuracy. Bold numbers indicate significantly better results. Tables 1 and 2 show that our ADMM algorithm outperforms existing methods in terms of objective values for most instances and has comparable test accuracy. In addition, we highlight the necessity of reselecting the learning rate for SGD and LSVRG algorithms, as the individual loss and dataset undergo modifications-a highly intricate procedure. Conversely, our proposed algorithm circumvents the need for any adjustments in this regard.

Datasets

Logistic Loss

Hinge Loss

ADMM

sADMM

LSVRG(NU)

LSVRG(U)

SGD

ADMM

sADMM

LSVRG(NU)

LSVRG(U)

SGD

SVMguide

0.6280 (0.0129)

0.6280 (0.0129)

0.6280 (0.0129)

0.6280 (0.0129)

0.6282 (0.0128)

0.6789 (0.0273 )

0.6789 (0.0273 )

0.9121 (0.0133)

0.6789 (0.0273)

0.8569 (0.0210)

0.9561 (0.0037)

0.9561 (0.0037)

0.9563 (0.0039)

0.9561 (0.0037)

0.9561 (0.0038)

0.9566 (0.0036)

0.9566 (0.0036)

0.9558 (0.0039)

0.9563 (0.0035)

0.9555 (0.0040)

AD

0.2601 (0.0079)

0.2602 (0.0079)

0.2614 (0.0080)

0.2614 (0.0080)

0.2641 (0.0105)

0.1352 (0.0094)

0.1370 (0.0096)

0.1457 (0.0110)

0.1449 (0.0100)

0.1402 (0.0101)

0.9673 (0.0054)

0.9673 (0.0054)

0.9680 (0.0054)

0.9680 (0.0054)

0.9673 (0.0039)

0.9589 (0.0060)

0.9589 (0.0060)

0.9644 (0.0045)

0.9639 (0.0039)

0.9635 (0.0023)

Figure 1 illustrates the relationship between time and sub-optimality of each algorithm in the ERM framework with logistic loss, which is a convex problem whose global optimum can be achieved. The learning rate of the other algorithms is the best one selected by the same method. The Figure 1 depicts that ADMM exhibits significantly faster convergence towards the minimum in comparison to the other algorithms. Conversely, although sADMM fails to achieve the accuracy level of ADMM, it nevertheless demonstrates accelerated convergence when compared to other algorithms.

5.2 Empirical Human Risk Minimization

Since [31] utilizes gradient-based algorithms akin to those in [36], we compare our algorithm with the one referenced in Section 5.1.

Results. The mean and standard deviation of the results are tabulated in Table 3. SPD, DI, EOD, AOD, TI, and FNRD are fairness metrics detailed in Appendix B.6. The bold numbers highlight the superior result. It is evident that our proposed algorithm outperforms the other algorithms in terms of objective values and test accuracy, and also shows enhanced performance across almost all fairness metrics.

5.3 Average of Ranked Range Aggregate Loss

As mentioned in Section 2.1, DCA can be used to solve for AoRR aggregate loss minimization [27]. Therefore, we compare our algorithm with the DCA algorithm in this section.

Results. Table 4 displays the mean and standard deviation of both the objective value and time results. Given that this is a nonconvex problem, the objective value may not be the optimal value.

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 1000 num_feature: 500

ADMM

SGD

LSVRG(NU)

LSVRG(U)

(a)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 2000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

(b)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 5000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

(c)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 10000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

(d)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 1000 num_feature: 500

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(e)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 2000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(f)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 5000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(g)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 10000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(h)

Nevertheless, as evidenced in Table 4, our proposed algorithm achieves better objective value in a shorter time for all the instances.

6 Conclusion

This paper considers rank-based loss optimization with monotonically increasing loss functions and weakly convex regularizers. We propose a unified ADMM framework for rank-based loss minimization. Notably, one subproblem of the ADMM is solved efficiently by the PAVA. Numerical experiments illustrate the outperformance of our proposed algorithm, with all three practical frameworks delivering satisfactory results. We also point out some limitations of our algorithm. To effectively utilize our algorithm, individual losses must exhibit monotonicity, as this allows us to use the PAVA to solve subproblems. If the sample size is large, the PA V A's computational efficiency may be hindered, potentially limiting its overall effectiveness. Future work may explore a variant using mini-batch samples, potentially improving early-stage optimization performance and overall computational efficiency.

Acknowledgement Rujun Jiang is partly supported by NSFC 12171100 and Natural Science Foundation of Shanghai 22ZR1405100.

References

Acerbi, C. (2002). Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of Banking & Finance , 26(7):1505-1518.

Acerbi, C. and Simonetti, P. (2002). Portfolio optimization with spectral measures of risk. arXiv preprint cond-mat/0203607 .

Bai, X., Sun, J., and Zheng, X. (2021). An augmented lagrangian decomposition method for chance-constrained optimization problems. INFORMS Journal on Computing , 33(3):1056-1069.

Beck, A. and Teboulle, M. (2009). A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences , 2(1):183-202.

Bellamy, R. K., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018). Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943 .

Best, M. J., Chakravarti, N., and Ubhaya, V. A. (2000). Minimizing separable convex functions subject to simple chain constraints. SIAM Journal on Optimization , 10(3):658-672.

Böhm, A. and Wright, S. J. (2021). Variable smoothing for weakly convex composite functions. Journal of optimization theory and applications , 188:628-649.

Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J., et al. (2011). Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning , 3(1):1-122.

Chow, Y., Tamar, A., Mannor, S., and Pavone, M. (2015). Risk-sensitive and robust decisionmaking: a CVAR optimization approach. Advances in neural information processing systems , 28.

Clarke, F. H. (1990). Optimization and nonsmooth analysis . SIAM.

Cotter, J. and Dowd, K. (2006). Extreme spectral risk measures: An application to futures clearinghouse margin requirements. Journal of Banking & Finance , 30(12):3469-3485.

Cui, X., Jiang, R., Shi, Y., and Yan, Y. (2023). Decision making under cumulative prospect theory: An alternating direction method of multipliers. arXiv preprint arXiv:2210.02626 .

Cui, X., Sun, X., Zhu, S., Jiang, R., and Li, D. (2018). Portfolio optimization with nonparametric value at risk: A block coordinate descent method. INFORMS Journal on Computing , 30(3):454471.

Curi, S., Levy, K. Y., Jegelka, S., and Krause, A. (2020). Adaptive sampling for stochastic risk-averse learning. Advances in Neural Information Processing Systems , 33:1036-1047.

Daouia, A., Gijbels, I., and Stupfler, G. (2019). Extremiles: A new perspective on asymmetric least squares. Journal of the American Statistical Association , 114(527):1366-1381.

Davis, D. and Drusvyatskiy, D. (2019). Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization , 29(1):207-239.

Derrac, J., Garcia, S., Sanchez, L., and Herrera, F. (2015). Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework. J. Mult. Valued Logic Soft Comput , 17.

Dua, D., Graff, C., et al. (2017). Uci machine learning repository, 2017. URL http://archive. ics. uci. edu/ml , 7(1).

Duchi, J. C., Hashimoto, T., and Namkoong, H. (2019). Distributionally robust losses against mixture covariate shifts. Under review , 2:1.

Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association , 96(456):1348-1360.

Fan, Y., Lyu, S., Ying, Y., and Hu, B. (2017). Learning with average top-k loss. Advances in neural information processing systems , 30.

Friedman, J., Hastie, T., and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of statistical software , 33(1):1.

Garcıa, J. and Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480.

Gopalan, A., Prashanth, L., Fu, M., and Marcus, S. (2017). Weighted bandits or: How bandits learn distorted values that are not expected. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31.

Holland, M. J. and Haress, E. M. (2022). Spectral risk-based learning using unbounded losses. In International Conference on Artificial Intelligence and Statistics , pages 1871-1886. PMLR.

HSU, C.-W. (2010). A practical guide to support vector classification. http://www. csie. ntu. edu. tw/˜ cjlin/papers/guide/guide. pdf .

Hu, S., Ying, Y., Lyu, S., et al. (2020). Learning by minimizing the sum of ranked range. Advances in Neural Information Processing Systems , 33:21013-21023.

Kamishima, T., Akaho, S., and Sakuma, J. (2011). Fairness-aware learning through regularization approach. In 2011 IEEE 11th International Conference on Data Mining Workshops , pages 643-650. IEEE.

Kushmerick, N. (1999). Learning to remove internet advertisements. In Proceedings of the third annual conference on Autonomous Agents , pages 175-181.

Laguel, Y., Pillutla, K., Malick, J., and Harchaoui, Z. (2021). Superquantiles at work: Machine learning applications and efficient subgradient computation. Set-Valued and Variational Analysis , 29(4):967-996.

Leqi, L., Prasad, A., and Ravikumar, P. K. (2019). On human-aligned risk minimization. Advances in Neural Information Processing Systems , 32.

Li, G. and Pong, T. K. (2015). Global convergence of splitting methods for nonconvex composite optimization. SIAM Journal on Optimization , 25(4):2434-2460.

Lin, Z., Li, H., and Fang, C. (2022). Alternating Direction Method of Multipliers for Machine Learning . Springer.

Ma, Y., Li, L., Huang, X., and Wang, S. (2011). Robust support vector machine using least median loss penalty. IFAC Proceedings Volumes , 44(1):11208-11213.

Maurer, A., Parletta, D. A., Paudice, A., and Pontil, M. (2021). Robust unsupervised learning via l-statistic minimization. In International Conference on Machine Learning , pages 7524-7533. PMLR.

Mehta, R., Roulet, V., Pillutla, K., Liu, L., and Harchaoui, Z. (2023). Stochastic optimization for spectral risk measures. In International Conference on Artificial Intelligence and Statistics , pages 10112-10159. PMLR.

Nurminskii, E. A. (1973). The quasigradient method for the solving of the nonlinear programming problems. Cybernetics , 9(1):145-150.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830.

Pflug, G. C. (2000). Some remarks on the value-at-risk and the conditional value-at-risk. Probabilistic constrained optimization: Methodology and applications , pages 272-281.

Phan, D. N. (2016). DCA based algorithms for learning with sparsity in high dimensional setting and stochastical learning . PhD thesis, Université de Lorraine.

Prashanth, L., Jie, C., Fu, M., Marcus, S., and Szepesvári, C. (2016). Cumulative prospect theory meets reinforcement learning: Prediction and control. In International Conference on Machine Learning , pages 1406-1415. PMLR.

Schmidt, M., Fung, G., and Rosales, R. (2007). Fast optimization methods for l1 regularization: Acomparative study and two new approaches. In Machine Learning: ECML 2007: 18th European Conference on Machine Learning, Warsaw, Poland, September 17-21, 2007. Proceedings 18 , pages 286-297. Springer.

Shalev-Shwartz, S. and Wexler, Y. (2016). Minimizing the maximal loss: How and why. In International Conference on Machine Learning , pages 793-801. PMLR.

Shen, Y., Wen, Z., and Zhang, Y. (2014). Augmented lagrangian alternating direction method for matrix separation based on low-rank factorization. Optimization Methods and Software , 29(2):239-263.

Tversky, A. and Kahneman, D. (1992). Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty , 5:297-323.

Vapnik, V. (2013). The Nature of Statistical Learning Theory . Springer Science & Business Media.

Vial, J.-P. (1983). Strong and weak convexity of sets and functions. Mathematics of Operations Research , 8(2):231-259.

Wang, Y., Yin, W., and Zeng, J. (2019). Global convergence of admm in nonconvex nonsmooth optimization. Journal of Scientific Computing , 78:29-63.

Williamson, R. and Menon, A. (2019). Fairness risk measures. In International Conference on Machine Learning , pages 6786-6797. PMLR.

Wnek, J. (1992). MONK's Problems. UCI Machine Learning Repository. DOI: 10.24432/ C5R30R .

Xu, Y., Yin, W., Wen, Z., and Zhang, Y. (2012). An alternating direction algorithm for matrix completion with nonnegative factors. Frontiers of Mathematics in China , 7:365-384.

Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics , pages 894-942.

Zhang, Z., Song, Y., and Qi, H. (2017). Age progression/regression by conditional adversarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5810-5818.

Appendix

A Proofs

A.1 Proofs for properties of Moreau envelope

Lemma 2 Let g : R n → R ∪ {∞} be a c -weakly convex function. For 0 < cγ ≤ 1 3 , the Moreau envelope M g,γ of g is γ -1 -weakly convex.

A.1.1 Proof of Lemma 2

Lemma 3 Let g : R n → R ∪ {∞} . Then g is c -weakly convex if and only if the following holds

for x , y ∈ R n , v ∈ ∂g ( x ) , w ∈ ∂g ( y ) .

Proof: See [16, Lemma 2.1].

Lemma 4 Suppose g ( w ) is c -weakly convex. If 0 < cγ ≤ 1 3 , then we have

Proof: Suppose x , y ∈ R n . By (8) and Lemma 3 we have

Thus we have

Note that (1 -2 cγ ) -1 ≤ 3 when 0 < cγ ≤ 1 3 . Thus we have

Proof of Lemma 2

Proof: For any x , y ∈ R n , we have

where the first equality follows from (8) and the last inequality follows from Lemma 4. Thus g is γ -1 -weakly convex by Lemma 3. ■

■

A.2 Proofs for ADMM in Section 3

A.2.1 Proof of Proposition 1

Proof: First we define h [ a,b ] ( z ) = ∑ b i = a θ i ( z ) . Due to convexity, we have

Noting that v [ m,n ] > v [ n +1 ,p ] , due to the strong convexity of h , we further have

where we use the convention that a set A > 0 (or < 0 ) denotes that all elements g ∈ A satisfies g > 0 (or < 0 ). Therefore we obtain that there exist s [ m,n ] , s [ n +1 ,p ] ∈ R such that

As h [ m,p ] is strongly convex in [ v [ n +1 ,p ] , v [ m,n ] ] , the subgradient ∂h [ m,p ] is strictly increasing in [ v [ n +1 ,p ] , v [ m,n ] ] . That is, ∀ x, y ∈ [ v [ n +1 ,p ] , v [ m,n ] ] , ∀ s x ∈ ∂h [ m,p ] ( x ) , s y ∈ ∂h [ m,p ] ( y ) , ( s x -s y )( x -y ) ≥ 0 . The above facts, together with the strong convexity of h [ m,p ] , the unique minimizer v [ m,p ] of h [ m,p ] must lie in the interval [ v [ n +1 ,p ] , v [ m,n ] ] . ■

A.2.2 Proof of Theorem 1

Proof: We use the extended formula for Clark generalized gradient of a sum of two functions in our proofs: ∂ ( f 1 + f 2 )( x ) ⊂ ∂f 1 ( x ) + ∂f 2 ( x ) if f 1 and f 2 are finite at x and f 2 is differentiable at x . The equality holds if f 1 is regular at x . [10, Theorem 2.9.8].

By Assumption 3, we have

and

Here the last equality of (15) follows from λ k +1 = λ k + ρ ( z k +1 -D w k +1 ) .

By Assumption 3 and ( r -c ) -strong convexity of the w -subproblem, we have

and

The second equality of (17) is due to the fact that weakly convex functions are regular [47, Proposition 4.5]. By the dual update we have

Summing (16), (18) and (19) we obtain that

By Assumptions 1 and 2 we have that

So L ρ ( z k , w k , λ k ) is bounded below by some L ⋆ . Moreover, ∑ K k =1 ∥ λ k +1 -λ k ∥ 2 ≤ ∑ ∞ k =1 ∥ λ k +1 -λ k ∥ 2 < ∞ for ∀ K ≥ 1 .

Let ˆ L := L ρ ( z 1 , w 1 , λ 1 ) -L ⋆ + ∑ ∞ k =1 2 ρ ∥ λ k +1 -λ k ∥ 2 < ∞ . Then telescoping (20) from k = 1 to K , we obtain that

This implies that

Letting k = arg min i ≤ K 2 r -c 2 ∥ w i +1 -w i ∥ 2 + 1 ρ ∥ λ i +1 -λ i ∥ 2 , we have

Letting K = 1 ϵ 2 , by (15) and (17) we further have

The proof of Theorem 1 is adapted from [33, Theorem 4.1].

■

A.3 Proofs for smoothed ADMM in Section 4

A.3.1 Proof of Proposition 2

Proof: By Assumption 3, similar to (15), we have

where the last equality follows from λ k +1 = λ k + ρ ( z k +1 -D w k +1 ) .

By the first order condition of the w -subproblem and Assumption 3, we have

Note that ∇ M g,γ ( w k +1 ) ∈ ∂g ( prox g,γ ( w k +1 )) = ∂g ( ˜ w k +1 ) . This completes the proof. ■

A.3.2 Proof of Lemma 1

Lemma 5 Under Assumption 4, if 0 < cγ ≤ 1 2 , then for ∀ k = 0 , 1 , . . . , we have

where σ is the smallest positive eigenvalue of DD ⊤ .

where the second inequality follows from (23) and the last inequality follows from the fact that M g,γ has γ -1 Lipschitz continuous gradient when 0 < cγ ≤ 1 2 [7]. ■

Proof of Lemma 1

Proof: By Assumption 3 we have

Since 0 < cγ ≤ 1 3 , M g,γ ( w ) is γ -1 -weakly convex by Lemma 2. By the optimality of w k +1 and ( r -γ -1 ) -strong convexity of the w -subproblem we have

By the update of dual variable

Summing (24), (25) and (26) we have

Finally, we obtain that

where the first inequality follows from (27) and the second inequality follows from Lemma 5. ■

A.3.3 Proof of Theorem 2

Proof: First note that 0 < cγ ≤ 1 3 , so Lemma 1 holds. Let E 0 := 1 3 c and E 1 := r -γ -1 2 -4 r 2 σρ -2 σργ 2 . Using the definitions of γ , ρ and r we have

Since { λ k } is bounded, we obtain that L ρ ( z k , w k , λ k ) is bounded below by a similar manner of (21) and Assumption 1. Thus Φ k is lower bounded by some Φ ⋆ as well, i.e., Φ k ≥ Φ ⋆ , for ∀ k ≥ 1 . Telescoping (14) from k = 1 to K we have:

Thus we have:

Let k = arg min i ≤ K E 1 ∥ w i +1 -w i ∥ 2 + 1 ρ ∥ λ i +1 -λ i ∥ 2 , we have

Recall ˜ w k = prox g,γ ( w k ) . Note that by (28), E -1 1 is upper bounded by the constant 2 σC 1 . Letting K = 1 ϵ 4 , by Propositon 2, the definations of r and ρ and the above two equations, we have

where the second inequality follows from (8) and Assumption 5. This completes the proof.

■

Our proof of Theorem 2 draws inspiration from [33, Theorem 4.1] and [32]. In our method, we employ the Moreau envelope M g,γ instead of g ( w ) to obtain a smoothed version of Problem (1). However, the solution obtained through Algorithm 1 may not satisfy the ϵ -KKT conditions of the original problem. Nonetheless, our analysis demonstrates that by setting ˜ w k = prox g,γ ( w k ) , our algorithm guarantees an ϵ -KKT point of the original problem in at most O (1 /ϵ 4 ) iterations.

B Additional Experimental Details

B.1 Source Code

The source code is available in the https://github.com/RufengXiao/ADMM-for-rank-based-loss.

B.2 Experimental Environment

All algorithms are implemented in Python 3.8 and all the experiments are conducted on a Linux server with 256GB RAM and 96-core AMD EPYC 7402 2.8GHz CPU.

B.3 Details for the PA V A

Proposition 3 Given two consecutive blocks [ m,n ] and [ n +1 , p ] , if they are out-of-order, then we have

Proof: Suppose on the contrary that v [ m,n ] > v [ n +1 ,p ] but

We use the conventions that h [ a,b ] ( z ) = ∑ b i = a θ i ( z ) . For any g ∈ ∂l ( v [ m,n ] ) , we must have g ≥ 0 because l is monotonically increasing (see the beginning of Section 3.1). Therefore we have

where the first inequality is due to the non-negativity of g , and the second inequality is due to the fact that m i ≤ m i +1 for i = m,m +1 , . . . , p . Note also that

and

We also have 0 ∈ ∂h [ m,n ] ( v [ m,n ] ) from the optimality condition. This implies that there exists some s ∈ ∂h [ n +1 ,p ] ( v [ m,n ] ) such that s ≤ 0 .

Now if there exists another s ′ ∈ ∂h [ n +1 ,p ] ( v [ m,n ] ) such that s ′ > 0 , then we have 0 ∈ ∂h [ n +1 ,p ] ( v [ m,n ] ) , which contradicts with the uniqueness of the optimal point v [ n +1 ,p ] to h [ n +1 ,p ] (due to its strongly convexity). So we must have s ≤ 0 for all s ∈ ∂h [ n +1 ,p ] ( v [ m,n ] ) . Then we must have v [ m,n ] ≤ v [ n +1 ,p ] from the optimality of v [ n +1 ,p ] to h [ n +1 ,p ] and convexity of h [ n +1 ,p ] . However, this contradicts the fact that [ m,n ] and [ n +1 , p ] are out-of-order . ■

Proposition 3 presents an additional opportunity to accelerate the PA V A, particularly for the topk loss and AoRR framework. Consider the case of topk loss, where σ i = 1 if i = k and σ i = 0 otherwise. In this scenario, the condition for merging blocks is satisfied only when i = k . Therefore, we only need to examine the block containing k , its preceding block, and its subsequent block. Consequently, the time complexity of searching for out-of-order blocks in line 5 of Algorithm 2 is reduced to O (1) .

By leveraging this insight, we enhance the efficiency of our algorithm, specifically for topk loss. An analogous acceleration can also be applied to the AoRR loss. This optimization dramatically reduces computational complexity, thereby facilitating faster execution of the PA V A for both topk loss and AoRR loss.

In our implementation of the PAVA, we use either the bisection method or Newton's method to find a minimizer of the convex function h [ m,n ] . Assuming the maximum number of iterations for these methods as T , the computation of the minimizer v [ m,n ] exhibits a time complexity of O ( T ) . In the PAVA, as we merge out-of-order blocks, the size of the index set J decreases by at least one.

Initially, J comprises n blocks. Hence, the minimizer needs to be computed no more than O ( n ) times throughout the algorithm. Furthermore, before each merge, we perform up to O ( n ) comparisons. However, this complexity is reduced to O (1) for topk loss and AoRR loss due to the structures of these losses. Overall, the time complexity of our PA V A is O ( n 2 + nT ) . Notably, for topk loss and AoRR loss, the time complexity is reduced to O ( n + nT ) .

Moreover, in the empirical human risk minimization with CPT-weight in (4), θ i ( z i ) is nonconvex with regard to z i because σ i is dependent on the value of z i . Specifically, θ i ( z i ) takes the form of a two-piece function for z i ∈ ( -∞ , B ] and ( B, ∞ ) , where B represents a certain threshold. However, despite its nonconvexity, θ i ( z i ) remains convex within each piece. Exploiting this property, we can determine the minimizer of such a function by comparing the minimizers of the two separate pieces. Considering this observation, the overall time complexity of the PA V A for the empirical human risk minimization is still O ( n 2 + nT ) .

B.4 Datasets Description

Datasets for our experiments are generated in two ways.

Synthetic data. We construct synthetic datasets where the data matrix X and label vector y are generated artificially. We utilize the datasets.make_classification() function from the Python package scikit-learn [38] to generate two-class classification problem datasets of various dimensions.

Real data. In the case of real data, the data matrix X and label vector y are derived from existing datasets. The ' SVMguide ' [26] dataset, frequently used in support vector machines, is included in our experiments. We also employ ' AD ' [29], which comprises potential advertisements for Internet pages, and ' Monks ' [50], the dataset based on the MONK's problem that served as the first international comparison of learning algorithms. The ' Splice ' dataset from the UCI [18] is used for the task of recognizing DNA sequences as exons or introns. We additionally include ' Australian ', ' Phoneme ', and ' Titanic ' dataset from [17]. Lastly, the ' UTKFace ' dataset [53] is used to predict gender based on facial images. Detailed statistics for each dataset are presented in Table 5.

B.5 Details of Our Algorithm Setting

In the experiments, we set the maximum iteration limit in our algorithm to 300. We utilize FISTA to solve the w -subproblem in our algorithm and L-BFGS for the smoothed w -subproblem. For varying frameworks, we adopt different choices of { ρ k }, which varies when the iteration number k increase, in Algorithm 1. To enable the replication of our experiments, we provide a suggested selection for { ρ k }. The specifics regarding these choices for { ρ k } are detailed in Table 6. We set the γ in Section 4 to γ k = max { 10 -5 × 0 . 9 k , 10 -9 } .

B.6 Details of Experiments Setting

Spectral Risk Measures. Our objective is to demonstrate the versatility of our algorithm and its ability to converge to a globally optimal solution in the convex problem. Consequently, we utilize two real binary classification problem datasets and several synthetic datasets to highlight the advantages of our algorithm. We compare our method with the algorithms in [36]. We adapt their implementation

by replacing the gradient of the ℓ 2 norm with the soft threshold operator to incorporate the ℓ 1 regularization. For simplicity, we set the regularization parameter µ as 0.01 and q as 0.8. The datasets are divided into 60% for training and 40% for testing. Both SGD and LSVRG are run for 2000 epochs and the batch size of SGD is set to 64, and the epoch lengths are set to 100 as recommended in [36]. We give an example of how we choose the learning rate. For comparative purposes, we have chosen the learning rate that yields the lowest objective value in a single randomized experiment among five different learning rates from { 10 -2 , 10 -3 , 10 -4 , 1 /N samples , 1 / ( N samples × N features ) } . We illustrate this approach using the AD dataset under the SRM superquantile framework as an example. The results are compiled in Table 7. The learning rate corresponding to the bold values, indicating the minimal objective value, is subsequently chosen.

Average of Ranked Range Aggregate Loss. We employ the same real datasets as those used in [27]. Consistent with their experimental setup, we randomly split each dataset into a 50% training set, a 25% validation set, and a 25% test set. As [27] only considered ℓ 2 regularization with µ = 10 -4 , our experiments in this section also utilize the AoRR aggregate loss with ℓ 2 regularization. The choice of hyper-parameters k and m and the selection of inner and outer epochs for each dataset follow [27].Table 8 presents the hyper-parameters for individual logistic loss, while Table 9 displays those for individual hinge loss.

Empirical Human Risk Minimization. We set γ = 0 . 61 and δ = 0 . 69 in (5) based on the recommendations in [45]. We set B = log (1 + exp ( -5)) in (4). For simplicity, we only use ℓ 2 regularization. We employ the 'UTKFace' dataset [53], also used in [31], for gender prediction. The experiments are performed five times with different random seeds. In this experimental segment, we exclusively use the logistic loss as the individual loss since hinge loss has most of the zeros so that it is not easy to determine the corresponding B in (4) which means that its distribution is not suitable for this framework. The cumulative distribution function of loss F is derived from synthetic data, and F ( B ) is approximately equal to 0 . 05 . The datasets are divided into 60% for training and 40% for testing. Other settings are similar to those in [31]. In accordance with [31], we divide the population into two groups based on race (white G 1 and other race G 2 ). However, for simplicity, we only obtain the parameters w for gender prediction by minimizing the problem (1) without using a neural network. The parameter settings mirror those in Section 5.1. We use the same method to get the learning rate, and the learning rate for all three algorithms is set to 1 / ( N samples × N features ) .

Let TPR denote the true positive rate, FPR the false positive rate, and FNR the false negative rate. Additionally, let X ∈ X represent the dataset and Y ∈ { 0 , 1 } the label. We utilize the same fairness metrics suggested by [5], as followed in [31]. Considering the privileged group G 1 ⊆ X and the unprivileged group G 2 ⊆ X , the definitions of the following metrics are from [31]:

Statistical Parity Difference (SPD): P ( Y = 1 | X ∈ G 2 ) -P ( Y = 1 | X ∈ G 1 ) .

Disparate Impact (DI): P ( Y =1 | X ∈ G 2 ) P ( Y =1 | X ∈ G 1 ) .

Equal Opportunity Difference (EOD): TPR( G 2 ) -TPR( G 1 ) .

Average Odds Difference (AOD): 1 2 (FPR( G 2 ) -FPR( G 1 ) + (TPR( G 2 ) -TPR( G 1 ))) .

Theil Index (TI): 1 n ∑ n i =1 b i µ ln ( b i µ ) where b i = ̂ Y i -Y i +1 and µ = 1 n ∑ n i =1 b i . Here, ̂ Y i is the prediction of X i and n represents the number of samples.

False Negative Rate Difference (FNRD): FNR( G 2 ) -FNR( G 1 ) .

C Additional Experimental Results

In this section, synthetic datasets labeled 1, 2, 3, and 4 are utilized. Detailed descriptions of datasets 1, 2, 3, and 4 can be found in Table 10.

C.1 Experiments of Synthetic Datasets with ERM

The experimental settings implemented in this section are consistent with those discussed in Section 5.1. The datasets are replaced with the synthetic datasets. We use the same method to get the learning rate for other algorithms. Tables 11, 12, and 13 enumerate the mean and standard deviation of the results, primarily focusing on the objective value and test accuracy. 'sADMM' stands for the ADMM applied to the smoothed problem. Tables 11, 12, and 13 show that under most scenarios in the ERM framework, our ADMM algorithm exhibits superior performance in terms of objective

values when compared to existing methods. This superiority is particularly evident in instances of hinge loss, with the algorithm demonstrating a test accuracy that is analogous to other methods.

Figure 2 graphically demonstrates the correlation between time and sub-optimality for each algorithm within the superquantile framework and hinge loss, which represents a convex problem whose global optimum is achievable. The figure indicates that ADMM displays a more rapid convergence towards the minimum relative to other algorithms. While sADMM does not match the performance of ADMM, it does achieve a commendable rate of convergence when compared to other algorithms.

C.2 Experiments of Synthetic Datasets with AoRR

In the application of AoRR to synthetic data, we have configured the parameters k and m to be 0 . 8 and 0 . 2 times the size of the training set, respectively. In other words, k = ⌊ 0 . 8 N train ⌋ and m = ⌈ 0 . 2 N train ⌉ . We maintain the learning rate in the DCA at 0.01, and set the outer epoch at 10, and the inner epoch at 2000. The parameters for SGD and LSVRG are identical to those employed in the preceding experiment (the learning rate is chosen from the same method). Tables 14 and 15

0

10

20

30

40

50

60

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 1000 num_feature: 500

ADMM

SGD

LSVRG(NU)

LSVRG(U)

(a)

0

10

20

30

40

50

60

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 2000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

(b)

0

10

20

30

40

50

60

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 5000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

(c)

0

10

20

30

40

50

60

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 10000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

(d)

0

10

20

30

40

50

60

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 1000 num_feature: 500

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(e)

0

10

20

30

40

50

60

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 2000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(f)

0

10

20

30

40

50

60

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 5000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(g)

0

10

20

30

40

50

60

70

80

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 10000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(h)

enumerate the mean and standard deviation of results for the objective value, accuracy, and time. A careful examination of Tables 14 and 15 suggests that our algorithm, in most scenarios, achieves lower objective values and comparable accuracy in a shorter time span relative to existing methods.

C.3 Experiments about Figure 1

Weincreased the sample size to further observe the performance of our algorithm and the experimental results are shown in Figure 3. It can be seen that both ADMM and sADMM exhibit relatively fast convergence compared to other algorithms. An interesting phenomenon is that in the case of a large amount of data, random algorithms are able to achieve decent results in the early stages of optimization. The underlying reason for this phenomenon may be that existing methods use mini-batch samples that are independent of the sample size to update model parameters, thereby reducing the computational cost per iteration. This allows existing methods to update parameters more frequently, leading to faster convergence in the early stages of optimization. In contrast, the proposed algorithm uses the entire batch of samples in each iteration, resulting in slower iteration speeds. This leads to suboptimal solutions in the early stages compared to existing methods. In Appendix B.3, we provide a detailed explanation of the time complexity of the PAVA, which is O ( n + nT ) for top-k loss and AoRR loss, where T represents the maximum number of iterations when solving each PAVA subproblem, and n represents the sample size. Therefore, with an increase in

0

10

20

30

40

50

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 20000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(a)

0

10

20

30

40

50

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 20000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(b)

0

20

40

60

80

100

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 50000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(c)

0

20

40

60

80

100

CPU time (secs.)

10

8

10

6

10

4

10

2

10 0

Sub-optimality gap

log

10(

F k

F

*

)

num_sample: 50000 num_feature: 1000

ADMM

SGD

LSVRG(NU)

LSVRG(U)

sADMM

(d)

sample size, the time required for our proposed algorithm will also increase. Nevertheless, compared to existing algorithms, we are still able to achieve higher accuracy within a reasonable time frame.

Peer reviews:
comments: SUMMARY:
This paper presents rank-based loss as a popular replacement for empirical loss. The work develops how
optimization of rank-based loss can be done by a proximal alternating direction method of multipliers. The authors also demonstrate the algorithm's features in terms of convergence under certain conditions. Experimentation which includes synthetic and real datasets is also added in the paper with some numerical simulations showing how the framework behaves. It supports the theoretical results.

STRENGTHS:
The paper is mathematically strong

WEAKNESSES:
The contribution doesn't seem significant enough
The applicability of the result is not clearly shown

QUESTIONS:
I would like the authors to compare the method with similar-purpose frameworks, even if not using rank-based loss, and see their differences in performance.

I would appreciate the inclusion of the applicability of the result to a real machine-learning problem.

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission focuses on efficient minimization of a group of loss functions called rank-based losses. It proposes to consider several related losses from the perspective of a genral unified framework with a regularizer. Then, focusing on the case of monotone increasing loss functions and wealy convex regularizers, it proposes an ADMM-based algorithm and shows its convergence rate under common assumptions. Furthermore, when the regularizer is non-smooth, it then extends the proposed algorithm with weaker assumptions and also shows its convergence rate. Sufficient numerical verification shows the satisfying empirical performance of the proposed algorithm compared with several existing methods.\n\nSTRENGTHS:\nOriginality:\n- The task provides a new perspective on an important problem and the proposed methods improving ADMM are novel.\n- Authors clearly address where exactly the improvements from existing methods are.\n- Related literature review is adequate and detailed.\n\nQuality:\n- The submission is technically rigid.\n- Claims are well-supported by its clear presentation. Adequate theoretical and empirical results are presented.\n\nClarity:\n- The submission is clearly written and easy to follow. It is very well organized and the story line expands naturally.\n\nSignificance:\n- The results are important to the field. Others are very likely to use the results as a baseline method or build extensions upon it.\n\nWEAKNESSES:\n- It would be more pursuative to elaborate on potential practical limitations of the proposed method. For example, under what case the proposed method may not function efficiently.\n\nQUESTIONS:\n- Is equation (1) first proposed by this paper, or it has been somehow mentioned in related publications?\n- In Figure 1 (h), it seems like existing methods can take advantage of more data at the beginning of optimization. How this trend changes when more samples are available? Is there some theoretical explainable available for the phenomenon?\n\nLIMITATIONS:\nNo potential negative societal impact needs to be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new ADMM algorithm that focuses on three specific cases of rank-based losses. The algorithm's convergence is theoretically analyzed in the paper. Additionally, the authors conducted comprehensive experiments to compare the new algorithm with traditional approaches. The results indicate that the proposed algorithm outperforms traditional methods in terms of both efficiency and effectiveness.\n\nSTRENGTHS:\n1. The authors proposed a new algorithm that can address three specific cases of rank-based losses, whereas traditional algorithms like SGD, DCA, and LSVRG can only handle one. Additionally, the new algorithm permits the regularization term to be a weakly convex function.\n2. The authors' theoretical analysis of the new algorithm's convergence is a significant contribution that has not been achieved in previous work.\n3. To demonstrate the advantages of the new algorithm, the authors conducted comprehensive experiments. The results show that the new algorithm generally outperforms existing methods in terms of both efficiency and effectiveness. Overall, this study provides valuable insights into the development of more efficient and effective algorithms for rank-based losses.\n\nWEAKNESSES:\nSignificant Issues (Detailed Explanation Required)\n1.The representation of the loss vector function in line 23 is unclear. If function l maps an n-dimensional column vector to a real number, how a comparison of magnitude can be made later on?\n\n2.As is widely known, due to the unknown underlying distribution of the data, we can only optimize the discrete form of risk expectation, which is the arithmetic mean of n observations. The theoretical basis for doing so is that the arithmetic mean converges in probability to the expectation. Therefore, could you please explain the theoretical basis for the discrete form of spectral risk that you employed in lines 78-79.\n\n3.Could you please clarify how the conclusion stated in line 116 was derived? If it is a result obtained from citing other papers, please provide the source. If it is a result you have proven yourself, please provide a detailed proof.\n\n4.In line 159, definitions for two consecutive blocks disorder are provided, but definitions for three or more consecutive block disorder are missing.\n\n5.In line 6 of Algorithm 2, it is not possible to select the disordered blocks because definitions for three or more consecutive blocks disorder have not been provided.\n\n6.In line 6 of Algorithm 2, you assume that the optimal solutions for  are equal to each other. Could you please a more detailed proof or a citation if the conclusion is derived from another paper?\n\n7.In line 225, you stated that Assumptions 4 and 5 are weaker than Assumption 2. Please provide a detailed explanation for this claim.\n\n8.Could you please provide specific information on the loss functions and regularizers used in the experiments in Sections 5.2 and 5.3?\n\nMinor issue (needs improvement).\n\n1.The \"l\" in line 23 and the \"l\" in line 71 have different meanings. It’s suggested to use distinct notations to indicate this difference.\n\n2.In Equation (1), the independent variable of  is , where  is a d-dimensional column vector. However, the independent variable of  is written as an n-dimensional column vector in line 24.\n\n3.Line 71 should use lowercase \"d\" instead of uppercase.\n\n4.In line 85, please provide a brief explanation of the term \"fairness\".\n\n5.There is a missing negative sign in line 127.\n\n6.In line 154, \"\" should be in lowercase.\n\n7.The explanations in lines 169-170 appear to contradict the explanations in lines 187-188.\n\n8.To avoid confusion among readers, please use a consistent notation for the loss function \"\" in lines 71, 125, and 194.\n\n9.Please provide an explanation for \"dist\" as used in line 249 at its initial occurrence in line 212.\n\n10.The placement of \"Dataset\" in Tables 1 and 2 is inconsistent.\n\n11.Please use the notation \"new-ADMM\" to distinguish the modified version of the traditional ADMM algorithm.\n\nQUESTIONS:\nPlease see the above comments about  the Weakness.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a unified framework for rank-based loss minimization based on the ADMM algorithm. The paper proposes to apply a pool adjacent violators (PAV) algorithm to solve one of the subproblems of ADMM. Numerical experiments show that the proposed algorithm outperforms the existing ones.

STRENGTHS:
+ The problem of rank-based loss minimization is very important.
+ The proposed PAV algorithm looks interesting.
+ Convergence of the algorithm is theoretically analyzed.

WEAKNESSES:
- It seems that Eq.(1) assumes a linear model and it is unclear how to generalize to non-linear settings.

QUESTIONS:
- Can the proposed method be applied to nonlinear models?

LIMITATIONS:
Limitations are not discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06