Title: Non-Smooth Weakly-Convex Finite-sum Coupled Compositional Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: non-smooth optimization

weakly-convex optimization

compositional optimization

AUC maximization
Abstract: This paper investigates new families of compositional optimization problems, called non-smooth weakly-convex finite-sum coupled compositional optimization (NSWC FCCO). There has been a growing interest in FCCO due to its wide-ranging applications in machine learning and AI, as well as its ability to address the shortcomings of stochastic algorithms based on empirical risk minimization. However, current research on FCCO presumes that both the inner and outer functions are smooth, limiting their potential to tackle a more diverse set of problems. Our research expands on this area by examining non-smooth weakly-convex FCCO, where the outer function is weakly convex and non-decreasing, and the inner function is weakly-convex. We analyze a single-loop algorithm and establish its complexity for finding an $\epsilon$-stationary point of the Moreau envelop of the objective function. Additionally, we also extend the algorithm for solving novel non-smooth weakly-convex tri-level finite-sum coupled compositional optimization problems,  which feature a nested arrangement of three functions. Lastly, we explore the applications of our algorithms in deep learning for two-way partial AUC maximization and multi-instance two-way partial AUC maximization, using empirical studies to showcase the effectiveness of the proposed algorithms.

Non-Smooth Weakly-Convex Finite-sum Coupled Compositional Optimization

Abstract

This paper investigates new families of compositional optimization problems, called n ons mooth w eaklyc onvex f inite-sum c oupled c ompositional o ptimization (NSWCFCCO). There has been a growing interest in FCCO due to its wide-ranging applications in machine learning and AI, as well as its ability to address the shortcomings of stochastic algorithms based on empirical risk minimization. However, current research on FCCO presumes that both the inner and outer functions are smooth, limiting their potential to tackle a more diverse set of problems. Our research expands on this area by examining non-smooth weakly-convex FCCO, where the outer function is weakly convex and non-decreasing, and the inner function is weakly-convex. We analyze a single-loop algorithm and establish its complexity for finding an ϵ -stationary point of the Moreau envelop of the objective function. Additionally, we also extend the algorithm to solving novel non-smooth weakly-convex tri-level finite-sum coupled compositional optimization problems, which feature a nested arrangement of three functions. Lastly, we explore the applications of our algorithms in deep learning for two-way partial AUC maximization and multi-instance two-way partial AUC maximization, using empirical studies to showcase the effectiveness of the proposed algorithms.

1 Introduction

In this paper, we consider two classes of non-convex compositional optimization problems. The first class is formulated as following:

where S denotes a finite set of n items and D i denotes a distribution that could depend on i . The second class is given by:

where S 1 denotes a finite set of n 1 items and S 2 denotes a finite set of n 2 items and D ij denotes a distribution that could depend on ( i, j ) . For simplicity of discussion, we denote by g i ( w ) =

Dixian Zhu

E ξ ∼D i [ g i ( w ; ξ )] : R d → R d 1 and by h i,j ( w ) = E ξ ∼D i,j [ h i,j ( w ; ξ )] : R d → R d 2 . For both classes of problems, we focus our attention on non-convex F with non-smooth non-convex functions f i and g i , which, to the best of our knowledge, has not been studied in any prior works.

The first problem (1) with smooth functions f i and g i has been explored in previous works [26, 15, 21, 33], which is known as finite-sum coupled compositional optimization (FCCO). It is subtly different from standard stochastic compositional optimization (SCO) [27] and conditional stochastic optimization (CSO) [13]. FCCO has been successfully applied to optimizing a wide range of X-risks [33] with convergence guarantee, including smooth surrogate losses of areas under the curves [20] and ranking measures [21], listwise losses [21], and contrastive losses [36]. The second problem (2) is a novel class and is referred to as tri-level finite-sum coupled compositional optimization (TCCO). Both problems differ from traditional two-level or multi-level compositional optimization due to the coupling of variables i, ξ in (1) or the coupling of variables i, j, ξ in (2) at the inner most level.

One limitation of prior works about non-convex FCCO is that their convergence analysis heavily rely on the smoothness conditions of f i and g i [26, 15]. This raises a concern about whether existing techniques can be leveraged for solving non-smooth non-convex FCCO problems with non-asymptotic convergence guarantee. Non-smooth non-convex FCCO and TCCO problems have important applications in ML and AI, e.g., group distributionally robust optimization [4] and two-way partial AUC maximization for deep learning [44]. We defer discussions and formulations of these problems to Section 5. The difficulty for solving smooth FCCO lies at high costs of computing a stochastic gradient ∇ g i ( w ) ∇ f i ( g i ( w )) for a randomly sampled i and the overall gradient ∇ F ( w ) . To approximate the stochastic gradient, a variance-reduced estimator of g i ( w t ) denoted by u i,t is usually maintained and updated for sampled data in the mini-batch i ∈ B t . As a result, the stochastic gradient can be approximated by ∇ g i ( w t ; ξ t ) ∇ f i ( u i,t ) , where ξ t ∼ D i is a random sample. The overall gradient can be estimated by averaging the stochastic gradient estimator over the mini-batch or using variance-reduction techniques. A key insight of the convergence analysis for smooth FCCO is to bound the following error using the L -smoothness of f i , which reduces to bounding the error of u i,t for estimating g i ( w t ) :

Acentral question to be addressed in this paper is 'Can these gradient estimators be used in stochastic optimization for solving non-smooth non-convex FCCO with provable convergence guarantee"? To address this question we focus our attention on a specific class of FCCO/TCCO called non-smooth weakly-convex (NSWC) FCCO/TCCO . This approach aligns with many established works on NSWC optimization [6-9]. Nevertheless, NSWC FCCO/TCCO is more complex than a standard weakly-convex optimization problem because an unbiased stochastic subgradient is not readily accessible. In addition, the convergence measure in terms of the gradient norm of smooth non-convex objectives is not applicable to weakly convex optimization, which will complicate the analysis involving the biased stochastic gradient estimator ∂g i ( w t ; ξ t ) ∂f i ( u t i ) 1 .

Contributions. A major contribution of this paper is to present novel convergence analysis of singleloop stochastic algorithms for solving NSWC FCCO/TCCO problems, respectively. In particular,

For non-smooth FCCO, we analyze the following single-loop updates:

where B t is a random mini-batch of B items, and u i,t is an appropriate variance-reduced estimator of g i ( w t ) that is updated only for i ∈ B t at the t -th iteration. To overcome the non-smoothness, we adopt the tool of Moreau envelop of the objective as in previous works [6, 7]. The key difference of our convergence analysis from previous ones for smooth FCCO is that we bound the inner product ⟨ E i ∂g i ( w ) ∂f i ( u i,t ) , ̂ w t -w t ⟩ , where ̂ w t is the solution of the proximal mapping of the objective at w t . To this end, specific conditions of f i , g i are imposed, i.e., f i is weakly convex and non-decreasing and g i ( w ) is weakly convex, under which we establish an iteration complexity of T = O ( ϵ -6 ) for finding an ϵ -stationary point of the Moreau envelope of F ( · ) .

For non-smooth TCCO, we analyze the following single-loop updates:

where B 1 t and B 2 t are random mini-batches of B 1 and B 2 items, respectively, and u i,t is an appropriate variance-reduced estimator of 1 n 2 ∑ j ∈S 2 g i ( h ij ( w t )) that is updated only for i ∈ B 1 t , and v i,j,t is an appropriate variance-reduced estimator of h i,j ( w t ) that is updated only for i ∈ B 1 t , j ∈ B 2 t . To prove the convergence, we impose conditions of f i , g i , h i,j , i.e., f i is weakly convex and non-decreasing and g i ( · ) is weakly convex and non-decreasing (or monotonic), h ij is weakly convex (or smooth), and establish an iteration complexity of T = O ( ϵ -6 ) for finding an ϵ -stationary point of the Moreau envelope of F ( · ) .

We extend the above algorithms to solving (multi-instance) two-way partial AUC maximization for deep learning, and conduct extensive experiments to verify the effectiveness of the both algorithms.

2 Related work

Smooth SCO. There are many studies about two-level smooth SCO [27, 38, 10, 19, 3, 28] and multi-level smooth SCO [32, 32, 1, 39]. The complexities of finding an ϵ -stationary point for twolevel smooth SCO have been improved from O ( ϵ -5 ) [27] to O ( ϵ -3 ) [19], and that for multi-level smooth SCO have been improved from a level-dependent complexity of O ( ϵ -(7+ K ) / 2 ) [32] to a level-independent complexity of O ( ϵ -3 ) [32], where K is the number of levels. The improvements mostly come from using advanced variance reduction techniques for estimating each level function or its Jacobian and for estimating the overall gradient. Two stochastic algorithms have been developed in [13] for CSO but suffer a limitation of requiring large batch sizes.

Smooth FCCO. FCCO was first introduced in [20] for optimizing average precision. Its algorithm and convergence analysis was improved in [26] and [15]. The former work [26] proposed an algorithm named SOX by using moving average (MA) to estimate the inner function values and the overall gradient. In the smooth non-convex setting, SOX is proved to achieve an iteration complexity of O ( ϵ -4 ) . The latter work [15] proposed a novel multi-block-single-probe variance reduced (MSVR) estimator for estimating the inner function values, which helps achieve a lower iteration complexity O ( ϵ -3 ) . Recently, [11] proposed an extrapolation based estimator for the inner function, which yields a method with a complexity that matches MSVR when n ≤ ϵ 2 / 3 . These techniques have been employed for optimizing various X-risks, including contrastive losses [36], ranking measures and listwise losses [21], and other objectives [26, 15]. However, all of these prior works assume the smoothness of f i and g i . Hence, their analysis is not applicable to NSWC FCCO problems. Our novel analysis of a simple algorithm for NSWC FCCO problems yields an iteration complexity of O ( ϵ -6 ) for using the MSVR estimators of the inner functions. The comparison with [26, 15] is shown in Table 1.

Non-smooth Weakly Convex Optimization. Analysis of weakly convex optimization with unbiased stochastic subgradients was pioneered by [6, 7]. Optimization of compositional functions that are weakly convex have been tackled in earlier works [8, 9], where the inner function is deterministic or does not involve coupling between two random variables. A closely related work to our NSWC FCCO is weakly-convex concave minimax optimization [22]. Assuming f i is convex, (1) can be written as: min w max π ∈ R n 1 n ∑ i ∈S ⟨ π i , g i ( w ) ⟩ -f ∗ i ( π i ) , where f ∗ i ( · ) is the convex conjugate of f i . It can be solved using existing methods [22, 31, 41, 43, 17] but with several limitations: (i) the algorithms in [22, 31, 41, 43] have a comparable complexity of O (1 /ϵ 6 ) but have unnecessary double loops which require setting the number of iterations for the inner loop; (ii) the algorithm in [17] is single loop but has a worse complexity of O (1 /ϵ 8 ) ; (iii) these existing algorithms and analysis does not account for complexity of updating all coordinates of π , which could be prohibitive in

many applications; iv) these approaches are not applicable to NSWC FCCO/TCCO with weakly convex f i . In fact, the double loop algorithm has been leveraged and extended to solving the two-way partial AUC maximization problem, a special case of NSWC FCCO [44], by sampling and updating a batch of coordinates of π at each iteration. However, it is less practical thus not implemented and its analysis did not explicitly show the convergence rate dependency on n + , n -and the block batch size.

They proposed two methods, SCS for smooth g ( x ) and SCS with SPIDER for non-smooth g ( x ) . For both proposed methods, they proved a sample complexity of O (1 /ϵ 6 ) for achieving an ϵ -stationary point of the objective's Moreau envelope 2 . We would like to remark that the above problem with a non-smooth g ( x ) is a special case of NSWC FCCO with only a convex outer function, one block and no coupled structure. Nevertheless, their algorithm for non-smooth g ( · ) suffers a limitation of requiring a large batch size in the order of O (1 /ϵ 2 ) for achieving the same convergence.

Finally, we would like to mention that non-smooth convex or strongly convex SCO problems have been considered in [27, 42, 26], which, however, are out of scope of the present work.

3 Preliminaries

Let ∥ · ∥ be the Euclidean norm of a vector and spectral norm of a matrix. We use Π C [ · ] to denote the Euclidean projection onto { v ∈ R m : ∥ v ∥ ≤ C } . For vectors, inequality notations including ≤ , ≥ , >, < are used to denote element-wise inequality. For an expectation function f ( · ) = E ξ [ f ( · ; ξ )] , let f ( · ; B ) = 1 |B| ∑ ξ ∈B f ( · ; ξ ) be its stochastic unbiased estimator evaluated on a sample batch B . A stochastic unbiased estimator is said to have bounded variance σ 2 if E ξ [ ∥ f ( · ) -f ( · ; ξ ) ∥ 2 ] ≤ σ 2 . The Jacobian matrix of function f : R m 1 → R m 2 is in dimension R m 1 × m 2 . We recall the definition of general subgradient and subdifferential following [6, 24].

Definition 3.1 (subgradient and subdifferential) . Consider a function f : R n → R ∪ {∞} and a point with f ( x ) finite. A vector v ∈ R n is a general subgradient of f at x , if

The subdifferential ∂f ( x ) is the set of subgradients of f at point x .

For simplicity, we abuse the notation and also use ∂f ( x ) to denote one subgradient from the corresponding subgradient set when no confusion could be caused. We use ∂f ( x ; B ) to represent a stochastic unbiased estimator of the subgradient ∂f ( x ) that is evaluated on a sample batch B . A function is called C 1 -smooth if it is continuously differentiable. A function f = ( f 1 , . . . , f m 2 ) : R m 1 → R m 2 is called monotone if ∀ i ∈ { 1 , . . . , m 2 } , f i : R m 1 → R is monotone with respect to each element of the input. Note that if a Lipschitz continuous function f : O → R m 2 is assumed to be non-increasing (resp. non-decreasing), where the domain O ⊂ R m 1 is open, then all subgradients of f are element-wise non-positive (resp. non-negative). We refer the details to Appendix D.1.

A function f is C -Lipschitz continuous if ∥ f ( x ) -f ( y ) ∥ ≤ C ∥ x -y ∥ . A differentiable function f is L -smooth if ∥∇ f ( x ) -∇ f ( y ) ∥ ≤ L ∥ x -y ∥ . A function f : R d → R ∪ {∞} is ρ -weakly-convex if the function f ( · ) + ρ 2 ∥ · ∥ 2 is convex. A vector-valued function f : R d →{ R ∪ {∞}} m is called ρ -weakly-convex if it is ρ -weakly-convex for each output. It is difficult sometimes impossible to find an ϵ -stationary point of a non-smooth weakly-convex function F , i.e., dist (0 , ∂F ( w )) ≤ ϵ . For example, an ϵ -stationary point of function f ( x ) = | x | does not exist for 0 ≤ ϵ < 1 unless it is the optimal solution. To tackle this issue, [6] proposed to use the stationarity of the problem's Moreau envelope as the convergence metric, which has become a standard metric for solving weakly-convex problems [7, 22, 31, 41, 43, 17]. Given a weakly-convex function φ : R m → R , its Moreau envelope and proximal map with λ > 0 are constructed as

The Moreau envelope is an implicit smoothing of the original problem. Thus it attains a continuous differentiation. As a formal statement, the following lemma follows from standard results [6, 18].

Lemma 3.2. Given a ρ -weakly-convex function φ and λ < ρ -1 , the envelope φ λ is C 1 -smooth with gradient given by ∇ φ λ ( x ) = λ -1 ( x -prox λφ ( x )) .

Algorithm 1 Stochastic Optimization algorithm for Non-smooth FCCO (SONX)

1: Initialization: w 0 , { u i, 0 : i ∈ S} .

2: for t = 0 , . . . , T -1 do

3: Draw sample batches B t 1 ∼ S , and B t 2 ,i ∼ D i for each i ∈ B t 1 .

4: u i,t +1 = { (1 -τ ) u i,t + τg i ( w t ; B t 2 ,i ) + γ ( g i ( w t ; B t 2 ,i ) -g i ( w t -1 ; B t 2 ,i )) , i ∈ B t 1 u i,t , i ̸∈ B t 1

5: Compute G t = 1 B 1 ∑ i ∈B t 1 ∂g i ( w t ; B t 2 ,i ) ∂f i ( u i,t )

6: Update w t +1 = w t -ηG t

7: end for

Moreover, for any point x ∈ R m , the proximal point ˆ x := prox λφ ( x ) satisfies [6]

Thus if ∥∇ φ λ ( x ) ∥ ≤ ϵ , we can say x is close to a point ˆ x that is ϵ -stationary, which is called nearly ϵ -stationary solution of φ ( x ) .

4 Algorithms and Convergence

4.1 Non-Smooth Weakly-Convex FCCO

In this section, we assume the following conditions hold for the FCCO problem (1).

Assumption 4.1. For all i ∈ S , we assume that

f i is ρ f -weakly-convex, C f -Lipschitz continuous and non-decreasing;

g i ( · ) is ρ g -weakly-convex and g i ( · ; ξ ) is C g -Lipschitz continuous;

Stochastic gradient estimators g i ( w ; ξ ) and ∂g i ( w ; ξ ) have bounded variance σ 2 .

Proposition 4.2. Under Assumption 4.1, F ( w ) in (1) is ρ F weakly convex with ρ F = √ d 1 ρ g C f + ρ f C 2 g .

One challenge in solving FCCO is the lack of access to unbiased estimation of the subgradients 1 n ∑ i ∈S ∂g i ( w ) ∂f i ( g i ( w )) due to the expectation form of g i ( w ) inside a non-linear function f i . A common solution in existing works for solving smooth FCCO is to maintain function value estimators { u i : i ∈ S} for { g i ( w ) : i ∈ S} , and approximate the true gradient by a stochastic version 1 B 1 ∑ i ∈B 1 ∂g i ( w ; B 2 ) ∂f i ( u i ) [26, 15], where B 1 , B 2 are sampled mini-batches. Simply using a mini-batch estimator of g i inside f i does not ensure convergence if mini-batch size is small.

Inspired by existing algorithms of smooth FCCO, a simple method for solving non-smooth FCCO is presented in Algorithm 1 referred to as SONX. A key step is the step 4, which uses the multiblock-single-probe variance reduced (MSVR) estimator proposed in [15] to update { u i : i ∈ S} in a block-wise manner. It is an advanced variance reduced update strategy for multi-block variable inspired by STORM [5]. In the update of MSVR estimator, for each sampled i ∈ B t 1 , u i,t is updated following a STORM-like rule with a specialized parameter γ = n -B 1 B 1 (1 -τ ) + (1 -τ ) for the error correction term. For the unsampled i ̸∈ B t 1 , no update for u i,t is needed. When γ = 0 , the estimator becomes the moving average estimator analyzed in [26] for smooth FCCO, which is also analyzed in the Appendix. With the function values of { g i ( w t ) : i ∈ S} well-estimated, the gradient can be approximated by G t in step 5. Next, we directly update w t by subgradient descent using the stochastic gradient estimator G t . Note that unlike existing works on smooth FCCO that often maintain a moving average estimator [26] or a STORM estimator [15] for the overall gradient to attain better rates, this is not possible in the non-smooth case as those variance reduction techniques for the overall gradient critically rely on the Lipschitz continuity of ∇ F , i.e., the smoothness of F .

4.2 Non-Smooth Weakly-Convex TCCO

In this section, we consider non-smooth TCCO problem and aim to extend Algorithm 1 to solve it. First of all, for convergence analysis and to ensure the weak convexity of F ( w ) in (2), we make the following assumptions.

Assumption 4.3. For all ( i, j ) ∈ S 1 ×S 2 , we assume that

Algorithm 2 Stochastic Optimization algorithm for Non-smooth TCCO (SONT)

1: Initialization: w 0 , { u i, 0 : i ∈ S 1 } , v i,j, 0 = h i,j ( w 0 ; B 0 3 ,i,j ) for all ( i, j ) ∈ S 1 ×S 2 .

2: for t = 0 , . . . , T -1 do

3: Sample batches B t 1 ⊂ S 1 , B t 2 ⊂ S 2 , and B t 3 ,i,j ⊂ D i,j for i ∈ B t 1 and j ∈ B t 2 .

7: Update w t +1 = w t -ηG t

8: end for

f i is C f -Lipschitz continuous, ρ f -weakly-convex and non-decreasing;

g i is ρ g -weakly-convex and C g -Lipschitz continuous. h i,j ( · ; ξ ) is C h -Lipschitz continuous.

Either g i is non-decreasing, h i,j is L h -weakly-convex or g i is monotone, h i,j is L h -smooth.

Stochastic estimators h i,j ( w , ξ ) and ∂h i,j ( w , ξ ) have bounded variance σ 2 , and ∥ h i,j ( w ) ∥ ≤ ˜ C h . E i ∥ g i ( v ) -1 n 2 ∑ j ∈S 2 g i ( v ) ∥ 2 ≤ σ 2 for any v .

The weak convexity of F ( w ) in (2) is guaranteed by the following Proposition.

Proposition 4.4. Under Assumption 4.3, F ( w ) in (2) is ρ F -weakly-convex with ρ F = √ d 1 ( √ d 2 L h C g + ρ g C 2 h ) C f + ρ f C 2 g C 2 h .

We extend SONX to Algorithm 2 for (2), which is referred to as SONT. For dealing with the extra layer of compositional problem, we maintain another multi-block variable to track the extra layer of function value estimation. To understand this, we first write down the true subgradient:

To approximate this subgradient, we need the estimations of 1 n 2 ∑ j ∈S 2 g i ( h i,j ( w )) and h i,j ( w ) , which can be tracked by using MSVR estimators denoted by { u i,t : i ∈ S 1 } and { v i,j,t : ( i, j ) ∈ S 1 × S 2 } , respectively. As a result, a stochastic estimation of ∂F ( w t ) is computed in step 6 of Algorithm 2, and the model parameter is updated similarly as before.

4.3 Convergence Analysis

In this section, we present the proof sketch of the convergence guarantee for Algorithm 1. The analysis for Algorithm 2 follows in a similar manner. The detailed proofs can be found in Appendix A (please refer to the supplement). Before starting the proof, we define a constant M 2 ≥ C 2 f C 2 g so that under Assumption 4.1 we have E t [ ∥ G t ∥ 2 ] ≤ M 2 . Then we start by giving the error bound of the MSVR estimator in Algorithm 1. The following norm bound of the estimation error follows from the squared-norm error bound in Lemma 1 from [15], whose proof is given in Appendix D.3.

Lemma 4.5. Consider the update for { u i,t : i ∈ S} in Algorithm 1. Assume g i is C g -Lipshitz for all i ∈ S . With γ = n -B 1 B 1 (1 -τ ) +(1 -τ ) , τ ≤ 1 2 , we have

For simplicity, denote by ˆ w t := prox F/ ¯ ρ ( w t ) . Then using the definition of Moreau envelope and the update rule of w t , we can obtain a bound for the change in the Moreau envelope,

where E t [ G t ] = 1 n ∑ i ∈ S 1 ∂g i ( w t ) ∂f i ( u i,t ) is the subgradient approximation based on the MSVR estimator u i,t of the inner function value. This is a standard result in weakly-convex optimization [6].

To bound the inner product ⟨ ˆ w t -w t , E t [ G t ] ⟩ on the right-hand-side of (5), we apply the assumptions that f i is weakly-convex, Lipschitz continuous and non-decreasing, and g i is weakly-convex. Its upper bound is given as follows.

Due to the ρ F -weak convexity of F ( w ) , we have ( ¯ ρ -ρ F )-strong convexity of w ↦→ F ( w )+ ¯ ρ 2 ∥ w t -w ∥ 2 . Then it follows F ( ˆ w t ) -F ( w t ) ≤ ( ρ F 2 -¯ ρ ) ∥ w t -ˆ w t ∥ 2 . Combining this with inequalities (5), (6), and setting ¯ ρ sufficiently large we have

Recall Lemma 3.2, we have ∥ w t -ˆ w t ∥ 2 = 1 ¯ ρ 2 ∥∇ F 1 / ¯ ρ ( w t ) ∥ 2 . Moreover, the last three terms on the R.H.S of inequality (7) can be bounded using the Lipschitz continuity of f i and the error bound given in Lemma 4.5. Then we can conclude the complexity of SONX with the following theorem.

O ( B 1 B 1 / 2 2 ϵ 4 n ) , and ¯ ρ = ρ F + ρ g C f +2 ρ f C 2 g , Algorithm 1 converges to an ϵ -stationary point of the Moreau envelope F 1 / ¯ ρ in T = O ( n B 1 B 1 / 2 2 ϵ -6 ) iterations.

Remark. Similar to the complexity for smooth FCCO problems [26, 15], Theorem 4.6 guarantees that SONX for NSWC FCCO has a parallel speed-up in terms of the batch size B 1 and linear dependency on n . The dependency of the complexity on the batch size B 2 is due to the use of MSVR estimator, which matches the results in [15]. If the MSVR estimator in SONX is replaced by moving average estimator, the complexity becomes O ( n B 1 B 2 ϵ -8 ) (cf. Appendix B).

Following a similar proof strategy, the convergence guarantee of Algorithm 2 is given below.

Theorem 4.7. (Informal) Under Assumption 4.3, with appropriate values of γ 1 , γ 2 , τ 1 , τ 2 , η and a proper constant ¯ ρ , Algorithm 2 converges to an ϵ -stationary point of the Moreau envelope F 1 / ¯ ρ in

Remark. In the worst case, the complexity has a worse dependency on n 1 /B 1 , i.e., O ( n 3 / 2 1 /B 3 / 2 1 ) . This is caused by the two layers of block-sampling update for { u i,t , i ∈ S 1 } and { v i,j,t : ( i, j ) ∈ S 1 ×S 2 } . When n 1 = B 1 = 1 and B 3 ≤ √ n 2 , the complexity of SONT becomes similar as SONX, which is understandable as the inner two levels in TCCO is the same as FCCO.

5 Applications

NSWC FCCO finds important applications in group distributionally robust optimization (group DRO) and two-way partial AUC (TPAUC) maximization.

Consider N groups with different distributions. Each group k has an averaged loss L k ( w ) = 1 n k ∑ n k i =1 ℓ ( f w ( x k i ) , y k i ) , where w is the the model parameter and ( x k i , y k i ) is a data point. It has been shown in previous study [23] that the group DRO problem can be formulated into

This formulation can be mapped into non-smooth weakly-convex FCCO under certain assumptions. Due to space limitation, we defer the comprehensive discussion of group DRO to Appendix E. The rest of this section focuses on TPAUC maximization.

Let X denote an input example and h w ( X ) denote a prediction of a parameterized deep net on data X . Denote by S + the set of n + positive examples and by S -the set of n -negative examples. TPAUC measures the area under ROC curve where the true positive rate (TPR) is higher than α and the false positive rate (FPR) is lower than an upper bound β . A surrogate loss for optimizing TPAUC

with FPR ≤ β , TPR ≥ α is given by [34]:

where ℓ ( · ) is a convex, monotonically non-decreasing surrogate loss of the indicator function I ( h w ( X j ) ≥ h w ( X i )) , S ↑ + [1 , k 1 ] is the set of positive examples with k 1 = ⌊ n + α ⌋ smallest scores, and S ↓ -[1 , k 2 ] is the set of negative examples with k 2 = ⌊ n -β ⌋ largest scores. To tackle the challenge of selecting examples from S ↑ + [1 , k 1 ] and S ↓ -[1 , k 2 ] , the above problem is cast into the following [44]:

where s = ( s 1 , . . . , s n + ) . We will consider two scenarios, namely regular learning scenario where X i ∈ R d 0 is an instance, and multi-instance learning (MIL) scenario where X i = { x 1 i , . . . , x m i i ∈ R d 0 } contains multiple instances (e.g., one patient has hundreds of high-resolution CT images). A challenge in MIL is that the number of instances m i for each data might be large such that it is difficult to load all instances into the memory for mini-batch training. It becomes more nuanced especially because MIL involves a pooling operation that aggregates the predicted information of individual instances into a single prediction, which can be usually written as a compositional function with the inner function being an average over instances from X . For simplicity of exposition, below we consider the mean pooling h w ( X ) = 1 | X | ∑ x ∈ X e ( w e ; x ) ⊤ w c , where e ( w e , x ) is the encoded feature representation of instance x with a parameter w e , and w c is the parameter of the classifier. We will map the regular learning problem as NSWC FCCO and the MIL problem as NSWC TCCO.

The problem (9) is slightly more complicated than (1) or (2) due to the presence of s ′ , s . In order to understand the applicability of our analysis and results to (9), we ignore s ′ , s for a moment. In the regular learning setting when h w ( X ) = e ( w e , X ) ⊤ w c can be directly computed, we can map the problem into NSWC FCCO, where f i ( g, s ′ ) is non-smooth, convex, and non-decreasing in terms of g , and g i ( w , s i ) = ψ i ( w , s i ) is non-smooth, and is proved to be weakly when ℓ ( · ) is convex and h w ( X ) is smooth in terms of w . In the MIL setting with mean pooling, we can map the problem into NSWC TCCO by defining h i ( w ) = 1 | X i | ∑ x ∈ X i e ( w e ; x ) ⊤ w c , h ij ( w ) = h j ( w ) -h i ( w ) and ( ℓ ( h ( w )) -s ) ( g -s ′ )

g i ( h i,j ( w ) , s i ) = s i + i,j i + β , and f i ( g i , s ′ ) = s ′ + i + α , where f i is non-smooth, convex, and non-decreasing in terms of g i , and g i ( h ij ( w ) , s i ) is non-smooth, convex, monotonic in terms of h ij ( w ) when ℓ ( · ) is convex and monotonically non-decreasing, and g i ( h ij ( w ) , s i ) is weakly convex in terms of w when h ij ( w ) is smooth and Lipchitz continuous in terms of w . Hence, the problem (9) satisfies the conditions in Assumption 4.1 for the regular learning setting and that in Assumption 4.3 for the MIL with mean pooling under mild regularity conditions of the neural network. We present full details in Appendix C.1 for interested readers.

To compute the gradient estimator w.r.t w , u i,t will be maintained for tracking g i ( w , s i ) in the regular setting or 1 n -∑ X j ∈S -g i ( h i,j ( w ) , s i ) in the MIL setting, v i,t will be maintained for tracking h i ( w ) in the MIL setting, which are updated similar to that in SONX and SONT. One difference from SONT is that v i,j,t is decoupled into v i,t and v j,t due to that h i,j can be decoupled. In terms of the extra variable s ′ , s , the objective function is convex w.r.t both s ′ and s , which allows us to simply update s ′ by SGD using the stochastic gradient estimator 1 B 1 ∑ i ∈B t 1 ∂ s ′ f i ( u i,t , s ′ t ) and we update [ ]

s i by SGD using the stochastic gradient estimator 1 B 2 ∑ j ∈B t 2 ∂ s i g i ( v j,t -v i,t , s i,t ) ∂ u f i ( u i,t , s ′ t ) . Detailed updates are presented in Algorithm 5 and Algorithm 6 in Appendix C.2. We can extend the convergence analysis of SONX and SONT to the two learning settings of TPAUC maximization, which is included in Appendix C.4. Finally, it is worth mentioning that we can also extend the results to other pooling operations, including smoothed max pooling and attention-based pooling [45]. Due to limit of space, we include discussions in Appendix C.3 as well.

6 Experimental Results

We justify the effectiveness of the proposed SONX and SONT algorithms for TPAUC Maximization in the regular learning setting and MIL setting [14, 45].

Baselines. For regular TPAUC maximization , we compare SONX with the following competitive methods: 1) Cross Entropy (CE) loss minimization; 2) AUC maximization with squared hinge loss (AUC-SH); 3) AUC maximization with min-max margin loss (AUC-M) [37]; 4) Mini-Batch based heuristic loss (MB) [16]; 5) Adhoc-Weighting based method with polynomial function (AWpoly) [35]; 5) a single-loop algorithm (SOTAs) for optimizing a smooth surrogate for TPAUC [44]. For MIL TPAUC maximization , we consider the following baselines: 1) AUC-M with attention-based pooling (AUC-M [att]); 2) SOTAs with attention-based pooling, which is a natural combination between advanced TPAUC optimization and MIL pooling technique; 3) the recently proposed provable multi-instance deep AUC maximization methods with stochastic smoothed-max pooling and attention-based pooling (MIDAM [smx] and MIDAM [att]) [45]. The first two baselines use naive mini-batch pooling for computing the loss function in AUC-M and SOTAs. We implement SONT for MIL TPAUC maximization with attention-based pooling, which is referred to as SONT (att).

Datasets. For regular TPAUC maximization, we use three molecule datasets as in [44], namely moltox21 (the No.0 target), molmuv (the No.1 target) and molpcba (the No.0 target) [29]. For MIL TPAUC maximization, we use four MIL datasets, including two tabular datasets MUSK2 and Fox, and two medical image datasets Colon and Lung. MUSK2 and Fox are two tabular datasets that have been widely adopted for MIL benchmark study [14]. Colon and Lung are two histopathology (medical image) datasets that have large image size (512 × 512) but local interests for classification [2]. For Colon dataset, the adenocarcinoma is regarded as positive label and benign is negative; for Lung dataset, we treat adenocarcinoma as positive and squamous cell carcinoma as negative 3 . For both of the histopathology datasets, we uniformly randomly sample 100 positive and 1000 negative data for experiments. For all MIL datasets, we uniformly randomly split 10% as the testing and the remaining as the training and validation. The statistics for all used datasets are summarized in Table 3and Table 4 in Appendix F.

Experiment Settings. For regular TPAUC maximization, we use the same setting as in [44]. The adopted backbone Graph Nueral Network (GNN) model is Graph Isomorphism Network (GIN), which has 5 mean-pooling layers with 64 number of hidden units and dropout rate 0.5 [30]. We utilize the sigmoid function for the final output layer to generate the prediction score, and set the surrogate loss ℓ ( · ) as squared hinge loss with a margin parameter. We follow the setups for model training and tuning exactly the same as the prior work [44]. Essentially, the model is trained by 60 epochs and the learning rate is decreased by 10-fold after every 20 epochs. The model is initialized as a pretrained model from CE loss on the training datasets. We fix the learning rate of SONX as 1e-2 and moving average parameter τ as 0.9; tune the parameter γ in {0, 1e-1,1e-2,1e-3}, the parameter α, β in {0.1,0.3,0.5} and fix the margin parameter of the surrogate loss ℓ as 1.0, which cost the same

0

10

20

30

40

50

60

Epochs

0.980

0.983

0.985

0.988

0.990

0.993

0.995

0.998

(0.5, 0.5)

=0

=1

e

1

=1

e

2

=1

e

3

(a) molmuv (t1)

0

10

20

30

40

50

60

Epochs

0.0

0.1

0.2

0.3

0.4

0.5

(0.5, 0.5)

=0

=1

e

1

=1

e

2

=1

e

3

0

20

40

60

80

100

Epochs

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

(0.5, 0.5)

=0

=1

e

1

=1

e

2

=1

e

3

(c) MUSK2

0

20

40

60

80

100

Epochs

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

(0.5, 0.5)

=0

=1

e

1

=1

e

2

=1

e

3

tuning effort as the other baselines. The weight decay is set as the same value (2e-4) with the other baselines. For baselines, we directly use the results reported in [44] since we use the same setting.

For MIL TPAUC maximization, we train a simple Feed Forward Neural Network (FFNN) with one hidden layer (the number of neurons equals to data dimension) for the two tabular datasets and ResNet20 for the two medical image datasets. Sigmoid transformation is adopted for the output layer to generate prediction score. The training epoch number is fixed as 100 epochs for all methods; the bag batch size is fixed as 16 (resp. 8) and the number of sampled instances per bag is fixed as 4 (resp. 128) for tabular (resp. medical image) datasets; the learning rate is tuned in {1e-2, 1e-3, 1e-4} and decreased by 10 folds at the end of 50-th and 75-th epoch for all baselines. For SONT (att), we set moving average parameter τ 1 = τ 2 as 0.9; tune the parameter γ 1 = γ 2 = γ in {0, 1e-1,1e-2,1e-3} and fix the margin parameter of the surrogate loss ℓ as 0.5, and the parameter α, β in {0.1,0.5,0.9}. Similar parameters in baselines are set the same or tuned similarly. For all experiments, we utilize 5-fold-cross-validation to evaluate the testing performance based on the best validation performance with possible early stopping choice.

Results. The testing results for the regular and MIL TPUAC maximization with different TPAUC measures are summarized in the Table 2. From Table 2, we observe that our method SONX achieves the best performance for regular TPAUC maximization. It is better than the state-of-the-art method SOTAs for TPAUC maximization. We attribute the better performance of SONX to the fact that the objective of SONX is an exact estimator of TPAUC while the smoothed objective of SOTAs is an inexact estimator of TPAUC. We also observe that SONT (att) achieves the best performance in all cases, which is not surprising since it is the only one that directly optimizes the TPAUC surrogate. In contrast, other baselines either optimizes a different objective (MIDAM) or does not ensure convergence due to the use of mini-batch pooling (AUC-M, SOTAs).

Ablation Study. We conduct ablation studies to demonstrate the effect of the error correction term on the training convergence by varying the γ value for SONX and SONT, where γ 1 = γ 2 = γ is set as the same value in SONT. The training convergence results are presented in Figure 1. We can see that an appropriate value of γ > 0 can yield a faster convergence than γ = 0 , which verifies the faster convergence of using MSVR estimators than using moving average estimators. However, we do observe a gap between theory and practice, as setting a large value of γ > 1 as in the theory might not yield convergence. This phenomenon is also observed in [12]. We conjecture that the gap could be fixed by considering convex objectives [40], which is left as future work.

7 Conclusions

In this paper, we have considered non-smooth weakly-convex two-level and tri-level finite-sum coupled compositional optimization problems. We presented novel convergence analysis of two stochastic algorithms and established their complexity. Applications in deep learning for two-way partial AUC maximization was considered and great performance of proposed algorithms were demonstrated through experiments on multiple datasets. A future work is to prove the convergence of both algorithms for convex objectives.

References

Krishnakumar Balasubramanian, Saeed Ghadimi, and Anthony Nguyen. Stochastic multilevel composition optimization algorithms with level-independent convergence rates. SIAM Journal on Optimization , 32(2):519-544, 2022.

Andrew A Borkowski, Marilyn M Bui, L Brannon Thomas, Catherine P Wilson, Lauren A DeLand, and Stephen M Mastorides. Lung and colon cancer histopathological image dataset (lc25000). arXiv preprint arXiv:1912.12142 , 2019.

Tianyi Chen, Yuejiao Sun, and Wotao Yin. Solving stochastic compositional optimization is nearly as easy as solving stochastic optimization. IEEE Transactions on Signal Processing , 69:4937-4948, 2021.

Sebastian Curi, Kfir Y. Levy, Stefanie Jegelka, and Andreas Krause. Adaptive sampling for stochastic risk-averse learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1036-1047. Curran Associates, Inc., 2020.

Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Damek Davis and Dmitriy Drusvyatskiy. Stochastic model-based minimization of weakly convex functions, 2018.

Damek Davis and Benjamin Grimmer. Proximally guided stochastic subgradient method for nonsmooth, nonconvex problems. SIAM Journal on Optimization , 29(3):1908-1930, 2019.

Dmitriy Drusvyatskiy and Courtney Paquette. Efficiency of minimizing compositions of convex functions and smooth maps. Math. Program. , 178(1-2):503-558, 2019.

John C. Duchi and Feng Ruan. Stochastic methods for composite and weakly convex optimization problems. SIAM Journal on Optimization , 28(4):3229-3259, 2018.

S. Ghadimi, Andrzej Ruszczy'nski, and Mengdi Wang. A single timescale stochastic approximation method for nested stochastic optimization. SIAM J. Optim. , 30:960-979, 2020.

Lie He and Shiva Prasad Kasiviswanathan. Debiasing conditional stochastic optimization, 2023.

Quanqi Hu, Zi-Hao Qiu, Zhishuai Guo, Lijun Zhang, and Tianbao Yang. Blockwise stochastic variance-reduced methods with parallel speedup for multi-block bilevel optimization. In Proceedings of the 39th International Conference on Machine Learning , 2023.

Yifan Hu, Siqi Zhang, Xin Chen, and Niao He. Biased stochastic first-order methods for conditional stochastic optimization and applications in meta learning. Advances in Neural Information Processing Systems , 33, 2020.

Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 2127-2136. PMLR, 10-15 Jul 2018.

Wei Jiang, Gang Li, Yibo Wang, Lijun Zhang, and Tianbao Yang. Multi-block-single-probe variance reduced estimator for coupled compositional optimization, 2022.

Purushottam Kar, Harikrishna Narasimhan, and Prateek Jain. Online and stochastic gradient methods for non-decomposable loss functions. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1 , NIPS'14, page 694-702, Cambridge, MA, USA, 2014. MIT Press.

Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 6083-6093. PMLR, 13-18 Jul 2020.

J.J. Moreau. Proximité et dualité dans un espace hilbertien. Bulletin de la Société Mathématique de France , 93:273-299, 1965.

Qi Qi, Zhishuai Guo, Yi Xu, Rong Jin, and Tianbao Yang. An online method for a class of distributionally robust optimization with non-convex objectives. Advances in Neural Information Processing Systems , 34, 2021.

Qi Qi, Youzhi Luo, Zhao Xu, Shuiwang Ji, and Tianbao Yang. Stochastic optimization of area under precision-recall curve for deep learning with provable convergence. In Advances in neural information processing systems , volume abs/2104.08736, 2021.

Zi-Hao Qiu, Quanqi Hu, Yongjian Zhong, Lijun Zhang, and Tianbao Yang. Large-scale stochastic optimization of NDCG surrogates for deep learning with provable convergence. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 18122-18152. PMLR, 17-23 Jul 2022.

Hassan Rafique, Mingrui Liu, Qihang Lin, and Tianbao Yang. Weakly-convex-concave min-max optimization: provable algorithms and applications in machine learning. Optimization Methods and Software , 37(3):1087-1121, 2022.

R. T. Rockafellar and S. Uryasev. Optimization of conditional valueat-risk. Journal of Risk , 2(3), 1999.

R.T. Rockafellar, M. Wets, and R.J.B. Wets. Variational Analysis . Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2009.

Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2020.

Bokun Wang and Tianbao Yang. Finite-sum coupled compositional stochastic optimization: Theory and applications. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 2329223317. PMLR, 17-23 Jul 2022.

Mengdi Wang, Ethan X Fang, and Han Liu. Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions. Mathematical Programming , 161(1-2):419-449, 2017.

Mengdi Wang, Ji Liu, and Ethan X. Fang. Accelerating stochastic composition optimization, 2016.

Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: a benchmark for molecular machine learning. Chemical science , 9(2):513-530, 2018.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations , 2019.

Yan Yan, Yi Xu, Qihang Lin, Wei Liu, and Tianbao Yang. Optimal epoch stochastic gradient descent ascent methods for min-max optimization. In Advances in Neural Information Processing Systems 33 (NeurIPS) , 2020.

Shuoguang Yang, Mengdi Wang, and Ethan X. Fang. Multi-level stochastic gradient methods for nested composition optimization, 2018.

Tianbao Yang. Algorithmic foundation of deep x-risk optimization. CoRR , abs/2206.00439, 2022.

Tianbao Yang and Yiming Ying. AUC maximization in the era of big data and AI: A survey. ACM Comput. Surv. , 55(8):172:1-172:37, 2023.

Zhiyong Yang, Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao, and Qingming Huang. When all we need is a piece of the pie: A generic framework for optimizing two-way partial auc. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 11820-11829. PMLR, 18-24 Jul 2021.

Zhuoning Yuan, Yuexin Wu, Zi-Hao Qiu, Xianzhi Du, Lijun Zhang, Denny Zhou, and Tianbao Yang. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 25760-25782. PMLR, 17-23 Jul 2022.

Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification, 2021.

Junyu Zhang and Lin Xiao. A stochastic composite gradient method with incremental variance reduction. In Advances in Neural Information Processing Systems , pages 9075-9085, 2019.

Junyu Zhang and Lin Xiao. Multilevel composite stochastic optimization via nested variance reduction. SIAM J. Optim. , 31(2):1131-1157, 2021.

Xuan Zhang, Necdet Serhat Aybat, and Mert Gurbuzbalaban. Robust accelerated primal-dual methods for computing saddle points. 2021.

Xuan Zhang, Necdet Serhat Aybat, and Mert Gurbuzbalaban. Sapd+: An accelerated stochastic method for nonconvex-concave minimax problems, 2023.

Zhe Zhang and Guanghui Lan. Optimal algorithms for convex nested stochastic composite optimization, 2022.

Renbo Zhao. A primal-dual smoothing framework for max-structured non-convex optimization, 2022.

Dixian Zhu, Gang Li, Bokun Wang, Xiaodong Wu, and Tianbao Yang. When AUC meets DRO: Optimizing partial AUC for deep learning with non-convex convergence guarantee. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 27548-27573. PMLR, 17-23 Jul 2022.

Dixian Zhu, Bokun Wang, Zhi Chen, Yaxing Wang, Milan Sonka, Xiaodong Wu, and Tianbao Yang. Provable multi-instance deep auc maximization with stochastic pooling. In International Conference on Machine Learning . PMLR, 2023.

Landi Zhu, Mert Gürbüzbalaban, and Andrzej Ruszczy´ nski. Distributionally robust learning with weakly convex losses: Convergence rates and finite-sample guarantees, 2023.

A Proofs of Theorem 4.6 and Theorem 4.7

In this section, we provide the detailed proofs for Theorem 4.6 and Theorem 4.7. We first give a basic property for weakly-convex functions.

Proposition A.1 (Proposition 2.1 in [7]) . Suppose function g : R d → R ∪ {∞} is lowersemicontinuous. Then g is ρ -weakly-convex if and only if

holds for all vectors v ∈ ∂g ( x ) and x, y ∈ R d

A.1 Proof of Theorem 4.6

Note that the proof of Lemma 4.5 also implies the following squared-norm error bound,

Proof of Theorem 4.6. Define ˆ w t := prox F/ ¯ ρ ( w t ) . For a given i ∈ S , we have

where (a) follows from the ρ f -weak-convexity of f i , (b) follows from that f i ( · ) is non-decreasing and the weak convexity of g i , (c) is due to 0 ≤ ∂f i ( u i,t ) ≤ C f . Then it follows

Now we consider the change in the Moreau envelope:

where

and the second inequality uses the bound of E [ ∥ G t ∥ 2 ] , which follows from the Lipschitz continuity and bounded variance assumptions and is denoted by M .

Combining inequality 29 and 30 yields

Due to the ρ F -weak convexity of F i ( w ) , we have ( ¯ ρ -ρ F )-strong convexity of w ↦→ F i ( w ) + ¯ ρ 2 ∥ w t -w ∥ 2 . Then it follows

Plugging inequality 32 into inequality 31 yields

Set ¯ ρ = ρ F + ρ g C f +2 ρ f C 2 g . We have

where inequality (a) follows from Lemma 3.2.

Using the Lipschitz continuity of f i , we have

By Lemma 4.5, the error bound of the MSVR update gives where

Then

Taking summation from t = 0 to T -1 yields

where (a) uses ∑ T -1 t =0 (1 -µ ) t ≤ 1 µ .

Lower bounding the left-hand-side by min w F ( w ) , we obtain

Thus

Setting

To reach an ϵ -stationary point, we need

A.2 Proof of Theorem 4.7

A formal statement in given below.

Theorem A.2. Under Assumption 4.3, with γ 1 = n 1 n 2 -B 1 B 2 B 1 B 2 (1 -τ 1 ) + (1 -τ 1 ) , γ 2 = n 1 -B 1 B 1 (1 -τ 2 ) + (1 -τ 2 ) , τ 1 = O ( min { B 3 , B 1 / 2 1 n 1 / 2 2 n 1 / 2 1 } ϵ 4 ) ≤ 1 2 , τ 2 = O ( B 2 ϵ 4 ) ≤ 1 2 , η = O ( min { B 1 / 2 3 , B 1 / 4 1 n 1 / 4 2 n 1 / 4 1 , B 1 / 2 1 n 1 / 2 2 n 1 / 2 1 } B 1 B 2 n 1 n 2 ϵ 4 ) , and ¯ ρ = ρ F + 4 ρ f C 2 g + 2 ρ g C f C 2 h + C f C g L h , Algorithm 2 converges to an ϵ -stationary point of the Moreau envelope F 1 / ¯ ρ in T = O ( max { 1 B 1 / 2 3 , n 1 / 4 1 B 1 / 4 1 n 1 / 4 2 , n 1 / 2 1 B 1 / 2 1 n 1 / 2 2 } n 1 n 2 B 1 B 2 ϵ -6 ) iterations.

We first define constant M 2 ≥ max { 3 C 2 f C 2 g σ 2 B 3 + 3 C 2 f C 2 g C 2 h B 2 + 3 C 2 f C 2 g C 2 h B 1 , ˜ C 2 h + σ 2 } so that E t [ ∥ G t ∥ 2 ] ≤ M 2 and ∥ v i,j,t ∥ 2 ≤ M 2 for all i ∈ S 1 , j ∈ S 2 and t . Then to prove Theorem A.2, we need the following Lemmas.

Lemma A.3. Consider MSVR update for v . Assume h i,j ( w ; ξ ) is C h -Lipshitz for all ( i, j ) ∈ S 1 × S 2 , and E [ ∥ G t ∥ 2 ] ≤ M 2 . With γ 1 = n 1 n 2 -B 1 B 2 B 1 B 2 (1 -τ 1 ) +(1 -τ 1 ) , and τ 1 ≤ 1 2 , we have

Lemma A.4. Consider MSVR update for u . Assume g i ( · ) is C g -Lipshitz for all i ∈ S 1 . With γ 2 = n + -B 1 B 1 (1 -τ 2 ) +(1 -τ 2 ) and τ 2 ≤ 1 2 , we have

Proof of Theorem A.2. Consider the change in the Moreau envelope:

Note that

and the second inequality uses the bound of E [ ∥ G t ∥ 2 ] , which follows from the Lipschitz continuity and bounded variance assumptions and is denoted by M .

where (a) follows from the convexity of f i , (b) uses the assumption that f i ( · ) is non-decreasing and g i is weak convex, (c) is due to 0 ≤ ∂f i ( u i,t ) ≤ C f .

The L h -smoothness assumption of h i,j ( w ) (or weakly-convexity of h i,j ( w ) , then only the second inequality holds) for all i, w implies

We first assume that g i ( · ) is non-increasing. Since ∂f i ( u i,t ) ≥ 0 and ∂g i ( v i,j,t ) ≤ 0 , we bound A 1 as following

where inequality (a) follows from the first inequality in (20), (b) follows from the Lipschitz continuity and monotone assumptions on f i , g i , h i,j . On the other hand, if we assume g i ( · ) is non-decreasing, we may use the second inequality in (20) and obtain the same result as (21). Now plugging the new formulation of A 1 back to inequality 19 yields

Combining inequality 22 and 18 yields

where (a) follows from the Lipschitz continuity of f i , g i , h i,j .

Due to the ρ F -weak convexity of F i ( w ) , we have ( ¯ ρ -ρ F )-strong convexity of w ↦→ F i ( w ) + ¯ ρ 2 ∥ w t -w ∥ 2 . Then it follows

Plugging inequality 23 back into A.2, we obtain E t [ F 1 / ¯ ρ ( w t +1 )]

where in inequality (a) we use ¯ ρ = ρ F + 4 ρ f C 2 g + 2 ρ g C f C 2 h + C f C g L h and C 1 = max { 2 C f C g , 2 C f , (2 ρ f C 2 g + ρ g C f ) , ρ f } , and equality (b) uses Lemma 3.2.

With general error bounds

where µ min = min { µ 1 , µ 2 } .

Taking summation from t = 0 to T -1 yields E [ F 1 / ¯ ρ ( w T )]

where we use ∑ T -1 t =0 (1 -µ min ) t ≤ 1 µ min and define constant ∆ 0 such that

Then it follows

where we define constant ∆ such that F 1 / ¯ ρ ( w 0 , s 0 , s ′ 0 ) -E [ F 1 / ¯ ρ ( w T , s T , s ′ T )] ≤ ∆ . With MSVR updates for v i,j,t and u i,t , following from Lemma A.3 and Lemma A.4, we have

Then

Algorithm 3 Stochastic Optimization algorithm for Non-smooth FCCO with coordinate moving average

1: Initialization: w 0 , { u i, 0 : i ∈ S} .

2: for t = 0 , . . . , T -1 do

3: Draw sample batches B t 1 ∼ S , and B t 2 ,i ∼ D i for each i ∈ B t 1 .

5: Compute G t = 1 B 1 ∑ i ∈B t 1 ∂g i ( w t ; B t 2 ,i ) ∂f i ( u i,t )

6: Update w t +1 = w t -ηG t

7: end for

8: return w ¯ t with uniformly sampled ¯ t ∈ { 0 , T -1 } .

Setting then with

we have

B Solving Non-smooth FCCO and TCCO with Coordinate Moving Average

In this section we consider solving non-smooth weakly-convex FCCO and TCCO without variance reduction method. To be specific, we use coordinate moving average updates for function values estimations instead of MSVR. This allows us to weaken the assumption on the Lipschitz continuity, i.e. the Lipschitz continuity of the stochastic function value estimation is not required, and can be replaced by the Lipschitz continuity of the function value. Moreover, compared with MSVR, coordinate moving average update does not need the stochastic evaluation from the previous iteration, and thus has a simpler implementation. However, as a result of not using variance reduction technique, the algorithms suffer from worse convergence rates in terms of ϵ .

B.1 Solving Non-smooth FCCO with Coordinate Moving Average

We first assume the followings assumptions hold.

Assumption B.1. For all i ∈ S , we assume that

f i ( · ) is ρ f -weakly-convex, C f -Lipschitz continuous and non-decreasing;

g i ( · ) is ρ g -weakly-convex and C g -Lipschitz continuous;

Stochastic gradient estimators g i ( w ; ξ ) and ∂g i ( w ; ξ ) have bounded variance σ 2 .

With coordinate moving average update, we present the following lemma of error bound.

Lemma B.2. Consider the coordinate moving average update for { u i,t : i ∈ S 1 } in Algorithm 3, assume g i ( w ) is C g -Lipschitz continuous for all i ∈ S 1 and τ ≤ 1 , then we have

Then we have a convergence analysis similar to Theorem 4.6.

Theorem B.3. Consider non-smooth weakly-convex FCCO problem, under Assumption B.1, setting τ = O ( B 2 ϵ 4 ) ≤ 1 , η = O ( B 1 B 2 n 1 ϵ 6 ) , Algorithm 3 converges to an ϵ -stationary point of the Moreau envelope F 1 / ¯ ρ in T = O ( n 1 B 1 B 2 ϵ -8 ) iterations.

Proof of Theorem B.3. Since the only difference between SONX and Algorithm 3 is the update for { u i,t : i ∈ S 1 } , the proof of Theorem 4.6 still holds with the error bound replaced by Lemma B.2, i.e.,

Then proof proceeds to

Setting iterations.

B.2 Solving Non-smooth TCCO with Coordinate Moving Average

We first assume the following assumptions hold.

Assumption B.4. For all ( i, j ) ∈ S 1 ×S 2 , we assume that

f i ( · ) is ρ f -weakly-convex, C f -Lipschitz continuous and non-decreasing;

g i ( · ) is ρ g -weakly-convex and C g -Lipschitz continuous. h i,j ( · ) is differentiable and C h -Lipschitz continuous.

Either g i is monotone and h i,j ( · ) is L h -smooth, or g i is non-decreasing and h i,j ( · ) is L h -weaklyconvex.

Stochastic estimators h i,j ( w , ξ ) , ∂h i,j ( w , ξ ) and g i ( v i,j ) have bounded variance σ 2 , and ∥ h i,j ( w ) ∥ ≤ ˜ C h .

With coordinate moving average update, we present the following lemmas of error bounds.

then to reach a nearly ϵ -stationary point, Algorithm 3 needs

Algorithm 4 Stochastic Optimization algorithm for Non-smooth TCCO with coordinate moving average

1: Initialization: w 0 , { u i, 0 : i ∈ S 1 } , v i,j, 0 = h i,j ( w 0 ; B 0 3 ,i,j ) for all ( i, j ) ∈ S 1 ×S 2 .

2: for t = 0 , . . . , T -1 do

3: Sample batches B t 1 ⊂ S 1 , B t 2 ⊂ S 2 , and B t 3 ,i,j ⊂ D i,j for i ∈ B t 1 and j ∈ B t 2 .

7:

Update

8: end for

9: return w ¯ t with uniformly sampled ¯ t ∈ { 0 , T -1 } .

Lemma B.5. Consider the coordinate moving average update for { v i,j,t : ( i, j ) ∈ S 1 × S 2 } in Algorithm 4, assume h i,j ( w ) is C h -Lipschitz continuous for all ( i, j ) ∈ S 1 ×S 2 and τ 1 ≤ 1 , then we have

Lemma B.6. Consider the coordinate moving average update for { u i,t : i ∈ S 1 } in Algorithm 4, assume g i ( · ) is C g -Lipschitz continuous for all i ∈ S 1 and τ 2 ≤ 1 , then we have

Then we have a convergence analysis similar to Theorem A.2.

Theorem B.7. Consider non-smooth weakly-convex TCCO problem, under Assumption B.4, setting τ 1 = O ( min { B 3 ϵ 4 , B 1 / 2 1 n 1 / 2 2 n 1 / 2 1 B 1 / 2 2 B 2 ϵ 6 }) ≤ 1 , τ 2 = O ( B 2 ϵ 4 ) ≤ 1 , η =

w

t

+1

=

w

t

-

ηG

t

Proof of Theorem B.7. Since the only difference between SONT and Algorithm 4 is the update for { u i,t : i ∈ S 1 } and { v i,j,t : ( i, j ) ∈ S 1 ×S 2 } , the proof of Theorem A.2 still holds with the error bound replaced by Lemma B.5 and Lemma B.6, i.e.,

Then the proof proceeds to

Setting

then to reach a nearly ϵ -stationary point, Algorithm 4 need

iterations.

C Details for TPAUC Maximization

C.1 Assumption Verification

We first present two lemmas about the weak convexity of the objective in the regular learning setting and in the multi-instance learning setting with mean pooling.

Lemma C.1. Consider the formulation in problem (9) in the regular learning setting and assume that function ℓ ( · ) is non-decreasing, C ℓ Lipschitz continuous and ρ ℓ -weakly-convex, and function h w ( X i ) is C h Lipschitz continuous and ρ h -weakly-convex. then the following statements are true:

f i ( g, s ′ ) is convex and C f -Lipschitz continuous w.r.t. ( g, s ′ ) , and non-decreasing w.r.t. g .

ψ i ( w , s i ) is ρ ψ -weakly-convex w.r.t. ( w , s i ) , and the stochastic estimator of the finite sum function value ψ i ( w , s i ) is C ψ -Lipschitz continuous w.r.t. ( w , s i ) .

1 n + ∑ i ∈S + f i ( ψ i ( w , s i ) , s ′ ) is ρ F -weakly-convex w.r.t. ( w , s , s ′ ) .

Lemma C.2. Consider the formulation in problem (9) in the multi-instance learning setting with mean pooling, and assume that function h i ( w ) = 1 | X i | ∑ x ∈ X i e ( w e ; x ) ⊤ w c is ˜ L h -smooth and is bounded by ˜ C h , and h i ( w ; ξ ) = e ( w e ; ξ ) ⊤ w c is C h -Lipschitz continuous and has bounded variance σ 2 , ℓ is non-decreasing and L ℓ -weakly-convex, then the followings are true:

f i ( g, s ′ ) is convex and C f -Lipschitz-continuous w.r.t. ( g, s ′ ) , and non-decreasing w.r.t. g ;

g i ( v, s i ) = s i + ( ℓ ( v ) -s i ) + β is ρ g -weakly convex and non-decreasing w.r.t. v , convex w.r.t. s i , and C g -Lipschitz continuous w.r.t. ( v, s i ) ;

h i,j ( w ) = h j ( w ) -h i ( w ) is L h -weakly-convex, and h i,j ( w ; ξ, ζ ) is C h -Lipschitz continuous;

1 n + ∑ X i ∈S + f i ( g i ( h i,j ( w ) , s i ) , s ′ ) is ρ F -weakly-convex w.r.t. ( w , s , s ′ ) .

C.1.1 Proof of Lemma C.1

Proof of Lemma C.1. The convexity of f i ( g, s ′ ) with respect to ( g, s ′ ) follows from the convexity definition. With subgradients ∂ s ′ f i ( g, s ′ ) ∈ [1 -1 α , 1] , ∂ g f i ( g, s ′ ) ∈ [0 , 1 α ] , we can see that f i ( g, s ′ ) is 1 α -Lipschitz continuous w.r.t. ( g, s ′ ) , and non-decreasing w.r.t. u .

We first show that ℓ ( h w ( X j ) -h w ( X i )) is weakly-convex w.r.t. w . ℓ ( h ˜ w ( X j ) -h ˜ w ( X i )) ≥ ℓ ( h w ( X j ) -h w ( X i )) + ⟨ ∂ℓ ( h w ( X j ) -h w ( X i )) , ( h ˜ w ( X j ) -h ˜ w ( X i )) -( h w ( X j ) -h w ( X i )) ⟩ + ρ ℓ 2 ∥ ( h ˜ w ( X j ) -h ˜ w ( X i )) -( h w ( X j ) -h w ( X i )) ∥ 2 ( a ) ≥ ℓ ( h w ( X j ) -h w ( X i )) + ⟨ ∂ℓ ( h w ( X j ) -h w ( X i )) , ⟨∇ h w ( X j ) -∇ h w ( X i ) , ˜ w -w ⟩⟩ +2 ρ ℓ C 2 h ∥ ˜ w -w ∥ 2

where (a) uses the weak-convexity of h w ( X i ) and h w ( X j ) ,

Thus ℓ ( h w ( X j ) -h w ( X i )) is 4 ρ ℓ C 2 h -weakly-convex w.r.t. w .

where (a) follows from the monotonicity of ψ i w.r.t. ℓ and weak-convexity of ℓ ( h w ( X j ) -h w ( X i )) , and (b) is due to the Lipschitz continuity of ( ℓ, s i ) ↦→ s i + ( ℓ -s i ) + β w.r.t. ℓ . Thus ψ i is 4 ρ ℓ C 2 h β -weaklyconvex w.r.t. ( w , s i ) .

With a similar argument using the convexity and Lipschitz continuity of f i ( g, s ′ ) w.r.t. ( g, s ′ ) and the weak-convexity of ψ i ( w , s i ) , we can show that f i ( ψ i ( w , s i ) , s ′ ) is 4 ρ ℓ C 2 h β -weakly-convex w.r.t. ( w , s i , s ′ ) . Thus, F ( w , s i , s ′ ) is ρ F = 4 ρ ℓ C 2 h β -weakly-convex w.r.t. ( w , s , s ′ ) .

Now we show the Lipschitz continuity of ψ i ( w , s i ; X j ) , i.e. an unbiased stochastic estimator of ψ i ( w , s i ) . We have

Thus ψ i ( w , s i ; X j ) is (2 + β 2 ) -Lipschitz continuous w.r.t. ( w , s i ) .

C.1.2 Proof of Lemma C.2

Proof of Lemma C.2. First of all, the convexity of f i ( u, s ′ ) w.r.t. ( u, s ′ ) and the convexity of g i ( v ij , s i ) w.r.t. ( ℓ, s i ) directly follows from the convexity definition. Moreover, one can see from the formulation that ∂ s ′ f i ( g, s ′ ) ∈ [1 -1 α , 1] , ∂ u f i ( g, s ′ ) ∈ [0 , 1 α ] , ∂ ℓ g i ( v ij , s i ) ∈ [1 -1 β , 1] , ∂ s i g i ( v ij , s i ) ∈ [0 , 1 β ] . Thus f i is C f = 1 α -Lipschitz continuous w.r.t. ( u, s ′ ) and non-decreasing w.r.t. u , g i is 1 β -Lipschitz continuous w.r.t. ( ℓ, s i ) and non-decreasing w.r.t. ℓ . Since ℓ ( · ) is nondecreasing, g i ( v ij , s i ) is non-decreasing w.r.t. v ij . As a result of Proposition 4.2, g i ( v ij , s i ) is ρ g = 1 β L ℓ -weakly-convex w.r.t. v ij . Due to the composition structure and the Lipschitz continuity of g i and ℓ , one can see that g i ( v ij , s i ) is C g = 1 β C ℓ -Lipschitz continuous w.r.t. ( v ij , s i ) .

The L h = 2 ˜ L h -weakly-convexity of h i,j ( w ) and C h = 2 ˜ C h -Lipschitz continuity of h i,j ( w ; ξ, ζ ) directly follows from the ˜ L h -smoothness of h i ( w ) and ˜ C h -Lipschitz continuity of h i ( w ; ξ ) . Finally,

we show the weakly-convexity of f i ( g i ( h i,j ( w ) , s i ) , s ′ ) :

f ′

where (a) uses the convexity of f i , (b) uses the monotonicity of f i w.r.t. u and convexity of g i ( ℓ, s i ) w.r.t. ( ℓ, s i ) , (c) uses monotonicity of f i w.r.t. u , monotonicity of g i w.r.t. ℓ and L ℓ -weak-convexity of ℓ , (d) uses the smoothness of h i,j . Thus f i ( g i ( h i,j ( w ) , s i ) , s ′ ) is ρ F = ( C f C g C 2 h L ℓ + C f C g L h ) -weakly-convex w.r.t. ( w , s i , s ′ ) . Therefore, 1 n + ∑ i ∈S + f i ( g i ( h i,j ( w ) , s i ) , s ′ ) is ρ F -weakly-convex w.r.t. ( w , s , s ′ ) .

C.2 Algorithms for TPAUC and Multi-instance TPAUC Maximization

Algorithm 5 SONX for TPAUC

1: Initialization: w 0 , { u i, 0 : i ∈ S + } , { s i, 0 : i ∈ S + } , s ′ 0

2: for t = 0 , . . . , T -1 do

3: Sample batches B t 1 ⊂ S + and B t 2 ⊂ S -.

8: Update w t +1 = w t -ηG t

9: end for

10: return w ¯ t with ¯ t uniformly sampled from { 0 , . . . , T -1 } .

Algorithm 6 SONT for Multi-instance TPAUC

1: Initialization: w 0 , { u i, 0 : i ∈ S + } , { s i, 0 : i ∈ S + } , s ′ 0 , { v i,j, 0 : ( i, j ) ∈ S + ×S -}

2: for t = 0 , . . . , T -1 do

3: Sample batches B t 1 ⊂ S + , B t 2 ⊂ S -, and B t 3 ,i ⊂ X i for i ∈ B t 1 ∪ B t 2 .

10:

Update

11: end for

12: return w ¯ t with ¯ t uniformly sampled from { 0 , . . . , T -1 } .

C.3 TPAUC in MIL with smoothed-max pooling and attention-based pooling

We can extend our results to smoothed-max pooling and attention-based pooling.

Smoothed-max Pooling. The smoothed-max pooling can be written as [45]:

where τ > 0 is a hyperparameter and ϕ ( w ; x ) = e ( w e , x ) ⊤ w c is the prediction score for instance x . We can see that h w ( X ) itself is a compositional function. To map the problem into TCCO, we define h i ( w ) = 1 | X i | ∑ x ∈ X i exp( ϕ ( w ; x ) /τ ) + C , where C > 0 is a constant. Then the objective function becomes

In this case we define g i ( ℓ ( v ) , s i ) = s i + ( ℓ ( τ log v 1 -τ log v 2 ) -s i ) + β and h i,j ( w ) = [ h i ( w ) , h j ( w )] . Wecan still prove that g i ( ℓ ( v ) , s i ) is monotone w.r.t to each component of v . It is not difficult to prove that ℓ ( τ log v 1 -τ log v 2 ) is weakly convex w.r.t v because τ log v 1 -τ log v 2 is a smooth mapping of v due to v ≥ C and ℓ is a convex function [8]. As a result, since g i ( ℓ, s i ) is non-decreasing and convex w.r.t to ℓ , it is easy to prove that g i ( ℓ ( v ) , s i ) is weakly convex w.r.t v and is monotone (either non-decreasing or non-increasing) w.r.t to each component of v . Hence, assuming h i ( w ) is a smooth and Lipchitz continuous function, we can prove that g i ( h i,j ( w ) , s i ) is weakly convex w.r.t. to w .

Attention-based Pooling. Attention-based pooling was recently introduced for deep MIL [14], which aggregates the feature representations using attention, i.e.,

w

t

+1

=

w

t

-

ηG

t

where g ( w ; x ) is a parametric function, e.g., g ( w ; x ) = w ⊤ a tanh ( V e ( w e ; x )) + C , where V ∈ R m × d o and w a ∈ R m . Based on the aggregated feature representation, the bag level prediction can be computed by

where δ ( w ; x ) = w ⊤ c e ( w e ; x ) .

We can see that h w ( X ) itself is a compositional function. To map the problem into TCCO, we define h 1 i ( w ) = 1 | X i | ∑ x ∈ X i exp( g ( w ; x )) δ ( w ; x ) , and h 2 i ( w ) = 1 | X i | ∑ x ′ ∈ X i exp( g ( w ; x ′ )) . Assume | w ⊤ a tanh ( V e ( w e ; x )) | ≤ C b then h 2 i ( w ) ≥ exp( C -C b ) . Then the objective function becomes

In this case we define g i ( ℓ ( v ) , s i ) = s i + ( ℓ ( v 3 v 4 -v 1 v 2 ) -s i ) + β and h i,j ( w ) = [ h 1 i ( w ) , h 2 i ( w ) , h 1 j ( w ) , h 2 j ( w )] . We can still prove that g i ( ℓ ( v ) , s i ) is monotone w.r.t to each component of v . It is not difficult to prove that ℓ ( v 3 v 4 -v 1 v 2 ) is weakly convex w.r.t v because v 3 v 4 -v 1 v 2 is a smooth mapping of v when v 2 , v 4 are lower bounded and ℓ is a convex function [8]. As a result, since g i ( ℓ, s i ) is non-decreasing and convex w.r.t to ℓ , it is easy to prove that g i ( ℓ ( v ) , s i ) is weakly convex w.r.t v and is monotone (either non-decreasing or non-increasing) w.r.t to each component of v . Hence, assuming h 1 i ( w ) , h 2 i ( w ) are smooth and Lipchitz continuous, we can prove that g i ( h i,j ( w ) , s i ) is weakly convex w.r.t. to w .

C.4 Convergence Analysis of TPAUC Maximization

C.4.1 Convergence analysis for Algorithm 5

We first consider TPAUC maximization in the regular learning setting. Define F ( w , s , s ′ ) := 1 n + ∑ X i ∈S + f i ( ψ i ( w , s i ) , s ′ ) . Due to the weak-convexity of F ( w , s , s ′ ) w.r.t. ( w , s , s ′ ) , we consider the following Moreau envelope and proximal map defined as

Following the same proof of Lemma 4.5, we have the following error bound

Lemma C.3. Consider the update for { u i,t : X i ∈ S + } in Algorithm 5. Assume ψ i ( w , s i ) is C ψ -Lipshitz continuous for all X i ∈ S + . Assume E t [ ∥ G t ∥ 2 ] ≤ M 2 and E t [ ∥ 1 B 1 ∑ X i ∈B t 1 ∂ s ψ i ( w t , s i,t ; B t 2 ) ∂ u f ( u i,t , s ′ t ) e i ∥ 2 ] ≤ M 2 , where e i is the n + -dimensional vector with 1 at the i -th entry and 0 everywhere else. With γ = n + -B 1 B 1 (1 -τ ) +(1 -τ ) and τ ≤ 1 2 , we have

Then we have following convergence guarantee.

Theorem C.4. Under the assumptions given in Lemma C.1, with γ = n + -B 1 B 1 (1 -τ ) + (1 -τ ) , τ = O ( B 2 ϵ 4 ) ≤ 1 2 , η = O ( B 1 B 1 / 2 2 ϵ 4 n + ) , and ¯ ρ = ρ F + ρ ψ C f , Algorithm 5 converges to an ϵ -stationary point of the Moreau envelope F 1 / ¯ ρ in T = O ( n + B 1 B 1 / 2 2 ϵ -6 ) iterations.

Proof of Theorem C.4. Define ( ˆ w t , ˆ s t , ˆ s ′ t ) := prox F/ ¯ ρ ( w t , s t , s ′ t ) . For a given X i ∈ S + , we have f i ( ψ i ( ˆ w t , ˆ s i,t ) , ˆ s ′ t ) -f i ( u i,t , s ′ t )

where (a) follows from the convexity of f i , (b) follows from the monotonicity of f i ( · , s ′ ) and weak convexity of ψ i , (c) is due to 0 ≤ ∂ u f i ( u i,t , s ′ t ) ≤ C f . Then it follows

Now we consider the change in the Moreau envelope:

where for simplicity we denote G 1 t = 1 B 1 ∑ X i ∈B t 1 ∂ u f i ( u i,t , s ′ t ) ∂ s ψ i ( w t , s i,t ; B t 2 ) and G 2 t = 1 B 1 ∑ X i ∈B t 1 ∂ s ′ f i ( u i,t , s ′ t ) . The second inequality in the above derivation uses the bounds of E [ ∥ G t ∥ 2 ] , E [ ∥ G 1 t ∥ 2 ] and E [ ∥ G 2 t ∥ 2 ] , which follow from the Lipschitz continuity and bounded variance

assumptions and are denoted by M . Moreover, we have

and

Combining inequality 29 and 30 yields

Due to the ρ F -weak convexity of F ( w , s , s ′ ) , we have ( ¯ ρ -ρ F )-strong convexity of ( w , s , s ′ ) ↦→ F ( w , s , s ′ ) + ¯ ρ 2 ∥ ( w t , s t , s ′ t ) -( w , s , s ′ ) ∥ 2 . Then it follows

Plugging inequality 32 into inequality 31 yields

Set ¯ ρ = ρ F + ρ ψ C f . We have

where inequality (a) follows from Lemma 3.2.

Using the Lipschitz continuity of f , we have

With the error bound from Lemma C.3, we have

Taking summation from t = 0 to T -1 yields

Lower bounding the left-hand-side by min w , s ,s ′ F 1 / ¯ ρ ( w , s , s ′ ) , we obtain

where we assume F 1 / ¯ ρ ( w 0 , s 0 , s ′ ) -min w , s ,s ′ F 1 / ¯ ρ ( w , s , s ′ ) ≤ ∆ and

0 C = max { 8∆ , 12¯ ρM 2 , 32 C f ¯ ρ ∑ X i ∈ S + ∥ ψ i ( w 0 , s i, 0 ) -u i, 0 ∥ , 16 C f ¯ ρ }

Plugging the expression of µ and R yields

Setting τ = O ( B 2 ϵ 4 ) and η = O ( B 1 B 1 / 2 2 n + ϵ 4 ) , with T = O ( n + B 1 B 1 / 2 ϵ -6 ) iterations, we have

C.4.2 Convergence analysis for Algorithm 6

We now consider MIL TPAUC maximization with mean pooling. Define F ( w , s , s ′ ) := 1 n + ∑ X i ∈S + f i ( g i ( h j ( w ) -h i ( w ) , s i ) , s ′ ) . Due to the weak-convexity of F ( w , s , s ′ ) w.r.t. ( w , s , s ′ ) , we consider the following Moreau envelope and proximal map defined as

Following the same proofs of Lemma A.3 and Lemma A.4, we have the following error bounds

Lemma C.5. Consider the update for { v i,t : X i ∈ S + ∪ S -} in Algorithm 6. Assume h i ( w ; ξ ) is C h -Lipshitz for all X i ∈ S + ∪ S -, and E [ ∥ G t ∥ 2 ] ≤ M 2 . With γ 1 = n + -B 1 B 1 (1 -τ 1 ) + (1 -τ 1 ) ,

LemmaC.6. Consider update for { u i,t : X i ∈ S + } in Algorithm 6. Assume g i ( v ij , s i ) is C g -Lipshitz w.r.t. ( v ij , s i ) for all X i ∈ S + and X j ∈ S -. With γ 3 = n + -B 1 B 1 (1 -τ 2 ) +(1 -τ 2 ) and τ 2 ≤ 1 2 , we have

where C 2 is a constant defined in the proof.

Then we have the following covnergence guarantee.

Theorem C.7. Under assumptions given in Lemma C.2, with γ 1 = n 1 -B 1 B 1 (1 -τ 1 ) + (1 -τ 1 ) , γ 2 = n 2 -B 2 B 2 (1 -τ 1 ) + (1 -τ 1 ) , γ 3 = n 1 -B 1 B 1 (1 -τ 2 ) + (1 -τ 2 ) , τ 1 = O ( min { B 3 , B 1 n + min { n 1 / 2 + B 1 / 2 1 , n 1 / 2 -B 1 / 2 2 } B 1 / 2 2 } ϵ 4 ) ≤ 1 / 2 , τ 2 = O ( B 2 ϵ 4 ) ≤ 1 / 2 , η = O ( min { min { B 1 n + , B 2 n -} min { B 1 / 2 3 , B 1 / 2 1 n 1 / 2 + min { n 1 / 4 + B 1 / 4 1 , n 1 / 4 -B 1 / 4 2 } B 1 / 4 2 } , B 1 n + min { B 1 / 2 1 n 1 / 2 + , B 1 / 2 2 n 1 / 2 -} B 1 / 2 3 } ϵ 4 ) , then after T ≥ O ( max { max { n + B 1 , n -B 2 } max { 1 B 1 / 2 3 , n 1 / 2 + B 1 / 2 1 max { B 1 / 4 1 n 1 / 4 + , B 1 / 4 2 n 1 / 4 -} 1 B 1 / 4 2 } , n + B 1 max { n 1 / 2 + B 1 / 2 1 , n 1 / 2 -B 1 / 2 2 } 1 B 1 / 2 2 } ϵ -6 ) iterations, Algorithm 6 gives ϵ -stationary point to the Moreau envelope, i.e., .

1 T T -1 ∑ t =0 ∥∇ F 1 / ¯ ρ ( w t , s t , s ′ t ) ∥ 2 ≤ ϵ 2 where ¯ ρ = ρ F + ρ g C f +8 ρ g C f C h + C f C g L h .

Proof of Theorem C.7. Consider the change in the Moreau envelope:

where for simplicity we denote G 2 t = 1 B 1 ∑ i ∈B t 1 ∂ s ′ f i ( u i,t , s ′ t ) , and G 1 t is a n + -dimensional vector whose i -th coordinate is defined as

The second inequality in the above derivation uses the bounds of E [ ∥ G t ∥ 2 ] , E [ ∥ G 1 t ∥ 2 ] and E [ ∥ G 2 t ∥ 2 ] , which follow from the Lipschitz continuity and bounded variance assumptions and are denoted by M .

where (a) follows from the convexity of f i , (b) follows from the monotonicity of f i ( · , s ′ ) and weak convexity of g i , (c) is due to 0 ≤ ∂ u f i ( u i,t , s ′ t ) ≤ C f .

The L h -smoothness assumption of h i ( w ) -h j ( w ) for all i, w implies

where inequality (a) follows from inequality 39, (b) follows from the Lipschitz continuity and monotone assumptions on f i , g i , h i , h j . Then plugging the new formulation of A 1 back to inequality 38

yields

Taking average over i ∈ S + gives

Combining inequality 37 and 40 yields

where (a) follows from the Lipschitz continuity of f i , g i , h i , h j and inequality 40.

Due to the ρ F -weak convexity of F ( w , s i , s ′ ) , we have ( ¯ ρ -ρ F )-strong convexity of ( w , s i , s ′ ) ↦→ F ( w , s , s ′ ) + ¯ ρ 2 ∥ ( w t , s t , s ′ t ) -( w , s , s ′ ) ∥ 2 . Then it follows

Plugging inequality 42 back into 41, we obtain

where in inequality (a) we use ¯ ρ = ρ F + ρ g C f + 8 ρ g C f C h + C f C g L h and C 1 = max { 2 C f C g , 2 ρ g C f , 2 C f } , and inequality (b) uses Lemma 3.2.

With general error bounds

we have

Then it follows

where we define constant ∆ such that F 1 / ¯ ρ ( w 0 , s 0 , s 0 ) -E [ F 1 / ¯ ρ ( w T , s T , s T )] ≤ ∆ . With MSVR updates for v i,t and u i,t , following from Lemma C.5 and Lemma C.6, we have

Then we have

Setting

iterations, we have

D Proofs of Lemmas and Propositions

D.1 Additional Proposition

Proposition D.1. Consider a Lipschitz continuous function f : O → R where O ⊂ R d is an open set. Assume f to be non-increasing (resp. non-decreasing) with respect to each element in the input, then all subgradients of f are element-wise non-positive (resp. non-negative).

Proof of Proposition D.1. Let D be the subset of O where f is differentiable. By Theorem 9.60 in [24], a Lipschitz continuous function f : O → R , where O ⊂ R d is an open set, is differentiable almost everywhere, i.e., D is dense in O . Then by Theorem 9.61 in [24], the subdifferential of f at x is defined as

where con denotes the convex hull. If we assume that f is non-increasing with respect to each element in the input, then ∇ f ( x ) ≤ 0 (element-wise) for all differentiable points x ∈ D . It implies that the all vectors in { v |∃ x k → x with x k ∈ D, ∇ f ( x k ) → v } are element-wise non-positive. Therefore, all subgradients of f are element-wise non-positive. On the other hand, if we assume that f is non-decreasing, one may follow the same argument and conclude that all subgradients of f are element-wise non-negative.

For functions f : O → R m where O ⊂ R d is an open set, one may write f = ( f 1 , . . . , f m ) and apply the above proposition for each f k , k = 1 , . . . , m .

D.2 Proofs of Proposition 4.2 and Proposition 4.4

To prove Proposition 4.2 and Proposition 4.4, we first present the following proposition on the weak-convexity of composition functions.

Proposition D.2. Assume f : R d → R is ρ 1 -weakly-convex and C 1 -Lipschitz continuous, g : R ¯ d → R d is C 2 -Lipschitz continuous, and either of the followings holds:

f ( · ) is monotone and g ( · ) is L 2 -smooth;

f ( · ) is non-decreasing and g ( · ) is L 2 -weakly-convex,

then f ◦ g is ˜ ρ -weakly-convex with ˜ ρ = √ dL 2 C 1 + ρ 1 C 2 2 .

Proof of Proposition D.2. The weak convexity of f implies

where v ∈ ∂f ( g ( x )) . Moreover, due to the smoothness of g ( · ) (or weakly-convexity of g ( · ) , then only the second inequality holds), we have

where v ( e ) denotes a d -dimensional vector with value e on each dimensions. We first assume that f is non-increasing, then we may use the first inequality in (43) and the Lipschitz continuity of g to get

On the other hand, if we assume f is non-decreasing, the same result follows from the second inequality in (43). Thus f ◦ g is ˜ ρ -weakly-convex with ˜ ρ g = √ dL 2 C 1 + ρ 1 C 2 2 .

Proof of Proposition 4.2. Under Assumption 4.1, Proposition D.2 directly implies the ρ F -weakconvexity of F ( w ) with ρ F = √ d 1 ρ g C f + ρ f C 2 g .

Proof of Proposition 4.4. Under Assumption 4.3, we first apply Proposition D.2 to the composite function g i ( h i,j ( · )) and obtain its ρ ˜ g = √ d 2 L h C g + ρ g C 2 h -weak-convexity. To show it Lipschitz continuity, we use the Lipschitz continuity of g i and h i,j to obtain

Thus g i ( h i,j ( w )) is C ˜ g = C g C h -Lipschitz-continuous

Since we assume f i ( · ) is non-decreasing, ρ f -weakly-convex and C f -Lipschitz continuous, and g i ( h i,j ( · )) is ρ ˜ g -weakly-convex and C ˜ g -Lipschitz-continuous, we apply Proposition D.2 again to conclude that F ( · ) is ρ F = √ d 1 ρ ˜ g C f + ρ f C 2 ˜ g -weakly-convex.

D.3 Proof of Lemma 4.5

Proof of Lemma 4.5. With γ = n 1 -B 1 B 1 (1 -τ ) +(1 -τ ) , τ ≤ 1 2 , MSVR update gives recursive error bound [15]

Applying this inequality recursively, we obtain

where we use ∑ t j =0 (1 -B 1 τ 2 n 1 ) 2( t -j ) ≤ 2 n 1 B 1 τ .

It follows

Thus

Taking summation over i ∈ S , we obtain the desired result

D.4 Proof of Lemma C.6

Proof of Lemma C.6. With γ 3 = n + -B 1 B 1 (1 -τ 2 ) +(1 -τ 2 ) and τ 2 ≤ 1 2 , MSVR update gives the following recursive error bound [15]

It remains to bound E [ ∥ v i,t -v i,t +1 ∥ 2 ] and E [ ∥ v j,t -v j,t +1 ∥ 2 ] . We bound the former, and the latter's bound naturally follows. Consider the update of v i,t +1 and we have

where inequality (a) uses τ 1 ≤ 1 / 2 and γ 1 = n + -B 1 B 1 (1 -τ 1 ) + (1 -τ 1 ) ≤ 2 n + B 1 . Plugging the above inequality back into inequality 44 gives

Applying this inequality recursively, we obtain

where we use ∑ t j =0 (1 -B 1 τ 2 2 n + ) 2( t -j ) ≤ 2 n + B 1 τ 1 and denotes C 2 2 = 2 max { 256 C 2 g M 2 , 256 C 2 g C 2 h M 2 , 16 C 2 g M 2 } . Taking average over i ∈ S + gives the squarednorm error bound.

To derive the norm error bound, we derive

Taking average over i ∈ S + , we obtain the norm error bound

D.5 Proof of Lemma A.3

Proof of Lemma A.3. With γ 1 = n 1 n 2 -B 1 B 2 B 1 B 2 (1 -τ 1 ) +(1 -τ 1 ) and τ 1 ≤ 1 2 , MSVR update has the following recursive error bound [15][15]

Applying this inequality recursively, we obtain

where we use ∑ t j =0 (1 -B 1 B 2 τ 1 2 n 1 n 2 ) 2( t -j ) ≤ 2 n 1 n 2 B 1 B 2 τ 1 . Taking average over ( i, j ) ∈ S 1 × S 2 squared-norm error bound.

To derive the norm error bound, we derive

Thus

Taking average over ( i, j ) ∈ S 1 ×S 2 , we obtain the norm error bound

D.6 Proof of Lemma A.4

Proof of Lemma A.4. With γ 2 = n 1 -B 1 B 1 (1 -τ 2 ) +(1 -τ 2 ) and τ 2 ≤ 1 2 , MSVR update has the following recursive error bound [15]

2

where inequality (a) uses τ 1 ≤ 1 / 2 and γ 1 = n 1 n 2 -B 1 B 2 B 1 B 2 (1 -τ 1 ) +(1 -τ 1 ) ≤ 2 n 1 n 2 B 1 B 2 . Plugging the above inequality back into inequality 45 gives

Applying this inequality recursively, we obtain

where we use ∑ t j =0 (1 -B 1 τ 2 2 n 1 ) 2( t -j ) ≤ 2 n 1 B 1 τ 1 . Taking average over i ∈ S 1 gives the squared-norm error bound.

To derive the norm error bound, we derive

Taking squared root on both sides and taking average over i ∈ S 1 , we obtain the norm error bound

D.7 Proof of Lemma B.2

Proof of Lemma B.2. Define

Then we have

It follows

where we use

Then

where we use B 1 τ 4 n 1 ≤ 1 . Applying this inequality recursively, we obtain

To obtain the absolute bound, we derive

The desired result follows by taking squared root on both sides.

D.8 Proof of Lemma B.5

Proof of Lemma B.5. The proof of Lemma B.5 is the same as Lemma B.2.

D.9 Proof of Lemma B.6

It follows

where we use

Then

Applying this inequality recursively, we obtain

To obtain the absolute bound, we derive

The desired result follows by taking squared root on both sides.

E Group Distributionally Robust Optimization

NSWC FCCO finds an important application in group distributionally robust optimization (group DRO), particularly valuable in addressing distributional shift [25]. Consider N groups with different distributions. Each group k has an averaged loss L k ( w ) = 1 n k ∑ n k i =1 ℓ ( f w ( x k i ) , y k i ) , where w is the the model parameter and ( x k i , y k i ) is a data point. For robust optimization, we assign different weights to different groups and form the following robust loss minimization problem:

where Ω ⊂ ∆ and ∆ denotes a simplex. A common choice for Ω is Ω = { p ∈ ∆ , p i ≤ 1 /K } where K is an integer, resulting in the so-called CVaR losses, i.e., average of top-K group losses. Consequently, the above problem can be equivalently reformulated as [23]:

This formulation can be mapped into non-smooth weakly-convex FCCO when the loss function ℓ ( · , · ) is weakly convex in terms of w . In comparison to directly solving the min-max problem, solving the above FCCO problem avoids the need of dealing with the projection onto the constraint Ω and expensive sampling as in existing works [4].

F More Information for Experiments

F.1 Dataset Statistics

F.2 Illustration for Histopathology Dataset on MIL Task

Colon Ade.

Colon SCC

Bag

Bag

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the finite-sum coupled compositional optimization (FCCO) scenario, relaxing the requirement of Lipschitz gradient for the involved functions. Instead, they consider weakly convex functions with certain monotonicity conditions. The paper introduces new algorithms and provides oracle complexity guarantees for computing a point with an epsilon-gradient norm relative to the Moreau envelop. The authors also demonstrate the applicability of these methods to AUC maximization tasks.\n\nSTRENGTHS:\nIt is always beneficial and sometimes nontrivial to extend the algorithm implementation and analytic technique for gradient Lipschitz functions to that of more general weakly convex functions. The authors also demonstrate practical applicability of the proposed new algorithms.\n\nWEAKNESSES:\nSince I'm not acquainted with the FCCO setting, I will concentrate on the broader technical aspects instead.\n\n* L137: I don't get the idea of the \"monotonic property\" of a multivariate mapping f here. What is the \"input\" referring here? Let the function be f: R^n -> R^m. Is the input here referring to a vector in R^n? If so, how can we define the \"monotonicity\" with respect to such a multivariate function? \n\n* The whole development in this paper is built on the weak convexity of F proved in Proposition 4.2 and 4.4. However, the arguments there are not transparent to me. Let me take the proof of Proposition 4.2 for example.\n\n(a) L809 first inequality: Why do we have v'(g(y) - g(x)) \\geq v'( ... )? It seems that the authors assumed v <= 0 here as f is assumed non-increasing in L808 (though the term \"non-increasing\" is not clearly defined in my opinion). However, it is not clear to me why such a \"monotonicity\" of f will promise the nonnegativity of subgradients v. For a smooth function f, the claim (v <= 0) holds trivially. But for a general non-smooth weakly convex function, even with the \"monotonic assumption\", this is nontrivial and a formal proof is required. \n\n(b) L809 third inequality: should L_2C_1 be L_2C_1*sqrt(n), where n is the input dimension of f? Please check. If that is true, then the weak convexity parameter would be depend on the dimension n of the input to f.\n\nMinor: \n\n* L805: should v'(g(x) - g(y)) be v'(g(y) - g(x))?\n* L145: \\epsilon > 0 should be 1 > \\epsilon >= 0.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a class of finite-sum coupled compositional optimization (FCCO) problems and a class of tri-level finite-sum coupled compositional optimization (TCCO) problems. Under the setting of non-smooth weakly-convex FCCO, the paper establishes the complexity of a single-loop algorithm with the tool of Moreau envelop. The algorithm is then extended to solve the non-smooth weekly-convex TCCO. Numerical results on the two-way partial AUC maximization and multi-instance two-way partial AUC maximization are also reported.\n\nSTRENGTHS:\nConvergence analysis of an single-loop stochastic algorithm for solving non-smooth weakly-convex FCCO is a nice contribution of the paper.\n\nWEAKNESSES:\n1. There lacks motivation for considering non-smooth weakly-convex FCCO and TCCO. Some additional applications other than the two-way partial AUC maximization should be provided.  \n\n2. In the nonconvex setting, definitions need to be rigorously given/referred to. For example, $f$ at line 133 can be nonconvex, which definition of \"subgradient\" and \"subdifferential\" are the paper using? There some ways to define a \"variance-reduced estimator\", which definition is the paper using? \n\n3. The experiments just partially support the theoretical results of the paper. Numerical results on the comparison of the evolution of the objective function over time when using the proposed algorithm and other known algorithms for solving one same non-smooth weakly-convex FCCO problem are expected. It would illustrate the importance of the obtained complexity.\n\nQUESTIONS:\nConsider the TPAUC problem in Section 5.The corresponding NSWC FCCO problem of the regular setting and the corresponding NSWC TCCO problem for the MIL setting still have convex $f_i$. Could the complexity of the proposed algorithms be improved if the weakly-convex assumption of $f_i$ is replaced by convexity?\n\nLIMITATIONS:\nn.n.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper handles the problems of non-smooth weakly-convex compositional optimization. The first problem, referred to as FCCO (Finite-sum coupled compositional minimization), is given by\n\n$$ \\min_{w \\in \\mathbb{R}^d} F(w) \\triangleq \\frac{1}{n} \\sum_{i=1}^n f_i(\\mathbb{E}_{\\xi\\sim \\mathcal{D}_i}[g_i(w; \\xi)])$$\n\nThe second problem, referred to as TCCO (Tri-level coupled compositional minimization), is given by,\n\n$$ \\min_{w \\in \\mathbb{R}^d} F(w) \\triangleq \\frac{1}{n_1} \\sum_{i=1}^{n_1} f_i\\left(\\frac{1}{n_2}\\sum_{j=1}^{n_2} g_i( \\underset{\\xi \\sim \\mathcal{D}{i,j}}{\\mathbb{E}} [h_{i,j}(w; \\xi)])\\right)$$\n\n\nHere, the outer functions $f_i$ and $g_i$ are Lipschitz and weakly convex which has not been considered by previous works and significantly complicates the analysis. \n\nThe authors propose two algorithms, SONX and SONT for the two problems respectively. For the first problem, FCCO, $u_{i,t}$ are the estimates of $g_i(w_t)$, $\\forall i \\in \\{1,2,\\ldots, n\\}$, which are required for an unbiased estimate of the the complete function $F(w)$. In each step, a batch $B$ of coordinates from $\\{1,2,\\ldots, n\\}$ is sampled and only the estimates in this batch, $u_{i,t}, i\\in B$, are updated using a Variance-reduced estimator. Using the estimates, a single gradient step is computed for $w_t$. The estimator has been adapted from [1].\n\nAs the functions are weakly convex, the authors show convergence to an $\\epsilon$-stationary point of the Moreau envelope of $F$ at a rate of $\\mathcal{O}(\\epsilon^{-6})$.\n\nThe authors show that both problems can be mapped exactly to Two-way partial AUC (TPAUC) maximization of a parametrized network, even deep networks, with the TCCO problem mapping to a multi-instance learning version. TPAUC is the area under the ROC curve when False Positive rate $\\leq \\beta$ and True Positive Rate $\\geq \\alpha$. Over a set of medical datasets, the authors show that SONX and SONT out-perform baselines for TPAUC maximization as the baselines use poor approximations of TPAUC.\n\n\n\n**References**\n1. Jiang et al 2022. Multi-block-single-probe variance reduced estimator for coupled compositional optimization\n\nSTRENGTHS:\n- **Easy to Implement**: The algorithms are single-loop and use moderately sized batches which makes it easy to implement. Also, the algorithms can also be parallelized over the coordinates $i$.\n- **Better Rates**: In addition to being easy to implement, the algorithms achieve the best possible rates for this class of functions. Existing works obtain similar  or worse rates under an easier problem setting or with multiple loops or using large batches.\n- **Mapping to TPAUC**: The authors describe the TPAUC problem comprehensively and show the exact mapping with the two optimization problems.\n- **Detailed presentation**: The convergence analysis, the mapping to TPAUC and the related works are very detailed which helps in understanding the paper.\n\nWEAKNESSES:\n- **Experimental and Theoretical Baselines do not match**: For theory, the baselines for comparisons are algorithms for simpler problem settings, for instance with smoothness or convexity. For experiments, the baselines for TPAUC maximization are used which do not match the theoretical baselines. It is unclear, if some theoretical baseline can be applied to TPAUC and perform better. This should not be the case ideally but it has not been verified.\n\nQUESTIONS:\n- Is there a lower bound for this problem setting, which can be used to verify if the rates are optimal and cannot be improved?\n- Why is the dependence on $\\frac{n_1}{B_1}$ worse for SONT as compared to SONT?\n\nLIMITATIONS:\nDescribed in Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe purpose of this paper is to introduce a new approach to solving a specific type of optimization problem called non-smooth weakly-convex finite-sum coupled compositional optimization (NSWC FCCO) problems. The authors specifically focus on a variation of FCCO problems where the outer function is weakly convex and non-decreasing, and the inner function is weakly convex.\n\nTo address these problems, the authors propose two new algorithms. The first algorithm is designed for two-level NSWC FCCO problems and operates using a single-loop. The second algorithm is an extension of the first algorithm and is intended for tri-level NSWC FCCO problems. The authors provide a comprehensive analysis. They establish the complexity of the algorithms in terms of finding an $ε$-stationary point of the Moreau envelope of the objective function.\n\nSTRENGTHS:\nThe paper is of high quality as it provides a detailed and rigorous mathematical analysis of the proposed algorithms. The technique of using Moreau envelope is intuitive and interesting. I have not checked the proof line by line but it looks good.  The authors also conduct extensive experiments to validate their theoretical findings. They compare the performance of their algorithms with other competitive methods across multiple datasets, demonstrating the effectiveness of their approach.\n\nWEAKNESSES:\nThe authors have presented new convergence analysis for FCCO and TCCO, which I acknowledge. However, it seems that other multistage algorithms have already achieved the optimal rate. Consequently, I would lower my evaluation score for the novelty aspect.\n\nQUESTIONS:\nI acknowledge and value the authors' dedication in exploring the intricacies of the technical aspects. Nevertheless, I fail to find the application of maximizing two-way partial AUC particularly inspiring. Could you please demonstrate any other compelling applications?\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

comments: SUMMARY:
The manuscript studies a class of non-smooth non-convex compositional optimization problems in which the objective function is given in a form of finite-sum composition where the functions are assumed to be weakly convex. The authors present stochastic approximation algorithms to solve this class of problems and establish their sample complexity bounds.

STRENGTHS:
The authors provide the sample complexity bound of $O(1/\epsilon^6)$ for a single-loop algorithm applied to non-smooth weakly convex compositional problems.

WEAKNESSES:
Given the existing works in the literature and assumptions made in the manuscript, the presented theoretical results are incremental.

QUESTIONS:
1. In Table 1, the authors compare their results with the existing ones for smooth problems. However, dependence on $n$ is hidden and the reader cannot compare this dependency. Moreover, they need to add more existing results (mentioned on page 3) to the table with their assumptions so that there is a clear picture for comparison.

2. The choice of bath sizes appear in the complexity bounds, however, there is a very limited discussion on its role.

LIMITATIONS:
.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-27