Title: Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generalization

algorithm stability

multi-objective optimization

gradient conflict
Abstract: Multi-objective learning (MOL) often arises in emerging machine learning problems when multiple learning criteria or tasks need to be addressed.  Recent works have developed various _dynamic weighting_ algorithms for MOL, including MGDA and its variants, whose central idea is to find an update direction that _avoids conflicts_ among objectives. Albeit its appealing intuition, empirical studies show that dynamic weighting methods may not always outperform static alternatives. To bridge this gap between theory and practice, we focus on a new variant of stochastic MGDA - the Multi-objective gradient with Double sampling (MoDo) algorithm and study its generalization performance and the interplay with optimization through the lens of algorithm stability. We find that the rationale behind MGDA -- updating along conflict-avoidant direction - may \emph{impede} dynamic weighting algorithms from achieving the optimal ${\cal O}(1/\sqrt{n})$ population risk, where $n$ is the number of training samples. We further highlight the variability of dynamic weights and their impact on the three-way trade-off among optimization, generalization, and conflict avoidance that is unique in MOL. Code is available at https://github.com/heshandevaka/Trade-Off-MOL.

Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance

Abstract

Multi-objective learning (MOL) often arises in emerging machine learning problems when multiple learning criteria or tasks need to be addressed. Recent works have developed various dynamic weighting algorithms for MOL, including MGDA and its variants, whose central idea is to find an update direction that avoids conflicts among objectives. Albeit its appealing intuition, empirical studies show that dynamic weighting methods may not always outperform static alternatives. To bridge this gap between theory and practice, we focus on a new variant of stochastic MGDA - the M ultio bjective gradient with Do uble sampling ( MoDo ) algorithm and study its generalization performance and the interplay with optimization through the lens of algorithm stability. We find that the rationale behind MGDA-updating along conflict-avoidant direction - may impede dynamic weighting algorithms from achieving the optimal O (1 / √ n ) population risk, where n is the number of training samples. We further highlight the variability of dynamic weights and their impact on the three-way trade-off among optimization, generalization, and conflict avoidance that is unique in MOL. Code is available at https://github.com/heshandevaka/Trade-Off-MOL .

1 Introduction

Multi-objective learning (MOL) emerges frequently in recent machine learning problems such as learning under fairness and safety constraints [49]; learning across multiple tasks, including multi-task learning [39] and meta-learning [47]; and, learning across multiple agents that may not share a global utility including federated learning [40] and multi-agent reinforcement learning [33].

This work considers solving the empirical version of MOL defined on the training dataset as S = { z 1 , . . . , z n } . The performance of a model x ∈ R d on a datum z for the m -th objective is denoted as f z,m : R d ↦→ R , and its performance on the entire training dataset S is measured by the m -th empirical objective f S,m ( x ) for m ∈ [ M ] . MOL optimizes the vector-valued objective, given by

(a) Objective 1

(b) Objective 2

(c) MGDA

(d) Static

(e) MoDo (dynamic)

10

10

10

10

5

0

0

0

10

5

13

15

10

10

10

10

10

18

10

X1

0

5

10

10

-5

X1

0

5

01

10

X1

0

10

01

X1

0

5

01

-10

-5

X1

0

5

10

One natural method for solving (1.1) is to optimize the (weighted) average of the multiple objectives, also known as static or unitary weighting [18, 45]. However, this method may face challenges due to potential conflicts among multiple objectives during the optimization process; e.g., conflicting gradient directions ⟨∇ f S,m ( x ) , ∇ f S,m ′ ( x ) ⟩ < 0 . A popular alternative is thus to dynamically weight gradients from different objectives to avoid conflicts and obtain a direction d ( x ) that optimizes all objective functions jointly that we call a conflict-avoidant (CA) direction. Algorithms in this category include the multi-gradient descent algorithm (MGDA) [9], its stochastic variants [30, 10, 52]. While the idea of finding CA direction in dynamic weighting-based approaches is very appealing, recent empirical studies reveal that dynamic weighting methods may not outperform static weighting in some MOL benchmarks [18, 45], especially when it involves stochastic updates and deep models. Specifically, observed by [18], the vanilla stochastic MGDA can be under-optimized, leading to larger training errors than static weighting. The reason behind this training performance degradation has been studied in [52, 10], which suggest the vanilla stochastic MGDA has biased updates, and propose momentum-based methods to address this issue. Nevertheless, in [45], it is demonstrated that the training errors of MGDA and static weighting are similar, while their main difference lies in the generalization performance. Unfortunately, the reason behind this testing performance degradation is not fully understood and remains an open question.

To gain a deeper understanding of the dynamic weighting-based algorithms, a natural question is

Q1: What are the major sources of errors in dynamic weighting-based MOL methods?

To answer this question theoretically, we first introduce a proper measure of testing performance in MOL - the Pareto stationary measure in terms of the population objectives, which will immediately imply stronger measures such as Pareto optimality under strongly convex objectives. We then decompose this measure into generalization error and optimization error and further introduce a new metric on the distance to CA directions that is unique to MOL; see Sections 2.1 and 2.2.

To characterize the performance of MOL methods in a unified manner, we introduce a generic dynamic weighting-based MOL method that we term stochastic Multi-Objective gradient with DOuble sampling algorithm ( MoDo ), which uses a step size γ to control the change of dynamic weights. Roughly speaking, by controlling γ , MoDo approximates MGDA (large γ ) and static weighting algorithm ( γ = 0 ) as two special cases; see Section 2.3. We first analyze the generalization error of the model learned by MoDo through the lens of algorithmic stability [3, 14, 24] in the framework of statistical learning theory. To our best knowledge, this is the first-ever-known stability analysis for MOL algorithms. Here the key contributions lie in defining a new notion of stability MOLuniform stability and then establishing a tight upper bound (matching lower bound) on the MOL uniform stability for MoDo algorithm that involves two coupled sequences; see Section 3.1. We then analyze the optimization error of MoDo and its distance to CA directions, where the key contributions lie in relaxing the bounded function value/gradient assumptions and significantly improving the convergence rate of state-of-the-art dynamic weighting-based MOL methods [10]; see Section 3.2.

Different from the stability analysis for single-objective learning [14], the techniques used in our generalization and optimization analysis allow to remove conflicting assumptions and use larger step sizes to ensure both small generalization and optimization errors, which are of independent interest.

Given the holistic analysis of dynamic weighting methods provided in Q1 , a follow-up question is

Q2: What may cause the empirical performance degradation of dynamic weighting methods?

Region

Opt.

Gen.

Conflict

Step sizes

α

&

γ

Iterations

T

&data size

n

I

γ

=

ω

(

α

)

T

=

o

(

n

)

II

γ

=

ω

(

α

)

T

=

ω

(

n

)

III

γ

=

o

(

T

-

1

)

T

=

ω

(

n

)

IV

γ

= Θ(

α

1

2

) = Θ(

T

-

1

4

)

T

= Θ(

n

4

5

)

Figure 2: An illustration of three-way trade-off among optimization, generalization, and conflict avoidance in the strongly convex case; α is the step size for x , γ is the step size for weights λ , where o ( · ) denotes a strictly slower growth rate, ω ( · ) denotes a strictly faster growth rate, and Θ( · ) denotes the same growth rate. Arrows ↓ and ↑ respectively represent diminishing in an optimal rate and growing in a fast rate w.r.t. n , while ↘ represents diminishing w.r.t. n , but not in an optimal rate. Visualizing MOL solution concepts. To reason the root cause for this, we first compare different MOL algorithms in a toy example shown in Figure 1. We find MGDA can navigate along CA directions and converge to the empirical Pareto front under all initializations, while static weighting gets stuck in some initializations; at the same time, the empirical Pareto solution obtained by MGDA may incur a larger population risk than the suboptimal empirical solution obtained by the static weighting method; finally, if the step size γ of dynamic weights is carefully tuned, MoDo can converge along CA directions to the empirical Pareto optimal solution that also generalizes well.

Aligned with this toy example, our theoretical results suggest a novel three-way trade-off in the performance of dynamic weighting-based MOL algorithm; see Section 3.3. Specifically, it suggests that the step size for dynamic weighting γ plays a central role in the trade-off among convergence to CA direction, convergence to empirical Pareto stationarity, and generalization error; see Figure 2. In this sense, MGDA has an edge in convergence to the CA direction but it could sacrifice generalization; the static weighting method cannot converge to the CA direction but guarantees convergence to the empirical Pareto solutions and their generalization. Our analysis also suggests that MoDo achieves a small population risk under a proper combination of step sizes and the number of iterations.

2 Problem Formulation and Target of Analysis

In this section, we first introduce the problem formulation of MOL, the target of analysis, the metric to measure its generalization, and then present the MGDA algorithm and its stochastic variant.

2.1 Preliminaries of MOL

Denote the vector-valued objective function on datum z as F z ( x ) = [ f z, 1 ( x ) , . . . , f z,M ( x )] . The training and testing performance of x can then be measured by the empirical objective F S ( x ) and the population objective F ( x ) which are, respectively, defined as F S ( x ) := 1 n ∑ n i =1 F z i ( x ) and F ( x ) := E z ∼D [ F z ( x )] . Their corresponding gradients are denoted as ∇ F S ( x ) and ∇ F ( x ) ∈ R d × M .

Analogous to the stationary solution and optimal solution in single-objective learning, we define the Pareto stationary point and Pareto optimal solution for MOL problem min x ∈ R d F ( x ) as follows.

̸

Definition 1 (Pareto stationary and Pareto optimal) . If there exists a convex combination of the gradient vectors that equals to zero, i.e., there exists λ ∈ ∆ M such that ∇ F ( x ) λ = 0 , then x ∈ R d is Pareto stationary. If there is no x ∈ R d and x = x ∗ such that, for all m ∈ [ M ] f m ( x ) ≤ f m ( x ∗ ) , with f m ′ ( x ) < f m ′ ( x ∗ ) for at least one m ′ ∈ [ M ] , then x ∗ is Pareto optimal. If there is no x ∈ R d such that for all m ∈ [ M ] , f m ( x ) < f m ( x ∗ ) , then x ∗ is weakly Pareto optimal.

By definition, at a Pareto stationary solution, there is no common descent direction for all objectives. A necessary and sufficient condition for x being Pareto stationary for smooth objectives is that min λ ∈ ∆ M ∥∇ F ( x ) λ ∥ = 0 . Therefore, min λ ∈ ∆ M ∥∇ F ( x ) λ ∥ can be used as a measure of Pareto stationarity (PS) [9, 11, 42, 30, 10]. We will refer to the aforementioned quantity as the PS population risk henceforth and its empirical version as PS empirical risk or PS optimization error . We next introduce the target of our analysis based on the above definitions.

2.2 Target of analysis and error decomposition

In existing generalization analysis for MOL, measures based on function values have been used to derive generalization guarantees in terms of Pareto optimality [7, 41]. However, for general

nonconvex smooth MOL problems, it can only be guaranteed for an algorithm to converge to Pareto stationarity of the empirical objective, i.e., a small min λ ∈ ∆ M ∥∇ F S ( x ) λ ∥ [9, 11, 30, 10]. Thus, it is not reasonable to measure population risk in terms of Pareto optimality in this case. Furthermore, when all the objectives are convex or strongly convex, Pareto stationarity is a sufficient condition for weak Pareto optimality or Pareto optimality, respectively, as stated in Proposition 1.

Proposition 1 ([42, Lemma 2.2]) . If f m ( x ) are convex or strongly-convex for all m ∈ [ M ] , and x ∈ R d is a Pareto stationary point of F ( x ) , then x is weakly Pareto optimal or Pareto optimal.

Next, we proceed to decompose the PS population risk.

Error Decomposition. Given a model x , the PS population risk can be decomposed into

where the optimization error quantifies the training performance, i.e., how well does model x perform on the training data; and the generalization error (gap) quantifies the difference between the testing performance on new data sampled from D and the training performance, i.e., how well the model x performs on unseen testing data compared to the training data.

Let A : Z n ↦→ R d denote a randomized MOL algorithm. Given training data S , we are interested in the expected performance of the output model x = A ( S ) , which is measured by E A,S [ R pop ( A ( S ))] . From (2.1) and linearity of expectation, it holds that

Distance to CA direction. As demonstrated in Figure 1, the key merit of dynamic weighting over static weighting algorithms lies in its ability to navigate through conflicting gradients. Consider an update direction d = -∇ F S ( x ) λ , where λ is the dynamic weights from a simplex λ ∈ ∆ M := { λ ∈ R M | 1 ⊤ λ = 1 , λ ≥ 0 } . To obtain such a steepest CA direction in unconstrained learning that maximizes the minimum descent of all objectives, we can solve the following problem [11]

Defining d λ ( x ) = -∇ F S ( x ) λ given x ∈ R d and λ ∈ ∆ M , we measure the distance to d ( x ) via [10]

With the above definitions of measures that quantify the performance of algorithms in different aspects, we then introduce a stochastic gradient algorithm for MOL that is analyzed in this work.

2.3 A stochastic algorithm for MOL

MGDA finds λ ∗ ( x ) in (2.3b) using the full-batch gradient ∇ F S ( x ) , and then constructs d ( x ) = -∇ F S ( x ) λ ∗ ( x ) , a CAdirection for all empirical objectives f S,m ( x ) . However, in practical statistical learning settings, the full-batch gradient ∇ F S ( x ) may be costly to obtain, and thus one may resort to a stochastic estimate of ∇ F S ( x ) instead. The direct stochastic counterpart of MGDA, referred to as the stochastic multi-gradient algorithm in [30], replaces the full-batch gradients ∇ f S,m ( x ) in (2.3b) with their

Algorithm 1 Stochastic MGDA - MoDo algorithm

1: input Training data S , initial model x 0 , weighting coefficient λ 0 , and their learning rates { α t } T t =0 , { γ t } T t =0 .

2: for t = 0 , . . . , T -1 do

3: for objective m = 1 , . . . , M do

4: Independent gradients ∇ f m,z t,s ( x t ) , s ∈ [3]

5: end for

6: Compute dynamic weight λ t +1 following (2.5a)

7: Update model parameter x t +1 following (2.5b)

8: end for

9: output x T

stochastic approximations ∇ f z,m ( x ) for z ∈ S , which, however, introduces a biased stochastic estimate of λ ∗ t +1 , thus a biased CA direction; see [10, Section 2.3].

To provide a tight analysis, we introduce a simple yet theoretically grounded stochastic variant of MGDA - stochastic Multi-Objective gradient with DOuble sampling algorithm (MoDo). MoDo obtains an unbiased stochastic estimate of the gradient of problem (2.3b) through double (independent) sampling and iteratively updates λ , because E z t, 1 ,z t, 2 [ ∇ F z t, 1 ( x t ) ⊤ ∇ F z t, 2 ( x t ) λ t ] = ∇ F S ( x t ) ⊤ ∇ F S ( x t ) λ t . At each iteration t , denote z t,s as an independent sample from S with s ∈ [3] , and ∇ F z t,s ( x t ) as a stochastic estimate of ∇ F S ( x t ) . MoDo updates x t and λ t as

where α t , γ t are step sizes, and Π ∆ M ( · ) denotes Euclidean projection to the simplex ∆ M . We have summarized the MoDo algorithm in Algorithm 1 and will focus on MoDo in the subsequent analysis.

3 Optimization, Generalization and Three-Way Trade-Off

This section presents the theoretical analysis of the PS population risk associated with the MoDo algorithm, where the analysis of generalization error is in Section 3.1 and that of optimization error is in Section 3.2. A summary of our main results is given in Table 1.

3.1 Multi-objective generalization and uniform stability

We first bound the expected PS generalization error by the generalization in gradients in Proposition 2, then introduce the MOL uniform stability and establish its connection to the generalization in gradients. Finally, we bound the MOL uniform stability.

Proposition 2. With ∥ · ∥ F denoting the Frobenious norm, R gen ( A ( S )) in (2.2) can be bounded by

With Proposition 2, next we introduce the concept of MOL uniform stability tailored for MOL problems and show that PS generalization error in MOL can be bounded by the MOL uniform stability. Then we analyze their bound in general nonconvex case and strongly convex case, respectively.

Definition 2 (MOL uniform stability) . A randomized algorithm A : Z n ↦→ R d , is MOL-uniformly stable with ϵ F if for all neighboring datasets S, S ′ that differ in at most one sample, we have

Next we show the relation between the upper bound of PS generalization error in (3.1) and MOL uniform stability in Proposition 3.

Proposition 3 (MOL uniform stability and generalization) . Assume for any z , the function F z ( x ) is differentiable. If a randomized algorithm A : Z n ↦→ R d is MOL-uniformly stable with ϵ F , then

where the variance is defined as V z ∼D ( ∇ F z ( A ( S ))) = E z ∼D [ ∥∇ F z ( A ( S )) -E z ∼D [ ∇ F z ( A ( S ))] ∥ 2 F ] .

Proposition 3 establishes a connection between the upper bound of the PS generalization error and the MOL uniform stability, where the former can be bounded above by the latter plus the variance of the stochastic gradient over the population data distribution. It is worth noting that the standard arguments of bounding the generalization error measured in function values by the uniform stability measured in function values [14, Theorem 2.2] is not applicable here as the summation and norm operators are not exchangeable. More explanations are given in the proof in Appendix B.1.

Theorem 1 (PS generalization error of MoDo in nonconvex case) . If sup z E A [ ∥∇ F z ( A ( S )) ∥ 2 F ] ≤ G 2 for any S , then the MOL uniform stability, i.e., ϵ 2 F in Definition 2 is bounded by ϵ 2 F ≤ 4 G 2 T / n . And the PS generalization error E A,S [ R gen ( A ( S ))] = O ( T 1 2 n -1 2 ) .

Compared to the function value uniform stability upper bound in [14, Theorem 3.12] for nonconvex single-objective learning, Theorem 1 does not require a step size decay α t = O (1 /t ) , thus can enjoy at least a polynomial convergence rate of optimization errors w.r.t. T . Combining Theorem 1 with Proposition 3, to ensure the generalization error is diminishing with n , one needs to choose T = o ( n ) , which lies in the 'early stopping' regime and results in potentially large optimization error. We then provide a tighter bound in the strongly convex case that allows a larger choice of T . Below we list the standard assumptions used to derive the introduced MOL stability.

Assumption 1 (Lipschitz continuity of ∇ F z ( x ) ) . For all m ∈ [ M ] , ∇ f z,m ( x ) is ℓ f, 1 -Lipschitz continuous for all z . And ∇ F z ( x ) is ℓ F, 1 -Lipschitz continuous in Frobenius norm for all z .

Assumption 2. For all m ∈ [ M ] , z ∈ Z , f z,m ( x ) is µ -strongly convex w.r.t. x , with µ > 0 .

Note that in the strongly convex case, the gradient norm ∥∇ F z ( x ) ∥ F can be unbounded in R d . Therefore, one cannot assume Lipschitz continuity of f z,m ( x ) w.r.t. x ∈ R d . We address this challenge by showing that { x t } generated by the MoDo algorithm is bounded as stated in Lemma 1. Notably, combining with Assumption 1, we can derive that the gradient norm ∥∇ F z ( x t ) ∥ F is also bounded, which serves as a stepping stone to derive the MOL stability bound.

Lemma 1 (Boundedness of x t for strongly convex and smooth objectives) . Suppose Assumptions 1, 2 hold. For { x t } , t ∈ [ T ] generated by MoDo algorithm or other dynamic weighting algorithm with weight λ ∈ ∆ M , step size α t = α , and 0 ≤ α ≤ ℓ -1 f, 1 , there exists a finite positive constant c x such that ∥ x t ∥ ≤ c x . And there exists finite positive constants ℓ f , ℓ F = √ Mℓ f , such that for all λ ∈ ∆ M , we have ∥∇ F ( x t ) λ ∥ ≤ ℓ f , and ∥∇ F ( x t ) ∥ F ≤ ℓ F .

With Lemma 1, the stability bound and PS generalization is provided below.

Theorem 2 (PS generalization error of MoDo in strongly convex case) . Suppose Assumptions 1, 2 hold. Let A be the MoDo algorithm (Algorithm 1). For the MOL uniform stability ϵ F of algorithm A in Definition 2, if the step sizes satisfy 0 < α t ≤ α ≤ 1 / (2 ℓ f, 1 ) , and 0 < γ t ≤ γ ≤ min { µ 2 120 ℓ 2 f ℓ g, 1 , 1 8(3 ℓ 2 f +2 ℓ g, 1 ) } /T , then it holds that

And there exist functions F z ( x ) that satisfy Assumptions 1, 2, neighboring datasets S , S ′ that differ in at most one sample, and MoDo algorithm with step sizes 0 < α t ≤ α ≤ 1 / (2 ℓ f, 1 ) , and 0 < γ t ≤ γ ≤ min { µ 2 120 ℓ 2 f ℓ g, 1 , 1 8(3 ℓ 2 f +2 ℓ g, 1 ) } /T such that

Theorem 2 provides both upper and lower bounds for the MOL uniform stability. In this case, we choose α = Θ( T -1 2 ) , γ = o ( T -1 ) , and T = Θ( n 2 ) to minimize the PS population risk upper bound, as detailed in Section 3.3. With this choice, the MOL uniform stability upper bound matches the lower bound in an order of n -2 , suggesting that our bound is tight. The generalization error bound in (3.4) is a direct implication from the MOL uniform stability bound in (3.4), Propositions 2, and 3.

It states that the PS generalization error of MoDo is O ( n -1 2 ) , which matches the generalization error of static weighting up to a constant coefficient [22]. Our result also indicates that when all the objectives are strongly convex, choosing small step sizes α and γ can benefit the generalization error.

3.2 Multi-objective optimization error

In this section, we bound the multi-objective PS optimization error min λ ∈ ∆ M ∥∇ F S ( x ) λ ∥ [9, 30, 10]. As discussed in Section 2.2, this measure being zero implies the model x achieves a Pareto stationarity for the empirical problem.

Below we list an additional standard assumption used to derive the optimization error.

Assumption 3 (Lipschitz continuity of F z ( x ) ) . For all m ∈ [ M ] , f z,m ( x ) are ℓ f -Lipschitz continuous for all z . Then F z ( x ) are ℓ F -Lipschitz continuous in Frobenius norm for all z with ℓ F = √ Mℓ f .

Lemma2 (Distance to CA direction) . Suppose either: 1) Assumptions 1, 3 hold; or 2) Assumptions 1, 2 hold, with ℓ f and ℓ F defined in Lemma 1. For { x t } , { λ t } generated by MoDo, it holds that

Lemma 2 analyzes convergence to the CA direction using the measure introduced in Section 2.2. By, e.g., choosing α = Θ( T -3 4 ) , and γ = Θ( T -1 4 ) , the RHS of (3.6) converges in a rate of O ( T -1 4 ) .

Theorem 3 (PS optimization error of MoDo) . Suppose either: 1) Assumptions 1, 3 hold; or, 2) Assumptions 1, 2 hold, with ℓ f defined in Lemma 1. Define c F such that E A [ F S ( x 0 ) λ 0 ] -min x ∈ R d E A [ F S ( x ) λ 0 ] ≤ c F . Considering { x t } generated by MoDo (Algorithm 1), with α t = α ≤ 1 / (2 ℓ f, 1 ) , γ t = γ , then under either condition 1) or 2), it holds that

The choice of step sizes α = Θ( T -3 4 ) , and γ = Θ( T -1 4 ) to ensure convergence to CA direction is suboptimal for the convergence to Pareto stationarity (see Theorem 3), exhibiting a trade-off between convergence to the CA direction and convergence to Pareto stationarity; see discussion in Section 3.3.

3.3 Optimization, generalization and conflict avoidance trade-off

Combining the results in Sections 3.1 and 3.2, we are ready to analyze and summarize the three-way trade-off of MoDo in MOL. With A t ( S ) = x t denoting the output of algorithm A at the t -th iteration, we can decompose the PS population risk R pop ( A t ( S )) as (cf. (2.1) and (3.1))

The general nonconvex case. Suppose Assumptions 1, 3 hold. By the generalization error in Theorem 1, and the optimization error bound in Theorem 3, the PS population risk of the output of MoDo can be bounded by

Discussion of trade-off. Choosing step sizes α = Θ( T -1 2 ) , γ = Θ( T -1 2 ) , and number of steps T = Θ( n 2 3 ) , then the expected PS population risk is O ( n -1 6 ) , which matches the PS population risk upper bound of a general nonconvex single objective in [22]. A clear trade-off in this case is between the optimization error and generalization error, controlled by T . Indeed, increasing T leads to smaller optimization errors but larger generalization errors, and vice versa. To satisfy convergence to CA

direction, it requires γ = ω ( α ) based on Lemma 2, and the optimization error in turn becomes worse, so does the PS population risk. Specifically, choosing α = Θ( T -1 2 ) , γ = Θ( T -1 4 ) , and T = Θ( n 4 5 ) leads to the expected PS population risk in O ( n -1 10 ) , and the distance to CA direction in O ( n -1 10 ) . This shows another trade-off between conflict avoidance and optimization error.

The strongly convex case. Suppose Assumptions 1, 2 hold. By the generalization error and the optimization error given in Theorems 2 and 3, MoDo's PS population risk can be bounded by

Discussion of trade-off. Choosing step sizes α = Θ( T -1 2 ) , γ = o ( T -1 ) , and number of steps T = Θ( n 2 ) , we have the expected PS population risk in gradients is O ( n -1 2 ) . However, choosing γ = o ( T -1 ) leads to large distance to the CA direction according to Lemma 2 because the term 4 γT in (3.6) increases with T . To ensure convergence to the CA direction, it requires γ = ω ( T -1 ) , under which the tighter bound in Theorem 2 does not hold but the bound in Theorem 1 still holds. In this case, the PS population risk under proper choices of α, γ, T is O ( n -1 6 ) as discussed in the previous paragraph. Therefore, to avoid conflict of gradients, one needs to sacrifice the sample complexity of PS population risk, demonstrating a trade-off between conflict avoidance and PS population risk.

4 Related Works and Our Technical Contributions

Multi-task learning (MTL). MTL, as one application of MOL, leverages shared information among different tasks to train a model that can perform multiple tasks. MTL has been widely applied to natural language processing, computer vision, and robotics [15, 38, 50, 43]. From the optimization perspective, a simple method for MTL is to take the weighted average of the per-task losses as the objective. However, as studied in [16], the static weighting method is not able to find all the Pareto optimal models in general. Alternatively, the weights can be updated dynamically during optimization, and the weights for different tasks can be chosen based on different criteria such as uncertainty [17], gradient norms [6], or task difficulty [13]. These methods are often heuristic and designed for specific applications. Another line of work tackles MTL through MOL [39, 48, 29, 12]. A foundational algorithm in this regard is MGDA [9], which takes dynamic weighting of gradients to obtain a CA direction for all objectives. Stochastic versions of MGDA have been proposed in [30, 52, 10]. Besides finding one single model, algorithms for finding a set of Pareto optimal models rather than one have been proposed in [28, 35, 31, 20, 46, 51, 32, 27, 34].

Theory of MOL. Optimization convergence analysis for the deterministic MGDA algorithm has been provided in [11]. Later on, stochastic variants of MGDA were introduced [30, 52, 10]. However, the vanilla stochastic MGDA introduces biased estimates of the dynamic weight, resulting in biased estimates of the CA directions during optimization. To address this, Liu et al. [30] proposed to increase the batch size during optimization, Zhou et al. [52] and Fernando et al. [10] proposed to use momentum-based bias reduction techniques. Compared to these works on the optimization analysis, we have improved the assumptions and/or the final convergence rate of the PS optimization error. A detailed comparison is summarized in Table 2. While the community has a rich history of investigating the optimization of MOL algorithms, their generalization guarantee remains unexplored until recently. In [7], a min-max formulation to solve the MOL problem is analyzed, where the weights are chosen based on the maximum function values, rather than the CA direction. More recently, [41] provides generalization guarantees for MOL for a more general class of weighting. These two works analyze generalization based on the Rademacher complexity of the hypothesis class, with generalization bound independent of the training process. Different from these works, we use algorithm stability to derive the first algorithm-dependent generalization error bounds, highlighting the effect of the training dynamics. In contrast to previous MOL theoretical works that focus solely on either optimization [10, 52] or generalization [7, 41], we propose a holistic framework to analyze three types of errors, namely, optimization, generalization, and CA distance in MOL with an instantiation of the MoDo algorithm. This allows us to study the impact of the hyperparameters on the theoretical testing performance, and their optimal values to achieve the best trade-off among the errors. Our theory and algorithm design can also be applied to algorithms such as PCGrad [48], CAGrad [29], and GradNorm [6]. Specifically, for example, the implementation of CAGrad takes iterative updates of the dynamic weight using a single stochastic estimate, resulting in a biased estimate of the update

direction, thus no guarantee of convergence for the stochastic algorithm. This issue can be addressed by the double sampling technique introduced in this paper. In addition, our analysis techniques have been applied to improve the analysis of convergence rates for other algorithms [44, 5].

Algorithm stability and generalization. Stability analysis dates back to the work [8] in 1970s. Uniform stability and its relationship with generalization were studied in [3] for the exact minimizer of the ERM problem with strongly convex objectives. The work [14] pioneered the stability analysis for stochastic gradient descent (SGD) algorithms with convex and smooth objectives. The results were extended and refined in [19] with data-dependent bounds, in [4, 37, 23] for non-convex objectives, and in [1, 24] for SGD with non-smooth and convex losses. However, all these studies mainly focus on single-objective learning problems. To our best knowledge, there is no existing work on the stability and generalization analysis for multi-objective learning problems and our results on its stability and generalization are the first-ever-known ones.

Challenges and contributions. Our contributions are highly non-trivial as summarized below.

The definition of PS testing risk in gradient (2.1) is unique in MOL, and overcomes the unnecessarily small step sizes usually brought by the classical function value-based risk analysis. Specifically, prior stability analysis in function values for single objective learning [14] requires 1 /t step size decay in the nonconvex case, otherwise, the generalization error bound will depend exponentially on the number of iterations. However such step sizes lead to very slow convergence of the optimization error. This is addressed by the definitions of gradient-based measures and sampling-determined MOL algorithms, which yield stability bounds in O ( T/n ) without any step size decay. See Theorem 1.

The stability of the dynamic weighting algorithm in the strongly convex (SC) case is non-trivial compared to single objective learning [14] because it involves two coupled sequences during the update. As a result, the classical contraction property for the update function of the model parameters that are often used to derive stability does not hold. This is addressed by controlling the change of λ t by the step size γ , and using mathematical induction to derive a tighter bound. See Appendix B.4.

In the SC case with an unbounded domain, the function is not Lipschitz or the gradients are unbounded, which violates the commonly used bounded gradient assumption for proving the stability and optimization error. We relax this assumption by proving that the iterates generated by dynamic weighting algorithms in the SC case are bounded on the optimization trajectory in Lemma 1.

5 Experiments

In this section, we conduct experiments to further demonstrate the three-way trade-off among the optimization, generalization, and conflict avoidance of the MoDo algorithm. An average of 10 random seeds with 0.5 standard deviation is reported if not otherwise specified.

5.1 Synthetic experiments

Our theory in the SC case is first verified through a synthetic experiment; see the details in Appendix D.1. Figure 3a shows the PS optimization error and PS population risk, as well as the distance to CA direction, decreases as T increases, which corroborates Lemma 2, and Theorem 3. In addition, the generalization error, in this case, does not vary much with T , verifying Theorems 2. In Figure 3b, the optimization error first decreases and then increases as α increases, which is consistent with Theorem 3. Notably, we observe a threshold for α below which the distance to the CA direction converges even when the optimization error does not converge, while beyond which the distance to the CA direction becomes larger, verifying Lemma 2. Additionally, Figure 3c demonstrates that

10

1

10

2

10

3

10

4

T

(log scale)

0

0.5

1

1.5

2

2.5

3

3.5

Errors

Ropt

Rgen

Rpop

E

ca

(a) Number of iterations

T

.

10

-3

10

-2

10

-1

,

(log scale)

0

0.5

1

1.5

2

2.5

3

3.5

Errors

(b) Different step size

α

.

10

-3

10

-2

10

-1

.

(log scale)

0

0.5

1

1.5

Errors

(c) Different step size

γ

.

10

2

10

3

10

4

10

5

T

(log scale)

0

1

2

3

4

5

6

7

Errors

#

10

-3

Ropt

Rgen

Rpop

E

ca

(a) Number of iterations

T

10

-2

10

-1

10

0

,

(log scale)

0

1

2

3

4

5

Errors

#

10

-3

(b) Different step size

α

10

-2

10

-1

10

0

.

(log scale)

0

1

2

3

4

5

Errors

#

10

-3

(c) Different step size

γ

Figure 4: Optimization, generalization, and CA direction errors of MoDo for MNIST image classification under different T , α , and γ . The default parameters are T = 1000 , α = 0 . 1 , and γ = 0 . 01 .

increasing γ enlarges the PS optimization error, PS generalization error, and thus the PS population risk, but decreases the distance to CA direction, which supports Lemma 2.

5.2 Image classification experiments

We further verify our theory in the NC case on MNIST image classification [21] using a multi-layer perceptron and three objectives: cross-entropy, mean squared error (MSE), and Huber loss. Following Section 2.2, we evaluate the performance in terms of R pop ( x ) , R opt ( x ) , R gen ( x ) , and E ca ( x, λ ) . The exact PS population risk R pop ( x ) is not accessible without the true data distribution. To estimate the PS population risk, we evaluate min λ ∈ ∆ M ∥∇ F S te ( x ) λ ∥ on the testing data set S te that is independent of training data set S . The PS optimization error R opt ( x ) is obtained by min λ ∈ ∆ M ∥∇ F S ( x ) λ ∥ , and the PS generalization error R gen ( x ) is estimated by min λ ∈ ∆ M ∥∇ F S te ( x ) λ ∥ -R opt ( x ) .

We examine the impact of different T , α , γ on the errors in Figure 4. Figure 4a shows that increasing T reduces optimization error and CA direction distance but increases generalization error, aligning with Theorems 1, 2, and 3. Figure 4b shows that increasing α leads to an initial decrease and subsequent increase

in PS optimization error and population risk. which aligns with Theorem 3 and (3.8). On the other hand, there is an overall increase in CA direction distance with α , which aligns with Theorem 2. Figure 4c shows that increasing γ increases both the PS population and optimization errors but decreases CA direction distance. This matches our bounds for PS optimization error in Theorem 3, PS population risk in (3.8), and CA direction distance in Theorem 2.

6 Conclusions

This work studies the three-way trade-off in MOL - among optimization, generalization, and conflict avoidance. Our results show that, in the general nonconvex setting, the traditional trade-off between optimization and generalization depending on the number of iterations also exists in MOL. Moreover, dynamic weighting algorithms like MoDo introduce a new dimension of trade-off in terms of conflict avoidance compared to static weighting. We demonstrate that this three-way trade-off can be controlled by the step size for updating the dynamic weighting parameter and the number of iterations. Proper choice of these parameters can lead to decent performance on all three metrics.

Broader impacts and limitations

This work has a potential impact on designing dynamic weighting algorithms and choosing hyperparameters such as step sizes and number of iterations based on the trade-off for MTL applications such as multi-language translation, and multi-agent reinforcement learning. No ethical concerns arise from this work. A limitation of this study is that the theory focuses on a specific algorithm, MoDo, for smooth objectives in unconstrained learning. Future research could explore the theory of other algorithms for non-smooth objectives or constrained learning, which would be interesting directions to pursue.

References

Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. In Proc. Advances in Neural Information Processing Systems , volume 33, virtual, 2020.

Amir Beck and Marc Teboulle. Gradient-based algorithms with applications to signal-recovery problems. In Convex Optimization in Signal Processing and Communications , 2010.

Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research , 2:499-526, March 2002.

Zachary Charles and Dimitris Papailiopoulos. Stability and generalization of learning algorithms that converge to global optima. In Proc. International Conference on Machine Learning , pages 744-753, Stockholm, Sweden, 2018.

Lisha Chen, Heshan Fernando, Yiming Ying, and Tianyi Chen. Three-way trade-off in multi-objective learning: Optimization, generalization and conflict-avoidance. arXiv preprint arXiv:2305.20057 , 2023.

Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proc. International Conference on Machine Learning , virtual, July 2018.

Corinna Cortes, Mehryar Mohri, Javier Gonzalvo, and Dmitry Storcheus. Agnostic learning with multiple objectives. In Proc. Advances in Neural Information Processing Systems , volume 33, pages 20485-20495, virtual, 2020.

Luc Devroye and Terry Wagner. Distribution-free performance bounds for potential function rules. IEEE Transactions on Information Theory , 25(5):601-604, 1979.

Jean-Antoine Désidéri. Multiple-gradient Descent Algorithm (MGDA) for Multi-objective Optimization. Comptes Rendus Mathematique , 350(5-6), 2012.

Heshan Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, and Tianyi Chen. Mitigating gradient bias in multi-objective learning: A provably convergent stochastic approach. In Proc. International Conference on Learning Representations , Kigali, Rwanda, 2023.

Jörg Fliege, A Ismael F Vaz, and Luís Nunes Vicente. Complexity of Gradient Descent for Multi-objective Optimization. Optimization Methods and Software , 34(5):949-959, 2019.

Xiang Gu, Xi Yu, Jian Sun, Zongben Xu, et al. Adversarial reweighting for partial domain adaptation. In Proc. Advances in Neural Information Processing Systems , virtual, December 2021.

Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision , Munich, Germany, July 2018.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Proc. International Conference on Machine Learning , pages 1225-1234, New York City, NY, 2016.

Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A Joint Manytask Model: Growing a Neural Network for Multiple NLP Tasks. arXiv preprint:1611.01587 , 2016.

Yuzheng Hu, Ruicheng Xian, Qilong Wu, Qiuling Fan, Lang Yin, and Han Zhao. Revisiting scalarization in multi-task learning: A theoretical perspective. In Proc. Advances in Neural Information Processing Systems , New Orleans, LA, 2023.

A Kendall, Y Gal, and R Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. arXiv preprint:1705.07115 , 2017.

Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, and Pawan K Mudigonda. In defense of the unitary scalarization for deep multi-task learning. In Proc. Advances in Neural Information Processing Systems , volume 35, pages 12169-12183, New Orleans, LA, 2022.

Ilja Kuzborskij and Christoph Lampert. Data-dependent stability of stochastic gradient descent. In Proc. International Conference on Machine Learning , pages 2815-2824, Stockholm, Sweden, 2018.

Panagiotis Kyriakis, Jyotirmoy Deshmukh, and Paul Bogdan. Pareto policy adaptation. In Proc. International Conference on Learning Representations , virtual, 2021.

Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ , 1998.

Yunwen Lei. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems. In Proc. Annual Conference on Learning Theory , Bangalore, India, July 2023.

Yunwen Lei, Rong Jin, and Yiming Ying. Stability and generalization analysis of gradient methods for shallow neural networks. In Proc. Advances in Neural Information Processing Systems , New Orleans, LA, 2022.

Yunwen Lei and Yiming Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. In Proc. International Conference on Machine Learning , pages 5809-5819, virtual, 2020.

Baijiong Lin, Ye Feiyang, Yu Zhang, and Ivor Tsang. Reasonable effectiveness of random weighting: A litmus test for multi-task learning. Transactions on Machine Learning Research , 2022.

Baijiong Lin and Yu Zhang. LibMTL: A python library for multi-task learning. arXiv preprint arXiv:2203.14338 , 2022.

Xi Lin, Zhiyuan Yang, and Qingfu Zhang. Pareto set learning for neural multi-objective combinatorial optimization. arXiv preprint arXiv:2203.15386 , 2022.

Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. In Proc. Advances in Neural Information Processing Systems , Vancouver, Canada, December 2019.

Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-Averse Gradient Descent for Multi-task Learning. In Proc. Advances in Neural Information Processing Systems , virtual, December 2021.

Suyun Liu and Luis Nunes Vicente. The Stochastic Multi-gradient Algorithm for Multi-objective Optimization and its Application to Supervised Machine Learning. Annals of Operations Research , pages 1-30, 2021.

Xingchao Liu, Xin Tong, and Qiang Liu. Profiling Pareto Front With Multi-Objective Stein Variational Gradient Descent. In Proc. Advances in Neural Information Processing Systems , virtual, December 2021.

Debabrata Mahapatra and Vaibhav Rajan. Exact pareto optimal search for multi-task learning: Touring the pareto front. arXiv preprint:2108.00597 , 2021.

Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. Journal of Machine Learning Research , 15(107):3663-3692, 2014.

Michinari Momma, Chaosheng Dong, and Jia Liu. A multi-objective/multi-task learning framework induced by pareto stationarity. In Proc. International Conference on Machine Learning , pages 15895-15907, Baltimore, MD, 2022.

Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik. Learning the pareto front with hypernetworks. In Proc. International Conference on Learning Representations , virtual, April 2020.

Yurii Nesterov. Introductory Lectures on Convex Optimization: A Basic Course . Springer Publishing Company, Incorporated, 1 edition, 2014.

Dominic Richards and Ilja Kuzborskij. Stability & generalisation of gradient descent for shallow neural networks without the neural tangent kernel. In Proc. Advances in Neural Information Processing Systems , volume 34, virtual, 2021.

Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint:1706.05098 , 2017.

Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Proc. Advances in Neural Information Processing Systems , Montreal, Canada, December 2018.

Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Proc. Advances in Neural Information Processing Systems , volume 30, Long Beach, CA, 2017.

Peter Súkeník and Christoph H Lampert. Generalization in multi-objective machine learning. arXiv preprint arXiv:2208.13499 , 2022.

Hiroki Tanabe, Ellen H. Fukuda, and Nobuo Yamashita. Proximal gradient methods for multiobjective optimization and their applications. Computational Optimization and Applications , 72(2):339-361, 2019.

Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Trans. Pattern Anal. Machine Intell. , 2021.

Peiyao Xiao, Hao Ban, and Kaiyi Ji. Direction-oriented multi-objective learning: Simple and provable stochastic algorithms. In Proc. Advances in Neural Information Processing Systems , New Orleans, LA, 2023.

Derrick Xin, Behrooz Ghorbani, Justin Gilmer, Ankush Garg, and Orhan Firat. Do current multi-task optimization methods in deep learning even help? In Proc. Advances in Neural Information Processing Systems , volume 35, pages 13597-13609, New Orleans, LA, 2022.

Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, and Yuhui Shi. Pareto policy pool for model-based offline reinforcement learning. In Proc. International Conference on Learning Representations , virtual, 2021.

Feiyang Ye, Baijiong Lin, Zhixiong Yue, Pengxin Guo, Qiao Xiao, and Yu Zhang. Multiobjective meta learning. In Proc. Advances in Neural Information Processing Systems , volume 34, pages 21338-21351, virtual, 2021.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Proc. Advances in Neural Information Processing Systems , virtual, December 2020.

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Proc. International Conference on Artificial Intelligence and Statistics , pages 962-970, Fort Lauderdale, FL, 2017.

Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Trans. Knowledge Data Eng. , 2021.

Yiyang Zhao, Linnan Wang, Kevin Yang, Tianjun Zhang, Tian Guo, and Yuandong Tian. Multi-objective optimization by learning space partitions. arXiv preprint arXiv:2110.03173 , 2021.

Shiji Zhou, Wenpeng Zhang, Jiyan Jiang, Wenliang Zhong, Jinjie Gu, and Wenwu Zhu. On the convergence of stochastic multi-objective gradient manipulation and beyond. In Proc. Advances in Neural Information Processing Systems , volume 35, pages 38103-38115, New Orleans, LA, December 2022.

Appendix for 'Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance "

Table of Contents

A Notations

A summary of notations used in this work is listed in Table 4 for ease of reference.

In the proof, we use ∥ · ∥ to denote the spectral norm, and ∥ · ∥ F to denote the Frobenius norm.

B Bounding the generalization error

B.1 Proof of Propositions 2-3

In this subsection, we prove Propositions 2-3, which establishes the relation between PS generalization error and MOL uniform stability.

Proof of Proposition 2. For a given model x , it holds that

where ( a ) follows from the subadditivity of max operator, ( b ) follows from triangle inequality, ( c ) follows from Cauchy-Schwartz inequality.

Proof of Proposition 3. The proof extends that of [22] for single objective learning to our MOL setting. Recall that S = { z 1 , . . . , z n } , which are drawn i.i.d. from the data distribution D . Define the perturbed dataset S ( i ) = { z 1 , . . . , z ′ i , . . . , z n } sampled i.i.d. from D with z ′ i independent of z j , for all i, j ∈ [ n ] . Let ˜ z be an independent sample of z j , z ′ j , for all j ∈ [ n ] , and from the same distribution D . We first decompose the difference of population gradient and empirical gradient on the algorithm output n ( ∇ F ( A ( S )) -∇ F S ( A ( S ))) as follows using the gradient on A ( S ( i ) ) . Since E ˜ z [ ∇ F ˜ z ( A ( S ))] = ∇ F ( A ( S )) , it holds that

where the last equality follows from rearranging and that z i , z ′ i , ˜ z are mutually independent. Applying triangle inequality to (B.3), it then follows that

Note S and S ( i ) differ by a single sample. By Definition 2, the MOL uniform stability ϵ F , and Jensen's inequality, we further get

We then proceed to bound E [∥ ∥ ∑ n i =1 ξ i ( S ) ∥ ∥ F ] , which satisfies

̸

For J 1 ,i , according to the definition of ξ i ( S ) in (B.3) and Jensen inequality, it holds that

̸

where ( a ) follows from Jensen's inequality, ( b ) follows from the symmetry between z i and z ′ i . To bound J 2 ,i,j with i = j , further introduce S ′′ = { z ′′ 1 , . . . , z ′′ n } which are drawn i.i.d. from the data distribution D . Then for each i, j ∈ [ n ] with i = j , introduce S j as a neighboring dataset of S by replacing its z j with z ′′ j , and S ( i ) j as a neighboring dataset of S ( i ) by replacing its z j with z ′′ j , i.e.,

̸

̸

Then the idea is to bound J 2 ,i,j using the newly introduced neighboring datasets S j and S ( i ) j , so as to connect to the definition of the stability ϵ F . We first show that E [ ⟨ ξ i ( S ) , ξ j ( S ) ⟩ ] = E [ ⟨ ξ i ( S ) -ξ i ( S j ) , ξ j ( S ) -ξ j ( S i ) ⟩ ] because for i = j ,

̸

For i = j , ( c ) follows from

where the second identity holds since ξ i ( S j ) is independent of z j and the last identity follows from E z j [ ξ j ( S )] = 0 due to the symmetry between ˜ z and z i , and their independence with S ( i ) , derived as

̸

In a similar way, for i = j , ( d ) and ( e ) follow from

̸

Based on (B.9), for i = j we have

̸

where we have used ab ≤ 1 2 ( a 2 + b 2 ) . According to the definition of ξ i ( S ) and ξ i ( S j ) we know the following identity for i = j

It then follows from the inequality ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) and the Jensen's inequality that

Since S ( i ) , S ( i ) j and S ( j ) , S ( j ) i are two pairs of neighboring datasets, it follows from the definition of stability that

̸

We can plug the above inequalities back into (B.14) and bound J 2 ,i,j by

̸

Combining the bound for J 1 ,i in (B.7) and J 2 ,i,j in (B.18) and substituting them back into (B.6), it then follows that

̸

Plugging the above inequality back into (B.5), using the subadditivity of the square root function, we get

The proof is complete.

B.2 Proof of Theorem 1 - PS generalization error in nonconvex case

In this subsection, we prove Theorem 1, which establishes the PS generalization error of MoDo in the nonconvex case.

Organization of proof. To prove the PS generalization error of MoDo, we first define the concept of Sampling-determined algorithms in Definition 3. This concept has been defined in [22] for the analysis in single-objective learning. Then we show that MoDo is sampling-determined in Proposition 4. Finally, combining Propositions 2-4, we can prove Theorem 1, the MOL uniform stability and PS generalization error of MoDo.

Definition 3 (Sampling-determined algorithm [22]) . Let A be a randomized algorithm that randomly chooses an index sequence I ( A ) = { i t,s } to compute stochastic gradients. We say a symmetric algorithm A is sampling-determined if the output model is fully determined by { z i : i ∈ I ( A ) } .

Proposition 4 (MoDo is sampling determined) . MoDo (Algorithm 1) is sampling determined. In other words, Let I ( A ) = { i t } be the sequence of index chosen by these algorithms from training set S = { z 1 , . . . , z n } , and z i i . i . d . ∼ P for all i ∈ [ n ] to build stochastic gradients, the output A ( S ) is determined by { z j : j ∈ I ( A ) } . To be precise, A ( S ) is independent of z j if j / ∈ I ( A ) .

Proof of Proposition 4. Let I ( A ) = { I 1 , . . . , I T } , I t = { i t,s } 3 s =1 and i t,s ∈ [ n ] for all 1 ≤ t ≤ T . And S I ( A ) = { z i t,s } . By the description in Algorithm 1, A ( S ) = G z I T ◦ · · · ◦ G z I 1 ( x 0 ) , where G z ( · ) is the stochastic update function of the model parameter given random sample z . Therefore, for all possible sample realization z , we have

where the last equality holds because z j / ∈ S I ( A ) , and z j is independent of all elements in S I ( A ) by i . i . d . sampling. Therefore, A ( S ) is independent of z j if j / ∈ I ( A ) . The proof is complete.

Note that, besides MoDo, other popular stochastic randomized MTL algorithms such as SMG [30] and MoCo [10] are also sampling-determined. Therefore, the result is also applicable to these algorithms.

Lemma 3 ([22, Theorem 5 (b)]) . Let A be a sampling-determined random algorithm (Definition 3) and S, S ′ be neighboring datasets with n data points that differ only in the i -th data point. If sup z E A [ ∥∇ F z ( A ( S )) ∥ 2 F | i ∈ I ( A ) ] ≤ G 2 for any S , then

Proof of Theorem 1. From Proposition 4, algorithm A , MoDo is sampling-determined. Then based on Lemma 3, its MOL uniform stability in Definition 2 can be bounded by

Let i t be the index of the sample selected by A at the t -th step, and i ∗ be the index of the data point that is different in S and S ′ . Then

Combining (B.23) and (B.24) gives

Then based on Propositions 2-3, we have

The proof is complete.

B.3 Proof of Lemma 1 - Boundedness of x t in strongly convex case

Technical challenges. In this work, we focus on analyzing stochastic MGDA-based MOL algorithms in the unconstrained setting. This is because in the constrained setting, MGDA with projected gradient descent on x has no guarantee to find the CA direction, and a new algorithm needs to be developed to achieve this [42]. However, a fundamental challenge in the unconstrained strongly convex setting is that a strongly convex function is not Lipschitz continuous on R d . We overcome this challenge by showing that { x t } T t =1 generated by the MoDo algorithm is bounded on the trajectory, so is the gradient ∥∇ f z,m ( x t ) ∥ for all m ∈ [ M ] , and z ∈ Z . Thereby, we can derive the upper bound of PS optimization and generalization errors without the Lipschitz continuity assumption for strongly convex objectives.

Organization of proof. Without loss of generality, we assume inf x ∈ R d f m,z ( x ) < ∞ for all m ∈ [ M ] and z ∈ Z in the strongly convex case. In Lemma 4, we show that the optimal solution of F z ( x ) λ given any stochastic sample z ∈ Z , and weighting parameter λ ∈ ∆ M , is bounded. In Lemma 5, we show that if the argument parameter is bounded, then the updated parameter by MoDo at each iteration is also bounded by exploiting the co-coerciveness of strongly convex and smooth objectives. Finally, based on Lemma 4 and Lemma 5, we first prove in Corollary 4 that with a bounded initialization x 0 , the model parameter { x t } T t =1 generated by MoDo algorithm is bounded on the trajectory. Then based on Lemma 6, by the Lipschitz smoothness assumption of f z,m ( x ) , we immediately have that ∥∇ f z,m ( x ) ∥ is bounded for x ∈ { x t } T t =1 generated by MoDo algorithm, which completes the proof of Lemma 1. Lemma 1 paves the way for deriving the MOL uniform stability and PS generalization error of MoDo in the strongly convex and unconstrained setting.

Lemma 4. Suppose Assumptions 1, 2 hold. WLOG, assume inf x ∈ R d f m,z ( x ) < ∞ for all m ∈ [ M ] and z ∈ Z . For any given λ ∈ ∆ M , and stochastic sample z ∈ Z , define x ∗ λ,z = arg min x ∈ R d F z ( x ) λ , then inf x ∈ R d F z ( x ) λ < ∞ and ∥ x ∗ λ,z ∥ < ∞ , i.e., there exist finite positive constants c F ∗ and c x ∗ such that

Proof. Under Assumption 2, for all m ∈ [ M ] , f m,z ( x ) is strongly convex w.r.t. x , thus has a unique minimizer. Define the minimizer x ∗ m,z = arg min x ∈ R d f m,z ( x ) . Since a strongly convex function is coercive, inf x ∈ R d f m,z ( x ) < ∞ , i.e., f m,z ( x ∗ m,z ) < ∞ , implies that ∥ x ∗ m,z ∥ < ∞ .

By Assumption 1, the ℓ f, 1 -Lipschitz smoothness of f m,z ( x ) , for x such that ∥ x ∥ < ∞

Since F z ( x ) λ is convex w.r.t. x , for all λ ∈ ∆ M , with λ = [ λ 1 , . . . , λ M ] ⊤ , we have

Therefore, for all λ ∈ ∆ M , we have

Since F z ( x ) λ is strongly convex, thus is coercive, we have ∥ x ∗ λ,z ∥ < ∞ , which proves the result.

Lemma 5. Suppose Assumptions 1, 2 hold, and define κ = 3 ℓ f, 1 /µ ≥ 3 . For any given λ ∈ ∆ M , and a stochastic sample z ∈ Z , define x ∗ λ,z = arg min x F z ( x ) λ . Then by Lemma 4, there exists a positive finite constant c x, 1 ≥ c x ∗ such that ∥ x ∗ λ,z ∥ ≤ c x ∗ ≤ c x, 1 . Recall the multi-objective gradient update is

-1 √ , we have that with step size 0 ≤ α ≤ ℓ f, 1 . Defining c x, 2 = (1 + 2 κ ) c x, 1

Proof. We divide the proof into two cases: 1) when ∥ x ∥ < c x, 1 ; and, 2) when c x, 1 ≤ ∥ x ∥ ≤ c x, 2 . 1) For the first case, ∥ x ∥ < c x, 1 ≤ c x, 2 , then we have

where ( a ) follows from ∇ F z ( x ∗ ) λ = 0 , and ( b ) follows from the non-expansiveness of the gradient update for strongly convex and smooth function.

2) For the second case, c x, 1 ≤ ∥ x ∥ ≤ c x, 2 , we first consider α = ℓ -1 f, 1 . Let µ ′ = µ/ 3 . Note that since F z ( x ) λ is µ -strongly convex, it is also µ ′ -strongly convex. By strong convexity and smoothness of F z ( x ) λ , the gradients are co-coercive [36, Theorem 2.1.12], i.e., for any x we have

Rearranging and applying Cauchy-Schwartz inequality, we have

By the definition of G λ,z ( x ) ,

Substituting (B.34) into (B.35) yields

Since κ ≥ 3 , thus 1 -κ -1 1+ κ -1 > 0 , then I 1 is a quadratic function w.r.t. τ , and is strictly concave, thus can be bounded above by

Substituting this back into (B.36) gives that

where the last inequality follows from κ ≥ 3 , thus 1+ κ -1 1 -κ -1 ≤ 2 , -2 κ -1 1+ κ -1 ≤ -κ -1 , and ∥ x ∗ ∥ ≤ c x, 1 ≤ ∥ x ∥ by assumption.

For c x, 1 ≤ ∥ x ∥ ≤ c x, 2 , I 2 is a strictly convex quadratic function of ∥ x ∥ , which achieves its maximum at ∥ x ∥ = c x, 1 or ∥ x ∥ = c x, 2 . Therefore,

where ( c ) follows from the definition that c x, 2 = (1 + √ 2 κ ) c x, 1 ; ( d ) follows from κ ≥ 3 , and thus 3 c 2 x, 1 < (1 + √ 2 κ ) 2 c 2 x, 1 = c 2 x, 2 .

We have proved the case for α = ℓ -1 f, 1 . The result for 0 ≤ α < ℓ -1 f, 1 follows by observing that,

The proof is complete.

Lemma 6. Suppose Assumptions 1, 2 hold. For all λ ∈ ∆ M and z ∈ S , define x ∗ λ,z = arg min x F z ( x ) λ , then there exist finite positive constants c F ∗ and c x ∗ such that F z ( x ∗ λ,z ) λ ≤ c F ∗ and ∥ x ∗ λ,z ∥ ≤ c x ∗ . And for x ∈ R d such that ∥ x ∥ is bounded, i.e., there exists a finite positive constant c x such that ∥ x ∥ ≤ c x , then

Proof. Under Assumptions 1, 2, by Lemma 4, there exist finite positive constants c F ∗ and c x ∗ such that F z ( x ∗ λ,z ) λ ≤ c F ∗ and ∥ x ∗ λ,z ∥ ≤ c x ∗ .

By Assumption 1, the ℓ f, 1 -Lipschitz continuity of the gradient ∇ F z ( x ) λ , we have

where the first equality uses the fact that ∇ F z ( x ∗ λ,z ) λ = 0 .

For the function value, by Assumption 1, the ℓ f, 1 -Lipschitz smoothness of F z ( x ) λ , we have

from which the proof is complete.

Corollary 4 ( x t bounded on the MoDo trajectory) . Suppose Assumptions 1, 2 hold. Define κ = 3 ℓ f, 1 /µ and x ∗ λ,z = arg min x F z ( x ) λ with λ ∈ ∆ M . Then there exists a finite positive constant c x ∗ such that ∥ x ∗ λ,z ∥ ≤ c x ∗ . Choose the initial iterate to be bounded, i.e., there exists a finite positive constant c x 0 such that ∥ x 0 ∥ ≤ c x 0 , then for { x t } generated by MoDo algorithm with α t = α and 0 ≤ α ≤ ℓ -1 f, 1 , we have

Proof of Corollary 4. Under Assumptions 1, 2, by Lemma 4, ∥ x ∗ λ,z ∥ < ∞ , i.e., there exists a finite positive constant c x ∗ such that ∥ x ∗ λ,z ∥ ≤ c x ∗ . Let c x, 1 = max { (1 + √ 2 κ ) -1 c x 0 , c x ∗ } , and c x, 2 = (1 + √ 2 κ ) c x, 1 = max { c x 0 , (1 + √ 2 κ ) c x ∗ } in Lemma 5. We then consider the following two cases:

1) If (1 + √ 2 κ ) c x ∗ ≤ c x 0 , then ∥ x ∗ λ,z ∥ ≤ c x ∗ ≤ (1 + √ 2 κ ) -1 c x 0 . Then it satisfies the condition in Lemma 5 that ∥ x ∗ λ,z ∥ ≤ c x, 1 and ∥ x 0 ∥ ≤ c x, 2 . Applying Lemma 5 yields ∥ x 1 ∥ ≤ c x, 2 .

2) If (1 + √ 2 κ ) c x ∗ > c x 0 , then ∥ x 0 ∥ ≤ c x 0 < (1 + √ 2 κ ) c x ∗ . Then it satisfies the condition in Lemma 5 that ∥ x ∗ λ,z ∥ ≤ c x, 1 and ∥ x 0 ∥ ≤ c x, 2 . Applying Lemma 5 yields ∥ x 1 ∥ ≤ c x, 2 .

Therefore, (B.44) holds for t = 1 . We then prove by induction that (B.44) also holds for t ∈ [ T ] . Assume (B.44) holds at 1 ≤ k ≤ T -1 , i.e.,

Since ∥ x 1 ∥ ≤ c x, 2 , for t = 0 , . . . , T -1 , we have

Therefore, by mathematical induction, ∥ x t ∥ ≤ c x, 2 = c x , for all t ∈ [ T ] . The proof is complete.

Then by Lemma 5, at k +1 ,

Proof of Lemma 1. By Corollary 4, for { x t } generated by MoDo algorithm with α t = α and 0 ≤ α ≤ ℓ -1 f, 1 , we have

According to Lemma 6, define ℓ f = ℓ f, 1 ( c x + c x ∗ ) , and ℓ F = √ Mℓ f , then it holds for all λ ∈ ∆ M

B.4 Proof of Theorem 2 - PS generalization error in strongly convex case

Technical challenges. One challenge that the strongly convex objectives are not Lipschitz continuous for x ∈ R d is addressed by Lemma 1. Another challenge compared to static weighting or single-objective learning is that the MoDo algorithm involves the update of two coupled sequences { x t } and { λ t } . Consequently, the traditional standard argument that the SGD update for strongly convex objectives has the contraction property [14] does not necessarily hold in our case since the weighting parameter λ is changing, as detailed in Section B.4.1. Nevertheless, we manage to derive a tight stability bound when γ = O ( T -1 ) , as detailed in Section B.4.3.

Organization of proof. In Section B.4.1, we prove the properties of the MoDo update, including expansiveness or non-expansiveness and boundedness. Building upon these properties, in Section B.4.3, we prove the upper bound of argument stability in Theorem 5, and the upper bound of MOL uniform stability. To show the tightness of the upper bound, in Section B.4.4, Theorem 6, we derive a matching lower bound of MOL uniform stability. Combining the upper bound in Section B.4.3 and the lower bound in Section B.4.4 leads to the results in Theorem 2, whose proof is in Section B.4.5.

B.4.1 Expansiveness and boundedness of MoDo update

In this section, we prove the properties of the update function of MoDo at each iteration, including boundedness and approximate expansiveness, which is then used to derive the algorithm stability. For z, z 1 , z 2 ∈ S , λ ∈ ∆ M , recall that the update functions of MoDo is

Lemma 7 (Boundedness of update function of MoDo) . Let ℓ f be a positive constant. If ∥∇ F z ( x ) λ ∥ ≤ ℓ f for all λ ∈ ∆ M , z ∈ S and x ∈ { x t } T t =1 generated by the MoDo algorithm with step size α t ≤ α , then G λ,z ( x ) is ( αℓ f ) -bounded on the trajectory of MoDo, i.e.,

Proof. For all x ∈ { x t } T t =1 , λ ∈ ∆ M , and z ∈ S , since ∥∇ F z ( x ) λ ∥ ≤ ℓ f , we have

which proves the boundedness.

Lemma 8 (Properties of update function of MoDo in convex case) . Suppose Assumptions 1, 2 hold. Let ℓ f be a positive constant. If for all λ, λ ′ ∈ ∆ M , z ∈ S , and x ∈ { x t } T t =1 , x ′ ∈ { x ′ t } T t =1 generated by the MoDo algorithm on datasets S and S ′ , respectively, we have ∥∇ F z ( x ) λ ∥ ≤ ℓ f , ∥∇ F z ( x ′ ) λ ′ ∥ ≤ ℓ f , and ∥∇ F z ( x ) ∥ ≤ ℓ F , ∥∇ F z ( x ′ ) ∥ ≤ ℓ F , and step sizes of MoDo satisfy α t ≤ α , γ t ≤ γ , it holds that

Proof. The squared norm of the difference of G λ,z ( x ) and G λ ′ ,z ( x ′ ) can be bounded by

where ( a ) follows from rearranging and that ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 ; ( b ) follows from the µ -strong convexity of F z ( x ) λ , ℓ f, 1 -Lipschitz continuity of ∇ F z ( x ) λ , and that ∥∇ F z ( x ′ ) ∥ ≤ ℓ F for x ′ ∈ { x ′ t } T t =1 ; and, ( c ) follows from Cauchy-Schwartz inequality.

And ∥ G x,z 1 ,z 2 ( λ ) -G x ′ ,z 1 ,z 2 ( λ ′ ) ∥ can be bounded by

where ( d ) follows from non-expansiveness of projection; ( e ) follows from triangle inequality, ( f ) follows from ∥∇ F z ( x ) ∥ ≤ ℓ F for x ∈ { x ′ t } T t =1 , ( g ) follows from triangle inequality; and ( h ) follows from ℓ f, 1 -Lipschitz continuity of ∇ F z ( x ) λ ′ , ℓ F, 1 -Lipschitz continuity of ∇ F z ( x ) , ∥∇ F z ( x ) ∥ ≤ ℓ F for x ∈ { x ′ t } T t =1 , and ∥∇ F z ( x ) λ ′ ∥ ≤ ℓ f for x ∈ { x t } T t =1 .

Let ℓ g, 1 = ℓ f ℓ F, 1 + ℓ F ℓ f, 1 . Taking square on both sides of (B.55) yields

The proof is complete.

B.4.2 Growth recursion

Lemma 9 (Growth recursion with approximate expansiveness) . Fix an arbitrary sequence of updates G 1 , . . . , G T and another sequence G ′ 1 , . . . , G ′ T . Let x 0 = x ′ 0 be a starting point in Ω and define δ t = ∥ x ′ t -x t ∥ where x t , x ′ t are defined recursively through

Let η t > 0 , ν t ≥ 0 , and ς t ≥ 0 . Then, for any p > 0 , and t ∈ [ T ] , we have the recurrence relation (with δ 0 = 0 )

Proof. When G t and G ′ t are ς t -bounded, we can bound δ t +1 by

Alternatively, when G t and G ′ t are ς t -bounded, G t is ( η t , ν t ) -approximately expansive, we have

When G t = G ′ t , is ( η t , ν t ) -approximately expansive in square, given δ 2 t , δ 2 t +1 can be bounded by

When G t and G ′ t are ς t -bounded, applying (B.57), we can bound δ 2 t +1 by

where p > 0 and the last inequality follows from ( a + b ) 2 ≤ (1 + p ) a 2 +(1 + 1 /p ) b 2 .

Alternatively, when G t and G ′ t are ς t -bounded, G t is ( η t , ν t ) -approximately expansive in square, the following holds

The proof is complete.

B.4.3 Upper bound of MOL uniform stability

In Theorem 5 we bound the argument stability, which is then used to derive the MOL uniform stability and PS generalization error in Theorem 2.

Theorem 5 (Argument stability bound in strongly convex case) . Suppose Assumptions 1, 2, hold. Let A be the MoDo algorithm in Algorithm 1. Choose the step sizes α t ≤ α ≤ min { 1 / (2 ℓ f, 1 ) , µ/ (2 ℓ 2 f, 1 ) } , and γ t ≤ γ ≤ min { µ 2 120 ℓ 2 f ℓ g, 1 , 1 8(3 ℓ 2 f +2 ℓ g, 1 ) } /T . Then it holds that

Proof of Theorem 5. Under Assumptions 1, 2, Lemma 1 implies that for { x t } generated by the MoDo algorithm, and for all λ ∈ ∆ M , and for all m ∈ [ M ] ,

For notation simplicity, denote δ t = ∥ x t -x ′ t ∥ , ζ t = ∥ λ t -λ ′ t ∥ , x T = A T ( S ) and x ′ T = A T ( S ′ ) . Denote the index of the different sample in S and S ′ as i ∗ , and the set of indices selected at the t -th iteration as I t , i.e., I t = { i t,s } 3 s =1 . When i ∗ / ∈ I t , for any c 1 > 0 , based on Lemma 8, we have

where the second last inequality is due to Young's inequality; the last inequality is due to choosing α t ≤ µ/ (2 ℓ 2 f, 1 ) .

When i ∗ ∈ I t , from Lemma 7, the ( α t ℓ f ) -boundedness of the update at t -th iteration, and Lemma 9, the growth recursion, for a given constant p > 0 , we have

Taking expectation of δ 2 t +1 over I t , we have

At each iteration of MoDo, we randomly select three independent samples (instead of one) from the training set S . Then the probability of selecting the different sample from S and S ′ at the t -th iteration, P ( i ∗ ∈ I t ) in the above equation, can be computed as follows

Consequently, the probability of selecting the same sample from S and S ′ at the t -th iteration is P ( i ∗ / ∈ I t ) = 1 -P ( i ∗ ∈ I t ) .

Let ℓ g, 1 = ℓ f ℓ F, 1 + ℓ F ℓ f, 1 . Recalling when i ∗ / ∈ I t , ζ t +1 ≤ (1+ ℓ 2 F γ t ) ζ t +2 γ t ℓ g, 1 δ t from Lemma 8, it follows that

where the last inequality follows from ℓ g, 1 γ t ≤ 1 , and ℓ 2 F γ t ≤ 1 .

And since ζ t and δ t are independent of I t , it follows that

Combining (B.66) and (B.69), we have

where we define η t = 1 -α t ( µ -ℓ F c 1 -3 c 2 ℓ g, 1 γ t ) P ( i / ∈ I t ) .

While when i ∗ ∈ I t , for a given constant p 2 > 0 , we have

Taking expectation of ζ 2 t +1 over I t gives

Based on linearity of expectation and applying (B.72) recursively yields

where ( a ) follows from choosing γ t ≤ γ ≤ 1 / (8 c 3 T ) , p 2 = n/ (8 T ) , ( b ) follows from t -t ′ ≤ T , and (1 + a T ) T ≤ e a , and the inequality ( c ) follows from e 1 2 < 2 .

Note that δ 0 = 0 , ζ 1 = 0 . Applying (B.66) at t = 0 gives

which together with (B.72) gives

Therefore, for 0 ≤ t ≤ 1 , it satisfies that

Next, we will prove by induction that (B.74) also holds for t > 1 .

Assuming that (B.74) holds for all 0 ≤ t ≤ k ≤ T -1 , we apply (B.70) to the case where t = k to obtain

where ( a ) follows from (B.73), and ( b ) follows from (B.74) for 0 ≤ t ≤ k and that γk ≤ γT ≤ 1 . The coefficient J 1 in (B.75) can be further bounded by

where ( c ) follows from β t ≤ β t +1 , γ t ≤ γ for all t = 0 , . . . , T ; ( d ) follows from the definition of η k ; ( e ) is because γ ≤ µ 2 / (120 ℓ 2 F ℓ g, 1 T ) , α ≤ 1 / (2 ℓ f, 1 ) ≤ 1 / (2 µ ) and choosing c 1 = µ/ (4 ℓ F ) leads to

Combining (B.75) and (B.76) implies

where the equality follows by the definition of β t given in (B.74). The above statements from (B.75)-(B.77) show that if (B.74) holds for all t such that 0 ≤ t ≤ k ≤ T -1 , it also holds for t = k +1 .

Therefore, we can conclude that for T ≥ 0 , it follows

where the last equality follows from taking p = αµn/ 12 , and compute the sum of geometric series. By plugging in c 1 = µ/ (4 ℓ F ) , c 2 = ℓ F c -1 1 +2 αℓ 2 F , c 3 = 3 ℓ 2 F +2 ℓ g, 1 , we have that

where the last inequality follows from c 2 = ℓ 2 F (4 µ -1 + 2 α ) ≤ 5 Mℓ 2 f µ -1 , and c 2 c -1 3 ≤ 5 ℓ 2 F µ -1 / (3 ℓ 2 F ) ≤ 2 µ -1 .

B.4.4 Lower bound of MOL uniform stability

In this section, we construct Example 1 with a lower bound of stability for the MoDo algorithm. Before proceeding to the example, we first define 1 as the all-one vector in R M , ˜ ∆ M := { λ ∈ R M | 1 ⊤ λ = 1 } , and P 1 := I -1 M 11 ⊤ . Then given any vector u ∈ R M , Π ˜ ∆ M ( u ) = P 1 u + 1 M 1 .

Example 1. Recall that S = { z 1 , z 2 , . . . , z j , . . . z n } , S ′ = { z 1 , z 2 , . . . , z ′ j , . . . , z n } , where S and S ′ differ only in the j -th data point. Define the m -th objective function as

where A is a symmetric positive definite matrix, µ = 16 n -1 3 > 0 is the smallest eigenvalue of A , and v is the corresponding eigenvector of A . For the datasets S , and S ′ , let z i = c i v , with E z ∈ S [ z ] = µv , E z ∈ S ′ [ z ] = µ ′ v , z j -z ′ j = v , i.e., µ -µ ′ = 1 n . For simplicity, let M = 2 , b = [1 , 1 + √ 2] ⊤ such that P 1 b = b P = [ -1 √ 2 , 1 √ 2 ] ⊤ , where b P is the eigenvector of P 1 with eigenvalue 1 .

Technical challenges. One challenge of deriving the lower bound lies in the projection operator when updating λ . Unlike deriving the upper bound, where the projection operator can be handled by its non-expansive property, due to the nature of inequality constrained quadratic programming, neither a simple closed-form solution can be obtained, nor a non-trivial tight lower bound can be derived in general. We overcome this challenge by showing that when γ = O ( T -1 ) , the Euclidean projection onto the simplex is equivalent to a linear operator in Lemma 10. Another challenge compared to deriving the lower bound for single-objective learning is that the update of MoDo involves two coupled sequences, { x t } and { λ t } . The update of x t and x ′ t involves different weighting parameters λ t and λ ′ t , where { x t } , { λ t } and { x ′ t } , { λ ′ t } are generated by the MoDo algorithm on neighboring training data S and S ′ , respectively. We overcome this challenge by deriving a recursive relation of the vector [ x t -x ′ t ; λ t -λ ′ t ] in Lemma 11.

Organization of proof. Lemma 10 proves that under proper choice of initialization of λ and step size γ , the projection of the updated λ onto simplex ∆ M is equal to the projection of that onto the set ˜ ∆ M := { λ ∈ R M | 1 ⊤ λ = 1 } . And thus the projection is equivalent to a linear transformation. Thanks to Lemma 10, we are able to derive a recursive relation the vector [ x t -x ′ t ; λ t -λ ′ t ] in Lemma 11. Finally, relying on the recursive relation, we derive a lower bound for the recursion of [ E A ∥ x t -x ′ t ∥ ; E A ∥ λ t -λ ′ t ∥ ] , depending on a 2 × 2 transition matrix. And based on its eigen decomposition, we could compute the T -th power of such a transition matrix, which is used to derive the final lower bound of E A ∥ A ( S ) -A ( S ′ ) ∥ in Theorem 6.

Lemma 10. Suppose Assumptions 1, 2 hold. For MoDo algorithm, choose λ 0 = 1 M 1 , γ ≤ 1 2 MTℓ F ℓ f , and define

then the update of λ t for MoDo algorithm is λ t +1 = Π ∆ M ( λ + t ) .

Define the set ˜ ∆ M := { λ ∈ R M | 1 ⊤ λ = 1 } , P 1 := I -1 M 11 ⊤ , λ P,t := Π ˜ ∆ M ( λ + t ) = P 1 λ + t + 1 M 1 . Then for t = 0 , . . . , T -1 , it holds that

Proof. By the update of λ t , we have

where the last inequality follows from Lemma 6, with ℓ f = ℓ f, 1 ( c x + c x ∗ ) .

Then for all t ∈ [ T -1] , it holds that

where the last inequality follows from γ ≤ 1 2 MTℓ F ℓ f .

Then for t ∈ [ T -1] , it holds that

By the update of λ t , and the definition of projection,

Let λ P,t = [ λ P,t, 1 , . . . , λ P,t,M ] ⊤ . Then it holds that

where ( a ) follows from non-expansiveness of projection and that λ 0 ∈ ˜ ∆ M ; ( b ) follows from (B.85). Therefore, each element of λ P,t satisfies

which shows that λ P,t = Π ˜ ∆ M ( λ + t ) ∈ ∆ M . Therefore it holds that,

where ( c ) is because λ P,t ∈ ∆ M ; ( d ) is because ∆ M ⊂ ˜ ∆ M by the definition of the simplex; ( e ) is because λ P,t = Π ˜ ∆ M ( λ + t ) . Then the equality holds that

Also we have and

The proof is complete.

With the help of Lemma 10, which simplifies the Euclidean projection operator as a linear operator, we then prove in Lemma 11, the recursive relation of x t -x ′ t and λ t -λ ′ t .

Lemma 11. Suppose Assumptions 1, 2 hold. Under Example 1, choose λ 0 = 1 M 1 , γ ≤ 1 2 MTℓ F ℓ f for the MoDo algorithm. Denote { x t } , { λ t } and { x ′ t } , { λ ′ t } as the sequences generated by the MoDo algorithm with dataset S and S ′ , respectively. Then it holds that

and φ x,t , φ λ,t satisfy the following recursion

where 1 ( · ) is the indicator function.

Proof. Denote z t,s , and z ′ t,s , s ∈ [3] , as the samples selected in the t -th iteration from S and S ′ , respectively. According to the MoDo algorithm update of x t , and the definition of the problem in (B.80), we have

The difference x t +1 -x ′ t +1 can be computed by

̸

where 1 ( · ) denotes the indicator function, and the last equation follows from that z t, 3 -z ′ t, 3 = 0 if i t, 3 = j , and z t, 3 -z ′ t, 3 = z j -z ′ j = v if i t, 3 = j .

By Lemma 10, in Example 1, λ t +1 = P 1 ( λ t -γ ∇ F z t, 1 ( x t ) ⊤ ∇ F z t, 2 ( x t ) λ t ) + 1 M 1 , which can be further derived as

where ( a ) follows from rearranging the equation and that P 1 λ t + 1 M 1 = Π ˜ ∆ M ( λ t ) = λ t as λ t ∈ ˜ ∆ M ; ( b ) follows from that P 1 b = b P and P 1 1 = 0 .

The difference λ t +1 -λ ′ t +1 can be derived as

̸

where ( c ) follows from rearranging the equation; ( d ) follows from that z t,s -z ′ t,s = 0 if i t,s = j , and z t,s -z ′ t,s = z j -z ′ j = v if i t,s = j , and that Av = µv .

Combining (B.96) and (B.98) gives

where the matrices are defined as

Next we show by induction that

First, when t = 0 , x 0 -x ′ 0 = 0 = φ x, 0 v and λ 0 -λ ′ 0 = φ λ, 0 b P with φ x, 0 = 0 and φ λ, 0 = 0 . Therefore (B.101) holds at t = 0 . Supposing that (B.101) holds for t = k , next we show that it also holds at t = k +1 .

At t = k +1 , applying (B.99) for x k +1 -x ′ k +1 , and substituting (B.100a), (B.100b) yields

where φ x,k +1 is computed by

Therefore, for all t ∈ [ T ] , it holds that x t -x = φ x,t v .

At t = k +1 , apply (B.99) for λ k +1 -λ , and substitute (B.100c), (B.100d) yields

′ t ′ k +1

where φ λ,k +1 is computed by

Lemma 11 provides the recursive relation of x t -x ′ t and λ t -λ ′ t . And Lemma 12 below provides another property used to derive the lower bound of E [ ∥ x T -x ′ T ∥ ] in Theorem 6.

Lemma 12. Suppose Assumptions 1, 2 hold. Under Example 1, choose x 0 = x ′ 0 = 7 v , α = 1 4 µT for the MoDo algorithm. Denote { x t } , { λ t } and { x ′ t } , { λ ′ t } as the sequences generated by the MoDo algorithm with dataset S and S ′ , respectively. Then it holds that

Proof. From the update of x ′ t , we have

Suppose x ′ t = c ′ x,t v , then x ′ t +1 = c ′ x,t +1 v with

Applying the above inequality recursively gives

Since x 0 = 7 v , µ ′ µ -1 ≤ 1 , it holds that

Then it follows that

By the update of λ ′ t from (B.97),

The proof is complete.

Theorem 6. Suppose Assumptions 1 and 2 hold. Under Example 1 with M = 2 , choose λ 0 = 1 M 1 , x 0 = x ′ 0 = 7 v , α = 1 4 µT , 0 < γ ≤ 1 2 MTℓ F ℓ f , and T ≤ 4 n 2 3 for the MoDo algorithm. Denote { x t } , { λ t } and { x ′ t } , { λ ′ t } as the sequences generated by the MoDo algorithm with dataset S and S ′ , respectively. Then it holds that

Proof. Denote δ t = ∥ x t -x ′ t ∥ , ζ t = ∥ λ t -λ ′ t ∥ . From Lemma 11, it holds that

where φ x,t , φ λ,t satisfy

The expectation of φ x,t +1 can be further bounded as

where ( a ) follows from E A [ c t,s ] = µ , E A [ c ′ t,s ] = µ ′ ≤ µ , E [ λ ′ t +1 ] ≥ E [ λ ′ t ] by Lemma 12, and the fact that c t,s is independent of φ x,t .

Similarly, the expectation of φ λ,t +1 can be further bounded as

where ( b ) follows from E A [ c t,s ] = µ , E A [ c ′ t,s ] = µ ′ ≤ µ , and Lemma 12.

The above arguments prove that

where the inequality for vectors denotes the inequality of each corresponding element in the vectors, and matrix B has v B, 1 = [1 , 1] ⊤ as an eigenvector associated with the eigenvalue 1 because

Similarly, since

then v B, 2 = [ α (1 -γµ 2 ) , -γµ ] ⊤ is another eigenvector of B with a positive eigenvalue (1 -αµ )(1 -γµ 2 ) < 1 . Let Q B = [ v B, 1 , v B, 2 ] , which can be expressed as

Then B has eigenvalue decomposition B = Q B Λ B Q -1 B , where Λ B = diag([1 , (1 -αµ )(1 -γµ 2 )]) , and thus B t = Q B Λ t B Q -1 B for t ∈ [ T ] .

Let [ α (1 -γµ 2 ) , -µ ′ γ ] ⊤ = Q B [ c B, 1 , c B, 2 ] ⊤ , where [ c B, 1 , c B, 2 ] ⊤ = Q -1 B [ α (1 -γµ 2 ) , -µ ′ γ ] ⊤ can be computed by

where the last inequality follows from µ -µ ′ = 1 n and α (1 -γµ 2 ) ≥ 1 2 α = 1 / (8 µT ) ≥ γµ for c B, 1 ≥ 1 2 n γ , and αµ 2 = µ/ (4 T ) = 4 n -1 3 T -1 ≥ n -1 = µ -µ ′ , so that α + µ ′ γ = α + γ ( µ ′ -µ + µ ) ≥ α + γ ( µ -αµ 2 ) = α (1 -γµ 2 ) + γµ for c B, 2 ≥ 1 .

Since all elements in B are positive, multiplying B on both sides preserves inequality. Applying (B.115) recursively yields

where the last inequality follows from c B, 1 ≥ 1 2 n γ , and c B, 2 ≥ 1 . Plugging in v B, 1 = [1 , 1] ⊤ and v B, 2 = [ α (1 -γµ 2 ) , -γµ ] ⊤ , and since (1 -γµ 2 ) ≥ 1 2 , it follows that E [ δ T ] ≥ γT 2 n 2 + 1 16 n .

B.4.5 Proof of Theorem 2

Proof of Theorem 2. Combining the argument stability in Theorem 5, and Assumption 1, the MOL uniform stability can be bounded by

Then based on Propositions 2-3, we have

The proof of the upper bound is complete. We then prove the MOL uniform stability lower bound based on the argument uniform stability lower bound in Theorem 6. By the strong convexity of the function f m,z ( x ) , for all m ∈ [ M ]

The proof of the lower bound is complete.

C Bounding the optimization error

C.1 Auxiliary lemmas

Lemma 13 (Uniqueness of CA direction) . Given Q ∈ R d × M , then d Q := Qλ ∗ with λ ∗ ∈ arg min λ ∈ ∆ M ∥ Qλ ∥ 2 exists, and d Q is unique.

Proof. This is a standard result due to convexity of the subproblem. Proof is given in [9, Section 2].

Lemma 14. For any x ∈ R d , define λ ∗ ( x ) such that

Then, for any x ∈ R d and λ ∈ ∆ M , it holds that

Proof. By the first order optimality condition for (C.1) , for any x ∈ R d and λ ∈ ∆ M , we have

By rearranging the above inequality, we obtain

which is precisely the first inequality in the claim. Furthermore, we can also have

which is the desired second inequality in the claim. Hence, the proof is complete.

Lemma 15 (Continuity of λ ∗ ρ ( x ) ) . Given any ρ > 0 and x ∈ R d , define λ ∗ ρ ( x ) = arg min λ ∈ ∆ M 1 2 ∥∇ F S ( x ) λ ∥ 2 + 1 2 ρ ∥ λ ∥ 2 , then the following inequality holds

Suppose either 1) Assumptions 1, 3 hold, or 2) Assumptions 1, 2 hold, with ℓ F defined in Lemma 1. Then for x ∈ { x t } T t =1 , x ′ ∈ { x ′ t } T t =1 generated by MoDo algorithm on training dataset S and S ′ , respectively, it implies that

Proof. We provide proof leveraging the convergence properties of the projected gradient descent algorithm on strongly convex objectives below. Consider the problem min λ ∈ ∆ M g ( λ ; x, ρ ) = 1 2 ∥∇ F S ( x ) λ ∥ 2 + 1 2 ρ ∥ λ ∥ 2 , which is ρ -strongly convex. Let { λ ρ,k ( x ) } for k = 0 , 1 , . . . , K denote the sequence obtained from applying projected gradient descent (PGD) on the objective g ( λ ; x, ρ ) = 1 2 ∥∇ F S ( x ) λ ∥ 2 + 1 2 ρ ∥ λ ∥ 2 , i.e.,

where η is the step size that η < 1 / ( ∥∇ F S ( x ) ⊤ ∇ F S ( x ) ∥ + ρ ) . Note that both ρ, η are independent of K . By the convergence result of PGD on strongly convex objective, we know that λ ∗ ρ ( x ) is the limit point of { λ ρ,k ( x ) } ∞ k =0 . By the non-expansiveness of projection, we have

Since λ ρ, 0 ( x ) = λ ρ, 0 ( x ′ ) = λ ρ, 0 , apply the above inequality recursively from k = 0 , 1 , . . . , K -1 , we have

Then it follows that

where ( a ) follows from lim K →∞ 1 -( ηρ ) K = 1 , and from the convergence of PGD [2, Theorem 1.1] on ρ -strongly convex objectives that

This proves (C.7).

In addition, under Assumptions 1, 3, the above result directly implies that

While under Assumptions 1 and 2, and for ℓ F defined in Lemma 1, and for x ∈ { x t } T t =1 , x ′ ∈ { x ′ t } T t =1 generated by MoDo algorithm on training dataset S and S ′ , respectively, ∥∇ F S ( x ) ∥ ≤ ℓ F , ∥∇ F S ( x ′ ) ∥ ≤ ℓ F , which along with (C.12) implies that

The proof is complete.

Lemma 16. For any ρ > 0 and x ∈ R d , define λ ∗ ( x ) = arg min λ ∈ ∆ M ∥∇ F S ( x ) λ ∥ 2 , and λ ∗ ρ ( x ) = arg min λ ∈ ∆ M ∥∇ F S ( x ) λ ∥ 2 + ρ ∥ λ ∥ 2 , then we have

Proof. Since λ ∗ ( x ) = arg min λ ∈ ∆ M ∥∇ F S ( x ) λ ∥ 2 , therefore ∥∇ F S ( x ) λ ∗ ρ ( x ) ∥ 2 -∥∇ F S ( x ) λ ∗ ( x ) ∥ 2 = ∥∇ F S ( x ) λ ∗ ρ ( x ) ∥ 2 -min λ ∈ ∆ M ∥∇ F S ( x ) λ ∥ 2 ≥ 0 . (C.15)

Since λ ∗ ρ ( x ) = arg min λ ∈ ∆ M ∥∇ F S ( x ) λ ∥ 2 + ρ ∥ λ ∥ 2 , therefore

Rearranging the above inequality gives

Lemma 17. Consider { x t } , { λ t } generated by the MoDo algorithm. For all λ ∈ ∆ M , it holds that

Proof. By the update of λ , for all λ ∈ ∆ M , we have

Taking expectation over z t, 1 , z t, 2 on both sides and rearranging proves (C.18).

By the convexity of the problem, min λ ∈ ∆ M 1 2 ∥∇ F S ( x t ) λ ∥ 2 , we have

Rearranging the above inequality proves (C.19).

C.2 Proof of Lemma 2 - Distance to CA direction

Organization of proof. In Lemma 18, we prove the upper bound of the distance to CA direction, 1 T ∑ T -1 t =0 E A [ ∥∇ F S ( x t ) λ t -∇ F S ( x t ) λ ∗ ( x t ) ∥ 2 ] , in terms of two average of sequences, S 1 ,T , and S 2 ,T . Then under either Assumptions 1, 3, or Assumptions 1, 2, we prove the upper bound of S 1 ,T , and S 2 ,T , and thus the average-iterate distance to CA direction in Lemma 2.

Lemma 18. Suppose Assumption 1 holds. Let { x t } , { λ t } be the sequences produced by the MoDo algorithm. With a positive constant ¯ ρ > 0 , define

Then it holds that

Proof. Define λ ∗ ¯ ρ ( x t ) = arg min λ ∈ ∆ M 1 2 ∥∇ F S ( x t ) λ ∥ 2 + ¯ ρ 2 ∥ λ ∥ 2 with ¯ ρ > 0 . Note that ¯ ρ is strictly positive and is used only for analysis but not for algorithm update.

Substituting λ = λ ∗ ¯ ρ ( x t ) in Lemma 17, (C.19), we have

Setting γ t = γ > 0 , taking expectation and telescoping the above inequality gives

where I 1 can be further derived as

where ∥ λ ∗ ¯ ρ ( x t +1 ) -λ ∗ ¯ ρ ( x t ) ∥ , by Lemma 15, can be bounded by

Hence, it follows that

plugging which into (C.24) gives

where the last inequality follows from Lemma 16. The proof is complete.

Proof of Lemma 2. Building on the result in Lemma 18, and by the convexity of the subproblem, min λ ∈ ∆ M 1 2 ∥∇ F S ( x t ) λ ∥ 2 , and Lemma 14, we have

By Assumptions 1, 3 or Assumptions 1, 2 and Lemma 1, we have

Substituting S 1 ,T , S 2 ,T in (C.30) with the above bound yields

Because ℓ F, 1 ℓ F ≤ Mℓ f, 1 ℓ f , choosing ¯ ρ = 2( αMℓ f, 1 ℓ 2 f /γ ) 1 2 yields

where ( a ) follows from Lemma 14. This proves the result.

C.3 Proof of Theorem 3 - PS optimization error

Technical contributions. The optimization error bound in Theorem 3 is improved with either relaxed assumption or improved convergence rate compared to prior stochastic MOL algorithms [52, 10, 30] (see Table 2). This is achieved by 1) instead of bounding the approximation error to λ ∗ ( x t ) , we bound that to the CA direction d ( x t ) = -∇ F S ( x t ) λ ∗ ( x t ) as a whole, and 2) instead of using the descent lemma of F S ( x t ) λ ∗ ( x t ) with a dynamic weight, we use that of F S ( x t ) λ with a fixed weight (see Lemma 19, (C.36)), thereby improving the tightness of the bound.

Organization of proof. In Lemma 19, we prove the upper bound of the PS optimization error, 1 T ∑ T -1 t =0 E A ∥∇ F S ( x t ) λ ∗ t ( x t ) ∥ 2 , in terms of three average of sequences, S 1 ,T , S 3 ,T , and S 4 ,T . Then we prove the upper bound of S 1 ,T , S 3 ,T , and S 4 ,T , and thus the PS optimization error either in the nonconvex case under Assumptions 1, 3 or in the strongly convex case under Assumptions 1, 2. Combining the results leads to Theorem 3.

Lemma 19. Suppose Assumption 1 holds. Consider the sequence { x t } , { λ t } generated by MoDo in unbounded domain for x . Define

Then it holds that

Proof. By the ℓ f, 1 -Lipschitz smoothness of F S ( x ) λ for all λ ∈ ∆ M , we have

Taking expectation over z t, 3 on both sides of the above inequality gives

By Lemma 17, (C.18), we have

Rearranging the above inequality and letting γ t = γ > 0 gives

Plugging the above inequality into (C.37), and setting α t = α > 0 , we have

Taking telescope sum and rearranging yields, for all λ ∈ ∆ M ,

Setting λ = λ 1 in the above inequality yields

Finally, the results follow from the definition of λ ∗ t ( x t ) such that 1 T ∑ T -1 t =0 E A ∥∇ F S ( x t ) λ ∗ t ( x t ) ∥ 2 ≤ 1 T ∑ T -1 t =0 E A ∥∇ F S ( x t ) λ t ∥ 2 .

Proof of Theorem 3. Lemma 19 states that, under Assumption 1, we have

Then we proceed to bound S 1 ,T , S 3 ,T , S 4 ,T . Under either Assumptions 1, 3, or Assumptions 1, 2 with ℓ f , ℓ F defined in Lemma 1, we have that for all z ∈ S and λ ∈ ∆ M , ∥∇ F z ( x t ) λ ∥ ≤ ℓ f , and ∥∇ F z ( x t ) ∥ ≤ ℓ F . Then S 1 ,T , S 3 ,T , S 4 ,T can be bounded below

which proves that

We arrive at the results by 1 T ∑ T -1 t =0 E A ∥∇ F S ( x t ) λ ∗ t ( x t ) ∥ ≤ ( 1 T ∑ T -1 t =0 E A ∥∇ F S ( x t ) λ ∗ t ( x t ) ∥ 2 ) 1 2 from the Jensen's inequality and the convexity of the square function, as well as the subadditivity of square root function.

(a) MGDA

(b) Static

(c) MoDo

5

5

5

0

0

0

-5

5

5

2

-10

2

01-

2

-10

EmpiricalParetoFront

fs

fs

fs

InitialIterate

-15

-15

-15

-20

-20

-20

-25

-25

-25

30

30

30

-30 -25-20 -15 -10-505

-30 25 20 -15 -10 5

505

-30-25-20-15-10-505

fs.1

fs.1

fs.1

5

0

0

0

PopulationParetoFront

2

-5

-5

f2

-5

Initial Iterate

-10

-10

-10

15

15

15

-20

-20

-20

-20

-15

-10

-5

0

5

20

-15

-10

5

0

5

-20

-15

-10

5

0

5

f1

f1

f1

D Additional experiments and implementation details

Compute. Experiments are done on a machine with GPU NVIDIA RTX A5000. We use MATLAB R2021a for the synthetic experiments in strongly convex case, and Python 3.8, CUDA 11.7, Pytorch 1.8.0 for other experiments. Unless otherwise stated, all experiments are repeated with 5 random seeds. And their average performance and standard deviations are reported.

D.1 Synthetic experiments

D.1.1 Experiments on strongly convex objectives

Implementation details. Below we provide the details of experiments that generate Figure 3. We use the following synthetic example for the experiments in the strongly convex case. The m -th objective function with stochastic data sample z is specified as

where b 1 ,m > 0 for all m ∈ [ M ] , and b 2 ,m is another scalar. We set M = 3 , b 1 = [ b 1 , 1 ; b 1 , 2 ; b 1 , 3 ] = [1; 2; 1] , and b 2 = [ b 2 , 1 ; b 2 , 2 ; b 2 , 3 ] = [1; 3; 2] . The default parameters are T = 100 , α = 0 . 01 , γ = 0 . 001 . In other words, in Figure 3a, we fix α = 0 . 01 , γ = 0 . 001 , and vary T ; in Figure 3b, we fix T = 100 , γ = 0 . 001 , and vary α ; and in Figure 3c, we fix T = 100 , α = 0 . 01 , and vary γ .

D.1.2 Experiments on nonconvex objectives

Implementation details. The toy example is modified from [29] to consider stochastic data. Denote the model parameter as x = [ x 1 , x 2 ] ⊤ ∈ R 2 , stochastic data as z = [ z 1 , z 2 ] ⊤ ∈ R 2 sampled from the standard multi-variate Gaussian distribution. And the individual empirical objectives are defined as:

(a)

γ

= 0

(b)

γ

= 10

-

12

(c)

γ

= 10

-

4

(d)

γ

= 10

-

2

10

5

10

10

10

5

3

0

5

3

0

5

0

3

0

3

0

-10

5

0

5

-10

0

-10

5

0

-10

5

13

13

13

13

-5

15

15

15

15

10

10

-5

X1

0

5

10

-18

10

-10

5

X1

0

5

10

-18

10

10

-5

X1

0

5

10

-18

10

10

5

X1

0

5

10

18

Since z is zero-mean, the individual population objectives are correspondingly:

The training dataset size is n = | S | = 20 . For all methods, i.e., MGDA, static weighting, and MoDo, the number of iterations is T = 50000 . The initialization of λ is λ 0 = [0 . 5 , 0 . 5] ⊤ . The hyperparameters for this experiment are summarized in Table 5.

Figure 5, in addition to Figure 1, shows the trajectories of different methods from different initializations to the empirical and population Pareto fronts (PF). With the visualized empirical and population PFs, it is clear in Figure 5a, the first row, that the three trajectories of MGDA all converge to the empirical PF, but, it stops updating the model parameter as soon as it reaches the empirical PF. However, due to the difference between the empirical and population PFs caused by finite stochastic training data, as shown in Figure 5a, the second row, not all three solutions from MGDA has small population risk, implied from the distance of the solution (yellow point) to the population PF colored in green. For static weighting method with uniform weights in Figure 5b, one trajectory is able to converge to the center of the empirical PF, which is the optimal solution of the uniform average of the two objectives. However, the other two get stuck and oscillate around suboptimal parameters for a long time, corresponding to the clusters of scattered points in the figure. Nevertheless, in the second row of Figure 5b, one empirically suboptimal solution (on the trajectory with red to yellow colormap) is able to achieve small population risk. This example demonstrates that even though static weighting method does not have small distance to CA direction, it might still be able to achieve small testing error. Finally, for MoDo in Figure 5c, the first row shows that MoDo is slower than MGDA in convergence to the empirical PF, since it only approximately solves the CA direction using

(a)

γ

= 0

ou

(b)

γ

= 10

-

12

(c)

γ

= 10

-

4

(d)

γ

= 10

-

2

5

0

2

2

-10

2

-10

2

EmpiricalPareto Front

15

fs

-15

fs

15

fs

Initial Iterate

20

20

20

-20

25

25

25

25

30

-30

30

-30

S0S-01-S1-07-S7-05-

-30-25-20-15-10-505

30 25 20 15 10505

30 25 20 15 10 5

05

fs.1

fs,1

fs.1

fs.1

0

0

0

0

f2

-5

-5

-5

5

PopulationParetoFront

Initial Iterate

-10

-10

10

01-

15

15

15

15

20

20

20

20

20

-15

10

5

0

5

20

15

-10

5

0

5

20

15

-10

5

0

20

15

0I-

5

0

5

f1

f1

f1

f1

stochastic gradient. The second row shows that all three solutions of MoDo can achieve relatively small population risk, demonstrating a good generalization ability.

To demonstrate how the choice of γ has an impact on the performance of MoDo, we further conduct experiments with different γ , and show that MoDo can recover static weighting ( γ = 0 ) and approximate MGDA ( γ = 10 -2 ). Results are visualized in Figure 6, which plots the trajectories over the contours of the average objectives, and Figure 7, which plots the empirical and population PFs. Figure 6a and Figure 7a together verify that MoDo with γ = 0 behaves the same as static weighting, cf. Figure 1d and Figure 5b. Like static weighting, for two of the initializations, they either cannot go through the valley or stuck in the valley for a long time. And if the trajectory converges, it will converge to the optimal solution of the average empirical objectives. In addition, Figure 6d and Figure 7d together indicate that MoDo with γ = 10 -2 behaves similarly as MGDA, cf. Figure 1c and Figure 5a. Like MGDA, it can go through the valley without getting stuck in it for a long time, and then converge to the empirical PF for all three initializations. But also like MGDA, it will stop updating or oscillating around the parameters as soon as it reaches the empirical PF, resulting in a worse population risk for one of the solutions with a trajectory colored from purple to pink to yellow. We also conduct experiments on 0 < γ < 10 -2 . With γ = 10 -12 , as shown in Figures 6b, and 7b, MoDo is able to go through the valley and converge to the empirical PF for all initializations, but very slowly. And all initializations converge to the optimal solution of the average objectives. While with γ = 10 -4 , as shown in Figures 6c, and 7c, MoDo is able to go through the valley and converge to the empirical PF for all initializations quickly. And it stops updating the parameters as soon as it reaches the empirical PF. The solutions of the three initializations are different from the optimal solution of the average objectives. But compared to the case when γ = 10 -2 , the three solutions are closer to the optimal solution of the average objectives. This is because the weighting parameter λ does not change too much during the optimization procedure with a small γ .

D.2 Multi-task supervised learning experiments

In this section we present experiment details and additional results for comparison of static weighting, MGDA, and MoDo algorithms, under synthetic and real world multi-task supervised learning problems. We use MNIST, Office-31, Office-home, and NYU-v2 datasets.

D.2.1 MNIST dataset experiments

Implementation details. Below we provide the details and additional experimental results on MNIST image classification. We simulate a synthetic multi-objective optimization problem using

different loss functions applied for training an image classifier for MNIST handwritten digit dataset. We consider three loss functions: cross entropy, mean squared error (MSE), and Huber loss. The model architecture is a two-layer multi-layer perceptron (MLP). Each hidden layer has 512 neurons, and no hidden layer activation. The input size is 784, and the output size is 10, the number of digit classes. The training, validation, and testing data sizes are 50k, 10k, and 10k, respectively. Hyperparameters such as step sizes are chosen based on each algorithm's validation accuracy performance, as given in Table 6.

We then discuss the results of the experiments in Table 7, which shows the performance of the last iterate for each method. Experiments are repeated 10 times with average performance and standard deviations reported. Observed from Table 7, for cross-entropy loss, MGDA performs the worst while static weighting performs the best. On the other hand, for Huber loss, MGDA performs the best while static weighting performs the worst. This is not surprising as cross-entropy loss has the largest scale and Huber loss has the smallest scale among the three losses. Since equal weights are assigned for all three objectives in the static weighting method, it tends to focus more on optimizing the loss with the largest scale. While for MGDA, it is the other way around. Compared to MGDA, MoDo performs much better on cross-entropy loss, and in the meantime, it achieves comparable performance on Huber loss. Comparing their PS population risks R pop and the decomposed PS optimization errors R opt and PS generalization errors R gen , MGDA has the smallest PS population risk and PS optimization error. One potential reason is that MGDA performs best on Huber loss, with smaller gradients. Another reason is that the generalization errors for all the algorithms are similar and not dominating compared to optimization errors in this setting, making the PS population risk close to the PS optimization error. Overall, MoDo demonstrates a balance among objectives with different scales and performs reasonably well on all three objectives since it combines the properties of static weighting and MGDA.

D.2.2 Office-31 and Office-home dataset experiments

Implementation details. We give the details for the experiments conducted using Office-31 and Office-home datasets, which consist of multi-domain image classification tasks. Both of these are multi-input single-task learning problems. Office-31 and Office-home consist of 31 and 65 image classes, respectively. The image domains for Office-31 are; 'Amazon', which includes object images from Amazon, 'DSLR', which includes high-resolution images of objects, and 'Webcam', which includes low-resolution images from objects. The image domains for Office-31 are 'Art', 'Clipart', 'Product', and 'Real-world' which include images of objects taken from the respective image domains. We use the MTL benchmark framework LibMTL [26] to run experiments on both of the aforementioned datasets. For both datasets, we tune the step size of x and weight decay parameters for Static and MGDA algorithms and tune the step sizes of x and λ for the MoDo algorithm. We use batch size 64 to update static weighting and MGDA, and use 2 independent samples of batch size 32 to update MoDo, for both Office-31 and Office-home. A summary of hyper-parameters used for

Office-31 and Office-home for each algorithm are given in Table 8 and Table 9, respectively. All other experiment setup is shared for all algorithms, and the default LibMTL configuration is used.

The results of Office-31 and Office-home experiments are given in Tables 3 and 10, respectively (average over 5 seeds, the error indicates standard deviation). Here, we use the average per-task performance drop of metrics S A ,m for method A with respect to corresponding baseline measures S B ,m as a measure of the overall performance of a given method. Specifically, this measure is

where M is the number of tasks. Here, ℓ m =1 if higher values for S A ,m are better and 0 otherwise. We use the best results for each task obtained by dedicated independent task learners of each task as S B ,m . The independent task learners are tuned for the learning rate and weight decay parameter. For Office-31, S B ,m values are 87 . 50% for 'Amazon', 98 . 88% for 'DSLR', and 97 . 32% for 'Webcam'. For Office-home, S B ,m values are 66 . 98% for 'Art', 82 . 02% for 'Clipart', 91 . 53% for 'Product', and 80 . 97% for 'Real-world'. It can be seen from Tables 3 and 10 that static weighting outperforms MGDA method in some tasks and also in terms of ∆ A % . However, by proper choices of hyperparameters, MoDo performs on par or better compared to both static weighting and MGDA, and hence achieves the best overall performance in terms of ∆ A % .

D.2.3 NYU-v2 dataset experiments

Implementation details. We give the details for the experiments conducted using NYU-v2 dataset, which consists of image segmentation, depth estimation, and surface normal estimation tasks. Unlike Office-31 and Office-home datasets, this is a single-input multi-task learning problem. The

dataset consists of images from indoor video sequences. We use the MTL benchmark framework LibMTL [26] to run experiments with this dataset. We tune step size of x and weight decay parameters for static weighting and MGDA algorithms, and tune step sizes of x and λ for MoDo algorithm. We use batch size 4 to update static weighting and MGDA, and use 2 independent batches of size 2 to update MoDo. Experiments were run for 50 epochs for all methods. Since there was no validation set for NYU-v2, we averaged and reported the test performance of the last 10 epochs. A summary of hyper-parameters used for each algorithm is given in Table 11. All other experiment setup is shared for all algorithms, and the default LibMTL configuration is used.

The results of NYU-v2 experiments are given in Tables 12 (average over 3 seeds, the error indicates standard deviation). Again, we use the average per-task performance drop of metrics S A ,m for method A with respect to corresponding baseline measures S B ,m as a measure of the overall performance of a given method. We use the best results for each task obtained by dedicated independent task learners of each task as S B ,m . The independent task learners are tuned for the learning rate and weight decay parameter. For segmentation task, S B ,m values are 53 . 94% for 'mIoU', and 75 . 67% for 'Pix Acc'. For depth estimation task, S B ,m values are 0 . 3949 for 'Abs Err', and 0 . 1634 for 'Rel Err'. For surface normal estimation task, S B ,m values are 22 . 12 for 'Angle Distance - Mean', 15 . 49 for 'Angle Distance - Median', 38 . 35% for 'Within 11 . 25 ◦ ', 64 . 30% for 'Within 22 . 5 ◦ ', and 74 . 70% for 'Within 30 ◦ '. It can be seen from Table 12 that MoDo outperforms both MGDA and static weighting in some tasks and also in terms of ∆ A % . The worse performance of MGDA in terms of ∆ A % can be due to the large bias towards the Surface normal estimation task, which seems to affect other tasks adversely.

D.2.4 Additional experiments for comparison with other MOL baselines

In this section, we provide a comparison between MoDo and other popular MOL baselines. For this purpose, we use the same benchmark datasets as the previous section and use the experiment setup provided in [25] to run experiments with MoDo. Hence, we use experiment results provided by [25] for other baselines for comparison. Additionally, we implement MoCo [10], which is not included in [25]. For results in this section we report two holistic measures: ∆ A st , which measures performance degradation w.r.t. static scalarization (similar to [25], but lower the better), and ∆ A id , which measures performance degradation w.r.t. independent task learners (as defined in our paper)

The results on Office-31, Office-home, and NYU-v2 are given in Tables 13, 14, and 15, respectively, where MoDo outperforms all the baselines for most tasks, and has a better overall performance in ∆ A st % and ∆ A id % . The hyper-parameters of MoDo for the above experiments are in Table 16.

Peer reviews:
comments: SUMMARY:
This paper has two contributions:
(1) MoDo algorithm which is a variant of MGDA with a double sampling to obtain an unbiased stochastic estimate of the gradient problem.
(2) A solid theoretically analysis on the error of multi-objective optimization.

STRENGTHS:
This paper has a very detailed analysis on the optimization error and generalization error of multi-objective optimization.

WEAKNESSES:
In the experiments, authors only compare MoDo with MGDA, but there are many other algorithms, like CAGrad, GradNorm, Uncertainty Weight. This baseline is not enough to demonstrate the effectiveness of the proposed method.

Overall, MoDo is better than MGDA. However, in two objectives out of three (table 2), MGDA is better than MoDo, also not convincing enough on the effectiveness of the proposed method.

QUESTIONS:
Could you summarize and simply explain the advantages of MoDo over MGDA here, smaller error or more efficient?

Why MoDo has an unbiased gradient estimation? line 142

LIMITATIONS:
Authors only discuss the situation of using Pareto optimal to get the aggregated gradient.
However, there are other methods other than Pareto to balance multi-objective optimization like balancing gradient magnitudes like AdaTask, GradNorm.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the multi-objective learning problem. The classic idea of dynamic weighting in MOL is to take gradients from each objective and to weight them using a fixed procedure to avoid conflicts between different objectives. Empirically, however, there often seems to be performance degradation when using these methods. They discover that this is due to a tradeoff between optimization, generalization, and conflict avoidance. They propose a new algorithm, MoDo, that interpolates between static weighting and dynamic weighting and find parameters that can control this tradeoff effectively.\n\nSTRENGTHS:\nOverall, this is a strong work that provides a lot of insight into dynamic weighting in multi-objective optimization. The highlights are as follows:\n- The paper is very intuitively written and easy to follow. Specifically, the three-way tradeoff is clear both intuitively and quantitatively. In addition, Figures 1 and 2 are very well done and extremely insightful.\n- The proposed MoDo algorithm is very simple and intuitive while brilliantly highlighting the three-way tradeoff inherent in dynamic weighting algorithms. I found the remark about being in the early stopping regime for generalization error to diminish (T = o(n)) interesting, and it is also empirically highlighted later in the work.\n- The tradeoffs for both strongly convex and nonconvex cases are analyzed\n- The findings in this work translate to a practical explanation of the behavior of dynamic weighting algorithms\n\nWEAKNESSES:\n- While MoDo is a great theoretically inspired algorithm that controls the aforementioned tradeoffs, I am not sure how well it performs empirically. While there is an empirical result, the tasks seem rather simple. In addition, as Table 2 shows, MoDo does not consistently outperform MGDA. In practice, I feel handling the conflict avoidance tradeoff shouldn't matter much as long as the accuracy for the task is good. Therefore, I am not sure how applicable the algorithm would be in practical situations. \n- As the authors mention in their limitations section, the analysis is only about one specific algorithm in one specific setting, which makes it unclear how general the principles in this work would apply when the assumptions are relaxed. I feel this given this work is meant to analyze a practical phenomenon, this is a substantial weakness, as it is unlikely that Assumptions 1-3 are all true in real-world settings (such as the image classification setting the authors test on)\n- There is no intuition of the proof in the main text, and while this is common in optimization papers, it would be useful to learn about the key insights towards the proof.\n\nQUESTIONS:\n- Can you compare the theoretical tradeoffs in Table 1 with the empirical findings in Figure 4? How well do they match with each other?\n- How can knowledge of this three-way tradeoff be used to design better algorithms in multi-task learning in the future?\n- How can we tune the parameters in practice? How sensitive is the algorithm to the parameters?\n\nLIMITATIONS:\nGreat!", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies three-way trade-off in multi-objective learning: 1) optimization error caused by sampling and stochastic training; 2) generalization error that measures the difference between source and target sets; 3) conflict-avoidance direction error that is the bias between the calculated direction and the right one. The authors propose the MoDo algorithm to optimize this trade-off in one algorithm. Plenty of theoretical analysis supports the claims.\n\nSTRENGTHS:\n1. This paper considers the multi-objective algorithmic design in a bigger picture by compositing three-way trade-off together, and provides solid analysis to solve it.\n\n2. This paper is well written. In particular, Table 1 and Figure 2 help to understand the contribution better.\n\nWEAKNESSES:\n1. The proposed algorithm needs to compute three batches of gradients to run an iteration, which is much less computationally efficient. \n\n2. Benchmarks are too limited that only contain static and MGDA (2018 proposed), experiments have not compared with other typical methods like PCGrad and CAGrad, which have better empirical performance. \n\n3. Several claims are out-of-date. In 34-35 \"Unfortunately, the reason behind this empirical performance\ndegradation is not fully understood and remains an open question\" is not accurate, since [1] has proved that the vanilla MGDA algorithm and also PCGrad and CAGrad will not converge to Pareto optimal. In 268 \"This can be overcome by increasing the batch size during optimization [25]\" is wrong, because the assumption on Lipschitzness for $\\lambda$ with respect to gradients has been proved to be wrong (Proposition 2 in [1]). Also, it can be proved that even a stochastic error of the gradient can cause a significant bias in the direction, so only increasing the batch size does not work.\n\n[1] S Zhou, W Zhang, J Jiang, W Zhong, J Gu, W Zhu - Advances in Neural Information Processing Systems, 2022\n\nQUESTIONS:\n1. Why use double sampling? Can MoCo completely solve the problem? Or is it possible to reduce the sample complexity?\n\n2. Does the proposed theory can help improve PCGrad and CAGrad?\n\nLIMITATIONS:\nSample efficiency of the proposed algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the multi-objective optimization, and in particular, focus on the generalization and stability analysis. By decomposing the Pareto stationarity error into the generalization and optimization error, the authors then analyze and upper-bound these two errors respectively. The distance to the conflict-avoidant direction, which optimizes al objectives jointly, is also analyzed. A stochastic variant of MGDA named MoDo has been developed and analyzed. Based on the theoretical derivations, the authors find a three-way tradeoff among the optimization error, generalization error and the CA error. Some implications on the parameter selection and illustrations are provided.\n\nSTRENGTHS:\n1. The studied topic on multi-objective optimization has received increasing attention thanks to the important applications like multi-task learning. Studying the generalization, stability and the tradeoff is important and under-explored.\n\n2. Although generalization has been studied for multi-objective optimization, stability has not been explored. This work seems to be the first one to fill this gap.\n\n3. The tradeoff among optimization and generalization is not surprising, but their tradeoffs with the CA direction error seem to be new given the multi-objective structure.\n\nWEAKNESSES:\n1. How is (4a) equivalent to (4b)? Can the authors provide some more details?\n\n2. Before (4a), it says that CA direction maximize the minimum descent of all objectives. But (4a) seems to say that the direction minimizes something. \n\n3. The upper bounds in the nonconvex case may not be very tight given the large exponential dependence, and hence may not be able to exactly capture the generalization and tradeoff behaviors. Is it possible to provide a lower bound in this case? Or what challenges in getting a tight bound? \n\n4. The analysis uses Frobenius norm, which may be large in practice. Instead, spectrum norm may be more proper. Can the authors comment on why use Frobenius norm rather than spectrum norm?\n\nOverall, I appreciate the studied problem and the analysis in this work, but I am also open to other reviewers’ comments.\n\nQUESTIONS:\nSee Weakness part.\n\nLIMITATIONS:\nSee Weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}