Title: Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-objective optimization

multi-task leaning

stochastic algorithms

convergence and complexity

Pareto stationarity
Abstract: Multi-objective optimization (MOO) has become an influential framework in many machine learning problems with multiple objectives such as learning with multiple criteria and multi-task learning (MTL). In this paper, we propose a new direction-oriented multi-objective formulation by regularizing the common descent direction within a neighborhood of a direction that optimizes a linear combination of objectives such as the average loss in MTL or a weighted loss that places higher emphasis on some tasks than the others. This formulation includes GD and MGDA as special cases, enjoys the direction-oriented benefit as in CAGrad, and facilitates the design of stochastic algorithms. To solve this problem, we propose Stochastic Direction-oriented Multi-objective Gradient descent (SDMGrad) with simple SGD type of updates, and its variant SDMGrad-OS with an efficient objective sampling. We develop a comprehensive convergence analysis for the proposed methods with different loop sizes and regularization coefficients. We show that both SDMGrad and SDMGrad-OS achieve improved sample complexities to find an $\epsilon$-accurate Pareto stationary point while achieving a small $\epsilon$-level distance toward a conflict-avoidant (CA) direction. For a constant-level CA distance, their sample complexities match the best known $\mathcal{O}(\epsilon^{-2})$ without bounded function value assumption. Extensive experiments show that our methods achieve competitive or improved performance compared to existing gradient manipulation approaches in a series of tasks on multi-task supervised learning and reinforcement learning. Code is available at https://github.com/ml-opt-lab/sdmgrad.

Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithms

Abstract

Multi-objective optimization (MOO) has become an influential framework in many machine learning problems with multiple objectives such as learning with multiple criteria and multi-task learning (MTL). In this paper, we propose a new direction-oriented multi-objective formulation by regularizing the common descent direction within a neighborhood of a direction that optimizes a linear combination of objectives such as the average loss in MTL or a weighted loss that places higher emphasis on some tasks than the others. This formulation includes GD and MGDA as special cases, enjoys the direction-oriented benefit as in CAGrad, and facilitates the design of stochastic algorithms. To solve this problem, we propose Stochastic Direction-oriented Multi-objective Gradient descent (SDMGrad) with simple SGD type of updates, and its variant SDMGrad-OS with an efficient objective sampling. We develop a comprehensive convergence analysis for the proposed methods with different loop sizes and regularization coefficients. We show that both SDMGrad and SDMGradOS achieve improved sample complexities to find an ϵ -accurate Pareto stationary point while achieving a small ϵ -level distance toward a conflict-avoidant (CA) direction. For a constant-level CA distance, their sample complexities match the best known O ( ϵ -2 ) without bounded function value assumption. Extensive experiments show that our methods achieve competitive or improved performance compared to existing gradient manipulation approaches in a series of tasks on multi-task supervised learning and reinforcement learning. Code is available at https://github.com/ml-opt-lab/sdmgrad .

1 Introduction

In recent years, multi-objective optimization (MOO) has drawn intensive attention in a wide range of applications such as online advertising [1], hydrocarbon production [2], autonomous driving [3], safe reinforcement learning [4], etc. In this paper, we focus on the stochastic MOO problem, which takes the formulation of

where m is the parameter dimension, K ≥ 2 is the number of objectives and L i ( θ ) := E ξ L i ( θ ; ξ ) . One important example of MOO in eq. (1) is the multi-task learning (MTL) [5, 6], whose objective

function is often regarded as the average loss over K objectives.

where L i ( θ ) is the loss function for task i . However, solving the MOO problem is challenging because it can rarely find a common parameter θ that minimizes all individual objective functions simultaneously. As a result, a widely-adopted target is to find the Pareto stationary point at which there is no common descent direction for all objective functions. In this context, a variety of gradient manipulation methods have been proposed, including the multiple gradient descent algorithm (MGDA) [7], PCGrad [8], CAGrad [9] and Nash-MTL [10]. Among them, MGDA updates the model parameter θ using a time-varying multi-gradient, which is a convex combination of gradients for all objectives. CAGrad further improves MGDA by imposing a new constraint on the difference between the common descent direction and the average gradient to ensure convergence to the minimum of the average loss of MTL. However, these approaches mainly focus on the deterministic case, and their stochastic versions still remain under-explored.

Stochastic MOO has not been well understood except for several attempts recently. Inspired by MGDA, [11] proposed stochastic multi-gradient (SMG), and established its convergence with convex objectives. However, their analysis requires the batch size to increase linearly. In the more practical nonconvex setting, [12] proposed a correlation-reduced stochastic multi-objective gradient manipulation (CR-MOGM) to address the non-convergence issue of MGDA, CAGrad and PCGrad in the stochastic setting. However, their analysis requires a restrictive assumption on the bounded function value. Toward this end, [13] recently proposed a method named MoCo as a stochastic counterpart of MGDA by introducing a tracking variable to approximate the stochastic gradient. However, their analysis requires that the number T of iterations is big at an order of K 10 , where K is the number of objectives, and hence may be unsuitable for the scenario with many objectives. Thus, it is highly demanding but still challenging to develop efficient stochastic MOO methods with guaranteed convergence in the nonconvex setting under mild assumptions.

1.1 Our Contributions

Motivated by the limitations of existing methods, we propose Stochastic Direction-oriented Multiobjective Gradient descent (SDMGrad), which is easy to implement, flexible with a new directionoriented objective formulation, and achieves a better convergence performance under milder assumptions. Our specific contributions are summarized as follows.

New direction-oriented formulation. We propose a new MOO objective (see eq. (6)) by regularizing the common descent direction d within a neighborhood of a direction d 0 that optimizes a

linear combination L 0 ( θ ) of objectives such as the average loss in MTL or a weighted loss that places higher emphasis on some tasks than the others. This formulation is general to include gradient descent (GD) and MGDA as special cases and takes a similar direction-oriented spirit as in CAGrad [9]. However, different from the objective in CAGrad [9] that enforces the gap between d and d 0 to be small via constraint, our regularized objective is easier to design near-unbiased multigradient, which is crucial in developing provably convergent stochastic MOO algorithms.

New stochastic MOO algorithms. The proposed SDMGrad is simple with efficient stochastic gradient descent (SGD) type of updates on the weights of individual objectives in the multi-gradient and on the model parameters with (near-)unbiased (multi-)gradient approximations at each iteration. SDMGrad does not require either the linearly-growing batch sizes as in SMG or the extra tracking variable for gradient estimation as in MoCo. Moreover, we also propose SDMGrad-OS (which refers to SDMGrad with objective sampling) as an efficient and scalable counterpart of SDMGrad in the setting with a large number of objectives. SDMGrad-OS samples a subset of data points and objectives simultaneously in all updates and is particularly suitable in large-scale MTL scenarios.

Convergence analysis and improved complexity. We provide a comprehensive convergence analysis of SDMGrad for stochastic MOO with smooth nonconvex objective functions. For a constantlevel regularization parameter λ (which covers the MGDA case), when an ϵ -level conflict-avoidant (CA) distance (see Definition 2) is required, the sample complexities (i.e., the number of samples to achieve an ϵ -accurate Pareto stationary point) of the proposed SDMGrad and SDMGrad-OS improve those of MoCo [13] and MoDo [14] by an order of ϵ -4 and ϵ -2 (see Table 1), respectively. In addition, when there is no requirement on CA distance, the sample complexities of SDMGrad and SDMGrad-OS are improved to O ( ϵ -2 ) , matching the existing best result in the concurrent work [14]. For an increasing λ , we also show this convergent point reduces to a stationary point of the linear combination L 0 ( θ ) of objectives.

Promising empirical performance. We conduct extensive experiments in multi-task supervised learning and reinforcement learning on multiple datasets and show that SDMGrad can achieve competitive or improved performance compared to existing state-of-the-art gradient manipulation methods such as MGDA, PCGrad, GradDrop, CAGrad, IMTL-G, MoCo, MoDo, Nash-MTL and FAMO, and can strike a better performance balance on different tasks. SDMGrad-OS also exhibits a much better efficiency than SDMGrad in the setting with a large number of tasks due to the efficient objective sampling.

2 Related Works

Multi-task learning. One important application of MOO is MTL, whose target is to learn multiple tasks with possible correlation simultaneously. Due to this capability, MTL has received significant attention in various applications in computer vision, natural language process, robotics, and reinforcement learning [5, 15, 16, 17]. A group of studies have focused on how to design better MTL model architectures. For example, [18, 19, 20] enhance the MTL models by introducing task-specific modules, an attention mechanism, and different activation functions for different tasks, respectively. Another line of research aims to learn a bunch of smaller models of local tasks split from the original problem, which are then aggregated into a single model via knowledge distillation [21]. Recent several works [22, 23] have explored the connection between MTL and gradient-based meta-learning. [24] and [25] highlight the importance of task grouping and unrelated tasks in MTL, respectively. This paper focuses on a single model by learning multiple tasks simultaneously with novel modelagnostic SDMGrad and SDMGrad-OS methods.

Gradient-based MOO. Various gradient manipulation methods have been developed to learn multiple tasks simultaneously. One popular class of approaches re-weight different objectives based on uncertainty [26], gradient norm [27], and training difficulty [28]. MOO-based approaches have received more attention due to their principled designs and training stability. For example, [6] viewed MTL as a MOO problem and proposed an MGDA-type method for optimization. A class of approaches has been proposed to address the gradient conflict problem. Among them, [8] proposed PCGrad by projecting the gradient direction of each task on the norm plane of other tasks. GradDrop randomly dropped out highly conflicted gradients [29], and RotoGrad rotated task gradients to alleviate the conflict [30]. [9] proposed CAGrad by constraining the common direction direction within a local region around the average gradient. These approaches mainly focus on the determin-

istic setting. In the stochastic case, [13] proposed MoCo as a stochastic counterpart of MGDA, and provided a comprehensive convergence and complexity analysis. More recently, a concurrent work [14] analyzed a three-way trade-off among optimization, generalization, and conflict avoidance, providing an impact on designing the MOO algorithm. In addition, [31] found that scalarization SGD could be incapable of fully exploring the Pareto front compared with MGDA-variant methods. In this paper, we propose a new stochastic MOO method named SDMGrad, which benefits from a direction-oriented regularization and an improved convergence and complexity performance.

Concurrent work. A concurrent work [14] proposed a multi-objective approach named MoDo, which uses a similar double sampling strategy (see Section 4.2). However, there are still several differences between this work and [14]. First, our method benefits from a direction-oriented mechanism and is general to include MGDA and CAGrad as special cases, whereas MoDo [14] does not have such features. Second, MoDo takes a single-loop structure, whereas our SDMGrad features a double-loop scheme with an improved sample complexity. Third, [14] focuses more on the trade-off among optimization, generalization and conflict-avoidance, whereas our work focues more on the optimization efficiency and performance balance in theory and in experiments.

3 Preliminaries

3.1 Pareto Stationarity in MOO

Differently from single-objective optimization, MOO aims to find points at which all objectives cannot be further optimized. Consider two points θ and θ ′ . It is claimed that θ dominates θ ′ if L i ( θ ) ≤ L i ( θ ′ ) for all i ∈ [ K ] and L ( θ ) = L ( θ ′ ) . In the general nonconvex setting, MOO aims to find a Pareto stationary point θ , at which there is no common descent direction for all objectives. In other words, we say θ is a Pareto stationary point if range ( ∇ L ( θ )) ∩ ( -R K ++ ) = ∅ where R K ++ is the positive orthant cone.

̸

3.2 MGDAand Its Stochastic Variants

Deterministic MGDA. The deterministic MGDA algorithm was first studied by [7], which updates the model parameter θ along a multi-gradient d = ∑ K i =1 w ∗ i g i ( θ ) = G ( θ ) w ∗ , where G ( θ ) = ( g 1 ( θ ) , g 2 ( θ ) , ..., g K ( θ )) are the gradients of different objectives, and w ∗ := ( w ∗ 1 , w ∗ 2 , ..., w ∗ K ) T are the weights of different objectives obtained via solving the following problem.

where W is the probability simplex. The deterministic MGDA and its variants such as PCGrad and CAGrad have been well studied, but their stochastic counterparts have not been understood well.

Stochastic MOO algorithms. SMG [11] is the first stochastic variant of MGDA by replacing the full gradient G ( θ ) in eq. (3) by its stochastic gradient G ( θ ; ξ ) := ( g 1 ( θ ; ξ ) , ..., g K ( θ ; ξ )) , and updates the model parameters θ along the direction given by

However, this direct replacement can introduce biased multi-gradient estimation, and hence SMG required to increase the batch sizes linearly with the iteration number. To address this limitation, [13] proposed MoCo by introducing an additional tracking variable y t,i as the stochastic estimate of the gradient ∇ L i ( θ ) , which is iteratively updated via

where β t is the step size, Π Y i denotes the projection on a bounded set Y i = { y ∈ R m |∥ y ∥ ≤ C y,i } for some constant C y,i , and h t,i denotes stochastic estimator of ∇ L i ( θ ) at t-th iteration. Then, MoCo was shown to achieve an asymptotically unbiased multi-gradient, but with a relatively strong assumption that the number T of iterations is much larger than the number K of objectives. Thus, it is important but still challenging to develop provable and easy-to-implement stochastic MOO algorithms with mild assumptions.

4 Our Method

We first provide a new direction-oriented MOO problem, and then introduce a new stochastic MOO algorithm named SDMGrad and its variant SDMGrad-OS with objective sampling.

4.1 Direction-oriented Multi-objective Optimization

MOOgenerally targets at finding a direction d to maximize the minimum decrease across all objectives via solving the following problem.

where the first-order Taylor approximation of L i is applied at θ with a small stepsize α .

In some scenarios, the target is to not only find the above common descent direction but also optimize a specific objective that is often a linear combination L 0 ( θ ) = ∑ i ˜ w i L i ( θ ) for some ˜ w ∈ W of objectives. For instance, MTL often takes the averaged loss over tasks as the objective function, and every element in ˜ w will be set as 1 K . In addition, it is quite possible that there is a preference for tasks. In this case, motivated by the framework in [32], we can regard ˜ w as a preference vector and tend to approach a preferred stationary point along this direction. To address this problem, we propose the following multi-objective problem formulation by adding an inner-product regularization λ ⟨ g 0 , d ⟩ in eq. (5) such that the common descent direction d stays not far away from a target direction g 0 = ∑ i ˜ w i g i .

In the above eq. (6), the term -1 2 ∥ d ∥ 2 is used to regularize the magnitude of the direction d and the constant λ controls the distance between the update vector d and the target direction g 0 .

Compared to CAGrad. We note that CAGrad also takes a direction-oriented objective but uses a different constraint-enforced formulation as follows.

where h 0 is the average gradient and c ∈ [0 , 1) is a constant. To optimize eq. (7), CAGrad involves the evaluations of the product ∥ h 0 ∥∥ g w ∥ and the relation ∥ h 0 ∥ ∥ g w ∥ with g w := ∑ i w i g i , both of which complicate the designs of unbiased stochastic gradient/multi-gradient in the w and θ updates. As a comparison, our formulation in eq. (6), as shown later, admits very simple and provable stochastic algorithmic designs, while still enjoying the direction-oriented benefit as in CAGrad.

To efficiently solve the problem in eq. (6), we then substitute the relation that min i ∈ [ K ] ⟨ g i , d ⟩ = min w ∈W ⟨ ∑ K i =1 g i w i , d ⟩ = min w ∈W ⟨ g w , d ⟩ into eq. (6), and obtain an equivalent problem as

By switching min and max in the above problem, which does not change the solution due to the concavity in d and the convexity in w , we finally aim to solve min w ∈W max d ∈ R m ⟨ g w + λg 0 , d ⟩ -1 2 ∥ d ∥ 2 , where the solution to the min problem on w is

and the solution to the max problem is d ∗ = g w ∗ λ + λg 0 .

Connection with MGDA and GD. It can be seen from eq. (8) that the updating direction d ∗ reduces to that of MGDA whenwe set λ = 0 , and is consistent with that of GD for λ large enough. This consistency is also validated empirically in Appendix A.2 by varying λ .

4.2 Proposed SDMGrad Algorithm

The natural idea to solve the problem in eq. (8) is to use a simple SGD-type method. The detailed steps are provided in algorithm 1. At each iteration t , we run S steps of projected SGD with warmstart initialization and with a double-sampling-based stochastic gradient estimator, which is unbiased by noting that

where ξ, ξ ′ are sampled data and g 0 ( θ t ) = G ( θ t ) ˜ w denotes the orientated direction. After obtaining the estimate w t,S , SDMGrad updates the model parameters θ t based on the stochastic counterpart G ( θ t ; ζ ) w t,S + λg 0 ( θ t ; ζ ) of the direction d ∗ = g w ∗ λ + λg 0 with a stepsize of α t . It can be seen that SDMGrad is simple to implement with efficient SGD type of updates on both w and θ without introducing other auxiliary variables.

4.3 SDMGrad with Objective Sampling (SDMGrad-OS)

Another advantage of SDMGrad is its simple extension via objective sampling to the more practical setting with a large of objectives, e.g., in large-scale MTL. In this setting, we propose SDMGradOS by replacing the gradient matrix G ( θ t ; ξ ) in Algorithm 1 by a matrix H ( θ t ; ξ ) with randomly sampled columns, which takes the form of

where h i ( θ t ; ξ ) = g i ( θ t ; ξ ) with a probability of n/K or 0 otherwise, S corresponds to the randomness by objective sampling, and n is the expected number of sampled objectives. Then, the stochastic gradient in the w update is adapted to

where h 0 ( · ) = H ( · ) ˜ w . Similarly, the updating direction d = K n H ( θ t ; ζ, ˜ S ) w t,S + K n λh 0 ( θ t ; ζ, ˜ S ) . In the practical implementation, we first use np.random.binomial to sample a 0 -1 sequence s following a Bernoulli distribution with length K and probability n/K , and then compute the gradient g i ( θ ; ξ ) of G ( θ ; ξ ) only if the i th entry of s equals to 1 . This sampling strategy greatly speeds up the training with a large number of objectives, as shown by the reinforcement learning experiments in Section 6.3.

5 Main results

5.1 Definitions and Assumptions

We first make some standard definitions and assumptions, as also adopted by existing MOO studies in [11, 12, 13]. Since we focus on the practical setting where the objectives are nonconvex, algorithms are often expected to find an ϵ -accurate Pareto stationary point, as defined below.

Definition 1. We say θ ∈ R m is an ϵ -accurate Pareto stationary point if E [ min w ∈W ∥ g w ( θ ) ∥ 2 ] ≤ ϵ where W is the probability simplex.

MGDA-variant methods find a direction d ( θ ) (e.g., d ∗ = g w ∗ λ + λg 0 with w ∗ λ given by eq. (8)) that tends to optimize all objective functions simultaneously, which is called a conflict-avoidant (CA)

direction [14]. Thus, it is important to measure the distance between a stochastic direction estimate ̂ d ( θ ) and the CA direction, which we call as CA distance, as defined below.

Definition 2. ∥ E ̂ d ( θ ) [ ̂ d ( θ )] -d ( θ ) ∥ denotes the CA distance.

The following assumption imposes the Lipschitz continuity on the objectives and their gradients.

Assumption 1. For every task i ∈ [ K ] , L i ( θ ) is l i -Lipschitz continuous and ∇ L i ( θ ) is l i, 1 -Lipschitz continuous for any θ ∈ R m .

We next make an assumption on the bias and variance of the stochastic gradient g i ( θ ; ξ ) .

Assumption 2. For every task i ∈ [ K ] , the gradient g i ( θ ; ξ ) is the unbiased estimate of g i ( θ ) . Moreover, the gradient variance is bounded by E ξ [ ∥ g i ( θ ; ξ ) -g i ( θ ) ∥ 2 ] ≤ σ 2 i .

Assumption 3. Assume there exists a constant C g > 0 such that ∥ G ( θ ) ∥ ≤ C g .

The bounded gradient condition in Assumption 3 is necessary to ensure the boundedness of the multi-gradient estimation error, as also adopted by [12, 13].

5.2 Convergence Analysis with Nonconvex Objectives

We first upper-bound the CA distance for our proposed method.

Proposition 1. Suppose Assumptions 1-3 are satisfied. If we choose β t,s = c/ √ s and S > 1 , then

where C 1 = O ( K (1 + λ )) and c is a constant.

Proposition 1 shows that CA distance is decreasing with the number S of iterations on w updates. Then, by selecting a properly large S , we obtain the following general convergence result.

Theorem 1 (SDMGrad) . Suppose Assumption 1-3 are satisfied. Set α t = α = Θ((1 + λ ) -1 K -1 2 T -1 2 ) , β t,s = c/ √ s , where c is a constant, and S = Θ((1 + λ ) -2 T 2 ) . Then the outputs of the proposed SDMGrad algorithm satisfy

where ˜ O omits the order of log T .

Theorem 1 establishes a general convergence guarantee for SDMGrad along the multi-gradient direction d ∗ = g w ∗ λ + λg 0 . Building on Theorem 1, we next show that with different choices of the regularization parameter λ , two types of convergence results can be obtained in the following two corollaries.

Corollary 1 (Constant λ ) . Under the same setting as in Theorem 1, choosing a constant-level λ > 0 , we have 1 T ∑ T -1 t =0 E [ ∥ G ( θ t ) w ∗ t ∥ 2 ] = ˜ O ( K 1 2 T -1 2 ) , where w ∗ t ∈ arg min w ∈W 1 2 ∥ G ( θ t ) w ∥ 2 . To achieve an ϵ -accurate Pareto stationary point, each objective requires O ( K 3 ϵ -6 ) samples in ξ ( ξ ′ ) and O ( Kϵ -2 ) samples in ζ , respectively. Meanwhile, the CA distance takes the order of ˜ O ( ϵ ) .

Corollary 1 covers the MGDA case when λ = 0 . In this setting, the sample complexity O ( ϵ -6 ) improves those of MoCo [13] and MoDo [14] by an order of ϵ -4 and ϵ -2 , respectively, while achieving an ϵ -level CA distance.

Corollary 2 (Increasing λ ) . Under the same setting as in Theorem 1 and choosing λ = Θ( T 1 2 ) , we have 1 T ∑ T -1 t =0 E [ ∥ g 0 ( θ t ) ∥ 2 ] = O ( K 1 2 T -1 2 ) . To achieve an ϵ -accurate stationary point, each objective requires O ( K 2 ϵ -4 ) samples in ξ ( ξ ′ ) and O ( Kϵ -2 ) samples in ζ , respectively. Meanwhile, the CA distance takes the order of ˜ O ( √ K ) .

Corollary 2 analyzes the case with an increasing λ = Θ( T 1 2 ) . We show that SDMGrad converges to a stationary point of the objective L 0 ( θ ) = ∑ i ˜ w i L i ( θ ) with an improved sample complexity, but with a worse constant-level CA distance. This justifies the flexibility of our framework that the λ can balance the worst local improvement of individual objectives and the target objective L 0 ( θ ) .

Convergence under objective sampling. We provide a convergence analysis for SDMGrad-OS.

Theorem 2 (SDMGrad-OS) . Suppose Assumption 1-3 are satisfied. Define γ = K n , α t = α = Θ((1+ λ 2 ) -1 2 γ -1 2 K -1 2 T -1 2 ) , β t,s = c/ √ s where c is a constant, and S = Θ((1+ λ 2 ) -2 γ -2 T 2 ) . Then, by choosing a constant λ , the iterates of the proposed SDMGrad-OS algorithm satisfy

Theorem 2 establishes the convergence guarantee for our SDMGrad-OS algorithm, which achieves a per-objective sample complexity of O ( ϵ -6 ) comparable to that of SDMGrad, but with a much better efficiency due to the objective sampling, as also validated by the empirical comparison in Table 4.

5.3 Lower sample complexity but constant-level CA distance

Without the requirement on the ϵ -level CA distance, we further improve the sample complexity of our method to O ( ϵ -2 ) , as shown in the following theorem.

Theorem 3. Suppose Assumptions 1-3 are satisfied. Set S = 1 , α t = α = Θ( K -1 2 T -1 2 ) , β t = β = Θ( K -1 T -1 2 ) and λ as constant. The iterates of the proposed SDMGrad satisfy

Theorem 3 shows that to achieve an ϵ -accurate Pareto stationary point, our method requires T = O ( K 2 ϵ -2 ) . In this case, each objective requires a number O ( K 2 ϵ -2 ) of samples in ξ ( ξ ′ ) and ζ .

Convergence under objective sampling. We next analyze the convergence of SDMGrad-OS.

Theorem 4. Suppose Assumptions 1-3 are satisfied. Set S = 1 , γ = K n , α t = α = Θ( K -1 2 γ -1 2 T -1 2 ) , β t = β = Θ( K -1 γ -1 T -1 2 ) and λ as a constant. The iterates of the proposed SDMGrad-OS algorithm satisfy

Theorem 4 shows that to achieve an ϵ -accurate Pareto stationary point, our algorithm requires T = O ( γ 2 K 2 ϵ -2 ) , and each objective requires a number O ( γ 2 K 2 ϵ -2 ) of samples in ξ ( ξ ′ ) and ζ .

6 Experiments

In this section, we first describe the implementation details of our proposed methods. Then, we demonstrate the effectiveness of the methods under a couple of multi-task supervised learning and reinforcement settings. The experimental details and more empirical results such as the twoobjective toy example, consistency with GD and MGDA, and ablation studies over λ can be found in the Appendix A.

6.1 Practical Implementation

Double sampling. In supervised learning, double sampling (i.e., drawing two samples simultaneously for gradient estimation) is employed, whereas in reinforcement learning, single sampling is used because double sampling requires to visit the entire episode twice a time, which is much more time-consuming.

Gradient normalization and rescale. During the training process, the gradient norms of tasks may change over time. Thus, directly solving the objective in eq. (8) may trigger numerical problems.

Inspired by CAGrad [9], we normalize the gradient of each task and rescale the final update d by multiplying a factor of 1 1+ λ to stabilize the training.

Projected gradient descent. The computation of the projection to the probability simplex we use is the Euclidean projection proposed by [33], which involves solving a convex problem via quadratic programming. The implementation used in our experiments follows the repository in [34], which is very efficient in practice.

6.2 Supervised Learning

For the supervised learning setting, we evaluate the performance on the Cityscapes [39] and NYUv2 [40] datasets. The former dataset involves 2 pixel-wise tasks: 7-class semantic segmentation and depth estimation, and the latter one involves 3 pixel-wise tasks: 13-class semantic segmentation, depth estimation and surface normal estimation. Following the experimental setup of [9], we embed a MTL method MTAN [36] into our SDMGrad method, which builds on SegNet [41] and is empowered by a task-specific attention mechanism. We compare SDMGrad with Linear Scalarization (LS) which minimizes the average loss, Scale-invariant (SI) which minimizes the average logarithmic loss, RLW [35], DWA [36], UW [26], MGDA [7], PCGrad [8], GradDrop [29], CAGrad [9], IMTL-G [37], MoCo [13], MoDo [14], Nash-MTL [10], and FAMO [38]. Following [42, 9, 13, 10, 38], we compute two metrics reflecting the overall performance: (1) ∆m % , the average per-task performance drop versus the single-task (STL) baseline b to assess method m : ∆ m % = 1 K ∑ K k =1 ( -1) l k ( M m,k -M b,k ) /M b,k × 100 , where K is the number of metrics, M b,k is the value of metric M k obtained by baseline b , and M m,k obtained by the compared method m . l k = 1 if the evaluation metric M k on task k prefers a higher value and 0 otherwise. (2) Mean Rank (MR) : the average rank of each method across all tasks.

We search the hyperparameter λ ∈ { 0 . 1 , 0 . 2 , · · · , 1 . 0 } for our SDMGrad method and report the results in Table 2 and Table 3. Each experiment is repeated 3 times with different random seeds and the average is reported. It can be seen that our proposed method is able to obtain better or comparable results than the baselines, and in addition, can strike a better performance balance on multiple tasks than other baselines. For example, although MGDA achieves better results on Surface Normal, it performs the worst on both Segmentation and Depth. As a comparison, our SDMGrad method can achieve more balanced results on all tasks.

6.3 Reinforcement Learning

For the reinforcement learning setting, we evaluate the performance on the MT10 benchmarks, which include 10 robot manipulation tasks under the Meta-World environment [43]. Following the experiment setup in [9, 13, 10], we adopt Soft Actor-Critic (SAC) [44] as the underlying training algorithm. We compare SDMGrad with Multi-task SAC [43], Multi-headed SAC [43], Multi-task SAC + Task Encoder [43], PCGrad [8], CAGrad [9], MoCo [13], Nash-MTL [10] and FAMO [38]. We search λ ∈ { 0 . 1 , 0 . 2 , · · · , 1 . 0 } and provide the success rate and average training time (in sec-

onds) per episode in Table 4. As shown, SDMGrad achieves the second best success rate among the compared baselines.

Wealso validate the efficiency of the proposed objective sampling strategy against other acceleration strategies including CAGrad-Fast [9], Nash-MTL with updating once per {50, 100} iterations [38]. Following [9], we choose the task sampling size n = 4 . As shown in Table 4, our SDMGrad-OS with objective sampling achieves approximately 1.4 × speedup on MT10, while achieving a success rate comparable to that of SDMGrad. We also observe that although our SDMGrad-OS requires more time than CAGrad-Fast [9] and Nash-MTL [38] (every 100), it reaches a higher or comparable success rate.

7 Conclusion

In this paper, we propose a new and flexible direction-oriented multi-objective problem formulation, as well as two simple and efficient MOO algorithms named SDMGrad and SDMGrad-OS. We establish the convergence guarantee for both algorithms in various settings. Extensive experiments validate the promise of our methods. We anticipate that our new problem formulation and the proposed algorithms can be applied in other learning applications with multiple measures, and the analysis can be of independent interest in analyzing other stochastic MOO algorithms.

References

Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1930-1939, 2018.

Junyu You, William Ampomah, and Qian Sun. Development and application of a machine learning based multi-objective optimization workflow for co2-eor projects. Fuel , 264:116758, 2020.

Xinyu Huang, Peng Wang, Xinjing Cheng, Dingfu Zhou, Qichuan Geng, and Ruigang Yang. The apolloscape open dataset for autonomous driving and its application. IEEE transactions on pattern analysis and machine intelligence , 42(10):2702-2719, 2019.

Philip S Thomas, Joelle Pineau, Romain Laroche, et al. Multi-objective spibb: Seldonian offline policy improvement with safety constraints in finite mdps. Advances in Neural Information Processing Systems , 34:2004-2017, 2021.

Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence , 44(7):3614-3633, 2021.

Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems , 31, 2018.

Jean-Antoine Désidéri. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique , 350(5-6):313-318, 2012.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems , 33:5824-5836, 2020.

Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems , 34:1887818890, 2021.

Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. arXiv preprint arXiv:2202.01017 , 2022.

Suyun Liu and Luis Nunes Vicente. The stochastic multi-gradient algorithm for multi-objective optimization and its application to supervised machine learning. Annals of Operations Research , pages 1-30, 2021.

Shiji Zhou, Wenpeng Zhang, Jiyan Jiang, Wenliang Zhong, Jinjie Gu, and Wenwu Zhu. On the convergence of stochastic multi-objective gradient manipulation and beyond. Advances in Neural Information Processing Systems , 35:38103-38115, 2022.

Heshan Devaka Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, and Tianyi Chen. Mitigating gradient bias in multi-objective learning: A provably convergent approach. In The Eleventh International Conference on Learning Representations , 2023.

Lisha Chen, Heshan Fernando, Yiming Ying, and Tianyi Chen. Three-way trade-off in multi-objective learning: Optimization, generalization and conflict-avoidance. arXiv preprint arXiv:2305.20057 , 2023.

Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587 , 2016.

Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098 , 2017.

Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering , 34(12):5586-5609, 2021.

Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3994-4003, 2016.

Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239 , 2017.

Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems , 33:4767-4777, 2020.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Haoxiang Wang, Han Zhao, and Bo Li. Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation. In International Conference on Machine Learning (ICML) , pages 10991-11002. PMLR, 2021.

Feiyang Ye, Baijiong Lin, Zhixiong Yue, Pengxin Guo, Qiao Xiao, and Yu Zhang. Multiobjective meta learning. Advances in Neural Information Processing Systems (NeurIPS) , 34:21338-21351, 2021.

Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems (NeurIPS) , 34:27503-27516, 2021.

Elliot Meyerson and Risto Miikkulainen. The traveling observer model: Multi-task learning through spatial variable embeddings. arXiv preprint arXiv:2010.02354 , 2020.

Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7482-7491, 2018.

Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning , pages 794-803. PMLR, 2018.

Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision (ECCV) , pages 270-287, 2018.

Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems , 33:2039-2050, 2020.

Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. arXiv preprint arXiv:2103.02631 , 2021.

Yuzheng Hu, Ruicheng Xian, Qilong Wu, Qiuling Fan, Lang Yin, and Han Zhao. Revisiting scalarization in multi-task learning: A theoretical perspective. arXiv preprint arXiv:2308.13985 , 2023.

Michinari Momma, Chaosheng Dong, and Jia Liu. A multi-objective/multi-task learning framework induced by pareto stationarity. In International Conference on Machine Learning , pages 15895-15907. PMLR, 2022.

Weiran Wang and Miguel Á. Carreira-Perpiñán. Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application, 2013.

Adrien Gaidon. Compute euclidean projections on the simplex or l1-ball. GitHub, 2011.

Baijiong Lin, Feiyang Ye, Yu Zhang, and Ivor W Tsang. Reasonable effectiveness of random weighting: A litmus test for multi-task learning. arXiv preprint arXiv:2111.10603 , 2021.

Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1871-1880, 2019.

Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. iclr, 2021.

Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization, 2023.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV , 2012.

Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence , 39(12):2481-2495, 2017.

Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos. Attentive single-tasking of multiple tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1851-1860, 2019.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning , pages 1094-1100. PMLR, 2020.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Shagun Sodhani and Amy Zhang. Mtrl - multi task rl algorithms. Github, 2021.

Jonathan M Borwein. A very complicated proof of the minimax theorem. Minimax Theory and its Applications , 1(1):21-27, 2016.

Ohad Shamir and Tong Zhang. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In International conference on machine learning , pages 71-79. PMLR, 2013.

Supplementary Materials

A Experiments

A.1 Toy Example

To demonstrate our proposed method can achieve better or comparable performance under stochastic settings, we provide an empirical study on the two-objective toy example used in CAGrad [9]. The two objectives L 1 ( x ) and L 2 ( x ) shown in Figure 1 are defined on x = ( x 1 , x 2 ) ⊤ ∈ R 2 ,

where the functions are given by

(a) Mean objective

(b) Objective 1

(c) Objective 2

(d) MGDA

(f) PCGrad

(g) CAGrad

(h) SDMGrad(Ours)

10

10

10

8

5

5

0

8-

Q

16

10

5

5

-10

-10

15

0

10

-10

-10

10

10

OT-

10

-5

10

-10

10

-5

10

10

10

10

10

5

5

5

0

-10

-10

-10

5

-10

15

15

15

15

10

-10

10

10

Q

5

10

10

10

10

-5

0

10

10

-5

U

10

for different methods and visualize the optimization trajectories in Figure 1. The starting point of every trajectory in Figure 1d-Figure 1h is given by the · symbol, and the color of every trajectory changes gradually from red to yellow. The gray line illustrates the Pareto front, and the ⋆ symbol denotes the global optimum. To simulate the stochastic setting, we add zero-mean Gaussian noise to the gradient of each objective for all the methods except MGDA. We adopt Adam optimizer with learning rate of 0.002 and 70000 iterations for each run. As shown, GD can get stuck due to the dominant gradient of a specific objective, which stops progressing towards the Pareto front. PCGrad and CAGrad can also fail to converge to the Pareto front in certain circumstances.

We choose 3 initializations

A.2 Consistency Verification

We conduct the experiment on the multi-task classification dataset Multi-Fashion+MNIST [45]. Each image contained in this dataset is constructed by overlaying two images randomly sampled from MNIST [46] and FashionMNIST [47] respectively. We adopt shrinked Lenet [48] as the shared base-encoder and a task-specific linear classification head for each task. We report the training losses obtained from different methods over 3 independent runs in Figure 2. As illustrated, the performance of SDMGrad with large λ is similar to GD, and the performance when λ is small resembles MGDA. With properly tuned λ , lower average training loss can be obtained. Generally, the results confirm the consistency of our formulation with the direction-oriented principle.

0.60

Task 1 Loss

Task 2 LosS

Avg. Loss

0.55

0.50

Training Loss

0.45

0.40

0.35

0.30

0.25

MGDA

SDMGrad

SDMGrad

SDMGrad

SDMGrad

SDMGrad

(=0)

( =0.5)

( =1)

(=5)

( =10)

The Multi-Fashion+MNIST[45] includes images constructed from FashionMNIST[47] and MNIST[46]. First, select one image from each dataset randomly, then transform the two images into a single image with one put in the top-left corner and the other in bottom-right corner. The dataset contains 120000 training images and 20000 test images. We use SGD optimizer with learning rate 0.001 and train for 100 epochs with batch size 256. We use multi-step scheduler with scale factor 0.1 to decay learning rate every 15 epochs. The projected gradient descent is performed with learning rate of 10 and momentum of 0.5 and 20 gradient descent steps are applied.

A.3 Supervised Learning

We implement the methods based on the library released by [10]. Following [9, 13, 10], we train our method for 200 epochs, using Adam optimizer with learning rate 0.0001 for the first 100 epochs and 0.00005 for the rest. The batch size for Cityscapes and NYU-v2 are 8 and 2 respectively. We compute the averaged test performance over the last 10 epochs as final performance measure. The inner projected gradient descent is performed with learning rate of 10 and momentum of 0.5 and 20 gradient descent steps are applied. The experiments on Cityscapes and NYU-v2 are run on RTX 3090 and Tesla V100 GPU, respectively. We also report additional experiment results over different λ and S = 1 in Table 5 and Table 6.

A.4 Reinforcement Learning

Following [9, 13, 10], we conduct the experiments based on MTRL codebase[49]. We train our method for 2 million steps with batch size of 1280. The inner projected gradient descent is performed with learning rate of 10 for MT10 benchmark and 20 gradient descent steps are applied. The method is evaluated once every 10000 steps and the best average test performance over 10 random seeds over the entire training process is reported. We search λ ∈ { 0 . 1 , 0 . 2 , · · · , 1 . 0 } for MT10 benchmark and the highest success rate is achieved when λ = 0 . 6 . For our objective sampling strategy, the number of sampled objectives is a random variable obeying binomial distribution whose expectation is n .

To compare with CAGrad-Fast[9], we choose n = 4 for MT10 benchmark. We cite the reported success rates of all baseline methods in Table 4, but independently run each experiment 5 times to calculate the average running time. All experiments on MT10 are run on RTX 2080Ti GPU. We also report addtional experiments results over S = 1 on MT10 in Table 7.

B Notations for Technical Proofs

In this part, we first summarize all the notations that we used in this paper in order to help readers understand. First, in multi-objective optimization, we have K ≥ 2 different objectives and each of them has the loss function L i ( θ ) . Let g i denote the gradient of objective i and g 0 denotes the target gradient. w = ( w 1 , ..., w K ) T ∈ R K and W denotes the probability simplex. Other useful notations are listed as below:

g w = ∑ i w i g i s.t. W = { w : ∑ i w i = 1 and w i ≥ 0 } w ∗ λ = arg min w ∈W 1 2 ∥ g w + λg 0 ∥ 2 , w ∗ = arg min w ∈W 1 2 ∥ g w ∥ 2 , w ∗ t = arg min w ∈W 1 2 ∥ G ( θ t ) w ∥ 2 w ∗ t,λ = arg min w ∈W F ( w ) = arg min w ∈W 1 2 ∥ G ( θ t ) w + λg 0 ( θ t ) ∥ 2 ∇ w F ( w ) = G ( θ t ) T ( G ( θ t ) w + λg 0 ( θ t )) , ∇ w ̂ F ( w ) = G ( θ t ; ξ ) T ( G ( θ t , ξ ′ ) w + λg 0 ( θ t , ξ ′ )) . (11) We use E [ · ] A | B to denote taking expectation over A conditioning on B and ˜ O omits the order of log .

C Detailed proofs for convergence analysis with nonconvex Objectives

We now provide some auxiliary lemmas for proving Proposition 1 and Theorem 1

Lemma 1. Let d ∗ be the solution of

then we have

In addition, w ∗ λ is the solution of

Proof. First, it can be seen that

Noting that the problem is concave w.r.t. d and convex w.r.t w and using the Von Neumann-Fan minimax theorem [50], we can exchange the min and max problems without changing the solution. Then, we can solve the following equivalent problem.

Then by fixing w , we have d ∗ = g w + λg 0 . Substituting this solution to the eq. (13) and rearranging the equation, we turn to solve the following problem.

Let w ∗ λ be the solution of the above problem, and hence the final updating direction d ∗ = g w ∗ λ + λg 0 . Then, the proof is complete.

Lemma 2. Suppose Assumption 2-3 are satisfied. According to the definition of g 0 ( θ ) in eq. (11) , we have the following inequalities,

Proof. Based on the definitions, we have

where the inequality follows from the fact that ∥ ˜ w ∥ ≤ 1 and Assumption 3. Then, we have

where σ 2 0 = max i σ 2 i and the proof is complete.

Lemma 3. Suppose Assumptions 2-3 are satisfied and recall that F ( w ) = 1 2 ∥ G ( θ t ) w + λg 0 ( θ t ) ∥ 2 is a convex function. Let w ∗ λ = arg min w ∈W 1 2 ∥ g w + λg 0 ∥ 2 and set step size β t,s = c/ √ s where c > 0 is a constant. Then for any S > 1 , it holds that,

Proof. This lemma mostly follows from Theorem 2 in [51]. However, we did not take that E [ ∥∇ w ̂ F ( w ) ∥ ] is bounded by a constant as an assumption. Therefore, we first provide a bound for it in our method. Based on the definition in Equation (11), ∇ w ̂ F ( w ) = G ( θ t ; ξ ) T ( G ( θ t ; ξ ′ ) w + λg 0 ( θ t ; ξ ′ )) . According to the fact that E [ X ] ≤ √ E [ X 2 ] , we have

where ( i ) follows from the Young's inequality. Next, we provide bounds for E [ A ] and E [ B ] , separately:

where ( i ) follows from Cauchy-Schwarz inequality and w ∈ W where W is the simplex, ( ii ) follows from Young's inequality and ( iii ) follows from Assumption 2 and Assumption 3. Then for term B, we have,

where ( i ) follows from Young's inequality, ( ii ) follows from Assumption 3 and Lemma 2. Then substituting eq. (15) and eq. (16) into eq. (14), we can obtain,

Meanwhile, since E [ ∥∇ w ̂ F ( w ) ∥ ] ≤ C 1 , sup w,w ′ ∥ w -w ′ ∥ ≤ 1 and by choosing step size β s = c/ √ s where c > 0 is a constant, we can obtain the following inequality from Theorem 2 in [51]:

Then after multiplying by 2 on both sides, the proof is complete.

C.1 Proof of Proposition 1

CAdistance. Now we show the upper bound for the distance to CA direction. Recall that we define the CA distance as ∥ E ζ,w t,S | θ t [ G ( θ t ; ζ ) w t,S + λg 0 ( θ t ; ζ )] -G ( θ t ) w ∗ t,λ -λg 0 ( θ t ) ∥ .

Proof. Based on the Jensen's inequality, we have

where ( i ) omits the subscript of taking expectation over w t,S conditioning on θ t , ( ii ) follows from optimality condition that

( iii ) follows from Lemma 3 whenwe choose β t,s = c/ √ s where c is a constant. Then take the square root on both sides, the proof is complete.

C.2 Proof of Theorem 1

Theorem 5 (Restatement of Theorem 1) . Suppose Assumptions 1-3 are satisfied. Set α t = α = Θ((1 + λ ) -1 K -1 2 T -1 2 ) , β t,s = c/ √ s where c is a constant, and S = Θ((1 + λ ) -2 T 2 ) . The outputs of the proposed SDMGrad algorithm satisfy

Proof. Recall that d = G ( θ t ; ζ ) w t,S + λg 0 ( θ t ; ζ ) . According to Assumption 1, we have for any i ,

where l ′ i, 1 = l i, 1 + λ max i l i, 1 = Θ(1 + λ ) . Then we bound the second and third terms separately on the right-hand side (RHS). First, for the second term, conditioning on θ t and taking expectation, we have

where ( i ) follows from Cauchy-Schwarz inequality and optimality condition in eq. (19), ( ii ) follows from the fact that E [ X ] ≤ √ E [ X 2 ] and ( iii ) follows from eq. (18).

Then for the third term,

where ( i ) follows from Young's inequality, and ( ii ) follows from Assumption 3 and Lemma 2. Note that C 2 = O ( K + Kλ 2 ) . Then taking expectation on eq. (20), substituting eq. (21) and eq. (22) into it, and unconditioning on θ t , we have

Then, choosing α t = α , and rearranging the above inequality, we have

Telescoping over t ∈ [ T ] in the above inequality yields

If we choose α = Θ((1 + λ ) -1 K -1 2 T -1 2 ) and S = Θ((1 + λ ) -2 T 2 ) , we have

where O means the order of logT

˜ is omitted. The proof is complete.

C.3 Proof of Corollary 1

Proof. Since λ > 0 and g 0 ( θ t ) = G ( θ t ) ˜ w , we have

where w ′ = 1 1+ λ ( w ∗ 1 ,t,λ + λ ˜ w 1 , w ∗ 2 ,t,λ + λ ˜ w 2 , ..., w ∗ K,t,λ + λ ˜ w K ) T such that w ′ ∈ W . According to parameter selection in Theorem 1 and by choosing a constant λ , we have

To achieve an ϵ -accurate Pareto stationary point, it requires T = ˜ O ( Kϵ -2 ) and each objective requires ˜ O ( K 3 ϵ -6 ) samples in ξ ( ξ ′ ) and ˜ O ( Kϵ -2 ) samples in ζ , respectively. Meanwhile, according to the choice of S and T , we have the following result for CA distance,

Remark. Our algorithm with a constant λ helps mitigate gradient conflict and it guarantees an ϵ -accurate Pareto stationary point and the CA distance takes the order of ˜ O ( ϵ ) simultaneously.

C.4 Proof of Corollary 2

Proof. According to the inequality ∥ a + b -b ∥ 2 ≤ 2 ∥ a + b ∥ 2 +2 ∥ b ∥ 2 , we have

where the last inequality follows from Assumption 3. Then we take the expectation on the above inequality and sum up it over t ∈ [ T ] such that

where the last inequality follows from Theorem 1. If we choose λ = Θ( T 1 2 ) , then we have

To achieve an ϵ -accurate stationary point, it requires T = ˜ O ( Kϵ -2 ) and each objective requires ˜ O ( K 2 ϵ -4 ) samples in ξ ( ξ ′ ) and ˜ O ( Kϵ -2 ) samples in ζ , respectively. Meanwhile, according to the choice of λ , S and T , we have the following result for CA distance,

Remark. With an increasing λ , our algorithm approaches GD and it has a faster convergence rate to the stationary point. However, the CA distance takes the order of ˜ O ( √ K ) .

C.5 Proof of Theorem 2

Now we provide the convergence analysis with nonconvex objectives with objective sampling.

Theorem 6 (Restatement of Theorem 2) . Suppose Assumptions 1-3 are satisfied. Set γ = K n , α t = α = Θ((1 + λ 2 ) -1 2 γ -1 2 K -1 2 T -1 2 ) , β t,s = c/ √ s and S = Θ((1 + λ ) -2 γ -2 T 2 ) . Then by choosing a constant λ , the iterates of the proposed SDMGrad-OS algorithm satisfy

Proof. Recall that updating direction for θ t is d ′ = K n H ( θ t ; ζ, ˜ S ) w t,S + K n λh 0 ( θ t ; ζ, ˜ S ) . Similarly, we have

Then for the inner product term on the RHS of eq. (26), conditioning on θ t and taking expectation, we have

where the last inequality follows from eq. (21). Then following the same step as in eq. (22), we can bound the last term on the RHS of eq. (26) as

Then taking expectation on eq. (26), substituting eq. (27) and eq. (28) into it and unconditioning on θ t , we have

Then choosing α t = α , telescoping the above inequality over t ∈ [ T ] , and rearranging the terms, we have

If we choose α = Θ((1 + λ 2 ) -1 2 γ -1 2 K -1 2 T -1 2 ) , and S = Θ((1 + λ ) -2 γ -2 T 2 ) , we can get 1 T ∑ T -1 t =0 E [ ∥ G ( θ t ) w ∗ t,λ + λg 0 ( θ t ) ∥ 2 ] = ˜ O ((1 + λ 2 ) K 1 2 γ 1 2 T -1 2 ) . Furthermore, by choosing λ as constant and following the same step as in Appendix C.3, we have

To achieve an ϵ -accurate Pareto stationary point, it requires T = ˜ O ( Kγϵ -2 ) . In this case, each objective requires a similar number of samples ˜ O ( K 3 γϵ -6 ) in ξ ( ξ ′ ) and ˜ O ( Kγϵ -2 ) samples in ζ , respectively. As far as we know, this is the first provable objective sampling strategy for stochastic multi-objective optimization.

D Lower sample complexity but higher CA distance

When we do not have requirements on CA distance, we can have a much lower sample complexity. In Algorithm 1, the update process for w is to reduce the CA distance, which increases the sample complexity. Thus, we will set S = 1 to make Algorithm 1 more sample-efficient. In addition, we will use w t +1 = w t, 1 and β t instead of β t,s in Algorithm 1 for simplicity. The following proof is mostly motivated by Theorem 3 in [14].

D.1 Proof of Theorem 3

Theorem 7 (Restatement of Theorem 3) . Suppose Assumptions 1-3 are satisfied and S = 1 . Set α t = α = Θ( K -1 2 T -1 2 ) , β t = β = Θ( K -1 T -1 2 ) and λ as constant. The iterates of the proposed SDMGrad algorithm satisfy,

Proof. Now we define a new function, with a fixed weight w ∈ W ,

For this new function, we have

where l ′ 1 = max i l i, 1 + λl i, 1 . Then taking expectations over ζ on both sides and rearranging the inequality, we have

where ( i ) follows from Cauchy-Schwarz inequlaity. Then we provide bound for term C and term D, respectively. For term C,

where C 3 = O ((1 + λ ) 2 K 2 ) . Then for term D, we first follow the non-expansive property of projection onto the convex set,

Then taking expectation on the above inequality, we can obtain,

where the last inequality follows from Lemma 3. Then by rearranging the above inequality, we can obtain,

Then substituting eq. (31) and eq. (32) into eq. (30), we can obtain,

Then we take α t = α and β t = β as constants, telescope and rearrange the above inequality,

where ( i ) follows from that we choose λ as a constant. If we choose α = Θ( K -1 2 T -1 2 ) and β = Θ( K -1 T -1 2 ) , we can get 1 T ∑ T -1 t =0 E [ ∥ G ( θ t ) w t + λg 0 ( θ t ) ∥ 2 ] = O ( KT -1 2 ) . Furthermore, following the same steps as in Appendix C.3, we have

To achieve an ϵ -accurate Pareto stationary point, it requires T = O ( K 2 ϵ -2 ) . In this case, each objective requires a similar number of samples O ( K 2 ϵ -2 ) in ξ ( ξ ′ ) and ζ , respectively.

Convergence under objective sampling. We next analyze the convergence of SDMGrad-OS.

Theorem 8 (Restatement of Theorem 4) . Suppose Assumptions 1-3 are satisfied and S = 1 . Set γ = K n , α t = α = Θ( K -1 2 γ -1 2 T -1 2 ) , β t = β = Θ( K -1 γ -1 T -1 2 ) and λ as a constant. The iterates of the proposed SDMGrad-OS algorithm satisfy,

Proof. In SDMGrad-OS, the vector for updating θ t is d ′ = K n H ( θ t ; ζ, ˜ S ) w t +1 + λK n h 0 ( θ t ; ζ, ˜ S ) . Using the same function defined in eq. (29), we have

Then by taking expectation over ζ and ˜ S , we have

Then following the same steps in eq. (31) and eq. (32), we can obtain,

where C ′ 2 1 = 4 γ 4 (1+ λ 2 )( nσ 2 0 + C g ) 2 , C ′ 2 = 4 γ 2 (1+ λ 2 )( nσ 2 0 + C 2 g ) , and C ′ 3 = γ 2 (1+ λ ) 2 C 2 g ( nσ 2 0 + C g ) 2 . Then we take α t = α and β t = β as constants and telescope the above inequality,

where ( i ) follows from that we choose λ as constant. Similarly, if we choose α = Θ( K -1 2 γ -1 2 T -1 2 ) and β = Θ( K -1 γ -1 T -1 2 ) , we can get 1 T ∑ T -1 t =0 E [ ∥ G ( θ t ) w t + λg 0 ( θ t ) ∥ 2 ] = O ( KγT -1 2 ) . Furthermore, following the same step as in Appendix C.3, we have

To achieve an ϵ -accurate Pareto stationary point, it requires T = O ( γ 2 K 2 ϵ -2 ) . In this case, each objective requires a similar number of samples O ( γ 2 K 2 ϵ -2 ) in ξ ( ξ ′ ) and ζ , respectively.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a gradient manipulation method named SDMGrad for multi-task learning (MTL). SDMGrad improves the previous MGDA method by using two constraints. The first one is to constrain the common descent direction nearby the one computed with a specific preference (such as the average direction), which is similar to the previous CAGrad method. The second one is to add a regularization term for the computed weights, which is the same as the previous MoCo method. To reduce the computational cost, partial tasks are randomly selected to compute their gradients in SDMGrad, which is called SDMGrad-OS and is similar to the previous RLW method.\n\nTheoretically, a stochastic non-convex convergence analysis for the proposed methods is provided. Empirically, experiments on multi-task supervised learning and reinforcement learning are conducted but some important baselines are missing.\n\nSTRENGTHS:\n1. This paper proposes a method for MTL.\n2. This paper provides a stochastic non-convex convergence analysis for the proposed methods.\n3. The code is provided.\n\nWEAKNESSES:\n1. The novelty of the proposed methods is limited. This paper aims to improve the previous MGDA method by adding several components. However, those components are similar to some existing work.\n2. Lack of some important baselines, especially Nash-MTL (ICML 2022), which outperforms the proposed SMDGrad in all benchmark datasets.\n3. The regularization term with a coefficient $\\frac{\\rho}{2}$ in Eq. (9) is one of the proposed improvements. However, the paper does not mention how to set $\\rho$. Besides, from the code in the supplementary material, it seems $\\rho=0$ in the implementation.\n\nQUESTIONS:\n1. The novelty of the proposed methods is limited. The proposed SDMGrad method improves the previous MGDA method by using two constraints.\n- constraining the common descent direction nearby the one computed with a specific preference, as shown in Eq. (6). This is very similar to CAGrad in Eq. (7). The only difference is the norm constraint used in CAGrad while the angle constraint used in SDMGrad. Thus, what is the motivation and advantage to use Eq. (6)?\n- adding a regularization term for the computed weights in Eq. (9), which is the same as MoCo. \n- SDMGrad-OS reduces the computational cost in SDMGrad by randomly sampling tasks, which is similar to RLW [1]. In RLW, loss/gradient weights are randomly sampled from a distribution (such as Normal and Bernoulli) at each iteration. If the sample distribution is Bernoulli-family, it is equivalent to randomly sampling task losses/gradients to optimize at each iteration.\n\n&ensp;&ensp; In a word, it is hard to find something new or interesting in Section 4.\n\n2. What is the meaning of \"propose a new direction-oriented multi-objective problem\" in Lines 3-4, Line 139, and Line 313? Maybe \"formulation\" is more appropriate than \"problem\".\n3. $L_i$ in the last of Line 23 is quite confusing. It is better to use more clear notation.\n4. Lines 24-25: \"MTL aims to solve an average loss for all $K$ tasks\" is wrong. \n5. Line 27: why MOO **cannot** find a common parameter $\\theta$ that achieves optima for all objectives? MOO can, but rarely. The description of \"parameter $\\theta$ that minimizes all objective functions\" is wrong. \n6. Eq. (4): what is the meaning of $h_{t, i}$?\n7. Line 133: $\\mathbb{R}^K\\rightarrow\\mathbb{R}^m$\n8. Lines 145-147: in what scenarios? It is better to provide some examples here.\n9. Line 176 in the text and Step 5 in Algorithm 1: why two different sampled data is used for the update of $\\omega$? From the code in the supplementary material, the same sampled data is used in the implementation.\n10. Proposition 1: it seems the bound is meaningless when $\\rho\\rightarrow0$.\n11. Lines 447-448 in Appendix: the experimental setting of adding zero-mean Gaussian noise is following MoCo but without citation. Besides, MoCo does not been compared in this toy example section.\n12. Section B in Appendix: \"$g_0$ denotes the average gradient\" in Line 483 but $g_0=G_0(\\theta)=G(\\theta)\\tilde{\\omega}$ in Eq. (13). What does $\\tilde{\\omega}$ denote? Is $G_0$ a vector here? It is confusing because capital $G$ denotes a matrix and lowercase $g$ denotes a vector in Eq. (13). The last line in Eq. (13): $G(\\theta_t)\\rightarrow G(\\theta_t)\\omega$.\n13. Eq. (17) in Appendix: Is it possible to provide a detailed derivation of step (i)? \n14. Hyperparameter selection.\n- how to set $\\rho$? From the code in the supplementary material, it seems $\\rho=0$ in the implementation. So why? If so, the regularization term in Eq. (9) and the bound in Proposition 1 are meaningless.\n- $\\lambda$ in the proposed SDMGrad is searched for the best. How about the hyperparameters of baselines? From Table 8 in Appendix, CAGrad performs significantly better than the results in Table 4 in the main text.\n- how to set $n$ for SMDGrad-OS?\n15. Line 289: the definition of $\\Delta m$ is wrong since some task has more than one metric (for example those tasks in Cityscapes and NYU-v2).\n16. Lack of some important baselines.\n- Nash-MTL [2] outperforms the proposed SMDGrad in all benchmark datasets, i.e., Cityscapes, NYU-v2, MT10, and MT50. Besides, the running time of Nash-MTL is shorter than SMDGrad-OS in MT10.\n- It is better to compare with more baselines such as RLW [1], GradNorm [3], IMTL [4], and so on.\n17. SegNet is out-of-dated and performs unsatisfactory. It is better to conduct experiments using a more powerful backbone, such as resnet50 and transformer.\n18. The results of SMDGrad-OS in Cityscapes and NYU-v2 datasets should be reported.\n19. The claim of SMDGrad-OS can deal with the case where the number of objectives is large (Lines 10-11) while MoCo does not (Lines 44-47). However, there is no experiment to support this. \n20. Lines 94-96: there is no experiment to support that the proposed SMDGrad is model-agnostic. \n \n\n\n\n\n----\n[1] Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning. TMLR 2022.\n\n[2] Multi-Task Learning as a Bargaining Game. ICML 2022.\n\n[3] GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks. ICML 2018.\n\n[4] Towards Impartial Multi-task Learning. ICLR 2021.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe contributions are as follows:\n- First, this work gives a new framework of direction-oriented multi-objective optimization.\n- Second, they propose an algorithm, SDMGrad (and an objective sampling version when the number of objectives is large)\n- Third, they give a convergence analysis for their algorithm and show that it has improved complexity compared to previous work\n- Finally, they show good empirical performance on both supervised learning (semantic segmentation + depth estimation) and reinforcement learning (robot manipulation task) paradigms\n\nSTRENGTHS:\nOverall, the paper is clearly written, the new algorithm is straightforwardly effective, and the convergence rates beat existing bounds with fewer convergence assumptions. More specifically:\n- The proposed formulation is a generalization of existing algorithms, CAGrad and SGD\n- Algorithm 1 is intuitive to understand and memory-efficient to implement \n- The convergence rate beats the previous MoCo by epsilon^2, and even is able to handle the case where the number of objectives is large (via the objective sampling version)\n- Experiments are conducted well and show the effectiveness of the method in practice\n\nWEAKNESSES:\n- I am unsure about the applicability of the newly proposed formulation in the context of multi-objective optimization. Could the authors share some examples where this direction based regularization would be useful? \n- Also, I am not sure how the -1/2||d||^2 term will affect the regularization, as it seems added for the convenience of the algorithm and convergence proof.\n- Combining the above two points, properly choosing $$\\lambda$$ also seems like it is important but would add an extra hyperparameter (making things more costly, empirically). For the examples you could give above, can you also share how this parameter would be chosen? \n- I believe the work would benefit from a short discussion of the theorems and proof challenges in the main text. Currently, there are four theorems listed with little discussion of any proof details.\n\nQUESTIONS:\n- Could you outline, in intuitive terms, a sketch of the proof of Theorem 1? Is the proof inspired by any previous work (such as MoCo or CR-MOGM)?\n- Is there any chance to remove or relax the bounded gradient assumption?\n- Why can the convergence rate (of both your algorithm and MoCo) improve so much with a bounded function assumption?\n- Do you have any sense how things would change if you remove the smoothness assumption?\n\nLIMITATIONS:\nNo limitation section is included.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors presented a new stochastic gradient method for multi-objective optimization (MOO) problem, called SDMGrad.\nCompared with previous SMG, they claimed SDMGrad dose not need to increase the batch size linearly with the iteration numbers.\nCompared with previous MoCo, they claimed SDMGrad could be applied into situations with large number of objectives.\n\nSTRENGTHS:\nThe authors presented a new stochastic gradient method for multi-objective optimization (MOO) problem, called SDMGrad.\nDetailed background is provided to help understand the history of MOO.\nAnd various experiments show that the new method can outperform the previous methods.\nBut I need to ask some questions to fully understand the novelty of this method.\n\nWEAKNESSES:\nI need to ask some questions to fully understand the novelty of this method.\nAlso, some technically details are presented very clear, I will also post some questions.\n\nQUESTIONS:\n1. Could you detail the advantages and novelty of SDMGrad over MoCo?\nFirst, Line 5 in Algorithm 1 seems similar with Equation 4 of MoCo.\nSecond, authors commented MoCo \"a relatively strong assumption that the number T of iterations is much larger than the number K of objectives.\" Well, in the real industrial systems, the number T of iterations could be millions, and the number K of objectives (maybe 3 or 10?) is obviously much smaller than T. So this is a natural assumption to me and hence weaken the necessity to improve MoCo from this perspective.\n\n2. Why the equation in the Line 5 of Algorithm 1 is an unbiased estimation? (Line 175)\n\n3. The equation after line 175 is the derivative of Equation 9, right? Why we need two different data samples: \\xi and \\xi\\prime?\nand how do we implement this in the mini-batch training? sample two mini-batches in each s?\n\n4.  Line 5 in Algorithm 1, why we need \\prod_{W}? what is the meaning of this? In Equation 3, W is a set with infinite possibilities of w. How does this multiply work? And this is element-wise multiple?\n\nLIMITATIONS:\nNo social impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper proposes a direction-oriented multi-objective gradient descent algorithm under stochastic gradient settings. The authors show that the algorithm can benefit from the direction-oriented mechanism and ensure optimal convergence. In addition, an objective sampling strategy is applied to the proposed algorithm for the case of many objectives.

STRENGTHS:
This paper studies an important problem in multi-objective optimization. It has solid analysis and experiments.

WEAKNESSES:
1. The key benefit of MoCo compared with CR-MOGM is that the direction can gradually be unbiased to the one calculated by full-batch gradients (Lemma 2). However, there is no respective analysis of the direction bias in the analysis.

2. I check that CR-MOGM has the same sample complexity as SDMGrad in the non-convex setting. Note that $O(T^{-1/4})$ in CR-MOGM is for the first-order convergence of $\|\|G_t w_t\|\|$, and for the second-order convergence $\|\|G_t w_t\|\|_2^2$ is $O(T^{-1/2})$. Usually, the first-order one is more often used.

3. The idea is very similar to [1], which studies multi-objective online learning that can be easily reduced to stochastic settings. [1] also uses a regularization to restrict the direction to be closed to a prior preference.

4. CR-MOGM has not been compared in the experiments.

5. I am not convinced with the motivation to face many objectives problems as stated in Line 46. It is known that the many objective problem is hard to be solved by Pareto optimization, since when the number of objectives becomes too large, the Pareto set will cover nearly the majority of the space, then optimizing towards Pareto optimality is meaningless.


[1] Jiang et al. Multi-Objective Online Learning. ICLR 23.

QUESTIONS:
No

LIMITATIONS:
Novelty is limited.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the stochastic direction-oriented multi-objective gradient descent (SDMGrad) and SDMGrad-OS (OS stands for objective sampling). The idea of SDMGrad is to make the objective “direction-oriented”, which is done by regularizing the descent direction towards a specific direction. This direction is chosen to be the average of all gradients, which would allow the regularizer to interpolate the algorithm between multiple-gradient descent and regular gradient descent. This formulation is similar to CAGrad, but results in a simpler stochastic analysis and a simpler algorithm. The experiments show that, for a well-tuned regularization parameter, SDMGrad’s performance is competitive with the state-of-the-art. The objective sampling procedure in SDMGrad-OS makes it run significantly faster than SDMGrad when the number of objectives is large. Analysis for both methods is provided with good sample complexities.\n\nSTRENGTHS:\n- The idea is clear, the formulation is sound, and the algorithm is simple to implement.\n- The formulation allows for a better stochastic analysis and simpler algorithm. The sample complexity achieved by the algorithm is good.\n- SDMGrad generalizes MGDA and GD, and is a better formulation of CAGrad.\n- The code is available, so experiments can be reproduced.\n- Performance is good overall, and can be better for fine-tuned lambda.\n- Objective sampling helps when the number of objective is large. The analysis also justifies this procedure, which is a good contribution.\n\nWEAKNESSES:\n- Need to tune lambda to get better results. It does not seem to be easy to pick a good starting point as the algorithm is not very robust to the choice of lambda. Also, it is not clear how to choose $\\rho$.\n- The algorithm interpolates between MGDA and GD (with $w$ regularization), so the algorithmic novelty is limited.\n- In equation 4, $h_{t,i}$ is not explained.\n- The improvements are not significant, and that is for the best choices of $\\lambda$. For other choices of $\\lambda$, it might be worse. It seems to be on par with CAGrad and MoCO, which is competitive.\n- In the appendix, the authors mention Young's inequality for some steps, but I think it's Cauchy-Schwartz, though I might be mistaken.\n- Note that the Lipschitzness (bounded gradient) assumption of $L_i$ would not work with strongly convex losses.\n\nQUESTIONS:\n- Which method do you use to project the weights onto the simplex? Is the projection costly?\n- Have you considered relaxing the simplex constraint? Would it make sense to use the closed-form minimum $w$ for unconstrained $w$?\n- Have you considered a different regularizer for $w$? How do you set $\\rho$ in practice?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes stochastic variants of the MGDA for multi-objective optimization (MOO) problem, named \"Stochastic Direction-Oriented Multi-objective Gradient descent\" (SDMGrad) and SDMGrad-OS with efficient sampling. Optimization convergence analysis to the Pareto stationary point is provided, with improved complexities compared to prior works on stochastic MOO. \nThe proof of optimization convergence mainly follows that of [12].\nExperiments on multi-task supervised learning and reinforcement learning justified the superior performance of the proposed method.\n\nSTRENGTHS:\n1. The paper proposes a stochastic variant of MGDA to address the MOO problem.\n\n2. Optimization convergence analysis is provided with improved complexity over [12] without bounded function values assumption, and with improved complexity over [11] with bounded function values assumption.\n\n3. Experiments on MTL benchmarks demonstrated the superior performance of the proposed methods.\n\nWEAKNESSES:\n## Soundness\n### Lack of justification of some claims\n\nSee __Questions__-1,2.\n\n\n### Some steps of the proof is ambiguous or unclear\n\nSee __Questions__-3,4.\n\n### Theoretical and practical benefits of the proposed algorithm over the simple unitary scalarization (GD or SGD) baseline are unclear\n\n1. The algorithm is not very efficient. In Theorems 1, 2, with a constant $\\lambda$, it requires the inner loop iterations $S = \\mathcal{O}(T^3)$, and each step of the inner loop needs to compute a projection to the simplex, which adds more time complexity to the MGDA based algorithms as they already require computing $K$ gradients at each outer iteration comparing to $1$ gradient for GD or SGD.  \nHow does your algorithm compare to the simple SGD baseline in terms of convergence to Pareto stationary point in clock time? It would be better if some evaluations and discussions regarding this can be provided.\nE.g., in Table 6 or other experiments, compare with the time and performance of the simple SGD baseline.\n\n5. What is the theoretical benefit of this proposed algorithm compared to SGD of unitary scalarization, as the latter can also converge to Pareto stationary (PS) point and is more efficient?   \nFor example, in [12], in addition to the convergence to PS point, Lemma 2 is provided to justify the convergence of MoCo to the desired MGDA direction, does similar results apply for the proposed algorithm also?\n\n2. In the experiments, one baseline method with unitary scalarization is needed, as it has been shown in prior works [a,b] that unitary scalarization outperforms MGDA-based algorithms in some practical applications.  \nI understand that experiments with varying $\\lambda$ are provided, which becomes close to GD with larger $\\lambda$, but they are still not the same. Instead I would expect an ablation study with varied $\\beta_t$, and especially $\\beta_t = 0$.\n\n>[a] Vitaly Kurin et al. \"In Defense of the Unitary Scalarization for Deep Multi-Task Learning\"\n\n>[b] Derrick Xin et al. \"Do Current Multi-Task Optimization Methods in Deep Learning Even Help?\"\n\n### Technical novelty is limited\n\nNovelty in terms of proof techniques is limited, the proof mainly follows that of [12].\n\n## Minor\n\n1. Appendix C.2, line 530, \"satisfies\" -> \"satisfy\"\n2. Move Table 4 to Section 6.2\n3. In Appendix D.1, LHS of Equation below line 591, \"$E$\" -> \"$\\mathbb{E}$\"\n\n\n\n================UPDATE==========================\n\nTo avoid unnecessary misunderstandings, and unevidenced accusations, I change the word *honest* to *accurate* in my previous comment and also update my final comments and suggestions below.\n\nI appreciate the reviewers carefully checking the code and I also appreciate the authors clarifying what they implemented in the rebuttal. \n\nMy current score is based on the theoretical contributions and also conditioned on the authors can correctly implement and report the results with double sampling. Otherwise it could be confusing.\n\n=================================================\n\nQUESTIONS:\n1. In Section 4.2, line 170-172, why is the case \"the objective in eq. (8) is merely convex w.r.t. $w$, it can only guarantee the convergence of at least one iterate (not necessarily the last one) to the solution $w_\\lambda^*$\"?  \nIn fact, there are works that prove the last iterate convergence of convex objectives, see [c].  \nSince this serves as a motivation for the algorithm design, it is important to clarify it.  \nAnd although it claims that \"to ensure the more practical last-iterate convergence\" in line 172, no guarantee of convergence of objective in eq. (9) for last iterate is provided in the paper.\n\n>[c] Ohad Shamir et al. \"Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes\".\n\n2. The use of smoothing constant $\\rho$ seems to be redundant since you already have another regularization term $\\lambda g_0$, which makes the update direction close to $g_0$. Using the regularization term $\\rho ||w||^2$ has a similar effect of making the update direction close to a uniform combination of gradients of all objectives.  \nSince the motivation \"to ensure the more practical last-iterate convergence\" is questionable, see __Question__-1,\nmore justification is needed for this design.\n\n3. The expectation operations used throughout the paper and proof are very unclear. \nThe same notation $\\mathbb{E}[\\cdot]$ is used for almost all expectations, whether conditioning on $\\theta_t$ or not, except for Eq. (11) in the main paper where $\\mathbb{E}[\\cdot\\mid \\theta_t]$ is used. This makes the arguments or proof steps ambiguous. \n   - In Proposition 1, Equation (11), the LHS has two expectations, one conditioned on $\\theta_t$, and the other not. What are the specific distributions for these two expectations? There is some ambiguity since there are multiple random variables, e.g. $\\zeta, w_{t,S}, \\theta_t$.  \nAlso in the proof of Proposition 1, Appendix C.1, line 522, how is the condition on $\\theta_t$ removed in the first equation of (21)?  \nThis does not seem correct if the inner expectation is not conditioned on $\\theta_t$.\n    - In most part of the proof, $\\mathbb{E}[\\cdot]$ is used for both conditioning on $\\theta_t$ and without conditioning on $\\theta_t$.  \n    E.g. line 533 in the Appendix, Eq (24) is \"conditioning on $\\theta_t$\", and later on in line 538 Eq(26) is \"unconditioning on $\\theta_t$\".\n\n1. In Appendix C.2, line 533-534, I did not see how the inequality (iii) is derived from eq. (21)?\n\nLIMITATIONS:\nI did not see specific discussions on the limitations and broader societal impacts. \n\nThis does not decrease my score, and it would be better if the authors provide some.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}