SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haibo Yang
Dept. of Comput. & Info. Sci.  
Rochester Institute of Technology  
Rochester, NY 14623  
hbycis@rit.edu
Zhuqing Liu
Dept. of ECE  
The Ohio State University  
Columbus, OH 43210  
liu.9384@osu.edu
Jia Liu
Dept. of ECE  
The Ohio State University  
Columbus, OH 43210  
liu@ece.osu.edu
Chaosheng Dong
Amazon.com Inc. Seattle, WA 98109 chaosd@amazon.com
Michinari Momma
Amazon.com Inc. Seattle, WA 98109 michi@amazon.com
In recent years, multi-objective optimization (MOO) emerges as a foundational problem underpinning many multi-agent multi-task learning applications. However, existing algorithms in MOO literature remain limited to centralized learning settings, which do not satisfy the distributed nature and data privacy needs of such multi-agent multi-task learning applications. This motivates us to propose a new federated multi-objective learning (FMOL) framework with multiple clients distributively and collaboratively solving an MOO problem while keeping their training data private. Notably, our FMOL framework allows a different set of objective functions across different clients to support a wide range of applications, which advances and generalizes the MOO formulation to the federated learning paradigm for the first time. For this FMOL framework, we propose two new federated multi-objective optimization (FMOO) algorithms called federated multi-gradient descent averaging (FMGDA) and federated stochastic multi-gradient descent averaging (FSMGDA). Both algorithms allow local updates to significantly reduce communication costs, while achieving the same convergence rates as those of their algorithmic counterparts in the single-objective federated learning. Our extensive experiments also corroborate the efficacy of our proposed FMOO algorithms.
In recent years, multi-objective optimization (MOO) has emerged as a foundational problem underpinning many multi-agent multi-task learning applications, such as training neural networks for multiple tasks [1], hydrocarbon production optimization [2], recommendation system [3], tissue engineering [4], and learning-to-rank [5, 6, 7]. MOO aims at optimizing multiple objectives simultaneously, which can be mathematically cast as:
$$
\min  _ {\mathbf {x} \in \mathcal {D}} \mathbf {F} (\mathbf {x}) := \left[ f _ {1} (\mathbf {x}), \dots , f _ {S} (\mathbf {x}) \right], \tag {1}
$$
where  $\mathbf{x} \in \mathcal{D} \subseteq \mathbb{R}^d$  is the model parameter, and  $f_s: \mathbb{R}^d \to \mathbb{R}$ ,  $s \in [S]$  is one of the objective functions. Compared to conventional single-objective optimization, one key difference in MOO is the coupling and potential conflicts between different objective functions. As a result, there may not exist a common  $\mathbf{x}$ -solution that minimizes all objective functions. Rather, the goal in MOO is to find a Pareto stationary solution that is not improvable for all objectives without sacrificing some objectives. For example, in recommender system designs for e-commerce, the platform needs to consider different
customers with substantially conflicting shopping objectives (price, brand preferences, delivery speed, etc.). Therefore, the platform's best interest is often to find a Pareto-stationary solution, where one cannot deviate to favor one consumer group further without hurting any other group. MOO with conflicting objectives also has natural incarnations in many competitive game-theoretic problems, where the goal is to determine an equilibrium among the conflicting agents in the Pareto sense.
Since its inception dating back to the 1950s, MOO algorithm design has evolved into two major categories: gradient-free and gradient-based methods, with the latter garnering increasing attention in the learning community in recent years due to their better performances (see Section 2 for more detailed discussions). However, despite these advances, all existing algorithms in the current MOO literature remain limited to centralized settings (i.e., training data are aggregated and accessible to a centralized learning algorithm). Somewhat ironically, such centralized settings do not satisfy the distributed nature and data privacy needs of many multi-agent multi-task learning applications, which motivates application of MOO in the first place. This gap between the existing MOO approaches and the rapidly growing importance of distributed MOO motivates us to make the first attempt to pursue a new federated multi-objective learning (FMOL) framework, with the aim to enable multiple clients to distributively solve MOO problems while keeping their computation and training data private.
So far, however, developing distributed optimization algorithms for FMOL with provable Pareto-stationary convergence remains uncharted territory. There are several key technical challenges that render FMOL far from being a straightforward extension of centralized MOO problems. First of all, due to the distributed nature of FMOL problems, one has to consider and model the objective heterogeneity (i.e., different clients could have different sets of objective functions) that is unseen in centralized MOO. Moreover, with local and private datasets being a defining feature in FMOL, the impacts of data heterogeneity (i.e., datasets are non-i.i.d. distributed across clients) also need to be mitigated in FMOL algorithm design. Last but not least, under the combined influence of objective and data heterogeneity, FMOL algorithms could be extremely sensitive to small perturbations in the determination of common descent direction among all objectives. This makes the FMOL algorithm design and the associated convergence analysis far more complicated than those of the centralized MOO. Toward this end, a fundamental question naturally arises:
Under both objective and data heterogeneity in FMOL, is it possible to design effective and efficient algorithms with Pareto-stationary convergence guarantees?
In this paper, we give an affirmative answer to the above question. Our key contribution is that we propose a new FMOL framework that captures both objective and data heterogeneity, based on which we develop two gradient-based algorithms with provable Pareto-stationary convergence rate guarantees. To our knowledge, our work is the first systematic attempt to bridge the gap between federated learning and MOO. Our main results and contributions are summarized as follows:
- We formalize the first federated multi-objective learning (FMOL) framework that supports both objective and data heterogeneity across clients, which significantly advances and generalizes the MOO formulation to the federated learning paradigm. As a result, our FMOL framework becomes a generic model that covers existing MOO models and various applications as special cases (see Section 3.2 for further details). This new FMOL framework lays the foundation to enable us to systematically develop FMOO algorithms with provable Pareto-stationary convergence guarantees.
- For the proposed FMOL framework, we first propose a federated multi-gradient descent averaging (FMGDA) algorithm based on the use of local full gradient evaluation at each client. Our analysis reveals that FMGDA achieves a linear  $\mathcal{O}(\exp(-\mu T))$  and a sublinear  $\mathcal{O}(1/T)$  Pareto-stationary convergence rates for  $\mu$ -strongly convex and non-convex settings, respectively. Also, FMGDA employs a two-sided learning rates strategy to significantly lower communication costs (a key concern in the federated learning paradigm). It is worth pointing out that, in the single-machine special case where FMOL degenerates to a centralized MOO problem and FMGDA reduces to the traditional MGD method [8], our results improve the state-of-the-art analysis of MGD by eliminating the restrictive assumptions on the linear search of learning rate and extra sequence convergence. Thus, our results also advance the state of the art in general MOO theory.
- To alleviate the cost of full gradient evaluation in the large dataset regime, we further propose a federated stochastic multi-gradient descent averaging (FSMGDA) algorithm based on the use of stochastic gradient evaluations at each client. We show that FSMGDA achieves  $\tilde{\mathcal{O}}(1/T)$  and  $\mathcal{O}(1/\sqrt{T})$  Pareto-stationary convergence rate for  $\mu$ -strongly convex and non-convex settings, respectively. We establish our convergence proof by proposing a new  $(\alpha, \beta)$ -Lipschitz continuous
Table 1: Convergence rate results (shaded parts are our results) comparisons.
Methods	Strongly Convex	Non-convex
Rate	Assumption*	Rate	Assumption*
MGD [8]	O(rT) #	Linear search & sequence convergence	O(1/T)	Linear search & sequence convergence
SMGD [9]	O(1/T)	First moment bound & Lipschitz continuity of λ	Not provided	Not provided
FMGDA	O(exp(-μT)) #	Not needed	O(1/T)	Not needed
FSMGDA	O(1/T)	(α,β)-Lipschitz continuous stochastic gradient	O(1/√T)	(α,β)-Lipschitz continuous stochastic gradient
\*Notes on constants:  $\mu$  is the strong convexity modulus;  $r$  is a constant depends on  $\mu$  , s.t.,  $r\in (0,1)$  . Assumption short-hands: "Linear search": learning rate linear search [8]; "Sequence convergence":  $\{\mathbf{x}_t\}$  converges to  $\mathbf{x}^*$  [8]; "First moment bound" (Asm. 5.2(b) [9]):  $\mathbb{E}[\| \nabla f(\mathbf{x},\xi) - \nabla f(\mathbf{x})\| ]\leq \eta (a + b\| \nabla f(\mathbf{x})\|);$  Lipschitz continuity of  $\lambda$  (Asm. 5.4 [9]):  $\| \pmb {\lambda}_k - \pmb {\lambda}_t\| \leq$ $\beta \| [\left(\nabla f_1(\mathbf{x}_k) - \nabla f_1(\mathbf{x}_t)\right)^T,\dots ,(\nabla f_S(\mathbf{x}_k) - \nabla f_S(\mathbf{x}_t))^T]\|$  ;  $(\alpha ,\beta)$  -Lipschitz continuous stochastic gradient": see Asm. 4.
stochastic gradient assumption (cf. Assumption 4), which relaxes the strong assumptions on first moment bound and Lipschitz continuity on common descent directions in [9]. We note that this new  $(\alpha, \beta)$ -Lipschitz continuous stochastic gradient assumption can be viewed as a natural extension of the classical Lipschitz-continuous gradient assumption and could be of independent interest.
The rest of the paper is organized as follows. In Section 2, we review related works. In Section 3, we introduce our FMOL framework and two gradient-based algorithms (FMGDA and FSMGDA), which are followed by their convergence analyses in Section 4. We present the numerical results in Section 5 and conclude the work in Section 6. Due to space limitations, we relegate all proofs and some experiments to supplementary material.
In this section, we will provide an overview on algorithm designs for MOO and federated learning (FL), thereby placing our work in a comparative perspective to highlight our contributions and novelty.
1) Multi-objective Optimization (MOO): As mentioned in Section 1, since federated/distributed MOO has not been studied in the literature, all existing works we review below are centralized MOO algorithms. Roughly speaking, MOO algorithms can be grouped into two main categories. The first line of works are gradient-free methods (e.g., evolutionary MOO algorithms and Bayesian MOO algorithms [10, 11, 12, 13]). These methods are more suitable for small-scale problems but less practical for high-dimensional MOO models (e.g., deep neural networks). The second line of works focus on gradient-based approaches [14, 15, 8, 16, 9, 17], which are more practical for high-dimensional MOO problems. However, while having received increasing attention from the community in recent years, Pareto-stationary convergence analysis of these gradient-based MOO methods remains in its infancy.
Existing gradient-based MOO methods can be further categorized as i) multi-gradient descent (MGD) algorithms with full gradients and ii) stochastic multi-gradient descent (SMGD) algorithms. It has been shown in [8] that MGD methods achieve  $\mathcal{O}(r^T)$  for some  $r\in (0,1)$  and  $\mathcal{O}(1 / T)$  Pareto-stationary convergence rates for  $\mu$ -strongly convex and non-convex functions, respectively. However, these results are established under the unconventional linear search of learning rate and sequence convergence assumptions, which are difficult to verify in practice. In comparison, FMGDA achieves a linear rate without needing such assumptions. For SMGD methods, the Pareto-stationary convergence analysis is further complicated by the stochastic gradient noise. Toward this end, an  $\mathcal{O}(1 / T)$  rate analysis for SMGD was provided in [9] based on rather strong assumptions on a first-moment bound and Lipschitz continuity of common descent direction. As a negative result, it was shown in [9] and [18] that the common descent direction needed in the SMGD method is likely to be a biased estimation, which may cause divergence issues.
In contrast, our FSMGDA achieves state-of-the-art  $\tilde{\mathcal{O}}(1/T)$  and  $\mathcal{O}(1/\sqrt{T})$  convergence rates for strongly-convex and non-convex settings, respectively, under a much milder assumption on Lipschitz continuous stochastic gradients. For easy comparisons, we summarize our results and the existing works in Table 1. It is worth noting recent works [18, 19, 20] established faster convergence rates in
the centralized MOO setting by using acceleration techniques, such as momentum, regularization and bi-level formulation. However, due to different settings and focuses, these results are orthogonal to ours and thus not directly comparable. Also, since acceleration itself is a non-trivial topic and could be quite brittle if not done right, in this paper, we focus on the basic and more robust stochastic gradient approach in FMOL. But for a comprehensive comparison on assumptions and main results of accelerated centralized MOO, we refer readers to Appendix A for further details.
Federated Learning (FL): Since the seminal work by [21], FL has emerged as a popular distributed learning paradigm. Traditional FL aims at solving single-objective minimization problems with a large number of clients with decentralized data. Recent FL algorithms enjoy both high communication efficiency and good generalization performance [21, 22, 23, 24, 25, 26]. Theoretically, many FL methods have the same convergence rates as their centralized counterparts under different FL settings [27, 28, 29, 30]. Recent works have also considered FL problems with more sophisticated problem structures, such as min-max learning [31, 32], reinforcement learning [33], multi-armed bandits [34], and bilevel and compositional optimization [35]. Although not directly related, classic FL has been reformulated in the form of MOO[36], which allows the use of a MGD-type algorithm instead of vanilla local SGD to solve the standard FL problem. We will show later that this MOO reformulation is a special case of our FMOL framework. So far, despite a wide range of applications (see Section 3.2 for examples), there remains a lack of a general FL framework for MOO. This motivates us to bridge the gap by proposing a general FMOL framework and designing gradient-based methods with provable Pareto-stationary convergence rates.
3 Federated multi-objective learning
3.1 Multi-objective optimization: A primer
As mentioned in Section 1, due to potential conflicts among the objective functions in MOO problem in (1), MOO problems adopt the the notion of Pareto optimality:
Definition 1 ((Weak) Pareto Optimality). For any two solutions  $\mathbf{x}$  and  $\mathbf{y}$ , we say  $\mathbf{x}$  dominates  $\mathbf{y}$  if and only if  $f_{s}(\mathbf{x}) \leq f_{s}(\mathbf{y}), \forall s \in [S]$  and  $f_{s}(\mathbf{x}) < f_{s}(\mathbf{y}), \exists s \in [S]$ . A solution  $\mathbf{x}$  is Pareto optimal if it is not dominated by any other solution. One solution  $\mathbf{x}$  is weakly Pareto optimal if there does not exist a solution  $\mathbf{y}$  such that  $f_{s}(\mathbf{x}) > f_{s}(\mathbf{y}), \forall s \in [S]$ .
Similar to solving single-objective non-convex optimization problems, finding a Pareto-optimal solution in MOO is NP-Hard in general. As a result, it is often of practical interest to find a solution satisfying Pareto-stationarity (a necessary condition for Pareto optimality) stated as follows [14, 37]:
Definition 2 (Pareto Stationarity). A solution  $\mathbf{x}$  is said to be Pareto stationary if there is no common descent direction  $\mathbf{d} \in \mathbb{R}^d$  such that  $\nabla f_s(\mathbf{x})^\top \mathbf{d} < 0, \forall s \in [S]$ .
Note that for strongly convex functions, Pareto stationary solutions are also Pareto optimal. Following Definition 2, gradient-based MOO algorithms typically search for a common descent direction  $\mathbf{d} \in \mathbb{R}^d$  such that  $\nabla f_s(\mathbf{x})^\top \mathbf{d} \leq 0, \forall s \in [S]$ . If no such a common descent direction exists at  $\mathbf{x}$ , then  $\mathbf{x}$  is a Pareto stationary solution. For example, MGD [15] searches for an optimal weight  $\lambda^*$  of gradients  $\nabla \mathbf{F}(\mathbf{x}) \triangleq \{\nabla f_s(\mathbf{x}), \forall s \in [S]\}$  by solving  $\lambda^*(\mathbf{x}) = \operatorname*{argmin}_{\lambda \in C} \| \lambda^\top \nabla \mathbf{F}(\mathbf{x}) \|^2$ . Then, a common descent direction can be chosen as:  $\mathbf{d} = \lambda^\top \nabla \mathbf{F}(\mathbf{x})$ . MGD performs the iterative update rule:  $\mathbf{x} \gets \mathbf{x} - \eta \mathbf{d}$  until a Pareto stationary point is reached, where  $\eta$  is a learning rate. SMGD [9] also follows the same process except for replacing full gradients by stochastic gradients. For MGD and SMGD methods, it is shown in [8] and [18] show that if  $\| \lambda^\top \nabla \mathbf{F}(\mathbf{x}) \| = 0$  for some  $\lambda \in C$ , where  $C \triangleq \{\mathbf{y} \in [0,1]^S, \sum_{s \in [S]} y_s = 1\}$ , then  $\mathbf{x}$  is a Pareto stationary solution. Thus,  $\| \mathbf{d} \|^2 = \| \lambda^\top \nabla \mathbf{F}(\mathbf{x}) \|^2$  can be used as a metric to measure the convergence of non-convex MOO algorithms [8, 18, 19]. On the other hand, for more tractable strongly convex MOO problems, the optimality gap  $\sum_{s \in [S]} \lambda_s [f_s(\mathbf{x}) - f_s(\mathbf{x}^*)]$  is typically used as the metric to measure the convergence of an algorithm [9], where  $\mathbf{x}^*$  denotes the Pareto optimal point. We summarize and compare different convergence metrics as well as assumptions in MOO, detailed in Appendix A.
3.2 A general federated multi-objective learning framework
With the MOO preliminaries in Section 3.1, we now formalize our general federated multi-objective learning (FMOL) framework. For a system with  $M$  clients and  $S$  tasks (objectives), our FMOL framework can be written as:
$$
\min  _ {\mathbf {x}} \quad \operatorname {D i a g} \left(\mathbf {F} \mathbf {A} ^ {\top}\right), \tag {2}
$$
$$
\mathbf {F} \triangleq \left[ \begin{array}{c c c} f _ {1, 1} & \dots & f _ {1, M} \\ \vdots & \ddots & \vdots \\ f _ {S, 1} & \dots & f _ {S, M} \end{array} \right] _ {S \times M}, \mathbf {A} \triangleq \left[ \begin{array}{c c c} a _ {1, 1} & \dots & a _ {1, M} \\ \vdots & \ddots & \vdots \\ a _ {S, 1} & \dots & a _ {S, M} \end{array} \right] _ {S \times M},
$$
where matrix  $\mathbf{F}$  groups all potential objectives  $f_{s,i}(\mathbf{x})$  for each task  $s$  at each client  $i$ , and  $\mathbf{A} \in \{0,1\}^{S \times M}$  is a binary objective indicator matrix, with each element  $a_{s,i} = 1$  if task  $s$  is of client  $i$ 's interest and  $a_{s,i} = 0$  otherwise. For each task  $s \in [S]$ , the global objective function  $f_{s}(\mathbf{x})$  is the average of local objectives over all related clients, i.e.,  $f_{s}(\mathbf{x}) \triangleq \frac{1}{|R_{s}|} \sum_{i \in R_{s}} f_{s,i}(\mathbf{x})$ , where  $R_{s} = \{i : a_{s,i} = 1, i \in [M]\}$ . Note that, for notation simplicity, here we use simple average in  $f_{s}(\mathbf{x})$ , which corresponds to the balanced dataset setting. Our FMLO framework can be directly extended to imbalanced dataset settings by using weighted average proportional to dataset sizes of related clients. For a client  $i \in [M]$ , its objectives of interest are  $\{f_{s,i}(\mathbf{x}) : a_{s,i} = 1, s \in [S]\}$ , which is a subset of  $[S]$ .
We note that FMOL generalizes MOO to the FL paradigm, which includes many existing MOO problems as special cases and corresponds to a wide range of applications.
- If each client has only one distinct objective, i.e.,  $\mathbf{A} = \mathbb{I}_M$ ,  $S = M$ , then  $\mathrm{Diag}(\mathbf{FA}^\top) = [f_1(\mathbf{x}), \ldots, f_S(\mathbf{x})]^\top$ , where each objective  $f_s(\mathbf{x}), s \in [S]$  is optimized only by client  $s$ . This special FMOL setting corresponds to the conventional multi-task learning and federated learning. Indeed, [1] and [38] formulated a multi-task learning problem as MOO and considered Pareto optimal solutions with various trade-offs. [36] also formulated FL as as distributed MOO problems. Other examples of this setting include bi-objective formulation of offline reinforcement learning [39] and decentralized MOO [40].
- If all clients share the same  $S$  objectives, i.e.,  $\mathbf{A}$  is an all-one matrix, then  $\mathrm{Diag}(\mathbf{FA}^{\top}) = \left[\frac{1}{M}\sum_{i\in [M]}f_{1,i}(\mathbf{x}),\dots,\frac{1}{M}\sum_{i\in [M]}f_{S,i}(\mathbf{x})\right]^{\top}$ . In this case, FMOL reduces to federated MOO problems with decentralized data that jointly optimizing fairness, privacy, and accuracy [41, 42, 43], as well as MOO with decentralized data under privacy constraints (e.g., machine reassignment among data centres [44] and engineering problems [45, 46, 47, 48]).
- If each client has a different subset of objectives (i.e., objective heterogeneity), FMLO allows distinct preferences at each client. For example, each customer group on a recommender system in e-commerce platforms might have different combinations of shopping preferences, such as product price, brand, delivery speed, etc.
3.3 Federated Multi-Objective Learning Algorithms
Upon formalizing our FMOL framework, our next goal is to develop gradient-based algorithms for solving large-scale high-dimensional FMOL problems with provable Pareto stationary convergence guarantees and low communication costs. To this end, we propose two FMOL algorithms, namely federated multiple gradient descent averaging (FMGDA) and federated stochastic multiple gradient descent averaging (FSMGDA) as shown in Algorithm 1. We summarize our key notation in Table 3 in Appendix to allow easy references for readers.
As shown in Algorithm 1, in each communication round  $t \in [T]$ , each client synchronizes its local model with the current global model  $\mathbf{x}_t$  from the server (cf. Step 1). Then each client runs  $K$  local steps based on local data for all effective objectives (cf. Step 2) with two options: i) for FMGDA, each local step performs local full gradient descent, i.e.,  $\mathbf{x}_{s,i}^{t,k+1} = \mathbf{x}_{s,i}^{t,k} - \eta_L \nabla f_{s,i}(\mathbf{x}_{s,i}^{t,k}), \forall s \in S_i$ ; ii) For FSMGDA, the local step performs stochastic gradient descent, i.e.,  $\mathbf{x}_{s,i}^{t,k+1} = \mathbf{x}_{s,i}^{t,k} - \eta_L \nabla f_{s,i}(\mathbf{x}_{s,i}^{t,k}, \xi_i^{t,k}), \forall s \in S_i$ , where  $\xi_i^{t,k}$  denotes a random sample in local step  $k$  and round  $t$  at client  $i$ . Upon finishing  $K$  local updates, each client returns the accumulated update  $\Delta_{s,i}^t$  for each effective objective to the server (cf. Step 3). Then, the server aggregates all returned  $\Delta$ -updates from
Algorithm 1 Federated (Stochastic) Multiple Gradient Descent Averaging (FMGDA/FSMGDA).
At Each Client  $i$ :
1. Synchronize local models  $\mathbf{x}_{s,i}^{t,0} = \mathbf{x}_t, \forall s \in S_i$ .
2. Local updates: for all  $s \in S_i$ , for  $k = 1, \dots, K$
$$
(F M G D A): \mathbf {x} _ {s, i} ^ {t, k} = \mathbf {x} _ {s, i} ^ {t, k - 1} - \eta_ {L} \nabla f _ {s, i} (\mathbf {x} _ {s, i} ^ {t, k - 1}).
$$
$$
(\text {F S M G D A}): \mathbf {x} _ {s, i} ^ {t, k} = \mathbf {x} _ {s, i} ^ {t, k - 1} - \eta_ {L} \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, k - 1}, \xi_ {i} ^ {t, k}\right).
$$
3. Return accumulated updates to server  $\{\Delta_{s,i}^t, s \in S_i\}$ :