SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} (\operatorname {F M G D A}): \Delta_ {s, i} ^ {t} = \sum_ {k \in [ K ]} \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, k}\right). \\ (\text {F S M G D A}): \Delta_ {s, i} ^ {t} = \sum_ {k \in [ K ]} \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, k}, \xi_ {i} ^ {t, k}\right). \\ \end{array}
$$
At the Server:
4. Receive accumulated updates  $\{\Delta_{s,i}^{t},\forall s\in S_{i},\forall i\in [M]\}$
5. Compute  $\Delta_s^t = \frac{1}{|R_s|}\sum_{i\in R_s}\Delta_{s,i}^t,\forall s\in [S]$ , where  $R_{s} = \{i:a_{s,i} = 1,i\in [M]\}$ .
6. Compute  $\lambda_t^* \in [0,1]^S$  by solving
$$
\min  _ {\lambda_ {t} \geq 0} \left\| \sum_ {s \in [ S ]} \lambda_ {s} ^ {t} \Delta_ {s} ^ {t} \right\| ^ {2}, \quad \text {s . t .} \sum_ {s \in [ S ]} \lambda_ {s} ^ {t} = 1. \tag {3}
$$
7. Let  $\mathbf{d}_t = \sum_{s\in [S]}\lambda_s^{t,*}\Delta_s^t$  and update the global model as:  $\mathbf{x}_{t + 1} = \mathbf{x}_t - \eta_t\mathbf{d}_t$ , with a global learning rate  $\eta_t$ .
the clients to obtain the overall updates  $\Delta_s^t$  for each objective  $s\in [S]$  (cf. Steps 4 and 5), which will be used in solving a convex quadratic optimization problem with linear constraints to obtain an approximate common descent direction  $\mathbf{d}_t$  (cf. Step 6). Lastly, the global model is updated following the direction  $\mathbf{d}_t$  with global learning rate  $\eta_t$  (cf. Step 7).
Two remarks on Algorithm 1 are in order. First, we note that a two-sided learning rates strategy is used in Algorithm 1, which decouples the update schedules of local and global model parameters at clients and server, respectively. As shown in Section 4 later, this two-sided learning rates strategy enables better convergence rates by choosing appropriate learning rates. Second, to achieve low communication costs, Algorithm 1 leverages  $K$  local updates at each client and infrequent periodic communications between each client and the server. By adjusting the two-sided learning rates appropriately, the  $K$ -value can be made large to further reduce communication costs.
4 Pareto stationary convergence analysis
In this section, we analyze the Pareto stationary convergence performance for our FMGDA and FSMGDA algorithms in Sections 4.1 and 4.2, respectively, each of which include non-convex and strongly convex settings.
4.1 Pareto stationary convergence of FMGDA
In what follows, we show FMGDA enjoys linear rate  $\tilde{\mathcal{O}} (\exp (-\mu T))$  for  $\mu$ -strongly convex functions and sub-linear rate  $\mathcal{O}\left(\frac{1}{T}\right)$  for non-convex functions.
1) FMGDA: The Non-convex Setting. Before presenting our Pareto stationary convergence results for FMGDA, we first state several assumptions as follows:
Assumption 1. (L-Lipschitz continuous) There exists a constant  $L > 0$  such that  $\| \nabla f_s(\mathbf{x}) - \nabla f_s(\mathbf{y}) \| \leq L \| \mathbf{x} - \mathbf{y} \|, \forall \mathbf{x}, \mathbf{y} \in \mathbb{R}^d, s \in [S]$ .
Assumption 2. (Bounded Gradient) The gradient of each objective at any client is bounded, i.e., there exists a constant  $G > 0$  such that  $\| \nabla f_{s,i}(\mathbf{x})\| ^2\leq G^2,\forall s\in [S],i\in [M]$ .
With the assumptions above, we state the Pareto stationary convergence of FMGDA for non-convex FMOL as follows:
Theorem 1 (FMGDA for Non-convex FMOL). Let  $\eta_t = \eta \leq \frac{3}{2(1 + L)}$ . Under Assumptions 1 and 2, if at least one function  $f_s, s \in [S]$  is bounded from below by  $f_s^{\min}$ , then the sequence  $\{\mathbf{x}_t\}$  output by FMGDA satisfies:  $\min_{t \in [T]} \| \bar{\mathbf{d}}_t \|^2 \leq \frac{16(f_s^0 - f_s^{\min})}{T\eta} + \delta$ , where  $\delta \triangleq \frac{16\eta_L^2 K^2 L^2 G^2 (1 + S^2)}{\eta}$ .
In non-convex functions, we use  $\left\| \bar{\mathbf{d}}_t\right\|^2$  as the metrics for FMOO, where  $\bar{\mathbf{d}}_t = \boldsymbol{\lambda}_t^T\nabla (\mathrm{Diag}(\mathbf{FA}^\top))$  and  $\lambda_{t}$  is calculated by the quadratic programming problem 3 based on accumulated (stochastic) gradients  $\Delta_t$ . We compare different metrics for MOO in Appendix A. The convergence bound in Theorem 1 contains two parts. The first part is an optimization error, which depends on the initial point and vanishes as  $T$  increases. The second part is due to local update steps  $K$  and data heterogeneity  $G$ , which can be mitigated by carefully choosing the local learning rate  $\eta_L$ . Specifically, the following Pareto stationary convergence rate of FMGDA follows immediately from Theorem 1 with an appropriate choice of local learning rate  $\eta_L$ :
Corollary 2. With a constant global learning rate  $\eta_t = \eta$ ,  $\forall t$ , and a local learning rate  $\eta_L = \mathcal{O}(1 / \sqrt{T})$ , the Pareto stationary convergence rate of FMGDA is  $(1 / T)\sum_{t\in [T]}\|\bar{\mathbf{d}}_t\|^2 = \mathcal{O}(1 / T)$ .
Several interesting insights of Theorem 1 and Corollary 2 are worth pointing out: 1) We note that FMGDA achieves a Pareto stationary convergence rate  $\mathcal{O}(1 / T)$  for non-convex FMOL, which is the same as the Pareto stationary rate of MGD for centralized MOO and the same convergence rate of gradient descent (GD) for single objective problems. This is somewhat surprising because FMGDA needs to handle more complex objective and data heterogeneity under FMOL; 2) The two-sided learning rates strategy decouples the operation of clients and server by utilizing different learning rate schedules, thus better controlling the errors from local updates due to data heterogeneity; 3) Note that in the single-client special case, FMGDA degenerates to the basic MGD algorithm. Hence, Theorem 1 directly implies a Pareto stationary convergence bound for MGD by setting  $\delta = 0$  due to no local updates in centralized MOO. This convergence rate bound is consistent with that in [8]. However, we note that our result is achieved without using the linear search step for learning rate [8], which is much easier to implement in practice (especially for deep learning models); 4) Our proof is based on standard assumptions in first-order optimization, while previous works require strong and unconventional assumptions. For example, a convergence of x-sequence is assumed in [8].
2) FMGDA: The Strongly Convex Setting. Now, we consider the strongly convex setting for FMOL, which is more tractable but still of interest in many learning problems in practice. In the strongly convex setting, we have the following additional assumption:
Assumption 3. ( $\mu$ -Strongly Convex Function) Each objective  $f_{s}(\mathbf{x})$ ,  $s \in [S]$  is a  $\mu$ -strongly convex function, i.e.,  $f_{s}(\mathbf{y}) \geq f_{s}(\mathbf{x}) + \nabla f_{s}(\mathbf{x})(\mathbf{y} - \mathbf{x}) + \frac{\mu}{2}\|\mathbf{y} - \mathbf{x}\|^{2}$  for some  $\mu > 0$ .
For more tractable strongly-convex FMOL problems, we show that FMGDA achieves a stronger Pareto stationary convergence performance as follows:
Theorem 3 (FMGDA for  $\mu$ -Strongly Convex FMOL). Let  $\eta_t = \eta$  such that  $\eta \leq \frac{3}{2(1 + L)}$ ,  $\eta \leq \frac{1}{2L + \mu}$  and  $\eta \geq \frac{1}{\mu T}$ . Under Assumptions 1-3, pick  $\mathbf{x}_t$  as the final output of the FMGDA algorithm with weights  $w_t = (1 - \frac{\mu\eta}{2})^{1 - t}$ . Then, it holds that  $\mathbb{E}[\Delta_Q^t] \leq \| \mathbf{x}_0 - \mathbf{x}_* \|^2 \mu \exp(-\frac{\eta \mu T}{2}) + \delta$ , where  $\Delta_Q^t \triangleq \sum_{s \in [S]} \lambda_s^{t,*}[f_s(\mathbf{x}_t) - f_s(\mathbf{x}_*)]$  and  $\delta = \frac{8\eta_L^2 K^2 L^2 G^2 S^2}{\mu} + 2\eta_L^2 K^2 L^2 G^2$ .
Theorem 3 immediately implies following Pareto stationary convergence rate for FMGDA with a proper choice of local learning rate:
Corollary 4. If  $\eta_L$  is chosen sufficiently small such that  $\delta = \mathcal{O}(\mu \exp (-\mu T))$ , then the Pareto stationary convergence rate of FMGDA is  $\mathbb{E}[\Delta_Q^t ] = \mathcal{O}(\mu \exp (-\mu T))$ .
Again, several interesting insights can be drawn from Theorem 3 and Corollary 4. First, for strongly convex FMOL, FMGDA achieves a linear convergence rate  $\mathcal{O}(\mu \exp(-\mu T))$ , which again matches those of MGD for centralized MOO and GD for single-objective problems. Second, compared with the non-convex case, the convergence bounds suggest FMGDA could use a larger local learning rate for non-convex functions thanks to our two-sided learning rates design. A novel feature of FMGDA for strongly convex FMOL is the randomly chosen output  $x_{t}$  with weight  $w_{t}$  from the  $\mathbf{x}_{t}$ -trajectory, which is inspired by the classical work in stochastic gradient descent (SGD) [49]. Note that, for implementation in practice, one does not need to store all  $\mathbf{x}_{t}$ -values. Instead, the algorithm can be implemented by using a random clock for stopping [49].
4.2 Pareto stationary convergence of FSMGDA
While enjoying strong performances, FMGDA uses local full gradients at each client, which could be costly in the large dataset regime. Thus, it is of theoretical and practical importance to consider the stochastic version of FMGDA, i.e., federated stochastic multi-gradient descent averaging (FSMGDA).
1) FSMGDA: The Non-convex Setting. A fundamental challenge in analyzing the Pareto stationarity convergence of FSMGDA and other stochastic multi-gradient descent (SMGD) methods stems from bounding the error of the common descent direction estimation, which is affected by both  $\lambda_t^*$  (obtained by solving a quadratic programming problem) and the stochastic gradient variance. In fact, it is shown in [9] and [18] that the stochastic common descent direction in SMGD-type methods could be biased, leading to divergence issues. To address these challenges, in this paper, we propose to use a new assumption on the stochastic gradients, which is stated as follows:
Assumption 4  $((\alpha, \beta)$ -Lipschitz Continuous Stochastic Gradient). A function  $f$  has  $(\alpha, \beta)$ -Lipschitz continuous stochastic gradients if there exist two constants  $\alpha, \beta > 0$  such that, for any two independent training samples  $\xi$  and  $\xi'$ ,  $\mathbb{E}[\|\nabla f(\mathbf{x}, \xi) - \nabla f(\mathbf{y}, \xi')\|^2] \leq \alpha \| \mathbf{x} - \mathbf{y}\|^2 + \beta \sigma^2$ .
In plain language, Assumption 4 says that the stochastic gradient estimation of an objective does not change too rapidly. We note that the  $(\alpha, \beta)$ -Lipschitz continuous stochastic gradient assumption is a natural extension of the classic  $L$ -Lipschitz continuous gradient assumption (cf. Assumption 1) and generalizes several assumptions of SMGD convergence analysis in previous works. We note that Assumption 1 is not necessarily too hard to satisfy in practice. For example, when the underlying distribution of training samples  $\xi$  has a bounded support (typically a safe assumption for most applications in practice due to the finite representation limit of computing systems), suppose that Assumption 1 holds (also a common assumption in the optimization literature), then for any given  $\mathbf{x}$  and  $\mathbf{y}$ , the left-hand-side of the inequality in Assumption 4 is bounded due to the L-smoothness in Assumption 1. In this case, there always exist a sufficiently large  $\alpha$  and a  $\beta$  such that the right-hand-side of the inequality in Assumption 1 holds. Please see Appendix A for further details. In addition, we need the following assumptions for the stochastic gradients, which are commonly used in standard SGD-based analyses [49, 50, 51, 52].
Assumption 5. (Unbiased Stochastic Estimation) The stochastic gradient estimation is unbiased for each objective among clients, i.e.,  $\mathbb{E}[\nabla f_{s,i}(\mathbf{x},\xi)] = \nabla f_{s,i}(\mathbf{x}),\forall s\in [S],i\in [M]$ .
Assumption 6. (Bounded Stochastic Gradient) The stochastic gradients satisfy  $\mathbb{E}[\| \nabla f_{s,i}(\mathbf{x},\xi)\| ^2 ]\leq D^2,\forall s\in [S],i\in [M]$  for some constant  $D > 0$
With the assumptions above, we now state the Pareto stationarity convergence of FSMGDA as follows:
Theorem 5 (FSMGDA for Non-convex FMOL). Let  $\eta_t = \eta \leq \frac{3}{2(1 + L)}$ . Under Assumptions 4-6, if an objective  $f_s$  is bounded from below by  $f_s^{\min}$ , then the sequence  $\{\mathbf{x}_t\}$  output by FSMGDA satisfies:  $\min_{t \in [T]} \mathbb{E} \left\| \bar{\mathbf{d}}_t \right\|^2 \leq \frac{8(f_s^0 - f_s^{\min})}{\eta T} + \delta$ , where  $\delta = (2S^2 + 4)(\alpha \eta_L^2 K^2 D^2 + \beta \sigma^2)$ .
Theorem 5 immediately implies an  $\mathcal{O}(1 / \sqrt{T})$  convergence rate of FSMGDA for non-convex FMOL:
Corollary 6. With a constant global learning rate  $\eta_t = \eta = \mathcal{O}(1 / \sqrt{T})$ ,  $\forall t$  and a local learning rate  $\eta_L = \mathcal{O}\left(1 / T^{1/4}\right)$ , and if  $\beta = \mathcal{O}(\eta)$ , the Pareto stationarity convergence rate of FSMGDA is  $\min_{t \in [T]} \mathbb{E} \| \bar{\mathbf{d}}_t \|^2 = \mathcal{O}(1 / \sqrt{T})$ .
2) The Strongly Convex Setting: For more tractable strongly convex FMOL problems, we can show that FSMGDA achieve stronger convergence results as follows:
Theorem 7 (FSMGDA for  $\mu$ -Strongly Convex FMOL). Let  $\eta_t = \eta = \Omega\left(\frac{1}{\mu T}\right)$ . Under Assumptions 3, 5 and 6, pick  $\mathbf{x}_t$  as the final output of the FSMGDA algorithm with weight  $w_t = (1 - \frac{\mu\eta}{2})^{1 - t}$ . Then, it holds that:  $\mathbb{E}[\Delta_Q^t] \leq \| \mathbf{x}_0 - \mathbf{x}_* \|^2 \mu \exp(-\frac{\eta}{2} \mu T) + \delta$ , where  $\Delta_Q^t = \sum_{s \in [S]} \lambda_s^{t,*} [f_s(\mathbf{x}_t) - f_s(\mathbf{x}_*)]$  and  $\delta = \frac{1}{\mu} S^2 (\alpha \eta_L^2 K^2 D^2 + \beta \sigma^2) + \frac{\eta S^2 D^2}{2}$ .
The following Pareto station convergence rate of FSMGDA follows immediately from Theorem 7:
Corollary 8. Choose  $\eta_L = \mathcal{O}\left(\frac{1}{\sqrt{T}}\right)$  and  $\eta = \Theta\left(\frac{\log(\max(1,\mu^2T))}{\mu T}\right)$ . If  $\beta = \mathcal{O}(\eta)$ , then the Pareto stationary convergence rate of FSMGDA is  $\mathbb{E}[\Delta_Q^t ]\leq \tilde{\mathcal{O}} (1 / T)$ .
Corollary 8 says that, With proper learning rates, FSMGDA achieves  $\tilde{\mathcal{O}}(1/T)$  Pareto stationary convergence rate (i.e., ignoring logarithmic factors) for strongly convex FMOL. Also, in the single-client special case with no local updates, FSMGDA reduces to the SMGD algorithm and  $\delta = \frac{4}{\mu}\beta S^2\sigma^2 + \frac{\eta S^2D^2}{2}$  in this case. Then, Theorem 7 implies an  $\tilde{\mathcal{O}}\left(\frac{1}{T}\right)$  Pateto stationarity convergence rate for SMGD for strongly convex MOO problems, which is consistent with previous works [9]. However, our convergence rate proof uses a more conventional  $(\alpha,\beta)$ -Lipschitz stochastic gradient assumption, rather than the unconventional assumptions on the first moment bound and Lipschitz continuity of common descent directions in [9].
5 Numerical results
In this section, we show the main numerical experiments of our FMGDA and FSMGDA algorithms in different datasets, while relegating the experimental settings and details to the appendix.
Figure 1: Training loss convergence comparison.
1) Ablation Experiments on Two-Tasks FMOL: 1-a) Impacts of Batch Size on Convergence: First, we compare the convergence results in terms of the number of communication rounds using the "MultiMNIST" dataset [53] with two tasks (L and R) as objectives. We test our algorithms with four different cases with batch sizes being [16, 64, 128, 256]. To reduce computational costs in this experiment, the dataset size for each client is limited to 256. Hence, the batch size 256 corresponds to FMGDA and all other batch sizes correspond to FSMGDA. As shown in Fig. 1(a), under non-i.i.d. data partition, both FMGDA and FSMGDA algorithms converge. Also, the convergence speed of the FSMGDA algorithm increases as the batch size gets larger. These results are consistent with our theoretical analyses as outlined in Theorems 1 and 5.
1-b) Impacts of Local Update Steps on Convergence: Next, we evaluate our algorithms with different numbers of local update steps  $K$ . As shown in Fig. 1(b) and Table 2, both algorithms converge faster as the number of the local steps  $K$  increases. This is because both algorithms effectively run more iterative updates as  $K$  gets large.
1-c) Comparisons between FMOL and Centralized MOO: Since this work is the first that investigates FMOL, it is also interesting to empirically compare the differences between FMOL and centralized MOO methods. In Fig. 2(a), we compare the training loss of FMGDA and FSMGDA with those of the centralized MGD and SMGD methods after 100 communication rounds. For fair comparisons, the centralized MGD and SMGD methods use  $\sum_{i}^{M}|S_{i}|$  batch-sizes and run  $K\times T$  iterations. Our results indicate that FMGDA and MGD produce similar results, while the performance of FSMGDA is slightly worse than that of SMGD due to FSMGDA's sensitivity to objective and data heterogeneity in stochastic settings. These numerical results confirm our theoretical convergence analysis.
Table 2: Communication rounds needed for  ${10}^{-2}$  loss.
i.i.d.	non-i.i.d.
Task L	Task R	Task L	Task R
K=1	82	84	96	82
K=5	18(4.6×)	20(4.2×)	24(4.0×)	20(4.1×)
K=10	10(8.2×)	9(9.3×)	13(7.4×)	10(8.2×)
K=20	5(16.4×)	5(16.8×)	6(16.0×)	5(16.4×)
(a) 100 communication rounds with various local steps  $K$ , corresponding federated and centralized settings share the same marker shape.
Figure 2: Training losses comparison
(b) Normalized loss with the River Flow datasets.
2) Experiments on Larger FMOL: We further test our algorithms on FMOL problems of larger sizes. In this experiment, we use the River Flow dataset[54], which contains eight tasks in this problem. To better visualize 8 different tasks, we illustrate the normalized loss in radar charts in Fig. 2(b). In this 8-task setting, we can again verify that more local steps  $K$  and a larger training batch size lead to faster convergence. In the appendix, we also verify the effectiveness of our FMGDA and FSMGDA algorithms in CelebA [55] (40 tasks), alongside with other hyperparameter tuning results.
6 Conclusion and discussions
In this paper, we proposed the first general framework to extend multi-objective optimization to the federated learning paradigm, which considers both objective and data heterogeneity. We showed that, even under objective and data heterogeneity, both of our proposed algorithms enjoy the same Pareto stationary convergence rate as their centralized counterparts. In our future work, we will go beyond the limitation in the analysis of MOO that an extra assumption on the stochastic gradients (and  $\lambda$ ). In this paper, we have proposed a weaker assumption (Assumption 4). We conjecture that using acceleration techniques, e.g., momentum, variance reduction, and regularization, could relax such assumption and achieve better convergence rate, which is a promising direction for future works. In addition, MOO in distributed learning gives rise to substantially expensive communication costs, which scales linearly with the number of clients and the number of objectives in each client. Developing communication-efficient MOO beyond typical gradient compression methods for distributed learning is also a promising direction for future works.
This work has been supported in part by NSF grants CAREER CNS-2110259 and CNS-2112471.
A Gradient-based methods in MOO
(Stochastic) Gradient-based methods in MOO have attracted much attention owing to simple update rules and less intensive computation recently, thus rendering them perfect candidates to underpin MOO applications in deep learning under first-oracle. However, their theoretical understandings remain less explored relative to their counterparts of single objective optimization. Hence, we highlight the existing works and corresponding assumptions alongside with convergence metrics.
Existing Works. Various works managed to explore the convergence rates under different assumptions in strongly-convex, convex, and non-convex functions as listed in Table 4. Using full gradient, MGD [8] could achieve tight convergence rates in strongly-convex and non-convex cases, i.e., linear rate  $\mathcal{O}(r^T)$ ,  $r\in (0,1)$  and sub-linear rate  $\mathcal{O}\left(\frac{1}{T}\right)$ . However, it requires linear search of learning rate in the algorithm and sequence convergence ( $\{\mathbf{x}_t\}$  converges to  $\mathbf{x}_{*}$ ). The linear search of learning rate is a classic technique, but does not fit in gradient-based algorithms in deep learning. Moreover, sequence convergence assumption is a too strong assumption. With no local step, our FMGDA degenerates to MGD. As a result, our analysis also provide the same order convergence rates in both strongly-convex and non-convex functions while avoiding strong and unpractical assumptions. If using stochastic gradient, SMGD methods makes a further complicated case. The stochastic gradient noise would complicate the analysis and thus it is still unclear whether SMGD is guaranteed to converge. [9] provided convergence rate for SMGD but extra assumptions and/or unreasonably large batch requirements were needed. On the other hand, [9] and [18] showed that the common descent direction provided by SMGD method is likely to be a biased estimation, rendering non-convergence issues. Recently, by utilizing momentum, MoCo [19] and CR-MOGM [18] were proposed with corresponding convergence guarantees. [20] utilized direction-oriented approach by a preference direction. However, these analyses do not shed light on pure SMGD despite its widespread application.
Assumptions. When applying stochastic gradient to MOO, common descent direction estimation  $\pmb{\lambda}^T\nabla \mathbf{F}(\mathbf{x},\xi)$ $(\mathbf{F}(\mathbf{x}) = [f_{1}(\mathbf{x}),\dots ,f_{S}(\mathbf{x})])$  is a biased estimation and thus rendering potential nonconvergence issues [9, 18]. This is a limitation for SMGD. However, SMGD does work well with a wide range of applications in practice. Understanding under what conditions can SMGD have convergence guarantee is thus an important problem. [56] assumes convexity property(H5):  $f(\mathbf{x},\xi) - f(\mathbf{x}^{*},\xi)\geq \frac{c}{2}\| \mathbf{x} - \mathbf{x}^{*}\|^{2}$  almost sure. [9] utilizes weaker assumptions but still needs first moment bound (Assumption 5.2(b)):  $\mathbb{E}[\| \nabla f(\mathbf{x},\xi) - \nabla f(\mathbf{x})\| ]\leq \eta (a + b\| \nabla f(\mathbf{x})\|)$  and Lipschitz continuity of  $\lambda$  (Assumption 5.4):  $\| \pmb {\lambda}_k - \pmb {\lambda}_t\| \leq \beta \| \big[(\nabla f_1(\mathbf{x}_k) - \nabla f_1(\mathbf{x}_t))^T,\ldots ,(\nabla f_S(\mathbf{x}_k) - \nabla f_S(\mathbf{x}_t))^T\big]\|$ .
In this paper, we use  $(\alpha, \beta)$ -Lipschitz continuous stochastic gradient (Assumption 4). In essence, we need the stochastic gradient estimation satisfying  $\mathbb{E}[\| \nabla f(\mathbf{x}, \xi) - \nabla f(\mathbf{y}, \xi')\|^2] \leq \alpha \| \mathbf{x} - \mathbf{y}\|^2 + \beta \sigma^2$  for any two independent samples  $\xi$  and  $\xi'$ . For the inequality  $\mathbb{E}[\| \nabla f(\mathbf{x}, \xi) - \nabla f(\mathbf{y}, \xi')\|^2] \leq \alpha \| \mathbf{x} - \mathbf{y}\|^2 + \beta \sigma^2$  in Assumption 4, the notation  $\sigma^2$  just represents a general positive constant. This  $\sigma^2$  does not denote the variance of the stochastic gradient variance. Thus, this inequality does not
Table 3: List of key notation.
Notation	Definition
i	Client index
M	Total number of clients
s	Objective/task index
S	Total number of Objectives/tasks
Si	Number of objectives/tasks of client i's interest
k	Local step index
K	Total number of local steps
t	Communication round index
T	Total number of communication rounds
x ∈ Rd	Global model parameters of FMOL in Problem (2)
x0 ∈ Rd	Initial solution of FMOL in Problem (2)
x* ∈ Rd	A Pareto optimal solution of FMOL in Problem (2)
ηL	The learning rate on the client side
ηt	The learning rate on the server side in round t