SlowGuess/NeurIPS_2023_OCR huggingface dataset:
depend on the batch size of the stochastic gradient. More specifically, unlike the assumption in [9] that characterizes the difference between a stochastic gradient and its full gradient (hence depending on the batch size), our Assumption 4 only measures the average norm square of two stochastic gradient difference  $\nabla f(\mathbf{x},\xi) - \nabla f(\mathbf{y},\xi^{\prime})$  given any two points  $\mathbf{x}$  and  $\mathbf{y}$  and any two samples  $\xi$  and  $\xi^{\prime}$ . In other words, Assumption 4 does not involve any full gradient, and hence no dependence on batch size.
It is a natural extension of the classic Lipschitz continuous gradient assumption and could generalize existing assumptions.
1. If  $\xi$  and  $\xi^{\prime}$  are the whole dataset, by setting  $\alpha = L^{2}$  and  $\beta = 0$ ,  $(\alpha, \beta)$ -Lipschitz continuous stochastic gradient condition generalizes the traditional Lipschitz continuous gradient assumption  $\| \nabla f(\mathbf{x}) - \nabla f(\mathbf{y}) \| \leq L \| \mathbf{x} - \mathbf{y} \|$ .
2. If  $\xi$  is one data sample,  $\xi'$  are the whole dataset and  $\mathbf{x} = \mathbf{y}$ , by setting  $\alpha = 0$  and  $\beta = 1$ ,  $(\alpha, \beta)$ -Lipschitz continuous stochastic gradient condition generalizes the traditional bounded variance assumption  $\| \nabla f(\mathbf{x}, \xi) - \nabla f(\mathbf{x}) \|^{2} \leq \sigma^{2}$ .
3. If  $\xi$  is one data sample,  $\xi^{\prime}$  are the whole dataset and  $\mathbf{x} = \mathbf{y}$ , by setting  $\beta = \alpha_{k}$ ,  $(\alpha ,\beta)$ -Lipschitz continuous stochastic gradient condition generalizes the bound on the first moment assumption (assumption 5.2(b)) and bounded sets assumption (assumption 5.3) [9]  $(\mathbb{E}[\| \nabla f(\mathbf{x},\xi) - \nabla f(\mathbf{x})\| ]\leq \alpha_{k}(C_{i} + \hat{C}_{i}\| \nabla f_{i}(\mathbf{x}_{k})\|)$  and  $\| \nabla f_i(\mathbf{x})\| \leq M_\nabla +L\Theta$ .
Metrics. For strongly-convex functions, we use  $\Delta_{Q}^{t} = \sum_{s\in [S]}\lambda_{s}^{t,*}[f_{s}(\mathbf{x}_{t}) - f_{s}(\mathbf{x}_{*})]$  as the metrics. We note similar metrics are used in other works. For example, [9] uses  $\min_{t = 1,\dots,T}\sum_{s\in [S]}\left[\lambda_s^{t,*}f_s(\mathbf{x}_t) - \bar{\lambda}_Tf_s(\mathbf{x}_*)\right]$  where  $\bar{\lambda}_T = \sum_{t = 1}^T\frac{t}{\sum_{t = 1}^Tt}\lambda_t$ . Here  $\lambda_s^{t,*}$  is calculated by the quadratic programming problem 3 with stochastic gradients. Rigorously speaking, the left-hand side is not guaranteed to be positive. But if we impose stronger assumptions as shown in [9, 56], we can have the same convergence metric as that in single objective optimization as an direct extension. In non-convex functions,  $\|\bar{\mathbf{d}}_t\|^2$  are used as the metrics for FMOO, where  $\bar{\mathbf{d}}_t = \boldsymbol{\lambda}_t^T\nabla \mathbf{F}(\mathbf{x}_t)$  and  $\boldsymbol{\lambda}_t$  is calculated based on accumulated (stochastic) gradients  $\Delta_t$ . We note, directly extended from MOO [18, 19],  $\mathbf{d}_t^* = \hat{\boldsymbol{\lambda}}_t^{*T}\nabla \mathbf{F}(\mathbf{x}_t)$  could also be used as the metrics in FMOO, where  $\hat{\boldsymbol{\lambda}}_t^*$  is calculated based on full gradients  $\nabla \mathbf{F}(\mathbf{x}_t)$ . However, we prefer  $\bar{\mathbf{d}}_t$  for the following reasons: i). For applying gradient descent with no local steps,  $\bar{\mathbf{d}}_t$  degenerates to  $\mathbf{d}_t^*$ . ii). Clearly,  $\| \bar{\mathbf{d}}_t\| ^2\geq \| \mathbf{d}_t^*\| ^2$  as  $\hat{\boldsymbol{\lambda}}_t^*$  is calculated based on gradients  $\nabla \mathbf{F}(\mathbf{x}_t)$ . Hence,  $\| \bar{\mathbf{d}}_t\|^2$  is stronger convergence measure for FMOO. iii).  $\boldsymbol{\lambda}_t$  is calculated in the algorithm and thus being more practical to use in practice, while  $\hat{\boldsymbol{\lambda}}_t^*$  is unknown. Also, the convergence of  $\bar{\mathbf{d}}_t$  implicitly indicates  $\boldsymbol{\lambda}_t$  converges to  $\hat{\boldsymbol{\lambda}}_t^*$ .
B Proof of gradient descent type methods
For gradient descent type methods, each step utilizes a full gradient to update and the corresponding parameter  $\lambda$  is deterministic. For clarity of notation, we drop  $^*$  for  $\lambda$ , that is, we use  $\lambda_t^s$  to represent the solution of quadratic problem (Step 6 in the algorithm) for task  $s$  in the  $t$ -th round.
Lemma 1. Under bounded gradient assumption, the local model updates for any client  $s$  could be bounded
$$
G _ {s, i} ^ {t, k} = \left\| \mathbf {x} _ {s, i} ^ {t, k} - \mathbf {x} _ {t} \right\| ^ {2} \leq 4 \eta_ {L} ^ {2} K ^ {2} G ^ {2}, \tag {4}
$$
$$
H _ {t, s} = \left\| \nabla f _ {s} (\mathbf {x} _ {t}) - \Delta_ {s} ^ {t} \right\| ^ {2} \leq 4 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2}. \tag {5}
$$
Proof. For one task  $s \in [S]$  and one client  $i \in R_s$ , the local update  $\left\| \mathbf{x}_t - \mathbf{x}_{s,i}^{t,k} \right\|^2$  could be further bounded.
$$
\begin{array}{l} \left\| \mathbf {x} _ {t} - \mathbf {x} _ {s, i} ^ {t, k} \right\| ^ {2} = \left\| \mathbf {x} _ {t} - \mathbf {x} _ {s, i} ^ {t, k - 1} + \eta_ {L} \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, k - 1}\right) \right\| ^ {2} (6) \\ \leq \left(1 + \frac {1}{K - 1}\right) \left\| \mathbf {x} _ {t} - \mathbf {x} _ {s, i} ^ {t, k - 1} \right\| ^ {2} + \eta_ {L} ^ {2} K \left\| \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, k - 1}\right) \right\| ^ {2} (7) \\ \end{array}
$$
Table 4: Convergence rate (shaded parts are our results) for strongly-convex and non-convex functions, respectively:
Methods	Rate	Assumption
Setting	Algorithm	SC	NC
Vanilla Gradient	MGD [8]	O(rT), r ∈ (0,1)	O(1/T)	Sequence convergence
MGD	O(exp(-μT))	O(1/T)	-
SMGD [9]	O(1/T)	-	Lipschitz continuity of λ
SMGD [56]	O(1/T)	-	Convexity property
SMGD [39]	-	O(1/√T)	Given exact solution λ*
SMGD	O(1/T)	O(1/√T)	Asm. 4
Momentum	MoCo [19]	-	O(1/√T)	-
CR-MOGM [18]	-	O(1/√T)	-
Federated Settings	FMGDA	O(exp(-μT))	O(1/T)	-
FSMGDA	O(1/T)	O(1/√T)	Asm. 4
Assumptions. Linear search [8]: stepsize linear search; sequence convergence [8]:  $\{\mathbf{x}_t\}$  converges to  $\mathbf{x}_{*}$ ; first moment bound (Asm. 5.2(b) [9]):  $\mathbb{E}[\| \nabla f(\mathbf{x},\xi) - \nabla f(\mathbf{x})\| ]\leq \eta (a + b\| \nabla f(\mathbf{x})\|)$ ; Lipschitz continuity of  $\lambda$  (Asm. 5.4 [9]):  $\| \lambda_k - \lambda_s\| \leq \beta \| \left[(\nabla f_1(\mathbf{x}_k) - \nabla f_1(\mathbf{x}_t))^T,\dots ,(\nabla f_m(\mathbf{x}_k) - \nabla f_m(\mathbf{x}_t))^T\right]\|$ ; convexity property(H5) [56]:  $f(\mathbf{x},\xi) - f(\mathbf{x}^{*},\xi)\geq \frac{c}{2}\| \mathbf{x} - \mathbf{x}^{*}\|^{2}$  almost sure;  $(\alpha ,\beta)$ -Lipschitz continuous stochastic gradient (Asm. 4).
$$
\leq \left(1 + \frac {1}{K - 1}\right) \left\| \mathbf {x} _ {t} - \mathbf {x} _ {s, i} ^ {t, k - 1} \right\| ^ {2} + \eta_ {L} ^ {2} K G ^ {2} \tag {8}
$$
$$
\leq \sum_ {\tau \in [ k - 1 ]} \left(2 \eta_ {L} ^ {2} K G ^ {2}\right) \left(1 + \frac {1}{K - 1}\right) ^ {\tau} \tag {9}
$$
$$
\leq (K - 1) \left[ \left(1 + \frac {1}{K - 1}\right) ^ {K} - 1 \right] \left(\eta_ {L} ^ {2} K G ^ {2}\right) \tag {10}
$$
$$
\leq 4 \eta_ {L} ^ {2} K ^ {2} G ^ {2}, \tag {11}
$$
where the first inequality comes from Young's inequality, the second inequality follows from bounded gradient assumption, and the last inequality follows if  $\left(1 + \frac{1}{K - 1}\right)^K - 1 \leq 4$  for  $K > 1$ .
We have the bound for local update for each task  $s$ ,  $H_{t,s}$ , as follows:
$$
H _ {t, s} = \left\| \nabla f _ {s} \left(\mathbf {x} _ {t}\right) - \Delta_ {s} ^ {t} \right\| ^ {2} \tag {12}
$$
$$
= \left\| \frac {1}{K} \sum_ {k \in [ K ]} \frac {1}{| R _ {s} |} \sum_ {i \in R _ {s}} \left[ \nabla f _ {s, i} \left(\mathbf {x} _ {t}\right) - \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, k}\right) \right] \right\| ^ {2} \tag {13}
$$
$$
\leq \frac {1}{K} \sum_ {k \in [ K ]} \frac {1}{| R _ {s} |} \sum_ {i \in R _ {s}} \left\| \nabla f _ {s, i} \left(\mathbf {x} _ {t}\right) - \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, k}\right) \right\| ^ {2} \tag {14}
$$
$$
\leq \frac {1}{K} L ^ {2} \sum_ {k \in [ K ]} \frac {1}{| R _ {s} |} \sum_ {i \in R _ {s}} \left\| \mathbf {x} _ {t} - \mathbf {x} _ {s, i} ^ {t, k} \right\| ^ {2} \tag {15}
$$
$$
\leq 4 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2}. \tag {16}
$$
Lemma 2. For general  $L$ -smooth functions  $\{f_s, s \in [S]\}$ , choose the learning rate  $\eta_t$  s.t.  $\eta_t \leq \frac{3}{2(1 + L)}$ , the update  $d_t$  of the algorithm satisfies:
$$
\frac {\eta_ {t}}{4} \left\| \mathbf {d} _ {t} \right\| ^ {2} \leq - f _ {s} \left(\mathbf {x} _ {t + 1}\right) + f _ {s} \left(\mathbf {x} _ {t}\right) + 6 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2} \tag {17}
$$
$$
\begin{array}{l} f _ {s} \left(\mathbf {x} _ {t + 1}\right) \leq f _ {s} \left(\mathbf {x} _ {t}\right) + \left\langle \nabla f _ {s} \left(\mathbf {x} _ {t}\right), - \eta_ {t} \mathbf {d} _ {t} \right\rangle + \frac {1}{2} L \| \eta_ {t} \mathbf {d} _ {t} \| ^ {2} (18) \\ = f _ {s} \left(\mathbf {x} _ {t}\right) + \left\langle \nabla f _ {s} \left(\mathbf {x} _ {t}\right) - \Delta_ {s} ^ {t}, - \eta_ {t} \mathbf {d} _ {t} \right\rangle - \eta_ {t} \left\langle \Delta_ {s} ^ {t}, \mathbf {d} _ {t} \right\rangle + \frac {1}{2} L \| \eta_ {t} \mathbf {d} _ {t} \| ^ {2} (19) \\ \leq f _ {s} \left(\mathbf {x} _ {t}\right) + \left\langle \nabla f _ {s} \left(\mathbf {x} _ {t}\right) - \Delta_ {s} ^ {t}, - \eta_ {t} \mathbf {d} _ {t} \right\rangle - \eta_ {t} \| \mathbf {d} _ {t} \| ^ {2} + \frac {1}{2} L \| \eta_ {t} \mathbf {d} _ {t} \| ^ {2} (20) \\ \leq f _ {s} \left(\mathbf {x} _ {t}\right) + \frac {1}{2} \| \nabla f _ {s} \left(\mathbf {x} _ {t}\right) - \Delta_ {s} ^ {t} \| ^ {2} + \frac {1}{2} \eta_ {t} ^ {2} \| \mathbf {d} _ {t} \| ^ {2} - \eta_ {t} \| \mathbf {d} _ {t} \| ^ {2} + \frac {1}{2} L \eta_ {t} ^ {2} \| \mathbf {d} _ {t} \| ^ {2} (21) \\ = f _ {s} \left(\mathbf {x} _ {t}\right) + \frac {1}{2} \| \nabla f _ {s} \left(\mathbf {x} _ {t}\right) - \Delta_ {s} ^ {t} \| ^ {2} - \eta_ {t} \left(1 - \frac {1}{2} \eta_ {t} - \frac {1}{2} L \eta_ {t}\right) \| \mathbf {d} _ {t} \| ^ {2} (22) \\ \leq f _ {s} \left(\mathbf {x} _ {t}\right) + 2 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2} - \eta_ {t} \left(1 - \frac {1}{2} \eta_ {t} - \frac {1}{2} L \eta_ {t}\right) \| \mathbf {d} _ {t} \| ^ {2}. (23) \\ \end{array}
$$
The third inequality follows from  $\langle \Delta_s^t, \mathbf{d}_t \rangle \geq \| \mathbf{d}_t \|^2$  since  $\mathbf{d}_t$  is a general solution in the convex hull of the family of vectors  $\{\Delta_s^t, s \in [S]\}$  (see Lemma 2.1 [15]). Here  $\mathbf{d}_t = \sum_{s \in [S]} \lambda_s^{t,*} \Delta_s^t$  and  $\lambda_s^{t,*}$  is calculated by  $\Delta_s^t$ , but we drop the  $^*$  of  $\lambda$  for simplicity.
By setting  $\left(1 - \frac{1}{2}\eta_t - \frac{1}{2} L\eta_t\right) \geq \frac{1}{4}$ , that is,  $\eta_t \leq \frac{3}{2(1 + L)}$ , we have
$$
\frac {\eta_ {t}}{4} \left\| \mathbf {d} _ {t} \right\| ^ {2} \leq - f _ {s} \left(\mathbf {x} _ {t + 1}\right) + f _ {s} \left(\mathbf {x} _ {t}\right) + 2 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2}. \tag {24}
$$
B.1 Strongly Convex Functions
$$
f _ {s} \left(\mathbf {x} _ {t + 1}\right) \leq f _ {s} \left(\mathbf {x} _ {t}\right) + \left\langle \nabla f _ {s} \left(\mathbf {x} _ {t}\right), - \eta_ {t} \mathbf {d} _ {t} \right\rangle + \frac {1}{2} L \| \eta_ {t} \mathbf {d} _ {t} \| ^ {2} \tag {25}
$$
$$
\leq f _ {s} \left(\mathbf {x} _ {*}\right) + \left\langle \nabla f _ {s} \left(\mathbf {x} _ {t}\right), \mathbf {x} _ {t} - \mathbf {x} _ {*} \right\rangle - \frac {\mu}{2} \| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} \tag {26}
$$
$$
+ \left\langle \nabla f _ {s} \left(\mathbf {x} _ {t}\right), - \eta_ {t} \mathbf {d} _ {t} \right\rangle + \frac {1}{2} L \| \eta_ {t} \mathbf {d} _ {t} \| ^ {2}, \tag {27}
$$
where the first inequality is due to  $L$ -smoothness, the second inequality follows from  $\mu$ -strongly convex.
$$
\begin{array}{l} \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \left[ f _ {s} \left(\mathbf {x} _ {t + 1}\right) - f _ {s} \left(\mathbf {x} _ {*}\right) \right] (28) \\ \leq \left\langle \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \nabla f _ {s} (\mathbf {x} _ {t}), \mathbf {x} _ {t} - \mathbf {x} _ {*} \right\rangle - \frac {\mu}{2} \| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} + \left\langle \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \nabla f _ {s} (\mathbf {x} _ {t}), - \eta_ {t} \mathbf {d} _ {t} \right\rangle + \frac {1}{2} L \| \eta_ {t} \mathbf {d} _ {t} \| ^ {2} (29) \\ = \left\langle \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \nabla f _ {s} \left(\mathbf {x} _ {t}\right), \mathbf {x} _ {t} - \mathbf {x} _ {*} - \eta_ {t} \mathbf {d} _ {t} \right\rangle - \frac {\mu}{2} \| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} + \frac {1}{2} L \| \eta_ {t} \mathbf {d} _ {t} \| ^ {2} (30) \\ = \left\langle \mathbf {d} _ {t}, \mathbf {x} _ {t} - \mathbf {x} _ {*} - \eta_ {t} \mathbf {d} _ {t} \right\rangle - \frac {\mu}{2} \| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} + \frac {1}{2} L \| \eta_ {t} \mathbf {d} _ {t} \| ^ {2} + \left\langle \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \nabla f _ {s} (\mathbf {x} _ {t}) - \mathbf {d} _ {t}, \mathbf {x} _ {t} - \mathbf {x} _ {*} - \eta_ {t} \mathbf {d} _ {t} \right\rangle (31) \\ \end{array}
$$
$$
\begin{array}{l} = \left\langle \mathbf {d} _ {t}, \mathbf {x} _ {t} - \mathbf {x} _ {*} \right\rangle - \eta_ {t} \| \mathbf {d} _ {t} \| ^ {2} - \frac {\mu}{2} \| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} + \frac {1}{2} L \eta_ {t} ^ {2} \| \mathbf {d} _ {t} \| ^ {2} + \left\langle \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \nabla f _ {s} (\mathbf {x} _ {t}) - \mathbf {d} _ {t}, \mathbf {x} _ {t + 1} - \mathbf {x} _ {*} \right\rangle (32) \\ \leq \frac {1}{2 \eta_ {t}} \left(\| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} - \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {*} \| ^ {2}\right) - \frac {1}{2} \eta_ {t} \| \mathbf {d} _ {t} \| ^ {2} - \frac {\mu}{2} \| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} + \frac {1}{2} L \eta_ {t} ^ {2} \| \mathbf {d} _ {t} \| ^ {2} (33) \\ + \frac {1}{4 \epsilon} \underbrace {\left\| \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \nabla f _ {s} (\mathbf {x} _ {t}) - \mathbf {d} _ {t} \right\| ^ {2}} _ {H _ {t}} + \epsilon \left\| \mathbf {x} _ {t + 1} - \mathbf {x} _ {*} \right\| ^ {2} (34) \\ \leq \frac {1}{2 \eta_ {t}} \left(\left\| \mathbf {x} _ {t} - \mathbf {x} _ {*} \right\| ^ {2} - \left\| \mathbf {x} _ {t + 1} - \mathbf {x} _ {*} \right\| ^ {2}\right) - \frac {1}{2} \eta_ {t} \| \mathbf {d} _ {t} \| ^ {2} - \frac {\mu}{2} \| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} + \frac {1}{2} L \eta_ {t} ^ {2} \| d _ {t} \| ^ {2} (35) \\ + \frac {1}{4 \epsilon} H _ {t} + \epsilon \left(2 \| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} + 2 \eta_ {t} ^ {2} \| \mathbf {d} _ {t} \| ^ {2}\right) (36) \\ \leq \frac {1}{2 \eta_ {t}} \left(\left(1 - \frac {\mu}{2} \eta_ {t}\right) \| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} - \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {*} \| ^ {2}\right) - \left(\frac {1}{2} \eta_ {t} - \frac {1}{2} L \eta_ {t} ^ {2} - \frac {\mu}{4} \eta_ {t} ^ {2}\right) \| \mathbf {d} _ {t} \| ^ {2} + \frac {2}{\mu} H _ {t}, (37) \\ \end{array}
$$
where  $\| \mathbf{x}_t - \mathbf{x}_*\|^2 - \| \mathbf{x}_{t + 1} - \mathbf{x}_*\|^2 = -\eta_t^2\| \mathbf{d}_t\|^2 + 2\langle \eta_t\mathbf{d}_t,\mathbf{x}_t - \mathbf{x}_*\rangle$ , and we choose  $\epsilon = \frac{\mu}{8}$  in the last inequality.
From Lemma 2, it is clear that
$$
\begin{array}{l} \left| f _ {s} \left(\mathbf {x} _ {t + 1}\right) - f _ {s} \left(\mathbf {x} _ {t}\right) \right| \leq \left| 2 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2} - \frac {\eta_ {t}}{4} \| \mathbf {d} _ {t} \| ^ {2} \right| (38) \\ \leq 2 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2} + \frac {\eta_ {t}}{4} \| \mathbf {d} _ {t} \| ^ {2}. (39) \\ \end{array}
$$
$$
\begin{array}{l} \Delta_ {Q} ^ {t} = \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \left[ f _ {s} \left(\mathbf {x} _ {t}\right) - f _ {s} \left(\mathbf {x} _ {*}\right) \right] \leq \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \left[ f _ {s} \left(\mathbf {x} _ {t + 1}\right) - f _ {s} \left(\mathbf {x} _ {*}\right) \right] + \left| f _ {s} \left(\mathbf {x} _ {t + 1}\right) - f _ {s} \left(\mathbf {x} _ {t}\right) \right| (40) \\ \leq \frac {1}{2 \eta_ {t}} \left(\left(1 - \frac {\mu}{2} \eta_ {t}\right) \| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} - \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {*} \| ^ {2}\right) - \left(\frac {1}{4} \eta_ {t} - \frac {1}{2} L \eta_ {t} ^ {2} - \frac {\mu}{4} \eta_ {t} ^ {2}\right) \| \mathbf {d} _ {t} \| ^ {2} + \frac {2}{\mu} H _ {t} + 2 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2}. (41) \\ \end{array}
$$
$$
\begin{array}{l} H _ {t} = \left\| \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \nabla f _ {s} \left(\mathbf {x} _ {t}\right) - \mathbf {d} _ {t} \right\| ^ {2} (42) \\ \leq S \sum_ {s \in [ S ]} \left(\lambda_ {t} ^ {s}\right) ^ {2} H _ {t, s} (43) \\ \leq 4 \eta_ {L} ^ {2} K ^ {- 2} L ^ {2} G ^ {2} S ^ {2}. (44) \\ \end{array}
$$
By setting  $\eta_t \leq \frac{1}{2L + \mu}$ , we have
$$
\begin{array}{l} \Delta_ {Q} ^ {t} = \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \left[ f _ {s} \left(\mathbf {x} _ {t + 1}\right) - f _ {s} \left(\mathbf {x} _ {*}\right) \right] (45) \\ \leq \frac {1}{2 \eta_ {t}} \left((1 - \frac {\mu}{2} \eta_ {t}) \| \mathbf {x} _ {t} - \mathbf {x} _ {*} \| ^ {2} - \| \mathbf {x} _ {t + 1} - \mathbf {x} _ {*} \| ^ {2}\right) + \underbrace {\frac {8 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2} S ^ {2}}{\mu} + 2 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2}} _ {\delta}. (46) \\ \end{array}
$$
Averaging using weight  $w_{t} = (1 - \frac{\mu\eta}{2})^{1 - t}$  and using such weight to pick output  $\mathbf{x}$ . By using Lemma 1 in [27] with  $\eta \geq \frac{1}{uR}$ , we have
$$
\mathbb {E} \left[ \Delta_ {Q} \right] \leq \left\| \mathbf {x} _ {0} - \mathbf {x} _ {*} \right\| ^ {2} \mu \exp \left(- \frac {\eta \mu T}{2}\right) + \delta \tag {47}
$$
$$
= \mathcal {O} (\mu \exp (- \mu T)) + \mathcal {O} (\delta). \tag {48}
$$
If we set  $\eta_L$  sufficiently small such that  $\delta = \mathcal{O}(\mu \exp (-\mu T))$ , then we have the convergence rate  $\mathbb{E}[\Delta_Q] = \mathcal{O}(\mu \exp (-\mu T))$ .
B.2 Non-Convex Functions
Theorem 1 (FMGDA for Non-convex FMOL). Let  $\eta_t = \eta \leq \frac{3}{2(1 + L)}$ . Under Assumptions 1 and 2, if at least one function  $f_s$ ,  $s \in [S]$  is bounded from below by  $f_s^{\min}$ , then the sequence  $\{\mathbf{x}_t\}$  output by FMGDA satisfies:  $\min_{t \in [T]} \| \bar{\mathbf{d}}_t \|^2 \leq \frac{16(f_s^0 - f_s^{\min})}{T\eta} + \delta$ , where  $\delta \triangleq \frac{16\eta_L^2 K^2 L^2 G^2 (1 + S^2)}{\eta}$ .
Proof. From Lemma 2, we have
$$
\frac {\eta_ {t}}{4} \left\| \mathbf {d} _ {t} \right\| ^ {2} \leq - f _ {s} \left(\mathbf {x} _ {t + 1}\right) + f _ {s} \left(\mathbf {x} _ {t}\right) + 2 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2}. \tag {49}
$$
With constant learning rate  $\eta_t = \eta$
$$
\frac {1}{T} \sum_ {t \in [ T ]} \| \mathbf {d} _ {t} \| ^ {2} \leq \frac {4 \left(f _ {s} ^ {0} - f _ {s} ^ {\min }\right)}{T \eta} + \frac {8 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2}}{\eta}. \tag {50}
$$
Note that  $\left\| \bar{\mathbf{d}}_t\right\|^2$  are used as the metrics for FMOO, where  $\bar{\mathbf{d}}_t = \boldsymbol{\lambda}_t^T\nabla (\mathrm{Diag}(\mathbf{FA}^\top))$  and  $\boldsymbol{\lambda}_t$  is calculated based on accumulated (stochastic) gradients  $\Delta_t$ . Then we have
$$
\left\| \bar {\mathbf {d}} _ {t} \right\| ^ {2} \leq 2 \left\| \sum_ {s \in [ S ]} \lambda_ {t} ^ {s} \nabla f _ {s} (\mathbf {x} _ {t}) - \mathbf {d} _ {t} \right\| ^ {2} + 2 \| \mathbf {d} _ {t} \| ^ {2}. \tag {51}
$$
$$
\frac {1}{T} \sum_ {t \in [ T ]} \| \bar {\mathbf {d}} _ {t} \| ^ {2} \leq \frac {1 6 \left(f _ {s} ^ {0} - f _ {s} ^ {\min }\right)}{T \eta} + \frac {1 6 \eta_ {L} ^ {2} K ^ {2} L ^ {2} G ^ {2} \left(1 + S ^ {2}\right)}{\eta}. \tag {52}
$$
With constant learning rate  $\eta$  and local learning rate  $\eta_{L} = \mathcal{O}\left(\frac{1}{\sqrt{T}KLGS}\right)$ , we have
$$
\frac {1}{T} \sum_ {t \in [ T ]} \| \bar {\mathbf {d}} _ {t} \| ^ {2} \leq \mathcal {O} \left(\frac {1}{T}\right) \tag {53}
$$
C Proof of stochastic gradient descent type methods
For stochastic gradient descent type methods, each step utilizes a stochastic gradient to update and the corresponding parameter  $\lambda$  is stochastic, depending on the random samples in each client. For clarity of notation, we drop  $^*$  for  $\lambda$ , that is, we use  $\lambda_t^s$  to represent the solution of quadratic problem (Step 6 in the algorithm) for task  $s$  in the  $t$ -th round.
Lemma 3. Under bounded stochastic gradient assumption, the local model updates could be bounded
$$
G _ {s, i} ^ {t, k} = \mathbb {E} \left\| \mathbf {x} _ {s, i} ^ {t, k} - \mathbf {x} _ {t} \right\| ^ {2} \leq 6 \eta_ {L} ^ {2} k ^ {2} \| \nabla f _ {s, i} (\mathbf {x} _ {t}) \| ^ {2}, \tag {54}
$$
$$
\mathbb {E} \left\| \sum_ {s \in [ S ]} \lambda_ {s} ^ {t} \Delta_ {s} ^ {t} \right\| ^ {2} \leq S ^ {2} D ^ {2}. \tag {55}
$$
Further with assumption 4, we have
$$
H _ {t, s} = \mathbb {E} \left\| \nabla f _ {s} \left(\mathbf {x} _ {t}, \xi_ {t}\right) - \Delta_ {s} ^ {t} \right\| ^ {2} \leq \alpha \eta_ {L} ^ {2} K ^ {2} D ^ {2} + \beta \sigma^ {2}. \tag {56}
$$
$$
\begin{array}{l} G _ {s, i} ^ {t, k} = \mathbb {E} \left\| \mathbf {x} _ {t} - \mathbf {x} _ {s, i} ^ {t, k} \right\| ^ {2} (57) \\ = \mathbb {E} \left\| \sum_ {\tau \in [ k ]} \eta_ {L} \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, \tau}, \xi_ {s, i} ^ {t, \tau}\right) \right\| ^ {2} (58) \\ \leq \eta_ {L} ^ {2} k ^ {2} D ^ {2}. (59) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left\| \sum_ {s \in [ S ]} \lambda_ {s} ^ {t} \Delta_ {s} ^ {t} \right\| ^ {2} \leq S \sum_ {s \in [ S ]} \mathbb {E} \left[ \left(\lambda_ {s} ^ {t}\right) ^ {2} \| \Delta_ {s} ^ {t} \| ^ {2} \right] (60) \\ \leq S \sum_ {s \in [ S ]} \mathbb {E} \left[ \left\| \Delta_ {s} ^ {t} \right\| ^ {2} \right] (61) \\ \leq S \sum_ {s \in [ S ]} \mathbb {E} \left\| \frac {1}{R _ {s}} \sum_ {i \in R _ {s}} \frac {1}{K} \sum_ {\tau \in [ K ]} \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, \tau}, \xi_ {s, i} ^ {t, \tau}\right) \right\| ^ {2} (62) \\ \leq S \sum_ {s \in [ S ]} \frac {1}{R _ {s}} \sum_ {i \in R _ {s}} \frac {1}{K} \sum_ {\tau \in [ K ]} \mathbb {E} \left\| \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, \tau}, \xi_ {s, i} ^ {t, \tau}\right) \right\| ^ {2} (63) \\ \leq S ^ {2} D ^ {2}. (64) \\ \end{array}
$$
$$
\begin{array}{l} H _ {t, s} = \mathbb {E} \left\| \nabla f _ {s} \left(\mathbf {x} _ {t}, \xi_ {t}\right) - \Delta_ {s} ^ {t} \right\| ^ {2} (65) \\ \leq \mathbb {E} \left\| \frac {1}{K} \sum_ {k \in [ K ]} \frac {1}{| R _ {s} |} \sum_ {i \in R _ {s}} \left(\nabla f _ {s, i} \left(\mathbf {x} _ {t}, \xi_ {t}\right) - \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, k}, \xi_ {s, i} ^ {t, k}\right)\right) \right\| ^ {2} (66) \\ \leq \frac {1}{K} \sum_ {k \in [ K ]} \frac {1}{| R _ {s} |} \sum_ {i \in R _ {s}} \mathbb {E} \left\| \nabla f _ {s, i} \left(\mathbf {x} _ {t}, \xi_ {t}\right) - \nabla f _ {s, i} \left(\mathbf {x} _ {s, i} ^ {t, k}, \xi_ {s, i} ^ {t, k}\right) \right\| ^ {2} (67) \\ \leq \frac {1}{K} \sum_ {k \in [ K ]} \frac {1}{| R _ {s} |} \sum_ {i \in R _ {s}} \left(\alpha \mathbb {E} \| \mathbf {x} _ {t} - \mathbf {x} _ {s, i} ^ {t, k} \| ^ {2} + \beta \sigma^ {2}\right) (68) \\ \leq \alpha \eta_ {L} ^ {2} K ^ {2} D ^ {2} + \beta \sigma^ {2}. (69) \\ \end{array}
$$
C.1 Strongly Convex Functions
Theorem 7 (FSMGDA for  $\mu$ -Strongly Convex FMOL). Let  $\eta_t = \eta = \Omega\left(\frac{1}{\mu T}\right)$ . Under Assumptions 3, 5 and 6, pick  $\mathbf{x}_t$  as the final output of the FSMGDA algorithm with weight  $w_t = (1 - \frac{\mu\eta}{2})^{1 - t}$ . Then, it holds that:  $\mathbb{E}[\Delta_Q^t] \leq \| \mathbf{x}_0 - \mathbf{x}_* \|^2 \mu \exp(-\frac{\eta}{2} \mu T) + \delta$ , where  $\Delta_Q^t = \sum_{s \in [S]} \lambda_s^{t,*}[f_s(\mathbf{x}_t) - f_s(\mathbf{x}_*)]$  and  $\delta = \frac{1}{\mu} S^2 (\alpha \eta_L^2 K^2 D^2 + \beta \sigma^2) + \frac{\eta S^2 D^2}{2}$ .
Proof. Taking expectation over random samples conditioning on  $\mathbf{x}_t$ , we have