SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. In Proc. Advances in Neural Information Processing Systems, volume 33, virtual, 2020.
[2] Amir Beck and Marc Teboulle. Gradient-based algorithms with applications to signal-recovery problems. In Convex Optimization in Signal Processing and Communications, 2010.
[3] Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2:499-526, March 2002.
[4] Zachary Charles and Dimitris Papailiopoulos. Stability and generalization of learning algorithms that converge to global optima. In Proc. International Conference on Machine Learning, pages 744-753, Stockholm, Sweden, 2018.
[5] Lisha Chen, Heshan Fernando, Yiming Ying, and Tianyi Chen. Three-way trade-off in multi-objective learning: Optimization, generalization and conflict-avoidance. arXiv preprint arXiv:2305.20057, 2023.
[6] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proc. International Conference on Machine Learning, virtual, July 2018.
[7] Corinna Cortes, Mehryar Mohri, Javier Gonzalvo, and Dmitry Storches. Agnostic learning with multiple objectives. In Proc. Advances in Neural Information Processing Systems, volume 33, pages 20485-20495, virtual, 2020.
[8] Luc Devroye and Terry Wagner. Distribution-free performance bounds for potential function rules. IEEE Transactions on Information Theory, 25(5):601-604, 1979.
[9] Jean-Antoine Désideri. Multiple-gradient Descent Algorithm (MGDA) for Multi-objective Optimization. Comptes Rendus Mathematique, 350(5-6), 2012.
[10] Heshan Fernando, Han Shen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, and Tianyi Chen. Mitigating gradient bias in multi-objective learning: A provably convergent stochastic approach. In Proc. International Conference on Learning Representations, Kigali, Rwanda, 2023.
[11] Jörg Fliege, A Ismael F Vaz, and Luís Nunes Vicente. Complexity of Gradient Descent for Multi-objective Optimization. Optimization Methods and Software, 34(5):949-959, 2019.
[12] Xiang Gu, Xi Yu, Jian Sun, Zongben Xu, et al. Adversarial reweighting for partial domain adaptation. In Proc. Advances in Neural Information Processing Systems, virtual, December 2021.
[13] Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision, Munich, Germany, July 2018.
[14] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Proc. International Conference on Machine Learning, pages 1225-1234, New York City, NY, 2016.
[50] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Trans. Knowledge Data Eng., 2021.
[51] Yiyang Zhao, Linnan Wang, Kevin Yang, Tianjun Zhang, Tian Guo, and Yuandong Tian. Multi-objective optimization by learning space partitions. arXiv preprint arXiv:2110.03173, 2021.
[52] Shiji Zhou, Wenpeng Zhang, Jiyan Jiang, Wenliang Zhong, Jinjie Gu, and Wenwu Zhu. On the convergence of stochastic multi-objective gradient manipulation and beyond. In Proc. Advances in Neural Information Processing Systems, volume 35, pages 38103-38115, New Orleans, LA, December 2022.
Appendix for "Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance"
A Notations 15
B Bounding the generalization error 16
B.1 Proof of Propositions 2-3 16
B.2 Proof of Theorem 1 - PS generalization error in nonconvex case 18
B.3 Proof of Lemma 1 - Boundedness of  $x_{t}$  in strongly convex case 19
B.4 Proof of Theorem 2 - PS generalization error in strongly convex case 23
C Bounding the optimization error 36
C.1 Auxiliary lemmas 36
C.2 Proof of Lemma 2 - Distance to CA direction 38
C.3 Proof of Theorem 3-PS optimization error 40
D Additional experiments and implementation details 43
D.1 Synthetic experiments 43
D.2 Multi-task supervised learning experiments 45
A summary of notations used in this work is listed in Table 4 for ease of reference.
Table 4: Notations and their descriptions.
Notations	Descriptions
x∈Rd	Model parameter, or decision variable
z∈Z	Data point for training or testing
S∈Zn	Dataset such that S={z1,...,zn}
fz,m(x), fS,m(x)	A scalar-valued objective function evaluated on data point z, with fz,m: Rd←R, or on dataset S, fz,m, with fz,m := 1|S| ∑z∈S fz,m(x)
fm(x)	A scalar-valued population objective function, fm(x) := Ez[fz,m(x)]
∇fm(x)	Gradient of fm(x), with ∇fm(x): Rd←Rd
Fz(x), F_S(x)	A vector-valued objective function evaluated on data point z, with Fz: Rd←RM, or on dataset S, with FS := 1|S| ∑z∈S Fz(x)
F(x)	A vector-valued population objective, F(x) := Ez[iz,m(x)]
∇F(x)	Gradient of F(x), with ∇F(x): Rd←Rd×M
λ ∈ ΔM	Weighting parameter in an (M-1)-simplex
λρ(x) ∈ ΔM	CA weight, optimal solution to (2.3), when ρ = 0, it is simplified as λ*(x)
1 ∈ RM	All-one vector with dimension M
α	Step size to update model parameter x
γ	Step size to update weight λ
In the proof, we use  $\| \cdot \|$  to denote the spectral norm, and  $\| \cdot \|_{\mathrm{F}}$  to denote the Frobenius norm.
B Bounding the generalization error
B.1 Proof of Propositions 2-3
In this subsection, we prove Propositions 2-3, which establishes the relation between PS generalization error and MOL uniform stability.
Proof of Proposition 2. For a given model  $x$ , it holds that
$$
\begin{array}{l} R _ {\text {g e n}} (x) = \min  _ {\lambda \in \Delta^ {M}} \| \nabla F (x) \lambda \| - \min  _ {\lambda \in \Delta^ {M}} \| \nabla F _ {S} (x) \lambda \| \\ = - \max  _ {\lambda \in \Delta^ {M}} - \| \nabla F (x) \lambda \| + \max  _ {\lambda \in \Delta^ {M}} - \| \nabla F _ {S} (x) \lambda \| \\ \stackrel {(a)} {\leq} \max  _ {\lambda \in \Delta^ {M}} \left(\| \nabla F (x) \lambda \| - \| \nabla F _ {S} (x) \lambda \|\right) \stackrel {(b)} {\leq} \max  _ {\lambda \in \Delta^ {M}} \left(\| (\nabla F (x) - \nabla F _ {S} (x)) \lambda \|\right) \\ \stackrel {(c)} {\leq} \max  _ {\lambda \in \Delta^ {M}} \left(\| \nabla F (x) - \nabla F _ {S} (x) \| _ {\mathrm {F}} \| \lambda \| _ {\mathrm {F}}\right) \leq \| \nabla F (x) - \nabla F _ {S} (x) \| _ {\mathrm {F}} \tag {B.1} \\ \end{array}
$$
where  $(a)$  follows from the subadditivity of max operator,  $(b)$  follows from triangle inequality,  $(c)$  follows from Cauchy-Schwartz inequality.
Setting  $x = A(S)$ , and taking expectation over  $A, S$  on both sides of the above inequality, we have
$$
\mathbb {E} _ {A, S} \left[ R _ {\text {g e n}} (A (S)) \right] \leq \mathbb {E} _ {A, S} \left[ \| \nabla F (A (S)) - \nabla F _ {S} (A (S)) \| _ {\mathrm {F}} \right]. \tag {B.2}
$$
Proof of Proposition 3. The proof extends that of [22] for single objective learning to our MOL setting. Recall that  $S = \{z_{1},\ldots ,z_{n}\}$ , which are drawn i.i.d. from the data distribution  $\mathcal{D}$ . Define the perturbed dataset  $S^{(i)} = \{z_1,\dots ,z_i',\dots ,z_n\}$  sampled i.i.d. from  $\mathcal{D}$  with  $z_{i}^{\prime}$  independent of  $z_{j}$  for all  $i,j\in [n]$ . Let  $\tilde{z}$  be an independent sample of  $z_{j},z_{j}^{\prime}$ , for all  $j\in [n]$ , and from the same distribution  $\mathcal{D}$ . We first decompose the difference of population gradient and empirical gradient on the algorithm output  $n(\nabla F(A(S)) - \nabla F_S(A(S)))$  as follows using the gradient on  $A(S^{(i)})$ . Since  $\mathbb{E}_{\tilde{z}}[\nabla F_{\tilde{z}}(A(S))] = \nabla F(A(S))$ , it holds that
$$
\begin{array}{l} n \left(\nabla F (A (S)) - \nabla F _ {S} (A (S))\right) = n \mathbb {E} _ {\tilde {z}} \left[ \nabla F _ {\tilde {z}} (A (S)) \right] - n \nabla F _ {S} (A (S)) \\ = n \mathbb {E} _ {\tilde {z}} [ \nabla F _ {\tilde {z}} (A (S)) ] - \left(\sum_ {i = 1} ^ {n} \nabla F _ {z _ {i}} (A (S))\right) + \sum_ {i = 1} ^ {n} \left(\mathbb {E} _ {z _ {i} ^ {\prime}} [ \nabla F (A (S ^ {(i)})) ] - \mathbb {E} _ {z _ {i} ^ {\prime}} [ \nabla F (A (S ^ {(i)})) ]\right) \\ + \sum_ {i = 1} ^ {n} \left(\mathbb {E} _ {z _ {i} ^ {\prime}} [ \nabla F _ {z _ {i}} (A (S ^ {(i)})) ] - \mathbb {E} _ {z _ {i} ^ {\prime}} [ \nabla F _ {z _ {i}} (A (S ^ {(i)})) ]\right) \\ = \sum_ {i = 1} ^ {n} \mathbb {E} _ {\tilde {z}, z _ {i} ^ {\prime}} [ \nabla F _ {\tilde {z}} (A (S)) - \nabla F _ {\tilde {z}} (A (S ^ {(i)})) ] + \sum_ {i = 1} ^ {n} \underbrace {\mathbb {E} _ {z _ {i} ^ {\prime}} [ \mathbb {E} _ {\tilde {z}} [ \nabla F _ {\tilde {z}} (A (S ^ {(i)})) ] - \nabla F _ {z _ {i}} (A (S ^ {(i)})) ]} _ {\xi_ {i} (S)} \\ + \sum_ {i = 1} ^ {n} \mathbb {E} _ {z _ {i} ^ {\prime}} \left[ \nabla F _ {z _ {i}} \left(A \left(S ^ {(i)}\right)\right) - \nabla F _ {z _ {i}} (A (S)) \right] \tag {B.3} \\ \end{array}
$$
where the last equality follows from rearranging and that  $z_{i},z_{i}^{\prime},\tilde{z}$  are mutually independent. Applying triangle inequality to (B.3), it then follows that
$$
\begin{array}{l} n \| \nabla F (A (S)) - \nabla F _ {S} (A (S)) \| _ {\mathrm {F}} \leq \sum_ {i = 1} ^ {n} \mathbb {E} _ {\tilde {z}, z _ {i} ^ {\prime}} [ \| \nabla F _ {\tilde {z}} (A (S)) - \nabla F _ {\tilde {z}} (A (S ^ {(i)})) \| _ {\mathrm {F}} ] + \left\| \sum_ {i = 1} ^ {n} \xi_ {i} (S) \right\| _ {\mathrm {F}} \\ + \sum_ {i = 1} ^ {n} \mathbb {E} _ {z _ {i} ^ {\prime}} [ \| \nabla F _ {z _ {i}} (A (S ^ {(i)})) - \nabla F _ {z _ {i}} (A (S)) \| _ {\mathrm {F}} ]. \tag {B.4} \\ \end{array}
$$
Note  $S$  and  $S^{(i)}$  differ by a single sample. By Definition 2, the MOL uniform stability  $\epsilon_{\mathrm{F}}$ , and Jensen's inequality, we further get
$$
n \mathbb {E} \left[ \| \nabla F (A (S)) - \nabla F _ {S} (A (S)) \| _ {\mathrm {F}} \right] \leq 2 n \epsilon_ {\mathrm {F}} + \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {n} \xi_ {i} (S) \right\| _ {\mathrm {F}} \right]. \tag {B.5}
$$
We then proceed to bound  $\mathbb{E}\big[\big\| \sum_{i = 1}^{n}\xi_{i}(S)\big\|_{\mathrm{F}}\big]$ , which satisfies
$$
\left(\mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {n} \xi_ {i} (S) \right\| _ {\mathrm {F}} \right]\right) ^ {2} \leq \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {n} \xi_ {i} (S) \right\| _ {\mathrm {F}} ^ {2} \right] = \sum_ {i = 1} ^ {n} \underbrace {\mathbb {E} \left[ \| \xi_ {i} (S) \| _ {\mathrm {F}} ^ {2} \right]} _ {J _ {1, i}} + \sum_ {i, j \in [ n ]: i \neq j} \underbrace {\mathbb {E} [ \langle \xi_ {i} (S) , \xi_ {j} (S) \rangle ]} _ {J _ {2, i, j}}. \tag {B.6}
$$
For  $J_{1,i}$ , according to the definition of  $\xi_i(S)$  in (B.3) and Jensen inequality, it holds that
$$
\begin{array}{l} J _ {1, i} = \mathbb {E} [ \| \xi_ {i} (S) \| _ {\mathrm {F}} ^ {2} ] = \mathbb {E} \left[ \left\| \mathbb {E} _ {z _ {i} ^ {\prime}} \left[ \mathbb {E} _ {\tilde {z}} [ \nabla F _ {\tilde {z}} (A (S ^ {(i)})) ] - \nabla F _ {z _ {i}} (A (S ^ {(i)})) \right] \right\| _ {\mathrm {F}} ^ {2} \right] \\ \stackrel {(a)} {\leq} \mathbb {E} \left[ \left| \left| \mathbb {E} _ {\tilde {z}} [ \nabla F _ {\tilde {z}} (A (S ^ {(i)})) ] - \nabla F _ {z _ {i}} (A (S ^ {(i)})) \right| \right| _ {\mathrm {F}} ^ {2} \right] \\ \stackrel {(b)} {=} \mathbb {E} \left[ \left\| \mathbb {E} _ {\tilde {z}} [ \nabla F _ {\tilde {z}} (A (S)) ] - \nabla F _ {z _ {i} ^ {\prime}} (A (S)) \right\| _ {\mathrm {F}} ^ {2} \right] \\ = \mathbb {E} \left[ \mathbb {V} _ {\tilde {z}} \left(\nabla F _ {\tilde {z}} (A (S))\right) \right], \tag {B.7} \\ \end{array}
$$
where  $(a)$  follows from Jensen's inequality,  $(b)$  follows from the symmetry between  $z_{i}$  and  $z_{i}^{\prime}$ . To bound  $J_{2,i,j}$  with  $i\neq j$ , further introduce  $S'' = \{z_1'',\ldots ,z_n''\}$  which are drawn i.i.d. from the data distribution  $\mathcal{D}$ . Then for each  $i,j\in [n]$  with  $i\neq j$ , introduce  $S_{j}$  as a neighboring dataset of  $S$  by replacing its  $z_{j}$  with  $z_{j}^{\prime \prime}$ , and  $S_j^{(i)}$  as a neighboring dataset of  $S^{(i)}$  by replacing its  $z_{j}$  with  $z_{j}^{\prime \prime}$ , i.e.,
$$
S _ {j} = \left\{z _ {1}, \dots , z _ {j - 1}, z _ {j} ^ {\prime \prime}, z _ {j + 1}, \dots , z _ {n} \right\}, \tag {B.8a}
$$
$$
S _ {j} ^ {(i)} = \left\{z _ {1}, \dots , z _ {i - 1}, z _ {i} ^ {\prime}, z _ {i + 1}, \dots , z _ {j - 1}, z _ {j} ^ {\prime \prime}, z _ {j + 1}, \dots , z _ {n} \right\}. \tag {B.8b}
$$
Then the idea is to bound  $J_{2,i,j}$  using the newly introduced neighboring datasets  $S_{j}$  and  $S_{j}^{(i)}$ , so as to connect to the definition of the stability  $\epsilon_{\mathrm{F}}$ . We first show that  $\mathbb{E}\left[\langle \xi_i(S),\xi_j(S)\rangle \right] = \mathbb{E}\left[\langle \xi_i(S) - \xi_i(S_j),\xi_j(S) - \xi_j(S_i)\rangle \right]$  because for  $i\neq j$
$$
\mathbb {E} \left[ \langle \xi_ {i} (S _ {j}), \xi_ {j} (S) \rangle \right] \stackrel {(c)} {=} 0, \mathbb {E} \left[ \langle \xi_ {i} (S _ {j}), \xi_ {j} (S _ {i}) \rangle \right] \stackrel {(d)} {=} 0, \mathbb {E} \left[ \langle \xi_ {i} (S _ {j}), \xi_ {j} (S _ {i}) \rangle \right] \stackrel {(e)} {=} 0. \tag {B.9}
$$
For  $i\neq j$ $(c)$  follows from
$$
\mathbb {E} \left[ \langle \xi_ {i} (S _ {j}), \xi_ {j} (S) \rangle \right] = \mathbb {E} \mathbb {E} _ {z _ {j}} \left[ \langle \xi_ {i} (S _ {j}), \xi_ {j} (S) \rangle \right] = \mathbb {E} \left[ \left\langle \xi_ {i} (S _ {j}), \mathbb {E} _ {z _ {j}} [ \xi_ {j} (S) ] \right\rangle \right] = 0, \tag {B.10}
$$
where the second identity holds since  $\xi_{i}(S_{j})$  is independent of  $z_{j}$  and the last identity follows from  $\mathbb{E}_{z_j}[\xi_j(S)] = 0$  due to the symmetry between  $\tilde{z}$  and  $z_{i}$ , and their independence with  $S^{(i)}$ , derived as
$$
\mathbb {E} _ {z _ {i}} \left[ \xi_ {i} (S) \right] = \mathbb {E} _ {z _ {i}} \left[ \mathbb {E} _ {z _ {i} ^ {\prime}} \left[ \mathbb {E} _ {\tilde {z}} \left[ \nabla F _ {\tilde {z}} \left(A \left(S ^ {(i)}\right)\right) \right] - \nabla F _ {z _ {i}} \left(A \left(S ^ {(i)}\right)\right) \right] \right] = 0, \quad \forall i \in [ n ]. \tag {B.11}
$$
In a similar way, for  $i \neq j$ ,  $(d)$  and  $(e)$  follow from
$$
\mathbb {E} \left[ \left\langle \xi_ {i} (S), \xi_ {j} (S _ {i}) \right\rangle \right] = \mathbb {E} \mathbb {E} _ {z _ {i}} \left[ \left\langle \xi_ {i} (S), \xi_ {j} (S _ {i}) \right\rangle \right] = \mathbb {E} \left[ \left\langle \xi_ {j} (S _ {i}), \mathbb {E} _ {z _ {i}} \left[ \xi_ {i} (S) \right] \right\rangle \right] = 0, \tag {B.12}
$$
$$
\mathbb {E} \left[ \left\langle \xi_ {i} \left(S _ {j}\right), \xi_ {j} \left(S _ {i}\right) \right\rangle \right] = \mathbb {E} _ {z _ {i}} \left[ \left\langle \xi_ {i} \left(S _ {j}\right), \xi_ {j} \left(S _ {i}\right) \right\rangle \right] = \mathbb {E} \left[ \left\langle \xi_ {j} \left(S _ {i}\right), \mathbb {E} _ {z _ {i}} \left[ \xi_ {i} \left(S _ {j}\right) \right] \right\rangle \right] = 0. \tag {B.13}
$$
Based on (B.9), for  $i \neq j$  we have
$$
\begin{array}{l} J _ {2, i, j} = \mathbb {E} \left[ \langle \xi_ {i} (S), \xi_ {j} (S) \rangle \right] = \mathbb {E} \left[ \langle \xi_ {i} (S) - \xi_ {i} (S _ {j}), \xi_ {j} (S) - \xi_ {j} (S _ {i}) \rangle \right] \\ \leq \mathbb {E} \left[ \| \xi_ {i} (S) - \xi_ {i} (S _ {j}) \| _ {\mathbf {F}} \| \xi_ {j} (S) - \xi_ {j} (S _ {i}) \| _ {\mathbf {F}} \right] \\ \leq \frac {1}{2} \mathbb {E} \left[ \| \xi_ {i} (S) - \xi_ {i} \left(S _ {j}\right) \| _ {\mathrm {F}} ^ {2} \right] + \frac {1}{2} \mathbb {E} \left[ \| \xi_ {j} (S) - \xi_ {j} \left(S _ {i}\right) \| _ {\mathrm {F}} ^ {2} \right] \tag {B.14} \\ \end{array}
$$
where we have used  $ab \leq \frac{1}{2} (a^2 + b^2)$ . According to the definition of  $\xi_i(S)$  and  $\xi_i(S_j)$  we know the following identity for  $i \neq j$
$$
\begin{array}{l} \mathbb {E} \left[ \| \xi_ {i} (S) - \xi_ {i} (S _ {j}) \| _ {\mathrm {F}} ^ {2} \right] = \mathbb {E} \left[ \left\| \mathbb {E} _ {z _ {i} ^ {\prime}} \mathbb {E} _ {\tilde {z}} \left[ \nabla F _ {\tilde {z}} (A (S ^ {(i)})) - \nabla F _ {\tilde {z}} (A (S _ {j} ^ {(i)})) \right] \right. \right. \\ \left. + \mathbb {E} _ {z _ {i} ^ {\prime}} \left[ \nabla F _ {z _ {i}} \left(A \left(S _ {j} ^ {(i)}\right)\right) - \nabla F _ {z _ {i}} \left(A \left(S ^ {(i)}\right)\right) \right] \right\| _ {\mathrm {F}} ^ {2} \bigg ]. \tag {B.15} \\ \end{array}
$$
It then follows from the inequality  $(a + b)^2 \leq 2(a^2 + b^2)$  and the Jensen's inequality that
$$
\begin{array}{l} \mathbb {E} \left[ \| \xi_ {i} (S) - \xi_ {i} (S _ {j}) \| _ {\mathrm {F}} ^ {2} \right] \leq 2 \mathbb {E} \left[ \| \nabla F _ {\tilde {z}} \left(A \left(S ^ {(i)}\right)\right) - \nabla F _ {\tilde {z}} \left(A \left(S _ {j} ^ {(i)}\right)\right) \| _ {\mathrm {F}} ^ {2} \right] \\ + 2 \mathbb {E} [ \| \nabla F _ {z _ {i}} (A (S _ {j} ^ {(i)})) - \nabla F _ {z _ {i}} (A (S ^ {(i)})) \| _ {\mathrm {F}} ^ {2} ]. \tag {B.16} \\ \end{array}
$$
Since  $S^{(i)}, S_{j}^{(i)}$  and  $S^{(j)}, S_{i}^{(j)}$  are two pairs of neighboring datasets, it follows from the definition of stability that
$$
\mathbb {E} \left[ \| \xi_ {i} (S) - \xi_ {i} (S _ {j}) \| _ {\mathrm {F}} ^ {2} \right] \leq 4 \epsilon_ {\mathrm {F}} ^ {2}, \text {a n d} \mathbb {E} \left[ \| \xi_ {j} (S) - \xi_ {j} (S _ {i}) \| _ {\mathrm {F}} ^ {2} \right] \leq 4 \epsilon_ {\mathrm {F}} ^ {2}, \quad \forall i \neq j. \tag {B.17}
$$
We can plug the above inequalities back into (B.14) and bound  $J_{2,i,j}$  by
$$
J _ {2, i, j} = \mathbb {E} \left[ \left\langle \xi_ {i} (S), \xi_ {j} (S) \right\rangle \right] \leq 4 \epsilon_ {\mathrm {F}} ^ {2}, \quad \forall i \neq j. \tag {B.18}
$$
Combining the bound for  $J_{1,i}$  in (B.7) and  $J_{2,i,j}$  in (B.18) and substituting them back into (B.6), it then follows that
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \sum_ {i = 1} ^ {n} \xi_ {i} (S) \right\| _ {\mathrm {F}} ^ {2} \right] = \mathbb {E} \left[ \sum_ {i = 1} ^ {n} \| \xi_ {i} (S) \| _ {\mathrm {F}} ^ {2} \right] + \sum_ {i, j \in [ n ]: i \neq j} \mathbb {E} [ \langle \xi_ {i} (S), \xi_ {j} (S) \rangle ] \\ \leq n \mathbb {E} \left[ \mathbb {V} _ {\tilde {z}} \left(\nabla F _ {\tilde {z}} (A (S))\right) \right] + 4 n (n - 1) \epsilon_ {\mathrm {F}} ^ {2}. \tag {B.19} \\ \end{array}
$$
Plugging the above inequality back into (B.5), using the subadditivity of the square root function, we get
$$
n \mathbb {E} [ \| \nabla F (A (S)) - \nabla F _ {S} (A (S)) \| _ {\mathrm {F}} ] \leq 4 n \epsilon_ {\mathrm {F}} + \sqrt {n \mathbb {E} \left[ \mathbb {V} _ {\tilde {z}} (\nabla F _ {\tilde {z}} (A (S))) \right]}. \tag {B.20}
$$
B.2 Proof of Theorem 1 - PS generalization error in nonconvex case
In this subsection, we prove Theorem 1, which establishes the PS generalization error of MoDo in the nonconvex case.
Organization of proof. To prove the PS generalization error of MoDo, we first define the concept of Sampling-determined algorithms in Definition 3. This concept has been defined in [22] for the analysis in single-objective learning. Then we show that MoDo is sampling-determined in Proposition 4. Finally, combining Propositions 2-4, we can prove Theorem 1, the MOL uniform stability and PS generalization error of MoDo.
Definition 3 (Sampling-determined algorithm [22]). Let  $A$  be a randomized algorithm that randomly chooses an index sequence  $I(A) = \{i_{t,s}\}$  to compute stochastic gradients. We say a symmetric algorithm  $A$  is sampling-determined if the output model is fully determined by  $\{z_i : i \in I(A)\}$ .
Proposition 4 (MoDo is sampling determined). MoDo (Algorithm 1) is sampling determined. In other words, Let  $I(A) = \{i_t\}$  be the sequence of index chosen by these algorithms from training set  $S = \{z_1,\dots ,z_n\}$ , and  $z_{i}\stackrel {\mathrm{i.i.d.}}{\sim}\mathcal{P}$  for all  $i\in [n]$  to build stochastic gradients, the output  $A(S)$  is determined by  $\{z_j:j\in I(A)\}$ . To be precise,  $A(S)$  is independent of  $z_{j}$  if  $j\notin I(A)$ .
Proof of Proposition 4. Let  $I(A) = \{I_1, \dots, I_T\}$ ,  $I_t = \{i_{t,s}\}_{s=1}^3$  and  $i_{t,s} \in [n]$  for all  $1 \leq t \leq T$ . And  $S_{I(A)} = \{z_{i_{t,s}}\}$ . By the description in Algorithm 1,  $A(S) = G_{z_{I_T}} \circ \dots \circ G_{z_{I_1}}(x_0)$ , where  $G_z(\cdot)$  is the stochastic update function of the model parameter given random sample  $z$ . Therefore, for all possible sample realization  $z$ , we have
$$
\begin{array}{l} \mathbb {P} (A (S) = x \mid z _ {j} = z, j \notin I (A)) = \mathbb {P} \left(G _ {z _ {I _ {T}}} \circ \dots \circ G _ {z _ {I _ {1}}} \left(x _ {0}\right) = x \mid z _ {j} = z, j \notin I (A)\right) \\ = \mathbb {P} \left(G _ {z _ {I _ {T}}} \circ \dots \circ G _ {z _ {I _ {1}}} \left(x _ {0}\right) = x \mid j \notin I (A)\right) \\ = \mathbb {P} (A (S) = x \mid j \notin I (A)) \tag {B.21} \\ \end{array}
$$
where the last equality holds because  $z_{j} \notin S_{I(A)}$ , and  $z_{j}$  is independent of all elements in  $S_{I(A)}$  by i.i.d. sampling. Therefore,  $A(S)$  is independent of  $z_{j}$  if  $j \notin I(A)$ . The proof is complete.
Note that, besides MoDo, other popular stochastic randomized MTL algorithms such as SMG [30] and MoCo [10] are also sampling-determined. Therefore, the result is also applicable to these algorithms.
Lemma 3 ([22, Theorem 5 (b)]). Let  $A$  be a sampling-determined random algorithm (Definition 3) and  $S, S'$  be neighboring datasets with  $n$  data points that differ only in the  $i$ -th data point. If  $\sup_z \mathbb{E}_A\left[\| \nabla F_z(A(S))\|_{\mathrm{F}}^2 \mid i \in I(A)\right] \leq G^2$  for any  $S$ , then
$$
\sup  _ {z} \mathbb {E} _ {A} [ \| \nabla F _ {z} (A (S)) - \nabla F _ {z} (A \left(S ^ {\prime}\right)) \| _ {\mathrm {F}} ^ {2} ] \leq 4 G ^ {2} \cdot \mathbb {P} \{i \in I (A) \}. \tag {B.22}
$$
Proof of Theorem 1. From Proposition 4, algorithm  $A$ , MoDo is sampling-determined. Then based on Lemma 3, its MOL uniform stability in Definition 2 can be bounded by
$$
\epsilon_ {\mathrm {F}} ^ {2} \leq 4 G ^ {2} \cdot \mathbb {P} \{i \in I (A) \}. \tag {B.23}
$$
Let  $i_{t}$  be the index of the sample selected by  $A$  at the  $t$ -th step, and  $i^{*}$  be the index of the data point that is different in  $S$  and  $S'$ . Then
$$
\mathbb {P} \{i ^ {*} \in I (A) \} \leq \sum_ {t = 0} ^ {T - 1} \mathbb {P} \left\{i _ {t} = i ^ {*} \right\} \leq \frac {T}{n}. \tag {B.24}
$$
Combining (B.23) and (B.24) gives
$$
\epsilon_ {\mathrm {F}} ^ {2} \leq \frac {4 G ^ {2} T}{n}. \tag {B.25}
$$
Then based on Propositions 2-3, we have
$$
\begin{array}{l} \mathbb {E} _ {A, S} \left[ R _ {\text {g e n}} (A (S)) \right] \leq \mathbb {E} _ {A, S} \left[ \| \nabla F (A (S)) - \nabla F _ {S} (A (S)) \| _ {\mathrm {F}} \right] \quad \text {b y} \\ \leq 4 \epsilon_ {\mathrm {F}} + \sqrt {n ^ {- 1} \mathbb {E} _ {S} \left[ \mathbb {V} _ {z \sim \mathcal {D}} (\nabla F _ {z} (A (S))) \right]} \quad \text {b y} \\ = \mathcal {O} \left(T ^ {\frac {1}{2}} n ^ {- \frac {1}{2}}\right) \quad \text {b y} \tag {B.25} \\ \end{array}
$$
B.3 Proof of Lemma 1 - Boundedness of  $x_{t}$  in strongly convex case
Technical challenges. In this work, we focus on analyzing stochastic MGDA-based MOL algorithms in the unconstrained setting. This is because in the constrained setting, MGDA with projected gradient descent on  $x$  has no guarantee to find the CA direction, and a new algorithm needs to be developed to achieve this [42]. However, a fundamental challenge in the unconstrained strongly convex setting is that a strongly convex function is not Lipschitz continuous on  $\mathbb{R}^d$ . We overcome this challenge by showing that  $\{x_t\}_{t=1}^T$  generated by the MoDo algorithm is bounded on the trajectory, so is the gradient  $\| \nabla f_{z,m}(x_t) \|$  for all  $m \in [M]$ , and  $z \in \mathcal{Z}$ . Thereby, we can derive the upper bound of PS optimization and generalization errors without the Lipschitz continuity assumption for strongly convex objectives.