SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this part, we first summarize all the notations that we used in this paper in order to help readers understand. First, in multi-objective optimization, we have  $K \geq 2$  different objectives and each of them has the loss function  $L_{i}(\theta)$ . Let  $g_{i}$  denote the gradient of objective  $i$  and  $g_{0}$  denotes the target gradient.  $w = (w_{1},\dots,w_{K})^{T} \in \mathbb{R}^{K}$  and  $\mathcal{W}$  denotes the probability simplex. Other useful notations are listed as below:
$$
\theta^ {*} = \arg \min  _ {\theta \in \mathbb {R} ^ {m}} \left\{L _ {0} (\theta) \triangleq \frac {1}{K} \sum_ {i = 1} ^ {K} L _ {i} (\theta) \right\}, g _ {0} = g _ {0} (\theta) = G (\theta) \widetilde {w}
$$
$$
g _ {w} = \sum_ {i} w _ {i} g _ {i} \quad s. t. \quad \mathcal {W} = \{w: \sum_ {i} w _ {i} = 1 \text {a n d} w _ {i} \geq 0 \}
$$
$$
w _ {\lambda} ^ {*} = \arg \min  _ {w \in \mathcal {W}} \frac {1}{2} \| g _ {w} + \lambda g _ {0} \| ^ {2}, w ^ {*} = \arg \min  _ {w \in \mathcal {W}} \frac {1}{2} \| g _ {w} \| ^ {2}, w _ {t} ^ {*} = \arg \min  _ {w \in \mathcal {W}} \frac {1}{2} \| G (\theta_ {t}) w \| ^ {2}
$$
$$
w _ {t, \lambda} ^ {*} = \arg \min  _ {w \in \mathcal {W}} F (w) = \arg \min  _ {w \in \mathcal {W}} \frac {1}{2} \| G (\theta_ {t}) w + \lambda g _ {0} (\theta_ {t}) \| ^ {2}
$$
$$
\nabla_ {w} F (w) = G \left(\theta_ {t}\right) ^ {T} \left(G \left(\theta_ {t}\right) w + \lambda g _ {0} \left(\theta_ {t}\right)\right), \nabla_ {w} \widehat {F} (w) = G \left(\theta_ {t}; \xi\right) ^ {T} \left(G \left(\theta_ {t}, \xi^ {\prime}\right) w + \lambda g _ {0} \left(\theta_ {t}, \xi^ {\prime}\right)\right). \tag {11}
$$
We use  $\mathbb{E}[\cdot ]_{A|B}$  to denote taking expectation over  $A$  conditioning on  $B$  and  $\tilde{\mathcal{O}}$  omits the order of log.
C Detailed proofs for convergence analysis with nonconvex Objectives
We now provide some auxiliary lemmas for proving Proposition 1 and Theorem 1
Lemma 1. Let  $d^{*}$  be the solution of
$$
\max _ {d \in \mathbb {R} ^ {m}} \min _ {i \in [ K ]} \langle g _ {i}, d \rangle - \frac {1}{2} \| d \| ^ {2} + \lambda \langle g _ {0}, d \rangle ,
$$
$$
d ^ {*} = g _ {w _ {\lambda} ^ {*}} + \lambda g _ {0}.
$$
In addition,  $w_{\lambda}^{*}$  is the solution of
$$
\min  _ {w \in \mathcal {W}} \frac {1}{2} \| g _ {w} + \lambda g _ {0} \| ^ {2}.
$$
Proof. First, it can be seen that
$$
\begin{array}{l} \max  _ {d \in \mathbb {R} ^ {m}} \min  _ {i \in [ K ]} \langle g _ {i}, d \rangle - \frac {1}{2} \| d \| ^ {2} + \lambda \langle g _ {0}, d \rangle \\ = \max  _ {d \in \mathbb {R} ^ {m}} \min  _ {w \in \mathcal {W}} \left\langle \sum_ {i} w _ {i} g _ {i}, d \right\rangle - \frac {1}{2} \| d \| ^ {2} + \lambda \langle g _ {0}, d \rangle \\ = \max  _ {d \in \mathbb {R} ^ {m}} \min  _ {w \in \mathcal {W}} g _ {w} ^ {T} d - \frac {1}{2} \| d \| ^ {2} + \lambda \langle g _ {0}, d \rangle . \tag {12} \\ \end{array}
$$
Noting that the problem is concave w.r.t.  $d$  and convex w.r.t  $w$  and using the Von Neumann-Fan minimax theorem [50], we can exchange the min and max problems without changing the solution. Then, we can solve the following equivalent problem.
$$
\min  _ {w \in \mathcal {W}} \max  _ {d \in \mathbb {R} ^ {m}} g _ {w} ^ {T} d - \frac {1}{2} \| d \| ^ {2} + \lambda \langle g _ {0}, d \rangle \tag {13}
$$
Then by fixing  $w$ , we have  $d^{*} = g_{w} + \lambda g_{0}$ . Substituting this solution to the eq. (13) and rearranging the equation, we turn to solve the following problem.
Let  $w_{\lambda}^{*}$  be the solution of the above problem, and hence the final updating direction  $d^{*} = g_{w_{\lambda}^{*}} + \lambda g_{0}$ . Then, the proof is complete.
Lemma 2. Suppose Assumption 2-3 are satisfied. According to the definition of  $g_0(\theta)$  in eq. (11), we have the following inequalities,
$$
\left\| g _ {0} (\theta) \right\| \leq C _ {g}, \quad \mathbb {E} [ \left\| g _ {0} (\theta ; \xi) - g _ {0} (\theta) \right\| ^ {2} ] \leq K \sigma_ {0} ^ {2}.
$$
Proof. Based on the definitions, we have
$$
\left\| g _ {0} (\theta) \right\| = \left\| G (\theta) \widetilde {w} \right\| \leq C _ {g},
$$
where the inequality follows from the fact that  $\| \widetilde{w}\| \leq 1$  and Assumption 3. Then, we have
$$
\mathbb {E} _ {\xi} [ \| g _ {0} (\theta ; \xi) - g _ {0} (\theta) \| ^ {2} ] \leq \mathbb {E} _ {\xi} [ \| G (\theta ; \xi) - G (\theta) \| ^ {2} ] \leq K \sigma_ {0} ^ {2}
$$
where  $\sigma_0^2 = \max_i\sigma_i^2$  and the proof is complete.
Lemma 3. Suppose Assumptions 2-3 are satisfied and recall that  $F(w) = \frac{1}{2} \| G(\theta_t)w + \lambda g_0(\theta_t)\|^2$  is a convex function. Let  $w_{\lambda}^{*} = \arg \min_{w\in \mathcal{W}}\frac{1}{2}\| g_w + \lambda g_0\|^2$  and set step size  $\beta_{t,s} = c / \sqrt{s}$  where  $c > 0$  is a constant. Then for any  $S > 1$ , it holds that,
$$
\mathbb {E} [ \| \nabla_ {w} \widehat {F} (w) \| ] \leq C _ {1},
$$
$$
\mathbb {E} [ \| G (\theta_ {t}) w _ {S} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} - \| G (\theta_ {t}) w _ {\lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] \leq (\frac {2}{c} + 2 c C _ {1}) \frac {2 + \log (S)}{\sqrt {S}}
$$
where  $C_1 = \sqrt{8(K\sigma_0^2 + C_g^2)^2 + 8\lambda^2(K\sigma_0^2 + C_g^2)^2}$  =  $\mathcal{O}(K + \lambda K), \nabla_w\widehat{F} (w) = G(\theta_t;\xi)^T (G(\theta_t;\xi ')w + \lambda g_0(\theta_t;\xi ')).$
Proof. This lemma mostly follows from Theorem 2 in [51]. However, we did not take that  $\mathbb{E}[\| \nabla_w\widehat{F} (w)\| ]$  is bounded by a constant as an assumption. Therefore, we first provide a bound for it in our method. Based on the definition in Equation (11),  $\nabla_w\widehat{F} (w) = G(\theta_t;\xi)^T (G(\theta_t;\xi ')w + \lambda g_0(\theta_t;\xi '))$ . According to the fact that  $\mathbb{E}[X]\leq \sqrt{\mathbb{E}[X^2]}$ , we have
$$
\begin{array}{l} \mathbb {E} [ \| \nabla_ {w} \widehat {F} (w) \| ] \leq \sqrt {\mathbb {E} [ \| \nabla_ {w} \widehat {F} (w) \| ^ {2} ]} = \sqrt {\mathbb {E} [ \| G (\theta_ {t} ; \xi) ^ {T} (G (\theta_ {t} ; \xi^ {\prime}) w + \lambda g _ {0} (\theta_ {t} ; \xi^ {\prime})) \| ^ {2} ]} \\ \stackrel {(i)} {\leq} \sqrt {2 \mathbb {E} [ \underbrace {\| G (\theta_ {t} ; \xi) ^ {T} G (\theta_ {t} ; \xi^ {\prime}) w \| ^ {2}} _ {A} + \lambda^ {2} \underbrace {\| G (\theta_ {t} ; \xi) ^ {T} g _ {0} (\theta_ {t} ; \xi^ {\prime})) \| ^ {2}} _ {B} ]}, \tag {14} \\ \end{array}
$$
where  $(i)$  follows from the Young's inequality. Next, we provide bounds for  $\mathbb{E}[A]$  and  $\mathbb{E}[B]$ , separately:
$$
\begin{array}{l} \mathbb {E} [ A ] \stackrel {(i)} {\leq} \mathbb {E} \left[ \left\| \left(G \left(\theta_ {t}; \xi\right) ^ {T} - G \left(\theta_ {t}\right) ^ {T} + G \left(\theta_ {t}\right) ^ {T}\right) \left(G \left(\theta_ {t}; \xi^ {\prime}\right) - G \left(\theta_ {t}\right) + G \left(\theta_ {t}\right)\right) \right\| ^ {2} \right] \\ = \mathbb {E} [ \| (G (\theta_ {t}; \xi) ^ {T} - G (\theta_ {t}) ^ {T}) (G (\theta_ {t}; \xi^ {\prime}) - G (\theta_ {t})) + (G (\theta_ {t}; \xi) ^ {T} - G (\theta_ {t}) ^ {T}) G (\theta_ {t}) \\ + G \left(\theta_ {t}\right) ^ {T} \left(G \left(\theta_ {t}; \xi^ {\prime}\right) - G \left(\theta_ {t}\right)\right) + G \left(\theta_ {t}\right) ^ {T} G \left(\theta_ {t}\right) \| ^ {2} ] \\ \stackrel {(i i)} {\leq} 4 \mathbb {E} \left[ \| G (\theta_ {t}; \xi) ^ {T} - G (\theta_ {t}) ^ {T} \| ^ {2} \| G (\theta_ {t}; \xi^ {\prime}) - G (\theta_ {t}) \| ^ {2} + \| G (\theta_ {t}; \xi) ^ {T} - G (\theta_ {t}) ^ {T} \| ^ {2} \| G (\theta_ {t}) \| ^ {2} \right. \\ + \| G \left(\theta_ {t}\right) ^ {T} \| ^ {2} \| \left(G \left(\theta_ {t}; \xi^ {\prime}\right) - G \left(\theta_ {t}\right) \| ^ {2} + \| G \left(\theta_ {t}\right) ^ {T} G \left(\theta_ {t}\right) \| ^ {2} \right] \\ \end{array}
$$
$$
\stackrel {(i i i)} {\leq} 4 K ^ {2} \sigma_ {0} ^ {4} + 8 K \sigma_ {0} ^ {2} C _ {g} ^ {2} + 4 C _ {g} ^ {4} = 4 (K \sigma_ {0} ^ {2} + C _ {g} ^ {2}) ^ {2}, \tag {15}
$$
where  $(i)$  follows from Cauchy-Schwarz inequality and  $w\in \mathcal{W}$  where  $\mathcal{W}$  is the simplex,  $(ii)$  follows from Young's inequality and  $(iii)$  follows from Assumption 2 and Assumption 3. Then for term B, we have,
$$
\begin{array}{l} \mathbb {E} [ B ] = \mathbb {E} [ \| (G (\theta_ {t}; \xi) ^ {T} - G (\theta_ {t}) ^ {T} + G (\theta_ {t}) ^ {T}) (g _ {0} (\theta_ {t}; \xi^ {\prime}) - g _ {0} (\theta_ {t}) + g _ {0} (\theta_ {t})) \| ^ {2} ] \\ \stackrel {(i)} {\leq} 4 \mathbb {E} [ \| (G (\theta_ {t}; \xi) ^ {T} - G (\theta_ {t}) ^ {T}) (g _ {0} (\theta_ {t}; \xi^ {\prime}) - g _ {0} (\theta_ {t})) \| ^ {2} + \| (G (\theta_ {t}; \xi) ^ {T} - G (\theta_ {t}) ^ {T}) g _ {0} (\theta_ {t}) \| ^ {2} \\ + \left\| G \left(\theta_ {t}\right) ^ {T} \left(g _ {0} \left(\theta_ {t}; \xi^ {\prime}\right) - g _ {0} \left(\theta_ {t}\right)\right) \right\| ^ {2} + \left\| G \left(\theta_ {t} ^ {T}\right) g _ {0} \left(\theta_ {t}\right) \right\| ^ {2} ] \\ \stackrel {(i i)} {\leq} 4 K ^ {2} \sigma_ {0} ^ {4} + 8 K \sigma_ {0} ^ {2} C _ {g} ^ {2} + 4 C _ {g} ^ {4} = 4 \left(K \sigma_ {0} ^ {2} + C _ {g} ^ {2}\right) ^ {2}, \tag {16} \\ \end{array}
$$
where  $(i)$  follows from Young's inequality,  $(ii)$  follows from Assumption 3 and Lemma 2. Then substituting eq. (15) and eq. (16) into eq. (14), we can obtain,
$$
\mathbb {E} [ \| \nabla_ {w} \widehat {F} (w) \| ] \leq \sqrt {8 (K \sigma_ {0} ^ {2} + C _ {g} ^ {2}) ^ {2} + 8 \lambda^ {2} (K \sigma_ {0} ^ {2} + C _ {g} ^ {2}) ^ {2}} = C _ {1}.
$$
Meanwhile, since  $\mathbb{E}[\| \nabla_w\widehat{F} (w)\| ]\leq C_1,\sup_{w,w'}\| w - w'\| \leq 1$  and by choosing step size  $\beta_{s} = c / \sqrt{s}$  where  $c > 0$  is a constant, we can obtain the following inequality from Theorem 2 in [51]:
$$
\mathbb {E} \left[ F \left(w _ {S}\right) - F \left(w _ {\lambda} ^ {*}\right) \right] \leq \left(\frac {1}{c} + c C _ {1}\right) \frac {2 + \log (S)}{\sqrt {S}} \tag {17}
$$
Then after multiplying by 2 on both sides, the proof is complete.
CA distance. Now we show the upper bound for the distance to CA direction. Recall that we define the CA distance as  $\| \mathbb{E}_{\zeta ,w_{t,S}|\theta_t}[G(\theta_t;\zeta)w_{t,S} + \lambda g_0(\theta_t;\zeta)] - G(\theta_t)w_{t,\lambda}^* -\lambda g_0(\theta_t)\|$ .
Proof. Based on the Jensen's inequality, we have
$$
\begin{array}{l} \left\| \mathbb {E} _ {\zeta , w _ {t, S} | \theta_ {t}} \left[ G (\theta_ {t}; \zeta) w _ {t, S} + \lambda g _ {0} (\theta_ {t}; \zeta) \right] - G (\theta_ {t}) w _ {t, \lambda} ^ {*} - \lambda g _ {0} (\theta_ {t}) \right\| ^ {2} \\ \leq \mathbb {E} _ {w _ {t, S} | \theta_ {t}} \left[ \left\| \mathbb {E} _ {\zeta} \left[ G (\theta_ {t}; \zeta) w _ {t, S} + \lambda g _ {0} (\theta_ {t}; \zeta) \right] - G (\theta_ {t}) w _ {t, \lambda} ^ {*} - \lambda g _ {0} (\theta_ {t}) \right\| ^ {2} \right] \\ \stackrel {(i)} {=} \mathbb {E} [ \| G (\theta_ {t}) w _ {t, S} - G (\theta_ {t}) w _ {t, \lambda} ^ {*} \| ^ {2} ] \\ = \mathbb {E} [ \| G (\theta_ {t}) w _ {t, S} + \lambda g _ {0} (\theta_ {t}) - G (\theta_ {t}) w _ {t, \lambda} ^ {*} - \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] \\ = \mathbb {E} [ \| G (\theta_ {t}) w _ {t, S} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} + \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} \\ \left. - 2 \mathbb {E} \langle G (\theta_ {t}) w _ {t, S} + \lambda g _ {0} (\theta_ {t}), G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \rangle \right] \\ = \mathbb {E} [ \| G (\theta_ {t}) w _ {t, S} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} + \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] \\ - 2 \mathbb {E} [ \langle G (\theta) w _ {t, S}, G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \rangle ] - 2 \mathbb {E} [ \langle \lambda g _ {0} (\theta_ {t}), G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \rangle ] \\ \stackrel {(i i)} {\leq} \mathbb {E} \left[ \| G (\theta_ {t}) w _ {t, S} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} + \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} \right] \\ - 2 \mathbb {E} [ \langle G (\theta) w _ {t, \lambda} ^ {*}, G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \rangle ] - 2 \mathbb {E} [ \langle \lambda g _ {0} (\theta_ {t}), G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \rangle ] \\ = \mathbb {E} [ \| G (\theta_ {t}) w _ {t, S} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} + \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] \\ - 2 \mathbb {E} [ \langle G (\theta) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}), G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \rangle ] \\ = \mathbb {E} [ \| G (\theta_ {t}) w _ {t, S} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} - \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] \\ \stackrel {(i i i)} {\leq} \left(\frac {2}{c} + 2 c C _ {1}\right) \frac {2 + \log (S)}{\sqrt {S}} \tag {18} \\ \end{array}
$$
where  $(i)$  omits the subscript of taking expectation over  $w_{t,S}$  conditioning on  $\theta_t$ ,  $(ii)$  follows from optimality condition that
$$
\langle w, G \left(\theta_ {t}\right) ^ {T} \left(G \left(\theta_ {t}\right) w _ {t, \lambda} ^ {*} + \lambda g _ {0} \left(\theta_ {t}\right)\right) \rangle \geq \langle w _ {t, \lambda} ^ {*}, G \left(\theta_ {t}\right) ^ {T} \left(G \left(\theta_ {t}\right) w _ {t, \lambda} ^ {*} + \lambda g _ {0} \left(\theta_ {t}\right)\right) \rangle . \tag {19}
$$
(iii) follows from Lemma 3 when we choose  $\beta_{t,s} = c / \sqrt{s}$  where  $c$  is a constant. Then take the square root on both sides, the proof is complete.
Theorem 5 (Restatement of Theorem 1). Suppose Assumptions 1-3 are satisfied. Set  $\alpha_{t} = \alpha = \Theta((1 + \lambda)^{-1}K^{-\frac{1}{2}}T^{-\frac{1}{2}})$ ,  $\beta_{t,s} = c / \sqrt{s}$  where  $c$  is a constant, and  $S = \Theta((1 + \lambda)^{-2}T^{2})$ . The outputs of the proposed SDMGrad algorithm satisfy
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] = \widetilde {\mathcal {O}} ((1 + \lambda^ {2}) K ^ {\frac {1}{2}} T ^ {- \frac {1}{2}}).
$$
Proof. Recall that  $d = G(\theta_t; \zeta) w_{t,S} + \lambda g_0(\theta_t; \zeta)$ . According to Assumption 1, we have for any  $i$ ,
$$
L _ {i} \left(\theta_ {t + 1}\right) + \lambda L _ {0} \left(\theta_ {t + 1}\right) \leq L _ {i} \left(\theta_ {t}\right) + \lambda L _ {0} \left(\theta_ {t}\right) + \alpha_ {t} \left\langle g _ {i} \left(\theta_ {t}\right) + \lambda g _ {0} \left(\theta_ {t}\right), - d \right\rangle + \frac {l _ {i , 1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \| d \| ^ {2}. \tag {20}
$$
where  $l_{i,1}^{\prime} = l_{i,1} + \lambda \max_{i} l_{i,1} = \Theta(1 + \lambda)$ . Then we bound the second and third terms separately on the right-hand side (RHS). First, for the second term, conditioning on  $\theta_t$  and taking expectation, we have
$$
\begin{array}{l} \mathbb {E} [ \langle g _ {i} (\theta_ {t}) + \lambda g _ {0} (\theta_ {t}), - G (\theta_ {t}; \zeta) w _ {t, S} - \lambda g _ {0} (\theta_ {t}; \zeta) \rangle | \theta_ {t} ] \\ = \mathbb {E} \left[ \left\langle g _ {i} \left(\theta_ {t}\right) + \lambda g _ {0} \left(\theta_ {t}\right), - G \left(\theta_ {t}\right) w _ {t, S} - \lambda g _ {0} \left(\theta_ {t}\right) \right\rangle \mid \theta_ {t} \right] \\ = \mathbb {E} [ \langle g _ {i} (\theta_ {t}) + \lambda g _ {0} (\theta_ {t}), G (\theta_ {t}) w _ {t, \lambda} ^ {*} - G (\theta_ {t}) w _ {t, S} \rangle - \langle g _ {i} (\theta_ {t}) + \lambda g _ {0} (\theta_ {t}), G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \rangle | \theta_ {t} ] \\ \end{array}
$$
$$
\stackrel {(i)} {\leq} \mathbb {E} [ (l _ {i} + \lambda C _ {g}) \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} - G (\theta_ {t}) w _ {t, S} \| | \theta_ {t} ] - \mathbb {E} [ \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} | \theta_ {t} ]
$$
$$
\stackrel {(i i)} {\leq} \left(l _ {i} + \lambda C _ {g}\right) \sqrt {\mathbb {E} [ \| G (\theta_ {t}) w _ {t , \lambda} ^ {*} - G (\theta_ {t}) w _ {t , S} \| ^ {2} | \theta_ {t} ]} - \mathbb {E} [ \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} | \theta_ {t} ]
$$
$$
\stackrel {(i i i)} {\leq} \left(l _ {i} + \lambda C _ {g}\right) \sqrt {\left(\frac {2}{c} + 2 c C _ {1}\right) \frac {2 + \log (S)}{\sqrt {S}}} - \mathbb {E} [ \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} | \theta_ {t} ] \tag {21}
$$
where  $(i)$  follows from Cauchy-Schwarz inequality and optimality condition in eq. (19),  $(ii)$  follows from the fact that  $\mathbb{E}[X] \leq \sqrt{\mathbb{E}[X^2]}$  and  $(iii)$  follows from eq. (18).
Then for the third term,
$$
\begin{array}{l} \mathbb {E} [ \| d \| ^ {2} ] = \mathbb {E} [ \| G (\theta_ {t}; \zeta) w _ {t, S} + \lambda g _ {0} (\theta_ {t}; \zeta) \| ^ {2} ] \\ = \mathbb {E} [ \| G (\theta_ {t}; \zeta) w _ {t, S} - G (\theta_ {t}) w _ {t, S} + G (\theta_ {t}) w _ {t, S} + \lambda g _ {0} (\theta_ {t}; \zeta) - \lambda g _ {0} (\theta_ {t}) + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] \\ \stackrel {(i)} {\leq} 4 \mathbb {E} [ \| G (\theta_ {t}; \zeta) - G (\theta_ {t}) \| ^ {2} ] + 4 \mathbb {E} [ \| G (\theta_ {t}) \| ^ {2} ] + 4 \lambda^ {2} \mathbb {E} [ \| g _ {0} (\theta_ {t}; \zeta) - g _ {0} (\theta_ {t}) \| ^ {2} ] \\ + 4 \lambda^ {2} \mathbb {E} [ \| g _ {0} (\theta_ {t}) \| ^ {2} ] \\ \end{array}
$$
$$
\leq \underbrace {4 K \sigma_ {0} ^ {2} + 4 C _ {g} ^ {2} + 4 \lambda^ {2} K \sigma_ {0} ^ {2} + 4 \lambda^ {2} C _ {g} ^ {2}} _ {C _ {2}} \tag {22}
$$
where  $(i)$  follows from Young's inequality, and  $(ii)$  follows from Assumption 3 and Lemma 2. Note that  $C_2 = \mathcal{O}(K + K\lambda^2)$ . Then taking expectation on eq. (20), substituting eq. (21) and eq. (22) into it, and unconditioning on  $\theta_t$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ L _ {i} \left(\theta_ {t + 1}\right) + \lambda L _ {0} \left(\theta_ {t + 1}\right) \right] \\ \leq \mathbb {E} \left[ L _ {i} \left(\theta_ {t}\right) + \lambda L _ {0} \left(\theta_ {t}\right) \right] + \alpha_ {t} \mathbb {E} \left[ \langle g _ {i} \left(\theta_ {t}\right) + \lambda g _ {0} \left(\theta_ {t}\right), - d \rangle \right] + \frac {l _ {i , 1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \mathbb {E} [ \| d \| ^ {2} ] \\ \leq \mathbb {E} \left[ L _ {i} \left(\theta_ {t}\right) + \lambda L _ {0} \left(\theta_ {t}\right) \right] + \alpha_ {t} \left(l _ {i} + \lambda C _ {g}\right) \sqrt {\left(\frac {2}{c} + 2 c C _ {1}\right) \frac {2 + \log (S)}{\sqrt {S}}} \\ - \alpha_ {t} \mathbb {E} [ \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] + \frac {l _ {i , 1} ^ {\prime} \alpha_ {t} ^ {2}}{2} C _ {2} \tag {23} \\ \end{array}
$$
Then, choosing  $\alpha_{t} = \alpha$ , and rearranging the above inequality, we have
$$
\begin{array}{l} \alpha \mathbb {E} \left[ \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} \right] \leq \mathbb {E} \left[ L _ {i} (\theta_ {t}) + \lambda L _ {0} (\theta_ {t}) - L _ {i} (\theta_ {t + 1}) - \lambda L _ {0} (\theta_ {t + 1}) \right] \\ + \alpha \left(l _ {i} + \lambda C _ {g}\right) \sqrt {\left(\frac {2}{c} + 2 c C _ {1}\right) \frac {2 + \log (S)}{\sqrt {S}}} + \frac {l _ {i , 1} ^ {\prime} \alpha^ {2}}{2} C _ {2}. \\ \end{array}
$$
Telescoping over  $t \in [T]$  in the above inequality yields
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \left\| G \left(\theta_ {t}\right) w _ {t, \lambda} ^ {*} + \lambda g _ {0} \left(\theta_ {t}\right) \right\| ^ {2} \right] \\ \leq \frac {1}{\alpha T} \mathbb {E} \left[ L _ {i} \left(\theta_ {0}\right) - \inf  L _ {i} (\theta) + \lambda \left(L _ {0} \left(\theta_ {0}\right) - \inf  L _ {0} (\theta)\right) \right] + \frac {l _ {i , 1} ^ {\prime} \alpha}{2} C _ {2} \\ + \left(l _ {i} + \lambda C _ {g}\right) \sqrt {\left(\frac {2}{c} + 2 c C _ {1}\right) \frac {2 + \log (S)}{\sqrt {S}}}, \\ \end{array}
$$
If we choose  $\alpha = \Theta ((1 + \lambda)^{-1}K^{-\frac{1}{2}}T^{-\frac{1}{2}})$  and  $S = \Theta ((1 + \lambda)^{-2}T^{2})$  , we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] = \widetilde {\mathcal {O}} ((1 + \lambda^ {2}) K ^ {\frac {1}{2}} T ^ {- \frac {1}{2}}),
$$
where  $\tilde{\mathcal{O}}$  means the order of  $logT$  is omitted. The proof is complete.
C.3 Proof of Corollary 1
Proof. Since  $\lambda > 0$  and  $g_0(\theta_t) = G(\theta_t)\widetilde{w}$ , we have
$$
\mathbb {E} [ \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] = (1 + \lambda) ^ {2} \mathbb {E} [ \| G (\theta_ {t}) w ^ {\prime} \| ^ {2} ] \geq (1 + \lambda) ^ {2} \mathbb {E} [ \| G (\theta_ {t}) w _ {t} ^ {*} \| ^ {2} ]
$$
where  $w' = \frac{1}{1 + \lambda} (w_{1,t,\lambda}^* + \lambda \widetilde{w}_1, w_{2,t,\lambda}^* + \lambda \widetilde{w}_2, \dots, w_{K,t,\lambda}^* + \lambda \widetilde{w}_K)^T$  such that  $w' \in \mathcal{W}$ . According to parameter selection in Theorem 1 and by choosing a constant  $\lambda$ , we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G (\theta_ {t}) w _ {t} ^ {*} \| ^ {2} ] = \widetilde {\mathcal {O}} \left(K ^ {\frac {1}{2}} T ^ {- \frac {1}{2}}\right). \tag {24}
$$
To achieve an  $\epsilon$ -accurate Pareto stationary point, it requires  $T = \widetilde{\mathcal{O}}(K\epsilon^{-2})$  and each objective requires  $\widetilde{\mathcal{O}}(K^3\epsilon^{-6})$  samples in  $\xi(\xi')$  and  $\widetilde{\mathcal{O}}(K\epsilon^{-2})$  samples in  $\zeta$ , respectively. Meanwhile, according to the choice of  $S$  and  $T$ , we have the following result for CA distance,
$$
\left\| \mathbb {E} _ {\zeta , w _ {t, S} \mid \theta_ {t}} \left[ G \left(\theta_ {t}, \zeta\right) w _ {t, S} + \lambda g _ {0} \left(\theta_ {t}; \zeta\right) \right] - G \left(\theta_ {t}\right) w _ {t, \lambda} ^ {*} - \lambda g _ {0} \left(\theta_ {t}\right) \right\| = \tilde {\mathcal {O}} (\sqrt {\frac {K}{T}}) = \tilde {\mathcal {O}} (\epsilon) \tag {25}
$$
Remark. Our algorithm with a constant  $\lambda$  helps mitigate gradient conflict and it guarantees an  $\epsilon$ -accurate Pareto stationary point and the CA distance takes the order of  $\widetilde{O}(\epsilon)$  simultaneously.
C.4 Proof of Corollary 2
Proof. According to the inequality  $\| a + b - b\| ^2\leq 2\| a + b\| ^2 +2\| b\| ^2$  , we have
$$
\begin{array}{l} \lambda^ {2} \| g _ {0} (\theta_ {t}) \| ^ {2} \leq 2 \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} + 2 \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} \| ^ {2} \\ \leq 2 \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} + 2 C _ {g} ^ {2} \\ \end{array}
$$
where the last inequality follows from Assumption 3. Then we take the expectation on the above inequality and sum up it over  $t \in [T]$  such that
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| g _ {0} (\theta_ {t}) \| ^ {2} ] \leq \frac {2}{\lambda^ {2} T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] + \frac {C _ {g} ^ {2}}{\lambda^ {2}} \\ = \tilde {\mathcal {O}} ((\lambda^ {- 2} + 1) K ^ {\frac {1}{2}} T ^ {- \frac {1}{2}} + \lambda^ {- 2}), \\ \end{array}
$$
where the last inequality follows from Theorem 1. If we choose  $\lambda = \Theta (T^{\frac{1}{2}})$ , then we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| g _ {0} (\theta_ {t}) \| ^ {2} ] = \widetilde {\mathcal {O}} (K ^ {\frac {1}{2}} T ^ {- \frac {1}{2}}).
$$
To achieve an  $\epsilon$ -accurate stationary point, it requires  $T = \tilde{\mathcal{O}}(K\epsilon^{-2})$  and each objective requires  $\tilde{\mathcal{O}}(K^2\epsilon^{-4})$  samples in  $\xi(\xi')$  and  $\tilde{\mathcal{O}}(K\epsilon^{-2})$  samples in  $\zeta$ , respectively. Meanwhile, according to the choice of  $\lambda$ ,  $S$  and  $T$ , we have the following result for CA distance,