SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\| \mathbb {E} _ {\zeta , w _ {t, S} | \theta_ {t}} [ G (\theta_ {t}, \zeta) w _ {t, S} + \lambda g _ {0} (\theta_ {t}; \zeta) ] - G (\theta_ {t}) w _ {t, \lambda} ^ {*} - \lambda g _ {0} (\theta_ {t}) \| = \widetilde {\mathcal {O}} (\sqrt {\frac {K (1 + \lambda) ^ {2}}{T}}) = \widetilde {\mathcal {O}} (\sqrt {K})
$$
Remark. With an increasing  $\lambda$ , our algorithm approaches GD and it has a faster convergence rate to the stationary point. However, the CA distance takes the order of  $\tilde{O}(\sqrt{K})$ .
C.5 Proof of Theorem 2
Now we provide the convergence analysis with nonconvex objectives with objective sampling.
Theorem 6 (Restatement of Theorem 2). Suppose Assumptions 1-3 are satisfied. Set  $\gamma = \frac{K}{n}$ ,  $\alpha_{t} = \alpha = \Theta((1 + \lambda^{2})^{-\frac{1}{2}}\gamma^{-\frac{1}{2}}K^{-\frac{1}{2}}T^{-\frac{1}{2}})$ ,  $\beta_{t,s} = c / \sqrt{s}$  and  $S = \Theta((1 + \lambda)^{-2}\gamma^{-2}T^{2})$ . Then by choosing a constant  $\lambda$ , the iterates of the proposed SDMGrad-OS algorithm satisfy
Proof. Recall that updating direction for  $\theta_t$  is  $d' = \frac{K}{n} H(\theta_t; \zeta, \widetilde{S}) w_{t,S} + \frac{K}{n} \lambda h_0(\theta_t; \zeta, \widetilde{S})$ . Similarly, we have
$$
L _ {i} \left(\theta_ {t + 1}\right) + \lambda L _ {0} \left(\theta_ {t + 1}\right) \leq L _ {i} \left(\theta_ {t}\right) + \lambda L _ {0} \left(\theta_ {t}\right) + \alpha_ {t} \left\langle g _ {i} \left(\theta_ {t}\right) + \lambda g _ {0} \left(\theta_ {t}\right), - d ^ {\prime} \right\rangle + \frac {l _ {i , 1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \| d ^ {\prime} \| ^ {2}. \tag {26}
$$
Then for the inner product term on the RHS of eq. (26), conditioning on  $\theta_t$  and taking expectation, we have
$$
\begin{array}{l} \mathbb {E} [ \langle g _ {i} (\theta_ {t}) + \lambda g _ {0} (\theta_ {t}), - d ^ {\prime} \rangle | \theta_ {t} ] = \mathbb {E} [ \langle g _ {i} (\theta_ {t}) + \lambda g _ {0} (\theta_ {t}), - G (\theta_ {t}) w _ {t, S} - \lambda g _ {0} (\theta_ {t}) \rangle | \theta_ {t} ] \\ \leq \left(l _ {i} + \lambda C _ {g}\right) \left(\sqrt {\left(\frac {2}{c} + 2 c C _ {1}\right) \frac {2 + \log (S)}{\sqrt {S}}}\right) - \mathbb {E} \left[ \| G \left(\theta_ {t}\right) w _ {t, \lambda} ^ {*} + \lambda g _ {0} \left(\theta_ {t}\right) \| ^ {2} \mid \theta_ {t} \right], \tag {27} \\ \end{array}
$$
where the last inequality follows from eq. (21). Then following the same step as in eq. (22), we can bound the last term on the RHS of eq. (26) as
$$
\mathbb {E} \left[ \| d ^ {\prime} \| ^ {2} \right] \leq \underbrace {4 \gamma^ {2} \left(1 + \lambda^ {2}\right) \left(n \sigma_ {0} ^ {2} + C _ {g} ^ {2}\right)} _ {C _ {2} ^ {\prime}}. \tag {28}
$$
Then taking expectation on eq. (26), substituting eq. (27) and eq. (28) into it and unconditioning on  $\theta_{t}$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ L _ {i} \left(\theta_ {t + 1}\right) + \lambda L _ {0} \left(\theta_ {t + 1}\right) \right] \leq \mathbb {E} \left[ L _ {i} \left(\theta_ {t}\right) + \lambda L _ {0} \left(\theta_ {t}\right) \right] + \alpha_ {t} \left(l _ {i} + \lambda C _ {g}\right) \left(\sqrt {\left(\frac {2}{c} + 2 c C _ {1}\right) \frac {2 + \log (S)}{\sqrt {S}}}\right) \\ - \alpha_ {t} \mathbb {E} \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} + \frac {l _ {i , 1} ^ {\prime} \alpha_ {t} ^ {2}}{2} C _ {2} ^ {\prime} \\ \end{array}
$$
Then choosing  $\alpha_{t} = \alpha$ , telescoping the above inequality over  $t \in [T]$ , and rearranging the terms, we have
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \| G (\theta_ {t}) w _ {t, \lambda} ^ {*} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} \right] \leq \frac {1}{\alpha T} \mathbb {E} \left[ L _ {i} \left(\theta_ {0}\right) - \inf  L _ {i} (\theta) + \lambda \left(L _ {0} \left(\theta_ {0}\right) - \inf  L _ {0} (\theta)\right) \right] + \frac {l _ {i , 1} ^ {\prime} \alpha}{2} C _ {2} ^ {\prime} \\ + \left(l _ {i} + \lambda C _ {g}\right) \left(\sqrt {\left(\frac {2}{c} + 2 c C _ {1}\right) \frac {2 + \log (S)}{\sqrt {S}}}\right) \\ \end{array}
$$
If we choose  $\alpha = \Theta((1 + \lambda^2)^{-\frac{1}{2}}\gamma^{-\frac{1}{2}}K^{-\frac{1}{2}}T^{-\frac{1}{2}})$ , and  $S = \Theta((1 + \lambda)^{-2}\gamma^{-2}T^2)$ , we can get  $\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}[||G(\theta_t)w_{t,\lambda}^* + \lambda g_0(\theta_t)\|^2] = \widetilde{\mathcal{O}}((1 + \lambda^2)K^{\frac{1}{2}}\gamma^{\frac{1}{2}}T^{-\frac{1}{2}})$ . Furthermore, by choosing  $\lambda$  as constant and following the same step as in Appendix C.3, we have
To achieve an  $\epsilon$ -accurate Pareto stationary point, it requires  $T = \widetilde{\mathcal{O}}(K\gamma\epsilon^{-2})$ . In this case, each objective requires a similar number of samples  $\widetilde{\mathcal{O}}(K^3\gamma\epsilon^{-6})$  in  $\xi(\xi')$  and  $\widetilde{\mathcal{O}}(K\gamma\epsilon^{-2})$  samples in  $\zeta$ , respectively. As far as we know, this is the first provable objective sampling strategy for stochastic multi-objective optimization.
D Lower sample complexity but higher CA distance
When we do not have requirements on CA distance, we can have a much lower sample complexity. In Algorithm 1, the update process for  $w$  is to reduce the CA distance, which increases the sample complexity. Thus, we will set  $S = 1$  to make Algorithm 1 more sample-efficient. In addition, we will use  $w_{t+1} = w_{t,1}$  and  $\beta_t$  instead of  $\beta_{t,s}$  in Algorithm 1 for simplicity. The following proof is mostly motivated by Theorem 3 in [14].
Theorem 7 (Restatement of Theorem 3). Suppose Assumptions 1-3 are satisfied and  $S = 1$ . Set  $\alpha_{t} = \alpha = \Theta(K^{-\frac{1}{2}}T^{-\frac{1}{2}})$ ,  $\beta_{t} = \beta = \Theta(K^{-1}T^{-\frac{1}{2}})$  and  $\lambda$  as constant. The iterates of the proposed SDMGrad algorithm satisfy,
Proof. Now we define a new function, with a fixed weight  $w \in \mathcal{W}$ ,
$$
l ^ {\prime} \left(\theta_ {t}\right) = L \left(\theta_ {t}\right) w + \lambda L _ {0} \left(\theta_ {t}\right). \tag {29}
$$
For this new function, we have
$$
\begin{array}{l} l ^ {\prime} \left(\theta_ {t + 1}\right) \leq l ^ {\prime} \left(\theta_ {t}\right) + \alpha_ {t} \langle G \left(\theta_ {t}\right) w + \lambda g _ {0} \left(\theta_ {t}\right), - d \rangle + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \| d \| ^ {2} \\ = l ^ {\prime} (\theta_ {t}) + \alpha_ {t} \langle G (\theta_ {t}) w + \lambda g _ {0} (\theta_ {t}), - G (\theta_ {t}; \zeta) w _ {t + 1} - \lambda g _ {0} (\theta_ {t}; \zeta) \rangle + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \| d \| ^ {2} \\ \end{array}
$$
where  $l_1' = \max_i l_{i,1} + \lambda l_{i,1}$ . Then taking expectations over  $\zeta$  on both sides and rearranging the inequality, we have
$$
\begin{array}{l} \mathbb {E} [ l ^ {\prime} (\theta_ {t + 1}) ] - \mathbb {E} [ l ^ {\prime} (\theta_ {t}) ] \leq \alpha_ {t} \mathbb {E} [ \langle G (\theta_ {t}) w + \lambda g _ {0} (\theta_ {t}), - G (\theta_ {t}) w _ {t + 1} - \lambda g _ {0} (\theta_ {t}) \rangle ] + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \mathbb {E} [ \| d \| ^ {2} ] \\ = - \alpha_ {t} \mathbb {E} [ \langle G (\theta_ {t}) w + \lambda g _ {0} (\theta_ {t}), G (\theta_ {t}) w _ {t + 1} - G (\theta_ {t}) w _ {t} \rangle ] \\ - \alpha_ {t} \mathbb {E} [ \langle G (\theta_ {t}) w + \lambda g _ {0} (\theta_ {t}), G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \rangle ] + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \mathbb {E} [ \| d \| ^ {2} ] \\ = - \alpha_ {t} \mathbb {E} [ \langle G (\theta_ {t}) w + \lambda g _ {0} (\theta_ {t}), G (\theta_ {t}) w _ {t + 1} - G (\theta_ {t}) w _ {t} \rangle ] \\ - \alpha_ {t} \mathbb {E} [ \langle G (\theta_ {t}) w - G (\theta_ {t}) w _ {t}, G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \rangle ] \\ - \alpha_ {t} \mathbb {E} [ \| G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \mathbb {E} [ \| d \| ^ {2} ] \\ \stackrel {(i)} {\leq} \alpha_ {t} \underbrace {\mathbb {E} \left[ \left\| \left(G \left(\theta_ {t}\right) w + \lambda g _ {0} \left(\theta_ {t}\right)\right) ^ {T} G \left(\theta_ {t}\right) \right\| \| w _ {t} - w _ {t + 1} \| \right]} _ {C} \\ + \alpha_ {t} \underbrace {\mathbb {E} [ \langle G (\theta_ {t}) w _ {t} - G (\theta_ {t}) w , G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \rangle ]} _ {D} \\ - \alpha_ {t} \mathbb {E} [ \| G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \mathbb {E} [ \| d \| ^ {2} ], \tag {30} \\ \end{array}
$$
where  $(i)$  follows from Cauchy-Schwarz inequality. Then we provide bound for term C and term D, respectively. For term C,
$$
\begin{array}{l} \mathbb {E} \left[ \| \left(G \left(\theta_ {t}\right) w + \lambda g _ {0} \left(\theta_ {t}\right)\right) ^ {T} G \left(\theta_ {t}\right) \| \| w _ {t} - w _ {t + 1} \| \right] \\ = \beta_ {t} \mathbb {E} [ \| (G (\theta_ {t}) w + \lambda g _ {0} (\theta_ {t})) ^ {T} G (\theta_ {t}) \| \| G (\theta_ {t}; \xi) ^ {T} (G (\theta_ {t}; \xi^ {\prime}) w _ {t} + \lambda g _ {0} (\theta ; \xi^ {\prime})) \| ] \\ \leq \beta_ {t} \mathbb {E} [ \| (G (\theta_ {t}) w + \lambda g _ {0} (\theta_ {t})) ^ {T} G (\theta_ {t}) \| (\| G (\theta_ {t}; \xi) ^ {T} (G (\theta_ {t}; \xi^ {\prime}) w _ {t} \| + \lambda \| G (\theta_ {t}; \xi) g _ {0} (\theta ; \xi^ {\prime}) \|) ] \\ \leq \beta_ {t} (1 + \lambda) ^ {2} C _ {g} ^ {2} \left(K \sigma_ {0} + C _ {g}\right) ^ {2} = \beta_ {t} C _ {3}, \tag {31} \\ \end{array}
$$
where  $C_3 = \mathcal{O}((1 + \lambda)^2 K^2)$ . Then for term D, we first follow the non-expansive property of projection onto the convex set,
$$
\begin{array}{l} \left\| w _ {t + 1} - w \right\| ^ {2} \leq \left\| w _ {t} - \beta_ {t} G \left(\theta_ {t}; \xi\right) ^ {T} \left(G \left(\theta_ {t}; \xi^ {\prime}\right) w _ {t} + \lambda g _ {0} \left(\theta_ {t}; \xi^ {\prime}\right)\right) - w \right\| ^ {2} \\ = \| w _ {t} - w \| ^ {2} - 2 \beta_ {t} \langle w _ {t} - w, G (\theta_ {t}; \xi) ^ {T} (G (\theta_ {t}; \xi^ {\prime}) w _ {t} + \lambda g _ {0} (\theta_ {t}; \xi^ {\prime})) \rangle \\ + \beta_ {t} ^ {2} \left\| G \left(\theta_ {t}; \xi\right) ^ {T} \left(G \left(\theta_ {t}; \xi^ {\prime}\right) w _ {t} + \lambda g _ {0} \left(\theta_ {t}; \xi^ {\prime}\right)\right) \right\| ^ {2} \\ \end{array}
$$
Then taking expectation on the above inequality, we can obtain,
$$
\mathbb {E} [ \| w _ {t + 1} - w \| ^ {2} ] \leq \mathbb {E} [ \| w _ {t} - w \| ^ {2} ] - 2 \beta_ {t} \mathbb {E} [ \langle w _ {t} - w, G (\theta_ {t}; \xi) ^ {T} (G (\theta_ {t}; \xi^ {\prime}) w _ {t} + \lambda g _ {0} (\theta_ {t}; \xi^ {\prime})) \rangle ]
$$
$$
\begin{array}{l} + \beta_ {t} ^ {2} \mathbb {E} [ \| G (\theta_ {t}; \xi) ^ {T} (G (\theta_ {t}; \xi^ {\prime}) w _ {t} + \lambda g _ {0} (\theta_ {t}; \xi^ {\prime})) \| ^ {2} ] \\ \leq \mathbb {E} [ \| w _ {t} - w \| ^ {2} ] - 2 \beta_ {t} \mathbb {E} [ \langle w _ {t} - w, G (\theta_ {t}) ^ {T} (G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t})) \rangle ] + \beta_ {t} ^ {2} C _ {1} ^ {2}, \\ \end{array}
$$
where the last inequality follows from Lemma 3. Then by rearranging the above inequality, we can obtain,
$$
\mathbb {E} \left[ \langle w _ {t} - w, G \left(\theta_ {t}\right) ^ {T} \left(G \left(\theta_ {t}\right) w _ {t} + \lambda g _ {0} \left(\theta_ {t}\right)\right) \rangle \right] \leq \frac {1}{2 \beta_ {t}} \mathbb {E} \left[ \| w _ {t} - w \| ^ {2} - \| w _ {t + 1} - w \| ^ {2} \right] + \frac {\beta_ {t}}{2} C _ {1} ^ {2} \tag {32}
$$
Then substituting eq. (31) and eq. (32) into eq. (30), we can obtain,
$$
\begin{array}{l} \mathbb {E} [ l ^ {\prime} (\theta_ {t + 1}) - l ^ {\prime} (\theta_ {t}) ] \leq \alpha_ {t} \beta_ {t} C _ {3} + \frac {\alpha_ {t}}{2 \beta_ {t}} \mathbb {E} [ \| w _ {t} - w \| ^ {2} - \| w _ {t + 1} - w \| ^ {2} ] + \frac {\alpha_ {t} \beta_ {t}}{2} C _ {1} ^ {2} \\ - \alpha_ {t} \mathbb {E} [ \| G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \mathbb {E} [ \| d \| ^ {2} ] \\ \stackrel {(i)} {\leq} \alpha_ {t} \beta_ {t} C _ {3} + \frac {\alpha_ {t}}{2 \beta_ {t}} \mathbb {E} [ \| w _ {t} - w \| ^ {2} - \| w _ {t + 1} - w \| ^ {2} ] + \frac {\alpha_ {t} \beta_ {t}}{2} C _ {1} ^ {2} \\ - \alpha_ {t} \mathbb {E} \left[ \| G \left(\theta_ {t}\right) w _ {t} + \lambda g _ {0} \left(\theta_ {t}\right) \| ^ {2} \right] + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} C _ {2} \tag {33} \\ \end{array}
$$
Then we take  $\alpha_{t} = \alpha$  and  $\beta_{t} = \beta$  as constants, telescope and rearrange the above inequality,
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] \leq \frac {1}{\alpha T} \mathbb {E} [ l ^ {\prime} (\theta_ {0}) - l ^ {\prime} (\theta_ {T}) ] + \frac {1}{2 \beta T} \mathbb {E} [ \| w _ {0} - w \| ^ {2} - \| w _ {T} - w \| ^ {2} ] \\ + \beta \left(C _ {3} + \frac {C _ {1} ^ {2}}{2}\right) + \frac {l _ {1} ^ {\prime} \alpha}{2} C _ {2} \\ \stackrel {(i)} {\leq} \mathcal {O} \left(\frac {1}{\alpha T} + \alpha K + \frac {1}{\beta T} + \beta K ^ {2}\right), \tag {34} \\ \end{array}
$$
where  $(i)$  follows from that we choose  $\lambda$  as a constant. If we choose  $\alpha = \Theta (K^{-\frac{1}{2}}T^{-\frac{1}{2}})$  and  $\beta = \Theta (K^{-1}T^{-\frac{1}{2}})$ , we can get  $\frac{1}{T}\sum_{t = 0}^{T - 1}\mathbb{E}[\| G(\theta_t)w_t + \lambda g_0(\theta_t)\| ^2 ] = \mathcal{O}(KT^{-\frac{1}{2}})$ . Furthermore, following the same steps as in Appendix C.3, we have
To achieve an  $\epsilon$ -accurate Pareto stationary point, it requires  $T = \mathcal{O}(K^2 \epsilon^{-2})$ . In this case, each objective requires a similar number of samples  $\mathcal{O}(K^2 \epsilon^{-2})$  in  $\xi(\xi')$  and  $\zeta$ , respectively.
Theorem 8 (Restatement of Theorem 4). Suppose Assumptions 1-3 are satisfied and  $S = 1$ . Set  $\gamma = \frac{K}{n}$ ,  $\alpha_{t} = \alpha = \Theta(K^{-\frac{1}{2}}\gamma^{-\frac{1}{2}}T^{-\frac{1}{2}})$ ,  $\beta_{t} = \beta = \Theta(K^{-1}\gamma^{-1}T^{-\frac{1}{2}})$  and  $\lambda$  as a constant. The iterates of the proposed SDMGrad-OS algorithm satisfy,
Proof. In SDMGrad-OS, the vector for updating  $\theta_t$  is  $d' = \frac{K}{n} H(\theta_t; \zeta, \widetilde{S}) w_{t+1} + \frac{\lambda K}{n} h_0(\theta_t; \zeta, \widetilde{S})$ . Using the same function defined in eq. (29), we have
$$
l ^ {\prime} (\theta_ {t + 1}) \leq l ^ {\prime} (\theta_ {t}) + \alpha_ {t} \langle G (\theta_ {t}) w + \lambda g _ {0} (\theta_ {t}), - d ^ {\prime} \rangle + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \| d ^ {\prime} \| ^ {2}.
$$
Then by taking expectation over  $\zeta$  and  $\widetilde{S}$ , we have
$$
\mathbb {E} [ l ^ {\prime} (\theta_ {t + 1}) - l ^ {\prime} (\theta_ {t}) ] \leq \alpha_ {t} \mathbb {E} [ \langle G (\theta_ {t}) w + \lambda g _ {0} (\theta_ {t}), - G (\theta_ {t}) w _ {t + 1} + \lambda g _ {0} (\theta_ {t}) \rangle ] + \frac {l ^ {\prime} \alpha_ {t} ^ {2}}{2} \mathbb {E} [ \| d ^ {\prime} \| ^ {2} ]
$$
$$
\begin{array}{l} = \alpha_ {t} E [ \langle G (\theta_ {t}) w + \lambda g _ {0} (\theta_ {t}), G (\theta_ {t}) (w _ {t} - w _ {t + 1}) \rangle ] \\ + \alpha_ {t} \mathbb {E} [ \langle G (\theta_ {t}) w _ {t} - G (\theta_ {t}) w, G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \rangle ] \\ - \mathbb {E} [ \| G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \mathbb {E} [ \| d ^ {\prime} \| ^ {2} ] \\ \leq \alpha_ {t} \mathbb {E} \left[ \left\| \left(G \left(\theta_ {t}\right) w + \lambda g _ {0} \left(\theta_ {t}\right)\right) ^ {T} G \left(\theta_ {t}\right) \right\| \right. \| w _ {t} - w _ {t + 1} \| ] \\ + \alpha_ {t} \mathbb {E} [ \langle G (\theta_ {t}) w _ {t} - G (\theta_ {t}) w, G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \rangle ] \\ - \mathbb {E} [ \| G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} \mathbb {E} [ \| d ^ {\prime} \| ^ {2} ] \tag {35} \\ \end{array}
$$
Then following the same steps in eq. (31) and eq. (32), we can obtain,
$$
\begin{array}{l} \mathbb {E} [ l ^ {\prime} (\theta_ {t + 1}) - l ^ {\prime} (\theta_ {t}) ] \leq \alpha_ {t} \beta_ {t} C _ {3} ^ {\prime} + \frac {\alpha_ {t}}{2 \beta_ {t}} \mathbb {E} [ \| w _ {t} - w \| ^ {2} - \| w _ {t + 1} - w \| ^ {2} ] + \frac {\alpha_ {t} \beta_ {t}}{2} C _ {1} ^ {\prime 2} \\ - \alpha_ {t} \mathbb {E} [ \| G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] + \frac {l _ {1} ^ {\prime} \alpha_ {t} ^ {2}}{2} C _ {2} ^ {\prime}, \tag {36} \\ \end{array}
$$
where  $C_1^{\prime 2} = 4\gamma^4 (1 + \lambda^2)(n\sigma_0^2 +C_g)^2$ ,  $C_2^\prime = 4\gamma^2 (1 + \lambda^2)(n\sigma_0^2 +C_g^2)$ , and  $C_3^\prime = \gamma^2 (1 + \lambda)^2 C_g^2 (n\sigma_0^2 + C_g)^2$ . Then we take  $\alpha_{t} = \alpha$  and  $\beta_{t} = \beta$  as constants and telescope the above inequality,
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| G (\theta_ {t}) w _ {t} + \lambda g _ {0} (\theta_ {t}) \| ^ {2} ] \leq \frac {1}{\alpha T} \mathbb {E} [ l ^ {\prime} (\theta_ {0}) - l ^ {\prime} (\theta_ {T}) ] + \frac {1}{2 \beta T} \mathbb {E} [ \| w _ {0} - w \| ^ {2} - \| w _ {T} - w \| ^ {2} ] \\ + \beta \left(C _ {3} ^ {\prime} + \frac {C _ {1} ^ {\prime 2}}{2}\right) + \frac {l _ {1} ^ {\prime} \alpha}{2} C _ {2} ^ {\prime} \\ \stackrel {(i)} {\leq} \mathcal {O} \left(\frac {1}{\alpha T} + \alpha \gamma K + \frac {1}{\beta T} + \beta \gamma^ {2} K ^ {2}\right), \tag {37} \\ \end{array}
$$
where  $(i)$  follows from that we choose  $\lambda$  as constant. Similarly, if we choose  $\alpha = \Theta (K^{-\frac{1}{2}}\gamma^{-\frac{1}{2}}T^{-\frac{1}{2}})$  and  $\beta = \Theta (K^{-1}\gamma^{-1}T^{-\frac{1}{2}})$ , we can get  $\frac{1}{T}\sum_{t = 0}^{T - 1}\mathbb{E}[\| G(\theta_t)w_t + \lambda g_0(\theta_t)\| ^2 ] = \mathcal{O}(K\gamma T^{-\frac{1}{2}})$ . Furthermore, following the same step as in Appendix C.3, we have
To achieve an  $\epsilon$ -accurate Pareto stationary point, it requires  $T = \mathcal{O}(\gamma^2 K^2 \epsilon^{-2})$ . In this case, each objective requires a similar number of samples  $\mathcal{O}(\gamma^2 K^2 \epsilon^{-2})$  in  $\xi(\xi')$  and  $\zeta$ , respectively.