SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Organization of proof. Without loss of generality, we assume  $\inf_{x\in \mathbb{R}^d}f_{m,z}(x) < \infty$  for all  $m\in [M]$  and  $z\in \mathcal{Z}$  in the strongly convex case. In Lemma 4, we show that the optimal solution of  $F_{z}(x)\lambda$  given any stochastic sample  $z\in \mathcal{Z}$ , and weighting parameter  $\lambda \in \Delta^{M}$ , is bounded. In Lemma 5, we show that if the argument parameter is bounded, then the updated parameter by MoDo at each iteration is also bounded by exploiting the co-coerciveness of strongly convex and smooth objectives. Finally, based on Lemma 4 and Lemma 5, we first prove in Corollary 4 that with a bounded initialization  $x_0$ , the model parameter  $\{x_t\}_{t = 1}^T$  generated by MoDo algorithm is bounded on the trajectory. Then based on Lemma 6, by the Lipschitz smoothness assumption of  $f_{z,m}(x)$ , we immediately have that  $\| \nabla f_{z,m}(x)\|$  is bounded for  $x\in \{x_t\}_{t = 1}^T$  generated by MoDo algorithm, which completes the proof of Lemma 1. Lemma 1 paves the way for deriving the MOL uniform stability and PS generalization error of MoDo in the strongly convex and unconstrained setting.
Lemma 4. Suppose Assumptions 1, 2 hold. WLOG, assume  $\inf_{x\in \mathbb{R}^d}f_{m,z}(x) < \infty$  for all  $m\in [M]$  and  $z\in \mathcal{Z}$ . For any given  $\lambda \in \Delta^{M}$ , and stochastic sample  $z\in \mathcal{Z}$ , define  $x_{\lambda ,z}^{*} = \arg \min_{x\in \mathbb{R}^{d}}F_{z}(x)\lambda$ , then  $\inf_{x\in \mathbb{R}^d}F_z(x)\lambda < \infty$  and  $\| x_{\lambda ,z}^{*}\| < \infty$ , i.e., there exist finite positive constants  $c_{F^{*}}$  and  $c_{x^{*}}$  such that
$$
\inf  _ {x \in \mathbb {R} ^ {d}} F _ {z} (x) \lambda \leq c _ {F ^ {*}} \quad \text {a n d} \quad \| x _ {\lambda , z} ^ {*} \| \leq c _ {x ^ {*}}. \tag {B.26}
$$
Proof. Under Assumption 2, for all  $m \in [M]$ ,  $f_{m,z}(x)$  is strongly convex w.r.t.  $x$ , thus has a unique minimizer. Define the minimizer  $x_{m,z}^{*} = \arg \min_{x \in \mathbb{R}^{d}} f_{m,z}(x)$ . Since a strongly convex function is coercive,  $\inf_{x \in \mathbb{R}^{d}} f_{m,z}(x) < \infty$ , i.e.,  $f_{m,z}(x_{m,z}^{*}) < \infty$ , implies that  $\| x_{m,z}^{*} \| < \infty$ .
By Assumption 1, the  $\ell_{f,1}$ -Lipschitz smoothness of  $f_{m,z}(x)$ , for  $x$  such that  $\| x\| < \infty$
$$
\begin{array}{l} f _ {m, z} (x) \leq f _ {m, z} \left(x _ {m, z} ^ {*}\right) + \left\langle \nabla f _ {m, z} \left(x _ {m, z} ^ {*}\right), x - x _ {m, z} ^ {*} \right\rangle + \frac {\ell_ {f , 1}}{2} \| x - x _ {m, z} ^ {*} \| ^ {2} \\ \leq f _ {m, z} \left(x _ {m, z} ^ {*}\right) + \frac {\ell_ {f , 1}}{2} \| x - x _ {m, z} ^ {*} \| ^ {2} <   \infty . \tag {B.27} \\ \end{array}
$$
Since  $F_{z}(x)\lambda$  is convex w.r.t.  $x$ , for all  $\lambda \in \Delta^{M}$ , with  $\lambda = [\lambda_1,\dots ,\lambda_M]^\top$ , we have
$$
F _ {z} \left(\frac {1}{M} \sum_ {m = 1} ^ {M} x _ {m, z} ^ {*}\right) \lambda \leq \frac {1}{M} \sum_ {m = 1} ^ {M} F _ {z} \left(x _ {m, z} ^ {*}\right) \lambda = \frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {m ^ {\prime} = 1} ^ {M} f _ {m ^ {\prime}, z} \left(x _ {m, z} ^ {*}\right) \lambda_ {m ^ {\prime}} <   \infty . \tag {B.28}
$$
Therefore, for all  $\lambda \in \Delta^M$ , we have
$$
\inf  _ {x \in \mathbb {R} ^ {d}} F _ {z} (x) \lambda \leq F _ {z} \left(\frac {1}{M} \sum_ {m = 1} ^ {M} x _ {m, z} ^ {*}\right) \lambda <   \infty . \tag {B.29}
$$
Since  $F_{z}(x)\lambda$  is strongly convex, thus is coercive, we have  $\| x_{\lambda ,z}^{*}\| <  \infty$  , which proves the result.
Lemma 5. Suppose Assumptions 1, 2 hold, and define  $\kappa = 3\ell_{f,1} / \mu \geq 3$ . For any given  $\lambda \in \Delta^M$  and a stochastic sample  $z \in \mathcal{Z}$ , define  $x_{\lambda,z}^{*} = \arg \min_{x} F_z(x)\lambda$ . Then by Lemma 4, there exists a positive finite constant  $c_{x,1} \geq c_{x^*}$  such that  $\| x_{\lambda,z}^{*} \| \leq c_{x^*} \leq c_{x,1}$ . Recall the multi-objective gradient update is
$$
G _ {\lambda , z} (x) = x - \alpha \nabla F _ {z} (x) \lambda \tag {B.30}
$$
with step size  $0 \leq \alpha \leq \ell_{f,1}^{-1}$ . Defining  $c_{x,2} = (1 + \sqrt{2\kappa})c_{x,1}$ , we have that
$$
i f \| x \| \leq c _ {x, 2}, \text {t h e n} \| G _ {\lambda , z} (x) \| \leq c _ {x, 2}. \tag {B.31}
$$
Proof. We divide the proof into two cases: 1) when  $\| x \| < c_{x,1}$ ; and, 2) when  $c_{x,1} \leq \| x \| \leq c_{x,2}$ .
1) For the first case,  $\| x \| < c_{x,1} \leq c_{x,2}$ , then we have
$$
\begin{array}{l} \left\| G _ {\lambda , z} (x) \right\| \leq \left\| G _ {\lambda , z} (x) - x ^ {*} \right\| + \left\| x ^ {*} \right\| \\ \stackrel {(a)} {=} \| G _ {\lambda , z} (x) - G _ {\lambda , z} (x ^ {*}) \| + \| x ^ {*} \| \stackrel {(b)} {\leq} \| x - x ^ {*} \| + \| x ^ {*} \| \\ \leq \| x \| + 2 \| x ^ {*} \| \leq 3 c _ {x, 1} \leq (1 + \sqrt {6}) c _ {x, 1} \leq (1 + \sqrt {2 \kappa}) c _ {x, 1} \leq c _ {x, 2} \tag {B.32} \\ \end{array}
$$
where  $(a)$  follows from  $\nabla F_z(x^*)\lambda = 0$ , and  $(b)$  follows from the non-expansiveness of the gradient update for strongly convex and smooth function.
2) For the second case,  $c_{x,1} \leq \| x \| \leq c_{x,2}$ , we first consider  $\alpha = \ell_{f,1}^{-1}$ . Let  $\mu' = \mu / 3$ . Note that since  $F_z(x)\lambda$  is  $\mu$ -strongly convex, it is also  $\mu'$ -strongly convex. By strong convexity and smoothness of  $F_z(x)\lambda$ , the gradients are co-coercive [36, Theorem 2.1.12], i.e., for any  $x$  we have
$$
\left(\nabla F _ {z} (x) \lambda\right) ^ {\top} \left(x - x ^ {*}\right) \geq \frac {\ell_ {f , 1} ^ {- 1} \| \nabla F _ {z} (x) \lambda \| ^ {2}}{1 + \kappa^ {- 1}} + \frac {\mu^ {\prime} \| x - x ^ {*} \| ^ {2}}{1 + \kappa^ {- 1}}. \tag {B.33}
$$
Rearranging and applying Cauchy-Schwartz inequality, we have
$$
\begin{array}{l} \left(\nabla F _ {z} (x) \lambda\right) ^ {\top} x \geq \left(\nabla F _ {z} (x) \lambda\right) ^ {\top} x ^ {*} + \frac {\ell_ {f , 1} ^ {- 1} \| \nabla F _ {z} (x) \lambda \| ^ {2}}{1 + \kappa^ {- 1}} + \frac {\mu^ {\prime} \| x - x ^ {*} \| ^ {2}}{1 + \kappa^ {- 1}} \\ \geq - c _ {x, 1} \| \nabla F _ {z} (x) \lambda \| + \frac {\ell_ {f , 1} ^ {- 1} \| \nabla F _ {z} (x) \lambda \| ^ {2}}{1 + \kappa^ {- 1}} + \frac {\mu^ {\prime} \| x - x ^ {*} \| ^ {2}}{1 + \kappa^ {- 1}}. \tag {B.34} \\ \end{array}
$$
By the definition of  $G_{\lambda ,z}(x)$
$$
\left\| G _ {\lambda , z} (x) \right\| ^ {2} = \left\| x - \frac {1}{\ell_ {f , 1}} \nabla F _ {z} (x) \lambda \right\| ^ {2} = \| x \| ^ {2} + \frac {1}{\ell_ {f , 1} ^ {2}} \| \nabla F _ {z} (x) \lambda \| ^ {2} - \frac {2}{\ell_ {f , 1}} \left(\nabla F _ {z} (x) \lambda\right) ^ {\top} x. \tag {B.35}
$$
Substituting (B.34) into (B.35) yields
$$
\begin{array}{l} \| G _ {\lambda , z} (x) \| ^ {2} \leq \| x \| ^ {2} + \frac {1}{\ell_ {f , 1} ^ {2}} \| \nabla F _ {z} (x) \lambda \| ^ {2} + \frac {2}{\ell_ {f , 1}} \left(c _ {x, 1} \| \nabla F _ {z} (x) \lambda \| - \frac {\ell_ {f , 1} ^ {- 1} \| \nabla F _ {z} (x) \lambda \| ^ {2}}{1 + \kappa^ {- 1}} - \frac {\mu^ {\prime} \| x - x ^ {*} \| ^ {2}}{1 + \kappa^ {- 1}}\right) \\ = \| x \| ^ {2} + \frac {2}{\ell_ {f , 1}} \left(c _ {x, 1} \| \nabla F _ {z} (x) \lambda \| - \frac {1}{2 \ell_ {f , 1}} \left(\frac {1 - \kappa^ {- 1}}{1 + \kappa^ {- 1}}\right) \| \nabla F _ {z} (x) \lambda \| ^ {2} - \frac {\mu^ {\prime}}{1 + \kappa^ {- 1}} \| x - x ^ {*} \| ^ {2}\right) \\ \leq \| x \| ^ {2} + \frac {2}{\ell_ {f , 1}} \sup  _ {\tau \in \mathbb {R}} \left(\underbrace {c _ {x , 1} \cdot \tau - \frac {1}{2 \ell_ {f , 1}} \left(\frac {1 - \kappa^ {- 1}}{1 + \kappa^ {- 1}}\right) \tau^ {2}} _ {I _ {1}} - \frac {\mu^ {\prime} \| x - x ^ {*} \| ^ {2}}{1 + \kappa^ {- 1}}\right). \tag {B.36} \\ \end{array}
$$
Since  $\kappa \geq 3$ , thus  $\frac{1 - \kappa^{-1}}{1 + \kappa^{-1}} > 0$ , then  $I_1$  is a quadratic function w.r.t.  $\tau$ , and is strictly concave, thus can be bounded above by
$$
\sup  _ {\tau \in \mathbb {R}} c _ {x, 1} \cdot \tau - \frac {1}{2 \ell_ {f , 1}} \left(\frac {1 - \kappa^ {- 1}}{1 + \kappa^ {- 1}}\right) \tau^ {2} \leq \frac {c _ {x , 1} ^ {2} \ell_ {f , 1}}{2} \frac {1 + \kappa^ {- 1}}{1 - \kappa^ {- 1}}. \tag {B.37}
$$
Substituting this back into (B.36) gives that
$$
\begin{array}{l} \left\| G _ {\lambda , z} (x) \right\| ^ {2} \leq \| x \| ^ {2} + \frac {2}{\ell_ {f , 1}} \left(\frac {c _ {x , 1} ^ {2} \ell_ {f , 1}}{2} \frac {1 + \kappa^ {- 1}}{1 - \kappa^ {- 1}} - \frac {\mu^ {\prime}}{1 + \kappa^ {- 1}} \| x - x ^ {*} \| ^ {2}\right) \\ = \| x \| ^ {2} + c _ {x, 1} ^ {2} \frac {1 + \kappa^ {- 1}}{1 - \kappa^ {- 1}} - 2 \frac {\kappa^ {- 1}}{1 + \kappa^ {- 1}} \| x - x ^ {*} \| ^ {2} \\ \leq \| x \| ^ {2} + c _ {x, 1} ^ {2} \frac {1 + \kappa^ {- 1}}{1 - \kappa^ {- 1}} - 2 \frac {\kappa^ {- 1}}{1 + \kappa^ {- 1}} (\| x \| - \| x ^ {*} \|) ^ {2} \\ \leq \underbrace {\left\| x \right\| ^ {2} + 2 c _ {x , 1} ^ {2} - \kappa^ {- 1} \left(\left\| x \right\| - c _ {x , 1}\right) ^ {2}} _ {I _ {2}} \tag {B.38} \\ \end{array}
$$
where the last inequality follows from  $\kappa \geq 3$ , thus  $\frac{1 + \kappa^{-1}}{1 - \kappa^{-1}} \leq 2, -2\frac{\kappa^{-1}}{1 + \kappa^{-1}} \leq -\kappa^{-1}$ , and  $\| x^{*} \| \leq c_{x,1} \leq \| x \|$  by assumption.
For  $c_{x,1} \leq \| x \| \leq c_{x,2}$ ,  $I_2$  is a strictly convex quadratic function of  $\| x \|$ , which achieves its maximum at  $\| x \| = c_{x,1}$  or  $\| x \| = c_{x,2}$ . Therefore,
$$
\begin{array}{l} \left\| G _ {\lambda , z} (x) \right\| ^ {2} \leq \max  \left\{3 c _ {x, 1} ^ {2}, c _ {x, 2} ^ {2} + 2 c _ {x, 1} ^ {2} - \kappa^ {- 1} \left(c _ {x, 2} - c _ {x, 1}\right) ^ {2} \right\} \\ \stackrel {(c)} {=} \max  \left\{3 c _ {x, 1} ^ {2}, c _ {x, 2} ^ {2} \right\} <   c _ {x, 2} ^ {(d)} \tag {B.39} \\ \end{array}
$$
where  $(c)$  follows from the definition that  $c_{x,2} = (1 + \sqrt{2\kappa})c_{x,1}$ ;  $(d)$  follows from  $\kappa \geq 3$ , and thus  $3c_{x,1}^2 < (1 + \sqrt{2\kappa})^2 c_{x,1}^2 = c_{x,2}^2$ .
We have proved the case for  $\alpha = \ell_{f,1}^{-1}$ . The result for  $0 \leq \alpha < \ell_{f,1}^{-1}$  follows by observing that,
$$
\begin{array}{l} \left\| G _ {\lambda , z} (x) \right\| = \left\| x - \alpha \nabla F _ {z} (x) \lambda \right\| \\ = \| \left(1 - \alpha \ell_ {f, 1}\right) x + \alpha \ell_ {f, 1} \left(x - \ell_ {f, 1} ^ {- 1} \nabla F _ {z} (x) \lambda\right) \| \\ \leq \left(1 - \alpha \ell_ {f, 1}\right) \| x \| + \alpha \ell_ {f, 1} \| x - \ell_ {f, 1} ^ {- 1} \nabla F _ {z} (x) \lambda \| \leq c _ {x, 2}. \tag {B.40} \\ \end{array}
$$
Lemma 6. Suppose Assumptions 1, 2 hold. For all  $\lambda \in \Delta^{M}$  and  $z \in S$ , define  $x_{\lambda,z}^{*} = \arg \min_{x} F_{z}(x)\lambda$ , then there exist finite positive constants  $c_{F^{*}}$  and  $c_{x^{*}}$  such that  $F_{z}(x_{\lambda,z}^{*})\lambda \leq c_{F^{*}}$  and  $\| x_{\lambda,z}^{*} \| \leq c_{x^{*}}$ . And for  $x \in \mathbb{R}^d$  such that  $\| x \|$  is bounded, i.e., there exists a finite positive constant  $c_{x}$  such that  $\| x \| \leq c_{x}$ , then
$$
\left\| \nabla F _ {z} (x) \lambda \right\| \leq \ell_ {f, 1} \left(c _ {x} + c _ {x ^ {*}}\right), \quad \text {a n d} \quad F _ {z} (x) \lambda \leq \frac {\ell_ {f , 1}}{2} \left(c _ {x} + c _ {x ^ {*}}\right) ^ {2} + c _ {F ^ {*}}. \tag {B.41}
$$
Proof. Under Assumptions 1, 2, by Lemma 4, there exist finite positive constants  $c_{F^*}$  and  $c_{x^*}$  such that  $F_z(x_{\lambda, z}^*) \lambda \leq c_{F^*}$  and  $\| x_{\lambda, z}^* \| \leq c_{x^*}$ .
By Assumption 1, the  $\ell_{f,1}$ -Lipschitz continuity of the gradient  $\nabla F_z(x)\lambda$ , we have
$$
\begin{array}{l} \| \nabla F _ {z} (x) \lambda \| = \| \nabla F _ {z} (x) \lambda - \nabla F _ {z} \left(x _ {\lambda , z} ^ {*}\right) \lambda \| \\ \leq \ell_ {f, 1} \| x - x _ {\lambda , z} ^ {*} \| \leq \ell_ {f, 1} (\| x \| + \| x _ {\lambda , z} ^ {*} \|) \leq \ell_ {f, 1} \left(c _ {x} + c _ {x ^ {*}}\right) \tag {B.42} \\ \end{array}
$$
where the first equality uses the fact that  $\nabla F_z(x_{\lambda ,z}^*)\lambda = 0$
For the function value, by Assumption 1, the  $\ell_{f,1}$ -Lipschitz smoothness of  $F_z(x)\lambda$ , we have
$$
\begin{array}{l} F _ {z} (x) \lambda \leq F _ {z} \left(x _ {\lambda , z} ^ {*}\right) \lambda + \left\langle \nabla F _ {z} \left(x _ {\lambda , z} ^ {*}\right) \lambda , x - x _ {\lambda , z} ^ {*} \right\rangle + \frac {\ell_ {f , 1}}{2} \| x - x _ {\lambda , z} ^ {*} \| ^ {2} \\ \leq F _ {z} \left(x _ {\lambda , z} ^ {*}\right) \lambda + \frac {\ell_ {f , 1}}{2} \| x - x _ {\lambda , z} ^ {*} \| ^ {2} \\ \leq c _ {F ^ {*}} + \frac {\ell_ {f , 1}}{2} \left(c _ {x} + c _ {x ^ {*}}\right) ^ {2} \tag {B.43} \\ \end{array}
$$
from which the proof is complete.
Corollary 4 ( $x_{t}$  bounded on the MoDo trajectory). Suppose Assumptions 1, 2 hold. Define  $\kappa = 3\ell_{f,1} / \mu$  and  $x_{\lambda ,z}^{*} = \arg \min_{x}F_{z}(x)\lambda$  with  $\lambda \in \Delta^M$ . Then there exists a finite positive constant  $c_{x^*}$  such that  $\| x_{\lambda ,z}^{*}\| \leq c_{x^{*}}$ . Choose the initial iterate to be bounded, i.e., there exists a finite positive constant  $c_{x_0}$  such that  $\| x_0\| \leq c_{x_0}$ , then for  $\{x_t\}$  generated by MoDo algorithm with  $\alpha_{t} = \alpha$  and  $0\leq \alpha \leq \ell_{f,1}^{-1}$ , we have
$$
\left\| x _ {t} \right\| \leq c _ {x}, \quad \text {w i t h} \quad c _ {x} = \max  \left\{\left(1 + \sqrt {2 \kappa}\right) c _ {x ^ {*}}, c _ {x _ {0}} \right\}. \tag {B.44}
$$
Proof of Corollary 4. Under Assumptions 1, 2, by Lemma 4,  $\| x_{\lambda ,z}^{*}\| < \infty$ , i.e., there exists a finite positive constant  $c_{x^*}$  such that  $\| x_{\lambda ,z}^{*}\| \leq c_{x^{*}}$ . Let  $c_{x,1} = \max \{(1 + \sqrt{2\kappa})^{-1}c_{x_0},c_{x^*}\}$ , and  $c_{x,2} = (1 + \sqrt{2\kappa})c_{x,1} = \max \{c_{x_0},(1 + \sqrt{2\kappa})c_{x^*}\}$  in Lemma 5. We then consider the following two cases:
1) If  $(1 + \sqrt{2\kappa})c_{x^*} \leq c_{x_0}$ , then  $\| x_{\lambda, z}^* \| \leq c_{x^*} \leq (1 + \sqrt{2\kappa})^{-1}c_{x_0}$ . Then it satisfies the condition in Lemma 5 that  $\| x_{\lambda, z}^* \| \leq c_{x,1}$  and  $\| x_0 \| \leq c_{x,2}$ . Applying Lemma 5 yields  $\| x_1 \| \leq c_{x,2}$ .
2) If  $(1 + \sqrt{2\kappa})c_{x^*} > c_{x_0}$ , then  $\| x_0\| \leq c_{x_0} < (1 + \sqrt{2\kappa})c_{x^*}$ . Then it satisfies the condition in Lemma 5 that  $\| x_{\lambda ,z}^{*}\| \leq c_{x,1}$  and  $\| x_0\| \leq c_{x,2}$ . Applying Lemma 5 yields  $\| x_1\| \leq c_{x,2}$ .
Therefore, (B.44) holds for  $t = 1$ . We then prove by induction that (B.44) also holds for  $t \in [T]$ . Assume (B.44) holds at  $1 \leq k \leq T - 1$ , i.e.,
$$
\left\| x _ {k} \right\| \leq c _ {x} = c _ {x, 2} \tag {B.45}
$$
Then by Lemma 5, at  $k + 1$
$$
\left\| x _ {k + 1} \right\| = \left\| G _ {\lambda_ {k + 1}, z _ {k, 3}} \left(x _ {k}\right) \right\| \leq c _ {x, 2}. \tag {B.46}
$$
Since  $\| x_1 \| \leq c_{x,2}$ , for  $t = 0, \dots, T - 1$ , we have
$$
\left\| x _ {t + 1} \right\| = \left\| G _ {\lambda_ {t + 1}, z _ {t, 3}} \left(x _ {t}\right) \right\| \leq c _ {x, 2}. \tag {B.47}
$$
Therefore, by mathematical induction,  $\| x_{t} \| \leq c_{x,2} = c_{x}$ , for all  $t \in [T]$ . The proof is complete.
Proof of Lemma 1. By Corollary 4, for  $\{x_{t}\}$  generated by MoDo algorithm with  $\alpha_{t} = \alpha$  and  $0\leq \alpha \leq \ell_{f,1}^{-1}$ , we have
$$
\left\| x _ {t} \right\| \leq c _ {x}, \quad \text {w i t h} \quad c _ {x} = \max  \left\{\left(1 + \sqrt {2 \kappa}\right) c _ {x ^ {*}}, c _ {x _ {0}} \right\}. \tag {B.48}
$$
According to Lemma 6, define  $\ell_f = \ell_{f,1}(c_x + c_{x^*})$ , and  $\ell_F = \sqrt{M}\ell_f$ , then it holds for all  $\lambda \in \Delta^{M}$
$$
\left\| \nabla F \left(x _ {t}\right) \lambda \right\| \leq \ell_ {f} \quad \text {a n d} \quad \left\| \nabla F \left(x _ {t}\right) \right\| \leq \left\| \nabla F \left(x _ {t}\right) \right\| _ {\mathrm {F}} \leq \ell_ {F}. \tag {B.49}
$$
B.4 Proof of Theorem 2 - PS generalization error in strongly convex case
Technical challenges. One challenge that the strongly convex objectives are not Lipschitz continuous for  $x \in \mathbb{R}^d$  is addressed by Lemma 1. Another challenge compared to static weighting or single-objective learning is that the MoDo algorithm involves the update of two coupled sequences  $\{x_t\}$  and  $\{\lambda_t\}$ . Consequently, the traditional standard argument that the SGD update for strongly convex objectives has the contraction property [14] does not necessarily hold in our case since the weighting parameter  $\lambda$  is changing, as detailed in Section B.4.1. Nevertheless, we manage to derive a tight stability bound when  $\gamma = \mathcal{O}(T^{-1})$ , as detailed in Section B.4.3.
Organization of proof. In Section B.4.1, we prove the properties of the MoDo update, including expansiveness or non-expansiveness and boundedness. Building upon these properties, in Section B.4.3, we prove the upper bound of argument stability in Theorem 5, and the upper bound of MOL uniform stability. To show the tightness of the upper bound, in Section B.4.4, Theorem 6, we derive a matching lower bound of MOL uniform stability. Combining the upper bound in Section B.4.3 and the lower bound in Section B.4.4 leads to the results in Theorem 2, whose proof is in Section B.4.5.
B.4.1 Expansiveness and boundedness of MoDo update
In this section, we prove the properties of the update function of MoDo at each iteration, including boundedness and approximate expansiveness, which is then used to derive the algorithm stability. For  $z, z_1, z_2 \in S$ ,  $\lambda \in \Delta^M$ , recall that the update functions of MoDo is
$$
G _ {x, z _ {1}, z _ {2}} (\lambda) = \Pi_ {\Delta^ {M}} \left(\lambda - \gamma \nabla F _ {z _ {1}} (x) ^ {\top} \nabla F _ {z _ {2}} (x) \lambda\right)
$$
$$
G _ {\lambda , z} (x) = x - \alpha \nabla F _ {z} (x) \lambda .
$$
Lemma 7 (Boundedness of update function of MoDo). Let  $\ell_f$  be a positive constant. If  $\| \nabla F_z(x)\lambda \| \leq \ell_f$  for all  $\lambda \in \Delta^M$ ,  $z \in S$  and  $x \in \{x_t\}_{t=1}^T$  generated by the MoDo algorithm with step size  $\alpha_t \leq \alpha$ , then  $G_{\lambda, z}(x)$  is  $(\alpha \ell_f)$ -bounded on the trajectory of MoDo, i.e.,
$$
\sup  _ {\infty} \| G _ {\lambda , z} (x) - x \| \leq \alpha \ell_ {f}. \tag {B.50}
$$
$$
x \in \left\{x _ {t} \right\} _ {t = 1} ^ {T}
$$
Proof. For all  $x \in \{x_t\}_{t=1}^T$ ,  $\lambda \in \Delta^M$ , and  $z \in S$ , since  $\|\nabla F_z(x)\lambda\| \leq \ell_f$ , we have
$$
\left\| G _ {\lambda , z} (x) - x \right\| \leq \left\| \alpha \nabla F _ {z} (x) \lambda \right\| \leq \alpha \ell_ {f} \tag {B.51}
$$
which proves the boundedness.
Lemma 8 (Properties of update function of MoDo in convex case). Suppose Assumptions 1, 2 hold. Let  $\ell_f$  be a positive constant. If for all  $\lambda, \lambda' \in \Delta^M$ ,  $z \in S$ , and  $x \in \{x_t\}_{t=1}^T$ ,  $x' \in \{x_t'\}_{t=1}^T$  generated by the MoDo algorithm on datasets  $S$  and  $S'$ , respectively, we have  $\|\nabla F_z(x)\lambda\| \leq \ell_f$ ,  $\|\nabla F_z(x')\lambda'\| \leq \ell_f$ , and  $\|\nabla F_z(x)\| \leq \ell_F$ ,  $\|\nabla F_z(x')\| \leq \ell_F$ , and step sizes of MoDo satisfy  $\alpha_t \leq \alpha$ ,  $\gamma_t \leq \gamma$ , it holds that
$$
\begin{array}{l} \left\| G _ {\lambda , z} (x) - G _ {\lambda^ {\prime}, z} \left(x ^ {\prime}\right) \right\| ^ {2} \leq \left(1 - 2 \alpha \mu + 2 \alpha^ {2} \ell_ {f, 1} ^ {2}\right) \| x - x ^ {\prime} \| ^ {2} \\ + 2 \alpha \ell_ {F} \| x - x ^ {\prime} \| \| \lambda - \lambda^ {\prime} \| + 2 \alpha^ {2} \ell_ {F} ^ {2} \| \lambda - \lambda^ {\prime} \| ^ {2} \tag {B.52} \\ \end{array}
$$
$$
\begin{array}{l} \left\| G _ {x, z _ {1}, z _ {2}} (\lambda) - G _ {x ^ {\prime}, z _ {1}, z _ {2}} \left(\lambda^ {\prime}\right) \right\| ^ {2} \leq \left((1 + \ell_ {F} ^ {2} \gamma) ^ {2} + (1 + \ell_ {F} ^ {2} \gamma) \ell_ {g, 1} \gamma\right) \| \lambda - \lambda^ {\prime} \| ^ {2} \\ + \left((1 + \ell_ {F} ^ {2} \gamma) \ell_ {g, 1} \gamma + \ell_ {g, 1} ^ {2} \gamma^ {2}\right) \| x - x ^ {\prime} \| ^ {2}. \tag {B.53} \\ \end{array}
$$
Proof. The squared norm of the difference of  $G_{\lambda, z}(x)$  and  $G_{\lambda', z}(x')$  can be bounded by
$$
\begin{array}{l} \left\| G _ {\lambda , z} (x) - G _ {\lambda^ {\prime}, z} \left(x ^ {\prime}\right) \right\| ^ {2} \\ = \| x - x ^ {\prime} \| ^ {2} - 2 \alpha \langle x - x ^ {\prime}, \nabla F _ {z} (x) \lambda - \nabla F _ {z} \left(x ^ {\prime}\right) \lambda^ {\prime} \rangle + \alpha^ {2} \| \nabla F _ {z} (x) \lambda - \nabla F _ {z} \left(x ^ {\prime}\right) \lambda^ {\prime} \| ^ {2} \\ \stackrel {(a)} {\leq} \| x - x ^ {\prime} \| ^ {2} - 2 \alpha \langle x - x ^ {\prime}, (\nabla F _ {z} (x) - \nabla F _ {z} (x ^ {\prime})) \lambda \rangle + 2 \alpha^ {2} \| (\nabla F _ {z} (x) - \nabla F _ {z} (x ^ {\prime})) \lambda \| ^ {2} \\ + 2 \alpha \langle x - x ^ {\prime}, \nabla F _ {z} \left(x ^ {\prime}\right) \left(\lambda^ {\prime} - \lambda\right) \rangle + 2 \alpha^ {2} \| \nabla F _ {z} \left(x ^ {\prime}\right) \left(\lambda - \lambda^ {\prime}\right) \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(b)} {\leq} (1 - 2 \alpha \mu + 2 \alpha^ {2} \ell_ {f, 1} ^ {2}) \| x - x ^ {\prime} \| ^ {2} + 2 \alpha \langle x - x ^ {\prime}, \nabla F _ {z} (x ^ {\prime}) (\lambda^ {\prime} - \lambda) \rangle + 2 \alpha^ {2} \ell_ {F} ^ {2} \| \lambda^ {\prime} - \lambda \| ^ {2} \\ \stackrel {(c)} {\leq} \left(1 - 2 \alpha \mu + 2 \alpha^ {2} \ell_ {f, 1} ^ {2}\right) \| x - x ^ {\prime} \| ^ {2} + 2 \alpha \ell_ {F} \| x - x ^ {\prime} \| \| \lambda^ {\prime} - \lambda \| + 2 \alpha^ {2} \ell_ {F} ^ {2} \| \lambda^ {\prime} - \lambda \| ^ {2} \tag {B.54} \\ \end{array}
$$
where  $(a)$  follows from rearranging and that  $\| a + b\| ^2\leq 2\| a\| ^2 +2\| b\| ^2$  ;  $(b)$  follows from the  $\mu$  -strong convexity of  $F_{z}(x)\lambda ,\ell_{f,1}$  -Lipschitz continuity of  $\nabla F_z(x)\lambda$  , and that  $\| \nabla F_z(x')\| \leq \ell_F$  for  $x^{\prime}\in \{x_{t}^{\prime}\}_{t = 1}^{T}$  ; and,  $(c)$  follows from Cauchy-Schwartz inequality.
And  $\| G_{x,z_1,z_2}(\lambda) - G_{x',z_1,z_2}(\lambda')\|$  can be bounded by
$$
\begin{array}{l} \left\| G _ {x, z _ {1}, z _ {2}} (\lambda) - G _ {x ^ {\prime}, z _ {1}, z _ {2}} \left(\lambda^ {\prime}\right) \right\| \\ = \| \Pi_ {\Delta^ {M}} (\lambda - \gamma (\nabla F _ {z _ {1}} (x) ^ {\top} \nabla F _ {z _ {2}} (x)) \lambda) - \Pi_ {\Delta^ {M}} (\lambda^ {\prime} - \gamma (\nabla F _ {z _ {1}} (x ^ {\prime}) ^ {\top} \nabla F _ {z _ {2}} (x ^ {\prime})) \lambda^ {\prime}) \| \\ \end{array}
$$
$$
\stackrel {(d)} {\leq} \left\| \lambda - \lambda^ {\prime} - \gamma \left(\nabla F _ {z _ {1}} (x) ^ {\top} \nabla F _ {z _ {2}} (x) \lambda - \nabla F _ {z _ {1}} \left(x ^ {\prime}\right) ^ {\top} \nabla F _ {z _ {2}} \left(x ^ {\prime}\right) \lambda^ {\prime}\right) \right\|
$$
$$
\stackrel {(e)} {\leq} \| \lambda - \lambda^ {\prime} \| + \gamma \| \nabla F _ {z _ {1}} (x) ^ {\top} \nabla F _ {z _ {2}} (x) (\lambda - \lambda^ {\prime}) \| + \gamma \| (\nabla F _ {z _ {1}} (x) ^ {\top} \nabla F _ {z _ {2}} (x) - \nabla F _ {z _ {1}} (x ^ {\prime}) ^ {\top} \nabla F _ {z _ {2}} (x ^ {\prime})) \lambda^ {\prime} \|
$$
$$
\stackrel {(f)} {\leq} \| \lambda - \lambda^ {\prime} \| + \gamma \ell_ {F} ^ {2} \| \lambda - \lambda^ {\prime} \| + \gamma \| \left(\nabla F _ {z _ {1}} (x) ^ {\top} \nabla F _ {z _ {2}} (x) - \nabla F _ {z _ {1}} \left(x ^ {\prime}\right) ^ {\top} \nabla F _ {z _ {2}} \left(x ^ {\prime}\right)\right) \lambda^ {\prime} \|
$$
$$
\begin{array}{l} \stackrel {(g)} {\leq} \left\| \lambda - \lambda^ {\prime} \right\| + \gamma \ell_ {F} ^ {2} \left\| \lambda - \lambda^ {\prime} \right\| + \gamma \left(\left\| \left(\nabla F _ {z _ {1}} (x) - \nabla F _ {z _ {1}} \left(x ^ {\prime}\right)\right) ^ {\top} \nabla F _ {z _ {2}} (x) \lambda^ {\prime} \right\| \right. \\ + \left\| \nabla F _ {z _ {1}} \left(x ^ {\prime}\right) ^ {\top} \left(\nabla F _ {z _ {2}} (x) - \nabla F _ {z _ {2}} \left(x ^ {\prime}\right)\right) \lambda^ {\prime} \right\| \\ \end{array}
$$