SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. From the update of  $x_{t}^{\prime}$ , we have
$$
\begin{array}{l} x _ {t + 1} ^ {\prime} = x _ {t} ^ {\prime} - \alpha A x _ {t} ^ {\prime} + \alpha z _ {t, 3} ^ {\prime} (b ^ {\top} \lambda_ {t + 1} ^ {\prime}) \\ = x _ {t} ^ {\prime} - \alpha A x _ {t} ^ {\prime} + \alpha c _ {t, 3} ^ {\prime} v \left(b ^ {\top} \lambda_ {t + 1} ^ {\prime}\right) \tag {B.107} \\ \end{array}
$$
Suppose  $x_{t}^{\prime} = c_{x,t}^{\prime}v$ , then  $x_{t + 1}^{\prime} = c_{x,t + 1}^{\prime}v$  with
$$
c _ {x, t + 1} ^ {\prime} = (1 - \alpha \mu) c _ {x, t} ^ {\prime} + \alpha c _ {t, 3} ^ {\prime} \left(b ^ {\top} \lambda_ {t + 1} ^ {\prime}\right)
$$
and  $\mathbb{E}_A[x_{t + 1}^{\prime}] = (1 - \alpha \mu)\mathbb{E}_A[x_t^{\prime}] + \alpha \mu^{\prime}v\mathbb{E}_A(b^\top \lambda_{t + 1}^{\prime})$
Applying the above inequality recursively gives
$$
\begin{array}{l} v ^ {\top} \mathbb {E} _ {A} \left[ x _ {t} ^ {\prime} \right] = v ^ {\top} (1 - \alpha \mu) ^ {t} x _ {0} + \alpha \mu^ {\prime} \Big (\sum_ {t ^ {\prime} = 0} ^ {t - 1} (1 - \alpha \mu) ^ {t - 1 - t ^ {\prime}} \mathbb {E} _ {A} \left(b ^ {\top} \lambda_ {t ^ {\prime} + 1} ^ {\prime}\right) \Big) \\ \geq v ^ {\top} (1 - \alpha \mu) ^ {t} x _ {0} + \alpha \mu^ {\prime} \frac {1 - (1 - \alpha \mu) ^ {t}}{\alpha \mu} \quad b ^ {\top} \lambda \geq 1 \text {f o r a l l} \lambda \in \Delta^ {M} \\ = (1 - \alpha \mu) ^ {t} \left(v ^ {\top} x _ {0} - \mu^ {\prime} \mu^ {- 1}\right) + \mu^ {\prime} \mu^ {- 1}. \\ \end{array}
$$
Since  $x_0 = 7v$ ,  $\mu' \mu^{-1} \leq 1$ , it holds that
$$
v ^ {\top} \mathbb {E} _ {A} \left[ x _ {t} ^ {\prime} \right] = (1 - \alpha \mu) ^ {t} \left(7 - \mu^ {\prime} \mu^ {- 1}\right) + \mu^ {\prime} \mu^ {- 1} \geq 6 (1 - \alpha \mu) ^ {t} + \mu^ {\prime} \mu^ {- 1}.
$$
$$
\begin{array}{l} v ^ {\top} \mathbb {E} _ {A} \left[ A x _ {t} ^ {\prime} - z _ {t, 2} b ^ {\top} \lambda_ {t} ^ {\prime} \right] = \mu \mathbb {E} _ {A} \left[ v ^ {\top} x _ {t} ^ {\prime} - b ^ {\top} \lambda_ {t} ^ {\prime} \right] \geq \mu \left(6 (1 - \alpha \mu) ^ {t} + \mu^ {\prime} \mu^ {- 1} - (1 + \sqrt {2})\right) \\ \geq \mu \left(6 (1 - \alpha \mu t) - (1 + \sqrt {2})\right) \quad \mu^ {\prime} \mu^ {- 1} \geq 0 \\ \geq \mu \Big (6 (1 - \frac {1}{4}) - (1 + \sqrt {2}) \Big) \geq 0 \quad \alpha = \frac {1}{4 \mu T} \\ \end{array}
$$
By the update of  $\lambda_t^\prime$  from (B.97),
$$
\begin{array}{l} b ^ {\top} \mathbb {E} _ {A} \left[ \lambda_ {t + 1} ^ {\prime} - \lambda_ {t} ^ {\prime} \right] = b ^ {\top} \gamma b _ {P} \mathbb {E} _ {A} \left[ z _ {t, 1} ^ {\prime \top} \left(A x _ {t} ^ {\prime} - z _ {t, 2} ^ {\prime} b ^ {\top} \lambda_ {t} ^ {\prime}\right) \right] \\ = \gamma \mathbb {E} _ {A} \left[ \mu^ {\prime} v ^ {\top} \left(A x _ {t} ^ {\prime} - z _ {t, 2} ^ {\prime} b ^ {\top} \lambda_ {t} ^ {\prime}\right) \right] \\ \mathbb {E} _ {A} \left[ z _ {t, 1} ^ {\prime} \right] = \mu^ {\prime} v, z _ {t, 1} ^ {\prime} \text {i n d e p e n d e n t o f} \lambda_ {t} ^ {\prime} \text {a n d} x _ {t} ^ {\prime} \\ = \gamma \mu^ {\prime} v ^ {\top} \mathbb {E} _ {A} \left[ A x _ {t} ^ {\prime} - \mu^ {\prime} v b ^ {\top} \lambda_ {t} ^ {\prime} \right] \quad \mathbb {E} _ {A} \left[ z _ {t, 2} ^ {\prime} \right] = \mu^ {\prime} v, z _ {t, 2} ^ {\prime} \text {i n d e p e n d e n t} \lambda_ {t} ^ {\prime} \\ \geq \gamma \mu^ {\prime} v ^ {\top} \mathbb {E} _ {A} \left[ A x _ {t} ^ {\prime} - \mu v b ^ {\top} \lambda_ {t} ^ {\prime} \right] \quad \mu^ {\prime} \leq \mu , b ^ {\top} \lambda \geq 1 \text {f o r a l l} \lambda \in \Delta^ {M} \\ = \gamma \mu^ {\prime} v ^ {\top} \mathbb {E} _ {A} \left[ A x _ {t} ^ {\prime} - z _ {t, 2} b ^ {\top} \lambda_ {t} ^ {\prime} \right] \geq 0. \quad \mathbb {E} _ {A} \left[ z _ {t, 2} \right] = \mu v, z _ {t, 2} \text {i n d e p e n d e n t} \lambda_ {t} ^ {\prime} \\ \end{array}
$$
Theorem 6. Suppose Assumptions 1 and 2 hold. Under Example 1 with  $M = 2$ , choose  $\lambda_0 = \frac{1}{M} \mathbf{1}$ ,  $x_0 = x_0' = 7v$ ,  $\alpha = \frac{1}{4\mu T}$ ,  $0 < \gamma \leq \frac{1}{2MT\ell_F\ell_f}$ , and  $T \leq 4n^{\frac{2}{3}}$  for the MoDo algorithm. Denote  $\{x_t\}$ ,  $\{\lambda_t\}$  and  $\{x_t'\}$ ,  $\{\lambda_t'\}$  as the sequences generated by the MoDo algorithm with dataset  $S$  and  $S'$ , respectively. Then it holds that
$$
\mathbb {E} \left[ \left\| x _ {T} - x _ {T} ^ {\prime} \right\| \right] \geq \frac {\gamma T}{2 n ^ {2}} + \frac {1}{1 6 n}. \tag {B.108}
$$
Proof. Denote  $\delta_t = \| x_t - x_t' \|$ ,  $\zeta_t = \| \lambda_t - \lambda_t' \|$ . From Lemma 11, it holds that
$$
\mathbb {E} [ \delta_ {t + 1} ] = \mathbb {E} [ | \varphi_ {x, t + 1} | | v | ] = \mathbb {E} [ | \varphi_ {x, t + 1} | ] \geq \mathbb {E} [ \varphi_ {x, t + 1} ] \tag {B.109}
$$
$$
\mathbb {E} [ \zeta_ {t + 1} ] = \mathbb {E} [ | \varphi_ {\lambda , t + 1} | | v | | ] = \mathbb {E} [ | \varphi_ {\lambda , t + 1} | ] \geq \mathbb {E} [ \varphi_ {\lambda , t + 1} ] \tag {B.110}
$$
where  $\varphi_{x,t},\varphi_{\lambda ,t}$  satisfy
$$
\begin{array}{l} \varphi_ {x, t + 1} = (1 - \alpha \mu + \alpha \gamma c _ {t, 3} c _ {t, 1} \mu) \varphi_ {x, t} + \alpha c _ {t, 3} (1 - \gamma c _ {t, 1} c _ {t, 2}) \varphi_ {\lambda , t} + \mathbb {1} (i _ {t, 3} = j) \alpha \left(b ^ {\top} \lambda_ {t + 1} ^ {\prime}\right) \\ + \mathbb {1} \left(i _ {t, 1} = j\right) \gamma \alpha c _ {t, 3} \left(\mu v ^ {\top} x _ {t} ^ {\prime} - c _ {t, 2} b ^ {\top} \lambda_ {t} ^ {\prime}\right) - \mathbb {1} \left(i _ {t, 2} = j\right) \gamma \alpha c _ {t, 3} c _ {t, 1} ^ {\prime} b ^ {\top} \lambda_ {t} ^ {\prime} \tag {B.111} \\ \end{array}
$$
$$
\begin{array}{l} \varphi_ {\lambda , t + 1} = \left(1 - \gamma c _ {t, 1} c _ {t, 2}\right) \varphi_ {\lambda , t} + \gamma c _ {t, 1} \mu \varphi_ {x, t} \\ + \mathbb {1} \left(i _ {t, 1} = j\right) \gamma \left(\mu \left(v ^ {\top} x _ {t} ^ {\prime}\right) - c _ {t, 2} \left(b ^ {\top} \lambda_ {t} ^ {\prime}\right)\right) - \mathbb {1} \left(i _ {t, 2} = j\right) \gamma c _ {t, 1} ^ {\prime} \left(b ^ {\top} \lambda_ {t} ^ {\prime}\right). \tag {B.112} \\ \end{array}
$$
The expectation of  $\varphi_{x,t + 1}$  can be further bounded as
$$
\begin{array}{l} \mathbb {E} \left[ \varphi_ {x, t + 1} \right] = \mathbb {E} \left[ \left(1 - \alpha \mu + \alpha \gamma c _ {t, 3} c _ {t, 1} \mu\right) \varphi_ {x, t} + \alpha c _ {t, 3} \left(1 - \gamma c _ {t, 1} c _ {t, 2}\right) \varphi_ {\lambda , t} + \frac {1}{n} \alpha \left(b ^ {\top} \lambda_ {t + 1} ^ {\prime}\right) \right. \\ + \frac {1}{n} \gamma \alpha c _ {t, 3} (\mu v ^ {\top} x _ {t} ^ {\prime} - c _ {t, 2} b ^ {\top} \lambda_ {t} ^ {\prime}) - \frac {1}{n} \gamma \alpha c _ {t, 3} c _ {t, 1} ^ {\prime} b ^ {\top} \lambda_ {t} ^ {\prime} ] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {\geq} (1 - \alpha \mu (1 - \gamma \mu^ {2})) \mathbb {E} [ \varphi_ {x, t} ] + \alpha \mu (1 - \gamma \mu^ {2}) \mathbb {E} [ \varphi_ {\lambda , t} ] + \frac {1}{n} \alpha b ^ {\top} \mathbb {E} [ \lambda_ {t + 1} ^ {\prime} ] - \frac {1}{n} \gamma \alpha \mu \mu^ {\prime} b ^ {\top} \mathbb {E} [ \lambda_ {t} ^ {\prime} ] \\ \geq (1 - \alpha \mu (1 - \gamma \mu^ {2})) \mathbb {E} [ \varphi_ {x, t} ] + \alpha \mu (1 - \gamma \mu^ {2}) \mathbb {E} [ \varphi_ {\lambda , t} ] + \frac {1}{n} \alpha b ^ {\top} \mathbb {E} [ \lambda_ {t} ^ {\prime} ] (1 - \gamma \mu^ {2}) \tag {B.113} \\ \end{array}
$$
where  $(a)$  follows from  $\mathbb{E}_A[c_{t,s}] = \mu$ ,  $\mathbb{E}_A[c_{t,s}^{\prime}] = \mu^{\prime}\leq \mu$ ,  $\mathbb{E}[\lambda_{t + 1}^{\prime}]\geq \mathbb{E}[\lambda_t^{\prime}]$  by Lemma 12, and the fact that  $c_{t,s}$  is independent of  $\varphi_{x,t}$ .
Similarly, the expectation of  $\varphi_{\lambda ,t + 1}$  can be further bounded as
$$
\begin{array}{l} \mathbb {E} \left[ \varphi_ {\lambda , t + 1} \right] = \mathbb {E} \left[ \left(1 - \gamma c _ {t, 1} c _ {t, 2}\right) \varphi_ {\lambda , t} + \gamma c _ {t, 1} \mu \varphi_ {x, t} + \frac {1}{n} \gamma \left(\mu \left(v ^ {\top} x _ {t} ^ {\prime}\right) - c _ {t, 2} \left(b ^ {\top} \lambda_ {t} ^ {\prime}\right)\right) - \frac {1}{n} \gamma c _ {t, 1} ^ {\prime} \left(b ^ {\top} \lambda_ {t} ^ {\prime}\right) \right] \\ \stackrel {(b)} {\geq} \left(1 - \gamma \mu^ {2}\right) \mathbb {E} \left[ \varphi_ {\lambda , t} \right] + \gamma \mu^ {2} \mathbb {E} \left[ \varphi_ {x, t} \right] - \frac {1}{n} \gamma \mu^ {\prime} b ^ {\top} \mathbb {E} \left[ \lambda_ {t} ^ {\prime} \right] \tag {B.114} \\ \end{array}
$$
where  $(b)$  follows from  $\mathbb{E}_A[c_{t,s}] = \mu$ ,  $\mathbb{E}_A[c_{t,s}^{\prime}] = \mu^{\prime}\leq \mu$ , and Lemma 12.
The above arguments prove that
$$
\left[ \begin{array}{c} \mathbb {E} [ \delta_ {t + 1} ] \\ \mathbb {E} [ \zeta_ {t + 1} ] \end{array} \right] \geq \left[ \begin{array}{c} \mathbb {E} [ \varphi_ {x, t + 1} ] \\ \mathbb {E} [ \varphi_ {\lambda , t + 1} ] \end{array} \right] \geq \underbrace {\left[ \begin{array}{c c} (1 - \alpha \mu (1 - \gamma \mu^ {2})) & \alpha \mu (1 - \gamma \mu^ {2}) \\ \gamma \mu^ {2} & (1 - \gamma \mu^ {2}) \end{array} \right]} _ {B} \left[ \begin{array}{c} \mathbb {E} [ \varphi_ {x, t} ] \\ \mathbb {E} [ \varphi_ {\lambda , t} ] \end{array} \right] + \frac {1}{n} \left[ \begin{array}{c} \alpha (1 - \gamma \mu^ {2}) \\ - \mu^ {\prime} \gamma \end{array} \right] \tag {B.115}
$$
where the inequality for vectors denotes the inequality of each corresponding element in the vectors, and matrix  $B$  has  $v_{B,1} = [1,1]^{\top}$  as an eigenvector associated with the eigenvalue 1 because
$$
B v _ {B, 1} = \left[ \begin{array}{c c} (1 - \alpha \mu (1 - \gamma \mu^ {2})) & \alpha \mu (1 - \gamma \mu^ {2}) \\ \gamma \mu^ {2} & (1 - \gamma \mu^ {2}) \end{array} \right] \left[ \begin{array}{l} 1 \\ 1 \end{array} \right] = \left[ \begin{array}{l} 1 \\ 1 \end{array} \right]. \tag {B.116}
$$
Similarly, since
$$
\begin{array}{l} B \left[ \begin{array}{c} \alpha (1 - \gamma \mu^ {2}) \\ - \gamma \mu \end{array} \right] = \left[ \begin{array}{c c} (1 - \alpha \mu (1 - \gamma \mu^ {2})) & \alpha \mu (1 - \gamma \mu^ {2}) \\ \gamma \mu^ {2} & (1 - \gamma \mu^ {2}) \end{array} \right] \left[ \begin{array}{c} \alpha (1 - \gamma \mu^ {2}) \\ - \gamma \mu \end{array} \right] \\ = \left[ \begin{array}{c} (1 - \alpha \mu (1 - \gamma \mu^ {2})) \alpha (1 - \gamma \mu^ {2}) - \gamma \mu \alpha \mu (1 - \gamma \mu^ {2}) \\ \gamma \mu^ {2} \alpha (1 - \gamma \mu^ {2}) - \gamma \mu (1 - \gamma \mu^ {2}) \end{array} \right] \\ = \left[ \begin{array}{c} (1 - \alpha \mu) (1 - \gamma \mu^ {2}) \alpha (1 - \gamma \mu^ {2}) \\ - (1 - \alpha \mu) (1 - \gamma \mu^ {2}) \gamma \mu \end{array} \right] = (1 - \alpha \mu) (1 - \gamma \mu^ {2}) \left[ \begin{array}{c} \alpha (1 - \gamma \mu^ {2}) \\ - \gamma \mu \end{array} \right], \tag {B.117} \\ \end{array}
$$
then  $v_{B,2} = [\alpha (1 - \gamma \mu^2), - \gamma \mu ]^\top$  is another eigenvector of  $B$  with a positive eigenvalue  $(1 - \alpha \mu)(1 - \gamma \mu^2) < 1$ . Let  $Q_{B} = [v_{B,1},v_{B,2}]$ , which can be expressed as
$$
Q _ {B} = \left[ v _ {B, 1}, v _ {B, 2} \right] = \left[ \begin{array}{c c} 1 & \alpha (1 - \gamma \mu^ {2}) \\ 1 & - \gamma \mu \end{array} \right]. \tag {B.118}
$$
Then  $B$  has eigenvalue decomposition  $B = Q_B \Lambda_B Q_B^{-1}$ , where  $\Lambda_B = \mathrm{diag}([1, (1 - \alpha \mu)(1 - \gamma \mu^2)])$  and thus  $B^t = Q_B \Lambda_B^t Q_B^{-1}$  for  $t \in [T]$ .
Let  $[\alpha (1 - \gamma \mu^2), - \mu '\gamma ]^\top = Q_B[c_{B,1},c_{B,2}]^\top$  , where  $[c_{B,1},c_{B,2}]\top = Q_B^{-1}[\alpha (1 - \gamma \mu^2), - \mu '\gamma ]^\top$  can be computed by
$$
\begin{array}{l} \left[ \begin{array}{c} c _ {B, 1} \\ c _ {B, 2} \end{array} \right] = Q _ {B} ^ {- 1} \left[ \begin{array}{c} \alpha (1 - \gamma \mu^ {2}) \\ - \mu^ {\prime} \gamma \end{array} \right] = - \frac {1}{\alpha (1 - \gamma \mu^ {2}) + \gamma \mu} \left[ \begin{array}{c c} - \gamma \mu & - \alpha (1 - \gamma \mu^ {2}) \\ - 1 & 1 \end{array} \right] \left[ \begin{array}{c} \alpha \\ - \mu^ {\prime} \gamma \end{array} \right] \\ = \frac {1}{\alpha \left(1 - \gamma \mu^ {2}\right) + \gamma \mu} \left[ \begin{array}{c} \alpha \gamma \left(\mu - \mu^ {\prime}\right) \left(1 - \gamma \mu^ {2}\right) \\ \alpha + \mu^ {\prime} \gamma \end{array} \right] \geq \left[ \begin{array}{c} \frac {1}{2 n} \gamma \\ 1 \end{array} \right] \tag {B.119} \\ \end{array}
$$
where the last inequality follows from  $\mu -\mu^{\prime} = \frac{1}{n}$  and  $\alpha (1 - \gamma \mu^{2})\geq \frac{1}{2}\alpha = 1 / (8\mu T)\geq \gamma \mu$  for  $c_{B,1}\geq \frac{1}{2n}\gamma$ , and  $\alpha \mu^2 = \mu /(4T) = 4n^{-\frac{1}{3}}T^{-1}\geq n^{-1} = \mu -\mu '$ , so that  $\alpha +\mu^{\prime}\gamma = \alpha +\gamma (\mu^{\prime} - \mu +$ $\mu)\geq \alpha +\gamma (\mu -\alpha \mu^{2}) = \alpha (1 - \gamma \mu^{2}) + \gamma \mu$  for  $c_{B,2}\geq 1$
Since all elements in  $B$  are positive, multiplying  $B$  on both sides preserves inequality. Applying (B.115) recursively yields
$$
\begin{array}{l} \left[ \begin{array}{c} \mathbb {E} [ \delta_ {T} ] \\ \mathbb {E} [ \zeta_ {T} ] \end{array} \right] \geq \sum_ {t = 0} ^ {T - 1} B ^ {T - 1 - t} \frac {1}{n} \left[ \begin{array}{c} \alpha (1 - \gamma \mu^ {2}) \\ - \mu^ {\prime} \gamma \end{array} \right] = \sum_ {t = 0} ^ {T - 1} B ^ {T - 1 - t} \frac {1}{n} \left[ \begin{array}{c} \alpha (1 - \gamma \mu^ {2}) \\ - \mu^ {\prime} \gamma \end{array} \right] = \sum_ {t = 0} ^ {T - 1} B ^ {T - 1 - s} \frac {1}{n} Q _ {B} \left[ \begin{array}{c} c _ {B, 1} \\ c _ {B, 2} \end{array} \right] \\ = \frac {1}{n} \sum_ {t = 0} ^ {T - 1} 1 ^ {T - 1 - t} c _ {B, 1} v _ {B, 1} + \frac {1}{n} \sum_ {t = 0} ^ {T - 1} \left((1 - \alpha \mu) (1 - \gamma \mu^ {2})\right) ^ {T - 1 - t} c _ {B, 2} v _ {B, 2} \\ \geq \frac {T}{n} c _ {B, 1} v _ {B, 1} + \frac {1}{8 n \alpha} c _ {B, 2} v _ {B, 2} \geq \frac {\gamma T}{2 n ^ {2}} v _ {B, 1} + \frac {1}{8 n \alpha} v _ {B, 2} \tag {B.120} \\ \end{array}
$$
where the last inequality follows from  $c_{B,1} \geq \frac{1}{2n}\gamma$ , and  $c_{B,2} \geq 1$ . Plugging in  $v_{B,1} = [1,1]^{\top}$  and  $v_{B,2} = [\alpha (1 - \gamma \mu^2), -\gamma \mu]^{\top}$ , and since  $(1 - \gamma \mu^2) \geq \frac{1}{2}$ , it follows that  $\mathbb{E}[\delta_T] \geq \frac{\gamma T}{2n^2} + \frac{1}{16n}$ .
B.4.5 Proof of Theorem 2
Proof of Theorem 2. Combining the argument stability in Theorem 5, and Assumption 1, the MOL uniform stability can be bounded by
$$
\begin{array}{l} \sup _ {z} \mathbb {E} _ {A} [ \| \nabla F _ {z} (A (S)) - \nabla F _ {z} (A (S ^ {\prime})) \| _ {\mathrm {F}} ^ {2} ] \\ \leq \mathbb {E} _ {A} \left[ \ell_ {F, 1} ^ {2} \| A (S) - A \left(S ^ {\prime}\right) \| ^ {2} \right] \quad \text {b y} \\ \leq \frac {4 8}{\mu n} \ell_ {f} ^ {2} \ell_ {F, 1} ^ {2} \left(\alpha + \frac {1 2 + 4 M \ell_ {f} ^ {2}}{\mu n} + \frac {1 0 M \ell_ {f} ^ {4} \gamma}{\mu}\right). \tag {B.121} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {A, S} \left[ R _ {\text {g e n}} (A (S)) \right] \leq \mathbb {E} _ {A, S} \left[ \| \nabla F (A (S)) - \nabla F _ {S} (A (S)) \| _ {\mathrm {F}} \right] \quad \text {b y} \\ \leq 4 \epsilon_ {\mathrm {F}} + \sqrt {n ^ {- 1} \mathbb {E} _ {S} \left[ \mathbb {V} _ {z \sim \mathcal {D}} (\nabla F _ {z} (A (S))) \right]} \quad \text {b y} \\ = \mathcal {O} \left(n ^ {- \frac {1}{2}}\right). \quad \text {b y} \tag {B.121} \\ \end{array}
$$
The proof of the upper bound is complete. We then prove the MOL uniform stability lower bound based on the argument uniform stability lower bound in Theorem 6. By the strong convexity of the function  $f_{m,z}(x)$ , for all  $m \in [M]$
$$
\begin{array}{l} \sup  _ {z} \mathbb {E} _ {A} [ \| \nabla F _ {z} (A (S)) - \nabla F _ {z} (A (S ^ {\prime})) \| _ {\mathrm {F}} ^ {2} ] \geq \mathbb {E} _ {A} [ M \mu^ {2} \| A (S) - A (S ^ {\prime}) \| ^ {2} ] \quad \text {b y} \\ \geq \frac {M \mu^ {2}}{2 5 6 n ^ {2}}. \quad \text {b y} \\ \end{array}
$$
The proof of the lower bound is complete.
C Bounding the optimization error
Lemma 13 (Uniqueness of CA direction). Given  $Q \in \mathbb{R}^{d \times M}$ , then  $d_{Q} \coloneqq Q\lambda^{*}$  with  $\lambda^{*} \in \arg \min_{\lambda \in \Delta^{M}} \|Q\lambda\|^{2}$  exists, and  $d_{Q}$  is unique.
Proof. This is a standard result due to convexity of the subproblem. Proof is given in [9, Section 2].  $\square$
Lemma 14. For any  $x\in \mathbb{R}^d$  , define  $\lambda^{*}(x)$  such that
$$
\lambda^ {*} (x) \in \arg \min  _ {\lambda \in \Delta^ {M}} \| \nabla F (x) \lambda \| ^ {2}. \tag {C.1}
$$
Then, for any  $x \in \mathbb{R}^d$  and  $\lambda \in \Delta^M$ , it holds that
$$
\langle \nabla F (x) \lambda^ {*} (x), \nabla F (x) \lambda \rangle \geq \| \nabla F (x) \lambda^ {*} (x) \| ^ {2}, \tag {C.2}
$$
$$
a n d \| \nabla F (x) \lambda - \nabla F (x) \lambda^ {*} (x) \| ^ {2} \leq \| \nabla F (x) \lambda \| ^ {2} - \| \nabla F (x) \lambda^ {*} (x) \| ^ {2}. \tag {C.3}
$$
Proof. By the first order optimality condition for (C.1), for any  $x \in \mathbb{R}^d$  and  $\lambda \in \Delta^M$ , we have
$$
\langle \nabla F (x) ^ {\top} \nabla F (x) \lambda^ {*} (x), \lambda - \lambda^ {*} (x) \rangle \geq 0. \tag {C.4}
$$
By rearranging the above inequality, we obtain
$$
\langle \nabla F (x) \lambda^ {*} (x), \nabla F (x) \lambda \rangle \geq \| \nabla F (x) \lambda^ {*} (x) \| ^ {2}, \tag {C.5}
$$
which is precisely the first inequality in the claim. Furthermore, we can also have
$$
\begin{array}{l} \| \nabla F (x) \lambda - \nabla F (x) \lambda^ {*} (x) \| ^ {2} = \| \nabla F (x) \lambda \| ^ {2} + \| \nabla F (x) \lambda^ {*} (x) \| ^ {2} - 2 \langle \nabla F (x) \lambda^ {*} (x), \nabla F (x) \lambda \rangle \\ \leq \| \nabla F (x) \lambda \| ^ {2} + \| \nabla F (x) \lambda^ {*} (x) \| ^ {2} - 2 \| \nabla F (x) \lambda^ {*} (x) \| ^ {2} \\ = \| \nabla F (x) \lambda \| ^ {2} - \| \nabla F (x) \lambda^ {*} (x) \| ^ {2}, \tag {C.6} \\ \end{array}
$$
which is the desired second inequality in the claim. Hence, the proof is complete.
Lemma 15 (Continuity of  $\lambda_{\rho}^{*}(x)$ ). Given any  $\rho > 0$  and  $x \in \mathbb{R}^d$ , define  $\lambda_{\rho}^{*}(x) = \arg \min_{\lambda \in \Delta^M} \frac{1}{2} \| \nabla F_S(x) \lambda \|^2 + \frac{1}{2} \rho \| \lambda \|^2$ , then the following inequality holds
$$
\left\| \lambda_ {\rho} ^ {*} (x) - \lambda_ {\rho} ^ {*} \left(x ^ {\prime}\right) \right\| \leq \rho^ {- 1} \left\| \nabla F (x) ^ {\top} \nabla F (x) - \nabla F \left(x ^ {\prime}\right) ^ {\top} \nabla F \left(x ^ {\prime}\right) \right\|. \tag {C.7}
$$
Suppose either 1) Assumptions 1, 3 hold, or 2) Assumptions 1, 2 hold, with  $\ell_F$  defined in Lemma 1. Then for  $x\in \{x_{t}\}_{t = 1}^{T}$ ,  $x^{\prime}\in \{x_{t}^{\prime}\}_{t = 1}^{T}$  generated by MoDo algorithm on training dataset  $S$  and  $S^{\prime}$ , respectively, it implies that
$$
\left\| \lambda_ {\rho} ^ {*} (x) - \lambda_ {\rho} ^ {*} \left(x ^ {\prime}\right) \right\| \leq 2 \rho^ {- 1} \ell_ {F, 1} \ell_ {F} \| x - x ^ {\prime} \|. \tag {C.8}
$$
Proof. We provide proof leveraging the convergence properties of the projected gradient descent algorithm on strongly convex objectives below. Consider the problem  $\min_{\lambda \in \Delta^M} g(\lambda; x, \rho) = \frac{1}{2} \| \nabla F_S(x) \lambda \|^2 + \frac{1}{2} \rho \| \lambda \|^2$ , which is  $\rho$ -strongly convex. Let  $\{\lambda_{\rho, k}(x)\}$  for  $k = 0, 1, \ldots, K$  denote the sequence obtained from applying projected gradient descent (PGD) on the objective  $g(\lambda; x, \rho) = \frac{1}{2} \| \nabla F_S(x) \lambda \|^2 + \frac{1}{2} \rho \| \lambda \|^2$ , i.e.,
$$
\begin{array}{l} \lambda_ {\rho , k + 1} (x) = \Pi_ {\Delta^ {M}} \left(\lambda_ {\rho , k} (x) - \eta \nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x) \lambda_ {\rho , k} (x) - \eta \rho \lambda_ {\rho , k} (x)\right) \\ = \Pi_ {\Delta^ {M}} \left(\left((1 - \eta \rho) I - \eta \nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x)\right) \lambda_ {\rho , k} (x)\right) \tag {C.9} \\ \end{array}
$$
where  $\eta$  is the step size that  $\eta < 1 / (\| \nabla F_S(x)^\top \nabla F_S(x)\| +\rho)$ . Note that both  $\rho, \eta$  are independent of  $K$ . By the convergence result of PGD on strongly convex objective, we know that  $\lambda_{\rho}^{*}(x)$  is the limit point of  $\{\lambda_{\rho ,k}(x)\}_{k = 0}^{\infty}$ . By the non-expansiveness of projection, we have
$$
\| \lambda_ {\rho , k + 1} (x) - \lambda_ {\rho , k + 1} \left(x ^ {\prime}\right) \|
$$
$$
\begin{array}{l} \leq \left\| \left((1 - \eta \rho) I - \eta \nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x)\right) \lambda_ {\rho , k} (x) - \left((1 - \eta \rho) I - \eta \nabla F _ {S} \left(x ^ {\prime}\right) ^ {\top} \nabla F _ {S} \left(x ^ {\prime}\right)\right) \lambda_ {\rho , k} \left(x ^ {\prime}\right) \right\| \\ \leq \| (1 - \eta \rho) I - \eta \nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x) \| \| \lambda_ {\rho , k} (x) - \lambda_ {\rho , k} \left(x ^ {\prime}\right) \| \\ + \eta \| \left(\nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x) - \nabla F _ {S} \left(x ^ {\prime}\right) ^ {\top} \nabla F _ {S} \left(x ^ {\prime}\right)\right) \lambda_ {\rho , k} \left(x ^ {\prime}\right) \| \\ \leq (1 - \eta \rho) \| \lambda_ {\rho , k} (x) - \lambda_ {\rho , k} \left(x ^ {\prime}\right) \| + \eta \| \left(\nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x) - \nabla F _ {S} \left(x ^ {\prime}\right) ^ {\top} \nabla F _ {S} \left(x ^ {\prime}\right)\right) \lambda_ {\rho , k} \left(x ^ {\prime}\right) \|. \tag {C.10} \\ \end{array}
$$
Since  $\lambda_{\rho,0}(x) = \lambda_{\rho,0}(x') = \lambda_{\rho,0}$ , apply the above inequality recursively from  $k = 0,1,\ldots,K-1$ , we have
$$
\begin{array}{l} \| \lambda_ {\rho , K} (x) - \lambda_ {\rho , K} \left(x ^ {\prime}\right) \| \leq \eta \| \left(\nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x) - \nabla F _ {S} \left(x ^ {\prime}\right) ^ {\top} \nabla F _ {S} \left(x ^ {\prime}\right)\right) \lambda_ {\rho , k} \left(x ^ {\prime}\right) \| \Big (\sum_ {k = 0} ^ {K - 1} \left(1 - \eta \rho\right) ^ {k} \Big) \\ = \eta \| (\nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x) - \nabla F _ {S} (x ^ {\prime}) ^ {\top} \nabla F _ {S} (x ^ {\prime})) \lambda_ {\rho , k} (x ^ {\prime}) \| \frac {1 - (\eta \rho) ^ {K}}{\eta \rho} \\ \leq \rho^ {- 1} (1 - (\eta \rho) ^ {K}) \| \nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x) - \nabla F _ {S} \left(x ^ {\prime}\right) ^ {\top} \nabla F _ {S} \left(x ^ {\prime}\right) \|. \tag {C.11} \\ \end{array}
$$
$$
\begin{array}{l} \| \lambda_ {\rho} ^ {*} (x) - \lambda_ {\rho} ^ {*} (x ^ {\prime}) \| \leq \lim  _ {K \rightarrow \infty} \left(\| \lambda_ {\rho} ^ {*} (x) - \lambda_ {\rho , K} (x) \| + \| \lambda_ {\rho} ^ {*} (x ^ {\prime}) - \lambda_ {\rho , K} (x ^ {\prime}) \| + \| \lambda_ {\rho , K} (x) - \lambda_ {\rho , K} (x ^ {\prime}) \|\right) \\ \leq \lim  _ {K \rightarrow \infty} \left(\| \lambda_ {\rho} ^ {*} (x) - \lambda_ {\rho , K} (x) \| + \| \lambda_ {\rho} ^ {*} (x ^ {\prime}) - \lambda_ {\rho , K} (x ^ {\prime}) \|\right) \\ + \lim  _ {K \rightarrow \infty} \rho^ {- 1} (1 - (\eta \rho) ^ {K}) \| \nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x) - \nabla F _ {S} \left(x ^ {\prime}\right) ^ {\top} \nabla F _ {S} \left(x ^ {\prime}\right) \| \\ \stackrel {(a)} {\leq} \rho^ {- 1} \| \nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x) - \nabla F _ {S} \left(x ^ {\prime}\right) ^ {\top} \nabla F _ {S} \left(x ^ {\prime}\right) \| + \lim  _ {K \rightarrow \infty} 2 \sqrt {\frac {4}{\rho \eta K}} \\ \leq \rho^ {- 1} \| \nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x) - \nabla F _ {S} \left(x ^ {\prime}\right) ^ {\top} \nabla F _ {S} \left(x ^ {\prime}\right) \| \tag {C.12} \\ \end{array}
$$
where  $(a)$  follows from  $\lim_{K\to \infty}1 - (\eta \rho)^{K} = 1$ , and from the convergence of PGD [2, Theorem 1.1] on  $\rho$ -strongly convex objectives that
$$
\left\| \lambda_ {\rho} ^ {*} (x) - \lambda_ {\rho , K} (x) \right\| ^ {2} \leq \frac {2}{\rho} \Big (g (\lambda_ {\rho , K} (x); x, \rho) - g (\lambda_ {\rho} ^ {*} (x); x, \rho) \Big) \leq \frac {2}{\rho} \frac {\left\| \lambda_ {\rho , 0} (x) - \lambda_ {\rho} ^ {*} (x) \right\| ^ {2}}{2 \eta K} \leq \frac {4}{\rho \eta K}.
$$
This proves (C.7).
In addition, under Assumptions 1, 3, the above result directly implies that