SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \delta_ {T} ^ {2} \right] \leq \left(\frac {3}{n} (1 + \frac {1 2}{\alpha \mu n}) 4 \alpha^ {2} \ell_ {f} ^ {2} + 2 4 M \ell_ {f} ^ {4} c _ {2} c _ {3} ^ {- 1} \frac {\alpha}{n ^ {2}} + 2 4 M \ell_ {f} ^ {4} c _ {2} \frac {\alpha \gamma}{n}\right) (\frac {1}{4} \alpha \mu) ^ {- 1} \\ \leq \frac {4 8}{\mu n} \ell_ {f} ^ {2} \left(\alpha + \frac {1 2}{\mu n} + \frac {2 M \ell_ {f} ^ {2} c _ {2} c _ {3} ^ {- 1}}{n} + 2 M \ell_ {f} ^ {2} c _ {2} \gamma\right) \\ \leq \frac {4 8}{\mu n} \ell_ {f} ^ {2} \left(\alpha + \frac {1 2 + 4 M \ell_ {f} ^ {2}}{\mu n} + \frac {1 0 M \ell_ {f} ^ {4} \gamma}{\mu}\right) \tag {B.79} \\ \end{array}
$$
where the last inequality follows from  $c_{2} = \ell_{F}^{2}(4\mu^{-1} + 2\alpha)\leq 5M\ell_{f}^{2}\mu^{-1}$ , and  $c_{2}c_{3}^{-1}\leq$ $5\ell_F^2\mu^{-1} / (3\ell_F^2)\leq 2\mu^{-1}$ .
B.4.4 Lower bound of MOL uniform stability
In this section, we construct Example 1 with a lower bound of stability for the MoDo algorithm. Before proceeding to the example, we first define  $\mathbf{1}$  as the all-one vector in  $\mathbb{R}^M$ ,  $\widetilde{\Delta}^M \coloneqq \{\lambda \in \mathbb{R}^M \mid \mathbf{1}^\top \lambda = 1\}$ , and  $P_1 \coloneqq I - \frac{1}{M} \mathbf{1} \mathbf{1}^\top$ . Then given any vector  $u \in \mathbb{R}^M$ ,  $\Pi_{\widetilde{\Delta}^M}(u) = P_1 u + \frac{1}{M} \mathbf{1}$ .
Example 1. Recall that  $S = \{z_1, z_2, \ldots, z_j, \ldots, z_n\}$ ,  $S' = \{z_1, z_2, \ldots, z_j', \ldots, z_n\}$ , where  $S$  and  $S'$  differ only in the  $j$ -th data point. Define the  $m$ -th objective function as
$$
f _ {z, m} (x) = \frac {1}{2} x ^ {\top} A x - b _ {m} z ^ {\top} x \tag {B.80}
$$
where  $A$  is a symmetric positive definite matrix,  $\mu = 16n^{-\frac{1}{3}} > 0$  is the smallest eigenvalue of  $A$ , and  $v$  is the corresponding eigenvector of  $A$ . For the datasets  $S$ , and  $S'$ , let  $z_i = c_i v$ , with  $\mathbb{E}_{z \in S}[z] = \mu v$ ,  $\mathbb{E}_{z \in S'}[z] = \mu' v$ ,  $z_j - z_j' = v$ , i.e.,  $\mu - \mu' = \frac{1}{n}$ . For simplicity, let  $M = 2$ ,  $b = [1, 1 + \sqrt{2}]^\top$  such that  $P_1 b = b_P = \left[-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right]^\top$ , where  $b_P$  is the eigenvector of  $P_1$  with eigenvalue 1.
Technical challenges. One challenge of deriving the lower bound lies in the projection operator when updating  $\lambda$ . Unlike deriving the upper bound, where the projection operator can be handled by its non-expansive property, due to the nature of inequality constrained quadratic programming, neither a simple closed-form solution can be obtained, nor a non-trivial tight lower bound can be derived in general. We overcome this challenge by showing that when  $\gamma = \mathcal{O}(T^{-1})$ , the Euclidean projection onto the simplex is equivalent to a linear operator in Lemma 10. Another challenge compared to deriving the lower bound for single-objective learning is that the update of MoDo involves two coupled sequences,  $\{x_{t}\}$  and  $\{\lambda_t\}$ . The update of  $x_{t}$  and  $x_{t}^{\prime}$  involves different weighting parameters  $\lambda_{t}$  and  $\lambda_t^\prime$ , where  $\{x_{t}\}$ ,  $\{\lambda_t\}$  and  $\{x_t^\prime\}$ ,  $\{\lambda_t^\prime\}$  are generated by the MoDo algorithm on neighboring training data  $S$  and  $S^{\prime}$ , respectively. We overcome this challenge by deriving a recursive relation of the vector  $[x_{t} - x_{t}^{\prime};\lambda_{t} - \lambda_{t}^{\prime}]$  in Lemma 11.
Organization of proof. Lemma 10 proves that under proper choice of initialization of  $\lambda$  and step size  $\gamma$ , the projection of the updated  $\lambda$  onto simplex  $\Delta^M$  is equal to the projection of that onto the set  $\widetilde{\Delta}^M \coloneqq \{\lambda \in \mathbb{R}^M \mid \mathbf{1}^\top \lambda = 1\}$ . And thus the projection is equivalent to a linear transformation. Thanks to Lemma 10, we are able to derive a recursive relation the vector  $[x_t - x_t'; \lambda_t - \lambda_t']$  in Lemma 11. Finally, relying on the recursive relation, we derive a lower bound for the recursion of  $[\mathbb{E}_A \| x_t - x_t' \|; \mathbb{E}_A \| \lambda_t - \lambda_t' \|]$ , depending on a  $2 \times 2$  transition matrix. And based on its eigen decomposition, we could compute the  $T$ -th power of such a transition matrix, which is used to derive the final lower bound of  $\mathbb{E}_A \| A(S) - A(S')\|$  in Theorem 6.
Lemma 10. Suppose Assumptions 1, 2 hold. For MoDo algorithm, choose  $\lambda_0 = \frac{1}{M}\mathbf{1}$ ,  $\gamma \leq \frac{1}{2MT\ell_F\ell_f}$  and define
$$
\lambda_ {t} ^ {+} := \lambda_ {t} - \gamma \nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right) \lambda_ {t} \tag {B.81}
$$
then the update of  $\lambda_{t}$  for MoDo algorithm is  $\lambda_{t + 1} = \Pi_{\Delta^M}(\lambda_t^+)$ .
Define the set  $\widetilde{\Delta}^M \coloneqq \{\lambda \in \mathbb{R}^M \mid \mathbf{1}^\top \lambda = 1\}$ ,  $P_1 \coloneqq I - \frac{1}{M} \mathbf{1} \mathbf{1}^\top$ ,  $\lambda_{P,t} \coloneqq \Pi_{\widetilde{\Delta}^M}(\lambda_t^+) = P_1 \lambda_t^+ + \frac{1}{M} \mathbf{1}$ . Then for  $t = 0, \dots, T-1$ , it holds that
$$
\lambda_ {t + 1} = P _ {\mathbf {1}} \left(\lambda_ {t} - \gamma \nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right) \lambda_ {t}\right) + \frac {1}{M} \mathbf {1} = P _ {\mathbf {1}} \lambda_ {t} ^ {+} + \frac {1}{M} \mathbf {1}. \tag {B.82}
$$
Proof. By the update of  $\lambda_{t}$ , we have
$$
\begin{array}{l} \left\| \lambda_ {t + 1} - \lambda_ {t} \right\| = \left\| \Pi_ {\Delta^ {M}} \left(\lambda_ {t} - \gamma \nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right) \lambda_ {t}\right) - \lambda_ {t} \right\| \\ \leq \left\| \lambda_ {t} - \gamma \nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right) \lambda_ {t} - \lambda_ {t} \right\| \\ \leq \gamma \| \nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t}) \lambda_ {t} \| \leq \gamma \ell_ {F} \ell_ {f} \tag {B.83} \\ \end{array}
$$
where the last inequality follows from Lemma 6, with  $\ell_f = \ell_{f,1}(c_x + c_{x^*})$ .
Then for all  $t \in [T - 1]$ , it holds that
$$
\left\| \lambda_ {t} - \lambda_ {0} \right\| = \left\| \sum_ {k = 0} ^ {t - 1} \lambda_ {k + 1} - \lambda_ {k} \right\| \leq \sum_ {k = 0} ^ {t - 1} \left\| \lambda_ {k + 1} - \lambda_ {k} \right\| \leq \gamma t \ell_ {F} \ell_ {f} \leq \frac {t}{2 M T} \tag {B.84}
$$
where the last inequality follows from  $\gamma \leq \frac{1}{2MT\ell_{F}\ell_{f}}$
Then for  $t \in [T - 1]$ , it holds that
$$
\begin{array}{l} \left\| \lambda_ {t} ^ {+} - \lambda_ {0} \right\| \leq \left\| \lambda_ {t} ^ {+} - \lambda_ {t} \right\| + \left\| \lambda_ {t} - \lambda_ {0} \right\| \\ \leq \gamma \| \nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right) \lambda_ {t} \| + \gamma t \ell_ {F} \ell_ {f} \leq \gamma (t + 1) \ell_ {F} \ell_ {f} \leq \gamma T \ell_ {F} \ell_ {f} \leq \frac {1}{2 M}. \tag {B.85} \\ \end{array}
$$
By the update of  $\lambda_{t}$ , and the definition of projection,
$$
\lambda_ {t + 1} = \Pi_ {\Delta^ {M}} \left(\lambda_ {t} ^ {+}\right) = \underset {\lambda \in \Delta^ {M}} {\arg \min } \| \lambda - \lambda_ {t} ^ {+} \| ^ {2}. \tag {B.86}
$$
Also we have
$$
\lambda_ {P, t} = \Pi_ {\widetilde {\Delta} ^ {M}} \left(\lambda_ {t} ^ {+}\right) = \underset {\lambda \in \widetilde {\Delta} ^ {M}} {\arg \min } \| \lambda - \lambda_ {t} ^ {+} \| ^ {2}. \tag {B.87}
$$
Let  $\lambda_{P,t} = [\lambda_{P,t,1},\dots,\lambda_{P,t,M}]^{\top}$ . Then it holds that
$$
\left| \lambda_ {P, t, m} - \lambda_ {0, m} \right| \leq \left\| \lambda_ {P, t} - \lambda_ {0} \right\| = \left\| \Pi_ {\tilde {\Delta} ^ {M}} \left(\lambda_ {t} ^ {+}\right) - \lambda_ {0} \right\| \overset {(a)} {\leq} \left\| \lambda_ {t} ^ {+} - \lambda_ {0} \right\| \overset {(b)} {\leq} \frac {1}{2 M}
$$
where  $(a)$  follows from non-expansiveness of projection and that  $\lambda_0\in \widetilde{\Delta}^M$  ;  $(b)$  follows from (B.85). Therefore, each element of  $\lambda_{P,t}$  satisfies
$$
0 \leq \frac {1}{M} - \frac {1}{2 M} \leq \lambda_ {0, m} - \left| \lambda_ {P, t, m} - \lambda_ {0, m} \right| \leq \lambda_ {P, t, m} \leq \lambda_ {0, m} + \left| \lambda_ {P, t, m} - \lambda_ {0, m} \right| \leq \frac {3}{2 M} \leq 1 \tag {B.88}
$$
which shows that  $\lambda_{P,t} = \Pi_{\widetilde{\Delta}^M}(\lambda_t^+) \in \Delta^M$ . Therefore it holds that,
$$
\left\| \lambda_ {P, t} - \lambda_ {t} ^ {+} \right\| ^ {2} \stackrel {(c)} {\geq} \min  _ {\lambda \in \Delta^ {M}} \| \lambda - \lambda_ {t} ^ {+} \| ^ {2} \stackrel {(d)} {\geq} \min  _ {\lambda \in \tilde {\Delta} ^ {M}} \| \lambda - \lambda_ {t} ^ {+} \| ^ {2} \stackrel {(e)} {=} \| \lambda_ {P, t} - \lambda_ {t} ^ {+} \| ^ {2} \tag {B.89}
$$
where  $(c)$  is because  $\lambda_{P,t} \in \Delta^M$ ;  $(d)$  is because  $\Delta^M \subset \widetilde{\Delta}^M$  by the definition of the simplex;  $(e)$  is because  $\lambda_{P,t} = \Pi_{\widetilde{\Delta}^M}(\lambda_t^+)$ . Then the equality holds that
$$
\left\| \lambda_ {P, t} - \lambda_ {t} ^ {+} \right\| ^ {2} = \min  _ {\lambda \in \Delta^ {M}} \left\| \lambda - \lambda_ {t} ^ {+} \right\| ^ {2} \tag {B.90}
$$
$$
P _ {1} \lambda_ {t} ^ {+} + \frac {1}{M} \mathbf {1} = \lambda_ {P, t} = \arg \min  _ {\lambda \in \Delta^ {M}} \| \lambda - \lambda_ {t} ^ {+} \| ^ {2} = \Pi_ {\lambda \in \Delta^ {M}} \left(\lambda_ {t} ^ {+}\right) = \lambda_ {t + 1}. \tag {B.91}
$$
With the help of Lemma 10, which simplifies the Euclidean projection operator as a linear operator, we then prove in Lemma 11, the recursive relation of  $x_{t} - x_{t}^{\prime}$  and  $\lambda_t - \lambda_t^\prime$ .
Lemma 11. Suppose Assumptions 1, 2 hold. Under Example 1, choose  $\lambda_0 = \frac{1}{M}\mathbf{1}$ ,  $\gamma \leq \frac{1}{2MT\ell_F\ell_f}$  for the MoDo algorithm. Denote  $\{x_{t}\}$ ,  $\{\lambda_t\}$  and  $\{x_t^\prime \}$ ,  $\{\lambda_t^\prime \}$  as the sequences generated by the MoDo algorithm with dataset  $S$  and  $S^{\prime}$ , respectively. Then it holds that
$$
x _ {t} - x _ {t} ^ {\prime} = \varphi_ {x, t} v, \quad a n d \quad \lambda_ {t} - \lambda_ {t} ^ {\prime} = \varphi_ {\lambda , t} b _ {P} \tag {B.92}
$$
and  $\varphi_{x,t}$ ,  $\varphi_{\lambda ,t}$  satisfy the following recursion
$$
\begin{array}{l} \varphi_ {x, t + 1} = (1 - \alpha \mu + \alpha \gamma c _ {t, 3} c _ {t, 1} \mu) \varphi_ {x, t} + \alpha c _ {t, 3} (1 - \gamma c _ {t, 1} c _ {t, 2}) \varphi_ {\lambda , t} + \mathbb {1} (i _ {t, 3} = j) \alpha \left(b ^ {\top} \lambda_ {t + 1} ^ {\prime}\right) \\ + \mathbb {1} \left(i _ {t, 1} = j\right) \gamma \alpha c _ {t, 3} \left(\mu v ^ {\top} x _ {t} ^ {\prime} - c _ {t, 2} b ^ {\top} \lambda_ {t} ^ {\prime}\right) - \mathbb {1} \left(i _ {t, 2} = j\right) \gamma \alpha c _ {t, 3} c _ {t, 1} ^ {\prime} b ^ {\top} \lambda_ {t} ^ {\prime} \tag {B.93} \\ \end{array}
$$
$$
\begin{array}{l} \varphi_ {\lambda , t + 1} = (1 - \gamma c _ {t, 1} c _ {t, 2}) \varphi_ {\lambda , t} + \gamma c _ {t, 1} \mu \varphi_ {x, t} \\ + \mathbb {1} \left(i _ {t, 1} = j\right) \gamma \left(\mu \left(v ^ {\top} x _ {t} ^ {\prime}\right) - c _ {t, 2} \left(b ^ {\top} \lambda_ {t} ^ {\prime}\right)\right) - \mathbb {1} \left(i _ {t, 2} = j\right) \gamma c _ {t, 1} ^ {\prime} \left(b ^ {\top} \lambda_ {t} ^ {\prime}\right) \tag {B.94} \\ \end{array}
$$
where  $\mathbb{1}(\cdot)$  is the indicator function.
Proof. Denote  $z_{t,s}$ , and  $z_{t,s}'$ ,  $s \in [3]$ , as the samples selected in the  $t$ -th iteration from  $S$  and  $S'$ , respectively. According to the MoDo algorithm update of  $x_t$ , and the definition of the problem in (B.80), we have
$$
\begin{array}{l} x _ {t + 1} = x _ {t} - \alpha \nabla F _ {z _ {t, 3}} (x _ {t}) \lambda_ {t + 1} = x _ {t} - \alpha \left[ A x _ {t} - b _ {1} z _ {t, 3}, \dots , A x _ {t} - b _ {M} z _ {t, 3} \right] \lambda_ {t + 1} \\ = x _ {t} - \alpha A x _ {t} + \alpha z _ {t, 3} \left(b ^ {\top} \lambda_ {t + 1}\right). \tag {B.95} \\ \end{array}
$$
The difference  $x_{t + 1} - x_{t + 1}'$  can be computed by
$$
\begin{array}{l} x _ {t + 1} - x _ {t + 1} ^ {\prime} = \big (I - \alpha A \big) \big (x _ {t} - x _ {t} ^ {\prime} \big) + \alpha \big (z _ {t, 3} b ^ {\top} \lambda_ {t + 1} - z _ {t, 3} ^ {\prime} b ^ {\top} \lambda_ {t + 1} ^ {\prime} \big) \\ = (I - \alpha A) \left(x _ {t} - x _ {t} ^ {\prime}\right) + \alpha z _ {t, 3} b ^ {\top} \left(\lambda_ {t + 1} - \lambda_ {t + 1} ^ {\prime}\right) + \alpha \left(z _ {t, 3} - z _ {t, 3} ^ {\prime}\right) b ^ {\top} \lambda_ {t + 1} ^ {\prime} \\ = (I - \alpha A) \left(x _ {t} - x _ {t} ^ {\prime}\right) + \alpha z _ {t, 3} b ^ {\top} \left(\lambda_ {t + 1} - \lambda_ {t + 1} ^ {\prime}\right) + \mathbb {1} \left(i _ {t, 3} = j\right) \alpha v b ^ {\top} \lambda_ {t + 1} ^ {\prime} \tag {B.96} \\ \end{array}
$$
where  $\mathbb{1}(\cdot)$  denotes the indicator function, and the last equation follows from that  $z_{t,3} - z_{t,3}' = 0$  if  $i_{t,3} \neq j$ , and  $z_{t,3} - z_{t,3}' = z_j - z_j' = v$  if  $i_{t,3} = j$ .
By Lemma 10, in Example 1,  $\lambda_{t + 1} = P_1\big(\lambda_t - \gamma \nabla F_{z_{t,1}}(x_t)^\top \nabla F_{z_{t,2}}(x_t)\lambda_t\big) + \frac{1}{M}\mathbf{1}$ , which can be further derived as
$$
\begin{array}{l} \lambda_ {t + 1} = P _ {\mathbf {1}} \left(\lambda_ {t} - \gamma \nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right) \lambda_ {t}\right) + \frac {1}{M} \mathbf {1} \\ = P _ {\mathbf {1}} \left(\lambda_ {t} - \gamma \left(\mathbf {1} x _ {t} ^ {\top} A - b z _ {t, 1} ^ {\top}\right) \left(A x _ {t} - z _ {t, 2} b ^ {\top} \lambda_ {t}\right)\right) + \frac {1}{M} \mathbf {1} \\ \stackrel {(a)} {=} \lambda_ {t} - \gamma \left(P _ {1} \mathbf {1} x _ {t} ^ {\top} A - P _ {1} b z _ {t, 1} ^ {\top}\right) \left(A x _ {t} - z _ {t, 2} b ^ {\top} \lambda_ {t}\right) \\ \stackrel {(b)} {=} \lambda_ {t} + \gamma b _ {P} z _ {t, 1} ^ {\top} \left(A x _ {t} - z _ {t, 2} b ^ {\top} \lambda_ {t}\right) \tag {B.97} \\ \end{array}
$$
where  $(a)$  follows from rearranging the equation and that  $P_{\mathbf{1}}\lambda_{t} + \frac{1}{M}\mathbf{1} = \Pi_{\widetilde{\Delta}^{M}}(\lambda_{t}) = \lambda_{t}$  as  $\lambda_t\in \widetilde{\Delta}^M$  (b) follows from that  $P_{\mathbf{1}}b = b_{P}$  and  $P_{\mathbf{1}}\mathbf{1} = 0$
The difference  $\lambda_{t + 1} - \lambda_{t + 1}^{\prime}$  can be derived as
$$
\lambda_ {t + 1} - \lambda_ {t + 1} ^ {\prime} = \left(\lambda_ {t} - \lambda_ {t} ^ {\prime}\right) + \gamma b _ {P} \left(z _ {t, 1} ^ {\top} A x _ {t} - z _ {t, 1} ^ {\prime \top} A x _ {t} ^ {\prime}\right) - \gamma b _ {P} \left(z _ {t, 1} ^ {\top} z _ {t, 2} b ^ {\top} \lambda_ {t} - z _ {t, 1} ^ {\prime \top} z _ {t, 2} ^ {\prime} b ^ {\top} \lambda_ {t} ^ {\prime}\right)
$$
$$
\begin{array}{l} \stackrel {(c)} {=} \left(\lambda_ {t} - \lambda_ {t} ^ {\prime}\right) + \gamma b _ {P} z _ {t, 1} ^ {\top} A \left(x _ {t} - x _ {t} ^ {\prime}\right) + \gamma b _ {P} \left(z _ {t, 1} - z _ {t, 1} ^ {\prime}\right) ^ {\top} A x _ {t} ^ {\prime} - \gamma b _ {P} z _ {t, 1} ^ {\top} z _ {t, 2} b ^ {\top} \left(\lambda_ {t} - \lambda_ {t} ^ {\prime}\right) \\ - \gamma b _ {P} (z _ {t, 1} - z _ {t, 1} ^ {\prime}) ^ {\top} z _ {t, 2} b ^ {\top} \lambda_ {t} ^ {\prime} - \gamma b _ {P} z _ {t, 1} ^ {\prime \top} (z _ {t, 2} - z _ {t, 2} ^ {\prime}) b ^ {\top} \lambda_ {t} ^ {\prime} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(d)} {=} \left(\lambda_ {t} - \lambda_ {t} ^ {\prime}\right) + \gamma b _ {P} z _ {t, 1} ^ {\top} A \left(x _ {t} - x _ {t} ^ {\prime}\right) - \gamma b _ {P} z _ {t, 1} ^ {\top} z _ {t, 2} b ^ {\top} \left(\lambda_ {t} - \lambda_ {t} ^ {\prime}\right) \\ + \mathbb {1} \left(i _ {t, 1} = j\right) \gamma b _ {P} \left(\mu v ^ {\top} x _ {t} ^ {\prime} - v ^ {\top} z _ {t, 2} b ^ {\top} \lambda_ {t} ^ {\prime}\right) - \mathbb {1} \left(i _ {t, 2} = j\right) \gamma b _ {P} z _ {t, 1} ^ {\prime \top} v b ^ {\top} \lambda_ {t} ^ {\prime} \tag {B.98} \\ \end{array}
$$
where  $(c)$  follows from rearranging the equation;  $(d)$  follows from that  $z_{t,s} - z_{t,s}' = 0$  if  $i_{t,s} \neq j$ , and  $z_{t,s} - z_{t,s}' = z_j - z_j' = v$  if  $i_{t,s} = j$ , and that  $Av = \mu v$ .
Combining (B.96) and (B.98) gives
$$
\begin{array}{l} \left[ \begin{array}{c} x _ {t + 1} - x _ {t + 1} ^ {\prime} \\ \lambda_ {t + 1} - \lambda_ {t + 1} ^ {\prime} \end{array} \right] = \left[ \begin{array}{c c} C _ {x, x, t} & C _ {x, \lambda , t} \\ C _ {\lambda , x, t} & C _ {\lambda , \lambda , t} \end{array} \right] \left[ \begin{array}{c} x _ {t} - x _ {t} ^ {\prime} \\ \lambda_ {t} - \lambda_ {t} ^ {\prime} \end{array} \right] + \mathbb {1} (i _ {t, 3} = j) \alpha \left[ \begin{array}{c} v b ^ {\top} \lambda_ {t + 1} ^ {\prime} \\ 0 \end{array} \right] - \mathbb {1} (i _ {t, 2} = j) \gamma \left[ \begin{array}{c} \alpha c _ {t, 3} c _ {t, 1} ^ {\prime} v b ^ {\top} \lambda_ {t} ^ {\prime} \\ b _ {P} c _ {t, 1} ^ {\prime} b ^ {\top} \lambda_ {t} ^ {\prime} \end{array} \right] \\ + \mathbb {1} \left(i _ {t, 1} = j\right) \gamma \left[ \begin{array}{c} \alpha c _ {t, 3} v \left(\mu v ^ {\top} x _ {t} ^ {\prime} - c _ {t, 2} b ^ {\top} \lambda_ {t} ^ {\prime}\right) \\ b _ {P} \left(\mu v ^ {\top} x _ {t} ^ {\prime} - c _ {t, 2} b ^ {\top} \lambda_ {t} ^ {\prime}\right) \end{array} \right] \tag {B.99} \\ \end{array}
$$
where the matrices are defined as
$$
C _ {x, x, t} = I - \alpha A + \alpha \gamma z _ {t, 3} b ^ {\top} b _ {P} z _ {t, 1} ^ {\top} A = I - \alpha A + \alpha \gamma c _ {t, 3} c _ {t, 1} \mu v v ^ {\top} \tag {B.100a}
$$
$$
C _ {x, \lambda , t} = \alpha z _ {t, 3} b ^ {\top} \left(I - \gamma b _ {P} z _ {t, 1} ^ {\top} z _ {t, 2} b ^ {\top}\right) = \alpha c _ {t, 3} v b ^ {\top} \left(1 - \gamma c _ {t, 1} c _ {t, 2}\right) \tag {B.100b}
$$
$$
C _ {\lambda , x, t} = \gamma b _ {P} z _ {t, 1} ^ {\top} A = \gamma c _ {t, 1} \mu b _ {P} v ^ {\top} \tag {B.100c}
$$
$$
C _ {\lambda , \lambda , t} = \left(I - \gamma b _ {P} z _ {t, 1} ^ {\top} z _ {t, 2} b ^ {\top}\right) = \left(I - \gamma c _ {t, 1} c _ {t, 2} b _ {P} b ^ {\top}\right). \tag {B.100d}
$$
Next we show by induction that
$$
x _ {t} - x _ {t} ^ {\prime} = \varphi_ {x, t} v, \quad \text {a n d} \quad \lambda_ {t} - \lambda_ {t} ^ {\prime} = \varphi_ {\lambda , t} b _ {P}. \tag {B.101}
$$
First, when  $t = 0$ ,  $x_0 - x_0' = 0 = \varphi_{x,0}v$  and  $\lambda_0 - \lambda_0' = \varphi_{\lambda,0}b_P$  with  $\varphi_{x,0} = 0$  and  $\varphi_{\lambda,0} = 0$ . Therefore (B.101) holds at  $t = 0$ . Supposing that (B.101) holds for  $t = k$ , next we show that it also holds at  $t = k + 1$ .
At  $t = k + 1$ , applying (B.99) for  $x_{k + 1} - x_{k + 1}'$ , and substituting (B.100a), (B.100b) yields
$$
\begin{array}{l} x _ {k + 1} - x _ {k + 1} ^ {\prime} = \left[ \begin{array}{c c} C _ {x, x, k} & C _ {x, \lambda , k} \end{array} \right] \left[ \begin{array}{c} x _ {k} - x _ {k} ^ {\prime} \\ \lambda_ {k} - \lambda_ {k} ^ {\prime} \end{array} \right] + \mathbb {1} (i _ {k, 3} = j) \alpha v b ^ {\top} \lambda_ {k + 1} ^ {\prime} \\ + \mathbb {1} (i _ {k, 1} = j) \gamma \alpha c _ {k, 3} v (\mu v ^ {\top} x _ {k} ^ {\prime} - c _ {k, 2} b ^ {\top} \lambda_ {k} ^ {\prime}) - \mathbb {1} (i _ {k, 2} = j) \gamma \alpha c _ {k, 3} c _ {k, 1} ^ {\prime} v b ^ {\top} \lambda_ {k} ^ {\prime} \\ = (I - \alpha A + \alpha \gamma c _ {k, 3} c _ {k, 1} \mu v v ^ {\top}) \varphi_ {x, k} v + \alpha c _ {k, 3} v (1 - \gamma c _ {k, 1} c _ {k, 2}) \varphi_ {\lambda , k} + \mathbb {1} (i _ {k, 3} = j) \alpha v b ^ {\top} \lambda_ {k + 1} ^ {\prime} \\ + \mathbb {1} (i _ {k, 1} = j) \gamma \alpha c _ {k, 3} v \left(\mu v ^ {\top} x _ {k} ^ {\prime} - c _ {k, 2} b ^ {\top} \lambda_ {k} ^ {\prime}\right) - \mathbb {1} (i _ {k, 2} = j) \gamma \alpha c _ {k, 3} c _ {k, 1} ^ {\prime} v b ^ {\top} \lambda_ {k} ^ {\prime} \\ = (1 - \alpha \mu + \alpha \gamma c _ {k, 3} c _ {k, 1} \mu) \varphi_ {x, k} v + \alpha c _ {k, 3} (1 - \gamma c _ {k, 1} c _ {k, 2}) \varphi_ {\lambda , k} v + \mathbb {1} (i _ {k, 3} = j) \alpha \left(b ^ {\top} \lambda_ {k + 1} ^ {\prime}\right) v \\ + \mathbb {1} (i _ {k, 1} = j) \gamma \alpha c _ {k, 3} v \left(\mu v ^ {\top} x _ {k} ^ {\prime} - c _ {k, 2} b ^ {\top} \lambda_ {k} ^ {\prime}\right) - \mathbb {1} (i _ {k, 2} = j) \gamma \alpha c _ {k, 3} c _ {k, 1} ^ {\prime} v b ^ {\top} \lambda_ {k} ^ {\prime} \\ = \varphi_ {x, k + 1} v \tag {B.102} \\ \end{array}
$$
where  $\varphi_{x,k + 1}$  is computed by
$$
\begin{array}{l} \varphi_ {x, k + 1} = (1 - \alpha \mu + \alpha \gamma c _ {k, 3} c _ {k, 1} \mu) \varphi_ {x, k} + \alpha c _ {k, 3} (1 - \gamma c _ {k, 1} c _ {k, 2}) \varphi_ {\lambda , k} + \mathbb {1} (i _ {k, 3} = j) \alpha \left(b ^ {\top} \lambda_ {k + 1} ^ {\prime}\right) \\ + \mathbb {1} \left(i _ {k, 1} = j\right) \gamma \alpha c _ {k, 3} \left(\mu v ^ {\top} x _ {k} ^ {\prime} - c _ {k, 2} b ^ {\top} \lambda_ {k} ^ {\prime}\right) - \mathbb {1} \left(i _ {k, 2} = j\right) \gamma \alpha c _ {k, 3} c _ {k, 1} ^ {\prime} b ^ {\top} \lambda_ {k} ^ {\prime}. \tag {B.103} \\ \end{array}
$$
Therefore, for all  $t \in [T]$ , it holds that  $x_{t} - x_{t}^{\prime} = \varphi_{x,t}v$ .
At  $t = k + 1$ , apply (B.99) for  $\lambda_{k + 1} - \lambda_{k + 1}^{\prime}$ , and substitute (B.100c), (B.100d) yields
$$
\begin{array}{l} \lambda_ {k + 1} - \lambda_ {k + 1} ^ {\prime} = [ C _ {\lambda , x, k} \quad C _ {\lambda , \lambda , k} ] \left[ \begin{array}{c} x _ {k} - x _ {k} ^ {\prime} \\ \lambda_ {k} - \lambda_ {k} ^ {\prime} \end{array} \right] \\ + \mathbb {1} (i _ {k, 1} = j) \gamma (\mu b _ {P} v ^ {\top} x _ {k} ^ {\prime} - c _ {k, 2} b _ {P} b ^ {\top} \lambda_ {k} ^ {\prime}) - \mathbb {1} (i _ {k, 2} = j) \gamma c _ {k, 1} ^ {\prime} b _ {P} b ^ {\top} \lambda_ {k} ^ {\prime} \\ = \gamma c _ {k, 1} \mu b _ {P} v ^ {\top} \varphi_ {x, k} v + (I - \gamma c _ {k, 1} c _ {k, 2} b _ {P} b ^ {\top}) b _ {P} \varphi_ {\lambda , k} \\ + \mathbb {1} (i _ {k, 1} = j) \gamma (\mu b _ {P} v ^ {\top} x _ {k} ^ {\prime} - c _ {k, 2} b _ {P} b ^ {\top} \lambda_ {k} ^ {\prime}) - \mathbb {1} (i _ {k, 2} = j) \gamma c _ {k, 1} ^ {\prime} b _ {P} b ^ {\top} \lambda_ {k} ^ {\prime} \\ = \gamma c _ {t, 1} \mu \varphi_ {x, k} b _ {P} + (1 - \gamma c _ {k, 1} c _ {k, 2}) \varphi_ {\lambda , k} b _ {P} \\ + \mathbb {1} (i _ {k, 1} = j) \gamma \left(\mu \left(v ^ {\top} x _ {k} ^ {\prime}\right) - c _ {k, 2} \left(b ^ {\top} \lambda_ {k} ^ {\prime}\right)\right) b _ {P} - \mathbb {1} (i _ {k, 2} = j) \gamma c _ {k, 1} ^ {\prime} \left(b ^ {\top} \lambda_ {k} ^ {\prime}\right) b _ {P} \\ = \varphi_ {\lambda , k + 1} b _ {P} \tag {B.104} \\ \end{array}
$$
where  $\varphi_{\lambda ,k + 1}$  is computed by
$$
\begin{array}{l} \varphi_ {\lambda , k + 1} = \gamma c _ {k, 1} \mu \varphi_ {x, k} + (1 - \gamma c _ {k, 1} c _ {k, 2}) \varphi_ {\lambda , k} \\ + \mathbb {1} \left(i _ {k, 1} = j\right) \gamma \left(\mu \left(v ^ {\top} x _ {k} ^ {\prime}\right) - c _ {k, 2} \left(b ^ {\top} \lambda_ {k} ^ {\prime}\right)\right) - \mathbb {1} \left(i _ {k, 2} = j\right) \gamma c _ {k, 1} ^ {\prime} \left(b ^ {\top} \lambda_ {k} ^ {\prime}\right). \tag {B.105} \\ \end{array}
$$
Therefore, for all  $t \in [T]$ , it holds that  $\lambda_t - \lambda_t' = \varphi_{\lambda,t}b_P$ .
Lemma 11 provides the recursive relation of  $x_{t} - x_{t}^{\prime}$  and  $\lambda_t - \lambda_t^\prime$ . And Lemma 12 below provides another property used to derive the lower bound of  $\mathbb{E}[\| x_T - x_T^\prime \| ]$  in Theorem 6.
Lemma 12. Suppose Assumptions 1, 2 hold. Under Example 1, choose  $x_0 = x_0' = 7v$ ,  $\alpha = \frac{1}{4\mu T}$  for the MoDo algorithm. Denote  $\{x_t\}$ ,  $\{\lambda_t\}$  and  $\{x_t'\}$ ,  $\{\lambda_t'\}$  as the sequences generated by the MoDo algorithm with dataset  $S$  and  $S'$ , respectively. Then it holds that
$$
v ^ {\top} \mathbb {E} _ {A} \left[ A x _ {t} ^ {\prime} - z _ {t, 2} b ^ {\top} \lambda_ {t} ^ {\prime} \right] \geq 0 \quad a n d \quad b ^ {\top} \mathbb {E} _ {A} \left[ \lambda_ {t + 1} ^ {\prime} \right] \geq b ^ {\top} \mathbb {E} _ {A} \left[ \lambda_ {t} ^ {\prime} \right]. \tag {B.106}
$$