SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \lambda_ {\rho} ^ {*} (x) - \lambda_ {\rho} ^ {*} \left(x ^ {\prime}\right) \right\| \leq \rho^ {- 1} \left\| \nabla F _ {S} (x) ^ {\top} \nabla F _ {S} (x) - \nabla F _ {S} \left(x ^ {\prime}\right) ^ {\top} \nabla F _ {S} \left(x ^ {\prime}\right) \right\| \\ \leq \rho^ {- 1} \| \nabla F _ {S} (x) + \nabla F _ {S} \left(x ^ {\prime}\right) \| \| \nabla F _ {S} (x) - \nabla F _ {S} \left(x ^ {\prime}\right) \| \\ \leq 2 \rho^ {- 1} \ell_ {F, 1} \ell_ {F} \| x - x ^ {\prime} \|. \tag {C.13} \\ \end{array}
$$
While under Assumptions 1 and 2, and for  $\ell_F$  defined in Lemma 1, and for  $x\in \{x_{t}\}_{t = 1}^{T}$ ,  $x^{\prime}\in \{x_{t}^{\prime}\}_{t = 1}^{T}$  generated by MoDo algorithm on training dataset  $S$  and  $S^{\prime}$ , respectively,  $\| \nabla F_S(x)\| \leq \ell_F$ ,  $\| \nabla F_S(x')\| \leq \ell_F$ , which along with (C.12) implies that
$$
\| \lambda_ {\rho} ^ {*} (x) - \lambda_ {\rho} ^ {*} (x ^ {\prime}) \| \leq \rho^ {- 1} \| \nabla F _ {S} (x) + \nabla F _ {S} (x ^ {\prime}) \| \| \nabla F _ {S} (x) - \nabla F _ {S} (x ^ {\prime}) \| \leq 2 \rho^ {- 1} \ell_ {F, 1} \ell_ {F} \| x - x ^ {\prime} \|.
$$
Lemma 16. For any  $\rho >0$  and  $x\in \mathbb{R}^d$ , define  $\lambda^{*}(x) = \arg \min_{\lambda \in \Delta^{M}}\| \nabla F_{S}(x)\lambda \|^{2}$ , and  $\lambda_{\rho}^{*}(x) = \arg \min_{\lambda \in \Delta^{M}}\| \nabla F_{S}(x)\lambda \|^{2} + \rho \| \lambda \|^{2}$ , then we have
$$
0 \leq \| \nabla F _ {S} (x) \lambda_ {\rho} ^ {*} (x) \| ^ {2} - \| \nabla F _ {S} (x) \lambda^ {*} (x) \| ^ {2} \leq \rho \left(1 - \frac {1}{M}\right). \tag {C.14}
$$
Proof. Since  $\lambda^{*}(x) = \arg \min_{\lambda \in \Delta^{M}}\| \nabla F_{S}(x)\lambda \|^{2}$ , therefore
$$
\left\| \nabla F _ {S} (x) \lambda_ {\rho} ^ {*} (x) \right\| ^ {2} - \left\| \nabla F _ {S} (x) \lambda^ {*} (x) \right\| ^ {2} = \left\| \nabla F _ {S} (x) \lambda_ {\rho} ^ {*} (x) \right\| ^ {2} - \min  _ {\lambda \in \Delta^ {M}} \| \nabla F _ {S} (x) \lambda \| ^ {2} \geq 0. \tag {C.15}
$$
Since  $\lambda_{\rho}^{*}(x) = \arg \min_{\lambda \in \Delta^{M}}\| \nabla F_{S}(x)\lambda \|^{2} + \rho \| \lambda \|^{2}$ , therefore
$$
\left\| \nabla F _ {S} (x) \lambda^ {*} (x) \right\| ^ {2} + \rho \left\| \lambda^ {*} (x) \right\| ^ {2} - \left\| \nabla F _ {S} (x) \lambda_ {\rho} ^ {*} (x) \right\| ^ {2} - \rho \left\| \lambda_ {\rho} ^ {*} (x) \right\| ^ {2} \geq 0. \tag {C.16}
$$
Rearranging the above inequality gives
$$
\left\| \nabla F _ {S} (x) \lambda_ {\rho} ^ {*} (x) \right\| ^ {2} - \left\| \nabla F _ {S} (x) \lambda^ {*} (x) \right\| ^ {2} \leq \rho \left\| \lambda^ {*} (x) \right\| ^ {2} - \rho \left\| \lambda_ {\rho} ^ {*} (x) \right\| ^ {2} \leq \rho \left(1 - \frac {1}{M}\right). \tag {C.17}
$$
Lemma 17. Consider  $\{x_{t}\},\{\lambda_{t}\}$  generated by the MoDo algorithm. For all  $\lambda \in \Delta^{M}$ , it holds that
$$
2 \gamma_ {t} \mathbb {E} _ {A} \left\langle \lambda_ {t} - \lambda , \left(\nabla F _ {S} \left(x _ {t}\right) ^ {\top} \nabla F _ {S} \left(x _ {t}\right)\right) \lambda_ {t} \right\rangle
$$
$$
\leq \mathbb {E} _ {A} \| \lambda_ {t} - \lambda \| ^ {2} - \mathbb {E} _ {A} \| \lambda_ {t + 1} - \lambda \| ^ {2} + \gamma_ {t} ^ {2} \mathbb {E} _ {A} \| \left(\nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right)\right) \lambda_ {t} \| ^ {2}, \tag {C.18}
$$
and  $\gamma_t\mathbb{E}_A(\| \nabla F_S(x_t)\lambda_t\| ^2 -\| \nabla F_S(x_t)\lambda \| ^2)$
$$
\leq \mathbb {E} _ {A} \| \lambda_ {t} - \lambda \| ^ {2} - \mathbb {E} _ {A} \| \lambda_ {t + 1} - \lambda \| ^ {2} + \gamma_ {t} ^ {2} \mathbb {E} _ {A} \| \left(\nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right)\right) \lambda_ {t} \| ^ {2}. \tag {C.19}
$$
Proof. By the update of  $\lambda$ , for all  $\lambda \in \Delta^M$ , we have
$$
\begin{array}{l} \left\| \lambda_ {t + 1} - \lambda \right\| ^ {2} \\ = \| \Pi_ {\Delta^ {M}} (\lambda_ {t} - \gamma_ {t} (\nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t})) \lambda_ {t}) - \lambda \| ^ {2} \\ \leq \left\| \lambda_ {t} - \gamma_ {t} \left(\nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right)\right) \lambda_ {t} - \lambda \right\| ^ {2} \\ = \| \lambda_ {t} - \lambda \| ^ {2} - 2 \gamma_ {t} \langle \lambda_ {t} - \lambda , (\nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t})) \lambda_ {t} \rangle + \gamma_ {t} ^ {2} \| (\nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t})) \lambda_ {t} \| ^ {2}. \\ \end{array}
$$
Taking expectation over  $z_{t,1}, z_{t,2}$  on both sides and rearranging proves (C.18).
By the convexity of the problem,  $\min_{\lambda \in \Delta^M}\frac{1}{2}\| \nabla F_S(x_t)\lambda \| ^2$  , we have
$$
\begin{array}{l} \gamma_ {t} \mathbb {E} _ {A} \left(\| \nabla F _ {S} (x _ {t}) \lambda_ {t} \| ^ {2} - \| \nabla F _ {S} (x _ {t}) \lambda \| ^ {2}\right) \\ \leq 2 \gamma_ {t} \mathbb {E} _ {A} \left\langle \lambda_ {t} - \lambda , \left(\nabla F _ {S} \left(x _ {t}\right) ^ {\top} \nabla F _ {S} \left(x _ {t}\right)\right) \lambda_ {t} \right\rangle \\ \leq \mathbb {E} _ {A} \| \lambda_ {t} - \lambda \| ^ {2} - \mathbb {E} _ {A} \| \lambda_ {t + 1} - \lambda \| ^ {2} + \gamma_ {t} ^ {2} \mathbb {E} _ {A} \| (\nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t})) \lambda_ {t} \| ^ {2}. \tag {C.20} \\ \end{array}
$$
Rearranging the above inequality proves (C.19).
C.2 Proof of Lemma 2 - Distance to CA direction
Organization of proof. In Lemma 18, we prove the upper bound of the distance to CA direction,  $\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}_A[\|\nabla F_S(x_t)\lambda_t - \nabla F_S(x_t)\lambda^*(x_t)\|^2]$ , in terms of two average of sequences,  $S_{1,T}$ , and  $S_{2,T}$ . Then under either Assumptions 1, 3, or Assumptions 1, 2, we prove the upper bound of  $S_{1,T}$  and  $S_{2,T}$ , and thus the average-iterate distance to CA direction in Lemma 2.
Lemma 18. Suppose Assumption 1 holds. Let  $\{x_{t}\}, \{\lambda_{t}\}$  be the sequences produced by the MoDo algorithm. With a positive constant  $\bar{\rho} > 0$ , define
$$
S _ {1, T} = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \left(\nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right)\right) \lambda_ {t} \| ^ {2} \tag {C.21a}
$$
$$
S _ {2, T} = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {S} \left(x _ {t + 1}\right) + \nabla F _ {S} \left(x _ {t}\right) \| \| \nabla F _ {z _ {t, 3}} \lambda_ {t + 1} \|. \tag {C.21b}
$$
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \left[ \| \nabla F _ {S} \left(x _ {t}\right) \lambda_ {t} \| ^ {2} - \| \nabla F _ {S} \left(x _ {t}\right) \lambda^ {*} \left(x _ {t}\right) \| ^ {2} \right] \leq \bar {\rho} + \frac {4}{\gamma T} \left(1 + \bar {\rho} ^ {- 1} \alpha \ell_ {F, 1} T S _ {2, T}\right) + \gamma S _ {1, T}. \tag {C.22}
$$
Proof. Define  $\lambda_{\bar{\rho}}^{*}(x_{t}) = \arg \min_{\lambda \in \Delta^{M}}\frac{1}{2}\| \nabla F_{S}(x_{t})\lambda \|^{2} + \frac{\bar{\rho}}{2}\| \lambda \|^{2}$  with  $\bar{\rho} >0$ . Note that  $\bar{\rho}$  is strictly positive and is used only for analysis but not for algorithm update.
Substituting  $\lambda = \lambda_{\hat{\rho}}^{*}(x_{t})$  in Lemma 17, (C.19), we have
$$
\begin{array}{l} \gamma_ {t} \mathbb {E} _ {A} \left(\left\| \nabla F _ {S} (x _ {t}) \lambda_ {t} \right\| ^ {2} - \left\| \nabla F _ {S} (x _ {t}) \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \right\| ^ {2}\right) \\ \leq \mathbb {E} _ {A} \| \lambda_ {t} - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \| ^ {2} - \mathbb {E} _ {A} \| \lambda_ {t + 1} - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \| ^ {2} + \gamma_ {t} ^ {2} \mathbb {E} _ {A} \| (\nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t})) \lambda_ {t} \| ^ {2}. \tag {C.23} \\ \end{array}
$$
Setting  $\gamma_{t} = \gamma > 0$ , taking expectation and telescoping the above inequality gives
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \left[ \| \nabla F _ {S} (x _ {t}) \lambda_ {t} \| ^ {2} - \| \nabla F _ {S} (x _ {t}) \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \| ^ {2} \right] \\ \leq \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \frac {1}{\gamma} \mathbb {E} _ {A} \left[ \| \lambda_ {t} - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \| ^ {2} - \| \lambda_ {t + 1} - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \| ^ {2} \right] + \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \gamma \mathbb {E} _ {A} \| (\nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t})) \lambda_ {t} \| ^ {2} \\ = \frac {1}{\gamma T} \underbrace {\left(\sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \left[ \left\| \lambda_ {t} - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \right\| ^ {2} - \left\| \lambda_ {t + 1} - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \right\| ^ {2} \right]\right)} _ {I _ {1}} + \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \gamma \mathbb {E} _ {A} \| (\nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t})) \lambda_ {t} \| ^ {2} \tag {C.24} \\ \end{array}
$$
where  $I_{1}$  can be further derived as
$$
\begin{array}{l} I _ {1} = \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \lambda_ {t} - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \| ^ {2} - \mathbb {E} _ {A} \| \lambda_ {t + 1} - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \| ^ {2} \\ = \mathbb {E} _ {A} \| \lambda_ {0} - \lambda_ {\bar {\rho}} ^ {*} (x _ {0}) \| ^ {2} - \mathbb {E} _ {A} \| \lambda_ {T} - \lambda_ {\bar {\rho}} ^ {*} (x _ {T}) \| ^ {2} + \sum_ {t = 0} ^ {T - 2} \mathbb {E} _ {A} [ \| \lambda_ {t + 1} - \lambda_ {\bar {\rho}} ^ {*} (x _ {t + 1}) \| ^ {2} - \| \lambda_ {t + 1} - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \| ^ {2} ] \\ \leq \mathbb {E} _ {A} \| \lambda_ {0} - \lambda_ {\bar {\rho}} ^ {*} (x _ {0}) \| ^ {2} - \mathbb {E} _ {A} \| \lambda_ {T} - \lambda_ {\bar {\rho}} ^ {*} (x _ {T}) \| ^ {2} \\ + \sum_ {t = 0} ^ {T - 2} \mathbb {E} _ {A} \left[ \| 2 \lambda_ {t + 1} - \lambda_ {\bar {\rho}} ^ {*} (x _ {t + 1}) - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \| \| \lambda_ {\bar {\rho}} ^ {*} (x _ {t + 1}) - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \| \right] \\ \leq 4 + 4 \sum_ {t = 0} ^ {T - 2} \mathbb {E} _ {A} \left\| \lambda_ {\bar {\rho}} ^ {*} \left(x _ {t + 1}\right) - \lambda_ {\bar {\rho}} ^ {*} \left(x _ {t}\right) \right\| \tag {C.25} \\ \end{array}
$$
where  $\| \lambda_{\bar{\rho}}^{*}(x_{t + 1}) - \lambda_{\bar{\rho}}^{*}(x_{t})\|$ , by Lemma 15, can be bounded by
$$
\begin{array}{l} \left\| \lambda_ {\bar {\rho}} ^ {*} (x _ {t + 1}) - \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \right\| \leq \bar {\rho} ^ {- 1} \left\| \nabla F _ {S} (x _ {t + 1}) + \nabla F _ {S} (x _ {t}) \right\| \left\| \nabla F _ {S} (x _ {t + 1}) - \nabla F _ {S} (x _ {t}) \right\| \\ \leq \bar {\rho} ^ {- 1} \ell_ {F, 1} \| \nabla F _ {S} (x _ {t + 1}) + \nabla F _ {S} (x _ {t}) \| \| x _ {t + 1} - x _ {t} \| \\ \leq \bar {\rho} ^ {- 1} \alpha \ell_ {F, 1} \| \nabla F _ {S} \left(x _ {t + 1}\right) + \nabla F _ {S} \left(x _ {t}\right) \| \| \nabla F _ {z _ {t, 3}} \lambda_ {t + 1} \|. \tag {C.26} \\ \end{array}
$$
$$
\begin{array}{l} I _ {1} \leq 4 + 4 \bar {\rho} ^ {- 1} \alpha \ell_ {F, 1} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {S} (x _ {t + 1}) + \nabla F _ {S} (x _ {t}) \| \| \nabla F _ {z _ {t, 3}} \lambda_ {t + 1} \| \\ = 4 + 4 \bar {\rho} ^ {- 1} \alpha \ell_ {F, 1} T S _ {2, T} \tag {C.27} \\ \end{array}
$$
plugging which into (C.24) gives
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \left[ \| \nabla F _ {S} \left(x _ {t}\right) \lambda_ {t} \| ^ {2} - \| \nabla F _ {S} \left(x _ {t}\right) \lambda_ {\bar {\rho}} ^ {*} \left(x _ {t}\right) \| ^ {2} \right] \leq \frac {4}{\gamma T} \left(1 + \bar {\rho} ^ {- 1} \alpha \ell_ {F, 1} T S _ {2, T}\right) + \gamma S _ {1, T}. \tag {C.28}
$$
Define  $\lambda^{*}(x_{t})\in \arg \min_{\lambda \in \Delta^{M}}\| \nabla F_{S}(x_{t})\lambda \|^{2}$ . Then
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \left[ \| \nabla F _ {S} (x _ {t}) \lambda_ {t} \| ^ {2} - \| \nabla F _ {S} (x _ {t}) \lambda^ {*} (x _ {t}) \| ^ {2} \right]
$$
$$
\begin{array}{l} = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \left[ \left\| \nabla F _ {S} (x _ {t}) \lambda_ {t} \right\| ^ {2} - \left\| \nabla F _ {S} (x _ {t}) \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \right\| ^ {2} + \left\| \nabla F _ {S} (x _ {t}) \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \right\| ^ {2} - \left\| \nabla F _ {S} (x _ {t}) \lambda^ {*} (x _ {t}) \right\| ^ {2} \right] \\ \stackrel {(C. 2 8)} {\leq} \frac {4}{\gamma T} (1 + \bar {\rho} ^ {- 1} \alpha \ell_ {F, 1} T S _ {2, T}) + \gamma S _ {1, T} + \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} [ \| \nabla F _ {S} (x _ {t}) \lambda_ {\bar {\rho}} ^ {*} (x _ {t}) \| ^ {2} - \| \nabla F _ {S} (x _ {t}) \lambda^ {*} (x _ {t}) \| ^ {2} ] \\ \leq \frac {4}{\gamma T} \left(1 + \bar {\rho} ^ {- 1} \alpha \ell_ {F, 1} T S _ {2, T}\right) + \gamma S _ {1, T} + \bar {\rho} \tag {C.29} \\ \end{array}
$$
where the last inequality follows from Lemma 16. The proof is complete.
Proof of Lemma 2. Building on the result in Lemma 18, and by the convexity of the subproblem,  $\min_{\lambda \in \Delta^M}\frac{1}{2}\| \nabla F_S(x_t)\lambda \| ^2$ , and Lemma 14, we have
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \left[ \| \nabla F _ {S} (x _ {t}) \lambda_ {t} - \nabla F _ {S} (x _ {t}) \lambda^ {*} (x _ {t}) \| ^ {2} \right] \leq \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \left[ \| \nabla F _ {S} (x _ {t}) \lambda_ {t} \| ^ {2} - \| \nabla F _ {S} (x _ {t}) \lambda^ {*} (x _ {t}) \| ^ {2} \right] \\ \leq \bar {\rho} + \frac {4}{\gamma T} \left(1 + \bar {\rho} ^ {- 1} \alpha \ell_ {F, 1} T S _ {2, T}\right) + \gamma S _ {1, T}. \tag {C.30} \\ \end{array}
$$
By Assumptions 1, 3 or Assumptions 1, 2 and Lemma 1, we have
$$
S _ {1, T} = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \left(\nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right)\right) \lambda_ {t} \| ^ {2} \leq \left(\ell_ {f} \ell_ {F}\right) ^ {2} \leq M \ell_ {f} ^ {4} \tag {C.31}
$$
$$
S _ {2, T} = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {S} \left(x _ {t + 1}\right) + \nabla F _ {S} \left(x _ {t}\right) \| \| \nabla F _ {z _ {t, 3}} \lambda_ {t + 1} \| \leq 2 \ell_ {f} \ell_ {F}. \tag {C.32}
$$
Substituting  $S_{1,T}$ ,  $S_{2,T}$  in (C.30) with the above bound yields
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \left[ \| \nabla F _ {S} \left(x _ {t}\right) \lambda_ {t} - \nabla F _ {S} \left(x _ {t}\right) \lambda^ {*} \left(x _ {t}\right) \| ^ {2} \right] \leq \bar {\rho} + \frac {4}{\gamma T} \left(1 + 2 \bar {\rho} ^ {- 1} \alpha T \ell_ {F, 1} \ell_ {f} \ell_ {F}\right) + \gamma M \ell_ {f} ^ {4}. \tag {C.33}
$$
Because  $\ell_{F,1}\ell_{F} \leq M\ell_{f,1}\ell_{f}$ , choosing  $\bar{\rho} = 2(\alpha M\ell_{f,1}\ell_{f}^{2} / \gamma)^{\frac{1}{2}}$  yields
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \left[ \| \nabla F _ {S} (x _ {t}) \lambda_ {t} - \nabla F _ {S} (x _ {t}) \lambda^ {*} (x _ {t}) \| ^ {2} \right] \stackrel {(a)} {\leq} \bar {\rho} + \frac {4}{\gamma T} \left(1 + 2 \bar {\rho} ^ {- 1} \alpha T M \ell_ {f, 1} \ell_ {f} ^ {2}\right) + \gamma M \ell_ {f} ^ {4} \\ = \frac {4}{\gamma T} + 6 \sqrt {M \ell_ {f , 1} \ell_ {f} ^ {2} \frac {\alpha}{\gamma}} + \gamma M \ell_ {f} ^ {4} \tag {C.34} \\ \end{array}
$$
where  $(a)$  follows from Lemma 14. This proves the result.
C.3 Proof of Theorem 3 - PS optimization error
Technical contributions. The optimization error bound in Theorem 3 is improved with either relaxed assumption or improved convergence rate compared to prior stochastic MOL algorithms [52, 10, 30] (see Table 2). This is achieved by 1) instead of bounding the approximation error to  $\lambda^{*}(x_{t})$ , we bound that to the CA direction  $d(x_{t}) = -\nabla F_{S}(x_{t})\lambda^{*}(x_{t})$  as a whole, and 2) instead of using the descent lemma of  $F_{S}(x_{t})\lambda^{*}(x_{t})$  with a dynamic weight, we use that of  $F_{S}(x_{t})\lambda$  with a fixed weight (see Lemma 19, (C.36)), thereby improving the tightness of the bound.
Organization of proof. In Lemma 19, we prove the upper bound of the PS optimization error,  $\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}_A\|\nabla F_S(x_t)\lambda_t^*(x_t)\|^2$ , in terms of three average of sequences,  $S_{1,T}, S_{3,T}$ , and  $S_{4,T}$ . Then we prove the upper bound of  $S_{1,T}, S_{3,T}$ , and  $S_{4,T}$ , and thus the PS optimization error either in the nonconvex case under Assumptions 1, 3 or in the strongly convex case under Assumptions 1, 2. Combining the results leads to Theorem 3.
Lemma 19. Suppose Assumption 1 holds. Consider the sequence  $\{x_{t}\}, \{\lambda_{t}\}$  generated by MoDo in unbounded domain for  $x$ . Define
$$
S _ {1, T} = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right) \lambda_ {t} \| ^ {2} \tag {C.35a}
$$
$$
S _ {3, T} = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t}) \lambda_ {t} \| \| \nabla F _ {S} (x _ {t}) ^ {\top} \nabla F _ {S} (x _ {t}) \lambda_ {1} \| \tag {C.35b}
$$
$$
S _ {4, T} = \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {z _ {t, 3}} (x _ {t}) \lambda_ {t + 1} \| ^ {2}. \tag {C.35c}
$$
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {S} (x _ {t}) \lambda_ {t} ^ {*} (x _ {t}) \| ^ {2} \leq \frac {1}{2 \alpha T} \mathbb {E} _ {A} [ F _ {S} (x _ {1}) - F _ {S} (x _ {T + 1}) ] \lambda_ {1} + \frac {1}{2} \gamma S _ {1, T} + \gamma S _ {3, T} + \frac {1}{2} \alpha \ell_ {f, 1} S _ {4, T}.
$$
Proof. By the  $\ell_{f,1}$ -Lipschitz smoothness of  $F_S(x)\lambda$  for all  $\lambda \in \Delta^M$ , we have
$$
\begin{array}{l} F _ {S} (x _ {t + 1}) \lambda - F _ {S} (x _ {t}) \lambda \leq \langle \nabla F _ {S} (x _ {t}) \lambda , x _ {t + 1} - x _ {t} \rangle + \frac {\ell_ {f , 1}}{2} \| x _ {t + 1} - x _ {t} \| ^ {2} \\ = - \alpha_ {t} \left\langle \nabla F _ {S} \left(x _ {t}\right) \lambda , \nabla F _ {z _ {t, 3}} \left(x _ {t}\right) \lambda_ {t + 1} \right\rangle + \frac {\ell_ {f , 1}}{2} \alpha_ {t} ^ {2} \| \nabla F _ {z _ {t, 3}} \left(x _ {t}\right) \lambda_ {t + 1} \| ^ {2}. \tag {C.36} \\ \end{array}
$$
Taking expectation over  $z_{t,3}$  on both sides of the above inequality gives
$$
\mathbb {E} _ {z _ {t, 3}} \left[ F _ {S} \left(x _ {t + 1}\right) \right] \lambda - F _ {S} \left(x _ {t}\right) \lambda \leq - \alpha_ {t} \left\langle \nabla F _ {S} \left(x _ {t}\right) \lambda , \nabla F _ {S} \left(x _ {t}\right) \lambda_ {t + 1} \right\rangle + \frac {\ell_ {f , 1}}{2} \alpha_ {t} ^ {2} \mathbb {E} _ {z _ {t, 3}} \| \nabla F _ {z _ {t, 3}} \left(x _ {t}\right) \lambda_ {t + 1} \| ^ {2}. \tag {C.37}
$$
By Lemma 17, (C.18), we have
$$
\begin{array}{l} 2 \gamma_ {t} \mathbb {E} _ {A} \left\langle \lambda_ {t} - \lambda , \left(\nabla F _ {S} \left(x _ {t}\right) ^ {\top} \nabla F _ {S} \left(x _ {t}\right)\right) \lambda_ {t} \right\rangle \\ \leq \mathbb {E} _ {A} \| \lambda_ {t} - \lambda \| ^ {2} - \mathbb {E} _ {A} \| \lambda_ {t + 1} - \lambda \| ^ {2} + \gamma_ {t} ^ {2} \mathbb {E} _ {A} \| (\nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t})) \lambda_ {t} \| ^ {2}. \tag {C.38} \\ \end{array}
$$
Rearranging the above inequality and letting  $\gamma_t = \gamma > 0$  gives
$$
\begin{array}{l} - \mathbb {E} _ {A} \left\langle \lambda , \nabla F _ {S} (x _ {t}) ^ {\top} \nabla F _ {S} (x _ {t}) \lambda_ {t} \right\rangle \leq - \mathbb {E} _ {A} \left\langle \lambda_ {t}, \left(\nabla F _ {S} (x _ {t}) ^ {\top} \nabla F _ {S} (x _ {t})\right) \lambda_ {t} \right\rangle + \frac {1}{2 \gamma} \mathbb {E} _ {A} \left(\left\| \lambda_ {t} - \lambda \right\| ^ {2} - \left\| \lambda_ {t + 1} - \lambda \right\| ^ {2}\right) \\ + \frac {1}{2} \gamma \mathbb {E} _ {A} \| (\nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t})) \lambda_ {t} \| ^ {2} \\ \leq - \mathbb {E} _ {A} \| \nabla F _ {S} (x _ {t}) \lambda_ {t} \| ^ {2} + \frac {1}{2 \gamma} \mathbb {E} _ {A} (\| \lambda_ {t} - \lambda \| ^ {2} - \| \lambda_ {t + 1} - \lambda \| ^ {2}) \\ + \frac {1}{2} \gamma \mathbb {E} _ {A} \| \left(\nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right)\right) \lambda_ {t} \| ^ {2}. \tag {C.39} \\ \end{array}
$$
Plugging the above inequality into (C.37), and setting  $\alpha_{t} = \alpha >0$ , we have
$$
\begin{array}{l} \mathbb {E} _ {A} [ F _ {S} (x _ {t + 1}) \lambda - F _ {S} (x _ {t}) \lambda ] \leq - \alpha \mathbb {E} _ {A} \bigl \langle \nabla F _ {S} (x _ {t}) \lambda , \nabla F _ {S} (x _ {t}) \lambda_ {t + 1} \bigr \rangle + \frac {\ell_ {f , 1}}{2} \alpha^ {2} \mathbb {E} _ {A} \| \nabla F _ {z _ {t, 3}} (x _ {t}) \lambda_ {t + 1} \| ^ {2} \\ \leq - \alpha \mathbb {E} _ {A} \| \nabla F _ {S} (x _ {t}) \lambda_ {t} \| ^ {2} + \frac {\alpha}{2 \gamma} \mathbb {E} _ {A} [ \| \lambda_ {t} - \lambda \| ^ {2} - \| \lambda_ {t + 1} - \lambda \| ^ {2} ] \\ + \alpha \mathbb {E} _ {A} \langle \nabla F _ {S} (x _ {t}) \lambda , \nabla F _ {S} (x _ {t}) (\lambda_ {t} - \lambda_ {t + 1}) \rangle + \frac {1}{2} \alpha^ {2} \ell_ {f, 1} \mathbb {E} _ {A} \| \nabla F _ {z _ {t, 3}} (x _ {t}) \lambda_ {t + 1} \| ^ {2} \\ + \frac {1}{2} \alpha \gamma \mathbb {E} _ {A} \| \left(\nabla F _ {z _ {t, 1}} \left(x _ {t}\right) ^ {\top} \nabla F _ {z _ {t, 2}} \left(x _ {t}\right)\right) \lambda_ {t} \| ^ {2}. \tag {C.40} \\ \end{array}
$$
Taking telescope sum and rearranging yields, for all  $\lambda \in \Delta^M$ ,
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {S} (x _ {t}) \lambda_ {t} \| ^ {2} \\ \leq \frac {1}{2 \gamma T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} [ \| \lambda_ {t} - \lambda \| ^ {2} - \| \lambda_ {t + 1} - \lambda \| ^ {2} ] + \frac {1}{\alpha T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} [ F _ {S} (x _ {t}) - F _ {S} (x _ {t + 1}) ] \lambda \\ + \frac {1}{2 T} \sum_ {t = 0} ^ {T - 1} \left(\gamma \mathbb {E} _ {A} \| \nabla F _ {z _ {t, 1}} (x _ {t}) ^ {\top} \nabla F _ {z _ {t, 2}} (x _ {t}) \lambda_ {t} \| ^ {2} + \alpha \ell_ {f, 1} \mathbb {E} _ {A} \| \nabla F _ {z _ {t, 3}} (x _ {t}) \lambda_ {t + 1} \| ^ {2} \right. \\ \left. + 2 \mathbb {E} _ {A} \langle \nabla F _ {S} (x _ {t}) \lambda , \nabla F _ {S} (x _ {t}) (\lambda_ {t} - \lambda_ {t + 1}) \rangle\right) \\ \leq \frac {1}{2 \gamma T} \mathbb {E} _ {A} \left[ \| \lambda_ {1} - \lambda \| ^ {2} - \| \lambda_ {T + 1} - \lambda \| ^ {2} \right] + \frac {1}{\alpha T} \mathbb {E} _ {A} \left[ F _ {S} \left(x _ {1}\right) - F _ {S} \left(x _ {T + 1}\right) \right] \lambda + \frac {1}{2} \gamma S _ {1, T} + \gamma S _ {3, T} + \frac {1}{2} \alpha \ell_ {f, 1} S _ {4, T}. \tag {C.41} \\ \end{array}
$$
Setting  $\lambda = \lambda_{1}$  in the above inequality yields
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {S} (x _ {t}) \lambda_ {t} \| ^ {2} \leq \frac {1}{\alpha T} \mathbb {E} _ {A} [ F _ {S} (x _ {1}) - F _ {S} (x _ {T + 1}) ] \lambda_ {1} + \frac {1}{2} \gamma S _ {1, T} + \gamma S _ {3, T} + \frac {1}{2} \alpha \ell_ {f, 1} S _ {4, T}
$$
Finally, the results follow from the definition of  $\lambda_t^*(x_t)$  such that  $\frac{1}{T} \sum_{t=0}^{T-1} \mathbb{E}_A \| \nabla F_S(x_t) \lambda_t^*(x_t) \|^2 \leq \frac{1}{T} \sum_{t=0}^{T-1} \mathbb{E}_A \| \nabla F_S(x_t) \lambda_t \|^2$ .
Proof of Theorem 3. Lemma 19 states that, under Assumption 1, we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {A} \| \nabla F _ {S} (x _ {t}) \lambda_ {t} ^ {*} (x _ {t}) \| ^ {2} \leq \frac {1}{\alpha T} \mathbb {E} _ {A} [ F _ {S} (x _ {1}) - F _ {S} (x _ {T + 1}) ] \lambda_ {1} + \frac {1}{2} \gamma S _ {1, T} + \gamma S _ {3, T} + \frac {1}{2} \alpha \ell_ {f, 1} S _ {4, T}.
$$