SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where for simplicity we denote  $G_{t}^{1} = \frac{1}{B_{1}}\sum_{X_{i}\in \mathcal{B}_{1}^{t}}\partial_{u}f_{i}(u_{i,t},s_{t}^{\prime})\partial_{\mathbf{s}}\psi_{i}(\mathbf{w}_{t},s_{i,t};\mathcal{B}_{2}^{t})$  and  $G_{t}^{2} = \frac{1}{B_{1}}\sum_{X_{i}\in \mathcal{B}_{1}^{t}}\partial_{s^{\prime}}f_{i}(u_{i,t},s_{t}^{\prime})$ . The second inequality in the above derivation uses the bounds of  $\mathbb{E}[\| G_t\|^2],\mathbb{E}[\| G_t^1\|^2]$  and  $\mathbb{E}[\| G_t^2\|^2]$ , which follow from the Lipschitz continuity and bounded variance
assumptions and are denoted by  $M$ . Moreover, we have
$$
\begin{array}{l} \mathbb {E} _ {t} [ \eta \langle \hat {\mathbf {w}} _ {t} - \mathbf {w} _ {t}, G _ {t} \rangle + \eta \langle \hat {\mathbf {s}} _ {t} - \mathbf {s} _ {t}, G _ {t} ^ {1} \rangle + \eta \langle \hat {s} _ {t} ^ {\prime} - s _ {t} ^ {\prime}, G _ {t} ^ {2} \rangle ] \\ = \eta \bigl \langle \hat {\mathbf {w}} _ {t} - \mathbf {w} _ {t}, \mathbb {E} _ {t} [ G _ {t} ] \bigr \rangle + \eta \bigl \langle \hat {\mathbf {s}} _ {t} - \mathbf {s} _ {t}, \mathbb {E} _ {t} [ G _ {t} ^ {1} ] \bigr \rangle + \eta \bigl \langle \hat {s} _ {t} ^ {\prime} - s _ {t} ^ {\prime}, \mathbb {E} _ {t} [ G _ {t} ^ {2} ] \bigr \rangle , \\ \end{array}
$$
$$
\mathbb {E} _ {t} \left[ G _ {t} \right] = \frac {1}{n _ {+}} \sum_ {X _ {i} \in \mathcal {S} _ {+}} \partial_ {u} f _ {i} \left(u _ {i, t}, s _ {t} ^ {\prime}\right) \partial_ {w} \psi_ {i} \left(\mathbf {w} _ {t}, s _ {i, t}\right)
$$
$$
\mathbb {E} _ {t} \left[ G _ {t} ^ {1} \right] = \frac {1}{n _ {+}} \sum_ {X _ {i} \in \mathcal {S} _ {+}} \partial_ {u} f _ {i} \left(u _ {i, t}, s _ {t} ^ {\prime}\right) \partial_ {\mathbf {s}} \psi_ {i} \left(\mathbf {w} _ {t}, s _ {i, t}\right)
$$
$$
\mathbb {E} _ {t} [ G _ {t} ^ {2} ] = \frac {1}{n _ {+}} \sum_ {X _ {i} \in \mathcal {S} _ {+}} \partial_ {s ^ {\prime}} f _ {i} (u _ {i, t}, s _ {t} ^ {\prime}).
$$
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t + 1}, \mathbf {s} _ {t + 1}, s _ {t + 1} ^ {\prime}\right) \right] \\ \leq F _ {1 / \bar {\rho}} (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) + \frac {3 \eta^ {2} \bar {\rho} M ^ {2}}{2} + \frac {\bar {\rho} \eta}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \left[ f _ {i} (\psi_ {i} (\hat {\mathbf {w}} _ {t}, \hat {s} _ {i, t}), \hat {s} _ {t} ^ {\prime}) - f _ {i} (u _ {i, t}, s _ {t} ^ {\prime}) \right] \\ \left. - \partial_ {u} f _ {i} (u _ {i, t}, s _ {t} ^ {\prime}) \big [ \psi_ {i} (\mathbf {w} _ {t}, s _ {i, t}) - u _ {i, t} \big ] + \frac {\rho_ {\psi} C _ {f}}{2} \left(\| \hat {\mathbf {w}} _ {t} - \mathbf {w} _ {t} \| ^ {2} + \| \hat {s} _ {i, t} - s _ {i, t} \| ^ {2}\right) \right] \\ \leq F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right) + \frac {3 \eta^ {2} \bar {\rho} M ^ {2}}{2} + \bar {\rho} \eta \left(F \left(\hat {\mathbf {w}} _ {t}, \hat {\mathbf {s}} _ {t}, \hat {s} _ {t} ^ {\prime}\right) - F \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right)\right) \tag {31} \\ + \frac {\bar {\rho} \eta}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \left[ f _ {i} (\psi_ {i} (\mathbf {w} _ {t}, s _ {i, t}), s _ {t} ^ {\prime}) - f _ {i} (u _ {i, t}, s _ {t} ^ {\prime}) - \partial_ {u} f _ {i} (u _ {i, t}, s _ {t} ^ {\prime}) [ \psi_ {i} (\mathbf {w} _ {t}, s _ {i, t}) - u _ {i, t} ] \right. \\ \left. + \frac {\rho_ {\psi} C _ {f}}{2} \left(\| \hat {\mathbf {w}} _ {t} - \mathbf {w} _ {t} \| ^ {2} + \| \hat {s} _ {i, t} - s _ {i, t} \| ^ {2}\right) \right] \\ \end{array}
$$
Due to the  $\rho_{F}$ -weak convexity of  $F(\mathbf{w},\mathbf{s},s^{\prime})$ , we have  $(\bar{\rho} -\rho_F)$ -strong convexity of  $(\mathbf{w},\mathbf{s},s^{\prime})\mapsto F(\mathbf{w},\mathbf{s},s^{\prime}) + \frac{\bar{\rho}}{2}\| (\mathbf{w}_t,\mathbf{s}_t,s_t^{\prime}) - (\mathbf{w},\mathbf{s},s^{\prime})\| ^2$ . Then it follows
$$
\begin{array}{l} F (\hat {\mathbf {w}} _ {t}, \hat {\mathbf {s}} _ {t}, \hat {s} _ {t} ^ {\prime}) - F (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) = \left[ F (\hat {\mathbf {w}} _ {t}, \hat {\mathbf {s}} _ {t}, \hat {s} _ {t} ^ {\prime}) + \frac {\bar {\rho}}{2} \| (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) - (\hat {\mathbf {w}} _ {t}, \hat {\mathbf {s}} _ {t}, \hat {s} _ {t} ^ {\prime}) \| ^ {2} \right] \\ - \left[ F \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right) + \frac {\bar {\rho}}{2} \left\| \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right) - \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right) \right\| ^ {2} \right] \tag {32} \\ - \frac {\bar {\rho}}{2} \left\| \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, \hat {s} _ {t} ^ {\prime}\right) - \left(\hat {\mathbf {w}} _ {t}, \hat {\mathbf {s}} _ {t}, \hat {s} _ {t} ^ {\prime}\right) \right\| ^ {2} \\ \leq \left(\frac {\rho_ {F}}{2} - \bar {\rho}\right) \| (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) - (\hat {\mathbf {w}} _ {t}, \hat {\mathbf {s}} _ {t}, \hat {s} _ {t} ^ {\prime}) \| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t + 1}, \mathbf {s} _ {t + 1}, s _ {t + 1} ^ {\prime}\right) \right] \\ \leq \mathbb {E} [ F _ {1 / \bar {\rho}} (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) ] + \frac {3 \eta^ {2} \bar {\rho} M ^ {2}}{2} + \bar {\rho} \eta (\frac {\rho_ {F}}{2} - \bar {\rho}) \| (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) - (\hat {\mathbf {w}} _ {t}, \hat {\mathbf {s}} _ {t}, \hat {s} _ {t} ^ {\prime}) \| ^ {2} \\ + \frac {\bar {\rho} \eta}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \left[ f _ {i} \left(\psi_ {i} \left(\mathbf {w} _ {t}, s _ {i, t}\right), s _ {t} ^ {\prime}\right) - f _ {i} \left(u _ {i, t}, s _ {t} ^ {\prime}\right) - \partial_ {u} f _ {i} \left(u _ {i, t}, s _ {t} ^ {\prime}\right) \left[ \psi_ {i} \left(\mathbf {w} _ {t}, s _ {i, t}\right) - u _ {i, t} \right] \right. \tag {33} \\ \left. + \frac {\rho_ {\psi} C _ {f}}{2} \left(\| \hat {\mathbf {w}} _ {t} - \mathbf {w} _ {t} \| ^ {2} + \| \hat {s} _ {i, t} - s _ {i, t} \| ^ {2}\right) \right] \\ \end{array}
$$
Set  $\bar{\rho} = \rho_F + \rho_\psi C_f$ . We have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t + 1}, \mathbf {s} _ {t + 1}, s _ {t + 1} ^ {\prime}\right) \right] \\ \leq F _ {1 / \bar {\rho}} (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) + \frac {3 \eta^ {2} \bar {\rho} M ^ {2}}{2} - \frac {\bar {\rho} ^ {2} \eta}{2} \| (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) - (\hat {\mathbf {w}} _ {t}, \hat {\mathbf {s}} _ {t}, \hat {s} _ {t} ^ {\prime}) \| ^ {2} \\ + \frac {\bar {\rho} \eta}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \left[ f _ {i} \left(\psi_ {i} \left(\mathbf {w} _ {t}, s _ {i, t}\right), s _ {t} ^ {\prime}\right) - f _ {i} \left(u _ {i, t}, s _ {t} ^ {\prime}\right) - \partial_ {u} f _ {i} \left(u _ {i, t}, s _ {t} ^ {\prime}\right) \left[ \psi_ {i} \left(\mathbf {w} _ {t}, s _ {i, t}\right) - u _ {i, t} \right] \right] \\ \stackrel {(a)} {\leq} F _ {1 / \bar {\rho}} (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) + \frac {3 \eta^ {2} \bar {\rho} M ^ {2}}{2} - \frac {\eta}{2} \| \nabla \varphi_ {1 / \bar {\rho}} (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) \| ^ {2} \\ + \frac {\bar {\rho} \eta}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \left[ f _ {i} \left(\psi_ {i} \left(\mathbf {w} _ {t}, s _ {i, t}\right), s _ {t} ^ {\prime}\right) - f _ {i} \left(u _ {i, t}, s _ {t} ^ {\prime}\right) - \partial_ {u} f _ {i} \left(u _ {i, t}, s _ {t} ^ {\prime}\right) \left[ \psi_ {i} \left(\mathbf {w} _ {t}, s _ {i, t}\right) - u _ {i, t} \right] \right] \\ \end{array}
$$
Using the Lipschitz continuity of  $f$ , we have
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t + 1}, \mathbf {s} _ {t + 1}, s _ {t + 1} ^ {\prime}\right) \right] \\ \leq F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right) + \frac {3 \eta^ {2} \bar {\rho} M ^ {2}}{2} - \frac {\eta}{2} \| \nabla F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right) \| ^ {2} \tag {34} \\ + \frac {\bar {\rho} \eta}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} 2 C _ {f} \| \psi_ {i} (\mathbf {w} _ {t}, s _ {i, t}) - u _ {i, t} \| \\ \end{array}
$$
With the error bound from Lemma C.3, we have
$$
\mathbb {E} \left[ \frac {1}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \| \psi_ {i} (\mathbf {w} _ {t}, s _ {i, t}) - u _ {i, t} \| \right] \leq (1 - \mu) ^ {t} \frac {1}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \| \psi_ {i} (\mathbf {w} _ {0}, s _ {i, 0}) - u _ {i, 0} \| + R
$$
with  $\mu = \frac{B_1\tau}{2n_+}$ ,  $R = \frac{2\tau^{1/2}\sigma}{B_2^{1/2}} + \frac{4n_+C_\psi M\eta}{B_1\tau^{1/2}} + \frac{4n_+^{1/2}C_\psi M\eta}{B_1\tau^{1/2}}$ . Then
$$
\begin{array}{l} \mathbb {E} \left[ F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t + 1}, \mathbf {s} _ {t + 1}, s _ {t + 1} ^ {\prime}\right) \right] \\ \leq F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right) + \frac {3 \eta^ {2} \bar {\rho} M ^ {2}}{2} - \frac {\eta}{2} \mathbb {E} \left[ \| \nabla F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right) \| ^ {2} \right] \tag {35} \\ + 2 C _ {f} \bar {\rho} \eta \left((1 - \mu) ^ {t} \frac {1}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \| \psi_ {i} (\mathbf {w} _ {0}, s _ {i, 0}) - u _ {i, 0} \| + R\right) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ F _ {1 / \bar {\rho}} (\mathbf {w} _ {T}, \mathbf {s} _ {T}, s _ {T} ^ {\prime}) ] \\ \leq F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {0}, \mathbf {s} _ {0}, s _ {0} ^ {\prime}\right) + \frac {3 \eta^ {2} \bar {\rho} M ^ {2} T}{2} - \frac {\eta}{2} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \| \nabla F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right) \| ^ {2} \right] \\ + 2 C _ {f} \bar {\rho} \eta \left(\sum_ {t = 0} ^ {T - 1} (1 - \mu) ^ {t} \frac {1}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \| \psi_ {i} \left(\mathbf {w} _ {0}, s _ {i, 0}\right) - u _ {i, 0} \| + R T\right) \tag {36} \\ \stackrel {(a)} {\leq} F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {0}, \mathbf {s} _ {0}, s _ {0} ^ {\prime}\right) + \frac {3 \eta^ {2} \bar {\rho} M ^ {2} T}{2} - \frac {\eta}{2} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| \nabla F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right) \| ^ {2} ] \\ + \frac {4 C _ {f} \bar {\rho} \eta}{\mu} \sum_ {X _ {i} \in S _ {+}} \frac {1}{n _ {+}} \| \psi_ {i} (\mathbf {w} _ {0}, s _ {i, 0}) - u _ {i, 0} \| + 2 C _ {f} \bar {\rho} \eta R T \\ \end{array}
$$
Lower bounding the left-hand-side by  $\min_{\mathbf{w},\mathbf{s},s'}F_{1 / \hat{\rho}}(\mathbf{w},\mathbf{s},s')$ , we obtain
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| \nabla F _ {1 / \bar {\rho}} (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) \| ^ {2} ] \\ \leq \frac {2}{\eta T} \left[ F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {0}, \mathbf {s} _ {0}, s _ {0} ^ {\prime}\right) - \min  _ {\mathbf {w}, \mathbf {s}, s ^ {\prime}} F _ {1 / \bar {\rho}} \left(\mathbf {w}, \mathbf {s}, s ^ {\prime}\right) + \frac {3 \eta^ {2} \bar {\rho} M ^ {2} T}{2} \right. \\ \left. + \frac {4 C _ {f} \bar {\rho} \eta}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \| \psi_ {i} (\mathbf {w} _ {0}, s _ {i, 0}) - u _ {i, 0} \| + 2 C _ {f} \bar {\rho} \eta R T \right] \\ \leq \frac {2 \Delta}{\eta T} + 3 \eta \bar {\rho} M ^ {2} + \frac {8 C _ {f} \bar {\rho}}{\mu T n _ {+}} \sum_ {X _ {i} \in S _ {+}} \| \psi_ {i} (\mathbf {w} _ {0}, s _ {i, 0}) - u _ {i, 0} \| + 4 C _ {f} \bar {\rho} R \\ \leq \frac {C}{T} \left(\frac {1}{\eta} + \frac {1}{\mu}\right) + C (\eta + R) \\ \end{array}
$$
where we assume  $F_{1 / \bar{\rho}}(\mathbf{w}_0,\mathbf{s}_0,s_0^{\prime}) - \min_{\mathbf{w},\mathbf{s},s^{\prime}}F_{1 / \bar{\rho}}(\mathbf{w},\mathbf{s},s^{\prime})\leq \Delta$  and
$$
C = \max  \{8 \Delta , 1 2 \bar {\rho} M ^ {2}, 3 2 C _ {f} \bar {\rho} \sum_ {X _ {i} \in S _ {+}} \| \psi_ {i} (\mathbf {w} _ {0}, s _ {i, 0}) - u _ {i, 0} \|, 1 6 C _ {f} \bar {\rho} \}.
$$
Plugging the expression of  $\mu$  and  $R$  yields
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| \nabla F _ {1 / \bar {\rho}} (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) \| ^ {2} ] \\ \leq \mathcal {O} \left(\frac {1}{T} \left(\frac {1}{\eta} + \frac {n _ {+}}{B _ {1} \tau}\right) + \left(\frac {\tau^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {n _ {+} \eta}{B _ {1} \tau^ {1 / 2}}\right)\right) \\ \end{array}
$$
Setting  $\tau = \mathcal{O}(B_2\epsilon^4)$  and  $\eta = \mathcal{O}\left(\frac{B_1B_2^{1 / 2}}{n_+}\epsilon^4\right)$ , with  $T = \mathcal{O}\left(\frac{n_{+}}{B_{1}B_{2}^{1 / 2}}\epsilon^{-6}\right)$  iterations, we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} [ \| \nabla F _ {1 / \bar {\rho}} (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) \| ^ {2} ] \leq \epsilon^ {2}.
$$
C.4.2 Convergence analysis for Algorithm 6
We now consider MIL TPAUC maximization with mean pooling. Define  $F(\mathbf{w}, \mathbf{s}, s') := \frac{1}{n_+} \sum_{X_i \in S_+} f_i(g_i(h_j(\mathbf{w}) - h_i(\mathbf{w}), s_i), s')$ . Due to the weak-convexity of  $F(\mathbf{w}, \mathbf{s}, s')$  w.r.t.  $(\mathbf{w}, \mathbf{s}, s')$ , we consider the following Moreau envelope and proximal map defined as
$$
\operatorname {p r o x} _ {\lambda F} (\mathbf {w}, \mathbf {s}, s ^ {\prime}) = \underset {\tilde {\mathbf {w}}, \tilde {\mathbf {s}}, \tilde {s} ^ {\prime}} {\arg \min} F (\tilde {\mathbf {w}}, \tilde {\mathbf {s}}, \tilde {s} ^ {\prime}) + \frac {1}{2 \lambda} \left(\| \tilde {\mathbf {w}} - \mathbf {w} \| ^ {2} + \| \tilde {\mathbf {s}} - \mathbf {s} \| ^ {2} + \| \tilde {s} ^ {\prime} - s ^ {\prime} \| ^ {2}\right).
$$
Following the same proofs of Lemma A.3 and Lemma A.4, we have the following error bounds
Lemma C.5. Consider the update for  $\{v_{i,t}:X_i\in S_+\cup S_-\}$  in Algorithm 6. Assume  $h_i(\mathbf{w};\xi)$  is  $C_h$ -Lipshtz for all  $X_{i}\in S_{+}\cup S_{-}$ , and  $\mathbb{E}[\| G_t\|^2 ]\leq M^2$ . With  $\gamma_{1} = \frac{n_{+} - B_{1}}{B_{1}(1 - \tau_{1})} + (1 - \tau_{1})$
$$
\gamma_ {2} = \frac {n _ {-} - B _ {2}}{B _ {2} (1 - \tau_ {1})} + (1 - \tau_ {1}) a n d \tau_ {1} \leq \frac {1}{2}, w e h a v e
$$
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n _ {+}} \sum_ {X _ {i} \in \mathcal {S} _ {+}} \| v _ {i, t + 1} - h _ {i} (\mathbf {w} _ {t + 1}) \| \right] \leq (1 - \frac {B _ {1} \tau_ {1}}{2 n _ {+}}) ^ {t + 1} \sum_ {X _ {i} \in \mathcal {S} _ {+}} \| v _ {i, 0} - h _ {i} (\mathbf {w} _ {t}) \| + 2 \tau_ {1} ^ {1 / 2} \sigma + \frac {4 n _ {+} C _ {h} M \eta}{B _ {1} \tau_ {1} ^ {1 / 2}} \\ \mathbb {E} \left[ \frac {1}{n _ {-}} \sum_ {X _ {j} \in \mathcal {S} _ {-}} \| v _ {j, t + 1} - h _ {j} (\mathbf {w} _ {t + 1}) \| \right] \leq (1 - \frac {B _ {1} \tau_ {1}}{2 n _ {-}}) ^ {t + 1} \frac {1}{n _ {-}} \sum_ {X _ {j} \in \mathcal {S} _ {-}} \| v _ {j, 0} - h _ {j} (\mathbf {w} _ {t}) \| + 2 \tau_ {1} ^ {1 / 2} \sigma + \frac {4 n _ {-} C _ {h} M \eta}{B _ {1} \tau_ {1} ^ {1 / 2}} \\ \mathbb {E} \left[ \frac {1}{n _ {+}} \sum_ {X _ {i} \in \mathcal {S} _ {+}} \| v _ {i, t + 1} - h _ {i} (\mathbf {w} _ {t + 1}) \| ^ {2} \right] \leq (1 - \frac {B _ {1} \tau_ {1}}{2 n _ {+}}) ^ {2 (t + 1)} \frac {1}{n _ {+}} \sum_ {X _ {i} \in \mathcal {S} _ {+}} \| v _ {i, 0} - h _ {i} (\mathbf {w} _ {t}) \| ^ {2} + 4 \tau_ {1} \sigma^ {2} + \frac {1 6 n _ {+} ^ {2} C _ {h} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} \tau_ {1}} \\ \mathbb {E} \bigg [ \frac {1}{n _ {-}} \sum_ {X _ {j} \in \mathcal {S} _ {-}} \| v _ {j, t + 1} - h _ {j} (\mathbf {w} _ {t + 1}) \| ^ {2} \bigg ] \leq (1 - \frac {B _ {1} \tau_ {1}}{2 n _ {-}}) ^ {2 (t + 1)} \frac {1}{n _ {-}} \sum_ {X _ {j} \in \mathcal {S} _ {-}} \| v _ {j, 0} - h _ {j} (\mathbf {w} _ {t}) \| ^ {2} + 4 \tau_ {1} \sigma^ {2} + \frac {1 6 n _ {-} ^ {2} C _ {h} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} \tau_ {1}} \\ \end{array}
$$
Lemma C.6. Consider update for  $\{u_{i,t}:X_i\in S_+\}$  in Algorithm 6. Assume  $g_{i}(v_{ij},s_{i})$  is  $C_g$ -Lipschitz w.r.t.  $(v_{ij},s_i)$  for all  $X_{i}\in S_{+}$  and  $X_{j}\in S_{-}$ . With  $\gamma_{3} = \frac{n_{+} - B_{1}}{B_{1}(1 - \tau_{2})} + (1 - \tau_{2})$  and  $\tau_{2}\leq \frac{1}{2}$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \| u _ {i, t + 1} - \frac {1}{n _ {-}} \sum_ {X _ {j} \in S _ {-}} g _ {i} \left(v _ {j, t + 1} - v _ {i, t + 1}, s _ {i, t + 1}\right) \| \right] \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{2 n _ {+}}\right) ^ {t + 1} \frac {1}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \| u _ {i, 0} - \frac {1}{n _ {-}} \sum_ {X _ {j} \in S _ {-}} g _ {i} \left(v _ {j, 0} - v _ {i, 0}, s _ {i, 0}\right) \| + 2 \tau_ {2} ^ {1 / 2} \sigma \\ + C _ {2} \frac {n _ {+}}{B _ {1}} \left(\frac {B _ {1} ^ {1 / 2}}{n _ {+} ^ {1 / 2}} + \frac {B _ {2} ^ {1 / 2}}{n _ {-} ^ {1 / 2}}\right) \frac {\tau_ {1}}{\tau_ {2} ^ {1 / 2}} + C _ {2} \frac {n _ {+}}{B _ {1}} \left(\frac {n _ {+} ^ {1 / 2}}{B _ {1} ^ {1 / 2}} + \frac {n _ {-} ^ {1 / 2}}{B _ {2} ^ {1 / 2}}\right) \frac {\eta}{\tau_ {2} ^ {1 / 2}} + C _ {2} \frac {n _ {+} ^ {1 / 2} \eta}{B _ {1} \tau_ {2} ^ {1 / 2}} \\ \end{array}
$$
Then we have the following coverage guarantee.
Theorem C.7. Under assumptions given in Lemma C.2, with  $\gamma_{1} = \frac{n_{1} - B_{1}}{B_{1}(1 - \tau_{1})}$
$$
\begin{array}{l} (1 - \tau_ {1}), \gamma_ {2} = \frac {n _ {2} - B _ {2}}{B _ {2} (1 - \tau_ {1})} + (1 - \tau_ {1}), \gamma_ {3} = \frac {n _ {1} - B _ {1}}{B _ {1} (1 - \tau_ {2})} + (1 - \tau_ {2}), \tau_ {1} = \\ \mathcal {O} \left(\min  \left\{B _ {3}, \frac {B _ {1}}{n _ {+}} \min  \left\{\frac {n _ {+} ^ {1 / 2}}{B _ {1} ^ {1 / 2}}, \frac {n _ {-} ^ {1 / 2}}{B _ {2} ^ {1 / 2}} \right\} B _ {2} ^ {1 / 2} \right\} \epsilon^ {4}\right) \leq 1 / 2, \tau_ {2} = \mathcal {O} (B _ {2} \epsilon^ {4}) \leq 1 / 2, \eta = \\ \mathcal{O}\left(\min \left\{\min \{\frac{B_{1}}{n_{+}},\frac{B_{2}}{n_{-}}\} \min \{B_{3}^{1 / 2},\frac{B_{1}^{1 / 2}}{n_{+}^{1 / 2}}\min \{\frac{n_{+}^{1 / 4}}{B_{1}^{1 / 4}},\frac{n_{-}^{1 / 4}}{B_{2}^{1 / 4}}\} B_{2}^{1 / 4}\} ,\frac{B_{1}}{n_{+}}\min \{\frac{B_{1}^{1 / 2}}{n_{+}^{1 / 2}},\frac{B_{2}^{1 / 2}}{n_{-}^{1 / 2}}\} B_{3}^{1 / 2}\right\} \epsilon^{4}\right), \\ \end{array}
$$
then after
$$
T \geq \mathcal {O} \left(\max  \left\{\max  \left\{\frac {n _ {+}}{B _ {1}}, \frac {n _ {-}}{B _ {2}} \right\} \max  \left\{\frac {1}{B _ {3} ^ {1 / 2}}, \frac {n _ {+} ^ {1 / 2}}{B _ {1} ^ {1 / 2}} \max  \left\{\frac {B _ {1} ^ {1 / 4}}{n _ {+} ^ {1 / 4}}, \frac {B _ {2} ^ {1 / 4}}{n _ {-} ^ {1 / 4}} \right\} \frac {1}{B _ {2} ^ {1 / 4}} \right\}, \frac {n _ {+}}{B _ {1}} \max  \left\{\frac {n _ {+} ^ {1 / 2}}{B _ {1} ^ {1 / 2}}, \frac {n _ {-} ^ {1 / 2}}{B _ {2} ^ {1 / 2}} \right\} \frac {1}{B _ {2} ^ {1 / 2}} \right\} \epsilon^ {- 6}\right)
$$
iterations, Algorithm 6 gives  $\epsilon$ -stationary point to the Moreau envelope, i.e.,
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla F _ {1 / \bar {\rho}} (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) \| ^ {2} \leq \epsilon^ {2}.
$$
where  $\bar{\rho} = \rho_{F} + \rho_{g}C_{f} + 8\rho_{g}C_{f}C_{h} + C_{f}C_{g}L_{h}$
Proof of Theorem C.7. Consider the change in the Moreau envelope:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t + 1}, \mathbf {s} _ {t + 1}, s _ {t + 1} ^ {\prime}\right) \right] \\ = \mathbb {E} _ {t} \left[ \min _ {\tilde {\mathbf {w}}, \tilde {\mathbf {s}}, \tilde {s} ^ {\prime}} F (\tilde {\mathbf {w}}, \tilde {\mathbf {s}} _ {t}, \tilde {s} _ {t} ^ {\prime}) + \frac {\bar {\rho}}{2} \left(\| \tilde {\mathbf {w}} - \mathbf {w} _ {t + 1} \| ^ {2} + \| \tilde {\mathbf {s}} - \mathbf {s} _ {t + 1} \| ^ {2} + \| \tilde {s} ^ {\prime} - \mathbf {s} _ {t + 1} ^ {\prime} \| ^ {2}\right) \right] \\ \leq \mathbb {E} _ {t} \left[ F (\hat {\mathbf {w}} _ {t}, \hat {\mathbf {s}} _ {t}, \hat {s} _ {t} ^ {\prime}) + \frac {\bar {\rho}}{2} \left(\| \hat {\mathbf {w}} _ {t} - \mathbf {w} _ {t + 1} \| ^ {2} + \| \hat {\mathbf {s}} _ {t} - \mathbf {s} _ {t + 1} \| ^ {2} + \| \hat {s} _ {t} ^ {\prime} - s _ {t + 1} ^ {\prime} \| ^ {2}\right) \right] \\ = F \left(\hat {\mathbf {w}} _ {t}, \hat {\mathbf {s}} _ {t}, \hat {s} _ {t} ^ {\prime}\right) + \mathbb {E} _ {t} \left[ \frac {\bar {\rho}}{2} \left(\| \hat {\mathbf {w}} _ {t} - \left(\mathbf {w} _ {t} - \eta G _ {t}\right) \| ^ {2} + \| \hat {\mathbf {s}} _ {t} - \left(\mathbf {s} _ {t} - \eta G _ {t} ^ {1}\right) \| ^ {2} \right. \right. \\ \left. + \left\| \hat {s} _ {t} ^ {\prime} - \left(s _ {t} ^ {\prime} - \eta G _ {t} ^ {2}\right) \right\| ^ {2}\right) \Bigg ] \tag {37} \\ \leq F (\hat {\mathbf {w}} _ {t}, \hat {\mathbf {s}} _ {\mathbf {t}}, \hat {s} _ {t} ^ {\prime}) + \frac {\bar {\rho}}{2} \left(\| \hat {\mathbf {w}} _ {t} - \mathbf {w} _ {t} \| ^ {2} + \| \hat {\mathbf {s}} _ {t} - \mathbf {s} _ {t} \| ^ {2} + \| \hat {s} _ {t} ^ {\prime} - s _ {t} ^ {\prime} \| ^ {2}\right) \\ + \bar {\rho} \mathbb {E} _ {t} \big [ \eta \langle \hat {\mathbf {w}} _ {t} - \mathbf {w} _ {t}, G _ {t} \rangle + \eta \langle \hat {\mathbf {s}} _ {t} - \mathbf {s} _ {t}, G _ {t} ^ {1} \rangle + \eta \langle \hat {s} _ {t} ^ {\prime} - s _ {t} ^ {\prime}, G _ {t} ^ {2} \rangle \big ] + \frac {3 \eta^ {2} \bar {\rho} M ^ {2}}{2} \\ = F _ {1 / \bar {\rho}} (\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}) + \bar {\rho} \mathbb {E} _ {t} [ \eta \langle \hat {\mathbf {w}} _ {t} - \mathbf {w} _ {t}, G _ {t} \rangle + \eta \langle \hat {\mathbf {s}} _ {t} - \mathbf {s} _ {t}, G _ {t} ^ {1} \rangle + \eta \langle \hat {s} _ {t} ^ {\prime} - s _ {t} ^ {\prime}, G _ {t} ^ {2} \rangle ] \\ + \frac {3 \eta^ {2} \bar {\rho} M ^ {2}}{2} \\ \end{array}
$$
where for simplicity we denote  $G_{t}^{2} = \frac{1}{B_{1}}\sum_{i\in B_{1}^{t}}\partial_{s^{\prime}}f_{i}(u_{i,t},s_{t}^{\prime})$ , and  $G_{t}^{1}$  is a  $n_+$ -dimensional vector whose  $i$ -th coordinate is defined as
$$
\left\{ \begin{array}{l l} \frac {1}{B _ {1}} \partial_ {u} f _ {i} (u _ {i, t}, s _ {t} ^ {\prime}) \left[ \frac {1}{B _ {2}} \sum_ {X _ {j} \in \mathcal {B} _ {2} ^ {t}} \partial_ {s _ {i}} g _ {i} (v _ {j, t} - v _ {i, t}, s _ {i, t}) \right], & X _ {i} \in \mathcal {B} _ {1} ^ {t} \\ 0, & X _ {i} \not \in \mathcal {B} _ {1} ^ {t} \end{array} \right..
$$
The second inequality in the above derivation uses the bounds of  $\mathbb{E}[\| G_t\|^2]$ ,  $\mathbb{E}[\| G_t^1\|^2]$  and  $\mathbb{E}[\| G_t^2\|^2]$ , which follow from the Lipschitz continuity and bounded variance assumptions and are denoted by  $M$ .
$$
\begin{array}{l} \mathbb {E} _ {t} [ G _ {t} ] \\ = \frac {1}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \partial_ {u} f _ {i} \left(u _ {i, t}, s _ {t} ^ {\prime}\right) \left[ \frac {1}{n _ {-}} \sum_ {X _ {j} \in S _ {-}} \partial_ {v} g _ {i} \left(v _ {j, t} - v _ {i, t}, s _ {i, t}\right) \left(\nabla h _ {i} (\mathbf {w}) - \nabla h _ {j} (\mathbf {w})\right) \right] \\ \end{array}
$$
$$
\mathbb {E} _ {t} \left[ G _ {t} ^ {1} \right] = \frac {1}{n _ {+}} \sum_ {X _ {i} \in \mathcal {S} _ {+}} \partial_ {u} f _ {i} \left(u _ {i, t}, s _ {t} ^ {\prime}\right) \left[ \frac {1}{n _ {-}} \sum_ {X _ {j} \in \mathcal {S} _ {-}} \partial_ {\mathbf {s}} g _ {i} \left(v _ {j, t} - v _ {i, t}, s _ {i, t}\right) \right]
$$
$$
\mathbb {E} _ {t} [ G _ {t} ^ {2} ] = \frac {1}{n _ {+}} \sum_ {X _ {i} \in S _ {+}} \partial_ {s ^ {\prime}} f _ {i} (u _ {i, t}, s _ {t} ^ {\prime})
$$
Define  $(\hat{\mathbf{w}}_t,\hat{\mathbf{s}}_t,\hat{s}_t^{\prime})\coloneqq \mathrm{prox}_{F / \bar{\rho}}(\mathbf{w}_t,\mathbf{s}_t,s_t^{\prime})$  . For a given  $i\in \{1,\ldots ,m\}$  , we have