SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} [ \| v _ {i, j, t + 1} - h _ {i, j} (\mathbf {w} _ {t + 1}) \| ] \\ \leq \left(1 - \frac {B _ {1} B _ {2} \tau_ {1}}{2 n _ {1} n _ {2}}\right) ^ {t + 1} \| v _ {i, j, 0} - h _ {i, j} (\mathbf {w} _ {0}) \| + \frac {2 \tau_ {1} ^ {1 / 2} \sigma}{B _ {3} ^ {1 / 2}} + \frac {4 n _ {1} n _ {2} C _ {h} M \eta}{B _ {1} B _ {2} \tau_ {1} ^ {1 / 2}} \\ \end{array}
$$
Taking average over  $(i,j)\in S_1\times S_2$ , we obtain the norm error bound
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n _ {1}} \sum_ {i \in \mathcal {S} _ {1}} \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} \| v _ {i, j, t + 1} - h _ {i, j} (\mathbf {w} _ {t + 1}) \| \right] \\ \leq \left(1 - \frac {B _ {1} B _ {2} \tau_ {1}}{2 n _ {1} n _ {2}}\right) ^ {t + 1} \frac {1}{n _ {1}} \sum_ {i \in \mathcal {S} _ {1}} \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} \| v _ {i, j, 0} - h _ {i, j} (\mathbf {w} _ {0}) \| + \frac {2 \tau_ {1} ^ {1 / 2} \sigma}{B _ {3} ^ {1 / 2}} + \frac {4 n _ {1} n _ {2} C _ {h} M \eta}{B _ {1} B _ {2} \tau_ {1} ^ {1 / 2}}. \\ \end{array}
$$
D.6 Proof of Lemma A.4
Proof of Lemma A.4. With  $\gamma_{2} = \frac{n_{1} - B_{1}}{B_{1}(1 - \tau_{2})} + (1 - \tau_{2})$  and  $\tau_{2} \leq \frac{1}{2}$ , MSVR update has the following recursive error bound [15]
$$
\begin{array}{l} \mathbb {E} [ \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t + 1}) \| ^ {2} ] \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{n _ {1}}\right) \mathbb {E} \left[ \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} \left(v _ {i, j, t}\right) \| ^ {2} \right] + \frac {2 \tau_ {2} ^ {2} B _ {1} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 n _ {1} C _ {g} ^ {2}}{B _ {1}} \mathbb {E} \left[ \| v _ {i, j, t + 1} - v _ {i, j, t} \| ^ {2} \right] \tag {45} \\ \end{array}
$$
It remains to bound  $\mathbb{E}[\| v_{i,j,t + 1} - v_{i,j,t}\| ^2 ]$ , which is done as following
$$
\begin{array}{l} \mathbb {E} \left[ \| v _ {i, j, t + 1} - v _ {i, j, t} \| ^ {2} \right] \\ \leq \mathbb {E} \left[ \frac {B _ {1} B _ {2}}{n _ {1} n _ {2}} \| \tau_ {1} v _ {i, j, t} - \tau_ {1} h _ {i, j} (\mathbf {w} _ {t}; \mathcal {B} _ {3, i, j} ^ {t}) - \gamma_ {1} (h _ {i, j} (\mathbf {w} _ {t}; \mathcal {B} _ {3, i, j} ^ {t}) - h _ {i, j} (\mathbf {w} _ {t - 1}; \mathcal {B} _ {3, i, j} ^ {t})) \| ^ {2} \right] \\ \leq \mathbb {E} \left[ \frac {2 B _ {1} B _ {2} \tau_ {1} ^ {2}}{n _ {1} n _ {2}} \| v _ {i, j, t} - h _ {i, j} (\mathbf {w} _ {t}; \mathcal {B} _ {3, i, j} ^ {t}) \| ^ {2} + \frac {2 B _ {1} B _ {2} \gamma_ {1} ^ {2}}{n _ {1} n _ {2}} \| h _ {i, j} (\mathbf {w} _ {t}; \mathcal {B} _ {3, i, j} ^ {t}) - h _ {i, j} (\mathbf {w} _ {t - 1}; \mathcal {B} _ {3, i, j} ^ {t}) \| ^ {2} \right] \\ \leq \mathbb {E} \left[ \frac {2 B _ {1} B _ {2} \tau_ {1} ^ {2}}{n _ {1} n _ {2}} \| v _ {i, j, t} - h _ {i, j} (\mathbf {w} _ {t}; \mathcal {B} _ {3, i, j} ^ {t}) \| ^ {2} + \frac {2 B _ {1} B _ {2} \gamma_ {1} ^ {2} C _ {h}}{n _ {1} n _ {2}} \| \mathbf {w} _ {t} - \mathbf {w} _ {t - 1} \| ^ {2} \right] \\ \stackrel {(a)} {\leq} \frac {8 B _ {1} B _ {2} \tau_ {1} ^ {2} M ^ {2}}{n _ {1} n _ {2}} + \frac {8 n _ {1} n _ {2} C _ {h} ^ {2} \eta^ {2} M ^ {2}}{B _ {1} B _ {2}} \\ \end{array}
$$
where inequality (a) uses  $\tau_{1} \leq 1/2$  and  $\gamma_{1} = \frac{n_{1}n_{2} - B_{1}B_{2}}{B_{1}B_{2}(1 - \tau_{1})} + (1 - \tau_{1}) \leq \frac{2n_{1}n_{2}}{B_{1}B_{2}}$ . Plugging the above inequality back into inequality 45 gives
$$
\begin{array}{l} \mathbb {E} [ \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t + 1}) \| ^ {2} ] \\ \leq (1 - \frac {B _ {1} \tau_ {2}}{n _ {1}}) \mathbb {E} [ \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} ] + \frac {2 \tau_ {2} ^ {2} B _ {1} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 n _ {1} C _ {g} ^ {2}}{B _ {1}} \left(\frac {8 B _ {1} B _ {2} \tau_ {1} ^ {2} M ^ {2}}{n _ {1} n _ {2}} + \frac {8 n _ {1} n _ {2} C _ {h} ^ {2} \eta^ {2} M ^ {2}}{B _ {1} B _ {2}}\right) \\ \leq (1 - \frac {B _ {1} \tau_ {2}}{n _ {1}}) \mathbb {E} [ \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in S _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} ] + \frac {2 \tau_ {2} ^ {2} B _ {1} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {6 4 B _ {2} \tau_ {1} ^ {2} M ^ {2} C _ {g} ^ {2}}{n _ {2}} + \frac {6 4 n _ {1} ^ {2} n _ {2} C _ {h} ^ {2} \eta^ {2} M ^ {2} C _ {g} ^ {2}}{B _ {1} ^ {2} B _ {2}} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t + 1}) \| ^ {2} ] \\ \leq (1 - \frac {B _ {1} \tau_ {2}}{2 n _ {1}}) ^ {2 (t + 1)} \| u _ {i, 0} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, 0}) \| ^ {2} + \sum_ {j = 0} ^ {t} (1 - \frac {B _ {1} \tau_ {2}}{2 n _ {1}}) ^ {t - j} \bigg (\frac {2 \tau_ {2} ^ {2} B _ {1} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {6 4 B _ {2} \tau_ {1} ^ {2} M ^ {2} C _ {g} ^ {2}}{n _ {2}} \bigg) \\ \left. + \frac {6 4 n _ {1} ^ {2} n _ {2} C _ {h} ^ {2} \eta^ {2} M ^ {2} C _ {g} ^ {2}}{B _ {1} ^ {2} B _ {2}}\right) \\ \leq (1 - \frac {B _ {1} \tau_ {2}}{2 n _ {1}}) ^ {2 (t + 1)} \| u _ {i, 0} - \frac {1}{n _ {2}} \sum_ {j \in S _ {2}} g _ {i} (v _ {i, j, 0}) \| ^ {2} + \frac {4 \tau_ {2} \sigma^ {2}}{B _ {2}} + \frac {1 2 8 n _ {1} B _ {2} \tau_ {1} ^ {2} M ^ {2} C _ {g} ^ {2}}{B _ {1} n _ {2} \tau_ {2}} + \frac {1 2 8 n _ {1} ^ {3} n _ {2} C _ {h} ^ {2} \eta^ {2} M ^ {2} C _ {g} ^ {2}}{B _ {1} ^ {3} B _ {2} \tau_ {2}} \\ \end{array}
$$
where we use  $\sum_{j=0}^{t} \left(1 - \frac{B_1 \tau_2}{2n_1}\right)^{2(t-j)} \leq \frac{2n_1}{B_1 \tau_1}$ . Taking average over  $i \in S_1$  gives the squared-norm error bound.
$$
\begin{array}{l} \mathbb {E} \left[ \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} \left(v _ {i, j, t + 1}\right) \| \right] ^ {2} \\ \leq \mathbb {E} [ \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t + 1}) \| ^ {2} ] \\ \leq (1 - \frac {B _ {1} \tau_ {2}}{2 n _ {1}}) ^ {2 (t + 1)} \| u _ {i, 0} - \frac {1}{n _ {2}} \sum_ {j \in S _ {2}} g _ {i} (v _ {i, j, 0}) \| ^ {2} + \frac {4 \tau_ {2} \sigma^ {2}}{B _ {2}} + \frac {1 2 8 n _ {1} B _ {2} \tau_ {1} ^ {2} M ^ {2} C _ {g} ^ {2}}{B _ {1} n _ {2} \tau_ {2}} + \frac {1 2 8 n _ {1} ^ {3} n _ {2} C _ {h} ^ {2} \eta^ {2} M ^ {2} C _ {g} ^ {2}}{B _ {1} ^ {3} B _ {2} \tau_ {2}} \\ \leq \left[ (1 - \frac {B _ {1} \tau_ {2}}{2 n _ {1}}) ^ {t + 1} \| u _ {i, 0} - \frac {1}{n _ {2}} \sum_ {j \in S _ {2}} g _ {i} (v _ {i, j, 0}) \| + \frac {2 \tau_ {2} ^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {8 \sqrt {2} n _ {1} ^ {1 / 2} B _ {2} ^ {1 / 2} \tau_ {1} M C _ {g}}{B _ {1} ^ {1 / 2} n _ {2} ^ {1 / 2} \tau_ {2} ^ {1 / 2}} + \frac {8 \sqrt {2} n _ {1} ^ {3 / 2} n _ {2} ^ {1 / 2} C _ {h} \eta M C _ {g}}{B _ {1} ^ {3 / 2} B _ {2} ^ {1 / 2} \tau_ {2} ^ {1 / 2}} \right] ^ {2} \\ \end{array}
$$
Taking squared root on both sides and taking average over  $i \in S_1$ , we obtain the norm error bound
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n _ {1}} \sum_ {i \in \mathcal {S} _ {1}} \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t + 1}) \| \right] \\ \leq (1 - \frac {B _ {1} \tau_ {2}}{2 n _ {1}}) ^ {t + 1} \frac {1}{n _ {1}} \sum_ {i \in \mathcal {S} _ {1}} \| u _ {i, 0} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, 0}) \| + \frac {2 \tau_ {2} ^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {C _ {2} n _ {1} ^ {1 / 2} B _ {2} ^ {1 / 2} \tau_ {1}}{B _ {1} ^ {1 / 2} n _ {2} ^ {1 / 2} \tau_ {2} ^ {1 / 2}} + \frac {C _ {2} n _ {1} ^ {3 / 2} n _ {2} ^ {1 / 2} \eta}{B _ {1} ^ {3 / 2} B _ {2} ^ {1 / 2} \tau_ {2} ^ {1 / 2}} \\ \end{array}
$$
where  $C_2 = \max \{8\sqrt{2} MC_g, 8\sqrt{2} C_h MC_g\}$ .
D.7 Proof of Lemma B.2
Proof of Lemma B.2. Define
$$
\tilde {u} _ {i, t} = (1 - \tau) u _ {i, t} + \tau g _ {i} (\mathbf {w} _ {t}; \mathcal {B} _ {2, i} ^ {t})
$$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {B} _ {2, i} ^ {2}} \left[ \| \tilde {u} _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} \right] \\ = \mathbb {E} _ {\mathcal {B} _ {2, i} ^ {t}} [ \| (1 - \tau) (u _ {i, t} - g _ {i} (\mathbf {w} _ {t})) + \tau (g _ {i} (\mathbf {w} _ {t}; \mathcal {B} _ {2, i} ^ {t}) - g _ {i} (\mathbf {w} _ {t})) \| ^ {2} ] \\ = \mathbb {E} _ {\mathcal {B} _ {2, i} ^ {t}} \left[ (1 - \tau) ^ {2} \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} + \tau^ {2} \| g _ {i} (\mathbf {w} _ {t}; \mathcal {B} _ {2, i} ^ {t}) - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} \right. \\ \left. + 2 (1 - \tau) \tau \langle u _ {i, t} - g _ {i} (\mathbf {w} _ {t}), g _ {i} (\mathbf {w} _ {t}; \mathcal {B} _ {2, i} ^ {t}) - g _ {i} (\mathbf {w} _ {t})) \right\rangle \\ \leq (1 - \tau) ^ {2} \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} + \frac {\tau^ {2} \sigma^ {2}}{B _ {2}} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {B} _ {2, i} ^ {t}} \mathbb {E} _ {\mathcal {B} _ {1} ^ {t}} \left[ \left\| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t}) \right\| ^ {2} \right] \\ = \frac {B _ {1}}{n _ {1}} \mathbb {E} _ {\boldsymbol {B} _ {2, i} ^ {t}} [ \| \tilde {u} _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} ] + (1 - \frac {B _ {1}}{n _ {1}}) \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} \\ \leq \frac {B _ {1}}{n _ {1}} (1 - \tau) ^ {2} \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} + \frac {B _ {1} \tau^ {2} \sigma^ {2}}{n _ {1} B _ {2}} + (1 - \frac {B _ {1}}{n _ {1}}) \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} \\ \leq \left(1 - \frac {B _ {1} \tau}{2 n _ {1}}\right) ^ {2} \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} + \frac {B _ {1} \tau^ {2} \sigma^ {2}}{n _ {1} B _ {2}} \\ \end{array}
$$
where we use
$$
\begin{array}{l} \frac {B _ {1}}{n _ {1}} (1 - \tau) ^ {2} + \left(1 - \frac {B _ {1}}{n _ {1}}\right) = \frac {B _ {1}}{n _ {1}} \left(1 - 2 \tau + \tau^ {2}\right) + 1 - \frac {B _ {1}}{n _ {1}} \\ = 1 - 2 \tau \frac {B _ {1}}{n _ {1}} + \tau^ {2} \frac {B _ {1}}{n _ {1}} \\ \leq 1 - \tau \frac {B _ {1}}{n _ {1}} \\ \leq 1 - \tau \frac {B _ {1}}{n _ {1}} + (\frac {\tau B _ {1}}{2 n _ {1}}) ^ {2} = (1 - \frac {\tau B _ {1}}{2 n _ {1}}) ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \| u _ {i, t + 1} - g _ {i} \left(\mathbf {w} _ {t + 1}\right) \| ^ {2} \right] \\ \leq \mathbb {E} _ {t} \left[ \left(1 + \frac {B _ {1} \tau}{4 n _ {1}}\right) \| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} + \left(1 + \frac {4 n _ {1}}{B _ {1} \tau}\right) \| g _ {i} (\mathbf {w} _ {t}) - g _ {i} (\mathbf {w} _ {t + 1}) \| ^ {2} \right] \\ \leq \left(1 + \frac {B _ {1} \tau}{4 n _ {1}}\right) \left(1 - \frac {B _ {1} \tau}{2 n _ {1}}\right) ^ {2} \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} + \left(1 + \frac {B _ {1} \tau}{4 n _ {1}}\right) \frac {B _ {1} \tau^ {2} \sigma^ {2}}{n _ {1} B _ {2}} \\ + \left(1 + \frac {4 n _ {1}}{B _ {1} \tau}\right) C _ {g} ^ {2} \mathbb {E} _ {t} \| \mathbf {w} _ {t} - \mathbf {w} _ {t + 1} \| ^ {2} ] \\ \leq \left(1 - \frac {B _ {1} \tau}{4 n _ {1}}\right) ^ {2} \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} + \frac {2 B _ {1} \tau^ {2} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 n _ {1}}{B _ {1} \tau} C _ {g} ^ {2} \mathbb {E} _ {t} \| \mathbf {w} _ {t} - \mathbf {w} _ {t + 1} \| ^ {2} ] \\ \leq \left(1 - \frac {B _ {1} \tau}{4 n _ {1}}\right) ^ {2} \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} + \frac {2 B _ {1} \tau^ {2} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 n _ {1} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} \tau} \\ \end{array}
$$
where we use  $\frac{B_1\tau}{4n_1} \leq 1$ . Applying this inequality recursively, we obtain
$$
\begin{array}{l} \mathbb {E} \left[ \left\| u _ {i, t + 1} - g _ {i} \left(\mathbf {w} _ {t + 1}\right) \right\| ^ {2} \right] \\ \leq \left(1 - \frac {B _ {1} \tau}{4 n _ {1}}\right) ^ {2} \mathbb {E} [ \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} ] + \frac {2 B _ {1} \tau^ {2} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 n _ {1} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} \tau} \\ \leq \left(1 - \frac {B _ {1} \tau}{4 n _ {1}}\right) ^ {2 (t + 1)} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| ^ {2} + \sum_ {j = 0} ^ {t} \left(1 - \frac {B _ {1} \tau}{4 n _ {1}}\right) ^ {2 (t - j)} \left[ \frac {2 B _ {1} \tau^ {2} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 n _ {1} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} \tau} \right] \\ \leq (1 - \frac {B _ {1} \tau}{4 n _ {1}}) ^ {2 (t + 1)} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| ^ {2} + \frac {8 \tau \sigma^ {2}}{B _ {2}} + \frac {3 2 n _ {1} ^ {2} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} \tau^ {2}} \\ \end{array}
$$
where we use  $\sum_{j=0}^{t}(1 - \frac{B_1\tau}{4n_1})^{2(t-j)} \leq \frac{4n_1}{B_1\tau}$ .
To obtain the absolute bound, we derive
$$
\begin{array}{l} \mathbb {E} \left[ \| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t + 1}) \| \right] ^ {2} \leq \mathbb {E} \left[ \| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t + 1}) \| ^ {2} \right] \\ \leq \left(1 - \frac {B _ {1} \tau}{4 n _ {1}}\right) ^ {2 (t + 1)} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| ^ {2} + \frac {8 \tau \sigma^ {2}}{B _ {2}} + \frac {3 2 n _ {1} ^ {2} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} \tau^ {2}} \\ \leq \left[ \left(1 - \frac {B _ {1} \tau}{4 n _ {1}}\right) ^ {t + 1} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| + \frac {2 \sqrt {2} \tau^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {4 \sqrt {2} n _ {1} C _ {g} M \eta}{B _ {1} \tau} \right] ^ {2} \\ \end{array}
$$
The desired result follows by taking squared root on both sides.
D.8 Proof of Lemma B.5
Proof of Lemma B.5. The proof of Lemma B.5 is the same as Lemma B.2.
D.9 Proof of Lemma B.6
Proof of Lemma B.6. Define
$$
\tilde {u} _ {i, t} = (1 - \tau_ {2}) u _ {i, t} + \tau_ {2} \frac {1}{B _ {2}} \sum_ {j \in \mathcal {B} _ {2, i} ^ {t}} g _ {i} (v _ {i, j, t})
$$
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {B} _ {2} ^ {t}} [ \| \tilde {u} _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} ] \\ = \mathbb {E} _ {\mathcal {B} _ {2} ^ {t}} [ \| (1 - \tau_ {2}) (u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t})) + \tau_ {2} (\frac {1}{B _ {2}} \sum_ {j \in \mathcal {B} _ {2} ^ {t}} g _ {i} (v _ {i, j, t}) - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t})) \| ^ {2} ] \\ = \mathbb {E} _ {\mathcal {B} _ {2} ^ {t}} [ (1 - \tau_ {2}) ^ {2} \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} + \tau_ {2} ^ {2} \| \frac {1}{B _ {2}} \sum_ {j \in \mathcal {B} _ {2} ^ {t}} g _ {i} (v _ {i, j, t}) - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} \\ + 2 (1 - \tau_ {2}) \tau_ {2} \langle u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}), \frac {1}{B _ {2}} \sum_ {j \in \mathcal {B} _ {2} ^ {t}} g _ {i} (v _ {i, j, t}) - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \rangle ] \\ \leq (1 - \tau_ {2}) ^ {2} \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} + \frac {\tau_ {2} ^ {2} \sigma^ {2}}{B _ {2}} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {B} _ {2, i} ^ {t}} \mathbb {E} _ {\mathcal {B} _ {1} ^ {t}} [ \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} ] \\ = \frac {B _ {1}}{n _ {1}} \mathbb {E} _ {\mathcal {B} _ {2} ^ {t}} [ \| \tilde {u} _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} ] + (1 - \frac {B _ {1}}{n _ {1}}) \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} \\ \leq \frac {B _ {1}}{n _ {1}} (1 - \tau_ {2}) ^ {2} \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} + \frac {B _ {1} \tau_ {2} ^ {2} \sigma^ {2}}{n _ {1} B _ {2}} + (1 - \frac {B _ {1}}{n _ {1}}) \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{2 n _ {1}}\right) ^ {2} \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in S _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} + \frac {B _ {1} \tau^ {2} \sigma^ {2}}{n _ {1} B _ {2}} \\ \end{array}
$$
$$
\frac {B _ {1}}{n _ {1}} (1 - \tau_ {2}) ^ {2} + (1 - \frac {B _ {1}}{n _ {1}}) \leq (1 - \frac {\tau_ {2} B _ {1}}{2 n _ {1}}) ^ {2}
$$
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} \left(v _ {i, j, t + 1}\right) \| ^ {2} \right] \\ \leq \mathbb {E} _ {t} \left[ (1 + \frac {B _ {1} \tau_ {2}}{4 n _ {1}}) \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} + (1 + \frac {4 n _ {1}}{B _ {1} \tau_ {2}}) \| \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t + 1}) \| ^ {2} \right] \\ \leq \left(1 + \frac {B _ {1} \tau_ {2}}{4 n _ {1}}\right) \left(1 - \frac {B _ {1} \tau_ {2}}{2 n _ {1}}\right) ^ {2} \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} \left(v _ {i, j, t}\right) \| ^ {2} + \left(1 + \frac {B _ {1} \tau_ {2}}{4 n _ {1}}\right) \frac {B _ {1} \tau_ {2} ^ {2} \sigma^ {2}}{n _ {1} B _ {2}} \\ + \left(1 + \frac {4 n _ {1}}{B _ {1} \tau_ {2}}\right) C _ {g} ^ {2} \mathbb {E} _ {t} \left[ \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} \| v _ {i, j, t} - v _ {i, j, t + 1} \| ^ {2} \right] \\ \leq (1 - \frac {B _ {1} \tau_ {2}}{4 n _ {1}}) ^ {2} \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} + \frac {2 B _ {1} \tau_ {2} ^ {2} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 C _ {g} ^ {2} M ^ {2} B _ {2} \tau_ {1} ^ {2}}{n _ {2} \tau_ {2}} \\ \end{array}
$$
where we use  $\frac{B_1\tau_2}{4n_1} \leq 1$ , and
$$
\mathbb {E} _ {t} [ \| v _ {i, j, t} - v _ {i, j, t + 1} \| ^ {2} ] = \frac {B _ {1} B _ {2}}{n _ {1} n _ {2}} \mathbb {E} _ {\mathcal {B} _ {3, i, j} ^ {t}} \| \tau_ {1} v _ {i, j, t} - \tau_ {1} h _ {i, j} (\mathbf {w} _ {t}; \mathcal {B} _ {3, i, j} ^ {t}) \| ^ {2} \leq \frac {B _ {1} B _ {2} \tau_ {1} ^ {2} M ^ {2}}{n _ {1} n _ {2}}.
$$
$$
\begin{array}{l} \mathbb {E} \left[ \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} \left(v _ {i, j, t + 1}\right) \| ^ {2} \right] \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{4 n _ {1}}\right) ^ {2} \mathbb {E} [ \| u _ {i, t} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t}) \| ^ {2} ] + \frac {2 B _ {1} \tau_ {2} ^ {2} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 C _ {g} ^ {2} M ^ {2} B _ {2} \tau_ {1} ^ {2}}{n _ {2} \tau_ {2}} \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{4 n _ {1}}\right) ^ {2 (t + 1)} \| u _ {i, 0} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, 0}) \| ^ {2} + \sum_ {j = 0} ^ {t} \left(1 - \frac {B _ {1} \tau_ {2}}{4 n _ {1}}\right) ^ {2 (t - j)} \left[ \frac {2 B _ {1} \tau_ {2} ^ {2} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 C _ {g} ^ {2} M ^ {2} B _ {2} \tau_ {1} ^ {2}}{n _ {2} \tau_ {2}} \right] \\ \leq (1 - \frac {B _ {1} \tau_ {2}}{4 n _ {1}}) ^ {2 (t + 1)} \| u _ {i, 0} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, 0}) \| ^ {2} + \frac {8 \tau_ {2} \sigma^ {2}}{B _ {2}} + \frac {3 2 C _ {g} ^ {2} M ^ {2} n _ {1} B _ {2} \tau_ {1} ^ {2}}{B _ {1} n _ {2} \tau_ {2} ^ {2}} \\ \end{array}
$$
where we use  $\sum_{j=0}^{t}(1 - \frac{B_1\tau_2}{4n_1})^{2(t-j)} \leq \frac{4n_1}{B_1\tau_2}$ .
$$
\begin{array}{l} \mathbb {E} \left[ \left\| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} \left(v _ {i, j, t + 1}\right) \right\| \right] ^ {2} \\ \leq \mathbb {E} [ \| u _ {i, t + 1} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, t + 1}) \| ^ {2} ] \\ \leq (1 - \frac {B _ {1} \tau_ {2}}{4 n _ {1}}) ^ {2 (t + 1)} \| u _ {i, 0} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, 0}) \| ^ {2} + \frac {8 \tau_ {2} \sigma^ {2}}{B _ {2}} + \frac {3 2 C _ {g} ^ {2} M ^ {2} n _ {1} B _ {2} \tau_ {1} ^ {2}}{B _ {1} n _ {2} \tau_ {2} ^ {2}} \\ \leq \left[ \left(1 - \frac {B _ {1} \tau_ {2}}{4 n _ {1}}\right) ^ {t + 1} \| u _ {i, 0} - \frac {1}{n _ {2}} \sum_ {j \in \mathcal {S} _ {2}} g _ {i} (v _ {i, j, 0}) \| + \frac {2 \sqrt {2} \tau_ {2} ^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {4 \sqrt {2} C _ {g} M n _ {1} ^ {1 / 2} B _ {2} ^ {1 / 2} \tau_ {1}}{B _ {1} ^ {1 / 2} n _ {2} ^ {1 / 2} \tau_ {2}} \right] ^ {2} \\ \end{array}
$$
E Group Distributionally Robust Optimization
NSWC FCCO finds an important application in group distributionally robust optimization (group DRO), particularly valuable in addressing distributional shift [25]. Consider  $N$  groups with different distributions. Each group  $k$  has an averaged loss  $L_{k}(w) = \frac{1}{n_{k}}\sum_{i=1}^{n_{k}}\ell(f_{w}(x_{i}^{k}),y_{i}^{k})$ , where  $w$  is the model parameter and  $(x_{i}^{k},y_{i}^{k})$  is a data point. For robust optimization, we assign different weights to different groups and form the following robust loss minimization problem:
$$
\min  _ {w} \max  _ {p \in \Omega} \sum_ {k = 1} ^ {N} p _ {k} L _ {k} (w),
$$
where  $\Omega \subset \Delta$  and  $\Delta$  denotes a simplex. A common choice for  $\Omega$  is  $\Omega = \{\mathbf{p} \in \Delta, p_i \leq 1 / K\}$  where  $K$  is an integer, resulting in the so-called CVaR losses, i.e., average of top-K group losses. Consequently, the above problem can be equivalently reformulated as [23]:
This formulation can be mapped into non-smooth weakly-convex FCCO when the loss function  $\ell(\cdot, \cdot)$  is weakly convex in terms of  $w$ . In comparison to directly solving the min-max problem, solving the above FCCO problem avoids the need of dealing with the projection onto the constraint  $\Omega$  and expensive sampling as in existing works [4].
F More Information for Experiments
F.1 Dataset Statistics
Table 3: Datasets Statistics. The percentage in parenthesis represents the proportion of positive samples.
Dataset	Train	Validation	Test
moltox21(t0)	5834 (4.25%)	722 (4.01%)	709 (4.51%)
molmuv(t1)	11466 (0.18%)	1559 (0.13%)	1709 (0.35%)
molpcba(t0)	120762 (9.32%)	19865 (11.74%)	20397 (11.61%)
Table 4: Data statistics for the MIL datasets.  $D_{+} / D_{-}$  is the positive/negative bag number.
Data Format	Dataset	D+	D-	average bag size	#features
Tabular	MUSK2	39	63	64.69	166
Fox	100	100	6.6	230
Histopathological	Lung	100	1000	256	32x32x3
Image	Lung	100	1000	256	32x32x3
F.2 Illustration for Histopathology Dataset on MIL Task
Figure 2: Illustration for Histopathology Dataset on MIL Task. Ade. is abbreviated for adenocarcinoma and SCC is short for squamous cell carcinoma. In this work, each RGB image is separated by  $32 \times 32$  non-overlapped patches, which constitute the bag.