SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
R _ {5} = \frac {2 \tau_ {2} ^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + C _ {2} \frac {n _ {+}}{B _ {1}} (\frac {B _ {1} ^ {1 / 2}}{n _ {+} ^ {1 / 2}} + \frac {B _ {2} ^ {1 / 2}}{n _ {-} ^ {1 / 2}}) \frac {\tau_ {1}}{\tau_ {2} ^ {1 / 2}} + C _ {2} \frac {n _ {+}}{B _ {1}} (\frac {n _ {+} ^ {1 / 2}}{B _ {1} ^ {1 / 2}} + \frac {n _ {-} ^ {1 / 2}}{B _ {2} ^ {1 / 2}}) \frac {\eta}{\tau_ {2} ^ {1 / 2}} + C _ {2} \frac {n _ {+} ^ {1 / 2} \eta}{B _ {1} \tau_ {2} ^ {1 / 2}}
$$
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla F _ {1 / \bar {\rho}} \left(\mathbf {w} _ {t}, \mathbf {s} _ {t}, s _ {t} ^ {\prime}\right) \| ^ {2} \\ \leq \mathcal {O} \left(\frac {1}{T} \left(\frac {1}{\eta} + \frac {1}{\mu_ {\text {m i n}}}\right) + \left(\frac {\tau_ {1} ^ {1 / 2}}{B _ {3} ^ {1 / 2}} + \frac {\tau_ {2} ^ {1 / 2}}{B _ {2} ^ {1 / 2}}\right) \sigma \right. \\ \left. + \frac {n _ {+} \eta}{B _ {1} \tau_ {1} ^ {1 / 2}} + \frac {n _ {-} \eta}{B _ {2} \tau_ {1} ^ {1 / 2}} + \frac {n _ {+}}{B _ {1}} \max  \left\{\frac {B _ {1} ^ {1 / 2}}{n _ {+} ^ {1 / 2}}, \frac {B _ {2} ^ {1 / 2}}{n _ {-} ^ {1 / 2}} \right\} \frac {\tau_ {1}}{\tau_ {2} ^ {1 / 2}} + \frac {n _ {+}}{B _ {1}} \max  \left\{\frac {n _ {+} ^ {1 / 2}}{B _ {1} ^ {1 / 2}}, \frac {n _ {-} ^ {1 / 2}}{B _ {2} ^ {1 / 2}} \right\} \frac {\eta}{\tau_ {2} ^ {1 / 2}}\right) \\ \end{array}
$$
$$
\tau_ {1} = \mathcal {O} \left(\min  \left\{B _ {3}, \frac {B _ {1}}{n _ {+}} \min  \left\{\frac {n _ {+} ^ {1 / 2}}{B _ {1} ^ {1 / 2}}, \frac {n _ {-} ^ {1 / 2}}{B _ {2} ^ {1 / 2}} \right\} B _ {2} ^ {1 / 2} \right\} \epsilon^ {4}\right), \quad \tau_ {2} = \mathcal {O} (B _ {2} \epsilon^ {4}),
$$
$$
\eta = \mathcal {O} \left(\min  \left\{\min  \left\{\frac {B _ {1}}{n _ {+}}, \frac {B _ {2}}{n _ {-}} \right\} \min  \left\{B _ {3} ^ {1 / 2}, \frac {B _ {1} ^ {1 / 2}}{n _ {+} ^ {1 / 2}} \min  \left\{\frac {n _ {+} ^ {1 / 4}}{B _ {1} ^ {1 / 4}}, \frac {n _ {-} ^ {1 / 4}}{B _ {2} ^ {1 / 4}} \right\} B _ {2} ^ {1 / 4} \right\}, \frac {B _ {1}}{n _ {+}} \min  \left\{\frac {B _ {1} ^ {1 / 2}}{n _ {+} ^ {1 / 2}}, \frac {B _ {2} ^ {1 / 2}}{n _ {-} ^ {1 / 2}} \right\} B _ {3} ^ {1 / 2} \right\} \epsilon^ {4}\right),
$$
Then with
iterations, we have
D Proofs of Lemmas and Propositions
D.1 Additional Proposition
Proposition D.1. Consider a Lipschitz continuous function  $f: O \to \mathbb{R}$  where  $O \subset \mathbb{R}^d$  is an open set. Assume  $f$  to be non-increasing (resp. non-decreasing) with respect to each element in the input, then all subgradients of  $f$  are element-wise non-positive (resp. non-negative).
Proof of Proposition D.1. Let  $D$  be the subset of  $O$  where  $f$  is differentiable. By Theorem 9.60 in [24], a Lipschitz continuous function  $f: O \to \mathbb{R}$ , where  $O \subset \mathbb{R}^d$  is an open set, is differentiable almost everywhere, i.e.,  $D$  is dense in  $O$ . Then by Theorem 9.61 in [24], the subdifferential of  $f$  at  $x$  is defined as
$$
\partial f (x) = \operatorname {c o n} \left\{v \mid \exists x _ {k} \rightarrow x \text {w i t h} x _ {k} \in D, \nabla f \left(x _ {k}\right)\rightarrow v \right\},
$$
where  $\text{con}$  denotes the convex hull. If we assume that  $f$  is non-increasing with respect to each element in the input, then  $\nabla f(x) \leq 0$  (element-wise) for all differentiable points  $x \in D$ . It implies that the all vectors in  $\{v | \exists x_k \to x \text{ with } x_k \in D, \nabla f(x_k) \to v\}$  are element-wise non-positive. Therefore, all subgradients of  $f$  are element-wise non-positive. On the other hand, if we assume that  $f$  is non-decreasing, one may follow the same argument and conclude that all subgradients of  $f$  are element-wise non-negative.
For functions  $f: O \to \mathbb{R}^m$  where  $O \subset \mathbb{R}^d$  is an open set, one may write  $f = (f_1, \ldots, f_m)$  and apply the above proposition for each  $f_k, k = 1, \ldots, m$ .
D.2 Proofs of Proposition 4.2 and Proposition 4.4
To prove Proposition 4.2 and Proposition 4.4, we first present the following proposition on the weak-convexity of composition functions.
Proposition D.2. Assume  $f: \mathbb{R}^d \to \mathbb{R}$  is  $\rho_1$ -weakly-convex and  $C_1$ -Lipschitz continuous,  $g: \mathbb{R}^d \to \mathbb{R}^d$  is  $C_2$ -Lipschitz continuous, and either of the followings holds:
1.  $f(\cdot)$  is monotone and  $g(\cdot)$  is  $L_{2}$ -smooth;
2.  $f(\cdot)$  is non-decreasing and  $g(\cdot)$  is  $L_{2}$ -weakly-convex,
then  $f\circ g$  is  $\tilde{\rho}$ -weakly-convex with  $\tilde{\rho} = \sqrt{d} L_2C_1 + \rho_1C_2^2$ .
Proof of Proposition D.2. The weak convexity of  $f$  implies
$$
\begin{array}{l} f (g (y)) \geq f (g (x)) + v ^ {\top} (g (y) - g (x)) - \frac {\rho_ {1}}{2} \| g (y) - g (x) \| ^ {2} \\ \geq f (g (x)) + v ^ {\top} (g (y) - g (x)) - \frac {\rho_ {1} C _ {2} ^ {2}}{2} \| x - y \| ^ {2} \\ \end{array}
$$
where  $v \in \partial f(g(x))$ . Moreover, due to the smoothness of  $g(\cdot)$  (or weakly-convexity of  $g(\cdot)$ , then only the second inequality holds), we have
$$
g (y) - g (x) \leq \nabla g (x) ^ {\top} (y - x) + \mathbf {v} \left(\frac {L _ {2}}{2} \| x - y \| ^ {2}\right), \tag {43}
$$
$$
g (y) - g (x) \geq \nabla g (x) ^ {\top} (y - x) - \mathbf {v} \left(\frac {L _ {2}}{2} \| x - y \| ^ {2}\right).
$$
where  $\mathbf{v}(e)$  denotes a  $d$ -dimensional vector with value  $e$  on each dimensions. We first assume that  $f$  is non-increasing, then we may use the first inequality in (43) and the Lipschitz continuity of  $g$  to get
$$
\begin{array}{l} f (g (y)) \geq f (g (x)) + v ^ {\top} \left[ \nabla g (x) ^ {\top} (y - x) + \mathbf {v} \left(\frac {L _ {2}}{2} \| x - y \| ^ {2}\right) \right] - \frac {\rho_ {1} C _ {2} ^ {2}}{2} \| x - y \| ^ {2} \\ \geq f (g (x)) + v ^ {\top} \nabla g (x) ^ {\top} (y - x) + v ^ {\top} \mathbf {v} \left(\frac {L _ {2}}{2} \| x - y \| ^ {2}\right) - \frac {\rho_ {1} C _ {2} ^ {2}}{2} \| x - y \| ^ {2} \\ \geq f (g (x)) + \langle v ^ {\top} \nabla g (x) ^ {\top} (y - x) - \frac {\sqrt {d} L _ {2} C _ {1} + \rho_ {1} C _ {2} ^ {2}}{2} \| x - y \| ^ {2}. \\ \end{array}
$$
On the other hand, if we assume  $f$  is non-decreasing, the same result follows from the second inequality in (43). Thus  $f \circ g$  is  $\tilde{\rho}$ -weakly-convex with  $\tilde{\rho}_g = \sqrt{d} L_2C_1 + \rho_1C_2^2$ .
Proof of Proposition 4.2. Under Assumption 4.1, Proposition D.2 directly implies the  $\rho_F$ -weak-convexity of  $F(\mathbf{w})$  with  $\rho_{F} = \sqrt{d_{1}}\rho_{g}C_{f} + \rho_{f}C_{g}^{2}$ .
Proof of Proposition 4.4. Under Assumption 4.3, we first apply Proposition D.2 to the composite function  $g_{i}(h_{i,j}(\cdot))$  and obtain its  $\rho_{\bar{g}} = \sqrt{d_2} L_hC_g + \rho_gC_h^2$ -weak-convexity. To show it Lipschitz continuity, we use the Lipschitz continuity of  $g_{i}$  and  $h_{i,j}$  to obtain
$$
\left\| g _ {i} \left(h _ {i, j} (\mathbf {w})\right) - g _ {i} \left(h _ {i, j} (\tilde {\mathbf {w}})\right) \right\| ^ {2} \leq C _ {g} ^ {2} C _ {h} ^ {2} \| \mathbf {w} - \tilde {\mathbf {w}} \| ^ {2}.
$$
Thus  $g_{i}(h_{i,j}(\mathbf{w}))$  is  $C_{\tilde{g}} = C_gC_h$ -Lipschitz-continuous
Since we assume  $f_{i}(\cdot)$  is non-decreasing,  $\rho_f$ -weakly-convex and  $C_f$ -Lipschitz continuous, and  $g_{i}(h_{i,j}(\cdot))$  is  $\rho_{\tilde{g}}$ -weakly-convex and  $C_{\tilde{g}}$ -Lipschitz-continuous, we apply Proposition D.2 again to conclude that  $F(\cdot)$  is  $\rho_F = \sqrt{d_1}\rho_{\tilde{g}}C_f + \rho_fC_{\tilde{g}}^2$ -weakly-convex.
D.3 Proof of Lemma 4.5
Proof of Lemma 4.5. With  $\gamma = \frac{n_1 - B_1}{B_1(1 - \tau)} + (1 - \tau)$ ,  $\tau \leq \frac{1}{2}$ , MSVR update gives recursive error bound [15]
$$
\begin{array}{l} \mathbb {E} [ \| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t + 1}) \| ^ {2} ] \\ \leq \left(1 - \frac {B _ {1} \tau}{n _ {1}}\right) \mathbb {E} [ \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} ] + \frac {2 \tau^ {2} B _ {1} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 n _ {1} C _ {g} ^ {2}}{B _ {1}} \mathbb {E} [ \| \mathbf {w} _ {t} - \mathbf {w} _ {t + 1} \| ^ {2} ] \\ \leq \left(1 - \frac {B _ {1} \tau}{n _ {1}}\right) \mathbb {E} \left[ \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} \right] + \frac {2 \tau^ {2} B _ {1} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 n _ {1} C _ {g} ^ {2}}{B _ {1}} \eta^ {2} \mathbb {E} \left[ \| G _ {t} \| ^ {2} \right] \\ \leq \left(1 - \frac {B _ {1} \tau}{2 n _ {1}}\right) ^ {2} \mathbb {E} [ \| u _ {i, t} - g _ {i} (\mathbf {w} _ {t}) \| ^ {2} ] + \frac {2 \tau^ {2} B _ {1} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 n _ {1} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1}} \\ \end{array}
$$
Applying this inequality recursively, we obtain
$$
\begin{array}{l} \mathbb {E} [ \| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t + 1}) \| ^ {2} ] \\ \leq \left(1 - \frac {B _ {1} \tau}{2 n _ {1}}\right) ^ {2 (t + 1)} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| ^ {2} + \sum_ {j = 0} ^ {t} \left(1 - \frac {B _ {1} \tau}{2 n _ {1}}\right) ^ {2 (t - j)} \left(\frac {2 \tau^ {2} B _ {1} \sigma^ {2}}{n _ {1} B _ {2}} + \frac {8 n _ {+} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1}}\right) \\ \leq \left(1 - \frac {B _ {1} \tau}{2 n _ {1}}\right) ^ {2 (t + 1)} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| ^ {2} + \frac {4 \tau \sigma^ {2}}{B _ {2}} + \frac {1 6 n _ {1} ^ {2} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} \tau} \\ \end{array}
$$
where we use  $\sum_{j=0}^{t} \left(1 - \frac{B_1\tau}{2n_1}\right)^{2(t-j)} \leq \frac{2n_1}{B_1\tau}$ .
$$
\begin{array}{l} \mathbb {E} \left[ \left| \left| u _ {i, t + 1} - g _ {i} \left(\mathbf {w} _ {t + 1}\right) \right| \right| \right] ^ {2} \\ \leq \mathbb {E} [ \| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t + 1}) \| ^ {2} ] \\ \leq \left(1 - \frac {B _ {1} \tau}{2 n _ {1}}\right) ^ {2 (t + 1)} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| ^ {2} + \frac {4 \tau \sigma^ {2}}{B _ {2}} + \frac {1 6 n _ {1} ^ {2} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} \tau} \\ \leq \left[ (1 - \frac {B _ {1} \tau}{2 n _ {1}}) ^ {t + 1} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| + \frac {2 \tau^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {4 n _ {1} C _ {g} M \eta}{B _ {1} \tau^ {1 / 2}} \right] ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left\| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t + 1}) \right\| \right] \\ \leq \left(1 - \frac {B _ {1} \tau}{2 n _ {1}}\right) ^ {t + 1} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| + \frac {2 \tau^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {4 n _ {1} C _ {g} M \eta}{B _ {1} \tau^ {1 / 2}} \\ \end{array}
$$
Taking summation over  $i \in S$ , we obtain the desired result
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n} \sum_ {i \in \mathcal {S}} \| u _ {i, t + 1} - g _ {i} (\mathbf {w} _ {t + 1}) \| \right] \\ \leq \left(1 - \frac {B _ {1} \tau}{2 n _ {1}}\right) ^ {t + 1} \frac {1}{n} \sum_ {i \in \mathcal {S}} \| u _ {i, 0} - g _ {i} (\mathbf {w} _ {0}) \| + \frac {2 \tau^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + \frac {4 n C _ {g} M \eta}{B _ {1} \tau^ {1 / 2}} \\ \end{array}
$$
D.4 Proof of Lemma C.6
Proof of Lemma C.6. With  $\gamma_{3} = \frac{n_{+} - B_{1}}{B_{1}(1 - \tau_{2})} + (1 - \tau_{2})$  and  $\tau_{2} \leq \frac{1}{2}$ , MSVR update gives the following recursive error bound [15]
$$
\begin{array}{l} \mathbb {E} \left[ \| u _ {i, t + 1} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {j, t + 1} - v _ {i, t + 1}, s _ {i, t + 1}\right) \| ^ {2} \right] \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{n _ {+}}\right) \mathbb {E} \left[ \| u _ {i, t} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {j, t} - v _ {i, t}, s _ {i, t}\right) \| ^ {2} \right] + \frac {2 \tau_ {2} ^ {2} B _ {1} \sigma^ {2}}{n _ {+} B _ {2}} \\ + \frac {8 n _ {+} C _ {g} ^ {2}}{B _ {1}} \mathbb {E} [ \| (v _ {j, t} - v _ {i, t}, s _ {i, t}) - (v _ {j, t + 1} - v _ {i, t + 1}, s _ {i, t + 1}) \| ^ {2} ] \tag {44} \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{n _ {+}}\right) \mathbb {E} \left[ \| u _ {i, t} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {j, t} - v _ {i, t}, s _ {i, t}\right) \| ^ {2} \right] + \frac {2 \tau_ {2} ^ {2} B _ {1} \sigma^ {2}}{n _ {+} B _ {2}} \\ + \frac {1 6 n _ {+} C _ {g} ^ {2}}{B _ {1}} \mathbb {E} [ \| v _ {i, t} - v _ {i, t + 1} \| ^ {2} + \| v _ {j, t} - v _ {j, t + 1} \| ^ {2} ] + \frac {8 C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1}} \\ \end{array}
$$
It remains to bound  $\mathbb{E}[\| v_{i,t} - v_{i,t + 1}\| ^2 ]$  and  $\mathbb{E}[\| v_{j,t} - v_{j,t + 1}\| ^2 ]$ . We bound the former, and the latter's bound naturally follows. Consider the update of  $v_{i,t + 1}$  and we have
$$
\begin{array}{l} \mathbb {E} \left[ \left\| v _ {i, t} - v _ {i, t + 1} \right\| ^ {2} \right] \\ \leq \mathbb {E} \left[ \frac {B _ {1}}{n _ {+}} \| \tau_ {1} v _ {i, t} - \tau_ {1} h ^ {(i)} \left(\mathbf {w} _ {t}; \mathcal {B} _ {3, i} ^ {t}\right) - \gamma_ {1} \left(h ^ {(i)} \left(\mathbf {w} _ {t}; \mathcal {B} _ {3, i} ^ {t}\right) - h ^ {(i)} \left(\mathbf {w} _ {t - 1}; \mathcal {B} _ {3, i} ^ {t}\right)\right) \| ^ {2} \right] \\ \leq \mathbb {E} \left[ \frac {2 B _ {1} \tau_ {1} ^ {2}}{n _ {+}} \| v _ {i, t} - h ^ {(i)} \left(\mathbf {w} _ {t}; \mathcal {B} _ {3, i} ^ {t}\right) \| ^ {2} + \frac {2 B _ {1} \gamma_ {1} ^ {2}}{n _ {+}} \| h ^ {(i)} \left(\mathbf {w} _ {t}; \mathcal {B} _ {3, i} ^ {t}\right) - h ^ {(i)} \left(\mathbf {w} _ {t - 1}; \mathcal {B} _ {3, i} ^ {t}\right) \| ^ {2} \right] \\ \leq \mathbb {E} \left[ \frac {2 B _ {1} \tau_ {1} ^ {2}}{n _ {+}} \| v _ {i, t} - h ^ {(i)} \left(\mathbf {w} _ {t}; \mathcal {B} _ {3, i} ^ {t}\right) \| ^ {2} + \frac {2 B _ {1} \gamma_ {1} ^ {2} C _ {h}}{n _ {+}} \| \mathbf {w} _ {t} - \mathbf {w} _ {t - 1} \| ^ {2} \right] \\ \stackrel {(a)} {\leq} \frac {8 B _ {1} \tau_ {1} ^ {2} M ^ {2}}{n _ {+}} + \frac {8 n _ {+} C _ {h} ^ {2} \eta^ {2} M ^ {2}}{B _ {1}} \\ \end{array}
$$
where inequality (a) uses  $\tau_{1} \leq 1/2$  and  $\gamma_{1} = \frac{n_{+} - B_{1}}{B_{1}(1 - \tau_{1})} + (1 - \tau_{1}) \leq \frac{2n_{+}}{B_{1}}$ . Plugging the above inequality back into inequality 44 gives
$$
\begin{array}{l} \mathbb {E} \left[ \| u _ {i, t + 1} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {j, t + 1} - v _ {i, t + 1}, s _ {i, t + 1}\right) \| ^ {2} \right] \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{n _ {+}}\right) \mathbb {E} \left[ \| u _ {i, t} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {j, t} - v _ {i, t}, s _ {i, t}\right) \| ^ {2} \right] + \frac {2 \tau_ {2} ^ {2} B _ {1} \sigma^ {2}}{n _ {+} B _ {2}} \\ + \frac {1 6 n _ {+} C _ {g} ^ {2}}{B _ {1}} \left(8 \tau_ {1} ^ {2} M ^ {2} \left(\frac {B _ {1}}{n _ {+}} + \frac {B _ {2}}{n _ {-}}\right) + 8 C _ {h} ^ {2} \eta^ {2} M ^ {2} \left(\frac {n _ {+}}{B _ {1}} + \frac {n _ {-}}{B _ {2}}\right)\right) + \frac {8 C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1}} \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{n _ {+}}\right) \mathbb {E} \left[ \| u _ {i, t} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {j, t} - v _ {i, t}, s _ {i, t}\right) \| ^ {2} \right] + \frac {2 \tau_ {2} ^ {2} B _ {1} \sigma^ {2}}{n _ {+} B _ {2}} \\ + 1 2 8 C _ {g} ^ {2} M ^ {2} \frac {n _ {+}}{B _ {1}} \left(\frac {B _ {1}}{n _ {+}} + \frac {B _ {2}}{n _ {-}}\right) \tau_ {1} ^ {2} + 1 2 8 C _ {g} ^ {2} C _ {h} ^ {2} M ^ {2} \frac {n _ {+}}{B _ {1}} \left(\frac {n _ {+}}{B _ {1}} + \frac {n _ {-}}{B _ {2}}\right) \eta^ {2} + \frac {8 C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1}} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \| u _ {i, t + 1} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {j, t + 1} - v _ {i, t + 1}, s _ {i, t + 1}\right) \| ^ {2} \right] \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{2 n _ {+}}\right) ^ {2 (t + 1)} \| u _ {i, 0} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {i, 0} - v _ {j, 0}, s _ {i, 0}\right) \| ^ {2} + \sum_ {j = 0} ^ {t} \left(1 - \frac {B _ {1} \tau_ {2}}{2 n _ {+}}\right) ^ {2 (t - j)} \left(\frac {2 \tau_ {2} ^ {2} B _ {1} \sigma^ {2}}{n _ {+} B _ {2}}\right) \\ + 1 2 8 C _ {g} ^ {2} M ^ {2} \frac {n _ {+}}{B _ {1}} \left(\frac {B _ {1}}{n _ {+}} + \frac {B _ {2}}{n _ {-}}\right) \tau_ {1} ^ {2} + 1 2 8 C _ {g} ^ {2} C _ {h} ^ {2} M ^ {2} \frac {n _ {+}}{B _ {1}} \left(\frac {n _ {+}}{B _ {1}} + \frac {n _ {-}}{B _ {2}}\right) \eta^ {2} + \frac {8 C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1}} \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{2 n _ {+}}\right) ^ {2 (t + 1)} \| u _ {i, 0} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} (v _ {i, 0} - v _ {j, 0}, s _ {i, 0}) \| ^ {2} + \frac {4 \tau_ {2} \sigma^ {2}}{B _ {2}} \\ + 2 5 6 C _ {g} ^ {2} M ^ {2} \frac {n _ {+} ^ {2}}{B _ {1} ^ {2}} \left(\frac {B _ {1}}{n _ {+}} + \frac {B _ {2}}{n _ {-}}\right) \frac {\tau_ {1} ^ {2}}{\tau_ {2}} + 2 5 6 C _ {g} ^ {2} C _ {h} ^ {2} M ^ {2} \frac {n _ {+} ^ {2}}{B _ {1} ^ {2}} \left(\frac {n _ {+}}{B _ {1}} + \frac {n _ {-}}{B _ {2}}\right) \frac {\eta^ {2}}{\tau_ {2}} + \frac {1 6 n _ {+} C _ {g} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} \tau_ {2}} \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{2 n _ {+}}\right) ^ {2 (t + 1)} \| u _ {i, 0} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {i, 0} - v _ {j, 0}, s _ {i, 0}\right) \| ^ {2} + \frac {4 \tau_ {2} \sigma^ {2}}{B _ {2}} + C _ {2} ^ {2} \frac {n _ {+} ^ {2}}{B _ {1} ^ {2}} \left(\frac {B _ {1}}{n _ {+}} + \frac {B _ {2}}{n _ {-}}\right) \frac {\tau_ {1} ^ {2}}{\tau_ {2}} \\ + C _ {2} ^ {2} \frac {n _ {+} ^ {2}}{B _ {1} ^ {2}} \left(\frac {n _ {+}}{B _ {1}} + \frac {n _ {-}}{B _ {2}}\right) \frac {\eta^ {2}}{\tau_ {2}} + C _ {2} ^ {2} \frac {n _ {+} \eta^ {2}}{B _ {1} ^ {2} \tau_ {2}} \\ \end{array}
$$
where we use  $\sum_{j=0}^{t}(1 - \frac{B_1\tau_2}{2n_+})^{2(t-j)} \leq \frac{2n_+}{B_1\tau_1}$  and denotes  $C_2^2 = 2\max\{256C_g^2M^2, 256C_g^2C_h^2M^2, 16C_g^2M^2\}$ . Taking average over  $i \in S_+$  gives the squared-norm error bound.
To derive the norm error bound, we derive
$$
\begin{array}{l} \mathbb {E} \left[ \| u _ {i, t + 1} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {j, t + 1} - v _ {i, t + 1}, s _ {i, t + 1}\right) \| \right] ^ {2} \\ \leq \mathbb {E} [ \| u _ {i, t + 1} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} (v _ {j, t + 1} - v _ {i, t + 1}, s _ {i, t + 1}) \| ^ {2} ] \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{2 n _ {+}}\right) ^ {2 (t + 1)} \| u _ {i, 0} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {i, 0} - v _ {j, 0}, s _ {i, 0}\right) \| ^ {2} + \frac {4 \tau_ {2} \sigma^ {2}}{B _ {2}} + C _ {2} ^ {2} \frac {n _ {+} ^ {2}}{B _ {1} ^ {2}} \left(\frac {B _ {1}}{n _ {+}} + \frac {B _ {2}}{n _ {-}}\right) \frac {\tau_ {1} ^ {2}}{\tau_ {2}} \\ + C _ {2} ^ {2} \frac {n _ {+} ^ {2}}{B _ {1} ^ {2}} \left(\frac {n _ {+}}{B _ {1}} + \frac {n _ {-}}{B _ {2}}\right) \frac {\eta^ {2}}{\tau_ {2}} + C _ {2} ^ {2} \frac {n _ {+} \eta^ {2}}{B _ {1} ^ {2} \tau_ {2}} \\ \leq \left[ (1 - \frac {B _ {1} \tau_ {2}}{2 n _ {+}}) ^ {t + 1} \| u _ {i, 0} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} (v _ {i, 0} - v _ {j, 0}, s _ {i, 0}) \| + \frac {2 \tau_ {2} ^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + C _ {2} \frac {n _ {+}}{B _ {1}} (\frac {B _ {1} ^ {1 / 2}}{n _ {+} ^ {1 / 2}} + \frac {B _ {2} ^ {1 / 2}}{n _ {-} ^ {1 / 2}}) \frac {\tau_ {1}}{\tau_ {2} ^ {1 / 2}} \right] \\ \left. + C _ {2} \frac {n _ {+}}{B _ {1}} \left(\frac {n _ {+} ^ {1 / 2}}{B _ {1} ^ {1 / 2}} + \frac {n _ {-} ^ {1 / 2}}{B _ {2} ^ {1 / 2}}\right) \frac {\eta}{\tau_ {2} ^ {1 / 2}} + C _ {2} \frac {n _ {+} ^ {1 / 2} \eta}{B _ {1} \tau_ {2} ^ {1 / 2}} \right] ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ \| u _ {i, t + 1} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {j, t + 1} - v _ {i, t + 1}, s _ {i, t + 1}\right) \| ] \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{2 n _ {+}}\right) ^ {t + 1} \| u _ {i, 0} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {i, 0} - v _ {j, 0}, s _ {i, 0}\right) \| + \frac {2 \tau_ {2} ^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} + C _ {2} \frac {n _ {+}}{B _ {1}} \left(\frac {B _ {1} ^ {1 / 2}}{n _ {+} ^ {1 / 2}} + \frac {B _ {2} ^ {1 / 2}}{n _ {-} ^ {1 / 2}}\right) \frac {\tau_ {1}}{\tau_ {2} ^ {1 / 2}} \\ + C _ {2} \frac {n _ {+}}{B _ {1}} \left(\frac {n _ {+} ^ {1 / 2}}{B _ {1} ^ {1 / 2}} + \frac {n _ {-} ^ {1 / 2}}{B _ {2} ^ {1 / 2}}\right) \frac {\eta}{\tau_ {2} ^ {1 / 2}} + C _ {2} \frac {n _ {+} ^ {1 / 2} \eta}{B _ {1} \tau_ {2} ^ {1 / 2}} \\ \end{array}
$$
Taking average over  $i \in S_{+}$ , we obtain the norm error bound
$$
\begin{array}{l} \mathbb {E} \left[ \frac {1}{n _ {+}} \sum_ {i \in S _ {+}} \| u _ {i, t + 1} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} \left(v _ {j, t + 1} - v _ {i, t + 1}, s _ {i, t + 1}\right) \| \right] \\ \leq \left(1 - \frac {B _ {1} \tau_ {2}}{2 n _ {+}}\right) ^ {t + 1} \frac {1}{n _ {+}} \sum_ {i \in S _ {+}} \| u _ {i, 0} - \frac {1}{n _ {-}} \sum_ {j \in S _ {-}} g _ {i} (v _ {i, 0} - v _ {j, 0}, s _ {i, 0}) \| + \frac {2 \tau_ {2} ^ {1 / 2} \sigma}{B _ {2} ^ {1 / 2}} \\ + C _ {2} \frac {n _ {+}}{B _ {1}} \left(\frac {B _ {1} ^ {1 / 2}}{n _ {+} ^ {1 / 2}} + \frac {B _ {2} ^ {1 / 2}}{n _ {-} ^ {1 / 2}}\right) \frac {\tau_ {1}}{\tau_ {2} ^ {1 / 2}} + C _ {2} \frac {n _ {+}}{B _ {1}} \left(\frac {n _ {+} ^ {1 / 2}}{B _ {1} ^ {1 / 2}} + \frac {n _ {-} ^ {1 / 2}}{B _ {2} ^ {1 / 2}}\right) \frac {\eta}{\tau_ {2} ^ {1 / 2}} + C _ {2} \frac {n _ {+} ^ {1 / 2} \eta}{B _ {1} \tau_ {2} ^ {1 / 2}} \\ \end{array}
$$
D.5 Proof of Lemma A.3
Proof of Lemma A.3. With  $\gamma_{1} = \frac{n_{1}n_{2} - B_{1}B_{2}}{B_{1}B_{2}(1 - \tau_{1})} + (1 - \tau_{1})$  and  $\tau_{1} \leq \frac{1}{2}$ , MSVR update has the following recursive error bound [15][15]
$$
\begin{array}{l} \mathbb {E} [ \| v _ {i, j, t + 1} - h _ {i, j} (\mathbf {w} _ {t + 1}) \| ^ {2} ] \\ \leq (1 - \frac {B _ {1} B _ {2} \tau_ {1}}{n _ {1} n _ {2}}) \mathbb {E} [ \| v _ {i, j, t} - h _ {i, j} (\mathbf {w} _ {t}) \| ^ {2} ] + \frac {2 \tau_ {1} ^ {2} B _ {1} B _ {2} \sigma^ {2}}{n _ {1} n _ {2} B _ {3}} + \frac {8 n _ {1} n _ {2} C _ {h} ^ {2}}{B _ {1} B _ {2}} \mathbb {E} [ \| \mathbf {w} _ {t} - \mathbf {w} _ {t + 1} \| ^ {2} ] \\ \leq (1 - \frac {B _ {1} B _ {2} \tau_ {1}}{2 n _ {1} n _ {2}}) ^ {2} \mathbb {E} [ \| v _ {i, j, t} - h _ {i, j} (\mathbf {w} _ {t}) \| ^ {2} ] + \frac {2 \tau_ {1} ^ {2} B _ {1} B _ {2} \sigma^ {2}}{n _ {1} n _ {2} B _ {3}} + \frac {8 n _ {1} n _ {2} C _ {h} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} B _ {2}} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ \| v _ {i, j, t + 1} - h _ {i, j} (\mathbf {w} _ {t + 1}) \| ^ {2} ] \\ \leq \left(1 - \frac {B _ {1} B _ {2} \tau_ {1}}{2 n _ {1} n _ {2}}\right) ^ {2 (t + 1)} \| v _ {i, j, 0} - h _ {i, j} (\mathbf {w} _ {0}) \| ^ {2} + \sum_ {j = 0} ^ {t} (1 - \frac {B _ {1} B _ {2} \tau_ {1}}{2 n _ {1} n _ {2}}) ^ {2 (t - j)} (\frac {2 \tau_ {1} ^ {2} B _ {1} B _ {2} \sigma^ {2}}{n _ {1} n _ {2} B _ {3}} + \frac {8 n _ {1} n _ {2} C _ {h} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} B _ {2}}) \\ \leq \left(1 - \frac {B _ {1} B _ {2} \tau_ {1}}{2 n _ {1} n _ {2}}\right) ^ {2 (t + 1)} \| v _ {i, j, 0} - h _ {i, j} (\mathbf {w} _ {0}) \| ^ {2} + \frac {4 \tau_ {1} \sigma^ {2}}{B _ {3}} + \frac {1 6 n _ {1} ^ {2} n _ {2} ^ {2} C _ {h} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} B _ {2} ^ {2} \tau_ {1}} \\ \end{array}
$$
where we use  $\sum_{j=0}^{t}(1 - \frac{B_1B_2\tau_1}{2n_1n_2})^{2(t-j)} \leq \frac{2n_1n_2}{B_1B_2\tau_1}$ . Taking average over  $(i,j) \in S_1 \times S_2$  gives the squared-norm error bound.
$$
\begin{array}{l} \mathbb {E} \left[ \left\| v _ {i, j, t + 1} - h _ {i, j} \left(\mathbf {w} _ {t + 1}\right) \right\| \right] ^ {2} \\ \leq \mathbb {E} [ \| v _ {i, j, t + 1} - h _ {i, j} (\mathbf {w} _ {t + 1}) \| ^ {2} ] \\ \leq \left(1 - \frac {B _ {1} B _ {2} \tau_ {1}}{2 n _ {1} n _ {2}}\right) ^ {2 (t + 1)} \| v _ {i, j, 0} - h _ {i, j} (\mathbf {w} _ {0}) \| ^ {2} + \frac {4 \tau_ {1} \sigma^ {2}}{B _ {3}} + \frac {1 6 n _ {1} ^ {2} n _ {2} ^ {2} C _ {h} ^ {2} M ^ {2} \eta^ {2}}{B _ {1} ^ {2} B _ {2} ^ {2} \tau_ {1}} \\ \leq \left[ (1 - \frac {B _ {1} B _ {2} \tau_ {1}}{2 n _ {1} n _ {2}}) ^ {t + 1} \| v _ {i, j, 0} - h _ {i, j} (\mathbf {w} _ {0}) \| + \frac {2 \tau_ {1} ^ {1 / 2} \sigma}{B _ {3} ^ {1 / 2}} + \frac {4 n _ {1} n _ {2} C _ {h} M \eta}{B _ {1} B _ {2} \tau_ {1} ^ {1 / 2}} \right] ^ {2} \\ \end{array}
$$