SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\stackrel {(1)} {=} 2 \left\| \left(\eta N K \sum_ {j = 1} ^ {N} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \sum_ {j = 1} ^ {N} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r})\right) \right\| ^ {2} + 2 \left\| \sum_ {n = 1} ^ {N - 1} \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) \boldsymbol {Q} _ {n + 1} \mathbf {H} _ {n + 1} \sum_ {j = 1} ^ {n} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| ^ {2}
$$
$$
\begin{array}{l} \stackrel {(2)} {\leq} \left(2 0 \eta^ {2} N ^ {2} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2} + 3 6 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4}\right) \| \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \| ^ {2} + 2 5 6 \eta^ {2} N ^ {3} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2} G ^ {2} \log (1 / p) \\ + 2 4 4 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4} G ^ {2} N \log (1 / p) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(3)} {\leq} \left(2 0 \eta^ {2} N ^ {2} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2} + 3 6 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4}\right) 2 L \left(\Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) - \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}))\right) \\ + \left(2 4 4 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4} + 2 5 6 \eta^ {2} N ^ {3} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2}\right) G ^ {2} N \log (1 / p), \\ \end{array}
$$
where in (1) we apply Jensen's inequality, in (2) we plug in Lemma 11 (a), and Lemma 12, and in (3) we use the  $L$ -smoothness of  $\Phi$ . Plugging above bound back in (19) yields:
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} \\ \leq (1 - \frac {1}{2} \mu \eta N K) \mathbb {E} \| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \| ^ {2} + \eta^ {2} N ^ {2} K ^ {2} e ^ {4} \delta^ {2} \\ - \underbrace {\left(\eta N K - 4 \eta^ {2} N ^ {2} K ^ {2} L - \left(\frac {1}{2 \mu \eta N K} + 2\right) \left(2 0 \eta^ {2} N ^ {2} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2} - 3 6 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4}\right)\right)} _ {T _ {1}} \mathbb {E} [ \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) - \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}})) ] \\ + \left(\frac {1}{2 \mu \eta N K} + 2\right) \left(2 4 4 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4} + 2 5 6 \eta^ {2} N ^ {3} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2}\right) G ^ {2} N \log (1 / p). \\ \end{array}
$$
Since we choose  $\eta = \frac{4\log(\sqrt{NK}R)}{\mu NKR}$ , and large enough epoch number:
$$
R \geq \max  \left\{\left(\frac {4 0}{\mu} + 1\right) e, 1 6 \log (\sqrt {N K} R), 6 4 \kappa \log (\sqrt {N K} R) \right\},
$$
we know that  $T_{1} \leq 0$ . We thus have:
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} \\ \leq \left(1 - \frac {1}{2} \mu \eta N K\right) \mathbb {E} \left\| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} + \eta^ {2} N ^ {2} K ^ {2} e ^ {4} \delta^ {2} \\ + \left(\frac {1}{2 \mu \eta N K} + 2\right) \left(2 4 4 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4} + 2 5 6 \eta^ {2} N ^ {3} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2}\right) G ^ {2} N \log (1 / p) \\ \end{array}
$$
Unrolling the recursion from  $r = R$  to 0:
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {v} ^ {R} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} \\ \leq \left(1 - \frac {1}{2} \mu \eta N K\right) ^ {R} \mathbb {E} \left\| \boldsymbol {v} ^ {0} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} + \frac {2}{\mu} \eta N K e ^ {4} \delta^ {2} \\ + \frac {1}{\mu} \left(\frac {1}{2 \mu \eta N K} + 2\right) \left(4 8 8 e ^ {6} \eta^ {3} N ^ {3} K ^ {3} L ^ {4} + 5 1 2 \eta N ^ {2} K \left(\frac {e}{4 R - e}\right) ^ {2}\right) G ^ {2} N \log (1 / p). \\ \end{array}
$$
Plugging in our choice of  $\eta$  will conclude the proof:
$$
\mathbb {E} \left\| \boldsymbol {v} ^ {R} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} \leq \tilde {O} \left(\frac {\mathbb {E} \left\| \boldsymbol {v} ^ {0} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2}}{N K R ^ {2}} + \frac {\delta^ {2}}{\mu^ {2} R} + \left(\frac {L ^ {4} + N}{\mu^ {4} R ^ {2}}\right) G ^ {2} N \log (1 / p)\right).
$$
Finally, according to Lemma 2 we can complete the proof:
$$
\begin{array}{l} \Phi \left(\boldsymbol {\alpha} _ {i} ^ {*}, \hat {\boldsymbol {v}} _ {i}\right) - \Phi \left(\boldsymbol {\alpha} _ {i} ^ {*}, \boldsymbol {v} _ {i} ^ {*}\right) \leq 2 L \left\| \boldsymbol {v} _ {i} ^ {R} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}} _ {i}) \right\| ^ {2} + \left(2 \kappa_ {\Phi} ^ {2} L + \frac {4 N G ^ {2}}{L}\right) \| \hat {\boldsymbol {\alpha}} _ {i} - \boldsymbol {\alpha} ^ {*} \| ^ {2} \\ \leq \tilde {O} \left(\frac {L D ^ {2}}{N K R ^ {2}} + \frac {L \delta^ {2}}{\mu^ {2} R} + \left(\frac {L ^ {4} + N}{\mu^ {4} R ^ {2}}\right) L G ^ {2} N \log (1 / p)\right) \\ + \kappa_ {\Phi} ^ {2} L \tilde {O} \left(\exp \left(- \frac {T _ {\boldsymbol {\alpha}}}{\kappa_ {g}}\right) + \kappa_ {g} ^ {2} \bar {\zeta} _ {i} (\boldsymbol {w} ^ {*}) L ^ {2} \left(\frac {D ^ {2}}{R K} + \frac {\kappa \zeta^ {2}}{\mu^ {2} R ^ {2}} + \frac {\delta^ {2}}{\mu^ {2} N R K}\right)\right), \\ \end{array}
$$
where we plug in the convergence result from Theorem 1 at last step.
C Proof of Convergence of Single Loop Algorithm
In this section, we turn to presenting the proof of single loop PERM algorithm (Algorithm 2) where the learning of mixing parameters and personalized models are coupled. Compared to Algorithm A2, here during the optimization of model, the mixing parameters are also being updated. As a result, we need to decouple the two updates which makes the analysis more involved. We begin with some technical lemmas that support the proof of main result.
C.1 Technical Lemmas
Proposition 4 (Basic Properties of SGD on Smooth Strongly Convex Function). Let  $\boldsymbol{w}^t$  to be the  $t$ th iterate of minibatch SGD on smooth and strongly convex function  $F$ , with minibatch size  $M$  and learning rate  $\gamma$ . Also assume the variance is bounded by  $\delta$ . Then the following statements hold true after  $T$  iterations of SGD:
$$
\mathbb {E} \| \nabla F (\boldsymbol {w} ^ {T}) \| ^ {2} \leq 2 L (1 - \mu \gamma) ^ {T} (F (\boldsymbol {w} ^ {0}) - F (\boldsymbol {w} ^ {*})) + \frac {2 \gamma \kappa \delta^ {2}}{M} \tag {15}
$$
$$
\mathbb {E} \left\| \boldsymbol {w} ^ {T + 1} - \boldsymbol {w} ^ {T} \right\| ^ {2} \leq 2 \gamma^ {2} L \left(1 - \mu \gamma\right) ^ {T} \left(F \left(\boldsymbol {w} ^ {0}\right) - F \left(\boldsymbol {w} ^ {*}\right)\right) + \frac {2 \gamma^ {3} \kappa \delta^ {2}}{M} + \frac {\gamma^ {2} \delta^ {2}}{M} \tag {16}
$$
$$
\mathbb {E} \left\| \boldsymbol {w} ^ {T} - \boldsymbol {w} ^ {*} \right\| ^ {2} \leq \frac {2}{\mu} (1 - \mu \gamma) ^ {T} (F (\boldsymbol {w} ^ {0}) - F (\boldsymbol {w} ^ {*})) + 2 \gamma \frac {\delta^ {2}}{\mu^ {2} M}. \tag {17}
$$
Lemma 14 (Bounded iterates difference of  $\alpha$ ). Let  $\{\alpha_i^r\}$  be iterates generated by Algorithm 2, then under conditions of Theorem 3, the following statement holds:
$$
\| \boldsymbol {\alpha} _ {i} ^ {r} - \boldsymbol {\alpha} _ {i} ^ {r - 1} \| ^ {2} \leq 6 \left(1 - \frac {1}{\kappa_ {g}}\right) ^ {T _ {\boldsymbol {\alpha}}} + O \left(\kappa_ {g} ^ {2} L ^ {2} \bar {\zeta_ {i}} (\boldsymbol {w} ^ {*})\right) \left(\gamma^ {2} L (1 - \mu \gamma) ^ {r} (F (\boldsymbol {w} ^ {0}) - F (\boldsymbol {w} ^ {*})) + \frac {\gamma^ {3} \kappa \delta^ {2}}{M} + \frac {\gamma^ {2} \delta^ {2}}{M}\right)
$$
$$
\boldsymbol {z} ^ {r} = \left[ \| \nabla f _ {i} (\boldsymbol {w} ^ {r}) - \nabla f _ {1} (\boldsymbol {w} ^ {r}) \| ^ {2}, \dots , \| \nabla f _ {i} (\boldsymbol {w} ^ {r}) - \nabla f _ {N} (\boldsymbol {w} ^ {r}) \| ^ {2} \right].
$$
According to updating rule of  $\alpha$  in Algorithm 2 and Lemma 3 we have:
$$
\begin{array}{l} \left\| \boldsymbol {\alpha} _ {i} ^ {r} - \boldsymbol {\alpha} _ {i} ^ {r - 1} \right\| ^ {2} \leq 3 \left\| \boldsymbol {\alpha} _ {i} ^ {r} - \boldsymbol {\alpha} _ {g _ {i}} ^ {*} (\boldsymbol {w} ^ {r}) \right\| ^ {2} + 3 \left\| \boldsymbol {\alpha} _ {g _ {i}} ^ {*} (\boldsymbol {w} ^ {r - 1}) - \boldsymbol {\alpha} _ {g _ {i}} ^ {*} (\boldsymbol {w} ^ {r}) \right\| ^ {2} + 3 \left\| \boldsymbol {\alpha} _ {g _ {i}} ^ {*} (\boldsymbol {w} ^ {r - 1}) - \boldsymbol {\alpha} _ {i} ^ {r - 1} \right\| ^ {2} \\ \leq 6 \left(1 - \mu_ {g} \eta_ {\alpha}\right) ^ {T _ {\alpha}} + 3 \left\| \boldsymbol {\alpha} _ {g _ {i}} ^ {*} (\boldsymbol {w} ^ {r - 1}) - \boldsymbol {\alpha} _ {g _ {i}} ^ {*} (\boldsymbol {w} ^ {r}) \right\| ^ {2} \\ \leq 6 \left(1 - \mu_ {g} \eta_ {\alpha}\right) ^ {T _ {\alpha}} + 3 \kappa_ {g} ^ {2} \left\| z ^ {r - 1} - z ^ {r} \right\| ^ {2} \\ \leq 6 \left(1 - \mu_ {g} \eta_ {\boldsymbol {\alpha}}\right) ^ {T _ {\boldsymbol {\alpha}}} + 3 \kappa_ {g} ^ {2} \sum_ {j = 1} ^ {N} \left\| \nabla f _ {i} (\boldsymbol {w} ^ {r}) - \nabla f _ {j} (\boldsymbol {w} ^ {r}) + \nabla f _ {i} (\boldsymbol {w} ^ {r - 1}) - \nabla f _ {j} (\boldsymbol {w} ^ {r - 1}) \right\| ^ {2} 4 L ^ {2} \left\| \boldsymbol {w} ^ {r} - \boldsymbol {w} ^ {r - 1} \right\| ^ {2} \\ \end{array}
$$
where the third inequality follows from (8). Since  $\| \nabla f_i(\pmb{w}^r) - \nabla f_j(\pmb{w}^r)\| \leq$ $\| \nabla f_i(\pmb{w}^*) - \nabla f_j(\pmb{w}^*)\| + 2L\| \pmb{w}^r - \pmb{w}^*\|$ , we can conclude that
$$
\begin{array}{l} \left\| \boldsymbol {\alpha} _ {i} ^ {r} - \boldsymbol {\alpha} _ {i} ^ {r - 1} \right\| ^ {2} \leq 6 \left(1 - \mu_ {g} \eta_ {\boldsymbol {\alpha}}\right) ^ {T _ {\boldsymbol {\alpha}}} \\ + 1 2 L ^ {2} \kappa_ {g} ^ {2} \sum_ {j = 1} ^ {N} \left(8 \| \nabla f _ {i} (\boldsymbol {w} ^ {*}) - \nabla f _ {j} (\boldsymbol {w} ^ {*}) \| ^ {2} + 8 L ^ {2} \| \boldsymbol {w} ^ {r} - \boldsymbol {w} ^ {*} \| ^ {2} + 8 L ^ {2} \| \boldsymbol {w} ^ {r - 1} - \boldsymbol {w} ^ {*} \| ^ {2}\right) \| \boldsymbol {w} ^ {r} - \boldsymbol {w} ^ {r - 1} \| ^ {2} \\ \leq 6 \left(1 - \frac {1}{\kappa_ {g}}\right) ^ {T _ {\alpha}} + O \left(\kappa_ {g} ^ {2} L ^ {2} \bar {\zeta} _ {i} (\boldsymbol {w} ^ {*}) \left\| \boldsymbol {w} ^ {r} - \boldsymbol {w} ^ {r - 1} \right\| ^ {2}\right) \\ \leq 6 \left(1 - \frac {1}{\kappa_ {g}}\right) ^ {T _ {\alpha}} + O \left(\kappa_ {g} ^ {2} L ^ {2} \bar {\zeta} _ {i} (\boldsymbol {w} ^ {*})\right) \left(\gamma^ {2} L \left(1 - \mu \gamma\right) ^ {r} \left(F (\boldsymbol {w} ^ {0}) - F (\boldsymbol {w} ^ {*})\right) + \frac {\gamma^ {3} \kappa \delta^ {2}}{M} + \frac {\gamma^ {2} \delta^ {2}}{M}\right) \\ \end{array}
$$
where at last step we plug in Proposition 4 (16).
Lemma 15 (Convergence of  $\alpha$ ). Let  $\{\hat{\alpha}_i\}_{i=1}^N$  be the mixing parameters generated by Algorithm 2. Then under the conditions of Theorem 3, the following statement holds:
$$
\left\| \hat {\boldsymbol {\alpha}} _ {i} - \boldsymbol {\alpha} ^ {*} \right\| ^ {2} \leq 2 \left(1 - \frac {1}{\kappa_ {g}}\right) ^ {T _ {\boldsymbol {\alpha}}} + O \left(\kappa_ {g} ^ {2} \bar {\zeta_ {i}} (\boldsymbol {w} ^ {*}) L ^ {2} \frac {2}{\mu} \left(1 - \mu \gamma\right) ^ {T} + 2 \gamma \frac {\delta^ {2}}{\mu^ {2} M}\right), i \in [ N ]
$$
Proof. We notice the following decomposition:
$$
\begin{array}{l} \left\| \hat {\boldsymbol {\alpha}} _ {i} - \boldsymbol {\alpha} ^ {*} \right\| ^ {2} = \left\| \boldsymbol {\alpha} _ {i} ^ {R} - \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {*}) \right\| ^ {2} \\ \leq 2 \left\| \boldsymbol {\alpha} _ {i} ^ {R} - \boldsymbol {\alpha} _ {g} ^ {*} (\boldsymbol {w} ^ {R}) \right\| ^ {2} + 2 \left\| \boldsymbol {\alpha} _ {g _ {i}} ^ {*} (\boldsymbol {w} ^ {R}) - \boldsymbol {\alpha} _ {g _ {i}} ^ {*} (\boldsymbol {w} ^ {*}) \right\| ^ {2} \\ \leq 2 \left(1 - \frac {1}{\kappa_ {g}}\right) ^ {T _ {\alpha}} + O \left(\kappa_ {g} ^ {2} \left(\bar {\zeta} _ {i} (\boldsymbol {w} ^ {*}) + N L ^ {2} \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2}\right) 4 L \left\| \boldsymbol {w} ^ {R} - \boldsymbol {w} ^ {*} \right\| ^ {2}\right) \\ \leq 2 \left(1 - \frac {1}{\kappa_ {g}}\right) ^ {T _ {\alpha}} + O \left(\kappa_ {g} ^ {2} \bar {\zeta} _ {i} (\boldsymbol {w} ^ {*}) L ^ {2} \frac {2}{\mu} (1 - \mu \gamma) ^ {T} + 2 \gamma \frac {\delta^ {2}}{\mu^ {2} M}\right), \\ \end{array}
$$
where in the second inequality we apply Lemma 3, and in the third inequality we use Proposition 4 (17).
Proof. According to Lemma 2, we have:
$$
\Phi (\boldsymbol {\alpha} _ {i} ^ {*}, \hat {\boldsymbol {v}} _ {i}) - \Phi (\boldsymbol {\alpha} _ {i} ^ {*}, \boldsymbol {v} _ {i} ^ {*}) \leq 2 L \left\| \boldsymbol {v} _ {i} ^ {R} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}} _ {i}) \right\| ^ {2} + \left(2 \kappa_ {\Phi} ^ {2} L + \frac {4 N G ^ {2}}{L}\right) \| \hat {\boldsymbol {\alpha}} _ {i} - \boldsymbol {\alpha} _ {i} ^ {*} \| ^ {2}.
$$
We first examine the convergence of  $\left\| \pmb{v}_i^R - \pmb{v}^*(\hat{\alpha}_i) \right\|^2$ . Applying Cauchy-Schwartz inequality yields:
$$
\begin{array}{l} \left\| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r + 1}) \right\| ^ {2} \leq \left(1 + \frac {1}{4 a - 2}\right) \left\| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \right\| ^ {2} + (1 + 4 a - 2) \left\| \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r + 1}) - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \right\| ^ {2} \\ \leq \left(1 + \frac {1}{4 a - 2}\right) \| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} \left(\boldsymbol {\alpha} ^ {r}\right) \| ^ {2} + (1 + 4 a - 2) \kappa_ {\Phi} ^ {2} \| \boldsymbol {\alpha} ^ {r + 1} - \boldsymbol {\alpha} ^ {r} \| ^ {2} \tag {18} \\ \end{array}
$$
where  $a = \frac{1}{\mu\eta NK}$ , and last step is due to that  $\pmb{v}^{*}(\pmb{\alpha})$  is  $\kappa_{\Phi} := \frac{\sqrt{N}G}{\mu}$  Lipschitz, as proven in Lemma 2. Similar to the proof of Theorem 2, we first define
$$
\boldsymbol {g} ^ {r} := \sum_ {j = 1} ^ {N} \prod_ {j ^ {\prime} = N - 1} ^ {j + 1} (\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}) \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}),
$$
$$
\boldsymbol {\delta} ^ {r} := \sum_ {j = 1} ^ {N} \prod_ {j ^ {\prime} = N - 1} ^ {j + 1} (\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}) \boldsymbol {\delta} _ {j}.
$$
Then we recall the updating rule of  $\pmb{v}$ :
$$
\boldsymbol {v} ^ {r + 1} = \mathcal {P} _ {\mathcal {W}} \left(\boldsymbol {v} ^ {r} - \boldsymbol {g} ^ {r} - \boldsymbol {\delta} ^ {r}\right).
$$
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \right\| ^ {2} = \mathbb {E} \left\| \mathcal {P} _ {\mathcal {W}} \left(\boldsymbol {v} ^ {r} - \boldsymbol {g} ^ {r} - \boldsymbol {\delta} ^ {r} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r})\right) \right\| ^ {2} \\ \leq \mathbb {E} \| \boldsymbol {v} ^ {r} - \boldsymbol {g} ^ {r} - \boldsymbol {\delta} ^ {r} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \| ^ {2} \\ \leq \mathbb {E} \| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \| ^ {2} - 2 \mathbb {E} \langle \boldsymbol {g} ^ {r}, \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \rangle + \mathbb {E} \| \boldsymbol {g} ^ {r} \| ^ {2} + \mathbb {E} \| \boldsymbol {\delta} ^ {r} \| ^ {2} \\ \leq \mathbb {E} \| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \| ^ {2} - 2 \mathbb {E} \langle \eta N K \nabla \Phi (\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {r}), \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \rangle \\ - 2 \mathbb {E} \langle \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {r}), \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \rangle + \mathbb {E} \| \boldsymbol {g} ^ {r} \| ^ {2} + \mathbb {E} \| \boldsymbol {\delta} ^ {r} \| ^ {2}. \\ \end{array}
$$
Now, applying strongly convexity of  $\Phi (\alpha^r,\cdot)$  and Cauchy-Schwartz inequality yields:
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \right\| ^ {2} \leq (1 - \mu \eta N K) \mathbb {E} \left\| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \right\| ^ {2} - \eta N K \mathbb {E} [ \Phi (\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {r}) - \Phi (\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}})) ] \\ + \frac {1}{2} \left(\frac {1}{\mu \eta N K} \mathbb {E} \| \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\hat {\alpha}, \boldsymbol {v} ^ {r}) \| ^ {2} + \mu \eta N K \mathbb {E} \| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\hat {\alpha}) \| ^ {2}\right) \\ + \mathbb {E} \| \boldsymbol {g} ^ {r} \| ^ {2} + \mathbb {E} \| \boldsymbol {\delta} ^ {r} \| ^ {2} \\ \leq \left(1 - \frac {1}{2} \mu \eta N K\right) \mathbb {E} \left\| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \right\| ^ {2} - \eta N K \mathbb {E} \left[ \Phi \left(\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {r}\right) - \Phi \left(\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}})\right) \right] \\ + \frac {1}{2 \mu \eta N K} \mathbb {E} \| \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {r}) \| ^ {2} \\ + 2 \mathbb {E} \| \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {r}) \| ^ {2} + 2 \mathbb {E} \| \eta N K \nabla \Phi (\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {r}) \| ^ {2} + \mathbb {E} \| \boldsymbol {\delta} ^ {r} \| ^ {2}. \\ \end{array}
$$
where in the first inequality we applied Cauchy-Schwartz inequality and strongly convexity. Since  $\Phi (\pmb{\alpha}^r,\cdot)$  is  $L$  smooth, we have:  $\mathbb{E}\left\| \nabla \Phi (\pmb {\alpha}^r,\pmb {v}^r)\right\| ^2\leq 2L\mathbb{E}[\Phi (\pmb {\alpha}^r,\pmb {v}^r) - \Phi (\pmb {\alpha}^r,\pmb {v}^* (\pmb {\alpha}^r))]$ . Therefore, it follows that:
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}) \right\| ^ {2} \leq \left(1 - \frac {1}{2} \mu \eta N K\right) \mathbb {E} \left\| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \right\| ^ {2} - (\eta N K - 4 \eta^ {2} N ^ {2} K ^ {2} L) \mathbb {E} [ \Phi (\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {r}) - \Phi (\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r})) ] \\ + \left(\frac {1}{2 \mu \eta N K} + 2\right) \mathbb {E} \| \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {r}) \| ^ {2} + \mathbb {E} \| \boldsymbol {\delta} ^ {r} \| ^ {2} \tag {19} \\ \end{array}
$$
Now, we examine the term  $\| \pmb{g}^r - \eta NK \nabla \Phi(\pmb{\alpha}^r, \pmb{v}^r) \|^2$  in the right hand side of abovee inequality. First according to summation by part (Lemma 8): we let  $A_j := \prod_{j' = N-1}^{j+1} (\mathbf{I} - Q_{j'} \mathbf{H}_{j'})$  and  $\mathbf{B}_j = Q_j \nabla f_{\sigma(j)}(\pmb{v}^r)$ , then we have:
$$
\begin{array}{l} \boldsymbol {g} ^ {r} = \sum_ {j = 1} ^ {N} \prod_ {j ^ {\prime} = N} ^ {j + 1} (\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}) \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \\ = \sum_ {j = 1} ^ {N} \boldsymbol {A} _ {j} \mathbf {B} _ {j} = \sum_ {j = 1} ^ {N} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \sum_ {n = 1} ^ {N - 1} \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right) - \prod_ {j ^ {\prime} = N} ^ {n + 1} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) \sum_ {j = 1} ^ {n} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \\ = \sum_ {j = 1} ^ {N} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \sum_ {n = 1} ^ {N - 1} \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - Q _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) Q _ {n + 1} \mathbf {H} _ {n + 1} \sum_ {j = 1} ^ {n} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}). \\ \end{array}
$$
$$
\left\| \boldsymbol {g} ^ {r} - \eta N K \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) \right\| ^ {2}
$$
$$
= \left\| \eta N K \nabla \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) - \sum_ {j = 1} ^ {N} \prod_ {j ^ {\prime} = N - 1} ^ {j + 1} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right) \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| ^ {2}
$$
$$
= \left\| \eta N K \sum_ {j = 1} ^ {N} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \left(\sum_ {j = 1} ^ {N} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \sum_ {n = 1} ^ {N - 1} \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - \boldsymbol {Q} _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) \boldsymbol {Q} _ {n + 1} \mathbf {H} _ {n + 1} \sum_ {j = 0} ^ {n} \boldsymbol {Q} _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r})\right) \right\| ^ {2}
$$
$$
\stackrel {(1)} {\leq} 2 \left\| \left(\eta N K \sum_ {j = 1} ^ {N} \hat {\alpha} (\sigma (j)) \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) - \sum_ {j = 1} ^ {N} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r})\right) \right\| ^ {2}
$$
$$
+ 2 \left\| \sum_ {n = 1} ^ {N - 1} \left(\prod_ {j ^ {\prime} = N} ^ {n + 2} \left(\mathbf {I} - Q _ {j ^ {\prime}} \mathbf {H} _ {j ^ {\prime}}\right)\right) Q _ {n + 1} \mathbf {H} _ {n + 1} \sum_ {j = 1} ^ {n} Q _ {j} \nabla f _ {\sigma (j)} (\boldsymbol {v} ^ {r}) \right\| ^ {2}
$$
$$
\begin{array}{l} \stackrel {(3)} {\leq} \left(2 0 \eta^ {2} N ^ {2} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2} + 3 6 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4}\right) 2 L \left(\Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {r}) - \Phi (\hat {\boldsymbol {\alpha}}, \boldsymbol {v} ^ {*} (\hat {\boldsymbol {\alpha}}))\right) \\ + \left(2 4 4 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4} + 2 5 6 \eta^ {2} N ^ {3} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2}\right) G ^ {2} N \log (1 / p) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {v} ^ {r + 1} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \right\| ^ {2} \\ \leq (1 - \frac {1}{2} \mu \eta N K) \mathbb {E} \| \boldsymbol {v} ^ {r} - \boldsymbol {v} ^ {*} (\boldsymbol {\alpha} ^ {r}) \| ^ {2} + \eta^ {2} N ^ {2} K ^ {2} e ^ {4} \delta^ {2} \\ \left. - \left(\eta N K - 4 \eta^ {2} N ^ {2} K ^ {2} L - \left(\frac {1}{2 \mu \eta N K} + 2\right) \left(2 0 \eta^ {2} N ^ {2} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2} - 3 6 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4}\right)\right) \right. \\ \times \mathbb {E} \left[ \Phi \left(\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {r}\right) - \Phi \left(\boldsymbol {\alpha} ^ {r}, \boldsymbol {v} ^ {*} \left(\boldsymbol {\alpha} ^ {r}\right)\right) \right] \\ + \left(\frac {1}{2 \mu \eta N K} + 2\right) \left(2 4 4 e ^ {6} \eta^ {4} N ^ {4} K ^ {4} L ^ {4} + 2 5 6 \eta^ {2} N ^ {3} K ^ {2} \left(\frac {e}{4 R - e}\right) ^ {2}\right) G ^ {2} N \log (1 / p). \\ \end{array}
$$
Since we choose  $\eta = \frac{4\log(\sqrt{NKR})}{\mu NKR}$ , and
$$
R \geq \max  \left\{\frac {3}{8} e, \sqrt [ 3 ]{\frac {6 4 \kappa^ {2} \log (\sqrt {N K} R) e ^ {6}}{9 \mu}} \right\},
$$
hence we have: