SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The work of YD and MM was partially supported by NSF CAREER Award # 2239374 and CNS Award #1956276.
[1] The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/.
[2] F. Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research, 18(1):629-681, 2017.
[3] P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
[4] P. L. Bartlett, A. Montanari, and A. Rakhlin. Deep learning: a statistical viewpoint. Acta Numerica, 2021.
[5] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine Learning, 79(1-2):151-175, 2010.
[6] N. Cesa-Bianchi and G. Lugosi. Prediction, learning, and games. Cambridge University Press, 2006.
[7] T. Chen, Y. Sun, and W. Yin. Solving stochastic compositional optimization is nearly as easy as solving stochastic optimization. IEEE Transactions on Signal Processing, 69:4937-4948, 2021.
[8] F. Cucker and D. X. Zhou. Learning theory: an approximation theory viewpoint, volume 24. Cambridge University Press, 2007.
[9] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054, 2018.
[10] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML), pages 1126–1135. PMLR, 2017.
[11] L. Györfi, M. Köhler, A. Krzyżak, and H. Walk. A distribution-free theory of nonparametric regression. Springer, 2006.
[12] S. Hanneke and S. Kpotufe. A no-free-lunch theorem for multitask learning. The Annals of Statistics, 50(6):3119-3143, 2022.
[13] E. Hazan and N. Megiddo. Online Learning with Prior Knowledge. In Learning Theory, pages 499-513. Springer, 2007.
[14] T. Hu, W. Wang, C. Lin, and G. Cheng. Regularization matters: A nonparametric perspective on overparametrized neural network. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 829-837. PMLR, 2021.
[15] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: convergence and generalization in neural networks. In Conference on Neural Information Processing Systems, 2018.
[16] J. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547, 2019.
[17] N. Konstantinov and C. H. Lampert. Robust learning from untrusted sources. In International Conference on Machine Learning (ICML), pages 3488-3498. PMLR, 2019.
[18] S. Kpotufe and F. Orabona. Regression-tree tuning in a streaming setting. Conference on Neural Information Processing Systems, 26, 2013.
[19] R. Krauthgamer and J. R. Lee. Navigating nets: simple algorithms for proximity search. In Proceedings of the 15th annual ACM-SIAM Symposium on Discrete algorithms, pages 798-807. Society for Industrial and Applied Mathematics, 2004.
[20] I. Kuzborskij and N. Cesa-Bianchi. Nonparametric Online Regression while Learning the Metric. In Conference on Neural Information Processing Systems, 2017.
[21] I. Kuzborskij and Cs. Szepesvári. Learning lipschitz functions by gd-trained shallow overparameterized relu neural networks. arXiv:2212.13848, 2022.
[22] T. Lin, C. Jin, and M. Jordan. Near-optimal algorithms for minimax optimization. arXiv preprint arXiv:2002.02417, 2020.
[23] T. Lin, C. Jin, and M. I. Jordan. On gradient descent ascent for nonconvex-concave minimax problems. arXiv preprint arXiv:1906.00331, 2019.
[24] Y. Mansour, M. Mohri, J. Ro, A. T. Suresh, and K. Wu. A theory of multiple-source adaptation with limited target labeled data. In Arindam Banerjee and Kenji Fukumizu, editors, International Conference on Artificial Intelligence and Statistics (AISTATS), volume 130 of Proceedings of Machine Learning Research, pages 2332-2340. PMLR, 13-15 Apr 2021.
[25] Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In Conference on Computational Learning Theory (COLT), 2009.
[26] H. B. McMahan, E. Moore, D. Ramage, and S. Hampson. Communication-efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629, 2016.
[27] A. Mokhtari, A. Ozdaglar, and S. Pattathil. Convergence rate of o  $(1 / k)$  for optimistic gradient and extra-gradient methods in smooth convex-concave saddle point problems. arXiv preprint arXiv:1906.01115, 2019.
[28] A. Mokhtari, A. Ozdaglar, and S. Pattathil. Proximal point approximations achieving a convergence rate of  $\mathrm{o}(1 / \mathrm{k})$  for smooth convex-concave saddle point problems: Optimistic gradient and extra-gradient methods. arXiv preprint arXiv:1906.01115, 2019.
[29] Q. Nguyen, M. Mondelli, and G. F. Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference on Machine Learning (ICML), pages 8119-8129. PMLR, 2021.
[30] M. Nouiehed, M. Sanjabi, J. D. Lee, and M. Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. arXiv preprint arXiv:1902.08297, 2019.
[31] S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms. Cambridge University Press, 2014.
[32] N. Srebro, K. Sridharan, and A. Tewari. Smoothness, low noise and fast rates. In Conference on Neural Information Processing Systems, pages 2199-2207, 2010.
[33] Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019.
[34] Z. Xu, H. Zhang, Y. Xu, and G. Lan. A unified single-loop alternating gradient projection algorithm for nonconvex-concave and convex-nonconcave minimax problems. Mathematical Programming, pages 1-72, 2023.
[35] C. Zhang, L. Zhang, and J. Ye. Generalization bounds for domain adaptation. Conference on Neural Information Processing Systems, 25, 2012.
A Proof of theorem 2: Stationarity of Algorithm 1
A.1 Proof Sketch
To prove the convergence of Algorithm 1 to a stationary point, we first need to relate our convergence measure to actual iterates during the algorithm dynamics. Defining
$$
G _ {w} (\pmb {\alpha}, \mathbf {w}) = \frac {1}{\gamma} \left(\mathbf {w} - \mathcal {P} _ {\mathcal {W}} \left(\mathbf {w} + \gamma \nabla_ {\mathbf {w}} F (\pmb {\alpha}, \mathbf {w})\right)\right), G _ {\alpha} (\pmb {\alpha}, \mathbf {w}) = \frac {1}{\eta} \left(\pmb {\alpha} - \mathcal {P} _ {\Lambda} \left(\pmb {\alpha} - \eta \nabla_ {\pmb {\alpha}} F (\pmb {\alpha}, \mathbf {w})\right)\right),
$$
the following statements hold:
$$
\mathbb {E} \left\| G _ {\alpha} (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} \leq \frac {2}{\eta^ {2}} \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t + 1} \right\| ^ {2} + 2 G _ {g} ^ {2} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t + 1} - (f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t}) - f _ {j} (\mathbf {w} ^ {t})) \right\| ^ {2},
$$
$$
\mathbb {E} \left\| G _ {w} (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} \leq \frac {2}{\gamma^ {2}} \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t + 1} \right\| ^ {2} + 4 G _ {g} ^ {2} \frac {\sigma^ {2}}{B} + 8 G _ {f} ^ {2} L _ {g} ^ {2} \sum_ {j = 1} ^ {N} \alpha^ {t} (j) \mathbb {E} \left\| z _ {j} ^ {t + 1} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t}) - f _ {j} (\mathbf {w} ^ {t})\right) \right\| ^ {2}.
$$
Bounds above show that the gradient norm can be bounded by the difference between two iterates generated by Algorithm 1, and the approximation error of our stochastic correction steps. Hence, we need to quantify how well the proposed stochastic correction step can approximate inner finite functions, i.e., the difference between  $z_{j}^{t + 1}$  and  $f_{\widehat{\mathcal{T}}}(\mathbf{w}^t) - f_j(\mathbf{w}^t)$ . This gap can be bounded following a standard result from [7]:
$$
\begin{array}{l} \mathbb {E} \left\| z _ {j} ^ {t + 1} - (f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t}) - f _ {j} (\mathbf {w} ^ {t})) \right\| ^ {2} \leq (1 - \beta) ^ {2} \mathbb {E} \left\| z _ {j} ^ {t} - (f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})) \right\| ^ {2} \\ + 4 (1 - \beta) ^ {2} G _ {f} ^ {2} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + 2 \beta^ {2} \frac {\sigma^ {2}}{B}. \\ \end{array}
$$
Besides above tracking error, we will also need to control the iterates gap of auxiliary variable  $z_{j}^{t}$ :
$$
\begin{array}{l} \mathbb {E} \left\| z _ {j} ^ {t + 1} - z _ {j} ^ {t} \right\| ^ {2} \leq \left(1 - \frac {\beta}{2}\right) ^ {2} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2} + 8 \left(1 + \frac {2}{\beta}\right) G _ {f} ^ {2} \left(\left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2}\right) \\ + 8 \left(1 + \frac {2}{\beta}\right) \beta^ {2} G _ {f} ^ {2} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2}. \\ \end{array}
$$
It shows that as the primal iterates approach the stationary point (i.e., iterates will not change too much), our auxiliary variable  $z_{j}^{t}$  also will not change too much.
Next, we are going to characterize the upper bound of  $\mathbb{E}\left\| \pmb{\alpha}^t - \pmb{\alpha}^{t + 1}\right\|^2$ ,  $\mathbb{E}\left\| \mathbf{w}^t - \mathbf{w}^{t + 1}\right\|^2$ . To this end, our proof relies on constructing two-level potential functions. Our first level potential function is defined as
$$
\begin{array}{l} \hat {F} ^ {t + 1} := F (\boldsymbol {\alpha} ^ {t + 1}, \mathbf {w} ^ {t + 1}) - \frac {2}{\eta^ {2} \mu} \| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \| ^ {2} - O \left(\frac {1}{4 \gamma} + L _ {g} ^ {2} G _ {f} ^ {4} \gamma + \frac {\eta L ^ {2}}{2} + \frac {N G _ {g} ^ {2}}{\mu \eta \beta}\right) \left\| \mathbf {w} ^ {t + 1} - \mathbf {w} ^ {t} \right\| ^ {2} \\ - O \left(\frac {1}{8 \gamma} + \frac {N G _ {g} ^ {2}}{\mu \eta \beta}\right) \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + O \left(\frac {7}{2 \eta} + \mu - \frac {\eta L ^ {2}}{2}\right) \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} \\ \end{array}
$$
Then we prove the following one-iteration relationship:
$$
\begin{array}{l} \mathbb {E} [ \hat {F} ^ {t + 1} - \hat {F} ^ {t} ] \geq C _ {1} \mathbb {E} \left\| \mathbf {w} ^ {t + 1} - \mathbf {w} ^ {t} \right\| ^ {2} + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2} \\ + C _ {2} \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} - (1 - \beta) ^ {2} O (\gamma + \eta) \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - (f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})) \right\| ^ {2} \\ - O \left(\gamma + \beta^ {2} \gamma\right) \frac {\sigma^ {2}}{B} - \left(1 - \frac {\beta}{2}\right) ^ {2} O \left(\frac {1}{\eta}\right) \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2}, \\ \end{array}
$$
where  $C_1$  and  $C_2$  are some constant depending on  $\eta, \gamma, L, \mu$ . To eliminate the approximation error  $\mathbb{E}\left\|z_j^t - (f_{\widehat{\mathcal{T}}}(\mathbf{w}^{t-1}) - f_j(\mathbf{w}^{t-1}))\right\|^2$  and difference of auxiliary iterates  $\mathbb{E}\left\|z_j^t - z_j^{t-1}\right\|^2$ , we need construct another potential function  $\tilde{F}^{t+1}$ :
$$
\tilde {F} ^ {t + 1} := \hat {F} ^ {t + 1} - \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t + 1} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t}) - f _ {j} (\mathbf {w} ^ {t})\right) \right\| ^ {2} - O \left(\frac {(1 - \frac {\beta}{2}) ^ {2}}{\beta} \frac {G _ {g} ^ {2}}{\mu^ {2} \eta}\right) \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t + 1} - z _ {j} ^ {t} \right\| ^ {2}
$$
Finally, we arrive at the following bounds.
$$
\begin{array}{l} \mathbb {E} [ \tilde {F} ^ {t + 1} - \tilde {F} ^ {t} ] \geq \frac {C _ {1}}{2} \gamma^ {2} \mathbb {E} \left\| \nabla_ {\mathbf {w}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} + \frac {C _ {2}}{2} \eta^ {2} \mathbb {E} \left\| \nabla_ {\boldsymbol {\alpha}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} \\ - \left(2 C _ {1} \gamma^ {2} L _ {g} ^ {2} N \beta^ {2} + 2 C _ {2} \eta^ {2} G _ {g} ^ {2} N ^ {2} \beta^ {2} + 4 \gamma G _ {g} ^ {2} + 2 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma + 2 \beta^ {2}\right) \frac {\sigma^ {2}}{B}. \\ \end{array}
$$
Conducting telescoping summation will yield the desired result.
A.2 Technical Lemmas
Proposition 2. If  $g(\cdot)$  is  $G_{g}$  Lipschitz and  $L_{g}$  smooth, and  $f_{\widehat{\tau}}(\cdot), f_{1}(\cdot), \ldots, f_{N}(\cdot)$  are all  $G_{f}$  Lipschitz and  $L_{f}$  smooth, then  $F(\alpha, \mathbf{w})$  is  $L := \max \left\{4G_{f}^{2}L_{g} + 2G_{g}L_{f}, \frac{2C}{m_{\min}}\right\}$  smooth, and  $\mu = \frac{2C}{m_{\max}}$  strongly convex in  $\alpha$ .
Proof. We first examine the Lipschitzness of gradient of  $F$  w.r.t.  $\alpha$ .
$$
\nabla_ {\boldsymbol {\alpha}} F (\boldsymbol {\alpha}, \mathbf {w}) = \left[ g \left(f _ {\hat {\mathcal {T}}} (\mathbf {w}) - f _ {1} (\mathbf {w})\right), \dots , g \left(f _ {\hat {\mathcal {T}}} (\mathbf {w}) - f _ {N} (\mathbf {w}))\right) \right] + 2 C \mathbf {M} \boldsymbol {\alpha}
$$
$$
\left\| \nabla_ {\boldsymbol {\alpha}} F (\boldsymbol {\alpha}, \mathbf {w}) - \nabla_ {\boldsymbol {\alpha}} F \left(\boldsymbol {\alpha} ^ {\prime}, \mathbf {w}\right) \right\| \leq 2 C \| \boldsymbol {\alpha} - \boldsymbol {\alpha} ^ {\prime} \|
$$
Similarly for gradient of  $F$  w.r.t.  $\mathbf{w}$
$$
\begin{array}{l} \left\| \nabla_ {\mathbf {w}} F (\boldsymbol {\alpha}, \mathbf {w}) - \nabla_ {\mathbf {w}} F (\boldsymbol {\alpha}, \mathbf {w} ^ {\prime}) \right\| \\ \leq \sum_ {i = 1} ^ {N} \alpha (i) \left\| \nabla g (f _ {\widehat {\mathcal {T}}} (\mathbf {w}) - f _ {i} (\mathbf {w})) (\nabla f _ {\widehat {\mathcal {T}}} (\mathbf {w}) - \nabla f _ {i} (\mathbf {w})) - \nabla g (f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {\prime}) - f _ {i} (\mathbf {w} ^ {\prime})) (\nabla f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {\prime}) - \nabla f _ {i} (\mathbf {w} ^ {\prime})) \right\| \\ \leq \sum_ {i = 1} ^ {N} \alpha (i) \left\| \nabla g \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w}) - f _ {i} (\mathbf {w})\right) \left(\nabla f _ {\widehat {\mathcal {T}}} (\mathbf {w}) - \nabla f _ {i} (\mathbf {w})\right) - \nabla g \left(f _ {\widehat {\mathcal {T}}} \left(\mathbf {w} ^ {\prime}\right) - f _ {i} \left(\mathbf {w} ^ {\prime}\right)\right) \left(\nabla f _ {\widehat {\mathcal {T}}} (\mathbf {w}) - \nabla f _ {i} (\mathbf {w})\right) \right\| \\ + \sum_ {i = 1} ^ {N} \alpha (i) \left\| \nabla g (f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {\prime}) - f _ {i} (\mathbf {w} ^ {\prime})) (\nabla f _ {\widehat {\mathcal {T}}} (\mathbf {w}) - \nabla f _ {i} (\mathbf {w})) - \nabla g (f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {\prime}) - f _ {i} (\mathbf {w} ^ {\prime})) (\nabla f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {\prime}) - \nabla f _ {i} (\mathbf {w} ^ {\prime})) \right\| \\ \leq \sum_ {i = 1} ^ {N} \alpha (i) L _ {g} \left\| f _ {\widehat {\mathcal {T}}} (\mathbf {w}) - f _ {i} (\mathbf {w}) - \left(f _ {\widehat {\mathcal {T}}} \left(\mathbf {w} ^ {\prime}\right) - f _ {i} \left(\mathbf {w} ^ {\prime}\right)\right) \right\| \cdot 2 G _ {f} + \sum_ {i = 1} ^ {N} \alpha (i) 2 L _ {f} \| \mathbf {w} - \mathbf {w} ^ {\prime} \| \cdot G _ {g} \\ \leq \sum_ {i = 1} ^ {N} \alpha (i) 4 G _ {f} ^ {2} L _ {g} \| \mathbf {w} - \mathbf {w} ^ {\prime} \| + \sum_ {i = 1} ^ {N} \alpha (i) 2 G _ {g} L _ {f} \| \mathbf {w} - \mathbf {w} ^ {\prime} \|. \\ \end{array}
$$
For strong convexity, we compute Hessian w.r.t.  $\alpha$ :
$$
\nabla_ {\boldsymbol {\alpha}} ^ {2} F (\mathbf {w}, \boldsymbol {\alpha}) = 2 C \mathbf {M} \succeq 2 C \frac {1}{m _ {\max}} \mathbf {I},
$$
so  $F(\alpha, \mathbf{w})$  is  $2CM$  strongly convex in  $\alpha$ .
The following lemma bounds the tracking error of the stochastic correction algorithm.
Lemma 2 (Tracking Error [7]). For Algorithm 1, under the assumptions of Theorem 2, the following statement holds true:
$$
\begin{array}{l} \mathbb {E} \left\| z _ {j} ^ {t + 1} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t}) - f _ {j} (\mathbf {w} ^ {t})\right) \right\| ^ {2} \leq (1 - \beta) ^ {2} \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})\right) \right\| ^ {2} \\ + 4 (1 - \beta) ^ {2} G _ {f} ^ {2} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + 2 \beta^ {2} \frac {\sigma^ {2}}{B}. \\ \end{array}
$$
An immediate implication of Lemma 2 is the following corollary:
Corollary 2. For Algorithm 1, under the assumptions of Theorem 2, the following statement holds true:
$$
\begin{array}{l} \mathbb {E} \left\| z _ {j} ^ {T + 1} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {T}) - f _ {j} (\mathbf {w} ^ {T})\right) \right\| ^ {2} \leq (1 - \beta) ^ {2 T} \mathbb {E} \left\| z _ {j} ^ {0} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {- 1}) - f _ {j} (\mathbf {w} ^ {- 1})\right) \right\| ^ {2} \\ + 4 \frac {(1 - \beta) ^ {4}}{1 - (1 - \beta) ^ {2}} \gamma^ {2} G _ {f} ^ {4} G _ {g} ^ {2} + 2 \beta^ {2} \frac {(1 - \beta) ^ {2}}{1 - (1 - \beta) ^ {2}} \frac {\sigma^ {2}}{B}. \\ \end{array}
$$
Proof. Due to Lemma 2 and updating rule for  $\mathbf{w}$ , we have
$$
\begin{array}{l} \mathbb {E} \left. \right.\left\| z _ {j} ^ {t + 1} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t}) - f _ {j} (\mathbf {w} ^ {t})\right)\right\| ^ {2} \leq (1 - \beta) ^ {2} \mathbb {E} \left.\left\| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})\right)\right\| ^ {2} \right. \\ + 4 (1 - \beta) ^ {2} G _ {f} ^ {2} \mathbb {E} \left\| \gamma \mathbf {g} _ {\mathbf {w}} ^ {t - 1} \right\| ^ {2} + 2 \beta^ {2} \frac {\sigma^ {2}}{B} \\ \leq (1 - \beta) ^ {2} \mathbb {E} \left| \left| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} \left(\mathbf {w} ^ {t - 1}\right) - f _ {j} \left(\mathbf {w} ^ {t - 1}\right)\right) \right| \right| ^ {2} \\ + 4 (1 - \beta) ^ {2} G _ {f} ^ {2} \gamma^ {2} G _ {g} ^ {2} G _ {f} ^ {2} + 2 \beta^ {2} \frac {\sigma^ {2}}{B}. \\ \end{array}
$$
Then we unroll the recursion in Lemma 2:
$$
\begin{array}{l} \mathbb {E} \left\| z _ {j} ^ {T + 1} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {T}) - f _ {j} (\mathbf {w} ^ {T})\right) \right\| ^ {2} \leq (1 - \beta) ^ {2 T} \mathbb {E} \left\| z _ {j} ^ {0} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {- 1}) - f _ {j} (\mathbf {w} ^ {- 1})\right) \right\| ^ {2} \\ + \sum_ {t = 0} ^ {T} (1 - \beta) ^ {2 t} \left(4 (1 - \beta) ^ {2} \gamma^ {2} G _ {g} ^ {2} G _ {f} ^ {4} + 2 \beta^ {2} \frac {\sigma^ {2}}{B}\right) \\ \leq (1 - \beta) ^ {2 T} \mathbb {E} \left\| z _ {j} ^ {0} - \left(f _ {\widehat {\gamma}} \left(\mathbf {w} ^ {- 1}\right) - f _ {j} \left(\mathbf {w} ^ {- 1}\right)\right) \right\| ^ {2} \\ + \frac {(1 - \beta) ^ {2}}{1 - (1 - \beta) ^ {2}} \left(4 (1 - \beta) ^ {2} \gamma^ {2} G _ {g} ^ {2} G _ {f} ^ {4} + 2 \beta^ {2} \frac {\sigma^ {2}}{B}\right). \\ \end{array}
$$
Besides the above lemma, we also need the following bound on tracking error between two consecutive iterates.
Lemma 3 (Second Order Tracking Error). For Algorithm 1, under the assumptions of Theorem 2, the following statement holds true:
$$
\begin{array}{l} \mathbb {E} \left\| z _ {j} ^ {t + 1} - z _ {j} ^ {t} \right\| ^ {2} \leq (1 - \frac {\beta}{2}) ^ {2} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2} + 8 \left(1 + \frac {2}{\beta}\right) G _ {f} ^ {2} \left(\left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2}\right) \\ + 8 \left(1 + \frac {2}{\beta}\right) \beta^ {2} G _ {f} ^ {2} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2}. \\ \end{array}
$$
Proof. For the ease of presentation, we define the following two auxiliary variables:
$$
\begin{array}{l} f _ {j} ^ {t} = f _ {i} \left(\mathbf {w} ^ {t}; \xi_ {i} ^ {t}\right) - f _ {j} \left(\mathbf {w} ^ {t}; \xi_ {j} ^ {t}\right), \\ f _ {j} ^ {t \mapsto t - 1} = f _ {i} \left(\mathbf {w} ^ {t}; \xi_ {i} ^ {t}\right) - f _ {j} \left(\mathbf {w} ^ {t}; \xi_ {j} ^ {t}\right) - \left(f _ {i} \left(\mathbf {w} ^ {t - 1}; \xi_ {i} ^ {t}\right) - f _ {j} \left(\mathbf {w} ^ {t - 1}; \xi_ {j} ^ {t}\right)\right). \\ \end{array}
$$
According to updating rule of  $z$ , we have:
$$
z _ {j} ^ {t + 1} - z _ {j} ^ {t} = (1 - \beta) (z _ {j} ^ {t} - z _ {j} ^ {t - 1}) + (1 - \beta) (f _ {j} ^ {t \mapsto t - 1} - f _ {j} ^ {t - 1 \mapsto t - 2}) + \beta (f _ {j} ^ {t} - f _ {j} ^ {t - 1})
$$
Taking expectation w.r.t.  $\xi_j^t,\xi_i^t,\xi_j^{t - 1}$  and  $\xi_i^{t - 1}$  yields:
$$
\mathbb {E} \left. \left\| z _ {j} ^ {t + 1} - z _ {j} ^ {t} \right\| ^ {2} \right.
$$
$$
= \mathbb {E} \left\| (1 - \beta) \left(z _ {j} ^ {t} - z _ {j} ^ {t - 1}\right) + (1 - \beta) \left(f _ {j} ^ {t \mapsto t - 1} - f _ {j} ^ {t - 1 \mapsto t - 2}\right) + \beta \left(f _ {j} ^ {t} - f _ {j} ^ {t - 1}\right) \right\| ^ {2}
$$
$$
\begin{array}{l} \stackrel {(\Delta)} {\leq} \left(1 + \frac {\beta}{2 - 2 \beta}\right) (1 - \beta) ^ {2} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} + \left(f _ {j} ^ {t \mapsto t - 1} - f _ {j} ^ {t - 1 \mapsto t - 2}\right) \right\| ^ {2} \\ + \left(1 + \frac {2 - 2 \beta}{\beta}\right) \left\| \beta \left(f _ {j} ^ {t} - f _ {j} ^ {t - 1}\right) \right\| ^ {2} \\ \end{array}
$$
$$
\stackrel {(\Theta)} {\leq} \left(1 - \frac {\beta}{2}\right) (1 - \beta) \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} + \left(f _ {j} ^ {t \mapsto t - 1} - f _ {j} ^ {t - 1 \mapsto t - 2}\right) \right\| ^ {2} + \left(1 + \frac {2}{\beta}\right) \beta^ {2} \left\| f _ {j} ^ {t} - f _ {j} ^ {t - 1} \right\| ^ {2}
$$
$$
\begin{array}{l} \stackrel {(\Lambda)} {\leq} \left(1 - \frac {\beta}{2}\right) (1 - \beta) \left(1 + \frac {\beta}{2 - 2 \beta}\right) \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2} \\ + \left(1 - \frac {\beta}{2}\right) (1 - \beta) \left(1 + \frac {2 - 2 \beta}{\beta}\right) \mathbb {E} \left\| f _ {j} ^ {t \mapsto t - 1} - f _ {j} ^ {t - 1 \mapsto t - 2} \right\| ^ {2} + \left(1 + \frac {2}{\beta}\right) \beta^ {2} \left\| f _ {j} ^ {t} - f _ {j} ^ {t - 1} \right\| ^ {2} \\ \end{array}
$$
$$
\stackrel {(\equiv)} {\leq} \left(1 - \frac {\beta}{2}\right) ^ {2} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1}) \right\| ^ {2} + \underbrace {\left(1 + \frac {2}{\beta}\right) \mathbb {E} \left\| f _ {j} ^ {t \mapsto t - 1} - f _ {j} ^ {t - 1 \mapsto t - 2} \right\| ^ {2}} _ {T _ {1}} + \underbrace {\left(1 + \frac {2}{\beta}\right) \beta^ {2} \left\| f _ {j} ^ {t} - f _ {j} ^ {t - 1} \right\| ^ {2}} _ {T _ {2}}
$$
where in (1) and (3) we use Young's inequality that  $\| \mathbf{a} + \mathbf{b}\|^2 \leq (1 + a)\|\mathbf{a}\|^2 + (1 + \frac{1}{a})\|\mathbf{b}\|^2$ . Now we bound  $T_1$  as follows:
$$
\begin{array}{l} T _ {1} \leq 4 \left(1 + \frac {2}{\beta}\right) \left(\left\| f _ {i} \left(\mathbf {w} ^ {t}; \xi_ {i} ^ {t}\right) - f _ {i} \left(\mathbf {w} ^ {t - 1}; \xi_ {i} ^ {t}\right) \right\| ^ {2} + \left\| f _ {j} \left(\mathbf {w} ^ {t}; \xi_ {j} ^ {t}\right) - f _ {j} \left(\mathbf {w} ^ {t - 1}; \xi_ {j} ^ {t}\right) \right\| ^ {2}\right) \\ + 4 \left(1 + \frac {2}{\beta}\right) \left(\left\| f _ {i} \left(\mathbf {w} ^ {t - 1}; \xi_ {i} ^ {t - 1}\right) - f _ {i} \left(\mathbf {w} ^ {t - 2}; \xi_ {i} ^ {t - 1}\right) \right\| ^ {2} + \left\| f _ {j} \left(\mathbf {w} ^ {t - 1}; \xi_ {j} ^ {t - 1}\right) - f _ {j} \left(\mathbf {w} ^ {t - 2}; \xi_ {j} ^ {t - 1}\right) \right\| ^ {2}\right) \\ \leq 8 \left(1 + \frac {2}{\beta}\right) G _ {f} ^ {2} \left(\left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2}\right), \\ \end{array}
$$
and for  $T_{2}$