[44] Y. E. Nesterov. A method of solving a convex programming problem with convergence rate  $o(1 / k^2)$ . In Doklady Akademii Nauk, volume 269, pages 543-547. Russian Academy of Sciences, 1983.
[45] F. Nogueira et al. Bayesian optimization: Open source constrained global optimization tool for python. 2014.
[46] C. Philippenko and A. Dieuleveut. Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees. ArXiv, 2022.
[47] S. Pu and A. Nedic. Distributed stochastic gradient tracking methods. Mathematical Programming, 187:409-457, 2021.
[48] X. Qian, P. Richtárik, and T. Zhang. Error compensated distributed sgd can be accelerated. Advances in Neural Information Processing Systems, 34:30401-30413, 2021.
[49] P. Richtárik, I. Sokolov, and I. Fatkhullin. Ef21: A new, simpler, theoretically better, and practically faster error feedback. ArXiv, abs/2106.05203, 2021.
[50] M. H. Safaryan, E. Shulgin, and P. Richtárik. Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor. Information and Inference: A Journal of the IMA, 2021.
[51] K. Scaman, F. R. Bach, S. Bubeck, Y. T. Lee, and L. Massoulié. Optimal algorithms for smooth and strongly convex distributed optimization in networks. In International Conference on Machine Learning, 2017.
[52] F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In INTERSPEECH, 2014.
[53] B. Song, I. Tsaknakis, C.-Y. Yau, H.-T. Wai, and M. Hong. Distributed optimization for overparameterized problems: Achieving optimal dimension independent communication complexity. In Advances in Neural Information Processing Systems, volume 35, pages 6147-6160, 2022.
[54] S. U. Stich. Local sgd converges fast and communicates little. In International Conference on Learning Representations, 2019.
[55] S. U. Stich, J.-B. Cordonnier, and M. Jaggi. Sparsified sgd with memory. In Advances in Neural Information Processing Systems, 2018.
[56] H. Tang, X. Lian, M. Yan, C. Zhang, and J. Liu.  $d^2$ : Decentralized training over decentralized data. In International Conference on Machine Learning, pages 4848-4856, 2018.
[57] J. Wangni, J. Wang, J. Liu, and T. Zhang. Gradient sparsification for communication-efficient distributed optimization. In Advances in Neural Information Processing Systems, 2018.
[58] W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. H. Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems, 2017.
[59] J. Wu, W. Huang, J. Huang, and T. Zhang. Error compensated quantized sgd and its applications to large-scale distributed optimization. In International Conference on Machine Learning, 2018.
[60] R. Xin, U. A. Khan, and S. Kar. An improved convergence analysis for decentralized online stochastic non-convex optimization. IEEE Transactions on Signal Processing, 2020.
[61] K. Yuan, S. A. Alghunaim, and X. Huang. Removing data heterogeneity influences network topology dependence of decentralized sgd. Journal of Machine Learning Research, 24(280):1-53, 2023.
[62] K. Yuan, S. A. Alghunaim, B. Ying, and A. H. Sayed. On the influence of bias-correction on distributed stochastic optimization. IEEE Transactions on Signal Processing, 2020.
[63] K. Yuan, Y. Chen, X. Huang, Y. Zhang, P. Pan, Y. Xu, and W. Yin. Decentlam: Decentralized momentum sgd for large-batch deep training. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3009-3019, 2021.
[64] K. Yuan, X. Huang, Y. Chen, X. Zhang, Y. Zhang, and P. Pan. Revisiting optimal convergence rate for smooth and non-convex stochastic decentralized optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[65] M. D. Zeiler. Adadelta: An adaptive learning rate method. ArXiv, abs/1212.5701, 2012.