[31] Shi, G., Li, B., Johansson, M., and Johansson, K. H. (2016). Finite-time convergent gossiping. In IEEE Transactions on Networking.
[32] Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations.
[33] Song, Z., Li, W., Jin, K., Shi, L., Yan, M., Yin, W., and Yuan, K. (2022). Communication-efficient topologies for decentralized learning with  $O(1)$  consensus rate. In Advances in Neural Information Processing Systems.
[34] Takezawa, Y., Bao, H., Niwa, K., Sato, R., and Yamada, M. (2023). Momentum tracking: Momentum acceleration for decentralized deep learning on heterogeneous data. In Transactions on Machine Learning Research.
[35] Tang, H., Gan, S., Zhang, C., Zhang, T., and Liu, J. (2018a). Communication compression for decentralized training. In Advances in Neural Information Processing Systems.
[36] Tang, H., Lian, X., Yan, M., Zhang, C., and Liu, J. (2018b).  $D^2$ : Decentralized training over decentralized data. In International Conference on Machine Learning.
[37] Vogels, T., He, L., Koloskova, A., Karimireddy, S. P., Lin, T., Stich, S. U., and Jaggi, M. (2021). Relaysum for decentralized deep learning on heterogeneous data. In Advances in Neural Information Processing Systems.
[38] Vogels, T., Karimireddy, S. P., and Jaggi, M. (2020). Practical low-rank communication compression in decentralized deep learning. In Advances in Neural Information Processing Systems.
[39] Wang, J., Sahu, A. K., Yang, Z., Joshi, G., and Kar, S. (2019). Matcha: Speeding up decentralized sgd via matching decomposition sampling. In Indian Control Conference.
[40] Wu, Y. and He, K. (2018). Group normalization. In European Conference on Computer Vision.
[41] Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. In arXiv.
[42] Xin, R. and Khan, U. A. (2020). Distributed heavy-ball: A generalization and acceleration of first-order methods with gradient tracking. In IEEE Transactions on Automatic Control.
[43] Ying, B., Yuan, K., Chen, Y., Hu, H., Pan, P., and Yin, W. (2021). Exponential graph is provably efficient for decentralized deep training. In Advances in Neural Information Processing Systems.
[44] Yu, H., Jin, R., and Yang, S. (2019). On the linear speedup analysis of communication efficient momentum SGD for distributed non-convex optimization. In International Conference on Machine Learning.
[45] Yuan, K., Chen, Y., Huang, X., Zhang, Y., Pan, P., Xu, Y., and Yin, W. (2021). Decentralized momentum sgd for large-batch deep training. In International Conference on Computer Vision.
[46] Yuan, K., Ying, B., Zhao, X., and Sayed, A. H. (2019). Exact diffusion for distributed optimization and learning—part i: Algorithm development. In IEEE Transactions on Signal Processing.
[47] Zhao, H., Li, B., Li, Z., Richtárik, P., and Chi, Y. (2022). BEER: Fast  $O(1 / T)$  rate for decentralized nonconvex optimization with communication compression. In Advances in Neural Information Processing Systems.
[48] Zhu, T., He, F., Zhang, L., Niu, Z., Song, M., and Tao, D. (2022). Topology-aware generalization of decentralized SGD. In International Conference on Machine Learning.