Boris Polyak and Y.Z. Tsypkin. Pseudogradient adaptation and training algorithms. Automation and Remote Control, 34:377-397, 01 1973.
Peter Richtárik and Martin Takáč. Stochastic reformulations of linear systems: Algorithms and convergence theory. SIAM J. Matrix Anal. Appl., 41(2):487-524, 2020. ISSN 0895-4798.
Peter Richtárik, Igor Sokolov, and Ilyas Fatkhullin. EF21: A new, simpler, theoretically better, and practically faster error feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Peter Richtárik, Igor Sokolov, Elnur Gasanov, Ilyas Fatkhullin, Zhize Li, and Eduard Gorbunov. 3PC: Three point compressors for communication-efficient distributed training and a better theory for lazy aggregation. In International Conference on Machine Learning (ICML), pages 18596-18648. PMLR, 2022.
Herbert Robbins and Sutton Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22:400-407, 1951.
Atal Sahu, Aritra Dutta, Ahmed M Abdelmoniem, Trambak Banerjee, Marco Canini, and Panos Kalnis. Rethinking gradient sparsification as total error minimization. Advances in Neural Information Processing Systems (NeurIPS, 34:8133-8146, 2021).
Amedeo Sapio, Marco Canini, Chen-Yu Ho, Jacob Nelson, Panos Kalnis, Changhoon Kim, Arvind Krishnamurthy, Masoud Moshref, Dan Ports, and Peter Richtárik. Scaling distributed machine learning with in-network aggregation. In In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), pages 785-808, 2021.
Mark Schmidt, Nicolas Roux, and Francis Bach. Convergence rates of inexact proximal-gradient methods for convex optimization. Advances in Neural Information Processing Systems (NeurIPS), 24:1458-1466, 2011.
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014.
Sebastian U Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rates for SGD with delayed gradients and compressed updates. The Journal of Machine Learning Research, 21(1):9613-9648, 2020.
Sebastian U. Stich, J.-B. Cordonnier, and Martin Jaggi. Sparsified SGD with memory. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Nikko Ström. Scalable distributed dnn training using commodity GPU cloud computing. In *Interspeech* 2015, 2015.
Ruo-Yu Sun. Optimization for deep learning: An overview. Journal of the Operations Research Society of China, 8(2):249-294, 2020.
Hanlin Tang, Xiangru Lian, Chen Yu, Tong Zhang, and Ji Liu. DoubleSqueeze: Parallel Stochastic Gradient Descent with double-pass error-compensated compression. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2020.
Rachael Tappenden, Peter Richtárik, and Jacek Gondzio. Inexact coordinate descent: Complexity and preconditioning. Journal of Optimization Theory and Applications, 170:144-176, 2016.
Jie Wang, Rui Gao, and Yao Xie. Sinkhorn distributionally robust optimization. arXiv preprint arXiv:2109.11926, 2021.
Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communication-efficient distributed optimization. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, pages 1306-1316, 2018.