[17] Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear Convergence of Gradient and ProximalGradient Methods Under the Polyak-Lojasiewicz Condition. arXiv:1608.04636, 2020.
[18] Ahmed Khaled and Peter Richtárik. Gradient descent with compressed iterates. arXiv preprint arXiv:1909.04716, 2019.
[19] Ahmed Khaled and Peter Rictarik. Better Theory for SGD in the Nonconvex World. Transactions on Machine Learning Research, 2023.
[20] Jakub Koněcný and Peter Richtárik. Semi-Stochastic Gradient Descent Methods. Frontiers in Applied Mathematics and Statistics 3:9, 2017.
[21] Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do Better ImageNet Models Transfer Better? Conference on Computer Vision and Pattern Recognition, 2019.
[22] Dmitry Kovalev, Samuel Horváth, and Peter Richtárik. Don't jump through hoops and remove those loops: SVRG and Katyusha are better without the outer loop. 31st International Conference on Learning Theory (ALT), 2020.
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. CiteSeer, 2009.
[24] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist/, 2010.
[25] Lihua Lei, Cheng Ju, Jianbo Chen, and Michael I. Jordan. Non-Convex Finite-Sum Optimization Via SCSG Methods. 31st Conference on Neural Information Processing Systems (NIPS), 2017.
[26] Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. Learning from noisy labels with distillation. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1928-1936. IEEE, 2017.
[27] Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503-528, 1989.
[28] David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. arXiv preprint arXiv:1511.03643, 2015.
[29] Grigory Malinovsky, Alibek Sailanbayev, and Peter Richtárik. Random reshuffling with variance reduction: new analysis and better rates. 39th Conference on Uncertainty in Artificial Intelligence (UAI), 2023.
[30] Ilia Markov, Adrian Vladu, Qi Guo, and Dan Alistarh. Quantized Distributed Training of Large Models with Convergence Guarantees. arXiv preprint arXiv:2302.02390, 2023.
[31] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 5191-5198, 2020.
[32] Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. arXiv preprint arXiv:1711.05852, 2017.
[33] Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett. Self-Distillation Amplifies Regularization in Hilbert Space. 34th Conference on Neural Information Processing Systems (NeurIPS), 2020.
[34] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, 2010.
[35] Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.
[36] Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In International Conference on Learning Representations, 2016.