[41] Wolfe, P. (1969). Convergence conditions for ascent methods. SIAM review, 11(2):226-235.
[42] Xie, Z., He, F., Fu, S., Sato, I., Tao, D., and Sugiyama, M. (2021a). Artificial neural variability for deep learning: On overfitting, noise memorization, and catastrophic forgetting. *Neural Computation*, 33(8).
[43] Xie, Z., Sato, I., and Sugiyama, M. (2021b). A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations.
[44] Xie, Z., Tang, Q., Sun, M., and Li, P. (2023). On the overlooked structure of stochastic gradients. In Thirty-seventh Conference on Neural Information Processing Systems.
[45] Xie, Z., Tang, Q.-Y., Cai, Y., Sun, M., and Li, P. (2022a). On the power-law spectrum in deep learning: A bridge to protein science. arXiv preprint arXiv:2201.13011.
[46] Xie, Z., Wang, X., Zhang, H., Sato, I., and Sugiyama, M. (2022b). Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 24430-24459.
[47] Xie, Z., Yuan, L., Zhu, Z., and Sugiyama, M. (2021c). Positive-negative momentum: Manipulating stochastic gradient noise to improve generalization. In International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11448-11458. PMLR.
[48] Yan, Y., Yang, T., Li, Z., Lin, Q., and Yang, Y. (2018). A unified analysis of stochastic momentum methods for deep learning. In *IJCAI International Joint Conference on Artificial Intelligence*.
[49] Zaheer, M., Reddi, S., Sachan, D., Kale, S., and Kumar, S. (2018). Adaptive methods for nonconvex optimization. In Advances in neural information processing systems, pages 9793-9803.
[50] Zaremba, W., Sutskever, I., and Vinyals, O. (2014). Recurrent neural network regularization. arXiv preprint arXiv:1409.2329.
[51] Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In International Conference on Machine Learning.
[52] Zhang, G., Wang, C., Xu, B., and Grosse, R. (2019a). Three mechanisms of weight decay regularization. International Conference on Learning Representations.
[53] Zhang, J., He, T., Sra, S., and Jadbabaie, A. (2019b). Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations.
[54] Zhang, X., Xu, R., Yu, H., Zou, H., and Cui, P. (2022). Gradient norm regularizer seeks flat minima and improves generalization.
[55] Zhao, Y., Zhang, H., and Hu, X. (2022). Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning.
[56] Zhou, P., Feng, J., Ma, C., Xiong, C., HOI, S., et al. (2020). Towards theoretically understanding why sgd generalizes better than adam in deep learning. arXiv preprint arXiv:2010.05627.
[57] Zhu, Z., Wu, J., Yu, B., Wu, L., and Ma, J. (2019). The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In ICML, pages 7654-7663.