An automatic learning rate decay strategy for stochastic gradient descent optimization methods in neural networks

International Journal of Intelligent SystemsEarly View RESEARCH ARTICLE An automatic learning rate decay strategy for stochastic gradient descent optimization methods in neural networks Kang Wang, Kang Wang orcid.org/0000-0002-0632-5901 National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorYong Dou, Yong Dou orcid.org/0000-0002-1256-8934 National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorTao Sun, Corresponding Author Tao Sun nudtsuntao@163.com orcid.org/0000-0001-5024-1900 National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, Changsha, China Correspondence Tao Sun, National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, 410073 Changsha, China. Email: nudtsuntao@163.comSearch for more papers by this authorPeng Qiao, Peng Qiao orcid.org/0000-0001-6752-7892 National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorDong Wen, Dong Wen orcid.org/0000-0002-1537-0077 National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, Changsha, ChinaSearch for more papers by this author Kang Wang, Kang Wang orcid.org/0000-0002-0632-5901 National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorYong Dou, Yong Dou orcid.org/0000-0002-1256-8934 National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorTao Sun, Corresponding Author Tao Sun nudtsuntao@163.com orcid.org/0000-0001-5024-1900 National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, Changsha, China Correspondence Tao Sun, National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, 410073 Changsha, China. Email: nudtsuntao@163.comSearch for more papers by this authorPeng Qiao, Peng Qiao orcid.org/0000-0001-6752-7892 National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, Changsha, ChinaSearch for more papers by this authorDong Wen, Dong Wen orcid.org/0000-0002-1537-0077 National Laboratory for Parallel and Distributed Processing, School of Computer, National University of Defense Technology, Changsha, ChinaSearch for more papers by this author First published: 31 March 2022 https://doi.org/10.1002/int.22883Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Stochastic Gradient Descent (SGD) series optimization methods play the vital role in training neural networks, attracting growing attention in science and engineering fields of the intelligent system. The choice of learning rates affects the convergence rate of SGD series optimization methods. Currently, learning rate adjustment strategies mainly face the following problems: (1) The traditional learning rate decay method mainly adopts manual manner during training iterations, the small learning rate produced from which causes slow convergence in training neural networks. (2) Adaptive method (e.g., Adam) has poor generalization performance. To alleviate the above issues, we propose a novel automatic learning rate decay strategy for SGD optimization methods in neural networks. On the basis of the observation that the convergence rate's upper bound enjoys minimization in a specific iteration concerning the current learning rate, we first present the expression of the current learning rate determined by historical learning rates. And merely one extra parameter is initialized to generate automatic decreasing learning rates during the training process. Our proposed approach is applied to SGD and Momentum SGD optimization algorithms, and concrete theoretical proof explains its convergence. Numerical simulations are conducted on the MNIST and Cifar-10 data sets with different neural networks. Experimental results show that our algorithm outperforms existing classical ones, achieving faster convergence rate, better stability, and generalization performance in neural network training. It also lays a foundation for large-scale parallel search of initial parameters in intelligent systems. Early ViewOnline Version of Record before inclusion in an issue RelatedInformation