Adaptive synchronous strategy for distributed machine learning

International Journal of Intelligent SystemsEarly View RESEARCH ARTICLE Adaptive synchronous strategy for distributed machine learning Miaoquan Tan, Miaoquan Tan orcid.org/0000-0002-8664-6181 Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of ChinaSearch for more papers by this authorWai-Xi Liu, Corresponding Author Wai-Xi Liu liuwaixi@sina.com orcid.org/0000-0002-7343-4948 Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of China Correspondence Wai-Xi Liu, Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of China. Email: liuwaixi@sina.comSearch for more papers by this authorJunming Luo, Junming Luo Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of ChinaSearch for more papers by this authorHaosen Chen, Haosen Chen Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of ChinaSearch for more papers by this authorZhen-Zheng Guo, Zhen-Zheng Guo Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of ChinaSearch for more papers by this author Miaoquan Tan, Miaoquan Tan orcid.org/0000-0002-8664-6181 Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of ChinaSearch for more papers by this authorWai-Xi Liu, Corresponding Author Wai-Xi Liu liuwaixi@sina.com orcid.org/0000-0002-7343-4948 Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of China Correspondence Wai-Xi Liu, Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of China. Email: liuwaixi@sina.comSearch for more papers by this authorJunming Luo, Junming Luo Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of ChinaSearch for more papers by this authorHaosen Chen, Haosen Chen Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of ChinaSearch for more papers by this authorZhen-Zheng Guo, Zhen-Zheng Guo Department of Electronic and Communication Engineering, Guangzhou University, Guangzhou, People's Republic of ChinaSearch for more papers by this author First published: 20 September 2022 https://doi.org/10.1002/int.23060Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract In distributed machine learning training, bulk synchronous parallel (BSP) and asynchronous parallel (ASP) are two main synchronization methods to help achieve gradient aggregation. However, BSP needs longer training time due to "stragglers" problem, while ASP sacrifices the accuracy due to "gradient staleness" problem. In this article, we propose a distributed training paradigm on parameter server framework called adaptive synchronous strategy (A2S) which improves the BSP and ASP paradigms by adaptively adopting different parallel training schemes for workers with different training speeds. Based on the stale value between the fastest and slowest worker, A2S adaptively adds a relaxed synchronous barrier for fast workers to alleviate gradient staleness, where a differentiated weighting gradient aggregation method is used to reduce the impact of slow gradients. Simultaneously, A2S adopts ASP training for slow workers to eliminate stragglers. Hence, A2S not only improves the "gradient staleness" and "stragglers" problems, but also obtains convergence stability and synchronous gain through synchronous and asynchronous parallel, respectively. Specially, we theoretically proved the convergence of A2S by deriving the regret bound. Moreover, experiment results show that A2S improves accuracy by up to 2.64% and accelerates training by up to 41% more than the state-of-the-art synchronization methods BSP, ASP, stale synchronous parallel (SSP), dynamic SSP, and Sync-switch. CONFLICT OF INTEREST The authors declare no conflict of interest. Open Research DATA AVAILABILITY STATEMENT Research data are not shared. Early ViewOnline Version of Record before inclusion in an issue RelatedInformation