Understanding adaptive gradient clipping in DP‐SGD, empirically

International Journal of Intelligent SystemsVolume 37, Issue 11 p. 9674-9700 RESEARCH ARTICLE Understanding adaptive gradient clipping in DP-SGD, empirically Guanbiao Lin, Guanbiao Lin orcid.org/0000-0002-7372-7345 Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, ChinaSearch for more papers by this authorHongyang Yan, Corresponding Author Hongyang Yan hyang_yan@gzhu.edu.cn hyang.yan@foxmail.com orcid.org/0000-0002-1493-9671 Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, China Correspondence Hongyang Yan, Institute of Artificial Intelligence and Blockchain, Guangzhou University, 510006 Guangdong, China. Email: hyang_yan@gzhu.edu.cn and hyang.yan@foxmail.comSearch for more papers by this authorGuang Kou, Guang Kou Artificial Intelligence Research Center, Defense Innovation Institute, Beijing, ChinaSearch for more papers by this authorTeng Huang, Teng Huang orcid.org/0000-0001-7261-6398 Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, ChinaSearch for more papers by this authorShiyu Peng, Shiyu Peng Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, ChinaSearch for more papers by this authorYingying Zhang, Yingying Zhang Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, ChinaSearch for more papers by this authorChangyu Dong, Changyu Dong Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, ChinaSearch for more papers by this author Guanbiao Lin, Guanbiao Lin orcid.org/0000-0002-7372-7345 Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, ChinaSearch for more papers by this authorHongyang Yan, Corresponding Author Hongyang Yan hyang_yan@gzhu.edu.cn hyang.yan@foxmail.com orcid.org/0000-0002-1493-9671 Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, China Correspondence Hongyang Yan, Institute of Artificial Intelligence and Blockchain, Guangzhou University, 510006 Guangdong, China. Email: hyang_yan@gzhu.edu.cn and hyang.yan@foxmail.comSearch for more papers by this authorGuang Kou, Guang Kou Artificial Intelligence Research Center, Defense Innovation Institute, Beijing, ChinaSearch for more papers by this authorTeng Huang, Teng Huang orcid.org/0000-0001-7261-6398 Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, ChinaSearch for more papers by this authorShiyu Peng, Shiyu Peng Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, ChinaSearch for more papers by this authorYingying Zhang, Yingying Zhang Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, ChinaSearch for more papers by this authorChangyu Dong, Changyu Dong Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangdong, ChinaSearch for more papers by this author First published: 25 August 2022 https://doi.org/10.1002/int.23001Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Differentially Private Stochastic Gradient Descent (DP-SGD) is a prime method for training machine learning models with rigorous privacy guarantees. Since its birth, DP-SGD has gained popularity and has been widely adopted in both academic and industrial research. One well-known challenge when using DP-SGD is how to improve utility while maintaining privacy. To this end, recently we have seen several proposals that clip the gradients with adaptive thresholds rather than a fixed one. Although each proposal comes with some theoretical justification, the theories often rely on strong assumptions and are not compatible with each other. It is hard to know whether they are good in practice and how good they are. In this paper, we investigate adaptive clipping in DP-SGD from an empirical perspective. With extensive experiments, we were able to gain some fresh insights and proposed two new adaptive clipping strategies based on them. We cross-compared the existing methods and our new strategies experimentally. Results showed that our strategies did provide a substantial improvement in model accuracy, and outperformed the state-of-the-art adaptive clipping methods consistently. Volume37, Issue11November 2022Pages 9674-9700 RelatedInformation