Parallel Successive Learning for Dynamic Distributed Model Training Over Heterogeneous Wireless Networks

Authors:
raw_author_name: Seyyedali Hosseinalipour
raw_affiliation_string: Department of Electrical Engineering, University at Buffalo-SUNY, Buffalo, NY, USA
raw_author_name: Nicolò Michelusi
raw_affiliation_string: School of Electrical, Computer, Energy Engineering, Arizona State University, Tempe, AZ, USA
raw_author_name: Vaneet Aggarwal
raw_affiliation_string: Schools of Industrial Engineering and Electrical and Computer Engineering, Purdue University, West Lafayette, USA
raw_author_name: Christopher G. Brinton
raw_affiliation_string: School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA
raw_author_name: David J. Love
raw_affiliation_string: School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA
raw_author_name: Mung Chiang
raw_affiliation_string: School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA

Abstract: Federated learning (FedL) has emerged as a popular technique for distributing model training over a set of wireless devices, via iterative local updates (at devices) and global aggregations (at the server). In this paper, we develop <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">parallel successive learning</i> (PSL), which expands the FedL architecture along three dimensions: (i) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Network</i> , allowing decentralized cooperation among the devices via device-to-device (D2D) communications. (ii) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Heterogeneity</i> , interpreted at three levels: (ii-a) Learning: PSL considers heterogeneous number of stochastic gradient descent iterations with different mini-batch sizes at the devices; (ii-b) Data: PSL presumes a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">dynamic environment</i> with data arrival and departure, where the distributions of local datasets evolve over time, captured via a new metric for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">model/concept drift</i> . (ii-c) Device: PSL considers devices with different computation and communication capabilities. (iii) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Proximity</i> , where devices have different distances to each other and the access point. PSL considers the realistic scenario where global aggregations are conducted with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">idle times</i> in-between them for resource efficiency improvements, and incorporates <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">data dispersion</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">model dispersion with local model condensation</i> into FedL. Our analysis sheds light on the notion of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cold</i> vs. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">warmed up</i> models, and model <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">inertia</i> in distributed machine learning. We then propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">network-aware dynamic model tracking</i> to optimize the model learning vs. resource efficiency tradeoff, which we show is an NP-hard signomial programming problem. We finally solve this problem through proposing a general optimization solver. Our numerical results reveal new findings on the interdependencies between the idle times in-between the global aggregations, model/concept drift, and D2D cooperation configuration.