Multi-Level Local SGD: Distributed SGD for Heterogeneous Hierarchical Networks

1 INTRODUCTION . Stochastic Gradient Descent ( SGD ) is a key algorithm in modern Machine Learning and optimization ( Amari , 1993 ) . To support distributed data as well as reduce training time , Zinkevich et al . ( 2010 ) introduced a distributed form of SGD . Traditionally , distributed SGD is run within a huband-spoke network model : a central parameter server ( hub ) coordinates with worker nodes . At each iteration , the hub sends a model to the workers . The workers each train on their local data , taking a gradient step , then return their locally trained model to the hub to be averaged . Distributed SGD can be an efficient training mechanism when message latency is low between the hub and workers , allowing gradient updates to be transmitted quickly at each iteration . However , as noted in Moritz et al . ( 2016 ) , message transmission latency is often high in distributed settings , which causes a large increase in overall training time . A practical way to reduce this communication overhead is to allow the workers to take multiple local gradient steps before communicating their local models to the hub . This form of distributed SGD is referred to as Local SGD ( Lin et al. , 2018 ; Stich , 2019 ) . There is a large body of work that analyzes the convergence of Local SGD and the benefits of multiple local training rounds ( McMahan et al. , 2017 ; Wang & Joshi , 2018 ; Li et al. , 2019 ) . Local SGD is not applicable to all scenarios . Workers may be heterogeneous in terms of their computing capabilities , and thus the time required for local training is not uniform . For this reason , it can be either costly or impossible for workers to train in a fully synchronous manner , as stragglers may hold up global computation . However , the vast majority of previous work uses a synchronous model , where all clients train for the same number of rounds before sending updates to the hub ( Dean et al. , 2012 ; Ho et al. , 2013 ; Cipar et al. , 2013 ) . Further , most works assume a hub-and-spoke model , but this does not capture many real world settings . For example , devices in an ad-hoc network may not all be able to communicate to a central hub in a single hop due to network or communication range limitations . In such settings , a multi-level communication network model may be beneficial . In flying ad-hoc networks ( FANETs ) , a network architecture has been proposed to improve scalability by partitioning the UAVs into mission areas ( Bekmezci et al. , 2013 ) . Here , clusters of UAVs have their own clusterheads , or hubs , and these hubs communicate through an upper level network , e.g. , via satellite . Multi-level networks have also been utilized in Fog and Edge computing , a paradigm de- ∗T . Castiglia , A. Das , and S. Patterson are with the Department of Computer Science , Rensselaer Polytechnic Institute , 110 8th St , Troy , NY 12180 , castit @ rpi.edu , dasa2 @ rpi.edu , sep @ cs.rpi.edu . signed to improve data aggregation and analysis in wireless sensor networks , autonomous vehicles , power systems , and more ( Bonomi et al. , 2012 ; Laboratory , 2017 ; Satyanarayanan , 2017 ) . Motivated by these observations , we propose Multi-Level Local SGD ( MLL-SGD ) , a distributed learning algorithm for heterogeneous multi-level networks . Specifically , we consider a two-level network structure . The lower level consists of a disjoint set of hub-and-spoke sub-networks , each with a single hub server and a set of workers . The upper level network consists of a connected , but not necessarily complete , hub network by which the hubs communicate . For example , in a Fog Computing application , the sub-network workers may be edge devices connected to their local data center , and the data centers act as hubs communicating over a decentralized network . Each subnetwork runs one or more Local SGD rounds , in which its workers train for a local training period , followed by model averaging at the sub-network ’ s hub . Periodically , the hubs average their models with neighbors in the hub network . We model heterogeneous workers using a stochastic approach ; each worker executes a local training iteration in each time step with a probability proportional to its computational resources . Thus , different workers may take different numbers of gradient steps within each local training period . Note since MLL-SGD averages every local training period , regardless of how many gradient steps each worker takes , slow workers do not slow algorithm execution . We prove the convergence of MLL-SGD for smooth and potentially non-convex loss functions . We assume data is distributed in an IID manner to all workers . Further , we analyze the relationship between the convergence error and algorithm parameters and find that , for a fixed step size , the error is quadratic in the number of local training iterations and the number of sub-network training iterations , and linear in the average worker operating rate . Our algorithm and analysis are general enough to encompass several variations of SGD as special cases , including classical SGD ( Amari , 1993 ) , SGD with weighted workers ( McMahan et al. , 2017 ) , and Decentralized Local SGD with an arbitrary hub communication network ( Wang & Joshi , 2018 ) . Our work provides novel analysis of a distributed learning algorithm in a multi-level network model with heterogeneous workers . The specific contributions of this paper are as follows . 1 ) We formalize the multi-level network model with heterogeneous workers , and we define the MLL-SGD algorithm for training models in such a network . 2 ) We provide theoretical analysis of the convergence guarantees of MLLSGD with heterogeneous workers . 3 ) We present an experimental evaluation that highlights our theoretical convergence guarantees . The experiments show that in multi-level networks , MLL-SGD achieves a marked improvement in convergence rate over algorithms that do not exploit the network hierarchy . Further , when workers have heterogeneous operating rates , MLL-SGD converges more quickly than algorithms that require all workers to execute the same number of training steps in each local training period . The rest of the paper is structured as follows . In Section 2 , we discuss related work . Section 3 introduces the system model and problem formulation . We describe MLL-SGD in Section 4 , and we present our main theoretical results in Section 5 . Proofs of these results are deferred to the appendix . We provide experimental results in Section 6 . Finally , we conclude in Section 7 . 2 RELATED WORK . Distributed SGD is a well studied subject in Machine Learning . Zinkevich et al . ( 2010 ) introduced parallel SGD in a hub-and-spoke model . Variations on Local SGD in the hub-and-spoke model have been studied in several works ( Moritz et al. , 2016 ; Zhang et al. , 2016 ; McMahan et al. , 2017 ) . Many works have provided convergence bounds of SGD within this model ( Wang et al. , 2019b ; Li et al. , 2019 ) . There is also a large body of work on decentralized approaches for optimization using gradient based methods , dual averaging , and deep learning ( Tsitsiklis et al. , 1986 ; Jin et al. , 2016 ; Wang et al. , 2019a ) . These previous works , however , do not address a multi-level network structure . In practice , workers may be heterogeneous in nature , which means that they may execute training iterations at different rates . Lian et al . ( 2017 ) addressed this heterogeneity by defining a gossipbased asynchronous SGD algorithm . In Stich ( 2019 ) , workers are modeled to take gradient steps at an arbitrary subset of all iterations . However , neither of these works address a multi-level network model . Grouping-SGD ( Jiang et al. , 2019 ) considers a scenario where workers can be clustered into groups , for example , based on their operating rates . Workers within a group train in a synchronous manner , while the training across different groups may be asynchronous . The system model differs significantly from that in MLL-SGD in that as the model parameters are partitioned vertically across multiple hubs , and workers communicate with every hub . Several recent works analyze Hierarchical Local SGD ( HL-SGD ) , an algorithm for training a model in a hierarchical network . Different from MLL-SGD , HL-SGD assumes the hub network topology is a hub-and-spoke and also that workers are homogeneous . Zhou & Cong ( 2019 ) and Liu et al . ( 2020 ) analyze the convergence error of HL-SGD , while Abad et al . ( 2020 ) analyzes convergence time . Unlike HL-SGD , MLL-SGD accounts for an arbitrary hub communication graph , and MLL-SGD algorithm execution does not slow down in the presence of heterogeneous worker operating rates . Several other works seek to encapsulate many variations of SGD under a single framework . Koloskova et al . ( 2020 ) created a generalized model that considers a gossip-based decentralized SGD algorithm where the communication network is time-varying . However , this work does not account for a multi-level network model nor worker heterogeneity . Wang et al . introduced the Cooperative SGD framework ( Wang & Joshi , 2018 ) , a model that includes communication reduction through local SGD steps and decentralized mixing between homogeneous workers . Cooperative SGD also allows for auxiliary variables . These auxiliary variables can be used to model SGD in a multi-level network , but only when sub-network averaging is immediately followed by hubs averaging with their neighbors in the hub network . Our model is more general ; it considers heterogeneous workers and it allows for an arbitrary number of averaging rounds within each sub-network between averaging rounds across sub-networks , which is more practical in multi-level networks where inter-hub communication is slow or costly . 3 SYSTEM MODEL AND PROBLEM FORMULATION . In this section , we introduce our system model , the objective function that we seek to minimize , and the assumptions we make about the function . We consider a set of D sub-networks D = { 1 , . . . , D } . Each sub-network d ∈ D has a single hub and a set of workersM ( d ) , with |M ( d ) | = N ( d ) . Workers inM ( d ) only communicate with their own hub and not with any other workers or hubs . We define the set of all workers in the system as M = ⋃D d=1M ( d ) . Let |M | = N . Each worker i holds a set S ( i ) of local training data . Let S = ⋃N i=1 S ( i ) . The set of all D hubs is denoted C. The hubs communicate with one another via an undirected , connected communication graph G = ( C , E ) . Let Nd = { j | ed , j ∈ E } denote the set of neighbors of the hub in sub-network d in the hub graph G. Let the model parameters be denoted by x ∈ Rn . Our goal is to find an x that minimizes the following objective function over the training set : F ( x ) = 1 | S | ∑ s∈S f ( x ; s ) ( 1 ) where f ( · ) is the loss function . The workers collaboratively minimize this loss function , in part by executing local iterations of SGD over their training sets . For each executed local iteration , a worker samples a mini-batch of data uniformly at random from its local data . Let ξ be a randomly sampled mini-batch of data and let g ( x ; ξ ) = 1|ξ| ∑ s∈ξ∇f ( x ; s ) be the mini-batch gradient . For simplicity , we use g ( x ) instead of g ( x ; ξ ) from here on . Assumption 1 . The objective function and the mini-batch gradients satisfy the following : 1a The objective function F : Rn → R is continuously differentiable , and the gradient is Lipschitz with constant L > 0 , i.e. , ‖∇F ( x ) −∇F ( y ) ‖2 ≤ L‖x− y‖2 for all x , y ∈ Rn . 1b The function F is lower bounded , i.e. , F ( x ) ≥ Finf > −∞ for all x ∈ Rn . 1c The mini-batch gradients are unbiased , i.e. , Eξ|x [ g ( x ) ] = ∇F ( x ) for all x ∈ Rn . 1d There exist scalars β ≥ 0 and σ ≥ 0 such that Eξ|x‖g ( x ) −∇F ( x ) ‖22 ≤ β||∇F ( x ) ||22+σ2 for all x ∈ Rn . Assumption 1a requires that the gradients do not change too rapidly , and Assumption 1b requires that our objective function is lower bounded by some Finf . Assumptions 1c and 1d assume that Algorithm 1 Multi-Level Local SGD 1 : Initialize : y ( d ) 1 for hubs d = 1 , . . . , D 2 : for k = 1 , . . . , K do 3 : parallel for d ∈ D do 4 : parallel for i ∈M ( d ) do 5 : x ( i ) k ← y ( d ) k . Workers receive updated model from hub 6 : for j = k , . . . , k + τ − 1 do 7 : x ( i ) k+1 ← x ( i ) k − ηg ( i ) k . Local iteration ( probabilistic ) 8 : end for 9 : end parallel for 10 : z ( d ) ← ∑ i∈M ( d ) v ( i ) x ( i ) k+1 . Hub d computes average of its workers ’ models 11 : if k mod q · τ = 0 then 12 : y ( d ) k+1 ← ∑ j∈N ( d ) Hj , dz ( j ) . Hub d averages its model with neighboring hubs 13 : else 14 : y ( d ) k+1 ← z ( d ) 15 : end if 16 : end parallel for 17 : end for the local data at each worker can be used as an unbiased estimate for the full dataset with the same bounded variance . These assumptions are common in convergence analysis of SGD algorithms ( e.g. , Bottou et al . ( 2018 ) ) .

Summary: This paper proposes a new variant of local SGD algorithm to make it be more realistic. In particular, (1) it allows workers to perform different number of local steps, depending on their computational resources; (2) workers are organized in a multi-level structure. Workers connected to one central hub can synchronize frequently and hubs are communicated in an infrequent and decentralized manor.