ANYTIME MINIBATCH: EXPLOITING STRAGGLERS IN ONLINE DISTRIBUTED OPTIMIZATION

1 INTRODUCTION . The advent of massive data sets has resulted in demand for solutions to optimization problems that are too large for a single processor to solve in a reasonable time . This has led to a renaissance in the study of parallel and distributed computing paradigms . Numerous recent advances in this field can be categorized into two approaches ; synchronous Dekel et al . ( 2012 ) ; Duchi et al . ( 2012 ) ; Tsianos & Rabbat ( 2016 ) ; Zinkevich et al . ( 2010 ) and asynchronous Recht et al . ( 2011 ) ; Liu et al . ( 2015 ) . This paper focuses on the synchronous approach . One can characterize synchronization methods in terms of the topology of the computing system , either master-worker or fully distributed . In a master-worker topology , workers update their estimates of the optimization variables locally , followed by a fusion step at the master yielding a synchronized estimate . In a fully distributed setting , nodes are sparsely connected and there is no obvious master node . Nodes synchronize their estimates via local communications . In both topologies , synchronization is a key step . Maintaining synchronization in practical computing systems can , however , introduce significant delay . One cause is slow processing nodes , known as stragglers Dean et al . ( 2012 ) ; Yu et al . ( 2017 ) ; Tandon et al . ( 2017 ) ; Lee et al . ( 2018 ) ; Pan et al . ( 2017 ) ; S. Dutta & Nagpurkar ( 2018 ) . A classical requirement in parallel computing is that all nodes process an equal amount of data per computational epoch prior to the initiation of the synchronization mechanism . In networks in which the processing speed and computational load of nodes vary greatly between nodes and over time , the straggling nodes will determine the processing time , often at a great expense to overall system efficiency . Such straggler nodes are a significant issue in cloud-based computing systems . Thus , an important challenge is the design of parallel optimization techniques that are robust to stragglers . To meet this challenge , we propose an approach that we term Anytime MiniBatch ( AMB ) . We consider a fully distributed topologyand consider the problem of stochastic convex optimization via dual averaging Nesterov ( 2009 ) ; Xiao ( 2010 ) . Rather than fixing the minibatch size , we fix the computation time ( T ) in each epoch , forcing each node to “ turn in ” its work after the specified fixed time has expired . This prevents a single straggler ( or stragglers ) from holding up the entire network , while allowing nodes to benefit from the partial work carried out by the slower nodes . On the other hand , fixing the computation time means that each node process a different amount of data in each epoch . Our method adapts to this variability . After computation , all workers get fixed communication time ( Tc ) to share their gradient information via averaging consensus on their dual variables , accounting for the variable number of data samples processed at each node . Thus , the epoch time of AMB is fixed to T + Tc in the presence of stragglers and network delays . We analyze the convergence of AMB , showing that the online regret achieves O ( √ m̄ ) performance , which is optimal for gradient based algorithms for arbitrary convex loss Dekel et al . ( 2012 ) . In here , m̄ is the expected sum number of samples processed across all nodes . We further show an upper bound that , in terms of the expected wall time needed to attain a specified regret , AMB is O ( √ n− 1 ) faster than methods that use a fixed minibatch size under the assumption that the computation time follows an arbitrary distribution where n is the number of nodes . We provide numerical simulations using Amazon Elastic Compute Cloud ( EC2 ) and show that AMB offers significant acceleration over the fixed minibatch approach . 2 RELATED WORK . This work contributes to the ever-growing body of literature on distributed learning and optimization , which goes back at least as far as Tsitsiklis et al . ( 1986 ) , in which distributed first-order methods were considered . Recent seminal works include Nedic & Ozdaglar ( 2009 ) , which considers distributed optimization in sensor and robotic networks , and Dekel et al . ( 2012 ) , which considers stochastic learning and prediction in large , distributed data networks . A large body of work elaborates on these ideas , considering differences in topology , communications models , data models , etc . Duchi et al . ( 2012 ) ; Tsianos et al . ( 2012 ) ; Shi et al . ( 2015 ) ; Xi & Khan ( 2017 ) . The two recent works most similar to ours are Tsianos & Rabbat ( 2016 ) and Nokleby & Bajwa ( 2017 ) , which consider distributed online stochastic convex optimization over networks with communications constraints . However , both of these works suppose that worker nodes are homogeneous in terms of processing power , and do not account for the straggler effect examined herein . The recent work Pan et al . ( 2017 ) ; Tandon et al . ( 2017 ) ; S. Dutta & Nagpurkar ( 2018 ) proposed synchronous fixed minibatch methods to mitigate stragglers for master-worker setup . These methods either ignore stragglers or use redundancy to accelerate convergence in the presence of stragglers . However , our approach in comparison to Pan et al . ( 2017 ) ; Tandon et al . ( 2017 ) ; S. Dutta & Nagpurkar ( 2018 ) utilizes work completed by both fast and slow working nodes , thus results in faster wall time in convergence . 3 SYSTEM MODEL AND ALGORITHM . In this section we outline our computation and optimization model and step through the three phases of the AMB algorithm . The pseudo code of the algorithm is provided in App . A . We defer discussion of detailed mathematical assumptions and analytical results to Sec . 4 . We suppose a computing system that consists of n compute nodes . Each node corresponds to a vertex in a connected and undirected graph G ( V , E ) that represents the inter-node communication structure . The vertex set V satisfies |V | = n and the edge set E tells us which nodes can communicate directly . Let Ni = { j ∈ V : ( i , j ) ∈ E , i 6= j } denote the neighborhood of node i . The collaborative objective of the nodes is to find the parameter vector w ∈W ⊆ Rd that solves w∗ = arg min w∈W F ( w ) where F ( w ) : = Ex [ f ( w , x ) ] . ( 1 ) The expectation Ex [ · ] is computed with respect to an unknown probability distribution Q over a set X ⊆ Rd . Because the distribution is unknown , the nodes must approximate the solution in ( 1 ) using data points drawn in an independent and identically distributed ( i.i.d . ) manner from Q. AMB uses dual averaging Nesterov ( 2009 ) ; Dekel et al . ( 2012 ) as its optimization workhorse and averaging consensus Nokleby & Bajwa ( 2017 ) ; Tsianos & Rabbat ( 2016 ) to facilitate collaboration among nodes . It proceeds in epochs consisting of three phases : compute , in which nodes compute local minibatches ; consensus , in which nodes average their dual variables together ; and update , in which nodes take a dual averaging step with respect to the consensus-averaged dual variables . We let t index each epoch , and each node i has a primal variable wi ( t ) ∈ Rd and dual variable zi ( t ) ∈ Rd . At the start of the first epoch , t = 1 , we initialize all primal variables to the same value w ( 1 ) as wi ( 1 ) = w ( 1 ) = arg min w∈W h ( w ) , ( 2 ) and all dual variables to zero , i.e. , zi ( 1 ) = 0 ∈ Rd . In here , h : W → R is a 1-strongly convex function . Compute Phase : All workers are given T fixed time to compute their local minibatches . During each epoch , each node is able to compute bi ( t ) gradients of f ( w , x ) , evaluated at wi ( t ) where the data samples xi ( t , s ) are drawn i.i.d . from Q . At the end of epoch t , each node i computes its local minibatch gradient : gi ( t ) = 1 bi ( t ) bi ( t ) ∑ s=1 ∇wf ( wi ( t ) , xi ( t , s ) ) . ( 3 ) As we fix the compute time , the local minibatch size bi ( t ) is a random variable . Let b ( t ) : = ∑n i=1 bi ( t ) be the global minibatch size aggregated over all nodes . This contrasts with traditional approaches in which the minibatch is fixed . In Sec . 4 we provide a convergence analysis that accounts for the variability in the amount of work completed by each node . In Sec . 5 , we presents a wall time analysis based on random local minibatch sizes . Consensus Phase : Between computational epochs each node is given a fixed amount of time , Tc , to communicate with neighboring nodes . The objective of this phase is for each node to get ( an approximation of ) the following quantity : 1 b ( t ) n∑ i=1 bi ( t ) [ zi ( t ) + gi ( t ) ] = 1 b ( t ) n∑ i=1 bi ( t ) zi ( t ) + 1 b ( t ) n∑ i=1 bi ( t ) ∑ s=1 ∇wf ( wi ( t ) , xi ( t , s ) ) : = z̄ ( t ) + g ( t ) . ( 4 ) The first term , z̄ ( t ) , is the weighted average of the previous dual variables . The second , g ( t ) , is the average of all gradients computed in epoch t. The nodes compute this quantity approximately via several synchronous rounds of average consensus . Each node waits until it hears from all neighbors before starting a consensus round . As we have fixed communication time Tc , the number of consensus rounds ri ( t ) varies across workers and epochs due to random network delays . Let P be a positive semi-definite , doubly-stochastic matrix ( i.e. , all entries of P are non-negative and all row- and column-sums are one ) that is consistent with the graph G ( i.e. , Pi , j > 0 only if i = j or if ( i , j ) ∈ E ) . At the start of the consensus phase , each node i shares its message m ( 0 ) i = nbi ( t ) [ zi ( t ) + gi ( t ) ] with its neighboring nodes . Let [ r ] stand for [ r ] ∈ { 1 , . . . r } . Then , in consensus iteration k ∈ [ ri ( t ) ] node i computes its update as m ( k ) i = n∑ j=1 Pi , jm ( k−1 ) j = n∑ j=1 ( Pi , j ) km ( 0 ) j . As long as G is connected and the second-largest eigenvalue of P is strictly less than unity , the iterations are guaranteed to converge to the true average . For finite ri ( t ) , each node will have an error in its approximation . Instead of ( 4 ) , at the end of the rounds of consensus , node i will have zi ( t+ 1 ) = z̄ ( t ) + g ( t ) + ξi ( t ) , ( 5 ) where ξi ( t ) is the error . We use D ( ri ( t ) ) ( { yj } j∈V , i ) to denote the distributed averaging affected by ri ( t ) rounds of consensus . Thus , zi ( t+ 1 ) = 1 b ( t ) D ( ri ( t ) ) ( { nbj ( t ) [ zj ( t ) + gj ( t ) ] ] } j∈V , i ) = 1 b ( t ) m ( ri ( t ) ) i . ( 6 ) We note that the updated dual variable zi ( t+ 1 ) is a normalized version of the distributed average solution , normalized by b ( t ) = ∑n i=1 bi ( t ) . Update Phase : After distributed averaging of dual variables , each node updates its primal variable as wi ( t+ 1 ) = arg min w∈W { 〈 w , zi ( t+ 1 ) 〉 + β ( t+ 1 ) h ( w ) } ; ( 7 ) where 〈· , ·〉 denotes the standard inner product . As will be discussed further in our analysis , in this paper we assume h : W → R to be a 1-strongly convex function and β ( t ) to be a sequence of positive non-decreasing parameters , i.e. , β ( t ) ≤ β ( t+ 1 ) . We also work in Euclidean space where h ( w ) = ‖w‖2 is a typical choice .

Summary: This paper studies distributed optimization in the presence of straggling computing nodes. In a synchronous distributed optimization approach, the stragglers delay the entire computation as the synchronization operation cannot be performed till every computing nod has completed its task. This paper aims to mitigate the effect of stragglers by proposing Anytime MiniBatch (AMB) approach, where each computing node is allowed to process the different number of samples between two synchronization steps. In particular, each node is given $T$ unit time to process as many samples as it can. After that, the nodes are allowed to aggregate the information among themselves through a consensus mechanism for another $T_c$ unit time. In contrast with this, the usual Fixed MiniBatch (FMB) approach requires each node to process a fixed number of samples before invoking aggregating step. The presence of stragglers can significantly increase the time between two synchronization step and slow down the overall optimization process.