Asynchronous Stochastic Subgradient Methods for General Nonsmooth Nonconvex Optimization

Asynchronous distributed methods are a popular way to reduce the communication and synchronization costs of large-scale optimization . Yet , for all their success , little is known about their convergence guarantees in the challenging case of general non-smooth , non-convex objectives , beyond cases where closed-form proximal operator solutions are available . This is all the more surprising since these objectives are the ones appearing in the training of deep neural networks . In this paper , we introduce the first convergence analysis covering asynchronous methods in the case of general non-smooth , non-convex objectives . Our analysis applies to stochastic sub-gradient descent methods both with and without block variable partitioning , and both with and without momentum . It is phrased in the context of a general probabilistic model of asynchronous scheduling accurately adapted to modern hardware properties . We validate our analysis experimentally in the context of training deep neural network architectures . We show their overall successful asymptotic convergence as well as exploring how momentum , synchronization , and partitioning all affect performance . 1 INTRODUCTION . Training parameters arising in Deep Neural Net architectures is a difficult problem in several ways ( Goodfellow et al. , 2016 ) . First , with multiple layers and nonlinear activation functions such as sigmoid and softmax functions , the ultimate optimization problem is nonconvex . Second , with ReLU activation functions and max-pooling in convolutional structures , the problem is nonsmooth , i.e. , it is not differentiable everywhere , although typically the set of non-differentiable points is a set of measure zero in the space of the parameters . Finally , in many applications it is unreasonable to load the whole sample size in memory to evaluate the objective function or ( sub ) gradient , thus samples must be taken , necessitating analysis in a probabilistic framework . The analysis of parallel optimization algorithms using shared memory architectures , motivated by applications in machine learning , was ushered in by the seminal work of Recht et al . ( 2011 ) ( although precursors exist , see the references therein ) . Further work refined this analysis , e.g . ( Liu & Wright , 2015 ) and expanded it to nonconvex problems , e.g . ( Lian et al. , 2015 ) . However , in all of these results , a very simplistic model of asynchronous computation is presented to analyze the problem . Notably , it is assumed that every block of the parameter , among the set of blocks of iterates being optimized , has a fixed , equal probability of being chosen at every iteration , with a certain vector of delays that determine how old each block is that is stored in the cache relative to the shared memory . As one can surmise , this implies complete symmetry with regards to cores reading and computing the different blocks . This does not correspond to asynchronous computation in practice . In particular , in the common Non-Uniform Memory Access ( NUMA ) setting , practical experience has shown that it can be effective for each core to control a set of blocks . Thus , the choice of blocks will depend on previous iterates , which core was last to update , creating probabilistic dependence between the delay vector and the choice of block . This exact model is formalized in Cannelli et al. , which introduced a new probabilistic model of asynchronous parallel optimization and presented a coordinate-wise updating successive convex approximation algorithm . In this paper , we are interested in studying parallel asynchronous stochastic subgradient descent for general nonconvex nonsmooth objectives , such as the ones arising in the training of deep neural network architectures . Currently , there is no work in the literature specifically addressing this problem . The closest related work is given by Zhu et al . ( 2018 ) and Li & Li ( 2018 ) , which consider asynchronous proximal gradient methods for solving problems of the form f ( x ) + g ( x ) , where f is smooth and nonconvex , and g ( x ) is nonsmooth , with an easily computable closed form prox expression . This restriction applies to the case of training a neural network which has no ReLUs or max pooling in the architecture itself , i.e. , every activation is a smooth function , and there is an additional regularization term , such as an ` 1 . These papers derive expected rates of convergence . In the general case , where the activations themselves are nonsmooth—for instance in the presence of ReLUs—there is no such additive structure , and no proximal operator exists to handle away the non-smoothness and remove the necessity of computing and using subgradients explicitly in the optimization procedure . This general problem of nonsmooth nonconvex optimization is already difficult ( see , e.g. , Bagirov et al . ( 2014 ) ) , and the introduction of stochastically uncertain iterate updates creates an additional challenge . Classically , the framework of stochastic approximation , with stochastic estimates of the subgradient approximating elements in a differential inclusion that defines a flow towards minimization of the objective function , is a standard , effective approach to analyzing algorithms for this class of problems . Some texts on the framework include Kushner & Yin ( 2003 ) , which we shall reference extensively in the paper , and Borkar ( 2008 ) . See also Ermol ’ ev & Norkin ( 1998 ) and Ruszczyński ( 1987 ) for some classical results in convergence of stochastic algorithms for nonconvex nonsmooth optimization . Interest in stochastic approximation has resurfaced recently sparked by the popularity of Deep Neural Network architectures . For instance , see the analysis of nonconvex nonsmooth stochastic optimization with an eye towards such models in Davis et al . ( 2018 ) and Majewski et al . ( 2018 ) . In this paper , we provide the first analysis for nonsmooth nonconvex stochastic subgradient methods in a parallel asynchronous setting , in the stochastic approximation framework . For this , we employ the state of the art model of parallel computation introduced in Cannelli et al. , which we map onto the analysis framework of Kushner & Yin ( 2003 ) . We prove show that the generic asynchronous stochastic subgradient methods considered are convergent , with probability 1 , for nonconvex nonsmooth functions . This is the first result for this class of algorithms , and it combines the state of the art in these two areas , while extending the scope of the results therein . Furthermore , given the success of momentum methods ( see , e.g. , Zhang et al . ( 2017 ) ) , we consider the momentum variant of the classical subgradient method , again presenting the first convergence analysis for this class of algorithms . We validate our analysis numerically by demonstrating the performance of asynchronous stochastic subgradient methods of different forms on the problem of ResNet deep network training . We shall consider variants of asynchronous updating with and without write locks and with and without block variable partitioning , showing the nuances in terms of convergence behavior as depending on these strategies and properties of the computational hardware . 2 PROBLEM FORMULATION . Consider the minimization problem min x f ( x ) , ( 1 ) where f : Rn → R is locally Lipschitz continuous ( but could be nonconvex and nonsmooth ) and furthermore , it is computationally infeasible to evaluate f ( x ) or an element of the Clarke subdifferential ∂f ( x ) . The problem ( 1 ) has many applications in machine learning , including the training of parameters in deep neural networks . In this setting , f ( x ) is loss function evaluated on some model with x as its parameters , and is dependant on input data A ∈ Rn×m and target values y ∈ Rm of high dimension , i.e. , f ( x ) = f ( x ; ( A , y ) ) , with x a parameter to optimize with respect to the loss function . In cases of practical interest , f is decomposable in finite-sum form , f ( x ) = 1 M M∑ i=1 l ( m ( x ; Ai ) ; yi ) where l : Rm × Rm → R represents the training loss and { ( Ai , yi ) } is a partition of ( A , y ) . We are concerned with algorithms that solve ( 1 ) in a distributed fashion , i.e. , using multiple processing cores . In particular , we are analyzing the following inconsistent read scenario : before computation begins , each core c is allocated a block of variables Ic , for which it is responsible to update . At each iteration the core modifies a block of variables ik , chosen randomly among Ic . Immediately after core c completes its k-th iteration , it updates the shared memory . A lock is only placed on the shared memory when a core writes to it , thus the process of reading may result in computations of the function evaluated at variable values that never existed in memory , e.g. , block 1 is read by core 1 , then core 3 updates block 2 , then core 1 reads block 2 , and now block 1 is operating on a vector with the values in blocks 1 and 2 not simultaneously at their present local values at any point in time in shared memory . We shall index iterations to indicate when a core writes a new set of values for the variable into memory . We let dk = { dkc1 , ... , dkcn } be the vector of delays for each component of the variable used to evaluate the subgradient estimate , thus the j-th component of x that is used in the computation of the update at k is actually not xkcj but x dkcj j . In this paper , we are interested in applying stochastic approximation methods , of which the classic stochastic gradient descent forms a special case . Since f in ( 1 ) is in general nonsmooth , we will exploit subgradient methods . Denote by ξk the set of mini-batches used to compute an element of the subgradient g ( ( xd kc 1 1 , ... , x dkcn n ) ; ξkc ) . The set of minibatches ξkc is chosen uniformly at random from ( A , y ) , independently at each iteration . By the central limit theorem , the error is asymptotically Gaussian as the total size of the data as well as the size of the mini-batches increases . Asynchronous System Specification . We consider a shared-memory system with p processes concurrently and asynchronously performing computations on independent compute-cores . We interchangeably use the terms process and core . The memory allows concurrent-read-concurrent-write ( CRCW ) 1 access . The shared-memory system offers word-sized atomic read and fetch-and-add ( faa ) primitives . Processes use faa to update the components of the variable . 2.1 ALGORITHM DESCRIPTION . We now recall the stochastic subgradient algorithm under asynchronous updating in Algorithm 1 , from the perspective of the individual cores . The update of the iterate performed by uikc = muikc + g kc ikc ; xkc+1 ikc = xkc ikc − ( 1−m ) γkcuikc where m is the momentum constant , required to satisfy 0 < m < 1 Algorithm 1 Asynchronous Stochastic Subgradient Method for an Individual Core Input : x0 , core c. 1 : while Not converged do 2 : Sample i from the variables Ic corresponding to c. Sample ξ . 3 : Read xkc from the shared memory 4 : Compute a subgradient estimate gkc , local to kc 5 : Write , with a lock , to the shared memory vector partition uikc = muikc + g kc ikc 6 : Update , with a lock , to the shared memory vector partition xikc = xikc − ( 1−m ) γkcuikc 7 : kc = kc + 1 8 : end while

Summary: This paper analyzes convergence of asynchronous methods on general non-smooth and non-convex functions (typically arising from deep leaning). Stochastic sub-gradient asynchronous methods are of particular challenging when coupled with complicated hardware behavior of modern NUMA architecture. To validate the analysis, and study the impact of momentum, variable partitioning, numerical experiments on deep learning training are given.