Understanding the effects of data parallelism and sparsity on neural network training

1 INTRODUCTION . Data parallelism is a straightforward and common approach to accelerate neural network training by processing training data in parallel using distributed systems . Being model-agnostic , it is applicable to training any neural networks , and the degree of parallelism equates to the size of mini-batch for synchronized settings , in contrast to other forms of parallelism such as task or model parallelism . While its utility has attracted much attention in recent years , however , distributing and updating large network models at distributed communication rounds still remains a bottleneck ( Dean et al. , 2012 ; Hoffer et al. , 2017 ; Goyal et al. , 2017 ; Smith et al. , 2018 ; Shallue et al. , 2019 ; Lin et al. , 2020 ) . Meanwhile , diverse approaches to compress such large network models have been developed , and network pruning – the sparsification process that zeros out many parameters of a network to reduce computations and memory associated with these zero values – has been widely employed ( Reed , 1993 ; Han et al. , 2015 ) . In fact , recent studies discovered that pruning can be done at initialization prior to training ( Lee et al. , 2019 ; Wang et al. , 2020 ) , and by separating the training process from pruning entirely , it not only saves tremendous time and effort in finding trainable sparse networks , but also facilitates the analysis of pruned sparse networks in isolation . Nevertheless , there has been little study concerning the subsequent training of these sparse networks , and various aspects of the optimization of sparse networks remain rather unknown as of yet . In this work , we focus on studying data parallelism and sparsity1 , and provide clear explanations for their effects on neural network training . Despite a surge of recent interest in their complementary 1For the purpose of this work , we equate data parallelism and sparsity to increasing batch size and pruning model parameters , respectively ; we explain these more in detail in Appendix E. benefits in modern deep learning , there is a lack of fundamental understanding of their effects . For example , Shallue et al . ( 2019 ) provide comprehensive yet empirical evaluations on the effect of data parallelism , while Zhang et al . ( 2019 ) use a simple noisy quadratic model to describe the effect ; for sparsity , Lee et al . ( 2020 ) approach the difficulty of training under sparsity solely from the perspective of initialization . In this regard , we first accurately measure their effects by performing extensive metaparameter search independently for each and every study case of batch size and sparsity level . As a result , we find a general scaling trend as the effect of data parallelism in training sparse neural networks , across varying sparsity levels and workloads of data set , model and optimization algorithm . Also , the critical batch size turns out to be no less with sparse networks , despite the general difficulty of training sparse networks . We formalize our observation and theoretically prove the effect of data parallelism based on the convergence properties of generalized stochastic gradient methods irrespective of sparsity levels . We take this result further to understand the effect of sparsity based on Lipschitz smoothness analysis , and find that pruning results in a sparse network whose gradient changes relatively too quickly . Notably , this result is developed under standard assumptions used in the optimization literature and generally applied to training using any stochastic gradient method with nonconvex objective and learning rate schedule . Being precise and general , our results could help understand the effects of data parallelism and sparsity on neural network training . 2 SETUP . We follow closely the experiment settings used in Shallue et al . ( 2019 ) . We describe more details including the scale of our experiments in Appendix B , and provide additional results in Appendix D. The code can be found here : https : //github.com/namhoonlee/effect-dps-public Experiment protocol . For a given workload ( data set , network model , optimization algorithm ) and study ( batch size , sparsity level ) setting , we measure the number of training steps required to reach a predefined goal error . We repeat this process for a budget of runs while searching for the best metaparameters involved in the optimization ( e.g. , learning rate , momentum ) , so as to record the lowest number of steps , namely steps-to-result , as our primary quantity of interest . To this end , we regularly evaluate intermediate models on the entire validation set for each training run . Workload and study . We consider the workloads as the combinations of the followings : ( data set ) MNIST , Fashion-MNIST , CIFAR-10 ; ( network model ) Simple-CNN , ResNet-8 ; ( optimization algorithm ) SGD , Momentum , Nesterov with either a fixed or decaying learning rate schedule . For the study setting , we consider a batch size from 2 up to 16384 and a sparsity level from 0 % to 90 % . Metaparameter search . We perform a quasi-random search to tune metaparameters efficiently . More precisely , we first generate Sobol low-discrepancy sequences in a unit hypercube and convert them into metaparameters of interest , while taking into account a predefined search space for each metaparameter . The generated values for each metaparameter is in length of the budget of trials , and the search space is designed based on preliminary experimental results . Pruning . Sparse networks can be obtained by many different ways , and yet , for the purpose of this work , they must not undergo any training beforehand so as to measure the effects of data parallelism while training from scratch . Recent pruning-at-initialization approaches satisfy this requirement , and we adopt the connection sensitivity criterion in Lee et al . ( 2019 ) to obtain sparse networks . 3 EXPERIMENTAL RESULTS . 3.1 MEASURING THE EFFECT OF DATA PARALLELISM . First of all , we observe in each and every sparsity level across different workloads a general scaling trend in the relationship between batch size and steps-to-result for the effects of data parallelism ( see the 1st and 2nd columns in Figure 1 ) : Initially , we observe a period of linear scaling where doubling the batch size reduces the steps to achieve the goal error by half ( i.e. , it aligns closely with the dashed line ) , followed by a region of diminishing returns where the reduction in the required number of steps by increasing the batch size is less than the inverse proportional amount ( i.e. , it starts to digress from the linear scaling region ) , which eventually arrives at a maximal data parallelism ( i.e. , it hits a plateau ) where increasing the batch size no longer decreases the steps to reach a goal error . The same trend is observed across various workloads of data set , network model , and optimization algorithm as well as different goal errors ( see Appendix D ) . We note that our observation is consistent with the results of regular network training presented in Shallue et al . ( 2019 ) ; Zhang et al . ( 2019 ) . When we put the results for all sparsity levels together , we observe that training a sparser network takes a longer time ; a data parallelism curve for higher sparsity usually lies above than that for lower sparsity ( see the 3rd column in Figure 1 ) . For example , compared to the case of sparsity 0 % ( i.e. , the dense , over-parameterized network ) , 90 % sparse network takes about 2 − 4 times longer training time ( or the number of training steps ) , consistently across different batch sizes ( see Figure 12 in Appendix D.1 for more precise comparisons ) . Recall that we tune all metaparameters independently for each and every study case of batch size and sparsity level without relying on a single predefined training rule in order to find the best steps-to-result . Therefore , this result on the one hand corroborates the general difficulty of training sparse neural networks against the ease of training overly parameterized neural networks . On the other hand , when we normalize the y-axis of each plot by dividing by the number of steps at the first batch size , we can see the phase transitions more clearly . As a result , we find that the regions of diminishing returns and maximal data parallelism appear no earlier when training sparse networks than the dense network ( see the 4th column in Figure 1 ) . This is quite surprising in that one could have easily guessed that the general optimization difficulty incurred by sparsification may influence the data parallelism too , at least to some degree ; however , it turns out that the effects of data parallelism on sparse network training remain no worse than the dense case . Moreover , notice that in many cases the breakdown of linear scaling regime occurs even much later at larger batch sizes for a higher sparsity case ; this is especially evident for Momentum and Nesterov optimizers ( e.g. , compare training 90 % sparse network using Momentum against 0 % dense network ) . In other words , for sparse networks , a critical batch size can be larger , and hence , when it comes to training sparse neural networks , one can increase the batch size ( or design a parallel computing system for distributed optimization ) more effectively , while better exploiting given resources . We find this result particularly promising since SGD with momentum is often the method of choice in practice . We further show that momentum optimizers being capable of exploiting large batch sizes hold the same across different sparsity levels by displaying all plots together in Figure 2 . Overall , we believe that it is important to confirm the robustness of the data parallelism in sparse neural network training , which has been unknown thus far and difficult to estimate a priori . 3.2 ANALYZING METAPARAMETER SEARCH . In this section , we analyze the metaparameter search used to measure the effect of data parallelism . We specifically investigate the workload { MNIST , Simple-CNN , Momentum } where there are two metaparameters to tune ( i.e. , learning rate and momentum ) , to visualize all metaparameters easily in a 2D figure ( see Appendix D.2 for other results ) . The results are presented in Figure 3 , and we summarize our key findings below : • Our quasi-random search samples metaparameters efficiently , so that they are distributed evenly ( without being cluttered in a log-space ) and flexibly ( rather than sitting in a grid with fixed spac- ing ) within the search spaces . Also , the best metaparameters to yield lowest steps ( marked by gold star ? ) are located in the middle of the search ranges rather than sitting at the search boundaries across different batch sizes and sparsity levels . This means that our experiments are designed reasonably well , and the results are reliable . • There are two distinguished regions ( i.e. , complete ( ) and incomplete ( N ) ) being separated by a seemingly linear boundary as per the relationship between learning rate and momentum . This indicates that the optimization is being done by an interplay between these two metaparameters ; if one metaparameter is not chosen carefully with respect to the other ( e.g. , increase learning rate for fixed momentum ) , the optimizer may be stuck in a region spending time oscillating and eventually results in incomplete runs . This highlights the importance of performing metaparameter search , although it is expensive , rather than relying on predetermined heuristic training strategies , in order to accurately measure the effect of data parallelism and avoid potentially suboptimal results . • The successful region ( filled with blue circles ) becomes larger as with increasing batch size , showing that large batch training reaches a given goal error in less number of training iterations than small batch training , and hence , yields more complete runs . Notably , the best learning rate tends to increase as with increasing batch size too . This aligns well with the classic result in learning theory that large batch training allows using bigger learning rates ( Robbins & Monro , 1951 ; Bottou , 1998 ; Krizhevsky , 2014 ) .

Summary: The manuscript studies the effect of batch size at different sparsity levels (achieved by applying connection sensitivity pruning) on the required number of optimisation steps to reach a certain accuracy. The goal is to understand the interplay between those fundamental parameters. The empirical evaluation is performed for different triples of dataset, network architecture and optimisation scheme. The theoretical analysis is based on established bounds on the expected gradient norm.