Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization

Modern deep neural networks are highly overparameterized , and often of huge sizes . A number of post-training model compression techniques , such as distillation , pruning and quantization , can reduce the size of network parameters by a substantial fraction with little loss in performance . However , training a small network of the post-compression size de novo typically fails to reach the same level of accuracy achieved by compression of a large network , leading to a widely-held belief that gross overparameterization is essential to effective learning . In this work , we argue that this is not necessarily true . We describe a dynamic sparse reparameterization technique that closed the performance gap between a model compressed through iterative pruning and a model of the post-compression size trained de novo . We applied our method to training deep residual networks and showed that it outperformed existing reparameterization techniques , yielding the best accuracy for a given parameter budget for training . Compared to existing dynamic reparameterization methods that reallocate non-zero parameters during training , our approach achieved better performance at lower computational cost . Our method is not only of practical value for training under stringent memory constraints , but also potentially informative to theoretical understanding of generalization properties of overparameterized deep neural networks . 1 INTRODUCTION . Deep neural network ’ s success is due in no small part to two of its surprising properties that starkly defy conventional wisdom . First , training of modern deep neural networks is a high-dimensional non-convex optimization problem , a kind prohibitively difficult in general from an optimization theoretic viewpoint . Yet first-order gradient-based methods , stochastic gradient descent ( SGD ) and its variants , prove to be very effective ; training seldom gets trapped in bad local minima . Second , overparameterization ( i.e . having more model parameters than training data examples ) does not undermine generalization ; highly overparameterized models in practice seldom overfit , a phenomenon at odds with traditional principles of statistical learning theory . Simply put , huge models are not hard to train , and trained huge models are not bad ones . Emerging evidence has attributed these effects to the geometry of high-dimensional loss landscapes of overparameterized deep neural networks ( Dauphin et al. , 2014 ; Choromanska et al. , 2014 ; Goodfellow et al. , 2014 ; Im et al. , 2016 ; Wu et al. , 2017 ; Liao & Poggio , 2017 ; Cooper , 2018 ; Novak et al. , 2018 ) , and to the implicit regularization properties of SGD ( Brutzkus et al. , 2017 ; Zhang et al. , 2018a ; Poggio et al. , 2017 ) , though a thorough theoretical understanding is not yet complete . On the other hand , practitioners , blessed in this reality , do not seem to shy away from building gigantic models . Models with parameters in the tens and hundreds of millions are common in practice , with the current record holder having a parameter count in the range of a hundred billion ( Shazeer et al. , 2017 ) . However , huge models are computationally expensive , both spatially ( in terms of memory requirements ) and temporally ( in terms of number of operations ) , at both training time and inference time . Thus , scalable methods for controlling the computational complexity of deep neural networks are key to making them more useful in practice . To achieve efficient inference , a number of well-established model compression techniques are highly effective in reducing the post-training model size with little degradation in performance . These include distillation ( e.g . Bucilua et al . ( 2006 ) ; Hinton et al . ( 2015 ) ) , quantization ( e.g . Hubara et al . ( 2016 ) ; McDonnell ( 2018 ) ) , low-rank decomposition ( e.g . Denil et al . ( 2013 ) ; Jaderberg et al . ( 2014 ) ) , and pruning ( e.g . Han et al . ( 2015 ) ; Zhang et al . ( 2018b ) ) , to name a few . Despite their success , post-training compression methods require the full overparameterized model to be trained in the first place . Training remains expensive . In order to make training more efficient , significant efforts have been invested into numerical innovations for training at limited precision ( e.g . Courbariaux et al . ( 2016 ) ; Köster et al . ( 2017 ) ) . In contrast , however , little progress has been made in effective training at significantly reduced number of parameters . One obvious alternative toward this goal is to seek novel network architectures , ones that are more parameter efficient 1 . In fact , recent architectural innovations in deep convolutional neural networks ( CNNs ) not only achieved better generalization performance , but parameter efficiency as well ( see Section 5 ) . Instead of inventing new networks , an alternative approach is to achieve higher parameter efficiency directly by reparameterizing an existing model architecture . In general , any differentiable reparameterization can be used to augment training of a given model . Let an original network ( or a layer therein ) be denoted by y = f ( x ; θ ) , parameterized by θ ∈ Θ. Reparameterize it by φ ∈ Φ through θ = g ( φ ; ψ ) , where g is differentiable w.r.t . φ but not necessarily w.r.t . ψ. Denote the reparameterized network by fψ , considering ψ as metaparameters 2 : y = fψ ( x ; φ ) , f ( x ; g ( φ ; ψ ) ) . ( 1 ) Training of fψ is done by backpropagation further through g , as ∂∂φ = ∂g ∂φ ∂ ∂g . If it is so chosen that dim ( Φ ) < dim ( Θ ) and fψ ≈ f in terms of its generalization performance , then fψ is a more parameter efficient approximation of f . Sparse reparameterization is a special case where g is a linear projection ; φ is the non-zero entries ( i.e . “ weights ” ) and ψ their indices ( i.e . “ connectivity ” ) in the original parameter θ . Likewise , parameter sharing ( or more generally linear constraints of parameters ) is a similar special case of linear reparameterization where φ is the tied parameters and ψ their indices . In this sense a convolution layer can be considered a reparameterization of a fully connected layer . Further , if metaparameters ψ are fixed during the course of training , the reparameterization is static , whereas if ψ is adjusted adaptively during training , we call it dynamic reparameterization . Now , the question is : given a full model f , is it possible to find a more efficient reparameterization fψ that , trained de novo , can generalize comparably well ? The success of various model compression techniques suggests that such reparameterizations might exist for most well-known models , and a recent study ( Frankle & Carbin , 2018 ) made successful post hoc identifications of sparse reparameterized small networks with precisely such properties . Nevertheless , attempts at training small networks de novo typically yield results significantly underperforming networks obtained by compressing larger models ( Zhu & Gupta , 2017 ) . This has led to a commonly held belief that gross overparameterization is necessary for effective training . Here we argue that this is not true by presenting a dynamic sparse reparameterization technique able to train sparse models de novo without the need to compress a large model , a desirable feature for training on memory- and power-constrained devices . Our contributions are listed as follows . 1 . We showed that it is possible to train a small sparse network directly without a larger than inference-time parameter footprint at any stages of training , yet still achieving generalization performance at least on par with post-training iterative pruning of large dense models , yielding the most parameter efficient model at a given sparsity . 2 . We showed that our method is more scalable and efficient , and leads to significantly better accuracy than existing dynamic sparse reparameterization training techniques . 1 We formally introduce the notion of parameter efficiency as follows : given a dataset and a task , if model family A achieves a specific level of generalization performance with fewer parameters than model family B , we say A is more parameter efficient than B at that performance level . 2 We use the term metaparameter to refer to the parameters ψ of the reparameterization function g. They differ from parameters φ in that they are not simultaneously optimized with model parameters , and they differ from hyperparameters in that they define meaningful structures of the model which are required for inference . 2 RELATED WORK . Training of differentiably reparameterized networks has been proposed in numerous studies before . Dense reparameterization Several dense reparameterization techniques sought to reduce the size of fully connected layers . These include low-rank decomposition ( Denil et al. , 2013 ) , fastfood transform ( Yang et al. , 2014 ) , ACDC transform ( Moczulski et al. , 2015 ) , HashedNet ( Chen et al. , 2015 ) , low displacement rank ( Sindhwani et al. , 2015 ) and block-circulant matrix parameterization ( Treister et al. , 2018 ) . Note that similar reparameterizations were also used to introduce certain algebraic properties to the parameters for purposes other than reducing model sizes , e.g . to make training more stable as in unitary evolution RNNs ( Arjovsky et al. , 2015 ) and in weight normalization ( Salimans & Kingma , 2016 ) , to inject inductive biases ( Thomas et al. , 2018 ) , and to alter ( Dinh et al. , 2017 ) or to measure ( Li et al. , 2018 ) properties of the loss landscape . All dense reparameterization methods to date are static . Sparse reparameterization Successful training of sparse reparameterized networks usually employs iterative pruning and retraining , e.g . Han et al . ( 2015 ) ; Narang et al . ( 2017 ) ; Zhu & Gupta ( 2017 ) 3 . Training typically starts with a large pre-trained model and sparsity is gradually increased during the course of fine-tuning . Training a small , static , and sparse model de novo always fared much worse than training a large one to begin with ( Zhu & Gupta , 2017 ) . Frankle & Carbin ( 2018 ) successfully identified small and sparse subnetworks post-training which , when trained in isolation , reached a similar accuracy as the enclosing big network . They further showed that these subnetworks were sensitive to initialization , and hypothesized that the role of overparameterization is to provide a large number of candidate subnetworks , thereby increasing the likelihood that one of these subnetworks will have the necessary structure and initialization needed for effective learning . Most closely related to our work are dynamic sparse reparameterization techniques that emerged only recently . Like ours , these methods adaptively alter , by certain heuristic rules , reparameterization during training . Sparse evolutionary training ( Mocanu et al. , 2018 ) used magnitude-based pruning and random growth at the end of each training epoch . NeST ( Dai et al. , 2017 ; 2018 ) iteratively grew and pruned parameters and neurons during training ; parameter growth was guided by gradient and pruning by magnitude . Deep rewiring ( Bellec et al. , 2017 ) combined sparse reparameterization with stochastic parameter updates for training . These methods were mostly concerned with sparsifying fully connected layers and applied to relatively small and shallow networks . We show that the method we propose in this paper is more scalable and computationally efficient than these previous approaches , while achieving better performance on deep convolutional networks .

Summary: The paper provides a dynamic sparse reparameterization method allowing small networks to be trained at a comparable accuracy as pruned network with (initially) large parameter spaces. Improper initialization along with a fewer number of parameters requires a large parameter model, to begin with (Frankle and Carbin, 2018).  The proposed method which is basically a global pooling followed by a tensorwise growth allocates free parameters using an efficient weight re-distribution scheme, uses an approximate thresholding method and provides automatic parameter re-allocation to achieve its goals efficiently. The authors empirically demonstrate their results on MNIST, CIFAR-10, and Imagenet and show that dynamic sparse provides higher accuracy than compressed sparse (and other) networks.