The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training

1 INTRODUCTION . Most recent breakthroughs in deep learning are fairly achieved with the increased complexity of over-parameterized networks ( Brown et al. , 2020 ; Raffel et al. , 2020 ; Dosovitskiy et al. , 2021 ; Fedus et al. , 2021. arXiv:2101.03961 ; Jumper et al. , 2021 ; Berner et al. , 2019 ) . It is well-known that large models train better ( Neyshabur et al. , 2019 ; Novak et al. , 2018 ; Allen-Zhu et al. , 2019 ) , generalize better ( Hendrycks & Dietterich , 2019 ; Xie & Yuille , 2020 ; Zhao et al. , 2018 ) , and transfer better ( Chen et al. , 2020 ; 2021 ) . However , the upsurge in the investment of large models exacerbates the gap between research and real-life applications since many practical applications demand compact and efficient networks . Neural network pruning , since proposed by ( Mozer & Smolensky , 1989 ; Janowsky , 1989 ) , has evolved as the most common technique in literature to reduce the computational and memory requirements of neural networks . Over the past few years , numerous pruning criteria have been proposed , including magnitude ( Mozer & Smolensky , 1989 ; Han et al. , 2015 ; Frankle & Carbin , 2019 ; Mocanu et al. , 2018 ) , Hessian ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ) , mutual information ( Dai et al. , 2018 ) , Taylor expansion ( Molchanov et al. , 2016 ) , movement ( Sanh et al. , 2020 ) , connection sensitivity ( Lee et al. , 2019 ) , etc . Motivated for different scenarios , pruning can occur after training ( Han et al. , 2015 ; Frankle & Carbin , 2019 ; Molchanov et al. , 2016 ; Lee et al. , 2021 ) , during training ( Zhu & Gupta , 2017 ; Gale et al. , 2019 ; Louizos et al. , 2018 ) , and even before training ( Mocanu et al. , 2018 ; Lee et al. , 2019 ; Gale et al. , 2019 ) . The last regime can be further categorized into “ static sparse training ” ( Mocanu et al. , 2016 ; Gale et al. , 2019 ; Lee et al. , 2019 ; Wang et al. , 2020 ) and “ dynamic sparse training ” ( Mocanu et al. , 2018 ; Bellec et al. , 2018 ; Evci et al. , 2020a ; Liu et al. , 2021b ) . While random pruning is a universal method that can happen at any stage of training , training a randomly pruned network from scratch is arguably the most appealing way , owing to its “ end-to-end ” saving potential for the entire training process besides the inference . Due to this reason , we focus on random pruning at initialization ( or sparse training ) in this paper . When new pruning approaches bloom , random pruning naturally becomes their performance ’ s empirical “ lower bound ” since the connections are randomly chosen without any good reasoning . Likely due to the same reason , the results of random pruning at initialization ( as “ easy to beat ” baselines to support fancier new pruning methods ) reported in the literature are unfortunately vague , often inconsistent , and sometimes casual . For instance , it is found in Liu et al . ( 2020b ) that randomly pruned sparse networks can be trained from scratch to match the full accuracy of dense networks with only 20 % parameters , whereas around 80 % parameters are required to do so in Frankle et al . ( 2021 ) . The differences may arise from architecture choices , training recipes , distribution hyperparameters/layer-wise ratios , and so on . In most pruning literature ( Gale et al. , 2019 ; Lee et al. , 2019 ; Frankle et al. , 2021 ; Tanaka et al. , 2020 ) , random pruning usually refers to randomly removing the same proportion of parameters per layer , ending up with uniform layer-wise sparsities . Nevertheless , researchers have explored other pre-defined layer-wise sparsities , e.g. , uniform+ ( Gale et al. , 2019 ) , Erdős-Rényi random graph ( ER ) ( Mocanu et al. , 2018 ) , and Erdős-Rényi-Kernel ( ERK ) ( Evci et al. , 2020a ) . These layerwise sparsities naturally fit the category of random pruning , as they require no training to obtain the corresponding sparsity ratios ( followed by sampling randomly ) . We assess random pruning at initialization with these layer-wise sparsity ratios , in terms of various perspectives besides the predictive accuracy . Our main findings during this course of study are summarized below : • We find that the network size matters for the effectiveness of random pruning at initialization . With small networks , random pruning at initialization hardly matches the full accuracy even at mild sparsities ( 10 % , 20 % ) . However , as the networks grow wider and deeper , the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent , even at high sparsity ratios . • We further identify that appropriate layer-wise sparsity ratios can be an important booster for the performance of random pruning at initialization , particularly for large networks . We investigate several options to pre-define layer-wise sparsity ratios before any training ; one of them is able to push the performance of a completely random sparse Wide ResNet-50 to the densely trained Wide ResNet-50 on ImageNet . • We systematically assess the performance of random pruning at initialization and are somehow surprised by its accuracy and robustness . The accuracy achieved by ERK ratio can even surpass the ones learned by complex criteria , e.g. , SNIP , GraSP . In addition , randomly pruned sparse networks are found to outperform dense networks in other favorable aspects , such as out-of-distribution ( OoD ) performance , adversarial robustness , and uncertainty estimation . 2 RELATED WORK . 2.1 STATIC SPARSE TRAINING . Static sparse training represents a class of methods that aim to train a sparse subnetwork with a fixed sparse connectivity pattern during the course of training . We divide the static sparse training into random pruning and non-random pruning according to whether the connection is randomly selected . Random Pruning . Static sparse training with random pruning samples masks within each layer in a random fashion based on pre-defined layer-wise sparsities . The most naive approach is pruning each layer uniformly with the same pruning ratio , i.e. , uniform pruning ( Mariet & Sra , 2016. arXiv:1511.05077 ; He et al. , 2017 ; Suau et al. , 2019 ; Gale et al. , 2019 ) . Mocanu et al . ( 2016 ) proposed a non-uniform and scale-free topology , showing better performance than the dense counterpart when applied to restricted Boltzmann machines ( RBMs ) . Later , expander graphs were introduced to build sparse CNNs and showed comparable performance against the corresponding dense CNNs ( Prabhu et al. , 2018 ; Kepner & Robinett , 2019 ) . While not initially designed for static sparse training , ER ( Mocanu et al. , 2018 ) and ERK ( Evci et al. , 2020a ) are two advanced layer-wise sparsities introduced from the field of graph theory with strong results . Non-Random Pruning . Instead of pre-choosing a sparsity ratio for each layer , many works utilize the proposed saliency criteria to learn the layer-wise sparsity ratios before training , also termed as pruning at initialization ( PaI ) . Lee et al . ( 2019 ) first introduced SNIP that chooses structurally important connections at initialization via the proposed connection sensitivity . Following SNIP , many efficient criteria have been proposed to improve the performance of non-random pruning at initialization , including but not limited to gradient flow ( GraSP ; Wang et al . ( 2020 ) ) , synaptic strengths ( SynFlow ; Tanaka et al . ( 2020 ) ) , neural tangent kernel ( Liu & Zenke , 2020 ) , and iterative SNIP ( de Jorge et al. , 2021 ; Verdenius et al. , 2020 ) . 2.2 DYNAMIC SPARSE TRAINING . In contrast to static sparse training , dynamic sparse training stems from randomly initialized sparse subnetworks , and meanwhile dynamically explores new sparse connectivity during training . Dynamic sparse training starts from Sparse Evolutionary Training ( SET ) ( Mocanu et al. , 2018 ; Liu et al. , 2020a ) which initializes the sparse connectivity with Erdős-Rényi ( Erdős & Rényi , 1959 ) topology and periodically explores the sparse connectivity via a prune-and-grow scheme during the course of training . While there exist numerous pruning criteria in the literature , simple magnitude pruning typically performs well in the field of dynamic sparse training . On the other hand , the criteria used to grow weights back differs from method to method , including randomness ( Mocanu et al. , 2018 ; Mostafa & Wang , 2019 ) , momentum ( Dettmers & Zettlemoyer , 2019 ) , gradient ( Evci et al. , 2020a ; Jayakumar et al. , 2020 ; Liu et al. , 2021b ) . Besides the prune-and-grow scheme , layer-wise sparsities are vital to achieving high accuracy . ( Mostafa & Wang , 2019 ; Dettmers & Zettlemoyer , 2019 ) reallocates weights across layers during training based on reasonable heuristics , demonstrating performance improvement . Evci et al . ( 2020a ) extended ER to CNNs and showed considerable performance gains to sparse CNN training with the Erdős-Rényi-Kernel ( ERK ) ratio . Liu et al . ( 2021a ) recently showed that starting from a relatively lower sparsity ( 50 % ) rather than the target sparsity , provides a larger exploration space for DST , enables better sparsity ratios with higher accuracy but fewer FLOPs . Even though dynamic sparse training achieves promising sparse training performance , it changes the sparse connectivity during training and is thus out of the scope of random pruning . The most closely related works of our work are Su et al . ( 2020 ) and Frankle et al . ( 2021 ) . Su et al . ( 2020 ) proposed several sanity-check and showed that SNIP , GraSP , and Lottery Tickets ( LT ) 1 hardly exploit any information from the training data and are very robust to mask shuffling . They further show that random pruning at initialization with hand-designed layer-wise sparsity ratios outperforms or attains a similar performance compared with PaI methods . Frankle et al . ( 2021 ) further confirmed that what PaI discovers are the layer-wise sparsities rather than the specific weights or values , whereas magnitude pruning after training learns both , reflecting a broader challenge inherent to pruning at initialization . While prior works have observed that random pruning can be more competitive in certain cases ( Mocanu et al. , 2018 ; Liu et al. , 2020b ; Su et al. , 2020 ; Frankle et al. , 2021 ) , they did not give principled guidelines on when and how it can become that good ; nor do they show it can match the performance of dense networks on ImageNet . Standing on the shoulders of giants , our work summarized principles by more rigorous studies , and demonstrate the strongest result so far , that randomly pruned sparse Wide ResNet-50 can be sparsely trained to match a dense Wide ResNet-50 , on ImageNet . Moreover , compared with the ad-hoc sparsity ratios used in ( Su et al. , 2020 ) , we show that ERK ( Evci et al. , 2020a ) and our modified ERK+ are more general sparsity ratios that consistently demonstrate competitive performance without careful layer-wise sparsity design for every architecture , while being data free , feed-forward free , and dense initialization free .

Summary: This paper revisits the OG and the simplest pruning baselines out there: Random Pruning. Random Pruning randomly zeros out the weights in a network with a target budget and is often used at a layer level based on the chosen sparsity ratios.   This paper is a detailed study into Random Pruning. As put forward in contributions, the discussion includes the dependence on network size, the importance of sparsity ratios, and lastly the general-purpose aspects like robustness and OOD, etc of these pruned networks.   The observations seem to point to the fact that as the model size increases the pruning method becomes obsolete and everything converges to the same solution. They also show that sparsity ratios matter more than anything else for pruning techniques. These two observations have been made in the past in different settings but still are valid observations.   Lastly, the paper extensively tries to evaluate the robustness aspects of these models + sparsity ratios for CIFAR.