Peek-a-Boo: What (More) is Disguised in a Randomly Weighted Neural Network, and How to Find It Efficiently

1 INTRODUCTION . Recent years have seen substantial efforts devoted to scaling neural networks ( NNs ) to enormous sizes ( He et al. , 2016 ; Devlin et al. , 2019 ) . Parameter-counts are frequently measured in billions rather than millions , with the time and financial outlay necessary to train these models growing in concert . Sparse NNs , whose large portions of parameters are zero , have been studied to address those gaps , saving storage , memory and computational costs . Conventional approaches first train dense NNs , and then prune the trained NNs to high levels of sparsity ( Han et al. , 2015 ; Guo et al. , 2016 ) . Those methods significantly reduce the inference complexity yet cost even greater computational resources and memory footprints at training . An emerging field has explored what roles sparsity can play in neural networks , spawning different directions of research . One of them follows the typical pruning-and-finetuning pipeline in conventional pruning methods , but explored the prospect of directly training smaller , sparse subnetworks in place of the full models without sacrificing performance . One key idea is to reuse the sparsity pattern found through pruning and training a sparse network from scratch . The seminal work of the Lottery ticket hypothesis ( LTH ) ( Frankle & Carbin , 2019 ) demonstrated that standard dense NNs contain sparse matching subnetworks ( called “ winning tickets ” ) capable of training in isolation to full accuracy . In other words , we could have trained smaller networks from the start if only we had known which subnetworks to choose . Additionally , other works also showed that sparsity might even emerge at initialization , before training ever starts and with minimal-to-no dependence on data , such as SNIP ( Lee et al. , 2019 ) , GraSP ( Wang et al. , 2020 ) , and SynFlow ( Tanaka et al. , 2020 ) . Those methods suggest a tantalizing possibility of “ pruning-at-initialization ” for finding sparse subnetworks without overhead , even though it remains under debate whether their found subnetworks have inferior quality to lottery tickets found with IMP ( Frankle et al. , 2020 ) . The consensus seems to be that , a “ good ” sparse topology can be very crucial and informative for successful training . Another set of efforts ( Zhou et al. , 2019 ; Wortsman et al. , 2020 ) suggest that “ masking ( pruning ) is training ” , e.g. , by strategically zeroing out randomly initialized weights , the network can reach far-better-than-chance accuracy . ( Ramanujan et al. , 2020 ) found that from a “ sufficiently overparameterized ” dense NN with random initialization , there exists a hidden subnetwork that achieves competitive accuracy ( w.r.t . to the fully trained dense NN ) , without any training on the remaining weights . Its theoretical proof was presented by ( Malach et al. , 2020 ) for the first time and followed by ( Pensia et al. , 2020 ; Orseau et al. , 2020 ) . ( Ivan & Florian , 2020 ) tried an alternative training scheme , by only flipping the weight signs in a dense NN while preserving their initialized magnitudes . All aforementioned works suggest that it is possible to only update the connectivity patterns without changing the magnitudes of the weights to achieve decent performance . However , many of them only tried on rather small NNs ( Zhou et al. , 2019 ; Ivan & Florian , 2020 ) , with notable accuracy drops compared to full gradient-based training . This is intuitively understandable because NNs ’ expressiveness will be strictly constrained if pruning is the only allowable operations during training . Under such rigid restrictions , to obtain hidden subnetworks with acceptable performance requires high-quality connectivity patterns with appropriate pruning ratios . Ramanujan et al . ( 2020 ) found that the hidden subnetworks with either too high or too low pruning ratios have poor performance . What is even worse is that the tedious optimization needed for identifying the hidden subnetworks . All of those works used some form of latent weights or score functions for all NN weights , which are cast into binary decisions to mask their corresponding weights or not during forward pass and are trained during backpropagation , e.g. , the edge-popupalgorithm in ( Ramanujan et al. , 2020 ) . Those latent weights or scores make their training even more costly than training standard NNs , and the non-differentiable binarization operation may also inject instability . 1.1 OUR CONTRIBUTIONS AND RATIONALES . Inspired by the recent advances in sparse NNs and the philosophy of “ masking is training ” , this paper first proposes the concept of disguised subnetworks in randomly initialized NNs , as the extension to the “ hidden subnetworks ” found in ( Ramanujan et al. , 2020 ) . At its name suggests , a disguised subnetwork is hidden as a sparse subnetwork in a dense NN at its random initialization . This subnetwork , without achieving good performance at the beginning , can have its “ disguise ” unmasked by learning some simple transformations on the remaining weights to lower the training loss . To preserve the randomness in the subnetwork at the best , we only consider transformations that do not alter the magnitudes of the remaining weights , such as shuffling and sign flipping . In this work , we will focus on sign flipping specifically . We argue that the extra unmasking step in disguised subnetworks grants essential advantages over plain hidden subnetworks by significantly enlarging the expressiveness of the subnetworks . Such advantages immediately bring several benefits . Firstly , with much better capacity and expressiveness , disguised subnetworks can easily achieve superior performances compared with hidden subnetworks and thus have better scalability to larger NNs and datasets . Secondly , the searching of the sparse subnetwork and the following unmasking process can compliment each other for better performance , which relaxes the quality requirement on the sparse mask . This benefit enables us to replace the unstable and expensive searching algorithms in previous works with more efficient pruning algorithms . The rationale behind is that only learning transformations on randomly initialized weights can not create sparsity that a trained NN often shows to possess ( Zhou et al. , 2019 ) while only pruning has too limited flexibility which can be drastically boosted by allowing for further weight processing . Based on this reasoning , we propose a two-phase algorithm called Peek-a-Boo ( PaB ) , which consists of the searching and unmasking phases involving two operations : ( 1 ) sparsifying a deep neural network with efficient pruning-at-initialization methods at its random initialization ; ( 2 ) learning to flip the signs of the resulting subnetwork ’ s remaining weights to lower the training loss . Note that we never modify weight magnitudes after being initialized aside from removing weights . PaB absorbs and improves the idea “ masking is training ” ( Zhou et al. , 2019 ) , by unifying the two streams of ideas : pruning ( i.e. , applying { 0,1 } masks to weights ) and flipping signs ( i.e. , applying { -1,1 } masks ) . Moreover , for its second stage of unmasking , we optimize weight signs without extra latent weights or scores ( Helwegen et al. , 2019 ) . Such optimization method frees us from using the heavy latent parameters nor the non-stable straight-through-estimators ( STEs ) , which were assumed by all previous methods . We further demonstrate that an even more efficient implementation could be achieved by using low-cost gradient predictors for approximated gradients ( Wang et al. , 2019 ) , as PaB might be tolerant to “ coarser ” gradient information since we only flip signs . Despite remaining as an empirical exploration , the success of PaB also urges theoretical reflections on whether sparse NNs have to follow the same optimization strategies as the dense ones , or whether they can utilize new optimizers as simple as sign flipping . On one hand , pruning can often cause more difficult optimization landscapes for gradient descents , as training a sparse NN essentially becomes a constrained and non-smooth optimization problem ( Evci et al. , 2019 ) . On the other hand , assuming a good sparse mask is known a priori , the possible solution space has also been significantly reduced , which may suggest the existence of “ shortcuts ” to reach the optima in this shrunken space , such as the “ solution locality ” of LTH recently observed by ( Liu et al. , 2021 ) . 2 RELATED WORKS . 2.1 SPARSE NEURAL NETWORKS . Pruning methods ( Hassibi et al. , 1993 ; LeCun et al. , 1989 ) have witnessed great advances to reduce the increasing inference complexities of NNs . Magnitude-based pruning methods ( CarreiraPerpiñán & Idelbayev , 2018 ; Guo et al. , 2016 ; Han et al. , 2015 ) have become mainstream due to their relative low overheads and good scalability to large NNs . However , they still hinge on the expensive train-prune-retrain cycles to restore performance , and do not lead to efficient training . LTH ( Frankle & Carbin , 2019 ) discovers sparse subnetworks at random initialization that can achieve similar or even better performance when trained in isolation . The winning tickets are found by the computationally intensive iterative magnitude-based pruning ( IMP ) , which involves tens of rounds of train-prune-retrain cycles . Rewinding was later found to be essential for stably identifying winning tickets in large networks ( Frankle et al. , 2019 ; 2020 ) . ( Morcos et al. , 2019 ; Chen et al. , 2020 ) studied the transferability of winning tickets between datasets and downstream tasks . Zhou et al . ( 2019 ) made further interesting observations on winning tickets , discovering that preserving the original weight signs is essential to successfully training the tickets , and that the pruned weights have to be set to zero instead of being frozen at initial values . They concluded that “ masking is training ” , i.e. , the masking operation tends to move weights in the direction they would have moved during training . Hence , simply applying an appropriate mask to the random initialization leads to reasonably good performance . The authors found such “ supermasks ” with both heuristics and Bernoulli sampler-based optimization . Their finding was empirically concurred by ( Ramanujan et al. , 2020 ) and theoretically supported by ( Malach et al. , 2020 ) . However , the original experiments in ( Zhou et al. , 2019 ) were restricted to NNs with a very small number of layers ( e.g. , Conv 4 ) . Later work ( Ramanujan et al. , 2020 ) identified untrained subnetworks from Wide ResNet-50 as “ mother NNs ” on ImageNet that can match the performance of ResNet-34 ( worse than ResNet-50 ) . Yet that is very different from our ( more ambitious ) goal - pruning a subnetwork from the “ mother NN ” , and having it match the original performance of the “ mother NN ” by only flipping-based training . Moreover , their sophisticated algorithm relies on a set of real-valued latent weights ( called “ scores ” associated with each NN weight ) , making their training essentially as expensive as full SGD . Another line of works has arisen to investigate pruning NNs directly at random initialization . Examples include SNIP ( Lee et al. , 2019 ) , GraSP ( Wang et al. , 2020 ) and SynFlow ( Tanaka et al. , 2020 ) , all of which exploit the weight magnitudes as well as first- or higher-order information of NNs at the initial point . ( Frankle et al. , 2021 ) revisited these methods and observed that the naive magnitude-based pruning at initialization already plays as a strong competitor , with a side observa- tion that delaying the pruning generally improves performance . All these algorithms train the pruned networks using the same standard SGD-type optimizers .

Summary: Optimizing sparse neural networks is an important topic due to their computational and space savings.  Building on the work of the lottery ticket hypothesis, others have shown there exist hidden subnetworks within randomly initialized NN that have good performance. The authors extend this definition to disguised subnetworks, which contain hidden subnetworks as a subclass. Moreover, the authors present a novel combination of existing methods  into a single algorithm they call Peek-a-Boo (PaB) which can efficiently find such networks.