Peek-a-Boo: What (More) is Disguised in a Randomly Weighted Neural Network, and How to Find It Efficiently

1 INTRODUCTION . Recent years have seen substantial efforts devoted to scaling neural networks ( NNs ) to enormous sizes ( He et al. , 2016 ; Devlin et al. , 2019 ) . Parameter-counts are frequently measured in billions rather than millions , with the time and financial outlay necessary to train these models growing in concert . Sparse NNs , whose large portions of parameters are zero , have been studied to address those gaps , saving storage , memory and computational costs . Conventional approaches first train dense NNs , and then prune the trained NNs to high levels of sparsity ( Han et al. , 2015 ; Guo et al. , 2016 ) . Those methods significantly reduce the inference complexity yet cost even greater computational resources and memory footprints at training . An emerging field has explored what roles sparsity can play in neural networks , spawning different directions of research . One of them follows the typical pruning-and-finetuning pipeline in conventional pruning methods , but explored the prospect of directly training smaller , sparse subnetworks in place of the full models without sacrificing performance . One key idea is to reuse the sparsity pattern found through pruning and training a sparse network from scratch . The seminal work of the Lottery ticket hypothesis ( LTH ) ( Frankle & Carbin , 2019 ) demonstrated that standard dense NNs contain sparse matching subnetworks ( called “ winning tickets ” ) capable of training in isolation to full accuracy . In other words , we could have trained smaller networks from the start if only we had known which subnetworks to choose . Additionally , other works also showed that sparsity might even emerge at initialization , before training ever starts and with minimal-to-no dependence on data , such as SNIP ( Lee et al. , 2019 ) , GraSP ( Wang et al. , 2020 ) , and SynFlow ( Tanaka et al. , 2020 ) . Those methods suggest a tantalizing possibility of “ pruning-at-initialization ” for finding sparse subnetworks without overhead , even though it remains under debate whether their found subnetworks have inferior quality to lottery tickets found with IMP ( Frankle et al. , 2020 ) . The consensus seems to be that , a “ good ” sparse topology can be very crucial and informative for successful training . Another set of efforts ( Zhou et al. , 2019 ; Wortsman et al. , 2020 ) suggest that “ masking ( pruning ) is training ” , e.g. , by strategically zeroing out randomly initialized weights , the network can reach far-better-than-chance accuracy . ( Ramanujan et al. , 2020 ) found that from a “ sufficiently overparameterized ” dense NN with random initialization , there exists a hidden subnetwork that achieves competitive accuracy ( w.r.t . to the fully trained dense NN ) , without any training on the remaining weights . Its theoretical proof was presented by ( Malach et al. , 2020 ) for the first time and followed by ( Pensia et al. , 2020 ; Orseau et al. , 2020 ) . ( Ivan & Florian , 2020 ) tried an alternative training scheme , by only flipping the weight signs in a dense NN while preserving their initialized magnitudes . All aforementioned works suggest that it is possible to only update the connectivity patterns without changing the magnitudes of the weights to achieve decent performance . However , many of them only tried on rather small NNs ( Zhou et al. , 2019 ; Ivan & Florian , 2020 ) , with notable accuracy drops compared to full gradient-based training . This is intuitively understandable because NNs ’ expressiveness will be strictly constrained if pruning is the only allowable operations during training . Under such rigid restrictions , to obtain hidden subnetworks with acceptable performance requires high-quality connectivity patterns with appropriate pruning ratios . Ramanujan et al . ( 2020 ) found that the hidden subnetworks with either too high or too low pruning ratios have poor performance . What is even worse is that the tedious optimization needed for identifying the hidden subnetworks . All of those works used some form of latent weights or score functions for all NN weights , which are cast into binary decisions to mask their corresponding weights or not during forward pass and are trained during backpropagation , e.g. , the edge-popupalgorithm in ( Ramanujan et al. , 2020 ) . Those latent weights or scores make their training even more costly than training standard NNs , and the non-differentiable binarization operation may also inject instability . 1.1 OUR CONTRIBUTIONS AND RATIONALES . Inspired by the recent advances in sparse NNs and the philosophy of “ masking is training ” , this paper first proposes the concept of disguised subnetworks in randomly initialized NNs , as the extension to the “ hidden subnetworks ” found in ( Ramanujan et al. , 2020 ) . At its name suggests , a disguised subnetwork is hidden as a sparse subnetwork in a dense NN at its random initialization . This subnetwork , without achieving good performance at the beginning , can have its “ disguise ” unmasked by learning some simple transformations on the remaining weights to lower the training loss . To preserve the randomness in the subnetwork at the best , we only consider transformations that do not alter the magnitudes of the remaining weights , such as shuffling and sign flipping . In this work , we will focus on sign flipping specifically . We argue that the extra unmasking step in disguised subnetworks grants essential advantages over plain hidden subnetworks by significantly enlarging the expressiveness of the subnetworks . Such advantages immediately bring several benefits . Firstly , with much better capacity and expressiveness , disguised subnetworks can easily achieve superior performances compared with hidden subnetworks and thus have better scalability to larger NNs and datasets . Secondly , the searching of the sparse subnetwork and the following unmasking process can compliment each other for better performance , which relaxes the quality requirement on the sparse mask . This benefit enables us to replace the unstable and expensive searching algorithms in previous works with more efficient pruning algorithms . The rationale behind is that only learning transformations on randomly initialized weights can not create sparsity that a trained NN often shows to possess ( Zhou et al. , 2019 ) while only pruning has too limited flexibility which can be drastically boosted by allowing for further weight processing . Based on this reasoning , we propose a two-phase algorithm called Peek-a-Boo ( PaB ) , which consists of the searching and unmasking phases involving two operations : ( 1 ) sparsifying a deep neural network with efficient pruning-at-initialization methods at its random initialization ; ( 2 ) learning to flip the signs of the resulting subnetwork ’ s remaining weights to lower the training loss . Note that we never modify weight magnitudes after being initialized aside from removing weights . PaB absorbs and improves the idea “ masking is training ” ( Zhou et al. , 2019 ) , by unifying the two streams of ideas : pruning ( i.e. , applying { 0,1 } masks to weights ) and flipping signs ( i.e. , applying { -1,1 } masks ) . Moreover , for its second stage of unmasking , we optimize weight signs without extra latent weights or scores ( Helwegen et al. , 2019 ) . Such optimization method frees us from using the heavy latent parameters nor the non-stable straight-through-estimators ( STEs ) , which were assumed by all previous methods . We further demonstrate that an even more efficient implementation could be achieved by using low-cost gradient predictors for approximated gradients ( Wang et al. , 2019 ) , as PaB might be tolerant to “ coarser ” gradient information since we only flip signs . Despite remaining as an empirical exploration , the success of PaB also urges theoretical reflections on whether sparse NNs have to follow the same optimization strategies as the dense ones , or whether they can utilize new optimizers as simple as sign flipping . On one hand , pruning can often cause more difficult optimization landscapes for gradient descents , as training a sparse NN essentially becomes a constrained and non-smooth optimization problem ( Evci et al. , 2019 ) . On the other hand , assuming a good sparse mask is known a priori , the possible solution space has also been significantly reduced , which may suggest the existence of “ shortcuts ” to reach the optima in this shrunken space , such as the “ solution locality ” of LTH recently observed by ( Liu et al. , 2021 ) . 2 RELATED WORKS . 2.1 SPARSE NEURAL NETWORKS . Pruning methods ( Hassibi et al. , 1993 ; LeCun et al. , 1989 ) have witnessed great advances to reduce the increasing inference complexities of NNs . Magnitude-based pruning methods ( CarreiraPerpiñán & Idelbayev , 2018 ; Guo et al. , 2016 ; Han et al. , 2015 ) have become mainstream due to their relative low overheads and good scalability to large NNs . However , they still hinge on the expensive train-prune-retrain cycles to restore performance , and do not lead to efficient training . LTH ( Frankle & Carbin , 2019 ) discovers sparse subnetworks at random initialization that can achieve similar or even better performance when trained in isolation . The winning tickets are found by the computationally intensive iterative magnitude-based pruning ( IMP ) , which involves tens of rounds of train-prune-retrain cycles . Rewinding was later found to be essential for stably identifying winning tickets in large networks ( Frankle et al. , 2019 ; 2020 ) . ( Morcos et al. , 2019 ; Chen et al. , 2020 ) studied the transferability of winning tickets between datasets and downstream tasks . Zhou et al . ( 2019 ) made further interesting observations on winning tickets , discovering that preserving the original weight signs is essential to successfully training the tickets , and that the pruned weights have to be set to zero instead of being frozen at initial values . They concluded that “ masking is training ” , i.e. , the masking operation tends to move weights in the direction they would have moved during training . Hence , simply applying an appropriate mask to the random initialization leads to reasonably good performance . The authors found such “ supermasks ” with both heuristics and Bernoulli sampler-based optimization . Their finding was empirically concurred by ( Ramanujan et al. , 2020 ) and theoretically supported by ( Malach et al. , 2020 ) . However , the original experiments in ( Zhou et al. , 2019 ) were restricted to NNs with a very small number of layers ( e.g. , Conv 4 ) . Later work ( Ramanujan et al. , 2020 ) identified untrained subnetworks from Wide ResNet-50 as “ mother NNs ” on ImageNet that can match the performance of ResNet-34 ( worse than ResNet-50 ) . Yet that is very different from our ( more ambitious ) goal - pruning a subnetwork from the “ mother NN ” , and having it match the original performance of the “ mother NN ” by only flipping-based training . Moreover , their sophisticated algorithm relies on a set of real-valued latent weights ( called “ scores ” associated with each NN weight ) , making their training essentially as expensive as full SGD . Another line of works has arisen to investigate pruning NNs directly at random initialization . Examples include SNIP ( Lee et al. , 2019 ) , GraSP ( Wang et al. , 2020 ) and SynFlow ( Tanaka et al. , 2020 ) , all of which exploit the weight magnitudes as well as first- or higher-order information of NNs at the initial point . ( Frankle et al. , 2021 ) revisited these methods and observed that the naive magnitude-based pruning at initialization already plays as a strong competitor , with a side observa- tion that delaying the pruning generally improves performance . All these algorithms train the pruned networks using the same standard SGD-type optimizers .

Summary: This paper proposes the idea of a "disguised subnetwork", which are hidden random subnetworks that can be transformed into a well-performing subnetwork. The paper introduces PaB as a way to uncover these subnetworks, by first searching for a mask over the random weights using pruning-at-initialization techniques, and then learning a transformation on the subnetwork. The paper further shows that this PaB process can be efficiently implemented, offering significant advantage over prior work.