Peek-a-Boo: What (More) is Disguised in a Randomly Weighted Neural Network, and How to Find It Efficiently
Authors: Xiaohan Chen
Jason Zhang
Zhangyang Wang
Venue: ICLR 2022 Poster (2022)
Keywords: Sparse Neural Network
Lottery Ticket Hypothesis
Efficient Machine Leanring

Abstract: Sparse neural networks (NNs) are intensively investigated in literature due to their appeal in saving storage, memory, and computational costs. A recent work (Ramanujan et al., 2020) showed that, different from conventional pruning-and-finetuning pipeline, there exist hidden subnetworks in randomly initialized NNs that have good performance without training the weights. However, such "hidden subnetworks" have mediocre performances and require an expensive edge-popup algorithm to search for them. In this work, we define an extended class of subnetworks in randomly initialized NNs called disguised subnetworks, which are not only "hidden" in the random networks but also "disguised" -- hence can only be "unmasked" with certain transformations on weights. We argue that the unmasking process plays an important role in enlarging the capacity of the subnetworks and thus grants two major benefits: (i) the disguised subnetworks easily outperform the hidden counterparts; (ii) the unmasking process helps to relax the quality requirement on the sparse subnetwork mask so that the expensive edge-popup algorithm can be replaced with more efficient alternatives. On top of this new concept, we propose a novel two-stage algorithm that plays a Peek-a-Boo (PaB) game to identify the disguised subnetworks with a combination of two operations: (1) searching efficiently for a subnetwork at random initialization; (2) unmasking the disguise by learning to transform the resulting subnetwork's remaining weights. Furthermore, we show that the unmasking process can be efficiently implemented (a) without referring to any latent weights or scores; and (b) by only leveraging approximated gradients, so that the whole training algorithm is computationally light. Extensive experiments with several large models (ResNet-18, ResNet-50, and WideResNet-28) and datasets (CIFAR-10, CIFAR-100 and ImageNet) demonstrate the competency of PaB over edge-popup and other counterparts. Our codes are available at: https://github.com/VITA-Group/Peek-a-Boo.

Published as a conference paper at ICLR 2022
PEEK-A-BOO: WHAT (MORE) IS DISGUISED IN A RANDOMLY WEIGHTED NEURAL NETWORK, AND HOW TO FIND IT EFFICIENTLY
Xiaohan Chen', Jason Zhang 1.2, Zhangyang Wangb
Unniersityy of Texas at Austin, CCrnnegie Mellon University (xiachan. chen, jzhang27144 /143, at (laswang/uttxxa .du
ABSTRACT
Sparse neural networks (NNs) are intensively investigated in literature due to their appeal in saving storage, memory, and computational costs. A recent work (Ramanujan et al., 2020) showed that, different from conventional pruning- and-finetuning pipeline, there exist hidden subnetworks in randomly initialized NNs that have good performance without training the weights. However, such "hidden subnetworks" have mediocre performances and require an expensive edge-popup algorithm to search for them. In this work, we define an ex- tended class subnetworks randomly initialized NNs called disguised subner- works, which are not only "hidden" in the random networks but also "disguised" - hence can only be "unmasked" with certain transformations on weights. We argue that the unmasking process plays important role in enlarging the pacity of the subnetworks and thus grants two major benefits: (i) the disguised subnetworks easily outperform the hidden counterparts; (ii) the unmasking pro- cess helps to relax the quality requirement on the sparse subnetwork mask so that the expensive edge edge-popul -oopup algorithm can be replaced with more efficient alternatives, On top of this new concept, we propose a novel two-stage algo- (PaB) game identify the disguised subnetworks
rithm that plays a Peek-a- with a combination of two operations: (1) searching efficiently for a subnet- work at random initialization; (2) unmasking the disguise by learning to trans- form the resulting subnetwork's 's remaining weights. Furthermore, we show that the unmasking process can be efficiently implemented (a) without referring any latent weights or scores; and (b) by only leveraging approximated gradients, so that the whole training algorithm is computationally light. Extensive exper- iments with several large models (ResNet-18, ResNet-50, and WideResNet-28)) and datasets (CIFAR-10, 10, CIFAR-10 and ImageNet) demonstrate the competency of PaB over edge rddge-popup popup and other counterparts. Our codes are available at: https: /tttt/bb.tttttt/////gg com D.om/VIn /t////VIAAAG-rrp Group/Peek a BBo..
INTRODUCTION
Recent years have seen substantial efforts devoted scaling neural networks (NNs) enormous sizes (He et al., 2016; Devlin et al., 2019). Parameter-counts are frequently measured in billions rather than millions, with the time and financial outlay necessary to train these models growing in concert. Sparse NNs, whose large portions of parameters are zero, have been studied to address those gaps, saving storage, memory and computational costs. Conventional approaches first train dense NNs, and then prune the trained NNs to high levels of sparsity (Han et al., 2015; Guo et al., 2016). Those methods significantly reduce the inference complexity yet cost even greater computational resources and memory footprints at training.
An emerging field has explored what roles sparsity can play in neural networks, spawning different directions of research. One of them follows the typical pruning-on and-finetuning pipeline in conven- tional pruning methods, but explored the prospect of directly training smaller, sparse subnetworks
Work done while the author was at the University of Texas Austin.

---

Published as a conference paper at ICLR 2022
in place of the full models without sacrificing performance. One key idea is to reuse the sparsity pattern found through pruning and training a sparse network from scratch. The seminal work of the lottery ticket hypothesis (LTH) (Frankle & Carbin, 2019) demonstrated that standard dense NNs contain sparse matching subnetworks (called "winning tickets") capable of training in isolation to full accuracy. In other words, we could have trained smaller networks from the start if only we had known which subnetworks to choose. Additionally, other works also showed that sparsity might even emerge initialization, before training ever starts and with minimal-to-no dependence on data, such as SNIP (Lee et al., 2019). GraSP (Wang et al., 2020), and SynFlow (Tanaka et al., 2020). Those methods suggest a tantalizing possibility of "pruning-at at attinitialization" for finding sparse sub- networks without overhead, even though it remains under debate whether their found subnetworks have inferior quality to lottery tickets found with IMP (Frankle et al., 2020). The consensus seems to be that, a "good" sparse topology can be very crucial and informative for successful training.
Another set of efforts (Zhou et al., 2019; Wortsman et al., 2020) suggest that "masking (pruning) is training". c.g., by strategically zeroing out randomly initialized weights, the network can reach far-beteer-taan-chance accuracy. (Ramanujan et al., 2020) found that from a "sufficiently over- parameterized" dense NN with random initialization, there exists hidden subnetwork that achieves competitive accuracy (w.rt. to the fully trained dense NN), without any training on the remaining weights This finding was later theoretically justified by (Malach et al., 2020; Pensia et al., 2020; Orseau et al., 2020). Ivan & Florian (2020) tried an alternative training scheme, by only flipping the weight signs in dense NN while preserving their initialized magnitudes. Sreenivasan et al. (2021) further showed that such hidden subnetworks also exist in networks with binary weights. All aforementioned works suggest that it is possible only update the connectivity patterns without changing the magnitudes of the weights to achieve decent performance.
However, many of them only tried on rather small NNs (Zhou et al., 2019; Ivan & Florian, 2020), with notable accuracy drops compared to full gradient-based training. This is intuitively understand- able because NNs' expressiveness will be strictly constrained pruning is the only allowable oper- ations during training. Under such rigid restrictions, to obtain hidden subnetworks with acceptable performance requires high-quality connectivity patterns with appropriate pruning ratios. Ramanujan et al. (2020) found that the hidden subnetworks with either too high or too low pruning ratios have poor performance. What is even worse is that the tedious optimization needed for identifying the hidden subnetworks. All of those works used some form of latent weights score functions for all NN weights, which are cast into binary decisions to mask their corresponding weights or not during forward pass and are trained during backpropagation, e.g., the Sedge-popup algorithm in (Ramanujan et al., 2020). Those latent weights or scores make their training even more costly than training standard NNs, and the -on-differentiable binarization operation may also inject instability.
1.1
OUR CONTRIBUTIONS AND RATIONALES
Inspired by the recent advances in sparse NNs and the philosophy of "masking is training", this paper first proposes the concept of disguised subnetworks in randomly initialized NNs, as the extension to the "hidden subnetworks" found in (Ramanujan al., 2020). At its name suggests, disguised subnetwork is hidden as a sparse subnetwork in a dense NN at its random initialization. This subnetwork, without achieving good performance the beginning, can have its "disguise" unmasked by learning some simple transformations on the remaining weights to lower the training loss. To preserve the randomness in the subnetwork at the best, we only consider transformations that do not alter the magnitudes of the remaining weights, such as shuffling and sign flipping. In this work, we will focus on sign flipping specifically.
We argue that the extra unmasking step in disguised subnetworks grants essential advantages over plain hidden subnetworks by significantly enlarging the expressiveness of the subnetworks. Such ad- vantages immediately bring several benefits. Firstly, with much better capacity and expressiveness, disguised subnetworks can easily achieve superior performances compared with hidden subnetworks and thus have better scalability to larger NNs and datasets. Secondly, the searching of the sparse subnetwork and the following unmasking process can compliment each other for better performance, which relaxes the quality requirement the sparse mask. This benefit enables us to replace the un- stable and expensive searching algorithms in previous works with more efficient pruning algorithms. randomly initialized weights cannot
The rationale behind is that only learning transformations create sparsity that a trained NN often shows to possess (Zhou et al., 2019) while only pruning has too limited flexibility which can be drastically boosted by allowing for further weight processing.
2

---

Published as a conference paper at ICLR 2022
Based on this reasoning, we propose a two-phase algorithm called Peek Booo (PaB), which consists of the searching and unmasking phases involving two operations: (1) sparsifying a deep neural network with efficient pruming-at-initin -t-iiaaiatittiiiiittionn methods at its random initialization; (2) learning to flip the signs of the resulting subnetwork's remaining weights to lower the training loss. Note that we never modify weight magnitudes after being initialized aside from removing weights. PaB absorbs and improves the idea "masking is training" (Zhou et al., 2019), by unifying the two streams of ideas: pruning (i.e., applying {0.1] masks to weights) and flipping signs (i.e., applying {-1.1} masks). Moreover, for its second stage of unmasking. we optimize weight signs without extra latent weights or scores (Helwegen et al., 2019). Such optimization method frees us from using the heavy latent parameters nor the unstable triight-trrough-sii estimatoos (STEs), which were assumed by all previous methods. We further demonstrate that an even more efficient implementation could be achieved by using low-cost gradient predictors for approximated gradients (Wang et al., 2019), as PaB might be tolerant to "coarser" gradient information since we only flip signs.
Despite remaining an empirical exploration, the success of PaB also urges theoretical reflections on whether sparse NNs have to follow the same optimization strategies as the dense ones, or whether they can utilize new optimizers as simple as sign flipping. On one hand, pruning can often cause more difficult optimization landscapes for gradient descents, as training a sparse NN essentially becomes constrained and non-smooth optimization problem (Evci et al., 2019). On the other hand, assuming a good sparse mask is known a priori, the possible solution space has also been significantly reduced, which may suggest the existence of "shortcuts" reach the optima in this shrunken space, such as the "solution locality" of LTH recently observed by (Liu et al., 2021).
2
RELATED WORKS
2.1 SPARSE NEURAL NETWORKS
Pruning methods (Hassibi et al., 1993; LeCun et al., 1989) have witnessed great advances to re- duce the increasing inference complexities of NNs. Magnitude-based pruning methods (Carreira- Perpiñáán & Idelbayev. 2018; Guo et al., 2016; Han et al., 2015) have become mainstream due to their relative low overheads and good scalability to large NNs. However, they still hinge on the ex- cycles to restore performance, and do not lead to efficient training. LTH (Frankle & Carbin, 2019) discovers sparse subnetworks random initialization that can achieve similar or even better performance when trained in isolation. The winning tickets are found by the computationally intensive iterative magnitude-based praning (IMP), which involves tens of rounds of train- prune-retraie cycles. Rewinding ning tickets in large networks (Frankle et al., 2019; 2020). (Morcos et al., 2019; Chen et al., 2020; 2021a;b) studied the transferability of winning tickets between datasets, tasks and architectures.
pensive
later found to be essential for stably identifying win-
Zhou et al. (2019) made further interesting observations on winning tickets, discovering that pre- serving the original weight signs is essential to successfully training the tickets, and that the pruned weights have to be set to zero instead of being frozen at initial values. They concluded that "mask- ing is training", i.e., the masking operation tends to move weights in the direction they would have moved during training. Hence, simply applying an appropriate mask to the random initialization leads to reasonably good performance. The authors found such "supermasks" with both heuristics and Bernoulli cumpler-based optimization. Their finding was empirically concurred by (Ramanujan et al., 2020) and theoretically supported by (Malach et al., 2020; Pensia et al., 2020; Orseau et al., 2020). Similar results are also observed and proved for binary neural networks.
However, the original experiments in (Zhou et al., 2019) were restricted NNs with a very small number of layers (e.g., Conv 4). Later work (Ramanujan et al., 2020) identified untrained sub- networks from Wide ResNet-50 as "mother NNs" on ImageNet that can match the performance of ResNet-34 (worse than ResNet-50). Yet that is very different from our (more ambitious) goal pruning a subnetwork from the "mother NN", and having it match the original performance of the "mother NN" by only flipping-based training. Moreover, their sophisticated algorithm relies set of real-valued latent weights (called "scores" associated with each NN weight), making their training essentially as expensive as full SGD.
Another line of works has arisen to investigate pruning NNs directly at random initialization. Ex- amples include SNIP (Lee et al., 2019), GraSP (Wang et al., 2020) and SynFlow (Tanaka et al.,
3

---

Published as a conference paper at ICLR 2022
2020), all of which exploit the weight magnitudes well as first- or higher-order information of NNs at the initial point. (Frankle et al., 2021) revisited these methods and observed that the naive magnitude-basee pruming initialization already plays strong competitor, with side observa- tion that delaying the pruning generally improves performance. All these algorithms train the pruned networks using the same standard SGD-type optimizers.
2.2 OPTIMIZATION SIGN FLIPPING
The mrin-by-flipphn scheme considered in this work closely related to the optimization of binary neural networks (BNNs), whose weights are only +1/-1. Optimization methods over BNNs can be categorized into two types - training with and without real-valued latent weights. The former category (Bengio al., 2013; Courbariaux et al., 2016) maintains and updates set of full-precision weights for back-pue poock--opggttion while forward passes only utilize binary weights. The downside of this method is the extra memory and resource overhead in training. The second category eliminates latent weights and only learns the decision rules for flipping or not, based on certain training statistics. For example, Bop (Helwegen et al., 2019) keeps running average of historical gradients, and flips weight's sign when its accumulated gradients surpasses a pre-defined threshold.
A more recent work (Ivan & Florian, 2020) studied the possibility of training NNs by only flip- ping the signs of the initialized weights, leaving the magnitudes unchanged. The authors proposed to update an extra set of latent parameters, whose signs determines the weight flipping decision throughout training. Their method struggled to scale up to large NNs, causing large performance drops. The potential reason behind this might be just as (Zhou et al., 2019) advocated, that zero would particularly good value to set many weights to. In other words, weight sparsity may be generally helpful inductive prior for well-trained NNs, which cannot be met by only flipping signs of weights. Besides, the training algorithm in (Ivan & Florian, 2020) also induces large overhead due to using full- full-precision latent weights and does not head towards efficient training. Moreover, neither (Ivan& & Florian, 2020) nor any BNN optimization methods have considered the intersection of training-bu seaninggby--fiipeee by-flipiing and sparse NNs.
3. PEEK-A-BOO- SEARCH AND UNMASK DISGUISED SUBNETWORKS IN RANDOMLY INITIALIZED NEURAL NETWORKS
3.1 DISGUISED SUBNETWORKS VERSUS HIDDEN SUBNETWORKS
In this subsection, we formally define hidden subnetworks and disguised subnetworks in randomly initialized NNs, and show that the latter, as an extended class of subnetworks to the former, contain better subnetworks that can outperform those in the former.
Consider a neural network fx; parameterized by (vectorized) weights w E R for input x. We denote the randomly initialized weights as (0)), and (())) as the weights after steps of training. We denote general transformation applied on the weights w as (ww) that takes any form.
Definition 1. A hidden subnetwork in the dense neural network far; w) is characterized by binary mask m E {0, 13"h and denoted as f(r; w O m), where O represents the Hadamard product. A transformed hidden subnetwork characterized by binary mask m E {0, 1]"h and a weight transformation U-), denoted as f(r U(w @ m))
disguised subnetwork is defined
Remark Here we omit the index for the number of training steps because we can take hidden subnetworks or disguised subnetworks at any time during training, while in this work our focus is on the subnetworks at random initializations, i.e., f(; (0) @ m) and fæ; U 3 m)).
To identify the optimal disguised subnetworks at a certain sparsity level S, we need to solve the optimization problem below:
minimize Lnl (w)()) @m)),, UEN, ||m]|
(1)
where U is the collection of all possible weight transformations and LD is the loss function dataset D. It is clear that the optimal disguised network achieves a minimum that is equal to or smaller than that achieved by the optimal hidden network, because the hidden networks are the
4

---

Published as a conference paper at ICLR 2022
special cases of disguised subnetworks if we constrain the weight transformation U to the identity mapping I. And the optimal hidden subnetwork is the minimizer of the following problem:
minimize Colf(x; (( @ m))]. ||m|o=
In this paper, we argue that constraining the weight transformation U to identity as in the hidden networks can significantly decrease the model capacity and thus degrade the performance, especially when n is large. i.e,, the model is large or s is small, i.c., the model is highly sparse. In contrast, if we can directly solve (1), we can find disguised subnetworks with much better performance.
However, directly optimizing (1) is computationally intractable due to the high-dimensionalityy of the networks and the complexity of the collection of weight transformations. Therefore, we propose to approximately solve (1) by decoupling it into a two-phase process. In the first phase, we find a suitable mask vi to sparsify a deep neural network at its random initialization. We call the searching phase. In the second phase, we learn the optimal weight transformation to further lower the training loss. We call it the unmasking phase. The whole process is formally formulated as:
i E argmin R R(f(x; @ m))
(3)
[mmllo=S
E argmin (xlf(x; U( o m))]
(4)
Veu
Here R is a score function that measures the quality of the binary mask m for the network f with random initialization ()(0) in some metrie. Different selections of this score function will lead to dif- ferent pruning methods. To efficiently solve this two-phase optimization in (3) and (4), we propose Peek-a-Bor Boo (PaB) framework as detailed below.
3.2 PEEK-A-BOO: EFFICIENT SEARCHING AND UNMASKING
In the searching phase of the PaB framework, we sparsify a neural network at its random initial- ization using a training-free pruning method. In the unmasking phase, we learn to flip signs of the remaining nonzero weights as the weight transformation U to lower the training loss.
Searching phase: training-free pruning. Our goal in the first step use pruning find a sparse mask mt, with two requirements. Firstly, the sparse mask provides a useful prior of "good structures" in the network for the sopping-based training that follows. Secondly, the pruning itself needs to be light enough to not incur overhead for training efficiency. Due to the second concern, we give up the expensive IMP (Frankle & Carbin, 2019; Zhou et al., 2019) and focus on the pruning-at-ing attinitiiliiztioon methods (Lee et al., 2019; Wang et al., 2020; Tanaka et al., 2020; Frankle et al., 2021).
We choose SynFlow (Tanaka et al., 2020) over SNIP (Lee et al., 2019) and GraSP (Wang et al., 2020), because the latter two depend on the first-order and/or second-order information of the neural network to calculate the weight scores. To accurately estimate those, one needs to run through training samples. In comparison, SynFlow is data-free and only uses an all-one input.
Unmasking phase: flipping the non-zero weights. In this step, we learn the optimal weight sign flipping Uz(w) want to optimize. Allowing to flip the signs of the remaining nonzero weights, the unmasking phase significantly alleviates the limitation in model capacity. That is, one sparse NN of le non-zero elements can be augmented to 2* possible candidates, if sign flipping is enabled. However, it is computationally intractable to exhaustively search for the optimal vector. To approximately solve (4) with sign flipping transformation U., we leverage a prior art called Bop from BNN optimization (Helwegen al., 2019). As formulated in Appendix A, Bop selects what weights and when to flip their signs by taking into account the exponential moving average of historical gradients. Different from the edge-popur algorithm used by Ramanujan al. (2020) and a similar method in (Ivan & Florian, 2020), Bop gets rid of additional latent weights, hence incurring more training overhead and deviating from our end goal of more efficient training. Bop also avoids the use of -thoogh estimator (STE) during backpropagation, which is known to unstabilize the training.
@ s to lower the training loss. Here is (+1,-1) binary mask that
3.3 FLIPPING USING CHEAP GRADIENTS: TOWARDS REAL COMPUTATIONAL EFFICIENCY
The training-by flippini scheme only seeks to optimize the sign configuration, not the the real- valued weights (they are fixed as initialized). Hence, educated guess is that this scheme can
5

---

Published as a conference paper at ICLR 2022
tolerate "coarser" gradient information than standard training methods. This leaves decent room for us to further reduce the training cost by replacing precise gradients with approximated ones. One way to implement this is to adopt low-precision gradient approximation called Predictive Sign Gradient, which was shown to save computation on hardware (Wang et al., 2019).
PSG predicts the signs of the precise gradients. using the signs of approximated gradients calculated from the most significant bits (MSB). PSG only corrects its predictions when the approximated gradients are close to zero, e.g., implying non-trivial chance of wrong predictions. Specifically, consider a one-layer neural network with input æ, weight matrix w and output wa. During back- propagation, we receive 9y aC/0y for the proceeding layer, and calculate the weight gradient Iw aC/0w gy and the input gradient 9N OC/0c w" Py. PSG replaces (a,w, w. gy) in
the above calculations with their bit-quantized versions (z, w, my), with bit-width (bx, bw, bg), and uses the resulting نhe o and O2 = w² By as the approximated weight and input gradient. Iz is further propagated to preceding layers run the back- prop-propag
In PaB. we use Iw itself instead of its sign, either in flipping with latent weights or flipping with Bop. This because without changing the weight magnitudes, we only have two actions to flip or not to flip. Too much training variance will be introduced if we directly use (āg)( to flip the weight signs, thus causing instability and difficulty in training. Moreover, we ignore elements in س. that are close to zero, rather than correct them with precise gradient calculation. This can reduce the training cost and also help to smooth the training process. Gradients to be ignored are selected using an adaptive thresholding technique in (Wang et al., 2019). This behavior is discussed in Section 4.4. The full PaB framework summarized in Algorithm 1.
Algorithm 1 PaB Efficient Training Framework
Require: Randomly initialized network f(x; (o)), dataset D sparsity S, threshold 6 total number of training steps T, PSG configuration [bz, bw. bgl-
{ /i loss function CD,
1: m < Prume (f w), D) with ||m]]oo = S
>> Get the sparse mask by pruning D Initialize sign configuration and time step
2: s(0) + (1,.... -1
3: while
T do
4: ((() + PSG (Ln, I (2; (0) O m + a(t),, (br, bw, by))
>> Approximate gradient using PSG
5: g(++1) Flip (s()), (
1t )E=))
Flip signs using historical gradient information
6: t=t+1
7:
end while
8: return f(x;w( m
4
EXPERIMENTS
4.1 MAIN RESULTS ON CIFAR-10 AND CIFAR-100
Comparison Methods. We compare our proposed PaB method with the standard SGD training for the dense network (Dense-SGD) and the edge edge-popup algorithm (Ramanujan al., 2020). For the non-fipping training for pruned networks, we choose standard SGD and SignSGD (Bernstein et al., 2018) as the baselines. For the proposed PaB methods, we compare: (1) PaB-Latent, which first prunes the network and trains the unpruned weights using the flipping with latent weights method in (Ivan & Florian, 2020) 1; (2) PaB-Lateni-P'S PSG, the variant PaB-Latent combined with PSG; (3) PaB-Bop, which trains the unpruned weights using Bop (Helwegen et al., 2019); and (4) PaB-Bop- PSG, the variant of PaB-Ba PaB-Bop combined with PSG. We also introduce a variant of Dense-SGD as baseline called Dense-SGD-SMe the number of training BitOps of that in PaB-Bop-PSG.
where cut down the training epochs of Dense-SGD to match
Experiment and evaluation settings: We evaluate the performance of these PaB methods on the CIFAR- 10 using large ResNet-18, ResNet-50 architectures, in contrast to previous papers on flipping based training (Ivan& & Florian, 2020) and lottery tickets (Zhou et al., 2019) where their methods
'The latest manuscript of (Ivan & Florian, 2020) just reported a new result on ResNet-18. However, their baseline accuracy number (90.45%) is inferior to what is commonly used by the community (over 95% using the implementation in ht attpp:: /github con. kuangliniii iu/ httppyyttrcchttooo cirr.. Their flipping algorithm ends up ~3% accuracy drop on this baseline. Without access to their official codes for large models, we are not sure whether we could draw fair comparison with their results.

---

Published as a conference paper at ICLR 2022
Table 1: Experiments on CIFAR-I0 using ResNet-18 and ResNet-50. Models are initialized with
Kaiming Normal distribution and pruned using SynFlow with pruning ratio 90% at initialization.
ResNet-18
ResNet-50
Model
Acc (%)
Size (MB)
BitOPs
Ace (%)
Size (MB)
BitOPs
Dense-SGD
95.10
42.59
1.71Tr
95.34
89.54
3 3,99Tr
Dense-SGD-Short
92.52
42.59
0.30Tr
86.19
89.54
0.55Tr
Edge-popup (Ramanujan et al., 2020)
58.54
1.46
1.20Tr
75.90
3.08
2.79Tr
SGD
94.39
5.59
0.72Ir
94.46
11.75
1.32Tr
SignSGD (Bernstein al., 2018)
93.15
5.59
0.72Tr
93.61
11.75
1.32Tr
PaB-Latent (Ivan & Florian, 2020)
91.82
1.46
0.72Ir
89.80
3,08
1.32Tr
PaB-Latent-PSC
91.84
1.46
0,30Tr
90.25
3,08
0.55Tr
PaB-Bop (Helwegen et al., 2019)
92.55
1.46
0.72Tr
92.13
3.08
1.32Tr
PaB-Bop-PSG
92.71
1.46
0.30Tr
91.61
3.08
0,55Tr
Table 2: Experiment results on CIFAR-I0 and CIFAR-I00 100 using WideResNet-28 as testbeds. All
models are initialized with Kaiming Normal distribution and pruned using SynFlow with pruning ratio 70% at initialization. * We perform a range of hyperparameter tuning on SignSGD and report the best accuracy observe.
WideResNet-28. CIFAR-10
WideResNet-26. CIFAR-100
Model
Acc (%)
Size (MB)
BitOPs
Acc (%)
Size (MB)
BitOPs
Dense-SGD
96.30
139.10
18.28Tr
81.11
139.10
18.28Tr
Dense-SGD-Shon
92.29
139.10
2.03Tr
77.04
139.10
4.20Tr
Edge-popup (Ramanujan et al., 2020)
40.77
4.78
12.80Tr
45.01
3.08
12.80Tr
SGD
95,50
18.26
4,86Tr
80.28
46.08
10.10Tr
SignSGD (Bernstein et al., 2018)
94.18
18.26
4.86Tr
77.82
46.08
10T
PaB-Latent (Ivan Florian, 2020)
4.78
4.86Tr
4.78
10.10Tr
PaB-Latent-PSC
89.29
4.78
2.03Tr
68.90
4.78
4.20Tr
PaB-Bop (Helwegen et al., 2019)
94.41
4.78
4.86Tr
77.59
4.78
10.10TT
PaB-Bop-PSG
94.26
4.78
2.03Tr
77.81
4.78
4.20Tr
scale only up to 6-layer conv networks. In all experiments, initialize networks using Kaiming Normal initialization (He et al., 2016), prune to 90% pruning ratio using SynFlow for 100 iterations (Tanaka et al., 2020) at random initialization, and train over 200 epochs. For Bop, start with adaptivity ratio = 10-3, which is decayed by 0.15 every 45 epochs, and use threshold r = 10-6 For non-Bop methods and non-Bop weights,
an initial learning rate of 0.1 and decay them by 0.1 at epoch 80 and 120. We use an initial learning rate of 0.01 for SignSGD as smaller learning rates are favored in (Bernstein al., 2018). Results are presented in Table 1, including accuracies, model sizes and training BitOPs (in trillions) for each method and network. Huffman coding is used encode the PaB models as described Appendix We calculate training BitOPs following (Jin et al., 2020) instead of FLOPs because PSG involves low-precision computation.
Main observations: Several observations can be drawn from the results:
As expected, the PaB framework results in impressive compression ratios (> 30x) on all mod- cls. thanks the fact that dipping-based methods leave weight magnitudes unchanged, which can be restored easily using the same random seed.
We see that PaB-Bop and its PSG variant are consistently better than PaB-Latent and the PSG variant. This shows the superiority of Bop-based methods over latent- latent-weight methods (Ivan & Florian, 2020) terms of the effectiveness when training sparse networks. We perceive this observation as an indication of the potential existence of even better sparse training methods than the Bop-based method that can exploit the sparse structure more effectively.
The integration of PSG with PaB further reduces the training cost by significant margins, saving 82~866% of the total BitOPs that is needed for training a dense counterpart.
PaB-Bop-PSG performs comparatively well for BesNet-18/50 achieving 1.68% and 2.33% accuracy gaps respectively compared to SGD and even smaller gaps compared to SignSGD.

---

Published as a conference paper at ICLR 2022
Test Accuracy 1%9
Test Accuracy (%%
Thxt Acturacy (%)
90
80-
80
Edge-
Edge-popup
70
Edge-poupp
PaB-Bop- Bsp-Bop-P
PaB-Bop-Pas Pap
PaB-Bop-Pso
Dense-SGD
Dense-SG SGD
60
Dense SGD
30
50 Pruning Ratio
90
30
50
70
70
Pruning
Pruning Ratio
(a) Conv 4
(b) Conv
(c) ResNet-18
Figure 2: Comparing the performance of PaB-Bop-Pos and the edge-popup algorithm (Ramanujan al., 2020) on Conv4, Conv6 and ResNet-18 the CIFAR-10 dataset.
Results wider networks: Besides scaling ResNet- 18 up to ResNet-50 in depth, also show the scalability of PaB in width by training WideResNet-22 with 10x width multiplier, on both CIFAR- 10 and CIFAR- 100 datasets. As from the results shown in Table 2, PaB- PaB-Bop- Bop-PSG scales very well to WideResNet-25 on both CIFAR-10 and CIFAR-100. On CIFAR-10, PaB-Bop-PSG incurs 1.24% gap compared SGD and even outperforms SignSGD. Similarly CIFAR-100, PaB- Bop-PSG incurs 2.47% gap from SGD and performs similarly to SignSGD. The accuracy gaps of PaB on WedeResNet-28
even smaller than on MesNett-1/50, demonstrating that PaB scales well to wide networks, potentially due to better gradient flow in wider architectures. We omit PaB- Latent results here because we fail train PaB-Latent to reasonable accuracies for WideResNet-28. which, in addition to the observed gaps to PaB-Bop on ResNet- 18 and -50, shows the superiority of Bop-based methods over flipping with latent weights.
4.2 COMPARISON WITH STATE-OF-THE-A ART EFFICIENT TRAINING METHODS
We compare PaB with state-of-the-art efficient training methods, E²-Train (Wang et al., 2019) and EB-Train (You et al., 2019). E²-Train improves training efficiency by combining data dropping, model layer skipping, and gradient quantization, and
93.0
PaB-Bop- PSG
E2-Train
Test Accuracy
99.55
EB-Train
TB-Train utilizes structured pruning and the early emergence of sparsity patterns. We train ResNet-18 on CIFAR-10 using all three methods with different trade-offs. As shown in Fig- ure PaB achieves better accuracy- BitOpps trade-off than Er-Train and EB-Train, especially when high training efficiency is demanded (small BitOps). The x-axis is the average BitOps for training one sample for one step. We present the detailed protocol for this comparison in Appendix.
92.0
91.5
91.0
0.2
0.4 Avg BitOps (Tr)
Figure Comparison between PaB and state-of-thee tf-th-a-tt efficient training methods. PaB achieves better trade-off between train- ing BitOps and accuracy.
4.3 COMPARISON WITH EDGE-POPUP ALGORITHM
We compare PaB with the rdge-popup algorithm proposed in (Ramanujan et al., 2020) to identify the hidden subnetworks in deep neural networks their random initializations. We adopt two small networks, Conv4 and Conv62 used (Ramanujan et al., 2020) and a larger network ResNet- 18 for comparison. All models trained CIFAR-10. We apply PaB and edge-popup to the testing networks with pruning ratios 30%. 50%, 70% and 90%. Results presented in Figure 2 show that PaB consistently finds better disguised subnetworks than the hidden networks found by edge -popp algorithm, especially when the pruning ratios are high.
We further implement PaB the ResNet-50 for ImageNet and compare with edge-popup popup (Ra- manujan et al., 2020). For both methods we train the ResNet-50 50 network for 100 epochs, following the same settings as (Ramanujan al., 2020). With 30% and 50% sparsity ratio (the percentage of pruned parameters), PaB-PSG can achieve 63.58% and 63. 63.25% accuracy, in contrast to the 39.5% and 58.4% accuracy achieved by
We follow the official implementation of the edge-popup algorithm but make slight modifications to the Conv4 and conv6 networks by adding batch normalization layers stabilize the training.
8

---

Published as a conference paper at ICLR 2022
Poo-famm
104
5.0
62.
000
175
920
Figure 3: PaB-Bop accuracy CIFAR- 10 us- ing different hyperparameter combinations of 7, with R8 Networks are pruned us- ing SynFlow with 90% pruning ratio.
Figure 4: PaB-Bop-PSG accuracy on CIFAR-10 using different PSG behaviors and thresholds. Networks are pruned using SynFlow with 90% pruning ratio.
4.4 ABLATION STUDIES ON PAB
Due to space limit, we only present the ablation study on two important sets of hyperparameters in PaB, namely the hyperparameters for BOP, and those that control the small gradient behavior in PSG. We leave more ablation studies on the pruning ratio, the epoch when pruning occurs, and the random distribution for initialization the Appendix. We use ResNer-18 and CIFAR-10 by default.
BOP hyperparameters: Different from PaB-ILatent, which in nature still trains latent weights using SGD, PaB-Bop consists of two parts: the accumulation of historical gradients and thresholding mechanism to select significant updates in the correct directions, controlled by adaptivity ratio and threshold т. respectively, as formulated in (6). Therefore, we conduct an ablation study on y and f and present the results in Figure 3. We prune ResNet-18 networks with pruning ratio 90% using SynFlow and then train them on CIFAR- 10 using PaB-Bop with different combinations of ? and т. When y is small, the latest gradient makes too small contributions in the accumulation to cause enough updates. In contrast, when 7 = 0.01, Bop needs a larger threshold r to reduce the training variance. The best combination is (7, = (10-3, 10-6).
Dealing with small gradients in PSG: the original PSG (Wang et al., 2019), when PSG obtains small gradient approximations (judged by threshold 0), PSG corrects them with precise gradients using full-precision calculation. Here we study three different PSG behaviors in the framework of PaB: (1) PSG-AII, where take whatever gradient approximated; (2)
the default behavior in (Wang et al., 2019), which corrects small approximated gradients with precise ones; and (3) PSG-Remove, which removes small approximated gradients. The accuracy results of applying PSG with these three different behaviors to ResNet-18 pruned by SynFlow are shown in Figure 4. Note that ESG-CorrectRRemoee reduces PSG-AII with threshold. The results show that PSG- Correct and PSG-Remove perform similarly at their best thresholds. Since PSG-Correct requires extra computation of precise small gradients, we prioritize efficiency and use the PSG-Remove behavior in all PSG experiments.
5 CONCLUSION, FUTURE WORKS AND DISCUSSION OF SOCIETAL IMPACTS
We study Peek-a-Boo (PaB) as new minimalist approach to training dense networks at their ran- dom initializations without altering weight magnitudes. Inspired by the recent exploration of sparse NNs, PaB achieves high training efficiency and provides a natural lossless compression after train- ing. While this work remains and datasets (e.g, up to ResNet-50 and ImageNet) with competitive trade-off between accuracy and training efficiency. We hope that PaB will evoke more theoretical reflections training. One of our immediate future work will explore alternatives to the sign flipping operation as the weight transfor- mation in the unmasking phase. For example, we extend sign flipping with learnable scaling, or replace it with guantization. In another direction, while this paper mainly uses unstructured sparsity following the convention of pruning at initialization, we can explore how structured sparsity (You et al., 2019) may fit in PaB for practical hard hardware acceleration.
pilot study, show it can already scale up to larger networks
LCKNOWLED
Z. W. is supported by a US Army Research Office Young Investigator Award (W911NF2010240) and NSF SCALE MoDL, project (2133861).

---

Published as a conference paper at ICLR 2022
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 130.iv:300.333 3432, 2013.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp. 560-569. PMLR, 2018.
Miguel Á. Perreina and Yerlan Idelbayev. "learning-compresion"" algorithms for neural net pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre- ceo-trained bert networks, 2020.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16306-16316, 2021a.
Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, and Zhangyang Wang. The elastic lottery ticket hypothesis. Advances in Neural Information Processing Systems, 34, 2021b.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Da-Ynnv,, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint arXiv: 2016.
J. Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In NAACL-HLT, 2019.
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv: /906. 10732, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https:/ openreview net/ / for uorum? www.mmiddixiii -1-
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv: 01631 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259-3266 3269. PMLR, 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural net- works initialization: Why are missing the mark? In International Conference on Learning Representations, 2021. URL https: http://pppeeeeeeeeee net forum? -eeeiii=--iiiiiiie VyQo-MyQaMaa MLK.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Yiwen Guo, Anbang Yao, and Yurong Chen. dnns. In NIPS, pp. 1379-1387. 2016. URL http://papez: nips cc/paper/ 6165-dynamic-netwoxz
Dynamic network surgery for efficient
Song Han, Jeff Pool, John Tran, and William J. Dally. both weights and connections for efficient neural network. pp. 1135-1143, 2015.
Learning NIPS,
URL http: /tto://paprraae nips cttbossscccpppp/////// and- and-connectiin connectionsssssoeeooo for-efficient wtwpriiiiennnttrrra---eeeeeteeee
Babak Hassibi, David G Stork, and Gregory J Wolff Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp. 293-299. IEFF, 1993.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang- Ting Cheng, and Roe- land Nusselder. Latent weights do not exist: Rethinking binarized neural network optimiza- tion. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: htttp::://prooeedngg..... hwwpeeeedddnns.euuuugeeeeee ips.ppss.oo cc/ wwtpppapae//21////ieeeeee 9ca8c9b0 0ww.099600000070000555550555 Paap:aaa-66666paaapaa..p..-
Cristian Ivan and Razvan Florian. Training highly effective connectivities within neural networks with randomly initialized, fixed weights. arXiv preprint arXiv:2006.1660 2020.
Qing Jin, Linjie Yang, and Zhenyu Liao. Adabits: Neural network quantization with adaptive bit- widths. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Yann LeCun, John S Denker, Sara A Solla, Richard E Howard, and Lawrence D Jackel. Optimal brain damage. In NIPs, volume 2, pp. 598-605. Citeseer, 1989.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHOP NETWORK
PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learn- ing Representations, 2019. URL https: / wpw.revie...eeeeeeeeeeeeeeoe.wee net.eee.eeeeeeeeeeeeeeeeeeeeeeee forum? i d=
Ning Liu, Geng Yuan, Zhengping Che, Xuan Shen, Xiaolong Ma. Qing Jin, Jian Ren, Jian Tang, Sijia Liu, and Yanzhi Wang. Lottery ticket implies accuracy degradation, is it a desirable phenomenon? arXiv preprint arXiv:2102. /068. 2021.
Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. International Conference on Machine Learning, pp. 6682- 6691. PMLR, 2020.
Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: gen- eralizing lottery ticket initializations across datasets and optimizers. In NeurlPS, pp. 4932 4942, 2019.
Laurent Orseau, Marcus Hutter, and Omar Rivasplata. Logarithmic pruning is all you need. Ad- vances in Neural Information Processing Systems, 33, 2020.
Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopoulos. Optimal lottery tickets via subsetsum: Logarithmic over-parameeerizzaionn is sufficient. arXiv preprint (tl1:20066.79900 2020.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari. What's hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Kartik Sreenivasan, Shashank Rajput, Jy-yong Sohn, and Dimitris Papailiopoulos. Finding every- thing within random binary networks. arXiv preprint arXiv:2110.08996, 2021.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006.05444 2020.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by
preserving gradient flow. In International Conference on Learning Representations, 2020. URL
https: wwwpreweeeeee.eee/////// net perpenrreeeweeet//ooou...eieede
Yue Wang. Ziyu Jiang, Xiaohan Chen, Pengfei Xu, Yang Zhao, Yingyan Lin, and Zhangyang Wang. E2-train: Training otate-of-the-an cnns with over 80% energy savings. arXiv preprint 1r1iv: 13349, 2019.
11

---

Published as a conference paper at ICLR 2022
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33, 2020.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint 1XXi::990.11999950 2019.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Cur- Associates, Inc., 2019. URL https: http::/ss//p//eeeiipn/eooo. neurips wccppppp//011/33 Ette://2.3.cafffrccaaa3333311113 wwppp.ppppa....eee.eeeee.ee.....
12

---

Published as a conference paper at ICLR 2022
A BOP FORMULATION
Bop (Helwegen et al., 2019) selects what weights and when to flip their signs by taking into ac- count past gradient information. particular, Bop accumulates the exponential moving average of historical gradients:
= -1 + +VV,.colfuu()
(5)
where y is called the adaptivity ratio. Bop flips weight signs when the moving average surpasses certain threshold and aligns with the weights in direction:
w((
|e 2 v and w()) > 0,
((++1)
(
otherwise.
B PROTOCOL OF COMPARING PAB WITH E2-TRAIN AND EB-TRAIN
EB-train (You al., 2019) is easier to control the computational costs and calculate its BitOps. EB- Train consists of three stages: (i) the searching stage where a dense model is trained with sparsity regularization applied on the scaling factors of the batch normalization layers; (ii) prune the filters when the convergence of the pruning mask is detected; (iii) continue training the pruned models. In the first stage, EB-Train has the same computational cost as a dense model, but only for a short period of training. After the pruning is performed in the second stage, we can easily calculate the reduced BitOps based on the layerwise pruning ratios induced by pruning. Such reduced computational cost will be sustained throughout the third stage. In the experiment shown in Figure 1, we change the pruning ratio in EB-Train to control the BitOps.
E2-Train Wang et al. (2019) is more tricky baseline because it consists of three components stochastic mini-batch dropping (SMD), selective layer update (SLU) and predictive sign gradient (PSG). We follow the setting of SMD ratio, i.e the ratio of randomly dropped mini-batches during training and fix it at 50%. The BitOps reduction brought by PSG is decided by the bit-width of the most significant bits. We use the same 8-8- 16 bit widths for inputs/activaations weights and gradients respectively as in (Wang et al., 2019) and use full pelliirccisoon computations for the forward passes. We then change the computation budget in SLU control the BitOps in Figure 1.
Although the computation reduction of PaB fixed given the sparsity ratio and the bit-width con- figuration in PSG, PaB is orthogonal to stochastic mini-bater dropping and can be combined with it. Hence, we change the ratio of dropping mini-batches to control the BitOps of PaB-Bop-PSG in Figure 1.
C ADDITIONAL ABLATION STUDIES
Pruning time point. Although pruning at initialization has shown success at training sparse net- works, (Frankle et al., 2021) identify that pruning different time points after initialization could further improve the performance
these networks. Using a fixed pruning ratio of 90%, we train ResNet-18 on CIFAR- 10 and observe the effects on the test accuracy of delaying pruning different training points from epoch 0/200 to epoch 60/200. Figure 5a shows the performance of our non- flipping baselines and PaB methods the pruning epoch varies. Different from the observations in (Frankle et al., 2021), while slight increases in accuracy can be observed with delayed pruning, the effect quite small the scale of this experiment. The reason might be that we fix the total number of epochs whenever we perform pruning, while (Frankle et al., 2021) )) goes through the whole train- ing process after pruning. With efficient training mind, we maintain pruning initialization in subsequent experiments since delaying pruning after initialization would require expensive training of dense networks before pruning.
Pruning ratios. We also evaluate the performance of our PaB methods against our baselines at different levels of sparsity. Figure 5b shows the test accuracies from training ResNet-18 on CIFAR- 10 at varying pruning ratios from 0% to 90%. Not surprisingly test accuracy when we gradually increase the pruning ratios for all methods, especially going from 80% 90%. However, PaB-Bop and its PSG variant more robust to high pruning ratios, while
decreasing tendency in
13

---

Published as a conference paper at ICLR 2022
94
Tex Accuuedy
Accua aauuuaayy
92
20
Pruning
Pruting Ratio
Pali-Latent
Pati-BOP PaB-BOR-PGG
5GD Sgr0GD
Pati-Latent
PaB-BOP PMB-BOF-PSG
Sgr0GD
(a) Ablation study of pruning time epoch
(b) Ablation study of pruning ratio
Figure 5: Accuracy of baseline and PaB methods for ResNet- 18 and CIFAR-10 with different prun- ing time epochs (0-60 epoch) and pruning ratios (0-90%).
PaB-Latent and PaB-Latent- have obvious degradation at 90% pruning ratio. Beyond 90%, the test accuracy starts to drop noticeably, and the resulting sparse networks becoming untrainable for all training methods due to layer collapse. By default hereinafter, we fix the pruning ratio to 90%.
Ablation on initialization. In previous experiments, we use Kaiming Normal dis- tribution for random initialization. We also try Kaiming Uniform and Xavier Nor- mal (Glorot& Bengio, 2010) distributions as alternatives. As shown in Table 3, there is generally little variation in performance across initialization methods, but able to achieve the best results across all PaB methods from PaB-Bop-PSG initial- ized using the Kaiming Normal distribu- tion. We can also see that all PaB methods consistently perform better under Kaiming Normal initialization than under Xavier Normal initialization.
Table Accuracy on CIFAR-10 with different initial- ization for ResNet-18. All networks are pruned using SynFlow initialization with 90% pruning ratio.
Kaiming Normal
Kaiming Uniform
Xavier Normal
Method
SGD SignSGD
94.39%
94.63%
94.31 %
93.15%
93.41% %
93.39%
PaB-Latent
91.82%
92.12%
91.75%
PaB-Latent-PSC
91.84%
92.08%
91.47%
PaB-Bop
92.55%
92.62%
91.91% %
PSG
92.71%
92.53%
92.21% %
Iterative pruning and sign-flipping training. Inspired by the iterative scheme of IMP, which is observed to be essential for finding winning tickets, we extend SynFlow from one-shot pruning to multiple-shot over period of the training process. Note that one shot of SynFlow can contain many iterations of pruning as described in Tanaka et al. (2020). Here by "multi-shot" we mean to apply SynFlow several times throughout the training process to gradually increase the sparsity level. Specifically, instead of applying SynFlow to the networks once at initialization, 2-55 times, evenly spread over the first 50 epochs, so that the sparsity ratio linearly increases to a certain level (90%). Between two runs of pruning after the last run of pruning, perform the flipping-based training using Bop.
run SynFlow for
This iterative multi-shot version of PaB can be seen alternating between the optimization pro- cesses of the searching and unmasking phase to discover the disguised networks. Compared to the one-shot variant, multi-shot PaB is closer a joint optimization scheme on the objective in (I). The results are shown in Table 4. We observe that such an alternative scheme provides little accuracy improvement, while apparently losing more of training efficiency. We do not intend say that the joint optimization is not good: to properly solve the joint optimization problem in (1) with itera- tive algorithms is interesting and potentially desirable. Our argument is that, with our goal to do "efficient" finding of disguised subnetworks, our current two-step relaxation is not only the most efficient; but also highly competitive accuracy with no sacrifice.
Changing the bit-widths in PSG. In PSG, the bit-width configuration (ba, bu. by) for layer inputs, layer weights and output gradients controls the trade-off between training efficiency and the pre- cision of approximated gradient, and thus the model performance. We apply PaB-Bop-PSG with varying bit bit-configurations and show the results in Table 5. We observe that the bit-width of output gradient by has strong influence the final performance, which is consistent with previous low- precision efficient training works (Wang et al., 2019). We choose (bz, bw. by) = (8, 8, 16) as it is a combination of good performance and training efficiency.
14

---

Published as a conference paper at ICLR 2022
Table 4: Results of the iterative multi-shot variant of PaB. We run SynFlow for 2-55 times, evenly spread over the first 50 epochs, so that the sparsity ratio linearly increases to certain level (90%). Between two runs of pruning or after the last run pruning, we perform Bop training.
Runs of SynFlow
2
SGD
94.79%
94.76%
94.48%
94.68%
SignSGD
93.20%
93.57%
93.51% %
93.77%
PnF-LLaeent 91.98%
92.05%
92.00%
92.29%
PnF-Latent-PSG
91.93%
91.94%
92.06%
91.89%
PnF-Bop
92.25%
92.18%
92.15%
92.48%
PnF-Bop-PSG 92.28%
91.89%
92.25%
92.32%
Table 5: Accuracy and training cost in BitOPs on CIFAR-10 with varying bit-width configurations for ResNet-18. All networks are pruned using SynFlow at initialization with 90% pruning ratio.
Configuration (4,4,4)
(4,4,8)
(8,8,8
(8,8,16)
(16,16,16)
(16,16,32)
Accuracy (%)
32.38
89.68
89.02
92.71
92.24
92.29
BitOPs
0.25Tr
0.25Tr
0.27Tr
0.30Tr
0.36Tr
0.48Tr
D HUFFMAN CODING FOR ENCODING MASK AND SIGN ENCODING
PaB methods, without changing the magnitudes of the random initialization, only induces two out- puts that need to be stored (i) the sparse mask generated by the pruning methods (SynFlow this work); and (ii) the sign flipping actions for all weights. These format, where the pruned weights are represented by 0, sustaining the sign by +1 and flipping the sign by -1. Considering that the pruned weights account for the majority of all weights (90% in our CIFAR-10 10 experiments and 70% in our CIFAR-10 100 experiments), we can adopt Huffman coding encode 0 using 1-bit representation, and the sign flipping actions with 2-bit representation, as illustrated in Figure 6.
outputs can be merged into one
Sign Corlig
Huffman
Erooded (1
Figure 6: Huffman encoding for PaB model compression.
15


Reviews:
[{"review_id": "orPXKxgINHE", "reviewer": "Reviewer_7V6H", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper is extremely well-written. There are no typos, the language and the presentation are very strong, and the flow is very fluent. Moreover, the literature is summarised very precisely and it includes most of the relevant papers to discuss, to the best of my knowledge. The algorithm is described well, and the contribution is discussed in a fair way without unrealistic claims.\n\nMy major concerns are as follows:\n\n1 - **Contribution.** Problem (2) formally describes the “optimal hidden subnetwork” problem, and it immediately follows from the definition of problem (1) that what the authors suggest will be an ‘improvement’ over the training data. This is not unimaginable, that is, it is already known that any transformation of weights we optimise over the training data will be an improvement. \n\n2 - **Novelty.** As discussed above, rather than the definition of disguised subnetworks, what matters most here is, (i) the classes of “U” (transformation) that can be handled in this work, and (ii) novelty of the solution algorithm the work proposes. Regarding these two points: \n-  (i) the class of U is constrained to sign flips, which is already well-studied and being applied in similar settings; \n- (ii) the solution algorithm splits the problem into two sub-problems. The first sub-problem solves for “m” (the masking variable) by using existing training data-independent methods (cf. SynFlow). After fixing “m”, the algorithm simply proceeds with the Bop method from the BNN optimisation literature. My concern here is that, since in the loss function the order of these variables is loss [ U( .. m ) ], first solving for \"m\" and then for \"U\" is the most trivial way, and while doing this using all of the well-known results is not bringing much novelty. \n\n3 - **Generality.** The paper is defining a very general problem of optimal disguised subnetworks and mentioning this in the title/abstract/ and throughout the paper. However, in my view, this work is more like a case-study where heuristic methods are being used iteratively. Namely, the contribution is actually: “taking a recent paper that finds a masking variable, and just flipping the signs w.r.t. the training loss”. This is still an interesting result, but I also would like to highlight the distinction from my perspective. Finally, I would like to highlight that, although the optimization formulation of \"disguised subnetworks\" finds U (transformation) and m (masking) jointly, the relaxation proposed (PaB) first splits the problem so that the solution algorithm first finds a sub-network (i.e., m is found) and then changes the weights. So this algorithm does not find a disguised subnetwork (as claimed in the introduction), rather finds a subnetwork and then re-assigns the weights, which is *a* disguised subnetwork, but not a new architecture. I believe the problem definition of disguised networks is very general but PaB first finds the hidden architecture so this is not a new way of finding subnetworks.\n\n4 - **Numerical Experiments.** This is a question for the author(s) rather than a direct concern: some of the algorithms in the numerical experiments, where PaB is being compared with, are training-independent algorithms. The fact that they are training-independent (or sometimes called “without training methods”) is the main focus in the literature around them. Malach et al. (2020), on the work(Ramanujan et al., 2019) comment: “within a sufficiently overparameterized neural network with random weights (e.g. at initialization), there exists a subnetwork that achieves competitive accuracy”. Is this mentioned in the paper? Otherwise, the fact that here there is an optimisation procedure after fixing the sparsity structure based on the training set already changes the setting of the problem here and introduces a bias in comparison?\n\nHowever, I still think the idea of combining these existing methods to find a solution for the optimal disguised subnetwork problem (and hopefully a good one), can pave the way for a new research direction. Here the key standpoint is, rather than the generality or the novelty of solutions, to demonstrate that instead of solving a dense network and “transforming the weights”, or to mask for a sparse network, one can combine these methods to have a sparse network that also performs better than the alternatives. The numerical experiments are conducted and reported carefully, and the results look promising. Hence, despite the lack of theoretical contributions, I would like to give my decision as \"marginally below the acceptance threshold\".\n\nMinor comments:\n- Page 2: “Even worse is” maybe better to re-write?\n- Page 3: “Unfortunately, the original experiments in (Zhou et al., 2019)” maybe change ‘unfortunately’?\n- Page 5: In the first sentence, “s” is not capital but previously it was. The same happens again in the “Unmasking phase” part of S3.2.\n- “Computationally infeasible” -> “computationally intractable”\n- “ = argmin “ -> “ \\in argmin” (or prove that argmin set is a singleton) \n- Page 5: “For example, one sparse NN of k non-zero elements  […]” is this called an example because k is named? Maybe it is better to connect with before “, that is, if a sparse NN …”?\n- Page 5: Optimisation over 2^k many vectors is called “NP-hard”. I am not sure but this terminology may be misleading. The optimisation problem itself can be tractable but there the input is just intractably large in “k”.  I am not very sure about whether it is common to say the problem is NP-hard. For example, with that logic, max{a_1, a_2, … , a_n }, that is found in linear time, is NP-hard in log(n)? \n- Page 7: “and the PSG variant,.” There is an additional comma (,) before the full stop (.)\n", "summary_of_the_paper": "This paper extends the definition of the hidden subnetworks in randomly initiated neural networks. The new notion of subnetworks, namely the disguised subnetworks, apply a transformation on the hidden subnetwork weights to obtain final weights (a posteriori finding the hidden subnetwork). \n\nMathematically speaking, the main decision variable of the underlying optimisation problem of finding “hidden subnetworks” is the so-called masking variable that is a binary vector, the variable that decides which components of the randomly initiated weights will be zero, constrained to a desired level of sparsity. On the other hand, the underlying optimisation problem of “disguised subnetworks” has an additional variable U that applies a transformation on the set of weights after being ‘sparsified’. The selection of U = I, for I being the identity transformation, recovers the problem of hidden subnetworks, showing that the latter is a generalisation.\n\nThe author(s) present a heuristic algorithm that solves the forenamed optimisation problem. The idea is to first find a solution for the masking variable where (i) U = I is taken, (ii) the objective function is independent of a training set. Afterward, the algorithm proceeds by solving the problem for U given the solution of the previous step, where the space of U is restricted to the class of transformations where only sign-flips are allowed. If we think of the above solution process as a two-phase problem, the author(s) use the literature on sparse neural networks for the first phase and use the literature on binary neural networks for the sign flipping phase.", "summary_of_the_review": "**Pros**\n- The paper has a very clear overview of the relevant research, and in general, the paper is written very well\n- The numerical experiments are very thorough\n- The idea of generalizing the notion of hidden subnetworks is relevant\n**Cons**\n- The contribution is limited -- the definition of disguised networks is straightforward and the solution method comprises a subsequent application of existing algorithms (not parallelized or nested, but applied sequentially)\n- The optimal disguised networks problem is restricted iteratively so that the end result is not very interesting anymore", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MT6KYdUIJvo", "reviewer": "Reviewer_QSy2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is well written, and does a fair job covering recent works. \nThe definition of disguised subnetworks is clear and properly motivated.\nThe presentation of the PaB is clear and well motivated with good context.\nThe results are quite convincing that PaB scaled well to larger networks,\nsomething other methods lack, while providing good accuracy for much smaller\nand easier to train models.\n\nWeakness:\n\n1. I would have liked to see some more theoretical discussion, though the authors\nadmit this work is primarily empirical.\n\n2. It would be nice to discussion of other weight transformations U in the unmasking phase,\n    that may be useful.\n\n3. Though the definition of disguised subnetworks is new, and the generalized\napproach is novel, and the results speak to the validity of the approach, the\nPaB algorithm is a relatively straightforward application of two existing\nmethods.\n", "summary_of_the_paper": "Optimizing sparse neural networks is an important topic due to their\ncomputational and space savings.  Building on the work of the lottery ticket\nhypothesis, others have shown there exist hidden subnetworks within randomly\ninitialized NN that have good performance. The authors extend this definition\nto disguised subnetworks, which contain hidden subnetworks as a subclass.\nMoreover, the authors present a novel combination of existing methods \ninto a single algorithm they call Peek-a-Boo (PaB) which can efficiently\nfind such networks.\n", "summary_of_the_review": "The paper is well written, and does a good job covering all the related material.\nThe definition of disguised subnetworks is novel and will be useful for future researchers.\nThe PaB algorithm clearly yields good results on larger NN, something other methods lack.\nMore exploration of theoretical understanding as well as other weight transformations would be \nuseful.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "9qzqAZWTJJ", "reviewer": "Reviewer_rMZy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an algorithm named peek-a-boo (PaB) to optimize network pruning (at initialization) and optimization (limited within flipping the sign of weights). This setting has not been studied by prior works. A two-step algorithm was designed -- pruning first, optimization second. Experiments show competitive performance compared to prior methods with similar optimization complexity.\n\nThough the setting is new and interesting, I shall say that its value to the community has not been perfectly revealed based on the existing experiments. In the studied cases, the neural networks and datasets are mostly small. The value of this work is supposed to be accelerating large-scale network training, but I am not sure that it scales up well.\n\nSecond, it seems that the technical contribution of the proposed algorithm is limited. When I see the objective function (1), I was expecting that pruning and optimization are jointly performed, however, according to Algorithm 1, they were performed separately, and the pruning part follows an existing algorithm. Separate optimization brings the issue that the best pruned architecture may not fit very well in the unmasking setting. Therefore, I am wondering if there is an iterative solution, that pruning is divided into several steps (e.g. one decides to keep 50% weights, so he/she can prune 10% weights every time for 5 rounds), and after each step, the flipping operation is performed. This can align both parts and hopefully improve the final performance. Of course, this may increase the training BitOps, but this shall be a tradeoff -- anyway, showing whether this strategy helps is not a bad trial.\n\nThird, the experiments are performed on CIFAR datasets. Since ImageNet results are missing, I am not sure if the approach can actually scale up. In particular, according to Tables 1 and 2, PaB does not show a significant advantage over SGD, which further limits the application of the proposed approach. I think ImageNet experiments might be a good add-on to this paper.", "summary_of_the_paper": "This paper presents an algorithm named peek-a-boo (PaB) to optimize network pruning (at initialization) and optimization (limited within flipping the sign of weights). This setting has not been studied by prior works. A two-step algorithm was designed -- pruning first, optimization second. Experiments show competitive performance compared to prior methods with similar optimization complexity.", "summary_of_the_review": "1. The setting is new, but the practical value to the community seems unclear.\n2. The algorithm is straightforward, joint optimization is not considered.\n3. Experiments are promising but not exciting, ImageNet results are absent.\n\nOverall, a bit below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0x2svroFpQ", "reviewer": "Reviewer_jduC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength:**\n- This paper is very well written and a pleasure to read.\n\n- The paper tackles an important question, which is to rethink the optimization strategies of sparse neural networks, especially when using the idea of masking as training. The observation that when we are only looking to flip signs of random weights during training, we can tolerate much coarser gradients is insightful. The results are significant as it reduces training cost compared to fully trained networks while maintaining competitive performance.\n\n- The method is compared to an appropriate set of baselines over a reasonable set of model architectures. \n\n**Weakness:**\n- How exactly is the compression ratio calculated? The authors mention that this is due to the fact that the random initialization can be stored using a single seed. While this may be true in some cases, I would be hesitant to rely on this attribute as the seed may not result in the same parameters in different machines. It would be more reasonable to assume that the weight values must be stored for a more conservative estimate.\n\n- Most of the work in supermasks consider the \"signed constant\" variant, where the weights are converted to a single signed value for each layer. This parameterization has often shown improved performance over the parameterization using actual weight values, and leads to significant reductions in model complexity and size. How would this variant work under the PaB framework?", "summary_of_the_paper": "This paper proposes the idea of a \"disguised subnetwork\", which are hidden random subnetworks that can be transformed into a well-performing subnetwork. The paper introduces PaB as a way to uncover these subnetworks, by first searching for a mask over the random weights using pruning-at-initialization techniques, and then learning a transformation on the subnetwork. The paper further shows that this PaB process can be efficiently implemented, offering significant advantage over prior work.", "summary_of_the_review": "The method is novel and achieves better efficiency-performance tradeoff than the various baselines. It tackles the question of rethinking optimization strategies for sparse NN training, which I think is an important research direction. I think the contributions of this paper are significant and would support its acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I recommend acceptance. This paper presents an interesting "in-between" of work on lottery tickets and work on supermasks, and I think it is sufficiently novel to merit acceptance even if the significance of the results will need to be left to the judgment of future researchers. The reviewers seem broadly in favor of acceptance, and I defer to their judgment as a proxy for that signal.

For a quick bit of context, work on "supermasks" (Zhou et al., 2019) has shown that randomly initialized networks contain subnetworks that can reach high accuracy without training the weights themselves. That is to say, within randomly initialized networks are high-accuracy subnetworks. This work is interesting in its own right and has had a number of interesting implications for the theoretical community. This work derives from work on the lottery ticket hypothesis (LTH; Frankle & Carbin 2019), which shows that randomly initialized networks contain subnetworks that can train to full accuracy on their own. The key distinctions between these two kinds of work are (1) the LTH trains the subnetworks, while supermask work does not and (2) the LTH work requires that the subnetworks train to full accuracy, while work on supermasks obtain high (but not full) accuracy in many cases. No one approach is "better" than the other; they simply showcase different properties of neural networks.

As far as I understand, this paper creates space for an "in-between:" high-accuracy subnetworks are created by finding subnetworks at random initialization and flipping the signs of some of the weights to improve accuracy. This is a limited modification to the subnetworks that falls short of actually training them (LTH work) but is more than leaving them at their random initializations (supermask work). Doing so appears to produce subnetworks that perform better than in supermask work but with a lighter-weight procedure than LTH work. The procedure for accomplishing this feat is different than either approach (using SynFlow to find the subnetwork and a binary neural network training scheme to find the signs), and there is probably significant room for improvement in this new algorithmic space (just as there was for both LTH and supermasks).

This is novel and interesting, and I defer to the reviewers who find it worthy of acceptance. I have reservations about the eventual significance of the work, but that determination will be made by future researchers.
Decision: Accept (Poster)