Signing the Supermask: Keep, Hide, Invert
Authors: Nils Koster
Oliver Grothe
Achim Rettinger
Venue: ICLR 2022 Poster (2022)
Keywords: Neural Networks
Supermask
Lottery Ticket Hypothesis
Pruning
Weight Initialization
Interpretation
Subnetworks

Abstract: The exponential growth in numbers of parameters of neural networks over the past years has been accompanied by an increase in performance across several fields. However, due to their sheer size, the networks not only became difficult to interpret but also problematic to train and use in real-world applications, since hardware requirements increased accordingly. 
Tackling both issues, we present a novel approach that either drops a neural network's initial weights or inverts their respective sign. 
Put simply, a network is trained by weight selection and inversion without changing their absolute values.
Our contribution extends previous work on masking by additionally sign-inverting the initial weights and follows the findings of the Lottery Ticket Hypothesis.
Through this extension and adaptations of initialization methods, we achieve a pruning rate of up to 99%, while still matching or exceeding the performance of various baseline and previous models.
Our approach has two main advantages.
First, and most notable, signed Supermask models drastically simplify a model's structure, while still performing well on given tasks.
Second, by reducing the neural network to its very foundation, we gain insights into which weights matter for performance. 
The code is available on GitHub.

Published as a conference paper at ICLR 2022
SIGNING THE SUPERMASK: KEEP, HIDE, INVERT
Nils Koster
Oliver Grothe
Department of Applied Econometrics
Department of Analytics and Statistics Karlsruhe Institute of Technology oliver, grotheakttkeooo ki kit. edu
Karlsruhe Institute of Technology
nils. kosterkkit.....k .du
Achim Rettinger
Department of Computational Linguistics and Digital Humanities
Trier University
bettingereuni-triee.
ABSTRACT
The exponential growth in numbers of parameters of neural networks over the past years has been accompanied by an increase in performance across several fields. However, due to their sheer size, the networks not only became difficult to interpret but also problematic to train and use in real-world applications, since hardware requirements increased accordingly. Tackling both issues, we present novel approach that either drops neural network's initial weights inverts their respective sign. Put simply. a network is trained by weight selection and inversion without changing their absolute values. Our contribution extends previous work on masking by additionally sign-inverting the initial weights and follows the findings of the Lottery Ticket Hypothesis. Through this extension and adaptations of initialization methods, we achieve a pruning rate of up to 99%, while still matching or exceeding the performance of various baseline and previous models. Our approach has two main advantages. First, and most notable, signed Supermask models drastically simplify a model's structure, while still performing well on given tasks. Second, by reducing the neural network to its very foundation, we gain insights into which weights matter for performance. The code available here.
INTRODUCTION
In recent years neural networks have established themselves versatile problem solvers far beyond the field of machine learning. However, the performance increases achieved are obtained by an even larger growth their size.This This not only leads to growing energy costs (Strubell et al., 2020), but also complicates interpretability considerably. There are different approaches to reduce various aspects of complexity within a neural network, which broadly can be categorized either as pruning, i.c., the technique of trimming down neural networks, or low-precision training, i.e., representing the weights by low precision numbers. Pruning has shown to significantly reduce the size of a neural network (in terms of "active" weights) without affecting performance considerably, once the training phase has ended. Compared to the original network, the pruned network is sparse, since many of the original, "useless" weights are to zero. In contrast, replacing weights by low precision numbers, reduces memory size, but does not reveal a sparse, likely easier to interpret network structure.
Many approaches of pruning (e.g., Janowsky (1989); LeCun al. (1990); Han et al. (2015); Li et al. (2016b); Molchanov et al. (2019) among others) exist to identify "uscless" weights. This raises the question of why not train a smaller network from scratch, since the training of large network the computationally most expensive step and pruning is another additional step. Frankle & Carbin (2018) present a possible solution to this question by formulating the Lottery Ticket Hypothesis (LTH), which states that any densely connected neural network contains smaller subnetworks that, when trained isolation, perform just as well or even better than the original network.
They able identify such subnetworks (winning tickets) by iteratively pruning the network based on the magnitude of each weight. For deeper architectures, Frankle et al. (2019) show, that pruning after few epochs of normal training works significantly better over pruning at initialization.

---

Published as a conference paper at ICLR 2022
Zhou et al. (2019) follow the seminal idea of the LTH and are able to train neural networks by only selecting untrained weights (i.c., weights are frozen after initialization). a concept they call Supermasks. In other words, they find a smaller subnetwork during training without adjusting the weights themselves. Although this approach did not match the performance of their baselines and the pruning rate is inconsistent, it revealed startling insight: weight values do not seem to be as important as the connection itself; a single, well-initialized value for each layer is sufficient. Ramanujan al. (2020) further develop Supermasks by modifying the way masks are calculated, which leads to significant performance improvements compared to Zhou et al. (2019). However, the number of parameters is still high. Recently, Chijiwa et al. (2021) modified the approach of Ramanujan et al. (2020) by randomizing the scores, but prune their networks only up to 60%.
In this paper, we propose a technique called signed Supermask, a natural extension of Zhou et al. (2019) and Ramanujan et al. (2020). We not only determine the importance of a weight by masking, but also learn the respective sign of a weight. Signed Supermasks aim at very sparse structures and are able to uncover subnetworks with far fewer parameters in the range of 0.5% to 4% of the original network requiring little additional computational effort without sacrificing performance.
This differs substantially from low precision training. In its most extreme form, low-precision training quantizes the weight values of a neural network to three constants, including zero or two constants, excluding zero. There, binarized neural networks (BNN) (Courbariaux et al., 2015) reduce complexity but not the size of the networks in terms of sparsity of weights. Ternryy neural networks (TNN), introduced by Li et al. (2016a) allow in principle for sparse subnetworks, however this literature focuses almost exclusively on optimizing computational costs while maintaining predictive performance. For TNNs to reduce computational complexity, sparsity (and interpretability) is of no importance, as they are focused on reducing the computational footprint only. The literature presents diverse approaches, for example Shayer al. (2017) utilize the local reparameterization trick (Kingma et al., 2015) to learn teraaiized weights stochastically. Zhu et al. (2016) ternarize the weights but scale them layer-wise by two learned real-valued scalars and Alemdar et al. (2016) employ a teacher-student approach. Deng & Zhang (2020) train their networks normally with an additional regularization term in the loss function and only ternarize at the end of training by rounding. To the best of our knowledge, this is the only work TNNs also reporting on sparsity. More recently, Diffenderfer & Kailkhura (2021) combined the edge-popup algorithm by Ramanujan et al. (2020) and the binarization of weights and achieve good results.
Thus, although TNNs and Supermasks in general appear similar at first glance, their goals are different: while the former attempt to reduce computational complexity, the latter attempt to find the smallest possible subnetworks within neural network to better understand neural networks in general and work towards facilitating interpretability.
This paper takes the Supermasks perspective and our experiments show, that signing the Supermask matches or outperforms baselines and state-of-the-art approaches on Supermasks and leads to very sparse representations. A convenient side effect of signed Supermask is the ternarization of weights, with implied reduced memory requirements (Li et al., 2016a) and further significant speedup in inference (Hidayetoglu et al., 2020; Brasoveanu et al., 2020). In summary, signed Supermasks provide two major advantages. First, the network structure is simplified while maintaining or improving the performance compared to their dense counterpart. Second, the reduction facilitates a better understanding of the inner mechanics of neural networks. Based on that, we might be able to build smaller but equally powerful models a priori. Additionally, once trained, signed Supermask models can be stored more efficiently and have the potential for faster inference.
SIGNED SUPERMASK
The idea of signed Supermask simple, but elegant: the network may not only switch off a given weight, but may also flip its sign if deemed beneficial. We do this by multiplying each weight with mask m E {-1, 0, 1}. in contrast to m E {0, 1} as in previous work (Zhou et al., 2019; Ramanujan et al., 2020).
For example, if single weight is initialized with negative sign but the performance of the network would increase if it was positive, the signed Supermask has the additional degree of freedom to flip the weight's 's sign. This step decreases the impact of random initialization, which matters specifically for signed constant initialization, as introduced by Zhou et al. (2019) in the context of Supermasks.
2

---

Published as a conference paper at ICLR 2022
Despite including -1 in m, have no additional information to store for the final sparse weight matrices (i.e., the final mask multiplied with the weight matrix), since already includes the weight value, the sign and the zero. In this regard, the additional flexibility in the training phase comes for free.
Note that signed Supermasks are applicable any neural network architecture. By masking, we neither alter the information flow, nor any architecture specific properties. For the sake of brevity, however, we focus in the following on the application on feed-forward neural networks only.
The general setup is as follows: Let Wi E Ru denote the frozen weight matrix of layer I E [1, ......, }.} of feed-forward neural network with L layers and ny denoting the number of neurons in layer I. We initialize Wi ~ Dw. Let further Mi E R= MM ~ Dn be learnable, real-valued matrix that transformed to values {-1, 0, 13 by a element-wise function g, where
1,
M S Tp
(MM
0,
Tn < M Tp
1, M 2 Tp
for some thresholds Tn and Tp as well as appropriate distributions D.., and D,,. that will be specified below. We now define the signed Supermask as M := -=g(M) i.e., a Supermask as in Zhou et al. (2019) with the additional degree of freedom of negative sign. Further note that, compared to Zhou et al. (2019), the function g(-) is deterministic. In order to make full use of the signed Supermask, we utilize the ELU (Exponential Linear Unit) activation function (Clevert et al., 2015) with hyperparameter a. Hereafter, we always set a --=,, Then, with @ symbolizing the Hadamard product, each layer I's output o with 2 = (Wi O M,)" 0 1 is computed follows:
Z1: - 21 <0 0
21 >0
The parameters Tn.p are hyperparameters that need to be tuned: the larger they are in absolute value, the more weights are pruned at initialization and the larger values in M need to grow during the course of training to "activate" its responding weight. The values of Tn.p might differ depending on the task and network architecture at hand. The gradient is estimated using the straight-through estimator (Hinton, 2012; Bengio al., 2013) as in previous work (Zhou et al., 2019; Ramanujan et al., 2020). That is, in the backward-pas pass, we regard the threshold function g(-) as the identity function. Thus, the gradient is going "straight through" 9(-).
Zhou et al. (2019) generate their masks by sampling the entries from Bernoulli distributions with learnable Bernoulli probabilities. Their M therefore resembles the probability, that weight way in some layer! 1 is active. Sampling from a Bernoulli distribution can also be interpreted as a stochastic threshold function. Our fixed threshold approach follows the same intuition with the difference of not using stochastic sampling method. That is, we define two thresholds Tn and Tp after the initialization phase which remain fixed during the course of training in contrast to Ramanujan et al. (2020) who update Tn and Tp after each epoch such that the pruning rate stays constant. Having fixed thresholds has three main advantages which are in line with the argumentation of Zhou et al. (2019): first, the network can choose the optimal number of remaining weights. Second, it can choose the optimal distribution of I's and - Third, this approach is fast, as we only have to compute тn and Tp once. Moreover, this method, with some modifications, is frequently used in the TNN literature. which underlines its legitimacy. As a result, an exact pruning rate cannot be determined a priori, but learned. However, we show in Appendix F, that the pruning rate can be indirectly influenced by adjusting other hyperparameters.
2.1 THOUGHTS ON INITIALIZATION
Signed Supermasks do not alter the value of any weight during training. Therefore, it especially important to initialize weights appropriately. Common weight initialization methods such as He (He et al., 2015) or Xavier (Glorot & Bengio, 2010) neither take masking into account nor the ELU activation function.
Weights We propose a weight initialization scheme that considers both aforementioned alterations, namely, the signed masking process and differing activation function. In simple terms, since each
3

---

Published as a conference paper at ICLR 2022
layer holds fewer weights, scale the weight value to counteract masking. Considering that the goal of initialization methods is to keep the variance in the forward- and backward-pase constant, we suggest to use the following variance:
1.5
Var [i]] nu(1 - ())
(3)
with fan-out ny and layer-wise equal, initial probabilities pà = P(M =0) for layer E {1..... L} and Vi E ,,..., R4: assumptions can be found in Appendix A. The result is still valid without masking, i.e., P6 = 0. He et al. (2015) come to similar conclusion for their Parametric ReLU (PReLU) activation function. Henceforth, we refer to this initialization method as ELUS (ELU Scaled) with masking or simply ELU in case of no masking. Notice that ELUS is simply scaled version of He initialization (He et al., 2015), where the scalar is only dependent on the hyperparameter a of the ELU activation function which is known before training (and set to I in this work).
1..... 74-1- A detailed formal motivation of Equation under specified
Mask Our assumptions and observations regarding the initialization of the real-valued mask M are as follows: We assume that Dm, symmetric around zero, a standard assumption for common initialization distributions. Given the properties of M, M and the gradient straight-through estimator, advantageous the values in M and W are roughly of the same magnitude, such that the gradient, which is independent of M, is neither too small nor too large. In other words, Dm should be chosen such that symmetrical around zero and has similar variance as DD.- In the case of mask initialization, we advice against signed constant approach but recommend a normal or uniform distribution instead.
EMPIRICAL ASSESSMENT
The following paragraphs present the experiments conducted in order to assess the validity and effectiveness of fixed threshold signed Supermasks. The code for the experiments can be found here.
Experimental Sctup et al. (2019) and Frankle & Carbin (2018) i.e., models FCN, Conv2, Conv4 and Conv6 (VGG-like architectures (Simonyan & Zisserman, 2014)) with the additional Conv8 model of Ramanujan et al. (2020) without batch normalization (loffe & Szegedy, 2015) and dropout (Srivastava et al., 2014). Since want to study the fumctionality of signed Supermasks in particular, any extra functionalities would disturb from the essential topic even though it might bump performance. Details about the aforementioned model architectures can be found in Table 7 in Appendix B. Furthermore, we study signed Supermasks on ResNet20, ResNet56 and ResNetl 10 (He et al., 2016) with the exception of replacing ReLU with ELU (similar to Shah et al. (2016)). The batch normalization layers in the ResNet models are frozen for signed Supermask training and are only used because its standard practice. For all experiments the same architectures as the respective signed Supermask models but trained conventionally, act as baselines.
To ensure comparability, we utilize the same experimental setup as Zhou
The fully-connected (FCN) model is trained on MNIST (LeCun et al., 2010) and all CNNs plus ResNer20 on 100 (Krizhevsky, 2009). We only apply per image standardization on both of those datasets. ResNet56 and ResNetl 10 are trained on CIFAR- (Krizhevsky, 2009), where we apply per image standardization, pad the image with 4 pixels on each side and randomly crop a 32x32 image out of the original or its horizontally flipped counterpart.
We ran each experiment 50 times (ResNet56/Resset) 10: 3 runs) with the same setup but different weights and masks drawn from the respectively same specified distribution. This ensures that our experimental results are not affected by the randomness of the initialization step. To obtain an intuition of the scale of memory reduction, we provide a simple metric by comparing the average network size in bytes stored in numpy arrays and scipy's psr_matrix sparse matrices after training. For matrices with more than 2 dimensions, we reshape the matrix to two dimensions. This does not give an exact result but provides good heuristic. Since we attempt to reduce complexity of neural network, we also provide the average time needed to train model for single epoch to further assess practicability.
'The choice of as is arbitrary. The purpose of this metric is not to show how we can most efficiently store the networks but give an intuition added value.
4

---

Published as a conference paper at ICLR 2022
Table I: FCN: Test accuracy and training time. We report the mean, 5% and 95% quantiles of 50 runs for each initialization method well as the results of Zhou et al. (2019) and Ramanujan et al. (2020). Mean training time per epoch reported in the last column.
Init
Test Accuracy [%]
Rem. Weights [%]
TT Epoch [s]
He
97.40 [97.3, 97.5]
100
1.23
Baseline
ELU
97.42 [97.3, 97.5]
100
1.21
Xavier
97.43 [97.4, 97.5]
100
1.21
He
97.12 [97.0, 97.3]
4.93 [4.9,5.0]
1.36
Signed Supermask
ELUS
97.48 [97.3, 97.7]
3.77
[3.7.3.8]
1.27
Xavier
96.89 [96.8, 97.0]
5.51
[5.5.5.6]
1.32
Thou et al.
Xavier (S.C.)
98.0
11 93
Initialization To initialize the weights of the signed Supermask models, we use signed constants in combination with Xavier, He and ELUS. Thus, the respective weight value has a 50% probability of being negative, where the absolute value of the constant is influenced by the respective initialization method. We initialize the baseline models with the same method within a uniform distribution as the respective signed Supermask model. The idea of using only one (well-chosen) signed constant is particularly compelling from the perspective of complexity reduction. For the ELUS initialization, we scale a He initialization by /3 for FCN and Conv architectures and VI.5 for our ResNets, which returned best results in the preliminary testing phase. All masks initialized with Xavier uniform initialization. Results experiments with different mask initializations and distributions can be found in Appendix C.
3.1 FULLY-CONNECTED NETWORKS
This section evaluates the performance of the fully-connected neural network (FCN) signed Super- mask model. Hyperparameters used for training the FCN are listed in Tables 8 and 10 in Appendix B. Note that compared to Zhou al. (2019), the parameter choices are in line with standard practice. Table summarizes the average test accuracy, remaining weights and training time per epoch with the respective 5% and 95% quantiles for both baseline and signed Supermask models with the different weight initializations. All models achieve similar test accuracy. Best performing is the ELUS signed Supermask model. Furthermore, signed Supermask models have wider confidence interval, indicating higher variance. In contrast, confidence intervals for the ratio of remaining weights are very small, demonstrating constant pruning rates throughout all runs. Furthermore, ELUS' ratio of remaining weights of 3.77% is considerably lower than for the other two initializations. This indicates superior initialization as the network is able to achieve a higher performance with fewer weights remaining. For another angle of observation, Figure 5 in Appendix C visualizes the average ratio of remaining weights over the course of training for each weight initialization. Regarding efficiency, ELUS signed Supermask models require about 5% more training time. However, the obtained compression rate for the ELUS model after training is 92.01% % on average, which makes up for the additional training effort. Although the performance of the ELUS signed Supermask is trailing 0.5pp behind the FCN model of Zhou et al. (2019), we argue that considering the pruning rate (they state the pruning rate only imprecisely, ranging from 7% 89%) and performance, signed Supermasks are advantageous.
To assess how the model interprets given dataset, we examine the first masking layer of the network, visualized in Figure! 1. The top image represents input vector (i.e., a flattened MNIST sample), with each vertical line representing one pixel. The bottom shows the mask of the first layer of randomly selected trained ELUS FCN model. The signed Supermask has learned which of the input nodes are more important than others. Those that match a background pixel are completely masked, while the input nodes that receive actual information are allowed to let the information pass through. Thus, the first layer acts as a filter and feature selector. We further analyze the behavior of the masks and draw comparisons to binary Supermasks in Appendix C.
The results so far show the potential signed Supermasks in relatively small architectures. Not only does the ELUS model outperform the respective baselines with only 3.8% of the original weights,
5

---

Published as a conference paper at ICLR 2022
Figure 1: FCN Signed Supermask: The upper figure shows randomly chosen (in this case the digit "8") flattened MNIST sample. Each line represents a single pixel where non-yellow lines represent the pixels containing a part of the handwritten digit. The lower figure visualizes the first mask of randomly picked and trained ELUS FCN model. We can clearly see the alignment of the "digit-pixels" in the MNIST sample and the non-zero mask elements.
also saves 92% of memory once trained. Furthermore, we gain insights into layer interpretability as pruning is not evenly distributed across the network.
3.2 CONVOLUTIONAL NETWORKS
Here, we present the main experimental results of convolutional neural networks (CNN) signed Supermasks. Overfitting was major issue in the development, especially for the baseline models. To counter that, the baseline models are only trained for 50 epochs. In contrast, signed Supermask models were Conv2, we reduced its learning rate compared to the other CNN models. Apart from those alterations, the hyperparameter choices are the same throughout all CNN architectures. All models are optimized with SGD with weight decay. Learning rate for Conv models are exponentially reduced over the course of training. Tables 8 and 10 in Appendix B summarize all training parameters. We have found in initial experiments that =0001 is a solid choice for the threshold parameters тn and Tp- This corresponds an initial overall pruning rate about 11% to 30%.Thu larger layers initially experience a slightly higher pruning rate than smaller layers, following the intuition that weights in smaller layers impact the output of the network to a greater extent than weights in larger layers. Table 2 presents test accuracy for our basclines, the models of Zhou al. (2019) and Ramanujan et al. (2020) and signed Supermasks as well as remaining weights (i.e. pruning rate) for the latter three. For broader comparison to literature, see Table 14 in Appendix C. While we able to reproduce the results of Zhou et al. (2019), this was not the case for Ramanujan et al. (2020) which not only delivered worse results than reported but also demanded a multiple of the training time. The interested reader can find additional results, experiments on different weight and initialization methods as well as investigations on extreme pruning rates Appendices C D. and F. First, apart from Conv2, all CNN models significantly outperform their respective basclines. Conv2 reaches similar accuracy to its baseline, the difference only being 0.4pp. The more our models grow in depth, the greater the benefit of signed Supermasks over the baselines. When considering the ratio of remaining weights, likely reason for the slight performance glitch of Conv2 becomes apparent: it utilizes only 0.6% of the original weights. In other words, Conv2 reaches similar performance with 99,4% of the original weights unused. Conv4, the smallest model in terms of absolute parameter count, keeps the highest percentage of weights in relation to the original count. As the networks grow in size and depth, can once again note an increase the pruning rate. Except for Conv6, there is hardly any variance at all in this metric, indicating robust behavior.
prone to overfitting. However, once we anticipated overfitting tendencies of

---

Published as a conference paper at ICLR 2022
Table Conv: Test accuracy and remaining weights. We report the mean, 5% and 95% quantiles of 50 runs for the ELUS signed Supermasks and baselines. The respective best results of Zhou al. (2019), Ramanujan al. (2020) (abbreviated "Ram") and Diffenderfer & Kailkhura (2021) (abbreviated "Diff"" are shown well. Evidently, signed Supermasks outperform the previous literature on Supermasks on CNN architectures, taking our extreme pruning rates of at least 97% into account. Apart from Conv2, we also gain higher performance than the baseline models.
Accuracy [%]
Rem. Ram.
Weights [%]]
Baseline
Zhou
Ram.
Diff.
Sig. Supermask
Zhou
DifI.
Sig. Supermask
Conv2
68.79 (68.4, 69.2]
66.0
65
70
68.37 [67.7_ 69.0]
11-93
10
10
0.60 (0.58. 0.62]
Conv4
74.50 [74.7, 75.3]
72.5
74
79
77.40 [76.7, 78.3]
11 11-93
10
10
2.91 [2.9, 3.0]
Conv6
75.91 [75.4, 76.4]
76.5
77
82
79.17 [78.1, 80.5]
11
10
10
2.36 [1.9, 2.6]
Conv8
76.24 (75.1. 77.2)
80.91 (79.9. 81.7]
10
1.17 [121
Table Conv Signed Supermask: Additional training time (abbreviated "TT" below) and compres- sion rate compared to the respective baselines. Signed Supermask CNN models train roughly 6 to 10% longer, however, once trained, their required memory can be reduced by least 3....
Add. TT/Epoch [%]
Compression Rate [%]
Conv2
10.14
98.41
Conv4
6.02
93.82
Conv6
6.25
95.07
Conv8
6.00
97.6
In terms of efficiency, we see in Table that the signed Supermask models require 6% to 10% more training time, which can be accounted to the additional calculations necessary. However, once the networks are trained the required memory can, on average, be compressed by 94% 98%. Signed Supermasks outperform not only their respective baselines but also the methods proposed in the related work. Compared to Zhou et al. (2019), our Conv 2, Couv4 and Conv 6 models achieve a higher performance of at least 2pp. Diffenderfer & Kailkhura (2021) do not report performance over a 90% pruning rate, however, their models achieve higher accuracy than ours. We account this to their lower pruning rate. Ramanujan et al. (2020) do report model performance below pruning rate of 90% as well. Indeed, in this case, all our Conv models outperform their equivalent models by 3 to 10 pp, while uncovering much smaller subnetworks.
Creri
Ceevi
Crevi
Cunri
L200
1200
(a) Average ELUS mask distribution. We can observe, that sparsity is very high in very wide layers, indicating that only few those weights are needed.
(b) Equality of all 50 final ELUS masks. Masks show high element-wise equality where sparsity is highest.
Figure 2: Conv Signed Supermask: Mask distribution and equality.
Figure 2a helps to interpret the masks by visualizing the average layer-wise mask distribution of each architecture. can be clearly seen that the first layers are least affected by pruning, while all other layers reach significantly higher pruning rate of over 99%. As the layer size increases, the pruning rate also increases and peaks in the largest layer of the network. The output layers are also relatively sparse, again indicating proper initialization. This finding confirms previous assumption that
7

---

Published as a conference paper at ICLR 2022
larger layers contain far fewer weights that are essential to the network. The relative element equality of the final masks over all 50 runs for all layers and architectures is shown in Figure 2b. This provides an intuition about the similarity of the masks and therefore subnetworks through different runs. We observe that pairwise equality is strongly correlated with the size and sparsity of a layer: the larger, the sparser, the more alike. To some degree this is expected as very sparse layers alike by chance. However, measuring similarity in matrices is very difficult task (for an overview on this literature from a graph-based perspective see, e.g. Emmert-Streib et al. (2016)), which is why we provide visualizations in Figure 9 in Appendix C that emphasize especially the similarity in structure. This shows, that signed Supermasks repeatedly find strikingly similar subnetworks for different runs, meaning that it might be possible to identify certain neurons that are more important than others and hence better understand neural networks general.
3.3 RESIDUAL NETWORKS
To further investigate the effectiveness of signed Supermasks, we turn our attention towards Residual networks which allow for deeper architectures. The investigated ResNet consist of 9,27,54 residual blocks, each in equal parts with 16, 32 and 64 filters, respectively. This sums up to comparatively small parameter count of roughly 0.25M, 0.84M and 1.7M. As hypothesized above, signed Super- masks work better in deeper architectures that contain enough parameters to find a good subnetwork. To investigate this, we multiply the number of filters by 1.5, 2, 2.5 and 3 to create wider variants of ResNet20. Note that even the largest of those variants is smaller than any of the studied VGG-like ar- chitectures above, suggesting less overparameterizaaton, especially for the more complex CIFAR- dataset. Hyperparameters for baseline and signed Supermask models are depicted in Tables and 11 in Appendix B. Compared to the other investigated architectures, we only reduce the learning rate once the train loss hits a plateau. Furthermore, since there is no large deviation between layer sizes, we set the threshold parameters Tn and Tp such that only 25% of the weights are left for ResNet20 and 15% for ResNet56 and ResNet1
Table 4 shows the test accuracy and remaining weights of both baselines and signed Supermasks trained on CIFAR- 10. Notably, signed Supermask always trail 2 to 3 pp behind their respective baseline. However, once the signed Supermask model gets wide enough it can reach the performance of the original ResNet20 with only a third of the parameter count. This finding is partially in line with Ramanujan et al. (2020) and their scaling of ResNet50. As expected, the pruning rate increases as the network grows in depth. We suspect that weight count could be further dropped in the deep architectures by adapting the weight decay parameter, for the sake of clarity however, we decided to keep all learning parameters the same. A possible reason for the slightly lower performance of signed Supermasks might be the frozen batch normalization layers, which would benefit learning greatly. We investigate this aspect in Section 3.3.1. The performance and remaining weights of ResNet56 and
Table 4: ResNet20: Test accuracy and remaining weights on CIFAR-10. We report the mean, 5% and 95% quantiles for the ELUS signed Supermasks and ELU baselines. ResNet20 signed Supermasks are not able reach the performance of the baselines. However, the wider the ResNet20s become, the closer they get to the respective baseline and the higher the pruning rate. Furthermore, with multiplier of 2.5 and 3, the signed Supermasks reach the performance of the original baseline with a third of the original weights.
Accuracy [%]
Rem. Weights
Baseline
Sig. Supermask
Baseline
Sig. Supermask
ResNet20
84.91 [84.39, 85.56]
81.68 [81.03, 82.62]
248624
53530 (21.13%)
ResNet20x1.55
86. 18 [85.59, 86.74]
83.76 [83.01, 84.31]
558600
64223 (11.93%)
ResNer20x2.0
86.80 [86.39, 87.21]
84.42 [83.67, 85.26]
992352
77280 (7.69%)
2esNet20x2.5
87.08 [86.72, 87.42]
84.71 [84.26, 85.32]
1549880
84446 (5.42%)
ResNet20x3.0
87.32 (87.00, 87.67]
84.89 [84.36, 85.48)
2231184
91798 (4.06%)
ResNet1 10 on CIFAR-100 depicted in Table 5. Signed Supermasks show, that they successfully learn from the dataset, even in deep and relatively small environments, However, they cannot reach the same test accuracy their respective baseline. This can be partially explained by the relatively
8

---

Published as a conference paper at ICLR 2022
small original network sizes. Furthermore, hypothesized above, batch normalization is of great importance especially in deep architectures like ResNet110, which thereby particularly suffered.
Table 5: ResNet56/1 10: Test accuracy and remaining weights on CIFAR-100. We report the mean, estimated 5% and 95% quantiles for the ELUS signed Supermasks and ELU baselines as well as the utilized weights. Signed Supermask models trail behind their bascline, however, utilizing maximum of 29% of the original weights.
Accuracy [%]
Rem. Weights
Baseline
Sig. Supermask
Baseline
Sig. Supermask
ResNet56
68.04 167.84, 68. 68.24]
60.01 [59.51, 60.52]
834992 1668272
245116 (29.39%)
ResNetl 10
62.70 [54.80, 68.61]
46.42 [46.31, 46.51]
346757 (20.64%)
3.3.1 THE ROLE OF BATCH NORMALIZATION
ResNets excel compared conventional CNNs because of their depth. A big role in being able to learn very deep architectures is batch normalization. Here, we want to study the influence of batch normalization on the ResNet signed Supermask architectures. For those experiments, we explicitly turn on the batch normalization layers in the respective architecture. This parts a bit from the minimalist idea of Supermasks in general, however, weights are still frozen initialization and this step might be necessary for deep architectures.
Table reports the results of the baseline models (as displayed above) and the signed Supermask with learnable batch normalization layers. Compared to the pure signed Supermask models, batch normalization helps to push performance while increasing pruning rate for all models. However, they unable to reach the performance of the baselines. Nevertheless, taking the remaining weights into account, the predictive performance impresses. More research is needed to investigate the performance gap. Besides this, the results underline that batch normalization helps signed Supermasks to gain better performance in very deep architectures while utilizing fewer weights.
Table 6: ResNet BN: Test accuracy and remaining weights on CIFAR-10 for ResNet 20 and CIFAR- 100 for ResNets 56 and 1 110. We report the mean, estimated 5% and 95% quantiles for the ELUS signed Supermasks and ELU baselines well as the utilized weights.
Accuracy [%]
Rem. Weights
Baseline
Sig. Supermask BN
Baseline
Sig. Supermask BN
ResNet20 BN
84.91 [84.39, 85.56]
83.38 [82.87, 83.92]
248624
37992 (15.54%)
ResNet56 BN
68.04 [67.84, (6,,84,6 68. 68.24]
64.24 (63.90, 64.58]
834992
231769 (27.59%)
ResNetl 10 BN
62.70 [54.80, 68.61]
53.67 [53.63, 53.72]
1668272
64663 (3.86%)
SUMMARY
In this work, we introduced the concept of signed Supermasks. Extending the original work on Supermasks (Zhou et al., 2019). we include the sign in the masking process and utilize a simple step-function to calculate the effective masks. Additionally, we gave theoretical and experimental justification for an adapted weight initialization scheme to the masking process, called ELUS. Our experiments on MNIST and CIFAR-IO 10 show, that signed Supermask models at best hold as little as 0.6% of the original weights, while performing our baselines and current literature (Zhou et al., 2019; Ramanujan et al., 2020). As a side effect, the trained signed Supermask models' memory footprint can be compressed by up to 98%. Further analyses on CIFAR- 100 with ResNet56 and ResNetl 10 suggest that signed Supermasks are generally able to work on complex datasets and architectures as well.
equal or superior level compared to

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
The authors cannot envision any socioeconomic or ethical downsides that are specific to signed Supermasks but neural nets machine learning in general. On the contrary, can see obvious benefits, since signed Supermasks facilitate a drastic reduction of energy consumption when deploying neural and they can contribute to the interpretability, explainability and ultimately trustworthiness of deep learning approaches.
REPRODUCIBILITY STATEMENT
The authors provide the complete code used to conduct all experiments in the supplementary material. An exemplary configuration and requirements file are attached as well, with which all experiments be easily reproduced. Importantly, all experiments were reproduction and strengthens the overall benefits of signed Supermasks. The code is written in Python, mainly using tensorflow and numpy. All datasets used in this work are provided by tensorflow here.
single GPU, which simplifies
REFERENCES
Hande Alemdar, Vincent Leroy, Adrien Prost-Boucle, and Frédéric Pétrot. Ternary neural networks for resource-efficient ai applications. 9 2016.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. 8 2013. URL http: /ttp:i//aiivvvv/ia org. /bs/ 1308.3433
Adrian Brasoveanu, Megan Moodie, and Rakshit Agrawal. Textual evidence for the perfunctoriness of independent medical reviews. CEUR Workshop Proceedings, 2657:1 -9, 2020. ISSN 16130073.
doi: 10.1 (145///mnon.ooommn.. onnnnnn..
Daiki Chijiwa, Shin's ya Yamaguchi, Yasutoshi Ida, Kenji Umakoshi, and Tomohiro Inoue. Pruning Randomly Initialized Neural Networks with Iterative Randomization. jun 2021. URL http:
//arxiv. ///aaxii.rrg/aa////22//.oo 09269.
Djórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). 11 2015. URL 07289.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. BinaryConnect: Training Deep Neural Networks with binary weights during propagations. nov 2015. URL http:://axivv..oo
abs/1511. 00363.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1. 2016.
Pau de Jorge, Amartya Sanyal, Harkirat S. Behl, Philip H. S. Torr, Gregory Rogez, and Puneet K. Dokania. Progressive Skeletonization: Trimming more fat from a network initialization. jum 2020. URL hhtpp:/// //ae:///axiiiii wwwg/bss/////.ooooooooo........r 2206.. 09081.
Xiang Deng and Zhongfei Zhang. An embarrassingly simple approach to training ternary weight networks. 11 2020.
James Diffenderfer and Bhavya Kailkhura. Multi- Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network. mar 2021. URL http: // //arxiv. www..org/abbo haagg//b/// /2103.2233.. 09377.
Frank Emmert-Streib. Matthias Dehmer, and Yongtang Shi. Fifty years of graph matching, network aligmment and network comparison. Information Sciences. 346-347:1800197, jun 2016. ISSN 00200255. doi: 100.1006/....2011..00 URL https: httpt:httttnmeeeeie httpp:ss/////nnnnniiiaa www.eier..cm.
retrieve/pii/ 0300000555110000000 6300100
10

---

Published as a conference paper at ICLR 2022
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the Lottery: Making All Tickets Winners. nov 2019. URL
http://wwwwgg/a//////aa/aa/e// 11134.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 3 2018. URL http:://axiv...tom////// org/ haaeg//bbss abs 03635.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. 3 2019. URL http: htwp/aaxivvv...o /aargg/bbbaa. abs 1903.00000 01611. 611.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, JMLR Workshop and Conference Proceedings, pp. 249-256, 2010. URL http: //proceedings mlr. press / glorot10e html.
Song Han, Jeff Pool, John Tran, and William J. Dally, Learning both weights and connections for efficient neural networks. 6 2015. URL http: arxiv org/abs 1506. 02626.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2 2015. 1502. 01 152.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016--ecemm77007771 2016. ISSN 10636919. doi: 10.1009/CPR.22016.000
Mert Hidayetoglu, Carl Pearson, Vikram Sharma Mailthody, Eiman Ebrahimi, Jinjun Xiong, Rakesh Nagi, and Wen Mei Hwu. At-scale sparse deep neural network inference with efficient gpu implementation. 2020 IEEE High Performance Extreme Computing Conference, HPEC 2020, pp. 1-7, 2020. doi: 10.1 (0.1000/HHECC45542200002222000
Geoffrey Hinton. Neural networks for machine learning. coursera, video lectures: Lecture 9c, 2012. URL httph: / http///wwww//..o cs tor.ottttooooooooooooooooo
cwurseraaaaa....o.. lecat_lccuusssss html.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 32nd International Conference on Machine Learning, ICML 2015, 1:448-456, 2015.
Steven A. Janowsky. Pruning versus clipping in neural networks. Physical Review A, 39:6600-6603,, 1989. ISSN 0556-2791. doi: 10.1 0..103/PyyRevvA.3..666 URL httts:://iinke aps org/ doi/10. 1 39. 6600.
Diederik P. Kingma, Tim Salimans, and Max Welling, Variational dropout and the local reparameteri- zation trick. 6 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. Advances in Neural Information Processing Systems 2, pp. 598-605, 1990. URL http: /ttp:://ppppee httpppaaeraaaa//aa nips. cc/ braainrin dwa.nnaaaaaaee.ammmmmaam .ddf.
paper/250-os
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://ynn.llecmm www.///ww..//ed/////ns.sss 2, 2010.
Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. 5 2016a.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. 8 2016b. URL htttth/aaaaiii org/ abs/1608 08710.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse Training via Boosting Pruning Plasticity with Neuroregeneration. jun 2021. URL http: arxiv. org/abs 2206. 10404.
11

---

wttp:/oo....ttttt.teeeeeee Published as a conference paper at ICLR 2022
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature Communications, 9(1):2383, dee 2018. ISSN 2041- 1723. doi: 10. 441467-018 11--0011--3-6--3
URL. hwtp:tttp//eee www. www.:..nttaeee nature. www.aaawe..oeoeeaaeaaee com.atiiclesss..ee...........ome
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. arXiv, pp. 11264-11272, 2019.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Randomly Weighted Neural Network? 2020. URL
Rastegari. What's Hidden
htttps:/gittii com/allenaailaaccaccacccmmamaaae
thecvf. com/ content_cveeeeeee 2020 / haaaaaaaaaaaaaaaaaaaaaaaaaaaaaee Whats Whats_Hidden__________h_ _ pdf.
Randoml Marroomly_Leeggttdd_/eeera ott.pe/eeeeereeeeeeoooorrrrrrrrr
Anish Shah, Eashan Kadam, Hena Shah, Sameer Shinde, and Sandip Shingade. Deep Residual Networks with Exponential Linear Unit. apr 2016. doi: 10.1 10.1145/298340..
2983406. URL sttp://arxta htta//xii..gg/aaaaa//eaaaaaeaaae http/////10...111ttt:tt:////: doi 1145 1145/2983402 2983406.
0ttp::////..ii..//111/110 org/10.
Oran Shayer, Dan Levi, and Ethan Fetaya. Learning discrete weights using the local reparameteriza- tion trick. 10 2017.
Mingzhu Shen, Xianglong Liu, Ruihao Gong, and Kai Han. Balanced Binary Neural Networks with Gated Residual. sep 2019. URL http: http/axxvvvv.///..a/a/a org/abb /alpaabbs///../ 119...119991.1111119...........1 12117.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. sep 2014. URL http:///aaxii wwtp:////gaa///////ee 1556.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting, 2014.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nip. ACI. 2019 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pp. 3645-3650 2020. doi: 10. 1e.11865/1111111333
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in Neural Information Processing Systems, 32, 2019. ISSN 10495258.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. Trained ternary quantization. 12 2016.
12

---

Published as a conference paper at ICLR 2022
A DERIVATION OF ELUS
In the following paragraphs, we revise the initialization methods He et al. (2015) and Glorot & Bengio (2010). This is needed because first, introduce masking and the ELU activation function. Both concepts alter the assumptions of the cited work. Here, we derive a novel adaption of the common He initialization (He et al., 2015).
In this section, we use the following notation:
Activation function (Ir)
Layer has ny neurons. This means that layer takes n4-1 inputs per neuron. In the literature, 74-1 is sometimes called fan-in and ny is sometimes referred to as fan-oul
Wi E | b, E _
Weighted layer imput 2, = Wio + bi, 21 E R
Layer output a E R
L denotes the network's loss function
Vo: 0c
do:
Vz dzc
ac
In order maintain clarity, we will denote the transposed weight matrix of layer W," as
W; in the derivation the backward-
Wi = Wi @ M, This leads to the suboptimal but necessary choice to denote the transpose of Wi Wi (instead of W ) to maintain clarity.
We will further assume, that the following holds:
(I) We use the ELU activation function, introduced by Clevert et al. (2015).
Variances of imput features æ are the same Var
(3) For each layer 1, the respective weight matrix Wi is drawn from independent and symmetric distribution with [[[]] = 0 and Var[Wi] It follows that [[z]] O.
(4) For each layer I, the respective mask matrix M; is drawn from an independent and symmetric distribution with E[M] 0. To make the derivations more tangible, we assume ph
P(M = 0) 0.5 and pi P(M 1) = P(M)) = -1) = 0.25 Vi l....,,,..
1,... 1 at initialization. In general, the variance is Var[M = pb-
(5) For each layer 1, all elements in z₁ and o share the same variance Var[z]] and Varloi] respectively.
Since we use ELU activation functions in our experiments, the question arises whether we can still use the initialization of He al. (2015). Clevert et (2015) do so and achieve good results. Let us therefore calculate the important values that might alter the assumptions necessary.
Like Glorot Bengio (2010) and He et al. (2015) differ in the choice of activation function, we can particularly investigate the parts of the derivation where the expected value of the activation function
used. particular, this is [E((o)-))? ? in the forward-pan ppsss and (V())) in the backward-pass
We calculate E[(()-1) 2] as follows:
E[(o:-1))] = B[(ae a) +|=21
From here on, we drop index / for the sake of clarity. For the calculation of [[[(ee a) we additionally assume that 2; (O, 1), although this is simplification and therefore only approximation. With this, we obtain
[(cc - a)2
(ac² - a)2
1 dz
22T
2
0
(622
2x² + 1).
dz
2m
(5)
a
0
2 dz
2a 0
dz
+
a²
0
c
1/2
dz
2T
/ 2n
22T
(a)
6)
3
13

---

Published as a conference paper at ICLR 2022
We can observe that the last term is half of the area under the standard normal probability density
function. Note that holds that -ax² bx -a(x and (b) as follows
20
+
6²
With this, we can re-write (a)
4a
= a
a²c²
0
(a)
dz
e
-1((=-)) 2/
dz
(6)
2t
2tT
-00
-00
2a²
0
2a²e:
e
(b) 2tT
c-((3-1)8+3 dz
e- -2=1=-1)² dz
V2n
-
We can see that (a) is the probability density function of (2, and (b) is the probability density
function (1, 1), which can be easily evaluated. With (a) = a ee ob ( = 0.168102a² and
(b) 2a²e ( 1 0.523157a have
El(ar= a) 0.1 1022aa - 0..231577 + 0.5a 0. 1.144495a-2 = =k
(8)
If we take the x 0 into account, i.e. [[[²]] Var[z], have
[[(o)-)) = ( k) Var(zz
Let us now move on to calculating E[(Vo:) Note that the derivative of ELU is simply if (which is the same for ReLU) and ae if second case:
0. Let us look at the squared expectation of the
El(a' (=))² Eia²
(10)
we assume again, that 21 ~ (O, 1). we immediately see that we already calculated this expected value in Equation 6. Therefore, we have
B[(o'(z))²] 0.168102a =
(11)
For the total expected value, we then have
B [Vo)) =
(12)
To summarize, in comparison to (He et al., 2015), we only have to scale Equations 9 and 12 with the constants h and k, respectively. The results above are equal Equation 15 in He et al. (2015), where they assume PReLU activation (PReL.U as activation function is also introduced in He et al. (2015)). However, they do not supply a derivation. Therefore we cannot guarantee equality with confidence.
Hence, we are careful with the pronoun "our" (initialization) and we will therefore simply refer to the initialization method that follows from the derivations above "ELU". Let us now use these results and calculate Var
Forward-Pass:
We start with the following equation
Var
Var WarlW, oi
(13)
Because W 0 we can re-write this as
Var[2ı] = n4-1 Var (or)E()
(14)
With Equation it follows that
Var [z]] = 1
Var Wi Var æ
II
1
k)r4-Varr(W)
(15)
2
we want to achieve constant variance throughout the network, we can set
2 + k)m=1 Var Mi]
(16)
- Var
( + 1))m-11111111) -(b)
14

---

Published as a conference paper at ICLR 2022
With Var = -
(--1) + ph 02 + pl 1 0.5 as stated above and we have
Var [i] = 1.5505
(17)
74-1
Note that for normal training, we would not have to account for the variance of the mask, which would result in Var [a]] = 1.5505
Backward-Pass:
We know that
n
Var Vojl
Σ
Var Wi 1j
(18)
=Va
Recall that ELU derivative is
11,
21 2120 0
('22) (ae"1
(19)
21
We know from Equation 12 that E[(('(21)) =
+ h and [[oo++]] = 0. It follows, that
Var[Vor = B[((o4+ resulting in
Var [ar] = ( + Var
(20)
With the obtained results we can now re-write Equation 18 as
1 2 + h)n, Var Wir 2
Var[Vo,]
(21)
Herewith we are able to calculate the variance throughout all layers as
Voi] = VarlVor+
II
h)m [ar[W]]
(22)
2
-=1
Like in the forward-pass, an easy way to keep the variance constant is to set
2 + Var Wi -
(23)
P + h) h)m((1 (p) 1.49678 2
= Var Wi]
mi
As above, if we use this initialization for training a normal neural network, we would use Var [i] = 1.49678 @
We will follow the intuition of He et al. (2015) by leaving the choice of Equation 23 or 17 to the user, as their initialization scheme has proven to outperform Glorot & Bengio (2010) practice and takes the
into account. However, since Equations 23 and 17 are relatively close each other, we suggest to use the following variance for the sake of simplicity
1.5
Var Wi]
(24)
mi(1 - pb)
15

---

Published as a conference paper at ICLR 2022
B MODELS AND HYPERPARAMETERS
Table Neural Network architectures used in the experiments of this work. To be able to compare the results the existing literature, the architectures equal Frankle & Carbin (2018); Zhou et al. (2019) in addition to the Conv8 model used in Ramanujan et al. (2020). Each CNN performs various convolutions (pool denotes max-pooling) with stated number of filters, followed by fully connected (FC) layers with specified output neurons, For the CNNs, we always use a filter size of 3 x 3. Total parameter count for each architecture is reported in the last row.
FCN
Conv2
Conv4
Conv6
Conv8
64, 64, pool
64, 64, pool
128, 128, pool
Conv
64, 64, pool 128, 128, pool
128, 128, pool
256, 256, pool
Layers
64, 64, pool
256, 256, pool
512, 512, pool
FC Layers
300,100,10
256, 256, 10
256, 256, 10
256, 256, 10
256, 256, 10
Parameter Count
266.200
4.300.992
2.425.024
2.261.184
5.275.840
Table Hyperparameter choices for training the baseline models of FCN and Conv2 Conv2-Conv8. For simplicity, we chose parameters that work well on all models
Conv8.
FCN
Conv2
Cony4
Conv6
Conv8
Learning Rate
0.008
0.008
0.008
0.002
Decay Rate Step
.66100
.9655
.6/100
.6/10
.6/100
Red. LR on Plat. (Patience)
Weight Decay
7e-4
7e-4
7e-4
7e-4
3e-4
Momentum
Iterations Optimizer
50
50
50
50
SGD
SGD
SGD
SGD
SGD
Table 9: Hyperparameter choices for training the baseline ResNet models.
ResNet20
ResNet56
ResNet110
Learning Rate
0.15
0.15
Decay Rate / Step
Red. LR on Plat. (Patience)
2 (15)
.2 (15)
(15)
Weight Decay
le-4
le-4
le-4
Momentum
Iterations
90
90
Optimizer
SGD
SGD
SGD
Table 10: Hyperparameter choices for training the signed Supermask models of FCN and Conv2 Conv8. For simplicity, chose parameters that work well on all models, except for Conv2 which suffered overfitting and we therefore reduced the learning rate.
ResNet20
ResNet56
ResNetl 10
Learning Rate
0.2
0.15
0.15
Decay Rate Step
Red. LR Plat. (Patience)
.2 (15)
(15)
.2 (15)
Weight Decay
le-4
le-4
le-5
Momentum
.9
.9
Iterations
110
120
120
Optimizer
SGD
SGD
SGD
Initial Pruning Rate
75%
85%
85%
16

---

Published as a conference paper at ICLR 2022
Table 11: Hyperparameter choices for training the signed Supermask ResNet models.
FCN
Conv2
Conv4
Conv6
Conv8
ResNet20
Learning Rate
0.05
0.02
0.05
0.05
0.05
0.2
Decay Rate / Step
96/10
.6/55
.96/10
96/10
96/10
Red. LR on Plat. (Patience)
.2 (15)
Weight Decay
5e-4
5e-4
5e-4
5e-4
5e-4
le44
Momentum
.9
.9
.9
.9
.9
.9
Iterations
100
100
100
100
100
110
Optimizer
SGD
SGD
SGD
SGD
SGD
SGD
Initial Pruning Rate
6.3%
29.5%
19.3%
11.9%
12.9%
75%
C FURTHER EXPERIMENTAL RESULTS
17

---

Published as a conference paper at ICLR 2022
Table 12: CNN Signed Supermask: Average test accuracy, test loss and required training time per epoch for each model architecture and weight initialization method. The 5% and 95% quantiles are given well.
Architecture
Init
Test Acc. [%]]
Test Loss
Time/Epoch [s]
He
68.79 [68.35, 69.19]
0.9083 [0.8986, 0.9166]
2.87 [2.86, 2.88]
Conv2 Baseline
ELU
68.60 [68.23, 68.99]
0.9189 [0.9047, 0.9331]
2.88 [2.85, 2.97]
Xavier
67.93 [67.39, 68.41]
0.9535 [0.9281, 0.9716]
2.86 [2.84, 2.88]
He ELUS
66.58 (63.92, 67.15]
0.95% [0.9452, 0.9741]
3.16 [3.14, 3.18]
Conv2 Su.
68.37 [67.74, 69.04]
0.9784 [0.9524, 1.0090]
3.16 [3.14, 3.18]
Xavier
54.24 [53.36, 55.39]
1.2862 [1.2608, 1.3067]
3.20 [3.15, 3.31]
He ELU
75.03 [74.71, 75.42]
0.7266 [0.7200, 0.7347]
3,99 [3.97. 4.01]
Conv4 Baseline
75.00 [74.65, 75.29]
0.7281 [0.7213, 0.7363)
3.98 [3.97, 4.00]
Xavier
74.20 [73.68, 74.75)
0.7542 [0.7390, 0.77
3.99 [3.96, 4.01)
He ELUS
75.92 [74.93, 76.78]
0.8084 [0.7695, 0,8438)
4.22 [4.19, 4.26]
Conv4 St. Su.
77.40 [76.73, 78.25]
0.8309 (0.7863, 0.8721]
4.22 [4.20, 4.25)
Xavier
73.99
[73.47, 74.64]
0.7544 [0.7390, 0.7680]
2.51 [4.21, 4.34]
He
75.54 [74.89, 76.03]
0.7159 70.7001, 0.7324]
4.98 [4.92, 5.321
Conv6 Baseline
ELJU
75.91 [75.43, 76.36]
0.7009 [0.6914, 0.7159]
4.84 [4.82, 4.85]
Xavier
75.54 [74.98, 76.13]
0.7151 7007007, 0.7300]
4.94 [4.92, 4.96]
He
78.42 [77.57, 79.27]
0.6544 [0.6306, 0.6791]
5.07 [5.05, 5.10]
Conv6 St.
ELUS
79.17 [78.05, 80.49]
0.6685 [0.6277, 0.7151]
5.14 [5.09, 5.21]
Xavier
78.47 [77.52, 79.23]
0.6724 [0.6397, 0.7252]
5.12 [5.09, 5.20]
He
73.82 [70. 18, 78.29]
0.6830 [0.6399, 0.7440J
6,38 [6.27 6.89]
Conv8 Baxeline
ELU
76.24 [75, 14, 77.18]
0.7787 [0. 7067, 0.8374]
6.27 [6.26 6.29]
Xavier
73.53 [71.82, 74,85]
0.9133 [0.8317, 1.0212)
6.31 [6.27 6.39]
He ELUS
78.92 [78.17] 17, 79.77]
0.6194 [0.5992, 0.6430]
6.67 [6.61 6.74]
Conv8 Si. Su.
80.91 [79.93, 81.71]
0.6057 [0.5769, 0.6511]
6.65 [6.60 6.77]
Xavier
78.51 [76.28, 79.92]
0.6360 [0.5957, 0.7147]
6.68 [6.64, 6.73)
Table 13: ResNet Signed Supermask: Additional training time (abbreviated "TT" below) and compression rate compared to the respective baselines. Signed Supermask require 3 14.5% more training time, which is made up for by the higher compression rate. Unsurprisingly, pruning rate correlates highly with the compression rate.
Add. TT/Epoch [%]
Compression Rate [%]
ResNer20
3.42
57.65
ResNer20x1.5
8.53
76.08
ResNer20x2.0
13.69
84.58
ResNet20x2.5
14.42
89.
ResNer20x3.0
12.57
91.85
18

---

Published as a conference paper at ICLR 2022
Table 14: Comparison with broader literature periphery with regard to the total number of remaining weights on CIFAR-10. We show those results that are also compared with other obtained results in the respective papers. Please note that, as already stated in the introduction, direct comparison of all those methods is not possible for multiple reasons: first, complexity between the final result varies. As an example, pruned networks allow trained weights whereas BNN only allows for binary weights and activations. Second, many works experiment with different networks and the usage of additional layers such as batch normalization dropout. you start with very overparameterized network, probability theory gives the network much higher chances to include better subnetwork than very small one. On top of that, batch normalization or dropout can have a big impact on predictive performance, which might hide the true performance of the proposed method.
The methods we compare our signed Supermasks to, are: Supermasks Zhou et al. (2019), edge- popup Ramanujan et al. (2020), IteRand Chijiwa et al. (2021), MP 1/32 Diffenderfer & Kailkhura (2021), FORCE de Jorge et al. (2020), SET Mocanu et al. (2018) 8), Rigl. Evei et al. (2019), GraNet Liu et al. (2021), TWN Li al. (201 6a), TTQ Zhu al. (2016), SCA Deng & Zhang (2020), LR-Net Shayer et al. (2017), BinaryConnect Courbariaux et al. (2015), BNN Courbariaux et al. (2016) and BBG Shen et al. (2019). Considered models are Conv6 and Conv8, VGG-19, ResNet-20, ResNet-18 and Wide ResNet 22-2. VGG-Small summarizes all small VGG-like architectures, i.e. they can slightly differ. We refer to the respective work for details. The handling of the weights can be categorized as either a binary mask ("Masked (B)"), pruned weights where the weight values change during training, ternarized and binarized weights as well as ternary masks ("Masked (T)") as is the case for our signed Supermasks.
Method
Model
Weights
Acc. [%]]
Init. Weights
Rem. Weights
Supermask
Conv6
Masked (B)
76.5
2.3 M
0.25 M 2.1 M
edge-popup
Conv8
Masked (B)
5.3
IteRand
Conv6 x2
Masked (B)
90
4.6 M
2.3
MPT-1/32
Conv8
Masked (B)
91.48
5.3 M
0.23 M
MPT-1/32 BN
ResNet- 18
Masked (B)
94.8
11.2 M
2.2 M
FORCE
VGG-19
Masked (B)
M
SET
VGG-Small
Pruned
0.47
RigL.
Wide ResNet 22-2
Pruned
94
26.8 M
13.4 M
GraNet
VGG-19
Pruned
93.1
20 M
1
TWN
VGG-Small
Temarized
92.56
5.1 M
5.1 M
TTQ
ResNet-20
Teraaiized
91.13
0,27 M
0.27
SCA
VGG-Small
Ternaiized
93.41
4,9 M
4,9
LR-Net
VGG-Small
Temariiedd
93.26
5.1 M
5.1 M
BinaryConnect
VGG-Small
Binarized
91.7
4,6 M
4.6 M
BNN
VGG-Small
Binarized
88.6
4.6 M
4.6
BBG
ResNet-20
Binarized
0.27 M
0.27 M
Sig. Supermask
Conv8
Masked (T)
80.91
0.061 M
Sig. Supermask BN
ResNet-20
Masked (T)
83.38
0.27 M
0.038 M
19

---

Published as a conference paper at ICLR 2022
Binary Supermaek
(Absolute) Signed Supermask
Figure 3: FCN (Signed) Supermask: The figure seeks draw a rough comparison between FCN trained with (ELUS) binary Supermask and (ELUS) signed Supermask. Note that comparison Supermasks of Zhou al. (2019) is not directly feasible as there are other important factors that differ, namely the ELUS initialization, the ELU activation function and the same configuration as used for the ELUS signed Supermask. However, comparing a binary with signed Supermask can uncover important differences.
20

---

Published as a conference paper at ICLR 2022
The above matrix depicts the sum of 50 trained binary Supermasks, i.c. during the training process the weights could either be pruned or "let through". The same is shown the bottom for the signed Supermask. A darker shade represents a larger value. In order for any negative and positive masking not to cancel each other out, the absolute masks were summed. Put simply, if single mask (out of the 50 runs) prunes a weights every time except once, the respective value in the matrices would be 1 and so on. Both versions have average pruning rate of 97.2% in the first layer. The shown matrices still exhibit pruning rate of 54. 111 % in case of the signed Supermask and 43.63% for the binary Supermask, i.c. 54% (43%) of the weights were always pruned. This number is impressive by itself, if we consider that normal neural networks could never reach similar number. The overall average pruning rates are 3.77% and 4.04% for the signed and binary Supermasks. respectively. While the signed Supermask reached an average performance of 97.48%, the binary Supermask reached an average test accuracy of 97.0 7.03%. In other words, the signed Supermask left the same number of weights in the first layer, while the distribution differed more in the other layers but reached higher performance with roughly 0.3pp less weights. The focus of this examination to compare the properties of the respective first masks.
Figure 3:
At first glance, the binary Supermask looks more erratic, while the signed Supermask inhibits a more ordered structure. The impression is not deceiving: out of the 784 columns (i.e. input neurons), 284 in case of the signed Supermask are completely masked, wheres the binary Supermasks only masks 144 (we strongly assume, that the fact that is, those columns were masked in every considered mask, which shows the robustness of Supermasks in general. Even if we allow 100 outliers (that is, column-wise maximum sum of 100) per column, binary Supermasks only count 260 of those columns (364 for signed Supermasks). We can observe this fact in two parts: the signed Supermask more sparse the edges and the O-columns in the center are wider and less noisy. This leads to the conclusion that, although average having the same pruning rate, signed Supermasks are more consistent (otherwise we would count fewer 0-columns) and more understanding of the given data, as it focuses specific inputs and mostly disregards the image background, which (in the flattened vector) is positioned at the sides. This strongly suggests that not the pruning rate alone is of importance, but also the distribution of pruning per layer. To summarize, although a comparison is not possible without compromise, we can see that the binary Supermask, although exhibiting the same average pruning rate is far less consistent compared to the signed Supermask.
number is almost double the other is a coincidence). That
21

---

Published as a conference paper at ICLR 2022
Baseline
Signed Supermask
98
97
FCN
Accuracy
96
96
95
95
94
0
10
20
30
40
50
20
40
60
100
80
Accuracy [[]
70
Conv2
70
60
50
60
30
50
20
40
100
80
Accuracy [S]]
75
Conv4
60
65
50
0
10
20
30
40
50
20
40
80
80
100
80
80
Accuracy [+]
70
Conv6
70
60
50
60
40
100
80
75
70
Couv8
Accuracy
60
65
50
0
10
20
30
40
50
20
40
60
80
100
Epoch
Epoch
ELU(()
Xavier
Figure 4: Signed Supermask: Average test accuracy over 50 runs of all baseline (left) and signed Supermask (right) models and architectures. We also report the 5% and 95% quantiles. We can observe the larger variance in test accuracy for the signed Supermask models. The ELUS signed Supermask models visually coincides with the baseline or lies above it, whereas He and Xavier cannot reach the same performance. Furthermore, for the deeper models, ELUS needs few epochs to "warm up".
22

---

Published as a conference paper at ICLR 2022
14
He: 4.93%
ELUS: 3.77%
Remaining Weights [%)
12-
Xavier: 5.51%
10
8
6
4
20
40
60
80
100
Epoch
Figure 5: FCN Signed Supermask: Average ratio of remaining weights. We also report the 5% and 95% quantile, although the interval barely visible indicating robust metric. Weight count drops early in the learning phase and plateaus thereafter.
Conv2
Conv4
Remaining Weights (88
4-
4-
2-
20
40
60
80
100
80
100
Conv6
Conv8
5
Remaining Weights (56
100 ()
20
40
100
80
80
100
Epach
Epoch
He
ELUS
Xavier
Figure 6: CNN Signed Supermask: Average ratio of remaining weights over 50 runs of the CNN signed Supermask models initialized with He, Xavier and ELUS and architectures during training. We also report the 5% and 95% quantiles. Besides for the Conv6 model, all confidence intervals are very narrow. Furthermore, all architectures drop the weights heavily during the first few epochs and plateau thereafter.
23

---

Published as a conference paper at ICLR 2022
Hc
ELUS
Xavier
FCN
5
2
0
2
Conv2
55
2
4
0
2
Cony4
2
Conv6
0
2
4
6
8
3
5
7
0
2
4
Conv8
5
0
5 Layer
6
10
1
2
7
8
9
10
0
4 5
9
10
Layer
Layer
Figure 7: Signed Supermask: Average mask distribution over 50 runs of all models and weight initializations. It holds for all configurations that the first layer undergoes only little pruning in contrast the large hidden layers, where almost all weights pruned. The behavior in the last layer differs for ELUS: there, the pruning rate is significantly higher in the last layer as compared to He and Xavier. The sparsity correlates with the size of the layer: the larger layer, the more it is pruned.
24

---

Published as a conference paper at ICLR 2022
He
ELUS
Xavier
FCN
.5
0
2
Conv2
-5
.3
11
4
2
3
Conv4
3
5
2
4
Conv6
4
6
3
5
2
Conv8
-5
.5
0
5
10
2
3
8
9
10
0
4
8
9 10
Layer
Layer
Layer
Pairwise Equality
Pairwise Equality (Abs. Mask)
Figure 8: Signed Supermask: Equality of masks and absolute (i.e. binary) masks for each architecture and weight initialization over the 50 conducted runs. Only looking at the equality of the signed Supermasks, FCN the first layer is the largest and attains the highest pruning rate (refer to Figure 7), whereas the hidden and output layer across all runs in about 50% of all elements. For the CNNs, we have almost no similarity in the first layers which can be partially attributed the nature of CNNs and the concept of weight-sharing. Here, similarity gradually increases as the layer size and pruning rate are increased, peaking at the respective largest layer.
that the picture is divided into FCN and CNN behavior. For the
pruned much. All initializations produce masks that similar
Taking the absolute signed Supermasks into account, do not any meaningful differences in those layers that gain high similarity in the signed Supermask case. However, for the output layers across almost all models that were initialized with He and Xavier we see high similarity. If contrasted with the pruming rate again, its notable that those layers are not pruned much. This can be seen most clearly in the FCNs: He and Xavier only prune the last layer very little, resulting in absolute signed Supermask that contains many I's. The same holds for the CNN output layers and some CNN input layers.
These findings suggest that the 0-elements are the main driver of mask similarity, i.e. the same weights are being pruned in every run.
Note that this metric is more takes a more heuristic-appin approach to analyze the problem. Comparing matrices or tensors is difficult task. Introducing more precise metrics would be far beyond the scope of this paper.
25

---

Published as a conference paper at ICLR 2022
Randorn Draw:
Randen: Draw 2
Coor2 (Laver
Charet (Layer
Coart :Lager 0)
Currs (Logee
Figure 9: Conv Signed Supermask: We show two randomly drawn trained signed Supermasks for each architecture. The chosen layers are the respective first fully connected layer in the respective architecture (which are of different shapes for each architecture). A green dot represents "1", red dot "-1". For each architecture, we notice unique patterns which are present both draws. For example, the Conv8 masks show grid-like pattern which has area with almost completely pruned weights the upper part the mask. The structure is similar for the two Conv6 masks. For Conv4, the masks show Both Conv2 samples are very sparse. The lower third both masks show very similar patterns, while wide horizontal band top is almost completely pruned again.
pruning activity on the left part of the mask and
row-wise structure.
Interpretation the masks, e.g. like Figure very difficult as the layers here are hidden and depend a lot on the previous and following layers. Still, for each architecture the layers remarkably similar which cannot be said for standard neural networks.
26

---

Published as a conference paper at ICLR 2022
25
50
Test Aceuracy [%6]
85
Remaining Weights
80
15-
73
20
100
80
100
Epoch
Epoch
ResNet20
ResNer20x15
ResNe120x20
ResNe120x25
ResNet20x330
Figure 10: Signed Supermask: Mean test accuracy and ratio of remaining weights in addition the respective 5% and 95% quantiles for ResNet20 and its wider siblings.
ResNer20
ResNer20x1.5
Rel. Count (S)
10
12
16
18
20
22
10
12
18
20
22
2es2222
ResNer20x2.5
Rel. Count (%)
10
16
18
20
22
10
14
15
200
22
ResNet20x3.0
Layer
Rel. Count (86]
10
22
Layer
Figure 11: ResNet Signed Supermask: Average mask distribution (over the 50 conducted runs) of the investigated ResNet20 models. The wider the layer, the higher the pruning rate. Layers 8 and 15 are 1x1 shorteut layers. Interestingly, those layers are relatively sparse compared to other hidden layers. As with the CNNs, the first and output layer are most sparse.
27

---

Published as a conference paper at ICLR 2022
D INFLUENCE OF WEIGHT DISTRIBUTION
In this section we analyze the influence of the weight distribution overall performance. Therefore, we train additional models with weights drawn from a uniform distributions with the standard thresholds of He (He et al., 2015). As baseline, we utilize the signed Supermask models with weights as signed constants presented in Section 3.
For both variants, we arbitrarily choose He initialization as we expect similar behavior for the other initialization methods. Hence in this Section, we call the former "He Uniform", the latter "He SC" acting as the baseline. Note that these experiments are not exhaustive, as we only examine the two mentioned distributions. The purpose of this section is to give an intuition on the behavior of different weight distributions. We first analyze the FCN architecture, followed by a summary of the CNN architectures.
FCN Figure 12 displays the average test accuracy and average ratio of remaining weights for both models with the respective 5% and 95%nquansi.es. We can note that the variance of He Uniform is lot higher than the variance of He SC. Moreover, He Uniform performs on average slightly worse than He SC. The final mean result for He SC is 97.12%, whereas He Uniform achieves 96.77%. A Welch's 's t-test for significance yields that He SC significantly (p < 0.01) outperforms He Uniform. The higher variance of He Uniform is also visible in the ratio of remaining weights. On average, He SC and He Uniform achieve almost the same level of sparsity, however the 5%- and 95%-quaniil are almost 2pp apart for He Uniform.
We postulate that a uniform distribution does not have additional value compared to signed constants for a dense neural network in the context of signed Supermask. Apparently, initializing weights with small values deteriorates performance. Recall that neither He SC nor He Uniform are scaled for the use of a signed Supermask. We further speculate that the higher variance is a cause of random initialization such that training cannot nullify the impact of randomness and ill-initialized weights. This leads to the conclusion that the value of a weight is important if not initialized well, as He Uniform holds more weight values compared to He SC and performs worse.
To summarize, using signed constants instead of a uniform distribution provides a much higher level of robustness for the FCN. Our results suggest that for signed Supermask and dense neural networks, a signed constant initialization surpasses the performance of uniform distribution.
CNN We now investigate the influence of weight distribution on our CNN architectures. Figure 13 visualizes the average test accuracy with the respective 5% and 95% quantiles during training for each model. We first note that He Uniform is not behaving consistently. While with Conv2 and Conv8 the variance of He Uniform much higher compared to He SC, it is roughly equal for Conv4 and conv6. The average test accuracy of He Uniform is of the same magnitude as He SC, and we can only register a significantly (p- < 0.01) better performance for Conv2 He SC compared to He Uniform. This is not sufficient to conclude on one distributions significance for the general case. The results indicate that the additional weight values delivered by a uniform distribution do not improve the learning capability of a signed Supermask model. This is in line with the findings of the FCN architecture. It seems sufficient to initialize the weight matrix with a well-chosen constant. We further speculate that due to the higher parameter count of the investigated CNN architectures compared to the FCN, the optimizer is to some extent able to neglect the influence of poorly initialized weights.
Let us consider the average ratio of remaining weights over the course of training for each CNN model. Figure 14 visualizes this metric for each network in addition to the 5% and 95% quantiles. Unlike observed in the case of the FCN, we cannot report a high variance for He Uniform within CNN models. In fact, the weight distribution does not seem to have an impact on the level of sparsity, as the ratios move almost equally over the course of training for each model.
Considering both the performance and level of sparsity, we do not find that the choice between uniform and signed constant distribution impacts the end result accuracy of He Uniform varies a lot more compared to He SC, the findings suggest to prefer signed constants over uniform distribution, signed constants bring positive side-effects such as efficient
average. However, since the test
28

---

Published as a conference paper at ICLR 2022
98
14
Uniform: 4.77%
He SC: 4.93%
12"
Remmaning Weights
94
10
92
He Uniform
SC
88
20
40
60
80
100
0
20
40
60
80
100
Epoch
Epoch
Figure 12: FCN Signed Supermask: Average test accuracy and average remaining weights with their respective 5% and 95%-quantile a FCN with weights drawn from a uniform distribution and as signed constants ("SC""). We can see that the uniform distribution yields in much higher variance of both, test accuracy and ratio of remaining weights. Furthermore, drawing weights from uniform distribution does not improve performance sparsity.
Conv2
Conv4
80
80
75
75
Accuracy (86
70
70
65
65
60
60
55
55
50
50
20
40
80
80
100
0
60
100
Conv6
Conv8
80
75
Accuracy [%)
60
55
50
0
20
40
60
80
100
0
20
40
60
80
100
Epoch
Epoch
He Uniform
He SC
Figure 13: CNN Signed Supermask: Average test accuracy for He Uniform and He SC during training, We also report the respective 5% and 95%-quantile. We find that the variance of He Uniform at least as large as the variance of He SC, but especially models Conv2 and Conv8 He Uniform has much larger variance. He SC outperforms He Uniform in all architectures.
storing and more robust behavior.
To summarize both FCN and CNN behavior, it be stated that uniform distribution is not advantageous over a signed constant approach. We further want to re-emphasize the simplicity, yet good performance of signed constants. They have shown very robust behavior with regards to the
29

---

Published as a conference paper at ICLR 2022
Conv2
Conv4
He Uniform: 1.64%
He Uniform: 2.9%
Remaining Weights (5%
SC: 1.69%
SC: 3.05%
8
8-
6-
6-
2-
20
80
100
20
40
100
Conv6
Conv8
Uniform: 2.75%
He Uniform: 1.15%
Remaining Weights (%6
He SC: 2.75%
He SC: 1.19%
8-
6
6
4
4
2-
20
100
40
100
Epoch
Epoch
Figure 14: CNN Signed Supermask: Average ratio of remaining weights for He Uniform and He SC during training in addition to the respective 5% and 95%-quantile. We find that the variance of He Uniform is at least as large as the variance of He SC but especially in models Conv2 and Conv8 He Uniform has much larger variance. He SC outperforms He Uniform in all architectures.
variance test aceuracy while performing equal or better compared to a uniform distribution. further simplifies the storage of network as we only have store each weight's sign, zero and single weight value for each layer.
E INFLUENCE OF MASK INITIALIZATION
We now pursue the question to what extent mask initialization plays role in performance. Therefore, we use the ELUS models of Sections 3 as baseline with Xavier uniform mask initialization (abbreviated this section as "ELUS/Xavier"). Furthermore, we train the same models with ELUS uniform mask initialization (abbreviated in this section "ELUS/ELUS"), We compare the results separately for the FCN and CNN architectures. As in the last section, we note that these experiments not exhaustive but give intuition the relationship between mask and weight initialization.
FCN Figure 15 visualizes the average accuracy and average remaining weight ratio with the respective 5% and 95% quantiles. We can note that there is almost no difference in variance and average test accuracy for ELUS/ELUS and ELUS/Xavier. The former achieves an average accuracy of 97.36,, whereas the latter marginally but significantly (p < 0.01) exceeds ELUS/ELUS with an average accuracy of 97.48%. As stated in Section 2.1, we presume that mask initialization is not as impactful, long as weight and mask values are of the same magnitude. The FCN results indicate to support this thesis.
We find slight deviation in training behavior in the ratio of remaining weights during training. ELUS/ELUS reduces weight count slightly later in training as ELUS/Xavier does. However, the results again very alike regarding mean and variance.
We can conclude that for the FCN, the difference mask initialization did not influence training behavior and robustness meaningfully, yet HLUS/Xavier outperformed ELUS/ELUS slightly but
30

---

Published as a conference paper at ICLR 2022
98
14
ELUS/ELUS: 3.87%
ELUS/Xavver:: 3.77%
96
12"
Remmaning Weights
94
10
92
ELUS/ELUS
ELUS/Xavier
88 20
40
60
80
100
0
20
40
60
80
100
Epoch
Epoch
Figure 15: FCN Signed Supermask: Average test accuracy and average remaining weights with their respective 5% and 95%-quantile of a ELUS/ELUS and ELUS/Xavier FCN. We can see that both models behave very similar with regards to performance and variance. ELUS//LUS drops its weights a few epochs later but the general behavior is equal.
significantly by 1.pp Moreover, the average ratio of remaining weights differs only by 0. Thus, we argue that both weight/mask initialization combinations might be useful, depending on the task hand.
Our focus shifts towards the CNN architectures. Figure 16 displays the average test accuracy during training with the respective 5% and 95% quantiles for each architecture. We can note very robust training behavior for both mask initializations, as variance is equal across all models. Observe that only Convó ELUS/ELUS performs significantly (p 0.01) better than ELUS/Xavier. We further find that for Conv4, Conv6 and Conv8 ELUS/ELUS begins to optimize effectively a few epochs later than ELUS//avver. Yet, this does not influence the final result. We assume this is caused by the optimizer and straight-through estimation, as the gradient might not be approximated properly in the very early learning phase.
CNN
Figure 17 visualizes the ratio of remaining weights for each CNN architecture with the respective 5% and 95% quantiles. First, we note that each model has a distinct behavior. Whereas Conv2 ascends its weights rapidly, it then almost linearly reduces the weights further until it plateaus. The other architectures exhibit similar behaviour towards the end of training, but without the linear section. This is in line with the findings in Section 3. Additionally, we observe that the choice of weight/mask initialization has little influence on the respective curves for each architecture. As already seen with the FCN model, ELUS/ELUS drops its weights few epochs later than ELUS/Xavier. Moreover, ELUS/Xavier utilizes 0.02pp to 0.4pp weights less than ELUS/ELUS, depending on the network architecture.
Both patterns are consistent for FCN and CNN models. We hypothesize, that the later rise of accuracy and later drop of weight count for ELUS/ELUS cause of the gradient estimation. The shift between the two initializations is consistent throughout all architectures. It is furthermore consistent that ELUS/ELUS results marginally larger ratio of remaining weights.
31

---

Published as a conference paper at ICLR 2022
Conv2
Conv4
80
Accuracy (98
70
70
60
60
50
50
20
40
60
80
100
0
20
40
60
80
100
Conv6
Conv8
80
80
75
75
Accuracy
70
65
60
60
55
.55
100
60
100
Epoch
Epoch
ELUS/ELUS
BLUS/Xavier
Figure 16: CNN Signed Supermask: Average test accuracy for ELUS/ELUS and ELUS/Xavier during training. We also report the respective 5% and 95%-quantile. We find that for all architec- tures, both combinations behave and perform very similar. However, ELUS/ELUS visibly exceeds ELUS/Xavier for conv6.
32

---

Published as a conference paper at ICLR 2022
Conv2
Conv4
12
12
ELUS/ELUS: 1.96%
ELUS/ELUS: 2.94%
Remaining Weights [%8]
ELUS/Xavier: 1.83%
ELUS/Xavier: 2.91%
8
6-
4
2
2
20
80
100
20
40
100
Conv6
Conv8
ELUS/ELUS: 2.75%
ELUS/ELUS: 1.1%%
Remaining Weights (%)
ELUS/Xavier: 2.36%
BLUS/Xavier: 1.17%
4-
20
80
100
40
100
Epoch
Epoch
Figure 17: CNN Signed Supermask: Average ratio of remaining weights for ELUS/ELUS and ELUS/Xavier during training in addition to the respective 5% and 95%-quantil.. For each archi- tecture, it is visible that the training behavior of both combinations very similar. Moreover, we note the different shapes of ascend of the four architectures. ELUS/ELUS drops its weights few epochs later and achieves marginally higher ratio at the end of training. Both initialization combinations behave very robust.
33

---

Published as a conference paper at ICLR 2022
F HOW FAR CAN WE GO?
Two important questions still need to be answered. Can we control the pruning rate? And if so, how small can the subnetworks get before the performance is impacted negatively? As we saw in Table 2, Conv2 achieved the lowest pruning rate of all CNN models. However, as described in the beginning of this section, we lowered the learning rate of Conv2 to tackle its tendencies overfit. Hence, we can investigate if all CNN models show a similar behavior when altering the learning
SiNN
SiNN 2
Learning Rate
0.01 5e-4
0.008
Weight Decay
3e-4
Table 15: Simple-minded Neural Networks: Altered Hyperparameter choices to minimize weight count during training.
parameters in the same way. Table 15 depicts two altered parameter settings, Why the learning rate somewhat influences pruning is for future work investigate. In the following, we will call the networks initialized with ELUS and trained with those hyperparameters SiNN (Simple-minded Neural Networks) models. As Figure 19 reports, we see that both models prune more than 99% of the original weights, SiNN I reaching even pruning rates above 99.5%. Considering the predictive performance we can note, that the performance of SiNN trails significantly behind SiNN for each model.
Let us compare the SiNN models to the baseline models. Figure depicts the test accuracy of the SiNN 2 and baseline models well remaining weights of SiNN 2 and ELUS models on the right side. As SiNN achieves higher accuracy than SiNN I with only few more weights, we do not further consider SiNN at this point. As the compression rate after training highly correlates with the ratio of remaining weights, we do not show this metric for the sake brevity. It can be observed that the SiNN 2 models achieve similar test accuracy with maximum of about 0.6% of the original weights. This is a further 50% reduction of numbers of weights compared to the ELUS models. Additionally, it is interesting see, that the remaining absolute weight count of each CNN model is around 14.000 weights. This suggests that this is the absolute minimum of weights that are needed to map the dataset accurately, independently of the model architecture. The performance difference for SiNN 2 between the different architectures then stems from deeper arrangement of weights, supporting the hypothesis that deeper architectures (with fewer weights) are beneficial. These results reveal three findings: First, are able to indirectly control the pruning rate of signed Supermask models by adapting the learning parameters. Second, SiNN 2 models perform similar to their respective baselines. The uncovered subnetworks however, utilize only 0.3% to 0.6% of the original weights. Third, although performance suffers compared to the ELUS models, the signed Supermasks still perform well, even when pushed the extreme. Ultimately, this demonstrate not only the performance quality of the models, but also how few parameters are actually needed in general to solve given task well.
34

---

Published as a conference paper at ICLR 2022
3.0
Remaining rel. Weights [%]]
2.5
Test Accuracy [96
60
40
1.5
20
0.5
0.0
Conv2
Conv4
Convó
Conv8
Conv2
Conv4
Conv6
Conv8
Model
Model
Baseline Accuracy
Rem. rel. Weights SiNN
Accuracy
Rem. Weights
Figure 18: Conv SiNN 2: Test Accuracy (left) for baselines and SiNN 2 models as well as average ratio of remaining weights the right side for SiNN 2 and ELUS models. 5% and 95% quantiles are reported in the form error bars for both metrics.
80
0.6
Remaining rel. Weights (5g
Test Accuracy [%6]
60
0.5
0.4
40
0.2
20
0.1
0.0
Conv2
Conv4
Convó
Conv8
Conv2
Cone4
Convó
Conv8
Model
Model
SiNN
Accuracy
Rem. rel. Weights
SINN Accuracy
Rem. rel. Weights SiNN
Figure 19: CNN SiNN Signed Supermasks: Average SiNN accuracy (left) and remaining weights (right) over 50 runs of all CNN architectures with error bars indicating the 5% and 95% quantile. SiNN is trumped by SiNN 1 in each architecture terms of test accuracy. Looking at the right side of the plot, the difference in performance that are left active: for this level of sparsity, a certain amount of weights is clearly needed to yield in acceptable performance. Combining this with the information seen Figure 21, we can hypothesize that the loss in performance of SiNN 1 compared to SiNN 1 comes from pruning the first layer even more. This further supports the thesis that the first layer in CNNs crucial for the network's performance, as already stated for the ELUS models. By comparing the masks of SiNNI and SiNN in greater detail, future work could find e.g. specific weights that influence the performance more than other weights and subsequently draw conclusions on network architectures in general.
intuitively be explained by the fewer weights
35

---

Published as a conference paper at ICLR 2022
0.98
6
Remaining Weights (58
0.96
Accuracy [%c]
0.94
0.92
2
0.90
0
20
40
60
80
100
20
40
60
80
100
Epcch
Epoch
BLUS
SiNN
SiNN
Figure 20: FCN SiNN Signed Supermask: Average SiNN accuracy (left) and relative remaining weights (right) over 50 runs for the FCN model. As a comparison, ELUS is visualized as well. Furthermore, we report the respective 5% and 95% quantiles. SiNN does not reach the test accuracy of neither SiNN 2 nor ELUS, but it also utilizes the fewest weights, around 1.4%. We argue however, that the trade-off of SiNN 2 is better, as the "cost" of performance is only a marginally lower pruning rate.
36

---

Published as a conference paper at ICLR 2022
ELUS
SiNN
SiNN
FCN
2
0
Conv2
2
Conv4
55
3
5
0
2
4
6
3
5
Conv6
.5
.5
0
2
4
6
8
3
5
7
8
0
2
4
6
8
Conv8
5
.5
0
5 Layer
2
8
10
Layer
Layer
Figure 21: SiNN Signed Supermask: Average SiNN & mask distribution over 50 runs of all CNN architectures and ELUS for comparison. For the FCN, the overall picture does not change: SiNN & prune the first two layers even more, while the last layer only experiences little more sparsity. The same is partially true for the CNN architectures: while the last last layer remains on an equal pruning level compared to ELUS, the amount remaining weights in the first layer is considerably smaller. On top, all hidden layers are pruned very heavily. Apart from more aggressive pruning in the first layer for SiNN 1, difference between SiNN and SiNN 2 cannot be detected visually.
37


Reviews:
[{"review_id": "sJZHycX0mJ", "reviewer": "Reviewer_UFKs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Previous studies on supermask are interesting because they provide insights that even untrained models have chances to achieve reasonable model accuracy only if we can find such supermask in advance. This reviewer is not sure whether minor improvements over the original supermask idea are critical if a large amount of training is still required.\n\nIf the authors can reveal that the model accuracy can be significantly improved even for untrained models compared to previous supermask, it would be helpful to understand the inherent characteristics of the neural networks initialization. But as shown in Table 1, compared to the work by Zhou et al, even test accuracy is slightly degraded even with additional \"-1\" mask while 'normal training' is still required. The original idea of Supermask is meaningful because it shows a new insight that untrained model has already a sub-network of reasonable accuracy. If the authors want to claim that we can generate a new Supermask generation method, it would not produce additionally new insight. Such concern is obvious for residual networks. Without training batch norm parameters, it is unavoidable to see noticeable accuracy degradation. Then, batch-norm hyperparameters are known to be affine hyperparameters that are superior to normal weights in terms of expressive power. Since the existence of Supermask is already known, the focus of new research in Supermask would need to be finding Supermask efficiently with minimal efforts. Otherwise, Supermask would not be practical in the field. Unfortunately, as indicated in Table 1, training time to find Supermask is quite slow. Then, why we don't just perform usual pruning method to achieve better model accuracy?\n\nComparison on the previous works is missing. At least, this paper needs to include \"What's Hidden in a Randomly Weighted Neural Network?\" in CVPR 2020.", "summary_of_the_paper": "This paper discusses the signed supermask that can improve the model accuracy of untrained neural networks significantly while enhancing sparsity compared to the original supermask idea. The authors introduce \"-1\" as an additional mask value to enable flipping a sign of an initialized weight and suggest related activation functions and fixed threshold hyperparameters to achieve sparse networks. Analysis on sparsity and corresponding accuracy is given for various CNN models including ResNet models. The role of batch normalization for large models is also studied.", "summary_of_the_review": "Since this paper introduces only marginal improvements over original Supermask work, this reviewer cannot find significantly new insights from this paper. Supermask needs to involve much higher model accuracy (than previous works) or minimal efforts to be computed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qjs8VSws3Jy", "reviewer": "Reviewer_okPF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##########################################################################\n\nPros: \n\n \n\n1. The paper proposes a method to find untrained subnetworks that can approach the performance achieved by trained networks. The performance improvement achieved by allowing weights to flip is interesting.\n\n \n\n2. Overall, the paper is well written. Readers can directly grasp the main idea of the paper.\n\n\n\n##########################################################################\n\nCons: \n\n1. My main concern is the motivation and the usage of the proposed method.  As the authors said in the introduction \"However, due to their sheer size, the networks not only became difficult to interpret but also problematic to train and use in real-world applications...\", I expect a method that can either reduce the number of trained parameters or require a fewer number of FLOPs to be proposed. While the authors claim that the subnetworks discovered are untrained, I find that it requires training mask matrices with the same size as the model weights. Considering that straight-through estimator is used to estimate gradients, I suppose the backward pass is also dense and thus leads to no acceleration for training. Compared with directly training a dense network, I suppose the overall training FLOPs required by Signed Supermask is similar. The larger \"TT / Epoch\" over the baseline in Table 1 confirms my concern. I believe it is necessary to explain the benefits of Signed Supermask compared with training a dense network and prune or directly training a sparse network. With the sparsity learned by Signed Supermask, I believe even directly training a sparse neural network with static or dynamic sparsity (e.g., Mocanu et al [1], Evci et al [2], Liu et al [3]) can have similar performance, while with much fewer training FLOPs. I encourage the authors to clarify this.\n\n2. There are some related works that are missing in the paper, e.g., Diffenderfer & Kailkhura [4] (accepted by ICLR 2021) and Chijiwa [5] (accepted by NeurIPS 2021). Since the number of works on this topic is quite small, I expect a good submission should at least do a good related work job by introducing them. What's more, as they are aiming to search for untrained subnetworks without adding another dimension, comparisons with these works are encouraged. The current experiments only include comparisons with Zhou2019 and Ramanujan2019 with small convolutional networks, which is too small to draw a solid conclusion. For instance,  Ramanujan2019, [4] and [5] all provide results on large scale dataset ImageNet.\n\n3. As I understand, the performance improvement of Signed Supermask comes from (1) allowing weight flip, (2) ELUS initialization. I expect ELUS  is a universal method that can improve all the related works. However, I didn't see any ablation study of these two components. It is not clear to me if the improvement is caused by one of them or both of them. \n\n4. As mentioned by the authors, the two threshold hyperparameters in equation (1) control the sparsity level. I expect to see more experiments with how these thresholds influence the sparsity level and the corresponding performance of Signed Supermask.\n\nMinor typos: \n\nInstead of starting with a new paragraph, there is an inexplicable blank after some lines. E.g., 2rd paragraph on page 1; 1st paragraph on page 7.\n\nA space is missing before this sentence \"The gradient is estimated\" on page 3.\n\nReference:\n\n[1] Mocanu, Decebal Constantin, et al. \"Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.\" Nature communications 9.1 (2018): 1-12.\n\n[2] Evci, Utku, et al. \"Rigging the lottery: Making all tickets winners.\" International Conference on Machine Learning. PMLR, 2020.\n\n[3] Shiwei  Liu, et al. ''Sparse training via boost-ing pruning plasticity with neuroregeneration.'' NeurIPS, 2021.\n\n[4] Diffenderfer, J., & Kailkhura, B. (2021). Multi-prize lottery ticket hypothesis: Finding accurate binary neural networks by pruning a randomly weighted network. ICLR, 2021.\n\n[5] Chijiwa, Daiki, et al. \"Pruning Randomly Initialized Neural Networks with Iterative Randomization.\" NeurIPS, 2021.\n\n", "summary_of_the_paper": "This paper proposes Signed Supermask,  an extension of the original Supermask work (Zhou2019) for finding more efficient untrained subnetworks. Instead of learning a binary mask, Signed Supermask claims and shows that adding another dimension -1 to the masks leads to higher sparsity with higher accuracy. The main contribution of this paper is the introduction of weight flipping, as an improvement over the existing approaches. The method is simple and effective. The empirical experiments validate the effectiveness of the proposal. ", "summary_of_the_review": "While the performance achieved by untrained NN is interesting, the motivation and experiments need more work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eAINb6vcAp4", "reviewer": "Reviewer_AEMP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Firstly, this paper is written clearly and easy to follow. The main technical contributions are clear. Performance improvements over (Zhou et al., 2019) and (Ramanujan et al., 2019) are clear.\n\nHowever,  there are several concerns I have for this work.\n\n1. I think more detailed ablation study is needed for the experiments. For example, it is imaginable that the threshold tau will have an influence on the number of remaining parameters but the corresponding experiments are lacking. Having some will be nice. Also, the new training technique is always coupled with the new initialization method. Ablation study to decouple the effect of these two contributions will be good for the readers to understand the which one is more important. This is crucial because in (Ramanujan et al., 2019) the authors also investigated a certain form of scaling of the parameters which is found to improve the performance significantly.\n\n2. In the abstract, the authors posed this work as tackling the issue of difficult interpretation and training/usage in real-world applications of highly overparameterized networks due to their size. However, I don't think the proposed method save anything during training, e.g., memory, computations (in FLOPs) or time because it uses latent weights.\n\nMinor point:\n - In table 2, should the baseline accuracy for conv 8 be 82+% instead of 72+%?", "summary_of_the_paper": "This work extends to (Zhou et al., 2019) and (Ramanujan et al., 2019) to allow sign flipping in the supermasks, without updating the magnitudes of weights. The main technical contributions are the new thresholding-based training method and the new weight initialization scheme that considers the supermasks. Empirical results show better performance on small conv nets and also residual networks on CIFAR-10/100 datasets.", "summary_of_the_review": "Overall I think this work proposes an interesting extension to previous supermask works but lack necessary experiments, making the results less convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bOYOmtl35SH", "reviewer": "Reviewer_sRRw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength:**\n\n- This paper is well written and the proposed method is clearly described. \n- The proposed method is fairly novel, and the resulting high performance and sparsity level of the Conv 2,4,6,8 models are inspiring. \n\n**Weakness:**\n\n- The paper motivates the signed supermask as a way to improve the interpretability of the trained model. While this would be very interesting, I find that the supporting interpretability analyses to be lacking in substance. The mask visualizations are limited to the first layer of a FC network on MNIST, and cannot be directly extended to more layers or other architectures. The observations of layer-wise pruning ratios are interesting, but are not new and have been well studied in the pruning literature. \n\n- Another motivation is in terms of efficiency due to sparsity and compression of the final trained model. To this point, I think it is more appropriate to compare performance to other tertiary network methods, as the supermask baselines have much less capacity due to maintaining the sign at initialization. Is signed supermask a competitive method for training tertiary networks? How does it compare to other tertiary methods in terms of the performance and efficiency tradeoff? Testing the limits of how barebones can a network be is an interesting question, and it may very well be the case that the signed supermask pushes this limit, but it needs to be made clear through a quantitative comparison to similar methods rather than just a discussion.\n\n- Comparisons to the other supermask baselines are missing in the ResNet models for some reason.\n", "summary_of_the_paper": "This paper introduces \"signed supermasks\", which builds on top of the supermasks line of work of finding binary masks on untrained networks that result in good performance. This work extends the original supermask by adding the ability to flip the sign of the weights. The proposed method learns parameters of a mask, and converts the mask parameters into a tertiary mask through the use of two thresholds. The paper further proposes to use ELU activation function and a ELUS initialization scheme that is more tailored to supermask training. Performance is compared against fully-trained models and prior work on supermasks on MNIST, CIFAR10, and CIFAR100, and the proposed method shows competitive results.", "summary_of_the_review": "This paper presents an interesting new approach of training tertiary neural networks with high sparsity levels. The paper is well written and several of the results look promising and would be of interest to the community. However, I take some issue with the motivations of this paper. As a method that is proposed to produce insights on neural network training and enable interpretablility of networks, I find a lack of new insights provided in the paper. As a method that improves efficiency and compressibility, I find it to be very intriguing and promising, but the paper currently lacks the appropriate performance comparisons with other tertiary neural networks to fully convince me of its advantages.\n\n### Update after rebuttal\n\nThank you to the author for the update. I agree that there is value in training the most bare-boned network that we can. Even though not all the value has been demonstrated in the present work, I think it will be useful in many, perhaps unexpected ways in the future. Although this is not really discussed, I'm particularly interested in how this form of training can act as a particular form of regularization that is orthogonal to most other forms. I'll raise my score from 5 to 8 and encourage the authors to continue in this direction.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper builds on previous work on supermasks. It  proposes to replace binary masks by a signed supermask, i.e. a trainable, trashold-based mask that can take values from {-1,0,1}. This change (in combination with the use of ELUS activation functions and an ELUS specific initialization strategy) leads to a significantly higher pruning rate while keeping competitive performance  in comparison to baseline models. 

Most reviewers agreed that the paper is well written and that the proposed approach and the experimental findings are interesting. The motivation to improve interpretability was commonly perceived as misleading. Another  downside that was mentioned is the training time/efficiency. This however, should not be taken too much into account since the work focusses on finding the smallest possible subnetwork that still performs well (without changing the weight values) and- in line with work on the lottery hypothesis-  aims at understanding more about the structure of the „winning tickets“ which is interesting for itself. The paper therefore should be accepted.
Decision: Accept (Poster)