SUPERMASK: KEEP, HIDE, INVERT
Year: 2022

Abstract:
The exponential growth in numbers of parameters of neural networks over the past years has been accompanied by an increase in performance across several fields. However, due to their sheer size, the networks not only became difficult to interpret but also problematic to train and use in real-world applications, since hardware requirements increased accordingly. Tackling both issues, we present a novel approach that either drops a neural network’s initial weights or inverts their respective sign. Put simply, a network is trained by weight selection and inversion without changing their absolute values. Our contribution extends previous work on masking by additionally sign-inverting the initial weights and follows the findings of the Lottery Ticket Hypothesis. Through this extension and adaptations of initialization methods, we achieve a pruning rate of up to 99%, while still matching or exceeding the performance of various baseline and previous models. Our approach has two main advantages. First, and most notable, signed Supermask models drastically simplify a model’s structure, while still performing well on given tasks. Second, by reducing the neural network to its very foundation, we gain insights into which weights matter for performance. The code is available here.

heading: 1 INTRODUCTION
text: In recent years neural networks have established themselves as versatile problem solvers far beyond the field of machine learning. However, the performance increases achieved are obtained by an even larger growth in their size.This not only leads to growing energy costs (Strubell et al., 2020), but also complicates interpretability considerably. There are different approaches to reduce various aspects of complexity within a neural network, which broadly can be categorized either as pruning, i.e., the technique of trimming down neural networks, or as low-precision training, i.e., representing the weights by low precision numbers. Pruning has shown to significantly reduce the size of a neural network (in terms of “active” weights) without affecting performance considerably, once the training phase has ended. Compared to the original network, the pruned network is sparse, since many of the original, “useless” weights are set to zero. In contrast, replacing weights by low precision numbers, reduces memory size, but does not reveal a sparse, likely easier to interpret network structure.
Many approaches of pruning (e.g., Janowsky (1989); LeCun et al. (1990); Han et al. (2015); Li et al. (2016b); Molchanov et al. (2019) among others) exist to identify “useless” weights. This raises the question of why not train a smaller network from scratch, since the training of a large network is the computationally most expensive step and pruning is another additional step. Frankle & Carbin (2018) present a possible solution to this question by formulating the Lottery Ticket Hypothesis (LTH), which states that any densely connected neural network contains smaller subnetworks that, when trained in isolation, perform just as well or even better than the original network. They are able to identify such subnetworks (winning tickets) by iteratively pruning the network based on the magnitude of each weight. For deeper architectures, Frankle et al. (2019) show, that pruning after a few epochs of normal training works significantly better over pruning at initialization.
Zhou et al. (2019) follow the seminal idea of the LTH and are able to train neural networks by only selecting untrained weights (i.e., weights are frozen after initialization), a concept they call Supermasks. In other words, they find a smaller subnetwork during training without adjusting the weights themselves. Although this approach did not match the performance of their baselines and the pruning rate is inconsistent, it revealed a startling insight: weight values do not seem to be as important as the connection itself; a single, well-initialized value for each layer is sufficient. Ramanujan et al. (2020) further develop Supermasks by modifying the way masks are calculated, which leads to significant performance improvements compared to Zhou et al. (2019). However, the number of parameters is still high. Recently, Chijiwa et al. (2021) modified the approach of Ramanujan et al. (2020) by randomizing the scores, but prune their networks only up to 60%.
In this paper, we propose a technique called signed Supermask, a natural extension of Zhou et al. (2019) and Ramanujan et al. (2020). We not only determine the importance of a weight by masking, but also learn the respective sign of a weight. Signed Supermasks aim at very sparse structures and are able to uncover subnetworks with far fewer parameters in the range of 0.5% to 4% of the original network requiring little additional computational effort without sacrificing performance.
This differs substantially from low precision training. In its most extreme form, low-precision training quantizes the weight values of a neural network to three constants, including zero or two constants, excluding zero. There, binarized neural networks (BNN) (Courbariaux et al., 2015) reduce complexity but not the size of the networks in terms of sparsity of weights. Ternary neural networks (TNN), introduced by Li et al. (2016a) allow in principle for sparse subnetworks, however this literature focuses almost exclusively on optimizing computational costs while maintaining predictive performance. For TNNs to reduce computational complexity, sparsity (and interpretability) is of no importance, as they are focused on reducing the computational footprint only. The literature presents diverse approaches, for example Shayer et al. (2017) utilize the local reparameterization trick (Kingma et al., 2015) to learn ternarized weights stochastically, Zhu et al. (2016) ternarize the weights but scale them layer-wise by two learned real-valued scalars and Alemdar et al. (2016) employ a teacher-student approach. Deng & Zhang (2020) train their networks normally with an additional regularization term in the loss function and only ternarize at the end of training by rounding. To the best of our knowledge, this is the only work on TNNs also reporting on sparsity. More recently, Diffenderfer & Kailkhura (2021) combined the edge-popup algorithm by Ramanujan et al. (2020) and the binarization of weights and achieve good results. Thus, although TNNs and Supermasks in general appear similar at first glance, their goals are different: while the former attempt to reduce computational complexity, the latter attempt to find the smallest possible subnetworks within a neural network to better understand neural networks in general and work towards facilitating interpretability.
This paper takes the Supermasks perspective and our experiments show, that signing the Supermask matches or outperforms baselines and state-of-the-art approaches on Supermasks and leads to very sparse representations. A convenient side effect of signed Supermask is the ternarization of weights, with implied reduced memory requirements (Li et al., 2016a) and further significant speedup in inference (Hidayetoglu et al., 2020; Brasoveanu et al., 2020). In summary, signed Supermasks provide two major advantages. First, the network structure is simplified while maintaining or improving the performance compared to their dense counterpart. Second, the reduction facilitates a better understanding of the inner mechanics of neural networks. Based on that, we might be able to build smaller but equally powerful models a priori. Additionally, once trained, signed Supermask models can be stored more efficiently and have the potential for faster inference.

heading: 2 SIGNED SUPERMASK
text: The idea of a signed Supermask is simple, but elegant: the network may not only switch off a given weight, but may also flip its sign if deemed beneficial. We do this by multiplying each weight with a mask m ∈ {−1, 0, 1}, in contrast to m ∈ {0, 1} as in previous work (Zhou et al., 2019; Ramanujan et al., 2020).
For example, if a single weight is initialized with a negative sign but the performance of the network would increase if it was positive, the signed Supermask has the additional degree of freedom to flip the weight’s sign. This step decreases the impact of random initialization, which matters specifically for a signed constant initialization, as introduced by Zhou et al. (2019) in the context of Supermasks.
Despite including -1 in m, we have no additional information to store for the final sparse weight matrices (i.e., the final mask multiplied with the weight matrix), since it already includes the weight value, the sign and the zero. In this regard, the additional flexibility in the training phase comes for free.
Note that signed Supermasks are applicable to any neural network architecture. By masking, we neither alter the information flow, nor any architecture specific properties. For the sake of brevity, however, we focus in the following on the application on feed-forward neural networks only.
The general setup is as follows: Let Wl ∈ Rnl×nl−1 denote the frozen weight matrix of layer l ∈ {1, . . . , L} of a feed-forward neural network with L layers and nl denoting the number of neurons in layer l. We initialize Wl ∼ Dw. Let further Ml ∈ Rnl×nl−1 , Ml ∼ Dm be a learnable, real-valued matrix that is transformed to values {−1, 0, 1} by a element-wise function g, where
g(M) =  −1, M ≤ τn 0, τn <M < τp
1, M ≥ τp (1)
for some thresholds τn and τp as well as appropriate distributions Dw and Dm that will be specified below. We now define the signed Supermask as M̄ := g(M), i.e., a Supermask as in Zhou et al. (2019) with the additional degree of freedom of a negative sign. Further note that, compared to Zhou et al. (2019), the function g(·) is deterministic. In order to make full use of the signed Supermask, we utilize the ELU (Exponential Linear Unit) activation function (Clevert et al., 2015) with hyperparameter α. Hereafter, we always set α = 1. Then, with symbolizing the Hadamard product, each layer l’s output ol with zl = (Wl M̄l)Tol−1 is computed as follows:
ol =
{ zl, zl > 0
ezl − 1, zl ≤ 0 (2)
The parameters τn,p are hyperparameters that need to be tuned: the larger they are in absolute value, the more weights are pruned at initialization and the larger values in M need to grow during the course of training to “activate” its responding weight. The values of τn,p might differ depending on the task and network architecture at hand. The gradient is estimated using the straight-through estimator (Hinton, 2012; Bengio et al., 2013) as in previous work (Zhou et al., 2019; Ramanujan et al., 2020). That is, in the backward-pass, we regard the threshold function g(·) as the identity function. Thus, the gradient is going “straight through” g(·). Zhou et al. (2019) generate their masks by sampling the entries from Bernoulli distributions with learnable Bernoulli probabilities. Their M therefore resembles the probability, that weight wij in some layer l is active. Sampling from a Bernoulli distribution can also be interpreted as a stochastic threshold function. Our fixed threshold approach follows the same intuition with the difference of not using a stochastic sampling method. That is, we define two thresholds τn and τp after the initialization phase which remain fixed during the course of training in contrast to Ramanujan et al. (2020) who update τn and τp after each epoch such that the pruning rate stays constant. Having fixed thresholds has three main advantages which are in line with the argumentation of Zhou et al. (2019): first, the network can choose the optimal number of remaining weights. Second, it can choose the optimal distribution of 1’s and −1’s. Third, this approach is fast, as we only have to compute τn and τp once. Moreover, this method, with some modifications, is frequently used in the TNN literature, which underlines its legitimacy. As a result, an exact pruning rate cannot be determined a priori, but is learned. However, we show in Appendix F, that the pruning rate can be indirectly influenced by adjusting other hyperparameters.

heading: 2.1 THOUGHTS ON INITIALIZATION
text: Signed Supermasks do not alter the value of any weight during training. Therefore, it is especially important to initialize weights appropriately. Common weight initialization methods such as He (He et al., 2015) or Xavier (Glorot & Bengio, 2010) neither take masking into account nor the ELU activation function.
Weights We propose a weight initialization scheme that considers both aforementioned alterations, namely, the signed masking process and a differing activation function. In simple terms, since each
layer holds fewer weights, we can scale the weight value to counteract masking. Considering that the goal of initialization methods is to keep the variance in the forward- and backward-pass constant, we suggest to use the following variance:
Var[Wl] = 1.5
nl(1− pl0) (3)
with fan-out nl and layer-wise equal, initial probabilities pl0 = P (M̄ l ij = 0) for layer l ∈ {1, ..., L} and ∀i ∈ 1, . . . , nl, j ∈ 1, . . . , nl−1. A detailed formal motivation of Equation 3 under specified assumptions can be found in Appendix A. The result is still valid without masking, i.e., pl0 = 0. He et al. (2015) come to a similar conclusion for their Parametric ReLU (PReLU) activation function. Henceforth, we refer to this initialization method as ELUS (ELU Scaled) with masking or simply ELU in case of no masking. Notice that ELUS is simply a scaled version of He initialization (He et al., 2015), where the scalar is only dependent on the hyperparameter α of the ELU activation function which is known before training (and set to 1 in this work).
Mask Our assumptions and observations regarding the initialization of the real-valued mask M are as follows: We assume that Dm is symmetric around zero, a standard assumption for common initialization distributions. Given the properties of M , M̄ and the gradient straight-through estimator, it is advantageous if the values in M and W are roughly of the same magnitude, such that the gradient, which is independent of M , is neither too small nor too large. In other words, Dm should be chosen such that it is symmetrical around zero and has a similar variance as Dw. In the case of mask initialization, we advice against a signed constant approach but recommend a normal or uniform distribution instead.

heading: 3 EMPIRICAL ASSESSMENT
text: The following paragraphs present the experiments conducted in order to assess the validity and effectiveness of fixed threshold signed Supermasks. The code for the experiments can be found here.
Experimental Setup To ensure comparability, we utilize the same experimental setup as Zhou et al. (2019) and Frankle & Carbin (2018) i.e., models FCN, Conv2, Conv4 and Conv6 (VGG-like architectures (Simonyan & Zisserman, 2014)) with the additional Conv8 model of Ramanujan et al. (2020) without batch normalization (Ioffe & Szegedy, 2015) and dropout (Srivastava et al., 2014). Since we want to study the functionality of signed Supermasks in particular, any extra functionalities would disturb from the essential topic even though it might bump performance. Details about the aforementioned model architectures can be found in Table 7 in Appendix B. Furthermore, we study signed Supermasks on ResNet20, ResNet56 and ResNet110 (He et al., 2016) with the exception of replacing ReLU with ELU (similar to Shah et al. (2016)). The batch normalization layers in the ResNet models are frozen for signed Supermask training and are only used because its standard practice. For all experiments the same architectures as the respective signed Supermask models but trained conventionally, act as baselines.
The fully-connected (FCN) model is trained on MNIST (LeCun et al., 2010) and all CNNs plus ResNet20 on CIFAR-10 (Krizhevsky, 2009). We only apply per image standardization on both of those datasets. ResNet56 and ResNet110 are trained on CIFAR-100 (Krizhevsky, 2009), where we apply per image standardization, pad the image with 4 pixels on each side and randomly crop a 32x32 image out of the original or its horizontally flipped counterpart.
We ran each experiment 50 times (ResNet56/ResNet110: 3 runs) with the same setup but different weights and masks drawn from the respectively same specified distribution. This ensures that our experimental results are not affected by the randomness of the initialization step. To obtain an intuition of the scale of memory reduction, we provide a simple metric by comparing the average network size in bytes stored in numpy arrays and scipy’s csr_matrix1 sparse matrices after training. For matrices with more than 2 dimensions, we reshape the matrix to two dimensions. This does not give an exact result but provides a good heuristic. Since we attempt to reduce complexity of a neural network, we also provide the average time needed to train a model for a single epoch to further assess practicability.
1The choice of csr is arbitrary. The purpose of this metric is not to show how we can most efficiently store the networks but to give an intuition on added value.
Initialization To initialize the weights of the signed Supermask models, we use signed constants in combination with Xavier, He and ELUS. Thus, the respective weight value has a 50% probability of being negative, where the absolute value of the constant is influenced by the respective initialization method. We initialize the baseline models with the same method within a uniform distribution as the respective signed Supermask model. The idea of using only one (well-chosen) signed constant is particularly compelling from the perspective of complexity reduction. For the ELUS initialization, we scale a He initialization by √ 3 for FCN and Conv architectures and √ 1.5 for our ResNets, which returned best results in the preliminary testing phase. All masks are initialized with Xavier uniform initialization. Results of experiments with different mask initializations and distributions can be found in Appendix C.

heading: 3.1 FULLY-CONNECTED NETWORKS
text: This section evaluates the performance of the fully-connected neural network (FCN) signed Supermask model. Hyperparameters used for training the FCN are listed in Tables 8 and 10 in Appendix B. Note that compared to Zhou et al. (2019), the parameter choices are in line with standard practice. Table 1 summarizes the average test accuracy, remaining weights and training time per epoch with the respective 5% and 95% quantiles for both baseline and signed Supermask models with the different weight initializations. All models achieve similar test accuracy. Best performing is the ELUS signed Supermask model. Furthermore, all signed Supermask models have a wider confidence interval, indicating higher variance. In contrast, confidence intervals for the ratio of remaining weights are very small, demonstrating constant pruning rates throughout all runs. Furthermore, ELUS’ ratio of remaining weights of 3.77% is considerably lower than for the other two initializations. This indicates a superior initialization as the network is able to achieve a higher performance with fewer weights remaining. For another angle of observation, Figure 5 in Appendix C visualizes the average ratio of remaining weights over the course of training for each weight initialization. Regarding efficiency, ELUS signed Supermask models require about 5% more training time. However, the obtained compression rate for the ELUS model after training is 92.01% on average, which makes up for the additional training effort. Although the performance of the ELUS signed Supermask is trailing 0.5pp behind the FCN model of Zhou et al. (2019), we argue that considering the pruning rate (they state the pruning rate only imprecisely, ranging from 7% to 89%) and performance, signed Supermasks are advantageous.
To assess how the model interprets a given dataset, we examine the first masking layer of the network, visualized in Figure 1. The top image represents an input vector (i.e., a flattened MNIST sample), with each vertical line representing one pixel. The bottom shows the mask of the first layer of a randomly selected trained ELUS FCN model. The signed Supermask has learned which of the input nodes are more important than others. Those that match a background pixel are completely masked, while the input nodes that receive actual information are allowed to let the information pass through. Thus, the first layer acts as a filter and feature selector. We further analyze the behavior of the masks and draw comparisons to binary Supermasks in Appendix C.
The results so far show the potential of signed Supermasks in relatively small architectures. Not only does the ELUS model outperform the respective baselines with only 3.8% of the original weights, it
also saves 92% of memory once trained. Furthermore, we gain insights into layer interpretability as pruning is not evenly distributed across the network.

heading: 3.2 CONVOLUTIONAL NETWORKS
text: Here, we present the main experimental results of convolutional neural networks (CNN) signed Supermasks. Overfitting was a major issue in the development, especially for the baseline models. To counter that, the baseline models are only trained for 50 epochs. In contrast, signed Supermask models were not as prone to overfitting. However, once we anticipated overfitting tendencies of Conv2, we reduced its learning rate compared to the other CNN models. Apart from those alterations, the hyperparameter choices are the same throughout all CNN architectures. All models are optimized with SGD with weight decay. Learning rate for Conv models are exponentially reduced over the course of training. Tables 8 and 10 in Appendix B summarize all training parameters. We have found in initial experiments that ±0.01 is a solid choice for the threshold parameters τn and τp. This corresponds to an initial overall pruning rate of about 11% to 30%.Thus, larger layers initially experience a slightly higher pruning rate than smaller layers, following the intuition that weights in smaller layers impact the output of the network to a greater extent than weights in larger layers. Table 2 presents test accuracy for our baselines, the models of Zhou et al. (2019) and Ramanujan et al. (2020) and signed Supermasks as well as remaining weights (i.e. 1 - pruning rate) for the latter three. For a broader comparison to literature, see Table 14 in Appendix C. While we were able to reproduce the results of Zhou et al. (2019), this was not the case for Ramanujan et al. (2020) which not only delivered worse results than reported but also demanded a multiple of the training time. The interested reader can find additional results, experiments on different weight and initialization methods as well as investigations on extreme pruning rates in Appendices C D, E and F. First, apart from Conv2, all CNN models significantly outperform their respective baselines. Conv2 reaches similar accuracy to its baseline, the difference only being 0.4pp. The more our models grow in depth, the greater the benefit of signed Supermasks over the baselines. When considering the ratio of remaining weights, a likely reason for the slight performance glitch of Conv2 becomes apparent: it utilizes only 0.6% of the original weights. In other words, Conv2 reaches similar performance with 99.4% of the original weights unused. Conv4, the smallest model in terms of absolute parameter count, keeps the highest percentage of weights in relation to the original count. As the networks grow in size and depth, we can once again note an increase in the pruning rate. Except for Conv6, there is hardly any variance at all in this metric, indicating a robust behavior.
In terms of efficiency, we see in Table 3 that the signed Supermask models require 6% to 10% more training time, which can be accounted to the additional calculations necessary. However, once the networks are trained the required memory can, on average, be compressed by 94% to 98%. Signed Supermasks outperform not only their respective baselines but also the methods proposed in the related work. Compared to Zhou et al. (2019), our Conv 2, Conv4 and Conv 6 models achieve a higher performance of at least 2pp. Diffenderfer & Kailkhura (2021) do not report performance over a 90% pruning rate, however, their models achieve higher accuracy than ours. We account this to their lower pruning rate. Ramanujan et al. (2020) do not report model performance below a pruning rate of 90% as well. Indeed, in this case, all our Conv models outperform their equivalent models by 3 to 10 pp, while uncovering much smaller subnetworks.
0 1 2 3 40
.5
1
R el
.C ou
nt [%
]
Conv2
0 1 2 3 4 5 60
.5
1 Conv4
0 1 2 3 4 5 6 7 8 Layer
0
.5
1
R el
.C ou
nt [%
]
Conv6
0 -1 1
0 1 2 3 4 5 6 7 8 9 10 Layer
0
.5
1 Conv8
(a) Average ELUS mask distribution. We can observe, that sparsity is very high in very wide layers, indicating that only few of those weights are needed.
0 1 2 3 40
.5
1
E qu
al E
le m
en ts
[% ]
Conv2
0 1 2 3 4 5 60
.5
1 Conv4
0 1 2 3 4 5 6 7 8 Layer
0
.5
1
E qu
al E
le m
en ts
[% ]
Conv6
0 1 2 3 4 5 6 7 8 9 10 Layer
0
.5
1 Conv8
(b) Equality of all 50 final ELUS masks. Masks show high element-wise equality where sparsity is highest.
Figure 2: Conv Signed Supermask: Mask distribution and equality.
Figure 2a helps to interpret the masks by visualizing the average layer-wise mask distribution of each architecture. It can be clearly seen that the first layers are least affected by pruning, while all other layers reach a significantly higher pruning rate of over 99%. As the layer size increases, the pruning rate also increases and peaks in the largest layer of the network. The output layers are also relatively sparse, again indicating proper initialization. This finding confirms our previous assumption that
larger layers contain far fewer weights that are essential to the network. The relative element-wise equality of the final masks over all 50 runs for all layers and architectures is shown in Figure 2b. This provides an intuition about the similarity of the masks and therefore subnetworks through different runs. We observe that pairwise equality is strongly correlated with the size and sparsity of a layer: the larger, the sparser, the more alike. To some degree this is expected as very sparse layers are alike by chance. However, measuring similarity in matrices is a very difficult task (for an overview on this literature from a graph-based perspective see, e.g. Emmert-Streib et al. (2016)), which is why we provide visualizations in Figure 9 in Appendix C that emphasize especially the similarity in structure. This shows, that signed Supermasks repeatedly find strikingly similar subnetworks for different runs, meaning that it might be possible to identify certain neurons that are more important than others and hence better understand neural networks in general.

heading: 3.3 RESIDUAL NETWORKS
text: To further investigate the effectiveness of signed Supermasks, we turn our attention towards Residual networks which allow for deeper architectures. The investigated ResNet consist of 9,27,54 residual blocks, each in equal parts with 16, 32 and 64 filters, respectively. This sums up to a comparatively small parameter count of roughly 0.25M, 0.84M and 1.7M. As hypothesized above, signed Supermasks work better in deeper architectures that contain enough parameters to find a good subnetwork. To investigate this, we multiply the number of filters by 1.5, 2, 2.5 and 3 to create wider variants of ResNet20. Note that even the largest of those variants is smaller than any of the studied VGG-like architectures above, suggesting less overparameterization, especially for the more complex CIFAR-100 dataset. Hyperparameters for baseline and signed Supermask models are depicted in Tables 9 and 11 in Appendix B. Compared to the other investigated architectures, we only reduce the learning rate once the train loss hits a plateau. Furthermore, since there is no large deviation between layer sizes, we set the threshold parameters τn and τp such that only 25% of the weights are left for ResNet20 and 15% for ResNet56 and ResNet110.
Table 4 shows the test accuracy and remaining weights of both baselines and signed Supermasks trained on CIFAR-10. Notably, signed Supermask always trail 2 to 3 pp behind their respective baseline. However, once the signed Supermask model gets wide enough it can reach the performance of the original ResNet20 with only a third of the parameter count. This finding is partially in line with Ramanujan et al. (2020) and their scaling of ResNet50. As expected, the pruning rate increases as the network grows in depth. We suspect that weight count could be further dropped in the deep architectures by adapting the weight decay parameter, for the sake of clarity however, we decided to keep all learning parameters the same. A possible reason for the slightly lower performance of signed Supermasks might be the frozen batch normalization layers, which would benefit learning greatly. We investigate this aspect in Section 3.3.1. The performance and remaining weights of ResNet56 and
ResNet110 on CIFAR-100 are depicted in Table 5. Signed Supermasks show, that they successfully learn from the dataset, even in deep and relatively small environments. However, they cannot reach the same test accuracy as their respective baseline. This can be partially explained by the relatively
small original network sizes. Furthermore, as hypothesized above, batch normalization is of great importance especially in deep architectures like ResNet110, which thereby particularly suffered.

heading: 3.3.1 THE ROLE OF BATCH NORMALIZATION
text: ResNets excel compared to conventional CNNs because of their depth. A big role in being able to learn very deep architectures is batch normalization. Here, we want to study the influence of batch normalization on the ResNet signed Supermask architectures. For those experiments, we explicitly turn on the batch normalization layers in the respective architecture. This parts a bit from the minimalist idea of Supermasks in general, however, weights are still frozen at initialization and this step might be necessary for deep architectures. Table 6 reports the results of the baseline models (as displayed above) and the signed Supermask with learnable batch normalization layers. Compared to the pure signed Supermask models, batch normalization helps to push performance while increasing pruning rate for all models. However, they are unable to reach the performance of the baselines. Nevertheless, taking the remaining weights into account, the predictive performance impresses. More research is needed to investigate the performance gap. Besides this, the results underline that batch normalization helps signed Supermasks to gain better performance in very deep architectures while utilizing fewer weights.

heading: 4 SUMMARY
text: In this work, we introduced the concept of signed Supermasks. Extending the original work on Supermasks (Zhou et al., 2019), we include the sign in the masking process and utilize a simple step-function to calculate the effective masks. Additionally, we gave theoretical and experimental justification for an adapted weight initialization scheme to the masking process, called ELUS. Our experiments on MNIST and CIFAR-10 show, that signed Supermask models at best hold as little as 0.6% of the original weights, while performing on an equal or superior level compared to our baselines and current literature (Zhou et al., 2019; Ramanujan et al., 2020). As a side effect, the trained signed Supermask models’ memory footprint can be compressed by up to 98%. Further analyses on CIFAR-100 with ResNet56 and ResNet110 suggest that signed Supermasks are generally able to work on complex datasets and architectures as well.
ETHICS STATEMENT
The authors cannot envision any socioeconomic or ethical downsides that are specific to signed Supermasks but not neural nets or machine learning in general. On the contrary, we can see obvious benefits, since signed Supermasks facilitate a drastic reduction of energy consumption when deploying neural nets and they can contribute to the interpretability, explainability and ultimately trustworthiness of deep learning approaches.
REPRODUCIBILITY STATEMENT
The authors provide the complete code used to conduct all experiments in the supplementary material. An exemplary configuration and requirements file are attached as well, with which all experiments can be easily reproduced. Importantly, all experiments were run on a single GPU, which simplifies reproduction and strengthens the overall benefits of signed Supermasks. The code is written in Python, mainly using tensorflow and numpy. All datasets used in this work are provided by tensorflow here.

heading: A DERIVATION OF ELUS
text: In the following paragraphs, we revise the initialization methods of He et al. (2015) and Glorot & Bengio (2010). This is needed because first, we introduce masking and a the ELU activation function. Both concepts alter the assumptions of the cited work. Here, we derive a novel adaption of the common He initialization (He et al., 2015). In this section, we use the following notation:
• Activation function σ(x) • Layer l has nl neurons. This means that a layer takes nl−1 inputs per neuron. In the
literature, nl−1 is sometimes called fan-in and nl is sometimes referred to as fan-out • Wl ∈ Rnl×nl−1 , bl ∈ Rnl×1 • Weighted layer input zl = Wlol−1 + bl, zl ∈ Rnl×1 • Layer output ol = σ(zl), ol ∈ Rnl×1 • L denotes the network’s loss function • ∇ol = ∂L∂ol • ∇zl = ∂L∂zl = σ
′(zl)∇ol+1 • In order to maintain clarity, we will denote the transposed weight matrix of layer lW Tl as Ŵl in the derivation of the backward-pass. • W̄l = Wl M̄l. This leads to the suboptimal but necessary choice to denote the transpose of W̄l as W̃l (instead of W̄l T ) to maintain clarity.
We will further assume, that the following holds:
(1) We use the ELU activation function, introduced by Clevert et al. (2015). (2) Variances of input features x are the same Var[x]. (3) For each layer l, the respective weight matrix Wl is drawn from an independent and
symmetric distribution with E[Wl] = 0 and Var[Wl]. It follows that E[zl] = 0. (4) For each layer l, the respective mask matrix Ml is drawn from an independent and symmetric
distribution with E[Ml] = 0. To make the derivations more tangible, we assume pl0 = P (M̄ lij = 0) = 0.5 and p l 1 = P (M̄ l ij = 1) = P (M̄ l ij = −1) = 0.25 ∀i ∈ 1, . . . , nl, j ∈
1, . . . , nl−1 at initialization. In general, the variance is Var[M̄ l] = 1− pl0. (5) For each layer l, all elements in zl and ol share the same variance Var[zl] and Var[ol],
respectively.
Since we use ELU activation functions in our experiments, the question arises whether we can still use the initialization of He et al. (2015). Clevert et al. (2015) do so and achieve good results. Let us therefore calculate the important values that might alter the assumptions necessary. Like Glorot & Bengio (2010) and He et al. (2015) differ in the choice of activation function, we can particularly investigate the parts of the derivation where the expected value of the activation function is used. In particular, this is E[(ol−1)2] in the forward-pass and E[(∇ol)2] in the backward-pass. We can calculate E[(ol−1)2] as follows:
E[(ol−1)2] = E[(αezl−1 − α)2] + E[z2l−1] (4) From here on, we drop index l for the sake of clarity. For the calculation of E[(αezl−1 − α)2], we additionally assume that zl ∼ N (0, 1), although this is a simplification and is therefore only an approximation. With this, we obtain
E[(αez − α)2] = ∫ 0 −∞ (αez − α)2 · 1√ 2π e− 1 2z 2 dz
= α2√ 2π ∫ 0 −∞ (e2z − 2ez + 1) · e− 12z 2 dz
= α2√ 2π ∫ 0 −∞ e− 1 2z
2+2z dz︸ ︷︷ ︸ =:(a)
− 2α 2
√ 2π ∫ 0 −∞ e− 1 2z
2+z dz︸ ︷︷ ︸ =:(b) + α2√ 2π
∫ 0 −∞ e− 1 2z 2
dz︸ ︷︷ ︸ = 12α 2Var[z]= 12α 2
(5)
We can observe that the last term is half of the area under the standard normal probability density function. Note that it holds that −ax2 + bx = −a(x − b2a ) 2 + b 2
4a . With this, we can re-write (a) and (b) as follows
(a) = α2√ 2π ∫ 0 −∞ e− 1 2 (z−2) 2+2 dz = α2e2√ 2π ∫ 0 −∞ e− 1 2 (z−2) 2 dz (6)
(b) = 2α2√
2π ∫ 0 −∞ e− 1 2 (z−1) 2+ 12 dz = 2α2e 1 2 √ 2π ∫ 0 −∞ e− 1 2 (z−1) 2 dz (7)
We can see that (a) is the probability density function of N (2, 1) and (b) is the probability density function of N (1, 1), which can be easily evaluated. With (a) = α2e2 · Φ(z−21 ) = 0.168102α
2 and (b) = 2α2e
1 2 · Φ(z−11 ) = 0.523157α 2 we have
E[(αez − α)2] = 0.168102α2 − 0.523157α2 + 0.5α2 = 0.144945α2 := k (8) If we take the case x > 0 into account, i.e. E[z2] = 12Var[z], we have
E[(ol−1)2] = ( 1
2 + k)Var[zl−1] (9)
Let us now move on to calculating E[(∇ol)2]. Note that the derivative of ELU is simply 1 if z > 0 (which is the same as for ReLU) and αez if z ≤ 0. Let us look at the squared expectation of the second case: E[(σ′(z))2] = E[α2e2z] (10) If we assume again, that zl ∼ N (0, 1), we immediately see that we already calculated this expected value in Equation 6. Therefore, we have
E[(σ′(z))2] = 0.168102α2 := h (11) For the total expected value, we then have
E[(∇ol)2] = 1
2 + h (12)
To summarize, in comparison to (He et al., 2015), we only have to scale Equations 9 and 12 with the constants h and k, respectively. The results above are equal to Equation 15 in He et al. (2015), where they assume a PReLU activation (PReLU as an activation function is also introduced in He et al. (2015)). However, they do not supply a derivation. Therefore we cannot guarantee equality with confidence. Hence, we are careful with the pronoun “our” (initialization) and we will therefore simply refer to the initialization method that follows from the derivations above as “ELU”. Let us now use these results and calculate Var[Wl].
Forward-Pass: We start with the following equation
Var[zjl ] = nl−1∑ k=0 Var[W̄l jk okl−1] (13)
Because E[W̄l] = 0 we can re-write this as
Var[zl] = nl−1Var[W̄l]E[(ol−1)2] (14) With Equation 9 it follows that
Var[zl] = ( 1
2 + k)nl−1Var[W̄l]Var[zl−1] = Var[x] l∏ i=1 ( 1 2 + k)ni−1Var[W̄i] (15)
If we want to achieve constant variance throughout the network, we can set
( 1
2 + k)nl−1Var[Wl]Var[M̄l]
! = 1
⇐⇒Var[Wl] = 1
( 12 + k)nl−1(1− p l 0)
(16)
With Var[M̄l] = pl1 · (−1)2 + pl0 · 02 + pl1 · 12 = 0.5 as stated above and α = 1 we have
Var[Wl] = 1.5505 · 2
nl−1 (17)
Note that for normal training, we would not have to account for the variance of the mask, which would result in Var[Wl] = 1.5505 · 1nl−1 .
Backward-Pass: We know that
Var[∇oil] = nl∑ j=0 Var[W̃l ij σ′(zjl )∇o j l+1︸ ︷︷ ︸
=∇zjl
] (18)
Recall that ELU derivative is
σ′(zl) = { 1, zl ≥ 0 αezl , zl < 0
(19)
We know from Equation 12 that E[(σ′(zl))2] = 12 + h and E[∇ol+1] = 0. It follows, that Var[∇ol+1] = E[(∇ol+1)2], resulting in
Var[∇zl] = ( 1
2 + h)Var[∇ol+1] (20)
With the obtained results we can now re-write Equation 18 as
Var[∇ol] = ( 1
2 + h)nlVar[W̃l]Var[∇ol+1] (21)
Herewith we are able to calculate the variance throughout all layers as
Var[∇o1] = Var[∇oL+1] L∏ l=1 ( 1 2 + h)nlVar[W̃l] (22)
Like in the forward-pass, an easy way to keep the variance constant is to set
( 1
2 + h)nlVar[W̃l]
! = 1
⇐⇒Var[Wl] = 1
( 12 + h)nl(1− p l 0)
= 1.49678 · 2 nl
(23)
As above, if we use this initialization for training a normal neural network, we would use Var[Wl] = 1.49678 1nl .
We will follow the intuition of He et al. (2015) by leaving the choice of Equation 23 or 17 to the user, as their initialization scheme has proven to outperform Glorot & Bengio (2010) in practice and takes the CNN-architecture into account. However, since Equations 23 and 17 are relatively close to each other, we suggest to use the following variance for the sake of simplicity
Var[Wl] = 1.5
nl(1− pl0) (24)

heading: B MODELS AND HYPERPARAMETERS
text: C FURTHER EXPERIMENTAL RESULTS
Figure 3: The above matrix depicts the sum of 50 trained binary Supermasks, i.e. during the training process the weights could either be pruned or “let through”. The same is shown on the bottom for the signed Supermask. A darker shade represents a larger value. In order for any negative and positive masking not to cancel each other out, the absolute masks were summed. Put simply, if a single mask (out of the 50 runs) prunes a weights every time except once, the respective value in the matrices would be 1 and so on. Both versions have an average pruning rate of 97.2% in the first layer. The shown matrices still exhibit a pruning rate of 54.11% in case of the signed Supermask and 43.63% for the binary Supermask, i.e. 54% (43%) of the weights were always pruned. This number is impressive by itself, if we consider that normal neural networks could never reach a similar number. The overall average pruning rates are 3.77% and 4.04% for the signed and binary Supermasks, respectively. While the signed Supermask reached an average performance of 97.48%, the binary Supermask reached an average test accuracy of 97.03%. In other words, the signed Supermask left the same number of weights in the first layer, while the distribution differed more in the other layers but reached a higher performance with roughly 0.3pp less weights. The focus of this examination is to compare the properties of the respective first masks. At first glance, the binary Supermask looks more erratic, while the signed Supermask inhibits a more ordered structure. The impression is not deceiving: out of the 784 columns (i.e. input neurons), 284 in case of the signed Supermask are completely masked, wheres the binary Supermasks only masks 144 (we strongly assume, that the fact that one number is almost double the other is a coincidence). That is, those columns were masked in every considered mask, which shows the robustness of Supermasks in general. Even if we allow 100 outliers (that is, a column-wise maximum sum of 100) per column, binary Supermasks only count 260 of those columns (364 for signed Supermasks). We can observe this fact in two parts: the signed Supermask is more sparse at the edges and the 0-columns in the center are wider and less noisy. This leads to the conclusion that, although on average having the same pruning rate, signed Supermasks are more consistent (otherwise we would count fewer 0-columns) and more understanding of the given data, as it focuses on specific inputs and mostly disregards the image background, which (in the flattened vector) is positioned at the sides. This strongly suggests that not the pruning rate alone is of importance, but also the distribution of pruning per layer. To summarize, although a comparison is not possible without compromise, we can see that the binary Supermask, although exhibiting the same average pruning rate is far less consistent compared to the signed Supermask.
D INFLUENCE OF WEIGHT DISTRIBUTION
In this section we analyze the influence of the weight distribution on overall performance. Therefore, we train additional models with weights drawn from a uniform distributions with the standard thresholds of He (He et al., 2015). As a baseline, we utilize the signed Supermask models with weights as signed constants presented in Section 3. For both variants, we arbitrarily choose He initialization as we expect similar behavior for the other initialization methods. Hence in this Section, we call the former “He Uniform”, the latter “He SC” acting as the baseline. Note that these experiments are not exhaustive, as we only examine the two mentioned distributions. The purpose of this section is to give an intuition on the behavior of different weight distributions. We first analyze the FCN architecture, followed by a summary of the CNN architectures.
FCN Figure 12 displays the average test accuracy and average ratio of remaining weights for both models with the respective 5%- and 95%-quantiles. We can note that the variance of He Uniform is a lot higher than the variance of He SC. Moreover, He Uniform performs on average slightly worse than He SC. The final mean result for He SC is 97.12%, whereas He Uniform achieves 96.77%. A Welch’s t-test for significance yields that He SC significantly (p < 0.01) outperforms He Uniform. The higher variance of He Uniform is also visible in the ratio of remaining weights. On average, He SC and He Uniform achieve almost the same level of sparsity, however the 5%- and 95%-quantile are almost 2pp apart for He Uniform. We postulate that a uniform distribution does not have additional value compared to signed constants for a dense neural network in the context of signed Supermask. Apparently, initializing weights with small values deteriorates performance. Recall that neither He SC nor He Uniform are scaled for the use of a signed Supermask. We further speculate that the higher variance is a cause of random initialization such that training cannot nullify the impact of randomness and ill-initialized weights. This leads to the conclusion that the value of a weight is important if not initialized well, as He Uniform holds more weight values compared to He SC and performs worse. To summarize, using signed constants instead of a uniform distribution provides a much higher level of robustness for the FCN. Our results suggest that for signed Supermask and dense neural networks, a signed constant initialization surpasses the performance of a uniform distribution.
CNN We now investigate the influence of weight distribution on our CNN architectures. Figure 13 visualizes the average test accuracy with the respective 5% and 95% quantiles during training for each model. We first note that He Uniform is not behaving consistently. While with Conv2 and Conv8 the variance of He Uniform is much higher compared to He SC, it is roughly equal for Conv4 and Conv6. The average test accuracy of He Uniform is of the same magnitude as He SC, and we can only register a significantly (p < 0.01) better performance for Conv2 He SC compared to He Uniform. This is not sufficient to conclude on one distributions significance for the general case. The results indicate that the additional weight values delivered by a uniform distribution do not improve the learning capability of a signed Supermask model. This is in line with the findings of the FCN architecture. It seems sufficient to initialize the weight matrix with a well-chosen constant. We further speculate that due to the higher parameter count of the investigated CNN architectures compared to the FCN, the optimizer is to some extent able to neglect the influence of poorly initialized weights. Let us consider the average ratio of remaining weights over the course of training for each CNN model. Figure 14 visualizes this metric for each network in addition to the 5% and 95% quantiles. Unlike observed in the case of the FCN, we cannot report a high variance for He Uniform within CNN models. In fact, the weight distribution does not seem to have an impact on the level of sparsity, as the ratios move almost equally over the course of training for each model. Considering both the performance and level of sparsity, we do not find that the choice between uniform and signed constant distribution impacts the end result on average. However, since the test accuracy of He Uniform varies a lot more compared to He SC, the findings suggest to prefer signed constants over a uniform distribution, as signed constants bring positive side-effects such as efficient
storing and more robust behavior.
To summarize both FCN and CNN behavior, it can be stated that a uniform distribution is not advantageous over a signed constant approach. We further want to re-emphasize the simplicity, yet good performance of signed constants. They have shown a very robust behavior with regards to the
variance in test accuracy while performing equal or better compared to a uniform distribution. It further simplifies the storage of a network as we only have to store each weight’s sign, zero and a single weight value for each layer.
E INFLUENCE OF MASK INITIALIZATION
We now pursue the question to what extent mask initialization plays a role in performance. Therefore, we use the ELUS models of Sections 3 as a baseline with Xavier uniform mask initialization (abbreviated in this section as “ELUS/Xavier”). Furthermore, we train the same models with ELUS uniform mask initialization (abbreviated in this section as “ELUS/ELUS”). We compare the results separately for the FCN and CNN architectures. As in the last section, we note that these experiments are not exhaustive but give an intuition on the relationship between mask and weight initialization.
FCN Figure 15 visualizes the average test accuracy and average remaining weight ratio with the respective 5% and 95% quantiles. We can note that there is almost no difference in variance and average test accuracy for ELUS/ELUS and ELUS/Xavier. The former achieves an average accuracy of 97.36%, whereas the latter marginally but significantly (p < 0.01) exceeds ELUS/ELUS with an average accuracy of 97.48%. As stated in Section 2.1, we presume that mask initialization is not as impactful, as long as weight and mask values are of the same magnitude. The FCN results indicate to support this thesis. We find a slight deviation in training behavior in the ratio of remaining weights during training. ELUS/ELUS reduces its weight count slightly later in training as ELUS/Xavier does. However, the results are again very alike regarding mean and variance. We can conclude that for the FCN, the difference in mask initialization did not influence training behavior and robustness meaningfully, yet ELUS/Xavier outperformed ELUS/ELUS slightly but
significantly by 0.12pp. Moreover, the average ratio of remaining weights differs only by 0.1pp. Thus, we argue that both weight/mask initialization combinations might be useful, depending on the task at hand.
CNN Our focus shifts towards the CNN architectures. Figure 16 displays the average test accuracy during training with the respective 5% and 95% quantiles for each architecture. We can note a very robust training behavior for both mask initializations, as variance is equal across all models. Observe that only Conv6 ELUS/ELUS performs significantly (p < 0.01) better than ELUS/Xavier. We further find that for Conv4, Conv6 and Conv8 ELUS/ELUS begins to optimize effectively a few epochs later than ELUS/Xavier. Yet, this does not influence the final result. We assume this is caused by the optimizer and straight-through estimation, as the gradient might not be approximated properly in the very early learning phase. Figure 17 visualizes the ratio of remaining weights for each CNN architecture with the respective 5% and 95% quantiles. First, we note that each model has a distinct behavior. Whereas Conv2 ascends its weights rapidly, it then almost linearly reduces the weights further until it plateaus. The other architectures exhibit a similar behaviour towards the end of training, but without the linear section. This is in line with the findings in Section 3. Additionally, we observe that the choice of weight/mask initialization has little influence on the respective curves for each architecture. As already seen with the FCN model, ELUS/ELUS drops its weights a few epochs later than ELUS/Xavier. Moreover, ELUS/Xavier utilizes 0.02pp to 0.4pp weights less than ELUS/ELUS, depending on the network architecture. Both patterns are consistent for FCN and CNN models. We hypothesize, that the later rise of accuracy and later drop of weight count for ELUS/ELUS is a cause of the gradient estimation. The shift between the two initializations is consistent throughout all architectures. It is furthermore consistent that ELUS/ELUS results in a marginally larger ratio of remaining weights.

heading: F HOW FAR CAN WE GO?
text: Two important questions still need to be answered. Can we control the pruning rate? And if so, how small can the subnetworks get before the performance is impacted negatively? As we saw in Table 2, Conv2 achieved the lowest pruning rate of all CNN models. However, as described in the beginning of this section, we lowered the learning rate of Conv2 to tackle its tendencies to overfit. Hence, we can investigate if all CNN models show a similar behavior when altering the learning
parameters in the same way. Table 15 depicts two altered parameter settings. Why the learning rate somewhat influences pruning is for future work to investigate. In the following, we will call the networks initialized with ELUS and trained with those hyperparameters SiNN (Simple-minded Neural Networks) models. As Figure 19 reports, we see that both models prune more than 99% of the original weights, SiNN 1 reaching even pruning rates above 99.5%. Considering the predictive performance we can note, that the performance of SiNN 1 trails significantly behind SiNN 2 for each model.
Let us compare the SiNN models to the baseline models. Figure 18 depicts the test accuracy of the SiNN 2 and baseline models as well as remaining weights of SiNN 2 and ELUS models on the right side. As SiNN 2 achieves higher accuracy than SiNN 1 with only few more weights, we do not further consider SiNN 1 at this point. As the compression rate after training highly correlates with the ratio of remaining weights, we do not show this metric for the sake of brevity. It can be observed that the SiNN 2 models achieve a similar test accuracy with a maximum of about 0.6% of the original weights. This is a further 50% reduction of numbers of weights compared to the ELUS models. Additionally, it is interesting to see, that the remaining absolute weight count of each CNN model is around 14.000 weights. This suggests that this is the absolute minimum of weights that are needed to map the dataset accurately, independently of the model architecture. The performance difference for SiNN 2 between the different architectures then stems from a deeper arrangement of weights, supporting the hypothesis that deeper architectures (with fewer weights) are beneficial. These results reveal three findings: First, we are able to indirectly control the pruning rate of signed Supermask models by adapting the learning parameters. Second, the SiNN 2 models perform similar to their respective baselines. The uncovered subnetworks however, utilize only 0.3% to 0.6% of the original weights. Third, although performance suffers compared to the ELUS models, the signed Supermasks still perform well, even when pushed to the extreme. Ultimately, this demonstrate not only the performance quality of the models, but also how few parameters are actually needed in general to solve a given task well.

Review summaries:
This work extends to (Zhou et al., 2019) and (Ramanujan et al., 2019) to allow sign flipping in the supermasks, without updating the magnitudes of weights. The main technical contributions are the new thresholding-based training method and the new weight initialization scheme that considers the supermasks. Empirical results show better performance on small conv nets and also residual networks on CIFAR-10/100 datasets.

This paper introduces "signed supermasks", which builds on top of the supermasks line of work of finding binary masks on untrained networks that result in good performance. This work extends the original supermask by adding the ability to flip the sign of the weights. The proposed method learns parameters of a mask, and converts the mask parameters into a tertiary mask through the use of two thresholds. The paper further proposes to use ELU activation function and a ELUS initialization scheme that is more tailored to supermask training. Performance is compared against fully-trained models and prior work on supermasks on MNIST, CIFAR10, and CIFAR100, and the proposed method shows competitive results.

This paper discusses the signed supermask that can improve the model accuracy of untrained neural networks significantly while enhancing sparsity compared to the original supermask idea. The authors introduce "-1" as an additional mask value to enable flipping a sign of an initialized weight and suggest related activation functions and fixed threshold hyperparameters to achieve sparse networks. Analysis on sparsity and corresponding accuracy is given for various CNN models including ResNet models. The role of batch normalization for large models is also studied.

This paper proposes Signed Supermask,  an extension of the original Supermask work (Zhou2019) for finding more efficient untrained subnetworks. Instead of learning a binary mask, Signed Supermask claims and shows that adding another dimension -1 to the masks leads to higher sparsity with higher accuracy. The main contribution of this paper is the introduction of weight flipping, as an improvement over the existing approaches. The method is simple and effective. The empirical experiments validate the effectiveness of the proposal.