Signing the Supermask: Keep, Hide, Invert

1 INTRODUCTION . In recent years neural networks have established themselves as versatile problem solvers far beyond the field of machine learning . However , the performance increases achieved are obtained by an even larger growth in their size.This not only leads to growing energy costs ( Strubell et al. , 2020 ) , but also complicates interpretability considerably . There are different approaches to reduce various aspects of complexity within a neural network , which broadly can be categorized either as pruning , i.e. , the technique of trimming down neural networks , or as low-precision training , i.e. , representing the weights by low precision numbers . Pruning has shown to significantly reduce the size of a neural network ( in terms of “ active ” weights ) without affecting performance considerably , once the training phase has ended . Compared to the original network , the pruned network is sparse , since many of the original , “ useless ” weights are set to zero . In contrast , replacing weights by low precision numbers , reduces memory size , but does not reveal a sparse , likely easier to interpret network structure . Many approaches of pruning ( e.g. , Janowsky ( 1989 ) ; LeCun et al . ( 1990 ) ; Han et al . ( 2015 ) ; Li et al . ( 2016b ) ; Molchanov et al . ( 2019 ) among others ) exist to identify “ useless ” weights . This raises the question of why not train a smaller network from scratch , since the training of a large network is the computationally most expensive step and pruning is another additional step . Frankle & Carbin ( 2018 ) present a possible solution to this question by formulating the Lottery Ticket Hypothesis ( LTH ) , which states that any densely connected neural network contains smaller subnetworks that , when trained in isolation , perform just as well or even better than the original network . They are able to identify such subnetworks ( winning tickets ) by iteratively pruning the network based on the magnitude of each weight . For deeper architectures , Frankle et al . ( 2019 ) show , that pruning after a few epochs of normal training works significantly better over pruning at initialization . Zhou et al . ( 2019 ) follow the seminal idea of the LTH and are able to train neural networks by only selecting untrained weights ( i.e. , weights are frozen after initialization ) , a concept they call Supermasks . In other words , they find a smaller subnetwork during training without adjusting the weights themselves . Although this approach did not match the performance of their baselines and the pruning rate is inconsistent , it revealed a startling insight : weight values do not seem to be as important as the connection itself ; a single , well-initialized value for each layer is sufficient . Ramanujan et al . ( 2020 ) further develop Supermasks by modifying the way masks are calculated , which leads to significant performance improvements compared to Zhou et al . ( 2019 ) . However , the number of parameters is still high . Recently , Chijiwa et al . ( 2021 ) modified the approach of Ramanujan et al . ( 2020 ) by randomizing the scores , but prune their networks only up to 60 % . In this paper , we propose a technique called signed Supermask , a natural extension of Zhou et al . ( 2019 ) and Ramanujan et al . ( 2020 ) . We not only determine the importance of a weight by masking , but also learn the respective sign of a weight . Signed Supermasks aim at very sparse structures and are able to uncover subnetworks with far fewer parameters in the range of 0.5 % to 4 % of the original network requiring little additional computational effort without sacrificing performance . This differs substantially from low precision training . In its most extreme form , low-precision training quantizes the weight values of a neural network to three constants , including zero or two constants , excluding zero . There , binarized neural networks ( BNN ) ( Courbariaux et al. , 2015 ) reduce complexity but not the size of the networks in terms of sparsity of weights . Ternary neural networks ( TNN ) , introduced by Li et al . ( 2016a ) allow in principle for sparse subnetworks , however this literature focuses almost exclusively on optimizing computational costs while maintaining predictive performance . For TNNs to reduce computational complexity , sparsity ( and interpretability ) is of no importance , as they are focused on reducing the computational footprint only . The literature presents diverse approaches , for example Shayer et al . ( 2017 ) utilize the local reparameterization trick ( Kingma et al. , 2015 ) to learn ternarized weights stochastically , Zhu et al . ( 2016 ) ternarize the weights but scale them layer-wise by two learned real-valued scalars and Alemdar et al . ( 2016 ) employ a teacher-student approach . Deng & Zhang ( 2020 ) train their networks normally with an additional regularization term in the loss function and only ternarize at the end of training by rounding . To the best of our knowledge , this is the only work on TNNs also reporting on sparsity . More recently , Diffenderfer & Kailkhura ( 2021 ) combined the edge-popup algorithm by Ramanujan et al . ( 2020 ) and the binarization of weights and achieve good results . Thus , although TNNs and Supermasks in general appear similar at first glance , their goals are different : while the former attempt to reduce computational complexity , the latter attempt to find the smallest possible subnetworks within a neural network to better understand neural networks in general and work towards facilitating interpretability . This paper takes the Supermasks perspective and our experiments show , that signing the Supermask matches or outperforms baselines and state-of-the-art approaches on Supermasks and leads to very sparse representations . A convenient side effect of signed Supermask is the ternarization of weights , with implied reduced memory requirements ( Li et al. , 2016a ) and further significant speedup in inference ( Hidayetoglu et al. , 2020 ; Brasoveanu et al. , 2020 ) . In summary , signed Supermasks provide two major advantages . First , the network structure is simplified while maintaining or improving the performance compared to their dense counterpart . Second , the reduction facilitates a better understanding of the inner mechanics of neural networks . Based on that , we might be able to build smaller but equally powerful models a priori . Additionally , once trained , signed Supermask models can be stored more efficiently and have the potential for faster inference . 2 SIGNED SUPERMASK . The idea of a signed Supermask is simple , but elegant : the network may not only switch off a given weight , but may also flip its sign if deemed beneficial . We do this by multiplying each weight with a mask m ∈ { −1 , 0 , 1 } , in contrast to m ∈ { 0 , 1 } as in previous work ( Zhou et al. , 2019 ; Ramanujan et al. , 2020 ) . For example , if a single weight is initialized with a negative sign but the performance of the network would increase if it was positive , the signed Supermask has the additional degree of freedom to flip the weight ’ s sign . This step decreases the impact of random initialization , which matters specifically for a signed constant initialization , as introduced by Zhou et al . ( 2019 ) in the context of Supermasks . Despite including -1 in m , we have no additional information to store for the final sparse weight matrices ( i.e. , the final mask multiplied with the weight matrix ) , since it already includes the weight value , the sign and the zero . In this regard , the additional flexibility in the training phase comes for free . Note that signed Supermasks are applicable to any neural network architecture . By masking , we neither alter the information flow , nor any architecture specific properties . For the sake of brevity , however , we focus in the following on the application on feed-forward neural networks only . The general setup is as follows : Let Wl ∈ Rnl×nl−1 denote the frozen weight matrix of layer l ∈ { 1 , . . . , L } of a feed-forward neural network with L layers and nl denoting the number of neurons in layer l. We initialize Wl ∼ Dw . Let further Ml ∈ Rnl×nl−1 , Ml ∼ Dm be a learnable , real-valued matrix that is transformed to values { −1 , 0 , 1 } by a element-wise function g , where g ( M ) = −1 , M ≤ τn 0 , τn < M < τp 1 , M ≥ τp ( 1 ) for some thresholds τn and τp as well as appropriate distributions Dw and Dm that will be specified below . We now define the signed Supermask as M̄ : = g ( M ) , i.e. , a Supermask as in Zhou et al . ( 2019 ) with the additional degree of freedom of a negative sign . Further note that , compared to Zhou et al . ( 2019 ) , the function g ( · ) is deterministic . In order to make full use of the signed Supermask , we utilize the ELU ( Exponential Linear Unit ) activation function ( Clevert et al. , 2015 ) with hyperparameter α. Hereafter , we always set α = 1 . Then , with symbolizing the Hadamard product , each layer l ’ s output ol with zl = ( Wl M̄l ) Tol−1 is computed as follows : ol = { zl , zl > 0 ezl − 1 , zl ≤ 0 ( 2 ) The parameters τn , p are hyperparameters that need to be tuned : the larger they are in absolute value , the more weights are pruned at initialization and the larger values in M need to grow during the course of training to “ activate ” its responding weight . The values of τn , p might differ depending on the task and network architecture at hand . The gradient is estimated using the straight-through estimator ( Hinton , 2012 ; Bengio et al. , 2013 ) as in previous work ( Zhou et al. , 2019 ; Ramanujan et al. , 2020 ) . That is , in the backward-pass , we regard the threshold function g ( · ) as the identity function . Thus , the gradient is going “ straight through ” g ( · ) . Zhou et al . ( 2019 ) generate their masks by sampling the entries from Bernoulli distributions with learnable Bernoulli probabilities . Their M therefore resembles the probability , that weight wij in some layer l is active . Sampling from a Bernoulli distribution can also be interpreted as a stochastic threshold function . Our fixed threshold approach follows the same intuition with the difference of not using a stochastic sampling method . That is , we define two thresholds τn and τp after the initialization phase which remain fixed during the course of training in contrast to Ramanujan et al . ( 2020 ) who update τn and τp after each epoch such that the pruning rate stays constant . Having fixed thresholds has three main advantages which are in line with the argumentation of Zhou et al . ( 2019 ) : first , the network can choose the optimal number of remaining weights . Second , it can choose the optimal distribution of 1 ’ s and −1 ’ s . Third , this approach is fast , as we only have to compute τn and τp once . Moreover , this method , with some modifications , is frequently used in the TNN literature , which underlines its legitimacy . As a result , an exact pruning rate can not be determined a priori , but is learned . However , we show in Appendix F , that the pruning rate can be indirectly influenced by adjusting other hyperparameters .

Summary: This work extends to (Zhou et al., 2019) and (Ramanujan et al., 2019) to allow sign flipping in the supermasks, without updating the magnitudes of weights. The main technical contributions are the new thresholding-based training method and the new weight initialization scheme that considers the supermasks. Empirical results show better performance on small conv nets and also residual networks on CIFAR-10/100 datasets.