Scaling Up Exact Neural Network Compression by ReLU Stability

1 Introduction . For the past decade , the computing requirements associated with state-of-art machine learning models have grown faster than typical hardware improvements [ 5 ] . Although those requirements are often associated with training neural networks , they also translate into larger models , which are challenging to deploy in modest computational environments , such as in mobile devices . Meanwhile , we have learned that the expressiveness of the models associated with neural networks— when measured in terms of their number of linear regions —grows polynomially on the number of neurons and occasionally exponentially on the network depth [ 69 , 65 , 73 , 81 , 36 , 37 ] . Hence , we may wonder if the pressing need for larger models could not be countered by such gains in model complexity . Namely , if we could not represent the same model using a smaller neural network . More specifically , we consider the following definition of equivalence [ 49 , 79 ] : Definition 1 . Two neural networks N1 and N2 with associated functions f1 : Rn0 → Rm and f2 : Rn0 → Rm are local equivalent with respect to a domain D ⊆ Rn0 if f1 ( x ) = f2 ( x ) ∀x ∈ D. There is an extensive literature on methods for compressing neural networks [ 18 , 11 ] , which is aimed at obtaining smaller networks that are nearly as good as the original ones . These methods generally produce networks that are not equivalent , and thus require retraining the neural network for better accuracy . They may also lead to models in which the relative accuracy for some classes is more affected than that of other classes [ 43 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Compressing a neural network while preserving its associated function is a relatively less explored topic , which has been commonly referred to as lossless compression [ 79 , 83 ] . However , that term has also been used for the more general case in which the overall accuracy of the compressed network is no worse than that of the original network regardless of equivalence [ 95 ] . Hence , we regard exact compression as a more appropriate term when equivalence is preserved . Exact compression has distinct benefits and challenges . On the one hand , there is no need for retraining and no risk of disproportionately affecting some classes more than others . On the other hand , optimization problems that are formulated for exact compression need to account for any valid input as opposed to relying on a sample of inputs . In this paper , we focus on how to scale such an approach to a point in which exact compression starts to become practical for certain applications . In particular , we introduce and evaluate a faster algorithm for exact compression based on identifying all neurons with Rectified Linear Unit ( ReLU ) activation that have linear behavior , which are denoted as stable . In other words , those are the neurons for which the mapping of inputs to outputs is always characterized by a linear function , which is either the constant value 0 or the preactivation output . We can remove or merge such neurons—and even entire layers in some cases—while obtaining a smaller but equivalent neural network . Our main contributions are the following : ( i ) We propose the algorithm ISA ( Identifying Stable Activations ) , which is based on solving a single Mixed-Integer Linear Programming ( MILP ) formulation to verify the stability of all neurons of a feedforward neural network with ReLU activations . ISA is faster than solving MILP formulations for every neuron—either optimally [ 90 ] or approximately [ 79 ] . Compared to [ 79 ] , the median improvement is of 83 times on MNIST dataset ( 183 times on CIFAR-10 dataset and 137 times on CIFAR-100 dataset ) —and in fact greater in larger networks . ( ii ) We reduce the runtime with a GPU-based preprocessing step that identifies neurons that are not stable with respect to the training set . The median improvement for that part alone is of 3.2 times on MNIST dataset . ( iii ) We outline and prove the correctness of a new compression algorithm , LEO++ ( Lossless Expressiveness Optimization , as in [ 79 ] ) , which leverages ( i ) to perform all compressions once per layer instead of once per stable neuron [ 79 ] . ( iv ) We leverage the scalability of our approach to investigate exact compressibility on classifiers that are deeper ( 5× 100 ) and wider ( 2× 800 ) than previously studied in [ 79 ] ( 2× 100 ) . We show that approximately 20 % of the neurons and 40 % of the connections can be removed from MNIST classifiers trained with an amount of ` 1 regularization that does not worsen accuracy . 2 Related work . There are many pruning methods for sparsifying or reducing the size of neural networks by removing connections , neurons , or even layers . They are justified by the significant redundancy among parameters [ 23 ] and the better generalization bounds of compressed networks [ 8 , 98 , 87 , 88 ] . Surveys such as [ 11 ] note that these methods are typically based on a tradeoff between model efficiency and quality : the models of compressed neural networks tend to have a comparatively lower accuracy , save some exceptions [ 35 , 95 , 87 ] . Nevertheless , the loss in accuracy due to compression is disproportionately distributed across classes and more severe in a fraction of them ; the most impacted inputs are those that the original network could not classify well ; and the overall robustness to noise or adversarial examples is diminished [ 43 ] . Furthermore , the amount of compression yielding similar performance to the original network can vary significantly depending on the task [ 59 ] . To make up for model changes and potential accuracy loss , one may rely on a three-step procedure consisting of ( 1 ) training the neural network ; ( 2 ) compression ; and ( 3 ) retraining . Nevertheless , the scope of compression methods is seldom restricted to the second step . For example , the compressibility of a neural network hinges on how it was trained , with regularizations such as ` 1 and ` 2 often used to make part of the network parameters negligible in magnitude—and hopefully in impact as well . In fact , the two main—and recurring—themes in this topic are pruning connections when the corresponding parameters are sufficiently small [ 38 , 66 , 47 , 35 , 34 , 57 , 29 , 31 , 89 ] and when the impact of the connection on the loss function is sufficiently small [ 54 , 39 , 52 , 64 , 25 , 96 , 97 , 56 , 91 , 92 , 82 ] . The main issue with the first approach is that small weights may nevertheless be important , although it is possible to empirically quantify their impact on the loss function [ 76 ] . The main issue with the second approach is the computational cost of calculating the second-order derivatives of the loss function in deep networks , which has lead to many approaches for approximating such values . Overlapping with such approximations , there is a growing literature on casting neural network compression as an optimization problem [ 41 , 62 , 1 , 96 , 79 , 26 ] . Most often , these formulations aim to minimize the impact of the compression on how the neural network performs on the training set . Other lines of work and overlapping themes in neural network compression include combining similar neurons [ 84 , 63 , 86 , 87 ] ; low-rank approximation , factorization , and random projection of the weight matrices [ 46 , 24 , 51 , 8 , 93 , 85 , 91 , 87 , 88 , 58 ] ; and statistical tests on the relevance of a connection to network output [ 95 ] . Many recent approaches focus on pruning at initialization instead of after training [ 56 , 55 , 92 , 89 , 30 ] as well as on what parameters to use when these networks are retrained [ 29 , 61 , 74 ] . Exact compression was only recently explored for fully-connected feedforward neural networks [ 79 ] and graph neural networks [ 83 ] . Nevertheless , we may associate it with the literature on neural network equivalency , which includes verifying that networks are equivalent [ 67 , 15 ] , identifying operations that produce equivalent networks [ 42 , 16 , 50 , 49 , 72 ] , reconstructing networks from their outputs [ 3 , 4 , 27 , 2 , 75 ] , and evaluating the effect of redundant representations on training [ 10 , 71 ] . 3 Setting and notation . We consider fully-connected feedforward neural networks with L hidden layers , in which we denote nl as the number of units—or width—of layer l ∈ L : = { 1 , 2 , . . . , L } and xli as the output of the i-th unit of layer l for i ∈ { 1 , 2 , . . . , nl } . For uniformity , we denote x0 ∈ Rn0 as the network input . We denote the output of the i-th unit of layer l as xli = σ l ( yli ) , where the pre-activation output yli : = w l i · xl−1 + bli is defined by the learned weights wli ∈ Rnl−1 and the bias bli ∈ R of the unit as well as the activation function σl : R→ R associated with layer l , which is σl ( u ) = max { 0 , u } —the ReLU [ 33 ] . The output layer may have a different structure , such as the softmax layer [ 13 ] , which is nevertheless irrelevant for our purpose of compressing the hidden layers . For every layer l ∈ L , let W l = [ wl1w l 2 . . .w l nl ] T be the matrix of weights , W lS be a submatrix ofW l consisting of the rows in set S , and bl = [ bl1bl2 . . . blnl ] T be the vector of biases . Finally , let Im ( S ) be an m×m diagonal matrix in which the i-th diagonal element is 1 if i ∈ S and 0 if i /∈ S . 4 Identifying stability for exact compression . This section explains the concept of stability and describes how MILP has been used to identify stable neurons . If the output of neuron i in layer l is always linear on its inputs , we say that the neuron is stable . This happens in two ways for the ReLU activation . When xli = 0 for any valid input , which implies that yli ≤ 0 , we say that the neuron is stably inactive . When xli = yli for any valid input , which implies that yli ≥ 0 , we say that the neuron is stably active . The qualifier valid is essential since not every input may occur in practice . If wli 6= 0 , there are nonempty halfspaces on xl−1 that would make that neuron active or inactive , { xl−1 : wli ·xl−1+bli ≤ 0 } and { xl−1 : wli · xl−1 + bli ≥ 0 } , but it is possible that valid inputs only map to one of them . For the first layer , we only need to account for the valid inputs to the neural network . For example , the domain of a network trained on the MNIST dataset is { x0 : x0 ∈ [ 0 , 1 ] 784 } [ 53 ] . For the remaining hidden layers , we also account for the combinations of outputs that can be produced by the preceding layers given their valid inputs and parameters . Hence , assessing stability is no longer straightforward . We can determine if a neuron of a trained neural network is stable by solving optimization problems to maximize and minimize its preactivation output [ 90 ] . The main decision variables in these problems are the inputs for which the preactivation output is optimized . Consequently , there is also a decision variable associated with the output of every neuron , in addition to other variables described below . MILP formulation of a single neuron For every neuron i of layer l , we map every input vector xl−1 to the corresponding output xli through a set of linear constraints that also include a binary variable zli denoting if the unit is active or not , a variable for the pre-activation output y l i , a variable χli : = max { 0 , −yli } denoting the output of a complementary fictitious unit , and positive constants M li and µ l i that are as large as x l i and χ l i can be . The formulation below is explained in Appendix A1 . wli · xl−1 + bli = yli = xli − χli ( 1 ) 0 ≤ xli ≤M lizli ( 2 ) 0 ≤ χli ≤ µli ( 1− zli ) ( 3 ) zli ∈ { 0 , 1 } ( 4 ) Using MILP to determine stability Let X ⊂ Rn0 be the set of valid inputs for the neural network , which we may assume to be bounded in every direction . We can obtain the interval [ Y l ′ i , Y l′ i ] for the preactivation output yl ′ i of neuron i in layer l ′ by solving the following optimization problems [ 90 ] : Y l ′ i : = { minwl ′ i · xl ′−1 + bl ′ i : x 0 ∈ X ; ( 1 ) − ( 4 ) ∀l ∈ [ l′ − 1 ] , i ∈ [ nl ] } ( 5 ) Y l ′ i : = { maxwl ′ i · xl ′−1 + bl ′ i : x 0 ∈ X ; ( 1 ) − ( 4 ) ∀l ∈ [ l′ − 1 ] , i ∈ [ nl ] } ( 6 ) When Y l ′ i ≤ 0 , then xl ′ i = 0 for every x 0 ∈ X and the neuron is stably inactive . When Y l ′ i ≥ 0 , then xl ′ i = y l′ i for every x 0 ∈ X and the neuron is stably active . Variations of the formulations above have been proposed for diverse tasks over neural networks , such as verifying them [ 17 ] , embedding their model into a broader decision-making problem [ 78 , 9 , 21 ] , and measuring their expressiveness [ 81 ] . When stable units are identified , other optimization problems over trained neural networks become easier to solve [ 90 ] . For example , weight regularization can induce neuron stability and facilitate adversarial robustness verification [ 94 ] . There is extensive work on the properties of such formulations and methods to solve them more effectively [ 28 , 7 , 12 , 80 , 6 ] . For the purpose of identifying stable neurons , however , it is not scalable to analyze large neural networks by solving such optimization problems for every neuron [ 90 ] —or even by just approximately solving each of them to ensure that Y l ′ i ≤ 0 or Y l′ i ≥ 0 [ 79 ] .

Summary: The paper builds upon prior work on MLPs with ReLU activations that leverage the insight that ReLUs are either linear or zero to formulate a MILP to solve the problem of _exact compression_, i.e., the process of reducing neurons in the networks while exactly preserving the output of the network for the input domain. The compression hinges on the insight that if a ReLU neuron is always activated it becomes an affine function while if it's always inactive it's simple zero and does not contribute to the output.  Their main contribution over prior work is a simple, but elegant insight on the problem formulation for the MILP that avoids solving the MILP exactly for each neuron in the network and instead solving it approximately at most N times (N = # neurons) while simplifying the problem (i.e. removing decision variables pertaining to neurons that are not stable) at each step so that only during the final call the MILP has to be solved exactly. The resulting exact compression problem thus has significantly reduced runtime compared to prior work leading to the ability to scale to larger networks (up to 2x800 or 5x100 in their experiments compared to 2x100 in prior work).