Max-Affine Spline Insights Into Deep Network Pruning

1 INTRODUCTION . Deep Networks ( DNs ) are powerful and versatile function approximators that have reached outstanding performances across various tasks , such as board-game playing ( Silver et al. , 2017 ) , genomics ( Zou et al. , 2019 ) , and computer vision ( Esteva et al. , 2019 ) . For decades , the main driving factor of DN performances has been progresses in their architectures , e.g . with the finding of novel nonlinear operators ( Glorot et al. , 2011 ; Maas et al. , 2013 ) , or by discovering novel arrangements of the succession of linear and nonlinear operators ( LeCun et al. , 1995 ; He et al. , 2016 ; Zhang et al. , 2018 ) . With a tremendously increasing need for DNs ’ practical deployments , one line of research aims to produce a simpler , energy efficient DN by pruning a dense and overparametrized one , e.g . removing either weights , nodes , filters , layers , or any combination of these options from a DN architecture , leading to a much reduced computational cost ( Frankle & Carbin , 2019b ; Han et al. , 2015 ; Chin et al. , 2020 ; Liu et al. , 2017 ) . Recent progresses ( You et al. , 2020 ; Molchanov et al. , 2016 ) in this direction allow to obtain models much more energy friendly while nearly maintaining the models ’ task accuracy ( Li et al. , 2020 ) . While tremendous empirical progress has been made regarding DN pruning , there remains a lack of explicit understanding of its impact on a DN ’ s decision boundary plus a lack of theoretical tools for deriving pruning techniques in a principled way . Such understandings are crucial for one to study the possible failure modes of pruning techniques , to better decide which to use based on a given application , or to design pruning techniques possibly guided by some a priori knowledge about the given task and data . Our core motivation is to bridge the affine spline formulation of DNs and the recently developed empirical pruning techniques , e.g. , lottery ticket hypothesis Frankle & Carbin ( 2019b ) ; You et al . ( 2020 ) ; Evci et al . ( 2019 ) ; Su et al . ( 2020 ) ; Blalock et al . ( 2020 ) . In this paper , we shed new light on the inner workings of pruning techniques from a spline perspective , by leveraging recent advances in DN understandings and spline formulation ( Montufar et al. , 2014 ; Balestriero & Baraniuk , 2018 ) . Specifically , current DNs are affine splines , that is the input-output mapping is affine in polytopal regions of the input space partition . From this viewpoint pruning acts upon a DN by removing/altering the partition boundaries as demonstrated in Fig . 1 , and therefore pruning affects the decision boundary which is constrained to be linear within the regions of the DN partition . We will demonstrate how this viewpoint allows us to interpret current pruning techniques ( e.g. , lottery tickets hypothesis ( Frankle & Carbin , 2019a ) ) by studying their impact on the DN input space partition , demonstrating how and when can pruning be used without sacrificing the final performances . Finally , we demonstrate how to derive a new pruning scheme based on our gained understandings that reaches competitive performances . In order to ease our development , we slightly abuse notations and refer to a DN as being overparametrized whenever it can be pruned while maintaining its performances and refer to a DN as being minimal whenever it can not be pruned without impacting its performances . We summarize our contributions as follows : [ C1 ] We discover and bridge the connection between spline theory and network pruning techniques . Specifically , we relate the pruning of DN nodes or weights to ( i ) the DN input space partition , ( ii ) the per-region affine parameters , and ( iii ) the decision boundary , providing the explicit interpretation of existing empirical pruning strategies at various granularity levels ( either structured or unstructured ) . [ C2 ] We further extend these insights by proposing a partition-based metric to quantify the evolution of the partition boundaries during training , which allows us to efficiently detect early-bird ( EB ) tickets when an overparametrize DN has been trained enough and can be pruned ; and as opposed to previous EB methods , ours detects EB tickets regardless of the employed pruning techniques or hyperparameters . [ C3 ] We leverage the new insights and the finding of [ C2 ] to derive an efficient pruning strategy from first principles , which only focuses on DN nodes whose corresponding spline partition boundaries contribute to the final decision boundary . A series of experiments on various benchmarked models and datasets validate that our pruning method achieves 3.5× training FLOPs reduction and maintains similar or even better accuracies over state-of-the-art pruning techniques , while being principled and interpretable . 2 BACKGROUND AND RELATED WORKS . A DN transforms an input x through a composition of L layers f ` , ` = 1 , . . . , L to form the final prediction : f ( x ) = ( fL ◦ ⋅ ⋅ ⋅ ◦ f1 ) ( x ) . Each layer is a ( nonlinear ) mapping taking as input a D ` −1-dimensional feature map and producing a D ` -dimensional one , where D ` is the ` th feature map ’ s dimension1 ; each layer ’ s parameters are collected in θ ` . For a fully-connected and activation function layer , θ ` comprises the D ` × D ` −1 dense matrix W ` and the D ` -dimensional bias vector b ` . For convolution operators ( LeCun et al. , 1995 ) , the dense matrix W ` is replaced with a circulant block circulant matrix C ` for channel-wise convolutions and summations . 1we consider matrices and tensors as flattened vectors for clarity Max-affine spline DNs . A key result of ( Montufar et al. , 2014 ; Balestriero & Baraniuk , 2018 ) is the reformulation of current DN layers with ( leaky- ) ReLu/Linear/Abs . Value activation functions as spline operators and in particular as Max-Affine Spline Operator ( MASO ) . With this formulation each layer ’ s input-output mapping can be written as f ` ( z ) = max r=1 , ... , R ( A ` rz + v ` r ) , ( 1 ) with the maximum taken coordinate-wise . By the form of Eq . ( 1 ) it is clear that there exists an underlying layer input space partition based on the realization of the maximum operator ; for a study of this partitioning , see ( Balestriero et al. , 2019 ) . Jointly , the layers combine their input space partition to form the DN input space partition Ω . A few work have focused on studying the relation between the number of regions and the DN architecture ( Hanin & Rolnick , 2019b ) , the analytical form of the partition ( Balestriero et al. , 2019 ) , the upper bound in the number of regions ( Montufar et al. , 2014 ; Montúfar et al. , 2021 ) .The fundamental property that we will leverage throughout this paper is that the internal weights of the layers are paired with their input space partition . As pruning impacts those weights and/or nodes directly , this will offer us new ways to study pruning in DNs ’ input space . More background on DNs , their spline formulation , and its difference with our work can be found in Appendix F. Network pruning . Pruning is a widely used DN compression technique reducing the number of activated nodes ( Liu et al. , 2019c ; LeCun et al. , 1990 ) in a given model . The common pruning scheme adopts a three-step routine : ( i ) training a large model with more parameters than the desired final DN , ( ii ) pruning this overly large trained DN , and ( iii ) fine-tuning the pruned model to adjust the remaining parameters and restore as best as possible the performance lost during the pruning step . Those three steps can be iterated to get a highly-sparse network ( Han et al. , 2015 ) . Within this routine , different pruning methods can be employed , each with a specific pruning criteria , granularity , and scheduling ( Liu et al. , 2019c ; Blalock et al. , 2020 ) . Those techniques roughly fall into two categories : unstructured pruning ( Han et al. , 2015 ; Frankle & Carbin , 2019b ) and structured pruning ( He et al. , 2018 ; Liu et al. , 2017 ; Chin et al. , 2020 ) . Regardless of pruning methods , the trade-offs lie between the amount of pruning performed on a model and the final accuracy . For various energy efficient applications , novel pruning techniques have been able to push this trade-off favorably . The most recent theoretical works on DN pruning relies on studying the existence of Winning Tickets . ( Frankle & Carbin , 2019b ) first hypothesized the existence of sub-networks ( pruned DNs ) , called winning tickets , that can produce comparable performances to their non-pruned counterpart . Later , ( You et al. , 2020 ) showed that those winning tickets could be identified in the early training stage of the unpruned model . Such sub-networks are denoted as early-bird ( EB ) tickets . Despite the above discoveries , the DN pruning literature lacks an explicit understanding and visualization via theoretical analysis that would bring insights into ( i ) current pruning techniques and ( ii ) observed phenomenons such as EB tickets , while leading to principled pruning techniques . We propose to approach this task by leveraging the spline viewpoint of DNs to provide novel interpretations of existing pruning techniques , study the conditions to their success and when should they be avoided , and finally , how to derive novel pruning strategies from first principles . 3 DEEP NETWORKS PARTITION , DECISION BOUNDARY AND PRUNING WORK HAND-IN-HAND . In this section , we first introduce our novel spline interpretation of pruning at various granularity levels . Then we extend such insights by detecting spline EB tickets through a pruning invariant partition-based metric . Finally , we leverage the new insights to derive an principle and efficient pruning strategy . 3.1 INTERPRETING PRUNING FROM A SPLINE PERSPECTIVE . We first propose to leverage the DN input space partition to study the difference between node and weight pruning , representing structured and unstructured pruning , respectively . In the former , nodes of different layers are removed , while in the latter , entries of the W ` matrices ( or C for convolutions ) are removed . We demonstrate in Fig . 1 ( b ) and ( c ) that node pruning removes entire subdivision splines while weight pruning ( or quantization ) can be thought as finer granular limitations on the slopes of subdivision splines , and will only remove the subdivision lines when all entries of a specific row in W ` are pruned , in which case node and weight pruning become identical . From this perspective , we can already identify the reason why pruned networks are less expressive than the overparametrized variants ( Sharir & Shashua , 2018 ) as pruned DNs ’ input space partition and final decision boundary shape are limited compared to their unpruned counterparts . Thus this limitation occurs first on the DN partition which in turn constrains the regions in the space where the decision boundary can be nonlinear as we recall that the DN decision boundary must be linear within the regions of the DN input space partition . Despite the constraints that pruning imposes on the DN input space partition , classification performances do not necessarily reduce when pruning is employed . In fact , the final decision boundary , while being tied with the DN input space partition , does not always depend on all the existing subdivision lines . That is , pruning will not degrade performances as long as the needed decision boundary geometry does not rely on the partition regions that are being affected by pruning . We demonstrate and provide explicit visualization of the above in Fig . 2 and Fig . 3 with fully-connected networks ( FCNets ) and convolutional networks ( ConvNets ) , respectively . The observation consistently shows that only parts of subdivision splines are useful for decision boundary ; and the goal of pruning is to remove those ( redundant ) subdivision splines and find winning tickets . For example , we observe that for a two-layer FCNet ( 20 nodes per layer ) , applying pruning ratios ranging from 20 % ∼ 95 % ( i.e. , prune 4 ∼ 19 nodes ) does not prevent solving the task as long as the remaining subdivision lines are positioned to allow the decision boundary geometry to remain intact . We also extend the above experiment to a high dimensional case with MNIST classification and a DN with two convolutional layers , 20 filters , and kernel sizes of 21 and 5 , respectively in Fig . 3 . By adopting the same channel pruning method as in ( Liu et al. , 2017 ) , we see that most of the pruned nodes remove subdivision lines that were not crucial for the decision boundary and thus only have a small impact on the final classification performance . Hence , as long as pruning leaves at least Iterations Lo ss Epochs Lo ss ( b ) Spline Trajectory for ConvNets ( a ) Spline Trajectory for FCNets those few subdivision lines , the final performances will remain high . Apart from the empirical study , we also provide some analysis about the connection between pruning ( e.g , lottery ticket hypothesis ( LTH ) ) and spline theory in Appendix G .

Summary: This paper conducts a series of empirical studies, which aim at relating node/filter/channel pruning with the partition of input space by neural nets using spline operators as activation functions. In particular, the final decision boundary is only determined by a few splines defined by a few filters of the network. This is verified in the paper through case studies of both fully-connected networks and convolutional networks, which implies that pruning does not severely degrade the accuracy as long as the important splines are not removed by pruning. Another observation is the early-bird tickets: the paper argues that the important splines do not change too much after a few early epochs since the binary activation patterns of data converge rapidly, so prunning can be applied after only a few epochs. The paper then proposes a pruning strategy that sequentially finds the most similar filter pairs and removes them. In experiments, they show that the proposed prunning method is more efficient than some recent pruning methods to achieve similar accuracy with pruning ratio <=70% for CIFAR10/100 and <=50% for ImageNet.