LEAN: graph-based pruning for convolutional neural networks by extracting longest chains

1 INTRODUCTION . In recent years , convolutional neural networks ( CNNs ) have become state-of-the-art for many imageto-image translation tasks ( LeCun et al. , 2015 ) , including image segmentation ( Ronneberger et al. , 2015 ) , and denoising ( Tian et al. , 2020 ) . They are increasingly used as a subcomponent of a larger system , e.g. , visual odometry ( Yang et al. , 2020 ) , as well as in energy-limited and real-time applications ( Yang et al. , 2017 ) . In these situations , the applicability of high-accuracy CNNs may be limited by large computational resource requirements . Small networks may be more applicable in such settings , but may lack accuracy . Neural network pruning ( Mozer & Smolensky , 1989 ; Karnin , 1990 ) has recently gained popularity as a technique to reduce the size of neural networks ( Blalock et al. , 2020 ) . Neural networks consist of learnable parameters , including the scalar components of the convolutional filters . When pruning , the neural network is reduced in size by removing such scalar parameters while trying to maintain high accuracy . We distinguish between individual parameter pruning ( Han et al. , 2016 ) , where each parameter of an operation is ranked and pruned separately , and structured pruning ( Li et al. , 2017 ; Luo et al. , 2017 ) , where entire convolutional filters are ranked and pruned . As convolution operators can only be removed once all scalar parameters of the filter kernel have been pruned , structured pruning is favored over individual pruning when aiming to improve computational performance ( Park et al. , 2017 ) . In the remainder of this paper , we focus on structured pruning . Although structured pruning methods take into account the division of a neural network into operations , they do not take into account the fact that the output of the network is formed by a sequence of such operations . This has two drawbacks . First , since the relative scaling of individual convolutions may vary without changing the output of the whole chain , pruning methods that prune individual operators could potentially prune a suboptimal set of operators from the chain . Second , to significantly reduce evaluation time , a severe pruning regime must be considered , i.e. , a pruning ratio ( percentage of remaining parameters after pruning ) of 1–10 % . In this regime , pruning can result in network disjointness , i.e. , the network contains sections that are not part of some path from the input to the network output . Some existing pruning methods take into account network structure to a limited degree ( Salehinejad & Valaee , 2021 ) . In practice , however , these methods do not contain safeguards to avoid network disjointness . In this paper , we present a novel pruning method called LongEst-chAiN ( LEAN ) pruning , which as opposed to conventional pruning approaches uses graph-based algorithms to keep or prune chains of operations collectively . In LEAN , a CNN is represented as a graph that contains all the CNN operators , with the operator norm of each operator as edge weights . We argue that strong subnetworks in a CNN can be discovered by extracting the longest ( multiplicative ) paths , using computationally efficient graph algorithms . The main focus of this work is to show how LEAN pruning can significantly improve the computation speed of CNNs for real-world image-to-image applications , and obtain high accuracy in the severe pruning regime that is difficult to achieve with existing approaches . This paper is structured as follows . In Section 2 , we explore existing pruning approaches . In Section 3 , we outline the preliminaries on CNNs , pruning filters , and the operator norm . Next , in Section 4 , we introduce LEAN pruning and describe how to calculate the operator norm of various convolutional operators . We discuss the setup of our experiments in Section 5 . In Section 6 , we demonstrate the results of the proposed pruning approach on a series of image segmentation problems and report practically realized wall time speedup . Our final conclusions are presented in Section 7 . 2 RELATED WORK . Reducing the size of neural networks by removing parameters has been studied for decades ( Mozer & Smolensky , 1989 ; Karnin , 1990 ; Hassibi et al. , 1993 ) . Several works take into account the structure of the network to some degree . In Lin et al . ( 2017 ) filters are pruned at runtime based on the feature maps ( Lin et al. , 2017 ) . Alternatively , one can prune entire channels ( He et al. , 2017 ) , or decide which channels to keep so that the feature maps approximate the output of the unpruned network over several training examples ( Luo et al. , 2017 ) . In recent work , a graph is built for each convolutional layer , and filters are pruned based on the properties of this graph ( Wang et al. , 2021 ) . In Salehinejad & Valaee ( 2021 ) a neural network is represented as a graph and interdependencies are determined using the Ising model . Many pruning approaches are aimed at reducing neural network size with little accuracy drop ( Dong & Yang , 2019 ; He et al. , 2019 ; Molchanov et al. , 2019 ; Zhao et al. , 2019 ) , as opposed to sacrificing accuracy in favor of computation speed . These approaches rarely exceed a pruning ratio of 12–50 % ( Blalock et al. , 2020 ; Luo et al. , 2017 ; Lin et al. , 2019 ) . When a high pruning ratio is used , e.g. , a range of 5–10 % ( Lin et al. , 2017 ; Liu et al. , 2019 ) , a significant drop in accuracy is observed . Pruning ratios of 2–10 % can be achieved with an accuracy drop of 1-3 % by learning-rate rewinding ( Renda et al. , 2020 ) . However , the reduction in FLOPs was less substantial ( 1.5–4.8 times ) . In Yeom et al . ( 2021 ) severe pruning ratios of up to 1 % have been considered , but the approach achieved limited improvements in terms of FLOPs reduction compared with existing pruning methods . Criteria for deciding which elements of a neural network to prune have been extensively studied . A parameter ’ s importance is commonly scored using its absolute value . Whether this is a reasonable metric has been questioned ( LeCun et al. , 1990 ) . Singular values ( which determine certain operator norms ) have been used to compress network layers ( Denton et al. , 2014 ) and to prune feed-forward networks ( Abid et al. , 2002 ) . Efficient methods for the computation of singular values have been developed for convolutional layers ( Sedghi et al. , 2019 ) . Furthermore , a definition of ReLU singular values was proposed recently with an accompanying upper bound ( Dittmer et al. , 2019 ) . 3 PRELIMINARIES . 3.1 CNNS FOR SEGMENTATION . A common image to image translation task is semantic image segmentation . The goal of semantic image segmentation is to assign a class label to each pixel in an image . A segmentation CNN computes a function f : Rm×n → [ 0 , 1 ] k×m×n , which specifies the probability of each pixel being in one of the k classes for an m× n image . CNNs are composed of layers of operations which pass images from one layer to the next . Every operation , e.g. , convolution , has an input x and output y . The input and output consist of one or more images , called channels . For clarity , we distinguish throughout this paper between an operation , which may have several input and output channels , and an operator , which computes the relation between a single input channel and a single output channel . For instance , in a convolutional operation with input channels x1 , . . . , xN , an output channel yj is computed by convolving input images with learned filters yj = ( N∑ i=1 hij ∗ xi ) + bj . ( 1 ) Here hij is the filter related to the convolution operator that acts between channel xi and yj , and bj is an additive bias parameter . In a similar way , every CNN operation produces an output which consists of a number of channels . The exact arrangement of operations , and connections between them , depends on the architecture . A common operator to downsample images is the strided convolution . The stride defines the step size of the convolution kernel . A convolution with stride s defines a map h : Rm×n → Rms ×ns . Upsampling images can be done by transposed convolutions . Transposed convolutions intersperse the input image pixels with zeroes so that the output image has larger dimensions . In addition to convolution operators , other common operators such as pooling and batch normalization are often used . A batch normalization operator ( Ioffe & Szegedy , 2015 ) normalizes the input images for convolutional layers . A batch normalization operator scales and shifts an image xi by yi = γ xi − µB√ σ2B + + β . ( 2 ) Here , γ and β are scaling and bias parameters which are learned during training , and µB and σ2B are the running mean and variance of the mini-batch , i.e. , the set of images used for the current training step . For an overview of CNN components we refer to Goodfellow et al . ( 2016 ) . 3.2 PRUNING CONVOLUTION FILTERS . Pruning techniques aim to remove extraneous parameters from a neural network . Several schemes exist to prune parameters from a network , but retraining the network after pruning is critical to avoid significantly impacting accuracy ( Han et al. , 2015 ) . Pruning a network once after training is called one-shot pruning . Alternatively , a network can be fine-tuned , where the network is repeatedly pruned by a certain percentage and is retrained for a few epochs after every pruning step . Fine-tuning typically gives better results than one-shot pruning ( Renda et al. , 2020 ) . Generic pruning algorithm : All pruning methods used in this work make use of the fine-tuning pruning algorithm outlined in Algorithm 1 . The selection criteria for determining which filters to keep for each step define the different pruning methods . The pruning ratio pRatio is the fraction of remaining convolutions we ultimately want to keep , and stepRatio is the fraction of convolutions that is pruned at each step . Algorithm 1 Fine-tuning pruning algorithm 1 : procedure PRUNE ( MODEL , PRATIO , NSTEPS , EPOCHS ) 2 : stepRatio← eln ( pRatio ) /nSteps 3 : for step← 0 to nSteps do 4 : pruneParams← selectPrunePars ( model , stepRatio ) 5 : model← removePars ( model , pruneParams ) 6 : for k← 0 to epochs do 7 : model← trainOneEpoch ( model , trainData ) return model Here , we focus on structured pruning . In structured pruning , a common approach to decide which filters to remove is structured magnitude pruning . When using structured magnitude pruning , a convolution filter h ∈ Rk×k is scored by its L1 vector norm ||h||1 . Filters with norms below a threshold are pruned . The threshold is determined by sorting a group of filters , and removing a percentage based on the pruning ratio . Thresholds can be set per layer or globally . Setting thresholds globally can give higher accuracy than setting thresholds per layer ( Blalock et al. , 2020 ) .

Summary: The authors propose a structured pruning method which turns structure pruning into a graph pruning problem.  The authors represent each input, output pair as an operator node, with edges measured by operator norms between operators.  The authors then propose an iterative structured pruning algorithm which prunes layers based on the longest path in the graph.  They then evaluate their method on three image segmentation tasks.