EXPLAINABLE AI-BASED DYNAMIC FILTER PRUNING OF CONVOLUTIONAL NEURAL NETWORKS

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have achieved rapid success in many image processing applications , including image classification [ 1 , 2 , 3 ] image segmentation [ 4 , 5 ] , object detection [ 6 ] , etc . The key ingredients in this success of DNN have been the usage of deeper networks and a large amount of training data . However , as the network gets deeper , the model complexity also increases rapidly . The training of DNNs can be carried out on high-performance clusters with Graphics Processing Unit ( GPU ) acceleration ; however , for implementing these networks on hardware , the complexity of the DNNs needs to be reduced . This includes decreasing the memory requirements , energy consumption , latency , or throughput of the implementation of DNNs on the hardware . Pruning is one of the main techniques that are utilized for reducing the complexity of DNNs . DNN Pruning refers to removing the undesired parameters of a DNN that have little influence on the output of the neural network [ 7 , 8 ] . This leads to fewer Multiply-Accumulate Operation ( MAC ) operations and fewer Number of Parameters ( NPs ) . Pruning can be neuron pruning , filter pruning , weight pruning , and layer pruning . In neuron pruning , individual neurons are removed , i.e . , all the incoming and outgoing connections to a neuron are also removed [ 9 ] . In filter pruning , CNN filters are removed [ 10 ] . In layer pruning , some of the layers can also be pruned [ 11 ] . Weight pruning is used synonymously for unstructured pruning , where the redundant weights are set to zero . The two fundamental objectives for pruning the model are : ( 1 ) reducing the memory by lowering the NPs and reducing the latency and energy consumption of the computation by reducing MACs . These two objectives are often conflicting in a DNN because the MACs are concentrated in the lower layers of a typical image classification network and the NPs are concentrated in the higher layers of a typical image classification network . Pruning can be structured or unstructured . In structured pruning , the filters and weights are eliminated by removing all their input and output connections , and this means that no additional compilation techniques or hardware optimization is required to obtain the gain on hardware in terms of reduction in the size of the model or reduction in the inference time of a sample . This is because the whole model structured is changed in unstructured pruning . In unstructured pruning , the unimportant filters or weights are set to zero , a compiler utilizes these zeros to skip some computations , thereby decreasing the inference time . Unstructured pruning has an additional cost in terms of compilation effort or computational effort in order to exploit the irregular sparsity . Typically , unstructured pruning provides a more pruning ratio as compared to structure pruning . Network Pruning can also be classified in terms of static pruning and dynamic pruning . In static pruning , the parameters of a DNN are removed permanently , while in dynamic pruning , the parameters of a DNN are not removed permanently , instead , they are selectively used for computation based on the input to a DNN . A DNN can be pruned using static pruning , thereby reducing MACs and NPs , and then the same DNN can be dynamically pruned as well . In [ 12 ] , the authors surveyed dynamic neural networks . They divided dynamic networks into three main categories : ( 1 ) instance-wise dynamic models that process each input sample or instance with data-dependent architectures or parameters , ( 2 ) spatial-wise dynamic networks that conduct adaptive computation with respect to different spatial locations of image data , and 3 ) temporal-wise dynamic models that perform adaptive inference along the temporal dimension for sequential data such as videos and texts . Our work comes under the category of instance-wise dynamic models . Within this category , the two types of dynamic networks that are relevant to our work are ( 1 ) dynamic depth models and ( 2 ) dynamic width models . In dynamic depth models , the sample could be predicted earlier in a network . Works such as [ 13 ] utilize this strategy . This strategy is based on the principle that an easier sample in a dataset can be predicted earlier within a network than a hard sample . In contrast to dynamic depth models , the dynamic width models change the number of filters or channels of a CNN , based on the input sample . In [ 14 ] , the authors utilized reinforcement learning for training an agent that judges the importance of each convolutional kernel and conducts channel-wise pruning conditioned on different samples such that the network is pruned more when the sample is easier . Their work utilizes hardness of sample for dynamic behavior in a DNN , whereas our work utilizes coarse prediction . Similar to our work is [ 15 ] , in which the authors proposed a Learning Kernel Activation Module ( LKAM ) , which is able to dynamically activate or deactivate a subset of filter kernels depending upon the input image content during the inference phase . Their method requires a bank of 1x1 convolutional kernels followed by average pooling and a sigmoid function in order to choose which filter kernels in a layer will be activated . Our work is unique in this regard that in addition to classifying easy samples earlier , we utilize an intermediate branch to perform a coarse prediction . The lower layers of a CNN output simpler features , while the higher layers output more categorical features that correlate with a specific class . In higher layers , different filters output features specific to various classes . The concept behind our proposed dynamic pruning is to perform a coarse prediction . Then based on this coarse prediction , we select the CNN filters only relevant to specific classes . explainable AI allows us to obtain filter importances relative to specific classes . Our approach allows us to reduce the average latency as well as the longest-path latency of inference while keeping the overhead low and hardware implementation easy . 1.1 CONTRIBUTIONS . Our contribution can be summarized in the following points : • We propose a novel method for dynamic pruning that utilizes early exit along with early coarse prediction and explainable AI . • The early coarse prediction branch is trained using deep top-k loss . If the branch predicts a sample with high confidence , the prediction is made . Otherwise , coarse prediction is obtained , thereby restricting the possible output to be within a subset of classes . • The coarse prediction is used to dynamically select CNN filters relevant for those classes . • The filters relevant for all output classes are obtained in prior using explainable AI . In run-time , only ranking of filters is done . • The dynamically pruned model is trainable and easily deployable on the end hardware . 2 BACKGROUND . In this section , we briefly describe some of the concepts that are utilized in our work . 2.1 RANKING CRITERIA FOR PRUNING . One of the most essential tasks in filter pruning is to rank the filters based on their importances . The importance of a filter can be obtained locally within a layer or it can be obtained globally within a network . Some ranking criteria for pruning are as follows : Magnitude-based metrics These methods utilize ℓ1-norm and ℓ2-norm of the model weights and have been shown to work reasonably well for general cases in works such as [ 16 ] and [ 17 ] . Loss-preservation based metrics These measures determine the effect of removing a set of parameters on model loss , for example , first-order Taylor decomposition has been used for this purpose by [ 18 ] . XAI-based metrics As described in [ 19 ] , explainable AI ( XAI ) seeks to explain why a neural network produces the output that it does for the input it gets . Explanations of DNNs can be of different types such as explaining which neurons are most sensitive ( saliency methods ) , which neurons have the most effect on output , which input excites which neurons ( signal methods ) , etc . For example , explainable AI has been used to guide the quantization and pruning of DNNs [ 20 ] . Saliency methods explain the decision of a neural network by assigning values that reflect the importance of input components in their contribution towards the output . These methods can be used to obtain the importance of both the features and the weights . Some of the methods in this category are for example : DeepLift [ 21 ] , Conductance [ 22 ] , IntegratedGradients [ 23 ] , etc . We choose DeepLIFT as our work is not aimed at comparing different explainable-AI methods specifically , rather it utilizes them in a novel pruning strategy . We choose DeepLIFT due to its robustness and less computational requirement . One of the most attractive aspects of explainable AI based algorithms is that they provide scores of different filters respective to each output class , which makes it possible for these methods to be utilized in our dynamic pruning architecture . DeepLIFT In [ 21 ] , the authors propose a method for decomposing the output prediction of a neural network on a specific input by backpropagating the contributions of all neurons in the network to every feature of the input . DeepLIFT compares the activation of each neuron to its reference activation and assigns contribution scores according to the difference . The choice of reference activation is important for the algorithm ’ s outcome , and it often requires domain-specific knowledge . To specify a reference activation , we must understand the intuition behind the DeepLIFT algorithm . It compares the effect of the features to a baseline of what the model would predict when it can not see the features . Therefore , a good reference activation for MNIST [ 24 ] is an all-black image . Mathematically , the DeepLIFT algorithm works as follows : Let t represent some target output neuron of interest and let x1 , x2 , . . . , xn′ represent some neurons in some intermediate layer or set of layers that are necessary and sufficient to compute t. Let t0 represent the reference activation of t. The quantity ∆t is defined as the difference-from-reference , that is ∆t = t − t0 . DeepLIFT assigns contribution scores C∆xi∆t to ∆xi s.t . : n′ ∑ i=1 C∆xi∆t = ∆t ( 1 ) C∆xi∆t can be thought of as the amount of difference-from-reference t that is attributed to the differencefrom-reference of xi . ∆t is the DeepLIFT score , which can be represented as follows : DL ( t , x ) = n′ ∑ i=1 C∆xi∆t ( 2 ) 2.2 OBTAINING FILTER IMPORTANCES FROM EXPLAINABLE AI ALGORITHMS . The explainable AI methods take a sample of a dataset as input and output sensitivity maps instead of activations/feature maps . These sensitivity maps have the same dimensions as the feature maps . Let Ω be a set of all indices of all feature map elements of all layers of a neural network , I ( m , Ω ) be the importances of all feature map elements from mth sample from the validation set , then we can define total feature map importance as the average of importances obtained from M samples : I ( θ ) = 1 M M−1 ∑ m=0 I ( m , θ ) ( 3 ) where M is the number of samples used by the XAI method for obtaining the importances . M varies depending upon the dataset , its typical value is 1–2 % of validation samples for CIFAR10 . These sensitivity maps are converted into the importances of the filter weights using different methods , which are as follows : ℓ1-norm criteria We can take ℓ1-norm of the sensitivity map . For example , for a d×d convolutional kernel , for each sample m∈M , the explainable AI method outputs a sensitivity map O of height H and width W , the sensitivity of the weights of this convolutional kernel is given by : W−1 ∑ w=0 H−1 ∑ h=0 |Ow , h| / ( H × W ) . Max-Min criteria We can take also get importances by subtracting max from min of a importances , this is useful in DeepLIFT which also outputs negative importances to indicate pixels that negate a class . This will be given by : W−1 ∑ w=0 H−1 ∑ h=0 max ( Ow , h ) −min ( Ow , h ) / ( H ×W ) .

Summary: The authors propose a set of steps in order to reduce the latency and the flops computation effort of neural networks. They add early exit prediction layers, a top-k prediction layer and dynamic pruning based on the set of classes from the top-k prediction layer. The pruning uses statistics precomputed from a validation dataset in order to decide which filters are to be used for which class. The statistics are obtained from deeplift scores.