INTERPRETABLE CONVOLUTIONAL FILTER PRUNING

1 INTRODUCTION . The great success of Convolutional Neural Network ( CNN ) is benefited from its advanced algorithm and architecture , which utilize inter-connected multi-layer network structure to hierarchically abstract the data feature for recognition tasks Krizhevsky & et al ( 2012 ) . However , this complex mechanism offers CNN outstanding performance at the price of intensive computation cost . Therefore , many CNN optimization works have been proposed to alleviate this computation cost Han & et al ( 2015 ) , Jaderberg & et al ( 2014 ) , etc . While those network redundancy leveraging works are mostly based on quantitative evaluation , the qualitative analysis based on network interpretation is highly overlooked . With the neural network interpretation , CNN can be well analyzed regarding its hierarchical structure and individual component ’ s functionality , and therefore no longer a “ black-box ” . However , most neural network interpretation works is merely considered as a post-analysis approach , and very few works have utilized it to analysis and guide the optimization directly ( Yosinski et al . ( 2015 ) ) . Considering the absence of the network interpretation analysis and optimization , in this work we utilized different CNN interpretation techniques to analysis the convolutional filter functionality and optimize the filter pruning method . In this work , we have the following major contributions : • We utilized CNN visualization techniques and backward-propagation gradients analysis to interpret the convolutional filter functionality , and demonstrate significant network redundancy due to repetitive filter functions rather than insignificant magnitude ; • We designed a hierarchical filter functionality-oriented pruning method to explore the interpretable neural network optimization ; • We observed the filter functionality transission with model tuning , and reveals the inappropriate pruning and retraining methods may significantly defect the filter functionality and cause unnecessary network reconstruction effort . Experiment results with various CNN models and image dataset show that , the proposed filter interpretation approach can effectively analysis the convolutional filters ’ the functionality and similarity . The proposed functionality-oriented pruning methods also achieves outstanding performance compared to traditional filter pruning methods with better training efficiency and interpretability 2 RELATED WORK . Convolutional Filter Pruning It is well known that the major computation cost in a CNN comes from the convolutional layers . Therefore , most CNN optimization works fall into convolutional filter pruning or compact CNN training with convolutional filter constraints : Li et al . ( 2016 ) ranked the convolutional filters with their absolute weight sum as the filter significance indication and pruned the insignificant filters for better computation efficiency , which is well-recognized and ` 1 ranking based prunning ; Hu et al . ( 2016 ) evaluated the filter significance calculating the occupation of zero valued weights for pruning guidance ; He et al . ( 2017 ) also applied Lasso-regression in the filter pruning for batched processing . The filter significance is also evaluated with the filter feature maps : Polyak & Wolf ( 2015 ) pruned the filters with small feature map values ; Luo et al . ( 2017 ) evaluate filter ’ s significance by testing its impact on feature map reconstruction errors after being pruned . Most of these works are based on quantitative analysis on the filter significance . However , such an approach is already questioned by some recent works : Huang et al . ( 2018 ) proved that certain redundancy also exists in filters with large “ significance ” ; while Ye et al . ( 2018 ) correspondingly demonstrated that filters with small values also significantly affect the neural network performance . Therefore , rather than merely evaluating the convolutional filters in a quantitative approach , we analysis the filter functionality in a qualitative manner to guide the neural network optimization . Convolutional Filter Visualization As the convolutional filters are designed to capture certain input features , the semantics of the captured feature can effectively indicate the functionality of each filter . However , as the convolutional filter are embodied by matrices , the functionality is hard to directly interpret , which significantly hinders the qualitative neural network optimization . Recently , many CNN visualization techniques have been proposed to qualitatively analysis the CNN in perspectives of network structure and component functionality , such as Activation Maximization ( AM ) ( Yosinski et al . ( 2015 ) ) , Network Inversion ( Mahendran & Vedaldi ( 2015 ) ) , Deconvolutional Neural Networks ( Zeiler & Fergus ( 2014 ) ) , and Network Dissection based Visualization ( Zhou et al . ( 2016 ) ) . Among those methods , AM offers the most efficient and effective interpretation for the convolutional filter functionality , which is defined by each filter ’ s maximum activation pattern . In this work , we utilize the AM as our major visualization tool for convolutional filter analysis . 3 CONVOLUTIONAL FILTER FUNCTIONALITY INTERPRETATION AND FILTER REDUNDANCY ANALYSIS . Different from previous quantitative filter significance analysis , the convolutional filter functionality is qualitatively interpreted in this paper . Based on the functionality interpretation , the neural network redundancy is further analyzed regarding the filter functionality . For preliminary demonstration , we adopt a VGG-16 model trained on CIFAR-10 dataset for intuitive result demonstration . 3.1 INTERPRETABLE CONVOLUTIONAL FILTER FUNCTIONALITY ANALYSIS . To qualitatively interpret the convolutional filter functionality , both AM visualization and backwardpropagation gradients analysis are utilized . AM visualization interprets a filter ’ s feature extraction preference from the neural network inputs , while the backward-propagation gradients evaluate the filters ’ contribution to classification outputs . In AM visualization , the feature extraction preference of a filter F li from layer Ll is represented by a synthesized input image X that causes the highest activation of F li ( i.e . the convolutional feature map value ) . Mathematically , the synthesis process of such an input can be formulated as : V ( F li ) = arg max X Ali ( X ) , X ← X + η · ∂Ali ( X ) ∂X , ( 1 ) where Ali ( X ) is the activation of filter F l i from an input image X , η is the gradient ascent step size . WithX initialized as an input image of random noises , each pixel of this input is iteratively changed along the ∂Ali ( X ) /∂X increment direction to achieve the highest activation . Eventually , X demonstrates a specific visualized pattern V ( F li ) , which contains the filter ’ s most sensitive input features with certain semantics , and represents the filter ’ s functional preference for feature extraction . Fig . 1 demonstrates several filters ’ visualized patterns from different layers . By interpreting the content of these patterns , it is clear to see that the filters in lower layers are more sensitive to fundamental colors and shapes . While with the layer depth increasing , the patterns tend to show specific objects and eventually recognizable class targets . In the backward-propagation gradients analysis , the gradients indicate the impact of each convolutional filter to individual classification targets : γi , yj = 1 N N∑ n=1 ∥∥∥∥ ∂P ( yj ) ∂Ali ( xn ) ∥∥∥∥ , ( 2 ) where P ( yj ) is the output probability of a sample image xn to the class yj , and Ali ( xn ) is the activation of filterF li for each test image xn . ∂P ( yj ) /∂Ai ( xn ) is the backward-propagation gradient of class yj respective to the filter ’ s activation , which indicates the dedicated contribution of filter F li to class yj . With preliminary experiments , we also find that some filters ’ visualized patterns also constrain obvious class objects , which indicates large impact to certain classes yj . Therefore , the connection between filters and dedicated class outputs can be also determined . Associating these two analysis methods , namely the CNN visualization and backward-propagation gradients analysis , the filters ’ functionalities can be well interpreted in a qualitative manner . 3.2 FUNCTION SIMILARITY BASED FILTER REDUNDANCY ANALYSIS . Based on the above analysis , the convolutional filter functionality can be well interpreted regarding the input feature extraction preference and output contribution . These qualitative interpretations also guide certain neural network redundancy analysis , where filters with similar functionalities may indicate unnecessary structural repetition . Such repetition can be well identified among the filters with similar visualized patterns , as shown in the Fig . 1 : as denoted by the red blocks , significant functionality similarities present among the convolutional filters in each layer . To quantitatively determinate possible filter functionality repetition , a similarity degree between filter F ( c , l ) k and F ( c , l ) i is formulated regarding both AM visualized patterns and the back-propagation gradients : SD [ V ( F ( c , l ) i ) , V ( F ( c , l ) k ) ] = ‖V ( F ( c , l ) i ) − V ( F ( c , l ) k ) ‖ 2 · ‖γi − γk ) ‖2 , ( 3 ) where ‖V ( F ( c , l ) i ) − V ( F ( c , l ) k ) ‖2 calculates the Euclidean distance between two visualized patterns , which indicate feature extraction preference similarity 1 . And ‖γi − γk ) ‖2 calculates the similarity between the two filter ’ s contribution to specific output classes . To equally evaluate these two criterion , both vectores are normalized , and the summation is utilized as the filter similarity degree . 1Although image similarity analysis tools like SSIM are more specialized , they can not be applied into low layer filters with extremely small resolutions . Therefore , the Euclidean distance is adopted for its generality . According to the qualitative analysis of Eq.3 , a set of matrices filter similarity degrees are constructed as shown in Fig.2 . The matrix axes represent the filter index in each layer ( e.g . 512x512 for filter Conv5 2 ) , and each cross-point indicates the similarity degree SD between two filters . The magnitudes of the similarity degrees are denoted by a color range , where the minimum is denoted as red and the maximum is denoted as white . From Fig.2 , it can be observed that the filter similarity widely present in different layers . Meanwhile , significant filter similarity exit in the shallowest and deepest layers of the neural network ( e.g . Conv1 1 and Conv5 2 ) . Those high similarity degrees indicate considerable functionality repetition among the convolutional filters , which might be leveraged for effective neural network optimization . In later sections , the filter functionality evaluation approach is widely applied for network redundancy analysis and filter pruning guidance . 4 HIERARCHICAL FILTER FUNCTIONALITY-ORIENTED PRUNING . Based on the convolutional filter functionality and redundancy analysis , we further propose a novel hierarchical filter pruning method oriented by the qualitative functionality interpretation . 4.1 PRUNING METHOD OVERVIEW . The proposed functionality-oriented filter pruning method is designed in a hierarchical manner to address the functionality redundancy in different neural network component levels . The method has the following major stages : ( 1 ) Filter Clustering – Based on the aforementioned filter similarity analysis with AM visualization and the backpropagation gradients , the filters with similar functionalities in each layer are clustered into multiple groups . The filters in the same cluster are considered to have repetitive functionalities ; ( 2 ) Filter Level Pruning – Inside each cluster , the filters ’ relative contribution to the output classes are qualitative evaluated by the backpropagation gradients analysis . The contribution index of each filter can be considered as the filter significance regarding its functionality ; ( 3 ) Cluster Level Pruning – Inside each layer , the relative cluster pruning rate is based on the cluster volume size to guide the filter pruning in each cluster ; ( 4 ) Layer Level Pruning – Guided by the layer sensitive to pruning , the relative pruning ratio for each layer is calculated to guide the pruning in each layer and therefore each cluster . Fig . 3 ( a ) illustrate an intuitive example for the filter clustering in the layer Conv5 1 , where each row represents one cluster in the layer , and the horizontal axis indicates the filter ’ s contribution index . Given a pruning rate for the model , the relative pruning rate for each layer and cluster can be determined , and the filters with least classification contribution will be pruned first ( marked by green crosses ) . Meanwhile , we also demonstrate ` 1 ranking based filter pruning in the Fig . 3 ( b ) . Considering ` 1 ranking ’ s quantitative filter significance evaluation , the pruned filters ( marked by red crosses ) have no interpretable selection patterns and no functionality correlations . Moreover , Algorithm 1 Visualization based k-means Filter Clustering Input : CNN , Number of layers L , and Number of filters in each layer Il Output : L Clusters 1 : for each Layer Ll do Listl= [ ] ; 2 : for each Filter F li do Generate the pattern : V ( F li ) =X li ; Listl.append ( V ( F li ) ) ; 3 : for c in range ( 1 , Il/2 ) do Grid search all possible cluster numbers : Cl=kmeans.cluster ( Listl , c ) ; 4 : for Ccl in Cl do Merge the single filter clusters : Cllocked = [ ] ; 5 : if Len ( Clc ) == 1 then Cllocked.append ( Clc ) ; k=k-1 . 6 : Select the maximal c as the optimal cluster number : c=Max ( c ) return Clc comparing to ` 1 ranking based filter pruning , the proposed functionality-oriented pruning requires extremely small effort for model retraining , which will be further explained in later sections . In the next section , the details for each processing stage will be presented .

Summary: In this paper, the authors propose a method for pruning the convolutional filters. This method first separates the filters into clusters based on similarities defined with both Activation Maximization (AM) and back-propagation gradients. Then pruning is conducted based on the clustering results, and the contribution index that is calculated based on backward-propagation gradients. The proposed method is compared with a baseline method in the experiments.