Training Interpretable Convolutional Neural Networks towards Class-specific Filters

1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) demonstrate extraordinary performance in various visual tasks ( Krizhevsky et al. , 2012 ; He et al. , 2016 ; Girshick , 2015 ; He et al. , 2017a ) . However , the strong expressive power of CNNs is still far from interpretable , which significantly limits its applications that require humans ’ trust or interaction , e.g . self-driving and medical image analysis ( Caruana et al. , 2015 ; Bojarski et al. , 2017 ) . In this paper , we argue that filter ambiguity is one of the most critical reasons that hampers the interpretability of CNNs . As a matter of fact , previous studies has shown that 1 ) filters in CNNs generally extract features of a mixture of various semantic concepts , including objects , parts , scenes , textures , materials and colors ( Zhang et al. , 2018b ; Bau et al. , 2017 ) ; and that 2 ) there is also redundant overlap between features extracted by different filters ( Prakash et al. , 2019 ) . The intricate many-to-many correspondence relationship between filters and features is so-called filter ambiguity as shown on the left of Figure 1 . Obviously , in high convolutional layers which might capture class-related feature , filter ambiguity contradicts our intention of an interpretable CNN , because it hinders humans from interpreting the concepts of a filter ( Zhang et al. , 2018b ) , which has been shown as an essential role in the visualization and analysis of networks ( Olah et al. , 2018 ) in human-machine collaborative systems ( Zhang et al. , 2017a ; c ) . Moreover , the unnecessary overlap between features extracted by different filters leads to under-utilization of a model ’ s expressiveness ( Prakash et al. , 2019 ) Therefore , reducing filter ambiguity is critical to obtain better feature with better interpretability and less redundancy . However , it is non-trivial to achieve such a goal barricaded by substantial challenges . First , most interpretability-related research simply focuses on post-hoc interpretation of filters ( Szegedy et al. , 2013 ; Bau et al. , 2017 ) , which manages to interpret the main semantic concepts captured by a filter but fails to alleviate the filter ambiguity prevalent in pretrained models . Second , many existing works such as VAEs ’ variants ( Higgins et al. , 2017 ; Burgess et al. , 2018 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ; Kumar et al. , 2017 ) and InfoGAN ( Chen et al. , 2016 ) try to disentangle data representation and obtain better interpretability in an unsupervised way . However , it is proved that unsupervised learning on disentangled features without inductive bias is impossible ( Locatello et al. , 2018 ) , which challenges the works above . Considering the aforementioned challenges , we shed light on enforcing the one-class for one-filter relationship during training ( instead of post-hoc ) in a supervised manner only with classification labels . To this end , we propose a novel training strategy that coerces each filter into extracting features from only one or few classes for classification tasks , namely disentangling filters towards classspecific . Specifically , we design a Label Sensitive Gate ( LSG ) structure on the top of convolutional filters , which limits each filter ’ s activation only to its specific input label ( s ) . In our training process , we periodically insert LSG into the CNN and jointly minimize the classification cross-entropy and the sparsity of LSG , so as to keep the model ’ s performance on classification and meanwhile encourage class-specific filters . Experiments proved the LGS filter yield better representation for images hence leading to promising applications . Our training method makes data representation sparse and highly correlated with the labeled class , which not only illustrates the alleviation of filter ambiguity but also enhances the interpretability of the network . The advantages of our method are concretely substantiated by sparser correlation between filters , sparser filter-class correlation , better explanation for misclassification and more precise localization of labeled objects . Contributions The contributions of this work can be summarized as : ( 1 ) we propose a novel training strategy for CNNs which forces each filter to extract features mainly from only one or a few classes ; ( 2 ) moreover , we propose a metric to evaluate filter ambiguity , which can also be used as a regularization in our training to encourage class-specific filters ; ( 3 ) finally , our training enables filters to output sparse and class-related representation , which helps alleviate filter redundancy , improve interpretability of the model . 2 RELATED WORKS . Existing works related to our work include post-hoc filter interpretation , learning disentangled representation and model pruning . Post-hoc Interpretation for Filters is widely studied , which aims to interpret the patterns captured by filters in pretrained CNNs . Plenty of works visualize the pattern of a neuron as an image , which is the gradient ( Zeiler & Fergus , 2014 ; Mahendran & Vedaldi , 2015 ; Simonyan et al. , 2013 ) or accumulated gradient ( Mordvintsev et al. , 2015 ; Olah et al. , 2018 ) of a certain score about the activation of the neuron . Bau et al . ( 2017 ) determine the main visual patterns extracted by a convolutional filter by treating it as a pattern detector . Some other works transfer the representation in CNN into an explanatory graph ( Zhang et al. , 2017b ; 2018a ) or a decision tree ( Zhang et al. , 2019 ) , which aim to figure out the visual patterns of filters and the relationship between co-activated patterns . Posthoc filter interpretation helps to understand the main patterns of a filter but makes no change to the existing filter ambiguity of the pretrained models , while our work aims to train interpretable models . Learning Disentangled Representation refers to learning data representation that encodes different semantic information into different dimensions . As a principle , it proves impossible to learn disentangled representation without inductive bias ( Locatello et al. , 2018 ) . Unsupervised methods such as variants of VAEs ( Kingma & Welling , 2014 ) and InfoGAN ( Chen et al. , 2016 ) rely on regularization . VAEs ( Kingma & Welling , 2014 ) are modified into many variants ( Higgins et al. , 2017 ; Burgess et al. , 2018 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ; Kumar et al. , 2017 ) , while their disentangling performance is sensitive to hyperparameters and random seeds . InfoGAN ( Chen et al. , 2016 ) learns disentangled hidden representations by maximizing the mutual information between input and generated images . Some other unsupervised methods rely on special network architectures including interpretable CNNs ( Zhang et al. , 2018b ) and CapsNet ( Sabour et al. , 2017 ) . As for supervised methods , Thomas et al . ( 2018 ) propose to disentangle with interaction with the environment ; Bouchacourt et al . ( 2018 ) apply weak supervision from grouping information , while our work applies weak supervision from classification labels . Model Pruning reduces structure redundancy and therefore is widely used for both model efficiency and better generalization , while our work uses it to alleviate filter ambiguity and improve interpretability . Many existing methods prune pretrained CNNs for better efficiency without significant deterioration in performance , including channel-wise pruning ( He et al. , 2017b ; Molchanov et al. , 2016 ; Liu et al. , 2017 ) and filter-wise pruning ( Luo et al. , 2017 ) . The methods mentioned imply that CNNs trained by standard training strategy tend to have redundant filters . Further research exploits redundant filters to improve generalization and suppress over-fitting by temporally pruning during training : Dense-Sparse-Dense ( Han et al. , 2016 ) encourages sparse weights by regularization and then recovers dense weights by removing the regularization ; RePr ( Prakash et al. , 2019 ) repeatedly prunes and reinitializes redundant filters to reduce filter overlap . 3 METHOD . Learning disentangled filters in CNNs alleviates filter ambiguity and meanwhile narrows the gap between human ’ s perception and CNN ’ s representations . In this section , we first present an ideal case of class-specific filters , which is a direction for our training , and then we elaborate on our method about how to train an interpretable network . 3.1 CLASS-SPECIFIC FILTERS . For CNNs in classification tasks , it is demonstrated that the filters in the last convolutional layer extract high-level features related to certain classes more or less ( Zeiler & Fergus , 2014 ) . We suppose the most ideal case , as shown in Figure 2 , is that each filter extracts features mainly from only one class , i.e . each filter is mainly relevant to one class . We call such filters ideally class-specific . Obviously , given the labeled class of an input , the classification performance changes little if we shut down the channels from the filters irrelevant to the class . This is the direction for our training method elaborated later . To have a rigorous definition of “ ideally class-specific ” , for a CNN , we use a matrixG ∈ { 0 , 1 } C×K to measure the relevance between filters and classes , where K is the number of filters , C is the number of classes , and each element Gkc represents the relevance between the k-th filter and c-th class , as shown in Figure 3 . The k-th filter extracts features mainly in the c-th class when and only when Gkc = 1 . Given the label y ∈ { 1 , 2 , ... , C } of an input , we can index a row Gy ∈ { 0 , 1 } K from the matrixG , which can be used as a gate multiplied to the feature maps to shut down those irrelevant channels . We call filters in a CNN as ideally class-specific filters , when the network ’ s prediction ỹG approaches that of original network ỹ after the feature maps from the last convolutional layer are multiplied by the gate Gy . 3.2 PROBLEM FORMULATION . In order to train a CNN towards disentangling filters to class-specific and meanwhile keep the original classification performance , as opposed to the standard training scheme that entangles filters , we introduce a Label-Sensitive Gate ( LSG ) path in addition to the standard path of forward propagation . In such a LSG path , some channels are shut down by multiplication with a learnable gate . This path ’ s classification performance is regarded as a regularization for disentanglement training . As shown in Figure 3 , let us denote θ as the network parameters . The network forward propagates in two paths : 1 ) the standard ( STD ) path predicting ỹθ , and 2 ) the LSG path with gate matrix G predicting ỹGθ where the learnable gate Gy for the input labeled with y is multiplied to the feature map before the linear layer . Initially , the elements in the gate matrix G for a CNN are unknown . Intuitively , we can search for them by exploring the binary space and find one solution that yields the best classification performance through LSG path , i.e. , to solve Φ0 ( θ ) = minG : Gkis one-hot CE ( y||ỹGθ ) , where the constraint on Gk applies to ∀k ∈ { 1 , 2 , ... , K } and Φ0 evaluates the performance of the CNN with filters disentangled . Therefore , it is natural to add Φ0 into training loss as a regularization that forces filters to be class-specific . Thus , we get the following formulation of the original problem to train a CNN towards ideally class-specific filters as min θ L0 ( θ ) = CE ( y||ỹθ ) + λΦ0 ( θ ) . ( 1 ) However , the problem is difficult to solve in practice . On the on hand , the assumption that each filter is ideally disentangled to extract only one class hardly holds , since it is usual for several classes to share one high-level feature in CNNs ; on the other hand , binary vectors in a non-continuous space are difficult to optimize with gradient descent . To address these two issues , we relax the constraints from two perspectives . First , we relax the one-hot vectorGk to sparse vector by L1 regularization ‖G‖1 where at least one element equals to 1 ( ∥∥Gk∥∥∞ = 1 ) . Since ||Gk||∞ = 1 ensures ‖G‖1 to be larger than K , we actually apply ∣∣ ‖G‖1− s∣∣ as the sparsity regularization term in order to make the norm of gate matrix close to a constant target norm s ≥ K. Second , we relax the binary vectorGk ∈ { 0 , 1 } C to a continuous vectorGk ∈ [ 0 , 1 ] C . Therefore Φ0 now becomes Φ ( θ ) = min G { CE ( y||ỹGθ ) + a ∣∣ ‖G‖1 − s∣∣ } s.t . ∥∥Gk∥∥∞ = 1 and Gkc ∈ [ 0 , 1 ] , ( 2 ) where a is a coefficient to balance classification and sparsity . Φ can be regarded as a metric of filter ambiguity , which means a CNN with higher class-specificity of filters will have a lower Φ ( contrary to common sense , here we expect Φ to be as small as possible ) . Replacing Φ0 in Equation 1 with Φ , we get an intermediate problem minθ ( CE ( y||ỹθ ) + λΦ ( θ ) ) . It is mathematically equivalent if we move minG within Φ to the left most ( see Appendix A for proof ) . Thus , we formulate a relaxed problem as min θ , G L ( θ , G ) = CE ( y||ỹθ ) + λ ( CE ( y||ỹGθ ) + a ∣∣ ‖G‖1 − s∣∣ ) , ( 3 ) s.t . ∥∥Gk∥∥∞ = 1 and Gkc ∈ [ 0 , 1 ] . The relaxed problem is easier to solve by jointly optimizing θ and G with gradient descent , compared to either the discrete optimization in the original problem , or the nested optimization in the intermediate problem . Solving the relaxed problem , we can obtain a CNN for classification with class-specific filters , where G precisely describes the correlation between filters and classes .

Summary: This paper proposes a method to improve the interpretability of a convolutional neural network (CNN). The main idea is to force the CNN filters to be class specific, i.e. to be associated to a specific class. This is accomplished by a gating function that enforces filters to be sparsely activated.  This would make the model more interpretable by allowing to check which filters/classes have been activated. Results are evaluated in terms of performance, sparsity of the filters and localization accuracy on CIFAR10.