Training Interpretable Convolutional Neural Networks towards Class-specific Filters
Authors: Haoyu Liang
Zhihao Ouyang
Hang Su
Yuyuan Zeng
Zihao He
Shu-tao Xia
Jun Zhu
Bo Zhang
Venue:  (2020)
Keywords: class-specific filters
interpretability
disentangled representation
filter ambiguity
gate

Abstract: Convolutional neural networks (CNNs) have often been treated as “black-box” and successfully used in a range of tasks. However, CNNs still suffer from the problem of filter ambiguity – an intricate many-to-many mapping relationship between filters and features, which undermines the models’ interpretability. To interpret CNNs, most existing works attempt to interpret a pre-trained model, while neglecting to reduce the filter ambiguity hidden behind. To this end, we propose a simple but effective strategy for training interpretable CNNs. Specifically, we propose a novel Label Sensitive Gate (LSG) structure to enable the model to learn disentangled filters in a supervised manner, in which redundant channels experience a periodical shutdown as flowing through a learnable gate varying with input labels. To reduce redundant filters during training, LSG is constrained with a sparsity regularization. In this way, such training strategy imposes each filter’s attention to just one or few classes, namely class-specific. Extensive experiments demonstrate the fabulous performance of our method in generating sparse and highly label- related representation of the input. Moreover, comparing to the standard training strategy, our model displays less redundancy and stronger interpretability.


Under review as a conference paper at ICLR 2020
TRAINING INTERPRETABLE CONVOLUTIONAL NEU- RAL NETWORKS TOWARDS CLASS-SPECIFIC FILTERS
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional neural networks (CNNs) have often been treated as "black-box" and successfully used in range of tasks. However, CNNs still suffer from the problem of filter ambiguity an intricate muny-lo-many correspondence relation- ship between filters and features, which undermines the models' interpretability. To interpret CNNs, most existing works attempt to interpret pre-trained model, while neglecting to reduce the filter ambiguity hidden behind. To this end, we propose simple but effective strategy for training interpretable CNNs. Specif- ically, we propose a novel Label Sensitive Gate (LSG) structure to enable the model to learn disentangled filters in supervised manner, in which redundant channels experience a periodical shutdown as flowing through a learnable gate varying with input labels. To reduce redundant filters during training, LSG is con- strained with sparsity regularization. In this way, such training strategy imposes each filter's attention just one or few classes, namely class-specific. Extensive experiments demonstrate the fabulous performance of our method in generating sparse and highly class-related representation of the input. Moreover, comparing to the standard training strategy, model displays less redundancy and stronger interpretability.
INTRODUCTION
Convolutional Neural Networks (CNNs) demonstrate extraordinary performance in various visual tasks (Krizhevsky et al. 2012 He et al. 2016 Girshick 2015 He et al. 2017a). However, the strong expressive power of CNNs still far from interpretable, which significantly limits applications that require humans' trust or interaction, e.g. self-driving and medical image analysis (Caruana cct al. 2015; Bojarski et al. 2017). In this paper, we argue that filter ambiguity is one of the most critical reasons that hampers the interpretability of CNNs. As matter of fact, previous studies has shown that 1) filters CNNs generally extract features mixture various semantic concepts, including objects, parts, scenes, textures, materials and colors (Zhang 2018|)))))) Bau al. 201 and that 2) there is also redundant overlap between features extracted by different filters (Prakash let al.) 2019). The intricate many-to-many correspondence relationship between filters and features so-called filter ambiguity as shown on the left of Figure [
Obviously, in high convolutional layers which might capture class-related feature, filter ambiguity contradicts our intention of an interpretable CNN, because it hinders humans from interpreting the concepts of filter (Zhang et 2018b), which has been shown an essential role in the visualiza- tion and analysis of networks (Olah et al., 2018) in human-machine collaborative systems (Zhang al. 2017a c). Moreover, the unnecessary overlap between features extracted by different filters leads under-utilization of a model's expressiveness (Prakash et al. (019)) Therefore, reducing filter ambiguity is critical to obtain better feature with better interpretabillity and less redundancy.
However, it is non-trivial to achieve such goal barricaded by substantial challenges. First, most interpretabillty-rlltee research simply focuses on post-hoc interpretation of filters (Szegedy et al., 2013 Bau et al. 2017), which manages to interpret the main semantic concepts captured by filter but fails to alleviate the filter ambiguity prevalent in pretrained models. Second, many existing works such VAEs* variants (Higgins al. 2017 Burgess et al. 2018 Kim & Mnih 2018 Chen al.) 2018 Kumar et al., (017) and InfoGAN (Chen et al. (0166) try to disentangle data representation and obtain better interpretability in unsupervised way. However, it is proved that unsupervised

---

Under review as a conference paper at ICLR 2020
activation maps
often activated by cat, dog out, ding plane, dog, ship
activation
often activated by
advantages in applications
convolationa filees 000G00000000
100000000
10p convolmiionll filees
maps
improving localization
plane
many-ko-manyy
class-related
<>
detecting adversarial samples
dog
representation
dog, ship cul. piane dog, cul, ship
ship
ambiguous filters
class spesi-se filters
normal CNN
our CNN
The motivation of learning class-specific filters. In a normal CNN, each filter corresponds to multiple classes, since it extracts a mixture of features from different classes et al. (20T8b), which is symptom of filter ambiguity. In contrast, we enforce each filter to correspond to one (or few) classes, namely to be class-specific, which brings better interpretability and class-related feature representation. Such representation not only facilitates understanding the inner logic of CNNs, but also improves different application tasks related to the mechanism of filters, which is verified by our exhaustive experiments.
Figure 1:
learning on disentangled features without inductive bias impossible (Locatello et al. 2018 which challenges the works above.
Considering the aforementioned challenges, we shed light on enforcing the one-class for one-filter relationship during training (instead of post-hoc) in a supervised manner only with classification la- bels. To this end, we propose a novel training strategy that coerces each filter into extracting features from only one or few classes for classification tasks, namely disentangling filters towards class- specific. Specifically, we design a Label Sensitive Gate (LSG) structure on the top of convolutional filters, which limits each filter's activation only to its specific input label(s). In our training pro- cess, we periodically insert LSG into the CNN and jointly minimize the classification cross-entropy and the sparsity of LSG, encourage speciicc filters.
to keep the model's 's performance classification and meanwhile
Experiments proved the LGS filter yield better representation for images hence leading promising applications. Our training method makes data representation sparse and highly correlated with the labeled class, which not only illustrates the alleviation of filter ambiguity but also enhances the interpretability of the network. The advantages of our method are concretely substantiated by sparser correlation between filters, sparser filter-class correlation, better explanation for misclassification and more precise localization of labeled objects.
Contributions The contributions of this work can be summarized as: (1) we propose novel training strategy for CNNs which forces each filter to extract features mainly from only one or a few classes; (2) moreover, we propose a metric to evaluate filter ambiguity, which can also be used as regularization in our training to encourage class-specific filters; (3) finally, our training enables filters to output sparse and class-related representation, which helps alleviate filter redundancy, improve interpretability of the model.
2 RELATED WORKS
Existing works related our work include post-hoc filter interpretation, learning disentangled rep- resentation and model pruning.
Post-hoc Interpretation for Filters is widely studied, which aims to interpret the patterns captured by filters in pretrained CNNs. Plenty of works visualize the pattern of neuron as an image, which the gradient (Zeiler & Fergus 2014 Mahendran & Vedaldi 2015 Simonyan et al. 2013) or aceu- mulated gradient (Mordvintsev et al. Olah et al. 20118) of certain score about the activation of the neuron. Bau et al. (2017) determine the main visual patterns extracted by a convolutional filter by treating it as pattern detector. Some other works transfer the representation in CNN into an explanatory graph (Zhang et al. 2017b; (a) or a decision tree (Zhang et al., 2019), which aim to figure out the visual patterns of filters and the relationship between co-activated patterns. Post- hoc filter interpretation helps to understand the main patterns of a filter but makes no change to the existing filter ambiguity of the pretrained models, while work aims to train interpretable models.
2

---

Under review as a conference paper at ICLR 2020
Learning Disentangled Representation refers to learning data representation that encodes differ- ent semantic information into different dimensions. As principle, it proves impossible to learn disentangled representation without inductive bias (Locatello et al.) 2018). Unsupervised methods such as variants of VAEs (Kingma & Welling 2014) and InfoGAN (Chen et al. (016) rely on regu- larization. VAEs (Kingma & 2014) are modified into many variants (Higgins et al. 2017] Burgess et al., 2018 Kim & Mnih 2018 Chen et al.] [2088] Kumar et al., 2017), while their dis- entangling performance is sensitive hyperparameters and random seeds. InfoGAN (Chen al. 2016) learns disentangled hidden representations by maximizing the mutual information between input and generated images. Some other unsupervised methods rely on special network architec- tures including interpretable CNNs (Zhang et al. 2018b) and CapsNet (Sabour et al. 2017). As for supervised methods, Thomas et al. (2018) propose to disentangle with interaction with the environ- ment: Boottacourtt et (2. 8) apply weak supervision from grouping information, while our work applics weak supervision from classification labels.
Model Pruning reduces structure redundancy and therefore is widely used for both model effi- ciency and better generalization, while our work uses it to alleviate filter ambiguity and improve interpretability. Many existing methods prune pretrained CNNs for better efficiency without signif- icant deterioration
performance, including channel-wise pruning (He et al.) 2017b] [Mocchanvv]
al. 2016 Liu et al., 2017) and filter-wise pruning (Luo et al. 2017). The methods mentioned im- ply that CNNs trained by standard training strategy tend to have redundant filters. Further research exploits redundant filters improve generalization and suppress over-fitting by temporally pruning during training: Penne-Spaase-Derte Dense (Han et al. 2016) encourages sparse weights by regulariza- tion and then recovers dense weights by removing the regularization; RePr (Prakash al. 2019) repeatedly prunes and reinitializes redundant filters to reduce filter overlap.
METHOD
Learning disentangled filters in CNNs alleviates filter ambiguity and meanwhile narrows the gap between human's perception and CNN's representations. In this section, we first present ideal case of class-specific filters, which is a direction for our training, and then we claborate on our method about how to train interpretable network.
channels from normal filter
channels from ideally class-specific filters
when inport belongs to "dog"
sbarod
ship
dog
cat
plane
ship
dog
cat
plane
The intuition of disentangling filters to class-specific. In a normal CNN, each filter extracts mixture of features from different classes (Zhang et al. (0188)) which is symptom of filter ambiguity. In contrast, when filters are ideally class-specific, each filter extracts features mainly from only one class. Given the labeled class of an input, the filters irrelevant to the class have weak activation, and hence classification performance changes little when they are shut down.
Figure 2:
3.1 CLASS-SPECIRE FILTERS
For CNNs in classification tasks, it is demonstrated that the filters in the last convolutional layer extract high-level features related to certain classes more less (Zeiler & Fergus 2014). We sup- pose the most ideal case, as shown in Figure 2 is that each filter extracts features mainly from only one class, i.e. each filter is mainly relevant one class. We call such filters ideally class- speciicc.... Obviously, given the labeled class of an input, the classification performance changes little if we shut down the channels from the filters irrelevant to the class. This the direction for our training method elaborated later.
To have a rigorous definition of "ideally class-specific", for a CNN, we use a matrix G E (0, 1] Cx to measure the relevance between filters and classes, where K is the number of filters, C' is the number of classes, and each element G represents the relevance between the k-th filter and c-th class, as shown in Figure The k-tth filter extracts features mainly in the c-th class when and only when Gk 1. Given the label y E {1, ..., C') of an input, we can index a row G, E (0, 1]* from the matrix G, which can be used a gate multiplied to the feature maps to shut down those irrelevant
3

---

Under review as a conference paper at ICLR 2020
channels. We call filters in a CNN ideally class- class-specific filters, when the network's 's prediction y approaches that of original network y after the feature maps from the last convolutional layer are multiplied by the gate G,-
prediction
prediction
lingar layer
channels
linear layer
0 0 1 1001
am
label
0000
0100
0100
0100
0.000
classes
1111
1110
0 .. 0 0.000
convolational layers
learnable label-sensitive gute (LSG) matrix
convolutional layers
(input label )
input
LSG Path
Standard Path
Figure 3: The framework of Label-Sensitive Gate (LSG) training. During LSG training, we al- ternately train CNN through both the LSG path and the standard (STD) path. In the STD path, network parameters are optimized to minimize the cross-entropy. In the LSG path, feature maps after the last convolutional layer pass through the learnable gate which is row vector in the LSG matrix indexed by the label of the input. Network parameters and LSG matrix are optimized to minimize the cross-entropy in conjunction with a sparsity regularization for the LSG matrix. When testing, we just run the STD path.
3.2 PROBLEM FORMULATION
In order to train a CNN towards disentangling filters to class class-speciff specific and meanwhile keep the origi- nal classification performance, as opposed to the standard training scheme that entangles filters, we introduce Label-Sensitive Gate (LSG) path in addition to the standard path of forward propaga- tion. such a I.SG path, some channels are shut down by multiplication with learnable gate. This path's 's classification performance is regarded as regularization for disentanglement training.
As shown in Figure 3 let us denote as the network parameters. The network forward propagates in two paths: 1) the standard (STD) path predicting De. and 2) the LSG path with gate matrix G predicting oh where the learnable gate G, for the input labeled with y is multiplied to the feature map before the linear layer.
Initially, the elements in the gate matrix G for a CNN are unknown. Intuitively, we can search for them by exploring the binary space and find one solution that yields the best classification perfor- mance through LSG path, i.e., to solve (f))) minG: one hot CH(yllu5)) where the constraint on G applies to vk E {1, ..., K) and oy evaluates the performance of the CNN with filters dis- entangled. Therefore, is natural add & into training loss as a regularization that forces filters to be class-specific. Thus, we get the following formulation of the original problem to train a CNN towards ideally class-specific filters
min Lo = CE ((0)) + (0))
However, the problem is difficult to solve in practice. On the on hand, the assumption that each filter ideally disentangled to extract only one class hardly holds, since it is usual for several classes to share one high-level feature in CNNs; the other hand, binary vectors in non-continuous space are difficult to optimize with gradient descent.
To address these two issues, relax the constraints from two perspectives. First, we relax the one-hot vector G to sparse vector by L1 regularization |G''], where at least one element equals to [| = 1). Since || = 1 ensures ||G]]] to be larger than K, we actually apply ||G]] - as the sparsity regularization term in order make the norm of gate matrix close constant target norm 2 K. Second, we relax the binary vector G* E {0, 1] to continuous vector G E [0, 1
Therefore O now becomes
((0) min (CE(( (D6) + al [|]]]
=/} s.t.
|G and G E 10, 1].
(2)
4

---

Under review as a conference paper at ICLR 2020
where a is coefficient to balance classification and sparsity. @ can be regarded metric of filter ambiguity, which means a CNN with higher class-specificity filters will have a lower (contrary to common sense, here expect & to be as small possible).
Replacing 2g in Equation with 2, we get an intermediate problem ming((E( (CE(y)lūa) (0)))) mathematically equivalent if we move ming within & to the left most (see Appendix for proof). Thus, we formulate a relaxed problem as
min L(0, G) CB(yllãa) + X(CB(vl (8) + [|]]]]]] s(), 8,6
s.t. ||*l| and G E [0. 1].
The relaxed problem easier solve by jointly optimizing 0 and G with gradient descent, com- pared to either the discrete optimization in the original problem, or the nested optimization in the intermediate problem. Solving the relaxed problem, we can obtain a CNN for classification with speciic filters, where G' precisely describes the correlation between filters and classes.
3.3 OPTIMIZATION
To solve the optimization problem formulated Equation 3 with gradient descent, we alternately optimize G and 0 to improve the classification performance of the model while ensuring the con- straints of sparsity on G, as is shown in Algorithm | In this scheme, we run the LSG path to update both G and 6 in some epochs of a period, and run the STD path to update in the other epochs. When G' is updated with gradient, G will be normalized by ||G* || to ensure || = 1,
and then clipped into range [0, 1].
After solving the relaxed problem, we can further tighten the constraint on LSG G E fo, c for stronger sparsity of G. In this work, we tighten into a simple and special case, where some filters are related to only one class (i.e. G is one-hot) for class-specific features, and the other filters are related to all classes (i.e. G is all-one) for class-sharing features. Specifically, if G has only one element G 2 (threshold h E (0, 1J) G3 will set to noth otherwise G& will be set to anl-one.
Algorithm LSG Training
for in epochs do for N in batches do
2:
3:
E%3 then
4:
T + prediction through the LSG path with G
5:
Cost + X(CE(vlūū)) + a ||| =I)
6:
G is updated using the gradient decent as G
0Cost
80
Each column Gk normalized as G G²
7:
8:
G + clip(G,O, G,0,
9:
else
10:
Úo + prediction through the STD path Cost + CB(vllue)
11:
12:
end if
13:
+ 0 #Cast 00
14:
end for
15:
end for
4 EXPERIMENT
In this section, we conduct five experiments. We first delve into LSG training from three aspects, so respectively study the effectiveness of LSG training, the class-specificity of filters and the cor- relation between filters train with LSG. After that, we demonstrate LSG's s application on improving
Another choice is naive scheme: in all epochs predict through both paths to directly calculate L(0, G) defined in equation [ and update i and G with gradients of it. We choose the alternating scheme hecause in our preliminary exploration shows hetter training stability and converging speed.
5

---

Under review as a conference paper at ICLR 2020
object localization and adversarial samples detection. In the following parts, we denote training method sabel-Sensitive Gate as LSG, the standard training as STD, and CNNs trained with them as LSG CNNs and STD CNNs, respectively.
Training setting We trained ResNet20s (He et al. 2016) PASCAL VOC 2010 (Everingham al.) on classification task with LSG/STD training.
2016) on CIFAR20 and ResNet152s (He et al,
ResNet20s are evaluated this 4.1
and 4.5 and 55s are evaluated in 4. and 4.4
For fair comparison, all the code implemented the Pytorch framework and tested on GTX 1080Ti GPUs. We report the validation performance on last epoch following common practice.
For ResNer20s, the default settings include: batch size is 256; the optimizer SGD with momentum of 0.9 (Sutskever et al. 2013); the initial learning rate is 0.1; and the total training epochs is 150. The ResNet152s are finetuned from model pretrained on ImageNet (Deng et al. 2009). Parameters are frozen except the top 2 bottleneck blocks, gate matrix and linear layers. The setting is: batch size is 32; the optimizer Adam (Kingma Ba 2014); the initial learning rate le-5 for STD path and le-3 for LSG path; the total training epochs is 150.
Beside, we preprocess PASCAL VOC to be classification dataset for training ResNet152s: we crop out images for the objects in 6 classes (bird, cat, dog, cow, horse and sheep) and resize the image to 128x 128; then randomly reassign 3644 objects for training and 1700 objects for testing. No segmentation label is used in training 52sNet152s.
4.1 EFFECTIVENESS OF LSG TRAINING
To begin with, we conduct experiments to demonstrate the effectiveness of our LSG training in learning sparse gate matrix and yielding low 4 that evaluates class-specificcis of filters.
Basic Quantitative Evaluation To compare the performance on classification and filter disen- tanglement of the STD CNN and the LSG CNN, calculate their test accuracy, cross entropy, L1 Norm ||G]|, and O as mentioned in Section 3.2, As shown in Table LSG CNN's even slightly out- performs STD CNN in test accuracy, and cross entropy of both is comparable, while LSG CNN has much lower LI Norm and &. The L1 Norm and the & indicate LSG learns class-specific filters that hence reduce filter ambiguity. These metrics quantitatively demonstrate LSG's s capability of learning a sparse gate matrix and disentangling filters without any sacrifice on classification accuracy.
Table I: Metrics of the STD CNN and the LSG CNN.
Dataset
Model
Training
Accuracy
Cross Entropy
1.1 Norm
+
LSG (Ours)
0.9062
6.9762
0.1742
0.2203
CIFAR-10
ResNet20
STD
0.9046
6.8657
0,4757
0.8234
LSG (Ours)
0.8506
0.0508
0.1996
0.0044
PASCAL VOC 2010
R5sNet1
STD
0.8429
0.0356
0.8488
0.2794
Visualization Label-Sensitive Gate Matrix To illustrate the relevance between classes and the
leamned filters described in the gate matrix, we visualize gate matrices in Figue + Subfigure (a) indicates that the LSG training yields a sparse LSG matrix where each filter is only related to one or few classes. Subfigure (b) comes from LSG training without the sparsity regularization to the gate matrix (L1 norm), from which we observe filter ambiguity where a filter would extract ambiguous features from multiple classes. Accordingly, LSG training effectively learn sparse gate matrix, and this characteristic originates from our sparsity regularization - the LI norm on the gate matrix.
(a) Gate Matrix LSG CNN using LI Regularization
(b) Gate Matrix of STD CNN LI Regularization
Class ID
Filter ID
Filter ID
Figure 4: Visualization of the gate matrices from LSG training w/o the sparsity regularization on the gate matrix. The x-axis is the filter from to 64, the y-axis is the class in CIFARIO from 0 to 9, and the color represents how much a filter is related to a class.

---

Under review as a conference paper at ICLR 2020
(42)
(121
plate
car
0.8
hind
cal
dear
horse
Indd
02
Kin
glaac Ca bes
CAT
bec
Figure 5: Classification confusion matrix for STD / LSG models when masking filters highly related the first class (al, bl), and the first and sixth classes (a2, b2).
4.2 CLASS-SPECIFIC FILTERS
To substantiate that LSG training learns class-specifis filters and reveal how it works, explore the LSG CNN with filters masked for certain class(es) from which we observe the similarity between feature vectors and gates. The experiments conducted the ResNet20 trained in 4.1
Masking Filters for Certain Class Although the sparse gate matrix implies filters are class- specific, further verify the LSG training learns class-specific filters, we remove the filters highly related to certain class(es) referencing the gate matrix (i.e. G greater than threshold), and then visualize the classification confusion matrices. As shown Figure 31 when filters highly related to "plane" are removed, to our surprise, the LSG CNN fails to recognize the first class "plane"; nev- ertheless, the STD CNN still manages recognize "plane". Analogously, when masking the filters highly related to the first and sixth classes, we observe similar phenomenon. This demonstrates that the LSG CNN, the features from certain class are almost extracted by the specific filters highly related to the class, while other filters extract little feature from the class. Therefore the filters trained to be class-specific under the encouragement of the sparse gate matrix.
Similarity Between Features and Gates This experiment explains how the LSG matrix encour- ages class-specific filters by analyzing the similarity between feature from each class and each row in the LSG matrix. In our models there is global average pooling (GAP) layer after the last convo- lutional layer, which yields feature vector denoted as a. We use inner production to measure the similarity Sy between the feature vectors ay of images in the class y and the e-th G, in the LSG matrix (see Appendix B for details). We calculate St over all true positive (TP) and false negative (FN) images respectively and obtain similarity matrices STP, SEN E CCC as shown in Figure
From the figure, we observe two phenomena and provide the following analysis. 1) TP similarity matrix is diagonally dominant significantly, which reveals how LSG works in disentangling filters to class-specific: LSG forces filters to yield feature vectors whose direction approaches that of the gate vector for its related class. 2) FN similarity matrix is far from diagonally dominant. Besides, two classes with shared features, such as car truck and ship & plane, have high similarity in the FN similarity matrix, which enlightens tends to be misclassified in LSG models. Thus, the mechanism of misclassification in the LSG CNN probably the features across classes extracted by the shared filters
that hard samples with ambiguous feature across classes
True Poethno
Takee Nervtse
tikk sip
0.15
0.13
<11g
41
plas
brd dea loce fng.
p0nr
bid
411
dec
0.10
0.15
0.09
fog
dog
0.66
Figure 6: The similarity between feature vectors from a class (y-axis) and a row in the LSG matrix
(x-axis), averaged over all TP/FN samples. We reorder classes in CIFAR10 for better visualization.
4.3 CORRELATION BETWEEN FILTERS
To study the filter correlation in LSG CNNs, we train a CNN with the LSG under a tighter constraint, so conveniently analyze the correlation between filters for different classes.
7

---

Under review as a conference paper at ICLR 2020
Constraint Tightening Before tightening the constraint on gate matrix G' E {0, 1) accord- ing to Section 3.3 we first conduct a statistic analysis on 20 different LSG models to figure out how many filters each class monopolizes. The result indicates that in CIFARIO, each class tends to monopolize 10% total filters (6 for ResNet20's last convolutional layer), and 10% from all filters shared by classes. Inspired by this phenomenon, we tighten the constraint accordingly by man- ually setting a fixed LSG matrix. In the fixed LSG matrix, for each row Ge there are 6 ones from 6 columns of one-hot Gt, and extra columns G* are all-one (see Appendix C for illustration). This provides a setup where each class monopolizes 6 filters and 4 extra filters are shared by all classes. Models trained with this constraint naturally inherits all features of previous LSG models, and moreover, has better disentangled filters.
Filter Orthogonality Analysis We evaluate the correlation between filters with the orthogonality of their weights. We train an AlexNet (Krizhevsky et al. 2012) and a ResNer20 with the fixed LSG matrix or with STD, and then calculate the correlation between filters in each models. The correlation between two filters is defined as the inner product of their normalized weights, visualized as matrices in Figure 7 In subfigures (a), (c) for STD models, the filters are randomly correlated with each other. On the contrary, subfigures (b), (d) for LSG models, the matrices are approximately block- diagona,,,,,,, which means the correlation between the filters is limited into several filter groups corresponding to classes. This implies that filters for the same class are highly correlated (non- orthogonal) for the co-occurrence of features extracted by them, while filters for different classes are almost uncorrelated (orthogonal) for the lack of co-occurrence.
Through intuitive reasoning, we can explain why LSG training encourages filters for different classes to become orthogonal. Given class and a gate matrix, that assigns the filter k for class and filter k' for other class. During training, filter k is closed (i.e. its activation is masked) in the LSG path when class c's images input. In order to ensure the STD path predict similar LSG path, the filter k' tends to be activated by class e as less as possible, which implies the weight of filter k' is approximately perpendicular to Ve, the linear space spanned by class c's features in a layer before. The filter & for class however, tends to be activated by class as saliently as possible so as to enable the CNN to recognize this class. So the weight of filter & is approximately within Ve- Overall, the weights of filter k and filter k' tends to be orthogonal
Filter Redundancy With further analysis based on filter correlation, we discover that LSG reduce the redundancy of filters. For each element di.j in the correlation matrix, we define that the i-th filter and j-th filter are correlated if az,, > s where $ is a varying threshold. We count the ratio of elements in the correlation matrix satisfying the constrain and plot the results in subfigure (e). It shows that LSG significantly reduces the redundant weak (e.g. threshold= 0.3) correlation between most filter pairs, which are mostly for different classes. We explain the reduction of filter redundancy natural consequence of encouraging filters (for different classes) to be orthogonal. For a set of filter groups orthogonal each other, a filter in any group can not be a linear representation with the filers from other groups This directly avoids redundant filter across groups, Besides, experiments in (Prakash et al. 2019) also verify the opinion that filter orthogonality reduces filter redundancy.
Bharati ofreeaaed who bolet warkes
STD AlciNe
LSC Aloahia (Oub)
Racherh
to
120 38c (e)
FL12200 30
10
FEE
82
035
66
(b)
(8)
(6)
Figure 7: The correlation (inner product) matrix of filters in AlexNet and ResNet20 trained with STD/LSG. (c) shows the ratio of each matrix' elements larger than a varying threshold.
Related Representation Based on filter correlation, we can even show that filters trained with LSG can yield highly class-related representation, namely the representation for an image tends to exactly correspond to its labeled class rather than to other classes. That's based on a reasoning that in the highly activated channel of representation, filters are less activated by other classes and less correlated to the filters for other classes. To verify this reasoning, we analyze the correlation between the filters highly activated by each class. First we pick out each class c's dominant m filters, denoted as group A, (class c has highest average activation on them; m = 25 for AlexNet; m = 6 for RestNer20). We define Inter-Class filter correlation the average correlation between
8

---

Under review as a conference paper at ICLR 2020
filters in different classes' group: Cic CXC-1)m² EcErge ExEA. ENEA, CK, where CR. the row & column k' of the filter correlation matrix in Figure 7 The results in Table 2 show that inter-class filter correlation in LSG is about half much as that in STD, both on AlexNet and ResNer20. This demonstrates that different classes tend to activate uncorrelated filters in LSG CNNs. As a result, the representations for different classes have less overlap. Thus we finally confirm representation from LSG CNN is highly class-related.
Table 2: Inter-Class Filter Correlation (Cic)
Model
STD AlexNet
LSG AlexNet (Ours)
STD ResNet20
LSG ResNer20 (Ours)
Cic
0.1224
0.0684
0.1648
0.1022
4.4 APPLICATION LOCALIZATION
In this subsection, we exploit our class-specific filters to improve the localization of CNNs. LSG training encourages top convolutional each filter focus fewer classes, which obviously implies its feature map can localize a class better. We conducts experiments below to verify it.
Localization method Resizing feature maps to input size is widely used method to determine the area of objects or visual concepts, which not only works in localization task without bounding box labels (Bau al., 2017). but also take important role in network visualization and under- standing the function of filters (Zhou et al. 2016).
We study LSG CNNs' performs on localizing object classes with the feature map from both single filter and all filters. For single filter, we bilinear interpolate its feature map and segment the region with values above the top 20% activation of the filter on the entire test dataset. For all filers, we sums up their feature maps with the weights of the linear connections between each channels and an output class] By bilinear interpolating the sum feature map to input size, we get a classification activation map (CAM) (Bau et al. 2017). We segment the region with values above the top 20% activation in a CAM. Thus we can get segmentation map for a class from feature maps.
Quantitative evaluation train ResNet1 to do classification preprocessed PASCAL VOC and use Avg (average intersection over union) and AP15 (average precision 15%) to evaluate their localization for each class and all classes. Higher metrics shows better localization. Especially, for localization with one filter, we report Avg-IoU and AP15 averaged over all filters. See Ap- pendix f for detailed definition of the metrics. The results for localization with or all filters shown in Table 3 For localization with one filter, most classes is localized better with LSG CNN. That's because filters trained with LSG tends be activated by the labeled class rather than many other classes, which alleviates other classes' interference on feature maps. Therefore, as a weighted sum of better one-filter feature maps, LSG CNN's CAMs also outperform LSG CNN's.
Table 3:
The performance of localization with resized feature maps in the LSG/STD CNN. For almost all classes, LSG CNN significantly outperforms STD CNN both on Avg-loU and AP15.
localized with
metric
training
bird
cat
dog
cow
horse
sheep
total
LSG
0.232
0.359
0.383
0.335
0.212
0.298
0.343
Avg-IoU
one filter's
STD
0.150
0.145
0.138
0.147
0.152
0.151
0.146
feature maps
LSG
0.689
0.939
0.961
0.923
0.650
0.852
0.918
AP15
STD
0.447
0.429
0.401
0.434
0.458
0.452
0.432
LSG
0.239
0.272
0.243
0.227
0.192
0.1.66
0.228
all filters' feature maps i.e. CAMs
Aog
STD
0.174
0.110
0.115
0.161
0.175
0.194
0.147
LSS
0.730
0.873
0.798
0.690
0.650
0.512
0.732
AP15
STD
0.505
0.280
0.307
0.503
0.607
0.630
0.438
Visualiziation Besides the quantitative evaluation above, in Figure X we also visualize sample some images and their CAMs from the STD/LSG CNN. We observe that the CAMs of STD CNN often activate extra or other semantic areas unrelated to the labeled class. However, LSG training
It only works for CNNs with global average pooling and one linear layer the last convolutional layer.

---

Under review as a conference paper at ICLR 2020
successfully helps the CNN find precise area of the labeled objects in image. Such
phenomenon vividly demonstrates that LSG training improves the performance in localization.
Cat
Dog
Bird
Car
Plane
Trunk
True Labels
STD CNN
Prodicted
LSG CNN Predicted (Ours)
Figure 8: Visualizing the localization STD CNN and LSG CNN with CAM (Zhou al. 2016).
4.5 APPLICATION ADVERSARIAL SAMPLE DETECTION
This subsection shows application of the highly class-related representation from our class- specific filters. It is studied (Wang et al. 2018) that adversarial samples can be detected based on the anomalous behavior of their representation on each layers: in low layers of neural network, the representation of an adversarial sample is similar to the original class, while on the high layers it similar the target class. With the highly class-related representation from our LSG training, will be much easier distinguish which class the high-layer representation is similar to. Therefore, we suppose this advantage will help in adversarial detection task.
To verify this judgment, train binary classifier with the features of normal samples and ad- versarial samples extracted by global average pooling after each convolution layers of ResNer20 trained in 4.1 As indicated in (Wang al. (0088). the non-targeted adversarial samples result in semantic- closer class with the original class, which are hard to discriminate. In this regard, we gen- erate targeted adversarial samples by commonly used PGD (Madry et al. (017)) attack with = 0.3 and iter = 40 and the adversarial target classes are from a random permutation of original classes besides each image's
class. In our experiments, the binary classifier. We randomly select 100, 500 and 1000 images for each class in CIFAR-10 dataset form different sizes of training datasets while the test data is collected by randomly se- lecting 100 images for each class. The experimental results are shown in Table We repeat each experiments five times and report the mean AUC scores. The experimental results demonstrate that the class-related representation can better distinguish the abnormal behavior of adversarial samples which can help improve the robustness of the model.
adopt random forest (Breiman) 2001) as
Table The mean Under-Curve (AUC) score for random forest on adversarial detection with features of STD CNN and LSG CNN. With the features of LSG CNN, we can achieve higher AUC score compared with using features STD CNN, which indicates that the highly class-specific representation makes it easier to distinguish the abnormal behaviour of adversarial samples.
Num. of training samples
100
500
1000
STD
72.86
80.77
83.64
LSG (Ours)
77.39
85.01
86.93
CONCLUSION
In this work, we propose a simply yet effective structure Label Sensitive Gate Matrix to disentangle the filters in CNNs. With reasonable assumptions about the behaviors of filters, we derive regular- ization terms to constrain the form of the gate matrix. As a result, the sparsity of the gate matrix encourages class-specific filters, and therefore yields sparse and highly class-related representations, which endows model with better interpretability. We believe LSG a promising architecture to dis- entangle filters in CNNs. Referring to LSG's successful utility and feasibility in classification prob- lem, we expect that LSG also have the potential to interpret other tasks like detection, segmentation etc, and networks more than CNNs, which is the direction of future work.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 6541-6549, 2017.
Mariusz Bojarski, Philip Yeres, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Lawrence Jackel, and Urs Muller. Explaining how deep neural network trained with end-to- end learning steers a car. arXiv preprint arXiv:/744.073 2017.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. Thirty-Second AAAI Con- ference Artificial Intelligence, 2018.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Christopher P Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in B-vae. arXiv preprint arXiv: 03599, 2018.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intel- ligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 1721-1730. ACM, 2015.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances Neural Information Processing Systems, pp- 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
J. Deng, W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. CVPR09, 2009.
M. Everingham. L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. http://www.paccal.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Peter Va- jda, Manohar Paluri, John Tran, et al. Dsd: Dense-sparse-dense training for deep neural networks. arXiv preprint arXiv:1607.0438/ 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017a.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017b.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. International Conference on Learning Representations. 2(5):
6, 2017.

---

Under review as a conference paper at ICLR 2020
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv: 1802.0588, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL
puttp://ariiva wwwpg//a//1...a.////
Diederik P Kingma and Max Welling. Stochastic gradient vb and the variational auto-encoder. In International Conference on Learning Representations, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00843 2017.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference Computer Vision, pp. 7736-2744, 2017.
Francesco Locatello, Stefan Bauer, Mario Lucie, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. arXiv preprint arXiv: 12335,, 2018
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: arXiv:1706.0608: 2017.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 5188-5196, 2015.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 2016.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks, 2015. URL https://researchh www..ebboggooooooooooooooooooo htt.b//oog..om/0/5//66./ inceptionisns -ileee--iinniegeeeeeeeeee dewpriggggeeee---------e------a wtt.:eeeraaaaa..ma...........m..
Nina Narodytska and Shiva Prasad Kasiviswanathan, Simple black-box adversarial perturbations for deep networks. arXiv preprint arXiv: 2016.
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 10.23915/ distil1.00010. ptnps://ditiill.ubbb2 Wbuilding-blocke
Aaditya Prakash, James Storer, Dinei Florencio, and Cha Zhang. Repr: Improved training of convo- lutional filters. Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 10666-10675, 2019.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint /3Xiv:/312.6034. 2013.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 2019.
12

---

Under review as a conference paper at ICLR 2020
Iya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning, pp- 1139-1147, 2013.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprini arXiv:/3/2.6199 2013.
Valentin Thomas, Emmanuel Bengio, William Fedus, Jules Pondard, Philippe Beaudoin, Hugo Larochelle, Joelle Pineau, Doina Precup, and Yoshua Bengio. Disentangling the independently controllable factors of variation by interacting with the world. arXiv preprint arXiv: arXiv:1802.09484, 2018.
Yulong Wang, Hang Su, Bo Zhang, and Xiaolin Hu. Interpret neural networks by identifying critical data routing paths. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8906-8914, 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818 088.833. Springer, 2014.
Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and Song-Chun Zhu. Mining object parts from cnns via active gresstionaansweingg In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 346-355, 2017a.
Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and Song-Chun Zhu. Growing interpretable part graphs convnets via multi-shot learning. In Thirty-Firsi AAAI Conference on Artificial Intelli- gence, 2017b.
Quanshi Zhang, Ruiming Cao, Shengming Zhang, Mark Redmonds, Ying Nian Wu, and Song- Chun Zhu. Interactively transferring enn patterns for part localization. arXiv preprint arXiv:1708.01730 2017c.
Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu, and Song- Chun Zhu. Interpreting enn knowledge via an explanatory graph. In Thirty-Secone Second AAAI Conference on Artificial Intelligence, 8a.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8827- 8836, 2018b.
Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. Interpreting cnns via decision trees. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6261 6270, 2019.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
13

---

Under review as a conference paper at ICLR 2020
A
EQUIVALENCE PROOF
This section is supplementary proof for the equivalence (mentioned in 3.2 between
min lãe) + A)()))
"
((0) min CH(yl / + a ||[| s| s.l. || 1, G E [0, 1],
and
min L(8, CE(yll)e) X(CE(vllb)) + + - =s) s.t. || 1, G [0.1]. 8.G
This equivalence is true referencing the Lemma below, if set X in the Lemma as 0 E R MM y E Y as G E {G E [0, 1](xK]_ f()) CE(yllūo). and g(z, y) CH(yl (g)) |||G]|, s|. Here, M the number of parameters in CNN, C' is number of classes, K the number of filters. is the label of input, de is the prediction of the CNN's prediction through the STD path, and y is the prediction of the CNN's prediction through the LSG path with G.
Lemma Given sets X, Y, and functions f: bounds, then
X.g:XXYN R. f. f and have lower
(/(x)) [f(a)) (x
min
g( v) min min g( v)]
=ex
yEY
Proof.
Given Va E fz) is constant if we take as the only variable,
(w) ming min(f(x) g(x,,)))
yEY
Denote F(x,y) [(x) + g(x, y), thus to prove equation 4 we only need to prove
min (x
F(I, min min F(x, y)
This is obvious, because for Vx E X,UEY,
min (a.y)eXxY
F(r,y) & F(z.y).
so for Vx E X
min (xx)
F(x.y) S min F(z,y), yEY
and then
min (x,))xx
F(r,y) S min min F(z,y). zEX 9EX
Reversely, for Vx E X. Y, we have
min min F(x, v) S F(x,)),
rex #eX geY
SO
min min F(z v) o
min
F(z,y).
zEX yEY (xx)exxx
14

---

Under review as a conference paper at ICLR 2020
B SIMILARITY BETWEEN FEATURE VECTORS AND GATES
We design similarity between the feature vector dy of image in class y and the e-th row Ge the LSG matrix. Noticing a, Ge R we use their inner production az G. as a similarity in direction. Thus, each pair of image in class y and class e will contribute an inner production oy Ge to St the average similarity between the feature vectors for class y and the e-th gate row. Thus, can further define the average similarity between the feature vectors for class y and the c-th gate row,
SD Dest meana Ge (x,z) D.
Here, D is a dataset to calculate the average on, and (xt y) a pair of image and label in the dataset. We take D as all positive or all false negative data, conducted in 4.2
C MANUALLY FIXED GATE MATRIX
In we use manually fixed gate matrix to train CNNs from scratch. The gate matrix is visualized Figure
1.0
Class ID
0.0
10
20
30
50
60
Filter
Figure 9: Manual fixed gate matrix. Each class monopolizes 6 filters and 4 extra filters are shared by all classes.
D CLUSTER CENTER EXPERIMENTS
Using the model with fixed LSG mentioned in 4.3 we run k-means clustering the feature vectors after the global average pooling in the STD CNN and the LSG CNN. The dataset we used is CIFAR- 10. We find the LSG CNN yields better clustering centers, which is almost the same the gate matrix we used (visualized in Figure with filter groups reordered.
LSS CNN (Ours)
STD CNN
S-
10
20
40
10
20
40 50
20
6.0
8.0
10.0
0.5 1.0 1.5 2.0
Figure 10: Cluster center experiment. The x-axis is the channel id, and the y-axis is class id. Each row is the mean of cluster in the feature vectors' space. and the color represents the value of an element in the mean.
E METRICS FOR LOCALIZATION
This section gives detailed definition of the metrics used in localization experiments.
15

---

Under review as a conference paper at ICLR 2020
E.1 LOCALIZATION WITH ONE FILTER
For an image æ in class e (denoted as a C De E D. D is the dataset), we denote the ground-truth segmentation map for x Sz, and the segmentation map given by the resized feature map from filter k as S The filter [1, K], where K is the number of filers the top convolutional layer.
The IoU (intersection over union) on this image is defined as
Sx Sx
IOU =
Sh U Sk
The Avg-IoU (average intersection over union) for filter lo to localize class is defined
Io no,
The AP15 (average precision 15%) for filter k to localize class i defined
AP15 meanzzp,,((loUg) 0.15).
When arg maxe AP15 we call filter k is for localizing class c, denoted as k E Fe, where F the set of filters for localizing class The performance for filter & is evaluated with IoU IoU and AP15* AP15
Thus we can define the Avg-IoU and AP15 for localizing class as
ToU, = meanker. ToU
and
AP15, m AP15
Besides, the Avg-IoU and AP15 for localizing all classes is defined as
IoU = mcanke(1,x]IoU]
and
AP15 mww. ke
E.2 LOCALIZATION WITH ALL FILTERS
For image in class (denoted as De), denote the ground -ruth segmentation map for x as Sz, and the segmentation map given by the resized classification activation map (CAM) (Bau et al. 2017) as Sz.
The loU (intersection union) this image is defined
S4 n
loU
S S
The Avg-IoU (average intersection over union) for localizing class c is defined as
IoU_- D, IoU
The AP15 (average precision 15%) for localizing class is defined
AP15. = meangen,T[Tots > 0.15).
Besides, the Avg-IoU and AP15 for localizing all classes is defined as
IoU mean
and
AP15 mean D(loo)) > 0.15).
16

---

Under review as a conference paper at ICLR 2020
F DEFENDING ADVERSARIAL SAMPLES
In this experiments, inspired by the highly class-related representation, we further explore LSG CNNs' potential in robustness for adversarial attacks. Two black box attacks are conducted, includ- ing one pixel attack (Su et al. 2019)) and local search attack (Narodytska & Kasiviswanathan 2016) They try to fool models according the model's predicted probability without access the models' parameters and architectures. From the results shown in Table we find both the attacks gain attack success rates on STD CNN much higher than on LSG CNN. This demonstrates that LSG training also improves robustness to CNNs. We guess the robustness is caused by the increase of within-class distance and the decrease of hetween-class distance, which requires further verification. Robustness another valuable characteristic of the highly class-related representation from the class-specific filters.
Table 5: Black Box Attack STD CNN and LSG CNN
Attack
Metric
STD CNN
LSG CNN
No Attack
Accuracy
88.03%
88.85%
Single Pixel Attack
14.00%
2.00%
Attack Success Rates
Local Search Attack
15.00%
2.00%
17


Reviews:
[{"review_id": "Syx3XXYAtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to improve the interpretability of a convolutional neural network (CNN). The main idea is to force the CNN filters to be class specific, i.e. to be associated to a specific class. This is accomplished by a gating function that enforces filters to be sparsely activated.  This would make the model more interpretable by allowing to check which filters/classes have been activated. Results are evaluated in terms of performance, sparsity of the filters and localization accuracy on CIFAR10.\n\nI lean to reject this paper because I consider the proposed motivations not clear and partially misleading. In the introduction it seems that the idea of enforcing class-specific filters makes sense in general because it avoids filters ambiguity and it reduces redundancy (not clear how). Instead, in the actual implementation, this idea is applied only at the last layer of a CNN. This makes sense, because in a CNN filters need to be shared among classes. It’s an important form of regularization, especially when the amount of training data is reduced. \nAdditionally, the advantage of enforcing the last layer filters to be class-specific is not clear. To me, instead of evaluating the activation of the classification layer, it is possible to check the activation of the filters of the last convolutional layers. However not much more interpretability is added. Furthermore, the improvement on localization as shown in Fig. 6 is only qualitative as the images could have been cherry-picked and there is no real localization measure.\n\nAdditional comments:\nIn table 1, performance of the proposed training is comparable to a standard training on CIFAR10. However, evaluating the proposed approach on a single dataset and only one network is not enough for a clear evidence. Additionally, the obtained performance is below the standard performance of ResNet on CIFAR10 without a clear reason.\nAnother way to enforce a similar pattern on the gating function would be by maximizing the mutual information between selected filters and classes.  \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eMnQEpFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an interesting idea of using Label Sensitive Gate (LSG) structure to enforce models to learn disentangled filters for better interpretability of the DNN model. By periodically training with the sparse LSG structure, the model is forced to extract features from only a few classes. The model is trained efficiently in an alternate fashion (with respect to both the network parameters and the sparse gate matrix.) By disentangling the class-specific filters, the model becomes less redundant and more interpretable.\n\nOverall the paper is well-written and well-organized. I  like the idea of imposing a class-specific gated structure for disentangling the representation. And numerical experiments verify the effectiveness of the proposed method in terms of\n1) Improved performance\n2) Disentangled representation (a small L1 norm on the gate matrix G)\n3) Consistent class activation map for different inputs.\n\nI do have some question though\n1) Table 1 also reports the L1 norm and \\Phi of a STD CNN. What is the gated matrix G here for a STD CNN?\n2) Is it very important to have a constraint $\\|G^k\\|_{\\infty}$ in the model? What if an L-1 norm is used directly?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgVZnT2YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions: The paper proposes a novel Label Sensitive Gate (LSG) structure to enable the model to learn disentangled filters in a supervised manner. The novelty of the paper is to introduce the Label-Sensitive Gate path during the training, on top of the standard training path. This encourages the filters to extract class-sensitive features. \n\nThe highlight of the experimental results is that by incorporating the LSG structure, the author is able to achieve better localization with CAM because the filters are trained to extract more relevant features. Additionally, filters from different classes are kind of \"orthogonal\" to each other.\n\nHowever, I have a few questions for the author:\n\na) In Figure 5, the author mentions \"TP/FN\" samples in the caption. However, for the plot at the right, it is titled with \"False Positive\". Is this a typo or do you talk about something else? \n\nb) In Figure 7, the author shows the correlation matrix of filters. Since the filters belonging to one class are highly correlated (block matrices in plot d), why do not train even less filters for each class? If we train less filters for each class, will that hurt the performance? It might be a good direction on how to interpret filters from one class and how to make intra-class filters more interpretable.\n\nOverall, the novelty of the paper is limited by the scope of the experimental results. Why would people care about interpretability of CNN filters is not explained clearly. The paper shows interpretable filters could help with localization. However, this is the only example in the paper I find useful. It would be better if the author could show any other applications. Moreover, I wish the author could spend more paragraph discussing why filters belonging to different classes tend to be orthogonal to each other. This is not clearly written (Eq(3)?). A more theoretical/intuitive explanation would also make the paper stronger as well.\n\nIf the paper could omtroduce more real applications of the proposed LSG model and give more theoretical/intuitive explanations of why the inter-class filters are orthogonal, I would be willing to raise up my scores.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to make the filters of the last conv layer more class-specific. The motivation for this is to improve upon the interpretability of the CNN, which is empirically shown by comparing the class activation maps (CAMs) of regular CNN and the proposed LSG-CNN. While the idea is interesting, one of the concerns from reviewers is about limited applicability of the method, at least the way it is shown in experiments -- a concern that I tend to agree with. As primary goal of the work is improving interpretability of CNNs, authors should test LSG-CNN with some more recent methods for producing the saliency maps other than CAM to convincingly establish the value of the method. Authors also mention lack of hyperparameter tuning and the use of SGD with limited training epochs as a reason for the drop in accuracy. It will be worth spending some effort so the accuracy matches the standard benchmarks -- this will help in arguing more convincingly about practical benefit of the method. 
Decision: Reject