Semantic Pruning for Single Class Interpretability
Authors: Kamila Abdiyeva
Martin Lukac
Kanat Alimanov
Venue:  (2020)
Keywords: deep learning
semantic pruning
filter correlation

Abstract: Convolutional Neural Networks (CNN) have achieved state-of-the-art performance in different computer vision tasks, but at a price of being computationally and power intensive. At the same time, only a few attempts were made toward a deeper understanding of CNNs. In this work, we propose to use semantic pruning technique toward not only CNN optimization but also as a way toward getting some insight information on convolutional filters correlation and interference. We start with a pre-trained network and prune it until it behaves as a single class classifier for a selected class. Unlike the more traditional approaches which apply retraining to the pruned CNN, the proposed semantic pruning  does not use retraining. Conducted experiments showed that a) for each class there is a pruning ration which allows removing filters with either an increase or no loss of classification accuracy, b) pruning can improve the interference between filters used for classification of different classes c) effect between classification accuracy and correlation between pruned filters groups specific for different classes. 

Under review as a conference paper at ICLR 2020
SEMANTIC PRUNING FOR SINGLE CLASS INTER- PRETABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional Neural Networks (CNN) have achieved state-of-the-art perfor- mance different computer vision tasks, but price of being computationally and power intensive. At the same time, only few attempts were made toward a deeper understanding of CNNs. In this work, we propose to use semantic prun- ing technique toward not only CNN optimization but also as way toward getting some insight information on convolutional filters correlation and interference. We start with pre-trained network and prune it until it behaves as single class clas- sifier for a selected class. Unlike the more traditional approaches which apply retraining to the pruned CNN, the proposed semantic pruning does not use re- training. Conducted experiments showed that a) for each class there a pruning ration which allows removing filters with either an increase no loss of clas- sification accuracy, b) pruning can improve the interference between filters used for classification of different classes c) effect between classification accuracy and correlation between pruned filters groups specific for different classes.
INTRODUCTION
Convolutional Neural Networks (CNN) provides high quality features that are used in vari-
ous tasks such as object recognition (Ouvang al. 2015: Wang et all 2015: Lt al. 2017a
Hu et all (001)). scene classification (Zhou et al_ 2014 Ren & Li, 2015). semantic segmen-
tation Girshick et al. (2014); Husain et all (017)) image captioning (Anderson et al.. 2017: Rennic et al. 2017: Yao et al. 2017 Lu et al. 2017). raw audio generation (van den Oord et al. 2016), data analytics (Najafahadi et al.. 2015). Unlike engineered features, where one or multiple features have been designed with specific domain knowledge (SIFT (Lowe, 1999). ,SURF (Bay et all, 2008), LBP (Ojala et all (1944), Canny (Cannyl 1986), etc.), features in CNN are obtained by di- rectly learning representations (filters) from the data without explicit domain expertise.
While using large number of learned filters in CNNs creates high-quality features, it also re- quires significant computational resources. The filter learning and the inference processes need resources such as GPU or TPUs to tackle the computational overhead. Recently, multiple opti- mization methods were proposed, with the target to reduce the computational overhead of either the learning or the inference processes. For instance, in the learning optimization approach, the following works have been done (Zhang et al., 2018 [in al., (2077)). On the other hand, the number of the current works targeting the optimization of the inference are: reduced precision computation (Rastegari al. 2016, Courbariaux et all. 2016 Zhou et al.. 2016), look-up and pre- computing (Abdiveva et al. 2018 Brendel & Bethge, 2019), pruning (Anwar et all, 2015 Li et al., 2016: Zhu & Guptal 2017 Raghu et al., 2017: Morcos al. 2018: Ma et al. 2019) and knowledge distillation (Hinton et al. 2015).
The work described in this paper continues the direction of inference optimization by pruning. However, the pruning is only used as tool for studying the interpretability of CNNs filter. Pruning
the process removing redundant filters from the network (Anwar et al. 2015: Li et al. 2016
Zhang et all 2 Zylberberg, 2017 7 Zhu & Guptal 2017 Raghu et all 2017 Morcos et all 2018
et 2019). In current approaches, the pruning process consists of the following steps: train CNN. remove redundant filters and retrains the network to preserve the original accuracy. The pro-
posed pruning method, however, does not involve retraining of the pruned network. The motivation of this work is the interpretation of CNNs: pruned and unpruned filters analyzed evaluate

---

Under review as a conference paper at ICLR 2020
class-wise relations and dependencies. In particular, the pruning applied to AlexNet is used to plore and study whether one can obtain a new single class classifier by dropping the selected filters in the original multi-class classifier instead of expensive training process of new classifier.
The interpretability of the network resulting from pruning can seen as a filter-wise and signal- wise network dissection: The removal of filters in controlled manner allows to determine how different classified objects affects other classes, how the density the classes the learned feature space affects the accuracy of classification and how the structure of the learned multi-class feature space allows to improve the accuracy of a single-class classification. Therefore, the pruning used in this work can also be seen an approach CNN understanding (Bau al. 2017: Zylberberg 2017: Kindermans et al. 2017: Morcos et al. 2018).
The proposed method, referred as semantic pruning, performed as follows: we start from CNN trained to distinguish classes. Then we reduce the multi-class CNN to the single class classification problem, by removing unrelated to the selected class labels. To remove filters, we measure each filter's contribution (activation magnitude) toward recognition of target object class. Filters with the highest contributions were preserved, and the remaining filters were dropped from the network. The pruned and unpruned filters are analyzed for correlation, overlap and other measures allowing us to determine class-wizs wise interactions.
The contribution of this paper can be summarized the following:
investigated the sensitivity of object classification to semantic pruning,
study of the interference between class wise filters in CNN,
This paper is organized as follows. Section 2 introduces semantic pruning concept. Section deseribes the experiments and the results and Section concludes the findings.
PROPOSED SEMANTIC PRUNING
Let F; RMXMXD be filter in layer of convolutional network. Let, B, E XXXXXD be an input tensor to the layer j. X and Y are spatial dimensions. To calculate the output tensor B of the layer j input tensor By convolved with the set of filters L {F ..., Fpr } of layer j. Lets denote the output of the convolution operation between input tensor B and a single filter F; as R₁
R = F. B
(1)
Then B Bi+1
Rs
Input Tensor
Filters
Output Tensor
Input Tonsor
Filters
h
Fyr
R
XxYx
Bj
M MaMrD
Lj
Bys
FxFxi
Figure 1: (a) Convolutional layer operation and (b) computation of the filter Activation Magnitude.
Definition 1 (Pruning). Let (Fi..... FD! } be a convolutional layer, F MxXD be single filter of the layer with an input tensor B, E R XXXxX
B B = L + B (eq. 7 with
Bj+1 R x Pruning layer L denoted (2TL.) is the process of removing F. from resulting in a new set of filters Q(I)) = L' (FI,... FDr), Hence, a new output is B B/41 = L' ② By with B' 3+1 E XxXXXX xY'x x1=" such that D" < D'.
In this work, we propose a semantic pruning method that is applied with respect to each selected class separately. We determine the amount of pruned filters for each object class X; A
2

---

Under review as a conference paper at ICLR 2020
fAI....,A,, } where q is the number of classes in the dataset. To determine which filters can be removed, the process of pruning requires a measure of contribution. In this work, we propose to use activation magnitude filter removal criteria.
Definition (Activation Magnitude r). Let E XXXXX be a result of convolution between imput lensor and single filter in the layer, then activation magnitude for each filter F; is computed as follows:
X
Y
F2
R
(2)
gam
with F E RIIIDD
The schema for calculating F is shown in Figure Tb
We propose to perform pruning for each class separately; hence, we split the dataset into subsets of images belonging to the same class. Let ID {d....., 6n) be the dataset containing the n data samples. Let A {AI..... Àg be the number of object classes in the dataset. Then, Xx_ C II be a subset of containing all data samples with class label Xe. The number of samples in xx, is denoted as p, such that p S n. Let Fi (6.)) be the activation magnitude of a filter F; resulting from using input sample 6, x We want to measure the response each filter w.r.t to the individual class. Therefore, we propose to accumulate the activation magnituees of all samples belonging to the selected class. The resultant value is denoted as Accumulated Response.
Definition 3 (Accumulated Response 7). Te is the Accumulated Response for filter F; and object class de and defined as shown in eq.
7
Fi(6))
(3)
The process of semantic pruning starts by separating the dataset ID into subsets Dx, {oi..... 6p3
with p being the number of samples in the subset Xx,, . for object label Ae E : A₂ with being the number of object classes. Let be the total number of filters in AlexNet. We calculate the accumulated response y^: { ..... pd: } for each filter in the network and for selected number of object classes from the dataset. The percentage of pruned filters is represented by the parameter 0 (pruning ratio) such that 00, 10..., 1]. For instance, for 0 0.1, the 10% of filters with lowest accumulated response will be pruned. In this work, let F, and Fa be the set of unpruned and pruned filters for a given y respectively. The process of filter pruning can be performed using two different approaches: network wise and layer-wise. Network wise pruning (NWP) is performed on filters from all layers of the network. Filters are sorted according their corresponding accumulated responses, and the filters with the smallest values are removed in correspondence to the pruning ratio O. On the other hand, Layer wise pruning (L.P) separates filters into groups, each containing filters from one layer. The pruning is then applied to each layer separately. However, as result of removing certain filters, the activation response for the filters in the consecutive layers can get affected, and therefore, higher potential of pruning can be achieved. Therefore, we also provided comparison between these two cases, referred to as Direct and Residual Pruning.
Two settings were used toward parying-parameter 0. These approaches will be referred to discrete and continuous pruning. For discrete pruning, the values are changes with the fixed step size, while for continues pruning the wider range of values is explored.
3 EXPERIMENTS
To illustrate the effectiveness of the semantic pruning approach, we applied the method on the model trained for an object classification task. In our experiments we used AlexNet (Krizhevsky et all 2012) trained on ImageNet (Russakovsky al. 2015) dataset. The AlexNet network consists of five convolutional and three fully connected layers. Fully connected layers are referred to as classifier. The convolutional layers are referred to feature extractor.
The ImageNet data set contains 1000 object classes; however, due to computational constraints, we evaluated method randomly chosen 50 classes, to show the further potential of the method.
3

---

Under review as a conference paper at ICLR 2020
Average accuracy drop vs Pruning Ratio
AlexNet
LWWP-33 TWP-M
classification accuracy
NWP.66 BR
0.0-
LWP-66
accuracy
67
0.
object classes
Pruning ratio
Figure 2: AlexNet average accuracy for differ- ent pruning rations
Figure Comparison of classification accuracy between unpruned AlexNet and pruned using NWP-M, LWP-33, NWP-66, LWP-66.
The experimental section is organized as follows:
Sensitivity to pruning
Study of the Filter Interference
3.1 SENSITIVITY TO PRUNING
We started our study by firstly evaluating the sensitivity of the single class classifier for pruning. In particular, we started with an evaluation of the discrete pruning approach. We applied both layer-wise (LWP) and cetwork-wise pruning(NWP).
3.1.1 DISCRETE PRUNING
The process of semantic pruning starts by separating the dataset into subsets of images belonging to the selected object label. Let m be the total number of filters in AlexNet. We calculated accumulated responses for all m filters in the network. Then we sorted the values in the descending order and applied four diffeerntt settings access the effect of discrete pruning:
0 score removing all filters with S 0
0.33 - removing 33% of Filters with lowest value of y
0.66 removing 66% of Filters with lowest value of
median (M)- Filters with y value lower than the median of all y were removed.
The procedure was repeated for all selected 50 classes separately.
showss the effect discrete pruning on the number parameters removed and preserved. Experiments indicated that pruning filters with NWP-O and NWP-33 affect the average network ac- curacy by around 10% (Figure while layer-wise pruning looks to be more resistant toward higher ratios. However, both the LWP-M and NWP-660 WWWP-66LWPP46 result in a serious drop of network accuracy (green, red and purple lines Figure 3. and the difference between layer-wise and network-wiss pruning are not as large as expected (red and purple lines).
TigureZalso shows that removing even the least active layer-wise filters reduces the general network accuracy. This can be seen when comparing NWP-33 and LWP-33 in Figure 2 and respectively. However, it can also be observed that the classification of certain classes becomes more accurate while other classes suffer much steeper loss in classification accuracy for LWP-33 (FigureByeelovr curve).
Finally, note that normalization of activity measure used. The reason for not using normal- ization is due to the fact that we are evaluating pruning with respect to one class at a time. There- fore, normalizing the activity with respect the whole data set would result in sample dependent
4

---

Under review as a conference paper at ICLR 2020
pruning and therefore remove the desired class wise pruning result. Additionally, normalizing with respect to one class at a time will introduce different minimal and maximal filter activities making magnitude--ased comparison between classes meaningless. Therefore, simply taking into account the NWP or LWP allows to account for both these effects and still provide reasonable information about the pruned and unpruned filters.
Figure shows how the pruning affects the individual layers of filters in the feature extractor. Fig- ure 4a shows the relative amount of pruned filters at ratio of 5 5%, Figure 4b at 25% and Figure 50%. The r axis indicates the layer and y axis the mount of filters.
Layersise Pruning
"thunder
Pruning "thunder
auights
1d0 priain
Totafael praitee
40000
Layer
Layer
Layer
(a)
(b)
(c)
Figure 4: The relative amount of filter pruned by layer and by pruning ratio (a) 0 = 0.05, (b)
0.25, (c) 0.5
In the Direct Pruning (DP) represents the number of filters removed by the semantic pruning method. The Residual Pruning (RP) represents the pruning resulting from the DP pruning. The RP pruning represents the filters that became inactive on average because their input becomes 0 on average after the DP pruning. Observe that similarly to the previous experiment, filters in 0th layer do not get pruned at all. Similarly, filters in the 4th layer do not get pruned as well at low 0. Filters from the 1 and 2nd layer the one most pruned in DP while the filters of the 3rd layer are pruned consequence in RP. Additionally, notice that at 50% of pruning ration almost all filters from the 2nd layer are removed as a result of pruning. It is very interesting to see that even with higher pruning ratios, the 0th layer is not pruned at all and the 4th layer is pruned only higher values of This means that the encoding provided by the first and last layer seems to be the most crucial and the densest.
3.1.2 CONTINUOUS PRUNING
The experiments with discrete pruning showed some important information about structural rela- tionships of filters and the accuracy of classification certain classes. However this information only partial complete and therefore continuous process of pruning spect to individual class accuracies. In the continuous pruning, was set to and was increased by 0.1 until reaching 0 0.6 (representing 60% pruning ratio). Figure 5 show the plot of class-wise accuracy for twelve selected object labels. It can be noticed that:
also evaluated with
Several classes are not affected by the pruning unless very large ratio (0 2 0.15) of filters is pruned (Figure 55e 5h).
Additionally, several classified classes has its accuracy increased after pruning. This is quite remarkable as this also entails that there can be interference between filters. (Fig-
ure 5d
3.2 STUDY OF THE INTERFERENCE
Semantic pruning showed that pruning least active filters from class result in increased accuracy of classification. The possible explanation for the phenomena is filters interference within overcrowded region of the feature space.
To investigate these phenomena in more details, we decided to investigate how many different and shared filters pruned, and how the pruning affected the class density within the feature space.
5

---

Under review as a conference paper at ICLR 2020
(a) switch
(b) sliding door
(c) jigsaw puzzle
(d) bullet proof vest
(c) apron
(f) badger
(g) Christmas stocking
(h) jeep
400
(i) cardigan
(i) maraca
(k) lighter
(l) ringneck snake
Figure 5: Examples of (a)-(d) object classes that improved for pruning ratio 0 >> (e)-(h) that for certain ratio 0 0 have same accuracy of classification as when 0 = O and (i)-(1) object classes for which classification accuracy was always worse than when classifying when 0 x-axis 0, y-axis accuracy.
In order to determine the overlap between object classes, inter-class feature overlap was computed by calculating the intersection of unpruned F. and pruned F, sets of filters. Let two object classes Aı and A2 be represented by F F and F>a F respectively. Then let Pr F n F and Por F FAx be double measure filter similarity. The reason for using both F, and F is to utilize two sources of information. In other words, we want to determine similarities of features required by the object class. Higher is the value of F.
which are required, and what features higher the similarity of filters used for classification of λ₁ and X2-
Figure provides the correlation between pruned and un-pruned filters for the three classes. (Only three classes were selected due to the space limitations.) All three classes represent different be- haviour toward pruning: performance improvement, pruning resistant and accuracy degradation.
The general trend for all three graphs for non-pruned filters is that the correlation between filters is constantly decreasing, as one can expect. More interesting trends, however, be observed for the correlation between pruned filters.
The first object class we would look at is jigsaw puzzle. According to Figure there is significant effect of the pruning on the classification accuracy of the jigsaw puzzle class. Possible explanation can be provided by the fact that the classes-neighlee neighbouss (Figure 6a| and Figure (d) for this class are semantically unrelated. Hence, the reduction in a correlation of pruned filters is reducing the interference with class-neighbours filters (0 E {0.075, 0.2}), which match the region of the sudden jump in accuracy according to Figure [5c] It is also can be noticed that for 0 0.3 semantically unrelated classes started to have increased in pruned-filte filter correlation to the jigsaw puzzle class, which reflected in the sudden accuracy drop in Figure 5c) A similar trend can be observed for another class jeep. At the region where {0.05, 0.15). we can see that correlation with more semantically related classes: beach vagon, recreational vehicle, is increasing. while and jigsaw

---

Under review as a conference paper at ICLR 2020
Nas fruasd Fitter Comoltaon for Class jig.er. jigoer__aaddo praddo
Neo Prused Fiker Cucrolation be Cless jesp
Nao Erused Fibor Gurrelaioos for Class Igilte
1,00
1,00
0.75
0.75
weich
E:24
0.77
reguect
0,752
budd. VAgE
0.60
Drwder
1.40
Cardgga
6.2
(a) jigsaw puzzle
jecp
lighter
Class 3giowe graale
Pruned Filter Correlation for Class: Jeep.
Filtar Orentan
Deadh vegen
Spear poade
Flt mabiou
Trusing
(d) jigsaw puzzle
(e) jeep
(1) lighter
Figure 6: Correlation of filters (a)-(c) left after pruning, (d)-(f) removed after pruning
spene
Pruzed Fileer Correlttion for Cluss agran
PCA Accumulated Responses
dw0.15
6-00
Tolblll
6075
befsen
valestgli
Fille Orettp
10
aproin dreppirr ginidor hacpeliet Chrstear rtckinn
sproa
nigleam Chitemee
20
grnvain
6023 606 0.2
sin Bla Eab
833 33s 646
X0
613
30
-10
20
db
(a) non-pruned filter correlation
(b) non-pruned filter correlation
(c) PCA projection of activation magni- tudes
Figure 7: Correlation of filters left after pruning (a), removed by pruning (b) and PCA mapping of accumulated response for class apron
puzzle is decreasing. It is possible assume that, the effect of interference from unrelated filters is reducing, and as a result, improves the model accuracy, as shown in Figure 5hh The last example is provided for the class from the group, which negatively affected by the pruning (Figure[5k). Looking at Figure [f we can see that lighter class has a high correlation for pruned filters with neighbouring classes; hence the effect of interference from the class-neighbours was not negated, which is reflected in the accuracy drop. However, one can observe that during the drop in pruned filter correlation for (0.15, 0.21) there a sudden jump in classification accuracy (Figure 55k)
More extended evaluation is provided in Figure the class apron. According to Figure 76 at the pruning rations 0 = 10 the correlation between pruned filters is increasing with such classes as hoopskirt, Christmas stoking and miniskirt, while the correlation is reducing for volleyball and sax. Semantically, the first three classes are more related to the class apron than the last In such way, pruning ratio, 0 00.1, 0.2] filters would be more associated with the related classes, which at the same time should positively affect the performance. That what can be observed in Figure (el There is performance jump in that region for 0 [0.1, 0.2]. Further, the point be supported by calculating the accumulated responses yapron = {n apron
:74 napran] for all filters in AlexNet and mapping them to 2D space through PCA dimensionality reduction. 2D projections
7

---

Under review as a conference paper at ICLR 2020
(Figure (c) shows that the pruning ratio 0.15 (which is in range of 0.1-0.2) class apron lay closer to the correlated classes, and moving away from unrelated "sax" and "volleyball" classes. While the pruning ratio 0.4, it got moved to the unrelated classes again, which also affected the model accuracy.
The general observation from this experiment leads to the conclusion that the objects are represented in the feature space in areas of different density. Interestingly, the accuracy of classes was improving for the pruning ratios at which the correlation of pruned filters with semantically related classes was increasing, while the correlation with semantically unrelated filters was decreasing. These observations are supporting the hypothesis about filter interference and giving a promising direction for further investigation and potential classification accuracy improvements. Conducted experiments showed an interesting observation about the correlation between accuracy and class-wist similarity between pruned filters Fg.
4 CONCLUSIONS
We presented study of semantic CNN pruning. The results indicate that classification classes are asymmetrically represented by filters resulting in the fact that some object classes have their classification accuracy increased when pruned for. We also observed that objects classes represented in densely populated parts of the feature space share very similar filters. Such object classes when pruned for the least active filters (small 7) have a tendency to affect close neighbouring classes more directly and therefore, the classification accuracy of the pruned object increases. Additionally, we also observed that the filters in the lowest layer are rarely pruned, similarly to the first layer. Finally, we observed that there is an interference between filters affecting classification accuracy, and there a correlation between accuracy and class-wise similarity between pruned filters. As future work, we plan to extend our work to more classes and more models. In addition, we will study the effect of filter overlap on convolutional networks safety and adversarial examples phenomena.
REFERENCES
Kamila Abdiyeva, Kim-Hui Yap. Gang Wang, Narendra Ahuja, and Martin Lukac Ql-net: Quantized-by-ooouup CNN. In ICARCV, pp. 413-418. IEEE, 2018.
Peter Anderson, Xiaodong He, Chris son, Stephen Gould, and Lei Zhang.
John- Bottom-Up and Top-Down Atten- Question Answering. 2017. URL
Buehler, Damien Teney Mark
tion for Image Captioning and Visual 079.8.798...
networks. CoRR, abs/1512.08571,, 2015. http://pffhtttpp/rrr...ooooo abs 707 00079.77988
https: http://wwwxwwoogpp////o//
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. of convoluttono neural 1512.. 08571
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition, 2017.
Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. Speeded-up robust features (surf). Computer Vision and Image Understanding, 110()):346 359, 2008. ISSN 1077-3142. Similarity Matching in Computer Vision and Multimedia.
Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-local-feature models works surprisingly well on imagenet. In Intermational Conference on Learning Representations, 2019.
J. Canny, A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8(6)6679-698, Nov 1986. ISSN 0162-8828. doi: 10. 10.1109/TPAMI. 1986.4767851.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training neural networks with weights and activations constrained to +1 or -1. arXiv preprint arXiv: 16X2::660..22833 2016.
8

---

Under review as a conference paper at ICLR 2020
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 580-587, 2014. ISBN 9781479951 178. doi: 10.1009CPP2.0014.8...
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. arXiv e-prints, MtXiv:1503.0251, Mar 2015.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-Excitaiie Networks.
2017.
URL
http:// http:://axii.ooom wwt./rgggbb///....o / 01507
Farzad Husain, Babette Dellen, and Carme Torras. Chapter 20 scene understanding us-
ing deep learning. In Pijush Samui, Sanjiban Sekhar, and Valentina E. Balas (eds.),
Handbook of Neural Computation, pp. 373 382. Academic Press, 2017. ISBN
978-0-12-8113188
doi:
URL
Pieter-Jan Kindermans, Kristof 'T. Schtt, Maximilian Alber, Klaus-Robert Miler, Dumitru Erhan, Been Kim, and Sven Dhne. Learning how to explain neural networks: Patternnet and patternat- tribution. ArXiv e-prints, 2017.
A. Krizhevsky, I. Sutskever, and G.E. Hinton. Imagenet classification with deep convolutional neural networks. in: Advances in neural information processing systems. NIPS, 2012.
Hao Li, Asim Kadav. Igor Durdanovic,
Hanan
Samet, and Hans Peter
Graf. URL
Pruming filters for efficient convnets.
CoRR, Abs/1608.08741 2016.
http://aat http:///wxvv.oogg/ooooooo w.b.r//bs////ooooo 608.0.6 08710
Tsung Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal Loss for Dense Object Detection. Proceedings of the IEEE International Conference on Computer Vision, 2017- Octob:2999-30000 2017a. 7a. ISSN 15505499. doi: 10. 10911CCCV20117..33
Tsung-Y Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2999- 2017b.
D. G. Lowe. Object recognition from local scale-invariant features. In Proceedings of the Seventh IEEE International Conference Compuler Vision, volume 2, pp. 1150-1157 vol.2, Sep. 1999. doi: 10. 1O0/ICCCV.199..9994444.
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing When to Look: Adaptive Attention via Visual Sentinel for Image Captioning. In CVPR, pp. 375-383, 2017. ISBN
978-1-5386-0457-1 URL http:.///aax...oogg/a////////.. 01880
Xiaolong Ma, Geng Yuan, Sheng Lin, Zhengang Li, Hao Sun, and Yanzhi Wang. ResNet Can Be Pruned 60x: Introducing Network Purification and Unused Path Removal (P-RM) after Weight Pruning. arXiv e-prints, art. arXiv: 10XXiv1105.000
1905.00011111 005.001130 00116 36, Apr 2019.
Ari S. Morcos, David G. T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. arXiv e-prints, arXiv:1803.006 180300699, Mar 2018.
Maryam M. Najafabadi, Flavio Villanustre, Taghi M. Khoshgoftaar, Naeem Seliya, Randall Wald, and Edin Muharemagic. Deep learning applications and challenges in big data analytics. Jour- nal of Big Data, 2(1):1, Feb 2015. ISSN 2196-1115. doi: 10. 1411186/-40557-014-0000 https: / org/ 10. 1186/ 10t..///////433-0-0000000
URL
Ojala, M. Pietikainen, and D. Harwood. Performance evaluation of texture measures with classification based on kullback discrimination of distributions. In Proceedings of 12th Inter- national Conference on Pattern Recognition, volume 1, pp. 582-585 vol.1, Oct 1994. doi: N0.1109/CCPR.9994.5776

---

Under review as a conference paper at ICLR 2020
Wanli Ouyang, Xiaogang Wang, Xingyu Zeng, Shi Qiu, Ping Luo, Yonglong Tian, Hongsheng Li, Shuo Yang, Zhe Wang. Chen- Change Loy, and Xiaoou Tang. DeepID-Net: Deformable deep convolutional neural networks for object detection. Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on, pp. 2403-2412, 2015. ISSN 1550-5499. doi: 10.1109/ CVPR.2015.7298855
M. Raghu, J. Gilmer, J. Yosinski, and Sohl-Dickstein. SVCCA: Singular Vector Canonical Cor- relation Analysis for Deep Learning Dynamics and Interpretability. ArXiv e-prints, June 2017.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. ECCV, 2016.
Haoyu Ren and Ze-Nian Li. Balanced Co-Occurrence Features. 2015 IEEE
Object Detection Using Generalization and Efficiency International Conference on Com- URL
puter Vision (ICCV), pp. 46-54, 2015. doi: 10.1 20.100//
http://ieeexploreiiees htt.://ww........oooooooooo////a wow.moet//1111111
Steven Remnie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self-critical Sequence Training for Image Captioning. In CVPR, pp. 7008 7024, 2017. doi: 10.1109/CVPR. 2017.131. URL
0atp://1122.000500 00563
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211222) 252, 2015. doi: 10. 1007/s1 00000/ss1163--000 126-0001-011--1111166666
Aron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In Arxiv, 2016. URRLitttps://areee http:///wawxww.oo.aa//////s///// wwwp/ogggg.ooooo/////.o/.eeeeeeo 6699 03499
Ziyu Wang. Tom Schaul, Mattco Hessel, Hado van Hasselt, Mare Lanctot, and Nando de Freitas. Dueling Network Architectures for Deep Reinforcement Learning. (9), 2015. ISSN 0163-6804. doi: 10. 10.1100 12.10//MCMM..20167773333223 UtL/tttp://rxxx.oom http://aaa/aa.....1..11 06581
Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei. Boosting Image Captioning with Attributes. In ICCV, pp. 4894-4902, 2017. doi: 10. (0.100//CCC22221..522.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable Convolutional Neural Networks. 2017.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learn- ing deep features for scene recognition using places database. In Proceedings of the 27th International Conference on Neural Information Processing Systems Vol- ume 1, NIPS' 14, pp. 487-495, Cambridge. MA, USA, 2014. MIT Press. URL http: / al. acm.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa- net: Training low bitwidth convolutional neural networks with low bitwidth gradients.
arXiv preprint 1oX6.:6606.66666 2016.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the ef- ficacy of pruning for model compression. https: /tttps://arxiv...
CoRR, abs/1710.01878. 2017. URL
Joel Zylberberg. Untuned but not irrelevant: The role of untuned neurons in
sensory information coding. bioRxiv, 2017. doi: 10.1101/134379. URL
https:
bttps://wee wwwp://wwwwwwwo////////////// wwwwxii.
10


Reviews:
[{"review_id": "rJldqmS6Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary\n---\n\n(motivation)\nThis paper proposes a new approach to pruning activations from neural networks,\nbut uses it to understand neural nets better rather than trying to make them\nmore efficient. It prunes filters per-class 1) to measure how sensitive image\nclasses are to pruning and 2) to measure how similar classes are by comparing\nthe filters they prune.\n\n(approach)\nUsing only images from class c, CNN pre-activations are aggregated across spatial\ndimensions and examples. This gives an average feature vector for that class.\nT percent of neurons are pruned. Features with lower pre-activations\n(possibly negative with large magnitude) are pruned first with successively\nhigher activations pruned later.\nNo re-training is performed.\n\n(experiments)\nExperiments use AlexNet and 50 of the 1000 ImageNet challenge classes to show:\n1. Pruning filters results in decreased accuracy with smaller decreases in accuracy as the first and last few filters are pruned.\n2. More filters from middle layers are pruned than are those from early and late layers.\n3. After some filters have been pruned, sometimes pruning can increase accuracy.\n4. I could not understand section 3.2.\n\n\nStrengths\n---\n\nThe proposed pruning method is simple and efficient.\n\nI like the broad goal of understanding CNNs. We could use more papers that just focus on analysis like this one.\n\nI think the idea of class conditional pruning is novel.\n\n\nWeaknesses\n---\n\n# Major Weaknesses\n\nI don't see why these results are significant. I do not find these results very surprising (see next comment) and I do not see why the community will find them useful.\n\n* In particular, consider the conclusion. Sentences 2, 3, 5, and 6 seem to all be observations about what happened in the experiments. The conclusion should re-iterate the results, but it should also say why they are important. How does the work relate to the goals of the community at large? Which goals? Will this enable important new capabilities? What general concepts did we learn from this that we didn't know before?\n\nSome of the positions in the paper could be more carefully considered. There are alternate explanations for many of these phenomena.\n\n* Pruning smallest activations doesn't make sense to me. Typically redundant or un\"important\" activations are pruned because doing so has negligible impact on accuracy. The smallest activations are pruned here. Should the smallest activations be redundant or unimportant in some way? Does this rely on ReLU activations following the pre-activations? I think these values are not necessarily redundant and could be quite important (e.g., as measured by some saliency explanation like Integrated Gradients), but I could be wrong. It would be useful to provide a baseline which removes highest instead of lowest activations.\n\n* Much of the surprise about figure 5 seems to be because it is not monotonic but it was expected to be monotonic. I agree that these curves should generally go down as theta increases, but I don't see why that relationship should be strict. I would be surprised if activations were not in some way dependent on one another. Furthermore, why does that dependence have to be interference? Couldn't it also be that some activations are complementary (and thus ineffective when only one is present)?\n\n* Does Network Wise Pruning (NWP) favor more layers than others? It may be that some filters have higher Accumulated Responses per filter simply because there are more feature maps in the previous layer (thus more things summed up) and not because of what information they capture or their relationships with other filters. Does this happen? This could be an alternative to the following conclusion: \"This means that the encoding provided by the first and last layer seems to be the most crucial and the densest.\"\n\n\n# Other General Weaknesses:\n\n* AlexNet is a rather old architecture to use for this analysis, so I can't be confident these methods or behaviors will generalize to other architectures. Does it hold for more modern architectures?\n\n\n# Missing details / Points of confusion:\n\n* The paper says the correlation from Figure 6 should be expected to decrease as theta increases. Why should this be expected?\n\n* What exactly does Figure 6 measure? I think it's the ratio of the size of the intersection rho_sigma to the size of the union of the same two sets. However, the text calls the metric \"correlation.\" This should be made clearer.\n\n\n# Minor presentation weaknesses\n\nSome parts of the notation/explanation don't make sense to me:\n* \"Let Lambda=... be the number of object classes in the dataset.\" But Lambda is defined as a set, not a numeral.\n* The number p needs more context/subscripts as it depends on class c and filter i.\n* \"θ = [0, . . . , 1]\" This defines theta as a finite set, but I think it's meant to be a number in the interval range (0, 1).\n* Why is lambda_c needed instead of just using the index c to specify a class? The variable lambda doesn't seem to be any different than a class index.\n* \"let Fσ and Fσ ̄ be the set of unpruned and pruned filters for a given γ...\" I don't see how this works. According to eq. 3 gamma depends on a particular filter i and class c, so the only filter F could contain is the ith one. (Later it became clear that F was only mean to class conditional, not filter conditional.)\n\nFigure 5: These plots shoul all have the same y axis range. This would make them comparable and allow readers to much more easily compare trends across classes. Similar steps should be taken so the same range is used any time theta is plotted on the x axis.\n\nFigure 3: I find it hard to get an overall ordering of the approaches in this figure because there is so much variance from class to class. It effectively conveys the variance, but I'd also like to know what the means across classes are for each method so I can compare the proposed approaches more effectively.\n\n\"The results indicate that classification classes are asymmetrically represented by filters resulting in the fact that some object classes have their classification accuracy increased when pruned for.\"\n* I'm not sure what it means to be asymmetrically represented by filters.\n\n\nSuggestions\n---\n\nThis analysis would have been more interesting with an existing pruning approach because we would already know that such an approach is good a removing unnecessary filters.\n\nFinal Evaluation\n---\n\nQuality: Experiments could have been cleaner, but they basically demonstrate the patterns the paper intended to show.\nClarity: I could understand most experiments at a high level, but I found it hard to understand the motivation and the rest of the experiments.\nSignificance: As explained above, I do not see why the paper is significant.\nOriginality: The experiments and the proposed class conditional pruning approach are somewhat novel.\n\nThe paper is somewhat novel, but do not find it very clear and I do not see why it is important, so I cannot reccomend it for acceptance.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkgMdNkFKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to prune filters in CNN model to interpret the correlation among different filters or classes. Though interpreting CNN filter is a well-studied topic, learning the interpretability through pruning is new and interesting. The proposed method is simple, by just using the averaging the value of the output of each filter as the indicator. The author claims that object classes represented in high feature density area usually share similar filters, which is in accordance with the common sense. Also, filters at lower layer are usually important. \n\nSome questions: \n1.\tSince each filter is still like a black box, is it possible to visualize some result of the discovered interpretability?\n2.\tI’m confused with the implementation of pruning. If the filter at layer j-1 is pruned, then the dimensions of the filters at layer j should also change. How this issue is dealt with? Further, will this dimension reduction, instead of the pruned filter itself, influence the model performance?\n3.\tWhy not use the absolute value of r_i? Any justification for this?\n4.\tThe author mentioned that no normalization across categories are applied. However, are r_i from different layers comparable under NWP? Also, How can you guarantee that the Eq.(2) is comparable for different filters? More discussion on the normalization is desired.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HylbYWO6tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to prune CNN networks for each class in order to study interpretability of the networks. However, there are several significant drawbacks:\n\n1) The pruning approach is too simplistic. Network pruning has been a field of very active research as the authors have acknowledged, however, the approach used in the paper is a very simplistic remove the ones with lowest response one, for which, there is no experiment to justify its validity w.r.t. state-of-the-art pruning approaches. In fact, from Fig. 2 and Fig. 3 one can almost conclude that this naive pruning method is significantly inferior to state-of-the-art.\n2) Lack of novel insights. For an explanation paper, we would expect to obtain some new insights about the classification. The results that this paper get to, such as few filters were pruned in the lower layers, similar classes share similar filters, are not necessarily new knowledge to the community. In terms of the result analysis, mostly simple correlation analysis was used which presented no novelty nor insights.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to use pruning to study/interpret learned CNNs. The reviewers believed the results were not surprising and/or had no practical relevance. Unlike in many cases, two of the reviewers acknowledged reading the rebuttals, but were unswayed.
Decision: Reject