Quantitative Performance Assessment of CNN Units via Topological Entropy Calculation
Authors: Yang Zhao
Hao Zhang
Venue: ICLR 2022 Poster (2022)
Keywords: interpretation of neural network units
computational topology
convolutional neural networks
entropy

Abstract:  Identifying the status of individual network units is critical for understanding the mechanism of convolutional neural networks (CNNs). However, it is still challenging to reliably give a general indication of unit status, especially for units in different network models. To this end, we propose a novel method for quantitatively clarifying the status of single unit in CNN using algebraic topological tools. Unit status is indicated via the calculation of a defined topological-based entropy, called feature entropy, which measures the degree of chaos of the global spatial pattern hidden in the unit for a category. In this way, feature entropy could provide an accurate indication of status for units in different networks with diverse situations like weight-rescaling operation. Further, we show that feature entropy decreases as the layer goes deeper and shares almost simultaneous trend with loss during training. We show that by investigating the feature entropy of units on only training data, it could give discrimination between networks with different generalization ability from the view of the effectiveness of feature representations.


Published as a conference paper at ICLR 2022
QUANTITATIVE PERFORMANCE ASSESSMENT OF CNN UNITS VIA TOPOLOGICAL ENTROPY CALCULATION
Yang Zhao & Hao Zhang
Department of Electronic Engineering
Tsinghua University zhao-yant anl 80mai 1s. www.sssiinnh..eeeeeeeeooe wewhnghuaaedu.ccom .dd..n, haozhangetsinghus. edu. cn
ABSTRACT
Identifying the status of individual network units is critical for understanding the mechanism of convolutional neural networks (CNNs). However, it is still chal- lenging to reliably give a general indication of unit status, especially for units in different network models. To this end, we propose a novel method for quantita- tively clarifying the status of single unit in CNN using algebraic topological tools. Unit status is indicated via the calculation of a defined topological-basee entropy, called feature entropy, which measures the degree of chaos of the global spatial pattern hidden in the unit for category. In this way, feature entropy could pro- vide an accurate indication of status for units in different networks with diverse situations like weight-rescaling operation. Further, we show that feature entropy decreases as the layer goes deeper and shares almost simultaneous trend with loss during training. We show that by investigating the feature entropy of units only training data, it could give discrimination between networks with different gener- alization ability from the view of the effectiveness of feature representations.
INTRODUCTION
Convolutional neural networks (CNNs) have achieved great success in various vision tasks (Szegedy et al., 2016; Redmon al., 2016; He et al., 2017a). The key to such success is the powerful ability of feature representations to input images, where network units' play a critical role. But impacted by the diverse training deployments and huge hypothesis space, networks even with the same archi- tecture may converge to different minima on a given task. Although units between these networks could present similar function for the same task, yet they may have completely different activation magnitudes. Consequently, this makes it fairly hard to give general indication of the status for a given network unit with respect how well features represented by it from images in the same class.
Being rough indicators in practice, magnitude responses of units are usually chosen simply (Zhang et al., 2018) or processed statistically (such as average mean) (Li et al., 2016; Luo et al., 2017) based on the idea of matched filtering. However, firstly these indicators are apparently sensitive to rescaling operations in magnitude. If performing a simply rescaling operation to the weights such as the strategy introduced Neyshabur et al. (2015), the results of the network and the function of each unit would all remain unchanged, but these indicators would vary along with the rescaling coefficient. Secondly, as the spatial information in the unit is completely discarded, they could not give discrimination between units with and without random patterns, for example units separately outputted by a well-trained and ing the mentioned situations, these indicators fail ensure the universal applicability for units in different network models.
CNN filter. Without a valid indication regard-
In this paper, attempt to investigate the status of units from new perspective. Roughly speak- ing, natural images in the same class have common features, and meanwhile the locations of these features are spatially correlated in global. For effective units, features picked out and represented
'Regarding the term unit, unit is the perceptive node in networks, which generally refers to the activated feature map outputted by a convolutional filter in CNNs.

---

Published as a conference paper at ICLR 2022
Images
Effective unis
values
Local operation
Topological characterization
eguubrzze pattom
Spataa pattom unchanged
Low ture entroyy
Ineffective units
Rescallnn values
Resco
Choosi sparbpppttoon
Spatbll paraer unchanged
High feature entropy
Figure 1: Comparisons between the effective units and ineffective units. For effective units, since the spatial pattern of the features in the images would be preserved, units should stably present this regularized spatial pattern. We propose a topological-basee quantity called feature entropy to indicate the unit status, giving reliable indication various situations like rescaling the values.
by high activation values the units. And due to the locality nature in feature extraction by convo- lution, this global spatial pattern between the common features would be preserved synchronously in the counterpart representations in the effective units. In contrast, for ineffective units, being in- capability of effectively representing these common features, representations would be in chaos and marks of this pattern is vague. This provides a valid road for performance assessment of individual and universally applicable to any CNN architecture.
units, and critically it rescaling-invaree
The investigation of such pattern could naturally lead to topological approaches because knowl- edge of topological data analysis such as barcodes (Ghrist, 2008) provides valuable tools to resolve the intrinsic patterns in raw data. Along this line, firstly we introduce a method for characterizing the spatial pattern of feature representations in units for a single sample by incorporating with the topological tools, and then use information entropy to evaluate the stability of this spatial character- izations for various images sampled from the same class, where we call feature entropy. this way, unit is judged to be effeective if its feature entropy is high, otherwise ineffective.
our experiments, we find that feature entropy would gradually decrease as the layer goes deeper and the evolution trends of feature entropy and losses are almost the same during network training. We show that the feature entropy could provide reliable indication of unit status in situations like weight-rescaling and the emergence of random pattern. Finally, show the value of feature entropy in giving discrimination between networks with different generalization ability by investigating only the training set.
2 RELATED WORKS
One line of research that attracts many researchers is seeking solutions in a way of visualizing what features have learned by the units (Zeiler & Fergus, 2014; Zhou al., 2014; Mahendran & Vedaldi, 2015: Simonyan et al., 2013). Status is generally identified depending on the degree of alignment between the visualized features and the human-visual concepts (Bau al., 2017: Zhou et al., 201 8a; Bau et al., 2020), On the one hand, they meanwhile give excellent visual interpretation of each unit; on the other hand, it hinders its universal application arbitrary tasks and models in which units' functionalities may be unrecognized to human (Wang et al., 2020).
Another related research trace lies in the field network pruming, where they concentrate on using simple methods to roughly select less important units within network. Typical approaches include the I.1-Norm of units (Luo et al., 2017), Average Percentage of Zeros (APoZ. units (Hu al., Despite
2016), some sparse-based methods (Li et al., 2019; Yoon & Hwang, 2017), and commonly used in practice, since without a specific processing on units diverse situations, they unable to provide general indication for units in different networks.
Besides, Morcos et al. (2018) introduce the class selectivity from neuroscience to investigate the selectivity classes for specific unit, the basis of calculating the mean units. Alain & Bengio (2016) propose linear classifier probe, where they report the degree of linear classification of units in intermediate layers could somehow characterize the status of units.
2

---

Published as a conference paper at ICLR 2022
Lastly, we would like to discuss some recent works related to topological approaches in deep learn- ing. Naitzat et al. (2020) demonstrate the superiority of using ReLu activation by studying the changes in Betti numbers of a two-class neural network. Montúfar al. (2020) use neural networks to predict the persistent homology features. In Gabrielsson & Carlsson (2019), by using barcode, they show the topological structure changes during training which correlates to the generalization of networks. Rieck et al. (2018) propose the neural persistence, a topological complexity measure of network structure that could give criterion on early stopping. Guss & Salakhutdinov (2018) empirically investigate the connection between neural network expressivity and the complexity of dataset topology. In Hofer et al. (2017), topologiaal signatures of data are evaluated and used to improve the classification of shapes.
3 METHOD
In general, input images for network model are commonly resized to square for processing, For input image sample I of a given class with size X be represented by unit U with size m X m via feature extraction processing f in CNN, we have,
f
For image I, features specifically arranged, where each feature has associated spatial location in the image. After perceived by U, features are represented by high activation values at the corre- sponding locations in the unit. Basically, there are two steps in our assessment of unit performance: firstly, characterize the spatial pattern hidden in these high activation values a unit for a single image; secondly, evaluate the stability of this characterization when giving multiple image samples.
3.1 CHARACTERIZING THE SPATIAL PATTERN OF FEATURE REPRESENTATION IN UNIT
For U2 with grid structure, the location of an element generally refers to its coordinate index (i,j). And intuitively, the spatial pattern hidden in the elements denotes certain regular relationship among their coordinate indices. So, it is natural to model such relationship with graph structure and tackle it with topological tools in the following.
Unit and graph We use the edge-weighted graphs (Mehmet et al., 2019) as our basic model and construct the weighted graph g (V, F.) from unit Ut.j. where the vertex set and E is the edge set. Define the adjacency matrix A of G as follows,
A E mm Aiij = =Us
(2)
should noted that the individual element of A the weight of edge in G, which conveys the intensity of corresponding point in the U.
A family of undirected graphs ())) with adjacency matrices At could be constructed by following the typical implementation of the sublevel set.
1 Ang 2a
where (1) is the oth value in the descend ordering of elements of A and 1()) is indicator function. Here, we take the adjustment of (1)) mar(A (1)), (A("))T) to ensure the adjacency matrices A of undirected graphs to be symmetric.
So ((v) = (V(+), Ele)) is the subgraph of G where (v) = V and E(r) C E only includes the edges whose weights are greater than or equal to a(*) We have the following graph filtration,
Gi G2 C G3 c...
To be more specifically, in this sublevel set filtration, it starts with the vertex set, then rank the edge weights from the maximum amar to minimum amin and let the threshold parameters decrease from amaz amin At each step, we add the corresponding edges obtain the threshold subgraph ((v).
Fig.2 illustrates the construction of certain subgraph through a toy example. Consider the unit Ur We circle the locations of the top 4 largest elements in Ut, (Ai.. Then the nonzero elements in adjacency matrix A (4), {(1, 2), (4, 3), (2, (3, 1)} is located (Fig.2B) and corresponding sub- graph ((1) is constructed 2Fi..2C).
3

---

Published as a conference paper at ICLR 2022
Coluum
2
3
1.3
15
0.6
3.5
TOW
1th
(1.2)
2
(2,4)
2.7
21
1.6
9.1
7
1.9
3.2
0.2
4th
A
1.1
0.5
10.6
27
2th
C
(3.1)
(4.3)
D
3
unit
adjacency matrix
undirect graph -V E
clique complex (4)
U
A (43), ()
G
Figure Example of the conversion from unit its clique complex.
Complex filtration To further reveal the correlation structure in the graphs, they are typically converted into certain kinds of topological objects, where topological invariants are calculated for capturing the high-level abstraction of correlation structure. Here, by following the common method in (Horak et al., 2009; Giovanni et al., 2013), each graph g(v) is converted to simplicial complex (also called clique complex) 7(*), as shown in Fig.2D. In this way, we have complex filtration corresponding to graph filtration (Eq.4).
(()) C 7() (3) CT
(5)
This filtration describes the evolution of correlation structure in graph & along with the decreasing of threshold parameter. Fig.3A shows the complex filtration of the previous example (Fig.2).
r (2)
20
(6)
1
2
2
36.. C. 8.)
birth point
1
1
4
1
A
3
B
3
Figure 3: Instance of complex filtration (A) and Betti curve (B).
So far, we have completed the characterization from unit to the topological objects. Other than our strategy, also discuss other alternative method, which maps the unit to the cubical complex (Kaczynski et al., 2004). See Appendix for more details.
Betti curve and its charaterization Next, kth Betti number (Hatcher, 2002) of each element in the complex filtration could be calculated using the typical computational approach of persistent homology (Ninna al., 2017).
(*) + (((()))
Intuitively, kth Betti number (()())) could be regarded as the number of k-dimensional 'circle's or 'hole's some higher order structures in complex r(v), On the other hand, many meaningful pat- terns in the unit would lead to the 'circle's or 'hole's ' of complexes in the filtration (Eq.5), see Fig.2 for illustration. In particular, the number of 'hole's is typically used as an important quantitative index for featuring such patterns. Hence, the kth Betti numbers (()(v)), ),v [1,... n) could be arranged into so called keth Betti curves (UU, , k) for the unit U. Fig.3B shows the 1th Betti curve of filtration in Fig.
Once having obtained the Betti curve, one needs to interpret the Betti curve and extract its core char- acterization. Although there exists many choices of distance between two topological diagrams such as persistence images (Adams et al., 2017), persistence landscape (Bubenik et al., 2015) and persis- tence entropy(Ninna al., 2017), we find that the simple birth time of the Betti curves 3(U, v, k) is sufficient in this characterization.
b(U k) inf (v)3((, #
4

---

Published as a conference paper at ICLR 2022
We call b(U, k) the birth time. Birth time is the indication of the critical element in complex filtration that begins to carry "hole" structure (Betti number is nonzero). It is an important sign that some essential change has occurred in complex filtration, which implies the appearance of regularized spatial pattern of notable components in the unit. Meanwhile, in some cases, no spatial pattern appear in the components in the unit, A(U, v, k) constantly equals zero, meaning that birth time doesn't I exist. In general, this would happen when the unit is unable to give representations for the image, where its values are almost all zeros.
3.2 ASSESSING THE UNIT PERFORMANCE USING FRATURE ENTROPY
For image samples in the same class C, an ideal unit U could perceive their common features. So, the spatial pattern of this unit should be similar between different image samples. In other words, the birth time obtained from each realization of units should be relatively close. That is to say, the performance of good unit for certain target class should be stable over all the samples of this class. It is the key idea for performance assessment of network unit.
Birth distribution Essentially, birth time (é U. is a random variable since sampling images from the specific class C could be regarded as statistical experiments. In fact, the probability space (22 E, P) could be constructed. The elements in sample space f2 are the unit U resulted from the image samples in dataset of class C. Σ could be set as common discrete o-field and probability measure uniformly distributed on Q. In other words, every image sample has an equal chance to be chosen the input of network model. Afterwards, be (i, U, k) is defined as a random variable on Ω (where the argument is i, and U and is are parameters),
be (e, U, ()) Ω -
(8)
with the probability distribution
by
Pe.u,(())= P(be(i,, U,, U,k) x)
(9)
#(2))
where
#(2)
b
Σ 5=1
(L(U,)))=
(10)
Here the composite mapping be (.) from Ω to Z is composed of all the operation mentioned above, including construct weighted graphs, building complex filtration, calculating Betti curve and extracting birth time.
The degree of concentration of (c.u.(()) gives a direct view about the performance of unit U on class C, as illustrated in 1. More specifically, if the distribution presents close to a degenerate- like style, it means that the underlying common features of the class C could be stably perceived by the unit U. On the contrary. the distribution presents close to a uniform-like style when features are perceived almost blindly, indicating that unit U is invalid for C. In summary, the degree of concentration of (C, is supposed to be an effective indicator of the performance of unit U.
Feature entropy To further quantize the degree of concentration of birth distribution &c.u.(()), we introduce its entropy Hc.U,& and call feature entropy,
He,u,k - Pcu,((t))) Pc.
(11)
should be noted that the birth time in Eq.7 may not exist for some input images in class C and unit U. For unit U, the percentage of images in class C having birth times, termed selective rate ec.u, is also a crucial factor to the effectiveness of U on C. If the ec.u is too low, it indicates that the unit could not perceive most of the image samples in this class. In this situation, extremely low eC.U would cause the feature entropy approach to zero, but the unit should be judged as completely invalid. Therefore, we rule out this extreme case by setting a threshold p and for completeness, and assign the feature entropy associated with the maximum of feature entropy 2 for the set of samples,
H.yUA =
H.J.kk (1-Ec.u) log [2]
CCTU 2 EC.U p
(12)
Here, p is prescribed as 0.1 in our computation.
5

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
For experiments, we use the VGGI6 network architecture to perform the image classification task on the ImageNet dataset. Unless otherwise stated, the exampled VGG16 model trained from scratch with the hyperparameters deployed in Simonyan & Zisserman (2014). For clarity, we only calculate birth times ( U, 1) based on Ith betti curve for all the units. Also, should noted that our method focuses the behaviors of feature extraction operations and has not utilized any kind of particular nature of VGG network architecture, and all our investigation could be applicable to other network architectures effortlessly.
4.1 CALCULATION FL.WW
input image
units
complex lex filtration
birth point
birth distribution
feature entro
feature entropy
2713
1.87
images
selective rate
1019 point
Figure Calculation flow of feature entropy.
As an example, the class partridge (wnid n01 8074966) in ImageNet is chosen for illustration. Here, sample 100 images from its training Fig.4 shows the calculation flow. It starts from extracting all the units for each image sample. By characterizing the unit with graph model, each unit corresponds to a specific filtration. Then, using formular 7, we can obtain the birth time of each unit. In this way, the distribution of birth time could be set up via Eq.9 over the sampled images. Fig.4 shows the histogram the birth time distribution for a specific unit in the the last convolution layer "block5_conv" Likewise, the feature entropy can be calculated via Eq. for all other units.
the image set for building the birth distribution.
4.2 LAYER AND TRAINING ANALYSIS
Layer analysis Here, we check the status of units in each convolutional layer, where we average the feature entropy across all the units within the layer indicate the overall status of units in this layer. Using the same image set the previous section, Fig.5A(1-2) give comparisons of results between the convergence model and the random-initialized model.
In Fig.5A(1), we could clearly see that for the convergence model, the feature entropy continually decrease as the layers go deeper. This as expected because the layer goes deeper, units considered to perceive more advanced features than previous layers, so the spatial pattern in these features would be more significant. As for the random-initialized model, since units are incapable to perceive the common features, we could not observe a clear decrease of feature entropy, and mean- while the feature entropy in every layer is higher than that the convergence model. Fig.5A(2), we could also find that each layer in the convergence model would present higher selective rate than the corresponding layer in the indoo-initialized model, except for the last convolutional layer "block5_conv3"" Also, the selective rate of the last convolutional layer is much more lower than other layers. The low feature entropy and fairly high selective rate indicate that comparing to units in other layers, units at the last convolutional layer in the convergence model would present strong specialization and exhibit the most effective representations features to this class.
Then, randomly choose 100 classes in ImageNet and average the feature entropy across all these classes on the convergence model. Fig.5A(3) shows the results. We could see that the results are very similar to F1g-5) which confirms the fact further.

---

Published as a conference paper at ICLR 2022
A
20
30
077
1.
Festurn aranery
wick coom
20
(3)
(2)
(3)
(4)
Figure 5: (A) Comparisons of feature entropy (1) and selective rate (2) of different layers between the convergence model and
model, where (3) shows the results over 100 classes. (B) Simultaneous evolution of training loss and feature entropy during training for the chosen class (1-2) and for the 100 classes (3-4).
Training analysis Then, we investigate the variation of feature entropy of the last convolutional layer during training. Fig. (B(-2) show the results the same example class used previously, and Fig.5B(3-4) show the results across 100 classes chosen previously. In both situations, we could find that the feature entropy would decrease during training. indicating that units gradually learned to be able to perceive the common features in the class. And remarkably, the decreasing pattern of feature entropy and that of training cross-entropy loss coincide approximately. Both of them experience a comparable big drop in the first epoch and gradually down to the convergence level. This means that feature entropy is valid indicator of network performance.
4.3 INDICATOR OF STATUS OF NETWORK UNIT
To investigate the ability of feature entropy as indicator of unit status, we make comparisons with some commonly-used analogous network indicators including Ll-norm (Li et al., 2016), APoZ (He et al., 2017b), and a more generalized form of class selectivity used in Zhou et al. (2018b). Here, the unit and the image set the previous subsection are still used in the following demonstration.
Rescaling investigation The comparison is implemented by rescaling the magnitude of values to half for all the input images or all the CNN filters connecting to the layer. Both the two implemen- tations could potentially cause the values in units within the layer vary with the same scale, but in general have no substantial impact on the network performance and the function of each unit. In other words, units should be indicated almost the same with or without such implementation.
Table I: Comparisons of unit status by rescaling the values in images CNN filters
Images
CNN filters
Accuracy
Ll-norm
APoZ
Class selectivity
Feature entropy
X
X
0.83
29.5
17.14%
0.58
1.87
Half scale X
X
0.81
14.7
17.15%
0.31
1.90
Half scale
0.83
14.6
17.14%
0.30
1.87
Half scale
Half scale
0.79
7.2
17.16%
0.03
1.92
Table 1 shows the results where X denotes rescaling operation for the item. As half scaling the magnitude in input images or units, the performance of the model fluctuates slightly. We could find that APoZ and feature entropy vary in the similar way with the performance, but Ll-norm and class selectivity vary terribly. Apparently, despite little effect for the network, rescaling operations would have major impact these magnitude-based indicators, like Li-norm and class selectivity. These indicators fail to give accurate and stable measure of unit status especially when facing images or units with different value scales.

---

Published as a conference paper at ICLR 2022
Table 2: Comparisons of unit status with respect well-trained units and random units
L1 L1-norm
APoZ
Class selectivity
Feature entropy
Well- unit Random initialized units
29.5
17.14%
0.58
1.87 2..00.21),0.833...2.
32.2(30.9)
(10%)
0.01(0.003)
Detecting randomness in units Next, we compare the status of this unit with random units (units yielded by annoom-nitiilizze models). Table presents the results. The random units are sampled 100 times and the presented results are averaged over the 100 samples where the value in the brackets denotes the standard deviation. Since random units are clearly incapable to perceive features well like those trained units, they are expected to be indicated as ineffective units. We could see that when using Ll-norm and APoZ indicators, they are impossible to give stable indication as the standard deviation is extremely large. In some samples, the random units judged much "better" than the trained units, which is obviously incorrect. Accordingly, it could be also misleading using APoZ the indicator of unit status. In contrast, the feature entropy would consistently be very high when random pattern exists in the unit, providing a well discrimination between trained units and random ones.
4.4 USING FEATURE ENTROPY TO INDICATE NETWORKS WITH DIFFERENT GENERALIZATION
In general, due to the large hypothesis space, CNNs could converge to variety of minima on the dataset. Since feature entropy could reliably indicate the status of network units, it is natural to use it to discriminate which minima could provide more effective feature representations.
Models In this subsection, we prepare two sets of VGG16 models. Model set A consists of four models trained from scratch with different hyperparameters consists of five models trained from scratch to almost zero training error with the same hyperparam- eters but on ImageNet dataset with different fractions of randomly corrupted labels as introduced in Zhang et al. (2017). Table 3 and 4 separately show the performance of models in the two model sets. In model set B, we use Model AD in Model A as the first model Model BA with corruption rate. Besides, it should be noted that all the calculation in this section is based on the image sampled from the training dataset.
ImageNet dataset, and Model set B
Table 4: Model set B
Table 3: Model set A
Train Ace
Test Acc
Corrupted
Train Acc
Test Acc
Model BA
0.996
0.378
0.0
Model AA
0.732
0.657
Model BB
0.992
0.297
0.2
Model AB
0,818
0.532
Model BC
0.994
0.166
0.4
Model AC
0.828
0.444
Model BD
0.992
0.074
0.6
Model AD
0.996
0.378
Model BE
0.993
0.010
0.8
Model set A Using the same image set in previous section, we start by investigating the feature entropy of units at different layers in the four models. Here, we still use the averaged feature entropy across all the units within layer indicate the overall level of how well the units in this layer could perceive the features. Fig. 6A(11)) shows the results of this class. We could see in the figure that there would not be significant difference of feature entropy between these models in layers except for the last convolutional layer. And in the last convolutional layer, for models with better generalization, their feature entropy would be lower than those with poor generalization, indicating that they would provide more effective feature representations. Besides, as for the selective rate, the four models are quite close.
Then, we randomly choose 100 classes in the ImageNet dataset and calculate the feature entropy of the units in the last convolutional layer. Fig.6A(3) presents the scatter plot for the four models, where each point stands for the feature entropy and selective rate of specific class. For each model, its points locate at an area separately from other models, giving a discrimination between models. Also similarly, models with better generalization have points with lower feature entropy.
8

---

Published as a conference paper at ICLR 2022
1.0
entropy
3.0
Selecive rate
0.8
selectuv
0.0
2.5
0.2
Model AD
Modil
Model
Model
0.3
Model AA
Mode AB
Mode AD
Mode AC
(1)
Grog
(2)
curl cood
feature entrspy
22
2.6
B
5.0
Modei BA
Model BA
4.5
Model
Model BB
1.0
Modal
Modal aB
entrapy
4.0
Model BD
Model BE
selectiv rore
0.8
selective rate
0.3
Model BD
Model BE
25
0.2
30-
Model
Model
Model
Model BD
Model BE
0.0
(1)
(2)
(3)
25
3.0
3.5 entrayy
4,0
4.5
(A) and model B (B). Com- pare the feature entropy (1) and selective rate (2) of units at different layers between models in the corresponding model set on the exampled class. (3) shows the scatter plot between feature entropy and selective rate of units at the last convolutional layer on the 100 sampled classes.
Figure 6: Comparisons between models in separately model
Model set B For model set B, set A, where the results are shown in Fig.6B, Comparing to the Model set A, since using the partially corrupted labels, units in the Model Set B are unable perceive the common features between samples in the same class, which causes that the selective rate of most units are extremely low as shown in Fig.6B(2). Due such low selective rate, we could also find in Fig. that feature entropy of the units in the last convolutional layer may abruptly reach to very high point. The more fraction the labels are corrupted, the higher feature entropy the units are and in the meantime the lower the selective rate the units are. This could be observed as well in Fig.6B(3) where the 100 classes are used for calculation.
the same implementation applied previously in the model
5 CONCLUSION
We propose novel method that could give quantitative identification of individual unit status, called feature entropy, for a specific class using algebraic topological tools. We show that feature entropy reliable indicator of unit status that could well cope with various cases such rescaling values or existence of randomness. Also we show that feature entropy behaves in the similar way as loss during the training stage and presents descending trend convolutional layers go deeper. Using feature entropy, we show that CNNs with different generalization could be discriminated by the effectiveness of feature representations of the units in the last convolutional layer. We suppose this would be helpful for further understanding the mechanism of convolutional neural networks.
ACKNOWLEDGMENTS
We would like thank Brain-Inspired Research Team at Tsinghua University for the discussions. We would like to thank the reviewers for their helpful comments.
REFERENCES
Henry Adams, Tegan Emerson, Michael Kirby, Rachel Neville, Chris Peterson, Patrick Shipman, Sofya Chepushtanova, Eric Hanson, Francis Motta, and Lori Ziegelmeier. Persistence images: A

---

Published as a conference paper at ICLR 2022
stable vector representation of persistent homology. Journal of Machine Learning Research, 18, 2017.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.
David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralb.. Network dissection: Quantifying interpretability of deep visual representations. Proceedings the IEEE conference on computer vision and pattern recognition, pp. 6541-6549, 2017.
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units deep neural network. Proceedings of the National Academy of Sciences, 117449:30071-30077 2020.
Kay Henning Brodersen, Cheng Soon Ong. Klaas Enno Stephan, and Joachim M. Buhmann. The balanced accuracy and its posterior distribution. 2010 20th International Conference on Pattern Recognition, pp. 3121-3124, 2010.
Peter Bubenik al. Statistical topological data analysis using persistence landscapes. J. Mach. Learn. Res., 16(1):77-102,, 2015.
Rickard Brüel Gabrielsson and Gunnar Carlsson. Exposition and interpretation of the topology of neural networks. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pp. 1069-1076. IEEE, 2019.
Robert Ghrist. Barcodes: the persistent topology of data. Bulletin of the American Mathematical Society, 45(1):61-75, 2008
Petri Giovanni, Scolamiero Martina, Donato Irene, and Vaccarino Francesco. Topological strata of weighted complex networks. PLOS ONE, 8(6):199 2013.
William H Guss and Ruslan Salakhutdinov. On characterizing the capacity of neural networks using algebraic topology. arXiv preprint 1802.0443, 2018.
Hatcher. Algebraic Topology. Cambridge University Press, London, 2002.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.11009CVP...016..99
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE In- ternational Conference on Computer Vision, ICCV 2017, Venice, Italy. October 22-29, 2017, pp- 2980-2988 IEEE Computer Society, 2017a. doi: 10.11099CCV22201..3.22
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Conference Computer Vision and Pattern Recognition, pp. 4340-4349, 2019.
Yihui Hc, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017b.
Christoph Hofer, Roland Kwitt, Marc Niethammer, and Andreas Uhl. Deep learning with topologi- cal signatures. In Advances in neural information processing systems, pp. 1634-1644, 2017.
Danijela Horak, Slobodan Maletić, and Milan Rajković. Persistent homology of complex networks. Journal of Statistical Mechanics: Theory and Experiment, P003(4)), 2009.
Hengyuan Hu, Rui Peng, Yu- YunWing Tai, and Chi- Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv: 0605000250, 2016.
Tomasz Kaczynski, Konstantin Michael Mischaikow, and Marian Mrozek. Computational homol- ogy, volume 3. Springer, 2004.
10

---

Published as a conference paper at ICLR 2022
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In Iternational Conference on Learning Representations, 2016.
Yuchao Li, Shaohui Lin, Baochang Zhang, Jianzhuang Liu, David Doermann, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Exploiting kernel sparsity and entropy for interpretable enn compres- sion. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 2800-2809, 2019.
Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 5188-5196, 2015.
Aktas Mehmet, Akbas Esra, and Fatmaoui Ahmed. Persistence homology of networks: methods and applications. Applied Network Science, 4(61):1-28, 2019.
Guido Montúfar, Nina Otter, and Yuguang Wang. Can neural networks learn persistent homology features? arXiv preprint arXiv:2011. 1..iv:2011.166880 2020.
Ari Morcos, David GT Barrett, Neil Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In Iternational Conference on Learning Representations, 2018.
Gregory Naitzat, Andrey Zhitnikov, and Lek-Heng Lim. Topology of deep neural networks. arXiv preprint arXiv:2004.06093, 2020.
Behnam Neyshabur, Ruslan Salakhutdinov, and Nathan Srebro. Path-sgd: Path-normalized opti- mization in deep neural networks. In Advances in Neural Information Processing, pp. 2422-2430, 2015.
Otter Ninna, A Porter Mason, Tillmann Ulrick, Grindrod Peter, and A Harrington Heather. A roadmap for the computation of persistent homology. EPJ Data Science, 6(17):1-38. 2017.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Bastian Rieck, Matteo Togninalli, Christian Bock, Michael Moor, Max Horn, Thomas Gumbsch, and Karsten Borgwardt. Neural persistence: complexity measure for deep neural networks using algebraic topology. arXiv preprint 11X1v:18/2.009764, 2018.
Giorgio Roffo, Simone Melzi, and Marco Cristani. Infinite feature selection. In International Con- ference on Computer Vision, pp. 4202 2015. doi: 10.1109/1CCV.2015..77
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
networs. Vi-
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens. and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2818-2828 IEEE Computer Society, 2016.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8684-8694, 2020.
Jaehong Yoon and Sung Ju Hwang. Combined group and exclusive sparsity for deep neural net- works. In International Conference on Machine Learning, pp. 3958-3966. 2017.
11

---

Published as a conference paper at ICLR 2022
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad 1. Morariu, Xintong Han, Mingfei Gao, Ching Yung Lin, and Larry S. Davis. NISP: pruning networks using neuron importance score propagation. In 2018 Conference Computer Vision and Pattern Recognition. pp. 9194-9203, 2018. doi: 10. 200.C/VP...08.0099...
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 118-833. Springer, 2014.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, 2017.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8827- 8836, 2018.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antomio Torralba. Object detectors emerge in deep scene cnns. arXiv preprint arXiv:1412.6886. 2014.
Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representations via network dissection. IEEE transactions on pattern analysis and machine intelligence, 41(9): 2131-2145, 2018a.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the importance of individual units in cnns via ablation. arXiv preprint (a0::1886.0899), 2018b.
A APPENDIX
A.1 IMPLEMENTATION DETAILS OF MODELS
In our experiments. the networks we used are the standard VGGI6 architecture. We simply resize all the mentioned sample images to 224 x 224 without implementing any data augment (like random crop) during all the experiments (including the feature entropy calculation and the reported perfor- mance). The model used for demonstration in the experiment section (besides the last subsection) is Model AA in model set A.
For model set A, we use the following implementations.
Model AA. The hyper-parameters are the same with them in the paper (Simonyan & Zis- serman, 2014).
Model AB. The hyper-parameters are the same with them in Model AA, except for chang- ing the momentum to and without using the data augmentation strategy.
are the same with them in Model AB, except for that only the first fully connected layer use the dropout with the rate of 0.3.
Model AC. The
Model AD. None of the conventional training enhancement technique is applied. Basically, is Model AC without using dropout and 12 regularization.
For model set B, we use the following implementations,
Model BA. is actually the Model AD.
Model BB. The hyper-parameters are the same with them in Model BA, except for corrupt-
ing the labels with 0.2 fraction.
Model BC. The hyper-parameters are the same with them in Model BA, except for corrupt-
ing the labels with 0.4 fraction.
Model BD. The hyper- parameerrtsssss are the same with them in Model BA, except for corrupt-
ing the labels with 0.6 fraction.
Model BE. The hyper-yareee parameters are the same with them in Model BA, except for corrupt- ing the labels with 0.8 fraction.
12

---

Published as a conference paper at ICLR 2022
A.2 SPATIAL CHARACTERIZATION USING CUBICAL COMPLEX
In our method, a unit is firstly convert to a set of graphs and then each graph could be converted to the corresponding clique complex. In this way, a unit is characterized by the filtration of clique complexes. Alternatively, the unit could be directly modeled as a cubical complex (Kaczynski et al., 2004), and then use the similar sublevel set implementation to acquire the filtration of cubical com- plexes. So here, we use cubical complex instead of clique complex and meanwhile keep the rest of the methods unchanged.
By investigating the birth distribution on the classes in the Imagenet with Model AA, we found that for every unit especially at relatively deeper layers, almost 95% images would have no birth time. In other words, no persistence homology emerges for almost all the units when using cubical complex. Thus, is impossible to give a further calculation the stability for images in the same class.
A.3 DISCUSSIONS OF USING BIRTH TIME
When characterizing the filtration of topological complexes, the birth time of the filtration. Comparing to some other characterizations, we suppose that the advantages of using birth time may lie in,
Birth time is very casy to compute. In practice, when using birth time, it is not necessary to go through the whole filtration process, where we could stop the calculation when the birth time emerges. In this way, could largely save the computation especially when the size of units is very high. This could be very helpful for calculating the large amount of units in neural networks.
Birth time is very convenient for the investigation of stability between samples because it is an integer essentially. So, the discrete distribution could be set up effortlessly. Besides, when using entropy to investigate the stability of birth times, the results are bounded strictly in the range from 0 to log N.
Birth time is suitable for the idea. Since the significant features are generally represented by the high activation values in the units, it is more expected that the regularized pattern could be formed by these values. So natural to focus on these high activation values, which leads to the use of birth time.
Then, considering the various topological features, we compare the effect of using birth time and two other characterizations the Betti curves which are its maximum value and its integration. During calculation, we would only change birth time to the use of given characterization, where all the other parts remain the same. Table shows the results. We find that the difference between using birth time and using other characterization we need to calculate the whole Betti curve. The computation cost may be hundreds times than using birth time.
acceptable. But when using other characterizations,
Table Comparisons of feature entropy by using different characterizations
Birth time
Maximum
Integration
Reference unit
1.87
1.89
1.92
Last convolutional layer
2.01
2.09
2.11
Last convolutional layer (100 classes)
2.07
2.13
2.17
A.4 STUDY ON SAMPLE SIZE
For an efficient computation in practice, we could use part of the training set for calculating the feature entropy. In this section, check the influence the feature entropy when using different sample sizes.
We test the sample size from the 50 to the full size of of the example class (about 1300 images) in the training set on the reference model (Model AA) used in the paper. We first investigate the variation of feature entropy of single unit when changing the sample size. We performed 100 times of
13

---

Published as a conference paper at ICLR 2022
sampling investigate the stability of feature entropy, and each time randomly sample 500, 100 and 50 images from the training set. Fig.7 shows the histogram of feature entropy the umit used in Section 4.1 for the 100 times of sampling, where the x-axis is feature entropy and y-axis is the frequency. We could see that for 500 and 100 samples, their feature entropy distribute closely to the However, for 50 samples, the feature
value of feature entropy for using the whole training entropy distribute scatteredly. Therefore, using 100 samples could give considerable feature entropy of the class and in the meantime largely reduce the computation cost.
0.50
133 sumples
6.2
sumpli
0.2
somple
0.20
0.00
9.20
0.0
0.15
0:35
0:00
0.00
6.08
0.05
00
1.8
1.8
24
0.00
1.8
1.9
7.
0.00
18
3.0
0.00
14
1.3
2.3
footuro entrapy
fosture entropy
feature antropy
feature enrrazy
Figure 7: Histogram of the feature entropy in 100 trials for different sample size.
Layer block5 conv3
1100 samples
500 samples
110 samples
www.aeeeegggeo. festure errryyy
0.3
8.10
0.13
festure entrrry
petcemooge
aeroemmangg
2.0
0.23
2.05
0.03
3.5
2.0
25
35
2.5
2.5
3.0
15
2.0
25
3.
600
803
1200
(1)
festure
(2)
feature
(3)
festure
(4)
samplo
Layer block5 conv2
0.22
503 samples
126 samples
averaged Rattre
0.23
porcersange
02
0:33
0.10
0.08
0.35
355
0:33
15
000
15
1200
festure entropy
feature entropy
feature entropy
sample number
Figure 8: (1-3) Histogram of the feature entropy of units in the layer "block5_conv3" conv3" and (ilocck__con22" for different sample s4ze.(4) Feature entropy averaged across all the units in the layer with respect to the sample size. Error bars stand for performing the sampling 100 times.
Next, we investigate the feature entropy of all the units at a layer when changing the sample size. All the units the layer "block5 "_oos3""" and "block5_conv2" in the reference model used, and 2ig.2A presents the results. As we could see in Fig.8(1-3), for 1300, 500 and 100 samples, their distributions of feature entropy of the 512 units at the layer are very close. And in Fig.8(4) average the feature entropy across the 512 units, and we could see that the feature entropy vary slightly from using 100 samples to 1300 samples. Besides, the error bar is acquired via performing the 100 times of sampling. And we could see the error bars decrease as the sample sizes increase, and even for 100 samples, its error bar still very low.
A.5 ADDITIONAL RESULTS ON RESNET
In this section, we perform the experiments on the ResNet34, which trained from scratch with the hyperparameters deployed in (He al., 2016).
14

---

Published as a conference paper at ICLR 2022
Rescaling and randomness comparison For rescaling and randomness investigation, the experi- ment setups are the same as those in Section 4.3. Here, the unit is chosen at the layer "conv5_3_ou which is the last layer before global pooling layer. The suffix "out" stands for the output of a residual convolutional block. Table shows the corresponding results when performing rescaling operation.
Table 6: Comparisons of unit status by rescaling the values in images CNN filters
Images
CNN filters
Accuracy
Ll-norm
APoZ
Class selectivity
Feature entropy
X
X
0.81
122.1
4.79%
0.47
1.71
Half scale
0.80
60.9
4.81% 1%
0.23
1.72
X
Half scale Half scale
0.81
61.1
4.79%
0.24
1.71
Half scale
0.77
30.4
4.83%
0.07
1.74
Table 7 shows the corresponding results of trained units and random initialized units.
Table Comparisons of unit status with respect to well-trained units and random units
L1-norm
APoZ.
Class selectivity
Feature entropy
Well-trained unit 122.1 678(317)
4.79%
0.47 0.02(0.013)
1.71 2.08(0.29)
Random initialized units
1.17%(1..54%)
Just as the results on VGG16 (shown in Table 1 and 2), due to the advantages of using topology, feature entropy could give stable indication of the status of units as expected, no matter with the scaling operation in the randomness situation.
Layer and training analysis We follow the implementation in the VGG16 network (Section 4.2) and first check the status of unit in each convolution layer in ResNet34. Fig.9 gives the comparisons between the results of the convergence ResNet34 model and the random-initialized model. For FAg. it shows the variation of feature entropy with respect the output of each convolutional block in the ResNet34 model. We could see that the feature entropy continually decrease as the layer goes deeper, similar to the results on VGG16. Besides, the feature entropy of each layer in the model are all larger than the corresponding layer in the convergence model.
Then, we check the variation of feature entropy for layers in convolutional block, where Fig.9B shows the results. Typically, for ResNet34 architecture, a convolutional block consists of two con- secutive convolutional layers and yields the output after the shorteut connection. We could see in the figure that the feature entropy at the first convolutional layer would increase at the second con- volutional layer, which is because of the non-activation of units at the second convolutional layer. After that, the shorteut connection would decrease the feature entropy even without being activated and finally reach at a lower value than that at the first convolutional layer. The shortcut connection plays important role in making the features more significant, which leads to the decrease in the feature entropy.
Residual I block
shortna coctintionn
come5 3 bbock
conv5 bbock
conv4 6 block
Barder Gasiediccaa
Cony 2 ino activation
Output lw h sctivsionn)
Figure 9: (A) Comparisons of feature entropy of different layers between the convergence model and random-ing aindom-iitiialized model. (B) Comparisons of feature entropy of the layers in separately three residual blocks between the convergence model and random- initialized model.
random-initialized
Next, Fig. 10 shows the variation of feature entropy during training. Similarly to those on the VGG16 model, the feature entropy decrease during training and behaves close to the training loss as well.
15

---

Published as a conference paper at ICLR 2022
21
Dostare criogg E 12
fosture CoBuR
Figure 10: (B) Simultaneous evolution of training loss and feature entropy during training for the chosen class (A-B) and for the 100 classes (C-D).
Indicating models with differnt generalization Further, a set of ResNet34 models with different generalization trained via the same partially corrupted strategy in Section 4.4. Table A.5 shows the corresponding performance and Fig. 11 gives the results of feature entropy. As we could see in the figure, results on ResNet model are quite similar those on VGG16 (Fig.4.4). The feature entropy of models would gradually increase as the generalization become worse.
Table 8: Performance of model set R
Train Ace
Test Acc
Corrupted
Model RA
0.823
0.714
0.0
Model RB
0.982
0.382
0.2
Model RC
0.991
0.229
0,4
Model RD
0.995
0.102
0.6
Model RE
0.991
0.036
0.8
Model
Model RA
4.5
Model
Ro
0.4
Model RC
Model RB
feature entoapy
4,0
Ro
RD
selective rate
0.0
selective rate:
0.3
Model RD
Model RE
33.5
0.6
3.0
0.2
0.4
Model
Modal
2.0
0.2
0.1
1.5
con Your
cowd oour
and mee
cond ou
0.0
(1)
and
and
(2)
comdl
conv
come
rod
and
and
and
feature entropy
Figure 11: Comparisons between models in model set Compare the feature entropy (1) and selective rate (2) of units at different layers between models in the corresponding model set on the exampled class. (3) shows scatter plot between feature entropy and selective rate of units last residual block the 100 sampled classes.
A.6 COMPARISONS WITH OTHER RELATED METHIODS USED PRUNING
In this section, we compare feature entropy with two other related strategies used in pruning. One is filter pruning via geometric median (FPGM) (He et al., 2019), which uses the norm distance the geometric median of the set of filters as the indicator of the importance of filter F; at layer (Eq.4 in He al. (2019)),
g(F.) = Σ
FFFF,12
(13)
FF
where denotes the total number of filters at the layer. A filter is considered as useless its (FF)) small.
16

---

Published as a conference paper at ICLR 2022
The other one is called neuron importance score propagation (NISP) (Yu et al., 2018), which mea- sures the importance of a unit by propagating the final feature selection score backwards based on coefficients of network weights at specific layer (Eq.19 in Yu et al. (2018)).
8k1 Σ W (k++)
8k41.
(14)
where | | is the element-wise absolute value, Sk,, denotes the importance score for unit i at layer ls and w denotes the weight matrix at layer (A + 1). The method needs a base metric to score the final feature selection and use it to calculate units in other layers backwards via this formula. Here, we follow the metric used in the original paper called infinite feature selection (Roffo et al., 2015).
i.j
We follow the same investigations of rescaling operations and randomness detection as deployed previously in Section 4.3. Table and Table 10 show the corresponding results. We could see in the Table 9 that just the other methods in Section 4.3, rescaling operations would have major impact on the two indicators. Apparently, from the above formulas of the two methods, we could easily find that they are still essentially based on the magnitude of weights. Therefore, it is nature that they fail to give valid indication in these complicated situations.
Table 9: Comparisons of unit status by rescaling the values in images or CNN filters
Images
CNN filters
Accuracy
FPGM
NISP
Feature entropy
X
X
0.83
0.88
72.37
1.87
Half scale
X
0.81
0.88
36.04
1.90
X
Half scale
0.83
0.44
36.17
1.87
Half scale
Half scale
0.79
0.44
17.81
1.92
In addition the rescaling operation, as we could see in Table 10, the FPGM method is also unable to give correct discrimination between the well-trained units and the aadom-initialized units.
Table 10: Comparisons of unit status with respect well-trained units and random units
FPGM
NISP
Feature entropy
Well-trained unit
0.88
72.37
1.87 2.87(0.22)
Random initialized units
1.44(0.044)
19.27(2.98)
A.7 STUDY OF PRUNING
In Section 4.4, feature entropy has shown the ability to reliably indicate the status of units between different models in various situations, which is the core motivation of the paper. In addition, also show the effectiveness of feature entropy to give comparisons between different units in the single model in a fixed situation. In this situation, since units are generally in the comparable scale (unless use some specific implementations on networks like Neyshabur et al. (2015))), the mentioned problems may be largely alleviated, so units would be compared directly via the typical methods like L1-Norm, etc. Besides, we would consider feature entropy and selective rate both to represent its effectiveness of units for an accurate estimation, where we would fuse these two factors by using H/c in the following calculations.
Here we are going to investigate the units in the single model two parts. The first part is the cu- mulative unit ablation and the second part is example of the implementation of network pruning.
A.7.1 CUMULATIVE UNIT ABLATION
Ablation test setting For a given class, cumulative unit ablation tests check the evolution of the network performance by progressively removing each unit within a layer according to the order of
"This implementation could somehow yield the same effect with performing the rescaling operation to the magnitude of a specific unit in Section 4.3, without changing the results of networks and other units. For more details about this implementation, please refer to the paper.
17

---

Published as a conference paper at ICLR 2022
certain kind of sorted attribute of units. Typically, removing unit refers to forcing the outputs of the unit to all zeros. Here, the unit's 's feature entropy is chosen as the attribute, and descending and ascending orders are considered simultaneously. According to our idea, the unit with lower feature entropy is more effective than that with higher feature entropy. The cumulative ablation tests reveal the relation between feature entropy of unit's output and its effectiveness where training sets are generally used for calculating the feature entropy of the units and testing sets are used for checking their impact on the network performance. Besides, the investigations are still using the last convolutional layer on the same image set in the previous section. In particular, for each image in the set, it would be randomly rescaled within the ratio from 0.5 1.5.
network performance,
A
18-
01.01
(1)
290
200
3
148
12
1.-
12
300
430
(1)
rumber dAhkol
(2]
numberd 17h alreted
[31
(4)
Figure 12: Cumulative ablation curves of accuracy (1) and loss (2) according to the descending rank (A) and the ascending rank (B), together with comparisons with other methods (3-4).
The cumulative ablation is performed via ascending order of feature entropy first. Fig. 12A shows the variation the testing accuracy (1) and loss (2) during ablation. We can see the accuracy drops rapidly in the beginning and arrives zero after ablating about 38 units. On the other side, the loss sharply increases in the beginning just like accuracy, afterwards slowly climbing and reaching its peak value (12.32) during ablation. It should be mentioned that the peak value exceeds that of removing all the units (8.71). Being somehow counterintuitive, the ablation has generated layer even worse than the all-zero layer.
Results
We have also compared with the performance evolutions resulted from APoZ, class selectivity, L1- norm, FPGM, NISP. We can see in 123-4)(3-4) that the accuracy drops (or the loss increases) more rapidly using feature entropy (red line) than others during the ablation process. This implies that those most effective units picked by feature entropy would be more impactful the network perfor- mance.
Then, the cumulative ablation is performed via descending order of feature entropy. From F2g. instead of decreasing, the testing accuracy features a slow increase from 0.54 to surprising 1.0 when 96% of the units are removed. These units do lie the head part of feature entropy rank and are considered as the less effective units. So removing them will promote the accuracy and enhance the network performance. Remarkably, starting with removing a certain unit, the accuracy dramatically drops to zero within only 30 units. It means that these units are critical and removing them will lead to breakdown of network. Likewise, the comparisons are also implemented with other attributes in 12A. We could also find that curve of feature entropy could reach much higher peak point in accuracy, which shows the advantage of feature entropy.
Interestingly, could see that the testing performance could be largely enhanced via the cumula- tive unit ablation for a single class. Then, all the 1000 classes in ImageNet are investigated in the same way. On the one hand, we would check whether the testing performance could be enhanced
18

---

Published as a conference paper at ICLR 2022
Table 11: Performance enhancement via cumulative ablation when using different methods
Unchanged network
Ablated network
Performance enhanced
LI-Norm
0.759!
0.928
0.167
Apoz
0.759
0.873
0.114
Class selectivity
0.759
0.917
0.158
FPGM
0.759
0.789
0.030
NISP
0.759
0.838
0.079
Feature entropy
0.759
0.945
0.186
All the performances reported as balanced accuracy
(Brodersen al., 2010).
in this way for the majority of the classes, on the other hand, we use this to give comparisons be- tween different methods under this cireumstance. Table 11 shows the corresponding results, which reported by averaging across all the classes. Note that all the results in the table are reported as the balanced accuracy, which is commonly used for assessing the performance where the task is im- balanced. We could see that after performing ablation on a single class, the network could generally acquire performance enhancement some extent. And compared to other methods, feature entropy shows a larger performance enhancement, which again demonstrates the superiority of our method.
A.7.2 IMPLEMENTATION OF NETWORK PRUNING
In the previous paragraph, we perform the ablation test on single class to show the effectiveness of feature entropy. Here, we are going to give implementation of channel-level network pruning for the whole dataset via feature entropy.
Network pruning setting The network is pruned by following the layer-by-layer strategy. which prunes the channels from the shallow the deep layers progressively in stages. In the first stage, unimportant units at a layer are selected to be pruned based on a given pruning ratio via feature entropy. For units at the layer, they are selected by averaging the feature entropy across numbers of classes,
1 K H(U)) = K Σ (U.)
(15)
where K is the total number of chosen classes. In this way, we would prune the corresponding filters whose outputted units have higher feature entropy. In the meantime, since the unit removed, channels of the filters that connect this unit and units at next layer would also be pruned. Then in the second stage, we would fine-tune the pruned model. Here, we still use the reference VGG16 model in the paper our target network. In the conventional VGG architecture, after the convolutional backbone, two fully connected layers are used as the classifier for the following decision making. Although parameters in the fully connected layers could account for over 80%. yet the majority of computation cost lie on the convolutional operation. So here, we remain this classifier and target to prune all the convolutional layers for computation reduction.
Additionally, during fine-tuning stage, we use the SGD optimizer with a learning rate which is initially set to 1e-3, decay 10 times after epoch fine-tuning and finally stop after 1e-5. All implementations are deployed on the Nvidia-A 100 station with a batch size of 512. Also, similar the setting in the paper, all the images simply resized 224 224. We the FLOPs of convolutional operation as the metric of computation cost and parameter counts as the metric of storage cost.
Results We first investigate the impact of the number of classes K to the pruning effect, where the pruning ratio is set to 50%. Fig. 13A shows the pruning results VGG16 with remaining fully connected layers. We could that as choose amount of classes for the calculation of
19

---

Published as a conference paper at ICLR 2022
feature entropy, the accuracy of the fine-tuned model gradually increase. And compared to using 100 classes, using 150 and 200 classes would increase the accuracy slightly. Thus, for an efficient computation of feature entropy, would use 100 classes for the following pruning implementation.
A
B
VG616 16 with layers
68
VGG16 layers
tecttng arcuracy
testing accuracy
66
64
62
60
150
0.4
0.5
0.6
number classes K
pruning ratio
Figure 13: (A) Accuracy of the fine-tuned models with respect to the number chosen classes used for feature entropy calculation. (B) Accuracy of the fine-tuned models on different pruning ratios.
Then, we give the results fine-tuned networks via the implementations of separately 40%, 50%, 60% pruning ratios, as shown in Fig. 13B. We could see that as we keep more channels, the fine- tuned network could reach higher accuracy and when the pruning ratio is 60%, the accuracy is comparable to that of unchanged network.
Finally, we make comparisons when using different methods with the pruning ratio of 50%. For other methods, we use the same pruning implementation as feature entropy. Table 12 gives the final results. As we could see in the table, the model pruned via feature entropy could reach competitive results compared to other methods. In conclusion, feature entropy shows the advantage of being an effective indicator for not only network units between different models but units a single model.
Table 12: Comparisons of the accuracy of fine-tuned models when using different methods
Accuracy (fine-tuned)
Pruned ratio
#Params
FLOPs
img/sec?
Unchanged network
65.71%
138M
30.96B
2,178
L1-Norm
1.03%
Apoz
1.59%
Class selectivity
0.38 %
50%
75.9M
7.87B
5,541
FPGM
0.59%
NISP
0.92%
0.09 %
40%
87.8M
11.15B
4,869
Feature entropy
0.42 %
50%
75.9M
7.87B
5,541
0.89%
60%
64.2M
5.02B
6,131
"M" and "B" denote the million and billion separately. Inference speed measured Nvidia A100 GPU. Since the pruned models are the same architecture, their inference time are very close in practice, so it is the average value across all the models.
20


Reviews:
[{"review_id": "ulVoEpr_3vV", "reviewer": "Reviewer_CZcz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. The paper is easy to follow and the authors clearly highlight the problems with prior approaches when measuring the importance of a unit.\n2. The experimental analysis corroborates these problems nicely and shows that they can be solved by the proposed feature entropy approach.\n\nConcerns:\n1. The advantage of feature entropy over the three baseline approaches is illustrated by conducting two tests (rescaling and randomness) and observing the behaviours. However, the comparisons mostly represent a sanity-check and do not illustrate the importance and practical benefit of the proposed approach. \n2. Previous approaches that measure the importance of a given node have often been compared in pruning scenarios. Is there a reason, why feature entropy can not be used in this setting?\n3. The method is compared to relatively old baselines. For instance, [1] and [2]  address a similar task of measuring neutron importance (in a pruning setting).\n\nMinor:\n- It is not quite clear from the start, what is meant my “status” of a unit. Maybe “importance” of a unit will be clearer.\n- Consider adding Model BA in Table 4 to improve clarity.\n\n[1] R. Yu, A. Li, C.-F. Chen, J.-H. Lai, V. I. Morariu, X. Han, M. Gao, C.-Y. Lin, and L. S. Davis. NISP: Pruning networks using neuron importance score propagation. In CVPR, 2018.\n[2] Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In CVPR, 2019.", "summary_of_the_paper": "The paper proposes a topology-based approach to measure the effectiveness of a unit in a neural network for a set of inputs (i.e. images from a specific class). A graph is constructed based on a given image and the units representation and the birth point is computed. The importance of a unit is then measured as the entropy over the birth point distribution, which is estimated by sampling a set of images. Experimental results highlight the methods advantage over three baseline approaches and illustrate its ability to discriminate models with different generalisation capability.", "summary_of_the_review": "Overall, the paper is interesting and the proposed approach of measuring unit importance is well motivated. However, as mentioned above, I do have some concerns regarding the empirical evaluation of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lhfBXLcCJV7", "reviewer": "Reviewer_qJ7D", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- The establishment of a technique for estimating the state of a neural network with respect to training will be useful for various applications, and the creation of a technique that can check the progress of training and compare trained NNs is expected to contribute to the development of AI.\n- Such measurement techniques are generally plagued by scaling and randomness problems, but the entropy-based approach is a significant contribution in that it solves these problems.\n- Being able to analyze the learning progress at the structural level of the NN can also provide information for ways to improve performance.\n- Although it is a natural idea to measure the distribution of the activity of each unit, it is an interesting idea to construct it on a TDA basis, because a simple measurement method has many problems.\n- Although the proposed method is heuristic and still in the observation stage, it is worthy of further development.\n\nWeakness\n- The authors state that the size of the unit can be assumed without loss of generality, but that seems to be a rather strong assumption. Since the proposed method can be computed only for square matrices, it cannot be applied directly when the number of elements in the data or filter is not square.\n- It is also questionable that the unit is two-dimensional. In general, the middle layer of a CNN is multiple images. One of the advantages of the proposed method is that it can be measured independently of the order of the elements (i.e., the numbering of i and j). This is because the order may change in each training trial. The single-sheet assumption may not be a problem if you compare each corresponding sheet, but the order of the sheets will be switched in each trial, making it less useful.\n- In TDA, we generally refer to the time of occurrence and disappearance of the simplex in filtration as birth time and death time, respectively, so calling the first non-zero state of the Betti curve the birth point may cause confusion.\n- The reasons for using birth points are explained in the appendix. However, if, for example, another 1-cycle is created immediately after the first 1-cycle is created, it means that significant activity has taken place in a different region, so ignoring it would be lacking critical information. The trade-off between ease of calculation and lack of information will need to be discussed.\n- Each cell of a CNN unit is fundamentally independent and does not represent any relationship. It may retain the positional relationships of the input images, but it is still counter-intuitive to represent the activity of the units in a relational graph. Therefore, it is difficult to understand why this construction method extracts the features of the activity of the unit.\n- This construction method may only extract information similar to the change in the number of active cells for a change in threshold. In this case, there is no effect of using TDA. These things need to be considered and verified in order to show that the proposed method makes sense.", "summary_of_the_paper": "In this paper, the authors propose a technique to measure the state of each neural network unit. The features of the neural network units are given by a method that applies TDA to a graph representation of the activity degree of each unit. The authors also experimentally measure the degree of learning and show its effectiveness against other methods in terms of scaling and randomness.", "summary_of_the_review": "The effects suggested by the proposed method are very useful and can be considered a sufficient contribution. On the other hand, the proposed method is heuristic and limited, and there are some questions about the theory behind feature extraction, which require deeper investigation in the future.\n\n[Additional comments after discussion]\nThrough discussion and revision, some of the unclear points were clarified and the paper became better. The proposed tool has the potential to be a very useful tool and its contribution is appreciated. On the other hand, this is something that needs to be developed in the future. These points were taken into account at the beginning, and although they are relatively positive, the score remains unchanged.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kgRGWgj9EkN", "reviewer": "Reviewer_LHvg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. I enjoyed reading this paper. The proposed method is introduced clearly step-by-step with figures.\n2. Main claims are well-supported. The motivation is explained in Section 1 by pointing out two flaws of previous indicators. And Section 4.3 shows how the proposed method address the flaws point-to-point through experiments.\n3. The finding from Section 4.4 is interesting. The proposed indicator seems to be a good replacement for extra testing data. This would be useful in some real cases where labels are expensive and we want to use all the labeled data to train a model for the potentially best performance.\n\nWeaknesses:\n1. It would be good to repeat the experiments on ResNet. I'm curious about how skip connections affect \"feature entropy\".\n2. Sampling 100 images seems to be too small. Any reason on the sampling size? Why not computing on all images?\n3. Network pruning is mentioned in the related work. It would be interesting to see how the proposed method guide pruning.\n4. Minor: It is not clear to me why the adjacency matrix needs to be symmetric. Thinking backwards, does this suggest symmetry in the feature map is important or not?\n5. Minor: There is no $w_v$ in equation (3).", "summary_of_the_paper": "This paper proposes a novel method to tell whether a unit (feature map) is effective in CNNs quantitatively, via \"feature entropy\". The proposed method builds a weighted graph based on a feature map and calculates the Betti curve. Then it defines the \"birth point\" that implies the appearance of regularized spatial pattern of notable components in the feature map. By computing the \"birth point\" based on a sample of images from a specified class, it can get a birth distribution, upon which the \"feature entropy\" is computed.", "summary_of_the_review": "Overall, it is a solid paper with clear introductions on the proposed method and good experiment design supporting the main claims. However, the fact that experiments are all based on one kind of CNNs is concerning. I would consider it as a board-line paper with the current version.\n\n-------------\nThe author feedbacks are satisfying. The improvements on the paper with answers to Q1&3 are substantial. I've raised my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "PrAMCzxPKU", "reviewer": "Reviewer_F2EM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main essence of the paper is to translate the activations of the neural network for a given pattern class into a graph representation whose topology evolution is analyzed (using TDA) as a function of activation threshold.  The main novelty is the combination of well known methods  from topological data analysis and apply it to neural unit characterization.   The feature entropy measure defined seems to be more stable across variations do the rescaling and perturbations and the measure seems to correlate with network hyperparameter changes and perturbations. Thus the measure may be suitable for characterization of relationships between input/output in a systematic way.  \n\nHowever, The main weakness of the paper is that the results in the paper are not conclusive on how this work will impact neural net design choices and optimization.  If this paper can be extended to address this, I believe the work will have impact. \n\nIt is not clear to me why the authors take the activations and perform a symmetry transformation (A^v = max (A_v, (A_v)^t) ) after equation 3.   It is not clear to my why you chose 0.1 for p in equation (12).  What is the rationale here?  \n\nThe experimental sections can be improved in clarity. While I follow the general results, I cannot completely comprehend the fact that the feature entropy drops rapidly and the value in the last layer is large.  Can you elaborate about this and the above points in the rebuttal?\n\nWhile the ideas in the paper are clear, the writing needs to be improved.  There are several places where the sentence structure is awkward and it is hard to comprehend what the authors state. \ne.g. 1)  In contrast, for ineffective units, since being incapability of effectively  ---> change to:   being incapable of effectively\n2) In Hofer et al. (2017), topological signatures of data are evaluated and used to improved\nthe classification of shapes. --> change to: used to improve\n3) rewrite line in section 3.2 -- \"For a specific target class C, consider image I(i) ...  By perceived\nwith an ideal unit U, it should present similar pattern with other image samples.\"\n4) section 4.2 -- when referring to figures and results the sentences are often awkward to the reader..\n5) section 4.3 -- \"Since random units are clearly incapable to well perceive features\nlike those trained units,\"   should be changed to 'perceive features well'\n6) section 4.3 -- reference to ApoZ seems to be wrong (As far as I understand ApoZ was proposed by Hu et al, not He et al).\n\n\n", "summary_of_the_paper": "The paper applies well known methods in topological data analysis for quantitative assessment of neural net unit performance. Specifically, a scheme is proposed wherein spatial patterns of activation in the network units are determined as function of a sequence of activation thresholds, patterns transformed to a graph representations (i.e. successive filtrations) and their corresponding kth betti number, (i.e. number of kth dimensional holes), evolution calculated. In this paper k is set to 1.  For units that the authors term, effective units for a given class and a given input image class population, the analysis is expected to yield to similar birth time of the holes. Thus, a feature entropy is defined on the first birth time in the betti curves computed on a given instance and aggregating the estimated birth times over a population of instances in a given class.  The main results include the illustration of invariance of the calculated feature entropy when one does reweighting / rescaling transformations in the inputs and the illustration of correlation between the choice of hyperparameters, perturbations in data sets, and the feature entropy measure. \n\n", "summary_of_the_review": "Overall, I think the paper is interesting and the ideas are worth pursuing. The writing needs significant improvement, the experiments are still limited.  I am not sure also not sure about the impact of the paper, in its current form, and its utility for enabling improvements in design of neural net architectures.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for understanding the role and importance of individual units in convolutional neural networks. The reviewers were in agreement that the technique is novel and provides potentially valuable insights into neural network behavior. The reviewers were less certain about the utility or significance of this idea; however, the authors partially addressed this concern by adding studies of using this technique as a pruning heuristic, and future researchers will be the best judge of the paper's eventual significance. With that in mind, I recommend acceptance so that this intriguing idea can become part of the research literature and future researchers will have this opportunity.
Decision: Accept (Poster)