On the Evolution of Neuron Communities in a Deep Learning Architecture
Authors: Sakib Mostafa
Debajyoti Mondal
Venue: ICLR 2022 Submitted (2022)
Keywords: explainable ai
deep learning

Abstract: Deep learning techniques are increasingly being adopted for classification tasks over the past decade, yet explaining how deep learning architectures can achieve state-of-the-art performance is still an elusive goal. While all the training information is embedded deeply in a trained model, we still do not understand much about its performance by only analyzing the model. This paper examines the neuron activation patterns of deep learning-based classification models and explores whether the models' performances can be explained through neurons' activation behavior. We propose two approaches: one that models neurons' activation behavior as a graph and examines whether the neurons form meaningful communities, and the other examines the predictability of neurons' behavior using entropy. Our comprehensive experimental study reveals that both  the community quality and entropy can provide new insights into the deep learning models' performances, thus paves a novel way of explaining deep learning models directly from the neurons' activation pattern
.

Under review as a conference paper at ICLR 2022
ON THE EVOLUTION OF NEURON COMMUNITIES IN A DEEP LEARNING ARCHITECTURE
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning techniques are inereasingly being adopted for classification tasks over the past decade, yet explaining how deep learning architectures can achieve state-of-the-ard performance is still an elusive goal. While all the training informa- tion is embedded deeply in a trained model, we do not yet understand much about its performance by only analyzing the model. This paper examines the neuron acti- vation patterns of deep learning-basse the models' performances can be explained through neurons' activation behavior. We propose two approaches: one that models neurons' activation behavior as graph and examines whether the neurons form meaningful communities, and the other examines the predictability of neurons' behavior using entropy. Our experi- mental study reveals that both the community quality and entropy can provide new insights into the deep learning models' performances, thus paves novel way of explaining deep learning models directly from the neurons' activation pattern.
classification models and explores whether
INTRODUCTION
Deep learning allows the researchers engineer better features and representation of data through representation learning (LeCun et al. 2015). Despite the widespread usage of deep learning methods, is still considered a black box, and there is lack of understanding of the working procedure of the models (Rai 2020 Oh et al. 2019 Tzeng & Ma 2005 Rudin 2019). Researchers often rely on intuition and domain knowledge when designing deep learning architectures (Shahriari et al. 2015). They use the models' accuracy and loss to evaluate the performance and tune the hyperparameters to optimize the models (Géron 2019 Gigante et al., 2019). However, to gain a deeper insight into the model, it is crucial to look beyond its accuracy and loss, i.e., we need to design additional evaluation metrics that can provide a level of confidence in the model predictions.
Exploring network architecture helps explain why some models perform better than others. For example, by comparing the weight optimization against model architecture, /aaerr & Hal (2019) showed that a model's performance depends mostly on its architecture. Zhang al. (2018) created an explanatory graph that can disentangle different part patterns from feature maps of the convolutional neural network (CNN). Visual analytics tools have also been used to understand the architectures better (Hohman et al., 2019 Gigante et al.) (019). However, such approaches sometimes require domain knowledge, and are also subject to human interpretation as they do not provide quantitative measures for the evaluation.
Motivation. In this paper, our main focus is on classification models. A deep learning model that performs classification task is optimized minimize the difference between the model's 's prediction and the actual value. The prediction is recorded as accurate if it is above a threshold, but this process does not fully leverage the information hidden inside the architecture. The neuron activation pattern that appears during the training contains rich information that, if understood well, can potentially be combined with model prediction to provide a level of confidence the prediction value. It may also allow us to design more efficient models and additional evaluation metrics to help reduce the number of false negatives and false positives. it is possible to relate the model's performance with the neuron activation pattern, may understand why some models perform better than others by analyzing the pattern. This motivated us to explore different ways to examine neurons' activation pattern and establish their relation to the training and test accuracy.

---

Under review as a conference paper at ICLR 2022
Contribution. We study the neuron activation pattern using two approaches: one based on a graph- theoretic model, and the other is based on an coommation-theorelee model. Our proposed metrics show correlation with the accuracy on both benchmark and real-life datasets in both cases.
For the theph-theoretic approach, we created a novel "activation pattern graph' model and showed that the set of neurons that are frequently and highly activated for class often forms a community (i.e., an induced subgraph that contains more edges within than outside of the subgraph) in the activation pattern graph. We observed that modularity, widely used metric to measure community quality, is closely related to the model's training and test accuracy.
For the cntermattionthhoreette model, we propose a novel method that allows us to measure the predictability of the neurons' activation behavior leveraging entropy (Gray) 201T). Our experimental results show that the entropy of the neurons relevant the model's 's training and test accuracy.
RELATED WORK
Neuron Activation Pattern. Researchers have investigated both neuron activation patterns and loss function to explain deep learning models. Olah et al. (2017) studied what the neurons respond to and proposed that neurons work in group. Li al. 2017) showed relationship between the performance of the deep learning model and the convexity of the loss function. |Mahendran & Vedaldi (2015) studied the feature maps of the CNN models understand the activation pattern by inverting the models and showed that there is indeed a relation between the performance of the model and the activation pattern. Kim et al. (2018) vectorized the activation values of hidden layers and created striped patterns to find relevance between pattern and decision. However, they experimented with a small dataset without directions for generalizability.
Bau et al. (017) studied the relation between the activation pattern and semantic concepts, They optimized the hyperparameters of the deep learning models and quantitively analyzed the effect of changing different parameters. The relevance between activation patterns and semantic concepts was further studied by Fong & Vedaldi (0188). They created vector response semantic concepts based on activation pattern and established that the neurons work in group, and the same neuron can represent multiple concepts. We found a similar characteristic of the neuron activation pattern in our study, where there training, a class was represented by more unique neurons.
overlap of neurons representing multiple classes. However, with
Another way of interpreting the deep learning models' learning is by selecting image patches that maximize the neuron activation (Zelerr & Fergus 2014). Zhou et al. (2014) described scenes using the units from the feature maps and also proposed that individual units behave object detectors. [Simon] et al. (2014) analyzed the deep learning model's feature maps and found a spatial relation between the activation centers and the semantic parts or bounding boxes of the ground truth images. ZZang et al. (2018) used hicrarchical explanatory graphs across layers to propose a way of maximizing a deep learning model's performance. Bau al. (2020) discovered that individual units of CNN can learn concepts from the images and some of them strongly influence the decision of the model. Although there have been several attempts to explain activation patterns, in this paper we take very different approach to the explanation of the neural network models by combining the graph-theoretic and informattion-heerrtii approaches, which allow us to relate various quantitative measures to the model's 's performance.
Visualization and Interpretability. Visualization is considered to be promising approach for explaining complex systems, and as result, there have been several attempts explain the deep learning models using visualization tools (Hohman et al. 2018 Chung et al. 2016). A dataflow diagram is a simple way to visualize deep learning model 's architecture (Wongsuphasawat et al.) 2017). However, dataflow diagrams do not tell much about the model's learning process.
Smilkov et al. (2017) proposed "Tensorflow Playground." " where users can train a model by changing the hyperparameters and observe the training process with various plots and charts. Harley 15)15) proposed an interactive visual inspection tool, where a model was trained on the MNIST dataset (Le Cun et al. 2010), and for input data, the tool showed which feature maps were activated and also which part of the input data was activating the maps. ACTIVIS (Kahng et al.) 2017) is an interactive visualization tool that, along with the dataflow diagram, provides projection of the neuron activation
2

---

Under review as a conference paper at ICLR 2022
for different instances. This only allows users to explore the activation pattern for different samples. classes, and subsets rather than explaining the model's 's working procedure. CNNComparator (Zeg et al. 2017) is tool to compare the performance metrics and the distribution of hyperparameters of Two deep learning models.
A rich body of research examines ways of visualizing feature maps of CNN models (Dobrescu et al., 2019; Simonyan et al.] 2013: Zeiler & Fergus) 2014). A common limitation of studies that rely on visualization is that the visual assessment subjective and often hard to evaluate quantitatively.
Entropy and Deep Learning, Entropy is commonly used to quantify errors, e.g., the cross entropy is widely used loss function (Gordon- Rodriguez et al. 2020). Many researchers proposed modifications to this loss function. Martinez & Stiefelhagen (2018) proposed Tamed Cross Entropy, which has the same convergence property as the cross entropy but is more robust against uniformly distributed label noise, Zhou et al. (2019) proposed Maximum Probability based Cross Entropy (MPCE) loss function, which uses a MPCE based gradient update algorithm and has less paokppropngttion error than cross entropy, and
et al. (2020) proposed to use log-likelihood of the continuous categorical distribution in the place of the cross entropy loss used in label smoothing and actor-mimic reinforcement learning.
Barbiero al. (2021) proposed an entropy- based linear layer for concept-based deep learning models, which utilizes entropy to choose limited subset of input concepts, allowing it to provide concise explanations of its predictions. (Huo et al. (2020) implemented maximum entropy regularizer that encourages uniform weight distribution. Li et al. (2020) approximated the gradient of the cross entropy loss function, which are robust against noise and can avoid the vanishing gradient problem.
Although entropy measures has been used for model optimization, we leveraged it to analyze the activation pattern of the models and propose performance metric that correlates to the model accuracy.
TECHNICAL BACKGROUND
Graph and Community Structure. A graph consists of a set of elements (nodes) and a set of pairs of elements (edges), where the nodes represent objects and edges represent pairwise relationships. A community of a graph is defined as a subgraph that contains more edges within than the edges connecting them to the rest of the graph. Community detection algorithms often define quality measure for the communities and then attempts to find a partition of the nodes that maximizes the will define
quality measure (Fortunato & Castellano! 2007 Newman & Girvan 2004). Later, graphs with nodes as neurons and edges as their simultancous activation. We will examine the communities of these graphs based on several quality measures as explained in the next section.
Modularity. Modularity a widely used metric to assess the quality of given set of non-overlapping communities [Newman] 2004 Duch & Arenas 2005; Clauset et al.] (004)). The idea of modularity is based on comparing the given graph G with a random graph. Given two communities C, and C₃ in G, we use the notation e(,,, C₁) to denote the number of edges between these two communities. Let ac, be the fraction of all the edges connecting the community C, to all other communities, i.e., ac, -, ec,c,- For a random graph, the fraction of the resulting edges that connect nodes within the community C. is ac," 2 Hence the modularity (Newman & Girvan 2004) can be formalized as Qno Qno-ouerlpp >, (ec.c, ac;" This is commonly expressed leveraging the adjacency matrix of the graph (Blondel et al., (0088. Let Agw be the adjacency matrix of G, and let m be the number of edges in G. For a vertex v (resp., w) in G, we denote its degree and community by ko and Co (resp., knp and C.,). Then
Q Qno-overlap 2m
1
Σ (Avm 0,30
kukw )8 Cwr), where 5(C,, C.n) 2m
1, if C, = C.
(I)
0. otherwise.
In Eq. ku represents the probability that two nodes v and w are connected in random graph, 2m
and Avw is 1 0 depending on whether e and w are adjacent in G not. The term 1 2m is for normalizing the modularity value, and 8(C., Cw) regulates the algorithm only to consider the edges of specific community. A larger modularity score indicates better quality for the communities.
The modularity above is defined only on a partition, i.c., when the communities are disjoint. Shen et al. (2099) proposed an extension of Eq. for undirected, unweighted graphs with overlapping
3

---

Under review as a conference paper at ICLR 2022
communities, as follows:
kigha 1
Σ (Apw 0,301
uuwnesghtted.oorrto
2m
2m
0,
where O, and Ow is the number of communities containing node v and w. respectively. Thus the modularity becomes larger when the communities have less overlap. [Cen et al. (2010) proposed further extension for weighted, undirected graphs with overlapping communities:
1
k
Q Qwiihhted,overlap
(Aew
)occokcos
(3)
2m
2m
NEC v.W
where Oevv is the belonging coefficient (Nicosia et al.
2009) for a community and vertex e is defined as O'ev = Eeco and hicu Epee Wep. where C' is the set community in graph G and
K..
E C. Wop the weight of the edge between node and p. In Eq. if node belongs only one community c, Qce is equal to 1; if node v does not belong to community c, Qcee is equal to O, which ensures the consistency communities similar to Eq. but unlike Eq. Di considers the edge weights too. The benefit of Eq. over other equations is, a pair of nodes within community with higher edge weight contributes more to the quality measure compared to the pairs with low edge weight.
Eq. with [q.[] Compared to Eq. I Eq. Bconniderss overlapping
Entropy. Entropy is metric that measures the level of uncertainty in system, and rich body of research examines different ways of measuring entropy (Borowska 2015). Shanon entropy is widely used metric in information theory (Shannon 2001). which calculates the average information available based on the probability of a variable's $ possible outcomes. Let X be a discrete random variable with possible outcomes 21, T22,..,, which occurs with probabilities P(x1), P(z))...., P(rx), then the Shanon entropy, H of the variable X is defined as follows:
H
Σ P(x:) log P(z.).
(4)
A higher value of Shanon's 's entropy indicates more uncertain outcome, which is difficult to predict.
4 HYPOTHESIS
We assume that if a neuron is frequently activated with a high activation value for a particular class, then the neuron is representative of that class. Consider graph with nodes neurons and edges representing simultancous activation of pairs of neurons. We refer to such graph as an 'activation pattern graph', and describe the details Section 3.1 Although one neuron can be representative for multiple classes, over the training, we expect the class representatives to have less overlap. Hence we also expect the activation pattern graph evolve such that the neurons representing a class form community. Furthermore, over the training, the neurons are expected to have more certainty in their activation pattern for various classes (rather than being random). Therefore, we expect the 'activation pattern entropy", a measure related to the predictability of neuron behavior, as described later in Section 5.2 to decrease. In particular, we examine the following hypotheses.
H1: The neurons that are frequently activated together form a community in activation pattern graph.
H2: The modularity of the activation pattern graph is related to a deep learning model's performance.
H3: The entropy of the activation pattern is related to a deep learning model's performance.
5
METHODOLOGY
In this section, we describe the activation pattern graph and activation pattern entropy. which are at the core of our methodology and experimental design.
5.1 ACTIVATION PATTERN GRAPH
Assume that there are C classes in the training dataset, and let D., where i k, be the subset corresponding the ith class. Let & be fully connected layer in the neural network with neurons.
4

---

Under review as a conference paper at ICLR 2022
By the notation vg(q, l, d), we denote the activation value of neuron 9 E for data d that belongs to the ith class. Thus the average activation value of a neuron q in layer & for a class is as follows:
V)la, EdED, si(g,ed)
(5)
|D:]
We construct the activation pattern graph G of a layer as follows. We first calculate each neuron's s average activation value for a class. Next, for every class, we select S neurons with the highest average activation value and take their union to create the vertex set of G. Let p. q be a pair of neurons in S and let óp.q be the number of data elements in D,, where both and q obtain activation values that are larger than their individual average activation values, i..., dp.g Kd d D., vs U, l, (,d)>V e) and U; (g. l, d.).))>aa Vi(g, H. Then, for class i, we create an adjacency matrix, A = (Ap.g)nxns where
Epg if p E S and g E e Di 0, otherwise.
dp.4
The final adjacency matrix, Ag of the activation pattern graph is computed as A - A.c. Thus intuitively, an activation pattern graph maintains set of neurons that are frequently activated, where a weighted edge between a pair of neurons represents the frequency of their simultaneous activation. We created a graph for every iteration and every fully connected layer (except the output layer).
5.2 ENTROPY OF ACTIVATION PATTERN
In a neural network with fully connected layers and ReLUU (Rectified Linear Unit) as the activation function, the activation of each neuron is calculated ReLLi maz(0.)) y), where y is the output of a layer. Let Wi and Bg be the weight and bias of layer E of neural network. Then for an input I, the output of the layer is calculated y, (We x I) BB, At the beginning of training, the weights and biases are randomly initialized, and over the training, these values are updated for improved predictions. The neuron activation thus gets more and more influenced by the classes present in the data. Due to random initial weights, the activation of the neurons also becomes unpredictable. In such a case, the activation patterns' entropy is high. reflecting the system's 's randomness. As the training progresses, the activation pattern is biased by the data, and thus the entropy will decrease.
We now compute the activation pattern entropy of particular neuron, which is based on the idea of measuring the predictability of its activation value. We create [D] x N activation pattern matrix, F, where |D] is the size of the training dataset, and N is the total number of neurons in the architecture, except for the neurons in the last layer (output layer). Each entry (i. of F contains the activation value of the jth neuron subject to the ith element of the dataset D. We then compute a normalized matrix F Frorm by dividing each column by the column sum, i.e., Fnorm (1,
F(i.)
ΣE F(J))
To examine the predictability of the activation value for jthh neuron, we categorized its normalized activation values using R equal size bins. In other words, we create a histogram for the jihh column values of Faorm j). The intuition is that if the neuron's activation is unpredictable, then the histogram will not have well-defined maxima or contain many local maxima. Otherwise, it will be activated for one or only a few classes and likely to produce a global peak. There is an exception, where a neuron may never be activated and will create a global maxima the bin that contains the value. Therefore, to create the histogram, we only consider the non-zero activation values. Let Bg and Hg. where < l < R. be the ktth bin and its number of elements. Let h2 be the normalized value, Hg Ej for the jih neuron. E,- = - R - -h, log(h;). The activation pattern entropy, E, of a deep learning model over all the neurons in all the fully connected layers is calculated as E Σ Ej.
i.c., h ΣR, H. We use the vector F.- [hr..... hR] to compute the activation pattern entropy
6 DATASET AND MODEL ARCHITECTURE
In our study, we used three benchmark datasets: MNIST (LeCun et al.) 2010), Fashion MNIST (Xiao et al. 2017), CIFAR-10 (Krizhevsky et al. 2009) and one real-life dataset Plant village (Mohanty 2018). We also created MNIST Mixed and Fashion MNIST Mixed by ran- domizing the labels of the respective datasets to examine the reliability of the proposed metrics.
5

---

Under review as a conference paper at ICLR 2022
The architecture for each dataset is illustrated in Table The MNIST, Fashion MNIST, CIFAR-10 datasets each consists of 10 classes. There 60,000 grayscale 28 x 28 im- ages for training and 10,000 for test- ing in MNIST and Fashion MNIST, and 50,000 grayscale 32 x 28 images for training and 10,000 for testing in CIFAR-10.
Table Details of the architecture of the deep learning mod- els for different datasets. In the model we only used fully connected (FC) and dropout layers.
Fully
Training Accuracy (%)
Testing Accuracy (%)
Daraset
Image
Size
Connected Layers
Output Layer
Itera- tions
MNIST
28x28
512, 512
99.56
98.24
MNIST
Mixed
28x28
512, 512
10
20
12.44
9.85
Fashion MNIST
28x28
512, 512
10
20
91.80
88.9
The Plant village dataset consists of 28693 segmented color images of the healthy apple, blueherry and cherry, corn, grape, orange, peach and pep- per, soybean, strawberry and squash, and tomato classes, and 6890 testing images. We transformed the color images grayscale and reshaped the images to size 256 x 256.
Fashion MNIST Mixed
28x28
512, 512
11.02
10.12
1024, 1024,
CIFAR-10
32x32
1024, 512,
46.24
43.43
256, 128
8192, 2048,
Plant Village
256x256
1024, 512.
100
80.65
79.55
2556, 128, 64
For all the models, apart from the input and output layers, we only used fully connected layers and dropout layer. For the fully connected layers, we used ReLU as the activation function, and Softmax activation for the output layer. For each model, we collected the weight values from 20 iterations at a uniform interval, where each iteration represents training the model with the dataset once. We denote the corresponding activation pattern graphs as G(1).G(2),
G(2)..... ...)....... G(20). Since different models needed a different number of training iterations, this interval length varies, e.g. the CIFAR-10 model collected every two iterations, whereas the Plant Village model every five iterations. See the supplementary materials for detailed model architecture.
RESULT AND DISCUSSION
In this section, we describe the experimental results. To create the activation pattern graphs, we took 50 neurons per class (i.e., choose |S| 50 in Eq. 6 with the largest average activation value. We repeated the experiments for the top 25 and top 100 neurons per class and achieved similar results.
7.1 COMMUNITY FORMATION (H1)
Let G(k) be pattern activation graph of a fully connected layer at iteration k and let N; be the set of neurons which are representative of the ith class, i.c., neurons that are frequently and highly activated for that class. The unique neurons of a pair of classes and j are the neurons that belong to both N₂ and Nj, but in other N₁ where k ji In other words, these neurons represents reeeeseeeeeeee both classes and j. but no other class. If i = j. then we obtain the unique neurons of class i. We observed that in most cases, the number of unique neurons for a single class increases with training, whereas unique neurons for pair of distinct classes decrease. This behavior indicates that the overlap between the communities is decreasing and the classes are forming stronger community among themselves. Table 2 compares the confusion metrics obtained from G(1) and G(20) for the MNIST dataset, where each entry the diagonal of matrix represents the unique neurons for a single class. We observed similar behavior for all the datasets.
Training Accuracy and Average Community Size: We used Gephi (Bastian et al., 2009 a widely used graph visualization software, to compute force directed layouts (Jacomy et al.) (014)) of the activation pattern graphs different iterations. The communities were detected based on the Louvain method (Blondel et al. 2008). Although the visualization revealed community structures in these graphs, the change in the communities over the number of iterations was not readily visible from the graph layouts (see the supplementary material). Hence, quantitatively examined how the number of nodes per community varies over the training. For each dataset, we used Gephi's modularity function with resolution value of 0.5 to find the communities in G(1), G(2), G(20). Table

---

Under review as a conference paper at ICLR 2022
Table 2: Two confusion metrics computed from layer for MNIST dataset. Each matrix represents
the number of unique neurons for pairwise classes. Darker red color represents higher value.
lteratonn
Iteration 20
20
13
13
13
17
17
11
4
19
6
5
12
13
10
18
4
13
7
4
4
11
11
11
14
14
28
3
8
19
5
10
10
12
24
14
12
18
5
15
14
20
4
10
19
15
14
14
22
7
3
20
13
16
13
12
15
10
10
21
12
3
20
10
24
10
21
Table 3: Number of nodes in different communities of the activation pattern graph (detected by Gephi). For each dataset, the communities of G(1) and ((00) are shown along with the PCC between the average node per community and training accuracy. Darker red color represents higher value.
MNIST
Fashion MNIST Mixed
Iteration
C1
C2
C3
C4
C5
C6
C7
C8
PCC
Iteration
C1
C2
C3
C4
C5
C6
C7
C8
PCC
33
34
25
26
9
7
6
5
4
10
7
0.28
MNIST Mixed
CIFAR-10
Iteration
C1
C2
C3
C4
C5
C6
C7
C8
PCC
Iteration
C1
C2
C3
C4
C5
C6
C7
C8
PCC
5
12
12
11
6
6
13
14
10
0.80
Pashion MNIST
Plant Village
Iteration
C1
C2
C3
C4 C5 C6
C7
C8
PCC
Iteration
C1
C2
C3
C4
C5
C6
C7
C8
PCC
40
26
26
21
6
3
20
32
27
25
21
15
15
12
11
0.71
34
shows the number of nodes in the eight largest communities C1, C......, Cs in layer 2 for different datasets and for two different iterations and 20, where the distribution of community sizes appears to be less skewed over the training. We then perform the Pearson Correlation Coefficient (PCC) between the training accuracy and average node per community over the 10 iterations. For most of the datasets, we observed the average node per community to have a positive correlation with the training accuracy higher than 0. However, the weak (near-zero) correlation
(Table 3), i.e., the PCC correlation coefficient for MNIST Mixed and Fashion MNIST Mixed represents the randomness of the formation of the communities of the model. This indicates that, for models with good performance, the number of well defined communities increases with training and supports hypothesis H1
7.2 MODULARITY AND ACCURACY (H2)
So far, we have observed that community detection can reveal meaningful clusters in the pattern activation graph, i.e., clusters are related representative neuron for different classes. We now examine the other side, i.e., can the representative neuron sets for different classes be seen as well as the defined communities? To assess this, we compute different modularity metries (Eq. [ for the activation pattern graphs G(I), .,, G(20). We also used different community detection algorithms available in the python library NetworkX (Hagberg et al., 2008) and iGraph (Csardi & Nepusz 2006). Note that these algorithms take an initial partition of the neurons and then change those partitions to optimize some quality measure. Therefore, they are suitable in this context, yet, the Kernighan Lin Bisection (KLB) (Kernighan & Lin 1970) could generate consistent results.
We computed the Spearman correlation coefficient (SCC) and Pearson correlation coefficient (PCC) to examine the potential relation between modularity and training accuracy (Table A positive correlation value indicates that the communities' quality is positively correlated with training accuracy. We observed the same relation with the test accuracy. A large number of strong positive correlation coefficients in Table supports hypothesis H2. A stronger correlation is observed in later layers for all the dataset. For most datasets, the unweighted overlap (Eq. (2). and weighted overlap metric (Eq.3) captures the relation better, which is expected since the no-overlap metric neither considers weight nor the community overlap.
7

---

Under review as a conference paper at ICLR 2022
Table 4: Spearman and Pearson correlation coefficient between modularity and training accuracy for different layers. Darker blue represents higher values and darker red represents lower values. (See the supplementary material for the relation with test accuracy.)
MNIST Mixed
Fashion MNIST
Fashion MNIST Mixed
MNIST
L1
L2
L1
L2
LI
L22
L1
L2
PCC
SCC
PCC
SCC
PCC
SCC
PCC
SOC
PCC
SCC
PCC
SCC
PCC
SCC
PCC
SCC
KLB
0.24
0.16
0.50
0.61
-0.76
-0.81
40.73
-0.84
-0.04
-0.10
0.42
0.44
-0.03
-0.07
-0.44
-0.38
Na-overlup
0.22
0.19
0.23
0.19
00.4
0.55
0.46
0.37
0.42
0.46
0.62
0.61
0.28
0.40
0.14
0.29
Unweighted,
Overlap
0.37
-0.70
-0.73
0.36
0.37
0.67
0.77
0.43
0.51
Waighted,
Overlup
0.33
0.77
0.57
-0.63
-0.69
0.48
0.47
0.56
0.46
0.78
0.79
0.43
0.51
0.26
0.38
CIFAR-1
L1
L22
1.3
1.4
L.5
L6
PCC
SCC
PCC
SCC
PCC
SCC
PCC
SCC
PCC
SCC
PCC
SCC
KLB
0.16
0.20
0.68
0.74
0.69
0.73
-0.18
-0.13
-0.29
-0.40
-0.01
0.06
No-overlap
-0.05
-0.11
0.62
0.66
0.29
0.27
0.06
-0.05
0.01
0.03
-0.49
-0.44
Unweighted,
Overlap
0.34
0.44
0.87
0.86
0.73
0.82
0.82
0.79
0.66
0.77
-0.37
-0.29
Weighted,
Overlap
0.42
0.55
0.78
0.68
0.71
0.78
0.71
0.70
0.71
0.79
0.02
00.03
Plant Village
L1
L2
133
L4
L55
L6
L7
PCC
SCC
PCC
SCC
PCC
SCC
PCC
SCC
POC
SCC
PCC
SCC
PCC
SCC
KLB
0.22
0.16
0.57
0.62
0.76
0.61
0.84
0.83
40.16
0.02
00.34
0..00
.0.21
-0.19
No-overlup
0.24
0.15
0.05
0.03
0.40
0.42
0.15
0.30
0.04
0.00
0.03
0.17
0.46
0.58
Unweighted,
Overlap
0.80
0.91
0.65
0.46
0.74
0.82
Weighted,
Overlap
0.87
0.88
0.78
0.66
0.72
Table 5: Spearman and Pearson correlation coefficient between the entropy and training accuracy. Darker blue represents higher values and darker red represents lower values.
MNIST
Fashion
Fashion
Plant Village
MNIST
Mixed
MNIST
MNIST
CIFAR-10
Mixed
PCC
0.87
0.22
-0.55
0.57
-0.55
-0.94
SCC
-0.44
0.51
-0.99
0.67
0.97
7.3 ENTROPY AND ACCURACY (H3)
A higher entropy value represents more randomness in the activation behavior of a neuron. At the beginning of the training, due to random weights, the activation of the neuron is random. So, the entropy of the activation pattern over all the fully connected layers should be higher. As the training progresses, the entropy should decrease, representing biased neuron activation behavior.
Table shows a negative correlation between the entropy of activation pattern and training accuracy over model training for most of the datasets. We observed the same relation with the test accuracy. Figure [ shows the change of normalized entropy with training and testing accuracy for different iterations for all the datasets, where clear relation between the entropy and model performance can be observed. However, MNIST Mixed and Fashion MNIST Mixed has positive correlation values, indicating the absence of activation pattern with training due to it's random labels, which is consistent with hypothesis H3. Note that the evidence for hypothesis H3 is supportive of H1 and H2, where we examined the community behavior of the neurons and observed better community structure except for MNIST mixed and Fashion MNIST mixed.
We also examine entropy change for individual classes (Table 6) where to compute the entropy of class, we take the same approach in Section 5.2, but use only the training data corresponding to that class. In this setting, we obtain meaningful results (i.e., a negative correlation between the entropy and training accuracy) only for all well-trained models (MNIST, Fashion MNIST, and Plant
8

---

Under review as a conference paper at ICLR 2022
Table 6: Spearman and Pearson correlation coefficient between entropy and training accuracy for
individual classes. Darker blue represents higher values, and darker red represents lower values.
MNIST
Fashion MNIST
Fashion
Plant
MNIST
Mixed
MNIST
CIFAR-10
Village
Mixed
Class
PCC
SCC
PCC
SCC
PCC
SCC
PCC
SCC
PCC
SCC
PCC
SCC
-0.5
-0.67
-0.06
-0.29
0.21
-0.13
-0.4
0.22
0.17
-0.85
-0.89
2
0.34
40.71
0.18
40.16
-0.24
-0.45
0.77
-0.15
0.7
0.66
-0.86
-0.93
3
0.22
-0.31
0.47
-0.02
0.37
0.22
0.47
0.31
0.38
-0.1
0.46
-0.42
0.35
-0.7
0.36
0.11
.0.57
0.84
0.25
-0.18
0.13
0.16
0.88
.0.22
-0.1
-0.73
0.33
-0.1
0.48
-0.40
-0.53
-0.39
-0.25
-0.35
-0.52
-0.14
-0.07
-0.67
-0.67
0.33
-0.17
-0.5
-0.51
-0.86
-0.36
0.16
0.19
0:44
.0.83
00.95
0.67
0.17
0.59
-0.17
-0.87
-0.89
-0.09
-0.29
0.33
-0.43
-0.62
0.7
0.3
0.16
0.19
-0.91
-0.89
0.67
40.8
0.46
0.62
0.9
0.96
0.63
0.33
0.01
0.14
.0.81
-0.86
10
0039
-0.77
0.09
0.01
-0.77
0.60
0.90
0.86
-0.18
-0.26
MNNST
MINIST Mixel
Finkioo MNNST
0.806
0330
0.200
14
19:2 Avmonyy
12
Thickng Boreeyy
Triking Senwery
Bording Aveniyy
Fishion MNIST Mixed
CIFAR-10
Plant Village
1,100
1100
1.000
0.8000
0330
0,400
0430
04003
18
15
18
Takikn Toding Autter
Figure 1:
Change of normalized entropy, training and testing accuracy over iterations. Since values normalized, a proper comparison should focus trends instead of individual peaks or drops.
Village). However, there are few classes with positive and small negative correlation between entropy and accuracy. This is due to frequent change in the training accuracy of that class. For CIFAR- 10, MNIST Mixed, Fashion MNIST Mixed, we observed a positive correlation values in most of the classes, and sometimes even found the correlation coefficient be strongly positive. This due to the inability of the model to separate the classes from one another.
LIMITATIONS AND FUTURE WORK
We have proposed novel methods explain the behavior of neurons in deep learning models as the model training progresses. We used graph theoretic and entropy-based methods to model the activation pattern of the neurons. We quantitatively showed that neurons that are highly activated for a class form a community in our graph model. For the entropy-based approach, we analyzed the activation pattern using the Shanon entropy and found the entropy show negative correlation with the training accuracy.
There are many scope for future research. We used deep learning models with only fully connected layers, using different model architecture may help understand the entropy and modularity behavior with more granularity. Although we examine a diverse set of datasets, adding more datasets could strengthen the results. Our experimental results show that the modularity of the activation pattern graph and entropy of the activation pattern is related to the model's performance. Although we used widely used quality measures for the modularity and entropy, there is still scope for designing better quality metrics specifically for neural network context. Another exciting direction of study can be to use the modularity and entropy better optimize the performance of the deep learning models. We believe that our results will inspire further research explain the deep learning models using graph and information theoretic methods.
regularization in the loss function (Leavitt & Morcos 2020) to
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pietro Lió, Marco Gori, and Stefano Mclacci. Entropy-based logic explanations of neural networks. arXiv preprint arXiv:2106. 066006604 2021.
Mathieu Bastian, Sebastien Heymann, and Mathieu Jacomy, Gephi: an open source software for exploring and manipulating networks. In Proceedings of the International AAAI Conference on Web and Social Media, volume 3, 2009.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. Proceedings of the IEEE conference computer vision and pattern recognition, pp.
2017.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role individual units in deep neural network. Proceedings of the National Academy of Sciences, 117(4):30071-3007 2020.
Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008
(10):P10008, 2008.
Marta Borowska. Entropy-based algorithms the analysis of biomedical signals. Studies in Logic, Grammar and Rhetoric, 43(1):21-32, 2015.
Duanbing Chen, Mingsheng Shang. Zehua Lv. and Yan Fu. Detecting overlapping communities of weighted networks via local algorithm. Physica A: Statistical Mechanics and its Applications, 389(19):417-41187 2010.
Sunghyo Chung, Sangho Suh, Cheonbok Park, Kyeongpil Kang. Jaegul Choo, and Bum Chul Kwon. Revacnn: Real-Time visual analytics for convolutional neural network. 2016.
Aaron Clauset, Mark Newman, and Cristopher Moore. Finding community structure very large networks. Physical review E, 70(6):06611) 2004.
Gabor Csardi and Tamas Nepusz. The igraph software package for complex network research. InterJournal, Complex Systems: 1695, 2006. URL https::////// URttttts::///rrraaaaa //inr/ph.///..../ org
Andrei Dobrescu, Mario Valerio Giuffrida, and Sotirios Tsaftaris. Understanding deep neural networks for regression in leaf counting. In Proceedings of IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Jordi Duch and Alex Arenas. Community detection in complex networks using extremal optimization. Physical review E, 72(2):027104 1000 2005.
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8730-8738, 2018
Santo Fortunato and Claudio Castellano. Community structure in graphs. arXiv preprint arXiv:0712.2716, 2007.
Adam Gaier and David Ha. Weight agnostic neural networks. arXiv preprint 19Xiv:1906.04355. 2019.
Aurélien Géron. Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems. O' O'Reilly Media, 2019.
Scott Gigante, Adam S Charles, Smita Krishnaswamy, and Gal Mishne. Visualizing the phate of neural networks. In NeurIPS, 2019.
Elliott Mordon--Rodrggun Gabriel Loaiza-Ganem, Geoff Pleiss, and John Patrick Cunningham. Uses and abuses of the cross-entropy loss: case studies in modern deep learning. 2020.
Robert M Gray. Entropy and information theory. Springer Science & Business Media, 2011.
10

---

Under review as a conference paper at ICLR 2022
Aric Hagberg, Pieter Swart, and Daniel Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LAN)) Los Alamos, NM (United States), 2008.
Adam W Harley. An interactive node-link visualization of convolutional neural networks. In International Symposium Visual Computing, pp. 867-877. Springer, 2015.
Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. Visual analytics in deep learning: An interrogative survey for the next frontiers. JEEE transactions visualization and computer graphics, 25(8):2674-25) 2693, 2018.
Fred Hohman, Haekyu Park, Caleb Robinson, and Duen Horng Polo Chau. ummit: Scaling deep learning interpretability by visualizing activation and attribution summarizations. IEEE Iransactions on visualization and computer graphics, 26(1):1096-1 1066, 2019.
Yujia Huo, Derek F. Wong. Lionel M. Ni, Lidia S. Chao, and Jing Zhang. Hetropy: Ex-
plainable learning diagnostics via heterogeneous maximum-entropy and multi-spatial knowl- edge representation. Knowledge- Based Systems, 207:106389, 2020. ISSN 0950-7051. doi:
http://orrrr...111...oooooooooo 1033300999 URLpttps: htttpp://www.cccr http://wwwwiieeeeerettttttte com/ sci Bccieneee/arrriill////////911103
Mathieu Jacomy, Tommaso Venturini, Sebastien Heymann, and Mathieu Bastian. Forceatlas2, continuous graph layout algorithm for handy network visualization designed for the gephi software. PloS one, 966):e98679, 2014.
Minsuk Kahng, Pierre Y Andrews, Aditya Kalro, and Duen Horng Chau. Activis: Visual exploration of industry-scale deep neural network models. IEEE transactions on visualization and computer graphics, 24(1 ::8 :1:8))887777 97, 2017.
Brian W Kernighan and Shen Lin. An efficient heuristic procedure for partitioning graphs. The Bell system technical journal, 49(2):291-307, 1970.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pp.
PMLR, 2018.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). 2009. URL TtRL:htp:///wwt /wwp: cs www.ntto.ed./kkkiii....... ftt.mlll
Matthew Leavitt and Ari Morcos. Selectivity considered harmful: evaluating the causal impact of class selectivity in dnns. arXiv preprint arXiv:2003.01262: 2020.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann. www.//an..lec.....om lecun. wwwp:/wwwww.......///e./.ccc. wwwp//ee////biissst.oooomomo 2, 2010.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 531(7553):436 444, 2015.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. arXiv preprint arXiv:1712.09911. 2017.
Li Li, Miloš Doroslovački, and Murray H Loew. Approximating the gradient of cross-entropy loss function. IEEE Access, 6:111626-111633 2020.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188-5196, 2015.
Manuel Martinez and Rainer Stiefelhagen. Taming the cross entropy loss. In German Conference on Pattern Recognition, pp. 628-637. Springer, 2018.
P. Mohanty. Plant village. Plantvi 1werlllageeaaa Dataset. 2018.
https://atpa: ctt.//t...ooooeeeaoooooooaoeeeee http://..coo/aaahht.......

---

Under review as a conference paper at ICLR 2022
Mark EJ Newman. Fast algorithm for detecting community structure in networks. Physical review E, 69(6):066133, 2004.
Mark EJ Newman and Michelle Girvan. Finding and evaluating community structure in networks. Physical review E, 69(2):26613, 2004.
Vincenzo Nicosia, Giuseppe Mangioni, Vincenza Carchiolo, and Michele Malgeri. Extending the definition of modularity to directed graphs with overlapping communities. Journal of Statistical Mechanics: Theory and Experiment, 2009(03):P03024, 2009.
Scong Joon Oh, Bernt Schiele, and Mario Fritz. Towards reverse-engineering black-box neural networks. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 121- 144. Springer, 2019.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7, 2017.
Arun Rai. Explainable ai: From black box to glass box. Journal of the Academy of Marketing Science, 48(1):137-141, 2020.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-22) 215, 2019.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148 175, 2015.
Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
Huawei Shen, Xueqi Cheng. Kai Cai, and Mao-Bin Hu. Detect overlapping and hicrarchical community structure in networks. Physica A: Statistical Mechanics and its Applications, 388(8): 1706-1712, 2009.
Marcel Simon, Erik Rodner, and Joachim Denzler. Part detector discovery in deep convolutional neural networks. Asian Conference on Computer Vision, pp. 162-177. Springer, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: 13/2.6034 2013.
Daniel Smilkov, Shan Carter, D Sculley, Fernanda B Viégas, and Martin Wattenberg, Direct- manipulation visualization of deep networks. arXiv preprint 1rXiv:708.033888 2017.
F-Y Tzeng and K-L Ma. Opening the black box-data driven visualization of neural networks. IEEE, 2005.
Kanit Wongsuphasawat, Daniel Smilkov, James Wexler, Jimbo Wilson, Dandelion Mane, Doug Fritz, Dilip Krishnan, Fernanda B Viégas, and Martin Wattenberg, Visualizing dataflow graphs of deep learning models in tensorflow. IEEE transactions on visualization and computer graphics, 24(1): 1-12, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1008.07747 2017.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 318-833. Springer, 2014.
Haipeng Zeng, Hammad Haleem, Xavier Plantaz, Nan Cao, and Huamin Qu. Cnncomparator: Comparative analytics of convolutional neural networks. arXiv preprint 1rXi::1710.05285, 2017.
Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu. and Long-Chun Zhu. Interpreting enn knowl- edge via an explanatory graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
12

---

Under review as a conference paper at ICLR 2022
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. arXiv preprint arXiv:4412.6856, 2014.
Yangfan Zhou, Xin Wang, Mingchuan Zhang, Junlong Zhu, Ruijuan Zheng, and Qingtao Wu. Mpce: maximum probability based cross entropy loss function for neural network classification. IEEE Access, 7:146331-146444 2019.
13


Reviews:
[{"review_id": "pDR6-1DbZcS", "reviewer": "Reviewer_Xmnm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Suitable and persuasive application of the graph and community structure to explain  the activation behavior and relations of the neurons.\n- Appropriate usage of the entropy and modularity score to effectively measure the tendency and the predictability of the neuron activation.\n- Intuitive hypotheses to validate the correlation between suggested methods and the behavior of the neurons.\nWeaknesses\n- As authors mentioned in the discussion section(8: Limitations and Future Work). The scalability of this work is suspicious. There ‘s no guarantee that proposed activation analysis methods will also show the same correlation tendency in other neural networks with more complicated architectures than  multi-layer perceptron networks.\n- To compute the entropy, activation pattern matrix is required for each layer. The spatial complexity of the matrix is the multiplication of the total number of neurons and the size of a training data. Computational overload will be inevitable when it’s trained on larger data and models.\n- Aside from the architecture, the authors only had done experiments on the dropout conditions. Experiments on other various hyperparameters/conditions that can affect neuron configurations or learning ability of the neural network(e.g., pruning) could add more validities to their ‘comprehensive experimental study’.\n", "summary_of_the_paper": "This paper examined novel neuron activation pattern analysis on neural network classification models via graph theoretic and entropy-based methods. The authors showed reliability of their work by approving hypotheses on examining the qualitative correlation between model performance and activation patterns. The main technical contribution of this paper comes from explaining the neural classifiers by combining the graph-theoretic and information-theoretic approaches.", "summary_of_the_review": "The suggested explanatory methodologies and approaches were pretty persuasive to explain the network behavior in microscopic context. Nevertheless, the scalability and practicality of the work should be examined carefully. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YIteKrC-X0m", "reviewer": "Reviewer_RKXh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Understanding why deep learning works by inspecting the internal representations beyond loss value or performance (accuracy) is an important topic. This paper tries to advance our understanding with two newly proposed metrics, which is a good point. The writting and logic is clear.\n\nHowever, several major concerns are obvious:\n\n1. Not enough literature survey. The authors claim this paper \"lay the initial groundwork for graph and entropy-based studies to analyze the deep learning models’ performance\", but entropy is prevalent in deep learning. Given the bold literature on both understanding deep learning and entropy utilization, it is hard to say this paper is the first to do so.\n\n2. No baseline methods to compare the significance of the two proposed metrics. There are many existing approaches in explaining deep learning models, such as turning deep models into decision trees in \"Distilling a Neural Network Into a Soft Decision Tree\". Then complexity measurement of the decision tree can be a baseline. Or counting the number of linear regions induced by neural networks like \"On the number of linear regions of deep neural networks\" is also a feasible idea. Frankly speaking, I'm not an expert in explaining deep learning, but I believe there have exist many methods to explain deep models' performance. The significance of methods proposed in this paper should be compared against existing approaches.\n\n3. Experiments are rather preliminary, which only involves some toy datasets and MLP networks. It is not clear whether the method scales to larger and real-world problems, as admitted by authors in the paper.\n\nBy the way, I'm curious about how the experiments are done. In section 6, what do you mean by \"iteration\"? Is it one mini-batch or one epoch? What is the batch-size? The results are quite strange because I see that 99.56% accuracy is achieved within just 20 iterations.", "summary_of_the_paper": "An important problem of understanding the performance of deep learning models is studied by this submission. Two graph-theoretic and information-theoretic metrics are checked in some datasets with MLP networks. Preliminary results with expected correlation are observed.", "summary_of_the_review": "I acknowledge the importance of the research direction in this paper. The contribution of this paper is not properly compared against existing methods, making it difficult to tell their significance. Meanwhile, the toy setting in experiments makes me doubt whether the techniques can be extended to real-world scenarios.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "V0zAsgYksTD", "reviewer": "Reviewer_Gvju", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose two methods to analyze the behavior of neurons in neural nets. The main idea is to study the neuron activation\npatterns of classification models and explore if the performance can be explained through neurons' activation behavior. The authors propose two approaches: one that models neurons' activation behavior as a graph and examines whether the neurons form meaningful communities, and the other that examines the predictability of neurons' behavior using entropy.\n\nThe paper is interesting, easy to read, and very well-structured. Furthermore, the subject of the paper is very relevant: the exploration of tools to increase the understanding and explainability of neural models. However, I have several important questions/comments to the authors:\n\n1) In page 6, the authors clearly state that \"we only used fully connected (FC) and dropout layers\". However, in the supplementary material (page 1), the authors present a graphical representation of the deep learning model architecture used, and they include convolutional blocks. Three questions in this regard:\n- do the authors employ or do not employ only fully connected layers?\n- if what is stated in the paper is correct, why to employ only fully connected layers if the problem tackled is image classification? I think it would be more natural to employ ConvNets. \n- do the authors think that the conclusions extracted would also apply to ConvNets and/or other neural models and problems?\n\n2) The authors discuss three main hypotheses:\nH1: The neurons that are frequently activated together form a community in the activation pattern graph.\nH2: The modularity of the activation pattern graph is related to a deep learning model's performance.\nH3: The entropy of the activation pattern is related to a deep learning model's performance.\n\nRegarding H1, the authors conclude that \"for models with good performance, the number of well defined communities increases with training\". In turn, H2 can contribute to find representative neuron sets for different classes, and authors conclude that \"the communities' quality is positively correlated with training accuracy\". However, if I'm not mistaken, the main conclusion related to H3 is that, at the beginning of the training, due to the randomness of the initial weights, the activation of the neurons becomes unpredictable (high entropy); while, as the training moves forward, the activation pattern is biased and thus the entropy will decrease. It is not totally clear to me how this hypothesis (H3), and the associated experimentation and conclusions, effectively contribute to a better understanding and evaluation of the performance and explainability of neural networks. \n\n3) Figure 3 of the supplementary material shows that the entropy, first, decreases with iterations (which fits the hypothesis managed in the paper: \"As the training progresses, the entropy should decrease, representing a biased neuron activation behavior\") but from the 8th-9th iteration it dramatically increases. What is the explanation of the authors for this, and how this fits the initial assumption?\n\n4) All experimental evaluation is performed on quite \"small\" architectures. Is it possible that the explanatory and exploratory capacities of the presented methods are diminished with much larger/deeper models? \n\n5) Also regarding the experimental configuration, it is unclear what exactly an \"iteration\" is, or what are the different hyperparameters involved (learning rate, batch size, etc. and how these were selected). In other words, I have the feeling that the results are hardly reproducible. On the other hand, there are no competitor approaches to compare the performance obtained by these metrics/methods. It would be interesting to see how the techniques presented in this paper compare to other techniques already present in the literature. \n\n", "summary_of_the_paper": "The authors propose two methods to analyze the behavior of neurons in neural nets. The main idea is to study the neuron activation\npatterns of classification models and explore if the performance can be explained through neurons' activation behavior. The authors propose two approaches: one that models neurons' activation behavior as a graph and examines whether the neurons form meaningful communities, and the other that examines the predictability of neurons' behavior using entropy.", "summary_of_the_review": "Interesting paper and relevant subject. But I think there are some inconsistencies in the paper, and the overall impact, conclusions and applicability of the paper is somewhat dubious.   ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TsFgNx0xVn8", "reviewer": "Reviewer_8T7r", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The metrics and evaluation procedures are clearly explained. Developing new metrics to measure models activations and relating them to model performance an exciting research direction. \n\nWeaknesses:\n- The metrics introduced in this paper do not immediately suggest a way to build higher performance models or provide significant insight into what the model's have learned. Just because a representational metric is correlated with a model's performance does not mean that it causes a model's performance improvement. It also does not mean that a representational metric is predictive of the performance of the fully trained model. I would be more convinced that these are important metrics if they could be incorporated as a regularization term in a loss function (Ex. Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs (Leavitt & Morcos 2020)) or they could be used to predict which models will generalize better (Ex. Fantastic Generalization Measures and Where to Find Them (Jiang et al. 2019))\n- I do not think that the entropy metric is entirely novel. Many past studies have demonstrated that individual units in neural networks become specialized during the course of training for detecting specific features or classes. (Ex. Understanding the Role of Individual Units in a Deep Network (Bau et. al. 2020))\n- The finding that modularity is correlated with training accuracy, and that the number of well defined neural communities increase with accuracy could only apply to a vanilla SGD model training setup. Model representations are often very flexible and it could be that regularization against modularity results in a model that is still equally high performance, but is much less modular. Such an experiment could demonstrate if the metrics studied in this paper are measuring an important model representation structure necessary for high performance or a representational coincidence of their particular training method.\n\n", "summary_of_the_paper": "This paper proposes new metrics based on the activation patterns of a model. It propose an activation pattern entropy metric and a graph theoretic metric of neuron communities. The authors then evaluated models trained on MNIST, Fashion MNIST, Fashion MNIST mixed, CIFAR-10, and plant village using their new metrics. They found that entropy is negatively correlated with training accuracy, modularity is correlated with training accuracy, and that the number of well defined neural communities increase with accuracy.", "summary_of_the_review": "Overall, the metrics in this paper are sensible and clearly explained, but further experiments are necessary to demonstrate insights into what models are learning or what directions for performance improvements the metrics can suggest.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Understanding neural networks once they have been trained is a big open problem for machine learning. This manuscript designed graph theoretic and information theoretic measures aimed at helping us understand community structure and function in trained networks. In particular, they measure community structure (modularity) and entropy for trained networks and related these to the performance of the networks. The manuscript runs experiments with fully connected networks on problems such as MNIST and CIFAR. Both community structure and entropy measures are shown to correlate (Spearman and Pearson correlation coefficients) with performance metrics in the networks studied. 
Reviewers tended to agree that the paper was well written and motivated by an interesting and timely question (understanding trained networks). However, on the whole, most of the reviewers believe that the manuscript is too preliminary for publication at ICLR and I agree. A central issue cited by most of the reviewers is that the experiments are performed on small/toy models for small tasks and under particular hyperparameter regimes. It is therefore unclear to what extent the results would generalize to other situations. E.g. would the results hold for larger dataset or for convolutional neural networks? Connected to this complaint, reviewers worry that there is not enough connection to the literature and baseline methods that could be used to predict performance given measures of trained network activity. Even allowing that the observed correlations are true and generalizable, are these measures better than those covered elsewhere in the literature? Additionally problematic, the measures are not theoretically justified either. Thus, we are missing both reasoned arguments for the metrics and robust quantification beyond a limitted experimental setting. One reviewer, Xmnm, is compelled by the work and recommends acceptance. However, they do not present a compelling case for acceptance, and even repeat several of the concerns raised by other reviewers. 
In sum, the work is on an interesting subject and timely, but needs further work to be ready for publication.
Decision: Reject