Year: 2021

Abstract:
Deep learning techniques are increasingly being adopted for classification tasks over the past decade, yet explaining how deep learning architectures can achieve state-of-the-art performance is still an elusive goal. While all the training information is embedded deeply in a trained model, we do not yet understand much about its performance by only analyzing the model. This paper examines the neuron activation patterns of deep learning-based classification models and explores whether the models’ performances can be explained through neurons’ activation behavior. We propose two approaches: one that models neurons’ activation behavior as a graph and examines whether the neurons form meaningful communities, and the other examines the predictability of neurons’ behavior using entropy. Our experimental study reveals that both the community quality and entropy can provide new insights into the deep learning models’ performances, thus paves a novel way of explaining deep learning models directly from the neurons’ activation pattern.

heading: 1 INTRODUCTION
text: Deep learning allows the researchers to engineer better features and representation of data through representation learning (LeCun et al., 2015). Despite the widespread usage of deep learning methods, it is still considered a black box, and there is a lack of understanding of the working procedure of the models (Rai, 2020; Oh et al., 2019; Tzeng & Ma, 2005; Rudin, 2019). Researchers often rely on intuition and domain knowledge when designing deep learning architectures (Shahriari et al., 2015). They use the models’ accuracy and loss to evaluate the performance and tune the hyperparameters to optimize the models (Géron, 2019; Gigante et al., 2019). However, to gain a deeper insight into the model, it is crucial to look beyond its accuracy and loss, i.e., we need to design additional evaluation metrics that can provide a level of confidence in the model predictions.
Exploring network architecture helps explain why some models perform better than others. For example, by comparing the weight optimization against model architecture, Gaier & Ha (2019) showed that a model’s performance depends mostly on its architecture. Zhang et al. (2018) created an explanatory graph that can disentangle different part patterns from feature maps of the convolutional neural network (CNN). Visual analytics tools have also been used to understand the architectures better (Hohman et al., 2019; Gigante et al., 2019). However, such approaches sometimes require domain knowledge, and are also subject to human interpretation as they do not provide quantitative measures for the evaluation.
Motivation. In this paper, our main focus is on classification models. A deep learning model that performs a classification task is optimized to minimize the difference between the model’s prediction and the actual value. The prediction is recorded as accurate if it is above a threshold, but this process does not fully leverage the information hidden inside the architecture. The neuron activation pattern that appears during the training contains rich information that, if understood well, can potentially be combined with model prediction to provide a level of confidence in the prediction value. It may also allow us to design more efficient models and additional evaluation metrics to help reduce the number of false negatives and false positives. If it is possible to relate the model’s performance with the neuron activation pattern, we may understand why some models perform better than others by analyzing the pattern. This motivated us to explore different ways to examine neurons’ activation pattern and establish their relation to the training and test accuracy.
Contribution. We study the neuron activation pattern using two approaches: one based on a graphtheoretic model, and the other is based on an information-theoretic model. Our proposed metrics show a correlation with the accuracy on both benchmark and real-life datasets in both cases.
For the graph-theoretic approach, we created a novel ‘activation pattern graph’ model and showed that the set of neurons that are frequently and highly activated for a class often forms a community (i.e., an induced subgraph that contains more edges within than outside of the subgraph) in the activation pattern graph. We observed that modularity, a widely used metric to measure community quality, is closely related to the model’s training and test accuracy.
For the information-theoretic model, we propose a novel method that allows us to measure the predictability of the neurons’ activation behavior leveraging entropy (Gray, 2011). Our experimental results show that the entropy of the neurons is relevant to the model’s training and test accuracy.

heading: 2 RELATED WORK
text: Neuron Activation Pattern. Researchers have investigated both neuron activation patterns and loss function to explain deep learning models. Olah et al. (2017) studied what the neurons respond to and proposed that neurons work in a group. Li et al. (2017) showed a relationship between the performance of the deep learning model and the convexity of the loss function. Mahendran & Vedaldi (2015) studied the feature maps of the CNN models to understand the activation pattern by inverting the models and showed that there is indeed a relation between the performance of the model and the activation pattern. Kim et al. (2018) vectorized the activation values of hidden layers and created striped patterns to find relevance between pattern and decision. However, they experimented with a small dataset without directions for generalizability.
Bau et al. (2017) studied the relation between the activation pattern and semantic concepts. They optimized the hyperparameters of the deep learning models and quantitively analyzed the effect of changing different parameters. The relevance between activation patterns and semantic concepts was further studied by Fong & Vedaldi (2018). They created vector response of semantic concepts based on activation pattern and established that the neurons work in a group, and the same neuron can represent multiple concepts. We found a similar characteristic of the neuron activation pattern in our study, where there was an overlap of neurons representing multiple classes. However, with training, a class was represented by more unique neurons.
Another way of interpreting the deep learning models’ learning is by selecting image patches that maximize the neuron activation (Zeiler & Fergus, 2014). Zhou et al. (2014) described scenes using the units from the feature maps and also proposed that individual units behave as object detectors. Simon et al. (2014) analyzed the deep learning model’s feature maps and found a spatial relation between the activation centers and the semantic parts or bounding boxes of the ground truth images. Zhang et al. (2018) used hierarchical explanatory graphs across layers to propose a way of maximizing a deep learning model’s performance. Bau et al. (2020) discovered that individual units of CNN can learn concepts from the images and some of them strongly influence the decision of the model. Although there have been several attempts to explain activation patterns, in this paper we take a very different approach to the explanation of the neural network models by combining the graph-theoretic and information-theoretic approaches, which allow us to relate various quantitative measures to the model’s performance.
Visualization and Interpretability. Visualization is considered to be a promising approach for explaining complex systems, and as a result, there have been several attempts to explain the deep learning models using visualization tools (Hohman et al., 2018; Chung et al., 2016). A dataflow diagram is a simple way to visualize a deep learning model’s architecture (Wongsuphasawat et al., 2017). However, dataflow diagrams do not tell much about the model’s learning process.
Smilkov et al. (2017) proposed “Tensorflow Playground,” where users can train a model by changing the hyperparameters and observe the training process with various plots and charts. Harley (2015), proposed an interactive visual inspection tool, where a model was trained on the MNIST dataset (LeCun et al., 2010), and for input data, the tool showed which feature maps were activated and also which part of the input data was activating the maps. ACTIVIS (Kahng et al., 2017) is an interactive visualization tool that, along with the dataflow diagram, provides a projection of the neuron activation
for different instances. This only allows users to explore the activation pattern for different samples, classes, and subsets rather than explaining the model’s working procedure. CNNComparator (Zeng et al., 2017) is a tool to compare the performance metrics and the distribution of hyperparameters of two deep learning models.
A rich body of research examines ways of visualizing feature maps of CNN models (Dobrescu et al., 2019; Simonyan et al., 2013; Zeiler & Fergus, 2014). A common limitation of studies that rely on visualization is that the visual assessment is subjective and often hard to evaluate quantitatively.
Entropy and Deep Learning. Entropy is commonly used to quantify errors, e.g., the cross entropy is a widely used loss function (Gordon-Rodriguez et al., 2020). Many researchers proposed modifications to this loss function. Martinez & Stiefelhagen (2018) proposed Tamed Cross Entropy, which has the same convergence property as the cross entropy but is more robust against uniformly distributed label noise, Zhou et al. (2019) proposed Maximum Probability based Cross Entropy (MPCE) loss function, which uses a MPCE based gradient update algorithm and has less back-propagation error than cross entropy, and Gordon-Rodriguez et al. (2020) proposed to use log-likelihood of the continuous categorical distribution in the place of the cross entropy loss used in label smoothing and actor-mimic reinforcement learning.
Barbiero et al. (2021) proposed an entropy-based linear layer for concept-based deep learning models, which utilizes entropy to choose limited subset of input concepts, allowing it to provide concise explanations of its predictions. Huo et al. (2020) implemented a maximum entropy regularizer that encourages uniform weight distribution. Li et al. (2020) approximated the gradient of the cross entropy loss function, which are robust against noise and can avoid the vanishing gradient problem.
Although entropy measures has been used for model optimization, we leveraged it to analyze the activation pattern of the models and propose performance metric that correlates to the model accuracy.

heading: 3 TECHNICAL BACKGROUND
text: Graph and Community Structure. A graph consists of a set of elements (nodes) and a set of pairs of elements (edges), where the nodes represent objects and edges represent pairwise relationships. A community of a graph is defined as a subgraph that contains more edges within than the edges connecting them to the rest of the graph. Community detection algorithms often define a quality measure for the communities and then attempts to find a partition of the nodes that maximizes the quality measure (Fortunato & Castellano, 2007; Newman & Girvan, 2004). Later, we will define graphs with nodes as neurons and edges as their simultaneous activation. We will examine the communities of these graphs based on several quality measures as explained in the next section.
Modularity. Modularity is a widely used metric to assess the quality of a given set of non-overlapping communities (Newman, 2004; Duch & Arenas, 2005; Clauset et al., 2004). The idea of modularity is based on comparing the given graph G with a random graph. Given two communities Ci and Cj in G, we use the notation e(Ci, Cj) to denote the number of edges between these two communities. Let aCi be the fraction of all the edges connecting the community Ci to all other communities, i.e., aCi = ∑ j eCiCj . For a random graph, the fraction of the resulting edges that connect nodes within the community Ci is aCi 2. Hence the modularity (Newman & Girvan, 2004) can be formalized as
Qno−overlap = ∑
i(eCiCi − aCi2. This is commonly expressed leveraging the adjacency matrix of the graph (Blondel et al., 2008). Let Avw be the adjacency matrix of G, and let m be the number of edges in G. For a vertex v (resp., w) in G, we denote its degree and community by kv and Cv (resp., kw and Cw). Then
Qno−overlap = 1
2m ∑ v,w (Avw − kvkw 2m )δ(Cv, Cw),where δ(Cv, Cw) = { 1, if Cv = Cw 0, otherwise.
(1)
In Eq. 1, kvkw2m represents the probability that two nodes v and w are connected in a random graph, and Avw is 1 or 0 depending on whether v and w are adjacent in G or not. The term 12m is for normalizing the modularity value, and δ(Cv, Cw) regulates the algorithm only to consider the edges of a specific community. A larger modularity score indicates better quality for the communities.
The modularity above is defined only on a partition, i.e., when the communities are disjoint. Shen et al. (2009) proposed an extension of Eq. 1 for undirected, unweighted graphs with overlapping
communities, as follows:
Qunweighted,overlap = 1
2m ∑ v,w (Avw − kvkw 2m ) 1 OvOw , (2)
where Ov and Ow is the number of communities containing node v and w, respectively. Thus the modularity becomes larger when the communities have less overlap. Chen et al. (2010) proposed a further extension for weighted, undirected graphs with overlapping communities:
Qweighted,overlap = 1
2m ∑ c∈C ∑ v,w (Avw − kvkw 2m )αcvαcw, (3)
where αcv is the belonging coefficient (Nicosia et al., 2009) for a community c and vertex v is defined as αcv = kcv∑
c∈C kcv and kcv =
∑ p∈c Wvp, where C is the set of community in graph G and
c ∈ C, Wvp is the weight of the edge between node v and p. In Eq. 3, if node v belongs to only one community c, αcv is equal to 1; if node v does not belong to community c, αcv is equal to 0, which ensures the consistency of Eq. 3 with Eq. 1. Compared to Eq. 1, Eq. 3 considers overlapping communities similar to Eq. 2, but unlike Eq. 1 it considers the edge weights too. The benefit of Eq. 3 over other equations is, a pair of nodes within a community with higher edge weight contributes more to the quality measure compared to the pairs with low edge weight.
Entropy. Entropy is a metric that measures the level of uncertainty in a system, and a rich body of research examines different ways of measuring entropy (Borowska, 2015). Shanon entropy is a widely used metric in information theory (Shannon, 2001), which calculates the average information available based on the probability of a variable’s possible outcomes. Let X be a discrete random variable with possible outcomes x1, x2, ..., xk which occurs with probabilities P (x1), P (x2), ..., P (xk), then the Shanon entropy, H of the variable X is defined as follows:
H = − k∑
i=1
P (xi) logP (xi). (4)
A higher value of Shanon’s entropy indicates a more uncertain outcome, which is difficult to predict.

heading: 4 HYPOTHESIS
text: We assume that if a neuron is frequently activated with a high activation value for a particular class, then the neuron is a representative of that class. Consider a graph with nodes as neurons and edges representing simultaneous activation of pairs of neurons. We refer to such a graph as an ‘activation pattern graph’, and describe the details in Section 5.1. Although one neuron can be representative for multiple classes, over the training, we expect the class representatives to have less overlap. Hence we also expect the activation pattern graph to evolve such that the neurons representing a class form a community. Furthermore, over the training, the neurons are expected to have more certainty in their activation pattern for various classes (rather than being random). Therefore, we expect the ‘activation pattern entropy’, a measure related to the predictability of a neuron behavior, as described later in Section 5.2, to decrease. In particular, we examine the following hypotheses.
H1: The neurons that are frequently activated together form a community in activation pattern graph.
H2: The modularity of the activation pattern graph is related to a deep learning model’s performance.
H3: The entropy of the activation pattern is related to a deep learning model’s performance.

heading: 5 METHODOLOGY
text: In this section, we describe the activation pattern graph and activation pattern entropy, which are at the core of our methodology and experimental design.

heading: 5.1 ACTIVATION PATTERN GRAPH
text: Assume that there are C classes in the training dataset, and let Di, where 1 ≤ i ≤ k, be the subset corresponding to the ith class. Let ℓ be a fully connected layer in the neural network with n neurons.
By the notation vi(q, ℓ, d), we denote the activation value of a neuron q ∈ ℓ for a data d that belongs to the ith class. Thus the average activation value of a neuron q in layer ℓ for a class i is as follows:
Vi(q, ℓ) =
∑ d∈Di vi(q, ℓ, d)
|Di| . (5)
We construct the activation pattern graph G of a layer ℓ as follows. We first calculate each neuron’s average activation value for a class. Next, for every class, we select S neurons with the highest average activation value and take their union to create the vertex set of G. Let p, q be a pair of neurons in S and let δp,q be the number of data elements in Di, where both p and q obtain activation values that are larger than their individual average activation values, i.e., δp,q = |{d : d ∈ Di, vi(p, ℓ, d)>Vi(p, ℓ) and vi(q, ℓ, d)>Vi(q, ℓ)}|. Then, for class i, we create an adjacency matrix, Ai,ℓ = (ap,q)n×n, where
ap,q = { δp,q |Di| , if p ∈ S and q ∈ S 0, otherwise.
(6)
The final adjacency matrix, Aℓ of the activation pattern graph is computed as Aℓ = ∑k
i=1 Ai,ℓ. Thus intuitively, an activation pattern graph maintains a set of neurons that are frequently activated, where a weighted edge between a pair of neurons represents the frequency of their simultaneous activation. We created a graph for every iteration and every fully connected layer (except the output layer).

heading: 5.2 ENTROPY OF ACTIVATION PATTERN
text: In a neural network with fully connected layers and ReLU (Rectified Linear Unit) as the activation function, the activation of each neuron is calculated as ReLU(y) = max(0, y), where y is the output of a layer. Let Wℓ and Bℓ be the weight and bias of layer ℓ of a neural network. Then for an input I , the output of the layer ℓ is calculated as yℓ = (Wℓ×I)+Bℓ. At the beginning of training, the weights and biases are randomly initialized, and over the training, these values are updated for improved predictions. The neuron activation thus gets more and more influenced by the classes present in the data. Due to random initial weights, the activation of the neurons also becomes unpredictable. In such a case, the activation patterns’ entropy is high, reflecting the system’s randomness. As the training progresses, the activation pattern is biased by the data, and thus the entropy will decrease.
We now compute the activation pattern entropy of a particular neuron, which is based on the idea of measuring the predictability of its activation value. We create a |D| ×N activation pattern matrix, F , where |D| is the size of the training dataset, and N is the total number of neurons in the architecture, except for the neurons in the last layer (output layer). Each entry (i, j) of F contains the activation value of the jth neuron subject to the ith element of the dataset D. We then compute a normalized matrix Fnorm by dividing each column by the column sum, i.e., Fnorm(i, j) =
F (i,j)∑|D| i=1 F (i,j) .
To examine the predictability of the activation value for jth neuron, we categorized its normalized activation values using R equal size bins. In other words, we create a histogram for the jth column values of Fnorm(i, j). The intuition is that if the neuron’s activation is unpredictable, then the histogram will not have well-defined maxima or contain many local maxima. Otherwise, it will be activated for one or only a few classes and likely to produce a global peak. There is an exception, where a neuron may never be activated and will create a global maxima at the bin that contains the 0 value. Therefore, to create the histogram, we only consider the non-zero activation values. Let Bk and Hk, where 1 ≤ k ≤ R, be the kth bin and its number of elements. Let hi be the normalized value, i.e., hi = Hk∑R
i=1 Hi . We use the vector Fv = [h1, . . . , hR] to compute the activation pattern entropy Ej for the jth neuron. Ej = − ∑R
i=1 −hi · log(hi). The activation pattern entropy, E, of a deep learning model over all the neurons in all the fully connected layers is calculated as E = ∑N j=1 Ej .

heading: 6 DATASET AND MODEL ARCHITECTURE
text: In our study, we used three benchmark datasets: MNIST (LeCun et al., 2010), Fashion MNIST (Xiao et al., 2017), CIFAR-10 (Krizhevsky et al., 2009) and one real-life dataset Plant village (Mohanty, 2018). We also created MNIST Mixed and Fashion MNIST Mixed by randomizing the labels of the respective datasets to examine the reliability of the proposed metrics.
MNIST 28×28 512, 512 10 20 91.80 88.9
Fashion MNIST Mixed
28×28 512, 512 10 20 11.02 10.12
CIFAR-10 32×32 1024, 1024, 1024, 512, 256, 128
10 40 46.24 43.43
Plant Village 256×256
8192, 2048, 1024, 512,
256, 128, 64 9 100 80.65 79.55
The architecture for each dataset is illustrated in Table 1. The MNIST, Fashion MNIST, CIFAR-10 datasets each consists of 10 classes. There are 60,000 grayscale 28 × 28 images for training and 10,000 for testing in MNIST and Fashion MNIST, and 50,000 grayscale 32×28 images for training and 10,000 for testing in CIFAR-10.
The Plant village dataset consists of 28693 segmented color images of the healthy apple, blueberry and cherry, corn, grape, orange, peach and pepper, soybean, strawberry and squash, and tomato classes, and 6890 testing images. We transformed the color images to grayscale and reshaped the images to size 256× 256. For all the models, apart from the input and output layers, we only used fully connected layers and dropout layer. For the fully connected layers, we used ReLU as the activation function, and Softmax activation for the output layer. For each model, we collected the weight values from 20 iterations at a uniform interval, where each iteration represents training the model with the dataset once. We denote the corresponding activation pattern graphs as G(1), G(2), . . . , G(20). Since different models needed a different number of training iterations, this interval length varies, e.g., the CIFAR-10 model collected every two iterations, whereas the Plant Village model every five iterations. See the supplementary materials for detailed model architecture.

heading: 7 RESULT AND DISCUSSION
text: In this section, we describe the experimental results. To create the activation pattern graphs, we took 50 neurons per class (i.e., we choose |S| = 50 in Eq. 6) with the largest average activation value. We repeated the experiments for the top 25 and top 100 neurons per class and achieved similar results.

heading: 7.1 COMMUNITY FORMATION (H1)
text: Let G(k) be a pattern activation graph of a fully connected layer at iteration k and let Ni be the set of neurons which are representative of the ith class, i.e., neurons that are frequently and highly activated for that class. The unique neurons of a pair of classes i and j are the neurons that belong to both Ni and Nj , but in no other set Nk where k ̸∈ {i, j}. In other words, these neurons represents both classes i and j, but no other class. If i = j, then we obtain the unique neurons of class i. We observed that in most cases, the number of unique neurons for a single class increases with training, whereas unique neurons for a pair of distinct classes decrease. This behavior indicates that the overlap between the communities is decreasing and the classes are forming a stronger community among themselves. Table 2 compares the confusion metrics obtained from G(1) and G(20) for the MNIST dataset, where each entry at the diagonal of a matrix represents the unique neurons for a single class. We observed similar behavior for all the datasets.
Training Accuracy and Average Community Size: We used Gephi (Bastian et al., 2009), a widely used graph visualization software, to compute force directed layouts (Jacomy et al., 2014) of the activation pattern graphs of different iterations. The communities were detected based on the Louvain method (Blondel et al., 2008). Although the visualization revealed community structures in these graphs, the change in the communities over the number of iterations was not readily visible from the graph layouts (see the supplementary material). Hence, we quantitatively examined how the number of nodes per community varies over the training. For each dataset, we used Gephi’s modularity function with a resolution value of 0.5 to find the communities in G(1), G(2), . . . , G(20). Table 3
shows the number of nodes in the eight largest communities C1, C2, . . . , C8 in layer 2 for different datasets and for two different iterations 1 and 20, where the distribution of community sizes appears to be less skewed over the training. We then perform the Pearson Correlation Coefficient (PCC) between the training accuracy and average node per community over the 10 iterations. For most of the datasets, we observed the average node per community to have a positive correlation with the training accuracy (Table 3), i.e., the PCC correlation was higher than 0. However, the weak (near-zero) correlation coefficient for MNIST Mixed and Fashion MNIST Mixed represents the randomness of the formation of the communities of the model. This indicates that, for models with good performance, the number of well defined communities increases with training and supports hypothesis H1.

heading: 7.2 MODULARITY AND ACCURACY (H2)
text: So far, we have observed that community detection can reveal meaningful clusters in the pattern activation graph, i.e., clusters are related to representative neuron sets for different classes. We now examine the other side, i.e., can the representative neuron sets for different classes be seen as well as the defined communities? To assess this, we compute different modularity metrics (Eq. 1– 3) for the activation pattern graphs G(1), . . . , G(20). We also used different community detection algorithms available in the python library NetworkX (Hagberg et al., 2008) and iGraph (Csardi & Nepusz, 2006). Note that these algorithms take an initial partition of the neurons and then change those partitions to optimize some quality measure. Therefore, they are not suitable in this context, yet, the Kernighan Lin Bisection (KLB) (Kernighan & Lin, 1970) could generate consistent results.
We computed the Spearman correlation coefficient (SCC) and Pearson correlation coefficient (PCC) to examine the potential relation between modularity and training accuracy (Table 4). A positive correlation value indicates that the communities’ quality is positively correlated with training accuracy. We observed the same relation with the test accuracy. A large number of strong positive correlation coefficients in Table 4 supports hypothesis H2. A stronger correlation is observed in later layers for all the dataset. For most datasets, the unweighted overlap (Eq. 2), and weighted overlap metric (Eq. 3) captures the relation better, which is expected since the no-overlap metric neither considers weight nor the community overlap.

heading: 7.3 ENTROPY AND ACCURACY (H3)
text: A higher entropy value represents more randomness in the activation behavior of a neuron. At the beginning of the training, due to random weights, the activation of the neuron is random. So, the entropy of the activation pattern over all the fully connected layers should be higher. As the training progresses, the entropy should decrease, representing a biased neuron activation behavior.
Table 5 shows a negative correlation between the entropy of activation pattern and training accuracy over model training for most of the datasets. We observed the same relation with the test accuracy. Figure 1 shows the change of normalized entropy with training and testing accuracy for different iterations for all the datasets, where a clear relation between the entropy and model performance can be observed. However, MNIST Mixed and Fashion MNIST Mixed has positive correlation values, indicating the absence of activation pattern with training due to it’s random labels, which is consistent with hypothesis H3. Note that the evidence for hypothesis H3 is supportive of H1 and H2, where we examined the community behavior of the neurons and observed better community structure except for MNIST mixed and Fashion MNIST mixed.
We also examine entropy change for individual classes (Table 6), where to compute the entropy of a class, we take the same approach as in Section 5.2, but use only the training data corresponding to that class. In this setting, we obtain meaningful results (i.e., a negative correlation between the entropy and training accuracy) only for all well-trained models (MNIST, Fashion MNIST, and Plant
Village). However, there are few classes with positive and small negative correlation between entropy and accuracy. This is due to frequent change in the training accuracy of that class. For CIFAR-10, MNIST Mixed, Fashion MNIST Mixed, we observed a positive correlation values in most of the classes, and sometimes even found the correlation coefficient to be strongly positive. This is due to the inability of the model to separate the classes from one another.

heading: 8 LIMITATIONS AND FUTURE WORK
text: We have proposed novel methods to explain the behavior of neurons in deep learning models as the model training progresses. We used graph theoretic and entropy-based methods to model the activation pattern of the neurons. We quantitatively showed that neurons that are highly activated for a class form a community in our graph model. For the entropy-based approach, we analyzed the activation pattern using the Shanon entropy and found the entropy to show a negative correlation with the training accuracy.
There are many scope for future research. We used deep learning models with only fully connected layers; using different model architecture may help understand the entropy and modularity behavior with more granularity. Although we examine a diverse set of datasets, adding more datasets could strengthen the results. Our experimental results show that the modularity of the activation pattern graph and entropy of the activation pattern is related to the model’s performance. Although we used widely used quality measures for the modularity and entropy, there is still scope for designing better quality metrics specifically for neural network context. Another exciting direction of study can be to use the modularity and entropy as a regularization in the loss function (Leavitt & Morcos, 2020) to better optimize the performance of the deep learning models. We believe that our results will inspire further research to explain the deep learning models using graph and information theoretic methods.

Review summaries:
This paper proposes new metrics based on the activation patterns of a model. It propose an activation pattern entropy metric and a graph theoretic metric of neuron communities. The authors then evaluated models trained on MNIST, Fashion MNIST, Fashion MNIST mixed, CIFAR-10, and plant village using their new metrics. They found that entropy is negatively correlated with training accuracy, modularity is correlated with training accuracy, and that the number of well defined neural communities increase with accuracy.

This paper examined novel neuron activation pattern analysis on neural network classification models via graph theoretic and entropy-based methods. The authors showed reliability of their work by approving hypotheses on examining the qualitative correlation between model performance and activation patterns. The main technical contribution of this paper comes from explaining the neural classifiers by combining the graph-theoretic and information-theoretic approaches.

The authors propose two methods to analyze the behavior of neurons in neural nets. The main idea is to study the neuron activation patterns of classification models and explore if the performance can be explained through neurons' activation behavior. The authors propose two approaches: one that models neurons' activation behavior as a graph and examines whether the neurons form meaningful communities, and the other that examines the predictability of neurons' behavior using entropy.

An important problem of understanding the performance of deep learning models is studied by this submission. Two graph-theoretic and information-theoretic metrics are checked in some datasets with MLP networks. Preliminary results with expected correlation are observed.