On the Evolution of Neuron Communities in a Deep Learning Architecture

1 INTRODUCTION . Deep learning allows the researchers to engineer better features and representation of data through representation learning ( LeCun et al. , 2015 ) . Despite the widespread usage of deep learning methods , it is still considered a black box , and there is a lack of understanding of the working procedure of the models ( Rai , 2020 ; Oh et al. , 2019 ; Tzeng & Ma , 2005 ; Rudin , 2019 ) . Researchers often rely on intuition and domain knowledge when designing deep learning architectures ( Shahriari et al. , 2015 ) . They use the models ’ accuracy and loss to evaluate the performance and tune the hyperparameters to optimize the models ( Géron , 2019 ; Gigante et al. , 2019 ) . However , to gain a deeper insight into the model , it is crucial to look beyond its accuracy and loss , i.e. , we need to design additional evaluation metrics that can provide a level of confidence in the model predictions . Exploring network architecture helps explain why some models perform better than others . For example , by comparing the weight optimization against model architecture , Gaier & Ha ( 2019 ) showed that a model ’ s performance depends mostly on its architecture . Zhang et al . ( 2018 ) created an explanatory graph that can disentangle different part patterns from feature maps of the convolutional neural network ( CNN ) . Visual analytics tools have also been used to understand the architectures better ( Hohman et al. , 2019 ; Gigante et al. , 2019 ) . However , such approaches sometimes require domain knowledge , and are also subject to human interpretation as they do not provide quantitative measures for the evaluation . Motivation . In this paper , our main focus is on classification models . A deep learning model that performs a classification task is optimized to minimize the difference between the model ’ s prediction and the actual value . The prediction is recorded as accurate if it is above a threshold , but this process does not fully leverage the information hidden inside the architecture . The neuron activation pattern that appears during the training contains rich information that , if understood well , can potentially be combined with model prediction to provide a level of confidence in the prediction value . It may also allow us to design more efficient models and additional evaluation metrics to help reduce the number of false negatives and false positives . If it is possible to relate the model ’ s performance with the neuron activation pattern , we may understand why some models perform better than others by analyzing the pattern . This motivated us to explore different ways to examine neurons ’ activation pattern and establish their relation to the training and test accuracy . Contribution . We study the neuron activation pattern using two approaches : one based on a graphtheoretic model , and the other is based on an information-theoretic model . Our proposed metrics show a correlation with the accuracy on both benchmark and real-life datasets in both cases . For the graph-theoretic approach , we created a novel ‘ activation pattern graph ’ model and showed that the set of neurons that are frequently and highly activated for a class often forms a community ( i.e. , an induced subgraph that contains more edges within than outside of the subgraph ) in the activation pattern graph . We observed that modularity , a widely used metric to measure community quality , is closely related to the model ’ s training and test accuracy . For the information-theoretic model , we propose a novel method that allows us to measure the predictability of the neurons ’ activation behavior leveraging entropy ( Gray , 2011 ) . Our experimental results show that the entropy of the neurons is relevant to the model ’ s training and test accuracy . 2 RELATED WORK . Neuron Activation Pattern . Researchers have investigated both neuron activation patterns and loss function to explain deep learning models . Olah et al . ( 2017 ) studied what the neurons respond to and proposed that neurons work in a group . Li et al . ( 2017 ) showed a relationship between the performance of the deep learning model and the convexity of the loss function . Mahendran & Vedaldi ( 2015 ) studied the feature maps of the CNN models to understand the activation pattern by inverting the models and showed that there is indeed a relation between the performance of the model and the activation pattern . Kim et al . ( 2018 ) vectorized the activation values of hidden layers and created striped patterns to find relevance between pattern and decision . However , they experimented with a small dataset without directions for generalizability . Bau et al . ( 2017 ) studied the relation between the activation pattern and semantic concepts . They optimized the hyperparameters of the deep learning models and quantitively analyzed the effect of changing different parameters . The relevance between activation patterns and semantic concepts was further studied by Fong & Vedaldi ( 2018 ) . They created vector response of semantic concepts based on activation pattern and established that the neurons work in a group , and the same neuron can represent multiple concepts . We found a similar characteristic of the neuron activation pattern in our study , where there was an overlap of neurons representing multiple classes . However , with training , a class was represented by more unique neurons . Another way of interpreting the deep learning models ’ learning is by selecting image patches that maximize the neuron activation ( Zeiler & Fergus , 2014 ) . Zhou et al . ( 2014 ) described scenes using the units from the feature maps and also proposed that individual units behave as object detectors . Simon et al . ( 2014 ) analyzed the deep learning model ’ s feature maps and found a spatial relation between the activation centers and the semantic parts or bounding boxes of the ground truth images . Zhang et al . ( 2018 ) used hierarchical explanatory graphs across layers to propose a way of maximizing a deep learning model ’ s performance . Bau et al . ( 2020 ) discovered that individual units of CNN can learn concepts from the images and some of them strongly influence the decision of the model . Although there have been several attempts to explain activation patterns , in this paper we take a very different approach to the explanation of the neural network models by combining the graph-theoretic and information-theoretic approaches , which allow us to relate various quantitative measures to the model ’ s performance . Visualization and Interpretability . Visualization is considered to be a promising approach for explaining complex systems , and as a result , there have been several attempts to explain the deep learning models using visualization tools ( Hohman et al. , 2018 ; Chung et al. , 2016 ) . A dataflow diagram is a simple way to visualize a deep learning model ’ s architecture ( Wongsuphasawat et al. , 2017 ) . However , dataflow diagrams do not tell much about the model ’ s learning process . Smilkov et al . ( 2017 ) proposed “ Tensorflow Playground , ” where users can train a model by changing the hyperparameters and observe the training process with various plots and charts . Harley ( 2015 ) , proposed an interactive visual inspection tool , where a model was trained on the MNIST dataset ( LeCun et al. , 2010 ) , and for input data , the tool showed which feature maps were activated and also which part of the input data was activating the maps . ACTIVIS ( Kahng et al. , 2017 ) is an interactive visualization tool that , along with the dataflow diagram , provides a projection of the neuron activation for different instances . This only allows users to explore the activation pattern for different samples , classes , and subsets rather than explaining the model ’ s working procedure . CNNComparator ( Zeng et al. , 2017 ) is a tool to compare the performance metrics and the distribution of hyperparameters of two deep learning models . A rich body of research examines ways of visualizing feature maps of CNN models ( Dobrescu et al. , 2019 ; Simonyan et al. , 2013 ; Zeiler & Fergus , 2014 ) . A common limitation of studies that rely on visualization is that the visual assessment is subjective and often hard to evaluate quantitatively . Entropy and Deep Learning . Entropy is commonly used to quantify errors , e.g. , the cross entropy is a widely used loss function ( Gordon-Rodriguez et al. , 2020 ) . Many researchers proposed modifications to this loss function . Martinez & Stiefelhagen ( 2018 ) proposed Tamed Cross Entropy , which has the same convergence property as the cross entropy but is more robust against uniformly distributed label noise , Zhou et al . ( 2019 ) proposed Maximum Probability based Cross Entropy ( MPCE ) loss function , which uses a MPCE based gradient update algorithm and has less back-propagation error than cross entropy , and Gordon-Rodriguez et al . ( 2020 ) proposed to use log-likelihood of the continuous categorical distribution in the place of the cross entropy loss used in label smoothing and actor-mimic reinforcement learning . Barbiero et al . ( 2021 ) proposed an entropy-based linear layer for concept-based deep learning models , which utilizes entropy to choose limited subset of input concepts , allowing it to provide concise explanations of its predictions . Huo et al . ( 2020 ) implemented a maximum entropy regularizer that encourages uniform weight distribution . Li et al . ( 2020 ) approximated the gradient of the cross entropy loss function , which are robust against noise and can avoid the vanishing gradient problem . Although entropy measures has been used for model optimization , we leveraged it to analyze the activation pattern of the models and propose performance metric that correlates to the model accuracy . 3 TECHNICAL BACKGROUND . Graph and Community Structure . A graph consists of a set of elements ( nodes ) and a set of pairs of elements ( edges ) , where the nodes represent objects and edges represent pairwise relationships . A community of a graph is defined as a subgraph that contains more edges within than the edges connecting them to the rest of the graph . Community detection algorithms often define a quality measure for the communities and then attempts to find a partition of the nodes that maximizes the quality measure ( Fortunato & Castellano , 2007 ; Newman & Girvan , 2004 ) . Later , we will define graphs with nodes as neurons and edges as their simultaneous activation . We will examine the communities of these graphs based on several quality measures as explained in the next section . Modularity . Modularity is a widely used metric to assess the quality of a given set of non-overlapping communities ( Newman , 2004 ; Duch & Arenas , 2005 ; Clauset et al. , 2004 ) . The idea of modularity is based on comparing the given graph G with a random graph . Given two communities Ci and Cj in G , we use the notation e ( Ci , Cj ) to denote the number of edges between these two communities . Let aCi be the fraction of all the edges connecting the community Ci to all other communities , i.e. , aCi = ∑ j eCiCj . For a random graph , the fraction of the resulting edges that connect nodes within the community Ci is aCi 2 . Hence the modularity ( Newman & Girvan , 2004 ) can be formalized as Qno−overlap = ∑ i ( eCiCi − aCi2 . This is commonly expressed leveraging the adjacency matrix of the graph ( Blondel et al. , 2008 ) . Let Avw be the adjacency matrix of G , and let m be the number of edges in G. For a vertex v ( resp. , w ) in G , we denote its degree and community by kv and Cv ( resp. , kw and Cw ) . Then Qno−overlap = 1 2m ∑ v , w ( Avw − kvkw 2m ) δ ( Cv , Cw ) , where δ ( Cv , Cw ) = { 1 , if Cv = Cw 0 , otherwise . ( 1 ) In Eq . 1 , kvkw2m represents the probability that two nodes v and w are connected in a random graph , and Avw is 1 or 0 depending on whether v and w are adjacent in G or not . The term 12m is for normalizing the modularity value , and δ ( Cv , Cw ) regulates the algorithm only to consider the edges of a specific community . A larger modularity score indicates better quality for the communities . The modularity above is defined only on a partition , i.e. , when the communities are disjoint . Shen et al . ( 2009 ) proposed an extension of Eq . 1 for undirected , unweighted graphs with overlapping communities , as follows : Qunweighted , overlap = 1 2m ∑ v , w ( Avw − kvkw 2m ) 1 OvOw , ( 2 ) where Ov and Ow is the number of communities containing node v and w , respectively . Thus the modularity becomes larger when the communities have less overlap . Chen et al . ( 2010 ) proposed a further extension for weighted , undirected graphs with overlapping communities : Qweighted , overlap = 1 2m ∑ c∈C ∑ v , w ( Avw − kvkw 2m ) αcvαcw , ( 3 ) where αcv is the belonging coefficient ( Nicosia et al. , 2009 ) for a community c and vertex v is defined as αcv = kcv∑ c∈C kcv and kcv = ∑ p∈c Wvp , where C is the set of community in graph G and c ∈ C , Wvp is the weight of the edge between node v and p. In Eq . 3 , if node v belongs to only one community c , αcv is equal to 1 ; if node v does not belong to community c , αcv is equal to 0 , which ensures the consistency of Eq . 3 with Eq . 1 . Compared to Eq . 1 , Eq . 3 considers overlapping communities similar to Eq . 2 , but unlike Eq . 1 it considers the edge weights too . The benefit of Eq . 3 over other equations is , a pair of nodes within a community with higher edge weight contributes more to the quality measure compared to the pairs with low edge weight . Entropy . Entropy is a metric that measures the level of uncertainty in a system , and a rich body of research examines different ways of measuring entropy ( Borowska , 2015 ) . Shanon entropy is a widely used metric in information theory ( Shannon , 2001 ) , which calculates the average information available based on the probability of a variable ’ s possible outcomes . Let X be a discrete random variable with possible outcomes x1 , x2 , ... , xk which occurs with probabilities P ( x1 ) , P ( x2 ) , ... , P ( xk ) , then the Shanon entropy , H of the variable X is defined as follows : H = − k∑ i=1 P ( xi ) logP ( xi ) . ( 4 ) A higher value of Shanon ’ s entropy indicates a more uncertain outcome , which is difficult to predict .

Summary: The authors propose two methods to analyze the behavior of neurons in neural nets. The main idea is to study the neuron activation patterns of classification models and explore if the performance can be explained through neurons' activation behavior. The authors propose two approaches: one that models neurons' activation behavior as a graph and examines whether the neurons form meaningful communities, and the other that examines the predictability of neurons' behavior using entropy.