On the Evolution of Neuron Communities in a Deep Learning Architecture

1 INTRODUCTION . Deep learning allows the researchers to engineer better features and representation of data through representation learning ( LeCun et al. , 2015 ) . Despite the widespread usage of deep learning methods , it is still considered a black box , and there is a lack of understanding of the working procedure of the models ( Rai , 2020 ; Oh et al. , 2019 ; Tzeng & Ma , 2005 ; Rudin , 2019 ) . Researchers often rely on intuition and domain knowledge when designing deep learning architectures ( Shahriari et al. , 2015 ) . They use the models ’ accuracy and loss to evaluate the performance and tune the hyperparameters to optimize the models ( Géron , 2019 ; Gigante et al. , 2019 ) . However , to gain a deeper insight into the model , it is crucial to look beyond its accuracy and loss , i.e. , we need to design additional evaluation metrics that can provide a level of confidence in the model predictions . Exploring network architecture helps explain why some models perform better than others . For example , by comparing the weight optimization against model architecture , Gaier & Ha ( 2019 ) showed that a model ’ s performance depends mostly on its architecture . Zhang et al . ( 2018 ) created an explanatory graph that can disentangle different part patterns from feature maps of the convolutional neural network ( CNN ) . Visual analytics tools have also been used to understand the architectures better ( Hohman et al. , 2019 ; Gigante et al. , 2019 ) . However , such approaches sometimes require domain knowledge , and are also subject to human interpretation as they do not provide quantitative measures for the evaluation . Motivation . In this paper , our main focus is on classification models . A deep learning model that performs a classification task is optimized to minimize the difference between the model ’ s prediction and the actual value . The prediction is recorded as accurate if it is above a threshold , but this process does not fully leverage the information hidden inside the architecture . The neuron activation pattern that appears during the training contains rich information that , if understood well , can potentially be combined with model prediction to provide a level of confidence in the prediction value . It may also allow us to design more efficient models and additional evaluation metrics to help reduce the number of false negatives and false positives . If it is possible to relate the model ’ s performance with the neuron activation pattern , we may understand why some models perform better than others by analyzing the pattern . This motivated us to explore different ways to examine neurons ’ activation pattern and establish their relation to the training and test accuracy . Contribution . We study the neuron activation pattern using two approaches : one based on a graphtheoretic model , and the other is based on an information-theoretic model . Our proposed metrics show a correlation with the accuracy on both benchmark and real-life datasets in both cases . For the graph-theoretic approach , we created a novel ‘ activation pattern graph ’ model and showed that the set of neurons that are frequently and highly activated for a class often forms a community ( i.e. , an induced subgraph that contains more edges within than outside of the subgraph ) in the activation pattern graph . We observed that modularity , a widely used metric to measure community quality , is closely related to the model ’ s training and test accuracy . For the information-theoretic model , we propose a novel method that allows us to measure the predictability of the neurons ’ activation behavior leveraging entropy ( Gray , 2011 ) . Our experimental results show that the entropy of the neurons is relevant to the model ’ s training and test accuracy . 2 RELATED WORK . Neuron Activation Pattern . Researchers have investigated both neuron activation patterns and loss function to explain deep learning models . Olah et al . ( 2017 ) studied what the neurons respond to and proposed that neurons work in a group . Li et al . ( 2017 ) showed a relationship between the performance of the deep learning model and the convexity of the loss function . Mahendran & Vedaldi ( 2015 ) studied the feature maps of the CNN models to understand the activation pattern by inverting the models and showed that there is indeed a relation between the performance of the model and the activation pattern . Kim et al . ( 2018 ) vectorized the activation values of hidden layers and created striped patterns to find relevance between pattern and decision . However , they experimented with a small dataset without directions for generalizability . Bau et al . ( 2017 ) studied the relation between the activation pattern and semantic concepts . They optimized the hyperparameters of the deep learning models and quantitively analyzed the effect of changing different parameters . The relevance between activation patterns and semantic concepts was further studied by Fong & Vedaldi ( 2018 ) . They created vector response of semantic concepts based on activation pattern and established that the neurons work in a group , and the same neuron can represent multiple concepts . We found a similar characteristic of the neuron activation pattern in our study , where there was an overlap of neurons representing multiple classes . However , with training , a class was represented by more unique neurons . Another way of interpreting the deep learning models ’ learning is by selecting image patches that maximize the neuron activation ( Zeiler & Fergus , 2014 ) . Zhou et al . ( 2014 ) described scenes using the units from the feature maps and also proposed that individual units behave as object detectors . Simon et al . ( 2014 ) analyzed the deep learning model ’ s feature maps and found a spatial relation between the activation centers and the semantic parts or bounding boxes of the ground truth images . Zhang et al . ( 2018 ) used hierarchical explanatory graphs across layers to propose a way of maximizing a deep learning model ’ s performance . Bau et al . ( 2020 ) discovered that individual units of CNN can learn concepts from the images and some of them strongly influence the decision of the model . Although there have been several attempts to explain activation patterns , in this paper we take a very different approach to the explanation of the neural network models by combining the graph-theoretic and information-theoretic approaches , which allow us to relate various quantitative measures to the model ’ s performance . Visualization and Interpretability . Visualization is considered to be a promising approach for explaining complex systems , and as a result , there have been several attempts to explain the deep learning models using visualization tools ( Hohman et al. , 2018 ; Chung et al. , 2016 ) . A dataflow diagram is a simple way to visualize a deep learning model ’ s architecture ( Wongsuphasawat et al. , 2017 ) . However , dataflow diagrams do not tell much about the model ’ s learning process . Smilkov et al . ( 2017 ) proposed “ Tensorflow Playground , ” where users can train a model by changing the hyperparameters and observe the training process with various plots and charts . Harley ( 2015 ) , proposed an interactive visual inspection tool , where a model was trained on the MNIST dataset ( LeCun et al. , 2010 ) , and for input data , the tool showed which feature maps were activated and also which part of the input data was activating the maps . ACTIVIS ( Kahng et al. , 2017 ) is an interactive visualization tool that , along with the dataflow diagram , provides a projection of the neuron activation for different instances . This only allows users to explore the activation pattern for different samples , classes , and subsets rather than explaining the model ’ s working procedure . CNNComparator ( Zeng et al. , 2017 ) is a tool to compare the performance metrics and the distribution of hyperparameters of two deep learning models . A rich body of research examines ways of visualizing feature maps of CNN models ( Dobrescu et al. , 2019 ; Simonyan et al. , 2013 ; Zeiler & Fergus , 2014 ) . A common limitation of studies that rely on visualization is that the visual assessment is subjective and often hard to evaluate quantitatively . Entropy and Deep Learning . Entropy is commonly used to quantify errors , e.g. , the cross entropy is a widely used loss function ( Gordon-Rodriguez et al. , 2020 ) . Many researchers proposed modifications to this loss function . Martinez & Stiefelhagen ( 2018 ) proposed Tamed Cross Entropy , which has the same convergence property as the cross entropy but is more robust against uniformly distributed label noise , Zhou et al . ( 2019 ) proposed Maximum Probability based Cross Entropy ( MPCE ) loss function , which uses a MPCE based gradient update algorithm and has less back-propagation error than cross entropy , and Gordon-Rodriguez et al . ( 2020 ) proposed to use log-likelihood of the continuous categorical distribution in the place of the cross entropy loss used in label smoothing and actor-mimic reinforcement learning . Barbiero et al . ( 2021 ) proposed an entropy-based linear layer for concept-based deep learning models , which utilizes entropy to choose limited subset of input concepts , allowing it to provide concise explanations of its predictions . Huo et al . ( 2020 ) implemented a maximum entropy regularizer that encourages uniform weight distribution . Li et al . ( 2020 ) approximated the gradient of the cross entropy loss function , which are robust against noise and can avoid the vanishing gradient problem . Although entropy measures has been used for model optimization , we leveraged it to analyze the activation pattern of the models and propose performance metric that correlates to the model accuracy . 3 TECHNICAL BACKGROUND . Graph and Community Structure . A graph consists of a set of elements ( nodes ) and a set of pairs of elements ( edges ) , where the nodes represent objects and edges represent pairwise relationships . A community of a graph is defined as a subgraph that contains more edges within than the edges connecting them to the rest of the graph . Community detection algorithms often define a quality measure for the communities and then attempts to find a partition of the nodes that maximizes the quality measure ( Fortunato & Castellano , 2007 ; Newman & Girvan , 2004 ) . Later , we will define graphs with nodes as neurons and edges as their simultaneous activation . We will examine the communities of these graphs based on several quality measures as explained in the next section . Modularity . Modularity is a widely used metric to assess the quality of a given set of non-overlapping communities ( Newman , 2004 ; Duch & Arenas , 2005 ; Clauset et al. , 2004 ) . The idea of modularity is based on comparing the given graph G with a random graph . Given two communities Ci and Cj in G , we use the notation e ( Ci , Cj ) to denote the number of edges between these two communities . Let aCi be the fraction of all the edges connecting the community Ci to all other communities , i.e. , aCi = ∑ j eCiCj . For a random graph , the fraction of the resulting edges that connect nodes within the community Ci is aCi 2 . Hence the modularity ( Newman & Girvan , 2004 ) can be formalized as Qno−overlap = ∑ i ( eCiCi − aCi2 . This is commonly expressed leveraging the adjacency matrix of the graph ( Blondel et al. , 2008 ) . Let Avw be the adjacency matrix of G , and let m be the number of edges in G. For a vertex v ( resp. , w ) in G , we denote its degree and community by kv and Cv ( resp. , kw and Cw ) . Then Qno−overlap = 1 2m ∑ v , w ( Avw − kvkw 2m ) δ ( Cv , Cw ) , where δ ( Cv , Cw ) = { 1 , if Cv = Cw 0 , otherwise . ( 1 ) In Eq . 1 , kvkw2m represents the probability that two nodes v and w are connected in a random graph , and Avw is 1 or 0 depending on whether v and w are adjacent in G or not . The term 12m is for normalizing the modularity value , and δ ( Cv , Cw ) regulates the algorithm only to consider the edges of a specific community . A larger modularity score indicates better quality for the communities . The modularity above is defined only on a partition , i.e. , when the communities are disjoint . Shen et al . ( 2009 ) proposed an extension of Eq . 1 for undirected , unweighted graphs with overlapping communities , as follows : Qunweighted , overlap = 1 2m ∑ v , w ( Avw − kvkw 2m ) 1 OvOw , ( 2 ) where Ov and Ow is the number of communities containing node v and w , respectively . Thus the modularity becomes larger when the communities have less overlap . Chen et al . ( 2010 ) proposed a further extension for weighted , undirected graphs with overlapping communities : Qweighted , overlap = 1 2m ∑ c∈C ∑ v , w ( Avw − kvkw 2m ) αcvαcw , ( 3 ) where αcv is the belonging coefficient ( Nicosia et al. , 2009 ) for a community c and vertex v is defined as αcv = kcv∑ c∈C kcv and kcv = ∑ p∈c Wvp , where C is the set of community in graph G and c ∈ C , Wvp is the weight of the edge between node v and p. In Eq . 3 , if node v belongs to only one community c , αcv is equal to 1 ; if node v does not belong to community c , αcv is equal to 0 , which ensures the consistency of Eq . 3 with Eq . 1 . Compared to Eq . 1 , Eq . 3 considers overlapping communities similar to Eq . 2 , but unlike Eq . 1 it considers the edge weights too . The benefit of Eq . 3 over other equations is , a pair of nodes within a community with higher edge weight contributes more to the quality measure compared to the pairs with low edge weight . Entropy . Entropy is a metric that measures the level of uncertainty in a system , and a rich body of research examines different ways of measuring entropy ( Borowska , 2015 ) . Shanon entropy is a widely used metric in information theory ( Shannon , 2001 ) , which calculates the average information available based on the probability of a variable ’ s possible outcomes . Let X be a discrete random variable with possible outcomes x1 , x2 , ... , xk which occurs with probabilities P ( x1 ) , P ( x2 ) , ... , P ( xk ) , then the Shanon entropy , H of the variable X is defined as follows : H = − k∑ i=1 P ( xi ) logP ( xi ) . ( 4 ) A higher value of Shanon ’ s entropy indicates a more uncertain outcome , which is difficult to predict .

Summary: An important problem of understanding the performance of deep learning models is studied by this submission. Two graph-theoretic and information-theoretic metrics are checked in some datasets with MLP networks. Preliminary results with expected correlation are observed.