Importance and Coherence: Methods for Evaluating Modularity in Neural Networks

1 INTRODUCTION . Deep neural networks have achieved state-of-the-art performance in a variety of applications , but this success contrasts with the challenge of making them more intelligible . As these systems become more advanced and widely-used , there are a number of reasons we may need to understand them more effectively . One reason is to shed light on better ways to build and train them . A second reason is the importance of transparency , especially in settings which involve matters of safety , trust , or justice ( Lipton , 2018 ) . More precisely , we want methods for analyzing a trained network that can be used to construct semantic and faithful descriptions of its inner mechanisms . We refer to this as mechanistic transparency . Toward this goal , we consider modularity as an organizing principle to achieve mechanistic transparency . In the natural sciences , we often try to understand things by taking them apart . Aside from subdivision into the standard architectural building blocks ( e.g. , neurons , channels , layers ) , are there other ways a trained neural network be meaningfully “ taken apart ” ? We aim to analyze a network via a partitioning of its neurons into disjoint sets with the hope of finding that these sets are “ modules ” with distinct functions . Since there are many choices for how to partition a network , we would like metrics for anticipating how meaningful a given partition might be . Inspired by the field of program analysis ( Fairley , 1978 ) , we apply the concepts of “ dynamic ” and “ static ” analysis to neural networks . Dynamic analysis includes performing forward passes and/or computing gradients , while static analysis only involves analyzing architecture and parameters . In a concurrent submission ( Anonymous et al. , 2021 ) , we use spectral clustering to study the extent to which networks form clusters of neurons that are highly connected internally but not externally and find that in many cases , networks are structurally clusterable . This approach is static because the partitioning is produced according to the network ’ s weights only , using neither activations nor gradients . Here , we build off of this concurrent submission by working to bridge graph-based clusterability and functional modularity . To see how well neurons within each cluster share meaningful similarities , we introduce two proxies : importance and coherence . Importance refers to how crucial clusters are to the network ’ s perfor- mance overall and lends insight into how well a partition identifies clusters that are individually key to the network ’ s function . Coherence refers to how consistently the neurons within a cluster correspond in their activations to particular features in data . We analyze coherence both with respect to input features and output labels . To measure these proxies , we utilize dynamic interpretability methods that have been conventionally used for single-neuron analysis to the study of these partitions . We conduct a set of experiments and hypothesis tests in networks scaling from the MNIST to the ImageNet level . In doing so , we show that spectral clustering is capable of identifying functionally important and coherent clusters of neurons . This new finding the and methods we present for combining spectral clustering with dynamic methods supports the use of modular decompositions of neurons toward mechanistic transparency . Our key contributions are threefold : 1 . Introducing two proxies , importance and coherence , to assess whether a given partitioning of a network exhibits modularity . 2 . Quantifying these two proxies with interpretability methods equipped with statistical hypothesis testing procedures . 3 . Applying our methods on the partitions produced by the spectral clustering technique of Anonymous et al . ( 2021 ) on a range of networks , and finding evidence of modularity among these clusters . 2 GENERATING PARTITIONINGS WITH SPECTRAL CLUSTERING . In our concurrent submission , we introduce and study in-depth a procedure to partition a neural network into disjoint clusters of neurons ( Anonymous et al. , 2021 ) based only on its weights . We found that trained networks are more clusterable than randomly initialized ones , and they are also often more clusterable than similar networks with identical weight distributions . The experimental procedure consists of three steps : ( 1 ) “ Graphification ” - transforming the network into an undirected edge-weighted graph ; ( 2 ) Spectral clustering - obtaining a partitioning via spectral clustering of the graph . Graphification : To perform spectral clustering , a network must be represented as an undirected graph with non-negative edges . For MLPs ( multilayer perceptrons ) , each graph vertex corresponds to a neuron in the network including input and output neurons . If two neurons have a weight connecting them in the network , their corresponding vertices are connected by an edge giving its absolute value . For CNNs ( convolutional neural networks ) , a vertex corresponds to a single feature map ( which we also refer to as a “ neuron ” ) in a convolutional layer . Here , we do not use input , output , or fully-connected layers . If two feature maps are in adjacent convolutional layers , their corresponding vertices are connected with an edge giving the L1 norm for the corresponding 2 dimensional kernel slice . If convolutional layers are separated by a batch normalization layer ( Ioffe & Szegedy , 2015 ) , we multiply weights by γ/ ( σ+ ε ) where γ is the scaling factor , σ is the moving standard deviation , and ε is a small constant . Spectral Clustering : We run normalized spectral clustering on the resulting graph ( Shi & Malik , 2000 ) to obtain a partition of the neurons into clusters . For all experiments , we set the number of clusters to 12 unless explicitly mentioned otherwise . We choose 12 because ( 1 ) it is computationally tractable , ( 2 ) it is larger than the number of classes in MNIST and CIFAR-10 , and ( 3 ) it is small compared to the number of neurons in the layers of all of our networks . However , in Appendix A.6 , we show results for k = 8 and k = 18 for a subset of experiments and find no major differences . We use the scikit-learn implementation ( Pedregosa et al. , 2011 ) with the ARPACK eigenvalue solver ( Borzı̀ & Borzı̀ , 2006 ) . Refer to appendix A.1 for a complete description of the algorithm . 3 EVALUATION OF MODULARITY USING IMPORTANCE AND COHERENCE . Clusters of neurons produced by spectral clustering span more than one layer . However , layers at different depths of a network tend to develop different representations . To control for these differences , we study the neurons in clusters separately per layer . We call these sets of neurons within the same cluster and layer “ sub-clusters. ” In our experiments , we compare these sub-clusters to other sets of random units of the same size and same layer . When discussing these experiments , we refer to the sub-clusters from the clustering algorithm as “ true sub-clusters ” and the sets composed of random neurons as “ random sub-clusters. ” Random sub-clusters form the natural control condition to test whether the specific partitioning of neurons exhibits importance or coherence compare to alternative partitions , while taking account location and size . As outlined in the Introduction , we study importance : how crucial each sub-cluster is to the network ; input coherence : how well neurons in a sub-cluster associate with similar input features ; and output coherence , how well they associate with particular output labels , as proxies for modularity . In this section , we present two types of experiments . First , we use visualization techniques on sub-clusters to measure input and output coherence , and second , we use “ lesion tests ” based on dropping out neurons in a sub-cluster to measure output coherence and importance . These techniques are scalable , and we experiment with a wide range of networks . For small-scale experiments , we train and analyze MLPs with four hidden layers of 256 neurons each and small convolutional networks with 3 layers of 64 neurons each followed by a dense layer of 128 neurons trained on the MNIST ( LeCun et al. , 1998 ) and Fashion-MNIST ( Xiao et al. , 2017 ) datasets . At a mid scale , we train and analyze VGG-style CNNs containing 13 convolutional layers using the architectures from Simonyan & Zisserman ( 2014 ) trained on CIFAR-10 ( Krizhevsky et al. , 2009 ) using the procedure from Liu & Deng ( 2015 ) . Finally , for the ImageNet ( Krizhevsky et al. , 2009 ) scale , we analyze pretrained ResNet18 , ResNet50 , ( He et al. , 2016 ) VGG-16 , and VGG-19 ( Simonyan & Zisserman , 2014 ) models . In our concurrent submission ( Anonymous et al. , 2021 ) we show that in some cases , weight pruning and dropout can each be used to promote graph-based clusterability . We use pruning in small MLPs but no other networks . We use dropout for MLPs in correlation-based visualization experiments in subsection 3.1.1 but no other MLPs . Also , for the mid-sized VGG-CNNs , we experiment both with versions that are unregularized and which are regularized using dropout and L2 regularization as done in Liu & Deng ( 2015 ) . Complete training details including testing accuracies are in the appendix A.2 . 3.1 FEATURE VISUALIZATION . 3.1.1 CORRELATION-BASED VISUALIZATION . First , we introduce here a simple method to provide visual examples and build intuition . In later subsections , we present a quantitative approach with statistical hypothesis testing . A simple way to visualize a sub-cluster is to identify what input features each of its neurons respond to and then use these to create an aggregated visualization . We do this for small MLPs in which we construct visualizations of neurons using their correlations with the input pixels across the test dataset . We use their post-ReLU activations , and consider the activation of a convolutional feature map to be its L1 norm . Instead of linear correlation , we use the Spearman correlation ( which is the linear correlation of ranks ) because it is able to capture relationships which tend to monotonically increase even if they are nonlinear . After obtaining visualizations for each neuron in a sub-cluster , we do not directly take their average to visualize the entire sub-cluster . To see why , consider two neurons which are highly anticorrelated across the testing set . These neurons are highly coherent , but averaging together their visualizations would obscure this by cancellation . To fix this problem , we align the signs of the visualizations for individual neurons using a variant of an algorithm from Watanabe ( 2019 ) . To visualize a sub-cluster , for a number of iterations ( we use 20 ) , we iterate over its neurons , and calculate for each the sum of cosines between its visualization and each of the other neurons ’ visualizations in vector form . If this sum is negative , we flip the sign of this neuron ’ s visualization . Refer to appendix A.3 for a complete algorithmic description . After this procedure , we take the mean of the visualizations within a sub-cluster . To see how much meaningful input coherence these sub-clusters exhibit , we compare them to random sub-clusters ( recall each of these are randomly selected sets of neurons of the same size from the same layer as a true sub-cluster ) . Figure 1a-b shows results from MLPs trained on MNIST and Fashion-MNIST . Here , these MLPs are trained with dropout which we found to be helpful for clearer visualizations . In the first row of each image are visualizations for true sub-clusters , and the bottom four rows show visualizations for random ones . The true sub-clusters in the top row produce more coherent visualizations with better-defined and higher-contrast features compared to the random ones in the bottom 4 rows . Next , we hypothesized that if we trained a network on a task that lent itself well to parallel processing , spectral clustering would capture specialized modules . To test this , we designed “ halves-same ” and “ halves-diff ” tasks for small MLPs based on the MNIST and Fashion-MNIST datasets . For the halves-same tasks , two images of the same class were resized to have half their original width and concatenated side-by-side in order to create a composite image of the same size as the originals . We gave these images the same label as their component halves . For the halves-diff tasks , this was done with two images from random classes , and the resulting image was labeled with the sum of their labels modulo 10 . Example images from each of the the halves-same/diff MNIST and Fashion-MNIST datasets are shown in figure 3 . We expected that the halves-diff task would be more economical to compute in a modular way by separately recognizing the two halves and computing their modular sum . In appendix A.3 , we show that our networks can compute this modular sum . Figure 1c-d shows these visualizations for MLPs trained with dropout on halves-same MNIST and without dropout on halves-diff MNIST . We did not use dropout to train the halves-diff networks because it resulted in poor accuracy . This is likely because while amenable to image classification , dropout is not amenable to modulo arithmetic . Columns are arranged from left to right in the order of the layer in which they appear in the network . Visualizations for the halves-same networks tend to result in similar left and right halves , but in the early ( leftmost ) layers of the networks trained on the halves-diff tasks , there is a tendency for true sub-clusters to be selective to one half . This method of understanding input coherence has the advantage of being able to provide intuitive visual examples and efficiently construct interpretable features for MLPs . However , it was not as effective for CNNs . In appendix A.3 we detail this process , and in figure 4 , we show visualizations for small CNNs in which we find less evidence of coherence among sub-clusters . To expand on the intuitive visual examples offered here , in the following section , we introduce a more versatile , scalable method along with hypothesis testing procedures for obtaining quantitative results .

Summary: The authors identify putative clusters of units/neurons in deep networks using spectral clustering on a graph defined by synaptic weights. The authors then argue that these structurally defined clusters of neurons have similar *functional representations*. Finding interpretable relationships between weight matrices and functional modules is challenging, and the authors should be applauded for attempting to tackle this challenging problem that few research groups are devoting energy to.