Detecting Modularity in Deep Neural Networks

A neural network is modular to the extent that parts of its computational graph ( i.e . structure ) can be represented as performing some comprehensible subtask relevant to the overall task ( i.e . functionality ) . Are modern deep neural networks modular ? How can this be quantified ? In this paper , we consider the problem of assessing the modularity exhibited by a partitioning of a network ’ s neurons . We propose two proxies for this : importance , which reflects how crucial sets of neurons are to network performance ; and coherence , which reflects how consistently their neurons associate with features of the inputs . To measure these proxies , we develop a set of statistical methods based on techniques conventionally used to interpret individual neurons . We apply the proxies to partitionings generated by spectrally clustering a graph representation of the network ’ s neurons with edges determined either by network weights or correlations of activations . We show that these partitionings , even ones based only on weights ( i.e . strictly from non-runtime analysis ) , reveal groups of neurons that are important and coherent . These results suggest that graph-based partitioning can reveal modularity and help us understand how deep neural networks function . 1 INTRODUCTION . Modularity is a common property of complex systems , both natural and artificial ( Clune et al. , 2013 ; Baldwin & Clark , 2000 ; Booch et al. , 2007 ) . It carries advantages including intelligibility and adaptivity . In particular , modularity implies that the structure of a system conveys information about its functionality . This leads us to the following definition : a neural network is modular to the extent that parts of its computational graph ( structure ) can be represented as performing some comprehensible subtask relevant to the overall task ( functionality ) . We call these parts modules . There exists a body of research for developing more modular networks which either have distinct architectural building blocks ( Alet et al. , 2018 ; Parascandolo et al. , 2018 ; Goyal et al. , 2019 ) or are trained in a way that promotes modularity via regularization or parameter isolation ( Kirsch et al. , 2018 ; De Lange et al. , 2019 ; Filan et al. , 2021 ) . Yet in machine learning , it is more common to encounter networks whose architecture and training are not guided by considerations of modularity . For example , in computer vision , networks are generally trained end-to-end with all of the filters in one layer connected to all filters in the next . Do such networks nonetheless develop modularity ? There is some evidence for this . For example , by methodical manual investigation , Cammarata et al . ( 2020 ) discovered modular subnetworks which perform human-explainable subtasks such as car detection via neurons which detect different car parts . More scalably detecting modularity in networks would help us to extend our understanding of their learning dynamics and to expand our interpretability toolbox by suggesting an additional level of abstraction beyond single-neuron methods . In this paper we systematically analyze the extent to which networks which are not explicitly trained to be modular exhibit it nonetheless . First , this requires a method for breaking down a network ’ s computational graph into different proposals for modules . For this , we use spectral clustering on a graph representation of the network , using an extension of the methods of ( Filan et al. , 2021 ) . Second , this requires a scalable method for approximating the degree to which a partitioning exhibits modularity . We do this by applying interpretability tools to clusters of neurons as a way of quantifying proxies for modularity . Operationalization of modularity : Following our notion of modularity , to say that a subset is a module , one must show that it performs some human-comprehensible subtask relevant to the overall task . Scaling this , however , would be difficult , as it would require a human in the loop . How could measuring modularity be automated ? Consider an idealized prototype of a highly modular network that has subsets of neurons , each performing subtasks in which ( 1 ) each subtask is necessary for high performance on the overall task , ( 2 ) each subtask is implemented by a single subset only , and ( 3 ) each subset executes only a single subtask . The combination of ( 1 ) and ( 2 ) suggests that the removal of one of the subsets from the network would harm performance because the network lacks the implementation of a necessary subtask . We say that such a subset is important . Next , given that neurons are frequently understood as feature detectors , ( 3 ) suggests that the neurons in a module should tend to be strongly activated by inputs which contain features relevant to the module ’ s subtask . We say that such a subset is coherent . Figure 1 provides an illustrative example of coherence in networks that are trained on a task that lends itself to parallel processing via subtasks . Measuring importance and coherence will give us a sense of the degree to which networks approximate these prototypical conditions , despite not perfectly satisfying them . Results and contributions : To measure these proxies , we utilize interpretability methods that have been conventionally used for single-neuron analysis to study these partitions . We find that the partitions have groups of neurons that are disproportionately likely to be important compared to random groups of neurons . We also find that the groups of neurons in the partitionings are reliably more coherent than random ones , though only with respect to features other than class label . By showing that our partitioning methods are able to reveal modularity , these results suggest that they can be used to understand deep networks . Our key contributions are threefold : 1 . Introducing two proxies , importance and coherence , to assess whether a partitioning of a network exhibits modularity and identify which subsets of neurons are the most responsible . 2 . Quantifying these proxies by applying single-neuron interpretability methods on subsets of neurons in an automated fashion 3 . Applying our methods on the partitions produced by spectral clustering on a range of neural networks and finding evidence of modularity captured by these partitions . 2 RELATED WORK . The work most closely related to ours is Filan et al . ( 2021 ) who also use spectral clustering to establish that deep networks are often clusterable and investigates what factors influence clusterability . They also introduce a method for regularization for modularity among clusters of neurons . We extend their work by bridging graphical clusterability and functional modularity . This line of work inherits insights from network science involving clustering in general ( Girvan & Newman , 2002 ; Newman & Girvan , 2004 ) , and spectral clustering ( Shi & Malik , 2000 ; von Luxburg , 2007 ) in particular . In our experiments we combine clustering with interpretability tools to measure importance and coherence . This highlights the usefulness of combining clustering with interpretability methods for more rigorously understanding networks . We use feature visualization ( Olah et al. , 2017 ; Watanabe , 2019 ) and lesions ( Zhou et al. , 2018 ) , but in a similar way , other interpretability techniques including analysis of selectivity ( Morcos et al. , 2018 ; Madan et al. , 2020 ) , network “ dissection ” ( Bau et al. , 2017 ; Mu & Andreas , 2020 ) , earth-mover distance ( Testolin et al. , 2020 ) , or intersection information ( Panzeri et al. , 2017 ) could also be combined with clustering-based partitionings under a similar framework . Relatedly , Cammarata et al . ( 2020 ) demonstrate that feature visualization and analysis of weights can be used to identify groups of neurons whose functionality is human-interpretable . This work adds to a body of research focused on modularity and compositionality in neural systems either at the neuron level ( You et al. , 2020 ; Mu & Andreas , 2020 ; Voss et al. , 2021 ) or at the subnetwork level ( Lake et al. , 2015 ; 2017 ; Csordás et al. , 2021 ) . There also exist techniques for developing more modular networks which either have an explicitly modular architecture ( Alet et al. , 2018 ; Parascandolo et al. , 2018 ; Goyal et al. , 2019 ) or are trained in a way that promotes modularity via regularization or parameter isolation ( Kirsch et al. , 2018 ; De Lange et al. , 2019 ) . 3 METHODS . To evaluate modularity , our procedural pipeline is as follows . ( 1 ) We begin with a trained neural network ; ( 2 ) construct a graph from it , treating each neuron as a node ; ( 3 ) perform spectral clustering on the graph to obtain a partitioning or “ clustering ” of neurons which we then further divide by layer to obtain a “ subclustering ” ; ( 4 ) analyze the subclusters according to our modularity proxies operationalized by lesioning neurons or feature visualization and comparing them to random subclusters ; and ( 5 ) aggregate results across the network to obtain a final p value and effect measure . This pipeline is outlined in Figure 2 and Figure 4 , and each step is explained in detail below . 3.1 GENERATING PARTITIONS WITH SPECTRAL CLUSTERING . To partition a network into clusters , we use an approach based on Filan et al . ( 2021 ) which consisted of two steps : “ graphification ” - transforming the network into an undirected , edge-weighted graph ; and clustering - obtaining a partitioning via spectral clustering . Graphification : To perform spectral clustering , a network must be represented as an undirected graph with non-negative edges . For MLPs ( multilayer perceptrons ) , each graph node corresponds to a neuron in the network including input and output neurons . For CNNs ( convolutional neural networks ) , a node corresponds to a single channel ( which we also refer to as a “ neuron ” ) in a convolutional layer.1 For CNNs , we do not use input , output , or fully-connected layers for clustering . For graphification , we test two ways of assigning adjacency edges between neurons : with weights and with correlations . For weight-based clustering with dense layers , if two neurons have a weight connecting them in the network , their corresponding vertices are connected by an edge with weight equal to the absolute value of the network ’ s weight between the neurons . For convolutional channels , we connect them by an edge with weight equal to the L1 norm for the corresponding 2D kernel slice . If layers are connected but with a batch-normalization layer in between , we mimic the scaling performed by the batch norm operation by multiplying weights by γ/ ( σ + ε ) where γ is the scaling factor , σ is the moving standard deviation , and ε is a small constant . Notably , this method of constructing the graphs requires no dataset or runtime analysis of the network . While graphification via weights only results in connections between neurons in adjacent layers , doing so with correlations creates more dense graphs . With this method , we connect the nodes for two neurons with their squared Spearman correlation across a validation set . Spearman correlation gives the Pearson ( linear ) correlation between ranks and reflects how well two sets of data can be related by a monotonically increasing function.2 Rather than using neurons ’ post-ReLU outputs to calculate these correlations , we use their pre-ReLU activations with the goal of extracting richer data from neurons.3 Again , for convolutional channels , we take the L1 norm of activations before calculating Spearman correlations . The fact that we take absolute valued weights or squared correlations to construct edges between neurons in graphification means that our analysis does not discriminate between positive and negative associations between neurons . In addition to graphification via weights versus activations , we also test two scopes in which to perform clustering : global and local . For global clustering , we cluster on one graph for the network as a whole . For local clustering , we produce a partitioning for each layer l individually by clustering on the graph of connections between l and the layers adjacent to l. Ultimately , we run 4 experiments on each network by clustering { weights , activations } × { globally , locally } . Spectral ( sub ) clustering : We perform normalized spectral clustering ( Shi & Malik , 2000 ) on the resulting graphs to obtain a partition of the neurons into clusters . For all experiments , we set the number of clusters to k = 16 . In Appendix A.6 , we reproduce a subset of experiments with k ∈ { 8 , 12 } , showing that results are robust to alternate choices of k. Refer to appendix A.1 for a complete description of the spectral clustering algorithm . Layers at different depths of a network tend to develop different representations . To control for these differences , we study the neurons in clusters separately per layer . Therefore , for global clustering in which clusters of neurons span more than one layer , we analyze clusters one layer at a time . We call these sets of neurons within the same cluster and layer subclusters . To ensure comparability between these clusterings when performing local clustering , we set the number of clusters per layer to be the same as the number produced in that layer with global clustering . In our experiments , we compare 1This is subtly different from treating each activational unit as a neuron in networks such as ResNets with skip connections . If a unit is used as inputs to multiple layers , we consider these inputs to be separate neurons . 2We use Spearman rather than Pearson correlation because networks are nonlinear , and there is no particular reason to expect associations between arbitrary neurons to be linear . 3Although all negative values are mapped to zero by a ReLU , we expect the degree of negativity to carry information about the possible presence ( or lack thereof ) of features which the neuron was meant to detect , especially for networks trained with dropout . these subclusters to other sets of random units of the same size in the same layer . When discussing these experiments , we refer to subclusters from the clustering algorithm as “ true subclusters ” and sets of random neurons as “ random subclusters. ” Random subclusters form the natural control condition to test whether the specific partitioning of neurons exhibits importance or coherence compared to alternative partitions , while taking account of layer and size .

Summary: This paper uses graph-based clustering methods to identify the humanly comprehensible modularities in the neural network. They thoroughly discussed how they construct the graph of neurons and carry out the clustering. They also proposed the fisher bates p-value for testing the significance of identified modules.