Importance and Coherence: Methods for Evaluating Modularity in Neural Networks
Authors: Shlomi Hod
Stephen Casper
Daniel Filan
Cody Wild
Andrew Critch
Stuart Russell
Venue:  (2021)
Keywords: interpretability
modularity

Abstract: As deep neural networks become more advanced and widely-used, it is important to understand their inner workings. Toward this goal, modular interpretations are appealing because they offer flexible levels of abstraction aside from standard architectural building blocks (e.g., neurons, channels, layers). In this paper, we consider the problem of assessing how functionally interpretable a given partitioning of neurons is. We propose two proxies for this: importance which reflects how crucial sets of neurons are to network performance, and coherence which reflects how consistently their neurons associate with input/output features. To measure these proxies, we develop a set of statistical methods based on techniques that have conventionally been used for the interpretation of individual neurons. We apply these methods on partitionings generated by a spectral clustering algorithm which uses a graph representation of the network's neurons and weights. We show that despite our partitioning algorithm using neither activations nor gradients, it reveals clusters with a surprising amount of importance and coherence. Together, these results support the use of modular interpretations, and graph-based partitionings in particular, for interpretability.

Under review as a conference paper at ICLR 2021
IMPORTANCE AND COHERENCE: METHODS FOR EVALUATING MODULARITY IN NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
As deep neural networks become more widely-used, it is important understand their inner workings. Toward this goal, modular interpretations are appealing be- cause they offer flexible levels of abstraction aside from standard architectural building blocks (e.g.. neurons, channels, layers). In this paper, we consider the problem of assessing how functionally interpretable given partitioning of neu- rons is. We propose two proxies for this: importance which reflects how crucial sets of neurons are to network performance, and coherence which reflects how consistently their neurons associate with input/output features. To measure these proxies, we develop a set of statistical methods based on techniques that have con- ventionally been used for the interpretation of individual neurons. We apply these methods on partitionings generated by a spectral clustering algorithm which uses graph representation of the network's neurons and weights. We show that de- spite our partitioning algorithm using neither activations nor gradients, it reveals clusters with surprising amount of importance and coherence. Together, these results support the use of modular interpretations, and graph-based partitionings in particular, for interpretability.
INTRODUCTION
Deep neural networks have achieved state-of-the-art performance in variety of applications, but this success contrasts with the challenge of making them more intelligible. As these systems become more advanced and widely-used, there are number of reasons we may need to understand them more effectively. One reason is to shed light on better ways to build and train them. second reason is the importance of transparency, especially in settings which involve matters of safety, trust, or justice (Lipton| 2018). More precisely, we want methods for analyzing a trained network that can be used to construct semantic and faithful deseriptions of its inner mechanisms. We refer to this as mechanistic transparency.
Toward this goal, we consider modularity as an organizing principle to achieve mechanistic trans- parency. In the natural sciences, we often try to understand things by taking them apart. Aside from subdivision into the standard architectural building blocks (e.g., neurons, channels, layers), are there other ways trained neural network be mcaningfully "taken apart"? We aim to analyze a network via a partitioning of its neurons into disjoint sets with the hope of finding that these sets are "mod- ules" with distinct functions. Since there are many choices for how to partition network, we would like metrics for anticipating how meaningful given partition might be.
Inspired by the field of program analysis (Fairley 1978), we apply the concepts of "dynamic" and "static" analysis neural networks. Dynamic analysis includes performing forward passes and/or computing gradients, while static analysis only involves analyzing architecture and parameters. In a concurrent submission (Anonymous et al. (022), we use spectral clustering to study the extent to which networks form clusters of neurons that are highly connected internally but not externally and find that in many cases, networks are structurally clusterable. This approach is static because the partitioning is produced according the network's weights only, using neither activations gradients. Here, we build off of this concurrent submission by working to bridge graph- based clus- terability and functional modularity.
To see how well neurons within each cluster share meaningful similarities, we introduce two proxies: importance and coherence. Importance refers to how crucial clusters are the network's perfor-

---

Under review as a conference paper at ICLR 2021
mance overall and lends insight into how well a partition identifies clusters that are individually key to the network's 's function. Coherence refers to how consistently the neurons within a cluster cor- respond in their activations to particular features in data. We analyze coherence both with respect to input features and output labels. To measure these proxies, we utilize dynamic interpretability methods that have been conventionally used for single-neuron analysis to the study of these parti- tions. We conduct a set of experiments and hypothesis tests in networks scaling from the MNIST to the ImageNet level. In doing so, we show that spectral clustering capable of identifying function- ally important and coherent clusters of neurons. This new finding the and methods we present for combining spectral clustering with dynamic methods supports the use of modular decompositions of neurons toward mechanistic transparency.
Our key contributions are threefold:
1. Introducing two proxies, importance and coherence, to assess whether a given partitioning of a network exhibits modularity.
2. Quantifying these two proxies with interpretability methods equipped with statistical hypothesis testing procedures.
3. Applying our methods on the partitions produced by the spectral clustering technique of Anonymous et al. (202T) on range of networks, and finding evi- dence of modularity among these clusters.
2 GENERATING PARTITIONINGS WITH SPECTRAL CLUSTERING
In our concurrent submission, we introduce and study in-depth a procedure to partition neural network into disjoint clusters of neurons (Anonymous et al.) 2021) based only on its weights. We found that trained networks are more clusterable than randomly initialized ones, and they are also often more clusterable than similar networks with identical weight distributions. The experimental procedure consists of three steps: (1) "Graphification" "- transforming the network into undirected edge-weighted graph; (2) Spectral clustering obtaining a partitioning via spectral clustering of the graph.
To perform spectral clustering, a network must be represented as an undirected graph with non-negative edges. For MLPs (multilayer perceptrons), each graph vertex corresponds to neuron in the network including input and output neurons. If two neurons have weight connect- ing them in the network, their corresponding vertices are connected by an edge giving its absolute value. For CNNs (convolutional neural networks). a vertex corresponds to a single feature map (which we also refer as a "neuron") "") in a convolutional layer. Here, we do not use input, output, or fully-connected layers. If two feature maps are in adjacent convolutional layers, their corresponding vertices are connected with an edge giving the L, norm for the corresponding 2 dimensional kernel slice. If convolutional layers are separated by a batch normalization layer (loffe & Szegedy (0155), we multiply weights by 7/(a + +s) where 7 is the scaling factor, o is the moving standard deviation, and is a small constant.
Graphification:
Spectral Clustering: We run normalized spectral clustering on the resulting graph (Shi & Malik 2000) to obtain partition of the neurons into clusters. For all experiments, clusters to 12 unless explicith
set the number of explicitly mentioned otherwise. We choose 12 because (1) it is computationally tractable, (2) it is larger than the number of classes in MNIST and CIFAR-10, and (3) it is small compared to the number of neurons in the layers of all of our networks. However, in Appendix[A.6) show results for k 8 and 18 for a subset of experiments and find major differences. We use the scikit-learn implementation (Pedregosa et al.] 2011) with the ARPACK eigenvalue solver
(Borzi & Borzi 2006). Refer to appendix A.I for complete description of the algorithm.
EVALUATION OF MODULARITY USING IMPORTANCE AND COHERENCE
Clusters of neurons produced by spectral clustering span more than one layer. However, layers at different depths of a network tend to develop different representations. To control for these differ- ences, we study the neurons in clusters separately per layer. We call these sets of neurons within the same cluster and layer "sub-clusters." In our experiments, we compare these sub-clusters to other sets of random units of the same size and same layer. When discussing these experiments, refer
2

---

Under review as a conference paper at ICLR 2021
to the sub-clusters from the clustering algorithm as "true sub-clusters" and the sets composed of random neurons as "random sub-clusters." Random sub-clusters form the natural control condition to test whether the specific partitioning of neurons exhibits importance or coherence compare to alternative partitions, while taking account location and size.
As outlined in the Introduction, we study importance: how crucial each sub-cluster is the network; input coherence: how well neurons sub-cluster associate with similar input features; and output coherence, how well they associate with particular output labels, as proxies for modularity, In this section, we present two types of experiments. First, we use visualization techniques on sub-clusters measure imput and output coherence, and second, we use "lesion tests" based on dropping out neurons in sub-cluster to measure output coherence and importance.
These techniques are scalable, and we experiment with a wide range of networks. For small-scale experiments, we train and analyze MLPs with four hidden layers of 256 neurons each and small convolutional networks with 3 layers of 64 neurons each followed by a dense layer of 128 neurons trained on the MNIST (LeCun et al.) 1998) and Fashion-MNIST (Xiao et al.) (017) datasets. At mid scale, we train and analyze VGG-style CNNs containing 13 convolutional layers using the archi-
tectures from Simonyan & Zisserman (2014) trained on CIFAR-10 (Krizhevsky et al.
2009) using
the procedure from Liu & Deng (2015). Finally, for the ImageNet (Krizhevsky et al.
2009) scale,
amalyze pretrained ResNet18, ResNet50, (He al. 2016) VGG-16, and VGG-19
Simonyan &
Zisserman 2014) models.
In our concurrent submission (Anonymous al. 2021) we show that in cases, weight pruning and dropout can each be used to promote graph-based clusterability. We use pruning in small MLPs but no other networks. We use dropout for MLPs in correlation-based visualization experiments in subsection 3.1.1 but no other MLPs. Also, for the mid-sized VGG-CNNs, we experiment both with versions that are unregularized and which are regularized using dropout and L2 regularization as done in Liu & Deng (2015). Complete training details including testing accuracies are in the appendix A.2
3.1 FEATURE VISUALIZATION
3.1.1
CORRELATION- -AADDD VISUALIZATION
First, we introduce here simple method to provide visual examples and build intuition. In later subsections, we present a quantitative approach with statistical hypothesis testing. A simple way to visualize a sub-cluster is to identify what input features each of its neurons respond to and then use these to create an aggregated visualization. We do this for small MLPs in which we construct visualizations of neurons using their correlations with the input pixels across the test dataset. We use their ReSt activations, and consider the activation of convolutional feature map to be its Lı norm. Instead linear correlation, we use the Spearman correlation (which is the linear correlation of ranks) because it is able to capture relationships which tend to monotonically increase even if they are nonlinear.
After obtaining visualizations for each neuron in sub-cluster, do not directly take their average to visualize the entire sub-cluster. To see why, consider two neurons which are highly anticorrelated across the testing set. These neurons are highly coherent, but averaging together their visualizations would obscure this by cancellation. To fix this problem, we align the signs of the visualizations for individual neurons using variant of an algorithm from Watanabe (2019). To visualize a sub-cluster, for a number of iterations (we use 20), we iterate over its neurons, and calculate for each the sum of cosines between its visualization and each of the other neurons' visualizations in vector form. If this sum is negative, we flip the sign of this neuron's visualization. Refer to appendix A.3 for a complete algorithmic description. After this procedure, we take the mean of the visualizations within sub-cluster.
To see how much meaningful input coherence these sub-clusters exhibit, we compare them to ran- dom sub-clusters (recall each of these are randomly selected sets of neurons of the same size from the same layer as a true sub-cluster). Figure - shows results from MLPs trained on MNIST and Fashion-MNIST Here, these MLPs are trained with dropout which we found to be helpful for clearer visualizations. In the first row of each image are visualizations for true sub-clusters, and the bottom four rows show visualizations for random ones. The true sub-clusters in the top row
3

---

Under review as a conference paper at ICLR 2021
produce more coherent visualizations with better-defined and higher-contrast features compared to the random ones in the bottom 4 rows.
Next, hypothesized that if we trained a network task that lent itself well to parallel process- ing. spectral clustering would capture specialized modules. To test this, we designed "halves-same" and "halves-diff" tasks for small MLPs based the MNIST and Fashion-MNIST datasets. For the halves-same tasks, two images of the same class were resized to have half their original width and concatenated side-by-side in order create a composite image of the same size the origi- nals. We gave these images the same label as their component halves. For the halves-diff tasks, this was done with two images from random classes, and the resulting image was labeled with the sum of their labels modulo 10. Example images from each of the the salves--aam/difff MNIST and Fashion- -NNIST datasets are shown in figure We expected that the halves-diff task would more economical to compute in modular way by separately recognizing the two halves and computing their modular sum. In appendix A.3 we show that our networks can compute this modular sum.
Figure - shows these visualizations for MLPs trained with dropout balves-same MNIST and without dropout on halves-diff MNIST. We did not use dropout to train the halves-diff networks because it resulted in poor accuracy. This is likely because while amenable to image classification, dropout is not amenable to modulo arithmetic. Columns are arranged from left to right in the order of the layer in which they appear in the network. Visualizations for the halves-sare to result in similar left and right halves, but in the early (leftmost) layers of the networks trained on the valves-diff tasks, there is tendency for true sub-clusters be selective one half.
networks tend
7
7
(a)
(b)
(c)
M DO
09 10 TO
D
(d)
O7 OEOUROO
Figure 1: Sub-cluster visualizations for small MIPs: (a) MNIST, (b) Fashion-MNIST, (c) halves- same MNIST, (d) halves-diff MNIST. In each image, the top row gives visualizations true sub- clusters and the bottom four for random ones of the same size. Columns are ordered left to right according to layer order. Pixel values are scaled from black to white for each column independently. All except (d) were trained with dropout.
This method of understanding input coherence has the advantage of being able to provide intuitive visual examples and efficiently construct interpretable features for MLPs. However, it was not as effective for CNNs. In appendix A.3 we detail this process, and in figure we show visualizations for small CNNs in which we find less evidence of coherence among sub-clusters. To expand on
4

---

Under review as a conference paper at ICLR 2021
the intuitive visual examples offered here, in the following section, we introduce versatile, scalable method along with hypothesis testing procedures for obtaining quantitative results.
3.1.2 INPUT CONSTRUCTION
Another way to visualize neurons in a network is to use gradient-based optimization create an input image which maximizes the activation of a neuron, or in our case, a sub-cluster of them. Pat- terns in the resulting visualizations can suggest what features the neurons respond to. We visualize sub-clusters with this method (Olah et al. 2017) using the Lucid package. Implementation details are in appendix A.8 Figure gives example visualizations.
To obtain quantitative results, we used two techniques. First, we analyzed the value of the maxi- mization objective for each image we produced, which we call the "score." This gives of one notion of how coherent sub-cluster may be with respect to input features, because if a single image can activate an entire sub-cluster well, this suggests that the neurons comprising can be activated by similar features. Second, we analyze the entropy of the softmax outputs of the network when these images are passed through it. If the entropy of the softmax distribution is low, this suggests that a cluster is coherent with respect to outputs.
We then test the null hypothesis that these sub-clusters are equally coherent as random sets of neu- rons. For each sub-cluster in network with at least three neurons and at most 80% of the neurons in a layer, we compare its visualization' 's score and output entropy to those of 9 random sub-clusters. We then obtain one-sided p values by taking the percentiles for the true sub-cluster's 's score and en- tropy relative to the random sub-clusters' score and entropy. We take right-sided values for scores and left-sided p values for output entropies so that lower p values indicate greater input/output co- herence in both cases. We then use two different methods to combine all sub-cluster p values to obtain combined p value for the entire network for either score or entropy. Both are presented here, but full details for both are in appendix A.4.
Fisher Method: First, we center the sub-cluster p values around 0.5 obtain a granular approxima- tion of the uniform distribution under the null, and then use the Fisher Method. The test statistic for a set of sub-cluster p values pI..pn is 2 Σm log pa which takes chi squared distribution with 2n degrees of freedom under the null hypothesis.
Second, since there are only a set number, m, of values which the p values can take (in our case m 10), we perform Chi Squared categorical test to see whether their distribution among these discrete values is nonuniform. The test statistic is Eis (a-=0]2 in which each 2; gives an observed count and each 14: gives a expected one. It will have a chi squared distribution with m degrees of freedom under the null hypothesis.
Chi Squared Method:
Me
These methods test for different things. The Fisher method indicates how low the values for sub- clusters tend to be across network and tests whether the true sub-clusters consistently more coherent than random ones. However, the distribution of sub-cluster values may be nonuniform but in a way that the Fisher Method is not designed to detect. For example, they may tend to be very high or follow a U-shaped distribution. The Chi Squared method adds additional resolution by detecting cases like this.
The top section of table I summarizes these results. For each network which we perform this test on, we provide the Fisher and Chi Squared categorical p values for both the score (input coherence) and output entropy (output coherence). For the non- ImageNet networks, we report results for each measure (separately) as median across networks. We find strong evidence of significant levels of input coherence in the VGG family of networks, and find that the unregularized VGGs trained on CIFAR-10 also seems to exhibit a significant amount of output coherence. In Appendix [A.8] we also present experiments for understanding variance of activations in true and random sub-clusters.
3.2 LESION TESTS
Another set of tools that has been used for understanding both biological GGazzaniga & Ivry 2013) and artificial (Zhou et al. 2018, Casper al. 2020) neural systems involves disrupting neurons dur-
5

---

Under review as a conference paper at ICLR 2021
Visualization Score:
Input Coherence
Softmax Entropy: Output Coherence
Network
Fisher Combined
Chi Squared p
Fisher Combined
Chi Squared p
MLP MNIST
0,575
0.810
0.377
0.365
CNN MNIST
0.671
0.328
0.504
0.135
VGG CIFAR (Unreg)
3.45 10 10-4
8.79 10 10-9
2.00 10 7
1.29 10
VGG CIFAR
0.035
3.19 x 10
0.891
0.082
VGG-16 ImageNet
2.54 10
1.41 x 10
0.118
0.281
VGG-19 ImageNet
0.003
5.69 x 10
0.633
0.888
ResNet50 ImageNet
0.355
0.437
0.435
0.812
Accuracy Change:
Importance
Classwise Range: Output Coherence
Network
Fisher Combined p
Chi Squared p
Fisher Combined p
Chi Squared p
MLP MNIST
1.34 x 10
4.40 x 10 14
0.968
0.035
MLP Fashion
7.73 10
1.76 x 11
0.513
0.003
CNN MNIST
0.013
0.035
0.480
0.561
CNN Fashion
0.025
0.483
0.738
0.283
VGG CIFAR (Unreg)
2.25 x 10 11
4.30 x 10
3.19 x 10 10
3.47 x 10 33
VGG CIFAR
7.17 10
1.55 x 10
5.49
10
7.93
10
ResNet-18 18 ImageNet
0.002
1.51 x 10-6
0.112
0.001
VGG- 16 ImageNet
9.81 x 10-13
1.56 x 10-40 40
1.000
1.62
10-23
Table 1: Combined p values for feature visualization and lesion experiments: Each value is a combination of p values for all of a network's sub-clusters created with the Fisher or Chi Squared method. For ImageNet networks, the values reflect results for single networks, but for all others, values (separately) give median across independently trained replicates. We do not ascribe any particular epistemic significance to the threshold of 0.05 (also note that significance thresholds normally need be adjusted when taking a median of p values) but we bold all p values less than it. In Appendix A.7 we present corrections for multiple comparisons. In Appendix [A.]] we show results for A = 8 and k = 18. Feature Visualization (Top): Results indicate tests for visualizations produced with Lucid. Lesions (Bottom): Results indicate tests involving lesions of sub-clusters. Due to computational demands, we limit our analysis here at the ImageNet scale to ResNet-18 and VGG-16 models.
ing inference. Whereas the images produced with feature visualization were optimized to maximally activate a sub-cluster, we perform a dual type of experiment with "lesion" tests in which we analyze network outputs when a sub-cluster is dropped out. When lesioning a sub-cluster, we set all weights incoming to the constituent neurons to O, while leaving the rest of the network untouched. Refer to figure 6 for example plots of the accuracy drops for a small MLP and CNN trained on Fashion- MNIST. We then determine the damage to the network's 's overall and per-class testing accuracy. This allows us to evaluate both importance and output coherence.
Importance Importance allows to identify which sub-clusters are key the network and there- fore of particular interest for scrutiny. To systematically quantify the importance of a network's sub-clusters in aggregated way, we combine the right-sided p values of all the network's true sub- clusters using the Fisher and Chi Squared categorical methods discussed above 3.1.2 and described
in detail in appendix A.4 Note that these experiments are analogous those in 3.1.2 The bottom
section of table gives results for these. We find strong evidence across the networks which we train that spectral clustering reveals important sub-clusters.
There is generally significant diversity among sub-clusters in their size, importance, and importance relative to random sub-clusters. To demonstrate this, we construct an example descriptive taxonomy, in which we consider three criteria for identifying particularly important sub-clusters. First, the sub- cluster should be at least 5% of the neurons of the layer. Second, the drop in accuracy under lesion should be greater than percentage point; and third, the drop should not simply be due to the number of damaged neurons. To evaluate the third criterion, we generate random sub-clusters with the same number of neurons as the true sub-cluster from the same layer, and collect the distribution of accuracy drops. We say that this criterion is met the accuracy drop for the true sub-cluster is greater than all of 20 random sub-clusters, i.e its p value is smaller than 1/20.

---

Under review as a conference paper at ICLR 2021
In figure we plot sub-cluster size versus accuracy drop for an MLP trained Fashion-MNIST and a VGG trained on CIFAR-10 10 that has been clustered into 8 clusters (we use for the sake of visualization here, but use 12 clusters for all quantitative experiments). Many sub-clusters are too small to be counted as important, and many are significantly impactful compared to random sub-clusters but not practically significant. However, some clearly are practically important for the functioning of the network.
MLP (with pruning): MNIST-Fashion
VGG (with reg. and dropout): CIFAR-10
3-7
2-0
3-0
-10
1-2
-20
Diffrrnce
-10
-30
-40
-15
A-7
-60
-20
-70
11-62-5
-25
-80
20
30
40
50
60
60
Proportion in Layer (%)
Proportion In Layer (%)
Important
Sig-out-not-diff
Diff-but-not-sig
Complete
Prunable
Small
Other
Figure 2: Plot of difffrent sub-clusters of two networks. Important sub-clusters are labeled first by their layer number and then their cluster number. The horizontal axis shows the proportion of their layer's neurons in the sub-cluster, and the vertical axis shows the reduction accuracy from lesioning it. 'Important" means that the sub-cluster makes up at least 5% of the layer, that the drop in accuracy is greater than one percentage point, and that was more important than all of 20 random sub-clusters it compared against-ther that is say, statistically significant. 'Sig- but-not-diff' means that the drop in accuracy is statistically significant but less than percentage point, DDiff-but-not-sig means that the lesioning damage was more than percentage point but not statistically significant, 'Prunable" means that the drop in the accuracy is smaller than all random shuffles and smaller than percentage point, 'Complete' means that the sub-cluster contains the whole layer, 'Small' means that the sub-cluster consists of less than 5% of the layer, and 'Other' means that the drop in accuracy is not statistically significant and less than percentage point. Both of the networks are partitioned into 8 clusters. The data is based on two tables which are included in appendix [A.9 Note that the 'Complete' category appears only in the right plot, and only a single point is shown although there are 8 such sub-clusters. Refer appendix A.9 for additional details.)
Coherence To measure the output coherence using lesions, analyze the accuracy changes for each of the output classes. For ten classes, we define d = (do, di....,dg)) where d, is the change in the i-th class accuracy due the lesioning of sub-cluster. In order obtain measurement independent of the overall importance, we divide these class-wise accuracy changes by their mean, d' d/d, and the then take their range A max d' min class-wise range. We compare true and random sub-clusters to obtain right-sided value for each sub-cluster based on the p values of the true A. We then combine these for the entire network using the Fisher and Chi Squared categorical methods as discussed above and detailed in appendix A.4
d'. We refer to this as the (normalized)
These results are the bottom section of table I The Chi Squared p values demonstrate that spectral clustering usually identifies sub-clusters with a significantly different distribution of impor- tances compared to random sub-clusters. Meanwhile, the Fisher tests suggests that at least in VGG networks trained on CIFAR-10, the sub-clusters exhibit more output coherence. Interestingly, for VGG-16s trained on ImageNet, the opposite seems to be the case. The Fisher p value is high, sug- gesting that the p values for its individual sub-clusters tend be high. However, the Chi Squared p value is low, suggesting nonuniformity among the sub-cluster values. Together, these indicate the the clusters are consistently less coherent than random ones.
7

---

Under review as a conference paper at ICLR 2021
4 RELATED WORK
The most closely-related work to this is our paper under concurrent submission (Anonymous et al.) 2021] which uses the same spectral clustering-basee approach to establish that deep networks are in many cases clusterable and investigates in depth methods can be used to control the development of clusterability. Both of these works inherit insights from network science involving clustering in general (Girvan & Newman [002] [
Newman & Girvan 2004), and spectral clustering (Shi & Malik, von Luxburg 2007 particular.
2000
Our experiments in which we combine spectral clustering with correlation-based visualization (Watanabe 2019), feature visualization (Olah et al. 2017), and lesions (Zhou et al., 2018) high- light the usefulness of combining multiple interpretability methods in order to build an improved set of tools for more rigorously understanding systems. In a similar way, other dynamic techniques for interpretability such as analysis of selectivity (Madan et al. (2220), network "dissection" (Bau
jet al. 2017 Mu & Andreas 2020), earth-mover distance (Testolin et al., 2020). or intersection information Panzeri et al. 2017) could also be combined with static graph-based partitionings un-
der a similar framework. There already exist examples of interpretability methods being used for the identification of unexpected adversarial weaknesses (Carter et al.) 2019 Mu & Andreas 2020)_ We expect that developing more powerful tools like these for scrutinizing networks will be helpful toward building more robust systems.
This work adds to a growing body of research focused on modularity and compositionality in neural systems (e.g. Lake et al. (2013 2017); Csordás et al. (2020); You al. (2020). This paradigm is useful both for interpretability and for building better models. Neural circuits with distributed, non-modular representations pose a litany of challenges including non-interpeetabliity. less useful representations, poorer generalization, catastrophic forgetting, and biological implausibility. One limitation of this work is a focus on clustering in models which have fairly monolithic architectures (e.g. all neurons/tilters in one layer being connected to all neurons/filters in the next). However, there exists a body of research focused specifically on developing more modular networks which either have an applicitly-modular architecture (Alet et al. 2018, Parascandolo al. 2018 Goyal et al.)
2019) or are trained in a way that promotes modularity via regularization or parameter isolation (Kirsch et al.
2018 De Lange al. 2019).
DISCUSSION
In this work, we introduce approach for evaluating whether partitioning of a network exhibits modular characteristics. Key to this is analyzing proxies: importance as a means of understanding what parts of network are crucial for performance, input/output coherence as measures for how specialized these parts are. We measure these proxies using statistical hypothesis testing procedures based on interpretability techniques which have conventionally been used for analyzing individual neurons. Though we analyze partitions produced by spectral clustering, static method, we find that these clusters exhibit significant amount of importance compared to random clusters. We also show that our networks in the VGG family also tend to exhibit a significant level of input coherence, and in some cases, output coherence. By and large, these findings, and those of a concurrent submission AAnnyyouss et al. 2021). support the analysis of modules, and in particular graph-based clusters of neurons, for developing a better understanding of neural networks' inner- workings.
Building framework for evaluating modularity in neural networks can can guide the development of new interpretability methods which examine networks at the module level. Toward this goal, compositionality how modules are combined and interact together, be another proxy of mod- ularity. For evaluating this, some of our methods can be extended to study dependencies between clusters. In appendix AA.0 we present exploratory lesion-based experiments for studying cluster interactions and constructing dependency graphs.
While we make progress here toward mechanistic transparency, neural systems are still complex, and more insights are needed to develop richer understandings. The ultimate goal would be to master the process of building compositional systems which lend themselves to simple and faithful semantic interpretations. We hope that using modularity as an organizing principle to achieve mechanistic transparency and expanding our interpretability toolbox with combined static and dynamic methods will lead to richer understanding of networks and better tools for building them to be reliable.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jazefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogencous systems, 2015. URL hitps: httpp:://ww..c........... available from tensorflow.org.
//wps://wwww..tm tww.orfoosr.....oo org/ Software
Ferran Alet, Tomás Lozano-Péree and Leslie P Kaelbling. Modular meta-learning. arXiv preprint arXiv: 10166, 2018.
Alice Anonymous, Bob Anonymous, Charlie Anonymous, and David Anonymous. Clusterability in neural networks (currently under submission). Association for the Advancement of Artificial Intelligence, 2021.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. Proceedings of IEEE conference computer vision and pattern recognition, pp. 65599, 2017.
Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-300, 1995.
Alfio Borz and Giuseppe Borz, Algebraic multigrid methods for solving generalized eigenvalue problems. International journal for numerical methods in engineering, 65(8): 186--11966,,1, 2006.
Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas.
Distill, 1(3) 2019.
Stephen Casper, Xavier Boix, Vanessa D' Amario, Ling Guo, Kasper Vinken, and Gabriel Kreiman. Frivolous units: Wider networks are not really that wide. arXiv preprint 19X.:::91920478. 2020.
François Chollet et al. Keras. htttp:://errs. 2015.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuher. Are neural nets modular? inspect- ing their functionality through differentiable weight masks. 2020.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. arXiv preprint 1eXivv/909.0S0 2019.
Richard Fairley. Tutorial: Static analysis and dynamic testing of computer software. Computer, 11(4):14-23, 1978.
Michael Gazzaniga and Richard B Ivry. Cognitive Neuroscience: The Biology of the Mind: Fourth International Student Edition. WW Norton, 2013.
Michelle Girvan and Mark EJ Newman. Community structure in social and biological networks. Proceedings of the national academy of sciences, (9(1)):821-7726, 2002.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:1909, 10Xx:3999..1093,, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian joanaiaa of statis- fies, pp. 65-70, 1979.

---

Under review as a conference paper at ICLR 2021
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 4484444444 456, 2015.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning decompose neural computation. In Advances in Neural Information Processing Systems, pp. 2408- 2418, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ily Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Brenden Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266) 1332-13323 2015.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffnnr. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
Zachary C Lipton. The mythos of model interpretability. Queue, 16(3)311-57,)) 2018.
Shuying Liu and Weihong Deng. Very deep convolutional neural network based image classification using small training sample size. In 3rd IAPR Asian Conference Pattern Recognition, ACPR 2015, Kuala Lumpur, Malaysia, November 3-6, 2015, pp. 730-734. IEEE, 2015. doi: 10.1 109/ JCPR.2015.7486599 /tLpttts::///ooo1 http:///oooooooo.///// 10 ACP.. 2015. 201...11111011111 7486599
Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. On the capability of neural networks gen- eralize to unseen category-poss combinations. arXiv preprint arXiv:2007.0800 080022 2020.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. arXiv:2006. /4032, 2020.
arXiv preprint
Mark EJ Newman and Michelle Girvan. Finding and evaluating community structure in networks. Physical review E, 69(2):026113, 2004.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11)x7, 2017.
Stefano Panzeri, Christopher D Harvey, Eugenio Piasini, Peter Latham, and Tommaso Fellin. Cracking the neural code for sensory perception by combining statistics, intervention, and behay- ior. Neuron, 93(3):491-507, 2017.
Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla, and Bernhard Schölkopf. Learning independent causal mechanisms. In International Conference on Machine Learning, pp. 4036- 4044. PMLR. 2018.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-leam:: Machine learning python. the Journal of machine Learning research, 12:2825-2830. 2011.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888-90) 2000.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
10

---

Under review as a conference paper at ICLR 2021
Alberto Testolin, Michele Piccolini, and Samir Suweis. Deep learning systems complex networks. Journal of Complex Networks, 8(1):en2018, 2020.
Ulrike von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17(4):395-416 2007.
Chihiro Watanabe. Interpreting layered neural networks via hierarchical modular representation. In International Conference on Neural Information Processing, pp. 376-388. Springer, 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint 1rXiv:1708..774,, 2017
Jiaxuan You, Jure Leskovec, Kaiming He, and Saining Xie. Graph structure of neural networks.
arXiv preprint arXiv22007.06559 2020.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the importance of individual units in CNNs via ablation. arXiv preprint (rXiv:180600289) 2018.
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710001878, 2017.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 SPECTRAL CLUSTERING ALGORITHIM
The spectral clustering algorithm on the graph = (V, produces a partition of its vertices, in which there are stronger connections within sets of vertices than between them (Shi & Malik] |alik]20000) does so by approximately minimizing the n-cut (normalized cut) of a partition. For disjoint, non- empty sets XI. where U V, it is defined by (von Luxburg 2007 as:
1
X(,, vol(X;)
Xv)
2
where A = (wij)).. is the adjacency matrix of the graph G; for two sets of vertices X, S V, we define W(X, Y) = Em,ex E Wiy; the degree of vertex or V is d, = Σド=1 Whys and the volume of a subset X C V is vol(X) == ->iex di.
Algorithm 1: Normalized spectral clustering according to (Shi & Malik| 2000), implemented in scikūt-learn (Pedregosa et al. 201 TD, description taken from von Luxburg (2007).
Input- : Weighttd adjacency matrix W RRx* number of clusters construct
Compute the unnormalized Laplacian L. Compute the first generalized eigenvectors U,,... Ug of the generalized eigenproblem
Lu = ADu.
Let U E x be the matrix containing the vectors u1, dg as columns.
For i = .., n, let y E I* be the vector corresponding to the 7th row of U. Cluster the points (y:)i=1....., in I2* with the k-means algorithm into clusters Cı,..., Cke
Output: Clusters A₁, Aq,... Ag with A, {ily E C.).
A.2 NETWORK TRAINING DETAILS
We use Tensorflow's 's implementation of the Keras API Abadi et al. (2015), ; Chollet et al. (2015). When training all networks, we use the Adam algorithm (Kingma & Ba 2014) with the standard Keras hyperparameters: learning rate 0.001, 31 = 0.9, 32 0.999, no amsgrad. The loss function was categorical cross-entropy.
Small MLPs (MNIST and MNNS):n-MNIST)): We train MI.ss with 4 hidden layers, each of width 256, for 20 epochs of Adam (Kingma & Ba 2014) with batch size 128. We then prune a poly- nomial decay schedule (Zhu & Gupta 2017) to 90% weight-sparsity for an additional 20 epochs after initial training. Initial and final sparsities were chosen due to their use in the TensorFlow Model Optimization Tutorial In cases where we use dropout (for correlation visualization experiments in- cluding halves-diff tasks), we apply it after each fully fully-connected layer with a rate of 0.5. All MLPs achieved testing accuracy on the MNIST and Fashion-MNIST datasets of at least 97% and 86% respectively except for the ones trained on the Halves-diff datasets which all achieved an accuracy of at least 92% and 71% respectively.
Small CNNS (MNIST and Fashion-MNISST:: These networks had 3 convolutional layers with 64 3 x 3 channels each with the second and third hidden layers being followed by max pooling with a 2 by 2 window. There was a final fully-connected hidden layer with 128 neurons. We train them with batch size of 64 for 10 epochs with no dropout pruning. All small CNNs achieved testing accuracy on the MNIST and Fashion- MNNIST datasets of at least 99% and 89% respectively except for the ones trained on the Halves-diff datasets which all achieved accuracy of at least 89% and 67% respectively.
Mid-sized VGG CNNs (CIFAR-100 10): We implement a version of VGG-16 described by (Simonyan & Zisserman 2014);
Liu & Deng 2015). We train these with Adam, and L2 regularization with
12

---

Under review as a conference paper at ICLR 2021
coefficient of 5 X 10 50-5mm for 200 epochs with batch size of 128. Training done with data augmentation which consisted of random rotations between and 15 degrees, random shifts both vertically and horizontally of up to 10% of the side length, and random horizontal flipping. In cases where we use dropout, we use a per-layer dropout rate as specified in Liu & Deng (0155) All of these networks achieved testing accuracies of at least 87%.
Large CNNs (ImageNet): We experimented with VGG-16 and 19 (Simonyan & Zisserman 2014) and ResNet-18, and 50 (He et al. 2016) networks. Weights were obtained from the Python image-claaeee fiers package, version 1.0.0.
A.3 CORRELATION BASED VISUALIZATION
Algorithm 2: Sign Alignment Algorithm (Similar to Watanabe (2019)
Result: Set of sign-aligned neuron visualizations.
Input Neuron visualizations Vin for iter in num iters do for in V do
Calculate sum of cosines, c = -Ejfi (o
c then
1/ 2/
end
end end
Algorithm 2 gives the sign alignment algorithm we use which is based on similar one from Watanabe 2019).
Figure 3 shows examples from the 'halves' and 'stack' datasets which we use for MLPs and CNNs respectively to test whether a parallelizable task can cause network develop clusters that cohere with one portion of the inputs or another. Details of the halves dataset experiments are detailed in Section 3 Analogous experiments for CNNs were done but with "stack" datasets. For CNNs with max pooling, object detection is insensitive spatial location, stack-diff datasets in an analogous way using channels instead of image-halves
design stack-same and
Visualizations for sub-clusters in the halves datasets are provided in section 3 However, here in figure 4 are visualization results for Small CNNs for the samc--f datasets. Unlike for the small MLPs, these visualizations do show obvious coherence among clusters, which part of our motivation for the subsequent input construction experiments.
For constructing all correlation- -aaeed visualizations, fined as the linear (Pearson) correlation of ranks. This measures how one series values of values can be expressed correlation because of the nonlinear nature of deep networks.
the Spearman correlation which is de-
monotonically increasing function of another. We used this rather than linear
Networks can Compute Modular Sums: A network an do this for M values by using an interme- diate layer of 2 neurons, each of which serve as detector of one of the possible combinations of inputs. Consider ReLU MLP with 2M inputs, single hidden layer with M neurons, and then M outputs. Suppose that it is given the task of mapping datapoints in which the input nodes numbered and M + are activated with value to an output in which the mod (i + M)th node is active with value 1. It could do so if each hidden neuron with ReLU activation detected one of the 2 possible input combinations via a bias of -1 and two weights of the input nodes in the combination is detects. A single weight from each hidden neuron to its corresponding output point would allow the network to compute the modular sum. In our networks, have M 10 classes, and all MLPs and CNNs have dense layer with > 102 neurons preceding the output layer. Thus, they are capable of computing a modular sum in the halves and stack-diff tasks give to them.
connecting it to each
13

---

Under review as a conference paper at ICLR 2021
3 33153333633313333333333
33
3
3
3
3
3
3
3
3
3
766 77
6
9
5
4
Figure 3: Examples images from halves/stack datasets: (Left) Samples from 'halves' datasets, and (Right) samples from 'stack' datasets, all of class 3. Each row has 10 images from the respective dataset. The first row is MNIST halves/stack-same second is MNIST halves/stack-diff third is Fashion halves/stack-sane and fourth is Fashion halves/stack-diff.
I
D
I
Figure 4: Sub-cluster visualizations for small CNNs trained on stack-same/diff' MNIST: (Left)
stack-same MNIST, (Right) stack-diff MNIST. The top rows give visualizations for true sub-clusters and the bottom four for random ones. Columns are ordered left to right according to layer order.
A.4 HYPOTHESIS TESTING
Here, we provide details for the hypothesis testing methods used for input construction and lesion experiments in Section 3 In each of these experiments, for all sub-clusters in a network, obtain quantities for the true sub-clusters and random sub-clusters. We compare these values to get a value in the form of a percentile for each true-cluster comparing it to the random ones. We then obtain a single combined p value for a network overall using two methods. Both methods involve constructing a test statistic which has chi squared distribution under the null hypothesis that true- sub-clusters have the same properties as random ones.
Fisher Method: This measures how low p values for the true sub-clusters are overall across network. In our case, because we use 9 random sub-clusters, the values for sub-clusters take values in 10.1, 0.0........ To obtain granular approximation of the uniform distribution under the null, we subtract 0.05 from them to center their distribution around 0.5 so that they give granular approximation to the continuous Uniform(0, 1) distribution under the null hypothesis that visualizations for true sub-clusters are good random. Then we obtain the Fisher method test statistic
2 -2 log Pc
Which for n and p values has a chi squared distribution on 2n degrees of freedom under the null. We then conduct right-sided test with respect this distribution. The fact that approximation of the uniform distribution makes this test conservative because when -2 times the sum of the logs of the p values is taken during the calculation of the test-statistic, the smallest p values will pull the test statistic toward the heavy tail of the Chi Squared distribution while the largest ones will pull it toward zero.
a granular
This measures how nonuniform the distribution of values were for sub-clusters were across network. The p values fall into discrete bins, so standard Chi Squared categorical test can be used to test to see whether the assortment across the sub-clusters for a network is consistent with randomness or not. The test statistic
Chi Squared Categorical Method:
(x)
m4)
14

---

Under review as a conference paper at ICLR 2021
Here, this is observations of each value while 14 gives the expected count under the null. This test statistic will take Chi Squared distribution a right-sided test with respect to this distribution.
discrete values which results take, and each 2, gives the count of
degrees of freedom under the null hypothesis. We conduct
A.5 COHERENCE IN UNTRAINED NETWORKS
In most cases, find that trained networks, exhibit significant levels of importance and/or coher- ence. However, in order to get a sense of how much importance and coherence result from the training process, it is also natural exhibit these. Here, we present the results for experiments with feature visualization as done in table Th. We do not this for lesion tests though because in expectation, any untrained network will have accuracy at the random guess baseline whether intact or lesioned. Table 2eeewsswww these results for untrained CIFAR-IO scale VGGs. Here, the p values for input coherence are not indicative of any sort of interesting phenomenon which contrasts with the corresponding input coherence values from table which are very low. These suggest that the training process promotes input coherence in these networks. For output coherence, the values here are lower than the regularized VGGs but higher than the unregularized VGGs from table
ask to what extent untrained, randomly-inin -iittliiiie networks
Visualization Score: Input Coherence
Softmax Entropy: Output Coherence
Network
Fisher Combined p Chi Squared p
Fisher Combined p Chi Squared p
VGG (Untrained)
0.519
0.358
0.002
0.006
Table 2:
Results of Lesion tests for randomly initialized mid-sized VGG networks: As in Table each cell gives a median for 5 independent networks, and values less than 0.05 are bolded.
A.6 LESION TESTS WITH ALTERNATE CHOICES OF l:
In all quantitative experiments in the main paper, we present results for k 12 clusters. However, to test the robustness of result to the choice of k, we present here in table 3 replicates of table I with 8 (50% more) and 18 (50% fewer) clusters. Overall, results are very similar with no apparent systematic differences. In table Bh. there are only 3 values which are different from table T by whether they are below the threshold of 0.05, and similarly, there is only such value 3p.
8=8
Accuracy Change: Importance
Classwise Range: Output Coherence
Network
Fisher Combined
Chi Squared
Fisher Combined
Chi Squared
MLP MNIST
1.59
10
1.05
10
0.304
0.098
MLP Fashion
1.88
10 10-5
8.85 x 10 10
0.586
0.002
CNN MNIST
0.008
0.028
0.570
0.587
CNN Fashion
0.005
0.005
0.700
0.582
VGG CIFAR (Unreg)
1.24
10
1.59 x 10
2.17 x 10
6.07
10 10-32 32
VGG CIFAR
3.99
10
7.08 x 10 29
7.97 x 10 9
1.80
10 38
ResNet-13 18 ImageNet
0.130
0.001
0.086
0.021
VGG- 16 ImageNet
3.36 x 10
6.08 x 10
0.998
0.006
Accuracy Change:
Importance
Classwise Range: Output Coherence
Network
Fisher Combined p
Chi Squared p
Fisher Combined p
Chi Squared p
MLP MNIST
1.23 10
1.21 x 10
0.370
0.027
MLP Fashion
5.67 x 10
6.31 x 10
0.115
0.019
CNN MNIST
0.042
0.088
0.503
0.569
CNN Fashion
0.035
0.328
0.372
0.866
VGG CIFAR (Unreg)
2.58 x 10 T3
1.42 x 10
1.84
10
1.49
10 -4
VGG CIFAR
4.07 10 10
1.63 x 10-17
4.69 x 10
2.72
10 33
ResNet- 18 ImageNet
4.08 10 10-4
7.50 10
0.411
0.001
VGG-16 ImageNet
2.94 x 10
1.41 x 10 66
1.000
3.34 10
Table 3: Combined values for lesion experiments with & 8 (top) and A: = 18 (bottom): Table
replicated with alternate choices of the number of clustering centers for the same networks.
15

---

Under review as a conference paper at ICLR 2021
Visualization Score: Input Coherence
Softmax Entropy: Output Coherence
Network
Fisher Combined
Chi Squared
Fisher Combined
Chi Squared
MLP MNIST
CNN MNIST
VGG CIFAR (Unreg)
VGG CIFAR
VGG-16 ImageNet
VGG- 19 ImageNet
ResNet50 ImageNet
Accuracy Change:
Importance
Classwise Range: Output Coherence
Network
Fisher Combined p
Chi Squared p
Fisher Combined p
Chi Squared y
MLP MNIST
MLP Fashion
CNN MNIST
CNN Fashion
VGG CIFAR (Unreg)
VGG CIFAR
ResNct-18 18 ImageNct
VGG-16 ImageNet
Table 4: Significance of results in table [ using the Benjamini- Hocchbiei- procedure to ensure that the false discovery rate per group-that is, the expectation under the data-generating distribution of the proportion results declared significant that came from the null distribution below 0.05, where all the Fisher tests are grouped together and all the chi squared tests are separately grouped together.
A.7 MULTIPLE COMPARISON ADJUSTMENT
In table I we report various p values that summarize the degree to which statistics of sub-clusters vary from those of random groups of neurons within network. For each network, can the p value to test whether the sub-cluster statistics are drawn from the same distribution of the statistics of random groups of neurons. However, when testing multiple networks, one might want to ensure that the experiment and significance-testing procedure are unlikely to generate false positives. In order do this, more complicated procedure to decide significance must be used.
procedure (Benjamini & Hochberg 1995) controls the false discovery rate: that is, the expected proportion of rejections of the null hypothesis that are false positives, where the expectation is taken under the data-generaie gentragenn distribution. It relies on all experiments being independent, and therefore it was run separately on the Fisher combined p values and on the Chi squared values. Results that are declared significant under this procedure when the maximum acceptable false discovery rate 1/20 is shown in table
The Senjamini-Hochheee
The coll-Boonerroni method (Holm, 1979) controls the family-wise error rate: that is, the prob- ability under the data- tata-gene distribution that any null hypotheses are falsely rejected. Results that are declared significant by the Holm-Bonfeen Bonfeerooi method run on the whole of table capping the family-wise error rate at 1/20 are shown in table 5
A.8 INPUT CONSTRUCTION
All visualizations were created using the Lucic package. The optimization objective for visualizing sub-clusters was the mean Rest- activation for all neurons inside the cluster (it was a mean of means for convolutional feature maps). For small MLPs, small CNNs, and mid-sized CNNs, generated images using random jittering and scaling, and for ImageNet models, we used Lucid's default transformations which consist of padding, jittering, rotation, and scaling with default hy- perparameters. For all networks, we used the standard pixel-based parameterization of the image and no regularization on the Adam optimizer. For visualizations in small MLPs and CNNs, we used versions of these networks trained on 3-channel versions of their datasets in which the same
16

---

Under review as a conference paper at ICLR 2021
Visualization Score: Input Coherence
Softmax Entropy: Output Coherence
Network
Fisher Combined
Chi Squared
Fisher Combined
Chi Squared
MLP MNIST
CNN MNIST
VGG CIFAR (Unreg)
VGG CIFAR
VGG-16 ImageNet
VGG- 19 ImageNet
ResNet50 ImageNet
Accuracy Change:
Importance
Classwise Range: Output Coherence
Network
Fisher Combined p
Chi Squared p
Fisher Combined p
Chi Squared y
MLP MNIST
MLP Fashion
CNN MNIST
CNN Fashion
VGG CIFAR (Unreg)
VGG CIFAR
ResNet-18 18 ImageNct
VGG-16 ImageNet
Table 5: Significance of results in table I using the Holm-Bonferroni method ensure that the family-wise error rate-that is, the probability under the data-generating distribution that any result falsely declared significant- is less than 1/20.
inputs were stacked thrice because Lucid requires networks to have 3-channel inputs. However, we show grayscaled versions of these in figure 5 Refer the main text (section 3.1.2p for quantitative analysis of the optimization objective values.
(a)
(b)
(0)
(
Figure 5: Example feature visualizations for true and random sub-clusters: In the left column are shown true sub-cluster visualizations, and the right column are visualizations of sub-clusters of random neurons of the same size in the same layer. (a) MLP, MNIST; (b) CNN. MNIST: (c) CNN-VGG, CIFAR-10; (d) VGG-16, ImageNet; (e) VGG-19, ImageNet: (f) ResNet-50, ImageNet.
Importantly these feature visualizations, while designed to maximally activate sub-cluster, will not necessarily highly activate all of the neurons inside of it. In order to get sense of how much
17

---

Under review as a conference paper at ICLR 2021
variance there is among these activations, analyze two properties of the distribution of sub-cluster activations when a visualization is passed through the network.
First, perform the same tests as for score and entropy in table [ but with the variance among neuron activations. The "Activational Variance" columns in table show these p values. Here, low Fisher p values reflect low variance for unit activations in true sub-cluster compared the variance for unit activations in random sub-clusters. In table there is significant evidence that some of the networks the CIFAR-IO and ImageNet scale have lower variance among the activations of true sub-clusters than random ones when a sub-cluster's visualization is passed through the network. This suggests that in the networks for which this is the case, neurons in true sub-clusters are more consistently activated by the same visualizations than those in random sub-clusters.
Second, we directly analyze the empirical coefficients of variation (CoVs) for the distributions of true sub-clusters. The CoV is the standard deviation of a distribution divided by its mean: 8 ji. As such, high CoVV means that the distribution has high standard deviation relative the mean. For each sub-cluster of network, we take the CoV of the distribution of post-ReLU activations. Then, for each network, we take the distribution of CoVs of its subclusters, and find the first quartile, median, and third quartile. For each training condition, we train five networks, rank the five by their median CoV, take the median network under this ranking, and report that network's ' CoV quartiles in the final three columns of table 6 We find that in some cases the CoVs relatively low, including the ImageNet models which indicates relatively consistent activations. In other networks though, many of the CoVs are above 1.
Activational Variance
Activational Coefs of Variation
Network
Fisher Combined p
Chi Squared p
Quartile
Median
Quartile 3
MLP MNIST
0.124
0.295
1.399
I.539
1.877
CNN MNIST
0.485
0.328
0.614
0.691
0.787
VGG CIFAR (Unreg)
0.061
4.39 x 10 4
0.929
1.214
3.690
VGG CIFAR
0.023
0.003
0.866
1.153
1.384
VGG-16 ImageNet
0.091
0.327
0.424
0.528
0.748
VGG-19 ImageNet
7.44 x 10 4
1.13 x 10
0.367
0.453
0,561
ResNet50 ImageNet
0.016
0.002
0.650
0.753
0.810
Table 6: Combined p values testing for differences in variance and coefficient of variation quartiles in feature visualization experiments: Compare to table Da. In the activational variance columns, p values are shown comparing true sub-cluster activations to random sub-cluster activa- tions. As in table [T p values for non-ImageNet networks are medians among 5 trials, and p values less than 0.05 are bolded. In the coefficients variance columns, quartiles for the CoVs for true sub-clusters shown.
A.9 LESION TESTS
presents the lesion test experiments. Example accuracy-change profiles for an MLP and small CNN in the Fashion datasets are shown here in figure [6] Table 7 and Table & show data on the importance of sub-clusters in the single lesion experiments, and is plotted in figure 2 "Acc. diff." means the difference in accuracy between the actual network and the network with that sub- cluster lesioned, while "Acc. diff. dist." shows the mean and standard deviation of the distribution of accuracy differentials between the actual network and one with a random set of neurons lesioned.
Section 3.2
The "Proportion" column denotes the proportion of the layer's neurons that the sub-cluster repre- sents. 'Important' means that the sub-cluster makes up at least 5% of the layer, that the drop in accuracy greater than one percentage point, and that was more important than all of 20 random sub-clusters it was compared against. "Sig-but-ne -oft-dff means that the drop in accuracy is signifi- cant but less than percentage point, 'Diff-but-not-sig means that the lesioning damage was more than percentage point but not significant, 'Prunable' means that the drop in the accuracy is smaller than all random shuffles and smaller than 1 percentage point, 'Complete' means that the sub-cluster contains the whole layer, 'Small' means that the sub-cluster consists of less than 5% of the layer, and 'Other' means that the drop in accuracy is not statistically significant and less than I percentage point.
18

---

Under review as a conference paper at ICLR 2021
One detail not included in the main paper is that for the sake of computational efficiency, two mea- sures were used for lesion experiments in the ImageNet models we used (ResNet- 18 and VGG-16). First, we used downsampled version of the ImageNet2012 dataset (Krizhevsky et al. 2012) with 10,000 instead of 50,000 images. Second, we omitted sub-clusters with fewer than neurons or more than 90% of the neurons in the layer (this is different from the thresholds of 3 units and 80% we used for input construction experiments).
MUS-IAHHON koud
1.
Layer 1 Clusser d 80.32%
Layer Ouster 3. 1667%
Layer Clatte O 297%
Layer 2 Guster 2: 6.79%
Layer 2 Custer 2 W6.a0h
CNN-FASHION kod
2.
Lover Chasee 99%
Custer 12.30%%
Dutter
Lover Chater
Lover Chater 40.82%
C6
38
Layer 2 Chtter 118.8%
Layer 2 Custer 2: 28315
Layw Outerrs h 25 19.5
Layer Cluber 25 6P%
Laper Ouster 1: 12 1m.9%
38
Loyer 3 20.5%
Layer 2 C 25,95
4.0
Figure 6: Example accuracy change profiles for lesion tests: (Top) Fashion-MNIST MLP and (Bottom) Fashion-MNIST Small CNN. Each subplot corresponds to sub-cluster in the network. For the sake of visualization here, 4 clusters were used instead of the normal 12 which was used in all quantitative experiments. The accuracies for each class plus the overall accuracies in the rightmost points are plotted. Green lines indicate the performance of the unperturbed network, red lines give results when lesioning true sub-clusters, and grey lines show results for lesioning 20 random sub- clusters.
A. EXPLORING THE "COMPOSABILITY" PROXY WITH DOUBLE LESION TEST
Given the lesion test presented in the main text, we know which sub-clusters are important, and it would be ideal to understand how the important sub-clusters depend on each other. To do this, we conduct experiments where we lesion two different important sub-clusters, which we'll call X and Y, in different layers. First, we measure the loss in accuracy when both are lesioned, which we'll ll call e(X UY). We then compare {(X U Y') to the loss in accuracy {(X U Y') if we take a random subset Y' of neurons of size |Y| from the same layer as Y, and check if E(X UY) is larger than 50 random samples of ((X U Y'). This tests if the damage from lesioning Y is statistically significant given how many neurons are contained in Y, and given that we are already lesioning X. We also calculate 8(Y,X) that X has been lesioned. If f(X U Y) is statistically significantly different to the distribution of (XUY"), and if 5(Y, X) is larger than one percentage point, we say that sub-cluster Y is important
f(X U Y)- {(X), which is the additional damage from lesioning Y given
19

---

Under review as a conference paper at ICLR 2021
Table 7: Table of sub-clusters of an MLP network trained on Fashion-MINIST using pruning and dropout. The network is partitioned into 8 clusters.
Layer
Label
Acc. diff.
p value
Proportion
Type
Acc. diff. dist.
0
0
0.476
0.005
small
-0.00009 = 0.00021
1
1
-0.2484
0.048
0.404
important
-0.00751 = 0.0048
1
2
-0.0685
0.048
0.154
important
-0.00179 + 0.00222
-0.0016
0.429
0.111
other
-0.00124 + 0.00105
5
0
0.429
0.005
small
-0.00011 + 0.00035
1
6
0.1761
0.048
0.313
important
0.00453 1 0.00315
1
7
0
0.476
0.01
small
0.00006 1 0.00039
2
0
-0.114
0.048
0.473
important
0.00362 1 0.00296
2
0.476
0.008
small
0.00002 +:: 0.00025
2
2
-0.0233
0.048
0.027
small
0.000005 + 0.00038
2
3
-0,006
0.048
0.113
sig-but-
ge-bbttnnot-diff not-diff
0.00045 = 0.00114
2
6
-0.0014
0.476
0.188
other
0.00126 = 0.00147
2
0.0014
0.191
prunable
0.00143 + 0.00131
3
-0.043
0.048
0.617
important
0.0005 0.00331
3
4
0.0005
0.905
0.023
small
0.00002 = 0.0003
3
7
-001129
0.048
0.359
important
0.00202 + 0.00186
4
O
-0.0379
0.143
0,563
diff-but-
0.02009 = 0.02002
4
0.0002
0.004
small
0.00005 0.00017
4
-0.0017
0.095
0.113
other
0.00089 + 0.00086
4
7
0.2048
0.048
0.32
important
00.00455 = 0.00138
Table 8: Table of sub-clusters of an VGG network trained on CIFAR-10 using dropout. The network is partitioned into 8 clusters.
Layer
Label
Acc. diff.
p value
Proportion
Type
Acc. diff. dist.
I
0.8043
0.048
I
complete
-0.80430
0
0.857
0.016
small
0.00532 0.00708
2
0.8043
0.143
0.984
diff-but-not-sig 0.80470 + 0.00166
3
0.8043
0.048
1
complete
-0.80430
4
1
0.8043
0.048
1
complete
-0.80430
5
0.8043
0.048
complete
-0.80430
0.8043
0.048
complete
-0.80430
7
0.8043
0.048
0.617
important
0. 1.2226 1: 0.03685
7
5
0
1
0.383
prunable
0.03632 +: 0.01111
8
0
0.586
prunable
0.02803 = 0.01497
O
0.238
0.002
small
-0.00005
0.00020
5
0.8043
0.048
0.412
important
00.07552 = 0.00451
3
0
0.822
prunable
0.21835 + 0.12/62
9
5
0.8043
0.048
0.178
important
0.00200 + 0.001-16
10
0
0.924
prunable
00.22127 + 0.15113
1T
008043
0.048
0.076
important
0..00433 O. 0.00289
1I
0
0.143
0.002
small
-.00001 + 0.00004
11
4
0
1
0.662
prunable
0.05353 + 0.03084
11
0.8043
0.048
0.336
important
0..0387 + 0.00240
12
0.8043
0.048
complete -0.80430
13
-0.8043
0.048
complete
-0.80430
conditioned on sub-cluster X. Similarly, we test if X important conditioned on Y by comparing ((XUY) to the distribution of ((X'UY), and by determining the size of 5(X, Y). Table shows the values and importances of different pairs of sub-clusters for an MLP trained on Fashion- Fashion-MNIST with pruning and dropout, when the number of cluster is set 8 for visualization.
By examining the importances sub-clusters conditioned on each other, we can attempt to construct dependency graph of sub-clusters by determining which sub-clusters send information to which
20

---

Under review as a conference paper at ICLR 2021
Figure 7: Four cases of dependency between sub-clusters. These cases characterize how informa- tion can flow from a sub-cluster X to a sub-cluster Y, where X is in an earlier layer than Y. U and V are two other sub-clusters, which reside in the same layer as and respectively. All of these sub-clusters are individually "important". Given the importance configuration of (X YY, (I),, we conjecture what is the relationship terms of information flow between the sub-clusters.
x
y
x
y
u
u
(a) x|y: x Y|X: x
(h) X|Y:
X|X:
X
u
U
u
(c) x x YIX: v
(d) X|Y:
Y|X:
others. Consider a pair of sub-clusters (X, Y) where X is in an earlier layer than Y, and where both are individually important (refer to figure for an elaborated visual illustration).
If X is not important conditioned on Y, and Y is not important conditioned on X, we reason that all of the information from X is sent to Y (since otherwise lesioning X would damage accuracy even conditioned on Y being lesioned), and that the only information that Y receives is sent via X (since otherwise lesioning Y would damage accuracy even conditioned on X being lesioned).
If X is not important conditioned on Y but Y is important conditioned on X, then reason that X sends information to Y and also to other sub-clusters.
If is not important conditioned on but X is important conditioned on Y, we reason that Y receives information from X and other sub-clusters.
We can draw no conclusion if both X and Y are important conditioned on the other.
These assumptions, together with data shown in figure let us draw some edges in dependency graph of sub-clusters, which is shown in figure Note that sub-clusters of cluster 0 seem send information to each other, which is what we would expect if modules were internally connected. The same holds for the sub-cluster of cluster
21

---

Under review as a conference paper at ICLR 2021
Table 9: Dependency information of pairs of "important" sub-clusters of MLP trained Fashion-MNIST with dropout. X and Y represent sub-clusters, X being the one earlier in the network. They numbered by first their layer and then their cluster number. The network is partitioned into 8 cluster.
X
Y
Y
YX
6(X. Y)
6(Y, X)
2-0
-0.261
-0,026
1-1
3-0
-0.237
-0.023
1-1
3-7
V
X
-0.249
-0.014
1-1
4-7
-0.244
-0.201
1-2
2-0
-0.095
-0.04
3-0
V
-0.092
-0.058
1-2
3-7
-0.054
+0.001
/
1-2
4-7
-0,088
-0.244
1-6
2-0
-0.157
0.006
3-0
-0.194
-0,052
3-7
-0.180
-0.017
1-6
4-7
-0.153
-0.182
2-0
3-0
-0.125
-0.146
2-0
3-7
-0.125
-0.124
4-7
-0.101
-0,292
3-0
47
V
-0.502
-0.673
3-7
4-7
X
V
-0.010
-02202
1-1
1-2
1-6
2-0
3-7
3-0
4-7
Figure 8: A plot of the edges of the sub-cluster dependency graph for an MLP trained on Fashion-MNIST and partitioned with 8 clusters. We derive this from data shown in figure Sub-clusters are identified first by their layer number, then by their module number.
22

---

Under review as a conference paper at ICLR 2021
(M0)
23
22
1.5
13
25
20
44 (TTH)
0.17
0.62
0:12
0.17
4.55
0.17
0.72
0.13
0.19
0.05
1.5
0.03
5.3
5.1
19
0.38
0.37
4-3
3.02
-0.01
0..02
-0.01
4..02
-0.01
4.00
-0.01
-0.03
0.00
-0.03
-..0
-9
-0
-0
-0.02
-.001
-0
4.2
3.8
3.8
3.7
5.1
23
-0
0..14
(00)
2- (PRJ)
3.14
-0.14
0.47
2-6 (
0.14
0.44
9.84
0:09
0.14
10.
0.13
23
0.003
0.09
0.01
0:34
6.1.
0.92
0.08
1.4
0.24
0.15
1.5
2-3 15KG)
0.6
0.21
0.51
0.09
0.6
0.62
2.6
0.6
0.52
0.6
0.47
0.99
0.62
0.00
0.95
006.1
0.48
-1.7
0.23
0.21
0.13
0.23
0.23
0:28
0.23
1.2
0.17
0.15
0.25
0.54
0.5
0.24
0.04
0.12
0.24
0.25
0.00
-20
1-1 6MA
1-6 (MP)
1-5 (MMA
0
-0
3-3 (HTH)
0.35
1.4
10:33
0.26
0:38
8.17
0.15
0:33
0.31
8.30
00:45
0:33
0.07
01.37
0.15
0.18
0.14
08.17
0.11
0.02
(RP)
15
6.9
9.1
9.5
6.8
6.8
6.5
9.2
5.4
7.2
6.9
7.3+
8.0
25
3-3 (099)
33 (5MA)
1:61099
3-2 (000)
2-0 1099
2.2 13-14
3.99
3-4 (000)
3-7 1099
4.3 1964
6-4 (070))))
4- (997)
3-2
246
aecond
Figure Dependency information for all pairs of sub-clusters for an MLP trained on Fashion- MNIST and partitioned with 8 clusters. Best viewed in color and zoomed in on screen. Sub- clusters are numbered by their layer and cluster. They are also labeled by their importance: IMP stands for "important". SMA stands for "small"" SIG stands for bii--but-not-dif"" DIFF stands for tdiff-but-not-siig." PRU stans for "prunable" and OTH stands for "other", terms which are defined at the beginning of Appendix A.9 Cells the diagonal are labeled by the accuracy damage in percents caused by zeroing out the corresponding sub-cluster in the single lesion experiments, and labeled by 100 ddfrstt second). Cells contain stars the damage caused by lesioning the first is statistically significant at the < 0.05 level conditional on the second being lesioned, and question marks if the damage caused by lesioning the first less than the damage caused by lesioning each of 50 random sets of neurons (i.c.,
have bolded text, while cells off the diagonal
1.00), conditional on the second being lesioned. Note that this plot includes sub-clusters in the same layer, where the meaning is questionable since there can be no direct dependency.
23


Reviews:
[{"review_id": "pgsBVKNObcP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the application of spectral clustering methods to assess modular organization in the emergent architecture of deep networks. In particular, sub-modules identified by spectral clustering are evaluated in terms of \"importance\" and \"coherence\", two metrics defined by the authors with the goal of capturing how crucial the neurons in the sub-module are to the classification accuracy, and how consistent their activation is across input and output patterns.\nOverall, the paper addresses important questions related to the way structural properties in a deep network might support the emergence of functional properties, which is a key issue given the relatively poor theoretical understanding we have about these self-organizing systems. The paper is comprehensible, though the general structure and the writing could be improved to improve readability. For example, in Section 3 it is not immediately clear how the importance and coherence metrics relate to the specific technique adopted for feature visualization, or to the lesioning method applied. The “Related Work” section should be moved at the beginning of the paper, and the contribution should be better framed in the context of other existing approaches.\nAlthough I appreciate the wide range of networks tested by the authors, I think that the way results are presented does not easily allow to establish whether we can identify a series of robust findings that are valid across all architectures / tasks, or whether specific cases entail peculiar findings. It would also be valuable to better assess the relationship between modularity and regularization techniques, such as dropout, L2 and pruning: I think that this is a very important point that should deserve further investigation, since it could give important insights about the role of regularizers in shaping the final network architecture.\nFinally, it would strengthen the paper if the proposed method is put in relation to other approaches that have proven effective to address this kind of question. This would greatly improve the robustness of the results, since the current baseline is basically constituted by a comparison with random sub-modules. For example, it would be nice to see if clustering coefficient and average path length (as defined in [1]) can provide useful information also for the analyses proposed by the authors. Note that in [1] the authors investigated deep networks with similar architectures (MLPs, CNNs, ResNets) trained on similar tasks (CIFAR-10, ImageNet).\n\nOther comments:\n- Pg. 3: the technique used to visualize a sub-cluster by creating an aggregate measure of the learned features can be discussed in relation to the method based on Earth-mover distance proposed by [2], where the authors also discuss other graph-based metrics that might be useful in the present setting.\n- Could the “intersection information” approach presented in [3] can be exploited also in the analyses of the sub-modules detected by the spectral clustering? Note that in [3] the authors also investigate “lesion tests” by means of interventional techniques, which would make that approach very interesting as a further benchmark.\n- What is the rationale for setting the number of clusters to 12? If this value is not theoretically motivated, further analyses should show that the results are robust to variations in this value.\n- When comparing “true sub-clusters” with “random sub-clusters”, a useful control analysis would be to create random sub-clusters by matching some connectivity property (e.g., same average strength, clustering coefficient and/or average path length).\n- Regarding the gradient-based method discussed in Olah et al. (2017), it would be useful to have some dispersion measure over the final optimization score, in order to better assess whether all neurons in the sub-cluster where in fact similarly activated by the optimized image.\n- It would be interesting to include as baseline some analysis on randomly connected networks, since it has been shown that subgraphs in large random networks can in fact support accurate task performance even without ever training the weight values [4].\n- Where the p-values corrected for multiple comparisons?\n- The authors consider “modularity  as an organizing principle to achieve mechanistic transparency”. Though I sympathize with this statement, I guess there are several cases where modular systems (or in general systems with localized representations) can develop complex emergent dynamics that still prevent interpretability.\n\nReferences\n[1]\tJ. You, J. Leskovec, K. He, and S. Xie, “Graph Structure of Neural Networks,” in International Conference on Machine Learning, 2020.\n[2]\tA. Testolin, M. Piccolini, and S. Suweis, “Deep learning systems as complex networks,” J. Complex Networks, vol. 0000, no. 1, pp. 1–21, Jun. 2019.\n[3]\tS. Panzeri, C. D. Harvey, E. Piasini, P. E. Latham, and T. Fellin, “Cracking the Neural Code for Sensory Perception by Combining Statistics, Intervention, and Behavior,” Neuron, vol. 93, no. 3, pp. 491–507, 2017.\n[4]\tV. Ramanujan, M. Wortsman, A. Kembhavi, A. Farhadi, and M. Rastegari, “What’s Hidden in a Randomly Weighted Neural Network?,” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11893-11902. 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IKY7SYg-6Cy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors identify putative clusters of units/neurons in deep networks using spectral clustering on a graph defined by synaptic weights. The authors then argue that these structurally defined clusters of neurons have similar *functional representations*. Finding interpretable relationships between weight matrices and functional modules is challenging, and the authors should be applauded for attempting to tackle this challenging problem that few research groups are devoting energy to.\n\nDespite these positive notes, I have reservations about the presentation and results of the paper. My main concerns are:\n\n(1) The results are largely qualitative and anecdotal. In figure 1, for example, the authors show slightly higher contrast in their identified clusters than random clusters. The results are limited to black and white images (MNIST and fashion-MNIST), and not all examples look great. Only 2 figures are shown in the main paper, with a lot of other details shoved into the supplement. Thus, the writing and presentation could be improved to highlight the most exciting and surprising findings.\n\n(2) The results crucially rely on a second paper which was concurrently submitted and can't be reviewed because it is anonymized. The results shown in this paper are thin and qualitative (see point 1), so in my view these two paper should be combined into a single paper which overall might tell a more comprehensive and compelling story.\n\n(3) The paper does not generate testable predictions or practical insights that could be used by used by practitioners. The only takeaway point for me was that some neurons / units show correlated representations, which is arguably already known (e.g. Csordas et al 2020). How to exploit this modularity to develop human-interpretable explanations of network function remains unclear to me.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7LXScpW7tW6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript introduces an approach, based on importance and coherence, for evaluation whether a partitioning of a network exhibits modular characteristics. \nImportance refers to how crucial is a neuron , or set of neurons, to the performance of a network on a given task, e.g. classification.\nCoherence refers to how consistently the neuron(s) in question are related to specific features.\nExperiments are conducted by considering sets of neurons identified via a spectral clustering algorithm.\n\nThe manuscript proposes a method to verify to what extent a partitioning of a network follows modular characteristics. To a good extent the proposed method is grounded on proper theoretical foundations which is highly desirable. The only part where this cannot be fully verified is its dependence on the method from [Anonymous, 2021] which cannot be verified.\n\nMy main concerns with the manuscripts are the following:\n\n- When conducting spectral clustering, the number of clusters is set to 12. Is there a procedure to set this value in a principled manner? is there an indication on the effect of this parameter? the manuscript would benefit from analyzing the effect of this parameter in the observation made on the reported experiments?\n\n- Visualizations discussed in Sec. 3.1.1 (Fig.1) are quite subjective. While in some cases some patterns are indeed visible, in other cases it is hard to make sense of what is being presented. Is it possible to evaluate the produced visualizations in a more objective manner?\nIn recent years, several methods ( Bau et al, 2016, Oramas et al. ICLR'19, Yang and Kim, arXiv:1907.09701 ) for quantitative evaluation of methods for visual interpretation and explanation have been processed. Perhaps one of these could be adopted in the manuscript with the goal of objectively evaluating the visualizations/explanations presented in Fig. 1.\n\n- In some cases design decision are made that seem to favor observations expected in some experiments. For instance, in order to favor clusterability small MLPs are pruned, to improve visualization MLPs are trained with dropout; and other factors relevant to the proposed method. Therefore my question by ensuring that some of these properties, e.g. cleaner cluster, clearer visualizations, don't you favor the measurement capabilities of the proposed importance/coherence metrics?\n\n- When analyzing the \"importance\" metric on the lesion tests (Sec. 3.2) there are new conditions that are applied to the clusters being considered in the analysis, e.g. the size of the cluster, minimum effect of the cluster on accuracy, etc. Keeping this present, my questions are: i) Were these conditions also applied when analyzing \"coherence\", and ii) why these type of condition were not applied in the experiments of Sec. 3.1? Ideally, some level of consistency is expected among the experiments. Otherwise it is hard to assess properly the origin of observations made on the results of the experiments.\n\n- At the end of Sec. 4, it is stated that the conducted experiments, combining spectral clustering with feature visualization,\nhighlight the usefulness of combining multiple interpretability methods in order to build an improved set of tools for rigorously understanding systems. However, from the observations made on the experiments I do not see the added value that the proposed method could bring to interpretability/explainability of the analyzed models (networks).\n\n- Very related, one paragraph later, it is stated that having modular networks is useful both for interpretability and for building better models. However, from the content of the manuscript it is not clear how having a modular network/representation does contribute with the two listed aspects.\n\n- Significant parts of the manuscript are delegated to the supplementary material. In addition, the third part of the proposed method, i.e. intrinsic partition evaluation, is part of another manuscript [Anonymous,2021] that does not seem to be published. For these reasons, to a good extent, the manuscript is not self-contained. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper present an approach for defending for, and search for, 'modularity' in neural networks, as a step to better interpretations of their functional structure. This is an interesting, and highly original approach, as recognised by the reviewers. However,  there was also some discussion about what exactly can be learned from the derived clusters/modules, and if and how they will lead to a better understanding of neural networks, or provide concrete ways of improving them.  While the authors addressed some issues during the review process, and provided additional results, the consensus (of all three reviewers) was finally that the paper did not reach the quality standards required by ICLR. I share this view-- the paper provides a refreshing perspective, but I still am not convinced that I see a clear, compelling 'use case' for their approach.  
Decision: Reject