Detecting Modularity in Deep Neural Networks
Authors: Shlomi Hod
Stephen Casper
Daniel Filan
Cody Wild
Andrew Critch
Stuart Russell
Venue: ICLR 2022 Submitted (2022)
Keywords: Modularity
clustering
interpretability
feature visualization
lesions

Abstract: A neural network is modular to the extent that parts of its computational graph (i.e. structure) can be represented as performing some comprehensible subtask relevant to the overall task (i.e. functionality). Are modern deep neural networks modular? How can this be quantified? In this paper, we consider the problem of assessing the modularity exhibited by a partitioning of a network's neurons. We propose two proxies for this: importance, which reflects how crucial sets of neurons are to network performance; and coherence, which reflects how consistently their neurons associate with features of the inputs. To measure these proxies, we develop a set of statistical methods based on techniques conventionally used to interpret individual neurons. We apply the proxies to partitionings generated by spectrally clustering a graph representation of the network's neurons with edges determined either by network weights or correlations of activations. We show that these partitionings, even ones based only on weights (i.e. strictly from non-runtime analysis), reveal groups of neurons that are important and coherent. These results suggest that graph-based partitioning can reveal modularity and help us understand how deep neural networks function.

Under review as a conference paper at ICLR 2022
DETECTING MODULARITY IN DEEP NEURAL NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
A neural network is modular to the extent that parts its computational graph (i.c. structure) can be represented performing some comprehensible subtask relevant to the overall task (i.e. functionality). Are modern deep neural networks modular? How can this be quantified? In this paper, we consider the problem of assessing the modularity exhibited by partitioning of a network's neurons. We propose two proxies for this: importance, which reflects how crucial sets of neurons are network performance; and coherence, which reflects how consistently their neurons associate with features of the inputs. To measure these proxies, we develop set of statistical methods based on techniques conventionally used to interpret individual neurons. We apply the proxies to partitionings generated by spectrally clustering graph representation of the network's 's neurons with edges determined either by network weights or correlations of activations. We show that these partitionings, even ones based only weights (i.e. strictly from non-runtime analysis), reveal groups of neurons that are important and coherent. These results suggest that graph-basee based partitioning can reveal modularity and help us understand how deep neural networks function.
INTRODUCTION
Modularity common property of complex systems, both natural and artificial (Clune al.) 2013, Baldwin & Clark, 2000; Booch et al. 2007). It carries advantages including intelligibility and adaptivity. In particular, modularity implies that the structure of system conveys information about its functionality. This leads us to the following definition: a neural network is modular to the extent that parts its computational graph (structure) can be represented as performing some comprchensible subtask relevant to the overall task (functionality). We call these parts modules.
There exists a body of research for developing more modular networks which either have distinet architectural building blocks (Alet et al.) 2018; Parascandolo et al.. 2018 Goyal et al.) 2019)
trained in a way that promotes modularity via regularization or parameter isolation (Kirsch et al.)
2018 De Lange et al. 2019 Filan et al., 2021). Yet in machine learning, it is more common encounter networks whose architecture and training are not guided by considerations of modularity. For example, computer vision, networks are generally trained end-to-end with all of the filters in one layer connected to all filters in the next. Do such networks nonetheless develop modularity? There some evidence for this. For example, by methodical manual investigation, Cammarata discovered modular subnetworks which perform cumaa--xpllinaable subtasks such as car detection via neurons which detect different car parts. More scalably detecting modularity in networks would help to extend our understanding of their learning dynamics and to expand our interpretability toolbox by suggesting an additional level of abstraction beyond single-neuron methods.
(2020)
In this paper we systematically analyze the extent to which networks which are not explicitly trained to modular exhibit nonetheless. First, this requires method for breaking down a network's computational graph into different proposals for modules. For this, we use spectral clustering a graph representation Second, this requires scalable method for approximating the degree to which partitioning exhibits modularity. We do this by applying interpretability tools to clusters neurons as a way of quantifying proxies for modularity.
the network, using an extension of the methods of (Filan et al. 2021).

---

Under review as a conference paper at ICLR 2022
7
6 6
0
/
2
9
4
7
7
6
6
7
6
4
9
8
3
F 4
D
2
<
7
2
/
0
5
/
9
6
7
/
0
D
An illustrative example of coherence. Randomly selected examples of class "3" from our "halves-MNISTT" dataset (top) and visualizations of neuron clusters in the first layer of networks trained to output the modular sum of digits in the images (bottom). We trained multilayer perceptrons on version of the MNIST dataset (LeCun et al. 1998) in which the images were two half-width digits side-by-side and the labels were their sum modulo 10. We clustered their first-layer neurons using approaches presented in Section 3.1] We then used a correlation- baasd method from Watanabe (2019) to create visualizations of the clusters. Almost all show selectivity to one half of the input. In Appendix A.4. we detail our approach and compare to visualizations of random sets of units.
Figure I:
Operationalization of modularity: Following our notion of modularity, to say that a subset is module, one must show that it performs some compreenmmmeee subtask relevant to the overall task. Scaling this, however, would be difficult, as it would require human in the loop. How could measuring modularity be automated? Consider an idealized prototype of a highly modular network that has subsets of neurons, each performing subtasks in which (1) each subtask is necessary for high performance on the overall task, (2) each subtask is implemented by single subset only, and (3) each subset executes only a single subtask. The combination of (1) and (2) suggests that the removal of one of the subsets from the network would harm performance because the network lacks the implementation of necessary subtask. We say that such a subset important. Next, given that neurons are frequently understood as feature detectors, (3) suggests that the neurons in a module should tend to be strongly activated by inputs which contain features relevant to the module's 's subtask. We say that such subset is coherent. Figure I provides an illustrative example of coherence in networks that are trained on a task that lends itself to parallel processing via subtasks. Measuring importance and coherence will give prototypical conditions, despite not perfectly satisfying them.
sense of the degree to which networks approximate these
Results and contributions: To measure these proxies, we utilize interpretability methods that have been conventionally used for single-neuron analysis to study these partitions. We find that the partitions have groups of neurons that are disproportionately likely to be important compared to random groups of neurons. We also find that the groups of neurons in the partitionings are reliably more coherent than random ones, though only with respect to features other than class label.
By showing that our partitioning methods are able to reveal modularity, these results suggest that they can be used to understand deep networks. Our key contributions are threefold:
1. Introducing two proxies, importance and coherence, to assess whether partitioning of network exhibits modularity and identify which subsets of neurons are the most responsible.
2. Quantifying these proxies by applying single- neuron interpretability methods on subsets of neurons in an automated fashion
3. Applying our methods on the partitions produced by spectral clustering on a range of neural networks and finding evidence of modularity captured by these partitions.
2

---

Under review as a conference paper at ICLR 2022
1. Network
2. Graph
Subcluster
4a. Analysis: Lesion
4b. Analysis: Feature Vis
Figure 2: Our procedural pipeline. The first three steps generate partitioning of the network into "subclusters" which we analyze using (4a) lesion and (4b) feature visualization methods to measure importance and coherence compared random subclusters. Finally, (not shown in the pipeline), aggregate results to produce Fisher-Bates p values and effect measures. These final steps are shown in Figure 4
RELATED WORK
The work most closely related to ours is Filan et al. 2022) who also use spectral clustering to establish that deep networks are often clusterable and investigates what factors influence clusterability. They also introduce a method for regularization for modularity among clusters neurons. We extend their work by bridging graphical clusterability and functional modularity. This line of work inherits
insights from network science involving clustering in general (Girvan & Newman 2002 Newman &
Girvan 2004), and spectral clustering (Shi & Malik 2000 von Luxburg 2007) particular.
In our experiments we combine clustering with interpretability tools to measure importance and coherence. This highlights the usefulness of combining clustering with interpretability methods for more rigorously understanding networks. We use feature visualization (Olah et al. 2017] Watanabe 2019) and lesions (Zhou al.) 2018). but in similar way, other interpretability techniques including analysis of selectivity (Morcos et al. 2018 Madan et al. (0200), network "dissection" (Bau et al.
2017 Mu & Andreas 2020), earth-mover distance (Testolin al. 2020), intersection information
Panzeri et al. (017)) could also be combined with clustering based partitionings under similar framework. Relatedly, Cammarata et al.) (2020) demonstrate that feature visualization and analysis of weights can be used to identify groups of neurons whose functionality is human-interpretable
This work adds to a body of research focused on modularity and compositionality in neural systems either at the neuron level (You et al., 2020; Mu & Andreas) 2020 Voss et al.) 2021) or at the subnetwork level (Lake et al., 2015 2017] Csordás et al. 20211. There also exist techniques for developing more modular networks which either have explicitly modular architecture (Alet et al.) 2018: Parascandolo al. 2018 Goyal et al. (219)) or are trained in way that promotes modularity
regularization parameter isolation (Kirsch al. 2018 De Lange et al. 2019).
3 METHODS
To evaluate modularity, procedural pipeline is follows. (1) We begin with trained neural network; (2) construct a graph from it, treating each neuron as node; (3) perform spectral clustering on the graph to obtain partitioning "clustering" of neurons which we then further divide by layer to obtain "subclustering", (4) analyze the subclusters according to our modularity proxies opera- tionalized by lesioning neurons or feature visualization and comparing them random subclusters;

---

Under review as a conference paper at ICLR 2022
and (5) aggregate results across the network to obtain final value and effect measure. This pipeline
is outlined in Figure 2 and Figure 4 and each step is explained in detail below.
3.1 GENERATING PARTITIONS WITH SPECTRAL CLUSTERING
To partition a network into clusters, we use approach based on Filan et al. (2021) which consisted of two steps: "graphification" " transforming the network into an undirected, edge-weighted graph; and clustering obtaining a partitioning via spectral clustering.
To perform spectral clustering, a network must be represented as an undirected graph with non-negative edges. For MLPs (multilayer perceptrons), each graph node corresponds to neuron in the network including input and output neurons. For CNNs (convolutional neural networks), node corresponds to single channel (which we also refer as a "neuron") in a convolutional layer] For CNNs, we do not use input, output, or fully-connected layers for clustering.
Graphification:
For graphification, test two ways of assigning adjacency edges between neurons: with weights and with correlations. For weight-based clustering with dense layers, if two neurons have a weight connecting them in the network, their corresponding vertices are connected by an edge with weight equal to the absolute value of the network's 's weight between the neurons. For convolutional channels, we connect them by an edge with weight equal to the L1 norm for the corresponding 2D kernel slice. If layers are connected but with a batch- norrhoooooaaiiionn layer in between, we mimic the scaling performed by the batch norm operation by multiplying weights by 7/ + g) where y is the scaling factor, a is the moving standard deviation, and E is a small constant. Notably, this method of constructing the graphs requires no dataset or runtime analysis of the network.
While graphification via weights only results in connections between neurons in adjacent layers, doing so with correlations creates more dense graphs. With this method, we connect the nodes for two neurons with their squared Spearman correlation across validation set. Spearman correlation gives the Pearson (linear) correlation between ranks and reflects how well two sets of data can be related by a monotonically increasing function Rather than using neurons' post-ReLU outputs to calculate these correlations, we use their pre-ReI.U activations with the goal of extracting richer data from neurons F Again, for convolutional channels, we take the L norm of activations before calculating Spearman correlations. The fact that we take absolute valued weights or squared correlations to construct edges between neurons in graphification means that our analysis does not discriminate between positive and negative associations between neurons.
In addition to graphification via weights versus activations, we also test two scopes in which to perform clustering: global and local. For global clustering, we cluster on one graph for the network as whole. For local clustering, we produce partitioning for each layer individually by clustering on the graph of connections between and the layers adjacent to I. Ultimately, we run 4 experiments on each network by clustering (weights, activations) x {globally, locally).
Spectral (sub)elustering: We perform normalized spectral clustering (Shi & Malik 2000) on the resulting graphs to obtain a partition of the neurons into clusters. For all experiments, we set the number of clusters to k= = 16. In Appendix A.6 we reproduce a subset of experiments with E {8, 12), showing that results are robust to alternate choices of k. Refer to appendix A.l for a complete description of the spectral clustering algorithm.
Layers at different depths of a network tend to develop different representations. To control for these differences, we study the neurons in clusters separately per layer. Therefore, for global clustering in which clusters of neurons span more than one layer, we analyze clusters one layer at a time. We call these sets of neurons within the same cluster and layer subclusters. To ensure comparability between these clusterings when performing local clustering, we set the number of clusters per layer to be the same as the number produced in that layer with global clustering. In our experiments, we compare
'This subtly different from treating each activational unit as neuron in networks such as ResNets with skip connections. a unit used as inputs to multiple layers, we consider these inputs to be separate neurons. We use Spearman rather than Pearson correlation because networks nonlinear, and there is particular reason to expect associations between arbitrary neurons to be lincar.
"Atthough all negative values are mapped to zero by a ReLU, we expect the degree of negativity to carry information about the possihle presence (or lack thereof) of features which the neuron was meant to detect, especially for networks trained with dropout.
4

---

Under review as a conference paper at ICLR 2022
these subclusters to other sets of random units of the same size in the same layer. When discussing these experiments, we refer to subclusters from the clustering algorithm as "true subclusters" and sets of random neurons as "random subclusters." Random subclusters form the natural control condition to test whether the specific partitioning of neurons exhibits importance or coherence compared to alternative partitions, while taking account of layer and size.
3.2 ANALYSIS PIPELINE
Summary: our experiments, we measure the degree to which subelusters identified via spectral clustering are more important and coherent than random sets of neurons of the same size and layer. Operationalizations are given in subsections and 4.2 but in brief, the measure for importance of a subcluster quantifies the performance reduction from dropping out "lesioning" the neurons in that subcluster, and the measures for coherence quantify the degree to which the neurons in a subcluster are mutually associated with some feature of an imput. For each subcluster greater than neuron in size and which did not include every neuron in the layer, we calculate a measure of importance or coherence and compare to that of 19 random subclusters for ImageNee networks or 9 random subclusters for ImageNet ones. These experiments and measures are discussed in detail in section 4 We present two measures how true and random subclusters compare under these proxies. First, we test whether true subclusters are disproportionately often more important or coherent than random subclusters, and report a p value for this test, which we call a Fisher-Bates p value. This value is our primary focus. However, for additional resolution, we also calculate an effect measure used to assess the importance and coherence of the 'average' subcluster. Importantly, the Fisher-Bate value and effect measure are based different statistics and measure different things.
and
partitioning methods find more modular subsets of neurons than simply choosing random subsets. To do this, for each subcluster measurement we take the percentile of each true subcluster relative to the distribution of measurements of random subclusters [ Next, we use the Fisher method to test whether the subclusters in single network are disproportionately more modular than random neurons. To do so, we first center the subcluster percentiles around 0.5, which under the null hypothesis would give a granular approximation of the uniform distribution. We then combine the centered percentiles {PI..... Pn into the test statistic - E log Pi, which under the null hypothesis is drawn from chi-squared distribution with 2n degrees of freedom. We can then produce a p value for each network, testing whether this statistic is higher than the null hypothesis would produce, which would mean that there were more low percentiles than if they were distributed uniformly This procedure is illustrated in Figure 4
Fisher-Bates values: We wish to test whether spectral clustering-
For all non-ImageNee networks, train and analyze 5 different versions of each which requires a second aggregation step. In each condition, we take the mean of the p values for each network and correct it using the corresponding quantile of Bates(n 5) distribution which gives the distribution of the mean of five independent uniformly-distribuuted random variables
This produces a value for every network architecture, partitioning method, and modularity metric, which we call Fisher-Bates values] 88] We next correct for multiple testing using the Benjamini Hochberg method (Benjamini & Hochberg 1995), controlling the false discovery rate at the 0.05 level. See Appendix A.7 for details. Figure 3 shows which values are significant after this correction.
In summary, (1) first aggregate each network's subcluster percentiles using the Fisher method, (2) then we aggregate these new values among identically-coniiuue replicates of the same experiment using the Bates method, and finally (3) we correct for multiple comparisons using the Sonjjamini-Hochbeee procedure.
'For metrics where high values indicate modularity, we take the percentile of the negative metric, that low percentiles consistently indicate modularity.
'The fact that we coarsely measure percentiles and then center them makes this test conservative, because our statistic is more sensitive to low pereentiles than high percentiles.
"The (n) quantile function is Fx = + Σ man (-1)* (nx (n)==)" Marengo et al. 2017).
Eked
We use the Bates method for aggregation here instead of using the Fisher method again because this case, all five p values are from identically configured experiments, and the Bates test less sensitive to low outliers.
FFor simplicity, we use this name even for ImageNet networks, where we do not have multiple networks to apply Bates aggregation
5

---

Under review as a conference paper at ICLR 2022
Obtaining effect measures: In addition p values, we also calculate effect measures which give sense of how different our results for true and random subclusters are on average. We calculate them by taking the mean of 2z/(2 + ()) where is true subcluster measure of importanee/ccherrece and ju is the mean over that of random subelusters ( We do this as opposed to simply taking z/μ in order to avoid divisions by zero. This results in effect measures in the interval [0,2], and which side of 1 they are on indicates whether the true subclusters are more important/coheren than random ones. For ease of interpretation, note that if 2z/(x + μ1) + y. then z/μ & + 2y for y < I, so an effect measure of 1.05 would mean that the measure of a true subcluster was 2 10% higher than the expected measure of random subcluster. Together with these effect measures, we also report their standard errors.
Differences between Fisher-Bates p value and effect measure results: In some of our experiments, the Fisher-Bates values and effect measures seem to "disagree" with one suggesting that modularity was detected and the other suggesting was not. This potential for disagreement is due to the fact that the Fisher-Bates p values are based on the percentile of subcluster measurements relative to the distribution of those of random subclusters, while the effect measures compare the value of true subcluster measurements to the mean value of random subcluster measurements. When the Fisher-Bate p value seems to indicate modularity but the effect measure does not, this means that on the relevant metric, there are more subclusters than would be expected under the null hypothesis whose metric value is higher than that of random subclusters, but the average subcluster has a metric value similar or less than an average random subeluster. In other words, the partitioning method has detected some subclusters that are modular, but the average subcluster found was not modular. We consider this positive result, indicating that our partitioning method is detecting some modularity,
EXPERIMENTS
To show the applicability of our methods different scales, we experiment with a range of networks. For small-scale experiments, we train MLPs with 4 hidden layers of 256 neurons each and small convolutional networks with 3 layers each of 64 neurons followed by a dense layer of 128 neurons trained on the MNIST (LeCun et al. 1998) dataset. At mid scale, we train VGG-style CNNs containing 13 convolutional layers using the architectures of Simonyan & Zisserman (2014) trained on CIFAR-10 (Krizhevsky & Hinton 2009) using the procedure of Liu & Deng (2015). which includes weight decay and dropout for regularization. Finally, for the ImageNet (Krizhevsky & Hinton 2009)
scale, we analyze pretrained ResNet18 (He et al.) 2016). VGG-16, and VGG-19 (Simonyan &
Zisserman 2014) models. Further details including hyperparameters and test performances are in Appendix A.2
4.1 LESION EXPERIMENTS
One approach that has been used for understanding both biological (Gazzaniga & Ivry 2013) and artificial (Zhou et al. 2018 Casper et al.) 2020) neural systems involves disrupting neurons during inference. We experiment with "lesion" tests in which we analyze network performance on the test set when a subcluster is dropped out. We then analyze the damage the network's performance. First, we measure importance by taking the drop in accuracy. Specifically, let 0 be the parameter vector of the neural network, be set of neurons, MM0, e) be masked version of where weights into or out of nodes in e have been set to O, and Acc(O, 0, D) be the accuracy of the network parameterized by on dataset D. Then, our measure for importance is Acc(O, test) Acc(M c), test). where test is test dataset that was not used to construct the activation- based partitionings.
Second, we measure the coherence in a subcluster with respect to class by taking the range of class-specific accuracy drops. Specifically, let test; be the subset of the test set with label i, and let (6,, c, == Ace(O, test,)- - Acc c). test,) be the drop in accuracy for examples with label i from lesioning Then, this measure of coherence is the range (max, A00 c, i)) (min, A(0, c, i)), of accuracy drops over classes. We use this to detect whether clusters are more crucial for some classes over others, which would suggest that they coherently act to correctly label those classes
We use the analysis pipeline from Section 3.2 to test for importance and coherence using these overall accuracy differences and Mlasswiiee ranges. Low Fisher-Bate p values suggest the detection of
IIf this value is ever less than 0 for subeluster, we conservatively replace it with

---

Under review as a conference paper at ICLR 2022
Class Range
Class Range
Softmax H
Softmax H
Acc. Drop
Vis Score
Acc. Drop
Vis Score
Weights Global
-17
n.s.
-3
n.5.
-5
n.5.
n.s.
n.s.
Weights / Local
-26
n.s.
n.s.
n.s.
n.s.
n.s.
Act. Global
-11
n.s.
n.s.
0
Act. / Local
-21
n.s.
-6
-2
-7
n.s.
n.s.
n.s.
-0
MLP, MNIST
CNN, MNIST
Partitioning
Weights Global
-7
-29
-7
-25
-7
n.s.
Weights / Local
-2
-4
n.s.
n.s.
-22
n.s.
33
n.s.
Act. / Global
-11
n.s.
n.s.
-80
Act. / Local
-23
-120
-2
-27
-34
VGG, CIFAR-10
VGG-16, ImageNet
-100
Weights Global
-10
-3
Weights / Local
-80
n.s.
-5
n.s.
Act. / Global
-21
n.s.
n.s.
Act. / Local
-57
-10
VGG-19, ImageNet
ResNet-18, ImageNet
Figure 3: Fisher-Bates p values for (1) lesion- experiments measuring importance via overall accuracy drops (Acc. Drop) and coherence via the range of class-wise accuracy drops (Class Range); and (2) feature visualization-based experiments measuring coherence via the optimization score of feature visualizations (Vis Score) and the entropy of network outputs (Softmax H). Each subfigure corresponds to type of network, where each row represents a partitioning method and each column modularity proxy measurement. The Fisher-Bates p values are shown a log scale rounded to nearest integer. If a value is significant at an a = 0.05 level after Benjamini Hochberg correction (p Pari 0.025 10-1.6), then its background is shaded. Otherwise, "n.s." (not significant) is reported. Results are calculated as described in Section 3.2
an unexpectedly high number of important/coheren subclusters. Mcanwhile, effect measures > 1 indicate more importaacee/cohrence among true subclusters on average compared random ones. Figure 3 shows -ates values, and [ shows effect measure data. Results are summarized section
4.2 FEATURE VISUALIZATION EXPERIMENTS
To further analyze coherence, we leverage another set of interpretability techniques based on feature visualization. We use gradient-baser based optimization norm of the Ree-ReI.U activations of the neurons in a subcluster. Letting the parameter vector be and the subcluster c, we write Act(a, 0, c) for the vector of pre-Rel.U activations neurons in in network on input æ, and denote this optimized input image as rr00, c). which approximately maximizes ||Act(a, 0, c)l The key insight is that propertics of these visualizations (00, c) can suggest what roles the subclusters play in the network. Implementation details are in Appendix A.5 and Figure shows example visualizations.
create an input image which maximizes the L,
We two techniques to analyze coherence using these visualizations of subclusters. First, analyzed the value of the maximization objective for each image we produced, ||Act(z(6, c), 9, c)ll, which call the "score" of the visualization. This gives one notion of how coherent subcluster may
7

---

Under review as a conference paper at ICLR 2022
Table Effect measures for (1) lesion-based experiments measuring importance via overall accuracy drops (Acc. Drop) and coherence via the class-wise range of accuracy drops (Class Range); and (2) Feature visualization-baat baseedd experiments networks measuring coherence via the optimization score (Vis Score) and the entropy of network outputs (Softmax H). Each row corresponds to network paired with a partitioning method. Results are calculated as explained in Section 3.2 For accuracy drop, class-wise range and visualization score experiments, an effect measure > corresponds to more importance/cchhrreee among true subclusters than random ones, while one of < 1 does for softmax entropy experiments. Entries where the effect measure is more than two standard errors away from in the direction of modularity are bolded.
Lesion
Feature Visualization
Network
Partitioning
Acc. Drop High-+Ipp
Class Range High-+Coh.
Vis Score
Softmax
Low->Coh
Weight/Global
1.123 + 0.058
0.701 +: 0.036
1.003 1 0.004
1.105 1 0.021
Weight/Local
1,061 + 0.048
0.676 + 0.029
1.024 + 0.004
0.931 + 0.016
MLP, MNIST
Act//Global
0.883 0.038
0.646 0.024
1.02 0.003
0.997 = 0.013
Act./Local
0.929 0.040
0.687 0.025
1.026 0.003
1.011 = 0.012
Weight/Global
0.837 + 0.048
0.527 + 0.031
0.998 + 0.004
1.026 + 0.008
Weight/Local
0.814 0.046
0.635 = 0.033
1.007 = 0.007
CNN, MNIST
Act//Global
1.078 + 0.061
0.543 0.039
0.933 0.006
1.025 = 0.011
Act/LLocal
0.939 + 0.060
0.625 - 0.043
0.925
0.970
Weight/Global
0.682 0.066
0.407 = 0.042
0.011
1.124 0.012
Weight/Local
0.808 + 0.041
0.692 + 0.033
1.013 - 0.006
0.992 = 0.012
VGG, CIFAR-10
Act/Global
0.926 + 0.032
0.679 0.023
1.327 0.005
0.950 - 0.009
Act./Local
0.956 0.030
0.695 0.021
1.379 0.004
0.930 + 0.008
Weight/Global
1.245 - 0.042
0.763 - 0.040
1.043 0.005
0.998 + 0.001
Weight/Local
1.166 0.019
0.823 - 0.020
1.076 10761:00 0.003
0.991 + 0.001
VGG-16, ImageNet
Act/Global
1.154 0.026
0.838 0.028
1.066 0.003
1.000 + 0.001
Act./Local
1.084 0.020
0.879 0.021
1.056 0.003
1.001 0.001
Weight/Global
1.003 1 0.001
Weight/Local
1.099 1 0.003
1.001 1 0.001
VGG-19, ImageNet
Act//Global
1.046 = 0.003
0.996 = 0.001
Act./Local
1.081 0.002
1.004 = 0.001
Weight/Global
0.915 + 0.045
0.957 0.04
Weight/Local
0.970 0.015
0.970 0.016
ResNet18, ImageNet
Act/Global
0.972 0.016
0.970 0.018
Act./Local
0.983 + 0.014
0.967 0.015
be with respect to input features, because if single image can strongly excite an entire subcluster, this suggests that the neurons comprising it are involved in detecting/processing similar features. Second, we obtain a measure of coherence by analyzing the entropy H (label 10(6, c); 0) of the softmax outputs of the network when these images are passed through. the entropy is low, this suggests that cluster is coherent with respect to output labels.
Just as with lesion experiments, we perform analysis using these two methods using the pipeline from Section 3.2 to measure how coherent true subclusters are compared to random ones. Low Fisher- Bates p va vulu suggest the detection of an unexpectedly high number of coherent subclusters. For visualization score experiments, effect measures > 1 indicate coherence while for the softmax H ones, effect measures < 1 indicate coherence. Figure 3 shows Misher-Bates p values, and TableT[shor effect measure data. Figure Shows shows histograms of feature visualization percentiles for partitionings of a VGG network trained on CIFAR-10, to illustrate the significance of the Fisher-Bates values.
4.3 FINDINGS
Our partitionings identify important subclusters. Fisher-Bates p values for lesion accuracy drops are low and significant, as shown in Figure 3 indicating that sub-clusters are more likely to be highly important relative to random groups of neurons. However, as shown in Table [ not all of the corresponding effect measures are below one, even when the Fisher-Bate p value is significant. This indicates that when we detect that an unusual number of subclusters are important, this does not necessarily correspond to importance on average.
8

---

Under review as a conference paper at ICLR 2022
Our partitionings identify subclusters that are coherent w.r.f. input features but not class label. Class- -lass-specific measures of coherence, class-wins wise lesion accuracy drop range and output entropy, showed significant coherence in almost conditions. The class-wise range measure even tended to show that subclusters were less coherent w.r.t. class than random groups of neurons. However, the visualization score measure of coherence was reliably significant as shown in Figure and the effect measures were reliably above 1. Together, these results offer evidence that subclusters tended to perform coherent sub-tasks, but not in a class-specific way.
All partitioning methods give similar results. In Figure 3 we find that activation-based clusterings have a weak tendency to produce lower Fisher-Bates p values in our analysis compared to weight- based ones. Mcanwhile, between local and global clustering methods, local methods seem to have a weak tendency to produce lower Fisher- Batees p values. However, neither tendency was consistent, and all clustering methods performed similarly overall. This is somewhat unexpected: one might have predicted that weight-baset methods' lack of runtime information or local methods' lack of global information would lead to lower quality clusterings, but this was not the case.
5 DISCUSSION
Contributions: In this work, we introduce several methods for partitioning networks into clusters of neurons and analyze the resulting partitions for modularity. Key to this is analyzing proxies: importance means of understanding what parts of a network are crucial for performance, and coherence as a measure for how specialized these parts are. We rigorously evaluate these proxies using statistical methods, finding that even the weights-only clustering methods are able reveal clusters with a significant degree of importance and coherence compared to random ones. In each network, we found evidence that our partitioning methods were able to identify modular subsets of neurons via measuring accuracy drops under lesions (importance) and feature visualization scores (coherence). To the best of our knowledge, ours the first method which is able quantitatively assess modularity exhibited by neural networks in a way that does not require a human in the loop.
Relation to other research: Having effective tools for interpreting networks has been a key goal. especially when this aids in the diagnosis of failure modes (c.g.. Carter et al. (2019); Mu & Andreas (2020). Our work relates to this goal, though indirectly. The tests we perform are based on data from lesions and feature visualizations, both of which are interpretability tools. But rather than directly using these data to interpret subclusters, our focus is one step higher: on automatedly testing whether these subclusters are worth analyzing at all, and finding ways to screen for ones that should be the subject of deeper investigation. By showing that the partitioning methods we use identify subclusters that exhibit modularity, these results suggest that clustering neurons offers useful level of abstraction through which to study networks.
Limitations: One limitation of our work is lack of assurance that importance and coherence reliably strong proxies for comreeccmprehensible forms of modularity. While they are sufficient to imply some degree of abstractability with respect to the task hand, they may not always be particularly useful for a prosaic understanding. Relatedly, our approach is also not designed to identify the subtask performed by a of units which exhibit modularity, nor does it identify relationships between modules. Given these limitations, the tools we introduce should be seen as methods for screening a network in search for evidence of modularity overall and for particular sets of units which seem to exhibit it and merit additional investigation. A final notable limitation is that these methods do not offer tools for building more modular networks beyond techniques for measuring Future work toward this may benefit from our techniques but should also hinge on architectural or regularization- based methods for promoting modularity in networks.
Conclusion:
While we make progress here toward better understanding how networks can be understood, neural systems are still complex, and more insights are needed to develop prosaic understandings of them. The ultimate goal would be to develop reliable methods for building models which are both dynamic and lend themselves to faithful semantic interpretation. We believe that using modularity an organizing principle and expanding our interpretability toolbox with more modular models and better-tuned partitioning methods will lead to a richer understanding of networks and better tools for building them to be reliable and verifiable.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro. Greg Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, lan Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: / www. Lensorflos org/
Software available from tensorflow.org.
Ferran Alet, Tomás Lozano-Pénn Pérez, and Leslie P Kaelbling. Modular meta-learning. 1rXiv:1806.10166, 2018.
arXiv preprint
Carliss Young Baldwin and Kim B Clark. Design rules: The power of modularity, volume 1. MIT press, 2000.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541-6549, 2017.
Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1))289-300, 1995.
Grady Booch, Robert A Maksimchuk, Michael W Engle, Bobbi Young. Jim Conallen, and Kelli A Houston. Objject-Oriente Analysis and Design with Applications. Madison-Weslog Professional, third edition, 2007.
Alfio Borzi and Giuseppe Borzi Algebraic multigrid methods for solving generalized eigenvalue problems. International journal for numerical methods in engineering, 68(8):1186-1111 196, 2006.
Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, and Ludwig Schubert. Thread: Circuits. Distill, 2020. doi: //311///iiilll000022 https::/disttil..pt/////
Shan Carter, Zan Armstrong, Ludwig Schubert, Johnson, and Chris Olah, Activation atlas. Distill, 1(3) 2019.
Stephen Casper, Xavier Boix, Vanessa D' A'mrri,, Ling Guo, Kasper Vinken, and Gabriel Kreiman. Frivolous units: Wider networks are not really that wide. arXiv preprint arXiv:1912.0478. 2020.
François Chollet et al. Keras, htttss://koras 10 2015.
Jeff Clune, Jean--aptnntee Baptiste Mouret, and Hod Lipson. The evolutionary origins of modularity. Proceedings of the Royal Society B: Biological sciences, 280(1755) 2013.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting their functionality through diffcrentiable weight masks. International Conference on Learning Representations, 2021.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. arXiv preprint (.X.:::909.083300 2019.
Daniel Filan, Stephen Casper, Shlomi Hod, Cody Wild, Andrew Critch, and Stuart Russell. Clusterability in neural networks. arXiv preprint arXiv:2103.03338 03386, 2021.
Michael Gazzaniga and Richard B Ivry. Cognitive Neuroscience: The Biology of the Mind: Fourth International Student Edition. WW Norton, 2013.
Michelle Girvan and Mark EJ Newman, Community structure in social and biological networks. Proceedings of the national academy of sciences, 99(12):7821 -8266 2002.
Anirudh Goyal, Alex Lamb, Jordan Hoffimann, Shagun Sodhani. Sergey Levine. Yoshua Bengio, and Bernhard Schälkopf. Recurrent independent mechanisms. arXiv preprint arXiv:/909 10893, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv 2014.
arXiv: 1472.6980,
10

---

Under review as a conference paper at ICLR 2022
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning decompose neural computation. In Advances in Neural Information Processing Systems, pp. 2408-2418, 2018
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabillistie program induction. Science, 15066266):3332-3338 2015.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behaviord and brain sciences, 40, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11) (1111):227-222)) 2324, 1998.
Shuying Liu and Weihong Deng. Very deep convolutional neural network based image classification using small training sample size. In 3rd IAPR Asian Conference on Pattern Recognition, ACPR 2015, Kuala Lampur, Malaysia, November 3-6, 2015, pp. 730-734. IEEE, 2015. doi: 10.11009/CC..
URL
http://cc...rg//////// 10. 1109/ 2015 5. 7486599
Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. On the capability neural networks to generalize to unseon category-poss combinations. arXiv preprint arXiv:2007.00D0 08032, 2020.
James E Marengo, David Farnsworth, and Lucas Stefamic. geometric derivation the irwin-hall distribution. International Journal of Mathematics and Mathematical Sciences, 2017, 2017.
Ari S Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. arXiv preprint arXiv: 1803.06959, 2018
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. arXiv preprint arXiv:2006.14032, 2020.
Mark EJ Newman and Michelle Girvan. Finding and evaluating community structure in networks. Physical review E, 69(2):026113 13, 2004.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7, 2017.
Stefano Panzeri, Christopher D Harvey, Eugenio Piasini, Peter E Latham, and Tommaso Fellin. Cracking the neural code for sensory perception by combining statistics, intervention, and behavior. Neuron, 93(3):
491-507, 2017.
Giambatista Parascandolo, Niki Kilbertus, Mateo Rojas Rojas-Carulla, Carulla, and Bernhard Schölkopf. Learning independent causal mechanisms. In International Conference on Machine Learning, pp. 4036-4044 PMLR. 2018.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-leamn Machine learning in
python. the Journal of machine Learning research, 12:2825-2830.. 2011.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on paltern analysis and machine intelligence, 22(8):888-905, 2000.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint orXiv:1409, 2014.
Alberto Testolin, Michele Piccolini, and Samir Suweis. Deep learning systems as complex networks. Journal of Complex Networks, 8(1)cez0188, 2020.
Ulrike von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17(4):395 416, 2007.
Chelsea Voss, Gahriel Goh, Nick Cammarata, Michael Petrov, Ludwig Schubert, and Chris Olah. Branch specialization. Distill, 6(4):c00024-00 2021.
Chihiro Watanabe. Interpreting layered neural networks via hierarchical modular representation. International Conference on Neural Information Processing, pp. 376-388 388. Springer, 2019.
Jiaxuan You, Jure Leskovee, Kaiming He, and Saining Xie. Graph structure of neural networks. arXiv preprint vxXiv2207.06550 2020.
Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralha. Revisiting the importance of individual units in CNNs via ablation. arXiv preprint / 2018.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 SPECTRAL CLUSTERING ALGORITHM
The spectral clustering algorithm on the graph G EV produces partition of its vertices in which there are stronger connections within sets of vertices than between them (Shi & Malik, (2000) It does by solving relaxation of the NP-Hard problem of minimizing the n-cut (normalized cut) for a partition. For disjoint, non-empty sets XI,...Xa where i V, this is defined by von Luxburg 2007 as:
( 11 my Xx) 1 Σ X.)
1
2 i=1
vol(X,)
for two sets of vertices X,Y S V, we define (XX, Y) = Enjex, Way: the degree of a vertex
U/ E V is d, Σラ= Ways and the volume of a subset X CV is vx) = Eiex d,
We use the scikit-lear implementation (Pedregosa et al. 2011) with the ARPACK eigenvalue solver (Borzi & Borzi 2006).
Algorithm 1: Normalized spectral clustering according to Shi Malik (2000), implemented in scikit-learn (Pedregosa al., T01T), description taken from von Luxburg (2007).
Input :eigghted adjacency matrix W E R number ks of clusters to construct
Compute the unnormalized Laplacian L.
Compute the first k generalized eigenvectors u1,..., Ug of the generalized eigenproblem
Lu = XDu.
Let U E x be the matrix containing the vectors U1...., Ug columns.
For i = 1, 1,,,, n, let y: R2t be the vector corresponding to the 8th row of U. Cluster the points (3)))s1...... in Ie with the k-means algorithm into clusters Cz,..., Ck+
Output: Clusters Al,..., Aş with A, = {ااز E C.}.
A.2 NETWORK TRAINING DETAILS
We use Tensorflow's 's implementation of the Keras API Abaai et al./ 2015 Chollet et al.) 2015). When training all networks, we use the Adam algorithm Kingma Ba 2014) with the standard Keras hyperparameters: learning rate 0.001, 31 0.9. 32 0.999, no amsgrad. The loss function was categorical cross-entropy. For all non-ImageNet networks, we train 5 identically-confiuured replicates.
Small MIPs (MNIST): We train MLPs with hidden layers, each of width 256, for 20 epochs with batch size 128. All MLPs achieved a test accuracy on MNIST between 97.6% and 98.2%. On Fashion-MNIST, they achieved test accuracies between 88.4% and 89.4%.
Small CNNs (MNIST): These networks had convolutional layers with 64 3 channels each with the second and third hidden layers being followed by max pooling with a 2 by 2 window. There was a final fully-connected hidden layer with 128 neurons. We train them with a batch size of 64 for 10 epochs. All small CNNs achieved a testing accuracy MNIST between 99.1 1% and 99.4%. On Fashion-MNISS they achieved test accuracies between 91.8% and 92.4%.
Mid-sized VGG CNNs (CIFAR-10): We implement a version of VGG-16 16 described by Simonyan & Zisserman 2) Liu Deng 2015). We train these with Adam, for 200 epochs with batch size of 128. These are trained using Le regularization with coefficient of 5 x 10-4 and dropout with a rate tuned per-layer as done in [imonnyan & Zisserman (2014): Liu & Deng (215). Training was done with data augmentation which consisted of random rotations between 0 and 15 degrees, random shifts both vertically and horizontally of up to 10% of the side length, and random horizontal flipping. The unregularized networks achieved testing accuracies between 90.3% 3% and 90.9% while the regularized ones did between 82.8% and 86.6%.
Large CNNs (ImageNet): We experimented with VGG-16 and 19 (Simonyan & Zisserman 2014) and ResNet-18, and 50 (He et al.) (016) networks. Weights were obtained from the Python image-cl assi fiers package, version 1.0.0.
12

---

Under review as a conference paper at ICLR 2022
A.3 PIPELINE SECOND PART
Figure Our extended procedural pipeline. This figure expands Figure and shows the successive steps after generating a partitioning of subclusters (step 3 in Figure 2). After performing either lesion or feature visualization analysis, the results from each true cluster and its random clusters are aggregated produce p values and effect measures. For simplicity, only the analysis for the lesion experiment is presented, but the same pipeline is used for the feature vis experiments.
For each true subcluster from the generated partitioning:
Generate m random subclusters within the sume layer and having the same number of neurons the subcluster:
True Subcluster
3'. Random Subeluster
4a. Analysis: Lesion
4a' Analysis: Lesion
Compute the drop accuracy Stree
Compute the drop in accuracy grand
Combine samples of frone to form the random subcluster distribution
Mean: pon!
Drop accuracy value
Compute the effect of subcluster
5b. Compute the percentile of subcluster é
d, = (g gradd
6a. Combine the subclusters' effect measures (d,) by taking their mean and computing the average effect measure
6b. Combine the subclusters' percentiles (p.) using the Fisher method to produce a pvalue
13

---

Under review as a conference paper at ICLR 2022
Figure Histograms of feature visualization percentiles for four partitionings of VGG CIFAR- 10 network. A VGG CIFAR-10 network is partitioned using four methods (f (weights, activations] fglobal, local}) and analyzed with the feature visualization experiment produce the collection of percentiles for each subcluster. This figure shows the percentile distribution for each clustering as a histogram with a single percentile value per bin. Recall that a lower percentile means that true subcluster is disproportionately often more coherent than random subclusters while controlling for layer and size. Under the null hypothesis, percentiles distribute uniformly. Visual inspection demonstrates that, for example, Weights / Global, Act Global, and Act / Local deviate from the this paper, we use the Fisher-Bates procedure to conduct this hypothesis
uniform distribution. testing rigorously. Indeed, Figure 3 shows that these clustering of the VGG CIFAR-10 10 network significant when aggregated over five models.
Weights Global
Weights Local
0.15
Probability
0.10
0.05
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Act. Global
Act. / Local
0.15
Probability
0.10
0.05
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Precentile
Precentile
A.4 CORRELATION BAASD VISUALIZATION
Figure 6 shows examples from halves-MNIST dataset. To create each example, two images were randomly selected, resized to have half their original width, and concatenated together. Each image was labeled with the sum of the two digits modulo 10. Examples are shown in Figure
Halves-MNIST Dataset:
Visualization: To create the images from Figure which show clusters of neurons in the first layer of an MLP trained on halves-MNIST. we use correlation- based visualization algorithm from Watanabe (2019) We construct visualizations of neurons using their correlations with the input pixels' pre ReLU activities across the test dataset. Instead of Pearson (linear) correlation, we use the Spearman correlation (which is the Pearson correlation of ranks) because it is able to capture relationships which tend monotonically increase even if they nonlinear.
After obtaining visualizations for each neuron in a subcluster, we do not directly take their average to visualize the entire subcluster. To see why, consider two neurons which are highly anticorrelated across the testing set. These neurons are highly coherent, but averaging together their visualizations would obseure this by cancellation. To fix this problem, align the signs of the visualizations for individual neurons using a variant stoahestic alignment algorithm from Watanabe (2019). To visualize subcluster, for number of iterations (we use 20), iterate over its neurons and calculate
14

---

end end Under review as a conference paper at ICLR 2022
for each the sum of cosines between its visualization and each of the other neurons' visualizations in vector form. If this sum is negative, we flip the sign of this neuron's 's visualization. After this procedure, take the mean of the visualizations within a subcluster. This process is detailed in Algorithm
Algorithm 2: Sign Alignment Algorithm (Similar to Watanabe (2019)
Result: Set of sign- eegn-alinned neuron visualizations.
Input Neuron visualizations Vi... for iter in num iters do for 0; in V do
Calculate sum of cosines, c= Σjtti (o
if c < 0 then
U, 2/
end
Comparison to Random Subclusters: To confirm that the visualization in Figure I show that our clustering algorithms are capturing modularity the first layer of the MLPs, we compare them to the same visualizations but for random subclusters in Figure 6 The visualizations that were used to produce Figure [ are each at the top of a column of one of the panels in Figure 6 [ndd are above visualizations for random subclusters of the same size and layer. Each column in each panel was independently scaled to have values in the interval [0,1]. each of the four panels, visualizations in the first row reflect the most selectivity to one side or the other meaning that compared to random subclusters, the ones found via these clustering methods tend to be significantly more coherent.
MLPs can Compute Modular Sums: A network an do this for M values by using an intermediate layer of 2 neurons, each of which serve as a detector of one of the possible combinations of inputs. Consider ReLU MLP with 2MM inputs, a single hidden layer with M neurons, and then M outputs. Suppose that it is given the task of mapping datapoints in which the input nodes numbered and M j are activated with value 1 to output in which the mod (i + J, M) node is active with value 1. It could do so if each hidden neuron with a ReLU activation detected one of the M possible input combinations via bias of 1 and two weights of connecting it to each of the input nodes in the combination is detects. A single weight from each hidden neuron to its corresponding output point would allow the network to compute the modular sum. our MIPs, we have M = 10 classes, and the MLPs have a dense layer with 102 neurons preceding the output layer. Thus, they are capable computing a modular sum in the halves and stack-diff tasks we give to them.
A.5 FEATURE VISUALIZATION
All visualizations were created using the Lucid package. The optimization objective for visualizing sub-clusters was the L1 norm of the Pre-ReLU inputs for all neurons inside the subcluster (it was an Ln norm of L1 norms for convolutional feature maps). For small MLPs, small CNNs, and mid-sized CNNs, we generated images using random jittering and scaling, and for ImageNet models, we used Lucid's default transformations which consist of padding, jittering, rotation, and scaling with default hyperparameters. For all networks, we used the standard and no regularization on the Adam optimizer for 100 steps. For visualizations in small MLPs and CNNs, we used versions of these networks trained on 3-channel versions of their datasets in which the same inputs were stacked thrice because Lucid requires networks to have 3-channel inputs. However, we show grayscaled versions of these in 7 Refer to Section of of the main paper for quantitative analysis of the optimization objective values.
10
parameterization of the image
A.6 ROBUSTNESS TO THE CHOICE OF CLUSTER NUMBER
In the main paper, we only present results form experiments in which the number of clusters, k; was set to 16. The fact that we find evidence of modularity in networks from the MNIST to ImageNet scale using 16 suggests that detecting it is robust However, here we also present direct
httpps: htttass//iitabb...cc///essfffffo
15

---

Under review as a conference paper at ICLR 2022
00
0
3
00
1
(b)
5
00
G 0
0000000
00
00
(c)
7
9
000
00
0
7
00
0
0
30
00
07
3
00
00
00
0
0
(d)
Figure 6: Comparison true and random subcluster visualizations for the first layer of MLPs trained on the halves-MNIST dataset. The first rows show visualizations for subclusters, and the bottom four show visualizations for random ones of the same size. Each panel gives results for different approach to clustering: (a) weights/global, (b) weights/local, (c) activity/global, and (d) activity/local.
comparison between results for CIFAR CNN-VGGs k: E {8, 12, 16). Tables and show these results for lesion and feature-visualizatio experiments respectively. In general, whether or not an experiment resulted in significant Fisher-Bi Batter--atessse p value or an effect measure suggesting modularity consistent across these values of k.
A.7 MULTIPLE TESTING CORRECTIONS
In the main paper in Tables and 5 we report various Fisher-Bates p values that summarize the degree which statistics of sub-clusters vary from those of random groups of neurons within network. For each network, the Fisher-Bates values one-sidedly test whether the true sub-cluster measures reflect more importance/ccherrnce than those of random subclusters. However, when testing multiple networks, one may want to reduce the chance of false positives due to the sheer number of tests performed. To incorporate this analysis into our results, we perform multiple testing correction within Tables 4 and (separately) using the Benjamini Benjamini-Hochberg procedure (Benjamini & Hochberg 1995). This procedure controls the false discovery rate: that is, the expected proportion of rejections of the null hypothesis that are false positives, where the expectation is taken under the data-generating distribution. It relies on all experiments being independent. For a false discovery rate and ordered p values {pI...m.., J. this procedure chooses critical value Pk where k arg max I([p, < jay where is indicator. All values greater than P&c deemed not significant this level. In Figure we demarcate p values significant at the a 0.05 level under this correction. In our case, the critical value Pk 0.025.
16

---

Under review as a conference paper at ICLR 2022
(a)
(b)
(d)
(0)
Figure 7: Example feature visualizations for true and random sub-clusters: In the left column are shown true sub-cluster visualizations, and in the right column are visualizations of sub-clusters of random neurons of the same size the same layer. (a) MI.P, MNIST; (b) CNN, MNIST; (c) CNN-VGG, CIFAR-10: (d) VGG-16, ImageNet: (c) VGG-19, ImageNet.
Table 2: Comparison for different cluster number values CNN-VGG networks for lesion-based experiments. Results are shown for & E {8, 12, 16). Each Fisher-Bates value which is significant under the Benjamini Hochberg multiple correction at a = 0.05 is bold. An effect measure corresponds to more importance/ccherrnee among true subclusters than random ones. All above 1 are bold.
Importance: Acc. Drop
Coherence: Class Range
Network
Partitioning
p Value
Effeet Meas. High +Ipp
p Value
Effect Meas.
k
High CCh.
Weight/Glooal
2.33
10 12
0.912
4.25 10 14
0.430
VGG, CIFAR-10
Weight/Local
0.266
0.851
0.021
0.869
Act//Global
2.55
10 10
0.969
0.578
0.764
Act./Local
1.06
10 21
1.055
0.866
0.772
Weight/Global
4.06
10
0.739
1.64
10
0.632
VGG, CIFAR-10
Weight/Local
0.204
0.873
0.101
0.795
k == 12
Act//Global
2.70
10 10-13
0.903
0.134
0.741
Act./Local
6.30
10 10-11 17
0.994
0.759
0.720
Weight/Glooba
4.40 x 10
0.682
2.44 x 10 10-29 29
0.407
VGG, CIFAR-10
Weight/L
0.021
0.808
2.24
10
0.692
k 16
Act//Global
1.62
10 10-11
0.926
0.845
0.679
Act./Local
2.26
10
0.956
0.266
0.695
A.8 FULL TABULAR DATA
Tables (lesion experiments data) and (feature visualization experiment data) show the value results from Figure and effect measure results from Table [ but do so in form that places the Fisher-Bates p value and effect measure data from each network in the same row. All Fisher- Bates p values significant according the Benjamini Hochberg method measures indicating greater average importance/coherrence among true subclusters compared to random subclusters are in bold.
bold, and all effect
17

---

Under review as a conference paper at ICLR 2022
Table 3: Comparison for different cluster values CNN-VGG networks for feature visualization- based experiments. Tesults are shown for E 8, 12, 16. Each Fisher-Bater Bater value which is significant 0.05 level under the Benjamini Hochberg multiple correction is bold. For vis score tests, an effect measure > I corresponds to more coherence, and for softmax H tests, one of < corresponds to more coherence. All effect measures on the size of indicating more coherence are bold
Coherence: Vis Score
Coherence: Softmax H
Network
Partitioning
Value
Effect Meas.
p Value
Effect Meas.
Highh+Coh.
Low-+Coh.
Weight/Global
5.85
10 10
1.011
0.032
1.028
VGG, CIFAR-10
Weight/L.oco
0.939
0.974
0.784
1.059
k =
Act./Global
1.58
10 67
1.345
0.368
0.940
Act./Local
1.358
0.179
0.946
Weight/Global
3.71
10
0.959
0.917
1.106
VGG, CIFAR-10
Weight/Looal
0.732
0.993
0.089
0.945
k = 12
Act./Global
2.96 x 10 78
1.363
0.065
0.917
Acta/Local
1.57 x 10 102
1.364
0.041
0.928
Weight/Global
6.98 x 10
0.871
0.613
1.124
VGG, CIFAR-10
Weight/Local
0.442
1.013
0.616
0.992
16
Act//Global
5.59 10
1.327
0.045
0.950
Act./Local
7.79 x 10 121
1.379
0.013
0.930
Table 4: Results for lesion-basee experiments in networks involving importance as measured through overall accuracy drops and coherence measured by the class-wise range of accuracy drops. Each row corresponds to network paired with a partitioning method. Results are calculated as explained an
0.05 level under the Benjamini Hochberg multiple correction is in bold. In this case, significance means S 0.025. For both accuracy drop and classwise range experiments, an effect measure 1 corresponds more importance/coheree among true subclusters than random ones. All such effect measures which are further than two standard errors above are in bold.
in Section 3.2 Each Fisher-Bates p value which is significant
Importance: Acc. Drop
Coherence: Class Range
Network
Partitioning
Value
Effect Meas.
p Value
Effect Meas.
High-Imp
High->Coh.
Weight/Global
1.19 x 10 107 17
1,123 0.058
0.817
0.701 = 0.036
Weight/T Local
5.18 x 10
1.061 + 0.048
0.914
0.676 = 0.029
MLP. MNIST
Act./Global
2.36 10
0.883 0.038
0.556
0.646 + 0.024
Aet./Local
5.80 x 10
0.929 + 0.040
0.315
0.687 = 0.025
Weight/Glohal
1.64 10
0.837 0.048
0.984
0.527 = 0.031
Weight/L
0.063
0.814 + 0.046
0.821
0.635 = 0.033
CNN, MNIST
Act//Global
3.52
10
1.078 + 0.061
1.000
0.543 = 0.039
Act./Local
1.56
10
0.939 0.060
0.841
0.625 = 0.043
Weight/Global
4.40
10
0.682 + 0.066
2.44
10 29
0.407 = 0.042
Weight/Local
0.021
0.808 0.041
2.24 x 10
0.692 = 0.033
VGG, CIFAR-10
/cto/ll
1.62
10-11
0.926 + 0.032
0.845
0.679 = 0.023
Act/Local
2.26
0.956 0.030
0.266
0.695 1 0.021
Weight/GGooal
9.44 x 10
1.245 0.042
1.000
0.763 = 0.040
VGG16, ImageNet
Weight/Locca
2.35 x 10 52
1.166 0.019
1.000
0.823 = 0.020
/eto/Global
1.84 x 10
1.154 1: 0.026
1.000
0.838 = 0.028
Act/Local
1.55
10
1.084 0.020
1.000
0.879 = 0.021
Weight/Global
0.002
0.915 101550.0 0.045
0.061
0.957 = 0.047
Weight/Local
3.57 10
0.970 + 0.015
0.743
0.970 = 0.016
ResNet18, ImageNet
/ctoollbbl
5.94 10
0.972 0.016
0.956
0.970 = 0.018
Acta/Local
1.18 x 10-10
0.983 + 0.014
0.958
0.967 + 0.015
18

---

Under review as a conference paper at ICLR 2022
Table 5: Results for feature Aeualizzttionbassel experiments in networks involving coherence measured by the optimization score of feature visualizations and the entropy network outputs. Each corresponds
network paired with partitioning method. Results are calculated as explained in Section 3.2 Each Fisher-Bates value which is significant at am a 0.05 level under the Benjamini Hochberg multiple correction is in bold. this case, significance means p S 0.025. For visualization score experiments, an effect measure true subclusters than random ones, while one of does for softmax entropy experiments. All effect measures which are more than two standard errors away from 1 on the size reflecting greater coherence among true subclusters are bold
corresponds to more coherence among
Coherence: Vis Score
Coherence: Softmax H
Network
Partitioning
p Value
Effeet Meas.
p Value
Effect Meas.
High-Cohh
Low->Coh.
Weight/Global
0.001
1.003 0.004
0.707
1.105 0.021
Weight/Local
0.002
1.024
0.004
0.023
0.931 + 0.016
MLP, MNIST
Act//Global
4.45
10
1.020
0.003
0.025
0.997 = 0.013
Act./Local
8.94
10
1.026
0.003
0.013
1.011 = 0.012
Weight/Global
0.082
0.998 1 0.004
0.863
1.026 + 0.008
Weight/Local
0.344
1.004 0.003
0.519
1.007 0.007
CNN, MNIST
Act//Global
0.266
0.933 0.006
0.831
1.025 = 0.011
Act./Local
0.219
0.925 0.005
0.668
0.970 = 0.010
Weight/Global
6.98
10
0.871 0.011
0.613
1.124 = 0.012
Weight/Local
0.442
1.013 0.006
0.616
0.992 = 0.012
VGG, CIFAR-10
Act./Global
5.59 x 10 84
1.327 + 0.005
0.045
0.950 = 0.009
Act/Local
7.79
10 10-121
1.379 0.004
0.013
0.930 1 0.008
Weight/Global
5.50 10
1.043 + 0.005
0.052
0.998 = 0.001
VGG-16, ImageNet
Weight/Locca
2.52 10
1.076 0.003
0.155
0.991 0.001
Act/Global
2.65 x 10 15
1.066 1 0.003
0.208
1.000 + 0.001
Aet/Local
1.36 10
1.056 0.003
0.633
1.001 0.001
Weight/Global
1.87 x 10 TU
1.061 0.004
0.398
1.003 = 0.001
Weight/Local
3.69 10
1.099 0.003
0.458
1.001 0.001
VGG-19, ImageNet
Act/Global
6.01 x 10 22
1.046 0.003
0.089
0.996 + 0.001
Act//ooca
2.63 x 10
1.081 0.002
0.645
1.004 + 0.001
19


Reviews:
[{"review_id": "v2jc5JQM-B", "reviewer": "Reviewer_ZAxP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The aim of this work is important and surely relevant. I find it likely that some degree of modularity exists in networks and that it is possible to empirically detect their existence.\n\nHowever I have some doubts on the methodology that is proposed in this paper.\n\n1) It is not immediately obvious that the 5 steps described above really provide a good indication of modularity.  As a counter example, imagine to train a network with most nodes masked to zero. In this scenario, at the end of training only a small group of nodes will be ‘active’ (connected by nonzero weights), and such group of nodes will be hugely more ‘important’ and ‘coherent’ than random sets of nodes, but this is *not* an indication of the network modularity. What is the authors’ opinion on this?\n\n2) Connected to the first point, I see an issue in the high level of arbitrariness in the definition of modularity given. In particular, for the scheme one needs to choose: the graph construction algorithm, the clustering algorithm and its parameters and the definition of importance and coherence scores. All these choices will give rise to *different modules* and I see no way in which one should prefer one scheme with respect to the other. \n\n3) I think that all the heuristic choices described in point 2 make the method difficult to reproduce and to extend, and hence there is a risk for this work to become an isolated publication generating little attention and very few further investigations.  \n\n3) What is the value in knowing that some 'modules' exist if we can never discover what these modules do in practice? The article would be hugely more impactful if the authors could provide an indication of the functioning of the modules detected. I would appreciate if the author could comment (both here and in the paper) on whether this could be done at all and, in case, on the possible ways in which it could be done.\n\n", "summary_of_the_paper": "In this work the authors provide an empirical definition of 'modularity' of trained deep networks. Their definition is ‘automated’ in the sense that is does not require any human identification and evaluation of the specific modules.\n\nThe proposed definition consists in the following steps.\n\n1) Construct an undirected graph from the nodes and weights of the trained-network\n\n2) Divide the graph in groups using a spectral clustering algorithm\n\n3)  Define ‘subclusters’ as the sets of all nodes that are in the same cluster and in the same layer\n\n4) For each subcluster compute an *importance* score and a *coherence* score\n\n5) If the sub-clusters are more important and coherent than randomly defined groups of nodes of the same size than the network is said to be ‘modular’.\n\nThe *importance* of a group of nodes is computed as the loss of accuracy that results from masking to zero that group of nodes. The greater the loss of accuracy, the greater the importance of the group.\n\nThe *coherence* of a group of nodes is computed as the maximum magnitude of the pre-activation weights of the group that is achievable with a single image.\n\nBy numerical testing, the authors find that it is indeed possible to find groups of nodes that are more important and coherent than random in a statistically significant way for different architectures.\n", "summary_of_the_review": "Detecting modularity in networks is surely an important and timely problem in deep learning. The authors proposal to address it contains some interesting elements of novelty, but I am uncertain that the method really measures modularity and I believe that the arbitrariness and the many heuristics involved in the definition of modularity make the methodology difficult to trust, reproduce, or extend. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OblE-WSrK20", "reviewer": "Reviewer_d1u7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1.\tThe choice of metrics for modularity seems well-motivated and proposal to quantify the two proxies is seemingly a great strategy. \n2.\tThe paper is very well-written and all the analyses and experiments along with different choices seem reasonable and interesting. \n3.\tThe question of detecting modularity is a very interesting one \n\n \nCritiques: \n1. The paper claims they aim to detect/quantify modularity in DNNs but the proposed technique only seems to address whether spectral clustering-based partitioning can identify modular subsets (to the extent they are `more modular’ than random groups of neurons). The broader question of whether a DNN is modular or not or how its modularity compares against other networks remains untouched in this paper. \n\n2. The results seem more about clusterability than modularity. For instance, an important conclusion I draw from the paper is that the clustering scheme is useful to identify subsets of neurons that jointly serve as a coherent and important subset for a particular task. The extent of modularity quantified by the partitioning is only compared against random groups of neurons. This seems like a very crude baseline and it would have been nice to know if the metrics can be used to compare modularity across networks. \n\n3. Modularity is defined on top of clusterability – but what if the partitioning itself is imperfect. Clusters define candidate modules but what additional value does functional modularity analyses provide in this setting?  \n\n4. In the feature visualization experiment, the value of the maximization objective (the ‘score’) might be highly sensitive to the gradient optimization technique – are the authors doing unconstrained optimization in the input space? Importantly, the features highlighted by the visualization technique do not reveal anything interpretable and the visualizations for the true subclusters and random groups of neurons look very similar in all the datasets the authors tested (MNIST, CIFAR10 or ImageNet). Given this, the merits of using partitioning to analyze/interpret DNNs remains unclear.  \n\n", "summary_of_the_paper": "This paper seeks to answer whether modern DNNs are modular and proposes statistical methods to quantify modularity. In this study, DNNs are described as modular to the extent they agree with 2 prototypical conditions of modularity, namely importance (which is mainly captured by the drop in accuracy after removing a candidate module) and coherence (which would measure the extent to which neurons in a single module are .activated by similar features). These proxies for modularity, importance (with respect to task performance) and coherence (degree of specialization), are computed based on two well-known DNN interpretability techniques, namely lesions and feature visualization.  ", "summary_of_the_review": "While I enjoyed reading the paper and liked how the authors operationalized modularity, I don't think the paper is ready for publication yet as the results and experiment don't seem to add value or strongly support the practical utility of the proposed metrics and means to quantify them in studying DNNs.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "InXHxiMkJsX", "reviewer": "Reviewer_3KTK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, and generally fair in reporting its findings without much exaggeration. The method proposed here is straightforward (perhaps naïve) on one hand, using rather standard tools for building a neuron similarity graph and clustering it, but on the other hand simplicity is not necessarily a bad thing. The results are intriguing and raise interesting questions, which the authors properly discuss and propose some future work for in Section 5. It is clear that the neuron subclusters here do isolate some encoded information in a way that differs from random clustering, and further it seems this property is robust to the design choices in the proposed procedure. \n\nHowever, I am not convinced this method really exposes any modularity in neural networks, which is the weakest point of this work and in some sense it does not fulfil its declared goal. The authors do acknowledge some limitations already in their discussion - stating there that it does not provide support for modular design. However, while they claim these results \"measure modularity\", it is unclear to me whether this is the case, or are they merely measuring redundancy in the network. Indeed, it is typically desired to have multiple pathways in the network encode similar information, as encouraged by dropout and leveraged in network pruning or sparsification (for example, based on the lottery ticket hypothesis). The authors show here that via their partitioning approach, they are able to isolate, and in the lesion case remove, all the neurons that relate to a certain aspect. When compared to random partitioning, the redundancy of information is clearly seen, as random partitions remains more robust or diverse, probably due to redundant information being encoded across many partitions. It is not clear how to go from this evidence of redundancy to saying the network is modular, which would imply that certain subnetworks can serve as individual \"modules\" that can operate in isolation (to some extent). To establish such evidence, one would at least expect to see some difference between the correlation-based construction (which clearly groups neurons together based on encoding similar information - i.e., providing redundancy) and the connectivity-based construction, which one would hope would provide some submodules of the network. The results on the differences between these approaches (or the global versus local methods) are inconclusive, and in any case there is no attempt here to extract clear modules from the network. ", "summary_of_the_paper": "This work aims to quantitatively evaluate modularity in neural networks (although, as detailed below, I am not sure this is what it ends up doing). To this end, the authors propose to cluster the neurons of the network using spectral clustering applied to a graph that is weighted by similarity between the neurons. They consider two approaches for defining similarity - either via similar responses to inputs (using Spearman or Pearson correlation) or by taking into account the weighted connections (leveraging the learned network weights) between them. There are some technical details regarding the division to layers, exact implementation, and especially the consideration of convolutional architectures (where certain neurons are grouped together along channels), which I will not refer to much here for brevity, but suffices to say that these are all well described in the manuscript and make sense without much need for justification. Once the clusters (or \"subclusters\" when divided to layers) are obtained, they are evaluated empirically to related them to the network's task. This portion is mostly based on image classification tasks, so clusters are evaluated based on their impact on classification accuracy, their importance in identifying each class, or coherence of visual features captured by them. The results are somewhat inconclusive (more details below), but it is appreciable that the authors do acknowledge this in a well balanced discussion section.", "summary_of_the_review": "The approach provided in this work is nice (albeit somewhat simple or even straightforward) and provides a way to cluster together neurons that have some redundancy in their captured information, and therefore distil important information pathways or features in the network. The results are intriguing, but in my opinion do not lead to a clear conclusion, and I don't see how to leverage them towards better understanding of neural networks, their properties, or their design. As such, the impact and significance of this work are rather minor. Most importantly, while the paper claims to establish some quantitative approach to measure modularity in neural networks, the results mostly establish the redundancy of information without relating it in a clear way to modularity per se. Therefore, I believe this work is unfortunately slightly below the bar for acceptance to ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4YZ3eD4NbW", "reviewer": "Reviewer_hnkg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper is very well written and easy to follow. The methods and results are convincing and clear. \n2. This paper carefully designed gratification methods for neural network neurons and proposed plausible methods for not only global clusters and also local clusters.\n3. They also provide a statistical method to identify the significant modules automatically. Though the power is very limited. \n\nSome comments (not exactly weakness):\n1. The global gratification is very interesting. I am wondering if the author looked into the special cases of clustering. For example, is there any chance that the global cluster is from disconnected layers? Specifically, a cluster that has neurons from layer 1 and layer 3 but does not have layer 2  neurons.\n2. What is the purpose of local clustering? Why the clustering number of local clustering has to be the same as the global clustering. And how to explain the discrepancy between local clustering and global clustering.\n3. The overall gratification method is somewhat heuristic. And I assume that is the reason why the p-value does not work. Because the gratification did not capture the very \"true\" connections of each neuron. But identifying such a \"true\" connection will be very difficult. I was wondering could the author consider using the unit interference of each neuron and test the impact of other neurons. And using the impact to reflect the connections of the neurons.\n4. Does the identified modules can be taken advantage of to train the neuron network in some way. Links to lottery tickets?\n5. Did the author consider other graph clustering methods? Like Louvain community detection, hierarchical clustering?   \n\n\n\n", "summary_of_the_paper": "This paper uses graph-based clustering methods to identify the humanly comprehensible modularities in the neural network. They thoroughly discussed how they construct the graph of neurons and carry out the clustering. They also proposed the fisher bates p-value for testing the significance of identified modules.", "summary_of_the_review": "A good paper to explore the explainability of neuron networks with graph clustering methods. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper studies if DNNs are modular and proposes statistical methods to quantify modularity.
cluster the neurons of the network using spectral clustering applied to a graph that is weighted by similarity between the neurons.

While the reviewers find the question of modularity relevant, they raise the issue that the results are inconclusive regarding the main stated contribution of the paper (i.e., if modularity is appropriately measured). After discussion, some concerns are answered. However, the main problem of inconclusive results stands. Therefore, this borderline paper is rejected.
Decision: Reject