Clustering units in neural networks: upstream vs downstream information
Authors: Richard D Lange
David Rolnick
Konrad Kording
Venue: Accepted by TMLR ()

Abstract: It has been hypothesized that some form of "modular" structure in artificial neural networks should be useful for learning, compositionality, and generalization. However, defining and quantifying modularity remains an open problem. We cast the problem of detecting functional modules into the problem of detecting clusters of similar-functioning units. This begs the question of what makes two units functionally similar. For this, we consider two broad families of methods: those that define similarity based on how units respond to structured variations in inputs ("upstream"), and those based on how variations in hidden unit activations affect outputs ("downstream"). We conduct an empirical study quantifying modularity of hidden layer representations of a collection of feedforward networks trained on classification tasks, across a range of hyperparameters. For each model, we quantify pairwise associations between hidden units in each layer using a variety of both upstream and downstream measures, then cluster them by maximizing their "modularity score" using established tools from network science. We find two surprising results: first, dropout dramatically increased modularity, while other forms of weight regularization had more modest effects. Second, although we observe that there is usually good agreement about clusters within both upstream methods and downstream methods, there is little agreement about the cluster assignments across these two families of methods. This has important implications for representation-learning, as it suggests that finding modular representations that reflect structure in inputs (e.g. disentanglement) may be a distinct goal from learning modular representations that reflect structure in outputs (e.g. compositionality).

Published in Transactions on Machine Learning Research (06/2022)
Clustering units in neural networks: upstream vs down- stream information
Richard D. Lange
rdlange@seas. www.ene....eeeeeed.eeee
Deportment of Newrobiology University of Pennsyloania Philadelphia, PA 19104
David S. Rolnick
Mila Québec AI Institute
McGill Universily
Montréal, Canada H3A 0G4
Konrad P. Kording
Department of Neurobiology University of Pennsyluania Philadelphia, PA 19104
Reviewed on OpenReview: /ttps://openreese optpr/veew.eeeeee.emeee net/ forum? id= 7Ko funk
Abstract
has been hypothesized that some form of "modular" structure in artificial neural net- works should be useful for learning, compositionality, and generalization. However, defining and quantifying modularity remains open problem. We cast the problem of detect- ing functional modules into the problem of detecting clusters of similar-functioning units. This begs the question of what makes two units functionally similar. For this, we consider two broad families of methods: those that define similarity based on how units respond to structured variations in inputs ("upstream"), and those based on how variations hidden unit activations affect outputs ("downstream"). We conduct an empirical study quantifying modularity of hidden layer representations of collection of feedforward networks trained classification tasks, across range of hyperparameters. For cach model, quantify pairwise associations between hidden units in each layer using a variely of both upstream and downstream measures, then cluster them by maximizing their "modularity score" using established tools from network science. We find two surprising results: first, dropout dra- matically increased modularity, while other forms of weight regularization had more modest effects. Second, although we observe that there is usually good agreement about clusters within both upstream methods and downstream methods, there little agreement about the cluster assignments across these two families of methods. This has important implications for representation-Leaniiie structure in inputs (c.g. disentanglement) may be distinct goal from learning modular representations that reflect structure in outputs (e.g. compositionality).
it suggests that finding modular representations that reflect
1 Introduction
Modularity a design principle in which systems that solve complex problems are decomposed into sub- systems that solve simpler problems and that can be independently analyzed, debugged, and recombined for new tasks. From an engineering perspective, modular design has many benefits, such as increased robustness and fast adaptation to new problems (Simon 1962.. It is well-known that learning systems benefit. from

---

Published in Transactions on Machine Learning Research (06/2022)
A
0125456769
B
Pairwise functional similarity
c
0125456789
GECOREE
(ii) downstream similarity
080000000000000ooooo
upstream similarity
D
Section 3.1
Scction 3.2
Network/lan /layer
score Q"
pairwise similarity
max modular clustering
Dataset
clusters P
A) Given a neural network, such as this schematic of a fully-connected network for classifying MNIST digits, we frame the problem of detecting modules that perform distinct functions inside the network as a problem of identifying clusters units a hidden layer (colors). B) Given some method for quantifying the pairwise functional similarity between any two units, we operationalize "module" as subset of units with high intra-module functional similarity and low similarity to units in other modules, which appears high block-diagonal values in the pairwise similarity matrix after sorting by cluster. C) In this setup, the main challenge is to define what makes any pair of two units "functionally similar" such that two units are "similar" if they are part of the same functional module. We distinguish between two broad classes of methods: (i) those that score similarity based on how "upstream" " factors influence the pair, and (ii) those based on how the pair influences "downstream" behavior the network. D) Schematic of our pipeline quantify modularity: given layer of a network, we compute the n matrix of "similarity" of all pairs of units in that layer. We then search for the maximally modular clustering such that the similarity matrix becomes heavily block-diagonal in B. The output is modularity score (Q) and cluster-assignments (P), for each layer and for each method of quantifying pairwise similarity.
Figure 1:
2

---

Published in Transactions on Machine Learning Research (06/2022)
structure adapted to the problem at hand [Denker et al.
1987 and many real-world problems indeed can be decomposed into sub-problems. It is no surprise, then, that modularity is considered a normative design
principle of evolved biological systems (Wagner et al. 2007 Lipson 2007), including in biological neural
networks Azam 2000 Kashtan & Alon 2005 Kashtan et al.
2007 Clune et al. 2013), and one from which
artificial neural networks (ANNs) can benefit.
Despite these strong intuitions, formally defining and quantifying the modularity of a given system remains an open problem (cf. [Simon (1962 ;: Lipson (007)). It is generally accepted that, by definition, modular systems decompose into sub-systems that implement functions that solve sub-problens problems. Characterizing modules in ANNs therefore begs the question: when are two parts of a network involved in the same "function" (Csordás et al. 2021)? In this paper, we ask this question at the level of pairs of hidden units. That is, we consider a variety of methods for measuring the "functional similarity" of any given pair of hidden units, and we operationalize "module" as a cluster of similarly-functioningg units (Figure [A-B). This definition is not intended to solve the question of what a module is, but to provide a tractable basis for experimenting with various ideas related to modularity, such as how it is affected by regularization.
One goal of this paper is simply to raise awareness of qualitative differences between "upstream" " and "down- stream" ways of thinking about neural representations and the functions they are involved in (Figure [ In section 3 we make these definitions quantitative and give details for our method for detecting and quan- tifying functional modules in the hidden layers of trained neural networks by clustering units into similarly- functioning groups. This framework allows us to directly compare a variety of proxies for the amount of modularity a network exhibits. Experimental results are described in section | In addition to quantitatively scoring modularity, we further investigate whether different similarity measures agree on which units belong to which module. Surprisingly, we find that clusters discovered according to "upstream" measures of fune- tional similarity are consistently distinct from those discovered using "downstream" " measures. Although we do not consider regularization methods explicitly designed to produce modular designs, these initial empirical results nonetheless call for a deoper look at how the "function" of a representation is defined, as well as why and in which sil uations modules may be useful.
2 Related Work
The study of modularity in neural networks has a long history (Clune et al. 2013 Amer & Maul 2019). One common source of inspiration from biology is the functional dissociation of "what and "where" pathways in the ventral and dorsal streams of the brain, respectively (Goodale & Milner 1992). Each of these pathways can be seen as a specialized module (and may be further decomposed into submodules), and many previous experiments on modularity in artificial neural networks have explored principles that would result in similarly distinct what/ what/where /where information processing in ANNs (Rueckl et al. 1989 Jacobs et al. 1991 Di Ferdinando et al. 2001 Bakhtiari et al. 20021. j. A key difference to this line of work is that, rather than specifying the functional role of modules ahead of time, such as one module being "what" " and another being "where," our work sceks to identify distinct functional clusters in trained networks.
Broadly speaking, there are two families of approaches to modularity in neural networks, corresponding to different ways of thinking about the function of parts of a network. The structural modularity family of ap- proaches defines function in terms of network weights and how sub-networks connect to other sub-networks, and so modules are defined as densely intra- and sparsely inter-connected sub-networks (Watanaee et al. 2018 Amer & Maul 2019 Csordás et al. 2021: Filan et al., 2021 Hod et al. (2021). ). The functional modular- dyy family of approaches emphasizes network activations rather than weights, or the information represented by those activations. These include disentanglement,, compositionality, invariance, and others (Bengio et al,
2013 Higgins et al.] 2018 Andreas 2019: Scholkopf et al. 2021 Watanabe] 2019 Watanabe et al.) 2019 The relationship between structural and functional modules is not clear while it certainly seems that they are (or ought. to be) correlated (Lipson 2007]. it has been observed empirically that even extremely sparse inter-module-or www.-mmodllldeeciiieeeeeeeee connectivity does not always guarantee functional separation of information snformation-processing (Béna & Goodman p021p. In this work, we take the functional approach, basod on the assumption that structural modularity is itself only useful insofar as it supports distinct functions of the units, and that often distinct functions must share information, making striet structural delineations potentially counterproduc-
2020
3

---

Published in Transactions on Machine Learning Research (06/2022)
tive. For example, knowing "what" an object is in challenging visual scene helps localize "where" it is, and vice versa.
Our work is perhaps most closely related to a series of papers by Watanabe and colleagues in which trained networks are decomposed into clusters of "similar" units with the aim of understanding and simplifying those networks. In some cases, they quantify the similarity of units structurally using a combination of both incoming and outgoing weights (Watanabe et al. 2018), and in other cases functionally using correlation between hidden unit activations |Watanabel 2019 Watanabe et al., 2019 2020). These are similar in spirit to our goal of idietifying modules by clustering units, but interesting contrast to our approach; while they combine "upstrcam" and "downstream" information for each pair of units, we find stark differences in results between the two.
3 Quantifying modularity by clustering similarity
We split the problem of identifying functional modules into two stages (Figure [p): scoring the pairwise similarity of units, then clustering based on similarity. To simplify, we only apply these steps separately to each hidden layer, but in principle modules could be scored in the same way after concatenating layers together. Section 3.1 below defines the set of pairwise functional similarity methods we use, and Section 3.2 describes the clustering stage.
While we focus on similarity between pairs of individual units, our approach is related to, and inspired by, the question of what makes neural representations "similar" when comparing entire populations of neurons to each other (Korublith et al. 2019). Rather than finding chusters-ofssimilla--eeroo as we do here, one terms of wwwpolarrribbbeeeeeenssssssseeee
could instead define modules we explored such a definition of functional modules, using representational (dis)similarity between sub- populations of neurons. The main challenge of this approach is that existing representatoonalssmillatyy methods are highly sensitive to dimensionality (the number of neurons in each cluster), and it is not obvious how best to control for this when computing dissimilarity between clusters such that the method is not biased to prefer large or small cluster sizes [
In preliminary work,
To further motivate our method, note that representational similarity analysis is closely related to tests for statistical (in)depenee between populations of neurons (Kornblith et al.] 2019 Gretton et al.] [005 Cortes et al.] 2012), and so the problem of finding mutually "dissimilar" modules is analogous to the problem of finding independent subspaces (Hyvärinen et al. 2001
Gutch& & Theis 2007). In Independent Subspace Analysis (ISA), there is an analogous problem of determining what constitutes a surprssing amount of dependence between subspaces of different dimensions, and various
methods have been proposed with different inductive biases (Wu et al. 2021 Bach & Jordan 2004 2003:
Póczos & Lörinez 2005). However, Palmer & Makeig showed that a solution to the problem of detecting independent subspaces is to simply cluster the individual dimensions of the space. This provides some justification for the methods we use here: some technicalities() notwithstanding, the problem of finding subspaces of neural activity with "dissimilar" representations is, many cases, reducible to the problem of clustering individual units based on pairwise similarity, as we do here.
3.1 Quantifying pairwise similarity of hidden units
What makes two hidden units "functionally similar"? In other words, we seek some similarity function S : N.D. T + Rax" that takes as input neural network N evaluated on a dataset ID to solve task T and produces an n x matrix of neg-tieetive similarity scores for all pairs among the hidden units. We further require that the resulting matrix is symmetric, or S,y = Sn²- Importantly, allowing S to depend on the task T opens the door to similarity measures where units are considered similar based on their downstream contribution to a particular loss function.
Note that dimensionality is not problem for other applications of representational (dis)similarity methods, since the population size is traditionally fixed in any single analysis.
FFirst, one can construct pathological examples of dependent subspaces but where all individual variables are independent, so pairwise independence between all clements in cluster canuot guarantee independent clusters. Sccond, summing averaging pairwise dissimilarity across clusters will give quantitatively different value than measuring dissimilarity between the populations, but this brings us back the dimensionality problem already discussed.
4

---

given dataset Published in Transactions on Machine Learning Research (06/2022)
normalized
gradient
upstream
sov
Loss
covariance
downstream
S
S
gless
unnormalized
Figure Color scheme and 2x2x2 visualization of the 8 proposed similarity methods.
Similarity by covariance, The first similarity measure we consider is the absolute value covariance of hidden unit activities across inputs. Let E Rd be the kth input in the dataset, and h,(x)) be the response of the ith hidden unit to input with i E {1,2,...,,3 Then, we define similarity as
K
SeDv
(h.(axs) - h.) (h) h1)
(1)
K
where K is the number of itemss in D and h, # Σ (xx) is the mean response of unit i on the given dataset. Intuitively, the absolute value covariance quantifies the statistical dependence of two units across inputs, making it an upstream measure of similarity. Seov corresponds to some classic notions of *disen- tanglement": to the extent that the rows and columns Stov can be sorted to have highly block-diagonal structure, this means that hidden units cluster into statistically independent or at least uncorrelated groups. This is also similar in spirit to Representational Similarity Analysis, where the similarity of sets of units is quantified using some (often kernel-based) measure of their independence
[Korbblihh et al. 2019
Similarity by input sensitivity. While Seov quantifies similarity of responses across inputs, we next consider measure of similar sensitivity to single inputs, which is then averaged over D. Let Jh denote the x d Jacobian matrix of partial derivatives of cach hidden unit with respect to each of the d input dimensions. Then, we say two units and j are similarly sensilive to input changes on input on if the dot product between the ith and jth row of Jh has high absolute value magnitude. In matrix notation over the entire datasct, we use
Si-sens
Σ jh T
(2)
K
where the superscript "i-sens" should be read as the "input sensitivity."
Similarity by last-layer sensitivity. Let y E R denote the last-layer activity of the network. Using the same Jacobian notation as above, let JH denote the x n matrix of partial derivatives of the last layer with respect to changes in the hidden activities h. Then, we define similarity by output sensitivity as
K
S0-Sems
1
Σ
JM
K
likewise with "o-sens" to be read as "ontput-sensitivity" Note that both h and y depend on the particular input 2s, but this has been left. implicit in the notation to reduce clutter.
Similarity by the loss Hessian. The "function" " of hidden unit might usefully be thought of in terms of its contribution to the task or tasks it was trained on. To quote Lipson
"In order to measure modularity, one must have quantitative definition of function... It is then possible to take arbitrary chunk of system and measure the dependency of
5

---

Published in Transactions on Machine Learning Research (06/2022)
the system function elements within that chunk. The more that the dependency itself depends elements outside the chunk, the less the function of that chunk is localized, and hence the less modular it is." [Lipson] 2007).
Lipson then goes on to suggest that the "dependence of system function on elements" can be expressed derivative or gradient, and that the dependence of that dependence on other parts of the system can be expressed the second derivative or Hessian. Towards this conception of modular functions particular lask, use the following definition of similarily:
0LL
Styee
Σ k=1
K
ah,āhj
where L is the scalar loss function for the task, and should be understood to depend the particular input mg. Importantly, each Hessian on the right hand side is laken with respect to the activily of hidden units, not with respect to the network parameters as it is typically defined.
To summarize, equations ① through 4 provide four different methods to quantify pairwise similarity of hidden units. Seov and Si-enn are upstream, while So-sens and Shess are downstream. All four take values in [0, oo), however, it not clear if the raw magnitudes matter, or only relative (normalized) magnitudes. Take Soo for instance. [Korbblihh et al.] argue that hidden-unit covariance is more meaningful than correlation, since the principal directions of variation both affect learning and have been shown to correlate with meaningful perceptual differences for human observers. On the other hand, we do not want our measure of modularity, which is a global property of an entire layer, to be skewed by small number of highly variable units. Further, the community-detectioo primarily developed and tested on unweighted and undirected graphs. For these reasons, we introduce an optional normalized version of cach of the above four un-normalized similarity measures:
algorithm we use for clustering below has been
Sij Súj max (V...))..c)
(5)
where is a small positive value included for numerical stability, Whereas S.y E [0, oo), the normalized values are restricted to S, E []. In total, this gives us eight methods to quantify pairwise similarity. These can be thought of as 2x2x2 product of methods, as shown in the color scheme in Figure 2 the upstream vs downstream axis, the unnormalized vs normalized axis, and the covariance vs gradient (i.e. sensitivity) axis. We group together both genv and ghess under the term "covariance" because the Hessian is closely related the covariance of gradient vectors of the loss across inputs.
3.2 Quantifying modularity by clustering
Decomposing set into clusters that are maximally similar within clusters and maximally dissimilar across clusters is a well-studied problem in graph theory and network science. In particular, Girvan & Newman proposed method that cuts graph into its maximally modular subgraphs (Girvan & Newman 2002
Newman & Girvan 2004 Newman 2006), and this tool has previously been used to study modular neural
networks (Amer & Maul 2019 Béna & Goodman 2021
We apply this tool from graph theory to our problem of detecting functional modules in neural networks by constructing an adjacency matrix from the similarity matrix by simply removing the diagonal selff-smilariiyy::
S. if
A,
0 otherwise
Given A, we can simplify later notation by first constructing the normalized adjacency matrix, Á, whose elements all sum to one:
A
Ā =
Σnj Ay
6

---

Published in Transactions on Machine Learning Research (06/2022)
or, more compactly, Á= A1, where 1, is a column vector of length containing all ones. Let P be n x matrix that represents cluster assignments for each of units to a maximum of e different clusters. Cluster assignments can be "hard" (P.) E {0) or "soft" (Pag E [0, 1J), but in either case the constraint Pl, = 1n must be met, i.c that the sum of cluster assignments for each unit is 1 3 If an entire column of P is zero, that cluster is unused, so e only provides an upper-limit to the number clusters, and in pract ice we set c= n. Girvan Newman propose the following score to quantify the level of "modularity" when partitioning the normalized adjacency matrix Á into the cluster assignments P:
Q(A. P) Tr(PT AP) - Tr(P , AP).
(6)
The first term sums the total connectivity (or, in our case, similarity) of units that share a cluster. By itself, this term is maximized when P assigns all units to a single cluster. The second term gives the expected connectivity within each cluster under a null model where the elements of Á are interpreted as the joint probability of a connection, and A1, A, Á is the product of marginal probabilities of each unit's connections. This second term cncourages P to place units into the same cluster only if they are more similar to each other than "chance." Together, equation (6 is maximized by partitioning Á into clusters that are strongly iotracconeected and weakly connece....
We define the modularity of a set of ncural network units as the maximum achievable Q over all P:
(*( = arg max Q(A, P)
P ((A)=Q(A)
(7)
To summarize, to divide a given pairwise similarity matrix S into modules, we first construct À from S, then we find the cluster assignments p that give the maximal value Q" Importantly, this optimization process provides two pieces of information: a modularity score Q' which quantifies the amount of modularity in set of neurons, for given similarity measure. We also get the actual cluster assignments Pt, which provide additional information and can be compared across different similarity measures. Given a set of cluster assignments P", we quantify the number of clusters by first getting the fraction of units in cach cluster, r(P)) P.p'' We then use the formula for discrete entropy to measure the dispersion of cluster sizes: H(r) = Dd ry logr, r, Finally we say that the number of clusters in p
"
num clusters H(r(P"))
(8)
We emphasize that discovering the number of clusters in P is included automatically in the optimization process; set the maximum number of clusters e equal to the mumber of hidden units n, but in our experiments we find that P* rarely uses more than 6 clusters for hidden layers with 64 units (Supplemental Figure
It is important to recognize that the sense of the word "modularity" in graph theory is in some important ways distinct from its meaning in terms of engineering functionally modular systems. Whether or not a high value of Q+ indicates more "modular" design requires additional interprotation beyond the choice of pairwise similarity measure S. In graph-theoretic terms, a "module" is a cluster of nodes that are highly intra-connected and weakly inter-connected to other parts of the network, defined formally by Q (Girvan & Newman 2002 Newman & Girvan 2004). This definition of graph modularity uses a particular idea of a "null model" based on random connectivity between nodes in graph (Amer & Maul 2019). While this mull-model of graph connectivity enjoys a good deal of historical precedence in the theory of candomly-connectdd graphs, where unweighted graphs are commonly studied in terms of the probability of connection between random pairs of nodes, it is not obvious that the same sort of null model applies to groups of "functionally similar" units in an ANN. This relates to the earlier discussion of ISA, and provides a possibly unsatisfying answer "surprising" amount of statistical independence between clusters; using
to the question of what counts Q makes the implicit choice that the product of average pairwise similarity, A1, 1 A, gives the "expected"
While we only consider hard cluster assignments our experiments below, we have encountered cases where Q maximized by soft cluster assignments, so we include in the definition here for completeness.
7

---

Published in Transactions on Machine Learning Research (06/2022)
similarity between units. An important problem for future work will be to closely reexamine the question of what makes neural populations functionally similar or dissimilar, above and beyond statistical similarity (Csordás et al. 2021 Kornblith et al. 2019), and what constitutes a surprising amount of (dis)similarity that may be indicative of modular design. Throughout this paper, our use of the term "modularity" to refer to Q" should be understood as only a quantitative proxy for truly modular design principles, but one that enjoys some precedence the study of modular neural networks (Kashtan & Alon 2005 Watanabe et al. 2018
Béna & Goodman 2021).
Finding P* exactly is comollee (Brandes et al. 2008 in practice we use variation on the approximate
method proposed by (Nwwman 2006). Briefly, the approximation works in two steps: first, an initial set of cluster assignments is constructed using a fast spectral initialization method that, similar to other spectral clustering algorithms, recursively divides units into clusters based on the sign of eigenvectors of the matrix
B A A 1 A and its submatrices. Only subdivisions that increase Q are kept. In the second step, we use Monte Carlo method that repeatedly selects a random unit i then resamples its cluster assignment, holding the other n assignments fixed. This resampling step involves a kind of exploration/expliitaiiin trade-off: Q may decrease slightly on each move potentially find a better global optimum. We found that it was beneficial to control the entropy of cach step using temperature parameter, to ensure that a good explore/explots balance was struck for all A. Supplemental Figure S5 shows that both the initialization and the Monte Carlo steps play a crucial role in finding P* consistent with the observations of (Newman 2006). Full algorithms are given in Appendix A.1
4 Experiments
4.1 Setup and initial hypotheses
We would like to understand the behavior of the various notions of modularity above, operationalized using Q", for each of the eight different methods for quantifying pairwise similarity introduced in the previous section. In initial experiments, we began by studying large collection of simple feedforward fully- connected networks trained on the MNIST dataset {LeCun et al.) 1998) across range of regularization schemes (Experiment 1). In follow-up experiments, we further explored the effects of network width (Experiment 2) and depth (Experiment 3). A summary of the architectures and hyperparameters used in each experiment can be found in Supplemental Figure ST and Table S1
In all experiments, we defined æ (input layer) as the raw pixel inputs and y (output layer) as the 10-dimensional class logits. We discarded models that achieved less than 80% test accuracy (this only happened at extreme values of regularization). Summary statistics of model behavior as function of reg- ularization strength are shown in Supplementary Figures $2 through SS4 For reference values, we also performed all analyses on 100 randomly initialized models from each architecture with zero training. Models were written and trained using Py Torch (Paszke et al. 2019) and PyTorch Lightning and all compute jobs were run on a private server and managed using GNU Parallel (Tange 2011). Code is publicly available at httpp://github... com.ub...oo//ie www//oodangLaaa/llllceece.e
Before running these experiments, we hypothesized that
Dropout would decrease modularity by encouraging functions to be "spread out" over many units.
2. L2 regularization (weight decay) would minimally impact modularity, since the L2 norm is invariant to rolation while modularity depends on axis-alignment.
3. L1 regularization on weights would increase modularity by encouraging sparsity between subnet- works.
4. All similarity measures would be qualitatively consistent with each other.
As shown below, all four of these hypotheses turned out to be wrong, to varying degrees.
https://www.pytorblggnnii
8

---

Published in Transactions on Machine Learning Research (06/2022)
0.128
Q 0.166
0.196
0.232
y
088
64
64
02
7
0.5
0.5
0.5
-0- la
Bess
- ly
0.4
moduarity X000
- drepout
0.3
0.3
0.2
0.0
0.0
0.0
0.0
0.25
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.50
1.00
0.00
0.25
0.50
0.75
1.00
S-sotis
mocutalty Score (0)
0.4
0.4
0.4
0.2-
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.25
regularization
% regulariztiion
regularization
Figure 3: Modularity score as function of regularization strength. Inset network diagram: Results for Experiment 1 are based on a simple feedforward network, trained on MNIST, with two hidden layers and 64 units per layer. Top row: example similarity matrices, sorted by cluster, and their associated Q" values. Thin red grid lines indicate boundaries between clusters discovered by the assignments p Similarity in these examples measured by Suov i.e. the absolute value correlation between hidden units' activity test items, across increasing values of dropout regularization (curved red lines). Bottom two rows: modularity score (Q") function of percent regularization, combined for both hidden layers. Each subplot shows a different one of our eight similarity measures. Errorbars indicate standard error of the mean (across seeds and layers). Gray shading shows the distribution of scores for untrained models, out to #3o. Note that the x-axis, "percent regularization," is different for each series: it identical to dropout probability for the dropout series, but log-spaced from 0% 1e- to 100% = 1e for T2 and log-spaced from 0% le-5 to 100% le-2 for L1.
cluster
4.2 How modularity depends on regularization (Experiment 1)
In Experiment 1, we trained a fully-connected neural network on MNIST with two hidden layers and 64 units per layer. Figure shows the dependence of trained networks' modularity score (Q") as a function of regularization strength
each of three types of regularization: an L.2 penalty on the weights (weight decay), an Ll penalty on the weights, and dropout. The top row of Figure shows four example Á matrices sorted by cluster, to help give an intuition behind the quantitative values of Q". In these examples, the increasing value of Q" is driven by increasing contrast between intra-cluster similarity and inter-cluster similarity. In this example, it also appears that the namber and size of clusters remains roughly constant; this observation is confirmed by plotting the number of clusters versus regularization strength in Supplemental Figure S7
Figure 3 shows a number surprising patterns that contradict our initial predictions. First, and most saliently, we had predicted that dropout would reduce modularity, but found instead that it has the greatest effect on Q' among the three regularization methods we tried. This is especially apparent in the upstream methods (first two columns of the figure), and is also stronger for the first hidden layer than the second

---

Published in Transactions on Machine Learning Research (06/2022)
o-
basss
y
mocutomty store (0)
4
crapeut
0:2
0.2
0.2
256
0.1
0.1
0.1
0.1
256
0.0
0.0
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
7
meduarty sore-100 (2)
0.3
0.3
0.1
0.1
0.1
0.1
0.0
0.0
0.0
0.0
0.50
0.75
1.00
0.00
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
% regularization
95 regulantation
N
regularization
regulbiization
y
0.5-
0.6
Sco
0.6
0.6
0.5
0.5
0.5
0.5
64
mooubrtyy zcoee
0.3
0.3
0.3
0.3
64
0.2-
64
0.1
0.1
0.1
0.0
0.0
0.0
0.0
64
64
modjarty Sctre (0)
0.5-
0.5
0.5
0.5
7
0.4
0.4
0.4
0.3
0.3
0.2
0.2
0.2
0.2
0.9
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1:00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
%
regulartration
96
regulartration
%
regularizaaion
regulavization
Figure 4: Modularity score function of regularization strength for wider (gray background, top) and deeper (white background, bottom) models. Subplots are formatted identically to Figure
(Supplemental Figure S6 In general, Q" can increase either if the network partitions into greater number of clusters, or if the contrast between clusters is exaggerated. We found that this dramatic effect of dropout on Q" was accompanied by only minor changes to the number of clusters (Supplemental Figure S7), and so we can conclude that dropout increases Q' by increasing the redundancy of hidden units. other words, hidden units become more clusterable because they are driven towards behaving like funational replicas of each other, scparately for cach cluster. This observation ochoes, and may explain, why dropout also increases the "clusterability" of network weights in separate study (Filan et al., 2021).
The second surprising result in Figure 3 is that L2 regularization the weights did, in fact, increase Q", whercas we had expected it to have impact. Third, L1 regularization had surprisingly weak effect, although its similarity to the L2 regularization results may be explained by the fact that they actually resulted in fairly commensurate sparsity in the trained weights (Supplemental Figure S2 bottom row). Fourth, we had expected few differences between the eight different methods for computing similarity, but there appear be distinctive trends by similarity type both in Figure B as well as in the number of clusters detected (Supplemental Figure 57 The next section explores the question of similarity in the results in more detail.
10

---

Published in Transactions on Machine Learning Research (06/2022)
4.3 Generalization to wider and deeper networks (Experiments 2 and 3)
We repeated this analysis of Q+ scores and clusters, as a function of regularization strength, for both wider and deeper version of the fully-connected network used in Experiment 1. Modularity scores for these further experiments are shown in Figure 4 and analogous plots showing their numbers of clusters can be found in Supplemental Figure [S
While many of the qualitative trends remain, two differences to Experiment 1 are worth noting. First, inspecting the results for wider models, although we again see that dropout increases Q+, its effect relative to the other normalization types is greatly attenuated. In fact, in this wider architecture, we begin to see some evidence for our original hypothesis that increasing L1 weight regularization would increase modularity, at least with respect to downstream measures of similarity. However, this is a fragile effect, as too-large L1 regularization quickly overpowers learning and degrades performance (Supplemental Figure S3 Second, the deeper models in Experiment show an even more exaggerated difference between dropout and other regularization types, at least for upstream similarity measures. We further found that this effect is primarily driven by layers early in the network (see Supplemental Figures S9 and S10 for a breakdown by layer).
Another interesting effect revealed by Experiment 3 is that the effect of dropout Q' was highest in early layers for upstream similarity measures, but highest in later layers for downstream similarity measures (Supplemental Figure S9). As will be explored in more detail in the next section, this provides some initial evidence that the upstream- and downstrean-familis of approaches are measuring different things.
4.4 Comparing modules discovered by different similarity methods
The previous two scctions discussed idiosyncratic trends in the modularity scores Q' function of both regularization strength and how pairwise similarity wet units (S) is computed. However, such differences in the quantitative value of Q" are difficult to interpret, and would largely be moot if the various mot agreed on the question of which units belong in which cluster. We now turn to the question of how similar the cluster assignments P are across our eight definitions of functional modules. To minimize ambiguity, we will use the term ffunctionallsimillrttt"" to refer to S, and "cluster-similarity"" to refer to the comparison of different cluster assignments P across different functional-simiilrriyy methods. Note that all comparisons between cluster assignments investigated in this section are performed separately per layer per model cluster assignments are never compared for units across different models.
Quantifying similarity between cluster assignments is a well-studied problem, and we tested a variety of methods in the clusim Python package (Correia et al. 2018). All cluster-similarity methods we investigated gave qualitatively similar results, so here we report only the "Element Similarity" method of Gates et al., which is value between 0 and that is small when two cluster assignments are unrelated, and large when one cluster assignment is highly prodictive of the other. However, is conceivable that two cluster assignments do not strongly predict each other, but monetheless they give rise similar values of Q. That is, to the extent that Q has many good maxima a function of p*, differences between cluster assignments is not indicative of strong differences in the similarity structures g For this reason, we included an additional "transfer" test of cluster aligument: let, PA be the optimal cluster assiguments computed for the pairwise similarities A and likewise P for AB- Then,
Q(AA, Pj) + Q(Ann, Pi)
(9)
quantifies how well the cluster assignments P* transfer between similarity measures, transferring both B to A and vice versa. We refer to 9 as the "transferability" of clusters between two similarity measures.
Note that all of these mussterssimilari analyses are applied only to P' cluster assignments computed in the same layer of the same model. Thus, any dissimilarity in clusters that we see is due entirely to the different choices for functional-simiiarrty, S, rather than to differences in regularization, seed, architecture, etc.
Figure Ja summarizes the results of this cluster-similriity analysis in Experiment 1: there is striking difference between clusters of units identified by "upstrcam" functiinal-simillaiiy methods
5 We indebted an anonymous reviewer for pointing this out.
11

---

Published in Transactions on Machine Learning Research (06/2022)
a) Similarity of clusters averaged over layers, regularization, and runs
d) Change in similarity relative to
untrained model
hess
0.9
bess
0.8
0-90115
0.6
cov
S oon
0.1
-0.2
0.4
-0.3
b) Expt 2 (wide)
Expt 3 (deep)
Expt 2 (wide)
Expt (deep)
80
50
Figure 5: Summary cluster-siillrrity analysis. a) Separately for each model, layer, and run, we computed (H) chuster-similarity scores between cluster assignments P for each pair of the cight sumetitnal-smiilarity methods (S) above. This plot shows the average cluster-similarity for each pair of funetional-similarity methods in Experiment models. d) We repeated the analysis in (a) on set of 100 random (untrained) models. This panel shows the difference between (a) and the random models. Where the difference is positive (green), training had the effect of increasing chuste-simillrity, and where it is zero (black), there little difference in cluster- similarity before and after training. e-f) Same as (d) but for wider (Experiment 2) and deeper (Experiment 3) models. In all cases, the untrained reference models were of the same architecture as the trained model being analyzed. Note the different colorbar scale in panel (f).
b-c) Same (a) but for wider (Experiment 2) and deeper (Experiment 3)
12

---

Published in Transactions on Machine Learning Research (06/2022)
(S Srov, Si-sens, Si-sens) compared to "downstream" functional-simillrity methods (Shess, Shess, So-sens, So-sene). This analysis also reveals secondary structure within each class of upstream and downstream methods, where the choice to normalize not (S vs $) appears to matter little, and where there is a moderate difference between moment-based methods (S SShees) and gradient- based mothods (St-sens, So-seus). It is worth noting that some of this secondary structure is not robust across all lypes and levels of regularization; particular, increasing L. or L1 regularization strength appears to lead to (i) stronger dependence on normalization in the downstream methods, and (ii) a stronger overall agreement among the upstream methods (Supplemental Figure [SiIp. We found that qualitatively similar division between clusters identified using upstream and downstream similarity methods persisted in Experiments 2 and 3 (Figure pcc).
We next asked to what extent these cluster-similarity results are driven by training. As shown in Figure(ftlt much of the structure in the downstream methods is unaffected by training (i.c. it is present in untrained models as well), while the cluster-siiilarity among different upstream methods only emerged as result of training. Interystingly, despite the seemingly sharp differences between upstream and downstream met shown in Figure Ba-c, this analysis further shows that training tended to weakly increase the agreement between upstrcam and downstream methods.
Overall, these trends in alignment between clusters were confirmed in the "transfer" " analysis defined in equation 9 (Supplemental Figure $12
5 Conclusions
An abundance of "modular" designs in engincered and evolved systems has led many to speculate about the usefulness of modularity as design principle [Lipson] 2007), and how modular designs might be discovered by learning agents such as artificial neural networks (Amer & Maul, 2019). Yet, precisely defining what a "module" is in a neural network is an open problem. Here operationalized modules in a neural network as clusters of hidden wnits that perform similar functions, which leads naturally to the related question of what makes any given pair of unils funclionally similar? We introduced eight lunctional-similarii measures intended to cover a variety of intuitions about what makes two units similar, and empirically evaluated based on each of these eight methods in a large number of trained models. should be emphasized that our quantitative results on increased or decreased "modularity" refer to changes in Q", which may not correspond to the kind of system-level "modularity" that motivates work in this area. Our goal is not to establish an indisputable dcfinition of modularity, but to call attention to surprising ways that inluitions about modularity can be wrong.
One such surprising observation was that dropout increases modularity (as defined by Q") Filan et al, 2021), although this has little to do with the common-sense definition of a "module." Instead, it is the byproduct of dropout causing subsets of units to behave like near-copies of each other, perhaps so that if one unit is dropped out, a copy of it provides similar information to the subsequent layer. This effect was roduced, but not absent, in the "wide" model of Experiment 2. To our knowledge, this rodundancy-inducing effect of dropout has not been noted in the literature previously.
Our main result is that there is a crucial difference between defining "function" in terms of how units are driven by upstream inputs, and how units drive downstream outputs. While we studied this distinction between upstream and downstream similarity in the context of modularity and clustering, it speaks to the deeper and more general problem of how best to interpret neural representations. For example, some sub- disciplines
(e.g. "disentanglementt) have long emphasized that a "good" neural representation is one where distinct features of the world drive distinct sub-populations sub-spaces of neural activity (Higgins et al. 2018 Eastwood & Williams 2018 Ridgeway & Mozer 2018). This is an upstream way of thinking about what is represented, since it depends only on the relationship between inputs and the unit activations and does not take into account what happens downstream. Meanwhile, many have argued that the defining characteristic of a neural representation is its causal role in downstream behavior (Garson & Papincaul 2019); this is, of course, a downstrcam way of thinking. At high level, one way to interpret our results that upstream and downstream ways of thinking about neural representations are not necessarily aligned, even in trained networks. This observation is reminiscent. of recent. empirical
pegresentation-leaniin
13

---

Published in Transactions on Machine Learning Research (06/2022)
work finding that "disentangled" representations in auto-cncoders (an upstream concept) do not necessarily lead to improved performance or generalization to novel tasks (a. downstream concept) (Locatello et al. 2019 Montero et al. 2021 (cf. (van Steenkiste et al. 2019)).
Despite its theoretical motivations, this is an empirical study. We trained over 300 feedforward, fully- connected neural networks on MNIST. While is not obvious whether MNIST admits a meaningful "mod- ular" solution, Experiments and 3 suggest that our main result the misalignment between upstream and downstream definitions of neural similarity is fairly robust to width and depth. Still, it is plausible that upstream and downstream ways of thinking about neural representations will become better aligned in more structured tasks and larger models.
Our work raises the important questions: are neural representations defined by their inputs or their outputs? And, in what contexts beneficial for these to be aligned? We look forward to futurr work applying our methods to larger networks trained on more structured data, as well as recurrent networks. We also believe it will be valuable to evaluate the effect of attempling maximize modularity, training, to to what extent this is possible and whether it leads to performance benefits. Note that maximizing Q during training is challenging because (i) computing S may require large batches, and more importantly (ii) optimizing Q is highly prone to local minima, since neural activity and cluster assignments will tend to reinforce each other, entrenching accidental clusters that appear at the beginning of training. We suspect that maintaining uncertainty over cluster assignments (e.g. using soft Pi E [0, rather than hard P E 1] cluster assignments) will be crucial if optimizing any of our proposed modularity metrics during training.
we have defined it, during
Unequivoallly defining and quantifying the modularity of neural network remains open problem. How- ever, we hope that the distinction we introduced between upstream and downstream ways of thinking about modules, and about neural representations more generally, will foster and sharpen future work on modular neural network design.
14

---

Published in Transactions on Machine Learning Research (06/2022)
References
Mohammed Amer and Tomás Maul. A review of modularization techniques in artificial neural networks Artificial Intelligence Review, 52(1):527 561, 2019. ISSN 15737462. doi: 10.1007//110422-01999906666
URL
Jacob Andreas. Measuring compositionality in representation learning. arv, pp. 1 15, 2019. ISSN 23318422.
Farooq Azam. Biologicolly inspired modulor neural networks. Phd, Virginia Polytechnic Institute and State
University, 2000. URL http::/taaaaaa......r..rc//////t
184.kreep
replætype=pdf
Francis R. Bach and Michael I. Jordan. Kernel independent component analysis. Journal of Machine
Learning Research, 3(1):1 -4, 2003. ISSN 15324435. doi: 10. (0.1162/1132244303766660000
Francis R. Bach and Michael Jordan. Beyond independent components: Trees and clusters. Journal of Machine Learning Research, 4(7-8):1205-1223, 2004. ISSN 15324435. doi: 10.1162/jmlr (0.1162/jm1:2203.44777 200...47-8. 12205.
Shahab Bakhtiari, Patrick Mineault, Tim Lillierap Deepmind, Christopher C Pack, and Blake A Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. NewrIPS, 3, 2021. URL https://doi. org/ 10. 1100/1000011100 06. 18 448989
Gabriel Béna and Dan F. M. Goodman. Extreme sparsity gives rise to functional specialization. arXiv, 2021.
URL http://arxivvwaete///////// attp:://arxww..rg/abb//// 2106. 02626
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798-182) 1828, 2013. ISSN 01628828. doi: N0.1109/TPAM..2013..
C. Brandes, D. Delling, M. Gaertler, R. Gorke, M. Hoefer, Z. Nikoloski, and D. Wagner. On Modularity Clustering. IEFE Transactions on Knowledge and Data Engineering, 20(2):172-188, 2008. doi: 10.1109/ 190069900..90669..
Jeff Clune, Jean Baptiste Mouret, and Hod Lipson. The evolutionary origins of modularity. Proceedings of the Royal Sociely B, 280, 2013. doi: 20.1099/spb.2212..20
Rion Correia, Alexander Gates, Xuan Wang, and Luis M Rocha. Cana: A python package for quantifying control and canalization in boolean networks. Frontiers in physiology, 9:1046, 2018.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. Journal of Machine Learning Research, 13:795-825 2012. ISSN 15324435.
Róbert Csordás, Sjocrd van Steenkiste, and Jürgen Schmidluber. Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks. JCLR, 2021. URL httpt://aaaxiv.rrrrrr/// org/abs/
2010. 2010.02066]
Denker, D Schwartz, B Wittner, S Solla, R Howard, I Jackel, and J Hopfield. Large Automatic Learning, Rule Extraction, and Generalization. Complex Systems, 1:877 922, 1987.
Andrea Di Ferdinando, Raffaele Calabretta, and Domenico Parisi. Evolving Modular Architectures for Neural Networks. Proceedings of the sizth Neural Computation and Psychology Workshop: Evolution, Learning, and Development, pp. 253 262, 2001. doi:
Cian Eastwood and Christopher K.I. Williams. A framework for the quantitative evaluation of discntangled representations. ICLR, 2018
Daniel Filan, Stephen Casper, Shlomi Hod, Cody Wild, Andrew Critch, and Stuart Russell. Clusterability in Neural Networks. arXie, 2021. URL http://arxiv... wttp:o.////........... 03386
Justin Garson and David Papineau. Teleosemantics, Selection and Novel Contents. Biology & Philosophy, 34(3), 2019. doi:
15

---

Published in Transactions on Machine Learning Research (06/2022)
Alexander J. Gates, Ian B. Wood, William P. Hetrick, and Yong Yeol Ahn. Element-centric clustering
comparison unifies overlaps and hierarchy. Scientific Reports, 9(1):1-13, 2019. ISSN 20452322. doi: 00.1026//44159--11-445883 448922y. URL http://dx httt:://dx.doi.orge. (00..//rg/00.100///1110000009990
M. Girvan and M. F.J. Newman. Community structure in social and biological networks. Proceedings of the National Academy of Sciences of the United States of America, (9(1))78821 7826, 2002. ISSN 00278424.
doi: 10.1073 10.107//pas.. http:/aaaa............2 122653799.
Melvyn A. Goodale and A. David Milner. Separate visual pathways for perception and action. TINS, 15(1): 20-25, 1992.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with Hilbert-Schmidt norms. In S. Jain, H. U. Simon, and E. Tomita (eds.), Lecture Notes in Artificial Intelligence, volume 3734, pp. 63 63-77. Springer-Verlatt Berlin, 2005. ISBN 354029242X. doi: 10.1007/ 11564089 7.
Harold W Gutch and Fabian J Thcis. Independent Subspace Analysis is Unique, Given Irreducibility. In Mike E Davies, Christopher J James, Samer A Abdallah, and Mark D Plumbley (eds.), Independent Component Analysis and Signal Separation, volume 7. Springer, Berlin, 2007.
Trina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loie Malthey, Danilo Rezende, and Alexander Lerchner. Towards a Definition of Disentangled Representations. arXiv, pp. 1-29, 2018.
Shlomi Hod, Daniel Filan, Stephen Casper, Andrew Critch, and Stuart Russell. Quantifying Local Special- ization in Deep Neural Networks. arXiv, 2021. URL http://axxivv.rr/abss////s/ 08050
Aapo Hyvärinen, Patrik O. Hoyer, and Mika Inki. Topographic independent component analysis. Neural Computation, 13(7):1527 1558, 2001. ISSN 08997667. doi: (LL162/08897600015226699999
Robert A Jacobs, Michael I Jordan, and Andrew G Barto. Task Decomposition Through Competition in Modular Connectionist Architecture:The What and Where Vision Tasks. Cognitive Science, pp. 219 250, 1991.
Nadav Kashtan and Uri Alon. Spontaneous evolution of modularity and network motifs. Proceedings of the National Academy of Sciences of the United States of America, 102(39): 137788 2005. ISSN 00278424. doi: 14.073/ppas.. 0503610000.
Nadav Kashtan, Elad Noor, and Uri Alon. Varying environments can speed up evolution. Proceedings of the National Academy of Sciences of the United States of America, 104(34):13711 13716, 2007. ISSN 00278424. doi: 10.1073/pmas 001.mm0000113000....
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited. ICML, 36, 2019. URL http:///rxii. org/ htggabb//905500111 00414
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278 22248, 1998.
H Lipson. Principles of modularity, regularity, and hierarchy for scalable systems. Journal of Biological Physics and Chemistry, 7(4):125- -28,, 2007. ISSN 15120856. doi: 40.4024/40700..bpc.00.0
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, arXiv, pp. 1-33, 2019.
Milton Llera Montero, Casimir JJ Ludwig, Rui Ponte Costa, Guarav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalization. ICLR, 2021.
M. F.J. Newman. Modularity and community structure networks. Proceedings of the National Academy of Sciences of the United States of America, 103(23):8577-8552 2006. ISSN 00278424. doi: 10.1073/pnas 0601602103,
16

---

Published in Transactions on Machine Learning Research (06/2022)
M. E.J. Newman and M. Girvan. Finding and evaluating community structure in networks. Physical Review E Statistical, Nonlinear, and Soft Moller Physics, 69(2 2):1-15, 2004. ISSN 1063651X. doi: 10.1103/PIia wew.p/eeee....9000.0..
Jason A. Palmer and Scott Makeig. Contrast functions for independent subspace analysis. In Fabian J. Theis, A. Cichocki, A. Yeredor, and M. Zibulevsky (eds.), Independent Component Analysis and Signal Separation, volume LNCS 7191, pp. 00.1007/978-3--222 3-4---8551----66 15.
Springer-Verte Berlin, 2012. ISBN 9783642285509, doi:
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 1924-8035 Curran Associates, Inc., 2019. URL http::/ wew.aa-eeeaaaeettlleeeebeee deeperraaeeeeee--eeeeeeeeee
9015
Barnabés Póczos and András Lörinez. Independent Subspace Analysis Using Geodesic Spanning Trees. ICML, 22:673 680, 2005.
Karl Ridgeway and Michael C. Mozer. Advances in Newral Information Processing Systems, pp. 185 194, 2018. ISSN 10495258.
Learning deep disentangled embeddings with the F-statistic loss.
J. G. Rueckl, K. R. Cave, and S. M. Kosslyn. Why are "what" and "where" processed by separate cortical visual systems? A computational investigation. Journal of Cognétive Newroscience, 1(2):171-186, 1989. ISSN 0898929X. doi: 10. 111.66//jocn1199....1111
Bernhard Scholkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward Causal Representation Learning. Proceedings of the IEEE, 109(5): 612-66 63,, 2021. ISSN 15582256. doi: 10.1109/JPRCC200 0tt.:/CC..221.33558000
Herbert A Simon. The Architecture of Complexity. Proceedings of the American Philosophical Society, 106 (6), 1962.
O. Tange. Gnu parallel - the command-line power tool. :login: The USENIX Magazine, 36(1):42-44 -7, Feb 2011. doi: 10.5281 /zenodoz 16303. URL htttt:://///t -guu gww.gg.......................... ggu.orrg/s/paralll
Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disentangled representations helpful for abstract visual reasoning? Advances in Neural Information Processing Systems, 32(NeurIPS), 2019. ISSN 10495258.
Günter P. Wagner, Mihaela Pavlicev, and James M. Cheverud. The road to modularity. Nature Reviews Genetics, 8(12):921 931, 2007. ISSN 14710056. doi: 10.1038/nrg222
Chihiro Watanabe. Interpreting Layered Neural Networks via. Hierarchical Modular Representation. Com- munications in Computer and Information Science, 1143 CCIS:376-388, 2019. ISSN 18650937. doi: 10.1007/978----000366600 40.
Chihiro Watanabe, Kaoru Hiramatsu, and Kunio Kashino. Modular representation of layered neural net- works. Neural Networks, 77:62-73, 2018. ISSN 18792782, doi: ne..et//..nmmmt.20177...0
Chihiro Watanabe, Kaoru Hiramatsu, and Kunio Kashino. Understanding community structure layered neural networks. Neurocomputing, 367:84-102, 2019. ISSN 18728286. doi: 10.1016/J.neucom..211 2009..00.............. 0200..
Chihiro Watanabe, Kaoru Hiramatsu, and Kunio Kashino. Knowledge discovery from layered neural networks based on non-negative task matrix decomposition. IEICE Transactions on Information and Systems, F2033(2):390- 397, 2020. ISSN 17451361. doi: 10. 10..588/uuaamsinn22000000D77
17

---

Published in Transactions on Machine Learning Research (06/2022)
Zongze Wu, Chunchen Su, Ming Yin, Zhigang Ren, and Shengli Xie. Subspace clustering via stacked independent subspace analysis networks with sparse prior information. Pattern Recognition Letters, 146: 165 171, 2021. ISSN 01678655. doi: 10..010/j.patree2222.....8
18

---

Published in Transactions on Machine Learning Research (06/2022)
A Appendix
A.1
Algorithms
This section gives pscudocode for the algorithm we used to compute clusters P from the normalized matrix of pairwise associations between units, A. Before running these algorithms, we always remove all-zero rows and columns from A; we consider these units to all be in a separate "unused" cluster.
Algorithm Full clustering algorithm.
Require: Normalized pairwise associations Á
1: P f- GIRFDYSPPECTRAMMMOUU..S A)
Initialize P using spectral method > Further refine P using Monte Carlo method
2: P t- MONTECARLLMOODULESSA A, P)
3: return P
Algorithm 2 Pseudocode for greedy, approximale, spectral method for finding modules
1: function
2:
B + A A T
> B is analogous to the graph Laplacian, but for modules (Newman 2006 > Initialize P to a single cluster, which will be (recursively) split in two D FILO queue keeping track of which cluster we'll try splitting next Compute Q for the initial P
3:
P [00....0]
4:
queue - [0]
TT
while queue is empty do queue.pop()
Pop the next (lcftmost) cluster id
8:
+ indices of all units currently in cluster according to P
v + eig( ii) > Get the leading eigenvector of the submatrix of B containing just units in & Split v by sign (if not possible, continue loop)
10:
i + subset
where a was positive
11:
1 + subset of where v was negative
12:
d' < index the next available (all zero) column of P
13:
P' P but with all i- units moved to cluster c' > Try splitting into c, d' based on sign of v
14:
Q' + Tr(P"T BP')
Compute updated Q for newly-split clusters P' D Did splitting e help? b Update Q and P b Push e and d' onto the queue to consider further subdividing them
15:
if Q' > Q then
16:
-', queue (peundapp))d(c.e))
17:
later.
18:
else
19:
Nothing to do splitting into d' did not improve Q, so don't 't add further subdivisions to the queue, and we keep the old P, Q values
20:
end if
21:
end while
22: return P
> Once the queue is emply, P contains a good initial set of cluster assignments
23:
end function
19

---

Published in Transactions on Machine Learning Research (06/2022)
Algorithm 3 Pseudocode for Monte Carlo method for improving clusters.
function MONTECARLOOMODULESSAA.. P. for steps do
+ index of a single a randomly selected unit
c - index the first empty cluster in
Q" P t Tr(P (À 1 P).
Keep track of best Q, pair found far Try moving unit to each cluster j, including cluster at p' - P with i reassigned to cluster j
for j = c. do
Q + Tr(PPT (À 11,,
Compute updated Q with re-assigned unit D Update Q", P' pair, even if we don't select this j later
if Q > Q" then
P* + Jj P end end for whatever temperature makes p @ /T have entropy 0.15
6//r
We found H = 0.15 strikes good balance between exploration and greedy
Σ, S/
ascent 6
Sample new cluster assignment j from categorical distribution p P P with unit reassigned to cluster j", ensuring only the leftmost columns have nonzero values
end for return p end function
A.2 Supplemental Figures
Experiment (model)
L2 (weight decay)
L1 weight penalty
dropout prob.
1-gspace-66,11,9
0.0
0.0
Experiment 1 (MNIST (64, 64))
1e-5
logspace (-5,-2,7)
0.0
seeds per hyperparamcter
1e-5
0.0
linspace (0.05,0.7,14)
(respace(55,-11
0.0
0.0
Experiment 2
(MNIST (256, 256)) seeds per hyperparameter
1e-5
10gspace(55,22
0.0
1e-5
0.0
(0.,,,.e00.100....
logspace (-5,-1,5)
0.0
0.0
Experiment 3
(MNIST (64, 64, 64, 64, 64)) 3 seeds per hyperparameter
1e-5
1ogspace(-5,-21.
0.0
1e-5
0.0
Linspace(0.1,0.6,pg
Table S1: Models and hyperparameters. Number of units in each hidden layer given in parentheses in the first column, L.c "MNIST (64, 64)" is MIP with two hidden layers with 64 units in each layer. Each row of the table describes one hyperparaneter sweep performed for the corresponding model. L2 regularization was always set to minimum le 5 to avoid weights growing unboundedly (see Figures through for performance metrics and weight norms of trained models).
20

---

Published in Transactions on Machine Learning Research (06/2022)
Experiment 1
Experiment 2
Experiment 3
y
y
y
Key
64
256
64
Linear
ReLU
64
256
64
Dropout
7
7
64
64
64
7
Figure S1: Graphical depiction of architectures used in Experiments through 3.
21

---

Published in Transactions on Machine Learning Research (06/2022)
1.00
1.00
1.00
Train Set Accuracy
0.95
0.95
0.95
0.90
0.90
0.90
0.80
0.80
0.80
10-4
10-2
10-5
10-4
10-3
10
0.2
Test Set
Accuracy
0.95
0.95
0.95
0.90
0.90
0.90
0.85
0.85
0.85
0.80
0.80
0.80
10
10
10
10
10
10
0.0
0.2
0.4
0.6
3000
3000
3000
L2 norm
of weights
2000
2000
2000
1000
1000
1000
10-4
10-2
10-3
10-4
10-3
10-2
0.2
0.6
of weights
L1 norm
6000
6000
6000
2000
2000
2000
10
10
10
10
10
10
0.0
0.2
0.4
0.6
Nuclear norm of weights
400
200
200
200
10-4
10-2
10-3
10-4
10-3
10
0.0
0.2
0.4
0.6
Sparsity of weights
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
10
2
10
10
10
10
0.0
0.2
0.4
0.6
L2 regularization strength
L1 regularization
Dropout
strength
probability
Figure S2: Basic performance metrics as function of regularization strength for basic MNIST model (Experiment Table [S].. Each column corresponds to different regularization method, in Table SI Each row shows metric calculated on the trained models. Thin colored lincs are individual seeds, and thick black line is the average + standard error across runs. Horizontal gray line shows each metric computed randomly initialized network. Sparsity (bottom row) is calculated as the fraction of weights in the interval [-1e-3, +1e-3].
22

---

Published in Transactions on Machine Learning Research (06/2022)
1.00
1.00
1.00
train acc
0.95
train acc
0.95
train acc
0.95
0.90
0.90
0.90
0.85
0.85
0.85
0.80
0.80
0.80
10
10
10
10
10
10
0.0
0.2
0.4
0.6
12
11
drop
1.00
1.00
1.00
test. acc
0.95
test. ace
0.95
test acc
0.95
0.90
0.90
0.90
0.85
0.85
0.85
10-4
10-2
10-5
10-1
10-3
10
0.0
0.2
0.4
0.6
12
11
drop
1500
1500
1500
1000
1000
1000
12
12
500
500
500
10
10
10
10
10"
0.0
0.2
0.6
12
drop
11 m_nom
10000
11 P_nor
10000
11. P_nom
10000
5000
5000
5000
10-4
10-2
10-5
10-3
10
0.0
0.6
drop
1.00
1.00
1.00
0.75
sparsity
sparsity
sparsity
0.25
0.25
0.25
0.00
0.00
0.00
10-4
10-2
10-5
10-4
10-3
10
12
11
drop
600
600
600
nuc Norm
nuc_not nomm
nuc_non nomm
400
400
400
200
200
200
0
10-4
10-2
10
10-4
10-3
10
0.0
0.4
0.6
12
11
drop
Figure S3: Basic performance motrics as function of regularization strength for "wide" MNIST model (Experiment in Table ST) plotted in the same format as Figure 82
23

---

Published in Transactions on Machine Learning Research (06/2022)
1.00
1.00
1.00
train acc
0.95
train aacc
0.95
train acacc
0.95
0.90
0.90
0.90
0.85
0.85
0.85
0.80
0.80
0.80
10
10
10
10
10
10
0.0
0.2
0.4
0.6
11
drop
1.00
1.00
1.00
lest acc
0.95
test. acc
0.95
test acc
0.95
0.90
0.90
0.90
0.85
0.85
0.85
0.80
0.80
0.80
10-4
10-2
10-5
10-1
10-3
10
0.0
0.2
0.4
0.6
11
drop
1000
1000
1000
12. normm
12. P_.nrrm
12 P_norm
500
500
500
10
10
10
10
10
0.0
0.2
0.4
0.6
drop
4000
4000
4000
11. 1_nor
11. L_nomm
11.
2000
2000
2000
10-4
10-2
10-5
10-3
10
0.4
0.6
drop
1.00
1.00
1.00
0.75
0.75
0.75
sparsity
Sparsity
sparsity
0.25
0.25
0.25
0.00
0.00
0.00
10-4
10-2
10-5
10-4
-0
10
0.6
drop
300
300
300
nuc_nom noom
200
200
200
0
0
10-4
10-2
10
10
10-3
10-2
0.2
0.4
0.6
12
11
drop
Figure S4: Basic performance motrics as function of regularization strength for "doep" MNIST model (Experiment in Table S1,, plotted in the same format as Figure 82
24

---

Published in Transactions on Machine Learning Research (06/2022)
0.6
0.5
0 (spectral + Monte Carlo)
0 (spectral + Monte Carlo)
0.4
0.4
0.3
0.3
0.2
0.1
0.1
0.0
0.0
0.1
0.4
0.5
0.1 0.2
0.4
0.5
Q (spectral alone)
Q (Monte Carlo alone)
Figure $5: Both spectral initialization and Monte Carlo optimization steps contribute to finding good value of Q", Left: The x-axis shows modularity scores (Q") achieved using only the greedy spectral method for finding P The y-axis shows the actual scores we used in the paper by combining the spectral method for initialization plus Monte Carlo search. The fact that all points are on or above the y=x line indicates that the Monte Carlo search slep improved modularity scores. Right: The x-axis now shows modularity scores (Q*) achieved using 1000 Monte Carlo steps, after initializing all units into single cluster (we chose that were analyzed in the main paper to re-run for this analysis, which is why there are fewer points in this subplot than in the left subplot). The fact that all points or a above the y= line indicates that using the spectral method to initialize improved the search.
random 5% of the similarity-matrica
25

---

Published in Transactions on Machine Learning Research (06/2022)
S
la
Jless
heess
0.4
l1
0.4
0.4
First hidden layer h
modulariyy score
0.3
0.3
0.3
0.3
00
0.0
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
T-sems
getts
0.3
0.3
0.3
0.3
modulantyy
0.2
0.00
0.25
0.50
0,75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
regularization
regularization
regularization
regularization
cov
CUY
hress
Liess
0.4
0.4
0.4
0.4
score
Second hidden layer h²
0.3
0.3
0.3
0.3
0.0
0.0
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Leeens
issens
O-Sona
modularity (
0.4
0.4
0.4
0.4
0.3
0.3
0.3
0.3
0.0
0.0
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
regularization
regularization
regularization
reguuarrzztiin
Figure S6: Modularity (Q") versus regularization, split by layer. Format is identical to Figure 3 which shows modularity scores averaged across layers. Here, we break this down further by plotting each layer separately. The network used in our experiments has two hidden layers. The first two rows (white background) shows modularity scores for the first hidden layer h1, and the last two rows (gray background) shows h?
26

---

Published in Transactions on Machine Learning Research (06/2022)
S
4
Cay
hess
hess
num clusters
drop
First hidden layer h
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
10
1.0
8-
8-
num clusters
6-
6-
2
regularication
% regullrization
% regularication
% regularication
10
cow
heas
10
hess
num clusters
8-
8-
6-
6
Second hidden layer h
2-
0.00
num clusters
Vorey
2-
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25 regularization
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
regularization
regulbrization
regularizztion
Figure S7: Number of clusters in P versus regularization, split by layer. Layout is identical to Figure Gray shading in the background shows 10, 20, and 30 quantiles of number of clusters in untrained (randomly initialized) networks. Note that, for the most part, training has little impact on the number of clusters detected, suggesting that consistently finding on the order of 2-6 clusters more a property of the
56
MNIST dataset itself than of training.
We computed the number of clusters using equation relative size of the clusters.
This measure is sensitive both the number and
27

---

Published in Transactions on Machine Learning Research (06/2022)
50
DIV
50
50
less
50
Inee
Li
cropout
256
20
20
20
10
10
10
256
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
7
30
30
30-
30
20
0.75
1.00
0.25
0.25
0.50
0.75
0.25
regulatzaatio
regulariation
regularization
regulanization
S
Inss
Lhee
64
64
61
0.0
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
1.0
64
3-dens
64
scuutor
7
0.8
1.0
0.4
0.5
0.8
0.4
0.0
0.2
regularization
16 regulartzation
1: regularization
regularization
Figure S8: Number of clusters in P versus regularization, for wider (Experiment 2) and deeper (Experiment 3) models. Layout is identical to Figure S6
28

---

Published in Transactions on Machine Learning Research (06/2022)
0.6
-0-
0.6
cov
0.5
hxpts
0.6
0.4
0.4
0.4
0.4
02-
0.2
0.2
0.2
0.1
0.6
0.6
0.6
0.6
0.4
0.4
9.4
mobisrity
0.5
0.3
0.2
0.1
0.1
0.1
0.1
1
2
2
Layer
Layer
Layer
Layer
Figure S9: Modularity score versus depth in 5-layer feedforward network (Experiment 3), using the max- imum "good" regularization strength of each lype, defined as largest value for each lype of regularization that achieved at least 80% performance with at least half of the seeds. Gray shaded backgrounds show distribution of Q" values for untrained models; note that this null distribution itself depends on depth.
Sew
-O- 1
cot
S
C
Layer
Layer
Layer
Layer
Figure S10: Number of clusters versus depth in 5-layer feedforward network (Experiment 3), using the maximum "good" regularization strength of cach type, defined as largest value for each type of regularization that achieved at least 80% performance with at least half of the seeds. Gray shaded backgrounds show distribution of Q* values for untrained models; note that this null distribution itself depends on depth.
29

---

Published in Transactions on Machine Learning Research (06/2022)
Lam0%
1
1om50% 50%
3.0
10
0.3
6.4
08
L2 Regularization
614
1,-25%
1,-25%
6:2
01
64
84
-0.1
-12
1.00%
1.-29%
5. -666
L1 Regularization
3.5
-0%
666%
1,-83%
6.2
6.1
-2
8.3
00
6.8
4.0
0.1
-0.
-0.
drop-5%
d5%p=25%
drop-44%
dropwesM
Dropout regularization
07
-.6
0.5
0.4
dropm25%
drog-44%
Grop-65%
0.
6.35
-11
0.15
Figure S11: Further breakdown chuster-similarity by regularization strength (increasing left to right) and type (L2/L1/drop) /drrpouut. Results in Figure 5 reflect an average of the results shown here. The six rows of this figure should be read in groups of two rows: in each group, the top row shows the similarity scores (averaged over layers and runs), and the bottom
shows the difference to untrained models. A number of features are noteworthy here: (i) at low values of all three types of regularization, there is little cluster- similarity within the upstream methods, but it becomes very strong at as regularization strength grows; (ii) at the highest values L.2 and L1 regularization, the pattern inside the 4x4 block of downstream methods changes to depend more strongly on normalization; (iii) a moderate amount of agreement between upstream and downsttren ream methods seen for large L1 regularization strength, but curiously only for unnormalized downstream methods.
30

---

Published in Transactions on Machine Learning Research (06/2022)
a) Transfcrability of clusters averaged over layers, regularization, and runs
d) Change in transferability relative untrained models
cur
S hess
hess
0.10
S liess
S
0-90715
0.05
0.14
S Co-eeti
0.12
0.10
pow
pow
-0.05
1-90
0.04
b) Expt 2 (wide)
Expt 3 (deep)
Expt 2 (wide)
f) Expt (deep)
630
13
234
-006
Figure S12: Cluster-transfer control for the cluster-aligument analysis, in Figure in the main text. "Transferabilityy" is defined in equation (9). Note that transferability is not normalized, but is defined in raw units of Q. This is why the diagonals in panels (a)-(c) are not one.
31


Reviews:
[{"review_id": "DjuLOhP0Hw", "reviewer": "Reviewer_TQWy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* The paper is well written and the arguments are clear. The authors are generally upfront about the potential weaknesses of their approach. They also provide a good background on modularity literature and clarify that it is not a well-defined concept that everyone agrees on.\n* The observations regarding the increased modularity score with increased dropout as well as the difference between the functional structure of the downstream vs upstream analysis are interesting.\n\n\n### Weaknesses\n\n* The effect of the width is not considered. It is possible that width 64 is too narrow to see the full effect of L1/L2 regularization. It is known that the behavior of training MLPs can be qualitatively different between narrow and wide limits. \n\nBarring the above, the paper would be completely acceptable if the authors just talk about the functional structure of the latent space. However, it is the relationship between these concepts and modularity that is in my opinion the greatest weakness of the paper:\n* The authors admit that this concept is not well-defined, however they continue to make claims about increased/decreased modularity. If the idea is not well-defined and the authors admit that they do are not making an attempt at constructing a well-defined definition, how can these statements be made, especially since the different measures of their modularity score do not always agree with each other. \n* Given the vagueness of the notion of modularity, a comparison with at least one structural modularity measure would have been useful. Especially since it seems like the functional and structural modularity scores would behave completely opposite to each other in some cases. For example, in the submission, dropout increases the modularity score by increasing redundancy. However, I would suspect that increased dropout would decrease the structural modularity by decreasing the sparseness of the inter-layer weights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission is an empirical study looking at functional similarity of hidden units. Specifically, The authors define define a series of (mostly linear) functional similarity measures and study how the structure of these functional similarities change with L1, L2 and dropout regularization. The authors motivate this study by appealing to the admittedly vague notion of \"modularity\". The empirical study is carried out via training fully connected networks on the MNIST dataset.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. In order to make general claims, the authors need to show that their results are not sensitive to the width. \n\n2. Given the comments regarding the vagueness of the definition of modularity above, I would suggest that the authors _either_ add significant evidence in support of their definitions of modularity (see below) _or_ remove all claims, explicit or implicit, regarding the effect of the hyper-parameters on modularity. This would include renaming the modularity score to something like structural similarity score.\n\nIf the authors would like to keep their claims regarding the change of modularity intact, I would recommend bolstering their definitions of modularity. Some suggestions would be:\n* Comparison with some structural modularity measures and more robust arguments as to why it is the functional one that matters. (I am not sure if I agree with the statement that the structural modularity is useful only in promoting functional modularity. A structurally sparse network with a well tuned module system can act as a great inductive bias for learning and generalization.)\n* Proof of principle examples where the modularity measures are applied to networks (hand-tuned or trained) which are generally agreed to be modular. The authors could even make a where/what model by concatenating networks pre-trained to be sensitive to where and what etc. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SST92g0iDp", "reviewer": "Reviewer_fDiT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written, the methodology is clearly presented, and the authors do a good job at discussing various implications of design choices as well as connecting to different bodies of literature. The experimental design appears sound, and I don’t have doubts about the reproducibility of these results. \n\nIn my view the most significant finding is that up-stream and down-stream similarity measures yield very different modularizations. This is an interesting finding, but perhaps also somewhat strange given the simplicity of the task under consideration and the similarity of the metrics considered. I think it would be good to analyze this in more depth and I have some suggestions for this down below.\n\nThe findings regarding dropout, as well as L1 and L2 regularization are less significant in my view. I didn’t think that the hypothesis regarding L1 and L2 were very intuitive (since eg. it is unclear how much sparsity L1 actually induces), and even then, their effect on modularization is quite limited. Regarding dropout, perhaps one may conclude that the MLPs under consideration are biased towards having separate units fulfill only a single functionality, since the redundancy induced by dropouts does not smear out functionality, but rather creates duplicates. However, it seems premature to extend this claim beyond MNIST and the fixed MLP under consideration. \n\nThe limited setting is in general the main weakness of this work:\n\n1. only a single toy task is considered (MNIST) and it is not clear whether this admits a modular solution in the first place\n2. only a single network architecture is considered (3-layer MLP) and changes to the capacity of the model (depth or width) or other designs (eg. convolution based) are not considered\n3. only similarity of units within the same layer is considered\n\nRegarding (1) it is written that “While it is not obvious whether MNIST admits a meaningful “modular” solution, we expect that the main results we show here are likely robust”, however, no evidence is presented to support this claim.\n\nRegarding (3) it is written that “[...] in principle modules could be scored in the same way after concatenating layers together.”, but no such effort is undertaken.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a study of “functional modularity” in neural networks. Here functional modules are obtained through clustering similar functioning units within a layer, and it is studied how:\n\n1. the modules induced by 4 distinct similarity metrics (and their normalized versions) differ\n2. how L1 regularization, L2 regularization, and dropout affect modularity\n\nThe first contribution essentially focuses on the differences between up-stream similarity metrics and down-stream similarity metrics. The main finding is that metrics belonging to these different classes induce a different modularization, suggesting that finding modular representations that reflect the structure of the inputs may be different from learning modular representations that reflect output structure.\n\nThe second contribution essentially measures the effect of these different regularizers. The main finding here is that dropout encourages a “false sense” of modularity due to encouraging different units to fulfill the same role (as to mitigate the effect of dropout). The validates a finding that was made previously regarding dropout benefiting clusterability of units (Filan et al., 2021). Regarding L1 and L2 regularization it is found that both only impact modularization in subtle ways.\n\nThe current study is limited to 3-layer MLPs with two hidden layers of 64 units that are trained on MNIST for handwritten digit classification.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "### Changes critical to securing my recommendation for acceptance\n\nIncluding evidence to support the claim that “[...] the main results we show here are likely robust”. This should focus on conducting similar experiments (but perhaps for a reduced set of metrics, regularizers):\n\n* on different dataset (eg. CIFAR100 or ImageNet)\n* using different network architectures (eg. CNN or at least varying width/depth for MNIST)\n\nI strongly recommend exploring a more complex / different architecture also on a more complex dataset, since MNIST is so simple that adding more layers or making them wider (i.e. increasing capacity) probably won’t tell much about the effect of this in the general case (i.e. when having a network with the right amount of capacity for the task).\n\nSome more investigation into the observation that up-stream and down-stream similarity measures yield very different modularizations. Perhaps, if a similar finding is obtained on other datasets (eg. as per the above) this won’t be needed, but else I would recommend conducting the following analysis. Basically, given a clustering obtained under one metric A, evaluate its modularity score under another metric B and see how far this is from the optimal value that can be achieved when clustering based on B. This should tell one whether two very different clusterings also yield modularity scores that are far apart. Currently it may well be (but please correct me in case my reasoning is wrong) that there are many possible different clusterings having a similar modularity score (close to optimal) and while different clusterings are found for different metrics, this is mainly a consequence of ending up of comparing two arbitrary candidates within these pools.\n\n\n### Changes to strengthen the work\n\n* I think it would be very interesting to consider similarity between units in different layers. This won’t require any new training, but just re-analysis of existing MLPs.\n\n* To strengthen the results regarding L1 norm it may be useful to compare the sparsity of the layer directly with the achieved modularity score, i.e. verifying that L1 actually succeeds at making the layer more sparse and thus the hypothesis is correctly falsified.\n\n### Minor changes\n\n* In terms of presentation I may be better to remove table 1 to the appendix \n\n* It would be good to comment on why L2 regularization was used for the L1 sweep and the L2 sweep? Seeing as how L2 regularization does influence modularity, this may have impacted results slightly.\n\n* Regarding the conclusion about disentanglement not being useful for downstream tasks, please also see the opposite findings presented in Van Steenkiste, S., Locatello, F., Schmidhuber, J. and Bachem, O., 2019. Are disentangled representations helpful for abstract visual reasoning?. Advances in Neural Information Processing Systems, 32.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3QFAZqCJuO", "reviewer": "Reviewer_g9XM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The biggest strength of the paper is the systematic evaluation of different hidden unity similarity measures. This section was clearly motivated and well written.\n\nI think the weakness of this work is that it only studies a very simple task (MNIST) and only a single architecture which is also very simple (relatively shallow, fully connected, feedforward network). Thus, it is not clear how general the results and conclusions of this work are. Do they still hold for convolutional networks? Do other architectural differences that are important in practice (e.g. batch norm, residual / skip connections, or choice of optimizer) affect the results? Similarly, do the results change with very large and complicated image datasets? This is particularly concerning because MNIST is a small dataset with exceedingly simple image features.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates a simple empirical question: how modular are neural network representations? That is, to what extent do trained artificial networks contain \"clusters of similar-functioning units.\"\n\nThe authors propose several alternative similarity measures and use these in concert with simple graph theoretic methods to quantify modularity. A thoughtful review of prior work is provided. In my mind, the main advance seems to be that the authors used a broader set of hidden unit similarity measures than prior work, and that these different measures seem to lead one to different empirical conclusions in practice.\n\nI did not find any technical errors. As I explain below I think the biggest weakness of the work is that there are rather limited empirical characterization of modularity across different architectures and different tasks.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Though it is not critical to secure my recommendation, I would suggest that the authors add more experiments with (a) convolutional architectures, and (b) in more complex tasks (e.g. CIFAR-10 and CIFAR-100). This would increase the impact of this work by investigating the generalizability of their observations and conclusions. If further experiments will not be performed, I request that the authors add a few more sentences discussing this point at the end of their paper so that readers can take appropriate caution when interpreting how general the implications of this work are.\n\nOne final minor point -- can the results and experiments be clarified to explain whether any across-network comparisons are made in this work? Looking at Figure 1, it seems like all of the analyses compare hidden layer units within a single neural network -- i.e., hidden layer units are not compared across networks. Is this correct? If so, it may help to state this a bit more explicitly in section 4.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: With three reviewers recommending acceptance, I'm going to agree.

The primary criteria for including in TMLR is that the claims made in the paper are judged to be accurate and correct, for which I feel this review process helped with.  The reviewers had some concerns initially about the generality of the results and the authors responded with an extended set of experiments.  All three reviewers are now recommending acceptance after the changes.

In addition to the accuracy, I believe this paper is of interest to some in the TMLR community, in particular those who are interested in modularity.
Decision: Accept (poster)