Title: Neural Sculpting: Uncovering hierarchically modular task structure in neural networks through pruning and network analysis
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural networks

Hierarchical modularity

Pruning

Sparsity
Abstract: Natural target functions and tasks typically exhibit hierarchical modularity -- they can be broken down into simpler sub-functions that are organized in a hierarchy. Such sub-functions have two important features: they have a distinct set of inputs (input-separability) and they are reused as inputs higher in the hierarchy (reusability). Previous studies have established that hierarchically modular neural networks, which are inherently sparse, offer benefits such as learning efficiency, generalization, multi-task learning, and transfer. However, identifying the underlying sub-functions and their hierarchical structure for a given task can be challenging. The high-level question in this work is: if we learn a task using a sufficiently deep neural network, how can we uncover the underlying hierarchy of sub-functions in that task? As a starting point, we examine the domain of Boolean functions, where it is easier to determine whether a task is hierarchically modular. We propose an approach based on iterative unit and edge pruning (during training), combined with network analysis for module detection and hierarchy inference. Finally, we demonstrate that this method can uncover the hierarchical modularity of a wide range of Boolean functions and two vision tasks based on the MNIST digits dataset.

Neural Sculpting: Uncovering hierarchically modular task structure in neural networks through pruning and network analysis

Abstract

Natural target functions and tasks typically exhibit hierarchical modularity - they can be broken down into simpler sub-functions that are organized in a hierarchy. Such sub-functions have two important features: they have a distinct set of inputs ( input-separability ) and they are reused as inputs higher in the hierarchy ( reusability ). Previous studies have established that hierarchically modular neural networks, which are inherently sparse, offer benefits such as learning efficiency, generalization, multi-task learning, and transfer. However, identifying the underlying sub-functions and their hierarchical structure for a given task can be challenging. The high-level question in this work is: if we learn a task using a sufficiently deep neural network, how can we uncover the underlying hierarchy of sub-functions in that task? As a starting point, we examine the domain of Boolean functions, where it is easier to determine whether a task is hierarchically modular. We propose an approach based on iterative unit and edge pruning (during training), combined with network analysis for module detection and hierarchy inference. Finally, we demonstrate that this method can uncover the hierarchical modularity of a wide range of Boolean functions and two vision tasks based on the MNIST digits dataset.

1 Introduction

Modular tasks typically consist of smaller sub-functions that operate on distinct input modalities, such as visual, auditory, or haptic inputs. Additionally, modular tasks are often hierarchical, with simpler sub-functions embedded in, or reused by, more complex functions [1]. Consequently, hierarchical modularity is a key organizing principle studied in both engineering and biological systems. In neuroscience, the hierarchical modularity of the brain's neural circuits is believed to play a crucial role in its ability to process information efficiently and adaptively [2, 3, 4, 5]. Translating this hierarchical modularity to artificial neural networks (NNs) can potentially lead to more efficient, adaptable and interpretable learning systems. Prior works have already shown that modular NNs efficiently adapt to new tasks [6, 7, 8] and display superior generalization over standard NNs [9, 10, 11]. However, those studies assume knowledge of the task's hierarchy, or hand-design, modular NNs at initialization. Given an arbitrary task, however, we normally do not know its underlying sub-functions or their hierarchical organization. The high-level question in this work is: If we learn a task using a sufficiently deep NN, how can we uncover the underlying hierarchical organization of sub-functions in that task?

Recent studies through NN unit clustering have demonstrated that certain modular structures can emerge during the training of NNs [12, 13, 14, 15, 16]. However, it is unclear whether the structures extracted reflect the underlying hierarchy of sub-functions in a task. Csordas et al. [17] proposed a method that identifies sub-networks in NNs that learn specific sub-functions. This method however,

requires knowledge of the exact sub-functions within a task's hierarchy. Ideally, modules corresponding to specific sub-functions should emerge through a training strategy, and a method should be available to detect these modules without explicit knowledge of the corresponding sub-functions.

Biological networks exhibit hierarchical modularity where clusters of nodes with relatively dense internal connectivity and sparse external connectivity learn specific sub-functions [4, 5, 18, 19]. Drawing inspiration from this, we propose Neural Sculpting , a novel approach to train and structurally organize NN units to reveal the underlying hierarchy of sub-functions in a task. Within a hierarchically modular task, we consider sub-functions that are input-separable and reused. We first show that conventionally trained NNs do not acquire structural properties that reflect the previous sub-function properties. To address this, we introduce a sequential unit and edge pruning method to train NNs. Unit pruning first conditions the NN to learn reused sub-functions, while edge pruning subsequently reveals the sparse connectivity between the learned sub-functions. Finally, we propose a network analysis tool to uncover modules and their hierarchical organization within the sparse NNs.

We demonstrate the capability of Neural Sculpting to reveal the structure of diverse hierarchically modular Boolean tasks and two MNIST digits based tasks. To the best of our knowledge, this paper is the first to analyze specific sub-function properties within a hierarchically modular task and propose an end-to-end methodology to uncover its structure. This work also sheds light on the potential of pruning methods to uncover and harness structural properties in NNs.

1.1 Preliminary

To represent the decomposition of a task, we visualize it as a graph. Initially, we consider Boolean functions and their corresponding function graphs. A Boolean function f : { 0 , 1 } n →{ 0 , 1 } m , maps n input bits to m output bits. We define the set of gates G as {∧ , ∨ , I } , where ∧ represents logical conjunction (AND), ∨ represents logical disjunction (OR), and I represents the identity function (ID). Additionally, we define the set of edge-types E as {→ , ¬} , where → represents a transfer edge and ¬ represents a negation edge.

Function Graph: A ( G,E ) -graph representing a Boolean function f : { 0 , 1 } n → { 0 , 1 } m is a directed acyclic graph comprising: n sequentially ordered vertices with zero in-degree, designated as the input nodes ; k vertices with non-zero in-degree, designated as the gate nodes , with each vertex associated with a gate in G and each of its in-edges associated with an edge-type in E ; and m sequentially-ordered vertices with zero out-degree, designated as the output nodes .

We use the G and E defined above, as NN units have been demonstrated to learn these universal gates [20]. Function graphs, which break down complex computations into simpler ones, are typically sparse. Modularity in sparse graphs refers to the structural organization of subsets of nodes that exhibit strong internal and weak external connectivity, which are grouped into modules.

Sub-function: A sub-function is a subset of nodes within the function graph that collectively perform a specific task. The sub-function is characterized by having strong internal connectivity within its nodes, meaning that they are highly interdependent and work together to achieve the desired output. At the same time, nodes in the sub-function have weak external connectivity, indicating that they are relatively independent from the rest of the graph.

Input Separable: Two sub-functions are considered input separable if their in-edges originate from distinct and non-overlapping subsets of nodes in the function graph.

Reused: A sub-function is reused if it has two or more outgoing edges in the function graph. 1

Training NNs on target Boolean functions: We start by obtaining the truth table of each function graph, which serves as our data source. The training set ( X t , Y t ) includes the complete truth table with added random noise ( N (0 , 0 . 1) ) during each iteration to increase the number of training samples. The validation set ( X v , Y v ) consists of the noise-free rows of the truth table. We use multi-layered perceptrons (MLPs) with ReLU activation for the hidden units and Kaiming weight initialization to learn the Boolean functions. The loss function is bitwise cross-entropy with Sigmoid activation. The NNs are trained using Adam optimizer with L2 regularization of 1 e -4 .

2 Standard training of NNs is not enough

We show that NNs through standard training, do not acquire structural properties that reflect the properties of sub-functions. Specifically, we consider the two properties in isolation by constructing two graphs: one with input separable sub-functions, and the other with a reused sub-function.

2.1 Input Separable

The first function graph we consider has 4 input nodes and 4 output nodes. The output nodes { y 1 , y 2 } depend only on { x 1 , x 2 } , while the output nodes { y 3 , y 4 } depend only on { x 3 , x 4 } (Figure 1a). We train various NN architectures to learn the target function with perfect accuracy on the validation set. The property of the function graph that reflects input-separable sub-functions is that there are no paths from input nodes { x 1 , x 2 } to output nodes { y 3 , y 4 } and from input nodes { x 3 , x 4 } to output nodes { y 1 , y 2 } . However, in a neural network, all input units are connected to all output units

Input separablesub-functions

Edge Weight Product of paths dense NN

Architecture:[4,48,48.48,4]

- 1000

593.07

873.50

721.21

818.03

- 950

- 900

n ndui

N

571.15

841.27

697.88

789.90

850

800

m

649.16

963.20

817.90

925.28

750

700

650

674.72

1000.26

849.64

960.89

600

X1

ZX

X3

X4

2

OutputUnit

3

t-statistic :0.44, p-value :0.66

OR

AND→Transfer

Negation

t-statistic :-0.68, p-value :0.27

through the same number of paths. Therefore, we analyze the strength of their learned relationship by considering the product of weight magnitudes along those paths.

Consider a trained NN parametrized by θ ∈ R a and the set of all paths, P . Each edge weight θ i is assigned a binary variable p i to indicate whether it belongs to a given path p ∈ P . We define the edge-weight product of a path as π p = ∏ a i =1 | θ p i i | , and π ( i, j ) = ∑ p ∈ P i → j π p represents the sum of π p for all paths from input unit i to output unit j . We evaluate π ( i, j ) for i = 1 , 2 and j = 3 , 4 , and π ( i, j ) for i = 1 , 2 and j = 1 , 2 . If the former is significantly smaller than the latter, then the input units 1 and 2 are not used by output units 3 and 4 .

We perform a two-sample mean test with unknown standard deviations, with ( µ 1 , s 1 ) representing the mean and sample standard deviation of π ( i, j ) for i = 1 , 2 and j = 3 , 4 , and ( µ 2 , s 2 ) representing the mean and sample standard deviation

of π ( i, j ) for i = 1 , 2 and j = 1 , 2 . The null hypothesis H 0 is µ 1 = µ 2 , and the alternative hypothesis is µ 1 < µ 2 . A similar test is performed for input units 3 and 4 . Figure 1b shows an example heat map for π ( i, j ) , where i, j ∈ { 1 , 2 , 3 , 4 } . For the example shown we cannot reject the null hypotheses. Similarly, we conducted statistical tests on nine different NNs with varying architectures and seed values, and the results are summarized in Table 1. For NNs with a single hidden layer, we can reject the null hypotheses, but for deeper NNs, both the null hypotheses cannot be rejected.

2.2 Reused

Consider the function graph shown in Figure 2a, consisting of 4 input nodes and 16 output nodes. First, we construct an intermediate sub-function g ( X ) such that the three gate nodes depend on all the inputs. This sub-function is utilized by two separate gates, f 1 ( g ) and f 2 ( g ) , which are then used 8 times by different output nodes. All paths to the output nodes pass through three gate nodes in the first hierarchical level and two gate nodes in the second. We analyze the edgeweight product of the paths from hidden units to output units. Let π l p denote the sum of π p for all paths originating from hidden layer l , which contains N l hidden units. We compute

Reusedsub-function

Number of hidden units that form

90% of edge weight product

40

35

Number of units

3025 2015

10

X1

OR

AND

ID

0

8

12

1620

24

36

48

→

Negation

NN width

the minimum number of units, N l P , that are necessary to achieve P % of the total π p . By comparing N l P to that of the function graph, we can conclude whether the NN has learned the reused states.

We independently trained NNs with two hidden layers and different widths on the target Boolean function. The resulting N l 90 values for different layers are shown in Figure 2b. Our analysis indicates that, as the width of the NN increases, so does N l 90 in both the hidden layers, and these values are consistently close to the actual width of the NN. We trained an NN with hidden layers of width 3 and 2, respectively, to confirm that NNs with those widths can learn the function well.

3 Iterative pruning of NNs

In the previous section, we demonstrated that standard NN training is inadequate for learning structural properties that reveal input separable or reused sub-functions. Nevertheless, we observed that NNs with a relatively low number of parameters could still acquire these properties. Since hierarchical modularity is a property of sparse networks, we explored NN pruning algorithms as a means of achieving this sparsity. Prior research has shown that pruning NNs can reduce their number of parameters without compromising their performance [21, 22, 23]. Edge pruning methods typically require specifying a target NN density or pruning ratio [21, 24], but determining a density value that preserves the NN's performance remains an open question. Recently, iterative magnitude pruning [25, 26] has emerged as a natural solution to this problem. The algorithm prunes edges iteratively, removing some edges in each iteration and then retraining the NN. This process can be repeated until the NN achieves the same validation performance as the dense NN while being as sparse as possible.

3.1 Iterative edge pruning

Consider the initial edge pruning step p e . We train the dense NN and prune p e % of the edges with the lowest weight magnitude. The sparse NN resulting from this pruning is then trained with the same number of epochs and learning rate schedule as the original NN. We repeat this process until the sparse NN can no longer achieve the same validation accuracy as the dense NN.

At this point, we rewind the NN to the previous sparse NN that achieved the same validation accuracy as the dense NN and update the value of p e as p e = p e / 2 . We repeat this process until p e becomes lower than the required step to prune a single edge, thereby ensuring that the lowest possible density is achieved.

We next apply the tests developed in the previous section to analyze whether edge-pruned NNs acquire structural properties resembling those of the subfunctions. Specifically, we train and

Edge Weight Product of paths edge pruned NN

Number of hidden units in edge pruned NN that

Architecture:[4,48,48,48.4]

- 70

form90%ofedge weightproduct

54.16

54.16

0.00

0.00

- 60

6

Input Unit

56.76

56.76

0.00

0.00

50

40

units

5

m

0.00

0.00

72.42

72.42

OE -

Number

3

20

Z

0.00

0.00

70.25

70.25

10

1

2

=m

0

1

t-statistic :-73.86, p-value :0.0

OutputUnit

0

8

12

16

20

24

36

48

t-statistic :-113.88, p-value :0.0

NN width

prune different NN architectures to learn the function graph with input-separable sub-functions. For the example shown in Figure 3a, as well as for NNs with different architectures, we can reject the null hypotheses in favor of the alternate hypotheses. This implies that edge-pruned NNs acquire structural properties that enable them to recognize input-separable sub-functions.

Consider the previous function graph that has reused sub-functions and NNs with increasing widths. We independently train and prune each NN architecture and determine the number of units N l 90 (Figure 3b). We find that in a majority of the trials, N 2 90 = 2 , which are reused by the output units. Although there is a significant reduction in N 1 90 , the edge-pruned NNs fail to identify 3 units that are being reused. We observe that edge-pruned NNs identify sparse connectivity between units and reuse those units. However, they do not identify units corresponding to sub-functions that are not sparsely connected yet reused. We hypothesize that this may be due to the initial pruning iterations where some edges from all units are removed, leaving no hidden units with dense connectivity to learn densely connected and reused sub-functions. Additionally, this could also be due to very large NN widths and the absence of an objective conditioning NNs to utilize as few hidden units as possible. Therefore, next we introduce an iterative unit pruning method to limit the number of units used.

3.2 Iterative hidden unit pruning

To prune hidden units, we first train the neural network and then assign each hidden unit a score. We eliminate p u % of the hidden units with the lowest scores and train the resulting pruned network for the same number of epochs and learning rate schedule as the original network. We repeat this two-step pruning process iteratively.

We use loss sensitivity as the scoring metric for the units [27, 28, 29, 30], which approximates the change in the network's loss when the activation of a particular hidden unit is set to zero. Specifically, for a unit i in hidden layer l , with activation a l i , the score is computed as :

The iterative unit pruning process continues until the NN can no longer achieve the same validation accuracy as the original NN. To ensure that we have pruned as many units as possible, we revert to the latest NN that achieved the same validation accuracy as the original NN and halve the value of p u .

This process is repeated until the unit pruning step becomes lower than the step size needed to prune a single unit. We perform unit pruning before the edge pruning process to identify the minimum widths required in each hidden layer. Once the minimum widths are determined, edges are pruned to reveal the sparse connectivity between those units. For additional details please refer to appendix section H.

We conducted the previous tests to analyze whether the unit-edge pruned NNs acquire structural properties resembling

Edge Weight Product of paths unit-edge pruned NN

Architecture:[4,48,48,48,4]

Number of hidden units in unit-edge pruned NN

thatform90%ofedgeweightproduct

48.95

48.17

0.00

0.00

- 60

3.0

- 50

Input Unit

49.71

48.91

0.00

0.00

40

units

2.5

30

2.0

m

0.00

0.00

65.11

66.11

20

er

1.5

0.00

0.00

60.85

61.79

10

1.0

2

=m

4

Q

0.5

Hidden Layer1

Hidden Layer2

t-statistic :-156.11, p-value :0.0

OutputUnit

0.0

8

12

16

20

24

36

48

t-statistic :-49.98, p-value :0.0

NN width

those of the sub-functions, as presented in the previous section. The results of these tests are shown in Figure 4, and show that the unit-edge pruned NNs do acquire structural properties resembling both input separable sub-functions and reused sub-functions.

A.PrunedNN

B. Layerwise unit clustering

C.Sequentialmergingof clusters

B1.Outgoing connectivity

B2.Clusters layer 1

B3.Clustering output

C1.Connectivity overlap

C2.Merged layers1&2

C3.Modulesdetected

4 Detecting modules within sparse NNs

In this section, we propose a method to uncover sub-networks or modules that approximate the sub-functions within the target function. We approach this by projecting the problem as a two-step partitioning problem due to the layered structure of NNs. First, we cluster the units belonging to the same layer. We assume that given a layer, there exist various subsets of units that participate in learning the same sub-function. Next, we merge the previous unit clusters across layers such that they exhibit strong connectivity. The overview of our proposed method is illustrated in Figure 5.

Layer-wise unit clustering: Let us consider a single layer l with N l units. For each unit, we construct a feature vector based on its outgoing connectivity. The feature vector for a unit i is a binary vector f l i ∈ { 0 , 1 } g , where g is the total number of units in all the later layers. If unit j is connected to unit i through at least one directed path, then f l i ( j ) = 1 , otherwise f l i ( j ) = 0 . Our hypothesis is that the units that participate in learning the same sub-function are reused by similar units in the later layers. To partition the units into clusters such that their feature vectors have low intra-cluster and high inter-cluster distances, we use the Agglomerative clustering method with cosine distance and

average linkage. To identify the optimal number of clusters K l , we use the modularity metric, which we have modified for our problem [31, 32, 33].

Let ˜ D be the normalized distance matrix of the unit feature vectors, where the sum of all elements is equal to 1 . Consider the units partitioned into k clusters and a matrix A ∈ R k × k , where A ij = ∑ a ∈ C i ,b ∈ C j ˜ D ab represents the sum of the distances between all pairs of units in clusters C i and C j . The modularity metric, denoted by M , measures the quality of the partitioning and is defined as:

The first term in this equation measures the total intra-cluster distance between the unit features while the second term is the expected intra-cluster distance under the null hypothesis that unit distances were randomly assigned based on joint probabilities in ˜ D . A negative value of M indicates that pair-wise distance within a cluster is lower than the random baseline. We iterate through values of k ranging from 2 to N l -1 , which are obtained through Agglomerative clustering, and select the value of k that minimizes the modularity metric.

The modularity metric can accurately detect the presence of multiple unit clusters ( K l = 2 , ..., N l -1 ). However, it fails for the edge cases where all units may belong to a single cluster ( K l = 1 ) or to separate individual clusters ( K l = N l ). To address those, we conduct a separability test if the modularity metric is lowest for k = 2 or k = N l -1 , or if the modularity metric values are close to zero. Modularity metric close to zero is an indicator that the intra-cluster distances are not significantly different from the random baseline. To determine this, we set a threshold on the lowest modularity value obtained.

Separability test: The unit separability test is designed to evaluate whether two units in a cluster can be separated into sub-clusters. Consider two units i and j , with o i = ∑ f l i , o j = ∑ f l j neighbors respectively, and o ij = f l i glyph[circledot] f l j common neighbors. We consider a random baseline that preserves o i and o j . The number of common neighbors is modeled as a binomial random variable with g trials and probability of success p = o i × o j g 2 . The units are separable if the observed value of o ij is less than the expected value E ( o ij ) under the random model.

Consider the partition where N l -1 clusters are obtained. If the two units that are found in the same cluster are separable, it implies that all units belong to separate clusters. Now let us consider the partition of units into two clusters. We merge the feature vectors of the two unit groups. If the two groups of units are not separable, it implies that all units must belong to the same cluster. In some cases, both tests yield positive results. We determine the optimal number of clusters by selecting the result that is more statistically significant.

Merging clusters across layers: Strongly connected clusters from adjacent layers are next merged to uncover multi-layered modules. Consider, C i l , i = 1 , 2 , ..., K l to be the clusters identified at layer l . Let e l i,j be the number of edges from cluster C i l to cluster C j l +1 . The two clusters are merged if :

The output units are merged with the previous layer's modules, ensuring that δ m fraction of incoming edges to the unit are from that module. This allows multiple output units to be matched to the same structural module.

Input separable sub-function

Reused sub-function

Input separable andreused

Single sub-function

X1

OR

AND

ID

Transfer

Negation

5 Experiments and Results

5.1 Modular and hierarchical Boolean function graphs

In this section, we conduct experiments on Boolean function graphs with different sub-function properties to validate our pipeline. We begin by testing the pipeline on four function graphs shown in Figure 6. These graphs include: 1) input separable sub-functions, 2) a reused sub-function, 3) sub-functions that are both input separable and reused, and 4) a function graph without any such sub-function where all nodes are strongly connected to every other node.

We perform 36 trials for each function graph by training neural networks with combinations of 3 width values, 3 depth values, and 4 seed values. A trial is considered successful if the proposed pipeline detects a module corresponding to an input separable or reused sub-function (Figure 5). For Boolean functions, we set the modularity metric threshold to -0.2 and the

Successrate:uncoveringmodules

corresponding tosub-functions

1.0

0.8

Success rate

0.6

0.4

1hiddenlayer

0.2

2hiddenlayers

3hiddenlayers

0.0

input

reused

both

none

separable

cluster merging threshold to 0.9. Figure 7 shows the success rates for each function graph and NNs with different depths. We observed that the proposed pruning and module detection pipeline has a high success rate when the depth of the NN exceeds that of the function graph. (see to appendix section B for NN visualizations)

5.2 Sub-functions that are reused many times are uncovered more accurately

We consider the function graph shown in Figure 6b, which contains a single reused sub-function. We vary the number of times the two intermediate gate nodes in the second hierarchical level are used by decreasing the number of output nodes and measure the success rate. The results are shown in Figure 8, where we observe an increasing trend in the success rate as the number of output nodes using the two gate nodes increases. As the number of output units using the two gate nodes decreases, learning the two gate nodes using the previous "dense" subfunction may not be efficient. We provide visualizations of the corresponding NNs in the appendix section C. Note that NNs with only one hidden layer recover a single module, as the function graphs require four hierarchical levels. If only three hierarchical levels are available, the function graph collapses to a dense graph.

Successrate:reusedsub-function

1.0

0.8

rate

Successr

0.6

0.4

1hiddenlayer

0.2

2hiddenlayers

3hiddenlayers

0.0

1

2

4

6

8

OR

AND

ID

Transfer

Negation

5.3 Sub-functions with higher input separability are detected more accurately

In this experiment, we consider a function graph with two input separable sub-functions shown in Figure 9a. We increase the overlap between the two separable input sets by decreasing the total number of input nodes and reusing them. We also vary the number of times each sub-function is replicated or used in the output nodes. Our goal is to uncover three properties of the structure: accurately separated input units into sub-function specific and reused, two output sub-functions accurately detected in later layers, and all hidden units belong to either of the two output modules. The success rate for each of these properties as a function of input overlap and sub-function use is shown in Figure 10.

input

nodes

Overlapping

N

m

0.97

1.00

0.64

0.92

0.94

Success rate:Input separability

1

1.00

1.00

0.72

0.94

1.00

2

1.00

1.00

0.78

1.00

1.00

-4

1.00

1.00

1.00

0.89

1.00

- 6

1.00

1.00

1.00

0.86

0.89

8

- 1.00

- 0.95

- 0.90

0.85

0.75

0.70

0.80

-0.65

input

nodes

ddel

g

m

寸

0.00

1.00

1.00

1.00

0.00

Success rate: Hidden units separability

1

1.00

1.00

1.00

0.00

0.00

2

1.00

1.00

1.00

0.03

0.06

4

1.00

1.00

1.00

0.06

0.17

1.00

1.00

1.00

0.06

0.17

8

- 0.8

- 1.0

0.6

0.2

0.4

0.0

qndu

pou

Overlapping

6

N

m

1

0.97

0.58

0.25

1.00

1.00

Success rate:Output sub-function

1

1.00

1.00

1.00

0.58

0.31

2

1.00

0.69

0.44

1.00

1.00

4

0.53

1.00

1.00

1.00

0.67

6

1.00

1.00

1.00

0.72

0.61

8

- 1.0

-0.4

0.9

0.8

0.7

0.6

0.5

0.3

Number of times used

Number of times used

Number of times used

We observe that our method has a high success rate for detecting these properties for sub-functions with high input separability. However, as the overlap between input node sets increases, the success rate for detecting input modules decreases. Furthermore, the success rate decreases as the number of times a sub-function is used decreases. We also observe that for sub-functions with low input separability (and high input overlap), intermediate units are often clustered into a single module due to the hidden units pruning step. Finally, we find that the same trend is observed for detecting output sub-functions, where increasing input overlap and decreasing sub-function use results in a low success rate for our method. (See appendix section D for visualization of these results).

6HierarchicalLevels

5Hierarchical Levels

4HierarchicalLevels

OR

AND

ID

→Transfer

Negation

5.4 Hierarchical structures uncovered vary depending on NN depth

The function graph in Figure 11a consists of two input separable sub-functions, the output of those sub-functions is then used by a single intermediate sub-function. This intermediate sub-function is then reused by two additional sub-functions to produce the final output. Interestingly, we observe that the proposed pipeline uncovers different hierarchical structures depending on the depth of the NN.

Figure 12a shows the success rate of uncovering the specific subfunctions segregated by the NN depth. We find that NNs with depth greater than or equal to the number of hierarchical levels (5) can uncover all three types of sub-functions in the function graph. However, for NNs with lower depth, only the input separable sub-functions are uncovered, while the intermediate and output sub-functions are merged into a single module.

Successrate:uncoveringmodules

Successrate:detectingallmodules

corresponding to sub-functions

corresponding to NN and hierarchical depth

1.0

1.0

0.8

0.8

Successrate

Success rate

0.6

0.6

0.4

0.4

2hiddenlayers

2hiddenlayers

0.2

3hiddenlayers

0.2

3hiddenlayers

4hiddenlayers

4hiddenlayers

0.0

input

single

output

0.0

2

4

6

8

separable

intermediate

Number of times used

This result highlights the importance of selecting an appropriate depth for the NN architecture to effectively uncover hierarchical structures in Boolean function graphs. (see appendix section E for NN visualizations) In addition, we report the success rates of uncovering the exact hierarchical structure that corresponds to the depth of the NN as we vary the number of times the output sub-functions

were used (Figure 12b). The findings demonstrate an increasing trend in the success rate, which is consistent with our previous results.

5.5 Modular and hierarchical functions with MNIST digits

In this section, we present an experimental evaluation of hierarchically modular tasks constructed using the MNIST handwritten digits dataset. We set the modularity metric threshold to -0.3 and the cluster merging threshold to 0.6 for tasks with MNIST digits.

Classifying two MNIST digits: We begin by considering a simple function constructed to classify two MNIST digits. The NN is presented with two images concatenated together, and the output is a vector of length 20. The first 10 indices of the output predict the class for the first image, and the remaining 10 predict the class for the second image. We construct the dataset such that each unique combination of labels has 1000 input data points generated by randomly selecting images. We split the data into training and validation sets with an 8:2 ratio, and then train nine neural networks with varying widths (392, 784, 1568) and number of hidden layers (2, 3, 4) to learn the function for four different seed values. We use the Adam optimizer with bit-wise cross-entropy as the loss function, and select 99% as the accuracy threshold for the pruning algorithm, as all dense NNs achieve at least 99% validation accuracy. Out of the 36 trials conducted, we obtain two completely separable modules for 33 out of 36 trials, indicating the high effectiveness of our approach. (see appendix F for NN visualizations)

Hierarchical and Modular MNIST Task: We consider next a hierarchical and modular task that uses the MNIST digits dataset (Figure 13a). The task takes two MNIST images as input, each belonging to the first 8 classes (digits 0-7). The digit values are represented as 3-bit binary vectors and used to construct three output Boolean sub-functions. To generate the dataset, we randomly select 1000 input data points for each unique combination of digits. The data is then split into training and validation sets with an 8:2 ratio. All the NNs that we experiment with reach a validation accuracy of 98% , which we use as the accuracy threshold for the pruning algorithm.

To analyze the modular structure uncovered by our methodology, we divide the success rates into three categories: (1) detecting two input separable modules, (2) detecting three output modules, and (3) middle layer unit-separability into either the input separable modules or the output separable modules. We observe that the NNs uncover the three output modules with high success rates. However, for NNs with lower depths, the NNs fail to recover the two input separable modules, and all the units in the early layers are clustered into a single

Successrates:Hierarchicaltask

1.0

0.8

Successrate

0.6

0.4

4

Inputseparablesub-function

0.2

Outputsub-functions

Middlelayersseparability

0.0

2

3

4

OR

AND→Transfer

→Negation

Numberofhiddenlayers

module. As the depth of the NN increases, we find that the input separable modules are recovered with high accuracy as well. We observe that the success rates for middle layer unit-separability are very low, even when the depth of the NN increases. The units belonging to those hidden layers are often clustered into a single module. These units may be learning representations required to approximate the two digits well. This finding indicates that the NN depths we experiment with may not be sufficient to capture the underlying function, despite the NNs learning the task with high validation accuracy. Please refer to appendix F for NN visualizations.

6 Related work

Emerging modular structures in trained NNs has been a subject of prior investigation, primarily through unit clustering algorithms. Previous methods for extracting such structures can be categorized

as either structural or functional . Structural methods organize units based on their structural characteristics, including connectivity and edge-weights [12, 13, 14], while functional methods consider attributes based on unit activation patterns [15, 16]. However, it remains unclear whether these extracted structures represent the underlying hierarchy of sub-functions in a given task.

To the best of our knowledge, this work is the first attempt to introduce an approach that combines training (pruning) and network analysis to unveil the hierarchical modularity of tasks. Our proposed method for clustering units and discovering modules aligns with the structural methods. A series of prior studies leveraged normalized spectral clustering to globally extract unit clusters and analyze their characteristics [14, 15]. Spectral clustering optimizes for N-cuts, quantifying the internal connectivity relative to the external connectivity of unit clusters. Our method is most similar to previously introduced layer-wise unit clustering techniques [12, 13]. These methods consider incoming and outgoing edge-weights to group units within a layer and consolidate edges into single incoming and outgoing connections. Importantly, all previous methods were tailored for conventionally trained NNs. In contrast, our approach is simpler and customized for the pruned NNs we obtain.

Pruning of NN units and edges has gained significant traction as a method to enhance NN computational efficiency and reduce memory demands while maintaining performance integrity [21, 24, 23, 34, 35, 30, 36, 37, 38]. Recent research has also focused on investigating how pruning impacts NN generalization [39, 40, 41]. However, prior studies have primarily concentrated on either unit pruning or edge pruning in isolation, without employing both in a sequential manner as proposed in our work. In our approach, we employ unit pruning as a means to condition or compel NNs to learn reused sub-functions effectively. Furthermore, edge pruning is employed to uncover the sparse connectivity between various units or modules.

Mechanistic interpretability seeks to reverse-engineer trained NNs to understand their internal mechanisms. One avenue of research within mechanistic interpretability involves identifying the circuits formed within NNs. Recent discoveries include curve-detecting circuits in vision models [42, 43], transformer circuits [44], induction heads [45], and indirect object identification [46], among others. Our proposed approach for uncovering the hierarchical modular task structure dissects the NN into modules responsible for learning specific sub-functions. Hence, simplifying the complexity of reverse engineering entire NNs, by focusing instead on smaller sub-networks.

Continual learning aims at learning multiple tasks presented sequentially while preserving performance on previously learned tasks [47, 48, 49, 50]. Modular NNs have been demonstrated to mitigate catastrophic forgetting by freezing and reusing certain modules while introducing and updating others to learn new tasks [6, 7, 8]. However, those methods entail manually designing modular NNs with fixed module sizes at initialization. NNs that naturally acquire a hierarchically modular structure mirroring that of the task may offer additional advantages compared to generic modular NNs. These potential benefits encompass enhanced generalization, efficient transfer learning through the reuse of modules corresponding to frequently utilized sub-functions, and informed module additions to prevent sub-linear increases in NN capacity.

7 Conclusion

We have introduced Neural Sculpting , a methodology to uncover the hierarchical and modular structure of target tasks in NNs. Our findings first demonstrated that NNs trained conventionally do not naturally acquire the desired structural properties related to input separable and reused subfunctions. To address this limitation, we proposed a training strategy based on iterative pruning of units and edges, resulting in sparse NNs with those previous structural properties. Building upon this, we introduced an unsupervised method to detect modules corresponding to various sub-functions while also uncovering their hierarchical organization. Finally, we validated our proposed methodology by using it to uncover the underlying structure of a diverse set of modular and hierarchical tasks.

As a future research direction, we could investigate the efficiency and theoretical underpinnings of modularity in function graphs, which could further motivate pruning NNs. One potential approach to overcome the computational costs and dependence on initial architecture depth could be to use neural architecture search algorithms to construct modular NNs during training. Additionally, exploring the use of attention mechanisms and transformers to uncover hierarchical modularity in tasks could be an interesting direction for future work. These approaches could provide a more efficient way of obtaining modular NNs that can also better capture the underlying structure of the target task.

References

Herbert A Simon. The architecture of complexity. In Facets of systems science , pages 457-476. Springer, 1991.

Günter P Wagner, Mihaela Pavlicev, and James M Cheverud. The road to modularity. Nature Reviews Genetics , 8(12):921-931, 2007.

David Meunier, Renaud Lambiotte, Alex Fornito, Karen Ersche, and Edward T Bullmore. Hierarchical modularity in human brain functional networks. Frontiers in neuroinformatics , 3:37, 2009.

Miguel Valencia, MA Pastor, MA Fernández-Seara, J Artieda, J Martinerie, and M Chavez. Complex modular structure of large-scale brain networks. Chaos: An Interdisciplinary Journal of Nonlinear Science , 19(2):023119, 2009.

David Meunier, Renaud Lambiotte, and Edward T Bullmore. Modular and hierarchically modular organization of brain networks. Frontiers in neuroscience , 4:200, 2010.

Tom Veniat, Ludovic Denoyer, and Marc'Aurelio Ranzato. Efficient continual learning with modular networks and task-driven priors. arXiv preprint arXiv:2012.12631 , 2020.

Jorge A Mendez and Eric Eaton. Lifelong learning of compositional structures. arXiv preprint arXiv:2007.07732 , 2020.

Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. Advances in Neural Information Processing Systems , 34:3029830312, 2021.

Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: what is required and can it be learned? arXiv preprint arXiv:1811.12889 , 2018.

Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning , pages 2873-2882. PMLR, 2018.

Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: how do neural networks generalise? Journal of Artificial Intelligence Research , 67:757-795, 2020.

Chihiro Watanabe, Kaoru Hiramatsu, and Kunio Kashino. Modular representation of layered neural networks. Neural Networks , 97:62-73, 2018.

Chihiro Watanabe. Interpreting layered neural networks via hierarchical modular representation. In International Conference on Neural Information Processing , pages 376-388. Springer, 2019.

Daniel Filan, Stephen Casper, Shlomi Hod, Cody Wild, Andrew Critch, and Stuart Russell. Clusterability in neural networks. arXiv preprint arXiv:2103.03386 , 2021.

Shlomi Hod, Stephen Casper, Daniel Filan, Cody Wild, Andrew Critch, and Stuart Russell. Detecting modularity in deep neural networks. arXiv preprint arXiv:2110.08058 , 2021.

Richard D Lange, David S Rolnick, and Konrad P Kording. Clustering units in neural networks: upstream vs downstream information. arXiv preprint arXiv:2203.11815 , 2022.

Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. arXiv preprint arXiv:2010.02066 , 2020.

Jeff Clune, Jean-Baptiste Mouret, and Hod Lipson. The evolutionary origins of modularity. Proceedings of the Royal Society b: Biological sciences , 280(1755):20122863, 2013.

Henok Mengistu, Joost Huizinga, Jean-Baptiste Mouret, and Jeff Clune. The evolutionary origins of hierarchy. PLoS computational biology , 12(6):e1004829, 2016.

R Ryan Williams. Limits on representing boolean functions by linear combinations of simple functions: Thresholds, relus, and low-degree polynomials. arXiv preprint arXiv:1802.09121 , 2018.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems , pages 1135-1143, 2015.

Steven A Janowsky. Pruning versus clipping in neural networks. Physical Review A , 39(12):6600, 1989.

Sejun Park*, Jaeho Lee*, Sangwoo Mo, and Jinwoo Shin. Lookahead: A far-sighted alternative of magnitude-based pruning. In International Conference on Learning Representations , 2020.

Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems , pages 598-605, 1990.

Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV) , pages 464-472. IEEE, 2017.

Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003.02389 , 2020.

Xiaoqin Zeng and Daniel S Yeung. Hidden neuron pruning of multilayer perceptrons using a quantified sensitivity measure. Neurocomputing , 69(7-9):825-837, 2006.

DANIEL S Yeung and Xiao-Qin Zeng. Hidden neuron pruning for multilayer perceptrons using a sensitivity measure. In Proceedings. International Conference on Machine Learning and Cybernetics , volume 4, pages 1751-1757. IEEE, 2002.

Bo Li and Cheng Chen. First-order sensitivity analysis for hidden neuron selection in layer-wise training of networks. Neural Processing Letters , 48:1105-1121, 2018.

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learning Representations , 2019.

Michelle Girvan and Mark EJ Newman. Community structure in social and biological networks. Proceedings of the national academy of sciences , 99(12):7821-7826, 2002.

Mark EJ Newman and Michelle Girvan. Finding and evaluating community structure in networks. Physical review E , 69(2):026113, 2004.

Mark EJ Newman. Modularity and community structure in networks. Proceedings of the national academy of sciences , 103(23):8577-8582, 2006.

Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks. arXiv preprint arXiv:1507.06149 , 2015.

Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250 , 2016.

Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations , 2019.

Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems , 33, 2020.

Shreyas Malakarjun Patil and Constantine Dovrolis. Phew: Constructing sparse networks that learn fast and generalize well without training data. In International Conference on Machine Learning , pages 8432-8442. PMLR, 2021.

Siddhaling Urolagin, Prema KV, and NV Subba Reddy. Generalization capability of artificial neural network incorporated with pruning method. In Advanced Computing, Networking and Security: International Conference, ADCONS 2011, Surathkal, India, December 16-18, 2011, Revised Selected Papers , pages 171-178. Springer, 2012.

Brian Bartoldson, Ari Morcos, Adrian Barbu, and Gordon Erlebacher. The generalizationstability tradeoff in neural network pruning. Advances in Neural Information Processing Systems , 33:20852-20864, 2020.

Tian Jin, Michael Carbin, Daniel M Roy, Jonathan Frankle, and Gintare Karolina Dziugaite. Pruning's effect on generalization through the lens of training and regularization. arXiv preprint arXiv:2210.13738 , 2022.

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill , 5(3):e00024-001, 2020.

Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah. Curve detectors. Distill , 2020. https://distill.pub/2020/circuits/curve-detectors.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895 , 2022.

Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. URL https://arxiv. org/abs/2211.00593 , 2022.

Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671 , 2016.

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence , 40(12):2935-2947, 2017.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning , pages 3987-3995. PMLR, 2017.

Fengxiang He, Tongliang Liu, and Dacheng Tao. Control batch size and learning rate to generalize well: Theoretical and empirical evidence. Advances in Neural Information Processing Systems , 32, 2019.

Simran Kaur, Jeremy Cohen, and Zachary Chase Lipton. On the maximum hessian eigenvalue and generalization. In Proceedings on , pages 51-65. PMLR, 2023.

Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in Neural Information Processing Systems , 32, 2019.

Stanislav Fort, Paweł Krzysztof Nowak, Stanislaw Jastrzebski, and Srini Narayanan. Stiffness: A new perspective on generalization in neural networks. arXiv preprint arXiv:1901.09491 , 2019.

Peter J Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics , 20:53-65, 1987.

Tadeusz Cali´ nski and Jerzy Harabasz. A dendrite method for cluster analysis. Communications in Statistics-theory and Methods , 3(1):1-27, 1974.

David L Davies and Donald W Bouldin. A cluster separation measure. IEEE transactions on pattern analysis and machine intelligence , 1(2):224-227, 1979.

A Appendix structure and details

In this section, we discuss the structure of the appendix. The code and datasets used can be found here: https://github.com/ShreyasMalakarjunPatil/Neural-Sculpting .

Appendix sections (B - F) accompany section 5 in the main paper and provide visualizations of neural networks (NNs) and the hierarchical modularity uncovered by our proposed pipeline. Each section includes NN visualizations corresponding to the five experiments summarized in the main paper. The main figures in these sections are organized such that the first row represents different function graphs that the NNs are trained to learn, with each column representing a distinct function graph relevant to the experiment. The subsequent rows show visualizations of the uncovered hierarchical modular structures for NNs with increasing depths.

Visualization figure details: In the appendix sections, each function graph is associated with visualizations from a specific NN trial at a particular depth value (with one width value and one seed value). This visualization represents the structure that is commonly recovered for the given function graph and NN depth. While the NN width value and seed value remain consistent in the visualizations, they may vary if the recovered structure does not represent the majority. We also present and discuss other structures that are frequently recovered, using other function graphs representing the same Boolean function. For success rate computations, such trials are typically considered failures in recovering overall structures. However, for individual sub-function detection success rates, they may be considered successful if they uncover a module corresponding to that sub-function. Additional visualizations and the hyperparameters used for training and pruning the NNs are detailed in the code base.

Function graph nomenclature: In the visualizations, the term "modularity" indicates the function graph being used. The modularity is represented as a list that indicates the number of sub-functions in the hierarchy. For example, if the modularity is [1 , 2] , it means there are two sub-functions in the upper hierarchical level that take as input the output of a single sub-function lower in the hierarchy. In the code base, we prepend the number of input nodes and append the number of output nodes to this list.

Appendix section G accompanies section 2 in the main paper. Section 2 presents two tests developed to demonstrate that NNs, through standard training, do not acquire structural properties reflecting input separable and reused sub-functions. In appendix section G, we provide detailed results for these tests conducted on nine different NN architectures to further support our claims. We also report the results of the same tests conducted on edge-pruned and unit-edge-pruned NNs.

Appendix section H accompanies section 3 in the main paper, which presents the method for pruning NNs. This appendix section provides a detailed explanation of the unit and edge pruning algorithms, along with the exploration strategy and results for hyperparameter tuning.

Appendix section I accompanies section 4 in the main paper, which introduces the proposed method for detecting modules corresponding to various sub-functions. In this appendix section, we present a detailed algorithm description and the methodology used for hyperparameter exploration.

Note that the pruning and module detection hyperparameters were explored concurrently. However, in the appendix section, we first present experiments with the pruning hyperparameters while keeping the module detection method fixed, and vice versa.

Resources: The unit and edge pruning processes for the Boolean functions were executed on a CPU, while the pruning for the MNIST experiment was performed using RTX6000 GPUs.

B NN visualizations: Function graphs used for validation

Inputseparable sub-function

Reused sub-function

Input separable and reused

Single sub-function

QQ

888

X5

Modularity[2]: Arch[4, 24, 4]: Density = 8.33%

Modularity[1, 2]: Arch[4, 24, 16]: Density = 8.75%

Modularity[3, 3]: Arch[6, 24, 3]: Density = 5.56%

Modularity[1]: Arch[4, 24, 4]: Density = 8.33%

Ir = 0.05, batch size =8, epochs =15, seed =0, arch =[4, 4, 4]

Ir= 0.1, batch size =8, epochs =40, sed =0, arch =[4, 3, 16]

Ir = 0.05, batch size =32, epochs =15, seed =0, arch =[6, 3, 3]

Ir = 0.1, batch size =8, epochs =40, seed =0, arch =[4, 2, 4]

P=70.0, P =2.5

P =50.0, P。 =2.5

P =70.0, P。 =1.0

P, =70.0, P =2.5

Modularity[2]: Arch[4, 24, 24, 4]: Density = 2.08%

Modularity[1, 2]: Arch[4, 24, 24, 16]: Density = 2.75%

Modularity[3, 3]: Arch[6, 24, 24, 3]: Density = 1.89%

Modularity[1]: Arch[4, 24, 24, 4]: Density = 1.82%

Ir = 0.05, batch size =8, epochs =10, seed =0, arch =[4, 4, 2, 4]

Ir = 0.1, batch size =8, epochs =40, seed =0, arch =[4, 3, 2, 16]

Ir = 0.05, batch size =32, epochs =10, seed =0, arch =[6, 3, 3, 3]

Ir = 0.05, batch size =8, epochs =30, seed =0, arch =[4, 2, 1, 4]

P =65.0, P。 =1.5

P =70.0, P。 =2.5

P =60.0, P。 =0.5

P。 =15.0, P。 =2.5

Modularity[2]: Arch[4, 24, 24, 24, 4]: Density = 1.26%

Modularity[1, 2]: Arch[4, 24, 24, 24, 16]: Density = 2.02%

Modularity[3, 3]: Arch[6, 24, 24, 24, 3]: Density = 1.32%

Modularity[1]: Arch[4, 24, 24, 24, 4]: Density = 1.04%

[‘e]= ypue ‘0= paas ‘oz= supoda ‘g= azis yoqeq ‘soo =

Ir = 0.1, batch size =8, epochs =40, seed =0, arch =[4, 3, 3, 2, 16]

Ir= 0.05, batch size =32, epochs =15, seed =0, arch =[6, 3, 3, 3, 3]

Ir = 0.1, batch size =8, epochs =30, seed =0, arch =[4, 1, 3, 1, 4]

P =70.0, P, =1.5

P =70.0, P =2.5

P =50.0, P =1.5

P =25.0, Ps =2.5

C NN visualizations: Function graphs with a single sub-function and increasing reuse

X1

X2

X3

X4

X1

X2

X3

X1

X2

X3

X4

X1

X2

X3

X4

X1

X2

X3

X4

X1

X2

X4

X1

X2

X3

X4

X1

X2

X3

X4

X1

ZX

X3

X4

X1

X2

X3

X4

Modularity[1., 2]: Arch[4, 24, 2]: Density = 9.72%

Modularity[1, 2]: Arch[4, 24, 4]: Density = 9.38%

Modularity[1, 2]: Arch[4, 24, 8]: Density = 9.03%

Modularity[1, 2]: Arch[4, 24, 12]: Density = 8.85%

Modularity[1, 2]: Arch[4, 24, 16]: Density = 8.75%

[r = 0.1, batch size =8, epochs =20, seed =0, arch =[4, 3, 2]

P, =35.0, P, =1.5

[r = 0.1, batch size =8, epochs =40, seed =5, arch =[4, 3, 4]

P, =65.0, P, =1.0

Ir = 0.1, batch size =8, epochs =40, seed =1, arch =[4, 3, 8]

P, =50.0, P, =2.5

Ir = 0.1, batch size =8, epochs =40, seed =0, arch =[4, 3, 12]

P, =65.0, P, =2.5

Ir = 0.1, batch size =8, epochs =40, seed =0, arch =[4, 3, 16]

P, =50.0, P, =2.5

Modularity[1, 2]: Arch[4, 24, 24, 2]: Density = 2.08%

Modularity[1, 2]: Arch[4, 24, 24, 4]: Density = 2.34%

Modularity[1, 2]: Arch[4, 24, 24, 8]: Density = 2.55%

Modularity[1, 2]: Arch[4, 24, 24, 12]: Density = 2.6%

Modularity[1, 2]: Arch[4, 24, 24, 16]: Density = 2.75%

[r= 0.1, batch size =8, epochs =30, seed =5, arch =[4, 3, 2, 2]

Ir= 0.1, batch size =8, epochs =40, seed =0, arch =[4, 3, 2, 4]

Ir = 0.1, batch size =16, epochs =30, seed =0, arch =[4, 3, 2, 8]

Ir = 0.1, batch size =8, epochs =40, seed =0, arch =[4, 3, 2, 12]

Ir = 0.1, batch size =8, epochs =40, seed =0, arch =[4, 3, 2, 16]

P, =45.0, P, =1.0

P, =55.0, P, =2.5

P, =40.0, P, =2.5

P =65.0, P, =2.0

P, =70.0, P, =2.5

Modularity[1. 2]: Arch[4, 24, 24, 24, 2]: Density = 1.31%

Modularity[1. 2]: Arch[4, 24, 24, 24, 4]: Density = 1.49%

Modularity[1, 2]: Arch[4, 24, 24, 24, 8]: Density = 1.6%

Modularity[1. 2]: Arch[4, 24, 24, 24, 12]: Density = 1.95%

Modularity[1. 2]: Arch[4, 24, 24, 24, 16]: Density = 2.02%

Ir= 0.1,batch size =16, epochs =20, seed =0, arch =[4, 3, 2, 2, 2]

P, =25.0, P, =1.5

Ir= 0.1, batch size =8, epochs =40, seed =0, arch =[4, 3, 2, 2, 4]

P, =10.0, P, =2.0

Ir = 0.1, batch size =8, epochs =40, seed =0, arch =[4, 2, 3, 2, 8]

P, =15.0, P, =1.5

Ir = 0.1, batch size =8, epochs =40, seed =0, arch =[4, 4, 2, 2, 12]

P, =15.0, P, =2.5

Ir= 0.1, batch size =8, epochs =40, seed =0, arch =[4, 3, 3, 2, 16]

P, =70.0, P, =2.5

D NN visualizations : Increasing input overlap and reuse of two input separable sub-functions

D.1 Completely separable sub-functions

88899228

[‘ *8]= ypue ’0= paas °0z= supoda ‘ze= azis upjeq 't°0 =J]

=5.0, P, =2.5

[r = 0.1, batch size =32, epochs =15, seed =0, arch =[8, 5, 4]

P, =65.0, P, =2.5

Ir = 0.1, batch size =32, epochs =15, seed =0, arch =[8, 4, 8]

r= 0.1, batch

= 0.1, ba

=0, arch =[8, 5, 16]

Modularity[2]: Arch[8, 24, 24, 2]: Density = 2.57%

Modularity[2]: Arch[8, 24, 24, 4]: Density = 2.78%

Modularity[2]: Arch[8, 24, 24, 8]: De

[r = 0.1, batch size =32, epochs =50, seed =0, arch =[8, 4, 3, 2]

P, =70.0, P, =1.5

[r = 0.1, batch size =32, epochs =40, seed =0, arch =[8, 5, 2, 4]

=60.0, P, =1.5

[r = 0.1, batch size =32, epochs =40, seed =0, arch =[8, 4, 2, 12]

=35.0, P, =2.5

r = 0.1, batch size =32, epochs =40, seed =0, arch =[8, 4, 2, 16]

P, =55.0, P, =1.5

= 0.05, batch size =32, epochs =40, seed =0, arch =[8, 3, 3, 2, 2]

1 =0, arch =[8, 5, 3, 2, 8]

=50.0, P, =2.5

。 =30.0, P, =2.0

=55.0, P, =0.5

P, =45.0, P, =0.5

D.2 Single overlapping input node

22

...·.

........

........

Modularity[3, 2]: Arch[7, 24, 2]: Density = 8.8%

Modularity[3, 2]: Arch[7, 24, 4]: Density = 8.71%

Modularity[3, 2]: Arch[7,

ity[3, 2]: Arch[7, 24, 16]: Density = 8.51%

Ir = 0.1, batch size =16, epochs =40, seed =0, arch =[7, 4, 2]

Ir = 0.1, batch size =32, epochs =20, seed =0, arch =[7, 4, 4]

Ir = 0.1, batch size =32, epochs =20, seed =0, arch =[7, 5, 8]

Ir = 0.1, batch size =32, epochs =20, seed =0, arch =[7, 4, 12]

Ir = 0.1, batch size =32, epochs =30, seed =0, arch =[7, 4, 16]

P。=70.0, P, =2.5

P, =45.0, P, =2.5

P, =70.0, P, =2.5

P, =35.0, P, =2.5

P, =70.0, P, =1.0

Modularity[3, 2]: Arch[7, 24, 24, 2]: Density = 2.27%

Modularity[3, 2]: Arch[7, 24, 24, 4]: Density = 2.5%

Modularity[3, 2]: Arch[7, 24, 24, 8]: Density = 2.88%

Modularity[3, 2]: Arch[7, 24, 24, 12]: Density = 3.0%

Modularity[3, 2]: Arch[7, 24, 24, 16]: Density = 3.1%

= 0.1, batch size =16, epochs =30, seed =0, arch =[7, 3, 3, 2]

P, =10.0, P, =2.5

Ir = 0.05, batch size =16, epochs =30, seed =0, arch =[7, 3, 3, 4]

P, =20.0, P, =2.0

Ir = 0.1, batch size =32, epochs =40, seed =0, arch =[7, 4, 2, 8]

P, =55.0, P, =1.0

Ir = 0.1, batch size =32, epochs =40, seed =0, arch =[7, 4, 2, 12]

P, =65.0, P, =2.5

Ir = 0.1, batch size =32, epochs =30, seed =0, arch =[7, 4, 2, 16

P, =60.0, P, =2.5

ularity[3, 2]: Arch[7, 24, 24, 24, 4]: Density = 1.62%

Modularity[3, 2]: Arch[7, 24, 24, 24, 12]: Density = 2.05%

r = 0.05, batch size =16, epochs =60, seed =0, arch =[7, 2, 4, 2, 2]

P =20.0, P, =1.5

[r = 0.1, batch size =64, epochs =60, seed =0, arch =[7, 3, 3, 2, 4]

P, =25.0, P, =2.5

r = 0.1, batch size =64, epochs =40, seed =0, arch =[7, 3, 3, 2, 8]

P, =30.0, P, =2.5

Ir= 0.1, batch size =64, epochs =60, seed =0, arch =[7, 4, 2, 2, 12]

P, =35.0, P, =2.5

Ir= 0.05, batch sze =64, pochs =60, seed =0. arch =[7, 5, 2. 2, 16]

P, =25.0, P, =0.5

D.3 Two overlapping input nodes

22

.......

Modularity[3, 2]: Arch[6, 24, 2]: Density = 9.9%

Modularity[3, 2]: Arch[6, 24, 4]: Density = 10.42%

Modularity[3, 2]: Arch[6, 24, 12]: Density = 9.03%

Modularity[3, 2]: Arch[6, 24, 16]: Density = 8.9%

Ir = 0.1, batch size =16, epochs =30, seed =0, arch =[6, 4, 2]

Ir = 0.1, batch size =32, epochs =40, seed =0, arch =[6, 5, 4]

P, =65.0, P, =2.5

Ir = 0.1, batch size =32, e

seed =0, arch =[6, 5, 8]

[r = 0.1, batch size =32, epochs =40, seed =0, arch =[6, 4, 12]

P, =65.0, P, =2.5

Ir = 0.1, batch size =32, epochs =30, seed =0, arch =[6, 4, 16]

P, =70.0, P, =2.5

= 0.1, batch size =32,

=1, arch =[6, 5, 2, 2]

= 0.1, batch size

=0, arch =[6, 5, 2, 4

arch =[6, 5, 2, 8

0, arch =[6, 5, 2, 12]

r = 0.1, batch size =32, epochs =60, seed =0, arch =[6, 5, 2, 16]

P, =30.0, P, =2.5

Modularity[3, 2]: Arch[6, 24, 24, 24, 16]: Density = 2.02%

, =15.0, P, =1.5

P, =55.0, P, =2.5

0.05, batch size =32, epochs =60, seed =0, arch =[6, 3, 4, 2, 8]

P, =25.0, P, =2.0

Ir = 0.05, batch size =32, epochs =80, seed =0, arch =[6, 3, 3, 2, 12]

P, =10.0, P, =0.5

Ir= 0.05, batch sie =32, eochs =120, sed =0., arch =[6,3 3 2, 16]

P, =20.0, P, =2.5

D.4 Three overlapping input nodes

.....

........

Ir = 0.1, batch size =

=30.0, P, =2.5

=0, arch =[5, 3, 2]

Ir = 0.1, batch size =

=65.0, P, =0.5

ochs =20, seed =5, arch =[5, 4, 4]

Ir = 0.1, batch size =16, e

P =55.0, P, =2.5

d =0, arch =[5, 4, 8]

Ir = 0.05, batch size =16, epochs =20, seed =0, arch =[5, 4, 12]

P, =50.0, P, =2.5

=30.0, P, =1.5

=0, arch =[5, 4, 16

Modularity[3, 2]: Arch[5, 24, 24, 2]: Density = 2.55%

r = 0.1, batch size =16, epochs =30, seed =1, arch =[5, 3, 3, 2]

r = 0.05, batch size =8

=0, arch =[5, 4, 2, 12]

r = 0.1, batch size =16,

=20.0, P, =2.0

Modularity[3, 2]: Arch[5

ns =30, se

=50.0, P, =1.5

I =2, arch =[5, 3, 3, 2, 4]

r = 0.05, batch size =16, epo

=0, arch =[5, 4, 2, 2, 8]

r = 0.05, batch size =16, epochs =30, seed =0, arch =[5, 4, 2, 2, 12]

P, =40.0, P, =2.5

Ir = 0.05, batch size =16, epochs =30, seed =0, arch =[5, 4, 2, 2, 16]

P, =70.0, P, =0.5

Modularity[3, 2]: Arch[5, 36, 12]: Density = 6.37%

Modularity[3,2]:Arch[5,36,36,12]:Density =1.73%

Ir = 0.05, batch size =8, epochs =30, seed =2, arch =[5, 4, 12]

Ir = 0.1, batch size =16, epochs =30, seed =2, arch =[5, 5, 2, 12]

P=60.0, P。=2.5

P =65.0, P。=2.5

Modularity[3, 2]: Arch[5, 48, 48, 48, 8]: Density = 0.5%

Modularity[3, 2]: Arch[5, 36, 36, 36, 12]: Density = 0.94%

Modularity[3, 2]: Arch[5, 48, 48, 48, 16]: Density = 0.61%

Ir = 0.1, batch size =16, epochs =40, seed =5, arch =[5, 3, 2, 2, 8]

Ir = 0.05, batch size =16, epochs =20, seed =5, arch =[5, 3, 3, 2, 12]

Ir = 0.1, batch size =16, epochs =60, seed =2, arch =[5, 3, 2, 2, 16]

P =60.0, P。=2.5

P =30.0, P。 =1.5

P=50.0,P。=0.5

D.5 All overlapping input nodes

2.

.....

Modularity[1, 2, 2]: Arch[4, 24, 2]: Density = 8.33%

Modularity[1, 2, 2]: Arch[4, 24, 4]: Density = 10.94%

Modularity[1, 2, 2]: Arch[4, 24, 12]: Density = 10.94%

Modularity[1, 2, 2]: Arch[4, 24, 16]: Density = 8.33%

Ir = 0.1, batch size =8, epochs =30, seed =0, arch =[4, 2, 2]

P, =35.0, P, =2.5

P, =30.0, P, =2.5

Ir = 0.1, batch size

=10.0, P, =1.5

=0, arch =[4, 3, 8]

Ir = 0.1, batch size

=50.0, P, =2.5

ed =2, arch =[4, 2, 16]

Modularity[1, 2, 2]: Arch[4, 24, 24, 2]: Density = 1.67%

Modularity[1, 2. 2]: Arch[4, 24, 24, 4]: Density = 2.6%

Modularity[1, 2, 2]: Arch[4, 24, 24, 8]: Density = 2.08%

Modularity[1. 2, 2]: Arch[4, 24, 24, 12]: Density

= 2.29%

[°t °z ]= qoue 0= paes 0e= suxodo “g= azis upqeq ‘t0 = )

ir= 0.05, batch size =16, epochs =30, seed =0, arch =[4, 2, 2, 4]

[r= 0.1, batch size =16, epochs =20, seed =5, arch =[4, 2,1, 8]

r = 0.1, batch size =16, epochs =30, seed =0, arch =[4, 2, 1, 12]

Ir= 0.1, batch size =8, pochs =15, seed =0, arch =[4, 2, 2, 16]

P =35.0, P, =2.5

P, =50.0, P, =2.5

P, =45.0, P, =2.5

P, =20.0, P, =2.5

P, =70.0, P, =2.5

Modularity(1. 2, 2]: Arch[4, 24, 24, 24, 2]: Density = 1.0%

Modularity[1, 2, 2]: Arch[4, 48, 48, 48, 8]: Density = 0.37%

Modularity[1. 2, 2]: Arch[4, 48, 48, 48, 12]: Density = 0.6%

Modularity[1. 2. 2]: Arch[4, 36, 36, 36, 16]: Density = 1.21%

P, =45.0, P, =2.5

=1, arch =[4, 2, 1, 1, 2]

P, =70.0, P, =2.5

ed =5, arch =[4, 2, 1, 2, 4]

d =1, arch =[4, 2, 1, 1, 8]

r = 0.1, batch size =16, e

P =60.0, P, =1.5

=5, arch =[4, 3, 2, 1, 12]

P, =65.0, P, =1.5

....

Modularity[1, 2, 2

10.76%

= 10.16%

ularity[1, 2, 2]: Arch[4, 24, 16]: Density = 9.79%

rch =[4,3.2.2]

20,

r = 0.1, batch size =8, epochs =15, seed =1, arch =[4, 4, 2, 16]

P, =25.0, P, =0.5

arity[1, 2, 2]: Arch[4, 48, 48, 48, 16]: Density

..

Modularity[1, 2, 2]: Arch[4, 36, 36, 4]: Density = 1.26%

Modularity[1, 2, 2]: Arch[4, 24, 24, 8]: Density = 3.12%

Modularity[1, 2, 2]: Arch[4, 24, 24, 12]: Density = 3.44%

Modularity[1, 2, 2]: Arch[4, 48, 48, 16]: Density = 1.23%

Ir = 0.1, batch size =16, epochs =20, seed =1, arch =[4, 3, 3, 4]

P, =60.0, P, =2.5

Ir = 0.1, batch size =16, epochs =20, seed =0, arch =[4, 3, 3, 8]

P, =65.0, P, =2.5

Ir = 0.1, batch size =16, epochs =30, seed =5, arch =[4, 3, 3, 12]

P, =15.0, P, =2.5

Ir = 0.1, batch size =16, epochs =20, seed =0, arch =[4, 4, 3, 16]

P =35.0, P, =1.5

Modularity[1. 2, 2]: Arch[4, 48, 48, 48, 4]: Density = 0.42%

Modularity[1, 2, 2]: Arch[4, 24, 24, 24, 8]: Density = 1.6%

Modularity[1, 2, 2]: Arch[4, 48, 48, 48, 12]: Density = 0.6%

Modularity[1, 2, 2]: Arch[4, 24, 24, 24, 16]: Density = 2.14%

Ir = 0.1, batch size =16, epochs =20, seed =1, arch =[4, 3, 2, 3, 4]

Ir = 0.1, batch size =16, epochs =20, seed =2, arch =[4, 3, 2, 2, 8]

Ir= 0.1, batch size =16, epochs =20, seed =0, arch =[4, 4, 2, 2, 12]

Ir = 0.1, batch size =8, epochs =40, seed =0, arch =[4, 4, 2, 2, 16]

P =15.0, P, =1.5

P =5.0, P, =2.5

P, =30.0, P, =2.0

P, =35.0, P, =2.0

E NN visualizations : Increasing the number of hierarchical levels

L1

X4

Modularity[2, 1, 2]: Arch[8, 48, 4]: Density = 6.42%

Modularity[2, 1, 2]: Arch[8, 48, 8]: Density = 7.42%

Modularity[2, 1, 2]: Arch[8, 48, 12]: Density = 8.23%

Modularity[2, 1, 2]: Arch[8, 48, 16]: Density = 8.59%

Ir = 0.1, batch size =32, epochs =40, seed =0, arch =[8, 5, 4]

Ir = 0.1, batch size =32, epochs =40, seed =0, arch =[8, 5, 8]

Ir = 0.1, batch size =32, epochs =40, seed =0, arch =[8, 5, 12]

Ir = 0.1, batch size =32, epochs =40, seed =0, arch =[8, 5, 16]

P =50.0, P。 =1.0

P =5.0, P。 =1.5

P =65.0, P。 =1.5

P =55.0, P。 =2.5

Modularity[2, 1, 2]: Arch[8, 48, 48, 4]: Density = 0.94%

Modularity[2, 1, 2]: Arch[8, 48, 48, 8]: Density = 1.3%

Modularity[2, 1, 2]: Arch[8, 48, 48, 12]: Density = 1.38%

Modularity[2, 1, 2]: Arch[8, 48, 48, 16]: Density = 1.5%

Ir= 0.05, batch size =64, epochs =40, seed =0, arch =[8, 4, 2, 4]

Ir= 0.05, batch size =64, epochs =40, seed =0, arch =[8, 5, 2,8]

Ir = 0.1, batch size =32, epochs =20, seed =0, arch =[8, 4, 3, 12]

Ir = 0.05, batch size =64, epochs =15, seed =1, arch =[8, 5, 2, 16]

P =70.0, Pe =2.0

Pu =10.0, Pe =2.5

P =20.0, Pe =1.5

P =60.0, P。 =1.5

Modularity[2, 1, 2]: Arch[8, 48, 48, 48, 4]: Density = 0.52%

Modularity[2. 1, 2]: Arch[8, 48, 48, 48, 8]: Density = 0.61%

Modularity[2. 1. 2]: Arch[8, 48, 48, 48, 12]: Density = 0.68%

Modularity[2, 1, 2]: Arch[8, 48, 48, 48, 16]: Density = 0.71%

Ir = 0.1, batch size =128, epochs =40, sed =0, arch =[8, 4, 2, 2, 4]

Ir = 0.05, batch size =16, epochs =40, seed =0, arch =[8, 6, 2, 2, 8]

Ir = 0.05, batch size =16, epochs =40, seed =0, arch =[8, 5, 2, 2, 12]

Ir = 0.1, batch size =32, epochs =60, seed =1, arch =[8, 5, 2, 2, 16]

P =35.0, P。 =1.0

P =60.0, P。 =1.0

P =50.0, P。 =2.0

P =65.0, P。 =1.0

Modularity[2, 1, 2]: Arch[8, 48, 48, 48, 48, 4]: Density = 0.4%

Modularity[2. 1. 2]: Arch[8, 48, 48, 48, 48, 8]: Density = 0.47%

Modularity[2, 1, 2]: Arch[8, 48, 48, 48, 48, 12]: Density = 0.47%

Modularity[2. 1. 2]: Arch[8, 48, 48, 48, 48, 16]: Density = 0.53%

Ir= 0.05, batch size =128, epochs =40, seed =0, arch =[8, 5, 2, 2, 2, 4]

Ir= 0.05, batch size =32, epochs =30, seed =0, arch =[8, 6, 2, 2, 2, 8]

Ir = 0.05, batch size =32, epochs =30, seed =2, arch =[8, 4, 2, 2, 2, 12]

[r= 0.05, batch size =32, epochs =30, seed =0, arch =[8, 5, 2, 2, 2, 16]

P =65.0, P, =0.5

P =35.0, P。 =0.5

P =55.0, P, =1.0

P, =10.0, P =1.0

E.1 Other structures frequently recovered

Modularity[2, 1, 2]: Arch[8, 60, 60, 60, 4]: Density = 0.4%

Modularity[2, 1, 2]: Arch[8, 60, 60, 60, 60, 4]: Density = 0.3%

Ir = 0.05, batch size =128, epochs =40, seed =1, arch =[8, 5, 3, 4, 2, 4]

Ir = 0.05, batch size =128, epochs =20, seed =1, arch =[8, 5, 2, 4, 4]

P=5.0, P。=2.0

P =45.0, P =0.5

Modularity[2, 1, 2]: Arch[8, 60, 60, 60, 4]: Density = 0.42%

Modularity[2, 1, 2]: Arch[8, 60, 60, 60, 60, 4]: Density = 0.3%

Ir = 0.05, batch size =128, epochs =20, seed =2, arch =[8, 5, 3, 3, 4]

Ir = 0.05, batch size =128, epochs =40, seed =1, arch =[8, 5, 3, 2, 3, 4]

P =60.0, Pe =1.0

P =40.0, Pe =2.0

E.2 Output sub-functions used once

Modularity[2, 1, 2]: Arch[8, 48, 48, 2]: Density = 0.9%

Modularity[2, 1, 2]: Arch[8, 48, 48, 48, 2]: Density = 0.51%

Modularity[2, 1. 2]: Arch[8, 48, 48, 48, 2]: Density = 0.49%

Modularity[2, 1, 2]: Arch[8, 48, 48, 48, 48, 2]: Density = 0.39%

Modularity[2, 1, 2]: Arch[8, 48, 48, 48, 48, 2]: Density = 0.38%

Ir = 0.1, batch size =64, epochs =40, seed =0, arch =[8, 5, 2, 2]

P =25.0, P, =2.5

Ir = 0.05, batch size =16, epochs =30, seed =1, arch =[8, 5, 2, 2, 2]

P, =30.0, P, =1.0

[r = 0.05, batch size =16, epochs =30, seed =0, arch =[8, 4, 2, 2, 2]

P, =65.0, P, =0.5

Ir = 0.05, batch si2e =32, epochs =40, seed =5, arch =[8, 4, 3, 3, 3, 2]

P, =10.0, P, =2.5

Ir = 0.05, batch si2e =32, epochs =40, seed =2, arch =[8, 5. 2, 2, 2, 2]

P, =55.0, P, =0.5

F NN visualizations for tasks with MNIST

F.1 MNIST task classifying two digits

Modularity[2]:Arch[1568,784,784,20]:Density=0.12%

Ir=0.001,batch size =512,epochs =40,seed =0,arch=[836,26,25,20]

Pu=30.0,Pe=1.0

Ir= 0.001, batch size =512,epochs =40, seed =0, arch =[827,24,17, 16, 20]

Pu=30.0, Pe=1.0

Pu=50.0,Pe=0.5

F.2 Hierarchical task with MNIST digits

m

888

G Structural properties of dense and sparse NNs

T-statisticvaluesfor denseNNswithvarying

T-statisticvaluesfor denseNNswithvarying

width and depth, Test 1

width and depth,Test 2

5

- 5.0

忆

-19.43

1.58

-1.77

-0

忆

-8.50

-6.35

2.08

2.5

0.0

-5

width

-20.31

5.62

3.91

width

-11.78

-6.44

-1.78

2.5

-10

5.0

7.5

9

-9.11

-2.35

0.44

15

9

5.15

-0.68

-10.0

20

12.5

2

3

2

hidden layers

hiddenlayers

PvaluesfordenseNNswithvarying

Pvaluesfor denseNNswithvarying

width and depth, Test 1

width and depth, Test 2

-10

-10

0.00

0.91

0.06

0.8

忆

0.00

0.00

0.96

0.8

0.6

0.6

width

0.00

1.00

1.00

width

0.00

0.00

0.06

0.4

0.4

0.00

0.03

0.66

0.2

0.00

1.00

0.27

0.2

2

0.0

2

0.0

hiddenlayers

hiddenlayers

T-statisticvaluesforedge pruned NNswithvarying

T-statisticvaluesforedge pruned NNswithvarying

width and depth, Test 1

width and depth,Test 2

--100

-25

忆

-58.44

-216.77

-36.66

-200

忆

-59.83

-211.92

-13.92

--50

300

-75

-242.41

-834.46

-27.27

400

width

-171.41

-31.55

-100

-500

-125

600

150

8

-169.24

-59.83

-73.86

-700

9

-89.25

-105.14

-113.88

175

-800

-200

2

3

2

hiddenlayers

hidden layers

Pvaluesforedgepruned NNs withvarying

Pvaluesforedgepruned NNs withvarying

width and depth,Test 1

widthand depth,Test2

-0.100

-0.100

0.00

0.00

0.00

0.075

忆

0.00

0.00

0.00

0.075

0.050

0.050

0.025

0.025

width

0.00

0.00

0.00

0.000

0.00

0.00

0.00

0.000

0.025

0.025

0.050

0.050

9

0.00

0.00

0.00

0.075

0.00

0.00

0.00

0.075

0.100

0.100

2

2

hidden layers hiddenlayers

T-statisticvaluesfor unit-edge pruned NNs withvarying

T-statisticvalues for unit-edge prunedNNs withvarying

width and depth, Test 1

width and depth,Test2

忆

-180.74

-56.39

-35.07

200

忆

-123.24

-78.66

-46.80

-40

-400

-60

width

-42.71

-164.27

-1217.05

-600

width

-73.81

-29.97

-40.43

80

-800

--100

9

-75.70

-65.75

-156.11

-1000

9

-63.88

-32.22

-49.98

1200

120

2

3

2

hiddenlayers

hiddenlayers

Pvaluesforunit-edgeprunedNNswithvarying

Pvaluesforunit-edgeprunedNNswithvarying

width and depth, Test 1

-0.100

widthanddepth,Test2

- 0.100

0.00

0.00

0.00

0.075

忆

0.00

0.00

0.00

0.075

0.050

-0.050

0.025

0.025

width

9E

0.00

0.00

0.00

0.000

0.00

0.00

0.00

0.000

0.025

0.025

0.050

-0.050

0.00

0.00

0.00

0.075

0.00

0.00

0.00

0.075

0.100

0.100

2

2

hiddenlayers hiddenlayers

H Pruning Algorithms

This appendix section is a companion to section 3 of the main paper. Here, we provide a detailed explanation of the pruning algorithm and outline the experiments conducted to explore hyperparameters for training and pruning NNs.

H.1 Details

The pseudo code for the unit pruning algorithm can be found in Algorithm 1, while the pseudo code for the edge pruning algorithm is provided in Algorithm 2. In each iteration, pruning is performed (either units or edges), followed by training the NN for the same number of epochs and using the same learning rate schedule.

The pruning procedure consists of two steps. First, a score is assigned to each unit/edge, and then a fraction of the units/edges is pruned based on these scores. It is important to note that the pruning fractions ( p u , p e , and p ) are always computed relative to the original number of hidden units or edges in the NN.

Both unit pruning and edge pruning are implemented using binary masks. Each edge weight is assigned a binary mask value, where a value of 1 indicates that the edge weight is not pruned, and a value of 0 indicates that it is pruned. When a unit is pruned, all incoming and outgoing edge weights associated with that unit are masked out by setting their binary mask values to 0.

Unit and edge pruning gradients: In our experiments, we conducted a grid search over the p u and p e values for each NN architecture, function graph, and seed value. We explored a range of 14 different p u values, ranging from 5% to 70% in increments of 5% , and 5 different p e values: 0 . 5 , 1 . 0 , 1 . 5 , 2 . 0 , 2 . 5 . It is important to note that the p u values used throughout the process are relative to the original number of units in the NN, while the p e values are relative to the number of edges in the original NN. After the grid search, we select the NN with the lowest density among all the sparse NNs obtained. Typically, we find that higher unit pruning gradients ( p u > 40 ) work best, as they allow for the removal of a lower number of units in each iteration by halving the gradient value.

Algorithm 1: Unit pruning algorithm

Algorithm 2: Edge pruning algorithm

Input separablesub-function

Reused sub-function

Input separable andreused

Single sub-function

X1

XE

OR

AND

ID

Transfer

Negation

H.2 Learning rate and batch size experiments

In this section, we conduct experiments to investigate the impact of learning rates and batch sizes on the structures obtained through pruning. Previous studies have examined the roles of these hyper-parameters in terms of convergence, generalization, and the ability of NNs to find global optima [51, 52]. However, most of these studies train NNs for a fixed number of epochs or iterations, which may not be suitable for our experiment as we require the original dense NNs to learn the task effectively.

To determine the appropriate number of epochs, we aim for the NNs with the four considered seed values to achieve 100% validation accuracy. We set a maximum of 120 epochs to avoid excessively long training or pruning times. Consequently, some combinations of learning rates and batch sizes do not achieve 100% validation accuracy. These combinations typically fall into two categories: 1) high learning rates and low batch sizes leading to unstable training, and 2) low learning rates and high batch sizes resulting in an insufficient number of training iterations to achieve 100% validation accuracy. Such cases are marked with an 7 in the result tables.

We validate and tune hyperparameters for our pipeline using four function graphs shown in Figure 37. These graphs include input separable sub-functions, reused sub-functions, a combination of input separable and reused sub-functions, and a non-modular function graph. We then train and prune NNs with a width of 24 and varying depths, reporting the success rates for detecting the same hierarchical structure as the function graph.

Tables 2 and 3 present the success rates for the four different function graphs considered. We observe that higher learning rates generally lead to more successful trials across all function graphs and NN depths. This aligns with previous findings indicating that higher learning rates result in more globally optimal solutions [53, 54]. On the other hand, lower learning rates tend to under-prune the NNs, resulting in inefficient hierarchical structures in terms of NN density.

When using high learning rates and low batch sizes, we find that training becomes highly unstable, often leading to NNs failing to achieve 100% validation accuracy for all seed values. Therefore, when training and pruning the NNs for other function graphs, we explore the training regime of high learning rates combined with high batch sizes.

I Unit clustering algorithm

I.1 Clustering algorithm hyper-parameter experiments

In this section, we discuss the hyperparameters used in the proposed unit clustering algorithm. These hyperparameters include the distance measure for clustering units, the linkage method in Agglomerative clustering, and the criteria for selecting the optimal number of clusters. The primary criterion for the algorithm's usefulness is its ability to uncover modules and a hierarchical structure that resembles the hierarchical modularity of the function when using a fixed set of hyperparameters.

We consider the four function graphs mentioned in the main part of the paper, which include graphs with input separable sub-functions, a reused sub-function, both input separable and reused subfunctions, and a non-modular function graph. We train and prune a wide range of NN architectures for each function graph and pass the resulting sparse NNs through all combinations of the aforementioned

0.005

0.01

0.05

0.1

hyperparameters. We then select the set of hyperparameters that successfully uncover the modular structure for the maximum number of NNs and function graphs.

We employ the Agglomerative clustering method on the feature vectors, initially assigning each unit to its own cluster. In each iteration, two clusters are merged into one, and this process continues until all clusters are merged into a single cluster. The specific method used to select the two clusters to merge in each iteration is known as the linkage method. We experiment with four types of linkage methods: average, single, complete, and ward. Average linkage selects the two clusters to merge based on the minimum average pairwise distance between unit features in the two clusters, while single uses the minimum pairwise distance and complete uses the maximum pairwise distance. Ward linkage selects the two clusters to merge based on the cluster variance of the newly formed cluster.

Agglomerative clustering produces multiple cluster assignments depending on the number of clusters. Each iteration of the clustering method yields a cluster assignment with a different number of clusters, ranging from k = 1 to N , where N is the number of units. To determine the optimal number of clusters, we experiment with scoring metrics and select the k value with the best cluster quality at each iteration. Specifically, we explore four different metrics: 1. Silhouette Index [55], 2. CH-index [56], 3. DB-index [57], and 4. Modularity metric (modified).

From Table 4, we observe that all clustering linkage methods combined with either the modularity metric or Silhouette index perform well. Therefore, we choose the combination of average linkage, cosine distances, and the modularity metric. We discard Euclidean distance as the feature vectors are binary, and we select cosine distance over Jaccard distance to ensure appropriate clustering even in the presence of noise in the feature vectors. Additionally, we prefer average linkage over complete and single linkage methods to enhance the clustering method's robustness to noise.

I.2 Edge cases

In the previous sub-section, we explored different metrics for determining the optimal number of clusters in the Agglomerative clustering algorithm. However, three out of the four metrics are not applicable when the number of clusters is equal to the number of units or when there is only one cluster. Unfortunately, even the modified modularity metric fails to detect these cases accurately.

Let ˜ D be the normalized distance matrix of the unit feature vectors, where the sum of all elements is equal to 1 . Consider the units partitioned into k clusters and a matrix A ∈ R k × k , where A ij = ∑ a ∈ C i ,b ∈ C j ˜ D ab represents the sum of the distances between all pairs of units in clusters C i and C j . The modularity metric, denoted by M , measures the quality of the partitioning and is defined as:

The first term in this equation measures the total intra-cluster distance between the unit features while the second term is the expected intra-cluster distance under the null hypothesis that unit distances were randomly assigned based on joint probabilities in ˜ D . A negative value of M indicates that pair-wise distance within a cluster is lower than the random baseline.

Graph structures for edge cases: The units within a given layer must be in the same cluster if the majority of their outgoing connections lead to the same units in the subsequent layers. That is, if the feature vectors are dense or if they are sparse but have low pairwise distances. On the other hand, the units must belong separate clusters if their outgoing connections predominantly target different units in the later layers. That is, if the unit feature vectors are sparse and have high pairwise cosine distances. Surprisingly, despite being at opposite ends of the clustering spectrum, these two cases can exhibit highly similar graph structures. In fact, both structures may closely resemble randomly shuffled graphs with the same number of edges as the original graph.

Behavior of the modularity metric: The modularity metric often fails to distinguish between the previously discussed graph structures and struggles to detect them accurately. Specifically, when there is only a single cluster ( k = 1 ), the modularity metric always yields a value of 0, irrespective of the pairwise cosine distances. Similarly, when k = N , the modularity metric values ( M ) are always less than or equal to 0, regardless of the pairwise cosine distances. Consequently, even a slightly sparse random graph can result in negative modularity metric values for k = N (as well as other k values).

For instance, consider the function graph, neural network, and modularity metric values depicted in Figure 38. In the input layer, the connectivity or unit feature vectors exhibit slight sparsity. Despite the clear expectation that all input units should belong to a single cluster, the modularity metric value is the lowest for k = 4 = N .

Reused sub-function

X1

X2

X3

X4

Modularity[1, 2]: Arch[4, 24, 24, 24, 16]: Density = 2.02%

Ir = 0.1, batch size =8, epochs =40, seed =0, arch =[4, 3, 3, 2, 16]

Pu =70.0, P。 =2.5

Inputseparable andreused

X1

X2

X3

X4

X5

9X

Modularity[3,3]:Arch[6,24,24,24,3]:Density =1.32%

Ir = 0.05, batch size =32, epochs =15, seed =0, arch =[6, 3, 3,3, 3]

Pu =50.0, Pe=1.5

X1

X2

X3

X4

]==== =

Pu =20.0, Pe =1.5

In Figure 39, we can observe that in layers 2, 3, and 4, the modularity metric value is lowest for k = 3 = N , which aligns with the ground truth. However, due to our previous observation regarding k = N consistently yielding the lowest modularity metric value, regardless of the ground truth being k = 1 , this becomes an unreliable outcome. The fact that k = N always generates negative modularity metric values undermines its effectiveness in detecting the case where k = N .

Lastly, Figure 40 illustrates that if the neural network is densely connected, the modularity metric values for all k are 0, further complicating the search for an exact solution. As a result of these observations, we have developed an additional tool to identify the edge cases of k = 1 and k = N .

Unit separability test: The unit separability test is designed to evaluate whether two units in a cluster can be separated into sub-clusters. Consider two units i and j , with o i = ∑ f i , o j = ∑ f j neighbors respectively, and o ij = f i glyph[circledot] f j common neighbors. We consider a random baseline that preserves o i and o j . The number of common neighbors is modeled as a binomial random variable with g trials and probability of success p = o i × o j g 2 , where g is the total number of units in the later layer. The units are separable if the observed value of o ij is less than the expected value E ( o ij ) under the random model.

We can observe in the previous examples that if those edge cases exist in the ground truth, usually k = 2 or k = N has the lowest modularity metric among k = 2 , ..., N .

Test 1: Consider the partition where N -1 clusters are obtained. If the two units that are found in the same cluster are separable, it implies that all units belong to separate clusters.

Test 2: Let us consider the partition of units into two clusters. We merge the feature vectors of the two unit groups. If the two groups of units are not separable, it implies that all units must belong to the same cluster.

In some cases, both tests yield positive results as the graph structures are similar (discussed above). Wedetermine the optimal number of clusters by selecting the result that is more statistically significant. The detailed pseudo code for unit clustering is shown in algorithm 3.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhen conventionally trained, neural networks do not demonstrate structural properties like input separable functions or reusability of sub-modules. The authors investigated this phenomenon and proposed iterative pruning to enhance structural properties of neural networks. They demonstrate the effectiveness of their method on boolean functions and a modified MNIST dataset.\n\nSTRENGTHS:\n* The paper is well-written with nice motivations and clear logical flow. \n* The paper investigates this interesting scientific question \"whether conventionally trained neural networks display structural properties\", which is an important question for network interpretability\n* The paper proposes iterative pruning to enhance structural properties, which is a technical contribution.\n\nWEAKNESSES:\n* The scope is a bit limited. The paper only discusses two properties, input separable functions and reuse of sub-modules. The examples are a bit too simple. Said so, maybe this is not too much of a problem for a scientific paper whose goal is to understand something. Still, I'd love to see your method applied to larger-scale experiments.\n* The novelty is not very clear to me. I'm glad to see that iterative pruning works pretty well for module reuse, but it seems there is not too many technical contribution. I suggest authors should highlight the technical contributions/comparisons to previous works.\n* Regarding the method, I feel encouraging modularity directly in training (e.g., adding Eq. (2) as a penalty in training) may further enhance structural properties. I'd love to see how this trick changes the outcomes (especially for the failed cases).\n\nQUESTIONS:\n* Figure 3 and Figure 4 seem to have the exact same titles and captions. Should distinguish them more clearly.\n* I'm not very sure about your definition of reusability (your definition sounds more like \"shared features\" to me). An example I would count as reuse of submodule is: consider input (x1, x2), output ((x1-x2)^2, (x1+x2)^2), the squared function should be learned twice (reused). However, there is probably no shared feature in a trained network. A trained network (e.g., a fully-connected network) can only learn the squared function twice and independently, even with your pruning strategy I guess.\n* Do you expect your method to generalize to larger models, e.g., large language models?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper conducts an investigation of hierarchical modularity in neural networks by studying boolean networks. The paper studies simple hierarchically modular boolean functions and learns them using MLPs. It then propose metrics to discover this modularity by examining 1) input separability and 2) reusability of sub-functions. Finally, the paper provides a clustering based method to identify the modules in neural networks applied to arbitrary tasks.

STRENGTHS:
1) Clear and scientifically principled investigation into an important yet understudied facet of neural networks 

2) Interesting results showing that hierarchical modularity in neural networks often doesn't emerge with standard training but with pruning (both edge and neuron), the sparsity forces the networks to become hierarchically modular. 

3) Novel method to find  with some empirical backing

WEAKNESSES:
1) The experiments on more realistic datasets e.g. MNIST can be expanded to link the findings of the paper more concretely to what is observed in practice.

QUESTIONS:
Currently, the experiments on MNIST to verify the method to discover modularity do serve as a proof of concept but more extensive experiments would be needed to confirm the effectiveness of the proposed approach to identify modules in neural networks in general. 

In particular, extensions to CNNs on more realistic image datasets might be very interesting. It might be useful to conduct experiments with a subset of classes from a dataset like CIFAR100 or ImageNet where the superclasses and subclasses often offer some natural opportunities for hierarchical modularity.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a methodology for uncovering hierarchical modularity in neural networks (NNs). It combines iterative pruning and network analysis to reveal the underlying hierarchy of sub-functions in tasks. The paper demonstrates the effectiveness of the method on both Boolean functions and vision tasks using the MNIST dataset. \n\nThe main contribution of the paper lies in providing a novel approach to uncover hierarchical modularity without prior knowledge of the task's hierarchy. \n\nThe methodology offers insights into efficient and interpretable learning systems and showcases the potential of pruning and network analysis methods in revealing and utilizing structural properties in NNs.\n\nSTRENGTHS:\nThe paper demonstrates a significant strength through its comprehensive experimental evaluation, encompassing modular and hierarchical Boolean function graphs, as well as tasks utilizing the MNIST digits dataset. The authors meticulously conduct numerous trials, systematically varying network parameters like depth, width, and seed values to thoroughly validate the efficacy of their proposed methodology. The experimental results substantiate the approach's ability to precisely uncover the hierarchical and modular structures within the tasks, serving as empirical evidence of the methodology's robustness and broad applicability.\n\nWEAKNESSES:\nOne potential weakness of the paper is the limited discussion and analysis of the results regarding the failures or limitations of the proposed methodology. While the experiments highlight the success rates in detecting modules and uncovering hierarchical structures, there is less exploration of cases where the methodology might not perform as well or situations where the detected modules do not align perfectly with the expected sub-functions. A more in-depth analysis of the challenges and limitations of the approach could provide valuable insights for further improvement and understanding.\n\nQUESTIONS:\n1. How does the proposed iterative pruning process affect the overall performance of the neural networks in terms of learning efficiency and accuracy?\n2. Are there any limitations or challenges encountered when applying the proposed methodology to more complex tasks beyond Boolean functions and MNIST digit classification?\n3. Could the approach be extended to tasks with dynamic or evolving hierarchical structures, where the sub-functions change over time?\n4. How does the proposed methodology compare to existing approaches in terms of accuracy, efficiency, and scalability when uncovering hierarchical modularity in neural networks?\n\nLIMITATIONS:\nMinor comments: \n- Figure 5B1 -> the text is hard to read on a printed paper, due to small fonts in the figure. This is a minor comment, because if one is reading the PDF paperlessly, one can zoom in.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}