SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we conduct experiments to investigate the impact of learning rates and batch sizes on the structures obtained through pruning. Previous studies have examined the roles of these hyper-parameters in terms of convergence, generalization, and the ability of NNs to find global optima [51, 52]. However, most of these studies train NNs for a fixed number of epochs or iterations, which may not be suitable for our experiment as we require the original dense NNs to learn the task effectively.
To determine the appropriate number of epochs, we aim for the NNs with the four considered seed values to achieve  $100\%$  validation accuracy. We set a maximum of 120 epochs to avoid excessively long training or pruning times. Consequently, some combinations of learning rates and batch sizes do not achieve  $100\%$  validation accuracy. These combinations typically fall into two categories: 1) high learning rates and low batch sizes leading to unstable training, and 2) low learning rates and high batch sizes resulting in an insufficient number of training iterations to achieve  $100\%$  validation accuracy. Such cases are marked with an  $\times$  in the result tables.
We validate and tune hyperparameters for our pipeline using four function graphs shown in Figure 37. These graphs include input separable sub-functions, reused sub-functions, a combination of input separable and reused sub-functions, and a non-modular function graph. We then train and prune NNs with a width of 24 and varying depths, reporting the success rates for detecting the same hierarchical structure as the function graph.
Table 3: Success rates for NNs with varying learning rate and batch size, used for training on function graph with a. input separable and reused sub-functions, and b. a single sub-function (non-modular graph).
lr	0.005	0.01	0.05	0.1
batch size
NNs with 1 Hidden Layer
1	0.75	1.0	1.0	x
2	0.75	0.75	1.0	1.0
4	0.75	0.5	1.0	1.0
8	0.5	0.75	1.0	1.0
16	0.75	1.0	1.0	1.0
32	0.75	1.0	1.0	1.0
64	x	0.75	1.0	1.0
lr	0.005	0.01	0.05	0.1
batch size	Single
1	1.0	1.0	1.0	1.0
2	1.0	1.0	1.0	1.0
4	1.0	1.0	1.0	1.0
8	X	1.0	1.0	1.0
16	X	X	1.0	1.0
NNs with 2 Hidden Layers
1	0.75	1.0	0.25	x
2	1.0	1.0	0.25	x
4	0.75	1.0	0.0	x
8	1.0	1.0	1.0	x
16	1.0	1.0	1.0	0.25
32	1.0	1.0	1.0	0.5
64	1.0	0.75	1.0	1.0
NNs with 2 Hidden Layers
1	1.0	1.0	1.0	x
2	0.75	0.75	1.0	1.0
4	1.0	0.75	1.0	1.0
8	0.75	1.0	1.0	1.0
16	1.0	1.0	1.0	1.0
NNs with 3 Hidden Layers
1	1.0	1.0	X	X
2	0.75	1.0	X	X
4	1.0	0.75	0.5	X
8	0.75	1.0	0.75	0.25
16	0.5	0.75	0.5	0.5
32	0.5	0.75	1.0	0.5
64	0.5	0.75	1.0	0.5
NNs with 3 Hidden Layers
1	0.25	1.0	1.0	x
2	0.5	0.75	0.5	x
4	0.75	1.0	1.0	x
8	1.0	1.0	1.0	1.0
16	1.0	0.75	0.75	1.0
Tables 2 and 3 present the success rates for the four different function graphs considered. We observe that higher learning rates generally lead to more successful trials across all function graphs and NN depths. This aligns with previous findings indicating that higher learning rates result in more globally optimal solutions [53, 54]. On the other hand, lower learning rates tend to under-prune the NNs, resulting in inefficient hierarchical structures in terms of NN density.
When using high learning rates and low batch sizes, we find that training becomes highly unstable, often leading to NNs failing to achieve  $100\%$  validation accuracy for all seed values. Therefore, when training and pruning the NNs for other function graphs, we explore the training regime of high learning rates combined with high batch sizes.
I Unit clustering algorithm
I.1 Clustering algorithm hyper-parameter experiments
In this section, we discuss the hyperparameters used in the proposed unit clustering algorithm. These hyperparameters include the distance measure for clustering units, the linkage method in Agglomerative clustering, and the criteria for selecting the optimal number of clusters. The primary criterion for the algorithm's usefulness is its ability to uncover modules and a hierarchical structure that resembles the hierarchical modularity of the function when using a fixed set of hyperparameters.
We consider the four function graphs mentioned in the main part of the paper, which include graphs with input separable sub-functions, a reused sub-function, both input separable and reused sub-functions, and a non-modular function graph. We train and prune a wide range of NN architectures for each function graph and pass the resulting sparse NNs through all combinations of the aforementioned
Algorithm 3: Unit clustering algorithm for a given layer  $l$
Data:  $f_{i}^{l}, i = 1, \dots, N_{l}, t_{m}$  (modularity metric threshold); /* Given the feature vectors */
*/
Result: Optimal number of clusters,  $K_{l}$ $k \gets 2$ ; /* Enumerate through different cluster assignments */
while  $k < N_{l}$  do
\[ C_{k} \gets Agglo(f_{l}, k); \quad /* Agglomerative clustering dendogram cut giving  $k$  clusters */
\[ s_{k} \gets Metric(C_{k}); \quad /* Compute the metric to asses clustering quality */
if  $s_{k} \leq s$  then
\[ K_{l} \gets k; \quad /*Find the number of clusters with lowest score */
s \gets s_{k}; \quad end \]
end
end
Zsin  $\leftarrow 0$ ;
Zsep  $\leftarrow 0$ ;
if  $K_{l} == 2$  or  $K_{l} == N_{l} - 1$  or  $s > t_{m}$  then
\[ Zsin \gets test 2; \quad /*Run test for single cluster detection */
Zsep \gets test 1; \quad /*Run test for separate clusters detection */
end\]
if  $Zsin > 0$  or  $Zsep > 0$  then
if  $Zsin \geq Zsep$  then
|  $K_{l} \gets 1$ ; /* If either one of the test result is positive */
else
|  $K_{l} \gets N_{l}$ ;
end
hyperparameters. We then select the set of hyperparameters that successfully uncover the modular structure for the maximum number of NNs and function graphs.
We employ the Agglomerative clustering method on the feature vectors, initially assigning each unit to its own cluster. In each iteration, two clusters are merged into one, and this process continues until all clusters are merged into a single cluster. The specific method used to select the two clusters to merge in each iteration is known as the linkage method. We experiment with four types of linkage methods: average, single, complete, and ward. Average linkage selects the two clusters to merge based on the minimum average pairwise distance between unit features in the two clusters, while single uses the minimum pairwise distance and complete uses the maximum pairwise distance. Ward linkage selects the two clusters to merge based on the cluster variance of the newly formed cluster.
Agglomerative clustering produces multiple cluster assignments depending on the number of clusters. Each iteration of the clustering method yields a cluster assignment with a different number of clusters, ranging from  $k = 1$  to  $N$ , where  $N$  is the number of units. To determine the optimal number of clusters, we experiment with scoring metrics and select the  $k$  value with the best cluster quality at each iteration. Specifically, we explore four different metrics: 1. Silhouette Index [55], 2. CH-index [56], 3. DB-index [57], and 4. Modularity metric (modified).
From Table 4, we observe that all clustering linkage methods combined with either the modularity metric or Silhouette index perform well. Therefore, we choose the combination of average linkage, cosine distances, and the modularity metric. We discard Euclidean distance as the feature vectors are binary, and we select cosine distance over Jaccard distance to ensure appropriate clustering even in the presence of noise in the feature vectors. Additionally, we prefer average linkage over complete and single linkage methods to enhance the clustering method's robustness to noise.
Table 4: Success rates for different clustering algorithm hyper-parameters on 4 function graphs and NNs with varying widths and depths (Figure 37)
Clustering Linkage	Distance Measure	Modularity metric	Silhouette index	CH-Index	DB-Index
Feature vector step size s = L
Complete	Jaccard	0.99	0.99	0.49	0.49
Single	Jaccard	0.99	0.99	0.49	0.49
Average	Jaccard	0.99	0.99	0.49	0.49
Complete	Cosine	0.99	0.99	0.49	0.49
Single	Cosine	0.99	0.99	0.49	0.49
Average	Cosine	0.99	0.99	0.49	0.49
Complete	Euclidean	0.99	0.99	0.49	0.49
Single	Euclidean	0.99	0.99	0.49	0.49
Average	Euclidean	0.99	0.99	0.49	0.49
Ward	Euclidean	0.99	0.99	0.49	0.49
I.2 Edge cases
In the previous sub-section, we explored different metrics for determining the optimal number of clusters in the Agglomerative clustering algorithm. However, three out of the four metrics are not applicable when the number of clusters is equal to the number of units or when there is only one cluster. Unfortunately, even the modified modularity metric fails to detect these cases accurately.
$$
M = \sum_ {i = 1} ^ {k} \left(A _ {i i} - \left[ \sum_ {j = 1} ^ {k} A _ {i j} \right] ^ {2}\right) \tag {3}
$$
The first term in this equation measures the total intra-cluster distance between the unit features while the second term is the expected intra-cluster distance under the null hypothesis that unit distances were randomly assigned based on joint probabilities in  $\tilde{D}$ . A negative value of  $M$  indicates that pair-wise distance within a cluster is lower than the random baseline.
Graph structures for edge cases: The units within a given layer must be in the same cluster if the majority of their outgoing connections lead to the same units in the subsequent layers. That is, if the feature vectors are dense or if they are sparse but have low pairwise distances. On the other hand, the units must belong separate clusters if their outgoing connections predominantly target different units in the later layers. That is, if the unit feature vectors are sparse and have high pairwise cosine distances. Surprisingly, despite being at opposite ends of the clustering spectrum, these two cases can exhibit highly similar graph structures. In fact, both structures may closely resemble randomly shuffled graphs with the same number of edges as the original graph.
Behavior of the modularity metric: The modularity metric often fails to distinguish between the previously discussed graph structures and struggles to detect them accurately. Specifically, when there is only a single cluster ( $k = 1$ ), the modularity metric always yields a value of 0, irrespective of the pairwise cosine distances. Similarly, when  $k = N$ , the modularity metric values ( $M$ ) are always less than or equal to 0, regardless of the pairwise cosine distances. Consequently, even a slightly sparse random graph can result in negative modularity metric values for  $k = N$  (as well as other  $k$  values).
For instance, consider the function graph, neural network, and modularity metric values depicted in Figure 38. In the input layer, the connectivity or unit feature vectors exhibit slight sparsity. Despite the clear expectation that all input units should belong to a single cluster, the modularity metric value is the lowest for  $k = 4 = N$ .
Reused sub-function
Figure 38: Row 1: a. Function graph with a reused sub-function, b. hierarchically modular structure detected through a NN with 3 hidden layers and width of 24. ;Row 2: Modularity metric values for various layers and varying number of clusters  $k$
Modularity[1, 2]: Arch[4, 24, 24, 24, 16]: Density = 2.02%
k	1	2	3	4
Layer
1	0.0	-0.16	-0.24	-0.26
2	0.0	-0.5	-0.375	X
3	0.0	-0.5	-0.375	X
4	0.0	0.5	X	X
Figure 39: Row 1: a. Function graph with 3 input separable and reused sub-functions, b. hierarchically modular structure detected through a NN with 3 hidden layers and width of 24. ;Row 2: Modularity metric values for various layers and varying number of clusters  $k$
Input separable and reused
Modularity[3, 3]: Arch[6, 24, 24, 24, 3]: Density = 1.32%
k	1	2	3	4	5	6
Layer
1	0.0	-0.22	-0.33	-0.28	-0.22	-0.17
2	0.0	-0.22	-0.33	X	X	X
3	0.0	-0.22	-0.33	X	X	X
4	0.0	-0.22	-0.33	X	X	X
Figure 40: Row 1: a. Function graph with a single densely connected sub-function, b. hierarchically modular structure detected through a NN with 3 hidden layers and width of 24. ;Row 2: Modularity metric values for various layers and varying number of clusters  $k$
Single sub-function
Modularity[1]: Arch[4, 24, 24, 24, 4]: Density = 1.26%
1	0.0	0.0	0.0	0.0
2	0.0	0.0	0.0	X
3	0.0	X	X	X
4	0.0	X	X	X
In Figure 39, we can observe that in layers 2, 3, and 4, the modularity metric value is lowest for  $k = 3 = N$ , which aligns with the ground truth. However, due to our previous observation regarding  $k = N$  consistently yielding the lowest modularity metric value, regardless of the ground truth being  $k = 1$ , this becomes an unreliable outcome. The fact that  $k = N$  always generates negative modularity metric values undermines its effectiveness in detecting the case where  $k = N$ .
Lastly, Figure 40 illustrates that if the neural network is densely connected, the modularity metric values for all  $k$  are 0, further complicating the search for an exact solution. As a result of these observations, we have developed an additional tool to identify the edge cases of  $k = 1$  and  $k = N$ .
Unit separability test: The unit separability test is designed to evaluate whether two units in a cluster can be separated into sub-clusters. Consider two units  $i$  and  $j$ , with  $o_i = \sum f_i$ ,  $o_j = \sum f_j$  neighbors respectively, and  $o_{ij} = f_i \odot f_j$  common neighbors. We consider a random baseline that preserves  $o_i$  and  $o_j$ . The number of common neighbors is modeled as a binomial random variable with  $g$  trials and probability of success  $p = \frac{o_i \times o_j}{g^2}$ , where  $g$  is the total number of units in the later layer. The units are separable if the observed value of  $o_{ij}$  is less than the expected value  $\mathbb{E}(o_{ij})$  under the random model.
We can observe in the previous examples that if those edge cases exist in the ground truth, usually  $k = 2$  or  $k = N$  has the lowest modularity metric among  $k = 2, \ldots, N$ .
Test 1: Consider the partition where  $N - 1$  clusters are obtained. If the two units that are found in the same cluster are separable, it implies that all units belong to separate clusters.
Test 2: Let us consider the partition of units into two clusters. We merge the feature vectors of the two unit groups. If the two groups of units are not separable, it implies that all units must belong to the same cluster.
In some cases, both tests yield positive results as the graph structures are similar (discussed above). We determine the optimal number of clusters by selecting the result that is more statistically significant. The detailed pseudo code for unit clustering is shown in algorithm 3.